People never seem to get this so I'll repeat it: The amount of memory only impacts performance when you run out. Up until you saturate 4 gigs there will be no difference between having 4 or 400 GB of system memory. You may see minor variations due to timings of individual units of memory or dual vs single channel configurations, but there's zero advantage to extra empty memory from a performance perspective.
Whether or not you need more RAM is determined entirely by what your normal usage consumes, so do some experimentation and figure it out. Remember that some of the system memory will be used as VRAM so if you're right up against 4GB it would be advisable to go with 8.
Honestly this sounds more like you went from a system with a spinning HDD to an SSD.
I'd agree with you about going by one's "normal" usage patterns, except here we're talking about 1) a very new kind of system that does more than the typical notebook, so past experience may not predict future success, and all that, and 2) the RAM isn't upgradable (natch) and so one is stuck with one's initial purchase.
It's for this reason that I strongly recommend getting 8GB, if you can at all afford it and if you plan on using a machine for more than a year. If you're really strapped for cash, or if you plan to upgrade a machine relatively quickly, then 4GB can certainly suffice--at worst, you'll have to carefully manage your machine for best results, as others have mentioned.
But otherwise, I've always found it better to have something and not need it, then need it and not be able to get it.