I'm not a pen guru, but the writing experience for me is noticeably better on the new pen, particularly the lack of any visible lag. Beyond that, the new pen cost twice as much, doesn't have a pocket clip, has a much stronger magnet to keep it attached, and has less latency, more granular sensitivity, and supports tilt. The pen tips are NOT interchangeable.
Regarding the nSP vs. SP4, I found three complaints about the SP4 that were fixed in the nSP. The edges are reshaped making it not feel like its cutting into your hands when using in tablet mode for a long time; the pen experience is noticeably better, and the battery life is much better. I'm getting around 8-9 hours of use, which covers my whole work day when I'm out of the office. The i5 is also fanless, so silent, and feels to me like it runs cooler than the SP4.
My experience with the SP4 was that I bought one and had it for the 30 day return period which ended a few days before the nSP was announced... so I didn't use it for a really long time, but put it through its paces pretty well.
One other thing to note is there are some folks who are reporting issues with the nSP with the screen quality and pen issues. I've not had any problems with either of these and am very happy with the upgrade.