Personally I wouldn't do them all in one go as replacing them will trigger a re-evaluation process which could really hurt videos that are performing well. I'd recommend starting with just 4-5 videos (probably the least performing ones) and newly published ones to see if you get the improvements you want.
I been updating to better thumbs on my channel. I do a few at a time every few days. I also am playing with new tags. The new thumbs are a huge improvement than the ones
I had and if I think they look good then others may feel the say. So yes its worth it and I have seen improvment. Not huge but a little.
Once you set the branding, it's good to change your thumbnails to reflect the new look Depending on the number of videos you have on your channel, it'd be good to change a handful thumbnails and then monitor those videos and see where you need to go from there! But, don't prioritize this over new content