So YouTube is all about retention right? Well, I think of that mostly on the video level but I'm now hearing that your entire channel possibly has a rank and a big part of that rank is your average retention.
Audience retention is important for an individual video but for the channel, YouTube looks more at the over-all watch time of your channel.
I wouldn't recommend deleting an older video that bothers you but rather making it unlisted. Yes, your over-all view-count will drop because that video isn't live but if that video isn't performing
YouTube SEO is not about retention (common myth). SEO is based on watch time, and in particular, per session watch time, which means that even if you get poor retention on a video, it can be rated highly if viewers stay on your channel and move onto other videos. Do not get rid of poor performing videos, as that will affect your watch time.
So, as you can see I didn't have to remove my lower-performing videos for "Kid Candy Review" to take off. Those views are spit in the ocean compared to the bigger videos we have.