Well you've described
what happens, but not how or why.
OP, without going into too much detail, it's one of:
1)The CTR (click through rate) was lower than it should be for whatever position it was in the right hand side. So the algorithm switched it out and put a better performing video in its place (one getting a higher CTR) Solution- make better thumbs and/or titles (but not clickbait - see number 2)
AND / OR
2)The audience retention was below average compared to other videos in the same niche and of similar length. This is different to number one. A video could be getting lots of clicks, but if people are clicking away very quickly, that's how the algorithm decides it's "not a good video". So the algorithm switched it out and put a better performing video in its place (one getting more view time) Solution 1: make better videos! Solution 2: Make sure the thumb and title accurately represent the content of the video. That's what people wanted when they clicked. If too clickbaity, audience retention will be low.
AND / OR
3)After watching your video, not enough viewers clicked on to another video compared to other videos in the same niche and of similar length. In other words, relative to other videos, yours "broke the chain" more often. Again the algorithm switched it out and put a better performing video in its place (one that "keeps the chain going") Solution: Not a great deal you can do about it but if people aren't clicking on YOUR videos after watching, then you need to work on video quality, outros, and metadata to ensure at least your videos are featuring YOUR videos in the suggested list.
So honestly, the solutions are everything that gets talked about in the forums and in YouTube's official guides: Make good engaging videos, good thumbs, accurate titles and descriptions, tag correctly and make use of outros. And more importantly make sure you do it all better than your competitors. There are only 10 coveted suggested video slots above the fold and only another 9-10 below the fold.
Look objectively at those key points in your big competitors. That's the benchmark. Making good videos isn't enough. They have to be better than the rest on those key points.