So far everybody only mentioned containers not formats except vizibilityzero who mentioned the best one to use.
H264 is the best codec to use for video, and YouTube also states this in some of its documentation somewhere.
for audio AAC is the best codec to use, its highly compressed and sounds great. Its pretty much in lamens terms 2x better than M3P so 2x the quality in the same file size or 1/2 the size with the same quality.
The best container to use is subjective, most people should stick with .MP4 but if you want to be special like me use .MKV as it is technically superior and also I read people have the occasional audio sync issue with .MP4 and I never have experienced that using .MKV
Take a look at some of my game capture videos (not the ones that were live streams) you will notice my 480P looks better than most peoples 720P and my 720P looks better than others 1080P and my 1080P is just like the original game and this is AFTER YouTube butchers the heck out of it with a trans-code
I have a very precise encoding technique and process I use that takes much longer than normal but final results are worth it to me. My final videos are H264 (encoded with x264) and AAC (encoded with NeroAAC) in a MKV container.