I've always been a FRAPS guy, it's the only program I've found that can keep a steady framerate and not lose quality. Sure, file sizes are an issue but there are easy ways around this with minimal quality loss.
In my brief experience with Dxtory, I was bombarded with options and features but performance wise it was meh.