Doesn't sound like you've shot the .500 enough to feel comfortable with it. Like a few other commenters, I suggest you shoot it more until you build the same level of confidence and then take it on the hunt.

If for some reason, you still don't have the confidence, then the .454 is always there.

I think it's all about knowing you can kill game with the gun you have.