If you say you felt good about the shot, especially after confirming your placement on the doe, I believe you. I have no experience with the 200 grain FTX, but like karl, everything I've read about it is mixed. Really mixed. Some guys say it won't open well except at really high velocities, some say it fragments severely no matter the velocity. I don't think any handgun bullet has such a wide range of opinions on it, not even the XTPs. I'd lean more towards the bullet's performance, or lack thereof, being a factor.
I don't think there is one easy answer to your problem. It is apparent something needs to change, however.
On the subject of the 460, I think you ought to try another bullet. As mentioned by the others, the 200 grain Barnes XPB, which performs similarly to the FTX ballistically. I don't know whether you reload or not, but I think Cor-Bon has some factory loads with this bullet. Yes, it's expensive even just as a reloading component, but in my experience it is pretty accurate, sub-MOA even:
If you find them a bit too expensive, only use them to develop you load and/or sight your gun in with the loads, then use the FTX to practice with during the off season. They hit fairly similarly at 100 yards.
Option #2 if you reload would be to develop a good load with your 30-30. . . guessing it's a Contender? Can't give you any advice here, but there's tons of information in this forum and plenty of guys with experience using a 30-30 handgun. Pretty sure it's capable of killing reliably at 150 yards and beyond with the right load.
One last thing you might try just to ensure it wasn't you would be to try to replicate the shot you made. Set up a blank piece of paper or paper plate (because deer don't have little orange dots on them to aim at) at the same distance as the buck, then set up the same type of shooting rest, wear the same gloves, and try to take a similar amount of time to get on target and fire. See where that first shot goes. Not the best way to determine what happened, but it might be helpful.
