I agree that it sounds like an issue with the bullet. I prefer a flat point bullet for any big bore round, either a JHP or JSP. In my experience (when used at the proper velocities) they transfer energy on impact better resulting in larger entrance/exit holes and impressive internal damage.

I don't own a .460 but I have used .454 Casulls on several deer. I have been very happy with XTP/Mags. They are designed to expand at velocities from 1100-2100fps. They don't have to be pushed fast to perform. I load the 240gr XTP/Mags at 1650fps and the 300gr XTP/Mags at 1450fps. All the deer I have shot with them (heart/lung shots) died within sight and didn't go far. By contrast the very hard 300gr Sierra JSP loaded at 1400fps acts like a hardcast and resulted in the only .454 Casull deer I shot that made it out of my sight before dying. It traveled about 75 yards shot through both lungs. The internal damage and entrance/exit holes were not as impressive as the XTP/Mags.

I know the 460 gets high velocities but high velocities are not needed to kill deer efficiently. Due to its high velocities the 460 was also sold as a "long range" hunting round. As velocities drop at longer ranges bullets with thick jackets designed for high velocity rounds often fail to perform as well as they do at shorter distances (higher velocities). Because of this I would recommend that you keep your shots on game inside 50-75 yards until you find a bullet that you're happy with and you build confidence in it. When you have a bullet/load you're happy with keep in mind the bullets limitations. Limit your shots to distances that the velocity is still high enough for the bullet to perform properly.