You can drive them too fast, but its hard to do. Here is a (now) funny story I'll relay.
In my very earliest days of reloading, many, many moons (30yrs) ago, I used to load Elmer's 22gr 2400 over a (then) heavy 255 LSWC bullet with excellent accuracy in my original 6 1/2" M29.
I ran out of them and substituted Hornady 240 SWAGED SWC under the same load.
recoil increased with each round fired and after the first cylinder full, I checked the barrel and the lands and grooves were almost gone; ie filled with lead!
I had a fix for that, I'd just load a cylinder of jacketed bullets and clean the barrel.
I am so glad the 240 Hornadys lead were extremely soft.
The first shot was a wrist-twister! It was fierce! I learned my lesson x2-never shoot soft bullets fast and never clean a heavy leaded barrel with a jacketed bullet!
The barrel was clean though! No damage was done and I still shoot this 6 1/2" .44 Smith today! The Hornady bullet at .44 Special velocities is very accurate and does not lead, my 285 Hard cast SWC are very fast (1450FPS) from my 10" TC and don't lead at all.
My SW 500 shoots the 440LSWC to 1725FPS with no ill effects and is extremely accurate.
On the other hand, the referenced 240 Hornady and 2400 load used to key-hole from my 10" TC. It was obviously deforming the bullet enough to destabilize it!