I've used the same Lee .454 set that you have for 15+ years. Only problem I've had is sometimes the seater die doesn't crimp well. I solved that by getting a Lee .45 cal. crimp die. Works great.
I use Lee shell holder for their priming tool and RCBS shell holder for reloading (already had it for .44 Mag.) I use Lee shell holders with some other RCBS dies (different calibers) with never any problems.
I've loaded and shot well over 2500 rounds for my 3 different .454s with no problems EVER by mixing shell holder and dies. I consider the Lee sizing die excellent. I use them in all my straight wall revolver cases except .475 which they don't make.
I've always looked at Lee dies as the Rodney Dangerfield of loading dies- no respect, but decent price and they always work.