There's always a practical lower end limit, & I would say that's probably 357 max or 41 mag with proper bullets. It's true that shot placement is #1, but if we carry that to it's conclusion then we will be condoning the Single Six in .22 LR as adequate if it's what you shoot best :-)
A proper saying in my old street racing/drag racing days was there's no replacement for displacement. I was a big block chevy guy.
I like the larger calibers, but my 480 is loaded relatively mild so I can shoot it well with it's 400gr at around 1100fps. I could push it from there but might not shoot it as well.
So, I guess I believe in shot placement with displacement!