I think in general most folks who would even consider themselves gun enthusiasts to begin with can't shoot a handgun very well at all. Being able to shoot a nice tight group with a handgun takes a lot more practice and skill than doing it with a rifle rested on a bench. From my many visits at the range it's painfully obvious that many have great difficulty shooting handguns with any remote degree of accuracy.

I agree that you have to be a shooter first and that it really does make a big difference.

Many also think that even the biggest of revolvers are just not powerful enough, in their eyes a .44 Mag pumping out 1000 ft-lbs could never be as effective on deer as their 1700 ft-lb .30-30 because all the care about is ft-lbs. There's more to it than that even, but I think the general view of handguns and most peoples inability to shoot them well is a big reason for it.