Just a couple questions. How much time do you have shooting single actions? Also how much time do you have shooting large bore guns/
Like some one else stated grip pressure on a single action can make or break group sizes. Spend a lot of time dry firing as if you were shhoting at a target. This will help develop muscle memory and allow you to determine if you are flinching. I cannot over stress making sure the gun is empty. Have someone watch while you are doing this to see if you are flinching.
If you reload start trying various weight bullets with different charges of powder and even different powders. The load that works great in my SBH might not group out of your gun. Each gun is different and does not always group the same load the same.
A trigger job could help as well. One other thing would be having someone else shoot the gun and check the results. I have had to do that before.
Mainly practice, practice, practice.
Good luck on resolving this.
Rick