Size matters: many factors contribute to how a rifle handles.
For several years in the '70s my favorite hunting rifle was a superbly accurate Winchester 70A in .270 Win. The 70A was no lightweight, with a rather beefy barrel. With a steel-tube Weaver 1.5-4.5X scope, steel base and rings, the weight with loaded magazine and sling was nearly 10 pounds. It was a bit sluggish and slow for fast shots at bounding whitetails in heavy brush. But on running game in the open it was old death and destruction, swinging smoothly and steadily.
Though the 70A was the rifle I trusted most, I was constantly fiddling with other rifles. One for which I had high hopes was a Remington 600 in .308 Winchester. All up it weighed three pounds less than the .270. I loved its lightweight and compact size. The 600 had a fine trigger, certainly equal to that of the model 70A. With its short, stiff barrel, from the bench this little Remington would just lay them in.
The only problem was, in the field I couldn't hit anything. It was so muzzle-light, so "jittery," it didn't settle down for offhand shots. On running shots I'd swing too fast and shoot ahead. Forty years ago, when synthetic stocks were uncommon, about the only way to make a reasonably-priced lightweight rifle was to use a short, light barrel.
Here were two quality rifles, both very accurate, both with fine triggers, yet in the field one was markedly easier to shoot well with. For a time I thought it was simply a matter of weight. Heavy rifle good, light rifle bad.
I don't always learn quickly, but I do learn. The difference was not so much the weight as the balance. The 70A with its semi-heavy barrel had its balance point about 6" ahead of the trigger. With its short barrel the 600's balance point was less than 4" ahead of the trigger. One was sluggish but steady, the other fast but wobbly.
The best compromise I've found is a rifle whose balance point is about 4 3/4" ahead of the trigger. Within broad limits weight doesn't matter. I have several light rifles weighing around seven pounds field ready, similar to the old Remington 600, but weight reduction was achieved with both stock and barrel. They're easier to shoot offhand because they balance better. At the other end of the scale, even a 9 to 9 1/2-pound rifle handles well if it's balanced right.
My Remington 700 Mountain Rifle in .280 Rem. balances 4 1/4" ahead of the trigger. I would not want it any less. Routing out some wood from under the buttpad would move the balance point forward, though I likely won't bother. On the other hand, a Winchester 70 Classic in .375 H&H balances over 6" ahead of the trigger. On this one I'll likely have the barrel fluted to move the balance point back.
Length Affects Handling
Another overlooked factor in rifle handling is stock's length of pull (LOP), measured from the center of the buttpad to the trigger. Most rifles sold in the U.S. have an lop of 13 1/2", plus or minus a 1/4" or so. Many shooters would be better fitted with shorter stocks. A stock even slightly too long is a miserable affair. It tends to catch on the shoulder, especially when a fast offhand shot is needed.
The average American male, according to web sites I checked, stands about 5'9" in his socks. Personally I'm just a couple of inches above average, with rather long arms. I can handle a 13 1/2" LOP but I wouldn't want it a fraction longer. A custom gunmaker once said about stock length "better an inch too short than a quarter-inch too long."
My wife once owned a rifle (another Remington 600, but in 6mm Rem.) with a stock shortened to a 12 1/2" LOP. I used to borrow it once in a while. The gunmaker was right. It felt a little odd, but I could shoot it. On the other hand I've tested rifles with 14" pulls and found them cumbersome.
If your stock tends to catch when bringing it to the shoulder (and remember to test wearing hunting clothes, not a summer shirt) a shorter LOP may be the answer. If you do have a stock shortened remember to move the scope ahead an equal amount to avoid the risk of "magnum eyebrow."