How much surface area a film is able to stick to (whether it's a paint film or something like a vinyl film) directly affects the shear strength and peel strength of the bond.

I guess you could think about a primed or sanded surface as a bunch of peaks and valleys. The peaks and valleys are different sizes, with matte paint or primer being much smaller than sanding scratches.

Paint sticks to sanded or primed surfaces because of the greater surface area.
But, paint is a liquid, so it can conform to whatever it comes in contact with, both peaks and valleys. The adhesive on vinyl isn't really liquid, and can't conform the same way, so will instead tend to stay on top of the peaks and not go into the valleys, so it actually uses much less surface area. So in this case with vinyl, because it can't actually use the greater surface area from sanding or a matte finish, you'd want to have the greatest surface area it can use, which is a gloss surface. This is why vinyl is a lot harder to get off after a few good hot days in the sun; the adhesive will get warmer (and more liquid) and will flow more into more of the surface features of whatever it is stuck to, increasing the surface area. It still won't ever get liquid enough to fill in a sanded or primed surface though, so a gloss surface is still better.