I was of a similar opinion but the more news filters through it seems like there are a lot of organised Russian gangs looking for trouble and a contingent of Marseille Ultras. On the flip side though it always seems like that minority of England fans get "innocently" caught up in this nonsense time and time again.
Joey Barton was on the radio before talking about the difficulty of getting to the ground without problems. Apparently he was part of a small group with a kid in tow and they had to stick to the back streets and even then there was a hairy moment or two with groups actively targeting England fans regardless of what they were up to.
Unfortunately, we seem to have that reputation which attracts the wrong element of opposing fans. And just as unfortunately we have a good few followers who seem to think upholding that rep is a good thing.