I absolutely think color guard should be an Olympic sport, but the indoor version known as Winter Guard. This is a fantastic sport that involves dance, artistry, choreography and tons of athleticism and skill from team members. There is a scoring system and there are competitions already, so this isn't something that needs to be invented. Yes, it's a little subjective, but they are judged on technique, complexity of routine, and other criteria, similar to figure skating. There are 5 judges (1 for Equipment, 1 for Ensemble, 1 for Movement and 2 for General Effect). Each judge provides a score from 1 to 20, for a maximum total of 100. Typical scores range from 50 to the high 90's.
The judging is not all that different from the floor exercise in gymnastics, figure skating or sports like freestyle skiing and snowboarding, all of which have a technical and artistic element to them. Winter guard also adds a team element - and it isn't just synchronization. Some elements of a routine may feature all team members in unison, but other parts may features some members dancing, some spinning sabre, some spinning flags, and others spinning rifles all at the same time. Anybody who thinks this is a simplistic sport, or doesn't have the complexity of gymnastics doesn't know what they're talking about.
If you want more information on it, check out the Wikipedia article and the WGI (Winter Guard International) site. I have to admit that I'm biased. My daughter is on her high school team. I had never seen it or heard of it before she got involved, and now I'm hooked. These kids put in more practice time than the basketball teams do. The sabres and rifles they spin are heavy and I have seen many bruises and cuts from missed catches - and don't think for a minute that 20 kids tossing a flag in unison is easy. My daughter spins sabre and i can tell you that catching that thing is like sticking your hand in a moving propeller. It's steel, heavy and rotating at a high rate of speed.
Also, if you have never seen winter guard before, you've got to check out some of these videos on YouTube.