Aerodynamics are a big reason, but yes the effect of shaving your legs is not big. It's just one of many aero marginal gains one can achieve for (almost) free. The big one is your riding position, but everything is a factor.
I did a few recon laps of the course on Wednesday, I averaged 21mph...