I've been wearing helmets in all weathers and in many conditions (0 to 60 mph) with lots of different gear for over six years. I wore one part time for almost two years before that. Lots of nasty wipeouts in that time too against water and a couple of notable ones on land. Still here and won't ride without one.
I interviewed some of the best riders in the world, the first time helmets were required in a competition in the USA. This was at the 2003 Velocity Games. Not one competitor from Martin Vari, to Andre Phillips to about a dozen other riders said the helmets impaired their style of riding or ability to compete. This has been repeated in subsequent competitions since.
If you do wear a lid, make sure it fits well (doesn't slip), is comfortable, has low drag without bills or projections and the best padding/shell you can find for our sport.
Some good stuff to look over on this subject appears below:
http://fksa.org/viewtopic.php?t=720
http://fksa.org/viewtopic.php?t=706
http://fksa.org/viewtopic.php?t=494
http://fksa.org/viewtopic.php?t=920
http://fksa.org/viewtopic.php?t=117
http://fksa.org/viewtopic.php?t=210
and many, many more.
Whether you wear a helmet or not is just one more choice. Do you homework, think it over and choose well.