Helmets.org

Volunteer staff - 100% Consumer-funded

Home Children Materials Statistics Laws Standards
Quick How2Buy Media Teachers About Search


Evaluating Helmet Campaigns



Summary: Campaign evaluations are important to your helmet program and to others who will follow. We are great believers in field observation rather than telephone sampling.




Introduction

One of the worst problems of helmet promotion campaigns is evaluation. Thorough evaluations are necessary for those who come next and need to know about the best practices that produce the best results. But those who have struggled to put together a campaign and see it through are ready to relax, and are already convinced by their experience that they have done good. So the evaluation phase can be neglected, and the lessons learned are lost. Even worse, the people in the next state are denied the ability to promote their campaign or law with hard numbers and demonstrated results. So we have begun to assemble some material so you don't have to reinvent this wheel.

Our approach is firmly oriented toward scientific observation, and toward field observation methods. Anyone who has worked with bicycle riders and surveys knows the futility of asking a rider how far they have ridden recently, or almost anything else about their riding, and expecting an accurate answer. Ask a room full of second-graders if they have bike helmets and if they use them, and a forest of hands will pop up to give you the answer they know you want. Just ask a teacher. The State of Georgia passed a helmet law, and before they passed it they called people and asked if their kids were using helmets. Then after the law was passed they called again and -- surprise! -- more people answered that their kids were obeying the law and using helmets. The only rider surveys we accept are those based on ride logs kept daily by the hardest-core riders, and those are few and far between.

From these assumptions, our bias is toward the field observation method: actual field counts of who is and who is not wearing a helmet. If all other things were equal, careful counts before and after your campaign or before and after your law passed should show whether or not your actions were generating results. All things may not be equal, but at least you have to try.


Don't miss this one!

For starters, we highly recommend the article on Measuring Community Bike Helmet Use Among Children from Public Health Reports (Vol. 116, No. 2, pp.113-121). It details the ins and outs of using various measurement techniques. The authors share our bias toward actual observation to determine helmet use and go on to say that it can be used to evaluate the risk of injury. We don't agree with all of their findings, but If you are doing helmet counts you want to take the time to read this excellent article first.

The first group of materials below comes from a New York State program that produced a helmet evaluation campaign manual in 1990. The original manual was showing its age, so we have replaced it with newer material from the current program contact: The New York State Department of Health, Bureau of Injury Prevention, email: injury@health.state.ny.us Following that is a summation of the points in a recent article by Les Becker et al in Injury Prevention, which discusses how to choose observation points and how to conduct the observations. It is less detailed than the New York materials, but has more information on methods of selecting observation points. Finally, we have added a chart prepared by the Centers for Disease Control summarizing evaluation efforts from several localities.


New York State Material

The New York State Department of Health (NYSDOH), Bureau of Injury Prevention (BIP), Division of Chronic Disease Control and Prevention, supports a statewide bicycle helmet safety project including helmet distributions, education and awareness for the public and medical professionals, and evaluation research. The project has been funded through various grant awards from the National Centers for Disease Control and Prevention (CDC), the New York State Governor's Traffic Safety Committee (GTSC), and the NYS Developmental Disabilities Planning Council (DDPC), as well as from private donations from insurance companies. It is coordinated with statewide legislation requiring helmets for all children under 14 years of age while bicycling or inline skating. The State has developed an evaluation method for this program, which is presented below.


Evaluation Research

In addition to process evaluation, including documentation of the number of helmets distributed, educational presentations conducted, and educational/promotional materials requested, various strategies have been developed to ascertain the effectiveness of the overall project and determine if there has been a change in bicycle helmet ownership and usage. A school-based survey was administered each year between 1989 and 1993 through a sample of Parent Teacher Associations in upstate counties and in a sample of schools in NYC. This survey has documented an increase in ownership and usage of bicycle helmets, most significantly in younger children.

A protocol for conducting an observational study to determine helmet usage has been developed by Bureau of Injury Prevention staff and distributed within NYS and nationally. Pre- and post- law observation studies were conducted statewide to investigate any changes in bicycle helmet usage due to implementation of the bicycle helmet legislation. In addition, the NYSDOH BIP, under cooperative agreement with the CDC, is conducting a bicycle helmet evaluation project to determine which strategy, or combination of strategies, is associated with increasing bicycle helmet usage.


Method for Taking Observations

  • Observational Study Design
  • Form for Recording Observations
  • General Guidelines for Taking Observations
  • Instructions for Observers
  • Slides from Observer Training Course


Observational Study Design

Bicycle helmet observations are conducted in six counties across NYS, three times a year, Spring, Summer and Fall. Each county has two observers who are trained on the proper methods of conducting observations, and each county has three observational sites (school, park, playground, etc.). Observers conduct a combination of stationary and driving observations at the three sites. Most often each observer picks one site each and they share the third site. Stationary observations are usually 30 minutes in duration and driving observations are usually 60 minutes although variations by county/site are possible.

An example of how the observations may be conducted is listed below:


          Site One                 Site Two         Site Three
----------------------------------------------------------------------

Observer 1. 30 min stationary+                     1. 30 min stationary
  A         60 min driving                             + 60 min driving

         2. 30 min stationary +               
            60 min driving		              
                   
----------------------------------------------------------------------

Observer                     1. 30 min stationary  1. 30 min stationary
  B                             + 60 min driving       + 60 min driving

                             2. 30 min stationary               
                                + 60 min driving	             

----------------------------------------------------------------------
Observations are conducted on non-rainy, relatively warm days. For each site, the observers conduct the observations at the same time of day and observe from the same vantage point during the stationary observation and follow the same route for the driving observation (determined ahead of time during site visits).


Bicycle Helmet Observation Form



COUNTY:
PLACE OBSERVED:			Stationary / Driving 
DATE:          TIME: Begin           End 
WEATHER: Overcast / Partly Cloudy / Sunny         Approx. Temp.
OBSERVER'S NAME:
Alone   or  Together	with (name) 

===================================================================

-------------------------------------------------------------------
BOYS	           		Helmet               No Helmet
-------------------------------------------------------------------
            |
            |  10+
            |
            |
            ------------------------------------------------------
            |
            | Under 10
            |            
            |
            ------------------------------------------------------

------------------------------------------------------------------
GIRLS 
-------------------------------------------------------------------
            |
            |  10+
            |
            |
            ------------------------------------------------------
            |
            | Under 10
            |            
            |
            ------------------------------------------------------


===================================================================

Straight line = Correctly worn helmet 
Squiggly line = Incorrectly worn helmet 

Top of box = Sure of age 
Bottom of box = Unsure of age 



General Guidelines for Driving Observations

Very Important: For safety concerns please identify a person, other than yourself, to drive while you are conducting the observations.

This driver can be different every time and no training is required. The driver will be reimbursed at $8.00 per hour. If a driving observation is conducted in association with a stationary observation and the driver accompanies the observer on a stationary observation, the driver's time IS reimbursable. Please be sure the driver fills out the driver reimbursement form.

Mileage is reimbursable at $0.31 per mile while on the driving route only. Mileage to and from the driving route is not reimbursable at this time. Please be sure the owner of the automobile fills out the mileage reimbursement form.

Plan the driving route ahead of time based on your knowledge of the community. If a driving route has been pre-established, please try to follow this route as closely as possible. Please submit a map with the driving route clearly marked for each driving observation, even if the route did not change. Please put the date you conducted the driving observation on the map.

Plan an overall loop where the route begins and ends at the same place.

Focus on residential areas rather than business districts or rural areas. Observe for the time allotted based on your instruction packet. If you finish the route before the allotted time is over, restart the route and continue until you reach the time limit. Mark on the map where you stopped. A substantial overlap or shortfall may require modifications the next time to lengthen/shorten the route.

Do not observe while on highways/main streets where traffic is traveling greater than 35 mph. If you must travel these roads to get from one residential area to another, stop the clock and resume once you get to the next area.

If you encounter a traffic jam or construction delay, stop the clock and resume once you are clear.

Avoid double counting children whenever possible. If restarting a route, try to notice if you are observing the same children. Do not count those you know you saw in the beginning. Only observe as you drive one-way on a dead end street. Observing while coming back out will count the same children seen on the way in. Stop the clock if it is a very long dead end and resume when you get back to the main route.

Do not observe within a 2-3 minute (4-5 block) radius of the stationary site. This should eliminate observing children who are riding to and from the site and who may be counted while you are doing the stationary observation.

If you have any questions, please contact the New York State Department of Health, Bureau of Injury Prevention, email: injury@health.state.ny.us.

THANK YOU FOR TAKING THE TIME TO REVIEW THESE INSTRUCTIONS. YOUR HELP IS GREATLY APPRECIATED!


Instructions: Bicycle Helmet Observational Study

1. Observations should be completed between the pre-determined dates. If you have trouble completing them by the end of the time period, please let us know.

2. Conduct stationary and/or driving observations for each site based on the specific instructions provided in your observation packet. (Note the date and time started and stopped, and whether it was a stationary or driving observation on the observation forms.)

3. Remember to observe for the specified number of minutes only. Stationary observations are usually 30 minutes. The length of time for the driving observation may vary by county/site. Please refer to your observation packet.

4. If you do not observe any riders, please call us immediately. You may be asked to do another observation if your schedule/weather/time permits. You will be paid if you are asked to do this extra observation.

5. Observe on non-rainy, relatively warm days. (Note the weather on the observation forms.)

6. Observe the same type of day (weekday/weekend) and time of day at each site.

7. When no reliability assessments are being conducted, all observations should be completed separately from the other observer. Please be sure to coordinate schedules so you do not inadvertently show up on the same day. (Check the appropriate 'alone' or 'together' space on the observation forms.)

8. Make one mark per child in one box noting presence or absence of a helmet, gender (boy/girl), age (10 and older or under 10).

9. When noting age, if you are sure about the age of the child, mark in the upper half of the box; if you are NOT sure about age, mark in the bottom half of the box.

10. If the child is wearing a helmet, note whether the helmet is used properly by marking a straight line for a correctly worn helmet and a squiggly line for an incorrectly worn helmet. (Diagram shows a correctly worn helmet as level on the head, not tilted back or forward or to the side, with the chin strap fastened snugly, not hanging down way below the chin.)

11. If the child is not wearing a helmet, the mark will always be a straight line. There is no way for a helmet to be worn incorrectly (squiggly line) if the rider is not wearing one! 12. Please return final site maps (if applicable), observation forms and reimbursement forms in the enclosed pre-addressed postage paid envelopes once all observations are completed. Please keep a photo copy or some other record of the observations in case forms get lost in the mail. Please limit number of pages to 4-5 per envelope due to postal weight regulations.

If you have any questions, please contact the New York State Department of Health, Bureau of Injury Prevention, email: injury@health.state.ny.us.

THANK YOU FOR TAKING THE TIME TO REVIEW THESE INSTRUCTIONS. YOUR HELP IS GREATLY APPRECIATED!


Helmet Observation Training

This is based on slides prepared to go with training conducted by Patricia O'Connor, PhD, consultant to the NY State Department of Health.

Slide 1

Purposes of Research: I
  • To Explain
  • To Predict
  • To Intervene

Slide 2

Purposes of Research: II
  • Discover
  • Demonstrate
  • Refute
  • Replicate

Slide 3

Essential Characteristics of Research
  • Reliability
  • Validity

Slide 4

Research Design
  • Methods used to study

Slide 5

Types of Research
  • Surveys
  • Interviews
  • Observations

Slide 6

Observational Studies are Scientific when they:
  • Serve a research purpose
  • Are planned deliberately
  • Are recorded Systematically
  • Are subjected to reliability and validity checks

Slide 7

Now to practice and to establish reliability of observation:
  • Food Court area: Coat on/off, gender, younger/older
  • Outside cars parking: Seat belt used/not, gender, younger/older
  • Mall area: glasses/not, gender, younger/older
  • Mall area: carrying bag/not, gender, younger/older
  • Mall area: climbing stairs: hold rail/not, gender, younger/older

    Slide 8

    Training Observation Form (above)


    Additional Forms

    The New York State material includes forms for driver reimbursement, mileage reimbursement and observer reimbursement. There is a bicycle helmet application form, and methods of calculating income to determine who qualifies for low-income helmet assistance.


    Maryland's Community-Based Approach

    Staffers of the Maryland Department of Health published an article in Injury Prevention, 1996; 2:283-285 by Les R. Becker et al, on experience in the State of Maryland gathering numbers before and after their helmet laws were passed. (The numbers will be available eventually.) Their conclusions:


    Selecting Sites for Observation

    The article details two methods: asking club cyclists where to find the most riders with some map study thrown in, and asking community informants where to find the most young bicycle riders. The second method, using community informants, yielded more cyclists per hour and more observations per dollar spent than asking club cyclists.


    Training Observers and Conducting Observations

    For the Maryland studies, high school and college students were trained as observers a day or two before their sessions. After about 30 minutes of training they were able to identify and number passing cyclists while recording their age group, gender, race and helmet use on a form. Observations were made from vehicles and from fixed points between 9 AM and 4 PM on summer days. The sites selected by knowledgeable community members generated from 50 to 450 percent more observations of riders in the under-16 age group than those selected using a combination of bicycle club member recommendations and studying maps for such features as parks and playgrounds. The study authors were unable to say if the increased size of the sample made the results more representative of the target group.


    Other Evaluations: Results of Some Evaluations of Helmet Laws

    This chart was produced by the Centers for Disease Control in Atlanta, a part of the U.S. Public Health Service. It attempts to summarize the data from some evaluations of bicycle helmet laws.

    Centers for Disease Control
    Evaluation of Legislation and Community Programs
    to Increase the use of Bicycle Helmets - Selected Locations

    Helmet-use rates are for children except for
    Victoria, Australia, which included adults.




    
                                                    Helmet use Rates 
    
                                                   Pre-        Post-
    Location   Years          Program type        Program     Program 
               Evaluated
    
    
    Victoria,  March 1983-    Community campaign  6 percent   36 percent 
    Australia  March 1990
    
               Comments: Included education, mass media publicity, 
               support by professional associations and community groups, 
               involvement of bicycling groups, and $10 government rebate 
               for helmet purchases.
    
    
    
     d.o.      March 1990     Helmet legislation  36 percent  73 percent
               March 1991     introduced
    
               Comments: Hospitalizations for bicycle-related
               head injuries also decreased by 37 per cent,
    
    
    
    Howard     1990-1991      Helmet legislation,  4 percent  47 percent
    County,                    community campaign 
    Maryland    
     
                Comments: Activity prompted by bicycling deaths of two
                children. Use determined by observation. School-based
                survey showed 11 percent and 37 per cent. 
    
    
    
    Montgomery 1990-1991      Community campaign   8 percent  19 percent
    County,
    Maryland 
    
                 Comments: Use determined by observation. Use determined
                  by school-based survey was 8 per cent and 13 per cent
    
    
    
    Baltimore  1990-1991      No specific helmet   19 percent 4 percent
    County,                    promotion activities 
    Maryland
               Comments: Served as control county. Use determined by
               observation. Use determined by school-based survey was 
               7 percent and 11 percent.
    
     
    
    
    Seattle,   1987-1988      Community campaign   5 percent  14 percent
    Washington
    
               Comments: Included education of parents by physicians;
               advertising in newspapers, on television, and on radio; 
               school presentations; and discount coupons for helmets.
    
    
    
    d.o.       1980-1990      Community campaign   14 percent 33 percent
    
               Comments: Follow-up evaluation of bicycle helmet campaign.
    
    
    
    d.o.       1990-1993      Community campaign   33 percent 60 percent
    
               Comments: Follow-up evaluation of bicycle helmet campaign.
               Bicycle-related head Injuries decreased approximately 67
               per cent among children 5-14 years of age who were members
               of a health maintenance organization.
    
    
    
    Portland,  1987-1988      No specific helmet   1 percent  4 percent
    Oregon                     promotion activities
    
               Comments: Control community, use determined by observation.
    
    
    
    
    Barrie,    1988-1989      Educational program  0 percent  0 percent
    Ontario
    Canada
    
               Comments: Use determined by limited number of observations.
     
    
    
    d.o.       1980-1989      Educational program  0 percent  22 percent
                               and helmet subsidy
     
               Comments: Use determined by limited number of observations.
    
    
    

    Source:
    Injury-Control Recommendations: Bicycle Helmets
    Morbidity and Mortality Weekly Report, Vol 44, No. RR-l February 17, 1995
    Centers for Disease Control and Prevention (CDC)
    U. S. Public Health Service, Department of Health and Human Services

    More Studies

    The best list of studies and references on educational campaign evaluations is the page titled "Bicycle Injury Interventions Programs to Increase Helmet Use: Education" now on archive.org and somewhat outdated, but originally from the Harborview Injury Prevention Center site.

    In 2016 a study was published titled "Challenges in the accurate surveillance of booster seat and bicycle helmet usage by children: lessons from the field". The abstract says: "..This paper examined the challenges that confront efforts to collect surveillance data relevant to child traffic safety, including observation, interview, and focus group methods. Strategies to address key challenges in order to improve the efficiency and accuracy of surveillance methods were recommended. The potential for new technology to enhance existing surveillance methods was also explored." The article is available free on the International Journal of Environmental Research and Public Health site.




    This page was reformatted on: October 4, 2017. BHSI logo
  • Home Children Promotions Statistics Laws
    About New Standards Teachers Search