A couple days ago, an article in the British Medical Journal (BMJ) reviewed results from an evaluation of the Young People's Development Programme (YPDP) in England. The underlying evaluation had actually been done a year ago, but results are making headlines and popping up in blogs now due to the recent BMJ article. In short, BMJ reports that there were virtually no positive impacts of the YPDP program and, more surprisingly, some negative impacts--like higher rates of teen pregnancy.
Since then, I've read wide-ranging conclusions about what this all means for the U.S. and our efforts to prevent teen pregnancy--that sex ed doesn't work, that abstinence education works better, or that youth development actually increases risky behaviors (and, by the way, youth development is not synonymous with sex ed).
So, what does the study tell us about U.S. teen pregnancy prevention programs? The results suggest that if a similar, comprehensive set of youth development services (including reproductive health education but also training and employment activities, life skills building, mentoring, sports, arts, volunteer work, and outside referrals), were offered to a similar set of high risk adolescents in a similar setting, at least one but maybe more of those services, either alone or in combination, would have an impact on teen pregnancy rates that was less desirable than the impact from a different set of youth development services that were already being offered in other similar locations. Assuming of course that our programs and our youth are like those in England. Sound complicated? It is.
Don't get me wrong--evaluation studies have a wealth of valuable insights to offer us, and using science-based evidence to guide our teen pregnancy prevention efforts is spot on. But if we truly want to benefit from this science, we need to be willing to fully digest its contents.
The original evaluation report is 100 pages long, and the authors get high marks for using the best statistical rigor at their disposal and discussing the limitations of their findings with great insight. True, the findings say that YPDP led to teen pregnancy rates that were more than two times higher, but the critical question here--if you really want to draw implications from this--is, higher than what? Does this mean that teens who received the YPDP intervention were suddenly twice as likely to get pregnant? No. It means that at-risk teens getting one bucket of services had higher teen pregnancy rates than at-risk teens getting another bucket of services. We don't really know which component(s) of those services made the difference, and we don't really know how either group compares to those who get no services at all. And, these results cannot tell us anything about the relative merits of other programmatic approaches that were not part of the evaluation.
My goal here isn't to support any particular program or approach--just to say that giving science its due means always understanding the counterfactual and the external validity--in other words, what program is being studied, what is it being compared to, and how is it similar or different to the programs that you care about.
I agree that we sometimes look at a study as telling us more that it actually does. This study, however, I hope will lead to an honest review of the program. The program was designed to help the youth and was not successful in their intervention. Further study should be done to isolate and understand which factors had the negative or at least lacked the positive impact they were seeking. Additionally, I would charge us all to use the same intellectual argument with all studies and not be quick to take less rigors with the one's that support our viewpoint. For example, the band wagon that jumped on mathmatica report as the final word that abstinence education doesn't work. I wouldn't expect any program administered once in middle school to have a lasting impact 4 years later into high school. That assumption is absurd, so why did we love the study so much?