«Healthy school meals and Educational Outcomes Michèle Belot Jonathan James Department of Economics Nuffield College University of Essex University ...»
It is important to note that only part of the pupils included in the analysis has truly been treated: those who actually eat school meals and experienced a change in diet because of the campaign. Unfortunately, we do not have individual information about who is eating school meals and who is not – thus our estimates measure the effects of the “intention-to-treat” and are likely to be a lower bound. As we mentioned earlier, 45% of the children eat school meals at school. We now investigate whether we can identify subgroups of the pupil population that have been more affected than others.
Heterogeneous effects First, we investigate whether we find any differences in effects across the free school meal (FSM) eligibility status11. We know from Table 2 that about 36% of children in Greenwich and 40% in the control LEAs are eligible for free school meals. Note that eligibility does not mean take-up or actual consumption of the meal. We have information on the take-up rates for eligible children and we will investigate the effects of the campaign on these take-up rates in the next section, but we do not have information on the take-up rates (or actual consumption) of those who are not eligible for free school meals. Thus, we do not know whether FSM children are more or less likely to have been “treated” by the campaign.
Table 4 presents DD estimates when we split the sample according to the free school meal status. We find that most of the positive significant effects decrease or disappear entirely for the FSM children. Thus, we fail to find evidence that the campaign specifically helped those children who benefit from free school meals. One possible 11 Free school meals eligibility criteria: Parents do not have to pay for school lunches if they receive any of the following: Income support, income-based Jobseeker's Allowance, support under Part VI of the Immigration and Asylum Act 1999, Child Tax Credit, provided they are not entitled to working tax credit and have an Annual income (as assessed by HM Revenue & Customs) that does not exceed £15,575, the Guarantee element of State Pension Credit.
story is that FSM children and parents from these children may be harder to get on board and that children from richer socio-economic backgrounds might be more receptive to the changes in school meals. This is important to point out though, in the context of using this policy as a possible mechanism to reduce disparities across children.
We investigate further whether we find evidence of heterogeneous effects according to gender, race and “special educational needs” status (remaining of Table 4). Again, we have a priori no clear reasons to expect some groups to be more affected than others, because we do not know the distribution of school meal consumption across these groups. We find no clear evidence of heterogeneous effects. Girls seem to have been more affected, but we cannot reject that the effect of the reform was identical across gender. Thus, we cannot conclude that the reform affected some students more than others, except according to their free school meal status.
b) Effects on absenteeism We now turn to the effects of campaign on absenteeism. We have information at the school level on the percentage of authorised and unauthorised absences. Authorised absences are those that are formally pre-authorised by the school, thus most likely linked with sickness. Table 5 shows the results of the DD analysis, both on the percentage of authorised an unauthorised absences. We find a substantial negative effect on authorised absences; the rate of absenteeism drops by about.80 percentage points, which corresponds to 15% of the average rate of absenteeism. On the other hand, we do not find a significant effect on unauthorised absences.
The relative fall in absenteeism could in itself drive part of the improvement in educational outcomes, although obviously only a small part of the population of pupils has presumably been affected by this fall. In Table 6, we compare the results we have presented earlier (in Tables 3, based on the school level data) with results controlling for authorised absenteeism at the school level. We find that the coefficients reported earlier remain very similar. Thus, the effects on educational achievements are not due to the change in absenteeism. However, it could be that for those children for whom absenteeism does change, the improvement in educational achievements is more substantial than for the others. Unfortunately, we are unable to identify those children in the pupil-level data.
c) Effect on take-up rates We now examine the effect of the campaign on the take-up rates of free school meals.
As we mentioned earlier, we do not have information on whether children did indeed consume the meals or not (the anecdotal information we have points that, indeed, children were far from enthusiastic at the beginning but did adjust relatively quickly to the new menus), nor do we have information on the overall take-up rates of school lunches. We do have, however, detailed information at the school level on the percentage of children taking up free school meals (conditional on eligibility).
Changes in take-up rates are important to look at because, obviously, falling take-up rates would jeopardise the success of the campaign. On the other hand, it could be that improvements in the quality of the food encourage take-up.
We report the results in Table 6. We find no evidence of a change in take-up rates.
Obviously, this does not mean that there has been no change in the actual consumption of school meals. As we discussed earlier, the change in menus had not been implemented easily and some children were reluctant to accept the new menus.
At least, these results show that there was no change in the recorded take-up rates.
d) Placebo effect One concern is that the campaign affected educational outcomes not through the improvement in diet, but simply through a “placebo-effect”. Indeed, the schools were very well aware they were part of a pilot experiment and the campaign received a lot of media attention. Thus, we should worry that the effect we measure is a placebo effect rather than an actual effect of the campaign.
We should note that any reform of this kind, that is, where one group of people is treated and another is not, is potentially subject to this placebo effect. In contrast to experiments in medicine for example, it is virtually impossible to think of a way of administering a placebo treatment to a control group. Any change in policy could affect outcomes simply because those who are treated know they are treated. There is usually no way researchers can be sure that the effect they estimate is truly due to the change in policy rather than a placebo effect.
In this particular case, it is not clear whether the effects we find could be driven by such a placebo effect. On the one hand, this campaign has received attention from the media, which possibly could trigger a placebo effect. On the other hand, the attention was very much focused on the health benefits, and in particular on tackling the problem of obesity, rather than improving school performance. Also, we are looking at outcomes more than a year after the campaign and have excluded the year of the campaign itself. It is hard to believe that school children would remain motivated by a placebo effect more than a year after the campaign has been implemented.
Our setting, nonetheless, gives us some scope to investigate the placebo effect to some extent. As the campaign was part of a programme broadcast on one of the major channels in the UK, we have good reasons to believe that some schools were probably more subject to a possible placebo effect than others. Some of the treated schools were explicitly mentioned in the program, such that one could expect that for those schools, the placebo-effect could be stronger than others. However, there were only 7 schools explicitly mentioned in the programme, so we should be careful in interpreting the results, as idiosyncratic changes in one of these schools will weigh more on the estimates.
We have extended the empirical analysis by adding an interaction term for those schools that were explicitly mentioned during the programme (note that some of them were just very briefly mentioned, there was no filming on location). We present the results in Table 7 for English, Maths and Science respectively. The evidence points in the direction of a “disruption effect” rather than a positive placebo effect. In the case of Maths, we find that the interaction coefficient is significant and negative, while we find no positive effect of the campaign overall. For English and Science, the interaction dummy is in most cases negative but is not significant. Additional evidence on this disruption is that there were many initial problems in the schools that took on the scheme early on. Further, as the programme was rolled out, a food week was introduced and tasting sessions for the parents were organised; hence those later schools would have had a slightly different treatment than the early schools.
Since there are only few of these schools, we do not wish to draw too much attention to these estimates, but we conclude that, at least, there is little evidence of a positive placebo effect.
e) Robustness checks To make sure that the effects we have identified are not a statistical coincidence, we run a number of robustness checks. Appendix B presents the results of the robustness analysis on Key Stage 2 scores and Appendix C presents the results of the robustness analysis on absenteeism. First, we conducted a placebo analysis attributing the role of treated successively to each LEA included in the control group (Tables B1-B5 and C1). The results we find are much less consistent. More precisely it is only in Greenwich that we find systematically and consistently positive DD estimates for Key Stage 2 scores and negative DD estimates for absenteeism. We find no such pattern in any of the other LEAs. Second, we conducted a placebo analysis by attributing the year of treatment successively to 2002-2003 and 2003-2004 (Tables B6-B7 and C2).
None of the coefficients are significant when the treatment year is attributed to a placebo year. Finally, we considered a wider group of control LEAs, including LEAs that are not as close to Greenwich as the ones we selected for the main analysis12 (Tables B8 and C3). The results remain almost identical. Altogether, the robustness analysis provides strong evidence that the effects we have identified are not a statistical coincidence and the only convincing explanation for the effects seems to be the Jamie Oliver campaign.
f) Spill-over and selection effects One legitimate concern regarding the analysis and the results is whether school meals did remain similar in the control LEAs after the campaign. As we mentioned earlier, the campaign was public and thus could have spilled over to the schools not directly Lambeth, Lewisham, Southwark, Tower Hamlets, Wandsworth, Bexley, Croydon, Kingston upon Thames, Merton, Newham, Richmond, Sutton; see Figure 1.
involved in the campaign. This seems very unlikely for two reasons: First, the campaign proved to be quite resource-intensive and not straightforward to implement, it involved the re-training of kitchen staff and the improvement of kitchen equipment.
Other schools could not realistically have implemented similar changes at the same time. Second, schools are involved in long-term contracts with catering services and thus could not directly renegotiate menus and food provision. Nevertheless, it could be that the campaign raised public awareness and this may have affected parental behaviour, possibly even at home. We have no information that such changes have taken place but, in any case, this would imply that our results provide a lower bound on the effects of diet on educational achievements.
Another issue may be whether the new menus made Greenwich schools more attractive for parents to send their children to relative to the other LEAs. Mobility across LEAs could introduce a selection problem and bias our estimates, for example if those children who move towards healthier schools are relatively better pupils in terms of educational performance and presence at school. Unfortunately, we do not have data on the number of applications to primary schools, but one indication of possible selection effects is the mean IDACI score – the mean socio-economic index.
Figure x shows the IDACI score remaining constant over the analysis period suggest that the composition of the households of Greenwich schools, and our treatment schools remained constant.
g) Costs and benefits The last exercise that we propose is a back-of-the-envelope costs and benefits analysis. Note that since we do not detailed information about health outcomes, our estimates probably provide also a lower bound on the overall benefits of the program.
As indicated by the relative fall in absenteeism, it is likely that children’s health improved as well, which could also have long-lasting consequences for the children involved not only through improved educational achievements, but also in terms of their life expectancy, quality of life, and productive capacity on the labour market. We can only provide an estimate of the long-term benefits accrued through better learning and better educational achievements. The effects we have identified are comparable in magnitude to those estimates by Machin and McNally (2008) for the “Literacy Hour”.
The “Literacy Hour” was a reform implemented in the nineties in the UK to raise standards of literacy in schools by improving the quality of teaching through more focused literacy instruction and effective classroom management. They found that the reform increased the proportion of pupils reaching level 4 or more in reading increased by 3.2 percentage points, an effect very similar to the effect we have estimated.
They calculated the overall benefit in terms of future labour market earnings using the British Cohort Study, that includes information on wages at age 30 and reading scores at age 10. They estimate the overall benefit of the reform to be between £75.40 and £196.32 (depending on the specification) per annum, and assuming a discount rate of 3% and a labour market participation of 45 years (between 20 and 65) implies an overall lifetime benefit between £2,103 and £5,476.
It is worthwhile discussing not only the benefits of the programs, but also the costs.