Grade inflation, educational achievement and the profit motive: a comment on the SNS report

Yesterday, SNS published a report reviewing the research on whether the introduction of competition in the Swedish public sector has generated large efficiency gains. It finds that there is little evidence of this. The chapter on free schools and competition, written by Jonas Vlachos, deserves a comment. In general, I agree entirely with Vlachos’s conclusion that decentralisation and competition require more accountability regarding grading practices in Swedish schools – which I have argued here.

In this post, however, I focus on Vlachos’s findings regarding the impact of for-profit/non-profit schools on educational achievement/grade inflation – which have been cited in Swedish media as evidence that for-profit schools underperform. I also present new evidence indicating that profit-seeking compulsory schools suffer from slightly higher grade inflation than other school types, although the quantitative estimates are small.

Vlachos shows that pupils in for-profit/non-profit compulsory schools perform better in terms of average grades than pupils in municipal schools, and also that the effect on scores on standardised tests is stronger than on the GPA. (There is no attempt to deal with potential endogeneity or two-way causality.)¹ Although certainly far from methodologically flawless, partly due to data availability, my study from 2010 – using OLS and IV models – also finds that for-profit schools perform better in terms of average ninth grade GPAs.

However, arguing that average grades and test scores may not be comparable between schools with different ownership structures – given Sweden’s extremely decentralised grading practice – Vlachos also finds another, negative effect of for-profit schools.  When employing standard control variables accounting for socio-economic background, age, sex, immigration etc, pupils attending for-profit compulsory schools obtain slightly lower GPAs in upper-secondary school – amounting to 0.085 of a standard deviation – than those who attended municipal compulsory schools (in relation to their ninth grade GPA). Vlachos suggests two possibilities for the results: (1) for-profit schools are worse than municipal schools or (2) for-profit schools may have higher grade inflation in ninth grade, presumably due to the profit motive.

Firstly, although not presented in the report but kindly supplied by Vlachos, the R² value for this regression is 0.03. In other words, despite the variables utilised, only 3% of the difference between pupils’ GPAs in ninth grade and upper-secondary school can be explained by the model. This strongly suggests that there are other, omitted – and much more important – reasons for why pupils perform better/worse in upper-secondary school in comparison to how they performed in ninth grade.

This leads to my second point. Vlachos’s suggestions regarding how one should interpret the regression ignore three years of upper-secondary education (and life changes!), which his model treats as a black box. For example, it could equally be the case that the differences in grades are generated by differences in upper-secondary schools (in terms of equally decentralised grading practices, quality of education, teacher quality, pupil motivation etc). There is also no control for whether the pupil attended a for-profit/non-profit upper-secondary school.

It should be pointed out that Vlachos certainly admits this to a certain extent: there could be systematic differences between schools with different ownership structures regarding the upper-secondary school programmes their pupils choose (i.e. natural or social sciences, or non-academic, such as the arts, media, and handicraft programmes etc).

But this has strong implications. Pupils’ GPAs are much less comparable in Swedish upper-secondary schools than in compulsory schools, due to vast differences between pupils’ education in the former (they choose programmes focusing on different subjects – such as social sciences, natural sciences, vocational subjects, performing arts etc). When analysing overall educational attainment at Swedish upper-secondary school, one compares pupils studying vastly different subjects. This is essentially tantamount to comparing apples with oranges.

Another issue is a strong potential for selection bias. For example, if pupils in for-profit compulsory schools systematically choose more difficult academic programmes, with more difficult courses, in upper-secondary school, this could have a negative impact on their GPAs. On the other hand, if they choose vocational programmes to a larger extent, the incentive to care about grades declines significantly since these are not as important for the pupils' future careers as for those who choose academic programmes. This, of course, is purely speculative. The point is that there are numerous possible reasons for why upper-secondary school pupils perform better/worse than they did in certain compulsory schools.

The overall conclusion, therefore, is that it is doubtful whether one can compare overall educational attainment at the upper-secondary school level – at least if one doesn’t take into consideration which programmes and courses pupils choose.

A better alternative (1) would be to compare the difference between pupils’ grades in a specific course – say Mathematics A after the first year of upper-secondary school – with their final maths grade in compulsory school. Alternatively (2), one could analyse differences between pupils’ GPAs in ninth grade and their GPAs after the first year in upper-secondary school, since the latter reflect achievement in mandatory courses for pupils in all programmes. Yet another alternative (3) is to also include the second and third year upper-secondary school GPA in the analysis, as Vlachos has done, but restrict the sample to pupils choosing academic tracks (see Böhlmark and Lindahl 2008).

Having showed that Vlachos’s results should be interpreted with caution, it is also important to compare scores on standardised tests and final grades in ninth grade. This is a measure to analyse the comparability of final grades between profit/non-profit and municipal schools. The advantages are that one doesn’t have to take into account confounding variables at the upper-secondary school level, which clearly are at play here, and also that one is analysing average achievement in subjects that are mandatory for all pupils. The disadvantage is that these tests are still marked by individual teachers so it is doubtful whether the scores are strictly comparable.

In a footnote on page 87 in the SNS report, Vlachos notes briefly that there are no differences between for-profit and municipal schools in this respect. However, in my analysis, presented below,  I do find that for-profit schools in 2009 set slightly higher grades than municipal schools compared to how pupils perform in standardised tests in mathematics and English (but not in Swedish). Yet, the differences are quantitatively small: for-profit schools give higher final grades to 2.5-3.3% more pupils compared to municipal schools.²

Differences between test scores and final grades (2009)
(% of pupils with higher  minus % of pupils with lower final grades compared to test scores)
	English	Swedish	Maths
For-profit free school	2.5**	1.3n.s.	3.3**
Non-profit free school	-0.9n.s.	-0.8n.s.	0.6n.s.
Pre-reform free school	4.3*	0.4	-0.05n.s.
Boys	-9.7**	-6.2n.s.	-2.9n.s.
Immigrant_1st_gen	10.6*	-1.8n.s.	4.8n.s.
Immigrant_2nd_gen	-11.9***	14.7**	1.4n.s.
Parents’ education	-12.8***	-0.8n.s.	-10.5***
Teacher density	-0.03n.s.	0.15**	-0.1*
Urban dummy	-0.07n.s.	1.1n.s.	0.18n.s.
Average income (municipality)	0.03**	0.02**	-0.0007n.s.
% in free schools (municipality)	0.02n.s.	-0.07n.s.	-0.007n.s.
N	1,275	1,266	1,269
Adj. R2	0.06	0.03	0.02
Note: *p<0.1, **p<0.05, ***p<0.01, unstandardised coefficients, robust standard errors

Furthermore, similarly to Vlachos’s results, the low adjusted R² values suggest that only about 2-6% of the differences can be explained by variables normally associated with pupil achievement. Again, therefore, the vast bulk of the differences in grading practices between compulsory schools seemingly depends on other factors.³

Since my study at the IEA analyses ninth grade school GPAs, the findings should of course also be interpreted with caution. Yet, given the small quantitative estimates regarding differences between test scores and final grades – as well as the overall lack of explanatory power of both Vlachos’s and my regressions here – there is little hard evidence suggesting strong differences in grading practice between compulsory schools with different ownership structures on average. This could imply that ninth grade GPAs across different school types are relatively comparable. It is certainly difficult to argue that there is strong evidence that for-profit schools are worse than municipal ones based on Vlachos’s findings.

Regardless, grading practices in Swedish education must be radically reformed. According to the SIRIS-database, some schools give higher final grades to pupils compared to how they perform on standardised tests in over 100% of the cases (!) – while for some schools, the figure is 0%. This strongly suggests that grades are currently not comparable across individual schools, which has to be remedied.

One alternative is to centralise all formal grading in both compulsory and upper-secondary schools; another is to allow upper-secondary schools and universities to accept more pupils by other means than the GPA, as I’ve argued here. Competition should always be accompanied by strict accountability, but Sweden’s decentralised grading system, in combination with its extremely centralised admissions system, does not allow for such accountability.

----

¹However, the effects of attending for-profit compulsory schools do not translate into higher GPAs in upper-secondary school (the effect is not statistically significant), and the non-profit effect is small. Furthermore, there is no evidence that pupils in free schools overall performed better in the latest PISA test. 

²It should also be noted that these differences may not exist – or be more significant – in other years, which is something I now am researching. I will also analyse data from 2010 for which grades in biology, physics and chemistry are available.

³However, it is interesting that the higher the level of parental education, the lower is grade inflation in mathematics and English (while there is no effect in Swedish). This suggests that the fear that grade inflation would disadvantage pupils from lower socio-economic backgrounds –since they purportedly do not tend to pressure teachers to get higher grades to the same extent as pupils from higher socio-economic backgrounds – is unfounded.