At the start of ‘Visible Learning’, John Hattie talks about the two different ways to calculate the Effect Size.
Effect Size = (Mean of group at end of intervention – Mean of group at start of intervention) / Standard Deviation
Effect Size = (Mean of intervention group – Mean of control group) / Standard deviation
Now, obviously, real Mathematical things don’t have two different definitions because of the confusion this causes as we shall see.
The problem is that the first definition measures actual improvement whereas the second measures relative improvement.
To give an example, imagine Ann and Bob are both travelling to an education conference. They set off at the same time, driving their cars down the motorway.
We know that Ann drives at an average speed of 40 mph. How can we tell if Bob will get there first?
We could give his actual speed of 50 mph.
Or we could give his relative speed, he’s travelling 10 mph faster than Ann.
It doesn’t matter which one we use as long as we all know which definition we’re talking about.
The problem comes when we start using the same words to mean two different things and start mixing them up.
When comparing Bob’s speed to Ann’s, a good actual speed would be anything over 40 mph, but a good relative speed would be anything over 0 mph.
Now, if I say that Cathy is going at a speed of 30 mph, is that an actual speed, in which case she’s going the slowest, or a relative speed in which case she’s travelling the fastest?
Hattie only mentions the two different types of Effect Size once, at the start of the book, but the way he talks later on “Everything works” and “We need to compare with 0.40” shows that the definition he is using is the first one, measuring actual improvement. However, has he made that distinction when he was choosing his studies? I suspect that he has not realised that the two different ways of measuring would produce very different answers and he has just thrown both types of study all in together.
For John Hattie, any Effect Size bigger than 0.40 would be ‘good’.
Now which version does the Education Endowment Foundation use?
In their recent Chatterbooks study they say
This and the fact that they use control groups show that they are using the second way of calculating the Effect Size, the relative way.
So, for the Education Endowment Foundation, anything better than zero would be ‘good’.
So, to sum up, the two major players using the ‘the Effect Size’, John Hattie and the Education Endowment Foundation, are actually using it to mean two completely different calculations, one actual and one relative. For one of them, anything above 0.40 would be ‘good’, the other anything above zero would be ‘good’.