# John Hattie admits that half of the Statistics in Visible Learning are wrong

At the researchED conference in September 2013, Professor Robert Coe, Professor of Education at Durham University, said that John Hattie’s book, ‘Visible Learning’,  is “riddled with errors”. But what are some of those errors?

The biggest mistake Hattie makes is with the CLE statistic that he uses throughout the book. In ‘Visible Learning, Hattie only uses two statistics, the ‘Effect Size’ and the CLE (neither of which Mathematicians use).

The CLE is meant to be a probability, yet Hattie has it at values between -49% and 219%. Now a probability can’t be negative or more than 100% as any Year 7 will tell you.

This was first spotted and pointed out to him by Arne Kare Topphol, an Associate Professor at the University of Volda and his class who sent Hattie an email.

In his first reply –  here , Hattie completely misses the point about probability being negative and claims he actually used a different version of the CLE than the one he actually referenced (by McGraw and Wong). This makes his academic referencing, hmm, the word I’m going to use here is ‘interesting’.

In his second reply –  here , Hattie reluctantly acknowledges that the CLE has in fact been calculated incorrectly throughout the book but brushes it off as no big deal that out of two statistics in the book he has calculated one incorrectly.

There are several worrying aspects to this –

Firstly, it took 3 years for the mistake to be noticed, and it’s not as though it’s a subtle statistical error that only a Mathematician would spot, he has probability as negative for goodness sake. Presumably, the entire Educational Research community read the book when it came out and they all completely missed it. So, the question must be asked, who is checking John Hattie’s work? As a Bachelor of Arts is he capable of spotting Mathematical errors himself?

In Mathematics, new or unproven work is handed over to unbiased judges who go through it with a fine toothcomb before it is considered to have the stamp of approval of the Mathematical community. Who is performing this function for the Educational community?

Secondly, despite the fact that John Hattie has presumably known about this error since last year there has been no publicity telling people that part of the book is wrong and should not be used. Surely he could have found time between flying round the world to his many Visible Learning conferences to squeeze in a quick announcement.

As one of the letter writer’s stepfather, a Professor of Statistics said

“People who don’t know that Probability can’t be negative, shouldn’t write books on Statistics”

Sources –

Book review – Visible Learning by @twistedsq

Can we trust educational research? – (“Visible Learning”: Problems with the evidence)

EDIT – Since this post we have also discovered why the CLEs are all wrong and the reason is shocking. Read about it here – John Hattie admits that half of the Statistics in Visible Learning are wrong (Part 2).

## 41 thoughts on “John Hattie admits that half of the Statistics in Visible Learning are wrong”

1. Hi, let me just mention that I sent the following mail to Hattie and colleagues more than 3 years ago (but did not get any answer).

———————————————————-
Dear Debra,

Unfortunately we did not get support for our application because it was outside the scope of the special Atlas funding programme. The focus should be more on carrying out a joint project with a school in another country.

However, we will not give up our efforts and in the meantime we will offer a very brief series of three seminars for all the teachers in our municipality, with an international, national and local perspective, starting with “visible learning” tomorrow.

Now, when preparing the very first seminar, I was very puzzled over the CLE:s in Hattie (2009), Visible learning… It seems to me most of the CLE:s are simply the effect size, d, divided by the square root of 2. I am not sure about the characters but here is a try:

CLE = d/√2 ?

* Should not CLE be some integral from minus infinity to d/√2, as e.g. in Dunlap, W.P., 1999, A program to compute McGraw and Wong’s common language effect size indicator, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc, 31(4), pp. 706-9?

* The CLE of 92% for d = 2,0 on page 9 seems OK but how should CLE of 102% for self report grades on page be interpreted? Should not CLE rather be 84,6% in this case?

* Or — am I missing some important point here?

I would highly appreciate any hint to this problem if you know any one interested in helping a struggling superintendent…

All the best,

Per-Daniel Liljegren, superintendent
———————————————————-

2. Clarification, here is the mail headers, corresponding to the previous post.

Från: Per-Daniel Liljegren
Ämne: Re: Evidence based education?
Datum: 26 april 2011 22:18:41 CEST
Till: Debra Masters

Dear Debra,

3. Reblogged this on Blogcollectief Onderzoek Onderwijs and commented:
What to make of this? John Hattie’s book is one of the most influential books on education that has appeared in the last few years. For some, it has almost the status of an education bible. If the statement “Half of the statistics in ‘Visible Learning’ are wrong, can be sustained, what consequences does it have forour teaching practice?
In this and the next post, reblogged from Ollieorange2, the author, a British mathematician and math teachter finds out exactly what is wrong with Hattie’s statistics.

4. Reblogged this on From experience to meaning… and commented:
The title is correct and than again a bit misleading, as it’s not about that the insights are per se wrong. Still I hope there will come a reaction by Hattie on this.

5. Reblogged this on Save Our Schools NZ and commented:
“At the researchED conference in September 2013, Professor Robert Coe, Professor of Education at Durham University, said that John Hattie’s book, ‘Visible Learning’, is “riddled with errors”. But what are some of those errors?

The biggest mistake Hattie makes is with the CLE statistic that he uses throughout the book. In ‘Visible Learning, Hattie only uses two statistics, the ‘Effect Size’ and the CLE (neither of which Mathematicians use).

The CLE is meant to be a probability, yet Hattie has it at values between -49% and 219%. Now a probability can’t be negative or more than 100% as any Year 7 will tell you.

This was first spotted and pointed out to him by Arne Kare Topphol, an Associate Professor at the University of Volga and his class who sent Hattie an email.

In his first reply – here , Hattie completely misses the point about probability being negative and claims he actually used a different version of the CLE than the one he actually referenced (by McGraw and Wong). This makes his academic referencing, hmm, the word I’m going to use here is ‘interesting’.

In his second reply – here , Hattie reluctantly acknowledges that the CLE has in fact been calculated incorrectly throughout the book but brushes it off as no big deal that out of two statistics in the book he has calculated one incorrectly.

There are several worrying aspects to this …”

6. If mathematics in a scientific book are wrong then the insights are indeed not necessarily wrong. On the other hand, the chance of them being right is – well I’m tempted to say “negative” here …

It is better to present insights based on anecdotal evidence or thought experiments than to back them up with quack mathematics. At least the theory will be intellectually honest. This is my belief. I haven’t researched it though.

• MH

The math isn’t wrong and the conclusions don’t change. He just called it CLE when technically, it isn’t. He was measuring the probability of a correlation and if the correlation is negative, then he gave that probability as negative. There is a difference between zero probability and a high probability of a negative correlation. It would have been bad math to not differentiate between those two. He was trying to simplify things to give a simple dashboard for effective versus ineffective. If you want to throw out the baby with the bathwater because he misused a statistics term, so be it.

7. Pingback: Release Notes

8. Pingback: Release Notes

9. Ann-Katrin

Arne Kåre Topphol is a professor at Volda, not Volga. It’s in Norway, and the place where I studied to become a teacher.

• Thanks.

10. Passionate Teacher

I think that it is really easy for people to bring people down in any way that they can. John Hattie is a highly respected academic that is committed to BEST teaching practice. Teachers all over the world have reflected on their own practice to ensure that they are making an impact in their classroom. I believe this is tall poppy syndrome. You should be ashamed of yourself! If Hattie is making such a positive difference for so many students and teachers, find the things that are effective rather than searching and trying to find ways to bring him down and discredit his exceptional work in Education. Hattie never would have said that that half of his work is wrong, what an absolute lie that you are sharing……

• Why do you think his work his exceptional and he should be respected? Half the book is statistical techniques that Mathematicians don’t recognise and the other half is wrong by his own admission.

• thetackler

Hattie’s famous for being controversial, not for being good at doing proper research.

• Jason

If you’re right then the scientific method is tall poppy syndrome at work. Please disclose any relationship with the University of Melbourne or Hattie.

• Alistair Williams

I am a passionate teacher too, a science teacher and I worry very much that much of what is foistered on teachers as “professional learning” and best practice is not backed up with real scientific evidence. Much of educational policy during the 1900s has been based on political philosophy and peoples beliefs and we are still being inundated with buzz words purporting to be the best thing since sliced pedagogy. To find that many of the conclusions drawn by a major educational researcher were based on faulty statistics is embarrassing at best and injects a huge amount of uncertainty into his work. He needs to redo his sums post haste and release an update to his book detailing what affects these corrected stats have on his findings. Has he done so?

11. Pingback: Forward | quidne

12. Recently I commented on this in a Swedish podcast, Didaktorn, http://www.ur.se/Produkter/190464-Didaktorn/Om-serien and feel somewhat obliged to fill in some details.

I told my story since 2010 about this and tried to explain the errors with CLE. BTW, you can easily get them right(?) by using e.g. this formula in Excel (the Swedish localization): =NORM.S.FÖRD(A2/ROT(2);1) where A2 holds the effect size d.

Now, it gets even worse: It seems that Hattie’s standard example of the difference of the height between men and women also has errors. If you try to reproduce the table in McGraw and Wong (1992) you will find a couple of lines in the table which is not what you expect when calculating CLE from d. I tried to reach the authors to get their opinion without success.

How do I know I am right? Well, in fact you never know (depending on ontological, epistemological, methodological and so on, questions and a deeper discussion about objectivity) but I recalculated all the 200 CLE values in Dunlap, W.P., 1999. A program to compute McGraw and Wong’s common language effect size indicator. Behavior Research Methods, Instruments, & Computers, 31(4), pp.706–709. Every one of the CLE values is exactly the same as “mine” or according to Excel to be precise.

Well, you may argue what’s the big deal, that doesn’t change the conclusions?

I agree – IF – these are the “only” almost 1000 errors i Hattie (2009)! The problem is the overall trust of the work combined with the slow admission of the errors.

If you publish results that contradict the most fundamental definitions of probability – and also seem ignorant about it: On what grounds should we then trust the rest of the data?

*

Finally I would like to add that Visible learning had a great impact on me. Actually, I started to cry when I just finished the last lines of it on a bench in Ystad the summer of 2010. I really mourned for the lost opportunities and my own shortcomings of teaching. However, that was before I discovered the errors.

To paraphrase a Swedish poet and entertainer, Tage Danielsson (that I think can be translated): “Without doubts you are not wise.”

All the best to you John, Debra and all others

(If you understand Swedish you kan listen to some more details in the podcast Didaktorn, coming soon.)

13. senseilance

Classic quantitative methodology applied to social sciences research, and getting it wrong. Up yours positivistic researchers!

14. Alistair Williams

Here is Hattie’s final reply to the Tophol class’s enquiry regarding his stats calculations.

John Hattie #
02.12.2012 10:37
1
Thanks for contacting me.
I have found the method I used and indeed Topphol is correct. As he noted in his article: “My conclusion so far is that the CLE-values Hattie offers must be wrongly calculated.” It is. To calculate CLE from d requires finding the integral of the normal distribution from-infinity to d/sqrt(2), and my programming let me down! Thanks for Arne Kåre Topphol for noting this error and it will be corrected in any update of Visible Learning. The error does not affect any of the effect-sizes, or change the story in the book. Readers can have confidence in these matters, but the use of the CLE should be discarded. I have since calculated the correct values of the CLE and these are available from me upon request. I am grateful to Topphol for noting this error, and for your team in contacting me so I could clarify.
John

So does this settle the matter?

• Per-Daniel Liljegren

In my point of view the ”CLE” seems to be simply (mis)calculated as d/√2 (and the integral over the normal distribution from 0 to d/√2 “forgotten” which would have fixed the issue). CLE in the sense Hattie wants to use it is based on McGraw & Wong (page 9)—but BEWARE: There seem to be some errors even in that very reference which I have tried to contact the authors about without success.

In contrast I get exactly the same results as the 200 values calculated in Dunlap, W.P., 1999. A program to compute McGraw and wong’s common language effect size indicator. Behavior Research Methods, Instruments, & Computers, 31(4), pp.706–709. It makes me feel reassured that at least two persons in this world was able to calculated the same CLEs (Hattie not included then but he seem to have found the error now).