Tuesday, December 07, 2004

PISA 2003: Survey Results

The Organisation for Economic Co-operation and Development (OECD) has conducted it's second study of learning skills among 15-year olds. The study (called PISA, to be conducted every three years) was first performed in 2000, and then again last year. Results from PISA 2003 have just been released by OECD. The main focus in this study was on mathematics, while PISA 2000 was only a reading assessment. Over 250,000 students from 41 countries participated in the study; countries that were not OECD members could also choose to be included.

Finnish students were overall the most proficient, though four other countries (Korea, Hong Kong, Liechtenstein and Japan; all in the top 8 overall) had a greater percentage of students at the highest proficiency level. Incidentally, Finland had also led in the reading assessment of PISA 2000. The U.S. was 28th out of 40 countries, just behind Latvia and ahead of Portugal and the Russian Federation. Interestingly, Canada has the third highest performance overall.

The full report is available as a PDF from the OECD website. So far I've only managed to skim through the entire 471-page report; I would read it through except for the fact that I was meant to be working on a unit project for the last 2 hours. They have some obvious conclusions ("Both students and schools perform best in a climate characterised by high expectations [and] supported through strong teacher-student relations"; "Students whose parents have better-paid jobs, are better educated ... perform on average significantly better in all countries than those without such advantages") and some interesting ones.

From the PDF or OECD article, some things that struck me were:
1. Australia, Canada, Finland and Japan stand out for high standards of both quality and equity, with above-average mathematics performance and below-average impact of socio-economic background on student performance.
2. Poland had a dramatic variation of performance between schools in the original study; this has shrunk drastically after the school system was integrated in the intervening period.
3. Most countries have more boys than girls among top mathematics performers, resulting in a slight overall advantage for boys in average terms. On the other hand, boys and girls tend to be equally represented among the low-performers. It's interesting to note that girls attend the higher performing, academically oriented tracks and schools at a higher rate than boys but, within schools, girls often perform significantly below boys. Girls also consistently report lower interest in and enjoyment of mathematics than boys.
4. Better performance was often related to an enjoyment of mathematics, but perception of ability was not as strongly correlated to ability as one might expect. About a third of US children (ranked 28th overall) did not feel as though they were good at math, but nearly two-thirds of Koreans (ranked 2nd) felt the same. (Editorializing: Has worrying about students self-esteem hurt academic performance? Increasingly, schools are refusing to differentiate between students for fear of hurting the image students have of themselves. Readers Digest recently ran a story on how schools are refusing to recognise true merit; some high schools had 50 to 100 valedictorians! In Nashville, one high school principal was told he couldn't release the names of high scorers at basketball games. Another school couldn't announce the winner of the spelling bee! But I digress... I'll leave the subject for a future post)
5. The US also seems to have the poorest outcomes per dollar spent on education. The performance in reading (18th out of 40) is better than that in math, though. "While spending on educational institutions is important," the report says, "it is not sufficient to achive high levels of outcomes." (I wish more administrators would realize that throwing money at the problem wouldn't work!)

Unfortunately, India isn't rated. I'm curious about how Indian students would have performed. I suspect that the reading assessment results would have been abysmal, but the mathematics results very good. This is entirely due to the fact that only students still in school at the age of 15 participate in the survey; the majority of Indian children have dropped out of school by this time, if they ever attended. (It's technically illegal in most states to not send children to school, but this is rarely, if ever, enforced.) Glancing through the questions, I would be shocked if the average 15-year old Indian still in school couldn't solve at least half of the problems listed as most difficult (Level 6). (See my previous post on high-school math curricula.)

From the section titled "The PISA approach to Assessing Mathematics Performance":
PISA therefore presents students with problems mainly set in real-world situations. These are crafted in such a way that aspects of mathematics would be of genuine benefit in solving the problem. The objective of the PISA assessment is to obtain measures of the extent to which students presented with these problems can activate their mathematical knowledge and competencies to solve such problems successfully.
This approach to mathematics contrasts with a traditional understanding of school mathematics which is often narrower. In schools, mathematical content is often taught and assessed in ways that are removed from authentic contexts – e.g., students are taught the techniques of arithmetic, then given an arithmetic computation to complete; they are shown how to solve particular types of equations, then given further similar equations to solve; they are taught about geometric properties and relationships, then given a theorem to prove. Having learned the relevant concepts, skills and techniques, students are typically given contrived mathematical problems that call for the application of that knowledge.

Indian (and Asian, in general) schools usually use the latter (traditional/narrower) approach, in contrast to many Western schools which teach mathematics the way PISA tests it. So I find it strange that Korea, Hong Jong, Macao and Japan would be in the top 8, and I believe India would be up there with them. (Assuming, of course, that questions were appropriately translated into the local language and cultural context; I can see the average Indian student having difficulty on the question about skateboarding.) Perhaps the PISA test doesn't measure what they think it does, then.

The New York Times story had some interesting insights; they've obviously spent a lot more time analysing it than I have... With luck I'll get to it after Finals week. (Don't forget all the exams you'll have to grade! - ed. Siiigh - you had to remind me, didn't you?)

No comments: