“Riding on unfinished rails” -designing PIAAC

OECD's PIAAC manager William Thorn shares experiences on designing the mass survey.

27.03.2014

Click to enlarge.

Sample from the PIAAC survey (from the section measuring problem-solving in technology-rich environments)

Photo: OECD

 

Designing and carrying out the PIAAC survey was a big international effort, spanning almost a decade. The OECD’s PIAAC manager William Thorn takes us behind the scenes of the survey.

Origins

The origins of PIAAC lie in previous international surveys of adult literacy such as the International Adult Literacy Survey (IALS) and the Adult Literacy and Life Skills Survey (ALL) as well as the OECD’s survey of 15 year old school students, PISA.

These surveys all reflect a general interest in understanding the level and distribution of cognitive skills such as literacy and numeracy among adults. The surveys also study their relationship to factors supporting the development and maintenance of these skills over the lifecycle – and ultimately to outcomes such as employment and income.

The idea of ‘human capital’ and the idea that skills such as literacy and numeracy are an important element of ‘human capital’ also provides an important intellectual backdrop to the survey.

The study was funded principally by the countries participating. The European Commission also supported the study through grants to the OECD and to European countries involved.

Click to enlarge.

Sample from the literacy section of PIAAC.

Timeline

Work on PIAAC began in 2002. The period from 2002 to 2007 involved preparatory, developmental and scoping work to identify the preferred options for the design of the study. The basic design was agreed in late 2007 and implantation of the study started in early 2008.

A field test took place in the first half of 2010, and data was collected between August 2011 and March 2012 in most countries. As we know, the first report on results together with the data from the study was released in October 2013.  

The team

PIAAC was a huge undertaking involving 24 countries over 6 years. A large number of people worked on the study. These included OECD staff, the people responsible for the implementation of PIAAC in participating countries, interviewers, staff from the international consortium as well as a range of experts.

The study was organized in the following way. The OECD was responsible for overall project management on behalf of participating countries. An international consortium, led by Educational Testing Service (ETS)  from the United States, undertook activities such as the development of instruments and the delivery platform, quality control, data processing and such. Countries implemented the study nationally and collected data.

There was no typical ‘working day’ during the PIAAC process. For example, what occurred at a meeting of the project’s steering group (the Board of Participating Countries), what occurred at a meeting of national project managers and what occurred at a session between the OECD and the international consortium were different things. The Board of Participating Countries would discuss matters such as the budget for the project, the way results were to be reported and the standards that countries were expected to meet in implementing the survey. National Programme Managers meetings were focused on operational matters and were often associated with training sessions. Discussions between the consortium covered progress with the survey and identifying solutions to the many problems that arose in the course of implementing PIAAC.

Click to enlarge.

Sound shoe investment? Sample from the numeracy section of PIAAC.

Challenges

One thing that characterized the entire process at all levels was the sense of being in a train riding on rails that were themselves in a process of construction. Early on, it was an effort to ensure that all the parts that needed to be in place for the next stage to commence were ready on time. Given the innovative nature of the project, all those involved had to learn what needed to be done. This took some time for processes to be ironed out.

What also stood out was the dedication of the people who worked on the project to find solutions to problems and keep to the timetable, even if the demands seemed excessive at times.

The component of the study that proved the most complex and difficult was the development and implementation of the computer delivery platform. PIAAC was the first large scale assessment to be delivered entirely on computer. Working out a process for testing the various versions of the delivery platform and implementing revisions and testing them in their turn was difficult.

The main lesson learnt from these challenges is that good project planning is essential from the start, but that you only know what you should have done at the end of the project. Large, complex and innovative  projects are always far more complex and resource intensive than one ever imagines at the start. Goodwill and trust between the people involved are essential for success. 

The philosophy

To be precise, PIAAC assesses the proficiency of adults in three key ‘information processing’ skills – literacy, numeracy and problem solving in technology-rich environments. These are relevant in most social activities (including work) in a modern, ‘information-rich’ society. PIAAC also collects information about the use of these skills in work and elsewhere as well as information on the use of a number of generic skills at work. These work-related generic skills include learning, influencing, cooperation, organisation and physical (strength and dexterity) skills.

The guiding philosophy behind the choice of the skills assessed  -as well as those about which information on skills use was collected – was that they needed to be ‘generic ’in the sense of being highly transportable and relevant to a wide range of contexts and situations. The focus on ‘generic’ skills was driven both by considerations of efficiency (to assess domain or occupation specific skills would be very costly) and policy relevance (the importance of generic skills both of a cognitive and social-emotional or ‘soft’ nature for employment is emphasized by many commentators).

Stakeholder involvement

Decisions about the content of the study were made by the steering group for the project – the Board of Participating Countries made up of representatives from the countries involved. At all points in the development of the study, a range of experts were consulted. For example, in the initial scoping phase of the project, a series of expert workshops were convened. Expert groups also provided advice on the development of the components of the project, such as the background questionnaire and assessment instruments

Policy recommendations

The first report from PIAAC – OECD Skills Outlook 2013: First Results from the Survey of Adult Skills http://skills.oecd.org/skillsoutlook.html , deliberately did not seek to make strong policy recommendations. This is largely because of the complexity of interpreting the results of a study which covers the 16-65 year age range or individuals born between 1945 and 1996. Future reports will examine specific questions in more depth than was possible in the first report and explore the policy implications of the findings.

Hindsight

The major innovation of PIAAC was the computer delivery of both the background questionnaire and the assessments of literacy, numeracy and problem-solving in technology-rich environments. This meant that interviewers read the questions forming the background questionnaire from a laptop computer and recorded answers on the computer. Once the background questionnaire had been completed, respondents were handed the computer and completed the various components of the assessment using the laptop. Computer delivery had real benefits in terms of data quality as it removed many of the sources for error that exist in paper-based assessments.

PIAAC also represents a study in which the concept of managing information in digital environments is central. The inclusion of digital texts in the assessment of literacy and the notion of problem solving in technology-rich environments constitute an important advance in assessing the capacity of adults to manage information in the modern world.

As in all research projects, there are things that you would do differently if you had your time again. Perhaps the main thing I would do differently is to conceive the project as a major IT development project and not just as an assessment project. In thinking this way, some of the problems that were linked to the IT development may have been able to be avoided or solutions found more quickly.

Did you find this article?
  • Interesting 
  • Useful 
  • Easy to read