Unburnished

Quick notes while reading Mapping the Field: a framework for measuring, monitoring and reporting on adult learning in Canada

This report is entitled "Mapping the Field", but it does not mention teachers, institutions or anything other than learners. What is actually about is mapping learner participation and activities.

The report is less useful than I had hoped.

In terms of monitoring, the report focuses on four areas:
· sphere of learning
· Field, i.e. formal, non-formal, informal
· specific programs or learning activities
· learning modalities, e.g. full-time or part-time, workplace, program information

Some learner categories that we may want to think about in terms of what should be monitored: older learners, second chance learners, upskilling e.g. for a career change, people with disabilities.

They propose to gather information from learners and learning institutions.

We may want to follow this report (page 50) in mentioning the low skills equilibrium, and economic strategy based on low-cost/low value-added approach to business. Here I believe that Alison Wolf and others have criticised the UK for this, arguing that the real problem in the country is not a lack of skills, but a business approach wherein employers are not interested in taking the risk of pursuing higher skills, higher risk routes. This may or may not be relevant, but I know the ITT asks us to contextualise what we are doing in broader policy framework, including employment and economic issues. Perhaps this could all appear in some sort of conversation about skills gaps, skills needs, skills surveys (e.g. of employers).

This report's proposals for "data generation strategies" focus on the use of existing secondary data produced by statistics Canada and regular surveys of providers. The authors do argue that the scale of the undertaking is daunting, and that cost would be “significant”.

UNESCO's four pillars of learning:
· learning to know
· learning to do
· learning to live together
· learning to be

When discussing informal learning, looks at "explicit" informal learning, which includes four categories:
· employment related
· community based/volunteer work related
· household related
· general interest

Definitions of adult learning hugely influence our picture of who is learning and how much. For example, NALL in Canada found a adult learning participation rate of 96%, compared to the Statistics Canada Adult Education and Training Survey finding of 28%.

First notes reading through our data from the EU survey: frequency of administrative and survey sources

We have 67 administrative sources and 68 surveys. These 135 sources, from 29 countries. Off the top of my head, I don't know how many countries reported themselves, and how many were gathered through desk research. I think only a couple came purely from desk research.

In terms of distribution, six countries account for more than half the total number of sources: 72. Those countries are Austria (10), Flemish Belgium (8), Germany (10), Norway (19), Sweden (8), and England/Wales (17). The average for these six countries is 12 data sources per country.

When doing analysis and graphs, we will sometimes need to provide a look at all countries combined, and at other times provide a look only at the 23 countries that account for an average of fewer than three data sources per country.

Within countries, what is the distribution of administrative and survey sources like? Taking a quick glance at the data, this is what first leaps out:

Austria has nine administrative sources and only one survey;
Flemish Belgium has seven administrative sources and only one survey;
Czech Republic has five administrative sources and only one survey;
Denmark, which is under reported, shows three administrative sources and one survey;

• Germany is the opposite of Austria, showing one administrative source and nine surveys;
• Ireland has two administrative sources and for surveys;
• the Netherlands, which I suspect is under reported, shows zero administrative sources and three surveys;
• Norway is almost perfectly balanced and is perfectly representative of the overall results, with 10 administrative sources and nine surveys;
• Slovenia reports three administrative sources and no surveys;
• Sweden is perfectly balanced with four administrative sources and four surveys;
• England/Wales is very heavily tilted towards surveys, with three administrative sources and 14 surveys;
• Northern Ireland shows three administrative sources and one survey.

Looking at the numbers above, England and Germany account for slightly more than one third of all of the surveys reported: 23/68. When you add in Norway's nine, those three countries account for close to half of all surveys: 32/68.

Is the spread of administrative data more even? A bit. Austria, Flemish Belgium and Norway are the only countries with more than five administrative sources. All told, these three countries account for 26 of 67 administrative sources.

For administrative and survey, we will put in a graph showing frequencies.

I would suggest that some future follow-up research would involve qualitative interviews with selected countries -- the sort of research that wasn't possible given our timeline and budget. One thing to pursue, for example, would be to talk with Austria and Norway about the advantages and disadvantages of administrative sources. It would be interesting to hear from Germany about why they are so survey-dominant.

Towards a European statistics for adult education -- DIE, 2000

This is a useful paper. They did a very good job of presenting information from each country -- but unlike in our survey their aim was to actually present data from each country, rather than determine what types of data were available in each country.

There are some good steps to inform our method, particularly with regard to the thoroughness with which these researchers undertook their Internet-based desk research.

See Page 7 for a handy chart of topics covered in each country that can be modified for my purposes. It's a chart simply showing various fields of analysis and providing a tick if a country has a data source for that field of analysis -- at least that's how it will be when I modify it. That can be my introductory chart. Within each country I can then do a more detailed chart providing the same breakdown but with numbers to represent each individual national data source.

On pages 7-8, they discuss eurybase country reports. Should we in our country discussions or in the introduction to the contrary section, discuss these reports and similar ones?

And on page 84 they give a good list of shortcomings, probably all of which are true today.

From the above findings, it is obvious that European continuing education statistics are “patchy”. The statistics

of the individual countries can barely be compared one with another, let alone aggregated. The following shortcomings

can be identified:

a) In most European countries, continuing education statistics are inadequate, and it is not possible to

reach an overall view of the situation because the data available in the various states cannot be aggregated.

Institutional statistics and figures from funding bodies do not coincide with the results of

surveys. The fragmentation of continuing education statistics is not just a European but a national

problem. Few countries (Finland, Germany and Switzerland) can provide complete figures.

b) There is no clear standard definition of continuing education. It is not always distinguished from

initial vocational training, schooling and higher education on the one hand, or from work and leisure

on the other, so that it is difficult to compare totals.

c) Different institutional structures have grown up historically in the various countries, and have their

own peculiarities. The institutional statistics deriving from them are generally not comparable,

partly because classifications of topic areas and staffing vary.

d) The statistics available cover a considerable period, and the currency of the data provided varies

widely. Any composite overview would have to allow for the “stragglers”, which would reduce the

meaningfulness of the data.

e) The level of development of continuing education statistics differs in the various countries. For reasons

of comparability, any comparative or cumulative European continuing education statistics

would be constrained by the “weakest link”, which might make the more advanced countries appear

ineffectual and imprecise.

f) Only a relatively small set of characteristics is recorded by all countries. In most countries, this includes

numbers of participants (or students) and some classification or grouping of institutions, for

example. There is also a huge range of data or characteristics recorded in only one or a handful of

countries.

Measuring lifelong learning for the new economy -- implications for the statistical system

That's the article title up there, the bit before the dash. In this article, the author discusses some challenges for the monitoring of lifelong learning in general, but obviously my focus is the adult learning phase of that.

"The wide-ranging orientation of lifelong learning poses nearly unsurpassable conceptual problems. Because it is not tied to any institutional context, it requires analyst to take a large, holistic perspective."

The statistical system is frontloaded, mostly focused on measuring the inputs to education. The author argues that a key challenge will be to figure out how to monitor common measure and collect data on key processes of learning, such as changes in pedagogy and the ways in which individual students are engaged. I think that our survey supports this. How much information if any do we get on processes aside from learner participation?

And he says that the statistical system has worked for many years to develop good measures to shed light on finance, enrolment and achievement in education. "Available statistics on adult education and vocational training sector tend to be far fewer and less satisfactory, mainly because they are derived from institutional or administrative sources." I have been singing the praises of administrative sources, but this is a fair point. Peopleinputting data administratively have far fewer incentives to get the data right than do professional researchers. There's going to be a good amount of error, unless they are powerful incentives to reduce or prevent this. What are the error rates in the ILR and SIR?

what about life wide learning? This is about moving beyond the formal institutional settings and end to non-formal learning, e.g. at work, and informal learning in daily life. He makes the good point that for adults learning pathways are individually defined and that adults, as opposed to schoolchildren, are a noncaptive population. If we are going to accurately measure adults' participation in and pathways through adult learning over the lifecourse, then we do need better measurement instruments. But as discussed at our seminar in Lyon, measuring informal learning is next to impossible and may well be practically meaningless if 95% of people say they have participated in it. If you have a 95% participation rate, you need to quit focusing on measuring participation and start focusing your attention is on other aspects of the process, e.g. what they participated in, duration, what they get out of it, etc. But how could that be measured, or even recalled by the respondent?

This author argues that because adults are noncaptive and unpredictable, we need "new measurement instruments combined with a labour force or household survey approach to data collection". He points favourably to the LFS module on lifelong learning as an example of such an approach. Maybe something for us to think about?

He places an even bigger emphasis on the importance of longitudinal data for measuring lifelong learning: it is a cumulative process that cannot be captured with cross-sectional, snapshot data. He says that longitudinal approaches allow for investigations of interactive as well as cumulative impacts of formal, non-formal and informal learning on outcomes such as economic returns, health returns and social returns. Impact such as this can only be captured over time, but good use of longitudinal data showed, he argues, enable strong contributions to policy.

An advantage of longitudinal research that he flags up is that it provides information on "transitions and pathways between initial learning, work and further learning as well as the complementarity or substitutability" of these types of learning. This is exactly what one of the Spanish data sources tries to shed light on.

He laments having to use qualifications as a proxy for skills, but accepts that accept for competencies and skills that can be measured directly, such as literacy and numeracy, such proxy measures are necessary despite their weaknesses. One obvious weakness is that qualifications in a sense have a half-life: the skills that enabled one to achieve a qualification 30 years ago may not be particularly relevant to one's work responsibilities today.

The mysterious case of the disappearing teacher CPD discussion

This paper looks at key competencies for learners and teachers in Finland. What's interesting about it is that while it addresses learners across the lifecourse, when it comes time to talk about teacher development, it completely omits adult learning.

Perhaps use as an example of how far off the policy spectrum this is? Well it's not that far off the policy spectrum: it comes up repeatedly in European Commission documents. But there's a lot of talk and very little action. And in this Finnish paper, they don't even bother with the talking.

Jibber jabber about indicators

"At their most noble, civic indicators are used as measuring systems to assist societies and communities towards the desired course, to clarify key issues and challenges, and to prioritise resources, especially spending. They do not just monitor progress; they help make it happen."

(Taken from Walters, Shirley (2006) "Researching indicators for a learning region". Paper presented at the 36th annual SCUTREA conference, 4-6 July 2006, Leeds. Her attribution for that quote is "ODA, DLL 2005, citing Reed, 2000", but she doesn't give a footnote for the original quote. Hers is ODA, DLL (2005) Learning Cape Indicators Final Report, ODA/DLL, Cape Town. I can dig up the original if we want to use it.)

Indicators can be used for accountability, Feedback, evaluation or development. Choosing indicators and core data to focus on are not politically neutral processes.

 

Core indicator on teachers' professional development: don't worry -- it's only schools

From Page 168 of "Progress towards the Lisbon objectives in education and training -- indicators and benchmarks -- 2008":

"In the case of the three other core indicator areas, new surveys are implemented in cooperation with other international organisations. In the area of Adult Skills and Teachers Professional Development, EU data needs can be satisfied with new surveys organised by OECD."

What is this CPD survey?

It's TALIS, and it only surveys teachers in compulsory schools

The first round of TALIS surveyed teachers of lower secondary education and the principals of the schools in which they work. Separate questionnaires for teachers and principals were developed by an international expert group

....

Within participating countries, schools as well as teachers within schools, were randomly selected to take part in TALIS. For each country – except for smaller countries – some 200 schools and 20 teachers within each of these schools were sampled.

....

Countries participating in the first round of TALIS chose to focus the survey on the following key aspects of the learning environment, which can influence the quality of teaching and learning in schools:

• The leadership and management of schools- the roles adopted by school leaders, given increasing accountability and devolution of educational authority and the impact this has on the learning environment in schools and the work of teachers (complementing the current OECD thematic review on school leadership).
• The appraisal of teachers’ work in schools and the form and nature of the feedback they receive, as well as the use of outcomes from these processes to reward and develop teachers. Linked to this, the professional development that teachers undertake and how this is connected to appraisal systems, how it is supported by school leaders and how it impacts on classroom practices.
• The profiles of countries with regard to teaching practices, activities, beliefs and attitudes, and how these vary according to teacher background characteristics.

Is there any way in which this should or could inform our recommendations about surveying or collecting data about teacher quality, training and CPD in adult learning?

 

LLL2010

We should mention LLL2010?

The project focuses on the contribution of the education system to the making lifelong learning a reality and its role as a potential agency of social integration.

Strategic objectives addressed:

• to develop and carry out a joint research agenda for a better understanding of the tensions between the knowledge-based society, lifelong learning and social inclusion in the context of enlargement of the EU and globalisation;

• to provide an analysis of the role played by education system in the enhancing of lifelong learning and dependency of this role on relevant institutions at micro, meso and macro levels;

• to provide an empirically based analysis of the adequacy of lifelong learning policies in Europe and their implications for different social groups, especially for socially excluded groups;

• to develop relevant policy proposals for lifelong learning strategies to decrease social exclusion on the European and national level and to identify their implications for relevant areas of social and economic policies;

• to strengthen cooperation and infrastructure for international and multidisciplinary comparative research in the area of lifelong learning;

• to develop transnational data sources, i.e. survey of adult learners in schools and universities, in-depth interviews with schools management, employers, analysis of policy documents.

Under LLL2010 project the dissemination of findings to policy makers, practitioners and service users is aimed not only to present final conclusions but to involve relevant actors into the process as well.

Project includes 5 subprojects. These subprojects will help to achieve the objectives of the whole project

Piggybacking on what has been written about vocational education and training

I'm reading an article entitled "internationally comparable statistics on education, training and skills: current state and prospects". I think it was originally published in Penthouse. Writing about the indicator for public expenditure on education as a percentage of GDP, the authors point out that the source for this indicator, the UOE questionnaire, does not distinguish type of education. This affects VET, and adult education more broadly.

Writing about expenditure on continuing vocational training, they say that information on costs of courses provided by enterprises can be obtained from the CVTS but data is limited to "sectors of economic activity covered and do not include other forms of training".

Public expenditures on active labour market policy measures including training are available from the labour market policy data collection and the OECD.

There is not much information on individual or family expenditure on education in general and cvt in particular. I assume this latter is true for adult learning in general.

The authors complain that simple indicators such as vocational education and training participation and graduation rates are not easily produced using the UOE questionnaire, although this would contribute to a better understanding of factors related to dropout rates and presumably other issues.

They also complain that indicators on completion of educational programs duplicate each other. Worse, the duplicate each other but with inconsistent results. I assume this means that the same thing is being measured but in two different ways or from two different angles.

UOE data and the Labour Force Survey lack comparability, they say "leading to major discrepancies".

The LFS apparently provides limited insight into unemployed people, particularly the long-term unemployed and the hidden unemployed. It would be useful for research, they say, if more detailed characteristics on these groups were available. I think we might want to recommend some sort of data collection on education and skills training for the unemployed, so this is something to consider.

They also say that there "is no specific comparable data source on skills shortages and mismatches (e.g. qualification requirements of vacancies or skill need analysis by sectors or occupations)". In our discussion, how do the employment sector and the adult education sector work together? Can I borrow an idea from education and talk about the triangle, with one arm being Employers, one adult education providers and the other the government?

According to the authors, because of the emphasis on the teaching profession in EU and national policies, detailed data on teachers and other staff involved in training are required, including sociodemographic characteristics, qualifications, earnings, status, roles and duties. They say that right now, "only information on personnel in educational programs is available from the UOE questionnaire".

They say that currently "there is no adequate source at European or international level" to provide data on the outcomes of vocational education and training and lifelong learning, e.g. to assess the efficiency and effectiveness of systems, programmes and measures. Perhaps to my surprise, they say that PISA is a step in this direction because it measures students' skills in selected domains. But there is no way to assess the specific outcomes of vocational education and training. Nor is there any way to assess the specific outcomes of adult learning in general. International adult literacy surveys, including the adult literacy and life skills survey (ALLS), measure performance in a few skills are domains and relate these skills to "a number of labour market success variables".

According to the authors, analyses relying on skills levels give a better indication of the benefits of education and skills then analyses relying on educational attainment levels, such as the LFS. A major problem with this, however, is that such measures are currently and perhaps prominently limited to basic skills domains, because these can be assessed in a fairly straightforward manner, compared to for example occupational skills. It may be the case that qualifications as a proxy are as good as it gets, at least beyond the domain of basic skills.

According to the authors, "detailed data on lifelong learning are needed, covering types, subjects and duration of courses, training providers (including micro-enterprises, public service, etc), characteristics of participants and nonparticipants [i like that second one], and impacts on for adult life, career and participation in learning activities."

The authors go on to say that data should include non-traditional forms of learning such as informal learning, distance learning, and e-learning.

They argue that "the current LFS indicator on lifelong learning "underestimate the extent of adult participation". CVTS excludes the public sector and the non-employed, as well as some industries because of difficulties with data collection. Except for gender, data on participants in continuing vocational training do not include sociodemographic characteristics. CVTS data also exclude "more informal and non-formal work-related learning such as job rotation or quality circles", covering only continuing vocational training courses.

However, the authors feel that the adult education survey should resolve part of the information gap.

We are not going to use the inputs, processes, outputs, outcomes framework in this report, but it is worth noting that most data sources mainly provide information about inputs. The authors note that the LFS and OECD provide data on outcomes, including educational attainment of populations, dropouts, skill levels and earnings. Euro barometer survey is add to this by providing some information on citizens' opinions about lifelong learning, but does this include adult learning? Another question: which OECD source is she referring to here?

The authors argue that developing better data sources and indicators on outcomes would help provide "crucial data on vulnerable and disadvantaged groups". The EU, they argue, should more systematically support efforts in this respect; "extensive research programmes on the issue should be financed".

The one complaint the authors make about the international sources that exist is that they provide a fragmented rather than a comprehensive picture of adult learning. Because information across sources is difficult to combine, each source only allows analysis of "single aspects" of the education and training system. I think this is a really interesting point. An advantage of a comprehensive administrative source like the staff individualised record is that it could enable us to look at a full range of teacher and quality (e.g. IAG) issues, which could perhaps be supplemented by complimented or checked against occasional surveys.

What's worse, they say is that results from different sources often provide different pictures of the same issue. This  happens for a variety of reasons, including

• sources focus on different subjects, e.g. the LFS focuses on labour force characteristics whereas the CVTS focuses on provision of training to employees
• sources use different definitions, e.g. of education or skills
• sources refer to different statistical units, e.g. households in LFS, individuals in IALS
• sources have different periodicity
• sources have different country coverage
• in most sources, and this is important, adult learning is not the main subject but just one aspect covered, which leads to limited coverage of issues of concern to us. This is much less of an issue at national level I think.

Here's an interesting titbit: in 2003 France adjusted the reference period for participation in lifelong learning from one week to 4 weeks, and this doubled their perceived participation rate.

Do we need to say something about EU SILC? it does have some variables about education and training.

Household budget survey includes education consumption expenditure, but can this be broken up into adult learning?

PIAAC will measure skills levels in several domains while also assessing the use of selected skills in the workplace. Interesting.

These authors argue that "despite significant progress in the provision and use of comparable data on education, training and skills, further developments at EU and international level or hampered by a lack of strong guidance and corporation. The concrete roles and responsibilities of the various actors [ e.g. Eurostat, Cedefop, European training foundation] are not always clear and distinct."

The authors argue that "the process of adapting current data sources and of developing new ones... should not only aim at filling gaps and covering upcoming needs. It should also have as an objective to improve methods and definitions".

Despite working in vocational education and training, they note that the provision of data specific to the area is not necessarily an efficient option given the high costs of surveys and analysis and the relatively limited use of this information. They argue that data on vocational education and training should be embedded in the whole process of getting more reliable information on lifelong learning. I would argue that there should be an adult learning data strategy which includes VET.

My long awaited post on adult learning data and indicators

Standard criteria for indicators is that they are based on data that are already available and on data that are valid and comparable. To me, this suggests that there is no way we are going to be able to suggest any new indicators. Was it agreed at the seminar that we would not have to do this -- that our goal should be to suggest five areas for core data collection? That is my memory.

And I think it is a correct one. As I have read in some EU paper: "where data are not available, the development of indicators is a long-term process." We are at the start of that process. Though to be fair we are further along in some areas than in others.

Could we suggest an indicator for adult literacy and numeracy achievement? Not achievement -- that's not what I mean to say: literacy and numeracy skills levels. But do the indicators for teenagers, e.g. 15-year-olds, already cover that? There are indicators based on the pisa reading literacy scale and the pisa mathematical literacy scale. Would it make sense to have separate indicators for adults? What are the pros and cons of this?

What would it add? Well, it would add a focus on improving the literacy and numeracy skills of immigrants, and older people. Because there is so much flow from country to country nowadays in Europe, perhaps it is shortsighted to focus on the literacy and numeracy skills of 15-year-olds, assuming that they are pretty much the only ones who are going to grow up to make the country's adult population. In truth, the adult population in some countries will include a large number of people who were not 15-year-olds in that country.

It indicators have a twin purpose -- to monitor progress towards agreed objectives and to function as a means of identifying good practice - having indicators on adult literacy and numeracy skills would seem sensible. It would also encourage countries not to drop this issue after the age of 15. And it has the advantage of being an issue that is very higher profile in many countries. I think this would be a good area in which to make a recommendation. Is this what Piaac is doing? If so, PIAAC could be used as the instrument. What are the barriers to that?

I'm reading some notes now on short, medium and long-term activities in the realm of indicator development. I can't think of anything we've looked at that would not fall under the final heading. In the discussion of that heading, it is implicitly observed that data which can be collected and administrative lay can more quickly go through the process of contributing to an indicator. Administrative is quicker: I think that maybe one of our observations.

How does the concept of indicators and data work with, say, the idea of health? For example, would you want countries to collect information on health related outcomes associated with participation in adult education? Or is that too much damn work? Wouldn't it make more sense to do a lot of studies to nd out that there are health related impacts of adult learning, then focus on your indicator on participation, knowing that the higher it is the more health benefits you will see.

Is part of the issue here the idea that it is much easier to measure outputs, e.g. graduations or qualifications, then it is to measure outcomes? Outcomes are dependent on having established that there is a causal relationship between, in this case, adult learning and e.g. earnings and employment. It is easier to measure outputs, because the entire purpose of the input and process stages is to achieve those outputs, generally speaking. But to achieve the desired outcomes, you have to deal with far more issues that are outside of you were closed system's control.

I've just read the phrase "decreasing half-life of acquired knowledge". Well put. Perhaps this could be juxtaposed with increased life expectancy -- and increasing life expectations. We definitely should say something about monitoring older people -- that is, collecting data on the participation of older people in adult learning. This is one of those areas where we know there are many positive outcomes, even if there are a few outputs e.g. in the form of qualifications.

Specific knowledge (e.g. occupational) might have a decreasing half-life, but soft skills such as teamworking, communications and self-efficacy do not. Neither, one hopes, do literacy and numeracy.

The EC says over and over again that policy development and monitoring is hampered by a severe lack of internationally comparable data sets in adult learning. This is true, but we should all have serious doubts about the capacity or willingness of many or even most European countries to engage in the rigourous data collection required to make this dream a reality. In adult learning is very difficult to monitor, and it is not clear what the gains from adult learning are. Why work really hard to count something if you're not sure that it's going to add up to anything? The EC argument is that if we count more we will find out that it is adding up. It's a bit chicken and egg, isn't it?

Should we recommend some sort of focus on data regarding language skills or language courses for immigrants? I think so. This is an example of a policy priority-led the theme or recommendation, rather than one led by the availability of existing data.

What about an indicator or data contributing to the "one step up" ideal?