Taking the multifaceted evaluation tool into use developed for open youth work (2018–2021)

The length of one data-gathering cycle is two years

The developed evaluation tool consists of four different focal points and one for summarising the results. The aims for each evaluation focal points are discussed — for example, the evaluation questions which are going to be used are carefully negotiated with the piloting organisations, after which the chosen questions are used and fine-tuned for later use, if necessary. The regular meetings of the piloting organisations play an important role. One data-gathering cycle during the piloting takes two years, while the length, order and timing of each different focal point is tested so that the tool can be launched henceforth to last the same amount of time, when enduring use of the tool starts afterwards. Because of the challenging, careful and laboursome nature of the evaluation, each focal point is evaluated no more than once every two years — unless there is a specific need for an organisation to obtain results from a specific focal point more frequently.

Focal points of the evaluation tool

The evaluation tool use has several strengths:

A) The data the evaluations are based on is gathered from different stakeholders: from young people, bodies offering youth work and decision-makers.

B) The data is gathered with a broad variety of tools such as interactive evaluation, survey, peer-, and self-evaluation.

C) The aim is to cover different kinds of aspects of youth work.

As mentioned above, the evaluation cycle consists of four different focal points:

1. The utilisation rate of the service

2. The responsiveness of the service to the needs

3. Young people’s satisfaction with the service and an overview of its impact

4. Results of accessibility, equality mappings, and attempts to bring about change.

Each of these are important in of themselves — each offering interesting results and key figures — but beyond these, there is a possibility to compose a fifth focal point by summarising the aforementioned information gathered into an additional one, which, in this evaluation model, is referred to as “The functional efficiency of the service” (see figure 1). The results of functional efficiency pertain to questions of quality (outreach, operability, usability and relevance) and impact, as well as the actions taken to improve the equal accessibility of the service.
The figure for pointing out the functional efficiency of the service is counted by summarising the other focal points. Reaching acceptable level requires success in every aspect, but the form this outcome takes can only be found out within the piloting organisations during the process. Each piloting organisation also takes part in validating the results of other organisations.

Figure 1. The figure pointing out the functional efficiency (max 100%) of open youth work service consists of four different focal points: service utilisation rate (20%), responsiveness of the service to needs (20 %), satisfaction with the service (20 %), and an overview of the impact (20 %). Aspects of equality are considered both as its own area of evaluation (20%) and is also included in the other focal points also.

Piloting timetable for two years from September 2018 to August 2020 (progression 3/5 focal points) (updated 15.11.2019)

1. The utilisation rate of the service

Defining the utilisation rate of the service means that the amount and role of young people using the service is considered in relation to the nature of the youth work process, which is evaluated, for example in terms of working hours available. This is important because there are very different kinds of open youth work services. Some of the services offer, for example, individual contact with young people, while some are more community oriented. In practice, this means there is also a written description, which goes beyond the numbers describing only the volume of visitors. Other elements looked at are: the aims of the service, its location, descriptions of other similar services, or a shortage of those in the area, the online, individual, group and community processes being offered, and the opportunities for young people to participate and voice their ideas for development. It also describes what kind of frame of action is needed for successful work: how many young people can attend at once, what types of space and personal resources are available, the educational needs of youth workers, the needs for development already known and related to, for example, the equal accessibility of the service.

Timetable:

The volume measurements of a three-month period during 2018, descriptions of utilisation rate January 2019, self-evaluation and peer validation of the results February-November 2019.

2. The responsiveness of the service to the needs

Services are often only evaluated from the perspective of “whether or not actions are carried out correctly”, instead of considering whether those realised are in fact the correct ones from the start, and even in the light of the future. In this focal point, young people, people producing the youth work services and decision-makers have opportunities through negotiation to both guide the direction in which the service ought to be developed, and to have a say about how to improve the service right here and now. For this focal point, interactive evaluation is organised. The method used is referred to as Deliberative Discussion Day (DDD).

Timetable:

Training for the DDD method was organised in Turku in February 2019. Discussion Days will be held in every pilot organisation between April 2019 and December 2019.

3. Young people’s satisfaction with the service and an overview of its impact

A questionnaire is used for gathering data about the levels of satisfaction and experienced impact related to the youth work services. Piloting organisations have worked together twice: firstly, to provide ideas for question development and then to modify the questions the researcher and project coordinators had been formulating The pilot organisations also decided that the questionnaire will be directed at the decision-makers so that there will be a possibility for comparison, as an opportunity for the decision-makers to assess how satisfied  young people are with the offered youth work service and how meaningful the services are from the impact point of view.  In the current evaluation round, the questionnaire is realised according to the focal point three (see the figure 1). In the forthcoming evaluation rounds, starting from autumn 2020, the purpose has been stated earlier as follows: the comparison of the results gathered from young people and decision-makers is a fruitful source for more of discussion to be held on the deliberative discussion days. 

Timetable:

The questionnaire was put together during the second half of 2019. The questionnaire is to be launched in spring 2020.

How was the evaluation tool developed?

In 2013-2015 there was a research project called Network-based Development of Youth Work Statistics and Key Characteristics where two kinds of services were evaluated in collaboration with young people, youth workers and decision-makers: in the city of Lahti the objects of evaluation were face-to-face services in the southern region, and online youth work services at the national level. The National Youth Centre Association later offered an opportunity to study the impact of international youth work. The results of the both projects showed that there is a need for a new kind of evaluation tool. Even when working well, youth work services were unable to document the importance of their work. Some of the well-working services were even in danger of losing their funding due to a lack of evidence about:

• the needs of young people
• the quality of the process,
• the impact of the service or
• a need for more resources (there was no way of showing how many young people were left out of the service because of the services’ current limitations).

The knowledge basis that decision-makers were mainly using was the number of visitors, which is not comparable due to the particular nature of the different types of youth work processes and the amount of young people who can reach the service within a certain geographical area, or online. With this new multi-faceted evaluation tool for open youth work, the work is evaluated from different perspectives within the one evaluation process.

More information

For more information about the tool and the related development work, see the following publications.

Gretschel, Anu (2017b) Nuorisotyön tarpeellisuutta kuntapäättäjille perustelemassa: ehdotus arvioinnin menetelmäpaketiksi. (Arguing the necessity of having youth work to decision-makers: suggestion about what kind the evaluation tool could be). In Tommi Hoikkala & Johanna Kuivakangas (toim.) Kenen nuorisotyö? Yhteisöpedagogiikan kentät ja mahdollisuudet. (Who the youth work belongs to? The fields and opportunities of community educator.) Humanistinen ammattikorkeakoulu, julkaisuja 42 & Nuorisotutkimusverkosto & Nuorisotutkimusseura, julkaisuja 196. Helsinki: Humanistinen ammattikorkeakoulu & Nuorisotutkimusverkosto/Nuorisotutkimusseura, 295–304. Available only in Finnish.

Gretschel, Anu (Ed.) & Cupar, Tina & Linno, Merle & Kiilakoski, Tomi & Hästbacka, Noora & Korhonen, Antti (2017) Studying the impact of international youth work – Towards developing an evaluation tool for youth centres. Helsinki: Finnish Youth Centres Association. Suomen nuorisokeskusyhdistyksen julkaisuja, 1/2017. Finnish Youth Research Society/Finnish Youth Research Network, Internet Publications 113.

Gretschel, Anu & Korhonen, Antti (2019) What the European co-operation teach us about the nature of tools used in evaluation of (open) international youth work. International Journal of Open Youth Work, Issue 3, 83-92. 

Gretschel, Anu (2016) Nuorisotyön verkko- ja lähipalveluiden arvioinnin kehittäminen. (Developing evaluation for youth work services both as near service and online). In Anu Gretschel, Pirjo Junttila-Vitikka ja Anne Puuronen. Suuntaviivoja nuorisotoimialan määrittelyyn ja arviointiin. (Guidelines for defining and evaluating the youth affairs sector) Verkkojulkaisuja 96. Helsinki: Nuorisotutkimusverkosto/-seura, 63–113. Abstract in English is included in the publication.

Cooper, Sue & Gretschel, Anu (2018) Evaluating youth work in its contexts. Teoksessa Pam Alldred, Fin Cullen, Kathy Edwards and Dana Fusco (toim.) SAGE Handbook of Youth Work Practice. Lontoo: Sage, 608–622.