Methodology for impact evaluation of digitalisation

  1. Deutsch nicht verfügbar
  2. English

The starting point for this output is the need to evaluate the impact of digitalisation, given that agri-food systems, forestry and rural areas are undergoing a process of digital transformation. However, there is a lack of comparable statistical datasets to carry out robust counterfactual impact assessments at regional or local levels, except for data on fixed broadband coverage.

This is why the DESIRA project developed a qualitative impact evaluation. The methodological approach for such a qualitative impact evaluation of digitalisation is a participatory one and is based on the conceptual framework of Socio-Cyber-Physical (SCPS) system. It links the analysis directly to the United Nation’s Sustainable Development Goals.

The methodology for a participatory qualitative impact evaluation of digitalisation includes two key components:

1. Creation of Living Labs to conduct a Needs, Expectations, and Impact Assessment (NEI)

To apply the participatory impact assessment, 21 DESIRA Living Labs were established composed of agriculture, forestry and rural stakeholders. Their objective was to gather stakeholders to conduct a Needs, Expectations, and Impact Assessment (NEI) of digitalisation.

The first step was for each Living Lab to elaborate their own focal question. These questions allowed Living Labs to frame the scope, identify the specific topic, unit of analysis, geographic area, and possible hypothesis or sub- questions.

The Living Labs applied the concept of Socio- Cyber-Physical Systems (SCPS) as an analytical lens to research and gain insights on its past and present impacts. To perform the NEI assessment, they used a mix of data collection tools: desk research, semi- structured interviews, online surveys, interactive workshops. These were applied in three phases:

  1. Living Labs' context analysis and assessment of main needs
  2. Living Labs' description and visualisation of the Socio- Cyber-Physical Systems
  3. Living Labs’ participatory impact assessment

Comparison analysis across the Living Labs.

The data colleted and analysed in the above three phases was compared across the 21 Living Labs using qualitative content analysis, summary statistics and literature review. The findings were firstly exchanged in a peer-to-peer meeting among Living Labs, then summarised in a Synthesis Report.

Concerning the Living Labs' assessment of impacts, the comparison questions were addressed:

  • What and how has digitalisation impacted the SCPS activities until today?
  • Who benefits, loses, promotes, opposes to digitalisation, and why (under which conditions)? 
  • How has digitalisation contributed to the achievement of the 17 SDGs?
Mapping Digittal technologies: the taxonomy and inventory of digital game changers
Needs, expectations and impacts of Digitalisation in European agriculture, forestry and rural areas: Synthesis report of Living Labs' Asessments
Relevance for monitoring and evaluation of the CAP

This qualitative method based on the creation of Living Labs composed of key stakeholders who work together to answer focal questions using different qualitative and quantitative data collection tools, is an innovative method. It can be used by evaluators to assess the impacts of digitalisation, in the absence of more robust quantitative methods. It can also serve as a method to triangulate the findings of quantitative methods if they exist.

The main benefits of this participatory method are:

  • it is structured, through the creation of LIving Labs that centre their work on a limited number of focal questions;
  • it is a comprehensive, i.e. it relies on a network of stakeholders established across Europe and selected to represent a variety of agricultural, rural and forestry domains;
  • it provides answers to the focal questions that are evidence based and validated by the network of relevant stakeholders.

The method can be used to assess the impacts of digitalisation on processes like supply chain management. It may assess effectiveness, i.e. the ability to achieve (or not) a desired goal or demand, or capacity to successfully execute an activity or process. For instance, the accomplishment of new tasks, the creation of new services and values, the achievement of undesired goals. It may also assess efficiency, i.e. the ratio of resource/output of an activity in a given time.

Another useful element produced by the Living Labs was clustering the impacts qualitatively under enabling and disenabling effects (that influence effectiveness), diminishing and boosting effects (that influence efficiency).

Finally, the method may be used to assess(qualitatively) the links between digitalisation the Sustainable Development Goals (SDGs). In the context of DESIRA, focus group discussions and interviews amongst Living Lab stakeholders identified the frequency of general perceptions on the extent of these contributions towards the SDGs.

Last modification date: 
10/12/2021