Data Management for Evaluation
Big Data, Small Data, Short Data, Long Data?
When it comes to evaluation one of the main pieces that can often determine whether an evaluation is even possible or not is data, but data often sounds like a boring or complicated topic, and so we instead often times focus our attention only on how we are going to implement great programmes that create jobs or better preserve our landscape and biodiversity. But how can we assess the results and impacts of these great ideas? We need data. And most likely we need a lot of it. But while this may sound like a daunting task if prepared properly and in advance Member States can make sure they have the data needed in order to make sure they know if they really are achieving what they intended.
The legal requirements in the 2014-2020 programming period underline the need for the Member States and programmes to put into place more systematic approaches for evaluation related data collection and the provision of data to evaluators. These problems suggest the need for a comprehensive data management strategy developed by the Member States and programmes.
When conceptualising evidenced-based evaluations of Rural Development Programmes (RDPs), one must not begin with the technical details but instead, first assess the overarching purpose of data management for evaluation. The aim of data management for evidence-based evaluations is to demonstrate the progress, achievements, results, impacts, relevance, effectiveness and efficiency of the EU’s rural development policy. With this stated, one quickly realises that evaluations are both vital to the policy process and challenging endeavours.
RDP evaluations begin with focus area-related and horizontal common evaluation questions (30), which have to be answered at regular milestones during the programming period and reported to EU institutions to meet the information needs at the European level. Evaluation questions are answered by means of a set of common indicators agreed upon between the EC and Member States. The common indicator list encompasses different types of data sets, which are linked to a variety of data sources.
- Output indicators (26) collected from the operations database on beneficiaries.
- Result indicators (25) including the complementary ones: collected through different data sources such as the operations database, secondary data sources, and field research by evaluators.
- Target indicators and performance framework indicators: a subset of output and result indicators, which must be thoroughly matched and synchronised with the operations data.
- Context indicators (45) collected from regional/national/ EU statistics.
- Impact indicators (16): form a part of the context indicators set and collect macro data (on the territory, and the sector). Impact indicators are updated together with context indicators. However, impact indicators are also used in the RDP assessment.
The comprehensive sets of common evaluation questions and the related indicator groups were designed to capture the broad thematic scope of the Rural Development Programmes represented by 18 focus areas and numerous key measures. These indicators also reflect an increased focus on a results-oriented approach for programmes using EU funds represented by target indicators, the performance framework and the requirement to isolate programme-specific effects measured by result, and impact indicators resulting from other influencing factors.
The set of common indicators are designed to meet the reporting needs at the EU level (for the EC, Council and European Parliament), but they were not devised for a fully developed evaluation at the national or local levels. Therefore, programmes need to develop programme-specific elements to assess aspects that are of particular interest to stakeholders. The European Commission encourages programme bodies to develop indicators ‘in addition’ to common indicators whenever the common indicators cannot fully answer the evaluation questions in a satisfactory manner. A list of roughly 20 to 30 agricultural, environmental and social data sources are necessary to answer the common evaluation questions taking into account programme-specific elements. This leads to specific data needs and an increase in the complexity of the data sources needed for evaluation.
The figure illustrates how the various indicators relate to objectives at different levels and where data for these indicators can be drawn from.
When building a data strategy some key topics for data management should be considered. This table breaks these topics down by those which are cross-cutting which relate to data management for all themes and then further considers some specificities by theme.
This practical example can shed light on how one might look at evaluating a specific evaluation question related to a specific focus area in this case (FA 2A).
Find more information on data management in the Helpdesk’s eLibrary.
For more information on indicators please refer to DG AGRI’s ‘CAP Interactive Dashboard’ recently updated with 2017 data.