163professionals implementing it or a larger society somehow affected by the policy (Vedung, 2009). Yet another way would be to evaluate the efficiency or effectiveness of the policy—focusing on the means spent to achieve a certain result (Vedung, 2009). Within each of these evaluative methods, more distinctions could of course be made. In the current study, we have coded the methods as either quantitative descriptive methods, qualitative methods, quantitative counterfactual (or experimental) methods, or a mix of either the first two or all three of the methods.2
The study results show that qualitative methods are used to the greatest extent among the evaluations studied—qualitative methods occur in 61 of the cases. The second most common is that of mixed methods 1 (quantitative descriptive and qualitative methods), which occurs in 31 of the cases.
2Hence, the five categories of methods in the evaluations are: (1) Quantitative descriptive methods; (2) Qualitative methods; (3) Quantitative contrafactual (or experimental) methods; (4) A mix of 1 and 2; and (5) A mix of 1, 2, and 3.
164E. Collin et al.
61 60 Number of evaluations 50 40 32 30 20 9 6 10 2 0 Quantitative Qualitative Quantitative Mixed Methods Mixed Methods descriptive methods counterfactual 1 2 methods.
Fig. 3 Evaluative methods used in evaluations of Swedish growth and innovation policies
The quantitative counterfactual method was used in 9 of the cases and the qualitative descriptive method was used in 6. In 3 of the cases, mixed methods 2 (quantitative descriptive, qualitative, and quantitative counterfactual methods) were utilized. Figure 3 shows the frequencies of each method in the studied evaluations. The fact that several of the evaluations utilize qualitative methods is an interesting observation. Several of the evaluations examined are not the type of goal and result evaluation usually associated with quantitative methods and the typical evaluation practice that characterizes New Public Management (NPM) (Hood, 1991). Rather, they are largely based on interpretation and understanding of user or stakeholder experiences. For example, in one of its reports, the public expert agency Growth Analysis examined how well state and regional business support responds to policy goals and the needs of entrepreneurs (Tynelius, 2016). This was done by comparing intentions and formulations in different documents with interview results and by interpreting and seeking an understanding of how entrepreneurs and prospective innovators perceive the support. Moreover, it should be mentioned that many of the evaluations studied are so-called mid-term evaluations, in which the evaluator examines whether established processes or application procedures match the goals of the policy. These mid-term evaluations are carried out when a project has begun or is half-finished and thus make it difficult to assess efficiency or effectiveness.
Finally, evaluators often base their reports on a mix of data sources. In our study, such data is defined as a combination of both objective data, defined as independent from the viewer and exemplified by, for instance, index data referring to company turnover, or gathered patents; and subjective data, like self-evaluations of people taking part in projects or other value statements from respondents. More than half of the evaluations studied, 67 of the 110, were based on mixed data. Twenty-three were based on subjective data (again, value statements from participants or beneficiaries) Evaluating Evaluations of Innovation Policy: Exploring Reliability,. . .
165Fig. 4 Data used in evaluations of Swedish
67
70
growth and innovation
Number of evaluations
policies
60
50
40
30
23
20
20
10
0
Subjective
Objective
Mixed
and 20 on objective data (index data). Figure 4 shows the frequencies of each data type used in the studied evaluations. Public policy programs such as innovation policies are often quite complex in nature and studying different types of data to evaluate effects from such a policy hence seems a plausible approach. Apart from the variables presented above, two