165additional control variables were studied: type of intervention evaluated and whether the evaluated program was ongoing or completed. Type of intervention was coded in accordance with three possible types of interventions: Financing intervention, for example, grants or subsidies; Rule changes, such as permission to research new materials or regulatory relieves; and Information efforts, such as training in patent application or entrepreneurship. The evaluations examined concerned both completed and ongoing initiatives, which were coded by the dummy coding ongoing or completed intervention.
4.3
Evaluating Actors and Employed Methods
The next step in the analysis was to study the variation in evaluation judgments shown when divided based on the different types of actors. Among the 56 evaluations carried out by consultants, 45 (80.4%) were positive, the remainder neutral. For other types of actors, the distribution was much more even between the judgments distributed. Among other agencies, 11 (35.5%) evaluations were positive, 15 (48.4%) neutral, and 5 (16.1%) negative. Among researchers, 7 were positive, 7 neutral, and 1 negative; and among self-evaluations, there are 4 positive and 4 neutral evaluations. The results thus show that consultants provide considerably more positive evaluations than other actors. Figure 5 shows the frequencies of each judgment based on the actor conducting the evaluation.
To probe whether this correlation is statistically significant, Fischer’s exact test was performed on the actor and judgments of evaluation variables ( p-value: 0.001).
The dataset shows no major variation in the evaluations depending on which methods or data type they utilized, but great variation depending on the evaluating E. Collin et al.
16645 s 40 n oita 35 a ula 30 v 25 e f o 20 15 re b 15 11 11 m u 7 7 10 N 5 4 4 5 1 0 0 0 Positive Neutral Negative Positive Neutral Negative Positive Neutral Negative Positive Neutral Negative
Evaluative Consultants Self-evaluation Researcher Authority
Fig. 5 Reviews of valuations of Swedish growth and innovation policies by evaluative actor
38 40 35
s
n
30
oita
25
ula
v
e
17 20
f
o
12
re
15
b
m
7 7 7 10
u
5
N
4 3 3 2 2 5 1 1 1 0 0 0 0 0 0
Qualitative methods
Mixed Methods 1
Mixed Methods 2
Qualitative methods
Mixed Methods 1
Mixed Methods 2
Qualitative methods
Mixed Methods 1
Mixed Methods 2
Qualitative methods
Quantitative contrafactual
Mixed Methods 1
Mixed Methods 2
Quantitative contrafactual
Quantitative contrafactual
Quantitative contrafactual
Quantitative descriptive methods.
Quantitative descriptive methods.
Quantitative descriptive methods.
Quantitative descriptive methods.
Evaluative Authority
Consultants
Self-evaluation
Researcher
Fig. 6 Methods used in Swedish growth and innovation policy evaluation by evaluative actor
actor type. Figures 6 and 7 show frequencies of methods and data type based on the actor conducting the evaluation. The figures show a clear propensity among consultants to use qualitative and mixed methods while evaluative agencies have a slightly more even distribution between methods. The high number of qualitative methods could be attributed to the fact that a lot of the evaluations are conducted on ongoing projects, which makes quantitative approaches, often based on measuring effects Evaluating Evaluations of Innovation Policy: Exploring Reliability,. . .
16743 45 40 s n oita 35 ula 30 v 25 e f o 20 re 14 13 b 15 m