1679 u 8 10 N 4 4 4 4 3 2 2 5 0 Subjective Subjective Mixed Subjective Objective Mixed Objective Subjective Objective Mixed Objective Mixed

Evaluative Consultants Self-evaluation Researcher Authority

Fig. 7 Data used in Swedish growth and innovation policy evaluations by evaluative actor

through indicators such as employment, company turnover, or patents, somewhat difficult to perform. To rule out other potential explanations and to map additional correlations, a logistical regression analysis was performed. The results, which are given in appendix 9.1, show high odds ratios and statistically significant p-values between a dummy for positive judgment calls in the evaluations and a dummy for the actor type consultant. We also observe a negative relationship between the method category qualitative methods and less positive evaluations, meaning that qualitative approaches are more like to result in positive evaluations. Notably, the “consultant effect” in terms of the strong correlation between the type of evaluating actor and their evaluations of policy remained statistically significant ( p ¼ 0.02), indicating that difference in, for instance, methods or data utilized in the evaluation, cannot explain the difference between different evaluators.

5

Discussion

Our results show that the vast majority of evaluations are positive, and few make use of quantitative evaluations in which real effects can actually be measured. Moreover, consultants are significantly more likely to conduct positive evaluations relative to the other evaluating actors. This does not seem to be due to the consultants using different methods, utilizing certain types of data material, or evaluating a certain type E. Collin et al.

168of political intervention. What affects the results rather seems to be that it is specifically consultants that carry out the work.

The strong positive relationship between consultants as actors conducting evaluations and an evaluation being positive is the major finding of our study. Indeed, it is not a trivial finding. We have yet to confirm a causal relation between the two. In what follows, we, therefore, discuss possible explanations of the results and draw implications for future research.

5.1

What May Underlie Differences in Evaluations

of Innovation Policy?

One plausible explanation could be found in public choice theory, according to which government agencies have an innate interest of looking out for themselves, partly through indicating positive results of their work (Niskanen Jr, 1994). Evaluated agencies can hence be expected to have strong incentives to choose evaluators they expect to give positive evaluations, since this gives them arguments for continued funding and support. Consulting companies are therefore likely, through competitive pressure, to become inclined to please their clients, which seems to mean that they come up with positive evaluations.

Vice versa, it can be argued that reviewing agencies such as the Swedish National Audit Office and Growth Analysis may have incentives to examine other agencies’ efforts carefully and potentially more critically in order to identify problems and shortcomings and thereby justify their assignment as an examining agency.

Another explanation as to why consultants provide significantly more positive evaluations compared to other actors could be that they are hired to evaluate interventions that agencies already know have yielded positive results and therefore are considered easier to evaluate. Interventions that are more difficult to evaluate, and therefore often detect neutral or negative results, would, according to the same logic, be entrusted to evaluation agencies whose opinions should thus differ according to what we have observed. What speaks against such an interpretation would be that the positive evaluations studied often use methods that do not make it possible to draw conclusions on a scientific basis.

A potentially more reasonable interpretation of our results would be that evaluators are aware that the result affects the possibilities of obtaining further assign-