The Impact of Living Lab Methodology on Open Innovation Contributions and Outcomes

In academic theory, open innovation has been regarded as the norm for studying innovation management ever since Chesbrough’s seminal and widely cited Open Innovation: The New Imperative for Creating and Profiting from Technology (2003). However, in practice, a balance should be found between open and closed innovation, which calls for innovation management approaches that deal with finding this balance (Lakhani & Panetta, 2007). Although many principles and phenomena from the open innovation literature, such as economic spill-overs (Arrow, 1962) and dynamic capabilities (Teece & Pisano, 1994), were already described a long time ago, and open innovation as a domain has already fostered a large body of research (West & Bogers, 2013), many companies and innovation practitioners are still struggling with the concrete implementation of strategies to cope with these distributed innovation processes (Chiaroni et al., 2011).


Introduction
In academic theory, open innovation has been regarded as the norm for studying innovation management ever since Chesbrough's seminal and widely cited Open Innovation: The New Imperative for Creating and Profiting from Technology (2003).However, in practice, a balance should be found between open and closed innovation, which calls for innovation management approaches that deal with finding this balance (Lakhani & Panetta, 2007).Although many principles and phenomena from the open innovation literature, such as economic spill-overs (Arrow, 1962) and dynamic capabilities (Teece & Pisano, 1994), were already described a long time ago, and open innovation as a domain has already fostered a large body of research (West & Bogers, 2013), many companies and innovation practitioners are still struggling with the concrete implementation of strategies to cope with these distributed innovation processes (Chiaroni et al., 2011).
Three main issues and gaps will be examined in this article.First, there still is a lack of adequate innovation management models for implementing open innovation (Lichtenthaler, 2011).Second, Enkel, Gassmann, and Chesbrough (2009) found that few studies try to put forward measurement systems and key performance indicators to evaluate open versus closed approaches.Third, Huizingh (2011) argues that open innovation became an umbrella that connected a range of already existing activities.However, most of the principles and research relating to open innovation are tailored to large companies with abundant resources (van de Vrande et al., 2006), despite the fact that SMEs are usually more flexible, less formalized, and quicker to make decisions -meaning that they present many opportunities for the implementation of open innovation (Lee et al., 2010).
Therefore, with this article, we will assist in filling these gaps by investigating living lab projects as an organized (as opposed to an ad hoc) approach to open innovation consisting of real-life experimentation and active user involvement by means of different methods involving multiple stakeholders (Leminen et al., 2014;Schuurman, 2015).More specifically, we will explore the value of a living lab approach for open innovation in SMEs.
Open innovation scholars as well as practitioners are still struggling with the practical implementation of open innovation principles in different contexts.In this article, we explore the value of a living lab approach for open innovation in small and medium-sized enterprises (SMEs).Using a case study approach, we compared 27 SME projects conducted by iMinds Living Labs from 2011 to 2015.The results suggest that a real-life intervention and a multi-method approach -both of which are methodological characteristics of living lab projects -increase the chance of generating actionable user contributions for the innovation under development.Moreover, the results also suggest that a living lab project yields maximal value when evolving from concept towards prototype.Besides these exploratory findings, this article also demonstrates that living lab projects are a perfect "playground" to test and validate assumptions from the open innovation literature.
If you look at history, innovation doesn't come just from giving people incentives; it comes from creating environments where their ideas can connect.
Steve Johnson Science author and media theorist

" "
The Impact of Living Lab Methodology on Open Innovation Contributions and Outcomes Dimitri Schuurman, Lieven De Marez, and Pieter Ballon We conducted a comparative case study analysis of 27 SME projects in the domain of new media and ICT conducted by iMinds Living Labs from 2011 to 2015.Our aim was to examine the impact of the methodological setup of living lab innovation projects on the innovation contribution of end-users and on the eventual outcome of the new product development process.In this way, we assist in addressing the need for impact assessment and measurement systems of open innovation approaches, and we demonstrate the viability of a living lab project as a "playground" to test and validate assumptions from the open innovation literature.

Living Labs as a Structured approach to Open Innovation
The first premise of open innovation is that, from the perspective of a single firm (the usual level of analysis in open innovation research), opening the internal innovation process of a firm yields extra value (Chesbrough et al., 2006).According to Chesbrough and Bogers (2014), the critical conceptual distinction between the previous literature on spillovers in innovation is that open innovation transforms these spillovers into inflows and outflows of knowledge that can and should be purposively managed.Many open innovation studies deal with the economic (pecuniary) implications and opportunities provided by external sources of innovation and commercialization, and mainly focus on the revenue-generating practices from a firm perspective (Brunswicker & Vanhaverbeke, 2014;van de Vrande et al., 2009).Enkel and colleagues (2009) conclude that the future of innovation processes lies in an appropriate balance between open and closed innovation approaches, because too much openness can lead to a negative impact on companies' long-term innovation success, loss of control, and loss of core competences, whereas a too closed innovation approach does not serve the demands of increasingly shorter innovation cycles and reduced time-to-market.Pisano and Verganti (2008) also add to this discussion by identifying four types of open innovation collaboration models, based on the governance model (hierarchical versus flat) and participation (open versus closed), which indicates that there are many options and trade-offs between "open" approaches.However, there seems to be a gap between theory and practice given that multiple studies have indicated that many companies struggle with implementing open innovation practices (Lichtenthaler, 2008;van de Vrande et al., 2009), and that there are major differences between different firms and organizations (Laursen & Salter, 2006).
From the introduction, we gathered that there is a lack of structural approaches and guidelines for implementing open innovation, a lack of measurement systems that allow impact assessment, and a lack of research into open innovation in SMEs.One specific approach that offers a structured approach to open innovation and that has been used specifically by startups and SMEs are "living labs" (Schuurman, 2015).Living labs are put forward as an institution to overcome the socalled "European Paradox" or the gap between research leadership and (commercial success of) innovation (Almirall & Wareham, 2011).Living labs are physical regions or virtual realities, interaction spaces, in which stakeholders form public-private-people partnerships (4Ps) of companies, public agencies, universities, institutes, users, and others that follow the philosophies of open and user innovation to collaborate for improving, developing, creating, prototyping, validating, and testing of current or new technologies, services, products, and systems in real-life contexts (Leminen et al., 2012), and are driven by two main factors: involving users in the early stages of the innovation process and experimentation in real-world settings that aim to provide structure to user participation (Almirall & Wareham, 2008).Therefore, living lab projects are a specific case of open innovation where companies open up their innovation processes to users or customers (Schuurman et al., 2013), which can be linked to the user innovation paradigm (von Hippel, 1976(von Hippel, , 2009)).
In terms of methodological deconstruction of the living labs approach, the work of Pierson and Lievens (2005) remains unique in describing the different phases of a living lab project: i) contextualization, ii) selection, iii) concretization, iv) implementation, and v) feedback.However, the methodological basis of these five phases is left unexplored, as are the actual outcomes and added value when engaging in living lab projects.Schuurman (2015) suggested that this methodology is very similar to a quasi-experimental design, with a pre-test, a real-life intervention, and a post-test (Figure 1).
By adopting this methodological approach, living lab projects would be able to overcome the barriers to user contribution, because it implies triangulation of different methods and a real-life contextualization (Frissen, 2000).However, only very few studies try to assess the impact of the methodological design of living lab projects (Veeckman et al., 2013).For living labs, there generally is a gap in measurement systems; as Katzy and Turgut (2010)

Methodology
Based on a comparative case study analysis, we wanted to assess the impact of the methodological set-up on innovation contribution of end users and on the outcome of the innovation project.To gather and analyze empirical data, we used the case study technique, which is a common method in social sciences to describe and explore poorly understood processes and events.Case studies are especially suited to such work because of their emphasis on detailed contextual analysis of a limited number of events or conditions and their relationships (Eisenhardt, 1989).
Within a case study design, careful consideration should be dedicated to the selection of the cases to be included in the analysis (Dion, 2003).We analyzed all SME living lab projects that have been carried out by iMinds Living Labs in the period from 2011 to 2015.This approach yields a slightly larger sample of cases than usual, but enables a more quantitative, yet still exploratory, analysis coupled with more in-depth qualitative investigation.Therefore, the case studies are neither prospective (i.e., criteria are established and cases fitting the criteria are included as they become available) nor retrospective (i.e., in which criteria are established for selecting cases from historical records for inclusion in the study), but can be labelled as comprehensive for the analyzed time frame (2011)(2012)(2013)(2014)(2015), which is in line with the "sustained period of time" criterion for data collection of Shepard (2001).The Impact of Living Lab Methodology on Open Innovation Contributions and Outcomes

Dimitri Schuurman, Lieven De Marez, and Pieter Ballon
As researchers from iMinds, we had access to the following data sources from the iMinds Living Lab projects: i) transcripts of semi-structured interviews with representatives of the SMEs and ii) all project deliverables.Within the living labs community, iMinds Living Labs has played an important role in developing, applying, and studying the living labs approach, and is regarded internationally as a "best practice" example (Almirall et al., 2012;Dell'Era & Landoni, 2014).This view is reinforced by the fact that iMinds Living Labs also acts as secretary of the European Network of Living Labs (ENoLL; openlivinglabs.eu).Therefore, the availability of rich data, first-hand experiences, and the leading role of iMinds Living Labs in the living labs landscape justifies the choice of these 27 projects (Yin, 1984).
Later in the article, we briefly summarize each case; more detailed descriptions are provided by Schuurman (2015) and the website of iMinds Living Labs (tinyurl.com/zzowv4m).
For this article, we gathered the following data for the 27 projects: i) the presence of the living lab methodology, ii) evolution in terms of stages of new product development (NPD), iii) user contribution generated by the living lab project, iv) and outcome of the innovation.

Presence of the living lab methodology
If the project included a quasi-experimental design (i.e., pre-test -real-life intervention -post-test) and a multi-method user involvement approach, this criterion was coded as "yes".If only one or none of these characteristics was present, it was coded as "no".

Evolution in terms of NPD stages
For all projects, the evolution of the innovation in terms of NPD stages was logged during the interviews with the project instigators.We discerned between the following stages, based on Jespersen (2008): ideaconcept -prototype -pre-launch -launch -postlaunch.We recoded the project into three categories (iMinds, 2015), which are also used to describe the type of living lab projects, which are summarized in Figure 2: 1. Exploration: a project where the innovation starts at the idea or concept stage and ends in the idea or concept stage.These projects focus on exploring new knowledge for innovation development.
2. Experimentation: a project that includes the prototype stage.These projects focus on experimenting with the innovation.
3. Evaluation: projects that start at the pre-launch stage or later.These projects focus on evaluating the innovation.

User contribution generated by the living lab project
This aspect indicates the instigators' perceptions of what has been done with the user contributions generated during the living lab project.We discern three categories of contribution -during, after, or none -to indicate whether the results were used to modify the innovation during or after the project or to indicate that the results did not modify the innovation.

Outcome
The final variable identifies the current status of the innovation (as of May, 2015).On the market indicates that the innovation is launched and available for end-users, pipeline indicates that the innovation is still planned to be launched, but is not available yet, and reoriented is used when the instigator has decided not to launch the innovation.

Results and Discussion
Table 1 below gives an overview of the variables for all the 27 projects.In terms of the methodological approach, we notice that the majority of the projects (14 of 27; 52%) did not include all the methodological elements of an "ideal" living lab project.Nineteen out of 27 projects contain a real-life intervention (70%), and only 13 out of 27 (48%) also include a post-assessment.
These methodological elements can be regarded as forms of user contextualization, which is proposed as a means to overcome barriers related to user involvement, or the so-called "real-life experience" of living labs (Frissen, 2000).Another method to overcome these barriers was to include triangulation of different methods, which reflects the "multi-method" characteristic of living labs.The majority of the projects adhere to this criterion, with 23 out of 27 projects (85%) containing triangulation of user involvement.
The absence of some of these characteristics can be ascribed to various reasons.First, startups and SMEs are constrained by time and budget, which did not allow them to have all elements in a project.Second, the NPD stage also impacted the possibility of a real-life intervention.Projects that remained in the exploration stage have greater difficulty in organizing a field trial given that there is no working prototype yet.As a solution, a proxy technology assessment (Pierson et al., 2006), which means a simulation of the innovation by means of existing technologies, can be used (e.g., the Veltion and Unicorn projects), but this requires extra effort and expertise.
In terms of the types of living lab projects, 7 projects (26%) can be labelled as exploration, which means that, at the end of the project, there was no working prototype, 15 (56%) are experimental in nature, including the prototype stage, and 5 (18%) were coded as evaluation because these projects consisted of innovations that were already in a pre-launch stage at the start of the project.
The first variable that refers to an outcome of the living lab project is user contribution: it indicates what happened with the user contributions generated during the project.For 13 cases (48%), modifications were made during the project; for 7 cases (26%), they were made after the project; and in the remaining 7 cases, instigators stated that they did not use the living lab results to modify the innovation.
The second outcome variable refers to the market introduction of the innovation after the living lab project.In total, 10 innovations (37%) were launched on the market, 8 innovations (30%) were still in development, and 9 instigators (33%) reoriented themselves and abandoned the innovation development.
These results indicate that, in nearly three-quarters of the projects, the user contribution had an impact on the innovation development, but that iteration of the innovation development during the living lab project, or the so-called "pivots" out of the lean startup literature, is less common and occurred in only about half of the projects.However, when comparing the projects in which the "full" living lab methodology was used, there are some pronounced differences, as shown in Table 2 and described below.
For the cases that did contain all methodological living lab elements, only 2 did not generate user contributions that led to modifications in the innovation.Stated differently, 85% of these projects generated actionable user contributions, and more than half of the cases included iterations during the project.In comparison, for projects that did not contain all methodological elements, two-thirds generated actionable user contributions.In terms of outcome, the "real" living lab projects For the other 14 projects, the number of reoriented cases is twice as high (6 or 43%), but the number of successful market introductions is also slightly higher (6).A potential explanation could be that the living lab projects generated more input for the innovation, which requires more time to implement these changes and induces a longer time to market.
Finally, when looking at the stages in the NPD process, the majority of the "full" living lab projects were labelled as "experimentation", whereas the other projects were more evenly distributed among the three categories.Given that the experimentation projects lend themselves more to incorporating all methodological living lab elements, our results seem to suggest that the best time for a living lab project is when advancing from concept to prototype, or at least to include this stage in the project.
Because of the time and budget constraints of startups and SMEs, this was not always realistic within the cases we studied.One of the strategies that was used to overcome this issue was to carry out multiple projects in sequence.As an example, the Coxo case was an exploratory project that did not include all methodological elements, but was aimed at studying all stakeholders in the complex ecosystem of the innovation.The Planidoo project was the follow-up project of Coxo, with the innovation carrying a changed name (something which followed out of the results of this first project).This project did carry all living lab characteristics, because it started in the pre-launch stage and evolved towards market introduction during the project.This illustrates that a lean and agile approach and attitude are necessary when carrying out these type of projects with SMEs, both from the researchers and from the project instigators.

Conclusion
Within this article, we looked at 27 innovation projects from Flemish startups and SMEs carried out within the iMinds Living Labs constellation.To conclude, we summarize and translate our findings in three propositions.First, the discussed living lab projects are aimed at opening up the company boundaries towards user con- When taking into account the methodological set-up of these projects, it seems that a real-life intervention, a quasi-experimental design, and a multi-method approach increase the chances of user contributions that lead to modifications in the innovation, as this was the case for 85% of the projects against 64% for the projects that lacked one or more of these elements.
Proposition 2: The living lab characteristics "real-life intervention" and "a multi-method approach" foster valuable user contributions Although the literature on living labs stresses the importance of real-life and multi-method research approaches, few actual evidence-based arguments can be found that support this claim.However, our research supports this proposition: within our studied sample, not all projects displayed all the methodological characteristics, but those that did scored higher in terms of user contribution.Moreover, the projects with the most positive outcomes could be characterized as "experimentation", which indicates a transition from concept to prototype during the living lab project.

Proposition 3: The ideal maturity level of an innovation for a living lab project is the transition towards a testable prototype
This proposition is also in line with the emphasis that is put on the "real-life" testing within the living labs literature.Moreover, our findings also support the thesis that triangulation and real-life experience lower the barriers for user contribution, as was suggested by Frissen (2000).

Future research
Future research should test these propositions and investigate these findings in greater detail.Other variables that might play a role should be taken into

The Impact of Living Lab Methodology on Open Innovation Contributions and Outcomes
Dimitri Schuurman, Lieven De Marez, and Pieter Ballon account, such as the attitude of the instigator, the characteristics of participating end users, or the nature of the innovation in development.This broader approach would enable assessment of their impact on the outcomes of living lab projects.In any case, the characteristics of a living lab, where a given constellation carries out multiple innovation projects following a given methodology, allows researchers to test hypotheses on a supra-case level.This facilitates grasping the mechanics and nature of open innovation processes and phenomena beyond a single project.Moreover, this also allows investigation of the iterative learning processes that take place on the constellation level when conducting multiple open innovation projects.
We regard this as the way to go for living lab researchers and practitioners, in order to more clearly understand the mechanisms by which living labs operate and to assess the added value they are able to generate.This type of knowledge is necessary to further develop the potential of living labs and to outgrow their status as a promising, but fuzzy innovation concept.
state, a valid research methodology stillThe Impact of Living Lab Methodology on Open Innovation Contributions and OutcomesDimitri Schuurman, Lieven De Marez, and  Pieter Ballon needs to be developed for the innovation performance of individual living labs.According to them, the measurement of the efficiency of living lab processes and structures would serve two purposes: i) legitimating the (EU) research budget that has been used to stimulate the establishment of living labs and ii) enabling modification of the concept, or at least certain aspects of it.Therefore, by studying living lab projects that are an emanation of open innovation, a more systematic study of open innovation processes and principles is enabled, which may help overcome one of the key problems with open innovation: conceptual ambiguity(Dahlander & Gann, 2010).Moreover, in terms of the typology ofPisano and Verganti (2008), the studied living lab projects can be considered as collaboration projects with hierarchical governance, given that the project instigator (the SME) decides how the user contributions are implemented within the innovation in development.In terms of participation, the studied living lab projects share both open and closed characteristics: in some research steps, an open call is sent to users to participate (e.g.surveys), whereas for other steps, specific user profiles are recruited for participation (e.g., in cocreation sessions).By regarding the methodological aspects and characteristics of living labs as structural elements of open innovation in SMEs, the impact generated by this type of open innovation project can be explored.This approach also serves as a test of a living lab as a structural approach for implementing open innovation, which fills the gap of open innovation research into SMEs.

Figure 1 .
Figure 1.Methodological design of a living lab project mapped against the three phases of a quasi-experimental design

Figure 2 .
Figure 2. The three types of living lab projects mapped against the six stages of new product development

Table 2 .
Comparison of variables and methodological differences across the 27 cases

The Impact of Living Lab Methodology on Open Innovation Contributions and Outcomes
DimitriSchuurman, Lieven De Marez, and Pieter Ballon

Table 1 .
Overview of the 27 living lab projects www.timreview.caTheImpact

of Living Lab Methodology on Open Innovation Contributions and Outcomes
PhD titled Diffusion of ICT Innovations: More Accurate User Insight for Better Introduction Strategies.His main expertise is in the development of "segmentation forecast" tools for prior-to-launch adoption potential forecasts for new media and ICT innovations.He continuously seeks to explore new methodologies and understand emerging media use patterns and the impact of new media and ICT and making media innovation more user-centric.At the department of Communication Studies, he founded and coordinates the Master's program on New Media & Society.Pieter Ballon is the International Secretary of the European Network of Living Labs (ENoLL).He specializes in business modelling, open innovation, and the mobile telecommunications industry.Formerly, he was Senior Consultant and Team Leader at TNO.From 2006 to 2007, he was the coordinator of the cross issue on business models of the Wireless World Initiative (WWI) that united five integrated projects in the EU 6th Framework Programme.He holds a PhD in Communication Sciences and an MA in Modern History.