Action Research as a Framework to Evaluate the Operations of a Living Lab

The fast pace of technological change and globalization – and the associated increase in access to knowledge – have enabled a growing number of users to engage in the innovation process. Companies, too, have sought out user contributions to their (new) products and services (Bogers et al., 2010). In addition, research has indicated that different kinds of users have different kinds of needs. People have a higher willingness to pay for a product or service that perfectly satisfies their personal needs (Franke & Piller, 2004). To develop these customized products or services, it is possible to let users adapt products themselves and thus become part of the innovation process itself (Franke & von Hippel, 2003).


Introduction
The fast pace of technological change and globalization -and the associated increase in access to knowledgehave enabled a growing number of users to engage in the innovation process.Companies, too, have sought out user contributions to their (new) products and services (Bogers et al., 2010).In addition, research has indicated that different kinds of users have different kinds of needs.People have a higher willingness to pay for a product or service that perfectly satisfies their personal needs (Franke & Piller, 2004).To develop these customized products or services, it is possible to let users adapt products themselves and thus become part of the innovation process itself (Franke & von Hippel, 2003).
Active end-user contribution is one of the building blocks of innovation processes in living labs.Living labs are public-private partnerships established to foster user-driven innovation and are supported by the European Commission through policy measures (Schuurman, 2015).Living lab research consists of user co-creation and experimentation of innovations in reallife contexts (Eriksson et al., 2005).The underlying idea is that people's ideas, experiences, and knowledge, as well as their daily needs for support from products, services, or applications, should be the starting point in innovation (Bergvall-Kåreborn & Ståhlbröst, 2009).The living lab approach is also a form of open innovation, because technology is developed and tested in a physical or virtual real-life context with multiple innovation stakeholders, and end users are important informants and co-creators during this process (Kusiak & Tang, 2006).
The efficiency of a living lab is based on the creative power of user communities.For this reason, it is important for the research activities and living lab operations to be aligned with the expectations of the participating users.Motivated users, willing to participate in research and co-creation activities, are essential for the functioning of a living lab, given that the underlying philosophy is that people's ideas, experiences, and knowledge, as well as their daily needs and wants, should be the starting point in innovation (Bergvall-Kåreborn & Ståhlbröst, 2009).However, in terms of the three layers within living lab activities (Schuurman, 2015;Schuurman et al., 2016), the input from users is In this article, we propose an action research approach to capture and act upon the delights and frustrations of panel members who participate in living lab research in order to optimize the operations of the living lab itself.We used this approach to test the effectiveness of action research in providing guidelines to practitioners to evaluate and design effective and sustainable user involvement processes in living labs.We conducted a focused literature review and an in-depth case study of both the integration of a researcher within the community and the implementation of an action research project within an existing living lab.This living lab is regarded as both a forerunner and a best-practice example in Europe.Based on our findings, we recommend co-creating the "operations" of a living lab with the users themselves following a combined action research and living lab approach.
Seeking objective truth, the modern worldview makes no connection between knowledge and power.This positivist worldview has outlived its usefulness: as Habermas announced, 'modernism is dead'.
Peter Reason and Hilary Bradbury In Handbook of Action Research (2001) " " Action Research as a Framework to Evaluate the Operations of a Living Lab

Sara Logghe and Dimitri Schuurman
collected at the micro level (living lab user involvement activities), but the panel management and strategy towards user involvement should be taken into account at the macro level of the living lab -meaning its internal organizational processes for coping with different living lab projects.
Although this active user involvement is regarded as essential, few studies have focused on the motivation, attrition, and behaviour of users in Living Labs (Baccarne et al., 2013;Logghe et al., 2014;Ståhlbröst & Bergvall-Kåreborn, 2011).Moreover, we are not aware of any literature that reports end-user involvement in the design of user activities and operations in living labs, although this might seem to be a logic step for living lab practitioners who regard active user involvement as cornerstone of the living lab philosophy.Therefore, within this study, users were empowered to participate in the design of living lab activities and operations -at the macro level (organization), rather than at the micro level (user activities).We propose action research as a method to iteratively capture and implement this feedback because it emerges over time in an evolutionary process, as individuals develop skills of inquiry, and as communities of inquiry develop within communities of practice.This process not only leads to new practical knowledge, but also to new abilities to create knowledge (Reason & Bradbury, 2001).
Therefore, as a first part of this article, we examined the literature on action research for frameworks that could guide this process of user involvement.Within a participatory action research process, "communities of inquiry and action evolve and address questions and issues that are significant for those who participate as co-researchers" (Reason & Bradbury, 2008).Contrary to other research methods, action research does not emphasize disinterested researchers and reproducibility of findings.Among others, Ståhlbröst (2008) has already used action research as a methodology within a living lab environment to involve users early and throughout the whole development process, including the design of new information technology systems based on these users' needs.What is not dealt with in the literature is how this research approach can be used to construct a framework for user involvement and participation in the construction and optimization of living lab operations.
This aspect is addressed in the second part of this article; following our review of the literature on action research, we put the selected frameworks to the test in a single case study carried out at imec.livinglabs (imec-int.com/en/livinglabs;previously iMinds Living Labs).The first step of our implementation of action research was to gain knowledge about the current situation, meaning we sought to identify the basis for the organization's desire to change or alter its behaviour (Baskerville & Pries-Heje, 1999).After six years of living lab research at imec.livinglabs, many of the operation processes have changed.Following Ståhlbröst (2008), we started our action research approach by describing a main research theme within imec.livinglabs: to get to know the delights and frustrations of our panel members regarding their participation in living lab projects.
In order to answer this research question, a user researcher from imec.livinglabs became a panel member within specific living lab projects.Thus, our case study uses action research as a method to involve panel members in the organization processes of a living lab.We conclude the article by drawing main conclusions and recommendations on the use of action research in living labs.Lewin (1946) first described action research as "comparative research on the conditions and effects of various forms of social action and research leading to social action" that uses "a spiral of steps, each of which is composed of a circle of planning, action, and fact-finding about the result of the action".Action research has become an established research method that is often used in social sciences, but it is now used to not only to build knowledge on a certain topic but also to bring about changes to the topic.In order to stimulate these changes, several authors suggest that the researcher should become part of the user panel so that the obtained knowledge can be immediately applied (Baskerville, 1999).Furthermore, Checkland and Holwell (1998) distinguish three main phases of the process of action research: i) the researcher enters a real-world situation, ii) actions begin, and iii) the researcher leaves the situation and reflects on it in order to find a variety of lessons learned (Checkland & Holwell, 1998;Rönnerman, 2004).Because of its foundation in practical action and its aim to solve an immediate problem while informing theory, action research is seen to produce highly relevant results (Ståhlbröst, 2008).

Action Research in Living Labs
Action research starts with a practical problem owned by a certain group of people.The aim of this methodology is to find a solution for this problem, but also to develop theoretical knowledge for the wider research www.timreview.ca

Action Research as a Framework to Evaluate the Operations of a Living Lab
Sara Logghe and Dimitri Schuurman community (Chiasson et al., 2009).It requires understanding the context of the field and bringing changes to a current situation in order to solve certain problems through collaboration (Donnelly & O'Keeffe, 2013).Therefore, action research -and especially participatory action research -is also associated with the tradition of citizen science (Hand, 2010), wherein "the crowd" participates in scientific data collection and processing.Next to solving the specific practical problem, this research approach is mostly used to facilitate the understanding of complex human processes, rather than constructing universal social laws (Baskerville, 1999).It is particularly relevant when trying to "solve an identified class of problems" and producing guidelines for best practice (Sein et al., 2011).Reason and Bradbury (2001) argue that the characteristics of action research lead to a more valid research output because the practical and theoretical outcomes of the research process are grounded in the perspective and interests of those immediately concerned (in this study, the living lab panel members), and they are not filtered through an outside researcher's preconceptions and interestsa process that normally characterizes the positivist research approach.
The action research methodology and the living lab approach both appear to be user-centric research approaches, although the former has been used from the starting point (Lewis, 1946) (Ståhlbröst, 2008), but mainly as a method on a meso (project) or micro (user activities) level, to gather insights during a living lab project by means of the different user activities.In this study, we used the framework of Ståhlbröst (2008) as a starting point for our research on the satisfaction and motivation of our panel members regarding their participation in living lab projects in general, which is part of the general operations of the living lab organization, or the macro level.
Previous research has shown that intrinsic motivation is very important for users to remain part of a living lab community (Baccarne et al., 2013;Ståhlbröst & Bergvall-Kåreborn, 2011).Not only are users empowered by living labs (Veeckman et al., 2013), living labs are dependent on the involvement and motivation of their participating users.It is not easy for living lab researchers to motivate possible end users to take part in their research activities (Logghe et al., 2014) and to retain them.Action research "seeks to bring together action and reflection, theory and practice, in participation with others, in the pursuit of practical solutions to issues of pressing concern to people, and more generally the flourishing of individual persons and their communities" (Reason & Bradbury, 2001).Thus, we decided to use action research as a framework to co-create policies and guidelines for long-term user involvement in living labs, and to validate and implement them through a case study of a particular living lab.
However, despite the advantages described above, action research is not without criticism in the literature.
In our literature review, we identified three main areas of criticism of action research: 1.Because of the fact that a researcher becomes part of the study, a more personal view can become dominant during observations and deductions (Donnelly & O'Keeffe, 2013;Baskerville & Pries-Heje, 1999;Baskerville & Wood-Harper, 1996).
2. There is a lack of a common theoretical description of action research, which results in various approaches to action research (Chiasson et al., 2009;Donnelly and O'Keeffe, 2013).
3. The results of an action research setup are very specific, which may cause action research to simply result in more action research (Baskerville & Wood-Harper, 1996;Donnelly & O'Keeffe, 2013).
During our action research process, we took these valid criticisms into account and tried to establish reliability and validity for action research as a useful framework for researching living lab operations.Therefore, we tried to take into account each criticism as follows: 1.In order to avoid injecting our personal views, we asked questions in different ways.In this way, we were able to detect certain needs that were mentioned after asking various questions.
2. We compared different descriptions and implementations of action research in order to deduct one consistent theoretical framework for our use cases.
3. We selected multiple projects to implement action research in order to detect needs on a meso level rather than on a micro level.In this way, we tried to avoid to specific insights in order to gather more general results.

Case Study
Case description, methodology, and overall process The case focused on three living lab projects conducted at imec.livinglabs (imec-int.com/en/livinglabs;previously iMinds Living Labs), the living lab division of the research institute imec.Based in Ghent, Belgium, the former iMinds research institute became a separate business unit of imec following their merger in October 2016.The organization's experience conducting living labs dates back to 2009, with a particular emphasis on projects for startups and SMEs.imec.livinglabs has its own group of users (about 18,000 panel members or users) who are invited to participate in its living lab projects.For every living lab project, these panel members are invited to become part of a project community.In this way, they belong to innovation communities hosted by a neutral actor, in this case imec.livinglabs.These panel members cooperate as private participants during their spare time.But, according to Ståhlbröst and Bergvall-Kåreborn (2011), it is not sufficient to merely implement an innovation community in an organization to make a user innovation approach successful: it is also important to take into account what motivates the panel members to be part of this community.
In order to understand these motivations from an action research point of view, the main author of this article (an imec.livinglabsresearcher) became a panel member from March 2015 until April 2016 and took part in different research steps in three living lab projects: 1. De Kopploeg (8 months): a living lab project focusing on online privacy issues 2. We Run (2 months): a living lab project about an application for runners 3. Spott (7 months): a living lab project about a new way to buy products with your smartphone you see during a TV show or TV commercial These living lab projects were selected because they lasted for several months (so we were able to gather enough data) and consisted of multiple research steps (so we were able to ask the same questions in different ways).These three projects were also the first projects within imec.livinglabs whereby we focused on community building as a considerable aspect of a living lab.This approach fits with the aspects of action research of doing research in communities and emphasizing participation and action.Every community was inquired about in a collective way.The researcher became a member of the panel member community in order to experience the main motivations and thresholds for people to become and remain a panel member in a more intense way.The imec.livinglabs panel members of these three living lab projects were invited to give feedback on the general operation of imec.livinglabs by means of a survey (299 participants) and two co-creation session (12 and 8 participants, respectively.This allowed us to compare the experience of the researcher relative to the other imec.livinglabspanel members. The goal of the survey (n=299) was to evaluate the experience of the panel members with the De Kopploeg living lab project.This survey resulted in insights on how to manage expectations from the panel members regarding a living lab project, on how to keep a panel engaged throughout the research track (short term vs. long term), and on how to keep the most active members engaged or involved during a research project.This was also the first living lab project where imec.livinglabsused a Facebook group to stimulate a community feeling between the panel members who joined this project.In the Facebook group, imec.livinglabs was able to collect reactions on certain statements from the survey, or the panel members asked both practical and substantive questions in this group and answered each other's questions faster than imec.livinglabcoworkers were able to do so.imec.livinglabs was able to apply the main insights on the use of a Facebook group to the two other living lab projects under study.
In order to reflect on the involvement of researchers and panel members (Baskerville & Pries-Heje, 1999;Checkland & Holwell, 1998), we organized a survey and invited experienced panel members from each imec.livinglabspersona type (Logghe et al., 2015) to co-creation sessions to create solutions for the current frustrations of our panel members.Each co-creation session included the creation of a mock-up of the most interesting solutions.These mock-ups were validated by means of a specific validation survey and were handed over to the imec.livinglabsMarketing and Communication team who are currently constructing the community platform to be used over all living lab projects.
In December 2016, a final session was organized to ask our panel members for feedback regarding the platform in order to give our panel members the possibility to give an indication about what features or aspects Action Research as a Framework to Evaluate the Operations of a Living Lab

Sara Logghe and Dimitri Schuurman
should be modified before the final launch of the community platform.Based on these final remarks, a static version of the platform is being created and will be launched in March 2017.By inviting experienced panel members to these co-creation sessions, they became part of the reflection process on the operations of imec.livinglabs.In this way, learning occurs as an ongoing process in our research while reflecting on the method and projects as a whole (Ståhlbröst, 2008).

Findings
As a researcher who becomes a panel member, it is important to separate the application of knowledge to a project from the derivation of knowledge from each case (Ståhlbröst, 2008).As an example, we used the outcomes from "De Kopploeg", where we saw that a blog did not add value and that a Facebook group was consulted frequently, to plan the community strategy for the "We Run" and "Spott" projects.We also tested and implemented other measures such as a more personal communication approach (including names and photos of the panel managers in the communication), a higher "fun factor" in the incentives (personal challenges were used as "prize questions") and a faster, clearer feedback loop (creating infographics that summarize the user contributions from the research activity).We were able to capture this input during the project because the researcher took part in actions (research steps) in the situation (research projects) and acted upon it on an iterative basis for the subsequent projects.Also, through being a panel member and having conversations with other panel members, we found out that the imec.livinglabspanel has multiple unfulfilled needs and wants: • They want more detailed information about the initiator of our living lab projects.
• They want more detailed results than they receive now.
• They need an overview of all the calls for research participation.They want to be able to look back at a finished project and ask: What was it about?What research steps were organized?In which research steps did I participate?
During these reflective co-creation sessions, we found the following underlying needs and wants: 1. Panel members want to define their role in the innovation process more explicitly.
2. They expect that the innovation will be implemented according to their inputs and feedback.
3. They accept that they are co-creating and experimenting with innovations that are not yet finished.
Based on these needs and wants, the panel members were asked to think about ways to address them.First, they indicated that it would be useful to have the opportunity to consult an online platform with a project flow for each living lab research project indicating in which research step users are needed and what will happen with their input.Second, they asked for more concrete follow-up information.Nowadays, panel members receive an infographic via email with the main conclusions of a research activity, but they want to receive a more detailed report with extensive findings.They want to consult this report online and not via email.Also, they want to receive more information about the final product when the research project is finished, including (where possible) a link to the app store or a newsletter of the company who created the innovation.Eventually, these sessions resulted in mock-ups of an imec.livinglabscommunity platform for panel members, which would provide the required information.At the end of the co-creation sessions, the participating panel members told us they felt appreciated because they were involved in a feedback moment regarding the living lab operations.

Discussion and Conclusion
In our study, we found it useful to have a researcher becoming a panel member and to directly gather feedback from panel members using more traditional methods (i.e., a survey and co-creation sessions).By combining living lab activities with an action research methodology, we were able to quickly gathering issues and frustrations on the one hand and rapidly co-create and implement practical solutions on the other.The added value of a researcher being part of the panel was the fact that the researcher was able to elucidate the results and put the insights into perspective.In this way, the action research methodology shed light on the differences between what people say and what people do (van Merrienboer, 2015).
Moreover, by means of a case study, we illustrated that the end users themselves can also be part of this action research process, which dealt with the question of how to improve the operations of a living lab.We facilitated the reflection process of panel members by adding a re- In other words, adopting action research as a framework to evaluate and improve the operations of our living lab yielded positive results.
A major take-away was the fact that the action research approach and the living lab methodology strengthened and reinforced each other: the action research allowed us to uncover needs and wants with regards to user involvement in living lab projects and made it easier to experiment with new ways of involvement, which in turn enhanced the quality of the living lab research itself as the panel members felt empowered and involved in the living lab activities.
However, action research also has some caveats: as an action researcher, it is tempting to try to "act" like one of the regular panel members (Baskerville & Pries-Heje, 1999;Baskerville & Wood-Harper, 1996;Donnelly & O'Keeffe, 2013).The action researcher must stay more or less impartial in this situation and try to avoid pushing other panel members towards their own delights, frustrations, or solutions.They must also try to make other panel members feel at ease when giving feedback.Equally, we found that it was very useful to take the various criticisms into account during the research period, although it was not always easy to note down every insight in a structured way.Therefore, we propose to conduct further research leading to a more detailed framework about methods to use during the "practical part" of the action research process.Moreover, the long-term effects of involving panel members in improving the operations of a living lab on their motivation to participate should also be investigated.

Action Research as a Framework to Evaluate the Operations of a Living Lab
Sara Logghe is a Living Lab Researcher at imec -MICT -Ghent University in Belgium.She holds master's degrees in History and Communication Sciences from Ghent University, and her research interests include the potential of social media for cultural institutions, the changing library landscape, and living labs.Dimitri Schuurman is the Team Lead in User Research at imec.livinglabs and a Senior Researcher at imec -MICT -Ghent University in Belgium.He holds a PhD and a Master's degree in Communication Sciences from Ghent University.Together with his imec colleagues, Dimitri developed a specific living lab offering targeted at entrepreneurs in which he has managed over 100 innovation projects.Dimitri is responsible for the methodology and academic valorization of these living lab projects and coordinates a dynamic team of living lab researchers.His main interests and research topics are situated in the domains of open innovation, user innovation, and innovation management.His PhD thesis was entitled Bridging the Gap between Open and User Innovation?Exploring the Value of Living Labs as a Means to Structure User Contribution and Manage Distributed Innovation.