July 2017 Download this article as a PDFAbstract

The organization of supported and sustainable urban interventions is challenging, with multiple actors involved, fragmented decision-making powers, and multiple values at stake. Globally, urban living labs have become a fashionable phenomenon to tackle this challenge, fostering the development and implementation of innovation, experimentation, and knowledge in urban, real-life settings while emphasizing the important role of participation and co-creation. However, although urban living labs could in this way help cities to speed up the sustainable transition, urban living lab experts agree that, in order to truly succeed in these ambitious tasks, the way urban living labs are being shaped and steered needs further research. Yet, they also confirm the existing variation and opaqueness in the definition of the concept. This article contributes to conceptual clarity by developing an operationalized definition of urban living labs, which has been used to assess 90 sustainable urban innovation projects in the city of Amsterdam. The assessment shows that the majority of the projects that are labelled as living labs do not include one or more of the defining elements of a living lab. In particular, the defining co-creation and development activities were found to be absent in many of the projects. This article makes it possible to categorize alleged living lab projects and distill the “true” living labs from the many improperly labelled or unlabelled living labs, allowing more specific analyses and, ultimately, better targeted methodological recommendations for urban living labs.

Introduction

By 1969, before the term “sustainable development” was commonly known, the United States Congress had already emphasized the need to “create and maintain conditions that permit fulfilling the social, economic and other requirements of present and future generations” (U.S. Congress, 1969).  It is apparent that the current systems of urban life and development do not succeed in achieving this sense of sustainability in cities, thereby demonstrating the need for urban innovation to contribute to the sustainability transition of cities (Joss, 2015; Wheeler & Beatley, 2015).

However, many sustainable urban innovations at present do not systemically find their way to the market, inhibiting their wider implementation and limiting their positive impact potential in the context of this urban sustainability transition (Ashuri & Durmus-Pedini, 2010; van Bueren & Broekhans, 2013). The living lab approach is believed to close this gap between production and actual market uptake by directly involving all end users and other urban stakeholders into the development of new products. This would ensure a match with the actual needs and aspirations of the users while taking into account the local and institutional contexts, cultures, and creativity potentials (ENoLL, 2006; Kresin, 2009; Lemke, 2009; Lesnikowski et al., 2016; van Bueren & De Jong, 2007). Furthermore, the new partnerships and the inclusion of new, multidisciplinary stakeholders in the development process is believed to provide new insights and offer new, more integrated solutions to the investigated problems, thereby advancing the generation of innovations (ENoLL, 2006; Vincent, 2016).

Given these alleged benefits, it is no surprise that, in recent years, the “urban living lab” has become a popular development approach on which hope is fixed to accelerate the generation and adoption of sustainable innovations in the urban system in the light of the urban sustainability transition. However, evaluative accounts of urban living labs in practice indicate that many of these experience difficulty in achieving the full benefits of this approach (Bulkeley & Castán Broto, 2013; Franz et al., 2015; Karré et al., 2015; Schuurman et al., 2016). These authors point to a lack of methodological deepening of the living lab approach as a reason for such difficulties. Yet, in order to provide targeted methodological recommendations, it is necessary to take a closer and more precise look at the aim of urban living labs. Analysis of the (urban) living lab literature and the variety of existing urban living labs, for example those presented at the ENoLL website, shows that there is no consensus on this issue. The way an urban living lab is defined and used is diffuse, with many different and often abstract definitions of the concept, leaving participants involved in urban living labs in the dark as to how the “living lab dimension” makes this project different from other urban innovation projects.

This article focuses on establishing a definition of urban living labs, which will show what can be expected from urban living labs and which will provide guidance to actors engaged in urban living labs or contemplating starting one.  The definition presented in this article has been based on the existing living lab literature. Additionally, a scan of 90 Amsterdam-based sustainable urban innovation projects has provided insights on the representation of “true” urban living labs, fostering co-creative innovation among the more traditional sustainable urban innovation initiatives. We conclude by identifying some key challenges to the implementation of urban living labs.

Defining “Urban Living Lab”

The term “urban living lab” is to refer to a variety of local experimental projects of a participatory nature. It is often used interchangeably with the terms “testing ground”, “hatchery”, “incubator”, “making space”, “testbed”, “hub”, “city laboratory”, “urban lab”, or “field lab”. Although there are many publications on living labs and urban living labs, even in these texts, this notion of the urban living lab has often not been defined clearly. In literature discussing the “theory” of (urban) living labs, they have been explained as a methodology (Eriksson et al., 2005), as an environment (Ballon et al., 2005; Schaffers et al., 2007), as a system (CoreLabs, 2007), and as a governance approach (Bulkeley et al., 2016).

To explore the variety of definitions and understandings of the concept, we have analyzed articles on (urban) living labs published in this journal, the Technology Innovation Management Review (TIM Review) and other journals. The articles in the TIM Review are particularly relevant because this journal has played a considerable role in the transformative debate on the essence, role, and shape of urban living labs, with, in addition to separate articles on this topic, special issues attributed to this topic in 2012, 2013, 2015, 2016, and 2017. Appendix A presents an overview of the definitions used in these issues.

Almost all the articles we found on (urban) living labs, including those published in other journals, referred to the variation and opaqueness in the definition of the concept. Often, these articles, which are presented in Appendix B, adopt existing definitions, such as the one used by the European Network of Living Labs (ENoLL): “Living labs are defined as user-centered, open innovation ecosystems based on systematic user co-creation approach, integrating research and innovation processes in real life communities and settings” (ENoLL, 2016). These definitions aim to grasp the essence of living labs in one sentence, mentioning the main aspects and the relations between these aspects, often using complex concepts. Besides these single-sentence definitions, there are authors who provide a rich description of the phenomenon, but do not define it. Other authors distinguish different “types” of living labs (e.g., Leminen, 2013) or imply that living labs are environments offering a “multitude of different projects” (Tuija Hirvikoski, President of ENoLL, in an open discussion on February 13, 2017). Finally, there are authors that simply refer to an empirical example (e.g., Femenías & Hagbert, 2013) or skip the usual explanatory paragraph altogether and discuss “living labs” as an already accepted notion (e.g., Coorevits & Jacobs, 2017). We see only a few examples where authors operationalize this definition by formulating key principles, defining elements, or building blocks of living labs (Schuurman et al., 2013; Ståhlbröst, 2012; Veeckman et al., 2013). Yet, gradually, in the current living lab literature, a stabilized conception of living labs with roughly similar characteristics has emerged.

Characteristics

As described in the subsections below, we have drawn on the living labs articles published in the TIM Review and in other journals and documents to capture the defining characteristics of urban living labs. In total, we identified nine characteristics in four dimensions: i) aim ii) activities, iii) participants, and iv) context. For each dimension, the defining characteristics of urban living labs are described in the subsections that follow, and the complete list of defining characteristics of urban living labs, presenting the full urban living lab definition, is summarized in Table 1.

Aim

The overall aim of living labs is to learn and experiment, by integrating processes of research and innovation (ASC, 2016; Bijsterveldt, 2016; ENoLL, 2016). The innovation aspect refers to the development of new products (i.e., an object, service, technology, application, process, or system) and to the discovery of new solutions to existing problems. Learning and experimenting (McCormick, 2016; Pallot & Pawar, 2012; Schaub, 2016; Vincent, 2016) refers to the production and exchange of knowledge among participants (ASC, 2016; Bijsterveldt, 2016; Friedrich et al., 2013; Heijden, 2016; McCormick, 2016; Pallot & Pawar, 2012). Thus, the aim is not only to learn from the experiences from the particular lab environment, but also to replicate the innovation elsewhere, in real life, or to further future innovation (ASC, 2016; Bijsterveldt, 2016; Franz et al., 2015; Juujärvi & Lund, 2016). It is this emphasis on formalized knowledge production – lessons that are formulated and that can be disseminated – that sets living labs apart from other policy experiments and niches of innovation (Evans & Karvonen, 2014).

“Urban” living labs distinguish themselves from living labs by unanimously displaying an explicit territorial focus on finding local sustainable solutions addressing wicked problems that tend to be global, such as climate change and energy transition. The use of cities or parts of cities as laboratories is well in line with the current emphasis on the city as the impactful governance level for economic development (e.g., Glaeser, 2011; Barber, 2013; Katz & Wagner, 2014) and for sustainable development (e.g., van Bueren et al., 2012; Bulkeley & Betsill, 2013). It also responds to calls for citizen empowerment (e.g., Saurugger, 2010; Fung, 2015).

Activities

There are a number of activities that are supposed to take place in living labs. Living labs should comprise the development of a product, which can be all sorts of artifacts, including process innovations, and not only, for example, testing or implementation (Budweg et al., 2011; Coenen et al., 2014; Feurstein et al., 2008; Gray et al., 2014; Leminen, 2015; Schaffers et al., 2007; Tang & Hämäläinen, 2012). A key element in this development process is co-creation (Bijsterveldt, 2016; ENoLL, 2016; Feurstein et al., 2008; Franz et al., 2015; Gómez-Barroso et al., 2009; Heijden, 2016; Schaub, 2016). The essence of a living lab is that the solution is sought together with the user, rather than just applying a fixed solution and involving the user only for testing. To qualify as co-creation, the targeted users need to be involved in the various development phases of the living lab process: not only should they be asked for their opinions, they should have decision-making power throughout the phases (Prahalad & Ramaswamy, 2004). Furthermore, the development process of living labs is iterative, which implies that, after being designed, the (prototype) product is used and evaluated by the stakeholders. The feedback and evaluation gathered from these steps are used to further develop and improve the product (Feurstein et al., 2008; Pallot & Pawar, 2012; Pierson & Lievens, 2005).

Participants

The literature further shows that the living lab is a systematic innovation approach in which all stakeholders – all actors who have a stake in the developed product and the process leading to it – participate directly in the development process. Stakeholder participation, including users, is an integral part of the development process in living labs (Feurstein et al., 2008).

The actors who need to participate in the living lab activities are, at a minimum: users (the end users of the final product that will be developed; in many cases, citizens), private actors (businesses, firms, companies), public actors (governments and public institutions), and knowledge institutes. The participation of knowledge institutes emphasizes the aim of formalized knowledge production (Krueger & Buckingham, 2009; Perry, 2006). As mentioned, these actors not only need to participate in these activities but also need to have power to influence the process (Prahalad & Ramaswamy, 2004). This power allows these actors, including the end users of the product, to be active partners in the innovation and development during the whole process, rather than just passive consumers and receivers and subjects of R&D activities (Almirall et al., 2012; Feurstein et al., 2008; Friedrich et al., 2013; Higgins & Klein, 2011; Leminen, 2015; Schaub, 2016).

Context

Authors on living labs unanimously agree that a defining characteristic of living labs is that they take place in real-life environments. The complexity and multi-contextuality that is connected to real-life environments is part of the challenge of living labs (Feurstein et al., 2008; Friedrich et al., 2013). Whether these real-life environments are limited to physical environments (Feurstein et al., 2008; Friedrich et al., 2013; Kieboom, 2013; Leminen, 2015; McCormick, 2016; Westerlund & Leminen, 2011) or also comprise virtual realities (Feurstein et al., 2008; Leminen, 2015) is contested. In this research, the criterion used is that the product central to the living lab is being developed and evaluated in its real-life use context. Given that this product can also be a virtual product, such as a digital data-gathering system, virtual realities can also form the context of living labs. More often, however, the living labs will take place in a physical place, such as a neighbourhood, city, or other area. The wide attention for urban living labs emphasizes the need or desire of involved stakeholders to capture the real-life context in all its complexity while assuming that such experiments, despite the highly uncontrolled conditions, nevertheless produce useful, transformative knowledge (Evans & Karvonen, 2013).

Table 1. Overview of the defining characteristics of living labs

Aspects

Characteristics

Aims

Aimed at innovation

Aimed at formal ​​learning for replication

For urban living labs: Aimed at increasing urban sustainability

Activities

Development (all phases of the product development process

Co-creation

Iteration (feedback, evaluation, and improvement)

Participants

Public actors, private actors, users and knowledge institutes participate in the living lab activities

All actors involved have decision-making power

Context

The living lab activities take place in the real-life use context of the innovation. In many urban living labs, this is a territory or a space-bound place.

 

An overview of the defining living lab characteristics mentioned in the living lab definitions presented in the literature is provided in Figure 1. This figure shows a high level of agreement among the authors of the texts studied on the characteristics of living labs, leading to the synthesised definition presented in Table 1. In addition, Appendix B lists how authors have worded these living lab characteristics, which shows the original variety of concepts to which authors referred to convey these characteristics and demonstrating the need for operationalization of these concepts to be of guidance to stakeholders involved in urban living labs.

Figure 1

Figure 1. Overview of the presence of the proposed defining living lab characteristics in the autonomous living lab definitions provided in literature, focusing only on definitions introducing self-worded determinations of an (urban) living lab (not or besides quoting previously proposed definitions)

 

Living labs vs. urban living labs

The characteristics of living labs and urban living labs are very similar, and we assume that the general characteristics of living labs and recommendations for their design and operation are generally applicable to urban living labs as well. The difference between living labs and urban living labs is the explicit focus on finding solutions meant to increase urban sustainability. The inclusion of this explicit aim of increasing urban sustainability in the problem and goal statement of a living lab is a concrete feature of urban living labs – a feature that is often vaguely referred to by urban living lab authors as the “urban” dimension. This view is also consistent with the intuitive conception of scholars and practitioners that there is a difference between “normal” living labs and “urban” living labs in terms of increased complexity. The intricate number of variables and relationships influencing the process and outcome of urban living labs compared to “normal” living labs is exactly the learning environment that urban stakeholders tend to look for and appreciate. It helps them to develop real-world solutions to real-world problems while emphasizing the need for these solutions to work.

The use of living labs in cities may also raise questions about the inclusiveness and democratic accountability of the selection of areas for living labs, and about the focus of the innovation. Indeed, many innovations tend to be technological and are driven by ICT and decentralized technologies with ambitions to become “smart” and “low carbon”. These are highly relevant questions, but they are beyond the focus of this article, which focuses on the living lab phenomenon itself.

Living Labs in Practice

Despite the popularity of urban living labs, stakeholders being engaged in living labs or wanting to start a living lab are struggling with what it actually is that they are supposed to do in an urban living lab. To answer this question, we have identified and quickly scanned the urban living labs in the Amsterdam metropolitan region. Given that urban living labs are not necessarily labelled as such, the identification of urban living labs in Amsterdam was not based on the projects calling themselves a living lab or an urban living lab, but on projects potentially being classified as an urban living lab based on their characteristics. This approach led to a sample of 90 sustainable urban innovation projects (for details, see Steen & van Bueren, 2017) with attention for the citizens or end users in their project descriptions. The projects were identified through snowball sampling using a mixture of sources: policy documents; research papers; the Internet; and suggestions from experts working at the municipality of Amsterdam, Amsterdam SmartCity (a public organization aimed to collect and disseminate knowledge on smart city initiatives in the Amsterdam region), AMS (the Amsterdam Institute for Advanced Metropolitan Solutions), and other Amsterdam-oriented researchers. Based on the project descriptions found in online and offline sources, one researcher verified which of the defining living lab characteristics (Table 1) were present in each of the 90 projects.

During this search, some notable observations were made. Of the nine characteristics of urban living labs, the characteristics of development taking place in the lab and of co-creation applying to the living lab activities were found to be the two urban living lab characteristics most frequently absent in sustainable urban innovation projects in Amsterdam. Of the 90 scanned projects, only 12 displayed these two characteristics. Of these 12 projects, all of them contained the remaining seven urban living lab characteristics (Table 1), thus qualifying as urban living labs. Surprisingly, most of these projects are not the ones calling themselves “living labs” or “labs”. In fact, only three of the 14 projects with “living lab” in their name and two of the 14 projects with “lab” in their name met the identified defining characteristics of an urban living lab. The other seven projects from the 90 urban innovation projects in Amsterdam that were identified as urban living labs did not use any reference to “lab” of “living lab” in their name.

In this article, we hone in on the challenges experienced with these two urban living lab characteristics of “developing an innovation” and “co-creation”, which have been identified as the two urban living lab characteristics most frequently absent in sustainable urban innovation projects.

Innovation challenges

The analysis of 90 sustainable urban innovation projects in Amsterdam shows that, besides development of an innovation, which is a criterion for urban living labs, a number of other innovation-related activities can be central in the projects.

From the literature on the innovation process – whether it concerns new product development (Cooper, 1988), agile development (Beck et al., 2001), or user-driven (open) innovation processes (Von Hippel, 2005; Mikkela, 2008) – we have identified five overarching phases in the innovation process and used them to categorize the projects:

  1. Research: “An investigative process of revising current knowledge employed to reach understanding of a subject for the purpose of making decisions” (Cambridge Dictionary).
     
  2. Development: “The process of the product being created, growing, and changing in order to become more advanced” (Business Dictionary).
     
  3. Testing: “Application of a product for the purpose of observing and assessing its functionality” (Merriam-Webster).
     
  4. Implementation: Implementation of a finished product in a real-life environment.
     
  5. Commercialization of the developed product: Demonstrating a previously developed and implemented product.

The first four phases largely coincide with the natural stages of product maturation in the product development process (Cooper, 1988). The fifth phase is emphasized in the literature on user-driven, open innovation, and new product development, where the convincing of users who have not been involved in the development process needs more attention. It should be noted that this phasing is presented as a linear process. In reality, the sequence of these phases is not fixed, and processes take place in iteration.

Ideally, an innovation project would cover all these phases. In practice, we observed that each of the projects focussed on a particular phase of the innovation process (according to the pattern visualised in Figure 2). The projects aimed specifically at researching, developing, testing, implementing, or commercializing a solution. According to the living lab characteristics, only the projects (also) conducting development activities qualify as urban living labs.

Figure 2

Figure 2. Classification of innovation process phase of 90 potential living lab projects in the Amsterdam region

 

User involvement challenges

Even though innovation in urban living labs theoretically should take place in co-creative processes in which participants can participate and influence decisions in all phases of the innovation process, this was rarely seen in the 90 investigated sustainable urban innovation projects in Amsterdam. When looking at the literature, we consider Sherry Arnstein’s “ladder of citizen participation” (1969) the most usable elementary scale for measuring the degree of user involvement in the projects studied, with the three main levels of involvement being decision-making power, tokenism, and non-participation. As in the literature on co-creation (Prahalad & Ramaswamy, 2004), Arnstein associates true participation with the possession of (decision-making) power. The second level of participation identified by Arnstein is tokenism: the action of symbolically allowing people to “participate” without actually providing them with power in the decision-making process. The third main level distinguished is non-participation.

When projecting Arnstein’s model onto the 90 potential living lab projects, we simply considered the first level as “participation” and the other two levels (tokenism and non-participation) as “no participation”, because decision-making power by stakeholders, including users, has been shown to be conditional for living labs. Following this criterion, user participation only occurred in 51 of the 90 urban innovation projects, and, although this may sound as if co-creation takes place, user participation in the project activities is in fact quite common, if not inherent, when the project activities concerned are testing, implementation, and demonstration. However, development with the user (i.e., with the user participating and having decision-making power in the development process of the developed product) is the only form of user involvement that actually enables co-creation, which is one of the characteristics of living labs. The scan of the 90 sustainable urban innovation projects in Amsterdam found development with the user in only 12 of the projects.

When looking more closely at the sustainable urban innovation projects in Amsterdam associated with a living lab approach not including user participation, these projects do display a significant focus on user-related activities (Figure 3). Two-thirds of these projects either clearly adopt a user perspective and specifically aim to deliver solutions for the end user rather than for public or industry partners, or they perform their project activities with the help of digital user data that the user actively or passively provides, for example through iBeacons, smartphone applications, or smart meters. The first form of this user focus – in which the user does not directly participate (i.e., there is no co-creation) but the user is included in other ways – can be referred to as “user oriented”, following the definition of “user-oriented design” by Veryzer and de Mozota (2005). The second form can be referred to as “user sourced”, which indicates that project activities are performed using virtual user data, which is actively or passively provided by the user.

Figure 3

Figure 3. Degree of user involvement in the 90 potential living lab projects in Amsterdam

 

Conclusions

Although urban living labs are widely implemented in urban contexts and are popular projects to allegedly catalyze sustainable urban innovation and the sustainable transition, their definition remains unclear, both in real life as well as in the literature. This article has set up a demarcating definition of the concept of urban living labs in order to allow researchers to further investigate this topic. After a literature review on the common denominators of living lab projects, a framework has been set up outlining the defining features of living labs in terms of goals, activities, participants, and context,  making it possible to categorize and diversify alleged living lab projects and distill the real living labs from the many improperly labelled or unlabelled living labs, allowing more specific analyses and recommendations.

Based on a scan of 90 sustainable urban innovation projects and alleged living lab projects in Amsterdam, the majority of the projects that are labelled as living labs in fact do not include one or more of the defining elements of living labs, and sometimes are not even so much different from traditional top-down product development processes. However, given that many of the benefits attributed to living labs are linked to precisely to these defining characteristics, with co-creation and innovation as the most distinguishing ones, the absence of some of these basic components of living labs will also lead to disappointing performance in the field of these alleged benefits.

In order to develop an urban living lab design that does achieve the sustainable outcomes that the texts and definitions on living labs outline, the methodology of urban living labs should be more closely studied. For example, in this context, it would be valuable to extract the details of the methods recommended in the living lab literature and how they compare to the methods being used in practice, what the drivers and barriers are for the emergence and implementation of sustainable urban innovations, and which other methodological lessons can be learned from previous experiences with urban living labs. Reformulating these findings in the form of recommendations regarding the methodology, design, and governance of urban living labs would enhance the ability of urban living labs to achieve successful development, adoption, and replication of sustainable urban innovations. These aspects were beyond the scope of the current study, but could be addressed in future research.

In closing, we emphasize that today’s popularity of living labs, also among subsidizing programmes preferring or demanding living labs as a research method, sometimes leads to a choice for a living lab approach without relating it to the project aims. With their specific co-creation requirement, living labs are certainly not applicable to every urban innovation project. Especially with regards to projects developing highly technological innovations, often pursued in the search for smart and sustainable cities, co-creative development, as is expected in living labs, can pose large challenges while not necessarily contributing to the quality of the solution.

 

Acknowledgements

This article is an outcome of a research project run under the auspices of the Amsterdam Institute for Advanced Metropolitan Solutions (AMS). We would like to thank the partners of the AMS Institute for their contribution to this research. 

 


References

Almirall, E., Lee, M., & Wareham, J. 2012. Mapping Living Labs in the Landscape of Innovation Methodologies. Technology Innovation Management Review, 2(9): 12–18.
http://timreview.ca/article/603

ASC. 2016. Urban Living Labs. Amsterdam Smart City (ASC): Amsterdam Economic Board, January 14, 2017. Accessed July 1, 2017:
https://www.amsterdameconomicboard.com/projecten/amsterdam-smart-city-2

Ashuri, B. & Durmus-Pedini, A. 2010. An Overview of the Benefits and Risk Factors of Going Green in Existing Buildings. International Journal of Facility Management, 1(1).

Ballon, P., Pierson, J., & Delaere, S. 2005. Test and Experimentation Platforms for Broadband Innovation: Examining European Practice. SSRN Electronic Journal.
http://dx.doi.org/10.2139/ssrn.1331557

Bergvall-Kåreborn, B., Ihlström Eriksson, C., & Ståhlbröst, A. 2015. Places and Spaces within Living Labs. Technology Innovation Management Review, 5(12): 37–47. 
http://timreview.ca/article/951

Bergvall-Kåreborn, B., & Ståhlbröst, A. 2009. Living Lab: An Open and Citizen-Centric Approach for Innovation. International Journal of Innovation and Regional Development, 1(4): 356–370.
https://dx.doi.org/10.1504/IJIRD.2009.022727

Brankaert, R., & den Ouden, E. 2017. The Design-Driven Living Lab: A New Approach to Exploring Solutions to Complex Societal Challenges. Technology Innovation Management Review, 7(1): 44–51. 
http://timreview.ca/article/1049

Broto, V. C., & Bulkeley, H. 2013. A Survey of Urban Climate Change Experiments in 100 Cities. Global Environmental Change, 23(1): 92–102.
https://dx.doi.org/10.1016/j.gloenvcha.2012.07.005

Budweg, S., Schaffers, H., Ruland, R., Kristensen, K., & Prinz, W. 2011. Enhancing Collaboration in Communities of Professionals Using a Living Lab Approach. Production Planning & Control, 22(5-6): 594–609.
http://dx.doi.org/10.1080/09537287.2010.536630

Buhr, K., Federley, M., & Karlsson, A. 2016. Urban Living Labs for Sustainability in Suburbs in Need of Modernization and Social Uplift. Technology Innovation Management Review, 6(1): 27–34. 
http://timreview.ca/article/958

Bulkeley, H., & Betsill, M. M. 2013. Revisiting the Urban Politics of Climate Change. Environmental Politics, 22(1): 136–154.
http://dx.doi.org/10.1080/09644016.2013.755797

Bulkeley, H., & Castán Broto, V. 2013. Government by Experiment? Global Cities and the Governing of Climate Change. Transactions of the Institute of British Geographers, 38(3): 361–375.
http://dx.doi.org/10.1080/09537287.2010.536630

Bulkeley, H., Coenen, L., Frantzeskaki, N., Hartmann, C., Kronsell, A., Mai, L., & Palgan, Y. V. 2016. Urban Living Labs: Governing Urban Sustainability Transitions. Current Opinion in Environmental Sustainability, 22: 13–17.
https://dx.doi.org/10.1016/j.cosust.2017.02.003

Coenen, T., van der Graaf, S., & Walravens, N. 2014. Firing Up the City-A Smart City Living Lab Methodology. Interdisciplinary Studies Journal, 3(4): 118.

Coenen, T., & Robijt, S. 2017. Heading for a FALL: A Framework for Agile Living Lab Projects. Technology Innovation Management Review, 7(1): 37–43. 
http://timreview.ca/article/1048

Cooper, R. G. 1988. The New Product Process: A Decision Guide for Management. Journal of Marketing Management, 3(3), 238–255.
http://dx.doi.org/10.1080/0267257X.1988.9964044

Coorevits, L., & Jacobs, A. 2017. Taking Real-Life Seriously: An Approach to Decomposing Context Beyond “Environment” in Living Labs. Technology Innovation Management Review, 7(1): 26–36. 
http://timreview.ca/article/1047

CoreLabs. 2007. Building Sustainable Competitiveness-Living Labs Roadmap 2007-2010. Luleå, Sweden: Luleå University of Technology, Centre for Distance-Spanning Technology.

ENoLL. 2006. What is a Living Lab? European Network of Living Labs (ENoLL), Accessed August 2, 2016:
http://www.openlivinglabs.eu/FAQ

Eriksson, M., Niitamo, V.-P., & Kulkki, S. 2005. State-of-the-Art in Utilizing Living Labs Approach to User-Centric ICT Innovation A European Approach. Luleå, Sweden: Luleå University of Technology, Centre for Distance-Spanning Technology.

Evans, J., & Karvonen, A. 2014. ‘Give Me a Laboratory and I Will Lower Your Carbon Footprint!’—Urban Laboratories and the Governance of Low‐Carbon Futures. International Journal of Urban and Regional Research, 38(2): 413–430.
https://dx.doi.org/10.1111/1468-2427.12077

Femenías, P., & Hagbert, P. 2013. The Habitation Lab: Using a Design Approach to Foster Innovation for Sustainable Living. Technology Innovation Management Review, 3(11): 15–21. 
http://timreview.ca/article/741

Feurstein, K., Hesmer, A., Hribernik, K. A., Thoben, K., & Schumacher, J. 2008. Living Labs: A New Development Strategy. In J. Schumacher &V.P. Niitamo (Eds.), European Living Labs – A New Approach for Human Centric Regional Innovation: 1–14. Berlin: Wissenschaftlicher Verlag.

Franz, Y., Tausz, K., & Thiel, S. K. 2015. Contextuality and Co-Creation Matter: A Qualitative Case Study Comparison of Living Lab Concepts in Urban Research. Technology Innovation Management Review, 5(12): 48–55.
http://timreview.ca/article/952

Friedrich, P., Karlsson, A., & Federley, M. 2013. SubUrbanLab Report D2.1: Boundary Conditions for Successful Urban Living Labs. SubUrbanLab. 
http://suburbanlab.eu/tulokset/?lang=en

Georges, A., Schuurman, D., & Vervoort, K. 2016. Factors Affecting the Attrition of Test Users During Living Lab Field Trials. Technology Innovation Management Review, 6(1): 35–44. 
http://timreview.ca/article/959

Gómez-Barroso, J.-L., Feijóo, C., Pascu, C., & van Lieshout, M. 2009. User-Led, Citizen Innovation at the Interface of Services. info, 11(6): 82–96.
https://dx.doi.org/10.1108/14636690910996731

Gray, M., Mangyoku, M., Serra, A., Sánchez, L., & Aragall, F. 2014. Integrating Design for All in Living Labs. Technology Innovation Management Review, 4(5): 50–59.
http://timreview.ca/article/793

Hakkarainen, L., & Hyysalo, S. 2013. How Do We Keep the Living Laboratory Alive? Learning and Conflicts in Living Lab Collaboration. Technology Innovation Management Review, 3(12): 16–22. 
http://timreview.ca/article/749

Hakkarainen, L., & Hyysalo, S. 2016. The Evolution of Intermediary Activities: Broadening the Concept of Facilitation in Living Labs. Technology Innovation Management Review, 6(1): 45–58. 
http://timreview.ca/article/960

Higgins A., Klein S. 2011. Introduction to the Living Lab Approach. In Y. H. Tan, N. Björn-Andersen, S. Klein, & B. Rukanova (Eds), Accelerating Global Supply Chains with IT-Innovation: 31–36. Berlin: Springer, Heidelberg.
https://dx.doi.org/10.1007/978-3-642-15669-4_2

Joss, S. 2015. Sustainable Cities: Governing for Urban Innovation. Basingstoke, UK: Palgrave Macmillan.

Juujärvi, S., & Lund, V. 2016. Enhancing Early Innovation in an Urban Living Lab: Lessons from Espoo, Finland. Technology Innovation Management Review, 6(1): 17–26.
http://timreview.ca/article/957

Juujärvi, S., & Pesso, K. 2013. Actor Roles in an Urban Living Lab: What Can We Learn from Suurpelto, Finland? Technology Innovation Management Review, 3(11): 22–27. 
http://timreview.ca/article/742

Karré, P. M., Vanhommerig, I., & van Bueren, E. 2015. De stad als lab voor sociale verandering. Bestuurskunde, 24(1): 1–11.
https://dx.doi.org/10.5553/Bk/092733872015024001001

Katzy, B. 2012. Designing Viable Business Models for Living Labs. Technology Innovation Management Review, 2(9): 19–24. 
http://timreview.ca/article/604

Kieboom, M. 2013. Living Labs: Innoveren in een proeftuin vraagt om nieuwe manieren van tuinieren. Kennisland, August 20, 2013. Accessed August 2, 2016:
https://www.kl.nl/opinie/living-labs-innoveren-in-een-proeftuin-vraagt-o...

Kresin, F. 2009. Living Labs: Hoop of Hype? Frankwatching, March 31, 2009. Accessed August 1, 2016.
https://www.frankwatching.com/archive/2009/03/31/living-labs-hoop-of-hype/

Krueger, R., & Buckingham, S. 2009. Creative-City Scripts, Economic Development, and Sustainability. Geographical Review, 99(1): iii–xii.
https://dx.doi.org/10.1111/j.1931-0846.2009.tb00414.x

Kusiak, A. 2007. Innovation: The Living Laboratory Perspective. Computer-Aided Design and Applications, 4(6): 863–876.
http://dx.doi.org/10.1080/16864360.2007.10738518

Leminen, S., Westerlund, M., & Nyström, A.-G. 2012. Living Labs as Open-Innovation Networks. Technology Innovation Management Review, 2(9): 6–11. 
http://timreview.ca/article/602

Leminen, S. 2013. Coordination and Participation in Living Lab Networks. Technology Innovation Management Review, 3(11): 5–14. 
http://timreview.ca/article/740

Leminen, S. 2015. Living Labs as Open Innovation Networks-Networks, Roles and Innovation Outcomes. Doctoral dissertation, Aalto University, Finland.
https://aaltodoc.aalto.fi/handle/123456789/17899

Leminen, S. K., Nyström, A.-G., Westerlund, M., Kortelainen, M. J., & Zolkiewski, J. 2016. The Effect of Network Structure on Radical Innovation in Living Labs. Journal of Business & Industrial Marketing, 31(6): 737–757.
https://dx.doi.org/10.1108/JBIM-10-2012-0179

Leminen, S., Turunen, T., & Westerlund, M. 2015. The Grey Areas Between Open and Closed in Innovation Networks. Technology Innovation Management Review, 5(12): 6–18. 
http://timreview.ca/article/948

Leminen, S., & Westerlund, M. 2017. Categorization of Innovation Tools in Living Labs. Technology Innovation Management Review, 7(1): 15–25. 
http://timreview.ca/article/1046

Lemke, M. 2009. Living Labs for User-Driven Open Innovation: An Overview of the Living Labs Methodology, Activities, Achievements. Brussels: European Commission: New Infrastructure Paradigms and Experimental Facilities.
https://dx.doi.org/10.2759/34481

Lesnikowski, A., Ford, J., Biesbroek, R., Berrang-Ford, L., & Heymann, S. J. 2016. National-Level Progress on Adaptation. Nature Climate Change, 6(3): 261–264.
https://dx.doi.org/10.1038/nclimate2863

McCormick, K. 2016. GUST Policy Brief: Urban Living Labs. Governance of Urban Sustainability Transitions (GUST).
http://www.urbanlivinglabs.net/p/publications.html

Mikkelä, K. 2008. Finnish Living Lab and User Driven Open Innovation Situation. Paper presented at the Workshop on Living Labs, Halmstad University, Sweden, May 29–30, 2008.

Mulder, I. 2012. Living Labbing the Rotterdam Way: Co-Creation as an Enabler for Urban Innovation. Technology Innovation Management Review, 2(9): 39–43. 
http://timreview.ca/article/607

Nevens, F., Frantzeskaki, N., Gorissen, L., & Loorbach, D. 2013. Urban Transition Labs: Co-Creating Transformative Action for Sustainable Cities. Journal of Cleaner Production, 50(1): 111–122.
https://dx.doi.org/10.1016/j.jclepro.2012.12.001

Niitamo, V. P., Kulkki, S., Eriksson, M., & Hribernik, K. A. 2006. State-of-the-Art and Good Practice in the Field of Living Labs. In Proceedings of the IEEE International Technology Management Conference (ICE) 2006: 1–8.
https://dx.doi.org/10.1109/ICE.2006.7477081

Niitamo, V.-P., Westerlund, M., & Leminen, S. 2012. A Small-Firm Perspective on the Benefits of Living Labs. Technology Innovation Management Review, 2(9): 44–49. 
http://timreview.ca/article/608

Pallot, M., & Pawar, K. 2012. A Holistic Model of User Experience for Living Lab Experiential Design. Paper presented to the 18th International ICE Conference on Engineering, Technology and Innovation, Munich, Germany.
https://dx.doi.org/10.1109/ICE.2012.6297648

Perry, B. 2006. Science, society and the University: A Paradox of Values. Social Epistemology, 20(3-4): 201–219.
https://dx.doi.org/10.1080/02691720600879798

Pierson, J., & Lievens, B. 2005. Configuring Living Labs for a 'Thick' Understanding of Innovation. In Proceedings  of the Ethnographic Praxis in Industry Conference, 2005: 114–127. 
https://dx.doi.org/10.1111/j.1559-8918.2005.tb00012.x

Prahalad, C. K., & Ramaswamy, V. 2004. Co‐Creation Experiences: The Next Practice in Value Creation. Journal of Interactive Marketing, 18(3): 5–14.
https://dx.doi.org/10.1002/dir.20015

Rits, O., Schuurman, D., & Ballon, P. 2015. Exploring the Benefits of Integrating Business Model Research within Living Lab Projects. Technology Innovation Management Review, 5(12): 19–27. 
http://timreview.ca/article/949

Schaffers, H., Cordoba, M. G., Hongisto, P., Kallai, T., Merz, C., & Van Rensburg, J. 2007. Exploring Business Models for Open Innovation in Rural Living Labs. In Proceedings of the IEEE International Technology Management Conference (ICE), 2007: 1–8.

Schaffers, H., & Turkama, P. 2012. Living Labs for Cross-Border Systemic Innovation. Technology Innovation Management Review, 2(9): 25–30. 
http://timreview.ca/article/605

Schaub, A. 2016. Amsterdam Living Lab. Waag Society, Accessed August 2, 2017:
https://www.waag.org/nl/project/amsterdam-living-lab

Schuurman, D., & De Marez, L. 2012. Structuring User Involvement in Panel-Based Living Labs. Technology Innovation Management Review, 2(9): 31–38. 
http://timreview.ca/article/606

Schuurman, D., De Marez, L., & Ballon, P. 2013. Open Innovation Processes in Living Lab Innovation Systems: Insights from the LeYLab. Technology Innovation Management Review, 3(11): 28–36.
http://timreview.ca/article/743

Schuurman, D., De Marez, L., & Ballon, P. 2016. The Impact of Living Lab Methodology on Open Innovation Contributions and Outcomes. Technology Innovation Management Review, 1(6): 7–16.
http://timreview.ca/article/956

Schuurman, D., & Tõnurist, P. 2017. Innovation in the Public Sector: Exploring the Characteristics and Potential of Living Labs and Innovation Labs. Technology Innovation Management Review, 7(1): 7–14. 
http://timreview.ca/article/1045

Schuurman, D., De Marez, L., & Ballon, P. 2016. The Impact of Living Lab Methodology on Open Innovation Contributions and Outcomes. Technology Innovation Management Review, 1(6): 7–16. 
http://timreview.ca/article/956

Ståhlbröst, A. 2012. A Set of Key Principles to Assess the Impact of Living Labs. International Journal of Product Development, 17(1-2): 60–75.
https://dx.doi.org/10.1504/IJPD.2012.051154

Ståhlbröst, A. 2013. A Living Lab as a Service: Creating Value for Micro-enterprises through Collaboration and Innovation. Technology Innovation Management Review, 3(11): 37–42. 
http://timreview.ca/article/744

Ståhlbröst, A., & Lassinantti, J. 2015. Leveraging Living Lab Innovation Processes through Crowdsourcing. Technology Innovation Management Review, 5(12): 28–36. 
http://timreview.ca/article/950

Steen, K., & van Bueren, E. 2017. Urban Living Labs: A Living Lab Way of Working. AMS Research Report 2016–2017. Amsterdam: AMS Institute.
http://www.ams-institute.org/news/out-now-urban-living-labs-a-living-lab...

Tang, T., & Hämäläinen, M. 2012. Living Lab Methods and Tools for Fostering Everyday Life Innovation. Paper presented at the 18th International ICE Conference on Engineering, Technology and Innovation (ICE), Munich, Germany.
https://dx.doi.org/10.1109/ICE.2012.6297644

Tukiainen, T., Leminen, S., & Westerlund, M. 2015. Cities as Collaborative Innovation Platforms. Technology Innovation Management Review, 5(10): 16–23. 
http://timreview.ca/article/933

U.S. Congress. 1969. National Environmental Policy Act of 1969. Public Law, 91(190), 1–5.

van Bijsterveldt, M. 2016. Living Labs. Ruimtelijke Adaptatie. Accessed August 2, 2016.
https://ruimtelijkeadaptatie.nl/living-labs/

van Bueren, E., & Broekhans, B. 2013. Individual Projects as Portals for Mainstreaming Niche Innovations. In R. L. Henn & A. J. Hofman (Eds.), Constructing Green: The Social Structures of Sustainability: 145–167. Cambridge, MA: MIT Press.
https://dx.doi.org/ 10.7551/mitpress/9780262019415.003.0007

van Bueren, E., & De Jong, J. 2007. Establishing Sustainability: Policy Successes and Failures. Building Research & Information, 35(5): 543–556.
http://dx.doi.org/10.1080/09613210701203874

van der Heijden, E. 2016. Urban Management Fieldlabs: Living Lab Description.  European Network of Living Labs. Accessed August 2, 2016:
http://www.openlivinglabs.eu/livinglab/urban-management-fieldlabs

Veeckman, C., Schuurman, D., Leminen, S., & Westerlund, M. 2013. Linking Living Lab Characteristics and Their Outcomes: Towards a Conceptual Framework. Technology Innovation Management Review, 3(12): 6–15.
http://timreview.ca/article/748

Veeckman, C., & van der Graaf, S. 2015. The City as Living Laboratory: Empowering Citizens with the Citadel Toolkit. Technology Innovation Management Review, 5(3): 6–17. 
http://timreview.ca/article/877

Veryzer, R. W., & de Mozota, B. B. 2005. The Impact of User-Oriented Design on New Product Development: An Examination of Fundamental Relationship. The Journal of Product Innovation Management, 22(2): 128–143.
http://dx.doi.org/10.1111/j.0737-6782.2005.00110.x

Vincent, S. 2016. Lab Builders: Some Thinking about the Next Generation of Lab Programmes. Design for Europe, June 21, 2016. Accessed August 2, 2016:
http://www.designforeurope.eu/news-opinion/lab-builders-some-thinking-ab...

Voytenko, Y., McCormick, K., Evans, J., & Schliwa, G. 2016. Urban Living Labs for Sustainability and Low Carbon Cities in Europe: Towards a Research Agenda. Journal of Cleaner Production, 123: 45–54.
https://doi.org/10.1016/j.jclepro.2015.08.053

Wallin, S. 2014. APRILab: Guidelines to Define and Establish an Urban Living Lab. Amsterdam: University of Amsterdam, Amsterdam Institute of Social Science Research.

Westerlund, M., & Leminen, S. 2011. Managing the Challenges of Becoming an Open Innovation Company: Experiences from Living Labs. Technology Innovation Management Review, 1(1): 19–25.
http://timreview.ca/article/489

Wheeler, S. M., & Beatley, T. (Eds.). 2014. Sustainable Urban Development Reader. London: Routledge.

 


 
Appendices

 

Share this article:

Cite this article:

Rate This Content: 
No votes have been cast yet. Have your say!

Keywords: characteristics, cities, definition, living labs, TIM Review, urban living labs

Add new comment

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Badbot Fields
If you see these fields, something is wrong.
If you see this field, something is wrong.
If you see this field, something is wrong.
If you see this field, something is wrong.