Developing Social Impact Evaluation Methods for Research: viewpoints on commercialization and sustainability

Academic institutions are nowadays considered as important drivers for national economies, since they are expected to spur innovations and thus stimulate economic growth (Weckowska, 2015; Rajalo & Vadi, 2017). These benefits may be delivered by means of industry utilizing the results of the research made in universities. For this reason, governments and national innovation policy makers are actively promoting the establishment and development of collaborative networks between universities and society, represented by industry, other private sector actors, the public sector, and non-profit organizations (Morlacchi & Martin, 2009; Perkmann et al., 2013; Rajalo & Vadi, 2017). In this way, national innovation policies nowadays often emphasize a so-called “third mission of universities”, which means that in addition to the fundamental goals of higher education and academic research, universities are also expected to make social contributions. These social contributions may include collaborative knowledge creation, transfer, and exchange between universities and external partners (Pennacchio, 2016). Policy makers are thus increasingly expecting publicly funded research not only to produce scientific and scholarly results, but also to enable clear social impacts.


Introduction
Academic institutions are nowadays considered as important drivers for national economies, since they are expected to spur innovations and thus stimulate economic growth (Weckowska, 2015;Rajalo & Vadi, 2017). These benefits may be delivered by means of industry utilizing the results of the research made in universities. For this reason, governments and national innovation policy makers are actively promoting the establishment and development of collaborative networks between universities and society, represented by industry, other private sector actors, the public sector, and non-profit organizations (Morlacchi & Martin, 2009;Perkmann et al., 2013;Rajalo & Vadi, 2017). In this way, national innovation policies nowadays often emphasize a so-called "third mission of universities", which means that in addition to the fundamental goals of higher education and academic research, universities are also expected to make social contributions. These social contributions may include collaborative knowledge creation, transfer, and exchange between universities and external partners (Pennacchio, 2016). Policy makers are thus increasingly expecting publicly funded research not only to produce scientific and scholarly results, but also to enable clear social impacts.
From the viewpoint of attempting to commercialize academic results, social impact now plays a central role. In this context, successful commercialization requires industrial firms to be able to absorb critical knowledge from universities, and together with university partners create new knowledge that may be critical for their future innovation and new product development (Kunttu & Neuvo, 2018, 2020. It has nevertheless been argued that the results of academic research seldom yield specific inventions or industrial products (D'Este & Patel 2007), while at the same time it has been difficult to empirically evaluate and show the direct impact of university collaboration on industrial innovation (Laursen & Salter, 2004).
In recent decades, methods and tools have been developed to evaluate the social impact of university/academic research. Since 1997, the National Science Foundation (NSF) in the United States of America has used specific evaluation criteria to assess the contribution of research to society by requiring grant applicants to discuss ways their research will have broader impacts on society. Since then, the so-called "broader impacts criteria" (BIC) has become a standard policy tool for the NSF to show policy makers and the public that government-funded research is useful from a social point of view. The BIC model requires researchers to show that their research makes a social contribution in terms of educational outreach or broad dissemination The social contributions of research activities have become more and more important in the rapidly changing innovation environment. Despite the fact that industrial commercialization of research results constitutes one of the most essential drivers for innovation and competitiveness, most generally used social impact evaluation criteria do not include clear metrics involving research commercialization possibilities. In a similar manner, principles regarding sustainable development have been largely omitted from the impact criteria. This paper considers the "broader impacts criteria" (BIC) model developed for social impact evaluation in the National Science Foundation in United States. We propose extensions to the BIC criteria related to commercialization and sustainable development viewpoints on impact evaluation. This paper also considers a newly introduced extension to BIC, called "inclusionimmediacy criteria" (IIC). Based on it, we propose an extended version of the model that aims to additionally evaluate the impact of research from commercialization point of view.
As highlighted by the OECD Innovation Strategy, better measurement of innovation and its impact on economic growth, sustainability and inclusiveness is key to fulfilling the promise of better co-ordinated innovation policies in the digital era.
Angel Gurría, OECD Oslo Manual, 2018 Developing Social Impact Evaluation Methods for Research: viewpoints on commercialization and sustainability Leena Kunttu, Helka Kalliomäki, Sorin Dan, Jari Kuusisto of results (also to non-academic audiences). The BIC model also highlights the collaboration with external stakeholders.
Woodson et al. (2021) have recently proposed an extension to BIC called "inclusion-immediacy criteria" (IIC). This model aims to determine how research impacts are distributed across various social groups. In the European Union's research funding instruments, the principles of "responsible research and innovation" (RRI) are used to evaluate the social contributions of research projects. RRI is used to evaluate the social impact of the research process by emphasizing viewpoints that involve ethical acceptability and ecological sustainability.
The existing general impact criteria such as BIC target evaluating the social impact of scientific research in general. Thus, these criteria do not consider impacts related to commercializing or developing innovations, which is often crucial to making the results of academic research useful for commercial products. In addition, the currently used version of BIC does not have links to sustainable development, which is considered one of the main themes in RRI. Likewise, the principles of sustainable development are strongly emphasized by the United Nations and its related organizations (Griggs, 2013).
In this paper, we discuss how the BIC model could be extended to better cover various aspects of commercialization and sustainable development. We also propose an extended version of the IIC model, which focuses on business and user viewpoints, both essential regarding commercialization. Both proposed extensions can be regarded as examples on how the currently used impact and evaluation criteria can be modified to meet the needs of commercializing academic research results.
The rest of the paper is organized as follows. Section 2 gives an overview of current assessments involving "social impact" and introduces the current BIC and IIC models. In section 3, we present our extended versions of these two criteria, and in section 4 discuss the contribution and conclusions of the proposed methods.

Literature Insights Assessing the Social Impact of Research
The priority of governments and innovation policy makers is to support research that not only increase scientific knowledge, but also contributes to society. For this reason, policy makers and other actors operating in the innovation policy domain have created policy tools that seek to evaluate the social impact of publicly funded research. In recent decades, connections between academic science and social impact have been emphasized in the national science policies, for example, in the USA and Europe. Behind this development are steady calls from politicians to make academic research more accountable, transparent, and applied, which has resulted in increasing demands from the public for demonstrable returns on investment in science (Woodson et al., 2021).

Broader impact criteria
In the USA, the National Science Foundation (NSF) introduced the broader impacts criteria (BIC) in 1997 to ensure that grant proposals would take the social impact of their research into account (Davis & Laas, 2014). BIC's European counterpart, Responsible Research and Innovation (RRI) was introduced in 2011, and has now been adopted into the European Commission's Horizon 2020 strategy. RRI has been defined as "a transparent, interactive process by which societal actors and innovators become mutually responsive to each other with a view on the (ethical) acceptability, sustainability and societal desirability of the innovation process and its marketable products (in order to allow a proper embedding of scientific and technological advances in our society)" (Burget et al., 2015). When comparing the BIC and RRI, it seems that BIC is concerned with more peripheral aspects of research that include, for example, inclusion and participation of disadvantaged groups, outreach activities and utilization of results. RRI, on the other hand, seeks to understand fundamental aspects about how research is conducted, including sustainability and equality considerations (Davis & Laas, 2014).
With pressure to increase the social contribution of science, both BIC and RRI have been relatively widely adopted as parts of impact evaluation systems. For example, both RRI and BIC offer guidelines on quite a general level, and there have been difficulties turning their general principles into specific guidelines (Davis & Laas, 2014 (2021) poverty or belonging to an underrepresented minority group, as well as a person's dialect, gender identification, or religion. Thus, research initiatives with inclusive impacts may include, for example, the participation of women in scientific fields where they are underrepresented, or developing new pedagogical methods for children with special needs. Table 3 summarizes the levels of inclusion.
In their newly proposed model: "Inclusion-Immediacy Criteria, IIC", for assessing the social impact of research, Woodson et al. (2021)

Research Method
In this paper, we extend the existing versions of BIC and IIC to highlight two distinctive themes: commercialization and sustainability. The reasons behind these proposed extensions are to underline the commercialization potential of research, and ensure that research takes into consideration demands related to sustainable development. Our research process started by making a review of the existing literature on impact evaluation with special emphasis on social impact.
We also benchmarked the most widely used evaluation approaches in the United States and Europe, namely BIC and RRI. These approaches were presented on quite a general level, and we identified several "blind spots", that is, relevant areas that were not considered in the evaluations. Especially in the BIC approach, we found topics related to sustainable development and commercialization to be blind spots, since BIC had nothing related to sustainable development, and commercialization came across as quite vague.
Finally, we examined how these two topics could best be integrated with existing evaluation approaches. This led us to decide that the most straightforward way to do this would be to propose two new sections to the BIC model. In the following sections, we describe the practical process in detail.  (2021) many other global actors. For this reason, we propose two additional criteria to BIC, as presented in Table 4.
Commercializing innovations based on scientific research has been considered as one of the most essential social impacts of academic work. This is because commercialization and business development constitute immediate and measurable market impacts based on the results of academic research (Markman et al., 2008;Perkmann et al., 2013). The commercialization process is directly linked to developing new products or services. This creates commercial relevance for the firm that develop their businesses based on innovation. This means that the industrial interests to utilize research results are directly related to commercialization (Markman et al., 2008). Because the current form of BIC does not include criteria related to business development or the commercialization of results, we propose two additional criteria for commercializing, as shown in Table 5.
Developing a commercialization viewpoint to IIC model As described earlier, the newly introduced model, "Inclusion-Immediacy Criteria" (IIC), was designed for Developing Social Impact Evaluation Methods for Research: viewpoints on commercialization and sustainability Leena Kunttu, Helka Kalliomäki, Sorin Dan, Jari Kuusisto Extending BIC and IIC models to cover commercialization and sustainability

Extending the BIC model with commercialization and sustainable development viewpoints
After an extensive preparation and inclusion process, the UN member states adopted 17 Sustainable Development Goals (SDGs) in 2015 (Griggs, 2013). These goals were based on an urgent call for action by all countriesdeveloped and developing -in a global partnership towards peace and prosperity for people and the planet. The goals recognize that ending poverty and other deprivations must go hand-in-hand with strategies that improve health and education, reduce inequalities, and spur economic growth -all while tackling climate change and working to preserve our oceans and forests.
The European RRI framework relies strongly on these general UN goals in its principles for evaluating the social impact research. The current form of BIC has no dimensions related to sustainable development, despite the fact that demand for impacts related to green thinking, sustainability, wicked global problems, resilience, and equality are emphasized by the UN and Table 4. Proposed new BIC criteria related to sustainability while the columns represent immediacy. The first row is used to express the universal impact of research. It considers how research is important to everyone and how anyone can benefit from research. The intrinsic level of universal impact describes what direct goals of the project benefit everyone. For example, research that is directly related to reducing carbon emissions belongs to this area, since everyone benefits from having a cleaner environment. The direct level is used to describe the universal benefits achieved while conducting research (even if they are not a part of the project goals). For example, if a research project focuses on developing low-emission engine technologies, then a better environment is not a direct goal of the research, even while the results of the research could contribute to improving the environment. The extrinsic level of universal impacts includes those that are separate from the actual research. Outreach activities like school visits or podcasts to wide audiences belong in this category.
assessing the social impact of research by combining criteria related to immediacy and inclusion (Woodson et al., 2021). The purpose of the IIC model is to complement the currently available BIC model by better determining how the impacts of research are distributed across social groups (see Table 2). In this paper we present an extended IIC criteria (EIIC) that focuses more on widespread social impacts involving three levels: universal, project stakeholder, and user. Based on this proposed model, social impacts related to practical usage and commercialization can be viewed and discussed within a wide scope, and evaluated in a systematic manner. In the EIIC model, the levels of immediacy remain the same as in the original IIC, whereas the modified inclusion levels of EIIC are summarized in Table 6.
The EIIC model can be presented as a 3x3 matrix, as shown in Table 7, where the rows represent inclusivity, Developing Social Impact Evaluation Methods for Research: viewpoints on commercialization and sustainability Leena Kunttu, Helka Kalliomäki, Sorin Dan, Jari Kuusisto Table 6. The levels of inclusion in the original IIC (Woodson, 2021) and in the proposed EIIC model direct outcomes from an end-user's point of view, and the direct impacts tell what kinds of benefits can occur while doing research. For example, research related to accessible and easy user interfaces causes intrinsic impacts to individual users. On the other hand, developing face recognition technologies aiming at new biometric solutions does not directly aim at consumer solutions, though the results may potentially end up in consumer photography applications. Extrinsic activities in the user category may be, for example, related to involving users or consumers in testing and evaluating newly developed innovations.

Discussion and Conclusion
As academic research makes contributions to society in many ways, it is important for funding bodies, policy makers and a general audience to understand the value of its social contribution. We considered this topic related to how the currently available impact evaluation approaches could better support commercializing the results of research. By developing dedicated evaluation approaches that emphasize commercialization, new possibilities arise to encourage the research community to pay more attention in considering applications for The second row describes how the project benefits stakeholders. The stakeholders include both direct and indirect project stakeholders, such as funders, research communities, companies, public sector actors, ecosystems, etc. The impacts also concern business . The intrinsic level of stakeholder impact describes how a project's goals benefit the project's stakeholders, who can be regarded as customers of the research. The direct level of stakeholder impact expresses how the project's stakeholders may benefit from the research outside of its initial goals. For example, if a project develops new algorithms for an IT company (intrinsic result), a direct result might be having a competent R&D workforce trained during the project period, which can potentially be recruited by the customer company. The extrinsic level of stakeholder impact includes impacts that are separate from the actual research. For example, presentations given by researchers at events organized by stakeholder organizations belong in this category.
The third row in the EIIC matrix describes the impact of the research outcomes to potential users. This category considers how the end-users of research results, or the individuals somehow affected by the research results, are involved. As on previous rows, intrinsic impacts are Developing Social Impact Evaluation Methods for Research: viewpoints on commercialization and sustainability Leena Kunttu, Helka Kalliomäki, Sorin Dan, Jari Kuusisto  In this paper, we also explored how sustainable development has a central role in the European RRI evaluation methodology. However, despite its growing importance, the principles of sustainability are absent in the American BIC evaluation concept. While the BIC criteria provide a good basis for assessing the social impact of funding proposals, they still lack important areas from the viewpoints of sustainable development and commercialization, both of which have become important themes for the social contribution of research. Because European RRI relies quite heavily on the principles of sustainable development, it would be beneficial also to BIC-related activities to somehow include these topics in impact evaluation considerations. However, we highlight that the extension proposals presented in this paper are just example ideas about how the current criteria can be extended based on the individual needs and focuses of the funding calls. It is up to the funding bodies and impact evaluators to decide how the extensions could serve the goal of the impact evaluation in the best possible manner, according to the case of each individual funding call.
As a second main topic in this paper, we presented an example of extending the newly introduced IIC model. As the IIC model has a practical two-dimensional structure for impact considerations in the inclusionimmediacy domain, it can also be utilized for considerations that focus on practical impacts related to commercialization. In our EIIC approach, we consider inclusivity on three levels: universal, stakeholder, and user. Especially the stakeholder and user levels are important from the practical commercialization point of view, since they measure what kinds of real benefits the research provides to direct project stakeholders, and what the benefits are for end-users of the innovations developed.
There is no single evaluation model that would fit all kinds of research. For this reason, we find it important to emphasize that the existing models can be modified, and that the models presented in this paper could work as examples of evaluation models for research with potential for commercialization in the short-term. The models for evaluating research impact tend to steer the planning of research projects, and thus effect the researchers' mindsets regarding their research activities. In this manner, new evaluation models emphasizing commercialization viewpoints may encourage researchers to consider practical viewpoints involving their research from business perspectives. This activity Developing Social Impact Evaluation Methods for Research: viewpoints on commercialization and sustainability Leena Kunttu, Helka Kalliomäki, Sorin Dan, Jari Kuusisto can foster an innovation mindset among university researchers, while facilitating interactions between academia and industry. research projects engaging stakeholders from public and private sector, nationally and internationally. She has also been involved in developing the internationally recognized Turku Urban Research Programme and its collaboration model that has been selected internationally as an best practice of university-city collaboration. Her research expertise is especially related to strategic urban development, innovation environments, and the knowledge base of urban development (e.g. university-city relations).
Dr. Sorin Dan is a postdoctoral researcher in public sector innovation and renewal at the Innovation and Entrepreneurship InnoLab research platform, University of Vaasa, and an expert in policy research and public management reform. Dr. Dan earned his PhD at the Public Governance Institute at KU Leuven (2015) and his master's in public policy and administration at the University of Massachusetts, Amherst (2009). Previously he was an in-house consultant in the Public Governance and Territorial Development Directorate of the OECD. His recent projects include "Digitalisation Academy: Creating and piloting a nationwide model to tackle talent shortage and improve digital competences (DA-PITO)", funded by the Ministry of Economic Affairs and Employment of Finland and "Support for Developing Better Country Knowledge on Public Administration and Institutional Capacity Building", funded by the EU.
Dr. Jari Kuusisto is a science and innovation policy expert with over 20 years of experience in providing in-depth strategic consulting to high-level government agencies, businesses, and international organisations in more than 15 countries, including the EU, OECD, and UN. His expertise focuses on science and innovation policy, university organisational development, strategic change management and research profile development and upgrading. His research has been published in highlevel scientific journals such as Research Policy, and he has authored several policy programmes with government agencies, ministries, the OECD, and European Commission. Kuusisto has demonstrable skills in orchestrating large-scale R&D projects, leveraging international resources and personnel across international boundaries, and a proven history of forging key partnerships with private and public sector partners throughout Europe to secure continued funding for research endeavours.
Developing Social Impact Evaluation Methods for Research: viewpoints on commercialization and sustainability Leena Kunttu, Helka Kalliomäki, Sorin Dan, Jari Kuusisto