1. Introduction

Research and development (R&D) is a key source of competitive advantage for high-technology organizations (Artz et al., 2010). However, investments in R&D and innovation development can be risky and costly. Research has shown that only one of four R&D projects is successful (Evanschitzky et al., 2012). One essential reason for this is the lack of flexibility in traditional R&D approaches, which are typically based on waterfall development processes. In these sequential processes, the contribution of each functional department (for example, mechanical, electrical, packaging, manufacturing, quality control) is done one after the other. These methods seem to work for the development of products that rely on systems and subsystems  having years of prior experience in testing, development, operation and known solutions that use state-of-the-art (SOA) or near SOA technology. Yet these methodologies fail miserably for situations where there is no known solution and/or the problem is complex (Mitchell, 2009; Camarda et. al., 2019).

Set-based concurrent engineering (SBCE) or set-based design (SBD) is a product development strategy (Singer et al., 2009). It transcends traditional point-based solution methods by looking at broad “sets” of functional solutions that are constrained by broad “ranges” of design requirements and specifications in parallel, as the design space is explored and understood. For example, the root cause of excessively dropped keystrokes in a manufacturer’s keyboard that passed all automated quality validation tests but failed with human test subjects was found to be the humans’ perception of “feel” that the keys had been deflected sufficiently (Cloft et al., 2018).  Once the root cause was identified, very simple force-deflection structural models of the keyboard could be used to rapidly evaluate numerous material and structural design alternatives to select a viable, optimum solution. However, complex, transdisciplinary engineering problems, like the cause of premature failure of a Space Shuttle’s thermal protection system (TPS) tiles prior to launch, require a tiger team of subject matter experts and lengthy study, in order to identify the root cause, which in this case was a highly coupled transdisciplinary problem (Cooper & Holloway, 1981; Camarda, 2014a; Camarda et. al., 2019). After the root cause was understood and a satisfactory solution  of densifying a small bottom layer of the fragile fibrous TPS tile prior to bonding developed, it still required over one year to solve the problem. This delayed the launch of the first Space Shuttle. The R&D knowledge construction methods were adequate to solve this problem, however, are there ways this process can be significantly accelerated?

This paper will discuss in detail the knowledge construction process used by NASA research teams to understand and rapidly solve the on-orbit repair of a damaged Space Shuttle wing leading edge (Camarda, 2007; Camarda, 2014b; Camarda et. al., 2019). It will compare and contrast this new method of R&D knowledge capture with other Space Shuttle case studies, which used more traditional programmatic methods. The rapid R&D approach used is a blend of set-based design and intelligent fast failure (failing and learning smart, fast, small, cheap, early, and often) (Matson, 1996). The key to this methodology’s success is the incorporation of a network of teams, or Team-of-Teams approach, using open and effective communication, and a flat organizational structure (McChrystal, 2015).

2. Rapid R&D

2.1 James Starnes’ View of Research, Design, and Knowledge Construction

NASA Langley Research Center (LaRC) in Virginia, USA, is a research organization that had a culture where failure was not only tolerated; it was accepted and expected.  NASA LaRC historian, James Schultz, described Langley’s greatest gift as its “permission to try and try again” (“permission to fail”). He noted that, “Learning by repeated attempts may appear cumbersome, but failures indicated areas where further research was needed to improve the understanding of flight phenomena. At Langley, the mistakes were just as important as the successes, for they sowed the seeds of future accomplishment” (Schultz, 2003).

Dr. James Starnes was a world-renowned structural mechanics expert at LaRC who espoused a view of research and design (Camarda, 2009) illustrated in figure 1, on the left diagram. We begin with a physical observation, then attempt to model that observation as best we know how analytically/mathematically. We evaluate our representation of that observation by a test/experiment. More often than not, we either “fail,” or our model of the problem’s physics is found to be lacking.  It could be our experimental representation of the “real” observation (initial conditions, boundary conditions, physical properties, etc.) or the errors could lie in our simplified, mathematical model (simplifying assumptions, numerical model, etc.). We iterate in these two worlds of experiment and analysis (double-ended arrows), until we understand the discrepancies and can correlate our analytical representation of behavior with what we observe in the laboratory to within some level of accuracy. 

Dr. Starnes always stressed the importance of testing to failure, and it is a critical part of his philosophy of how research in structures should be done.  The true test of our understanding of structural concepts occurs when we are able to anticipate every significant failure mechanism, and can accurately predict when failure will occur.  Once we believe we truly understand the physics of a problem, we proceed to step into the world of design.

We can now use our analytical or numerical representation of the problem to rapidly and systematically vary design variables and wander through design space, avoiding constraints (that is, boundaries of regions in the design space where failure will occur), until we arrive at a design that satisfies all constraints (avoids failure) and produces an “optimum” solution to our function. We then go back to the laboratory to determine if we can reliably predict the behavior and failure of our “optimum” design. Once again, we may fail and, in the process, discover or learn that we have exceeded our understanding of the problem by moving beyond the bounds of applicability for our prior assumptions about the design space, or we have encountered an unanticipated “failure mechanism” that needs to be included in our analytical/numerical representation.  This process is repeated many times. Each time we fail, we also learn. Thus, we develop a much better understanding of the problem and, more importantly, an understanding of our limitations in predicting the behavior of an actual, imperfect artifact, given the use of an idealized, and also imperfect, model. 

In Dr. Starnes’ research philosophy, one cycles through a structural concept at ever-increasing scales, starting with the simplest of material property tests of coupons, and proceeding, in a building-block fashion, to sub-element, component, full-scale section, and finally to a full-scale test article (right-hand illustration in Figure 1). Research engineers use a building-block approach at a very elemental level to understand basic principles which can be observed with very simple experiments. The observations made during these simple experiments are then compared with very simple initial models of the problem, which are modified until they agree, and the analyses and tests can be said to be correlated. This validates that the simple analytical model is sufficient to accurately describe the behaviors witnessed in the experiment. More complex tests of these simple structural elements combined into a larger-scale structure are designed to explore the potentially nonlinear effects of interactions between simpler elements when integrated together. It leads to an improved physical understanding that is more complete, such that eventually a full-scale representation of the real, full-scale problem can be accurately modeled. These tests may include interfaces between multiple components, attachments, and manufacturing details. The closer the test article approaches the real embodiment and operational use of the concept, the more rigorous the analysis must be in order to not only predict behavior/performance, but also to predict failure.

Figure 1.  Knowledge construction methods used in Rapid R&D (Camada et. al., 2019).

Because the problems we are attempting to solve are complex and can have undetermined results as we integrate at larger and larger scales, the resulting outcomes cannot be predicted from only running small-scale tests.  It is therefore terribly important that we follow a stepwise building-block process, even within a single discipline such as structures. When a new and multidisciplinary concept or application is the focus of development, the team of researchers must be adept at working collaboratively, in a converged, transdisciplinary way, in order to understand all the potential interactions of key disciplines that can be highly coupled and lead to premature failure. 

An example of such a transdisciplinary team was the Thermal Structures Branch (TSB) at NASA LaRC. It was much more than a conventional integrated product development team (IPDT). Rather, it was a collection of subject-matter-experts with cross-disciplinary skills in multiple key domains related to hypersonic vehicle structures and related systems, and subsystems such as TPS, cryogenic tankage, hot, passive, and cooled structures. We call this type of research team or branch, an integrated systems research branch (see Figure 2). The collection of team members with cross-disciplinary skills (shown by the dashed oval) are connected, in a “team-of-teams” (McChrystal, 2015), with an integrated network of key subject-matter-experts (SMEs) in each individual discipline to rapidly assess potential failure mechanisms and anomalies. The success of this network of teams and the open and rapid collective learning and dissemination of knowledge is a function of many factors. Due to the length limitations of this paper, only a few will be highlighted.

Figure 2.  Definition of an integrated systems research branch and a team-of-teams network to solve complex problems.

The seemingly subtle differences in what we are describing as a converged, transdisciplinary team, or integrated systems research branch, will become evident when we compare the methodologies used by such a team with more traditional “groups” assigned to mature complex systems and/or to identify root causes of complex problems.

2.2 NASA Case Study:  Space Shuttle Thermal Protection System (TPS) Tile Development Anomaly

The Space Shuttle was an amazing and unique spacecraft that transported crewmembers, supplies, equipment, experiments, and large payloads to and from low Earth orbit (LEO) (Camarda, 2014a).  The history of the Space Shuttle Program (SSP) clearly demonstrates the drawbacks of a serial, point-based design approach to designing a complex technical product. Among the technical, scientific, political, and programmatic challenges during its development were ambiguous and everchanging design requirements, multiple stakeholders (NASA, military, industry, civilian, and scientific communities) to satisfy to secure sufficient funding and support, severe budget constraints, and schedule pressures.

It was hoped that this very complex, highly sophisticated space vehicle could usher in a new era of safe, low-cost access to space, which would enable effective commercial and private usage in space for everyone.

The engineers, scientists, and program managers that helped develop the thousands of components, sub-systems, and systems for the Space Shuttle, used a reductionist approach to functionally decompose the problem. They then used conventional systems engineering principles to relate the elements and to predict the integrated behavior as if it was a “complicated”, deterministic problem as opposed to a “complex” problem (Mitchell, 2009). 

Prior to the launch of Space Transportation System-1 (STS-1) on April 12, 1981, after flight profiles and air loads were refined, it became apparent that while the TPS material satisfied loading requirements, the TPS as a “system” had inadequate tensile strength as illustrated at the bottom of Figure 3. This meant that many of the Shuttle tiles would exceed structural limits and fail. At the time, Rockwell had already installed over 24,000 tiles on the vehicle before the root cause of problem was finally discovered, and a solution found. In fact, on the Shuttle ferry flight from Palmdale, California where the Shuttle was built, to its launch site in Florida at Kennedy Space Center (KSC), a large number of tiles fell off. In an actual mission, loss of even one TPS tile in a critical location could cause a burn through of the aluminum structure that would lead to loss of the vehicle during the high heating phase of entry. This Shuttle TPS anomaly was discovered late in the program and is notionally shown in Figure 3 by the steep increase of the “Knowledge Gap” (Kgap) curve. Because the anomaly was unanticipated, it caused delays, redesign and rework (which is called an engineering “loop-back” and illustrated by the blue arc denoted by 1000x) in the Shuttle development effort.  NASA created a “Tiger Team” led by Dr. Paul Cooper at LaRC, which included scientists and engineers from multiple NASA Centers, to determine the root cause of this very serious problem and recommend a solution (Cooper & Holloway, 1981). It is very important to emphasize that it required a small team of key SMEs in distinct areas of materials science, structural mechanics, structural dynamics, material and geometric nonlinear behavior, and advanced experimental techniques (in this case photoelasticity), to identify the elevated stress concentrations caused by the transverse fiber bundles of the strain isolator pad (SIP) material (Figure 3), which was the root cause of the reduction in transverse ultimate load of the bonded system. Only then was it possible to conceive a solution (in this case the densification of a thin layer of the tile adjacent to SIP) and conduct the necessary validation and verification testing. The real question, however, should have been: how could teams of professional thermal-structural and materials scientists and engineers not test the complete system of LI-900 ceramic tile, bonded to the strain isolator pad (SIP), and the aluminum structure of the Shuttle? A very simple and inexpensive pull test of the complete system, as part of a building block approach described earlier, would have rapidly and inexpensively identified the complex interactions which caused the emergent system properties to be degraded by over 50%.

Figure 3. Space Shuttle TPS bonding issue caused by complex systems interaction and subsequent property degradation, identified late in the product development life cycle (just prior to the first launch).

The case study above serves to highlight the drawbacks of a point-based design, phased-gated approach to product development with a NASA Space Shuttle design problem. It also highlights the value of an effective analysis/experiment building block approach to fail and learn as effectively as possible. The case study which follows, also taken from NASA experiences, provides some additional examples of experienced NASA research engineers raised in an R&D culture using principles of a SBCE/rapid research and development approach.

2.3 Knowledge Construction for Rapid R & D. Reinforced Carbon-Carbon (RCC) R&D Repair Effort Post Space Shuttle Columbia Disaster   

2.3.1 Early Exploration Phase Rapid R & D

After the loss of the Space Shuttle Columbia, one of the Columbia Accident Investigation Board (CAIB) recommendations for future flights was for NASA to develop methods to repair on-orbit damage similar to that which led to tragedy (Gehman, 2003). Damage to the reinforced carbon-carbon (RCC) wing leading edge (WLE) resulting from impact of insulation foam on ascent was believed to be the damage that led to the loss of the vehicle and crew during Earth entry on February 1, 2003 ( Gehman, 2003; Camarda, 2014b). 

A Technical Exchange Forum (TEF) was held at NASA Johnson Space Center (JSC) June 3-4, 2003 to investigate ideas for on-orbit repair of the Shuttle WLE. Numerous repair concepts and methods were suggested in multiple categories related to the size and type of damage and the suggested repair method.  The SSP was interested in repair ideas that spanned the smallest possible critical damage (for example, small cracks, holes, and/or SiC coating losses) to very large holes typical of the tests reported in (Camarda, 2014b)(approximately 16 in. [41 cm] square). For small damage, a spreadable pre-ceramic polymer was eventually developed that could be applied over a damage site by astronauts during a spacewalk (more correctly called an Extra-Vehicular Activity or EVA). The polymer would adhere to the damaged surface and cure on orbit, and then convert to a high-temperature ceramic barrier that would protect the surface from heating during entry. For larger damage (small to medium size holes or cracks), a “patch” that could be bonded to the outer surface, or a “plug” that would be mechanically attached to the surface of the RCC WLE and conform to the WLE outer moldline were developed (Camarda, 2007). The program also evaluated ideas for repairing larger holes, but discontinued pursuing them as they were deemed impractical. Some of these on-orbit, RCC WLE repair concepts and categories are shown in Figure 4.

Figure 4. Categories of types of reinforced carbon-carbon (RCC) wing leading edge on-orbit repair concepts.

In early June 2004 (over one year after the accident), it became obvious that the formal program TPS repair teams were struggling to develop concepts for on-orbit tile or RCC repair that would survive Earth entry. Charles Camarda approached the Orbiter Project Manager with a plan for developing a team to brainstorm new ideas for solving the problem. He also convinced astronaut Don Pettit (friend and fellow NASA astronaut classmate) to begin exploring ideas in secret in his garage/laboratory, and to work together in their spare time, while not interfering with their primary duties as Astronauts (see Figure 5).

Figure 5.  Covert exploratory on-orbit thermal protection system (TPS) design activity.

This duo expanded into a very small team of key SMEs in the areas of high-temperature structures, materials, and TPS from other government, industry, and academic connections.  This network helped to provide not only key ideas (for example, Francis Schwind of C-CAT, a carbon-carbon [C-C] material manufacturer, had the idea for drilling and tapping a hole in the C-C, and filling it with a C-C fastener and/or plug). Using this network, the team rapidly obtained materials, built and tested numerous leading edge wing and tile TPS repair concepts. They also explored approaches to drill holes in C-C and a doubly-curved, thin C-C plug, which would conform to numerous curvatures on the leading-edge surface.  The size of the circles in the network diagram in Figure 5 relate to the number of team members and the thickness of the lines is the relative communication traffic and/or strength of collaboration. This network was called the “Friends of Charlie” (FOC) network because it relied on trusted SMEs with whom Dr. Camarda had many years of experience working with, as well as a detailed understanding of personnel, and both the knowledge and experiences that each person had related to a specific domain (the “know who with the know how”, from Larsson, 2005).

The next phase of the rapid R&D effort called for the creation of a larger, more technically rigorous repair concept development team to close critical technical and manufacturing knowledge gaps. A two-and-a-half-day innovative design workshop was held at NASA LaRC in June 2004, with a small group of key researchers, engineers, designers (see Figure 6) from around the country. The approach used in this workshop later developed into a curriculum that became the foundation for the Innovative Conceptual Engineering Design (ICED) methodology (Camarda et al., 2010). The goal of the meeting was to exchange knowledge – each participant shared essential information from their perspective – and to define a broad design space by brainstorming concepts that could be used in a repair. Because of the varied backgrounds of the attendees, a good “cross-pollination” of ideas occurred.

The workshop was held at a facility experienced at such meetings, and it supplied floor-to-ceiling white boards, A/V equipment, computer capabilities, supplies, IT support, and a facilitator. The meeting first reviewed the current status of the RCC Repair Project, summarized the design requirements (for example, cost, schedule, technical requirements, constraints), presented the technology challenges with respect to several key disciplines (for example, aerothermodynamics, thermal, materials, and structures, etc.), reviewed the status of several key concepts such as crack repair and plug repair, and presented a short review of effective techniques for enhancing innovative thinking, such as brainstorming and TRIZ (Altshuller, 2001). Several brainstorming sessions were then held to identify potential repair approaches. To build the comradery needed for effective group dynamics, social activities such as dinners with keynote speakers were also included. As a result, over 70 individual repair concepts were generated in real time, with another 30 added the following week. These concepts were later categorized into classes having specific salient features in common.

Figure 6.  Diverse team of participants for a 3-day R & D On-Orbit Repair Workshop at NASA Langley Research Center (June 4, 2004).

Following the innovative design workshop at LaRC, a program team, called the R&D Repair Team, was formed with Dr. Stephen Scotti as its leader. The team then worked virtually and interacted several times a week to identify the critical knowledge gaps for each concept class, and to define and begin to execute the tasks necessary to close the gaps. 

Categories of repair concepts resulting from the workshop included metallic and ceramic shells that could deform to fit different surface curvatures, large flexible refractory metal and ceramic sheets that could cover the largest damage areas, soft gaskets and pastes to prevent hot-gas ingress though gaps, and many different types of fasteners and means to drill and tap holes in the leading edge. The critical knowledge gaps for each class were identified, both during the meeting and afterwards, and separate teams that “championed” a given class of repair concepts were formed to close the knowledge gaps. The most critical knowledge gaps dealt with were: 1) how a repair concept could be installed and verified by a space-walking astronaut, 2) whether or not the concept could withstand the temperatures and pressures of reentry, and 3) whether the concept could prevent the hot plasma formed during reentry from entering the interior of the leading edge.

Concept “gates”, defined as simple tests and analyses that establish concept feasibility, were established for each repair concept that provided goals for each team’s efforts. Within each concept class, the set-based design philosophy of eliminating the “weaker” solutions was performed within the team championing the concept. Weaker solutions were determined in several ways. They could have inferior performance as demonstrated by a quantitative metric, such as maximum operating temperature, they could have a larger number of knowledge gaps that could not be easily addressed, or they could have less applicability to the different damage scenarios than other alternatives within the class. However, when a solution was also applicable to teams outside their development team, such as a high temperature fastener that could be used with several repair concepts, it was not eliminated.  Each repair class team was allowed to continue their development in parallel as long as possible because the teams didn’t all proceed at the same pace, and a “show-stopper” in one class could be revealed late in the development. 

Following this approach, the network shown in Figure 6 was reconfigured several times during the effort to include additional participants (such as C-CAT mentioned earlier), as well as to “prune” branches that worked on approaches which were found infeasible. Some illustrative examples of team products for one repair concept, and of the many gaps that were closed, are shown in Figure 7. The rapid, set-based design approach utilized allowed the feasibility of this concept to be fully evaluated in only 3 months. Additional technologies and repair concepts that were developed by the team are shown in Figure 8. The development of these innovations followed the methodology described above. The R&D Repair Team closed the capability gap for repairing a large leading edge hole, a capability desired by the Shuttle program but initially believed to be infeasible, and a contingency repair kit flew on the Space Shuttle for the Hubble repair mission (STS-125).

Figure 7.  Rapid concept development efforts of one concept for Space Shuttle on-orbit wing leading edge repair.

Figure 8. Assorted technologies and concepts developed by the R&DR Space Shuttle Wing Leading Edge Repair Team to repair small to large on-orbit damage to the reinforced carbon-carbon (RCC) wing leading edge.

3. Discussion and Conclusions

In this paper, we have presented how set-based methods in product development may significantly improve product development, production, and quality assurance, as well as sophisticated problem solving in challenging environments. This paper relies on case studies from design challenges faced by NASA.

The two case studies from NASA’s real-world design activities highlighted several lessons-learned that illustrate the value of a set-based design approach for challenging and complex problems. In particular, they showed the value of starting with a broad design space, understanding the knowledge gaps (Kgaps) that characterize each potential design solution, using a systematic approach to closing the Kgaps (for example, the building block approach), and the pitfalls of selecting a point-design before the Kgaps are closed (for example, Space Shuttle Design). Also, it was shown that for situations where there is no known solution, and/or the problem is complex, the most significant Kgaps require understanding the root cause of failures so that a design can be tailored or modified to close them.

---

References

Altshuller, Genrich 2001.  And Suddenly the Inventor Appeared – TRIZ, the Theory of Inventive Problem Solving. Technical Innovation Center, Inc.  Worcester Massachusetts.

Artz, K. W., Norman, P. M., Hatfield, D. E., & Cardinal, L. B. 2010. A Longitudinal Study of the Impact of R&D, Patents, and Product Innovation on Firm Performance. Journal of Product Innovation Management, 27: 725–740.

Camarda, C. J. 2007. A Return to Innovative Engineering Design, Critical Thinking and Systems Engineering. The International Thermal Conductivity Conference (ITCC) and the International Thermal Expansion Symposium (ITES) Birmingham, Alabama June 24-27, 2007 Charles, 1–44.

Camarda, Charles J. 2009.  Failure is not an Option… It's a Requirement. AIAA Paper Number 2009-2255.  Presented at the 50^th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Palm Springs California, May 2009.

Camarda, C. J. 2014a. Space Shuttle Design and Lessons Learned. NATO Science and Technology Organization Lecture Series on “Hypersonic Flight Testing.” von Karman Institute, Rhodes-St-Genese, Belgium.

Camarda, C. J. 2014b. Space Shuttle Return-to-Flight Following the Columbia Tragedy. NATO Science and Technology Organization Lecture Series on “Hypersonic Flight Testing.” von Karman Institute, Rhodes-St-Genese, Belgium.

Camarda, Charles J., Scotti, Stephen, Kunttu, Iivari, and Perttula, Antti 2019.  "Rapid product development methods in practice - case studies from industrial production and tehnology development.'  Paper presented at ISPIM Connects Ottawa, Ottawa, Canada on 7-10 April, 2019.

Camarda, Charles J., Bilen, Sven, de Weck, Olivier, Yen, Jeannette, and Matson. ?? 2010.  "Innovative Conceptual Engineering Design - A Template to Teach Problem Solving of Complex Multidisciplinary Problems." ASEE Louisville, Kentucky.

Cloft, P. W., Kennedy, M. N., & Kennedy, B. M. 2018. Success is Assured - Satisfy your Customers on Time and On Budget by Optimizing Decisions Collaboratively Using Reusable Visual Models. Routledge Productivity Press.

Cooper, P. A., & Holloway, P. F. 1981. The Shuttle Tile Story. Aeronautics and Astronautics, 19(1): 24–34.

Evanschitzky, H., Eisend, M., Calantone, R. J., & Jiang, Y. 2012. Success factors of product innovation: An updated meta-analysis. Journal of Product Innovation Management, 29(1994): 21–37.

Gehman, H. W. 2003. Columbia Accident Investigation Board, Report Volume 1. U. S. Government Printing Office, Washington D. C.

Larsson, Andreas:  Engineering Know-Who - Why social Connectedness matters to Global Design Teams.  PhD Dssertation, Luleå University of Technology, 2005.

Matson, Jack V. 1996. Innovate or Die - A Personal Perspective on the Art of Innovation."  Paradigm Press Ltd.

McChrystal, General Stanley. 2015. Team of Teams - New Rules of Engagement for a Complex World.:  Penguin Publishing Group.

Mitchell, M. 2009. Complexity – A Guided Tour. Oxford University Press.

Schultz, James:  Crafting Flight - Aircraft Pioneers and the Contributions of the Men and Women of NASA Langley Research Center.  NASA History Series, SP 2003-4316.

Singer, D. J., Doerry, N., & Buckley, M. E. 2009. What is set-based design? Naval Engineers Journal, 121(4): 31–43.