On the Technological, Human, and Managerial Issues in Sharing Organizational Lessons

Rosina Weber,1,2 Leonard Breslow2, and Nabil Sandhu3

2

Department of Computer Science, University of Wyoming, Laramie, WY 82071-3682

Navy Center for Applied Research in Artificial Intelligence, Naval Research Laboratory, Washington, DC 20375

3

ITT Industries AES Division, Alexandria, VA 22303

{lastname}@aic.nrl.navy.mil

1

Abstract

Lessons learned systems (LLS) are systems that support a lessons learned process (LLP) to collect, verify, store, disseminate, and reuse organizational lessons. In this pa-per we examine technological, human, and managerial problems that contribute to the limited reuse of lessons in deployed LLS. This analysis results in the identification of a set of requirements that when met tend to improve the reuse of lessons. These requirements are mainly re-lated to the identification and representation of a lesson’s reuse components, i.e., what in a lesson is essential to promote reuse. We present and demonstrate a standard-ized format for lessons and a lesson elicitation tool (LET) that uses this format to collect lessons from human users and addresses some of the requirements while contribut-ing to the satisfaction of other requirements. This tool il-lustrates how technological solutions can impact human and managerial problems.

Introduction

Lessons learned systems (LLS) support a lessons learned process (LLP) to collect, verify, store, disseminate, and reuse organizational lessons. Existing systems support collection with textual input tools, store lessons in text databases, and support dissemination through standalone retrieval tools that offer variants of hierarchical browsing and keyword search. Verification and reuse are performed manually.

Knowledge artifacts (e.g., lessons learned, alerts, best practices) are knowledge structures that embed underlying concepts along with their conditions of applicability and rationale (Weber et al. 2001b). Organizational lessons (also called lessons learned, lessons identified) are or-ganization-specific knowledge artifacts that (a) originate from both successful and unsuccessful experiences, (b) are validated, (c) are applicable to a work practice of an organizational process, and (d) generate a positive impact on the process when reused (Weber, Aha and Becerra-Fernandez 2001a). They embody a suggestion that em-beds relevant knowledge that has been learned through working experiences and can be reused to generate a posi-tive impact on the process to which applied. Lessons must include a set of reuse-oriented components: the sug-__________

*. In Proceedings of the Fourteenth Annual Conference of the InternationalFlorida Artificial Intelligence Research Society, 334-338. Menlo Park, CA:AAAI Press. Copyright  2001, American Association for Artificial Intelli-gence (www.aaai.org). All rights reserved.

gestion, the applicable action (i.e., a work practice) it is intended to impact, the conditions for its applicability, and the originating event rationale. We suggest a standardized format for organizational lessons (Table 1) based on these components.

In a recent survey on lessons learned systems, Weber, Aha and Becerra-Fernandez (2001a) identified two main problems in that prevent these systems from distributing knowledge. First, the knowledge in organizational lessons is typically collected and stored in textual form. Textual lessons are difficult to retrieve and have to be interpreted (i.e., assessed for relevance) before they can be reused. Secondly, lessons learned repositories are built as stand-alone tools and therefore are never disseminated in the context in which they are used.

In this paper, we reexamine these technological prob-lems and analyze the corresponding human and manage-rial problems that obstruct knowledge sharing. When searching for remedies and appropriate techniques to im-prove lesson reuse, we describe solutions that also address human and managerial aspects.

content field name what is the suggestion to reuse, what to suggestion avoid or repeat

in which work practice (task, decision) to applicable action reuse it

under which context/circumstances the conditions lesson applies (or not), what are the restric-tions of its applicability

why, its rationale, what was the originating originating event event that caused that lesson to be learned

Table 1. Standardized format for organizational lessons.

The examination of these three aspects converge on the common requirement for a standardized format for les-sons that allows the implementation of other requirements (e.g., distribution in context, intelligent retrieval). In Sec-tion 3, we demonstrate the applicability of a standardized lesson format to four types of lessons. Among other things, this format underlies the implementation of a Les-son Elicitation Tool (LET) that offers a solution to tech-nological, human, and managerial problems we have identified.

Problems in the Lessons Learned Process

In this section we analyze the causes preventing knowl-edge sharing and their interrelations, focusing on the technological, managerial, and human problems.

Technological Issues

The LLP (Weber et al. 2000a) consists of five sub-proc-esses: collect, verify, store, disseminate, and reuse. In this section we describe technological barriers to lesson reuse that are related to all sub-processes of the LLP.

One problem confronting computerized lesson collec-tion stems from the fact that humans communicate their experiences in natural language. The verification of les-sons requires interpretation, what is neither precise nor easy, given textual lessons. While lesson storage can be implemented easily with textual format, intelligent re-trieval methods cannot be. Lesson dissemination and re-use are obstructed mainly because of standalone reposito-ries that are divorced from the context of reuse, i.e., in the context where and when knowledge is needed.

These technological issues can all be addressed with lessons represented in a standardized format that identi-fies explicitly a lesson’s reuse components: the sugges-tion, the applicable action it is intended to impact, the conditions for its applicability, and the originating event. Thus, we suggest a standardized format for organizational lessons (Table 1).

The adoption of this format tend to differentially affect the sub-processes of the LLP. Lessons collection can be implemented by eliciting directly the reuse components from users. The analysis of semantic patterns of such components allows the use of template-based methods (Weber et al. 2001b) that reduces dependency on natural language.

When using a standardized format, lesson verification is much facilitated. It even becomes possible to use auto-matic methods, as suggested by Vandeville and Shaikh (1999). Lesson storage in a standardized format enables the implementation of intelligent retrieval methods to in-crease the quality of retrieval.

Lesson dissemination and reuse benefit significantly from a standardized format because it allows dissemina-tion in context, which can substantially improve reuse. Lessons are disseminated in context when they are prompted to users in the context of distribution systems (e.g., decision support system) when and where users carry on their work, executing plans or making decisions (Reimer 1998). A standardized format makes the dissemi-nation in context possible because it simplifies the con-version of the lesson knowledge into the format of the distribution system.

Human Issues

Human issues related to the LLP are the ones concerned with organizational members who are responsible for submitting and reusing lessons.

When submitting lessons today, users typically use a collection tool to submit textual descriptions. In such tools, guidance is practically non-existent and the instruc-tions are not stated in terms of a known set of essential reuse components and thus are often unclear or ambigu-ous. In the absence of explicit and clear instructions re-lated to well understood reuse components, users tend to rely on concepts they already have in mind. In addition, users are invited to transcribe their experiences after re-turning from missions thus adding another tedious and exhausting task to an already difficult day. After lesson authors submit their lessons, their immediate superiors are in charge of verifying and validating them. The lack of a limited set of well-defined reuse components and explicit guidelines also affect the verifiers who tend to add more text, making lessons even longer, and thus more difficult to interpret, retrieve, and reuse.

These problems in lesson submission can be substan-tially decreased with the use of an elicitation tool that is built upon a standardized format that identifies a lesson’s reuse components. Such a tool can provide guidance to the user on what these components are and how lessons should be submitted in terms of them. The semantic pat-terns in such format can be identified in order to build lexicons so that the user instead of composing a textual description, can simply answer questions and select from lists. This allows an efficient verification and validation, reduces the author’s effort and supports consistency of representation. The quality of the lessons can be further improved if instructions (i.e., explaining what are the components and how they are supposed to be entered) are clear, simple, and supported by examples.

From the perspective of prospective users of lessons, the existing standalone architectures place the burden of lesson dissemination on the user. The user has to know the repository exists; has to believe that it will offer useful lessons; has to believe that lesson reuse is beneficial; has to know where and how to access the repository; has to have the time to access it; and has to learn how to use it (Weber et al. 2000a). All these difficulties are bypassed with dissemination in context, which is facilitated by the standardized format.

Finally, prospective lesson users need to retrieve les-sons and interpret them. Textual retrieval has low levels of precision (many non-relevant lessons are retrieved) and recall (many relevant lessons are not retrieved). Case re-trieval has the potential to increase quality of the retrieval. Once lessons are represented in the standardized format they can be used as cases (Aha and Weber, 2000). The requirement for interpretation is also overcame by the standardized format.

Managerial Issues

The managerial issues in the LLP are related to the man-agement’s responsibilities in knowledge creation (Mar-shall, Prusak, and Shpilberg 1996): • determine the knowledge • enable knowledge collection

• represent knowledge

• embed knowledge in targeted processes • verify and validate knowledge • oversee knowledge reuse • monitor knowledge transfer

• In the context of the LLP, all these managerial issues create infrastructure for the preceding responsibilities are directly related to the technological and human barri-ers that are preventing lesson reuse. Therefore, the solu-tions discussed in these respective sections already cover all the manager’s responsibilities. The definition of a les-son’s reuse components and a standardized format deter-mine knowledge content and format. This also enables lesson collection in conjunction with explicit guidelines that describe the content of the components. The repre-sentation also allows knowledge to be embedded in distri-bution systems, facilitates verification and validation of lessons, provides the conditions for lesson reuse and transfer and the infrastructure necessary for all these re-sponsibilities to be met.

In order to have a better LLP, managers should define a lesson’s reuse components, communicate guidelines with respect to these components, provide guidance to lesson authors, and enforce the guidelines. This is similar to what Davenport and Prusak (1998) refer to as setting the roles and skills. The elicitation tool described later covers these goals.

Having demonstrated the relevance of a standardized format, we now present the format that embeds the defini-tion of a lesson’s essential reuse components and illustrate its application to four types of lessons.

Representing Organizational Lessons

In this section, we demonstrate the applicability of the proposed standardized format (Table 1) to four different types of lessons we have identified. These lessons are exemplified in terms of the four reuse components of the standardized format: suggestion, applicable action, condi-tions of applicability, and originating action. This repre-sentation format was originally defined by Weber et al. (2000b) for organizational lessons. Analogous representa-tions can be defined for other knowledge artifacts (e.g., alerts, best practices, incident reports).

The three first elements suggestion, applicable action, and conditions of applicability are sine qua non for lesson reuse. However, humans are not easily persuaded to adopt a new work practice without a good justification. Thus, the justification is provided in the originating event that describes the event when the lesson was learned, specifies whether the event was a success or a failure experience, or an advice-giving event, and why the given suggestion should be followed. The motivation for including the originating event arises from the definition of lessons as originating from previous relevant experiences (i.e., suc-cesses, failures, advice).

These four elements are further decomposed for speci-ficity and verification. For example, the applicable action also includes an operation type. A lesson can also be

categorized in terms of the type of change that it suggests to the original work practice. For example, the suggestion may refer to a change of method, an inclusion of a new step, or a reorganization of the resources in the targeted work practice.

Variations in the originating event and the applicable action also define different types of lessons. This classifi-cation and its characteristics are intuitive and based on observations of organizational repositories. The types of lessons we exemplify are instance, generalization, innova-tion, and analogy. Instance and generalization types clas-sify lessons in terms of their coverage: instance lessons are applicable to one only action while generalization lessons are applicable to many. Analogy and innovation types designate the quality of the applicability of lessons. Analogy type are applicable because of a similarity be-tween the originating event and the applicable action, while innovation types are applicable in actions that are different from their originating events.

Instance Type. In an instance type lesson the action car-ried out in the originating event is equal to the applicable action. This lesson cannot be generalized to cover differ-ent variations of the applicable action; it is applicable exclusively to one action. The set of conditions that was present when the lesson was learned have to be similar to those present in time of reuse. Following is an example. Suggestion: Make sure to pave one lane at a time. Applicable action: pave lanes.

Originating event: successful; we were paving lanes and there was an emergency in the building which the road gave access to; the fire engine was able to use one of the lanes because we paved one at a time.

Conditions of applicability: Road lanes are the only ac-cess to a building.

Generalization Type. In a generalization type lesson the knowledge learned will be useful in several different tasks. This type is typical in organizations that are in the early stages of the maturity cycle. When the experience curve of an organization levels off, this type of lesson tends to become less frequent. However, they remain use-ful by adaptations to novel technologies and environ-ments.

In terms of reuse, lessons of the generalization type will typically be learned once; that is, the knowledge is easily incorporated into the regular procedures of the or-ganization. In this type of lesson, conditions are less im-portant, since its applicability is more dependent on the applicable action (that may vary) than on the contextual conditions. Generalization type lessons are useful for training because of their generality. This type of lesson tends to be self-justifying because it reflects common sense, as illustrated by the following example.

Suggestion: Make sure to review agreements between US and other countries to determine from which other coun-tries you will be forced to accept evacuees when estimat-ing the total number of evacuees.

Applicable action: Estimate the total number of evacuees. Originating event: advice; NEO planners need to be aware of formal and informal agreements between US and other nations.

Conditions of applicability: There were other countries’ nationals in the evacuation site.

Due to the generality of the suggestion, this lesson is ap-plicable whenever there might be agreements directing one’s conduct in a military operation.

Innovation Type. A lesson is learned in an originating event and the knowledge learned is applicable to a task that is different from task carried out in the originating event. This type of lesson is not very frequent since it characterizes innovation or discovery. That is, if we learn something useful for a given task in the context of per-forming a different one, it is likely that the action we have performed has more than one purpose or functionality. As long as a new function (lesson) has been identified and validated, it is immediately incorporated into the organ-izational knowledge.

Suggestion: Install light bulbs in a cabinet to reduce hu-midity.

Applicable action: Reduce humidity inside a storage compartment.

Originating event: successful; install light bulbs under the cabinet for the purpose of illumination and discover that it reduces humidity too.

Conditions of applicability: Power is available.

In lessons of types instance, generalization, and inno-vation, the suggestion recommends repeating or avoiding an originating event that was, respectively, a success or a failure, or simply following a piece of advice. This is not the case for the analogy type.

Analogy Type. In analogy type lessons, the suggestion recommends repeating or avoiding an interpretation or the strategy used in the originating event, instead of repeating or avoiding the action itself. The key characteristic in this type of lesson is that the originating event is different from the applicable action but the same strategy is appli-cable to both.

An analogy type of lesson can be either of instance or generalization type. Its utility has to be determined by the lesson user, since its interpretation or strategy is not rep-resented explicitly. The following example is expressed at an abstract level so that its strategy is explicit.

Suggestion: Make sure to assign high visibility to some-thing if you want to distract attention from something else that should be kept inconspicuous.

Applicable action: Accomplishing a task with a covert resource.

Originating event: advice; using covert special operation forces together with high visibility conventional forces when pulling out evacuees. Special operation forces pro-

vided clandestine assistance and the enemy did not sus-pect their presence.

Conditions of applicability: It is necessary to use an in-conspicuous resource.

These four types of lessons were described in terms of a lesson’s reuse components and they indicate the suffi-ciency of these components to represent lessons. How-ever, to guarantee its completeness, it is necessary to evaluate this representation in an experiment with actual prospective users. In next section we describe an AI tool that uses this standardized format.

Implementing an AI Solution

The assessment of human, technological, and managerial aspects indicated requirements (e.g., representation and submission of lessons; definition, communication, and enforcement of guidelines for lesson submission) with the potential to remedy the current limited reuse of lessons. In this section we present an elicitation tool that meets such requirements thus counteracting technological, human, and managerial barriers to lesson reuse.

Lesson Elicitation Tool

The lesson elicitation tool (LET) performs lesson collec-tion by interacting directly with users who have an organ-izational lesson to submit (i.e., lesson authors). This tool implements the LLP's collection sub-process, in particu-lar, an interactive collection sub-process (Weber, Aha and Becerra-Fernandez 2001a). The following goals guided its implementation: to guide the user in submitting a lesson, to ensure the user has in fact a valid lesson to submit, to orient the user in what a valid lesson is (i.e., what the re-use components are), to restrict the language of the con-versation, to enforce the guidelines by giving drop-down lists to the user to select from and by giving examples, to capture the reuse components, to check for spelling, cor-rectness, consistency, etc.

Figure 1 illustrates how the LET orients users during lesson submission by offering an interface that explains the purpose of each of the reuse components in clear and precise language. These instructions are reinforced by a set of examples that illustrate how users should respond to questions and communicate lessons. The questions are asked and the user answers them by filling out templates and selecting from drop-down lists (list of verbs in Figure 1).

When filling out templates, users are describing a les-son’s reuse components. The tool uses a subset of natural language to restrict user’s input, to facilitate understand-ing, to avoid ambiguous representations that require inter-pretation, and to exclude irrelevant content. The goal is to disambiguate the input as it is submitted to reduce ambi-guity in the stored content thus providing more reliable retrieval.

Figure 1. LET interface to elicit applicable action field.

The main benefit from this collection method is that it extracts relevant information and builds a model (stan-dardized format) of the lesson that is computationally usable. It starts from the definition of a lesson’s reuse components and it uses knowledge of these components to allow the use of methods that reduce the impact of natural language understanding problems. It can verify user’s input and clarify ambiguities during collection. This verifica-tion is supported by the use of a domain ontology. The LET standardizes the knowledge content and format facilitating disambiguated acquisition and producing a description that shows explicitly what to re-use, when, how, and why. Short sentences that describe directly the reuse components do not challenge interpreta-tion. Using the LET, users type less and do not need to compose textual descriptions of their experiences. Sub-mission becomes an easier task with more quality as the instructions and examples orient, requirements for con-firmations verify, and the restricted input restrict the sub-mission of irrelevant content. Verification and validation are facilitated and have the potential to result in better quality lessons instead of additional text. Moreover, the standardized format allows more reliable retrieval in em-bedded architectures, permitting lesson dissemination in context (as explained in the Section Technological Is-sues). Finally, all these capabilities represent the neces-sary infrastructure that management is supposed to pro-vide.

Further details of the implementation of the elicitation tool are described in (Weber et al. 2001b).

Concluding Remarks

In this paper, we extended the examination of causes of limited lesson reuse in lessons learned systems. We pre-sented a lesson elicitation tool (LET) that uses a standard-ized format for lessons that models a lesson in a form that is computationally treatable. LET represents a technologi-cal solution to problems in the sub-processes collect, ver-ify, store, disseminate, and reuse. We conclude that hu-man and managerial issues can be addressed with technological solutions such as the one described.

We described a standardized format tailored to organiza-tional lessons that can be adapted to other knowledge arti-

facts (e.g., alerts, best practices, incident reports) that can be described in terms of reuse components. The standard-ized format is demonstrated in four types of lessons that characterize different relations between an originating event (when a lesson was learned) and an applicable ac-tion (when a lesson can be reused). This classification is intuitive and based on observations of organizational re-positories.

Acknowledgements

We would like to thank David W. Aha for his continuous guidance. This research was sponsored by grants from the Office of Naval Research, the Naval Research Labora-tory, and the University of Wyoming.

References

Aha, D. W. and Weber, R. 2000 Intelligent Lessons Learned Systems: Papers from the AAAI Workshop (Tech. Rep. WS-00-08). Menlo Park, CA: AAAI Press. Davenport, T.H., and Prusak, L. 1998 Working knowl-edge: How organizations manage what they know. Bos-ton, MA: Harvard Business School Press.

Marshall, C., Prusak, L. and Shpilberg, D. 1996. Financial Risk and the Need for Superior Knowledge Management. California Management Review 38 3: 77-101.

Reimer, U. 1998. Knowledge Integration for Building Organizational Memories. In Proceedings of the Eleventh Banff Knowledge Acquisition for Knowledge-Based Sys-tems Workshop. Banff, Alberta, Canada, April 18-23,

1998. URL http://research.swisslife.ch/Papers/data/orgmem/kaw98/kaw_paper.html

Vandeville, J.V. and Shaikh, M. A. 1999 A structured approximate reasoning-based approach for gathering “les-sons learned” information from system development pro-jects. Systems Engineering 2(4): 242-247.

Weber, R., Aha, D.W., Branting, L.K., Lucas, J.R., and Becerra-Fernandez, I. 2000a. Active case-based reasoning for lessons delivery systems. In Proceedings of the Thir-teenth Annual Conference of the International Florida Artificial Intelligence Research Society, 170-174. Menlo Park, Calif.: AAAI Press.

Weber, R., Aha, D.W., Muñoz-Avila, H., and Breslow, L.A. 2000b. An Intelligent lessons learned process. In Z.W. Rás and S. Ohsuga eds. Foundations of Intelligent Systems 12th International Symposium, ISMIS'2000, 358-367. Berlin Heidelberg: Springer-Verlag.

Weber, R., Aha, D.W., and Becerra-Fernandez, I. 2001a. Intelligent lessons learned systems. International Journal of Expert Systems Research and Applications 20 1:17-34. Weber, R., Aha, D.W., Sandhu, N. and Muñoz-Avila, H. 2001b. A Textual Case-Based Reasoning Framework for Knowledge Management Applications. GWCBR 2001. Forthcoming.