Lead Users (Von Hippel)

Embed Size (px)

Citation preview

  • 8/9/2019 Lead Users (Von Hippel)

    1/21

    Performance Assessment of the Lead User Idea-Generation Process for New

    Product Development

    Gary L. Lilien; Pamela D. Morrison; Kathleen Searls; Mary Sonnack; Eric von Hippel

    Management Science, Vol. 48, No. 8. (Aug., 2002), pp. 1042-1059.

    Stable URL:

    http://links.jstor.org/sici?sici=0025-1909%28200208%2948%3A8%3C1042%3APAOTLU%3E2.0.CO%3B2-6

    Management Science is currently published by INFORMS.

    Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available athttp://www.jstor.org/about/terms.html. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtainedprior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content inthe JSTOR archive only for your personal, non-commercial use.

    Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained athttp://www.jstor.org/journals/informs.html.

    Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printedpage of such transmission.

    The JSTOR Archive is a trusted digital repository providing for long-term preservation and access to leading academicjournals and scholarly literature from around the world. The Archive is supported by libraries, scholarly societies, publishers,and foundations. It is an initiative of JSTOR, a not-for-profit organization with a mission to help the scholarly community takeadvantage of advances in technology. For more information regarding JSTOR, please contact [email protected].

    http://www.jstor.orgFri Nov 23 04:49:01 2007

    http://links.jstor.org/sici?sici=0025-1909%28200208%2948%3A8%3C1042%3APAOTLU%3E2.0.CO%3B2-6http://www.jstor.org/about/terms.htmlhttp://www.jstor.org/journals/informs.htmlhttp://www.jstor.org/journals/informs.htmlhttp://www.jstor.org/about/terms.htmlhttp://links.jstor.org/sici?sici=0025-1909%28200208%2948%3A8%3C1042%3APAOTLU%3E2.0.CO%3B2-6
  • 8/9/2019 Lead Users (Von Hippel)

    2/21

    Performance Assessment of the LeadUser Idea-Generation Process

    for New Product DevelopmentGary L. Lilien Pamela D. Morrison Kathleen SearlsMary Sonnack Eric von HippelSmeal College of Business Adm inistra tion, Pennsylvania S tate U nivers ity,

    University Park, Pennsylvania 16802Faculty of Commerce and Economics, University of New South Wales, Sydn ey NSW 2052, Australia

    Strategic Services, AS1 Associates, 333 N . Washing ton Avenue, M inneapolis, Minnesota 554013M Company, 7840 Mississippi Lane, Brooklyn Park, Minnesota 55444-2152MIT Sloan School of Management, Cambridge, Massachusetts [email protected] [email protected] [email protected]

    [email protected] [email protected]

    T raditional idea generation techniques based on customer input usually collect informa-tion on new product needs from a random or typical set of customers. The "lead userprocess" takes a different approach. It collects information about both needs and solutionsfrom users at the leading edges of the target market, as well as from users in other marketsthat face similar problems in a more extreme form. This paper reports on a natural experi-ment conducted within the 3M Company on the effect of the lead user ( L U )idea-generationprocess relative to more traditional methods. 3M is known for its innovation capabilities-and we find that the LU process appears to improve upon those capabilities. Annual salesof LU product ideas generated by the average LU project at 3M are conservatively projectedto be $146 million after five years-more than eight times higher than forecast sales for theaverage contemporaneously conducted "traditional" project. Each funded LU project is pro-jected to create a new major product line for a 3M division. As a direct result, divisionsfunding LU project ideas are projecting their highest rate of major product line generationin the past 50 years.(N ew Product Development; Lead Users; Idea Gen eration)

    1. IntroductionMany firms generate new product ideas based oninformation collected from current or potential users.What distinguishes such "customer-focused" pro-cesses across companies is the kind of informationthey collect and the respondents from whom theycollect it. Traditional market research techniques col-lect information from users at the center of the targetmarket. The "lead user" ( L U )process takes a differ-ent approach, collecting information about both needs

    and solutions from the leading edges of the targetmarket and from markets facing similar problems ina more extreme form.The research question we address in this paper is:How does the LU process actually perform relativeto more traditionally used methods? We are moti-vated to explore this matter by accumulating evi-dence that ideas generated by traditionally used pro-cesses are rarely breakthroughs, tending instead tobe marginal contributors to firms' product portfolios

    MANAGEMENTCIENCEO 2002 INFORMSVol. 48, No. 8, August 2002 pp. 1042-1059 0025-1909/02/4808/1042$5.001526-5501 electronic ISSN

  • 8/9/2019 Lead Users (Von Hippel)

    3/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPELPerformance Assessment of Lead User Data

    (Eliashberg et al. 1997). This evidence has heightenedinterest in nontraditional techniques in general, and innovel idea-generation techniques in particular (Gold-enberg et al. 2001). Our research involves a naturalexperiment conducted within 3M divisions to quan-titatively compare outcomes produced by LU idea-generation projects with those of projects using moreconventional approaches.

    The LU method as conducted at 3M involves iden-tifying and learning from lead users both within thetarget market and in "advanced analog" markets thathave similar needs in a more extreme form. We findthat this method generates breakthrough new prod-ucts at a higher rate than methods traditionally usedat 3M. Annual sales for the average funded LU projectidea was forecast by management to be $146 millionin year five-more than eight times higher than pro-jected sales for contemporaneous traditional projects.Funded ideas from a total of only five LU idea-generation projects were conservatively projected toyield $730 M in incremental annual sales for 3M.

    In 52 of this paper, we review the literature on ideageneration and the LU method. In 53 we present ourresearch setting, qualitative pilot research, and ourresearch hypotheses. In 54, we describe our quantita-tive research methods. In 55 we present our findings.In 56 we discuss the implications of the study, and in57 we offer suggestions for further research.

    2. Background andLiterature ReviewDespite the strategic importance of effective newproduct development as a source of competitiveadvantage, most new product development activi-ties fail to achieve their anticipated level of marketsuccess. Thus, Eliashberg et al. (1997) report on asurvey of 154 senior marketing officers of U.S. cor-porations, 79% of whom believe their new productdevelopment process could be significantly improved.A major complaint they identified was that most ofthe products developed tended to be marginal con-tributors to the firm's portfolio, rarely involving verynew or "breakthrough" ideas. Eliasberg et al. thenassess the range of market research methods that

    MANAGEMENT 48, N o. 8, August 2002C I E N C E /V O~.

    might enhance the output of new product develop-ment processes, classifying them by type of productsought (radically new vs. marginally new) and natureof the research method (traditional vs. nontraditional).They point out that traditional marketing researchmethods obtain information from respondents at thecenter of the market-respondents whose thinkingis limited by their current experience and environ-ment. They also note that these traditional methodshave generally failed to produce radical new productbreakthroughs, leading to interest in nontraditionalmethods.

    A review of marketing literature on techniques andtools used to generate ideas for new products andservices makes clear two major points of differencebetween methods traditionally used and the LU idea-generation method: the kind of respondents fromwhom information is collected and the type of infor-mation that is collected. What we will term "tradi-tional" idea-generation methods have two character-istics in common. First, traditional methods obtaindata from representative users or customers at ornear the center of the intended target market. Sec-ond, they obtain need information only, and assignthe task of generating ideas for solutions leading tonew products to manufacturers (Griffin 1997, Haman1996, Rangaswamy and Lilien 1997, Eliashberg et al.1997, Lonsdale et al. 1996). In contrast, LU idea-generation methods collect information on both needsand ideas for solutions from "lead users." Lead usersare defined as users of a given product or servicetype that combine two characteristics: (1) They expectattractive innovation-related benefits from a solutionto their needs and so are motivated to innovate,and (2) they experience needs for a given innova-tion earlier than the majority of the target market(von Hippel 1986).

    It is possible that traditional methods' focus onrepresentative users is a cause of their apparentlypoor ability to generate ideas for "breakthroughs"-defined here as new product ideas that form the basisfor an entire new line of products or services. Classicalresearch on problem solving shows that subjects arestrongly constrained by their real-world experience,an effect called "functional fixedness." Thus, thosewho use an object or see it used in a familiar way

    1043

  • 8/9/2019 Lead Users (Von Hippel)

    4/21

    LILIEN, MORRISON, SEARLS, SONN ACK, A ND VO N HIPPELPerformance Asse ssm ent of Lead U ser Data

    are blocked from using that object in a novel way(Dunker 1945, Birch and Rabinowitz 1951, Adam-son 1952). Furthermore, the more recently objects orproblem-solving strategies have been used in a famil-iar way, the more difficult subjects find it to employthem in a novel way (Adamson and Taylor 1954).Thisfixedness effect is also displayed in innovation prac-tice (Allen and Marquis 1964).

    If familiarity with existing product attributes anduses interferes with the ability to conceive of novelattributes and uses, then representative target-marketcustomers, users of today's products, would seem tobe poorly situated to envision novel needs or solu-tions. In contrast, lead users would seem to be bettersituated in this regard-they "live in the future" rela-tive to representative target-market users, experienc-ing today what representative users will experiencemonths or years later.

    As was proposed by von Hippel (1986), LU idea-generation studies can identify and learn from leadusers, both within and well beyond intended targetmarkets. Lead users found outside of a target marketoften encounter even more extreme conditions on atrend relevant to that target market. They may, there-fore, be forced to develop solutions that are novelenough to represent "breakthroughs" when appliedto the target market. For example, auto manufac-turers seeking to improve auto efficiency by reduc-ing weight with lighter, stronger materials might findbreakthroughs in the aerospace industry, which facesa very similar problem in an even more extreme form.

    The second major difference between traditionalmarketing research idea-generation methods and leadusers' methods is that the former only collect dataon needs from users or customers. Manufacturer-basedproduct developers then have the task of creatingideas for new products and services based upon thatneed data. Tools used to support ideation includebrainstorming (Rossiter and Lilien 1994, Hargadon1996) focus groups, customer visits (McQuarrie 1998),conjoint analysis and "channeled ideation" (Golden-berg et al. 1999).

    The value of assigning idea generation to manu-facturers, as is traditionally done, is not supportedby research. As Goldenberg et al. (2001) point out,". . the marketing literature has paid little attention

    to the way an idea is generated as a possible pre-dictor of its success." (p. 71). They themselves findthat ideas based on solution spotting (where the solu-tion either precedes traditional market research orwhere the solution and the need are spotted simulta-neously) are superior to those associated with priorneed assessment (need spotting). Similar results arereported by Finke et al. (1992).

    Two findings from innovation research also sup-port the potential utility of the LU approach. Thefirst shows that users rather than manufacturers areoften the initial developers of-the "idea generators"for-what later become commercially significant newproducts and processes (e.g., Enos 1962, Freeman1968, von Hippel 1988, and Shaw 1985). The secondfinding shows that innovation by users tends to beconcentrated among lead users of those products andprocesses (Urban and von Hippel 1988, Morrison et al.2000, Shah 1999, Luthje 2000).2.1. The LU Idea-Generation ProcessIn general overview, the LU process involves fourmajor phases.

    Phase 1: Goal Generation and Team Formation.A firm provides an ideation-goal for the project andforms a LU project team. Teams typically are com-posed of three to five experienced people from mar-keting and technical departments, with one memberserving as project leader. Team members typicallyspend 12 to 20 hours per week on a project. Typi-cal project duration is four to six months. During theinitial project phase, the LU project team works withrelevant key company stakeholders to select the gen-eral market it will target and the type and level ofinnovation desired.

    Phase 2: Trend Research. During Phase 2, LUteams focus on identifying and deeply understand-ing important market and technical trends in thefield being explored. Team members begin by review-ing conventional information sources. Next, they sys-tematically identify and interview leading experts inthe marketplace that they are exploring-people whohave a broad view of emerging technologies andleading-edge applications in that field or fields. Theythen select an important trend or trends as the centralfocus of further project work.

    1044 S C I E N C E / V O ~ .ANAGEMENT 48, No . 8 , August 2002

  • 8/9/2019 Lead Users (Von Hippel)

    5/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPELPerformance Assessm ent of Lead User Data

    Phase 3: LU Pyramid Networking. In Phase 3,unique to the LU process, the project team engagesin a "pyramid" networking exercise to identify andlearn from users at the leading edge of the importanttrends selected for as a study focus. They identify leadusers both in the target market and in other marketsthat face similar needs in an even more extreme form,and learn from those lead users about needs andsolutions they are encountering at the leading edge.The pyramid networking technique, a modified ver-sion of the "snowballing" technique, relies on the factthat people with a strong interest in a topic or fieldtend to know people more expert than themselves: Forexample, good computer science professionals tend toknow the identity of computer scientists who are evenmore skilled than they are (von Hippel et al. 1999).

    Phase 4: LU Workshop and Idea Improvement.In the fourth and final phase of a LU project, activ-ity centers around a LU workshop in which invitedlead users work with company personnel to improvethe preliminary concepts generated in Phase 3-andsometimes to generate entirely new concepts. Typi-cally, 10 to 15 people attend this workshop, of whicha third may come from the firm sponsoring the study.Participants first work in small groups, then work as awhole to design final concepts that fit the company'sneeds. Thereafter, the entire group evaluates the con-cepts in terms of technical feasibility, market appeal,and management priorities.2.2. The LU Method Track RecordThe three empirical studies of LU idea-generationprocesses published to date all utilized inputs fromusers at the leading edge of the target market, andall reported developing concepts for valuable, "nextgeneration" products. Thus, Urban and von Hippel(1988) evaluated the LU idea-generation method ina laboratory setting in the field of computer-aidedsystems for the design of printed circuit boards-PC-CAD. They followed the four-phase LU processdescribed earlier, drawing information from a sampleof 136 lead and representative users identified withinthe PC-CAD market. The new products that emergedfrom LU need and solution data were greatly pre-ferred by potential users compared to product con-cepts generated by more traditional methods.

    MANAGEMENT 48, N o . 8, August 2002C I E N C E /V O~.

    Herstatt and von Hippel (1992) documented aLU project seeking to develop a new line ofpipehangers-hardware used to attach pipes to theceilings of commercial buildings. Hilti, a major globalmanufacturer of construction-related equipment andproducts, conducted the project. The firm introduceda new line of pipehanger products based on the LUconcept, and a poststudy evaluation has shown thatthis line has become a major commercial success forHilti, according to private communication from thefirst author.

    Olson and Bakke (2001) report upon two LU studiescarried out by Cinet, a leading IT systems integratorin Norway, for the firm's two major product areas-desktop PCs, and Symfoni application Groupware. Inboth studies, data collection focused on lead usersin the target markets. These projects were very suc-cessful, providing most of the ideas incorporated intonext-generation products. However, one year aftertheir LU method trial, the firm had reverted to itstraditional pattern of learning from suppliers ratherthan customers, indicating the difficulties of changingtraditional practices within firms.

    3. Research Goals, Setting, PilotResearch, and HypothesesPast evaluations of the LU process assessed projects

    utilizing input from lead users at the leading edgeof target markets and reportedly produced conceptsfor valuable "next-generation" products. Our researchseeks to extend these findings in two ways. First,we investigate whether LU projects that identify andlearn from lead users from fields in advance of the lead-ing edge of target markets develop concepts for newproducts that are more valuable-i.e., are more likelyto produce "breakthroughs" that set the basis for newproduct lines. Second, we want to compare the effec-tiveness of the LU procedure in a field setting againstprocedures used by non-LU teams in the same setting.3.1. Research SettingFollowing Cook and Campbell (1979), we sought afield situation that could closely approximate a pre-post/test-control situation, with at least quasi-random

    1045

  • 8/9/2019 Lead Users (Von Hippel)

    6/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPELPerformance Assessment of Lead User Data

    assignments to treatment cells, satisfying three con-ditions. First, the site should be one for which inno-vation has historically been strategically important.This condition would ensure that the LU interventionwould not signal a concurrent increase in the strategicincentive to innovate, a change that might improveinnovation performance independent of any effectcaused by the method itself. Second, the site must beusing both LU methods seeking lead users beyond thetarget market and traditional idea-generation meth-ods, which could act as a control within the sameorganizational units in parallel with the LU method.Third, the site must have data on a sufficient num-ber of projects utilizing both LU and traditional idea-generation processes to permit statistical distinctionbetween the effect of the methods from other firm-orindustry-specific characteristics. (Henceforth, we willuse the term "non-LU" methods to refer to the rangeof more mainstream methods 3M has traditionallyemployed for idea generation.)

    At the time of our study, we found that 3M satisfiedthese three requirements as follows:

    (1) 3M has historically been known for, and hasalways placed major emphasis on, innovation. Asthey note in their 1999 Annual Report, "For nearly100 years, 3M has grown by pioneering innovativetechnologies and products. . . . Innovation remains thedriving force of 3M culture and growth." (3M, AnnualReport, 1999. p. 12.)

    (2) 3M first began using the LU method in one divi-sion in 1996. All 3M LU projects involved searchesfor lead users both within and beyond intended tar-get markets. By May 2000, idea-generation projectsusing the LU method had been carried out in severaldivisions, each of which was at the same time alsoemploying more traditional idea-generation methods.These divisions therefore could play the role of exper-imental units.(3) By May 2000, 3M had completed seven LUprojects and had funded further development of theideas generated by five of these. The divisions carry-ing out LU projects also had 42 contemporaneouslyfunded projects available for study. These compari-son projects used a range of non-LU idea-generationprocesses, described later.

    While 3M cooperated in our study and permittedaccess to company records and to new product devel-opment team members, the firm did not offer us acontrolled experimental setting. Rather, the organiza-tion agreed to serve as an environment for a naturalexperiment where we as researchers would have toaccount for any naturally occurring differences afterthe fact.3.2. Pilot Research and HypothesesWe conducted an inductive, qualitative research phaseat 3M to generate more specific research hypothe-ses and associated measurements. Our approach uti-lizes a research tradition informed by both ethno-graphic (Douglas 1986, Willis 1981, Lave 1988, VanMaanen 1988, and Workman 1993) and grounded the-ory (Glaser and Strauss 1967) perspectives, and incor-porated three goals:

    (1) Develop trust-based relationships between 3M newproduct development professionals and researchers. Toimplement this goal, we began our grounded researchactivities only after obtaining strong top manage-ment support and permissions for access to confi-dential company data. These permissions enabled ourresearch team to develop unguarded and informalrelationships with 3M product development teams.

    (2) All ow data and the field to drive theory creation.Product developers throughout the firm shared theirtime and helped us understand how innovative prod-uct concept generation is practiced, understood, andcommunicated at 3M. Their insights were an impor-tant input into hypothesis formulation (Wenger 1998).

    (3) Use fieldwork to drive measurement development.We studied the variables and scales used internally by3M related to new product development and resourc-ing decisions and integrated them with those weacquired from the academic literature. To the extentpossible, our measures incorporated the types of datacustomarily collected by 3M.In the course of our grounded research at 3M, wespecifically sought balance between LU and non-LUparticipants. We interviewed more than 20 managersand related new product development professionalsin several 3M divisions. These internal experts rep-resented the technical/scientific side of new prod-uct development (the professional group historically

    MANAGEMENT 48, No . 8, August 2002C I EN C E/V O~.046

  • 8/9/2019 Lead Users (Von Hippel)

    7/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPELPerformance Assessme nt of Lead User Data

    most credited as being the source of the company'sinnovative product offerings) and the business side(including marketing professionals). Research teammembers also observed and participated in LU train-ing sessions and studied relevant internal companydocumentation.

    Our fieldwork showed that all LU teams weretaught the same process by the same small group ofcoaches working from the same set of written train-ing materials. We also found that both instruction andproject team practice drove 3M teams to and thenbeyond the leading edges of the target market to iden-tify and learn from lead users in "advanced analog"markets.

    Three clear concerns about the LU process emergedduring the course of our interviews. First, some inter-viewees expressed concern that ideas developed viathe LU process might have low "organizational fit"with the 3M technical, production, and market envi-ronment, and hence, be less likely to be judged wor-thy of funding by 3M management. If this concernproved correct, it would represent a major barrier togaining actual marketplace benefit from any break-through ideas generated by the LU method. Second,some managers expressed a concern that a processbuilt upon distilling new ideas from LU needs andprototype solutions would result in ideas that couldnot be effectively patented by 3M, a major draw-back to the method if true. Third, many LU partici-pants expressed concern about the greater time andeffort involved in the LU idea generation method rel-ative to alternative approaches, and the impact of thattime and effort on managers' willingness to use themethod.

    Following our grounded research, we generatedfive hypotheses. Hypotheses 1 and 2 measure basicoutputs, and their motivation derives from the liter-ature cited earlier. Hypotheses 3-5, derived from ourgrounded theory approach, express the three majorconcerns of interviewees, described above, in hypoth-esis form.

    HYPOTHESISI. LU methods will generate ideas withgreater commercial potential than will non-LU methods.

    HYPOTHESIS2. LU methods ide nt~ fyi ng nd learningfrom lead users outside of the target market wil l increase

    MANAGEMENT 48, No. 8, August 2002CIENCE/VOI.

    the overall rate at which the organization generates majornew product lines.

    HYPOTHESIS3. Projects from LU methods willexhibit a lower level of organizational 'Ifit" than will non-LU projects.

    HYPOTHESIS4. Ideas generated by LU methods willbe less protectable by patent or other means of intellectualproperty protection than will ideas generated by non-LUmethods.

    HYPOTHESIS5. LU methods will cost more in moneyand time than non-LU methods to generate ideas formingthe basis for funded projects.

    4. Research Samples, DataCollection Instruments,and Methods4.1. Research SamplesWe sought samples of development projects in 3Mdivisions that differed with respect to their use of theLU process, but that were in other respects as simi-lar as possible to those from the LU process. Whereunavoidable differences did exist between our sam-ples, we attempt to err in the conservative directionto ensure that we do not overly favor the outcomes ofthe LU process. We control for the impact of division-related variables by selecting projects only from divi-sions that were carrying out both LU and non-LUprojects. Within these divisions, we selected two typesof samples.

    Cross-Sectional Sample. Our first sample con-sists of "funded ideas." The earliest stage involving3M critical management review is the initial requestfor funding; a "funded idea," therefore, has receivedcareful review and positive evaluations from 3Mdivisional managers and has received, at a minimum,initial funding within the 32 product developmentsystem Our sample of funded ideas was a completecensus of all ideas receiving funding in the 3M divi-sions that funded both LU and non-LU project ideas

    1047

  • 8/9/2019 Lead Users (Von Hippel)

    8/21

    LILIEN, MOR RISON, SEARLS, SONNA CK, AN D VON HIPPELPerformance Assessm ent of Lead User Data

    during our data collection period of February 1999-May 2000. We identified funded ideas via divisionalrecords and generated a sample of five funded ideasdeveloped by LU studies ("LU ideas") and 42 fundedideas developed by other, more traditional, divisionalprocesses ("non-LU ideas). This process controls forvariation over time in criteria that management mightapply to the funding decision.

    We next had to address the issue of possible differ-ences between project staffing and performance incen-tives. To explore possible staff differences, we com-pared the backgrounds, personalities, and skills of LUproject participants with those in non-LU groups viaa survey. We found that those staffing LU and non-LU groups were not significantly different in termsof job level, years employed at 3M, rate of careeradvancement, and other background variables. How-ever, they did differ significantly on two self-reportedskill dimensions. LU project participants gave them-selves significantly higher ratings with respect to gen-eral marketing/sales skills (the means differed by0.72 points on a 5-point scale, p < 0.03). Also, non-LU project participants gave themselves significantlyhigher ratings on technology skills (the means dif-fered by 0.62 points on a 5-point scale, p < 0.05). Inaddition, LU group participants differed significantlyfrom non-LU group participants on one of four MyersBriggs scales: They rated significantly higher on theextrovert/introvert dimension (the means differed by11 points on a 100-point scale, p < 0.08).

    The differences noted are a potential source ofconcern with respect to the validity of our naturalexperiment on the impact of the LU idea-generationprocess. Perhaps 3M LU project teams could gen-erate better ideas simply because they containedmore skilled (and extroverted) marketing researchers.However, a more detailed look suggests that this isunlikely. Although the difference was statistically sig-nificant, LU team members ranked themselves onaverage only 15% higher on the marketing/sales skillsscale than did non-LU team members. In addition,the overlap between conventional marketing researchskills and the activities prescribed for LU teams islow-and the skill level differences reported did notcorrelate significantly with differences in the activities

    the team members reported carrying out during theirpr0jects.l

    We next sought differences in the motivation ofLU and non-LU team members to achieve a break-through and found none. Our interviews with 3Mmanagement revealed that concerns about corporategrowth and margins had led to intense pressure beingput on all personnel at the levels of LU and non-LUteam members to achieve new "breakthrough" prod-ucts and product lines. We could find no evidencethat there was any difference in a priori attractivenessof the areas of focus between LU and non-LU teams.Managers said that there was no difference, and a con-tent analysis of formal annual performance goals setfor the individual LU and non-LU team members inthe one division that allowed us access to this data,supported their views that equivalent opportunitiesfor and pressure for breakthroughs existed for all.

    Even given equivalent incentives being appliedto LU and non-LU project teams, it is possiblethat Hawthorne or placebo effects were impactingthese teams differentially. (For our purposes here, theHawthorne effect can be described as "I do betterbecause extra attention is being paid to me or to myperformance." The placebo effect can be described as"I expect this process will work and so I will striveto get the results that have been described to me.")While such effects are possible, our fieldwork sug-gests that neither effect was likely to positively anddifferentially affect the performance of teams usingthe LU method. At 3M, as at most companies, high-profile activities are burdened by high managementattention paid to those activities, resulting in more

    'To explore this matter, we first ran a principal component analysison a battery of 21 product development activities. We then usedthe first principal component (representing twice the variance ofthe second principal component) as a summary measure of criti-cal product development capabilities, and regressed it against LUvs. non-LU training, using background, personality, and skills ascovariates. We found that the LU vs. non-LU training explained thebulk of the variance on this factor, with none of the covariates notedabove statistically significant. While some of these covariates didexplain significant variance in the other principal components, asthose components are of much less importance than the first com-ponent, the impact of those covariates can be regarded as minoroverall.

    MANAGEMENT 48, No. 8, August 2002CIENCE/VO~.

  • 8/9/2019 Lead Users (Von Hippel)

    9/21

    LILIEN, MORRISON, SEARLS, SONNA CK, A ND VON HIPPELPerformance Assessme nt of Lead U ser Data

    frequent reporting and restricted horizons. Also, ourqualitative research showed that at 3M, employeesgenerally view new processes introduced to the com-pany with skepticism rather than with the expectationthat they will perform well.With respect to the intended difference understudy-use of LU methods within projects-all LUteams employed an identical LU process taught tothem with identical coaching materials and withcoaching provided by members of the same smallset of internal 3M coaches. In accordance with tradi-tional 3M practice, idea generation leading to fundednon-LU projects was an internal process using infor-mal methods and carried out by marketing and labpersonnel. Non-LU teams used data sources for ideageneration that varied from project to project andincluded market data collected by outside organiza-tions, data from focus groups with major customersand from customer panels, and information from labpersonnel. Non-LU teams never reported collectinginformation from lead users-they collected marketinformation only from target markets. We refer tothese traditional 3M idea-generation practices as non-LU idea-generation practices.

    In sum, while not satisfying the random assign-ment criterion for experimental design, these samplesappear to satisfy the "rough equivalence" criteria intest and control conditions associated with natural orquasi experimentation.Longitudinal Sample. Preliminary analysis of the

    outcomes of the LU process showed that LU teamswere generally developing ideas for major new prod-uct lines, suggesting that it was important to gen-erate an additional sample of non-LU comparisonprojects. As major new product lines generally emergerarely, we chose this second sample to consist of allmajor new product lines introduced to the market-place between 1950-2000 by the 3M divisions that hadexecuted one or more LU studies. (1950 was as farback as we could go and still find company employ-ees who could provide some data about the innova-tion histories of these major product lines.) Followingthe advice of 3M controllers (and in line with Stalket al. 1996), we operationalized the concept of majorproduct lines as those that were separately reported

    MANAGEMENT 48, N o . 8, August 2002CIENCE/VOI.

    upon in divisional financial statements. In 1999 in the3M divisions we studied, sales of individual majorproduct lines ranged from 7% to 73% of total divi-sional sales. Major product lines at 3M tend to be longlived: We were able to identify only two in our sam-pled divisions that had been discontinued after beinginitiated in the period 1950-2000. Each major prod-uct line showed similar patterns of strong growth andgood profit margins.

    Our sample of major product lines developed vianon-LU methods contains 21 cases that met our crite-ria and that had been developed and funded by thedivisions executing LU studies during the 1950-2000period. We will compare data on these non-LU majorproduct lines with data related to the five fundedideas developed by LU methods that producedoutcomes meeting the criteria for major productlines.

    The data for our longitudinal sample of major prod-uct lines is neither as complete nor likely to be asaccurate as were the data collected for our cross-sectional sample. Specifically, we have no data onthe characteristics of the personnel involved in majorproduct line developments prior to 1997, nor do wehave data on incentives applied to or any specificsregarding idea-generation methods used by these per-sonnel. The data collected for our longitudinal sam-ples depends upon the recall of a small numberof informants (usually one or two) for each case.While these informants had been actual participantsin the projects they were describing, the accuracy ofthe reports is questionable (and might even involveupwardly biased recall of the degree of past inno-vation successes), as, in many cases, the innovationsthey described were made decades earlier. An addi-tional concern is that conditions within 3M and alsowithin the general U.S. and world economies havevaried significantly over the 50-year analysis timeframe.

    In sum, the quasi-experimental field setting for thisresearch involves several potential threats to valid-ity and generalizability of findings. While we believethat these threats are small in the case of our cross-sectional sample, they are more significant in the caseof our longitudinal sample.

  • 8/9/2019 Lead Users (Von Hippel)

    10/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPELPerformance Assessment of Lead User Data

    4.2. Data Collection InstrumentsWe developed two data collection instruments for useduring our study, as follows:

    Outcomes Survey. We developed a New Prod-uct Idea Description Form from measures used inprevious academic research: novelty of ideas, original-itylnewness of customer needs, and potential for prolifer-ating into an entire product line from the New ProductCreativity scale (Moorman and Miner 1997); companysales/market sharefvom the new idea in year 5, and prob-ability of business success from NPD Success Criteria(Griffin and Page 1996); and global market potentialfor all competitors from Cooper (1993). We supple-mented these measures with items revealed duringthe exploratory phase of the research, including sev-eral in general use within 3M, as part of their internalnew product idea-assessment procedure. We carefullypretested all items and refined them during prelimi-nary instrument development meetings with partici-pating staff at 3M. (The Appendix lists the data itemscollected for each idea.)

    Process Survey. We developed a process instru-ment to measure individual skills in idea-generationactivities, personality traits, and individual char-acteristics such as job level, time in the job, andR&D/marketing/accounting/manufacturing exper-tise. We developed items for the individual skillsidentified in the literature to be associated withgenerating new product ideas (Thomas 1993) andsupplemented those items with skills identified frominterviews with LU team leaders and their primaryLU trainer. We measured personality traits usinga Web-based version of Myers-Brigg's PersonalityInventory (www.persona1itypage.com).4.3. Data Collection M ethodsWe began our quantitative data collection with face-to-face meetings with each LU project team leader.In these meetings we asked each project leader to:(a) identify and recruit the members of their teamas respondents; (b) identify and recruit appropri-ate respondents within the same division to pro-vide non-LU idea method data; and (c) describe thenew product development goals and outputs fromthe LU project, provide organizational information

    about their division, and provide process informa-tion. We then contacted the respondents identifiedin (a) and (b) above. The respondents from non-LUidea-generation methods provided detailed informa-tion both on funded ideas as well as on process data.Respondents returned all (100%) of the Idea Descrip-tion Forms for LU projects and 79% for non-LUprojects, and 94% of the process instruments for LUprojects and 86% for non-LU projects. These responserates are sufficiently high to suggest that validitychecks for nonresponse bias are not needed (Malhotra1996).

    We collected data on major new product ideasdeveloped early in the 1950-2000 period throughin-depth interviews conducted with long-tenureemployees in each of the LU study divisions whohad a good knowledge of their division's and productline histories. We converted all historical dollar fig-ures to 1999 dollar equivalents using U.S. consumerprice index data (Council of Economic Advisors 2000).

    5. Analysis and ResultsWe performed two sets of analysis: a cross-sectionalanalysis of the census of all contemporaneous fundedideas (that is, currently funded ideas in the datacollection period February 1999-May 2000), and anintertemporal analysis of major product lines, forwhich we will compare the LU results with a 50-yearhistory of major product line development at 3M.

    In several of our analyses, we compare forecastdata (projected sales for a line of products not yetlaunched) with actual, historical sales. To develop aconservative basis for these comparisons, we exploredboth the general literature and 3M historical data.In the general literature, Armstrong's (2001) reviewon forecast bias for new product introduction indi-cates that forecasts for new product sales are gener-ally optimistic, but that upward bias decreases as themagnitude of the sales forecast increases. Coller andYoln (1998) review the literature on bias in accuracyof management earnings forecasts and find that littlesystematic bias occurs. Tull's (1967) model calculates$15 million in revenue as a level above which fore-casts actually become pessimistic on average.

    We collected data from five 3M division controllersresponsible for authorizing investment expenditure

    MANAGEMENT 48, No. 8, August 2002CIENCE/VO~.050

  • 8/9/2019 Lead Users (Von Hippel)

    11/21

    LILIEN, MORRISON, SEARLS, SONNACK, A N D V O N HIPPELPerformance Assessment of Lead User Data

    for developing new ideas into products and alsoobtained data from a 1995 internal study that com-pared such forecasts with actual sales. We combinedthis information to develop a distribution of forecasterrors for a number of 3M divisions, as well as over-all forecast errors across the full corporation. Thoseerrors range from forecast/actual of +3O0/0 (overfore-cast) to -13% (underforecast). Based on the infor-mation just described, and in consultation with 3Mmanagement, we chose a 25% sales-forecast deflatorto apply to all projected sales data in the analysisthat follows. That deflator is consistent with 3M's his-torical experience and, given Tull's (1967) findings,should provide conservative sales projection figure^.^5.1. Cross-Sectional Findings: Comparison ofContemporaneously Funded IdeasGenerated by LU and Non-LU MethodsWe compared all "funded ideas" generated by LU andnon-LU methods during the time period of our datacollection (February 1999 to May 2000). Table 1 pro-vides a census of all funded ideas during the notedperiod in the five divisions that funded LU ideas.During that time, five ideas generated by LU projectswere being funded along with 42 ideas generated bynon-LU idea-generation methods.

    From Table 1, we see that LU ideas are signifi-cantly more novel than are ideas generated by non-LU methods, they address more original/newer cus-tomer needs, have significantly higher market share,have greater potential to develop into an entire prod-uct line, and are more strategically important. We alsofind that the LU ideas have projected annual sales inYear 5 that are eight times higher than those of ideasgenerated by non-LU methods-an average of $146million versus an average of $18 million in forecastannual sales. Thus, we find support for Hypothesis 1:LU methods do appear to generate ideas with greater

    We find no reason to apply a different deflator to LU vs. non-LU project sales projections. Even if LU project personnel were forsome reason more likely to be optimistic with respect to such pro-jections than non-LU project personnel, that would not significantlyimpact our findings. In addition, over 60% of the total dollar valueof sales forecasts made for LU projects were actually made by per-sonnel not associated with those ~r oi ec ts outside consulting firmsA , -or business analysts from other divisions).MANAGEMENT 48, No. 8, August 2002CIENCE/VO~.

    Table 1 LU vs. Non-LU Funded Ideas (Census)LU ideas Non-LU ideas(n= 5)2 (n = 42)3 Sig.

    Factors related tovalue of idea

    Novelty comparedwith competition1

    Originalitylnewness ofcustomer needs addressed1

    % market share in Year 5Estimated sales in Year 5

    (deflated for forecast error)Potential for entire product family1Operating profi tProbability of successStrategic importance1Intellectual property protection1 7.1 6.7 0.80Factors related to

    organizational fit of ideaFit with existing distribution 8,8 8.0 0.61Fit with existing manufacturingcaoabilitiesl 7.8 6.7 0.92it with existing strategic plani 9.8 8.4 0.24

    'These items were measured using a 10-point rating scale, where 10 =hiah, = low,

    2Funded LU ideas: all are for major product lines.3Funded non-LU ideas: one is for a major product line, 41 are incremental

    ideas.commercial potential than do non-LU methods in thissample (p < 0.005).

    We also found that LU ideas differed in kind fromthose ideas produced by non-LU methods. Non-LUmethods produced mainly funded ideas for prod-uct improvements and extensions to existing productlines, while the LU method produced funded ideasthat fit 3M divisional criteria for major product lines.Those ideas produced projected sales that fell within(and sometimes exceeded) the proportion of divi-sional sales accounted for by existing individual divi-sional major product lines: projected sales five yearsafter introduction for funded LU ideas, conservativelydeflated as discussed above, ranged from 25% to over300% of current total divisional sales. Table 2 showsthe qualitative differencein the type of product (kcre-mental vs. major product line/breakthrough) that theLU process has been generating. Using a chi-square

    1051

  • 8/9/2019 Lead Users (Von Hippel)

    12/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPELPerforman ce Asses sme nt of Lead U ser Data

    Table 2 Idea Types Generated by LU and Non-LU MethodsIncremental Major product line

    LU method 0 5Nan-LU method 41 1

    test, the probability of this outcome occurring bychance is p < 0.005, supporting Hypothesis 2.

    To illustrate what the "major product line" innova-tions that the LU process teams generated at 3M werelike, we briefly describe four:

    (1) A new approach to the prevention of infec-tions associated with surgical operations. The newapproach replaces the traditional "one size fits all"approach to infection prevention with a portfolio ofpatient-specific measures based upon each patient'sindividual biological susceptibilities. This innovationinvolves new product lines plus related business andstrategy innovations made by the team to bring thisnew approach to market successfully and profitably.

    (2) Electronic test and communication equipmentthat, for the first time, enables physically isolatedworkers such as telecommunication equipment repairpeople to carry out their problem-solving work as ateam. Linked workgroup members can contribute tothe solution of a problem being worked upon by asingle, physically isolated worker in real time.

    (3) A new approach (implemented via novel equip-ment) to the application of commercial graphics filmsthat cuts the time of application from 48 hours toless than 1 hour. (Commercial graphics films are used,for example, to cover entire truck trailers, buses, andother vehicles with advertising or decorative graph-ics.) The LU team ideas involve technical innovationsplus related channel and business model changes tohelp diffuse the innovation rapidly.

    (4) A new approach to packaging fragile items inshipping cartons to replace current packaging mate-rials such as foamed "plastic peanuts." The newproduct lines implementing the approach are moreenvironmentally friendly and much faster and moreconvenient for both shippers and package recipientsthan are present products and methods.

    Following Hypothesis 3, our qualitative interview-ees proposed that the apparently more ambitiousoutcomes from the LU process would be less com-patible with key organization-fit criteria than would

    non-LU ideas. The last three items in Table 1 addressthis issue: We find no statistical difference in qualityof fit of LU and non-LU ideas with respect to existingdivisional distribution channels, manufacturing capa-bilities, or divisional strategic plans. Hence, we cannotreject the Null Hypothesis @ that LU and non-LUmajor product lines are equal on these critical dimen-sions of organizational fit (p > 0.10).

    In line with Hypothesis 4, interviewees proposedthat ideas generated by the LU method would be lessprotectable by patents than would ideas generatedby other methods. A single item in Table 3 tests thishypothesis and we cannot reject the Null HypothesisH, that LU and non-LU major product lines are equalon intellectual property protection (p > 0.10).

    Finally, we hypothesized (H,) that LU idea-generation methods would cost more in time andmoney than would idea-generation methods used inour sample of comparison projects. This hypothesis issupported by our data. An audit of idea developmenttime shows that the generation of a funded LU ideaconsumed 154 (sd.= 82) person days on average. Incontrast, generation of an average non-LU idea con-sumed an average of 60 (sd.= 43) person days, reject-ing the hypothesis of equality of these two means atthe p < 0.05 level. Using internal 3M data for aver-age costs of professional development personnel, weobtained a total cost of approximately $100,000 perfunded LU project (including additional costs associ-ated with the LU training) versus $30,000 for non-LUprojects. After accounting for the different probabil-ity of success for LU projects (Table 1: 80/o for LUprojects vs. 66% for the census of non-LU projects),we determined that 3M pays about $80,000 more, onaverage, for a successful LU project than it does for asuccessful non-LU project.5.2. Longitudinal Findings: Major Product

    Line Ideas Generated by LUand Non-LU MethodsWe now compare the major product line ideas gener-ated by the LU method with those generated earlier-during the period 1950-2000-by the 3M divisions inour study using non-LU methods. The 3M divisionswe studied produced 21 major product lines duringthe 1950-2000 period. During the 1997-2000 period,

    MANAGEMENT 48, N o . 8, August 2002C I E N C E / V O ~ .052

  • 8/9/2019 Lead Users (Von Hippel)

    13/21

    LILIEN, MORRISON, SEARLS, SONNACK, A ND VO N HIPPELPerformance Assessment of Lead User Data

    they produced five of those major product lines usingLU methods and two using non-LU methods. Exam-ples of major product lines generated by non-LUmethods in our 1950-2000 sample include:

    (1) Scotch tape: A line of transparent mendingtapes that was a major success in many householdand commercial applications.

    (2 ) Disposable patient drapes for operating roomuse: A pioneering line of disposable products for themedical field now sold in many variations.

    (3) Box-sealing tapes: The first type of tape strongenough to reliably seal corrugated shipping boxes,it replaced stapling in most "corrugated shipper"applications.

    (4) Commercial graphics films: Plastic films capa-ble of withstanding outdoor environments that couldbe printed upon and adhered to vehicles. This prod-uct line changed the entire approach to outdoorsignage.

    Our sample of ideas for new major product linesprovides additional tests of HI through H,. However,that sample does not permit an alternative means totest Hypothesis 5, as we have no data on the costsof idea generation for major product lines developedprior to 1997. We begin with alternative tests for HI,the hypothesis that LU methods will generate ideaswith greater commercial potential than will non-LUmethods. Here we consider an alternative to HI, callit HI,, that focuses on major new product lines only.HI, proposes that ideas for major product lines devel-oped by LU methods will have greater commercialpotential than those generated by non-LU methods.TO proceed, we make the following two assumptions:

    ASSUMPTION lines. Ret urn s from all major productthat have emerged at 3M using methods other than the LUmethod can be considered as draws from the same proba-bility distribution.

    ASSUMPTION. Revenue projections for LU majorproduct lines are biased to the same degree on average ashistorical projections have been for other major productlines.

    Assumption us 'Ompare the major prod-uct lines from the LU m&od with all major productlines in these divisions during the prior four decades.Assumption 2 allows us to use historical 3M figures

    MANAGEMENT 48, No. 8, August 2002CIENCE/VO~.

    on major product line forecast error to adjust the fore-casts from the LU data. Following Assumptions 1and 2 for major product lines introduced to mar-ket in 1994 or earlier we used as a reference "actualsales five years after introduction (including loss/gainfrom sales of related products in the division)." 3Mmanagement maintains such records and the five-yearsales goal is part of 3M's project justification process.To provide comparable data, we translated all salesdata into 1999 dollars. Following Assumption 2 forthe major product line ideas generated from the LUprocess (and for two major product line ideas recentlygenerated by non-LU processes), we have forecastsof five-year sales vs. actual figures for the historicalmajor product lines.

    After deflation of the forecasts as discussed earlier,we find that the average "sales in Year 5" for LUmajor product lines ( n= 5), is $146m, while the similarfigure for major product lines generated by non-LUmethods ( n= 16, as we were able to obtain detaileddata on only 16 of the 21 non-LU major product lines)is $62m. (Table 3).

    We next test whether the mean LU major productline yields higher sales than those derived from non-LU sources. At the p = 0.05 level we reject the hypoth-esis that these values are equivalent (Table 4), provid-ing support for HI,, conditional on Assumptions 1and 2. Table 4 also provides profiles of the 5 LU majorproduct lines and the 16 non-LU major product linesfor which we were able to collect data on the mat-ters listed in that table. We find that the ideas forTable 3 What Is a M ajor (New) Product Line (MNP L) Worth?

    LU MNPL five-year sales Non-LU MNPL five-year salesforecasts* forecasts and actual*

    1997-2000 (n= 5) 1950-2000 (n= 16)Mean $146m $62mMedian $124m $38mRange $67.5m-$232.5111 $1 1.7m-$276mNote. Five-year sales forecasts for all major product lines commercializedin 1994 or later (five LU and two non-LU major product lines) have beendeflated by 25% in line with 3M histo rical forecast error experience (see text) .Five-year sales figures for major product lines commercialized before 1994are actual historical sales data. This data has been converted to 1999 dollarsusing the Consumer Price Index from the Economic Report of the President(Council of Ec~nomic dvisors 2000).

  • 8/9/2019 Lead Users (Von Hippel)

    14/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPEL Pevformance Assessment of Lead U ser Data

    Table 4 Dimensions of Di f ference Between LU Major (New) ProductLines (MNPL) and Non-LU M ajor Product Lines

    Lead UserMNPLs Non-LU MNPLs( n= 5) (n= 16) Sig.

    Novelty compared with competition-Originalitylnewness of customer

    needs addressed*OO market share in Year 5Estimated sales i n Year 5

    (deflated for forecast error)Potential for entire pro duct family*Operating profitProbability of successStrategic importance *Intellectual property protection*Fit with distrib ution channels'Fit with manufacturing capabilities* 7.8 6.7 0.53Fit with Strategic Plan* 9.8 8.7 0.32Note. These items noted with an asterisk were measured using a 10-pointrating scale, where 10 = high, 1 = low.

    major new product lines developed by LU and non-LU methods are relatively similar on most dimensionsexamined. However, as a comparison with the data inTable 1 shows, both are very different from our sam-ple of currently funded ideas that were generated bynon-LU methods.To test the hypothesis that the LU method producesmajor new product line ideas at a higher rate thando non-LU methods (H,) we look at the entire four-year period during which the LU process was imple-mented at 3M and compare it (on a rolling basis) withall other four-year periods during 1950-2000. Theaverage rate of major product line development dur-ing this half-century for the divisions studied was 1.64per four-year period. As Figure 1 shows, the highestrate achieved over a four-year period was seven majorproduct lines, and this result was achieved during the1997-2000 period during which the LU process wasimplemented by those divisions. During this period,five of the seven major product lines developed weregenerated by LU methods. Non-LU methods gener-ated five or more major product lines in two of the47, four-year rolling periods.

    We next look at three comparisons: (1)LU outputvs. pre-1997 output; (2) LU output vs. non-LU output,

    Figure 1 Majo r Product Lines Generation Rate in 3 M Div is ions UnderStudy on a Four-Year Roll ing Basis (1950-2000)

    Frequency of OccurrenceNote. Development of seven major (new) product lines (MNPLs) within afour-year period has occu rred only once in the last half century-during theperiod corresponding to the use of the LU method.

    1997-2000; and (3) pre-1997 output vs. all post-1997output in the divisions studied. For Comparison (I),assume that 3M generates major product lines at a(constant) rate that represents the null hypothesis ofno difference in rate of major product line genera-tion over time. Using the four-year data in Figure l ,that rate is 1.64 major product lines per four-yearperiod for non-LU methods. Then, a formal test ofour Hypothesis H, is: "What is the likelihood thata process that generates h = 1.64 events on averageper study period could generate five (or more) in aperiod?" (1997-2000 for LU). Using an assumption ofa Poisson generation process (Grassman 1981), we cal-culate that P(5 or more Ih = 1.64)= 0.025, providingsupport for H, (rejecting the null hypothesis at the0.05 level).

    For Comparison (2), if we compare the 1997-2000period (five major product lines for LU methods vs.two major product lines for non-LU methods), we getP(5 or more lA = 4 x 0.5= 2.0)= 0.052, again providingsupport for H,. For Comparison (3), if we considerthe LU method an organizational intervention (Mills1999), we can compare the post-1997 period for 3M(blending both LU and non-LU ideas) with the resultsof the previous regime. These results give seven majorproduct lines (five LUs and two non-LUs) in the four-year 1997-2000 period, versus the historical average(adjusted to account for four fewer intervals and twofewer major product lines) of 1.79, giving P(7 or more

    MANAGEMENT 48, No . 8, August 2002CIENCE/VOI.054

  • 8/9/2019 Lead Users (Von Hippel)

    15/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPELPerfo rmance Assessment of Lead Llser Data

    Ih = 1.79)= 0.003. Thus, all three approaches providesupport for H,: We find that the rate of major newproduct line generation using the LU process is signif-icantly higher than the rate historically seen at 3M inthe 1950-2000 period using non-LU idea-generationprocesses.

    Finally, we can perform additional tests ofHypotheses 3 and 4 by referring back to Table 4. Weinvestigated earlier how major product line ideas gen-erated by the LU method differed from those gen-erated by non-LU methods. We reject the hypothe-sis (H,) regarding a significant difference between LUand non-LU major product lines on dimensions oforganizational "fit" (see the last three rows of Table 4).Also, we reject the hypothesis (H,) that ideas gen-erated by LU methods will be significantly less pro-tectable by patent or other means of intellectual prop-erty protection than will ideas generated by non-LUmethods.

    6. DiscussionOur study of LU idea-generation projects at 3M hasyielded several interesting empirical findings. Fromour cross-sectional sample, we have seen that ideasgenerated by LU processes had forecast sales in Year 5that were more than eight times higher than the salesof the contemporaneously funded projects: $146 mil-lion annual sales on average versus $18 million. Wealso found that funded projects emerging from 3MLU studies had significantly higher novelty (usuallybeing judged "new to the world"), addressed moreoriginal newer customer needs, and also had signif-icantly higher forecasted market share in Year 5 (onaverage, 68% vs. 33% for non-LU ideas) than didthose from more conventional methods.

    With respect to findings from our longitudinal sam-ple, we found both sets of ideas to be relatively sim-ilar in most respects, but forecast sales from majornew product ideas generated by LU methods weresignificantly higher than sales forecasts for productsgenerated by non-LU methods. We also found thatthe rate of introduction of products forecast to growinto a major new product line was significantly higher

    MANAGEMENT 48, No. 8, August 2002CIENCE/VOI.

    after use of the LU method began at 3M in 1997.The key finding here is not the rate itself: Clearly, therate of major new product line ideas resulting fromLU projects will go up or down depending upon thenumber of LU projects funded by management, butrather that ideas for major new product lines seemto be consistently produced by LU idea-generationprojects-at least as 3M practices this method. Thisoutcome differs from the outcomes of earlier studiesof LU projects that reported generating ideas for valu-able next-generation products rather than for break-throughs. We speculate that this difference in out-comes is due to the 3M practice of identifying andlearning from lead users outside and well in advanceof the target market, a speculation that requires fur-ther investigation.In the case of both the cross-sectional and longitu-dinal samples, we found that funded "breakthrough"ideas generated via the LU process offered as gooda fit to existing divisional goals and competencies asdid ideas generated by traditional methods. Insightsfrom our grounded research leads us to suggest thatthis is because both LU and non-LU team develop-ers at 3M know that ideas with "good fits" have agreater chance of acceptance and funding. They there-fore work to select and shape the ideas they proposeto achieve that fit.We also found that LU ideas involved as high alevel of intellectual property protection as did theideas generated by non-LU methods. On the face ofit, this finding seems puzzling: How can 3M expectto protect ideas that lead users have developed andprobably revealed elsewhere? Our fieldwork suggestsan answer. LU studies at 3M did not find completeideas for breakthrough products from single leadusers. Instead, LU project teams assessed ideas froma number of lead users and used Phase 4 of the LUprocess to combine those ideas. For example, a 3MLU study designed to identify better ways to preventinfections associated with surgery created a break-through product concept by combining insights from,among others, an expert in wound healing, a lead-ing veterinary surgeon, and a specialist in theatricalmakeup with special expertise in adhering materi-als to skin. These compound ideas were then novel

    1055

  • 8/9/2019 Lead Users (Von Hippel)

    16/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPELPerformance Asses smen t of Lead User Data

    intellectual property that could be protected by 3M.(In 3M's LU process, lead users retain rights to ideasgenerated prior to Phase 4 LU workshops, but assignrights to ideas generated at LU workshops to 3M.)

    Finally, note that LU projects can and do fail.Because we restricted membership in both our cross-sectional and longitudinal samples to funded ideas,we did not include any LU or non-LU ideas thatfailed to cross this initial evaluation hurdle. We doknow, however, that only five of the seven LU projectsinitiated during our period of study at 3M gener-ated ideas that were considered worthy of fundingby management. We have no data about the num-ber of non-LU idea-generation projects failing to crossthis hurdle, but our fieldwork suggests that the non-LU failure rate was at least as high as the LU failurerate.

    Relying most heavily upon our cross-sectionalresults, we suggest that the question that moti-vated this study is answered in the affirmative:The LU idea-generation method does appear togenerate better results than traditional methods. Ittherefore appears to merit further investigation andde~elopment .~

    3M top management seems to share this assessment as two quotescollected during our fieldwork suggest:We were using traditional methods of marketingresearch in our division, and were able to achieveour corporate growth targets. However, we found thattraditional techniques are not able to identify newlyemerging breakthroughs in a rapidly-moving fieldlike Telecoms. Now the LU process has been made thecenterpiece of our new idea generation activities forproduct breakthroughs, and we have an abundanceof radical new ideas. The challenge now is findingresources for all of them. -Roger Lacey, VP of 3MTelecoms Division.This is probably the best process I've seen for replicat-ing what originally made this company great. Whatmade 3M was our people going out and creating solu-tions with leading-edge customers. I think that, for aperiod of time, we lost a lot of that. It's very hard tocreate a process that will do it. But this [the LU Pro-cess] is the closest that I've se en.. . I'm glad that it'sbeing adopted across the company.-Bill Coyne, 3MSenior VP of R&D.

    7. Suggestions for Future ResearchThe LU paradigm opens up a number of researchopportunities, including: (1) further empirical studyof the process in other organizations, (2) new methoddevelopment regarding how to identi& users hold-ing leading-edge information of commercial value;(3) new methods to obtain information from lead usersand build that information into commercially viablenew product and service offerings, and (4) designingand studying the organizational metrics and struc-tures that lead to successful implementation of newprocesses like the LU method.

    With respect to Opportunity 1, we suggest that itwould be useful to conduct empirical research tracingkey elements of new ideas generated to specific inputsand individuals, including lead users both inside andoutside the target market. Such research could deter-mine whether there is a causal relationship betweenseeking information from lead users in advanced ana-log markets and the likelihood of identifying ideasfor new products and services that will be "break-throughs" for the target market.

    With respect to Opportunities 2 and 3, considerthat in the traditional idea-generation paradigm, ideageneration involves first identifying and quantifyingthe intensity of needs shared by many users andthen having internal manufacturer personnel lookfor a novel product that users will find respon-sive to those general needs. The LU idea-generationparadigm assumes that key elements of the desiredcreative idea for a breakthrough already exist amongleading-edge users, with the problem being to findit and develop its potential. In other words, the keychallenge in the traditional paradigm is idea genera-tion, and in the LU paradigm it is idea search.

    The LU process analyzed here utilizes a network-ing procedure to identify and learn from a few care-fully selected lead users both within and outside thetarget market. Other possibilities exist for mining LUinformation that also deserve exploration, drawingon emerging evidence that LU innovation is not rare.Empirical studies of user innovation in four verydifferent areas show a significant fraction of users,ranging from 10% to 3610, reporting that they have

    1056 SCIENCE/VO~.ANAGEMENT 48, No. 8, August 2002

  • 8/9/2019 Lead Users (Von Hippel)

    17/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPELPerformance Assessment o Lead User Data

    developed or modified product^.^ In light of this evi-dence, methods to process information from manylead users simultaneously could be developed andexplored.

    Concerning Opportunity 4, note that even giventhe apparent successes at 3M , diffusion and furtherimplementation is a challenge within the firm. Recallthat Olson and Bakke (2001) reported on the lackof extensive implementation of the LU method at

    9.8% of users of outdoor consumer products reported innovat-ing for own use (Luthje 2000); 26% of users of library informationsystems reported innovating for own use (Morrison et al. 2000);24.3% of users of printed circuit CAD software reported innovatingfor own use (Urban and von Hippel 1988); 36% of users of pipehanger hardware reported innovating for own use (Herstatt andvon Hippel 1992).

    Appendix: Idea Description FormWe are interested in information about:

    This new product idea has the followin g characteristics:Novelty: Relative to competition our productlservice will beOriginalitylnewness of customer needs addressed

    Cinet in spite of demonstrated success, and suggestthat it is "necessary to pressure or reward personnelin order to make permanent changes in establishedroutines" (p. 380). Resistance to innovation withincorporations is a well-known phenomenon. Studies ofhow to address that resistance and make successfulnew corporate practices "stick" are clearly important.

    In sum, we hope that these results will stimulateother researchers to explore and develop what wesee as a promising paradigm for the idea-generationphase of new product and service development.

    AcknowledgmentsThe authors gratefully acknowledge comments and suggestions byJohn Roberts and Christophe Van den Bulte, and appreciate thesupport of Penn State's Institute for the Study of Business Marketsand MIT's Innovation Lab.

    10-point scale, 1 = low, 10= high10-point scale, 1 =low, 10 =hig h

    Global market potential (all competitors) in $ millions $$

    $$

    M % Our company's global market share in Year 5 OEstimated company sales in Year 5: $ millions MProfit potential (estimated % profit-operating income before taxes-5 years after introduct ion) YOProbability of business success (% chance of overall success) YOEstimated LOSS on sales of related products/services for this business unit in 5 years (loss due to new

    productlservice replacing existing productlservice sales) Estimated GAIN on sales of related products/services for this business unit in 5 years (gain due to increased

    MMsales of related existing products/services for this business unit)

    Fit with this business unit's current sales and distribution channels 10-poin t scale, 1 = ow, 10= highFit with this business unit's current manufacturing capabilities 10-point scale, 1 = ow, 10 =highFit with current strategic plan of our business unit 10-point scale, 1 = low, 10= highStrategic importance to our business unit, regardless of fi t with current plans 10-poin t scale, 1 = low, 10= highImportance of projected intellectual property protection obtained 10-point scale, 1 = ow, 10= highPotential to proliferate int o an entire product line 10-point scale, 1 = ow, 10 =hig h

    MANAGEMENT 48, N o . 8, August 2002CIENCE/VO~ .

  • 8/9/2019 Lead Users (Von Hippel)

    18/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPELPerformance Asse ssme nt of Lead U ser Data

    ReferencesAdamson, R. E. 1952. Functional fixedness as related to problem

    solving: A repetition of three experiments. 1.Experiment. Psych.44(4) 288-291.-, -, D. W. Taylor. 1954. Functional fixedness as related toelapsed time and to set. 1.Experiment. Psych. 47(2) 122-126.

    Allen, T. J., D. G. Marquis. 1964. Positive and negative biasing sets:The effects of prior experience on research performance. lEEETrans. Engrg. Managem ent EM-ll(4) 158-161.

    Armstrong, J. S., ed. 2001. Principles of Forecasting: A Handbook forResearchers and Practitioners. Kluwer, Norwell, MA.

    Birch, H. G., H. J. Rabinowitz. 1951. The negative effect of previousexperience on productive thinking. J. Experiment. Psych. 41(2)121-125.

    Coller, M., T. L. Yohn. 1998. Management forecasts: What do weknow? Financial Analysts J. 54(1) 58-62.

    Cook, T. D., D. T. Campbell. 1979. Quasi-Experimentation: Design andAnalysis Issues. Houghton-Mifflin, Boston, MA.

    Cooper, R. G. 1993.Win ning at Ne w Products: Accelerating the Processfrom Idea to Launch. Addison Wesley, Reading, MA.Council of Economic Advisors. 2000. Economic Report of the Presi-

    dent. (www.access.gpo.gov/usbudget/fy2001/erp.html).Douglas, M. 1986.How Institutions Think. Syracuse University Press,

    Syracuse, NY.Dunker, K. 1945.On problem-solving. Lynne S. Lees, trans Psych.

    Monographs 58(5) (Whole No. 270).Eliashberg, J., G. L. Lilien, V. Rao. 1997. Minimizing technologi-

    cal oversights: A marketing research perspective. R. Garud,P. Nayyar, Praveen Rattah, Z. B. Shapira, eds. Technologi-cal Innovation: Oversights and Foresights. Cambridge UniversityPress, New York, 214-230.

    Enos, J. L. 1962. Petroleum Progress and Profits: A History of ProcessInnovation. MIT Press, Cambridge, MA.Finke, R. A,, T. B. Ward, S. M. Smith. 1992. Creative Cognition. MIT

    Press, Cambridge, MA.Freeman, C. 1968. Chemical process plant: Innovation and the

    world market. National Inst. Econom. R m 45(August) 29-57.Glaser, B. G., A. L. Strauss. 1967. The Discovery of Grounded

    Theory: Strategies for Quali tativ e Research. Aldine de Gruyter,Hawthorne, NY.

    Goldenberg, J., D. R. Lehrnann, D. Mazursky. 2001. The idea itselfand the circumstances of its emergence as predictors of newproduct success. Management Sci. 47(1) 69-84.

    , D. Mazursky, S. Solomon. 1999. Toward identifying theinventive templates of new products: A channeled ideationapproach. 1.Marketing Res. 34(2) 200-210.

    Grassman, W. K. 1981. Stochastic Systems for Management. NorthHolland, New York.

    Griffin, A. 1997. Drivers of NPD Success: The 1997 PDMA Report.Product Development & Management Association, Chicago,IL.

    -, A. L. Page. 1996. PDMA success measurement project: Rec-ommended measures for product development success andfailure. 1.Product lnnovation M anagement 13 478496.

    Haman, G. 1996. Techniques and tools to generate breakthroughproducts. M. D. Rosenau Jr., ed. P D M A Handbook of New Prod-uct Development. John Wiley and Sons, New York, 167-178.

    Hargadon, A. 1996. Brainstorming groups in context: Effectivenessin a product design firm. with R. I. Sutton. Admin. Sci. Quart.41 685-718.

    Herstatt, C., E. von Hippel. 1992. From experience: Developing newproduct concepts via the lead user method: A case study in a"low t ec h field.J. Product lnnovation Managem ent. 9(3) 213-221.

    Lave, J. 1988.Cognition in Practice: Min d, Mathematics and C ulture inEveryday Life. Cambridge University Press, Cambridge, MA.

    Lonsdale, R. T., N. M. Noel, S. F. Stasch. 1996. Classification ofsources of new product ideas. M. D. Rosenau Jr., ed. P D M AHandbook of New Product Development. John Wiley and Sons,New York, 179-194.

    Luthje, C. 2000. Characteristics of innovating users in a consumergoods field: An empirical study of sport-related product con-sumers. Working paper, MIT Sloan School of Management,Cambridge, MA.

    Malhotra, N. 1996. Marketing Research: A n Applied Orientation, 2nded. Prentice Hall, Englewood Cliffs,NJ.

    McQuarrie,E. F. 1998.Customer Visits: Building a Better Market Focus.Sage, Newbury Park, CA.

    Mills, T. C. 1999. Th e Econometric Modeling of Financial Time Series.Cambridge University Press, Cambridge, MA.

    Moorman, C., A. S. Miner. 1997. The impact of organizational mem-ory on new product performance and creativity. 1.MarketingRes. 34(February) 91-106.

    Morrison, P. D. , J. H. Roberts, E. von Hippel. 2000. Determinantsof user innovation and innovation sharing in a local market.Management Sci. 46(12) 1513-1527.

    Olson, E. L., G. Bakke. 2001. Implementing the lead user methodin a high technology firm: A longitudinal study of intentionsversus actions. J. Product lnnovation M anagement. 18(2)388-395.

    Rangaswamy, A,, G. L. Lilien. 1997. Software tools for new productdevelopment. J. Marketing Res. 34(1) 177-184.

    Rossiter, J. R., G. L. Lilien. 1994. New "brainstorming" principles.Austral. 1.Management. 19(1) 61-72.

    Shah, S. 1999. Sources and patterns of innovation in a consumerproducts field: Innovations in sporting equipment. WorkingPaper #410 (May), MIT Sloan School of Management, Cam-bridge, MA.

    Shaw, B. 1985. The role of the interaction between the user and themanufacturer in medical equipment innovation. R&D Manage-ment. 15(4) 283-292.

    Stalk Jr., G., D. K. Pecaut, B. Burnett. 1996. Breaking compromises,breakaway growth. Harvard Bus. Rev. (September-October.)

    Thomas, R. J. 1993. New Product Development: Managing and Fore-casting for N ew Product Suc cess. Wiley, New York.

    Tull, D. 1967. The relationship of actual and predicted sales andprofits in new product introductions. 1.Bus. 40 233-250.

    1058 S C I E N C E / V O ~ .ANAGEMENT 48, No. 8, August 2002

  • 8/9/2019 Lead Users (Von Hippel)

    19/21

    LILIEN, MORRISON, SEARLS, SONNACK, AND VON HIPPELPerformance Assessment of Lead User Data

    Urban, G. L., E. von Hippel. 1988. Lead user analyses for the -, S. Thomke, M. Sonnack. 1999. Creating breakthroughs at 3M.development of new industrial products. Management Sci. 34(5) Harvard Business Rmiew 77 (September-October) 47-57.569-582. Wenger, E. 1998. Comm unities of Practice: Learning, Meaning and Iden-

    Van Maanen, J. 1988. Tales of the Field: On Wr itin g Ethnography. The tity. Cambridge University Press, New York.University of Chicago Press, Chicago, IL. Willis, P. 1981. Learning to Labor. Morningside Edition, Columbia

    von Hippel, E. 1986. Lead users: A source of novel product con- University Press, New York.cepts. Management Sci. 32(7) 791-805. Workman, J. P. Jr. 1993. Marketing's limited role in new product

    -. 1988. The Sources of Innovation. Oxford University Press, development in one computer systems firm. 1. Marketing Res.New York. 33(4) 405-421.

    Accepted by Ralph Katz; received September 2001. This paper was wit h the authors 3 monthsfor 2 revisions.

    MANAGEMENT 48, No . 8, August 2002CIENCE/VO~.

  • 8/9/2019 Lead Users (Von Hippel)

    20/21

    You have printed the following article:

    Performance Assessment of the Lead User Idea-Generation Process for New ProductDevelopment

    Gary L. Lilien; Pamela D. Morrison; Kathleen Searls; Mary Sonnack; Eric von Hippel

    Management Science, Vol. 48, No. 8. (Aug., 2002), pp. 1042-1059.

    Stable URL:

    http://links.jstor.org/sici?sici=0025-1909%28200208%2948%3A8%3C1042%3APAOTLU%3E2.0.CO%3B2-6

    This article references the following linked citations. If you are trying to access articles from anoff-campus location, you may be required to first logon via your library web site to access JSTOR. Pleasevisit your library's website or contact a librarian to learn about options for remote access to JSTOR.

    References

    Toward Identifying the Inventive Templates of New Products: A Channeled IdeationApproach

    Jacob Goldenberg; David Mazursky; Sorin Solomon

    Journal of Marketing Research, Vol. 36, No. 2. (May, 1999), pp. 200-210.

    Stable URL:

    http://links.jstor.org/sici?sici=0022-2437%28199905%2936%3A2%3C200%3ATITITO%3E2.0.CO%3B2-0

    The Impact of Organizational Memory on New Product Performance and Creativity

    Christine Moorman; Anne S. Miner

    Journal of Marketing Research, Vol. 34, No. 1, Special Issue on Innovation and New Products.(Feb., 1997), pp. 91-106.

    Stable URL:

    http://links.jstor.org/sici?sici=0022-2437%28199702%2934%3A1%3C91%3ATIOOMO%3E2.0.CO%3B2-L

    Software Tools for New Product Development

    Arvind Rangaswamy; Gary L. Lilien

    Journal of Marketing Research, Vol. 34, No. 1, Special Issue on Innovation and New Products.(Feb., 1997), pp. 177-184.

    Stable URL:

    http://links.jstor.org/sici?sici=0022-2437%28199702%2934%3A1%3C177%3ASTFNPD%3E2.0.CO%3B2-W

    http://www.jstor.org

    LINKED CITATIONS- Page 1 of 2 -

    http://links.jstor.org/sici?sici=0025-1909%28200208%2948%3A8%3C1042%3APAOTLU%3E2.0.CO%3B2-6&origin=JSTOR-pdfhttp://links.jstor.org/sici?sici=0022-2437%28199905%2936%3A2%3C200%3ATITITO%3E2.0.CO%3B2-0&origin=JSTOR-pdfhttp://links.jstor.org/sici?sici=0022-2437%28199702%2934%3A1%3C91%3ATIOOMO%3E2.0.CO%3B2-L&origin=JSTOR-pdfhttp://links.jstor.org/sici?sici=0022-2437%28199702%2934%3A1%3C177%3ASTFNPD%3E2.0.CO%3B2-W&origin=JSTOR-pdfhttp://links.jstor.org/sici?sici=0022-2437%28199702%2934%3A1%3C177%3ASTFNPD%3E2.0.CO%3B2-W&origin=JSTOR-pdfhttp://links.jstor.org/sici?sici=0022-2437%28199702%2934%3A1%3C91%3ATIOOMO%3E2.0.CO%3B2-L&origin=JSTOR-pdfhttp://links.jstor.org/sici?sici=0022-2437%28199905%2936%3A2%3C200%3ATITITO%3E2.0.CO%3B2-0&origin=JSTOR-pdfhttp://links.jstor.org/sici?sici=0025-1909%28200208%2948%3A8%3C1042%3APAOTLU%3E2.0.CO%3B2-6&origin=JSTOR-pdf
  • 8/9/2019 Lead Users (Von Hippel)

    21/21

    The Relationship of Actual and Predicted Sales and Profits in New-Product Introductions

    Donald S. Tull

    The Journal of Business, Vol. 40, No. 3. (Jul., 1967), pp. 233-250.

    Stable URL:

    http://links.jstor.org/sici?sici=0021-9398%28196707%2940%3A3%3C233%3ATROAAP%3E2.0.CO%3B2-C

    Marketing's Limited Role in New Product Development in One Computer Systems Firm

    John P. Workman, Jr.Journal of Marketing Research, Vol. 30, No. 4. (Nov., 1993), pp. 405-421.

    Stable URL:

    http://links.jstor.org/sici?sici=0022-2437%28199311%2930%3A4%3C405%3AMLRINP%3E2.0.CO%3B2-T

    http://www.jstor.org

    LINKED CITATIONS- Page 2 of 2 -

    http://links.jstor.org/sici?sici=0021-9398%28196707%2940%3A3%3C233%3ATROAAP%3E2.0.CO%3B2-C&origin=JSTOR-pdfhttp://links.jstor.org/sici?sici=0022-2437%28199311%2930%3A4%3C405%3AMLRINP%3E2.0.CO%3B2-T&origin=JSTOR-pdfhttp://links.jstor.org/sici?sici=0022-2437%28199311%2930%3A4%3C405%3AMLRINP%3E2.0.CO%3B2-T&origin=JSTOR-pdfhttp://links.jstor.org/sici?sici=0021-9398%28196707%2940%3A3%3C233%3ATROAAP%3E2.0.CO%3B2-C&origin=JSTOR-pdf