Copernicus Institute - NUSAP net...• Both promote decision strategies that are not fit for meeting the challenges posed by the uncertainties and complexities faced. • This delays

1

Universiteit Utrecht

Copernicus Institute

HEIMTSA, INTARESE, 2-FUN, and NoMiracle Joint Winter School on UncertaintyLoch Tay 20 - 22 January 2010

The NUSAP method for uncertainty assessment

Centre d'Economie et d'Ethique pour l'Environnementet le Développement, Université de Versailles Saint-Quentin-en-Yvelines, France

Dr. Jeroen van der Sluijs [email protected]

Copernicus Institute for Sustainable Development and InnovationUtrecht University

&



Download this PPT

www.nusap.net/lochtay

2



Daily practice of dealing with uncertain science in policy making

Two dominant strategies: uncertainties are either• downplayed to promote political decisions (enforced

consensus), or • overemphasised to prevent political action

• Both promote decision strategies that are not fit for meeting the challenges posed by the uncertainties and complexities faced.

• This delays a transition to sustainability.

• We need new ways to deal with uncertainty, scientific dissent & plurality in sustainability science.



Goals• Be able to apply the NUSAP approach on own case studies• Understand how NUSAP complements quantitative uncertainty

assessment • Be able to recognize situations where it is appropriate to apply

NUSAP• Have a good appreciation of the resources required to apply

NUSAP and the practicalities of doing so• Have a firm grasp on the process of applying NUSAP

– Initial screening of uncertainties and assumptions– Choice whether or not to perform expert or stakeholder elicitation– Selection of experts– Choice of pedigree criteria– Plotting a diagnostic diagram– Reporting and communicating results– Interpreting results and using them in decision making

3



Complex - uncertain - risksTypical characteristics (Funtowicz & Ravetz):• Decisions will need to be made before conclusive

scientific evidence is available;• Potential impacts of ‘wrong’ decisions can be huge• Values are in dispute • Knowledge base is characterized by large (partly

irreducible, largely unquantifiable) uncertainties, multi-causality, knowledge gaps, and imperfect understanding;

• More research less uncertainty; unforeseen complexities!• Assessment dominated by models, scenarios,

assumptions, extrapolations• Many (hidden) value loadings reside in problem frames,

indicators chosen, assumptions made

Knowledge Quality Assessment is essential



A practical problem:

Protecting a strategic fresh-water resource

5 scientific consultants addressed same question:

“which parts of this area are most vulnerable to nitrate pollution and need to be protected?”

(Refsgaard, Van der Sluijs et al, 2006)

4



3 framings of uncertainty'deficit view'• Uncertainty is provisional• Reduce uncertainty, make ever more complex models• Tools: quantification, Monte Carlo, Bayesian belief networks

'evidence evaluation view'• Comparative evaluations of research results• Tools: Scientific consensus building; multi disciplinary expert panels• focus on robust findings

'complex systems view / post-normal view'• Uncertainty is intrinsic to complex systems• Uncertainty can be result of production of knowledge• Acknowledge that not all uncertainties can be quantified• Openly deal with deeper dimensions of uncertainty

(problem framing indeterminacy, ignorance, assumptions, value loadings, institutional dimensions)

• Tools: Knowledge Quality Assessment• Deliberative negotiated management of risk



How to act upon such uncertainty?• Bayesian approach: 5 priors. Average and

update likelihood of each grid-cell being red with data (but oooops, there is no data and we need decisions now)

• IPCC approach: Lock the 5 consultants up in a room and don’t release them before they have consensus

• Nihilist approach: Dump the science and decide on an other basis

• Precautionary robustness approach: protect all grid-cells

• Academic bureaucrat approach: Weigh by citation index (or H-index) of consultant.

• Select the consultant that you trust most• Real life approach: Select the consultant that

best fits your policy agenda• Post normal: explore the relevance of our

ignorance: working deliberatively within imperfections

5



The certainty trough(McKenzie, 1990)



Funtowicz and Ravetz, Science for the Post Normal age, Futures, 1993

6



The alternative model: PNSExtended participation: working deliberatively within imperfections• Science is only one part of relevant evidence• Critical dialogue on strength and relevance of

evidence• Interpretation of evidence and attribution of

policy meaning to knowledge is democratized• Tools for Knowledge Quality Assessment

empower all stakeholders to engage in this deliberative process

(Funtowicz, 2006; Funtowicz & Strand, 2007)



Elements of Post Normal Science

• Appropriate management of uncertainty quality and value-ladenness

• Plurality of commitments and perspectives

• Internal extension of peer community (involvement of other disciplines)

• External extension of peer community (involvement of stakeholders in environmental assessment & quality control)

7



Insights on uncertainty• More research tends to increase uncertainty

– reveals unforeseen complexities– Complex systems exhibit irreducible uncertainty (intrinsic or

practically)• Omitting uncertainty management can lead to scandals,

crisis and loss of trust in science and institutions• In many complex problems unquantifiable uncertainties

dominate the quantifiable uncertainty• High quality low uncertainty• Quality relates to fitness for function (robustness, PP)• Shift in focus needed from reducing uncertainty towards

reflective methods to explicitly cope with uncertainty and quality



NL Environmental Assessment Agency (RIVM/MNP) Guidance: Systematic reflection on uncertainty & quality in:

8



Uncertainty analysis = Mapping assumptionsonto inferencesSensitivity analysis = The reverse process

(slide borrowed from Andrea Saltelli)



Do we know enough to quantify?Risbey & Kandlikar (2007): What format is in accordance

with the level of knowledge on the quantity?• Full probability density function

– Robust, well defended distribution

• Bounds– Well defended percentile bounds

• First order estimates– Order of magnitude assessment

• Expected sign or trend– Well defended trend expectation

• Ambiguous sign or trend– Equally plausible contrary trend expectations

• Effective ignorance– Lacking or weakly plausible expectations

9



Reliability intervals normal distributions

= 68 % 2 = 95 % 3 = 99.7 %



Total NH3 emission in 1995 as reported in successive SotE reports

0

50

100

150

200

250

1996 1997 1998 1999 2000 2001 2002

Year of State of Environment Report

mlj

kg a

mm

on

iak

95%confidence-interval

10



Dimensions of uncertainty

• Technical (inexactness)• Methodological (unreliability)• Epistemological (ignorance)• Societal (limited social

robustness)



Value-ladenness

• Value-ladenness refers to the notion that value orientations and biases of an analyst, an institute, a discipline or a culture can co-shape the way scientific questions are framed, data are selected, interpreted, and rejected, methodologies are devised, explanations are formulated and conclusions are formulated.

• Since theories are always underdetermined by observation, the analysts' biases will fill the epistemic gap which makes any assessment to a certain degree value-laden.

11



Uncertainty in Relative Risk of child leukemia as a function of magnetic field strength of overhead power lines

Technical: 95% - interval

Methodological:- Indirect exposure metric- small sample size

> 0.4 T, n=106 - selection bias

Epistemological- Causality unknown

Societal- Distrust in outcomes that

point at low risks measure



NUSAPQualified Quantities

• Numeral • Unit• Spread • Assessment • Pedigree

(Funtowicz and Ravetz, 1990)

12



NUSAP: Pedigree

Evaluates the strength of the number by looking at:

• Background history by which the number was produced

• Underpinning and scientific status of the number



Code Proxy Empirical Theoretical basis Method Validation

4 Exactmeasure

Large sampledirect mmts

Well establishedtheory

Best availablepractice

Compared withindep. mmts ofsame variable

3 Good fit ormeasure

Small sampledirect mmts

Accepted theorypartial in nature

Reliable methodcommonlyaccepted

Compared withindep. mmts ofclosely relatedvariable

2 Wellcorrelated

Modeled/deriveddata

Partial theorylimitedconsensus onreliability

Acceptablemethod limitedconsensus onreliability

Compared withmmts notindependent

1 Weakcorrelation

Educated guesses/ rule of thumbest

Preliminarytheory

Preliminarymethodsunknownreliability

Weak / indirectvalidation

0 Not clearlyrelated

Crudespeculation

Crudespeculation

No discerniblerigour

No validation

Example Pedigree matrix parameter strength

13



Example: Air Quality



Knol et al, submitted;Knol et al 2008

14



Selection of experts

• Types of experts– generalists, – subject-matter experts– normative experts

• Balancetwo-step nomination procedure. – List authors of at least two peer-reviewed

papers on the subject (literature search) – Ask them to nominate experts having the

expertise necessary for the elicitation. – Invite those experts who got most nominations



Should one involve stakeholders in the elicitation?

Roles:• Co-framer of problem• Co-producer of knowledge• Co-producer of quality

Examples:• Critical review of assumptions in HIA: YES• Assess concentration response function: NO

15



How many experts?“There is no absolute guideline on which to base the

number of experts to be invited. According to a panel of expert elicitation practitioners [Cooke & Probst2006], as a rule of thumb the number of experts for most studies they conducted was targeted to lie between 6 and 12, although constraints of a given study may lead to different numbers of experts. Generally, however, at least six experts should be included; otherwise there may be questions about the robustness of the results. The feeling of the practitioners was that beyond 12 experts, the benefit of including additional experts begins to drop off.”

Cooke, R. M. and Probst, KN. Highlights of the Expert Judgment Policy Symposium and Technical Workshop. Conference Summary. 2006.



Case 1

VOC emissions from paint in the Netherlands

16





How is VOC from paint monitored?

VOC emission calculated from:• VVVF national sales statistics NL-paint

in NL per sector• CBS paint import statistics• Estimates of paint-related thinner use• Assumption of VOC% imported paint• Attribution imported paint over sectors

17



General steps in analysis

• Identification and classification• Disaggregation• Identification of expertise• Assessment of assumptions• Qualitative assessment• Quantitative assessment• Uncaptured assumptions• Calculation of Monte Carlo and Pedigree

results• Communication



Code Proxy Empirical Method Validation

4 Exact measure Large sampledirect mmts



3 Good fit formeasure




2 Well correlated Modeled/deriveddata



1 Weak correlation Educated guesses/ rule of thumb est

Preliminarymethods unknownreliability



Crude speculation No discerniblerigour

No validation

18



Expert Elicitation

Goal• To systematically make explicit

and utilizable unwritten knowledge in the heads of experts, including insight in the limitations, strengths and weaknesses of that knowledge



Pitfalls in expert elicitation

• Overconfidence• Representativeness• Anchoring• Bounded rationality• Availability / lamp posting• Implicit assumptions• Motivational bias

– Possibility of strategic answers– Interests with regard to outcome of analysis

19



Expert Elicitationproces

OverconfidenceRepresentativeAvailability

IPCCNUSAP



Sources of error

• Definitional inconsistency• Interpretation of definitions• Boundaries between raw materials, products,

assortment• Miscategorization• Misreporting via unit confusion• Deliberate misreporting• Miscoding• Non-response

20



Sources of error (continued)

• Not counting small firms (reporting threshold CBS)

• Not counting non-VVVF members• Firm dynamics• Paint dynamics• Computer code errors



Conceivable sources of motivational bias

• Cross-check with CBS data by tax authorities• Anonymity • Membership due VVVF depends on sales

figures• Paint sales statistics is market-sensitive

information; possibility of strategic reporting to mislead competitors

• KWS2000 - reputation of paint industry

21



Added during the elicitation

• Import below reporting threshold• Gap between NS VVVF-members

and total NS• Overlap CBS import figure / VVVF

NS figure

• Attribution of Thinner % DIY & Decoration to paint



Frequency Chart

kton

.000

.006

.012

.019

.025

0

481.2

962.5

1925

41.5 46.6 51.6 56.7 61.8

77,540 Trials 2,664 Outliers

Forecast: Total VOC Paint

22





23



Proxy Empirical Method Validation StrengthNS-SHI 3 3.5 4 0 0.66NS-B&S 3 3.5 4 0 0.66NS-DIY 2.5 3.5 4 3 0.81NS-CAR 3 3.5 4 3 0.84NS-IND 3 3.5 4 0.5 0.69Th%-SHI 2 1 2 0 0.31Th%-B&S 2 1 2 0 0.31Th%-DIY 1 1 2 0 0.25Th%-CAR 2 1 2 0 0.31Th%-IND 2 1 2 0 0.31VOS % import 1 2 1.5 0 0.28Attribution import 1 1 2 0 0.25

Pedigree scores

Trafic-light analogy <1.4 red; 1.4-2.6 amber; >2.6 green



Criticality

Pedigreeweakstrong

low

high

NUSAP Diagnostic Diagram

Dangerzone

Safezone

24



0.001

0.01

0.1

1

00.20.40.60.81

Strenght

Rel

ativ

e co

ntr

ibu

tio

n t

o v

aria

nce

VOC% imp.paint

Thin % Ind

Overlap VVVF/CBS imp

NUSAP Diagnostic Diagram

Imp. Below threshold

Thin.% DIY-rest

Gap VVVF-RNSThin.% Car

NS Decor

NS IndImp. Paint

NS DIY

NS Car

NS Ship

Th.% decor



Conclusions• NUSAP provides a strong diagnostic tool

which yields a richer insight in sources and nature of uncertainty than Monte Carlo analysis alone

• Helpful in priority setting for uncertainty management and quality improvement

• Time intensive – need for two-step approach:

quick-scan / thorough analysis

25



Case 2Chains of models

EO5 Environmental Indicators



RIVM Environmental Outlook

• Scenario study issued every 4 years

• hundreds of environmental indicators

• basis for NL Environmental Policy Plan

• Strongly based on chains of model calculations

26



The problem of assumptions: example of ozone health risks



Calculation chain deaths and hospital admittances

due to ozone

1 Societal/demographical developments2 VOC and NOx emissions in the

Netherlands and abroad3 Ozone concentrations4 Potential exposure to ozone5 Number of deaths/hospital

admittances due to exposure

27



Pedigree criteria for reviewing assumptions

• Influence of situational restrictions (time, money, etc.)

• Plausibility• Choice space• Agreement amongst peers• Agreement amongst stakeholders• Sensitivity to view and interests of analyst• Estimated influence on results



Kloprogge et al. pedigree matrix voor het karakteriseren van aannames

28



Workshop reviewing assumptions

• Completion of list of key assumptions

• Rank assumptions according to importance

• Elicit pedigree scores

• Evaluate method



Key assumptions deaths and hospital admittances

due to ozone• Uncertainty mainly determined by uncertainty

in Relative Risk (RR)• No differences in emissions abroad between

the two scenarios• Ozone concentration homogeneously

distributed in 50 x 50 km grid cells• Worst case meteo now = worst case future• RR constant over time (while air pollution

mixture may change!)• Linear dose-effect relationship

29



Example EMF caseAssumption 19 "Magnetic field is the causal agent"



Case 2

The IMAGE/TIMER B1 scenario

30



IMAGE 2: Framework of models and Linkages

World Economy(WorldScan)

Population(Popher)

Change in GDP, population & others(i.e. scenario assumptions)

Land-useemissions

Energy demand & supply (TIMER)

Energy & industry emissions

Land demand, use & cover

Emissions & land-use changes

Carboncycle

Atmosphericchemistry

Concentration changes

Climate(Zonal Climate Model or ECBilt)

Climatic changes

Naturalsystems

AgriculturalImpacts

Water Impacts

Land degradation

Sea levelrise

Feedbacks

Impacts



TIMER Model : five submodelsEnergy

Demand (ED)

Liquid Fuel

supply (LF)

Gaseous Fuelsupply (GF)

Electric PowerGeneration (EPG)

Solid Fuelsupply (SF)

Population(POPHER)

Inputs: Population, GDP capita-1, activity in energy sectors, assumptions regarding technologicaldevelopment, depletion and others.

Outputs: End-use energy consumption, primary energyconsumption.

Fuel demand

Prices

Economy(WorldScan)

Electricitydemand

31



Main objectives TIMER

• To analyse the long-term dynamics of the energy system, and in particular changes in energy demand and the transition to non-fossil fuels within an integrated modeling framework;

• To construct and simulate greenhouse gas emission scenarios that are used in other submodels of IMAGE 2.2 or that are used in meta-models of IMAGE;



Global CO2 emission from fossil fuels

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

40.00

1990 2000 2010 2020 2030 2040 2050 2060 2070 2080 2090 2100

Year

CO

2 E

mis

sio

n (

in G

tC)

Mar

iaM

essa

geA

imM

inic

amA

SFIm

age

(Van Vuuren et al. 2000)

(SRES scenarios reported to IPCC (2000) by six different modelling groups)

B1-marker

32



Key questions

What are key uncertainties in TIMER? What is the role of model structure

uncertainties in TIMER? Uncertainty in which input variables and

parameters dominate uncertainty in model outcome?

What is the strength of the sensitive parameters (pedigree)?



Method Checklist for model quality assistance Meta-level analysis SRES scenarios to

explore model structure uncertainties Global sensitivity analysis (Morris) NUSAP expert elicitation workshop to

assess pedigree of sensitive model components

Diagnostic diagram to prioritiseuncertainties by combination of criticality (Morris) and strength (pedigree)

33



Applying NUSAP to a complex model was quite

a challengeTIMER model:• 300 variables• 19 world regions• 5 economic sectors• 5 types of energy carriers• 2 forms of energy• some are time series

about 160,000 numbers



Checklist

• Assist in quality control in complex models

• Not models are good or bad but ‘better’and ‘worse’ forms of modelling practice

• Quality relates to fitness for function• Help guard against poor practice• Flag pittfalls

34



Checklist structure

• Screening questions• Model & problem domain• Internal strength• Interface with users• Use in policy process• Overall assessment



Morris (1991)• facilitates global sensitivity analysis in

minimum number of model runs• covers entire range of possible

values for each variable• parameters varied one step at a time in

such a way that if sensitivity of one parameter is contingent on the values that other parameters may take, Morris captures such dependencies

35



Most sensitive model components:

• Population levels and economic activity• Intra-sectoral structural change• Progress ratios for technological

improvements• Size and cost supply curves of fossil

fuels resources• Autonomous and price-induced energy

efficiency improvement• Initial costs and depletion of renewables



Parameter Pedigree

• Proxy• Empirical basis• Theoretical understanding• Methodological rigour• Validation

36



Code Proxy Empirical Theoretical basis Method Validation

4 Exactmeasure

Large sampledirect mmts

Well establishedtheory



3 Good fit ormeasure


Accepted theorypartial in nature



2 Wellcorrelated

Modeled/deriveddata

Partial theorylimitedconsensus onreliability



1 Weakcorrelation

Educated guesses/ rule of thumbest

Preliminarytheory

Preliminarymethodsunknownreliability



Crudespeculation

Crudespeculation

No discerniblerigour

No validation



Elicitation workshop

• Focussed on 40 key uncertain parameters grouped in 18 clusters

• 18 experts (in 3 parallel groups of 6) discussed parameters, one by one, using information & scoring cards

• Individual expert judgements, informed by group discussion

37





Instructions

• Do the Pedigree assessment as an individual expert judgement, we do not want a group judgement

• Main function of group discussion is clarification of concepts

• Group works on one card at a time• If you feel you cannot judge the

pedigree scores for a given parameter, leave it blank

38





Example result gas depletion multiplier

Same data represented as kite diagram:Green = min. scores, Amber= max scores, Light green = min. scores if outliers omitted(Traffic light analogy)

Radar diagram:Each coloured line represents scores given by one expert

39



Average scores (0-4)• proxy 2½ ±½• empirical 2 ±½• theory 2 ±½• method 2 ±½• validation 1 ±½

• valueladeness 2½ ±1• competence 2 ±½



40



Conclusions (1)

Global sensitivity analysis supplemented with expert elicitation constitutes an efficient selection mechanism to further focus the diagnosis of key uncertainties.

Our pedigree elicitation procedure yields a differentiated insight into parameter strength.



Conclusions (2)

The diagnostic diagram puts spread and strength together to provide guidance in prioritisation of key uncertainties.

41



Conclusions (3)

NUSAP method:• can be applied to complex models

in a meaningful way• helps to focus research efforts on

the potentially most problematic model components

• pinpoints specific weaknesses in these components



The Pedigree Exploring Tool (PET)

Serafín Corral Quintana, 2000, 2002

42



In summary, NUSAP• Has a strong theoretical foundation in the theory of knowledge and

the philosophy of science• Addresses all three dimensions of uncertainty: technical

(inexactness), methodological (unreliability) and epistemological (border with ignorance) in an coherent way

• Provides a systematic framework for synthesising qualitative andquantitative assessments of uncertainty

• Can act as a bridge between the quantitative mathematical disciplines and traditions and the qualitative discursive and participatory disciplines and traditions in the field of uncertainty management.

• Helps to focus research efforts on the potentially most problematic model components

• Pinpoints specific weaknesses in these components• Provides those who produce, use and are affected by policy-relevant

knowledge a tool for a critical self-awareness of their engagement with that knowledge. It thereby fosters extended peer review processes.



Science for Policy book (2009)

• Edited by Dr. Angela Guimaraes Pereira and Dr. SilvioFuntowicz

• The essays deal with policy issues relating to environment governance in a situation where relevant scientific knowledge is uncertain and contextual. Recent debates in the context include climate change, genetically modified foods, and sustainable development. When stakes are high and decisions are urgent, new approaches are required for knowledge production and policy making.

Divided into five parts, the volume features: I. methodological perspectivesII. GMO policies covering bio-safety lawsIII. climate change policyIV. energy policy: nuclear waste, corporate energy strategiesV. sustainable development: assessment tools, S&T Policymaking, and water governance

• Readership : Teachers and students of development economics, environmental studies, and sociology and policymakers seeking links between sustainability and development.

http://www.oupcanada.com/catalog/9780195698497.html

43



ReferencesFuntowicz SO, Ravetz JR. Uncertainty and quality in science for policy. Dordrecht: Kluwer; 1990. 229 pp.Funtowicz SO, Ravetz JR. Science for the post-normal age. Futures 1993;25(7):735–55.Funtowicz S, Strand R. Models of science and policy. in: Traavik, T. and Lim, L.C. (eds.) Biosafety First: Holistic

Approaches to Risk and Uncertainty in Genetic Engineering and Genetically Modified Organisms (forthcoming) Tapir Academic Press, Trondheim

Funtowicz SO. 2006. Why knowledge assessment? In: Interfaces between Science and Society (GuimarãesPereira Â, Guedes Vaz S, Tognetti, S, eds). Sheffield: Greenleaf Publishers, 138–145.

Guimarães Pereira Â, Guedes Vaz S, Tognetti, S, (eds), 2006. Interfaces between Science and Society. Sheffield: Greenleaf Publishers

Janssen PHM, Petersen AC, van der Sluijs JP, Risbey JS, Ravetz JR. 2005. A guidance for assessing and communicating uncertainties. Water Sci Technol 52:125–131.

P. Kloprogge, J.P. van der Sluijs and A.C. Petersen, A method for the analysis of assumptions in model-based environmental assessments, Environmental Modelling & Software, published online 19 July 2009.

A.B. Knol, A.C. Petersen, J.P. van der Sluijs, E. Lebret, Dealing with uncertainties: The case of environmental burden of disease assessment, Environmental Health 8:21

Ravetz, J. 2006, The no-nonsense guide to science, New Internationalist.Saltelli A, Chan K, Scott M. 2000, Sensitivity analysis. Probability and statistics series. John Wiley & Sons

Publishers; 475 pp.Saltelli A, Tarantola S, Campolongo F, Ratto M. 2004, Sensitivity analysis in practice: a guide to assessing

scientific models. JohnWiley & Sons Publishers; 219 pp.van der Sluijs, JP, 2002, A way out of the credibility crisis around model-use in Integrated Environmental

Assessment, Futures, 34 133-146.van der Sluijs JP. 2005. Uncertainty as a monster in the science policy interface: four coping strategies. Water Sci

Technol 52:87–92.van der Sluijs JP, Craye M, Funtowicz SO, Kloprogge P, Ravetz JR, Risbey JS. 2005. Combining quantitative and

qualitative measures of uncertainty in model based environmental assessment: the NUSAP system. Risk Anal. 25:481–492.

van der Sluijs, J.P. 2006, Uncertainty, assumptions, and value commitments in the knowledge-base of complex environmental problems, in: Ângela Guimarães Pereira, Sofia Guedes Vaz and Sylvia Tognetti, Interfaces between Science and Society, Green Leaf Publishing, 2006, p.67-84.

van der Sluijs JP, 2007, Uncertainty and Precaution in Environmental Management: Insights from the UPEM conference, Environmental Modelling and Software, 22, (5), 590-598.

J.P. van der Sluijs, A.C. Petersen, P.H.M. Janssen, James S Risbey and Jerome R. Ravetz (2008) Exploring the quality of evidence for complex and contested policy decisions, Environmental Research Letters, 3 024008 (9pp)



Exercise

• Apply the pedigree matrix and the uncertainty matrix to a model used in the Crystal Ball exercise

44



Pedigree matrix for evaluating models



Instructions

• Do the Pedigree assessment as an individual expert judgement, we do not want a group judgement

• Main function of group discussion is clarification of concepts

• Group works on one card at a time• If you feel you cannot judge the

pedigree scores for a given parameter, leave it blank

Documents

Copernicus Institute - NUSAP net...• Both promote decision strategies that are not fit for meeting the challenges posed by the uncertainties and complexities faced. • This delays