View
0
Download
0
Category
Preview:
Citation preview
1
Universiteit Utrecht
Copernicus Institute
HEIMTSA, INTARESE, 2-FUN, and NoMiracle Joint Winter School on UncertaintyLoch Tay 20 - 22 January 2010
The NUSAP method for uncertainty assessment
Centre d'Economie et d'Ethique pour l'Environnementet le Développement, Université de Versailles Saint-Quentin-en-Yvelines, France
Dr. Jeroen van der Sluijs j.p.vandersluijs@uu.nl
Copernicus Institute for Sustainable Development and InnovationUtrecht University
&
Copernicus Institute
Universiteit Utrecht
Download this PPT
www.nusap.net/lochtay
2
Copernicus Institute
Universiteit Utrecht
Daily practice of dealing with uncertain science in policy making
Two dominant strategies: uncertainties are either• downplayed to promote political decisions (enforced
consensus), or • overemphasised to prevent political action
• Both promote decision strategies that are not fit for meeting the challenges posed by the uncertainties and complexities faced.
• This delays a transition to sustainability.
• We need new ways to deal with uncertainty, scientific dissent & plurality in sustainability science.
Copernicus Institute
Universiteit Utrecht
Goals• Be able to apply the NUSAP approach on own case studies• Understand how NUSAP complements quantitative uncertainty
assessment • Be able to recognize situations where it is appropriate to apply
NUSAP• Have a good appreciation of the resources required to apply
NUSAP and the practicalities of doing so• Have a firm grasp on the process of applying NUSAP
– Initial screening of uncertainties and assumptions– Choice whether or not to perform expert or stakeholder elicitation– Selection of experts– Choice of pedigree criteria– Plotting a diagnostic diagram– Reporting and communicating results– Interpreting results and using them in decision making
3
Copernicus Institute
Universiteit Utrecht
Complex - uncertain - risksTypical characteristics (Funtowicz & Ravetz):• Decisions will need to be made before conclusive
scientific evidence is available;• Potential impacts of ‘wrong’ decisions can be huge• Values are in dispute • Knowledge base is characterized by large (partly
irreducible, largely unquantifiable) uncertainties, multi-causality, knowledge gaps, and imperfect understanding;
• More research less uncertainty; unforeseen complexities!• Assessment dominated by models, scenarios,
assumptions, extrapolations• Many (hidden) value loadings reside in problem frames,
indicators chosen, assumptions made
Knowledge Quality Assessment is essential
Copernicus Institute
Universiteit Utrecht
A practical problem:
Protecting a strategic fresh-water resource
5 scientific consultants addressed same question:
“which parts of this area are most vulnerable to nitrate pollution and need to be protected?”
(Refsgaard, Van der Sluijs et al, 2006)
4
Copernicus Institute
Universiteit Utrecht
3 framings of uncertainty'deficit view'• Uncertainty is provisional• Reduce uncertainty, make ever more complex models• Tools: quantification, Monte Carlo, Bayesian belief networks
'evidence evaluation view'• Comparative evaluations of research results• Tools: Scientific consensus building; multi disciplinary expert panels• focus on robust findings
'complex systems view / post-normal view'• Uncertainty is intrinsic to complex systems• Uncertainty can be result of production of knowledge• Acknowledge that not all uncertainties can be quantified• Openly deal with deeper dimensions of uncertainty
(problem framing indeterminacy, ignorance, assumptions, value loadings, institutional dimensions)
• Tools: Knowledge Quality Assessment• Deliberative negotiated management of risk
Copernicus Institute
Universiteit Utrecht
How to act upon such uncertainty?• Bayesian approach: 5 priors. Average and
update likelihood of each grid-cell being red with data (but oooops, there is no data and we need decisions now)
• IPCC approach: Lock the 5 consultants up in a room and don’t release them before they have consensus
• Nihilist approach: Dump the science and decide on an other basis
• Precautionary robustness approach: protect all grid-cells
• Academic bureaucrat approach: Weigh by citation index (or H-index) of consultant.
• Select the consultant that you trust most• Real life approach: Select the consultant that
best fits your policy agenda• Post normal: explore the relevance of our
ignorance: working deliberatively within imperfections
5
Copernicus Institute
Universiteit Utrecht
The certainty trough(McKenzie, 1990)
Copernicus Institute
Universiteit Utrecht
Funtowicz and Ravetz, Science for the Post Normal age, Futures, 1993
6
Copernicus Institute
Universiteit Utrecht
The alternative model: PNSExtended participation: working deliberatively within imperfections• Science is only one part of relevant evidence• Critical dialogue on strength and relevance of
evidence• Interpretation of evidence and attribution of
policy meaning to knowledge is democratized• Tools for Knowledge Quality Assessment
empower all stakeholders to engage in this deliberative process
(Funtowicz, 2006; Funtowicz & Strand, 2007)
Copernicus Institute
Universiteit Utrecht
Elements of Post Normal Science
• Appropriate management of uncertainty quality and value-ladenness
• Plurality of commitments and perspectives
• Internal extension of peer community (involvement of other disciplines)
• External extension of peer community (involvement of stakeholders in environmental assessment & quality control)
7
Copernicus Institute
Universiteit Utrecht
Insights on uncertainty• More research tends to increase uncertainty
– reveals unforeseen complexities– Complex systems exhibit irreducible uncertainty (intrinsic or
practically)• Omitting uncertainty management can lead to scandals,
crisis and loss of trust in science and institutions• In many complex problems unquantifiable uncertainties
dominate the quantifiable uncertainty• High quality low uncertainty• Quality relates to fitness for function (robustness, PP)• Shift in focus needed from reducing uncertainty towards
reflective methods to explicitly cope with uncertainty and quality
Copernicus Institute
Universiteit Utrecht
NL Environmental Assessment Agency (RIVM/MNP) Guidance: Systematic reflection on uncertainty & quality in:
8
Copernicus Institute
Universiteit Utrecht
Uncertainty analysis = Mapping assumptionsonto inferencesSensitivity analysis = The reverse process
(slide borrowed from Andrea Saltelli)
Copernicus Institute
Universiteit Utrecht
Do we know enough to quantify?Risbey & Kandlikar (2007): What format is in accordance
with the level of knowledge on the quantity?• Full probability density function
– Robust, well defended distribution
• Bounds– Well defended percentile bounds
• First order estimates– Order of magnitude assessment
• Expected sign or trend– Well defended trend expectation
• Ambiguous sign or trend– Equally plausible contrary trend expectations
• Effective ignorance– Lacking or weakly plausible expectations
9
Copernicus Institute
Universiteit Utrecht
Reliability intervals normal distributions
= 68 % 2 = 95 % 3 = 99.7 %
Copernicus Institute
Universiteit Utrecht
Total NH3 emission in 1995 as reported in successive SotE reports
0
50
100
150
200
250
1996 1997 1998 1999 2000 2001 2002
Year of State of Environment Report
mlj
kg a
mm
on
iak
95%confidence-interval
10
Copernicus Institute
Universiteit Utrecht
Dimensions of uncertainty
• Technical (inexactness)• Methodological (unreliability)• Epistemological (ignorance)• Societal (limited social
robustness)
Copernicus Institute
Universiteit Utrecht
Value-ladenness
• Value-ladenness refers to the notion that value orientations and biases of an analyst, an institute, a discipline or a culture can co-shape the way scientific questions are framed, data are selected, interpreted, and rejected, methodologies are devised, explanations are formulated and conclusions are formulated.
• Since theories are always underdetermined by observation, the analysts' biases will fill the epistemic gap which makes any assessment to a certain degree value-laden.
11
Copernicus Institute
Universiteit Utrecht
Uncertainty in Relative Risk of child leukemia as a function of magnetic field strength of overhead power lines
Technical: 95% - interval
Methodological:- Indirect exposure metric- small sample size
> 0.4 T, n=106 - selection bias
Epistemological- Causality unknown
Societal- Distrust in outcomes that
point at low risks measure
Copernicus Institute
Universiteit Utrecht
NUSAPQualified Quantities
• Numeral • Unit• Spread • Assessment • Pedigree
(Funtowicz and Ravetz, 1990)
12
Copernicus Institute
Universiteit Utrecht
NUSAP: Pedigree
Evaluates the strength of the number by looking at:
• Background history by which the number was produced
• Underpinning and scientific status of the number
Copernicus Institute
Universiteit Utrecht
Code Proxy Empirical Theoretical basis Method Validation
4 Exactmeasure
Large sampledirect mmts
Well establishedtheory
Best availablepractice
Compared withindep. mmts ofsame variable
3 Good fit ormeasure
Small sampledirect mmts
Accepted theorypartial in nature
Reliable methodcommonlyaccepted
Compared withindep. mmts ofclosely relatedvariable
2 Wellcorrelated
Modeled/deriveddata
Partial theorylimitedconsensus onreliability
Acceptablemethod limitedconsensus onreliability
Compared withmmts notindependent
1 Weakcorrelation
Educated guesses/ rule of thumbest
Preliminarytheory
Preliminarymethodsunknownreliability
Weak / indirectvalidation
0 Not clearlyrelated
Crudespeculation
Crudespeculation
No discerniblerigour
No validation
Example Pedigree matrix parameter strength
13
Copernicus Institute
Universiteit Utrecht
Example: Air Quality
Copernicus Institute
Universiteit Utrecht
Knol et al, submitted;Knol et al 2008
14
Copernicus Institute
Universiteit Utrecht
Selection of experts
• Types of experts– generalists, – subject-matter experts– normative experts
• Balancetwo-step nomination procedure. – List authors of at least two peer-reviewed
papers on the subject (literature search) – Ask them to nominate experts having the
expertise necessary for the elicitation. – Invite those experts who got most nominations
Copernicus Institute
Universiteit Utrecht
Should one involve stakeholders in the elicitation?
Roles:• Co-framer of problem• Co-producer of knowledge• Co-producer of quality
Examples:• Critical review of assumptions in HIA: YES• Assess concentration response function: NO
15
Copernicus Institute
Universiteit Utrecht
How many experts?“There is no absolute guideline on which to base the
number of experts to be invited. According to a panel of expert elicitation practitioners [Cooke & Probst2006], as a rule of thumb the number of experts for most studies they conducted was targeted to lie between 6 and 12, although constraints of a given study may lead to different numbers of experts. Generally, however, at least six experts should be included; otherwise there may be questions about the robustness of the results. The feeling of the practitioners was that beyond 12 experts, the benefit of including additional experts begins to drop off.”
Cooke, R. M. and Probst, KN. Highlights of the Expert Judgment Policy Symposium and Technical Workshop. Conference Summary. 2006.
Universiteit Utrecht
Copernicus Institute
Case 1
VOC emissions from paint in the Netherlands
16
Copernicus Institute
Universiteit Utrecht
Copernicus Institute
Universiteit Utrecht
How is VOC from paint monitored?
VOC emission calculated from:• VVVF national sales statistics NL-paint
in NL per sector• CBS paint import statistics• Estimates of paint-related thinner use• Assumption of VOC% imported paint• Attribution imported paint over sectors
17
Copernicus Institute
Universiteit Utrecht
General steps in analysis
• Identification and classification• Disaggregation• Identification of expertise• Assessment of assumptions• Qualitative assessment• Quantitative assessment• Uncaptured assumptions• Calculation of Monte Carlo and Pedigree
results• Communication
Copernicus Institute
Universiteit Utrecht
Code Proxy Empirical Method Validation
4 Exact measure Large sampledirect mmts
Best availablepractice
Compared withindep. mmts ofsame variable
3 Good fit formeasure
Small sampledirect mmts
Reliable methodcommonlyaccepted
Compared withindep. mmts ofclosely relatedvariable
2 Well correlated Modeled/deriveddata
Acceptablemethod limitedconsensus onreliability
Compared withmmts notindependent
1 Weak correlation Educated guesses/ rule of thumb est
Preliminarymethods unknownreliability
Weak / indirectvalidation
0 Not clearlyrelated
Crude speculation No discerniblerigour
No validation
18
Copernicus Institute
Universiteit Utrecht
Expert Elicitation
Goal• To systematically make explicit
and utilizable unwritten knowledge in the heads of experts, including insight in the limitations, strengths and weaknesses of that knowledge
Copernicus Institute
Universiteit Utrecht
Pitfalls in expert elicitation
• Overconfidence• Representativeness• Anchoring• Bounded rationality• Availability / lamp posting• Implicit assumptions• Motivational bias
– Possibility of strategic answers– Interests with regard to outcome of analysis
19
Copernicus Institute
Universiteit Utrecht
Expert Elicitationproces
OverconfidenceRepresentativeAvailability
IPCCNUSAP
Copernicus Institute
Universiteit Utrecht
Sources of error
• Definitional inconsistency• Interpretation of definitions• Boundaries between raw materials, products,
assortment• Miscategorization• Misreporting via unit confusion• Deliberate misreporting• Miscoding• Non-response
20
Copernicus Institute
Universiteit Utrecht
Sources of error (continued)
• Not counting small firms (reporting threshold CBS)
• Not counting non-VVVF members• Firm dynamics• Paint dynamics• Computer code errors
Copernicus Institute
Universiteit Utrecht
Conceivable sources of motivational bias
• Cross-check with CBS data by tax authorities• Anonymity • Membership due VVVF depends on sales
figures• Paint sales statistics is market-sensitive
information; possibility of strategic reporting to mislead competitors
• KWS2000 - reputation of paint industry
21
Copernicus Institute
Universiteit Utrecht
Added during the elicitation
• Import below reporting threshold• Gap between NS VVVF-members
and total NS• Overlap CBS import figure / VVVF
NS figure
• Attribution of Thinner % DIY & Decoration to paint
Copernicus Institute
Universiteit Utrecht
Frequency Chart
kton
.000
.006
.012
.019
.025
0
481.2
962.5
1925
41.5 46.6 51.6 56.7 61.8
77,540 Trials 2,664 Outliers
Forecast: Total VOC Paint
22
Copernicus Institute
Universiteit Utrecht
Copernicus Institute
Universiteit Utrecht
23
Copernicus Institute
Universiteit Utrecht
Proxy Empirical Method Validation StrengthNS-SHI 3 3.5 4 0 0.66NS-B&S 3 3.5 4 0 0.66NS-DIY 2.5 3.5 4 3 0.81NS-CAR 3 3.5 4 3 0.84NS-IND 3 3.5 4 0.5 0.69Th%-SHI 2 1 2 0 0.31Th%-B&S 2 1 2 0 0.31Th%-DIY 1 1 2 0 0.25Th%-CAR 2 1 2 0 0.31Th%-IND 2 1 2 0 0.31VOS % import 1 2 1.5 0 0.28Attribution import 1 1 2 0 0.25
Pedigree scores
Trafic-light analogy <1.4 red; 1.4-2.6 amber; >2.6 green
Copernicus Institute
Universiteit Utrecht
Criticality
Pedigreeweakstrong
low
high
NUSAP Diagnostic Diagram
Dangerzone
Safezone
24
Copernicus Institute
Universiteit Utrecht
0.001
0.01
0.1
1
00.20.40.60.81
Strenght
Rel
ativ
e co
ntr
ibu
tio
n t
o v
aria
nce
VOC% imp.paint
Thin % Ind
Overlap VVVF/CBS imp
NUSAP Diagnostic Diagram
Imp. Below threshold
Thin.% DIY-rest
Gap VVVF-RNSThin.% Car
NS Decor
NS IndImp. Paint
NS DIY
NS Car
NS Ship
Th.% decor
Copernicus Institute
Universiteit Utrecht
Conclusions• NUSAP provides a strong diagnostic tool
which yields a richer insight in sources and nature of uncertainty than Monte Carlo analysis alone
• Helpful in priority setting for uncertainty management and quality improvement
• Time intensive – need for two-step approach:
quick-scan / thorough analysis
25
Universiteit Utrecht
Copernicus Institute
Case 2Chains of models
EO5 Environmental Indicators
Copernicus Institute
Universiteit Utrecht
RIVM Environmental Outlook
• Scenario study issued every 4 years
• hundreds of environmental indicators
• basis for NL Environmental Policy Plan
• Strongly based on chains of model calculations
26
Copernicus Institute
Universiteit Utrecht
The problem of assumptions: example of ozone health risks
Copernicus Institute
Universiteit Utrecht
Calculation chain deaths and hospital admittances
due to ozone
1 Societal/demographical developments2 VOC and NOx emissions in the
Netherlands and abroad3 Ozone concentrations4 Potential exposure to ozone5 Number of deaths/hospital
admittances due to exposure
27
Copernicus Institute
Universiteit Utrecht
Pedigree criteria for reviewing assumptions
• Influence of situational restrictions (time, money, etc.)
• Plausibility• Choice space• Agreement amongst peers• Agreement amongst stakeholders• Sensitivity to view and interests of analyst• Estimated influence on results
Copernicus Institute
Universiteit Utrecht
Kloprogge et al. pedigree matrix voor het karakteriseren van aannames
28
Copernicus Institute
Universiteit Utrecht
Workshop reviewing assumptions
• Completion of list of key assumptions
• Rank assumptions according to importance
• Elicit pedigree scores
• Evaluate method
Copernicus Institute
Universiteit Utrecht
Key assumptions deaths and hospital admittances
due to ozone• Uncertainty mainly determined by uncertainty
in Relative Risk (RR)• No differences in emissions abroad between
the two scenarios• Ozone concentration homogeneously
distributed in 50 x 50 km grid cells• Worst case meteo now = worst case future• RR constant over time (while air pollution
mixture may change!)• Linear dose-effect relationship
29
Copernicus Institute
Universiteit Utrecht
Example EMF caseAssumption 19 "Magnetic field is the causal agent"
Universiteit Utrecht
Copernicus Institute
Case 2
The IMAGE/TIMER B1 scenario
30
Copernicus Institute
Universiteit Utrecht
IMAGE 2: Framework of models and Linkages
World Economy(WorldScan)
Population(Popher)
Change in GDP, population & others(i.e. scenario assumptions)
Land-useemissions
Energy demand & supply (TIMER)
Energy & industry emissions
Land demand, use & cover
Emissions & land-use changes
Carboncycle
Atmosphericchemistry
Concentration changes
Climate(Zonal Climate Model or ECBilt)
Climatic changes
Naturalsystems
AgriculturalImpacts
Water Impacts
Land degradation
Sea levelrise
Feedbacks
Impacts
Copernicus Institute
Universiteit Utrecht
TIMER Model : five submodelsEnergy
Demand (ED)
Liquid Fuel
supply (LF)
Gaseous Fuelsupply (GF)
Electric PowerGeneration (EPG)
Solid Fuelsupply (SF)
Population(POPHER)
Inputs: Population, GDP capita-1, activity in energy sectors, assumptions regarding technologicaldevelopment, depletion and others.
Outputs: End-use energy consumption, primary energyconsumption.
Fuel demand
Prices
Economy(WorldScan)
Electricitydemand
31
Copernicus Institute
Universiteit Utrecht
Main objectives TIMER
• To analyse the long-term dynamics of the energy system, and in particular changes in energy demand and the transition to non-fossil fuels within an integrated modeling framework;
• To construct and simulate greenhouse gas emission scenarios that are used in other submodels of IMAGE 2.2 or that are used in meta-models of IMAGE;
Copernicus Institute
Universiteit Utrecht
Global CO2 emission from fossil fuels
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
40.00
1990 2000 2010 2020 2030 2040 2050 2060 2070 2080 2090 2100
Year
CO
2 E
mis
sio
n (
in G
tC)
Mar
iaM
essa
geA
imM
inic
amA
SFIm
age
(Van Vuuren et al. 2000)
(SRES scenarios reported to IPCC (2000) by six different modelling groups)
B1-marker
32
Copernicus Institute
Universiteit Utrecht
Key questions
What are key uncertainties in TIMER? What is the role of model structure
uncertainties in TIMER? Uncertainty in which input variables and
parameters dominate uncertainty in model outcome?
What is the strength of the sensitive parameters (pedigree)?
Copernicus Institute
Universiteit Utrecht
Method Checklist for model quality assistance Meta-level analysis SRES scenarios to
explore model structure uncertainties Global sensitivity analysis (Morris) NUSAP expert elicitation workshop to
assess pedigree of sensitive model components
Diagnostic diagram to prioritiseuncertainties by combination of criticality (Morris) and strength (pedigree)
33
Copernicus Institute
Universiteit Utrecht
Applying NUSAP to a complex model was quite
a challengeTIMER model:• 300 variables• 19 world regions• 5 economic sectors• 5 types of energy carriers• 2 forms of energy• some are time series
about 160,000 numbers
Copernicus Institute
Universiteit Utrecht
Checklist
• Assist in quality control in complex models
• Not models are good or bad but ‘better’and ‘worse’ forms of modelling practice
• Quality relates to fitness for function• Help guard against poor practice• Flag pittfalls
34
Copernicus Institute
Universiteit Utrecht
Checklist structure
• Screening questions• Model & problem domain• Internal strength• Interface with users• Use in policy process• Overall assessment
Copernicus Institute
Universiteit Utrecht
Morris (1991)• facilitates global sensitivity analysis in
minimum number of model runs• covers entire range of possible
values for each variable• parameters varied one step at a time in
such a way that if sensitivity of one parameter is contingent on the values that other parameters may take, Morris captures such dependencies
35
Copernicus Institute
Universiteit Utrecht
Most sensitive model components:
• Population levels and economic activity• Intra-sectoral structural change• Progress ratios for technological
improvements• Size and cost supply curves of fossil
fuels resources• Autonomous and price-induced energy
efficiency improvement• Initial costs and depletion of renewables
Copernicus Institute
Universiteit Utrecht
Parameter Pedigree
• Proxy• Empirical basis• Theoretical understanding• Methodological rigour• Validation
36
Copernicus Institute
Universiteit Utrecht
Code Proxy Empirical Theoretical basis Method Validation
4 Exactmeasure
Large sampledirect mmts
Well establishedtheory
Best availablepractice
Compared withindep. mmts ofsame variable
3 Good fit ormeasure
Small sampledirect mmts
Accepted theorypartial in nature
Reliable methodcommonlyaccepted
Compared withindep. mmts ofclosely relatedvariable
2 Wellcorrelated
Modeled/deriveddata
Partial theorylimitedconsensus onreliability
Acceptablemethod limitedconsensus onreliability
Compared withmmts notindependent
1 Weakcorrelation
Educated guesses/ rule of thumbest
Preliminarytheory
Preliminarymethodsunknownreliability
Weak / indirectvalidation
0 Not clearlyrelated
Crudespeculation
Crudespeculation
No discerniblerigour
No validation
Copernicus Institute
Universiteit Utrecht
Elicitation workshop
• Focussed on 40 key uncertain parameters grouped in 18 clusters
• 18 experts (in 3 parallel groups of 6) discussed parameters, one by one, using information & scoring cards
• Individual expert judgements, informed by group discussion
37
Copernicus Institute
Universiteit Utrecht
Copernicus Institute
Universiteit Utrecht
Instructions
• Do the Pedigree assessment as an individual expert judgement, we do not want a group judgement
• Main function of group discussion is clarification of concepts
• Group works on one card at a time• If you feel you cannot judge the
pedigree scores for a given parameter, leave it blank
38
Copernicus Institute
Universiteit Utrecht
Copernicus Institute
Universiteit Utrecht
Example result gas depletion multiplier
Same data represented as kite diagram:Green = min. scores, Amber= max scores, Light green = min. scores if outliers omitted(Traffic light analogy)
Radar diagram:Each coloured line represents scores given by one expert
39
Copernicus Institute
Universiteit Utrecht
Average scores (0-4)• proxy 2½ ±½• empirical 2 ±½• theory 2 ±½• method 2 ±½• validation 1 ±½
• valueladeness 2½ ±1• competence 2 ±½
Copernicus Institute
Universiteit Utrecht
40
Copernicus Institute
Universiteit Utrecht
Conclusions (1)
Global sensitivity analysis supplemented with expert elicitation constitutes an efficient selection mechanism to further focus the diagnosis of key uncertainties.
Our pedigree elicitation procedure yields a differentiated insight into parameter strength.
Copernicus Institute
Universiteit Utrecht
Conclusions (2)
The diagnostic diagram puts spread and strength together to provide guidance in prioritisation of key uncertainties.
41
Copernicus Institute
Universiteit Utrecht
Conclusions (3)
NUSAP method:• can be applied to complex models
in a meaningful way• helps to focus research efforts on
the potentially most problematic model components
• pinpoints specific weaknesses in these components
Copernicus Institute
Universiteit Utrecht
The Pedigree Exploring Tool (PET)
Serafín Corral Quintana, 2000, 2002
42
Copernicus Institute
Universiteit Utrecht
In summary, NUSAP• Has a strong theoretical foundation in the theory of knowledge and
the philosophy of science• Addresses all three dimensions of uncertainty: technical
(inexactness), methodological (unreliability) and epistemological (border with ignorance) in an coherent way
• Provides a systematic framework for synthesising qualitative andquantitative assessments of uncertainty
• Can act as a bridge between the quantitative mathematical disciplines and traditions and the qualitative discursive and participatory disciplines and traditions in the field of uncertainty management.
• Helps to focus research efforts on the potentially most problematic model components
• Pinpoints specific weaknesses in these components• Provides those who produce, use and are affected by policy-relevant
knowledge a tool for a critical self-awareness of their engagement with that knowledge. It thereby fosters extended peer review processes.
Copernicus Institute
Universiteit Utrecht
Science for Policy book (2009)
• Edited by Dr. Angela Guimaraes Pereira and Dr. SilvioFuntowicz
• The essays deal with policy issues relating to environment governance in a situation where relevant scientific knowledge is uncertain and contextual. Recent debates in the context include climate change, genetically modified foods, and sustainable development. When stakes are high and decisions are urgent, new approaches are required for knowledge production and policy making.
Divided into five parts, the volume features: I. methodological perspectivesII. GMO policies covering bio-safety lawsIII. climate change policyIV. energy policy: nuclear waste, corporate energy strategiesV. sustainable development: assessment tools, S&T Policymaking, and water governance
• Readership : Teachers and students of development economics, environmental studies, and sociology and policymakers seeking links between sustainability and development.
http://www.oupcanada.com/catalog/9780195698497.html
43
Copernicus Institute
Universiteit Utrecht
ReferencesFuntowicz SO, Ravetz JR. Uncertainty and quality in science for policy. Dordrecht: Kluwer; 1990. 229 pp.Funtowicz SO, Ravetz JR. Science for the post-normal age. Futures 1993;25(7):735–55.Funtowicz S, Strand R. Models of science and policy. in: Traavik, T. and Lim, L.C. (eds.) Biosafety First: Holistic
Approaches to Risk and Uncertainty in Genetic Engineering and Genetically Modified Organisms (forthcoming) Tapir Academic Press, Trondheim
Funtowicz SO. 2006. Why knowledge assessment? In: Interfaces between Science and Society (GuimarãesPereira Â, Guedes Vaz S, Tognetti, S, eds). Sheffield: Greenleaf Publishers, 138–145.
Guimarães Pereira Â, Guedes Vaz S, Tognetti, S, (eds), 2006. Interfaces between Science and Society. Sheffield: Greenleaf Publishers
Janssen PHM, Petersen AC, van der Sluijs JP, Risbey JS, Ravetz JR. 2005. A guidance for assessing and communicating uncertainties. Water Sci Technol 52:125–131.
P. Kloprogge, J.P. van der Sluijs and A.C. Petersen, A method for the analysis of assumptions in model-based environmental assessments, Environmental Modelling & Software, published online 19 July 2009.
A.B. Knol, A.C. Petersen, J.P. van der Sluijs, E. Lebret, Dealing with uncertainties: The case of environmental burden of disease assessment, Environmental Health 8:21
Ravetz, J. 2006, The no-nonsense guide to science, New Internationalist.Saltelli A, Chan K, Scott M. 2000, Sensitivity analysis. Probability and statistics series. John Wiley & Sons
Publishers; 475 pp.Saltelli A, Tarantola S, Campolongo F, Ratto M. 2004, Sensitivity analysis in practice: a guide to assessing
scientific models. JohnWiley & Sons Publishers; 219 pp.van der Sluijs, JP, 2002, A way out of the credibility crisis around model-use in Integrated Environmental
Assessment, Futures, 34 133-146.van der Sluijs JP. 2005. Uncertainty as a monster in the science policy interface: four coping strategies. Water Sci
Technol 52:87–92.van der Sluijs JP, Craye M, Funtowicz SO, Kloprogge P, Ravetz JR, Risbey JS. 2005. Combining quantitative and
qualitative measures of uncertainty in model based environmental assessment: the NUSAP system. Risk Anal. 25:481–492.
van der Sluijs, J.P. 2006, Uncertainty, assumptions, and value commitments in the knowledge-base of complex environmental problems, in: Ângela Guimarães Pereira, Sofia Guedes Vaz and Sylvia Tognetti, Interfaces between Science and Society, Green Leaf Publishing, 2006, p.67-84.
van der Sluijs JP, 2007, Uncertainty and Precaution in Environmental Management: Insights from the UPEM conference, Environmental Modelling and Software, 22, (5), 590-598.
J.P. van der Sluijs, A.C. Petersen, P.H.M. Janssen, James S Risbey and Jerome R. Ravetz (2008) Exploring the quality of evidence for complex and contested policy decisions, Environmental Research Letters, 3 024008 (9pp)
Copernicus Institute
Universiteit Utrecht
Exercise
• Apply the pedigree matrix and the uncertainty matrix to a model used in the Crystal Ball exercise
44
Copernicus Institute
Universiteit Utrecht
Pedigree matrix for evaluating models
Copernicus Institute
Universiteit Utrecht
Instructions
• Do the Pedigree assessment as an individual expert judgement, we do not want a group judgement
• Main function of group discussion is clarification of concepts
• Group works on one card at a time• If you feel you cannot judge the
pedigree scores for a given parameter, leave it blank
Recommended