Privacy with Secondary Use of Personal Information

PrivacywithSecondaryUseofPersonalInformation

MKWI2016

Sicherheit,ComplianceundVerfügbarkeit vonGeschäftsprozessen

March9,2016,Ilmenau,Germany

Dr.SvenWohlgemuth (VisitingResearcherGoetheUniversität Frankfurt,Germany)

Dr.KazuoTakaragi (NationalInstituteofAdvancedIndustrialScienceandTechnology,Japan)

Prof.Dr.IsaoEchizen (NationalInstituteofInformatics,Japan)

Helper

TheGreatEastJapanEarthquake

03.08.16 Privacywith Secondary Use of PersonalInformation 2Urushidani andAoki2011,JAISA2015

Refugee

NationalacademicICTinfrastructure(SINET)wasavailable

Physical

Cyber

SINET4:Cloud-typeservicesfor>700organizations

Telemedicine

Insufficientinformationinreal-timeforresponseandrecovery

Agenda

I. ResilienceandSafety• Lessonslearned• Safety:AZero-KnowledgeProof?

II. TowardsProvableSafety• Language-BasedInformationFlowControl• LanguageforICTResilience

III. ProofSystemforICTResilience• Zero-KnowledgeProofwithOpenData• CryptographicBuildingBlocks

IV. LookingforPartners!

03.08.16 Privacywith Secondary Use of PersonalInformation 3

I.ResilienceandSafety


Urushidani etal.2015,JAISA2015

ResiliencebypredictiveITriskmanagementwithpersonaldata

HelperRefugee

Physical

Cyber

SINET5:CloudComputingwithPKIandMarketplace

Telemedicine

GroundTruth 5

Courtesy of Tsukuba Univ.

Kostadinka Bizheva, et al., J. of Biomedical Optics, July/ 2004 Vol.9 No.4

Petra Wilder-Smith, et al. J. of Biomedical Optics Sep/ 2005 Vol.10 No.5

BrainEye

Tooth

Oral

Skin

Z.P.Chen, et al.,Opt. Express, Aug/ 2007 Vol. 15 No. 16

Esophagus

Alexander Popp, et al., J. of Biomedical Optics, Jan/ 2004 Vol.11 No.1

Lung

Guillermo J. Tearney, et al. J. of Biomedical Optics Mar/ 2006 Vol.11 No.2

CardiovascularPancreas

Pier Alberto, et al. J Pancreas (Online)

2007 Vol.8 No.2 Cervix

Ilya V. Turchin, et al., J. of Biomedical Optics, Nov/ 2005 Vol.10 No.6

Blood flow

Bradley A. Bower., J. of Biomedical Optics, Jul/ 2007 Vol.12 No.4

Stomach

Yonghong He, et al. J. of Biomedical Optics

Jan/ 2004 Vol.9 No.1

Trachea

Matthew Brenner, et al., J. of Biomedical Optics, Sep/ 2007 Vol.12 No.5

Cochlea

Fangyi Chen, et al., J. of Biomedical Optics, Mar/ 2007 Vol.12 No.2

Bladder

Ying T. Pan, et al. J. of Biomedical Optics

Sep/ 2007 Vol.12 No.5

Colon

Alexandre R. Tumlinson, et al., J. of Biomedical Optics, Nov/ 2006 Vol.11 No.6

Kidney

Yu Chen, et al. J. of Biomedical Optics

Sep/ 2007 Vol.12 No.3

Bone

santec confidential SS-OCT System Inner Vision 16Application to Biometrics:Non-invasive measurement of iris, retina, fingerprint, vascular image under skin.

OCT(Optical Coherence Tomography)

図：santec株式会社提供資料より

RequirementsonSafety

03.08.16 PrivacywithSecondaryUseofPersonalInformation 5

Compliance• End-to-endsecurity

• Declassification

• Accountabilityandpenalty

• Adequateriskmanagementwithauthenticreporting

PersonalRiskManagement

• Transaction-specificsafety

• Just-in-timescalableknowledgecreationfromdata

• Optimizinguser’sriskwithdataminimization

User-centricsafety(Completeness)

Integrityofcomputation(Soundness)

User-centricsafeinformationflow

JAISA2015

HIPAA,(J-)SOX,KonTraG,EUGDPD,JapanPersonalInformationProtectionLaw

Safety:AZero-KnowledgeProof?


......

• Multilateralsecurity⇒ User-centricsafeinformationflow

dd,d*

Dataprovider/consumer

Dataconsumer

Dataconsumer/provider

Dataprovider

SecondaryusePrimaryuse

• Vulnerabilityinreal-timebyinevitable,hiddendependencies

Safety:AZero-KnowledgeProof?


......

• Multilateralsecurity⇒ User-centricsafeinformationflow

d

SecondaryusePrimaryuse

• Vulnerabilityinreal-timebyinevitable,hiddendependencies

Safetybyobscurity– Noreliablestatementoninformation

d,d*Dataprovider


Dataconsumer Dataprovider

Safety:Decidability


State-of-the-art:ISO270xx,IETFAAA(accesscontrol)

......

d

Dataprovider/consumer

Dataconsumer


Dataprovider

d,d*?

o1 =d o2 =d* …s1 own, r,w ?own,r,w?s2 r,w own,r,ws3 ?r,w? r…

Generalsecuritysystem

Decidabilityonsafetyingeneral⇒ HaltingproblemofTuringMachine

Probabilityofacorrectstatementonsafetyinthefuture=50%

Harrisonetal.1976Hamlen etal.2006

Enforcement

ThreattoCompleteness


• Informationflowfromdifferentsourcesinreal-time• Aggregationofanonymizedpersonaldata

Lossofcontrolonconfidentiality(ofhonestprover)

Bob David

Explicit/friendship

Implicitly assumed friendship

Sweeney2002

JerniganandMistree,2007

ThreattoSoundness


Lossofcontrolonclassification(ofhonestverifier)

• Knowledgecreationfrompersonaldatabysecondaryuse• “Faulty”dataincreaseserrorprobabilityofmachinelearning

Biggio etal2012;Huangetal2011

Supervised machine learning(e.g.SVM)

Poisoning Attacks against SVMs

0 2 4 6 80

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

% of attack points in training data

classification error (7 vs 1)

validation error

testing error

0 2 4 6 80

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4



validation error

testing error

0 2 4 6 80

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4



validation error

testing error

Figure 3. Results of the multi-point, multi-run experimentson the MNIST data set. In each plot, we show the clas-sification errors due to poisoning as a function of the per-centage of training contamination for both the validation(red solid line) and testing sets (black dashed line). Thetopmost plot is for the 7 vs.1 classifier, the middle is forthe 9 vs. 8 classifier, and the bottommost is for the 4 vs.0 classifier.

ing many tiny gradient steps. It would be interestingto investigate a more accurate and e�cient computa-tion of the largest possible step that does not alter thestructure of the optimal solution.

Another direction for research is the simultaneous opti-mization of multi-point attacks, which we successfullyapproached with sequential single-point attacks. Thefirst question is how to optimally perturb a subset ofthe training data; that is, instead of individually opti-mizing each attack point, one could derive simultane-ous steps for every attack point to better optimize theiroverall e↵ect. The second question is how to choosethe best subset of points to use as a starting pointfor the attack. Generally, the latter is a subset selec-tion problem but heuristics may allow for improved ap-proximations. Regardless, we demonstrate that evennon-optimal multi-point attack strategies significantlydegrade the SVM’s performance.

An important practical limitation of the proposedmethod is the assumption that the attacker controlsthe labels of the injected points. Such assumptionsmay not hold when the labels are only assigned bytrusted sources such as humans. For instance, a spamfilter uses its users’ labeling of messages as its groundtruth. Thus, although an attacker can send arbitrarymessages, he cannot guarantee that they will have thelabels necessary for his attack. This imposes an ad-ditional requirement that the attack data must satisfycertain side constraints to fool the labeling oracle. Fur-ther work is needed to understand these potential sideconstraints and to incorporate them into attacks.

The final extension would be to incorporate the real-world inverse feature-mapping problem; that is, theproblem of finding real-world attack data that canachieve the desired result in the learner’s input space.For data like handwritten digits, there is a direct map-ping between the real-world image data and the inputfeatures used for learning. In many other problems(e.g., spam filtering) the mapping is more complex andmay involve various non-smooth operations and nor-malizations. Solving these inverse mapping problemsfor attacks against learning remains open.

Acknowledgments

This work was supported by a grant awarded to B. Big-gio by Regione Autonoma della Sardegna, and bythe project No. CRP-18293 funded by the same in-stitution, PO Sardegna FSE 2007-2013, L.R. 7/2007“Promotion of the scientific research and technolog-ical innovation in Sardinia”. The authors also wishto acknowledge the Alexander von Humboldt Founda-

Unsupervised machine learning(e.g.PCA)

0.0

0.2

0.4

0.6

0.8

1.0

Single Poisoning Period: Evading PCA

Mean chaff volumeE

vasi

on s

ucc

ess

(F

NR

)

0% 10% 20% 30% 40% 50%

UninformedLocally−informedGlobally−informed

10

0 5 10 15 20

0.0

0.2

0.4

0.6

0.8

1.0

Boiling Frog Poisoning: Evading PCA

Attack duration (weeks)

Eva

sion s

ucc

ess

(ave

rage test

FN

R)

Growth rates

1.011.021.051.15

Figure 3: Effect of poisoning attacks on the PCA-based detector [36]. Left: Evasion success of PCA versusrelative chaff volume under Single-Training Period poisoning attacks using three chaff methods: uninformed(dotted black line) locally-informed (dashed blue line) and globally-informed (solid red line). Right: Evasionsuccess of PCA under Boiling Frog poisoning attacks in terms of the average FNR after each successive weekof locally-informed poisoning for four different poisoning schedules (i.e., a weekly geometric increase in thesize of the poisoning by factors 1.01, 1.02, 1.05, and 1.15 respectively). More aggressive schedules (e.g., growthrates of 1.05 and 1.15) significantly increase the FNR within a few weeks while less aggressive schedules takemany weeks to achieve the same result but are more stealthy in doing so.

sequent DoS attack. When trained on this poisoned data,the detector learned a distorted set of principal componentsthat are unable to effectively discern these DoS attacks—atargeted attack. Because PCA estimates the data’s princi-pal subspace solely on the covariance of the link traffic, weexplored poisoning schemes that add chaff (additional traf-fic) into the network along the flow targeted by the attackerto systematically increase the targeted flow’s variance. Inso doing, the attacker caused the estimated subspace to un-duly shift toward the target flow making large-volume eventsalong that flow less detectable. We considered three gen-eral categories of attacks based on the attacker’s capabilities:uninformed attacks, locally-informed attacks, and globally-informed attacks. Each of these reflect different levels ofknowledge and resources available to the attacker; see Sec-tion 3.3 and 3.1 for more detailed discussion of these models.In the above attacks, chaff was designed to impact a sin-

gle period (one week) in the training cycle of the detector,but we also considered the possibility of episodic poisoningwhich are carried out over multiple weeks of retraining thesubspace detector; see Section 3.4.2 for further discussionof iterative retraining. Multi-week poisoning strategies varythe attack according to the time horizon over which they arecarried out. As with single-week attacks, during each weekthe adversary inserts chaff along the target flow through-out the training period according to his poisoning strategy.However, in the multi-week attack the adversary increasesthe total amount of chaff used during each subsequent weekaccording to a poisoning schedule. This poisons the modelover several weeks by initially adding small amounts of chaffand increasing the chaff quantities each week so that thedetector is gradually acclimated to chaff and fails to ade-quately identify the eventually large amount of poisoning.

We call this type of episodic poisoning the Boiling Frog poi-soning method after the folk tale that one can boil a frog byslowly increasing the water temperature over time. The goalof Boiling Frog poisoning is to gradually rotate the normalsubspace, injecting low levels of chaff relative to the previ-ous week’s traffic levels so that PCA’s rejection rates staylow and a large portion of the present week’s poisoned trafficmatrix is trained on. Although PCA is retrained each week,the training data will include some events not caught bythe previous week’s detector. Thus, more malicious trainingdata will accumulate each successive week as the PCA sub-space is gradually shifted. This process continues until theweek of the DoS attack, when the adversary stops injectingchaff and executes their desired DoS.

In our prior work [57], we empirically demonstrated ourattacks against PCA and in Figure 3 we reproduce the re-sults of our experiments. These graphs depict the effec-tiveness of the Single-Training Period (leftmost figure) andBoiling Frog attacks (rightmost figure) in causing false nega-tives in terms of the percent of average increase in the meanlink rates due to chaff (see Section 3.2 for discussion of theattacker’s capabilities) and the length of the attack dura-tion, respectively. For the Boiling Frog attacks, we assumedthat the PCA-subspace method is retrained on a weekly ba-sis using the traffic observed in the previous week to retrainthe detector at the beginning of the new week. Further, wesanitize the data from the prior week before retraining sothat all detected anomalies are removed from the data. Asthe Figure 3 demonstrates, these attacks cause high rates ofmisdetection with relatively small increases in the volumeof traffic: e.g., a locally-informed attacker can increase hisevasion success to 28% from the baseline of 3.67% via a 10%average increase in the mean link rates due to chaff.

II.TowardsProvableSafety


StatusQuo:Language-basedinformationflowcontrol

RigorousNaturalLanguagePolicy

High-LevelPolicy

Language

Intermediate-LevelSecurityPolicyFlowGraph

Low-LevelEnforcement

InPractice

Take-grant,type-safety,lattice-basedaccesscontrol,

obligations

Identity,cryptography,safepublicdirectory,monitor,

proof-carryingcode

Decentralizedtrustmanagement

HIPAA,(J-)SOX,KonTraG,95/46/EC,JPPIIProtectionLaw,…

Enforcementclasses,Ponder,ExPDT

Computationalcomplexity,PKI,virtualization,testing

ISO/IEC270xx,BSIIT-BaselineProtection,IETF

AAA,NISTSCAP

Social/knowledgegraph,stickypolicies

securedelegationofrights

ZKP-carryinginformation

cf.Sandhu 1993,Myersand Liskov,1997;Schneider,Morrisett and Harper,2001;Sabelfeld andMyers,2003

Accesscontrol doesn‘t scale for resilience

Errorpropagation

Joined by Ground Truth

Rolechangeofsecondaryuse

Ext.:Reliable”BigBrother”

Int.:Errorpropagation

Rolechangeofsecondaryuse

(DP,DC,data,DS,time,…)

Dataminimization

SpecialCasesforSafety


• Strictorder

NaturalLanguagePolicy

High-LevelPolicy

Language



• Symmetricaccesstree

• Safetyiftreesareseparate • Availabilityofdatabydeclassification

Lattice-basedAccessControl

Sandhu1993

Take-grant

LiptonandSnyder1977

S1:u

S2:u S3:v

O:oS3:w

• Acyclicgraph

• x<=3parameter

• Norevocation

Type-safety

Sandhu1992

S1:u

S2:u S3:v

O:oS3:w

Example:Chinese-Wall


Conflictclasses

Personaldatasets

Syshigh

GroundTruth Registrationoffice

Medicaltreatment

Requiredinformationforenforcement(centralbySyshigh)



Conflictclasses

Personaldatasets

Syshigh


Medicaltreatment




Conflictclasses

Personaldatasets

Syshigh


Medicaltreatment

Bob David

Explicit/friendship





Conflictclasses

Personaldatasets

Syshigh


Medicaltreatment

Bob David

Explicit/friendship




High-LevelPolicy

Language


Low-LevelEnforcementLanguageforICTResilience


Safetyforsecondaryuse:Soundness(safety)∧ Completeness(safety+liveness)

d,d*d

Prover/Verifier

Verifier

Verifier/Prover

Prover

AccesscontrolProvisions

Provisions +observableobligationsUsagecontrol

Enforcement⇒ OpenDataofpersonalsecurityinformation(GroundTruth)

OpenDataonobligations

adaptedfromParkandSandhu2004;Pretschner,Hilty,andBasin2006

III.ProofSystemforICTResilience


Inpractice:Inevitablevulnerabilitybydependencies

Safeinformationaccountability⇒ Zero-KnowledgeProofonoriginofvulnerability


High-LevelPolicy

Language



d,d*d

Prover/Verifier

Verifier

Verifier/Prover

Prover

Scheduler(OpenData)






High-LevelPolicy

Language



d,d*d

Prover/Verifier

Verifier

Verifier/Prover

Prover

Scheduler(OpenData)

secd,d*

secd,d*

secd,d*

Knowledgeextractor

Knowledgeextractor

Knowledgeextractor

Ground Truth with secd,d*

Zero-Knowledge Proof (ZKP)

• Probabilistic proof system between 2 parties on graph isomorphism

• No additional knowledge for the verifier on original graph

• ICT Resilience: obligations + witnesses + compensation ⟼ Open Data

Prover Verifier

1. t random, a:=gt

2. a

3. c random out of {0,1}4. c

5. r:=t + cm mod q6. r

7. Check if gr = ahc

pkVerifier := (p, q, g, h) pkVerifier := (p, q, g, h)

Cha

lleng

eR

espo

nse

Goldwasser et al. 1989, Bellare and Goldreich 1993






High-LevelPolicy

Language



d,d*d

Prover/Verifier

Verifier

Verifier/Prover

Prover

Scheduler(OpenData)

secd,d*

secd,d*

secd,d*

Knowledgeextractor

Knowledgeextractor

Knowledgeextractor

Ground Truth with secd,d*

Scheduler:GlobalAAA(A) Service


OpenInternetStandardRFC2904AAAAuthorization Framework

1:Authentication

2:Authorization

3:Accounting

+Witness for InformationAccountability

4:Accountability



Dataconsumer/providerAAA(A)

service

Schedulersecd,d*

secd,d*

secd,d*

Scheduler:ReliableBroadcast


Self-organizedconsensusbycryptography




secd,d*…

secd,d*

secd,d*

• Userscheckusers(Usersas“miner“checktransactionsandgetreward)

Nakamoto 2009

• BlockchainforsafepublicdirectorywitheCoin forriskcompensation

Scheduler

Witness:Authorization


• Completeness:Non-linkabledelegationofrights• Soundness:Cryptographicprotocols(ISO/IECJTC1/SC27WG2)

GroundTruth:ISO/IEC24761ACBio – BiometricswithPKI

Dataprovider



service

Scheduler

d


Sonehara,Echizen,andWohlgemuth2011





Dataprovider



service

Scheduler

d


Issuer:Public Key:Attributes:

Ground TruthZKP on Xa23r,w, own on dDelegation, Purpose, …

Credential of data subject







Dataprovider



service

Scheduler

d



Ground TruthZKP on Xa23r,w, own on dDelegation, Purpose, …

Credential of data subject Issuer:

Public Key:Attributes:

MinerZKP on Xa23r on dDelegation to helper, medical, Time, Price …

Credential on d


MinerZKP on Xa23r on d*Delegation to logistics, transport, Time, Price,…

Credential on d*

Witness:Accountability


• Completeness:User’sdataprovenancewithasymmetricfingerprinting• Soundness:Users’cryptographiccommitmentondataprocessing


Dataprovider



service

Scheduler

d


d

RefugeeGround Truth

Wohlgemuth,Echizen,Sonehara,andMüller2010





d

Dataprovider



service

Scheduler

d


d

RefugeeGround Truth

d

RefugeeGround Truth

HelperWohlgemuth,Echizen,Sonehara,andMüller2010





d

d*

Dataprovider



service

Scheduler

d


d

RefugeeGround Truth

d

RefugeeGround Truth

Helper

d*

RefugeeGround Truth

Logistics

Wohlgemuth,Echizen,Sonehara,andMüller2010

KnowledgeExtractor:Accounting


• Reduceerrorprobabilitybydifferentwitnessesonusers

d,d*d

Prover/Verifier

Verifier

Verifier/Prover

Prover

trust rightscert dataprovenancerec delegationofrightsconf benchmarkingcomp profitsharing

• Probabilisticlogicalstatementonsafetyfromuser‘sview(onaPKI)

AAA(A)service

Scheduler

AutDC,DPd ?

AutDC,DPd,d*?AutDP,DCd,d*?

AutDP,DCd,d*,d**?

adapted fromMaurer1996,Wohlgemuth2015

KnowledgeExtractor:Accounting


• Reduceerrorprobabilitybydifferentwitnessesonusers

d,d*d

Prover/Verifier

Verifier

Verifier/Prover

Prover

trust rightscert dataprovenancerec delegationofrightsconf benchmarkingcomp profitsharing

• Probabilisticlogicalstatementonsafetyfromuser‘sview(onaPKI)

ICTResilience=Completeness+SoundnessICTResilience=Informationalself-determination+Compliance

AAA(A)service

Scheduler

AutDC,DPd ?

AutDC,DPd,d*?AutDP,DCd,d*?

AutDP,DCd,d*,d**?

IV.LookingforPartners!


Challenge:CreatingaSustainableSociety

MultilateralSecurity

IoT IntegratedSociety




secd,d* … secd,d*

secd,d*

Scheduler

Resilient Risk Assessment (RA1)

Resilient ICT Services (RA2)

Resilient ICT Infrastructure (RA3)

Technical

Human Legal et al. Privacy

byDesign