23
www.kit.edu 04.08 KIT – the cooperation of Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH) Cloud Computing mit mathematischen Anwendungen Vorlesung SoSe 2009 Dr. Marcel Kunze Karlsruhe Institute of Technology (KIT) Steinbuch Centre for Computing (SCC)

Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

www.kit.edu

04.0

8

KIT – the cooperation of Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH)

Cloud Computing mit mathematischen Anwendungen

Vorlesung SoSe 2009

Dr. Marcel KunzeKarlsruhe Institute of Technology (KIT) Steinbuch Centre for Computing (SCC)

Page 2: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze2

Agenda Cloud Computing1. Einleitung

Was ist Cloud Computing?2. Grundlagen

Virtualisierung, Web Services,…3. Cloud Architekturen

Infrastruktur, Plattform, Anwendung4. Cloud Services

Amazon Web Services, Google App Engine5. Aufbau einer Cloud

OpenCirrus Projekt, Eucalyptus, Hadoop6. Cloud Algorithmen

MapReduce, Optimierungsverfahren, …

Praktische Übungen und AnwendungenVorlesung im Web: http://www.mathematik.uni-karlsruhe.de/mitglieder/l ehre/cloud2009s/

2

Page 3: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze3

Commercial Cloud Offerings (Small Excerpt)

Problem: Commercial offerings are proprietary and u sually not open for cloud systems research and development

Page 4: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze4

Cloud Systems Research

Simple, transparent, controllable cloud computing infr astructureWhat types of interfaces are appropriate for clouds ?How should cloud networks be constructed/managed?How are security concerns addressed in “the cloud”?How are various workloads most efficiently transfer red?What types of applications can run in clouds?What types of service level agreements are appropri ate/possible?

Research requirementsPerform experiments also on a low system levelFlexible cloud computing frameworkCompare different methodologies and implementations

Page 5: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze5

Cloud Computing

� Cloud computing R&D: OpenCirrus TM project

OpenCirrusTM

Page 6: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze6

Page 7: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze7

OpenCirrus™ Cloud Computing Research Testbedhttp://opencirrus.org

An open, internet-scale global testbed for cloud computing research

Data center management & cloud servicesSystems level researchApplication level research

Structure: a loose federationSponsors: HP Labs, Intel Research, Yahoo!Partners: UIUC, Singapore IDA, KIT, NSFMembers: System and application development

Great opportunity for cloud R&D

Page 8: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze8

Where are the OpenCirrus™ sites? Six sites initially:

Sites distributed world-wide: HP Research, Yahoo!, UIUC, Intel Research Pittsburgh, KIT, Singapore IDA1000-4000 processor cores per siteNew CMU site coming in 2009

HPYahoo(sf)

UIUC

Intel(pgh)

KIT(de)

IDA(sg)

CMU (coming in ‘09)

Page 9: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze9

HC3: OpenCirrus Installation at KIT (July)

2656 Nehalem Cores / 332 Servers / 10 TB MemoryInfiniband QDR Interconnect192 TB Lustre Filesystem

MSA2000fc

Infniband QDR Interconnect

Page 10: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze10

OpenCirrus Global Services

Federation of sites needs common global services

Global SignOnSingle authentication at all sitesBased on distribution of ssh keys

Global MonitoringBased on Ganglia ( http://ganglia.info/ )

Scalable distributed monitoring systemSupports federations of clusters

OpenCirrus service at http://12.161.200.222/ganglia

Global DataTBD

Page 11: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze11

Cloud Architecture

Source: S.Tai

Page 12: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze12

OpenCirrus TM Blueprint

IT infrastructure layer (Physical Resource Sets)

Cloud infrastructure services

Cloud application services

Virtual Resource Sets

Eucalyptus

Page 13: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze13

Physical Resource Sets (PRS)

PRS service goalsProvide mini-datacenters to researchersIsolate experiments from each otherStable base for other research

PRS service approachAllocate sets of physical co-located nodes, isolate d inside VLANs.Leverage existing software (e.g. Utah Emulab, HP Op sWare)Start simple, add features as we goBase to implement virtual resource sets

Hardware as a Service (HaaS)

Page 14: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze14

Virtual Resource Sets (VRS)

Basic idea: Abstract from physical resource by introduct ion of avirtualization layerConcept applies to all IT aspects: CPU, storage, net works and applications, …Main advantages

Implement IT services exactly fitting customer‘s va rying needDeploy IT services on demandAutomated resource managementEasily guarantee service levelsLive migration of servicesReduce both: CapEx and OpEx

Infrastructure as a Service (IaaS)Implement Compute and Storage servicesDe-facto standard: Amazon Web Services interface

Page 15: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze15

Eucalyptus: A potential VRS Layerhttp://eucalyptus.cs.ucsb.edu

EUCALYPTUS - Elastic Utility Computing Architecture fo r Linking Your Programs To Useful Systems.

Open-Source software infrastructure for implementin g cloud computing on clusters from UC Santa Barbara.Impelements Infrastructure as a Service (IaaS) – gives the user theability to run and control entire virtual machine instanc es (Xen) deployed across a variety of physical resources.

Interface compatible with Amazon EC2 and S3.Potential to interact with the same tools, known to work withAmazon EC2 and S3.Linux image hosting ala Amazon Web Services.

Eucalyptus is an important step for archiving an open and established cloud computing infrastructure standard.

Spin-off Eucalyptus Systems with venture capital

Page 16: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze16

Eucalyptus: A potential VRS Layerhttp://eucalyptus.cs.ucsb.edu

Source: R.Wolski

Schedules the distribution

of virtual machines to the

NC. Collects (free) resource

information.

Collects resource

information from the

CC. Operates like a

meta-scheduler in the

cloud.

Runs on every node in

the cloud. Xen-

Hypervisor running.

Provides Information

about free resources to

the CC.

Page 17: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze17

R&D Cloud Installation at SCC/KIT

Eucalyptus R&D Cloud I2x IBM Blade LS20

Dual Core Opteron (2,4GHz)4GB RAM

2x IBM Blade HS21Dual Core Xeon (2,33GHz)16GB RAM

Eucalyptus R&D Cloud II5x HP Blade ProLiant BL2x220cEach Blade: 2 Server Nodes

2x Intel Quad-Core Xeon (2,33GHz) 16GB RAM

Page 18: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze18

Comparing Storage Performance between S3 and Eucalyptus

Sequential OutputPer-Character: file is written usind putc()

Block: file is written using write()

Rewrite: read() and write()

Sequential InputPer-Character: file is read usind getc()

Blockwise: file is written using read()

WOW!

Source: C.Baun

Page 19: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze19

Realistic values…

The RAM of the Eucalyptus Node Controller was reduced to overcome memory caching.The storage performance of Eucalyptus depends on the avai lable storage sub-system.

Source: C.Baun

Page 20: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze20

Performance of Random Seeks and File Creation

The RAM of the Eucalyptus Node Controller was reduced to overcome memory caching.The storage performance of Eucalyptus depends on the avai lable storage sub-system.

Source: C.Baun

Page 21: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze21

How is OpenCirrus™ different from other testbeds?

OpenCirrus TM supports both system- and app-level research

n/a at Google/IBM and EC2/S3OpenCirrus TM researchers will have complete access to the underlying hardware and software platform. OpenCirrus TM allows Intel platform features that support cloud computing (e.g. DCMI, NM) to be exposed, and exploited .

Virtualmachines

Hadoop

Map-Reduce apps

Google/IBMcluster

Virtual or physical machines

Cluster mgmt software

Open Cirrus cluster

Hadoop

Cloud apps and services

Map-Reduceapps

Cannot be modified by users

Can be modified by users

Can be modified by users

Page 22: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze22

Summary

OpenCirrus TM offers interesting R&D opportunitiesCloud systems developmentCloud application developmentAccepting research proposals

OpenCirrus TM workshop at HP Palo Alto on June 8/9https://opencirrus.org/content/agenda-open-cirrus-summit -palo-alto-june-8-9-2009

Page 23: Cloud Computing mit mathematischen Anwendungen · 2015-11-10 · Cloud Computing mit mathematischen Anwendungen ... Open-Source software infrastructure for implementing cloud computing

Cloud Computing Teil 8 | SoSe 2009 | Dr. M.Kunze23

Karlsruhe Institute of Technology

Thank you for your attention.Steinbuch Centre for Computing (SCC)