Höchstleistungsrechnen in Forschung und Entwicklung › ... › HLRS_Presentation.pdf · 2013-02-11 · Graduate & PhD Students Chair for High Performance HLRS Computing Center for

1Storage Consortium 2007

1 High Performance Computing Center StuttgartPeter W. Haas

Höchstleistungsrechnenin Forschung und Entwicklung

Peter W. Haas, Michael M. Resch

Anwenderkonferenz des Storage Consortiums14. Februar 2007, Garching

Storage Consortium 20072 High Performance Computing Center Stuttgart

Peter W. Haas

Table of Contents

1. Höchstleistungsrechenzentrum Stuttgart (HLRS)– The HLRS Framework– Key Application Scenarios

2. The Teraflop-Workbench Concept– Concepts and Workflow Examples– NEC SX-8 Configuration

3. Teraflop-Workbench Projects– Methods and Application Areas– Individual Application Projects

4. Terascale Storage and Communication– The BelWü 40-Gbit/s Research Network



Höchstleistungsrechenzentrum Stuttgart

synergy@work



The HLRS Framework

hwwhww

Production

Teaching &Research

Computational Science & EngineeringSchool for Mechanical Engineering

Computational Science & EngineeringSchool for Mechanical Engineering

Undergraduate,Graduate & PhDStudents

Chair for Chair for High High Performance Performance ComputingComputingHLRS

Center for Center for Scientific ComputingScientific ComputingUniversity of StuttgartUniversity of Stuttgart

hkzhkz--bwbwStuttgart Stuttgart –– Karlsruhe Karlsruhe -- HeidelbergHeidelberg

Services


Peter W. Haas

Porschenetworks

Networks

networksT-Systemsnetworks

Untertürkheim

4 Gbit/sd ~20 km

Vaihingen

DaimlerChryslernetworks

Universität Stuttgart

n x 10 Gbit/s

100 Mbit/s

1 Gbit/s

Universität Karlsruhe

BelWü40 Gbit/s


Peter W. Haas

Supercomputer

Fileserver

Stuttgart

NECAzusa (16)

HP zx6000 (16)

Appro(40)

NEC SX-6(48)

BullNovaScale (16)

NEC(416)

Cray(256)

IA64 (48) IA32 (446) Opteron (256)

NEC SX-8 (576)

Karlsruhe

HP XC 6000(328)

Ulm

SNU Fire 6800(96)


Peter W. Haas

Medical Application Example

GRID


Peter W. Haas

Ulrich Ray, Director EP/QI, Mercedes Car Group, EDM 2003, July 16-17, 2003



The Teraflop-Workbench Concept

HLRS - synergy@work


Peter W. Haas

Workbench Concept

File-System

Super-Computer

PP-Server

PP-Server

PP-Server

Vis-ServerAccess


Peter W. Haas

Execute Simulation

If Success

Notify user

Postprocessing

Filtering

Export Data

Create Input Data

Import Data

Preprocessing JobCluster

Client

Compute Task

Workflow Example


Peter W. Haas

Parallel Post-Processing

• Interactive visualization

• Simulation steering

Sim-Cluster Vis-Cluster Compositor

Sim-Cluster Vis-Cluster Compositor

Par.-FS.

Post-Processing + Rendering

Simulation + Post-Processing Post-Processing + Rendering


Peter W. Haas

Requirements

• Coordinated with industry and user community– 50% memory bandwidth dominated

applications -> vector systems– 50% scalability dominated applications -> big

clusters

• 3 TFLOP/s sustained (expected peak of 8-15 TFLOP/s)

• Highly integrated and balanced system• New technology approaches welcome• Close co-operation between vendor and HLRS• Common planning for a 5 year project


Peter W. Haas

Final NEC Configuration at HLRS

SX-8 72 nodes

FC Switch

FC Disk

IXS

114ports

#1 #2

FC Disk

FC Disk

FC Disk

FC Disk

FC Disk

FC Disk

FC Switch114ports

FC Switch114ports

GbEther SW

FC Switch114ports

GbEther SW

SX-8#3

SX-8#1

SX-8#2

SX-8#4

SX-8#1

SX-8#6

SX-8 SX-8#71

SX-8#72#70

GFS Server TX-7

NoconaNoconaNoconaNoconaNocona

NoconaNoconaNocona

Infiniband216 ports


Peter W. Haas

Teraflop WorkbenchVector SMP ClusterIA64 SMP Cluster PC Cluster

SAN• Integrated workbench for preprocessing, simulation, postprocessing

and vizualisation• Choose the best platform for each task• Demonstrate feasibility of workbench approach


Peter W. Haas

NEC SX-8

~ 400 m²~ 750 kW

Overall SystemNumber of Nodes 72IXS Switch 128x128 16 GB/sProcessorsNumber of Processor 576Peak Performance 12,67 TF/sMemorySize of Memory 9,2 TBSpeed of Memory 36,8 TB/s


Peter W. Haas

Current Cisco MDS Configuration

MDS 9509112port

Disk#1

Disk#36

Disk#37

Disk#72

MDS 9509112port

MDS 9509112port

MDS 9509112port

MDS 950664port

MDS 950664port

～～

SX#1

SX#72

TX7#1

TX7#1-2・・・

x36 x36 x36 x36

x72 x72 x72 x72x4 x4 x4 x4

x12

x8x8

x12

MDS9506/ 64port x2MDS9509/112port x4

TX7#1

PCCluster

x N x N


Peter W. Haas

Fabric Topology

Disk#1

MDS 9509domain 2

Disk

MDS 9509domain 3

MDS 9509domain 2

MDS 9509domain 3

MDS 9506domain 1

MDS 9506domain 1

SX TX7#1TX7 TX7

#1

PCCluster

Disk#1

Disk

Fabric 1 Fabric 2

SX

Domain number must be uniquewithin a Fabric.VSAN number may be same for all MDS (use VSAN 1 only).


Peter W. Haas

Zoning Configuration (SX-8 host)

MDS 9509Domain 2

MDS 9509Domain 3

SX#1

SX#2

SX#72

Disk#1

Disk#2

Disk#37

Disk#38

Disk#72

Disk#36

Same zone Same zone

Different zone

Different zone

Zone policy of recommendation is One Zone per One HBA.

20Storage Consortium 200720 High Performance Computing Center Stuttgart

Peter W. Haas

Teraflop-WorkbenchProject

HLRS – synergy@work


Peter W. Haas

Areas

• Aeroacoustics• Aerodynamics• Astrophysics• Bioinformatics• Combustion• Fluid-Structure

Interaction• Helicopter Aerodynamics• Nanotechnology• Solid State Physics• Turbo Machinery• Turbulence Phenomena


Peter W. Haas

FENFLOSS

IHS – University of StuttgartHLRS – synergy@work


Peter W. Haas

Projects: Fenfloss

• IHS, University Stuttgart• Scientific Area: Turbo

Machinery• Method: CFD

•Two and three dimensional on unstructured grids

• Laminar and turbulent flows

• Reynolds-averaged Navier-Stokes

• various turbulence models

• 50% efficiency on single CPU SX-6

• estimated 2TF on SX-8

• Integrated work environment

• Example:

•controllable pitch propeller tailored to a specific ship to reduce noise and avoid cavitation.


Peter W. Haas

Optimization of geometry using Virtual Reality

and „online“ flow simulation

DEMONSTRATION in the HLRS-Cave


Peter W. Haas


Peter W. Haas

CCARAT

LNM, TU-MunichHLRS – synergy@work


Peter W. Haas

Projects: CCarat – what is it?

• IBS, University of Stuttgart and LNM, TU München • Scientific Area: Fluid Structure Interaction• Method: Finite Element

• Transient interaction of incompressible viscousflows and nonlinear flexible structures

• GLS stabilised fluid elements in 2d and 3d on a moving mesh (ALE formulation)

• Sophisticated structural elements (e.g. shell, brickand beam)

• External solvers (e.g. AZTEC, SPOOLES, UMFPACK and TRILINOS)

• Performance limited due to iterative solver

• Example: 2d simulation of the failure of Tacoma Narrows suspension bridge.


Peter W. Haas

CCarat Simulation of Tacoma Bridge

Simulation IBS, University of Stuttgart


Peter W. Haas

Terascale Storage&

Communication


Peter W. Haas

Unified Heterogeneous File Systems

Shannon Filing Cabinet, Schlicht & Field Co., Rochester, NY, 1886http://www.officemuseum.com


Peter W. Haas

File Systems for Clusters

Symmetric ClusteredFile Systeme.g. GPFS

Lock management Is bottleneck

C C C

C C C

ComponentServer

Parallel File System like PFS

AsymmetricMD Server is bottleneck

ComponentServer

C C C

Server

Distributed File Systeme.g. NFS/CIFS

Server is bottleneck

SAN basedFile Systemslike SANergy

Server is bottleneckScale limited

sanServer

C C C

MetadataServer


Peter W. Haas

Lustre Solution

C C C

OST

Asymmetric Cluster File SystemScalable MDS handles object allocation,

OSTs handle block allocation

OST MDS Cluster


Peter W. Haas

HLRS Data Management Strategy

Backup

100 km

hkz-bw

KarlsruheStuttgart

DataManagement

System

VIS

PPVIS

HSM

PP

HP/CFS Lustre

IBM HPSS


Peter W. Haas

HLRS Storage Configuration

100 kmKarlsruhe Stuttgart

Metadata &Storage Servers

Meta data bases &Storage Servers

SSCK HPC SystemHP XC

HLRS HPC SystemsNEC, Cray, et.al.

FC Network FC Network

...RAID Systems

RAID Systems

System Area Network(10GE, 10GFC, 4 x IB)

...

... ...

Parallel Tapes


Peter W. Haas

10-Gbit/s Fibre Channel Remote VSANs

FC

Cisco MDS 9513With 4 port 10G FC card

In Stuttgart

Cisco MDS 9509With 4 port 10G FC cardand Inter-VSAN-Routing

in Hohenheim

Disk VSAN

Server VSAN

Inter-VSANRouting

10 Gig FC 10 Gig FC

DWDM DWDM

10G λ – 12km

Storage

Server

Feb 23, 2006:

Ralf Eberhardt, CiscoJoachim Thül, ADVAJoseph Michl, UoHPeter Haas, HLRS


Peter W. Haas

Generic Set of Client Interfaces

43 Gbit/soptical:B&W orcolored

43 Gbit/sDe/mux,

Transmitter,Receiver,adaptive

Regenerator

with FEC

43 Gbit/sDe/mux,

...

- 10GE LAN PHY, 10.3G

- SDH STM-64c, 9.9G

- OTN G709, OTU2, 10.7G

- 10GFC, 10.5G or via 10GE

- 4xLX IB, 10.0G or via 10GE

40-Gbit/s Multiplexing Transponder


Peter W. Haas

Transmission Impairment Mitigation

FEC FEC

MUX

and

REGENDEMUX

MUX

and

REGENDEMUX

Analog 40Gbit/slink performance

Regenerated error rate

Corrected error rate

10G Ethernet packet performance

a) End-to-End Monitoring of all Layers . . .


Peter W. Haas

Reduction of TCO

Directly modulated 1550-nm DFB laser diode module

DCF1

43 Gbit/sbinary, NRZ

Laser diode module

1686WMEDFA

ChannelFilter

43 Gbit/sReceiver

104 kmstandard fibre

DCF2

106 kmstandard fibre

43 Gbit/sinput

210 km standard fibre bridged at 43 Gbit/s with optical amplifiers and chromatic dispersion compensation out of Alcatel 1686WM product. Courtesy SEL/Alcatel, Stuttgart


Peter W. Haas

Transparent Switching of 10G Channels

40-Gbit/s Add/Drop Node

4:1MUX

1:4DE

MUX

40GOTX

40GORX

4:1MUX

1:4DE

MUX

40GOTX

40GORX

40 Gbit/soptical interface

40 Gbit/soptical interface

1 2 3 410-Gbit/s add/drop

40 Gbit/swest

40 Gbit/seast10-Gbit/s

transparentmatrix


Peter W. Haas

Thank you very muchThank you very muchfor your attention!for your attention!

Special Special thanksthanks areare duedue to:to:

ADVA ADVA OpticalOptical NetworkingNetworkingAlcatel SEL AG, StuttgartAlcatel SEL AG, StuttgartCiscoCisco SystemsSystemsDeutsches Zentrum fDeutsches Zentrum füür Luftr Luft-- und Raumfahrtund RaumfahrtEuropean HPC Technology CentreEuropean HPC Technology CentreInstitute of Institute of AerodynamicsAerodynamics and and GasdynamicsGasdynamicsInstitute Institute forfor HydraulicHydraulic MachineryMachineryInstitute of Institute of ProcessProcess Engineering and Power Plant TechnologyEngineering and Power Plant TechnologyInstitute of Institute of StructuralStructural MechanicsMechanicsHLRS HLRS VisualizationVisualization DepartmentDepartment

Documents

Höchstleistungsrechnen in Forschung und Entwicklung › ... › HLRS_Presentation.pdf · 2013-02-11 · Graduate & PhD Students Chair for High Performance HLRS Computing Center for