The evolution of ENEAGRID/CRESCO HPC infrastructure · 2014-10-30 · Development] are integrated in ENEAGRID infrastructure, a production quality, service oriented system for high

G. Bracco Evolution ENEAGRID/CRESCO - Portici 22/5/2013

Portici22 maggio 2013

The evolution of ENEAGRID/CRESCO HPC infrastructure

[email protected]

ENEA Centro Ricerche Frascati V. Enrico Fermi 45, Frascati (ROMA)

S.Migliori,A.Quintiliani,S.Podda,R.Guadagni,F.Ambrosino,F.Beone,M.Caporicci,P.D'Angelo,A.Funel,G.Ponti,G.Furini,A.Mariano,

G.Mencuccini,P.Ornelli,A.Perozziello,S.Pierattini,D.Abate,F.Poggi,D.Giammattei,M.DeRosa,S.Pecoraro,F.Simoni,S.Giusepponi,

G.Guarnieri,A.Petricca,A.Rocchi,C.Sciò,A.Italiano,A.Colavincenzo,G.Giannini


ENE

The scientific computing resources of ENEA [The Italian National Agency for New Technologies,Energy and Sustainable Economic Development] are integrated in ENEAGRID infrastructure, a production quality, service oriented system for high performance and high throughput computing. The main computing resources are the CRESCO clusters (Linux x86_64), the main site is Portici (NA), the location of the original CRESCO project (2008, Computational RESearch center for COmplex systems), funded by the Italian Minister of Research and the Education,in the framework of PON 2000-2006. This presentation describes the infrastructure and illustrates its evolution as funded by the new projects (PON 2007-2013 framework) where ENEA-UTICT, the ICT Unit of ENEA, is one of the partners.

Overview


Outline

• ENEAGRID– The architecture & the computing resources–Users and applications–User interface and Virtual Labs–CRESCO clusters

• New projects PON 2007-2013– IT@CHA, TEDAT, LAMRECOR

• New infrastructures–CRESCO3–CRESCO4

• Conclusions


ENE

The infrastructure is based on “mature” multiplatform software components assuring reliability and easy administration. Web interfaces have been developed/customized for a friendly user environment:– Kerberos 5 authentication– File systems: • AFS/OpenAFS geographic file system (HOME)• GPFS: parallel file system (also WAN)

– Resource manager: LSF Multicluster– User Web graphical interface:

• NX/FARO• Jobrama: job monitoring & Accounting

– System monitoring: Zabbix–Web management of users/projects: WARC

ENEAGRID architecture


ENE

ENEAGRID offers to the users computing resources based on Linux x86_64 ( ~5800 cores), AIX SP5 (~256 cpu), special systems (e.g. GPUs), virtualized hosts and distributed storage. The resources are located in 6 sites, connect by GARR network.

ENEAGRID: computing resources

ENE

ENE


ENE

ENEAGRID: CRESCO & Portici Site

ENEA Research Center PorticiBuildings designed by Vittorio Gregotti,1982/86


ENE

ENEAGRID users: – serial or small scale parallel jobs: ~200 users– large scale HPC parallel jobs: ~50 users

Application domains: – combustion CFD– aerospace CFD– computational chemistry– climate modeling– atmospheric pollusion diffusion simulation– nuclear technologies– nuclear fusion physics– bioinformatics– ...

ENEAGRID: users and applications


ENE

ENEAGRID: applications

Compilers: Intel, PGI, AMD Open64,... MPI Flavours:mvapich,openmpi,Intel,...

Applications:Abaqus, Amira, Ansys, Ansys CFX, Ansys Fluent, AVS Express Developer Edition, COMSOL, CPMD, E-cell, Fluent, FreeMat, Gambit, Grass GIS, Gsharp, IcemCFD, IDL, LynxPrime, Marc, Mathematica, Matlab, MeshLab, ModeFrontier, MpCCI, MultiGen-Paradigm VegaPrime, Nastran, OpenFOAM, OpenSceneGraph, Paraview, Patran, Prince, Quantum Espresso, Quantum GIS, Scalasca, Scilab, Scirun, Siap, Starccm+, StarDesign, StarView, Tgrid, Totalview, VIsit, Visual Molecular Dynamics, Workbench, XILINX...


ENE

CRESCO utilization by domain of activity

Combustion CFD 621.87 ENEA

New materials 531.9 ENEA, INFN/RM, INFN/NA,UniFI,UniSaUniSS, Numonyx

Climate modeling 357.57 ENEA,Ylichron

Nuclear fission 122.9 ENEA,ISS

HPC Support activity 103.11 ENEA

Atmospheric pollution 65.62 ENEA,AriaNet

Aerospace CFD 32.77 Avio,AAPS,UniROMA1

University 22.13 UniROMA1,CERI

Nuclear fusion 19.99 ENEA,PoliTo

Bioinformatics 18.3 ENEA, CNR-ITB, Ylichron,CNR-ISA

EFDA- Fusion 4.69 ENEA,EFDA

Industry 0.37 NICE,CETMA


FARO - Fast Access to Remote Objects

Web access to data, applications and virtual machines, a solution based on the integration of NX and a java interface


FARO & Virtual Labs

Thematic portals integrating the access to data and to the applications for a specific context/activity


Virtual Labs (1)


Virtual Labs (2)


Virtual Labs (3)


Virtual Labs (4)


Virtual Labs (5)


Total displace

ment (mm)

Post

Processor

Total displace

ment (mm)

Total displace

ment (mm)

Post

Processor

Post

Processor

FARO: 3D visualization

3D Remote Rendering


HPC: scalability test of various applications

Fluent

OpenFoam

Commercial code

Combustion

Open Source:OpenFoam

Processors

Use

r Cod

e

procs

cpmd


ENE

CRESCO clusters

>Portici CRESCO1(672 cores 4U) CRESCO2(2720 cores blades HS21,GPFS)IB CISCO 70xx, DDR Intel Clovertown,Tigerton,Nehalem,Westmere

>Casaccia CRESCOC (192 cores, twin 1U,Supermicro)IB Qlogic Silverstorm DDR, AMD 2427 Istanbul

>Frascati CRESCOF (480 cores twin square 2U, GPFS)IB Qlogic 12300 QDR, AMD 6172 Magnycours

>Brindisi CRESCOB (80 cores, 4 U, GPFS)GEthernet, Intel Tigerton

>Trisaia CRESCOT (16 cores,4U)Intel Tigerton


ENE

CRESCO1/2 Clusters - DDR IB & Storage

SFS-7000D(D) SFS-7000D (e)

SFS 7000D

SFS 7000D

SFS 7000D

SFS-7024-288P

4 graphic FRONT END

2X GPFS NODE

SFS 7012-144-P

4X FRONT END

FCFC

(2)

4

4

4

4

42 X 3850M2

10 X 3755

340 X HS21SECTION 2

SM (ACTIVE) SM (standby

4

4

4 X FRONT END GRAFICI

4X FRONT END

2X GPFS NODE

4)

4

4

2) 2)

2)2)

2) 2)

4

BACK-UP

4

IBM DCS 9550 -180TBDDN S2A9900 120TB 2X GPFS NODE

IB 36 ports

4

20 TBServerSTORAGE

20 TBServerSTORAGE

20 TBServerSTORAGE

20 TBServerSTORAGE

CRESCO2

Storage: IBM/DDN9550 120 TB; DDN9900 90 TB; disk servers 80 TB

CRESCO1


New projects : PON 2007-2013

• PON/1 Ricerca Industriale DDR MIUR 01/Ric 18/1/2010– IT@CHA - (application of new technologies to the conservation and

valorization of cultural and artistic heritage)– LAMRECOR - (Advanced logistics for persons and goods)– DIRECTFOOD - Gestione integrata filiere alimentari e canali innovativi

produttore -consumatore

• PON/2 Distretti/Laboratori Pubblico Privati DDR MIUR 713/Ric 29/10/2010

– VIS4FACTORY - Sistemi Informativi Visuali per i processi di fabbrica nel settore dei trasporti

– DATABENC - Distretto ad ALta Tecnologia per i beni culturali nella regione Campania

• PON/3 Infrastrutture DDR MIUR 254/Ric 18/5/2011– TEDAT - Centro di eccellenza per le Tecnologie e la diagnostica avanzata nel

settore dei trasporti (Advanced technologies for transport,new materials for aerospace, automotive,..)


PON 2007-2013: Projects started 2012

Budget for infrastructures:

• IT@CHA [Ricerca industriale] ENEA-UTICT 320 k€• LAMRECOR [Ricerca industriale] ENEA-UTICT 548 k€

On these projects: new CRESCO3 cluster

• TEDAT [Infrastrutture] ENEA-UTICT 2.221 k€– Portici (2.056 k€)– Brindisi (165 k€)

On this project: new CRESCO4 cluster


Portici: new cluster CRESCO3

In the framework of IT@CHA and LAMRECOR projects the new CRESCO3 cluster has been set up in CRESCO computer room:

84 AMD nodes, 2016 cores, Twin Square SuperMicro & Acer, 19.3 TFlops peakInfiniband QDR QLogic 12800-040 (96 ports)2 Racks (wide)

The cluster is now opened for general use.

HPL Benchmark: 75 % efficiency; compilers with Interlagos support (Open64 specific flags, Intel 12 specific flags; ACML Libraries)

Storage: DDN S2A9900 600 TB; GPFS


ENE

AMD Iterlagos 6234 2.4 GHz,64 GB,24 cores,84 nodes,2016 cores

CRESCO3:SMicro/Acer twinsquare QDR IB


Portici: CRESCO4 cluster

In the framework of TEDAT project the procurament of the new CRESCO4 cluster is currently under way. An European tender has been set up, evaluated and the contract has been awarded (E4 Company).

– 304 computing nodes Intel E2670, 2.6 GHz, 4864 cores– Supermicro Fat-Twin chassis (8 nodes in 4U)– 101 TFlops Peak– 5 Racks for computing nodes, 1 network rack – Storage: DDN S2A9900 600 TB– Infiniband QDR QLogic/Intel 12800-180 (432 ports)– New computer room – Conditioning system taking advantage of free cooling technogy– Delivered in Portici by July 2013– Final test completed October 2013


ENE

Portici: CRESCO4 - IB network


ENE

Computer rooms: CRESCO & CRESCO4

CRESCO computer room

new CRESCO4 computer

room


ENE

CRESCO computer room (1)


ENE

CRESCO computer room (2)


ENE

CRESCO4 computer room (1)


ENE



ENE



ENE



ENE



ENE

Conclusions

A significant update of the ENEAGRID HPC infrastructure is currently under way, with the new CRESCO3 and CRESCO4 clusters, funded by several PON 2007-2013 projects

The architecture of ENEAGRID provides the framework for an easy introduction of the new computing resources both from the point of view of the user and the administrator.

By the end of the year the integrated peak computing power of ENEAGRID will increase of a factor 5 up to ~150 Tflops, the most powerfull system being CRESCO4 with ~100 Tflops.


Links

• www.enea.itwww.enea.it

• www.cresco.enea.itwww.cresco.enea.it

• www.eneagrid.enea.itwww.eneagrid.enea.it

• www.afs.enea.itwww.afs.enea.it

Documents

The evolution of ENEAGRID/CRESCO HPC infrastructure · 2014-10-30 · Development] are integrated in ENEAGRID infrastructure, a production quality, service oriented system for high