The Storage Evolution: From Blocks, Files and …konferenz-nz.dlr.de/pages/storage2008/present/2. Konferenztag/06_01... · The Storage Evolution: From Blocks, Files and Objects to

The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems

Storagetechnology 2008 04. - 06.06.2008im Radisson SAS Hotel Schloss Fleesenseeveranstaltet durch das Deutsche Zentrum für Luft- und Raumfahrt e.V.

Christian BanduletPrincipal EngineerGlobal Technical Advisory TeamSun Microsystems Inc.

Sun Microsystems, [email protected]

Topics• Block-Based Data Access• File-Based Data Access• Object-Based Data Access

> Object-Based Storage Devices (OSD)> Object Storage Systems

> Object Storage Server (OSS)> Content Addressable Storage (CAS)> Content Aware Storage (CAS)

• Intelligent Storage Nodes (ISN)







The Data Access Taxonomy

DataAccess

Block ? ?

SCSI, SAS,FCP, iSCSI,SRP, ATA

SATA


The Block Paradigm

Physical Blocks: e.g. 512 bytes

SCSI, SAS, FCP, SRP, iSCSI, ATA, SATA

Client Interface

0 1 2 3 4

5 6 7 8 9

10 11 12 13 14

15 16 17 18 19


Logical Blocks & Physical BlocksLet's have a closer look....

Logical Blocks 0

Physical Blocks

Mapping

Client Interface

16 17 18 19 20

0 1 2 3 4

5 6 7 8 9

10 11 12 13 14

15 16 17 18 19

1 2 3 4 5 6 7 8 9 10 11 12 13

SCSI, SAS, FCP, SRP, iSCSI, ATA, SATA


Magnetic Hard Disk Drives

Source:


Millipede

~ 10nmHuman Hair ~80.000nm

http://domino.research.ibm.com/Comm/bios.nsf/pages/millipede.html/$FILE/milli_cantilevers.jpg


Racetrack• http://www.almaden.ibm.com/spinaps/research/sd/?racetrack• Spintronics Devices Research

http://www.almaden.ibm.com/spinaps/research/sd/?racetrack


Where do you want to go?An example from nature

Information storage density 4 x 105 bits per π (50 x 10-9)2 m-2

-or- about 3x1016 bits/inch2 (~ 30 Pbit bits/inch2)A factor of 105 times today's state-of-the-art (x100,000) -or- equivalent to > 30 years of development

Source:

http://de.wikipedia.org/wiki/Zytomegalievirus


Competing with Viruses

Source:

(~ 30 Pbit bits/inch2)


Disk to CPU Discontinuity• Moore’s Law is outstripping disk

drive performance (rotational speed)

• As a result, servers and storage systems are hopelessly unbalanced between CPU/controller capability and storage pool performance

• The objective of modern systems design is to rebalance the CPU-storage ecosystem while optimizing both low $/GB and $/IOPS

Source: MASS002 Intel Developer Forum 2008


€€€€

€

PerformanceLOW HIGH

DRAM SSD

Storage TiersRAM

EnterpriseHDD

SATA

Archive

Tape

MAID

NAND-Flash SSD


Storage IPC LAN

ProcessorMemory

I/O I/O I/O

Storage IPC LAN

I/O Subsystem

ProcessorMemory

I/O consolidation


Ethernet vs. Fibre Channel19

82

1984

1986

1988

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

0

1000020000300004000050000

60000700008000090000

100000Ethernet vs FibreChannel

EthernetEthernetFibreChannelFibreChannel

Year

MBi

t/s

100 Gb/s

2011?

16 Gb/s

2011?


Speed And Feeds – Fibre Channel Roadmap

*Base2 used throughout all applications for Fibre Channel infrastructure and devices. Each speed maintains backward compatibility at least two previous generations (I.e., 8GFC backward compatible to 4GFC and 2GFC)

**Base10 is for ISLs, core connections, and other high speed applications demanding maximum bandwidth. Except for 100GFC (which follow Ethernet standards and compatibility guidelines), each Base10 speed is expected to be compatible at least one previous generation.

***FCoE: Fibre Channel over Ethernet tunnels FC through Ethernet and is compatible with all existing Fibre Channel fabric environments. FCoE ports follow Ethernet standards and compatibility guidelines.


I/O Consolidation Benefits

> Adaptor: NIC for Ethernet/IP, HCA for InfiniBand, Converged Network Adaptor (CNA) for FCoE

> Customer Benefit: Fewer NIC’s, HBA’s and cables, lower CapEx, OpEx (power, cooling)

CNA

CNA

FC HBA

FC HBA

NIC

NIC

FC Traffic

FC TrafficEnet Traffic

Enet Traffic

All traffic

go over

10GE


FCoE EnablersEt

hern

etH

eade

r

FCoE

Hea

der

FCH

eade

r

FC Payload CR

C

EOF

FCS

Same as a physical FC frame

Control information: version, ordered sets (SOF, EOF)

Ethernet V2 Frame, Ethertype = FCoE=%x8906

• 10 GbE• Data Center Ethernet (Lossless Ethernet)• Ethernet jumbo frames (2180 bytes)

> Max FC frame payload (2112 bytes)


Fewer NICs and Cables Same Management Model as Native FC

NoGateway

Less Power and Cooling

FC TrafficFC Traffic

Enet TrafficEnet Traffic

FCoEFCoE

FC Storage FC Switch FCoESwitch

FCHBA

DisplayFCoE

AdapterFCoE

Adapter

Server

FCoE SAN

SAN A

SAN BFCoE

FCoE

I/O Consolidation: Benefits to Customers


FCoE Switch – Simplified Architecture

Ethport

Ethport

Ethport

Ethport

Ethport

Ethport

Ethport

Ethport

Ethernet Bridge

FCport

FCport

FCport

FCport

FCFFCoE

*FCF: Fibre Channel Forwarder


I/O Consolidation

• Slower I/O• Different service needs –• different fabrics• No flexibility• More ports to manage• More power• More space• Higher TCO

• High bandwidth pipe for capacity provisioning• Dedicated I/O channels enable convergence

> For Networking, Storage, Management> Application compatibility> QoS - differentiates different traffic types> Partitions – logical fabrics, isolation

• Gateways - S hare remote Fibre Channel and E th ports> Design based on average load across multiple servers> Scale incrementally – add Ethernet/FC/Server blades> Scale independently


Upper Layer Protocols• ULPs connect InfiniBand to common interfaces• Supported on mainstream operating systems• Clustering

> MPI (Message Passing Interface)> RDS (Reliable Datagram Socket)

• Network> IPoIB (IP over InfiniBand)> SDP (Socket Direct Protocol)

• Storage> SRP (SCSI RDMA Protocol - www.T10.org)> iSER (iSCSI Extensions for RDMA) – IETF

IP Storage WG> NFSoRDMA (NFS over RDMA)> FcoIB

• ISER, SRP and NFSoRDMA are part of mainline Linux kernel


Physical Layer


InfiniBand Resources• InfiniBand software is developed under OpenFabrics Open source Alliance

> http://www.openfabrics.org/> The mission of the OpenFabrics Alliance (OFA) is to develop, distribute

and promote a unified, transport-independent, open-source software stack for RDMA-capable fabrics and networks, including InfiniBand and Ethernet. The software stack is developed for many hardware architectures and operating systems, including Linux and Windows. Upper level protocols in the stack support IP, Sockets, SAN, NAS, Clustered File Systems and Database application environments

• InfiniBand standard is developed by the InfiniBand® Trade Association> http://www.infinibandta.org/home

http://www.openfabrics.org/








DataAccess

Block ? File


SATA

Local FS,Distributed FS:

(NFS, CIFS, WAFS)Shared FS:

(Cluster FS, SAN FS)Global, Distributed

& Parallel FS


Application Server

Data File system structure (i.e. inodes)

Physical Blocks on Disk

Logical Blocks on Disk0

16 17 18 19 20

0 1 2 3 45 6 7 8 9

10 11 12 13 1415 16 17 18 19

1 2 3 4 5 6 7 8 9 10 11 12 13

Mapping

Physical Disk

Local File Systems


SAN

MDS*

Shared FS with integrated volume management

Concatenate, RAID0, RAID1, RAID5, ...

Shared FS Shared FS Shared FSMDS*

Shared FS with Shared Device

Shared FS Shared FS Shared FS

Shared File Systems – Shared Disk• e.g. SAN FS or Cluster FS• Dedicated Meta Data Server (MDS) optional


IPIP

SANSAN

NAS Appliance

Data

VDISK

Might becomea bottleneck – hard

to scale.

Physical DiskPhysical Disk

11 12 13 1

Disk Array

11 12 13 1


11 12 13 1

11 12 13 1


11 12 13 1

Disk Array

11 12 13 1


11 12 13 1

11 12 13 1


11 12 13 1

Disk Array

11 12 13 1


11 12 13 1

11 12 13 1


11 12 13 1

Disk Array

11 12 13 1


11 12 13 1

11 12 13 1

Application Server Application Server Application Server Application Server Application Server Application Server

e.g. NAS with NFS/CIFS protocol

Distributed File Systems


Why should accessing and scaling file storage be any different in enterprises?

NAS N

AS

NAS Aggregation/Virtualization Global Namespace


FS Virtualization - NAS AggregationIn-Band Solution

NAS Appliance

IPIP

Provides Global

Namespace

Global Namespace

SAN

Data

NAS Appliance

SAN

Data

NAS Appliance

SAN

Data

NAS Appliance

SAN

Data

NAS Appliance

SAN

Data

NAS Appliance


NAS Router

Data


Application Server

IP

StorageDevice

In-Band NAS:

IP

Out-of-Band NAS:

Scalable NAS - Loosely Coupled NAS Cluster

Application Server Application ServerApplication Server Application ServerApplication Server Application ServerApplication Server Application ServerApplication Server Application ServerApplication Server

SAN SAN

StorageDevice

Data

NAS Appliance

Data

NAS Appliancewith NFSv4.1

pNFS extensions

Storage Protocol:SCSI, NFS, OSD

FS Virtualization – NFS4.1 pNFSOut-of-Band Solution

NFSv4.1 client with pNFS

NFSv4.1 client with pNFS

NFSv4.1 client with pNFSNFSv4 client NFSv4 client NFSv4 client


NAS Cluster – NAS Grid (Scalable NAS)aka Tightly Coupled NAS


SANSAN

Single Data ImageGlobal Namespace

VIP

NAS Appliance

Data

NAS Appliance

Data

NAS Appliance

Data

NAS Appliance

Data


Knowledge Workers in the Information AgeIdeas

CommunitiesNetworks Individual

• Information:> capture... > annotate...> store...> share...

making sense ofinformation,

organize ideas,creativity...

awarenesssexposure

conversations,collaboration

establishing andmaintaining

relations


• How to find one thing out of 100 million things?

The New Challenge: Unstructured Data


Knowledge Workers in the Information AgeIdeas

CommunitiesNetworks Individual

• Information:> capture... > annotate...> store...> share...

making sense ofinformation,

organize ideas,creativity...

awarenesssexposure

conversations,collaboration

establishing andmaintaining

relations








DataAccess

Block Object File


SATA

Local FS,Distributed FS:

(NFS, CIFS, WAFS)Shared FS:

(Cluster FS, SAN FS)Global, Distributed

& Parallel FS

OSDCASOSS


Data

IN

F

O R

M

A

T

I

ON


The Old Block ParadigmWrite Block #Write Block #Write Block #Write Block #Write Block #Write Block #Write Block #

StatusReturn 0

Store

Read Block #Read Block #Read Block #Read Block #Read Block #Read Block #Read Block # Data

Retrieve

0 1 2 3 4

5 6 7 8 9

10 11 12 13 14

15 16 17 18 19

0 1 2 3 4

5 6 7 8 9

10 11 12 13 14

15 16 17 18 19


The New Object Paradigm

Data

Object

OID

Store

OID

Object

Data

Retrieve


The New Object Paradigm (cont'd)

Data

Object

OID

Store

• WRITE 26,763 Bytes• QoS = High• Description = “X-Ray”• Retention = 50 years• Access Key = *&^%#• Data Payload........

● Object Storage Responsibilities:● Space Management● Access Control (Identity Mgmt)● QoS Management● Cache, Backup● Policy Migration, Retention


Inodes vs. Objects

Block Device

1 2

5 5

3 4

6 8

iSCSIFile System

Inodes:Block PointersFile Attributes

Object Device

iSCSIFile System

Objects:Block Pointers

Object Attributes

1 2

3

4

56

iSCSIObject IDs

Host Host


Object Autonomy

Block Device

iSCSIFile System


Object Device

iSCSIFile System


Object Attributes

1 2

3

4

56

iSCSIObject IDs

Host HostStorage becomes autonomous:

> capacity planning> load balancing> backup> QoS, SLAs> understand data/object grouping> aggressive pre-fetching> thin provisioning> search> compression/de-duplication,

commonality factoring> strong security, encryption> compliance/retention/secure delete> availability/replication> Audit> self-healing . . .

1 2

5 5

3 4

6 8


Data SharingHomogeneous/Heterogeneous

Object Device

iSCSIFile System


Object Attributes

1 2

3

4

56

iSCSIObject IDs

Host

Block Device

iSCSIFile System


HostiSCSIFile System


HostiSCSIFile SystemiSCSIObject IDs

Host

Cluster/SANfile system

1 2

5 5

3 4

6 8


Additional Layer of Security

Block Device

iSCSI

Object Device

1 2

3

4

56

Host

1 2

5 5

3 4

6 8

Host

• strong security via external service> authentication> authorization > NIS, LDAP....

• fine granularity> per object


0 1 2 3 45 6 7 8 9

10 11 12 13 14

15 16 17 18 19

Object

ObjectObject

Object

Object

Object

ObjectObject

Living in a Flat Namespace

TraditionalHierarchical

Flat

File names / inodes Objects / OIDs


Multiple Referenced Objects

iSCSIOID Y

iSCSICustomer ID

iSCSIPreferences

iSCSINext Store

iSCSIRevenue

iSCSICredit History

ObjectData

iSCSILink to OID X

iSCSIOID Z

iSCSIPatient ID

Metadata Record

iSCSIExamination

iSCSIDate

iSCSIDiagnose

iSCSITherapy

iSCSILink to OID X

Metadata Record Reference 2

Reference 3

iSCSIOID W

iSCSIEmployeee ID

Metadata Record

iSCSIJob Level

iSCSISalary

iSCSIStock Options

iSCSIDepartment

iSCSILink to OID X

Reference 1


0 1 2 3 45 6 7 8 9

10 11 12 13 14

15 16 17 18 19

Virtual View / Virtual File Systems (cont'd)

Traditional Flat Virtual

Virtual View A Virtual View B

Objects / OIDs

File names / inodes

Object

ObjectObject

Object

Object

Object

ObjectObject

Object

ObjectObject

Object

Object

Object

ObjectObject


Object Decomposition

Object Layout

OSD OSD OSD OSD

ObjectComponent

ObjectComponent

ObjectComponent

ObjectComponent

MDSOID:LayoutOID:LayoutOID:LayoutOID:LayoutOID:LayoutOID:LayoutOID:Layout

OID

Layout







SCSI Standards Architecture

Source: www.t10.org


SCSI Commands

Source: www.t10.org


SCSI Transports

Source: www.t10.org


SCSI OSD/OSD-2

Source: www.t10.org


SCSI OSD/OSD-2 over Fibre Channel

Source: www.t10.org


SCSI OSD/OSD-2 over InfiniBand

Source: www.t10.org


SCSI OSD/OSD-2 over Ethernet

Source: www.t10.org


SCSI OSD/OSD-2 over SAS

Source: www.t10.org


ANSI T10 OSD SCSI Targets

Object

ObjectObject

Object

Object

Object

ObjectObject

OSD SCSI Controller OSD SCSI Controller

Disk Array

iSCSIFile SystemiSCSIObject IDs

Host

SAS, FCP, iSCSI,SRP, iSER, SCSI, ...


Host







Object Storage Server - OSS

OSS could be a migration path to provide object technologies to legacy block devices

OS (BSD, Solaris, Linux, Windows..)

Object InterfaceFile System

Block Back-End Device Driver


Host


Distributed Parallel Cluster File SystemWith Object Storage Server (OSS)

IP

Single Data Image

Application ServerApplication Server Application ServerApplication Server Application ServerApplication Server

StorageDevice

StorageDevice

StorageDevice

MDS

Object StorageServer

SAN



Ethernet, IB,Myrinet, Quadrics...


Lustre™ Cluster File System

• The worlds most scalable parallel filesystem

• 10,000’s of clients• Proven technology at major HPC

installations:> Tokyo Tech, TACC (Sun),

LANL, LLNL, Sandia, PNNL, NCSA, etc.

• 70% of Top10 run Lustre• 50% of Top30 run Lustre• 15% of Top500 run Lustre

World’s Largest Network-Neutral Data Storage and Retrieval System


Tier 2 Near-LineArchive

Sun HPC Storage SolutionsCompute Engine

Data CacheLong-Term

Retention and Archive

Tier 1 Archive and Home Directories

Tier 2 FixedContent Archive

ScalableStorage Cluster

Computer Cluster

ArchiveLoadAutomated Migration

Data Movers

LAN and SANIB Network


Scale-Up

Vertical Scaling


http://www.emc.com/digital_universe


The Enterprise Dilemma• While 70% or more of the

digital universe is created, captured, or replicated by individuals — consumers and desk and information workers toiling far away from the datacenter — enterprises, at some point in time, have responsibility or liability for 85%.

• Responsibility includes: security, privacy protection, copyright protection, screening for obscenety, detecting fraud, archiving content, searching and retrieval, disposal...


• Greg Papadopoulos, Chief Technology Officer and Executive Vice President of Research and Development Sun Microsystems

• blogs.sun.com/Gregp/entry/the_world_needs_only_five

„THE WORLD NEEDS ONLY FIVE COMPUTERS“

„Let's see, the Google grid is one. Microsoft's live.com is two. Yahoo!, Amazon.com, eBay, Salesforce.com are three, four, five and six.

(Well, that's O(5) ;))Of course there are many, many more service providers

but they will almost all go the way of YouTube; they'll get eaten by one of the majors.

And, I'm not placing any wagers that any of these six will be one of the Five Computers

(nor that, per the above examples, they are all U.S. West Coast based --- I'll bet at least one,

maybe the largest, will be the Great Computer of China).I'm just saying that there will be, more or less, five hyperscale,

pan-global broadband computing services giants. There will be lots of regional players, of course; mostly,

they will exist to meet national needs.“


Storage in iPods • iPod classic 2007 = 160GB• Moore's law for storage.... • 270.000 iPods first weekend• i.e. Xmas 2011: 270PB iPod storage on a single weekend – 1Gbit wireless each...

> 4 years = 48 month / 18 = 2.66 [(((160 + 100%)+100%)+70%) ~ 1TB

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 20120

100200300400500600700800900

100011001200

Year

GBy

te







Content Addressable Storage - CAS

OS (BSD, Solaris, Linux...)

Proprietary Object API , XAM, WebDAV, NFS, FTP, CIFS...

File systemDatabase

• OIDs are hash values derived from the objects' content• Used as digital archive systems for long term fixed content data• ECM applications used as data injection machines


Data PlacementImproved Resilience

Node Node Node Node

Node Node Node Node

Node Node Node Node

Node Node Node Node

Object 1

= Data

= Parity

Object 2

Object 3


RAINRedundant Array of Inexpensive/Independent Nodes


CAS

File systemDatabase


CAS

File systemDatabase


CAS

File systemDatabase


CAS

File systemDatabase


CAS

File systemDatabase


CAS

File systemDatabase


CAS

File systemDatabase


CAS

File systemDatabase


CAS

File systemDatabase

Single Data Image







CAS: “Content Addressable Storage”

Enterprise Content ManagementInjection Engine

1100010110101111000

Object ID


CAS: “Content Aware Storage”

Enterprise Content ManagementInjection Engine

1100010110101111000

Object ID

Extensible Metadata

Trigger Applications

MD/Content Search







Traditional Storage Architecture (cont'd)

SAN

IPClients

ServerServer CPU& Memory Complex

Controller &128 drives

Up to256 drives

Disk Controller Frame Disk Array FRAME

Logic-Box

(750 mm) (650 mm)

HDD Box

AC-DCPower Supply (25.59 in.)

AC-DCPower Supply


Traditional Storage Architecture (cont'd)

SAN

IPClients

ServerServer CPU& Memory Complex

SCSI Port to OSiSCSITCP/IP

Adapter Driver

HW RAIDController

HW RAIDController


The New Challenges• I/O Performance• Availability • Reliability• Easiness to store, organize, retrieve and dispose• Scale to billions of objects across PB of storage• Horizontal scale (cloud storage, infinite disk, ...)• Complex data operations

> aggregate, join, view, sort, convert, encrypt...• Enhanced search operations (indexing)• Flexibility to present data• Customized storage behavior• Maximize capacity through data reduction technologies (de-dup, classification, ...• Retain objects for the prescribed period, delete expired data securely• Guard against data theft or loss, authentication, authorization• Reduced administration costs• Lengthy storage requirements can exceed normal life of media• Policy managers • Power savings (tape, MAID, ...)• Dynamic metadata: reputation management, access patterns, trending....


Storage

The Vertical Consolidation• Storage and server• Migration of data processing applications• No I/O is best I/O

Server

Storage

Server

Storage

Server

Storage

Server

Storver

The Storage Evolution: From Blocks, Files and Objects to Object Storage Systems

Storagetechnology 2008 04. - 06.06.2008im Radisson SAS Hotel Schloss Fleesenseeveranstaltet durch das Deutsche Zentrum für Luft- und Raumfahrt e.V.

Christian BanduletPrincipal EngineerGlobal Technical Advisory TeamSun Microsystems Inc.

Documents

The Storage Evolution: From Blocks, Files and …konferenz-nz.dlr.de/pages/storage2008/present/2. Konferenztag/06_01... · The Storage Evolution: From Blocks, Files and Objects to