of 23 /23
Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison Report: ZFS on Linux Performance Evaluation Norbert Schramm Arbeitsbereich Wissenschaftliches Rechnen Fachbereich Informatik Fakultät für Mathematik, Informatik und Naturwissenschaften Universität Hamburg 2016-03-30 Norbert Schramm Report: ZFS on Linux Performance Evaluation 1 / 23

Report: ZFS on Linux Performance Evaluation...Report: ZFS on Linux Performance Evaluation NorbertSchramm Arbeitsbereich Wissenschaftliches Rechnen Fachbereich Informatik Fakultät

  • Author
    others

  • View
    21

  • Download
    0

Embed Size (px)

Text of Report: ZFS on Linux Performance Evaluation...Report: ZFS on Linux Performance Evaluation...

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Report: ZFS on Linux Performance Evaluation

    Norbert Schramm

    Arbeitsbereich Wissenschaftliches RechnenFachbereich Informatik

    Fakultät für Mathematik, Informatik und NaturwissenschaftenUniversität Hamburg

    2016-03-30

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 1 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Agenda

    1 Introduction

    2 Comparing ZFS

    3 ZFS on Linux

    4 Lustre on ZFS

    5 Comparison

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 2 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Goal of the Project

    ZFS on Linux: Linux-Version of original ZFS by Sunstable-released in April 2013benefits against traditional file systemssupported by Lustre since May 2013How good is ZFS on Linux Compared to other ZFS-Versions?

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 3 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Tested ZFS on 3 Operating SystemsOpenIndiana (based on Illumos, Former Solaris)FreeBSDLinux (Ubuntu)

    HardwareCore-i5 2500k (4x3.3 GHz), 8 GB RAM, 1 TB WD BlackXeon E3110 (2x3.0 GHz), 8 GB RAM, 1 TB WD BlackXeon X5677 (4x3.46 GHz), 32 GB RAM, 136 GB 15KSAS-Drive

    Benchmarkbonnie++

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 4 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Figure: Core i5, bonnie++, ZFS initial

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 5 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Ubuntu uses relatimealso available on ZFS (initial: disabled)

    Figure: Core i5, atime optimisation

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 6 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    bonnie++ Data Compression Rate: > 130x

    Figure: Core i5, LZ4 Compression

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 7 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Figure: Core i5, Deduplication

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 8 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Figure: Core-i5, Compression % Deduplication

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 9 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Figure: Core i5, Comparison Write

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 10 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Figure: Comparison 3 Processor Generations with LZ4 CompressionNorbert Schramm Report: ZFS on Linux Performance Evaluation 11 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    ZoL Analysis

    recompiled Linux-Kernel with LOCK_STATUsing wrstat for detailed analysis

    lock_statoprofile (debug kernel needed)/proc file system

    analyzed run of bonnie++

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 12 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Example: ZFS no Compression/Dedup

    Figure: ZFS initial: RAM Usage

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 13 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Example: ZFS no Compression/Dedup

    Figure: ZFS initial: CPU States

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 14 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Example: ZFS no Compression/Dedup

    Figure: ZFS initial: Locks, when ARC is saturated

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 15 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: Preparation

    Test Setup3 NodesGBit Uplink to central SwitchXeon X5560 4(8)x2.8 GHz, 12 GB RAM, 2TB HDD for ZFSnehalem1: MGS, MDS, OSSnehalem2, nehalem3: OSS

    Benchmarkserial: bonnie++parallel: ior (with mpi)

    Operating SystemCentOS 7

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 16 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: bonnie++

    Figure: Bonnie++ Benchmark on Lustre on ZFS-Basis

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 17 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: bonnie++

    Problem: Data transfered uncompressed over the Network

    Figure: Network Limiting Problem

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 18 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: ior

    6x ior (2 of each node)Dedup very goodLZ4 fastests, Gzip slowest

    Figure: Ior-Benchmark auf Lustre auf ZFS-BasisNorbert Schramm Report: ZFS on Linux Performance Evaluation 19 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: ior

    Bursty Traffic when ARC is fullCompressratio of ior-data: ca. 3.9x

    more work on HDD

    Figure: Network and HDD on nehalem1 while ior-run withZFS-Compression

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 20 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: ior

    slow write performance thus parallel accesstxg_sync while no data is transfered

    ZIL is located on disk, slows disk down

    Figure: Network and HDD on nehalem1 while ior-run withoutZFS-Compression

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 21 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    ZoL is comparable to other versions (functionality,performance)actual Version: 0.6.5.6Version 1.0 when zvols are implementetUses a lot of Locks while caching, can possibly slow downseparate Device for L2ARC and SLOG can improve performanceLustre can get benefits from ZFS (e.g. Compression)

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 22 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Thank you for your attention.Questions?

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 23 / 23

    IntroductionComparing ZFSZFS on LinuxLustre on ZFSComparison