Report: ZFS on Linux Performance Evaluation ... Report: ZFS on Linux Performance Evaluation NorbertSchramm

  • View
    0

  • Download
    0

Embed Size (px)

Text of Report: ZFS on Linux Performance Evaluation ... Report: ZFS on Linux Performance Evaluation...

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Report: ZFS on Linux Performance Evaluation

    Norbert Schramm

    Arbeitsbereich Wissenschaftliches Rechnen Fachbereich Informatik

    Fakultät für Mathematik, Informatik und Naturwissenschaften Universität Hamburg

    2016-03-30

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 1 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Agenda

    1 Introduction

    2 Comparing ZFS

    3 ZFS on Linux

    4 Lustre on ZFS

    5 Comparison

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 2 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Goal of the Project

    ZFS on Linux: Linux-Version of original ZFS by Sun stable-released in April 2013 benefits against traditional file systems supported by Lustre since May 2013 How good is ZFS on Linux Compared to other ZFS-Versions?

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 3 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Tested ZFS on 3 Operating Systems OpenIndiana (based on Illumos, Former Solaris) FreeBSD Linux (Ubuntu)

    Hardware Core-i5 2500k (4x3.3 GHz), 8 GB RAM, 1 TB WD Black Xeon E3110 (2x3.0 GHz), 8 GB RAM, 1 TB WD Black Xeon X5677 (4x3.46 GHz), 32 GB RAM, 136 GB 15K SAS-Drive

    Benchmark bonnie++

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 4 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Figure: Core i5, bonnie++, ZFS initial

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 5 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Ubuntu uses relatime also available on ZFS (initial: disabled)

    Figure: Core i5, atime optimisation

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 6 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    bonnie++ Data Compression Rate: > 130x

    Figure: Core i5, LZ4 Compression

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 7 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Figure: Core i5, Deduplication

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 8 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Figure: Core-i5, Compression % Deduplication

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 9 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Figure: Core i5, Comparison Write

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 10 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Comparing ZFS

    Figure: Comparison 3 Processor Generations with LZ4 Compression Norbert Schramm Report: ZFS on Linux Performance Evaluation 11 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    ZoL Analysis

    recompiled Linux-Kernel with LOCK_STAT Using wrstat for detailed analysis

    lock_stat oprofile (debug kernel needed) /proc file system

    analyzed run of bonnie++

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 12 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Example: ZFS no Compression/Dedup

    Figure: ZFS initial: RAM Usage

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 13 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Example: ZFS no Compression/Dedup

    Figure: ZFS initial: CPU States

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 14 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Example: ZFS no Compression/Dedup

    Figure: ZFS initial: Locks, when ARC is saturated

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 15 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: Preparation

    Test Setup 3 Nodes GBit Uplink to central Switch Xeon X5560 4(8)x2.8 GHz, 12 GB RAM, 2TB HDD for ZFS nehalem1: MGS, MDS, OSS nehalem2, nehalem3: OSS

    Benchmark serial: bonnie++ parallel: ior (with mpi)

    Operating System CentOS 7

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 16 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: bonnie++

    Figure: Bonnie++ Benchmark on Lustre on ZFS-Basis

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 17 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: bonnie++

    Problem: Data transfered uncompressed over the Network

    Figure: Network Limiting Problem

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 18 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: ior

    6x ior (2 of each node) Dedup very good LZ4 fastests, Gzip slowest

    Figure: Ior-Benchmark auf Lustre auf ZFS-BasisNorbert Schramm Report: ZFS on Linux Performance Evaluation 19 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: ior

    Bursty Traffic when ARC is full Compressratio of ior-data: ca. 3.9x

    more work on HDD

    Figure: Network and HDD on nehalem1 while ior-run with ZFS-Compression

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 20 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Lustre on ZFS: ior

    slow write performance thus parallel access txg_sync while no data is transfered

    ZIL is located on disk, slows disk down

    Figure: Network and HDD on nehalem1 while ior-run without ZFS-Compression

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 21 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    ZoL is comparable to other versions (functionality, performance) actual Version: 0.6.5.6 Version 1.0 when zvols are implementet Uses a lot of Locks while caching, can possibly slow down separate Device for L2ARC and SLOG can improve performance Lustre can get benefits from ZFS (e.g. Compression)

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 22 / 23

  • Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison

    Thank you for your attention. Questions?

    Norbert Schramm Report: ZFS on Linux Performance Evaluation 23 / 23

    Introduction Comparing ZFS ZFS on Linux Lustre on ZFS Comparison