Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24

Viele Server – wenig Arbeit

Berlin | 07.06.2013 | Schlomo Schapiro | @SchlomoSchapiroSystemarchitekt, Open Source Evangelist

License: http://creativecommons.org/licenses/by-nc-nd/3.0/

Betriebsautomation bei ImmobilienScout24

| Betriebsautomation bei ImmobilienScout24 | Schlomo Schapiro

www.ImmobilienScout24.de

>2 billion PI per month

2 data center with ~1400 VM

total of ~600 employees

~30 crossfunctional IT teams

~160 in IT

15 years in business

part of Deutsche Telekom

Papa,was machst Du eigentlich auf Arbeit?

Lego City

Orchestrierung & Automation

Spielen die wirklich nur mit LEGO rum?

BusinessDecision

To goLive

All Humans are on the Same Side

BuildConfigDeployTestSystems-Management

Automation

Scale OutData Centers

through packages

PROInfrastructureBuild

ApplicationBuild

ConfigurationBuild

Interface

github.com/ImmobilienScout24/yum-repo-server

“Any relevant file should be either

deployed via a package

completely managed by an

application that is thus deployed.”

„Every package must be verifyable –

stay away from package scripts.“

“Reducing the config package

reduces the deployment risk.”

ConfigSVN

post-commit → YUM repos

Infrastructureas

TSTWEB05Location & Environment

Function Group

Instance

Configuration over Convention

Big Picture – Static Structure

ConfigSVN

YUMRepository

IS24 softwareAutomated RPM

creation

config├── host│ └── tstweb02│├── loc│ └── tst│ ├── VARIABLES│ │ ├── RPM_REQUIRES│ │ ├── RPM_PROVIDES│ │ ├── DB_HOST│ │ └── DB_USER│ └── etc│ └── is24│ ├── web.properties│ └── db.properties├── typ│ └── web│├── loctyp│ └── tstweb│ └── etc│ └── is24│ └── web.properties└── all ├── VARIABLES │ └── SYSLOG_HOST └── etc └── is24 └── system.properties

is24-config-tstweb01-1.0-$rev.rpm:/etc/is24/system.properties/etc/is24/db.properties/etc/is24/web.properties

[root@tstweb01 ~]# yum update

svn ci

db.host=@@@DB_HOST@@@db.user=@@@DB_USER@@@db.port=3306

Post-commit hook createsis24-config-$hostname RPM● svn export● patch VARIABLES● fill in metadata

svn co

loghost=log.domain.com

Static Structure – Pros and Cons

Variables follow samestructure as config data

Only one tool (SVN) needed

GUI talks only to SVN

Versioning and changetracking for variables

Variables and config fileschange together (atomic)

easy rollback→

Is this simple overlaying structure sufficient?

Simplify the world so that it is good enough!

ConfigSVN

YUMRepository

IS24 softwareAutomated RPM

creation

config├── host│ └── tstweb02│├── loc│ └── tst│ ├── VARIABLES│ │ ├── RPM_REQUIRES│ │ ├── RPM_PROVIDES│ │ ├── DB_HOST│ │ └── DB_USER│ └── etc│ └── is24│ ├── web.properties│ └── db.properties├── typ│ └── web│├── loctyp│ └── tstweb│ └── etc│ └── is24│ └── web.properties└── all ├── VARIABLES │ └── SYSLOG_HOST └── etc └── is24 └── system.properties

is24-config-tstweb01-1.0-$rev.rpm:/etc/is24/system.properties/etc/is24/db.properties/etc/is24/web.properties

[root@tstweb01 ~]# yum update

svn ci

db.host=@@@DB_HOST@@@db.user=@@@DB_USER@@@db.port=3306

Post-commit hook createsis24-config-$hostname RPM● svn export● patch VARIABLES● fill in metadata

svn co

loghost=log.domain.com

yadt-config-rpm-maker

File Management with RPM & YUM

InfrastructureBuild

ApplicationBuild

ConfigurationBuild

YUMRepos

Example: is24-tomcat Packages

is24-tomcat-welcome = 6.0.35

is24-tomcat-standalone is24-tomcat-behind-httpd

is24-tomcat-base

is24-tomcat = 6.0.35

Req Req

is24-tomcat-welcome = 7.0.40

is24-tomcat = 7.0.40

Example: Apache HTTPD

Design Goals:

Use and extend upstream httpd RPM

Configure MPM and service user per application

IS24 standard configuration everywhere

httpd RPM

/etc/httpd/conf/httpd.conf

/etc/sysconfig/httpd

Example: is24-httpd RPM

if ! echo '# IS24 HTTPD conf framework. Read IS24_README!# This file is managed by %{name}%{version}.%{release}# Put your stuff in /etc/conf/*/is24*.conf files!ServerRoot "/etc/httpd"Include conf/basic/is24*.confInclude conf/main/is24*.confInclude conf/other/is24*.conf' >/etc/httpd/conf/httpd.conf ; then logger p user.err s t %name "ERROR: …"fi

Requires: httpdRequires(Pre): httpd%post

Example: is24-httpd RPM

if ! echo '# HTTPD options can be configured in# additional /etc/httpd/conf/env/*.sh files# This file is managed by %{name}%{version}.%{release}# Please add extra options to the OPTIONS Bash Arrayshopt s nullglobHTTPD=/usr/sbin/httpd.workerfor f in /etc/httpd/conf/env/*.sh ; do source $fdoneOPTIONS="${OPTIONS[*]}" # flatten array' >/etc/sysconfig/httpd ; then logger p user.err s t %name "ERROR: ..."fi

%post (continued)

Systems Management

yadtshell

YADT – Unified Dependency Tree

Web Server

Tomcat

App Server

Load Balancer

Tomcat

web-app

config

tomcat

YADT – Unified Dependency Tree

initscript

/etc/yadt.services- httpd: is_frontservice: true needs_services: - tomcat- tomcat:

YADT – Large Scale System Management

Web Server

Tomcatweb-app

config

tomcat

App Server

Java App

config

LB: www.immobilienscout24.de

Web Server

Tomcatweb-app

config

tomcat

App Server

Java App

config

Web Server

Tomcatweb-app

config

tomcat

App Server

Java App

config

Web Server

Tomcatweb-app

config

tomcat

App Server

Java App

config

Web Server

Tomcatweb-app

config

tomcat

App Server

Java App

config

Web Server

Tomcatweb-app

config

tomcat

App Server

Java App

config

Web Server

Tomcatweb-app

config

tomcat

App Server

Java App

config

2nd chunk

yadtshell Wave Deployment

Systems Management with YADT

Reduce Systemic Complexity

Keep It Simple

Continuous Delivery Platform

BusinessDecisionTo GoLive

All Humans are on the Same Side

BuildConfigDeployTestSystems-Management

Automation

Scale OutData Centers

DevOps

Abstraktion

Starke Vereinfachung

Standardbausteine

Lösungen wiederverwenden

Kleine Komponenten zu Großem verbinden

Alles ist machbar

www.yadt-project.org

go.schapiro.org/slides

| Betriebsautomation bei ImmobilienScout24 | Schlomo Schapirobit.ly/is24techjobs

Kontakt:Immobilien Scout GmbHAndreasstraße 1010243 Berlin

Fon: +49 30 243 01-1229 Email: schlomo.schapiro@immobilienscout24.deURL: www.immobilienscout24.de

Thank you very much!Please contact me for further questions and discussions.

Photo URLs:http://skepchick.org/wp-content/uploads/lego-1.jpghttp://smithduck.home.comcast.net/~smithduck/bmw/kwiring/Early%20K75%20K100/K100_Early_Wiring_Diagram.jpeghttp://brickbrowse.com/creator/small-cottage-31009/http://static.desktopnexus.com/owallpaper/15044-1024x768-4097LinuxSkeleton1024768.jpg (John Schriener)http://images.brickbrowse.com/technic/motorized-excavator-8043/technic-motorized-excavator-8043-3.jpghttp://images.brickbrowse.com/technic/container-truck-8052/technic-container-truck-8052-3.jpghttp://www.flickr.com/photos/dhardacre/5754596620/http://images.brickbrowse.com/architecture/brandenburg-gate-21011/architecture-brandenburg-gate-21011-1.jpghttp://www.gtaberlin.de/models/brandenburger_tor_1.jpg

All remaining images are custom-made or from http://openclipart.org

Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24

Technology

Verzeichnisbasiertes Benutzer- und … · Proxy (Squid) Fax-Server (GOfax) FTP-Server (ProFTP) ThinClients (GOto) DHCP-Server DNS-Server ... GOsa ist Freie Software von GONICUS LDAP

Thomas Hirt FileMaker Server in Windows Server Umgebungen FileMaker Konferenz 2014 Winterthur FileMaker Server in Windows Server

Informationsvorlage zum Versand · tal/Feri, des Dynamik- und Niveaurankings der Initiative Neue Soziale Marktwirtschaft und IW Con-sult/Immobilienscout24, des Nachhaltigkeitsrankings

Exchange Server 2010 Installation und Konfiguration ... · PDF fileExchange Server 2010 Installation und Konfiguration Systemvoraussetzungen Windows Server 2008 oder Windows Server

Live-Demo: ionas-Server Home, ionas-Server Small Business

Seminarvortrag - Hinweise zur Präsentation · - Example: compute server with minimal load server server server client central manager service layer Central manager - acts as a "trader"

Bürofläche im Usingen Gründerzentrum · DSC_0293 Powered by ImmobilienScout24. Bürofläche im Usingen Gründerzentrum Raiffeisenstraße 10 61250 Usingen Miet-/Kaufobjekt: Miete

Monitoring Microsoft SQL Server - netways.de · Software / SQL Server / Performancecounter

Dns server & dhcp server

Modulare Oberstufe am BRG9 - Kurzinfo 2008. Ausgangspunkt: SchülerInnenkritik: -> zu wenig Wahlmöglichkeiten -> zu wenig Aktuelles -> zu starr -> zu wenig

Ein wenig malen, ein wenig animieren – aber ganz viel GIMP · Ein wenig malen, ein wenig animieren – aber ganz viel GIMP wir basteln eine Weihnachtskarte mit GIMP Dieses Tutorial

ImmobilienScout24 beschleunigt Geschäfts- prozesse mit ... · ImmobilienScout24, die Nr. 1 rund um Immobilien, setzt bereits seit 2010 auf die Salesforce . ... Computer-Telefonie-Integration

Keischrift Wenig Gewicht

100 Tage Bestellerprinzip Eine Online-Befragung unter 1.307 Maklern & gewerblichen Vermietern von ImmobilienScout24, der Immobilien Zeitung und IMMO MEDIA

Exchange Server 2003 - MSXFORUMMicrosoft Exchange Server User Monitor Kategorie : Exchange Server 2003 Veröffentlicht von webmaster am 20.04.2005 Der Microsoft Exchange Server User

ISA Server 2004 – ISA Server 2004 Zertifikatmanagement ... · ISA Server 2004 – ISA Server 2004 Zertifikatmanagement - Von Marc Grote----- Die

Server - Procube · 2012. 4. 18. · Server Server. Created Date: 4/18/2012 10:57:09 AM

FileMaker Server 13 · 2013. 11. 21. · Über FileMaker Server FILEMAKER SERVER 13 HILFE 8 Über FileMaker Server FileMaker® Server ist ein dedizierter Datenbank-Server, der FileMaker

Virtualisierungslösungen für VoIP Application Server zur ... · Quelle: 3GPP TS 23.203: ... CDR Processor Status Server Proxy Server Registry / ... 19 ... SSOA 0 CONFIG SERVER SIP

Kompetenzen eines Agile Coaches warum ImmobilienScout24 ... Kompeten… · Kompetenzen eines Agile Coaches ... • Einführung von Scrum • Scrum Master Kristina Müller Automobilkauffrau