Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24

Viele Server – wenig Arbeit

Berlin | 07.06.2013 | Schlomo Schapiro | @SchlomoSchapiroSystemarchitekt, Open Source Evangelist

License: http://creativecommons.org/licenses/by-nc-nd/3.0/

Betriebsautomation bei ImmobilienScout24

http://creativecommons.org/licenses/by-nc-nd/3.0/

Slide 3 | Betriebsautomation bei ImmobilienScout24 | Schlomo Schapiro

www.ImmobilienScout24.de

>2 billion PI per month

2 data center with ~1400 VM

total of ~600 employees

~30 crossfunctional IT teams

~160 in IT

15 years in business

part of Deutsche Telekom


Papa,was machst Du eigentlich auf Arbeit?







Lego City


Orchestrierung & Automation


Spielen die wirklich nur mit LEGO rum?


BusinessDecision

To goLive

All Humans are on the Same Side

BuildConfigDeployTestSystems-Management

Automation

Scale OutData Centers

through packages


PROInfrastructureBuild

ApplicationBuild

ConfigurationBuild

QAYUM

Repos

Interface

DEV

github.com/ImmobilienScout24/yum-repo-server

http://github.com/ImmobilienScout24/yum-repo-server


“Any relevant file should be either

deployed via a package

or

completely managed by an

application that is thus deployed.”


„Every package must be verifyable –

stay away from package scripts.“

“Reducing the config package

reduces the deployment risk.”


ConfigSVN

post-commit → YUM repos

Infrastructureas

Code


TSTWEB05Location & Environment

Function Group

Instance

Configuration over Convention


Big Picture – Static Structure

ConfigSVN

YUMRepository

IS24 softwareAutomated RPM

creation

config├── host│ └── tstweb02│├── loc│ └── tst│ ├── VARIABLES│ │ ├── RPM_REQUIRES│ │ ├── RPM_PROVIDES│ │ ├── DB_HOST│ │ └── DB_USER│ └── etc│ └── is24│ ├── web.properties│ └── db.properties├── typ│ └── web│├── loctyp│ └── tstweb│ └── etc│ └── is24│ └── web.properties└── all ├── VARIABLES │ └── SYSLOG_HOST └── etc └── is24 └── system.properties

is24-config-tstweb01-1.0-$rev.rpm:/etc/is24/system.properties/etc/is24/db.properties/etc/is24/web.properties

[root@tstweb01 ~]# yum update

svn ci

db.host=@@@DB_HOST@@@db.user=@@@DB_USER@@@db.port=3306

Post-commit hook createsis24-config-$hostname RPM● svn export● patch VARIABLES● fill in metadata

svn co

loghost=log.domain.com


Static Structure – Pros and Cons

Variables follow samestructure as config data

Only one tool (SVN) needed

GUI talks only to SVN

Versioning and changetracking for variables

Variables and config fileschange together (atomic)

easy rollback→

Is this simple overlaying structure sufficient?

Simplify the world so that it is good enough!

ConfigSVN

YUMRepository

IS24 softwareAutomated RPM

creation

config├── host│ └── tstweb02│├── loc│ └── tst│ ├── VARIABLES│ │ ├── RPM_REQUIRES│ │ ├── RPM_PROVIDES│ │ ├── DB_HOST│ │ └── DB_USER│ └── etc│ └── is24│ ├── web.properties│ └── db.properties├── typ│ └── web│├── loctyp│ └── tstweb│ └── etc│ └── is24│ └── web.properties└── all ├── VARIABLES │ └── SYSLOG_HOST └── etc └── is24 └── system.properties

is24-config-tstweb01-1.0-$rev.rpm:/etc/is24/system.properties/etc/is24/db.properties/etc/is24/web.properties

[root@tstweb01 ~]# yum update

svn ci

db.host=@@@DB_HOST@@@db.user=@@@DB_USER@@@db.port=3306

Post-commit hook createsis24-config-$hostname RPM● svn export● patch VARIABLES● fill in metadata

svn co

loghost=log.domain.com

yadt-config-rpm-maker










File Management with RPM & YUM

PROD

DEV

InfrastructureBuild

ApplicationBuild

ConfigurationBuild

QA

YUMRepos


Example: is24-tomcat Packages

is24-tomcat-welcome = 6.0.35

is24-tomcat-standalone is24-tomcat-behind-httpd

is24-tomcat-base

is24-tomcat = 6.0.35

XOR

Req

Req Req

Req

is24-tomcat-welcome = 7.0.40

is24-tomcat = 7.0.40


Example: Apache HTTPD

Design Goals:

Use and extend upstream httpd RPM

Configure MPM and service user per application

IS24 standard configuration everywhere

httpd RPM

/etc/httpd/conf/httpd.conf

/etc/sysconfig/httpd


Example: is24-httpd RPM

if ! echo '# IS24 HTTPD conf framework. Read IS24_README!# This file is managed by %{name}%{version}.%{release}# Put your stuff in /etc/conf/*/is24*.conf files!ServerRoot "/etc/httpd"Include conf/basic/is24*.confInclude conf/main/is24*.confInclude conf/other/is24*.conf' >/etc/httpd/conf/httpd.conf ; then logger p user.err s t %name "ERROR: …"fi

Requires: httpdRequires(Pre): httpd%post


Example: is24-httpd RPM

if ! echo '# HTTPD options can be configured in# additional /etc/httpd/conf/env/*.sh files# This file is managed by %{name}%{version}.%{release}# Please add extra options to the OPTIONS Bash Arrayshopt s nullglobHTTPD=/usr/sbin/httpd.workerfor f in /etc/httpd/conf/env/*.sh ; do source $fdoneOPTIONS="${OPTIONS[*]}" # flatten array' >/etc/sysconfig/httpd ; then logger p user.err s t %name "ERROR: ..."fi

...

%post (continued)


Systems Management

yadtshell


YADT – Unified Dependency Tree

Web Server

HTTPD

Tomcat

App Server

Load Balancer


HTTPD

Tomcat

web-app

httpd

config

tomcat

YADT – Unified Dependency Tree

initscript

initscript

/etc/yadt.services- httpd: is_frontservice: true needs_services: - tomcat- tomcat:


YADT – Large Scale System Management

Web Server

HTTPD

Tomcatweb-app

httpd

config

tomcat

App Server

Java App

app

config



LB: www.immobilienscout24.de

Web Server

HTTPD

Tomcatweb-app

httpd

config

tomcat

App Server

Java App

app

config

Web Server

HTTPD

Tomcatweb-app

httpd

config

tomcat

App Server

Java App

app

config




Web Server

HTTPD

Tomcatweb-app

httpd

config

tomcat

App Server

Java App

app

config

Web Server

HTTPD

Tomcatweb-app

httpd

config

tomcat

App Server

Java App

app

config

1st

chun

k




Web Server

HTTPD

Tomcatweb-app

httpd

config

tomcat

App Server

Java App

app

config

Web Server

HTTPD

Tomcatweb-app

httpd

config

tomcat

App Server

Java App

app

config

1st

chun

k

2nd chunk


yadtshell Wave Deployment


Systems Management with YADT

Reduce Systemic Complexity

Keep It Simple


Continuous Delivery Platform

BusinessDecisionTo GoLive

All Humans are on the Same Side

BuildConfigDeployTestSystems-Management

Automation

Scale OutData Centers


DevOps


Abstraktion

Starke Vereinfachung

Standardbausteine

Lösungen wiederverwenden

Kleine Komponenten zu Großem verbinden

Alles ist machbar


www.yadt-project.org

go.schapiro.org/slides

http://www.yadt-project.org/

http://go.schapiro.org/slides

Slide 47 | Betriebsautomation bei ImmobilienScout24 | Schlomo Schapirobit.ly/is24techjobs

http://bit.ly/is24techjobs


Kontakt:Immobilien Scout GmbHAndreasstraße 1010243 Berlin

Fon: +49 30 243 01-1229 Email: [email protected]: www.immobilienscout24.de

Thank you very much!Please contact me for further questions and discussions.


Photo URLs:http://skepchick.org/wp-content/uploads/lego-1.jpghttp://smithduck.home.comcast.net/~smithduck/bmw/kwiring/Early%20K75%20K100/K100_Early_Wiring_Diagram.jpeghttp://brickbrowse.com/creator/small-cottage-31009/http://static.desktopnexus.com/owallpaper/15044-1024x768-4097LinuxSkeleton1024768.jpg (John Schriener)http://images.brickbrowse.com/technic/motorized-excavator-8043/technic-motorized-excavator-8043-3.jpghttp://images.brickbrowse.com/technic/container-truck-8052/technic-container-truck-8052-3.jpghttp://www.flickr.com/photos/dhardacre/5754596620/http://images.brickbrowse.com/architecture/brandenburg-gate-21011/architecture-brandenburg-gate-21011-1.jpghttp://www.gtaberlin.de/models/brandenburger_tor_1.jpg

All remaining images are custom-made or from http://openclipart.org

http://skepchick.org/wp-content/uploads/lego-1.jpg

http://smithduck.home.comcast.net/~smithduck/bmw/kwiring/Early%20K75%20K100/K100_Early_Wiring_Diagram.jpeg

http://brickbrowse.com/creator/small-cottage-31009/

http://static.desktopnexus.com/owallpaper/15044-1024x768-4097LinuxSkeleton1024768.jpg

http://images.brickbrowse.com/technic/motorized-excavator-8043/technic-motorized-excavator-8043-3.jpg

http://images.brickbrowse.com/technic/container-truck-8052/technic-container-truck-8052-3.jpg

http://www.flickr.com/photos/dhardacre/5754596620/

http://images.brickbrowse.com/architecture/brandenburg-gate-21011/architecture-brandenburg-gate-21011-1.jpg

http://www.gtaberlin.de/models/brandenburger_tor_1.jpg

http://openclipart.org/

Technology

Viele Server - Wenig Arbeit: Betriebsautomation bei ImmobilienScout24