Kubernetes: Wie Chefkoch.de mit Containern arbeitet

Wie Chefkoch.demit Containernarbeitetein Vortrag von Per Bernhardt

Mein Name ist Perhttp://perprogramming.de

Ich bin ein Chefkochhttp://www.chefkoch.de

Agenda

Agenda1. Container? WTF?

Agenda1. Container? WTF?

2. Kubernetes

Agenda1. Container? WTF?

2. Kubernetes3. Herausforderungen

Agenda1. Container? WTF?

2. Kubernetes3. Herausforderungen

4. Fazit

Agenda1. Container? WTF?

2. Kubernetes3. Herausforderungen

5. Fragen??

4. Fazit

Container? WTF?

ALT

NEU

ALT

Deploy Deploy DeployDeploy

Deploy Deploy DeployDeploy

NEU

ALTOps Dev

NEUOps Dev

NEUOps Dev

?

Orchestrierung!

Helios

Apache

Helios

Apache

Helios

Apache

Helios

Apache

Helios

Apache

Helios

ECS

Apache

Helios

ECS

Apache

Helios

ECS…

Kubernetes

Cluster

core@core-01 ~ $ kubectl --server=https://core01.fra.chefkoch.net:6443 get nodesNAME LABELS STATUS10.10.0.100 kubernetes.io/hostname=10.10.0.100,name=core01.cgn Ready10.10.0.101 kubernetes.io/hostname=10.10.0.101,name=core01.fra Ready10.10.0.102 kubernetes.io/hostname=10.10.0.102,name=core02.fra Ready10.10.0.104 kubernetes.io/hostname=10.10.0.104,name=core03.fra Ready10.10.0.105 kubernetes.io/hostname=10.10.0.105,name=bigdata02.fra Ready10.10.0.106 kubernetes.io/hostname=10.10.0.106,name=bigdata01.cgn Ready10.10.0.107 kubernetes.io/hostname=10.10.0.107,name=bigdata01.fra Ready10.10.0.108 kubernetes.io/hostname=10.10.0.108,name=core06.fra Ready10.10.0.109 kubernetes.io/hostname=10.10.0.109,name=core05.fra Ready10.10.0.110 kubernetes.io/hostname=10.10.0.110,name=core04.fra Ready

Namespaces

core@core-01 ~ $ kubectl get namespacesNAME LABELS STATUSdefault <none> Active

core@core-01 ~ $ kubectl get namespacesNAME LABELS STATUSdefault <none> Active

core@core-01 ~ $ kubectl --namespace=default get allCONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICASNAME LABELS SELECTOR IP(S) PORT(S)NAME READY STATUS RESTARTS AGENAME LABELS STATUS VOLUME

core@core-01 ~ $ kubectl config viewapiVersion: v1kind: Configclusters:- name: production cluster: server: https://core01.fra.chefkoch.net:6443contexts:- name: foobar.production context: cluster: production namespace: foobar user: john.doeusers:- name: john.doe user: username: john.doe password: p4ssw0rdcurrent-context: ""

core@core-01 ~ $ kubectl config viewapiVersion: v1kind: Configclusters:- name: production cluster: server: https://core01.fra.chefkoch.net:6443contexts:- name: foobar.production context: cluster: production namespace: foobar user: john.doeusers:- name: john.doe user: username: john.doe password: p4ssw0rdcurrent-context: ""

core@core-01 ~ $ kubectl --context=foobar.production get all

core@core-01 ~ $ kubectl config viewapiVersion: v1kind: Configclusters:- name: production cluster: server: https://core01.fra.chefkoch.net:6443contexts:- name: foobar.production context: cluster: production namespace: foobar user: john.doeusers:- name: john.doe user: username: john.doe password: p4ssw0rdcurrent-context: ""

core@core-01 ~ $ kubectl --context=foobar.production get allcore@core-01 ~ $ kubectl --cluster=production get all

core@core-01 ~ $ kubectl config viewapiVersion: v1kind: Configclusters:- name: production cluster: server: https://core01.fra.chefkoch.net:6443contexts:- name: foobar.production context: cluster: production namespace: foobar user: john.doeusers:- name: john.doe user: username: john.doe password: p4ssw0rdcurrent-context: ""

core@core-01 ~ $ kubectl --context=foobar.production get allcore@core-01 ~ $ kubectl --cluster=production get allcore@core-01 ~ $ kubectl --username=john.doe --password=p4ssw0rd get all

core@core-01 ~ $ kubectl config viewapiVersion: v1kind: Configclusters:- name: production cluster: server: https://core01.fra.chefkoch.net:6443contexts:- name: foobar.production context: cluster: production namespace: foobar user: john.doeusers:- name: john.doe user: username: john.doe password: p4ssw0rdcurrent-context: ""

core@core-01 ~ $ kubectl config use-context foobar.production

core@core-01 ~ $ kubectl --context=foobar.production get allcore@core-01 ~ $ kubectl --cluster=production get allcore@core-01 ~ $ kubectl --username=john.doe --password=p4ssw0rd get all

Pods

apiVersion: v1kind: Podmetadata: name: my-symfony-app labels: name: my-symfony-appspec: containers: - name: nginx image: my-symfony-app command: [nginx] volumeMounts: - {name: socket, mountPath: /run} - name: php-fpm image: my-symfony-app command: [php5-fpm] volumeMounts: - {name: socket, mountPath: /run} volumes: - {name: socket, emptyDir: {medium: Memory}}

my-symfony-app.yml

core@core-01 ~ $ kubectl create -f my-symfony-app.yml

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app 2/2 Running 0 2s

core@core-01 ~ $ kubectl create -f my-symfony-app.yml

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app 2/2 Running 0 2s

core@core-01 ~ $ kubectl create -f my-symfony-app.yml

core@core-01 ~ $ kubectl logs my-symfony-app -c php-fpm[15-Oct-2015 15:29:45] NOTICE: fpm is running, pid 1[15-Oct-2015 15:29:45] NOTICE: ready to handle connections

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app 2/2 Running 0 2s

core@core-01 ~ $ kubectl create -f my-symfony-app.yml

core@core-01 ~ $ kubectl delete pod my-symfony-app

core@core-01 ~ $ kubectl logs my-symfony-app -c php-fpm[15-Oct-2015 15:29:45] NOTICE: fpm is running, pid 1[15-Oct-2015 15:29:45] NOTICE: ready to handle connections

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app 2/2 Running 0 2s

core@core-01 ~ $ kubectl create -f my-symfony-app.yml

core@core-01 ~ $ kubectl delete pod my-symfony-app

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGE

core@core-01 ~ $ kubectl logs my-symfony-app -c php-fpm[15-Oct-2015 15:29:45] NOTICE: fpm is running, pid 1[15-Oct-2015 15:29:45] NOTICE: ready to handle connections

Features, Features, Features!

Features, Features, Features!• Readiness-Probe, Liveness-Probe

Features, Features, Features!• Readiness-Probe, Liveness-Probe• Livecycle-Hooks

Features, Features, Features!• Readiness-Probe, Liveness-Probe• Livecycle-Hooks• ImagePull-Policy

Features, Features, Features!• Readiness-Probe, Liveness-Probe• Livecycle-Hooks• ImagePull-Policy• Ressourcen-Limits

Features, Features, Features!• Readiness-Probe, Liveness-Probe• Livecycle-Hooks• ImagePull-Policy• Ressourcen-Limits• Verschiedene Mounts

Features, Features, Features!• Readiness-Probe, Liveness-Probe• Livecycle-Hooks• ImagePull-Policy• Ressourcen-Limits• Verschiedene Mounts

• GCE

Features, Features, Features!• Readiness-Probe, Liveness-Probe• Livecycle-Hooks• ImagePull-Policy• Ressourcen-Limits• Verschiedene Mounts

• GCE• AWS EBS

Features, Features, Features!• Readiness-Probe, Liveness-Probe• Livecycle-Hooks• ImagePull-Policy• Ressourcen-Limits• Verschiedene Mounts

• GCE• AWS EBS• iSCSI

Features, Features, Features!• Readiness-Probe, Liveness-Probe• Livecycle-Hooks• ImagePull-Policy• Ressourcen-Limits• Verschiedene Mounts

• GCE• AWS EBS• iSCSI• NFS

Features, Features, Features!• Readiness-Probe, Liveness-Probe• Livecycle-Hooks• ImagePull-Policy• Ressourcen-Limits• Verschiedene Mounts

• GCE• AWS EBS• iSCSI• NFS• Glusterfs

Features, Features, Features!• Readiness-Probe, Liveness-Probe• Livecycle-Hooks• ImagePull-Policy• Ressourcen-Limits• Verschiedene Mounts

• GCE• AWS EBS• iSCSI• NFS• Glusterfs• Git-Repo

ReplicationController

apiVersion: v1kind: ReplicationControllermetadata: name: my-symfony-appspec: selector: name: my-symfony-app replicas: 5 template: // Inhalt von my-symfony-app.yml

my-symfony-app-rc.yml

core@core-01 ~ $ kubectl create -f my-symfony-app-rc.yml

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app-jzd97 2/2 Running 0 2smy-symfony-app-193aw 2/2 Running 0 2s my-symfony-app-bicex 2/2 Running 0 2s my-symfony-app-r2in0 2/2 Running 0 2s my-symfony-app-lbo54 2/2 Running 0 2s

core@core-01 ~ $ kubectl create -f my-symfony-app-rc.yml

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app-jzd97 2/2 Running 0 2smy-symfony-app-193aw 2/2 Running 0 2s my-symfony-app-bicex 2/2 Running 0 2s my-symfony-app-r2in0 2/2 Running 0 2s my-symfony-app-lbo54 2/2 Running 0 2s

core@core-01 ~ $ kubectl create -f my-symfony-app-rc.yml

core@core-01 ~ $ kubectl delete pods --all

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app-jzd97 2/2 Running 0 2smy-symfony-app-193aw 2/2 Running 0 2s my-symfony-app-bicex 2/2 Running 0 2s my-symfony-app-r2in0 2/2 Running 0 2s my-symfony-app-lbo54 2/2 Running 0 2s

core@core-01 ~ $ kubectl create -f my-symfony-app-rc.yml

core@core-01 ~ $ kubectl delete pods --all

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app-9tbfb 2/2 Running 0 2smy-symfony-app-ui6yv 2/2 Running 0 2s my-symfony-app-cx9te 2/2 Running 0 2s my-symfony-app-ui6yv 2/2 Running 0 2s my-symfony-app-tgquh 2/2 Running 0 2s

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app-jzd97 2/2 Running 0 2smy-symfony-app-193aw 2/2 Running 0 2s my-symfony-app-bicex 2/2 Running 0 2s my-symfony-app-r2in0 2/2 Running 0 2s my-symfony-app-lbo54 2/2 Running 0 2s

core@core-01 ~ $ kubectl create -f my-symfony-app-rc.yml

core@core-01 ~ $ kubectl delete pods --all

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app-9tbfb 2/2 Running 0 2smy-symfony-app-ui6yv 2/2 Running 0 2s my-symfony-app-cx9te 2/2 Running 0 2s my-symfony-app-ui6yv 2/2 Running 0 2s my-symfony-app-tgquh 2/2 Running 0 2s

core@core-01 ~ $ kubectl scale --replicas=2 rc my-symfony-app

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app-jzd97 2/2 Running 0 2smy-symfony-app-193aw 2/2 Running 0 2s my-symfony-app-bicex 2/2 Running 0 2s my-symfony-app-r2in0 2/2 Running 0 2s my-symfony-app-lbo54 2/2 Running 0 2s

core@core-01 ~ $ kubectl create -f my-symfony-app-rc.yml

core@core-01 ~ $ kubectl delete pods --all

core@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app-9tbfb 2/2 Running 0 2smy-symfony-app-ui6yv 2/2 Running 0 2s my-symfony-app-cx9te 2/2 Running 0 2s my-symfony-app-ui6yv 2/2 Running 0 2s my-symfony-app-tgquh 2/2 Running 0 2s

core@core-01 ~ $ kubectl scale --replicas=2 rc my-symfony-appcore@core-01 ~ $ kubectl get podsNAME READY STATUS RESTARTS AGEmy-symfony-app-9tbfb 2/2 Running 0 10smy-symfony-app-ui6yv 2/2 Running 0 10s

Features, Features, Features!

Features, Features, Features!• Rescheduling

Features, Features, Features!• Rescheduling• Rolling Updates

Features, Features, Features!• Rescheduling• Rolling Updates• Multiple Release Tracks

Services

apiVersion: v1kind: Servicemetadata: name: my-symfony-appspec: selector: name: my-symfony-app ports: - port: 80

my-symfony-app-svc.yml

core@core-01 ~ $ kubectl create -f my-symfony-app-svc.yml

core@core-01 ~ $ kubectl get servicesNAME LABELS SELECTOR IP(S) PORT(S)my-symfony-app name=my-symfony-app name=my-symfony-app 80/TCP

core@core-01 ~ $ kubectl create -f my-symfony-app-svc.yml

core@core-01 ~ $ kubectl get servicesNAME LABELS SELECTOR IP(S) PORT(S)my-symfony-app name=my-symfony-app name=my-symfony-app 80/TCP

core@core-01 ~ $ kubectl create -f my-symfony-app-svc.yml

core@core-01 ~ $ kubectl exec -ti my-symfony-app -c nginx bash

root@my-symfony-app:/# curl -I my-symfony-appHTTP/1.1 200 OKServer: nginx/1.9.3 (Ubuntu)Date: Fri, 16 Oct 2015 06:46:35 GMTContent-Type: text/htmlContent-Length: 177Connection: keep-alive

Features, Features, Features!

Features, Features, Features!• Renaming

Features, Features, Features!• Renaming• Load-Balancing

Features, Features, Features!• Renaming• Load-Balancing• Node-Port

Features, Features, Features!• Renaming• Load-Balancing• Node-Port• Static Endpoints

Features, Features, Features!• Renaming• Load-Balancing• Node-Port• Static Endpoints• Bidirektionale Verbindungen

Features, Features, Features!• Renaming• Load-Balancing• Node-Port• Static Endpoints• Bidirektionale Verbindungen• Das alles in Echtzeit!

Alles zusammen

Development (Vagrant Cluster)

NodePortServiceRCPod

NamespaceCluster

Testcluster

NodePortServiceRCPod

NamespaceCluster

Prodcluster

NodePortServiceRCPod

NamespaceCluster

Herausforderungen

Bugs, Bugs, Bugs…

Bugs, Bugs, Bugs…• Namespace-Isolation

Bugs, Bugs, Bugs…• Namespace-Isolation• Concurrent-Pull

Bugs, Bugs, Bugs…• Namespace-Isolation• Concurrent-Pull• Orphan-Pull

Bugs, Bugs, Bugs…• Namespace-Isolation• Concurrent-Pull• Orphan-Pull• Exec-Timeout

Bugs, Bugs, Bugs…• Namespace-Isolation• Concurrent-Pull• Orphan-Pull• Exec-Timeout• …

Fehlende Features

Fehlende Features• kubectl wait

Fehlende Features• kubectl wait• Variablen-Expansion

Fehlende Features• kubectl wait• Variablen-Expansion• …

Fazit

Container forcieren DevOps!

Container forcieren DevOps!

Container forcieren CD!

Container forcieren DevOps!

Container forcieren CD!

Orchestrierung ist komplex

Container forcieren DevOps!

Container forcieren CD!

Tooling ist Bleeding Edge

Orchestrierung ist komplex

Container forcieren DevOps!

Container forcieren CD!

Tooling ist Bleeding Edge

Orchestrierung ist komplex

Wir vertrauen auf Kubernetes!

?Fragenhttp://chefkoch.jobs - We are hiring ;)

http://perprogramming.de info@perprogramming.de

Danke!