65
Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

  • Upload
    others

  • View
    11

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Kubernetes und Container – Aber Sicher!

Container / K8s SecurityAndreas Falk

Page 2: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

[email protected] / @andifalk

https://www.novatec-gmbh.de/beratung/agile-security

Vorstellung

2

Andreas FalkNovatec Consulting

Page 3: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

3

https://www.novatec-gmbh.de/schulung/application-security-training-for-developers-by-jim-manico

Page 4: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Agenda

1. What can go wrong2. Application Security3. Container Security4. Kubernetes Security5. Kubernetes Secrets

4

Page 5: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Presentation Slides and Demo Code:

https://github.com/andifalk/secure-development-on-kubernetes

Where are the Slides and the Code?

5

Page 6: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Introduction

6

What can go wrong?

Page 7: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Top Challenges in Kubernetes

7

Source: https://thenewstack.io

Page 8: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Severe Vulnerability in Kubernetes

8

Source: https://blog.aquasec.com

Page 9: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Crypto Mining Via K8s Dashboard

9

Source: https://blog.heptio.com

Page 10: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Open ETCD Ports in Kubernetes (1)

10

https://shodan.io

Page 11: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Open ETCD Ports in Kubernetes (2)

11 https://github.com/etcd-io/etcd/releases

$ etcdctl --endpoints=http://xx.xx.xx.xx:2379 cluster-health

member b97ee4034db41d17 is healthy: got healthy resultfrom http://xx.xx.xx.xx:2379cluster is healthy

Page 12: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Vulnerable Docker Images

12

Source: The state of open source security report (snyk.io)

Page 13: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

All is Root

13

Page 14: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

14

Source:Kubernetes Security, O’Reilly, 2018

Kubernetes attack vectors

Page 15: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Operational / Development Kubernetes Security

15

https://kubernetes.io/docs/concepts/security/overview/#the-4c-s-of-cloud-native-securityhttps://learnk8s.io/production-best-practices/

K8s Operational Security

K8s Development Security

API Server

Etcd

Scheduler

Controller Manager

Master Node

Kubelet

Kube Proxy

Container Runtime

Worker Node

AuthAuthz

TLS

AuthAuthz

TLS

Page 16: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Application- / Docker- / K8s-Security

So what can we do as developers?

16

Page 17: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

The Path for Secure Development on K8s

17

Application Security

Container Security

Kubernetes Security

Kubernetes Secrets

Page 18: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

The Path for Secure Development on K8s

18

Application Security

Container Security

Kubernetes Security

Kubernetes Secrets

Page 19: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Application Security

19

Web Application

Authentication

Authorization

SQL Injection

Cross Site Scripting (XSS)

Cross Site Request Forgery (CSRF)

Data Protection (Crypto)

...

Page 20: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Application Security

20

Page 21: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Iteration 1: Application Securityhttps://github.com/andifalk/secure-development-on-kubernetes

Live Demo: Show me the code

21

Page 22: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

The Path for Secure Development on K8s

22

Application Security Container

SecurityKubernetes

SecurityKubernetes

Secrets

Page 23: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Docker Security Basics

23

Page 24: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ Process ID (pid) ▪ Network (net)▪ Filesystem/mount (mnt)▪ Inter-Process Communication (ipc)▪ User (user)▪ UTS (hostname)

Linux Kernel Namespaces

24

Page 25: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ Resource Limits− CPU − Memory− Devices− Processes− Network

Linux Control Groups (CGroups)

25

For Java this only works with container aware JDK versions as of OpenJDK 8u192 or above

Page 26: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ Break up root privileges into smaller units− CAP_SYS_ADMIN− CAP_NET_ADMIN− CAP_NET_BIND_SERVICE− CAP_CHOWN− ...

Linux Capabilities

26

http://man7.org/linux/man-pages/man7/capabilities.7.html

$ docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE

Page 27: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ AppArmor▪ Security Enhanced Linux (SELinux)

Mandatory Access Control (MAC)

27

https://gitlab.com/apparmor/apparmor/wikis/homehttps://github.com/SELinuxProject

Page 28: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ Deny critical system calls by default− reboot− mount− swapon− ...

Secure Computing Mode (SecComp)

28

http://man7.org/linux/man-pages/man2/seccomp.2.htmlhttps://docs.docker.com/engine/security/seccomp

Page 29: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

1. Secure User Mapping2. Patch Management Strategy3. Network Segmentation and Firewalling4. Secure Defaults and Hardening5. Maintain Security Contexts6. Protect Secrets7. Resource Protection8. Container Image Integrity and Origin9. Follow Immutable Paradigm

10. Logging

OWASP Docker Top 10

29

https://github.com/OWASP/Docker-Security

Page 30: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Docker Images

30

Page 31: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Docker Image Security

31

Page 32: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

USER directive in Dockerfile

Say No To Root!

32

https://opensource.com/article/18/3/just-say-no-root-containers

FROM openjdk:11-jre-slimCOPY hello-spring-kubernetes-1.0.0-SNAPSHOT.jar app.jarEXPOSE 8080RUN addgroup --system --gid 1002 app && adduser --system --uid 1002 --gid 1002 appuserUSER 1002ENTRYPOINT java -jar /app.jar

Page 33: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Use JIB and Distroless Images

Say No To Root!

33https://github.com/GoogleContainerTools/jib

plugins { id 'com.google.cloud.tools.jib' version '...'}

jib { container { user = 1002 }}

Page 34: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ Perform Image Scanning− Anchore− Clair− Trivy

▪ Regularly Update Base Images

Keep Being Secure

34

https://anchore.com/opensource/https://github.com/coreos/clairhttps://github.com/aquasecurity/trivy

Page 35: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Iteration 2: Container Securityhttps://github.com/andifalk/secure-development-on-kubernetes

Live Demo: Show me the code

35

Page 36: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

The Path for Secure Development on K8s

36

Application Security

Container Security Kubernetes

SecurityKubernetes

Secrets

Page 37: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Kubernetes Basics

37

Ingress Service Deployment Replica Set

PodPod

Pod

https://kubernetes.io/docs/conceptshttps://www.aquasec.com/wiki/display/containers/70+Best+Kubernetes+Tutorials

Page 38: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Kubernetes Security

Network Policies

Role Based Access Control (RBAC)

Resource Limits

Pod Security Context

Pod Security Policy

Kubernetes Auditing

Page 39: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Resource Limits

39

https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resourcehttps://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource

spec: ... containers: resources: limits: cpu: "1" memory: "512Mi" requests: cpu: 500m memory: "256Mi" ...

Page 40: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Pod/Container Security Context

40https://kubernetes.io/docs/tasks/configure-pod-container/security-context

spec: securityContext: runAsNonRoot: true containers: securityContext: allowPrivilegeEscalation: false privileged: false runAsNonRoot: true readOnlyRootFilesystem: true capabilities: drop: - ALL

Page 41: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Pod Security Policy (Still In Beta!)

41https://kubernetes.io/docs/concepts/policy/pod-security-policy

apiVersion: policy/v1beta1kind: PodSecurityPolicymetadata: name: no-root-policyspec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL runAsUser: rule: 'MustRunAsNonRoot' ...

Page 42: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Pod Security Policy (Policy Order)

42

https://kubernetes.io/docs/concepts/policy/pod-security-policy/#policy-orderhttps://kubernetes.io/docs/reference/access-authn-authz/admission-controllers

Policy order selection criteria:

1. Policies which allow the pod as-is are preferred2. If pod must be defaulted or mutated, the first

policy (ordered by name) to allow the pod is selected.

Page 43: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Kubernetes Role Based Access Control (RBAC)

43

ClusterRole ClusterRoleBinding

Role RoleBinding

SubjectAPI Groups

Resources

Verbs

Cluster-Wide

Namespace

- User- Group- ServiceAccount

https://kubernetes.io/docs/reference/access-authn-authz/rbac/

Page 44: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Kubernetes Role Based Access Control (RBAC)

44

https://kubernetes.io/docs/reference/access-authn-authz/rbac/

apiGroups extensions, apps, policy, ...

resources pods, deployments, configmaps, secrets, nodes,services, endpoints, podsecuritypolicies, ...

verbs get, list, watch, create, update, patch, delete, use, ...

Page 45: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Service Account

45

https://kubernetes.io/docs/concepts/policy/pod-security-policy/#authorizing-policies

apiVersion: v1kind: ServiceAccountmetadata: name: deploy-pod-security-policy namespace: default

Page 46: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Pod Security Policy Role

46

https://kubernetes.io/docs/concepts/policy/pod-security-policy/#authorizing-policies

apiVersion: rbac.authorization.k8s.io/v1kind: Rolemetadata: name: no-root-policy-role namespace: defaultrules: - apiGroups: ['policy'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: - no-root-policy

Page 47: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Pod Security Policy Role Binding

47

apiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata: name: deploy-pod-security-policy namespace: defaultroleRef: kind: Role name: no-root-policy-role apiGroup: rbac.authorization.k8s.iosubjects: - kind: ServiceAccount name: deploy-pod-security-policy namespace: default

Page 48: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Helm 3 Is Here!

48

https://v3.helm.shhttps://helm.sh/docs/faq/#removal-of-tiller

Page 49: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Iteration 3: Kubernetes Securityhttps://github.com/andifalk/secure-development-on-kubernetes

Live Demo: Show me the code

49

Page 50: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

The Path for Secure Development on K8s

50

Application Security

Container Security

Kubernetes Security Kubernetes

Secrets

Page 51: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Kubernetes Secrets

Secrets

KMS

Secrets

Secrets

Etcd

Page 52: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Kubernetes Secrets

52

https://kubernetes.io/docs/concepts/configuration/secret

apiVersion: v1kind: Secretmetadata: name: hello-spring-cloud-kubernetes namespace: defaulttype: Opaquedata: user.username: dXNlcg== user.password: azhzX3VzZXI= admin.username: YWRtaW4= admin.password: azhzX2FkbWlu

Page 53: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ Encrypt Secret Data at Rest Only Base64 Encoded by Default!

▪ Applications interacting with secrets API should be limited using RBAC

▪ Mount secrets instead of ENV Mapping

Kubernetes Secrets - Best Practices

53

https://kubernetes.io/docs/concepts/configuration/secret/#best-practiceshttps://kubernetes.io/docs/tasks/administer-cluster/encrypt-data

Page 54: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Pay Attention to Spring Boot Actuator

54 http://localhost:8080/actuator/env

{ "name": "applicationConfig: ...", "properties": { "greet.my-sec": { "value": "geheim", "origin": "class path resource ..." }, "greet.password": { "value": "******", "origin": "class path resource ..." }}

Page 55: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Encryption Layers

55

Page 56: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Envelope Encryption On Kubernetes

56

https://cloud.google.com/kms/docs/envelope-encryptionhttps://kubernetes.io/docs/tasks/administer-cluster/kms-provider

Page 57: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ Azure Key Vault (Key Vault FlexVolume)▪ Google Cloud KMS▪ AWS KMS▪ ...

Key Management System (KMS) Cloud Providers

57

https://github.com/Azure/kubernetes-kmshttps://github.com/Azure/kubernetes-keyvault-flexvolhttps://cloud.google.com/kmshttps://aws.amazon.com/de/kms

Page 58: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ Sealed Secrets▪ Helm Secrets▪ Kamus▪ Sops▪ Hashicorp Vault

What about Secrets in

58

https://learnk8s.io/kubernetes-secrets-in-githttps://github.com/bitnami-labs/sealed-secretshttps://github.com/futuresimple/helm-secretshttps://github.com/Soluto/kamushttps://github.com/mozilla/sopshttps://www.vaultproject.io

Page 59: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Summary

59

Page 60: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ Containers use Linux Namespaces+Caps▪ Say NO to root on K8s▪ “Least privilege” for service accounts▪ Keep K8s up-to-date and scan for security▪ Ensure your secrets are encrypted in K8s▪ Keep K8s and container images up-to-date

Summary / Key Insights

60

Page 61: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

61

Feedback

Page 62: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Books and Online References

62

Page 63: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ Kubernetes Security, O’Reilly, 2018, ISBN: 978-1-492-04600-4▪ Cloud Native DevOps with Kubernetes, O’Reilly, 2019, ISBN: 978-1492040767▪ https://github.com/andifalk/secure-development-on-kubernetes▪ Crafty Requests: Deep Dive Into Kubernetes CVE-2018-1002105 - Ian Coldwater (Video)▪ Ship of Fools: Shoring Up Kubernetes Security - Ian Coldwater (Video)▪ https://kubernetes.io/docs/concepts/security/overview/#the-4c-s-of-cloud-native-security▪ https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster▪ https://opensource.com/article/18/3/just-say-no-root-containers▪ https://github.com/GoogleContainerTools/jib▪ https://anchore.com/opensource/▪ https://github.com/coreos/clair▪ https://github.com/aquasecurity/trivy▪ https://www.owasp.org/index.php/OWASP_Docker_Top_10

Books and Online References (1)

63

Page 64: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

▪ https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource▪ https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource▪ https://kubernetes.io/docs/tasks/configure-pod-container/security-context▪ https://kubernetes.io/docs/concepts/policy/pod-security-policy▪ https://kubernetes.io/docs/reference/access-authn-authz/rbac/▪ https://kubernetes.io/docs/concepts/configuration/secret▪ https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data▪ https://cloud.google.com/kms/docs/envelope-encryption▪ https://kubernetes.io/docs/tasks/administer-cluster/kms-provider▪ https://github.com/Azure/kubernetes-kms▪ https://cloud.google.com/kms▪ https://aws.amazon.com/de/kms

Books and Online References (2)

64

Page 65: Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container / K8s Security Andreas Falk

Novatec Consulting GmbHDieselstraße 18/1D-70771 Leinfelden-Echterdingen

T. +49 711 [email protected]

65

Managing ConsultantAndreas Falk

Mobil: +49 151 46146778E-Mail: [email protected]