Kubernetes und Container – Aber Sicher! · Kubernetes und Container – Aber Sicher! Container /...

Kubernetes und Container – Aber Sicher!

Container / K8s SecurityAndreas Falk

andreas.falk@novatec-gmbh.de / @andifalk

https://www.novatec-gmbh.de/beratung/agile-security

Vorstellung

2

Andreas FalkNovatec Consulting

3

https://www.novatec-gmbh.de/schulung/application-security-training-for-developers-by-jim-manico

Agenda

1. What can go wrong2. Application Security3. Container Security4. Kubernetes Security5. Kubernetes Secrets

4

Presentation Slides and Demo Code:

https://github.com/andifalk/secure-development-on-kubernetes

Where are the Slides and the Code?

5

Introduction

6

What can go wrong?

Top Challenges in Kubernetes

7

Source: https://thenewstack.io

Severe Vulnerability in Kubernetes

8

Source: https://blog.aquasec.com

Crypto Mining Via K8s Dashboard

9

Source: https://blog.heptio.com

Open ETCD Ports in Kubernetes (1)

10

https://shodan.io

Open ETCD Ports in Kubernetes (2)

11 https://github.com/etcd-io/etcd/releases

$ etcdctl --endpoints=http://xx.xx.xx.xx:2379 cluster-health

member b97ee4034db41d17 is healthy: got healthy resultfrom http://xx.xx.xx.xx:2379cluster is healthy

Vulnerable Docker Images

12

Source: The state of open source security report (snyk.io)

All is Root

13

14

Source:Kubernetes Security, O’Reilly, 2018

Kubernetes attack vectors

Operational / Development Kubernetes Security

15

https://kubernetes.io/docs/concepts/security/overview/#the-4c-s-of-cloud-native-securityhttps://learnk8s.io/production-best-practices/

K8s Operational Security

K8s Development Security

API Server

Etcd

Scheduler

Controller Manager

Master Node

Kubelet

Kube Proxy

Container Runtime

Worker Node

AuthAuthz

TLS

AuthAuthz

TLS

Application- / Docker- / K8s-Security

So what can we do as developers?

16

The Path for Secure Development on K8s

17

Application Security

Container Security

Kubernetes Security

Kubernetes Secrets

The Path for Secure Development on K8s

18

Application Security

Container Security

Kubernetes Security

Kubernetes Secrets

Application Security

19

Web Application

Authentication

Authorization

SQL Injection

Cross Site Scripting (XSS)

Cross Site Request Forgery (CSRF)

Data Protection (Crypto)

...

Application Security

20

Iteration 1: Application Securityhttps://github.com/andifalk/secure-development-on-kubernetes

Live Demo: Show me the code

21

The Path for Secure Development on K8s

22

Application Security Container

SecurityKubernetes

SecurityKubernetes

Secrets

Docker Security Basics

23

▪ Process ID (pid) ▪ Network (net)▪ Filesystem/mount (mnt)▪ Inter-Process Communication (ipc)▪ User (user)▪ UTS (hostname)

Linux Kernel Namespaces

24

▪ Resource Limits− CPU − Memory− Devices− Processes− Network

Linux Control Groups (CGroups)

25

For Java this only works with container aware JDK versions as of OpenJDK 8u192 or above

▪ Break up root privileges into smaller units− CAP_SYS_ADMIN− CAP_NET_ADMIN− CAP_NET_BIND_SERVICE− CAP_CHOWN− ...

Linux Capabilities

26

http://man7.org/linux/man-pages/man7/capabilities.7.html

$ docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE

▪ AppArmor▪ Security Enhanced Linux (SELinux)

Mandatory Access Control (MAC)

27

https://gitlab.com/apparmor/apparmor/wikis/homehttps://github.com/SELinuxProject

▪ Deny critical system calls by default− reboot− mount− swapon− ...

Secure Computing Mode (SecComp)

28

http://man7.org/linux/man-pages/man2/seccomp.2.htmlhttps://docs.docker.com/engine/security/seccomp

1. Secure User Mapping2. Patch Management Strategy3. Network Segmentation and Firewalling4. Secure Defaults and Hardening5. Maintain Security Contexts6. Protect Secrets7. Resource Protection8. Container Image Integrity and Origin9. Follow Immutable Paradigm

10. Logging

OWASP Docker Top 10

29

https://github.com/OWASP/Docker-Security

Docker Images

30

Docker Image Security

31

USER directive in Dockerfile

Say No To Root!

32

https://opensource.com/article/18/3/just-say-no-root-containers

FROM openjdk:11-jre-slimCOPY hello-spring-kubernetes-1.0.0-SNAPSHOT.jar app.jarEXPOSE 8080RUN addgroup --system --gid 1002 app && adduser --system --uid 1002 --gid 1002 appuserUSER 1002ENTRYPOINT java -jar /app.jar

Use JIB and Distroless Images

Say No To Root!

33https://github.com/GoogleContainerTools/jib

plugins { id 'com.google.cloud.tools.jib' version '...'}

jib { container { user = 1002 }}

▪ Perform Image Scanning− Anchore− Clair− Trivy

▪ Regularly Update Base Images

Keep Being Secure

34

https://anchore.com/opensource/https://github.com/coreos/clairhttps://github.com/aquasecurity/trivy

Iteration 2: Container Securityhttps://github.com/andifalk/secure-development-on-kubernetes

Live Demo: Show me the code

35

The Path for Secure Development on K8s

36

Application Security

Container Security Kubernetes

SecurityKubernetes

Secrets

Kubernetes Basics

37

Ingress Service Deployment Replica Set

PodPod

Pod

https://kubernetes.io/docs/conceptshttps://www.aquasec.com/wiki/display/containers/70+Best+Kubernetes+Tutorials

Kubernetes Security

Network Policies

Role Based Access Control (RBAC)

Resource Limits

Pod Security Context

Pod Security Policy

Kubernetes Auditing

Resource Limits

39

https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resourcehttps://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource

spec: ... containers: resources: limits: cpu: "1" memory: "512Mi" requests: cpu: 500m memory: "256Mi" ...

Pod/Container Security Context

40https://kubernetes.io/docs/tasks/configure-pod-container/security-context

spec: securityContext: runAsNonRoot: true containers: securityContext: allowPrivilegeEscalation: false privileged: false runAsNonRoot: true readOnlyRootFilesystem: true capabilities: drop: - ALL

Pod Security Policy (Still In Beta!)

41https://kubernetes.io/docs/concepts/policy/pod-security-policy

apiVersion: policy/v1beta1kind: PodSecurityPolicymetadata: name: no-root-policyspec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL runAsUser: rule: 'MustRunAsNonRoot' ...

Pod Security Policy (Policy Order)

42

https://kubernetes.io/docs/concepts/policy/pod-security-policy/#policy-orderhttps://kubernetes.io/docs/reference/access-authn-authz/admission-controllers

Policy order selection criteria:

1. Policies which allow the pod as-is are preferred2. If pod must be defaulted or mutated, the first

policy (ordered by name) to allow the pod is selected.

Kubernetes Role Based Access Control (RBAC)

43

ClusterRole ClusterRoleBinding

Role RoleBinding

SubjectAPI Groups

Resources

Verbs

Cluster-Wide

Namespace

- User- Group- ServiceAccount

https://kubernetes.io/docs/reference/access-authn-authz/rbac/

Kubernetes Role Based Access Control (RBAC)

44

https://kubernetes.io/docs/reference/access-authn-authz/rbac/

apiGroups extensions, apps, policy, ...

resources pods, deployments, configmaps, secrets, nodes,services, endpoints, podsecuritypolicies, ...

verbs get, list, watch, create, update, patch, delete, use, ...

Service Account

45

https://kubernetes.io/docs/concepts/policy/pod-security-policy/#authorizing-policies

apiVersion: v1kind: ServiceAccountmetadata: name: deploy-pod-security-policy namespace: default

Pod Security Policy Role

46

https://kubernetes.io/docs/concepts/policy/pod-security-policy/#authorizing-policies

apiVersion: rbac.authorization.k8s.io/v1kind: Rolemetadata: name: no-root-policy-role namespace: defaultrules: - apiGroups: ['policy'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: - no-root-policy

Pod Security Policy Role Binding

47

apiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata: name: deploy-pod-security-policy namespace: defaultroleRef: kind: Role name: no-root-policy-role apiGroup: rbac.authorization.k8s.iosubjects: - kind: ServiceAccount name: deploy-pod-security-policy namespace: default

Helm 3 Is Here!

48

https://v3.helm.shhttps://helm.sh/docs/faq/#removal-of-tiller

Iteration 3: Kubernetes Securityhttps://github.com/andifalk/secure-development-on-kubernetes

Live Demo: Show me the code

49

The Path for Secure Development on K8s

50

Application Security

Container Security

Kubernetes Security Kubernetes

Secrets

Kubernetes Secrets

Secrets

KMS

Secrets

Secrets

Etcd

Kubernetes Secrets

52

https://kubernetes.io/docs/concepts/configuration/secret

apiVersion: v1kind: Secretmetadata: name: hello-spring-cloud-kubernetes namespace: defaulttype: Opaquedata: user.username: dXNlcg== user.password: azhzX3VzZXI= admin.username: YWRtaW4= admin.password: azhzX2FkbWlu

▪ Encrypt Secret Data at Rest Only Base64 Encoded by Default!

▪ Applications interacting with secrets API should be limited using RBAC

▪ Mount secrets instead of ENV Mapping

Kubernetes Secrets - Best Practices

53

https://kubernetes.io/docs/concepts/configuration/secret/#best-practiceshttps://kubernetes.io/docs/tasks/administer-cluster/encrypt-data

Pay Attention to Spring Boot Actuator

54 http://localhost:8080/actuator/env

{ "name": "applicationConfig: ...", "properties": { "greet.my-sec": { "value": "geheim", "origin": "class path resource ..." }, "greet.password": { "value": "******", "origin": "class path resource ..." }}

Encryption Layers

55

Envelope Encryption On Kubernetes

56

https://cloud.google.com/kms/docs/envelope-encryptionhttps://kubernetes.io/docs/tasks/administer-cluster/kms-provider

▪ Azure Key Vault (Key Vault FlexVolume)▪ Google Cloud KMS▪ AWS KMS▪ ...

Key Management System (KMS) Cloud Providers

57

https://github.com/Azure/kubernetes-kmshttps://github.com/Azure/kubernetes-keyvault-flexvolhttps://cloud.google.com/kmshttps://aws.amazon.com/de/kms

▪ Sealed Secrets▪ Helm Secrets▪ Kamus▪ Sops▪ Hashicorp Vault

What about Secrets in

58

https://learnk8s.io/kubernetes-secrets-in-githttps://github.com/bitnami-labs/sealed-secretshttps://github.com/futuresimple/helm-secretshttps://github.com/Soluto/kamushttps://github.com/mozilla/sopshttps://www.vaultproject.io

Summary

59

▪ Containers use Linux Namespaces+Caps▪ Say NO to root on K8s▪ “Least privilege” for service accounts▪ Keep K8s up-to-date and scan for security▪ Ensure your secrets are encrypted in K8s▪ Keep K8s and container images up-to-date

Summary / Key Insights

60

61

Feedback

Books and Online References

62

▪ Kubernetes Security, O’Reilly, 2018, ISBN: 978-1-492-04600-4▪ Cloud Native DevOps with Kubernetes, O’Reilly, 2019, ISBN: 978-1492040767▪ https://github.com/andifalk/secure-development-on-kubernetes▪ Crafty Requests: Deep Dive Into Kubernetes CVE-2018-1002105 - Ian Coldwater (Video)▪ Ship of Fools: Shoring Up Kubernetes Security - Ian Coldwater (Video)▪ https://kubernetes.io/docs/concepts/security/overview/#the-4c-s-of-cloud-native-security▪ https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster▪ https://opensource.com/article/18/3/just-say-no-root-containers▪ https://github.com/GoogleContainerTools/jib▪ https://anchore.com/opensource/▪ https://github.com/coreos/clair▪ https://github.com/aquasecurity/trivy▪ https://www.owasp.org/index.php/OWASP_Docker_Top_10

Books and Online References (1)

63

▪ https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource▪ https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource▪ https://kubernetes.io/docs/tasks/configure-pod-container/security-context▪ https://kubernetes.io/docs/concepts/policy/pod-security-policy▪ https://kubernetes.io/docs/reference/access-authn-authz/rbac/▪ https://kubernetes.io/docs/concepts/configuration/secret▪ https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data▪ https://cloud.google.com/kms/docs/envelope-encryption▪ https://kubernetes.io/docs/tasks/administer-cluster/kms-provider▪ https://github.com/Azure/kubernetes-kms▪ https://cloud.google.com/kms▪ https://aws.amazon.com/de/kms

Books and Online References (2)

64

Novatec Consulting GmbHDieselstraße 18/1D-70771 Leinfelden-Echterdingen

T. +49 711 22040-700info@novatec-gmbh.dewww.novatec-gmbh.de

65

Managing ConsultantAndreas Falk

Mobil: +49 151 46146778E-Mail: andreas.falk@novatec-gmbh.de