Zentrales Logging mit dem Elastic Stack

Simon SchneiderFroscon 2019

Simon Schneider

Zentrales vs Dezentrales Loggingi

Zentrales LoggingDezentrales Logging

Elastic Stacki

Beats

Logstash

Beats Logstash Ingest

Elasticsearch

Kibana

Elasticsearch

Beats Logstash

SaaS Self Managed

Elastic Stack

Visualize & Manage

Store, Search & Analyze

Ingest

DeploymentElastic Cloud Elastic CloudEnterprise

Standalone

Hot Node Warm NodeMaster Node

● Nur Management ● Keine Datenhaltung

● SSD (RAID 0)● Query● Ingest● Letzten 1-3 Tage

● HDD● Query● > 3 Tage

Hot-Warm Architektur

Docker NodesIngest Nodes

Data NodesKibanaNodes

REWE Digital Ecom - Elastic Stack

REWE Digital Big Data - Elastic Stack

https://github.com/elastic/cloud-on-k8s

Live Coding

Web Server● Static HTML● Proxy Route

Such-Backend● Such-Endpunkt● Bleve Search

Filebeat Logstash Elasticsearch

● Access Log Aufbereitung● Geo IP Parser● User Agent Parser

➔ Großer Initialaufwand / Kosten

➔ Hot-Warm Architektur zur Kostensenkung

➔ JSON Logs sind einfach zu verarbeiten

➔ Filebeat + Elasticsearch kann ausreichend sein

➔ Sicherheit, Datenschutz und Schnelligkeit gewonnen

Wrap Up

Thank youSpecial Thanks to:

- Peter Larsen- Paul Puschmann

twitter.com/raynigongithub.com/raynigon

● Verschiedene Logos de.wikipedia.org● RRZE Icon Set● Morning Coffee Photo by David Mao on Unsplash● Shock Photo by David Mao on Unsplash● Work Photo by Glenn Carstens-Peters on Unsplash● Office Photo by Tim van der Kuip on Unsplash● Folders Photo by Samuel Zeller on Unsplash● Light Building Photo by Lysander Yuen on Unsplash● Building a House Photo by Randy Fath on Unsplash● Trophy Photo by Fauzan Saari on Unsplash● Ecommerce by Mein Deal on Unsplash● Big Data by ev on Unsplash● Burning Money by Jp Valery on Unsplash● Rocket by Bill Jelen on Unsplash

Credits