Big Data zum AnfassenKai Voigt, Cloudera Inc.

AIShttps://en.wikipedia.org/wiki/Automatic_Identification_System

Sailing Yacht A ("White Pearl")Länge: 140+ m, Höhe: 100+ m, Kosten: 400+ Mio EUR

Dynamische Daten

{"msgid":1,"rot":0,"sub_message":806,"aisclient_id":2051,"slot_timeout":6,"spare":2,"userid":211704630,"pos_acc":1,"sentences":["!AIVDM,1,1,,A,139qK=R009PffhLO9MrU9@4v8H<V,0*38"],"nav_status":2,"sync_state":0,"time_received":1443099141551,"repeat":0,"pos":[10.20845,54.43495],"utc_sec":31,"sog":9,"time_captured":1443099141000,"cog":1317,"raim":0,"true_heading":2,"smi":0}

Statische Daten{"imo":1012141,"dim_port":15,"aisclient_id":2216,"pos_type":1,"userid":211704630,"draught":76,"sentences":["!AIVDM,2,1,2,A,539qK=T0?LJl@l4H001LPU@F10D58h000000000T:0w?:6J`hC4iA@PDP000,0*13","!AIVDM,2,2,2,A,00000000000,2*26"],"dim_starboard":10,"callsign":"DMAF@@@","repeat":0,"time_received":1443098166711,"version":1,"name":"WHITE PEARL@@@@@@@@@","dte":0,"eta_minute":48,"dim_bow":80,"msgid":5,"eta_hour":8,"ship_type":36,"spare":0,"eta_month":9,"dim_stern":63,"dest":"SEEBAR@@@@@@@@@@@@@@","eta_day":21,"time_captured":1443098166000}

http://www.vesseltracker.com/

Satellite AIS

Big Data?

• Nord- und Ostsee, 500 Records/Sekunde

• 250 KByte/Sekunde

• Weltweit: 10.000 Records/Sekunde

• 5 MByte/Sekunde = ~400 GByte/Tag

λApache Hadoop

AIS Data Stream

Flume/Kafka

Spark StreamingHBase/HDFS

Impala/Spark Near Real Time

Kollisionserkennung• 2 Schiffe zur gleichen Zeit am gleichen Ort

Illegaler Fischfang

Illegaler Fischfang

Illegaler Fischfang

Weitere Anwendungsfälle• Tracking

• Protokollierung von Unfällen

• Piraterie

• Routen-Optimierung

• Hafen-Logistik

• Flotten-Logistik

• Umweltschutz

• Search & Rescue

• Navigationshilfe

Danke!Kai Voigt, kai@cloudera.com