Big Data & Geoposicionamiento

Talk about how Big Data and geospatial processing worlds are merging to get the best insights

Transcript

1. Jorge López-Malla Matute Senior Data Engineer
4. SKILLS JORGE LÓPEZ-MALLA @jorgelopezmalla Arquitecto Big Data, certificado número 13 de Spark, riojano y miope. Después de años tratando de solventar problemas modernos con tecnologías tradicionales lo intenté con el Big Data y, ¡vi que lo resolvían!
5. What we do Geoblink
6. COLLECTING DATA internal data external data proprietary location data TRANSFORMING DATA using advanced analytics artificial intelligence PROVIDING INSIGHTS user-friendly platform data-driven decisions How we do it
8. • • • • •
12. 3G 1G 15G 15G 6G 12G 12G 12G
13. • • •
15. • • • • •
16. • • • •
20. • • • •
23. • • •
25. • • •
27. • • •
28. • • • •
30. • • • •
32. • • • •
34. • several • • 1:1 mapping •
35. • datasources and processing engines • • •
36. (U1 ,Madrid, Point(x1, y1)) (U2 ,Logroño, Point(x2, y2)) (U1 ,Cadiz, Point(x3, y3)) (U3 ,Logroño, Point(x4, y4)) (U1 ,Ávila, Point(x5, y5)) (U2 ,Huelva, Point(x6, y6)) (U3 ,Huelva, Point(x7, y7)) (U2 ,Logroño, Point(x8, y8)) HBase Master Spark Executor-1 Spark Executor-2 Point(x1, y1), [U1 ,Madrid] Point(x5, y5), [U1 ,Ávila] Point(x2, y2), [U21 ,Logroño] Point(x4, y4), [U31 ,Logroño] Point(x6, y6), [U1 ,Huelva] Point(x7, y7), [U3 ,Huelva] Point(x3, y3), [U1 ,Cadiz] Point(x8, y8), [U21 ,Logroño] Region Server-1 Region Server-2 Region Server-3
37. ECLQuery.toCQL(“people between 1000, 200”) HBase Master Client.java Region Server-1 Region Server-2 Region Server-3 Point(x1, y1), [U1 ,Madrid] Point(x5, y5), [U1 ,Ávila] Point(x2, y2), [U21 ,Logroño] Point(x4, y4), [U31 ,Logroño] Point(x6, y6), [U1 ,Huelva] Point(x7, y7), [U3 ,Huelva] Point(x3, y3), [U1 ,Cadiz] Point(x8, y8), [U21 ,Logroño]
38. HBase Master val dataFrame = sparkSession.read .format("geomesa") .options(dsParams) .option("geomesa.featur e", "spain") .load() Spark Driver Region Server-1 Region Server-2 Region Server-3 Point(x1, y1), [U1 ,Madrid] Point(x5, y5), [U1 ,Ávila] Point(x2, y2), [U21 ,Logroño] Point(x4, y4), [U31 ,Logroño] Point(x6, y6), [U1 ,Huelva] Point(x7, y7), [U3 ,Huelva] Point(x3, y3), [U1 ,Cadiz] Point(x8, y8), [U21 ,Logroño]
39. • really • • • •
42. ★ ◦ https://www.geoblink.com/work-with-us/ ★ ◦ jobs@geoblink.com ◦ jlmalla@geoblink.com