Big data and Machine learning APIs

Transcript

1. 1.

   Big Data and Machine Learning APIs

2. 2.

   Sam Bessalah @samklr Software Engineer, Freelance Data Engineering, Distributed systems,

   Machine Learning Paris Data Geek Meetup @DataParis me :
3. 3.

   None
4. 4.

   None
5. 5.

   None
6. 6.

   Big Data Legends ….

7. 7.

   Big Data Legends … Web logs Sensors Other Data source

   .. . . .
8. 8.

   A Big Data Legend … Web logs Sensors Other Data

   sources .. . . .
9. 9.

   A Big Data Legend … Web logs Sensors Other Data

   sources .. . . .
10. 10.

    A Big Data Legend … Web logs Sensors Other Data

    sources .. . . .
11. 11.

    A Big Data Legend … Web logs Sensors Other Data

    sources .. . . .
12. 12.

    A Big Data Legend … Web logs Sensors Other Data

    sources .. . . . Data Driven Decisions Smart Applications
13. 13.

    BUT ….

14. 14.

    - Building big data infrastructures is no easy task. -

    Leveraging data for decision making requires a mix of multiples skills : . System Engineering . Distributed computing . Statistics . Machine Learning
15. 15.

    Solutions …. - Build Data platforms as a service. -

    Build robust and consistent APIs to bring big data to the masses. - Leverages fluent APIs for fast data science
16. 16.

    None
17. 17.

    Big Data is not just about throwing data to Hadoop.

18. 18.

    It’s also about data pipelines

19. 19.

    Data Sources

20. 20.

    Data Sources

21. 21.

    Data Sources - High Throughput distributed mssaging platform - Publish

    Subscribe Model - Modelled as a distributed replicated log - Persists messages to disk - Categorizes messages into Topics - Allows message retention for long specified amount of time - Allows stream replay in case of failure
22. 22.

    Data Sources Machine Learning High Latency Batch Apps Real Time

    Processing
23. 23.

    How do you build an API around that?

24. 24.

    None
25. 25.

    /ingest REST API

26. 26.

    /ingest

27. 27.

    /ingest /query /trainModel /process

28. 28.

    Things to be careful with - Multitenancy (Yarn, Mesos, Docker…)

    - Job Scheduling - Security - Serialisation : ProtoBuf, Thrift, Avro - Storage Format : Optimize queries with columnar storage. - Compression : LZO, Snappy
29. 29.

    Making sense of data …

30. 30.

    None
31. 31.

    What is Machine Learning?

32. 32.

    http://dilbert.com/strips/comic/2013-02-02

33. 33.

    None
34. 34.

    https://speakerdeck.com/nivdul/lightning-fast-machine-learning-with-spark-1

35. 35.

    Machine Learning workflow

36. 36.

    Machine Learning workflow Text, Images, etc

37. 37.

    Machine Learning workflow Text, Images, etc Feature Extraction

38. 38.

    Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm

    Training
39. 39.

    Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm

    Training Predictive Model
40. 40.

    Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm

    Training Predictive Model New Data Feature Vector Prediction
41. 41.

    Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm

    Predictive Model New Data Prediction
42. 42.

    Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm

    Predictive Model New Data Prediction BLACK BOX
43. 43.

    Machine Learning Libraries and Frameworks

44. 44.

    scikit-learn.org

45. 45.

    Text, Images, etc Feature Extraction Predictive Model New Data Prediction

    X = vect.fit_transform(input) clf.fit(X,y) X_new = vect.fit_transform(input) y_new= clf.predict(X_new)
46. 46.

    http://arxiv.org/abs/1309.0238

47. 47.

    From library to web APIs

48. 48.

    Machine Learning workflow Text, Images, etc Feature Extraction Learning algorithm

    Predictive Model New Data Prediction BLACK BOX
49. 49.

    Machine Learning workflow Text, Images, etc Transformed Data Application Prediction

    Predictive API
50. 50.

    Predictive Web APIs

51. 51.

    Some examples

52. 52.

    Challenges of Predictive APIs

53. 53.

    http://www.r-bloggers.com/data-science-toolbox-survey-results-surprise-r-and-python-win/

54. 54.

    Modeling and Prediction are just a small part of the

    process
55. 55.

    - Data locality and data gravity - Support the full

    workflow - Verticalization of platforms - Scalability - Collaboration and interoperability - Black boxing of implementations
56. 56.

    Explore machine learning for APIs orchestration. Talk to Ori @OriPekelman

    Next Frontier ? Or actual reality ?
57. 57.

    None
58. 58.

    http://speakerdeck.com/samklr