SAMOA: A Platform for Mining Big Data Streams

Streaming data analysis in real time is becoming the fastest and most efficient way to obtain useful knowledge from what is happening now, allowing organizations to react quickly when problems appear or to detect new trends helping to improve their performance. In this talk, we present SAMOA, an upcoming platform for mining big data streams. SAMOA is a platform for online mining in a cluster/cloud environment. It features a pluggable architecture that allows it to run on several distributed stream processing engines such as S4 and Storm. SAMOA includes algorithms for the most common machine learning tasks such as classification and clustering.

Transcript

1. SAMOA
   A Platform for Mining Big Data Streams
   

   Gianmarco De Francisci Morales

   Yahoo Labs Barcelona

   gdfm@yahoo-inc.com
   RGB color version - for online/web use
   3D Y-Bang Logo
   1
   

   View Slide

2. Taxonomy
   Machine
   Learning
   Distributed
   Batch
   Hadoop
   Mahout
   Stream
   S4, Storm
   SAMOA
   Non
   Distributed
   Batch
   R,
   WEKA,
   …
   Stream
   MOA
   2
   RGB color version - for online/web use
   3D Y-Bang Logo
   

   View Slide

3. Research
   Scientist @
   Yahoo Labs
   Committer for
   Apache Pig.
   Contributor for
   Apache Hadoop,
   Giraph, S4.
   3
   RGB color version - for online/web use
   3D Y-Bang Logo
   

   View Slide

4. SAMOA Team
   4
   RGB color version - for online/web use
   3D Y-Bang Logo
   

   View Slide

5. Big Data Stream
   Volume + Velocity (+ Variety)
   Too large for single commodity server main memory
   Too fast for single commodity server CPU
   A solution should be:
   Distributed
   Scalable
   5
   RGB color version - for online/web use
   3D Y-Bang Logo
   

   View Slide

6. 6
   RGB color version - for online/web use
   3D Y-Bang Logo
   Applications
   

   View Slide

7. Personalization
   6
   RGB color version - for online/web use
   3D Y-Bang Logo
   Applications
   

   View Slide

8. Personalization
   Spam detection
   6
   RGB color version - for online/web use
   3D Y-Bang Logo
   Applications
   

   View Slide

9. Personalization
   Spam detection
   Recommendation
   6
   RGB color version - for online/web use
   3D Y-Bang Logo
   Applications
   

   View Slide

10. Data Science Lifecycle
    Old school’s

    data mining
    From data to insight
    From insight to model
    From model to value
    And repeat!
    7
    RGB color version - for online/web use
    3D Y-Bang Logo
    Gather
    Clean
    Model
    Deploy
    

    View Slide

11. Big Data Tools
    8
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

12. Problems
    Operational
    Need to rerun the pipeline and redeploy the model
    when new data arrives
    Paradigmatic
    New data lies in storage without generating new
    value until the new model is retrained
    9
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

13. Stream
    Batch data is a
    snapshot of
    streaming data
    10
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

14. Examples
    User clicks
    Search queries
    News
    Emails
    Tumblr posts
    Flickr photos
    Finance stocks
    Credit card transactions
    Wikipedia edit logs
    Facebook statuses
    Twitter updates
    Name your own…
    11
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

15. But we have Hadoop!
    “Mapreduce is Good Enough? If All You Have is a
    Hammer, Throw Away Everything That’s Not a Nail!”

    [J. Lin, in Big Data, 1(1):28–37, 2013]

    “Data whose characteristics forces us to look beyond
    the traditional methods that are prevalent at the time”

    [A. Jacobs, in ACM Queue, 7(6):10,2009]
    12
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

16. Big Data
    Too big to handle
    13
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

17. Future of big data
    Like drinking from a firehose
    14
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

18. Importance$of$O
    •  As$spam$trends$change
    retrain$the$model$with
    Importance
    Spam detection in
    comments on 

    Yahoo News
    Trends change in time
    Need to retrain model
    with new data
    15
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

19. Streaming
    Sequence is potentially infinite
    High amount of data, high speed of arrival
    Change over time (concept drift)
    Approximation algorithms

    (small error with high probability)
    Single pass, one data item at a time
    Sublinear space and time per data item
    16
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

20. Evolution of SPEs
    17
    —2003
    —2004
    —2005
    —2006
    —2008
    —2010
    —2011
    —2013
    Aurora
    STREAM
    Borealis
    SPC
    SPADE
    Storm
    S4
    1st generation
    2nd generation
    3rd generation
    Abadi et al., “Aurora: a new model and architecture for
    data stream management,” VLDB Journal, 2003
    Arasu et al., “STREAM: The Stanford Data Stream
    Management System,” Stanford InfoLab, 2004.
    Abadi et al., “The Design of the Borealis Stream
    Processing Engine,” in CIDR ’05
    Amini et al., “SPC: A Distributed, Scalable Platform
    for Data Mining,” in DMSSP ’06
    Gedik et al., “SPADE: The System S Declarative
    Stream Processing Engine,” in SIGMOD ’08
    Neumeyer et al., “S4: Distributed Stream Computing
    Platform,” in ICDMW ’10
    http://storm-project.net
    RGB color version - for online/web use
    3D Y-Bang Logo
    Samza http://samza.incubator.apache.org
    

    View Slide

21. Actors Model
    Live Streams
    Stream 1
    Stream 2
    Stream 3
    PE
    PE
    PE
    PE
    PE
    External
    Persister
    Output 1
    Output 2
    Event
    routing
    18
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

22. S4 Example
    status.text:"Introducing #S4: a distributed #stream processing system"
    PE1
    PE2 PE3
    PE4
    RawStatus
    null
    text="Int..."
    EV
    KEY
    VAL
    Topic
    topic="S4"
    count=1
    EV
    KEY
    VAL
    Topic
    topic="stream"
    count=1
    EV
    KEY
    VAL
    Topic
    reportKey="1"
    topic="S4", count=4
    EV
    KEY
    VAL
    TopicExtractorPE (PE1)
    extracts hashtags from status.text
    TopicCountAndReportPE (PE2-3)
    keeps counts for each topic across
    all tweets. Regularly emits report
    event if topic count is above
    a configured threshold.
    TopicNTopicPE (PE4)
    keeps counts for top topics and outputs
    top-N topics to external persister
    19
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

23. Paradigm Shift
    20
    RGB color version - for online/web use
    3D Y-Bang Logo
    Gather
    Clean
    Model
    Deploy
    + =
    

    View Slide

24. SAMOA
    Scalable Advanced Massive Online Analysis
    21
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

25. Concept
    SAMOA is a platform
    A framework for developing distributed streaming
    machine learning algorithms for researchers
    A library of state-of-the-art distributed streaming
    machine learning algorithms for practitioners
    22
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

26. Is SAMOA useful for you?
    Only if you need to deal with:
    Big fast data
    Evolving data (model updates)
    What is happening now?
    Use feedback in real-time
    Adapt to changes faster
    23
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

27. Architecture
    SAMOA
    S4 Storm …
    SAMOA
    Classifier
    Methods
    Clustering
    Methods
    Frequent
    Pattern
    Mining
    24
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

28. Advantages
    Program once, run everywhere
    Reuse existing computational infrastructure
    Model is always up to date
    No system downtime
    No complex backup/update procedures
    No need to choose update frequency
    25
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

29. What about Mahout?
    Think SAMOA = Mahout for streaming
    But SAMOA…
    More than JBoA (just a bunch of algorithms)
    Provides a common platform
    Easy to port to new computing engines
    26
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

30. Current Status
    Parallel algorithms
    Vertical Hoeffding Tree (classification)
    Clustream (clustering)
    PARMA (frequent pattern mining) [pending]
    Platforms
    S4 & Storm (Samza coming soon)
    Alpha version at https://github.com/yahoo/samoa
    27
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

31. Long-Term Goals
    Easy to integrate add-ons with packages (like R)
    Most common algorithms implemented (like Mahout)
    Large community in industry & academia (like Hadoop)
    Become reference platform for big data stream mining
    (like Weka)
    Lively open-source project (Apache Incubator)
    28
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

32. Challenges
    Algorithmic
    Platform design
    Implementation
    29
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

33. Algorithmic
    Case study: Vertical Hoeffding Tree
    What kind of parallelism?
    Task
    Data
    Horizontal
    Vertical
    30
    RGB color version - for online/web use
    3D Y-Bang Logo
    Instance
    Attributes
    Class
    

    View Slide

34. Task Parallelism
    31
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

35. Horizontal Parallelism
    Stats
    Stats
    Stats
    Stream
    Histograms
    Model
    Instances
    Y. Ben-Haim and E. Tom-Tov, “A Streaming Parallel Decision Tree Algorithm,”
    The Journal of Machine Learning Research, vol. 11, pp. 849–872, Mar. 2010. 32
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

36. Vertical Parallelism
    Stats
    Stats
    Stats
    Stream
    Model
    Attributes
    Splits
    33
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

37. Hoeffding Tree Profiling
    34
    RGB color version - for online/web use
    3D Y-Bang Logo
    Other
    6%
    Split
    24%
    Learn
    70%
    Training CPU time

    100 nominal and 100
    numeric attributes
    

    View Slide

38. Vertical Parallelism
    High number of attributes (e.g., documents) results in
    high level of parallelism
    Parallelism is observed immediately

    (compared to task parallelism)
    Localized failure handling and model updates

    (model is kept in one node)
    Less memory usage compared to horizontal
    partitioning (no model replication)
    35
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

39. Vertical Hoeffding Tree
    Control
    Split
    Result
    Source (n) Model (1) Stats (n) Evaluator (1)
    Instance
    Stream
    Shuffle Grouping
    Key Grouping
    All Grouping
    36
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

40. Accuracy
    37
    No. Leaf Nodes VHT2 –
    tree-100
    30
    Very close and
    very high accuracy
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

41. Performance
    38
    35
    0
    50
    100
    150
    200
    250
    MHT VHT2-par-3
    Execution Time (seconds)
    Classifier
    Profiling Results for text-10000
    with 100000 instances
    t_calc
    t_comm
    t_serial
    Throughput
    VHT2-par-3: 2631 inst/sec
    MHT : 507 inst/sec
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

42. Platform Design
    What is the right level of abstraction?
    Application building
    Computation
    Communication
    39
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

43. ML Developer API
    40
    RGB color version - for online/web use
    3D Y-Bang Logo
    Processing Item
    Processor
    Stream
    

    View Slide

44. ML Developer API
    TopologyBuilder builder;
    Processor sourceOne = new SourceProcessor();
    builder.addProcessor(sourceOne);
    Stream streamOne = builder.createStream(sourceOne);
    !
    Processor sourceTwo = new SourceProcessor();
    builder.addProcessor(sourceTwo);
    Stream streamTwo = builder.createStream(sourceTwo);
    !
    Processor join = new JoinProcessor());
    builder.addProcessor(join)
    .connectInputShuffle(streamOne)
    .connectInputKey(streamTwo);
    41
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

45. Implementation
    How to hide platform differences?
    Deployment
    Runtime
    How to isolate platform-related code?
    Build and release architecture
    42
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

46. Deployment
    SAMOA-S4.jar
    SAMOA-API.jar
    SAMOA-Storm.jar
    samoa-storm-deployable.jar
    samoa-s4-deployable.s4r
    S4 bindings
    Storm bindings
    API. Algorithm developer
    depends only on this
    To S4 cluster
    To Storm cluster
    43
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

47. Conclusions
    SAMOA: A Platform for Mining Big Data Streams
    Runs on existing distributed stream processing engines
    Parallel algorithms for mining data streams
    Pluggable architecture, flexible, extensible
    Open source and available (alpha release)
    44
    RGB color version - for online/web use
    3D Y-Bang Logo
    

    View Slide

48. Thanks!
    45
    RGB color version - for online/web use
    3D Y-Bang Logo
    gdfm@yahoo-inc.com
    https://github.com/yahoo/samoa
    @samoa_project
    

    View Slide