$30 off During Our Annual Pro Sale. View Details »

TME Introduction at OSDC.tw 2012

TME Introduction at OSDC.tw 2012

Trend Message Exchange realtime big data processing platform by Trend Micro. Now it is open sourced at http://trendmicro.github.com/tme/

lunastorm

April 18, 2012
Tweet

Other Decks in Programming

Transcript

  1. Trend Message Exchange:
    Open Source Realtime Big Data Processing Platform
    Kenneth Ho / fr3@k @ Trend Micro

    View Slide

  2. TME Highlight
    • Not another MQ middleware
    • A new breed of message bus
    • Easy as hell integration model
    • Modeled after Unix's pipeline (pipe and filter)
    • Chain processes by file I/O (typically stdio
    streams)

    View Slide

  3. Example
    • Word count in a Unix shell
    • $ cat log | wc -w
    • Distributed word count, producer(s) side:
    • $ cat log | mist-sink OPTIONS
    • Distributed word count, consumer(s) side:
    • $ mist-source OPTIONS | wc -w

    View Slide

  4. Example
    • By distributing load to consumers, you get:
    • Application load balancing
    • Application high availability
    • For free

    View Slide

  5. Key Features
    • Highly scalable and available
    • At least once delivery
    • Topology oriented routing (TME Graph)
    • No endpoint specific routing by design
    • No message ordering
    • Favors stateless processing
    • Enables failover in scale

    View Slide

  6. TME: A Scalable Distributed
    Messaging System
    Scott Wang / lunastorm @ Trend Micro

    View Slide

  7. Production since 2011

    View Slide

  8. Agenda
    • Introduction to the TME Ecosystem
    • Messaging Integration STandard (MIST)
    • TME Graph (Workflow Management)
    • Related Work
    • Future Work

    View Slide

  9. Traditional MQs
    Brokers
    Producer
    Consumer

    View Slide

  10. Traditional MQs
    Peering

    View Slide

  11. Traditional MQs

    View Slide

  12. Traditional MQs

    View Slide

  13. Traditional MQs
    Sacrifice
    Latency for
    Availability

    View Slide

  14. Traditional MQs
    Cannot
    scale out
    horizontally

    View Slide

  15. TME is a
    ecosystem!

    View Slide

  16. View Slide

  17. TME Brokers
    http://zookeeper.apache.org/
    Apache ZooKeeper

    View Slide

  18. TME Brokers
    Sync Information

    View Slide

  19. TME Brokers
    goto
    foo
    [foo]

    View Slide

  20. TME Brokers
    [foo] receive
    from foo

    View Slide

  21. TME Brokers
    [foo]

    View Slide

  22. TME Brokers
    [foo]
    goto
    bar
    [bar]

    View Slide

  23. TME Brokers
    [foo]
    [bar]
    receive
    from bar

    View Slide

  24. TME Brokers
    [foo]
    [bar]

    View Slide

  25. TME Brokers
    [foo]
    [bar]
    Scalability!

    View Slide

  26. TME Brokers
    [foo]
    [bar]
    What if 恁
    data center
    火燒厝? 

    View Slide

  27. TME Brokers
    [foo]
    [bar]

    View Slide

  28. TME Brokers
    [foo, bar]
    Availability!

    View Slide

  29. ZooKeeper
    MIST
    Producers
    Consumers
    MIST
    Producers
    Consumers
    MIST
    Producers
    Consumers
    MIST
    Producers
    Consumers
    Brokers
    TME Stack
    Overview

    View Slide

  30. MIST
    MIST
    MIST
    MIST
    Message Integration
    STandard

    View Slide

  31. View Slide

  32. 4-Byte
    Network Order
    Integer Length
    Message Streaming Format
    message MessageBlock
    {
    required string id = 1;
    required bytes message = 2;
    optional int64 ttl = 3;
    repeated KeyValuePair properties = 4;
    }
    4-Byte
    Network Order
    Integer Length
    Message Block Message Block

    View Slide

  33. You Call This Dead
    Easy?

    View Slide

  34. 余憶童稚時...
    #include
    #include
    int main() {
    std::string name;
    std::cin >> name;
    std::cout << "Hello! " << name;
    }

    View Slide

  35. 曾幾何時...
    #include
    #include
    #include
    #include
    #include
    #include
    #include
    #include

    int main(void){
    struct sockaddr_in stSockAddr;
    int SocketFD = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);
    memset(&stSockAddr, 0, sizeof(stSockAddr));
    stSockAddr.sin_family = AF_INET;
    stSockAddr.sin_port = htons(5566);
    stSockAddr.sin_addr.s_addr = INADDR_ANY;
    bind(SocketFD,(const void *)&stSockAddr, sizeof(stSockAddr))
    listen(SocketFD, 10)
    int ConnectFD = accept(SocketFD, NULL, NULL);

    //do something
    }

    View Slide

  36. You do not have to
    implement the protocol!
    • We have a set of command line tools
    • Use simple file IO for everything!

    View Slide

  37. MIST Tools
    • A set of command line tools to get / send
    messages from TME
    • In the UNIX world...
    • ls | grep "something" | wc -l
    • With MIST, we can also...
    • mist-source | some_filter | mist-sink

    View Slide

  38. MIST Tools
    • Hides all complexity of connecting to
    message brokers
    • Just specify what queue you would like to
    subscribe, and your output message IDs

    View Slide

  39. MIST Tools
    • Dump messages to a file
    • mist-source > messages.dump
    • Send the messages from a file
    • mist-sink < messages.dump

    View Slide

  40. MIST Tools
    • Easy to test application logic
    • From
    • mist-source | foo | mist-sink -> TME ->
    mist-source | bar
    • To
    • foo | bar

    View Slide

  41. One-Linerholic
    • src_id=$(mist-session) && sink_id=$(mist-
    session) && mist-source $src_id --mount
    apache_log && mist-source $src_id --attach |
    mist-decode --line | sed -e '/404/!d' | mist-
    encode --wrap 404_request --line | mist-sink
    $sink_sid --attach ; mist-session --destroy
    $src_id ; mist-session --destroy $sink_id

    View Slide

  42. TME Graph
    Let's talk about workflow

    View Slide

  43. foo.out
    ProducerFoo LogArchiver
    一個發生在企業內部的
    小小故事

    View Slide

  44. foo.out
    ProducerFoo LogArchiver
    ProducerBar
    bar.out

    View Slide

  45. foo.out
    ProducerFoo
    LogArchiver
    ProducerBar
    bar.out

    View Slide

  46. foo.out
    ProducerFoo
    LogArchiver
    ProducerBar bar.out
    log.in
    Router

    View Slide

  47. foo.out
    ProducerFoo
    LogArchiver
    ProducerBar bar.out
    log.in
    Router

    View Slide

  48. foo.out
    ProducerFoo
    LogArchiver
    ProducerBar bar.out
    log.in
    Implicit Message
    Forwarding

    View Slide

  49. foo.out
    bar.out
    log.in
    apache_log.out
    syslog.out
    intrusion
    detect.in
    intrusion
    detect.out
    alert.in
    deadlink
    detect.in
    deadlink
    detect.in
    Extend Your App Freely

    View Slide

  50. Future Work
    • More Documents
    • More Platforms
    • CI Server
    • Tests

    View Slide

  51. Future Work
    #include
    #include
    using namespace std;

    void main(){
    string line;
    while(getline(cin, line)){
    // do something with the message
    if(line.find("法轮大法好") != string::npos){
    cout << line << endl;
    }
    }
    }
    import sys, java.lang
    from org.apache.hadoop.hbase import HBase
    Configuration, client.HTable

    table = HTable(HBaseConfiguration.create(),
    'gold_shield')
    while True:
    line = sys.stdin.readline()
    if not line:
    break
    table.incrementColumnValue(row_key,
    family , log_dict[family], 1)
    table.close()
    Web IDE for cloud!

    View Slide

  52. Contribute?
    • Issue tracker on GitHub
    • Fork & Exec
    • Pull requests!!
    • Easy enough, huh?

    View Slide

  53. Thank You!
    Questions?

    View Slide

  54. • https://github.com/trendmicro/tme/downloads
    • http://trendmicro.github.com/tme/
    3rd_parties.html

    View Slide