TME Introduction at OSDC.tw 2012

Trend Message Exchange realtime big data processing platform by Trend Micro. Now it is open sourced at http://trendmicro.github.com/tme/

Transcript

1. Trend Message Exchange:
   Open Source Realtime Big Data Processing Platform
   Kenneth Ho / fr3@k @ Trend Micro
   

   View Slide

2. TME Highlight
   • Not another MQ middleware
   • A new breed of message bus
   • Easy as hell integration model
   • Modeled after Unix's pipeline (pipe and filter)
   • Chain processes by file I/O (typically stdio
   streams)
   

   View Slide

3. Example
   • Word count in a Unix shell
   • $ cat log | wc -w
   • Distributed word count, producer(s) side:
   • $ cat log | mist-sink OPTIONS
   • Distributed word count, consumer(s) side:
   • $ mist-source OPTIONS | wc -w
   

   View Slide

4. Example
   • By distributing load to consumers, you get:
   • Application load balancing
   • Application high availability
   • For free
   

   View Slide

5. Key Features
   • Highly scalable and available
   • At least once delivery
   • Topology oriented routing (TME Graph)
   • No endpoint specific routing by design
   • No message ordering
   • Favors stateless processing
   • Enables failover in scale
   

   View Slide

6. TME: A Scalable Distributed
   Messaging System
   Scott Wang / lunastorm @ Trend Micro
   

   View Slide

7. Production since 2011
   

   View Slide

8. Agenda
   • Introduction to the TME Ecosystem
   • Messaging Integration STandard (MIST)
   • TME Graph (Workflow Management)
   • Related Work
   • Future Work
   

   View Slide

9. Traditional MQs
   Brokers
   Producer
   Consumer
   

   View Slide

10. Traditional MQs
    Peering
    

    View Slide

11. Traditional MQs
    

    View Slide

12. Traditional MQs
    

    View Slide

13. Traditional MQs
    Sacrifice
    Latency for
    Availability
    

    View Slide

14. Traditional MQs
    Cannot
    scale out
    horizontally
    

    View Slide

15. TME is a
    ecosystem!
    

    View Slide

16. View Slide

17. TME Brokers
    http://zookeeper.apache.org/
    Apache ZooKeeper
    

    View Slide

18. TME Brokers
    Sync Information
    

    View Slide

19. TME Brokers
    goto
    foo
    [foo]
    

    View Slide

20. TME Brokers
    [foo] receive
    from foo
    

    View Slide

21. TME Brokers
    [foo]
    

    View Slide

22. TME Brokers
    [foo]
    goto
    bar
    [bar]
    

    View Slide

23. TME Brokers
    [foo]
    [bar]
    receive
    from bar
    

    View Slide

24. TME Brokers
    [foo]
    [bar]
    

    View Slide

25. TME Brokers
    [foo]
    [bar]
    Scalability!
    

    View Slide

26. TME Brokers
    [foo]
    [bar]
    What if 恁
    data center
    火燒厝？　
    

    View Slide

27. TME Brokers
    [foo]
    [bar]
    

    View Slide

28. TME Brokers
    [foo, bar]
    Availability!
    

    View Slide

29. ZooKeeper
    MIST
    Producers
    Consumers
    MIST
    Producers
    Consumers
    MIST
    Producers
    Consumers
    MIST
    Producers
    Consumers
    Brokers
    TME Stack
    Overview
    

    View Slide

30. MIST
    MIST
    MIST
    MIST
    Message Integration
    STandard
    

    View Slide

31. View Slide

32. 4-Byte
    Network Order
    Integer Length
    Message Streaming Format
    message MessageBlock
    {
    required string id = 1;
    required bytes message = 2;
    optional int64 ttl = 3;
    repeated KeyValuePair properties = 4;
    }
    4-Byte
    Network Order
    Integer Length
    Message Block Message Block
    

    View Slide

33. You Call This Dead
    Easy?
    

    View Slide

34. 余憶童稚時...
    #include
    #include
    int main() {
    std::string name;
    std::cin >> name;
    std::cout << "Hello! " << name;
    }
    

    View Slide

35. 曾幾何時...
    #include
    #include
    #include
    #include
    #include
    #include
    #include
    #include
    
    int main(void){
    struct sockaddr_in stSockAddr;
    int SocketFD = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP);
    memset(&stSockAddr, 0, sizeof(stSockAddr));
    stSockAddr.sin_family = AF_INET;
    stSockAddr.sin_port = htons(5566);
    stSockAddr.sin_addr.s_addr = INADDR_ANY;
    bind(SocketFD,(const void *)&stSockAddr, sizeof(stSockAddr))
    listen(SocketFD, 10)
    int ConnectFD = accept(SocketFD, NULL, NULL);
    
    //do something
    }
    

    View Slide

36. You do not have to
    implement the protocol!
    • We have a set of command line tools
    • Use simple file IO for everything!
    

    View Slide

37. MIST Tools
    • A set of command line tools to get / send
    messages from TME
    • In the UNIX world...
    • ls | grep "something" | wc -l
    • With MIST, we can also...
    • mist-source | some_filter | mist-sink
    

    View Slide

38. MIST Tools
    • Hides all complexity of connecting to
    message brokers
    • Just specify what queue you would like to
    subscribe, and your output message IDs
    

    View Slide

39. MIST Tools
    • Dump messages to a file
    • mist-source > messages.dump
    • Send the messages from a file
    • mist-sink < messages.dump
    

    View Slide

40. MIST Tools
    • Easy to test application logic
    • From
    • mist-source | foo | mist-sink -> TME ->
    mist-source | bar
    • To
    • foo | bar
    

    View Slide

41. One-Linerholic
    • src_id=$(mist-session) && sink_id=$(mist-
    session) && mist-source $src_id --mount
    apache_log && mist-source $src_id --attach |
    mist-decode --line | sed -e '/404/!d' | mist-
    encode --wrap 404_request --line | mist-sink
    $sink_sid --attach ; mist-session --destroy
    $src_id ; mist-session --destroy $sink_id
    

    View Slide

42. TME Graph
    Let's talk about workflow
    

    View Slide

43. foo.out
    ProducerFoo LogArchiver
    一個發生在企業內部的
    小小故事
    

    View Slide

44. foo.out
    ProducerFoo LogArchiver
    ProducerBar
    bar.out
    

    View Slide

45. foo.out
    ProducerFoo
    LogArchiver
    ProducerBar
    bar.out
    

    View Slide

46. foo.out
    ProducerFoo
    LogArchiver
    ProducerBar bar.out
    log.in
    Router
    

    View Slide

47. foo.out
    ProducerFoo
    LogArchiver
    ProducerBar bar.out
    log.in
    Router
    

    View Slide

48. foo.out
    ProducerFoo
    LogArchiver
    ProducerBar bar.out
    log.in
    Implicit Message
    Forwarding
    

    View Slide

49. foo.out
    bar.out
    log.in
    apache_log.out
    syslog.out
    intrusion
    detect.in
    intrusion
    detect.out
    alert.in
    deadlink
    detect.in
    deadlink
    detect.in
    Extend Your App Freely
    

    View Slide

50. Future Work
    • More Documents
    • More Platforms
    • CI Server
    • Tests
    

    View Slide

51. Future Work
    #include
    #include
    using namespace std;
    
    void main(){
    string line;
    while(getline(cin, line)){
    // do something with the message
    if(line.find("法轮大法好") != string::npos){
    cout << line << endl;
    }
    }
    }
    import sys, java.lang
    from org.apache.hadoop.hbase import HBase
    Configuration, client.HTable
    
    table = HTable(HBaseConfiguration.create(),
    'gold_shield')
    while True:
    line = sys.stdin.readline()
    if not line:
    break
    table.incrementColumnValue(row_key,
    family , log_dict[family], 1)
    table.close()
    Web IDE for cloud!
    

    View Slide

52. Contribute?
    • Issue tracker on GitHub
    • Fork & Exec
    • Pull requests!!
    • Easy enough, huh?
    

    View Slide

53. Thank You!
    Questions?
    

    View Slide

54. • https://github.com/trendmicro/tme/downloads
    • http://trendmicro.github.com/tme/
    3rd_parties.html
    

    View Slide