TME Introduction at OSDC.tw 2012

TME Introduction at OSDC.tw 2012

Trend Message Exchange realtime big data processing platform by Trend Micro. Now it is open sourced at http://trendmicro.github.com/tme/

3f44db9af00adf3dd3e209f1ca59d7fe?s=128

lunastorm

April 18, 2012
Tweet

Transcript

  1. Trend Message Exchange: Open Source Realtime Big Data Processing Platform

    Kenneth Ho / fr3@k @ Trend Micro
  2. TME Highlight • Not another MQ middleware • A new

    breed of message bus • Easy as hell integration model • Modeled after Unix's pipeline (pipe and filter) • Chain processes by file I/O (typically stdio streams)
  3. Example • Word count in a Unix shell • $

    cat log | wc -w • Distributed word count, producer(s) side: • $ cat log | mist-sink OPTIONS • Distributed word count, consumer(s) side: • $ mist-source OPTIONS | wc -w
  4. Example • By distributing load to consumers, you get: •

    Application load balancing • Application high availability • For free
  5. Key Features • Highly scalable and available • At least

    once delivery • Topology oriented routing (TME Graph) • No endpoint specific routing by design • No message ordering • Favors stateless processing • Enables failover in scale
  6. TME: A Scalable Distributed Messaging System Scott Wang / lunastorm

    @ Trend Micro
  7. Production since 2011

  8. Agenda • Introduction to the TME Ecosystem • Messaging Integration

    STandard (MIST) • TME Graph (Workflow Management) • Related Work • Future Work
  9. Traditional MQs Brokers Producer Consumer

  10. Traditional MQs Peering

  11. Traditional MQs

  12. Traditional MQs

  13. Traditional MQs Sacrifice Latency for Availability

  14. Traditional MQs Cannot scale out horizontally

  15. TME is a ecosystem!

  16. None
  17. TME Brokers http://zookeeper.apache.org/ Apache ZooKeeper

  18. TME Brokers Sync Information

  19. TME Brokers goto foo [foo]

  20. TME Brokers [foo] receive from foo

  21. TME Brokers [foo]

  22. TME Brokers [foo] goto bar [bar]

  23. TME Brokers [foo] [bar] receive from bar

  24. TME Brokers [foo] [bar]

  25. TME Brokers [foo] [bar] Scalability!

  26. TME Brokers [foo] [bar] What if 恁 data center 火燒厝? 

  27. TME Brokers [foo] [bar]

  28. TME Brokers [foo, bar] Availability!

  29. ZooKeeper MIST Producers Consumers MIST Producers Consumers MIST Producers Consumers

    MIST Producers Consumers Brokers TME Stack Overview
  30. MIST MIST MIST MIST Message Integration STandard

  31. None
  32. 4-Byte Network Order Integer Length Message Streaming Format message MessageBlock

    { required string id = 1; required bytes message = 2; optional int64 ttl = 3; repeated KeyValuePair properties = 4; } 4-Byte Network Order Integer Length Message Block Message Block
  33. You Call This Dead Easy?

  34. 余憶童稚時... #include <iostream> #include <string> int main() { std::string name;

    std::cin >> name; std::cout << "Hello! " << name; }
  35. 曾幾何時... #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <arpa/inet.h> #include

    <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> int main(void){ struct sockaddr_in stSockAddr; int SocketFD = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP); memset(&stSockAddr, 0, sizeof(stSockAddr)); stSockAddr.sin_family = AF_INET; stSockAddr.sin_port = htons(5566); stSockAddr.sin_addr.s_addr = INADDR_ANY; bind(SocketFD,(const void *)&stSockAddr, sizeof(stSockAddr)) listen(SocketFD, 10) int ConnectFD = accept(SocketFD, NULL, NULL); //do something }
  36. You do not have to implement the protocol! • We

    have a set of command line tools • Use simple file IO for everything!
  37. MIST Tools • A set of command line tools to

    get / send messages from TME • In the UNIX world... • ls | grep "something" | wc -l • With MIST, we can also... • mist-source | some_filter | mist-sink
  38. MIST Tools • Hides all complexity of connecting to message

    brokers • Just specify what queue you would like to subscribe, and your output message IDs
  39. MIST Tools • Dump messages to a file • mist-source

    > messages.dump • Send the messages from a file • mist-sink < messages.dump
  40. MIST Tools • Easy to test application logic • From

    • mist-source | foo | mist-sink -> TME -> mist-source | bar • To • foo | bar
  41. One-Linerholic • src_id=$(mist-session) && sink_id=$(mist- session) && mist-source $src_id --mount

    apache_log && mist-source $src_id --attach | mist-decode --line | sed -e '/404/!d' | mist- encode --wrap 404_request --line | mist-sink $sink_sid --attach ; mist-session --destroy $src_id ; mist-session --destroy $sink_id
  42. TME Graph Let's talk about workflow

  43. foo.out ProducerFoo LogArchiver 一個發生在企業內部的 小小故事

  44. foo.out ProducerFoo LogArchiver ProducerBar bar.out

  45. foo.out ProducerFoo LogArchiver ProducerBar bar.out

  46. foo.out ProducerFoo LogArchiver ProducerBar bar.out log.in Router

  47. foo.out ProducerFoo LogArchiver ProducerBar bar.out log.in Router

  48. foo.out ProducerFoo LogArchiver ProducerBar bar.out log.in Implicit Message Forwarding

  49. foo.out bar.out log.in apache_log.out syslog.out intrusion detect.in intrusion detect.out alert.in

    deadlink detect.in deadlink detect.in Extend Your App Freely
  50. Future Work • More Documents • More Platforms • CI

    Server • Tests
  51. Future Work #include<iostream> #include<string> using namespace std; void main(){ string

    line; while(getline(cin, line)){ // do something with the message if(line.find("法轮大法好") != string::npos){ cout << line << endl; } } } import sys, java.lang from org.apache.hadoop.hbase import HBase Configuration, client.HTable table = HTable(HBaseConfiguration.create(), 'gold_shield') while True: line = sys.stdin.readline() if not line: break table.incrementColumnValue(row_key, family , log_dict[family], 1) table.close() Web IDE for cloud!
  52. Contribute? • Issue tracker on GitHub • Fork & Exec

    • Pull requests!! • Easy enough, huh?
  53. Thank You! Questions?

  54. • https://github.com/trendmicro/tme/downloads • http://trendmicro.github.com/tme/ 3rd_parties.html