TME Introduction at OSDC.tw 2012

Trend Message Exchange realtime big data processing platform by Trend Micro. Now it is open sourced at http://trendmicro.github.com/tme/

Transcript

1. Trend Message Exchange: Open Source Realtime Big Data Processing Platform Kenneth Ho / fr3@k @ Trend Micro
2. TME Highlight • Not another MQ middleware • A new breed of message bus • Easy as hell integration model • Modeled after Unix's pipeline (pipe and filter) • Chain processes by file I/O (typically stdio streams)
3. Example • Word count in a Unix shell • $ cat log | wc -w • Distributed word count, producer(s) side: • $ cat log | mist-sink OPTIONS • Distributed word count, consumer(s) side: • $ mist-source OPTIONS | wc -w
4. Example • By distributing load to consumers, you get: • Application load balancing • Application high availability • For free
5. Key Features • Highly scalable and available • At least once delivery • Topology oriented routing (TME Graph) • No endpoint specific routing by design • No message ordering • Favors stateless processing • Enables failover in scale
6. TME: A Scalable Distributed Messaging System Scott Wang / lunastorm @ Trend Micro
7. Production since 2011
8. Agenda • Introduction to the TME Ecosystem • Messaging Integration STandard (MIST) • TME Graph (Workflow Management) • Related Work • Future Work
9. Traditional MQs Brokers Producer Consumer
10. Traditional MQs Peering
11. Traditional MQs
12. Traditional MQs
13. Traditional MQs Sacrifice Latency for Availability
14. Traditional MQs Cannot scale out horizontally
15. TME is a ecosystem!
17. TME Brokers http://zookeeper.apache.org/ Apache ZooKeeper
18. TME Brokers Sync Information
19. TME Brokers goto foo [foo]
20. TME Brokers [foo] receive from foo
21. TME Brokers [foo]
22. TME Brokers [foo] goto bar [bar]
23. TME Brokers [foo] [bar] receive from bar
24. TME Brokers [foo] [bar]
25. TME Brokers [foo] [bar] Scalability!
26. TME Brokers [foo] [bar] What if 恁 data center 火燒厝？　
27. TME Brokers [foo] [bar]
28. TME Brokers [foo, bar] Availability!
29. ZooKeeper MIST Producers Consumers MIST Producers Consumers MIST Producers Consumers MIST Producers Consumers Brokers TME Stack Overview
30. MIST MIST MIST MIST Message Integration STandard
32. 4-Byte Network Order Integer Length Message Streaming Format message MessageBlock { required string id = 1; required bytes message = 2; optional int64 ttl = 3; repeated KeyValuePair properties = 4; } 4-Byte Network Order Integer Length Message Block Message Block
33. You Call This Dead Easy?
34. 余憶童稚時... #include <iostream> #include <string> int main() { std::string name; std::cin >> name; std::cout << "Hello! " << name; }
35. 曾幾何時... #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <arpa/inet.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> int main(void){ struct sockaddr_in stSockAddr; int SocketFD = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP); memset(&stSockAddr, 0, sizeof(stSockAddr)); stSockAddr.sin_family = AF_INET; stSockAddr.sin_port = htons(5566); stSockAddr.sin_addr.s_addr = INADDR_ANY; bind(SocketFD,(const void *)&stSockAddr, sizeof(stSockAddr)) listen(SocketFD, 10) int ConnectFD = accept(SocketFD, NULL, NULL); //do something }
36. You do not have to implement the protocol! • We have a set of command line tools • Use simple file IO for everything!
37. MIST Tools • A set of command line tools to get / send messages from TME • In the UNIX world... • ls | grep "something" | wc -l • With MIST, we can also... • mist-source | some_filter | mist-sink
38. MIST Tools • Hides all complexity of connecting to message brokers • Just specify what queue you would like to subscribe, and your output message IDs
39. MIST Tools • Dump messages to a file • mist-source > messages.dump • Send the messages from a file • mist-sink < messages.dump
40. MIST Tools • Easy to test application logic • From • mist-source | foo | mist-sink -> TME -> mist-source | bar • To • foo | bar
41. One-Linerholic • src_id=$(mist-session) && sink_id=$(mist- session) && mist-source $src_id --mount apache_log && mist-source $src_id --attach | mist-decode --line | sed -e '/404/!d' | mist- encode --wrap 404_request --line | mist-sink $sink_sid --attach ; mist-session --destroy $src_id ; mist-session --destroy $sink_id
42. TME Graph Let's talk about workflow
43. foo.out ProducerFoo LogArchiver 一個發生在企業內部的 小小故事
44. foo.out ProducerFoo LogArchiver ProducerBar bar.out
45. foo.out ProducerFoo LogArchiver ProducerBar bar.out
46. foo.out ProducerFoo LogArchiver ProducerBar bar.out log.in Router
47. foo.out ProducerFoo LogArchiver ProducerBar bar.out log.in Router
48. foo.out ProducerFoo LogArchiver ProducerBar bar.out log.in Implicit Message Forwarding
49. foo.out bar.out log.in apache_log.out syslog.out intrusion detect.in intrusion detect.out alert.in deadlink detect.in deadlink detect.in Extend Your App Freely
50. Future Work • More Documents • More Platforms • CI Server • Tests
51. Future Work #include<iostream> #include<string> using namespace std; void main(){ string line; while(getline(cin, line)){ // do something with the message if(line.find("法轮大法好") != string::npos){ cout << line << endl; } } } import sys, java.lang from org.apache.hadoop.hbase import HBase Configuration, client.HTable table = HTable(HBaseConfiguration.create(), 'gold_shield') while True: line = sys.stdin.readline() if not line: break table.incrementColumnValue(row_key, family , log_dict[family], 1) table.close() Web IDE for cloud!
52. Contribute? • Issue tracker on GitHub • Fork & Exec • Pull requests!! • Easy enough, huh?
53. Thank You! Questions?
54. • https://github.com/trendmicro/tme/downloads • http://trendmicro.github.com/tme/ 3rd_parties.html