TME Introduction at OSDC.tw 2012

Transcript

1. Trend Message Exchange: Open Source Realtime Big Data Processing Platform

   Kenneth Ho / fr3@k @ Trend Micro

2. TME Highlight • Not another MQ middleware • A new

   breed of message bus • Easy as hell integration model • Modeled after Unix's pipeline (pipe and filter) • Chain processes by file I/O (typically stdio streams)

3. Example • Word count in a Unix shell • $

   cat log | wc -w • Distributed word count, producer(s) side: • $ cat log | mist-sink OPTIONS • Distributed word count, consumer(s) side: • $ mist-source OPTIONS | wc -w

4. Example • By distributing load to consumers, you get: •

   Application load balancing • Application high availability • For free

5. Key Features • Highly scalable and available • At least

   once delivery • Topology oriented routing (TME Graph) • No endpoint specific routing by design • No message ordering • Favors stateless processing • Enables failover in scale

6. TME: A Scalable Distributed Messaging System Scott Wang / lunastorm

   @ Trend Micro

7. Production since 2011

8. Agenda • Introduction to the TME Ecosystem • Messaging Integration

   STandard (MIST) • TME Graph (Workflow Management) • Related Work • Future Work

9. Traditional MQs Brokers Producer Consumer

10. Traditional MQs Peering

11. Traditional MQs

12. Traditional MQs

13. Traditional MQs Sacrifice Latency for Availability

14. Traditional MQs Cannot scale out horizontally

15. TME is a ecosystem!

16. None

17. TME Brokers http://zookeeper.apache.org/ Apache ZooKeeper

18. TME Brokers Sync Information

19. TME Brokers goto foo [foo]

20. TME Brokers [foo] receive from foo

21. TME Brokers [foo]

22. TME Brokers [foo] goto bar [bar]

23. TME Brokers [foo] [bar] receive from bar

24. TME Brokers [foo] [bar]

25. TME Brokers [foo] [bar] Scalability!

26. TME Brokers [foo] [bar] What if 恁 data center 火燒厝？　

27. TME Brokers [foo] [bar]

28. TME Brokers [foo, bar] Availability!

29. ZooKeeper MIST Producers Consumers MIST Producers Consumers MIST Producers Consumers

    MIST Producers Consumers Brokers TME Stack Overview

30. MIST MIST MIST MIST Message Integration STandard

31. None

32. 4-Byte Network Order Integer Length Message Streaming Format message MessageBlock

    { required string id = 1; required bytes message = 2; optional int64 ttl = 3; repeated KeyValuePair properties = 4; } 4-Byte Network Order Integer Length Message Block Message Block

33. You Call This Dead Easy?

34. 余憶童稚時... #include <iostream> #include <string> int main() { std::string name;

    std::cin >> name; std::cout << "Hello! " << name; }

35. 曾幾何時... #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <arpa/inet.h> #include

    <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> int main(void){ struct sockaddr_in stSockAddr; int SocketFD = socket(PF_INET, SOCK_STREAM, IPPROTO_TCP); memset(&stSockAddr, 0, sizeof(stSockAddr)); stSockAddr.sin_family = AF_INET; stSockAddr.sin_port = htons(5566); stSockAddr.sin_addr.s_addr = INADDR_ANY; bind(SocketFD,(const void *)&stSockAddr, sizeof(stSockAddr)) listen(SocketFD, 10) int ConnectFD = accept(SocketFD, NULL, NULL); //do something }

36. You do not have to implement the protocol! • We

    have a set of command line tools • Use simple file IO for everything!

37. MIST Tools • A set of command line tools to

    get / send messages from TME • In the UNIX world... • ls | grep "something" | wc -l • With MIST, we can also... • mist-source | some_filter | mist-sink

38. MIST Tools • Hides all complexity of connecting to message

    brokers • Just specify what queue you would like to subscribe, and your output message IDs

39. MIST Tools • Dump messages to a file • mist-source

    > messages.dump • Send the messages from a file • mist-sink < messages.dump

40. MIST Tools • Easy to test application logic • From

    • mist-source | foo | mist-sink -> TME -> mist-source | bar • To • foo | bar

41. One-Linerholic • src_id=$(mist-session) && sink_id=$(mist- session) && mist-source $src_id --mount

    apache_log && mist-source $src_id --attach | mist-decode --line | sed -e '/404/!d' | mist- encode --wrap 404_request --line | mist-sink $sink_sid --attach ; mist-session --destroy $src_id ; mist-session --destroy $sink_id

42. TME Graph Let's talk about workflow

43. foo.out ProducerFoo LogArchiver 一個發生在企業內部的 小小故事

44. foo.out ProducerFoo LogArchiver ProducerBar bar.out

45. foo.out ProducerFoo LogArchiver ProducerBar bar.out

46. foo.out ProducerFoo LogArchiver ProducerBar bar.out log.in Router

47. foo.out ProducerFoo LogArchiver ProducerBar bar.out log.in Router

48. foo.out ProducerFoo LogArchiver ProducerBar bar.out log.in Implicit Message Forwarding

49. foo.out bar.out log.in apache_log.out syslog.out intrusion detect.in intrusion detect.out alert.in

    deadlink detect.in deadlink detect.in Extend Your App Freely

50. Future Work • More Documents • More Platforms • CI

    Server • Tests

51. Future Work #include<iostream> #include<string> using namespace std; void main(){ string

    line; while(getline(cin, line)){ // do something with the message if(line.find("法轮大法好") != string::npos){ cout << line << endl; } } } import sys, java.lang from org.apache.hadoop.hbase import HBase Configuration, client.HTable table = HTable(HBaseConfiguration.create(), 'gold_shield') while True: line = sys.stdin.readline() if not line: break table.incrementColumnValue(row_key, family , log_dict[family], 1) table.close() Web IDE for cloud!

52. Contribute? • Issue tracker on GitHub • Fork & Exec

    • Pull requests!! • Easy enough, huh?

53. Thank You! Questions?

54. • https://github.com/trendmicro/tme/downloads • http://trendmicro.github.com/tme/ 3rd_parties.html