MongooseIM - Messaging that Scales

MongooseIM provides a unique instant messaging solution when massive volume is required. The growth of social media communication has resulted in the fact that the ceiling is quickly reached when successful projects take off. MongooseIM secures performance and stability.

In this talk I present lessons learnt when scaling messaging servers using common PC, cloud environments or a dedicated data centre. I demonstrate what bottlenecks we encountered and how to work around them in order to scale to millions of users. I also explain why MongooseIM and Erlang are a good baseline for building a custom messaging solution.

Talk objectives: to educate about MongooseIM, which is built around proven technology Jabber/XMPP, as well as discuss how to build a messaging solution, which will scale to big numbers.

Target audience: People who know Jabber/XMPP or are interested in messaging servers.

Transcript

1. © 1999-2013 Erlang Solutions Ltd. MongooseIM - Messaging that Scales Michał Ślaski Friday, 24 May 13
2. © 1999-2013 Erlang Solutions Ltd. What is MongooseIM? 2 Friday, 24 May 13
3. © 1999-2013 Erlang Solutions Ltd. What is MongooseIM? •Instant messaging for Social Media, Gaming and Telecommunications 2 Friday, 24 May 13
4. © 1999-2013 Erlang Solutions Ltd. What is MongooseIM? •Instant messaging for Social Media, Gaming and Telecommunications •Solution designed for high volume 2 Friday, 24 May 13
5. © 1999-2013 Erlang Solutions Ltd. What is MongooseIM? •Instant messaging for Social Media, Gaming and Telecommunications •Solution designed for high volume •Easily scalable distributed system 2 Friday, 24 May 13
6. © 1999-2013 Erlang Solutions Ltd. What is MongooseIM? •Instant messaging for Social Media, Gaming and Telecommunications •Solution designed for high volume •Easily scalable distributed system •Highly customisable platform thanks to industry standards and Open Source technology 2 Friday, 24 May 13
7. © 1999-2013 Erlang Solutions Ltd. Target domains 3 Friday, 24 May 13
8. © 1999-2013 Erlang Solutions Ltd. Target domains 3 Telecoms (chat features are a part of standard commmunication software) Friday, 24 May 13
9. © 1999-2013 Erlang Solutions Ltd. Target domains 3 Telecoms (chat features are a part of standard commmunication software) Social Media (messaging is the key part of social web sites) Friday, 24 May 13
10. © 1999-2013 Erlang Solutions Ltd. Target domains 3 Telecoms (chat features are a part of standard commmunication software) Social Media (messaging is the key part of social web sites) Gaming (multi-user chats are an integral part of the gameplay and enhance the user experience) Friday, 24 May 13
11. © 1999-2013 Erlang Solutions Ltd. Easy to work with •Support of XMPP - a protocol used for Facebook chat and Google Talk - seamless integration with many libraries and client applications 4 Friday, 24 May 13
12. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure Flexible Decentralised Efficient Proven 5 Friday, 24 May 13
13. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure Flexible Decentralised Efficient Proven 5 Standard formalised by the IETF through RFCs and extensions are published by the XSF Friday, 24 May 13
14. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure Flexible Decentralised Efficient Proven 5 Channel encryption, strong authentication, may be isolated from the public network Friday, 24 May 13
15. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure Flexible Decentralised Efficient Proven 5 Custom functionality can be built on top of XMPP Friday, 24 May 13
16. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure Flexible Decentralised Efficient Proven 5 Anyone can run XMPP server and there is no central master server Friday, 24 May 13
17. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure Flexible Decentralised Efficient Proven 5 Solves issues of polling approaches Friday, 24 May 13
18. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure Flexible Decentralised Efficient Proven 5 In use since 1998, many implementations and deployments, millions of end users Friday, 24 May 13
19. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure Flexible Decentralised Efficient Proven 5 Friday, 24 May 13
20. © 1999-2013 Erlang Solutions Ltd. Features •Support for WebSockets - ready for modern chat- enabled web applications •Multi-User Chat (XEP-0045) •Service Discovery (XEP-0030) •vCard (XEP-0054) •Privacy Lists (XEP-0016) •Private XML Storage (XEP-0049) 6 Friday, 24 May 13
21. © 1999-2013 Erlang Solutions Ltd. Extensible •Customisable - provides a base for bespoke solution to one’s specific needs - push notifications to mobile devices - authentication plugins - reliable message delivery in mobile networks •Has open sourced regression tests - github.com/esl/ejabberd_tests 7 Friday, 24 May 13
22. © 1999-2013 Erlang Solutions Ltd. Scalable •Configurable database backends - Mnesia for simple deployments - MySQL for persistent data - Mnesia or Redis for transient data 8 Friday, 24 May 13
23. © 1999-2013 Erlang Solutions Ltd. Scalable 9 Friday, 24 May 13
24. © 1999-2013 Erlang Solutions Ltd. Load tests - environment Test environment Erlang/OTP R15B02 MongooseIM 1.1 MySQL 5.5.24 Ubuntu 12.04 LTS 3.2.0-23-generic Tsung 1.5.0a redis 2.6.10 ejabberd 2.1.11 10 Friday, 24 May 13
25. © 1999-2013 Erlang Solutions Ltd. Load tests - dedicated box 11 MongooseIM AMD 8x3.3GHz, 32GB RAM MySQL Friday, 24 May 13
26. © 1999-2013 Erlang Solutions Ltd. Load tests - “max users” User arrival rate: 150/s User count: 400k Roster size: 100 No message exchange The aim was to check resource usage with maximum possible count of online users connected to server. 12 Friday, 24 May 13
27. © 1999-2013 Erlang Solutions Ltd. Load tests - “max users” 13 Friday, 24 May 13
28. © 1999-2013 Erlang Solutions Ltd. Load tests - “max users” 14 Friday, 24 May 13
29. © 1999-2013 Erlang Solutions Ltd. Load tests - “max message rate” User arrival rate: 100/s User count: 75k Roster size: 100 Message rate: ~21k per second The aim was to check resource usage when maximum message traffic is generated. 15 Friday, 24 May 13
30. © 1999-2013 Erlang Solutions Ltd. Load tests - “max message rate” 16 Friday, 24 May 13
31. © 1999-2013 Erlang Solutions Ltd. Load tests - “max message rate” 17 Friday, 24 May 13
32. © 1999-2013 Erlang Solutions Ltd. Load tests - Amazon EC2 •M1 Extra Large Instance - 15 GiB memory - 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each) - I/O Performance: High 18 Friday, 24 May 13
33. © 1999-2013 Erlang Solutions Ltd. Load tests - Amazon EC2 19 MongooseIM m1.xlarge MongooseIM m1.xlarge MongooseIM m1.xlarge Friday, 24 May 13
34. © 1999-2013 Erlang Solutions Ltd. Load tests - 3 MongooseIM + Mnesia User arrival rate: 130/s User count: 50k Roster size: 100 Presence rate: 120 per second (12k msg/s) The aim was to check resource usage with all user data like passwords and rosters in Mnesia. 20 Friday, 24 May 13
35. © 1999-2013 Erlang Solutions Ltd. 21 %" 10%" 20%" 30%" 40%" 50%" 60%" 70%" 80%" 90%" 100%" 1" 23" 45" 67" 89" 111" 133" 155" 177" 199" 221" 243" 265" 287" 309" 331" 353" 375" 397" 419" 441" 463" 485" 507" 529" 551" 573" 595" 617" 639" 661" 683" 705" 727" 749" 771" 793" 815" 837" 859" 881" 903" 925" 947" 969" 991" 1013" 1035" 1057" 1079" 1101" 1123" 1145" 1167" 1189" 1211" 1233" 1255" 1277" 1299" 1321" 1343" 1365" 1387" 1409" 1431" CPU$U%liza%on$(%)$ Time$(s)$ MiM"1" MiM"2" MiM"3" Friday, 24 May 13
36. © 1999-2013 Erlang Solutions Ltd. 22 0.00# 2,000,000.00# 4,000,000.00# 6,000,000.00# 8,000,000.00# 10,000,000.00# 12,000,000.00# 14,000,000.00# 16,000,000.00# 1# 26# 51# 76# 101# 126# 151# 176# 201# 226# 251# 276# 301# 326# 351# 376# 401# 426# 451# 476# 501# 526# 551# 576# 601# 626# 651# 676# 701# 726# 751# 776# 801# 826# 851# 876# 901# 926# 951# 976# 1001# 1026# 1051# 1076# 1101# 1126# 1151# 1176# 1201# 1226# 1251# 1276# 1301# 1326# 1351# 1376# 1401# 1426# Free$memory$(KB)$ Time$(s)$ MiM#1# MiM#2# MiM#3# Friday, 24 May 13
37. © 1999-2013 Erlang Solutions Ltd. Load tests - Amazon EC2 23 MySQL m1.xlarge MongooseIM m1.xlarge MongooseIM m1.xlarge MongooseIM m1.xlarge Friday, 24 May 13
38. © 1999-2013 Erlang Solutions Ltd. Load tests - 3 MongooseIM + MySQL User arrival rate: 130/s User count: 330k Roster size: 100 Presence rate: 120 per second (12k msg/s) The aim was to check resource usage with all user data like passwords and rosters in MySQL 24 Friday, 24 May 13
39. © 1999-2013 Erlang Solutions Ltd. 25 %" 10%" 20%" 30%" 40%" 50%" 60%" 70%" 80%" 90%" 100%" 1" 62" 123" 184" 245" 306" 367" 428" 489" 550" 611" 672" 733" 794" 855" 916" 977" 1038" 1099" 1160" 1221" 1282" 1343" 1404" 1465" 1526" 1587" 1648" 1709" 1770" 1831" 1892" 1953" 2014" 2075" 2136" 2197" 2258" 2319" 2380" 2441" 2502" 2563" 2624" 2685" 2746" 2807" 2868" 2929" 2990" 3051" 3112" 3173" 3234" 3295" 3356" 3417" 3478" 3539" 3600" 3661" 3722" 3783" 3844" 3905" 3966" 4027" CPU$U%liza%on$(%)$ Time$(s)$ MIM"1" MiM"2" MiM"3" Friday, 24 May 13
40. © 1999-2013 Erlang Solutions Ltd. 26 0.00# 2,000,000.00# 4,000,000.00# 6,000,000.00# 8,000,000.00# 10,000,000.00# 12,000,000.00# 14,000,000.00# 16,000,000.00# 1# 65# 129# 193# 257# 321# 385# 449# 513# 577# 641# 705# 769# 833# 897# 961# 1025# 1089# 1153# 1217# 1281# 1345# 1409# 1473# 1537# 1601# 1665# 1729# 1793# 1857# 1921# 1985# 2049# 2113# 2177# 2241# 2305# 2369# 2433# 2497# 2561# 2625# 2689# 2753# 2817# 2881# 2945# 3009# 3073# 3137# 3201# 3265# 3329# 3393# 3457# 3521# 3585# 3649# 3713# 3777# 3841# 3905# 3969# 4033# Free$memory$(KB)$ Time$(s)$ MiM#1# MiM#2# MiM#3# Friday, 24 May 13
41. © 1999-2013 Erlang Solutions Ltd. Load tests - Amazon EC2 27 redis m1.xlarge MongooseIM m1.xlarge MongooseIM m1.xlarge MongooseIM m1.xlarge MySQL m1.xlarge Friday, 24 May 13
42. © 1999-2013 Erlang Solutions Ltd. Load tests - 3 MIM + redis + MySQL User arrival rate: 2000/s User count: 380k Message rate: 8k per second The aim was to check characteristics of a cluster of 3 MongooseIM nodes with session data in redis instead of mnesia. 28 Friday, 24 May 13
43. © 1999-2013 Erlang Solutions Ltd. 29 Friday, 24 May 13
44. © 1999-2013 Erlang Solutions Ltd. 30 Friday, 24 May 13
45. 31 ooVoo is the largest independent video communication service provider • 70mm users • Free, up to 12way high-quality service. Leader in group calling • Apps for PC. Mac. Web. iPhone. iPad. Android. Facebook • 10+ billion video minutes in 2012 • Platform for Shared Experiences courtesy of Alex Fok, System Architect at ooVoo Friday, 24 May 13
46. 32 Main Challenge - Scale • 2010 – 600K online users – 10M registered users – 500 messages/sec • Due to constant and rapid usage grow – infinite scale requirement courtesy of Alex Fok, System Architect at ooVoo Friday, 24 May 13
47. 33 Today (2 years after ) • 11 scale units in 2 geographical locations • x10 Usage growth – 2.2M online users (connected) – 70M registered users – 5,000+ messages per second • 99.98% uptime courtesy of Alex Fok, System Architect at ooVoo Friday, 24 May 13
48. 34 courtesy of Alex Fok, System Architect at ooVoo Friday, 24 May 13
49. 35 courtesy of Alex Fok, System Architect at ooVoo Friday, 24 May 13
50. © 1999-2013 Erlang Solutions Ltd. Benefits •Proven solution from an Erlang-focused company with expertise in instant messaging (over 300 clients since 1999) •Rapid deployment •Interoperability ensured by industry standards •Minimal capital expenditure due to efficient runtime platform •Pluggable authentication for ease of integration 36 Friday, 24 May 13
51. © 1999-2013 Erlang Solutions Ltd. New release 1.2.2 •BOSH support (XMPP over HTTP) •WSS (WebSocekt Secure) •35 metrics for Zabbix, Nagios, Cacti) - successful logins, auth fails, message sent, presence sent, etc. •alarm handler •changes to ease back porting of ejabberd Community Edition modules to MongooseIM 37 Friday, 24 May 13
52. © 1999-2013 Erlang Solutions Ltd. MongooseIM Download www.erlang-solutions.com/downloads/ Fork and contribute https://github.com/esl/ejabberd Contact us ejabberd@erlang-solutions.com 38 Friday, 24 May 13