MongooseIM - Messaging that Scales

MongooseIM - Messaging that Scales

MongooseIM provides a unique instant messaging solution when massive volume is required. The growth of social media communication has resulted in the fact that the ceiling is quickly reached when successful projects take off. MongooseIM secures performance and stability.

In this talk I present lessons learnt when scaling messaging servers using common PC, cloud environments or a dedicated data centre. I demonstrate what bottlenecks we encountered and how to work around them in order to scale to millions of users. I also explain why MongooseIM and Erlang are a good baseline for building a custom messaging solution.

Talk objectives: to educate about MongooseIM, which is built around proven technology Jabber/XMPP, as well as discuss how to build a messaging solution, which will scale to big numbers.

Target audience: People who know Jabber/XMPP or are interested in messaging servers.

69639a461e92e590acdc1b554934bd8d?s=128

Michał Ślaski

May 23, 2013
Tweet

Transcript

  1. © 1999-2013 Erlang Solutions Ltd. MongooseIM - Messaging that Scales

    Michał Ślaski Friday, 24 May 13
  2. © 1999-2013 Erlang Solutions Ltd. What is MongooseIM? 2 Friday,

    24 May 13
  3. © 1999-2013 Erlang Solutions Ltd. What is MongooseIM? •Instant messaging

    for Social Media, Gaming and Telecommunications 2 Friday, 24 May 13
  4. © 1999-2013 Erlang Solutions Ltd. What is MongooseIM? •Instant messaging

    for Social Media, Gaming and Telecommunications •Solution designed for high volume 2 Friday, 24 May 13
  5. © 1999-2013 Erlang Solutions Ltd. What is MongooseIM? •Instant messaging

    for Social Media, Gaming and Telecommunications •Solution designed for high volume •Easily scalable distributed system 2 Friday, 24 May 13
  6. © 1999-2013 Erlang Solutions Ltd. What is MongooseIM? •Instant messaging

    for Social Media, Gaming and Telecommunications •Solution designed for high volume •Easily scalable distributed system •Highly customisable platform thanks to industry standards and Open Source technology 2 Friday, 24 May 13
  7. © 1999-2013 Erlang Solutions Ltd. Target domains 3 Friday, 24

    May 13
  8. © 1999-2013 Erlang Solutions Ltd. Target domains 3 Telecoms (chat

    features are a part of standard commmunication software) Friday, 24 May 13
  9. © 1999-2013 Erlang Solutions Ltd. Target domains 3 Telecoms (chat

    features are a part of standard commmunication software) Social Media (messaging is the key part of social web sites) Friday, 24 May 13
  10. © 1999-2013 Erlang Solutions Ltd. Target domains 3 Telecoms (chat

    features are a part of standard commmunication software) Social Media (messaging is the key part of social web sites) Gaming (multi-user chats are an integral part of the gameplay and enhance the user experience) Friday, 24 May 13
  11. © 1999-2013 Erlang Solutions Ltd. Easy to work with •Support

    of XMPP - a protocol used for Facebook chat and Google Talk - seamless integration with many libraries and client applications 4 Friday, 24 May 13
  12. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure

    Flexible Decentralised Efficient Proven 5 Friday, 24 May 13
  13. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure

    Flexible Decentralised Efficient Proven 5 Standard formalised by the IETF through RFCs and extensions are published by the XSF Friday, 24 May 13
  14. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure

    Flexible Decentralised Efficient Proven 5 Channel encryption, strong authentication, may be isolated from the public network Friday, 24 May 13
  15. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure

    Flexible Decentralised Efficient Proven 5 Custom functionality can be built on top of XMPP Friday, 24 May 13
  16. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure

    Flexible Decentralised Efficient Proven 5 Anyone can run XMPP server and there is no central master server Friday, 24 May 13
  17. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure

    Flexible Decentralised Efficient Proven 5 Solves issues of polling approaches Friday, 24 May 13
  18. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure

    Flexible Decentralised Efficient Proven 5 In use since 1998, many implementations and deployments, millions of end users Friday, 24 May 13
  19. © 1999-2013 Erlang Solutions Ltd. XMPP Properties Open standard Secure

    Flexible Decentralised Efficient Proven 5 Friday, 24 May 13
  20. © 1999-2013 Erlang Solutions Ltd. Features •Support for WebSockets -

    ready for modern chat- enabled web applications •Multi-User Chat (XEP-0045) •Service Discovery (XEP-0030) •vCard (XEP-0054) •Privacy Lists (XEP-0016) •Private XML Storage (XEP-0049) 6 Friday, 24 May 13
  21. © 1999-2013 Erlang Solutions Ltd. Extensible •Customisable - provides a

    base for bespoke solution to one’s specific needs - push notifications to mobile devices - authentication plugins - reliable message delivery in mobile networks •Has open sourced regression tests - github.com/esl/ejabberd_tests 7 Friday, 24 May 13
  22. © 1999-2013 Erlang Solutions Ltd. Scalable •Configurable database backends -

    Mnesia for simple deployments - MySQL for persistent data - Mnesia or Redis for transient data 8 Friday, 24 May 13
  23. © 1999-2013 Erlang Solutions Ltd. Scalable 9 Friday, 24 May

    13
  24. © 1999-2013 Erlang Solutions Ltd. Load tests - environment Test

    environment Erlang/OTP R15B02 MongooseIM 1.1 MySQL 5.5.24 Ubuntu 12.04 LTS 3.2.0-23-generic Tsung 1.5.0a redis 2.6.10 ejabberd 2.1.11 10 Friday, 24 May 13
  25. © 1999-2013 Erlang Solutions Ltd. Load tests - dedicated box

    11 MongooseIM AMD 8x3.3GHz, 32GB RAM MySQL Friday, 24 May 13
  26. © 1999-2013 Erlang Solutions Ltd. Load tests - “max users”

    User arrival rate: 150/s User count: 400k Roster size: 100 No message exchange The aim was to check resource usage with maximum possible count of online users connected to server. 12 Friday, 24 May 13
  27. © 1999-2013 Erlang Solutions Ltd. Load tests - “max users”

    13 Friday, 24 May 13
  28. © 1999-2013 Erlang Solutions Ltd. Load tests - “max users”

    14 Friday, 24 May 13
  29. © 1999-2013 Erlang Solutions Ltd. Load tests - “max message

    rate” User arrival rate: 100/s User count: 75k Roster size: 100 Message rate: ~21k per second The aim was to check resource usage when maximum message traffic is generated. 15 Friday, 24 May 13
  30. © 1999-2013 Erlang Solutions Ltd. Load tests - “max message

    rate” 16 Friday, 24 May 13
  31. © 1999-2013 Erlang Solutions Ltd. Load tests - “max message

    rate” 17 Friday, 24 May 13
  32. © 1999-2013 Erlang Solutions Ltd. Load tests - Amazon EC2

    •M1 Extra Large Instance - 15 GiB memory - 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each) - I/O Performance: High 18 Friday, 24 May 13
  33. © 1999-2013 Erlang Solutions Ltd. Load tests - Amazon EC2

    19 MongooseIM m1.xlarge MongooseIM m1.xlarge MongooseIM m1.xlarge Friday, 24 May 13
  34. © 1999-2013 Erlang Solutions Ltd. Load tests - 3 MongooseIM

    + Mnesia User arrival rate: 130/s User count: 50k Roster size: 100 Presence rate: 120 per second (12k msg/s) The aim was to check resource usage with all user data like passwords and rosters in Mnesia. 20 Friday, 24 May 13
  35. © 1999-2013 Erlang Solutions Ltd. 21 %" 10%" 20%" 30%"

    40%" 50%" 60%" 70%" 80%" 90%" 100%" 1" 23" 45" 67" 89" 111" 133" 155" 177" 199" 221" 243" 265" 287" 309" 331" 353" 375" 397" 419" 441" 463" 485" 507" 529" 551" 573" 595" 617" 639" 661" 683" 705" 727" 749" 771" 793" 815" 837" 859" 881" 903" 925" 947" 969" 991" 1013" 1035" 1057" 1079" 1101" 1123" 1145" 1167" 1189" 1211" 1233" 1255" 1277" 1299" 1321" 1343" 1365" 1387" 1409" 1431" CPU$U%liza%on$(%)$ Time$(s)$ MiM"1" MiM"2" MiM"3" Friday, 24 May 13
  36. © 1999-2013 Erlang Solutions Ltd. 22 0.00# 2,000,000.00# 4,000,000.00# 6,000,000.00#

    8,000,000.00# 10,000,000.00# 12,000,000.00# 14,000,000.00# 16,000,000.00# 1# 26# 51# 76# 101# 126# 151# 176# 201# 226# 251# 276# 301# 326# 351# 376# 401# 426# 451# 476# 501# 526# 551# 576# 601# 626# 651# 676# 701# 726# 751# 776# 801# 826# 851# 876# 901# 926# 951# 976# 1001# 1026# 1051# 1076# 1101# 1126# 1151# 1176# 1201# 1226# 1251# 1276# 1301# 1326# 1351# 1376# 1401# 1426# Free$memory$(KB)$ Time$(s)$ MiM#1# MiM#2# MiM#3# Friday, 24 May 13
  37. © 1999-2013 Erlang Solutions Ltd. Load tests - Amazon EC2

    23 MySQL m1.xlarge MongooseIM m1.xlarge MongooseIM m1.xlarge MongooseIM m1.xlarge Friday, 24 May 13
  38. © 1999-2013 Erlang Solutions Ltd. Load tests - 3 MongooseIM

    + MySQL User arrival rate: 130/s User count: 330k Roster size: 100 Presence rate: 120 per second (12k msg/s) The aim was to check resource usage with all user data like passwords and rosters in MySQL 24 Friday, 24 May 13
  39. © 1999-2013 Erlang Solutions Ltd. 25 %" 10%" 20%" 30%"

    40%" 50%" 60%" 70%" 80%" 90%" 100%" 1" 62" 123" 184" 245" 306" 367" 428" 489" 550" 611" 672" 733" 794" 855" 916" 977" 1038" 1099" 1160" 1221" 1282" 1343" 1404" 1465" 1526" 1587" 1648" 1709" 1770" 1831" 1892" 1953" 2014" 2075" 2136" 2197" 2258" 2319" 2380" 2441" 2502" 2563" 2624" 2685" 2746" 2807" 2868" 2929" 2990" 3051" 3112" 3173" 3234" 3295" 3356" 3417" 3478" 3539" 3600" 3661" 3722" 3783" 3844" 3905" 3966" 4027" CPU$U%liza%on$(%)$ Time$(s)$ MIM"1" MiM"2" MiM"3" Friday, 24 May 13
  40. © 1999-2013 Erlang Solutions Ltd. 26 0.00# 2,000,000.00# 4,000,000.00# 6,000,000.00#

    8,000,000.00# 10,000,000.00# 12,000,000.00# 14,000,000.00# 16,000,000.00# 1# 65# 129# 193# 257# 321# 385# 449# 513# 577# 641# 705# 769# 833# 897# 961# 1025# 1089# 1153# 1217# 1281# 1345# 1409# 1473# 1537# 1601# 1665# 1729# 1793# 1857# 1921# 1985# 2049# 2113# 2177# 2241# 2305# 2369# 2433# 2497# 2561# 2625# 2689# 2753# 2817# 2881# 2945# 3009# 3073# 3137# 3201# 3265# 3329# 3393# 3457# 3521# 3585# 3649# 3713# 3777# 3841# 3905# 3969# 4033# Free$memory$(KB)$ Time$(s)$ MiM#1# MiM#2# MiM#3# Friday, 24 May 13
  41. © 1999-2013 Erlang Solutions Ltd. Load tests - Amazon EC2

    27 redis m1.xlarge MongooseIM m1.xlarge MongooseIM m1.xlarge MongooseIM m1.xlarge MySQL m1.xlarge Friday, 24 May 13
  42. © 1999-2013 Erlang Solutions Ltd. Load tests - 3 MIM

    + redis + MySQL User arrival rate: 2000/s User count: 380k Message rate: 8k per second The aim was to check characteristics of a cluster of 3 MongooseIM nodes with session data in redis instead of mnesia. 28 Friday, 24 May 13
  43. © 1999-2013 Erlang Solutions Ltd. 29 Friday, 24 May 13

  44. © 1999-2013 Erlang Solutions Ltd. 30 Friday, 24 May 13

  45. 31 ooVoo is the largest independent video communication service provider

    • 70mm users • Free, up to 12way high-quality service. Leader in group calling • Apps for PC. Mac. Web. iPhone. iPad. Android. Facebook • 10+ billion video minutes in 2012 • Platform for Shared Experiences courtesy of Alex Fok, System Architect at ooVoo Friday, 24 May 13
  46. 32 Main Challenge - Scale • 2010 – 600K online

    users – 10M registered users – 500 messages/sec • Due to constant and rapid usage grow – infinite scale requirement courtesy of Alex Fok, System Architect at ooVoo Friday, 24 May 13
  47. 33 Today (2 years after ) • 11 scale units

    in 2 geographical locations • x10 Usage growth – 2.2M online users (connected) – 70M registered users – 5,000+ messages per second • 99.98% uptime courtesy of Alex Fok, System Architect at ooVoo Friday, 24 May 13
  48. 34 courtesy of Alex Fok, System Architect at ooVoo Friday,

    24 May 13
  49. 35 courtesy of Alex Fok, System Architect at ooVoo Friday,

    24 May 13
  50. © 1999-2013 Erlang Solutions Ltd. Benefits •Proven solution from an

    Erlang-focused company with expertise in instant messaging (over 300 clients since 1999) •Rapid deployment •Interoperability ensured by industry standards •Minimal capital expenditure due to efficient runtime platform •Pluggable authentication for ease of integration 36 Friday, 24 May 13
  51. © 1999-2013 Erlang Solutions Ltd. New release 1.2.2 •BOSH support

    (XMPP over HTTP) •WSS (WebSocekt Secure) •35 metrics for Zabbix, Nagios, Cacti) - successful logins, auth fails, message sent, presence sent, etc. •alarm handler •changes to ease back porting of ejabberd Community Edition modules to MongooseIM 37 Friday, 24 May 13
  52. © 1999-2013 Erlang Solutions Ltd. MongooseIM Download www.erlang-solutions.com/downloads/ Fork and

    contribute https://github.com/esl/ejabberd Contact us ejabberd@erlang-solutions.com 38 Friday, 24 May 13