Erlang Awesomeness

Introducing Erlang and some awesome success stories.

Transcript

1. ÓÄÈÂÈÒÅËÜÍÛÉ È ÍÅÂÅÐÎßÒÍÛÉ ERLANG À.Ä. ×è÷èãèí June 2011
2. Êðàòêàÿ èñòîðèÿ Erlang • Ïåðâàÿ âåðñèÿ  1986 ãîä, Äæî Àðìñòðîíã • Implemented in Prolog • JAM Virtual Machine  1988-1989 • Distribution  1990 • BEAM development started in 1992 • BEAM replaced JAM in 1997 • Open sourced in 1998 • Earlier in 1998  Ericsson AXD301 switch with nine 9 • HiPE integrated into Erlang/OTP  2001 • SMP support in 2006
3. Success Stories Any suciently complicated concurrent program in another language contains an ad hoc informally-specied bug-ridden slow implementation of half of Erlang.
4. RabbitMQ • POC - a few weeks of hacking in summer 2006 • rst public release, under MPL, in Feb 2007 • complete, conformant and interoperable implementation of the published AMQP spec • good throughput and latency • reliable and scalable • platform-neutral distribution, plus platform-specic packages and bundles • several extensions - HTTP, STOMP, XMPP, ...
5. RabbitMQ
6. RabbitMQ
7. RabbitMQ
8. AOL's Erlang Framework for Real-time Computational Advertising. Pero Subasic Äàííûå î ðåêëàìå ïðîõîäÿò ÷åðåç Flume, ðàñïðåäåëåííóþ ñèñòåìó ñáîðà è àãðåãàöèè äàííûõ. Ïîñëå àãðåãàöèè â Flume ýòè äàííûå ñêèäûâàþòñÿ äëÿ õðàíåíèÿ â Hadoop. Òàêèì îáðàçîì â Hadoop ñêèäûâàåòñÿ äî 16 ÒÁ äàííûõ â äåíü.
9. AOL's Erlang Framework for Real-time Computational Advertising. Pero Subasic Hadoop ïîëåçåí òîëüêî äëÿ õðàíåíèÿ ðåäêî èçìåíÿþùèõñÿ, ñåçîííûõ äàííûõ. Äëÿ åæåäíåâíî, à òî è åæåñåêóíäíî ìåíÿþùèõñÿ äàííûõ, Ïåðî Ñóáàñè÷ ïðèìåíÿåò Erlang. Ïî ñóòè Erlang ÿâëÿåòñÿ ýòàêîé ïðîñëîéêîé ìåæäó Flume è Hadoop, êîòîðàÿ çàõâàòûâàåò è ïðîñ÷èòûâàåò ïåðåñûëàåìûå äàííûå â ðåàëüíîì âðåìåíè.
10. AOL's Erlang Framework for Real-time Computational Advertising. Pero Subasic •  ñèñòåìå 10 óçëîâ, ïî 16 ÿäåð è 128 ÃÁ ïàìÿòè íà óçëå •  öåëîì, ñèñòåìà îáðàáàòûâàåò äî 5 ìèëëèàðäîâ ñîáûòèé â äåíü (íîâàÿ ðåêëàìà, èçìåíåíèÿ â ðåêëàìå, èçìåíåíèÿ â ïðåäïî÷òåíèÿõ ïîëüçîâàòåëåé è ò.ï.) • Ñèñòåìà, ïî ñóòè, ïðåäñòàâëÿåò ñîáîé ôðåéìâîðê, êîòîðûé ïðåäîñòàâëÿåò ñâîèì ïîëüçîâàòåëÿì íåñêîëüêî äåñÿòêîâ òûñÿ÷ ïàðàìåòðîâ • Ïîñëå çàõâàòà äàííûå ïðîõîäÿò ÷åðåç ìîäóëè-ñêàíåðû (80 ðàçíûõ ñêàíåðîâ), ïåðåêèäûâàþùèå îáðàáîòàííóþ èíôîðìàöèþ íà 120-150 àãðåãàòîðîâ, êîòîðûå â èòîãå ãåíåðèðóþò äî 2.5 ìèëëèîíîâ îáíîâëåíèé â ñåêóíäó • Ïðè îêíå äëÿ àïäåéòà â 90 ñåêóíä, ãåíåðàöèÿ íåîáõîäèìûõ äàííûõ çàíèìàåò âñåãî ëèøü 300 ìèëëèñåêóíä
11. AOL's Erlang Framework for Real-time Computational Advertising. Pero Subasic Ïÿòü ìåñÿöåâ, îäèí ÷åëîâåê  ÿ
12. Erlang in Klarna. David Craelius Klarna çàíèìàåòñÿ îáåñïå÷åíèåì è èíòåãðàöèåé îíëàéí-îïëàò è íà äàííûé ìîìåíò ÿâëÿåòñÿ îäíèì èç êðóïíåéøèõ ñêîïëåíèé ïðîãðàììèñòîâ íà Erlang'å  â Klarna ðàáîòàþò 55 ïðîãðàììèñòîâ, ïèøóùèõ íà Erlang.
13. Erlang in Klarna. David Craelius Âñÿ âíóòðåííÿÿ êóõíÿ êîìïàíèè áûëà ïåðåïèñàíà íà Erlang, ÷òî ïîëîæèòåëüíî ñêàçàëîñü íà êîìïàíèè â öåëîì: ñ 2007-ãî êîìïàíèÿ åæåãîäíî óäâàèâàåò ñâîþ ïðèáûëü, ðàñòåò è ðàñïðîñòðàíÿåòñÿ â äðóãèå ñòðàíû, è óæå îáàíêðîòèëà äâóõ ñâîèõ ãëàâíûõ êîíêóðåíòîâ íà øâåäñêîì ðûíêå.
14. Erlang in Klarna. David Craelius Äî Erlang'à ñèñòåìà äîëæíà áûëà ñïðàâëÿòüñÿ ñ 20 000 ñîáûòèé, ðàñïðåäåëåííûìè ìåæäó 50 000 êëèåíòîâ. Java íå ñïðàâëÿëàñü. Îäíà èç ãëàâíûõ ïðè÷èí  Java íå îñâîáîæäàåò ïàìÿòü òàê áûñòðî, êàê õîòåëîñü áû/íàäî. Òàê, ôèíàíñîâûå äàííûå íå ðàïñðåäåëåíû ïî âðåìåíè, à ïðèõîäÿò âîëíàìè. Java íå óñïåâàëà ðåàãèðîâàòü íà ïðîèñõîäÿùåå è âûñâîáîæäàòü ïàìÿòü äëÿ ñîáûòèé.
15. Erlang in Klarna. David Craelius Íà çàìåíó ñóùåñòâóþùåé ñèñòåìû ñ Java íà Erlang ïîíàäîáèëîñü 6 ìåñÿöåâ.  òî âðåìÿ, êàê â Java ïðèõîäèòñÿ áîðîòüñÿ ñ èíôðàñòðóêòóðîé, êîìïîíåíòàìè, jar'àìè è ò.ï., â Erlang'å ýòîãî íåò. Òåõíîëîãèÿ îòíîñèòåëüíî äåøåâàÿ è òðåáóåòñÿ ìåíüøåå êîëè÷åñòâî ëþäåé äëÿ äîñòèæåíèÿ òåõ æå ðåçóëüòàòîâ.
16. Erlang in Klarna. David Craelius Áëàãîäàðÿ Erlang'ó îòñóòñòâóþò ïðîáëåìû ñ óïðàâëåíèåì ðåëèçàìè. Ïðîâîäèòñÿ ãîðÿ÷àÿ çàìåíà êîäà ïðÿìî íà production'å, ñèñòåìà âñåãäà æèâàÿ. Ïîýòîìó îáíîâëåíèÿ ïðîâîäÿòñÿ ìèíèìóì ðàç â äåíü, æåëàòåëüíî ÷àùå.
17. Erlang in Klarna. David Craelius 20 Java-ìàøèí áûëè çàìåíåíû 2-ìÿ ñ Erlang'îì. Çà ïîñëåäíèå òðè ãîäà áûëî 48 ñåêóíä ïðîñòîÿ. Ñèñòåìà íåóáèâàåìà.
18. Large Scale Messaging at IMVU in social entertainment and gaming. Jon Watte IMVU  ýòî ìíîãîïîëüçîâàòåëüñêèé îíëàéí-ìèð/èãðà, â êîòîðîé åæåäíåâíî ïðèñóòñâóåò äî 150 000 ïîñåòèòåëåé. Êàæäûé èç èãðîêîâ ãåíåðèðóåò îãðîìíîå êîëè÷åñòâî ñîáûòèé: ñîîáùåíèÿ ìåæäó èãðîêàìè, èçìåíåíèÿ ñòàòóñîâ èãðîêîâ (çàøåë/âûøåë/idle è ò.ï.), èçìåíåíèÿ â ïðîôèëå èãðîêîâ (íàãðàäû è achievements).
19. Large Scale Messaging at IMVU in social entertainment and gaming. Jon Watte Âñå ýòè èçìåíåíèÿ â ðåàëüíîì âðåìåíè äîíîñÿòñÿ äî âñåõ èãðîêîâ, ñ êîòîðûìè èçìåíåíèÿ òàê èëè èíà÷å ñâÿçàíû. Ñèñòåìà, îáðàáàòûâàþùàÿ ýòè ñîáûòèÿ, íàïèñàíà íà Erlang'å è ðàáîòàåò íà íåñêîëüêèõ äåñÿòêàõ ñåðâåðîâ.
20. Large Scale Messaging at IMVU in social entertainment and gaming. Jon Watte •  äåíü â ÷àòàõ íà ñàéòå ñîçäà¼òñÿ ∼500 000 êîìíàò è ãåíåðèðóåòñÿ ∼20 000 000 ñîáûòèé ïîëüçîâàòåëü çàøåë/ïîëüçîâàòåëü âûøåë • Ñîñòîÿíèå õðàíèòñÿ â ëåãêîâåñíûõ ñòðóêòóðàõ â ïàìÿòè • 1 000 000 ïîëüçîâàòåëåé = 40GB RAM = 3-4 ñåêóíäû íà îáðàáîòêó ñîîáùåíèÿ îò ïîëüçîâàòåëÿ • Áîëüøàÿ ÷àñòü âðåìåíè CPU òðàòèòñÿ íà ìàðøàëèíã äàííûõ (èñïîëüçóåòñÿ Protocol Buers), òàê êàê ñèñòåìà ãåòåðîãåííà è åñòü êîä, íàïèñàííûé íà äðóãèõ ÿçûêàõ ïðîãðàììèðîâàíèÿ.
21. Large Scale Messaging at IMVU in social entertainment and gaming. Jon Watte • â äåíü íà production çàëèâàåòñÿ äî 50 îáíîâëåíèé • âåñü êîä ïðîõîäèò ÷åðåç àâòîìàòè÷åñêîå òåñòèðâîàíèå, êîòîðîå äëèòñÿ 9-12 ìèíóò.  ñëó÷àå îòñóòñòâèÿ òåñòîâ èëè åñëè òåñòû íå ïðîõîäÿòñÿ, êîä íà production íå èä¼ò • âñåãî îêîëî 30 òûñÿ÷ òåñòîâ, ñîäåðæàùèõ áîëåå 2 ìëí. asserts • êîä çàëèâàåòñÿ íà 700 production-ñåðâåðîâ. Ïîñëå îáíîâëåíèÿ ñèñòåìà òåñòèðóåòñÿ íà production'å â òå÷åíèå 2 ìèíóò (òàê êàê ïîëüçîâàòåëåé î÷åíü ìíîãî, òî â ñëó÷àå ïðîáëåì îíè âûëåçóò ñðàçó) • åäèíñòâåííîå, ÷òî îáíîâëÿåòñÿ ïðàêòè÷åñêè âðó÷íóþ  ýòî áàçû äàííûõ. Äëÿ íèõ ïèøóòñÿ update-ñêðèïòû, êîòîðûå ïðîâåðÿþòñÿ ìèíèìóì äâóìÿ ëþäüìè ïåðåä òåì, êàê ïîïàñòü íà production
22. Êðîìå òîãî • Echo  http://www.aboutecho.com/streamserver • ErlyVideo  https://github.com/erlyvideo • Ejabberd  http://www.ejabberd.im/
23. Êàê îíè ýòî äåëàþò?!
24. Ñïàñèáî çà âíèìàíèå! Âîïðîñû?