Erlang/OTP Awesomeness

Success stories of several Erlang applications and overview of OTP principles and behaviours. Mostly in Russian.

Transcript

1. ÓÄÈÂÈÒÅËÜÍÛÉ È ÍÅÂÅÐÎßÒÍÛÉ ERLANG À.Ä. ×è÷èãèí June 2011
2. Êðàòêàÿ èñòîðèÿ Erlang Ïåðâàÿ âåðñèÿ  1986 ãîä, Äæî Àðìñòðîíã Implemented in Prolog JAM Virtual Machine  1988-1989 Distribution  1990 BEAM development started in 1992 BEAM replaced JAM in 1997 Open sourced in 1998 Earlier in 1998  Ericsson AXD301 switch with nine 9 HiPE integrated into Erlang/OTP  2001 SMP support in 2006
3. Success Stories Any suciently complicated concurrent program in another language contains an ad hoc informally-specied bug-ridden slow implementation of half of Erlang.
4. RabbitMQ POC - a few weeks of hacking in summer 2006 rst public release, under MPL, in Feb 2007 complete, conformant and interoperable implementation of the published AMQP spec good throughput and latency reliable and scalable platform-neutral distribution, plus platform-specic packages and bundles several extensions - HTTP, STOMP, XMPP, ...
5. RabbitMQ
6. RabbitMQ
7. RabbitMQ
8. AOL's Erlang Framework for Real-time Computational Advertising. Pero Subasic Äàííûå î ðåêëàìå ïðîõîäÿò ÷åðåç Flume, ðàñïðåäåëåííóþ ñèñòåìó ñáîðà è àãðåãàöèè äàííûõ. Ïîñëå àãðåãàöèè â Flume ýòè äàííûå ñêèäûâàþòñÿ äëÿ õðàíåíèÿ â Hadoop. Òàêèì îáðàçîì â Hadoop ñêèäûâàåòñÿ äî 16 ÒÁ äàííûõ â äåíü.
9. AOL's Erlang Framework for Real-time Computational Advertising. Pero Subasic Hadoop ïîëåçåí òîëüêî äëÿ õðàíåíèÿ ðåäêî èçìåíÿþùèõñÿ, ñåçîííûõ äàííûõ. Äëÿ åæåäíåâíî, à òî è åæåñåêóíäíî ìåíÿþùèõñÿ äàííûõ, Ïåðî Ñóáàñè÷ ïðèìåíÿåò Erlang. Ïî ñóòè Erlang ÿâëÿåòñÿ ýòàêîé ïðîñëîéêîé ìåæäó Flume è Hadoop, êîòîðàÿ çàõâàòûâàåò è ïðîñ÷èòûâàåò ïåðåñûëàåìûå äàííûå â ðåàëüíîì âðåìåíè.
10. AOL's Erlang Framework for Real-time Computational Advertising. Pero Subasic  ñèñòåìå 10 óçëîâ, ïî 16 ÿäåð è 128 ÃÁ ïàìÿòè íà óçëå  öåëîì, ñèñòåìà îáðàáàòûâàåò äî 5 ìèëëèàðäîâ ñîáûòèé â äåíü (íîâàÿ ðåêëàìà, èçìåíåíèÿ â ðåêëàìå, èçìåíåíèÿ â ïðåäïî÷òåíèÿõ ïîëüçîâàòåëåé è ò.ï.) Ñèñòåìà, ïî ñóòè, ïðåäñòàâëÿåò ñîáîé ôðåéìâîðê, êîòîðûé ïðåäîñòàâëÿåò ñâîèì ïîëüçîâàòåëÿì íåñêîëüêî äåñÿòêîâ òûñÿ÷ ïàðàìåòðîâ Ïîñëå çàõâàòà äàííûå ïðîõîäÿò ÷åðåç ìîäóëè-ñêàíåðû (80 ðàçíûõ ñêàíåðîâ), ïåðåêèäûâàþùèå îáðàáîòàííóþ èíôîðìàöèþ íà 120-150 àãðåãàòîðîâ, êîòîðûå â èòîãå ãåíåðèðóþò äî 2.5 ìèëëèîíîâ îáíîâëåíèé â ñåêóíäó Ïðè îêíå äëÿ àïäåéòà â 90 ñåêóíä, ãåíåðàöèÿ íåîáõîäèìûõ äàííûõ çàíèìàåò âñåãî ëèøü 300 ìèëëèñåêóíä
11. AOL's Erlang Framework for Real-time Computational Advertising. Pero Subasic Ïÿòü ìåñÿöåâ, îäèí ÷åëîâåê  ÿ
12. Erlang in Klarna. David Craelius Klarna çàíèìàåòñÿ îáåñïå÷åíèåì è èíòåãðàöèåé îíëàéí-îïëàò è íà äàííûé ìîìåíò ÿâëÿåòñÿ îäíèì èç êðóïíåéøèõ ñêîïëåíèé ïðîãðàììèñòîâ íà Erlang'å  â Klarna ðàáîòàþò 55 ïðîãðàììèñòîâ, ïèøóùèõ íà Erlang.
13. Erlang in Klarna. David Craelius Âñÿ âíóòðåííÿÿ êóõíÿ êîìïàíèè áûëà ïåðåïèñàíà íà Erlang, ÷òî ïîëîæèòåëüíî ñêàçàëîñü íà êîìïàíèè â öåëîì: ñ 2007-ãî êîìïàíèÿ åæåãîäíî óäâàèâàåò ñâîþ ïðèáûëü, ðàñòåò è ðàñïðîñòðàíÿåòñÿ â äðóãèå ñòðàíû, è óæå îáàíêðîòèëà äâóõ ñâîèõ ãëàâíûõ êîíêóðåíòîâ íà øâåäñêîì ðûíêå.
14. Erlang in Klarna. David Craelius Äî Erlang'à ñèñòåìà äîëæíà áûëà ñïðàâëÿòüñÿ ñ 20 000 ñîáûòèé, ðàñïðåäåëåííûìè ìåæäó 50 000 êëèåíòîâ. Java íå ñïðàâëÿëàñü. Îäíà èç ãëàâíûõ ïðè÷èí  Java íå îñâîáîæäàåò ïàìÿòü òàê áûñòðî, êàê õîòåëîñü áû/íàäî. Òàê, ôèíàíñîâûå äàííûå íå ðàïñðåäåëåíû ïî âðåìåíè, à ïðèõîäÿò âîëíàìè. Java íå óñïåâàëà ðåàãèðîâàòü íà ïðîèñõîäÿùåå è âûñâîáîæäàòü ïàìÿòü äëÿ ñîáûòèé.
15. Erlang in Klarna. David Craelius Íà çàìåíó ñóùåñòâóþùåé ñèñòåìû ñ Java íà Erlang ïîíàäîáèëîñü 6 ìåñÿöåâ.  òî âðåìÿ, êàê â Java ïðèõîäèòñÿ áîðîòüñÿ ñ èíôðàñòðóêòóðîé, êîìïîíåíòàìè, jar'àìè è ò.ï., â Erlang'å ýòîãî íåò. Òåõíîëîãèÿ îòíîñèòåëüíî äåøåâàÿ è òðåáóåòñÿ ìåíüøåå êîëè÷åñòâî ëþäåé äëÿ äîñòèæåíèÿ òåõ æå ðåçóëüòàòîâ.
16. Erlang in Klarna. David Craelius Áëàãîäàðÿ Erlang'ó îòñóòñòâóþò ïðîáëåìû ñ óïðàâëåíèåì ðåëèçàìè. Ïðîâîäèòñÿ ãîðÿ÷àÿ çàìåíà êîäà ïðÿìî íà production'å, ñèñòåìà âñåãäà æèâàÿ. Ïîýòîìó îáíîâëåíèÿ ïðîâîäÿòñÿ ìèíèìóì ðàç â äåíü, æåëàòåëüíî ÷àùå.
17. Erlang in Klarna. David Craelius 20 Java-ìàøèí áûëè çàìåíåíû 2-ìÿ ñ Erlang'îì. Çà ïîñëåäíèå òðè ãîäà áûëî 48 ñåêóíä ïðîñòîÿ. Ñèñòåìà íåóáèâàåìà.
18. Large Scale Messaging at IMVU in social entertainment and gaming. Jon Watte IMVU  ýòî ìíîãîïîëüçîâàòåëüñêèé îíëàéí-ìèð/èãðà, â êîòîðîé åæåäíåâíî ïðèñóòñâóåò äî 150 000 ïîñåòèòåëåé. Êàæäûé èç èãðîêîâ ãåíåðèðóåò îãðîìíîå êîëè÷åñòâî ñîáûòèé: ñîîáùåíèÿ ìåæäó èãðîêàìè, èçìåíåíèÿ ñòàòóñîâ èãðîêîâ (çàøåë/âûøåë/idle è ò.ï.), èçìåíåíèÿ â ïðîôèëå èãðîêîâ (íàãðàäû è achievements).
19. Large Scale Messaging at IMVU in social entertainment and gaming. Jon Watte Âñå ýòè èçìåíåíèÿ â ðåàëüíîì âðåìåíè äîíîñÿòñÿ äî âñåõ èãðîêîâ, ñ êîòîðûìè èçìåíåíèÿ òàê èëè èíà÷å ñâÿçàíû. Ñèñòåìà, îáðàáàòûâàþùàÿ ýòè ñîáûòèÿ, íàïèñàíà íà Erlang'å è ðàáîòàåò íà íåñêîëüêèõ äåñÿòêàõ ñåðâåðîâ.
20. Large Scale Messaging at IMVU in social entertainment and gaming. Jon Watte  äåíü â ÷àòàõ íà ñàéòå ñîçäà¼òñÿ ∼500 000 êîìíàò è ãåíåðèðóåòñÿ ∼20 000 000 ñîáûòèé ïîëüçîâàòåëü çàøåë/ïîëüçîâàòåëü âûøåë Ñîñòîÿíèå õðàíèòñÿ â ëåãêîâåñíûõ ñòðóêòóðàõ â ïàìÿòè 1 000 000 ïîëüçîâàòåëåé = 40GB RAM = 3-4 ñåêóíäû íà îáðàáîòêó ñîîáùåíèÿ îò ïîëüçîâàòåëÿ Áîëüøàÿ ÷àñòü âðåìåíè CPU òðàòèòñÿ íà ìàðøàëèíã äàííûõ (èñïîëüçóåòñÿ Protocol Buers), òàê êàê ñèñòåìà ãåòåðîãåííà è åñòü êîä, íàïèñàííûé íà äðóãèõ ÿçûêàõ ïðîãðàììèðîâàíèÿ.
21. Large Scale Messaging at IMVU in social entertainment and gaming. Jon Watte â äåíü íà production çàëèâàåòñÿ äî 50 îáíîâëåíèé âåñü êîä ïðîõîäèò ÷åðåç àâòîìàòè÷åñêîå òåñòèðâîàíèå, êîòîðîå äëèòñÿ 9-12 ìèíóò.  ñëó÷àå îòñóòñòâèÿ òåñòîâ èëè åñëè òåñòû íå ïðîõîäÿòñÿ, êîä íà production íå èä¼ò âñåãî îêîëî 30 òûñÿ÷ òåñòîâ, ñîäåðæàùèõ áîëåå 2 ìëí. asserts êîä çàëèâàåòñÿ íà 700 production-ñåðâåðîâ. Ïîñëå îáíîâëåíèÿ ñèñòåìà òåñòèðóåòñÿ íà production'å â òå÷åíèå 2 ìèíóò (òàê êàê ïîëüçîâàòåëåé î÷åíü ìíîãî, òî â ñëó÷àå ïðîáëåì îíè âûëåçóò ñðàçó) åäèíñòâåííîå, ÷òî îáíîâëÿåòñÿ ïðàêòè÷åñêè âðó÷íóþ  ýòî áàçû äàííûõ. Äëÿ íèõ ïèøóòñÿ update-ñêðèïòû, êîòîðûå ïðîâåðÿþòñÿ ìèíèìóì äâóìÿ ëþäüìè ïåðåä òåì, êàê ïîïàñòü íà production
22. Êðîìå òîãî Echo  http://www.aboutecho.com/streamserver ErlyVideo  https://github.com/erlyvideo Ejabberd  http://www.ejabberd.im/
23. Êàê îíè ýòî äåëàþò?!
24. OTP for the resque! Open Telecom Platform Ôèëîñîôèÿ Ïîâåäåíèÿ Ìåòîäîëîãèÿ
25. OTP - Ôèëîñîôèÿ The world is concurrent Things in the world don't share data Things communicate with messages Things fail - Joe Armstrong
26. OTP - Ôèëîñîôèÿ Fail fast and Let it crash Supervision trees
27. OTP - ïîâåäåíèÿ gen_server gen_event gen_fsm supervisor
28. OTP - ïîâåäåíèÿ application release
29. gen_server example -module(key_value). -behaviour(gen_server). % Simple gen_server % (c) Alexandr Alexeev 2012 | http://eax.me/ -export([start/0, set/2, get/1]). -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]). % vvvv API vvvv start() -> gen_server:start_link({global, ?MODULE}, ?MODULE, [], []). set(Key, Value) -> gen_server:call({global, ?MODULE}, { set, Key, Value }). get(Key) -> gen_server:call({global, ?MODULE}, { get, Key }).
30. gen_server example % vvvv Gen Server Implementation vvvv init([]) -> State = dict:new(), {ok, State}. handle_call({ set, Key, Value }, _From, State) -> NewState = dict:store(Key, Value, State), { reply, ok, NewState }; handle_call({ get, Key }, _From, State) -> Resp = dict:find(Key, State), { reply, Resp, State }; handle_call(_Message, _From, State) -> { reply, invalid_command, State }. handle_cast(_Message, State) -> { noreply, State }. handle_info(_Message, State) -> { noreply, State }. terminate(_Reason, _State) -> ok. code_change(_OldVersion, State, _Extra) -> { ok, State }.
31. OTP - ìåòîäîëîãèÿ Huyak-huyak in production driven development
32. Other issues Robustness Soft real-time Introspection Remote debug Not a ghetto
33. Interoperability c-nodes ports linked-in drivers NIFs
34. Interoperability Sockets HTTP RESTful services Protocol Buers etc.
35. Ñïàñèáî çà âíèìàíèå! Âîïðîñû?