Gearman Deep Dive - PHPTek 2012

With every passing year the data that companies and application store grows astronomically, both through more data logging for business analysis and the consumption of 3rd party data (mashups), but how can those companies make sense of all that data and turn it into something useful that is constantly updating and evolving? Gearman is a very good and simple solution to this data problem. Gearman allows you to farm out work to other machines better suited to do the job at hand in a language and platform agnostic manner. During the talk I will tell stories of 2 different products where I faced a huge data processing challenges and how I solved them with Gearman. I will do demos and live coding of Gearman workers, a deep dive into the system!

Welcome to the Age of Data, we have cookies.

Transcript

1. Deep Dive into Gearman Helgi Þormar Þorbjörnsson PHPTek, Chicago, 24th May, 2012
2. Helgi
3. Co-founded Orchestra.io Helgi
4. Co-founded Orchestra.io Work at EngineYard Helgi
5. Co-founded Orchestra.io Work at EngineYard PEAR Developer Helgi
6. Co-founded Orchestra.io Work at EngineYard PEAR Developer From Iceland Helgi
7. Co-founded Orchestra.io Work at EngineYard PEAR Developer From Iceland @h on Twitter Helgi
8. What is Gearman
9. Job Queue
10. Task Processing
11. Your Client Code Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Gearman Job Server gearmand Gearman Worker API (C, PHP, Perl, Python, ...) Your Worker Code Your App Gearman
12. Good for... what?
13. Good for
14. Good for Processing (Images, text)
15. Good for Processing (Images, text) Logs (Web Server, etc)
16. Good for Processing (Images, text) Logs (Web Server, etc) Data Mining
17. Good for Processing (Images, text) Logs (Web Server, etc) Data Mining Mass Emails
18. Good for Processing (Images, text) Logs (Web Server, etc) Data Mining Mass Emails Intensive transformation
19. Good for Processing (Images, text) Logs (Web Server, etc) Data Mining Mass Emails Intensive transformation Search
20. Perception Helps with
21. Perception
22. Perception
23. Perception Defer intensive processes
24. Perception Defer intensive processes Give instant feedback
25. Perception Defer intensive processes Give instant feedback Users keep on browsing
26. Perception Defer intensive processes Give instant feedback Users keep on browsing
27. “It all depends on how we look at things, and not how they are in themselves.” - Carl G. Jung
29. Farm out work
30. Language Agnostic
31. Platform Agnostic
32. Your Client Code Gearman Job Server gearmand Your Worker Code Your App Gearman Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Gearman Worker API (C, PHP, Perl, Python, ...)
33. Your Client Code Gearman Job Server gearmand Your Worker Code Your App Gearman Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Gearman Worker API (C, PHP, Perl, Python, ...)
34. Your Client Code Gearman Job Server gearmand Your Worker Code Your App Gearman Gearman Client API (C, PHP, Perl, MySQL UDF, ...) Gearman Worker API (C, PHP, Perl, Python, ...)
35. Userland Code pear.php.net/net_gearman PHP Extension pecl.php.net/gearman
36. PHP Releases
37. PHP Releases In Theory
38. •Linux •RPM •DEB •Windows •OS X •Solaris Multi Platform releases
39. Daily / Hourly Snapshots
40. Push out to mirrors / CDNs
41. A Story!
42. Financial Software
43. 3000+ Clients
44. Each one has 5 external data sources
45. Each data source is a web service
46. Original solution was a big PHP script
47. Took over 24 hours
48. New solution is gearman
49. Ran every 6 hours every day
50. Cron
51. Cron Gearman
52. Cron Gearman Update
53. Cron Gearman Update 1 2 3 4 5 Web Services
54. Cron Gearman Update 1 2 3 4 5 Web Services 5 Processing 4 1 2 3
55. But! That wasn’t enough
56. Job kicked off on login
57. 1 2 3 4 5 Update Web Services Processing 5 4 3 2 1
58. 1 2 3 4 5 Update Web Services Processing 5 4 3 2 1
59. Teach me the magic!
60. Workers
61. Process Reuse
62. Multiple Capabilities
63. WS Update Worker-1 Process
64. WS Update Worker-1 Process
65. WS Update Worker-1 Worker-2 Worker-3 Process
66. <?php define('NET_GEARMAN_JOB_PATH', __DIR__ . '/../tasks/'); define('NET_GEARMAN_JOB_CLASS_PREFIX', ''); require_once 'Net/Gearman/Worker.php'; try { $worker = new Net_Gearman_Worker(array('localhost:4370', 'localhost:4371')); $worker->addAbility('Update'); $worker->beginWork(); } catch (Net_Gearman_Exception $e) { echo $e->getMessage() . "\n"; exit; }
67. <?php define('NET_GEARMAN_JOB_PATH', __DIR__ . '/../tasks/'); define('NET_GEARMAN_JOB_CLASS_PREFIX', ''); require_once 'Net/Gearman/Worker.php'; try { $worker = new Net_Gearman_Worker(array('localhost:4370', 'localhost:4371')); $worker->addAbility('Update'); $worker->addAbility('SaveDB'); $worker->addAbility('SaveMemcache'); $worker->beginWork(); } catch (Net_Gearman_Exception $e) { echo $e->getMessage() . "\n"; exit; }
68. Workers pick tasks based on type from top to bottom
69. Pass in -R to Server for capabilities to be assigned in a round-robin order
70. Helps when one worker has many capabilities
71. ✓ High Priority ✓ Normal Priority (default) ✓ Low Priority Task Priorities
72. Task Priorities 3 different queues
73. Task Priorities 3 different queues First in - First out within each queue
74. Server Persistence
75. Server Persistence Local memory by default
76. ✓ Memcache ✓ Drizzle ✓ MySQL ✓ PostgresSQL ✓ SQLite 3 ✓ TokyoCabinet ✓ Redis (in dev) ✓ MongoDB (in dev) Server Persistence
77. gearmand -d -q libsqlite3 --libsqlite3-db=/tmp/demo/gearman.db --listen=localhost --port=4370
78. Server HA
79. Server HA What if the server dies?
80. Server HA Keep multiple servers
81. gearmand -d -q libsqlite3 --libsqlite3-db=/tmp/demo/gearman.db --listen=localhost --port=4370 gearmand -d -q libsqlite3 --libsqlite3-db=/tmp/demo/gearman.db --listen=localhost --port=4371
82. Server HA Workers cycle through them
83. Worker 1 Client Client Server 1 Server 2 Client Client Worker 2 Worker 3
84. Worker 1 Client Client Server 1 Server 2 Client Client Worker 2 Worker 3
85. Worker 1 Client Client Server 1 Server 2 Client Client Worker 2 Worker 3
86. Code, Code, Code...
87. <?php define('NET_GEARMAN_JOB_PATH', __DIR__ . '/../tasks/'); define('NET_GEARMAN_JOB_CLASS_PREFIX', ''); require_once 'Net/Gearman/Worker.php'; try { $worker = new Net_Gearman_Worker(array('localhost:4370', 'localhost:4371')); $worker->addAbility('Update'); $worker->addAbility('SaveDB'); $worker->addAbility('SaveMemcache'); $worker->beginWork(); } catch (Net_Gearman_Exception $e) { echo $e->getMessage() . "\n"; exit; }
88. <?php try { // Do all the gearman magic here. kick off a job. Be the client. $servers = array( 'localhost:4370', 'localhost:4371' ); require_once 'Net/Gearman/Client.php'; $client = new Net_Gearman_Client($servers); // Emulate that a single client signs in. $clients = range(1, 50); $c = $clients[rand(0, 49)]; $jobs = $client->Update($c); echo "Added client $c to the queue with the handler $jobs\n"; } catch (Net_Gearman_Exception $e) { echo 'Could not connect to any server'; }
89. <?php require_once __DIR__ . '/../Base.php'; class Update extends Base { public function run($args) { $this->connect(); // Fetch Salary information for this client $this->client->WebserviceSalary($args); // Fetch all the Point of Sale information for this client $this->client->WebservicePos($args); return true; } }
90. <?php class WebserviceSalary extends Net_Gearman_Job_Common { public function run($args) { $data = @file_get_contents('http://gearman.talks.conf/salary.json'); if (!$data) { return false; } $file = 'salary-' . rand(1,9999) . '-' . time(); file_put_contents('/tmp/demo/raw/' . $file, $data); return true; } }
91. Foreground vs Background
92. How to keep it all alive and running?
93. Supervisord
94. Manages Processes
96. ✓ Makes sure processes are running
97. ✓ Makes sure processes are running ✓ Can start more than one of same process
98. ✓ Makes sure processes are running ✓ Can start more than one of same process ✓ Restarts crashed processes
99. ✓ Makes sure processes are running ✓ Can start more than one of same process ✓ Restarts crashed processes ✓ Simple to configure and manage
100. ✓ Makes sure processes are running ✓ Can start more than one of same process ✓ Restarts crashed processes ✓ Simple to configure and manage ✓ Each worker type has a different config
101. Great for long running process!
102. j.mp/supvervisord
103. [program:gearman-worker-save-update] command=php /var/www/gearman/workers/worker1.php numprocs=10 process_name=%(program_name)s-%(process_num)s autorestart=true [program:gearman-worker-ws-1] command=php /var/www/gearman/workers/worker2.php numprocs=5 process_name=%(program_name)s-%(process_num)s autorestart=true [program:gearman-worker-ws-2] command=php /var/www/gearman/workers/worker3.php numprocs=5 process_name=%(program_name)s-%(process_num)s autorestart=true
104. gearman_manager
105. gearman_manager j.mp/gearman_manager
106. ✓ Makes sure processes are running ✓ Can start more than one of same process ✓ Restarts crashed processes ✓ Simple to configure and manage ✓ Each worker type has a different config
107. Sounds familiar?
108. Does the worker heavy lifting for you
109. You can focus on writing amazing tasks
110. [GearmanManager] host=localhost:4370,localhost:4371 ; workers can be found in this dir worker_dir=/var/www/gearman/tasks ; Reload workers as new code is available auto_update=1 ; 20 workers will do all jobs count=20 ; Each job will have minimum 1 worker that does only that job dedicated_count=1 [Webservice_Pos] count=10 [Webservice_Salary] count=10
112. gearman_manager + Supervisord = Awesome!
114. Another Story!
115. CloudSplit
116. Near Real Time Cloud Analytics
117. Clients install logging agent locally
118. syslogd
119. Public API
120. Multiple Persistent Gearman Servers
121. Internal DB API
122. Agent
123. Agent syslogd
124. Agent syslogd API
125. Agent syslogd API Load Balanced
126. Agent syslogd API Gearman Gearman Load Balanced
127. Agent syslogd API Gearman Gearman Load Balanced Persistent
128. Agent syslogd API Gearman Gearman Worker Worker Worker Load Balanced Persistent
129. Agent syslogd API Gearman Gearman Worker Worker Worker Internal API Load Balanced Persistent
130. Agent syslogd API Gearman Gearman Worker Worker Worker Internal API Load Balanced Load Balanced Persistent
131. Agent syslogd API Gearman Gearman CouchDB Worker Worker Worker Internal API Load Balanced Load Balanced Persistent
132. engineyardjaunt.eventbrite.com Tomorrow: Tonight: Hackathon!
133. Questions? @h helgi@engineyard.com Please rate at joind.in/6496