Deep Dive
into
Gearman
Helgi Þormar Þorbjörnsson
PHPTek, Chicago, 24th May, 2012
Slide 2
Slide 2 text
Helgi
Slide 3
Slide 3 text
Co-founded Orchestra.io
Helgi
Slide 4
Slide 4 text
Co-founded Orchestra.io
Work at EngineYard
Helgi
Slide 5
Slide 5 text
Co-founded Orchestra.io
Work at EngineYard
PEAR Developer
Helgi
Slide 6
Slide 6 text
Co-founded Orchestra.io
Work at EngineYard
PEAR Developer
From Iceland
Helgi
Slide 7
Slide 7 text
Co-founded Orchestra.io
Work at EngineYard
PEAR Developer
From Iceland
@h on Twitter
Helgi
Slide 8
Slide 8 text
What is Gearman
Slide 9
Slide 9 text
Job Queue
Slide 10
Slide 10 text
Task Processing
Slide 11
Slide 11 text
Your Client Code
Gearman Client API
(C, PHP, Perl, MySQL UDF, ...)
Gearman Job Server
gearmand
Gearman Worker API
(C, PHP, Perl, Python, ...)
Your Worker Code
Your App Gearman
Slide 12
Slide 12 text
Good for... what?
Slide 13
Slide 13 text
Good for
Slide 14
Slide 14 text
Good for
Processing (Images, text)
Slide 15
Slide 15 text
Good for
Processing (Images, text)
Logs (Web Server, etc)
Slide 16
Slide 16 text
Good for
Processing (Images, text)
Logs (Web Server, etc)
Data Mining
Slide 17
Slide 17 text
Good for
Processing (Images, text)
Logs (Web Server, etc)
Data Mining
Mass Emails
Slide 18
Slide 18 text
Good for
Processing (Images, text)
Logs (Web Server, etc)
Data Mining
Mass Emails
Intensive transformation
Slide 19
Slide 19 text
Good for
Processing (Images, text)
Logs (Web Server, etc)
Data Mining
Mass Emails
Intensive transformation
Search
Slide 20
Slide 20 text
Perception
Helps with
Slide 21
Slide 21 text
Perception
Slide 22
Slide 22 text
Perception
Slide 23
Slide 23 text
Perception
Defer intensive processes
Slide 24
Slide 24 text
Perception
Defer intensive processes
Give instant feedback
Slide 25
Slide 25 text
Perception
Defer intensive processes
Give instant feedback
Users keep on browsing
Slide 26
Slide 26 text
Perception
Defer intensive processes
Give instant feedback
Users keep on browsing
Slide 27
Slide 27 text
“It all depends on how we
look at things, and not how
they are in themselves.”
- Carl G. Jung
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
Farm out work
Slide 30
Slide 30 text
Language Agnostic
Slide 31
Slide 31 text
Platform Agnostic
Slide 32
Slide 32 text
Your Client Code
Gearman Job Server
gearmand
Your Worker Code
Your App Gearman
Gearman Client API
(C, PHP, Perl, MySQL UDF, ...)
Gearman Worker API
(C, PHP, Perl, Python, ...)
Slide 33
Slide 33 text
Your Client Code
Gearman Job Server
gearmand
Your Worker Code
Your App Gearman
Gearman Client API
(C, PHP, Perl, MySQL UDF, ...)
Gearman Worker API
(C, PHP, Perl, Python, ...)
Slide 34
Slide 34 text
Your Client Code
Gearman Job Server
gearmand
Your Worker Code
Your App Gearman
Gearman Client API
(C, PHP, Perl, MySQL UDF, ...)
Gearman Worker API
(C, PHP, Perl, Python, ...)
Update($c);
echo "Added client $c to the queue with the handler $jobs\n";
} catch (Net_Gearman_Exception $e) {
echo 'Could not connect to any server';
}
Slide 89
Slide 89 text
connect();
// Fetch Salary information for this client
$this->client->WebserviceSalary($args);
// Fetch all the Point of Sale information for this client
$this->client->WebservicePos($args);
return true;
}
}
Slide 90
Slide 90 text
Slide 91
Slide 91 text
Foreground
vs
Background
Slide 92
Slide 92 text
How to keep it all
alive and running?
Slide 93
Slide 93 text
Supervisord
Slide 94
Slide 94 text
Manages Processes
Slide 95
Slide 95 text
No content
Slide 96
Slide 96 text
✓ Makes sure processes are running
Slide 97
Slide 97 text
✓ Makes sure processes are running
✓ Can start more than one of same process
Slide 98
Slide 98 text
✓ Makes sure processes are running
✓ Can start more than one of same process
✓ Restarts crashed processes
Slide 99
Slide 99 text
✓ Makes sure processes are running
✓ Can start more than one of same process
✓ Restarts crashed processes
✓ Simple to configure and manage
Slide 100
Slide 100 text
✓ Makes sure processes are running
✓ Can start more than one of same process
✓ Restarts crashed processes
✓ Simple to configure and manage
✓ Each worker type has a different config
✓ Makes sure processes are running
✓ Can start more than one of same process
✓ Restarts crashed processes
✓ Simple to configure and manage
✓ Each worker type has a different config
Slide 107
Slide 107 text
Sounds familiar?
Slide 108
Slide 108 text
Does the worker heavy
lifting for you
Slide 109
Slide 109 text
You can focus on writing
amazing tasks
Slide 110
Slide 110 text
[GearmanManager]
host=localhost:4370,localhost:4371
; workers can be found in this dir
worker_dir=/var/www/gearman/tasks
; Reload workers as new code is available
auto_update=1
; 20 workers will do all jobs
count=20
; Each job will have minimum 1 worker that does only that job
dedicated_count=1
[Webservice_Pos]
count=10
[Webservice_Salary]
count=10
Slide 111
Slide 111 text
No content
Slide 112
Slide 112 text
gearman_manager
+
Supervisord
=
Awesome!
Slide 113
Slide 113 text
No content
Slide 114
Slide 114 text
Another Story!
Slide 115
Slide 115 text
CloudSplit
Slide 116
Slide 116 text
Near Real Time Cloud
Analytics
Slide 117
Slide 117 text
Clients install logging
agent locally
Slide 118
Slide 118 text
syslogd
Slide 119
Slide 119 text
Public API
Slide 120
Slide 120 text
Multiple Persistent
Gearman Servers
Slide 121
Slide 121 text
Internal DB API
Slide 122
Slide 122 text
Agent
Slide 123
Slide 123 text
Agent syslogd
Slide 124
Slide 124 text
Agent syslogd
API
Slide 125
Slide 125 text
Agent syslogd
API
Load
Balanced
Slide 126
Slide 126 text
Agent syslogd
API
Gearman
Gearman
Load
Balanced
Slide 127
Slide 127 text
Agent syslogd
API
Gearman
Gearman
Load
Balanced
Persistent