Don't keep customer waiting

How to process tasks in the background while the customer drinks some coffee, using the queue.

Some tricks about laravel performance.
#laravel #performing #queues

Transcript

1. DON’T KEEP CUSTOMER WAITING Eduardo de Brito Colombo eduardobcolombo@gmail.com about.me/eduardobcolombo
3. HAVE YOU HEARD ABOUT JOBS, QUEUES, EVENTS...?
6. YOU MUST THINK • How am I planning software? • Do not just do what everyone else does; • Try to understand the labyrinth first and then plot your route; • Why use this pattern? Pablo said that it is the best. But it works for my solution too?
7. LARAVEL QUEUES Laravel queues provide a unified API across a variety of different queue backends, such as Beanstalk, Amazon SQS, Redis, or even a relational database. Queues allow you to defer the processing of a time consuming task, such as sending an email, until a later time. Deferring these time consuming tasks drastically speeds up web requests to your application.
8. http://www.mashpy.me/2017/03/add-queue-in-laravel-5.1-using-beanstalkd-and-keep-it-live-with-supervisor.html
9. ADVANTAGES • Asynchronous: Queue it now, run it later; • Decoupling: Separates application logic; • Resilience: Won't take down your whole application if part of it fails; • Redundancy: Can retry jobs if they fail; • Guarantees: Makes sure that jobs will be processed; • Scalable: Many workers can process individual jobs in a queue; • Profiling: Can aid in identifying performance issues;
10. DISADVANTAGES • Asynchronous: you have to wait until a job is complete; • Load: each job in the queue must wait its turn before it can be processed. If one job overruns, it affects each subsequent job; • Architecture: the application needs to be designed with queues in mind.
11. USE CASES • Any time consuming process can be placed in a queue; • Sending/receiving data from a third-party APIs; • Sending an e-mail; • Generating reports; • Running labour intensive processes.
12. HANDS-ON
13. CONFIG/QUEUE.PHP 'default' => env('QUEUE_CONNECTION', ‘database') 'connections' => [
 'sync' => [ driver' => ‘sync’ ],
 'database' => […],
 'beanstalkd' => […],
 'sqs' => […],
 'redis' => […],
 ],
 'failed' => […]
14. CONFIG/QUEUE.PHP 'database' => [
 'driver' => 'database',
 'table' => 'jobs',
 'queue' => 'default',
 'retry_after' => 90,
 ],
15. DATABASE $ php artisan queue:table Schema::create('jobs', function (Blueprint $table) { $table->bigIncrements('id'); $table->string('queue')->index(); $table->longText('payload'); $table->unsignedTinyInteger('attempts'); $table->unsignedInteger('reserved_at')->nullable(); $table->unsignedInteger('available_at'); $table->unsignedInteger(‘created_at'); }); $ php artisan migrate
16. WELL DONE DATABASE CONFIGURED
17. CREATING JOBS You can use the artisan command to create your Job classes if you prefer. $ php artisan make:job ProcessVideo By default, all of the queueable jobs for your application are stored in the app/Jobs directory
18. namespace App\Jobs; use App\Video; use App\VideoProcessor; use Illuminate\Bus\Queueable; use Illuminate\Queue\SerializesModels; use Illuminate\Queue\InteractsWithQueue; use Illuminate\Contracts\Queue\ShouldQueue; use Illuminate\Foundation\Bus\Dispatchable; class ProcessVideo implements ShouldQueue { use Dispatchable, InteractsWithQueue, Queueable, SerializesModels; public function __construct(Video $video) { $this->video = $video; } public function handle(VideoProcessor $processor) { // process uploaded video } }
19. CLASS STRUCTURE Job classes are very simple, normally containing only a handle method which is called when the job is processed by the queue.
20. SERIALIZES MODELS When the job is actually handled, the queue system will automatically re-retrieve the full model instance from the database. It's all totally transparent to your application and prevents issues that can arise from serializing full Eloquent model instances.
21. DISPATCHING JOBS Dispatch it using the dispatch method on the job itself. The arguments passed to the dispatch method will be given to the job's constructor.
22. namespace App\Http\Controllers; use App\Jobs\ProcessVideo; use Illuminate\Http\Request; use App\Http\Controllers\Controller; class VideoController extends Controller { public function store(Request $request) { // Upload/Create Video… ProcessVideo::dispatch($video); } }
23. namespace App\Jobs; use App\Video; use App\VideoProcessor; use Illuminate\Bus\Queueable; use Illuminate\Queue\SerializesModels; use Illuminate\Queue\InteractsWithQueue; use Illuminate\Contracts\Queue\ShouldQueue; use Illuminate\Foundation\Bus\Dispatchable; class ProcessVideo implements ShouldQueue { use Dispatchable, InteractsWithQueue, Queueable, SerializesModels; public function __construct(Video $video) { $this->video = $video; } public function handle(VideoProcessor $processor) { // process uploaded video } }
24. DELAYED DISPATCHING If you would like to delay the execution of a queued job, you may use the delay method when dispatching a job.
25. namespace App\Http\Controllers; use App\Jobs\ProcessVideo; use Illuminate\Http\Request; use App\Http\Controllers\Controller; class VideoController extends Controller { public function store(Request $request) { // Upload/Create Video… ProcessVideo::dispatch($video) ->delay(now()->addMinutes(10)); } }
26. JOB CHAINING Job chaining allows you to specify a list of queued jobs that should be run in sequence. If one job in the sequence fails, the rest of the jobs will not be run.
27. namespace App\Http\Controllers; use App\Jobs\ProcessVideo; use Illuminate\Http\Request; use App\Http\Controllers\Controller; class VideoController extends Controller { public function store(Request $request) { // Upload/Create Video… ProcessVideo::withChain([ new OptimizeVideo, new ReleaseVideo ]) ->dispatch($video) ->delay(now()->addMinutes(10)); } }
28. CUSTOMIZING THE QUEUE & CONNECTION ProcessVideo::dispatch($video) ->onConnection('redis') ->onQueue(‘encode');
29. http://www.mashpy.me/2017/03/add-queue-in-laravel-5.1-using-beanstalkd-and-keep-it-live-with-supervisor.html
30. DEVELOPMENT ENV Laravel web app: http://localhost:8000
 $ php artisan serve Laravel Queue Worker $ php artisan queue:work
31. TIMEOUT $ php artisan queue:work --timeout=60 The --timeout option specifies how long the Laravel queue master process will wait before killing off a child queue worker that is processing a job. Sometimes a child queue process can become "frozen" for various reasons, such as an external HTTP call that is not responding. The --timeout option removes frozen processes that have exceeded that specified time limit:
32. ATTEMPTS $ php artisan queue:work --tries=3 One approach to specifying the maximum number of times a job may be attempted is via the --tries.
33. QUEUE CONNECTION $ php artisan queue:work redis $ php artisan queue:work sqs You may also specify which queue connection the worker should utilize. The connection name passed to the work command should correspond to one of the connections defined in your config/queue.php configuration file:
34. MULTIPLE/PRIORITY $ php artisan queue:work \ --queue=high,csv,encoder,default Some applications may not need to ever push jobs onto multiple queues. However, pushing jobs to multiple queues can be especially useful for applications that wish to prioritize or segment how jobs are processed, since the Laravel queue worker allows you to specify which queues it should process by priority.
35. CONNECTION & QUEUE $ php artisan queue:work redis --queue=video You may customize your queue worker even further by only processing particular queues for a given connection. For example, if all of your emails are processed in an emails queue on your redis queue connection.
36. Have you heard about Supervisor? “Note that once the queue:work command has started, it will continue to run until it is manually stopped or you close your terminal.”
37. SUPERVISOR $ sudo apt-get install supervisor Supervisor is a process monitor for the Linux operating system, and will automatically restart your queue:work process if it fails.
38. SUPERVISOR [program:laravel-worker] process_name=%(program_name)s_%(process_num)02d command=php /../artisan queue:work redis --sleep=3 — tries=3 autostart=true autorestart=true user=laravel numprocs=10 redirect_stderr=true stdout_logfile=/home/forge/app.com/worker.log In this example, the numprocs directive will instruct Supervisor to run 10 queue:work processes and monitor all of them, automatically restarting them if they fail.
39. DEALING WITH FAILED JOBS $ php artisan queue:failed-table $ php artisan migrate After a job has exceeded this amount of attempts, it will be inserted into the failed_jobs database table.
40. HELPFUL $ php artisan queue:work --once $ php artisan queue:work --stop-when-empty $ php artisan queue:restart $ php artisan queue:work --sleep=3 $ php artisan queue:failed $ php artisan queue:retry 5
41. HELPFUL $ php artisan queue:retry all $ php artisan queue:forget 5 $ php artisan queue:flush
42. Cal Evans // @calevans “Frameworks are tools, your language is your toolbox.”
44. EVENTS Laravel's events provide a simple observer implementation, allowing you to subscribe and listen for various events that occur in your application.
45. EVENTS Event classes are typically stored in the 
 app/Events directory, while their listeners are stored in app/Listeners.
46. EVENTS A single event can have multiple listeners that do not depend on each other.
 For example, you may wish to send a Slack notification to your user each time an order has shipped. Instead of coupling your order processing code to your Slack notification code, you can raise an OrderShipped event, which a listener can receive and transform into a Slack notification.
47. REGISTERING EVENTS & LISTENERS File: App\Providers\EventServiceProvider protected $listen = [ 'App\Events\OrderShipped' => [‘App\Listeners\SendShipmentNotification'] ];
48. EVENTSERVICEPROVIDER File: App\Providers\EventServiceProvider use App\Events\OrderShipped; use App\Listeners\SendShipmentNotification; protected $listen = [ OrderShipped::class => [ SendShipmentNotification::class ], ];
49. THE EVENT File: App\Events\OrderShipped
 class OrderShipped { use SerializesModels; public $order; public function __construct(Order $order) { $this->order = $order; } }
50. LISTENERS Event listeners receive the event instance in their handle method.
51. THE LISTENER File: App\Listeners\ SendShipmentNotification
 namespace App\Listeners; use App\Events\OrderShipped; class SendShipmentNotification { public function __construct() { // } public function handle(OrderShipped $event) { // Access the order using $event->order... } }
52. FIRING THE EVENT namespace App\Http\Controllers; use Illuminate\Http\Request; use App\Http\Controllers\Controller; class OrderController extends Controller { public function store(Request $request) { // The order process… event(new OrderShipped($order)); } }
53. QUEUED EVENT LISTENERS
54. QUEUED EVENT LISTENERS Queueing listeners can be beneficial if your listener is going to perform a slow task such as sending an e- mail or making an HTTP request.
55. QUEUED EVENT LISTENERS File: App\Listeners\ SendShipmentNotification
 namespace App\Listeners; use App\Events\OrderShipped; class SendShipmentNotification { public function __construct() { // } public function handle(OrderShipped $event) { // Access the order using $event->order... } }
56. QUEUED EVENT LISTENERS File: App\Listeners\ SendShipmentNotification
 namespace App\Listeners; use App\Events\OrderShipped; use Illuminate\Contracts\Queue\ShouldQueue; class SendShipmentNotification implements ShouldQueue { public function __construct() { // } public function handle(OrderShipped $event) { // Access the order using $event->order... } }
57. THAT'S IT! Now, when this listener is called for an event, it will be automatically queued by the event dispatcher using Laravel's queue system. If no exceptions are thrown when the listener is executed by the queue, the queued job will automatically be deleted after it has finished processing.
59. THE N+1 QUERY PROBLEM This problem occurs when the code needs to load the children of a parent-child relationship (the “many” in the “one-to-many”).
60. THE N+1 QUERY PROBLEM ... public function index() { $videos = Video::get(); foreach($videos as $video) { $quality = $video->qualities; // do something with $quality } ...
61. THE N+1 QUERY PROBLEM 1. select * from `videos`; 2. select * from `qualities` where `qualities`.`video_id` = 1 and `qualities`.`user_id` is not null; 3. select * from `qualities` where `qualities`.`video_id` = 2 and `qualities`.`user_id` is not null; … N. select * from `qualities` where `qualities`.`video_id` = N and `qualities`.`user_id` is not null;
62. EAGER LOADING ... public function index() { $videos = Video::with('qualities')->get(); foreach($videos as $video) { $quality = $video->qualities; // do something with $quality } ...
63. EAGER LOADING 1. select * from `videos`; 2. select * from `qualities` where `qualities`.`video_id` in (‘1’,’2’,’…’,’N’);
64. DATABASE CHUNKING ... public function index() { // it return 100.000 records in one query $videos = Video::get(); //do something with $videos ... // it could spend around 100MB
65. DATABASE CHUNKING ... public function index() { // it return 10.000 records per query (10) Video::chunk(10000, function($videos) { //do something with $videos }); … // it could spend around 10MB This method is very useful for writing Artisan commands that process thousands of records.
66. OPTIMIZATION • Autoloader Optimization • composer install --optimize-autoloader —no-dev • Optimizing Configuration Loading • php artisan config:cache • Optimizing Route Loading • php artisan route:cache
67. ROUTE CACHING • In some cases, your route registration may even be up to 100x faster. • php artisan route:cache
68. CONFIGURATION CACHING To give your application a speed boost, you should cache all of your configuration files into a single file using the config:cache Artisan command. This will combine all of the configuration options for your application into a single file which will be loaded quickly by the framework. $ php artisan config:cache
69. CACHING Sometimes you may wish to retrieve an item from the cache, but also store a default value if the requested item doesn't exist. For example, you may wish to retrieve all users from the cache or, if they don't exist, retrieve them from the database and add them to the cache.
70. OBJECT CACHING $cacheKey = md5(vsprintf(‘%s.%s.%s’, [ auth()->user->id, $course, $actived ];
 $value = Cache::remember($cacheKey, 60, function () { return DB::table(‘users’) ->where(// Do something fancy here...) ->get(); } );
71. MYSQL INDEXING Database indexes in MySQL enable you to accelerate the performance of SELECT query statements. For small tables, an index does not help much. However, if you have tables with a large amount of data, indexes can dramatically improve performance.
72. THANK YOU Eduardo de Brito Colombo eduardobcolombo@gmail.com about.me/eduardobcolombo “ Community works. When it doesn’t, you are the community.” Cal Evans // @calevans
73. REFERÊCIAS • https://laravel.com/docs/5.7/queues • https://laravel.com/docs/5.7/events • http://supervisord.org/index.html • https://www.sitepoint.com/message-queues-comparing- beanstalkd-ironmq-amazon-sqs/ • https://www.a2hosting.com/kb/developer-corner/mysql/ using-indexes-to-improve-mysql-query-performance •