Parallel batch

Parallel batch helps to distribute batches across servers. It works great with Rails app.

Transcript

1. Parallel batch http://github.com/officialfm/parallel_batch
2. I am Alexis Bernard and work at Official.fm as chief technical architect. alexis@official.fm github.com/alexisbernard twitter.com/alexis_bernard
3. What is a batch? A batch is a processing that runs on a group of records.
4. How do we code batches with Rails on Rails? Track.not_encoded.each do |track| track.not_encoded.encode_mp3 end Loads all instances in memory Not usable on large tables
5. Track.not_encoded.find_each do |track| track.encode_mp3 end Load instances per batch of 1000, but not parallelizable Since it is not parallelized, this batch can take many days or weeks ! Improved version
6. How can we parallelize ? Threads and processes are limited by resources of a single device. Distributing has no resources limitation, since it run on many devices. Parallel batch handles everything to distribute your batches across servers.
7. class TrackBatch < ParallelBatch def scope Track.not_encoded end def perform(track) track.encode_mp3 end end TrackBatch.start(5) # Start 5 workers
8. How it works under the hood ? Parallel batch relies on the database as a mutex. It prevents workers to process a record multiple times.
9. ParallelBatch is an ActiveRecord sub class irb :001 > ParallelBatch ParallelBatch( id: integer, type: string, offset: string, created_at: datetime, updated_at: datetime)
10. class ParallelBatch < ActiveRecord::Base def run while records = next_batch records.each { |record| perform(record) } end end def next_batch transaction do reload(lock: true) next unless (records = find_records).last update_attributes!(offset: records.last.id) records end end end
11. Thanks for your attention ! Any question ?