Shrine: Handle file uploads like it's 2017

Shrine: Handle file uploads like it's 2017

376e4eb9dc6c2e33d1330262edc4f109?s=128

Janko Marohnić

February 22, 2017
Tweet

Transcript

  1. Shrine Handle file uploads like it’s 2017 janko-m @jankomarohnic

  2. Shrine Best practices for handling file uploads in Ruby janko-m

    @jankomarohnic
  3. Paperclip CarrierWave ★7,253 Refile ★2,314 ★8,357 Dragonfly ★ 1,982 Attache

    ★184 Paperdragon ★ 116
  4. Paperclip CarrierWave ★7,253 Refile ★2,314 ★8,357 Dragonfly ★ 1,982 Attache

    ★184 Paperdragon ★ 116
  5. Flow

  6. Choose a file… picture.jpg Submit POST /photos Content-Type: multipart/form-data …

    […file content…] { name: "file", filename: "picture.jpg", type: "image/jpeg", tempfile: #<Tempfile:/var/…/RackMultipart-286> } Rack
  7. Choose a file… picture.jpg Submit POST /albums Content-Type: multipart/form-data …

    […file content…] #<ActionDispatch::Http::UploadedFile @original_filename=“picture.jpg”, @content_type=“image/jpeg”, @tempfile=#<Tempfile:/var/…/RackMultipart-286>> Rails
  8. Shrine.storages = { store: Shrine::Storage::FileSystem.new( "public", prefix: "uploads/store") } Shrine.plugin

    :sequel # or :activerecord class ImageUploader < Shrine # image attachment logic end class Photo < Sequel::Model include ImageUploader[:image] end
  9. photo = Photo.create(image: File.open("…")) photo.image_data #=> { "storage": "store", "id":

    "fd9sdf0ag32kf1s.jpg", "metadata": { "filename": "nature.jpg", "size": 883954, "mime_type": "image/jpeg" } }
  10. IMPROVEMENT A: Retaining file on form redisplays

  11. Choose a file… picture.jpg Submit Album title Choose a file…

    No file selected Submit Album title can’t be blank (Waiting for file to be uploaded…)
  12. Shrine.storages = { cache: Shrine::Storage::FileSystem.new(…), store: Shrine::Storage::FileSystem.new(…), } {"storage":"cache", "id":"8ag8fdg.jpg",

    "metadata":{...}} Choose a file… No file selected Submit Album title can’t be blank
  13. IMPROVEMENT B: Direct uploads

  14. Choose a file… picture.jpg Submit

  15. Shrine.plugin :direct_upload Rails.application.routes.draw do mount ImageUploader::UploadEndpoint, to: "/images" end POST

    /images/cache/upload jQuery-File-Upload Dropzone FineUploader …
  16. Choose a file… Submit 32% POST /images/cache/upload

  17. Choose a file… Submit 100% {"storage":"cache", "id":"8ag8fdg.jpg", …}

  18. • User has a better idea how long upload will

    take • User can fill in other fields while file is uploading • Multiple files can be uploaded in parallel Benefits
  19. IMPROVEMENT C: External storage

  20. • Receiving file uploads uses server resources • Stored files

    are served through the app • Doesn’t work with multiple servers • Heroku doesn’t support storing on disk • Heroku has a 30-second request timeout Downsides of filesystem
  21. Shrine.storages = { cache: Shrine::Storage::S3.new(…), store: Shrine::Storage::S3.new(…), } Shrine.storages =

    { cache: Shrine::Storage::FileSystem.new(…), store: Shrine::Storage::FileSystem.new(…), }
  22. 1. GET /images/cache/presign { "url": "https://my-bucket.s3.amazonaws.com", "fields": { "key": "b7d575850ba61b44c8a9ff889dfdb14d88cdc25",

    "policy": "eyJleHBpcmF0aW9uIjoiMjAxNS0QwMToxMTo...", "x-amz-credential": "5TQ/eu-west-1/s3/aws4_request", "x-amz-algorithm": "AWS4-HMAC-SHA256", "x-amz-date": "20151024T001129Z", "x-amz-signature": "c1eb634f83f96b69bd675f535b3..." } } 2. POST https://my-bucket.s3.amazonaws.com Content-Type: multipart/form-data […file content…]
  23. IMPROVEMENT D: Resumable uploads

  24. • Flaky connection terminates the upload • User needs to

    change location during upload • Computer freezes in the middle of upload Large files => User needs to retry the whole upload
  25. Chunking 5MB 5MB 5MB 5MB 5MB …

  26. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5
  27. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 32%
  28. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅
  29. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ 17%
  30. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ ❌
  31. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ 43%
  32. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ ✅
  33. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ ✅ 17% 54% 32%
  34. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ ✅ ✅ ✅ ✅
  35. • S3 & Azure • Concurrent chunking • Numerous other

    features • Well-designed protocol • iOS & Android • Ruby implementation
  36. # config.ru require "tus/server" map("/files") { run Tus::Server } janko-m/tus-ruby-server

    Shrine.storages = { cache: Shrine::Storage::Url.new, store: Shrine::Storage::S3.new(…), } janko-m/shrine-url
  37. • Uploads can be resumed in case of interruption •

    Concurrent chunking can speed up the upload • Checksums can ensure that no bytes were lost Benefits
  38. IMPROVEMENT E: Backgrounding

  39. 2048×1365 800×800 500×500 300×300

  40. previous_cover_photo = album.cover_photo album.update(cover_photo: File.open("...")) previous_cover_photo.exists? #=> false

  41. album.destroy album.cover_photo.exists? #=> false

  42. Deleting ↓ Processing (= downloading + processing + uploading) ↓

    SLOW SLOW
  43. class PromoteJob include Sidekiq::Worker def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin

    :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end mperham/sidekiq
  44. class PromoteJob include Sidekiq::Worker def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin

    :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_in(3, data) end mperham/sidekiq
  45. class PromoteJob include Sidekiq::Worker def perform(data) ActiveRecord::Base.with_connection do Shrine::Attacher.promote(data) end

    end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end mperham/sidekiq
  46. class PromoteJob include SuckerPunch::Job def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin

    :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end brandonhilkert/sucker_punch
  47. class PromoteJob < RocketJob::Job key :data, Hash def perform(data) Shrine::Attacher.promote(data)

    end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.create!(data: data) end rocketjob/rocketjob
  48. class DeleteJob < RocketJob::Job key :data, Hash def perform(data) Shrine::Attacher.delete(data)

    end end Shrine.plugin :backgrounding Shrine::Attacher.promote { |data| … } Shrine::Attacher.delete { |data| … } rocketjob/rocketjob
  49. album.cover_photo #=> nil album.cover_photo = File.open("...") album.save # background job

    is kicked off album.cover_photo.storage_key #=> "cache" # … background job finishes album.reload album.cover_photo.storage_key #=> "store"
  50. album.destroy album.cover_photo.exists? #=> true # … background job finishes album.cover_photo.exists?

    #=> false
  51. Benefits • Fast for the user • Removes impact on

    request throughput • Can be scaled individually • Can automatically retry timeouts • Easier to manually retry failures that require action • Easier to monitor performance
  52. Demo

  53. Credits to Refile @jnicklas

  54. janko-m/shrine