Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Shrine: Handle file uploads like it's 2017

Shrine: Handle file uploads like it's 2017

Janko Marohnić

February 22, 2017
Tweet

More Decks by Janko Marohnić

Other Decks in Programming

Transcript

  1. Shrine Handle file uploads like it’s 2017 janko-m @jankomarohnic

  2. Shrine Best practices for handling file uploads in Ruby janko-m

    @jankomarohnic
  3. Paperclip CarrierWave ★7,253 Refile ★2,314 ★8,357 Dragonfly ★ 1,982 Attache

    ★184 Paperdragon ★ 116
  4. Paperclip CarrierWave ★7,253 Refile ★2,314 ★8,357 Dragonfly ★ 1,982 Attache

    ★184 Paperdragon ★ 116
  5. Flow

  6. Choose a file… picture.jpg Submit POST /photos Content-Type: multipart/form-data …

    […file content…] { name: "file", filename: "picture.jpg", type: "image/jpeg", tempfile: #<Tempfile:/var/…/RackMultipart-286> } Rack
  7. Choose a file… picture.jpg Submit POST /albums Content-Type: multipart/form-data …

    […file content…] #<ActionDispatch::Http::UploadedFile @original_filename=“picture.jpg”, @content_type=“image/jpeg”, @tempfile=#<Tempfile:/var/…/RackMultipart-286>> Rails
  8. Shrine.storages = { store: Shrine::Storage::FileSystem.new( "public", prefix: "uploads/store") } Shrine.plugin

    :sequel # or :activerecord class ImageUploader < Shrine # image attachment logic end class Photo < Sequel::Model include ImageUploader[:image] end
  9. photo = Photo.create(image: File.open("…")) photo.image_data #=> { "storage": "store", "id":

    "fd9sdf0ag32kf1s.jpg", "metadata": { "filename": "nature.jpg", "size": 883954, "mime_type": "image/jpeg" } }
  10. IMPROVEMENT A: Retaining file on form redisplays

  11. Choose a file… picture.jpg Submit Album title Choose a file…

    No file selected Submit Album title can’t be blank (Waiting for file to be uploaded…)
  12. Shrine.storages = { cache: Shrine::Storage::FileSystem.new(…), store: Shrine::Storage::FileSystem.new(…), } {"storage":"cache", "id":"8ag8fdg.jpg",

    "metadata":{...}} Choose a file… No file selected Submit Album title can’t be blank
  13. IMPROVEMENT B: Direct uploads

  14. Choose a file… picture.jpg Submit

  15. Shrine.plugin :direct_upload Rails.application.routes.draw do mount ImageUploader::UploadEndpoint, to: "/images" end POST

    /images/cache/upload jQuery-File-Upload Dropzone FineUploader …
  16. Choose a file… Submit 32% POST /images/cache/upload

  17. Choose a file… Submit 100% {"storage":"cache", "id":"8ag8fdg.jpg", …}

  18. • User has a better idea how long upload will

    take • User can fill in other fields while file is uploading • Multiple files can be uploaded in parallel Benefits
  19. IMPROVEMENT C: External storage

  20. • Receiving file uploads uses server resources • Stored files

    are served through the app • Doesn’t work with multiple servers • Heroku doesn’t support storing on disk • Heroku has a 30-second request timeout Downsides of filesystem
  21. Shrine.storages = { cache: Shrine::Storage::S3.new(…), store: Shrine::Storage::S3.new(…), } Shrine.storages =

    { cache: Shrine::Storage::FileSystem.new(…), store: Shrine::Storage::FileSystem.new(…), }
  22. 1. GET /images/cache/presign { "url": "https://my-bucket.s3.amazonaws.com", "fields": { "key": "b7d575850ba61b44c8a9ff889dfdb14d88cdc25",

    "policy": "eyJleHBpcmF0aW9uIjoiMjAxNS0QwMToxMTo...", "x-amz-credential": "5TQ/eu-west-1/s3/aws4_request", "x-amz-algorithm": "AWS4-HMAC-SHA256", "x-amz-date": "20151024T001129Z", "x-amz-signature": "c1eb634f83f96b69bd675f535b3..." } } 2. POST https://my-bucket.s3.amazonaws.com Content-Type: multipart/form-data […file content…]
  23. IMPROVEMENT D: Resumable uploads

  24. • Flaky connection terminates the upload • User needs to

    change location during upload • Computer freezes in the middle of upload Large files => User needs to retry the whole upload
  25. Chunking 5MB 5MB 5MB 5MB 5MB …

  26. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5
  27. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 32%
  28. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅
  29. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ 17%
  30. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ ❌
  31. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ 43%
  32. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ ✅
  33. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ ✅ 17% 54% 32%
  34. Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4

    5 ✅ ✅ ✅ ✅ ✅
  35. • S3 & Azure • Concurrent chunking • Numerous other

    features • Well-designed protocol • iOS & Android • Ruby implementation
  36. # config.ru require "tus/server" map("/files") { run Tus::Server } janko-m/tus-ruby-server

    Shrine.storages = { cache: Shrine::Storage::Url.new, store: Shrine::Storage::S3.new(…), } janko-m/shrine-url
  37. • Uploads can be resumed in case of interruption •

    Concurrent chunking can speed up the upload • Checksums can ensure that no bytes were lost Benefits
  38. IMPROVEMENT E: Backgrounding

  39. 2048×1365 800×800 500×500 300×300

  40. previous_cover_photo = album.cover_photo album.update(cover_photo: File.open("...")) previous_cover_photo.exists? #=> false

  41. album.destroy album.cover_photo.exists? #=> false

  42. Deleting ↓ Processing (= downloading + processing + uploading) ↓

    SLOW SLOW
  43. class PromoteJob include Sidekiq::Worker def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin

    :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end mperham/sidekiq
  44. class PromoteJob include Sidekiq::Worker def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin

    :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_in(3, data) end mperham/sidekiq
  45. class PromoteJob include Sidekiq::Worker def perform(data) ActiveRecord::Base.with_connection do Shrine::Attacher.promote(data) end

    end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end mperham/sidekiq
  46. class PromoteJob include SuckerPunch::Job def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin

    :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end brandonhilkert/sucker_punch
  47. class PromoteJob < RocketJob::Job key :data, Hash def perform(data) Shrine::Attacher.promote(data)

    end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.create!(data: data) end rocketjob/rocketjob
  48. class DeleteJob < RocketJob::Job key :data, Hash def perform(data) Shrine::Attacher.delete(data)

    end end Shrine.plugin :backgrounding Shrine::Attacher.promote { |data| … } Shrine::Attacher.delete { |data| … } rocketjob/rocketjob
  49. album.cover_photo #=> nil album.cover_photo = File.open("...") album.save # background job

    is kicked off album.cover_photo.storage_key #=> "cache" # … background job finishes album.reload album.cover_photo.storage_key #=> "store"
  50. album.destroy album.cover_photo.exists? #=> true # … background job finishes album.cover_photo.exists?

    #=> false
  51. Benefits • Fast for the user • Removes impact on

    request throughput • Can be scaled individually • Can automatically retry timeouts • Easier to manually retry failures that require action • Easier to monitor performance
  52. Demo

  53. Credits to Refile @jnicklas

  54. janko-m/shrine