Shrine: Handle file uploads like it's 2017

https://www.youtube.com/watch?v=plD-RkKEay0

Transcript

1. Shrine
   Handle file uploads like it’s 2017
   janko-m @jankomarohnic
   

   View Slide

2. Shrine
   Best practices for handling file uploads in Ruby
   janko-m @jankomarohnic
   

   View Slide

3. Paperclip
   CarrierWave ★7,253
   Refile ★2,314
   ★8,357
   Dragonfly ★ 1,982
   Attache ★184
   Paperdragon ★ 116
   

   View Slide

4. 
   Paperclip
   CarrierWave ★7,253
   Refile ★2,314
   ★8,357
   Dragonfly ★ 1,982
   Attache ★184
   Paperdragon ★ 116
   

   View Slide

5. Flow
   

   View Slide

6. Choose a file… picture.jpg Submit
   POST /photos
   Content-Type: multipart/form-data …
   […file content…]
   { name: "file",
   filename: "picture.jpg",
   type: "image/jpeg",
   tempfile: # }
   Rack
   

   View Slide

7. Choose a file… picture.jpg Submit
   POST /albums
   Content-Type: multipart/form-data …
   […file content…]
   #@original_filename=“picture.jpg”,
   @content_type=“image/jpeg”,
   @tempfile=#>
   Rails
   

   View Slide

8. Shrine.storages = {
   store: Shrine::Storage::FileSystem.new(
   "public", prefix: "uploads/store")
   }
   Shrine.plugin :sequel # or :activerecord
   class ImageUploader < Shrine
   # image attachment logic
   end
   class Photo < Sequel::Model
   include ImageUploader[:image]
   end
   

   View Slide

9. photo = Photo.create(image: File.open("…"))
   photo.image_data #=>
   {
   "storage": "store",
   "id": "fd9sdf0ag32kf1s.jpg",
   "metadata": {
   "filename": "nature.jpg",
   "size": 883954,
   "mime_type": "image/jpeg"
   }
   }
   

   View Slide

10. IMPROVEMENT A:
    Retaining file on form redisplays
    

    View Slide

11. Choose a file… picture.jpg
    Submit
    Album title
    Choose a file… No file selected
    Submit
    Album title can’t be blank
    (Waiting for file to be uploaded…)
    
    

    View Slide

12. Shrine.storages = {
    cache: Shrine::Storage::FileSystem.new(…),
    store: Shrine::Storage::FileSystem.new(…),
    }
    {"storage":"cache", "id":"8ag8fdg.jpg", "metadata":{...}}
    Choose a file… No file selected
    Submit
    Album title can’t be blank
    

    View Slide

13. IMPROVEMENT B:
    Direct uploads
    

    View Slide

14. Choose a file… picture.jpg
    Submit
    

    View Slide

15. Shrine.plugin :direct_upload
    Rails.application.routes.draw do
    mount ImageUploader::UploadEndpoint,
    to: "/images"
    end
    POST /images/cache/upload
    jQuery-File-Upload
    Dropzone
    FineUploader
    …
    

    View Slide

16. Choose a file…
    Submit
    32%
    POST /images/cache/upload
    

    View Slide

17. Choose a file…
    Submit
    100%
    {"storage":"cache", "id":"8ag8fdg.jpg", …}
    

    View Slide

18. • User has a better idea how long upload will take
    • User can fill in other fields while file is uploading
    • Multiple files can be uploaded in parallel
    Benefits
    

    View Slide

19. IMPROVEMENT C:
    External storage
    

    View Slide

20. • Receiving file uploads uses server resources
    • Stored files are served through the app
    • Doesn’t work with multiple servers
    • Heroku doesn’t support storing on disk
    • Heroku has a 30-second request timeout
    Downsides of filesystem
    

    View Slide

21. Shrine.storages = {
    cache: Shrine::Storage::S3.new(…),
    store: Shrine::Storage::S3.new(…),
    }
    Shrine.storages = {
    cache: Shrine::Storage::FileSystem.new(…),
    store: Shrine::Storage::FileSystem.new(…),
    }
    

    View Slide

22. 1. GET /images/cache/presign
    {
    "url": "https://my-bucket.s3.amazonaws.com",
    "fields": {
    "key": "b7d575850ba61b44c8a9ff889dfdb14d88cdc25",
    "policy": "eyJleHBpcmF0aW9uIjoiMjAxNS0QwMToxMTo...",
    "x-amz-credential": "5TQ/eu-west-1/s3/aws4_request",
    "x-amz-algorithm": "AWS4-HMAC-SHA256",
    "x-amz-date": "20151024T001129Z",
    "x-amz-signature": "c1eb634f83f96b69bd675f535b3..."
    }
    }
    2. POST https://my-bucket.s3.amazonaws.com
    Content-Type: multipart/form-data
    […file content…]
    

    View Slide

23. IMPROVEMENT D:
    Resumable uploads
    

    View Slide

24. • Flaky connection terminates the upload
    • User needs to change location during upload
    • Computer freezes in the middle of upload
    Large files
    => User needs to retry the whole upload
    

    View Slide

25. Chunking
    5MB 5MB 5MB 5MB 5MB …
    

    View Slide

26. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5
    

    View Slide

27. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5
    32%
    

    View Slide

28. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5
    ✅
    

    View Slide

29. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5
    ✅
    17%
    

    View Slide

30. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5
    ✅
    ❌
    

    View Slide

31. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5
    ✅
    43%
    

    View Slide

32. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5
    ✅
    ✅
    

    View Slide

33. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5
    ✅
    ✅
    17%
    54%
    32%
    

    View Slide

34. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5
    ✅
    ✅
    ✅
    ✅
    ✅
    

    View Slide

35. • S3 & Azure
    • Concurrent chunking
    • Numerous other features
    • Well-designed protocol
    • iOS & Android
    • Ruby implementation
    

    View Slide

36. # config.ru
    require "tus/server"
    map("/files") { run Tus::Server }
    janko-m/tus-ruby-server
    Shrine.storages = {
    cache: Shrine::Storage::Url.new,
    store: Shrine::Storage::S3.new(…),
    }
    janko-m/shrine-url
    

    View Slide

37. • Uploads can be resumed in case of interruption
    • Concurrent chunking can speed up the upload
    • Checksums can ensure that no bytes were lost
    Benefits
    

    View Slide

38. IMPROVEMENT E:
    Backgrounding
    

    View Slide

39. 2048×1365
    800×800
    500×500
    300×300
    

    View Slide

40. previous_cover_photo = album.cover_photo
    album.update(cover_photo: File.open("..."))
    previous_cover_photo.exists? #=> false
    

    View Slide

41. album.destroy
    album.cover_photo.exists? #=> false
    

    View Slide

42. Deleting
    ↓
    Processing
    (= downloading + processing + uploading)
    ↓
    SLOW
    SLOW
    

    View Slide

43. class PromoteJob
    include Sidekiq::Worker
    def perform(data)
    Shrine::Attacher.promote(data)
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote do |data|
    PromoteJob.perform_async(data)
    end
    mperham/sidekiq
    

    View Slide

44. class PromoteJob
    include Sidekiq::Worker
    def perform(data)
    Shrine::Attacher.promote(data)
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote do |data|
    PromoteJob.perform_in(3, data)
    end
    mperham/sidekiq
    

    View Slide

45. class PromoteJob
    include Sidekiq::Worker
    def perform(data)
    ActiveRecord::Base.with_connection do
    Shrine::Attacher.promote(data)
    end
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote do |data|
    PromoteJob.perform_async(data)
    end
    mperham/sidekiq
    

    View Slide

46. class PromoteJob
    include SuckerPunch::Job
    def perform(data)
    Shrine::Attacher.promote(data)
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote do |data|
    PromoteJob.perform_async(data)
    end
    brandonhilkert/sucker_punch
    

    View Slide

47. class PromoteJob < RocketJob::Job
    key :data, Hash
    def perform(data)
    Shrine::Attacher.promote(data)
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote do |data|
    PromoteJob.create!(data: data)
    end
    rocketjob/rocketjob
    

    View Slide

48. class DeleteJob < RocketJob::Job
    key :data, Hash
    def perform(data)
    Shrine::Attacher.delete(data)
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote { |data| … }
    Shrine::Attacher.delete { |data| … }
    rocketjob/rocketjob
    

    View Slide

49. album.cover_photo #=> nil
    album.cover_photo = File.open("...")
    album.save # background job is kicked off
    album.cover_photo.storage_key #=> "cache"
    # … background job finishes
    album.reload
    album.cover_photo.storage_key #=> "store"
    

    View Slide

50. album.destroy
    album.cover_photo.exists? #=> true
    # … background job finishes
    album.cover_photo.exists? #=> false
    

    View Slide

51. Benefits
    • Fast for the user
    • Removes impact on request throughput
    • Can be scaled individually
    • Can automatically retry timeouts
    • Easier to manually retry failures that require action
    • Easier to monitor performance
    

    View Slide

52. Demo
    

    View Slide

53. Credits to Refile
    @jnicklas
    

    View Slide

54. janko-m/shrine
    

    View Slide