$30 off During Our Annual Pro Sale. View Details »

Shrine: Handle file uploads like it's 2017

Shrine: Handle file uploads like it's 2017

Janko Marohnić

February 22, 2017
Tweet

More Decks by Janko Marohnić

Other Decks in Programming

Transcript

  1. Shrine
    Handle file uploads like it’s 2017
    janko-m @jankomarohnic

    View Slide

  2. Shrine
    Best practices for handling file uploads in Ruby
    janko-m @jankomarohnic

    View Slide

  3. Paperclip
    CarrierWave ★7,253
    Refile ★2,314
    ★8,357
    Dragonfly ★ 1,982
    Attache ★184
    Paperdragon ★ 116

    View Slide


  4. Paperclip
    CarrierWave ★7,253
    Refile ★2,314
    ★8,357
    Dragonfly ★ 1,982
    Attache ★184
    Paperdragon ★ 116

    View Slide

  5. Flow

    View Slide

  6. Choose a file… picture.jpg Submit
    POST /photos
    Content-Type: multipart/form-data …
    […file content…]
    { name: "file",
    filename: "picture.jpg",
    type: "image/jpeg",
    tempfile: # }
    Rack

    View Slide

  7. Choose a file… picture.jpg Submit
    POST /albums
    Content-Type: multipart/form-data …
    […file content…]
    #@original_filename=“picture.jpg”,
    @content_type=“image/jpeg”,
    @tempfile=#>
    Rails

    View Slide

  8. Shrine.storages = {
    store: Shrine::Storage::FileSystem.new(
    "public", prefix: "uploads/store")
    }
    Shrine.plugin :sequel # or :activerecord
    class ImageUploader < Shrine
    # image attachment logic
    end
    class Photo < Sequel::Model
    include ImageUploader[:image]
    end

    View Slide

  9. photo = Photo.create(image: File.open("…"))
    photo.image_data #=>
    {
    "storage": "store",
    "id": "fd9sdf0ag32kf1s.jpg",
    "metadata": {
    "filename": "nature.jpg",
    "size": 883954,
    "mime_type": "image/jpeg"
    }
    }

    View Slide

  10. IMPROVEMENT A:
    Retaining file on form redisplays

    View Slide

  11. Choose a file… picture.jpg
    Submit
    Album title
    Choose a file… No file selected
    Submit
    Album title can’t be blank
    (Waiting for file to be uploaded…)

    View Slide

  12. Shrine.storages = {
    cache: Shrine::Storage::FileSystem.new(…),
    store: Shrine::Storage::FileSystem.new(…),
    }
    {"storage":"cache", "id":"8ag8fdg.jpg", "metadata":{...}}
    Choose a file… No file selected
    Submit
    Album title can’t be blank

    View Slide

  13. IMPROVEMENT B:
    Direct uploads

    View Slide

  14. Choose a file… picture.jpg
    Submit

    View Slide

  15. Shrine.plugin :direct_upload
    Rails.application.routes.draw do
    mount ImageUploader::UploadEndpoint,
    to: "/images"
    end
    POST /images/cache/upload
    jQuery-File-Upload
    Dropzone
    FineUploader

    View Slide

  16. Choose a file…
    Submit
    32%
    POST /images/cache/upload

    View Slide

  17. Choose a file…
    Submit
    100%
    {"storage":"cache", "id":"8ag8fdg.jpg", …}

    View Slide

  18. • User has a better idea how long upload will take
    • User can fill in other fields while file is uploading
    • Multiple files can be uploaded in parallel
    Benefits

    View Slide

  19. IMPROVEMENT C:
    External storage

    View Slide

  20. • Receiving file uploads uses server resources
    • Stored files are served through the app
    • Doesn’t work with multiple servers
    • Heroku doesn’t support storing on disk
    • Heroku has a 30-second request timeout
    Downsides of filesystem

    View Slide

  21. Shrine.storages = {
    cache: Shrine::Storage::S3.new(…),
    store: Shrine::Storage::S3.new(…),
    }
    Shrine.storages = {
    cache: Shrine::Storage::FileSystem.new(…),
    store: Shrine::Storage::FileSystem.new(…),
    }

    View Slide

  22. 1. GET /images/cache/presign
    {
    "url": "https://my-bucket.s3.amazonaws.com",
    "fields": {
    "key": "b7d575850ba61b44c8a9ff889dfdb14d88cdc25",
    "policy": "eyJleHBpcmF0aW9uIjoiMjAxNS0QwMToxMTo...",
    "x-amz-credential": "5TQ/eu-west-1/s3/aws4_request",
    "x-amz-algorithm": "AWS4-HMAC-SHA256",
    "x-amz-date": "20151024T001129Z",
    "x-amz-signature": "c1eb634f83f96b69bd675f535b3..."
    }
    }
    2. POST https://my-bucket.s3.amazonaws.com
    Content-Type: multipart/form-data
    […file content…]

    View Slide

  23. IMPROVEMENT D:
    Resumable uploads

    View Slide

  24. • Flaky connection terminates the upload
    • User needs to change location during upload
    • Computer freezes in the middle of upload
    Large files
    => User needs to retry the whole upload

    View Slide

  25. Chunking
    5MB 5MB 5MB 5MB 5MB …

    View Slide

  26. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5

    View Slide

  27. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5
    32%

    View Slide

  28. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5

    View Slide

  29. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5

    17%

    View Slide

  30. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5


    View Slide

  31. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5

    43%

    View Slide

  32. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5


    View Slide

  33. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5


    17%
    54%
    32%

    View Slide

  34. Chunking
    5MB
    5MB
    5MB
    5MB
    5MB
    1
    2
    3
    4
    5





    View Slide

  35. • S3 & Azure
    • Concurrent chunking
    • Numerous other features
    • Well-designed protocol
    • iOS & Android
    • Ruby implementation

    View Slide

  36. # config.ru
    require "tus/server"
    map("/files") { run Tus::Server }
    janko-m/tus-ruby-server
    Shrine.storages = {
    cache: Shrine::Storage::Url.new,
    store: Shrine::Storage::S3.new(…),
    }
    janko-m/shrine-url

    View Slide

  37. • Uploads can be resumed in case of interruption
    • Concurrent chunking can speed up the upload
    • Checksums can ensure that no bytes were lost
    Benefits

    View Slide

  38. IMPROVEMENT E:
    Backgrounding

    View Slide

  39. 2048×1365
    800×800
    500×500
    300×300

    View Slide

  40. previous_cover_photo = album.cover_photo
    album.update(cover_photo: File.open("..."))
    previous_cover_photo.exists? #=> false

    View Slide

  41. album.destroy
    album.cover_photo.exists? #=> false

    View Slide

  42. Deleting

    Processing
    (= downloading + processing + uploading)

    SLOW
    SLOW

    View Slide

  43. class PromoteJob
    include Sidekiq::Worker
    def perform(data)
    Shrine::Attacher.promote(data)
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote do |data|
    PromoteJob.perform_async(data)
    end
    mperham/sidekiq

    View Slide

  44. class PromoteJob
    include Sidekiq::Worker
    def perform(data)
    Shrine::Attacher.promote(data)
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote do |data|
    PromoteJob.perform_in(3, data)
    end
    mperham/sidekiq

    View Slide

  45. class PromoteJob
    include Sidekiq::Worker
    def perform(data)
    ActiveRecord::Base.with_connection do
    Shrine::Attacher.promote(data)
    end
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote do |data|
    PromoteJob.perform_async(data)
    end
    mperham/sidekiq

    View Slide

  46. class PromoteJob
    include SuckerPunch::Job
    def perform(data)
    Shrine::Attacher.promote(data)
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote do |data|
    PromoteJob.perform_async(data)
    end
    brandonhilkert/sucker_punch

    View Slide

  47. class PromoteJob < RocketJob::Job
    key :data, Hash
    def perform(data)
    Shrine::Attacher.promote(data)
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote do |data|
    PromoteJob.create!(data: data)
    end
    rocketjob/rocketjob

    View Slide

  48. class DeleteJob < RocketJob::Job
    key :data, Hash
    def perform(data)
    Shrine::Attacher.delete(data)
    end
    end
    Shrine.plugin :backgrounding
    Shrine::Attacher.promote { |data| … }
    Shrine::Attacher.delete { |data| … }
    rocketjob/rocketjob

    View Slide

  49. album.cover_photo #=> nil
    album.cover_photo = File.open("...")
    album.save # background job is kicked off
    album.cover_photo.storage_key #=> "cache"
    # … background job finishes
    album.reload
    album.cover_photo.storage_key #=> "store"

    View Slide

  50. album.destroy
    album.cover_photo.exists? #=> true
    # … background job finishes
    album.cover_photo.exists? #=> false

    View Slide

  51. Benefits
    • Fast for the user
    • Removes impact on request throughput
    • Can be scaled individually
    • Can automatically retry timeouts
    • Easier to manually retry failures that require action
    • Easier to monitor performance

    View Slide

  52. Demo

    View Slide

  53. Credits to Refile
    @jnicklas

    View Slide

  54. janko-m/shrine

    View Slide