Tweet
Tweet

Slide 1

Slide 1 text

Shrine Handle file uploads like it’s 2017 janko-m @jankomarohnic

Slide 2

Slide 2 text

Shrine Best practices for handling file uploads in Ruby janko-m @jankomarohnic

Slide 3

Slide 3 text

Paperclip CarrierWave ★7,253 Refile ★2,314 ★8,357 Dragonfly ★ 1,982 Attache ★184 Paperdragon ★ 116

Slide 4

Slide 4 text

Paperclip CarrierWave ★7,253 Refile ★2,314 ★8,357 Dragonfly ★ 1,982 Attache ★184 Paperdragon ★ 116

Slide 5

Slide 5 text

Flow

Slide 6

Slide 6 text

Choose a file… picture.jpg Submit POST /photos Content-Type: multipart/form-data … […file content…] { name: "file", filename: "picture.jpg", type: "image/jpeg", tempfile: # } Rack

Slide 7

Slide 7 text

Choose a file… picture.jpg Submit POST /albums Content-Type: multipart/form-data … […file content…] #> Rails

Slide 8

Slide 8 text

Shrine.storages = { store: Shrine::Storage::FileSystem.new( "public", prefix: "uploads/store") } Shrine.plugin :sequel # or :activerecord class ImageUploader < Shrine # image attachment logic end class Photo < Sequel::Model include ImageUploader[:image] end

Slide 9

Slide 9 text

photo = Photo.create(image: File.open("…")) photo.image_data #=> { "storage": "store", "id": "fd9sdf0ag32kf1s.jpg", "metadata": { "filename": "nature.jpg", "size": 883954, "mime_type": "image/jpeg" } }

Slide 10

Slide 10 text

IMPROVEMENT A: Retaining file on form redisplays

Slide 11

Slide 11 text

Choose a file… picture.jpg Submit Album title Choose a file… No file selected Submit Album title can’t be blank (Waiting for file to be uploaded…)

Slide 12

Slide 12 text

Shrine.storages = { cache: Shrine::Storage::FileSystem.new(…), store: Shrine::Storage::FileSystem.new(…), } {"storage":"cache", "id":"8ag8fdg.jpg", "metadata":{...}} Choose a file… No file selected Submit Album title can’t be blank

Slide 13

Slide 13 text

IMPROVEMENT B: Direct uploads

Slide 14

Slide 14 text

Choose a file… picture.jpg Submit

Slide 15

Slide 15 text

Shrine.plugin :direct_upload Rails.application.routes.draw do mount ImageUploader::UploadEndpoint, to: "/images" end POST /images/cache/upload jQuery-File-Upload Dropzone FineUploader …

Slide 16

Slide 16 text

Choose a file… Submit 32% POST /images/cache/upload

Slide 17

Slide 17 text

Choose a file… Submit 100% {"storage":"cache", "id":"8ag8fdg.jpg", …}

Slide 18

Slide 18 text

• User has a better idea how long upload will take • User can fill in other fields while file is uploading • Multiple files can be uploaded in parallel Benefits

Slide 19

Slide 19 text

IMPROVEMENT C: External storage

Slide 20

Slide 20 text

• Receiving file uploads uses server resources • Stored files are served through the app • Doesn’t work with multiple servers • Heroku doesn’t support storing on disk • Heroku has a 30-second request timeout Downsides of filesystem

Slide 21

Slide 21 text

Shrine.storages = { cache: Shrine::Storage::S3.new(…), store: Shrine::Storage::S3.new(…), } Shrine.storages = { cache: Shrine::Storage::FileSystem.new(…), store: Shrine::Storage::FileSystem.new(…), }

Slide 22

Slide 22 text

1. GET /images/cache/presign { "url": "https://my-bucket.s3.amazonaws.com", "fields": { "key": "b7d575850ba61b44c8a9ff889dfdb14d88cdc25", "policy": "eyJleHBpcmF0aW9uIjoiMjAxNS0QwMToxMTo...", "x-amz-credential": "5TQ/eu-west-1/s3/aws4_request", "x-amz-algorithm": "AWS4-HMAC-SHA256", "x-amz-date": "20151024T001129Z", "x-amz-signature": "c1eb634f83f96b69bd675f535b3..." } } 2. POST https://my-bucket.s3.amazonaws.com Content-Type: multipart/form-data […file content…]

Slide 23

Slide 23 text

IMPROVEMENT D: Resumable uploads

Slide 24

Slide 24 text

• Flaky connection terminates the upload • User needs to change location during upload • Computer freezes in the middle of upload Large files => User needs to retry the whole upload

Slide 25

Slide 25 text

Chunking 5MB 5MB 5MB 5MB 5MB …

Slide 26

Slide 26 text

Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5

Slide 27

Slide 27 text

Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 32%

Slide 28

Slide 28 text

Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅

Slide 29

Slide 29 text

Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ 17%

Slide 30

Slide 30 text

Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ ❌

Slide 31

Slide 31 text

Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ 43%

Slide 32

Slide 32 text

Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ ✅

Slide 33

Slide 33 text

Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ ✅ 17% 54% 32%

Slide 34

Slide 34 text

Chunking 5MB 5MB 5MB 5MB 5MB 1 2 3 4 5 ✅ ✅ ✅ ✅ ✅

Slide 35

Slide 35 text

• S3 & Azure • Concurrent chunking • Numerous other features • Well-designed protocol • iOS & Android • Ruby implementation

Slide 36

Slide 36 text

# config.ru require "tus/server" map("/files") { run Tus::Server } janko-m/tus-ruby-server Shrine.storages = { cache: Shrine::Storage::Url.new, store: Shrine::Storage::S3.new(…), } janko-m/shrine-url

Slide 37

Slide 37 text

• Uploads can be resumed in case of interruption • Concurrent chunking can speed up the upload • Checksums can ensure that no bytes were lost Benefits

Slide 38

Slide 38 text

IMPROVEMENT E: Backgrounding

Slide 39

Slide 39 text

2048×1365 800×800 500×500 300×300

Slide 40

Slide 40 text

previous_cover_photo = album.cover_photo album.update(cover_photo: File.open("...")) previous_cover_photo.exists? #=> false

Slide 41

Slide 41 text

album.destroy album.cover_photo.exists? #=> false

Slide 42

Slide 42 text

Deleting ↓ Processing (= downloading + processing + uploading) ↓ SLOW SLOW

Slide 43

Slide 43 text

class PromoteJob include Sidekiq::Worker def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end mperham/sidekiq

Slide 44

Slide 44 text

class PromoteJob include Sidekiq::Worker def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_in(3, data) end mperham/sidekiq

Slide 45

Slide 45 text

class PromoteJob include Sidekiq::Worker def perform(data) ActiveRecord::Base.with_connection do Shrine::Attacher.promote(data) end end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end mperham/sidekiq

Slide 46

Slide 46 text

class PromoteJob include SuckerPunch::Job def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.perform_async(data) end brandonhilkert/sucker_punch

Slide 47

Slide 47 text

class PromoteJob < RocketJob::Job key :data, Hash def perform(data) Shrine::Attacher.promote(data) end end Shrine.plugin :backgrounding Shrine::Attacher.promote do |data| PromoteJob.create!(data: data) end rocketjob/rocketjob

Slide 48

Slide 48 text

class DeleteJob < RocketJob::Job key :data, Hash def perform(data) Shrine::Attacher.delete(data) end end Shrine.plugin :backgrounding Shrine::Attacher.promote { |data| … } Shrine::Attacher.delete { |data| … } rocketjob/rocketjob

Slide 49

Slide 49 text

album.cover_photo #=> nil album.cover_photo = File.open("...") album.save # background job is kicked off album.cover_photo.storage_key #=> "cache" # … background job finishes album.reload album.cover_photo.storage_key #=> "store"

Slide 50

Slide 50 text

album.destroy album.cover_photo.exists? #=> true # … background job finishes album.cover_photo.exists? #=> false

Slide 51

Slide 51 text

Benefits • Fast for the user • Removes impact on request throughput • Can be scaled individually • Can automatically retry timeouts • Easier to manually retry failures that require action • Easier to monitor performance

Slide 52

Slide 52 text

Demo

Slide 53

Slide 53 text

Credits to Refile @jnicklas

Slide 54

Slide 54 text

janko-m/shrine