Handling File Uploads for a modern developer

Handling File Uploads for a modern developer

In Ruby we have an abundance of options for handling file attachments. Unfortunately, most file attachment gems fall short when it comes to tailoring the attachment flow to your needs and preferences, often leaving your app with suboptimal user experience.

This talk will introduce the Shrine gem and go over some of the modern best practices for handling file attachments. We will cover topics such as image processing (“Is ImageMagick always the best option?”), asynchronous file uploads (“Which JavaScript solution should I choose?”), and handling large uploads (“How do I help users that have a flaky internet connection?”).

376e4eb9dc6c2e33d1330262edc4f109?s=128

Janko Marohnić

September 07, 2019
Tweet

Transcript

  1. Handling File Uploads For a modern developer

  2. 2007 Paperclip 2008 CarrierWave 2014 Refile 2018 ActiveStorage 2009 Dragonfly

    2015 ! Shrine
  3. Janko Marohnic • from Croatia ", living in Czechia #

    • Ruby off Rails developer • creator of Shrine @janko @jankomarohnic
  4. Active Storage?

  5. Active Storage Shrine Rails & Active Record only Rails, Hanami,

    Sinatra, Roda... Active Record, Sequel, ROM... framework library integrated experience more control more opinionated more features
  6. Metadata & Validation

  7. class PhotosController < AC::Base def create photo = Photo.new(photo_params) if

    photo.valid? photo.save # ... else # ... end end end
  8. class ImageUploader < Shrine TYPES = %w[image/jpeg image/png ...] EXTENSIONS

    = %w[jpg jpeg png ...] Attacher.validate do validate_max_size 10*1024*1024 validate_mime_type TYPES validate_extension EXTENSIONS end end
  9. \377\330\377························· (JPEG) \211PNG······························ (PNG) ········WEBP························· (WebP) MIME Type `file --mime-type

    -b #{file.path}` #=> "image/jpeg" MimeMagic.by_magic(file).type #=> "image/jpeg" Marcel::MimeType.for(file) #=> "image/jpeg"
  10. class PhotosController < AC::Base def create file = params[:photo][:image] file.read

    #=> "<?php ... ?>" file.filename #=> "nature.jpg" file.content_type #=> "image/jpeg" photo = Photo.new(image: file) photo.image.mime_type #=> "text/x-php" photo.valid? #=> false end end MIME Type
  11. https://www.bamsoftware.com/hacks/deflate.html $ identify spark.png ... 225000x225000 ... 6.1MB Attacher.validate do

    # ... if validate_mime_type IMAGE_TYPES validate_max_dimensions [5000, 5000] end end Image Dimensions
  12. class VideoUploader < Shrine add_metadata :duration do |file| FFMPEG::Movie.new(file.path).duration end

    Attacher.validate do if file.duration > 5*60*60 errors << "is too long (max is 5h)" end end end Custom Metadata
  13. { "id": "b93777246f10e509a553.mp4", "storage": "store", "metadata": { "size": 234837, "filename":

    "matrix.mp4", "mime_type": "video/mp4", "duration": 9000, ... } } SELECT video_data FROM movies;
  14. Processing

  15. ImageMagick

  16. image = MiniMagick::Image.open(file) image.resize("500x500") image.path #=> "/path/to/resized.jpg" $ gem install

    mini_magick
  17. Resize to fit Resize to limit Resize to fill

  18. magick = ImageProcessing::MiniMagick
 magick = magick.source(image) magick.resize_to_limit! 500, 500 magick.resize_to_fit!

    500, 500 magick.resize_to_fill! 500, 500 magick.resize_and_pad! 500, 500 $ gem install image_processing
  19. thumbnail = ImageProcessing::MiniMagick
 .source(image) .resize_to_limit!(500, 500) $ gem install image_processing

    $ convert source.jpg -auto-orient -resize 500x500> -sharpen 0x1 destination.jpg
  20. thumbnail = ImageProcessing::MiniMagick
 .source(image) .resize_to_limit!(500, 500) $ gem install image_processing

    $ convert source.jpg -auto-orient -resize 500x500> -sharpen 0x1 destination.jpg
  21. libvips⚡

  22. ImageProcessing::MiniMagick
 .source(image) .resize_to_limit!(500, 500) $ brew install imagemagick

  23. ImageProcessing::Vips
 .source(image) .resize_to_limit!(500, 500) $ brew install vips

  24. THUMBNAILS = { xs: [200, 200], s: [400, 400], m:

    [600, 600], l: [800, 800], xl: [1200, 1200], } THUMBNAILS.each do |name, (width, height)| ImageProcessing::Backend .source(original) .resize_to_limit!(width, height) end
  25. 0,47s 2,15s ImageMagick libvips

  26. Rails 6.0

  27. Processing On-the-fly

  28. Active Storage https://.../<base64-encoded-steps>/... photo.image.variant( resize_to_limit: [600, 400], strip: true, quality:

    85, interlace: "JPEG", )
  29. Shrine https://.../thumbnail/600/400/... photo.image.derivation_url( :thumbnail, # <-- name 600, 400 #

    <-- args ) Shrine.derivation :thumbnail do |file, w, h| ImageProcessing::Vips.source(file) .strip .saver(quality: 85, interlace: "JPEG") .resize_to_limit!(w.to_i, h.to_i) end
  30. Processing on attachment

  31. class ImageUploader < Shrine derivatives_processor :thumbs do |file| vips =

    ImageProcessing::Vips.source(file) { large: vips.resize_to_fit!(800, 800), medium: vips.resize_to_fit!(500, 500), small: vips.resize_to_fit!(300, 300) } end end photo = Photo.new(image: file) photo.image_derivatives!(:thumbs) photo.image_derivatives #=> # { large: #<Shrine::File @id="ld7.jpg">, # medium: #<Shrine::File @id="jg4.jpg">, # small: #<Shrine::File @id="02k.jpg"> }
  32. class VideoUploader < Shrine derivatives_processor :transcode do |file| movie =

    FMMPEG::Movie.new(file.path) transcoded = Tempfile.new(["", ".mp4"]) screenshot = Tempfile.new(["", ".jpg"]) movie.transcode(transcoded.path) movie.screenshot(screenshot.path) { transcoded: transcoded, screenshot: screenshot } end end
  33. Direct Uploads

  34. refile.js activestorage.js shrine.js ? ❌

  35. jQuery-File-Upload Dropzone.js FineUploader ❌ ❌ ❌

  36. Uppy

  37. Uppy() .use(Dashboard, { /* ... */ })

  38. Uppy() .use(DragDrop, { /* ... */ }) .use(StatusBar, { /*

    ... */ })
  39. Uppy() .use(FileInput, { /* ... */ }) .use(ProgressBar, { /*

    ... */ })
  40. { "id": "xyz.jpg", ... } submit { "id": "xyz.jpg", ...

    } POST /upload App S3
  41. mount Shrine.upload_endpoint(:cache), to: "/upload" POST /upload uppy.use(Uppy.XHRUpload, { endpoint: '/upload'

    })
  42. { "url": "...", "fields": "..." } submit { "id": "xyz.jpg",

    ... } App S3 GET /s3/params POST s3.aws.com
  43. mount Shrine.presign_endpoint(:cache), to: "/s3/params" GET /s3/params uppy.use(Uppy.AwsS3, { companionUrl: '/'

    })
  44. Resumable Uploads

  45. 0% ❌ 80% 80%

  46. 0% 0% 0% 0% 0%

  47. 0% 0% 0% 0% 0%

  48. 100% 80% ✅ ❌

  49. App S3 S3 Multipart Upload

  50. $ gem install uppy-s3_multipart uppy.use(Uppy.AwsS3Multipart, { companionUrl: '/' }) mount

    Shrine.uppy_s3_multipart(:cache), to: "/s3/multipart"
  51. tus.io • HTTP protocol for resumable uploads • Numerous implementations:

    • Client – JavaScript, iOS, Android, ... • Server – Ruby, Node, Go, Python, Java, PHP, ...
  52. S3 tus protocol storage API tus server tus client App

  53. $ gem install tus-ruby-server uppy.use(Uppy.Tus, { endpoint: '/files' }) mount

    Tus::Server, to: "/files"
  54. Recap • Validate and persist metadata • Processing "on attachment"

    or "on-the-fly" ImageProcessing – ImageMagick & libvips • Direct uploads Uppy simple uploads (to your app or to cloud) resumable uploads (S3 multipart upload or tus.io)
  55. • https://shrinerb.com/ • https://github.com/janko/image_processing • https://libvips.github.io/libvips/ • https://uppy.io/ • https://tus.io/

    • https://github.com/janko/tus-ruby-server • https://github.com/janko/uppy-s3_multipart • https://twin.github.io