Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Handling File Uploads for a modern developer

Handling File Uploads for a modern developer

In Ruby we have an abundance of options for handling file attachments. Unfortunately, most file attachment gems fall short when it comes to tailoring the attachment flow to your needs and preferences, often leaving your app with suboptimal user experience.

This talk will introduce the Shrine gem and go over some of the modern best practices for handling file attachments. We will cover topics such as image processing (“Is ImageMagick always the best option?”), asynchronous file uploads (“Which JavaScript solution should I choose?”), and handling large uploads (“How do I help users that have a flaky internet connection?”).

Janko Marohnić

September 07, 2019
Tweet

More Decks by Janko Marohnić

Other Decks in Programming

Transcript

  1. Handling File Uploads
    For a modern developer

    View Slide

  2. 2007
    Paperclip 2008
    CarrierWave
    2014
    Refile
    2018
    ActiveStorage
    2009
    Dragonfly
    2015
    ! Shrine

    View Slide

  3. Janko Marohnic
    • from Croatia ", living in Czechia #
    • Ruby off Rails developer
    • creator of Shrine
    @janko
    @jankomarohnic

    View Slide

  4. Active Storage?

    View Slide

  5. Active Storage Shrine
    Rails & Active Record only
    Rails, Hanami, Sinatra, Roda...
    Active Record, Sequel, ROM...
    framework library
    integrated experience more control
    more opinionated more features

    View Slide

  6. Metadata

    &

    Validation

    View Slide

  7. class PhotosController < AC::Base
    def create
    photo = Photo.new(photo_params)
    if photo.valid?
    photo.save
    # ...
    else
    # ...
    end
    end
    end

    View Slide

  8. class ImageUploader < Shrine
    TYPES = %w[image/jpeg image/png ...]
    EXTENSIONS = %w[jpg jpeg png ...]
    Attacher.validate do
    validate_max_size 10*1024*1024
    validate_mime_type TYPES
    validate_extension EXTENSIONS
    end
    end

    View Slide

  9. \377\330\377························· (JPEG)
    \211PNG······························ (PNG)
    ········WEBP························· (WebP)
    MIME Type
    `file --mime-type -b #{file.path}`
    #=> "image/jpeg"
    MimeMagic.by_magic(file).type
    #=> "image/jpeg"
    Marcel::MimeType.for(file)
    #=> "image/jpeg"

    View Slide

  10. class PhotosController < AC::Base
    def create
    file = params[:photo][:image]
    file.read #=> ""
    file.filename #=> "nature.jpg"
    file.content_type #=> "image/jpeg"
    photo = Photo.new(image: file)
    photo.image.mime_type #=> "text/x-php"
    photo.valid? #=> false
    end
    end
    MIME Type

    View Slide

  11. https://www.bamsoftware.com/hacks/deflate.html
    $ identify spark.png
    ... 225000x225000 ... 6.1MB

    Attacher.validate do
    # ...
    if validate_mime_type IMAGE_TYPES
    validate_max_dimensions [5000, 5000]
    end
    end
    Image Dimensions

    View Slide

  12. class VideoUploader < Shrine
    add_metadata :duration do |file|
    FFMPEG::Movie.new(file.path).duration
    end
    Attacher.validate do
    if file.duration > 5*60*60
    errors << "is too long (max is 5h)"
    end
    end
    end
    Custom Metadata

    View Slide

  13. {
    "id": "b93777246f10e509a553.mp4",
    "storage": "store",
    "metadata": {
    "size": 234837,
    "filename": "matrix.mp4",
    "mime_type": "video/mp4",
    "duration": 9000,
    ...
    }
    }
    SELECT video_data FROM movies;

    View Slide

  14. Processing

    View Slide

  15. ImageMagick

    View Slide

  16. image = MiniMagick::Image.open(file)
    image.resize("500x500")
    image.path #=> "/path/to/resized.jpg"
    $ gem install mini_magick

    View Slide

  17. Resize to fit
    Resize to limit
    Resize to fill

    View Slide

  18. magick = ImageProcessing::MiniMagick

    magick = magick.source(image)
    magick.resize_to_limit! 500, 500
    magick.resize_to_fit! 500, 500
    magick.resize_to_fill! 500, 500
    magick.resize_and_pad! 500, 500
    $ gem install image_processing

    View Slide

  19. thumbnail = ImageProcessing::MiniMagick

    .source(image)
    .resize_to_limit!(500, 500)
    $ gem install image_processing
    $ convert source.jpg -auto-orient -resize
    500x500> -sharpen 0x1 destination.jpg

    View Slide

  20. thumbnail = ImageProcessing::MiniMagick

    .source(image)
    .resize_to_limit!(500, 500)
    $ gem install image_processing
    $ convert source.jpg -auto-orient -resize
    500x500> -sharpen 0x1 destination.jpg

    View Slide

  21. libvips⚡

    View Slide

  22. ImageProcessing::MiniMagick

    .source(image)
    .resize_to_limit!(500, 500)
    $ brew install imagemagick

    View Slide

  23. ImageProcessing::Vips

    .source(image)
    .resize_to_limit!(500, 500)
    $ brew install vips

    View Slide

  24. THUMBNAILS = {
    xs: [200, 200],
    s: [400, 400],
    m: [600, 600],
    l: [800, 800],
    xl: [1200, 1200],
    }
    THUMBNAILS.each do |name, (width, height)|
    ImageProcessing::Backend
    .source(original)
    .resize_to_limit!(width, height)
    end

    View Slide

  25. 0,47s
    2,15s
    ImageMagick libvips

    View Slide

  26. Rails 6.0

    View Slide

  27. Processing

    On-the-fly

    View Slide

  28. Active Storage
    https://...//...
    photo.image.variant(
    resize_to_limit: [600, 400],
    strip: true,
    quality: 85,
    interlace: "JPEG",
    )

    View Slide

  29. Shrine
    https://.../thumbnail/600/400/...
    photo.image.derivation_url(
    :thumbnail, # <-- name
    600, 400 # <-- args
    )
    Shrine.derivation :thumbnail do |file, w, h|
    ImageProcessing::Vips.source(file)
    .strip
    .saver(quality: 85, interlace: "JPEG")
    .resize_to_limit!(w.to_i, h.to_i)
    end

    View Slide

  30. Processing

    on attachment

    View Slide

  31. class ImageUploader < Shrine
    derivatives_processor :thumbs do |file|
    vips = ImageProcessing::Vips.source(file)
    { large: vips.resize_to_fit!(800, 800),
    medium: vips.resize_to_fit!(500, 500),
    small: vips.resize_to_fit!(300, 300) }
    end
    end
    photo = Photo.new(image: file)
    photo.image_derivatives!(:thumbs)
    photo.image_derivatives #=>
    # { large: #,
    # medium: #,
    # small: # }

    View Slide

  32. class VideoUploader < Shrine
    derivatives_processor :transcode do |file|
    movie = FMMPEG::Movie.new(file.path)
    transcoded = Tempfile.new(["", ".mp4"])
    screenshot = Tempfile.new(["", ".jpg"])
    movie.transcode(transcoded.path)
    movie.screenshot(screenshot.path)
    { transcoded: transcoded,
    screenshot: screenshot }
    end
    end

    View Slide

  33. Direct
    Uploads

    View Slide

  34. refile.js
    activestorage.js
    shrine.js ?

    View Slide

  35. jQuery-File-Upload
    Dropzone.js
    FineUploader



    View Slide

  36. Uppy

    View Slide

  37. Uppy()
    .use(Dashboard, { /* ... */ })

    View Slide

  38. Uppy()
    .use(DragDrop, { /* ... */ })
    .use(StatusBar, { /* ... */ })

    View Slide

  39. Uppy()
    .use(FileInput, { /* ... */ })
    .use(ProgressBar, { /* ... */ })

    View Slide

  40. { "id": "xyz.jpg", ... }
    submit
    { "id": "xyz.jpg", ... }

    POST /upload
    App S3

    View Slide

  41. mount Shrine.upload_endpoint(:cache),
    to: "/upload"
    POST /upload
    uppy.use(Uppy.XHRUpload, {
    endpoint: '/upload'
    })

    View Slide

  42. { "url": "...", "fields": "..." }
    submit
    { "id": "xyz.jpg", ... }
    App S3
    GET /s3/params

    POST s3.aws.com

    View Slide

  43. mount Shrine.presign_endpoint(:cache),
    to: "/s3/params"
    GET /s3/params
    uppy.use(Uppy.AwsS3, {
    companionUrl: '/'
    })

    View Slide

  44. Resumable
    Uploads

    View Slide

  45. 0%

    80%
    80%

    View Slide

  46. 0% 0% 0% 0% 0%

    View Slide

  47. 0%
    0%
    0%
    0%
    0%

    View Slide

  48. 100%
    80%


    View Slide

  49. App
    S3

    S3 Multipart Upload

    View Slide

  50. $ gem install uppy-s3_multipart
    uppy.use(Uppy.AwsS3Multipart, {
    companionUrl: '/'
    })
    mount Shrine.uppy_s3_multipart(:cache),
    to: "/s3/multipart"

    View Slide

  51. tus.io
    • HTTP protocol for resumable uploads
    • Numerous implementations:
    • Client – JavaScript, iOS, Android, ...
    • Server – Ruby, Node, Go, Python, Java, PHP, ...

    View Slide

  52. S3

    tus protocol storage API
    tus server
    tus client
    App

    View Slide

  53. $ gem install tus-ruby-server
    uppy.use(Uppy.Tus, {
    endpoint: '/files'
    })
    mount Tus::Server, to: "/files"

    View Slide

  54. Recap
    • Validate and persist metadata
    • Processing "on attachment" or "on-the-fly"
    ImageProcessing – ImageMagick & libvips
    • Direct uploads
    Uppy
    simple uploads (to your app or to cloud)
    resumable uploads (S3 multipart upload or tus.io)

    View Slide

  55. • https://shrinerb.com/
    • https://github.com/janko/image_processing
    • https://libvips.github.io/libvips/
    • https://uppy.io/
    • https://tus.io/
    • https://github.com/janko/tus-ruby-server
    • https://github.com/janko/uppy-s3_multipart
    • https://twin.github.io

    View Slide