Upgrade to Pro — share decks privately, control downloads, hide ads and more …

File Upload 2015

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →

File Upload 2015

Chew Choon Keat

October 16, 2015

Transcript

94 slides
  1. How’s that predefined styles doing for you? has_attached_file :asset, styles:

    { thumb: "100x100#", medium: "320x>", large: "1024x>" }
  2. How’s that predefined styles doing for you? has_attached_file :asset, styles:

    { thumb: "100x100#", medium: "320x>",
 big: "640x>", large: "1024x>" } Names starting to lose meaning…
  3. How’s that predefined styles doing for you? has_attached_file :asset, styles:

    { thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>",
 big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>" }
  4. How’s that predefined styles doing for you? has_attached_file :asset, styles:

    { thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>",
 big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>" } Reprocess all the production files, each time, we make changes. 404 while rake runs or do at midnight?
  5. How’s that transformation juggling doing for you? class MyUploader <

    CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] end end
  6. How’s that transformation juggling doing for you? class MyUploader <

    CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] end end Did your users wait in the foreground or background?
  7. How’s that file path config doing for you? class Avatar

    < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end
  8. How’s that file path config doing for you? class Avatar

    < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end You sure this is the format? Or will they need to change?
  9. How’s that file path config doing for you? class Avatar

    < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end
  10. How’s that file path config doing for you? class Avatar

    < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end
  11. How’s that file path config doing for you? CarrierWave.configure do

    |config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :file end
  12. How’s that file path config doing for you? CarrierWave.configure do

    |config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :file end Did you configure your MySQL data file in your app too?
  13. How’s that file path config doing for you? class Avatar

    < ActiveRecord::Base self.table = { name: "avatars", data: "/var/lib/mysql/data/avatars.MYD", index: "/var/lib/mysql/data/avatars.MYI" } end Did you configure your MySQL data file in your app too?
  14. How’s that form validation error dance doing for you? •

    User chooses a file • Submit & wait for file to upload ⌛… • Validation error: “Username is already taken!” • Re-render form
  15. How’s that form validation error dance doing for you? •

    User chooses a file • Submit & wait for file to upload ⌛… • Validation error: “Username is already taken!” • Re-render form Where dat file go?
  16. How’s that form validation error dance doing for you? “Is

    there a standard approach? This seems like a very common use case.”
  17. How’s the schema pollution doing for you? class StoreMetadata <

    ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string end end
  18. How’s the schema pollution doing for you? class StoreMetadata <

    ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string end end
  19. How’s that multiple files doing for you? class Post <

    ActiveRecord::Base has_many :images, dependent: :destroy end
  20. How’s that multiple files doing for you? class Post <

    ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end
  21. How’s that multiple files doing for you? class Post <

    ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end
  22. How’s that multiple files doing for you? class Post <

    ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end Is this what you want or just what you’re accustomed to?
  23. How’s Amazon Lambda doing for you? • User chooses a

    file • Submit & wait for file to upload ⌛… • Success! • Render page with thumbnail… How many thumbnails - 404?
  24. How’s Amazon Lambda doing for you? • User chooses a

    file • Submit & wait for file to upload ⌛… • Success! • Render page with thumbnail… Direct upload to AWS? Cancel form submit - delete files & thumbnails? Deep integration & assumption
  25. Take a step back • We want to store a

    bunch of attributes in a model • e.g. Title, Body, Tags, Photo
 
 
 
 
 
 
 

  26. Take a step back • Why should photo be a

    disproportionately complicated attribute in my Article model? • stored file path • conversion • background job • aws config • clean up on delete
  27. Take a step back • Why should photo be a

    disproportionately complicated attribute in my Article model? • stored file path • conversion • background job • validation error dance • aws config
  28. Take a step back • Frankly photo_url is best; least

    intrusive
 
 
 
 
 
 
 
 
 

  29. Take a step back • Frankly photo_url is best; least

    intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data
  30. Take a step back • Frankly photo_url is best; least

    intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data
  31. Take a step back • Frankly photo_url is best; least

    intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data
  32. Take a step back • Frankly photo_url is best; least

    intrusive • Solutions • Exclusive server for your app • Upload to that server • On-the-fly resize based on URL • Store url with meta data: photo_json instead?
  33. • PostgreSQL, MySQL • Redis • Memcached • SMTP server

    (Mail)
 
 
 
 You are already Generic server to do specialised work Not specific to your business logic
  34. • Not a new pattern • Mostly commercial services
 


    
 
 
 • Maybe it has to be Free & Open Source to become a default pattern Image server
  35. Want • Move the “concern” out of my app •

    photo is a regular attribute • configure my app & forget it exist
 
 
 

  36. Want • Move the “concern” out of my app •

    photo is a regular attribute • configure my app & forget it exist
 
 
 
 What would my app look like in a better world?
  37. My app: Bare minimum create_table "users" do |t| t.string "name"

    t.json "avatar" t.json "photos" # multiple files in a column end
  38. My app: Bare minimum Image server Rails app Browser {

    avatar: #<File..> } {“path”:“x.jpg”, “geometry”:“200x600”} #<File..>
  39. My app: Bare minimum Image server Rails app Browser {“path”:“x.jpg”,

    “geometry”:“200x600”} #<File..> user.avatar={“path”: “x.jpg”…} user.save <img src=“x.jpg”> { avatar: #<File..> }
  40. My app: Bare minimum Image server Rails app Browser {“path”:“x.jpg”,

    “geometry”:“200x600”} <img src=“x.jpg”> #<File..> GET x.jpg #<File..> user.avatar={“path”: “x.jpg”…} user.save { avatar: #<File..> }
  41. My app: Bare minimum • Browser render <file> field; regular

    form submit • Receive binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png
 
 

  42. { avatar: {“path”:“x.jpg”, … } Image server Rails app Browser

    #<File..> {“path”:“x.jpg”, “geometry”:“200x600”} user.avatar={“path”: “x.jpg”…} user.save <img src=“x.jpg”> Progressive Enhancement
  43. • Browser render <file> field; regular form submit • Receive

    binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png
 
 
 Progressive Enhancement
  44. • Browser render <file> field; regular form submit • Receive

    binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png
 
 
 • JS upload directly to attache server; “Direct upload” in AWS parlance • No binary in params; receive and store json attribute Progressive Enhancement • When after_update & after_destroy removes obsolete file from attache via delete API
  45. • Just use Ruby; just use your framework • Pre-render

    multiple sizes • fetch the urls, server will generate and cache • Validation • validating a regular json attribute How do I…
  46. • Move the “concern” out of my app • photo

    is a regular attribute • configure my app & forget it exist
 
 
 
 Want (cont’d)
  47. Want (cont’d) • Move the “concern” out of my app

    • photo is a regular attribute • configure my app & forget it exist • Separate, standalone server • Minimal / zero ongoing administration
  48. Want (cont’d) • Move the “concern” out of my app

    • photo is a regular attribute • configure my app & forget it exist • Separate, standalone server • Minimal / zero ongoing administration How does this server work?
  49. Attache File Server • HTTP server with simple APIs •

    upload • download • delete • Rack app + ImageMagick • Go? Node? C++? PHP? • GraphicsMagick? MyResizer.bash?
  50. • Uploaded files are stored locally • Resize local file

    on-the-fly • configurable pool size to limit concurrent resizing • Sync upload to cloud storage • 2 hop problem vs complexity • Fixed local storage, purge LRU (zero maintenance) • Spin up new fresh servers anytime… 
 Attache File Server
  51. • When requested file does not exist locally • fetch

    from cloud storage & write locally • resume operations
 
 
 
 
 
 Attache File Server
  52. • Remove obsolete file is “best effort” • If photo

    delete failed, do you Error 500 or stop the world?
 
 
 
 • OCDs can schedule rake task remove dangling files? Attache File Server
  53. • Caching in production • Browser → CDN → Varnish

    → Disk → Cloud
 
 
 
 
 
 
 Attache File Server
  54. tus.io • Open protocol for resumable uploads built on HTTP

    • Perfect for mobile apps • Rack middleware implemented in choonkeat/ attache#10
  55. Dragonfly & refile • Rack middleware in your Rails app

    by default • performing on-the-fly image resize • Rack standalone end point • Dragonfly.app - upload, download, delete • Refile::App - upload, download, delete • Downloads are unthrottled
  56. Dragonfly & refile • BEFORE: Rails integrate with AWS •

    AFTER: Rails integrate with AWS + Rack app • Maintain identical AWS config in both apps • Rails app couldn’t shed the “concern” • Multiple images still require multiple models
  57. refile class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for

    :images, attachment: :file end class Image < ActiveRecord::Base belongs_to :post attachment :file end
  58. refile class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for

    :images, attachment: :file end class Image < ActiveRecord::Base belongs_to :post attachment :file end “Note it must be possible to persist images given only the associated post and a file. There must not be any other validations or constraints which prevent images from being saved” i.e. Must be pure dummy wrapper; no validations here
  59. refile upload https://github.com/refile/refile/blob/master/lib/refile/app.rb post "/:backend" do halt 404 unless upload_allowed?

    tempfile = request.params.fetch("file").fetch(:tempfile) file = backend.upload(tempfile) content_type :json { id: file.id }.to_json end def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.download end
  60. post "/:backend" do halt 404 unless upload_allowed? tempfile = request.params.fetch("file").fetch(:tempfile)

    file = backend.upload(tempfile) content_type :json { id: file.id }.to_json end def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.download end refile upload https://github.com/refile/refile/blob/master/lib/refile/app.rb 2 hops problem when uploading and downloading • 3mb file becomes 6mb each way • #create and #show becomes 12mb process • has_many :images?
  61. refile • CarrierWave-styled named processors (e.g. :fill, :thumb) vs passing

    through syntax to underlying ImageMagick • personally prefer leveraging off existing knowledge • instead of manually configured syntax sugar • “2 hop problem” however provide higher consistency when running multiple refile servers • upload-here-download-there problem • (considering to perform 2-hop upload instead of async)
  62. refile • Can upload to S3 direct and/or upload to

    refile • Redundancy interesting, but prefer less concern in Rails app • Concept of cache and store to manage “dangling file problem” is worth considering • Download urls are signed-only (this practice should be adopted) • can partly counter motivation to abuse “dangling file problem” (aka free file hosting, whee!)

More Decks by Chew Choon Keat

Other Decks in Programming