Share
Embed
File Upload 2015
Chew Choon Keat
October 16, 2015
Transcript
94 slides
-
How’s that predefined styles doing for you? has_attached_file :asset, styles:
{ thumb: "100x100#" } -
How’s that predefined styles doing for you? has_attached_file :asset, styles:
{ thumb: "100x100#", medium: "320x>" } -
How’s that predefined styles doing for you? has_attached_file :asset, styles:
{ thumb: "100x100#", medium: "320x>", large: "1024x>" } -
How’s that predefined styles doing for you? has_attached_file :asset, styles:
{ thumb: "100x100#", medium: "320x>", big: "640x>", large: "1024x>" } Names starting to lose meaning… -
How’s that predefined styles doing for you? has_attached_file :asset, styles:
{ thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>", big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>" } -
How’s that predefined styles doing for you? has_attached_file :asset, styles:
{ thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>", big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>" } Reprocess all the production files, each time, we make changes. 404 while rake runs or do at midnight? -
How’s that transformation juggling doing for you? class MyUploader <
CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] end end -
How’s that transformation juggling doing for you? class MyUploader <
CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] end end Did your users wait in the foreground or background? -
How’s that file path config doing for you? class Avatar
< ActiveRecord::Base has_attached_file :image, url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end -
How’s that file path config doing for you? class Avatar
< ActiveRecord::Base has_attached_file :image, url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end You sure this is the format? Or will they need to change? -
How’s that file path config doing for you? class Avatar
< ActiveRecord::Base has_attached_file :image, url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end -
How’s that file path config doing for you? class Avatar
< ActiveRecord::Base has_attached_file :image, url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end -
How’s that file path config doing for you? CarrierWave.configure do
|config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :file end -
How’s that file path config doing for you? CarrierWave.configure do
|config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :file end Did you configure your MySQL data file in your app too? -
How’s that file path config doing for you? class Avatar
< ActiveRecord::Base self.table = { name: "avatars", data: "/var/lib/mysql/data/avatars.MYD", index: "/var/lib/mysql/data/avatars.MYI" } end Did you configure your MySQL data file in your app too? -
How’s that form validation error dance doing for you? •
User chooses a file • Submit & wait for file to upload ⌛… • Validation error: “Username is already taken!” • Re-render form -
How’s that form validation error dance doing for you? •
User chooses a file • Submit & wait for file to upload ⌛… • Validation error: “Username is already taken!” • Re-render form Where dat file go? -
How’s that form validation error dance doing for you? “Is
there a standard approach? This seems like a very common use case.” -
How’s the schema pollution doing for you? class StoreMetadata <
ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string end end -
How’s the schema pollution doing for you? class StoreMetadata <
ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string end end -
How’s that multiple files doing for you? class Post <
ActiveRecord::Base has_many :images, dependent: :destroy end -
How’s that multiple files doing for you? class Post <
ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end -
How’s that multiple files doing for you? class Post <
ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end -
How’s that multiple files doing for you? class Post <
ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end Is this what you want or just what you’re accustomed to? -
How’s Amazon Lambda doing for you? • User chooses a
file • Submit & wait for file to upload ⌛… • Success! • Render page with thumbnail… How many thumbnails - 404? -
How’s Amazon Lambda doing for you? • User chooses a
file • Submit & wait for file to upload ⌛… • Success! • Render page with thumbnail… Direct upload to AWS? Cancel form submit - delete files & thumbnails? Deep integration & assumption -
Take a step back • We want to store a
bunch of attributes in a model • e.g. Title, Body, Tags, Photo -
Take a step back • Why should photo be a
disproportionately complicated attribute in my Article model? • stored file path • conversion • background job • aws config • clean up on delete -
Take a step back • Why should photo be a
disproportionately complicated attribute in my Article model? • stored file path • conversion • background job • validation error dance • aws config -
Take a step back • Frankly photo_url is best; least
intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data -
Take a step back • Frankly photo_url is best; least
intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data -
Take a step back • Frankly photo_url is best; least
intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data -
Take a step back • Frankly photo_url is best; least
intrusive • Solutions • Exclusive server for your app • Upload to that server • On-the-fly resize based on URL • Store url with meta data: photo_json instead? -
• PostgreSQL, MySQL • Redis • Memcached • SMTP server
(Mail) You are already Generic server to do specialised work Not specific to your business logic -
• Not a new pattern • Mostly commercial services
• Maybe it has to be Free & Open Source to become a default pattern Image server -
Want • Move the “concern” out of my app •
photo is a regular attribute • configure my app & forget it exist -
Want • Move the “concern” out of my app •
photo is a regular attribute • configure my app & forget it exist What would my app look like in a better world? -
My app: Bare minimum create_table "users" do |t| t.string "name"
t.json "avatar" t.json "photos" end -
My app: Bare minimum create_table "users" do |t| t.string "name"
t.json "avatar" t.json "photos" # multiple files in a column end -
My app: Bare minimum Image server Rails app Browser {
avatar: #<File..> } -
My app: Bare minimum Image server Rails app Browser {
avatar: #<File..> } {“path”:“x.jpg”, “geometry”:“200x600”} #<File..> -
My app: Bare minimum Image server Rails app Browser {“path”:“x.jpg”,
“geometry”:“200x600”} #<File..> user.avatar={“path”: “x.jpg”…} user.save <img src=“x.jpg”> { avatar: #<File..> } -
My app: Bare minimum Image server Rails app Browser {“path”:“x.jpg”,
“geometry”:“200x600”} <img src=“x.jpg”> #<File..> GET x.jpg #<File..> user.avatar={“path”: “x.jpg”…} user.save { avatar: #<File..> } -
My app: Bare minimum • Browser render <file> field; regular
form submit • Receive binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png -
Image server Rails app Browser #<File..> {“path”:“x.jpg”, “geometry”:“200x600”} Progressive Enhancement
{ avatar: {“path”:“x.jpg”, … } -
{ avatar: {“path”:“x.jpg”, … } Image server Rails app Browser
#<File..> {“path”:“x.jpg”, “geometry”:“200x600”} user.avatar={“path”: “x.jpg”…} user.save <img src=“x.jpg”> Progressive Enhancement -
• Browser render <file> field; regular form submit • Receive
binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png Progressive Enhancement -
• Browser render <file> field; regular form submit • Receive
binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png • JS upload directly to attache server; “Direct upload” in AWS parlance • No binary in params; receive and store json attribute Progressive Enhancement • When after_update & after_destroy removes obsolete file from attache via delete API -
• Just use Ruby; just use your framework • Pre-render
multiple sizes • fetch the urls, server will generate and cache • Validation • validating a regular json attribute How do I… -
• Move the “concern” out of my app • photo
is a regular attribute • configure my app & forget it exist Want (cont’d) -
Want (cont’d) • Move the “concern” out of my app
• photo is a regular attribute • configure my app & forget it exist • Separate, standalone server • Minimal / zero ongoing administration -
Want (cont’d) • Move the “concern” out of my app
• photo is a regular attribute • configure my app & forget it exist • Separate, standalone server • Minimal / zero ongoing administration How does this server work? -
Attache File Server • HTTP server with simple APIs •
upload • download • delete • Rack app + ImageMagick • Go? Node? C++? PHP? • GraphicsMagick? MyResizer.bash? -
• Uploaded files are stored locally • Resize local file
on-the-fly • configurable pool size to limit concurrent resizing • Sync upload to cloud storage • 2 hop problem vs complexity • Fixed local storage, purge LRU (zero maintenance) • Spin up new fresh servers anytime… Attache File Server -
• When requested file does not exist locally • fetch
from cloud storage & write locally • resume operations Attache File Server -
• Remove obsolete file is “best effort” • If photo
delete failed, do you Error 500 or stop the world? • OCDs can schedule rake task remove dangling files? Attache File Server -
• Caching in production • Browser → CDN → Varnish
→ Disk → Cloud Attache File Server -
tus.io • Open protocol for resumable uploads built on HTTP
• Perfect for mobile apps • Rack middleware implemented in choonkeat/ attache#10 -
Dragonfly & refile • tldr: we can fuss over implementation,
but it is mostly about architecture -
Dragonfly & refile • Rack middleware in your Rails app
by default • performing on-the-fly image resize • Rack standalone end point • Dragonfly.app - upload, download, delete • Refile::App - upload, download, delete • Downloads are unthrottled -
Dragonfly & refile • BEFORE: Rails integrate with AWS •
AFTER: Rails integrate with AWS + Rack app • Maintain identical AWS config in both apps • Rails app couldn’t shed the “concern” • Multiple images still require multiple models -
refile class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for
:images, attachment: :file end class Image < ActiveRecord::Base belongs_to :post attachment :file end -
refile class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for
:images, attachment: :file end class Image < ActiveRecord::Base belongs_to :post attachment :file end “Note it must be possible to persist images given only the associated post and a file. There must not be any other validations or constraints which prevent images from being saved” i.e. Must be pure dummy wrapper; no validations here -
refile download https://github.com/refile/refile/blob/master/lib/refile/app.rb get "/:token/:backend/:processor/:id/:filename" do halt 404 unless download_allowed?
stream_file processor.call(file) end How many ImageMagick can you run in parallel? has_many :images? -
refile upload https://github.com/refile/refile/blob/master/lib/refile/app.rb post "/:backend" do halt 404 unless upload_allowed?
tempfile = request.params.fetch("file").fetch(:tempfile) file = backend.upload(tempfile) content_type :json { id: file.id }.to_json end def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.download end -
post "/:backend" do halt 404 unless upload_allowed? tempfile = request.params.fetch("file").fetch(:tempfile)
file = backend.upload(tempfile) content_type :json { id: file.id }.to_json end def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.download end refile upload https://github.com/refile/refile/blob/master/lib/refile/app.rb 2 hops problem when uploading and downloading • 3mb file becomes 6mb each way • #create and #show becomes 12mb process • has_many :images? -
refile • CarrierWave-styled named processors (e.g. :fill, :thumb) vs passing
through syntax to underlying ImageMagick • personally prefer leveraging off existing knowledge • instead of manually configured syntax sugar • “2 hop problem” however provide higher consistency when running multiple refile servers • upload-here-download-there problem • (considering to perform 2-hop upload instead of async) -
refile • Can upload to S3 direct and/or upload to
refile • Redundancy interesting, but prefer less concern in Rails app • Concept of cache and store to manage “dangling file problem” is worth considering • Download urls are signed-only (this practice should be adopted) • can partly counter motivation to abuse “dangling file problem” (aka free file hosting, whee!)