File Upload 2015

Transcript

1. File Upload 2015 @choonkeat
 choonkeat.com jollygoodcode.com
2. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#" }
3. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>" }
4. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>", large: "1024x>" }
5. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#", medium: "320x>",
 big: "640x>", large: "1024x>" } Names starting to lose meaning…
6. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>",
 big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>" }
7. How’s that predefined styles doing for you? has_attached_file :asset, styles: { thumb: "100x100#", thumb2x: "200x200#", medium: "320x>", medium2x: "640x>",
 big: "640x>", big2x: "1280x>", large: "1024x>", large2x: "2048x>" } Reprocess all the production files, each time, we make changes. 404 while rake runs or do at midnight?
8. How’s that transformation juggling doing for you?
9. How’s that transformation juggling doing for you? class MyUploader < CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] end end
10. How’s that transformation juggling doing for you? class MyUploader < CarrierWave::Uploader::Base version :thumb do process resize_to_fill: [280, 280] end version :small_thumb, from_version: :thumb do process resize_to_fill: [20, 20] end end Did your users wait in the foreground or background?
11. How’s that file path config doing for you?
12. How’s that file path config doing for you? class Avatar < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end
13. How’s that file path config doing for you? class Avatar < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end You sure this is the format? Or will they need to change?
14. How’s that file path config doing for you? class Avatar < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end
15. How’s that file path config doing for you? class Avatar < ActiveRecord::Base has_attached_file :image,
 url: '/system/:class/:attachment/:id/:hash-:style.:extension', hash_secret: Rails.application.secrets.paperclip end
16. How’s that file path config doing for you? CarrierWave.configure do |config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :file end
17. How’s that file path config doing for you? CarrierWave.configure do |config| config.permissions = 0666 config.directory_permissions = 0777 config.storage = :file end Did you configure your MySQL data file in your app too?
18. How’s that file path config doing for you? class Avatar < ActiveRecord::Base self.table = { name: "avatars", data: "/var/lib/mysql/data/avatars.MYD", index: "/var/lib/mysql/data/avatars.MYI" } end Did you configure your MySQL data file in your app too?
19. How’s that form validation error dance doing for you?
20. How’s that form validation error dance doing for you? • User chooses a file • Submit & wait for file to upload ⌛… • Validation error: “Username is already taken!” • Re-render form
21. How’s that form validation error dance doing for you? • User chooses a file • Submit & wait for file to upload ⌛… • Validation error: “Username is already taken!” • Re-render form Where dat file go?
22. How’s that form validation error dance doing for you? http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err Closed: Won’t Fix
23. How’s that form validation error dance doing for you? http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err
24. How’s that form validation error dance doing for you? http://stackoverflow.com/questions/5198602/not-losing-paperclip-attachment-when-model-cannot-be-saved-due-to-validation-err Answer: use CarrierWave
25. How’s that form validation error dance doing for you?
26. How’s that form validation error dance doing for you? “Is there a standard approach? This seems like a very common use case.”
27. How’s the schema pollution doing for you?
28. How’s the schema pollution doing for you? class StoreMetadata < ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string end end
29. How’s the schema pollution doing for you? class StoreMetadata < ActiveRecord::Migration def change add_column :users, :profile_image_filename, :string add_column :users, :profile_image_size, :integer add_column :users, :profile_image_content_type, :string end end
30. How’s that multiple files doing for you?
31. How’s that multiple files doing for you? class Post < ActiveRecord::Base has_many :images, dependent: :destroy end
32. How’s that multiple files doing for you? class Post < ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end
33. How’s that multiple files doing for you? class Post < ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end
34. How’s that multiple files doing for you? class Post < ActiveRecord::Base has_many :images, dependent: :destroy end class Image < ActiveRecord::Base belongs_to :post attachment :file end Is this what you want or just what you’re accustomed to?
35. How’s Amazon Lambda doing for you? • User chooses a file • Submit & wait for file to upload ⌛… • Success! • Render page with thumbnail… How many thumbnails - 404?
36. How’s Amazon Lambda doing for you? • User chooses a file • Submit & wait for file to upload ⌛… • Success! • Render page with thumbnail… Direct upload to AWS? Cancel form submit - delete files & thumbnails? Deep integration & assumption
37. How’s Dragonfly doing for you? http://markevans.github.io/dragonfly/
38. How’s refile doing for you? https://github.com/refile/refile
39. http://thecooperreview.com/10-tricks-appear-smart-meetings/ 10 Tricks to Appear Smart During Meetings
40. Take a step back
41. Take a step back • We want to store a bunch of attributes in a model • e.g. Title, Body, Tags, Photo
 
 
 
 
 
 
 

42. Take a step back {photo} {title} {body}
43. Take a step back <img src={photo}> <h1>{title}</h1> {body}
44. Take a step back • Why should photo be a disproportionately complicated attribute in my Article model? • stored file path • conversion • background job • aws config • clean up on delete
45. Take a step back • Why should photo be a disproportionately complicated attribute in my Article model? • stored file path • conversion • background job • validation error dance • aws config
46. Take a step back • Frankly photo_url is best; least intrusive
 
 
 
 
 
 
 
 
 

47. Take a step back • Frankly photo_url is best; least intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data
48. Take a step back • Frankly photo_url is best; least intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data
49. Take a step back • Frankly photo_url is best; least intrusive • Problems • Remote url 404? (not exclusive to your app) • Asking users to give us a URL is a hard sell • Need to render other sizes • Filter by meta data
50. Take a step back • Frankly photo_url is best; least intrusive • Solutions • Exclusive server for your app • Upload to that server • On-the-fly resize based on URL • Store url with meta data: photo_json instead?
51. Just add server
52. • PostgreSQL, MySQL • Redis • Memcached • SMTP server (Mail)
 
 
 
 You are already Generic server to do specialised work Not specific to your business logic
53. • Not a new pattern • Mostly commercial services
 
 
 
 
 • Maybe it has to be Free & Open Source to become a default pattern Image server
54. Want • Move the “concern” out of my app • photo is a regular attribute • configure my app & forget it exist
 
 
 

55. Want • Move the “concern” out of my app • photo is a regular attribute • configure my app & forget it exist
 
 
 
 What would my app look like in a better world?
56. My app: Bare minimum create_table "users" do |t| t.string "name" t.json "avatar" t.json "photos" end
57. My app: Bare minimum create_table "users" do |t| t.string "name" t.json "avatar" t.json "photos" # multiple files in a column end
58. My app: Bare minimum Image server Rails app Browser { avatar: #<File..> }
59. My app: Bare minimum Image server Rails app Browser { avatar: #<File..> } {“path”:“x.jpg”, “geometry”:“200x600”} #<File..>
60. My app: Bare minimum Image server Rails app Browser {“path”:“x.jpg”, “geometry”:“200x600”} #<File..> user.avatar={“path”: “x.jpg”…} user.save <img src=“x.jpg”> { avatar: #<File..> }
61. My app: Bare minimum Image server Rails app Browser {“path”:“x.jpg”, “geometry”:“200x600”} <img src=“x.jpg”> #<File..> GET x.jpg #<File..> user.avatar={“path”: “x.jpg”…} user.save { avatar: #<File..> }
62. My app: Bare minimum • Browser render <file> field; regular form submit • Receive binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png
 
 

63. Image server Rails app Browser #<File..> {“path”:“x.jpg”, “geometry”:“200x600”} Progressive Enhancement
64. Image server Rails app Browser #<File..> {“path”:“x.jpg”, “geometry”:“200x600”} Progressive Enhancement { avatar: {“path”:“x.jpg”, … }
65. { avatar: {“path”:“x.jpg”, … } Image server Rails app Browser #<File..> {“path”:“x.jpg”, “geometry”:“200x600”} user.avatar={“path”: “x.jpg”…} user.save <img src=“x.jpg”> Progressive Enhancement
66. • Browser render <file> field; regular form submit • Receive binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png
 
 
 Progressive Enhancement
67. • Browser render <file> field; regular form submit • Receive binary param, uploads to attache server and stores the json response instead • Your app render <img src> requesting for image in certain size, e.g. http://example/200x/ file.png
 
 
 • JS upload directly to attache server; “Direct upload” in AWS parlance • No binary in params; receive and store json attribute Progressive Enhancement • When after_update & after_destroy removes obsolete file from attache via delete API
68. • Just use Ruby; just use your framework • Pre-render multiple sizes • fetch the urls, server will generate and cache • Validation • validating a regular json attribute How do I…
69. • Move the “concern” out of my app • photo is a regular attribute • configure my app & forget it exist
 
 
 
 Want (cont’d)
70. Want (cont’d) • Move the “concern” out of my app • photo is a regular attribute • configure my app & forget it exist • Separate, standalone server • Minimal / zero ongoing administration
71. Want (cont’d) • Move the “concern” out of my app • photo is a regular attribute • configure my app & forget it exist • Separate, standalone server • Minimal / zero ongoing administration How does this server work?
72. Attache File Server • HTTP server with simple APIs • upload • download • delete • Rack app + ImageMagick • Go? Node? C++? PHP? • GraphicsMagick? MyResizer.bash?
73. • Uploaded files are stored locally • Resize local file on-the-fly • configurable pool size to limit concurrent resizing • Sync upload to cloud storage • 2 hop problem vs complexity • Fixed local storage, purge LRU (zero maintenance) • Spin up new fresh servers anytime… 
 Attache File Server
74. • When requested file does not exist locally • fetch from cloud storage & write locally • resume operations
 
 
 
 
 
 Attache File Server
75. • Remove obsolete file is “best effort” • If photo delete failed, do you Error 500 or stop the world?
 
 
 
 • OCDs can schedule rake task remove dangling files? Attache File Server
76. • Caching in production • Browser → CDN → Varnish → Disk → Cloud
 
 
 
 
 
 
 Attache File Server
77. Demo https://attache-demo.herokuapp.com/
78. https://github.com/choonkeat/attache_api ATTACHE_URL=http://localhost:9292 ATTACHE_SECRET_KEY=topsecret rake Compatibility Check
79. tus.io • Open protocol for resumable uploads built on HTTP • Perfect for mobile apps • Rack middleware implemented in choonkeat/ attache#10
80. Responsive Images with Client Hints http://blog.imgix.com/2015/10/13/next-generation-responsive-images-with-client.html
81. SmartCrop https://github.com/jwagner/smartcrop.js/
82. Dragonfly & refile
83. Dragonfly & refile • tldr: we can fuss over implementation, but it is mostly about architecture
84. Dragonfly & refile • Rack middleware in your Rails app by default • performing on-the-fly image resize • Rack standalone end point • Dragonfly.app - upload, download, delete • Refile::App - upload, download, delete • Downloads are unthrottled
85. Dragonfly & refile • BEFORE: Rails integrate with AWS • AFTER: Rails integrate with AWS + Rack app • Maintain identical AWS config in both apps • Rails app couldn’t shed the “concern” • Multiple images still require multiple models
86. refile class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for :images, attachment: :file end class Image < ActiveRecord::Base belongs_to :post attachment :file end
87. refile class Post < ActiveRecord::Base has_many :images, dependent: :destroy accepts_attachments_for :images, attachment: :file end class Image < ActiveRecord::Base belongs_to :post attachment :file end “Note it must be possible to persist images given only the associated post and a file. There must not be any other validations or constraints which prevent images from being saved” i.e. Must be pure dummy wrapper; no validations here
88. refile download https://github.com/refile/refile/blob/master/lib/refile/app.rb get "/:token/:backend/:processor/:id/:filename" do halt 404 unless download_allowed? stream_file processor.call(file) end
89. refile download https://github.com/refile/refile/blob/master/lib/refile/app.rb get "/:token/:backend/:processor/:id/:filename" do halt 404 unless download_allowed? stream_file processor.call(file) end How many ImageMagick can you run in parallel? has_many :images?
90. refile upload https://github.com/refile/refile/blob/master/lib/refile/app.rb post "/:backend" do halt 404 unless upload_allowed? tempfile = request.params.fetch("file").fetch(:tempfile) file = backend.upload(tempfile) content_type :json { id: file.id }.to_json end def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.download end
91. post "/:backend" do halt 404 unless upload_allowed? tempfile = request.params.fetch("file").fetch(:tempfile) file = backend.upload(tempfile) content_type :json { id: file.id }.to_json end def file file = backend.get(params[:id]) unless file.exists? log_error("Could not find attachment by id: #{params[:id]}") halt 404 end file.download end refile upload https://github.com/refile/refile/blob/master/lib/refile/app.rb 2 hops problem when uploading and downloading • 3mb file becomes 6mb each way • #create and #show becomes 12mb process • has_many :images?
92. refile • CarrierWave-styled named processors (e.g. :fill, :thumb) vs passing through syntax to underlying ImageMagick • personally prefer leveraging off existing knowledge • instead of manually configured syntax sugar • “2 hop problem” however provide higher consistency when running multiple refile servers • upload-here-download-there problem • (considering to perform 2-hop upload instead of async)
93. refile • Can upload to S3 direct and/or upload to refile • Redundancy interesting, but prefer less concern in Rails app • Concept of cache and store to manage “dangling file problem” is worth considering • Download urls are signed-only (this practice should be adopted) • can partly counter motivation to abuse “dangling file problem” (aka free file hosting, whee!)
94. Questions? Attache server
 https://github.com/choonkeat/attache Gem for Rails
 https://github.com/choonkeat/attache_rails Demo
 https://attache-demo.herokuapp.com/