un-uploader-demo

An example full-stack upload workflow for Ember applications

Transcript

1. Hello Toronto Ember Carsten - @heycarsten Mattia - @ghedamat
2. Un-Uploader An example full-stack upload workflow for Ember applications
3. File uploads with Ember FileReader.readAsDataURL "image":{"data":"data:image/png;base64,44Xj…"} NO NO NO NO NO NO
4. It’s not just an Ember problem, it’s a full-stack problem Why?
5. Server Client S3 Pusher 3. Upload to signed URL 2. Generate signed URL for S3 4. Persist image model 6. Notify of completed / failed processing 5. Fetch uploaded file for processing The Flow 1. User selects files
6. • Provides file upload workflow management • Displays processed content The Parts Server S3 Pusher • Generates signed S3 upload URL • Processes uploaded files via worker queue • Provides REST API for Ember Data • Very cost effective redundant storage • Handles direct upload from browser • Stores processed versions of uploads • Simple and robust pub/sub message platform • Notifies client of processing events
7. Advantages • Direct upload from client to S3 • Upload progress notification • Async processing on the server • Notify client on completed processing • Notify other connected clients of newly created objects
8. Demo http://uploader.unspace.ca
9. Client / 1 - File-picker component • File-input (with multiple files) • Sends an event to the controller with the list of images • Styled to be hidden behind a button
10. Client / 2 - ArrayController • Has the list of the uploaded images • Wraps images with an item-controller • Every image is handled independently • Handles the action from the component ◦ Creates Ember Data records ◦ Sends the startUpload action to each added image
11. Client / 3 - Item Controller uploading Upload to S3 signing Get signed url processing Transcode file processed failed initialized start upload process complete error error (if file is invalid) (if processing fails) When a state is entered an action is performed
12. Client / 4 - Notes and Caveats • IE 10+ Uses native XHR2 upload, CORS • Possible to support older browsers with extra code • Basic drag-n-drop thanks to Chrome/Firefox • Pusher is available to the controller via an initializer, fancier solutions like ember-pusher exist • Support for drop file outside of add image button also possible, not implemented in example • You might want to use separate arrays for the existing images and the ones under upload, moving them from one list to the other on complete
13. Server / 1 - Rails • POST /api/uploads endpoint to get a signed-url for the upload • POST /api/images endpoint to create the server-side model when the upload is completed • After the Image is created a job for processing is enqueued • Rails can be easily swapped out for another server side technology (Node, Go, etc.)
14. Server / 2 - Sidekiq and Pusher • Sidekiq processes the images (with ImageMagick) • A Pusher event is sent when the processing succeeds/fails • (Replace Pusher with Faye, Socket.io, message_bus)
15. Server / 3 - Notes and Caveats • Ember app is currently built by ember-cli and served by Nginx / Rails • A better approach would be to push assets to S3 and serve only the index from Rails (See Luke Melia’s great talk) • S3 images sometimes are not immediately available, the current background job implementation does polling, web-hooks are another viable option
16. Final Notes • This is a mid-level complexity system • Designed for more than resizing images: ◦ Audio processing ◦ Video processing ◦ Data analysis ◦ … • If S3 is a concern, direct uploads are possible
17. The Future • Solid testing for upload process (state machine is tested and UI too) • Libraryification (omg gross I just typed that) of the client and server ◦ {{un-uploader}} ◦ Un.Uploader.configure({ … }) • Add other Server-side options (PRs are welcome :) )
18. Thanks Carsten - @heycarsten Mattia - @ghedamat http://uploader.unspace.ca https://github.com/unspace/un-uploader-demo