Validating and processing the content of a file with Active Storage

Validating and processing the content of a file with Active Storage

0722f1ff8d0a69bce57ebdb93dafc395?s=128

Claudio B.

January 31, 2020
Tweet

Transcript

  1. Validating and processing the content of a file with Active

    Storage Claudio Baccigalupo
  2. Summary 1. Uploading and downloading with Active Storage has_one_attached, rails_blob_url

    2. Extracting metadata and processing attachments download_blob_to_tempfile, prepending callbacks 3. Validating the content of an attachment attachment_changes, attached?
  3. Part 1: uploading and downloading files with Active Storage (5

    minutes)
  4. github.com/clutter/mdma Uploading files Downloading files

  5. Basics of Active Storage :package :version Started POST "/builds" Parameters:

    {"build"=>{"package"=>#<ActionDispatch::Http::UploadedFile:47e0>, "version"=>"42"} :version :package
  6. Uploading an attachment # app/views/builds/new.html.erb <%= form_with(model: @build) do |form|

    %> <legend>1. Choose a file to upload:</legend> <%= form.file_field :package %> # app/controllers/builds_controller.rb def build_params params.require(:build).permit :package, :version end # app/models/build.rb class Build < ActiveRecord::Base has_one_attached :package end
  7. Downloading an attachment # config/environments/production.rb Rails.application.configure do config.active_storage.service = :s3

    end # app/views/builds/index.html.erb <table> <% @builds.each do |build| %> <td><%= build.version %></td> <td><%= link_to 'Download', rails_blob_url(build.package, disposition: 'attachment') %> </td> <% end %> </table>
  8. Basic attachment validations # app/models/build.rb class Build < ActiveRecord::Base has_one_attached

    :package validates :package, presence: true end
  9. Proposed attachment validations # app/models/build.rb class Build < ActiveRecord::Base has_one_attached

    :package validates :package, presence: true, attachment_size: { maximum: 90.megabytes }, attachment_content_type: "application/zip" end
  10. Part 2: extracting metadata and processing an attachment (10 minutes)

  11. From downloading to installing https://localhost:3000/rails/active_storage/blobs/ [sha]/demo_v42.ipa?disposition=attachment Simply downloading the package

    file to an iOS device is not enough to install the app
  12. From downloading to installing itms-services://?action=download- manifest&url=https%3A%2F%2Flocalhost%3A300 0%2Frails%2Factive_storage%2Fblobs%2F[sha] %2Fmanifest.plist%3Fdisposition%3Dattachment 1. Extract

    URL and metadata from the package (app name, version, minimum OS version, …) 2. Create and upload a "manifest.plist" file with all these properties in XML format 3. Share a link to the manifest, not the package
  13. Taking inspiration from ImageAnalyzer # app/models/active_storage/attachment.rb class Attachment < ActiveRecord::Base

    after_create_commit { AnalyzeJob.perform_later(blob) } end # app/jobs/active_storage/analyze_job.rb class AnalyzeJob < BaseJob def perform(blob) # Pick an analyzer based on the blob content-type analyzer = Analyzer::ImageAnalyzer.new(blob) blob.update! metadata: analyzer.metadata end end # lib/active_storage/analyzer/image_analyzer.rb class Analyzer::ImageAnalyzer < Analyzer def metadata download_blob_to_tempfile do |file| image = MiniMagick::Image.new(file.path) { width: image.width, height: image.height } end end end 1. Whenever a model with an attachment is created, Rails enqueues a background job 2. The job looks for the right analyzer for the blob. Metadata returned by analyzer is stored in the DB 3. Analyzers run in a background job, so they must download the file to memory first. The file analysis is typically delegated to external libraries (MiniMagick, FFMpeg, …)
  14. Working with a custom analyzer # app/models/build.rb class Build <

    ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job
  15. Working with a custom analyzer # app/models/build.rb class Build <

    ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job what?
  16. Extracting metadata from a package # lib/cfpropertybundle.rb class CFPropertyBundle class

    Error < StandardError; end def initialize(path) @path = path end def properties Zip::File.open(@path) do |package| data = extract_plist_data_from package { display_name: data['CFBundleDisplayName'], identifier: data['CFBundleIdentifier'], version: data['CFBundleVersion'], minimum_os_version: data['MinimumOSVersion'], short_version: data['CFBundleShortVersionString'] } end rescue Zip::Error, SystemCallError => e raise Error, e.message end end The CFPropertyBundle library unzips the package, parses the information stored in XML format and returns a Hash with the properties needed to generate a manifest. If the package is not a Zip file or the content doesn’t match the .ipa format, a custom CFProperty::Error is raised.
  17. Working with a custom analyzer # app/models/build.rb class Build <

    ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job why?
  18. Working with a custom analyzer # app/models/build.rb class Build <

    ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job the manifest can be generated here
  19. Processing an attachment # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def

    perform(build) metadata = PackageAnalyzer.new(build.package.blob).metadata build.package.blob.update! metadata: metadata Tempfile.open ['manifest', '.plist'] do |file| file.write <<~MANIFEST <?xml version="1.0" encoding="UTF-8"?>[…] <key>url</key><string>#{CGI.escapeHTML build.package.service_url}</string> <key>title</key><string>#{metadata[:bundle_display_name]}</string> <key>bundle-version</key><string>#{metadata[:short_version]}</string>[…] MANIFEST build.manifest.attach io: File.open(file.path), filename: 'manifest.plist' end end end # app/models/build.rb class Build < ActiveRecord::Base has_one_attached :package has_one_attached :manifest end Attaching the manifest to the build grants access to all the convenient Active Storage methods (attach, rails_blob_url, …)
  20. From downloading to installing itms-services://?action=download- manifest&url=https%3A%2F%2Flocalhost%3A300 0%2Frails%2Factive_storage%2Fblobs%2F[sha] %2Fmanifest.plist%3Fdisposition%3Dattachment manifest_url =

    rails_blob_url( build.manifest, disposition: 'attachment') <%= link_to 'Install', "itms-services:// ?action=download-manifest &url=#{CGI.escape manifest_url)}" %>
  21. Part 3: validating the content of an attachment (5 minutes)

  22. Validating the content of an attachment

  23. Validating the content of an attachment # app/analyzers/package_analyzer.rb class PackageAnalyzer

    < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties # extracts properties from the iOS bundle { display_name: "iOS app", identifier: "com.example.app", version: "42", minimum_os_version: "8.0", short_version: "1.0" } end end end
  24. Validating the content of an attachment

  25. Validating the content of an attachment # app/models/build.rb class Build

    < ActiveRecord::Base validates :version, presence: true, uniqueness: true end
  26. Validating the content of an attachment # app/analyzers/package_analyzer.rb class PackageAnalyzer

    < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties # extracts properties from the iOS bundle { display_name: "iOS app", identifier: "com.example.app", version: "42", minimum_os_version: "8.0", short_version: "1.0" } end end end
  27. Validating the content of an attachment # app/models/build.rb class Build

    < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end
  28. Validating the content of an attachment # app/models/build.rb class Build

    < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end
  29. Validating the content of an attachment # app/models/build.rb class Build

    < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end
  30. Validating the content of an attachment # app/models/build.rb class Build

    < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end Started POST "/builds" Parameters: {"build"=>{"package"=>#<ActionDispatch::Http::UploadedFile:47e0 @tempfile=#<Tempfile:/var/t14t7.ipa>}
  31. Validating the content of an attachment # app/models/build.rb class Build

    < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end
  32. Validating the content of an attachment # app/models/build.rb class Build

    < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end
  33. Summary 1. Uploading and downloading with Active Storage has_one_attached, rails_blob_url

    2. Extracting metadata and processing attachments download_blob_to_tempfile, prepending callbacks 3. Validating the content of an attachment attachment_changes, attached?
  34. Thanks! Questions?