Validating and processing the content of a file with Active Storage Claudio Baccigalupo

Summary 1. Uploading and downloading with Active Storage has_one_attached, rails_blob_url 2. Extracting metadata and processing attachments download_blob_to_tempfile, prepending callbacks 3. Validating the content of an attachment attachment_changes, attached?

Part 1: uploading and downloading files with Active Storage (5 minutes)

github.com/clutter/mdma Uploading files Downloading files

Basics of Active Storage :package :version Started POST "/builds" Parameters: {"build"=>{"package"=>#, "version"=>"42"} :version :package

Uploading an attachment # app/views/builds/new.html.erb <%= form_with(model: @build) do |form| %> 1. Choose a file to upload: <%= form.file_field :package %> # app/controllers/builds_controller.rb def build_params params.require(:build).permit :package, :version end # app/models/build.rb class Build < ActiveRecord::Base has_one_attached :package end

Downloading an attachment # config/environments/production.rb Rails.application.configure do config.active_storage.service = :s3 end # app/views/builds/index.html.erb <% @builds.each do |build| %> <%= build.version %> <%= link_to 'Download', rails_blob_url(build.package, disposition: 'attachment') %> <% end %>

Basic attachment validations # app/models/build.rb class Build < ActiveRecord::Base has_one_attached :package validates :package, presence: true end

Proposed attachment validations # app/models/build.rb class Build < ActiveRecord::Base has_one_attached :package validates :package, presence: true, attachment_size: { maximum: 90.megabytes }, attachment_content_type: "application/zip" end

Part 2: extracting metadata and processing an attachment (10 minutes)

From downloading to installing https://localhost:3000/rails/active_storage/blobs/ [sha]/demo_v42.ipa?disposition=attachment Simply downloading the package file to an iOS device is not enough to install the app

From downloading to installing itms-services://?action=download- manifest&url=https%3A%2F%2Flocalhost%3A300 0%2Frails%2Factive_storage%2Fblobs%2F[sha] %2Fmanifest.plist%3Fdisposition%3Dattachment 1. Extract URL and metadata from the package (app name, version, minimum OS version, …) 2. Create and upload a "manifest.plist" file with all these properties in XML format 3. Share a link to the manifest, not the package

Taking inspiration from ImageAnalyzer # app/models/active_storage/attachment.rb class Attachment < ActiveRecord::Base after_create_commit { AnalyzeJob.perform_later(blob) } end # app/jobs/active_storage/analyze_job.rb class AnalyzeJob < BaseJob def perform(blob) # Pick an analyzer based on the blob content-type analyzer = Analyzer::ImageAnalyzer.new(blob) blob.update! metadata: analyzer.metadata end end # lib/active_storage/analyzer/image_analyzer.rb class Analyzer::ImageAnalyzer < Analyzer def metadata download_blob_to_tempfile do |file| image = MiniMagick::Image.new(file.path) { width: image.width, height: image.height } end end end 1. Whenever a model with an attachment is created, Rails enqueues a background job 2. The job looks for the right analyzer for the blob. Metadata returned by analyzer is stored in the DB 3. Analyzers run in a background job, so they must download the file to memory first. The file analysis is typically delegated to external libraries (MiniMagick, FFMpeg, …)

Working with a custom analyzer # app/models/build.rb class Build < ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job

Working with a custom analyzer # app/models/build.rb class Build < ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job what?

Extracting metadata from a package # lib/cfpropertybundle.rb class CFPropertyBundle class Error < StandardError; end def initialize(path) @path = path end def properties Zip::File.open(@path) do |package| data = extract_plist_data_from package { display_name: data['CFBundleDisplayName'], identifier: data['CFBundleIdentifier'], version: data['CFBundleVersion'], minimum_os_version: data['MinimumOSVersion'], short_version: data['CFBundleShortVersionString'] } end rescue Zip::Error, SystemCallError => e raise Error, e.message end end The CFPropertyBundle library unzips the package, parses the information stored in XML format and returns a Hash with the properties needed to generate a manifest. If the package is not a Zip file or the content doesn’t match the .ipa format, a custom CFProperty::Error is raised.

Working with a custom analyzer # app/models/build.rb class Build < ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job why?

Working with a custom analyzer # app/models/build.rb class Build < ActiveRecord::Base after_create_commit(prepend: true) do PackageAnalyzeJob.perform_later(self) end end # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) analyzer = PackageAnalyzer.new(build.package.blob) build.package.blob.update! metadata: analyzer.metadata end end # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties end end end 2. The package analyzer is invoked and the returned metadata are stored in the database 3. The file is downloaded to memory and the analysis is run by the CFPropertyBundle library 1. Whenever a build with a package is created, enqueue a background job the manifest can be generated here

Processing an attachment # app/jobs/package_analyze_job.rb class PackageAnalyzeJob < ActiveStorage::BaseJob def perform(build) metadata = PackageAnalyzer.new(build.package.blob).metadata build.package.blob.update! metadata: metadata Tempfile.open ['manifest', '.plist'] do |file| file.write <<~MANIFEST […] url#{CGI.escapeHTML build.package.service_url} title#{metadata[:bundle_display_name]} bundle-version#{metadata[:short_version]}[…] MANIFEST build.manifest.attach io: File.open(file.path), filename: 'manifest.plist' end end end # app/models/build.rb class Build < ActiveRecord::Base has_one_attached :package has_one_attached :manifest end Attaching the manifest to the build grants access to all the convenient Active Storage methods (attach, rails_blob_url, …)

From downloading to installing itms-services://?action=download- manifest&url=https%3A%2F%2Flocalhost%3A300 0%2Frails%2Factive_storage%2Fblobs%2F[sha] %2Fmanifest.plist%3Fdisposition%3Dattachment manifest_url = rails_blob_url( build.manifest, disposition: 'attachment') <%= link_to 'Install', "itms-services:// ?action=download-manifest &url=#{CGI.escape manifest_url)}" %>

Part 3: validating the content of an attachment (5 minutes)

Validating the content of an attachment

Validating the content of an attachment # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties # extracts properties from the iOS bundle { display_name: "iOS app", identifier: "com.example.app", version: "42", minimum_os_version: "8.0", short_version: "1.0" } end end end

Validating the content of an attachment

Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base validates :version, presence: true, uniqueness: true end

Validating the content of an attachment # app/analyzers/package_analyzer.rb class PackageAnalyzer < ActiveStorage::Analyzer def metadata download_blob_to_tempfile do |file| CFPropertyBundle.new(file.path).properties # extracts properties from the iOS bundle { display_name: "iOS app", identifier: "com.example.app", version: "42", minimum_os_version: "8.0", short_version: "1.0" } end end end

Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end

Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end

Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end

Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end Started POST "/builds" Parameters: {"build"=>{"package"=>#}

Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end

Validating the content of an attachment # app/models/build.rb class Build < ActiveRecord::Base with_options if: -> { package.attached? } do before_validation :set_version, on: :create validates :version, presence: true, uniqueness: true end def set_version file = attachment_changes['package'].attachable metadata = CFPropertyBundle.new(file.path).properties self.version = metadata[:version] rescue CFPropertyBundle::Error message = 'cannot be extracted from the attachment' errors.add(:version, :invalid, message: message) end end

Summary 1. Uploading and downloading with Active Storage has_one_attached, rails_blob_url 2. Extracting metadata and processing attachments download_blob_to_tempfile, prepending callbacks 3. Validating the content of an attachment attachment_changes, attached?

Thanks! Questions?