Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Validating and processing the content of a file with Active Storage

Validating and processing the content of a file with Active Storage

Claudio B.

January 31, 2020
Tweet

More Decks by Claudio B.

Other Decks in Programming

Transcript

  1. Validating and processing
    the content of a file
    with Active Storage
    Claudio Baccigalupo

    View Slide

  2. Summary
    1. Uploading and downloading with Active Storage
    has_one_attached, rails_blob_url
    2. Extracting metadata and processing attachments
    download_blob_to_tempfile, prepending callbacks
    3. Validating the content of an attachment
    attachment_changes, attached?

    View Slide

  3. Part 1:
    uploading and downloading
    files with Active Storage
    (5 minutes)

    View Slide

  4. github.com/clutter/mdma
    Uploading files Downloading files

    View Slide

  5. Basics of Active Storage
    :package
    :version
    Started POST "/builds"

    Parameters: {"build"=>{"package"=>#, "version"=>"42"}
    :version :package

    View Slide

  6. Uploading an attachment
    # app/views/builds/new.html.erb
    <%= form_with(model: @build) do |form| %>
    1. Choose a file to upload:
    <%= form.file_field :package %>
    # app/controllers/builds_controller.rb
    def build_params
    params.require(:build).permit :package, :version
    end
    # app/models/build.rb
    class Build < ActiveRecord::Base
    has_one_attached :package
    end

    View Slide

  7. Downloading an attachment
    # config/environments/production.rb
    Rails.application.configure do
    config.active_storage.service = :s3
    end
    # app/views/builds/index.html.erb

    <% @builds.each do |build| %>
    <%= build.version %>
    <%= link_to 'Download',
    rails_blob_url(build.package,
    disposition: 'attachment') %>

    <% end %>

    View Slide

  8. Basic attachment validations
    # app/models/build.rb
    class Build < ActiveRecord::Base
    has_one_attached :package
    validates :package, presence: true
    end

    View Slide

  9. Proposed attachment validations
    # app/models/build.rb
    class Build < ActiveRecord::Base
    has_one_attached :package
    validates :package, presence: true,
    attachment_size: { maximum: 90.megabytes },
    attachment_content_type: "application/zip"
    end

    View Slide

  10. Part 2:
    extracting metadata and
    processing an attachment
    (10 minutes)

    View Slide

  11. From downloading to installing
    https://localhost:3000/rails/active_storage/blobs/
    [sha]/demo_v42.ipa?disposition=attachment
    Simply downloading the package file to an iOS
    device is not enough to install the app

    View Slide

  12. From downloading to installing
    itms-services://?action=download-
    manifest&url=https%3A%2F%2Flocalhost%3A300
    0%2Frails%2Factive_storage%2Fblobs%2F[sha]
    %2Fmanifest.plist%3Fdisposition%3Dattachment
    1. Extract URL and metadata from the package
    (app name, version, minimum OS version, …)
    2. Create and upload a "manifest.plist" file with
    all these properties in XML format
    3. Share a link to the manifest, not the package

    View Slide

  13. Taking inspiration from ImageAnalyzer
    # app/models/active_storage/attachment.rb
    class Attachment < ActiveRecord::Base
    after_create_commit { AnalyzeJob.perform_later(blob) }
    end
    # app/jobs/active_storage/analyze_job.rb
    class AnalyzeJob < BaseJob
    def perform(blob)
    # Pick an analyzer based on the blob content-type
    analyzer = Analyzer::ImageAnalyzer.new(blob)
    blob.update! metadata: analyzer.metadata
    end
    end
    # lib/active_storage/analyzer/image_analyzer.rb
    class Analyzer::ImageAnalyzer < Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    image = MiniMagick::Image.new(file.path)
    { width: image.width, height: image.height }
    end
    end
    end
    1. Whenever a model with an attachment is
    created, Rails enqueues a background job
    2. The job looks for the right analyzer for the blob.
    Metadata returned by analyzer is stored in the DB
    3. Analyzers run in a background job, so they must
    download the file to memory first.

    The file analysis is typically delegated to external
    libraries (MiniMagick, FFMpeg, …)

    View Slide

  14. Working with a custom analyzer
    # app/models/build.rb
    class Build < ActiveRecord::Base
    after_create_commit(prepend: true) do
    PackageAnalyzeJob.perform_later(self)
    end
    end
    # app/jobs/package_analyze_job.rb
    class PackageAnalyzeJob < ActiveStorage::BaseJob
    def perform(build)
    analyzer = PackageAnalyzer.new(build.package.blob)
    build.package.blob.update! metadata: analyzer.metadata
    end
    end
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    end
    end
    end
    2. The package analyzer is invoked and the
    returned metadata are stored in the database
    3. The file is downloaded to memory and the
    analysis is run by the CFPropertyBundle library
    1. Whenever a build with a package is created,
    enqueue a background job

    View Slide

  15. Working with a custom analyzer
    # app/models/build.rb
    class Build < ActiveRecord::Base
    after_create_commit(prepend: true) do
    PackageAnalyzeJob.perform_later(self)
    end
    end
    # app/jobs/package_analyze_job.rb
    class PackageAnalyzeJob < ActiveStorage::BaseJob
    def perform(build)
    analyzer = PackageAnalyzer.new(build.package.blob)
    build.package.blob.update! metadata: analyzer.metadata
    end
    end
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    end
    end
    end
    2. The package analyzer is invoked and the
    returned metadata are stored in the database
    3. The file is downloaded to memory and the
    analysis is run by the CFPropertyBundle library
    1. Whenever a build with a package is created,
    enqueue a background job
    what?

    View Slide

  16. Extracting metadata from a package
    # lib/cfpropertybundle.rb
    class CFPropertyBundle
    class Error < StandardError; end
    def initialize(path)
    @path = path
    end
    def properties
    Zip::File.open(@path) do |package|
    data = extract_plist_data_from package
    {
    display_name: data['CFBundleDisplayName'],
    identifier: data['CFBundleIdentifier'],
    version: data['CFBundleVersion'],
    minimum_os_version: data['MinimumOSVersion'],
    short_version: data['CFBundleShortVersionString']
    }
    end
    rescue Zip::Error, SystemCallError => e
    raise Error, e.message
    end
    end
    The CFPropertyBundle library unzips the
    package, parses the information stored in XML
    format and returns a Hash with the properties
    needed to generate a manifest.

    If the package is not a Zip file or the content
    doesn’t match the .ipa format, a custom
    CFProperty::Error is raised.

    View Slide

  17. Working with a custom analyzer
    # app/models/build.rb
    class Build < ActiveRecord::Base
    after_create_commit(prepend: true) do
    PackageAnalyzeJob.perform_later(self)
    end
    end
    # app/jobs/package_analyze_job.rb
    class PackageAnalyzeJob < ActiveStorage::BaseJob
    def perform(build)
    analyzer = PackageAnalyzer.new(build.package.blob)
    build.package.blob.update! metadata: analyzer.metadata
    end
    end
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    end
    end
    end
    2. The package analyzer is invoked and the
    returned metadata are stored in the database
    3. The file is downloaded to memory and the
    analysis is run by the CFPropertyBundle library
    1. Whenever a build with a package is created,
    enqueue a background job
    why?

    View Slide

  18. Working with a custom analyzer
    # app/models/build.rb
    class Build < ActiveRecord::Base
    after_create_commit(prepend: true) do
    PackageAnalyzeJob.perform_later(self)
    end
    end
    # app/jobs/package_analyze_job.rb
    class PackageAnalyzeJob < ActiveStorage::BaseJob
    def perform(build)
    analyzer = PackageAnalyzer.new(build.package.blob)
    build.package.blob.update! metadata: analyzer.metadata
    end
    end
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    end
    end
    end
    2. The package analyzer is invoked and the
    returned metadata are stored in the database
    3. The file is downloaded to memory and the
    analysis is run by the CFPropertyBundle library
    1. Whenever a build with a package is created,
    enqueue a background job
    the manifest can be generated here

    View Slide

  19. Processing an attachment
    # app/jobs/package_analyze_job.rb
    class PackageAnalyzeJob < ActiveStorage::BaseJob
    def perform(build)
    metadata = PackageAnalyzer.new(build.package.blob).metadata
    build.package.blob.update! metadata: metadata
    Tempfile.open ['manifest', '.plist'] do |file|
    file.write <<~MANIFEST
    […]
    url#{CGI.escapeHTML build.package.service_url}
    title#{metadata[:bundle_display_name]}
    bundle-version#{metadata[:short_version]}[…]
    MANIFEST
    build.manifest.attach io: File.open(file.path), filename: 'manifest.plist'
    end
    end
    end
    # app/models/build.rb
    class Build < ActiveRecord::Base
    has_one_attached :package
    has_one_attached :manifest
    end
    Attaching the manifest to the build grants access to all the
    convenient Active Storage methods (attach, rails_blob_url, …)

    View Slide

  20. From downloading to installing
    itms-services://?action=download-
    manifest&url=https%3A%2F%2Flocalhost%3A300
    0%2Frails%2Factive_storage%2Fblobs%2F[sha]
    %2Fmanifest.plist%3Fdisposition%3Dattachment
    manifest_url = rails_blob_url(
    build.manifest,
    disposition: 'attachment')
    <%= link_to 'Install',
    "itms-services://
    ?action=download-manifest
    &url=#{CGI.escape manifest_url)}" %>

    View Slide

  21. Part 3:
    validating the content
    of an attachment
    (5 minutes)

    View Slide

  22. Validating the content of an attachment

    View Slide

  23. Validating the content of an attachment
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    # extracts properties from the iOS bundle
    {
    display_name: "iOS app",
    identifier: "com.example.app",
    version: "42",
    minimum_os_version: "8.0",
    short_version: "1.0"
    }
    end
    end
    end

    View Slide

  24. Validating the content of an attachment

    View Slide

  25. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    validates :version, presence: true, uniqueness: true
    end

    View Slide

  26. Validating the content of an attachment
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    # extracts properties from the iOS bundle
    {
    display_name: "iOS app",
    identifier: "com.example.app",
    version: "42",
    minimum_os_version: "8.0",
    short_version: "1.0"
    }
    end
    end
    end

    View Slide

  27. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end

    View Slide

  28. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end

    View Slide

  29. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end

    View Slide

  30. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end
    Started POST "/builds"

    Parameters: {"build"=>{"package"=>#}

    View Slide

  31. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end

    View Slide

  32. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end

    View Slide

  33. Summary
    1. Uploading and downloading with Active Storage
    has_one_attached, rails_blob_url
    2. Extracting metadata and processing attachments
    download_blob_to_tempfile, prepending callbacks
    3. Validating the content of an attachment
    attachment_changes, attached?

    View Slide

  34. Thanks!
    Questions?

    View Slide