Validating and processing the content of a file with Active Storage

Transcript

1. Validating and processing
   the content of a file
   with Active Storage
   Claudio Baccigalupo
   

   View Slide

2. Summary
   1. Uploading and downloading with Active Storage
   has_one_attached, rails_blob_url
   2. Extracting metadata and processing attachments
   download_blob_to_tempfile, prepending callbacks
   3. Validating the content of an attachment
   attachment_changes, attached?
   

   View Slide

3. Part 1:
   uploading and downloading
   files with Active Storage
   (5 minutes)
   

   View Slide

4. github.com/clutter/mdma
   Uploading files Downloading files
   

   View Slide

5. Basics of Active Storage
   :package
   :version
   Started POST "/builds"
   
   Parameters: {"build"=>{"package"=>#, "version"=>"42"}
   :version :package
   

   View Slide

6. Uploading an attachment
   # app/views/builds/new.html.erb
   <%= form_with(model: @build) do |form| %>
   1. Choose a file to upload:
   <%= form.file_field :package %>
   # app/controllers/builds_controller.rb
   def build_params
   params.require(:build).permit :package, :version
   end
   # app/models/build.rb
   class Build < ActiveRecord::Base
   has_one_attached :package
   end
   

   View Slide

7. Downloading an attachment
   # config/environments/production.rb
   Rails.application.configure do
   config.active_storage.service = :s3
   end
   # app/views/builds/index.html.erb
   
   <% @builds.each do |build| %>
   <%= build.version %>
   <%= link_to 'Download',
   rails_blob_url(build.package,
   disposition: 'attachment') %>
   
   <% end %>
   
   

   View Slide

8. Basic attachment validations
   # app/models/build.rb
   class Build < ActiveRecord::Base
   has_one_attached :package
   validates :package, presence: true
   end
   

   View Slide

9. Proposed attachment validations
   # app/models/build.rb
   class Build < ActiveRecord::Base
   has_one_attached :package
   validates :package, presence: true,
   attachment_size: { maximum: 90.megabytes },
   attachment_content_type: "application/zip"
   end
   

   View Slide

10. Part 2:
    extracting metadata and
    processing an attachment
    (10 minutes)
    

    View Slide

11. From downloading to installing
    https://localhost:3000/rails/active_storage/blobs/
    [sha]/demo_v42.ipa?disposition=attachment
    Simply downloading the package file to an iOS
    device is not enough to install the app
    

    View Slide

12. From downloading to installing
    itms-services://?action=download-
    manifest&url=https%3A%2F%2Flocalhost%3A300
    0%2Frails%2Factive_storage%2Fblobs%2F[sha]
    %2Fmanifest.plist%3Fdisposition%3Dattachment
    1. Extract URL and metadata from the package
    (app name, version, minimum OS version, …)
    2. Create and upload a "manifest.plist" file with
    all these properties in XML format
    3. Share a link to the manifest, not the package
    

    View Slide

13. Taking inspiration from ImageAnalyzer
    # app/models/active_storage/attachment.rb
    class Attachment < ActiveRecord::Base
    after_create_commit { AnalyzeJob.perform_later(blob) }
    end
    # app/jobs/active_storage/analyze_job.rb
    class AnalyzeJob < BaseJob
    def perform(blob)
    # Pick an analyzer based on the blob content-type
    analyzer = Analyzer::ImageAnalyzer.new(blob)
    blob.update! metadata: analyzer.metadata
    end
    end
    # lib/active_storage/analyzer/image_analyzer.rb
    class Analyzer::ImageAnalyzer < Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    image = MiniMagick::Image.new(file.path)
    { width: image.width, height: image.height }
    end
    end
    end
    1. Whenever a model with an attachment is
    created, Rails enqueues a background job
    2. The job looks for the right analyzer for the blob.
    Metadata returned by analyzer is stored in the DB
    3. Analyzers run in a background job, so they must
    download the file to memory first.
    
    The file analysis is typically delegated to external
    libraries (MiniMagick, FFMpeg, …)
    

    View Slide

14. Working with a custom analyzer
    # app/models/build.rb
    class Build < ActiveRecord::Base
    after_create_commit(prepend: true) do
    PackageAnalyzeJob.perform_later(self)
    end
    end
    # app/jobs/package_analyze_job.rb
    class PackageAnalyzeJob < ActiveStorage::BaseJob
    def perform(build)
    analyzer = PackageAnalyzer.new(build.package.blob)
    build.package.blob.update! metadata: analyzer.metadata
    end
    end
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    end
    end
    end
    2. The package analyzer is invoked and the
    returned metadata are stored in the database
    3. The file is downloaded to memory and the
    analysis is run by the CFPropertyBundle library
    1. Whenever a build with a package is created,
    enqueue a background job
    

    View Slide

15. Working with a custom analyzer
    # app/models/build.rb
    class Build < ActiveRecord::Base
    after_create_commit(prepend: true) do
    PackageAnalyzeJob.perform_later(self)
    end
    end
    # app/jobs/package_analyze_job.rb
    class PackageAnalyzeJob < ActiveStorage::BaseJob
    def perform(build)
    analyzer = PackageAnalyzer.new(build.package.blob)
    build.package.blob.update! metadata: analyzer.metadata
    end
    end
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    end
    end
    end
    2. The package analyzer is invoked and the
    returned metadata are stored in the database
    3. The file is downloaded to memory and the
    analysis is run by the CFPropertyBundle library
    1. Whenever a build with a package is created,
    enqueue a background job
    what?
    

    View Slide

16. Extracting metadata from a package
    # lib/cfpropertybundle.rb
    class CFPropertyBundle
    class Error < StandardError; end
    def initialize(path)
    @path = path
    end
    def properties
    Zip::File.open(@path) do |package|
    data = extract_plist_data_from package
    {
    display_name: data['CFBundleDisplayName'],
    identifier: data['CFBundleIdentifier'],
    version: data['CFBundleVersion'],
    minimum_os_version: data['MinimumOSVersion'],
    short_version: data['CFBundleShortVersionString']
    }
    end
    rescue Zip::Error, SystemCallError => e
    raise Error, e.message
    end
    end
    The CFPropertyBundle library unzips the
    package, parses the information stored in XML
    format and returns a Hash with the properties
    needed to generate a manifest.
    
    If the package is not a Zip file or the content
    doesn’t match the .ipa format, a custom
    CFProperty::Error is raised.
    

    View Slide

17. Working with a custom analyzer
    # app/models/build.rb
    class Build < ActiveRecord::Base
    after_create_commit(prepend: true) do
    PackageAnalyzeJob.perform_later(self)
    end
    end
    # app/jobs/package_analyze_job.rb
    class PackageAnalyzeJob < ActiveStorage::BaseJob
    def perform(build)
    analyzer = PackageAnalyzer.new(build.package.blob)
    build.package.blob.update! metadata: analyzer.metadata
    end
    end
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    end
    end
    end
    2. The package analyzer is invoked and the
    returned metadata are stored in the database
    3. The file is downloaded to memory and the
    analysis is run by the CFPropertyBundle library
    1. Whenever a build with a package is created,
    enqueue a background job
    why?
    

    View Slide

18. Working with a custom analyzer
    # app/models/build.rb
    class Build < ActiveRecord::Base
    after_create_commit(prepend: true) do
    PackageAnalyzeJob.perform_later(self)
    end
    end
    # app/jobs/package_analyze_job.rb
    class PackageAnalyzeJob < ActiveStorage::BaseJob
    def perform(build)
    analyzer = PackageAnalyzer.new(build.package.blob)
    build.package.blob.update! metadata: analyzer.metadata
    end
    end
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    end
    end
    end
    2. The package analyzer is invoked and the
    returned metadata are stored in the database
    3. The file is downloaded to memory and the
    analysis is run by the CFPropertyBundle library
    1. Whenever a build with a package is created,
    enqueue a background job
    the manifest can be generated here
    

    View Slide

19. Processing an attachment
    # app/jobs/package_analyze_job.rb
    class PackageAnalyzeJob < ActiveStorage::BaseJob
    def perform(build)
    metadata = PackageAnalyzer.new(build.package.blob).metadata
    build.package.blob.update! metadata: metadata
    Tempfile.open ['manifest', '.plist'] do |file|
    file.write <<~MANIFEST
    […]
    url#{CGI.escapeHTML build.package.service_url}
    title#{metadata[:bundle_display_name]}
    bundle-version#{metadata[:short_version]}[…]
    MANIFEST
    build.manifest.attach io: File.open(file.path), filename: 'manifest.plist'
    end
    end
    end
    # app/models/build.rb
    class Build < ActiveRecord::Base
    has_one_attached :package
    has_one_attached :manifest
    end
    Attaching the manifest to the build grants access to all the
    convenient Active Storage methods (attach, rails_blob_url, …)
    

    View Slide

20. From downloading to installing
    itms-services://?action=download-
    manifest&url=https%3A%2F%2Flocalhost%3A300
    0%2Frails%2Factive_storage%2Fblobs%2F[sha]
    %2Fmanifest.plist%3Fdisposition%3Dattachment
    manifest_url = rails_blob_url(
    build.manifest,
    disposition: 'attachment')
    <%= link_to 'Install',
    "itms-services://
    ?action=download-manifest
    &url=#{CGI.escape manifest_url)}" %>
    

    View Slide

21. Part 3:
    validating the content
    of an attachment
    (5 minutes)
    

    View Slide

22. Validating the content of an attachment
    

    View Slide

23. Validating the content of an attachment
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    # extracts properties from the iOS bundle
    {
    display_name: "iOS app",
    identifier: "com.example.app",
    version: "42",
    minimum_os_version: "8.0",
    short_version: "1.0"
    }
    end
    end
    end
    

    View Slide

24. Validating the content of an attachment
    

    View Slide

25. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    validates :version, presence: true, uniqueness: true
    end
    

    View Slide

26. Validating the content of an attachment
    # app/analyzers/package_analyzer.rb
    class PackageAnalyzer < ActiveStorage::Analyzer
    def metadata
    download_blob_to_tempfile do |file|
    CFPropertyBundle.new(file.path).properties
    # extracts properties from the iOS bundle
    {
    display_name: "iOS app",
    identifier: "com.example.app",
    version: "42",
    minimum_os_version: "8.0",
    short_version: "1.0"
    }
    end
    end
    end
    

    View Slide

27. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end
    

    View Slide

28. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end
    

    View Slide

29. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end
    

    View Slide

30. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end
    Started POST "/builds"
    
    Parameters: {"build"=>{"package"=>#}
    

    View Slide

31. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end
    

    View Slide

32. Validating the content of an attachment
    # app/models/build.rb
    class Build < ActiveRecord::Base
    with_options if: -> { package.attached? } do
    before_validation :set_version, on: :create
    validates :version, presence: true, uniqueness: true
    end
    def set_version
    file = attachment_changes['package'].attachable
    metadata = CFPropertyBundle.new(file.path).properties
    self.version = metadata[:version]
    rescue CFPropertyBundle::Error
    message = 'cannot be extracted from the attachment'
    errors.add(:version, :invalid, message: message)
    end
    end
    

    View Slide

33. Summary
    1. Uploading and downloading with Active Storage
    has_one_attached, rails_blob_url
    2. Extracting metadata and processing attachments
    download_blob_to_tempfile, prepending callbacks
    3. Validating the content of an attachment
    attachment_changes, attached?
    

    View Slide

34. Thanks!
    Questions?
    

    View Slide