EAD Ingest and Heracles Workflow

Transcript

1. EAD Ingest and Heracles Workflow System Rajesh Balekai Donald Brower

   Digital Library Services Hesburgh Libraries University of Notre Dame Hydra Partners’ Meeting September 2012

2. EAD Ingest Have many Encoded Archival Description (EAD) files describing

   various special collections. Every night we scan each EAD file and Ingest any new or changed item descriptions Ingest any new or changed images Create a new finding aid document, if needed

3. Collection Component FindingAid FileAsset (HTML File) Image :has_members :has_parts :is_member_of

   :is_member_of :is_part_of :has_members :has_parts :is_part_of :has_parts :is_part_of FEDORA OBJECT MODEL

4. EAD Process Flow Chart

5. EAD Process Flow Chart—Top      

                   

6. EAD Process Flow Chart—Bottom      

                        

7. Image Ingest        

             

8. Heracles We implement the EAD ingestion process on top of

   our new background job system, Heracles. We plan to develop more services to use it: Ingest EAD files and content specified therein Handle digital reformatting of brittle items Automate as much ingest processing as possible Oversee scanning & digitization of workflows Run background integrity checks of stored content

9. How Each job has a state machine to control execution

   of the job’s tasks Job state and information are stored in a relational database Tasks are idempotent units of execution that do the actual work Distributing the tasks is handled by Resque Worker tasks use ActiveFedora to interface with Fedora and Solr
10. None

11. Example State Machine DSL Diagram is_file_different ingest_inquisition_image :yes done :no

    :ok

12. Rabbit Test State Machine DSL Diagram spawn_new_rabbit get_random_yes_or_no :ok done

    :leveled_out :no wait_one_second :yes :ok

13. Open Issues Have not determined best way to automate a

    test of the whole stack Still working out deployment details: Using Jenkins and Capistrano and Resque-pool

14. Sharing the Code Code will be on GitHub. Two repositories:

    Heracles The workflow state machine and general framework TBD Our worker tasks. Goal is to make tasks reusable between workflows. https://github.com/ndlib/heracles