Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Digital Repositories Infrastructure

CETA-Ciemat
December 15, 2011

Digital Repositories Infrastructure

CETA-Ciemat

December 15, 2011
Tweet

More Decks by CETA-Ciemat

Other Decks in Technology

Transcript

  1. www.epikh.eu
    The EPIKH Project
    (Exchange Programme to advance e-Infrastructure Know-How)
    Digital Repositories Infractucture
    Jorge Sevilla Cedillo([email protected])
    CETA-Ciemat
    University of Stellenbosch
    10th June 2011

    View full-size slide

  2. Stellenbosch, EPIKH Workshop , 10.06.2011 2
    Outline
    • What is DRI?
    • Architecture
    • How it works?
    • API as library
    • API as Resful access
    • Jobs API.

    View full-size slide

  3. Stellenbosch, EPIKH Workshop , 10.06.2011 3
    What is DRI?
    • DRI, Digital Repositories Infrastructure
    – Provide store digital repositories on different stores.
     Store metadata and digital content.
     Abstraction
     Easy commands
    – Extensible.
     Add news modules.
    • Browsing
    • Storage
     Default hibernate modules.
    • FTP, File and SE drivers.
    – Projects.
     IMED. Portugal
     BIOMED. Colombia.
     AHT. Spain
    WEB
    DRI
    DB SE
    http

    View full-size slide

  4. Stellenbosch, EPIKH Workshop , 10.06.2011 4
    Architecture
    DRI-ENGINE
    Hppt
    REST-WS
    WEB-DRI Client Client
    DB
    Storage
    Browsing
    Storage
    Browsing
    Storage
    Browsing
    Storage
    Browsing
    SE DB
    FTP FILE
    DB SE
    AMGA
    Repository-1 Repository-2 Repository-3 Repository-N
    JSON
    DRINode
    DRINode
    Persistent
    Node
    DRI-JOB
    SERVICE
    Local glite
    Client Client

    View full-size slide

  5. Desing Datamodel
    • Depends of the modules.
    – Each different module uses his own repository format.
    • A repository had.
     Datamodel.
     Configured instance of the browsing module.
     Configured instance of the storage.
    • DRI manage several repositories.
     Repositories and their modules are configured in DRI-Engine
    configuration.
    • Provided default tool hibernate-based.
     Input: Standar hibernate xml schemas.
     Ouput: standar hibernate repository.
    Location, Meeting title, dd.mm.yyyy 5

    View full-size slide

  6. Datamodel - Storage
    – DRISchema
     Datamodel schema
    • Entities
    • Attribute names.
    • Attribute Types.
    – DRINode
     Id: entityName+number
     Properties
    – DRIDigitalAsset
     Id:
     Type of asset.
     Size
     Digital Content
    Stellenbosch, EPIKH Workshop , 10.06.2011 6
    [
    {
    "entity":"Cliente", ← Entity Name
    «name":"String", ← String property
    "setPhone":"Set", ← Set of property
    "direccion":"Dirección", ← relation with other entity
    "setSubject":"Subject", ← relation with a set of
    entities
    "picture":"DigitalAsset", ← DRIDigitalAsset Property
    "age":"int" ← int property
    },
    → Other entity←
    {
    "entity":"Pedido",
    "property1" :"float",

    }

    ]
    DRINode node =
    new DRINode("es.cetaciemat.dri_test.datamodel.Cliente");
    node.setProperty(“name”,”Paco”);
    node.setProperty(“age”,”45”);
    node.setProperty(“picture”,new DigitalAsset("image/jpe"));
    • Storage Data Transfer Objects.

    View full-size slide

  7. Datamodel - Browsing
    – DRITreeModel
     Datamodel schema
    • Entities
    • Attribute names.
    • Attribute Types.
    – DRITreeNode
     Id
     Parent id
     Display Name
    – Query:
     Dialect
     Query (String)
    – QueryResult
     Array DRINodes
    Stellenbosch, EPIKH Workshop , 10.06.2011 7
    [
    • Browsing Data Transfer Objects.

    View full-size slide

  8. Browser Module
    • Any modules which follow the browsing API.
    • Communicated with DTOs
    – DRITreeNode Object
    – Query Object
    • Feeding with repository
    configuration.
    • Several dialect support.
    • Provided DRIBrowsingDefault.
    – Hibernate browsing.
    – HQL and SQL dialect.
    Stellenbosch, EPIKH Workshop , 10.06.2011 8
    Browsing
    XML
    REPO
    ENGINE
    DRITree
    Node
    Query

    View full-size slide

  9. Storage Module
    • Any modules which follow the Storage API.
    – Digital data and metadata follow different ways
    • Metadata and digital asset store operations.
    • Datamodel configured from the repository.
    • Receives DTOs
    • DRIHibernateStorage
    – Hibernate persistent operation.
    – Different digital asset drivers.
     Storage Elements.
     FTP servers.
     File System.
     On-memory
    Stellenbosch, EPIKH Workshop , 10.06.2011 9
    Storage
    ENGINE
    DB REPO
    DTOs
    Persistent
    format

    View full-size slide

  10. DRI-Engine
    • Engine of the platform
    – Access to the modules.
     Communicate browsing and storage module.
    – Management of the repositories.
     Internal security.
     Logging.
    Stellenbosch, EPIKH Workshop , 10.06.2011 10
    Storage Browsing
    ENGINE
    Conf
    DTOs

    View full-size slide

  11. API as Library
    • DRI can be integrated as library.
    – Configured by environment variables
     Different repositories
    • Storage Module
    • Browsing Module
    Stellenbosch, EPIKH Workshop , 10.06.2011 11
    DRINode node = new DRINode("es.cetaciemat.dri_test.datamodel.Cliente");
    node.setProperty(“name”,”Paco”);
    node.setProperty(“age”,”45”);
    node.setProperty(“picture”,”image/jpeg”);
    Serializable id = DRIEngineFactory.getInstance().saveNode(session, repoId, node);
    DRI
    byte[] content = file.getBytes();
    DRIEngineFactory.getInstance().
    uploadDigitalContent(session, repoId, new DRIDigitalAsset("mime", 1), content, 0);
    DRI
    Medadata Digital Asset
    DB SE

    View full-size slide

  12. Stellenbosch, EPIKH Workshop , 10.06.2011 12
    API Public Methods (1)
    • startSession
    – DRISession session = DRIEngineFactory.getInstance().startSession();
    • endSession
    – DRIEngineFactory.getInstance().endSession(session);
    • getEntitySchema
    – DRIEntitySchema schema = DRIEngineFactory.getInstance().getEntitySchema(session, repoId, entityName);
    • getRepositorySchema
    – Collection schema = DRIEngineFactory.getInstance().getRepositorySchema(session,repoId);
    • getRepositories
    – DRIEngineFactory.getInstance().getRepositories(session);
    • getTreeModel
    – DRIEngineFactory.getInstance().getTreeNames(session, repoId);
    • getTreeNode
    – DRITreeModel treeNames = DRIEngineFactory.getInstance(). getTree(session,repoId,treeName);

    View full-size slide

  13. Stellenbosch, EPIKH Workshop , 10.06.2011 13
    API Public Methods(2)
    • saveNode
    – Serializable id = DRIEngineFactory.getInstance().saveNode(session, repoId, node);
    • loadNode
    – DriNode node=
    DRIEngineFactory.getInstance().loadNode(session,repoId,"es.cetaciemat.dri_test.datamodel.Cliente", 0, true);
    • removeNode
    – DriNode node=
    DRIEngineFactory.getInstance().removeNode(session,repoId,"es.cetaciemat.dri_test.datamodel.Cliente", 0);
    • uploadDigitalContent
    – DRIEngineFactory.getInstance().uploadDigitalContent(session, repoId,new DRIDigitalAsset("mime", 1),content,0);
    • downloadDigitalContet
    – DRIBinaryResponse binary = DRIEngineFactory.getInstance().downloadDigitalContent(session, repoId, da, 0, 0);
    • executeQuery
    – DRIQueryResult result=executeQuery(session,repiId,query);

    View full-size slide

  14. API as Restful access
    • Web Service REST-Based (Representational State Transfer)
    • Resources Access with HTTP URIs.
    – URIs identify the operation.
    – PUT, GET, POST, DELETE.
    • Feeding with JSON object.
    • Ej: http://dri/repositories/imed/entities/patient

    Head parameters.
    Ex: session token: ljlk4j4444hiua998as

    Body in JSON
    Ex: { patientName: “Pepito”,
    patientAge: 65 }
    Stellenbosch, EPIKH Workshop , 10.06.2011 14
    REST-WS
    JSON
    DRI-ENGINE
    DRINode

    View full-size slide

  15. Stellenbosch, EPIKH Workshop , 10.06.2011 15
    Https access(1)
    • startSession. PUT
    – http://host/dri-restws/resources/session  response object with http code which contains the session token
    • endSession. DELETE
    – http://host/dri-restws/resources/session  response object with http code
    • saveNode. PUT
    – http://host/dri-restws/resources/repositories/ {repoID}/entities/{entity}  response object with http code and ID
     Head:
    • session token
    • "Content-Type", "application/json”
     Body: Node in JSON format
    • loadNode. GET
    – http://host/dri-restws/resources/repositories/ {repoID}/entities/{entity}/ID  response object with http code and
    node in JSON format
     Head: session token
    • removeNode. DELETE
    – http://host/dri-restws/resources/repositories/ {repoID}/entities/{entity} /ID  response object with http code
     Head: session token
    • executeQuery. GET
    – http://host/dri-restws/resources/repositories/ {repoID}/query/{queryID} ?index=0&size=10  response object with
    http code and list of nodes in JSON format
     Head: session token
     Queryid id the identification of the predefined queries in the repository

    View full-size slide

  16. Stellenbosch, EPIKH Workshop , 10.06.2011 16
    Https access(2)
    • uploadDigitalContent. PUT
    – http://host/dri-restws/resources/repositories/ {repoID}/dc/{digAssetID}  response object with http code
     Head:
    • session token
    • "Content-Type", "image/jpeg"
    • "Content-Transfer-Encoding", "binary"
     Body: Digital Content, byte array
    • downloadDigitalContent. GET
    – http://host/dri-restws/resources/repositories/ {repoID}/dc/{digAssetIdID}  response object with http code and
    Response Object with the byte array
     Head: session token
    • executeQuery. GET
    – http://host/dri-restws/resources/repositories/ {repoID}/query/{queryID} ?index=0&size=10  response object with
    http code and list of nodes in JSON format
     Head: session token
     Queryid id the identification of the predefined queries in the repository

    View full-size slide

  17. DRI-JOB SERVICE
    Location, Meeting title, dd.mm.yyyy 17
    • Submit jobs from
    personal computers.
    – Compose mode.
    – Your own scripts.
    • Monitoring.
    – Follow de job.
    – Get output
    • Drivers.
    – Local.
     Background Forks.
    – Glite
     Type
     Executable
     Dependencies
     Parameters
     Output
     Extra Parameters
    DRI-JOB
    SERVICE
    Local glite
    JobType=Normal
    Executable=hostname
    Dependencies=[lib,lib2,li4]
    Parameters
    Output=hostname.err,hostname.output

    View full-size slide

  18. Summary
    • Direct access from the client.
    – Access to the library.
    – Http access.
    • Store and manage medadata
    and digital asset.
    – Separate Management
     Metadata
     Digital asset.
    – Digital Asset linked to medatada
    • We are working in
    – DRI-JOB SERVICE
    – AMGA / SE modules.
    • You can integrate your own modules
    Location, Meeting title, dd.mm.yyyy 18
    WEB
    DRI
    DB SE
    http

    View full-size slide

  19. Stellenbosch, EPIKH Workshop , 10.06.2011 19
    References
    • Contact to get the tutorials:
    – Architecture of Applications Departament
    [email protected]
    – Jorge Sevilla Cedillo
    [email protected]
    – www.ceta-ciemat.es
    • Twitter
    – cetauaa, follow as.

    View full-size slide

  20. 20
    Thank you for your kind attention !
    Any questions ?
    Stellenbosch, EPIKH Workshop , 10.06.2011

    View full-size slide