Exporting NDAR Data to the Cloud - Robert Arrington

Advancing Autism Discovery Workshop, Robert Arrington

Transcript

1. EXPORTING NDAR DATA TO THE CLOUD Robert Arrington
2. Intro  NDAR is rolling out 2 new ways to download data  Databases hosted by you in your cloud  Databases hosted by NDAR  Advantages/Disadvantages of each method
3. Availability Zone #1 NDAR Create Package “ndar_pipe” Security Group Security Group Root Volume Data Volume S3 Bucket EC2 Instance RDS Pusher MySQL DB Instance Encrypted Object Hosted by You
4. Final Product “ndar_pipe” Security Group MySQL DB Instance Data Volume Contains genomic and/or imaging data Contains genomic and/or imaging data - The remaining data volume contains the data package you would receive through the previous download manager - This can be attached to a new EC2 instance (One running NITRC, BioLinux, or Galaxy) - Data descriptions are contained in comma separated text files - The MySQL database is pre-loaded with all of the tabular data. - This section mirrors NDAR tables. - Use for direct query
5. Advantages/Disadvantages Feature Advantage Disadvantage User Hosted Easy integration in to computation pipelines Account management and hosting costs (10 TB = $800/month) Block Storage Ease of use/speed Slight cost increase over S3* Read/Write Database Stored Procedures Full Package Download All data available once complete 1 TB limit * Data can be pushed to your own S3 buckets.
6. Hosted by NDAR Table Data (read only) User Package Data S3 Package Data (Read Only) User Create Package NDAR Amazon Cloud Permission Grant to NDAR data
7. Final Product - The user is receives a login/password and database endpoint. - Within the database, S3 objects are referenced. (full URI address) - The user will have read access on all database items - Database will expire after 30 days. (Can be restarted)
8. Advantages/Disadvantages Feature Advantage Disadvantage Download using URI Simple web access to objects More resilient coding needed for programs Download on demand Pull objects as you need them Download required when objects accessed Static Objects Download originals as needed Local copy will be needed to process files Database expires and Read-only Ease of AWS account management Need to refresh database, limits on DB computation No hosting cost Data is in NDAR’s AWS account Downloads are billed to your AWS account Account Number Access Access granted by Account Number, not key/secret key Account Number required to be held by NDAR