SHARE Notification Service Development

An update from the Center for Open Science for the SHARE Fall Meeting 2014.

Transcript

1. SHARE Notification Service Development Update from the Center for Open Science
2. Center for Open Science
3. Community Metascience Infrastructure
4. http://osf.io
5. Modular Reliable Flexible Scalable
6. COS-SHARE Development Team Brian Geiger Chris Seto Coral Sheldon-Hess Erin Braswell Fabian Von Feilitzsch
7. COS-SHARE Developer Alumni Casey Rollins Erica Baranski Faye Huynh Lauren Revere Ginny Huang Michelle Yao Peter Fan Saman Ehsan Xander Herrick
8. Gather Notify Providers Consumers
9. How are we getting data? Application Programming Interface
10. GET https://frogworld. com/api/frogs Request Response [ { name: Kermit, color: green, type: felt }, { name: Travis, color: green, type: tree } ]
11. POST https://osf.io/api/share Request Response { title: Easy Being Green?, contributors: [ Kermit, Travis ], source: frogworld, id: 10.100/frogworld.102 } Success
15. Resource 1 Resource 2 { title: Easy Being Green?, contributors: [ Kermit, Travis ], source: frogworld, id: 10.100/frogworld.102 } { title: Easy Being Green?, contributors: [ Kermit, Travis ], origin: frogworld, id: 10.100/frogworld.102, description: Exploring greenness. }
16. Resource 1 Resource 2 { title: Easy Being Green?, contributors: [ Kermit, Travis ], source: frogworld, id: 10.100/frogworld.102 } { name: Easy Being Green?, contributors: [ Frog Scientists Intl, Amphibians United, ], origin: frogworld, doi: 10.100/frogworld.102 }
17. Resource 1 Resource 2 { title: Easy Being Green?, contributors: [ Kermit, Travis ], source: frogworld, id: 10.100/frogworld.102 } { title: Easy Being Green?, source: frogworld, id: 10.100/frogworld.102 }
18. Resource 1 Resource 2 { title: Easy Being Green?, contributors: [ Kermit, Travis ], source: frogworld, id: 10.100/frogworld.102 } { title: Easy Being Green?, contributors: [ No Contributors ], source: frogworld, id: 10.100/frogworld.102 }
19. Gather Notify Providers Consumers
20. Request GET http://repository.stcloudstate.edu/do/oai/? verb=ListRecords &metadataPrefix=oai_dc &from=2014-10-02T00:00:00Z
21. Request Response GET http://repository.stcloudstate.edu/do/oai/? verb=ListRecords &metadataPrefix=oai_dc &from=2014-10-02T00:00:00Z <record> <date> … </date> <title> … </title> <contributors> <name>...</name> <name>...</name> </contributors> <description> … </description> <id>...</id> </record>
22. Provider: stcloudstate
23. Provider: stcloudstate 11:05
24. Provider: stcloudstate 11:05 GET stcloud.edu/api/
25. Provider: stcloudstate 11:05 GET stcloud.edu/api/ GET plos.org/api/ GET columbia.edu/api/ GET mit.edu/api/ GET vtech.edu/api/ GET dataONE.org/api/ Queue
26. Provider: stcloudstate 11:05 GET stcloud.edu/api/ GET plos.org/api/ GET columbia.edu/api/ GET mit.edu/api/ GET vtech.edu/api/ GET dataONE.org/api/ Queue worker server image by RRZEicons
27. Provider: stcloudstate 11:05 GET stcloud.edu/api/ GET plos.org/api/ GET columbia.edu/api/ GET mit.edu/api/ GET vtech.edu/api/ GET dataONE.org/api/ Queue worker worker worker worker server image by RRZEicons
28. GET stcloud.edu/api/ Archive Response Archive Data Archive Request
29. GET stcloud.edu/api/ Archive Response Archive Data Normalize Archive Request
30. <record> <header> <identifier>oai:repository.stcloudstate.edu:comm_facpubs-1001</identifier> <datestamp>2014-10-01T17:54:11Z</datestamp> </header> <metadata> <oai_dc:dc> <dc:title>The Press as an Elite Power Group in Japan</dc:title> <dc:creator>Akhavan-Majid, Roya</dc:creator> <dc:description> A lengthy field study in Japan using interviews and other sources and focusing on the nation's five leading national newspapers and Tokyo's major television stations finds evidence of much overlap between industry and the news media, through interlocking directorships and social club memberships </dc:description> <dc:date>1990-12-01T08:00:00Z</dc:date> <dc:identifier>http://repository.stcloudstate.edu/comm_facpubs/3</dc:identifier> <dc:source>Mass Communication Faculty Publications</dc:source> <dc:publisher>the Repository at St. Cloud State</dc:publisher> <dc:subject>Mass Communication</dc:subject> </oai_dc:dc> </metadata> </record>
31. { "source": "stcloud", "description": "A lengthy field study in Japan using interviews and other sources and focusing on the nation's five leading national newspapers and Tokyo's major television stations finds evidence of much overlap between industry and the news media, through interlocking directorships and social club memberships,", "contributors": { "given": "Roya", "family": "Akhavan-Majid", "middle": "", "email": "" }, "title": "The Press as an Elite Power Group in Japan", "dateCreated": "1970-01-01T00:00:00", "dateUpdated": "2014-10-01T17:54:11+00:00", "tags": [ "mass communication"], "id": { "url": "http://repository.stcloudstate.edu/comm_facpubs/3", "serviceID": "oai:repository.stcloudstate.edu:comm_facpubs-1001", "doi": "" } }
32. GET stcloud.edu/api/ Archive Response Archive Data Normalize Send to OSF Archive Request
33. Gather Notify Providers Consumers
34. Notify Consumers
35. OSF Metadata OSF Metadata and Search APIs
36. Notify Consumers RSS PubSubHubbub ResourceSync OSF Search Engine OSF API
37. RSS
38. /rss/?q=open AND science AND source:scitech
39. PubSubHubbub
40. ResourceSync
41. /resync/resourcelist/?q=open AND science AND source:scitech
45. Inside the Notification Service 3:30 in this room
46. Recap of lessons learned
47. Today…. Over 20 content providers Robust backend providing: ➢ archive of all research release events ➢ monitoring and extensive logging ➢ recovery from service interruptions Solid foundation to enable building of other services like curation and networking of research projects
48. More institutional repositories Tests with complementary types of content: Providers able to push content to SHARE NS Coming soon...
49. Today Early 2015 Fall 2015 Beta release Full release 20 + service providers Robust backend Output formats RSS ResourceSync Search More service providers Complementary data Refined metadata schema and push of content
50. Try it: http://share-dev.osf.io Follow on GitHub: https://github.com/ CenterForOpenScience/SHARE See more details: https://osf.io/29ypx/wiki/
51. [ Intentionally Left Blank ]
52. Inside the Notification Service
53. Elasticsearch RabbitMQ MongoDB/ TokuMX Celery Workers OSF App Servers ScrAPI Load Balancers Caching Proxies Raw User Provider Provider Provider Fluentd Elasticsearch Kibana User User
54. • Reliable ◦ Messages persist ◦ Redundant message storage • Scalable ◦ easily clustered • Supported ◦ Most used message broker RabbitMQ
55. Celery • Asynchronous job execution • Scalable ◦ Allows for any number of worker machines • Fault resistant • Large community ◦ Tight integration with RabbitMQ
56. MongoDB/TokuMX • Fast • Transaction-based • Improve document storage • Document-level locking
57. Elasticsearch • Flexible ◦ Schema-less • Built to be horizontally extensible • Robust API
58. Fluentd • Most robust log aggregation system • Converts log messages to JSON for easy processing • Emphasizes simplicity
59. Sentry • No exceptions uncaught • Real-time logging • Easy triage
60. Kibana • Nice visualizations • Easy setup • Simple to use • Stats at a glance • In depth progress monitoring
62. Harvesters Gathering metadata from providers
64. RSS Feed
68. SHARE in the world
69. SHARE on GitHub https://github.com/CenterForOpenScience/SHARE
70. Ways to access SHARE
71. Search API
74. Knockout - Observables
77. Knockout - Functions
80. HTML