LHCbPR V2

Talk at 5th LHCb Computing Workshop

Transcript

1. LHCbPR V2 Sasha Mazurov, Amine Ben Hammou, Ben Couturier 5th LHCb Computing Workshop 2015-05-19
2. Content • Objectives – LHCbPR V1 Experience – Why V2? • New architecture and tools • RESTful service • Web frontend 5th LHCb Computing Workshop 2 2015-05-19
3. LHCbPR • “LHCb Performance and Regression(PR) is a service designed to record important measures about releases of the version of the LHCb software” • Current version (V1): https://lhcb- pr.web.cern.ch/lhcb-pr/ 2015-05-19 5th LHCb Computing Workshop 3
4. LHCbPR V1 experience (1) Web frontend Main form for jobs searching: lack of dynamic filtering for options, platforms, ... 5th LHCb Computing Workshop 4 2015-05-19
5. LHCbPR V1 experience (2) 5th LHCb Computing Workshop 5 Admin backend Automatic admin site scaffolding is not usable for creating a new job description – too many unobvious steps. A wizard for such task is neede to be written 2015-05-19
6. LHCbPR V1 experience (3) 5th LHCb Computing Workshop 6 New handlers and analysis development • Lack of python and javascript libraries for common LHCbPR programming tasks. A lot of copy-paste code. • Unreadable raw sql in the source code. To get a readable code we need to use standard Django database model classes 2015-05-19
7. Why V2? • First version of LHCbPR is a good program for understanding the real needs for regression analysis. • We have collected a list of features that hard to implement in V1 (thresholds, attributes management, separate versions and applications,…). • We need to have a common javascript libraries and components specific for LHCbPR that can be shared among developers (now we have a spaghetti code) • Decouple data access application and presentation sites. • Avoid using raw SQL. 5th LHCb Computing Workshop 7 2015-05-19
8. New database entities and changes • Versions table – separate application and its version entities. Has timestamp field that is inserted from its first job data, so we can arrange releases’ and nightly builds’ versions. • Nightly slots table – store name of nightly slot (“lhcb-head”, “lhcb- gaudi-head”,…). Versions table can have a link to this table. • Thresholds table for attributes – we can check attributes ranges • Attributes’ group table – attribute can belong to several groups We have a tool that import data from V1 database. 5th LHCb Computing Workshop 8 2015-05-19
9. New architecture • LHCbPR as a RESTful data service: – Separate data access and data representation. Interoperate using JSON format – Web site is only one of the clients of the service – Easy to write command line scripts (for nightlies, reports,…) 9 2015-05-19 5th LHCb Computing Workshop
10. Tools • RESTful service – Django 1.8 with rest framework application. • Web site – AngularJS JavaScript MVC framework – Bootstrap based CSS framework – No backend 10 2015-05-19 5th LHCb Computing Workshop
11. REST API 11 http://amazurov.web.cern.ch/amazurov/lhcbpr-api - outputs all possible service endpoints 2015-05-19 5th LHCb Computing Workshop
12. REST API features • We can use it for all CRUD operations (create, read, update, delete) • Can output arbitrary JSON (examples later) • Authentication plugins: e.g. shibboleth for web and tokens for console applications • Output pagination, throttling and requests’ cache allow to avoid service overloading. 5th LHCb Computing Workshop 12 2015-05-19
13. Examples • Jobs list - /jobs • Job details - /jobs/178 – We can expand any object’s relationships. • Nested requests - /jobs/178/results/ – Output job attributes’ values • Output filtering - /compare/?ids=32,33&contains=Event – Show and group attributes of jobs with id 32 and 32. Attribute name should contain “Event” string 5th LHCb Computing Workshop 13 2015-05-19
14. Web Frontend 5th LHCb Computing Workshop 14 Only http requests from browser to REST server, no backend behind http://amazurov.web.cern.ch/amazurov/lhcbpr-dev - DEMO 2015-05-19
15. Reusable Web Components 15 = <div search-jobs="search-jobs" on-found="onJobsFound(searchParams)”> </div> onJobsFound – client’s custom callback function that get search form results: ids of application, options, Versions More components in development 2015-05-19 5th LHCb Computing Workshop
16. Command line / Scripts 16 Can be easily parsed with standard python libraries $> curl http://amazurov.web.cern.ch/amazurov/lhcbpr-api/jobs/ 2015-05-19 5th LHCb Computing Workshop
17. For Developers • We have prepared a set of reusable JavaScript components that simplifies work with REST service and introduce common UI elements for LHCbPR • Well organized modules’ directory structure 5th LHCb Computing Workshop 17 2015-05-19
18. Modules 5th LHCb Computing Workshop 18 • Controllers are written in JavaScript • Views use Jedi template engine • Modules support lazy JavaScript loading, so only required libraries are used in modules. 2015-05-19
19. Summary • API – Updated database schema for proper support of application versions, build slots, thresholds, attribute groups. Import script from V1 database – Ready for production RESTful service that can be used from any application (currently in readonly mode) – https://github.com/LHCbDev/lhcbpr-api • Frontend – Fixed a structure of modules in web frontend – Prepared a set of web components and services. – Compare jobs’ results module. Framework is ready for developing (migration) of the rest of analysis modules. – https://github.com/LHCbDev/lhcbpr-www 5th LHCb Computing Workshop 19 2015-05-19