Hilary James Oliver: The Cylc Suite Engine

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Hilary James Oliver:
The Cylc Suite Engine
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2013 - Sunday, 08 Sep 2013 - Track 2
http://nz.pycon.org/

**Audience level**

Intermediate

**Description**

The Cylc Suite Engine schedules complex distributed suites of cycling tasks. Developed for weather forecasting applications at NIWA, cylc has since been adopted by the UK Met Office and others, and was a finalist in the 2012 NZ Open Source Awards. Lead developer Hilary Oliver will talk about cylc and the many advantages that the Python programming language brought to the project.

**Abstract**

The Cylc Suite Engine is a metascheduler or workflow engine that makes it relatively easy to construct, monitor, and control large complex distributed suites of interdependent cycling tasks. Cylc was developed, entirely in Python, to automate modern environmental forecasting systems in an HPC environment at NIWA in New Zealand. Since its release as an Open Source project on GitHub in 2011 cylc has been adopted by the UK Met Office to control their large and varied forecasting systems; it is also used by groups at the Marine Meteorology Division of the Naval Research Laboratory in the US, the Max Planck Institute for Meteorology in Germany, and others; and it was a finalist in the 2012 NZ Open Source Awards. Lead developer Hilary Oliver will give an overview of what cylc does and how it works, and the particular advantages gained by our decision to use the Python programming language.

**YouTube**

http://www.youtube.com/watch?v=Mv-hBTcncKM

Transcript

1. The Cylc Suite Engine and Python Kiwi PyCon, 8 September 2013 Hilary James Oliver hilary.oliver@niwa.co.nz
2. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
3. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
4. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
5. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
6. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
7. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
8. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
9. NIWA Forecasting Credit: Dave Allen, NIWA
10. NIWA Forecasting Credit: Alan Blacklock, NIWA
11. NIWA Forecasting Credit: Dave Allen, NIWA
12. NIWA Forecasting Credit: Dave Allen, NIWA
13. NIWA Forecasting Credit: Dave Allen, NIWA
14. NIWA Forecasting Credit: Dave Allen, NIWA
15. NIWA Forecasting Credit: Dave Allen, NIWA
16. NIWA Forecasting Supercomputer: IBM p575/p6 (3456 * 4.6Ghz cores)
17. NIWA Forecasting http://ecoconnect.niwa.co.nz
18. NIWA Forecasting http://weather.niwa.co.nz
19. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
20. CYLC - Communist Youth League of China
21. Cylc is a Suite Engine Cylc allows you to construct, run, monitor, and control distributed SUITES of inter-dependent CYCLING (and non-cycling) TASKS.
22. Cylc is a Suite Engine Cylc allows you to construct, run, monitor, and control distributed SUITES of inter-dependent CYCLING (and non-cycling) TASKS. my.suite GetData 2013080806 Model 2013080806 Model 2013080812 PostA 2013080806 PostB 2013080806 GetData 2013080812 PostA 2013080812 Model 2013080800 PostA 2013080800 ColdModel 2013080800 GetData 2013080800 Prep 2013080800
23. Cylc is a Suite Engine Cylc allows you to construct, run, monitor, and control distributed SUITES of inter-dependent CYCLING (and non-cycling) TASKS. Things a suite engine should help with: suite preparation: design editing validation visualization . . .
24. Cylc is a Suite Engine Cylc allows you to construct, run, monitor, and control distributed SUITES of inter-dependent CYCLING (and non-cycling) TASKS. Things a suite engine should help with: suite preparation: design editing validation visualization . . . suite run time: (meta-)scheduling monitoring control failure recovery restart from previous state . . .
25. Cylc is a Suite Engine Cylc allows you to construct, run, monitor, and control distributed SUITES of inter-dependent CYCLING (and non-cycling) TASKS.
26. ecox OPS_AIRS 2012070406 VAR_AnalysePF 2012070406 UM_nzlam 2012070406 nzlam_var_monitoring 2012070406 OPS_AIRS 2012070412 VAR_AnalysePF 2012070412 UM_nzlam 2012070412 nzlam_var_monitoring 2012070412 OPS_AIRS 2012070418 VAR_AnalysePF 2012070418 UM_nzlam 2012070418 nzlam_var_monitoring 2012070418 OPS_AIRS 2012070500 VAR_AnalysePF 2012070500 UM_nzlam 2012070500 nzlam_var_monitoring 2012070500 OPS_ATOVS 2012070406 OPS_ATOVS 2012070412 OPS_ATOVS 2012070418 OPS_ATOVS 2012070500 OPS_AircraftSonde 2012070406 OPS_AircraftSonde 2012070412 OPS_AircraftSonde 2012070418 OPS_AircraftSonde 2012070500 OPS_GPSRO 2012070406 OPS_GPSRO 2012070412 OPS_GPSRO 2012070418 OPS_GPSRO 2012070500 OPS_IASI 2012070406 OPS_IASI 2012070412 OPS_IASI 2012070418 OPS_IASI 2012070500 OPS_Satwind 2012070406 OPS_Satwind 2012070412 OPS_Satwind 2012070418 OPS_Satwind 2012070500 OPS_SurfaceScatt 2012070406 OPS_SurfaceScatt 2012070412 OPS_SurfaceScatt 2012070418 OPS_SurfaceScatt 2012070500 UM_NZtoNZ_recon 2012070500 ukmodata_housekeeping 2012070500 get_um_output 2012070500 VAR_ConfigureLS 2012070412 get_um_output 2012070406 ukmodata_housekeeping 2012070406 VAR_ConfigureLS 2012070418 get_um_output 2012070412 ukmodata_housekeeping 2012070412 nzlam_conv_met 2012070406 nzlam_conv_sls 2012070406 nzlam_conv_tn 2012070406 nzlam_conv_escape 2012070406 cycle_check 2012070406 ukmodata_housekeeping 2012070418 VAR_ConfigureLS 2012070500 get_um_output 2012070418 nzlam_conv_tn 2012070412 nzlam_conv_escape 2012070412 nzlam_conv_sls 2012070412 nzlam_conv_met 2012070412 cycle_check 2012070412 cycle_check 2012070418 nzlam_conv_met 2012070418 nzlam_conv_escape 2012070418 nzlam_conv_tn 2012070418 nzlam_conv_sls 2012070418 VAR_ConfigureLS 2012070406 globalwave 2012070500 globalwave_ncl 2012070500 globalwave_products 2012070500 nzwave 2012070406 nztide 2012070406 ocean_phosphate 2012070406 ocean_tauranga 2012070406 nzwave_ncl 2012070406 nzwave 2012070412 nzwave 2012070418 nzwave 2012070500 nztide_ncl 2012070406 nzwave_products 2012070406 nztide 2012070418 ocean_tauranga 2012070418 ocean_tangaroa 2012070418 ocean_phosphate 2012070418 nzwave_ncl 2012070418 cycle_check 2012070500 cycle_check_transfer 2012070412 nztide_ncl 2012070418 nzwave_products 2012070418 cycle_check_transfer 2012070500 ricom 2012070406 ricom_ncl 2012070406 ricom 2012070418 ricom_products 2012070406 ricom_ncl 2012070418 ricom_products 2012070418 topnet 2012070406 topnet_ncl 2012070406 topnet 2012070409 topnet 2012070412 topnet 2012070415 topnet 2012070418 topnet 2012070421 topnet 2012070500 topnet 2012070503 topnet_input 2012070409 topnet_products 2012070406 topnet_ncl 2012070409 topnet_input 2012070412 topnet_ncl 2012070412 topnet_input 2012070415 topnet_ncl 2012070415 topnet_input 2012070418 topnet_ncl 2012070418 topnet_input 2012070421 topnet_ncl 2012070421 topnet_input 2012070500 topnet_ncl 2012070500 topnet_input 2012070503 topnet_ncl 2012070503 topnet_control 2012070409 topnet_products 2012070409 topnet_control 2012070412 topnet_products 2012070412 topnet_control 2012070415 topnet_products 2012070415 topnet_control 2012070418 topnet_products 2012070418 topnet_control 2012070421 topnet_products 2012070421 topnet_control 2012070500 topnet_products 2012070500 topnet_control 2012070503 topnet_products 2012070503 topnet_input 2012070406 topnet_control 2012070406 web_transfer_monitoring 2012070406 web_transfer_monitoring 2012070412 web_transfer_monitoring 2012070418 web_transfer_monitoring 2012070500 nzlam_conv_sls 2012070500 nzlam_conv_met 2012070500 nzlam_conv_tn 2012070500 nzlam_conv_escape 2012070500 mos_nzlam_apply 2012070406 weather_phosphate 2012070406 weather_tauranga 2012070406 kvh_stream_nzlam 2012070406 nzlam_verification 2012070406 psa_risk_nzlam_field 2012070406 nzlam_ncl 2012070406 tn_llclean 2012070406 cycle_check_transfer 2012070406 tn_llclean 2012070412 nzlam_verification 2012070412 weather_tangaroa 2012070412 kvh_stream_nzlam 2012070412 mos_nzlam_apply 2012070418 weather_phosphate 2012070418 nzlam_ncl 2012070418 weather_tauranga 2012070418 nzlam_verification 2012070418 psa_risk_nzlam_field 2012070418 kvh_stream_nzlam 2012070418 tn_llclean 2012070418 cycle_check_transfer 2012070418 nzlam_verification 2012070500 kvh_stream_nzlam 2012070500 tn_llclean 2012070500 UM_nzlam_coldstart 2012070406 jobsubmission_housekeeping 2012070406 jobsubmission_housekeeping 2012070412 jobsubmission_housekeeping 2012070418 jobsubmission_housekeeping 2012070500 ecan_rain_obs 2012070406 ecan_rain_obs 2012070412 ecan_rain_obs 2012070418 ecan_rain_obs 2012070500 ext_check_bgerr_ukmet 2012070406 ext_get_bgerr_ukmet 2012070406 unpack_bgerr 2012070406 ext_check_bgerr_ukmet 2012070412 ext_get_bgerr_ukmet 2012070412 unpack_bgerr 2012070412 ext_check_bgerr_ukmet 2012070418 ext_get_bgerr_ukmet 2012070418 unpack_bgerr 2012070418 ext_check_bgerr_ukmet 2012070500 ext_get_bgerr_ukmet 2012070500 unpack_bgerr 2012070500 ext_check_frames_ukmet 2012070406 ext_get_frames_ukmet 2012070406 unpack_frames 2012070406 make_alabc 2012070406 ext_check_frames_ukmet 2012070412 ext_get_frames_ukmet 2012070412 unpack_frames 2012070412 make_alabc 2012070412 ext_check_frames_ukmet 2012070418 ext_get_frames_ukmet 2012070418 unpack_frames 2012070418 make_alabc 2012070418 ext_check_frames_ukmet 2012070500 ext_get_frames_ukmet 2012070500 unpack_frames 2012070500 make_alabc 2012070500 ext_check_globalice_ukmet 2012070500 ext_get_globalice_ukmet 2012070500 globalnwp_conv_ice 2012070500 sls_globalice_copy 2012070500 ext_check_globalwind_ukmet 2012070500 ext_get_globalwind_ukmet 2012070500 globalnwp_conv_wind 2012070500 psa_risk_global_field 2012070500 sls_globalwind_copy 2012070500 psa_risk_global_site 2012070500 globalnwp_ncl 2012070500 kvh_stream_global 2012070500 ext_check_obstore_ukmet 2012070406 ext_get_obstore_ukmet 2012070406 unpack_obstores 2012070406 ext_check_obstore_ukmet 2012070412 ext_get_obstore_ukmet 2012070412 unpack_obstores 2012070412 ext_check_obstore_ukmet 2012070418 ext_get_obstore_ukmet 2012070418 unpack_obstores 2012070418 ext_check_obstore_ukmet 2012070500 ext_get_obstore_ukmet 2012070500 unpack_obstores 2012070500 ext_check_sst_ukmet 2012070406 ext_get_sst_ukmet 2012070406 ext_create_streamflow 2012070406 ext_get_streamq 2012070406 ext_create_streamflow 2012070409 ext_get_streamq 2012070409 ext_create_streamflow 2012070412 ext_get_streamq 2012070412 ext_create_streamflow 2012070415 ext_get_streamq 2012070415 ext_create_streamflow 2012070418 ext_get_streamq 2012070418 ext_create_streamflow 2012070421 ext_get_streamq 2012070421 ext_create_streamflow 2012070500 ext_get_streamq 2012070500 ext_create_streamflow 2012070503 ext_get_streamq 2012070503 ext_get_clidbdata 2012070406 mos_nzlam_coeffs 2012070406 nzlam_products 2012070406 psa_risk_nzlam_site 2012070406 mos_nzlam_vp_curr 2012070406 ext_get_clidbdata 2012070418 mos_nzlam_coeffs 2012070418 mos_nzlam_vp_curr 2012070418 nzlam_products 2012070418 psa_risk_nzlam_site 2012070418 psa_risk_map_global 2012070500 psa_risk_global_site_web 2012070500 globalnwp_products 2012070500 web_transfer_verification 2012070406 psa_risk_map_nzlam 2012070406 web_transfer_verification 2012070412 web_transfer_verification 2012070418 psa_risk_map_nzlam 2012070418 web_transfer_verification 2012070500 globalnwp_housekeeping 2012070500 clean_tmp 2012070500 globalwave_housekeeping 2012070500 mos_housekeeping 2012070406 mos_housekeeping 2012070418 nzlam_housekeeping 2012070406 psa_risk_nzlam_site_web 2012070406 mos_nzlam_vp_5day 2012070406 mos_nzlam_vp_5day 2012070418 nzlam_housekeeping 2012070418 psa_risk_nzlam_site_web 2012070418 ncep_get 2012070406 ncep_run 2012070406 ncep_upload_thotter 2012070406 ncep_xml 2012070406 ncep_housekeeping 2012070406 ncep_products 2012070406 nztide_products 2012070406 nztide_housekeeping 2012070406 nztide_products 2012070418 nztide_housekeeping 2012070418 nzwave_housekeeping 2012070406 nzwave_housekeeping 2012070418 nzwave_cold 2012070406 ricom_housekeeping 2012070406 ricom_housekeeping 2012070418 ricom_cold 2012070406 topnet_housekeeping 2012070406 topnet_housekeeping 2012070409 topnet_housekeeping 2012070412 topnet_housekeeping 2012070415 topnet_housekeeping 2012070418 topnet_housekeeping 2012070421 topnet_housekeeping 2012070500 topnet_housekeeping 2012070503 topnet_cold 2012070406
27. OPS_AIRS 2012070500 2 OPS_SurfaceScatt 2012070500 ukmodata_housek 201207050 globalwave 2012070500 globalwave_ncl 2012070500 globalwave_products 2012070500 topnet 2012070412 topnet 2012070415 topnet 2012070418 topnet_ncl 2012070412 topnet_input 2012070415 topnet_ncl 2012070415 topnet_input 2012070418 topnet_input 2012070421 topnet_products 2012070412 topnet_control 2012070415 topnet_products 2012070415 topnet_control 2012070418 topnet_control 2012070421 web_transfe 20120 ext_check_globalice_ukmet 2012070500 ext_get_globalice_ukmet 2012070500 globalnwp_conv_ice 2012070500 sls_globalice_copy 2012070500 ext_check_globalwind_ukmet 2012070500 ext_get_globalwind_ukmet 2012070500 globalnwp_conv_wind 2012070500 psa_risk_global_field 2012070500 sls_globalwind_copy 2012070500 psa_risk_global_site 2012070500 globalnwp_ncl 2012070500 kvh_ ext_check_obsto 201207050 ext_get_obsto 20120705 unp ext_get_streamq 2012070415 ext_create_streamflow 2012070418 ext_get_streamq 2012070418 ext_create_streamflow 2012070421 ext_get_streamq 2012070421 ext_create_streamflow 2012070500 psa_risk_map_global 2012070500 psa_risk_g 201 globalnwp_products 2012070500 globalnwp_housekeeping 2012070500 clean_tmp 2012070500 globalwave_housekeeping 2012070500 topnet_housekeeping 2012070409 topnet_housekeeping 2012070412 topnet_housekeeping 2012070415
28. UM_nzlam 2012070412 nzlam_var_ 20120 OPS_AIRS 2012070418 VAR_AnalysePF 2012070418 UM_nzlam 2012070418 nzlam_var_monitoring 2012070418 _AnalysePF 12070500 UM_nzlam 2012070500 itoring 00 OPS_ATOVS 2012070418 OPS_AircraftSonde 2012070418 OPS_GPSRO 2012070418 OPS_GPSRO 2012070500 OPS_IASI 2012070418 OPS_IASI 2012070500 OPS_Satwind 2012070418 OPS_Satwind 2012070500 OPS_SurfaceScatt 2012070418 UM_NZtoNZ_recon 2012070500 get_um_output 2012070500 VAR_ConfigureLS 2012070418 get_um_output 2012070412 ukmodata_housekeeping 2012070412 VAR_ConfigureLS 2012070500 get_um_output 2012070418 nzlam_conv_tn 2012070412 nzlam_conv_escape 2012070412 nzlam_conv_sls 2012070412 nzlam_conv_met 2012070412 cycle_check 2012070412 nzlam_conv_met 2012070418 nzlam_conv_tn 2012070418 nzlam_conv_sls 2012070418 nzwave 2012070412 nzwave 2012070418 ocean_tauranga 2012070418 ocean_tangaroa 2012070418 nzwave_ncl 2012070418 cycle_check_transfer 2012070412 nzwave_products 2012070418 ricom 2012070418 ricom_ncl 2012070418 ricom_products 2012070418 topnet_ncl 2012070418 web_transfe 20120 web_transfer_monitoring 2012070418 nzlam_conv_sls 2012070500 nzlam_conv_met 2012070500 conv_tn 070500 tn_llclean 2012070412 nzlam_verification 2012070412 weather_tangaroa 2012070412 kvh_stream_nzlam 2012070412 mos_nzlam_apply 2012070418 weather_phosphate 2012070418 nzlam_ncl 2012070418 weather_tauranga 2012070418 nzlam_verification 2012070418 psa_risk_nzlam_field 2012070418 kvh_stream_nzlam 2012070418 tn_llclean 2012070418 jobsubmission_housekeeping 2012070412 e unpack_bgerr 2012070418 et ext_get_frames_ukmet 2012070418 unpack_frames 2012070418 make_alabc 2012070418 ext_check_frames_ukmet 2012070500 ext_get_frames_ukmet 2012070500 unpack_frames 2012070500 make_alabc 2012070500 unpack_obstores 2012070418 ext_get_clidbdata 2012070418 mos_nzlam_coeffs 2012070418 mos_nzlam_vp_curr 2012070418 nzlam_products 2012070418 psa_risk_nzlam_site 2012070418 web_transfer_verification 2012070412 web_transfer_verification 2012070418 psa_risk_map_nzlam 2012070418
29. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
30. The Cycling Suite Problem a small suite . . . a b c e d f Time x
31. The Cycling Suite Problem . . . repeats on regular forecast cycles (in real time operation) a b c e d f a b c e d f Time x x
32. The Cycling Suite Problem typically there is also some inter-cycle dependence . . .
33. The Cycling Suite Problem . . . but this is always ignored (by others!) . . .
34. The Cycling Suite Problem . . . which does not matter in real time operation, BUT . . .
35. The Cycling Suite Problem Q: after a delay, can we start the next cycle early? a b c e f Time d ?
36. The Cycling Suite Problem A: in general, NO, if inter-cycle dependence is ignored . . . d a b c e d f a b c e f Time d
37. The Cycling Suite Problem . . . the best we can do is sequential whole cycles
38. The Cycling Suite Problem So: for efficient scheduling after delays (etc.) we need to handle inter- cycle dependence properly.
39. The Cycling Suite Problem What if a, b, and c have inter-cycle dependence:
40. The Cycling Suite Problem What if a, b, and c have inter-cycle dependence:
41. The Cycling Suite Problem Too difficult?!
42. The Cycling Suite Problem Deconstruct graph edges . . . a b T 1 (b.1 triggers off a.1)
43. The Cycling Suite Problem Deconstruct graph edges . . . to get task inputs and outputs a b T 1 task b input: "a.1 succeeded" task a output: "a.1 succeeded"
44. The Cycling Suite Problem Deconstruct graph edges . . . to get task inputs and outputs
45. The Scheduling Algorithm An evolving pool of task proxies interact to match completed outputs with unsatisfied inputs so that correct scheduling emerges naturally at run time.
46. Result Delayed/catch-up job schedules: grey: normal on-time operation coloured: delayed (catching up)
47. Result Delayed/catch-up job schedules: grey: normal on-time operation coloured: delayed (catching up)
48. Result Other benefits of this “task pool” approach to scheduling: flexibility - add or remove tasks from the system at run time adaptability - the system “flows around” failed or delayed tasks
49. Result Other benefits of this “task pool” approach to scheduling: flexibility - add or remove tasks from the system at run time adaptability - the system “flows around” failed or delayed tasks
50. Result Other benefits of this “task pool” approach to scheduling: flexibility - add or remove tasks from the system at run time adaptability - the system “flows around” failed or delayed tasks
51. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
52. A Cylc Hello World Suite A suite with one task . . . [scheduling] [[dependencies]] graph = "hello"
53. A Cylc Hello World Suite . . . that runs HelloWorld.exe . . . [scheduling] [[dependencies]] graph = "hello" [runtime] [[hello]] # run an external program command scripting = HelloWorld.exe
54. A Cylc Hello World Suite . . . that prints “Hello!” . . . [scheduling] [[dependencies]] graph = "hello" [runtime] [[hello]] # an inlined task command scripting = "echo Hello!"
55. A Cylc Hello World Suite . . . that prints a $GREETING . . . [scheduling] [[dependencies]] graph = "hello" [runtime] [[hello]] command scripting = "echo $GREETING" [[[environment]]] GREETING = Hello!
56. A Cylc Hello World Suite . . . that prints a $GREETING on a remote host . . . [scheduling] [[dependencies]] graph = "hello" [runtime] [[hello]] command scripting = "echo $GREETING" [[[environment]]] GREETING = Hello! [[[remote]]] host = bob.niwa.co.nz
57. A Cylc Hello World Suite . . . that prints a $GREETING on a remote host, under SLURM [scheduling] [[dependencies]] graph = "hello" [runtime] [[hello]] command scripting = "echo $GREETING" [[[environment]]] GREETING = Hello! [[[remote]]] host = bob.niwa.co.nz [[[job submission]]] method = slurm
58. Task Triggering [scheduling] [[dependencies]] graph = """ foo => bar & baz => post baz => upload """ bar 1 post 1 baz 1 upload 1 foo 1
59. Task Triggering [scheduling] [[dependencies]] graph = """ foo => bar & baz bar & baz => post baz => upload """ bar 1 post 1 baz 1 upload 1 foo 1
60. Task Triggering [scheduling] [[dependencies]] graph = """ foo => bar foo => baz bar => post baz => post baz => upload """ bar 1 post 1 baz 1 upload 1 foo 1
61. Task Triggering [scheduling] [[dependencies]] graph = """ foo => bar foo => baz # comment bar => post # comment baz => post baz => upload """ bar 1 post 1 baz 1 upload 1 foo 1
62. More Complex Triggering conditional triggers [scheduling] [[dependencies]] graph = """ ( foo:fail | bar:submit ) & baz:start => qux """
63. More Complex Triggering flexible family triggers [scheduling] [[dependencies]] graph = """ family:finish-all & family:succeed-any => foo """
64. Cycling Tasks [scheduling] [[dependencies]] graph = "prep" prep 1
65. Cycling Tasks [scheduling] [[dependencies]] graph = "prep" [[[0,6,12,18]]] graph = """ prep => foo => bar """ foo 2010080806 bar 2010080806 foo 2010080812 bar 2010080812 foo 2010080800 bar 2010080800 prep 1
66. Cycling Tasks [scheduling] [[dependencies]] graph = "prep" [[[0,6,12,18]]] graph = """ prep => foo => bar """ [[[6,18]]] graph = "bar => baz" foo 2010080806 bar 2010080806 baz 2010080806 foo 2010080812 bar 2010080812 foo 2010080800 bar 2010080800 prep 1
67. Cycling Tasks [scheduling] [[dependencies]] graph = "prep" [[[0,6,12,18]]] graph = """ prep & foo[T-6] => foo => bar """ [[[6,18]]] graph = "bar => baz" foo 2010080800 bar 2010080800 foo 2010080806 bar 2010080806 foo 2010080812 baz 2010080806 bar 2010080812 prep 1
68. Automatic Recovery [scheduling] [[dependencies]] graph = """ pre => model model:fail => diagnose => recover model => !diagnose & !recover model | recover => post """ diagnose 1 recover 1 post 1 model 1 pre 1
69. Automatic Recovery ˜[scheduling] ˜ [[dependencies]] ˜ graph = """ "pre => model" ˜ model:fail => diagnose => recover ˜ model => !diagnose & !recover ˜ model | recover => post ˜ """ model 1 pre 1
70. Automatic Recovery ˜[scheduling] ˜ [[dependencies]] ˜ graph = """ ˜ pre => model "model:fail => diagnose => recover" ˜ model => !diagnose & !recover ˜ model | recover => post ˜ """ diagnose 1 recover 1 model 1
71. Automatic Recovery ˜[scheduling] ˜ [[dependencies]] ˜ graph = """ ˜ pre => model ˜ model:fail => diagnose => recover "model => !diagnose & !recover" ˜ model | recover => post ˜ """ diagnose 1 recover 1 model 1
72. Automatic Recovery ˜[scheduling] ˜ [[dependencies]] ˜ graph = """ ˜ pre => model ˜ model:fail => diagnose => recover ˜ model => !diagnose & !recover "model | recover => post" ˜ """ recover 1 post 1 model 1
73. Automatic Recovery [scheduling] [[dependencies]] graph = """ pre => model model:fail => diagnose => recover model => !diagnose & !recover model | recover => post """ diagnose 1 recover 1 post 1 model 1 pre 1
74. Jinja2 Template Engine Support Jinja2 allows us to generate suite definitions programmatically. Great for repetition (graph or runtime), structural changes, . . . title = "simple ensemble" [scheduling] [[dependencies]] graph = """ foo => memb_1 => post_1 => bar """ foo 1 memb_1 1 post_1 1 bar 1
75. Jinja2 Template Engine Support Jinja2 allows us to generate suite definitions programmatically. Great for repetition (graph or runtime), structural changes, . . . #!Jinja2 title = "simple ensemble" {% set N = 4 %} [scheduling] [[dependencies]] graph = """ {%for I in range(0,N) %} foo => memb_{{I}} => \ post_{{I}} => bar {%endfor %} """ foo 1 memb_1 1 post_1 1 bar 1
76. Jinja2 Template Engine Support Jinja2 allows us to generate suite definitions programmatically. Great for repetition (graph or runtime), structural changes, . . . # "cylc view -p pres.jinja2" title = "simple ensemble" [scheduling] [[dependencies]] graph = """ foo => memb_0 => post_0 => bar foo => memb_1 => post_1 => bar foo => memb_2 => post_2 => bar foo => memb_3 => post_3 => bar """ foo 1 memb_2 1 memb_1 1 memb_3 1 memb_0 1 post_2 1 post_1 1 post_3 1 post_0 1 bar 1
77. Efficient Task Configuration The suite [runtime] section, which specifies runtime properties of tasks and task families, is a multiple inheritance hierarchy efficient - all commonality can be factored out of related tasks cylc uses the C3 MRO algorithm (adapted from Python!)
78. Efficient Task Configuration The suite [runtime] section, which specifies runtime properties of tasks and task families, is a multiple inheritance hierarchy efficient - all commonality can be factored out of related tasks cylc uses the C3 MRO algorithm (adapted from Python!)
79. Efficient Task Configuration The suite [runtime] section, which specifies runtime properties of tasks and task families, is a multiple inheritance hierarchy efficient - all commonality can be factored out of related tasks cylc uses the C3 MRO algorithm (adapted from Python!)
80. Efficient Task Configuration [scheduling] [[dependencies]] graph = "prep => FAM" [runtime] [[FAM]] [[[job submission]]] method = loadleveler [[SERIAL]] [[[directives]]] job_type = serial [[PARALL]] [[[directives]]] job_type = parallel [[m_s1, m_s2]] inherit = FAM, SERIAL [[m_p1, m_p2]] inherit = FAM, PARALL
81. Efficient Task Configuration [scheduling] [[dependencies]] graph = "prep => FAM" ˜[runtime] ˜ [[FAM]] ˜ [[[job submission]]] ˜ method = loadleveler ˜ [[SERIAL]] ˜ [[[directives]]] ˜ job_type = serial ˜ [[PARALL]] ˜ [[[directives]]] ˜ job_type = parallel ˜ [[m_s1, m_s2]] ˜ inherit = FAM, SERIAL ˜ [[m_p1, m_p2]] ˜ inherit = FAM, PARALL
82. Efficient Task Configuration ˜[scheduling] ˜ [[dependencies]] ˜ graph = "prep => FAM" [runtime] [[FAM]] [[[job submission]]] method = loadleveler ˜ [[SERIAL]] ˜ [[[directives]]] ˜ job_type = serial ˜ [[PARALL]] ˜ [[[directives]]] ˜ job_type = parallel ˜ [[m_s1, m_s2]] ˜ inherit = FAM, SERIAL ˜ [[m_p1, m_p2]] ˜ inherit = FAM, PARALL
83. Efficient Task Configuration ˜[scheduling] ˜ [[dependencies]] ˜ graph = "prep => FAM" [runtime] ˜ [[FAM]] ˜ [[[job submission]]] ˜ method = loadleveler [[SERIAL]] [[[directives]]] job_type = serial [[PARALL]] [[[directives]]] job_type = parallel ˜ [[m_s1, m_s2]] ˜ inherit = FAM, SERIAL ˜ [[m_p1, m_p2]] ˜ inherit = FAM, PARALL
84. Efficient Task Configuration ˜[scheduling] ˜ [[dependencies]] ˜ graph = "prep => FAM" [runtime] ˜ [[FAM]] ˜ [[[job submission]]] ˜ method = loadleveler ˜ [[SERIAL]] ˜ [[[directives]]] ˜ job_type = serial ˜ [[PARALL]] ˜ [[[directives]]] ˜ job_type = parallel [[m_s1, m_s2]] inherit = FAM, SERIAL [[m_p1, m_p2]] inherit = FAM, PARALL
85. Efficient Task Configuration [scheduling] [[dependencies]] graph = "prep => FAM" [runtime] [[FAM]] [[[job submission]]] method = loadleveler [[SERIAL]] [[[directives]]] job_type = serial [[PARALL]] [[[directives]]] job_type = parallel [[m_s1, m_s2]] inherit = FAM, SERIAL [[m_p1, m_p2]] inherit = FAM, PARALL
86. Efficient Task Configuration [scheduling] [[dependencies]] graph = "prep => FAM" [runtime] [[FAM]] [[[job submission]]] method = loadleveler [[SERIAL]] [[[directives]]] job_type = serial [[PARALL]] [[[directives]]] job_type = parallel [[m_s1, m_s2]] inherit = FAM, SERIAL [[m_p1, m_p2]] inherit = FAM, PARALL
87. Efficient Task Configuration [scheduling] [[dependencies]] graph = "prep => FAM" [runtime] [[FAM]] [[[job submission]]] method = loadleveler [[SERIAL]] [[[directives]]] job_type = serial [[PARALL]] [[[directives]]] job_type = parallel [[m_s1, m_s2]] inherit = FAM, SERIAL [[m_p1, m_p2]] inherit = FAM, PARALL
88. Efficient Task Configuration [scheduling] [[dependencies]] graph = "prep => FAM" [runtime] [[FAM]] [[[job submission]]] method = loadleveler [[SERIAL]] [[[directives]]] job_type = serial [[PARALL]] [[[directives]]] job_type = parallel [[m_s1, m_s2]] inherit = FAM, SERIAL [[m_p1, m_p2]] inherit = FAM, PARALL List tasks $ cylc list multi: m_p1 m_p2 m_s1 m_s2 prep
89. Efficient Task Configuration [scheduling] [[dependencies]] graph = "prep => FAM" [runtime] [[FAM]] [[[job submission]]] method = loadleveler [[SERIAL]] [[[directives]]] job_type = serial [[PARALL]] [[[directives]]] job_type = parallel [[m_s1, m_s2]] inherit = FAM, SERIAL [[m_p1, m_p2]] inherit = FAM, PARALL First-parent inheritance (used in visualization): $ cylc list -t multi root |-FAM | |-m_p1 | |-m_p2 | |-m_s1 | ‘-m_s2 ‘-prep
90. Efficient Task Configuration [scheduling] [[dependencies]] graph = "prep => FAM" [runtime] [[FAM]] [[[job submission]]] method = loadleveler [[SERIAL]] [[[directives]]] job_type = serial [[PARALL]] [[[directives]]] job_type = parallel [[m_s1, m_s2]] inherit = FAM, SERIAL [[m_p1, m_p2]] inherit = FAM, PARALL Inheritance order (C3 linearization): $ cylc list -m multi m_p1 FAM PARALL root m_p2 FAM PARALL root m_s1 FAM SERIAL root m_s2 FAM SERIAL root prep root
91. Efficient Task Configuration [scheduling] [[dependencies]] graph = "prep => FAM" [runtime] [[FAM]] [[[job submission]]] method = loadleveler [[SERIAL]] [[[directives]]] job_type = serial [[PARALL]] [[[directives]]] job_type = parallel [[m_s1, m_s2]] inherit = FAM, SERIAL [[m_p1, m_p2]] inherit = FAM, PARALL Print computed config: $ cylc get-config multi ’[m_s1]directives’ job_type = serial
92. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
93. cylc GUI All functionality can be accessed by CLI or GUI.
94. cylc GUI All functionality can be accessed by CLI or GUI.
95. cylc GUI All functionality can be accessed by CLI or GUI.
96. cylc GUI All functionality can be accessed by CLI or GUI.
97. cylc GUI All functionality can be accessed by CLI or GUI.
98. cylc GUI All functionality can be accessed by CLI or GUI.
100. Applications Outside of Meteorology? Cylc was designed to solve the inter-cycle dependence problem, to achieve fast catch-up scheduling for cycling suites. But intuitive suite design, flexibility, and ease of use make it a nice way to manage non-cycling distributed workflows too.
101. x1 2012010112 Outline Background What is Cylc? Scheduling Suite Design Running Suites Cylc and Python
102. Cylc and Python Cylc contains about 50,000 lines of code in about 300 files Python is great for doing complex things fast without a mess. high-level language and data structures great documentation nice GUI toolkits - PyGTK easy to debug and profile - pdb, cProfile Leveraging existing Python code: Pyro-3 https://pypi.python.org/pypi/Pyro Jinja2 http://jinja.pocoo.org/ C3 (MRO for Multiple Inheritance) http://www.python.org/download/releases/2.3/mro/ xdot - https://github.com/jrfonseca/xdot.py pygraphviz - pygraphviz.github.io ordereddict http://anthon.home.xs4all.nl/Python/ordereddict/
103. Cylc and Python Cylc contains about 50,000 lines of code in about 300 files Python is great for doing complex things fast without a mess. high-level language and data structures great documentation nice GUI toolkits - PyGTK easy to debug and profile - pdb, cProfile Leveraging existing Python code: Pyro-3 https://pypi.python.org/pypi/Pyro Jinja2 http://jinja.pocoo.org/ C3 (MRO for Multiple Inheritance) http://www.python.org/download/releases/2.3/mro/ xdot - https://github.com/jrfonseca/xdot.py pygraphviz - pygraphviz.github.io ordereddict http://anthon.home.xs4all.nl/Python/ordereddict/
104. Cylc Project: http://cylc.github.com/cylc Cylc is open source software (GPLv3). Contributors: NIWA (NZ) Met Office (UK) NRL Marine Meteorology (US) MPI Meteorology (Germany) Bureau of Meteorology (AU) NCAS (UK) ARC Centre of Excellence for Climate Science (AU) NCI,ANU (AU)
105. Other Bells And Whistles internal queues remote control family triggers task poll and kill conditional triggers suite definition reload inter-suite dependence suite restart with polling automatic retry on failure simulation and dummy modes stepped hourly, daily, monthly, yearly cycling “broadcast” new settings (e.g. communication between tasks)