OACIS_Tutorial_Session2_20170614

Transcript

1. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE Computer simulations create the future  L VacYS @c`OaS WaQ`SbS 8dS b FW c ObW ESaSO`QV GSO E<>8A 4<6F B46<F O Ra aSaaW )" 
      
   
   
2. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE 4US RO  • aSaaW ( , W " – P`WST dS`dWSe B46<F – VO Ra ( • e b acP Wb X Pa caW U B46<F • aSaaW ) W " – VO Ra ) • e b aSb c g c` aW c Ob ` • e b aSb c aba • aSaaW W " – VO Ra • B46<F 4C<a b Ocb ObS O`O SbS` aSO`QV
3. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE 4US RO T FSaaW )  • aSbbW U c O FW c Ob ` – V e X Pa O`S SfSQcbSR Pg B46<F – bVS `S_cW`S S ba T ` FW c Ob ` • T ` Ob T W cb O`O SbS`a • T ` Ob T cb cb TW Sa – O Ra • aSbbW U c ab O R ab:` c – V e B46<F O`S Q SQbSR b V aba – RS ab`ObW / aSbbW U c O ab • FF UW O R OcbVS bWQObW caW U YSg • facP • `SUWabS` O ab B46<F – RS ab`ObW / RSTW W U O ab:` c • D 4
4. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE aSbbW U c O FW c Ob `  • ESTS`S QS – Vbb /&&Q`Sab QOaaWO UWbVcP W & OQWa&S &Q TWUc`W UMaW c Ob ` Vb – Vbb /&&Q`Sab QOaaWO UWbVcP W & OQWa&XO&Q TWUc`W UMaW c Ob ` Vb
5. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE V e X Pa O`S SfSQcbSR 1. Work directory and shell script are created. Jobs are submitted to the scheduler. 2. Jobs are executed by the job scheduler. Current directory is set to the work directory. 3. Results are downloaded to OACIS server. The files are parsed and the records are updated. database job scheduler (ex. Torque) OACIS Computational Host qsub SSH login sh execute in the work directory the work directory is compressed into a single file. tar download (scp) & expand parse job status & update the record
6. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE `S_cW`S S ba T ` FW c Ob ` FW c Ob ` cab aObWaTg bVS T eW U `S_cW`S S ba • < cb O`O SbS`a cab PS UWdS Oa Q O R W S O`Uc S ba ` Pg O FBA TW S SbOW a O`S aV e W bVS Sfb a WRS " • Bcb cb TW Sa cab PS Q`SObSR c RS` bVS Qc``S b RW`SQb `g • GVSaS TW Sa O`S Q`SObSR Pg B46<F FW c Ob ` cab b Q T WQb eWbV bVSaS TW Sa – M cb cb Xa MabObca Xa MbW S bfb MdS`aW bfb M U bfb MabR cb bfb MabRS`` bfb • ESbc` a eVS Wb TW WaVSR acQQSaaTc g ESbc` O hS` Q RS eVS Wb VOa O S`` `
7. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE V e b aSb W cb O`O SbS`a • 5g Q O R W S O`Uc S ba – < cb O`O SbS`a O R bVS `O R c PS` aSSR O`S O S RSR b bVS Q O R • k&aW c Ob ` cb 0 (1 0 )1 0 1 0aSSR1 • 5g O FBA TW S – JVS `c W U bVS Q O R O TW S lMW cb Xa m Wa Q`SObSR W eVWQV W cb O`O SbS`a O`S e`WbbS • i ( / ) / ( MaSSR /() ,j
8. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE C`S O`ObW T FW c Ob `a • < `RS` b `SUWabS` g c` aW c Ob ` b B46<F Q`SObS O aQ`W b eVWQV OYSa g c` aW c Ob ` Q T ` b bVS W cb T ` Ob T B46<F ESUWabS` bVS aQ`W b Oa O FW c Ob ` T B46<F wrap script simulation program run.py _input.json a.out 1. Input parameters given by OACIS 2. parse command line args and make an XML file 3. launch the simulation program as a subprocess
9. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE http://crest-cassia.github.io/oacis/en/configuring_simulator.html Other samples for these scripts are found in our doc page. Load the input JSON file Make an XML file Spawn a subprocess
10. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE Bcb cb TW S T ` Ob  • 4a U Oa cb cb TW Sa O`S Q`SObSR W bVS Qc``S b RW`SQb `g g c VOdS bVW U b R • <T g c e c R WYS b b aQO O` dO cSa W B46<F bVS dO cSa aV c R PS e`WbbS W lM cb cb Xa m TW S • <T g c e c R WYS b caS TWUc`S dWSeS` e`WbS TW Sa W P U X U O R bVS` TWUc`S T ` Ob S a TW Sa O`S b ac `bSR " {"flow": 0.235, "velocity": 1.245 }
11. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE b `SUWabS` O FW c Ob `  Simulator name Definition of Parameters name, type, default value preprocess command 1optional2 command of Simulator JSON or Argument input A note of the Simulator List of executable hosts
12. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE SPcUUW U GW a  • JVS bVS aSb c Wa W O ` `WObS `c a eW TOW < acQV QOaSa aSS lMabRS`` bfbm T ` RSPcUUW U
13. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE c WQObW U O FW c Ob `  • L c Og eO b b b`g O aW c ObW RS eVWQV Wa a WUVb g RWTTS`S b T` bVS Qc``S b S < acQV QOaS g c QO Q`SObS O Se aW c Ob ` POaSR bVS SfWabW U S Duplicating Simulator
14. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE O Ra (  • ?Sb ca `SUWabS` lSQV m Q O R Oa O aW c Ob ` • Ec O X P O R aSS lMabR cb bfb
15. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE  Simulator name Definition of Parameters name, type, default value “echo” Select “Argument” Select “localhost”
16. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE O Ra )  • ?Sb ca `SUWabS` bVS AOUS FQV`SQYS PS`U aW c Ob ` Pg c`aS dSa – ? UW b bVS Q bOW S` – 8fSQcbS bVS aW c Ob ` Pg g c`aS T Key Description l Road length v Maximum velocity rho Car density p deceleration probability t_init initialization steps t_measure measurement steps mkdir temp && cd temp ~/sim_ns_model/run.sh 200 5 0.3 0.1 100 300 12345 docker exec -it -u oacis my_oacis bash -l
17. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE  • ` bVS b OUS T B46<F Q WQY lASe FW c Ob `m Pcbb O R TW W bVS T ` Item Contents Name MySimulator Definition of Parameters L, v, rho, p, t_init, t_measure Command ~/sim_ns_model/run.sh Input Type Argument Executable_on localhost Key Type Default Val l Integer 200 v Integer 5 rho Float 0.3 p Float 0.1 t_init Integer 100 t_measure Integer 300 Click “Add Parameter” to increase the number of parameters.
18. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE  • 8fSQcbS bVS Se g Q`SObSR FW c Ob ` – IS`WTg bVOb g c USb bVS aO S `Sac b Oa bVS `SdW ca aSaaW
19. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE `SUWabS`W U O ab O R ab:` c  • ESTS`S QS – Vbb /&&Q`Sab QOaaWO UWbVcP W & OQWa&S &Q TWUc`W UMV ab Vb – Vbb /&&Q`Sab QOaaWO UWbVcP W & OQWa&XO&Q TWUc`W UMV ab Vb
20. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE FSbbW U c aba  4 bg WQO aS_cS QS T ` aSbbW U c Q cbObW O V ab Wa Oa T ea • FSbbW U c FF OcbV `WhObW YSg FF Q SQbW a cab PS SabOP WaVSR eWbV cb S bS`W U Oaae `R • < abO KFH5 • `SUWabS` V ab W T ` ObW See http://crest-cassia.github.io/oacis/en/install.html
21. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE HaW U FF OUS b  • FSbbW U c FF OcbV `WhObW YSg a bVOb FF Q SQbW Wa OdOW OP S eWbV cb S bS`W U O Oaae `R ` Oaa V`OaS (At OACIS host) ssh-keygen -t rsa # ~/.ssh/id_rsa ~/.ssh/id_rsa.pub are created scp ~/.ssh/id_rsa.pub [USER]@[HOST_NAME]:~ # copy your public key to the remote host (At Computational host) cat ~/id_rsa.pub >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys # add your public key to the list of authorized keys (At OACIS host) eval `ssh-agent` # launch SSH agent ssh-add ~/.ssh/id_rsa # add key to agent. Enter your passphrase. ssh [USER]@[HOST_NAME] # verify that you don’t need to enter the passphrase bundle exec rake daemon:restart # launch OACIS
22. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE JVOb Wa KFH52  • F SQWTWQObW T X P aQVSRc S`a O`S RWTTS`S b T ` SOQV agabS • KFH5 Wa O a O aQ`W b eVWQV OPa `Pa bVS RWTTS`S QS T bVS a SQWTWQObW T X P aQVSRc S`a B46<F caSa KFH5 b acP Wb O X P <b cab PS W abO SR W ORdO QS • EcPg ( . ` ObS` Wa `S_cW`SR Refer to https://github.com/crest-cassia/xsub
23. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE < abO W U KFH5  • B Q cbObW O V ab – UWb Q S Vbb a/&&UWbVcP Q &Q`Sab QOaaWO&facP UWb • 8RWb k& POaVM ` TW S • 6c``S b g S b `_cS K( > FE(- B6HF ac S` Q cbW U agabS O`S ac `bSR export PATH="$HOME/xsub/bin:$PATH" export XSUB_TYPE="none"
24. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE aSbbW U c ab  host information for SSH connection Directory used for running jobs. OACIS will create sub directories under this. Maximum number of concurrent jobs. Worker checks the status of the remote host with this interval. The available range of the number of MPI processes and threads. List of executable simulators on that host. http://crest-cassia.github.io/oacis/en/tutorial.html#registering-a-host
25. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE  Parameters required by XSUB (Host Parameters) are automatically set.
26. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE  Host Parameters are required when creating a PS.
27. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE RS ab`ObW  • ` bVS Q bOW S` eVS`S B46<F Wa e `YW U SfSQcbS aW c ObW bVS V ab OQVW S docker exec -u my_oacis cat /home/oacis/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys # testing connection docker exec -it -u oacis my_oacis bash -l ssh $USER@`hostname` # installing xsub git clone https://github.com/crest-cassia/xsub.git ~/xsub # edit ~/.bash_profile. Add the following two lines. export PATH="$HOME/xsub/bin:$PATH" export XSUB_TYPE="none"
28. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE STW W U O ab:` c  We can define a HostGroup, a group of Hosts. When creating a job, HostGroup can be specified as a destination of the submission. The jobs are submitted to one of the hosts in the HostGroup.
29. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE  click Hosts click “New Host Group” We can select a HostGroup as the place to which jobs are submitted.
30. RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE 6 Q caW  • JS VOdS aV e V e b `SUWabS` FW c Ob `a aba O R ab:` c a – GVWa WUVb PS O PWb Q WQObSR Pcb QS g c VOdS R S bVSaS `SUWab`ObW bVS `S OW W U e `YT e PSQ Sa cQV `S ` RcQbWdS • <T g c VOdS O g _cSabW O R TSSRPOQYa – OQWa RSd3U U SU` c a Q n < bVS Sfb aSaaW eS eW RS ab`ObS V e b caS 4C<a b Ocb ObS O`O SbS` a OQS Sf `ObW