pfm

Transcript

1. Tiny Port Forward Manger for Data Scientists Takahiko Ito

2. Self-Introduction • Software engineer working in Cookpad Inc. • Ph.D

   • Research topics: graph analysis, natural Language processing, data mining etc… • Today I talk about a tool applicable to machine learning experiments. 2

3. Preliminaries: port forwarding • Limited resources in local machines •

   Data scientists work in multiple remote computers with ssh • To connect the UI of the servers (Jupyter Notebook) we need to add port forwarding settings. 3

4. Sample: ssh parameters ssh -A -L 8888:localhost:8888 takahi-i-workbench-001.ml.aws.ckpd.co 4 Port

   forward setting Remote host We need to remember the combinations among task, remote host, and forwarded port for each project

5. Solution: pfm • I made a tiny command line tool

   to manage port forward settings for data scientists. • URL: https://github.com/takahi-i/pfm 5

6. pfm keeps a table Table contains port forward settings •

   Task name • Type of port forwarding • Name of remote host • Remote port number • Login user • ssh-server 6

7. Example: pfm table pfm shows the list of port forward

   settings with `pfm list` command. 7 my-ml-instance-001.aws.com my-ml-instance-001.aws.com

8. Register the settings 8 $ pfm add --name food-nonfood --local-port

   7777 --remote-port 8888 --ssh-server my-ml-instance-2.aws.com --remote-host localhost --login-user takahi-i

9. Ssh connection with port forward settings with pfm • Simplify

   ssh commands • Example: • ssh -A `pfm param recipe-calssification` or • pfm connect recipe-classification 9

10. Demo: port forwarding to Jupyter server in Docker container with

    pfm Note: port forward settings from a Docker container is done with Cookiecutter Docker Science • URL: https://docker-science.github.io/ 10

11. Usage: provided commands Usage: pfm [OPTIONS] COMMAND [ARGS]... Options: -c,

    --config TEXT configuration file (DEFAULT $HOME/.pfm) --help Show this message and exit. Commands: add Add port forward setting delete Delete specified setting list List existing port forward settings param Generate ssh port forward parameters update Update registered port forward setting version Show version number 11

12. Future work • Register working directory in remote machines •

    Make `pfm connect` go to the working directory • Make pfm shows available list of port used in port forwarding 12

13. Thank you for your attention!