Upgrade to Pro — share decks privately, control downloads, hide ads and more …


Takahiko Ito
December 01, 2018


Takahiko Ito

December 01, 2018


  1. Self-Introduction • Software engineer working in Cookpad Inc. • Ph.D

    • Research topics: graph analysis, natural Language processing, data mining etc… • Today I talk about a tool applicable to machine learning experiments. 2
  2. Preliminaries: port forwarding • Limited resources in local machines •

    Data scientists work in multiple remote computers with ssh • To connect the UI of the servers (Jupyter Notebook) we need to add port forwarding settings. 3
  3. Sample: ssh parameters ssh -A -L 8888:localhost:8888 takahi-i-workbench-001.ml.aws.ckpd.co 4 Port

    forward setting Remote host We need to remember the combinations among task, remote host, and forwarded port for each project
  4. Solution: pfm • I made a tiny command line tool

    to manage port forward settings for data scientists. • URL: https://github.com/takahi-i/pfm 5
  5. pfm keeps a table Table contains port forward settings •

    Task name • Type of port forwarding • Name of remote host • Remote port number • Login user • ssh-server 6
  6. Example: pfm table pfm shows the list of port forward

    settings with `pfm list` command. 7 my-ml-instance-001.aws.com my-ml-instance-001.aws.com
  7. Register the settings 8 $ pfm add --name food-nonfood --local-port

    7777 --remote-port 8888 --ssh-server my-ml-instance-2.aws.com --remote-host localhost --login-user takahi-i
  8. Ssh connection with port forward settings with pfm • Simplify

    ssh commands • Example: • ssh -A `pfm param recipe-calssification` or • pfm connect recipe-classification 9
  9. Demo: port forwarding to Jupyter server in Docker container with

    pfm Note: port forward settings from a Docker container is done with Cookiecutter Docker Science • URL: https://docker-science.github.io/ 10
  10. Usage: provided commands Usage: pfm [OPTIONS] COMMAND [ARGS]... Options: -c,

    --config TEXT configuration file (DEFAULT $HOME/.pfm) --help Show this message and exit. Commands: add Add port forward setting delete Delete specified setting list List existing port forward settings param Generate ssh port forward parameters update Update registered port forward setting version Show version number 11
  11. Future work • Register working directory in remote machines •

    Make `pfm connect` go to the working directory • Make pfm shows available list of port used in port forwarding 12