pfm

Transcript

1. Tiny Port Forward Manger
   for Data Scientists
   Takahiko Ito
   

   View Slide

2. Self-Introduction
   • Software engineer working in Cookpad Inc.
   • Ph.D
   • Research topics: graph analysis, natural
   Language processing, data mining etc…
   • Today I talk about a tool applicable to machine
   learning experiments.
   2
   

   View Slide

3. Preliminaries: port forwarding
   • Limited resources in local
   machines
   • Data scientists work in
   multiple remote
   computers with ssh
   • To connect the UI of the
   servers (Jupyter Notebook)
   we need to add port
   forwarding settings.
   3
   

   View Slide

4. Sample: ssh parameters
   ssh -A -L 8888:localhost:8888
   takahi-i-workbench-001.ml.aws.ckpd.co
   4
   Port forward
   setting
   Remote host
   We need to remember the combinations
   among task, remote host, and forwarded port
   for each project
   

   View Slide

5. Solution: pfm
   • I made a tiny command line tool to manage port
   forward settings for data scientists.
   • URL: https://github.com/takahi-i/pfm
   5
   

   View Slide

6. pfm keeps a table
   Table contains port forward settings
   • Task name
   • Type of port forwarding
   • Name of remote host
   • Remote port number
   • Login user
   • ssh-server
   6
   

   View Slide

7. Example: pfm table
   pfm shows the list of port forward settings with `pfm
   list` command.
   7
   my-ml-instance-001.aws.com
   my-ml-instance-001.aws.com
   

   View Slide

8. Register the settings
   8
   $ pfm add
   --name food-nonfood
   --local-port 7777
   --remote-port 8888
   --ssh-server my-ml-instance-2.aws.com
   --remote-host localhost
   --login-user takahi-i
   

   View Slide

9. Ssh connection with port
   forward settings with pfm
   • Simplify ssh commands
   • Example:
   • ssh -A `pfm param recipe-calssification`
   or
   • pfm connect recipe-classification
   9
   

   View Slide

10. Demo: port forwarding to Jupyter
    server in Docker container with pfm
    Note: port forward settings from a Docker container is done with
    Cookiecutter Docker Science
    • URL: https://docker-science.github.io/
    10
    

    View Slide

11. Usage: provided commands
    Usage: pfm [OPTIONS] COMMAND [ARGS]...
    Options:
    -c, --config TEXT configuration file (DEFAULT $HOME/.pfm)
    --help Show this message and exit.
    Commands:
    add Add port forward setting
    delete Delete specified setting
    list List existing port forward settings
    param Generate ssh port forward parameters
    update Update registered port forward setting
    version Show version number
    11
    

    View Slide

12. Future work
    • Register working directory in remote machines
    • Make `pfm connect` go to the working directory
    • Make pfm shows available list of port used in port
    forwarding
    12
    

    View Slide

13. Thank you for your attention!
    

    View Slide