$30 off During Our Annual Pro Sale. View Details »

Containers @ Hyperspeed

Containers @ Hyperspeed

(talk given at container.camp london -- 2015-09-11)

Containers are awesome! But storing and moving them is painful! It's slow! Containers are huge! There's little deduplication! No cryptographic versioning! Images are lost! Worst, we must depend on other services -- or having fast connection to them...

No more! starship puts your containers on hyperspeed! Instead of old-school HTTP, starship uses shiny IPFS -- a new protocol, Git meets Bittorrent. With IPFS, starship (a) deduplicates all data between containers, (b) uses p2p download from fastest hosts, and (c) can limit connections to private cloud(s) or local networks.

Juan Batiz-Benet

September 11, 2015
Tweet

More Decks by Juan Batiz-Benet

Other Decks in Technology

Transcript

  1. ipfs.io
    @juanbenet
    Protocol Labs
    Containers @ Hyperspeed!

    View Slide

  2. View Slide

  3. IP
    UDP
    TCP SCTP
    Ethernet
    CSMA WIFI
    TP
    Coax Satellite
    Fiber Radio
    PPP
    HTTP SMTP
    FTP
    NFS Skype
    Pigeons
    Chat

    View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. 1. containers
    2. hyperspeed
    3. starship

    View Slide

  8. 1. containers
    2. hyperspeed
    3. starship
    4. car

    View Slide

  9. 1. containers
    2. hyperspeed
    3. starship
    4. car

    View Slide

  10. 1. containers
    2. hyperspeed
    3. starship

    View Slide

  11. View Slide

  12. represent a program

    View Slide

  13. have everything needed to run

    View Slide

  14. have everything needed to run
    (sort of)

    View Slide

  15. containers have flavors

    View Slide

  16. just a bunch of files

    View Slide

  17. just a bunch of files
    & standard protocols to run them

    View Slide

  18. protocols!

    View Slide

  19. protocols!
    runC
    Open Container
    Initiative

    View Slide

  20. some containers are small

    View Slide

  21. some containers are big

    View Slide

  22. some containers are big
    ~ 50 MB
    ~ 300 MB

    View Slide

  23. some containers are HUGE

    View Slide

  24. some containers are HUGE
    ~ 5 GB
    ~ 10 GB

    View Slide

  25. View Slide

  26. View Slide

  27. View Slide

  28. View Slide

  29. View Slide

  30. View Slide

  31. View Slide

  32. View Slide

  33. View Slide

  34. View Slide

  35. View Slide

  36. View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. View Slide

  42. View Slide

  43. View Slide

  44. View Slide

  45. View Slide

  46. View Slide

  47. View Slide

  48. View Slide

  49. View Slide

  50. View Slide

  51. View Slide

  52. View Slide

  53. View Slide

  54. View Slide

  55. View Slide

  56. View Slide

  57. 1. containers
    2. hyperspeed
    3. starship

    View Slide

  58. View Slide

  59. View Slide

  60. What is IPFS?

    View Slide

  61. A protocol to upgrade the web

    View Slide

  62. A protocol to upgrade the web

    View Slide

  63. View Slide

  64. 2,387,990,609

    View Slide

  65. 2,387,990,609
    477.6 PB

    View Slide

  66. View Slide

  67. View Slide

  68. View Slide

  69. View Slide

  70. View Slide

  71. View Slide

  72. A protocol to upgrade the web

    View Slide

  73. Permanent
    Distributed Safer
    Smarter
    Offline
    Faster

    View Slide

  74. View Slide

  75. hyperlinks

    View Slide

  76. hyperlinks media

    View Slide

  77. a hypermedia transport protocol
    HTTP

    View Slide

  78. a hypermedia transport protocol
    HTTP IPFS

    View Slide

  79. View Slide

  80. SFS
    web
    DHT
    +

    View Slide

  81. routing
    network
    exchange
    merkledag
    naming
    applications
    IPNS
    DNS Namecoin
    Bitswap HTTP
    BitTorrent
    Kad DHT
    Chord mDNS
    Gossip Delegated
    FTP
    TOR
    QUIC TOR
    TCP
    uTP WebRTC WebSockets
    Git Bitcoin
    VLC
    Etherpad Chat
    I2P
    I2P
    CJDNS UDT

    View Slide

  82. View Slide

  83. View Slide

  84. View Slide

  85. View Slide

  86. View Slide

  87. View Slide

  88. cvs/svn

    View Slide

  89. cvs/svn

    View Slide

  90. cvs/svn

    View Slide

  91. View Slide

  92. View Slide

  93. View Slide

  94. View Slide

  95. View Slide

  96. View Slide

  97. View Slide

  98. View Slide

  99. just a bunch of files

    View Slide

  100. just a bunch of files

    View Slide

  101. just a bunch of files

    View Slide

  102. just a bunch of files

    View Slide

  103. just a bunch of files

    View Slide

  104. just a bunch of files

    View Slide

  105. just a bunch of files

    View Slide

  106. just a bunch of files
    (tar) (tar) (tar)

    View Slide

  107. just a bunch of files
    (tar) (tar) (tar)

    View Slide

  108. 1. containers
    2. hyperspeed
    3. starship

    View Slide

  109. shp - starship
    container hypertransport

    View Slide

  110. shp - starship
    container hypertransport

    View Slide

  111. shp - starship
    container hypertransport

    View Slide

  112. shp - starship
    container hypertransport

    View Slide

  113. View Slide

  114. View Slide

  115. DEMO

    View Slide

  116. 1. as a docker/registry storage driver
    2. as a docker/docker graph driver
    3. standalone (load/save)
    dkr-shp

    View Slide

  117. jbenet @ earth : ~ > dkr-shp
    shp - containers at hyperspeed
    shp (starship) uses IPFS, a p2p merkle-dag transport, to
    move around container images.
    This is dkr-shp, the docker-specific tool for shp.
    COMMANDS
    dkr-shp init initialize a shp station
    dkr-shp start start the shp station
    dkr-shp push publish given docker image on this host
    dkr-shp pull pull down docker image, add it to docker

    View Slide

  118. jbenet @ earth : ~ > dkr-shp init
    initialized shp station (registry) at /var/lib/shp

    View Slide

  119. jbenet @ earth : ~ > dkr-shp init
    initialized shp station (registry) at /var/lib/shp
    jbenet @ earth : ~ > dkr-shp start
    started shp station (registry) 951f267861f99a79

    View Slide

  120. jbenet @ earth : ~ > dkr-shp init
    initialized shp station (registry) at /var/lib/shp
    jbenet @ earth : ~ > dkr-shp start
    started shp station (registry) 951f267861f99a79
    jbenet @ earth : ~ > docker images | grep ipfs
    jbenet/go-ipfs latest c1d9bf8d8844 4 hours ago 29.76 MB
    thelinuxkid/ipfs-reg latest 8bd6f334dca4 2 days ago 254.2 MB

    View Slide

  121. jbenet @ earth : ~ > dkr-shp init
    initialized shp station (registry) at /var/lib/shp
    jbenet @ earth : ~ > dkr-shp start
    started shp station (registry) 951f267861f99a79
    jbenet @ earth : ~ > docker images | grep ipfs
    jbenet/go-ipfs latest c1d9bf8d8844 4 hours ago 29.76 MB
    thelinuxkid/ipfs-reg latest 8bd6f334dca4 2 days ago 254.2 MB
    jbenet @ earth : ~ > dkr-shp push c1d9bf8d8844
    published as /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844

    View Slide

  122. jbenet @ earth : ~ > dkr-shp init
    initialized shp station (registry) at /var/lib/shp
    jbenet @ earth : ~ > dkr-shp start
    started shp station (registry) 951f267861f99a79
    jbenet @ earth : ~ > docker images | grep ipfs
    jbenet/go-ipfs latest c1d9bf8d8844 4 hours ago 29.76 MB
    thelinuxkid/ipfs-reg latest 8bd6f334dca4 2 days ago 254.2 MB
    jbenet @ earth : ~ > dkr-shp push c1d9bf8d8844
    published as /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844
    jbenet @ earth : ~ > dkr-shp push jbenet/go-ipfs
    published as /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/jbenet/go-ipfs

    View Slide

  123. jbenet @ mars : ~ > dkr-shp init
    initialized shp station (registry) at /var/lib/shp

    View Slide

  124. jbenet @ mars : ~ > dkr-shp init
    initialized shp station (registry) at /var/lib/shp
    jbenet @ mars : ~ > dkr-shp start
    started shp station (registry) eac13c827004cf7a

    View Slide

  125. jbenet @ mars : ~ > dkr-shp init
    initialized shp station (registry) at /var/lib/shp
    jbenet @ mars : ~ > dkr-shp start
    started shp station (registry) eac13c827004cf7a
    jbenet @ mars : ~ > dkr-shp pull c1d9bf8d8844
    pulled /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844

    View Slide

  126. jbenet @ mars : ~ > dkr-shp init
    initialized shp station (registry) at /var/lib/shp
    jbenet @ mars : ~ > dkr-shp start
    started shp station (registry) eac13c827004cf7a
    jbenet @ mars : ~ > dkr-shp pull c1d9bf8d8844
    pulled /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844
    jbenet @ mars : ~ > dkr-shp peers
    /ip4/104.236.32.22/tcp/4006/ipfs/QmNxzC53nTdTJaFivaev5mu2zFJmzcAAfQXJqTLarB8omm
    /ip4/162.243.139.61/tcp/4006/ipfs/QmNaVrdSKnZ8qpUhhGJeJuo7AjMHJBDE2xSWMvBau34g8P
    /ip4/104.236.176.52/tcp/4001/ipfs/QmSoLnSGccFuZQJzRadHn95W2CrSFmZuTdDWP8HXaHca9z
    /ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
    /ip4/178.62.158.247/tcp/4001/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd
    /ip4/178.62.61.185/tcp/4001/ipfs/QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3

    View Slide

  127. jbenet @ mars : ~ > dkr-shp addrs
    /ip4/127.0.0.1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
    /ip4/172.17.0.25/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
    /ip6/::1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH

    View Slide

  128. jbenet @ mars : ~ > dkr-shp addrs
    /ip4/127.0.0.1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
    /ip4/172.17.0.25/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
    /ip6/::1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
    jbenet @ mars : ~ > dkr-shp connect $addr

    View Slide

  129. jbenet @ mars : ~ > dkr-shp addrs
    /ip4/127.0.0.1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
    /ip4/172.17.0.25/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
    /ip6/::1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
    jbenet @ mars : ~ > dkr-shp connect $addr
    jbenet @ mars : ~ > # setup in a completely private network
    jbenet @ mars : ~ > dkr-shp init --net QmPzugFx3ysr8BujxZ3dkBiRYnuS2D63ndkAdq3W7J
    initialized shp station (registry) at /var/lib/shp

    View Slide

  130. ipfs.io
    @juanbenet
    Protocol Labs
    Containers @ Hyperspeed!

    View Slide