Containers @ Hyperspeed

Containers @ Hyperspeed

(talk given at container.camp london -- 2015-09-11)

Containers are awesome! But storing and moving them is painful! It's slow! Containers are huge! There's little deduplication! No cryptographic versioning! Images are lost! Worst, we must depend on other services -- or having fast connection to them...

No more! starship puts your containers on hyperspeed! Instead of old-school HTTP, starship uses shiny IPFS -- a new protocol, Git meets Bittorrent. With IPFS, starship (a) deduplicates all data between containers, (b) uses p2p download from fastest hosts, and (c) can limit connections to private cloud(s) or local networks.

A9670c143716320893863524a0efbaff?s=128

Juan Batiz-Benet

September 11, 2015
Tweet

Transcript

  1. ipfs.io @juanbenet Protocol Labs Containers @ Hyperspeed!

  2. None
  3. IP UDP TCP SCTP Ethernet CSMA WIFI TP Coax Satellite

    Fiber Radio PPP HTTP SMTP FTP NFS Skype Pigeons Chat
  4. None
  5. None
  6. None
  7. 1. containers 2. hyperspeed 3. starship

  8. 1. containers 2. hyperspeed 3. starship 4. car

  9. 1. containers 2. hyperspeed 3. starship 4. car

  10. 1. containers 2. hyperspeed 3. starship

  11. None
  12. represent a program

  13. have everything needed to run

  14. have everything needed to run (sort of)

  15. containers have flavors

  16. just a bunch of files

  17. just a bunch of files & standard protocols to run

    them
  18. protocols!

  19. protocols! runC Open Container Initiative

  20. some containers are small

  21. some containers are big

  22. some containers are big ~ 50 MB ~ 300 MB

  23. some containers are HUGE

  24. some containers are HUGE ~ 5 GB ~ 10 GB

  25. None
  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. None
  34. None
  35. None
  36. None
  37. None
  38. None
  39. None
  40. None
  41. None
  42. None
  43. None
  44. None
  45. None
  46. None
  47. None
  48. None
  49. None
  50. None
  51. None
  52. None
  53. None
  54. None
  55. None
  56. None
  57. 1. containers 2. hyperspeed 3. starship

  58. None
  59. None
  60. What is IPFS?

  61. A protocol to upgrade the web

  62. A protocol to upgrade the web

  63. None
  64. 2,387,990,609

  65. 2,387,990,609 477.6 PB

  66. None
  67. None
  68. None
  69. None
  70. None
  71. None
  72. A protocol to upgrade the web

  73. Permanent Distributed Safer Smarter Offline Faster

  74. None
  75. hyperlinks

  76. hyperlinks media

  77. a hypermedia transport protocol HTTP

  78. a hypermedia transport protocol HTTP IPFS

  79. None
  80. SFS web DHT +

  81. routing network exchange merkledag naming applications IPNS DNS Namecoin Bitswap

    HTTP BitTorrent Kad DHT Chord mDNS Gossip Delegated FTP TOR QUIC TOR TCP uTP WebRTC WebSockets Git Bitcoin VLC Etherpad Chat I2P I2P CJDNS UDT
  82. None
  83. None
  84. None
  85. None
  86. None
  87. None
  88. cvs/svn

  89. cvs/svn

  90. cvs/svn

  91. None
  92. None
  93. None
  94. None
  95. None
  96. None
  97. None
  98. None
  99. just a bunch of files

  100. just a bunch of files

  101. just a bunch of files

  102. just a bunch of files

  103. just a bunch of files

  104. just a bunch of files

  105. just a bunch of files

  106. just a bunch of files (tar) (tar) (tar)

  107. just a bunch of files (tar) (tar) (tar)

  108. 1. containers 2. hyperspeed 3. starship

  109. shp - starship container hypertransport

  110. shp - starship container hypertransport

  111. shp - starship container hypertransport

  112. shp - starship container hypertransport

  113. None
  114. None
  115. DEMO

  116. 1. as a docker/registry storage driver 2. as a docker/docker

    graph driver 3. standalone (load/save) dkr-shp
  117. jbenet @ earth : ~ > dkr-shp shp - containers

    at hyperspeed shp (starship) uses IPFS, a p2p merkle-dag transport, to move around container images. This is dkr-shp, the docker-specific tool for shp. COMMANDS dkr-shp init initialize a shp station dkr-shp start start the shp station dkr-shp push <docker-id> publish given docker image on this host dkr-shp pull <ipfs-hash> pull down docker image, add it to docker
  118. jbenet @ earth : ~ > dkr-shp init initialized shp

    station (registry) at /var/lib/shp
  119. jbenet @ earth : ~ > dkr-shp init initialized shp

    station (registry) at /var/lib/shp jbenet @ earth : ~ > dkr-shp start started shp station (registry) 951f267861f99a79
  120. jbenet @ earth : ~ > dkr-shp init initialized shp

    station (registry) at /var/lib/shp jbenet @ earth : ~ > dkr-shp start started shp station (registry) 951f267861f99a79 jbenet @ earth : ~ > docker images | grep ipfs jbenet/go-ipfs latest c1d9bf8d8844 4 hours ago 29.76 MB thelinuxkid/ipfs-reg latest 8bd6f334dca4 2 days ago 254.2 MB
  121. jbenet @ earth : ~ > dkr-shp init initialized shp

    station (registry) at /var/lib/shp jbenet @ earth : ~ > dkr-shp start started shp station (registry) 951f267861f99a79 jbenet @ earth : ~ > docker images | grep ipfs jbenet/go-ipfs latest c1d9bf8d8844 4 hours ago 29.76 MB thelinuxkid/ipfs-reg latest 8bd6f334dca4 2 days ago 254.2 MB jbenet @ earth : ~ > dkr-shp push c1d9bf8d8844 published as /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844
  122. jbenet @ earth : ~ > dkr-shp init initialized shp

    station (registry) at /var/lib/shp jbenet @ earth : ~ > dkr-shp start started shp station (registry) 951f267861f99a79 jbenet @ earth : ~ > docker images | grep ipfs jbenet/go-ipfs latest c1d9bf8d8844 4 hours ago 29.76 MB thelinuxkid/ipfs-reg latest 8bd6f334dca4 2 days ago 254.2 MB jbenet @ earth : ~ > dkr-shp push c1d9bf8d8844 published as /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844 jbenet @ earth : ~ > dkr-shp push jbenet/go-ipfs published as /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/jbenet/go-ipfs
  123. jbenet @ mars : ~ > dkr-shp init initialized shp

    station (registry) at /var/lib/shp
  124. jbenet @ mars : ~ > dkr-shp init initialized shp

    station (registry) at /var/lib/shp jbenet @ mars : ~ > dkr-shp start started shp station (registry) eac13c827004cf7a
  125. jbenet @ mars : ~ > dkr-shp init initialized shp

    station (registry) at /var/lib/shp jbenet @ mars : ~ > dkr-shp start started shp station (registry) eac13c827004cf7a jbenet @ mars : ~ > dkr-shp pull c1d9bf8d8844 pulled /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844
  126. jbenet @ mars : ~ > dkr-shp init initialized shp

    station (registry) at /var/lib/shp jbenet @ mars : ~ > dkr-shp start started shp station (registry) eac13c827004cf7a jbenet @ mars : ~ > dkr-shp pull c1d9bf8d8844 pulled /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844 jbenet @ mars : ~ > dkr-shp peers /ip4/104.236.32.22/tcp/4006/ipfs/QmNxzC53nTdTJaFivaev5mu2zFJmzcAAfQXJqTLarB8omm /ip4/162.243.139.61/tcp/4006/ipfs/QmNaVrdSKnZ8qpUhhGJeJuo7AjMHJBDE2xSWMvBau34g8P /ip4/104.236.176.52/tcp/4001/ipfs/QmSoLnSGccFuZQJzRadHn95W2CrSFmZuTdDWP8HXaHca9z /ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu /ip4/178.62.158.247/tcp/4001/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd /ip4/178.62.61.185/tcp/4001/ipfs/QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3
  127. jbenet @ mars : ~ > dkr-shp addrs /ip4/127.0.0.1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH /ip4/172.17.0.25/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH

    /ip6/::1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
  128. jbenet @ mars : ~ > dkr-shp addrs /ip4/127.0.0.1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH /ip4/172.17.0.25/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH

    /ip6/::1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH jbenet @ mars : ~ > dkr-shp connect $addr
  129. jbenet @ mars : ~ > dkr-shp addrs /ip4/127.0.0.1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH /ip4/172.17.0.25/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH

    /ip6/::1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH jbenet @ mars : ~ > dkr-shp connect $addr jbenet @ mars : ~ > # setup in a completely private network jbenet @ mars : ~ > dkr-shp init --net QmPzugFx3ysr8BujxZ3dkBiRYnuS2D63ndkAdq3W7J initialized shp station (registry) at /var/lib/shp
  130. ipfs.io @juanbenet Protocol Labs Containers @ Hyperspeed!