Containers @ Hyperspeed

(talk given at container.camp london -- 2015-09-11)

Containers are awesome! But storing and moving them is painful! It's slow! Containers are huge! There's little deduplication! No cryptographic versioning! Images are lost! Worst, we must depend on other services -- or having fast connection to them...

No more! starship puts your containers on hyperspeed! Instead of old-school HTTP, starship uses shiny IPFS -- a new protocol, Git meets Bittorrent. With IPFS, starship (a) deduplicates all data between containers, (b) uses p2p download from fastest hosts, and (c) can limit connections to private cloud(s) or local networks.

Transcript

1. ipfs.io
   @juanbenet
   Protocol Labs
   Containers @ Hyperspeed!
   

   View Slide

2. View Slide

3. IP
   UDP
   TCP SCTP
   Ethernet
   CSMA WIFI
   TP
   Coax Satellite
   Fiber Radio
   PPP
   HTTP SMTP
   FTP
   NFS Skype
   Pigeons
   Chat
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. 1. containers
   2. hyperspeed
   3. starship
   

   View Slide

8. 1. containers
   2. hyperspeed
   3. starship
   4. car
   

   View Slide

9. 1. containers
   2. hyperspeed
   3. starship
   4. car
   

   View Slide

10. 1. containers
    2. hyperspeed
    3. starship
    

    View Slide

11. View Slide

12. represent a program
    

    View Slide

13. have everything needed to run
    

    View Slide

14. have everything needed to run
    (sort of)
    

    View Slide

15. containers have flavors
    

    View Slide

16. just a bunch of files
    

    View Slide

17. just a bunch of files
    & standard protocols to run them
    

    View Slide

18. protocols!
    

    View Slide

19. protocols!
    runC
    Open Container
    Initiative
    

    View Slide

20. some containers are small
    

    View Slide

21. some containers are big
    

    View Slide

22. some containers are big
    ~ 50 MB
    ~ 300 MB
    

    View Slide

23. some containers are HUGE
    

    View Slide

24. some containers are HUGE
    ~ 5 GB
    ~ 10 GB
    

    View Slide

25. View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. View Slide

35. View Slide

36. View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. View Slide

42. View Slide

43. View Slide

44. View Slide

45. View Slide

46. View Slide

47. View Slide

48. View Slide

49. View Slide

50. View Slide

51. View Slide

52. View Slide

53. View Slide

54. View Slide

55. View Slide

56. View Slide

57. 1. containers
    2. hyperspeed
    3. starship
    

    View Slide

58. View Slide

59. View Slide

60. What is IPFS?
    

    View Slide

61. A protocol to upgrade the web
    

    View Slide

62. A protocol to upgrade the web
    

    View Slide

63. View Slide

64. 2,387,990,609
    

    View Slide

65. 2,387,990,609
    477.6 PB
    

    View Slide

66. View Slide

67. View Slide

68. View Slide

69. View Slide

70. View Slide

71. View Slide

72. A protocol to upgrade the web
    

    View Slide

73. Permanent
    Distributed Safer
    Smarter
    Offline
    Faster
    

    View Slide

74. View Slide

75. hyperlinks
    

    View Slide

76. hyperlinks media
    

    View Slide

77. a hypermedia transport protocol
    HTTP
    

    View Slide

78. a hypermedia transport protocol
    HTTP IPFS
    

    View Slide

79. View Slide

80. SFS
    web
    DHT
    +
    

    View Slide

81. routing
    network
    exchange
    merkledag
    naming
    applications
    IPNS
    DNS Namecoin
    Bitswap HTTP
    BitTorrent
    Kad DHT
    Chord mDNS
    Gossip Delegated
    FTP
    TOR
    QUIC TOR
    TCP
    uTP WebRTC WebSockets
    Git Bitcoin
    VLC
    Etherpad Chat
    I2P
    I2P
    CJDNS UDT
    

    View Slide

82. View Slide

83. View Slide

84. View Slide

85. View Slide

86. View Slide

87. View Slide

88. cvs/svn
    

    View Slide

89. cvs/svn
    

    View Slide

90. cvs/svn
    

    View Slide

91. View Slide

92. View Slide

93. View Slide

94. View Slide

95. View Slide

96. View Slide

97. View Slide

98. View Slide

99. just a bunch of files
    

    View Slide

100. just a bunch of files
     

     View Slide

101. just a bunch of files
     

     View Slide

102. just a bunch of files
     

     View Slide

103. just a bunch of files
     

     View Slide

104. just a bunch of files
     

     View Slide

105. just a bunch of files
     

     View Slide

106. just a bunch of files
     (tar) (tar) (tar)
     

     View Slide

107. just a bunch of files
     (tar) (tar) (tar)
     

     View Slide

108. 1. containers
     2. hyperspeed
     3. starship
     

     View Slide

109. shp - starship
     container hypertransport
     

     View Slide

110. shp - starship
     container hypertransport
     

     View Slide

111. shp - starship
     container hypertransport
     

     View Slide

112. shp - starship
     container hypertransport
     

     View Slide

113. View Slide

114. View Slide

115. DEMO
     

     View Slide

116. 1. as a docker/registry storage driver
     2. as a docker/docker graph driver
     3. standalone (load/save)
     dkr-shp
     

     View Slide

117. jbenet @ earth : ~ > dkr-shp
     shp - containers at hyperspeed
     shp (starship) uses IPFS, a p2p merkle-dag transport, to
     move around container images.
     This is dkr-shp, the docker-specific tool for shp.
     COMMANDS
     dkr-shp init initialize a shp station
     dkr-shp start start the shp station
     dkr-shp push publish given docker image on this host
     dkr-shp pull pull down docker image, add it to docker
     

     View Slide

118. jbenet @ earth : ~ > dkr-shp init
     initialized shp station (registry) at /var/lib/shp
     

     View Slide

119. jbenet @ earth : ~ > dkr-shp init
     initialized shp station (registry) at /var/lib/shp
     jbenet @ earth : ~ > dkr-shp start
     started shp station (registry) 951f267861f99a79
     

     View Slide

120. jbenet @ earth : ~ > dkr-shp init
     initialized shp station (registry) at /var/lib/shp
     jbenet @ earth : ~ > dkr-shp start
     started shp station (registry) 951f267861f99a79
     jbenet @ earth : ~ > docker images | grep ipfs
     jbenet/go-ipfs latest c1d9bf8d8844 4 hours ago 29.76 MB
     thelinuxkid/ipfs-reg latest 8bd6f334dca4 2 days ago 254.2 MB
     

     View Slide

121. jbenet @ earth : ~ > dkr-shp init
     initialized shp station (registry) at /var/lib/shp
     jbenet @ earth : ~ > dkr-shp start
     started shp station (registry) 951f267861f99a79
     jbenet @ earth : ~ > docker images | grep ipfs
     jbenet/go-ipfs latest c1d9bf8d8844 4 hours ago 29.76 MB
     thelinuxkid/ipfs-reg latest 8bd6f334dca4 2 days ago 254.2 MB
     jbenet @ earth : ~ > dkr-shp push c1d9bf8d8844
     published as /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844
     

     View Slide

122. jbenet @ earth : ~ > dkr-shp init
     initialized shp station (registry) at /var/lib/shp
     jbenet @ earth : ~ > dkr-shp start
     started shp station (registry) 951f267861f99a79
     jbenet @ earth : ~ > docker images | grep ipfs
     jbenet/go-ipfs latest c1d9bf8d8844 4 hours ago 29.76 MB
     thelinuxkid/ipfs-reg latest 8bd6f334dca4 2 days ago 254.2 MB
     jbenet @ earth : ~ > dkr-shp push c1d9bf8d8844
     published as /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844
     jbenet @ earth : ~ > dkr-shp push jbenet/go-ipfs
     published as /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/jbenet/go-ipfs
     

     View Slide

123. jbenet @ mars : ~ > dkr-shp init
     initialized shp station (registry) at /var/lib/shp
     

     View Slide

124. jbenet @ mars : ~ > dkr-shp init
     initialized shp station (registry) at /var/lib/shp
     jbenet @ mars : ~ > dkr-shp start
     started shp station (registry) eac13c827004cf7a
     

     View Slide

125. jbenet @ mars : ~ > dkr-shp init
     initialized shp station (registry) at /var/lib/shp
     jbenet @ mars : ~ > dkr-shp start
     started shp station (registry) eac13c827004cf7a
     jbenet @ mars : ~ > dkr-shp pull c1d9bf8d8844
     pulled /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844
     

     View Slide

126. jbenet @ mars : ~ > dkr-shp init
     initialized shp station (registry) at /var/lib/shp
     jbenet @ mars : ~ > dkr-shp start
     started shp station (registry) eac13c827004cf7a
     jbenet @ mars : ~ > dkr-shp pull c1d9bf8d8844
     pulled /ipfs/QmNsF3nR6kTz3PnKWH1uZP97ESYsLP9uJVqV9EpNfzRKSk/c1d9bf8d8844
     jbenet @ mars : ~ > dkr-shp peers
     /ip4/104.236.32.22/tcp/4006/ipfs/QmNxzC53nTdTJaFivaev5mu2zFJmzcAAfQXJqTLarB8omm
     /ip4/162.243.139.61/tcp/4006/ipfs/QmNaVrdSKnZ8qpUhhGJeJuo7AjMHJBDE2xSWMvBau34g8P
     /ip4/104.236.176.52/tcp/4001/ipfs/QmSoLnSGccFuZQJzRadHn95W2CrSFmZuTdDWP8HXaHca9z
     /ip4/128.199.219.111/tcp/4001/ipfs/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu
     /ip4/178.62.158.247/tcp/4001/ipfs/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd
     /ip4/178.62.61.185/tcp/4001/ipfs/QmSoLMeWqB7YGVLJN3pNLQpmmEk35v6wYtsMGLzSr5QBU3
     

     View Slide

127. jbenet @ mars : ~ > dkr-shp addrs
     /ip4/127.0.0.1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
     /ip4/172.17.0.25/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
     /ip6/::1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
     

     View Slide

128. jbenet @ mars : ~ > dkr-shp addrs
     /ip4/127.0.0.1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
     /ip4/172.17.0.25/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
     /ip6/::1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
     jbenet @ mars : ~ > dkr-shp connect $addr
     

     View Slide

129. jbenet @ mars : ~ > dkr-shp addrs
     /ip4/127.0.0.1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
     /ip4/172.17.0.25/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
     /ip6/::1/tcp/4001/ipfs/QmUm6cjdSw7GFUVCxDoos9S5ikAMcjtug7J5syw74AovLH
     jbenet @ mars : ~ > dkr-shp connect $addr
     jbenet @ mars : ~ > # setup in a completely private network
     jbenet @ mars : ~ > dkr-shp init --net QmPzugFx3ysr8BujxZ3dkBiRYnuS2D63ndkAdq3W7J
     initialized shp station (registry) at /var/lib/shp
     

     View Slide

130. ipfs.io
     @juanbenet
     Protocol Labs
     Containers @ Hyperspeed!
     

     View Slide