We can mirror it for you, wholesale

How do you build world-class, highly scalable web applications without access to the Internet? How can you take the useful pieces of the Internet with you to a Napoleonic sea fort… and why would you even want to?!

After spending a year building a suite of open-source tools to clone large chunks of the Internet, Steve realised there are far wider applications for this than “I want to live in a fort without Internet access”, and that decoupling can be a good thing for you, your team, your infrastructure, and your software.

Transcript

1. We can mirror it for you wholesale @SteveMarshall
2. How I started mirroring Why would you mirror? Tools for mirroring What can we learn?
12. You can take it with you
15. How I started mirroring Why would you mirror? Tools for mirroring What can we learn?
16. Avoid dependencies’ downtime
17. Avoid dependencies’ downtime Slow (or no!) network
18. Avoid dependencies’ downtime Slow (or no!) network Control dependency changes
19. Avoid dependencies’ downtime Slow (or no!) network Control dependency changes Improve performance
20. How I started mirroring Why would you mirror? Tools for mirroring What can we learn?
21. Index of / localhost:8000
23. $ rsync --archive --delete \ > --compress --progress \ > $SOURCE_URI \ > $TARGET_DIR
24. $ rsync --archive --delete \ > --compress --progress \ > cpan.org::CPAN \ > /data/cpan $ rsync --archive --delete \ > --compress --progress \
25. $ rsync --archive --delete \ > --compress --progress \ > ietf.org::internet-drafts \ > /data/ietf
26. $ rsync --archive --delete \ > --compress --progress \ > ftp.funet.fi::ftp/pub/\ > mirrors/ftp.imdb.com/pub \ > /data/imdb
27. $ git clone --shared --bare \ > $SOURCE_URI \ > $TARGET_DIR $ cd $TARGET_DIR $ git update-server-info
28. $ wget $SOME_FILE
29. $ wget --mirror \ > --convert-links \ > --page-requisites \ > --no-parent \ > $SOURCE_URI
30. $ some_proprietary_cmd \ > --for $REASONS
31. $ apt-mirror # rsync wrapper $ gem mirror $ bandersnatch mirror # PyPI
33. 1) The Mirror’s IP must be registered at pear.php.net as a valid mirror 2) The Mirror must be registered as a DNS alias in the form xx.pear.php.net (us.pear.php.net, de.pear.php.net, etc.) —Mirroring pear.php.net
34. The central repository is over 180 GB and growing. To save us bandwidth and you time, mirroring the entire central repository is not allowed. —Creating Your Own [Maven] Mirror
37. How I started mirroring Why would you mirror? Tools for mirroring What can we learn?
38. WORKS LIKE MAGIC DON’T KNOW IF IT WORKS ≈
39. DON’T REINVENT THE WHEEL
40. DISTRIBUTED IS BETTER THAN CENTRALISED
42. Tools rsync.samba.org, git-scm.com, www.gnu.org/software/wget, github.com/devfort?query=cookbook Websites devfort.com, mostlyfinal.com, behabitual.com, spacelog.org