Camlistore: Your, Personal Datastore for Life

A side project in GoLang by Brad Fitzpatrick. Maintainer Mathieu Lonjaret. Reprised by Clive Boulton for Google Developer Groups https://www.meetup.com/gdgnyc/events/234963670/

Transcript

1. Camlistore.org Your, Personal Datastore for Life A side project in GoLang by Brad Fitzpatrick. Maintainer Mathieu Lonjaret. Reprised by Clive Boulton for Google Developer Groups NYC. Jan 9, 2017. https://www.meetup.com/gdgnyc/events/234963670/
2. Entrepreneurs: personal memories are priceless.
4. Demo
5. How Brad got Camlistore going Brad Fitzpatrick @bradfitz is a member of the Go team at Google, working primarily on the standard library. • Brad wanted to create a system to store all of the content he’s created, which is currently scattered across multiple sites and disks. He started hacking in this directory: $ Content Addressable Multi Level Indexed Storage = Camlistore
6. Camlistore Goals • Content-addressable • Searchable • Web UI • Extensibility: FUSE, etc. Majority of Camlistore is pure Go. (Ed. note: It’s a great repo to read and learn from.)
7. Content-addressable advantages • Sync any which way, no conflicts • (merge resolution at higher layers) • corruption verification/repair • de-duping Content Addressable: What things are named depends on their content. Two identical things have the same name. For example, the "name" or "key" for the data is the SHA-1 for the data, ala git.
8. Multi-level • blob storage • indexing • search • web UI, FUSE, CU, apps are just clients of search + blob storage Multi-Layer: The whole storage stack is built out of several layers. The blob store sits on the bottom, and only knows about bytes, and access is via the SHA-1 of those bytes. Things that you might store (Files, directories, sets, collections of tweets, social graphs, etc) build on top of the blob store by additional blobs that hold pointers to data blobs. Again, it's sort of like git. A front-end might sit on top of that abstraction.
9. Indexed • ask the indexer • tags • search = find Indexed: blobs of JSON that have a few special attributes are recognized and indexed. So, you might have a bunch of blobs with these special attributes (ie, "tag") and be able to ask the indexer "Give me all blobs with tag equal to foo", rather than having to search through the blobs directly.
10. Overview Permanode random # Set-attribute camliContent Directory JSON File Bytes File Bytes File Bytes File Bytes Directory JSON File JSON File JSON unassigned blobs object Set-attribute camliContent
11. Logically
12. Features of @camlistore • Personal Datastore for life: Like Git for Social Backup. • Security is private by default (share if you wish, more sharing in v.10) • FUSE file system • Data versioning, time travel • Snappy UI (react.js like Instagram moving to gopherjs) • Clients in JavaScript, Java, Obj-C, • Deploy Web, Phone, Home, S3, GCE (Docker) • Targets Desktop Chrome, Firefox, Safari, Edge (last two stable versions). • Targets Safari and Chrome on Android and iOS tablets (last two stable ver)
13. Why Brad chose Go for Camlistore • He considered a few other candidate languages: • Perl and Python: too slow and single-threaded • C++ and Java: tedious and not fun (slow builds) • Go? Yeah. As Brad hacked on Camlistore, he realized he needs to implement support for various formats and protocols (like mime/multipart) and improve stdlib packages (like os/exec nee exec, database/sql, etc.). Camlistore ⇔ 20 per cent time (now a side project)
14. East deployment to GCP (now) Makes firing up your own Camlistore server on GCE+MySQL+Cloud Storage a few clicks from a web app, using CoreOS, Docker, and the various Google Cloud Platform APIs.. Visit https://camlistore.org/launch/ which lets you create your own camlistore instance on GCE. The Camlistore website will say: Step 1) go to https://console.developers.google.com/project Step 2) click "Create Project". remember its name. Step 3) Enable billing. (tell Google your credit card info) [Ed note. Dev’s sign up for GCP get $x00 credit for 60 days] Step 4) Enter your project name: [ ] [ Create ] You click Create, and then we send you on an OAuth2 voyage to get access to create VMs & DBs & Buckets on your behalf, and then we create it all, including creating a minimal CoreOS VM that boots right up into a tiny Docker container just running camlistore. [Ed note, the API forms are also Camlistore open source]
15. Web UI • React → gopherjs • Infinite scroll • Lightbox • HTML history API • Search … Camlistore Android app: demo time! https://play.google.com/store/apps/details?id=org.camlistore
16. Command line tools $ • camget (blobs, trees, shares) • camput (blobs, files, dirs, shares, permanodes, claims) • camtool {init, sync, init, describe, claims…) • cammount
17. Camlistore is pure Go: (no C) • HTTP client/server, TLS, AES, … • JSON, XML • WebSockets • SMTP server (coming) • Rolling checksums • GIF/JPG/PNG en/dec, resizing, thumbnails • FUSE • Notification bus • database/sql, MySQL, Postgres, MongoDB • OpenPGP signing & verification Ed note: NYC is adoption center for Blockchain technologies, also written in GoLang
18. More @camlistore https://camlistore.org Reprised by clive boulton twitter @iC