Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building an API with Go at Microco.sm

Building an API with Go at Microco.sm

18k lines of Go code later, I decided to condense some of what I'd learned in the last six months into a half-hour talk. Given at Go London User Group, July 2013

Matt Cottingham

July 10, 2013

More Decks by Matt Cottingham

Other Decks in Programming


  1. Platform overview • Hosted platform for community forums • API

    is 18k lines of Go code • API documentation: http: //microcosm-cc.github.io • Alpha site: http://meta.microco.sm Microco.sm
  2. Engineering overview Requirements • API does everything: all clients (including

    ours) built on it • Performance constraints (sub 50ms response times) • Runtime configurable • Scheduled tasks • Up-front schema design • Pluggable authentication • Websocket support API DATA CLIENT AUTH
  3. Why Go? Plusses Potential problems? • Third-party packages (e.g. oAuth,

    database support) • How much will change around 1.1? • Who's using it? Will we be able to hire Go programmers? • Type system • Powerful standard library • Small core language, easy to learn • Concurrency primitives (this is how things looked before I knew much about Go, i. e. six months ago)
  4. Implementation Stack • PostgreSQL 9.3 • Go 1.1 (we were

    following tip up to 1.1. We'll upgrade to 1.1.1 shortly) • Memcached • Nginx (TLS, SPDY, gzip, etc.)
  5. Implementation • Request processor inspired by goweb • gorilla/mux: request

    routing • lib/pq: pure Go PostgreSQL driver Go packages we use • Memcached client: bradfitz/gomemcache • Cron: robfig/cron • Configuration: laurent22/toml-go
  6. Implementation Persistence • What about struct-to-database-table mappings? • Several ORM

    options, e.g. • https://github.com/coopernurse/gorp/ • https://github.com/eaigner/hood • but with lib/pq we just talk to PostgreSQL through prepared statements (more maintenance but finer query control)
  7. Implementation Caveats/compromises • Documented examples are quite sparse for database

    access • database/sql could not (and should not) model all column datatypes, so some are provided by lib/pq • Dealing with SQL NULL is quite simple, `thing.Valid` idiom:
  8. S3 file storage • We use the package from Juju

    (by Canonical) • There are lots of packages, well worth reading the source: • https://launchpad.net/juju-core • Go and Juju at Canonical Implementation
  9. Deployment • `scp binary server` (yep...) • upstart to manage

    processes (may move to supervisord) • No graceful restarts. There's some work going on for this, but we'll likely just remove nodes from the load balancer, restart, add them back in. Systems
  10. Systems Logging and monitoring • stdlib logging package does syslog

    or user-defined formats • fine for capturing panics etc. • riemann or heka are possibilities for application monitoring • must accept a payload over UDP, not Go specific
  11. Flotsame & Jetsam Performance • Web framework benchmarks (uses Go

    1.1) • Benchmark source • Go compares favourably in tests that look realistic • We haven't (yet) tested memory usage at e.g. high concurrency levels
  12. Reading • the official Go blog, Dave Cheney's blog, and

    go-nuts • Go at Apcera • Russ Cox on regex implementation is an interesting read • Go Newsletter (disclaimer: by me!) Close