Building an API with Go at

Building an API with Go at

18k lines of Go code later, I decided to condense some of what I'd learned in the last six months into a half-hour talk. Given at Go London User Group, July 2013


Matt Cottingham

July 10, 2013


  1. APIs at Matt Cottingham @mattrco Image credit: Renée

  2. Platform overview • Hosted platform for community forums • API

    is 18k lines of Go code • API documentation: http: // • Alpha site:
  3. Engineering overview Requirements • API does everything: all clients (including

    ours) built on it • Performance constraints (sub 50ms response times) • Runtime configurable • Scheduled tasks • Up-front schema design • Pluggable authentication • Websocket support API DATA CLIENT AUTH
  4. Why Go? Plusses Potential problems? • Third-party packages (e.g. oAuth,

    database support) • How much will change around 1.1? • Who's using it? Will we be able to hire Go programmers? • Type system • Powerful standard library • Small core language, easy to learn • Concurrency primitives (this is how things looked before I knew much about Go, i. e. six months ago)
  5. Implementation Stack • PostgreSQL 9.3 • Go 1.1 (we were

    following tip up to 1.1. We'll upgrade to 1.1.1 shortly) • Memcached • Nginx (TLS, SPDY, gzip, etc.)
  6. Implementation • Request processor inspired by goweb • gorilla/mux: request

    routing • lib/pq: pure Go PostgreSQL driver Go packages we use • Memcached client: bradfitz/gomemcache • Cron: robfig/cron • Configuration: laurent22/toml-go
  7. Example route declarations Implementation

  8. Implementation Simple request Handler (switch on HTTP method)

  9. Implementation Basic controller

  10. Implementation Persistence • What about struct-to-database-table mappings? • Several ORM

    options, e.g. • • • but with lib/pq we just talk to PostgreSQL through prepared statements (more maintenance but finer query control)
  11. Implementation Storing data (receiver enables `m.Insert()` syntax)

  12. Implementation Caveats/compromises • Documented examples are quite sparse for database

    access • database/sql could not (and should not) model all column datatypes, so some are provided by lib/pq • Dealing with SQL NULL is quite simple, `thing.Valid` idiom:
  13. S3 file storage • We use the package from Juju

    (by Canonical) • There are lots of packages, well worth reading the source: • • Go and Juju at Canonical Implementation
  14. Deployment • `scp binary server` (yep...) • upstart to manage

    processes (may move to supervisord) • No graceful restarts. There's some work going on for this, but we'll likely just remove nodes from the load balancer, restart, add them back in. Systems
  15. Systems Logging and monitoring • stdlib logging package does syslog

    or user-defined formats • fine for capturing panics etc. • riemann or heka are possibilities for application monitoring • must accept a payload over UDP, not Go specific
  16. Flotsame & Jetsam Performance • Web framework benchmarks (uses Go

    1.1) • Benchmark source • Go compares favourably in tests that look realistic • We haven't (yet) tested memory usage at e.g. high concurrency levels
  17. Reading • the official Go blog, Dave Cheney's blog, and

    go-nuts • Go at Apcera • Russ Cox on regex implementation is an interesting read • Go Newsletter (disclaimer: by me!) Close