$30 off During Our Annual Pro Sale. View Details »

Building an API with Go at Microco.sm

Building an API with Go at Microco.sm

18k lines of Go code later, I decided to condense some of what I'd learned in the last six months into a half-hour talk. Given at Go London User Group, July 2013

Matt Cottingham

July 10, 2013
Tweet

More Decks by Matt Cottingham

Other Decks in Programming

Transcript

  1. APIs at Matt Cottingham http://microco.sm matt@microcosm.cc @mattrco Image credit: Renée

    French
  2. Platform overview • Hosted platform for community forums • API

    is 18k lines of Go code • API documentation: http: //microcosm-cc.github.io • Alpha site: http://meta.microco.sm Microco.sm
  3. Engineering overview Requirements • API does everything: all clients (including

    ours) built on it • Performance constraints (sub 50ms response times) • Runtime configurable • Scheduled tasks • Up-front schema design • Pluggable authentication • Websocket support API DATA CLIENT AUTH
  4. Why Go? Plusses Potential problems? • Third-party packages (e.g. oAuth,

    database support) • How much will change around 1.1? • Who's using it? Will we be able to hire Go programmers? • Type system • Powerful standard library • Small core language, easy to learn • Concurrency primitives (this is how things looked before I knew much about Go, i. e. six months ago)
  5. Implementation Stack • PostgreSQL 9.3 • Go 1.1 (we were

    following tip up to 1.1. We'll upgrade to 1.1.1 shortly) • Memcached • Nginx (TLS, SPDY, gzip, etc.)
  6. Implementation • Request processor inspired by goweb • gorilla/mux: request

    routing • lib/pq: pure Go PostgreSQL driver Go packages we use • Memcached client: bradfitz/gomemcache • Cron: robfig/cron • Configuration: laurent22/toml-go
  7. Example route declarations Implementation

  8. Implementation Simple request Handler (switch on HTTP method)

  9. Implementation Basic controller

  10. Implementation Persistence • What about struct-to-database-table mappings? • Several ORM

    options, e.g. • https://github.com/coopernurse/gorp/ • https://github.com/eaigner/hood • but with lib/pq we just talk to PostgreSQL through prepared statements (more maintenance but finer query control)
  11. Implementation Storing data (receiver enables `m.Insert()` syntax)

  12. Implementation Caveats/compromises • Documented examples are quite sparse for database

    access • database/sql could not (and should not) model all column datatypes, so some are provided by lib/pq • Dealing with SQL NULL is quite simple, `thing.Valid` idiom:
  13. S3 file storage • We use the package from Juju

    (by Canonical) • There are lots of packages, well worth reading the source: • https://launchpad.net/juju-core • Go and Juju at Canonical Implementation
  14. Deployment • `scp binary server` (yep...) • upstart to manage

    processes (may move to supervisord) • No graceful restarts. There's some work going on for this, but we'll likely just remove nodes from the load balancer, restart, add them back in. Systems
  15. Systems Logging and monitoring • stdlib logging package does syslog

    or user-defined formats • fine for capturing panics etc. • riemann or heka are possibilities for application monitoring • must accept a payload over UDP, not Go specific
  16. Flotsame & Jetsam Performance • Web framework benchmarks (uses Go

    1.1) • Benchmark source • Go compares favourably in tests that look realistic • We haven't (yet) tested memory usage at e.g. high concurrency levels
  17. Reading • the official Go blog, Dave Cheney's blog, and

    go-nuts • Go at Apcera • Russ Cox on regex implementation is an interesting read • Go Newsletter (disclaimer: by me!) Close