Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building an API with Go at Microco.sm

Building an API with Go at Microco.sm

18k lines of Go code later, I decided to condense some of what I'd learned in the last six months into a half-hour talk. Given at Go London User Group, July 2013

Matt Cottingham

July 10, 2013
Tweet

More Decks by Matt Cottingham

Other Decks in Programming

Transcript

  1. APIs at
    Matt Cottingham
    http://microco.sm
    [email protected]
    @mattrco
    Image credit: Renée French

    View full-size slide

  2. Platform overview
    ● Hosted platform for
    community forums
    ● API is 18k lines of Go code
    ● API documentation: http:
    //microcosm-cc.github.io
    ● Alpha site:
    http://meta.microco.sm
    Microco.sm

    View full-size slide

  3. Engineering overview
    Requirements
    ● API does everything: all clients
    (including ours) built on it
    ● Performance constraints (sub
    50ms response times)
    ● Runtime configurable
    ● Scheduled tasks
    ● Up-front schema design
    ● Pluggable authentication
    ● Websocket support
    API
    DATA
    CLIENT
    AUTH

    View full-size slide

  4. Why Go?
    Plusses Potential problems?
    ● Third-party packages (e.g.
    oAuth, database support)
    ● How much will change around
    1.1?
    ● Who's using it? Will we be able
    to hire Go programmers?
    ● Type system
    ● Powerful standard library
    ● Small core language, easy
    to learn
    ● Concurrency primitives
    (this is how things looked before I knew much about Go, i.
    e. six months ago)

    View full-size slide

  5. Implementation
    Stack
    ● PostgreSQL 9.3
    ● Go 1.1 (we were following tip up to 1.1. We'll upgrade to 1.1.1 shortly)
    ● Memcached
    ● Nginx (TLS, SPDY, gzip, etc.)

    View full-size slide

  6. Implementation
    ● Request processor
    inspired by goweb
    ● gorilla/mux: request
    routing
    ● lib/pq: pure Go
    PostgreSQL driver
    Go packages we use
    ● Memcached client:
    bradfitz/gomemcache
    ● Cron: robfig/cron
    ● Configuration:
    laurent22/toml-go

    View full-size slide

  7. Example route declarations
    Implementation

    View full-size slide

  8. Implementation
    Simple request Handler (switch on HTTP method)

    View full-size slide

  9. Implementation
    Basic controller

    View full-size slide

  10. Implementation
    Persistence
    ● What about struct-to-database-table mappings?
    ● Several ORM options, e.g.
    ● https://github.com/coopernurse/gorp/
    ● https://github.com/eaigner/hood
    ● but with lib/pq we just talk to PostgreSQL through prepared statements
    (more maintenance but finer query control)

    View full-size slide

  11. Implementation
    Storing data (receiver enables `m.Insert()` syntax)

    View full-size slide

  12. Implementation
    Caveats/compromises
    ● Documented examples are quite sparse for database access
    ● database/sql could not (and should not) model all column datatypes,
    so some are provided by lib/pq
    ● Dealing with SQL NULL is quite simple, `thing.Valid` idiom:

    View full-size slide

  13. S3 file storage
    ● We use the package from Juju (by Canonical)
    ● There are lots of packages, well worth reading the source:
    ● https://launchpad.net/juju-core
    ● Go and Juju at Canonical
    Implementation

    View full-size slide

  14. Deployment
    ● `scp binary server` (yep...)
    ● upstart to manage processes (may move to supervisord)
    ● No graceful restarts. There's some work going on for this, but we'll
    likely just remove nodes from the load balancer, restart, add them back
    in.
    Systems

    View full-size slide

  15. Systems
    Logging and monitoring
    ● stdlib logging package does syslog or user-defined formats
    ● fine for capturing panics etc.
    ● riemann or heka are possibilities for application monitoring
    ● must accept a payload over UDP, not Go specific

    View full-size slide

  16. Flotsame & Jetsam
    Performance
    ● Web framework benchmarks (uses Go 1.1)
    ● Benchmark source
    ● Go compares favourably in tests that look realistic
    ● We haven't (yet) tested memory usage at e.g. high concurrency levels

    View full-size slide

  17. Reading
    ● the official Go blog, Dave Cheney's blog, and go-nuts
    ● Go at Apcera
    ● Russ Cox on regex implementation is an interesting read
    ● Go Newsletter (disclaimer: by me!)
    Close

    View full-size slide