Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Go at The New York Times

Using Go at The New York Times

The New York Times has a small team of engineers dedicated to maintaining and enhancing an internal email platform that sends and tracks billions of messages each year. JP Robinson, senior software engineer, walks through how this team has come to adopt Go for almost all back end development over the last two years.

JP covers the first Go process in production, transitioning several legacy services to Go and also a quick look at how other teams within the company are using the language.

(Presented at the golang NYC meetup. Watch the screencap.)

More Decks by The New York Times Developers

Other Decks in Technology

Transcript

  1. @NYTDevs | developers.nytimes.com About JP • Sr Software Engineer •

    email and alert platforms • Writing Go for about 2 yrs • github.com/jprobinson • @jprbnsn
  2. @NYTDevs | developers.nytimes.com Go @ NYT • Paperboy • Before

    Go • Our first Go application • Writing bad Go and learning • Replacing legacy code with Go • Testing, building, deploying and monitoring • Other teams and Go
  3. @NYTDevs | developers.nytimes.com • Est. 2002 • 40+ newsletters •

    Transactional emails • 3-15 Million emails/day • ~2.5 Billion emails/year • Breaking News Alert ◦ sends ~1.3 Million unique emails in ~10 mins • Today’s Headlines ◦ ~2.2 million emails every morning Paperboy - NYT’s Email Platform
  4. @NYTDevs | developers.nytimes.com Paperboy’s Services • Campaign management ◦ audience

    + template + schedule ◦ A/B Testing • Audience management ◦ drag-drop user segmentation tool ◦ demographic, subscription, behavior data • Template management ◦ WYSIWYG template editor • Reporting ◦ delivery, clicks & opens ◦ Charts, Geo click maps, click heat maps
  5. @NYTDevs | developers.nytimes.com Brian Seitz • Smart man • Introduced

    me to Go • Launched first Go production process • Writes Go for enigma. io now
  6. @NYTDevs | developers.nytimes.com recpull • 1st production Go process •

    Small and simple • No database connections ◦ Picks up file of IDs ◦ Hit Recommendation Engine API ◦ Drops files on NFS • Static binary made it easy to sneak into prod • Most didn’t know, those who did didn’t mind
  7. @NYTDevs | developers.nytimes.com Writing Go • My first commit ◦

    set up GOPATH ◦ git clone ◦ - fmt.Sprint ◦ + fmt.Sprintf ◦ go run main.go ◦ git commit/push ◦ put it on the resume! • Decided to try it out in my free time ◦ Read from DB, hit API ◦ Serve API, write files to disk
  8. @NYTDevs | developers.nytimes.com • How to not use goroutines: ◦

    Oprah-style with goroutines ◦ ‘too many open files’! ◦ Learn to pool goroutines ◦ channels of channels • How to not use database/sql ◦ Oprah-style with *sql.DB ◦ RTFM http://golang.org/pkg/database/sql/#DB • Benchmarks looked good (5-7x PHP) Writing Bad Go
  9. @NYTDevs | developers.nytimes.com PowerMTA and Paperboy • MTA servers ◦

    connect to Email Service Providers ◦ queue/throttle/retry by domain • .csv accounting files ◦ who, what, when ◦ delivery/bounce response • TODO ◦ bounce management ◦ archiving ◦ reporting
  10. @NYTDevs | developers.nytimes.com PMTA Problems • 3 languages • Logic

    duplicated in Java & Python • Only 1 server can run processes at a time • Lots of state • Files ‘processing’ for hours • Lots of files to deploy and manage • Problem? wipe data and reload
  11. @NYTDevs | developers.nytimes.com pmta-service pros • Simple deployment • goroutines

    ◦ Easily fit several small tasks into 1 binary ◦ concurrency FTW • Reused logic for EMR • Smarter archival management • Distributed processing • Testing
  12. @NYTDevs | developers.nytimes.com My Alerts and the Newstracker • nytimes.com/myalerts

    • Free article notification service • Follow Times Topics, Orgs & key phrases • Started in 2002 (before Google Alerts!) • Originally paid service, free since 2008 • ~85 million emails/year • ~150 million article links/year
  13. @NYTDevs | developers.nytimes.com Newstracker Problems • 3 languages • Duplicated

    data structures • Lots of cron coordination • Only one server can run at a time • Slow
  14. @NYTDevs | developers.nytimes.com newstrackerd pros • Simple deployment • goroutines

    • Shared structs • Distributed processing • Speed • Testing
  15. @NYTDevs | developers.nytimes.com • Build management ◦ working with Hudson/Jenkins

    ◦ test coverage reports ◦ rpmbuild, rhnpush • Service management ◦ init.d & Puppet • Deployment ◦ yum, Puppet, Monit & Slack • Monitoring ◦ log, logrus, go-metrics, Graphite, Grafana Tools for Enterprise Adoption
  16. @NYTDevs | developers.nytimes.com • Jenkins/Hudson jobs • poll Github repo

    and NYT dependencies • go-testcover ◦ runs go test, lint & vet over entire repo ◦ if no tests, drop a blank_test.go ◦ using t-yuki/gocover-cobertura ◦ should switch to axw/gocov ◦ push progress to Slack • auto-deploy in dev and staging Testing and Building
  17. @NYTDevs | developers.nytimes.com Service Management • init.d bash script ◦

    check/drop pid file ◦ runs as system user, sets up env variables ◦ redirects stderr to error.log for panic stack traces [paperboy-profileapi01 ~]# service paperboy-profile-api status paperboy-profile-api is running… [paperboy-profileapi01 ~]# service paperboy-profile-api stop paperboy-profile-api stopped [paperboy-profileapi01 ~]# service paperboy-profile-api status paperboy-profile-api is not running [paperboy-profileapi01 ~]# service paperboy-profile-api start paperboy-profile-api started
  18. @NYTDevs | developers.nytimes.com Service Management • puppet services ◦ puppet

    agent - no daemon ◦ On package update, Puppet initiates a restart
  19. @NYTDevs | developers.nytimes.com Deployment • EC2 ◦ ephemeral ◦ provision

    with puppet and run ◦ remove old boxes from LB and terminate • NYT Data Centers ◦ persistent ◦ remove, update, add, carry on • Puppet agent to start • Monit to finish (in NYT) • Slack along the way
  20. @NYTDevs | developers.nytimes.com Monitoring • Most use “log” • rcrowley/go-metrics

    ◦ rcrowley/go-tigertonic/metrics.go ◦ Timers, status and panic counters for APIs ◦ goroutine counters ◦ Memory, CPU usage • Graphite and Grafana • Sirupsen/logrus • Looking at Sumo Logic • Nagios • Monit
  21. @NYTDevs | developers.nytimes.com Go and Paperboy So Far • 4

    EMR streaming jobs • 3 JSON API server stacks • 2 Report aggregation services • Transactional email RPC server • Bulk email scheduler daemon • My Alerts backend daemon • Following (beta) backend daemon • Behavior based audience service • Email confirmation service
  22. @NYTDevs | developers.nytimes.com Other Teams Using Go • R&D (github.com/nytlabs)

    • Interactive News (github.com/newsdev) • NYT Now (github.com/nytimes) • NYT Mobile • Comments • Messaging • Data Universe • Games • Recommendation Engine • Registration Services
  23. @NYTDevs | developers.nytimes.com Research & Development • github.com/nytlabs • streamtools

    (st-core) ◦ Explore, analyse, modify and learn from streams of data • Hive ◦ A platform for backing crowdsourcing websites, built in golang for elasticsearch • Tick ◦ An application that tracks characteristics of data-streams over time. • Colony ◦ A lightweight microservice framework for NSQ
  24. @NYTDevs | developers.nytimes.com Interactive News • DevOps • github.com/newsdev •

    Context ◦ Securely stores and conveniently retrieves environment variables in etcd or Redis • Promise ◦ An active HTTP reverse-proxy backed by etcd • longshore ◦ build server for docker
  25. @NYTDevs | developers.nytimes.com NYT Mobile • Samizdat ◦ Scala ◦

    Bundles requests for mobile apps ◦ Tested with custom Go benchmarking tool ▪ uses go-metrics
  26. @NYTDevs | developers.nytimes.com NYT Now • github.com/nytimes/gziphandler • m9t ◦

    serves custom fields to app ◦ takes big JSON, makes little JSON
  27. @NYTDevs | developers.nytimes.com Games • Preparing ourselves for a complete

    rewrite of the Games API using Go • Real time leaderboards, multiplayer collaborative puzzles, content APIs, in-app purchase support...
  28. @NYTDevs | developers.nytimes.com Rec Engine, Registration • Eyeing Go for

    upcoming projects ◦ api for whitelisting experiments ◦ reporting/dashboards ◦ data collection, aggregation and filtering ◦ infrastructure and monitoring • mainly APIs but also messaging