Using Go at The New York Times

The New York Times has a small team of engineers dedicated to maintaining and enhancing an internal email platform that sends and tracks billions of messages each year. JP Robinson, senior software engineer, walks through how this team has come to adopt Go for almost all back end development over the last two years.

JP covers the first Go process in production, transitioning several legacy services to Go and also a quick look at how other teams within the company are using the language.

(Presented at the golang NYC meetup. Watch the screencap.)

Transcript

1. @NYTDevs | developers.nytimes.com Go @nytimes.com JP Robinson
2. @NYTDevs | developers.nytimes.com About JP • Sr Software Engineer • email and alert platforms • Writing Go for about 2 yrs • github.com/jprobinson • @jprbnsn
3. @NYTDevs | developers.nytimes.com Go @ NYT • Paperboy • Before Go • Our first Go application • Writing bad Go and learning • Replacing legacy code with Go • Testing, building, deploying and monitoring • Other teams and Go
4. @NYTDevs | developers.nytimes.com • Est. 2002 • 40+ newsletters • Transactional emails • 3-15 Million emails/day • ~2.5 Billion emails/year • Breaking News Alert ◦ sends ~1.3 Million unique emails in ~10 mins • Today’s Headlines ◦ ~2.2 million emails every morning Paperboy - NYT’s Email Platform
5. @NYTDevs | developers.nytimes.com Paperboy’s Services • Campaign management ◦ audience + template + schedule ◦ A/B Testing • Audience management ◦ drag-drop user segmentation tool ◦ demographic, subscription, behavior data • Template management ◦ WYSIWYG template editor • Reporting ◦ delivery, clicks & opens ◦ Charts, Geo click maps, click heat maps
6. @NYTDevs | developers.nytimes.com Before Go
7. @NYTDevs | developers.nytimes.com Brian Seitz • Smart man • Introduced me to Go • Launched first Go production process • Writes Go for enigma. io now
8. @NYTDevs | developers.nytimes.com He left his Gophers!
9. @NYTDevs | developers.nytimes.com recpull • 1st production Go process • Small and simple • No database connections ◦ Picks up file of IDs ◦ Hit Recommendation Engine API ◦ Drops files on NFS • Static binary made it easy to sneak into prod • Most didn’t know, those who did didn’t mind
10. @NYTDevs | developers.nytimes.com Writing Go • My first commit ◦ set up GOPATH ◦ git clone ◦ - fmt.Sprint ◦ + fmt.Sprintf ◦ go run main.go ◦ git commit/push ◦ put it on the resume! • Decided to try it out in my free time ◦ Read from DB, hit API ◦ Serve API, write files to disk
11. @NYTDevs | developers.nytimes.com • How to not use goroutines: ◦ Oprah-style with goroutines ◦ ‘too many open files’! ◦ Learn to pool goroutines ◦ channels of channels • How to not use database/sql ◦ Oprah-style with *sql.DB ◦ RTFM http://golang.org/pkg/database/sql/#DB • Benchmarks looked good (5-7x PHP) Writing Bad Go
12. @NYTDevs | developers.nytimes.com PowerMTA and Paperboy • MTA servers ◦ connect to Email Service Providers ◦ queue/throttle/retry by domain • .csv accounting files ◦ who, what, when ◦ delivery/bounce response • TODO ◦ bounce management ◦ archiving ◦ reporting
13. @NYTDevs | developers.nytimes.com An Opportunity for Go
14. @NYTDevs | developers.nytimes.com PMTA Problems • 3 languages • Logic duplicated in Java & Python • Only 1 server can run processes at a time • Lots of state • Files ‘processing’ for hours • Lots of files to deploy and manage • Problem? wipe data and reload
15. @NYTDevs | developers.nytimes.com A Go Solution
16. @NYTDevs | developers.nytimes.com pmta-service
17. @NYTDevs | developers.nytimes.com pmta-service pros • Simple deployment • goroutines ◦ Easily fit several small tasks into 1 binary ◦ concurrency FTW • Reused logic for EMR • Smarter archival management • Distributed processing • Testing
18. @NYTDevs | developers.nytimes.com My Alerts and the Newstracker • nytimes.com/myalerts • Free article notification service • Follow Times Topics, Orgs & key phrases • Started in 2002 (before Google Alerts!) • Originally paid service, free since 2008 • ~85 million emails/year • ~150 million article links/year
19. @NYTDevs | developers.nytimes.com My Alerts
20. @NYTDevs | developers.nytimes.com My Alerts
21. @NYTDevs | developers.nytimes.com Another Opportunity for Go
22. @NYTDevs | developers.nytimes.com Newstracker Problems • 3 languages • Duplicated data structures • Lots of cron coordination • Only one server can run at a time • Slow
23. @NYTDevs | developers.nytimes.com A Go Solution
24. @NYTDevs | developers.nytimes.com newstrackerd pros • Simple deployment • goroutines • Shared structs • Distributed processing • Speed • Testing
25. @NYTDevs | developers.nytimes.com • Build management ◦ working with Hudson/Jenkins ◦ test coverage reports ◦ rpmbuild, rhnpush • Service management ◦ init.d & Puppet • Deployment ◦ yum, Puppet, Monit & Slack • Monitoring ◦ log, logrus, go-metrics, Graphite, Grafana Tools for Enterprise Adoption
26. @NYTDevs | developers.nytimes.com • Jenkins/Hudson jobs • poll Github repo and NYT dependencies • go-testcover ◦ runs go test, lint & vet over entire repo ◦ if no tests, drop a blank_test.go ◦ using t-yuki/gocover-cobertura ◦ should switch to axw/gocov ◦ push progress to Slack • auto-deploy in dev and staging Testing and Building
27. @NYTDevs | developers.nytimes.com Testing and Building
28. @NYTDevs | developers.nytimes.com Test Artifacts
29. @NYTDevs | developers.nytimes.com Test Artifacts
30. @NYTDevs | developers.nytimes.com Building RPMs
31. @NYTDevs | developers.nytimes.com Service Management • init.d bash script ◦ check/drop pid file ◦ runs as system user, sets up env variables ◦ redirects stderr to error.log for panic stack traces [paperboy-profileapi01 ~]# service paperboy-profile-api status paperboy-profile-api is running… [paperboy-profileapi01 ~]# service paperboy-profile-api stop paperboy-profile-api stopped [paperboy-profileapi01 ~]# service paperboy-profile-api status paperboy-profile-api is not running [paperboy-profileapi01 ~]# service paperboy-profile-api start paperboy-profile-api started
32. @NYTDevs | developers.nytimes.com Service Management • puppet services ◦ puppet agent - no daemon ◦ On package update, Puppet initiates a restart
33. @NYTDevs | developers.nytimes.com Deployment • EC2 ◦ ephemeral ◦ provision with puppet and run ◦ remove old boxes from LB and terminate • NYT Data Centers ◦ persistent ◦ remove, update, add, carry on • Puppet agent to start • Monit to finish (in NYT) • Slack along the way
34. @NYTDevs | developers.nytimes.com Monitoring • Most use “log” • rcrowley/go-metrics ◦ rcrowley/go-tigertonic/metrics.go ◦ Timers, status and panic counters for APIs ◦ goroutine counters ◦ Memory, CPU usage • Graphite and Grafana • Sirupsen/logrus • Looking at Sumo Logic • Nagios • Monit
35. @NYTDevs | developers.nytimes.com Paperboy’s Tech Today
36. @NYTDevs | developers.nytimes.com Go and Paperboy So Far • 4 EMR streaming jobs • 3 JSON API server stacks • 2 Report aggregation services • Transactional email RPC server • Bulk email scheduler daemon • My Alerts backend daemon • Following (beta) backend daemon • Behavior based audience service • Email confirmation service
37. @NYTDevs | developers.nytimes.com Paperboy’s Tech Tomorrow
38. @NYTDevs | developers.nytimes.com Other Teams Using Go • R&D (github.com/nytlabs) • Interactive News (github.com/newsdev) • NYT Now (github.com/nytimes) • NYT Mobile • Comments • Messaging • Data Universe • Games • Recommendation Engine • Registration Services
39. @NYTDevs | developers.nytimes.com Research & Development • github.com/nytlabs • streamtools (st-core) ◦ Explore, analyse, modify and learn from streams of data • Hive ◦ A platform for backing crowdsourcing websites, built in golang for elasticsearch • Tick ◦ An application that tracks characteristics of data-streams over time. • Colony ◦ A lightweight microservice framework for NSQ
40. @NYTDevs | developers.nytimes.com Interactive News • DevOps • github.com/newsdev • Context ◦ Securely stores and conveniently retrieves environment variables in etcd or Redis • Promise ◦ An active HTTP reverse-proxy backed by etcd • longshore ◦ build server for docker
41. @NYTDevs | developers.nytimes.com NYT Mobile • Samizdat ◦ Scala ◦ Bundles requests for mobile apps ◦ Tested with custom Go benchmarking tool ▪ uses go-metrics
42. @NYTDevs | developers.nytimes.com NYT Now • github.com/nytimes/gziphandler • m9t ◦ serves custom fields to app ◦ takes big JSON, makes little JSON
43. @NYTDevs | developers.nytimes.com Comments • Working on a complete API rewrite
44. @NYTDevs | developers.nytimes.com Messaging • New development in Go • Writing push notification processors
45. @NYTDevs | developers.nytimes.com Games • Preparing ourselves for a complete rewrite of the Games API using Go • Real time leaderboards, multiplayer collaborative puzzles, content APIs, in-app purchase support...
46. @NYTDevs | developers.nytimes.com Data Universe • Movies API • Looking to migrate existing APIs from PHP
47. @NYTDevs | developers.nytimes.com Rec Engine, Registration • Eyeing Go for upcoming projects ◦ api for whitelisting experiments ◦ reporting/dashboards ◦ data collection, aggregation and filtering ◦ infrastructure and monitoring • mainly APIs but also messaging
48. @NYTDevs | developers.nytimes.com • Deployment • Concurrency • Testing • Performance • Fun Why Go @ NYT?
49. @NYTDevs | developers.nytimes.com Thanks! Questions?
50. @NYTDevs | developers.nytimes.com We’re Hiring! developers.nytimes.com/careers