@NYTDevs | developers.nytimes.com Go @nytimes.com JP Robinson 

@NYTDevs | developers.nytimes.com About JP ● Sr Software Engineer ● email and alert platforms ● Writing Go for about 2 yrs ● github.com/jprobinson ● @jprbnsn 

@NYTDevs | developers.nytimes.com Go @ NYT ● Paperboy ● Before Go ● Our first Go application ● Writing bad Go and learning ● Replacing legacy code with Go ● Testing, building, deploying and monitoring ● Other teams and Go 

@NYTDevs | developers.nytimes.com ● Est. 2002 ● 40+ newsletters ● Transactional emails ● 3-15 Million emails/day ● ~2.5 Billion emails/year ● Breaking News Alert ○ sends ~1.3 Million unique emails in ~10 mins ● Today’s Headlines ○ ~2.2 million emails every morning Paperboy - NYT’s Email Platform 

@NYTDevs | developers.nytimes.com Paperboy’s Services ● Campaign management ○ audience + template + schedule ○ A/B Testing ● Audience management ○ drag-drop user segmentation tool ○ demographic, subscription, behavior data ● Template management ○ WYSIWYG template editor ● Reporting ○ delivery, clicks & opens ○ Charts, Geo click maps, click heat maps 

@NYTDevs | developers.nytimes.com Before Go 

@NYTDevs | developers.nytimes.com Brian Seitz ● Smart man ● Introduced me to Go ● Launched first Go production process ● Writes Go for enigma. io now 

@NYTDevs | developers.nytimes.com He left his Gophers! 

@NYTDevs | developers.nytimes.com recpull ● 1st production Go process ● Small and simple ● No database connections ○ Picks up file of IDs ○ Hit Recommendation Engine API ○ Drops files on NFS ● Static binary made it easy to sneak into prod ● Most didn’t know, those who did didn’t mind 

@NYTDevs | developers.nytimes.com Writing Go ● My first commit ○ set up GOPATH ○ git clone ○ - fmt.Sprint ○ + fmt.Sprintf ○ go run main.go ○ git commit/push ○ put it on the resume! ● Decided to try it out in my free time ○ Read from DB, hit API ○ Serve API, write files to disk 

@NYTDevs | developers.nytimes.com ● How to not use goroutines: ○ Oprah-style with goroutines ○ ‘too many open files’! ○ Learn to pool goroutines ○ channels of channels ● How to not use database/sql ○ Oprah-style with *sql.DB ○ RTFM http://golang.org/pkg/database/sql/#DB ● Benchmarks looked good (5-7x PHP) Writing Bad Go 

@NYTDevs | developers.nytimes.com PowerMTA and Paperboy ● MTA servers ○ connect to Email Service Providers ○ queue/throttle/retry by domain ● .csv accounting files ○ who, what, when ○ delivery/bounce response ● TODO ○ bounce management ○ archiving ○ reporting 

@NYTDevs | developers.nytimes.com An Opportunity for Go 

@NYTDevs | developers.nytimes.com PMTA Problems ● 3 languages ● Logic duplicated in Java & Python ● Only 1 server can run processes at a time ● Lots of state ● Files ‘processing’ for hours ● Lots of files to deploy and manage ● Problem? wipe data and reload 

@NYTDevs | developers.nytimes.com A Go Solution 

@NYTDevs | developers.nytimes.com pmta-service 

@NYTDevs | developers.nytimes.com pmta-service pros ● Simple deployment ● goroutines ○ Easily fit several small tasks into 1 binary ○ concurrency FTW ● Reused logic for EMR ● Smarter archival management ● Distributed processing ● Testing 

@NYTDevs | developers.nytimes.com My Alerts and the Newstracker ● nytimes.com/myalerts ● Free article notification service ● Follow Times Topics, Orgs & key phrases ● Started in 2002 (before Google Alerts!) ● Originally paid service, free since 2008 ● ~85 million emails/year ● ~150 million article links/year 

@NYTDevs | developers.nytimes.com My Alerts 

@NYTDevs | developers.nytimes.com My Alerts 

@NYTDevs | developers.nytimes.com Another Opportunity for Go 

@NYTDevs | developers.nytimes.com Newstracker Problems ● 3 languages ● Duplicated data structures ● Lots of cron coordination ● Only one server can run at a time ● Slow 

@NYTDevs | developers.nytimes.com A Go Solution 

@NYTDevs | developers.nytimes.com newstrackerd pros ● Simple deployment ● goroutines ● Shared structs ● Distributed processing ● Speed ● Testing 

@NYTDevs | developers.nytimes.com ● Build management ○ working with Hudson/Jenkins ○ test coverage reports ○ rpmbuild, rhnpush ● Service management ○ init.d & Puppet ● Deployment ○ yum, Puppet, Monit & Slack ● Monitoring ○ log, logrus, go-metrics, Graphite, Grafana Tools for Enterprise Adoption 

@NYTDevs | developers.nytimes.com ● Jenkins/Hudson jobs ● poll Github repo and NYT dependencies ● go-testcover ○ runs go test, lint & vet over entire repo ○ if no tests, drop a blank_test.go ○ using t-yuki/gocover-cobertura ○ should switch to axw/gocov ○ push progress to Slack ● auto-deploy in dev and staging Testing and Building 

@NYTDevs | developers.nytimes.com Testing and Building 

@NYTDevs | developers.nytimes.com Test Artifacts 

@NYTDevs | developers.nytimes.com Test Artifacts 

@NYTDevs | developers.nytimes.com Building RPMs 

@NYTDevs | developers.nytimes.com Service Management ● init.d bash script ○ check/drop pid file ○ runs as system user, sets up env variables ○ redirects stderr to error.log for panic stack traces [paperboy-profileapi01 ~]# service paperboy-profile-api status paperboy-profile-api is running… [paperboy-profileapi01 ~]# service paperboy-profile-api stop paperboy-profile-api stopped [paperboy-profileapi01 ~]# service paperboy-profile-api status paperboy-profile-api is not running [paperboy-profileapi01 ~]# service paperboy-profile-api start paperboy-profile-api started 

@NYTDevs | developers.nytimes.com Service Management ● puppet services ○ puppet agent - no daemon ○ On package update, Puppet initiates a restart 

@NYTDevs | developers.nytimes.com Deployment ● EC2 ○ ephemeral ○ provision with puppet and run ○ remove old boxes from LB and terminate ● NYT Data Centers ○ persistent ○ remove, update, add, carry on ● Puppet agent to start ● Monit to finish (in NYT) ● Slack along the way 

@NYTDevs | developers.nytimes.com Monitoring ● Most use “log” ● rcrowley/go-metrics ○ rcrowley/go-tigertonic/metrics.go ○ Timers, status and panic counters for APIs ○ goroutine counters ○ Memory, CPU usage ● Graphite and Grafana ● Sirupsen/logrus ● Looking at Sumo Logic ● Nagios ● Monit 

@NYTDevs | developers.nytimes.com Paperboy’s Tech Today 

@NYTDevs | developers.nytimes.com Go and Paperboy So Far ● 4 EMR streaming jobs ● 3 JSON API server stacks ● 2 Report aggregation services ● Transactional email RPC server ● Bulk email scheduler daemon ● My Alerts backend daemon ● Following (beta) backend daemon ● Behavior based audience service ● Email confirmation service 

@NYTDevs | developers.nytimes.com Paperboy’s Tech Tomorrow 

@NYTDevs | developers.nytimes.com Other Teams Using Go ● R&D (github.com/nytlabs) ● Interactive News (github.com/newsdev) ● NYT Now (github.com/nytimes) ● NYT Mobile ● Comments ● Messaging ● Data Universe ● Games ● Recommendation Engine ● Registration Services 

@NYTDevs | developers.nytimes.com Research & Development ● github.com/nytlabs ● streamtools (st-core) ○ Explore, analyse, modify and learn from streams of data ● Hive ○ A platform for backing crowdsourcing websites, built in golang for elasticsearch ● Tick ○ An application that tracks characteristics of data-streams over time. ● Colony ○ A lightweight microservice framework for NSQ 

@NYTDevs | developers.nytimes.com Interactive News ● DevOps ● github.com/newsdev ● Context ○ Securely stores and conveniently retrieves environment variables in etcd or Redis ● Promise ○ An active HTTP reverse-proxy backed by etcd ● longshore ○ build server for docker 

@NYTDevs | developers.nytimes.com NYT Mobile ● Samizdat ○ Scala ○ Bundles requests for mobile apps ○ Tested with custom Go benchmarking tool ■ uses go-metrics 

@NYTDevs | developers.nytimes.com NYT Now ● github.com/nytimes/gziphandler ● m9t ○ serves custom fields to app ○ takes big JSON, makes little JSON 

@NYTDevs | developers.nytimes.com Comments ● Working on a complete API rewrite 

@NYTDevs | developers.nytimes.com Messaging ● New development in Go ● Writing push notification processors 

@NYTDevs | developers.nytimes.com Games ● Preparing ourselves for a complete rewrite of the Games API using Go ● Real time leaderboards, multiplayer collaborative puzzles, content APIs, in-app purchase support... 

@NYTDevs | developers.nytimes.com Data Universe ● Movies API ● Looking to migrate existing APIs from PHP 

@NYTDevs | developers.nytimes.com Rec Engine, Registration ● Eyeing Go for upcoming projects ○ api for whitelisting experiments ○ reporting/dashboards ○ data collection, aggregation and filtering ○ infrastructure and monitoring ● mainly APIs but also messaging 

@NYTDevs | developers.nytimes.com ● Deployment ● Concurrency ● Testing ● Performance ● Fun Why Go @ NYT? 

@NYTDevs | developers.nytimes.com Thanks! Questions? 

@NYTDevs | developers.nytimes.com We’re Hiring! developers.nytimes.com/careers