Engineering Report (2007)

Presentation covering my first 6 months as VP Engineering at Zattoo

Transcript

1. This document is confidential and is intended solely for use by its original recipient for informational purposes. Neither the document nor any of the information contained in this document may be reproduced or disclosed to other persons without the prior written approval of Zattoo Inc. Engineering Report 2007 Dug Song, VP Engineering
2. 2 Strategic Goals for 2007
   • Outlined in my first official Zattoo e-mail: • Build a foundation for organizational growth (People) • Foster technology innovation (Technology) • Improve our capacity to support the business (Process) Date: Mon, 11 Jun 2007 11:38:54 -0400 From: dugsong@zattoo.com To: dev@zattoo.com, team@zattoo.com Subject: Quality, Innovation, and Organizational Excellence
3. 3 Startup Engineering • Engine of creation, fueled by ideas • Produce high achievement through intense team collaboration under extreme conditions • Balance delivery and discovery (creativity in the face of deadlines) • Hard economic result through “soft” means - jelled teams, inspirational work environment – Product/technology can be copied, teams can’t • Offer accountability – That which is not measured does not improve
4. 4 People First • Organize and build teams to jell – Application (user), infrastructure (internal), enterprise (ad, content, service partners) • Identify and backfill gaps • Build the engine for talent acquisition • Plan to develop our talent – Training, mentorship, management • Aim for 100% retention rate – Must preserve continuity in culture and institutional knowledge during high growth period • Keep up morale – Burnout is contagious, bad attitudes can kill
5. 5 Starting Concern: Rotating Door? • 7 departed engineers – spatrick, zhihengw, mtzajac, dmbutler, matt, eric, steiza • 7 full-time – dkonson, hschang, jdparker, jonathan, kupapatt, peter, wenjiew • 9 part-time / interns – alvaro, akgood, ewuch, bwtimm, halinar, jaschwei, kdubois, rcgrant, roycroft • More students than full-timers • What happens in September?
6. 6 Hiring Goals • Build a network of talent to tap into – Recruiting is not just about filling headcount – Develop relationships and standing in the community to draw top talent • Institute a t
   eam-oriented hiring process – Preserve the culture and organizational momentum, and seek to expand our potential – Hire for cultural fit, motivation, and talent, in order • Professional, reliable process to land big fish – Hiring process must be actively managed – Good people refer their friends - and also warn them
7. 7 Recruiting Summary 0 ArbCamp, gEECS Talk $1125 (+33 days) 2 / 1 3 (+3?) 11 20 160 Total 21 UW Campus Recruiting 1 3 46 UM Campus Recruiting $0 1 (Finance) 1 4 SPARK HotShots UIUC HKN Tech Talk UM HKN Tech Fair M-Powered Career Fair UM SWE/TBP Career Fair Event $350 0 $100 (1?) 0 / 1 3 12 $75 1 / 0 1 (+1?) 3 / 0 6 21 $600 1 / 1 2 (+1?) 3 / 2 7 56 Cost Accepted Offered Interviewed (FTE / Intern) Screened Recruited
8. 8 Referrals 5 / 1 6 (+1?) / 2 21 / 4 30 / 4 Total 1 / 0 Paul R. 1 / 1 3 / 1 Eric W. 1 / 0 1 / 0 Dirt 1 (+1?) / 0 6 / 0 7 / 0 Jonathan & Peter 10 (+2 advisors) 3 / 3 Referred (FTE / Intern) 5 / 0 5 / 0 7 / 0 Dug 0 / 1 0 / 2 3 / 3 Sugih Accepted Offered Interviewed Employee
9. 9 Hiring Results • 4 departures reversed / retained – recruited interns back from Cisco, Microsoft, etc. • 16 full-time – akgood, alvaro, bdimchef, ccassell, ckurecka, dirt, dkonson, eric, hschang, jdparker, jonathan, kupapatt, mjpizz, peter, steiza, wenjiew – Administrative support: jsyang • 11 part-time / interns – arvindal, ewuch, bwtimm, halinar, jaschwei, kdubois, milesck, mzlee, roycroft, smkent, trhaynes • Doubled full-time staff, shifting balance – Also let 1 intern go - keeping a high bar for talent
10. 10 Hiring Lessons • Referrals are key – Engineers need professional networks – 2 recruited interns were also Sugih’s referrals - nice to have the top picks from the University :-) • Compensation is only half the equation – You have to pay engineers what they’re worth, not what they can negotiate – But money doesn’t motivate better creative work - people, projects, and environment do – Engineers who are thinking about money aren’t thinking about engineering • Culture, values, and ownership are key • Must offer opportunity for career development
11. 11 Culture, Values, Ownership • Onboarding process for new hires – Mentorship for interns • Tech talks, conferences, training – Both as attendees and speakers – Help engineers build their professional networks • Promote open-source culture and values – Support flex time, remote work, mentorship, social events – Stand on the shoulders of giants / Don’t reinvent wheels • Invest in the best tools for our engineers – Our people are always worth the investment – Hardware, software, books, internal systems • Delegate authority and responsibility – Grow our engineers into positions of internal and external leadership
12. 12 Technical Career Track Architecture (Architect): Develop and promote appropriate technology choices, tradeoffs, strategy, and design, reference work and documentation E13: Software Architect Competance (Practitioner): Make significant contributions to individual projects and acquire professional-level skills in software engineering E10: Software Engineer II Leadership (Leader): Provide significant leadership to individual projects and make regular, strong contributions to the company E11: Senior Software Engineer Mastery (Master): Make extraordinary contributions to the company and significant or extraordinary contributions to the field of software engineering E12: Principal Software Engineer Introductory (Apprentice): Acquire proficiency in fundamental skills of excellent software development E9: Software Engineer I Internship (Trainee): Practical, supervised, on-the-job training and evaluation for full-time hire E8: Technical Intern Role / Responsibility Title
13. 13 The Learning Organization • Build knowledge systems to raise the entire team up – Grow high-performing teams, not just individuals – Look for mentorship, management potential in every hire • Introduced tech talks – Internal talks on core Zattoo technology – External talks from world-class speakers (Sun, Slashdot, RedHat, Sony Imageworks, Arbor, etc.) • Introduced code, change, and design reviews – Develop shared standards and values – Ensure idiomatic, maintainable, bug-scrubbed code • Introduced tools for constant, transparent, and asynchronous communication – IRC, Confluence, SVN commit mails, conference bridges • Document and measure everything
14. 14 Supporting Growth • Facilities – Desks, chairs, whiteboards – Snacks and drinks, helicopters • Infrastructure – Network, VPN, VOIP, Skype, IRC, automated builds, local testbed, EC2 testbed and S3 failover – Wiki, unified LDAP auth, Zimbra (coming) • Communication – Company newsletter – Departmental status reports – Daily scrum, weekly 1-on-1 meetings (in addition to existing design meetings)
15. 15 Technology Goals • Balance delivery and discovery – Not everything has a deadline – But everything must be managed – Sugih is directing technical invention • Foster value-driven innovation – Market news, competitive analysis, research – Give people room to think and invent • Protect our intellectual property – IP and license audit started • Build a strong IP portfolio – Set up a patent program for the company
16. 16 Process • Manage time, cost, quality – Software engineering primer: Ship It! – New business requirements dictate more explicit project / release management – Grow project management capacity • Fail fast, fail forward – Iterative, incremental delivery – Iterative, collaborative design process: debug on paper (or the wiki) first - not software* – Detect and mitigate risk early – Construct test plans and infrastructure • Minimize user impact of failures – Deployment plans essential – Staged deployment – Offer stable AND experimental releases *prototyping is the exception, not the rule
17. 17 Releases in 2006 • 21 releases in 6 months, averaging 9 days between releases (6 median, 35 max) • Rapid, reactive code-and-fix release cycle – No bug tracking, release branches, requirements • 1.0 product release churn (what was 1.0)?
18. 18 Releases in 2007 • 18 releases in 12 months, averaging 19 days between releases (16 median, 53 max) • Process changes introduced gradually during GODZILLA, in the midst of major backend rework, new development, and new projects
19. 19 GODZILLA code churn EC2 testbed
20. 20 GODZILLA defect rate GODZILLA: 80 bugs DURDEN / ELMO: 10 bugs
21. 21 GODZILLA staged deployment 3.0.8 3.0.7 BE 3.0.8 3.0.5 DE 3.0.8 NO 3.0.8 3.0.5 2.2.10 CH 3.0.8 3.0.6 3.0.5 3.0.4 3.0.2 3.0.3 ES 3.0.8 3.0.7 3.0.5 3.0.4 3.0.1 DK 3.0.8 3.0.7 3.0.5 3.0.4 3.0.1 3.0.0 UK 29 22 15 8 1 24 17 10 3 27 20 13 6 1 October September August
22. 22 GODZILLA lessons • New code == new bugs – Isolate codelines for stability – Plan releases off stable branches when needed • Systematically improve quality – Automated builds (thanks jonathan!) – Regression testbed (thanks steiza!) – Test and deployment plans – Code and design reviews – Scoped deployment • Manage process to parallelize execution – Handle interrupts, pipeline releases, hiring
23. 23 Project Portfolio Q3-4 2007 MoU response + Trial CanalSat Req Analysis + PoC Plan + Kickoff Swisscom Req Analysis + Design Reporting Content Zurich CMS Code + Design Integration ccassell Hiring Design Proposals, Selection, Plan Agency CMS Cost Design RFP Website Design, Cost, Beta Plan Automated Builds, EPG Integration + Test A1 Plan Impasse Interim EPG Plan, Code Req Analysis, Design, Cost BATMAN .8 EC2 Test .7 .6 .5 .4 RC 7 .2, .3 RC 5-6 .0, .1 Deployment Plan, Test, RC 1-4 Code GODZILLA .9 UM UM Test UM Plan RC 2 RC 1 Integration Test Code + Testbed Setup, A1 Plan SHAQ / UM .7 .6 RC 1-3, .4, .5 Plan, Code, Test ELMO .2, .3 .0, .1 Plan, Code, Test DURDEN .0 Plan, Code, Test CUJO IP Geo Req Analysis + Eval 25 18 11 4 28 21 14 7 30 23 16 9 2 26 19 12 5 29 22 15 8 1 November October September August July
24. 24 Handling Interrupts • Country launches and updates – Germany (9/13) – Belgium (9/25) – Norway (10/29) – United Kingdom (12/19) – German radio stations (10/15) – Swiss freestyle.ch (9/22) – Danish channel updates (10/22 moved to 11/27) – German channel blackouts (11/26) • New projects – CanalSat, Swisscom – New technical and management challenges • Operational failures (35+ since 8/20)
25. 25 Operational Failures • Some software issues, but predominantly not – 12 Unknown / Undiagnosed – 9 Misconfiguration – 3 Software flaw/failure – 2 Unscheduled outage – 2 Set-top box crash – 1 Unscheduled reconfiguration – 1 Unnoticed maintenance announcement – 1 Provider outage – 1 Announced power maintenance – 1 Poor planning – 1 Insufficient repeater capacity – 1 Disk full – 1 Change in acquisition signal – etc.
26. 26 Plans for 2008 • Continue to improve basic execution – Formalize and communicate development process – Improve communication paths and tools with the rest of the company - wiki, videoconferencing, etc. – Budget, measure, analyze our work • Focus on user experience – Aggressively innovate through
    preview releases – Improve product and service quality – Systematize operations • More organizational development – Training and cross-functional work to build up teams technical and management capability
27. 27 Teams for 2008 Projects, releases, people, metrics, process, … Deployment, monitoring, facilities, capacity, reliability, … DRM, privacy, authn/authz, audit, geolocation, … SVC, digital ingest, subtitles, multi- channel audio, … Adserver, Chewie, EPG, reporting, sub/credit, … Modulo protocol, subsidy, rhost, STUN, stats, … Mac/Win/Linux player, website, social network, … kupapatt, umbaugh? Management dirt, fredy, kupapatt, julian, jaschwei, smkent? Operations akgood, bwtimm Security eric, kdubois, aravind, jaschwei? Video jonathan, jdparker, bdimchef, peter, dkonson, mjpizz?, halinar? Services wenjiew, hschang, steiza, alvaro, ewuch, roycroft P2P dkonson, peter, ccassell, ckurecka? UI
28. 28 Thanks! • Questions, comments, concerns? • Visit the new engineering homepage at http://wiki/display/eng