Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Engineering Report (2007)

Dug Song
November 25, 2007

Engineering Report (2007)

Presentation covering my first 6 months as VP Engineering at Zattoo

Dug Song

November 25, 2007
Tweet

More Decks by Dug Song

Other Decks in Technology

Transcript

  1. This document is confidential and is intended solely for use by its original recipient for informational
    purposes. Neither the document nor any of the information contained in this document may be
    reproduced or disclosed to other persons without the prior written approval of Zattoo Inc.
    Engineering Report
    2007
    Dug Song, VP Engineering

    View Slide

  2. 2
    Strategic Goals for 2007

    • Outlined in my first official Zattoo e-mail:
    • Build a foundation for organizational growth
    (People)
    • Foster technology innovation (Technology)
    • Improve our capacity to support the business
    (Process)
    Date: Mon, 11 Jun 2007 11:38:54 -0400
    From: [email protected]
    To: [email protected], [email protected]
    Subject: Quality, Innovation, and Organizational Excellence

    View Slide

  3. 3
    Startup Engineering
    • Engine of creation, fueled by ideas
    • Produce high achievement through intense
    team collaboration under extreme conditions
    • Balance delivery and discovery (creativity in
    the face of deadlines)
    • Hard economic result through “soft” means -
    jelled teams, inspirational work environment
    – Product/technology can be copied, teams can’t
    • Offer accountability
    – That which is not measured does not improve

    View Slide

  4. 4
    People First
    • Organize and build teams to jell
    – Application (user), infrastructure (internal),
    enterprise (ad, content, service partners)
    • Identify and backfill gaps
    • Build the engine for talent acquisition
    • Plan to develop our talent
    – Training, mentorship, management
    • Aim for 100% retention rate
    – Must preserve continuity in culture and institutional
    knowledge during high growth period
    • Keep up morale
    – Burnout is contagious, bad attitudes can kill

    View Slide

  5. 5
    Starting Concern: Rotating Door?
    • 7 departed engineers
    – spatrick, zhihengw, mtzajac, dmbutler, matt, eric,
    steiza
    • 7 full-time
    – dkonson, hschang, jdparker, jonathan, kupapatt,
    peter, wenjiew
    • 9 part-time / interns
    – alvaro, akgood, ewuch, bwtimm, halinar, jaschwei,
    kdubois, rcgrant, roycroft
    • More students than full-timers
    • What happens in September?

    View Slide

  6. 6
    Hiring Goals
    • Build a network of talent to tap into
    – Recruiting is not just about filling headcount
    – Develop relationships and standing in the
    community to draw top talent
    • Institute a t
    eam-oriented hiring process
    – Preserve the culture and organizational
    momentum, and seek to expand our potential
    – Hire for cultural fit, motivation, and talent, in order
    • Professional, reliable process to land big fish
    – Hiring process must be actively managed
    – Good people refer their friends - and also warn
    them

    View Slide

  7. 7
    Recruiting Summary
    0
    ArbCamp,
    gEECS Talk
    $1125
    (+33 days)
    2 / 1
    3 (+3?)
    11
    20
    160
    Total
    21
    UW Campus
    Recruiting
    1
    3
    46
    UM Campus
    Recruiting
    $0
    1 (Finance)
    1
    4
    SPARK
    HotShots
    UIUC HKN
    Tech Talk
    UM HKN Tech
    Fair
    M-Powered
    Career Fair
    UM SWE/TBP
    Career Fair
    Event
    $350
    0
    $100
    (1?)
    0 / 1
    3
    12
    $75
    1 / 0
    1 (+1?)
    3 / 0
    6
    21
    $600
    1 / 1
    2 (+1?)
    3 / 2
    7
    56
    Cost
    Accepted
    Offered
    Interviewed
    (FTE / Intern)
    Screened
    Recruited

    View Slide

  8. 8
    Referrals
    5 / 1
    6 (+1?) / 2
    21 / 4
    30 / 4
    Total
    1 / 0
    Paul R.
    1 / 1
    3 / 1
    Eric W.
    1 / 0
    1 / 0
    Dirt
    1 (+1?) / 0
    6 / 0
    7 / 0
    Jonathan &
    Peter
    10 (+2
    advisors)
    3 / 3
    Referred
    (FTE / Intern)
    5 / 0
    5 / 0
    7 / 0
    Dug
    0 / 1
    0 / 2
    3 / 3
    Sugih
    Accepted
    Offered
    Interviewed
    Employee

    View Slide

  9. 9
    Hiring Results
    • 4 departures reversed / retained
    – recruited interns back from Cisco, Microsoft, etc.
    • 16 full-time
    – akgood, alvaro, bdimchef, ccassell, ckurecka,
    dirt, dkonson, eric, hschang, jdparker, jonathan,
    kupapatt, mjpizz, peter, steiza, wenjiew
    – Administrative support: jsyang
    • 11 part-time / interns
    – arvindal, ewuch, bwtimm, halinar, jaschwei,
    kdubois, milesck, mzlee, roycroft, smkent,
    trhaynes
    • Doubled full-time staff, shifting balance
    – Also let 1 intern go - keeping a high bar for talent

    View Slide

  10. 10
    Hiring Lessons
    • Referrals are key
    – Engineers need professional networks
    – 2 recruited interns were also Sugih’s referrals -
    nice to have the top picks from the University :-)
    • Compensation is only half the equation
    – You have to pay engineers what they’re worth, not
    what they can negotiate
    – But money doesn’t motivate better creative work -
    people, projects, and environment do
    – Engineers who are thinking about money aren’t
    thinking about engineering
    • Culture, values, and ownership are key
    • Must offer opportunity for career development

    View Slide

  11. 11
    Culture, Values, Ownership
    • Onboarding process for new hires
    – Mentorship for interns
    • Tech talks, conferences, training
    – Both as attendees and speakers
    – Help engineers build their professional networks
    • Promote open-source culture and values
    – Support flex time, remote work, mentorship, social events
    – Stand on the shoulders of giants / Don’t reinvent wheels
    • Invest in the best tools for our engineers
    – Our people are always worth the investment
    – Hardware, software, books, internal systems
    • Delegate authority and responsibility
    – Grow our engineers into positions of internal and external
    leadership

    View Slide

  12. 12
    Technical Career Track
    Architecture (Architect): Develop and promote
    appropriate technology choices, tradeoffs, strategy,
    and design, reference work and documentation
    E13: Software Architect
    Competance (Practitioner): Make significant
    contributions to individual projects and acquire
    professional-level skills in software engineering
    E10: Software Engineer II
    Leadership (Leader): Provide significant leadership to
    individual projects and make regular, strong
    contributions to the company
    E11: Senior Software Engineer
    Mastery (Master): Make extraordinary contributions to
    the company and significant or extraordinary
    contributions to the field of software engineering
    E12: Principal Software Engineer
    Introductory (Apprentice): Acquire proficiency in
    fundamental skills of excellent software development
    E9: Software Engineer I
    Internship (Trainee): Practical, supervised, on-the-job
    training and evaluation for full-time hire
    E8: Technical Intern
    Role / Responsibility
    Title

    View Slide

  13. 13
    The Learning Organization
    • Build knowledge systems to raise the entire team up
    – Grow high-performing teams, not just individuals
    – Look for mentorship, management potential in every hire
    • Introduced tech talks
    – Internal talks on core Zattoo technology
    – External talks from world-class speakers (Sun, Slashdot,
    RedHat, Sony Imageworks, Arbor, etc.)
    • Introduced code, change, and design reviews
    – Develop shared standards and values
    – Ensure idiomatic, maintainable, bug-scrubbed code
    • Introduced tools for constant, transparent, and
    asynchronous communication
    – IRC, Confluence, SVN commit mails, conference bridges
    • Document and measure everything

    View Slide

  14. 14
    Supporting Growth
    • Facilities
    – Desks, chairs, whiteboards
    – Snacks and drinks, helicopters
    • Infrastructure
    – Network, VPN, VOIP, Skype, IRC, automated
    builds, local testbed, EC2 testbed and S3 failover
    – Wiki, unified LDAP auth, Zimbra (coming)
    • Communication
    – Company newsletter
    – Departmental status reports
    – Daily scrum, weekly 1-on-1 meetings (in addition
    to existing design meetings)

    View Slide

  15. 15
    Technology Goals
    • Balance delivery and discovery
    – Not everything has a deadline
    – But everything must be managed
    – Sugih is directing technical invention
    • Foster value-driven innovation
    – Market news, competitive analysis, research
    – Give people room to think and invent
    • Protect our intellectual property
    – IP and license audit started
    • Build a strong IP portfolio
    – Set up a patent program for the company

    View Slide

  16. 16
    Process
    • Manage time, cost, quality
    – Software engineering primer: Ship It!
    – New business requirements dictate more explicit project /
    release management
    – Grow project management capacity
    • Fail fast, fail forward
    – Iterative, incremental delivery
    – Iterative, collaborative design process: debug on paper (or
    the wiki) first - not software*
    – Detect and mitigate risk early
    – Construct test plans and infrastructure
    • Minimize user impact of failures
    – Deployment plans essential
    – Staged deployment
    – Offer stable AND experimental releases
    *prototyping is the exception, not the rule

    View Slide

  17. 17
    Releases in 2006
    • 21 releases in 6 months, averaging 9 days
    between releases (6 median, 35 max)
    • Rapid, reactive code-and-fix release cycle
    – No bug tracking, release branches, requirements
    • 1.0 product release churn (what was 1.0)?

    View Slide

  18. 18
    Releases in 2007
    • 18 releases in 12 months, averaging 19 days
    between releases (16 median, 53 max)
    • Process changes introduced gradually during
    GODZILLA, in the midst of major backend
    rework, new development, and new projects

    View Slide

  19. 19
    GODZILLA code churn
    EC2 testbed

    View Slide

  20. 20
    GODZILLA defect rate
    GODZILLA: 80 bugs
    DURDEN / ELMO: 10 bugs

    View Slide

  21. 21
    GODZILLA staged deployment
    3.0.8
    3.0.7
    BE
    3.0.8
    3.0.5
    DE
    3.0.8
    NO
    3.0.8
    3.0.5
    2.2.10
    CH
    3.0.8
    3.0.6
    3.0.5
    3.0.4
    3.0.2
    3.0.3
    ES
    3.0.8
    3.0.7
    3.0.5
    3.0.4
    3.0.1
    DK
    3.0.8
    3.0.7
    3.0.5
    3.0.4
    3.0.1
    3.0.0
    UK
    29
    22
    15
    8
    1
    24
    17
    10
    3
    27
    20
    13
    6
    1
    October
    September
    August

    View Slide

  22. 22
    GODZILLA lessons
    • New code == new bugs
    – Isolate codelines for stability
    – Plan releases off stable branches when needed
    • Systematically improve quality
    – Automated builds (thanks jonathan!)
    – Regression testbed (thanks steiza!)
    – Test and deployment plans
    – Code and design reviews
    – Scoped deployment
    • Manage process to parallelize execution
    – Handle interrupts, pipeline releases, hiring

    View Slide

  23. 23
    Project Portfolio Q3-4 2007
    MoU response + Trial
    CanalSat
    Req Analysis + PoC
    Plan + Kickoff
    Swisscom
    Req Analysis + Design
    Reporting
    Content
    Zurich
    CMS Code + Design
    Integration
    ccassell
    Hiring
    Design Proposals, Selection, Plan
    Agency
    CMS Cost
    Design RFP
    Website
    Design, Cost,
    Beta Plan
    Automated Builds, EPG
    Integration + Test
    A1
    Plan Impasse
    Interim EPG Plan, Code
    Req Analysis, Design, Cost
    BATMAN
    .8
    EC2 Test
    .7
    .6
    .5
    .4
    RC
    7
    .2,
    .3
    RC
    5-6
    .0,
    .1
    Deployment Plan,
    Test, RC 1-4
    Code
    GODZILLA
    .9
    UM
    UM
    Test
    UM
    Plan
    RC
    2
    RC
    1
    Integration
    Test
    Code + Testbed Setup, A1
    Plan
    SHAQ /
    UM
    .7
    .6
    RC 1-3,
    .4, .5
    Plan, Code, Test
    ELMO
    .2,
    .3
    .0,
    .1
    Plan, Code, Test
    DURDEN
    .0
    Plan, Code, Test
    CUJO
    IP Geo Req Analysis + Eval
    25
    18
    11
    4
    28
    21
    14
    7
    30
    23
    16
    9
    2
    26
    19
    12
    5
    29
    22
    15
    8
    1
    November
    October
    September
    August
    July

    View Slide

  24. 24
    Handling Interrupts
    • Country launches and updates
    – Germany (9/13)
    – Belgium (9/25)
    – Norway (10/29)
    – United Kingdom (12/19)
    – German radio stations (10/15)
    – Swiss freestyle.ch (9/22)
    – Danish channel updates (10/22 moved to 11/27)
    – German channel blackouts (11/26)
    • New projects
    – CanalSat, Swisscom
    – New technical and management challenges
    • Operational failures (35+ since 8/20)

    View Slide

  25. 25
    Operational Failures
    • Some software issues, but predominantly not
    – 12 Unknown / Undiagnosed
    – 9 Misconfiguration
    – 3 Software flaw/failure
    – 2 Unscheduled outage
    – 2 Set-top box crash
    – 1 Unscheduled reconfiguration
    – 1 Unnoticed maintenance announcement
    – 1 Provider outage
    – 1 Announced power maintenance
    – 1 Poor planning
    – 1 Insufficient repeater capacity
    – 1 Disk full
    – 1 Change in acquisition signal
    – etc.

    View Slide

  26. 26
    Plans for 2008
    • Continue to improve basic execution
    – Formalize and communicate development process
    – Improve communication paths and tools with the
    rest of the company - wiki, videoconferencing, etc.
    – Budget, measure, analyze our work
    • Focus on user experience
    – Aggressively innovate through
    preview releases
    – Improve product and service quality
    – Systematize operations
    • More organizational development
    – Training and cross-functional work to build up
    teams technical and management capability

    View Slide

  27. 27
    Teams for 2008
    Projects, releases,
    people, metrics,
    process, …
    Deployment,
    monitoring,
    facilities, capacity,
    reliability, …
    DRM, privacy,
    authn/authz, audit,
    geolocation, …
    SVC, digital ingest,
    subtitles, multi-
    channel audio, …
    Adserver, Chewie,
    EPG, reporting,
    sub/credit, …
    Modulo protocol,
    subsidy, rhost,
    STUN, stats, …
    Mac/Win/Linux
    player, website,
    social network, …
    kupapatt, umbaugh?
    Management
    dirt, fredy, kupapatt, julian, jaschwei, smkent?
    Operations
    akgood, bwtimm
    Security
    eric, kdubois, aravind, jaschwei?
    Video
    jonathan, jdparker, bdimchef, peter, dkonson,
    mjpizz?, halinar?
    Services
    wenjiew, hschang, steiza, alvaro, ewuch, roycroft
    P2P
    dkonson, peter, ccassell, ckurecka?
    UI

    View Slide

  28. 28
    Thanks!
    • Questions, comments, concerns?
    • Visit the new engineering homepage at
    http://wiki/display/eng

    View Slide