Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Sysadmins view your stinkin' code

How Sysadmins view your stinkin' code

Talk given at Zendcon `12 discussing the dangers of devops in PHP and some tools to bring dev in harmony with ops.

Shawn Stratton

October 25, 2012
Tweet

More Decks by Shawn Stratton

Other Decks in Technology

Transcript

  1. How Sysadmins View Your
    Stinkin’ Code
    Shawn Stratton
    Senior Systems Engineer / Architect
    Monday, October 29, 12

    View Slide

  2. Who Am I?
    Currently work for Discovery Communications, Inc as a
    Systems Architect / Engineer.
    Became a Systems Architect 2011.
    Worked as a developer for NationalGuard.com,
    HowStuffWorks.com.
    PHP Developer since early 2000’s.
    Linux user since 1998.
    Monday, October 29, 12

    View Slide

  3. Sysadmin at Work!
    http://xkcd.com/705/
    Sysadmins are
    neurotic, OCD, and
    more than a little
    Paranoid. Beware!
    Monday, October 29, 12

    View Slide

  4. So what is a Sysadmin?
    Server monkey.
    Administrates servers to ensure uptime, stability, and
    performance.
    Responds to service outages, power outages, errors,
    etc. Often after hours.
    Monday, October 29, 12

    View Slide

  5. Great Sysadmin Qualities
    Calm during the inevitable firestorm.
    Analytical and understanding of everything that is being
    ran.
    Obsession about Single Points of Failure, Bus Factor,
    anything that impacts downtime.
    Monday, October 29, 12

    View Slide

  6. So who’s our nemesis?
    Monday, October 29, 12

    View Slide

  7. Developers! Developers!
    Developers!
    Monday, October 29, 12

    View Slide

  8. What is a Developer?
    Compiler, uses caffeine to compile business
    requirements into software.
    Constantly trying to “improve” software and add
    “functionality.”
    Monday, October 29, 12

    View Slide

  9. What’s wrong with that?
    Nothing, but...
    Sysadmins care about stability, developers
    introduce instability.
    Monday, October 29, 12

    View Slide

  10. And then there are devops...
    Monday, October 29, 12

    View Slide

  11. So what do we care about?
    Fault tolerance.
    Error handling.
    Configurability.
    Scalability.
    Minimal resource
    utilization.
    Sanity.
    Monday, October 29, 12

    View Slide

  12. Fault Tolerance
    System
    Multiple Servers.
    “Self Healing”
    systems.
    Software
    Exception handling.
    Tolerance just means try to handle errors as
    well as possible.
    Monday, October 29, 12

    View Slide

  13. Error Reporting
    Meaningful errors that we can act on.
    Tell us what failed, not just that a failure has
    occurred.
    Clear messages that we can understand.
    You have a parse error at line x of file y.
    Concise errors that don’t overwhelm.
    Parseable so we can automate responses.
    Monday, October 29, 12

    View Slide

  14. Configurability / Flexibility
    Make values configurable.
    Database endpoints.
    HTTP endpoints.
    Directories you write to/read from.
    Cache endpoints.
    Monday, October 29, 12

    View Slide

  15. Scalability
    In our world, we talk about Horizontal Scalability (scale
    out vs. up.)
    Basically means the ability to handle more traffic by
    adding additional servers.
    Monday, October 29, 12

    View Slide

  16. Disk Access
    Disk access is per server. Never assume you’ll talk to
    the same server twice in a row.
    NFS is slooow.
    Sticky sessions break more things than they fix.
    Monday, October 29, 12

    View Slide

  17. Database
    Master / Slave
    Master should be
    write-only.
    Dedicated slave for
    reporting / backups.
    Clustered
    Don’t expect instant
    consistency.
    Be aware of what
    you’ll lose.
    Monday, October 29, 12

    View Slide

  18. That’s all great but...
    Where are the tools?
    Monday, October 29, 12

    View Slide

  19. Disclaimer
    I’m a Linux administrator, a Linux/Mac
    user, and the majority of PHP Serving is
    done via Linux. The tools discussed are
    Linux tools. I’ve written everything for
    what’s included in Ubuntu (the common
    denominator).
    Sorry Windows guys.
    Monday, October 29, 12

    View Slide

  20. Making Requests
    Browser, like everyone else.
    curl from the command line.
    Monday, October 29, 12

    View Slide

  21. Curl - Test output
    Monday, October 29, 12

    View Slide

  22. Curl - Better than a Browser
    Monday, October 29, 12

    View Slide

  23. Checking CPU /Memory
    utilization
    xDebug + Cache Grind.
    Zend Debugger + Zend
    Studio.
    XHProf.
    uptime
    top
    htop
    Profiling Monitoring
    Monday, October 29, 12

    View Slide

  24. Load Averages - wtf?
    Commonly shown as 3 values, e.g. 1.06 1.14 1.36
    Numbers are load over 1 minute, 5 minutes, 15
    minutes
    Numbers represent load for a single core 1 = 100%
    Magic numbers (when you’re good):
    .7 per core is good utilization.
    1 per core is fully utilized (start planning new hw.)
    Monday, October 29, 12

    View Slide

  25. xDebug
    http://xdebug.org/docs/install
    Simple, efficient, open source.
    Integrated into PHPStorm, Netbeans, and most every
    other IDE (beside Zend Studio).
    Monday, October 29, 12

    View Slide

  26. xDebug + Cache Grind
    Monday, October 29, 12

    View Slide

  27. Zend Debugger
    Very well integrated to Zend Studio & Zend Server.
    Dead simple to use.
    Monday, October 29, 12

    View Slide

  28. Zend Debugger
    Monday, October 29, 12

    View Slide

  29. XHProf
    https://github.com/facebook/xhprof
    Runs great constantly, but does not integrate well with
    an IDE.
    Monday, October 29, 12

    View Slide

  30. Traditional (top)
    Monday, October 29, 12

    View Slide

  31. Newer (htop)
    Monday, October 29, 12

    View Slide

  32. Checking Disk IO
    inclued.
    dtrace.
    strace.
    Monday, October 29, 12

    View Slide

  33. inclued
    Get a map of your includes. This is more useful
    than it seems.
    Monday, October 29, 12

    View Slide

  34. dtrace
    From solaris, this will show you every system call,
    command is a bit complicated.
    Monday, October 29, 12

    View Slide

  35. strace
    Different output and interaction than dtrace,
    same purpose.
    Monday, October 29, 12

    View Slide

  36. Some other tools for
    Operations & Development.
    Monday, October 29, 12

    View Slide

  37. CPU / System Metrics
    Too many to list but Cacti & Munin are common
    and have low barrier to entry.
    Monday, October 29, 12

    View Slide

  38. Icinga / Nagios
    Alerts you to problems
    Monday, October 29, 12

    View Slide

  39. Wrapping Up
    Monday, October 29, 12

    View Slide

  40. Don’t
    Tell us it works on this machine or that machine.
    Forget to tell us your requirements while you’re making
    a deadline in the past.
    Claim you’ve found yet another [PHP|Apache|MySQL]
    bug when something doesn’t work.
    Do foolish things. e.g. Serve dynamically resized
    images (this is what gearman and caching is for.)
    Monday, October 29, 12

    View Slide

  41. Do
    Control your IO & Network access.
    Be aware of your environment.
    Use the tools you have available and learn new ones.
    Monday, October 29, 12

    View Slide