Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ops Tool made by Perl Beginner #yapcasia 2012

riywo
September 28, 2012

Ops Tool made by Perl Beginner #yapcasia 2012

You can talk using this slide without any permission:)

riywo

September 28, 2012
Tweet

More Decks by riywo

Other Decks in Programming

Transcript

  1. Ops Tool
    made by
    Perl Beginner
    @riywo
    Sep, 28, 2012
    YAPC::Asia 2012

    View full-size slide

  2. http://www.publicdomainpictures.net/view-image.php?image=4314&picture=t-shirt-image

    View full-size slide

  3. Perl
    Beginner

    View full-size slide

  4. DataBase
    Engineer
    MySQL

    View full-size slide

  5. Operation
    Engineer
    DevOps

    View full-size slide

  6. Operation
    Engineer?

    View full-size slide

  7. Trouble Shooting
    Design System
    Setup Server
    On Calling
    Isu nage
    Sync Info
    Operation
    Inspection
    Develop Tools
    Capacity Plan

    View full-size slide

  8. Trouble Shooting
    Design System
    Setup Server
    On Calling
    Isu nage?
    Sync Info
    Operation
    Inspection
    Develop Tools
    Capacity Plan

    View full-size slide

  9. =Throwing
    Chair :-p

    View full-size slide

  10. F*cking Query
    SELECT SQL_CALC_FOUD_ROWS *
    FROM table
    WHERE c1 > 1000 OR c2 != 100
    ORDER BY c3
    LIMIT 10 OFFSET 100000;

    View full-size slide

  11. http://www.flickr.com/photos/andrew_freese/2200774154/
    Operation
    Engineer

    View full-size slide

  12. dev> I did nothing..., but....

    View full-size slide

  13. ops> What did you do?
    dev> I did nothing..., but....

    View full-size slide

  14. http://www.flickr.com/photos/jaydoubleyougee/2259260845/
    Operation
    Engineer

    View full-size slide

  15. But...., do they only happen
    in Dev side?

    View full-size slide

  16. Operation
    Error!!!!!!!!!!!!!

    View full-size slide

  17. Operation Mistake
    # rm -fr /tmp/*
    # rm -fr / tmp/*

    View full-size slide

  18. Capacity Over
    Sorry,
    we have no more servers....

    View full-size slide

  19. ...and... Setup Error

    View full-size slide

  20. server_id = 1212
    default_character_set=sjis
    innodb_buffer_pool_size=2G
    server_id = 1213
    default_character_set=utf8
    innodb_buffer_pool_size=3G

    View full-size slide

  21. Can’t copy the same file to
    all servers
    => edit manually

    View full-size slide

  22. Can’t copy the same file to
    all servers
    => edit manually
    M
    istake!!!

    View full-size slide

  23. Current servers’
    sysctl changed
    But setup operation
    did not follow it...

    View full-size slide

  24. Current servers’
    sysctl changed
    But setup operation
    did not follow it...
    M
    istake!!!

    View full-size slide

  25. http://www.flickr.com/photos/clickflashphotos/3402287993/

    View full-size slide

  26. Why?
    Are you a Professional?

    View full-size slide

  27. Someone has
    “Setup Procedure Manual”
    1. OS install
    1.1 IP address
    1.2 nameserver
    1.3 OS user
    2. SSH Configuration
    2.1 ...
    3. Install Perl
    3.1 ...

    View full-size slide

  28. Someone have
    “Setup Procedure Manual”
    1. OS install
    1.1 IP address
    1.2 nameserver
    1.3 OS user
    2. SSH Configuration
    2.1 ...
    3. Install Perl
    3.1 ...
    Operate
    Manually
    Check
    Visually

    View full-size slide

  29. # ssh $TGT cat /etc/my.cnf | diff -u /etc/my.cnf -
    - server_id = 2323
    + server_id = 2324
    - log_slave_updaets
    Yes, this difference is OK
    If $TGT is Master DB
    this difference is NG

    View full-size slide

  30. # ssh $TGT cat /etc/my.cnf | diff -u /etc/my.cnf -
    - server_id = 2323
    + server_id = 2324
    - log_slave_updaets
    Yes, this difference is OK
    If $TGT is Master DB
    this difference is NG
    Impossible

    View full-size slide

  31. OK,
    Setup scripts help us!

    View full-size slide

  32. OK,
    Setup scripts help us!

    View full-size slide

  33. Patch, Patch, Patch...
    patch for a new component many times,
    setup script become a spaghetti code...

    View full-size slide

  34. Script Bug
    OpsEngi usually not good at programing,
    the script will have many many bugs...

    View full-size slide

  35. Unfollow current change
    For trouble, servers’ config were changed,
    but setup script isn’t changed...

    View full-size slide

  36. No Test!!!! F*ck!!!
    usually setup script is simple shell script,
    only listed exec commands
    no test

    View full-size slide

  37. No Test!!!! F*ck!!!
    usually setup script is simple shell script,
    only listed exec commands
    no test

    View full-size slide

  38. Check return code
    Check status of servers

    View full-size slide

  39. Server Correct Status
    /etc/sysconfig/ /etc/my.cnf /etc/sysctrl.conf
    /etc/ssh/ssh_config / /etc/cron.d
    /etc/resolv.conf /etc/yum.repos.d/
    /service/app/run chkconfig ulimit httpd.conf
    rpms

    View full-size slide

  40. If you describe the correct status
    you can also test current servers!

    View full-size slide

  41. Correct
    Status
    New Server
    Server
    Server
    Server
    Server
    Server

    View full-size slide

  42. Correct
    Status
    New Server
    Server
    Server
    Server
    Server
    Server
    Test
    Setup

    View full-size slide

  43. Break Time :)
    Remain 10min?

    View full-size slide

  44. Correct
    Status
    New Server
    Server
    Server
    Server
    Server
    Server
    Test
    Setup

    View full-size slide

  45. http://www.flickr.com/photos/stocktoc/4539506889/
    Touryo
    ౩ྊNBTUFS
    DBSQFOUFSPGTFSWFST

    View full-size slide

  46. Orchestration
    Configuration
    Bootstrapping
    Server Setup

    View full-size slide

  47. Orchestration
    Configuration
    Bootstrapping
    Server Setup
    Data sync, LB configuration
    OS install

    View full-size slide

  48. Orchestration
    Configuration
    Bootstrapping
    Server Setup
    Configure, Install Middleware

    View full-size slide

  49. Describe test/setup procedures
    DSL = blueprint.pl
    (cron,httpd,mysql,...)
    test {
    my $c = shift;
    ok ...
    };
    run {
    my $c = shift;
    $c->ssh(...)
    };

    View full-size slide

  50. server_id = <: $server_id :>
    default_character_set= <: $char :>
    Server’s my.cnf
    Text::Xslate::render
    Test example
    diff

    View full-size slide

  51. server_id = <: $server_id :>
    default_character_set= <: $char :>
    Server’s my.cnf
    Text::Xslate::render
    Run example
    SSH install

    View full-size slide

  52. server1 => mbga:web
    server2 => mbga:db:master
    server3 => game1:web
    Describe role of nodes

    View full-size slide

  53. Describe blueprints of roles
    mbga:web => [cron, httpd]
    mbga:db:master => [cron, mysql]
    game1:web => [cron, httpd]

    View full-size slide

  54. Describe attribute of roles
    mbga:web => {
    cron_hour => 1,
    proc_num => 50,
    }
    mbga:db:master => {...}
    game1:web => {...}

    View full-size slide

  55. How to
    develop?

    View full-size slide

  56. Small snippet

    View full-size slide

  57. Lessons for
    Manage Tool
    Development

    View full-size slide

  58. Maintenance
    Can you hand over its maintenance?

    View full-size slide

  59. Touryo blueprint are written
    by other staffs :)

    View full-size slide

  60. Simple
    The UNIX philosophy

    View full-size slide

  61. Touryo only care about
    server configuration.
    Combine other tools
    ex. AdminTool(node-role)

    View full-size slide

  62. Replace
    Is it able to replace easily?

    View full-size slide

  63. By Touryo,
    undocumented procedures
    have been described.
    Easier to replace!

    View full-size slide

  64. Any
    Questions?

    View full-size slide

  65. @riywo
    blog.riywo.com
    weblog.riywo.com

    View full-size slide