Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ops Tool made by Perl Beginner #yapcasia 2012

riywo
September 28, 2012

Ops Tool made by Perl Beginner #yapcasia 2012

You can talk using this slide without any permission:)

riywo

September 28, 2012
Tweet

More Decks by riywo

Other Decks in Programming

Transcript

  1. Ops Tool
    made by
    Perl Beginner
    @riywo
    Sep, 28, 2012
    YAPC::Asia 2012

    View Slide

  2. @riywo

    View Slide

  3. http://www.publicdomainpictures.net/view-image.php?image=4314&picture=t-shirt-image

    View Slide

  4. View Slide

  5. Perl
    Beginner

    View Slide

  6. I Love
    Cats

    View Slide

  7. DataBase
    Engineer
    MySQL

    View Slide

  8. Operation
    Engineer
    DevOps

    View Slide

  9. Operation
    Engineer?

    View Slide

  10. Trouble Shooting
    Design System
    Setup Server
    On Calling
    Isu nage
    Sync Info
    Operation
    Inspection
    Develop Tools
    Capacity Plan

    View Slide

  11. Trouble Shooting
    Design System
    Setup Server
    On Calling
    Isu nage?
    Sync Info
    Operation
    Inspection
    Develop Tools
    Capacity Plan

    View Slide

  12. =Throwing
    Chair :-p

    View Slide

  13. F*cking Query
    SELECT SQL_CALC_FOUD_ROWS *
    FROM table
    WHERE c1 > 1000 OR c2 != 100
    ORDER BY c3
    LIMIT 10 OFFSET 100000;

    View Slide

  14. http://www.flickr.com/photos/andrew_freese/2200774154/
    Operation
    Engineer

    View Slide

  15. dev> I did nothing..., but....

    View Slide

  16. ops> What did you do?
    dev> I did nothing..., but....

    View Slide

  17. http://www.flickr.com/photos/jaydoubleyougee/2259260845/
    Operation
    Engineer

    View Slide

  18. I Love
    Cats

    View Slide

  19. But...., do they only happen
    in Dev side?

    View Slide

  20. Operation
    Error!!!!!!!!!!!!!

    View Slide

  21. Operation Mistake
    # rm -fr /tmp/*
    # rm -fr / tmp/*

    View Slide

  22. Capacity Over
    Sorry,
    we have no more servers....

    View Slide

  23. ...and... Setup Error

    View Slide

  24. server_id = 1212
    default_character_set=sjis
    innodb_buffer_pool_size=2G
    server_id = 1213
    default_character_set=utf8
    innodb_buffer_pool_size=3G

    View Slide

  25. Can’t copy the same file to
    all servers
    => edit manually

    View Slide

  26. Can’t copy the same file to
    all servers
    => edit manually
    M
    istake!!!

    View Slide

  27. Current servers’
    sysctl changed
    But setup operation
    did not follow it...

    View Slide

  28. Current servers’
    sysctl changed
    But setup operation
    did not follow it...
    M
    istake!!!

    View Slide

  29. http://www.flickr.com/photos/clickflashphotos/3402287993/

    View Slide

  30. Why?
    Are you a Professional?

    View Slide

  31. Someone has
    “Setup Procedure Manual”
    1. OS install
    1.1 IP address
    1.2 nameserver
    1.3 OS user
    2. SSH Configuration
    2.1 ...
    3. Install Perl
    3.1 ...

    View Slide

  32. Someone have
    “Setup Procedure Manual”
    1. OS install
    1.1 IP address
    1.2 nameserver
    1.3 OS user
    2. SSH Configuration
    2.1 ...
    3. Install Perl
    3.1 ...
    Operate
    Manually
    Check
    Visually

    View Slide

  33. # ssh $TGT cat /etc/my.cnf | diff -u /etc/my.cnf -
    - server_id = 2323
    + server_id = 2324
    - log_slave_updaets
    Yes, this difference is OK
    If $TGT is Master DB
    this difference is NG

    View Slide

  34. # ssh $TGT cat /etc/my.cnf | diff -u /etc/my.cnf -
    - server_id = 2323
    + server_id = 2324
    - log_slave_updaets
    Yes, this difference is OK
    If $TGT is Master DB
    this difference is NG
    Impossible

    View Slide

  35. OK,
    Setup scripts help us!

    View Slide

  36. OK,
    Setup scripts help us!

    View Slide

  37. Patch, Patch, Patch...
    patch for a new component many times,
    setup script become a spaghetti code...

    View Slide

  38. Script Bug
    OpsEngi usually not good at programing,
    the script will have many many bugs...

    View Slide

  39. Unfollow current change
    For trouble, servers’ config were changed,
    but setup script isn’t changed...

    View Slide

  40. No Test!!!! F*ck!!!
    usually setup script is simple shell script,
    only listed exec commands
    no test

    View Slide

  41. No Test!!!! F*ck!!!
    usually setup script is simple shell script,
    only listed exec commands
    no test

    View Slide

  42. Check return code
    Check status of servers

    View Slide

  43. Describe

    View Slide

  44. Server Correct Status
    /etc/sysconfig/ /etc/my.cnf /etc/sysctrl.conf
    /etc/ssh/ssh_config / /etc/cron.d
    /etc/resolv.conf /etc/yum.repos.d/
    /service/app/run chkconfig ulimit httpd.conf
    rpms

    View Slide

  45. If you describe the correct status
    you can also test current servers!

    View Slide

  46. Correct
    Status
    New Server
    Server
    Server
    Server
    Server
    Server

    View Slide

  47. Correct
    Status
    New Server
    Server
    Server
    Server
    Server
    Server
    Test
    Setup

    View Slide

  48. Break Time :)
    Remain 10min?

    View Slide

  49. Correct
    Status
    New Server
    Server
    Server
    Server
    Server
    Server
    Test
    Setup

    View Slide

  50. http://www.flickr.com/photos/stocktoc/4539506889/
    Touryo
    ౩ྊNBTUFS
    DBSQFOUFSPGTFSWFST

    View Slide

  51. Describe

    View Slide

  52. Orchestration
    Configuration
    Bootstrapping
    Server Setup

    View Slide

  53. Orchestration
    Configuration
    Bootstrapping
    Server Setup
    Data sync, LB configuration
    OS install

    View Slide

  54. Orchestration
    Configuration
    Bootstrapping
    Server Setup
    Configure, Install Middleware

    View Slide

  55. Describe test/setup procedures
    DSL = blueprint.pl
    (cron,httpd,mysql,...)
    test {
    my $c = shift;
    ok ...
    };
    run {
    my $c = shift;
    $c->ssh(...)
    };

    View Slide

  56. server_id =
    default_character_set=
    Server’s my.cnf
    Text::Xslate::render
    Test example
    diff

    View Slide

  57. server_id =
    default_character_set=
    Server’s my.cnf
    Text::Xslate::render
    Run example
    SSH install

    View Slide

  58. server1 => mbga:web
    server2 => mbga:db:master
    server3 => game1:web
    Describe role of nodes

    View Slide

  59. Describe blueprints of roles
    mbga:web => [cron, httpd]
    mbga:db:master => [cron, mysql]
    game1:web => [cron, httpd]

    View Slide

  60. Describe attribute of roles
    mbga:web => {
    cron_hour => 1,
    proc_num => 50,
    }
    mbga:db:master => {...}
    game1:web => {...}

    View Slide

  61. How to
    develop?

    View Slide

  62. Make time

    View Slide

  63. Research

    View Slide

  64. Small snippet

    View Slide

  65. Ask Devs!

    View Slide

  66. Lessons for
    Manage Tool
    Development

    View Slide

  67. Maintenance
    Can you hand over its maintenance?

    View Slide

  68. Touryo blueprint are written
    by other staffs :)

    View Slide

  69. Simple
    The UNIX philosophy

    View Slide

  70. Touryo only care about
    server configuration.
    Combine other tools
    ex. AdminTool(node-role)

    View Slide

  71. Replace
    Is it able to replace easily?

    View Slide

  72. By Touryo,
    undocumented procedures
    have been described.
    Easier to replace!

    View Slide

  73. Any
    Questions?

    View Slide

  74. Thanks!

    View Slide

  75. @riywo
    blog.riywo.com
    weblog.riywo.com

    View Slide