Ops Tool made by Perl Beginner #yapcasia 2012

415d71aaeaa22d9ecfef343bda1ebde1?s=47 riywo
September 28, 2012

Ops Tool made by Perl Beginner #yapcasia 2012

You can talk using this slide without any permission:)

415d71aaeaa22d9ecfef343bda1ebde1?s=128

riywo

September 28, 2012
Tweet

Transcript

  1. Ops Tool made by Perl Beginner @riywo Sep, 28, 2012

    YAPC::Asia 2012
  2. @riywo

  3. http://www.publicdomainpictures.net/view-image.php?image=4314&picture=t-shirt-image

  4. None
  5. Perl Beginner

  6. I Love Cats

  7. DataBase Engineer MySQL

  8. Operation Engineer DevOps

  9. Operation Engineer?

  10. Trouble Shooting Design System Setup Server On Calling Isu nage

    Sync Info Operation Inspection Develop Tools Capacity Plan
  11. Trouble Shooting Design System Setup Server On Calling Isu nage?

    Sync Info Operation Inspection Develop Tools Capacity Plan
  12. =Throwing Chair :-p

  13. F*cking Query SELECT SQL_CALC_FOUD_ROWS * FROM table WHERE c1 >

    1000 OR c2 != 100 ORDER BY c3 LIMIT 10 OFFSET 100000;
  14. http://www.flickr.com/photos/andrew_freese/2200774154/ Operation Engineer

  15. dev> I did nothing..., but....

  16. ops> What did you do? dev> I did nothing..., but....

  17. http://www.flickr.com/photos/jaydoubleyougee/2259260845/ Operation Engineer

  18. I Love Cats

  19. But...., do they only happen in Dev side?

  20. Operation Error!!!!!!!!!!!!!

  21. Operation Mistake # rm -fr /tmp/* # rm -fr /

    tmp/*
  22. Capacity Over Sorry, we have no more servers....

  23. ...and... Setup Error

  24. server_id = 1212 default_character_set=sjis innodb_buffer_pool_size=2G server_id = 1213 default_character_set=utf8 innodb_buffer_pool_size=3G

  25. Can’t copy the same file to all servers => edit

    manually
  26. Can’t copy the same file to all servers => edit

    manually M istake!!!
  27. Current servers’ sysctl changed But setup operation did not follow

    it...
  28. Current servers’ sysctl changed But setup operation did not follow

    it... M istake!!!
  29. http://www.flickr.com/photos/clickflashphotos/3402287993/

  30. Why? Are you a Professional?

  31. Someone has “Setup Procedure Manual” 1. OS install 1.1 IP

    address 1.2 nameserver 1.3 OS user 2. SSH Configuration 2.1 ... 3. Install Perl 3.1 ...
  32. Someone have “Setup Procedure Manual” 1. OS install 1.1 IP

    address 1.2 nameserver 1.3 OS user 2. SSH Configuration 2.1 ... 3. Install Perl 3.1 ... Operate Manually Check Visually
  33. # ssh $TGT cat /etc/my.cnf | diff -u /etc/my.cnf -

    - server_id = 2323 + server_id = 2324 - log_slave_updaets Yes, this difference is OK If $TGT is Master DB this difference is NG
  34. # ssh $TGT cat /etc/my.cnf | diff -u /etc/my.cnf -

    - server_id = 2323 + server_id = 2324 - log_slave_updaets Yes, this difference is OK If $TGT is Master DB this difference is NG Impossible
  35. OK, Setup scripts help us!

  36. OK, Setup scripts help us! ✗

  37. Patch, Patch, Patch... patch for a new component many times,

    setup script become a spaghetti code...
  38. Script Bug OpsEngi usually not good at programing, the script

    will have many many bugs...
  39. Unfollow current change For trouble, servers’ config were changed, but

    setup script isn’t changed...
  40. No Test!!!! F*ck!!! usually setup script is simple shell script,

    only listed exec commands no test
  41. No Test!!!! F*ck!!! usually setup script is simple shell script,

    only listed exec commands no test
  42. Check return code Check status of servers ✗

  43. Describe

  44. Server Correct Status /etc/sysconfig/ /etc/my.cnf /etc/sysctrl.conf /etc/ssh/ssh_config / /etc/cron.d /etc/resolv.conf

    /etc/yum.repos.d/ /service/app/run chkconfig ulimit httpd.conf rpms
  45. If you describe the correct status you can also test

    current servers!
  46. Correct Status New Server Server Server Server Server Server

  47. Correct Status New Server Server Server Server Server Server Test

    Setup
  48. Break Time :) Remain 10min?

  49. Correct Status New Server Server Server Server Server Server Test

    Setup
  50. http://www.flickr.com/photos/stocktoc/4539506889/ Touryo ౩ྊNBTUFS DBSQFOUFSPGTFSWFST

  51. Describe

  52. Orchestration Configuration Bootstrapping Server Setup

  53. Orchestration Configuration Bootstrapping Server Setup Data sync, LB configuration OS

    install
  54. Orchestration Configuration Bootstrapping Server Setup Configure, Install Middleware

  55. Describe test/setup procedures DSL = blueprint.pl (cron,httpd,mysql,...) test { my

    $c = shift; ok ... }; run { my $c = shift; $c->ssh(...) };
  56. server_id = <: $server_id :> default_character_set= <: $char :> Server’s

    my.cnf Text::Xslate::render Test example diff
  57. server_id = <: $server_id :> default_character_set= <: $char :> Server’s

    my.cnf Text::Xslate::render Run example SSH install
  58. server1 => mbga:web server2 => mbga:db:master server3 => game1:web Describe

    role of nodes
  59. Describe blueprints of roles mbga:web => [cron, httpd] mbga:db:master =>

    [cron, mysql] game1:web => [cron, httpd]
  60. Describe attribute of roles mbga:web => { cron_hour => 1,

    proc_num => 50, } mbga:db:master => {...} game1:web => {...}
  61. How to develop?

  62. Make time

  63. Research

  64. Small snippet

  65. Ask Devs!

  66. Lessons for Manage Tool Development

  67. Maintenance Can you hand over its maintenance?

  68. Touryo blueprint are written by other staffs :)

  69. Simple The UNIX philosophy

  70. Touryo only care about server configuration. Combine other tools ex.

    AdminTool(node-role)
  71. Replace Is it able to replace easily?

  72. By Touryo, undocumented procedures have been described. Easier to replace!

  73. Any Questions?

  74. Thanks!

  75. @riywo blog.riywo.com weblog.riywo.com