webis chronicle.pdf

Transcript

1. What we did What we learned What we planned 杨镭 <yanglei@snda.com>
2. ž  a deployment system: http://deploy.sdo.com ž  has a Web UI ž  already serves all SDO's core applications ž  forked by 盛付通, they are using it now! ž  till 2010-06-03: 49 projects, 229 modules 660 servers 6,749 deployments
3. 2011-05 Average Daily Deployents : 50 2010-04-20 Max Daily Deployments: 158
4. Problems: 1.  too many servers 2.  mixed environment: Windows/Linux; C+ +, ASP.NET, PHP etc.
5. A parallel SSH execution system on steroid ž  cygwin On Windows ž  Prevent reinvent wheel( BatchManager) ž  parallel execution features: max_hosts, nopty “SSH is the lowest common denominator, just wrap it, build abstraction upon it”
6. Simple DSL similar to ruby's rake: DSL scripts are first interpreted by ruby then send to remote machines for execution “raw shell commands are NOT enough, sometimes you want to generate them on the fly”
7. Deploy related concepts such as roles, tasks, stages, recipes : ž  roles: web servers, app servers, databases ž  tasks: execution unit, descriptive name for messy shell scripts ž  stages: testing, staging, production ž  recipes: collection of tasks related to some topic
8. SSH tunneling connections via gateway set :gateway, 'ip:port' ž exempt needs to setup VPN, open holes in firewall “SSH offers lots of possibilities”
9. Problems: 3.  file consistency across servers(usually for web servers) 4.  prepare deployment package is time- consuming 5.  revert is extremely painful
10. git is a DVCS for developer but a content management tool for op 1.  It does not track filenames, it tracks content using sha1 hash 2.  Given a commit id, one can ensure not a single bit is different between repositories
11. view each application's deployment directory as a git repository, then: 1.  updating is the same to pull new files from git server 2.  reverting is the same to undo last commit
12. Deployment Flow: 1.  push new version to git repositories 2.  production servers using rsync to synchronize files from a central git server “Rsync is more light-weight than git, so use it to synchronize files”(bad decision)
13. 2010-05 Webistrano(abbreviated as webis), an open source project 2010-06 刘江南<liujiangnan@snda.com> start maintaining Webistrano , Fix lots of bugs and several improvements 2010-07 Using Gitosis to manage git repositories (security) 2010-08 Standardized revert strategy 2010-09 prefer git over rsync to do all file related tasks 2010-11 Separate privilege for each project
14. GET THINGS DONE, NO MORE MANY IMPORTANT FEATURES
15. ž keep track of who deployed what when to which servers
16. ž Centralized management of deployment scripts
17. ž Stage specific configuration, Alert on deploy
18. Hosts list (not in original webistrano)
19. Things Learned: “a web UI is a MUST for enterprise environment , get it as early as possible”
20. Problem: At first, dirty tricks were employed to ensure git repo’s security. Then, complexity appears, administration cost is HIGH. Solution: Gitosis: The Easy (and Secure) Way http://scie.nti.st/2007/11/14/hosting-git-repositories-the-easy-and-secure-way
21. Git is great but it exposes to many ways to revert commit: 1.  git checkout HEAD~1 && git checkout --b new_master 2.  git revert && git push 3.  git reset --merge commit_id (winner) “There should be one—and preferably only one— obvious way to do it.”
22. 1.  Deploy git is easy: it turns out deploying git on Linux and windows are easy. e.g., cygwin includes the latest version of git by default 2.  Git has better performance: since git only compares sha1 so it can stop traversing an entire directory when sha1 matches. Rsync eats up too much CPU 3.  Unify protocol(only SSH now): rsync requires its own protocol which complicates firewall/ iptables setup
23. THE argument that kicks rsync out: Git stores a completely history in each copy so there is no need to communicate with central server before reverting Rsync works on vanilla file system. Revert central repo first, then rsync with it. But revert is usually in urgent situation, Hurry!! “To make every decision right is hard, but we should evaluate tools to evolve”
24. 2010-11 specify a general scheme to name tag 2010-12 merge deploy/revert tasks into one(templating recipe) 2011-01 need a general configuration layer : gifonc 2011-02 Stages become modules, project summary page 2011-04 import existing projects into webis are painful, explicitly write down specifications(with dev) 2011-04 alert on deploy improved, better comment, deploy on which servers 2011-05 mine deploy stats to evaluate the quality of new version
25. ž Every deployment shall have a tag ž Specified tag format: pt_ptdy_v1.0.1.16933_R_08 ž Gains more understanding of git, introduce a template tasks
26. tag 1 tag 2 tag 3 1 2 3
27. 1 2 revert to previous version 3 1 normal deploy tag sha1hash of application’s deployment directory 1.  The content of tag is the sha1 hash of applications deployment directory. 2.  Move HEAD pointer can emulate any operation.
28. ž Encapsulate commands like stop/start service, reload configuration to _before/ _after stage specific variables ž All projects shall follow the pattern of template recipe. “There should be one—and preferably only one—obvious way to do it”
30. ž  Originally, stages indicates testing, staging, production environment ž  But we failed to integrate all environments into webis. That is: we can’t reach agreement with QA and Dev. (Political Issue) ž  Currently, stages are modules in disguise
31. 1.  Project is a container for modules, it’s like namespace 2.  Module is the deployable target, usually one project has several modules. “Module should be a central concept in deployment system”
32. Migrate existing applications into webis is painful: 1.  Some Windows apps filename with GB2312 characters, git is UTF-8 only 2.  Application create temp files at runtime which are untracked by git and might cause git merge fail 3.  Log are created directly under application's deployment directory
33. Solution: 1.  Applied several git features: .gitignore, git reset -- hard, git clean -d -f 2.  Explicitly ask developers to change whatever interfere with migrating into webis Lessons: “Write down deployment specifications early . Deployment system should be opinionated. Have a culture where dev/op agree on the importance of these seemingly unimportant details”
34. UGLY COMMIT ID(SHA1 HASH)
35. 1.  Some module has complex deployment requirements. 2.  Some release has lower quality which cause repeat deployments. Deployment system should detect those releases automatically then we humans can act upon it.
36. ž  Change tag format to contain a sequence number which increases by one for every deployment associated with a specific task. ž  Webis counts sequence number and send daily report “Make your data meaningful and use them to improve system”
37. 1.  Connect webis with VCS, e.g., SVN, GIT, etc,. 2.  Connect webis with Profiling System e.g., mSLA 3.  Connect webis with CI, e.g., jenkins 4.  Configuration/Code Review BEFORE/AFTER deployment (why we want to connect webis with other system) 5.  Vote for deployment. There must be at least three dev
38. DERIVE MORE MEANINGFUL DATA FROM SOURCECODE OTHERWISE, YOU HAVE ONLY BINARIES