Share
Embed
Transcript
38 slides
-
a deployment system: http://deploy.sdo.com has a Web UI
already serves all SDO's core applications forked by 盛付通, they are using it now! till 2010-06-03: 49 projects, 229 modules 660 servers 6,749 deployments -
Problems: 1. too many servers 2. mixed environment: Windows/Linux; C+
+, ASP.NET, PHP etc. -
A parallel SSH execution system on steroid cygwin On
Windows Prevent reinvent wheel( BatchManager) parallel execution features: max_hosts, nopty “SSH is the lowest common denominator, just wrap it, build abstraction upon it” -
Simple DSL similar to ruby's rake: DSL scripts are first
interpreted by ruby then send to remote machines for execution “raw shell commands are NOT enough, sometimes you want to generate them on the fly” -
Deploy related concepts such as roles, tasks, stages, recipes :
roles: web servers, app servers, databases tasks: execution unit, descriptive name for messy shell scripts stages: testing, staging, production recipes: collection of tasks related to some topic -
SSH tunneling connections via gateway set :gateway, 'ip:port' exempt needs
to setup VPN, open holes in firewall “SSH offers lots of possibilities” -
Problems: 3. file consistency across servers(usually for web servers) 4.
prepare deployment package is time- consuming 5. revert is extremely painful -
git is a DVCS for developer but a content management
tool for op 1. It does not track filenames, it tracks content using sha1 hash 2. Given a commit id, one can ensure not a single bit is different between repositories -
view each application's deployment directory as a git repository, then:
1. updating is the same to pull new files from git server 2. reverting is the same to undo last commit -
Deployment Flow: 1. push new version to git repositories 2.
production servers using rsync to synchronize files from a central git server “Rsync is more light-weight than git, so use it to synchronize files”(bad decision) -
2010-05 Webistrano(abbreviated as webis), an open source project 2010-06 刘江南<[email protected]>
start maintaining Webistrano , Fix lots of bugs and several improvements 2010-07 Using Gitosis to manage git repositories (security) 2010-08 Standardized revert strategy 2010-09 prefer git over rsync to do all file related tasks 2010-11 Separate privilege for each project -
Things Learned: “a web UI is a MUST for enterprise
environment , get it as early as possible” -
Problem: At first, dirty tricks were employed to ensure git
repo’s security. Then, complexity appears, administration cost is HIGH. Solution: Gitosis: The Easy (and Secure) Way http://scie.nti.st/2007/11/14/hosting-git-repositories-the-easy-and-secure-way -
Git is great but it exposes to many ways to
revert commit: 1. git checkout HEAD~1 && git checkout --b new_master 2. git revert && git push 3. git reset --merge commit_id (winner) “There should be one—and preferably only one— obvious way to do it.” -
1. Deploy git is easy: it turns out deploying git
on Linux and windows are easy. e.g., cygwin includes the latest version of git by default 2. Git has better performance: since git only compares sha1 so it can stop traversing an entire directory when sha1 matches. Rsync eats up too much CPU 3. Unify protocol(only SSH now): rsync requires its own protocol which complicates firewall/ iptables setup -
THE argument that kicks rsync out: Git stores a completely
history in each copy so there is no need to communicate with central server before reverting Rsync works on vanilla file system. Revert central repo first, then rsync with it. But revert is usually in urgent situation, Hurry!! “To make every decision right is hard, but we should evaluate tools to evolve” -
2010-11 specify a general scheme to name tag 2010-12 merge
deploy/revert tasks into one(templating recipe) 2011-01 need a general configuration layer : gifonc 2011-02 Stages become modules, project summary page 2011-04 import existing projects into webis are painful, explicitly write down specifications(with dev) 2011-04 alert on deploy improved, better comment, deploy on which servers 2011-05 mine deploy stats to evaluate the quality of new version -
Every deployment shall have a tag Specified tag format: pt_ptdy_v1.0.1.16933_R_08
Gains more understanding of git, introduce a template tasks -
1 2 revert to previous version 3 1 normal deploy
tag sha1hash of application’s deployment directory 1. The content of tag is the sha1 hash of applications deployment directory. 2. Move HEAD pointer can emulate any operation. -
Encapsulate commands like stop/start service, reload configuration to _before/ _after
stage specific variables All projects shall follow the pattern of template recipe. “There should be one—and preferably only one—obvious way to do it” -
Originally, stages indicates testing, staging, production environment But
we failed to integrate all environments into webis. That is: we can’t reach agreement with QA and Dev. (Political Issue) Currently, stages are modules in disguise -
1. Project is a container for modules, it’s like namespace
2. Module is the deployable target, usually one project has several modules. “Module should be a central concept in deployment system” -
Migrate existing applications into webis is painful: 1. Some Windows
apps filename with GB2312 characters, git is UTF-8 only 2. Application create temp files at runtime which are untracked by git and might cause git merge fail 3. Log are created directly under application's deployment directory -
Solution: 1. Applied several git features: .gitignore, git reset --
hard, git clean -d -f 2. Explicitly ask developers to change whatever interfere with migrating into webis Lessons: “Write down deployment specifications early . Deployment system should be opinionated. Have a culture where dev/op agree on the importance of these seemingly unimportant details” -
1. Some module has complex deployment requirements. 2. Some release
has lower quality which cause repeat deployments. Deployment system should detect those releases automatically then we humans can act upon it. -
Change tag format to contain a sequence number which
increases by one for every deployment associated with a specific task. Webis counts sequence number and send daily report “Make your data meaningful and use them to improve system” -
1. Connect webis with VCS, e.g., SVN, GIT, etc,. 2.
Connect webis with Profiling System e.g., mSLA 3. Connect webis with CI, e.g., jenkins 4. Configuration/Code Review BEFORE/AFTER deployment (why we want to connect webis with other system) 5. Vote for deployment. There must be at least three dev