Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Administer the devops pill across the enterprise

Administer the devops pill across the enterprise

Presented at devopsdays Berlin May 2013
devops addresses the last mile of continuous delivery. It advocates the tearing down of development and operations silos. It kind of assumes that silos don't exist elsewhere. This is often not true in enterprise IT where the number of silos correlates well with the number of vice presidents.

This anti-pattern of org design takes two forms. One is when you have a VP sw-dev, VP Data, VP configuration management, VP Testing, VP UX, VP Deployment. The other is when you have a product manager along with a VP Sales, VP Marketing, VP Support and VP Training. On the other hand, an org design based on true cross-functional teams has teams accountable for business outcomes and puts one person in clear charge of each team. This part of getting things right can be classifed as CD for execs and basically involves applying the devops pattern to general org design. There is also a part two.

Tooling can also hinder cross functional behaviour. Especially when access to tools are granted on a strict need-to-use basis. So for example, only sales guys get salesforce access. Or when marketing disallows self-service publishing from the product teams. Or when the devs use one tool for continuous integration and the deployment teams uses a completely different disconnected tool for release automation. Sometimes commercial considerations (licensing costs) encourage this behaviour but it doesn't serve the cause of tearing down silos. So we need a new kind of culture that restricts access to tools only where unavoidable but is open otherwise.


Sriram Narayan

May 27, 2013


  1. Hello. I am going to talk about organizing for continuous

    delivery in places with largish IT teams – say 50 or more - and about how our tooling choices and access decisions can have an impact. The ideas and opinions expressed here aren’t necessarily endorsed by my employer. They aren’t groundbreakingly original or based on hard data or research either. But then advances in the methods of software development have never waited for the results of controlled experiments with objectively quantifiable measures. We find something wrong with the status quo, we make a change and since we are part of the experiment, we can sense if the change is having the desired effect. 1
  2. Ok. So getting on with the presentation. Devops addresses the

    last mile of CD. But because it gets all the press these days, the journey before the last mile has faded away from view. For example, many organizations have started to adopt tools like Chef and Puppet but they don’t consider long lived feature branches to be a problem. Continuous s/w delivery needs more than devops. And a successful product needs more than CD. So if we care about shortening end-to-end cycle times, we need to look the at the end-to-end flow of work from product management and marketing to development to operations and sales. Now devops brings with it certain recommendations for organization design – for example cross-functional teams. But is it enough if a team is cross functional just with respect to development and operations? 2
  3. Many big orgs are still organized along functional verticals. Each

    vertical is typically headed by a VP who then reports to a CXO. A VP’s org consists of functional specialists that are made available to different projects. Product owners have a tough time getting their work done through the different layers from left to right. Occasional power struggles break out between product owners and VPs. In this illustration, each vertical has some people or some capacity dedicated to a given product. It is bad in the sense that there are Too may handoffs the tendency for batch size to go up Does not encourage continuous collaboration (so meetings) But it gets worse if in the name of efficiency and greater utilization, we get rid of the dedicated capacity and make every vertical a global pool of fungible resources to be allocated just in time. This is a good example of how not to design for efficiency in a knowledge process. If outsourcing, don’t outsource different verticals to different vendors. To get better cycle time, responsiveness matters more than utilization In my experience, a matrix org can at best give you a release every 6 months. If you have a matrix org in which the product owners are happy with the cycle time, then 3
  4. perhaps there is no need for change. I haven’t come

    across one yet. Continuous deployment talks about multiple releases in a day. My guess is – to get anywhere near one release a month – we’ll have to refactor the matrix. 3
  5. OTOH, we can have self-sufficient cross-functional product teams. The product

    team is fully accountable for the success of the product. It is almost like a different business unit except that they still depend on external shared verticals such as finance, admin, legal and HR. Each product team has one person in charge. This is generally the product owner or product manager. I’m not arguing against specialization. These teams still consist of specialists. I’m just arguing against organizing along the lines of specialization. Will specialists dedicated to product teams be underutilized? Probably. But as a side-effect of being part of a cross-functional team, they usually start to acquire adjacent skills. So developers pick up infrastructure skills while product analysts pick up testing skills. Pure specialists start morphing into generalizing specialists. Their skill profile changes shape from the letter I to the letter T. These cross-functional teams should be co-located, sitting side-by-side as far as possible. It is true that this structure is also siloed. It is a silo per product. Any org design can become siloed if it represents the only means for communication and collaboration. The structure here does not help business outcomes that are dependent on the synergies between the products. But that is usually a second order concern. First we care about the products being individually successful. 4
  6. By far the most beautiful slide in my presentation. Rijksmuseum

    in Amsterdam was re-opened to the public just over a month ago after being closed for renovation for over 10 years. Traditionally, museum galleries have a functional organization. So one gallery for sculpture, another for ceramics, a third one for paintings and so on. Each gallery is then curated by a functional specialist. But the new Rijksmuseum has opted for a more integrated or shall we say “cross-functional” organization. Each section in now devoted to a different century and within that section you will find all the artifacts from that period arranged in an integrated holistic display that effectively conveys the story of the age. Give an example In a gallery devoted to the young Rembrandt, for example, examples of his early work are set near finely worked glass and silver by makers Rembrandt knew, furniture of the period, and a portrait by his friend who was an important patron of the arts. Gives us a much better idea of the context in which he did his work "You get a sense of the world in which Rembrandt was producing his art.“ 5
  7. Much as A product analyst gets a sense of the

    world the product is deployed into by working in a cross-functional team that includes deployment specialists. 5
  8. Product quality: Hurts a lot if dev and test are

    different organizations. Time to market: pm, ux, dev, test, ops Customer experience : dev and support –(ability for support folks to quickly check with product dev before making a reply) Product innovation : pm and dev – we learn when we iterate Content rich messaging: Hurts quite a bit if sales, marketing and product mgmt are different orgs. Hurts less if finance, HR, legal and admin are different orgs. Typical objections: consistency of marketing, sales across product lines ; scarcity of talent – so need to centralize The silos book (Silos, Politics and Turf Wars by Patrick Lencioni) has a few other solutions. - A thematic goal - Define shared, quantitative, time-bound objectives + std. operating objectives Surely this is worth a shot but my feeling is - if just that was enough, we wouldn’t have such a strong devops movement advocating the merger of dev and ops. If we want cycle times of a month or less, we have to make sure that a given business outcome only depends on one power center. We need to re-imagine the role of 6
  9. functional vice presidents - the lynchpins of the silos, to

    use a mixed metaphor. 6
  10. The main reason for having a functional VP is let

    them wield org wide influence on their area of expertise. We want to give them autonomy. Dan Pink’s trinity of autonomy, mastery and purpose is now fairly well accepted as key motivating factors for knowledge workers. I find it interesting that he didn’t say authority, mastery and purpose. Autonomy is important but when you give someone autonomy along with control over resources and administrative authority, there is a chance that it sets the stage for territorial behavior. It is problematic when the product owner is the person accountable for business outcome and the VPs control the resources needed for the outcome. So I think it is worth separating autonomy from control over resources. Let VPs have autonomy without control over resources. Nobody reports to them. They don’t control anyone’s time. No small fiefdoms. They should have influence along the lines of their functional specialization. Make them work like how an agile coach would work – by going to the teams – not by forming a team. 7
  11. To compensate for reduction in their authority, they should be

    slightly less accountable than product managers. Approximate description: Influential, respected and powerless. Still on the same level in the hierarchy. Give them title but not control over resources. Treat them on par with product owners – invite them to company off-sites and other things that product managers get invited to. After all, hierarchy manifests in the protocols of communication. Need POs who can respect this kind of autonomy. If POs abuse the situation and disregard the VPs, the product business will soon suffer. The other option is to pair up VPs with product owners – depending on interest and ability. By product owner, I mean the first line of direct accountability for a product’s success. Some call it Product Manager. The role usually has a wide ambit – product vision, pulse of market, interfacing with sales and marketing, keeping in touch with customers, prioritizing features and providing direction to the development teams, keeping a tab on support. There is more than enough scope of work for a two-in-a- box set up. 8
  12. And finally… 9

  13. Next we look at how the tooling landscape and access

    to tooling may result in unintended silos. In one sense, we may say that people can collaborate if they really want to despite all this. That is true for individual initiative. But in the aggregate, the presence of silo-inducing org design will lead to silos. The presence of silo-inducing tooling and access landscape will lead to silos. 10
  14. Marshall McLuhan was one of the most influential voices behind

    the argument that technology isn’t value neutral. He famously said: we shape our tools and thereafter our tools shape us. Therefore it matters a lot how we shape our tools. This argument can be extended beyond technology to processes and structures. For instance, Conway’s law suggests that the interface structure of a software system will reflect the social structure of the organization(s) that produced it. So that’s an example of team structure influencing the shape of software. Therefore, we shouldn’t dismiss the problem of silos as merely a people problem. Yes it is a people problem but organization structure, culture and policies have the ability to amplify or attenuate the problem. 11
  15. So I think it is fair to say : 12

  16. The outer circle represents the sum of all tooling and

    information. Black is the restricted area. Blue is the permitted area. In a traditional set up, we look at what someone needs to do their daily work and just provide access to that part. In a collaboration friendly set up, we should rather look at what needs to be off limits and only restrict that much. When orgs share freely, the individuals within the org are also likely to. If the org norms are about sharing on a strict need basis, then the individuals within the org will also adopt the same policy with respect to each other. 13
  17. So what’s the problem with this team here? We have

    a cross functional team alright. But they are isolated from each other in terms of their tooling and access. Developers can’t access monitoring dashboards to see how the production servers are holding up – they have to ask their ops colleagues. Note that I did not say ssh access to production servers. Just read access to monitoring servers. The product analysts can’t access the sales lead management tool to see why certain prospects did not convert. The marketing and sales guys can’t access the product backlog mgmt tool to see the most up to date set of feature priorities or a likely launch date. This isn’t so bad if everyone is sitting in the same room. But that is often not the case. Not having access is limiting in many ways. It reduces my visibility of the world that my colleagues inhabit. Sometimes I make poorer decisions for want of timely information. Sometimes I am not even aware that some information is available. Repeatedly asking for information feels bureaucratic. Reasons for no access: licensing costs, territorial behaviour – need to know basis Workaround – shared logins – may or may not be permitted by vendor Negotiate for tiered fee structure for observers vs. active users Make it a criteria during product evaluation. Providing universal read access is the tooling equivalent of what Kent Beck refers to as serendipitous communication. Opening up access to tools and information is one way to widen the horizons of team members. Many devs are aware of the benefits of common vocabulary or ubiquitous language as described in the DDD book. The equivalent here is common tooling as far 14
  18. as possible. Different roles within a CF shouldn’t use different

    tools for doing the same things e.g. version control, wiki, knowledge sharing. In fact, infrastructure-as- code is a great example of this 14
  19. When we adopt the notion of infrastructure-as-code, we create a

    common ground of code between the application people and the infrastructure people. This common ground of code, being text is uniformly versionable, diffable and mergable. Contrast this with the case when the infrastructure people use some proprietary infrastructure automation tool that stores its artifacts in a binary format. We now have two very different worlds. 15
  20. 16

  21. Segregation of duties doesn’t mean people can’t talk to each

    other. In fact, collaboration is an essential element of effective risk management. So everybody in the PCI environment still follows devops principles – there’s no physical separation of the teams, and Etsy have hired more people into the various roles (dev, sysadmin, dba, manager, networking) to facilitate collaboration; In the payments environment they “still have to follow the rules: a developer still doesn’t have access to a production database”, but they’ll have dbas working alongside them who they can ask for help, and graphs showing metrics from the database; They use a similar deployment process in their PCI environment to their non-PCI environment, but it includes much more logging on who did what when, and they have roles for QA pusher and production pusher; 17