Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Best Practices Scaling UrbanCode Deploy

Sean Wilbur
February 24, 2016

Best Practices Scaling UrbanCode Deploy

Presentation from InterConnect 2016 on scaling and general best practices for design and maintenance of UrbanCode Deploy servers.

Sean Wilbur

February 24, 2016
Tweet

More Decks by Sean Wilbur

Other Decks in Technology

Transcript

  1. Please Note: 2 • IBM’s statements regarding its plans, directions,

    and intent are subject to change or withdrawal without notice at IBM’s sole discretion. • Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. • The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. • The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. • Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
  2. Agenda • Enterprise Concerns • Component Subsystems • Maintenance •

    Disaster Recovery • High Availability • Monitoring/Performance • Best Practices 4
  3. Enterprise Concerns • How do I backup and plan for

    disaster recovery? • What are the bottlenecks to scaling the solution? • How do I guarantee availability of my server and agents? • What are some best practices in topology-design based on business requirements and constraints? • How do I manage large-scale and very large-scale deployments? • How do I know when I need to scale and where my breaking points are? • How do I plan for upgrades, downtime, and coordinate with the enterprise on a release process? 5
  4. Maintenance: Regular Scheduled Downtime • Upgrade/Fixpacks/Patches ( delivered ~quarterly from

    IBM) cannot be done online • Full backup recommended process involves downtime and complete backups • Any database maintenance • UrbanCode Deploy console can be put in maintenance mode to stop any new deployments or any artifact imports from starting to bleed traffic from the server 7
  5. Maintenance: Complete and Incremental Backup • The documentation provides some

    basic guidance to get started • Simplify your process to cover both backup and disaster recovery • What do we need to recover – Database – Configuration Files – CodeStation – [+ Patterns] Plus OpenStack components KeyStone and HEAT Engine • Complete backups should be done regularly • Can use Console “Maintenance Mode” to bleed transactions off the server for taking the server down and online backup windows • Incremental backup should be happening continuously( <15m interval) 8
  6. Disaster Recovery: Have a failover plan • A global DNS/Load

    Balancer solution is generally used to cut over to failover servers manually during a failure • Understand your SLA for fail over, this will effect how involved your process must be to support it. • Data to restore or host is generally built by extending our backup solution to include replication to the cold standby solution. • Licensing must be part of your plan, the products does not function without a license. IBM Support can assist you in generating a duplicate copy of your production licenses to host with the disaster recovery server at no additional cost. • Ensure that we understand how to fall back to production, reverse replication process and play forward to catch up database/filesystems 9
  7. High-Availability of UrbanCode Deploy • Provides a clustering solution for

    hosting UrbanCode Deploy across multiple servers • Spread traffic across multiple servers • Improved throughput to handle more concurrency for complex and long running processes • Not a magic bullet for performance issues 12
  8. High-Availability: Best Practices and Tips • Shared connectivity to the

    same database means we will be putting more load on a single database/server/instance so plan accordingly • Shared file system connectivity and latency, means that things like it is easy to saturate a single network card while moving only a few large files across the network. • Spanning data centers is possible but recommend that you co-locate servers, coordination overhead needs to be taken into consideration. • Agent Relays should also be configured for failover where resiliency is required. 13
  9. Monitoring: System Level • Server(s) CPU, Memory, Disk I/O, Network

    I/O • Agent Relay(s) CPU, Memory, Disk I/O, Network I/O • Network Storage System I/O to UCD Servers • Database Services Resource and Space Utilization • Notification is important to ensure you find out when there are issues – CPU > 50% for 5 minutes – Disk < 50G • Goal here is to provide a high-level water mark to ensure that the UrbanCode operations team is aware of any infrastructure level issues before they get reported ( proactive versus reactive ) 14
  10. Monitoring: Application Introspection/Troubleshooting • Various tools can be used to

    provide a deeper level of inspection of what is going on in the applications • UrbanCode level tracing can be enabled for specific issues, but debugging and tracing features will affect product performance. • Application Level: Want to look at the “Application” specific data so we can see time spent in the web, app, and database tiers of the applications • Database level monitoring: temporary space, buffer pools, expensive queries, large tables, missing indexes • Example monitoring tools IBM APM Suite, NewRelic, DynaTrace, 16
  11. Performance: Setting a baseline • Every customer is a snowflake

    here as this depends so much on what your environment looks like • One major reason for monitoring is for you to develop a baseline of what is “normal” for your deployment • Internal UrbanCode Performance team formed in 2015 to help facilitate our focus on improving • Performance tests are built using API to build out, monitor, and validate • Suite being used as part of the UrbanCode Deploy’s internal build/ release process to help identify any changes positive or negative 18
  12. Performance: How far can we scale? • HW Sizing Recommendations

    • Part of the Performance Team work is to work on break the system – Number of Agents ( 10-15K Agents internally our maximum testing ) – Number of Applications – Components per Applications – Component Version Sizes – Concurrency • Practical limitations with setting up this many agents, we are scaling out to SoftLayer from our physical lab in Cleveland to do this • How many servers do you need to get here? 19
  13. Best Practices: What is your Goal State? • Understanding what

    your target is will drive the decisions • Who will manage workflow changes? – Central vs self-directed teams • What are your governance points? – Preparing for continuous delivery means these will need to be automated eventually • How to handle inter-Application dependencies? – Release management strategy affects how you design your automated deployment processes 20
  14. Best Practices: General • Server Configuration • Naming Conventions •

    Organization • Re-use via Component and Application Templates • Design of Applications and Components – provide a logical representation of path to production – Number of Components to Applications • CodeStation usage for smart staging and staying on top of cleanup • Resource organization, limited use of top level Resources, common standards here help at scale ( /<Business Unit>/<App>/<Env>/… ) 21
  15. Best Practices: UrbanCode Staging Environment • Environment that is a

    clone of your Production server including, database, files ( CodeStation, process logs), and a similar configuration ( HA vs non-HA) • Having a pseudo-production environment is crucial to being able for a few key scenarios – Validating Production updates/upgrades – Workflow development – User Training – Destructive Configuration/Integration Testing ( Security, API testing, plugin development ) • Can generally be built using a combination of the existing backup and/or Disaster Recovery process to re-create production at a point in time. 22
  16. Best Practices: Agent Relays • Yes you should use them

    ☺ • Roughly estimate 1 Agent Relay for every ~1000 Agents • Implementing Agent Relays is part of a best practice deployment • Minimizes concurrent connections to the server • When using CodeStation you can see immediate benefits from implementing Artifact Caching and/or Artifact Staging to minimize the re- download of the same files from the server during deployments. 23
  17. Best Practices: Database and FileSystem Cleanup • Component Cleanup Settings

    – To archive or not – System / Component /Environment values • Audit Cleanup – Cleanup audit table by date via System Settings – 6.2.1 – Adding support for regular process request cleanup • Resources – Cleaning up offline agents – Removing unused items from retired or completed projects 24
  18. Real World Examples: Prototypical Deployment • Single virtual server hosted

    in VMWare 8-12 cores, 8-16G, 6-12G Heap • ~1000 agents • 1-4 Agent Relays • Linux OS RHEL 6/7 • Oracle Java Virtual Machine 1.7/1.8 • Oracle Database via shared service • Oracle DataGuard for database replication • NAS hosted CodeStation • rsync for offsite CodeStation replication • 1-3 hour SLA for Disaster Recovery 25
  19. Thank You Your Feedback is Important! Text TRANSFORM to 41411


    to receive a copy of the paper,
 “Streamlined Enterprise Release Management” Access the InterConnect 2016 Conference Attendee Portal to complete your session surveys from your smartphone, laptop or conference kiosk.
  20. Resources • UrbanCode Information Centers • UrbanCode Microsite • UrbanCode

    Forums • UrbanCode Deploy Performance Whitepaper – Coming Soon** 27
  21. Notices and Disclaimers 28 Copyright © 2016 by International Business

    Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law
  22. Notices and Disclaimers Con’t. 29 Information concerning non-IBM products was

    obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.