Source Control, Versioning and Concurrent Development for OBIEE using Git

In this session we will look at options for integrating OBIEE 11g with source control tools including Subversion and Git, focusing on Git as our preferred SCM tool. We will look at how Git can make use of the new MDS XML Documents repository format, and see how it permits versioning of individual repository element over time. We'll also look at how Git supports concurrent development including branching and merging of repositories, and whether Git + MDS XML can replace MUDE as a platform for concurrent development of the repository.

Transcript

1. www.rittmanmead.com inquiries@rittmanmead.com @rittmanmead www.facebook.com/rittmanmead Source Control and Concurrent Development for OBIEE Robin Moffatt, Principal Consultant Rittman Mead Riga Dev Day, 22 January 2015
2. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com About Me •Principal Consultant with Rittman Mead •Previously OBIEE/DW developer at large UK retailer •Previously SQL Server DBA, Business Objects, 
 DB2, COBOL….
 •Newly minted Oracle ACE! 
 •Frequent blogger for Rittman Mead : http://ritt.md/rmoff •Twitter: @rmoff •IRC: rmoff / #obihackers / freenode
3. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com •Oracle BI and DW Gold partner •Winner of five UKOUG Partner of the Year awards in 2013 and 2014 - including BI •World leading specialist partner for technical excellence, 
 solutions delivery and innovation in Oracle BI •Approximately 80 consultants worldwide •All expert in Oracle BI and DW •Offices in US (Atlanta), Europe, Australia and India •Skills in broad range of supporting Oracle tools: ‣OBIEE, OBIA ‣ODIEE ‣Essbase, Oracle OLAP ‣GoldenGate ‣Endeca About Rittman Mead
4. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com OBIEE Software Development Life Cycle (SDLC) •A successful OBIEE implementation is more than just data modelling and analytics.
 •Software Development LifeCycle (SDLC) is of great importance to all involved in OBIEE: ‣Developers ‣Release Managers ‣Testing ‣Project & Delivery Management 
 •Two key areas of importance: ‣Source Control ‣Concurrent Development
5. info@rittmanmead.com www.rittmanmead.com @rittmanmead Source Control in OBIEE 
 (Source Control == Revision Control == Version Control) 5
6. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Without Source Control We Have Chaos!
7. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Why Do We Need Source Control? •Every mature OBIEE development should be using Source Control!
 ‣Reduce risk when deploying ‣Improve ease of maintenance and development ‣Pre-requisite to concurrent development method (discussed later)
8. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Why Do We Need Source Control? Who changed it? What did they change? When did they change it?
9. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com •Label (tag) points in the code line
 ‣Mark code bundles for release to a given environment -Audit what went where when
 ‣Revert to “known-good” point Why Do We Need Source Control?
10. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Isn’t Source Control Scary and Complicated? •No! •Well… maybe a bit •But you don’t really have a choice
11. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Without Source Control We Have Chaos!
12. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source Control is Mandatory •Any mature OBIEE development should be using Source Control
 •One developer or one hundred, still relevant
13. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com What Are the Tools for Source Control in OBIEE? •Familiarity with the tool and organisational acceptance is key
 •Two common ones (but plenty of others) ‣git ‣Subversion (SVN) •If you are wanting to do concurrent development, git is excellent ‣Modern technology ‣Distributed
14. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Just Use Source Control Already
15. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com OBIEE Artefacts in Source Control •OBIEE primarily stores its objects in binary form ‣We’ll talk about MDS XML later
 •Source Control tools and their merge workflows are designed around plaintext objects
 •Therefore Source Control becomes a snapshot / point-in-time capture of your OBIEE environment RPD Presentation Catalog Security Component configuration Custom skin/style ETL, DB schemas, etc
16. info@rittmanmead.com www.rittmanmead.com @rittmanmead Concurrent RPD Development in OBIEE 16
17. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com The RPD •At the core of OBIEE is the RPD - the brains of the operation! •A metadata model that generates SQL requests at runtime from user report requests •Most concurrent OBIEE development problems centre around the RPD
18. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com The Problem With Serial RPD Development •It doesn't scale well •Releases are complicated and risky Test Prod Dev Multiple
 Developers
19. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Feature Driven Development •Development is broken up into “Features”, such as: •New logical column •Add a dimension •Fix a bug •Great for flexible (Agile/agile) approach to delivery •Features only released when ready •Features don't block others •Selectively release features
20. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com OBIEE Development Nirvana •RPD development rate that scales with your team
 •Feature-Driven Development •Full source control integration •Because of the necessary framework, makes Continuous Integration very possible
21. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com The Harsh Reality •OBIEE does not natively support concurrent development integrated with source control ‣And it never will until Oracle re-engineer OBIEE and/or the RPD storage format •All options require a defined process to be followed and associated training for staff •There is no perfect software-based solution that will force ‘bad’ developers to be good •You will always need a “Dungeon Master” to oversee and track developments, manage conflicts, and support developers.
22. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com The Harsh Reality •There are four options to consider for 
 concurrent development in OBIEE ‣Online editing ‣MUD ‣Source control + MDS XML RPD ‣Source control + Binary RPD •They all suck! •This is 2015, not 1995!
 •Which sucks the least?… ‣What can we do to mitigate the suckiness?
23. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Online RPD edits •Zero setup required •RPD updated automatically - no redeployment or restarts needed •Concurrent development limited by high level locks 
 taken when objects checked out •Everyone's favourite error : "Transactional update failed” Developer #1 Developer #2 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
24. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Online RPD edits •No audit of changes made •No native integration with source control •No tagging or packaging for releases •No rollback of changes / restore to point in time •Conclusion : ‣Good for individual developers in isolation making sandbox changes ‣Useless anywhere else in the development cycle process 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
25. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Multi User Development Environment (MUD) •Administrator divides main repository into projects; self-contained RPD subsets
 •Master repository is then published to a network share
 •Projects are worked on independently, 
 and then merged back into the master RPD
 •Uses the repository compare and 
 merge features under the covers OBIEE Sandbox Environment Developer #1 OBIEE Sandbox Environment Developer #2 MUD Administrator Master 
 Repository Subset
 RPD Subset
 RPD 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
26. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Multi User Development Environment (MUD) •Closest thing there is to native concurrent development functionality in OBIEE
 •Onus and power for conflict resolution is with the developer, not source master
 •Known bugs (eg Variables, parent-child tables) •Doesn’t easily support feature-driven development •No native integration with source control •No tagging or packaging for releases •No rollback of changes / restore to point in time 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
27. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com We’ve Got To Have Source Control! •Any project should always be using source control •To do properly flexible concurrent development, we need to take a feature-driven approach ➡Thus we use branch based source control •But - this requires the ability to merge files within the source control tool from multiple branches into one develop feature/RM-x feature/RM-y
28. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source Control Merging •When branches are combined in source control, the tool will automagically merge files and folders
 •Source control tools can only automatically merge files that are not binary…
29. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Is MDS XML the answer? •The RPD is broken up into a hierarchical set of XML files representing object types such as Logical Tables, Logical Table Sources, etc •Because there are multiple plain text files, can the source control tool can carry out the merge itself?
30. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + MDS XML RPD •Package up all artefacts in Source Control, providing release and rollback capabilities •Store the RPD in MDS XML, a plaintext alternative to the binary RPD format •Deceptively alluring option for concurrent 
 development: 1. An RPD stored in MDS XML is plain text 2. git can merge code that is plain text from 
 multiple branches 3. Let's merge MDS XML with git! 4. Uh oh…. 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
31. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + MDS XML RPD - the good ‣It works in simple and specific use-cases Branch A Branch B Merged RPD Dim Stores Dim Staff 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
32. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + MDS XML RPD ‣Separate commits in git 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
33. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + MDS XML RPD ‣Happily merged! Dim Stores Dim Staff 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
34. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + MDS XML RPD - the bad •Even though MDS XML is not binary, it is still a structured file format 
 containing with application logic ‣Only the Administration Tool understands the RPD object model •Source control tools are not intelligent enough to be able to parse it to understand conflicts 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
35. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + MDS XML RPD - the bad •Let’s add new columns to same table in separate branches
 
 
 
 
 
 
 •Should be simple - there’s no logical conflict Branch A Branch B 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
36. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + MDS XML RPD ‣Source control tool looks at the two versions of MDS XML file as dumb plaintext, and throws a conflict ‣User now has to manually fix the XML, which is tricky and error prone ‣It’s also unnecessary - merging through the Administration Tool would be automatic 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
37. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + MDS XML RPD - the ugly ‣MDS XML files are named based on internal GUIDs (mdsid) ‣Same logical object can have different mdsid, and so source control sees it as unique ‣Allowing source control tool to merge files can create a corrupt RPD ‣For example, an identical object created in multiple branches Branch A Branch B 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
38. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + MDS XML RPD - the ugly ‣Same object gets a different mdsid, thus a different filename and so gets duplicated in the resulting merge, because source control doesn’t see it as the same object
 
 
 
 ‣Result: dodgy RPD, user cannot edit the SALES table.
 
 
 
 
 
 ‣Exactly what the Equalize function was designed for in the AdminTool merge! The same object definition! 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
39. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + MDS XML RPD - Summary •You cannot reliably merge the RPD except through Oracle’s own tools ‣MDS XML, whilst plaintext, is still structured -Unnecessary conflicts when merging -Potential inconsistencies / corruption ‣MDS XML uses GUIDs to name files, so source control may
 inadvertently duplicate objects without notification -RPD problems may not always be immediately apparent •MDS XML merging may work for simple isolated changes to 
 individual objects; as a development strategy this is
 restrictive. 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
40. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + Binary RPD •Package up all artefacts in Source Control, providing release and rollback capabilities •Use the common git-flow branching method, with development divided up into features ‣Provides clear structure and process for managing branches and releases ‣RPD stored as binary. Three-way merge with OBIEE tools done where necessary. develop feature/RM-x feature/RM-x Original Current Modified 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD ‘Secret Sauce’ : merge.bat script
41. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Source control + Binary RPD •Conflict resolution script: ‣Uses git-flow to determine 3-way merge candidates ‣Attempts automatic merge (comparerpd/patchrpd) ‣Launches Administration Tool for user interaction if necessary 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD
42. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com So which methods sucks the least? Multiple developers Integration with source control Feature driven development Ease of use Overall Source control + Binary RPD Yes Yes Yes Medium Not perfect, but the best option MUD Yes Manual No Medium Could be worse Source control + MDS XML RPD Yes Yes Only if objects fully isolated Medium Requires too much process to mitigate against RPD corruption and unnecessary merge conflicts Unnecessary conflict merges. Online editing Not really No No Easy Not viable
43. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com The Harsh Reality •OBIEE does not natively support concurrent development integrated with source control ‣And it never will until Oracle re-engineer OBIEE and/or the RPD storage format •All options require a defined process to be followed and associated training for staff •There is no perfect software-based solution that will force ‘bad’ developers to be good •You will always need a “Dungeon Master” to oversee and track developments, manage conflicts, and support developers.
44. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Checking It All Still Works •For concurrent development to be successful, you need automated testing Before After Test Result: Fail
45. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com Summary •If you're not doing source control, go and do it. ‣Now. ‣Even for serial development ‣Even if you’re the only developer •If you are doing source control, then you can use it to support concurrent development ‣But don't go anywhere near concurrent development until you've sorted source control •The only way to do safe and reliable concurrent development is using the binary RPD stored in source control such as git
46. T : +44 (0) 1273 911 268 (UK) E : info@rittmanmead.com W : www.rittmanmead.com #EOF ✴email: robin.moffatt@rittmanmead.com ✴web: http://ritt.md/rmoff ✴twitter: @rmoff ✴IRC: rmoff / #obihackers / freenode
47. www.rittmanmead.com inquiries@rittmanmead.com @rittmanmead www.facebook.com/rittmanmead Source Control and Concurrent Development for OBIEE Robin Moffatt, Principal Consultant Rittman Mead Riga Dev Day January 2015