Slide 1

Slide 1 text

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead Source Control and Concurrent Development for OBIEE Robin Moffatt, Principal Consultant Rittman Mead Riga Dev Day, 22 January 2015

Slide 2

Slide 2 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com About Me •Principal Consultant with Rittman Mead •Previously OBIEE/DW developer at large UK retailer •Previously SQL Server DBA, Business Objects, 
 DB2, COBOL….
 •Newly minted Oracle ACE! 
 •Frequent blogger for Rittman Mead : http://ritt.md/rmoff •Twitter: @rmoff •IRC: rmoff / #obihackers / freenode

Slide 3

Slide 3 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com •Oracle BI and DW Gold partner •Winner of five UKOUG Partner of the Year awards in 2013 and 2014 - including BI •World leading specialist partner for technical excellence, 
 solutions delivery and innovation in Oracle BI •Approximately 80 consultants worldwide •All expert in Oracle BI and DW •Offices in US (Atlanta), Europe, Australia and India •Skills in broad range of supporting Oracle tools: ‣OBIEE, OBIA ‣ODIEE ‣Essbase, Oracle OLAP ‣GoldenGate ‣Endeca About Rittman Mead

Slide 4

Slide 4 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com OBIEE Software Development Life Cycle (SDLC) •A successful OBIEE implementation is more than just data modelling and analytics.
 •Software Development LifeCycle (SDLC) is of great importance to all involved in OBIEE: ‣Developers ‣Release Managers ‣Testing ‣Project & Delivery Management 
 •Two key areas of importance: ‣Source Control ‣Concurrent Development

Slide 5

Slide 5 text

[email protected] www.rittmanmead.com @rittmanmead Source Control in OBIEE 
 (Source Control == Revision Control == Version Control) 5

Slide 6

Slide 6 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Without Source Control We Have Chaos!

Slide 7

Slide 7 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Why Do We Need Source Control? •Every mature OBIEE development should be using Source Control!
 ‣Reduce risk when deploying ‣Improve ease of maintenance and development ‣Pre-requisite to concurrent development method (discussed later)

Slide 8

Slide 8 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Why Do We Need Source Control? Who changed it? What did they change? When did they change it?

Slide 9

Slide 9 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com •Label (tag) points in the code line
 ‣Mark code bundles for release to a given environment -Audit what went where when
 ‣Revert to “known-good” point Why Do We Need Source Control?

Slide 10

Slide 10 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Isn’t Source Control Scary and Complicated? •No! •Well… maybe a bit •But you don’t really have a choice

Slide 11

Slide 11 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Without Source Control We Have Chaos!

Slide 12

Slide 12 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source Control is Mandatory •Any mature OBIEE development should be using Source Control
 •One developer or one hundred, still relevant

Slide 13

Slide 13 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com What Are the Tools for Source Control in OBIEE? •Familiarity with the tool and organisational acceptance is key
 •Two common ones (but plenty of others) ‣git ‣Subversion (SVN) •If you are wanting to do concurrent development, git is excellent ‣Modern technology ‣Distributed

Slide 14

Slide 14 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Just Use Source Control Already

Slide 15

Slide 15 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com OBIEE Artefacts in Source Control •OBIEE primarily stores its objects in binary form ‣We’ll talk about MDS XML later
 •Source Control tools and their merge workflows are designed around plaintext objects
 •Therefore Source Control becomes a snapshot / point-in-time capture of your OBIEE environment RPD Presentation Catalog Security Component configuration Custom skin/style ETL, DB schemas, etc

Slide 16

Slide 16 text

[email protected] www.rittmanmead.com @rittmanmead Concurrent RPD Development in OBIEE 16

Slide 17

Slide 17 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com The RPD •At the core of OBIEE is the RPD - the brains of the operation! •A metadata model that generates SQL requests at runtime from user report requests •Most concurrent OBIEE development problems centre around the RPD

Slide 18

Slide 18 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com The Problem With Serial RPD Development •It doesn't scale well •Releases are complicated and risky Test Prod Dev Multiple
 Developers

Slide 19

Slide 19 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Feature Driven Development •Development is broken up into “Features”, such as: •New logical column •Add a dimension •Fix a bug •Great for flexible (Agile/agile) approach to delivery •Features only released when ready •Features don't block others •Selectively release features

Slide 20

Slide 20 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com OBIEE Development Nirvana •RPD development rate that scales with your team
 •Feature-Driven Development •Full source control integration •Because of the necessary framework, makes Continuous Integration very possible

Slide 21

Slide 21 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com The Harsh Reality •OBIEE does not natively support concurrent development integrated with source control ‣And it never will until Oracle re-engineer OBIEE and/or the RPD storage format •All options require a defined process to be followed and associated training for staff •There is no perfect software-based solution that will force ‘bad’ developers to be good •You will always need a “Dungeon Master” to oversee and track developments, manage conflicts, and support developers.

Slide 22

Slide 22 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com The Harsh Reality •There are four options to consider for 
 concurrent development in OBIEE ‣Online editing ‣MUD ‣Source control + MDS XML RPD ‣Source control + Binary RPD •They all suck! •This is 2015, not 1995!
 •Which sucks the least?… ‣What can we do to mitigate the suckiness?

Slide 23

Slide 23 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Online RPD edits •Zero setup required •RPD updated automatically - no redeployment or restarts needed •Concurrent development limited by high level locks 
 taken when objects checked out •Everyone's favourite error : "Transactional update failed” Developer #1 Developer #2 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 24

Slide 24 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Online RPD edits •No audit of changes made •No native integration with source control •No tagging or packaging for releases •No rollback of changes / restore to point in time •Conclusion : ‣Good for individual developers in isolation making sandbox changes ‣Useless anywhere else in the development cycle process 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 25

Slide 25 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Multi User Development Environment (MUD) •Administrator divides main repository into projects; self-contained RPD subsets
 •Master repository is then published to a network share
 •Projects are worked on independently, 
 and then merged back into the master RPD
 •Uses the repository compare and 
 merge features under the covers OBIEE Sandbox Environment Developer #1 OBIEE Sandbox Environment Developer #2 MUD Administrator Master 
 Repository Subset
 RPD Subset
 RPD 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 26

Slide 26 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Multi User Development Environment (MUD) •Closest thing there is to native concurrent development functionality in OBIEE
 •Onus and power for conflict resolution is with the developer, not source master
 •Known bugs (eg Variables, parent-child tables) •Doesn’t easily support feature-driven development •No native integration with source control •No tagging or packaging for releases •No rollback of changes / restore to point in time 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 27

Slide 27 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com We’ve Got To Have Source Control! •Any project should always be using source control •To do properly flexible concurrent development, we need to take a feature-driven approach ➡Thus we use branch based source control •But - this requires the ability to merge files within the source control tool from multiple branches into one develop feature/RM-x feature/RM-y

Slide 28

Slide 28 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source Control Merging •When branches are combined in source control, the tool will automagically merge files and folders
 •Source control tools can only automatically merge files that are not binary…

Slide 29

Slide 29 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Is MDS XML the answer? •The RPD is broken up into a hierarchical set of XML files representing object types such as Logical Tables, Logical Table Sources, etc •Because there are multiple plain text files, can the source control tool can carry out the merge itself?

Slide 30

Slide 30 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + MDS XML RPD •Package up all artefacts in Source Control, providing release and rollback capabilities •Store the RPD in MDS XML, a plaintext alternative to the binary RPD format •Deceptively alluring option for concurrent 
 development: 1. An RPD stored in MDS XML is plain text 2. git can merge code that is plain text from 
 multiple branches 3. Let's merge MDS XML with git! 4. Uh oh…. 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 31

Slide 31 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + MDS XML RPD - the good ‣It works in simple and specific use-cases Branch A Branch B Merged RPD Dim Stores Dim Staff 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 32

Slide 32 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + MDS XML RPD ‣Separate commits in git 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 33

Slide 33 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + MDS XML RPD ‣Happily merged! Dim Stores Dim Staff 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 34

Slide 34 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + MDS XML RPD - the bad •Even though MDS XML is not binary, it is still a structured file format 
 containing with application logic ‣Only the Administration Tool understands the RPD object model •Source control tools are not intelligent enough to be able to parse it to understand conflicts 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 35

Slide 35 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + MDS XML RPD - the bad •Let’s add new columns to same table in separate branches
 
 
 
 
 
 
 •Should be simple - there’s no logical conflict Branch A Branch B 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 36

Slide 36 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + MDS XML RPD ‣Source control tool looks at the two versions of MDS XML file as dumb plaintext, and throws a conflict ‣User now has to manually fix the XML, which is tricky and error prone ‣It’s also unnecessary - merging through the Administration Tool would be automatic 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 37

Slide 37 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + MDS XML RPD - the ugly ‣MDS XML files are named based on internal GUIDs (mdsid) ‣Same logical object can have different mdsid, and so source control sees it as unique ‣Allowing source control tool to merge files can create a corrupt RPD ‣For example, an identical object created in multiple branches Branch A Branch B 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 38

Slide 38 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + MDS XML RPD - the ugly ‣Same object gets a different mdsid, thus a different filename and so gets duplicated in the resulting merge, because source control doesn’t see it as the same object
 
 
 
 ‣Result: dodgy RPD, user cannot edit the SALES table.
 
 
 
 
 
 ‣Exactly what the Equalize function was designed for in the AdminTool merge! The same object definition! 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 39

Slide 39 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + MDS XML RPD - Summary •You cannot reliably merge the RPD except through Oracle’s own tools ‣MDS XML, whilst plaintext, is still structured -Unnecessary conflicts when merging -Potential inconsistencies / corruption ‣MDS XML uses GUIDs to name files, so source control may
 inadvertently duplicate objects without notification -RPD problems may not always be immediately apparent •MDS XML merging may work for simple isolated changes to 
 individual objects; as a development strategy this is
 restrictive. 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 40

Slide 40 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + Binary RPD •Package up all artefacts in Source Control, providing release and rollback capabilities •Use the common git-flow branching method, with development divided up into features ‣Provides clear structure and process for managing branches and releases ‣RPD stored as binary. Three-way merge with OBIEE tools done where necessary. develop feature/RM-x feature/RM-x Original Current Modified 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD ‘Secret Sauce’ : merge.bat script

Slide 41

Slide 41 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Source control + Binary RPD •Conflict resolution script: ‣Uses git-flow to determine 3-way merge candidates ‣Attempts automatic merge (comparerpd/patchrpd) ‣Launches Administration Tool for user interaction if necessary 1. Online edit 2. MUD 3. Source control + MDS XML RPD 4. Source control + Binary RPD

Slide 42

Slide 42 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com So which methods sucks the least? Multiple developers Integration with source control Feature driven development Ease of use Overall Source control + Binary RPD Yes Yes Yes Medium Not perfect, but the best option MUD Yes Manual No Medium Could be worse Source control + MDS XML RPD Yes Yes Only if objects fully isolated Medium Requires too much process to mitigate against RPD corruption and unnecessary merge conflicts Unnecessary conflict merges. Online editing Not really No No Easy Not viable

Slide 43

Slide 43 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com The Harsh Reality •OBIEE does not natively support concurrent development integrated with source control ‣And it never will until Oracle re-engineer OBIEE and/or the RPD storage format •All options require a defined process to be followed and associated training for staff •There is no perfect software-based solution that will force ‘bad’ developers to be good •You will always need a “Dungeon Master” to oversee and track developments, manage conflicts, and support developers.

Slide 44

Slide 44 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Checking It All Still Works •For concurrent development to be successful, you need automated testing Before After Test Result: Fail

Slide 45

Slide 45 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com Summary •If you're not doing source control, go and do it. ‣Now. ‣Even for serial development ‣Even if you’re the only developer •If you are doing source control, then you can use it to support concurrent development ‣But don't go anywhere near concurrent development until you've sorted source control •The only way to do safe and reliable concurrent development is using the binary RPD stored in source control such as git

Slide 46

Slide 46 text

T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com #EOF ✴email: [email protected] ✴web: http://ritt.md/rmoff ✴twitter: @rmoff ✴IRC: rmoff / #obihackers / freenode

Slide 47

Slide 47 text

www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead Source Control and Concurrent Development for OBIEE Robin Moffatt, Principal Consultant Rittman Mead Riga Dev Day January 2015