Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Source Control and Concurrent Development for OBIEE

Source Control and Concurrent Development for OBIEE

In this session we will look at options for integrating OBIEE 11g with source control tools including Subversion and Git, focusing on Git as our preferred SCM tool. We will look at how Git can make use of the new MDS XML Documents repository format, and see how it permits versioning of individual repository element over time. We'll also look at how Git supports concurrent development including branching and merging of repositories, and whether Git + MDS XML can replace MUDE as a platform for concurrent development of the repository.

Robin Moffatt

January 22, 2015
Tweet

More Decks by Robin Moffatt

Other Decks in Technology

Transcript

  1. www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
    Source Control and
    Concurrent Development for OBIEE
    Robin Moffatt, Principal Consultant
    Rittman Mead
    Riga Dev Day, 22 January 2015

    View Slide

  2. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    About Me
    •Principal Consultant with Rittman Mead
    •Previously OBIEE/DW developer at large UK retailer
    •Previously SQL Server DBA, Business Objects, 

    DB2, COBOL….

    •Newly minted Oracle ACE! 

    •Frequent blogger for Rittman Mead : http://ritt.md/rmoff
    •Twitter: @rmoff
    •IRC: rmoff / #obihackers / freenode

    View Slide

  3. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    •Oracle BI and DW Gold partner
    •Winner of five UKOUG Partner of the Year awards in 2013 and 2014 - including BI
    •World leading specialist partner for technical excellence, 

    solutions delivery and innovation in Oracle BI
    •Approximately 80 consultants worldwide
    •All expert in Oracle BI and DW
    •Offices in US (Atlanta), Europe, Australia and India
    •Skills in broad range of supporting Oracle tools:
    ‣OBIEE, OBIA
    ‣ODIEE
    ‣Essbase, Oracle OLAP
    ‣GoldenGate
    ‣Endeca
    About Rittman Mead

    View Slide

  4. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    OBIEE Software Development Life Cycle (SDLC)
    •A successful OBIEE implementation is more than just data modelling and analytics.

    •Software Development LifeCycle (SDLC) is of great importance to all involved in OBIEE:
    ‣Developers
    ‣Release Managers
    ‣Testing
    ‣Project & Delivery Management 

    •Two key areas of importance:
    ‣Source Control
    ‣Concurrent Development

    View Slide

  5. [email protected] www.rittmanmead.com @rittmanmead
    Source Control in OBIEE

    (Source Control == Revision Control == Version Control)
    5

    View Slide

  6. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Without Source Control We Have Chaos!

    View Slide

  7. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Why Do We Need Source Control?
    •Every mature OBIEE development should be using Source Control!

    ‣Reduce risk when deploying
    ‣Improve ease of maintenance and development
    ‣Pre-requisite to concurrent development method (discussed later)

    View Slide

  8. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Why Do We Need Source Control?
    Who changed it?
    What did they change?
    When did they change it?

    View Slide

  9. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    •Label (tag) points in the code line

    ‣Mark code bundles for release to a given environment
    -Audit what went where when

    ‣Revert to “known-good” point
    Why Do We Need Source Control?

    View Slide

  10. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Isn’t Source Control Scary and Complicated?
    •No!
    •Well… maybe a bit
    •But you don’t really have a choice

    View Slide

  11. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Without Source Control We Have Chaos!

    View Slide

  12. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source Control is Mandatory
    •Any mature OBIEE development should be using Source Control

    •One developer or one hundred, still relevant

    View Slide

  13. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    What Are the Tools for Source Control in OBIEE?
    •Familiarity with the tool and organisational acceptance is key

    •Two common ones (but plenty of others)
    ‣git
    ‣Subversion (SVN)
    •If you are wanting to do concurrent development, git is excellent
    ‣Modern technology
    ‣Distributed

    View Slide

  14. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Just Use Source Control Already

    View Slide

  15. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    OBIEE Artefacts in Source Control
    •OBIEE primarily stores its objects in binary form
    ‣We’ll talk about MDS XML later

    •Source Control tools and their merge workflows are designed around plaintext objects

    •Therefore Source Control becomes a snapshot / point-in-time capture of your OBIEE
    environment
    RPD
    Presentation Catalog
    Security
    Component configuration
    Custom skin/style
    ETL, DB schemas, etc

    View Slide

  16. [email protected] www.rittmanmead.com @rittmanmead
    Concurrent RPD Development in OBIEE
    16

    View Slide

  17. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    The RPD
    •At the core of OBIEE is the RPD - the brains of the operation!
    •A metadata model that generates SQL requests at runtime
    from user report requests
    •Most concurrent OBIEE development problems centre
    around the RPD

    View Slide

  18. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    The Problem With Serial RPD Development
    •It doesn't scale well
    •Releases are complicated and risky
    Test Prod
    Dev
    Multiple

    Developers

    View Slide

  19. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Feature Driven Development
    •Development is broken up into “Features”, such as:
    •New logical column
    •Add a dimension
    •Fix a bug
    •Great for flexible (Agile/agile) approach to delivery
    •Features only released when ready
    •Features don't block others
    •Selectively release features

    View Slide

  20. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    OBIEE Development Nirvana
    •RPD development rate that scales with your team

    •Feature-Driven Development
    •Full source control integration
    •Because of the necessary framework, makes
    Continuous Integration very possible

    View Slide

  21. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    The Harsh Reality
    •OBIEE does not natively support concurrent
    development integrated with source control
    ‣And it never will until Oracle re-engineer
    OBIEE and/or the RPD storage format
    •All options require a defined process to be
    followed and associated training for staff
    •There is no perfect software-based solution
    that will force ‘bad’ developers to be
    good
    •You will always need a “Dungeon Master”
    to oversee and track developments, manage
    conflicts, and support developers.

    View Slide

  22. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    The Harsh Reality
    •There are four options to consider for 

    concurrent development in OBIEE
    ‣Online editing
    ‣MUD
    ‣Source control + MDS XML RPD
    ‣Source control + Binary RPD
    •They all suck!
    •This is 2015, not 1995!

    •Which sucks the least?…
    ‣What can we do to mitigate the suckiness?

    View Slide

  23. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Online RPD edits
    •Zero setup required
    •RPD updated automatically - no redeployment or restarts needed
    •Concurrent development limited by high level locks 

    taken when objects checked out
    •Everyone's favourite error : "Transactional update failed”
    Developer #1
    Developer #2
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  24. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Online RPD edits
    •No audit of changes made
    •No native integration with source control
    •No tagging or packaging for releases
    •No rollback of changes / restore to point in time
    •Conclusion :
    ‣Good for individual developers in isolation making sandbox changes
    ‣Useless anywhere else in the development cycle process
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  25. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Multi User Development Environment (MUD)
    •Administrator divides main repository into
    projects; self-contained RPD subsets

    •Master repository is then published to a
    network share

    •Projects are worked on independently, 

    and then merged back into the master
    RPD

    •Uses the repository compare and 

    merge features under the covers
    OBIEE Sandbox Environment
    Developer #1
    OBIEE Sandbox Environment
    Developer #2
    MUD Administrator
    Master 

    Repository
    Subset

    RPD
    Subset

    RPD
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  26. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Multi User Development Environment (MUD)
    •Closest thing there is to native concurrent development functionality in OBIEE

    •Onus and power for conflict resolution is with the developer, not source master

    •Known bugs (eg Variables, parent-child tables)
    •Doesn’t easily support feature-driven development
    •No native integration with source control
    •No tagging or packaging for releases
    •No rollback of changes / restore to point in time
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  27. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    We’ve Got To Have Source Control!
    •Any project should always be using source control
    •To do properly flexible concurrent development, we need to take a feature-driven approach
    ➡Thus we use branch based source control
    •But - this requires the ability to merge files within the source control tool from multiple
    branches into one
    develop
    feature/RM-x
    feature/RM-y

    View Slide

  28. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source Control Merging
    •When branches are combined in source control, the tool will automagically merge files and
    folders

    •Source control tools can only automatically merge files that are not binary…

    View Slide

  29. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Is MDS XML the answer?
    •The RPD is broken up into a hierarchical set of XML files representing
    object types such as Logical Tables, Logical Table Sources, etc
    •Because there are multiple plain text files, can the source control tool
    can carry out the merge itself?

    View Slide

  30. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + MDS XML RPD
    •Package up all artefacts in Source Control, providing release and rollback capabilities
    •Store the RPD in MDS XML, a plaintext alternative to the binary RPD format
    •Deceptively alluring option for concurrent 

    development:
    1. An RPD stored in MDS XML is plain text
    2. git can merge code that is plain text from 

    multiple branches
    3. Let's merge MDS XML with git!
    4. Uh oh….
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  31. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + MDS XML RPD - the good
    ‣It works in simple and specific use-cases
    Branch A
    Branch B
    Merged RPD
    Dim Stores
    Dim Staff
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  32. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + MDS XML RPD
    ‣Separate commits in git
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  33. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + MDS XML RPD
    ‣Happily merged!
    Dim Stores
    Dim Staff
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  34. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + MDS XML RPD - the bad
    •Even though MDS XML is not binary, it is still a structured file format 

    containing with application logic
    ‣Only the Administration Tool understands the RPD object model
    •Source control tools are not intelligent enough to be able to parse it to understand conflicts
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  35. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + MDS XML RPD - the bad
    •Let’s add new columns to same table in separate branches







    •Should be simple - there’s no logical conflict
    Branch A Branch B
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  36. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + MDS XML RPD
    ‣Source control tool looks at the two versions
    of MDS XML file as dumb plaintext, and
    throws a conflict
    ‣User now has to manually fix the XML, which is
    tricky and error prone
    ‣It’s also unnecessary - merging through the
    Administration Tool would be automatic
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  37. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + MDS XML RPD - the ugly
    ‣MDS XML files are named based on internal GUIDs (mdsid)
    ‣Same logical object can have different mdsid, and so source control sees it as unique
    ‣Allowing source control tool to merge files can create a corrupt RPD
    ‣For example, an identical object created in multiple branches
    Branch A Branch B
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  38. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + MDS XML RPD - the ugly
    ‣Same object gets a different mdsid, thus a different filename and so gets duplicated in the
    resulting merge, because source control doesn’t see it as the same object




    ‣Result: dodgy RPD, user cannot edit the SALES table.






    ‣Exactly what the Equalize function was designed for in the AdminTool merge!
    The same object
    definition!
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  39. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + MDS XML RPD - Summary
    •You cannot reliably merge the RPD except through Oracle’s own tools
    ‣MDS XML, whilst plaintext, is still structured
    -Unnecessary conflicts when merging
    -Potential inconsistencies / corruption
    ‣MDS XML uses GUIDs to name files, so source control may

    inadvertently duplicate objects without notification
    -RPD problems may not always be immediately apparent
    •MDS XML merging may work for simple isolated changes to 

    individual objects; as a development strategy this is

    restrictive.
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  40. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + Binary RPD
    •Package up all artefacts in Source Control, providing release and rollback capabilities
    •Use the common git-flow branching method, with development divided up into features
    ‣Provides clear structure and process for managing branches and releases
    ‣RPD stored as binary. Three-way merge with OBIEE tools done where necessary.
    develop
    feature/RM-x
    feature/RM-x
    Original
    Current
    Modified
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD
    ‘Secret Sauce’ :
    merge.bat script

    View Slide

  41. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Source control + Binary RPD
    •Conflict resolution script:
    ‣Uses git-flow to determine 3-way merge candidates
    ‣Attempts automatic merge (comparerpd/patchrpd)
    ‣Launches Administration Tool for user interaction if necessary
    1. Online edit
    2. MUD
    3. Source control + MDS XML RPD
    4. Source control + Binary RPD

    View Slide

  42. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    So which methods sucks the least?
    Multiple
    developers
    Integration
    with source
    control
    Feature
    driven
    development
    Ease of
    use
    Overall
    Source
    control +
    Binary RPD
    Yes Yes Yes Medium Not perfect, but the best option
    MUD Yes Manual No Medium Could be worse
    Source
    control +
    MDS XML
    RPD
    Yes Yes Only if
    objects fully
    isolated
    Medium Requires too much process to
    mitigate against RPD corruption
    and unnecessary merge conflicts
    Unnecessary conflict merges.
    Online
    editing
    Not really No No Easy Not viable

    View Slide

  43. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    The Harsh Reality
    •OBIEE does not natively support concurrent
    development integrated with source control
    ‣And it never will until Oracle re-engineer
    OBIEE and/or the RPD storage format
    •All options require a defined process to be
    followed and associated training for staff
    •There is no perfect software-based solution
    that will force ‘bad’ developers to be
    good
    •You will always need a “Dungeon Master”
    to oversee and track developments, manage
    conflicts, and support developers.

    View Slide

  44. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Checking It All Still Works
    •For concurrent development to be successful, you need automated testing
    Before
    After
    Test Result: Fail

    View Slide

  45. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    Summary
    •If you're not doing source control, go and do it.
    ‣Now.
    ‣Even for serial development
    ‣Even if you’re the only developer
    •If you are doing source control, then you can use it to support
    concurrent development
    ‣But don't go anywhere near concurrent development until you've
    sorted source control
    •The only way to do safe and reliable concurrent development is
    using the binary RPD stored in source control such as git

    View Slide

  46. T : +44 (0) 1273 911 268 (UK) E : [email protected] W : www.rittmanmead.com
    #EOF
    ✴email: [email protected]
    ✴web: http://ritt.md/rmoff
    ✴twitter: @rmoff
    ✴IRC: rmoff / #obihackers / freenode

    View Slide

  47. www.rittmanmead.com [email protected] @rittmanmead www.facebook.com/rittmanmead
    Source Control and
    Concurrent Development for OBIEE
    Robin Moffatt, Principal Consultant
    Rittman Mead
    Riga Dev Day January 2015

    View Slide