DevOps and OBIEE: Do it Before it's too late!

A23789f299ed06fe7d9f1c6940440bfa?s=47 FTisiot
June 11, 2018

DevOps and OBIEE: Do it Before it's too late!

A23789f299ed06fe7d9f1c6940440bfa?s=128

FTisiot

June 11, 2018
Tweet

Transcript

  1. info@rittmanmead.com www.rittmanmead.com @rittmanmead !1 DevOps and OBIEE: Do it Before

    it's Too Late!
  2. info@rittmanmead.com www.rittmanmead.com @rittmanmead !2 Verona, Italy Rittman Mead Blog 10

    Years Experience in BI/Analytics francesco.tisiot@rittmanmead.com @FTisiot Oracle ACE No Cappuccino After 11 Knight Photo by Matt Hoffman on Unsplash Francesco Tisiot BI Tech Lead at Rittman Mead
  3. info@rittmanmead.com www.rittmanmead.com @rittmanmead !3 Photo by Pascal van de Vendel

    on Unsplash Mode 1 Analytics IT Driven Long Cycle Times Centralised
  4. info@rittmanmead.com www.rittmanmead.com @rittmanmead !4 Photo by Kyle Henderson on Unsplash

    Mode 2 Analytics Fast Business Driven Data Discovery
  5. info@rittmanmead.com www.rittmanmead.com @rittmanmead !5 Photo by Jovica Smileski on Unsplash

    Centralised • Unique Source of Truth • KPIs Definition • Development • Test • Release
  6. info@rittmanmead.com www.rittmanmead.com @rittmanmead !6 Photo by Chris Sabor on Unsplash

    TTD Development Isolation Manual Deployment Manual Testing
  7. info@rittmanmead.com www.rittmanmead.com @rittmanmead !7 Photo by chage tang on Unsplash

    Agile DevOps
  8. info@rittmanmead.com www.rittmanmead.com @rittmanmead !8 https://www.01net.it/sviluppare-software-metodo-agile/ Collaboration Frequent Delivery Focus Iteration

  9. info@rittmanmead.com www.rittmanmead.com @rittmanmead !9 DevOps https://www.niceideas.ch/roller2/badtrash/entry/devops-explained

  10. info@rittmanmead.com www.rittmanmead.com @rittmanmead !10 https://www.niceideas.ch/roller2/badtrash/entry/devops-explained Waste Before it’s Too Late?

  11. info@rittmanmead.com www.rittmanmead.com @rittmanmead !11 Before it’s Too Late? https://www.niceideas.ch/roller2/badtrash/entry/devops-explained Risk

  12. info@rittmanmead.com www.rittmanmead.com @rittmanmead !12 Before it’s Too Late! The Results

    are Wrong! Production is Pointing to UAT! We Locked the DB User! Which RPD can we Promote? Photo by R. Mac Wheeler on Unsplash
  13. info@rittmanmead.com www.rittmanmead.com @rittmanmead !13 OBIEE *1996 - ✝2018 Photo by

    Jerry Kiesewetter on Unsplash Before it’s Too Late!
  14. info@rittmanmead.com www.rittmanmead.com @rittmanmead !14 Agenda • Default Situation • Code

    Versioning • Development Isolation • Testing • Promoting • Process Automation
  15. info@rittmanmead.com www.rittmanmead.com @rittmanmead !15 RPD Source Catalog OBIEE Security Config

    Artefacts
  16. info@rittmanmead.com www.rittmanmead.com @rittmanmead !16 The Original OBIEE Sins • No

    Code Versioning • Concurrent Online RPD Editing • Manual Regression Testing • Limited Scripting • Manual Deployment Photo by David Clode on Unsplash
  17. info@rittmanmead.com www.rittmanmead.com @rittmanmead !17 Photo by Erwan Hesry on Unsplash

    DevOps is a Methodology
  18. info@rittmanmead.com www.rittmanmead.com @rittmanmead !18 Photo by Fidel Fernando on Unsplash

    Start Simple - Code Versioning • RPD • Catalog • Security • Config Files • Artefacts
  19. info@rittmanmead.com www.rittmanmead.com @rittmanmead !19 Gitflow Method Master Branch -> Production

    Develop Branch -> Development Features -> New Developments
  20. info@rittmanmead.com www.rittmanmead.com @rittmanmead !20 RPD • RPD Binary • MDS

    XML • How to Merge?
  21. info@rittmanmead.com www.rittmanmead.com @rittmanmead !21 Catalog • Set of XML Files

    • Stored as .catalog? • What about .atr? • How to deploy?
  22. info@rittmanmead.com www.rittmanmead.com @rittmanmead !22 Security • system-jazn-data in 11g •

    bar file in 12c (contains system-jazn-data) • WLST Scripting (infrastructure as code)
  23. info@rittmanmead.com www.rittmanmead.com @rittmanmead !23 Config Files • Automatically promoted? •

    Distinct files per environment
  24. info@rittmanmead.com www.rittmanmead.com @rittmanmead !24 Photo by raquel raclette on Unsplash

    Development Isolation Local Instances VMs Docker
  25. info@rittmanmead.com www.rittmanmead.com @rittmanmead !25 Photo by Matthew Henry on Unsplash

    Original Situation Online RPD/Catalog Editing No History No Change Control No Automated Regression Testing
  26. info@rittmanmead.com www.rittmanmead.com @rittmanmead !26 Photo by Aaron Burden on Unsplash

    Start Simple • Offline RPD • Dev Server for Catalog No test of the RPD Before Merging!
  27. info@rittmanmead.com www.rittmanmead.com @rittmanmead !27 Photo by James & Carol Lee

    on Unsplash And Evolve… • Fully Separated Dev Envs • Content pushed/pulled from Git
  28. info@rittmanmead.com www.rittmanmead.com @rittmanmead !28 Photo by James & Carol Lee

    on Unsplash And Evolve… • Automated Environment Creation • Docker ‣ https:/ /github.com/gianniceresa/docker-images
  29. info@rittmanmead.com www.rittmanmead.com @rittmanmead !29 OBIEE + DB OBIEE F1 OBIEE

    F2 DB F1_BIPLATFORM F2_BIPLATFORM Docker Options
  30. info@rittmanmead.com www.rittmanmead.com @rittmanmead !30 Photo by Timothy Rhyne on Unsplash

    Testing
  31. info@rittmanmead.com www.rittmanmead.com @rittmanmead !31 RPD Consistency Check Validates RPD Logic

    Errors Warnings
  32. info@rittmanmead.com www.rittmanmead.com @rittmanmead !32 RPD Command Line Consistency Check validaterpd

    -R <repositoryName> -O/C/X <outputfile> Text CSV XML
  33. info@rittmanmead.com www.rittmanmead.com @rittmanmead !33 RPD Understanding Differences comparerpd -C <modifiedRPD>

    -G <originalRPD> -O/F <outputfile> Validating Differences
  34. info@rittmanmead.com www.rittmanmead.com @rittmanmead !34 Catalog Consistency Check runcat.cmd -cmd validate

    -items Report -links Report -offline <path_to_catalog> Valid XMLs No Broken Links
  35. info@rittmanmead.com www.rittmanmead.com @rittmanmead !35 Photo by Vincent van Zalinge on

    Unsplash Regression Testing Before After
  36. info@rittmanmead.com www.rittmanmead.com @rittmanmead !36 Photo by Tanner Larson on Unsplash

    Testing Suite • Static • Comprehensive • Security Sensitive Separate Catalog Folder
  37. info@rittmanmead.com www.rittmanmead.com @rittmanmead !37 BVT Baseline Validation Tool • Automated

    Testing • Set of Prebuilt Test ‣ Logic SQL (Discontinued) ‣ Catalog ‣ Reports ‣ Dashboards ‣ UI ‣ BI Publisher ‣ Visual Analyser
  38. info@rittmanmead.com www.rittmanmead.com @rittmanmead !38 BVT Baseline Validation Tool • Deployment

    ‣ Deployment Name ‣ Connections (analytics/bip/va) ‣ User • Test ‣ Set of Tests to run ‣ Catalog Folder • Results ‣ Results Folder textconfig.xml
  39. info@rittmanmead.com www.rittmanmead.com @rittmanmead !39 BVT Baseline Validation Tool • Thresholds

    ‣ UI Comparison
  40. info@rittmanmead.com www.rittmanmead.com @rittmanmead !40 BVT Baseline Validation Tool • Baseline

    • Changes • Compare Results bin\obibvt.bat -config testconfig.xml -deployment PRE bin\obibvt.bat -config testconfig.xml -deployment AFTER bin\obibvt.bat -compareresults Results\PRE Results\AFTER -config testconfig.xml
  41. info@rittmanmead.com www.rittmanmead.com @rittmanmead !41 Photo by Jordan Whitt on Unsplash

    Code Promotion
  42. info@rittmanmead.com www.rittmanmead.com @rittmanmead !42 Promoting • Push ‣ RPD ‣

    Catalog ‣ Security ‣ Configuration ‣ Custom Files/Images
  43. info@rittmanmead.com www.rittmanmead.com @rittmanmead !43 Photo by Rob Schreckhise on Unsplash

    Do you Remember the Password?
  44. info@rittmanmead.com www.rittmanmead.com @rittmanmead !44 RPD • UploadRPD • Patch RPD

    - Change Connection Pools datamodel.cmd uploadrpd -I <file> -D -U <user> -SI ssi -S <server> -N <port> admintool /command <filename> datamodel.sh updateConnectionpool -SI sis -U <user> -P <password> -C <filejson> patchrpd ... -C <curr_rpd> -G <origin_rpd> -I <patch_file> -O <dest_rpd>
  45. info@rittmanmead.com www.rittmanmead.com @rittmanmead !45 Catalog • File System Copy •

    Webservices /shared /system WebCatalogService.pasteItem2(<.catalog_file>,…)
  46. info@rittmanmead.com www.rittmanmead.com @rittmanmead !46 Security • WLST scripts • system-jazn

    wlst.sh <wlst_script.py> Config • File System Copy
  47. info@rittmanmead.com www.rittmanmead.com @rittmanmead !47 Custom Files AnalyticsRes - File System

    Copy
  48. info@rittmanmead.com www.rittmanmead.com @rittmanmead !48 Photo by Armando Tobar on Unsplash

    Sticking Everything Together
  49. info@rittmanmead.com www.rittmanmead.com @rittmanmead !49 Automation Tools Jenkins Gitlab

  50. info@rittmanmead.com www.rittmanmead.com @rittmanmead b !50 Environment Creation New Feature/New Ticket

    in Jira New Git Branch Jenkins Flow Docker OBIEE Image RPD+Catalog Deployment Development Baseline Creation
  51. info@rittmanmead.com www.rittmanmead.com @rittmanmead !51 Finish Feature Finish Feature (Commit to

    work Branch) RPD & Catalog Validation Jenkins Flow Baseline Validation Tool - Against Baseline Pre-Changes Push to Dev Review/Approve Fix
  52. info@rittmanmead.com www.rittmanmead.com @rittmanmead b !52 Merge Into Dev Commit to

    Dev RPD & Catalog Validation Jenkins Flow Baseline Validation Tool Push to Test Review/Approve RPD & Catalog Merge - Manual Intervention when Conflicts Fix
  53. info@rittmanmead.com www.rittmanmead.com @rittmanmead !53 Production Release Commit to Master Jenkins

    Flow Automatic Push to UAT/Production
  54. info@rittmanmead.com www.rittmanmead.com @rittmanmead !54 Rittman Mead BI Developer Toolkit https://www.rittmanmead.com/blog/2017/02/concurrent-rpd-development-with-git/

    • Accelleration Framework • Set of pre-packaged functions ‣ Git versioning ‣ Testing Suite • Written in Python - Easy to Integrate ‣ OBIEE Webservices, command line, file system • Git/SVN Versioning ‣ Binary RPD ‣ .catalog files
  55. info@rittmanmead.com www.rittmanmead.com @rittmanmead !55 Rittman Mead BI Developer Toolkit •

    Creation of Testing Suite - Parameter Modification python modify_reports.py /shared/Test /shared/Test_Suite/2014_2015 "Date Period" Year 2014 2015 • Regression Testing ‣ Comparison of LSQL and Results ‣ Parameter Modification ‣ Push Results to Database ‣ Prebuilt RPD/catalog
  56. info@rittmanmead.com www.rittmanmead.com @rittmanmead !56 Developer Toolkits available for ‣ BI

    ‣ ODI ‣ Informatica ‣ Essbase Apply DevOps to the Whole BI Cycle!