DevOps for Movie and Cartoon Production

DevOps is a well documented topic on the internet with a lot of resources. However they mainly focus on companies selling Software as a Service and very little information may be found addressing more specific domains. In this course we want to address the specificity of cartoon and movie production companies regarding their DevOps needs.
We will show how the scale of the development departments in production companies are too small considering their expected output. That's one of the main reasons production companies cannot work properly without extensive automation in their DevOps framework.
We will explain how the development for host applications (Digital Content Creation tools like Maya, Nuke or Houdini) generates some specific challenges and requires some tailored solutions to be really efficient. We will explain how to build, test and deploy is impacted when developing for those kinds of platforms.
Creating software for in-house customers, especially artists that are always busy creating new animation and arts, requires specific strategies to plan and monitor the required tools. Gathering information, in a backlog, working in an Agile way to adapt to project changes and new projects starts is crucial.
Attendees should expect to get a detailed description of the problems one can face in DevOps in a production company with answers to how to implement a fully working DevOps solution based on common technologies like git, CMake, python and Jenkins.

Transcript

1. DevOps for Movie and Cartoon Production OLM Digital, Inc Marc Salvati Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). SA Courses '24, December 03-06, 2024, Tokyo, Japan ACM 979-8-4007-1135-0/24/12. 10.1145/3680532.3689583
2. Disclaimer • Lots of Dev, little of Ops • Still learning • Cannot cover all topics and technologies • Technologies we talk about ◦ Windows/Linux ◦ Git/Github/Github Actions ◦ CMake ◦ C/C++, Python ◦ VcPkg ◦ Jenkins ◦ Maya, After Effects, Arnold, Nuke, Photoshop
3. What is a movie and cartoon production company? • Produce movies and cartoons ◦ Own IP, contractor ◦ Use Digital Content Creation (DCC) tools ▪ Extends their features using Software Development Kits (SDK) ◦ Multiple movie/cartoon projects in parallel • 90-100% of staff is non IT ◦ Some companies have no IT department ◦ Machine/servers maintenance outsourced ◦ Software engineer in IT companies: 40-60% ◦ Production company : 5-10% • R&D ◦ SaaS companies going IPO 23% ◦ IT computer services 11 % ◦ Movie and cartoon production company up to 5% R&D in Creative Industries Survey 2020 (UK)
4. IT departments in production companies • System maintenance ◦ Servers ◦ Network ◦ Software licences and installation • IT support • Pipeline • Research and Development More Ops than Dev More Dev than Ops
5. Overview • Introduction to DevOps in production • DevOps in production ◦ Dev ◦ Ops • Conclusion Code Plan Build Test Release Deploy Operate Monitor DEV OPS
6. Introduction to DevOps in production
7. What is DevOps Wikipedia: DevOps is a methodology in the software development and IT industry. Used as a set of practices and tools, DevOps integrates and automates the work of software development (Dev) and IT operations (Ops) as a means for improving and shortening the systems development life cycle • Production pipeline ◦ Let artists focus on art • DevOps is the pipeline of IT ◦ Let programmers focus on programming • Pipeline is hard! ◦ Calude, Coull, Lewis, Can we solve the pipeline problem? ACM Digipro 2014
8. Code Plan Build Test Release Deploy Operate Monitor DEV OPS
9. • Dev ◦ Plan: ▪ Study the market: size and target, needs ▪ Estimate cost and possible benefits ▪ Do it if it will make money ▪ Break it down • Define the project goal, roadmap, milestones • Organize the work, tickets, Agile ◦ Code: program with IDE, source versioning control ◦ Build: compile, packaging, multiple OS ◦ Test: unit, integration, automated • Ops ◦ Release: prepare to be used by final users, quality assurance ◦ Deploy: get to the final user, installer, web services update ◦ Operate: customer support, maintenance of servers ◦ Monitor: logging and metrics, crash reports General Definition
10. • Dev ◦ Plan: ▪ Study the market: size and target, needs ▪ Estimate cost and possible benefits ▪ Do it if it will make money ▪ Break it down • Define the project goal, roadmap, milestones • Organize the work, tickets, Agile ◦ Code: program with IDE, source versioning control, DCC tool plugins ◦ Build: compile, packaging, multiple OS, multiple DCC tool versions ◦ Test: unit, integration, automated, test in DCC tools • Ops ◦ Release: prepare to be used by final users, quality assurance ◦ Deploy: get to the final user, installer, web services update ◦ Operate: customer support, maintenance of servers ◦ Monitor: logging and metrics, crash reports CG Software company
11. • Dev ◦ Plan: ▪ Gather requests and frustrations, survey/monitor artists needs ▪ Estimate potential time and quality improvement • Current and future needs ▪ Need to fit the schedule ▪ Break it down • Define the project goal, roadmap, milestones • Organize the work, tickets, Agile ◦ Code: program with IDE, source versioning control, DCC tool plugins ◦ Build: compile, packaging, multiple OS, multiple DCC tool versions ◦ Test: unit, integration, automated, test in DCC tools • Ops ◦ Release: test by developer and selected artists, quality assurance ◦ Deploy: ▪ Installer ▪ Setup DCC tool so plugins may be used (no installer) ▪ Dev, Test, production environment ◦ Operate: maintenance of servers, artist feedback, render farm, DCC licences ◦ Monitor: logging and metrics, crash reports Movie production company
12. A production company / A CG software company • OLM ◦ 328 employees ◦ IT 21 ▪ System: 4, Research: 2, Pipeline: 4 ▪ Development: 9 permanent + 4 interns • Ephere • 8 employees ◦ 4-5 software engineers? • Currently support only Ornatrix Development projects (excluding pipeline) DCC Active Support Inactive DCC Versions OS Maya 3 30 57 4 2 Arnold 3 18 8 4 2 AE 3 25 26 3 1 Nuke 1 6 13 2 2 Photoshop 1 3 - 2 1 Standalone 2 3 - 2 DCC Active Support Inactive DCC Versions OS Maya 1 - 9 3 Max 4 9 1 C4D - 7 2 Unreal Engine 1 2 3 Tools: Active 13, Support 75, Tool versions: 3~4 Targets: Active 67 x 3~4 Support 495 x 3~4 Tools: 1, Tool Version:3~4 Targets: Active + Support 56 x 3~4
13. Conclusion • The business of production companies is NOT software ◦ If the picture right then it's right ◦ Everything can be done without inhouse development • Clients = Artists ◦ Easy access ◦ Easy to lose trust ◦ Low IT literacy • Exotic platforms ◦ DCC tools + SDKs ◦ Plugins and scripts ◦ Specific builds, tests, deploy • Many tools and target platforms ◦ Combination of OS and DCC tools Scale Specific Communication
14. About other approaches • NoOps/AiOps ◦ Release/Deployment: Full automation ◦ Monitor: Use AI to collect and analyse data, send feedback ◦ Operation: separate from the Dev team • DevSecOps ◦ Don't forget about security in the DevOps approach • Platform engineering ◦ Integrate Agile concept into the DevOps implementation • SRE (Site Reliability Engineering): ◦ Implementation of DevOps ◦ More automation? but DevOps is about automation
15. Dev in production
16. Overview • Type of development in production company ◦ Pipeline ◦ Research and Development • Plan • Code • Build • Test
17. Pipeline • Artists daily support • Automation ◦ Within on step • Asset management ◦ One step to the next • Save time for quality • Mostly python scripting 2D 2D Layout Key animation In-betweens 2D Clean-up Compositing 3D Modeling Rigging Animation Lighting Rendering FX Matte Painting
18. Research and Development • Create new possibilities ◦ Visual ◦ Functional • Wide range of development ◦ Standalone applications ◦ Automation scripts ◦ DCC tools plugins ◦ DCC tools scripts ◦ Server based applications ◦ DevOps • C/C++, python
19. Dev: Plan
20. Plan • Organize the tasks ◦ Define ◦ Priority ◦ Track • Software support ◦ Text, spreadsheets (Word, Excel, Google apps…) ▪ Linear representations ◦ Web application (Trello, Redmine, Jira, Github projects…) ▪ Easy to search ▪ Easy to share ▪ Connection with other tools (Github..) ▪ Often uses 2D representations ▪ May be associated to code ▪ More or less customization • Work Agile?
21. Plan in production • Artists cannot write IT specifications • When there is time/money (2-5 years production) ◦ Identify development needs during pre production ◦ Develop and deliver tools before production starts • When there is no time/money (6 month- 1 year production) ◦ Maybe no need right now, but when they need it it's too late ◦ Identify development needs ▪ During production and across productions • Keep communication channels open ▪ Synthesis of needs • Select the one that can help most productions ▪ Keep track in the backlog/global backlog ◦ Develop and deliver to test at a production that may use it
22. Task status • Backlog ◦ Recent request or idea ◦ May move to the global backlogs or pending • Pending: waiting to be assigned to someone • Todo: assigned to someone • Doing: the assigned person is actively working on it ◦ What to do when switched task and not working recently on it? ▪ Granularity? Other status? ▪ Is it that important to not go back? • Done: implemented but not released yet • Released ◦ Artists can use it ◦ Waiting for feedback (may be after project ends) • Archived: only once we got the feedback and created tickets if needed
23. The global backlogs • Existing project ◦ Current issues and idea ◦ To tackle when we have time or need for that project • Non existing project ◦ Conference paper reference ◦ Production problem to tackle ◦ Organized by topic • Participate/survey conferences and journals ◦ Siggraph / Siggraph Asia ◦ Digipro ◦ Eurographics ◦ High Performance Graphics ◦ Computer graphics forum
24. Feedback • In-house client ◦ Feedback loop track • Client is usually not prone to use tracking systems directly • Link between user request docs and implementation specs complicated ◦ Get docs, chat comments… anything ◦ Receive excel sheets, pdfs, slides from various client in various forms ◦ Manually linking
25. Dev: Plan About Agile
26. About Agile: The Agile Manifesto We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. Trust Communication Adapt
27. About Agile: Principles behind the Agile Manifesto Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. The best architectures, requirements, and designs emerge from self-organizing teams. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Business people and developers must work together daily throughout the project. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Trust Communication Adapt
28. About Agile: Principles behind the Agile Manifesto Working software is the primary measure of progress. Simplicity--the art of maximizing the amount of work not done--is essential. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. Continuous attention to technical excellence and good design enhances agility. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Refactor CI/CD
29. Summary of Agile • What is there ◦ Trust, communication, Adapt ◦ Refactor ▪ Code ▪ Organization ◦ Automation to focus on work ⇦ DevOps needed ▪ Continuous delivery • What is not there ◦ Scrum, Kanban, Scrum master, product owner… ◦ Metrics, granularity, ticket speed, daily meeting… • Going Agile or not, DevOps is needed! • Agile for devops in production ◦ Because lots of issues arise during production ◦ Done when done ▪ No deadlines ◦ Regular meeting and roadmaps ◦ Constant update of time estimation Depend on your Agile implementation Not in Agile itself
30. Scrum and Kanban in IT • Like processes ◦ Role and responsibilities ◦ Process and tools ▪ Sprint ▪ Meeting ▪ Tables ◦ Stories and point, evaluation ▪ Granularity of tasks ▪ Ticket flow optimization ▪ Dangerous link to evaluation (quantity vs quality) • Kanban comes from car industry ◦ Repetitive tasks, stock management • The differences between Agile principles and its implementation… ◦ There is no scrum master in agile manifesto ▪ Scrum is like a war room situation ◦ Agile 11th principle: The best architectures, requirements, and designs emerge from self-organizing teams.
31. Dev: Code
32. Code • Programming rules ◦ Naming conventions ◦ Coding conventions for each programming language • Enforces programming rules ◦ Code reviews ◦ Automatic check • Efficient coding ◦ Use IDE/debugger/profilers • Source versioning control ◦ About Git • Semantic versioning
33. Programming rules • Important to have a rule, which rule is less important • Naming conventions ◦ PascalCase, camelCase, snake_case, kebab-case… ▪ Per programming word type (class, variable…) ◦ Be consistent • Coding conventions ◦ Which feature you use or not, how to use them ▪ Ex: exceptions or not? std::expected? Smart pointer? Branchless programming? C++ modules? ◦ Code layout for readability ▪ Logical structure, comments • Specific to each programming language • May vary per project • Examples ◦ Google C++ Style Guide (no exceptions) ◦ C++ Core Guidelines (use exceptions) ◦ Python PEP 8 (standard)
34. Naming conventions in production • Each DCC tool SDK has its own conventions ◦ Maya C++ class method = camelCase ◦ Nuke C++ class method = mix of camelCase and snake_case ◦ After Effects C functions = PascalCase • Two choices ◦ Adapt to the convention of each DCC tool ▪ What about your libraries? ▪ Many rules to follow ▪ Not always well defined (Nuke) ◦ Use a unique convention ▪ Simple ▪ Easy to spot your code from the SDKs code ▪ Same convention for libraries
35. Coding conventions in production • May vary with the DCC tool ◦ DCC tool have a recommended compiler ▪ May limit the features you can use, ex: need vc2015 for c++11, ▪ Reference for C++ standard ◦ AE mainly C, use error code, own memory management system ▪ Would need a wrapping to use exceptions ▪ Need custom allocator to use C++ STL ◦ Maya use C++, use error code, no exceptions ◦ Arnold use C • Need a base coding conventions ◦ Use standard one ▪ C++: C++ Core Guidelines ▪ Python: PEP8 ◦ Make your own for each DCC tool • Create new conventions for specific languages ◦ DCC tool have specific language embedded ▪ Maya MEL and expressions, Nuke Blink Scripts, Houdini VEX ◦ Exotic languages: OSL, GLSL, TCL
36. Enforcing rules • C/C++/Java/Javascript: Clang Format, Clang Tidy • Python: pep8, autopep8 • Automate process ◦ At push to Version Control Software (VCS) ◦ At build time • Code reviews • Hard to automate anything on exotic languages ◦ Maya Mel, Houdini VEX…
37. • Code review = taking time to look at someone else code and talk about it ◦ Encourage discussion about code and algorithms ◦ Improve quality of code ◦ Help to find bug, optimization, refactoring possibilities ◦ Helps to enforce coding rules • Pull requests ◦ Provided by cloud git hosts ◦ Pros ▪ Systematic code review, someone checks! ▪ Organize push in bunch ◦ Cons ▪ Wait before the merge • Need someone able to check the pull request ◦ Easy if pool of many specialized people available ◦ Easy if many people work on the same part of the code ▪ But then they already saw the code? ◦ Hard if you need to get into the code, study it before validation… • Change in the meantime, complicates the merges ▪ Systematic code review may lower its quality? Code reviews
38. Code reviews in production • Use pull requests? ◦ Often one developer on one project ◦ Not enough staff ◦ Drawbacks may outweigh the benefits • Code reviews ◦ A must • Make people work overlap ◦ Need of two pair of eyes ◦ Alternate the programmer on each tool • Trunk based development ◦ Frequent push ◦ Continuous integration ▪ Automated compilation ▪ Automated tests Works well together Works well together
39. Dev: Code Coding tools
40. Efficient coding tools • IDE ◦ Visual studio/ Visual studio code ◦ Eclipse, Qt Creator… ◦ XCode • Debugger ◦ Integrated to the IDE ◦ GDB • Profiler ◦ VTune (Windows/Linux) ◦ Valgrind (Linux) ◦ Instruments (Mac)
41. Efficient coding tools in production • Plugins are programs executed in host applications ◦ Applications we don't have the source code for • Debugger ◦ Attach the debugger to the DCC tool/host application process • Profiler ◦ DCC tools are not instrumented, hard to profile ◦ Some DCC tool have their own profiler ▪ Maya profiler, DG profiler ▪ AE render time evaluation ▪ Nuke profile node ▪ Houdini performance monitor
42. Getting answers when stuck • General c++ ◦ C++ reference ◦ Google search>>stack overflow ◦ ChatGPT, AI? Not used yet… • Production ◦ Dedicated forum ▪ Some need an account to search (free) ◦ After Effects forum ◦ Arnold ◦ Houdini ◦ Maya ◦ Nuke
43. Dev: Code Source versioning
44. Version Control Software (VCS) • Also called Source Control Management (SCM) • Features ◦ Keep track of every change to code ◦ Rollback changes ◦ Merge changes ▪ Multiple programmer programming in parallel ◦ Centralized or decentralized • Examples ◦ Git (de facto standard Market share 89%) ◦ Subversion ◦ Mercurial • Can be hosted in the cloud ◦ Offers Continuous Integration solution ◦ Examples ▪ Github ▪ Gitlab
45. About Git • Need to have rules • Git configuration ◦ Merge or rebase ◦ LFS handling ▪ What file to upload ◦ Git ignore • Git flow VS Trunk based development ◦ Git flow ▪ Good for open source software ◦ Trunk based development ▪ Simpler ◦ In the reality use a mix of branches ▪ Main/Develop ▪ Feature ▪ Hotfixes ▪ Release
46. Trunk based development • Commit often ◦ Atomic commits ◦ Tag+branch for release • Good for senior developers • Good with CI Main v1.0.x 1.0.0 1.0.1 v1.1.x 1.1.0 1.1.1 Head Merge Bug Fix Bug Fix
47. Git flow • Develop in develop and tag in main • Feature branches ◦ When need to test some big changes ◦ Isolate feature influence on next release • Release branches ◦ When close to release state • Hotfix branches ◦ When problems occurs in the release Main Head Develop
48. • Isolate some development from the main branch • Merge when ready • Use pull requests to merge • Good for large scale change • Good for experimenting new designs Feature branch Main/Develop Feature 1 Head Merge Feature 2
49. Release branch • Buffer between main and develop • Develop can still go on with new features Main Head Trunk based Merge Main Head Develop Release Git flow
50. Hot fixes branch • Problem in a release, fix it and merge in develop and release • Develop can still go on with new features Main Head Develop Hotfix Main Head Git flow Trunk based Merge
51. Git branching in production • Hotfixes branch ◦ Merging a bugfix from an older version is usually hard ◦ Need specific port usually ◦ Hotfixes branches may not make it • Release branch ◦ One branch for one released is practical ◦ Hotfixes directly in those branches • Feature branches ◦ May be used when multiple developers on the same project • Separation of main and develop ◦ A matter of preference ◦ Personally i prefer the trunk based approach ▪ Different version of the tools used at the same time • Need to patch the released version often ◦ Change version in the middle of a production may change output • One branch per release, hotfix directly in the release branch ▪ Branch naming convention is simple
52. Dev: Code Tool versioning
53. Tool version names • Many patterns ◦ Numbers separated by dots: 1.2.3 ◦ Add - for specific version ▪ -a = alpha ▪ -b = beta ▪ -rc = release candidate ◦ Number order: 8<9<80<90 or 8<81<9<92 • Internal version number VS release number ◦ Internal version number may not be marketable • Exotic pattern ◦ TeX new version adds a new Pi number (current version 3.141592653) ◦ Odd number = development, even = release
54. Tool version names • Some standardization ◦ Dot separated numbers recommended • Semantic versioning ◦ Makes a lot of sense for libraries, less for softwares ◦ Most software companies have their own schemes and rules 2 1 3 . . API non compatible change Backward compatible feature Backward compatible bug fixes
55. Tool versioning in production • DCC tool and tool version may be set for 3-5 years ◦ High budget production ◦ TV series lasting for more than 2 seasons • What the artists want to know about the new version of a tool? ◦ Can i open my files? ◦ Is the output picture the same?
56. Semantic versioning for production 2 1 3 2 . . . Data (API) non Compatible change Output Change Data compatible Output stable feature/bug fixes Production specific Hotfixes (usually data compatible and output stable ) Error when opening file Rendering changes Anything else Anything may happened! Production surprise
57. Version 1.0.0.1 The patch number Project A Tool A Version 1 Version 2 Version 2.0.1 Start with Tool A version 1 Find bug with crash Correction in version 2.0.1 Encounter crash Impossible to upgrade to version 2 Create patch version 1.0.0.1 Time Projects end Throw away the patch
58. Dev: Build
59. Build definition Code Internal Library Code External Library Code Internal Library Binary External Library Binary Compiled Code Executable Code Compile Link Plan Test Install …
60. Build overview • We focus on C/C++ ◦ Mainly used in production • Build ◦ Compile ◦ Linking ◦ About API and ABI • Build system ◦ Build + install + test +... ◦ May be generated • Dependencies ◦ Diamond ◦ ABI problem • Automation and continuous integration
61. Build • Compile and link ◦ Windows: Visual c++, MingGW ◦ Linux: gcc ◦ Mac: clang • Many flags ◦ Optimization level ◦ Warning level ◦ Features ▪ C++ language standard (11, 14, 17, 20) ▪ SSE, AVX ▪ OpenMP • Libraries ◦ Shared or static ◦ C++ runtime/standard library ▪ Usually mixing static and dynamic runtime is not a good idea
62. Build in production • Compile and link ◦ All plugins are shared libraries ◦ Compiler specified by the DCC tool ◦ C++ runtime libraries ▪ Dynamic runtime ▪ Reuse the c++ dynamic runtime the DCC tool already installed • Flags ◦ Some are specific to the DCC tool SDK ◦ Features ▪ C++ language standard (11, 14, 17, 20) • Minimum requirement depending on the SDK ▪ OpenMP • Maya ◦ Need to use syncNumOpenMPThreads if using non intel openmp ◦ Maya recommends to use Tbb instead • After effects has its own parallelization functions but can use OpenMP anyways
63. Dev: Build ABI and API
64. About ABI and API breaks • API: Application Programming Interface ◦ Write a program using an API ◦ API stable >> source code portable ▪ can be compiled against new version without change ◦ Non stable API >> has to change the code >> API break • ABI: Application Binary Interface ◦ Data layout, call conventions… ◦ Write a program using an API ◦ Compile to machine code ◦ Library ABI = compiler ABI + library API and data layout ▪ Non stable API >> Non Stable ABI >> machine code is not portable ▪ Stable API + Stable ABI >> machine code is portable ▪ Stable API + Non stable ABI >> machine code is not portable • Break = break compatibility
65. API Break header.h #include “header.h” int main() { void fly(10.0f); return 0; } mylib Version 1 library void fly(float speed); mylib Version 2 library void fly(float speed, float altitude); myprogram.c Execution environment Compiled with Version 1 myprogram.exe myprogam Compiled with Version 2 mylib.dll mylib.so
66. ABI Break • C/C++ has the One Definition Rule ◦ If two entity with same name but different definition >> Undefined behavior Execution environment Compiled with Version 1 myprogram.exe myprogam Compiled with Version 2 mylib.dll mylib.so mylib Version 1 library struct MyStruct{ int a; int b; }; mylib Version 2 library struct MyStruct{ int a; int b; Int c; }; header.h #include “header.h” int main() { MyStruct s; s.a = 1; s.b = 2; return 0; } myprogram.c
67. How to break ABI • Easy to break – Data layout change – Class structure change – Change method to virtual – … • References – KDE、Oracle、WG21
68. API and ABI Break in production: The SDKs • Source code uses DCC tool SDKs ◦ C/C++ Headers (Maya, 3DS Max, Nuke, Houdini, Arnold, AE, Photoshop) ◦ Libraries ◦ 3rd party libraries binaries ◦ Documentation ▪ API ▪ Build instructions • Plugins compatibility ◦ Usually compatible within a major version ▪ Maya, Nuke, Arnold, Nuke ▪ ABI and API stable ▪ Read the docs ◦ Rarely compatible across major versions ▪ After effects • Depends on the feature set used • API is based on suites and opaque types7
69. • C++ runtime /standard library break • Plugins are executed on the client machine • Plugins uses C/C++ >> C++ runtime library • Windows: used to recompile until 2015 ◦ Compatible dll VC2015 ▪ Can mix VC++ versions libraries if link with most recent one ▪ Exceptions for /GL and /LTCG (optimization options) • Linux: libstdc++.so ◦ About C++ ABI and the C++11 ABI break ◦ Shipped with two version of the ABI ▪ Switch using _GLIBCXX_USE_CXX11_ABI=0 or 1 API and ABI Break in production: C++ runtime VC++2017 error Error C2338… Before VS 2017 15.8, the member type would non-conformingly have an alignment of only alignof(max_align_t). VS 2017 15.8 was fixed to handle this correctly, but the fix inherently changes layout and breaks binary compatibility…
70. Libstdc++ ABI version • C++11 String changed with GCC 5.1(2015) ◦ Performance reasons (Copy on write) ◦ ABI break ◦ Can check ABI ▪ objdump -T lib.so |grep cxx11 ▪ nm-D lib.so |grep cxx11 • VFX Reference platform ◦ Switched in 2023 ◦ Maya 2024 ◦ Nuke 15 ◦ Build with and without new ABI ▪ We still use Maya 2023 ▪ Arnold does not depend on C++ • But MtoA does and load Arnold shaders ▪ How to do it >> more details in the dependency section
71. Libstdc++ ABI version • You have to use the right compiler • Each DCC tools recommends a compiler • Maya 2024 linux: DTS-11 + GCC 11.2.1 ◦ Not GCC 11.2.1! DTS-11 (Development toolset) • Maya 2024 recommended linux ◦ Rocky Linux 8.7>> gcc 8.5 >> libstdc++ ABI GLIBCXX_3.4.25 ◦ GCC 11.2.1 >> libstdc++ ABI GLIBCXX_3.4.29 ▪ Can get version `GLIBCXX_3.4.26' not found error message ◦ DTS-11 ▪ Link to system library dynamically ▪ Link statically for newer features ▪ Uses GCC11 but libstdc++ ABI GLIBCXX_3.4.25 ▪ DTS 6-7 ignore the ABI maccro
72. • Juts recompile everything ◦ Most DCCs require plugins build for each SDK major release ▪ Recompile against each SDK version ▪ Use recommended compiler version ◦ Two flavors on linux ▪ _GLIBCXX_USE_CXX11_ABI=1 ▪ _GLIBCXX_USE_CXX11_ABI=0 ▪ Even for arnold plugins • Loaded through MtoA • ◦ OS major update may mean need to recompile • Third party libraries ◦ Compile from sources ◦ Need to propagate the ABI flavor recursively to dependencies ▪ Use the right flavor during the build • Don't expect ABI stability in general ◦ Glibc had abi break ◦ Deprecated attribute since c++14 Conclusion about API and ABI Break in production
73. Dev: Build About CMake
74. Build system • Manually create build files ◦ Make files ◦ Visual studio ◦ Ninja • Generate the build system ◦ Scons ◦ Cmake ◦ Cross platform • CMake is a defacto standard ◦ Used to generate Ninja, VC++… ◦ Has to be Modern CMake Code Internal Library Code External Library Code Internal Library Binary External Library Binary Compiled Code Executable Code Compile Link Build System How to compile How to Link
75. Build definition with CMake Code Internal Library Code External Library Code Internal Library Binary External Library Binary Compiled Code Executable Code Compile Link … add_executable(application1) target_sources(application1 PRIVATE src.cpp src.h) target_compile_features(application1 PRIVATE cxx_std_14) find_package(library1) target_link_libraries(application1 PRIVATE library1)
76. Modern CMake • So many ways to do Cmake ◦ From cryptic one to modern cmake • Modern CMake is about Best Practices ◦ Use subset of CMake the right way ▪ Mainly target based function ◦ Use find_package for dependencies ◦ De-facto standard • Cmake documentation is great ◦ Everything is there! Read carefully! ◦ Reference ◦ Mastering CMake • Lots of references ◦ Modern CMake for Modular Design (Cpp con, 2017) ◦ Effective CMake (C++ now, 2017) ◦ More Modern CMake (Meeting C++, 2018) ◦ Embracing Modern CMake (NDC 2019) ◦ Oh No! More Modern CMake (Meeting C++, 2019) ◦ Modern CMake Modules (CppCon, 2021) ◦ An Introduction to Modern Cmake (2023) • Can generate the Visual Studio files for the developer ◦ Guarantees binary consistency
77. Modern CMake in production • Most of third party libraries have CMake support ◦ OpenCV, Qhull, Eigen… • Each DCC use specific compiler ◦ Use Generator ◦ Setting specific GCC is not simple when multiple GCC installed! ▪ Prepend the PATH ▪ Use scl • DCC support of cmake is variable ◦ Maya ▪ Cmake but not modern ▪ Not defined as targets ▪ Impossible to use the Qt cmake out of the box ◦ Nuke ▪ Modern CMake with add_nuke_plugin wrapping ◦ Houdini ▪ Modern CMake ◦ Arnold, After Effects, Photoshop ▪ No cmake only Visual Studio file template • Need CMake for Maya, AE, Arnold, Photoshop
78. Dev: Build Dependencies
79. Dependencies • About libraries • Resolution • Diamond problem • With Cmake • With Package managers • ABI issues • In production ◦ About SDKs
80. About libraries • Why use libraries ◦ Avoid to reinvent the wheel ◦ Needed to use API • Type of libraries ◦ Dynamic or static ◦ With or without source code
81. Dynamic/shared libraries • Pro ◦ Can be updated ▪ If backward compatible ▪ Enable to correct bug, security issues ◦ Enable plugin system ▪ All DCC plugins are dynamic libraries • Cons ◦ Update may cause problems ▪ Change output ▪ Introduce bugs ◦ May be updated without your knowledge ▪ OS update ▪ Software install ◦ More ABI problem ◦ Need to deploy
82. Static libraries • Pro ◦ No need to deploy ◦ Reduce conflicts for internal libraries ◦ Reduce conflicts for third party libraries ▪ May use third party library in their static fashion ◦ Historical reasons from DLL hell on windows • Cons ◦ Usually not distributed so need to compile yourself ◦ Large size binaries ▪ Using Opencv_world from OpenCV in static mode results in huge binaries
83. Execution environment The dependencies resolution Code Library Binary Compiled Code Executable Code Library Header … Compile Required libraries Description Dependency resolution Dynamic Library Binary Library Binary Link
84. The dependencies resolution Compile Required libraries Description Required libraries Description Found binary? Found source? Library Binary Yes No Yes Library Code No Error Required libraries Description Dependency resolution Provides Execution environment Code Library Binary Compiled Code Executable Code Library Header … Compile Dynamic Library Binary Library Binary Link
85. • Two different paths of dependencies • Same library • Same version • In CMake ◦ config/module should make it ok ◦ Library target defined only once ◦ Safeguard with target existence check ▪ Example in OpenCV config The diamond dependency problem: simple version Code Library A v1 Library B v1 Library C v1 Uses Uses Uses Uses if(NOT TARGET opencv_core) include(${CMAKE_CURRENT_LIST_DIR} /OpenCVModules ${OpenCV_MODULES_SUFFIX} .cmake) endif()
86. • Two different paths of dependencies • Same library • Different versions • In Cmake ◦ First call to find_package finds a compatible library ▪ Use the config version file ◦ Second call usually not change the targets ▪ May depends on the config and version cmake • Just DON'T ◦ ABI issues ▪ Get an object from Library C through Library A ▪ Use the object with library B ▪ Maybe result in undefined behavior ◦ Find a common version compatible with all The diamond dependency problem: complex version Code Library A v1 Library B v1 Library C v1 Uses Uses Uses Uses Library C v2
87. The dependencies resolution in CMake Compile Required libraries Description Required libraries Description Found binary? Found source? Library Binary Yes No Yes Library Code No Error Required libraries Description Dependency resolution Provides Execution environment Code Library Binary Compiled Code Executable Code Library Header … Compile Dynamic Library Binary Library Binary Link FetchContent find_package
88. CMake find_package: how package are found? • In the command line ◦ Add each library CMake to the CMAKE_PREFIX_PATH and CMAKE_MODULE_PATH ◦ Very long and tedious to type ▪ Duplicate dependencies/ Need to know all of them ▪ Ok for one project, not for automation ▪ Rather use a package manager • Module mode ◦ In the CMAKE_MODULE_PATH ◦ For libraries that don’t know CMake ◦ For libraries that don’t have CMake config files ◦ Usually written by the consumer of the library ◦ Pre defined cmake modules ▪ System ones and often installed ones ▪ OpenMP, OpenGL, Git… • Config mode ◦ In the CMAKE_PREFIX_PATH ◦ Recommended one ◦ Provided by the library authors
89. CMake find_package: how package are found? • Forward call to dependency provider ◦ Can be a package manager ▪ Vcpkg, Conan ▪ Use chain "CMAKE_TOOLCHAIN_FILE" ▪ Use CMake presets to make it simple ◦ A custom dependency provider • May use a CMake Package Registry ◦ Local repository of libraries ◦ Setup is OS dependent
90. CMake FetchContent • Download a CMake project • Compile the project as if it was part of the sources • All project may not be FetchContent compatible ◦ Same behavior if installed or built from source • Download option if dependency not found through find_package ◦ Added in the project find_package(googletest NAMES GTest) ) include(FetchContent) FetchContent_Declare( googletest GIT_REPOSITORY https://github.com/google/googletest.git GIT_TAG 703bd9caab50b139428cea1aaff9974ebee5742e # release-1.10.0 FIND_PACKAGE_ARGS NAMES GTest ) FetchContent_MakeAvailable(googletest) add_executable(ThingUnitTest thing_ut.cpp) target_link_libraries(ThingUnitTest GTest::gtest_main)
91. • Third party libraries should provide CMake ◦ If installed ▪ Add to CMAKE_PREFIX_PATH ▪ Use find_package ◦ Not installed ▪ Use FetchContent • Bring in the project and build from sources • Equivalent to add_subdirectory • Internal libraries ◦ Process as third party ▪ Create config package, Install, add to CMAKE_PREFIX_PATH ▪ Use find_package ◦ Build from source ▪ Use FetchContent to download and include ▪ Include in the project directly with add_subdirectory • Need to be in the directory tree of the main CMakeList.txt • Ideally want to only consume libraries through find_package!! CMake dependency resolution in practice
92. How to fallback to FetchContent • Add download if necessary ◦ Just wrap FetchContent • Create a FindXXX.cmake for each library ◦ Just call the wrapping function • Recompile from source function (get_lib_from_github git_repo_name lib_name lib_version source_lib) set(git_url "https://github.com/${git_repo_name}.git ") #may need to add some authentication to the URL #set(git_url "https://x-access-token:${GITHUB_TOKEN}@github.com/ ${git_repo_name} .git") set(search_dir ${lib_name} /${lib_version} ) find_package (Git) if(IS_DIRECTORY "${search_dir}" ) #repo already downloaded once execute_process (COMMAND ${GIT_EXECUTABLE} pull ${git_url} ${lib_version} WORKING_DIRECTORY ${search_dir} ) set (${lib_name} _FOUND ON PARENT_SCOPE) set (${lib_name} _SEARCH_DIR ${search_dir} PARENT_SCOPE) if (source_lib) add_subdirectory (${search_dir} ${lib_name} _bin) endif () else () #try to download from git if (NOT TARGET ${lib_name} ) include (FetchContent) FetchContent_Declare( ${lib_name} GIT_REPOSITORY ${git_url} GIT_TAG ${lib_version} #OVERRIDE_FIND_PACKAGE SOURCE_DIR ${search_dir} ) FetchContent_MakeAvailable( ${lib_name} ) set (${lib_name} _FOUND ON PARENT_SCOPE) set (${lib_name} _SEARCH_DIR ${search_dir} PARENT_SCOPE) endif () endif () endfunction () Function to fetch from github
93. Simpler dependencies using package managers • Vcpkg ◦ Compile from source ◦ Binary cache ◦ Concept of baseline and triplet ▪ Sure to always have the right library version? ▪ Triplets are actually not triplets ◦ Port concept may be complicated ◦ Has a cmake toolchain ◦ Need to duplicate dependencies in the manifest vcpkg.json if want specific version ▪ Use override to get specific version • Conan ◦ May compile from source or download pre-build package ◦ Duplicate dependencies in conanfile.txt or conanfile.py ◦ Custom libraries/cache using artifactory ◦ Has a cmake toolchain • CPM ◦ Setup free ◦ Specific instruction CPMAddPackage • CMake Package Registry ◦ Just a binary cache to avoid recompilation ◦ Need to call export(PACKAGE) from the project ▪ Does not work from external libraries who don't support it
94. Simpler dependencies using package managers Compile Required libraries Description Required libraries Description Download binary? Download source? Library Binary Yes No Yes Library Code No Error Required libraries Description Dependency resolution Provides Conan In the cache? Yes No Execution environment Code Library Binary Compiled Code Executable Code Library Header … Compile Dynamic Library Binary Library Binary Link
95. Simpler dependencies using package managers Compile Required libraries Description Required libraries Description In the cache? Download source? Library Binary Yes No Yes Library Code No Error Required libraries Description Dependency resolution Provides Vcpkg Execution environment Code Library Binary Compiled Code Executable Code Library Header … Compile Dynamic Library Binary Library Binary Link
96. Simpler dependencies using package managers Compile Required libraries Description Source in the cache? Download source? Yes No Yes Library Code No Error Required libraries Description Dependency resolution CPM Execution environment Code Library Binary Compiled Code Executable Code Library Header … Compile Dynamic Library Binary Library Binary Link
97. Using package managers • Plus ◦ Easy to get library when change in library version ◦ Easy to get library when change in compiler version ◦ Use only find_package to get dependencies ▪ CPM has specific instructions >> X ▪ Wrap around FetchContent • Minus ◦ Requires setup (Vcpkg, Conan) ▪ Any kind of automation would requires some setup ◦ Requires a lot of compilation ▪ Takes time ▪ On each user machine ▪ Using cache may help (Conan, Vcpkg, CMake) • CMake Package Registry requires project to use export(PACKAGE) • Complicated to share cache updates across machines ◦ Dependencies duplication ▪ Vcpkg ▪ Conan
98. The simplest package managing • External libraries manually compiled and stored in a local repo folder ◦ May share archive among developer ◦ May be on the company file servers ◦ Common folder structure or specific environment variable name pointing to the repo • Internal libraries have their own git repository ◦ Use FetchContent wrap to download and compile • CMakes utilities modules on a repo in the cloud ◦ FindXXX.cmake files for each library ◦ Each file just point to the directory in the repository ◦ Can include utilities modules to manage ABI, FetchContent wraps… • Add only the module folder to CMAKE_MODULE_PATH if (WIN32) set(OpenCV_DIR "${Path_to_OpenCV_Windows}")#windows, version else() set(OpenCV_DIR "${Path_to_OpenCV_Linux}" endif() find_package(OpenCV ${OpenCV_FIND_VERSION} CONFIG REQUIRED)#because OpenCv has config files Ex: FindOpenCV.cmake
99. Package managers and ABI • Compile from source will use the global flag _GLIBCXX_USE_CXX11_ABI • How to retrieve the correct ABI library? ◦ Conan manages it in the profile? Just setting the compile? ▪ Compiler.libcxx ◦ Not sure how vcpkg manages it neither ▪ Can compile, but how to retrieve ◦ Find_package has no parameter to retrieve a library with a specific ABI • Not sure • Maybe use package manager in the future…
100. • Dependencies requirements are transitively applied by CMake B requires c++11 OpenMP Managing dependencies and ABI Library A Library B Program Uses Uses Union of A, B, C requirements C++17 OpenGL OpenMP A requires c++14 A requires from B OpenMP C requires C++17 OpenGL Library C Uses
101. • ABI has to be transmitted to dependencies Managing dependencies and ABI Library A Library B Program Uses with new ABI Requires New ABI Library C Uses with new ABI Uses with new ABI
102. How to get transitive new/old ABI at compile time? • Define in the cmake command line! ◦ New ABI: -D CMAKE_CXX_FLAGS='-D_GLIBCXX_USE_CXX11_ABI=1' ◦ Old ABI: -D CMAKE_CXX_FLAGS='-D_GLIBCXX_USE_CXX11_ABI=0' ◦ Compilation from source will be ok • Package managers support ◦ Conan supports it ▪ Tools.gnu:define_libcxx11_abi ▪ Is it transitive to dependencies? hopefully ◦ Vcpkg compiles from sources ▪ Need to define _GLIBCXX_USE_CXX11_ABI in VCPKG_CXX_FLAGS ▪ Compile from sources, so should be transitive? • However Cmake find_package does not support it ▪ Need to ensure that find_package get the right pre-compiled dependencies!
103. Managing dependencies and ABI • How to make find_package retrieve the correct ABI version? ◦ Create/add to a module file ◦ Add a test to the ABI flags, set the search directory accordingly • Required for each library you use ◦ Works well in conjunction with the simple package manager if (WIN32) set(OpenCV_DIR "${Path_to_OpenCV_Windows}" )#windows, version else() list(FIND CMAKE_CXX_FLAGS "-D_GLIBCXX_USE_CXX11_ABI=1" index_flags) if(index_flags GREATER_EQUAL 0) set(OpenCV_DIR "${Path_to_OpenCV_Linux_new_ABI}" else() set(OpenCV_DIR "${Path_to_OpenCV_Linux_old_ABI}" endif() endif() find_package (OpenCV ${OpenCV_FIND_VERSION} CONFIG REQUIRED) #because OpenCv has config files Ex: FindOpenCV.cmake Config mode avoids recursion Find the ABI global flag Point to the right folder according to the ABI flag
104. Dependencies in production • All plugins are shared libraries ◦ Shared libraries: need to deploy the dependencies with the plugin ◦ Static libraries: easier deployment • Need library exact version ◦ SDKs are usually not compatible between versions ◦ Guarantee image output ▪ Semantic versioning does not take that into account ◦ Use find_package(<PackageName> [version] EXACT) ▪ Exact version is never guaranteed ▪ Depends on the Module/Config processing of the parameters… • How to handle DCC SDKs? ◦ Package manager don't have DCC tool SDKs ▪ Need to setup ◦ Need a system to manage the SDKs
105. Dependencies in production: shared libraries • Symbol conflicts ◦ Different plugins, different version of the same library ▪ Useful third party library like boost, opencv… ▪ Internal libraries • Plugins may have common dependencies with the host application ◦ Maya lists ▪ Boost, OpenimageIO, Ffmpeg, Qt… ◦ Houdini lists ▪ Boost, libtiff, zlib… ◦ Nuke list ▪ Jpeg, libpng ◦ May result in diamond dependencies • Vcpkg build shared libraries by default on Windows
106. How to manage SDKs • SDKs are private ◦ Cannot be put in public repositories in the cloud ▪ Licences issues ▪ Private repositories are ok ◦ Cannot be downloaded directly from the web ▪ Authentication process needed • SDK may be huge ◦ Maya SDK > 600Mb ▪ X 8 (Windows, Linux, 3-4 Maya versions) = >4Gb • Need to share across projects ◦ Locally: machine of the programmer ▪ Download an archive for developer to work locally ▪ Download a repo from the cloud ▪ May use LFS for binaries ▪ Download limits (On Github: 2Gb, may use datapack) ▪ File size limits (On Github: 100 Mb) ▪ Repo size limits (On Github: around 5Gb?) ◦ Remotely: use from file servers ▪ Had issues on windows • Too slow when using library files stored on linux servers • Had to make a local copy…
107. How to manage SDKs with CMake • Modern CMake with config file ◦ Nuke ◦ Houdini • Cmake but no config file ◦ Maya • No Cmake ◦ Arnold, After Effects, Photoshop • Specific compilation steps ◦ After effects and Photoshop uses PiPL resources files ◦ Qt Moc files for Maya ▪ Maya Qt is not useable out of the box • Need FindXXX.cmake modules for the SDKs ◦ Enable find_package use • Need to wrap compilation specifics
108. • No lib, only headers • Some common source files Example: After Effects set(AESDK_SEARCH_DIR ${EXT_LIB_FOLDER} /AESDK/Windows/ ${AESDK_FIND_VERSION} ) if (EXISTS "${AESDK_SEARCH_DIR}" ) message ("-- Found AESDK ${AESDK_FIND_VERSION}: ${AESDK_SEARCH_DIR}" ) set (AESDK_FOUND ON) if(NOT TARGET AESDK::headers ) add_library (AESDK::headers INTERFACE IMPORTED) target_include_directories (AESDK::headers INTERFACE "${AESDK_SEARCH_DIR}/Headers" "${AESDK_SEARCH_DIR}/Headers/SP" "${AESDK_SEARCH_DIR}/Headers/adobesdk" "${AESDK_SEARCH_DIR}/Util" ) target_compile_definitions (AESDK::headers INTERFACE _WINDOWS _WINDLL MSWindows) endif () if(NOT TARGET AESDK::source_utils) add_library (AESDK::source_utils INTERFACE IMPORTED) target_sources (AESDK::source_utils INTERFACE "${AESDK_SEARCH_DIR}/Util/AEGP_SuiteHandler.cpp" "${AESDK_SEARCH_DIR}/Util/MissingSuiteError.cpp" "${AESDK_SEARCH_DIR}/Util/Smart_Utils.cpp" "${AESDK_SEARCH_DIR}/Util/AEFX_SuiteHelper.c" ) target_include_directories (AESDK::source_utils INTERFACE "${AESDK_SEARCH_DIR}/Util" ) endif () else () message ("-- AESDK ${AESDK_FIND_VERSION} not found: ${AESDK_SEARCH_DIR}" ) set (AESDK_FOUND OFF) endif() FindAESDK.cmake Search in the lib repo Guard to avoid double inclusion Guard to avoid double inclusion Headers Common source code
109. Example: After Effects cmake_minimum_required (VERSION 3.20) project (MyPlugin LANGUAGES C CXX) add_library (aex_plugin SHARED) add_subdirectory (src) setup_pipl(aex_plugin src/pipl_file.r ${AE_VERSION} ) set_target_properties ( ${target_name} PROPERTIES SUFFIX .aex OUTPUT_NAME ${CMAKE_PROJECT_NAME} ) find_package (AESDK ${AESDK_VERSION} EXACT REQUIRED) target_link_libraries (${target_name} PRIVATE AESDK::headers) target_link_libraries (${target_name} PRIVATE AESDK::source_utils) install (TARGETS ${target_name} RUNTIME DESTINATION plugins/ ${AE_VERSION} /64/windows) set(out_dir ${CMAKE_BINARY_DIR} /$<CONFIG>/ae) set_target_properties (${target_name} PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${out_dir} LIBRARY_OUTPUT_DIRECTORY ${out_dir} ARCHIVE_OUTPUT_DIRECTORY ${out_dir} Plugin sources PiPL processing Link to the SDK function (setup_pipl target_name pipl_file) target_sources (${target_name} PRIVATE ${pipl_file} ) add_custom_command (TARGET ${target_name} PRE_BUILD COMMAND cl.exe /D "${ae_macro}" /I "${GIT_LIB_FOLDER}/AESDK/${AESDK_VERSION}-windows/Headers" /I "${GIT_LIB_FOLDER}/AESDK/${AESDK_VERSION}-windows/Resources" /I "${CMAKE_CURRENT_SOURCE_DIR}/src" /EP ${pipl_file} > "${CMAKE_BINARY_DIR}/ae_effect_pipl.rr" COMMAND "${GIT_LIB_FOLDER}/AESDK/${AESDK_VERSION}-windows/Resources/PiPLtool.exe" "${CMAKE_BINARY_DIR}/ae_effect_pipl.rr" "${CMAKE_BINARY_DIR}/ae_effect_pipl.rrc" COMMAND cl.exe /D "MSWindows" /EP "${CMAKE_BINARY_DIR}/ae_effect_pipl.rrc" > "${CMAKE_BINARY_DIR}/ae_effect_pipl.rc" BYPRODUCTS "${CMAKE_BINARY_DIR}/ae_effect_pipl.rc" ) target_sources (${target_name} PRIVATE ${CMAKE_BINARY_DIR} /ae_effect_pipl.rc ) endfunction () Root CMakeFile.txt Function to process PiPL file SDK version is a variable
110. Summary about dependencies • Use only find_package to get dependencies ◦ Easier to switch to use a package manager later • Avoid diamond dependencies • Using package managers seems to be the future ◦ Not sure about the C++11 ABI problem for now ◦ Does not look very easy though ▪ How to use Vcpkg post ▪ First time i used VcPkg, i ran in an issue • Dependencies folder is not that bad • In production ◦ Manage the SDKs ▪ Create the cmake modules to find and configure them ◦ Use static libraries is usually ok • What about Git Submodules? ◦ Seems hard to manage
111. Dev: Build CI
112. Automation and continuous integration • Build automation ◦ On each push ◦ On various OS/Platform (Windows, Linux, iOS, Android, Game consoles…) ◦ Manually trigger • Use CI solution ◦ Jenkins ◦ Github actions ◦ Gitlab CI • Need script automation ◦ Configuration of each repo with actions or CI/Jenkins task is cumbersome ◦ Python
113. The production build matrix is huge • Multiple OS support ◦ Windows workstation for users ◦ Mac for artists ◦ Linux for render farm • Multiple projects in parallel ◦ Multiple tools version to support ◦ Multiple DCC tools version to support • Plugins ◦ Target a specific SDK ◦ Just like another OS support but… ▪ Less stable than OS ▪ More frequent breaking update ▪ New version with new requirements • OS • Compiler/C++ standard version Development projects (excluding pipeline) DCC Active Support Inactive DCC Versio ns OS Maya 3 30 57 4 2 Arnold 3 18 8 4 2 AE 3 25 26 3 1 Nuke 1 6 13 2 2 Photoshop 1 3 - 2 1 Standalone 2 3 - 2 Tools: Active 13, Support 75, Targets: Active ~ 250 Support ~2000 New Year (each new dcc version): ~200 New Tool Version: ~15 Tools x Tool Versions x DDC versions x OS DCC Number of targets =
114. Jenkins • Declarative pipeline • Groovy script ◦ Java based language • Allow to launch compilation on various machines ◦ Helps to automate linux/windows compilation • Use python call from jenkins to automate tasks ◦ Easy loop ◦ Avoid to configure each and every tool in jenkins ◦ Allow for easy and massive automation Jenkins CMake commands Python Execute Execute
115. Example of build Automation using Jenkins x Python • Create task list from user input ◦ Tool repository name ◦ Branch name = tool version ◦ DCC names and versions ◦ “all” wildcard to compile available version/dccs ◦ Deploy options ▪ More on that in the Deploy section • Launch compilations on relevant OS ◦ Trigger the python build script ◦ Trigger the python install and deploy ▪ Linux compilation files ▪ Windows compilation files ▪ Be careful about permission when you copy them • Cmake didn't seem to work as expected ◦ Created directories didn't get the group permissions • We copy linux files from windows
116. • Need tool listing file in readable/editable format ◦ Json ◦ Python • Information about compilation ◦ For each DCC + Version ▪ Each OS >>> Compiler ▪ Linux • New C++11 ABI • Old C++11 ABI • Defined by DCC version ◦ Specific info per DCC ▪ Arnold <> Maya couples ▪ Installed MtoA/Arnold list How to create the task list? maya_to_compiler_version = { "Windows": { "2019": "14.0", "2020": "14.1", "2022": "14.2", "2023": "14.2", "2024": "14.3", "2025": "14.3", }, "Linux": { "2019": "6.3.1", "2020": "6.3.1", "2022": "9.3.1", "2023": "9.3.1", "2024": "11.2.1", "2025": "11.2.1", } } nuke_to_compiler_version = { "Windows": { "13.2.3": "14.1", "14.1.2": "14.2", "15.0.1": "14.2"}, "Linux": { "13.2.3": "6.3.1", "14.1.2": "9.3.1", "15.0.1": "11.2.1"}, }
117. … "olm-toon ": { "tool_name ": "OLMToon ", "versions ": { "3.4": { "compile ": "scons", "dcc": {"arnold": {"dcc_versions ": config.arnold6 + config.arnold7, "sub_dir ": "maya"}, "maya": {"dcc_versions ": ["2022", "2023"]}, "mtoa": {"dcc_versions ": config.arnold6 + config.arnold7, "sub_dir ": "maya", }, }, }, … "4.2": { "compile ": "cmake", "dcc": { "arnold": {"dcc_versions ": config. versions_from ("arnold", "6.1.0.1 "),"sub_dir ": "maya"}, "maya": {"dcc_versions ": config. versions_from ("maya", "2022")}, "mtoa": {"dcc_versions ": config. versions_from ("arnold", "6.1.0.1 "),"sub_dir ": "maya"}, }, }, }, }, • Tool listing contains ◦ Repo name ◦ Tool name ◦ For each version ▪ Supported DCC + versions How to create the task list?
118. • From tool listing, compiler and SDKs info generate the task list • Each task has the following information ◦ Repo name ◦ Repo branch ◦ OS to compile for ◦ If linux use old c++ abi or new one ◦ For deployment/file naming ▪ Tool name ▪ Install resources ◦ Need to update the repository or not (run a clone or pull) How to create the task list?
119. Launch the tasks using Jenkins • Automate what can be on Jenkins side ◦ Trigger multiple python command line tool • Be careful at what can be done sequentially and in parallel • Clean in parallel ◦ The previous build/installed files • Update the repo in parallel • Configure sequentially ◦ Download of dependencies to the repo may result in disk writing collisions • Build and Install in parallel • Deploy in parallel ◦ Only once all build are finished
120. Jenkins automation job_map_main = [:] //os:command:job_name:job job_map_deploy = [:] def add_job (job_map , os, command , job_name , job){ if (!job_map.containsKey(os)){ job_map[os] = [:] } if (!job_map[os].containsKey(command)){ job_map[os][command] = [:] } job_map[os][command][job_name]= job } def generate_stage (operation , repo_name , branch_name , dcc, dcc_version , install_resources , os, new_abi , olm_setup_install_dir , olm_setup_deploy_dir ) { return { stage( "${operation } ${repo_name } ${branch_name } ${dcc} ${dcc_version } ${install_resources } ${os} ${new_abi }") { script{ execute(operation, repo_name, branch_name, dcc, dcc_version, install_resources, os, new_abi, olm_setup_install_dir, olm_setup_deploy_dir) } } } } def create_job (job_map , operation , repo_name , branch_name , dcc, dcc_version , install_resources , os, new_abi , olm_setup_install_dir , olm_setup_deploy_dir ){ job_id = repo_name + " " + operation + " " + branch_name + " " + dcc + " " + dcc_version + " "+install_resources+ " "+os + " " + new_abi add_job(job_map, os, operation, job_id, generate_stage(operation, repo_name, branch_name, dcc, dcc_version, install_resources, os, new_abi, olm_setup_install_dir, olm_setup_deploy_dir)) } Generate closure to be evaluated later Store in maps
121. Jenkins automation pipeline { (…) stages { stage( 'Sequential ') { (…) script { def tool_def_list = get_compile_tasks(params.REPO_NAME, params.DCC_NAME, params.BRANCH_NAME, params.DCC_VERSION) for (d in tool_def_list){ (…) create_job(job_map_main, "clean",repo_name, version, dcc, dcc_version, install_resources, os, new_abi, olm_setup_install_dir, olm_setup_deploy_dir) if (update_repository){ create_job(job_map_main, "update",repo_name, version, dcc, dcc_version, install_resources, os, new_abi, olm_setup_install_dir, olm_setup_deploy_dir) } create_job(job_map_main, "configure ",repo_name, version, dcc, dcc_version, install_resources, os, new_abi, olm_setup_install_dir, olm_setup_deploy_dir) create_job(job_map_main, "compile",repo_name, version, dcc, dcc_version, install_resources, os, new_abi, olm_setup_install_dir, olm_setup_deploy_dir) if (os=="windows"){ create_job(job_map_main, "deploy",repo_name, version, dcc, dcc_version, install_resources, os, new_abi, olm_setup_install_dir, olm_setup_deploy_dir) } (…) } } Generate each job to execute with python
122. Jenkins automation (…) stage( 'Cleaning ') { parallel { stage( 'Clean Windows '){ agent {label "windows"} steps{ script { job_chunk_windows_clean = get_job_map(job_map_main, "windows", "clean") execute_job_parallel(job_chunk_windows_clean, get_chunk_size()) } } } stage( 'Clean Linux ') { agent {label "linux"} steps{ script { job_chunk_linux_clean = get_job_map(job_map_main, "linux", "clean") execute_job_parallel(job_chunk_linux_clean, get_chunk_size()) } } } } } (…) Clean on windows and linux in parallel Remap task to chunk Jenkins would have memory problem if not def execute_job_parallel(job_list, chunk_size){ chunk_list = map_to_list_chunk(job_list, chunk_size) for (chunk in chunk_list){ catchError() { parallel chunk } } }
123. Jenkins automation (…) stage( 'Launch jobs ') { parallel { stage( 'Launch windows cmake jobs ') { agent {label "windows"} steps { script { job_chunk_windows_update = get_job_map(job_map_main, "windows", "update") job_chunk_windows_configure = get_job_map(job_map_main, "windows", "configure ") job_chunk_windows_compile = get_job_map(job_map_main, "windows", "compile") execute_job_parallel(job_chunk_windows_update, get_chunk_size()) execute_job_serial(job_chunk_windows_configure) execute_job_parallel(job_chunk_windows_compile, get_chunk_size()) } } } stage( 'Launch linux cmake jobs ') { (…) } } } Launch the tasks in parallel on windows and linux Execute the python tasks Update in parallel Configure serial Compile in parallel We may run into Disk writing collision if we run two tasks at the same time with same tools Easiest work around is to disable run to instance of the pipeline concurrently
124. Python command line • Low level command • Execute only atomic part of the build ◦ Clean|update|configure|compile|deploy python <path to script>/cmake_main.py clean|update|configure|compile|deploy -root_folder <Global repo> -source_dir <Where the source has been downloaded> -install_dir <Where to install the build> -deploy_dir <Where to deploy> -repo_name <The git url> -branch_name <The git branch name> -dcc <The DCC name> -dcc_version <The DCC version> -libstd_new_abi <if set use the new abi, if not old one> -install_resources <if set install the scripts and resources other than the build>
125. Python command line • Wraps Cmake call • Sets the CMAKE_MODULE_PATH • Sets the CMAKE_INSTALL_PREFIX • Sets the generator • Sets CMAKE_C_COMPILER, CMAKE_CXX_COMPILER • Sets the "CMAKE_CXX_FLAGS='-D_GLIBCXX_USE_CXX11_ABI=1/0'", • Sets DCC version macro since we compile for each dependency ◦ ARNOLD_VERSION, MAYA_VERSION… • Sets authentication token for Github ◦ OLM_GITHUB_TOKEN ◦ More details in the security part
126. About the build machines • Use machine at your company is safer ◦ Control on the configuration, libraries ◦ Easier to debug build • Build in the cloud ◦ Legal issues ▪ Maya: 2.1.1 No License Granted; Unauthorized Activities. ..(d) for Installation of the Licensed Materials on any Computer other than a Computer owned or leased, and controlled, by Licensee, unless otherwise authorized in writing by Autodesk ▪ AE • in the code :Dissemination of this information or reproduction of this material is strictly forbidden unless prior written permission is obtained from Adobe Systems Incorporated. • Developers are free to distribute plugins built using SDK headers and source files; you just can't distribute our SDK headers and source files. ▪ Houdini • In the code This software is proprietary to Side Effects Software Inc., and is not to be reproduced, transmitted, or disclosed in any way without written permission. ◦ Money issues ▪ Image available • Split compile per compiler? ▪ Cost for data transfer? Not sure how much. • Dependencies are multiple Gb ◦ Maya is 1GB! OpenCV is multiple Gb. Qt? Opencv? Boost? Eigen? • Action cache 10 Gb limit, link to a single repo ▪ Cost for github hosted
127. Summary of build for production • Maybe in some years we don't care about ABI problem anymore? ◦ Deprecated attribute since c++14 • Hopefully we use package manager for everything ◦ Custom dependency provider for the SDKs? • Massive automation is needed ◦ Now using Jenkins ◦ May move to github self hosted runner ▪ More secure ◦ Need script based automation • Still need to studies the compilation in the cloud ◦ Self hosted runners seems to be fine but slow
128. Dev: test
129. Test • Ensure software quality • Test ◦ Unit Tests ◦ Integration tests ◦ User tests • Better to have tests automated • Test libraries ◦ Doctest ◦ Catch ◦ Google Test ◦ Typical use ▪ Use your API and the test API to write test program ▪ Build test as independent executable ▪ Run the tests at/after build
130. Test in production • Plugins run within host applications ◦ Unit testing is challenging ◦ Need a lot of user input and context ▪ Could use mock? • Movies and cartoon primary quality is the picture! ◦ Between two versions of a tool ▪ Is output the same? ▪ Is time the same? • Test the image output ◦ All DCCs output images ◦ Need to run renders and compare them ◦ 3D tools ▪ Modeling/Animation/Rigging: same output >> same shape >> same render ▪ Shading/Lighting: same output >> same color >> same render ◦ 2D tools ▪ Drawing: same strokes >> same render ▪ Compositing: same picture generation/processing >> same render
131. How to test rendering output • Create the reference ◦ Choose tool version reference ◦ Choose DCC version reference ◦ Create reference DCC file ▪ Using the tool to test ▪ Use and animate all parameters ▪ Keep the number of frame to render low ◦ Render the reference • Create the variations ◦ Variation on DCC version ◦ Variation on tool version ◦ Render the variation • Compare the reference and the variations • Requirements ◦ Use lossless image format ◦ Program to compare images (count different pixels, use metrics…) ◦ Need automation of rendering (command line, render farm…)
132. How to use results and iterate Test Name DCC/Tool VS Reference Comparison to Reference Meaning S/S Same Normal Different Should not happen. Error in your reference or in your test process S/D Same Tool version stable on this DCC version Update the reference if D/S fails. Different Unexpected: check you code. Expected: update the reference. D/S Same DCC output stable. Can try D/D test to check tool stability. Different DCC output unstable. Update reference or create multiple references. D/D Same DCC and tool stable. No need to update the reference. Different DCC or tool is unstable. Run D/S to check. S: same D: Different
133. Example of a test framework in After Effects • What we want to test ◦ Tool features ▪ Vary parameter, animate… ◦ AE feature support ▪ Render in 8/16/32 bits ▪ Mask ▪ Downsampling, non square pixels ▪ Time remap ▪ Applied after gaussian blur • Create a test composition ◦ Use a specific version of the tool ◦ Generate multiple composition to cover the AE feature to tests • Render using output modules ◦ Use Tiff with lzw compression (works well for anime pictures) ◦ Bit settings also set by output module ◦ Create 8/16/32 bit output modules, install to rendering machine ▪ Quite annoying step… • Compare using libtiff custom compare tool ◦ Imagemagick was too slow
134. Conclusion about test • Needed • When to update the reference may be tricky ◦ Make tools to update references when needed • Automation of tests run with build • Each DCC may requires some specific adjustments ◦ Photoshop does not have command line rendering capabilities ◦ AE parameters cannot all be animated and changed by scripts • How to test UI interaction? ◦ By hand? …
135. Ops in production
136. Overview • Installing software and plugins • Release ◦ The development environment • Deploy ◦ The test and production environment • Operate • Monitor
137. Installing software and plugins • Software install ◦ Use an installer ▪ Content and installer need to be signed ◦ Multiple machines need command line installer + automation script • Plugin installation ◦ Depend on the host applications ▪ Copy in a directory ▪ Set environment variables
138. Installing plugins in production • Maya ◦ Modules ▪ .mod file to describe environment variables, platforms, local, module version ▪ Can set every other environment variables ◦ Environment variables ▪ Plugins: MAYA_PLUG_IN_PATH • Need to get your own Maya node IDs ▪ Mel Script: MAYA_SCRIPT_PATH ▪ Python script: PYTHONPATH ▪ Bifrost:BIFROST_LIB_CONFIG_FILES for components and plugins ▪ Icons: XBMLANGPATH ▪ Presets:MAYA_PRESET_PATH ◦ Maya.env: set of maya environment variables, don't override system ones ▪ One for each maya version in user home directory ◦ userSetup.mel, userSetup.py: script executed at startup of the application • Arnold ◦ Nodes/Procedurals/Shaders: ARNOLD_PLUGIN_PATH ◦ Python templates: MTOA_TEMPLATES_PATH • MtoA ◦ Translators: MTOA_EXTENSIONS_PATH
139. Installing plugins in production • AE ◦ Plugins: copy to the plugin folder ◦ Scripts: load from the script folder • Photoshop ◦ CEP extensions: copy to the extension folder ◦ Plugins: copy to the plugin folder • Nuke ◦ Plugins and Gizmo: NUKE_PATH ◦ OpenFX plugins: OFX_PLUGIN_PATH ◦ Python scripts: PYTHONPATH • Houdini ◦ Various environment variables (HOUDINI_PATH, HOUDINI_DSO_PATH, HOUDINI_MENU_PATH…)
140. Release • Copy the plugins locally ◦ File will be locked at DCC tool startup • Update environment variables if necessary • Launch the application • Everything should be automated
141. Release is more complicated • Multiple project in parallel ◦ Platform (DCC tool versions) set at the start of the project ◦ Similar to video games • Multiple tool versions X multiple DCC tool version • Application launcher ◦ Input ▪ Tool list and version for each project as input ▪ Repository: dev, test, prod ◦ Automate the setup of the DCC Tool • Need 3 different release environments ◦ For developers to test >> Dev ◦ For artists to test >> Test ◦ For artist to use in production >> Production
142. Example of application launcher • Command line mode for the render farm • UI mode for the users <?xml version="1.0" encoding="UTF-8"?> <profile project="DEV" version="Marc2024" olmsetup="dev"> <application name="MAYA" version="2024"/> <tool name="ArnoldMtoA" version="5.4.1.2"/> <tool name="CDVis" version="1.9"/> <tool name="OLMArrayMathematics" version="0.89.10"/> <tool name="OLMNoise" version="2.12.2"/> <tool name="ScriptLibrary" version="1.0"/> <tool name="ShotgunScripts" version="3.0"/> <tool name="OLMDirmap" version="latest"/> (...) </profile> Application to launch Vendor plugins process as other tools
143. Release and Deploy • Release ◦ Use in real conditions ◦ Quality assurance tests ◦ Prepare to send to final user ◦ In production ▪ Release = install the plugins to the development repository • Deploy ◦ Send to final user ▪ Installer, download ▪ Web service update ◦ In production ▪ Release = install the plugins to the test/production repository
144. Deploy and versions in production • Develop from last release in production to the next one • Increment version compared to last one ◦ May change during development according to version semantic ◦ Freeze version after release to production Tool x.y.z Tool x.y.z Tool u.v.w Tool x.y.z With x = u+1 OR y=v+1 OR z=w+1 Production to Development Test to Production Test to Development Development to test
145. Operate in production • The system guys side ◦ Client machine ◦ Servers ◦ Network ◦ Software install and licences • The pipeline side ◦ Assets management ◦ Rendering wrangler ◦ Pipeline tools… • Everyone cares about the artist feedback ◦ Tracking and manage ◦ Using tools described in the plan phase
146. What could be monitor in production • Crash report and logs • How the tool is used? • How often the tool is used? • How fast is the rendering? • Who uses the tool? • Can derives metrics ◦ Tool popularity and impact ◦ Identify bottlenecks • Be aware ◦ Some tools are essentially used at specific phases ▪ Ex: rigging tool only used at rig stage ◦ Some tools are used by very specialized people ▪ Ex: hair grooming tools
147. Security
148. Code Plan Build Test Release Deploy Operate Monitor DEV OPS DCC licenses Binary/Installer sign CI solution (Jenkins, Github Actions) SCM Servers, Machines, Licences Trello, Jira, Figma Google Apps VC++ DCC licenses SCM Developer account (VC++, Qt) OS Account SCM (Github, Gitlab) Remote connection (Splashtop, VPN, SSH) Many logins and authentication
149. About security • Login in various account and authentication ◦ User to machine ▪ On site machine ▪ Remotely • From home • At the company (unifies the way to access pc) ◦ User to web service ▪ SCM in the cloud (Github, Gitlab…) ▪ Google apps ◦ Service to service ▪ Github <> Google Cloud ▪ Jenkins <> Github ▪ Github self hosted Runner <> Github • Installer on windows ◦ Requires binaries and DLLs to be signed ◦ CPack is not enough
150. Many technologies… • Old world ◦ VPN ◦ SSH • New world ◦ HTTPS ◦ Two step authentication ◦ Security Assertion Markup Language (SAML) ◦ Single sign on ◦ Open iD Connect (OiDC) ▪ Workload Identity (Google) ▪ AWS, Azure
151. Example: SAML • User logins using identity provider ◦ Windows account, google account • When access to any service provider SAML triggered ◦ Check with identity provider ◦ If ok grants access with a token • Benefits ◦ No need for the user to sign in for each service ◦ Used by single sign on
152. Principles • Add security on the authentication ◦ Connect identity provider to service provider ◦ Identity provider emit a token ▪ Limited permissions/credentials ▪ Limited in time ▪ Limit damages in case of break ◦ Service provider trusts identity provider • Simplify user experience ◦ Less password to remember or change
153. Service to service authentication • Should use the same principles as user to service • OiDC to connect Identity provider to Service provider ◦ On top of OAuth2.0 ◦ Uses HTTPS • HTTPS ◦ Need to get a SSL certificate from a certificate authority (ex: GlobalSign) ◦ Need to manage certificate (they expire) ◦ Use port 80, 443 ▪ Usually open, useable in most environment ◦ Recommended by github • Identity provider using OiDC ◦ Google workload identity ◦ AWS, Azure • Service provider able to use OiDC ◦ Github ◦ Gitlab ◦ Google, AWS, Azure • Identity provider manage the time limited tokens for you
154. Service to service authentication in production • Lots of automations • Authentication with SCM in the cloud (Github, Gitlab…) ◦ Compile machine ◦ CI solutions • Sign of binaries and installers ◦ Requires access to certificate stored in a secure place ▪ With authentication ▪ Ex: Google drive • Ideally: implement OiDC • Sometimes ◦ Use PAT, github app authentication, … ◦ SSH
155. Authentication at OLM • Google Account ◦ Authentication: Single Sign On + SAML ◦ Two step authentication • Splashtop ◦ With single sign on (from google account) ◦ Connect to PC through Splashtop even at the company ▪ Less latency than home • Jenkins ◦ Internal server ◦ Our instance does not support HTTPS ▪ Cannot connect to Workload identity • Github ◦ With single sign on (from google account) ◦ Two step authentication ▪ Mandatory since 2023 ◦ Https for users ◦ CI solution connects through Github apps ▪ Cannot use a specific user account • not the right way to do it, what about the password? ▪ Generate temporary token use in every git commands ▪ Generate a pem file • Does not expire! Like ssh private key • Better than PAT but not ideal ▪ Plan to move to Github self hosted runners + Github actions + Google Workload Identity
156. Not recommended solutions • VPN ◦ Security: attack target ◦ Update requires network stop ▪ No update >> attack ◦ Balance between connection speed and cost • SSH ◦ No screen, slow ▪ Remote desktop ▪ VNC ◦ Need to open 22 port ◦ Private key ▪ Per user ▪ User can generate and create ▪ Use password manager ▪ Leak of private key problem ▪ Maybe no time limits • Personal Access Token PAT ◦ Couldn't use Fine-grained PAT in the past ◦ Need to decide how much right to give, the user can decide ◦ Related to one person: what when the persons leaves the company?
157. • C++ modules ◦ Need more experience with it ◦ Has to flag the module files (Compiler, cmake…) • SVN, Mercurial, Scons ◦ Not used that much ones • Github actions ◦ Working on it to replace Jenkins • Gitlab ◦ CI seems to be popular and very well designed • Ninja ◦ Seem to speed up compilation time • Rust ◦ Some effort to use Rust in CG existed but Still alive? • Qt ◦ de facto standard for UI. Moc generation may need some specific management • Javascript ◦ Specific versions in After effects, Photoshop • Python ◦ Used in every major DCC package ◦ Switch from python 2 to 3 was quite painful What we didn't talk about: techs • MacOS ◦ Used by a lot on artists. We can requests to support macos for our open tools • Conan ◦ Don't know the share of vcpkg VS Conan, Seem to have more flexibility than Vcpkg, but seem more complex • Jira ◦ Integrates tickets, reports and docs together. Good to manage the feedback and specifications? Tried some years ago… • SPK and SPFS ◦ For complex management of packages in CG pipelines • Rez ◦ Prepare environment to launch application. Used in production • Docker ◦ latform for developing, shipping, and running applications • Other DCCs ◦ Houdini, Cinema4D, 3dsMax, Katana, Blender… cannot cover all of them. Maybe in the future.
158. What we didn't talk about: methods • Waterfall VS Agile: we think agile is more appropriate for production • Specifications ◦ So many levels of abstraction ▪ UI concepts ▪ Functionality overviews ▪ … ▪ UML diagrams ▪ C++ function signature + expected output ◦ How to write? ▪ Most docs i found are either too high or too low level ▪ In the end … ad hoc ▪ For UI • Figma • Google slides
159. Conclusion • Plan ◦ Manage the backlog ◦ Be sure to keep communication going on with artists • Code ◦ Rules are important ◦ Specific semantic versioning for production • Build ◦ Many build to achieve ◦ Automation is mandatory ◦ ABI!! • Test ◦ Testing code in a host application is complicated ◦ Testing picture is different than testing code • Release/Deploy ◦ Application launcher is mandatory • Operate ◦ Be sure that the artists can focus on art ◦ Always listen to what they have to say • Monitor ◦ Automation of report and logging ◦ Use logs for future improvements
160. DevOps Journey at OLM
161. Evolution of DevOps in 15 years at OLM Digital 15 years ago Now WIP Staff 2 developers, 1 researcher 10+ developers, 1 researcher Issue tracking Nothing Trello Trello + Git projects Source code management Nothing SVN/Git Git Package manager Nothing In-house In-House (Git, CMake) Build Visual studio VS, GCC, Scons, Jenkins VS, GCC, Cmake, Github actions Deployment Manual copy /installers (wix) to clients Automatic copy to render servers Automatic DCC tool configuration Automatic install to clients/servers Continuous integration Nothing Jenkins (legacy jobs) Jenkins (pipeline scripts) Integration tests for CG Tool versioning Nothing Semantic versioning for CG tool
162. Why changing technology? • Move from SVN to Git ◦ Central server on site at OLM ◦ Access from outside with splashtop ▪ Latency problems with remote work • Foreign internship student • Foreign employees working from outside of Japan ◦ Up to 550 ms latency using simple web page test • Working from different parts in Japan ◦ Collaboration with other companies • Move from scons to cmake ◦ Our Scons solution was not implemented correctly ▪ Needed rework ◦ Cmake became a standard in the meantime ◦ Cmake affinity with git is great
163. Future works at OLM • Plan/Operate ◦ Connect report/feedback with coding: Jira? ◦ How to write spec? • Build ◦ Self hosted runner with Github actions ◦ Use package manager? vcpkg? • Test ◦ Testing using user simulation through system like AutoHotKey • Release/Deploy ◦ More automation to manage the project tool needs ◦ Installer through Self hosted runner with Github actions • Monitor ◦ Actually monitor and generate some useful statistics
164. Future works for this course • Other techs? ◦ Gitlab, Jira. Rez, Conan, Docker… • Other DCCs? ◦ Houdini, Katana… • How to switch from SVN to Git? • How to switch from Scons to Cmake? • More CMake ◦ Details about each DCCs CMakes? ◦ Dependencies? ◦ Link to package managers? • Github actions • More case studies? • Who wanna collaborate for a 3 hours DevOps for production course?
165. Code Plan Build Test Release Deploy Operate Monitor DEV OPS Release to test environment Release to production environment Automatic test after build Servers, Machines, Licences Messages, Automatic/Manual data collection Getting the artist in the room Survey,Track issues Multiple DCCs, DCCs versions, Multiple OS Multiple host languages for plugins
166. Reference about Agile • Scrum is like a war room situation • https://devops.com/agile-scrum-is-a-failure-heres-why/ • https://stackoverflow.blog/2020/06/29/does-scrum-ruin-great-engineers-or-are -you-doing-it-wrong/ • https://techbeacon.com/app-dev-testing/why-scrum-sucks • https://bootcamp.uxdesign.cc/scrum-is-just-as-bad-as-safe-ac4f1d0f1193 • https://blog.logrocket.com/why-scrum-is-becoming-irrelevant/ • https://www.linkedin.com/pulse/why-does-scrum-make-programmers-hate-co ding-jayme-edwards/ • https://www.linkedin.com/pulse/kanban-considered-harmful-chris-nicola/ • https://www.lean.org/the-lean-post/articles/how-can-kanban-be-useful-in-softw are-since-we-never-produce-the-same-part-twice/