Algorithmic task planning: How to automate an entire company

Every agency needs to prioritize and sort incoming work in some way or another. After years of manual sorting and lots of discussions and emotional debates about which task (and which customer) is the most important and why, we decided to implement an automatic work prioritization script into our Jira workflow. In this talk, we will present all factors playing into the automatic sorting process, the concept behind it and give some insights into how we implemented it.

Talk held at the TYPO3 Developer Days 2022 (https://t3dd22.typo3.com)

Transcript

1. ALGORITHMIC TASK PLANNING HOW TO AUTOMATE AN ENTIRE COMPANY 06th August 2022
2. WHAT WE ARE GOING TO TALK ABOUT TODAY • BACKGROUND ◦ Project management @MFC ◦ What problems did we run into? ◦ What needed to change? • ALGORITHMIC TASK PLANNING ◦ Criteria for prioritization ◦ Why the hell would we automate that? ◦ General principles of function ◦ Validate for correctness ◦ General architecture • LEARNINGS AGENDA
3. QUICK INTRODUCTION WHO ARE WE? Luisa Faßbender (26) • Senior Project Manager @ Marketing Factory • manages several customer projects, takes care of corporate channels • Marketing & Communications Masters’ student • TYPO3 Marketing Team Lead since 2019 • in her private life she enjoys going to the gym, cooking and visiting festivals Christian Spoo (33) • Senior Developer @ Marketing Factory • develops, designs architectures for customer projects, maintains systems • professionally interested in low-level hardware / OS-related topics • in his private life he likes playing ninepins and enjoys over-engineering his home
4. QUICK INTRODUCTION WHERE ARE WE WORKING? • Est. in 1996 • Based in Düsseldorf • Full Service Digital Agency • Focus on TYPO3, Shopware and Symfony • 20 employees • 50% female employees • TYPO3 Association Gold Member • Shopware Business Partner
5. BACKGROUND IN THE BEGINNING, THERE WAS CHAOS…
6. PROJECT MANAGEMENT @MFC BACK IN THE DARK, DARK DAYS… “Agile” project management (<2020) • employee resources were collected on a weekly basis • time spend in meetings and on open tasks was deducted • additional 30% for daily business tasks was also deducted • the remaining time got booked with a plethora of tasks • developers provided estimations on tickets • project managers had weekly meetings to plan tasks ➔ 1 meeting á 20 people: 20 hours of weekly work time ➔ 1 designated employee for collecting the resources weekly ➔ LOTS of discussions, tears and fights
8. PROBLEMS WHAT WERE THE PROBLEMS WITH THE OLD PROCESS? • still practicing push and not pull, • almost never able to replenish new tickets, despite recruiting several new employees, • only had little ticket throughput, • can hardly provide support for several smaller customers, • pull tickets according to "I just feel like it” / “I just don't feel like it", • not taking (customer) prioritizations into account,
9. 9 PROBLEMS WHAT WERE THE PROBLEMS WITH THE OLD PROCESS? • back to departmental thinking, • no knowledge transfer, • felt that we are working very inefficiently, • had lots of weekly, heartfelt discussions about capacities, • required endless, recurring, meetings dealing with planning of work, • unhappy with the way of working and experienced low motivation
10. PROBLEMS: CONCLUSION … THE PAIN POINTS WHICH CAUSED FRICTION SLOW AND UNHAPPY Lots of work in progress, minimal ticket throughput, no motivation and heightened annoyance. MEETINGS, MEETINGS.. We spent a minimum of 21 hours every week just in meetings, planning work and resources. CLEAR BOUNDARIES Decided tasks for backend, frontend, project management, sysadmin & co. No knowledge transfer. RANDOM PROCESSING Tasks were pulled randomly, didn’t follow customer and project manager prioritization and just weren’t done.
11. WHAT NEEDED TO CHANGE? HOW COULD WE MAKE EVERYONE’S LIFE EASIER? TALK LESS, DO MORE Reduce time spent in meetings talking about tickets to a minimum and free time to be used for actual work. LESS BOUNDARIES Break down departmental boundaries and enable all employees to broaden their horizons. CLEAR STRUCTURE Define a clear, overarching task prioritization for all team members. MORE EQUALITY Ensure equal work on all customer projects to improve customer happiness and regular output.
12. MARKETING FACTORY CONSULTING GMBH AGENTURVORSTELLUNG WE CAN DO BETTER… LET US INTRODUCE: ALGORITHMIC TASK PLANNING
13. CRITERIA FOR PRIORITIZATION WHICH CRITERIA DID WE TAKE INTO ACCOUNT? We defined the following criteria to be relevant when it comes to correctly sorting them: 1. Is the ticket a fastlane ticket? 2. Is this ticket a no-brainer? 3. Does the ticket have a deadline? 4. Does the ticket unblock other tickets? 5. Is there a monthly budget / number of tickets committed for certain customers? 6. Have certain tickets been prioritized by the customer? 7. How old is the ticket? How long has it been waiting in line? 8. When was the last time a customer's ticket was pulled?
14. 1 fastlane 3 unblocks deadline ticket 4 deadline 5 unblocks ticket 6 monthly budget 7 customer prioritization CRITERIA FOR PRIORITIZATION WHICH CRITERIA DID WE TAKE INTO ACCOUNT? time since last customer ticket was pulled 8 2 no-brainer
15. CRITERIA FOR PRIORITIZATION WHICH INFORMATION DO WE NEED TO ADD TO THE TICKETS?
16. WHERE TO START? REQUIREMENTS FOR ALGORITHMIC TASK PLANNING Agree with people that you want to delegate task planning to a machine • not everybody might feel good about this. Discuss concerns with your colleagues • will only work if everybody acts in concert Work need to be represented by a ticket in order to minimize invisible work • issue creation must be cheap • know the purpose and the goal of the issue and when it will be considered done • Bonus: if you can, let your monitoring tools create tickets themselves
17. WHERE TO START? REQUIREMENTS FOR ALGORITHMIC TASK PLANNING Collect contractual support and maintenance budgets • document and update them if they change Budget separation needs to be known: Which task belongs to which budget? • Jira projects ./. cost centers • are there any Jira projects that share the same budget?
18. WHY ON EARTH …? HOW DID WE END UP LETTING A PIECE OF SOFTWARE DECIDE WHAT TO DO NEXT? • Planning work as such is already a technical problem on its own • Analogy: multitasking computer systems ◦ your operating system has to decide which process should be given control of the CPU next ◦ an OS uses a scheduler to manage that ◦ nowadays systems have multiple CPUs, i.e. execution units that can perform work in parallel ◦ some processes could be run, some might be blocked. ◦ the OS tries to be fair to all processes, i.e. each process should receive computation time and no process shall starve Put simply, we needed a scheduler for our company.
19. GENERAL PRINCIPLE OF FUNCTION HOW DOES OUR ISSUE SORTER WORK? 1. Fetch all queued issues from Jira, … ◦ but except issues belonging to an upcoming sprint … ◦ and except issues belonging to certain projects
20. GENERAL PRINCIPLE OF FUNCTION HOW DOES OUR ISSUE SORTER WORK? 2. Fetch budgeted hours per project that already have been expended
21. GENERAL PRINCIPLE OF FUNCTION HOW DOES OUR ISSUE SORTER WORK? 3. Select customer whose turn it is next. ◦ We determine the previous head of the queue (i.e. the issue that led the field) ◦ We then restore the previous order among all customers ◦ Last, we continue after the customer whose ticket was previously at the top of the queue
22. GENERAL PRINCIPLE OF FUNCTION HOW DOES OUR ISSUE SORTER WORK? 3. Divide pool of upcoming issues into several “draw pots” ◦ fastlane ◦ unblocking ◦ etc. 4. Sort issues in each pot 5. Concatenate pots together to form the new queue 6. Calculate strictly monotonic queue priority number for each issue and update Jira accordingly 7. Predefined issue filters in Jira display issues sorted by their assigned ordinal number
23. 23 VALIDATE FOR CORRECTNESS WHAT IF SOMETHING GOES WRONG? • Unit tests to ensure basic functionality of internal classes (e.g. loading of configuration parameters) • Functional tests written in Gherkin to describe the expected behaviour ◦ primarily used for regression testing ◦ anyone at MFC should be able to specify new tests ◦ everyone should be able to understand the function principles • Important: some sorting criteria are time-dependent, i.e. one has to simulate a certain point in time for some logic to work properly
24. GENERAL ARCHITECTURE LOOKING UNDER THE HOOD OF THE APPLICATION • Sorter application is implemented as a command-line Symfony application ◦ Symfony’s architecture is a good fit for our purposes ◦ many things available without much effort (DI, integration of tests, logging, console output) • You’ll need programmatic access to your tools ◦ we access Jira’s HTTP REST API ◦ we’ve got REST-based access to our time tracker and accounting application • The application is run regularly through our GitLab instance • Test cases implemented in Gherkin and run through Codeception
25. MARKETING FACTORY CONSULTING GMBH AGENTURVORSTELLUNG LEARNINGS… OUR LEARNINGS SO FAR
26. LEARNINGS: TECHNOLOGY WHAT DID WE LEARN WITH REGARDS TO IT-RELATED TOPICS? Jira’s API • both the API and JQL are incredibly powerful, … • but on the other hand extremely inconsistent MOCO, our accounting application • first: If you have programmatic access to your accounting data - make use of it • even beyond dealing with daily work you might want to gather interesting numbers The project was fun because we could basically apply development principles to a company • describe organizational problems with software tests • learn more about our own processes and question them • remove every process detail that could not objectively be decided by a machine
27. LEARNINGS: ORGANIZATIONAL WHAT DID WE LEARN WITH REGARDS TO ORGANIZATION-RELATED TOPICS? Overcoming scepticism • our idea didn’t meet a lot of enthusiasm within the team at first • team members were sceptic and didn’t think it would be possible to automate something, that has heavily relied on human input for years (even decades) • BUT: discussions and scepticism died down eventually and the process has been accepted More time for actual work • project managers aren’t stuck in heated meetings anymore, don’t need to create lists of tasks for said meetings and are able to get more done Way less discussions • project managers are able to provide more concrete estimations • the open and transparent process leads to more trust into it and enables a clear understanding of why certain things are the way they are
28. THANK YOU! ANY QUESTIONS?