Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Algorithmic task planning: How to automate an entire company

Algorithmic task planning: How to automate an entire company

Every agency needs to prioritize and sort incoming work in some way or another. After years of manual sorting and lots of discussions and emotional debates about which task (and which customer) is the most important and why, we decided to implement an automatic work prioritization script into our Jira workflow. In this talk, we will present all factors playing into the automatic sorting process, the concept behind it and give some insights into how we implemented it.

Talk held at the TYPO3 Developer Days 2022 (https://t3dd22.typo3.com)

Christian Spoo

August 10, 2022
Tweet

More Decks by Christian Spoo

Other Decks in Programming

Transcript

  1. WHAT WE ARE GOING TO TALK ABOUT TODAY • BACKGROUND

    ◦ Project management @MFC ◦ What problems did we run into? ◦ What needed to change? • ALGORITHMIC TASK PLANNING ◦ Criteria for prioritization ◦ Why the hell would we automate that? ◦ General principles of function ◦ Validate for correctness ◦ General architecture • LEARNINGS AGENDA
  2. QUICK INTRODUCTION WHO ARE WE? Luisa Faßbender (26) • Senior

    Project Manager @ Marketing Factory • manages several customer projects, takes care of corporate channels • Marketing & Communications Masters’ student • TYPO3 Marketing Team Lead since 2019 • in her private life she enjoys going to the gym, cooking and visiting festivals Christian Spoo (33) • Senior Developer @ Marketing Factory • develops, designs architectures for customer projects, maintains systems • professionally interested in low-level hardware / OS-related topics • in his private life he likes playing ninepins and enjoys over-engineering his home
  3. QUICK INTRODUCTION WHERE ARE WE WORKING? • Est. in 1996

    • Based in Düsseldorf • Full Service Digital Agency • Focus on TYPO3, Shopware and Symfony • 20 employees • 50% female employees • TYPO3 Association Gold Member • Shopware Business Partner
  4. PROJECT MANAGEMENT @MFC BACK IN THE DARK, DARK DAYS… “Agile”

    project management (<2020) • employee resources were collected on a weekly basis • time spend in meetings and on open tasks was deducted • additional 30% for daily business tasks was also deducted • the remaining time got booked with a plethora of tasks • developers provided estimations on tickets • project managers had weekly meetings to plan tasks ➔ 1 meeting á 20 people: 20 hours of weekly work time ➔ 1 designated employee for collecting the resources weekly ➔ LOTS of discussions, tears and fights
  5. PROBLEMS WHAT WERE THE PROBLEMS WITH THE OLD PROCESS? •

    still practicing push and not pull, • almost never able to replenish new tickets, despite recruiting several new employees, • only had little ticket throughput, • can hardly provide support for several smaller customers, • pull tickets according to "I just feel like it” / “I just don't feel like it", • not taking (customer) prioritizations into account,
  6. 9 PROBLEMS WHAT WERE THE PROBLEMS WITH THE OLD PROCESS?

    • back to departmental thinking, • no knowledge transfer, • felt that we are working very inefficiently, • had lots of weekly, heartfelt discussions about capacities, • required endless, recurring, meetings dealing with planning of work, • unhappy with the way of working and experienced low motivation
  7. PROBLEMS: CONCLUSION … THE PAIN POINTS WHICH CAUSED FRICTION SLOW

    AND UNHAPPY Lots of work in progress, minimal ticket throughput, no motivation and heightened annoyance. MEETINGS, MEETINGS.. We spent a minimum of 21 hours every week just in meetings, planning work and resources. CLEAR BOUNDARIES Decided tasks for backend, frontend, project management, sysadmin & co. No knowledge transfer. RANDOM PROCESSING Tasks were pulled randomly, didn’t follow customer and project manager prioritization and just weren’t done.
  8. WHAT NEEDED TO CHANGE? HOW COULD WE MAKE EVERYONE’S LIFE

    EASIER? TALK LESS, DO MORE Reduce time spent in meetings talking about tickets to a minimum and free time to be used for actual work. LESS BOUNDARIES Break down departmental boundaries and enable all employees to broaden their horizons. CLEAR STRUCTURE Define a clear, overarching task prioritization for all team members. MORE EQUALITY Ensure equal work on all customer projects to improve customer happiness and regular output.
  9. CRITERIA FOR PRIORITIZATION WHICH CRITERIA DID WE TAKE INTO ACCOUNT?

    We defined the following criteria to be relevant when it comes to correctly sorting them: 1. Is the ticket a fastlane ticket? 2. Is this ticket a no-brainer? 3. Does the ticket have a deadline? 4. Does the ticket unblock other tickets? 5. Is there a monthly budget / number of tickets committed for certain customers? 6. Have certain tickets been prioritized by the customer? 7. How old is the ticket? How long has it been waiting in line? 8. When was the last time a customer's ticket was pulled?
  10. 1 fastlane 3 unblocks deadline ticket 4 deadline 5 unblocks

    ticket 6 monthly budget 7 customer prioritization CRITERIA FOR PRIORITIZATION WHICH CRITERIA DID WE TAKE INTO ACCOUNT? time since last customer ticket was pulled 8 2 no-brainer
  11. WHERE TO START? REQUIREMENTS FOR ALGORITHMIC TASK PLANNING Agree with

    people that you want to delegate task planning to a machine • not everybody might feel good about this. Discuss concerns with your colleagues • will only work if everybody acts in concert Work need to be represented by a ticket in order to minimize invisible work • issue creation must be cheap • know the purpose and the goal of the issue and when it will be considered done • Bonus: if you can, let your monitoring tools create tickets themselves
  12. WHERE TO START? REQUIREMENTS FOR ALGORITHMIC TASK PLANNING Collect contractual

    support and maintenance budgets • document and update them if they change Budget separation needs to be known: Which task belongs to which budget? • Jira projects ./. cost centers • are there any Jira projects that share the same budget?
  13. WHY ON EARTH …? HOW DID WE END UP LETTING

    A PIECE OF SOFTWARE DECIDE WHAT TO DO NEXT? • Planning work as such is already a technical problem on its own • Analogy: multitasking computer systems ◦ your operating system has to decide which process should be given control of the CPU next ◦ an OS uses a scheduler to manage that ◦ nowadays systems have multiple CPUs, i.e. execution units that can perform work in parallel ◦ some processes could be run, some might be blocked. ◦ the OS tries to be fair to all processes, i.e. each process should receive computation time and no process shall starve Put simply, we needed a scheduler for our company.
  14. GENERAL PRINCIPLE OF FUNCTION HOW DOES OUR ISSUE SORTER WORK?

    1. Fetch all queued issues from Jira, … ◦ but except issues belonging to an upcoming sprint … ◦ and except issues belonging to certain projects
  15. GENERAL PRINCIPLE OF FUNCTION HOW DOES OUR ISSUE SORTER WORK?

    2. Fetch budgeted hours per project that already have been expended
  16. GENERAL PRINCIPLE OF FUNCTION HOW DOES OUR ISSUE SORTER WORK?

    3. Select customer whose turn it is next. ◦ We determine the previous head of the queue (i.e. the issue that led the field) ◦ We then restore the previous order among all customers ◦ Last, we continue after the customer whose ticket was previously at the top of the queue
  17. GENERAL PRINCIPLE OF FUNCTION HOW DOES OUR ISSUE SORTER WORK?

    3. Divide pool of upcoming issues into several “draw pots” ◦ fastlane ◦ unblocking ◦ etc. 4. Sort issues in each pot 5. Concatenate pots together to form the new queue 6. Calculate strictly monotonic queue priority number for each issue and update Jira accordingly 7. Predefined issue filters in Jira display issues sorted by their assigned ordinal number
  18. 23 VALIDATE FOR CORRECTNESS WHAT IF SOMETHING GOES WRONG? •

    Unit tests to ensure basic functionality of internal classes (e.g. loading of configuration parameters) • Functional tests written in Gherkin to describe the expected behaviour ◦ primarily used for regression testing ◦ anyone at MFC should be able to specify new tests ◦ everyone should be able to understand the function principles • Important: some sorting criteria are time-dependent, i.e. one has to simulate a certain point in time for some logic to work properly
  19. GENERAL ARCHITECTURE LOOKING UNDER THE HOOD OF THE APPLICATION •

    Sorter application is implemented as a command-line Symfony application ◦ Symfony’s architecture is a good fit for our purposes ◦ many things available without much effort (DI, integration of tests, logging, console output) • You’ll need programmatic access to your tools ◦ we access Jira’s HTTP REST API ◦ we’ve got REST-based access to our time tracker and accounting application • The application is run regularly through our GitLab instance • Test cases implemented in Gherkin and run through Codeception
  20. LEARNINGS: TECHNOLOGY WHAT DID WE LEARN WITH REGARDS TO IT-RELATED

    TOPICS? Jira’s API • both the API and JQL are incredibly powerful, … • but on the other hand extremely inconsistent MOCO, our accounting application • first: If you have programmatic access to your accounting data - make use of it • even beyond dealing with daily work you might want to gather interesting numbers The project was fun because we could basically apply development principles to a company • describe organizational problems with software tests • learn more about our own processes and question them • remove every process detail that could not objectively be decided by a machine
  21. LEARNINGS: ORGANIZATIONAL WHAT DID WE LEARN WITH REGARDS TO ORGANIZATION-RELATED

    TOPICS? Overcoming scepticism • our idea didn’t meet a lot of enthusiasm within the team at first • team members were sceptic and didn’t think it would be possible to automate something, that has heavily relied on human input for years (even decades) • BUT: discussions and scepticism died down eventually and the process has been accepted More time for actual work • project managers aren’t stuck in heated meetings anymore, don’t need to create lists of tasks for said meetings and are able to get more done Way less discussions • project managers are able to provide more concrete estimations • the open and transparent process leads to more trust into it and enables a clear understanding of why certain things are the way they are