Algorithmic task planning: How to automate an entire company

Every agency needs to prioritize and sort incoming work in some way or another. After years of manual sorting and lots of discussions and emotional debates about which task (and which customer) is the most important and why, we decided to implement an automatic work prioritization script into our Jira workflow. In this talk, we will present all factors playing into the automatic sorting process, the concept behind it and give some insights into how we implemented it.

Talk held at the TYPO3 Developer Days 2022 (https://t3dd22.typo3.com)

Transcript

1. ALGORITHMIC TASK PLANNING
   HOW TO AUTOMATE AN ENTIRE COMPANY
   06th August 2022
   

   View Slide

2. WHAT WE ARE GOING TO TALK ABOUT TODAY
   ● BACKGROUND
   ○ Project management @MFC
   ○ What problems did we run into?
   ○ What needed to change?
   ● ALGORITHMIC TASK PLANNING
   ○ Criteria for prioritization
   ○ Why the hell would we automate that?
   ○ General principles of function
   ○ Validate for correctness
   ○ General architecture
   ● LEARNINGS
   AGENDA
   

   View Slide

3. QUICK INTRODUCTION
   WHO ARE WE?
   Luisa Faßbender (26)
   ● Senior Project Manager @ Marketing Factory
   ● manages several customer projects, takes care
   of corporate channels
   ● Marketing & Communications Masters’ student
   ● TYPO3 Marketing Team Lead since 2019
   ● in her private life she enjoys going to the gym,
   cooking and visiting festivals
   Christian Spoo (33)
   ● Senior Developer @ Marketing Factory
   ● develops, designs architectures for customer
   projects, maintains systems
   ● professionally interested in low-level hardware
   / OS-related topics
   ● in his private life he likes playing ninepins and
   enjoys over-engineering his home
   

   View Slide

4. QUICK INTRODUCTION
   WHERE ARE WE WORKING?
   ● Est. in 1996
   ● Based in Düsseldorf
   ● Full Service Digital Agency
   ● Focus on TYPO3, Shopware and Symfony
   ● 20 employees
   ● 50% female employees
   ● TYPO3 Association Gold Member
   ● Shopware Business Partner
   

   View Slide

5. BACKGROUND
   IN THE BEGINNING, THERE WAS CHAOS…
   

   View Slide

6. PROJECT MANAGEMENT @MFC
   BACK IN THE DARK, DARK DAYS…
   “Agile” project management (<2020)
   ● employee resources were collected on a weekly basis
   ● time spend in meetings and on open tasks was deducted
   ● additional 30% for daily business tasks was also deducted
   ● the remaining time got booked with a plethora of tasks
   ● developers provided estimations on tickets
   ● project managers had weekly meetings to plan tasks
   ➔ 1 meeting á 20 people: 20 hours of weekly work time
   ➔ 1 designated employee for collecting the resources weekly
   ➔ LOTS of discussions, tears and fights
   

   View Slide

7. View Slide

8. PROBLEMS
   WHAT WERE THE PROBLEMS WITH THE OLD PROCESS?
   ● still practicing push and not pull,
   ● almost never able to replenish new tickets, despite recruiting
   several new employees,
   ● only had little ticket throughput,
   ● can hardly provide support for several smaller customers,
   ● pull tickets according to "I just feel like it” / “I just don't feel
   like it",
   ● not taking (customer) prioritizations into account,
   

   View Slide

9. 9
   PROBLEMS
   WHAT WERE THE PROBLEMS WITH THE OLD PROCESS?
   ● back to departmental thinking,
   ● no knowledge transfer,
   ● felt that we are working very inefficiently,
   ● had lots of weekly, heartfelt discussions about
   capacities,
   ● required endless, recurring, meetings dealing with
   planning of work,
   ● unhappy with the way of working and experienced
   low motivation
   

   View Slide

10. PROBLEMS: CONCLUSION
    … THE PAIN POINTS WHICH CAUSED FRICTION
    SLOW AND UNHAPPY
    Lots of work in progress,
    minimal ticket
    throughput, no
    motivation and
    heightened annoyance.
    MEETINGS, MEETINGS..
    We spent a minimum
    of 21 hours every week
    just in meetings,
    planning work and
    resources.
    CLEAR BOUNDARIES
    Decided tasks for
    backend, frontend,
    project management,
    sysadmin & co. No
    knowledge transfer.
    RANDOM PROCESSING
    Tasks were pulled
    randomly, didn’t follow
    customer and project
    manager prioritization
    and just weren’t done.
    

    View Slide

11. WHAT NEEDED TO CHANGE?
    HOW COULD WE MAKE EVERYONE’S LIFE EASIER?
    TALK LESS, DO MORE
    Reduce time spent in
    meetings talking about
    tickets to a minimum
    and free time to be
    used for actual work.
    LESS BOUNDARIES
    Break down
    departmental
    boundaries and enable
    all employees to
    broaden their horizons.
    CLEAR STRUCTURE
    Define a clear,
    overarching task
    prioritization for all
    team members.
    MORE EQUALITY
    Ensure equal work on
    all customer projects
    to improve customer
    happiness and regular
    output.
    

    View Slide

12. MARKETING FACTORY
    CONSULTING GMBH
    AGENTURVORSTELLUNG
    WE CAN DO BETTER…
    LET US INTRODUCE: ALGORITHMIC TASK PLANNING
    

    View Slide

13. CRITERIA FOR PRIORITIZATION
    WHICH CRITERIA DID WE TAKE INTO ACCOUNT?
    We defined the following criteria to be relevant when it comes to correctly sorting them:
    1. Is the ticket a fastlane ticket?
    2. Is this ticket a no-brainer?
    3. Does the ticket have a deadline?
    4. Does the ticket unblock other tickets?
    5. Is there a monthly budget / number of tickets committed for certain customers?
    6. Have certain tickets been prioritized by the customer?
    7. How old is the ticket? How long has it been waiting in line?
    8. When was the last time a customer's ticket was pulled?
    

    View Slide

14. 1
    fastlane
    3
    unblocks deadline ticket
    4
    deadline
    5
    unblocks ticket
    6
    monthly budget
    7
    customer prioritization
    CRITERIA FOR PRIORITIZATION
    WHICH CRITERIA DID WE TAKE INTO ACCOUNT?
    time since last customer ticket was pulled
    8
    2
    no-brainer
    

    View Slide

15. CRITERIA FOR PRIORITIZATION
    WHICH INFORMATION DO WE NEED TO ADD TO THE TICKETS?
    

    View Slide

16. WHERE TO START?
    REQUIREMENTS FOR ALGORITHMIC TASK PLANNING
    Agree with people that you want to delegate task planning to a machine
    ● not everybody might feel good about this. Discuss concerns with your colleagues
    ● will only work if everybody acts in concert
    Work need to be represented by a ticket in order to minimize invisible work
    ● issue creation must be cheap
    ● know the purpose and the goal of the issue and when it will be considered done
    ● Bonus: if you can, let your monitoring tools create tickets themselves
    

    View Slide

17. WHERE TO START?
    REQUIREMENTS FOR ALGORITHMIC TASK PLANNING
    Collect contractual support and maintenance budgets
    ● document and update them if they change
    Budget separation needs to be known: Which task belongs to which budget?
    ● Jira projects ./. cost centers
    ● are there any Jira projects that share the same budget?
    

    View Slide

18. WHY ON EARTH …?
    HOW DID WE END UP LETTING A PIECE OF SOFTWARE DECIDE WHAT TO DO NEXT?
    ● Planning work as such is already a technical problem on its own
    ● Analogy: multitasking computer systems
    ○ your operating system has to decide which process should be given
    control of the CPU next
    ○ an OS uses a scheduler to manage that
    ○ nowadays systems have multiple CPUs, i.e. execution units that can
    perform work in parallel
    ○ some processes could be run, some might be blocked.
    ○ the OS tries to be fair to all processes, i.e. each process should
    receive computation time and no process shall starve
    Put simply, we needed a scheduler for our company.
    

    View Slide

19. GENERAL PRINCIPLE OF FUNCTION
    HOW DOES OUR ISSUE SORTER WORK?
    1. Fetch all queued issues from Jira, …
    ○ but except issues belonging to an upcoming
    sprint …
    ○ and except issues belonging to certain
    projects
    

    View Slide

20. GENERAL PRINCIPLE OF FUNCTION
    HOW DOES OUR ISSUE SORTER WORK?
    2. Fetch budgeted hours per project that already have been expended
    

    View Slide

21. GENERAL PRINCIPLE OF FUNCTION
    HOW DOES OUR ISSUE SORTER WORK?
    3. Select customer whose turn it is next.
    ○ We determine the previous head of the queue (i.e. the issue that led the field)
    ○ We then restore the previous order among all customers
    ○ Last, we continue after the customer whose ticket was previously at the top of the queue
    

    View Slide

22. GENERAL PRINCIPLE OF FUNCTION
    HOW DOES OUR ISSUE SORTER WORK?
    3. Divide pool of upcoming issues into several “draw
    pots”
    ○ fastlane
    ○ unblocking
    ○ etc.
    4. Sort issues in each pot
    5. Concatenate pots together to form the new queue
    6. Calculate strictly monotonic queue priority number for
    each issue and update Jira accordingly
    7. Predefined issue filters in Jira display issues sorted by
    their assigned ordinal number
    

    View Slide

23. 23
    VALIDATE FOR CORRECTNESS
    WHAT IF SOMETHING GOES WRONG?
    ● Unit tests to ensure basic functionality of internal
    classes (e.g. loading of configuration parameters)
    ● Functional tests written in Gherkin to describe the
    expected behaviour
    ○ primarily used for regression testing
    ○ anyone at MFC should be able to specify new
    tests
    ○ everyone should be able to understand the
    function principles
    ● Important: some sorting criteria are time-dependent,
    i.e. one has to simulate a certain point in time for some
    logic to work properly
    

    View Slide

24. GENERAL ARCHITECTURE
    LOOKING UNDER THE HOOD OF THE APPLICATION
    ● Sorter application is implemented as a command-line Symfony
    application
    ○ Symfony’s architecture is a good fit for our purposes
    ○ many things available without much effort (DI,
    integration of tests, logging, console output)
    ● You’ll need programmatic access to your tools
    ○ we access Jira’s HTTP REST API
    ○ we’ve got REST-based access to our time tracker and
    accounting application
    ● The application is run regularly through our GitLab instance
    ● Test cases implemented in Gherkin and run through
    Codeception
    

    View Slide

25. MARKETING FACTORY
    CONSULTING GMBH
    AGENTURVORSTELLUNG
    LEARNINGS…
    OUR LEARNINGS SO FAR
    

    View Slide

26. LEARNINGS: TECHNOLOGY
    WHAT DID WE LEARN WITH REGARDS TO IT-RELATED TOPICS?
    Jira’s API
    ● both the API and JQL are incredibly powerful, …
    ● but on the other hand extremely inconsistent
    MOCO, our accounting application
    ● first: If you have programmatic access to your accounting data - make use of it
    ● even beyond dealing with daily work you might want to gather interesting numbers
    The project was fun because we could basically apply development principles to a company
    ● describe organizational problems with software tests
    ● learn more about our own processes and question them
    ● remove every process detail that could not objectively be decided by a machine
    

    View Slide

27. LEARNINGS: ORGANIZATIONAL
    WHAT DID WE LEARN WITH REGARDS TO ORGANIZATION-RELATED TOPICS?
    Overcoming scepticism
    ● our idea didn’t meet a lot of enthusiasm within the team at first
    ● team members were sceptic and didn’t think it would be possible to automate something, that has heavily relied on
    human input for years (even decades)
    ● BUT: discussions and scepticism died down eventually and the process has been accepted
    More time for actual work
    ● project managers aren’t stuck in heated meetings anymore, don’t need to create lists of tasks for said meetings and are
    able to get more done
    Way less discussions
    ● project managers are able to provide more concrete estimations
    ● the open and transparent process leads to more trust into it and enables a clear understanding of why certain things are
    the way they are
    

    View Slide

28. THANK YOU!
    ANY QUESTIONS?
    

    View Slide