Augmenting Creative Work with Dynamic Checklists, Automated Quality Assurance, and Contextual Reviewer Feedback

Checklists and guidelines have played an increasingly important role in complex tasks ranging from the cockpit to the operating theater. Their role in creative tasks like design is less explored. In a needfinding study with expert web designers, we identified designers' challenges in adhering to a checklist of design guidelines. We built Critter, which addressed these challenges with three components: Dynamic Checklists that progressively disclose guideline complexity with a self-pruning hierarchical view, AutoQA to automate common quality assurance checks, and guideline-specific feedback provided by a reviewer to highlight mistakes as they appear. In an observational study, we found that the more engaged a designer was with Critter, the fewer mistakes they made in following design guidelines. Designers rated the AutoQA and contextual feedback experience highly, and provided feedback on the tradeoffs of the hierarchical Dynamic Checklists. We additionally found that a majority of designers rated the AutoQA experience as excellent and felt that it increased the quality of their work. Finally, we discuss broader implications for supporting complex creative tasks.

Transcript

1. Cri er: Augmenting Creative Work with Dynamic Checklists, Automated ality Assurance, and Contextual Reviewer Feedback Aditya Bharadwaj1, Pao Siangliulue2, Adam Marcus2, Kurt Luther1 Presented by: Aditya Bharadwaj May 7, 2019 1 2
2. Life-saving power of a simple checklist A study indicated a 37% decrease in post-surgical deaths in Scotland since the introduction of 19-item surgical safety checklist in 2008
3. Checklists are e ective in structured tasks Facilitate the enforcement of guidelines E ective in areas that require precision
4. Checklists are e ective in structured tasks Facilitate the enforcement of guidelines E ective in areas that require precision Is there room to explore checklists in creative and semi-structured tasks?
5. Creative tasks Website design So ware development Music composition
6. Creative tasks Website design So ware development Music composition While we o en pay more a ention to the novelty of creative work, the quality and the repeatability of creative products is no less important
7. Creative processes are not linear like checklists Creative processes are complex, dynamic, and non-linear Designers can have drastically di erent workflows Designers vary their approach on projects of di erent complexity or scope Getting Started Website Structure Content Global Style Website QA Wrapping Up
8. Creative processes are not linear like checklists Creative processes are complex, dynamic, and non-linear Designers can have drastically di erent workflows Designers vary their approach on projects of di erent complexity or scope Getting Started Website Structure Content Global Style Website QA Wrapping Up Can we e ectively use checklists to enforce guidelines in creative tasks?
9. Real world deployment of checklists
10. Real world deployment of checklists B12, a startup that works with web design experts to create websites for their customers
11. Real world deployment of checklists B12, a startup that works with web design experts to create websites for their customers Text cut-off! Contact Button About Page Link to wrong page! Unhappy client
12. Real world deployment of checklists B12, a startup that works with web design experts to create websites for their customers Use best practices and customer requirements as design guidelines Make sure the hyperlinks/bu ons link to the correct page Make sure the text is not cut-o on mobile Text cut-off! Contact Button About Page Link to wrong page! Unhappy client
13. Real world deployment of checklists A 153-point checklist to remind workers about important to-do items and best practices An automated checker to detect errors in design Expert reviewers’ feedback on areas that need improvement
14. Real world deployment of checklists A 153-point checklist to remind workers about important to-do items and best practices An automated checker to detect errors in design Expert reviewers’ feedback on areas that need improvement Designers missed explicit customer preferences or best practices
15. Needfinding Study 4 web designers Shadowed each designer for approximately 3–4 hours Interviews before and a er the shadow session
16. Needfinding Study 4 web designers Shadowed each designer for approximately 3–4 hours Interviews before and a er the shadow session Identified five needs for a new system
17. Needfinding Study 4 web designers Shadowed each designer for approximately 3–4 hours Interviews before and a er the shadow session Identified five needs for a new system 1. Each project has specific requirements that must be noted and adhered to
18. Needfinding Study 4 web designers Shadowed each designer for approximately 3–4 hours Interviews before and a er the shadow session Identified five needs for a new system 1. Each project has specific requirements that must be noted and adhered to 2. Only a subset of the design guidelines apply to a project
19. Needfinding Study 4 web designers Shadowed each designer for approximately 3–4 hours Interviews before and a er the shadow session Identified five needs for a new system 1. Each project has specific requirements that must be noted and adhered to 2. Only a subset of the design guidelines apply to a project 3. Designers vary the order in which they complete a checklist depending on customer priorities and their areas of expertise
20. Needfinding Study 4 web designers Shadowed each designer for approximately 3–4 hours Interviews before and a er the shadow session Identified five needs for a new system 1. Each project has specific requirements that must be noted and adhered to 2. Only a subset of the design guidelines apply to a project 3. Designers vary the order in which they complete a checklist depending on customer priorities and their areas of expertise 4. Designers desire and appreciate high-quality human feedback
21. Needfinding Study 4 web designers Shadowed each designer for approximately 3–4 hours Interviews before and a er the shadow session Identified five needs for a new system 1. Each project has specific requirements that must be noted and adhered to 2. Only a subset of the design guidelines apply to a project 3. Designers vary the order in which they complete a checklist depending on customer priorities and their areas of expertise 4. Designers desire and appreciate high-quality human feedback 5. Designers overestimate their capacity to remember project-specific details and design guidelines
22. Cri er Dynamic Checklists Each project has specific requirements that must be noted and adhered to Only a subset of the design guidelines apply to a project Designers vary the order in which they complete a checklist depending on customer priorities and their areas of expertise Contextual reviewer-provided feedback Designers desire and appreciate high-quality human feedback AutoQA Designers overestimate their capacity to remember project-specific details and design guidelines
23. Dynamic Checklists Designers can add a checklist template that includes all relevant guidelines and can be applied to any project
24. Dynamic Checklists Dynamic Checklist analyzes customer details and automatically skips irrelevant checklist items
25. Dynamic Checklists Dynamic Checklist analyzes customer details and automatically skips irrelevant checklist items
26. Dynamic Checklists Designers can further identify irrelevant checklist items that were not automatically detected as “[not relevant]” and move them to “Skipped todo items” column
27. Dynamic Checklists An expert can add individual checklist item related to this project by writing its details and clicking “Add todo”
28. Dynamic Checklists
29. Dynamic Checklists
30. Dynamic Checklists Designers can drill down to any level of specificity and swi ly navigate di erent parts of the checklist
31. AutoQA Captures a report of the errors detected in the project
32. Contextual reviewer-provided feedback Reviewers can leave notes on individual checklist items to give specific and structured feedback
33. Contextual reviewer-provided feedback Displays the feedback received on previous projects as a recommended todo
34. Mixed-methods evaluation 6 web designers 5 B12 client websites per designer Asked the designers to use Cri er as they saw fit
35. Mixed-methods evaluation 6 web designers 5 B12 client websites per designer Asked the designers to use Cri er as they saw fit A reviewer evaluated the websites based on their adherence to the 153-item template checklist
36. Mixed-methods evaluation 6 web designers 5 B12 client websites per designer Asked the designers to use Cri er as they saw fit A reviewer evaluated the websites based on their adherence to the 153-item template checklist Data collected Online survey Ratings on five-point Likert scales Open-ended questions Follow-up interviews
37. Mixed-methods evaluation 6 web designers 5 B12 client websites per designer Asked the designers to use Cri er as they saw fit A reviewer evaluated the websites based on their adherence to the 153-item template checklist Data collected Online survey Ratings on five-point Likert scales Open-ended questions Follow-up interviews Interaction logs
38. How did the designers use Cri er? 0 50 100 (A) Checking items off (#completed items) D1 D2 D3 D4 D5 D6 Designers 0 1 2 (B) Skipping (#skipped/#completed) 0 2 (C) Drilling deep (Avg. completed item depth) 0 100 (D) Toggling focus (#Expand/Collapse) 0 50 100 (F) Resolving AutoQA errors (%Successful AutoQA) 0 50 100 (E) Applying reviewer recommendations (%Successful Recommendations) Low quality website High quality website
39. How did the designers use Cri er? 0 50 100 (A) Checking items off (#completed items) D1 D2 D3 D4 D5 D6 Designers 0 1 2 (B) Skipping (#skipped/#completed) 0 2 (C) Drilling deep (Avg. completed item depth) 0 100 (D) Toggling focus (#Expand/Collapse) 0 50 100 (F) Resolving AutoQA errors (%Successful AutoQA) 0 50 100 (E) Applying reviewer recommendations (%Successful Recommendations) Low quality website High quality website Observation: D1, D2, D3 were more engaged with Cri er than D5, D6
40. How did the designers use Cri er? 0 50 100 Experience prior to the study (#websites) D1 D2 D3 D4 D5 D6 Designers 0 5 10 Mistakes (quality 1/#mistakes) D1 D2 D3 D4 D5 D6 Low quality website High quality website
41. How did the designers use Cri er? 0 50 100 Experience prior to the study (#websites) D1 D2 D3 D4 D5 D6 Designers 0 5 10 Mistakes (quality 1/#mistakes) D1 D2 D3 D4 D5 D6 Low quality website High quality website Observation: D1, D2, D3 had less prior experience than D5, D6
42. How did designers’ use of Cri er a ect the design quality? 0 50 100 Experience prior to the study (#websites) D1 D2 D3 D4 D5 D6 Designers 0 5 10 Mistakes (quality 1/#mistakes) D1 D2 D3 D4 D5 D6 Low quality website High quality website
43. How did designers’ use of Cri er a ect the design quality? 0 50 100 Experience prior to the study (#websites) D1 D2 D3 D4 D5 D6 Designers 0 5 10 Mistakes (quality 1/#mistakes) D1 D2 D3 D4 D5 D6 Low quality website High quality website Observation: Novice designers who adhered to the system made fewer mistakes than experienced designers who didn’t
44. Key design implications Hierarchical checklists “It was convenient because it was logically divided and allows the designer to open up one branch and close it when it is done” — D1 “It is nicer to be able to collapse them, and allows me to mentally compartmentalize the work” — D2
45. Key design implications Hierarchical checklists “It was convenient because it was logically divided and allows the designer to open up one branch and close it when it is done” — D1 “It is nicer to be able to collapse them, and allows me to mentally compartmentalize the work” — D2 Flat checklists “overwhelming” and “hard to use” — D1 “lot of scrolling” — D4
46. Key design implications Hierarchical checklists “It was convenient because it was logically divided and allows the designer to open up one branch and close it when it is done” — D1 “It is nicer to be able to collapse them, and allows me to mentally compartmentalize the work” — D2 Flat checklists “overwhelming” and “hard to use” — D1 “lot of scrolling” — D4 Hierarchical checklists support task non-linearity
47. Key design implications Contextual reviewer feedback “It actually helped me to be er understand some todo items and and what results are expected” — D1 “It would be an issue you were struggling with and by showing up, it reminds you to work on it” — D5
48. Key design implications Contextual reviewer feedback “It actually helped me to be er understand some todo items and and what results are expected” — D1 “It would be an issue you were struggling with and by showing up, it reminds you to work on it” — D5 Checklist as rubrics for feedback support learning and creative work
49. Generalization to other domains Design guidelines, AutoQA, and reviewer feedback have analogies in many creative fields Writing Design guidelines ↔ wri en style guides AutoQA ↔ Grammerly/LanguageTool Reviewer feedback ↔ editor’s feedback So ware development Design guidelines ↔ programming style guides AutoQA ↔ linters Reviewer feedback ↔ code review
50. Generalization to other domains Design guidelines, AutoQA, and reviewer feedback have analogies in many creative fields Writing Design guidelines ↔ wri en style guides AutoQA ↔ Grammerly/LanguageTool Reviewer feedback ↔ editor’s feedback So ware development Design guidelines ↔ programming style guides AutoQA ↔ linters Reviewer feedback ↔ code review Need more exploration of dynamic checklists and Cri er in other creative fields
51. Acknowledgements Pao Siangliulue Adam Marcus Kurt Luther 1UH2CA203768-01 B12 team and design experts
52. Summary A needfinding study highlighting gaps in traditional checklists A novel approach that combines automation with human-driven planning and reflection to help experts navigate complex creative tasks A system, Cri er, that exhibits this approach with three components An empirical, mixed-methods evaluation of web designers using Cri er to create websites for 30 B12 clients
53. Appendix
54. Design Guidelines as quality rubrics There is a rich history of using guidelines to evaluate quality in design [Yuan 2016] and other fields [Kulkarni 2013] While design guidelines may not capture all of a client’s quality considerations, we ensured that the guidelines were reasonably comprehensive Designers pushed back on reviewer judgment in only one instance In future work, it might be interesting to use customer ratings as quality measure.
55. Dynamic checklist usage model Finish (do) majority of the tasks Check (confirm) corresponding checklist item Go through (read) the list of remaining incomplete items Actively skip irrelevant items Finish (do) relevant checklist items
56. Ratings