USDA SNAP QC - Discovery Findings Report

Transcript

1. Prepared by: Rock Creek Strategic Marketing Date: April 8, 2014 USDA SNAP QC Training Discovery Findings Report
2. USDA SNAP QC Training | APR 8 2 Discovery Findings Report Table of Contents 03 Project Overview 04 Why We’re Doing This 05 Approach 06 Project Status 07 Phase 1 // Discovery 08 Approach 09 Conducted Research Activities 11 Research Synthesis 13 Personas 18 General Trends 22 Insights vs Observations 23 Key Insight #1 26 Key Insight #2 30 Key Insight #3 33 Key Insight #4 35 Key Insight #5 37 Observation #1 38 Observation #2 39 Observation #3 40 Observation #4 41 Concept Preview 44 Next Steps xx Phase 2 // Concept xx xx xx xx Phase 3 // Prototype xx xx xx 45 Appendix // Discovery xx Appendix // Concept xx Appendix // Prototype
3. USDA SNAP QC Training | APR 8 3 Project Overview
4. USDA SNAP QC Training | APR 8 4 Project Overview Why We’re Doing This There’s currently a need for a better implemented solution for training QC staff. We’re looking to design and develop efficient, effective, and evolving solutions to train, teach, and transfer knowledge to QC staff.
5. USDA SNAP QC Training | APR 8 5 Project Overview Approach
6. USDA SNAP QC Training | APR 8 6 Phase 1 // Discovery Phase 2 // Concept Phase 3 // Prototype Project Overview Current Status
7. USDA SNAP QC Training | APR 8 7 Phase 1 // Discovery
8. USDA SNAP QC Training | APR 8 8 Phase 1 // Discovery Approach We conducted several information gathering activities to get a comprehensive understanding of the content, the resources, and the QC staff and their responsibilities. These activities included: Stakeholder Interviews & 3-Day Training Focus Group with QC Trainers Focus Group with QC Coordinators Interviews with Program Specialists
9. USDA SNAP QC Training | APR 8 9 Phase 1 // Discovery Conducted Research Activities Stakeholder Interviews Stakeholder interviews are one-on-one, in-depth talks with people who have a vested interest in the project. These people often include executives, board members, program staff, and so on. Stakeholder interviews help the team understand the organizational goals the project must support. The team uses interview findings to align organizational goals and user needs, which steer the project’s development. Details • Participant: Fran Heil • Conducted a series of three in-person interviews Focus Groups A focus group is a structured conversation with a small sampling of a project’s target audience or key stakeholders. They are always led by a facilitator, who engages the group in a moderated discussion. Focus groups are used to gain insights and observe trends from vested interest groups to inform the project’s concept phase. Focus groups also serve as a way to engage participants in a healthy and open dialogue. QC Trainer Focus Group Details • Participants: Lana Rae Tapia, Dave Young, Mary Black-Finke, Fran Heil • Conducted one in-person session QC Coordinator Focus Group Details • Participants: QC Coordinators & Team Leads • Dawn Baker, Western Region (WRO) • Lori Kelly, Southwest Region (SWRO) • Brenda Kuykendall, Southeast Region (SERO) • Sara May, Mountains Plains Region (MPRO) • Lorraine Pilla, Mid-Atlantic Region (MARO) • Joanne Rando, Northeast Region (NERO) • Lynn Sims, Western Region (WRO) • Conducted one session over the phone • Also interviewed Janet Davis (SWRO), who retired prior to the focus group • The Midwest Region was not represented in this activity. Program Specialist Interviews User interviews are one-on-one, in-depth talks with individuals who will interact with the solution. Interviews help gather a clearer understanding of user needs, wants, pain points, and tasks. User interviews help the team learn about why and how people interact with each other and current resources. These insights inform persona development and help influence the solutions. Details • Participants: 11 Program Specialists of varying experience levels, ages, and locations. • Conducted individual interviews over the phone
10. USDA SNAP QC Training | APR 8 10 Phase 1 // Discovery Program Specialist Interviews Program Specialists were selected at random based on their experience in the role and their location. Participants were encouraged to speak freely since their interviews were confidential and the focus was on the content of what they had to say, not who said what from which region. Experience Level • 0-3 years: 5 individuals • 4-10 years: 3 individuals • 10+ years: 3 individuals Representation by Region • MARO: 1 individual • MPRO: 1 individual • MWRO: 2 individuals • NERO: 2 individuals • SERO: 1 individual • SWRO: 2 individuals • WRO: 2 individuals
11. USDA SNAP QC Training | APR 8 11 Phase 1 // Discovery Research Synthesis & Sense-making
12. USDA SNAP QC Training | APR 8 12 Phase 1 // Discovery Research Synthesis & Sense-making
13. USDA SNAP QC Training | APR 8 13 Phase 1 // Discovery Research Synthesis & Sense-making: Personas Personas are fictitious, but evidence-based characters that are compiled from a variety of sources (e.g., interviews, contextual inquiries, and so on) to provide a generalized archetype of each user group. Personas can include imagery, demographics (e.g., age, gender, occupation), habits and goals (e.g., frequency of web use, reasons for visiting a resource), and so on. Personas symbolize the target audience and help the team better understand the audience’s needs, wants, and motivations. Personas make use-case scenarios more realistic and help validate decisions on potential solutions. new SNAP BACKGROUND EXPERIENCE seasoned yes no
14. USDA SNAP QC Training | APR 8 14 Phase 1 // Discovery Persona #1: Tom New Program Specialist with no prior SNAP background “It’s a tough job - there’s a lot to remember. I’m not sure how long I see myself doing this.” - Program Specialist Tools and Resources • Tom uses a laptop and is really tech savvy; he has the latest iPhone and iPad • He consults the 310 Handbook frequently, but keeps a notebook full of his own personal notes next to his computer. • When he has questions, he goes to the QC Coordinator for assistance. Training Experience Tom was provided formal, one-on-one training with the QC Coordinator. They went over the basics of the SNAP program, then the basics of SNAP QC, before finally moving onto the 310 Handbook and sample cases. His recurring training is through emails, but on occasion the team will meet in person or on conference call to discuss the new policy changes. Overall Thoughts Tom would love a “Cliffs Notes” version of the 310 Handbook so he can better navigate to the information he needs quickly. Working in both QC and Policy has made keeping up difficult, especially when he struggles writing disagrees. He also worries that even with a heavy workload, there’s no room for career advancement. Experience: Prior experience: Work location: Workload: Training received: Traits: 6 months Worked at another government agency Works at the office 25% on QC, works on active cases only Formal training with QC Coordinator Tech savvy, poor writer
15. USDA SNAP QC Training | APR 8 15 Phase 1 // Discovery Persona #2: Sharon New Program Specialist with a prior SNAP background “I may be new, but I’m motivated and have been able to find the answers I need.” - Sharon Tools and Resources • Sharon uses a laptop and loves her big monitor at the office since Program Specialists have multiple documents and windows to switch between. • She uses the 310 Handbook, the PartnerWeb, and other cheat sheets that her peers have passed onto her. • She often goes to her peers for questions, but her QC Coordinator still reviews 100% of her work. Training Experience Sharon had a 3-hour training session with the QC Coordinator, where they reviewed the 310, completed a few worksheets, and went through a few cases step by step. She has yet to receive recurring training since starting this position. Overall Thoughts Sharon recognizes she still has a lot to learn and is up to the challenge. She’s self-sufficient in finding information, but wishes the resources such as the 310 Handbook and PartnerWeb were easier to navigate. She’s not sure if she could do this job without having a prior SNAP background. Experience: Prior experience: Work location: Workload: Training received: Traits: 2 months SNAP Policy Reviewer for 7 years Works at the office, occasionally from home 50% on QC, for now only does active cases Formal training, 3-hour training session with QC Coordinator Self-starter
16. USDA SNAP QC Training | APR 8 16 Phase 1 // Discovery Persona #3: Robert Experienced Program Specialist with no prior SNAP background “I love my job. I’m a problem solver and like to get things done.” - Program Specialist Tools and Resources • Robert uses a laptop, but does not own a smartphone or tablet. • He consults the 310 Handbook and occasionally references the PartnerWeb. His primary resources are cheat sheets, which he’s created from a variety of SNAP documents. • He consults his peers first for help and sometimes his QC Coordinator for further assistance. Training Experience Robert went through a 3-4 day, classroom-style experience, which included an overview of the QC process and working through some sample cases. He receives recurring training through emails from the QC Coordinator and sometimes they will have a conference call. The calls are mostly information on changing policy and how to apply them to their work. Overall Thoughts Robert loves his job and feels that this is his true calling. He recognizes there’s a lack of consistency in the QC program, where other Specialists do reviews differently from him. He also notices that the state and federal supervisors provide conflicting information about policy recommendations. Experience: Prior experience: Work location: Workload: Training received: Traits: 11 years Private sector and military Works from home 100% on QC, works on all cases Formal training, 3-4 day bootcamp Problem solver, organized, task oriented
17. USDA SNAP QC Training | APR 8 17 Phase 1 // Discovery Persona #4: Arline Experience Program Specialist with a prior SNAP background “I have a lot of work. It’s a lot to remember. I would love tools to make my job easier.” - Arline Tools and Resources • Arline uses a laptop, but at the office has a dual monitor set up. • She uses her personal iPad to read and find information in the 310 Handbook. • For complicated cases, she turns to her peers who work on the same state and very rarely seeks help from the QC Coordinator. Training Experience Arline received training from a senior employee, who provided lots of information at once. Her background in SNAP made it easier to sort what was relevant at the time. Refresher training comes in the form of emails, but sometimes these emails are a few months late. Overall Thoughts Arline is dedicated to her job, but feels overwhelmed by the amount of work. She’s trained new staff members before, but it’s only been when she can squeeze in time. She feels it would be more beneficial if the National Office provided a standard baseline training for them instead. Experience: Prior experience: Work location: Workload: Training received: Traits: 4 years Caseworker and State QC Reviewer for 10 years Works from home and at the office 100% on QC, works on all cases for 4 different states Informal training by a senior employee Meticulous, team player
18. USDA SNAP QC Training | APR 8 18 Phase 1 // Discovery General Trends: Learning Methods QCC PEER SELF Current QCC PEER SELF Desired
19. USDA SNAP QC Training | APR 8 19 Phase 1 // Discovery General Trends: Learning Model more help (guided) less help (self-directed) often (repetitive) seldom (resource) TASK FREQUENCY ASSISTANCE ROQCTS Disagree Letter Arbitration
20. USDA SNAP QC Training | APR 8 20 Phase 1 // Discovery General Trends: Learning Model TASK FREQUENCY ASSISTANCE ROQCTS SNAP Basics Arbitration QCC Peer Guide more help (guided) less help (self-directed) often (repetitive) seldom (resource)
21. USDA SNAP QC Training | APR 8 21 Phase 1 // Discovery General Trends: Confidence Level Program Specialists that were interviewed, who have worked at least one year in the role. Informally trained Formally trained 0 3 6 9 12 Amount Time to be Confident Performing Their Job Months Program Specialists* Confidence Level From the interviews, there was no clear trend that formal or informal training made a difference in how confident Program Specialists were in performing their job. *
22. USDA SNAP QC Training | APR 8 22 Phase 1 // Discovery Insights vs. Observations From the activities and synthesis, we extracted insights and observations. Insights are findings that can be affected by the training solutions. Observations are findings that cannot be affected by the training solutions, but some can influence the training solutions.
23. USDA SNAP QC Training | APR 8 23 Phase 1 // Discovery Key Insight #1: Program Specialists have complicated jobs, are task loaded, and have strict deadlines. To be successful, they need to be extremely competent in basic SNAP knowledge and professional skills.
24. USDA SNAP QC Training | APR 8 24 Phase 1 // Discovery Key Insight #1: SUPPORTING EVIDENCE Work Allotment Some Program Specialists spend 50-75% of their time doing other types of reviews (e.g., Policy reviews) aside from QC reviews. What they have to know for QC reviews is a small portion of their needed knowledge base. 0 25 50 75 100 QC Reviews Other Reviews Program Specialists “It’s hard to switch gears and be in a million directions all at once.” - Program Specialist
25. USDA SNAP QC Training | APR 8 25 Phase 1 // Discovery Key Insight #1: SUPPORTING EVIDENCE “A [SNAP] policy background has helped me ramp up on this job. I would be completely lost without the policy background.” - Program Specialist Having a background in SNAP is helpful in doing QC reviews. Basic writing, grammar, and time management skills are necessary. • QC Coordinators repeatedly mentioned the need for training on writing and grammar skills. • Program Specialists noted the considerable amount of writing required to complete some reviews. • Program Specialists indicated time management is an important skill to successfully complete their designated caseloads. “[My] eligibility [background] has helped me learn to navigate a case.” - Program Specialist
26. USDA SNAP QC Training | APR 8 26 Phase 1 // Discovery Key Insight #2: It’s critical to establish a training framework that provides context, shared understanding, and a holistic view of what QC does, so Program Specialists can learn effectively.
27. USDA SNAP QC Training | APR 8 27 Phase 1 // Discovery Key Insight #2: SUPPORTING EVIDENCE Training methods and materials are inconsistent. Formal (7) Informal (4) Staff has been trained differently. Of the Program Specialists interviewed, 7 received formal training by a QC Coordinator and 4 received informal training by a peer. • Formal: 3-hour, “classroom style” course by the QC Coordinator, which included: • Walking through a case step- by-step • Seeing the 310 Handbook • Completing a review sheet, a few worksheets, and a few budgets Training experiences • Informal: Specialist walks around from peer to peer asking for assistance for various tasks (e.g., using ROQCTS, writing a disagree). Peers are not as responsive or helpful, so the Specialist looks for information elsewhere.
28. USDA SNAP QC Training | APR 8 28 Phase 1 // Discovery Key Insight #2: SUPPORTING EVIDENCE Training methods and materials are inconsistent. Materials are “Frankenstein-ed” together to form a training solution. • Powerpoint presentations from previous training • Powerpoint presentations created by the QC Coordinator • 310 Handbook • State manuals • State profiles Materials used include: • Photocopies of the 310 • Previous disagree letters • Reference sheet on elements • Clarification emails • Policy memos • Personal notes on completing forms • Print outs from state websites • QC Coordinators curate materials from past training or from coordinators of other regions • Program Specialists seek possible learning materials on the PartnerWeb for self-study
29. USDA SNAP QC Training | APR 8 29 Phase 1 // Discovery Key Insight #2: SUPPORTING EVIDENCE “Don’t know what’s most important and what’s least important; just kind of learned everything at once.” - Program Specialist on initial training There is a lot of information to understand and digest.
30. USDA SNAP QC Training | APR 8 29 Phase 1 // Discovery Key Insight #2: SUPPORTING EVIDENCE “Don’t know what’s most important and what’s least important; just kind of learned everything at once.” - Program Specialist on initial training There is a lot of information to understand and digest.
31. USDA SNAP QC Training | APR 8 30 Phase 1 // Discovery Key Insight #3: Content that is accessible, structured, and intuitive allows Program Specialists to self-direct their SNAP QC training. This empowers them and builds confidence, and also creates organizational effectiveness.
32. USDA SNAP QC Training | APR 8 31 Phase 1 // Discovery Key Insight #3: SUPPORTING EVIDENCE “I would like to see a standardized manual with sample cases - to see what to look for, what the most common occurrence with the cases are, what are the red flags - a quick check.” - Program Specialist Program Specialists noted: • Other specialists do things differently - some provide more documentation, some provide less; it leads to inconsistencies. • They are notified of policy changes, but not the steps to take for them. Staff will then interpret policy differently. • State QC staff and Federal QC staff are told different things - errors were caused because a state was not informed of a new policy change. The lack of standardization makes staff feel hesitant or frustrated about their work. “Training is like folklore in some ways. People pass on what people heard [on how to do things] and the information gets diluted.” - Program Specialist
33. USDA SNAP QC Training | APR 8 32 Phase 1 // Discovery Key Insight #3: SUPPORTING EVIDENCE “PartnerWeb could be better organized.... [It’s] hard to locate the exact thing you need.” - Program Specialist • Program Specialists would like to see a “Cliffs Notes” version of the 310 Handbook that is written in plain language. • One Program Specialist has photocopied pages of the 310 Handbook at their desk for a quick reference guide. • Program Specialists found it hard to find information on the PartnerWeb. Current resources are difficult to navigate.
34. USDA SNAP QC Training | APR 8 33 Phase 1 // Discovery Key Insight #4: Transparent communication - throughout the organization and specific to policy and process - prevents confusion, second guessing, and the “rumor mill” and promotes consistency, role clarity, and shared understanding.
35. USDA SNAP QC Training | APR 8 34 Phase 1 // Discovery Key Insight #4: SUPPORTING EVIDENCE • QC Coordinators admitted to not using it in months and even years. • It takes anywhere from one week to one year to get a response. • They are less inclined to use it since the answers sound “snarky” or berating. Inquiry box reputation • QC Coordinators felt like they’re on their own for training staff. • Regions would like to be notified in advance of policy changes because some don’t learn about them until red flags come up when going through reviews. • Program Specialists think it would be beneficial to see how other regions handle policy changes or know if there are any general trends to be aware of. Staff feel isolated from Headquarters and other regions.
36. USDA SNAP QC Training | APR 8 35 Phase 1 // Discovery Key Insight #5: It’s critical to establish a governance plan for the training framework and content - to evaluate and improve it on an ongoing basis - to ensure its continuous success.
37. USDA SNAP QC Training | APR 8 36 Phase 1 // Discovery Key Insight #5: SUPPORTING EVIDENCE QC Coordinators and QC Trainers mentioned: • There’s no way to evaluate training until it’s too late - the effectiveness of a specialist is showing up in the quality of work and the questions being asked. • Improving the training is difficult because a specialist won’t directly mention how effective or poor the training is. • There is no assessment to see whether or not someone knows the material. The lack of evaluation makes it impossible to measure the success of training solutions. • One Program Specialist was given the 2008 version of the 310 Handbook, but was not aware it was out of date until receiving an electronic version. • One Program Specialist used to pull state profiles from the National Office’s drive, but received a new computer that isn’t connected to the drive. This individual finds the information on the regional drive now. • Arbitrations, policy memos, clarifications, and other information are distributed via email. Information is scattered and not version controlled well.
38. USDA SNAP QC Training | APR 8 37 Phase 1 // Discovery Observation #1: Certain personality traits lend themselves to more successful Program Specialists. “Self-starter” SUPPORTING EVIDENCE “Organized” “Team player” “Go-getter” “Meticulous” “Detail oriented”
39. USDA SNAP QC Training | APR 8 38 Phase 1 // Discovery Observation #2: A dual-monitor set up is helpful for Program Specialists to do their work. SUPPORTING EVIDENCE • Those with dual monitors said it was helpful to pull up the case documents on one screen and then have the review form on the other screen. • Those working on laptops with no desktop monitor wished they had bigger laptop monitors so it would be easier to access all the documents and windows for their case review. • Those with laptops and desktop monitors said it was nice having the real estate of a big monitor to access all their work.
40. USDA SNAP QC Training | APR 8 39 Phase 1 // Discovery Observation #3: The Program Specialist job isn’t readily appealing. • The tasks can be very complicated. • The work is tedious and repetitive. • Regional QC is viewed as a “dumping ground,” where staff is placed here after being passed around from other departments. • There’s a high turnover rate for this job. “Once I train someone and they are successful, they leave QC.” • There is very little room for advancement in this position and Program Specialists are often at a lower pay grade than other types of reviewers. “It’s hard to convince people that this [job] is interesting.” - QC Coordinator SUPPORTING EVIDENCE
41. USDA SNAP QC Training | APR 8 40 Phase 1 // Discovery Observation #4: Budgetary and travel constraints have hindered training efforts. • The last national training was in Dallas about five years ago. • There are no dedicated trainers. • Currently there is no basic SNAP QC training or refresher training from the National Office. SUPPORTING EVIDENCE
42. USDA SNAP QC Training | APR 8 41 Phase 1 // Discovery Concept Preview
43. USDA SNAP QC Training | APR 8 42 Phase 1 // Discovery Concept Preview STRATEGY LEVEL OF EFFORT more less traditional (low risk) innovative (high risk)
44. USDA SNAP QC Training | APR 8 42 Phase 1 // Discovery Concept Preview STRATEGY LEVEL OF EFFORT more less traditional (low risk) innovative (high risk)
45. USDA SNAP QC Training | APR 8 43 Phase 1 // Discovery Concept Preview
46. USDA SNAP QC Training | APR 8 44 Phase 1 // Discovery Next Steps • Brainstorm solutions • Identify resources to provide subject matter expertise • Identify potential opportunities as well as users to test the prototypes
47. USDA SNAP QC Training | APR 8 45 Appendix // Discovery
48. USDA SNAP QC Training | APR 8 46 Phase 1 // Discovery QC Trainer Focus Group Questions Purpose We used some of these questions to start the discussion and asked additional, follow-up questions based on the participants’ responses. • How would you define the QC training problem we are facing? • What are the biggest challenges of the current QC training? • What does training look like? • What positive feedback have you received from the current training? • What do you get the most questions about? • What are the trainee’s biggest challenges? • What are their biggest complaints? • What if we removed all the training challenges? What would the training scenario look like? • How would you change the training? Questions The purpose of this focus group was to understand QC training from the perspective of the QC Trainers. The insight we gained from the group influenced questions to ask the QC Coordinators and Program Specialists and will help shape possible training solutions.
49. USDA SNAP QC Training | APR 8 47 Phase 1 // Discovery QC Coordinator Focus Group Questions Purpose We used some of these questions to start the discussion and asked additional, follow-up questions based on the participants’ responses. • What does QC training look like at the regional level? • Please walk through a typical training scenario. • Is this the same of all the regions? • Who on the staff receives training? • When does training occur and how often? • How long does it take to train someone? • What are types of content are included in the training? • What training materials (electronic, books, handouts, presentations, etc.) are used? • What positive feedback have you received from the current training? Where is it successful? • What do you get the most questions about? • What are their biggest complaints? • How do you know whether the training was effective? • Are there any gaps in the current training program? Questions The purpose of this particular focus group was to gain an understanding of the training at the regional level. We were interested in learning what is and is not working. The insight we gained from the group influenced user interview questions to ask the Program Specialists and will help shape possible training solutions.
50. USDA SNAP QC Training | APR 8 Phase 1 // Discovery Program Specialist Interview Questions Purpose • How long have you been with SNAP QC? • Did you have a prior SNAP or QC background? • Are you located at the regional office or elsewhere? • What kinds of technology do you use at work and at home? • What does a typical day look like for you? • What tools and resources do you use to do your job? • What competes for your time? Questions The purpose of these interviews was to gain an understanding of the training at the regional level from the perspective of the Program Specialist. The insight we gained from these interviews will help influence possible training solutions. • When you started, how were you trained? (buddy/mentor system, official training, etc.) • How long were you given training for? • How was the training? • Did you feel the training you received was effective in doing your work? • Based on your training, how long would you say it took for you to be confident in doing your job? • Is there recurring (refresher or new policy) training? • What do you like about the current training? • What don’t you like about the current training? We used some of these questions to start the discussion and asked additional, follow-up questions based on their responses. • If you have questions, how do you get them answered? • Do you go to your team lead, QC coordinator, or peers for help? • What are your biggest complaints and challenges for doing your work? • How do you think QC Training could be improved? • What training would be most beneficial to you? • What tools, resources, (and people) would you like to see to make you more empowered at your job? 48