Phoenix Data Conference 2014 - Nitin Mulimani

Adaptive Interventions @ UoP

Transcript

1. Adaptive Interventions for non-traditional students
2. Who  is  Apollo?   •  Founded in 1973 •  Publicly listed on NASDAQ (Ticker: APOL) •  One of the world’s largest private education providers © 2014 Apollo Education Group, Inc. All Rights Reserved.
3. Ni,n  Mulimani   u  Engineering  and  Product   responsibili,es   u  Developing  Data  Driven  products   u  Passionate  about  Agile   methodologies   u  Love  to  watch  sports  and  movies  –   based  on  allowance  earned  from   family!      
4. Apollo  Technology   Key  objec)ves:   –  Deliver  deeply  personalized  experience  and  targeted  interven,ons  to  help  students   –  Create  insights  into  the  learning  challenges  students  face  and  how  interac,ons  with  their   learning  environment  affect  them   –  Create  insights  into  the  course  and  program  content  efficacy     Significant  progress:   –  State-­‐of-­‐art  data  plaKorm  in  place  (hybrid  data  warehouse  and  big  data  plaKorm)     –  Apollo  Classroom  connected  to  the  big  data  plaKorm       –  Data-­‐Driven  Applica,ons,  such  as  risk-­‐alerts  and  personalized  study  guide,  rolled  out    
5. Apollo  Data  PlaKorm   •  We  created  our  data  pla:orm  to:   –  Enable  large-­‐scale  learning  analy,cs     –  Build  predic,ve  models  by  analyzing  student  behavior  and  measuring   learning     •  Strengths  of  this  approach   –  Scalability  at  commodity  compu,ng  prices   –  Flexibility  to  handle  structured  and  unstructured  data   –  Reusable  infrastructure  for  all  future  applica,ons  
6. Apollo  Data  PlaKorm  
7. Hadoop  Cluster  details   Current CDH3u5 Prod - 54 nodes physical hardware QA - 8 nodes physical hardware Dev - 12 nodes vm In Pipeline CDH5 Prod - 30 nodes physical QA - 30 nodes physical Dev - 13 nodes physical
8. Adap,ve  Interven,ons  Architecture  
9. Ecosystem  
10. Data Driven Products and Experiments
11. 11 The problem: Who  is  at-­‐risk  now?   How  can  we  intervene  as  quickly  as  possible?   How  should  we  intervene?     Business Case
12. Intervene  effec,vely   Low   aVendance   Struggle  with  a   learning  objec,ve   Direct  links  to  instruc,onal   resource   (textbook  readings,  videos,   other  media)   Contact  from   an  academic   counselor   Synchronous  resources:   -­‐  Faculty-­‐led  sessions     -­‐  Live  1:1  tutoring   -­‐  Live  labs   Struggle  with  a   learning  objec,ve   Counselor   ,p  about  a   job  lead   Life   circumstance   Quiz   results   Course   ac,vity   Phoenix  Career   Services  inquiry   Assignment   grade   Journey  toward  a  degree   Example   Events   Risks   Interven,ons  
13. At-­‐Risk  Alerts   •  Data  analy,cs  and  an  intelligent   inference  engine  allow  us  to  build,   test  and  deploy  models  predic,ng  a   student’s  con,nued  aVendance  at   the  university,  the  key  reten,on   metric.     •  A  variety  of  in-­‐course  student   behaviors  contribute  to  the   predic,ve  model.   Data collected for associates, bachelors and graduate students Program Progress Number of courses passed in program Number of courses passed prior to program Level of activity Number of weeks of missed attendance Missed attendance in previous week Number of days since last post Number of days submitting discussion posts this week Total number of posts in the current week No assignments submitted in the most recent week Missed assignment, weighted by assignment size Number of days content was accessed in most recent week Mastery Current course points earned divided by attempted points Student  data  can  power  high-­‐value   predic,ve  analy,cs  that  result  in   improved  student  outcomes.  
14. At-­‐Risk  Alerts:  Ac,onable  Item   •  Data-­‐driven  Risk  Alerts  allow   student  advisors  to  quickly   intervene  with  students  at   risk  of  withdrawing.  
15. Architecture  for  At-­‐Risk  Alerts  
16. 1.  Knowledge  Checks  and     2.  Personalized  Study  Guides   Personalized  concept  level   feedback  to  students  
17. Misconcep,on  Analysis     •  Iden,fy  common   misconcep,ons  about  a   ques,on     •  Improve  our  curriculum    
18. Classroom  discussion:     Biography  Mining   •  We  find  all  of  the  classroom  discussions  are  about   introduc,ons.  These  are  essen,ally  biographical   entries.     •  We  used  some  natural  language  processing  to   extract  key  terms  about  work,  home,  hobbies  and   school,  etc.   •  From  this  we  built  a  “social”  model  of  the  student   based  on  their  self  proclaimed  social  aVributes   •  Next  steps   –  Segment  the  student  popula,on  according   to  a  social  profile   –  Do  an  auto-­‐cohor,ng  experiments   –  Suggest  “students  who  may  have  similar   interests”    
19. Why do we leverage a graph database?
20. Learnings: Be specific Laser focus on the business case that you intend to solve
21. Learnings: Early Feedback •  Obtained key business sponsorship •  Ran a pilot for 6 months •  Actionable item showed key data points •  Leveraged team of early adopters to become product promoters
22. Learnings: Recommend or not Currently student advisor used their judgement to decide on specific intervention As Next Step venture into prescriptive analytics One Challenge involves culture change for frontline staff One Baby Step try thumbs-up and thumbs- down
23. Learnings: Collaborators Good working relationships with application owners (business and tech) will take you a long way to get product adopted!
24. Learnings: Engineers, Scientists and Developers on same team Data Engineers •  Data Aggregation skills - Hadoop, Warehouse •  Business rules behind data •  Ensure High Data Quality Data Scientist •  Build Features •  Test Models Integration Engineers •  Web Application Developers Manager •  Remove Roadblocks •  Strong business understanding
25. Closing   “We  capture  fine  grained  data  through  our   systems  and  leverage  our  data  pla:orm  to   develop  products  and  services  that  help  our   students  achieve  be?er  outcomes.”