Lessons Learnt on a Lean Startup Journey

A presentation on using Lean Startup to create Open Universities Australia's Open2Study MOOCs platform. Created by Elabor8 consultants James Pulling and Cecilia Waters and Open Universities Australia Product Manager Takis Diakoumis and Agile Lead Erik van Eekelen.

1. 1 Lessons Learnt on a Lean Start Up Journey Using Lean Startup methodology to create Open Universities Australia's Open2Study MOOCs platform Takis Diakoumis, Erik Van Eekelen, Cecilia Waters, James Pulling
2. 2 Outline 1.  Introductions 2.  What on earth is a MOOC? 3.  What is Open2Study? 4.  The Lean Startup methodology 5.  Lean Startup Lessons Learned 6.  Questions
3. 3 1. Introductions
4. 4 Trailer video removed for size. Video can be found here: https://www.youtube.com/watch?v=Bw8HkjGQb3U
5. 5 2. What on earth is a MOOC?
6. 6 What is a MOOC?
7. 7 3. What is Open2Study?
8. 8 This thing is big!
10. 10 The Vision 1.  The first Australian MOOC platform 2.  Unlike ALL the other MOOC platforms 3.  Offer a more Social & Active Learning environment 4.  The first to offer a combined University Level Education & Professional Vocational courses 5.  Offer courses that are fit for the purpose for the digital space
11. 11 Module  1   Module  2   Module  3   Module  4   10  topics  (with  quiz)   1  assessment   1  video  
12. 12 Module  1   Module  2   Module  3   Module  4   10  topics  (with  quiz)   1  assessment   1  video  
13. 13 How it all happened 5th Nov 2012 3 weeks later Dec 2012 Mar 21st 2013 •  CEO raised need for MOOC •  Team formed for Business Case Definition •  Product Definition •  Elevator pitch •  Personas •  Key user journeys •  Competitor analysis Feb 2013
14. 14 How it all happened •  Business case endorsed and partial funding approved •  Initial project team created 5th Nov 2012 3 weeks later Dec 2012 Mar 21st 2013 Feb 2013
15. 15 How it all happened •  Project kicked off •  Streams and teams scaled up •  Use case models •  UX vision and wireframes •  Instructors engaged for production 5th Nov 2012 3 weeks later Dec 2012 Mar 21st 2013 Feb 2013
16. 16 How it all happened •  Business case approved •  Regression tested •  Platform user tested 5th Nov 2012 3 weeks later Dec 2012 Mar 21st 2013 Feb 2013
17. 17 How it all happened Go live! 5th Nov 2012 3 weeks later Dec 2012 Mar 21st 2013 Feb 2013
18. 18 How is it all going? 8361 Initial registrations 112,370 Enrolments 96% user satisfaction 9.4% industry average 26% our average 57,731 Current registrations
19. 19 4. The Lean Start Up methodology
20. 20 Definition •  "Lean Startup" is a method for developing businesses and products first proposed in 2011 by Eric Ries •  The Lean Startup by Eric Ries •  Derived from Lean Manufacturing •  Identifying which activities are value-creating and which are wasteful •  The application of Lean thinking to the process of innovation
21. 21 ‘A startup is a human institution designed to deliver a new product or service under conditions of extreme uncertainty.’ -- Eric Ries --
22. 22 Minimum Viable Product The question is not "Can this product be built?" Instead, the question is: "Should this product be built?"
24. 24 2007 2008 The Point -  Social platform for groups to solve problems -  Very modest traction -  Customers wanted to save money buying in a group 2010 Groupon -  One deal a day -  Deal doesn’t start until enough people buy the voucher -  Vouchers ideally for local businesses Groupon -  Valued at $1.35 billion in April 2010 Pivot example: Groupon’s first pivot
25. 25 5. Lean Startup Lessons learned
26. 26 User testing FAIL FAST Fail often MVP must be TRULY MINIMAL MEASURE ALL THE TIME Continually tune EVERYONE is an entrepreneur Actionable metrics Take the time to Learn from measurements Rapid releases, Agile & Lean practices Flat mgmt structures Working in a Lean start up presented many opportunities to Learn… Stop wasting people’s time
27. 27 Lesson 1: Focus the build •  One of our main drivers was ‘Time to Market’ – which is important to any Lean Start Up •  This driver made us start building without full or confirmed requirements •  Was this TOO FAST?!? •  More speed and less haste
28. 28 At first we only had: -  Business case -  UC: Discovered -  Wireframes -  Some visual designs We quickly specified: -  UC: bulleted outlines -  Independent leadership (workshop/ decisions) -  Prioritised to create MVP We then: -  UC’s broken down to User Stories -  Prioritised Backlog created (focused on getting priority requirements to the fine grain level) Lesson 1: Focus the build •  We wanted to keep going fast so we had to focus development •  Stop wasting people’s time and only do the things you have to do
29. 29 Lesson 2: Make MVP, really MVP •  System and feature screen test supporting fail fast, fail often in order to learn and adapt quickly •  Validate or invalidate your assumptions •  First release: -  No classroom -  No way to begin or complete any course -  Just enough to register and enrol users •  We could have gone live with less
30. 30 Lesson 2: Make MVP, really MVP
31. 31 LEARN: Missing assessments Want to keep moving MEASURE: Student feedback BUILD: Assessment model: Open Thursday - Sunday Lesson 3: Tune as often and as quickly as you can
32. 32 Extension email had to be sent BUILD 2: Assessment model: Open Monday – Sunday LEARN: Improved feedback Spike after extension email MEASURE: Student feedback Completion metrics Lesson 3: Tune as often and as quickly as you can
33. 33 BUILD 3: Assessment model: Open 1st day of week – Close Reminder email LEARN: Keep learning and pivoting MEASURE: Student feedback Completion metrics Lesson 3: Tune as often and as quickly as you can
34. 34 Lesson 4: Everyone is an entrepreneur Luckily we had a number of budding entrepreneurs of our own in the team
35. 35 Lesson 4: Everyone is an entrepreneur
36. 36 Lesson 4: Everyone is an entrepreneur
37. 37 Lesson 4: Everyone is an entrepreneur •  Extended enrolment period •  Enrolments closed on first day of course – no one saw this as an issue as the next course would start in 4 weeks
38. 38 Lesson 4: Everyone is an entrepreneur •  Extended enrollment period •  Inviting friends to the platform was a key feature •  Extend enrollment period to allow more students to join
39. 39 Lesson 5: Build, and build the ability to measure •  Customer surveys showed students wanted more ways to collaborate •  A new connections feature was built to provide scaffolding to connection to connection features •  We had performed workshops and analysis and were well into the next sprint when it was flagged •  ‘Did we have any user stories written to measure user engagement with our new Connections feature?’
40. 40 Lesson 5: Build, and build the ability to measure •  Connections shipped & we had to measure manually •  Results showed uptake was low •  Build + build ability to Measure in order to Learn •  Lesson learnt – all features must have measure stories •  Stakeholders engaged to learn from these measurements
41. 41 Lesson 6: Actionable insights over vanity metrics Vanity metric: •  Business case KPI of 100K enrolments in 2013 •  Vanity metrics make you look good •  Needed to understand more about our students Actionable Accessible Auditable metrics: •  Required to hold entrepreneurs accountable to real metrics, not vanity metrics
42. 42 Lesson 6: Actionable insights over vanity metrics •  LMS data was manipulated in excel pivot tables •  New project to develop and deploy an open source BI Tool Pentaho, on a read only RDS •  Slice & dice data from cohort to student level •  Actionable insights were easily accessed, eg: for course Y the average mark for module 1 assessment was 15%
43. 43 Elementary, my dear Watson Our deductions again: 1.  Focus the build 2.  Make MVP really MVP 3.  Tune as often and as quickly as you can 4.  Everyone is an entrepreneur 5.  Always build, and build the ability to measure 6.  Actionable insights over vanity metrics
44. 44 Open2StudyEduca@on   Open2Study   @open2study   Open2Study   @elabor8   Elabor8   6. Questions?