– GETTING RESULTS what did we find out? 4 – LESSONS LEARNED ...and what does it mean? AGENDA SAKAI MARKET SHARE MAKE IT COUNT MAIN METHOD LESSONS LEARNED what determines market share? how do we get good data? GETTING RESULTS
one vote: enrollment rates are not reflected since enrollments vary, sample data can't be generalized commercial LMS providers won't share client lists; open source registries are poorly managed THE ISSUES MAKE IT COUNT MAIN METHOD LESSONS LEARNED MAKE IT COUNT GETTING RESULTS 3 – ABSENT PROVIDER DATA SAKAI MARKET SHARE
as calculated by The CCP, based on a one system per institution measure for all higher ed respondents, including Community Colleges. COURSE MENU MAKE IT COUNT THE COUNT GETTING RESULTS SAKAI MARKET SHARE
as calculated by The CCP, based on a one system per institution measure for all higher ed respondents, with Community Colleges removed. COURSE MENU MAKE IT COUNT THE COUNT GETTING RESULTS SAKAI MARKET SHARE
an average response rate of 28.4%....making it impossible to generalize about student usage across colleges and universities with widely varying enrollment rates. COURSE MENU MAKE IT COUNT RESPONDENTS GETTING RESULTS SAKAI MARKET SHARE
control over implementation and client downloads, so cannot provide accurate usage statistics. COURSE MENU MAKE IT COUNT LMS PROVIDERS SAKAI MARKET SHARE
users, learners should be tallied to determine market share. By completion of their fourth year in college, the ratio of students who will have used a school's primary LMS approaches 100%. COURSE MENU MAKE IT COUNT THE USERS GETTING RESULTS SAKAI MARKET SHARE
METHOD select schools and get enrollment numbers how to count 100% of the student population of interest? 3 – ESTABLISH A PROCESS the plan, pitfalls, and (striving for) perfection THE STEPS MAKE IT COUNT MAIN METHOD LESSONS LEARNED MAKE IT COUNT MAIN METHOD GETTING RESULTS SAKAI MARKET SHARE
using the U.S. Department of Education's Institute of Education Sciences database... COURSE MENU MAKE IT COUNT IPEDS GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE
criteria: • 4-year+ not-for- profit • degree granting • based in the U.S. • 1,000 or more students • have a physical campus COURSE MENU MAKE IT COUNT IPEDS GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE
lot of work to do: how to determine the LMS in use by over 1,500 schools? COURSE MENU MAKE IT COUNT IPEDS RESULTS GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE
a large data set requiring repetitive search operations and intelligent evaluation of results, we had an ideal use case for Amazon's Mechanical Turk. COURSE MENU MAKE IT COUNT MECHANICAL TURK GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE
when the instructions are standardized, so we had to establish a web search that would return optimal results. We tested the term 'learning management system' and related terms suggested by Google and other sources... COURSE MENU MAKE IT COUNT THE SEARCH GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE
IT COUNT THE SEARCH GETTING RESULTS MAIN METHOD 1 – MTurk TESTING 2 – WORKER QUALIFICATION • first run: 100 items, 2 Master Classification Workers each (200 HITs) • second run: 50 items, 3 Master Workers each (150 HITs) • third run: 400 items, 3 Master Workers each (1200 HITs) • goal: create workforce with 95% accuracy or greater • reality: standards had to be set at 85% accuracy to generate a large enough worker base SAKAI MARKET SHARE
IT COUNT GETTING RESULTS MAIN METHOD • use 3 workers per item; verify LMS choice for all worker disagreements • do 5% audit on all worker agreements • compare final LMS list to provider information found on web 4 – DATA VERIFICATION • 3 Custom Qualification Workers per item • remaining institution list split in half and run sequentially in two sets 3 – DATA COLLECTION THE SEARCH SAKAI MARKET SHARE
analytics for completion velocity, average hourly pay rate, and percent HITs submitted. COURSE MENU MAKE IT COUNT MTurk TESTING GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE
workers who attempted our HITs, only 10 performed well enough to meet our qualification standards. COURSE MENU MAKE IT COUNT WORKER QUALIFICATION GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE
a 95% accuracy rate for worker agreements. • Workers: .95 x 3621 = 3440 • In-house search: 1.0 x 1263 = 1263 • Accuracy: 4703/4884 = 96.29% We are confident that our results are not off more than +/- 5% COURSE MENU MAKE IT COUNT DATA VERIFICATION GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE
share based on 2010-2011 1-year headcount how do our results differ from the CCS count? 3 – THE SPREADSHEET our data by LMS THE RESULTS MAKE IT COUNT MAIN METHOD LESSONS LEARNED MAKE IT COUNT MAIN METHOD GETTING RESULTS SAKAI MARKET SHARE
share since founding in 2008, with many recent migrations 8.3% market share since deployment in 2005, with few recent migrations and many previous Sakai installations having been replaced by other systems 3 – OPEN SOURCE 26.9% of primary LMSs by enrollment are open source, less than the 33.6% reported by the CCS count; Blackboard is often the primary LMS at schools with smaller parallel or test installations of other systems CONCLUSIONS MAKE IT COUNT MAIN METHOD MAKE IT COUNT MAIN METHOD GETTING RESULTS LESSONS LEARNED SAKAI MARKET SHARE
OF DATA Include analysis as to statistical significance of comparison survey This will allow schools to correct any errors as well as report a change in LMS 3 – SHARE DATA FUTURE WORK MAKE IT COUNT MAIN METHOD MAKE IT COUNT MAIN METHOD GETTING RESULTS LESSONS LEARNED SAKAI MARKET SHARE