Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Server & Survey Data to Understand Users at the University of Florida

Using Server & Survey Data to Understand Users at the University of Florida

Analytics was used to evaluate Sakai usage at UF.

Jose Lorenzo Silva-Lugo

June 04, 2013
Tweet

Other Decks in Technology

Transcript

  1. Using Server & Survey Data to Understand Users in France

    and Florida University of Florida Jose L. Silva-Lugo, Douglas Johnson, & John Boekenoogen 1
  2. Applying Analytics to evaluate Sakai Usage at the University of

    Florida 3 Analytics Use of Data Statistical Analysis Exploratory Models Predictive Models Research Questions or Hypotheses Create & Organize Knowledge Gain Insight & Take Actions on Complex Issues
  3. Research Questions What were the percentages of courses and instructors

    using Sakai per college/year? What were the colleges with percentages less than 50% used of Sakai? Were the percentages of courses and instructors per semester increasing through time? 4
  4. Server Data: Methods • Data was retrieved from the network:

    – Registrar Office: IBM DB2 database – Middleware: MySQL database • Language used to query the database: – A PHP script queried both database (< 5 seconds). – The same script merged and compared the results (5- 20 seconds). – Results were accurate and error free. 6
  5. • Statistical Analysis: – One-sample binomial test for proportion comparison:

    • Between years (2010, 2011, 2012) for courses and instructors per college • Between semesters for courses and instructors – Categorical regression for making predictions – Alpha level = 0.05 – Two-tailed test – IBM SPSS v.21 7 Server Data: Methods
  6. 8 Percentages of Courses Developed in Sakai per College per

    Year COLLEGES 2010 2011 2012 Total DESIGN, CONSTRUCTION AND PLANNING 29.64 73.56 100.00 52.98 HEALTH AND HUMAN PERFORMANCE 23.76 66.67 71.65 46.28 JOURNALISM AND COMMUNICATIONS 18.33 66.09 62.50 41.33 BUSINESS ADMINISTRATION 13.64 62.17 67.17 38.06 PHARMACY 17.21 50.68 76.30 35.56 PUBLIC HEALTH & HEALTH PROFESSIONS 14.22 53.49 52.35 33.88 NURSING 0.00 67.69 69.85 33.46 ENGINEERING 19.82 39.49 61.48 29.77 LIBERAL ARTS AND SCIENCES 14.50 43.07 43.51 28.15 AGRICULTURAL AND LIFE SCIENCES 13.30 34.79 42.32 24.19 MEDICINE 17.23 22.35 15.08 19.75 VETERINARY MEDICINE 5.02 19.32 27.65 12.24 FINE ARTS 5.48 17.26 16.82 11.41 EDUCATION 2.86 8.16 9.02 5.57 CROSS-COLLEGE 11.11 0.00 0.00 4.82 LEVIN COLLEGE OF LAW 0.00 2.00 5.85 1.49 DENTISTRY 0.41 1.79 3.33 1.15 Total 12.77 35.42 41.28 24.25
  7. 10

  8. 11 Courses: Proportion Comparison between Semesters Semester π1 π2 Π

    σ π1- π2 |Z| Zα P-value Result Spring 2010 - Summer 2010 0.00 0.04 0.0166 0.0031 13.4318 1.96 < 0.0001 Significant Summer 2010 - Fall 2010 0.04 0.30 0.2081 0.0099 26.1197 1.96 < 0.0001 Significant Fall 2010 - Spring 2011 0.30 0.37 0.3344 0.0099 6.4253 1.96 < 0.0001 Significant Spring 2011 - Summer 2011 0.37 0.29 0.3363 0.0111 6.7244 1.96 < 0.0001 Significant Summer 2011 - Fall 2011 0.29 0.38 0.3469 0.0111 8.2315 1.96 < 0.0001 Significant Fall 2011 - Spring 2012 0.38 0.41 0.3983 0.0103 2.8859 1.96 0.0039 Significant Spring 2012 -Summer 2012 0.41 0.36 0.3962 0.0126 4.0575 1.96 < 0.0001 Significant Summer 2012 - Fall 2012 0.36 0.44 0.4126 0.0127 5.9358 1.96 < 0.0001 Significant Spring 2012 – Fall 2012 041 0.44 0.4252 0.0103 2.3206 1.96 0.0203 Significant
  9. 12

  10. 14 COLLEGES 2010 2011 2012 Total HEALTH AND HUMAN PERFORMANCE

    25.45 65.54 70.75 47.43 ENGINEERING 25.25 55.01 52.04 40.76 JOURNALISM AND COMMUNICATIONS 19.58 56.60 55.28 39.07 DESIGN, CONSTRUCTION, AND PLANNING 21.10 54.66 65.20 38.55 PUBLIC HEALTH & HEALTH PROFESSIONS 15.14 36.49 34.78 26.33 AGRICULTURAL AND LIFE SCIENCES 14.66 36.95 39.46 26.06 LIBERAL ARTS AND SCIENCES 15.43 33.54 37.08 25.09 BUSINESS ADMINISTRATION 19.75 29.95 53.04 24.91 FINE ARTS 8.05 21.40 20.97 14.92 CROSS-COLLEGE 27.78 0.00 0.00 13.70 EDUCATION 3.60 11.53 12.94 7.56 MEDICINE 4.94 9.06 9.14 7.51 VETERINARY MEDICINE 5.47 6.54 9.98 6.17 PHARMACY 1.83 3.39 3.51 2.75 LEVIN COLLEGE OF LAW 0.00 3.03 3.37 1.63 NURSING 0.00 2.93 2.12 1.58 DENTISTRY 0.76 0.70 1.10 0.71 Total 13.52 28.63 30.97 21.75 Percentages of Instructors Teaching in Sakai per College per Year
  11. 16

  12. 17 Instructors: Proportion Comparison between Semesters Semester π1 π2 Π

    σ π1- π2 |Z| Zα P-value Result Spring 2010 - Summer 2010 0.00 0.07 0.0235 0.0044 14.5203 1.96 < 0.0001 Significant Summer 2010 - Fall 2010 0.07 0.29 0.2174 0.0117 19.0266 1.96 < 0.0001 Significant Fall 2010 - Spring 2011 0.29 0.32 0.3065 0.0103 3.5117 1.96 0.0004 Significant Spring 2011 - Summer 2011 0.32 0.23 0.2881 0.0115 8.4099 1.96 < 0.0001 Significant Summer 2011 - Fall 2011 0.23 0.28 0.2636 0.0111 5.1051 1.96 < 0.0001 Significant Fall 2011 - Spring 2012 0.28 0.29 0.2883 0.0096 0.9968 1.96 0.3189 Not Significant Spring 2012 - Summer 2012 0.29 0.29 0.2936 0.0117 0.1203 1.96 0.9042 Not Significant Summer 2012 - Fall 2012 0.29 0.33 0.3204 0.0119 3.2948 1.96 0.0010 Significant Fall 2012 – Spring 2013 0.33 0.34 0.3373 0.0100 0.6843 1.96 0.4938 Not Significant Spring 2011 - Spring 2013 0.32 0.34 0.3326 0.0102 1.6455 1.96 0.0999 Not Significant
  13. 18

  14. Actions Survey to know faculties’ satisfaction toward Sakai Ph.D. dissertation

    to find out faculties’ barriers toward the use of Sakai 20 % Courses/year < 50% % Instructors/year < 50% % Instructors/semester not increasing through time Predictions had low expectations
  15. Survey Data: Methods • A survey was developed to address

    mainly faculties’ satisfaction toward the use of Sakai. • The survey was structured as follows: – 15 questions about users’ satisfaction toward: • Sakai • Customer service – 13 questions about: • Demographic • Skills/knowledge in online/blended courses • Attitudes toward online/blended courses • Assistant and training 22
  16. • Research question: – What were the percentages of faculty

    satisfied with Sakai regarding overall, availability, quickness, easy to use, and reliability? • Statistical Analyses: – Power Sample 3.0: Power Analysis – IBM SPSS v.21, α = 0.05, and two-tails: • Proportion comparison (one-sample binomial test) 23 Research Question & Statistical Analyses
  17. 24 Power Analysis for Proportions Power Alpha Proportion 1 Proportion

    2 Effect Size Sample Size 80% 0.10 0.55 0.50 0.05 620 0.65 0.50 0.15 67 0.75 0.50 0.25 23 80% 0.05 0.55 0.50 0.05 785 0.65 0.50 0.15 85 0.75 0.50 0.25 27 80% 0.01 0.55 0.50 0.05 1165 0.65 0.50 0.15 126 0.75 0.50 0.25 43
  18. For the selected medium effect size (0.15), we needed a

    sample size at least of 85 surveys. Total sample size = 232 surveys The sample size was large enough to reject the Ho with a Power = 80% for the proportion analysis We could make inference from the sample to the population 25 Faculties’ Response to Survey
  19. 26 How satisfied/dissatisfied are you regarding Sakai “overall”? Faculties’ Satisfaction

    All Dataset # Answer Response % 1 Very dissatisfied 14 6% 2 Dissatisfied 20 9% 3 Somewhat dissatisfied 35 15% 4 Neutral 18 8% 5 Somewhat satisfied 63 27% 6 Satisfied 66 28% 7 Very satisfied 16 7% Total 232 100%
  20. Faculties’ Satisfaction All Dataset Question Satisfied Dissatisfied |Z| P-value Results

    Overall 62% 30% 7.31 <0.0001 Significant Availability 80% 9% 15.32 <0.0001 Significant Quickness 66% 24% 9.02 <0.0001 Significant Easy to Use 51% 36% 3.30 0.001 Significant Reliability 60% 22% 8.27 <0.0001 Significant 27 Percentage comparison between satisfied and dissatisfied faculties regarding 5 criteria of satisfaction
  21. 28 How satisfied/dissatisfied are you regarding Sakai “overall”? Faculties’ Satisfaction

    All Dataset # Answer Response % 1 Very dissatisfied 14 6% 2 Dissatisfied 20 9% 3 Somewhat dissatisfied 35 15% 4 Neutral 18 8% 5 Somewhat satisfied 63 27% 6 Satisfied 66 28% 7 Very satisfied 16 7% Total 232 100%
  22. Faculties’ Satisfaction Dissatisfied Group Question Satisfied Dissatisfied |Z| P-value Results

    Availability 43% 28% 1.95 0.0504 Not Significant Quickness 22% 61% 4.67 <0.0001 Significant Easy to Use 10% 78% 8.06 <0.0001 Significant Reliability 20% 52% 3.90 <0.0001 Significant 29 Percentage comparison between satisfied and dissatisfied faculties regarding 4 criteria of satisfaction
  23. The percentages of courses and instructors per year were <

    50%, but significantly increasing through time. The percentages of courses per semester are significantly increasing through time. The percentage of instructors per semester did not significantly increase from spring 2011 to spring 2013. We have identified 8 colleges with percentages < 50% used of Sakai. 33 Server Data Conclusions
  24. Predictions We do not have enough data to make predictions

    for the % of courses and instructors per year. The % of courses per semester might increase by 4%. The % of instructors per semester has a logistic distribution with carrying capacity fluctuating between 28% and 35%. 34
  25. Actions Survey to know faculties’ satisfaction toward Sakai. Ph.D. dissertation

    to find out faculties’ barriers toward the use of Sakai. 35
  26. Survey Data Conclusions The majority of faculties were satisfied with

    Sakai regarding Overall (63%) Availability (80%) Quickness (66%) Easy to use (51%) Reliability (60%) The group of dissatisfied faculties did not like Sakai because it was Slow (61%) Uneasy (78%) Unreliable (52%) 36
  27. 38