Data Mining Accessibility

Everyone agrees the best approach to accessibility issues is to avoid them in the first place. What if we could predict problems before they happen? Tenon wants to find out.

Everyone agrees the best approach to accessibility issues is to avoid them in the first place. What if we could predict problems before they happen? Tenon wants to find out. Informed buying decisions—such as researching a product’s accessibility before buying the product or adopting a technology—is the true path to saving money and time on auditing and remediating issues after the system has been deployed. Using a variety of data sources, it is possible to weigh how accessible a site is while also identifying which technologies it uses. With such approaches, Tenon is analyzing the accessibility of the Web and we’re using that data to correlate each technology to its respective error data. The aggregated data can be used to determine the error rates, error densities, and categories of errors. This information can allow organizations to make informed decisions when they choose which technologies to use on their sites. There are many ways that the implementation of technologies can affect the accessibility of the resulting interface. The use of large-scale data gathering will provide the opportunity for the impact of implementation differences from being a barrier to effective analysis. The data that we gathered in this process confirms how pervasive accessibility problems are on the web. It can also show risk patterns that can help organizations learn about potential accessibility problems related to specific technologies before they adopt those technologies. This presentation will discuss this ongoing research effort, including our current findings and our future research plans.

Transcript

1. DATA MINING ACCESSIBILITY ISSUE PATTERNS ON THE MODERN WEB Karl Groves, Tenon.io
2. DATA SOURCES ▸ Tenon.io ▸ Nearly 10,000 users worldwide ▸ In use 24/ 7/ 365 ▸ 477 cities in 61 countries on every continent ▸ 6,000,000 URLs tested ▸ 470,000 unique domains ▸ Tenon.io/research ▸ Ongoing process 24/ 7/ 365 ▸ Alexa 1,000,000 ▸ Customers who’ve consented to automatic stats ▸ 14,600,000 errors logged
3. DATA SOURCES: TENON.IO/RESEARCH Total Technologies 11.109 Total Tech Categories 204 Total Verticals 25 Avg. Alexa Rank 47442 Avg. Quantcast Rank 51350
7. DATA SOURCES: TENON.IO/RESEARCH Avg. Issues Per Page 260 Avg. Density Per Page 47 % Avg. Tests Failing Per Page 27 Avg. WCAG Level A Issues Per Page 276 Percentage of Error-Free Pages 0,03 %
8. But, we know automation isn’t enough! *
9. WE KNOW AUTOMATIC TESTING ISN’T ENOUGH WHAT CAN BE TESTED AND HOW? WCAG Level Auto Semi-Auto (verification req’d) Manual Only Level A 25 % 29 % 46 % Level AA 17 % 41 % 41 % Level AAA 28 % 24 % 53 % Average 18 % 31 % 47 %
10. DATA SOURCES: AUDITS Avg. Components Per Audit 23 Avg. Total Issues Per Audit 139 Avg. Issues Per Component 6
17. TOP 90% OF ISSUES BY CATEGORY ▸ Automated 1. Images and Non-text Content 2. Navigation 3. Language, typography, and content 4. Custom Controls 5. Document Structure ▸ Manual 1. Keyboard Accessibility and Focus Control 2. Document Structure 3. Color 4. Language, typography, and content 5. Role, State, and Accessibility Properties 6. Forms 7. Images and Non-text Content 8. Navigation
18. TOP 10 FAILING SCS (AUTO) 80/20 ▸ 1.1.1 Non-text Content (Level A) ▸ 2.4.4 Link Purpose (In Context) (Level A) ▸ 1.3.1 Info and Relationships (Level A) ▸ 4.1.2 Name, Role, Value (Level A) ▸ 3.3.2 Labels or Instructions (Level A) ▸ 1.4.4 Resize text (Level AA) ▸ 3.2.4 Consistent Identification (Level AA) ▸ 1.4.5 Images of Text (Level AA) ▸ 2.5.5 Target Size (Level AAA) ▸ 1.4.3 Contrast (Minimum) (Level AA)
19. TOP 11 FAILING SCS (MANUAL) 80/20 ▸ 1.3.1 Info and Relationships (Level A) ▸ 4.1.2 Name, Role, Value (Level A) ▸ 2.1.1 Keyboard (Level A) ▸ 1.1.1 Non-text Content (Level A) ▸ 1.3.3 Sensory Characteristics (Level A) ▸ 1.4.3 Contrast (Minimum) (Level AA) ▸ 1.4.4 Resize text (Level AA) ▸ 3.3.2 Labels or Instructions (Level A) ▸ 2.4.6 Headings and Labels (Level AA) ▸ 2.4.7 Focus Visible (Level AA) ▸ 2.4.4 Link Purpose (In Context) (Level A)
20. THE PARETO PRINCIPLE IN ACTION 7 OF THE TOP 10* FAILING SCS ARE SHARED AMONG AUTO AND MANUAL ▸ 1.1.1 Non-text Content (Level A) ▸ 1.3.1 Info and Relationships (Level A) ▸ 1.4.3 Contrast (Minimum) (Level AA) ▸ 1.4.4 Resize text (Level AA) ▸ 2.4.4 Link Purpose (In Context) (Level A) ▸ 3.3.2 Labels or Instructions (Level A) ▸ 4.1.2 Name, Role, Value (Level A)
21. WAIT. WHAT?
22. WHAT WE REALLY KNOW IS THIS: ▸ There is an extremely strong correlation between the automatically created data and the manually created data for: ▸ Issues by Content Type ▸ Issues by WCAG Principle ▸ Issues by WCAG Level ▸ Top failing SCs There are very few instances where audits find something we’ve never seen before.
23. DATA SOURCES: TENON.IO ▸90% of issues come from 20 tests ▸80% of issues come from 12 tests
24. DATA SOURCES: TENON.IO ▸All of the top 50 test failures fit into 5 meta issues: ▸Link quality ▸Text alternatives ▸Insufficient contrast ▸Object labels/ label quality ▸Table/ header structure
25. <!--// Pass //--> <span role="img" aria-label="Foo"></span> <!--// Fail //--> <span role="img">Bar</span> <!--// Passes only on Safari //--> <span role="img" alt="Bar2"></span> <!--// Pass //--> <span id="bat">Bat</span> <span role="img" aria-labelledby="bat"></span> <!--// Pass (ish) //--> <span role="img" title="Baz"></span>
27. HOW DO WE TURN THIS DATA INTO ACTION?
28. ISSUE DISCOVERY VS. BUDGET 0 25 50 75 100 0 25 50 75 100 New Issues Budget
29. TURNING DATA INTO ACTION ▸ Do automated testing first. This saves money and time. ▸ Estimated time-to-bug: .01 second per issue ▸ During Manual Testing: Learn to recognize the point at which you need to pivot from discovery to guidance ▸ During Reporting: Provide ultra-clear guidance up front and expect customers to avail themselves of that guidance ▸ Auditing is a speed bump that slows the eventual resolution and reduces learning opportunities ▸ Education is the #1 approach to resolving accessibility long term
30. THANKS! Karl Groves Founder & President, Tenon Ph: +1 443.875.7343 karl@tenon.io @karlgroves http://www.tenon.io http://www.karlgroves.com