Test development
Mapping tests in WCAG 2.0 Techniques and Failures onto the WCAG-Test


WCAG 2.0 has hundreds of Techniques and Failures. Many contain at their end Tests with a single step-by-step Procedure and Expected Results. To make sure that BIK's WCAG-Test (under preparation) can claim equivalence to the tests in WCAG 2.0 Techniques and Failures, a mapping excercise is currently being conducted.

Different approaches to testing

Structure of tests in WCAG 2.0 Techniques and Failures

The tests in WCAG 2.0 Techniques and Failures are characterised by the following points:

  • The tests in WCAG 2.0 techniques are generic, they do not specify tools. They specify what should be checked, not how this is done on the level of browser, Accessibility toolbar, Firefox Add-on or bookmarklet. While this level of generality may facilitate a mapping onto any number of test tools, it also implies that a separate more detailed instrument is indeed required to translate the test into a documented process that people can use (often involving the application, then interpretation of automatic checks with particular tool functions).
  • The tests in WCAG 2.0 techniques have a binary outcome: Results are either true or false. This implies that tests happen on the atomic level of elements on a page (not necessary atomic elements in the DOM tree — the term "element" is used here to signify a semantic unit, for example, the existence of a form input field and a corresponding label, or the existence of an image and an alternative text). Since many pages contain multiple instances to be checked, this implies that such a test cannot, without further aggregation, be used to provide a quantifiable outcome on the level of an individual page (unique URI).
  • The sucess or failure of any individual test in a WCAG 2.0 technique cannot be aggregated in an accessibility score on the level of success criterion. The reason is that often, several techniques (requiring different checks to determine whether they have been implemented correctly) can be used to satisfy a particular success criterion. Therefore, an individual test at the end of a particular technique is not sufficient to determine whether or not a success criterion has been met: other techniques may still satisfy the criterion in other ways. An example are skiplinks: if used, the test in Technique G1 applies, but the Success Criterion 2.4.1 (Bypass Blocks) may be met in different ways: in this case, by using an appropriate heading structure (WAI-ARIA landmarks are not yet referenced as a suitable technique).
Structure of checkpoints in the BIK WCAG-Test

The BIK WCAG-Test (under preparation) is based on the test instrument and methodology developed for the BITV-Test - an instrument that over the years has been successfully applied in more than 1000 tests for various institutions. The main approach of the WCAG-Test is to arrive at a point score (maximum is 100) that adequately reflects all relevant WCAG 2.0 success criteria. Sites that reach 90 points or more are considered accessible. Individual barriers (such as keyboard traps or lack of alternative text on important graphic navigation elements) lead to the downgrading of the site as 'badly accessible' irrespective of the overall score.

The test's checkpoints check compliance not on the level of the indivdual WCAG 2.0 technique used, but on the level of success criteria. In some cases, as in SC 1.3.1 Info and Relationships, success criteria are very general and can only usefully be tested in a single checkpoint when they are broken down to the programmatic elements (such as headlines, lists, tables, frames) to which the same criterion applies in different ways.

This aggregation means that when checking the compliance with at a particular success criterion (take alternative text of images), a number of tests that are spread across WCAG 2.0 techniques are actually conducted at the same time and in conjunction. Looking, for example, at the images on a page, a function in the WAT Toolbar will list all images and corresponding alternative texts, while swiching to another colour scheme within the same checkpoint will indicate whether background images have been applied for decorative purposes only, or may replace adequate alternative text in an accessible manner.

Is just using the tests in WCAG 2.0 Techniques and Failures a practical alternative?

If one were to test for meaningful alternative text of images (Success Criterion 1.1.1 (Non-text Content) based solely on the tests provided at the end of WCAG Techniques and Failures, the following Techniques and Failures would need to be considered:

The sheer number of tests that might apply is overwhelming. Checking out the tests at the end of Techniques, a significant amount of redundancy is quite apparent. Since so many tests may apply to an image, it seems clear that they are simply not meant to be carried out in a linear and complete fashion.

A more practical approach would therefore start with the actual images on the page to be tested, then assess the appropriateness of the alt-attribute (or image replacement technique) which will map onto one (often several) of the many atomic tests listed above. 

The crucial point however is that not aggregating various tests within one Success Criteria checkpoint makes it impossible to establish in a straightforward way the compliance of a page across all relevant Techniques.

Aggregation of atomic tests in checkpoints

The BIK WCAG-Test aggregates in each checkpoint a well documented process that involves several atomic checks which are often complementary since technically, there are different ways of meeting a particular success criterion. This aggregation ensures that testing does not merely reveal a true/false result for a particular application of an often non-exclusive atomic technique, but a weighted assessment of the degree to which a success criterion is actually being met by the entire page. (We ignore at this point the complexity of other states of a page, triggered by revealing or generating additional content through user interaction.)

Mapping the tests in WCAG 2.0 techniques onto the 50 checkpoints of the WCAG-Test

To make sure that BIK's upcoming WCAG-Test covers the full range of tests documented in WCAG 2.0 Techniques and Failures, a mapping excercise is currently being conducted. The aim is to link from tests in WCAG 2.0 Techniques and Failures to the corresponding point in BIK WCAG-Test checkpoints (technically this is accomplished by including a fragment identifier to link from a mapping table straight into the WCAG-Test checkpoint). The inverse mapping is already available: Each checkpoint of the current BITV-Test contains a section listing the relevant WCAG 2.0 Techniques and Failures.

Likely outcome of mapping

We expect that most WCAG 2.0 Techniques (with the exception of FLASH techniques - Flash is currently not tested) are already included within the checkpoints of the WCAG-Test.

A tricky area is the testing of script-generated content. The existing checkpoints detect various problems related to inaccessible implementations of scripted content, for example, keyboard traps, elements not receiving focus or not being in the right position within the source code, or moving/animated content that cannot be stopped by appropriate interface controls. The checking of some techniques — especially when content is dynamically generated and inserted through scripts — requires tools such as Firebug or Dirk Ginader's LogFocus booklet (hat tip to einfach für alle) that are so far not used in the existing BITV-Test. Here, further work is needed to define and document approriate test steps.

Thoughts or comments?

If you have any thoughts on this or would like to comment, please use the comments section below. We suggest to follow @wcagtest on Twitter if you want to keep informed about the progress of the development of WCAG-Test.