User research

To add:

  • screen-magnification will not carry across
  • speech-recognition
  • BSL users

Contents

Introduction

We are not our users. This phrase is often used when talking about the merits of conducting user research. Putting a product in front of a user is a sure-fire way to discover issues which despite being in constant contact with it for months were missed by the entire development team. This is not a judgement, just how it is.

This is doubly true for users with access needs. Whilst you can design and test your product for accessibility, nothing can replicate the real-world experience of actual users. But all too often this aspect of user testing is overlooked.

Why recruit for access needs?

There is a popular phrase used in accessibility - “Nothing for us without us”. Simply put, don't design and release things without talking to actual real-world disabled users, you are likely to get something wrong due to misunderstanding, biases or lack of knowledge.

Users with conditions or disabilities come up against barriers every day as a result of products not consulting these groups. Perhaps a design just will not work for those who rely on keyboards, or have ADHD, or have memory issues. Perhaps the way content is written makes it difficult for users with certain cognitive conditions to understand it. Or how something is coded may mean speech-recognition users cannot easily complete a task. It is very easy for us to just forget about users who are not using the product in the same way we would. Accessibility testing can take us so far using best practices but we need to go beyond that.

The range of different conditions and the impact those conditions may have are so varied, and that is before we factor in any assistive technology or users with multiple conditions.

Then there is the standard user-testing question of “does the user understand what we are trying to communicate with this design or code?”. Whilst we might be able to test this code works with this screen-reader, we cannot determine if the user will understand what is being communicated or if they will even pick up on how it is being communicated. Lived experience is very important when it comes to accessibility - this is how we build our best practice knowledge.

Recruiting users

It can be difficult to recruit users with access needs who are likely to be future users of your product. This is more pronounced the more niche or specialised the product is, especially where some domain knowledge is assumed.

Help with recruitment

Your company might have an employee resource group which acts as a community for different groups of users within your organisation. This may be a good first choice for some initial testing, more-so if it is an internal product.

Specialist recruitment companies can help with this too and can even advise as to the users you may want to include. Other sources of potential participants may be online communities (check with the group moderator first) and local communities which may form around charities such as the RNIB.

There are also companies which offer accessibility testing as a commercial venture (often marketed as accessibility auditing), but these are not going to provide the insights which observing and communicating with a user in real-time will. These types of services are best left to later on when the product is nearing completion.

Just as with any other user research, participants should be fairly compensated for their time and expenses. With this group of users in-particular, if you normally provide gift-cards as compensation, check that they are able to use them as they may be restricted which shops they can get to (or some online shops may just be inaccessible to them).

Inclusive user research: recruiting participants

How many users should we recruit?

1 in 5 people in the UK are disabled, so ideally we would be looking at a similar ratio when performing user testing.

Look to recruit users with different access needs. They should include users who make use of assistive technology but also users with a range of conditions.

Take a look at some accessibility surveys to get an idea of the types of users and different assistive technology:

Be aware that some people will use adaptations which they don't consider as accessibility tools.

When to test with users

Ideally you should have enough resources to test at different stages of the product lifecycle, even if at the start it is more of a conversation about the product and what they would and would not like to see.

Preparing for a research session

Is the product ready?

This may seem odd especially when talking about something which is in development, but you should be mindful that these users are a rare resource. Putting them in front of a development version which has had no accessibility testing is going to likely throw up a lot of issues, potentially a few blockers.

What you should be doing as a team is makng sure that whatever you are testing with users has been fully tested for accessibility as far as you can. That may involve going outside your immediate team and asking for a peer review or contacting the accessibility team if you have one.

For example you have a chart with two colours which do not have enough contrast. This is something your own testing can easily pick up. Don't let it be a blocker during the session for a user who has colour-blindness as a secondary condition.

This is especially true for particular groups which rely on software which uses the accessibility tree, such as screen-reader users. How the product is coded becomes very important to these users so extra care should be taken to ensure the product is in a testable state.

Not being prepared is not respecting the time of your user and is not going to present you with the learnings which are the whole point of the session.

Location

Ideally you will be conducting the review at the users normal location, either their office or home. Doing this will provide the most realistic result as it will put the user at ease and avoid additional set-up and journey time for them. It will also mean they can easily use the equipment they would normally use.

If doing a remote session (via Teams or Zoom for example) then a short call a week or so in advance of the session will allow you to iron out any technical or practical issues. Bear in mind remote sessions may be more difficult with screen-reader users in particular as you will need to be able to hear what the screen-reader is announcing. If you cannot get the audio working, most screen-readers have some sort of visual log which displays the announcement, but bear in mind this is not the same as hearing what the user hears. You would need to ask the user to turn this on in their system settings - you may need to have the location of this in case the user does not know where the setting is.

If they need to travel to your offices consider the user and any issues they may have:

  • do they need to bring someone else with them? You will need to cover their expenses too and make sure there is somewhere for them to sit, perhaps next to the participant
  • is your location accessible, for example is there a lift, is there a long walk from reception to your office
  • can they access the building or office - do they need to go through a security process
  • make the user aware of the layout of the building in advance so they can be prepared, including the location of the toilets. This will also help reduce any anxiety on the day.
  • if they need to bring a service animal (such as a guide dog or emotional support animal) check there are no barriers for them and make sure there is space where the participant will be sitting for the animal to sit too
  • if the participant brings a walking frame or other aid such as a mobility scooter make sure there is somewhere secure it can be put, prefereably in the same room
  • check if they prefer particular lighting as office lights may be too bright for some participant's conditions
  • be aware of any trip hazards, particularly for users with low or no vision or mobility issues

Allow users to bring their own devices

Letting a participant use their own device to access the product will improve the results you get. It will reflect real-world usage more and put the participant more at ease than using unfamiliar devices.

Be aware they may bring several devices or use several adaptations, so think how you may need to adjust how you record the screens. Talk about this before the session so you can plan effectively.

Ask the participant in advance what software they use (such as browser plugins, screen-readers, speech-recognition) and what version it is if they can access that information. Also take note of the browser (and version) they use if the product is a website as this can help replicating any issues found. Some users may be using older versions of software, especially if it is expensive, and may turn off auto-updating for browsers.

If you need to access the participants device for any reason, such as connecting to wifi, adding a VPN or installing a mobile app for them to test, always ask first. Ensure you put their system back to excactly how it was before the session once testing has completed.

Depending on the complexity of the participant's set-up it might be possible to have a back-up device ready with their software loaded, just in case they have difficulties connecting with their own. Knowing the software they use in advance will help determining if this is possible.

Do some research yourself

Think about the user, do you need to be aware of any ways of interacting which would put them more at ease? For example, asking a blind user if they need assistance before guiding them to a seat.

Learn how should you describe their condition or disability and if in doubt ask.

Take time to run through the product you will be testing with the software the user will be using, get help with this if you are unfamiliar with it. Being familiar with how a user will interact with the product during a session, or how the product will look with particular user preferences (for example Windows High Contrast or a screen magnifier), or how different components may be announced by a screen-reader will help you understand any difficulties they may have during the session.

Think about materials

Be mindful that your standard document format (non-disclosure agreement, release forms) may not be suitable for all users. Where possible provide these a week or more in advance of the session so the user has time to read them and return them. Paper copies may not be practical. Take this as an opportunity to update your material to make them accessible.

If you use an online survey tool, check beforehand if it is likely to cause issues for the participant. Not all survey tools are accessible. Have an alternative option ready just in case.

During the session

No two users are going to be alike. You may need to adjust your prompts based on the user's conditions and how they are interacting on the day. For example will your prompts make sense mto a user who cannot see the screen, or who can only see a small section of it at a time?

Screen-reader users will need to listen to the announcements from the software as they navigate and read content, so do not interrupt them. Allow time for the participant to process what they heard before asking them a question or pressing them for comment.

Recording sessions

As already mentioned, participants may require multiple devices and may switch between them, so you may need to allow for changes in camera set-up if this is the case.

Screen-reader users may not be using a screen (or even have one if you are doing a home visit). You may need to ask them if they mind adapting their set-up slightly (this may be as simple as asking them to turn their laptop screen brightness up) or be prepared to bring equipment with you.

Good visibility of the screen becomes more important when recording assistive technology or adaptation users than for other users. For example being able to see a screen-reader's visual highlight, a speech-recognition mouse-grid, a keyboard focus indicator or how tremors affect use of touch targets. A direct feed from the screen is best otherwise a clear tight shot of the entire screen. You may also need a wider recording of the user, especially if they are using a touch device.

You may need to ask a screen-reader user to slow down the announcements so you can interpret them during the session and for later analysis.

If a screen-reader user has a cochlear implant they may normally send sound directly to it, so again they may need to adapt their normal set-up so you can hear it and capture the sound of the screen-reader.

Allowing more time

Build additional time around the session in case the participant has issues getting there (for example needing to wait for the next bus due to lack of wheelchair space).

You may also need to allow additional time for equipment set-up if the participant is bringing their own, for example connecting to the office wifi.

The participant may take longer to process the information on the screen or take longer to input data.

You may not get through a standard session task in the time allotted, so it may be you need to arrange for repeat visits or split a task across more than one participant if appropriate.

The participant may tire quickly, so a session may end incomplete and earlier than you expect. Be sure to be thoughtful if this happens. Have some questions prepared if they feel unable to carry on with screen-work but are happy to chat. Even if a session ends early be sure to fully compensate them.

Reviewing the session

Always treat any health information as confidential.

Review the issues raised. However don't be tempted to treat the words of a single user at face value. Assess the information just as you would any other participant. It can be tempting to make widespread changes because one screen-reader user had issues. Bear in mind this was one user.

For each issue raised ask yourself questions: - is the issue a blocker? - is the issue more of a preference? - did the participant offer a solution? Treat that as only one possible fix. - did the participant just miss something? Review the video. - set up a session with another participant if you need qualification about an issue - was the issue raised because they are an 'expert' user? - was the issue raised because they are a new user? - how might fixing this make things worse for users with different conditions?

The results of the session will not tell you how all users with that condition or who use that assistive technology will interact with the product. It will just tell you how that one person did on that day in that environment when they were feeling the way they were. It is up to you to look at those findings and decide if any of them can be extrapolated to changes being required.

If you think you have some issues which need addressing, especially if they involve assistive technology, look to have the issues reviewed by someone who uses that software regularly (such as an accessibility team member). Don't jump to assumptions - if the issue is valid there may be several ways in which it can be remediated.

Inclusive user research: analysing findings

Share your findings

Quite often user research findings are kept within the product team. They are far too valuable to just do this. Anonymise the data (you may not be able to share the videos) and share your findings with other teams. Set up a library to showcase what users flagged as issues and what was decided to do about it and why.

Wrap-up

User testing with people with disabilities and access needs is a crucial step when developing a product. The insights the participants can give you may mean the difference between an accessible and inaccessible outcome and even change how you approach future designs. But approach findings with caution too. Rushing headlong into applying fixes can make things worse.

Further reading