Analysing Qualitative and Quantitative UX Research

Analyising UX Research can be a tricky and time consuming aspect of the research process. In this article we provide you with the most important metrics and a step-by-step guide for analysing user tests, interviews and focus groups. 

The role of analysis in UX Research

UX Research enables organisations to understand how users experience their products and services. These insights form the basis for improvements and innovations. Analysis is the bridge between raw data and actionable insights. It is the process by which you reveal the meaning behind the numbers and uncover deep insights into user behaviour, preferences and motivations.

When to conduct the analysis?

Conducting analysis is a structured process, and the timing is crucial. In most cases, analysis takes place after all the data has been collected. This ensures a complete picture of the user experience and avoids premature conclusions.

4 Common mistakes when analysing UX Research

• Premature conclusions. In the modern world, there is often pressure to present results quickly. This can lead to rushing the analysis. Researchers should be aware of these pressures and focus on thorough and careful analysis rather than jumping to conclusions quickly.
• Confirmation bias. The tendency to consciously or unconsciously seek confirmation of existing beliefs. To avoid confirmation bias, researchers should consciously strive for objectivity. This includes actively looking for contradictory data and being open to new insights, even if they conflict with previous ones. 
• Positivity bias. This includes the tendency to focus on the positive findings, reducing the exposure of negative findings. This can lead to distorted insights and missed opportunities for improvement.
• Lack of context. Analysis without proper context can lead to misinterpretations. It is essential to understand the situations and circumstances in which users interacted with a product or service. To overcome this, it is important to conduct qualitative research in addition to quantitative research. For additional insight into user context, testers can be asked to participate in a study from their own home. 

Analysing Quantitative UX Research

Quantitative analysis is a research method in which numerical data are collected and evaluated to identify patterns, trends and statistical inferences. In UX research, quantitative analysis involves collecting and analysing measurable data about users' interaction with a product or service. This data is often expressed in numerical values - usability metrics - and may include variables such as time, frequency and scores.

Quantitative usability metrics

While most of the analysis is qualitative in nature, there are a number of quantitative usability metrics that you definitely don't want to miss on. 

Task completion rate or success rate

The success rate, also known as the task completion rate or task success rate, is one of the most well-known usability metrics. When the task that the respondent has clearly formulated and has a measurable end goal (for example placing an order), the success rate shows how well users can handle the product.

The success rate is the ratio of the number of users who have successfully performed the tasks of the usability test. You calculate the success rate by dividing the total number of successful tasks of all respondents by the total number of tasks performed by all respondents.

Time to task completion rate

The time to task completion provides insight into how much time respondents (on average) needed to complete a task. It helps to set objectives prior to the usability test, so that you can compare your expectations versus reality. 

In general, the longer a user needs, the more complex the task or process. You calculate the time to task completion by adding up the time users needed and dividing it by the total number of respondents.

User Error Rate

It goes without saying that it is super interesting to zoom in deeper ornto mistakes that users are making when performring the usability tests.  It gives you an idea of ​​the user-friendliness of the product and shows you what users encounter.

Apart from the total number of mistakes that respondents make, you can also compare which interfaces promote errors or provide more clarity (e.g. with a/b testing).

But when does the user make a mistake?

Before you can calculate the error rate, you want to be clear on when something is an error and when it is not. You can think of opening the wrong page, selecting the wrong product, entering incorrect information in the contact form, et cetera.

This depends on the type of product you develop. To ensure that the error rate does not paint a distorted picture, it is advisable to record this in advance in the usability test plan.

When you are clear when something is an error, you can calculate the error rate with the formula below. For this you sum up the total number of mistakes made. You divide this by multiplying the total number of errors possible and the total number of respondents.

Keep in mind that it is normal for users and respondents to make mistakes. A study by Jeff Sauro, who analyzed 719 tasks, showed that the average error rate per task is 0.7 - meaning 2 out of 3 users made mistakes.

Additional usability metrics

In the addition to the above shown metrics, there are four other metrics often used to measure the user experience:

• System Usability Scale (SUS)
• Single Ease Question (SEQ)
• SUPR-Q
• Net Promoter Score (NPS)

Analysing Qualitative UX Research

Qualitative UX analysis is a research method that evaluates results from interviews, focus groups and user tests to gain in-depth insights and understanding. In UX research, qualitative analysis focuses on understanding users' experiences, attitudes and perceptions, as well as identifying patterns and themes in the data collected. 

The analysis process for user tests, interviews and focus groups is quite extensive compared to the analysis in quantitative UX research. Below, we have detailed the process for you:

1 - Look back at the research questions

Before you start your analysis, it is advisable to take a moment to look back at your research objectives and research questions. This way, you will know exactly what to pay attention to and make sure you can take relevant notes.

2 - Take as many notes as possible

When looking back at the session recordings, make sure you take as many notes as possible to ensure that all interesting findings are included in the analysis. To make the process easier later, you can use hashtags to indicate what kind of finding it is. For example, you can think of hashtags such as: #navigation #lackofinformation #homepage. 

3 - Group your notes by theme or task 

After you have analysed all the sessions, chances are you have a large amount of notes. The notes may differ in form and content, which probably means you don't have a good overview and it is difficult to draw conclusions. Therefore, the next step in the process is to start categorising the findings. There are two ways to do this:

Thematic analysis
Thematic analysis is a method for identifying, analysing and reporting patterns (themes) within qualitative data. The process involves coding text or observations to discover repeated ideas and concepts. Themes can then be interpreted to generate insights. For example:

Annotation: Validation of email field does not work properly
Annotation: Link behind general terms and conditions does not work
Example theme: Pain points in registration form

Task analysis
Task analysis is a method that groups insights based on the tasks performed during user tests. Here, problems and observations related to specific tasks are identified and evaluated.

Measure the frequency

Make sure to measure the frequency of issues on the website. This can be done on annotation level, but it is recommended to first group your annotations by theme and then record the frequency. There are multiple ways of going about this, but the easiest way is to do so by using a spreadsheet. Make sure to track the following details:

• Use a unique ID per note
• A short clear description of the problem / theme
• Indicate how many users encountered the same issue

4 - Go from themes to real insights

The next step in the process is to start interpreting the findings from the research. In other words, you are now going to turn the findings into insights.

The difference between findings and insights

A finding describes what happened, without offering any further explanation or solution. An insight, on the other hand, describes an aspect of human behaviour or motivation. This makes it clear what the solution might look like.

For example, a finding could be 'Testers report using different websites to book a holiday'. This finding could lead to the following insight: "There is currently no all-in-one solution for booking a holiday, which means multiple websites have to be consulted."

Thus, if it turns out that the above insight is prevalent among many users, you might decide to develop a new concept through which someone can book an entire holiday on the same website or app. 

Turn findings into insights

Now is the time to go through the findings you have grouped by theme or task. The aim here is to look for insights. In doing so, you will of course keep the research questions at hand and start with the themes that connect to them.

An example of what that might look like:

Theme: Pain points in booking -> Finding: Testers have trouble selecting the right dates -> Insight: The date selection module is not user-friendly on mobile devices

On giving priority to insights

Not every insights has the same impact and to determine which insights have the highest priority, you can to take the following factors into account:

• The critical score. How important the functionality is for the company and the end user.
• The frequency ratio. How many respondents experienced the problem.
• The impact score. The extent to which the problem influenced the respondent.

In a step-by-step plan it looks like this:

1. Determine the critical score. Assess how important this functionality is to the business and the end user. Use a scale from 1 to 5
2. Calculate the frequency score. To do this, divide the number of respondents who have experienced this problem by the total number of respondents who have taken the usability test. You will then get a score between 0 and 1.
3. Determine the impact score. Use the criteria below:
   • 5 (Blocker) Problem prevents the user from completing the task
   • 3 (Major problem) Problem causes frustration and delay
   • 2 (Minor problem) Problem has minimal impact
   • 1 (Suggestion) Suggestion of the respondent
4. Calculate the priority score. Do this by multiplying the critical, frequency and impact score. The insight with the highest score, then has the hightest priority. 

Reporting for UX Research

When you have analysed all data the next step is to present the results in a clear format. Consider using the following options:

• UX Research Report. Write a UX Research Report in which all insights are presented. Make sure to use a lot of screenshots and quotes from users to really bring your point across.
• Highlight video. If you have conducted qualitative UX Research, it is highly recommended to create a highlight video in which the most important fragments are shown. You can do so by using our platform.