Problem
My thesis project, conducted in two phases, investigates the effectiveness of allowing older adults to self-tailor, or personalize, a website's information architecture (IA) structure. The content used on the website are dementia awareness videos created for a separate research project in collaboration with Cyber-Seniors. View the entire thesis here.
This page describes the second phase of the project, during which I conducted a usability evaluation of the refined prototype design created in the first phase. The goal of the usability evaluation was to determine the impact that being able to self-tailor the IA structure had on the participants' experiences while using the website.
My Responsibilities
Prototyping, Research Study Design, Usability Evaluation
Tools
Framer
Zoom
Microsoft Forms
Microsoft Excel
My thesis project, conducted in two phases, investigated the effectiveness of allowing older adults to self-tailor or personalize a website's information architecture (IA) structure. The content used on the website is dementia awareness videos from Cyber-Seniors, who supported this project. View the entire thesis here.
This case study describes the second phase of the project, during which I conducted in-depth usability evaluations with older adults, using the redesigned prototype created during the first phase of the project. The goal of the evaluations was to determine the participants' preferences for linear or matrix IA structures and evaluate the impact that being able to self-tailor the IA structure had on the participants' experience while using the website
Problem Statement
How do older adults envision the option to switch between linear and matrix IA structures and how can their perspectives be reflected in the website's design?
Team
Trevor Cross
Tools
Figma
Framer
Zoom
Microsoft Excel
Timeframe
5 Weeks
My Responsibilities
This project was done completely independently. My thesis supervisor, Dr. Olivier St-Cyr, consistently gave valuable feedback, but all of the work shown below was completed by me.
Constraints
Based on the approval I received from the research ethics board, the usability evaluations needed to be conducted remotely. This increased the complexity of the evaluations, as each individual had a different computer and system settings, and made it slightly more challenging to gather design and research insights. However, the website was designed to be responsive, which made it usable by all of the participants.
Study Design
The usability evaluation required participants to complete a series of tasks on three different versions of the prototype:
One with a linear structure
One with a matrix structure
One with a self-tailoring element, allowing participants to freely switch between a linear and matrix structure.
The participants' experiences were measured using a post-task Likert scale questionnaire and post-evaluation semi-structured interview.
Instruments
Questionnaire
During the usability evaluation, each participant completed a post-task questionnaire after the first and second task. The purpose of the questionnaire was to inform which IA structure, linear or matrix, was preferred by the participants by identifying which of the two they found more engaging.
The questionnaire used was a modified version of the UES-SF developed by O’Brien et al. (2018), using a 5-point Likert scale.
Post-Evaluation Interview
Following the completion of all three tasks in the usability evaluation, a semi-structured interview was conducted with each participant. Emphasis was placed on understanding the participants' experience using the self-tailoring feature on the website (task three), particularly for the questions that focused on functionality.
Procedure
Participants
A total of 12 older adults participated in the usability evaluation. In order to participate in the study, participants had to meet a set of inclusion criteria:
you are over the age of 65
you are fluent in English
you have previous experience using a computer and browsing websites (basic IT support to be provided by the researcher)
you have access to a desktop computer or laptop
you are interested in an online self-guided dementia awareness program
you do not have a form of dementia
you consent to have your audio and screen recorded
Additionally, participants were required to complete a consent form before they could participate in the study.
Evaluation Protocol
Counterbalancing
Given that the usability evaluation design, the order in which the participants were exposed to the different versions of the website prototype was counterbalanced to account for potential order and sequence effects and improve the internal validity of the evaluation.
A visual representation of the counterbalancing can be seen here:
Data Analysis & Results
The usability evaluations provided quantitative and qualitative data that could be used to derive design and research insights. For each participant, the two sets of questionnaire scores were calculated by averaging the scores given to each individual question (in accordance with the original questionnaire instructions).
The participants' average questionnaire scores for the two structures were very similar:
Linear IA Structure: 4.19
Matrix IA Structure: 4.40
In fact, there is not a statistically significant difference between the participants' scores for either IA structure. This result reveals that self-tailoring may be beneficial, since the whole participant group did not have one clear IA structure preference
When comparing the participant scores with their answers to the interview questions and my observations of how they used the self-tailoring option, I determined that 10 out of 12 participants used the self-tailoring option to select the IA structure they preferred.
Interview Data Analysis
Similar to the first phase of the project, I conducted a qualitative thematic analysis (a rigorous but less visually appealing affinity diagram) of the interview responses using NVivo to identify themes and insights.
Example of how direct quotes were grouped into themes
Summary of Interview Findings
Based on my analysis of the interviews, the participants' insights can be grouped into two categories:
Factors that influence IA Preference:
Desired amount of information
Perceived complexity of their interaction
Familiarity with the prototype and websites in general
The type of task being perform (are they searching for specific information?)
Benefits of Self-Tailoring Option:
Desire to use both IA structures
Self-tailoring provides a greater sense of control over their experience
Design Insights & Implications
Overall, the findings of this phase of the project and my thesis as a whole can be summarized by the following:
No single IA structure is best for older adults in all scenarios. Preferences can change based on the content being presented and the users themselves.
Older adults can successfully choose, or self-tailor, an IA structure based on their own preferences, which gives them greater control while navigating a website and improves their overall experience.