-
Notifications
You must be signed in to change notification settings - Fork 0
phaseIII
The last phase left us with design goals to create a prototype in Figma which solved the key issues found through cognitive walkthroughs. This resulted in a more flushed out workflow with more details added to help the user understand the flow of the program. Once this new prototype was made, we needed to get more information on what users thought was Satisfying, Efficient, and Effective. This main goal of this phase was to create and execute a user study, including test protocols and IRB approvals.
A small group (n = 5) of participants were invited to take part in a moderated user study evaluating an early prototype of the application. Each session consisted of a one-on-one interview in which the participant was asked to interact directly with the prototype. During the session, the facilitator guided the participant through a predefined series of tasks designed to exercise the core functionality and typical user workflows.
As participants worked through the tasks, they were encouraged to think aloud and describe their expectations, decisions, and any points of confusion. Following task completion, participants were asked a set of semi-structured questions to gather feedback on usability, clarity of interactions, and perceived value of the features. The facilitator recorded observations, task outcomes, and participant responses for later analysis. The full study protocol and task list can be found here.
-
Upload a PDF for use
This task evaluates discoverability and affordance of the upload functionality. Timing how long it takes users to identify the upload button reveals how visually prominent and intuitive the entry point is.
-
Task 2: Identify accessibility issues in the uploaded PDF
This task assesses system feedback, clarity of results, and conceptual understanding of accessibility diagnostics. It reveals whether users can easily tell when analysis has occurred and how to interpret the output.
-
Task 3: Add missing alt text for an image
This task focuses on learnability, guidance, and interaction clarity for a core feature of the application. It is especially important for understanding how well the interface supports users who may be unfamiliar with accessibility concepts.
-
Task 4: Save the updated PDF
This task evaluates closure, confirmation, and workflow completion. It determines whether users understand how to finalize their work and feel confident that their changes are preserved.
The primary findings from the user study show that most participants were able to complete the tasks successfully and generally rated them positively, indicating that the core workflow of the prototype was understandable. However, several usability issues emerged, particularly around accessibility-specific terminology. Participants with no prior a11y experience expressed confusion about concepts such as "compliance" versus "accessibility issues," which led to uncertainty about where to navigate and how to interpret results. Some users also felt that certain screens contained too much information or had a convoluted layout.
The full data can be found in this spreadsheet.
Overall the platform provided to the platform were able to easily able to complete each task and recognize that they completed them. Each task was completed Efficiently only taking a few seconds per task. The workflow of the prototype was effective and allowed users to complete each task seamlessly. The largest issue with the prototype was in the terminology. A particular tool, which checks if the pdf is compliant to a set of standards, had terminology specific to accessibility. This led to brief confusion on how to complete the task. Despite this, tasks were completed.
The test was only done with students that are enrolled in the CSCI 431W class. This led to participants who may not have used other accessibility software before. There were also limitations in the prototype itself. Tagging images is reliant on inputting text however Figma's prototype framework does not support physically typing into textboxes.