Project: Piñata

an image of a computer and a mouse

The State of the World

Using a mouse or a pointing device to interact with computers can be challenging for individuals with motor impairments. Our interfaces are not intelligent enough to detect when a person is having difficulty pointing, nor do they do anything about it.

What I did

Building on previous research on individuals with motor impairments and a corpus of data collecting around their clicking behavior, I, along with a team of researchers at the University of Maryland Baltimore County created a system called Piñata (Pointing Interaction Notifications and AdapTAtions.)

The system is made up of a web browser extension that monitored cursor movement inside the browser. The browser, sends that information to a PHP server, which used a machine learning model to predict if the user's cursor movement is consistent with that of a person with a motor impairment—say someone with tremors.

I designed the system to alert the user about their pointing behavior using a three-tiered design approach to notifications based on the level of disruption to user flows:

Design A. An abstract indicator notification

Design B. A dialog Box notification

Design C. A pop-up notification from the add-on section of the browser.

an image showing 3 notification designs

We conducted numerous research studies for this project.

The Team

Dr. Abdullah X. Ali – User experience design | User interface design | Software development | User Research | Publication and communication

Catherine Hornback – Machine Learning development

Dr. Aqueasha Martin-Hammond – Publication and communication

Dr. Amy Hurst – Project leadership and mentorship

What I delivered

Phase I:

This project contributed insight into user preferences and design considerations around how adaptive user interfaces for pointing are both perceived and accepted by users.

A poster showing notification design preferences

The poster to the left summarizes the notification preferences of the three designs as a result of a user studies with 38 users.

We also created a list of personas that illustrate preferences in notification designs and how they would like their interface to adapt, whether automatically or by adjusting settings themselves.

A table showing three personas and their notifications and adaptations preferences

Phase II

An image of updated notification designs

We followed the insights from phase I by adapting the design of the notification system and adding interface adaptation options as shown in the image to the right here.

Our research emphasizes factors beyond performance and accuracy that can impact user perception and acceptance of personalized systems and complements other research on personalized systems that aim to support users who are undergoing changes in their abilities. Much of our findings indicate that understanding and integrating support for individual user goals as well as understanding and supporting individual user preferences for involvement in the pointing task is important for encouraging adoption.

My Impact on the World

The team published the phase I work at the Web for All (W4A) conference in 2015.

You can read the publication here.

Understanding Design Considerations
Download PDF • 482KB

We also published a short paper at Pervasive Health in 2016

You can read the publication here.

Supporting Awareness of Pointing Behavior among Diverse Groups
Download PDF • 312KB

The team published the phase II work at the Web for All (W4A) conference in 2018.

You can read the publication here.

Designing an Adaptive Web Navigation Interface for Users with Variable Pointing Performanc
Download • 1.03MB

dr. ax ali