Automating Mouse Transfer Function Design Using Active Learning

Hintergrund

A wide range of systems and user interfaces depend on users pointing with an indirect pointing device, such as a mouse or a touchpad, to complete tasks within the system [1]. To achieve this kind of interaction, users’ physical movement must be translated to the movement of the on-screen pointer, which requires a pointing device transfer function. This function defines the relation between the physical movement and the pointer movement as seen on the screen [5]. An exemplary implementation of a pointing device transfer function, which is often used by researchers, is using a control-to-display (CD) gain, which describes a scale factor between the real-world movement and the pointer movement on the screen. The CD gain can either be constant or dynamically adjusted, for example according to the velocity of the mouse [2]. Studying this kind of CD gain, previous work shows that the pointing transfer function impacts user performance, and a constant CD gain is often outperformed by a dynamically adjusted CD gain [3, 6].

Designing a good transfer function is challenging, as it requires manual trial and error [6]. Typically, in an iterative process, researchers design a function which they believe improves performance and then validate this function in an empirical study. They then improve their design with the new findings.

To better understand the pointing process, researchers not only study different pointing device transfer functions but also the pointer movement itself. Aranovskiy et al. [1] propose a switched dynamic model which is able to describe the pointer movement during a pointing task while the most popular model, Fitts Law, only takes the outcomes of a pointing task into account [4]. To validate and test their pointing transfer functions, researchers often use the Fitts’ Law paradigm. While participants complete a two-dimensional pointing task with varying difficulties, the measured movement time, error rate or the calculated throughput indicate their pointing performance. A higher throughput indicates higher performance [7, 8] and therefore a better suited transfer function for target selection tasks.

Automating the design process could help with the tedious pointing transfer function design task, but since there likely are several parameters in such a function [6], traditional machine learning is not suitable for finding the best set of parameters. More advanced approaches such as active learning however would be able to predict the user’s performance and the confidence in the prediction for a given function. Through a sampling technique, the model then provides the most promising parameters. For example, using uncertainty sampling, this would allow to explore the parameter space by trying the sets of parameters where the model is least confident [9, 10]. Utilizing this kind of active learning provides an interesting opportunity to automate all sorts of design challenges.

Zielsetzung der Arbeit

The thesis aims to utilize an active learning approach to explore the design space of pointing transfer functions for mouse input with the goal of designing an optimal pointing transfer function for the given task. This includes building a process of automating the otherwise tedious design of pointing transfer functions. To achieve these goals, a prototype system should be developed and later evaluated through an empirical study. The system should be able to iteratively propose new pointing transfer functions for which the user performance is then determined for a pointing task. Through this method, the system should be able to gain knowledge about the problem space and in turn use this knowledge for proposing the next pointing transfer function. This system consists of a front- and backend, where the frontend serves the pointing task to the user while the backend handles exchanging pointing transfer functions and performance metrics between the frontend and the pointing transfer function optimizer.

Konkrete Aufgaben

Concept and implementation

Reviewing previous work on pointing device transfer functions, CD gain, active learning, automated design and related topics

Conceptualize a suitable system providing a frontend for the user and a backend for managing transfer functions and results as well as an optimizer for choosing and improving the next transfer function

Find a suitable way of determining the first mouse transfer function (start function for optimization)

Create the model, implement and test the application

User Study

Desing an empirical study suited for answering the underlying research question

Do a test run of the study to ensure everything is working and the collected data is as expected

Execute the study

Analyze the results to give insight on the topic

Thesis writing

Erwartete Vorkenntnisse

JavaScript

Web-Development

Client-Server

Python

Machine Learning

Active Learning

Weiterführende Quellen

[1] Stanislav Aranovskiy, Rosane Ushirobira, Denis Efimov, and Géry Casiez. 2020. A switched dynamic model for pointing tasks with a computer mouse. Asian J. Control 22, 4 (2020), 1387–1400. https://doi.org/10.1002/asjc.2063

[2] Géry Casiez and Nicolas Roussel. 2011. No more bricolage! methods and tools to characterize, replicate and compare pointing transfer functions. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST ’11), October 16, 2011. Association for Computing Machinery, New York, NY, USA, 603–614. https://doi.org/10.1145/2047196.2047276

[3] Géry Casiez, Daniel Vogel, Ravin Balakrishnan, and Andy Cockburn. 2008. The Impact of Control-Display Gain on User Performance in Pointing Tasks. Human–Computer Interact. 23, 3 (August 2008), 215–250. https://doi.org/10.1080/07370020802278163

[4] Paul M. Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol. 47, 6 (1954), 381–391. https://doi.org/10.1037/h0055392

[5] Raiza Hanada, Damien Masson, Géry Casiez, Mathieu Nancel, and Sylvain Malacria. 2021. Relevance and Applicability of Hardware-independent Pointing Transfer Functions. In The 34th Annual ACM Symposium on User Interface Software and Technology (UIST ’21), October 12, 2021. Association for Computing Machinery, New York, NY, USA, 524–537. https://doi.org/10.1145/3472749.3474767

[6] Byungjoo Lee, Mathieu Nancel, Sunjun Kim, and Antti Oulasvirta. 2020. AutoGain: Gain Function Adaptation with Submovement Efficiency Optimization. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), April 23, 2020. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376244

[7] I. Scott MacKenzie. 1992. Fitts’ Law as a Research and Design Tool in Human-Computer Interaction. Human–Computer Interact. 7, 1 (March 1992), 91–139. https://doi.org/10.1207/s15327051hci0701_3

[8] I. Scott MacKenzie and William Buxton. 1992. Extending Fitts’ law to two-dimensional tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’92), June 01, 1992. Association for Computing Machinery, New York, NY, USA, 219–226. https://doi.org/10.1145/142750.142794

[9] Margaux Masson-Forsythe. 2024. Active Machine Learning with Python: Refine and elevate data quality over quantity with active learning. Packt Publishing Ltd.

[10] Robert (Munro) Monarch. 2021. Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster.