Thursday, September 8, 2011

Blog - Paper Reading #4

Gestalt: Integrated Support for Implementation and Analysis in Machine Learning

Authored by Kayur Patel, Nayomi Bancroft, Steven M. Drucker, James Fogarty, Andrew J. Ko and James A. Landay.

Kayur Patel is pursuing a PhD at Washington University. Naomi Bancroft graduated from Washington University and now works for Google. Seven M. Drucker, the only member not affiliated with WU, works for Microsoft. The other researchers all currently work at Washinton University in some fashion. James Landay is a professor, while Andrew Ko and James Fogarty are assistant professors. This paper was presented at the 23nd annual ACM symposium on User interface software and technology in New York.

Summary

Hypothesis
The authors hypothesized that Gestalt would improve a user's ability to find and fix errors in machine learning code by clarifying the relationships between steps in the classification pipeline.

Methods
Each tester was given one of two problems, one based on gesture recognition the other based on discovering movie review sentiment. On some problems a general purpose program akin to matlab was used as a baseline. On others, Gestalt was utilized. The subjects using Gestalt were allowed to ask questions of the researchers, as usability was not being tested. The test was meant to simulate salvaging another programmer's work. Each participant was given a problem with five bugs and had one hour to find and fix as many as possible. They were also asked to narrate their thought process as they worked on the code.

Results
The results seemed relatively positive. Most users expressed that the visualization tools provided by Gestalt would greatly assist them in their own work. This was generally supported by the data that showed an increase in bug correction in tests using Gestalt.

Discussion

This paper demonstrates the effectiveness of a new general purpose development system for machine learning programs. As programmers we are really only as good as the tools that we use, and sometimes it behooves us to set aside our work and build new tools. A new coding environment that provides gains across the board is a very good tool indeed.
The authors observed in testing that there were slightly greater rates of success on the second problem as compared to the first. Additionally they hypothesized that their representation would be of greatest benefit to the user at the start of a project. Their experimental design, however, provided users with a finished piece of code, and therefore did not test this seemingly important element. Also from what I can tell their experiment did not actually test any users on both the baseline and Gestalt so as to acquire a comparison.
While there were a few holes in the experimental procedure, the feedback was wholly positive, and the results support their conclusions. If further developed, Gestalt may see active use. The effect of a new and better tool entering wide use, is a general improvement in the field across the board. The less that we as programmers have to focus on locating errors and grasping abstract program structures, the more efficiently we can create them.




No comments:

Post a Comment