Blog - Paper Reader #6

TurKit: Human Computation Algorithms on Mechanical Turk

Authored by Greg Little, Lydia B. Chilton, Max Goldman and Robert C. Miller.

The authors are all graduate students at MIT with the exception of Robert Miller who is an associate professor. He leads the User Interface Design Group. This paper was presented at the UIST '10 Proceedings of the 23nd annual ACM.

Summary

Hypothesis 
The authors hypothesized that TurKit would improve prototyping of algorithmic tasks on MTurk at the expense of efficiency.

Methods
The design seems to be centered about what the authors call the "crash and re-run" paradigm. Basically when the program runs to failure, rather than dumping it's state, a user can select which of the completed steps are to be rerun and which are not. This saves on expensive human computation. They also demonstrate how TurKit provides an interface to the now abstracted MTurk. Additionally they did a small performance evaluation that demonstrated the efficiency levels were acceptable for prototyping.

Results
Results were generally positive. A few sample tests done with TurKit demonstrated it's usefulness in managing MTurk. There were some drawbacks however. While easy to implement, users described several usability issues. Scripts for example were necessarily deterministic, and users were also unaware of the parallel features of TurKit.

Contents
This paper describes TurKit. A user interface to be used in conjunction with MTurk. It allows for simple imperative programming techniques to be used. TurKit provides a means of incremental programming, allowing modifications to be made to original scripts before being rerun.  The paper steps through a series of tasks to which MTurk would traditionally be applied and describes the benefits and shortcomings of Turkit.

Discussion

The paper is interesting as it expands on a technology used to utilize human computational skills. It is much akin to distributed computing except the resources being exploited are the users themselves. The software has apparently fallen short in the category of usability, but not cripplingly so. It seems that the developers simply need to pay a bit more attention to the intuitiveness of their program. I am a bit unclear as the operation at large of MTurk and so it makes recognizing future work difficult. I do not know how valuable humans are as a computation resource and whether or not that this trend will tend upwards or downwards. It is nonetheless and interesting concept and TurKit makes it more accessible.