Reading #21: Human model evaluation in interactive supervised learning

References
Human model evaluation in interactive supervised learning by Rebecca Fiebrink, Perry R. Cook, and Daniel Trueman. Published in the CHI '11 Proceedings of the 2011 annual conference on Human factors in computing systems


Author Bios:
Rebecca Fiebrink is an assistant professor in Computer Science at Princeton University. 
Perry R. Cook is a professor at Princeton University.
Daniel Trueman is a professional musician.


Summary:
Hypothesis:
Finding the criteria for a model that is most important to a user will help develop better interactive machine learning systems.


Methods:
Several subjects were studied while they participated in some learning work with a system called The Wekinator. This allows for the subjects to train the system given certain input, usually gestures. The authors created three different studies to test this system. The first was to have several composers study the system and after using it for a set period of time, gave feedback to the researchers. The second experiment involved children from 1st to 4th year. They were told to create two interfaces on the machine, one was interaction based, the other duration based. The final experiment involved a cellist and teaching the machine to track and record the motions of the bow correctly.


Results:
Participants from the first study complained about the controls. Commenting that they were confusing to use and not intuitive. Cross validation was a feature that was only used in the latter two studies, and in those it was indicated to be of high importance. But participants from all studies used direct rather than cross validation most of the time.


Contents:
Here the authors observe how people interact with a machine learning system.  They discuss the different ways in which people work with the system, and which ones are most effective and why.


Discussion:
An interesting paper. I think something like this could definitely have cool applications down the line. Anything having to do with machine learning is something that i really feel can make a huge impact. Being able to designate tasks to a machine could lead to some cool developments in machines assisting humans.