Amazon's Mechanical Turk has just introduced a Categorization App that should make it easier to get accurate results on categorization projects.
Given that Mechanical Turk has been going since 2005 it is a bit surprising that it has taken so long to devise an app that streamlines the process of specifying a categorization task - but then this crowdsourcing venture that recruits ten of thousands of "workers" to complete "HITs" - Human Intelligence Tasks for "Requesters" is still in beta.
The new facility is designed to help Requesters who have tasks that involve categorization. This type of project is a common one on Mechanical Turk and is often used by organizations to clean dirty data. According to the blogpost announcing the new app:
Categorization is one of the most common use cases for Mechanical Turk. A categorization HIT is one that asks a Worker to select from a list of options. Categorization projects are used by many companies for many different reasons. For instance, a retailer‘s categorization project may ask Workers to "Select the product category for this item.” A listing or directory service may ask Workers “Is this the correct web address for this business?” Or a stock photo site may ask Workers “Is this image a photo or illustration?”
The app is intended to help with every part of defining the HIT - including recruiting pre-qualified Master Worker, eliminating the step of devising a test to screen Workers before accepting them and also providing a pricing recommendation. Essentially it uses forms to go through the steps of supplying categories and providing instruction without requiring Requesters to edit HTML to add radio buttons and label.
The App also arranges that a HIT will be completed by two Master Workers because experience has shown that when two Masters agree on an answer, there is a higher level of accuracy. And where there's disagreement the instructions can be modified to clarify what is required.
The new app should make Mechanical Turk more attractive to new Requesters and, by encouraging Workers to become Categorization Masters who will receive fair remuneration based on the complexity of the task, should also appeal to Workers.
