Blog‎ > ‎

mTurk in Organizations

posted Jan 10, 2012, 4:01 AM by Evan Morrison   [ updated Jan 10, 2012, 4:09 AM ]
I was excited after the last post, so I thought to myself, can I repeat this process and do the timing. I had a second collection of turk responses to another question asking if it was possible run a business using mTurk.

My own answer to the question: "Is it possible to run a business using micro jobs? What are the short comings? How could it become possible? What sort of businesses would benefit from micro jobs? "

Started 10:26pm
I believe that it is possible to process low-level jobs using micro workers. In fact, for the last month I have been deep in thought over the potential architecture of a system of workers where content is first tagged, processed into meaningful statements and then low level step based activities performed. The system I have in mind is a choose your own adventure style application that is executed by a collection of workers and whose result is then verified by a new set of workers. The output of such a system is the execution scenario that has the most feed back. However, there are issues in this model. In my early experiments with mTurk, I'm finding that there is a need for some filters. Firstly, I have found that having one text input is not enough. Each collection form needs to have multiple entry points. I believe this should force the user to think about what they're doing. Secondly, limiting text entry is extremely beneficial. I have found that generally over a point of about 400 words the average turk response is spammy garbage. Thirdly, I am also particularly interested in running a check of the incidence of coincidence over the text (http://en.wikipedia.org/wiki/Index_of_coincidence) to see if I can throw out some of the harder to edit results. And finally to make this achievable I've pondered the thought of hiring a freelance worker to oversee the results so that I only receive the final edited execution instances.

I know; I know. The final step is cheating. So to make it more micro-jobish, I would have to ask the free lancer to describe his process for assessing quality to re-encode the checking procedure into further micro jobs.

The short comings that I have faced so far include in my first attempt leaving the qualifications of workers open. Of the few jobs I allowed to come through using open qualifications, I found that I rejected approx. 80% (simple writing tasks). I have been increasing the qualification level and have found that the number of hits completed is not a bad thing so far. However, the number of approved jobs should be around 60%. Next I have found that paying more than 5c means that I get quicker results. For some of the more technical questions I've asked, this speed is necessary (otherwise they sit unanswered for weeks). For simple opinions, 5c is adequate, though I am interested in trying to set the price lower but with more tasks. I find the interface arduous to use quickly and generally have to spend about 15 minutes a day of my time clicking through each hit and then approving or rejecting work. Amazon really has a lot to do to make the system interface simpler. If I was going to do more work on the system, I would use the API to tap into the functionality to get better reports.

In terms of benefits, I feel more confident that if you are able to clearly express your requirements, then there is a possibility of getting some terrific input. For the courier question from my previous blog, I was hugely impressed with some of the interesting stories that the turkers gave. I don't believe that if I was writing for 30 minutes that I would have come up with nearly as many scenarios as they had.

And now, as the clock ticks away it is 10:48pm. Time to do my own editing and come back. I think that in comparison for 30minutes of work. If it was possible to speed up the editing time for the work that mTurk then there is a positive future in using micro jobs. Where we could complete large projects using some divide and conquer methodology. Perhaps one day, we could run an entire enterprise in the cloud. 
End of editing 11:01pm

The result of reformatting and editing results in (approx 20minutes of editing):

It is truly possible to run a business using mTurk. There are no short comings, you will just want to have an in-house team of experienced researchers or analysts who can analyze and process the flow of information. Businesses that do SEO research (keywords, meta tags writing), content creation and website image processing(tagging) would undoubtedly benefit from micro job workers.

I think we can make money by using micro jobs, by taking large works and by using mechanical turk you can divide the project to the qualified peoples. There are a lot of workers from all around the world so your works can be done in hours. For starting businesses using mechanical turk, I think surveys, voting, facebook, etc, is a excellent place to start. It can help you become established with a fantastic website and attract more business. If you can provide steady work, there will be more people waiting for your work. The main factor in mechanical turk is the payment, the more you spend more work can be done in less time.

Businesses that service web based companies can benefit tremendously from MTURK and if the "requester (you in this HIT)" remains fair and understanding. The business can get fast and accurate service. However, if the requester chooses to be unfair, then they quickly earn a negative rep on turkopticon and on Turker Nation which would cause quality workers to stay away. Businesses that need data collected or information verified can benefit. The short comings are: you may not always receive quality work. Just like there are scam requesters, there are scam workers. How could it be possible? Hard to answer or I would have done it. Just hard work to get the business, positive feedback and worthy pay towards workers, and good structure.

It is, look at CastingWords, SpeechInk, DoloresLabs. The shortcomings are: you can't be sure you'll get the job done fast, unless you pay a lot; there will be some cheaters, and you have to deal with it (not a significant problem though); you have to check the quality all the time, because most of the the mturkers are Indians, most of the other parts are not native English speakers. Obviously, it's something where you can split tasks between a lot of not-so-qualified workers, e.g. audio transcribing, filling spreadsheets etc. Verifying of results could be done by turkers too.

Yes it is certainly possible to run business micro jobs. When the business needs a large collection of data or they need a job done within a short period without manpower cost and so on. It is possible to run business using mechanical turk.

From (cost $2.75)

very much possible. no short comings, you will just need to have an in-house team of expert researchers or analysts who can analyse and process the flow of data. Businesses belonging to the seo research (keywords, meta tags writing), content creation and website image processing( tagging) would definately benefit from microjobs...

  i think we can make money by using microjobs, by taking large works and by using mechanical turk u can devide the job to the qualified peoples. there are lot of workers from all around the world in mechanical turk so your works can be done in hours. For starting business in mechanical turk i think surveys, voting, facebook like, vedio like ...etc, is good to start at the first stage and after that get established with a good website and attract more business.if u can provide continuous work there will be more people waiting for your work. main thing in mechanical turk is the payment more u pay more work can be done in less time. best wishes..........
  
   Business that service web based companies can benefit greatly from MTURK and if the "requester (you in this HIT)" remains fair and understanding, the business can provide fast and accurate work. However if the requester chooses to scheme, quickly they gain a negative rep on turkopticon and on Turker Nation which would cause quality workers to stay away. Business that need data collected or information verified can benefit. Short comings = You may not always receive quality work. Just like there are scam requesters, there are scam workers. How could it becomes possible? Hard to answer or I would have done it. Just hard work to get the business, positive feedback and worthy pay towards workers, and good structure.
   
    It is, look at CastingWords, SpeechInk, DoloresLabs. Shortcomings are: you can't be sure you'll get job done fast, unless you pay a lot; there will be some cheaters and you have to deal with it (not a very big problem though); you have to check the quality all the time, 'cause most of mturkers are indians, most of the other part are not native English speakers too. Obviously it's something where you can split tasks between a lot of not-so-qualified workers, e. g. audio transribing, filling spreadshits etc. Verifying of results could be done by turkers too.
    
     Yes definitely possible to run business microjobs. Where the business needs huge collection of data or to get job done within short period without manpower cost and so on. it is possible to run business using mechanical turk.
Comments