How to do Market and Customer Research With Mechanical Turk

When building products it's important to reach out to people. You must ask them what problems they are having in their day-to-day actives, so that you can implement the right solutions. That's at least what everybody is saying. It was my first time I tried to perform user research with Amazon's Mechanical Turk and SurveyMonkey.

I ran a survey of about 30 questions on Mechanical Turk to perform user research for the upcoming product I'm building. I'm very happy with the results and I have a few interesting takeaways from this whole experience which I'm going to share in this post.

First of all, you want to have most, if not all, survey questions to be required. This guarantees that people cannot skip to the end of the survey and collect rewards without actually doing any work. You want to have a survey code at the end of your survey on the last page which workers must enter into mechanical turk to collect their reward. The code ideally should be unique and mapped to individual worker ids. I decided to skip operational details and use static code in my survey.

The price per HIT is really important. Lower prices will give you short, non-quality responses or no responses at all. This is especially true if you have lots of open ended questions. It's a good idea to start with low priced HITs, collect some data and keep increasing HIT price until you start getting more/quality results, tweaking the survey along the way. I separated my survey process into daily batches to gather data.

The structure of your survey is also very important. If multiple choice questions is all you have, then the structure may not matter too much. It becomes more tricky when you have lots of open ended questions. These types of questions require more brain power to complete them and might discourage people form completing your survey if HIT price is not high enough. The survey that I ran had 3 distinct sections:

• Demographics section. Fairly standard stuff like age, gender, income. Just to get a sense of who our user is. All questions were simple multiple choice questions.
• Multiple choice questions section.
• Open ended questions section.

I chose such structure, so that I can still collect partial results even when survey takers leave without going through the entire thing. Structuring your questionnaire such way may not work for everyone. It did work pretty well for me especially when HIT price was increased to about half dollar.

HIT prices are important. Higher prices will bring you more and better results faster. You should also be generous with time allotted to complete the survey. You want people to be comfortable and not rushed.