Why we have this Process

Surveying can help prioritize the product roadmap by providing insight into user needs and feedback on the current product or product ideas. 

Context and Scope

Surveying is part of both the Discovery phase, where we try to understand user needs as well as the Design phase where we use it to get feedback on existing features or feature ideas. 

When should the process be performed?

Recommended frequency is quarterly. It's OK a survey is partly the same as last quarter. 

Who performs the process

Who performs the process: Product Manager (or Product Researcher / Marketeer) 
Who is monitors if process is done (correctly): Head of Product

Process steps

  1. Decide on survey goal.
    1. For existing products I recommend a mix of understanding customer challenges (Section 3), getting product feedback (Section 4,5) and testing out new ideas (Section 6,7 ). 
  2. Determine survey length and related compensation you can offer (see FAQ question 1)
  3. Open our survey generation template in Google Sheets. (added benefit: Many answer options are provided here too)
  4. Delete the questions you don't need.
  5. Add / adjust questions  you need. 
  6. Search for the "[" sign to ensure you removed all comments.
  7. Generate the Google Survey or Surveymonkey Survey as needed. 
  8. Test your survey yourselves. 
  9. Any fine tuning from now on do it directly in Google sheets/Surveymonkey/etc. 
  10. Ask your manager and a colleague for a review
  11. Once all review comments have been processed send out the survey invite to all our customers using our mailing tool (maybe exclude customers who filled the the last survey if it was conducted recently). 
  12. Create a reminder mail for user who haven't clicked the survey link 2, 5 and 8 days later.
  13. Close the survey after 2 weeks
  14. Generate a survey report and put in it in the Google drive under..
  15. Present the key findings in the Product Council and propose new/changed roadmap themes if appropriate. 
  16. If you promised a gift card (to qualified entries), this is the time to send those out. 

FAQ

What is the optimal length for a survey?

If you are unable to offer any compensation to the respondents than I recommend staying under 10 questions as otherwise you’ll see a steep drop in submission. Some survey software allows you to partly filled in (draft) survey responses and thus after how many questions people are dropping out.

For a paid survey (i.e. 10 Euro gift card) keep it under 30 questions. Such a survey will cost people 10–15 minutes roughly. Up to 60 questions, a compensation of 20 euros will be needed to get people to take the time. Surveys longer than 60 questions / 25 minutes are not a good idea as the answer quality will degrade rapidly. Consider splitting up your survey into certain topics that you send to only a part of your audience.

Where do I find people to fill in my survey?

The first source would be emailing your current customer base (make sure you mention the reward). You can also add a popup to your site, i.e. Mailchimp can create this signup popup for you and can send an email with a link to the survey.

How often to Survey?

As a starting point I recommend a quarterly survey to make it a bit into a habit. Teams who are blessed with a researcher should really do monthly surveys. If you are conserved about ‘over asking’ your best customers exclude participants from the last 3 months in your survey mailing.

Which questions to select for quick results, save time and do it the LEAN/Agile way?

The iterative, lean approach is to see surveys as part of an ongoing learning process. Each survey move you one step further. If you assumption about the customers, or his high level needs or a certain solution is right you can go one level deeper: I.e. within a higher level need (ie save time in a certain job) you can zoom into into the job steps to figure out where the biggest pain is. Same for solutions. If you assumptions are proven to be wrong, the more high level and open ended questions will give you inspiration about where the real need is or what a better solution could be. I recommend doing surveys quarterly in the beginning, so you turn it into a habit, you keep learning, and you reduce the pressure of creating this perfect survey the first time.

Why survey if I can Interview and Experiment?

Surveys sit between User Interviews and experiments. Interviewing will give you great qualitative insights but a few interviews do not provide enough qualitative data to base decisions on. Experiments (actually building a prototype or bit of production software) will give you the hard proof if a solution is working or not. However, experiments will take you hours to create and days to weeks to run. Surveys will effectively help you prioritize your much more limited experiments bandwidth.

Further reading

My original Medium posts: Part 1, Part 2
Some more survey questions: https://www.fieldboom.com/survey-questions/



Survey Sections and Questions

Section 1: Introduction

A survey introduction page is essential to motivate people to participate and take away any concerns that might have them drop out. For this B2B example I recommend a 10–20 euro gift card, so you take the survey up till circa 20 minutes and dig a little deeper.

Hi!
First of all, let me thank you for taking our survey. You are of great help!
We at [company name] are trying to better understand the needs of our [target audience]. This is why we chose you! The survey will take about [10 to 20 minutes] and as a little thank you we will send you a 10 Euro Amazon gift card within 2 weeks after the survey!
And don’t worry, your data is just for anonymous usage by [company name] and we take your privacy seriously.
Click continue below to get started!

Section 2: Getting to know you

Asking some questions about the respondent will help you identify what persona has a certain point point. This will make your marketing, research and product development far more effective as the customers with the biggest pain will become your most eager and helpful early MVP customers.

If your survey also targets non-customers or users of competitor products these answers will help you segment your results and gain insights if your or competitor products are actually reducing certain pains.

In this section, you also ask qualifying questions to see if the respondent is relevant to your company (i.e. does he indeed manage a team). Limit this to the max five most important questions.

  1. Are you male or female?
  2. How old are you?
  3. What is your job title?
  4. In one sentence what is your core job responsibility (i.e. “Outbound marketing for Canada”)?
  5. [Specific questions relevant to your product, i.e.] B2B: What are your company’s revenues (ranges)? How many monthly uniques visitors does your website have? Do you use process x / tool y within your company?
  6. Industry your company operates in [see the survey template for a industry list. See process step 3 for the link]
  7. Company size in terms of [employees or revenue]
  8. [Any qualifying questions to see if they are relevant to your company, ideally combine it right away with a usage question i.e.:] How often do you conduct outbound sales at your company?
  9. [Conditional question if previous answer is “I don’t do this know”] What is keeping you from doing X?
  10. What products/services do you currently use, or used before, for [process your investigating]?

Section 3: Understanding your Challenges

This section is about understanding challenges a respondent has in relation to the task, i.e. hiring a freelance developer or doing employee performance reviews. To prevent overlooking potential needs make sure you focus these question on the whole task, or the bigger outcome, i.e. hiring a freelancer, and not ‘vetting freelancers’. However also don’t make it to broad, i.e. ‘finding talent’ as this will deliver too diverse and generic pains to start with.

Section 4: Improving Our Product

This section will help you understand what current customers think your product is missing. If you don’t have a product yet (for this process) you can either leave this section out or use it to see where a competitor product is dropping the ball.

Section 5: Improving Our Product — Part 2

This section goes deeper into your current product functions to find out, in a quantifiable way, which area needs the most attention. The bigger Job/process is split into smaller steps for which each the user need is measured. This section is based on JTBD (Jobs To Be Done) style questions which are always a importance and satisfaction question pair as shown below. I recommend also recommend include the step the user has to do just before, and just after, using your service, to look for opportunities to extend your product.

Section 6: Feedback on Product Ideas

This section will show you the best way to test any feature ideas that you might have. I highly recommend splitting up any solution idea (feature) into a question about if the respondent has the related problem (need) and if he would like the solution. This allows you to identify important needs which need a better solution.

  1. How relevant is this to you? [Here goes the PROBLEM you are trying to solve, based on a insight you have on your (potential) consumers. A well-written problem speaks to what the consumer thinks, feels, or believes before he/she knows about your product. Example: When finding freelance onsite developers it is hard to find freelancers who would partially work for equity. ]
  2. If a product delivered this benefit how relevant would this be to you? [Here goes your SOLUTION idea including the benefit(s) of your product. It can be functional or emotional. When possible give proof to they will actually be able to achieve the benefit and consider adding a visual. Shortened example: Based on your requirements we will find the the right freelancer for you and negotiate a contract for you and take the paperwork out of your hands].
  3. Based on the description how would you rate the value of this product (service)?
  4. Is this idea unique and different?
  5. How likely would you share word about this idea with your peers?
  6. How much time (or money) would this feature save you? Or how much is -some outcome, i.e. a lead- worth to you? (and take 15% of that value as a price).
  7. What do you think the average [job role, i.e. broker / hiring manager] would find a fair and reasonable price for [this service product?] [This is not a great predictor nor great question, especially for uncommon/new services.]

Section 7: Feedback on Feature ideas

Let’s say using earlier research you discovered that the product area ‘selecting a freelancer/bid’ is the biggest sub-problem in your product. You can now dive deeper in this area to figure out of your hypothesis of the problems are correct and test if the respondent likes your solution. Like the previous sections splitting up problem and solution hypotheses will help you find valid user problems that just need another solution. Problems can be both functional (i.e. too something is too time consuming) as emotional (uncomfortable, embarrassing, uneasy, overwhelming, concerned, etc).

  1. How relevant is this problem to you? [Example: It costs me too much time to evaluate the years of relevant experience of the freelancers who bid on my project]
  2. How relevant is this problem to you? [Example: It don’t know how to optimally …. I don’t have the experience to optimally… It is overwhelming when… I don’t know how to best start/implement/etc]
  3. How relevant is this problem to you? [Examples: It is uncomfortable for me to / too costly to / ….]
  4. How relevant is this solution to you? [Example: Evaluate all interested freelancers for me and propose a top 3.]

In Closing

The closing page should thank the participant and provide information on how and when an eventual compensation will be handed over. It’s also a great please to recruit people onto a mailing list for your customer panel mailing list that you can use to recruit people for beta tests, in skype interviews, etc.

Thank you for completing this survey. Your feedback helps us create an even better product for you. You will receive an 10 Euro Amazon Gift Card within two weeks from today.


Questions that Don't belong in this type of survey