Why we have this Process
Surveying can help prioritize the product roadmap by providing insight into user needs and feedback on the current product or product ideas.
Context and Scope
Surveying is part of both the Discovery phase, where we try to understand user needs as well as the Design phase where we use it to get feedback on existing features or feature ideas.
When should the process be performed?
Recommended frequency is quarterly. It's OK a survey is partly the same as last quarter.
Who performs the process
Who performs the process: Product Manager (or Product Researcher / Marketeer)
Who is monitors if process is done (correctly): Head of Product
- Decide on survey goal.
- For existing products I recommend a mix of understanding customer challenges (Section 3), getting product feedback (Section 4,5) and testing out new ideas (Section 6,7 ).
- Determine survey length and related compensation you can offer (see FAQ question 1)
- Open our survey generation template in Google Sheets. (added benefit: Many answer options are provided here too)
- Delete the questions you don't need.
- Add / adjust questions you need.
- Search for the "[" sign to ensure you removed all comments.
- Generate the Google Survey or Surveymonkey Survey as needed.
- Test your survey yourselves.
- Any fine tuning from now on do it directly in Google sheets/Surveymonkey/etc.
- Ask your manager and a colleague for a review
- Once all review comments have been processed send out the survey invite to all our customers using our mailing tool (maybe exclude customers who filled the the last survey if it was conducted recently).
- Create a reminder mail for user who haven't clicked the survey link 2, 5 and 8 days later.
- Close the survey after 2 weeks
- Generate a survey report and put in it in the Google drive under..
- Present the key findings in the Product Council and propose new/changed roadmap themes if appropriate.
- If you promised a gift card (to qualified entries), this is the time to send those out.
What is the optimal length for a survey?
If you are unable to offer any compensation to the respondents than I recommend staying under 10 questions as otherwise you’ll see a steep drop in submission. Some survey software allows you to partly filled in (draft) survey responses and thus after how many questions people are dropping out.
For a paid survey (i.e. 10 Euro gift card) keep it under 30 questions. Such a survey will cost people 10–15 minutes roughly. Up to 60 questions, a compensation of 20 euros will be needed to get people to take the time. Surveys longer than 60 questions / 25 minutes are not a good idea as the answer quality will degrade rapidly. Consider splitting up your survey into certain topics that you send to only a part of your audience.
Where do I find people to fill in my survey?
The first source would be emailing your current customer base (make sure you mention the reward). You can also add a popup to your site, i.e. Mailchimp can create this signup popup for you and can send an email with a link to the survey.
How often to Survey?
As a starting point I recommend a quarterly survey to make it a bit into a habit. Teams who are blessed with a researcher should really do monthly surveys. If you are conserved about ‘over asking’ your best customers exclude participants from the last 3 months in your survey mailing.
Which questions to select for quick results, save time and do it the LEAN/Agile way?
The iterative, lean approach is to see surveys as part of an ongoing learning process. Each survey move you one step further. If you assumption about the customers, or his high level needs or a certain solution is right you can go one level deeper: I.e. within a higher level need (ie save time in a certain job) you can zoom into into the job steps to figure out where the biggest pain is. Same for solutions. If you assumptions are proven to be wrong, the more high level and open ended questions will give you inspiration about where the real need is or what a better solution could be. I recommend doing surveys quarterly in the beginning, so you turn it into a habit, you keep learning, and you reduce the pressure of creating this perfect survey the first time.
Why survey if I can Interview and Experiment?
Surveys sit between User Interviews and experiments. Interviewing will give you great qualitative insights but a few interviews do not provide enough qualitative data to base decisions on. Experiments (actually building a prototype or bit of production software) will give you the hard proof if a solution is working or not. However, experiments will take you hours to create and days to weeks to run. Surveys will effectively help you prioritize your much more limited experiments bandwidth.
My original Medium posts: Part 1, Part 2
Some more survey questions: https://www.fieldboom.com/survey-questions/
Survey Sections and Questions
Section 1: Introduction
A survey introduction page is essential to motivate people to participate and take away any concerns that might have them drop out. For this B2B example I recommend a 10–20 euro gift card, so you take the survey up till circa 20 minutes and dig a little deeper.
First of all, let me thank you for taking our survey. You are of great help!
We at [company name] are trying to better understand the needs of our [target audience]. This is why we chose you! The survey will take about [10 to 20 minutes] and as a little thank you we will send you a 10 Euro Amazon gift card within 2 weeks after the survey!
And don’t worry, your data is just for anonymous usage by [company name] and we take your privacy seriously.
Click continue below to get started!
Section 2: Getting to know you
Asking some questions about the respondent will help you identify what persona has a certain point point. This will make your marketing, research and product development far more effective as the customers with the biggest pain will become your most eager and helpful early MVP customers.
If your survey also targets non-customers or users of competitor products these answers will help you segment your results and gain insights if your or competitor products are actually reducing certain pains.
In this section, you also ask qualifying questions to see if the respondent is relevant to your company (i.e. does he indeed manage a team). Limit this to the max five most important questions.
- Are you male or female?
- How old are you?
- What is your job title?
- In one sentence what is your core job responsibility (i.e. “Outbound marketing for Canada”)?
- [Specific questions relevant to your product, i.e.] B2B: What are your company’s revenues (ranges)? How many monthly uniques visitors does your website have? Do you use process x / tool y within your company?
- Industry your company operates in [see the survey template for a industry list. See process step 3 for the link]
- Company size in terms of [employees or revenue]
- [Any qualifying questions to see if they are relevant to your company, ideally combine it right away with a usage question i.e.:] How often do you conduct outbound sales at your company?
- [Conditional question if previous answer is “I don’t do this know”] What is keeping you from doing X?
- What products/services do you currently use, or used before, for [process your investigating]?
Section 3: Understanding your Challenges
This section is about understanding challenges a respondent has in relation to the task, i.e. hiring a freelance developer or doing employee performance reviews. To prevent overlooking potential needs make sure you focus these question on the whole task, or the bigger outcome, i.e. hiring a freelancer, and not ‘vetting freelancers’. However also don’t make it to broad, i.e. ‘finding talent’ as this will deliver too diverse and generic pains to start with.
- What are you top 5 responsibilities / processes you are responsible for as a [job role] in just a few words per area? [ If your product doesn’t address one of these top 5 processes it not a so call “hair on fire” problem and it will be difficult find buyers. ]
- How often do you [process you’re improving] per week? [Or: How much time do you spend on [process you’re improving] [per…]?]
- Do you find it hard to [(new) process]? [It’s easy to do and time required is minimal, Somehwat hard / time intensive, Hard / Time intensive, Very challenging to get the results I want]
- When trying to [find a freelance marketer] what is important to you? And what would be the kind of results you are looking for?
- Think back to the last time you did [process]. What’s the hardest part about [process / achieving related outcome]?
- What challenges did you run into the last time you did [process]?
- What, if anything, have you done to solve that problem?
- In short, How are you currently doing [this process], i.e. steps/tools?
- What are your most important goals in [process your solution supports, i.e. writing project proposals, hiring freelancers, giving employee feedback]?
- What was happening in your world that led you to sign up for the product/service you are currently using for [process]?
- What happened during your [trial / or first time use in case you don’t have trail] that convinced you [PRODUCT] was the right solution?
- What were you skeptical or anxious about when you signed up, that would have prevented you from using [PRODUCT] long-term?
- Did you try or use a similar service before? (no, I tried these but stop using them, I actively used these)
Section 4: Improving Our Product
This section will help you understand what current customers think your product is missing. If you don’t have a product yet (for this process) you can either leave this section out or use it to see where a competitor product is dropping the ball.
- How likely is it that you would recommend [Product] to a friend or colleague? [0–10] [This is the famous NPS question, to calculate the NPS score do not average the score but use this formula: https://goo.gl/33sZnq
- How would you feel if you could no longer use [your product]? [This the Product Market fit formula. 40% very disappointed is considered market fit] (Very disappointed / Somehwat disappointed / Not disappointed / Not applicable — I no longer use your service.
- What is the primary benefit that you have received from our product, and in general what do you love most about our product?
- Have you achieved [core benefit of your product] using [Product]? [i.e. hired a freelancer, got insight into customer satisfaction, etc] (scale)
- What motivates you to continue using this product?
- How quickly is the product evolving?
- How satisfied are you with customer support and working with the company? [The last two question indicate churn risk. It’s also a great diagnostic tool for exposing overselling. Do the salespeople in the company overpromise to attain quota, only for the product to disappoint later on?]
- What’s the hardest part about using this product?
- How would you improve our [product/service]? What might improve your experience using the product?
- What would make you want to tell your peers about this product?
- If you were not allowed to use our service, how would you do [process]?
Section 5: Improving Our Product — Part 2
This section goes deeper into your current product functions to find out, in a quantifiable way, which area needs the most attention. The bigger Job/process is split into smaller steps for which each the user need is measured. This section is based on JTBD (Jobs To Be Done) style questions which are always a importance and satisfaction question pair as shown below. I recommend also recommend include the step the user has to do just before, and just after, using your service, to look for opportunities to extend your product.
- [Example sub-need/outcome: Setting the optimal project budget (other examples: Writing the project brief]
- How important is this to you? 1. Not at all important / 2. Slightly important / 3. Moderately important / 4. Very important / 5. Extremely Important. [There is science behind these two scales, I recommend to keep them as is]
- How satisfied are you with the way you are currently doing this (with our product or otherwise)? 1. Completely dissatisfied / 2. Mostly dissatisfied / 3. Somewhat dissatisfied / 4. Neither satisfied nor dissatisfied / 5. Somewhat satisfied / 6. Mostly satisfied / 7. Completely satisfied
- [Example sub-need/outcome: Selecting the best freelancers by scoring their relevant experience, certificates, customers ratings, etc] (and then the same 2 questions as above (importance and satisfaction))
- How important is this to you?
- How satisfied are you with the way you are currently doing this (with our product or otherwise)?
- [Example sub-need/outcome: Negotiating price and contract terms with my preferred freelancer(s). Another example: Interview / question freelancers in order to judge them]
- How important is this to you?
- How satisfied are you with the way you are currently doing this (with our product or otherwise)?
Section 6: Feedback on Product Ideas
This section will show you the best way to test any feature ideas that you might have. I highly recommend splitting up any solution idea (feature) into a question about if the respondent has the related problem (need) and if he would like the solution. This allows you to identify important needs which need a better solution.
- How relevant is this to you? [Here goes the PROBLEM you are trying to solve, based on a insight you have on your (potential) consumers. A well-written problem speaks to what the consumer thinks, feels, or believes before he/she knows about your product. Example: When finding freelance onsite developers it is hard to find freelancers who would partially work for equity. ]
- If a product delivered this benefit how relevant would this be to you? [Here goes your SOLUTION idea including the benefit(s) of your product. It can be functional or emotional. When possible give proof to they will actually be able to achieve the benefit and consider adding a visual. Shortened example: Based on your requirements we will find the the right freelancer for you and negotiate a contract for you and take the paperwork out of your hands].
- Based on the description how would you rate the value of this product (service)?
- Is this idea unique and different?
- How likely would you share word about this idea with your peers?
- How much time (or money) would this feature save you? Or how much is -some outcome, i.e. a lead- worth to you? (and take 15% of that value as a price).
- What do you think the average [job role, i.e. broker / hiring manager] would find a fair and reasonable price for [this service product?] [This is not a great predictor nor great question, especially for uncommon/new services.]
Section 7: Feedback on Feature ideas
Let’s say using earlier research you discovered that the product area ‘selecting a freelancer/bid’ is the biggest sub-problem in your product. You can now dive deeper in this area to figure out of your hypothesis of the problems are correct and test if the respondent likes your solution. Like the previous sections splitting up problem and solution hypotheses will help you find valid user problems that just need another solution. Problems can be both functional (i.e. too something is too time consuming) as emotional (uncomfortable, embarrassing, uneasy, overwhelming, concerned, etc).
- How relevant is this problem to you? [Example: It costs me too much time to evaluate the years of relevant experience of the freelancers who bid on my project]
- How relevant is this problem to you? [Example: It don’t know how to optimally …. I don’t have the experience to optimally… It is overwhelming when… I don’t know how to best start/implement/etc]
- How relevant is this problem to you? [Examples: It is uncomfortable for me to / too costly to / ….]
- How relevant is this solution to you? [Example: Evaluate all interested freelancers for me and propose a top 3.]
The closing page should thank the participant and provide information on how and when an eventual compensation will be handed over. It’s also a great please to recruit people onto a mailing list for your customer panel mailing list that you can use to recruit people for beta tests, in skype interviews, etc.
Thank you for completing this survey. Your feedback helps us create an even better product for you. You will receive an 10 Euro Amazon Gift Card within two weeks from today.
- Can we keep you in the loop on how our product develops? If you answer is yes please fill in your email here and we’ll keep you posted. You can unsubscribe at any time and we will keep your email private.
- Can we approach you for future beta tests, interviews and surveys? If so fill your email address below so we can reach out to you. [Or Alternatively:]
- We are looking for a few people to participate in a follow up skype conversation about how they shop for art (30 minutes long). If you are picked, you will be compensated for your participation (20 Euro Amazon Gift Card). If you would like to participate, please fill your email address below so we can reach out to you.
Questions that Don't belong in this type of survey
- What problem were you trying to solve when you initially came across our product or service? [This question belongs in a short post signup survey you send directly to new customers 1 hour after they sign up, so they have a better recollection of what they were looking for]
- How much would you pay for service X? This question doesn't result in reliable answers. It's better to just try out what people would actually pay by either increasing your price for small customer group (please don't pick the germans ) , or for new product make a (fake) payment page and see if people pay. You could ask "How much time (or money) would this feature save you? Or how much is -some outcome- worth to you? (and take 15% of that as a price).