6 Types of Survey Respondents to Be Cautious About

6 Types of Survey Respondents to Be Cautious About

A survey is essentially a research methodology employed to collect data from a predefined group of respondents to gain insights into specific topics of interest.

For the most part, survey research aims to assess a specific market or user needs or determine whether or not particular business objectives have been met. This helps to establish baselines against which future comparisons can be made.

However, a vital ingredient that is central to the success of any survey research is appropriate respondents recruitment in order to get representative respondents for a target market. 

types of Survey Respondents

Who are survey respondents?

A survey respondent is essentially any individual who answers a survey distributed via email, mobile apps, websites, QR codes, or social media. 

Survey respondents are typically sought from samples of the population. However, it’s important to point out that surveys only provide estimates for the true population. Not exact measurements!

Why are survey respondents needed for user research?

Survey respondents can make or break a user research project. The data provided from respondents generate a number of variables that can be actionably studied. For example, the right sample size allows researchers to make fair generalisations. However, a small sample size limits their ability to identify patterns and spot trends.

As a result, finding the right survey respondents is mission-critical to the success of your survey. However, this is sometimes difficult to do. 

Survey respondents to watch out for

Generally speaking, survey respondents come in multiple forms, shapes, and sizes. This means that sometimes there are bad apples, good ones, and some in between. 

Consequently, since survey respondents aren’t one and the same, researchers have to be cautious of how their disparities might impact survey results. Here are some notable respondent types to look out for. 

Speedy respondents

Speedsters move too fast through a survey to provide thoughtful and honest answers. In all honesty, speedsters aren’t really motivated and only aim to complete a survey to receive their incentive. 

Fortunately, parameters around the length of time a respondent is required to spend on a question typically discourage this type of respondent. 

Flatliners

Sometimes called straighteners, such respondents can go one of two ways. Overly positive flatliners frequently select top box answers like “strongly agree”.

On the other hand, negative flatliners typically choose bottom box answers like “strongly disagree.” All in all, flatliners respondents typically have some sort of unwanted acquiescence bias that makes them respond in such a way. 

Cheaters

Cheaters are typically both real respondents and fake ones. Fake ones come in the form of software bots that gain access to online surveys to redeem the rewards. This is without any human having to take the actual survey.

On the other hand, real cheater respondents create multiple accounts to take the same study more than once. Or attempt to take the same study as many times as they possibly can.

To address this, most panels and platforms typically deploy technology parameters to identify and remove bots. Or simply check that respondents aren’t coming from the same IP address in multiple instances. 

Additionally, thoroughly reading through open ends as well as double-checking contact information can help catch cheaters that may have fluked through.

Posers

Posers are the hardest low-quality respondents to identify. Poser respondents, unfortunately, don’t give honest feedback. In some instances, they choose to follow group discussions because of social desirability bias. 

This means they don’t provide their true thoughts or feelings for fear of being different from the crowd. Or even fear of being different from what they assume the survey provider wants to hear.

Professionals

Professionals are survey respondents who are usually categorised as good. These survey takers frequently take different studies and treat them as a job. 

However, sometimes their consistent participation can at times lead to biased results. Especially if they are repeating studies on similar subjects. 

To avoid this, it is important to screen professionals by querying if they have recently taken a survey. Especially if it’s specific to the subject on which you’re conducting research. Then terminate them if they have.

Confused Rule Breakers

Such survey respondents have a difficult time following instructions. Some may intentionally break survey rules, while others could be misinterpreting questions. 

To avoid them, consider performing quality checks by using different screening questions. 

These would include questions that ensure that the respondents belong to the right category and also requires them to read the question before choosing the answer. 

Let us understand this with the following example:

Screening questions can be of two types: Behavioural and industry-specific

For the former, the questions include discussing certain behavioural aspects of the respondents (while also ensuring that they read the question and not just answer in yes/no).

exercise

Photo by kike vega on Unsplash

Say, for example, you’re selecting respondents for an exercising app and have multiple questions, some of which could look like these:

 How often do you exercise?

  • Once a week
  • Thrice a week
  • More than 45 minutes every day
  • Rarely

What is your preferred exercise type?

  • Aerobics
  • Swimming
  • Gym 
  • Yoga
  • Running/Jogging
  • None

Such questions naturally help you to eliminate the respondents who choose the answers “rarely”/ “none” as those who would not be using your app. These are, therefore, not the best-qualified respondents to provide accuracy to your surveys. 

Similarly for industry-specific screening, questions ensure that the respondents who are selected fit the requirements of understanding the niche for which the survey is conducted. 

Example: Do you work in any one of the following industries?

  • Teaching
  • Educator
  • Special needs educator
  • Academic writing
  • None of these

Now, for a website/app that’s being tested within the educational niche, the respondents choosing “none of these” are naturally not as qualified as those choosing the other options. 

Screening questions like these ensure that the chosen respondents are aware of your purpose, following the survey instructions and reading the questions before selecting the response. 

Respondent Recruitment Services For Representative User Insights

NX logo
NX logo

Respondent Recruitment Services For Representative User Insights

Key Highlights
  • Dedicated and experienced project manager.
  • Comprehensive briefing call with client to fully understand respondent screener criteria.
  • Screening shortlisted participants and conduct quality check.
  • Scheduling the shortlisted & qualified participants, as well as backups.
  • Liaison with the participants and reminders.
  • Recruitment expert will assist the participants throughout the entire study.

Conclusion 

In successful surveys, choosing the right respondents is critical. This is important to avoid biased opinions that could negatively influence the outcome of the research/study. The nature and accuracy of responses matter.

Remember that even when a study is well written, analysed, and executed, the outcome is only as good as its respondents.

The article is a part of our series on “How to recruit the right respondents for user research?

How to Recruit the Right Respondents for User Research?

How to Recruit the Right Respondents for User Research?

Finding the right respondents for user research studies is no easy task. Typically, researchers have to attract interested participants, carefully vet them, then schedule a time to engage them. 

To complicate issues, at times, some respondents are not good candidates for the research as they cannot deliver meaningful feedback or insight. Unfortunately, suboptimal respondents negatively impact the quality of the study research.

What is Respondent Recruitment?

Respondent recruitment is a series of activities that focuses on recruiting the right respondents for focus groups, surveys, and in-depth interviews. 

What are the differences between participants and respondents?

In practice, participants and respondents are commonly used interchangeably, so people are referring to the same. However, if we are really particular about it, there is a slight distinction, a respondent is an individual who answers/responds to questions (either written or oral). While a participant is an individual who voluntarily joins to be part of a study as a subject.

Respondents mainly ‘respond’ to the researcher’s structured and closed-ended questions. But, on the other hand, the participants go beyond simply responding to a series of questions.

For example, a participant elaborates on the researcher’s questions. Or can even change the topic if they want to convey an idea. As a result, research participants tend to provide more qualitative data than respondents.

Tips to consider when recruiting respondents for user research 

1. Understand your audience

Your respondents should suitably represent your target group. To achieve this, it’s essential to fully understand your audience. 

So, ask yourself: Who exactly are your users? Who do you seek to target? What are the actions or traits your users have? Then, based on the data you gather, you should be able to fully achieve the objective of your user research.

2. Create target groups

Respondents have different kinds of motivations, habits, and behaviours. In some instances, you need respondents who meet a specific requirement. 

For example, individuals who make food orders on a mobile phone instead of a computer. This can provide more comprehensive insight when surveying a food delivery service.

It is imperative to define what your exact criteria for respondents are. In essence, the choice of criteria typically depends on the particular requirements of your study.

This means that it’s also important to carefully create target groups in a manner that avoids underrepresentation of your market. 

3. Use screeners/ screening questions

Consider using a screener (screening questions) to sieve out the right respondents. You can achieve this by writing a short questionnaire that helps to determine whether testers are suitable. 

However, take note that long questionnaires might scare away your respondents. Remember, a single question can help you determine whether you have found a suitable respondent or not. For example, questions about the occupation, age, location, or education. 

4. Don’t include colleagues/family members

Research or testing colleagues or family members isn’t the best idea as human beings aren’t 100% rational. 

Colleagues and family members are more likely to have inherent biases and can get emotional. Therefore, will typically lack the objectivity to deliver actionable insight.

5. Pay attention to how you compensate

Simply put, money matters because respondents who feel fairly compensated rarely give much trouble. They don’t get grumpy about multiple phone calls, long screeners, or not qualifying at times. So, whenever they are eligible to participate, they are more engaged and excited. 

6. The recruiting process can deliver some actionable research insights. 

The recruitment process can reveal unexpected insights that could be useful. Because of this, researchers should always be open to learning and retooling the recruiting process accordingly, to get better insights.

7. Repeat respondents can sometimes be good. 

Individuals with good experience as respondents tend to be more reliable. Additionally, they understand that the process isn’t about seeking positive points of view, rather honest opinions based on real-life experiences. 

However, whether repeating a respondent will really depend on your user research objective. 

For example, repeated respondents will not be suitable if the objective is to validate the usability of a mobile app interface for first time users, since they have already seen the mobile app user interface in the previous study. 

How do I get respondents for user research?

1. Solicit website respondents

Similar to generating email leads, sourcing respondents on your website is a good idea. Visitors to your website can be politely asked to opt into a survey if they have extra time.

This can be achieved via a landing or squeeze page because the visitor has fewer decisions to make. However, on the web homepage, visitors have hundreds of distractions such as videos, or product demos. But a landing page isolates the call to action for your survey.

2. Pre-qualification calls

If you have collected a list of prospective respondents, the subsequent step is to determine whether they meet your criteria. 

Unfortunately, some behavioural qualities aren’t easy to determine over an online form. However, speaking to these individuals on the phone always takes it a step further and reveals more.

For example, you can determine whether an individual can articulate their thoughts well. Or even quickly ask their views on a general topic like — “Is Starbucks coffee overpriced?”

3. Use a screener questionnaire

A screener questionnaire is a series of approximately 10–12 questions with different sequences of branching logic and termination points. The idea is that if, at any point during the screener call, the individual doesn’t qualify for your criteria. Then you must politely end the call.

However, specific things have to be handled while designing an effective screening questionnaire. The most critical point to remember is that you’re screening for behaviours, not demographics. Another vital point with screener questionnaires is to ensure that the recruitment criteria aren’t revealed to the potential respondent prior.

screener questionnaire

Photo by Alex Green from Pexels

4. Hire a recruitment agency

One way to avoid many recruitment hassles is to simply solicit the services of a recruitment firm. Most recruitment agencies have their own lead-time to recruitment services. 

But, remember to carefully review their screening question list to avoid mistakes that could lead to the wrong respondents for your research initiative.

5. Use social media channels

You can share your survey widely across different social media platforms to find respondents. 

Social sharing gives you a lot of visibility, depending on your topic. For example, your topic might suit a LinkedIn audience, since it’s dedicated to professional networking and job seekers. 

Or your topic might fit well with a Facebook group that regularly discusses Agriculture. So, it is imperative to use your discretion to decide which platform best serves your objectives.  

6. Utilise online panel vendors

Many recruitment vendors offer an online service to acquire participants. However, in some cases, their pricing could be high. If you decide to engage any, carefully check participants’ demographics and psychographics within their databases to ensure it’s up-to-date.

7. Use dedicated panels

Relatedly, dedicated panels are fundamentally large databases of prospective research participants. Typically, for a fee, an organisation can screen and recruit the participants they need for your research. 

In some cases, dedicated online panels will deal with the logistics of paying out incentives. However, like most online panels, they often maintain many professional research participants. This can work against you if you want a variety and objective results.

So, ensure to utilise a well-crafted screener technique, and prepare to deal with participants who might not fully represent your target users.

8. In-person polling

This approach involves physically meeting your potential participants where they are. For example, if you’re looking at a specific target audience situated at particular locations, go meet them where they are. 

Especially, if you’re seeking feedback from a specific demographic less likely to respond online. For instance, demographics around a supermarket branded store or a physical location. Also, depending on participants’ qualifications, you can set up a table at a local trade fair, or conference to get respondents for your qualitative research.

9. Employ third-party channels and partners

If you have marketing partners, ask them if you can use their networks or marketing channels to contact potential survey participants. Most marketing resellers and suppliers will have access to multiple social media accounts and marketing systems with prospective participants.

How to get respondents for an online quantitative survey?

  1. Organic website traffic- depending on the nature of your study, you can host a survey invitation on your web homepage. Then allow prospective survey respondents to choose whether they qualify to take your survey.
  2. Random Device Engagement. 
  3. Use online community platforms like Reddit, Craigslist, QuestionPro Communities, etc.
  4. Share your survey on your website, social media, or blogs.
  5. Hire a dedicated Market Research agency
  6. Utilise your existing customers’ database since they already have first-hand experience using your platform. 
  7. Send surveys via email campaigns

Respondent Recruitment Services For Representative User Insights

NX logo
NX logo

Respondent Recruitment Services For Representative User Insights

Key Highlights
  • Dedicated and experienced project manager.
  • Comprehensive briefing call with client to fully understand respondent screener criteria.
  • Screening shortlisted participants and conduct quality check.
  • Scheduling the shortlisted & qualified participants, as well as backups.
  • Liaison with the participants and reminders.
  • Recruitment expert will assist the participants throughout the entire study.

Conclusion

The success of user research depends on its respondents. More precisely, on one’s choice of respondents. This means that finding the right participants is vital to extracting valuable results from your user research. 

Remember that participants have to be able to fairly represent your end-users. So, carefully consider the criteria you have for your research participants before commencing recruitment. For example, when your criteria are quite general — like age and location — recruiting participants is considerably easy. 

However, when you have more particular requirements for candidates to participate in your study, it can be very difficult and time-consuming. Though by balancing extreme and mainstream users, you can improve your chances of getting objective and appreciable research outcomes.

If you require focus groups, surveys, or in-depth interviews, our experienced team will handle the entire recruitment process, saving you time and effort. Contact us today to learn more about how Respondence Recruitment Services can enhance your user research projects.

How To Conduct Usability Testing For Websites?

How To Conduct Usability Testing For Websites?

Theoretically, an ideal website should be immediately clear to new visitors, easy to use (even after long breaks from the site) and error-free. All these while delivering a pleasant experience for all users. 

While this might seem obvious, in practice, it’s quite challenging for many organisations to achieve. Unfortunately, very few web interfaces are perfect as many websites are typically poorly laid-out, therefore making user navigation counter-intuitive. 

For instance, their web components comprise poorly labelled buttons, or unclear prompts, which frustrate new or returning users. One way to avert such problems is usability testing. 

Usability testing for websites

What is usability testing?

To accurately define usability testing, we need to first define usability. Usability can be generally defined as the degree to which a device or software assists a person to accomplish a specific task. This is in contrast to becoming an additional impediment to the task’s accomplishment. 

That being said, usability testing is a set of non-functional testing techniques employed to measure how easily a software system or hardware device can be utilised by end-users.

What is website usability?

Website usability testing, also sometimes referred to as User Experience (UX) testing or user testing, involves using testing techniques to measure how easy, intuitive, and user-friendly a software application is. 

Website usability test revolves around exposing usability defects by determining users’ ease of using an application, the flexibility of the controls, and the capacity of the application to meet its objectives. 

The human-computer interaction characteristics of software are measured, and weaknesses are identified for correction.

As such, usability refers particularly to how well people engage with a website.

Jakob Nielsen, probably the most famous usability expert, believes that usability constitutes five key components: learnability, efficiency, memorability, minimisation of errors, and satisfaction. These can be broken down as: 

  • Learnability: How easy is it for website users to accomplish basic tasks the first time on the site.
  • Satisfaction: How pleasant and easy is it to use the website?
  • Efficiency: Once visitors have learned the design, how quickly can they perform tasks?
  • Memorability: How easily can return users re-establish proficiency? 
  • Error rates: How many, severe and permanent are user errors? 

Why is website usability testing needed?

Usability tests are typically conducted for multiple reasons, namely:

  • To determine the time it takes to complete a task compared against established benchmarks.
  • To ascertain user satisfaction by understanding user pain points to come up with applicable design solutions to improve user performance.
  • To evaluate if users can navigate your website. 
  • To identify potential problems with website functionality.
  • To establish if a website is accomplishing an organisation’s goals.

Overall, the primary goal of usability testing is to ensure that people can utilise your website.

Website usability testing methods

There are numerous methods and techniques for usability testing, namely:

Comparative Usability Testing

In comparative usability testing, end users are essentially required to choose the best of two or more options. It’s principally performed to check the effectiveness of a website against that of its competitors to identify any shortcomings, then address them.

It involves observing metrics like error rates, task completion, and time spent on each specific task.

Explorative Usability Testing

Unlike comparative tests, explorative usability tests are more open-ended as participants are encouraged to give their opinions and explain how they feel about a specific design or user flow.

It is usually performed during the initial stages of website development to judge how users might react to the design. This is to create a fully informed process with users’ exact needs in mind.

Cognitive Walkthrough

This testing approach is designed to assess how a website interface supports first-time users while they learn how to complete a task. 

The designers and developers basically imagine the possible steps that would be performed by a user to complete a particular task. They then evaluate the website system’s responses to those tasks. 

Focus Groups 

Here, a small group of people (participants) sit around a table and interactively react to ideas and designs that are shown to them.

Heuristic Evaluation

In this testing approach, testers examine a website to judge how well it conforms to standard usability principles. Evaluators systematically compare a website against a set-list of defined criteria to determine how closely its design follows recognised usability principles.

Think Aloud Testing

The thinking aloud approach entails observing test users’ interactions with the website as they navigate through the website, and vocalise their thoughts.

In practice, users complete a particular task, vocalising their thoughts as they navigate while explaining why they performed each action and what they are looking for. As such, anything that seems confusing or frustrating is marked upon and recorded for action.

Remote Usability Testing

In this testing approach, the testers and users could be in different countries and time zones. As such, remote testing is, at times, performed using video conferencing software, while other times, the website user works separately from the evaluator. 

Currently, remote testing software allows remote usability testing to be performed even by observers who are not usability experts. Usually, this software automatically records the click locations and streams of the users. 

As well as any critical incidents that happened while they were exploring the website, along with any actionable feedback the user submits.

Guerrilla Testing

The guerilla testing approach of a usability test is probably the cheapest and simplest of the lot. In practice, it involves gathering feedback from random people.  

Sometimes, this method involves paper prototyping as an alternative to creating a functional prototype website. Overall, guerilla testing is a time-efficient technique since you do not have to recruit qualified or vetted participants, then wait for their responses.

Read more in our article on “Guerrilla testing

Phone Interview Testing

As a usability testing approach, phone interviews involve a researcher asking questions to testing participants. It could also be done by instructing them on how to perform a specific task over the phone. In turn, the participants give feedback regarding the website in question.

Overall, this technique is helpful when reaching out to participants and gathering data from wider geographic areas. This is because it allows you to obtain a broader perspective of the website’s potential issues.

Steps to conduct website usability testing

1. Clearly determine metrics and create task analysis

It is imperative to figure out the precise metrics you’ll consider during your testing. Since usability testing can uncover a host of issues, if it’s not being targeted around specific metrics, it won’t be an effective use of your time or money. 

As such, ensure to clearly state what you seek to achieve with the usability testing and the exact information you hope to gather. For instance, if you intend to know if website users can order a product successfully, then you should test the entire process from the front page to completing the order. 

Overall, usability metrics play a crucial role as statistics to measure a user’s performance on a given set of tasks. For example: 

  • Success Rate: This judges whether the user was able to complete the task. 
  • Error Rate: This metric looks at the errors tripped up by users most. They can be divided into either: critical and noncritical. In practice, critical errors prevent a user from completing a task. While noncritical errors lower the efficiency with which they complete tasks.
  • Time to Completion: This measures the amount of time a user takes to complete a task.  
  • Subjective Measures: This metric numerically ranks a user’s self-determined satisfaction, ease-of-use, information availability, etc.

2. Identify the best applicable test method

Usability testing takes multiple forms that come in a range of difficulty and investment requirements. That being said, it is important to determine the best type of test for your website depending on the metrics and tasks you have curated in step one.

After you set the test’s goal, decide which usability testing method is the most suitable. Also, consider the resources you have to do a website usability test.

3. Create task scenarios and set your success rate

Ensure to create different task scenarios that participants should accomplish. For example, a series of tasks related to the aforementioned product ordering example we’ve used in step two. 

4. Find suitable participants

From the outset, it is important to carefully determine the number of users you need to conduct a satisfactory usability test. Typically, the recommended industry standard is a minimum of five users.

As you source users, ensure that they are fair representations and approximations to real users. In practice, not validating website design changes with a unique user base can have some drastic implications to your site’s usability. 

5. Decide when, where, and who

Essentially, this step involves choosing between two key decisions: 

  • Remote or in-person testing?
  • Moderated or unmoderated testing?

Website usability testing checklist

As you embark on your usability testing initiative, ensure that you check this checklist to increase your chances of success: 

  • Written website goals.
  • Written tasks users take to accomplish goals. 
  • Written scenarios and situations in which users would engage in these tasks.
  • A list of the usability metrics you will capture during the test.
  • A list of selected test methods. 
  • What point(s) during your project will your tests take place?
  • Number of testers you will need if a moderator is involved.
  • If you are conducting an online test or a paper test? 

Conclusion

Usability testing is an absolutely necessary activity for any business with an online presence.  As we have established, the three key metrics that are important for usability testing are satisfaction, efficiency, and effectiveness. 

Fundamentally, these metrics will determine the best type of usability test to perform for your website.

That being said, usability testing should be more than checking a list of product requirements, but an exercise to support your website design decisions.

Overall, by listening to users and understanding how they interact with your website, you can avoid spending unnecessary time and resources. And, in turn, better serve your users. Remember, usability testing should principally determine whether a website is:

  • Useful
  • Findable
  • Accessible
  • Usable
  • Desirable

The article is a part of our comprehensive series on “Usability testing”.