No Secrets in Sample – Learn about InnovateMR’s Philosophy on Quality

There has been some recent buzz about a report entitled, “More Dirty Little Secrets of Online Panels.” This report, like previous ones released by the firm, Grey Matter Research, paints a rather depressing and alarming picture for the sample industry. The report suggested that up to 46% of the sample traffic furnished in today’s surveys should be deemed unacceptable; riddled with programmatic bots and other undesirable behaviors. While we don’t know which panel suppliers were used for this research, the report indicated that 5 of the 10 largest sample companies in our industry were used for the test. I would be remiss if I didn’t say that the title of this report caught my attention and troubled me greatly.

As a sample practitioner, armed with nearly 20 years in the online research space, my intention is to never mislead or keep critical information from our buyers. Instead of leading by fear and provocation, we need to have open and productive conversations about the sa

mple universe that we all greatly rely on.

If you know me personally, you know that I have been a fierce advocate for education and transparency about the threats facing our ecosystem. I have written several thought pieces and presented on countless conference stages about the topic of cyberfraud. I even provided a live tour of the dark web where I discussed the extraordinary efforts and technological sophistication employed by these nefarious actors. According to Forbes, cyberfraud is slated to cost over 6 trillion dollars annually to the world economy by the end of 2021. Plainly put, cyberfraud is never going away. Instead, it is taking down multi-billion dollar brands and government agencies every day. So there are no dirty secrets here, but some firms are doing a better job than others in fighting this battle. Instead of pointing fingers, we need to work together as a research community to understand the threats that exist and implement various strategies at every stage of our process. Relying solely on your sample provider is a naïve approach. Just like there are new strains of the virus facing humankind today, there are new permutations of cyberfraud and other undesirable behaviors that surface all the time. By nature, there will always be noise and dissonance in your data. As researchers, a scrutinizing eye must be our standard not a happenstance. As Steve Jobs once said, “Deciding what to NOT to do is as important as deciding what to do.” As such, the researcher’s role is critical in turning data into information and information into insight. So what can you do?

Don’t get scared, get smart. Take control and implement effective and proactive strategies today.

• First, evaluate your sample providers closely. Ask the tough questions and ensure that they are employing a multitude of layers to validate their recruitment channels and communities.
• Next, never forget the important role that survey design plays in producing quality insights. Review your survey questions carefully, ensure there is nothing that leads the wrong participant in. Change the blueprint for success. By nature, our surveys are instruments to collect consistent data across a universe of people, but that doesn’t mean that the survey needs to be the same for everyone. Fraudsters prey on those that make it easy and repeatable, at scale. Develop consistency checks and red-herring questions that rotate from a larger battery of questions. This solution will help you test for domain expertise and operate uniquely for each person that enters your survey. It’s like changing the lock each time a person comes through the front door.
• Remember, there is a delicate balance that must be achieved when it comes to red-herrings. If you go too simple, bots can easily decipher the code. If you create questions that are too complex, you will create unnecessary fatigue and risk losing worthy participants. One effective strategy is to test awareness of real vs. fake brands. If fake brands are selected you know these individuals don’t have the domain expertise required or at a minimum, are not right for your survey. This is especially important for surveys that offer healthier incentives or target B2B audiences.
• Next, leverage open-ends and find a solution that will help you evaluate this data in an easy and efficient way. More and more bots are demonstrating a command for the correct survey language, they provide an adequate length of interview and non-patterned responses. Where they often fall short are in the open-ends. Scrutinize these responses and be mindful of “word salad.” This occurs when sophisticated bots or human networks provide answers that technically appear correct, but upon deeper review, the words don’t quite make sense to the question at hand. Remember, context matters. InnovateMR offers a Text Analyzer™ that can help weed out poor open-end responses at scale, and flag participants for poor responses.
• Lastly, develop a plan for your data analysis. Waiting until your survey is out of field is waiting too long. Have a plan going into your research so that you can conduct interim data reviews. Create flags that can be programmed into your survey so you can evaluate the performance of participants over a series of checks while you field. This strategy will help you remain proactive so you don’t have surprises that you get you in hot water when it is too late.

Like I said before, there is dissonance in data, that is part of the fun. Think of yourself as a sleuthing detective setting out to solve a case. Take these proactive actions to be a champion for quality, not a victim of cyberfraud. And remember, cyberfraud is not going away, there is no magic bullet, none of us are impervious and there are no dirty little secrets. Let’s work together to help you create valid and compelling insights.