My colleague Justin Sutton recently put his finger on something important: fraud isn’t the only major data quality risk in market research today – disengagement is also a big concern.

The data quality conversation has long centered on bad actors — fraudulent respondents, bots, duplicate completes. And while those problems are very real, they've overshadowed a more stubborn issue: legitimate respondents who are technically qualified but mentally checked out. Real people. Real answers. Just not real thinking.

Justin framed it well: we've spent years learning to manage the gap between what people say and what they do. Now we have to manage a second gap, the one between what people say and what they mean.

That second gap is what this blog is about. Specifically: how do we actually detect it?

The Problem with "Looks Fine to Me"

Low-effort responses are insidious precisely because they don't look broken. A respondent who's moving quickly through a survey to collect their incentive will still complete in a reasonable timeframe, still pass attention checks, still produce coherent-looking data. Nothing flags. Nothing trips a quality filter.

But the signal quality is degraded. Subtle distinctions between concepts get flattened. Opinions are muted and spread across multiple dimensions rather than gathering around clear winners. Or metrics come back cleaner than real human opinion ever actually is, which should itself be a red flag.

The data isn't fraudulent. But it isn't fully honest either. It's what Justin Sutton aptly calls "System 0" thinking: responses generated with minimal cognitive engagement, filling in the form without really engaging with it.

What Other Fields Already Know

This problem isn't unique to market research. We see this across industries and it got me wondering – how are others handling it and what can we learn from them?

I had the chance to speak with Dr. Justin Miller, a Neuropsychologist from the University of Washington who specializes in psychometric testing. In his field, psychological assessment is a critical part of the diagnostic process and effort measurement isn't optional. It's standard practice, backed by a lot scientific evidence.

"We build effort measures into every assessment," he told me. "Before we interpret any results, we need to know whether the patient was genuinely trying. Because a low-effort performance looks almost identical to a performance deficit. And the conclusions you'd draw from each are completely different."

That distinction hit home. In neuropsychological testing, calling a patient cognitively impaired when they were actually just disengaged could have major consequences. Dr. Miller once saw a patient diagnosed with dementia, told they could no longer drive or live alone. But the family knew something was off. When he dug deeper, he found the patient's pet had died days before the original evaluation. They were grief-stricken, sleep-deprived, and disengaged. This wasn’t mentioned anywhere in the records and some of the effort indicators in the original data looked inconsistent. When Dr. Miller retested them, the patient was in a better place and the results told a completely different story. The patient didn’t have dementia – they were grief-stricken. In market research, recommending a product reposition or a messaging strategy based on low-effort survey data is no less consequential, it's just more puzzling when it goes wrong.

The parallel is uncomfortable and instructive.

How the Panel Side Is Responding

The good news is some parts of the research supply chain are already developing tools to address this.

I spoke with Alexandrine de Montera, Chief Product Officer and ISO Quality Officer at Full Circle Research, a panel and data quality company that is pioneering a more proactive approach to respondent quality. Rather than relying solely on technical checks and post-survey data quality screening, Full Circle is focused on understanding respondent behavior and identifying signs of disengagement before they contaminate the data through measures embedded within the pre-survey environment itself.

“Fraud detection remains critical, but it only addresses part of the data quality challenge. We're increasingly focused on identifying respondents who are technically legitimate but may not be in the right state to provide thoughtful, reliable feedback. By measuring engagement signals before a survey begins, including patterns associated with acquiescence and low cognitive effort, we can better understand whether a respondent is prepared to contribute meaningful data. The goal isn't to exclude people unnecessarily; it's to create conditions where our clients can make decisions based on responses that genuinely reflect what consumers think and feel.”

One example she shared: a series of questions specifically designed to detect acquiescence bias — the tendency to agree with statements regardless of content. Acquiescence can stem from several sources (some people are just dispositionally agreeable), but it's also a reliable signal of cognitive fatigue. When someone is rushing or disengaged, they default to agreement because it's the path of least resistance.

By surfacing these patterns before the survey itself even begins, Full Circle can flag or redirect respondents whose engagement profile suggests their data won't be trustworthy, not because they're bad people, but because they're not in a state to give you good data.

It's a smart approach. And it's the kind of upstream intervention that Justin Sutton was pointing toward when he argued that real data quality has to be addressed in research design, not just cleaned up afterward. Especially in a reality when research is being pushed to deliver more, clearer, and faster insights.

What This Means for Brand-Side Insights Teams

If you're running quantitative research (tracking studies, concept tests, segmentations, brand equity) this issue is probably already affecting your data. Not catastrophically, but at the margins. And in research, the margins are often exactly where the most important decisions live.

A few things worth asking:

Are your data quality standards catching disengagement, or just fraud? Most standard quality check protocols are designed to catch the obvious stuff: speeders, straight-liners, failed attention checks, duplicate IPs, bots. Those of course matter, but they don't catch the respondent who is giving just enough effort to pass the speeder filter and not trip the straight-lining flag.

What does your panel partner actually do before the survey starts? The pre-survey environment is an underutilized opportunity to assess respondent readiness — before they've already answered 20 questions.

Are you treating data quality as a vendor issue or a design issue? Both, ideally. But often, quality is treated as something you outsource to the panel company and your full-service provider, and revisit only when something looks obviously off. The research design itself, the length, structure, cognitive load, has a direct impact on how much engagement you can realistically expect to sustain. This is an important thing to keep at the front of your mind when designing surveys. And it’s not just about length, even short surveys can become monotonous or disengaging enough to lull a respondent into a low-effort mindset.

We're Going Deeper on This at Quirks NYC

This topic has a lot of layers and we're going to unpack them in detail at our upcoming Quirks NYC presentation:

"Clean Data, Wrong Answers: The New Data Quality Crisis in Market Research"

📅 July 29, 2026 12-12:30pm at Quirks NYC

If you're thinking about how to protect the integrity of your quantitative research in an environment where respondent engagement is increasingly variable, this is a session worth attending.

We'll be covering detection methods, design principles, and the frameworks we're developing to help clients make better decisions about data quality,  before the data gets pulled, not after.

Don’t fret though if you aren’t attending Quirks NYC, the presentation deck and recording will be available afterward the conference concludes.  

The industry has done a lot of work to protect against fraudulent data. The next focus is protecting against hollow data. Data that looks clean, passes every filter, but just doesn't fully reflect what your consumers actually think.

That gap is closeable but only if we approach it with our eyes open and ready to embrace change.  

Questions on this topic? We'd love to talk. Contact us at hello@catapultinsights.com.

JILL MILLER

CO-FOUNDER
CATAPULT INSIGHTS