The market research industry is ripe with opportunities in AI innovation. In this article, we provide an overview of what it means to create synthetic survey responses for quantitative research using the latest and greatest of AI technology. We'll go over the two main methods of creating synthetic responses and identify where each method can be helpful for market research. We'll also go through some pitfalls to watch out for when adopting these methods.
Two ways to create synthetic survey responses: using an LLM or training your own AI model
The term "synthetic survey response" refers to a survey response that is produced by an AI system as opposed to an actual human respondent. This is an umbrella term that encapsulates different types of AI, and each can fundamentally change how you think about and use synthetic responses. We classify synthetic survey responses into two general buckets: Responses created by an LLM and responses created by training your own AI model. It's worth exploring each of these methods.
On one hand, you can use an LLM to generate survey responses. At a basic level, all the LLM needs is the survey template itself. It can use this to infer what survey responses could look like for different types of individuals. More advanced setups may include providing the LLM with more context that is specific to your particular industry in the hopes of nudging the LLM closer into the direction of your consumers. This might include providing general research insights or prior results. At the end of the day, you're relying on the LLM engine to generate synthetic survey responses.
On the other hand, there is a completely different type of setup where you train a smaller lightweight AI model yourself. This AI model is not an LLM. It's a smaller lightweight algorithm – for example, a statistical ML model or smaller neural network – that learns explicitly from responses you've collected so far in order to understand the patterns unique to your target consumers. Then, it's able to generate a variety of similar responses. In the end, the synthetic survey responses are based on the output of the lightweight AI model.
This isn't to say that LLMs and trained AI models are mutually exclusive options. A hybrid approach may allow you to derive benefits from both types of AI. However as we will discuss in the later sections, the two methods represent different philosophies of synthetic survey responses. Any hybridized setup will lean either towards using the LLM engine to generate responses or else an AI model's statistical engine. There are broader implications to what you are able to accomplish with each approach.
Using LLMs to generate synthetic survey responses
Using LLMs to create synthetic survey responses is enticing because they allow you to start receiving results even before you actually send your survey out. There is no major setup involved in using a popular LLM like OpenAI's ChatGPT or Anthropic's Claude. You can directly ask it to create synthetic responses based on your survey questions. You can also use other emerging products that interact with these LLMs on your behalf. This approach is known in some circles as zero-shot learning because it requires zero actual survey responses to get started, meaning that there is exactly zero expenditure for you to actually collect survey responses – just hand over your newly created survey questions to the LLM.
The key assumption is that LLMs should already contain a broad swath of knowledge about your industry as well as the psychological profiles of your consumers, and that the LLM is capable of producing responses based on that understanding. At this time, it's still unclear whether this assumption holds true and even if it does, whether it's enough to answer the questions that are important to market research.
Top 3 pitfalls of LLM-based synthetic data generation
It's certainly true that LLMs have a broad knowledge base that likely covers your general industry and some portions of your target consumer demographics. But LLMs don't have full coverage and are likely to encounter some common pitfalls.
LLMs have an implicit bias towards people who publish online. LLMs are trained on information that is available on the web. There is an implicit bias towards the types of consumer profiles who are likely to publish information this way. For example, people who have strong, negative opinions about the products are more likely to review them; the media often portrays women as being younger than their male counterparts; and wealthy, English speakers are generally overrepresented on the internet. These profiles may not correspond to your exact consumer target.
Some setups try to work around this blind spot by providing an LLM with additional context that is specific to your product or target consumer group. For example, providing more demographic information or even prior results. This can nudge the LLM in the right direction, though it may not be enough to fully counteract the bias and has been known to cause issues of its own when it attempts to create a new bias.
LLMs are trained on past data, not for predicting the future. Historically published information may not exactly correspond to the attitudes and profiles of consumers in the future. Yet, these are exactly the types of insights that you are looking for when performing market research. For example, you could be researching an emerging demographic of users like the Gen Alpha age group or attitudes about products that are on the cutting edge of innovation. Nobody knows if consumers are going to like them; these consumers haven't spoken publicly about these topics. An LLM is not a panacea to this problem. If it's not something that is already talked about in the world, then an LLM is not likely to have the insights.
LLMs struggle to create a variety of data. At their core, LLMs – particularly GPT models – are language prediction machines that are built to predict the most likely word that follows the previous text. For a quantitative survey, this means predicting the most likely numerical response based on the previous questions for that respondent. But they are not fundamentally meant to understand statistics. Intuitively, we know that individuals with the exact same demographic background (e.g. white females in the US between the ages of 30 and 35) will not necessarily respond to questions in the same way. And yet, it is difficult to get an LLM to reproduce the natural variation that exists within a population. Achieving this with an LLM right now would require additional workarounds such as careful prompting of the LLM or further manipulation of the responses that an LLM produces.
Are LLMs useful for creating synthetic survey responses?:
our verdict
The bottom line is that LLM responses summarize the state of the world today based on the information the LLM can access. Our impression is that using an LLM as a primary engine for generating synthetic survey responses won't provide much utility, but it could be useful for gaining general insights while crafting your survey. This might be something your organization is already doing: Ask an LLM about general insights such as "What is the net promoter score for brands X, Y, and Z in the US amongst women in their 30s?" Ideally the result would be an aggregated summary based on the resources the LLM can access. Then again, an LLM's response is not a defensible research methodology, so you would want to run your survey anyway.
Training your own AI model to generate synthetic survey responses
Responses from a trained AI model follow a different paradigm. This setup requires you to create the survey, send it out, and collect some responses to begin with. Then you can train a lightweight AI model on the responses you've received so far and use it to generate additional synthetic responses. This is especially useful when those responses contain quantitative information such as Likert scales or net promoter scores.
The main question with an AI model setup is why you'd need additional synthetic responses at all if your survey is already out. But there are many situations where collecting enough survey responses, or at least enough responses from a particular demographic, can be resource intensive. For example, if you are running physical studies or focus groups, recruiting additional respondents can be cost prohibitive. If it's a digital survey, then you may be subject to low response rates (or high dropout rates) from your target demographic. A dearth of responses quickly becomes an analytics problem. Common statistical analysis methods such as linear regression just won't work. So the idea is to supplement the actual responses with synthetic responses for the purposes of using more robust analytical tools.
The key assumption here is that your synthetic data software should be able to learn whatever patterns exist in your collected responses, and then emulate or extrapolate them when generating synthetic responses.
Challenges of training your own AI model
The approach of training your own AI model has its own set of challenges, but most of these challenges can be overcome with the right statistical frameworks.
Synthetic data software should learn and extrapolate from a limited sense of responses. The idea of training an AI model can seem a bit tautological. How can the AI model extract more information out of an already limited amount of responses – or handle idiosyncrasies such as the data sparsity that comes out of incomplete responses? Answering this requires a look at other techniques that you might consider using for an analytics problem. These methods include oversampling, which essentially means duplicating existing responses and altering them slightly, or interpolating between specific responses to create additional possible combinations (a method known as SMOTE). Compared to this, the AI model can be a more robust engine because it can interpolate between the responses in new ways and even extrapolate outside of the known responses based on weaker patterns that are already present. It's designed to create a variety of responses while sticking to the overall locality of the known statistical patterns. Essentially, AI models are a smart way to fill in the gaps between the limited responses that you have.
AI models need modifications to follow deterministic logic. Out-of-the-box, publicly available models involve classical statistical methods or neural network-based approaches. These can work well if there are strongly established patterns and correlations within your responses. However, these models are not likely to capture the survey control flow logic that includes skip patterns, option forwarding, and advanced question types. These require deterministic rules as opposed to probabilistic trends that are more native to the AI models. (Admittedly, breaking a rule of the survey – such as answering a question that is supposed to be skipped based on a previous response – may not entirely matter for downstream analytical purposes, but it erodes trust in the AI model.)
These concerns can be addressed with an additional survey control flow layer that's built alongside the foundational AI modeling software. The control flow layer is responsible for intercepting the probabilistic decisions of the AI model and injecting the deterministic control flow logic based on the survey template.
Should you use AI models for synthetic survey responses?: our verdict
We believe that synthetic data software can be specifically designed to address the challenges posed by survey data. When done successfully, we believe AI models are the best option for creating synthetic survey responses to aid in your market research analysis.
It's worth noting that properly addressing the challenges requires intrinsic knowledge of how the AI models are meant to work, so we expect the most promising solutions to be built by the owners of the original AI software – for example, synthetic data software vendors. (As the authors of one of the most popular synthetic data software, we already have a solution to make our AI models completely accustomed to the idiosyncrasies of survey data and able to follow survey logic at the same time.)
Future evolutions of synthetic survey responses
AI models and LLMs are not mutually exclusive solutions. The future points to hybrid approaches that are able to capture the benefits of both solutions. When designing a hybrid solution, we believe that the AI model should be the underlying response engine that LLMs can eventually integrate into. That is to say, a hybrid approach is more likely to work when the AI model is ultimately responsible for generating the responses within input from an LLM.
For example, an LLM may be able to use its (biased and somewhat incomplete) knowledge to seed the initial parameters of an AI model. Then as you collect more responses from actual, human respondents, the AI model will be able to adjust.
How to adopt synthetic data for market research
Ultimately, we see both the promise and the business case for synthetic survey responses being adopted into market research workflows, with a special emphasis on using trained AI models as the generation engines. Here at DataCebo, we are piloting an early access program for trained survey response synthesis, which involves adding a control flow layer to our home-grown AI modeling software. This would allow you to create a variety of synthetic responses that adhere to your original survey flow structure and logic.
To anyone interested in using synthetic survey responses, we recommend a systematic evaluation of AI models based on previously collected survey responses. With a proper experimental setup, it's possible to simulate the problem of not having enough survey responses and measure whether the trained responses match up with actual responses that were collected later. A good AI model is able to generate quality responses that are robust and trustworthy – enough so that you can feel comfortable cutting down the responses you need to collect for future surveys.
DataCebo is the company behind the Synthetic Data Vault (SDV), a platform for generating high-quality synthetic data. Are you interested in exploring synthetic survey responses? The DataCebo team is piloting an early access program for synthetic survey responses. To get started, Contact Us.
