Your test data deserves a true synthetic data product
When building complex applications that affect millions of users, comprehensive testing is important. High-quality test data is an especially crucial component of testing, as it can help you catch and mitigate major issues. However, we've heard many times that good test data is hard to come by. Most organizations run into the same challenge when trying to obtain test data:
Millions of data examples (even terabytes of data) can exist in the production environment … and yet, for security reasons, none of it can actually be used for testing!
The main reason for this is that production data contains personal identifiable information (PII) and other sensitive values that shouldn't be shared for testing and development. In other words, high security means that your production data is stuck in your production environment. As a result, you may look to solutions from external vendors to help create test data. In recent years, synthetic data has proven to be a state-of-the-art test data solution.
Synthetic test data is created by an AI model that understands and emulates the overall patterns from your production data. Unfortunately, many products out there claim to provide synthetic data … but they are actually just simpler, data transformation and masking ETL pipelines that move data around. Pursuing these solutions does not fully solve the problem and can lead to additional problems: massive data infrastructure requirements, computational overload, and cumbersome workflows. So when researching test data solutions, it's important to understand the difference between a true, synthetic data product and a data masking ETL tool in disguise.
In this article, we will cover how data masking using simple ETL works, how AI model-based generation works, and the differences between them. We'll also present the top 5 challenges faced by our customers who were using data masking/ETL solutions before migrating to AI model-based generation.
Data masking: how does it work?
A data masking and transformation tool is something that copies your production data and adds some extra security measures to it – for example, it masks PII, adds noise to values, and applies other anonymization techniques. It is essentially an ETL pipeline that transforms your production data into a safer version of it. Such tools have existed for a long time. We think that data masking products are now marketing themselves as "synthetic data" due to synthetic data's newfound popularity. But these tools are not actually creating synthetic data. Even if they use AI to help identify and transform some of the data, the test data is still fundamentally just a variation of the real data. It's more accurate to say that they are applying a transformation.
One consequence of this is that the test data is the exact same size as the production data that you put into the tool. So if, for example, you wanted 200GB of test data for a performance test, you'd need to first extract and input 200GB of production data into the tool. The volume can lead to issues with storage or compute depending on the exact method the ETL pipeline uses. Vendors typically offer two choices.
Masking in-place: For this method, the ETL pipeline makes a temporary copy of your production data and dumps it as-is into a similarly-protected environment (for example, a special shadow environment). Then the pipeline modifies that copy by applying anonymization techniques. When it's ready, it exports newly-anonymized data into its final destination (typically a test environment). This presents some challenges:
- This process requires additional storage space for the temporary data copy.
- It also requires additional infrastructure to set up the shadow environment for temporarily hosting the data. You need to work with your security team to set up and enforce rules for this environment.
- If you'd like to refresh your test data frequently, storage costs and infrastructure complexity can add up.
Masking in-flight: For this method, the ETL pipeline extracts the production data and applies the anonymization techniques to it in-memory before exporting it into its final destination (the test environment). This solves the storage and complexity issues from in-place masking, because a temporary copy of the production data is no longer necessary. However, it can create a new performance problem instead:
- For high volumes of production data, the ETL pipeline can be extremely slow due to its in-flight setup. If it crashes, you then have to set up a batched approach, which can add more complexity.
- The problem gets worse if you'd like to refresh the test data frequently. Unless you add computational resources, it can result in a bottleneck of unprocessed data.
We've heard from many organizations that neither of these options is a great solution. No matter which one they choose, the storage, infrastructure, or computational costs are so high that they don't end up refreshing their test data very frequently. And stale test data means sub-par testing.
AI model-based generation: how does it work?
In contrast to data transformation and masking ETL pipelines, synthetic data is part of a completely new paradigm that involves creating AI models. In this approach, you'll install a synthetic data software on-prem, such as SDV Enterprise. You then sample a subset of your production data and feed it into SDV Enterprise. It learns general patterns about the data and gives you a lightweight AI model. That AI model is capable of producing unlimited amounts of synthetic data on demand. This is why we refer to the synthetic data process as AI model-based generation.
Every single synthetic datapoint that the AI model generates represents completely new examples. Unlike anonymized data, it isn't possible to trace back a synthetic datapoint to a single instance of your production data. This makes it safe to use the synthetic data for testing. (Some argue that synthetic data is actually safer than anonymized data because it can't be traced back to an individual.)
Among AI model-based generation techniques, SDV Enterprise is optimized to provide you the best experience:
- SDV Enterprise does not require high compute resources. SDV Enterprise is designed to run on a standard, CPU-enabled machine. The algorithms are optimized and efficient, meaning that you won't have to pay for GPU time or resources.
- An SDV Enterprise model is extremely lightweight. You can think of this AI model as a condensed summary of the patterns in your production data. For example, it stores statistics such as the min, max, and average values per column, as well as information about how columns are related to each other. These statistics are fixed no matter the size of your production data. So if you have a database with 30 tables, SDV Enterprise will create a model that is roughly the same size regardless of the production data being 10MB or 100GB.
- You can input a small subset of production data and upsample it. SDV Enterprise's models do not require large amounts of production data. We've seen that the patterns become well-established with as few as 2500 rows in each table. This means that you can put in a small subset of your production data to create the model, and the model is capable of creating an infinite supply of synthetic data test data for you. At ING Belgium, for example, SDV Enterprise trained on just 5,000 production payments—less than 1% of daily volume—and generated 10,000 synthetic payments in under two minutes, enabling 100x the test coverage in one-tenth the time.
Ultimately, there is no need to run a massive ETL job when you have a lightweight model. It's also much faster to regenerate a model as your production data changes. As a result, organizations that adopt the synthetic data approach (AI model-based generation) are able to refresh their test data more frequently than those who are still using data masking. The result of this is better testing, as well as fewer escalations and problems in your application.
5 data masking ETL challenges solved by AI model-based generation
- Upsampling production data. Data masking and ETL-based solutions can only produce the same amount of data as what they've been fed from production. This is because they are basically transformation tools — they just transform the data they have received. Because AI model-based generation tools generate data based on learned patterns, they can create unlimited data with a small training set.
- Keeping storage costs low. Data masking and ETL-based solutions require extra storage space for transformed/masked data. In-place masking also requires the creation and storage of a copy of the production data. With AI model-based generation techniques, only the model must be stored, because data can be generated on demand. Generated data can be deleted at any time, because it's easy to create it again.
- Keeping infrastructure simple. Data masking and ETL-based solutions require complex infrastructure, including shadow environments, powerful network connectivity (for data transfers) and batched jobs. It can require an entire devops-style infrastructure management system just to create test data. With AI model-based generation, it's only necessary to train one model. Highly optimized tools like SDV Enterprise let you do this on a single CPU machine using just a small sample of the production data.
- Keeping computational cost low. Inflight masking requires a high level of computational resources to prevent bottlenecks. An AI model can be built on a single CPU-based machine.
- Frequent data refreshes. Because data masking and ETL-based solutions are so complex, it's not possible to frequently update the test data. With AI model-based generation, it's easy to generate new data or create a lightweight AI model, so you can update your test data regularly.
Data masking vs. AI model-based synthetic data: a cheat sheet
Even if a product is being sold to you as "synthetic data", it's best to perform some due diligence to determine whether it's truly a model-based generation software or just a data masking ETL pipeline. An easy way to tell:
- Does the product always create the same amount of test data as production? In this case, it's likely a transformation tool that uses data masking with an ETL pipeline.
- Does the product produce a lightweight AI model that you can use to generate unlimited test data? In this case, you have an actual, synthetic data software. Products may use different terminology to refer to this model – for example, SDV Enterprise calls AI models synthesizers because they are capable of creating unlimited amounts of synthetic data.
Are you ready to jump into AI model-based generation for test data? SDV Enterprise is a solution for creating synthetic data for testing applications. Create an account to get started.
