Assessing Large Language Models for Synthetic Survey Data Generation

Overview

Development stage

In development or acquisitionSource: a) Pre-deployment – The use case is in a development or acquisition status.

High-impact designation

Not high-impactSource: c) Not high-impact

Impact justification

Not reported

Start date

Not reported

Withheld from public reporting

NoSource: a) No

Mission

Topic area

Science

Operational functions

No function tag matched Derived by keyword rules; see methodology

Problem the AI is intended to solve

Survey data de-identification is crucial for the NCHS to maximize data utility while protecting privacy, but determining and applying modern best practices requires further research. NCHS conducts national surveys and releases microdata (data containing information about individuals) for public use. To protect survey participants’ confidentiality, statistical disclosure limitation techniques have been used to de-identify data, but these methods have drawbacks of losing statistical properties of the original data and thus limiting useful analyses. Additionally, these methods are not designed for very large data or text data. Use of synthetic data may offer another option. The goal of synthetic data is to preserve essential statistical features and variable relationships of the original data such that statistical inference based on the synthetic data is close to that of the original data. Large language models (LLMs) may be able to address limitations of statistical methods for synthetic data creation, especially for natural language data. We aim to advance knowledge of this application of LLMs to enable staff to select the optimal tools for synthetic data generation.

Expected benefits

Current statistical methods for synthetic data generation have drawbacks such as difficulty handling very large datasets, steep learning curve for people with less statistics or coding background, and inability to generate natural language data. Thus, if LLMs are evaluated to be successful at synthetic survey data generation, this alternative method would enable more data synthesis at scale, more data synthesis that can be conducted by staff with various levels of statistics backgrounds, and the first ever release of synthetic survey text data.

System outputs

Continuous, categorical, and free text data that matches properties of original survey data.

Technology

AI classification

Generative AISource: Generative AI: AI that generates new or synthetic content (e.g., images, videos, audio, text, code).

System name(s)

Not reported

Custom-developed code

Not reported

Public source code

Not reported

Sourcing

How it was built

Not reported

Vendor (as reported)

Not reported

Vendors (standardized)

None on the standard list

A vendor is the supplier named by the agency. It does not identify the underlying model or AI technology.

Data and privacy

Involves PII

Not reported

Privacy Impact Assessment

Not reported

Authorization to Operate

Not reported

Demographic features

Not reported

Training and evaluation data

Not reported

Federal Data Catalog

Not reported

Governance

0 of 8 minimum-practice questions answered. A blank answer means the agency reported nothing; it does not mean the practice is absent.

Potential impacts and how they were identified

Not reported