GenAIMeta: Generative AI CDC Metadata Query Application

Overview

Development stage

PilotSource: b) Pilot – The use case has been deployed in a limited test or pilot capacity.

High-impact designation

Not high-impactSource: c) Not high-impact

Impact justification

Not reported

Start date

2025-01 (month precision)Source: Jan-25

Withheld from public reporting

NoSource: a) No

Mission

Topic area

Health and Medical

Operational functions

No function tag matched Derived by keyword rules; see methodology

Problem the AI is intended to solve

Metadata plays a crucial role in enhancing public understanding and usage of CDC data. Usable metadata are essential not only for making data easy to find, understand, and use on data.cdc.gov but also for synchronizing with other federal catalogs. Metadata on data.cdc.gov spans 1,056 datasets with ~20 fields each, syncing nightly with federal catalogs. Manual validation, normalization and monitoring of this volume, and the inconsistent quality and completeness of those fields creates bottlenecks for data discovery, governance and downstream analytics. The aim is to leverage EDAV’s Azure OpenAI-powered models to automate metadata validation, standardization and monitoring at scale, replacing error-prone manual checks with real-time, AI-driven oversight.

Expected benefits

Objective: Phase 1- Automate metadata validation and monitoring on data.cdc.gov using EDAV’s Azure OpenAI API Phase 2 - Make useful AI agents that are user centric and make application of the tool broader with tested agent efficacy Evaluation: Approach: Asked both general and domain-trained models the same set of real-world questions. Benchmarked responses on four metrics: Accuracy: Match to ideal answers Relevance: Alignment with user needs Clarity: Readability and actionability Completeness: Coverage of all aspects of the question Results: Domain-trained models outscored general models on every metric. Trained models delivered more precise, context-aware, and fully-formed answers. General models tended toward vague or overly broad responses. Conclusion: Targeted, domain-specific training significantly boosts an LLM’s ability to meet specialized user requirements. Key Benefits: Actionable insights for better decision-making during time-sensitive scenarios. Optimized resource allocation for improved efficiency. Enhanced trust in decision-making frameworks through consistent performance.

System outputs

Phase -1 Implementation ? EDAV’s Azure AI Infrastructure • Ingest and preprocess data from data.cdc.gov • Extract vector embeddings for model training • Build and fine-tune LLM and ML models focused on metadata usage and quality monitoring ? Monitoring Dashboard • Connects directly to Azure AI outputs • Provides real-time data-quality checks and metadata health metrics • Features interactive interfaces for key metrics and insights Phase-2 Implementation ? Domain-Specific Model Training • Built a targeted dataset of real-world questions with paired ideal answers • Fine-tuned and evaluated models against accuracy, relevance, clarity and completeness benchmarks ? Multi-User, Multi-Agent Framework • Deployed specialized agents for distinct roles • Enabled simultaneous support for diverse users including data-quality managers, data scientists, epidemiologists, etc. ensuring scalable, task-focused collaboration

Technology

AI classification

Generative AISource: Generative AI: AI that generates new or synthetic content (e.g., images, videos, audio, text, code).

System name(s)

EDAV (Enterprise Data Analytics and Visualization Platform)

Custom-developed code

No

Public source code

Not reported

Sourcing

How it was built

Contract and in-houseSource: c) Developed with both contracting and in-house resources

Vendor (as reported)

Not reported

Vendors (standardized)

None on the standard list

A vendor is the supplier named by the agency. It does not identify the underlying model or AI technology.

Data and privacy

Involves PII

No

Privacy Impact Assessment

Not reported

Authorization to Operate

Yes

Demographic features

k) None of the above

Training and evaluation data

Not reported

Federal Data Catalog

Not reported

Governance

0 of 8 minimum-practice questions answered. A blank answer means the agency reported nothing; it does not mean the practice is absent.

Potential impacts and how they were identified

Not reported