Using LLMs to assist with data extraction

Overview

Development stage

In development or acquisitionSource: a) Pre-deployment – The use case is in a development or acquisition status.

High-impact designation

Not high-impactSource: c) Not high-impact

Impact justification

Not reported

Start date

Not reported

Withheld from public reporting

Not reported

Mission

Topic area

Health and Medical

Operational functions

Clinical and medical Derived by keyword rules; see methodology

Problem the AI is intended to solve

We are investigating the ability of Large Language Models (LLMs) to automate the extraction of discrete data elements from clinical notes. The National Radiation Oncology Program (NROP) office is using Quality Measures to evaluate the quality of care to veterans receiving radiotherapy inside of the VA and in the community. While a much value was found in the data collected in prior projects, the manual data extraction process was expensive, time consuming and still error prone. The purpose of this work is to determine the feasibility of using modern AI technology to reduce the burden of collecting this information from clinical notes. AI will be used to automate data extraction from free text notes which is not possible at scale because of the cost and the level of human labor required.

Expected benefits

Extracting this discrete data will help to track changes in treatment quality over time, identify care gaps and provide data which could later be used for building predictive models to improve care and become a resource for VA researchers.

System outputs

The primary output of this system will be discrete data found in free text clinical notes.

Technology

AI classification

Natural language processingSource: Natural Language Processing: AI that processes, interprets, and shares information in human language.

System name(s)

Not reported

Custom-developed code

Not reported

Public source code

Not reported

Sourcing

How it was built

Not reported

Vendor (as reported)

Not reported

Vendors (standardized)

None on the standard list

A vendor is the supplier named by the agency. It does not identify the underlying model or AI technology.

Data and privacy

Involves PII

Not reported

Privacy Impact Assessment

Not reported

Authorization to Operate

Not reported

Demographic features

Not reported

Training and evaluation data

Not reported

Federal Data Catalog

Not reported

Governance

0 of 8 minimum-practice questions answered. A blank answer means the agency reported nothing; it does not mean the practice is absent.

Potential impacts and how they were identified

Not reported