Use of Large Language Models to improve classification of identified text snippets from the CLEVER natural language processing pipeline

Overview

Development stage

DeployedSource: c) Deployed – The use case is being actively authorized or utilized to support the functions or mission of an agency.

High-impact designation

Not high-impactSource: c) Not high-impact

Impact justification

Not reported

Start date

Not reported

Withheld from public reporting

Not reported

Mission

Topic area

Health and Medical

Operational functions

Clinical and medical Derived by keyword rules; see methodology

Problem the AI is intended to solve

Use of text matching approaches like CLEVER to identify clinical information of interest in free-text clinical notes is an efficient approach but can be over-inclusive (e.g. including terms when used with an alternative meaning, for example, a search for “arms” may pick up mentions of forearms and firearms). A variety of approaches can be used to further filter identified text snippets to those more likely to be of interest for a given use case. Here we use large language model queries to classify text snippets as being relevant to a concept of interest.

Expected benefits

Improved classification of text snippets from clinical notes will increase clinician and management efficiency in use of decision support to summarize and review information extracted from free-text clinical notes. For example, improved filtering of text snippets for clinical risk factors of interest will reduce clinician time in obtaining clinically needed information about risk factors of interest by filtering out text snippets that are unlikely to be meaningful to their immediate clinical decision. In short, the clinician will have fewer candidate text snippets to read through to get the information they need. Likewise, summary views of identified possible concept mentions will be more accurate, reducing noise in views to support strategic decisions.

System outputs

This AI use case adds additional LLM-derived classification labels to text snippets from free-text clinical notes identified by the CLEVER NLP pipeline. These labels indicate the LLMs classification of the text snippet as being related to a concept of interest. For example, in our first deployed use of this method, an LLM model was used to improve classification of text snippets identified as including the term “xylazine” in the note text. The LLM classified identified text snippets into one of three labels: (1) "Other" (OTH) for snippets without evidence of suspected xylazine exposure, (2) "Suspected-Positive" (SUS-P) for snippets with positively asserted evidence of suspected exposure, and (3) "Suspected-Negative" (SUS-N) for snippets where the suspected exposure was negated. The OTH category included, for example, the many cases in which providers educated patients about the presence of xylazine in the illicit drug supply. The LLM-derived classification labels were used to narrow down text snippets for display as mentions of “possible xylaxine exposure” on the STORM decision support display (to help clinicians find information of relevance to opioid risk management across VA medical records) and in maps of possible xylazine exposures for strategic planning. In both cases, only those text snippets labeled as “SUS-P” were included on the displays.

Technology

AI classification

Natural language processingSource: Natural Language Processing: AI that processes, interprets, and shares information in human language.

System name(s)

Not reported

Custom-developed code

Yes

Public source code

Not reported

Sourcing

How it was built

Developed in-houseSource: b) Developed in-house

Vendor (as reported)

Not reported

Vendors (standardized)

None on the standard list

A vendor is the supplier named by the agency. It does not identify the underlying model or AI technology.

Data and privacy

Involves PII

Not reported

Privacy Impact Assessment

Not reported

Authorization to Operate

Not reported

Demographic features

Not reported

Training and evaluation data

Not reported

Federal Data Catalog

Not reported

Governance

0 of 8 minimum-practice questions answered. A blank answer means the agency reported nothing; it does not mean the practice is absent.

Potential impacts and how they were identified

Not reported