Deep-Learning approaches to develop candidate lists of terms for use in text searches

Overview

Development stage

DeployedSource: c) Deployed – The use case is being actively authorized or utilized to support the functions or mission of an agency.

High-impact designation

Not high-impactSource: c) Not high-impact

Impact justification

Not reported

Start date

Not reported

Withheld from public reporting

Not reported

Mission

Topic area

Health and Medical

Operational functions

Clinical and medical Derived by keyword rules; see methodology

Problem the AI is intended to solve

The intended use is to increase the comprehensiveness of term lists to be used in text-matching approaches to identifying concepts of interest in free-text medical record notes. Text-matching approaches are an efficient method for identifying text of interest in medical notes. The success of text-matching approaches depends on the quality of term lists used to search for a concept of interest. While humans can generate word lists for text-matching approaches, the generated word lists are likely to be limited and biased by the vocabulary, language habits, population exposure, and local dialects of the human generating the list. Deep learning methods can help to augment candidate term lists to help overcome some of these challenges. Once generated these augmented candidate terms are reviewed by subject matter experts for appropriateness and accuracy of the suggested terms for the concept and use case of interest.

Expected benefits

Here we (1) query pre-trained large language models (e.g., ChatGPT or Llama) through a user interface that allows prompting (such as the OpenAI API or CodeLlama) and/or (2) query word and phrase “embedding” models (e.g., Word2Vec and Phrase2Vec) to expand candidate term lists for review by subject matter experts. Our team has found both approaches effective for expediting the generation of more comprehensive term to concept mappings for use in text-matching algorithms that increase the accuracy of clinical concept extraction. We expect this use of AI to improve initial brainstorming of terms related to a given concept, resulting in a more sensitive approach to finding information of interest in free-text clinical notes. Experience to date has found that this approach enriches term sets in ways that increase sensitivity for identification of text of interest in clinical TIU notes when used in our CLEVER natural language processing pipeline.

System outputs

The outputs of this use of AI is a candidate list of terms related to a concept of interest. This candidate list is combined with suggestions generated by human experts and then these are reviewed by a subject matter expert to generate a final list of terms for use in text-matching algorithms to extract medical record free text mentions of interest. For the pre-trained Large Language model approach, the input is a query prompting for synonyms of terms related to the concept of interest. The output is the answer provided by the LLM, including the suggested candidate terms and the original terms that were stated in the query. For the word and phrase embedding models, the input is an initial list of terms that are relevant to the concept of interest and the output are the suggested candidate terms, ranked by their statistical similarity to the original terms, based on the transformation of a clinical corpus into vector space or an “embedding model”. Regardless of the type of model used, all suggested candidate terms are reviewed by subject matter experts to ensure quality term lists are used in all text searches.

Technology

AI classification

Natural language processingSource: Natural Language Processing: AI that processes, interprets, and shares information in human language.

System name(s)

Not reported

Custom-developed code

Yes

Public source code

github.com ↗

Sourcing

How it was built

Developed in-houseSource: b) Developed in-house

Vendor (as reported)

Not reported

Vendors (standardized)

None on the standard list

A vendor is the supplier named by the agency. It does not identify the underlying model or AI technology.

Data and privacy

Involves PII

Not reported

Privacy Impact Assessment

Not reported

Authorization to Operate

Not reported

Demographic features

Not reported

Training and evaluation data

Not reported

Federal Data Catalog

Not reported

Governance

0 of 8 minimum-practice questions answered. A blank answer means the agency reported nothing; it does not mean the practice is absent.

Potential impacts and how they were identified

Not reported