Open-source topic identification for the Public Engagement with Science Initiative data analysis.

Overview

Development stage

In development or acquisitionSource: a) Pre-deployment – The use case is in a development or acquisition status.

High-impact designation

Not reported

Impact justification

Not reported

Start date

2025 (year precision)

Withheld from public reporting

Not reported

Mission

Topic area

Not reported

Operational functions

Summarization Derived by keyword rules; see methodology

Problem the AI is intended to solve

OIA team proposes using unsupervised Natural Language Processing (NLP) to identify topical themes within responses submitted to the NSF public engagement questionnaire. The AI system will include multiple open-source techniques used together to achieve this goal. Details of the system are as follows: To ensure high-quality topics, the system will first programmatically remove responses that are blank or non-informative. The National Center for Health Statistics (NCHS) Semi-Automated Non-Response Detection for Surveys (SANDS) model will be used to filter out non-response responses (e.g., ""sdfsdfa,"" "" idkkkkk""). To reduce duplicate responses potentially caused by bots or submitted to boost relative importance, responses with the exact duplicate question wording will be removed over a threshold of three responses per IP address. If an individual has several different questions they would like to submit, all of their questions will be included. The system will use an open-source profanity filter such as Python’'s profanity-filter to identify toxic language. Any responses that are primarily toxic language will be dropped from analysis. To group questions into themes, we will use a word-embedding model to convert text responses into multidimensional space for topic modeling. Because we anticipate responses in multiple languages, the system will use a multilingual will use a multilingual word-embedding model, such as "paraphrase-multilingual-MiniLM-L12-v2" or the most up-to-date version of this model at the time of analysis. Embeddings will be reduced in dimensions using a dimension reduction algorithm such as t-SNE (t-distributed Stochastic Neighbor Embedding) to prepare for clustering. Reduced response embeddings will be clustered into groups using the ClassTfidfTransformer clustering algorithm from the Python package BertTopic. The algorithm will be seeded with words corresponding to a list of NSF programs corresponding to different research areas to map the clusters to these established categorizations. To aid in the review of topics generated by the clustering algorithm, an open-source large language model (LLM), such as Meta Llama, will be used to summarize the responses grouped within a cluster. Using a prompt like the example below, this strategy will produce a plain language summary and title of each of the clusters of responses. The clusters produced will be reviewed by Research Triangle Institute (RTI) staff for accuracy and appropriateness of naming. Staff will make edits to the LLM-generated summaries as necessary and note any clusters that are candidates for merging with other clusters into a single topic. The topics will then be validated by NSF experts and the advisory committee. Once the existing cluster model has been finalized, the cluster model will be applied to all responses. Where feasible, the RTI Science, Technology, Engineering, and Mathematics (STEM) education experts/analysts will manually map the research topics to NSF research programs.

Expected benefits

Not reported

System outputs

Not reported

Technology

AI classification

Not reported

System name(s)

Not reported

Custom-developed code

Not reported

Public source code

Not reported

Sourcing

How it was built

Not reported

Vendor (as reported)

Not reported

Vendors (standardized)

None on the standard list

A vendor is the supplier named by the agency. It does not identify the underlying model or AI technology.

Data and privacy

Involves PII

Not reported

Privacy Impact Assessment

Not reported

Authorization to Operate

Not reported

Demographic features

Not reported

Training and evaluation data

Not reported

Federal Data Catalog

Not reported

Governance

0 of 8 minimum-practice questions answered. A blank answer means the agency reported nothing; it does not mean the practice is absent.

Potential impacts and how they were identified

Not reported