STATE · no agency ID · record R3213

AI-Augmented Declassification Review

Department of State · A/SKS

Overview

Development stage: DeployedSource: c) Deployed – The use case is being actively authorized or utilized to support the functions or mission of an agency.
High-impact designation: High-impactSource: a) High-impact
Impact justification: Not reported
Start date: 2023-02-02 (day precision)Source: 02/02/2023 00:00:00
Withheld from public reporting: NoSource: a) No

Mission

Topic area: Administrative Functions
Operational functions: No function tag matched Derived by keyword rules; see methodology

Problem the AI is intended to solve

The amount of documents, particularly cables and emails, that require declassification review increases exponentially in the next few years. Manual review is unsustainable and expensive given the number of cables (in the hundreds of thousands) and emails (increasing from hundreds of thousands to millions).

Expected benefits

The expected benefits and positive outcomes from using AI are cost savings by reducing the need of manual reviews, reduce labor in cable review by up to 80%, reduce the time needed for annual review, and create more consistency in the review process.

System outputs

The AI system's outputs are binary classification predictions for documents on whether a document should be declassified or exempt from declassification and multiclassification for reasons for exemption.

Technology

AI classification: Classical / predictive MLSource: Classical/Predictive Machine Learning: Models trained on data to make predictions or classifications based on identified patterns or relatio…
System name(s): Not reported
Custom-developed code: Yes
Public source code: Not reported

Sourcing

How it was built: Contract and in-houseSource: c) Developed with both contracting and in-house resources
Vendor (as reported): Deloitte
Vendors (standardized): Deloitte

A vendor is the supplier named by the agency. It does not identify the underlying model or AI technology.

Data and privacy

Involves PII: No
Privacy Impact Assessment: Not reported
Authorization to Operate: No
Demographic features: k) None of the above
Training and evaluation data: The data used to train the model are cables from 1995-1999 that have completed manual review with metadata on decisions from manual declassification review. Additional data includes classification/declassification guides and associated glossaries to improve model performance. Performance evaluation is measured by a human Quality Control reviewer.
Federal Data Catalog: Not reported

Governance

8 of 8 minimum-practice questions answered. A blank answer means the agency reported nothing; it does not mean the practice is absent.

Pre-deployment testingReported in placeSource: a) Yes

AI impact assessmentReported in progressSource: b) In-progress

Independent reviewReported in progressSource: d) In-progress

Ongoing monitoringReported in placeSource: a) Yes, sufficient monitoring protocols have been established

Operator trainingReported in placeSource: a) Yes, sufficient and periodic training has been established

Fail-safeReported in placeSource: a) Yes

Appeal processReported in placeSource: a) Yes, an appropriate appeal process has been established

User and public consultationReported in placeSource: a) Direct usability testing

Potential impacts and how they were identified

The model could incorrectly predict to declassify a document. The model could predict to exempt a document that should have been declassified, reducing public visibility. All exempted documents are reviewed by a human.