Overview
The AI TrueCheck Dashboard provides comprehensive visibility into the performance and reliability of GenAI-powered document extraction across production and tuning document sets. It uses precision-driven metrics to measure extraction performance and identifies records requiring human expert validation.
When documents are uploaded through the Pramata Platform and extraction is triggered, the metrics dashboard delivers complete visibility into automated data extraction workflows, building confidence in AI-driven processes through transparent reporting and verification capabilities.
Run Extraction
The Run Extraction feature introduces intelligent automation to the document processing workflow, enabling Pramata and Partner users to trigger comprehensive data extraction on uploaded in-scope documents with a single click. Once clicked, the system automatically processes the documents through a comprehensive extraction workflow.
This powerful capability transforms how users interact with their document repositories by providing on-demand extraction with complete transparency and control.
Access Control Requirements:
- User Access: The Run Extraction button is exclusively visible to Partners and Pramata users.
- Document Limit: Run Extraction is enabled only when the total in-scope Pramata numbers (P#s) remains below 25k in the repository.
- Scope Filtering: The system automatically processes only documents that are not assigned to any milestone, are in-scope, are yet to be triggered, and are legible. Note that, document that are already triggered once will be skipped.
Configuring a Run Extraction
When Run Extraction is clicked, a configuration modal appears where users set up three options before confirming. The system remembers the Processing Workflow selection for future runs.
Step 1: Processing Workflow
Choose how documents move through the extraction pipeline
| Option | Best For | Credits |
| Cleanse and Organize | Ongoing daily runs — documents pass through both phases in one go | 4 Contract AI Credits per document |
| Only Cleanse | Large one-time loads where you want to review and remove documents before deeper extraction | 1 Contract AI Credit now; +3 if Organized later after de-scoping |
Step 2: Milestone Type
Select the milestone under which extracted documents will be grouped and reviewed:
- Production Milestone: Only low-confidence AI data is shown for human validation. Recommended for standard project runs.
- Tuning Milestone: All attributes are shown for human validation. Run at the start of a project to confirm AI extraction matches your guidelines. Identified with a T label and formatted as "Tuning Milestone <number>", with automatic assignment to the user who initiated the extraction.
Step 3: Document Selection
Choose which documents to include in the extraction run:
- All Documents: Processes all available documents. The count of eligible documents is shown (e.g., All documents (10)).
- Upload CSV: Specify exact Pramata #s to process. Limits and formatting depend on the milestone type selected.
Notes:
- If you have more documents than the limit for either milestone type, you can run extraction multiple times.
- For CSV uploads, list Pramata #s in Column A with each row containing a separate Pramata #.
Once all three sections are configured, click Confirm to initiate extraction.
OCR Pending Notification
When clicking Run Extraction, if any documents in the selected scope have not yet completed OCR processing, an OCR in Progress warning banner appears at the bottom of the Confirm Extraction dialog.
- The banner displays the count of pending documents as a clickable link, which downloads a file listing the affected Pramata numbers.
- This applies to both All Documents and Upload CSV selection modes. When using CSV upload, an Awaiting CSV Upload message is shown first; once the CSV is uploaded, the system checks OCR readiness for the selected documents.
- The user can choose to
- Cancel – Close the dialog and wait for OCR to complete.
- Exclude & Continue – Proceed with extraction, automatically excluding the pending documents. These documents remain available for the next extraction run once OCR is complete.
Extraction Process
The automated extraction workflow combines intelligent document processing with comprehensive tracking and assignment capabilities. Once initiated, the system executes a series of coordinated steps to ensure accurate data extraction and proper user assignment for review and validation.
Workflow Steps
- Automated Processing: The system identifies and processes documents that meet specific criteria - those not assigned to any milestone, marked as in-scope, and confirmed as legible documents from the repository.
- Milestone Creation: Upon clicking the Run Extraction button, users have the option to select or deselect the tuning milestone before proceeding:
- Tuning Milestone: If selected, a new tuning milestone is automatically created in Digitizer. You can identify this with a T label and formatted as "Tuning Milestone <number>", with automatic assignment to the user who initiated the extraction. Results will reflect in the tuning metrics table.
- Normal Milestone: If the tuning option is deselected, the system will set up a normal milestone, and the extraction results will reflect in the first metrics table.
- Auto Extraction: All Metadata and Document Family attributes are automatically triggered, along with Foundation Term & Renewal Model Tagging, which is auto-triggered at the same time.
- Database Integration: All extraction results are seamlessly saved to the database
- User Assignment: Extracted documents are systematically assigned to the initiating user for manual review and validation.
- Performance Monitoring: Users get access to detailed Quality Metrics that provide comprehensive insights into process performance and success rates.
Key Features
- Extraction Status: A dynamic status bar displays real-time completion progress (e.g., "26/236 in-scope documents extracted") with visual progress indicator for immediate status assessment.
- Date Range Filtering: Filter extraction metrics by Document Upload Date using start and end date selectors to view performance trends for specific time periods.
- Last Refreshed: Displays the Last Updated Date & Time stamp, along with a refresh icon that allows users to manually update the display to view the latest extraction data.
Attribute Level Confidence
- Validation-Based Categorization: The system automatically categorizes extracted data into two distinct validation levels, giving users immediate insight into the quality and reliability of the extraction results:
- Positive Accuracy & Negative Accuracy columns, expand further to view detailed validation results.
- Accurate: Data extracted and validated with strong reliability, confirmed for use
- Pending Validation: Data requiring manual review and validation
- Interactive Metrics Dashboard: Users can view aggregate counts for each confidence category at a glance, providing a quick overview of extraction performance across the entire document set.
- Detailed Drill-Down Capability: By clicking on any confidence category, users can access granular, page-by-page (p#) data that shows exactly what information was extracted from the document.
- Citation and Audit Trail Integration: The detailed view includes comprehensive citation information and complete Audit Trails, enabling users to trace back to the original source material and verify the accuracy of extracted data.
Dashboard Metrics
The AI TrueCheck Dashboard uses precision-driven metrics to evaluate the accuracy of document attribute extraction. The following metrics provide key insights into extraction performance:
Note: AI Interpretation of False Positive or False Negative will go into True Positive or True Negative.
| Metrics | Definition | Formula |
| Positive Accuracy | Measures how accurately the value is extracted when it actually exists in the document | 2 × (Precision × Recall) / (Precision + Recall) |
| Negative Accuracy | Measures how often the system correctly avoids extracting a value when it does not exist in the document. Note: Negative Accuracy is relevant for optional attributes and not for mandatory attributes. | True Negatives / (True Negatives + False Positives) |
| Precision | How often is the extracted attribute value actually correct | True Positives / (True Positives + False Positives) |
| Recall | How often the value is extracted when it actually exists. | True Positives / (True Positives + False Negatives) |
| Accurate | Number of correctly extracted values, both when it's there and when it's not there. | Formula: True Positives + True Negatives |
| Pending Validation | Number of extracted values with questionable accuracy plus when extraction failed altogether. These are pending human expert validation. | Formula: False Positives + False Negatives + Failed Extractions |
| Human Validated | Number of extracted values manually reviewed by a human expert. After validation:
| Formula: Human True Positives + Human True Negatives |
Data Sample: Understanding Positive and Negative Accuracy
To illustrate how these metrics work in practice, consider the following example with two attributes: Doc Title (expected on every document) and Doc ID (optional attribute, not present on every document).
Doc Title Attribute (Required Field - Every Document Must Have One)
| Pramata Number | Extracted Value | Actual Value | AI Classification |
| 12344 | Service Agreement 2024 | Service Agreement 2024 | True Positive ✓ |
| 12333 | License Agreement | License Agreement | True Positive ✓ |
| 12565 | (blank) | Master Agreement | False Negative ✗ |
| 12554 | Service Contract (Draft) | Service Contract | False Positive ✗ |
Doc Title Results (from 20 documents):
True Positives: 16 | False Positives: 1 | False Negatives: 2 | True Negatives: 1
Positive Accuracy Calculation (Doc Title):
Precision = 16 / 17 = 94.1% | Recall = 16 / 18 = 88.9%
Positive Accuracy = 2 × (0.941 × 0.889) / (0.941 + 0.889) = 91.3%
Doc ID Attribute (Optional Field - Not All Documents Have One)
When an attribute is optional, the system must correctly determine when it is present AND when it is absent. This creates balanced positive and negative accuracy metrics:
| Pramata Number | Extracted Doc ID | Actual Doc ID | AI Classification | Present? |
| 45123 | DOC-2024-001 | DOC-2024-001 | True Positive ✓ | Yes |
| 44123 | (blank) | None | True Negative ✓ | No |
| 46344 | ID-5678 (incomplete) | DOC-ID-5678 | False Positive ✗ | Yes |
| 46122 | (blank) | DOC-2024-006 | False Negative ✗ | Yes |
| 44552 | CONTRACT-12 (wrong) | DOC-2024-012 | False Positive ✗ | Yes |
Doc ID Results (from 20 documents):
True Positives: 12 | False Positives: 3 | False Negatives: 2 | True Negatives: 3
Positive Accuracy Calculation (Doc ID):
Precision = 12 / 15 = 80% | Recall = 12 / 14 = 85.7%
Positive Accuracy = 2 × (0.800 × 0.857) / (0.800 + 0.857) = 82.8%
Account and Document Family Visibility
The Account and Document Family pages now feature enhanced AI data integration with comprehensive validation indicators, providing users with immediate visibility into data quality and processing status.
Account Page Enhancements
Auto-Published AI Account Names: AI-generated account names are automatically published on Account pages, streamlining account identification while maintaining data quality through integrated validation indicators.
Document Family Page Improvements
- Comprehensive Metadata Display: Document Family pages showcase all AI-extracted metadata with full validation status indicators, providing users with complete visibility into document processing results and data reliability.
- Document Summary Access: Users can click on any document to access detailed summary information, with extracted data clearly categorized by validation status through the integrated flag system.
Visual Data Quality Indicators:
- Green highlighting: Indicates data validated with high confidence, representing information that has been verified and approved for use.
- Yellow highlighting: Marks data pending validation that requires manual review, ensuring users can quickly identify information needing attention. Additionally, all data points of documents belonging to a Tuning Milestone will be displayed as yellow until human validation is completed.
Note: The indicators are only on metadata terms right now and will eventually be extended to Contract Term and Renewal.