Skip to main content
Effective document management is crucial for building high-quality knowledge bases. This guide covers how to upload, process, organize, and maintain documents in AI Knowledge to ensure optimal retrieval performance.

​
Supported Document Types

AI Knowledge supports a wide range of document formats:
CategoryFormatsNotes
Text DocumentsPDF, DOCX, DOC, RTF, TXTFull text extraction with formatting preservation where possible
PresentationsPPTX, PPT, KEYExtracts text, slide structure, and notes
SpreadsheetsXLSX, XLS, CSV, TSVProcesses tabular data with cell relationships
Web ContentHTML, MHT, XMLPreserves content structure and extracts relevant text
ImagesPNG, JPG, TIFF, GIFOCR for text extraction from images
MarkdownMD, MARKDOWNPreserves structure and formatting
CodeVarious source code filesMaintains code structure and comments

​
Document Upload Methods

  • Direct Upload
  • Bulk Import
  • Connector Import
  • API Upload
Upload files directly through the web interface:
  • Select individual files or entire folders
  • Drag and drop multiple files
  • Monitor upload progress
  • Receive immediate processing feedback
Best for:
  • Small to medium document collections
  • Initial knowledge base setup
  • Ad-hoc document additions
  • Documents stored locally

​
Document Processing Pipeline

1

Upload & Initial Validation

Documents are transferred to the system and validated.This stage includes:
  • Format verification
  • Size and content checking
  • Initial metadata extraction
2

Text Extraction

Content is extracted from various document formats.Techniques include:
  • PDF text layer extraction
  • OCR for images and scanned documents
  • Document structure parsing
  • Table and chart content extraction
  • Formatting preservation
3

Document Enrichment

Additional information and structure are added.Enrichment includes:
  • Metadata enhancement
  • Language detection
  • Entity identification
  • Topic classification
  • Summarization
  • Structure annotation
  • Content typing
4

Chunking

Documents are divided into retrievable segments.Chunking strategies include:
  • Semantic chunking (based on meaning)
  • Fixed-size chunking (token count)
  • Structure-based chunking (sections)
  • Paragraph-level chunking
  • Sliding window approaches
  • Hierarchical chunking
5

Embedding Generation

Vector representations are created for chunks.This process includes:
  • Embedding model application
  • Vector generation for each chunk
  • Multi-vector approaches (where applicable)
  • Embedding verification
  • Quality assessment
  • Optimization for retrieval
6

Indexing

Chunks and embeddings are organized for efficient retrieval.Indexing includes:
  • Vector database storage
  • Metadata indexing
  • Full-text search indexing
  • Relationship mapping
  • Access control implementation
  • Query optimization structures
7

Quality Verification

Processing results are checked for quality and completeness.Verification includes:
  • Content extraction validation
  • Chunking quality assessment
  • Embedding consistency checks
  • Missing content detection
  • Error logging and reporting
  • Sample query testing

​
Document Management Interface

The document management interface in AI Knowledge provides comprehensive tools for organizing and maintaining your document collection:
  • Document Library
  • Upload & Import
  • Document Details
  • Batch Operations
The main document view provides:
  • Comprehensive document listing
  • Sorting and filtering options
  • Status indicators
  • Batch operations
  • Search functionality
  • Version history access
Key features:
  • Preview documents directly in the interface
  • Check processing status and health
  • View document metadata
  • Manage document tags and categories
  • Track document usage statistics

​
Document Organization

Effective document organization improves retrieval quality and knowledge base maintenance:
Apply flexible tags to documents to filter on these during queries.
  • On each document, you can specify its tags
  • You can use these tags:
    • Automatically, in AI > Self Query > Enabled, so the AI dynamically choose which tags to use for a query
    • Set by the user, In AI > Self Query > Enabled by the user. This will then enable adding tags using the new ”+” button that will appear in the AI Store.
    • Set mandatory tags for user or groups, in the User sharing page.
The tags set in the “mandatory” section per user/group operate as an AND with any other tag. If multiple tags are selected from the user input, they operate as an OR together.The tags selected by the user in the AI Store can be used by any other custom tool, with the value stored in the metadata.aikTags field

​
Document Processing Settings

Customize how documents are processed to optimize for your specific knowledge base needs:
  • Extraction Settings
  • Chunking Configuration
  • Embedding Options
  • Index Configuration
Configure how content is extracted from documents:
  • OCR Settings:
    • OCR engine selection
    • Language optimization
    • Image preprocessing
    • Confidence thresholds
  • Structure Handling:
    • Table extraction methods
    • Header/footer treatment
    • Layout preservation
    • Image handling
  • Content Filtering:
    • Element inclusion/exclusion
    • Content type prioritization
    • Noise reduction
    • Redundancy handling

​
Document Maintenance

Keep your knowledge base current and optimized with these document maintenance practices:
1

Regular Content Updates

Keep information current and accurate.Maintenance activities:
  • Schedule regular document reviews
  • Update outdated information
  • Add new versions of documents
  • Remove obsolete content
  • Track document freshness
2

Version Management

Track document changes over time.Key capabilities:
  • Maintain version history
  • Compare document versions
  • Restore previous versions
  • Track change audit trail
  • Manage version relevance
3

Content Health Monitoring

Proactively identify and address issues.Monitoring areas:
  • Processing error detection
  • Broken document identification
  • Chunking quality analysis
  • Embedding anomalies
  • Retrieval performance issues
4

Reprocessing & Optimization

Refresh processing to improve quality.Optimization activities:
  • Reprocess with improved settings
  • Apply new chunking strategies
  • Update to better embedding models
  • Enhance metadata and structure
  • Optimize based on performance analytics

​
Automated Document Processing

Set up automated workflows for efficient document management:
Automatically import documents on a regular basis:
  • Configure recurring import jobs
  • Set source locations and credentials
  • Define processing parameters
  • Schedule optimal import times
  • Configure notification preferences
Use cases:
  • Regular knowledge base updates
  • Synchronization with document repositories
  • Periodic report processing
  • Automated content refreshes
Monitor specific locations for new documents:
  • Set up folder monitoring for local or network locations
  • Configure cloud storage monitoring
  • Define instant processing triggers
  • Set up filtering rules
  • Configure error handling
Benefits:
  • Real-time knowledge updates
  • Reduced manual intervention
  • Streamlined document workflows
  • Consistent processing application
Create customized document workflows:
  • Define multi-stage processing
  • Set up conditional processing paths
  • Configure enrichment steps
  • Implement validation checkpoints
  • Create custom post-processing
Advanced capabilities:
  • Document classification and routing
  • Conditional metadata application
  • Multi-format conversions
  • Specialized content extraction
  • Custom data integration
Connect document processing to external systems:
  • Configure webhook notifications for events
  • Set up bidirectional system integrations
  • Implement custom API workflows
  • Create event-driven processing
  • Enable cross-system synchronization
Integration types:
  • Content management systems
  • Document repositories
  • Workflow systems
  • Enterprise applications
  • Custom business systems

​
Best Practices for Document Management

Consistent Organization

Establish and maintain a logical, consistent document organization scheme

Quality Over Quantity

Focus on high-quality, authoritative documents rather than sheer volume

Rich Metadata

Add comprehensive metadata to enhance context and retrieval

Optimal Chunking

Tune chunking strategies to preserve context and meaning

Regular Maintenance

Schedule routine updates, reviews, and optimizations

Automated Workflows

Implement automation for consistent, efficient processing

Versioning Strategy

Maintain clear version control for evolving documents

Performance Monitoring

Track and optimize document retrieval effectiveness

​
Troubleshooting Document Issues

If documents fail to upload:
  • Check file format compatibility
  • Verify file isn’t corrupted or password-protected
  • Ensure file size is within system limits
  • Check network connectivity and stability
  • Verify upload permissions
  • Examine client-side browser issues
Resolution steps:
  • Convert to a standard format
  • Use smaller batch sizes
  • Try alternative upload methods
  • Check system logs for detailed errors
When documents upload but fail during processing:
  • Review document structure and complexity
  • Check for unsupported elements or formatting
  • Verify text extraction capability for the format
  • Examine system resource availability
  • Check for timeout issues with large documents
  • Review processing logs for specific error messages
Resolution steps:
  • Simplify complex documents
  • Pre-process problematic files
  • Adjust extraction settings
  • Split very large documents
  • Use alternative processing approaches
If extracted content has quality problems:
  • Check original document formatting and structure
  • Review OCR settings for scanned documents
  • Examine table and image extraction results
  • Verify language support for the content
  • Check for unusual characters or formatting
  • Review chunking results for context preservation
Resolution steps:
  • Improve original document quality
  • Adjust OCR and extraction settings
  • Modify chunking parameters
  • Add manual metadata to compensate
  • Consider document preprocessing
When document retrieval isn’t meeting expectations:
  • Review document relevance to query needs
  • Check chunking strategy appropriateness
  • Examine embedding model suitability
  • Verify index configuration
  • Assess query processing effectiveness
  • Evaluate content quality and coverage
Resolution steps:
  • Adjust chunking strategy
  • Try different embedding models
  • Enhance metadata for better context
  • Implement hybrid search approaches
  • Add missing content
  • Fine-tune retrieval parameters

​
Security and Compliance

Ensure your document management practices meet security and compliance requirements:
Control who can access and manage documents:
  • Document-level permissions
  • Role-based access control
  • Group-based permissions
  • Temporary access grants
  • Inherited vs. explicit permissions
Implementation options:
  • Apply permissions during upload
  • Inherit from knowledge base settings
  • Set up custom access rules
  • Implement approval workflows
  • Configure visibility restrictions
Protect sensitive information in documents:
  • PII detection and handling
  • Automated redaction capabilities
  • Data classification implementation
  • Privacy policy enforcement
  • Consent management
Privacy features:
  • Sensitive information detection
  • Configurable redaction rules
  • Audit trails for privacy actions
  • Policy-based information handling
  • Restricted processing options
Meet regulatory and organizational requirements:
  • Retention policy implementation
  • Legal hold capabilities
  • Compliance tagging and tracking
  • Regulatory metadata
  • Audit log maintenance
Compliance tools:
  • Document lifecycle management
  • Approval and certification workflows
  • Chain of custody tracking
  • Evidence preservation
  • Compliance reporting
Protect document content and processing:
  • Encryption for documents at rest
  • Secure processing environments
  • Malware scanning and prevention
  • Data loss prevention integration
  • Secure deletion capabilities
Security implementation:
  • End-to-end encryption
  • Secure temporary storage
  • Isolated processing environments
  • Authentication requirements
  • Security event monitoring

​
Document Analytics

Gain insights into your document collection and usage:
  • Content Analytics
  • Usage Analytics
  • Performance Analytics
  • Health Monitoring
Understand your document content:
  • Document type distribution
  • Content age analysis
  • Topic clustering and trends
  • Language and terminology patterns
  • Content complexity metrics
  • Duplication identification
Use insights to:
  • Identify knowledge gaps
  • Prioritize content updates
  • Optimize document organization
  • Plan maintenance activities

​
Advanced Document Processing Features

Document Transformation

Convert documents between formats and structures for optimal processing.

Options include format conversion, structure normalization, template application, and content standardization.

Content Enrichment

Enhance documents with additional information and context.

Features include entity extraction, topic classification, sentiment analysis, and relationship mapping.

Multi-Language Support

Process and retrieve from documents in multiple languages.

Capabilities include language detection, multi-lingual embeddings, translation integration, and language-specific processing.

Document Summarization

Automatically generate summaries of document content.

Options include executive summaries, section summaries, key point extraction, and customizable summary lengths.

Content Deduplication

Identify and manage duplicate or similar content.

Features include similarity detection, content comparison, redundancy management, and optimized storage.

Intelligent Redaction

Automatically identify and protect sensitive information.

Capabilities include PII detection, configurable redaction rules, entity-based protection, and compliance support.

​
Integration with External Systems

Connect your document management with other enterprise systems:
Integrate with existing document repositories:
  • SharePoint and OneDrive connections
  • Google Workspace integration
  • Box and Dropbox connectors
  • Enterprise DMS connectors
  • ECM system integration
Integration capabilities:
  • Bidirectional synchronization
  • Metadata mapping
  • Permission alignment
  • Version synchronization
  • Change detection and updates
Connect with tools where documents are created:
  • Microsoft Office integration
  • Google Docs/Sheets connectors
  • Adobe Creative Cloud connection
  • CMS system integration
  • Email platform connectors
Integration features:
  • Direct publishing to knowledge bases
  • Creation-time metadata capture
  • Version control alignment
  • Workflow integration
  • Collaborative authoring support
Connect with key business systems:
  • CRM integration (Salesforce, Dynamics)
  • ERP system connections
  • ITSM platforms (ServiceNow, Jira)
  • HR systems integration
  • Industry-specific application connectors
Integration capabilities:
  • Document context enrichment
  • Cross-system knowledge alignment
  • Business process integration
  • Metadata synchronization
  • Workflow orchestration
Build specialized connections for unique needs:
  • REST API for document operations
  • Webhook support for events
  • Custom connector development
  • Scripting and automation
  • ETL pipeline integration
Development options:
  • API documentation and SDKs
  • Integration templates
  • Event-driven architecture
  • Authentication mechanisms
  • Data transformation tools

​
Document Visualization

Understand your document collection through visual analytics:
  • Content Map
  • Document Structure
  • Usage Patterns
  • Health & Performance
Visualize document relationships and topics:
  • Topic clustering visualization
  • Document similarity mapping
  • Knowledge domain visualization
  • Content coverage analysis
  • Gap identification
Benefits:
  • Understand knowledge distribution
  • Identify related content
  • Discover connection patterns
  • Plan content development

​
Next Steps

Now that you understand document management in AI Knowledge, explore these related topics:

Create Knowledge Base

Follow a step-by-step guide to creating your first knowledge base

RAG Configuration

Fine-tune retrieval and response settings

Analytics

Track and improve knowledge base performance