Text File

The Text File loader enables you to load and process content from various text-based file formats. It supports multiple file types and provides flexible options for text splitting and metadata handling.

Features

• Support for multiple text-based file formats
• Multiple file loading capability
• Text splitting support
• Customizable metadata handling
• Storage integration support
• Base64 file handling
• Multiple output formats

Supported File Types

The loader supports a wide range of text-based file formats:

• Text files (.txt)
• Web files (.html, .aspx, .asp, .css)
• Programming languages:
  • C/C++ (.cpp, .c, .h)
  • C# (.cs)
  • Go (.go)
  • Java (.java)
  • JavaScript/TypeScript (.js, .ts)
  • PHP (.php)
  • Python (.py, .python)
  • Ruby (.rb, .ruby)
  • Rust (.rs)
  • Scala (.sc, .scala)
  • Solidity (.sol)
  • Swift (.swift)
  • Visual Basic (.vb)
• Markup/Style:
  • CSS/LESS/SCSS (.css, .less, .scss)
  • Markdown (.md, .markdown)
  • XML (.xml)
  • LaTeX (.tex, .ltx)
• Other:
  • Protocol Buffers (.proto)
  • SQL (.sql)
  • RST (.rst)

Inputs

Required Parameters

• Txt File: One or more text files to process
  • Accepts files from local upload or storage
  • Supports multiple file selection

Optional Parameters

• Text Splitter: A text splitter to process the extracted content
• Additional Metadata: JSON object with additional metadata to add to documents
• Omit Metadata Keys: Comma-separated list of metadata keys to exclude
  • Format: key1, key2, key3.nestedKey1
  • Use * to remove all default metadata

Outputs

• Document: Array of document objects containing:
  • metadata: File metadata and custom fields
  • pageContent: Extracted text content
• Text: Concatenated string of all extracted content

Document Structure

Each document contains:

• pageContent: The main content from the text file
• metadata:
  • Default file metadata
  • Additional custom metadata (if specified)
  • Filtered metadata (based on omitted keys)

Usage Examples

Single File Processing

{
  "txtFile": "example.txt",
  "metadata": {
    "source": "local",
    "category": "documentation"
  }
}

Multiple Files Processing

{
  "txtFile": ["doc1.txt", "doc2.md", "code.py"],
  "metadata": {
    "batch": "docs-2024",
    "processor": "text-loader"
  },
  "omitMetadataKeys": "source, timestamp"
}

Storage Integration

The loader supports two file source modes:

1. Direct Upload: Files uploaded directly through the interface
2. Storage Integration: Files accessed through the storage system
   • Format: FILE-STORAGE::filename.txt
   • Supports organization and chatflow-specific storage

Notes

• Handles both single and multiple file processing
• Supports base64 encoded file content
• Automatically handles different file encodings
• Memory-efficient processing of large files
• Preserves file metadata when needed
• Supports text splitting for large documents
• Handles escape characters in output text
• Integrates with organization-specific storage