logo

Data Handlers

Data Handlers provide a comprehensive framework for processing various data sources, including files, API requests, and folders, enabling flexible data ingestion and transformation capabilities.

Data Handlers Architecture

Data Handlers Architecture

1.1 File Handler

Process individual files with advanced configuration options for file management and error handling.

1.1.1 Input Configuration

  • Path: File path to process
  • Server File Path: Optional server-side storage location
  • Silent Errors: Toggle error suppression
  • Delete Server File: Remove after processing
  • Ignore Unsupported Extensions: Skip unsupported file types
  • Ignore Unspecified Files: Skip files not explicitly listed
  • Processing Concurrency: Number of concurrent operations

1.2 API Request Handler

Make HTTP requests with comprehensive configuration options for API integration.

1.2.1 Input Configuration

  • URLs: Target endpoint URLs
  • CURL: CURL command string
  • Method: HTTP method (GET, POST, etc.)
  • Headers: Custom HTTP headers
  • Body: Request payload
  • Query Parameters: URL parameters
  • Timeout: Request timeout duration
  • Follow Redirects: Handle URL redirections
  • Save to File: Store response to file
  • Include HTTPX Metadata: Additional request metadata

1.3 Folder Handler

Process entire directories with customizable file filtering and traversal options.

1.3.1 Input Configuration

  • Path: Directory path to process
  • File Types: Allowed file extensions
  • Depth: Directory traversal depth
  • Max Concurrency: Parallel processing limit
  • Load Hidden: Include hidden files
  • Recursive: Process subdirectories
  • Silent Errors: Toggle error suppression
  • Use Multithreading: Enable parallel processing

Output Data (All Handlers)

  • Processed content
  • Metadata (file info, HTTP headers, etc.)
  • Processing statistics
  • Error reports (if any)

Note: Always implement proper error handling and validation for reliable data processing.

Tip: Use response caching and implement rate limiting to optimize API usage and prevent overload.