Parse Data Component

The Parse Data component provides powerful data parsing capabilities with customizable templates and separators. It transforms raw data into structured formats while maintaining data integrity and providing detailed parsing results.

Parse Data workflow and architecture

Configuration Parameters

Required Parameters

template: Parsing template configuration
- format: Template format (JSON, CSV, XML, etc.)
- schema: Data structure definition
- rules: Parsing rules and validations
data: Input data to parse

Optional Parameters

separator: Custom data separator
- character: Single character separator
- regex: Regular expression pattern
- custom: Custom separator function

Output Format

{
  "message": {
    "status": "success" | "partial" | "error",
    "summary": string,
    "details": {
      "parsed_count": number,
      "failed_count": number,
      "warning_count": number
    }
  },
  "data_list": [
    {
      "id": string,
      "parsed_data": {
        // Structured data based on template
        [key: string]: any
      },
      "metadata": {
        "source_line": number,
        "parse_time": number,
        "validation_results": {
          "passed": boolean,
          "errors": array,
          "warnings": array
        }
      }
    }
  ],
  "statistics": {
    "total_processing_time": number,
    "memory_usage": number,
    "template_matches": number,
    "separator_splits": number
  },
  "errors": [
    {
      "type": string,
      "message": string,
      "location": {
        "line": number,
        "column": number,
        "context": string
      }
    }
  ]
}

Features

Template-based parsing
Custom separators
Data validation
Error handling
Performance metrics
Batch processing
Format conversion
Data normalization

Note: Define clear templates with proper validation rules. Handle parsing errors gracefully for robust data processing.

Tip: Use custom separators for complex data formats. Implement batch processing for large datasets.

Example Usage

const parser = new DataParser({
  template: {
    format: "JSON",
    schema: {
      name: { type: "string", required: true },
      age: { type: "number", min: 0 },
      email: { type: "string", pattern: "email" },
      tags: { type: "array", items: "string" }
    },
    rules: [
      {
        field: "email",
        validate: "isValidEmail",
        message: "Invalid email format"
      }
    ]
  },
  separator: {
    type: "regex",
    pattern: "\n(?=\{)"
  }
});

const result = await parser.parse({
  data: rawDataString,
  options: {
    skipInvalid: true,
    normalizeStrings: true,
    maxErrors: 10
  }
});

Common Templates:

// JSON Template
{
  "format": "JSON",
  "schema": { ... }
}

// CSV Template
{
  "format": "CSV",
  "headers": ["name", "age", "email"],
  "types": ["string", "number", "string"]
}

// XML Template
{
  "format": "XML",
  "rootElement": "records",
  "schema": { ... }
}