logo

Documentation

Google Cloud Storage & Authentication Connectors

These connectors enable seamless integration with Google Cloud Storage services and authentication mechanisms, allowing your workflows to securely access and process data from Google's cloud infrastructure.

1.1 GCS Bucket Loader

The GCS Bucket Loader connects to Google Cloud Storage to retrieve files from a specified bucket. It allows your workflow to access multiple files within a bucket based on prefix filtering.

GCS Bucket Loader Interface

GCS Bucket Loader Interface

Use Cases

  • Batch processing documents stored in cloud storage
  • Loading multiple data files for analysis
  • Processing log files with common prefixes
  • Working with collections of related files
  • Integrating with data lake architectures

Inputs

  • Service Account Key File: Path to service account key file (required)

    Example: Upload a file...

  • GCP Project Name: Google Cloud project ID (required)

    Example: my-project-name

  • Bucket Name: GCS bucket name (required)

    Example: my-bucket

  • Prefix: File prefix filter within the bucket (required)

    Example: path/to/docs/

Outputs

A list of files from the bucket matching the prefix, with options to process each file individually or as a batch.

Example Output:

[
  {
    "name": "path/to/docs/document1.pdf",
    "size": 1245678,
    "updated": "2023-05-15T10:30:25Z",
    "contentType": "application/pdf"
  },
  {
    "name": "path/to/docs/document2.pdf",
    "size": 2356789,
    "updated": "2023-06-20T14:15:30Z",
    "contentType": "application/pdf"
  }
]

Implementation Notes

  • Use specific prefixes to limit the number of files returned
  • Consider implementing pagination for buckets with many files
  • Filter by file type or naming conventions when appropriate
  • Process files in parallel for better performance with large datasets

1.2 Gmail Loader

The Gmail Loader connects to Google's Gmail service to retrieve emails and their attachments. It enables access to email content for processing, analysis, or integration with other systems.

Gmail Loader Interface

Gmail Loader Interface

Use Cases

  • Processing incoming customer inquiries from email
  • Extracting data from structured email reports
  • Analyzing email communication patterns
  • Retrieving and processing email attachments
  • Building email-based workflow automations

Inputs

  • JSON String of the Service Account Token: OAuth token for Gmail API (required)

    Example: token : ya29.a0AfB_byC...

  • Label IDs: Gmail labels to filter messages by (required)

    Example: INBOX,SENT,UNREAD,IMPORTANT

  • Max Results: Maximum number of emails to retrieve (required)

    Example: 10

Outputs

Email messages with their content, metadata, and attachments.

Example Output:

[
  {
    "id": "18abc123def456",
    "threadId": "18abc123def456",
    "labelIds": ["INBOX", "UNREAD", "IMPORTANT"],
    "snippet": "Hello, I'd like to inquire about your services...",
    "payload": {
      "mimeType": "multipart/alternative",
      "headers": [
        {"name": "From", "value": "John Doe <john.doe@example.com>"},
        {"name": "To", "value": "support@yourcompany.com"},
        {"name": "Subject", "value": "Service Inquiry"},
        {"name": "Date", "value": "Mon, 15 May 2023 10:30:25 +0000"}
      ],
      "body": {...},
      "parts": [...]
    },
    "sizeEstimate": 15432
  }
]

Implementation Notes

  • Use specific label combinations to filter relevant emails
  • Implement proper handling for various email formats and encodings
  • Extract attachments appropriately based on their MIME types
  • Consider privacy implications when processing email content

1.3 Google OAuth Token

The Google OAuth Token connector provides authentication capabilities for Google services. It generates and manages OAuth tokens that can be used by other Google service connectors in your workflow.

Google OAuth Token Interface

Google OAuth Token Interface

Use Cases

  • Authenticating with Google services that require OAuth
  • Managing access tokens for multiple Google APIs
  • Implementing secure delegated access to user data
  • Enabling user-specific access to Google services
  • Supporting multi-user scenarios in applications

Inputs

  • Scopes: OAuth scopes needed for the required permissions (required)

    Example: https://www.googleapis.com/auth/drive.readonly,https://www.googleapis.com/auth/gmail.readonly

  • Credentials File: OAuth client credentials file (required)

    Example: Upload a file...

Outputs

OAuth token information that can be used with other Google API connectors.

Example Output:

{
  "token": "ya29.a0AfB_byC...",
  "refresh_token": "1//xEoDL4iW3cxlI...",
  "expiry": "2023-06-15T14:30:15Z",
  "scopes": [
    "https://www.googleapis.com/auth/drive.readonly",
    "https://www.googleapis.com/auth/gmail.readonly"
  ]
}

Implementation Notes

  • Request only the minimum required scopes for security best practices
  • Store refresh tokens securely for long-term access
  • Implement token refresh logic to handle expiration
  • Follow Google's OAuth best practices for handling credentials

Authentication & Security

  • Secure all credentials and tokens in environment variables or secret managers
  • Follow the principle of least privilege when requesting permissions
  • Regularly audit and rotate service account keys
  • Implement proper token refresh mechanisms for OAuth flows
  • Use service accounts for backend applications and OAuth for user-centric flows

Best Practices

  • Use specific file patterns and prefixes when working with storage
  • Implement retry logic for transient failures
  • Monitor API usage to stay within quotas and limits
  • Cache frequently accessed data to reduce API calls
  • Consider data locality for improved performance