Databricks SQL Agent

The Databricks SQL Agent provides an intelligent interface for Databricks SQL warehouses. It translates natural language queries into optimized SQL, handles connection management, and processes analytics queries while maintaining Databricks-specific optimizations.

Databricks SQL Agent workflow and architecture

Configuration Parameters

Required Input Parameters

databricks_hostname: Databricks workspace URL
access_token: Databricks personal access token
http_path: SQL warehouse HTTP path
natural_language_query: User query in natural language

Optional Configuration

llm: Language model configuration
- model_name: Name of the language model
- temperature: Response creativity (0-1)
- max_tokens: Maximum response length
connection_params: Additional connection parameters
- catalog: Databricks catalog name
- schema: Database schema
- warehouse_id: SQL warehouse ID
- timeout: Query timeout in seconds

Output Format

{
  "query_result": {
    "data": array,
    "schema": [
      {
        "name": string,
        "type": string,
        "nullable": boolean
      }
    ],
    "metadata": {
      "row_count": number,
      "bytes_processed": number,
      "execution_time": number,
      "warehouse_size": string
    }
  },
  "generated_query": {
    "sql": string,
    "analysis": {
      "tables_referenced": array,
      "columns_used": array,
      "filters_applied": array
    },
    "performance_info": {
      "photon_enabled": boolean,
      "partition_pruning": boolean,
      "cache_hit": boolean
    },
    "query_plan": {
      "plan_json": string,
      "estimated_cost": number,
      "optimization_applied": array
    }
  },
  "execution_stats": {
    "compilation_time": number,
    "execution_time": number,
    "data_scanned": number,
    "peak_memory_usage": number
  }
}

Features

Natural language to SQL translation
Photon query optimization
Delta Lake integration
Query performance monitoring
Schema inference
Resource usage tracking
Error handling and recovery
Query plan optimization

Note: Ensure proper warehouse sizing for your workload. Consider using Delta caching and Photon engine for better performance.

Tip: Use appropriate cluster sizing and auto-scaling settings. Monitor query performance and adjust configurations based on workload patterns.

Example Usage

const databricksAgent = new DatabricksSQLAgent({
  databricks_hostname: "your-workspace.cloud.databricks.com",
  access_token: "dapi****",
  http_path: "/sql/1.0/warehouses/abc123",
  llm: {
    model_name: "gpt-4",
    temperature: 0.3
  },
  connection_params: {
    catalog: "main",
    schema: "default",
    warehouse_id: "abc123",
    timeout: 300
  }
});

const results = await databricksAgent.query({
  natural_language_query: "What were the top 10 selling products last quarter?"
});