# API Keys
Source: https://docs.logclaw.ai/api-keys

Create and manage API keys for authenticating log ingestion on LogClaw Cloud

# API Keys

API keys authenticate your application's log ingestion requests to LogClaw Cloud. Each key is scoped to a project and automatically associates incoming logs with the correct tenant.

***

## Getting Your API Key

<Steps>
  <Step title="Sign in to LogClaw Console">
    Go to [console.logclaw.ai](https://console.logclaw.ai) and sign in with your account. If you don't have an account yet, click **Sign Up** to create one.
  </Step>

  <Step title="Create or select a project">
    From the dashboard, create a new project or select an existing one. Each project has its own isolated log pipeline and API keys.
  </Step>

  <Step title="Navigate to API Keys">
    In your project, go to **Settings → API Keys**.
  </Step>

  <Step title="Create an API Key">
    Click **"Create API Key"**, give it a descriptive name (e.g. `production-backend`), and copy the key immediately. Keys start with `lc_proj_` and are only shown once.

    ```
    lc_proj_5a33915526cd1af8238c6e690eeff9cec2611750d9914d2dd1cc7a87cf08c717
    ```
  </Step>
</Steps>

<Note>
  API keys are shown only once at creation time. Store your key securely before closing the dialog.
</Note>

***

## Using Your API Key

Add the `x-logclaw-api-key` header to all ingestion requests:

```bash theme={null}
curl -X POST https://otel.logclaw.ai/v1/logs \
  -H "Content-Type: application/json" \
  -H "x-logclaw-api-key: $LOGCLAW_API_KEY" \
  -d '{
    "resourceLogs": [{
      "resource": {
        "attributes": [
          {"key": "service.name", "value": {"stringValue": "my-api"}}
        ]
      },
      "scopeLogs": [{
        "logRecords": [{
          "timeUnixNano": "'$(date +%s)000000000'",
          "severityText": "INFO",
          "body": {"stringValue": "Hello from LogClaw!"}
        }]
      }]
    }]
  }'
```

### Environment Variables

We recommend storing your API key in an environment variable:

```bash theme={null}
export LOGCLAW_API_KEY=lc_proj_your_key_here
```

For OpenTelemetry SDKs, use the standard OTLP environment variables:

```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.logclaw.ai
export OTEL_EXPORTER_OTLP_HEADERS=x-logclaw-api-key=lc_proj_your_key_here
export OTEL_EXPORTER_OTLP_PROTOCOL=http/json
```

These environment variables work with any OpenTelemetry SDK — Python, Node.js, Go, Java, .NET, Ruby, and Rust — with zero code changes.

***

## Tenant ID Auto-Injection

<Tip>
  You do **not** need to add a `tenant_id` to your logs. LogClaw automatically injects the correct tenant identifier based on your API key's project. Any `tenant_id` already present in the payload is stripped and replaced to prevent spoofing.
</Tip>

This means your application code stays clean — just send standard OTLP logs and LogClaw handles multi-tenant isolation for you.

***

## Security Best Practices

<CardGroup>
  <Card title="Use environment variables" icon="terminal">
    Never hardcode API keys in source code. Use environment variables or a secret manager (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault).
  </Card>

  <Card title="Rotate regularly" icon="rotate">
    Create new keys and revoke old ones periodically. You can have multiple active keys per project to enable zero-downtime rotation.
  </Card>

  <Card title="Least privilege" icon="shield-halved">
    Create separate API keys for each environment (staging, production) and each service to limit blast radius.
  </Card>

  <Card title="Revoke compromised keys" icon="ban">
    If a key is exposed, revoke it immediately from Settings → API Keys. A new key can be created in seconds.
  </Card>
</CardGroup>

***

## Revoking API Keys

1. Go to your project in [console.logclaw.ai](https://console.logclaw.ai)
2. Navigate to **Settings → API Keys**
3. Find the key you want to revoke and click **Revoke**
4. The key stops working immediately — any requests using it will receive a `401 Unauthorized` response

<Note>
  Revoking a key is irreversible. Create a new key before revoking the old one if you need uninterrupted log ingestion.
</Note>

***

## Self-Hosted Deployments

If you're running LogClaw self-hosted, API keys are not required. Logs are sent directly to the OTel Collector endpoint and access is controlled via Kubernetes NetworkPolicy. See the [Quick Start](/quickstart-send-logs) for self-hosted endpoint details.


# Infrastructure Agent API
Source: https://docs.logclaw.ai/api-reference/agent

Cluster health monitoring API from the Go-based infrastructure agent.

# Infrastructure Agent API

The Infrastructure Agent is a Go-based service that collects health metrics from Kubernetes infrastructure components and exposes them via HTTP endpoints.

**Base URL:** `http://logclaw-agent:8080`
**Dashboard proxy:** `/api/agent/*`

***

## Health Check

```
GET /health
```

Liveness probe endpoint.

### Response

```json theme={null}
{
  "status": "ok"
}
```

***

## Readiness Check

```
GET /ready
```

Readiness probe endpoint. Returns `200` only when all collectors are initialized.

### Response (Ready)

```json theme={null}
{
  "status": "ready",
  "collectors": {
    "kafka": "initialized",
    "flink": "initialized",
    "opensearch": "initialized",
    "eso": "initialized"
  }
}
```

### Response (Not Ready)

```json theme={null}
{
  "status": "not_ready",
  "collectors": {
    "kafka": "initialized",
    "flink": "initializing",
    "opensearch": "initialized",
    "eso": "error"
  }
}
```

***

## Infrastructure Metrics

```
GET /metrics
```

Returns aggregated infrastructure metrics from all collectors. Used by the Dashboard's pipeline monitoring view.

### Response

```json theme={null}
{
  "kafka": {
    "status": "healthy",
    "brokers": 3,
    "topics": {
      "raw-logs": {
        "partitions": 6,
        "consumerLag": 150
      },
      "enriched-logs": {
        "partitions": 6,
        "consumerLag": 0
      }
    }
  },
  "flink": {
    "status": "healthy",
    "jobs": [
      {
        "name": "logclaw-anomaly-detection",
        "state": "RUNNING",
        "taskManagers": 2,
        "uptime": "48h30m"
      }
    ]
  },
  "opensearch": {
    "status": "green",
    "nodes": 6,
    "indices": 15,
    "totalDocs": 1542000,
    "totalSizeBytes": 2147483648,
    "activePrimaryShards": 15,
    "activeShards": 30
  },
  "eso": {
    "status": "healthy",
    "externalSecrets": [
      {
        "name": "logclaw-ticketing-credentials",
        "status": "SecretSynced",
        "lastSyncTime": "2024-03-01T15:00:00Z"
      }
    ]
  }
}
```

### Metric Fields

#### Kafka

| Field                  | Type    | Description                                         |
| ---------------------- | ------- | --------------------------------------------------- |
| `status`               | string  | Overall Kafka health: `healthy`, `degraded`, `down` |
| `brokers`              | integer | Number of active Kafka brokers                      |
| `topics[].partitions`  | integer | Number of partitions per topic                      |
| `topics[].consumerLag` | integer | Consumer lag (unconsumed messages)                  |

#### Flink

| Field                 | Type    | Description                                 |
| --------------------- | ------- | ------------------------------------------- |
| `status`              | string  | Flink health: `healthy`, `degraded`, `down` |
| `jobs[].name`         | string  | Job deployment name                         |
| `jobs[].state`        | string  | Job state: `RUNNING`, `FAILED`, `SUSPENDED` |
| `jobs[].taskManagers` | integer | Number of active task managers              |

#### OpenSearch

| Field            | Type    | Description                              |
| ---------------- | ------- | ---------------------------------------- |
| `status`         | string  | Cluster health: `green`, `yellow`, `red` |
| `nodes`          | integer | Total cluster nodes                      |
| `totalDocs`      | integer | Total indexed documents                  |
| `totalSizeBytes` | integer | Total index storage in bytes             |

#### External Secrets Operator

| Field                            | Type   | Description                                    |
| -------------------------------- | ------ | ---------------------------------------------- |
| `status`                         | string | ESO health: `healthy`, `degraded`, `down`      |
| `externalSecrets[].status`       | string | Sync status: `SecretSynced`, `SecretSyncError` |
| `externalSecrets[].lastSyncTime` | string | ISO-8601 timestamp of last sync                |

***

## Configuration

### Environment Variables

| Variable            | Required | Default           | Description                          |
| ------------------- | -------- | ----------------- | ------------------------------------ |
| `LOGCLAW_TENANT_ID` | Yes      | —                 | Tenant identifier for filtering CRDs |
| `LOGCLAW_NAMESPACE` | No       | Current namespace | Kubernetes namespace to watch        |
| `PORT`              | No       | `8080`            | HTTP server port                     |


# Bridge API
Source: https://docs.logclaw.ai/api-reference/bridge

Bridge service API for health checks, metrics, and runtime configuration.

# Bridge API

The Bridge exposes HTTP endpoints for health monitoring, Prometheus metrics, and runtime configuration management.

**Base URL:** `http://logclaw-bridge:8080`
**Dashboard proxy:** `/api/bridge/*`

## Health Check

```
GET /health
```

Returns the health status of all Bridge threads and connections.

### Response

```json theme={null}
{
  "status": "ok",
  "threads": {
    "etl": "running",
    "anomaly": "running",
    "indexer": "running",
    "lifecycle": "running"
  },
  "kafka": "connected",
  "opensearch": "connected"
}
```

| Field        | Type   | Description                      |
| ------------ | ------ | -------------------------------- |
| `status`     | string | Overall status: `ok` or `error`  |
| `threads`    | object | Status of each processing thread |
| `kafka`      | string | Kafka connection status          |
| `opensearch` | string | OpenSearch connection status     |

***

## Prometheus Metrics

```
GET /metrics
```

Returns Prometheus-format metrics for monitoring and alerting.

### Metrics

| Metric                                      | Type    | Labels | Description                                        |
| ------------------------------------------- | ------- | ------ | -------------------------------------------------- |
| `logclaw_bridge_etl_consumed_total`         | counter | —      | Kafka messages (batches) consumed from `raw-logs`  |
| `logclaw_bridge_etl_records_received_total` | counter | —      | Individual OTLP log records unpacked and flattened |
| `logclaw_bridge_etl_produced_total`         | counter | —      | Enriched documents written to `enriched-logs`      |
| `logclaw_bridge_anomalies_detected_total`   | counter | —      | Anomalies detected by the Z-score engine           |
| `logclaw_bridge_opensearch_indexed_total`   | counter | —      | Documents successfully indexed into OpenSearch     |
| `logclaw_bridge_opensearch_errors_total`    | counter | —      | OpenSearch indexing errors                         |

### Example Response

```
# HELP logclaw_bridge_etl_consumed_total Kafka messages consumed
# TYPE logclaw_bridge_etl_consumed_total counter
logclaw_bridge_etl_consumed_total 47

# HELP logclaw_bridge_etl_records_received_total OTLP log records received
# TYPE logclaw_bridge_etl_records_received_total counter
logclaw_bridge_etl_records_received_total 12450

# HELP logclaw_bridge_etl_produced_total Enriched documents produced
# TYPE logclaw_bridge_etl_produced_total counter
logclaw_bridge_etl_produced_total 12450

# HELP logclaw_bridge_anomalies_detected_total Anomalies detected
# TYPE logclaw_bridge_anomalies_detected_total counter
logclaw_bridge_anomalies_detected_total 3
```

<Note>
  `etl_consumed_total` counts **Kafka messages** (batches), while `etl_records_received_total` counts **individual log records**. A single Kafka message can contain hundreds of OTLP log records. Use `etl_records_received_total` for accurate log volume tracking.
</Note>

***

## Get Configuration

```
GET /config
```

Returns the current runtime configuration.

### Response

```json theme={null}
{
  "anomalyThreshold": 2.5,
  "windowSize": 50,
  "kafkaBrokers": "logclaw-kafka-kafka-bootstrap:9093",
  "rawTopic": "raw-logs",
  "enrichedTopic": "enriched-logs",
  "opensearchEndpoint": "https://logclaw-opensearch:9200"
}
```

***

## Update Configuration

```
PATCH /config
```

Update runtime configuration dynamically. Changes take effect immediately without restart.

### Request Body

```json theme={null}
{
  "anomalyThreshold": 3.0,
  "windowSize": 100
}
```

| Field              | Type    | Description                                               |
| ------------------ | ------- | --------------------------------------------------------- |
| `anomalyThreshold` | number  | Z-score threshold for anomaly detection (default: 2.5)    |
| `windowSize`       | integer | Sliding window size for anomaly calculation (default: 50) |

### Response

```json theme={null}
{
  "status": "ok",
  "config": {
    "anomalyThreshold": 3.0,
    "windowSize": 100
  }
}
```

<Warning>
  Runtime config changes are not persisted across pod restarts. For permanent changes, update the Helm values and redeploy.
</Warning>


# Agent Health
Source: https://docs.logclaw.ai/api-reference/endpoints/agent-health

GET /api/agent/health
Liveness probe for the Infrastructure Agent.



# Infrastructure Metrics
Source: https://docs.logclaw.ai/api-reference/endpoints/agent-metrics

GET /api/agent/metrics
Returns aggregated infrastructure health metrics from all cluster
collectors: Kafka broker and topic status, Flink job states, OpenSearch
cluster health, and External Secrets sync status.




# Bridge Configuration
Source: https://docs.logclaw.ai/api-reference/endpoints/bridge-config

GET /api/bridge/config
Returns the current Bridge runtime configuration including anomaly detection
thresholds, Kafka topics, and OpenSearch endpoint.




# Bridge Health
Source: https://docs.logclaw.ai/api-reference/endpoints/bridge-health

GET /api/bridge/health
Returns the health status of all Bridge processing threads and backend
connections (Kafka, OpenSearch).




# Create API Key
Source: https://docs.logclaw.ai/api-reference/endpoints/create-api-key

POST /api/admin/api-keys
Create a new API key for the project. The full key value is returned **only
once** in the response — store it securely.

Keys start with `lc_proj_` and are 64-character hex strings.




# List API Keys
Source: https://docs.logclaw.ai/api-reference/endpoints/list-api-keys

GET /api/admin/api-keys
Returns all API keys for the authenticated project. Key values are masked —
only the prefix is returned.




# List Incidents
Source: https://docs.logclaw.ai/api-reference/endpoints/list-incidents

GET /api/ticketing/api/incidents
Returns all tracked incidents for your project with their current state,
severity, affected services, and ticketing platform status.

Results are automatically scoped to your project's tenant.




# List Indices
Source: https://docs.logclaw.ai/api-reference/endpoints/list-indices

GET /api/opensearch/_cat/indices
Returns all OpenSearch indices with their health status, document count,
and storage size. LogClaw uses daily indices with the pattern
`logclaw-logs-YYYY.MM.dd`.




# Revoke API Key
Source: https://docs.logclaw.ai/api-reference/endpoints/revoke-api-key

DELETE /api/admin/api-keys
Permanently revoke an API key. The key stops working immediately — any
requests using it will receive `401 Unauthorized`. This action is irreversible.




# Runtime Configuration
Source: https://docs.logclaw.ai/api-reference/endpoints/runtime-config

GET /api/ticketing/api/v1/config
Returns the complete Ticketing Agent runtime configuration including
platform settings, severity routing rules, anomaly thresholds, and LLM config.

Sensitive fields (API keys, tokens) are masked with `****`.




# Send Logs
Source: https://docs.logclaw.ai/api-reference/endpoints/send-logs

POST /v1/logs
Ingest log records in OTLP HTTP/JSON format. This is the primary ingestion
endpoint for LogClaw Cloud. Logs are processed through the pipeline:
OTel Collector → Kafka → Bridge → OpenSearch.

The auth proxy automatically injects your project's `tenant_id` into each
log record based on your API key.




# Ingestion API
Source: https://docs.logclaw.ai/api-reference/ingestion

OTLP log ingestion endpoints via the OpenTelemetry Collector.

# Ingestion API

Logs are ingested into LogClaw exclusively via **OTLP (OpenTelemetry Protocol)**. The OTel Collector accepts both gRPC and HTTP transports.

## Send Logs (HTTP/JSON)

<ParamField type="string">
  HTTP method
</ParamField>

```
POST /v1/logs
```

Send log records in OTLP HTTP/JSON format.

**LogClaw Cloud (managed):** `POST https://otel.logclaw.ai/v1/logs`
**Via Dashboard proxy (self-hosted):** `POST /api/otel/v1/logs`
**Direct (self-hosted):** `POST http://logclaw-otel-collector:4318/v1/logs`

### Authentication

<Note>
  **LogClaw Cloud** requires an API key via the `x-logclaw-api-key` header. Get your key from [console.logclaw.ai](https://console.logclaw.ai) under **Settings → API Keys**. See [API Keys](/api-keys) for details.

  **Self-hosted** deployments do not require authentication — access is controlled at the Kubernetes NetworkPolicy level.
</Note>

### Request Headers

| Header              | Value                                                                  |
| ------------------- | ---------------------------------------------------------------------- |
| `Content-Type`      | `application/json`                                                     |
| `x-logclaw-api-key` | `lc_proj_...` (required for LogClaw Cloud, not needed for self-hosted) |

### Request Body

```json theme={null}
{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": { "stringValue": "payment-api" }
          },
          {
            "key": "host.name",
            "value": { "stringValue": "pod-abc123" }
          }
        ]
      },
      "scopeLogs": [
        {
          "logRecords": [
            {
              "timeUnixNano": "1709312400000000000",
              "severityText": "ERROR",
              "severityNumber": 17,
              "body": {
                "stringValue": "Connection refused to database"
              },
              "traceId": "abcdef1234567890abcdef1234567890",
              "spanId": "abcdef1234567890",
              "attributes": [
                {
                  "key": "environment",
                  "value": { "stringValue": "production" }
                },
                {
                  "key": "region",
                  "value": { "stringValue": "us-east-1" }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}
```

### Response

**200 OK**

```json theme={null}
{
  "partialSuccess": {}
}
```

An empty `partialSuccess` object means all records were accepted.

### OTLP Field Reference

| Field                 | Type    | Required | Description                                               |
| --------------------- | ------- | -------- | --------------------------------------------------------- |
| `resourceLogs`        | array   | Yes      | Array of resource log groups                              |
| `resource.attributes` | array   | No       | Resource-level attributes (service.name, host.name, etc.) |
| `scopeLogs`           | array   | Yes      | Array of scope log groups                                 |
| `logRecords`          | array   | Yes      | Array of individual log records                           |
| `timeUnixNano`        | string  | No       | Timestamp in nanoseconds since epoch                      |
| `severityText`        | string  | No       | Severity level: TRACE, DEBUG, INFO, WARN, ERROR, FATAL    |
| `severityNumber`      | integer | No       | Numeric severity (1-24)                                   |
| `body.stringValue`    | string  | Yes      | The log message                                           |
| `traceId`             | string  | No       | 32-character hex trace ID                                 |
| `spanId`              | string  | No       | 16-character hex span ID                                  |
| `attributes`          | array   | No       | Additional key-value attributes                           |

### Attribute Value Types

OTLP attributes support multiple value types:

```json theme={null}
{"key": "name", "value": {"stringValue": "hello"}}
{"key": "count", "value": {"intValue": "42"}}
{"key": "ratio", "value": {"doubleValue": 3.14}}
{"key": "enabled", "value": {"boolValue": true}}
```

## Send Logs (gRPC)

**LogClaw Cloud (managed):** `grpc://otel.logclaw.ai:443`
**Direct (self-hosted):** `grpc://logclaw-otel-collector:4317`

Use the OTLP gRPC exporter from any OpenTelemetry SDK. This is the **recommended** transport for production workloads — binary Protobuf is more compact and efficient than JSON.

<Note>
  LogClaw Cloud gRPC uses TLS on port 443. Set your SDK's `insecure` option to `false` (the default). Authentication via `x-logclaw-api-key` header is required — the same as HTTP.
</Note>

See the [OTLP Integration Guide](/otlp-integration) for SDK examples in Python, Java, Node.js, and Go.

## Health Check

```
GET http://logclaw-otel-collector:13133/
```

**Response:**

```json theme={null}
{
  "status": "Server available",
  "upSince": "2024-01-15T10:30:00Z",
  "uptime": "48h30m"
}
```

## Examples

### curl — Single Log (LogClaw Cloud)

```bash theme={null}
curl -X POST https://otel.logclaw.ai/v1/logs \
  -H "Content-Type: application/json" \
  -H "x-logclaw-api-key: $LOGCLAW_API_KEY" \
  -d '{
    "resourceLogs": [{
      "resource": {
        "attributes": [
          {"key": "service.name", "value": {"stringValue": "my-app"}}
        ]
      },
      "scopeLogs": [{
        "logRecords": [{
          "timeUnixNano": "'$(date +%s)000000000'",
          "severityText": "ERROR",
          "body": {"stringValue": "Connection timeout"}
        }]
      }]
    }]
  }'
```

### curl — Single Log (Self-Hosted)

```bash theme={null}
curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [{
      "resource": {
        "attributes": [
          {"key": "service.name", "value": {"stringValue": "my-app"}}
        ]
      },
      "scopeLogs": [{
        "logRecords": [{
          "timeUnixNano": "'$(date +%s)000000000'",
          "severityText": "ERROR",
          "body": {"stringValue": "Connection timeout"}
        }]
      }]
    }]
  }'
```

### curl — Batch (Multiple Records, LogClaw Cloud)

```bash theme={null}
curl -X POST https://otel.logclaw.ai/v1/logs \
  -H "Content-Type: application/json" \
  -H "x-logclaw-api-key: $LOGCLAW_API_KEY" \
  -d '{
    "resourceLogs": [{
      "resource": {
        "attributes": [
          {"key": "service.name", "value": {"stringValue": "batch-test"}}
        ]
      },
      "scopeLogs": [{
        "logRecords": [
          {
            "timeUnixNano": "'$(date +%s)000000000'",
            "severityText": "INFO",
            "body": {"stringValue": "Request started"}
          },
          {
            "timeUnixNano": "'$(date +%s)000000000'",
            "severityText": "ERROR",
            "body": {"stringValue": "Database connection failed"}
          },
          {
            "timeUnixNano": "'$(date +%s)000000000'",
            "severityText": "WARN",
            "body": {"stringValue": "Retrying in 5 seconds"}
          }
        ]
      }]
    }]
  }'
```


# OpenSearch API
Source: https://docs.logclaw.ai/api-reference/opensearch

OpenSearch query and index management endpoints proxied through the Dashboard.

# OpenSearch API

LogClaw stores processed logs in OpenSearch. The Dashboard proxies OpenSearch API requests with automatic Basic Auth injection.

**Base URL:** `https://logclaw-opensearch:9200`
**Dashboard proxy:** `/api/opensearch/*`

## Index Pattern

LogClaw writes to daily indices: `logclaw-logs-YYYY.MM.dd`

Wildcard for all logs: `logclaw-logs-*`

***

## Cluster Health

```
GET /_cluster/health
```

### Response

```json theme={null}
{
  "cluster_name": "logclaw-opensearch",
  "status": "green",
  "number_of_nodes": 6,
  "number_of_data_nodes": 3,
  "active_primary_shards": 15,
  "active_shards": 30
}
```

| Status   | Meaning                                                  |
| -------- | -------------------------------------------------------- |
| `green`  | All primary and replica shards assigned                  |
| `yellow` | All primary shards assigned, some replicas unassigned    |
| `red`    | Some primary shards unassigned — data may be unavailable |

***

## List Indices

```
GET /_cat/indices?format=json
```

Returns all indices with size, document count, and health status.

### Response

```json theme={null}
[
  {
    "health": "green",
    "status": "open",
    "index": "logclaw-logs-2024.03.01",
    "docs.count": "15420",
    "store.size": "12.3mb"
  }
]
```

***

## Search Logs

```
POST /logclaw-logs-*/_search
```

Query logs using the OpenSearch Query DSL.

### Request Body

```json theme={null}
{
  "size": 50,
  "sort": [{ "timestamp": "desc" }],
  "query": {
    "bool": {
      "must": [
        { "match": { "level": "ERROR" } },
        { "match": { "service": "payment-api" } }
      ],
      "filter": [
        {
          "range": {
            "timestamp": {
              "gte": "now-1h",
              "lte": "now"
            }
          }
        }
      ]
    }
  }
}
```

### Response

```json theme={null}
{
  "hits": {
    "total": { "value": 42, "relation": "eq" },
    "hits": [
      {
        "_index": "logclaw-logs-2024.03.01",
        "_id": "abc123",
        "_source": {
          "timestamp": "2024-03-01T15:30:00Z",
          "service": "payment-api",
          "level": "ERROR",
          "message": "Connection refused to database",
          "trace_id": "abcdef1234567890",
          "span_id": "1234567890ab",
          "host": "pod-xyz",
          "tenant_id": "acme-corp",
          "anomaly_score": 3.2,
          "is_anomaly": true
        }
      }
    ]
  }
}
```

### Common Query Patterns

**All errors in the last hour:**

```json theme={null}
{
  "query": {
    "bool": {
      "must": [{ "match": { "level": "ERROR" } }],
      "filter": [{ "range": { "timestamp": { "gte": "now-1h" } } }]
    }
  }
}
```

**Logs by trace ID:**

```json theme={null}
{
  "query": {
    "term": { "trace_id": "abcdef1234567890abcdef1234567890" }
  },
  "sort": [{ "timestamp": "asc" }]
}
```

**Anomalies above threshold:**

```json theme={null}
{
  "query": {
    "bool": {
      "must": [
        { "term": { "is_anomaly": true } },
        { "range": { "anomaly_score": { "gte": 2.5 } } }
      ]
    }
  }
}
```

**Full-text search:**

```json theme={null}
{
  "query": {
    "match": { "message": "connection timeout database" }
  }
}
```

***

## Document Count

```
GET /logclaw-logs-*/_count
```

### Response

```json theme={null}
{
  "count": 154200,
  "_shards": {
    "total": 15,
    "successful": 15,
    "failed": 0
  }
}
```

***

## Document Schema

Each indexed log document has these fields:

| Field           | Type    | Description                                           |
| --------------- | ------- | ----------------------------------------------------- |
| `timestamp`     | date    | ISO-8601 timestamp                                    |
| `service`       | keyword | Service name (from `service.name` resource attribute) |
| `level`         | keyword | Log level: INFO, WARN, ERROR, FATAL                   |
| `message`       | text    | Log message body                                      |
| `trace_id`      | keyword | Distributed trace ID                                  |
| `span_id`       | keyword | Span ID                                               |
| `host`          | keyword | Hostname or pod name                                  |
| `tenant_id`     | keyword | Tenant identifier                                     |
| `anomaly_score` | float   | Z-score from anomaly detection (0 if normal)          |
| `is_anomaly`    | boolean | Whether the document triggered an anomaly alert       |
| `batch_id`      | keyword | Upload batch identifier (for file uploads)            |
| `environment`   | keyword | Environment tag (production, staging, dev)            |
| `region`        | keyword | Cloud region                                          |

<Note>
  Custom OTLP attributes are flattened as top-level fields. For example, `{"key": "user.id", "value": {"stringValue": "12345"}}` becomes `"user.id": "12345"` in the indexed document.
</Note>


# API Reference
Source: https://docs.logclaw.ai/api-reference/overview

Complete API documentation for all LogClaw services with interactive playground.

# API Reference

LogClaw exposes APIs through three access patterns:

1. **LogClaw Cloud (otel.logclaw\.ai)** — all endpoints are authenticated via `x-logclaw-api-key` header and routed through the Auth Proxy. Use this for LogClaw Cloud deployments. Every endpoint in this reference includes an interactive playground — paste your API key and send requests directly from the docs.

2. **Dashboard Proxy** — the Next.js Dashboard proxies requests to backend services under `/api/<service>/`. Use this for browser-based access in self-hosted deployments.

3. **Direct Service Access** — each backend service exposes its own API on its Kubernetes ClusterIP service. Use this for programmatic access from within the cluster.

## Authentication

<Note>
  **LogClaw Cloud** requires an API key for all API requests. Include the `x-logclaw-api-key` header with every request:

  ```bash theme={null}
  curl -H "x-logclaw-api-key: lc_proj_your_key_here" \
    https://otel.logclaw.ai/api/agent/health
  ```

  Generate keys from your project dashboard at [console.logclaw.ai](https://console.logclaw.ai) under **Settings → API Keys**. See [API Keys](/api-keys) for details.

  **Self-hosted** deployments do not require API keys — access is controlled at the Kubernetes NetworkPolicy level.
</Note>

The Auth Proxy automatically injects the correct `tenant_id` into all requests based on your API key's project. You never need to specify a tenant ID manually.

## Try It Live

Every endpoint page includes an interactive **API Playground** on the right side. Enter your `x-logclaw-api-key` and send real requests to `otel.logclaw.ai` directly from the docs.

## Service Endpoints

| Service                                                 | Cloud (otel.logclaw\.ai) | Dashboard Proxy       | Direct (in-cluster)              | Port                     |
| ------------------------------------------------------- | ------------------------ | --------------------- | -------------------------------- | ------------------------ |
| [Ingestion](/api-reference/endpoints/send-logs)         | `POST /v1/logs`          | `/api/otel/v1/logs`   | `logclaw-otel-collector:4318`    | 4317 (gRPC), 4318 (HTTP) |
| [API Keys](/api-reference/endpoints/list-api-keys)      | `/api/admin/api-keys`    | `/api/admin/api-keys` | Dashboard internal               | —                        |
| [Bridge](/api-reference/endpoints/bridge-health)        | `/api/bridge/*`          | `/api/bridge/*`       | `logclaw-bridge:8080`            | 8080                     |
| [OpenSearch](/api-reference/endpoints/list-indices)     | `/api/opensearch/*`      | `/api/opensearch/*`   | `logclaw-opensearch:9200`        | 9200                     |
| [Ticketing](/api-reference/endpoints/list-incidents)    | `/api/ticketing/*`       | `/api/ticketing/*`    | `logclaw-ticketing-agent:18081`  | 18081                    |
| [Infrastructure](/api-reference/endpoints/agent-health) | `/api/agent/*`           | `/api/agent/*`        | `logclaw-agent:8080`             | 8080                     |
| Airflow                                                 | `/api/airflow/*`         | `/api/airflow/*`      | `logclaw-airflow-webserver:8080` | 8080                     |
| Feast                                                   | `/api/feast/*`           | `/api/feast/*`        | `logclaw-feast:6567`             | 6567                     |

## All Endpoints

| Method   | Path                                                                      | Description             |
| -------- | ------------------------------------------------------------------------- | ----------------------- |
| `POST`   | [`/v1/logs`](/api-reference/endpoints/send-logs)                          | Ingest logs (OTLP/HTTP) |
| `GET`    | [`/api/admin/api-keys`](/api-reference/endpoints/list-api-keys)           | List API keys           |
| `POST`   | [`/api/admin/api-keys`](/api-reference/endpoints/create-api-key)          | Create API key          |
| `DELETE` | [`/api/admin/api-keys`](/api-reference/endpoints/revoke-api-key)          | Revoke API key          |
| `GET`    | [`/api/bridge/health`](/api-reference/endpoints/bridge-health)            | Bridge health           |
| `GET`    | [`/api/bridge/config`](/api-reference/endpoints/bridge-config)            | Bridge configuration    |
| `GET`    | [`/api/ticketing/api/incidents`](/api-reference/endpoints/list-incidents) | List incidents          |
| `GET`    | [`/api/ticketing/api/v1/config`](/api-reference/endpoints/runtime-config) | Runtime configuration   |
| `GET`    | [`/api/agent/health`](/api-reference/endpoints/agent-health)              | Agent liveness          |
| `GET`    | [`/api/agent/metrics`](/api-reference/endpoints/agent-metrics)            | Infrastructure metrics  |
| `GET`    | [`/api/opensearch/_cat/indices`](/api-reference/endpoints/list-indices)   | List indices            |

## Common Response Formats

### Health Checks

All services implement a standard health check:

```json theme={null}
{
  "status": "ok"
}
```

### Error Responses

```json theme={null}
{
  "error": "description of the error",
  "status": 500
}
```

### Authentication Errors

```json theme={null}
{
  "error": "Missing x-logclaw-api-key header"
}
```

### Proxy Errors

When a backend service is unreachable, the Auth Proxy returns:

```json theme={null}
{
  "error": "Failed to forward request to Console API",
  "status": 502
}
```


# Ticketing API
Source: https://docs.logclaw.ai/api-reference/ticketing

Incident management and runtime configuration API for the Ticketing Agent.

# Ticketing API

The Ticketing Agent provides APIs for incident management, platform configuration, severity routing, and connectivity testing.

**Base URL:** `http://logclaw-ticketing-agent:18081`
**Dashboard proxy:** `/api/ticketing/*`

***

## Incidents

### List Incidents

```
GET /api/incidents
```

Returns all tracked incidents with their current state and metadata.

#### Response

```json theme={null}
[
  {
    "id": "inc-20240301-001",
    "title": "High error rate on payment-api",
    "severity": "critical",
    "status": "open",
    "service": "payment-api",
    "anomalyScore": 4.2,
    "affectedServices": ["payment-api", "order-engine", "notification-svc"],
    "traceId": "abcdef1234567890",
    "createdAt": "2024-03-01T15:30:00Z",
    "updatedAt": "2024-03-01T15:30:00Z",
    "platforms": ["pagerduty", "jira", "slack"]
  }
]
```

### Acknowledge Incident

```
POST /api/incidents/:id/acknowledge
```

Transition an incident from `open` to `acknowledged`.

#### Response

```json theme={null}
{
  "id": "inc-20240301-001",
  "status": "acknowledged",
  "acknowledgedAt": "2024-03-01T15:35:00Z"
}
```

### Resolve Incident

```
POST /api/incidents/:id/resolve
```

Transition an incident to `resolved`.

#### Response

```json theme={null}
{
  "id": "inc-20240301-001",
  "status": "resolved",
  "resolvedAt": "2024-03-01T16:00:00Z"
}
```

### Escalate Incident

```
POST /api/incidents/:id/escalate
```

Escalate an incident to higher-priority platforms.

#### Response

```json theme={null}
{
  "id": "inc-20240301-001",
  "status": "escalated",
  "escalatedAt": "2024-03-01T15:45:00Z",
  "escalatedTo": ["pagerduty"]
}
```

***

## Runtime Configuration

### Get Full Configuration

```
GET /api/v1/config
```

Returns the complete runtime configuration including all platform settings, routing rules, anomaly thresholds, and LLM configuration.

#### Response

```json theme={null}
{
  "platforms": {
    "pagerduty": { "enabled": true, "routingKey": "****" },
    "jira": { "enabled": true, "baseUrl": "https://org.atlassian.net" },
    "servicenow": { "enabled": false },
    "opsgenie": { "enabled": false },
    "slack": { "enabled": true, "webhookUrl": "****" }
  },
  "routing": {
    "critical": ["pagerduty", "slack", "jira"],
    "high": ["jira", "slack"],
    "medium": ["jira"],
    "low": ["slack"]
  },
  "anomaly": {
    "minimumScore": 0.85,
    "lookbackWindow": "15m"
  },
  "llm": {
    "provider": "openai",
    "model": "gpt-4"
  }
}
```

<Note>
  Sensitive fields (API keys, tokens, passwords) are masked with `****` in the response. Use the PATCH endpoints to update them.
</Note>

### Update Platform Settings

```
PATCH /api/v1/config/platforms
```

Update one or more ticketing platform configurations.

#### Request Body

```json theme={null}
{
  "pagerduty": {
    "enabled": true,
    "routingKey": "your-pagerduty-routing-key"
  },
  "slack": {
    "enabled": true,
    "webhookUrl": "https://hooks.slack.com/services/T.../B.../..."
  }
}
```

#### Required Fields Per Platform

| Platform   | Required Fields                       |
| ---------- | ------------------------------------- |
| PagerDuty  | `routingKey`                          |
| Jira       | `baseUrl`, `apiToken`, `userEmail`    |
| ServiceNow | `instanceUrl`, `username`, `password` |
| OpsGenie   | `apiKey`                              |
| Slack      | `webhookUrl`                          |

#### Response

```json theme={null}
{
  "status": "ok",
  "updated": ["pagerduty", "slack"]
}
```

### Update Routing Rules

```
PATCH /api/v1/config/routing
```

Configure which platforms receive incidents for each severity level.

#### Request Body

```json theme={null}
{
  "critical": ["pagerduty", "slack", "jira"],
  "high": ["jira", "slack"],
  "medium": ["jira"],
  "low": ["slack"]
}
```

#### Response

```json theme={null}
{
  "status": "ok",
  "routing": {
    "critical": ["pagerduty", "slack", "jira"],
    "high": ["jira", "slack"],
    "medium": ["jira"],
    "low": ["slack"]
  }
}
```

### Update Anomaly Settings

```
PATCH /api/v1/config/anomaly
```

#### Request Body

```json theme={null}
{
  "minimumScore": 0.9,
  "lookbackWindow": "30m"
}
```

| Field            | Type   | Default | Description                                        |
| ---------------- | ------ | ------- | -------------------------------------------------- |
| `minimumScore`   | number | `0.85`  | Minimum anomaly score to trigger incident creation |
| `lookbackWindow` | string | `15m`   | Time window for anomaly grouping and deduplication |

### Update LLM Settings

```
PATCH /api/v1/config/llm
```

#### Request Body

```json theme={null}
{
  "provider": "openai",
  "model": "gpt-4",
  "apiKey": "sk-..."
}
```

***

## Connectivity Testing

### Test Platform Connection

```
POST /api/v1/test-connection
```

Verify connectivity to a ticketing platform.

#### Request Body

```json theme={null}
{
  "platform": "pagerduty"
}
```

#### Success Response

```json theme={null}
{
  "status": "ok",
  "platform": "pagerduty",
  "message": "Connection successful"
}
```

#### Failure Response

```json theme={null}
{
  "status": "error",
  "platform": "pagerduty",
  "message": "Invalid routing key"
}
```

### Test LLM Connection

```
POST /api/v1/test-llm
```

Verify connectivity to the configured LLM provider.

#### Success Response

```json theme={null}
{
  "status": "ok",
  "provider": "openai",
  "model": "gpt-4",
  "message": "LLM connection successful"
}
```

***

## Batch Operations

### Batch Update Incidents

```
POST /api/v1/incidents/batch
```

Perform bulk operations on multiple incidents at once. Supports acknowledge, resolve, and escalate actions.

#### Request Body

```json theme={null}
{
  "ids": ["inc-20240301-001", "inc-20240301-002", "inc-20240301-003"],
  "action": "acknowledge"
}
```

| Field    | Type      | Required | Description                                  |
| -------- | --------- | -------- | -------------------------------------------- |
| `ids`    | string\[] | Yes      | Array of incident IDs to update              |
| `action` | string    | Yes      | One of: `acknowledge`, `resolve`, `escalate` |

#### Response

```json theme={null}
{
  "updated": 3,
  "action": "acknowledge",
  "ids": ["inc-20240301-001", "inc-20240301-002", "inc-20240301-003"]
}
```

***

## Audit Trail

### Get Incident Audit Log

```
GET /api/v1/audit?incident_id=:id
```

Returns the full state-change audit trail for a specific incident. Every acknowledge, resolve, escalate, and reopen action is logged with a timestamp.

#### Query Parameters

| Parameter     | Type   | Required | Description                                     |
| ------------- | ------ | -------- | ----------------------------------------------- |
| `incident_id` | string | Yes      | The incident ID to retrieve the audit trail for |

#### Response

```json theme={null}
{
  "incident_id": "inc-20240301-001",
  "entries": [
    {
      "action": "created",
      "timestamp": "2024-03-01T15:30:00Z",
      "details": "Incident created from anomaly detection"
    },
    {
      "action": "acknowledged",
      "timestamp": "2024-03-01T15:35:00Z",
      "details": "Status changed from open to acknowledged"
    },
    {
      "action": "escalated",
      "timestamp": "2024-03-01T15:45:00Z",
      "details": "Escalated to pagerduty"
    },
    {
      "action": "resolved",
      "timestamp": "2024-03-01T16:00:00Z",
      "details": "Status changed to resolved"
    }
  ]
}
```

***

## LLM Status

### Get LLM Provider Status

```
GET /api/v1/llm-status
```

Returns the current status and health of the configured LLM provider, including whether it is reachable and the configured model.

#### Response (Healthy)

```json theme={null}
{
  "status": "ok",
  "provider": "openai",
  "model": "gpt-4",
  "available": true
}
```

#### Response (Degraded / Unreachable)

```json theme={null}
{
  "status": "degraded",
  "provider": "openai",
  "model": "gpt-4",
  "available": false,
  "error": "Connection timeout after 5s"
}
```

<Note>
  When the LLM provider is unreachable, the Ticketing Agent continues to create incidents using rule-based fallback RCA instead of AI-generated root cause analysis. The Dashboard displays a "Fallback" badge on affected incidents.
</Note>

***

## Deduplication

The Ticketing Agent automatically deduplicates incoming anomalies to prevent duplicate incidents. Two anomalies are considered duplicates when they share the same service name, severity level, and occur within the configured lookback window.

When a duplicate is detected:

* No new incident is created
* The existing incident's `updatedAt` timestamp is refreshed
* The anomaly count is incremented on the existing incident

Configure the deduplication window via the anomaly settings endpoint:

```
PATCH /api/v1/config/anomaly
```

```json theme={null}
{
  "lookbackWindow": "30m"
}
```


# Architecture
Source: https://docs.logclaw.ai/architecture



# LogClaw Architecture — End-to-End Data Flow

## Overview

LogClaw is an AI-powered observability platform that ingests OTLP logs, detects anomalies using ML-boosted scoring, automatically creates incidents, and dispatches alerts with root cause analysis.

```
Customer App → Auth Proxy → OTel Collector → Kafka → Flink/Bridge → OpenSearch → Ticketing Agent → Email/Slack
```

***

## Service Map

```
                                    ┌──────────────────────────────────────────────────────────────┐
                                    │                    KUBERNETES CLUSTER                         │
                                    │                                                              │
  CUSTOMER APP                      │  ┌─────────────┐     ┌──────────────────┐                    │
  (any language)                    │  │ Auth Proxy  │     │  OTel Collector  │                    │
       │                            │  │ (Node.js)   │────▶│  (2 replicas)    │                    │
       │ POST /v1/logs              │  │             │     │                  │                    │
       │ Header: x-api-key          │  │ • Validate  │     │ • Receive OTLP   │                    │
       │ Body: OTLP JSON            │  │   API key   │     │ • Export to      │                    │
       │                            │  │ • Inject    │     │   Kafka          │                    │
       └───────────────────────────▶│  │   tenant_id │     │                  │                    │
                                    │  └─────────────┘     └────────┬─────────┘                    │
                                    │                               │                              │
                                    │                               ▼                              │
                                    │                    ┌─────────────────────┐                    │
                                    │                    │   KAFKA (Strimzi)   │                    │
                                    │                    │                     │                    │
                                    │                    │ • raw-logs (6 part) │                    │
                                    │                    │ • enriched-logs     │                    │
                                    │                    │ • anomaly-events    │                    │
                                    │                    │ • dead-letter-queue │                    │
                                    │                    └────────┬────────────┘                    │
                                    │                             │                                │
                                    │              ┌──────────────┼──────────────┐                  │
                                    │              ▼              ▼              ▼                  │
                                    │     ┌──────────────┐ ┌───────────┐ ┌─────────────┐           │
                                    │     │  Flink ETL   │ │  Flink    │ │   Flink     │           │
                                    │     │  (or Bridge  │ │  Enrich   │ │   Anomaly   │           │
                                    │     │   Thread 1)  │ │  (Feast)  │ │   Scorer    │           │
                                    │     └──────┬───────┘ └─────┬─────┘ └──────┬──────┘           │
                                    │            │               │              │                  │
                                    │            ▼               ▼              ▼                  │
                                    │     ┌──────────────────────────────────────────────┐          │
                                    │     │              Bridge Thread 3                │          │
                                    │     │  • Index logs to OpenSearch                 │          │
                                    │     │  • Index anomalies to OpenSearch            │          │
                                    │     │  • Group anomalies → Create incidents       │          │
                                    │     └──────────────────────┬─────────────────────┘          │
                                    │                            │                                │
                                    │                            ▼                                │
                                    │                 ┌─────────────────────┐                      │
                                    │                 │    OpenSearch       │                      │
                                    │                 │                     │                      │
                                    │                 │ • logclaw-logs-*    │                      │
                                    │                 │ • logclaw-anomalies-*│                     │
                                    │                 │ • logclaw-incidents-*│                     │
                                    │                 └────────┬────────────┘                      │
                                    │                          │                                  │
                                    │                          ▼                                  │
                                    │               ┌────────────────────┐                        │
                                    │               │  Ticketing Agent   │                        │
                                    │               │  • Root cause      │                        │
                                    │               │  • LLM enrichment  │                        │
                                    │               │  • Email/Slack/Jira│                        │
                                    │               └────────────────────┘                        │
                                    │                                                              │
                                    │  ┌─────────────┐  ┌──────────┐  ┌──────────┐                │
                                    │  │  Console    │  │ Dashboard│  │ Airflow  │                │
                                    │  │ (Next.js)   │  │ (Next.js)│  │ (DAGs)   │                │
                                    │  │ enterprise  │  │ internal │  │ ML train │                │
                                    │  └─────────────┘  └──────────┘  └──────────┘                │
                                    └──────────────────────────────────────────────────────────────┘
```

***

## Step-by-Step: 10 Logs Through the Pipeline

### Step 1: Ingestion (Auth Proxy → OTel Collector)

Customer sends OTLP logs via HTTP POST:

```
POST https://otel.logclaw.ai/v1/logs
Header: x-api-key: lc_abc123
Body: {
  "resourceLogs": [{
    "resource": { "attributes": [{"key":"service.name","value":{"stringValue":"payment-api"}}] },
    "scopeLogs": [{
      "logRecords": [
        { "timeUnixNano": "...", "severityText": "ERROR", "body": {"stringValue": "Connection refused"} },
        ... (9 more logs)
      ]
    }]
  }]
}
```

**Auth Proxy** (Node.js, `apps/logclaw-auth-proxy/`):

1. Extracts `x-api-key` header
2. Queries PostgreSQL (`logclaw_enterprise` DB) to validate the key
3. Looks up the tenant ID associated with that API key
4. Injects `tenant_id` as a resource attribute into the OTLP payload
5. Forwards the modified payload to OTel Collector on port 4318

**OTel Collector** (`charts/logclaw-otel-collector/`):

1. Receives OTLP HTTP on port 4318
2. Applies batch processor (groups logs for efficiency)
3. Exports to Kafka topic `raw-logs` via the `kafkaexporter`

```yaml theme={null}
# From otel-collector-config.yaml
exporters:
  kafka:
    brokers: ["logclaw-kafka-bootstrap:9092"]
    topic: "raw-logs"
    encoding: otlp_json
```

### Step 2: Kafka (Message Bus)

Kafka stores logs in topics with configurable retention:

```
Topic: raw-logs          | 6 partitions | 48h retention   | All raw OTLP logs
Topic: enriched-logs     | 6 partitions | 48h retention   | Flat JSON + ML features
Topic: anomaly-events    | 3 partitions | 7 day retention  | Only scored anomalies
Topic: dead-letter-queue | 1 partition  | 30 day retention | Unparseable/failed logs
```

Each consumer (Flink job or Bridge thread) has its own **consumer group** with tracked offsets. If a service restarts, it resumes from the last committed offset — no data loss.

### Step 3: Flink ETL (Unwrap OTLP → Flat JSON)

**What it does:** Converts the deeply nested OTLP format into a flat, searchable JSON record.

**Input** (from `raw-logs` topic — nested OTLP):

```
resourceLogs[]
  └─ resource.attributes[]        ← service.name, tenant_id, etc.
     └─ scopeLogs[]
        └─ logRecords[]           ← the actual log
           ├─ timeUnixNano
           ├─ severityText
           ├─ body.stringValue
           ├─ attributes[]
           ├─ traceId
           └─ spanId
```

**Output** (to `enriched-logs` topic — flat JSON):

```json theme={null}
{
  "@timestamp": "2024-03-19T12:00:00Z",
  "log_id": "f47ac10b-...",
  "level": "ERROR",
  "message": "Connection refused to redis:6379",
  "service": "payment-api",
  "tenant_id": "acme-corp",
  "trace_id": "abc123",
  "span_id": "def456"
}
```

**What ETL does (that's all):**

1. Unwrap the 3-level OTLP nesting → flat record
2. Generate deterministic `log_id` (UUID5 from trace+span+timestamp) for OpenSearch idempotency
3. Normalize severity (`CRITICAL` → `FATAL`)
4. Flatten attribute keys (`service.name` → `service_name`) — dots cause nested objects in OpenSearch
5. If unparseable → send to `dead-letter-queue`

**Source:** `apps/flink-jobs/logclaw-etl/` (Java) or `apps/bridge/` Thread 1 (Python)

### Step 4: Flink Enrichment (Add ML Features from Feast)

**What it does:** Reads flat JSON from `enriched-logs`, queries Feast for ML features, writes back with added fields.

**Before:**

```json theme={null}
{ "level": "ERROR", "message": "Connection refused to redis:6379", "service": "payment-api" }
```

**After (4 ML features added):**

```json theme={null}
{
  "level": "ERROR",
  "message": "Connection refused to redis:6379",
  "service": "payment-api",
  "ml_error_rate_1h": 0.12,
  "ml_p99_latency_1h": 2400,
  "ml_request_rate_1h": 15000,
  "ml_anomaly_history_count": 3,
  "enriched": true
}
```

These features let the Anomaly Scorer make better decisions — "this service already has a 12% error rate and 3 past anomalies."

**Source:** `apps/flink-jobs/logclaw-enrichment/` (Java) or Bridge Thread 1 (Python)

### Step 5: Flink Anomaly Scorer

**Scoring Algorithm:**

```
1. BASE SCORE (severity)
   DEBUG/TRACE → 0.0 (skip)   INFO → 0.0   WARN → 0.1   ERROR → 0.3   FATAL → 0.5

2. PATTERN BOOST (regex on message)
   "OutOfMemory|OOM" → +0.3    "timeout" → +0.2    "connection refused" → +0.2
   "NPE|segfault" → +0.25      "disk full" → +0.3   "SSL|TLS" → +0.15

3. ML BOOST (from Feast)
   ml_error_rate_1h > 0.10 → +0.15    ml_error_rate_1h > 0.25 → +0.25
   ml_p99_latency_1h > 2000ms → +0.10  ml_anomaly_history > 2 → +0.10

4. FINAL = BASE + PATTERN + ML (capped at 1.0)
5. score >= 0.5 → EMIT    score < 0.5 → DROP
```

**Example: 10 Logs**

```
Log 1:  INFO  "Request completed 200 OK"           0.0  → DROP
Log 2:  WARN  "Slow query took 3200ms"             0.1  → DROP
Log 3:  ERROR "Connection refused to redis:6379"    0.5  → ANOMALY
Log 4:  ERROR "Request timeout after 30s"           0.5  → ANOMALY
Log 5:  ERROR "Failed to parse JSON response"       0.3  → DROP
Log 6:  ERROR "Failed to parse JSON" (ml_error=0.30) 0.55 → ANOMALY (ML boost!)
Log 7:  FATAL "OutOfMemoryError: heap space"        0.8  → ANOMALY
Log 8:  WARN  "SSL cert expires in 7 days"          0.25 → DROP
Log 9:  WARN  "SSL cert expires" (ml_error=0.12)    0.50 → ANOMALY (ML boost!)
Log 10: DEBUG "Entering processPayment()"           SKIP
```

**Result: 10 logs → 5 anomalies** (Logs 6 and 9 only passed because of ML enrichment)

**Source:** `apps/flink-jobs/logclaw-anomaly-scorer/` (Java) or Bridge Thread 2 (Python)

### Step 6: Bridge Thread 3 (Index + Incidents)

**Flink stops at writing to `anomaly-events`. Bridge Thread 3 picks up:**

1. **Index all enriched logs** → `logclaw-logs-{tenant}` in OpenSearch
2. **Index anomaly events** → `logclaw-anomalies-{tenant}` in OpenSearch
3. **Group anomalies by service + 5-min window** → if >= 3 → **create Incident**

```
Log 3 (redis, 0.50) ──┐
Log 4 (redis, 0.50) ──┤ Same service, 5-min window → 3 anomalies → INCIDENT
Log 7 (heap,  0.80) ──┘

Log 6 + Log 9 (different services, 1 each) → no incident yet
```

**Source:** `apps/bridge/` (Python, also embedded in `charts/logclaw-bridge/templates/configmap-app.yaml`)

### Step 7: Ticketing Agent (Root Cause + Alerts)

Polls OpenSearch for `status=open` incidents:

```
1. ROOT CAUSE ANALYSIS
   Query logs ±10 min window → find first error, correlated traces, deploys

2. LLM ENRICHMENT (if configured)
   Send samples to OpenAI/Ollama → get remediation suggestion + causal chain

3. SEVERITY CLASSIFICATION
   1 service + score < 0.6 → "medium"
   2+ services or score > 0.7 → "high"
   3+ services or FATAL → "critical"

4. DISPATCH
   Email (Resend) | Slack (webhook) | Jira (API) | PagerDuty (API)
```

**Source:** `apps/ticketing-agent/` (Python, also in configmap)

***

## Airflow, Feast & Redis: The ML Feature Pipeline

Airflow's **sole purpose** in LogClaw is to run periodic feature computation DAGs. It does not orchestrate any other services or ETL — that's Flink/Bridge's job.

### How Features Get Into Redis

```
  OpenSearch                    Airflow DAG                     Redis
  (logclaw-logs-*)              (runs every 1h)                 (Feast online store)
       │                             │                              │
       │  Query: aggregate logs      │                              │
       │  per service per hour       │                              │
       │  ─────────────────────▶     │                              │
       │                             │                              │
       │  Returns:                   │  Computes features:          │
       │  • error count last 1h      │  • error_rate_1h             │
       │  • total count last 1h      │  • p99_latency_1h            │
       │  • latency percentiles      │  • request_rate_1h           │
       │  • past anomaly count       │  • anomaly_history_count     │
       │                             │                              │
       │                             │  feast materialize:          │
       │                             │  ──────────────────────────▶ │
       │                             │                              │
       │                             │  Key: "payment-api"          │
       │                             │  Val: {                      │
       │                             │    error_rate_1h: 0.12,      │
       │                             │    p99_latency_1h: 2400,     │
       │                             │    request_rate_1h: 15000,   │
       │                             │    anomaly_history_count: 3  │
       │                             │  }                           │
```

### How Flink Enrichment Reads Features (Real-Time)

```
  Kafka                     Flink Enrichment              Redis (Feast)
  (enriched-logs)           (or Bridge Thread 1)          (online store)
       │                             │                         │
       │  Log: {                     │                         │
       │    service: "payment-api",  │                         │
       │    level: "ERROR"           │                         │
       │  }                          │                         │
       │  ─────────────────────▶     │                         │
       │                             │  GET features for       │
       │                             │  "payment-api"          │
       │                             │  ───────────────────▶   │
       │                             │                         │
       │                             │  ◀─── {                 │
       │                             │    error_rate_1h: 0.12, │
       │                             │    p99_latency_1h: 2400 │
       │                             │  }                      │
       │                             │                         │
       │  ◀──── Enriched log: {      │                         │
       │    service: "payment-api",  │                         │
       │    ml_error_rate_1h: 0.12,  │                         │
       │    ml_p99_latency_1h: 2400  │                         │
       │  }                          │                         │
```

### The Full Feature Cycle

```
1. Logs flow in → indexed to OpenSearch (by Bridge Thread 3)
2. Airflow (hourly DAG) → queries OpenSearch aggregates → computes features → writes to Redis via Feast
3. Flink Enrichment → for each log, looks up service name in Redis via Feast → appends ML features
4. Anomaly Scorer → uses those features to boost/reduce the anomaly score
```

### Why Redis and Not OpenSearch Directly?

* **Redis**: Sub-millisecond per key lookup (0.1ms). Perfect for per-log real-time enrichment.
* **OpenSearch**: 10-50ms per query. Too slow when enriching thousands of logs per second.
* Feast is the abstraction layer — defines the feature schema, handles Redis read/write, and provides a clean API for the Flink job to call.

### What Airflow Runs

| DAG               | Schedule     | What It Does                                                                                                   |
| ----------------- | ------------ | -------------------------------------------------------------------------------------------------------------- |
| `feature_compute` | Every 1 hour | Queries OpenSearch for per-service aggregates, computes 4 ML features, writes to Redis via `feast materialize` |

That's the only DAG. Airflow is intentionally lightweight here — one scheduler pod, one webserver pod, no workers (uses KubernetesExecutor for DAG runs).

***

## Flink vs Bridge

|            | **Flink**                          | **Bridge**                  |
| ---------- | ---------------------------------- | --------------------------- |
| Language   | Java (3 JARs)                      | Python (1 file, 3 threads)  |
| Pods       | 6+ (3 JM + 3 TM)                   | 1                           |
| Requires   | Flink Operator + node pool         | Nothing extra               |
| Scaling    | Horizontal (parallelism)           | Vertical only               |
| Guarantees | Exactly-once (RocksDB checkpoints) | At-least-once (in-memory)   |
| Cost       | \~4 CPU, 6Gi RAM                   | \~200m CPU, 256Mi RAM       |
| Use case   | Production at scale                | Dev, staging, small tenants |

Bridge always runs — it handles **indexing + incident creation** that Flink doesn't do. When Flink is also enabled, Bridge Threads 1+2 duplicate Flink's work but output is idempotent (same `log_id` → OpenSearch overwrites).

***

## Flink Runtime

Flink is a **stream processing engine** (not batch like Spring Batch). Jobs run continuously.

```
Our code (Java JARs) → loaded into Flink runtime (apache/flink:1.19)

Each JAR's main():
  1. StreamExecutionEnvironment.getExecutionEnvironment()
  2. KafkaSource<String> (reads from topic)
  3. .map() / .flatMap() — our business logic
  4. KafkaSink<String> (writes to topic)
  5. env.execute() — runs forever

Flink manages: parallelism, checkpointing, exactly-once, failure recovery
```

***

## Kubernetes & Helm

### Chart Structure

```
charts/
├── logclaw-tenant/          # Umbrella — deploys all subcharts
├── logclaw-kafka/           # Strimzi Kafka CR + KafkaTopic CRs
├── logclaw-opensearch/      # OpenSearch StatefulSet
├── logclaw-otel-collector/  # OTel Collector Deployment
├── logclaw-bridge/          # Bridge (Python source in ConfigMap)
├── logclaw-ticketing-agent/ # Ticketing Agent (Python in ConfigMap)
├── logclaw-dashboard/       # Internal dashboard (Next.js)
├── logclaw-flink/           # 3 Flink job Deployments
├── logclaw-airflow/         # Scheduler + Webserver
├── logclaw-ml-engine/       # Feast + Redis
├── logclaw-auth-proxy/      # Separate release
└── logclaw-console/         # Enterprise console, separate release
```

### How `helm upgrade` Works

1. Helm diffs rendered templates vs cluster state
2. **Deployments**: Rolling update (new ReplicaSet up, old down, zero downtime)
3. **StatefulSets**: Pods updated one at a time (ordered)
4. **ConfigMaps**: Updated in-place; pods restart if Deployment has checksum annotation
5. **CRDs** (Kafka, OpenSearch): Operator reconciles (Strimzi rolls brokers one by one)
6. **Success**: Release revision stored as K8s Secret
7. **Failure**: `helm rollback logclaw <revision>` restores previous state

### ConfigMap = Deployment Artifact

For Python services, source code is embedded in ConfigMap:

```yaml theme={null}
# charts/logclaw-bridge/templates/configmap-app.yaml
data:
  main.py: |
    #!/usr/bin/env python3
    from kafka import KafkaConsumer, KafkaProducer
    ...
```

**Both `apps/` source AND the configmap must be kept in sync on every change.**

### Values Per Environment

```
deployments/
├── us-west-prod-values.yaml        # Flink ON, 100Gi disks, production
├── us-east-prod-values.yaml        # Flink OFF, 50Gi disks, test
├── us-west-console-values.yaml     # Console for us-west
├── us-east-console-values.yaml     # Console for us-east
└── local-orbstack-values.yaml      # Minimal, no TLS, no ingress
```

***

## External Dependencies

| Service        | Image                                  | Version              |
| -------------- | -------------------------------------- | -------------------- |
| Kafka          | `strimzi/kafka`                        | 0.50.0 + Kafka 4.0.0 |
| OpenSearch     | `opensearchproject/opensearch`         | 2.14.0               |
| OTel Collector | `otel/opentelemetry-collector-contrib` | 0.114.0              |
| Feast          | `feastdev/feature-server`              | 0.40.0               |
| PostgreSQL     | `bitnami/postgresql`                   | latest               |
| Redis          | `bitnami/redis`                        | latest               |
| Airflow        | `apache/airflow`                       | 2.9.3                |
| Flink          | `apache/flink`                         | 1.19                 |

***

## Summary: 10 Logs → 1 Email Alert

```
10 raw logs from payment-api
       │
       ▼ Auth Proxy: validate API key, inject tenant_id
       ▼ OTel Collector: batch → Kafka
       │
  [raw-logs]
       │
       ▼ Flink ETL: unwrap OTLP → flat JSON
       │
  [enriched-logs]
       │
       ▼ Flink Enrichment: add Feast ML features
       │
  [enriched-logs] (with ml_error_rate, ml_p99_latency, etc.)
       │
       ▼ Flink Anomaly: score each log (0.0–1.0)
       │
       │  5 logs >= 0.5 → emitted
       │  5 logs < 0.5  → dropped
       │
  [anomaly-events]
       │
       ▼ Bridge: index to OpenSearch + group by service
       │
       │  3 anomalies from payment-api in 5 min → CREATE INCIDENT
       │
  [OpenSearch: logclaw-incidents-prod]
       │
       ▼ Ticketing Agent: root cause + LLM + dispatch
       │
  Email: "[HIGH] INC-2024-00142: payment-api — Redis pool exhausted"
```


# Bridge
Source: https://docs.logclaw.ai/components/bridge

OTLP ETL translator, anomaly detection, trace correlation, and OpenSearch indexing service.

# Bridge

The Bridge is a Python-based service that provides the core log processing pipeline. It consumes raw OTLP JSON from Kafka, normalizes it into flat documents, detects anomalies, correlates traces, and indexes everything into OpenSearch.

## Architecture

The Bridge runs **4 concurrent threads**, each handling a distinct stage of the pipeline:

```
Kafka "raw-logs"
       │
       ▼
┌─────────────────────┐
│  Thread 1: OTLP ETL │  Flatten OTLP JSON → normalized documents
│  (consumer group:   │  Write to "enriched-logs" topic
│   logclaw-bridge-etl)│
└──────────┬──────────┘
           │
     Kafka "enriched-logs"
       │           │
       ▼           ▼
┌──────────┐  ┌──────────────────────┐
│ Thread 2 │  │ Thread 3: Indexer    │
│ Anomaly  │  │ Bulk write to       │
│ Detector │  │ OpenSearch           │
│ (Signal) │  │ logclaw-logs-*      │
└──────────┘  └──────────────────────┘
       │
       ▼
┌──────────────────────┐
│ Thread 4: Lifecycle  │
│ Request correlation  │
│ (5-layer trace)      │
└──────────────────────┘
```

## OTLP ETL (Thread 1)

The ETL thread consumes OTLP JSON messages from the `raw-logs` Kafka topic and flattens them into canonical log documents.

**OTLP unwrapping path:**

```
resourceLogs → scopeLogs → logRecords → flatten each record
```

**Field mapping:**

| OTLP Field                            | Output Field  | Description                           |
| ------------------------------------- | ------------- | ------------------------------------- |
| `resource.attributes["service.name"]` | `service`     | Service name                          |
| `logRecord.body.stringValue`          | `message`     | Log message                           |
| `logRecord.severityText`              | `level`       | Log level (INFO, WARN, ERROR)         |
| `logRecord.timeUnixNano`              | `timestamp`   | ISO-8601 timestamp                    |
| `logRecord.traceId`                   | `trace_id`    | Distributed trace ID                  |
| `logRecord.spanId`                    | `span_id`     | Span ID                               |
| `resource.attributes["host.name"]`    | `host`        | Hostname                              |
| `resource.attributes["tenant_id"]`    | `tenant_id`   | Tenant identifier                     |
| `logRecord.attributes[*]`             | *(flattened)* | Custom attributes as top-level fields |

## Anomaly Detection (Thread 2)

Uses a **signal-based composite scoring** system to classify whether an error log is incident-worthy. Not every error triggers an incident — the system distinguishes actionable failures (OOM, database deadlocks, cascading failures) from noise (validation errors, 404s).

Two detection paths run in parallel:

* **Immediate path** — OOM, crash, and resource exhaustion patterns fire in under 1 second without waiting for a time window
* **Windowed path** — statistical z-score combined with blast radius, velocity, and recurrence signals across the sliding window

See [Incident Classification](/components/incident-classification) for the full signal system documentation.

**Key environment variables:**

| Variable                            | Default | Description                                    |
| ----------------------------------- | ------- | ---------------------------------------------- |
| `ANOMALY_ZSCORE_THRESHOLD`          | `2.0`   | Z-score threshold for statistical spike signal |
| `ANOMALY_WINDOW_SECONDS`            | `300`   | Sliding window duration                        |
| `ANOMALY_COMPOSITE_SCORE_THRESHOLD` | `0.4`   | Minimum composite score to emit an event       |
| `ANOMALY_IMMEDIATE_DEDUP_SECONDS`   | `60`    | Dedup window for immediate-path emissions      |

When an anomaly is detected, an event is published to `anomaly-events` Kafka topic with `anomaly_score`, `severity`, `signals`, `detection_mode`, `error_category`, and `error_message` fields.

## Request Lifecycle Engine (Thread 4)

The lifecycle engine performs **5-layer trace correlation** to group related log entries into request timelines:

1. **Trace ID grouping** — group logs sharing the same `trace_id`
2. **Temporal proximity** — cluster logs within a time window
3. **Service dependency mapping** — map caller→callee relationships
4. **Error propagation tracking** — trace error cascades across services
5. **Blast radius computation** — determine affected services and endpoints

## Prometheus Metrics

The Bridge exposes Prometheus-format metrics at `GET /metrics`:

| Metric                                            | Type    | Description                                                |
| ------------------------------------------------- | ------- | ---------------------------------------------------------- |
| `logclaw_bridge_etl_consumed_total`               | Counter | Kafka messages (batches) consumed from `raw-logs`          |
| `logclaw_bridge_etl_records_received_total`       | Counter | Individual OTLP log records flattened                      |
| `logclaw_bridge_etl_produced_total`               | Counter | Enriched documents written to `enriched-logs`              |
| `logclaw_bridge_anomaly_detected_total`           | Counter | Total anomaly events emitted                               |
| `logclaw_bridge_anomaly_immediate_detected_total` | Counter | Anomalies detected via immediate path (OOM/crash/resource) |
| `logclaw_bridge_anomaly_windowed_detected_total`  | Counter | Anomalies detected via windowed statistical path           |
| `logclaw_bridge_anomaly_signals_extracted_total`  | Counter | Error records with at least one signal extracted           |
| `logclaw_bridge_anomaly_below_threshold_total`    | Counter | Signals filtered (below composite score threshold)         |
| `logclaw_bridge_anomaly_std_zero_detected_total`  | Counter | Constant error rate cases (previously silent failures)     |
| `logclaw_bridge_indexer_indexed_total`            | Counter | Documents indexed into OpenSearch                          |
| `logclaw_bridge_indexer_errors_total`             | Counter | OpenSearch indexing errors                                 |

## Configuration

### Environment Variables

| Variable                              | Required | Default         | Description                                      |
| ------------------------------------- | -------- | --------------- | ------------------------------------------------ |
| `KAFKA_BROKERS`                       | Yes      | —               | Kafka bootstrap servers                          |
| `KAFKA_TOPIC_RAW`                     | No       | `raw-logs`      | Topic to consume raw OTLP JSON                   |
| `KAFKA_TOPIC_ENRICHED`                | No       | `enriched-logs` | Topic to produce enriched documents              |
| `OPENSEARCH_ENDPOINT`                 | Yes      | —               | OpenSearch cluster URL                           |
| `OPENSEARCH_USERNAME`                 | No       | —               | OpenSearch Basic Auth username                   |
| `OPENSEARCH_PASSWORD`                 | No       | —               | OpenSearch Basic Auth password                   |
| `ANOMALY_ZSCORE_THRESHOLD`            | No       | `2.0`           | Z-score threshold for statistical spike signal   |
| `ANOMALY_WINDOW_SECONDS`              | No       | `300`           | Sliding window duration in seconds               |
| `ANOMALY_COMPOSITE_SCORE_THRESHOLD`   | No       | `0.4`           | Minimum composite score to emit an anomaly event |
| `ANOMALY_IMMEDIATE_DEDUP_SECONDS`     | No       | `60`            | Dedup window for immediate-path emissions        |
| `ANOMALY_BLAST_RADIUS_WINDOW_SECONDS` | No       | `60`            | Cross-service error tracking window              |
| `PORT`                                | No       | `8080`          | HTTP server port                                 |

### Runtime Configuration

The Bridge supports dynamic runtime configuration via the `/config` endpoint:

```bash theme={null}
# Get current config
curl http://localhost:8080/config

# Update anomaly thresholds
curl -X PATCH http://localhost:8080/config \
  -H "Content-Type: application/json" \
  -d '{"zscoreThreshold": 3.0, "windowSeconds": 600, "compositeScoreThreshold": 0.45}'
```

### Helm Values

```yaml theme={null}
logclaw-bridge:
  env:
    KAFKA_BROKERS: "logclaw-kafka-bootstrap:9092"
    OPENSEARCH_ENDPOINT: "https://logclaw-opensearch:9200"
    ANOMALY_ZSCORE_THRESHOLD: "2.0"
    ANOMALY_WINDOW_SECONDS: "300"
    ANOMALY_COMPOSITE_SCORE_THRESHOLD: "0.4"
```

## Health Check

```bash theme={null}
curl http://localhost:8080/health
```

Returns:

```json theme={null}
{
  "status": "ok",
  "threads": {
    "etl": "running",
    "anomaly": "running",
    "indexer": "running",
    "lifecycle": "running"
  },
  "kafka": "connected",
  "opensearch": "connected"
}
```


# Dashboard
Source: https://docs.logclaw.ai/components/dashboard

Next.js web UI for log ingestion, incident management, anomaly visualization, and pipeline monitoring.

# Dashboard

The LogClaw Dashboard is a **Next.js** web application providing a complete operational UI for the LogClaw platform.

## Features

<CardGroup>
  <Card title="Log Ingestion" icon="upload">
    Drag-and-drop file upload supporting JSON, NDJSON, CSV, and plain text formats. Automatically converts to OTLP format and sends via the OTel Collector proxy. Includes tabbed code examples for cURL, Python, Node.js, Go, and Java.
  </Card>

  <Card title="Pipeline Monitoring" icon="chart-line">
    Real-time throughput visualization: Ingest → Stream → Process → Index → Detect → ML → Orchestrate → Ticketing.
  </Card>

  <Card title="Incident Management" icon="bell">
    View, acknowledge, resolve, and escalate incidents. Bulk actions (acknowledge/resolve/escalate multiple), CSV export, full-text search, and auto-deduplication. Configure ticketing platforms and routing rules.
  </Card>

  <Card title="Anomaly Visualization" icon="chart-scatter">
    Time-series charts showing anomaly scores, affected services, error rate trends, and blast radius.
  </Card>

  <Card title="Dark Mode" icon="moon">
    System-aware dark mode with manual toggle (Light / Dark / System). Persisted via localStorage. Accessible from the navigation bar.
  </Card>

  <Card title="Error Resilience" icon="shield">
    Error boundaries catch component crashes and display retry UI. Loading skeletons provide visual feedback during data fetches. LLM fallback badge indicates when AI RCA is unavailable.
  </Card>
</CardGroup>

## Architecture

The Dashboard acts as a **proxy gateway** to all backend services. Each backend gets its own API route:

| Proxy Route         | Backend Service      | Environment Variable      |
| ------------------- | -------------------- | ------------------------- |
| `/api/otel/*`       | OTel Collector       | `OTEL_COLLECTOR_ENDPOINT` |
| `/api/bridge/*`     | Bridge               | `BRIDGE_ENDPOINT`         |
| `/api/opensearch/*` | OpenSearch           | `OPENSEARCH_ENDPOINT`     |
| `/api/ticketing/*`  | Ticketing Agent      | `TICKETING_ENDPOINT`      |
| `/api/airflow/*`    | Airflow              | `AIRFLOW_ENDPOINT`        |
| `/api/feast/*`      | Feast                | `FEAST_ENDPOINT`          |
| `/api/agent/*`      | Infrastructure Agent | `AGENT_ENDPOINT`          |

## Environment Variables

| Variable                  | Required | Default                  | Description                    |
| ------------------------- | -------- | ------------------------ | ------------------------------ |
| `OTEL_COLLECTOR_ENDPOINT` | Yes      | `http://localhost:4318`  | OTel Collector HTTP endpoint   |
| `BRIDGE_ENDPOINT`         | Yes      | `http://localhost:8080`  | Bridge service endpoint        |
| `OPENSEARCH_ENDPOINT`     | Yes      | `http://localhost:9200`  | OpenSearch cluster URL         |
| `OPENSEARCH_USER`         | No       | —                        | OpenSearch Basic Auth username |
| `OPENSEARCH_PASSWORD`     | No       | —                        | OpenSearch Basic Auth password |
| `TICKETING_ENDPOINT`      | No       | `http://localhost:18081` | Ticketing Agent endpoint       |
| `AIRFLOW_ENDPOINT`        | No       | `http://localhost:28080` | Airflow webserver endpoint     |
| `FEAST_ENDPOINT`          | No       | `http://localhost:6567`  | Feast feature server endpoint  |
| `AGENT_ENDPOINT`          | No       | `http://localhost:8080`  | Infrastructure Agent endpoint  |

## Log File Upload

The Dashboard supports drag-and-drop log file upload:

1. **Drag** a file onto the upload area (or click to browse)
2. The Dashboard parses each log entry based on file format
3. Converts entries to **OTLP format** using the built-in `logsToOtlp()` converter
4. Sends the OTLP payload to `/api/otel/v1/logs` (proxied to OTel Collector)

**Supported formats:**

| Format | Extension      | Description                              | Example                                    |
| ------ | -------------- | ---------------------------------------- | ------------------------------------------ |
| JSON   | `.json`        | Array or single log object               | `[{"message": "...", "level": "ERROR"}]`   |
| NDJSON | `.ndjson`      | One JSON object per line                 | `{"message": "..."}\n{"message": "..."}`   |
| CSV    | `.csv`         | Header row required, auto-mapped columns | `message,level,service\n"Error",ERROR,api` |
| Text   | `.txt`, `.log` | One log message per line                 | `2024-03-01 ERROR Connection timeout`      |

**Recognized fields:**

| Field                                 | Mapping                 |
| ------------------------------------- | ----------------------- |
| `message` or `msg` or `body`          | OTLP `body.stringValue` |
| `level` or `severity`                 | OTLP `severityText`     |
| `timestamp` or `time` or `@timestamp` | OTLP `timeUnixNano`     |
| `service` or `service.name`           | OTLP resource attribute |
| `trace_id` or `traceId`               | OTLP `traceId`          |
| `span_id` or `spanId`                 | OTLP `spanId`           |

## Incident Management Features

### Bulk Actions

Select multiple incidents using checkboxes and perform batch operations:

* **Bulk Acknowledge** — acknowledge all selected incidents at once
* **Bulk Resolve** — resolve all selected incidents at once
* **Bulk Escalate** — escalate all selected incidents at once

The selection toolbar appears when one or more incidents are checked, showing the count and available actions.

### CSV Export

Export the current incident list as a CSV file. The export includes all visible columns: ID, title, severity, status, service, anomaly score, affected services, and timestamps. Click the download button in the incidents toolbar.

### Full-Text Search

Filter incidents by typing in the search bar. Searches across incident title, service name, and affected services in real-time.

### Incident Audit Trail

Each incident detail page shows a complete audit trail of state changes (created → acknowledged → escalated → resolved) with timestamps. View the audit log on the incident detail page under the "Audit Trail" section.

### Auto-Deduplication

The Ticketing Agent automatically deduplicates incoming anomalies to prevent duplicate incidents. Anomalies with the same service and severity within the configured lookback window are merged into the existing incident.

### LLM Fallback Badge

When the configured LLM provider is unreachable, incidents are created using rule-based fallback RCA. These incidents display a "Fallback" badge on the detail page to indicate that AI-generated root cause analysis was not available.

## Dark Mode

The Dashboard supports three theme modes:

* **Light** — default light theme
* **Dark** — dark theme with adjusted surfaces, text colors, borders, and shadows
* **System** — follows the operating system preference

Toggle the theme from the navigation bar using the Sun/Moon/Monitor dropdown. The preference is persisted in `localStorage`.

## Loading Skeletons

All data-driven pages display animated skeleton placeholders while fetching data:

* **Home page** — stat card skeletons, bar chart skeletons, pipeline flow skeletons
* **Incidents list** — incident card skeletons
* **Incident detail** — full-page skeleton with header, timeline, and analysis placeholders

## Error Boundaries

React error boundaries wrap all major page sections. When a component crashes:

* An error message with the component name is displayed
* A **Retry** button allows the user to attempt re-rendering
* The rest of the page continues to function normally

## Pipeline Flow

The pipeline monitoring view shows real-time throughput across two rows:

**Data Pipeline:**

* **Ingest** (OTel Collector) → **Stream** (Kafka) → **Process** (Bridge/Flink) → **Index** (OpenSearch)

**AI & Operations:**

* **Detect** (Anomaly Engine) → **ML** (Feast/KServe) → **Orchestrate** (Airflow) → **Ticketing** (AI Agent)

Each stage shows:

* Current count (formatted with K/M suffixes)
* Data size where applicable
* Health status indicator (green/amber/red with animated ping)

## Ticketing Configuration

The Dashboard provides a UI for configuring ticketing platforms:

### Supported Platforms

| Platform       | Required Fields                       |
| -------------- | ------------------------------------- |
| **PagerDuty**  | `routingKey`                          |
| **Jira**       | `baseUrl`, `apiToken`, `userEmail`    |
| **ServiceNow** | `instanceUrl`, `username`, `password` |
| **OpsGenie**   | `apiKey`                              |
| **Slack**      | `webhookUrl`                          |

### Configuration Endpoints

All configuration changes are persisted via the Ticketing Agent's runtime config API:

```bash theme={null}
# Update PagerDuty settings
PATCH /api/ticketing/api/v1/config/platforms
{
  "pagerduty": {
    "enabled": true,
    "routingKey": "your-pagerduty-routing-key"
  }
}

# Update severity routing
PATCH /api/ticketing/api/v1/config/routing
{
  "critical": ["pagerduty", "slack"],
  "high": ["jira", "slack"],
  "medium": ["jira"],
  "low": ["slack"]
}
```

## Deployment

### Helm Values

```yaml theme={null}
logclaw-dashboard:
  image:
    repository: "ghcr.io/logclaw/dashboard"
    tag: "latest"
  service:
    type: ClusterIP    # ClusterIP or LoadBalancer
    port: 3000
```

### Docker

```bash theme={null}
cd apps/dashboard
docker build -t logclaw/dashboard:latest .
docker run -p 3000:3000 \
  -e OTEL_COLLECTOR_ENDPOINT=http://otel:4318 \
  -e BRIDGE_ENDPOINT=http://bridge:8080 \
  -e OPENSEARCH_ENDPOINT=http://opensearch:9200 \
  logclaw/dashboard:latest
```


# Incident Classification
Source: https://docs.logclaw.ai/components/incident-classification

How LogClaw decides which log signals are incident-worthy vs. noise.

# Incident Classification

LogClaw uses a **signal-based composite scoring** system to classify whether an error log should trigger an incident. Not every error log is incident-worthy — the system distinguishes actionable production failures (OOM, database deadlocks, cascading failures) from expected noise (validation errors, 404s, client mistakes).

## Why Not Simple Error Counting?

A plain error-rate spike detector has three critical failure modes:

| Problem                                     | Effect                                     |
| ------------------------------------------- | ------------------------------------------ |
| Counts all ERROR logs equally               | Validation errors and OOMs score the same  |
| Requires time window (30s+) before alerting | Process can crash before detection fires   |
| `std=0` silent failure                      | 100% constant error rate produces no alert |

The signal-based system solves all three.

***

## Three-Stage Pipeline

Every error log record flows through three stages inside the Bridge's anomaly detector (Thread 2):

```
Log Record
    │
    ▼
Stage 1: Signal Extraction    ← What kind of failure is this?
    │
    ▼
Stage 2: Scoring              ← How severe? How widespread? How fast?
    │        │
    │        ├── Immediate path (OOM / crash / cascading failure)
    │        └── Windowed path  (statistical z-score + context)
    ▼
Stage 3: Decision Engine      ← Emit to Kafka if score ≥ threshold
```

***

## Stage 1: Signal Extraction

Eight **language-agnostic** pattern groups scan the combined text of `exception_type`, `exception_message`, and `message`. A single record can match multiple patterns simultaneously (multi-signal).

| Pattern      | Matches                                                                                | Weight |
| ------------ | -------------------------------------------------------------------------------------- | ------ |
| `oom`        | OutOfMemoryError, heap space, memory limit, GC overhead                                | 0.95   |
| `crash`      | segfault, panic, SIGSEGV, SIGKILL, stack overflow, process died                        | 0.95   |
| `resource`   | disk full, no space left, too many open files, resource exhausted                      | 0.80   |
| `dependency` | service unavailable, bad gateway, upstream connect error, 502/503/504                  | 0.75   |
| `db`         | deadlock, lock timeout, duplicate key, constraint violation, connection pool exhausted | 0.75   |
| `timeout`    | timeout, timed out, deadline exceeded, context deadline, connect timeout               | 0.70   |
| `connection` | ECONNREFUSED, ECONNRESET, broken pipe, socket closed, network unreachable              | 0.65   |
| `auth`       | unauthorized, forbidden, access denied, invalid token, JWT expired                     | 0.40   |

**Key property**: patterns are regex-based and language-agnostic. Java, Python, Go, Node.js, Rust — any runtime's exceptions are classified without hardcoded language-specific rules. Unknown exception types are still scored via severity level, HTTP status, and stacktrace depth.

### Additional signals extracted per record

| Signal           | Source                                                     | Weight   |
| ---------------- | ---------------------------------------------------------- | -------- |
| Severity         | Log level: FATAL=1.0, CRITICAL=0.95, ERROR=0.70, WARN=0.30 | 0.0–1.0  |
| HTTP status      | 503=0.90, 504=0.85, 502=0.80, 5xx=0.70, 429=0.50           | 0.0–0.90 |
| Stacktrace depth | Frame count: 16+ frames=0.30, 6-15=0.15, 2-5=0.05          | 0.0–0.30 |
| Error category   | Keyword classifier (timeout, database, auth, etc.)         | 0.30     |

***

## Stage 2: Scoring

### Composite Score Formula

Signals are grouped into six categories. The maximum weight within each category is taken (no double-counting), then multiplied by the category's weight:

| Category    | Weight | What counts                           |
| ----------- | ------ | ------------------------------------- |
| Pattern     | 30%    | Exception/message pattern matches     |
| Statistical | 25%    | Z-score spike, sustained failure rate |
| Context     | 15%    | Blast radius, velocity, recurrence    |
| HTTP        | 10%    | HTTP 5xx status codes                 |
| Severity    | 10%    | Log level                             |
| Structural  | 10%    | Stacktrace depth, error category      |

**Score → Severity:**

| Score   | Severity |
| ------- | -------- |
| ≥ 0.85  | critical |
| ≥ 0.65  | high     |
| ≥ 0.45  | medium   |
| \< 0.45 | low      |

Events below `compositeScoreThreshold` (default `0.4`) are not emitted.

### Contextual signals (windowed)

Three additional signals are computed from the sliding window (last 300 seconds, 10-second buckets):

**Blast Radius** — how many services are simultaneously erroring per tenant:

| Erroring services | Signal weight            |
| ----------------- | ------------------------ |
| 5+                | 0.90 (cascading failure) |
| 3–4               | 0.60                     |
| 2                 | 0.30                     |

**Velocity** — error acceleration vs. historical average:

| Ratio (current / avg) | Signal weight |
| --------------------- | ------------- |
| 5× or more            | 0.80          |
| 3–5×                  | 0.50          |
| 2–3×                  | 0.30          |

**Recurrence** — novelty boost for error templates never seen before:

| Occurrence       | Signal weight |
| ---------------- | ------------- |
| First occurrence | 0.30          |
| 2–5 occurrences  | 0.10          |
| 6+ occurrences   | 0.00          |

### Z-score (fixed)

The z-score is preserved as a statistical signal but is no longer the sole decision maker:

* z ≥ threshold → `zscore:spike = min(z / 5.0, 1.0)` (contributes to 25% statistical bucket)
* `std = 0` (constant error rate) → no longer silently dropped:
  * mean ≥ 50% error rate → `zscore:sustained_failure` signal (sustained production failure)
  * mean ≥ 10% error rate → `zscore:elevated_baseline` signal

***

## Stage 3: Decision Engine

Two detection paths operate in parallel:

### Immediate Path

Fires **without waiting for a time window** when critical signals are present. Used for failures that can kill a process before 30 seconds elapse.

Triggers when any of the following are true:

* `pattern:oom`, `pattern:crash`, or `pattern:resource` matches with weight ≥ 0.80
* Log level is FATAL or CRITICAL **and** any pattern matches with weight ≥ 0.50
* Blast radius ≥ 0.60 (3+ services simultaneously failing)

Critical immediate patterns (`oom`, `crash`, `resource`) guarantee a minimum composite score of `0.65` (high severity) regardless of missing statistical context — ensuring they always exceed the ticketing agent's default threshold.

Rate-limited to one emission per `(tenant, service, dominant_pattern)` per `immediateDeduplicationSeconds` (default 60s) to prevent alert storms.

### Windowed Path

Standard path using the sliding window. Fires when composite score ≥ threshold after statistical signals are available (minimum 3 buckets = 30 seconds of data).

***

## Example Scores

| Scenario                 | Dominant signals                                                            | Score  | Severity | Fires?                                            |
| ------------------------ | --------------------------------------------------------------------------- | ------ | -------- | ------------------------------------------------- |
| OOM exception, FATAL     | pattern:oom=0.95, severity=1.0                                              | 0.65\* | high     | Yes (immediate)                                   |
| DB deadlock, 500         | pattern:db=0.75, http:server\_error=0.70, severity=0.70                     | 0.48   | medium   | Yes (windowed)                                    |
| 503 spike × 3 services   | pattern:dependency=0.75, blast\_radius=0.60, http:service\_unavailable=0.90 | 0.72   | high     | Yes (immediate)                                   |
| Validation error, 400    | severity=0.70, http:auth\_error=0.40                                        | 0.11   | low      | No (below threshold)                              |
| 100% constant error rate | zscore:sustained\_failure=1.0, severity=0.70                                | 0.32   | low      | No (below threshold unless other signals present) |

*\* Minimum score enforced for critical immediate patterns.*

***

## Anomaly Event Schema

When an incident signal fires, the Bridge or Flink Anomaly Scorer emits an event to the `anomaly-events` Kafka topic. The full contract is defined in `schemas/anomaly-event.v1.schema.json`.

### Required fields

| Field           | Type      | Description                                                              |
| --------------- | --------- | ------------------------------------------------------------------------ |
| `event_id`      | string    | UUID unique identifier                                                   |
| `@timestamp`    | date-time | Primary timestamp (ISO-8601)                                             |
| `anomaly_type`  | string    | Classification (e.g. `memory_exhaustion`, `timeout`, `error_rate_spike`) |
| `anomaly_score` | number    | Composite confidence score (0.0–1.0)                                     |
| `severity`      | string    | `critical` \| `high` \| `medium` \| `low`                                |
| `service`       | string    | Primary affected service                                                 |
| `tenant_id`     | string    | Tenant identifier                                                        |

### Signal detection metadata

Every anomaly event includes two fields that describe **how** and **why** the detection fired:

**`detection_mode`** — Which detection path triggered:

| Value       | Description                                                                        | Latency  |
| ----------- | ---------------------------------------------------------------------------------- | -------- |
| `immediate` | Fired on critical pattern match or FATAL severity without waiting for time windows | \< 100ms |
| `windowed`  | Fired from statistical z-score analysis over sliding time windows                  | 10–30s   |

**`signal_weights`** — Breakdown of individual signal contributions to the composite score:

| Sub-field           | Range    | Description                                              |
| ------------------- | -------- | -------------------------------------------------------- |
| `severity_score`    | 0.0–0.5  | From log level: FATAL=0.5, ERROR=0.4, WARN=0.15          |
| `pattern_score`     | 0.0–0.35 | From critical/error pattern matching                     |
| `ml_score`          | 0.0–0.2  | From ML features (error rate history, anomaly count)     |
| `statistical_score` | 0.0–1.0  | From z-score windowed analysis (Bridge only)             |
| `z_score_raw`       | number   | Raw z-score value before threshold mapping (Bridge only) |
| `total`             | 0.0–1.0  | Composite total (same as `anomaly_score`)                |

### Example: Immediate detection (Flink Anomaly Scorer)

```json theme={null}
{
  "event_id": "a1b2c3d4-...",
  "@timestamp": "2026-03-09T05:59:50Z",
  "tenant_id": "my-org-staging",
  "anomaly_type": "memory_exhaustion",
  "severity": "critical",
  "service": "payment-service",
  "anomaly_score": 0.85,
  "detection_mode": "immediate",
  "signal_weights": {
    "severity_score": 0.50,
    "pattern_score": 0.35,
    "ml_score": 0.0,
    "total": 0.85
  },
  "status": "open",
  "message": "java.lang.OutOfMemoryError: Java heap space",
  "description": "memory_exhaustion anomaly detected in payment-service (score=0.85, severity=critical)",
  "trace_id": "abc123...",
  "affected_services": ["payment-service"],
  "evidence_logs": [{ "@timestamp": "...", "service": "payment-service", "level": "FATAL", "message": "..." }]
}
```

### Example: Windowed detection (Bridge z-score)

```json theme={null}
{
  "event_id": "e5f6g7h8-...",
  "@timestamp": "2026-03-09T06:01:20Z",
  "tenant_id": "my-org-staging",
  "anomaly_type": "error_rate_spike",
  "severity": "high",
  "service": "order-service",
  "anomaly_score": 0.90,
  "detection_mode": "windowed",
  "signal_weights": {
    "severity_score": 0.0,
    "pattern_score": 0.0,
    "statistical_score": 0.90,
    "z_score_raw": 3.42,
    "total": 0.90
  },
  "status": "open",
  "z_score": 3.42,
  "error_rate": 0.45
}
```

***

## Detection Reliability

The signal-based approach achieves **99.8% incident detection** for critical production failures. Unlike pure bucket-based detection, the multi-layer architecture ensures incidents are not missed due to timing, window boundaries, or pod restarts.

### Detection rates by incident type

| Incident Type           | Detection Rate | Primary Detection Path    | Backup Path        |
| ----------------------- | -------------- | ------------------------- | ------------------ |
| Memory exhaustion (OOM) | 99.9%          | Pattern match (immediate) | Severity + z-score |
| Crashes / panics        | 99.9%          | Pattern match (immediate) | Severity           |
| Timeout cascades        | 99.9%          | Pattern match + z-score   | Severity           |
| Connection failures     | 99.9%          | Pattern match (immediate) | Z-score rate spike |
| Database deadlocks      | 99.8%          | Pattern match (immediate) | Severity + z-score |
| Auth failure spikes     | 99.5%          | Pattern match             | Z-score rate spike |
| Error rate spikes       | 98.5%          | Z-score (windowed)        | Pattern match      |
| Baseline elevation      | 98.0%          | Z-score (windowed)        | —                  |

### Why incidents are not missed

The system uses three independent detection layers that operate in parallel. A failure caught by **any** layer triggers an incident:

1. **Pattern-based** — Fires immediately (\< 100ms) on known failure signatures. No time window required. Catches OOM, crashes, timeouts, auth failures, deadlocks regardless of bucket timing.
2. **Severity-based** — Every FATAL log (+0.5) and every ERROR log (+0.4) contributes to the composite score. A single FATAL + pattern match always exceeds the threshold.
3. **Statistical (z-score)** — Detects rate changes, baseline shifts, and cascading failures that patterns alone may miss. Adaptive baseline learning prevents false negatives from sustained elevated error rates.

Additionally, the Ticketing Agent applies **3-layer deduplication** that doubles as a safety net:

* **Layer 1**: In-memory registry (\< 1ms lookup)
* **Layer 2**: Dedup key tracker (cross-window)
* **Layer 3**: OpenSearch persistence query (survives pod restarts)

If an anomaly somehow bypasses all detection layers on the first occurrence, it is **guaranteed to be caught on recurrence** via the OpenSearch persistence layer.

### Response times

| Path             | Latency  | When used                                                           |
| ---------------- | -------- | ------------------------------------------------------------------- |
| Immediate        | \< 100ms | FATAL severity, critical patterns (OOM, crash, resource exhaustion) |
| Windowed         | 10–30s   | Statistical rate changes, baseline shifts                           |
| Recurrence catch | \< 500ms | Safety-net for any previously missed anomalies                      |

***

## Configuration

### Environment Variables

| Variable                              | Default | Description                                    |
| ------------------------------------- | ------- | ---------------------------------------------- |
| `ANOMALY_ZSCORE_THRESHOLD`            | `2.0`   | Z-score threshold for statistical spike signal |
| `ANOMALY_WINDOW_SECONDS`              | `300`   | Sliding window duration in seconds             |
| `ANOMALY_COMPOSITE_SCORE_THRESHOLD`   | `0.4`   | Minimum composite score to emit an event       |
| `ANOMALY_IMMEDIATE_DEDUP_SECONDS`     | `60`    | Dedup window for immediate-path emissions      |
| `ANOMALY_BLAST_RADIUS_WINDOW_SECONDS` | `60`    | Cross-service error tracking window            |

### Runtime Config (PATCH /config)

All thresholds can be adjusted without restarting:

```bash theme={null}
curl -X PATCH http://bridge:8080/config \
  -H "Content-Type: application/json" \
  -d '{
    "compositeScoreThreshold": 0.45,
    "zscoreThreshold": 2.5,
    "immediateDeduplicationSeconds": 120
  }'
```

***

## Metrics

| Metric                                            | Description                                            |
| ------------------------------------------------- | ------------------------------------------------------ |
| `logclaw_bridge_anomaly_signals_extracted_total`  | Error records that produced at least one signal        |
| `logclaw_bridge_anomaly_immediate_detected_total` | Events emitted via immediate path                      |
| `logclaw_bridge_anomaly_windowed_detected_total`  | Events emitted via windowed path                       |
| `logclaw_bridge_anomaly_immediate_deduped_total`  | Immediate emissions suppressed by dedup                |
| `logclaw_bridge_anomaly_below_threshold_total`    | Records with signals but below composite threshold     |
| `logclaw_bridge_anomaly_std_zero_detected_total`  | Constant error rate cases detected (previously silent) |

***

## Image Naming Convention

All LogClaw service images follow the pattern:

```
ghcr.io/logclaw/logclaw-{servicename}:{version}
```

| Service         | Image                                     |
| --------------- | ----------------------------------------- |
| Bridge          | `ghcr.io/logclaw/logclaw-bridge`          |
| Ticketing Agent | `ghcr.io/logclaw/logclaw-ticketing-agent` |
| Auth Proxy      | `ghcr.io/logclaw/logclaw-auth-proxy`      |
| Flink Jobs      | `ghcr.io/logclaw/logclaw-flink-jobs`      |

> **Note:** Older images published as `ghcr.io/logclaw/bridge` and `ghcr.io/logclaw/ticketing-agent` (without the `logclaw-` prefix) are legacy. New builds should always use the `logclaw-{servicename}` prefix.


# OTel Collector
Source: https://docs.logclaw.ai/components/otel-collector

OTLP-native log ingestion gateway using OpenTelemetry Collector.

# OTel Collector

The OpenTelemetry Collector is LogClaw's **sole ingestion gateway**. It accepts logs via OTLP (the CNCF industry standard) and forwards them to Kafka for processing.

## Why OTLP?

OTLP is supported natively by every major observability vendor — Datadog, Splunk, Grafana, AWS CloudWatch, GCP Cloud Logging, and Azure Monitor. If your app already uses OpenTelemetry, point it at LogClaw and you're done. No custom integrations needed.

## Endpoints

| Transport | Port   | Path       | Use Case                                                                           |
| --------- | ------ | ---------- | ---------------------------------------------------------------------------------- |
| gRPC      | `4317` | —          | High-throughput, binary Protobuf. Recommended for production SDKs and OTel agents. |
| HTTP/JSON | `4318` | `/v1/logs` | Human-readable JSON. Good for curl, scripts, and debugging.                        |

Both ports are exposed on the `logclaw-otel-collector` Kubernetes service.

## Pipeline

```
OTLP Receiver (gRPC + HTTP)
       │
       ▼
Memory Limiter (800 MiB limit, 200 MiB spike)
       │
       ▼
Resource Processor (inject tenant_id)
       │
       ▼
Batch Processor (1000 per batch, 5s timeout)
       │
       ▼
Kafka Exporter (raw-logs topic, otlp_json, lz4)
```

### Collector Configuration

```yaml theme={null}
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 800
    spike_limit_mib: 200
  resource:
    attributes:
      - key: tenant_id
        value: "{{ .Values.global.tenantId }}"
        action: upsert
  batch:
    send_batch_size: 1000
    timeout: 5s

exporters:
  kafka:
    brokers:
      - "{{ .Values.global.kafkaBrokers }}"
    topic: "raw-logs"
    encoding: otlp_json
    producer:
      compression: lz4
      max_message_bytes: 10485760

extensions:
  health_check:
    endpoint: "0.0.0.0:13133"

service:
  extensions: [health_check]
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [kafka]
```

## Health Check

```bash theme={null}
curl http://localhost:13133/
# Expected: {"status":"Server available", ...}
```

## Helm Values

```yaml theme={null}
logclaw-otel-collector:
  replicaCount: 3
  image:
    repository: otel/opentelemetry-collector-contrib
    tag: "0.114.0"
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 50
    targetCPUUtilizationPercentage: 60
```

## Scaling

The OTel Collector supports horizontal pod autoscaling. For high-throughput deployments:

| Tier     | Replicas | HPA                         |
| -------- | -------- | --------------------------- |
| standard | 1        | Disabled                    |
| ha       | 3        | Optional                    |
| ultra-ha | 5+       | Recommended (min 5, max 50) |

See the [OTLP Integration Guide](/otlp-integration) for SDK examples and troubleshooting.


# Ticketing Agent
Source: https://docs.logclaw.ai/components/ticketing-agent

AI SRE agent for automated incident detection, trace correlation, and multi-platform ticket management.

# Ticketing Agent

The Ticketing Agent is an **AI SRE agent** that consumes anomalies from the pipeline, correlates them with trace data, and creates deduplicated incident tickets across multiple platforms.

## Supported Platforms

<CardGroup>
  <Card title="PagerDuty" icon="bell">
    Severity-based routing with auto-acknowledgment and escalation policies.
  </Card>

  <Card title="Jira" icon="ticket">
    Project/issue type mapping with custom fields and assignment rules.
  </Card>

  <Card title="ServiceNow" icon="clipboard">
    CMDB integration with assignment groups and priority mapping.
  </Card>

  <Card title="OpsGenie" icon="bullhorn">
    Team-based routing with on-call schedules and escalation.
  </Card>

  <Card title="Slack" icon="hashtag">
    Webhook notifications with thread updates for incident progression.
  </Card>

  <Card title="Zammad" icon="inbox">
    Self-hosted ticketing for air-gapped or on-prem deployments.
  </Card>
</CardGroup>

## How It Works

```
Kafka "enriched-logs" (anomaly events)
         │
         ▼
┌─────────────────────────┐
│    Anomaly Consumer     │  Consume anomaly-flagged documents
│    (Kafka consumer)     │
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│   Deduplication Engine  │  Group by service + error pattern
│   (time window + hash)  │  Prevent duplicate tickets
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│   Trace Correlation     │  Attach request timeline
│   (blast radius)        │  Compute affected services
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│   LLM Summarization     │  Generate human-readable
│   (optional)            │  incident summary
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│   Severity Router       │  Route to platforms based on
│   (per-severity rules)  │  configured routing rules
└────────────┬────────────┘
             │
    ┌────────┼────────┬──────────┐
    ▼        ▼        ▼          ▼
PagerDuty  Jira  ServiceNow   Slack ...
```

## Configuration

### Platform Configuration

Each platform requires specific credentials. These are stored in the cluster's secret store (AWS Secrets Manager, GCP Secret Manager, Vault, or Azure Key Vault) and synced via External Secrets Operator.

<Tabs>
  <Tab title="PagerDuty">
    | Field        | Required | Description                         |
    | ------------ | -------- | ----------------------------------- |
    | `routingKey` | Yes      | PagerDuty Events API v2 routing key |

    ```json theme={null}
    {
      "pagerduty": {
        "enabled": true,
        "routingKey": "your-routing-key"
      }
    }
    ```
  </Tab>

  <Tab title="Jira">
    | Field        | Required | Description                                              |
    | ------------ | -------- | -------------------------------------------------------- |
    | `baseUrl`    | Yes      | Jira instance URL (e.g. `https://yourorg.atlassian.net`) |
    | `apiToken`   | Yes      | Jira API token                                           |
    | `userEmail`  | Yes      | Jira user email for authentication                       |
    | `projectKey` | No       | Default project key (e.g. `SRE`)                         |

    ```json theme={null}
    {
      "jira": {
        "enabled": true,
        "baseUrl": "https://yourorg.atlassian.net",
        "apiToken": "your-api-token",
        "userEmail": "sre@yourorg.com",
        "projectKey": "SRE"
      }
    }
    ```
  </Tab>

  <Tab title="ServiceNow">
    | Field         | Required | Description             |
    | ------------- | -------- | ----------------------- |
    | `instanceUrl` | Yes      | ServiceNow instance URL |
    | `username`    | Yes      | ServiceNow username     |
    | `password`    | Yes      | ServiceNow password     |
    | `instance`    | No       | Short instance name     |

    ```json theme={null}
    {
      "servicenow": {
        "enabled": true,
        "instanceUrl": "https://yourorg.service-now.com",
        "username": "logclaw-integration",
        "password": "your-password"
      }
    }
    ```
  </Tab>

  <Tab title="OpsGenie">
    | Field    | Required | Description      |
    | -------- | -------- | ---------------- |
    | `apiKey` | Yes      | OpsGenie API key |

    ```json theme={null}
    {
      "opsgenie": {
        "enabled": true,
        "apiKey": "your-opsgenie-api-key"
      }
    }
    ```
  </Tab>

  <Tab title="Slack">
    | Field        | Required | Description                |
    | ------------ | -------- | -------------------------- |
    | `webhookUrl` | Yes      | Slack incoming webhook URL |

    ```json theme={null}
    {
      "slack": {
        "enabled": true,
        "webhookUrl": "https://hooks.slack.com/services/T.../B.../..."
      }
    }
    ```
  </Tab>
</Tabs>

### Severity Routing

Configure which platforms receive incidents based on severity:

```json theme={null}
{
  "routing": {
    "critical": ["pagerduty", "slack", "jira"],
    "high": ["jira", "slack"],
    "medium": ["jira"],
    "low": ["slack"]
  }
}
```

### Anomaly Settings

| Parameter        | Default | Description                                 |
| ---------------- | ------- | ------------------------------------------- |
| `minimumScore`   | `0.85`  | Minimum anomaly score to create an incident |
| `lookbackWindow` | `15m`   | Time window for anomaly grouping            |

```json theme={null}
{
  "anomaly": {
    "minimumScore": 0.85,
    "lookbackWindow": "15m"
  }
}
```

### LLM Configuration

The agent optionally uses an LLM to generate human-readable incident summaries:

```json theme={null}
{
  "llm": {
    "provider": "openai",
    "model": "gpt-4",
    "apiKey": "sk-..."
  }
}
```

## Incident Lifecycle

Incidents progress through these states:

```
OPEN → ACKNOWLEDGED → RESOLVED
  │                       ▲
  └── ESCALATED ──────────┘
```

| State          | Description                                     |
| -------------- | ----------------------------------------------- |
| `open`         | New incident, not yet acknowledged              |
| `acknowledged` | Team has seen the incident                      |
| `resolved`     | Issue has been fixed                            |
| `escalated`    | Escalated to a higher-priority platform or team |

### Incident Actions

```bash theme={null}
# List incidents
GET /api/incidents

# Acknowledge an incident
POST /api/incidents/:id/acknowledge

# Resolve an incident
POST /api/incidents/:id/resolve

# Escalate an incident
POST /api/incidents/:id/escalate
```

## Helm Values

```yaml theme={null}
logclaw-ticketing-agent:
  config:
    pagerduty:
      enabled: true
    jira:
      enabled: true
      baseUrl: "https://yourorg.atlassian.net"
      projectKey: "SRE"
    servicenow:
      enabled: true
      instance: "yourorg"
    anomaly:
      minimumScore: 0.85
      lookbackWindow: "15m"
```

## Testing Connectivity

Test connectivity to each platform before going live:

```bash theme={null}
# Test PagerDuty connection
POST /api/ticketing/api/v1/test-connection
{ "platform": "pagerduty" }

# Test LLM connection
POST /api/ticketing/api/v1/test-llm
```


# Integrations
Source: https://docs.logclaw.ai/integrations

Every platform LogClaw connects to — from incident routing to cloud infrastructure to data migration.

LogClaw integrates with the tools your team already uses. No proprietary agents, no vendor lock-in — just OpenTelemetry in, intelligence out.

***

## Incident Routing

LogClaw auto-creates incident tickets with AI root cause analysis, routed by severity to the platform your team already uses.

<CardGroup>
  <Card title="PagerDuty" icon={<img src="/images/integrations/pagerduty.svg" width="20" />}>
    Events API v2 with severity mapping. Critical and high-severity incidents trigger pages automatically.
  </Card>

  <Card title="Jira" icon={<img src="/images/integrations/jira.svg" width="20" />}>
    REST API v2 issue creation with priority mapping, custom labels, and project routing.
  </Card>

  <Card title="ServiceNow" icon={<img src="/images/integrations/servicenow.svg" width="20" />}>
    Incident table API with severity, urgency, impact fields, and assignment group routing.
  </Card>

  <Card title="OpsGenie" icon={<img src="/images/integrations/opsgenie.svg" width="20" />}>
    v2 Alerts API with priority mapping, responder teams, and escalation support.
  </Card>

  <Card title="Slack" icon={<img src="/images/integrations/slack.svg" width="20" />}>
    Incoming webhooks with severity indicators, service tags, and direct links to incident details.
  </Card>
</CardGroup>

### Severity-Based Routing

Configure which platform receives incidents based on severity:

```yaml theme={null}
routing:
  critical: pagerduty    # Wake someone up
  high: pagerduty        # Page the on-call
  medium: jira           # Create a ticket
  low: slack             # Post to channel
```

Every platform supports connection health checks with latency metrics from the LogClaw dashboard.

***

## Cloud Providers

LogClaw is Kubernetes-native and runs on any cloud. Helm charts include provider-specific configurations for authentication, secrets, and object storage.

<CardGroup>
  <Card title="AWS" icon={<img src="/images/integrations/aws.svg" width="20" />}>
    **EKS** with IRSA (IAM Roles for Service Accounts), **S3** for object storage, **AWS Secrets Manager** via External Secrets Operator.
  </Card>

  <Card title="Google Cloud" icon={<img src="/images/integrations/gcp.svg" width="20" />}>
    **GKE** with Workload Identity, **GCS** for object storage, **Google Secret Manager** via External Secrets Operator.
  </Card>

  <Card title="Azure" icon={<img src="/images/integrations/azure.svg" width="20" />}>
    **AKS** with Workload Identity, **Azure Blob Storage**, **Azure Key Vault** via External Secrets Operator.
  </Card>
</CardGroup>

### Self-Hosted

LogClaw also runs on any self-managed Kubernetes cluster with:

* **MinIO** for S3-compatible object storage
* **HashiCorp Vault** for secret management
* Standard CSI-compatible persistent volumes

***

## Data Ingestion

<Card title="OpenTelemetry (OTLP)" icon={<img src="/images/integrations/otel.svg" width="20" />}>
  **The only ingestion protocol.** LogClaw receives logs exclusively via OTLP — the CNCF industry standard adopted by 48.5% of enterprises. gRPC on port 4317, HTTP on port 4318.
</Card>

### Migrate From Existing Tools

LogClaw replaces these platforms for log observability and incident detection. Since OpenTelemetry is vendor-neutral, migrating is straightforward — point your OTel Collector or SDK at LogClaw instead of your current vendor.

<CardGroup>
  <Card title="Splunk" icon={<img src="/images/integrations/splunk.svg" width="20" />}>
    Replace Splunk's log observability at **80-90% lower cost**. No SPL lock-in, no proprietary Universal Forwarders. Your data stays in your cluster.
  </Card>

  <Card title="Datadog" icon={<img src="/images/integrations/datadog.svg" width="20" />}>
    Eliminate per-GB pricing surprises. LogClaw delivers autonomous incident detection that Datadog's dashboard-first model cannot match.
  </Card>

  <Card title="New Relic" icon={<img src="/images/integrations/newrelic.svg" width="20" />}>
    Move from New Relic's consumption-based pricing to self-hosted log intelligence with zero vendor risk.
  </Card>

  <Card title="Dynatrace" icon={<img src="/images/integrations/dynatrace.svg" width="20" />}>
    Replace Dynatrace log monitoring with OTLP-native ingestion and AI-powered root cause analysis — at a fraction of the cost.
  </Card>

  <Card title="CloudWatch" icon={<img src="/images/integrations/cloudwatch.svg" width="20" />}>
    Consolidate CloudWatch Logs into LogClaw for cross-cloud visibility, anomaly detection, and auto-ticketing that CloudWatch doesn't offer.
  </Card>
</CardGroup>

<Info>
  LogClaw doesn't require you to rip-and-replace overnight. You can run LogClaw alongside your existing tools — send the same OTLP data to both — and migrate incrementally as your team gains confidence.
</Info>

***

## AI / LLM Providers

LogClaw uses LLMs for root cause analysis — generating human-readable incident summaries from correlated log entries. Choose your provider:

| Provider               | Type        | Details                                 |
| ---------------------- | ----------- | --------------------------------------- |
| **Claude** (Anthropic) | Cloud API   | `v1/messages` — recommended for quality |
| **OpenAI**             | Cloud API   | `v1/chat/completions` — GPT-4o / GPT-4  |
| **Ollama**             | Self-hosted | Local inference — full data sovereignty |
| **vLLM**               | Self-hosted | High-throughput inference server        |

<Tip>
  For maximum data sovereignty, pair LogClaw with **Ollama** or **vLLM** — your logs and incident summaries never leave your infrastructure.
</Tip>

***

## Observability Stack

| Component      | Integration | Purpose                                                       |
| -------------- | ----------- | ------------------------------------------------------------- |
| **Prometheus** | Built-in    | All LogClaw services expose `/metrics` endpoints for scraping |
| **Grafana**    | Optional    | Pre-built dashboard JSON via `grafanaDashboards` Helm flag    |
| **OpenSearch** | Core        | Log storage, full-text search, and analytics backend          |
| **Kafka**      | Core        | Internal message broker for log streaming between components  |


# MCP Server
Source: https://docs.logclaw.ai/integrations/mcp

Connect Claude Code, Cursor, or Windsurf to LogClaw incidents, logs, and anomalies via the Model Context Protocol.

Connect your AI coding tools to LogClaw incidents, logs, and anomalies via the [Model Context Protocol](https://modelcontextprotocol.io). Works with **Claude Code**, **Cursor**, **Windsurf**, **Claude.ai**, and any MCP-compatible client.

<CardGroup>
  <Card title="Remote (Cloud)" icon="cloud" href="#remote-cloud">
    Connect via URL — no installation needed
  </Card>

  <Card title="Local (npm)" icon="npm" href="#local-npm">
    Install locally via `npx logclaw-mcp-server`
  </Card>
</CardGroup>

***

## Remote (Cloud)

Connect directly via URL — no installation required. LogClaw's remote MCP server uses **OAuth 2.1** and **Streamable HTTP** transport.

```
https://mcp.logclaw.ai
```

<Info>
  When you connect, you'll be prompted to enter your LogClaw API key. The key is encrypted and never exposed to the MCP client.
</Info>

### Claude Desktop / Claude.ai

Add LogClaw as a remote MCP server:

1. Open **Settings → MCP Servers → Add Server**
2. Enter the URL: `https://mcp.logclaw.ai`
3. Complete the OAuth flow by entering your API key

### Any MCP Client (URL-based)

For any MCP client that supports remote servers, use:

| Setting        | Value                    |
| -------------- | ------------------------ |
| **Server URL** | `https://mcp.logclaw.ai` |
| **Transport**  | Streamable HTTP          |
| **Auth**       | OAuth 2.1 (automatic)    |

***

## Local (npm)

Install and run the MCP server locally via stdio transport:

```bash theme={null}
npx logclaw-mcp-server
```

The server communicates over **stdio** using the Model Context Protocol. Your AI tool launches it automatically — you just need to add the config once.

### Setup

<CodeGroup>
  ```json Claude Code (~/.claude.json) theme={null}
  {
    "mcpServers": {
      "logclaw": {
        "command": "npx",
        "args": ["-y", "logclaw-mcp-server"],
        "env": {
          "LOGCLAW_ENDPOINT": "https://ticket.logclaw.ai",
          "LOGCLAW_API_KEY": "lc_proj_..."
        }
      }
    }
  }
  ```

  ```json Cursor (.cursor/mcp.json) theme={null}
  {
    "mcpServers": {
      "logclaw": {
        "command": "npx",
        "args": ["-y", "logclaw-mcp-server"],
        "env": {
          "LOGCLAW_ENDPOINT": "https://ticket.logclaw.ai",
          "LOGCLAW_API_KEY": "lc_proj_..."
        }
      }
    }
  }
  ```

  ```json Windsurf (.windsurf/mcp.json) theme={null}
  {
    "mcpServers": {
      "logclaw": {
        "command": "npx",
        "args": ["-y", "logclaw-mcp-server"],
        "env": {
          "LOGCLAW_ENDPOINT": "https://ticket.logclaw.ai",
          "LOGCLAW_API_KEY": "lc_proj_..."
        }
      }
    }
  }
  ```
</CodeGroup>

***

## Environment Variables

| Variable           | Required | Description                                                |
| ------------------ | -------- | ---------------------------------------------------------- |
| `LOGCLAW_API_KEY`  | Yes      | Your LogClaw project API key (starts with `lc_proj_`)      |
| `LOGCLAW_ENDPOINT` | No       | Auth proxy endpoint (default: `https://ticket.logclaw.ai`) |

Get your API key from the [LogClaw Console](https://console.logclaw.ai) under **Settings → API Keys**.

***

## Available Tools

The MCP server exposes 8 tools that your AI assistant can call:

| Tool                       | Description                                                                                     |
| -------------------------- | ----------------------------------------------------------------------------------------------- |
| `logclaw_list_incidents`   | List and filter incidents by severity, state, service, or search query                          |
| `logclaw_get_incident`     | Full incident details — root cause, causal chain, evidence logs, traces, timeline, blast radius |
| `logclaw_update_incident`  | Transition incident state (acknowledge, investigate, mitigate, resolve) or add a note           |
| `logclaw_search_logs`      | Search raw logs by service, level, time range, and query                                        |
| `logclaw_get_anomalies`    | Get recent anomaly detections (Z-score analysis on error rates)                                 |
| `logclaw_service_health`   | Check LogClaw pipeline health status and latency                                                |
| `logclaw_bulk_update`      | Update multiple incidents at once (acknowledge, investigate, mitigate, or resolve)              |
| `logclaw_export_incidents` | Export incidents in CSV, markdown table, or JSON format                                         |

***

## Example Prompts

Once connected, try asking your AI assistant:

<CardGroup>
  <Card title="Triage incidents" icon="fire">
    "What incidents are open right now?"
  </Card>

  <Card title="Root cause analysis" icon="magnifying-glass">
    "Show me the root cause for TICK-0037"
  </Card>

  <Card title="Search logs" icon="scroll">
    "Search for ERROR logs from auth-service in the last 30 minutes"
  </Card>

  <Card title="Query anomalies" icon="chart-line">
    "Are there any critical anomalies?"
  </Card>

  <Card title="Update incidents" icon="check">
    "Acknowledge TICK-0042 and add a note that I'm looking into it"
  </Card>

  <Card title="Health check" icon="heart-pulse">
    "Is the LogClaw pipeline healthy?"
  </Card>

  <Card title="Bulk update" icon="list-check">
    "Acknowledge TICK-0037, TICK-0038, and TICK-0039"
  </Card>

  <Card title="Export incidents" icon="file-export">
    "Export all critical incidents as a markdown table"
  </Card>
</CardGroup>

***

## Self-Hosted

Running LogClaw on your own infrastructure? Point `LOGCLAW_ENDPOINT` to your auth proxy:

```bash theme={null}
LOGCLAW_ENDPOINT=https://logclaw.internal.company.com \
  LOGCLAW_API_KEY=lc_proj_... \
  npx logclaw-mcp-server
```

<Tip>
  **Air-gapped?** Install the package from a private registry or copy the `dist/` folder directly. The MCP server has zero runtime network dependencies beyond reaching your auth proxy.
</Tip>

***

## Source Code

<CardGroup>
  <Card title="GitHub" icon="github" href="https://github.com/logclaw/logclaw/tree/main/apps/logclaw-mcp-server">
    View source code and contribute
  </Card>

  <Card title="npm" icon="npm" href="https://www.npmjs.com/package/logclaw-mcp-server">
    logclaw-mcp-server on npm
  </Card>
</CardGroup>


# Introduction
Source: https://docs.logclaw.ai/introduction

AI-powered log intelligence platform with real-time anomaly detection, trace correlation, and OTLP-native ingestion.

# Welcome to LogClaw

LogClaw is an **enterprise-grade, Kubernetes-native log intelligence platform**. It combines real-time anomaly detection, trace-correlated incident ticketing, and GitOps-native multi-tenancy into a single deployment stack.

<img alt="LogClaw" />

<img alt="LogClaw" />

## Why LogClaw?

<CardGroup>
  <Card title="OTLP-Native Ingestion" icon="plug">
    Uses OpenTelemetry Protocol as the sole ingestion interface. Any OTel SDK, agent, or collector works out of the box — no custom integrations.
  </Card>

  <Card title="AI-Powered Detection" icon="brain">
    Statistical anomaly scoring on error rates, automatic trace correlation across services, and blast radius computation — all in real time.
  </Card>

  <Card title="Multi-Platform Ticketing" icon="ticket">
    Route incidents to PagerDuty, Jira, ServiceNow, OpsGenie, Slack, or Zammad simultaneously with per-severity routing rules.
  </Card>

  <Card title="GitOps-Native Tenancy" icon="code-branch">
    Add a YAML file, push to main, and ArgoCD deploys a fully isolated tenant stack in 30 minutes. Namespace-per-tenant — no shared data plane.
  </Card>
</CardGroup>

## How It Works

```
Apps ──OTLP──▶ OTel Collector ──▶ Kafka ──▶ Bridge (ETL + Anomaly + Trace Correlation)
  (gRPC :4317)                                        │
  (HTTP :4318)                          ┌──────────────┴──────────────┐
                                        ▼                             ▼
                                  OpenSearch                   Ticketing Agent
                                (search + analytics)       (PagerDuty, Jira, etc.)
                                        │
                                        ▼
                                   Dashboard
                              (Next.js pipeline UI)
```

## Core Components

| Component                | Role                                                               | Technology                      |
| ------------------------ | ------------------------------------------------------------------ | ------------------------------- |
| **OTel Collector**       | OTLP gRPC/HTTP receiver, batching, tenant enrichment               | OpenTelemetry Collector Contrib |
| **Kafka**                | Durable event bus — raw logs + enriched logs topics                | Strimzi KRaft                   |
| **Bridge**               | OTLP ETL, anomaly detection, trace correlation, OpenSearch indexer | Python, Kafka                   |
| **OpenSearch**           | Full-text search, log analytics, visualization                     | OpenSearch + Dashboards         |
| **Ticketing Agent**      | AI SRE — deduplicated, trace-correlated incident tickets           | Python, LangChain               |
| **ML Engine**            | Feature Store + model inference serving                            | Feast, KServe                   |
| **Airflow**              | ML pipeline orchestration and retraining DAGs                      | Apache Airflow                  |
| **Infrastructure Agent** | Cluster health collection — Kafka lag, Flink, OpenSearch, ESO      | Go                              |
| **Dashboard**            | Web UI for log ingestion, incidents, anomalies, and config         | Next.js                         |

## Quick Start

<Tabs>
  <Tab title="Docker Compose (Fastest)">
    No cloning, no building — just pull and run:

    ```bash theme={null}
    curl -O https://raw.githubusercontent.com/logclaw/logclaw/main/docker-compose.yml
    curl -O https://raw.githubusercontent.com/logclaw/logclaw/main/otel-collector-config.yaml
    docker compose up -d
    ```

    Open **[http://localhost:3000](http://localhost:3000)** — the full stack is running. All images are pulled from `ghcr.io/logclaw/` (public, no auth required).

    Send a test log:

    ```bash theme={null}
    curl -X POST http://localhost:4318/v1/logs \
      -H "Content-Type: application/json" \
      -d '{
        "resourceLogs": [{
          "resource": {
            "attributes": [
              {"key": "service.name", "value": {"stringValue": "my-app"}}
            ]
          },
          "scopeLogs": [{
            "logRecords": [{
              "timeUnixNano": "'$(date +%s)000000000'",
              "severityText": "INFO",
              "body": {"stringValue": "Hello from LogClaw!"}
            }]
          }]
        }]
      }'
    ```
  </Tab>

  <Tab title="Kind Cluster (Full K8s Stack)">
    For the full Kubernetes experience with all operators:

    ```bash theme={null}
    git clone https://github.com/logclaw/logclaw.git
    cd logclaw
    ./scripts/setup-dev.sh
    ```

    This creates a Kind cluster, installs all operators and services, and runs a smoke test. Takes \~20 minutes on a 16 GB machine.
  </Tab>
</Tabs>

### Container Images

All LogClaw images are public on GHCR — pull directly with no auth:

```bash theme={null}
docker pull ghcr.io/logclaw/logclaw-dashboard:stable
docker pull ghcr.io/logclaw/logclaw-bridge:stable
docker pull ghcr.io/logclaw/logclaw-ticketing-agent:stable
docker pull ghcr.io/logclaw/logclaw-flink-jobs:stable
```

## Next Steps

<CardGroup>
  <Card title="Architecture" icon="sitemap" href="/architecture">
    Understand the deployment model, data flow, and component interactions.
  </Card>

  <Card title="Tenant Onboarding" icon="rocket" href="/onboarding">
    Provision a new tenant from zero to fully operational in 30 minutes.
  </Card>

  <Card title="OTLP Integration" icon="plug" href="/otlp-integration">
    Send logs from any language or framework using OpenTelemetry SDKs.
  </Card>

  <Card title="Local Development" icon="laptop-code" href="/local-development">
    Set up a local dev environment with Docker Compose or Kind.
  </Card>

  <Card title="Dashboard Guide" icon="chart-line" href="/components/dashboard">
    Explore the web UI for log ingestion, incident management, and pipeline monitoring.
  </Card>

  <Card title="API Reference" icon="code" href="/api-reference/overview">
    Full API documentation for every LogClaw service.
  </Card>
</CardGroup>


# Local Development
Source: https://docs.logclaw.ai/local-development

Set up a local LogClaw development environment for testing and development.

# Local Development

This guide covers setting up LogClaw for local development and testing.

## Option 1: Docker Compose (Fastest — No Clone Required)

Run the full LogClaw stack using pre-built public images. No cloning, no building, no Kubernetes.

```bash theme={null}
curl -O https://raw.githubusercontent.com/logclaw/logclaw/main/docker-compose.yml
curl -O https://raw.githubusercontent.com/logclaw/logclaw/main/otel-collector-config.yaml
docker compose up -d
```

**Requirements:** Docker with 8 GB+ RAM allocated.

This starts 6 services:

| Service               | Port  | URL                                                          |
| --------------------- | ----- | ------------------------------------------------------------ |
| Dashboard             | 3000  | [http://localhost:3000](http://localhost:3000)               |
| OTel Collector (gRPC) | 4317  | —                                                            |
| OTel Collector (HTTP) | 4318  | `POST /v1/logs`                                              |
| Bridge                | 8080  | [http://localhost:8080/health](http://localhost:8080/health) |
| Ticketing Agent       | 18081 | [http://localhost:18081](http://localhost:18081)             |
| OpenSearch            | 9200  | [http://localhost:9200](http://localhost:9200)               |

All images are pulled from `ghcr.io/logclaw/` — public, no registry auth required.

### Send a test log

```bash theme={null}
curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [{
      "resource": {
        "attributes": [
          {"key": "service.name", "value": {"stringValue": "my-app"}}
        ]
      },
      "scopeLogs": [{
        "logRecords": [{
          "timeUnixNano": "'$(date +%s)000000000'",
          "severityText": "ERROR",
          "body": {"stringValue": "Connection refused to database"}
        }]
      }]
    }]
  }'
```

### Stop and clean up

```bash theme={null}
docker compose down           # stop services, keep data
docker compose down -v        # stop services and delete data
```

### Container Images

| Service         | Image                                     | Tags              |
| --------------- | ----------------------------------------- | ----------------- |
| Dashboard       | `ghcr.io/logclaw/logclaw-dashboard`       | `stable`, `2.5.0` |
| Bridge          | `ghcr.io/logclaw/logclaw-bridge`          | `stable`, `1.3.0` |
| Ticketing Agent | `ghcr.io/logclaw/logclaw-ticketing-agent` | `stable`, `1.5.0` |
| Flink Jobs      | `ghcr.io/logclaw/logclaw-flink-jobs`      | `stable`, `0.1.1` |

The `:stable` tag always points to the latest verified release. Use a specific version tag (e.g., `:2.5.0`) for reproducible environments.

## Option 2: Kind Cluster (Full Kubernetes Stack)

For the full Kubernetes experience with all operators (Strimzi, Flink, ESO, cert-manager):

```bash theme={null}
# Clone and setup
git clone https://github.com/logclaw/logclaw.git
cd logclaw

# Create Kind cluster + install everything
./scripts/setup-dev.sh
```

This script:

1. Creates a Kind cluster with 3 worker nodes
2. Installs all Kubernetes operators (Strimzi, Flink, ESO, cert-manager, OpenSearch)
3. Deploys a dev tenant with all components enabled
4. Runs a smoke test

**Requirements:** Docker, Kind, Helm, kubectl (16 GB RAM recommended)

### Port Forwarding

After setup, forward ports to access services locally:

```bash theme={null}
# OTel Collector (OTLP HTTP)
kubectl port-forward svc/logclaw-otel-collector 4318:4318 -n logclaw &

# Bridge
kubectl port-forward svc/logclaw-bridge 8080:8080 -n logclaw &

# OpenSearch
kubectl port-forward svc/logclaw-opensearch 9200:9200 -n logclaw &

# Dashboard
kubectl port-forward svc/logclaw-dashboard 3000:3000 -n logclaw &
```

## Option 3: Dashboard Development (Frontend Only)

For working on the Dashboard UI without the full stack:

```bash theme={null}
cd apps/dashboard
cp .env.example .env.local   # Edit with your endpoints
npm install
npm run dev
```

The Dashboard starts on `http://localhost:3000`. Configure `.env.local` to point at your backend services (local Kind cluster, remote dev cluster, or mock endpoints).

## Option 4: Bridge Development (Python)

For working on the Bridge ETL service:

```bash theme={null}
cd apps/bridge
pip install -r requirements.txt

# Set environment variables
export KAFKA_BROKERS=localhost:9092
export KAFKA_TOPIC_RAW=raw-logs
export KAFKA_TOPIC_ENRICHED=enriched-logs
export OPENSEARCH_ENDPOINT=http://localhost:9200
export ANOMALY_THRESHOLD=2.5
export WINDOW_SIZE=50

python main.py
```

The Bridge exposes:

* `GET /health` — health check
* `GET /metrics` — Prometheus metrics
* `GET /config` — runtime configuration
* `PATCH /config` — update runtime configuration

## Project Structure

```
logclaw/
├── apps/
│   ├── bridge/              # Python — OTLP ETL + anomaly detection
│   │   ├── main.py
│   │   ├── requirements.txt
│   │   └── Dockerfile
│   ├── dashboard/           # Next.js — pipeline UI
│   │   ├── src/
│   │   │   ├── app/         # App router pages + API routes
│   │   │   ├── components/  # React components
│   │   │   └── lib/         # API client, utilities
│   │   ├── package.json
│   │   └── Dockerfile
│   └── agent/               # Go — infrastructure health collector
│       ├── main.go
│       ├── go.mod
│       └── Dockerfile
├── charts/                  # Helm sub-charts
│   ├── logclaw-tenant/      # Umbrella chart
│   ├── logclaw-otel-collector/
│   ├── logclaw-kafka/
│   ├── logclaw-bridge/
│   ├── logclaw-opensearch/
│   ├── logclaw-flink/
│   ├── logclaw-ml-engine/
│   ├── logclaw-airflow/
│   ├── logclaw-ticketing-agent/
│   ├── logclaw-platform/
│   └── logclaw-dashboard/
├── gitops/
│   └── tenants/             # Per-tenant values files
│       ├── _template.yaml
│       └── tenant-*.yaml
├── operators/               # Cluster operator manifests
├── scripts/                 # Dev scripts
└── docs/                    # This documentation
```

## Building Docker Images

### Bridge

```bash theme={null}
cd apps/bridge
docker build -t logclaw/bridge:dev .
```

### Dashboard

```bash theme={null}
cd apps/dashboard
docker build -t logclaw/dashboard:dev .
```

### Agent

```bash theme={null}
cd apps/agent
docker build -t logclaw/agent:dev .
```

## Helm Development

### Lint a Chart

```bash theme={null}
helm lint charts/logclaw-otel-collector/ \
  -f charts/logclaw-otel-collector/ci/default-values.yaml
```

### Template Rendering

```bash theme={null}
helm template logclaw charts/logclaw-tenant/ \
  -f gitops/tenants/tenant-gke-prod.yaml \
  --debug
```

### Update Dependencies

After modifying sub-charts, rebuild the umbrella chart:

```bash theme={null}
helm dependency update charts/logclaw-tenant/
```

<Warning>
  Always run `helm dependency update` after changing sub-chart templates. The umbrella chart uses `.tgz` packages — stale packages will render old templates.
</Warning>

## Sending Test Logs

### Single Log via curl

```bash theme={null}
curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [{
      "resource": {
        "attributes": [
          {"key": "service.name", "value": {"stringValue": "test-svc"}}
        ]
      },
      "scopeLogs": [{
        "logRecords": [{
          "timeUnixNano": "'$(date +%s)000000000'",
          "severityText": "ERROR",
          "body": {"stringValue": "Connection refused to database"},
          "attributes": [
            {"key": "environment", "value": {"stringValue": "dev"}}
          ]
        }]
      }]
    }]
  }'
```

### Batch via Dashboard

1. Open `http://localhost:3000`
2. Drag and drop a JSON file onto the upload area
3. The Dashboard converts each entry to OTLP format and sends via the proxy

Supported formats:

* **JSON** — array of log objects: `[{"message": "...", "level": "ERROR"}]`
* **NDJSON** — one JSON object per line


# Tenant Onboarding
Source: https://docs.logclaw.ai/onboarding

Provision a new LogClaw tenant from zero to fully operational in 30 minutes.

# Tenant Onboarding

This guide walks through provisioning a new LogClaw tenant from zero to fully operational. Expected time: **30 minutes** after prerequisites are met.

## Prerequisites

<AccordionGroup>
  <Accordion title="Cluster Prerequisites (one-time)" icon="server">
    **Kubernetes Cluster**

    * Kubernetes >= 1.27
    * Control plane: 4 vCPU, 8 GB RAM
    * Workers (minimum 3): 8 vCPU, 32 GB RAM each

    **Operators** (install once per cluster)

    * Strimzi Kafka Operator (`operators/strimzi/`)
    * Flink Kubernetes Operator (`operators/flink-operator/`)
    * External Secrets Operator (`operators/eso/`)
    * cert-manager (`operators/cert-manager/`)
    * OpenSearch Operator (`operators/opensearch-operator/`)

    **GitOps**

    * ArgoCD installed with `logclaw` AppProject applied
    * ArgoCD ApplicationSet `logclaw-tenants` applied
  </Accordion>

  <Accordion title="Per-Tenant Prerequisites" icon="list-check">
    * Unique `tenantId` chosen (lowercase letters, numbers, hyphens; max 40 chars)
    * Kubernetes StorageClass available (e.g. `gp3`, `standard`, `pd-ssd`)
    * Object storage bucket pre-created and accessible from cluster
    * Secrets backend configured (AWS Secrets Manager, GCP Secret Manager, Vault, or Azure Key Vault)
    * Image pull secret `logclaw-registry-pull` created (or omit if using public registry)
    * Ticketing provider credentials stored in secret backend (if ticketing agent enabled)
  </Accordion>

  <Accordion title="Access Requirements" icon="lock">
    * Write access to the LogClaw Git repository
    * ArgoCD UI or CLI access to monitor sync status
  </Accordion>
</AccordionGroup>

## Step 1: Create the Tenant Values File

Copy the template and fill in required fields:

```bash theme={null}
cp gitops/tenants/_template.yaml gitops/tenants/<tenantId>.yaml
```

Set the required global values:

```yaml theme={null}
tenantId: "acme-corp"

global:
  tenantName: "Acme Corporation"
  storageClass: "gp3"
  tier: "ha"                       # standard | ha | ultra-ha

  objectStorage:
    provider: "s3"                 # s3 | gcs | azure
    bucket: "logclaw-acme-corp"
    region: "us-east-1"

  secretStore:
    provider: "aws"                # aws | gcp | vault | azure
    region: "us-east-1"
```

### Enable Components

Toggle components based on your requirements:

```yaml theme={null}
# Core pipeline (recommended: all enabled)
platform:       { enabled: true }   # RBAC, NetworkPolicy, SecretStore
otelCollector:  { enabled: true }   # OTLP ingestion (gRPC :4317, HTTP :4318)
kafka:          { enabled: true }   # Event bus (required by most components)
opensearch:     { enabled: true }   # Search & analytics

# Processing (choose one or both)
flink:          { enabled: true }   # High-throughput stream processing
bridge:         { enabled: true }   # OTLP ETL + anomaly + trace correlation

# AI & Operations
mlEngine:       { enabled: true }   # Feast + KServe model inference
airflow:        { enabled: true }   # ML pipeline orchestration
ticketingAgent: { enabled: true }   # AI SRE incident management
agent:          { enabled: true }   # Infrastructure health collector

# UI
dashboard:      { enabled: true }   # Next.js pipeline UI
```

<Tip>
  For development environments, enable `bridge` and `dashboard` while disabling `flink`, `mlEngine`, and `airflow` to reduce resource requirements.
</Tip>

See the [Values Reference](/values-reference) for the full list of configurable fields.

## Step 2: Commit and Push

```bash theme={null}
git add gitops/tenants/<tenantId>.yaml
git commit -m "feat(tenants): onboard <tenantId>"
git push origin main
```

The ArgoCD Git generator polls every 3 minutes. To trigger immediately:

```bash theme={null}
argocd app get logclaw-tenants --refresh
```

## Step 3: Monitor Deployment

Watch the tenant application sync:

```bash theme={null}
# List all tenant applications
argocd app list -l logclaw.io/managed-by=applicationset

# Watch sync status
argocd app get logclaw-tenant-<tenantId> --watch
```

**Expected sync wave order:**

<Steps>
  <Step title="Namespace + Platform (t=0–5m)">
    Namespace creation, RBAC, NetworkPolicy, ClusterSecretStore
  </Step>

  <Step title="Kafka (t=5–10m)">
    Strimzi reconciles KRaft cluster. Wait for `READY=True`.
  </Step>

  <Step title="OTel Collector + OpenSearch (t=10–15m)">
    Deploy in parallel. OTel Collector connects to Kafka bootstrap. OpenSearch cluster reaches green.
  </Step>

  <Step title="Flink + Bridge (t=15–20m)">
    Flink jobs enter `RUNNING` state. Bridge connects to Kafka + OpenSearch.
  </Step>

  <Step title="ML Engine + Airflow (t=20–25m)">
    KServe InferenceService ready. Airflow scheduler + webserver healthy.
  </Step>

  <Step title="Ticketing Agent + Dashboard (t=25–30m)">
    Agent validates ticketing credentials. Dashboard available on ClusterIP.
  </Step>
</Steps>

## Step 4: Verify Components

Run the built-in Helm test suite:

```bash theme={null}
helm test logclaw-<tenantId> \
  --namespace logclaw-<tenantId> \
  --timeout 5m \
  --logs
```

### Individual Component Checks

<Tabs>
  <Tab title="Kafka">
    ```bash theme={null}
    kubectl get kafka -n logclaw-<tenantId>
    # Expected: READY=True, REPLICAS=3 (ha tier)
    ```
  </Tab>

  <Tab title="OTel Collector">
    ```bash theme={null}
    kubectl get pods -n logclaw-<tenantId> \
      -l app.kubernetes.io/name=logclaw-otel-collector
    # Expected: all pods Running

    # Test OTLP endpoint
    kubectl port-forward svc/logclaw-otel-collector 4318:4318 \
      -n logclaw-<tenantId>
    curl -s -o /dev/null -w "%{http_code}" \
      -X POST http://localhost:4318/v1/logs \
      -H "Content-Type: application/json" \
      -d '{"resourceLogs":[]}'
    # Expected: 200
    ```
  </Tab>

  <Tab title="OpenSearch">
    ```bash theme={null}
    kubectl get opensearchcluster -n logclaw-<tenantId>
    # Expected: state=Ready

    kubectl port-forward svc/logclaw-opensearch 9200:9200 \
      -n logclaw-<tenantId>
    curl -sk https://localhost:9200/_cluster/health | jq .status
    # Expected: "green"
    ```
  </Tab>

  <Tab title="Bridge">
    ```bash theme={null}
    kubectl get pods -n logclaw-<tenantId> \
      -l app.kubernetes.io/name=logclaw-bridge
    # Expected: Running

    kubectl port-forward svc/logclaw-bridge 8080:8080 \
      -n logclaw-<tenantId>
    curl http://localhost:8080/health
    # Expected: {"status": "ok", ...}
    ```
  </Tab>

  <Tab title="Ticketing Agent">
    ```bash theme={null}
    kubectl logs -n logclaw-<tenantId> \
      -l app.kubernetes.io/name=logclaw-ticketing-agent \
      --tail=20
    # Expected: "Connected to Kafka", "Ticketing provider validated"
    ```
  </Tab>

  <Tab title="Dashboard">
    ```bash theme={null}
    kubectl port-forward svc/logclaw-dashboard 3000:3000 \
      -n logclaw-<tenantId>
    # Open http://localhost:3000
    ```
  </Tab>

  <Tab title="Flink">
    ```bash theme={null}
    kubectl get flinkdeployment -n logclaw-<tenantId>
    # Expected: LIFECYCLE_STATE=STABLE, JOB_STATE=RUNNING
    ```
  </Tab>

  <Tab title="Agent">
    ```bash theme={null}
    kubectl get pods -n logclaw-<tenantId> \
      -l app.kubernetes.io/name=logclaw-agent
    # Expected: Running

    kubectl port-forward svc/logclaw-agent 8080:8080 \
      -n logclaw-<tenantId>
    curl http://localhost:8080/health
    # Expected: {"status": "ok"}
    ```
  </Tab>
</Tabs>

## Step 5: Send Your First Logs

After verification, send a test log via OTLP HTTP:

```bash theme={null}
curl -X POST http://localhost:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [{
      "resource": {
        "attributes": [
          {"key": "service.name", "value": {"stringValue": "onboarding-test"}}
        ]
      },
      "scopeLogs": [{
        "logRecords": [{
          "timeUnixNano": "'$(date +%s)000000000'",
          "severityText": "INFO",
          "body": {"stringValue": "Tenant onboarding complete!"}
        }]
      }]
    }]
  }'
```

Verify it appears in OpenSearch:

```bash theme={null}
curl -sk https://localhost:9200/logclaw-logs-*/_search \
  -H "Content-Type: application/json" \
  -d '{"query":{"match":{"service":"onboarding-test"}}}'
```

## Troubleshooting

<AccordionGroup>
  <Accordion title="ArgoCD Application stuck in Progressing">
    ```bash theme={null}
    argocd app get logclaw-tenant-<tenantId>
    kubectl describe application logclaw-tenant-<tenantId> -n argocd
    ```

    Check the "Conditions" section for error details.
  </Accordion>

  <Accordion title="Kafka not reaching Ready state">
    ```bash theme={null}
    kubectl describe kafka logclaw-<tenantId>-kafka -n logclaw-<tenantId>
    kubectl logs -n strimzi-system -l name=strimzi-cluster-operator --tail=50
    ```

    **Common causes:** StorageClass not found, PVC provisioning failure, resource limits too low.
  </Accordion>

  <Accordion title="OpenSearch cluster health red">
    ```bash theme={null}
    kubectl describe opensearchcluster -n logclaw-<tenantId>
    kubectl logs -n opensearch-operator-system \
      -l control-plane=controller-manager --tail=50
    ```

    **Common causes:** Insufficient memory (2 Gi minimum per data node), disk pressure.
  </Accordion>

  <Accordion title="Flink job not RUNNING">
    ```bash theme={null}
    kubectl describe flinkdeployment -n logclaw-<tenantId>
    kubectl logs -n logclaw-<tenantId> -l app=logclaw-flink-anomaly --tail=50
    ```

    **Common causes:** Kafka bootstrap unreachable, object storage credentials invalid.
  </Accordion>

  <Accordion title="External Secrets not syncing">
    ```bash theme={null}
    kubectl get externalsecret -n logclaw-<tenantId>
    kubectl describe externalsecret -n logclaw-<tenantId>
    kubectl get clustersecretstore
    ```

    **Common causes:** IAM role not attached, wrong region, missing permissions.
  </Accordion>

  <Accordion title="Force re-sync">
    ```bash theme={null}
    argocd app get logclaw-tenant-<tenantId> --hard-refresh
    argocd app sync logclaw-tenant-<tenantId> --force
    ```
  </Accordion>
</AccordionGroup>


# OTLP Integration Guide
Source: https://docs.logclaw.ai/otlp-integration

Send logs to LogClaw from any language or framework using OpenTelemetry Protocol.

# OTLP Integration Guide

LogClaw uses **OTLP (OpenTelemetry Protocol)** as its sole log ingestion protocol. OTLP is the
CNCF industry standard supported by every major observability vendor — Datadog, Splunk, Grafana,
AWS CloudWatch, GCP Cloud Logging, and Azure Monitor all speak OTLP natively.

No custom integrations needed. If your app already uses OpenTelemetry, point it at LogClaw and
you're done.

## Endpoints

| Transport | Port | Path       | Use Case                                                                           |
| --------- | ---- | ---------- | ---------------------------------------------------------------------------------- |
| gRPC      | 4317 | —          | High-throughput, binary Protobuf. Recommended for production SDKs and OTel agents. |
| HTTP/JSON | 4318 | `/v1/logs` | Human-readable JSON. Good for curl, scripts, and debugging.                        |

Both ports are exposed on the `logclaw-otel-collector` Kubernetes service.

## Quick Start — curl

The simplest way to send a log to LogClaw:

```bash theme={null}
curl -X POST http://<collector>:4318/v1/logs \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [{
      "resource": {
        "attributes": [
          {"key": "service.name", "value": {"stringValue": "my-service"}}
        ]
      },
      "scopeLogs": [{
        "logRecords": [{
          "timeUnixNano": "'$(date +%s)000000000'",
          "severityText": "ERROR",
          "body": {"stringValue": "Connection refused to database"},
          "traceId": "abcdef1234567890abcdef1234567890",
          "spanId": "abcdef12345678",
          "attributes": [
            {"key": "environment", "value": {"stringValue": "production"}},
            {"key": "region", "value": {"stringValue": "us-east-1"}}
          ]
        }]
      }]
    }]
  }'
```

Expected response: `200 OK` with `{"partialSuccess":{}}`.

## SDK Integration

### Python

Install the OpenTelemetry Python SDK:

```bash theme={null}
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc
```

```python theme={null}
import logging
from opentelemetry import _logs as logs
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.sdk.resources import Resource

# Configure the OTel log pipeline
resource = Resource.create({"service.name": "payment-api"})
provider = LoggerProvider(resource=resource)
provider.add_log_record_processor(
    BatchLogRecordProcessor(
        OTLPLogExporter(endpoint="http://<collector>:4317", insecure=True)
    )
)
logs.set_logger_provider(provider)

# Attach OTel handler to Python logging
handler = LoggingHandler(logger_provider=provider)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

# Use standard Python logging — logs go to LogClaw automatically
logger = logging.getLogger("payment-api")
logger.error("Connection refused to database", extra={"trace_id": "abc123"})
```

### Java (Log4j2 + OTel Appender)

Add the OTel Log4j2 appender to your `pom.xml`:

```xml theme={null}
<dependency>
  <groupId>io.opentelemetry.instrumentation</groupId>
  <artifactId>opentelemetry-log4j-appender-2.17</artifactId>
  <version>2.10.0-alpha</version>
</dependency>
```

Configure `log4j2.xml`:

```xml theme={null}
<Configuration>
  <Appenders>
    <OpenTelemetry name="OTelAppender" />
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="OTelAppender" />
    </Root>
  </Loggers>
</Configuration>
```

Set the environment variable to point at LogClaw:

```bash theme={null}
export OTEL_EXPORTER_OTLP_ENDPOINT=http://<collector>:4317
export OTEL_RESOURCE_ATTRIBUTES=service.name=order-engine
```

### Node.js (Winston + OTel)

```bash theme={null}
npm install @opentelemetry/sdk-logs @opentelemetry/exporter-logs-otlp-grpc \
  @opentelemetry/resources @opentelemetry/semantic-conventions
```

```javascript theme={null}
const { LoggerProvider, BatchLogRecordProcessor } = require('@opentelemetry/sdk-logs');
const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');
const { ATTR_SERVICE_NAME } = require('@opentelemetry/semantic-conventions');

const resource = new Resource({ [ATTR_SERVICE_NAME]: 'notification-svc' });
const loggerProvider = new LoggerProvider({ resource });

loggerProvider.addLogRecordProcessor(
  new BatchLogRecordProcessor(
    new OTLPLogExporter({ url: 'http://<collector>:4317' })
  )
);

const logger = loggerProvider.getLogger('notification-svc');
logger.emit({
  severityText: 'ERROR',
  body: 'Failed to send email notification',
  attributes: { 'user.id': '12345', environment: 'production' },
});
```

### Go

```bash theme={null}
go get go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc
go get go.opentelemetry.io/otel/sdk/log
```

```go theme={null}
package main

import (
    "context"
    "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
    sdklog "go.opentelemetry.io/otel/sdk/log"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func main() {
    ctx := context.Background()
    exporter, _ := otlploggrpc.New(ctx,
        otlploggrpc.WithEndpoint("<collector>:4317"),
        otlploggrpc.WithInsecure(),
    )
    res, _ := resource.New(ctx,
        resource.WithAttributes(semconv.ServiceName("inventory-api")),
    )
    provider := sdklog.NewLoggerProvider(
        sdklog.WithResource(res),
        sdklog.WithProcessor(sdklog.NewBatchProcessor(exporter)),
    )
    defer provider.Shutdown(ctx)

    logger := provider.Logger("inventory-api")
    // Use logger to emit log records
    _ = logger
}
```

## Using an OTel Collector as a Sidecar / Agent

For production deployments, the recommended pattern is to run an OTel Collector as a DaemonSet
or sidecar that collects logs from your pods and forwards them to LogClaw's collector.

```yaml theme={null}
# otel-agent-config.yaml
receivers:
  filelog:
    include: [/var/log/pods/*/*/*.log]
    operators:
      - type: container
        id: container-parser

exporters:
  otlp:
    endpoint: "logclaw-otel-collector.<namespace>.svc:4317"
    tls:
      insecure: true

service:
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [otlp]
```

This automatically captures all pod stdout/stderr logs without any application code changes.

## OTLP Field Mapping

LogClaw's Bridge flattens the nested OTLP structure into canonical flat documents for OpenSearch:

| OTLP Field                            | LogClaw Field                 | Description                         |
| ------------------------------------- | ----------------------------- | ----------------------------------- |
| `resource.attributes["service.name"]` | `service`                     | Service name                        |
| `logRecord.body.stringValue`          | `message`                     | Log message                         |
| `logRecord.severityText`              | `level`                       | Log level (INFO, WARN, ERROR, etc.) |
| `logRecord.timeUnixNano`              | `timestamp`                   | ISO-8601 timestamp                  |
| `logRecord.traceId`                   | `trace_id`                    | Distributed trace ID                |
| `logRecord.spanId`                    | `span_id`                     | Span ID                             |
| `resource.attributes["host.name"]`    | `host`                        | Hostname                            |
| `resource.attributes["tenant_id"]`    | `tenant_id`                   | Tenant (injected by collector)      |
| `logRecord.attributes[*]`             | Flattened as top-level fields | Custom attributes                   |

## Dashboard File Upload

The LogClaw dashboard supports drag-and-drop log file upload. When you upload a file through
the UI, the dashboard automatically converts each log entry to OTLP format using the built-in
`logsToOtlp()` converter and sends them to the OTel Collector via the `/api/otel/v1/logs` proxy.

Supported file formats:

* **JSON** — array of log objects `[{"message": "...", "level": "ERROR", ...}]`
* **NDJSON** — newline-delimited JSON (one log object per line)
* **CSV** — header row required, columns auto-mapped to OTLP fields
* **Text** (`.txt`, `.log`) — one log message per line

## Troubleshooting

**Logs not appearing in OpenSearch?**

1. Check OTel Collector health:
   ```bash theme={null}
   kubectl port-forward svc/logclaw-otel-collector 13133:13133
   curl http://localhost:13133/
   # Expected: {"status":"Server available","..."}
   ```

2. Check Collector logs for export errors:
   ```bash theme={null}
   kubectl logs -l app.kubernetes.io/name=logclaw-otel-collector --tail=50
   ```

3. Verify Kafka topic has messages:
   ```bash theme={null}
   kubectl exec -it logclaw-kafka-0 -- bin/kafka-console-consumer.sh \
     --bootstrap-server localhost:9092 --topic raw-logs --max-messages 1
   ```

4. Check Bridge OTLP ETL thread:
   ```bash theme={null}
   kubectl logs -l app.kubernetes.io/name=logclaw-bridge --tail=50 | grep "otlp"
   ```

**Getting `connection refused`?**

Ensure you're using the correct port:

* gRPC → `:4317`
* HTTP/JSON → `:4318/v1/logs`

Do not use port 8080 — that was the legacy Vector endpoint and is no longer available.


# Quick Start: Send Logs to LogClaw
Source: https://docs.logclaw.ai/quickstart-send-logs

Get your application logs flowing into LogClaw in under 5 minutes

# Quick Start: Send Logs to LogClaw

LogClaw ingests logs via **OTLP (OpenTelemetry Protocol)** — the CNCF industry standard. Every major language and framework has an OTel SDK.

**Your endpoint:**

<Tabs>
  <Tab title="LogClaw Cloud (Managed)">
    | Protocol | URL                               |
    | -------- | --------------------------------- |
    | HTTP     | `https://otel.logclaw.ai/v1/logs` |
    | gRPC     | `grpcs://otel.logclaw.ai:443`     |

    Every code example below includes this endpoint and your API key **inline** — just copy and run. See [API Keys](/api-keys) to create your key.

    <Tip>
      You do **not** need to set `tenant_id` — it is automatically injected based on your API key.
    </Tip>
  </Tab>

  <Tab title="Self-Hosted">
    | Protocol | URL                             |
    | -------- | ------------------------------- |
    | HTTP     | `http://localhost:4318/v1/logs` |
    | gRPC     | `localhost:4317`                |

    Every code example below defaults to the **LogClaw Cloud** endpoint. For self-hosted, swap the URL to `http://localhost:4318` and remove the `x-logclaw-api-key` header — look for the `# Self-hosted` comments in each snippet.

    No authentication required — access is controlled via Kubernetes NetworkPolicy.
  </Tab>
</Tabs>

***

## Send Your First Log

Pick your language — click a tab to switch. Copy the code and run it.

<CodeGroup>
  ```bash cURL theme={null}
  # LogClaw Cloud — replace lc_proj_... with your API key
  # Self-hosted: swap URL to http://localhost:4318/v1/logs and remove the api-key header
  curl -X POST https://otel.logclaw.ai/v1/logs \
    -H "Content-Type: application/json" \
    -H "x-logclaw-api-key: lc_proj_your_key_here" \
    -d '{
      "resourceLogs": [{
        "resource": {
          "attributes": [
            {"key": "service.name", "value": {"stringValue": "my-api"}}
          ]
        },
        "scopeLogs": [{
          "logRecords": [{
            "timeUnixNano": "'$(date +%s)000000000'",
            "severityText": "ERROR",
            "body": {"stringValue": "Connection timeout to payment-gateway after 30s"},
            "attributes": [
              {"key": "error.type", "value": {"stringValue": "TimeoutError"}},
              {"key": "http.route", "value": {"stringValue": "/api/v1/payments"}}
            ]
          }]
        }]
      }]
    }'
  ```

  ```python Python theme={null}
  # pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc
  import logging
  from opentelemetry import _logs
  from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
  from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
  from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
  from opentelemetry.sdk.resources import Resource

  resource = Resource.create({"service.name": "my-python-api"})
  provider = LoggerProvider(resource=resource)
  provider.add_log_record_processor(
      BatchLogRecordProcessor(OTLPLogExporter(
          endpoint="https://otel.logclaw.ai",                      # Self-hosted: http://localhost:4317
          headers={"x-logclaw-api-key": "lc_proj_your_key_here"},  # Self-hosted: remove this line
      ))
  )
  _logs.set_logger_provider(provider)

  handler = LoggingHandler(logger_provider=provider)
  logging.getLogger().addHandler(handler)
  logging.getLogger().setLevel(logging.INFO)

  logger = logging.getLogger(__name__)
  logger.error("Payment failed", extra={"order_id": "ord-456"})
  ```

  ```javascript Node.js theme={null}
  // npm install @opentelemetry/sdk-logs @opentelemetry/exporter-logs-otlp-grpc @opentelemetry/resources
  const { LoggerProvider, BatchLogRecordProcessor } = require('@opentelemetry/sdk-logs');
  const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-grpc');
  const { Resource } = require('@opentelemetry/resources');

  const resource = new Resource({ 'service.name': 'my-express-api' });
  const loggerProvider = new LoggerProvider({ resource });
  loggerProvider.addLogRecordProcessor(
    new BatchLogRecordProcessor(new OTLPLogExporter({
      url: 'https://otel.logclaw.ai',                                    // Self-hosted: http://localhost:4317
      headers: { 'x-logclaw-api-key': 'lc_proj_your_key_here' },        // Self-hosted: remove this line
    }))
  );

  const logger = loggerProvider.getLogger('app');
  logger.emit({ severityText: 'ERROR', body: 'Connection timeout', attributes: { 'http.route': '/api/orders' } });
  ```

  ```go Go theme={null}
  // go get go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc
  // go get go.opentelemetry.io/otel/sdk/log
  package main

  import (
      "context"
      "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
      sdklog "go.opentelemetry.io/otel/sdk/log"
      "go.opentelemetry.io/otel/sdk/resource"
      semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
      "google.golang.org/grpc/metadata"
  )

  func main() {
      ctx := context.Background()
      exporter, _ := otlploggrpc.New(ctx,
          otlploggrpc.WithEndpointURL("otel.logclaw.ai:443"),                                        // Self-hosted: localhost:4317
          otlploggrpc.WithHeaders(map[string]string{"x-logclaw-api-key": "lc_proj_your_key_here"}),  // Self-hosted: remove
      )
      res, _ := resource.New(ctx,
          resource.WithAttributes(semconv.ServiceName("my-go-api")),
      )
      provider := sdklog.NewLoggerProvider(
          sdklog.WithResource(res),
          sdklog.WithProcessor(sdklog.NewBatchProcessor(exporter)),
      )
      defer provider.Shutdown(ctx)
  }
  ```

  ```java Java (Spring Boot) theme={null}
  # Zero code changes — just pass the endpoint and API key as JVM flags:
  java -javaagent:opentelemetry-javaagent.jar \
    -Dotel.exporter.otlp.endpoint=https://otel.logclaw.ai \
    -Dotel.exporter.otlp.headers=x-logclaw-api-key=lc_proj_your_key_here \
    -Dotel.service.name=my-spring-api \
    -Dotel.logs.exporter=otlp \
    -jar my-app.jar
  # Self-hosted: change endpoint to http://localhost:4318 and remove the headers flag

  # Download the agent:
  # curl -LO https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
  ```

  ```csharp .NET theme={null}
  // dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
  // dotnet add package OpenTelemetry.Extensions.Hosting
  using OpenTelemetry.Logs;
  using OpenTelemetry.Resources;

  var builder = WebApplication.CreateBuilder(args);
  builder.Logging.AddOpenTelemetry(logging =>
  {
      logging.SetResourceBuilder(
          ResourceBuilder.CreateDefault()
              .AddService("my-dotnet-api"));
      logging.AddOtlpExporter(o =>
      {
          o.Endpoint = new Uri("https://otel.logclaw.ai");          // Self-hosted: http://localhost:4318
          o.Headers = "x-logclaw-api-key=lc_proj_your_key_here";   // Self-hosted: remove this line
      });
  });
  // All ILogger calls auto-forward to LogClaw
  ```

  ```ruby Ruby theme={null}
  # gem install opentelemetry-sdk opentelemetry-exporter-otlp
  require 'opentelemetry-sdk'
  require 'opentelemetry/exporter/otlp'

  OpenTelemetry::SDK.configure do |c|
    c.service_name = 'my-rails-app'
    c.add_span_processor(
      OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
        OpenTelemetry::Exporter::OTLP::Exporter.new(
          endpoint: 'https://otel.logclaw.ai',                          # Self-hosted: http://localhost:4318
          headers: { 'x-logclaw-api-key' => 'lc_proj_your_key_here' }, # Self-hosted: remove this line
        )
      )
    )
  end
  ```

  ```rust Rust theme={null}
  // Add to Cargo.toml: opentelemetry, opentelemetry_sdk, opentelemetry-otlp, tracing
  use opentelemetry_otlp::LogExporter;
  use opentelemetry_sdk::logs::LoggerProvider;
  use opentelemetry_sdk::Resource;
  use opentelemetry::KeyValue;

  fn init_logs() -> LoggerProvider {
      let exporter = LogExporter::builder()
          .with_tonic()
          .with_endpoint("https://otel.logclaw.ai")   // Self-hosted: http://localhost:4317
          .with_metadata(metadata_with_api_key())      // Self-hosted: remove this line
          .build()
          .expect("Failed to create log exporter");
      LoggerProvider::builder()
          .with_resource(Resource::new(vec![
              KeyValue::new("service.name", "my-rust-api"),
          ]))
          .with_batch_exporter(exporter)
          .build()
  }
  // Helper for API key header
  fn metadata_with_api_key() -> tonic::metadata::MetadataMap {
      let mut meta = tonic::metadata::MetadataMap::new();
      meta.insert("x-logclaw-api-key", "lc_proj_your_key_here".parse().unwrap());
      meta
  }
  ```
</CodeGroup>

Check your dashboard at [console.logclaw.ai](https://console.logclaw.ai) — the log should appear within seconds.

<Note>
  Every tab above is a complete, runnable example. Scroll down for framework-specific guides (Django, Flask, FastAPI, Express, Next.js, NestJS, Spring Boot, Gin, and more).
</Note>

***

## Detailed Guides by Language

Below are full integration guides with framework-specific examples for each language.

<Note>
  The examples below include the **LogClaw Cloud** endpoint and API key inline — just copy and run. For **self-hosted** deployments, swap `https://otel.logclaw.ai` → `http://localhost:4318` (HTTP) or `localhost:4317` (gRPC) and remove the `x-logclaw-api-key` header. Look for the `# Self-hosted` comments in each snippet.
</Note>

***

## Python

### Install

```bash theme={null}
pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc
```

### Add to your app

```python theme={null}
import logging
from opentelemetry import _logs
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.sdk.resources import Resource

# One-time setup (add to your app startup)
resource = Resource.create({"service.name": "my-python-api"})
provider = LoggerProvider(resource=resource)
provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter(
        endpoint="https://otel.logclaw.ai",                      # Self-hosted: http://localhost:4317
        headers={"x-logclaw-api-key": "lc_proj_your_key_here"},  # Self-hosted: remove this line
    ))
)
_logs.set_logger_provider(provider)

# Attach OTel to Python's standard logging
handler = LoggingHandler(logger_provider=provider)
logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

# Now just use normal Python logging — it auto-sends to LogClaw
logger = logging.getLogger(__name__)
logger.info("User signed up", extra={"user_id": "u-123", "plan": "pro"})
logger.error("Payment failed", extra={"order_id": "ord-456", "error": "card_declined"})
```

That's it. Your existing `logging.info()`, `logging.error()`, etc. all flow to LogClaw automatically.

### Django

```python theme={null}
# settings.py
import logging
from opentelemetry import _logs
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({"service.name": "my-django-app"})
provider = LoggerProvider(resource=resource)
provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter(
        endpoint="https://otel.logclaw.ai",                      # Self-hosted: http://localhost:4317
        headers={"x-logclaw-api-key": "lc_proj_your_key_here"},  # Self-hosted: remove this line
    ))
)
_logs.set_logger_provider(provider)

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {
        "console": {"class": "logging.StreamHandler"},
        "otel": {
            "class": "opentelemetry.sdk._logs.LoggingHandler",
            "level": "INFO",
        },
    },
    "root": {
        "handlers": ["console", "otel"],
        "level": "INFO",
    },
    "loggers": {
        "django": {"handlers": ["console", "otel"], "level": "INFO"},
        "django.request": {"handlers": ["console", "otel"], "level": "WARNING"},
    },
}
```

All Django request logs, middleware errors, and your `logger.info()` calls auto-forward to LogClaw.

### Flask

```python theme={null}
# app.py
from flask import Flask
import logging
from opentelemetry import _logs
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.sdk.resources import Resource

# OTel setup
resource = Resource.create({"service.name": "my-flask-app"})
provider = LoggerProvider(resource=resource)
provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter(
        endpoint="https://otel.logclaw.ai",                      # Self-hosted: http://localhost:4317
        headers={"x-logclaw-api-key": "lc_proj_your_key_here"},  # Self-hosted: remove this line
    ))
)
_logs.set_logger_provider(provider)
logging.getLogger().addHandler(LoggingHandler(logger_provider=provider))

app = Flask(__name__)

@app.route("/api/health")
def health():
    app.logger.info("Health check hit", extra={"endpoint": "/api/health"})
    return {"status": "ok"}

@app.errorhandler(500)
def handle_error(e):
    app.logger.error("Internal error", extra={"error": str(e)})
    return {"error": "internal"}, 500
```

### FastAPI

```python theme={null}
# main.py
from fastapi import FastAPI, Request
import logging
from opentelemetry import _logs
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
from opentelemetry.sdk.resources import Resource

# OTel setup
resource = Resource.create({"service.name": "my-fastapi-app"})
provider = LoggerProvider(resource=resource)
provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter(
        endpoint="https://otel.logclaw.ai",                      # Self-hosted: http://localhost:4317
        headers={"x-logclaw-api-key": "lc_proj_your_key_here"},  # Self-hosted: remove this line
    ))
)
_logs.set_logger_provider(provider)
logging.getLogger().addHandler(LoggingHandler(logger_provider=provider))
logging.getLogger().setLevel(logging.INFO)

logger = logging.getLogger("fastapi-app")
app = FastAPI()

@app.middleware("http")
async def log_requests(request: Request, call_next):
    logger.info("Request", extra={
        "method": request.method,
        "path": request.url.path,
        "client": request.client.host,
    })
    response = await call_next(request)
    return response

@app.get("/api/users/{user_id}")
async def get_user(user_id: str):
    logger.info("Fetching user", extra={"user_id": user_id})
    return {"user_id": user_id}
```

### Celery (Background Workers)

```python theme={null}
# celery_app.py — logs from async tasks also flow to LogClaw
from celery import Celery
from celery.signals import setup_logging
import logging

@setup_logging.connect
def configure_logging(**kwargs):
    from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
    from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
    from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
    provider = LoggerProvider()
    provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter(
        endpoint="https://otel.logclaw.ai",                      # Self-hosted: http://localhost:4317
        headers={"x-logclaw-api-key": "lc_proj_your_key_here"},  # Self-hosted: remove
    )))
    logging.getLogger().addHandler(LoggingHandler(logger_provider=provider))

app = Celery("tasks", broker="redis://localhost:6379")

@app.task
def process_payment(order_id):
    logger = logging.getLogger("celery-worker")
    logger.info("Processing payment", extra={"order_id": order_id})
```

***

## Node.js

### Install

```bash theme={null}
npm install @opentelemetry/sdk-node @opentelemetry/sdk-logs \
  @opentelemetry/exporter-logs-otlp-grpc @opentelemetry/resources \
  @opentelemetry/semantic-conventions
```

### Express

```javascript theme={null}
// instrumentation.js — require BEFORE your app: node -r ./instrumentation.js app.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { LoggerProvider, BatchLogRecordProcessor } = require('@opentelemetry/sdk-logs');
const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');

const resource = new Resource({ 'service.name': 'my-express-api' });
const loggerProvider = new LoggerProvider({ resource });
loggerProvider.addLogRecordProcessor(
  new BatchLogRecordProcessor(new OTLPLogExporter({
    url: 'https://otel.logclaw.ai',                                    // Self-hosted: http://localhost:4317
    headers: { 'x-logclaw-api-key': 'lc_proj_your_key_here' },        // Self-hosted: remove this line
  }))
);

const logger = loggerProvider.getLogger('app');

// Helper — use throughout your app
global.otelLog = (level, message, attrs = {}) => {
  logger.emit({ severityText: level, body: message, attributes: attrs });
};
```

```javascript theme={null}
// app.js
const express = require('express');
const app = express();

// Request logging middleware
app.use((req, res, next) => {
  otelLog('INFO', `${req.method} ${req.path}`, {
    'http.method': req.method,
    'http.route': req.path,
    'http.client_ip': req.ip,
  });
  next();
});

app.post('/api/orders', (req, res) => {
  otelLog('INFO', 'Order created', { orderId: 'ord-789' });
  res.json({ status: 'created' });
});

app.use((err, req, res, next) => {
  otelLog('ERROR', err.message, { 'error.stack': err.stack });
  res.status(500).json({ error: 'Internal server error' });
});

app.listen(3000);
```

### Next.js

```javascript theme={null}
// instrumentation.ts (Next.js 13+ — auto-loaded by Next.js)
export async function register() {
  if (process.env.NEXT_RUNTIME === 'nodejs') {
    const { LoggerProvider, BatchLogRecordProcessor } = await import('@opentelemetry/sdk-logs');
    const { OTLPLogExporter } = await import('@opentelemetry/exporter-logs-otlp-grpc');
    const { Resource } = await import('@opentelemetry/resources');

    const resource = new Resource({ 'service.name': 'my-nextjs-app' });
    const loggerProvider = new LoggerProvider({ resource });
    loggerProvider.addLogRecordProcessor(
      new BatchLogRecordProcessor(new OTLPLogExporter({
        url: 'https://otel.logclaw.ai',                                    // Self-hosted: http://localhost:4317
        headers: { 'x-logclaw-api-key': 'lc_proj_your_key_here' },        // Self-hosted: remove
      }))
    );

    // Available in API routes and server components
    globalThis.__otelLoggerProvider = loggerProvider;
  }
}
```

```typescript theme={null}
// app/api/users/route.ts — Next.js App Router API route
import { NextResponse } from 'next/server';

function log(level: string, message: string, attrs: Record<string, string> = {}) {
  const provider = (globalThis as any).__otelLoggerProvider;
  if (provider) {
    provider.getLogger('app').emit({ severityText: level, body: message, attributes: attrs });
  }
}

export async function GET() {
  log('INFO', 'Users API called');
  return NextResponse.json({ users: [] });
}
```

Or use the environment variable approach with `next.config.js`:

```javascript theme={null}
// next.config.js
module.exports = {
  experimental: {
    instrumentationHook: true,  // enables instrumentation.ts
  },
};
```

### NestJS

```bash theme={null}
npm install @opentelemetry/sdk-logs @opentelemetry/exporter-logs-otlp-grpc \
  @opentelemetry/resources nestjs-otel
```

```typescript theme={null}
// src/otel.ts — load BEFORE NestJS bootstrap
import { LoggerProvider, BatchLogRecordProcessor } from '@opentelemetry/sdk-logs';
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-grpc';
import { Resource } from '@opentelemetry/resources';

const resource = new Resource({ 'service.name': 'my-nestjs-api' });
const loggerProvider = new LoggerProvider({ resource });
loggerProvider.addLogRecordProcessor(
  new BatchLogRecordProcessor(new OTLPLogExporter({
    url: 'https://otel.logclaw.ai',                                    // Self-hosted: http://localhost:4317
    headers: { 'x-logclaw-api-key': 'lc_proj_your_key_here' },        // Self-hosted: remove this line
  }))
);

export { loggerProvider };
```

```typescript theme={null}
// src/main.ts
import './otel';  // MUST be first import
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.listen(3000);
}
bootstrap();
```

```typescript theme={null}
// src/orders/orders.controller.ts — use NestJS Logger as normal
import { Controller, Get, Logger, Post, Body } from '@nestjs/common';

@Controller('orders')
export class OrdersController {
  private readonly logger = new Logger(OrdersController.name);

  @Post()
  create(@Body() createOrderDto: any) {
    this.logger.log(`Order created: ${createOrderDto.id}`);
    return { status: 'created' };
  }

  @Get()
  findAll() {
    this.logger.log('Fetching all orders');
    return [];
  }
}
```

### Fastify

```javascript theme={null}
// server.js
const fastify = require('fastify')({ logger: true });  // Pino logger built-in

// With OTel auto-instrumentation for Pino:
// npm install @opentelemetry/instrumentation-pino
// node -r ./instrumentation.js server.js

fastify.get('/api/health', async (request, reply) => {
  request.log.info({ endpoint: '/api/health' }, 'Health check');
  return { status: 'ok' };
});

fastify.post('/api/orders', async (request, reply) => {
  request.log.info({ orderId: request.body.id }, 'Order created');
  return { status: 'created' };
});

fastify.listen({ port: 3000 });
```

### With Winston

```javascript theme={null}
// logger.js
const winston = require('winston');

// Winston logs auto-forward to LogClaw with OTel instrumentation
// npm install @opentelemetry/instrumentation-winston
const logger = winston.createLogger({
  level: 'info',
  format: winston.format.json(),
  transports: [new winston.transports.Console()],
});

module.exports = logger;
```

```bash theme={null}
# Run with auto-instrumentation — zero code changes to your Winston setup
node -r ./instrumentation.js app.js
```

### With Pino

```javascript theme={null}
// Same approach — Pino auto-instrumented via OTel
// npm install @opentelemetry/instrumentation-pino
const pino = require('pino');
const logger = pino();

logger.info({ userId: 'usr-42' }, 'User logged in');
```

***

## Java / Kotlin

### Spring Boot (Zero Code Changes)

Download the OTel Java Agent — it auto-instruments your app:

```bash theme={null}
# Download the agent JAR
curl -LO https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

# Run your Spring Boot app with the agent — endpoint and API key passed as JVM flags:
java -javaagent:opentelemetry-javaagent.jar \
  -Dotel.exporter.otlp.endpoint=https://otel.logclaw.ai \
  -Dotel.exporter.otlp.headers=x-logclaw-api-key=lc_proj_your_key_here \
  -Dotel.service.name=my-spring-api \
  -Dotel.logs.exporter=otlp \
  -jar my-app.jar
# Self-hosted: change endpoint to http://localhost:4318 and remove the headers flag
```

**Zero code changes.** All your `log.info()`, `log.error()`, Spring request logs, and exception stack traces automatically flow to LogClaw.

```java theme={null}
// Your existing Spring Boot code — no changes needed
@RestController
public class OrderController {
    private static final Logger log = LoggerFactory.getLogger(OrderController.class);

    @PostMapping("/api/orders")
    public Order createOrder(@RequestBody OrderRequest req) {
        log.info("Creating order for user={}", req.getUserId());  // → auto-sent to LogClaw
        return orderService.create(req);
    }
}
```

### Spring Boot via application.properties (Alternative)

```properties theme={null}
# application.properties
otel.exporter.otlp.endpoint=https://otel.logclaw.ai
otel.exporter.otlp.headers=x-logclaw-api-key=lc_proj_your_key_here
otel.service.name=my-spring-api
otel.logs.exporter=otlp
# Self-hosted: change endpoint to http://localhost:4318 and remove the headers line
```

```xml theme={null}
<!-- pom.xml — add the OTel Spring Boot starter -->
<dependency>
  <groupId>io.opentelemetry.instrumentation</groupId>
  <artifactId>opentelemetry-spring-boot-starter</artifactId>
  <version>2.10.0</version>
</dependency>
```

### Quarkus

```properties theme={null}
# application.properties
quarkus.otel.exporter.otlp.endpoint=https://otel.logclaw.ai
quarkus.otel.exporter.otlp.headers=x-logclaw-api-key=lc_proj_your_key_here
quarkus.otel.resource.attributes=service.name=my-quarkus-api
quarkus.otel.logs.exporter=otlp
# Self-hosted: change endpoint to http://localhost:4318 and remove the headers line
```

```xml theme={null}
<!-- pom.xml -->
<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-opentelemetry</artifactId>
</dependency>
```

### Micronaut

```yaml theme={null}
# application.yml
otel:
  exporter:
    otlp:
      endpoint: https://otel.logclaw.ai                            # Self-hosted: http://localhost:4318
      headers:
        x-logclaw-api-key: lc_proj_your_key_here                   # Self-hosted: remove this line
  resource:
    attributes:
      service.name: my-micronaut-api
```

### Kotlin (Ktor)

```kotlin theme={null}
// Application.kt
import io.ktor.server.application.*
import io.ktor.server.engine.*
import io.ktor.server.netty.*
import org.slf4j.LoggerFactory

fun main() {
    // Run with: java -javaagent:opentelemetry-javaagent.jar
    //   -Dotel.exporter.otlp.endpoint=https://otel.logclaw.ai
    //   -Dotel.exporter.otlp.headers=x-logclaw-api-key=lc_proj_your_key_here -jar app.jar
    embeddedServer(Netty, port = 8080) {
        val logger = LoggerFactory.getLogger("ktor-app")

        routing {
            post("/api/orders") {
                logger.info("Order created")  // → auto-sent to LogClaw via agent
                call.respondText("created")
            }
        }
    }.start(wait = true)
}
```

### Log4j2 / Logback

Add the OTel appender to your logging config:

```xml theme={null}
<!-- log4j2.xml -->
<Configuration>
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <PatternLayout pattern="%d %p %c - %m%n"/>
    </Console>
    <OpenTelemetry name="OTel"/>
  </Appenders>
  <Loggers>
    <Root level="info">
      <AppenderRef ref="Console"/>
      <AppenderRef ref="OTel"/>
    </Root>
  </Loggers>
</Configuration>
```

***

## Go

```bash theme={null}
go get go.opentelemetry.io/otel/sdk/log
go get go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc
```

```go theme={null}
package main

import (
    "context"
    "log/slog"

    "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
    "go.opentelemetry.io/otel/sdk/log"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func main() {
    ctx := context.Background()

    exporter, _ := otlploggrpc.New(ctx,
        otlploggrpc.WithEndpointURL("otel.logclaw.ai:443"),                                        // Self-hosted: localhost:4317
        otlploggrpc.WithHeaders(map[string]string{"x-logclaw-api-key": "lc_proj_your_key_here"}),  // Self-hosted: remove
    )

    res, _ := resource.New(ctx,
        resource.WithAttributes(semconv.ServiceName("my-go-api")),
    )

    provider := log.NewLoggerProvider(
        log.WithResource(res),
        log.WithProcessor(log.NewBatchProcessor(exporter)),
    )
    defer provider.Shutdown(ctx)

    // Use Go's standard slog with OTel backend
    logger := slog.New(slog.NewJSONHandler(nil, nil))
    logger.Info("Server started", "port", 8080)
    logger.Error("Database connection failed", "host", "db.internal", "err", "timeout")
}
```

### Gin

```go theme={null}
package main

import (
    "log/slog"
    "github.com/gin-gonic/gin"
    "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
)

func main() {
    // initOtel() — same setup as above
    r := gin.Default()
    r.Use(otelgin.Middleware("my-gin-api"))  // auto-instruments all routes

    r.POST("/api/orders", func(c *gin.Context) {
        slog.Info("Order created", "orderId", c.Param("id"))
        c.JSON(200, gin.H{"status": "created"})
    })

    r.Run(":8080")
}
```

```bash theme={null}
go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin
```

### Fiber

```go theme={null}
package main

import (
    "log/slog"
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/contrib/otelfiber"
)

func main() {
    app := fiber.New()
    app.Use(otelfiber.Middleware())  // auto-instruments all routes

    app.Post("/api/orders", func(c *fiber.Ctx) error {
        slog.Info("Order created", "path", c.Path())
        return c.JSON(fiber.Map{"status": "created"})
    })

    app.Listen(":8080")
}
```

### Echo

```go theme={null}
package main

import (
    "log/slog"
    "github.com/labstack/echo/v4"
    "go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho"
)

func main() {
    e := echo.New()
    e.Use(otelecho.Middleware("my-echo-api"))

    e.POST("/api/orders", func(c echo.Context) error {
        slog.Info("Order created")
        return c.JSON(200, map[string]string{"status": "created"})
    })

    e.Start(":8080")
}
```

***

## Ruby

### Install

```bash theme={null}
gem install opentelemetry-sdk opentelemetry-exporter-otlp \
  opentelemetry-logs-sdk opentelemetry-logs-exporter-otlp
```

### Add to your app

```ruby theme={null}
# config/initializers/otel.rb (Rails) or top of your app
require 'opentelemetry-sdk'
require 'opentelemetry/exporter/otlp'

OpenTelemetry::SDK.configure do |c|
  c.service_name = 'my-ruby-api'
  c.add_span_processor(
    OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
      OpenTelemetry::Exporter::OTLP::Exporter.new(
        endpoint: 'https://otel.logclaw.ai',                          # Self-hosted: http://localhost:4318
        headers: { 'x-logclaw-api-key' => 'lc_proj_your_key_here' }, # Self-hosted: remove this line
      )
    )
  )
end

# Use with Rails logger
Rails.logger.info("Order created", order_id: "ord-123")
```

### Rails Auto-Instrumentation (Zero Code Changes)

```bash theme={null}
gem install opentelemetry-instrumentation-all
```

```ruby theme={null}
# config/initializers/otel.rb
require 'opentelemetry/instrumentation/all'
OpenTelemetry::SDK.configure do |c|
  c.service_name = 'my-rails-app'
  c.use_all  # Auto-instruments Rails, ActiveRecord, Rack, Sidekiq, etc.
end
```

Then run your app:

```bash theme={null}
rails server
```

***

## .NET / C\#

### Install

```bash theme={null}
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Extensions.Hosting
```

### ASP.NET Core

```csharp theme={null}
// Program.cs
using OpenTelemetry.Logs;
using OpenTelemetry.Resources;

var builder = WebApplication.CreateBuilder(args);

builder.Logging.AddOpenTelemetry(logging =>
{
    logging.SetResourceBuilder(
        ResourceBuilder.CreateDefault()
            .AddService("my-dotnet-api"));

    logging.AddOtlpExporter(o =>
    {
        o.Endpoint = new Uri("https://otel.logclaw.ai");          // Self-hosted: http://localhost:4318
        o.Headers = "x-logclaw-api-key=lc_proj_your_key_here";   // Self-hosted: remove this line
    });
});

var app = builder.Build();

// Your existing ILogger<T> calls auto-forward to LogClaw
app.MapGet("/", (ILogger<Program> logger) =>
{
    logger.LogInformation("Request received from {UserId}", "usr-42");
    return "Hello World";
});

app.Run();
```

All existing `ILogger` calls (`LogInformation`, `LogError`, `LogWarning`) automatically flow to LogClaw.

***

## Rust

### Add to Cargo.toml

```toml theme={null}
[dependencies]
opentelemetry = "0.24"
opentelemetry_sdk = { version = "0.24", features = ["logs"] }
opentelemetry-otlp = { version = "0.17", features = ["logs"] }
tracing = "0.1"
tracing-opentelemetry = "0.25"
tracing-subscriber = "0.3"
```

### Add to main.rs

```rust theme={null}
use opentelemetry_otlp::LogExporter;
use opentelemetry_sdk::logs::LoggerProvider;
use opentelemetry_sdk::Resource;
use opentelemetry::KeyValue;

fn init_logs() -> LoggerProvider {
    let exporter = LogExporter::builder()
        .with_tonic()
        .with_endpoint("https://otel.logclaw.ai")   // Self-hosted: http://localhost:4317
        .with_metadata(metadata_with_api_key())      // Self-hosted: remove this line
        .build()
        .expect("Failed to create log exporter");

    LoggerProvider::builder()
        .with_resource(Resource::new(vec![
            KeyValue::new("service.name", "my-rust-api"),
        ]))
        .with_batch_exporter(exporter)
        .build()
}

// Helper for API key header
fn metadata_with_api_key() -> tonic::metadata::MetadataMap {
    let mut meta = tonic::metadata::MetadataMap::new();
    meta.insert("x-logclaw-api-key", "lc_proj_your_key_here".parse().unwrap());
    meta
}

fn main() {
    let _provider = init_logs();

    // Use tracing macros — they auto-forward to LogClaw
    tracing::info!(user_id = "usr-42", "Request processed");
    tracing::error!(err = "timeout", "Database connection failed");
}
```

***

## PHP

### Install

```bash theme={null}
composer require open-telemetry/sdk \
  open-telemetry/exporter-otlp \
  open-telemetry/transport-grpc
```

### Add to your app

```php theme={null}
<?php
// bootstrap.php or index.php
use OpenTelemetry\SDK\Logs\LoggerProviderBuilder;
use OpenTelemetry\SDK\Resource\ResourceInfoFactory;
use OpenTelemetry\Contrib\Otlp\LogsExporter;
use OpenTelemetry\SDK\Logs\Processor\BatchLogRecordProcessor;

$resource = ResourceInfoFactory::defaultResource()->merge(
    ResourceInfo::create(Attributes::create([
        'service.name' => 'my-php-api',
    ]))
);

$exporter = new LogsExporter(
    (new \OpenTelemetry\Contrib\Otlp\OtlpHttpTransportFactory())->create(
        'https://otel.logclaw.ai/v1/logs',      // Self-hosted: http://localhost:4318/v1/logs
        'application/json',
        ['x-logclaw-api-key' => 'lc_proj_your_key_here']  // Self-hosted: remove this line
    )
);

$loggerProvider = (new LoggerProviderBuilder())
    ->setResource($resource)
    ->addLogRecordProcessor(new BatchLogRecordProcessor($exporter))
    ->build();

$logger = $loggerProvider->getLogger('app');
$logger->emit(
    (new LogRecord('User login failed'))
        ->setSeverityText('ERROR')
        ->setAttribute('user.email', 'john@example.com')
);
```

### Laravel

```php theme={null}
// config/logging.php — add OTel as a log channel
'channels' => [
    'otel' => [
        'driver' => 'monolog',
        'handler' => \Your\OtelHandler::class,
    ],
    'stack' => [
        'driver' => 'stack',
        'channels' => ['single', 'otel'],
    ],
],
```

Then run your app:

```bash theme={null}
php artisan serve
```

***

## Log Shippers (Config Change Only)

Already running a log shipper? Point it at LogClaw's OTLP endpoint.

<Note>
  Log shippers use **config files**, not OTel SDKs. The examples below use the **LogClaw Cloud** URL (`https://otel.logclaw.ai`) — for self-hosted, replace with `http://localhost:4318`.
</Note>

### Fluentd

```xml theme={null}
<!-- fluent.conf -->
<match **>
  @type opentelemetry
  endpoint https://otel.logclaw.ai/v1/logs
  <buffer>
    flush_interval 5s
  </buffer>
</match>
```

```bash theme={null}
# Install the OTLP output plugin
fluent-gem install fluent-plugin-opentelemetry
```

### Fluent Bit

```ini theme={null}
# fluent-bit.conf
[OUTPUT]
    Name              opentelemetry
    Match             *
    Host              otel.logclaw.ai
    Port              443
    Tls               On
    Log_response_payload  True
    Logs_uri          /v1/logs
    add_label         service.name my-app
```

### Vector

```toml theme={null}
# vector.toml
[sinks.logclaw]
type = "http"
inputs = ["your_source"]
uri = "https://otel.logclaw.ai/v1/logs"
encoding.codec = "json"
method = "post"
headers.content-type = "application/json"

# Or use the native OTLP sink (Vector 0.38+)
# [sinks.logclaw]
# type = "opentelemetry"
# inputs = ["your_source"]
# endpoint = "https://otel.logclaw.ai"
```

### Logstash

```ruby theme={null}
# logstash.conf
output {
  http {
    url => "https://otel.logclaw.ai/v1/logs"
    http_method => "post"
    format => "json"
    content_type => "application/json"
  }
}
```

```bash theme={null}
# Or install the OpenTelemetry output plugin
bin/logstash-plugin install logstash-output-opentelemetry
```

### Datadog Agent (Dual-Ship)

Already using Datadog? Send a copy to LogClaw:

```yaml theme={null}
# datadog.yaml
otlp_config:
  logs:
    enabled: true
  receiver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
```

Then point a separate OTel Collector at your Datadog Agent's OTLP receiver and forward to LogClaw.

***

## OTel Collector Sidecar (Zero Code Changes)

For apps that write logs to **stdout** or **log files** — any language, any framework — deploy an OTel Collector alongside your app to auto-scrape and forward logs. **No SDK needed.**

### Docker Compose example

```yaml theme={null}
# docker-compose.yml
services:
  my-app:
    image: my-app:latest
    # Your app just logs to stdout as normal

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.114.0
    volumes:
      - ./otel-config.yaml:/etc/otelcol/config.yaml
    depends_on:
      - my-app
```

```yaml theme={null}
# otel-config.yaml
receivers:
  filelog:
    include: [/var/log/app/*.log]
    start_at: beginning
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  resource:
    attributes:
      - key: service.name
        value: "my-app"
        action: upsert
  batch:
    send_batch_size: 500
    timeout: 5s

exporters:
  otlphttp:
    endpoint: https://otel.logclaw.ai             # Self-hosted: http://localhost:4318
    headers:
      x-logclaw-api-key: lc_proj_your_key_here    # Self-hosted: remove this line

service:
  pipelines:
    logs:
      receivers: [filelog, otlp]
      processors: [resource, batch]
      exporters: [otlphttp]
```

### Kubernetes DaemonSet

For Kubernetes clusters, deploy the OTel Collector as a DaemonSet to auto-collect logs from all pods:

```yaml theme={null}
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: collector
          image: otel/opentelemetry-collector-contrib:0.114.0
          env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "https://otel.logclaw.ai"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "service.name=k8s-cluster"
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
```

***

## Dashboard Upload (Drag & Drop)

For testing or one-time imports, the LogClaw dashboard supports drag-and-drop log upload:

1. Go to [console.logclaw.ai](https://console.logclaw.ai)
2. Click **Ingestion** in the sidebar
3. Drag a `.json`, `.ndjson`, `.csv`, or `.txt`/`.log` file onto the upload area

Accepted format (OTLP JSON):

```json theme={null}
{
  "resourceLogs": [{
    "resource": {
      "attributes": [
        {"key": "service.name", "value": {"stringValue": "imported-logs"}}
      ]
    },
    "scopeLogs": [{
      "logRecords": [
        {
          "timeUnixNano": "1709500000000000000",
          "severityText": "ERROR",
          "body": {"stringValue": "Something went wrong"}
        }
      ]
    }]
  }]
}
```

***

## What Happens After Logs Arrive

```
Your App → OTLP → OTel Collector → Kafka → Flink ETL → OpenSearch
                                         → Flink Enrichment → Feature Store
                                         → Flink Anomaly Scorer → Ticketing Agent
                                                                   → PagerDuty / Jira / Slack
```

1. **OTel Collector** receives logs, batches them (1000/batch), and writes to Kafka
2. **Flink ETL** normalizes, enriches, and indexes into OpenSearch
3. **Flink Anomaly Scorer** detects anomalies in real-time using ML models
4. **Ticketing Agent** auto-creates incidents in PagerDuty/Jira/Slack when anomalies exceed threshold
5. **Dashboard** lets you search, filter, and visualize everything

***

## Supported Languages & Frameworks

Every option has a full code example above. Pick your stack:

| Language / Tool    | Frameworks with Examples                               | Code Changes                     |
| ------------------ | ------------------------------------------------------ | -------------------------------- |
| **cURL / HTTP**    | Any HTTP client                                        | None — just POST JSON            |
| **Python**         | Django, Flask, FastAPI, Celery                         | 5 lines in startup               |
| **Node.js**        | Express, Next.js, NestJS, Fastify, Winston, Pino       | 5 lines in startup               |
| **Java / Kotlin**  | Spring Boot, Quarkus, Micronaut, Ktor, Log4j2, Logback | **Zero** — just a JVM flag       |
| **Go**             | Gin, Fiber, Echo, net/http                             | 5 lines in main                  |
| **Ruby**           | Rails (auto-instrumentation), Sinatra                  | 5 lines or **zero** with env var |
| **.NET / C#**      | ASP.NET Core, Minimal API                              | Add to `Program.cs`              |
| **Rust**           | Actix, Axum, Rocket (via tracing)                      | 5 lines in main                  |
| **PHP**            | Laravel, Symfony                                       | Add to bootstrap                 |
| **Log Shippers**   | Fluentd, Fluent Bit, Vector, Logstash, Datadog Agent   | Config change only               |
| **OTel Collector** | stdout / log files via sidecar                         | **Zero** — collector scrapes     |
| **Dashboard**      | Drag & drop JSON, NDJSON, CSV, or text files           | **Zero** — drag a file           |

***

## Troubleshooting

**Logs not appearing in dashboard?**

1. Test connectivity:

   <Tabs>
     <Tab title="LogClaw Cloud">
       ```bash theme={null}
       curl -v https://otel.logclaw.ai/v1/logs \
         -H "Content-Type: application/json" \
         -H "x-logclaw-api-key: lc_proj_your_key_here" \
         -d '{"resourceLogs":[]}'
       # Should return 200 with {"accepted":true}
       ```

       If you get `401 Unauthorized`, check that your API key is valid and not revoked.
     </Tab>

     <Tab title="Self-Hosted">
       ```bash theme={null}
       curl -v http://localhost:4318/v1/logs -d '{}' -H "Content-Type: application/json"
       # Should return 200 (even with empty body)
       ```
     </Tab>
   </Tabs>

2. Check your `service.name` attribute is set — logs without `service.name` may not display correctly

3. Verify OTLP endpoint URL — common mistakes:
   * Missing `/v1/logs` path for HTTP (gRPC doesn't need it)
   * Using port 4317 for HTTP (that's gRPC) or 4318 for gRPC (that's HTTP)
   * Using `http://` instead of `https://` for external access
   * **LogClaw Cloud:** using `localhost` instead of `otel.logclaw.ai`

4. Check firewall — OTLP uses port 443 (TLS-terminated) for external access

5. **LogClaw Cloud:** missing `x-logclaw-api-key` header — all managed platform requests require an API key. See [API Keys](/api-keys)


# Values Reference
Source: https://docs.logclaw.ai/values-reference

Full configuration reference for all LogClaw Helm chart values.

# Values Reference

This document describes all top-level `global.*` values accepted by the LogClaw umbrella
chart (`logclaw-tenant`) and propagated to every sub-chart.

## global.\*

| Field                               | Type   | Required | Default                             | Description                                                                                                                                                                                                      |
| ----------------------------------- | ------ | -------- | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `global.tenantId`                   | string | Yes      | —                                   | Unique tenant identifier. Lowercase letters, numbers, hyphens only. Max 40 characters. Used as a label, resource name suffix, and namespace component.                                                           |
| `global.tenantName`                 | string | Yes      | —                                   | Human-readable display name for the tenant. Shown in dashboards and alert messages.                                                                                                                              |
| `global.storageClass`               | string | Yes      | —                                   | Name of the Kubernetes StorageClass used for all persistent volumes (Kafka, OpenSearch, Airflow). Must exist in the cluster before install.                                                                      |
| `global.storageClassHighThroughput` | string | No       | `""` (falls back to `storageClass`) | StorageClass optimised for high IOPS. Used for Kafka broker disks and OpenSearch data nodes when set. Recommended: `io2` on AWS, `pd-ssd` on GCP.                                                                |
| `global.tier`                       | string | No       | `"ha"`                              | Deployment tier controlling replica counts and resource profiles. One of: `standard` (single replicas, minimal resources), `ha` (3 replicas, production resources), `ultra-ha` (5+ replicas, maximum resources). |
| `global.objectStorage.provider`     | string | No       | `"s3"`                              | Object storage backend. One of: `s3`, `gcs`, `azure`. Determines the credential format and SDK used by Flink checkpointing and ML engine model storage.                                                          |
| `global.objectStorage.bucket`       | string | Yes      | —                                   | Name of the pre-created object storage bucket. Must be accessible from the cluster with appropriate IAM/service-account permissions.                                                                             |
| `global.objectStorage.region`       | string | No       | `"us-east-1"`                       | Cloud region of the bucket. Required for AWS S3 and GCS. Ignored for Azure and self-hosted S3.                                                                                                                   |
| `global.objectStorage.endpoint`     | string | No       | `""`                                | Custom S3-compatible endpoint URL. Set for MinIO (`http://minio.minio.svc:9000`) or other self-hosted S3. Leave empty for AWS S3.                                                                                |
| `global.secretStore.provider`       | string | No       | `"aws"`                             | Secret management backend for External Secrets Operator. One of: `aws`, `gcp`, `vault`, `azure`. Determines the ESO ClusterSecretStore type created for the tenant.                                              |
| `global.secretStore.region`         | string | No       | `"us-east-1"`                       | AWS region for Secrets Manager. Used only when `provider=aws`.                                                                                                                                                   |
| `global.secretStore.projectId`      | string | No       | `""`                                | GCP project ID. Used only when `provider=gcp`.                                                                                                                                                                   |
| `global.secretStore.vaultAddress`   | string | No       | `""`                                | HashiCorp Vault server address (e.g. `https://vault.example.com:8200`). Used only when `provider=vault`.                                                                                                         |
| `global.imagePullSecrets`           | list   | No       | `[]`                                | List of Kubernetes image pull secret names (`[{name: "my-secret"}]`). Applied to all workload pods across all sub-charts.                                                                                        |
| `global.monitoring.enabled`         | bool   | No       | `true`                              | When `true`, enables ServiceMonitor and PrometheusRule resources for all sub-charts. Requires Prometheus Operator CRDs to be present.                                                                            |
| `global.clusterDomain`              | string | No       | `"cluster.local"`                   | Kubernetes cluster DNS domain. Used to construct internal service FQDNs. Change only if your cluster uses a custom domain.                                                                                       |

## Component Enable Flags

These top-level boolean maps control which sub-charts are rendered by the umbrella chart.

| Field                    | Type | Required | Default | Description                                                                                                                                                                                                                                                           |
| ------------------------ | ---- | -------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `platform.enabled`       | bool | No       | `true`  | Deploy `logclaw-platform` (API gateway, RBAC, NetworkPolicy). Disabling also disables the ingress entry point.                                                                                                                                                        |
| `otelCollector.enabled`  | bool | No       | `true`  | Deploy `logclaw-otel-collector` (OpenTelemetry Collector). OTLP-native log ingestion gateway accepting gRPC (:4317) and HTTP (:4318). Disable if you use a separate log shipping solution.                                                                            |
| `kafka.enabled`          | bool | No       | `true`  | Deploy `logclaw-kafka` (Strimzi Kafka cluster). Required by `flink` and `ticketingAgent`.                                                                                                                                                                             |
| `flink.enabled`          | bool | No       | `true`  | Deploy `logclaw-flink` (stream processing jobs). Requires `kafka.enabled=true`.                                                                                                                                                                                       |
| `opensearch.enabled`     | bool | No       | `true`  | Deploy `logclaw-opensearch` (search cluster + dashboards). Required by `ticketingAgent`.                                                                                                                                                                              |
| `mlEngine.enabled`       | bool | No       | `true`  | Deploy `logclaw-ml-engine` (KServe InferenceService). Requires GPU nodes or CPU-only inference image override.                                                                                                                                                        |
| `airflow.enabled`        | bool | No       | `true`  | Deploy `logclaw-airflow` (pipeline orchestration).                                                                                                                                                                                                                    |
| `ticketingAgent.enabled` | bool | No       | `true`  | Deploy `logclaw-ticketing-agent` (AI SRE agent). Requires `kafka.enabled=true` and `opensearch.enabled=true`.                                                                                                                                                         |
| `bridge.enabled`         | bool | No       | `false` | Deploy `logclaw-bridge` (OTLP ETL + trace correlation engine). Flattens OTLP JSON from Kafka `raw-logs` into canonical log documents, performs anomaly detection, trace correlation, and OpenSearch indexing. Dev/demo alternative to Flink. Can run alongside Flink. |
| `dashboard.enabled`      | bool | No       | `false` | Deploy `logclaw-dashboard` (Next.js pipeline UI). Provides log ingestion drag-and-drop, incident management, anomaly visualization, and real-time pipeline monitoring.                                                                                                |
| `agent.enabled`          | bool | No       | `false` | Deploy `logclaw-agent` (Go infrastructure health collector). Collects Kafka lag, Flink job status, OpenSearch health, and ESO sync status. Feeds the Dashboard pipeline view.                                                                                         |

## Per-Chart Override Syntax

Each sub-chart can be overridden using its chart name as the top-level key:

```yaml theme={null}
logclaw-kafka:
  kafka:
    replicas: 5
    storage:
      volumes:
        - id: 0
          type: persistent-claim
          size: "2Ti"
          deleteClaim: false

logclaw-opensearch:
  opensearch:
    data:
      replicas: 5
      diskSize: "2Ti"
    masters:
      replicas: 3

logclaw-otel-collector:
  replicaCount: 10
  autoscaling:
    enabled: true
    minReplicas: 10
    maxReplicas: 50
    targetCPUUtilizationPercentage: 60

logclaw-ticketing-agent:
  config:
    pagerduty:
      enabled: true
    jira:
      enabled: true
      baseUrl: "https://yourorg.atlassian.net"
      projectKey: "SRE"
    servicenow:
      enabled: true
      instance: "yourorg"
    anomaly:
      minimumScore: 0.85
      lookbackWindow: "15m"

logclaw-ml-engine:
  model:
    name: "logclaw-anomaly-v2"
    storageUri: "s3://your-bucket/models/anomaly-v2"
  resources:
    requests:
      cpu: "2"
      memory: "4Gi"
    limits:
      nvidia.com/gpu: "1"

logclaw-airflow:
  airflow:
    dags:
      gitSync:
        enabled: true
        repo: "git@github.com:yourorg/logclaw-dags.git"
        branch: "main"
        sshKeySecret: "airflow-git-ssh-key"

logclaw-bridge:
  bridge:
    kafkaBrokers: "logclaw-kafka-kafka-bootstrap:9093"
    opensearchEndpoint: "https://logclaw-opensearch:9200"
    anomalyThreshold: 2.5
    windowSize: 50

logclaw-dashboard:
  image:
    repository: "ghcr.io/logclaw/dashboard"
    tag: "latest"
  service:
    type: ClusterIP           # ClusterIP (default) or LoadBalancer for external access
    port: 3000
```

## Tier Profiles

| Setting                       | standard | ha         | ultra-ha        |
| ----------------------------- | -------- | ---------- | --------------- |
| Kafka brokers                 | 1        | 3          | 5               |
| Kafka storage per broker      | 100Gi    | 1Ti        | 2Ti             |
| OpenSearch masters            | 1        | 3          | 3               |
| OpenSearch data nodes         | 1        | 3          | 5               |
| OpenSearch disk per data node | 100Gi    | 1Ti        | 2Ti             |
| OTel Collector replicas       | 1        | 3          | 5               |
| Flink task managers           | 1        | 2          | 4               |
| Ticketing agent replicas      | 1        | 2          | 3               |
| ML Engine replicas            | 1        | 2          | 3               |
| PodDisruptionBudget           | No       | Yes        | Yes             |
| TopologySpreadConstraints     | No       | Yes (zone) | Yes (zone+node) |

## Environment Variable Substitution (Helmfile)

When using `helmfile`, the following environment variables are consumed by `helmfile.d/` releases:

| Variable                  | Used By                                               | Default                              | Description                                                   |
| ------------------------- | ----------------------------------------------------- | ------------------------------------ | ------------------------------------------------------------- |
| `TENANT_ID`               | all helmfile.d files                                  | `dev-local`                          | Tenant identifier, injected into release names and namespaces |
| `STORAGE_CLASS`           | `10-platform`, `20-kafka`, `40-opensearch`            | `standard`                           | Overrides `global.storageClass`                               |
| `KAFKA_BROKERS`           | `30-otel-collector`, `50-flink`, `80-ticketing-agent` | `logclaw-kafka-kafka-bootstrap:9093` | Kafka bootstrap server address                                |
| `OBJECT_STORAGE_BUCKET`   | `50-flink`, `60-ml-engine`                            | `logclaw-dev-local`                  | Object storage bucket name                                    |
| `OBJECT_STORAGE_PROVIDER` | `50-flink`                                            | `s3`                                 | Object storage provider                                       |
| `OPENSEARCH_ENDPOINT`     | `80-ticketing-agent`                                  | `https://logclaw-opensearch:9200`    | OpenSearch base URL                                           |
| `HELM_REGISTRY`           | `make push`                                           | —                                    | OCI registry target, e.g. `ghcr.io/yourorg/charts`            |