Logs And Monitoring API Workflows

Use this page when you need real-time and historical visibility into stream activity through the API.

This page focuses on stream and system monitoring across live stream logs, current stream stats, persisted logs for completed runs, and run history. For live SQL query activity from streams, Data Explorer, or SQL Console, see SQL Logs API Workflows.

Monitoring model

Operational visibility typically combines:

• live logs from GET /api/v1/logs/stream
• current execution state from GET /api/v1/streams/{id}/stats
• persisted logs from GET /api/v1/streams/{id}/logs
• historical run records from GET /api/v1/stream-configs/{id}/history

Live logs with SSE

Use GET /api/v1/logs/stream when you need browser-friendly live stream logs or want to stream structured monitoring events into your own tooling.

Authentication: the current backend exposes /logs/stream as a public SSE route, and the checked-in Swagger contract marks it with security: []. In production deployments, access should be controlled at the network or proxy level.

Example:

curl -N "$API_URL/logs/stream"

SSE event types

The stream uses named SSE events to distinguish payload families:

Common structured categories within log events include progress, stat, error, and general.

Example SSE frame

event: log
data: {"level":"info","message":"STAGE:1 | Setting up stream.","timestamp":"2026-03-20T09:41:30Z","type":"reader","nodeId":"reader-1","streamId":"stream_123","category":"progress"}

Verification:

• confirm structured log payloads arrive as event: log frames with fields such as level, message, timestamp, type, and optionally streamId
• confirm categories such as progress or stat appear while a stream is active

Get current stream status

Use GET /api/v1/streams/{id}/stats when you need the current execution state for a running or completed stream.

This route returns current status and counters for one stream. It can reflect live execution while the stream is running, but it is a point-in-time API response, not an SSE event flow.

Example:

curl "$API_URL/streams/$STREAM_ID/stats" \
  -H "X-API-Key: $API_KEY" \
  -H "X-Install-ID: $INSTALL_ID"

Example response:

{
  "configID": "cfg_abc",
  "streamID": "stream_123",
  "status": "RUNNING",
  "nodes": [
    {
      "id": "reader-1",
      "type": "source",
      "stat": {
        "counter": 15000,
        "failedCounter": 0,
        "prevDataSize": 0,
        "sumDataSize": 2048576,
        "reportingInterval": 1000000000,
        "start": "2026-03-20T09:41:30Z",
        "duration": 120000000000,
        "avgRate": 17067,
        "status": "RUNNING"
      }
    }
  ]
}

Stream status values

Node stat fields

Use it when you need:

• the current state of one stream
• a point-in-time status check for automation
• polling-friendly status for external dashboards

Get persisted logs for one stream run

Use GET /api/v1/streams/{id}/logs when you need historical logs for a specific stream execution after the fact.

This route returns persisted log entries from the stream's log file. It is not a live feed — use the SSE stream for real-time logs.

Query parameters

Example:

curl "$API_URL/streams/$STREAM_ID/logs?level=error&limit=100" \
  -H "X-API-Key: $API_KEY" \
  -H "X-Install-ID: $INSTALL_ID"

Example response:

[
  {
    "level": "error",
    "message": "failed to insert batch: duplicate key",
    "timestamp": "2026-03-20T09:42:15Z",
    "type": "writer",
    "nodeId": "writer-1",
    "streamId": "stream_123",
    "category": "error"
  }
]

Use this route when you need:

• post-run troubleshooting
• filtered error review
• historical log retrieval for one stream execution

Get run history for a stream configuration

Use GET /api/v1/stream-configs/{id}/history when you need the run timeline for one saved stream configuration.

The response is a JSON array of all runs for the configuration, sorted newest first. The backend returns up to 10,000 runs in a single response — pagination and filtering are handled client-side.

Example:

curl "$API_URL/stream-configs/$CONFIG_ID/history" \
  -H "X-API-Key: $API_KEY" \
  -H "X-Install-ID: $INSTALL_ID"

Example response:

[
  {
    "id": "stream_456",
    "configId": "cfg_abc",
    "streamId": "stream_456",
    "timestamp": 1710928800,
    "durationMs": 12500,
    "status": "FINISHED",
    "dataSize": "11.7 KB",
    "rowsInserted": 1500,
    "rowsSkipped": 0
  },
  {
    "id": "stream_455",
    "configId": "cfg_abc",
    "streamId": "stream_455",
    "timestamp": 1710842400,
    "durationMs": 8200,
    "status": "FAILED",
    "dataSize": "4.2 KB",
    "rowsInserted": 600,
    "errorMessage": "connection refused"
  }
]

Run fields

Use it when you need:

• newest-first run history
• execution counts across multiple runs
• historical monitoring for one stream definition

API vs UI

Use the UI for interactive monitoring through Observability.

Use the API when you need to integrate logs or stream status into external systems such as dashboards, alerting pipelines, or custom automation.

Related docs

• Observability
• SQL Logs API Workflows
• Streams API
• API Reference