Streams

A Kafka topic, queried with ordinary SQL. SQLDesk keeps a window of the last few minutes and nothing older — and consumes only while somebody is watching.

What this is, and what it is not

It is not a pipeline. SQLDesk does not land your topic anywhere, does not replay history, and does not promise to have seen every message. If you want the last six months, land the topic in your warehouse and point SQLDesk at that. What this is for is the question you have during an incident, about the last ten minutes.

What it keeps is a window, held in DuckDB, sized from the row budget and the rate actually observed: a slow topic gets an hour, a fast one gets four minutes. Above a ceiling the stream is sampled, and everything drawn from it says so — a count from a sampled stream is an estimate, and silence about that is the worst outcome.

Clusters and topics

An administrator adds a cluster as a data source: brokers, and whatever the cluster needs to let us in. Its topics are its tables.

Which topics can be queried is chosen under Settings → Streaming Data Sources, by somebody with the streams permission — it sits beside Data Sources because it configures a connection, and the cluster itself is added on the tab next to it. That is the real access decision: the data source permission is per cluster, so whatever is enabled there is what everybody with access to that cluster can query. A cluster carrying a topic not everyone should see wants a second data source of its own with a restricted list, granted to another group.

Analyse a topic before enabling it. It reads the last few hundred messages and says what the columns are, how many could not be parsed, how fast they arrive, how much history that buys, and whether it will be sampled. None of those can be guessed from a topic’s name, and finding out an hour later is the bad way.

Querying one

Create → New Streaming Query opens the query editor, with the schema browser listing the enabled topics and the same visualizations as anywhere else. Two differences: the button says Start streaming rather than Execute, and no result is ever stored.

A streaming query is a query. It is listed under Queries → Streaming Queries, with the same All, Favorites, Mine and Archived views as any other — and nowhere else: the two halves of that list do not overlap, so a query has one place it lives. The same is true of Dashboards → Streaming Dashboards; a dashboard built on streams carries a streaming chip beside its name and a band down the side of the page, which turns green while its topics are actually being consumed.

The query is saved like any other; the rows are not. A window is what a consumer saw while somebody was watching, and writing it down would be making a copy of the one thing whose point is that it is current.

Streams cannot be scheduled and cannot be alerted on, for the same reason: at three in the morning nobody is watching, so there is nothing to refresh and nothing for an alert to check.

Every topic has _received_at

The time SQLDesk took the event off the topic, added to every window. It is the column to group by for a rate, and the column to order by for anything drawn over time:

select _received_at, region, sum(amount)
from orders
group by 1, 2
order by 1

Not the time the event happened — that is whatever field your producer puts in the message, and SQLDesk does not guess which one. Use yours where the difference matters; a late or replayed event has an arrival time of now.

Running, paused, cold

A stream is running while somebody has checked in within the last 45 seconds; paused when nobody has, keeping its window so coming back resumes in seconds rather than from an empty chart; and cold after ten minutes, when the window is dropped and starting again is a deliberate act.

Resuming seeks to the end of the topic, as starting does. Replaying the gap would stamp old events with an arrival time of now, which would be a chart that lies — so a pause leaves a hole, and the chart says it was paused.

Slots

A slot is a topic, not a viewer. Five people watching orders share one consumer and one window and take one slot between them, which is what makes a limit mean five topics rather than five tabs — and makes a team round one dashboard during an incident nearly free.

There are three limits: how many topics may be consumed at once across the install, how many one person may start, and how long one runs before somebody says they still want it. Past them you are told what is running and who started it, because “no slots” is only actionable if you can see who to ask. The editor says all N stream slots in use beside the button that will not start, and Admin → Streaming Queries shows every topic, who is watching it, and the budgets.

Who may do what

Administrators

Add the cluster, and pin a topic so it consumes with nobody watching.

manage_streams

Choose which topics can be queried, and how much of each is kept.

use_streams

Start a stream, which takes a slot.

Anybody with access to the cluster

Query what is enabled, and watch a stream somebody else started — no slot is taken and nothing new becomes visible.

On a dashboard

A dashboard shows streams or saved queries, never both, and a streaming one is still until it is made live.