Data sources

A data source is a connection to somewhere data lives. SQLDesk ships runners for more than thirty-five of them; which are available depends on what was installed with the image.

Adding one

Settings → Data Sources → New. Pick the type, fill in the connection, press Test Connection before saving — it is the difference between finding a typo now and finding it in somebody's dashboard later.

Credentials are encrypted at rest with the instance's secret key. Rotating that key makes existing credentials unreadable, which is worth knowing before you rotate it.

Who can use it

Permission is by group. A data source is attached to one or more groups; whoever is in one of those groups can query it, and whoever is not cannot see it exists. There is no per-user grant.

A group can be given view only access, which lets people see results without being able to write new queries against the source.

What each kind needs

Kafka: a stream is a window

A topic is a data source like any other, but what you query is not the topic — it is the window SQLDesk is holding. It keeps the last few minutes of raw events and rolls everything older into per-minute buckets it keeps for a day. A dashboard asking “what is happening now” reads the raw window; one asking “how does now compare to this morning” reads the rollup.

For history, land the topic in your warehouse with Kafka Connect or Flink and point SQLDesk at the warehouse. That is not a limitation being apologised for — a broker already has retention and a warehouse already has history, and a dashboard tool that tried to be either would be a worse version of both while being the thing that falls over.

The window sizes itself

You set a budget in rows, not minutes, and the window is whatever that buys at the rate the topic is actually doing. A fixed window is wrong in both directions: thirty minutes of a 50,000/s topic is gigabytes, five minutes of a 3/s topic is a chart with nine points on it. A quiet topic gets the full half hour; a busy one settles at five minutes. The chart says which: “last 6 minutes · 4.9M events”.

Over the ceiling, it samples — and says so

Past the events-per-second ceiling, events are kept one in N and every chart carries the figure. The decision is a hash of the message key, so the same keys stay in from one second to the next and a per-key count can be scaled back up honestly — random sampling would give you a different set of users every second and no count worth reporting. A topic with no keys is sampled by position instead, which is still one in N.

Nothing is ever buffered on the way in. An unbounded queue in front of a slow writer is the crash, not the protection against it. Whatever arrives in a second is either written or sampled away within that second.

Rollups, and the tail

A rollup names the columns to group by and what to measure. Distinct groups are capped at a thousand and everything past the cap lands in one other bucket the chart shows rather than hides: a chart that silently dropped the long tail would make a total wrong and look right. Grouped by something like a user id on a ten-million-user stream, a rollup is not a rollup — it is the stream again with extra steps, which is what the cap is for.

Distinct counts are approximate by default (HyperLogLog), because an exact distinct over a minute of a busy topic can cost more than everything else together. Ask for exact on the one measure that needs it.

It only runs while somebody is watching

A stream is consumed while something that uses it has been looked at in the last fifteen minutes, or while an administrator has pinned it. Always-on burns a worker and a disk for a dashboard nobody has open — a stream is the one thing here that costs money continuously rather than per query — and consuming only while a dashboard is literally open means arriving to an empty chart every time. The pin covers the few streams that genuinely have to be there at 3am.

On resume it starts from now, not from where it left off. The window is “now”, so replaying an hour of backlog to fill five minutes would stamp hour-old events with an arrival time of now and draw a chart that is a lie.

Schema

The columns are inferred from the first flush and then left alone; an administrator corrects and freezes them. A field a producer adds later is ignored until somebody re-infers, deliberately: a chart should not change shape because a producer shipped a field. A message SQLDesk cannot read increments a malformed counter on the stream's page and never stops the consumer. JSON only for now — Avro and Protobuf need a schema registry, which is its own connection and its own credentials.

Needs confluent-kafka, which is in the optional data-source dependency group with every other SDK. Without it the connector does not appear, which is the same as every other optional one.

Schema and autocomplete

SQLDesk periodically reads the table and column list and caches it in Redis so the editor can autocomplete. It is refreshed on a schedule and can be refreshed by hand from the editor. A missing table in autocomplete usually means the cache is older than the table.

When a connection fails

• From the browser it works, from SQLDesk it does not — the workers connect, not your laptop. The host has to be reachable from the container or pod.
• It worked yesterday — check whether the secret key changed, which makes stored credentials undecryptable.
• Only some queries fail — usually a per-query timeout rather than the connection.