Why Querying 100 EB
Takes Seconds

The first question I get from architects evaluating Vantage is always some version of the same thing: how does querying 2 TB of raw packet capture return results in 38 milliseconds?

The short answer is: because we read 412 KB, not 2 TB.

None of it involves a database. That distinction matters more than it might seem.

Why "Database for Packets" Is the Wrong Mental Model

When people first hear about querying exabytes of packet data in seconds, the instinct is to reach for a familiar frame: it must be some kind of database. A time-series store, maybe. A columnar engine. Something like Elasticsearch but faster.

This instinct leads to the wrong expectations and the wrong architecture questions.

Every database approach shares a property: the index tells you where the data is, but the query still reads it. Faster indexes reduce the scan. Compressed columns reduce the I/O. Sharded clusters parallelize the work. At the bottom, query time remains proportional to the rows that match.

A targeted packet query against 2.1 TB should not read 21 GB (1%) or even 210 MB (0.01%). It should read the handful of kilobytes that actually contain the answer, and prove the rest is irrelevant without looking at it. That's the problem the Query Analytics Tree solves — by being a hierarchical index written as a byproduct of capture, costing under a quarter percent of the underlying data, navigating by elimination rather than search.

The Query Analytics Tree: Structure

Sentinel writes a .index file alongside every PCAP capture it produces. That file contains the Query Analytics Tree, or QAT — a hierarchical beacon tree that covers the full capture at multiple levels of resolution simultaneously.

The 8× Window Scaling

At the leaf level — LOD-0 — each node covers a 1 MB window of the capture. Each level above covers exactly 8× the window of the level below:

LOD  0   (leaf)       1 MB windows
LOD  1                8 MB windows
LOD  2               64 MB windows
LOD  3              512 MB windows
LOD  4                4 GB windows
LOD  5               32 GB windows
LOD  6              256 GB windows
LOD  7                2 TB windows
LOD  8               16 TB windows
  ...
LOD 15               32 EB windows   (single-tree maximum)

A 2.1 TB capture  →  9-level tree rooted at LOD-8
A 32 EB capture   →  16-level tree rooted at LOD-15
Beyond 32 EB      →  multi-file / distributed mode, same algorithm

A 2.1 TB capture produces a tree roughly 9 levels deep. The root promotes to LOD-8 because LOD-7's 2 TB window is too small for 2.1 TB, and LOD-8's 16 TB window has room to spare. A 32 EB deployment — the maximum single-tree capacity — produces a 16-level tree rooted at LOD-15. Captures larger than 32 EB use multi-file or distributed mode, where the same tree format is sharded across files or nodes. Same tree shape, same navigation algorithm, same query performance characteristics at every scale.

Each node stores a compact bitmap of keys present in its window. The keys are not hashes or derived summaries — they are literal tokens from the jNetWorks Token Stream, the same tokens analyzers emit during decoding, copied byte-for-byte into the QAT with an 8-byte routing suffix. Every analyzer that emits a token automatically gets QAT indexability. New protocols, new IDS engines, new custom analyzers: no format change, no schema migration, no coordination. The tree picks up the output automatically.

The Sub-bitmap AND Pruning Algorithm

This is where the performance comes from.

A Vantage Query is translated into a set of required keys. At every branch node, the engine walks the node's key chain and performs a bitwise AND between each required key's 8-bit sub-bitmap and a running candidate mask. Children whose bit is clear in the final mask are pruned — the engine never descends into them.

-- Vantage Query
FIND packets WHERE ip.src == 185.234.72.19

-- Multi-predicate query at a single branch node
src_cidr key           sub_bitmap = 0b10110101
dst_port key           sub_bitmap = 0b11010111
app_proto key          sub_bitmap = 0b00110101

candidate_mask       = 0b10110101
                     & 0b11010111
                     & 0b00110101
                     = 0b00010101

Descend into children 0, 2, 4 only.
Five of eight children pruned at this level alone.

At each level, a single bitwise AND across the required keys eliminates most of the children. Across 9 levels of tree for a 2.1 TB capture — or 16 levels for 32 EB — compound pruning eliminates virtually all irrelevant data before any raw packet byte is touched.

The PROFILE output from a real query against 2.1 TB of capture makes this concrete:

quarry> PROFILE FIND packets WHERE ip.src == 185.234.72.19

  QAT nodes visited     847
  Packet data read      412 KB  (of 2.1 TB total)
  Data fraction         0.000019%
  Query time            38 ms

Adaptive Coarsening Keeps Nodes Bounded

Some windows produce a lot of values. An IP scan might touch 5,000 distinct /24 networks in a single 1 MB window. A DNS tunnel might emit thousands of distinct subdomain queries. If every value occupied its own key, node size would explode on adversarial traffic.

Instead, the QAT writer coarsens. Key chains per type are capped (64 entries by default). When a chain exceeds the cap, the writer emits coarsened keys covering the same space — 5,000 /24 keys become fewer /16 keys; thousands of distinct subdomain.slytechs.com queries become a single *.slytechs.com wildcard. Each key's coarseness byte records how many aggregation steps it represents.

The precision is preserved at child nodes below. Since each child window is 8× smaller, each holds 8× fewer distinct values and coarsening is less likely to trigger. Precision is recovered on descent. The 0.021% storage overhead holds at every scale because coarsening absorbs cardinality explosions structurally.

Query Results Are Virtual PCAP Files

When a Vantage Query completes, the result is a virtual PCAP file — a real file path that any PCAP-aware tool (Wireshark, tcpdump, Zeek) can open directly. The virtual file is a zero-overhead map into the original capture data — read operations are served directly from the underlying bytes. Nothing is extracted. Nothing is copied. The original capture continues recording uninterrupted, and multiple analysts can hold independent query results against the same capture simultaneously without any additional storage cost.

SILO Correlation: One Namespace, No JOINs

The QAT handles packet data efficiently. But packet data is never the whole picture. A realistic investigation requires correlating packets with IDS alerts from Suricata or Zeek, flow metadata from NetFlow or IPFIX, and threat intelligence from MISP or VirusTotal.

In most environments, this correlation happens manually: query Wireshark for the packet, pivot to the SIEM for the Suricata alert, cross-reference the flow record in the NetFlow tool, look up the IP in VirusTotal. Four tools. Four namespaces. Four query languages.

Vantage handles this through SILOs — Source Integration Layer Overlays. External data sources are organized into four archetypes: a Capture SILO owns the raw PCAP, the .index file, and the .events token stream; Analysis SILOs merge IDS alerts (Suricata, Zeek) directly into the same QAT; Telemetry SILOs (NetFlow, IPFIX) maintain their own QATs cross-referenced by 5-tuple at query time; Enrichment SILOs (MISP, VirusTotal, MaxMind) append threat scores and campaign tags to matching flows without a separate lookup step.

Each SILO source emits into its own dedicated domain in the Token Stream — SURICATA, ZEEK, VIRUSTOTAL, MISP, NETFLOW. Downstream consumers filter by source with a single domain match rather than parsing composite type fields.

Honest Provenance via Carrier Tokens

Correlation against live packet data is best-effort in one specific sense: if an IDS alert arrives after the packet window it corresponds to has left the live correlation cache, synchronous binding isn't possible. When this happens, Vantage emits a carrier token tagged with one of four confidence states: FULL (verified against live cache), PARTIAL (probable but not fully verified), INFERRED (structural similarity match), or UNKNOWN (placeholder pending resolution).

When retrospective correlation against stored data completes, a refined token supersedes the carrier with confidence elevated and the binding pinned to the specific frame. An anomaly detector can weight inputs by confidence. A training pipeline can restrict to FULL-confidence tokens for ground truth. The stream gives consumers the information to handle uncertainty explicitly, rather than hiding it in a vendor's enrichment layer.

What This Looks Like to the Analyst

-- A single Vantage Query spanning four data sources
FIND flows
  WHERE suricata.alert IS PRESENT
  AND tls.anomaly IS PRESENT
  AND netflow.bytes > 1000000
  AND threat_intel.score > 75
  DURING LAST 24h
  FROM captures, suricata-silo, netflow-silo, misp-silo

PCAPNG Compatible by Design

The .index file is not a proprietary container. It's a single PCAPNG copyable custom block, type 0x00000BAD, with the Sly Technologies Private Enterprise Number as the custom block's identifier. The same is true of the .events token stream. Custom blocks are defined by the PCAPNG specification to be skippable by readers that don't recognize their PEN, so tcpdump, Wireshark, Zeek, or any tool built on libpcap or pcapng reads the capture without issue — they encounter the blocks and skip them. Vantage-aware tools — Lynx, the Quarry virtual filesystem, the Vantage Query engine — recognize the PEN and unwrap the block.

This means the .index file can live inside the capture file itself as embedded blocks, or as a separate sidecar file alongside the capture. Either way, the format is a valid PCAPNG.

Why This Scales to Exabytes

The properties that make the QAT fast at 2.1 TB are structural, not tunable — they derive from the tree shape and the pruning algorithm, not from hardware or cache optimization. At 32 EB, the tree is deeper (16 levels) but the same algorithm applies. The number of nodes visited scales with the specificity of the query, not with the capture size.

The 0.021% storage overhead is a structural property of the tree's branching factor. It doesn't grow with capture complexity or the number of analyzers running — all analyzers write to the same QAT structure in the same format. Retaining the QAT and .events stream permanently, even after packet data is released, preserves full query capability indefinitely at negligible storage cost.

One More Time: This Is Not a Database

I want to be specific about this because the confusion has real consequences for how teams evaluate and deploy packet intelligence infrastructure.

Database index    Built after data arrives
QAT               Built as packets arrive — zero additional latency

Database query    Scans rows, applies filters — reads data
Vantage Query     Navigates tree, prunes branches — eliminates data

Database schema   Defines indexable fields before data arrives
QAT               Accepts any analyzer output — no schema change needed

Database store    Structured records
QAT               Keys derived from raw packets in native PCAP format

Database overhead 2–5× source data for rich indexing
QAT               0.021% structural overhead

Everything described in this article is running today on a single server. The full incident response workflow — alert to evidence in 24 minutes, two infected hosts identified, zero packets moved off server — is documented in our forensic readiness article. The feature stream that feeds the QAT's keys is described in the Token Stream article. The full architectural treatment is in the QAT whitepaper.