Why Querying 100 EB
Takes Seconds

The first question I get from architects evaluating Vantage is always some version of the same thing: how does querying 2 TB of raw packet capture return results in 38 milliseconds?

The short answer is: because we read 412 KB, not 2 TB.

None of it involves a database. That distinction matters more than it might seem.

Why "Database for Packets" Is the Wrong Mental Model

When people first hear about querying exabytes of packet data in seconds, the instinct is to reach for a familiar frame: it must be some kind of database. A time-series store, maybe. A columnar engine. Something like Elasticsearch but faster.

This instinct leads to the wrong expectations and the wrong architecture questions.

Databases are designed around schemas, rows, and indexes built after data arrives. A database ingesting packet data at 800 Gbps would need to ingest roughly 100 GB per second into a structured store, maintain indexes across billions of rows, and handle concurrent queries against a dataset that is growing faster than most databases are designed to handle.

We did not build a faster database. We built something structurally different: a hierarchical index that is written as a byproduct of the capture process itself, costs less than 1% of the underlying data in storage, and navigates by elimination rather than by search.

The Query Analytics Tree: Structure

Sentinel writes a .index file alongside every PCAP capture it produces. That file contains the Query Analytics Tree, or QAT — a hierarchical beacon tree that covers the full capture at multiple levels of resolution simultaneously.

The 8× Window Scaling

At the leaf level, the QAT stores keys for 1 MB capture windows. Each level above covers 8 times the window of the level below:

Level  0   (leaf)    1 MB windows
Level  1             8 MB windows
Level  2            64 MB windows
Level  3           512 MB windows
Level  4             4 GB windows
Level  5            32 GB windows
  ...
Level 10        ~32 EB windows  (theoretical maximum)

A 2.1 TB capture  →  QAT ~10 levels deep
A 100 EB capture  →  same tree shape, same algorithm

Each node stores a compact bitmap of keys present in its window. The keys are drawn from every analyzer running against the capture: TCP flow 5-tuples, HTTP URIs and status codes, DNS queries and response codes, TLS cipher suites and anomaly flags, GTP tunnel IDs, Suricata rule SIDs, Zeek connection UIDs, and any user-defined VQC rules.

Every analyzer writes to the same QAT structure in the same format. Adding a new analyzer does not require a schema change, a reindex, or a format migration. The tree picks up its output automatically.

The Sub-bitmap AND Pruning Algorithm

This is where the performance comes from.

A VQC query is translated into a set of required keys. The query engine begins at the root of the QAT and evaluates each branch node by performing a sub-bitmap AND between the query's required keys and the node's stored keys. If the result is zero — meaning none of the required keys are present in that branch's entire window — the branch is pruned. The engine never descends into it.

-- VQC query
FIND packets WHERE ip.src == 185.234.72.19

-- QAT traversal (simplified)
root node bitmap:   11010110...
query key bitmap:   00000100...
AND result:         00000100  →  non-zero, descend

  branch A bitmap:  11010010...
  query key bitmap: 00000100...
  AND result:       00000000  →  ZERO, prune entire branch

  branch B bitmap:  00000110...
  query key bitmap: 00000100...
  AND result:       00000100  →  non-zero, descend

At each level, a single bitwise AND operation eliminates up to 7 of 8 children. Across 10 levels of tree, compound pruning eliminates virtually all irrelevant data before a single raw packet byte is read.

The PROFILE output from a real query against 2.1 TB of capture makes this concrete:

PROFILE FIND packets WHERE ip.src == 185.234.72.19

  QAT nodes visited     847
  Packet data read      412 KB  (of 2.1 TB total)
  Data fraction         0.000019%
  Query time            38ms

Query Results Are Virtual PCAP Files

When a VQC query completes, the result is not a report, a table, or a summary. It is a virtual PCAP file — a real file path that any PCAP-aware tool can open directly. Wireshark, tcpdump, Zeek, or any analysis tool your team already uses reads it without modification.

A query against a 2.1 TB capture that matches 4,218 packets across a 30-minute window produces a virtual PCAP covering exactly that window, containing exactly those packets. Nothing is extracted. Nothing is copied. The virtual file is a zero-overhead map into the original capture data — read operations are served directly from the underlying bytes.

The storage overhead of a virtual PCAP result is zero. The original capture continues recording uninterrupted. Multiple analysts can hold independent query results against the same capture simultaneously, each with their own virtual file, without any additional storage cost.

SILO Correlation: One Namespace, No JOINs

The QAT handles packet data efficiently. But packet data is never the whole picture. A realistic investigation requires correlating packets with IDS alerts from Suricata or Zeek, flow metadata from NetFlow or IPFIX, and threat intelligence from MISP or VirusTotal.

In most environments, this correlation happens manually: query Wireshark for the packet, pivot to the SIEM for the Suricata alert, cross-reference the flow record in the NetFlow tool, look up the IP in VirusTotal. Four tools. Four namespaces. Four query languages.

Vantage handles this through SILOs — Source Integration Layer Overlays. A SILO is a declaration of intent: it describes where a data source lives, what format it uses, and how it relates to other sources in the namespace.

The Four SILO Archetypes

Capture SILO. The anchor. Owns the .index file, the .events stream, and the raw PCAP data. When Sentinel captures a packet, it writes the PCAP byte and simultaneously commits keys to the QAT and a structured event to the .events stream: TCP state transitions, HTTP request/response pairs, DNS query/answer cycles, TLS handshake outcomes.

Analysis SILO. Suricata, Zeek, Snort. The Analysis SILO does not sit alongside the Capture SILO — it merges into it. Alert keys from Suricata rule matches are committed directly into the Capture SILO's QAT at the time they fire. When an analyst queries "show packets where suricata.alert is present", they are navigating the same QAT that holds the packet keys. IDS alerts and packets share a namespace. No pivot required.

Telemetry SILO. NetFlow, IPFIX, cloud flow logs. The Telemetry SILO maintains its own QAT, cross-referenced by 5-tuple at query time. When a query asks for traffic involving a specific IP, the engine navigates both the Capture QAT and the Telemetry QAT in parallel and returns a unified result. The analyst sees one answer, not two.

Enrichment SILO. MISP, VirusTotal, MaxMind, internal threat intel. Enrichment SILOs append threat scores, geographic annotations, and campaign tags to matching flows at query time without a separate lookup step. An IP that appears in a VirusTotal feed as a known C2 infrastructure host will surface that tag automatically when it appears in query results.

What This Looks Like to the Analyst

-- A single VQC query spanning four data sources
FIND flows
  WHERE suricata.alert IS NOT NULL
  AND tls.anomaly IS PRESENT
  AND netflow.bytes > 1000000
  AND threat_intel.score > 75
  DURING LAST 24h
  FROM captures, suricata-silo, netflow-silo, misp-silo

Why This Scales to Exabytes

The properties that make the QAT fast at 2.1 TB are structural, not tunable. They hold at any scale because they derive from the tree shape and the pruning algorithm, not from hardware or cache optimization.

At 100 EB, the QAT is deeper — more levels — but the same algorithm applies. A query navigates from the root, pruning aggressively at each level, and touches only the leaf nodes that contain confirmed matches. The number of nodes visited scales with the specificity and selectivity of the query, not with the total capture size.

The 0.59% metadata overhead is also a structural property. It does not grow with capture complexity or the number of analyzers running. A capture with 20 analyzers active produces the same metadata ratio as a capture with 2 analyzers, because all analyzers write to the same QAT structure in the same format.

This has a practical consequence for retention. The metadata that powers queries costs less than 1% of the underlying data. Retaining the QAT and .events stream permanently — even after packet data is released — preserves full query capability for everything the analyzers recorded, indefinitely, at negligible storage cost.

One More Time: This Is Not a Database

I want to be specific about this because the confusion has real consequences for how teams evaluate and deploy packet intelligence infrastructure.

Database index    Built after data arrives
QAT               Built as packets arrive — zero additional latency

Database query    Scans rows, applies filters — reads data
VQC query         Navigates tree, prunes branches — eliminates data

Database schema   Defines indexable fields before data arrives
QAT               Accepts any analyzer output — no schema change needed

Database store    Structured records
QAT               Keys derived from raw packets in native PCAP format

Everything described in this article is running today on a single server. The full incident response workflow — alert to evidence in 24 minutes, two infected hosts identified, zero packets moved off server — is documented in our forensic readiness article.