> ## Documentation Index
> Fetch the complete documentation index at: https://docs.samschroeder.lu/llms.txt
> Use this file to discover all available pages before exploring further.
# SIEM Integrations: Splunk, QRadar, and Elasticsearch
> ThreatLab ships exercise log archives to your SIEM so analysts investigate real events. Supports Splunk HEC, syslog TCP, and Elasticsearch bulk indexing.
When you start an exercise, ThreatLab connects to your configured SIEM destination and ships the exercise log archive so you investigate real events in the tooling you use every day. No manual data loading, no synthetic replays — the logs land in your SIEM index and you work the case from there. Your administrator configures the destination connections; you just hit **Start** and begin investigating.
## Supported Drivers
ThreatLab supports three shipping drivers, each targeting a different SIEM stack:
| Driver | Protocol | Typical Use Case |
| -------------- | -------- | --------------------------------------------------------------------------------------------------------- |
| `splunk_hec` | HTTPS | Splunk HTTP Event Collector — sends events to a Splunk index via the HEC token endpoint. |
| `syslog_tcp` | TCP | QRadar and any other syslog-compatible SIEM — delivers events as syslog frames over a raw TCP connection. |
| `elastic_bulk` | HTTPS | Elasticsearch — indexes documents using the Elasticsearch Bulk API. |
## Payload Formats
The driver you use determines which log payload format ThreatLab ships:
**Log Event Extended Format.** Default for `splunk_hec` and `syslog_tcp`. LEEF archives contain structured key-value event lines compatible with Splunk and QRadar parsers.
**Elastic Common Schema.** Used by `elastic_bulk`. ECS archives contain JSON documents pre-formatted for Elasticsearch bulk indexing.
An exercise can carry **both LEEF and ECS archives simultaneously**. ThreatLab selects the correct format at exercise start based on the `payload_format` configured on your SIEM destination — so the same exercise works across different analyst environments without any changes.
### When SyslogCanon is required
ThreatLab uses the **SyslogCanon** sidecar to canonicalise raw logs and produce LEEF and ECS archive pairs. Authors need SyslogCanon at publish time whenever the raw input does not already cover the format an enabled destination needs:
* **Convert to LEEF is enabled.** SyslogCanon parses every row and emits both LEEF and (where required) ECS.
* **Raw rows are detected as ECS** but a LEEF destination is enabled (e.g. `splunk_hec`, `syslog_tcp`). SyslogCanon generates the LEEF side.
* **Raw rows are detected as LEEF** but an ECS destination is enabled (e.g. `elastic_bulk`). SyslogCanon generates the ECS side.
* **Any row is unknown.** SyslogCanon canonicalises it — ThreatLab no longer silently labels unknown raw archives as LEEF.
When the publish-time readiness check determines that SyslogCanon is required for a given format and it is either unavailable or fails to dual-parse, the publish is blocked. The error names both the affected SIEM destination and the format (LEEF or ECS) so authors know which side to fix.
## Syslog TCP Envelope Settings
When your destination uses the `syslog_tcp` driver, your administrator can configure the syslog envelope on a per-destination basis. These settings control how ThreatLab frames each event before it goes on the wire:
| Setting | Options | Notes |
| ------------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| **Header mode** | `preserve_raw`, `synthesize_if_missing`, `force_synthesize` | Controls whether the original log header is kept, filled in if absent, or always overwritten. |
| **Header format** | `rfc3164`, `rfc5424` | Selects the syslog framing standard. |
| **Source identity** | String | The hostname or IP ThreatLab presents as the log source. |
| **PRI** | Integer 0–191 | RFC 5424 PRIVAL encoding facility and severity. |
| **App name** | String | The application name field in the syslog header. |
| **Process ID** | String | The `procid` field in the syslog header. |
| **Message ID** | String | The `msgid` field in the syslog header. |
A QRadar-compatible destination profile typically uses **RFC 5424** header format with full syslog metadata populated in the defaults. Ask your administrator to verify the QRadar destination is configured with `rfc5424` and matching source identity values.
## QRadar API Management
For `syslog_tcp` destinations, administrators can optionally enable **QRadar API management** so ThreatLab calls the QRadar REST API to provision log sources and close offenses tied to wiped sessions. Event delivery still flows over syslog/LEEF — the API token only authorizes management actions. See [SIEM resources](/admin/siem-resources#adding-a-siem-destination) for the field reference.
When **Provision log source** is enabled, ThreatLab creates one QRadar log source per exercise section, scoped to every enabled QRadar `syslog_tcp` destination. Each log source is keyed by exercise and section, so the same section never reuses a log source from a different exercise, and rerunning provisioning is idempotent — an already-provisioned section returns its existing log source rather than creating a duplicate. Sectionless exercises continue to provision a single log source per exercise. Provisioning runs per section: if one section's destination fails (for example, because the QRadar API is unreachable), the other sections still return their own success or failure result.
QRadar destinations on the `syslog_tcp` driver can optionally use the QRadar REST API for two console-side tasks. Both are opt-in and configured per destination by your administrator — see [QRadar API management](/admin/siem-resources#qradar-api-management) for setup details.
* **Provision log source.** ThreatLab creates a per-exercise log source on the QRadar console the first time it ships logs, so events route to a dedicated log source instead of a shared one.
* **Close offenses on completion.** When an analyst finishes an exercise attempt, ThreatLab closes the QRadar offenses that were opened against the exercise log source during the attempt window. Cleanup runs after scoring and never blocks completion — if QRadar is unreachable, the analyst still gets credit and the failure is recorded for administrators.
## QRadar Log Source Provisioning
When your `syslog_tcp` destination targets QRadar, your administrator can let ThreatLab manage a dedicated QRadar log source per exercise so every event you investigate is tied back to a single exercise run. Provisioning is configured on the destination — see [QRadar log source provisioning](/admin/siem-resources#qradar-log-source-provisioning) for the admin steps.
### How it works
When **Enable log source provisioning** is on for the destination, ThreatLab runs the following flow at exercise start:
ThreatLab derives a deterministic source identifier in the form `threatlab-{exercise}-{siem}`. The same exercise on the same destination always uses the same identifier.
ThreatLab calls the QRadar REST API to either reuse an existing log source with that identifier or create a new one using the configured log source type (default `Universal LEEF`), protocol type (default `Syslog`), and event collector. A `409 Conflict` from QRadar is treated as a successful reuse.
For the rest of the run, ThreatLab forces synthesized syslog headers with the deterministic identifier as the source host, regardless of the destination's default **Source identity**. Every event for that exercise lands on the same QRadar log source.
The provisioning state (`pending`, `provisioned`, or `failed`), the QRadar log source ID, and whether QRadar requires a deploy are recorded so reruns of the same exercise reuse the same log source without re-calling the API.
Provisioning is a hard prerequisite for shipping. If the QRadar API is unreachable, the token is invalid, or no event collector can be resolved, **exercise start fails** with a clear error and no logs are shipped. Disable provisioning on the destination if you want shipping to proceed without an API round trip.
### When provisioning is disabled
If **Enable log source provisioning** is off, ThreatLab loads any previously stored source identity for the exercise and reuses it as the syslog source host. Destinations that have never been provisioned fall back to the default **Source identity** configured on the destination.
When **Provision log source** is enabled, ThreatLab creates a dedicated QRadar log source for **each section** of an exercise (not just one per exercise) so staged section release can identify which logs belong to which step of the investigation. Provisioning happens at **publish time** — when an author toggles an exercise to **Active** — as a preflight gate:
* **Provisioning failed** or **QRadar requires deploy** — the exercise is saved but kept in **Draft**, and warnings name the destination, the section, and the reason. The author must resolve each warning (re-check QRadar credentials, or run **Deploy Changes** in the QRadar console) and publish again.
* **Provisioning succeeded and QRadar is already deployed** — the exercise activates normally.
Cached QRadar log sources that still have `requires_deploy` set are re-checked against the QRadar API before the preflight short-circuits, so an administrator running **Deploy Changes** in QRadar is enough to clear the gate on the next publish attempt — no manual cache reset is needed. See [QRadar log-source preflight](/guides/authoring-exercises#qradar-log-source-preflight) in the authoring guide for the author-facing flow.
#### Per-section log sources
When **Provision log source** is enabled, ThreatLab provisions one QRadar log source per exercise section so each section's events arrive under a distinct syslog identity. At exercise start, redo, and **Ship logs to SIEM**, ThreatLab gates shipping on the readiness of every section's log source:
* A section is **ready** when its log source has `provisioned` status, a QRadar log source ID, and does not require a QRadar deploy.
* A section is **not ready** when its provisioning is missing, failed, or still pending a QRadar deploy.
If any section is not ready, the operation fails fast with an error that names each blocking section and the reason — for example `section 2 (web-access): QRadar deploy required`. Trigger the QRadar deploy (or fix the upstream provisioning error) and start the exercise again.
Exercises that have no per-section QRadar rows fall back to the legacy exercise-level log source, so older exercises that pre-date per-section provisioning continue to work without change.
#### Offense cleanup across sections
When **Close offenses** is enabled, session wipe and redo close every offense tied to any provisioned section log source for the exercise — plus any legacy exercise-level log source. Offenses linked to more than one section's log source are closed once and deduplicated in the cleanup summary.
## Shipping Models
ThreatLab supports two shipping models for an exercise's archive sections:
The default. Every section ships to your SIEM destination the moment the analyst clicks **Start**. Best for classic single-archive scenarios.
Sections ship individually as their release rules are satisfied — a per-section delay, an artifact gate, or both. Best for multi-phase scenarios where you want later evidence to appear only after earlier steps are solved.
Staged shipping is configured per section on the exercise itself — see [Section release rules](/concepts/exercises#section-release-rules) for the model and [Authoring exercises](/guides/authoring-exercises#staged-section-release) for the editor controls. ThreatLab snapshots each shipment per attempt, so a section that already shipped during one attempt is not re-shipped if the analyst hits **Ship logs to SIEM** again — the platform returns an explicit `already_shipped`, `not_due`, `waiting_on_artifacts`, or similar reason instead.
## Publish-Time Readiness
Setting an exercise to **Active** triggers a publish-time readiness check that mirrors the runtime resolution rules. The check derives required formats from **enabled** SIEM destinations (`driver` + `payload_format` + `enabled`) and the assigned SIEM, and verifies every section has a matching archive in every required format. If SyslogCanon is needed to generate a format and is unavailable or cannot dual-parse, publish is blocked per-format and per-section, with errors that name the affected SIEM and format.
This means an exercise can only go live if it is actually shippable to every destination it targets — there is no path that produces a partially-formatted exercise at publish time.
## Archive Resolution Order
At exercise start, ThreatLab resolves which archive to ship based on whether the exercise has section rows.
### Sectioned exercises (authoritative)
If the exercise defines any sections, per-section archives are **authoritative** for the requested payload format. Every section must carry a matching archive in the format your SIEM destination expects (LEEF or ECS).
* If any section is missing an archive in the requested format, shipping fails with a clear error that names the SIEM destination and the sections that are missing the format.
* ThreatLab will **not** silently fall back to a legacy per-exercise archive or to legacy exercise columns when section rows are present. A partial section set, or a section set that does not cover the requested format, always fails fast.
This applies to both active runs and staged exercises. Staged runs additionally validate against the full set of enabled SIEM destinations, so a staged exercise that targets both LEEF and ECS destinations must have section archives in both formats.
### Legacy exercises (no section rows)
If the exercise has no section rows at all, ThreatLab falls back to the legacy resolution order:
A single archive attached to the exercise record as a whole, in the requested payload format.
Archive metadata read directly from the legacy columns on the exercise row itself.
If neither legacy source has an archive in the requested format, shipping fails with an error that names the SIEM destination and the missing format.
Legacy fallback exists only to keep older, non-sectioned exercises working. Once an exercise has any section row, it is treated as sectioned and the legacy bundle is ignored — even if one is still attached.
## Publish-Time Guardrail
To prevent the resolution errors above from surfacing at exercise start, ThreatLab enforces a matching guardrail when an exercise is **saved or published** as a staged run:
* ThreatLab reads every **enabled** SIEM destination and groups them by payload format.
* For each format with at least one enabled destination, every section must have a matching archive.
* If any section is missing an archive in a required format, the save or publish fails with an error that names the affected SIEM destinations and sections.
The practical effect: a staged exercise cannot be saved with incomplete per-section archives. If you enable an ECS destination, every section needs an ECS archive; if you enable a LEEF destination, every section needs a LEEF archive. To unblock a save, either upload the missing section archives or disable the SIEM destinations that require the format you are not shipping.
## Wipe and Re-Fire
When you delete an active session, ThreatLab wipes the corresponding SIEM index so the environment is clean for the next attempt. Noise log jobs that have `propagate_on_wipe` enabled automatically **re-fire** after a wipe to refill your SIEM with background noise events — keeping the environment realistic for the next analyst who starts the exercise.
Deleting a session permanently removes the SIEM index for that exercise run. Any notes or in-progress work not saved to your investigation notebook will be lost. Use the redo flow if you want to restart while keeping your notebook.
SIEM destinations are configured by your ThreatLab administrator under **Admin > Resources**. Contact them to add a new connection or update an existing one.