Skip to content

Import the CTI database

The self-hosted-controller (SHC) imports the Sekoia CTI database into the ArangoDB database of your platform with the ImportArangoCollection command. It reads the latest CTI export from the Sekoia release bucket in online mode, or from the orchestration node in air-gapped mode, and loads the STIX objects and relationships it contains.

The imported database feeds contextualized alerts and the SEKOIA Intelligence Feed detection rule. It does not contain the observables and their tags, so observable tags enrichment is not available. The Intelligence module is not available in Sekoia Self-Hosted 1.0.1.

Prerequisites

  • The platform is installed and the SHC can retrieve its kubeconfig.
  • In online mode, the S3_ACCESS_KEY and S3_SECRET_KEY environment variables hold the Sekoia release bucket credentials provided by Sekoia, and global.version.fetch points to the Sekoia release bucket, as described in Configure the deployment.
  • In air-gapped mode, global.version.fetch.offline is true and the CTI archive is extracted on the orchestration node, as described in Prepare the CTI export in air-gapped mode.

Prepare the CTI export in air-gapped mode

The release archive does not include the CTI database. Sekoia publishes it as a separate archive in the release bucket, every day at 12:00 UTC: data/cti/archives/cti-latest.tar, with its SHA-256 checksum. The archive is an uncompressed .tar file that holds the latest export of each collection.

  1. On a machine with internet access, download the archive and its checksum with the bucket credentials provided by Sekoia, then verify its integrity. Keep the cti-latest.tar file name: the checksum file refers to it.

    export AWS_ACCESS_KEY_ID=""
export AWS_SECRET_ACCESS_KEY=""
export AWS_DEFAULT_REGION="fr-par-2"
export ENDPOINT="https://self-hosted.delivery.sekoia.io"
export AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED
export AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED

aws --endpoint-url "$ENDPOINT" s3 cp s3://self-hosted/data/cti/archives/cti-latest.tar cti-latest.tar
aws --endpoint-url "$ENDPOINT" s3 cp s3://self-hosted/data/cti/archives/cti-latest.tar.sha256 cti-latest.tar.sha256

sha256sum -c cti-latest.tar.sha256

    Expected output: cti-latest.tar: OK

  2. To transfer the archive to the orchestration node, run:

    scp cti-latest.tar user@<ORCHESTRATION_NODE>:$SEKOIA_LOCAL_DIR

    Disk space

    The parts of the export are already compressed, so the extracted files take about as much space as the archive. Before you transfer it, make sure the orchestration node has free space for twice the size of the archive. You can delete the archive once it is extracted.

  3. To extract the archive on the orchestration node, run:

    tar -xf $SEKOIA_LOCAL_DIR/cti-latest.tar -C $SEKOIA_LOCAL_DIR

The archive extracts to $SEKOIA_LOCAL_DIR/data/cti/, with one directory per collection. Each directory holds a latest.json file that names the export to import, and the export folder itself:

$SEKOIA_LOCAL_DIR/data/cti/
├── stix_objects/
│   ├── latest.json
│   └── <export ID>/
│       ├── checkpoint.json
│       └── part-<N>.jsonl.zst
└── stix_relationships/
    ├── latest.json
    └── <export ID>/
        ├── checkpoint.json
        └── part-<N>.jsonl.zst

The SHC reads each collection from <global.version.data.cti.path>/cti/<collection>/. The execution script mounts $SEKOIA_LOCAL_DIR on /opt/sekoia, which matches the default value of global.version.data.cti.path (/opt/sekoia/data/).

Import the CTI database

Retry with the default import policy

With modules.import_arango_collection.truncate=false and modules.import_arango_collection.on_duplicate=update (the defaults), the import updates documents with the same _key without clearing the collections first. If an import fails, fix the reported cause, then run it again. Check the import safety settings before retrying with a custom configuration.

To import the CTI database, run:

exec ImportArangoCollection

The SHC reads the ArangoDB root password of the platform from the arangodb-operator-dashboard secret of the support namespace. You do not pass it to the SHC. To use another password or another secret, set the following optional fields in config.yml:

Field Description Default
utils.arango.password ArangoDB password. When set, the SHC does not read the secret. Empty
utils.arango.password_secret.namespace Namespace of the secret holding the ArangoDB password. support
utils.arango.password_secret.name Name of the secret. arangodb-operator-dashboard
utils.arango.password_secret.key Key of the password in the secret. password

Import safety and tuning

These settings apply to every collection in the import. Set them in config.yml and restart the SHC after editing, or override them for one run with exec ImportArangoCollection --set KEY=VALUE. Runtime overrides do not change config.yml or later runs.

Field Description Default
modules.import_arango_collection.truncate true clears each target collection before importing its parts. Keep false for routine updates and failure retries. false
modules.import_arango_collection.on_duplicate Policy for documents whose _key already exists: update merges the imported fields, replace replaces the document, ignore keeps the existing document, and error reports a duplicate as an import error. update
modules.import_arango_collection.threads Positive number of part files imported concurrently. Lower it to reduce memory use in the SHC and concurrent load on ArangoDB. 8
modules.import_arango_collection.retries Maximum attempts per part file, including the first attempt, not additional retries. Must be at least 1. 5
modules.import_arango_collection.retry_backoff_seconds Non-negative base delay in seconds between failed attempts, doubled for each subsequent retry. 2

For example, to reduce concurrent imports for one run:

exec ImportArangoCollection --set modules.import_arango_collection.threads=4

Truncation deletes existing data before the import succeeds

With truncate=true, the SHC clears each target collection that has parts to import before loading its first part. A failure can leave that collection empty or only partly imported. There is no rollback. Each new invocation clears the collections again; automatic retries of a part do not. Use truncation only for a planned full replacement, with Sekoia support and a verified ArangoDB backup. See Back up the platform. Do not leave truncate=true in the configuration for routine imports.

Imports are not atomic, even with truncation disabled. Successful writes remain when another part or collection fails. A retry re-reads the whole part and can encounter documents written by a previous attempt: update applies the imported fields again, while error can fail on those duplicates. A duplicate policy is not a rollback mechanism. Inspect the final per-part and per-collection errors before retrying; increasing the retry count does not repair invalid input documents.

Select a CTI version

By default, the command imports the latest CTI export. To import another export, set global.version.data.cti.version:

Value Imported export
latest The latest completed export. This is the default.
A day, for example 2026-09-03 The most recent completed export of that day, for each collection.
An exact export, for example 2026-09-03T09 The export of that ID, for every collection. The SHC refuses it unless its checkpoint shows a completed export.

For example, to import the export of a given day for one run:

exec ImportArangoCollection --set global.version.data.cti.version=2026-09-03

In air-gapped mode, the SHC only finds the exports extracted on the orchestration node, and cti-latest.tar holds the latest export only. To import an older export, download the dated archive of that day, data/cti/archives/cti-<YYYY-MM-DD>.tar, instead of cti-latest.tar, and keep latest. The release bucket keeps the dated archives of the last 10 days.

Update the CTI database

The import is a snapshot: the CTI database of your platform does not update on its own. To refresh it, run the import again. In air-gapped mode, download and extract a new archive first.

Result

The command logs an Import progress line for each part it loads, then a summary of the documents created and updated per collection. Once every collection is imported, it deletes the relationships whose source or target object is missing from the database, and logs their count.

To confirm which export was imported, look for the Resolved export run log line of each collection. Its location field ends with the folder of the imported export, named after the export ID, an hourly timestamp such as 2026-09-03T09. With latest, the line also gives the run_id and the completed_at time read from latest.json. With a pinned day, check that the folder name starts with that day.

If a collection fails, the command still imports the other ones, skips the deletion of the relationships, and lists every failure at the end.