Storage and Indexes

An index is the “directory�?that stores parsed logs. This page shows how to review index volume in DataOri / Qinglong SIEM and control rollover, retention, and delete with an index storage policy. It is for SecOps and implementation engineers who must balance compliance keep-time, disk capacity, and search experience.

Version note (2.4): Current product documentation. Behavior follows the 2.4 environment.

What you can do

• In Index Management, see which indexes exist, how much space each uses, and whether they grow
• In Index Storage Policy, set rollover limits (size, document count, age) and delete retention
• Rough-size capacity from daily growth so you neither keep too long (slow search) nor too short (compliance risk)
• Before shortening retention or enabling delete, spot-check boundaries and confirm backup—reduce accidental wipe risk
• After the policy applies, verify in Security Data Search what is still queryable inside vs outside the window

Before you start

• Permission for Index Management and Index Storage Policy
• Typical menu path: System Settings �?Data Management �?Index Management / Index Storage Policy (UI labels win)
• Ingest and parsing must work: logs arrive via Data Collection and are written by Parsing rules / ETL before meaningful volume appears
• Know the minimum retention days required by compliance or business rules before you change the policy
• This page covers in-product index and policy actions only. Paths and capacity notes: Storage component configuration. No deep cluster tuning here

Common tasks

1. Review indexes and volume (Index Management)

1. Open System Settings �?Data Management �?Index Management
2. Check index name and storage size (or equivalent columns)
3. Filter by name to find business indexes; confirm recent writes and growth
4. Compare with yesterday / last week (or your ops notes): a sudden spike often means a new source or ETL volume change—review that before tightening the policy
5. Note the largest indexes; focus on them when you change the policy later
6. If the list stays empty or size stays 0: fix collection and ETL first—do not start by shortening delete days

2. Configure the index storage policy (rollover / retention / delete)

1. Open Index Storage Policy in the same area
2. Read current values; screenshot or copy them to a change ticket for rollback
3. Under basic settings, set rollover triggers (any one may trigger—follow on-screen help):
   • Maximum index size: size limit before the active index rolls to a new one
   • Maximum log count: document-count limit that triggers rollover
   • Maximum period: age limit that triggers rollover (often in days)
4. Under delete, confirm whether Activate Delete is on:
   • When enabled, set time to delete (keep N more days after expiry, then delete; field label follows UI)
   • Turning delete off means no automatic cleanup; disk keeps growing—use only with a clear archive/backup plan
5. Check: the new retention window is �?your compliance minimum; do not cut below that just to free disk once
6. Save. Rollover and delete run on the policy schedule; saving does not wipe history instantly
7. Re-open the page and confirm values match what you saved

3. Decide retention with capacity and search together

1. From Index Management or ops monitoring, take recent daily growth (size). Estimate: daily growth × target retention days × (1.2�?.3) �?disk needed (with headroom)
2. Open Security Data Search and try wide ranges (e.g. last 7 / 30 days). If searches often slow or time out, shrink the query window first and check whether indexes are far past expected size
3. When compliance allows, shorten online retention usually stabilizes search better than keeping every day online; long-term keep uses backup/archive
4. Rare rollover (one huge active index) also hurts search: tighten size/period slightly from site defaults so indexes stay smaller chunks
5. After changes, watch a few days: disk should track the retention window—not “only grow forever�?

4. Prevent accidental deletion (before shorter retention or enabling delete)

1. Confirm target retention days �?compliance/business floor; keep a ticket or written basis
2. Confirm a recent backup or archive is usable if you rely on offline keep; do not combine “no backup + first-time delete on�?with a large cut
3. In Security Data Search, spot-check the boundary about to fall out of the window; note reproducible queries and hit counts
4. If the UI shows preview, effective-time, or countdown hints, read them before confirming save
5. Within 24�?2 hours after save, recheck: data inside the window still searchable; clearly outside should be gone online (or restorable from archive)
6. If you set retention too short by mistake: restore a compliant value and save ASAP; already deleted data needs backup restore—the product cannot “undo delete�?
7. On production, have a second person review the key fields in the parameter table before save

Parameter reference (Index Storage Policy)

Do not put real production passwords, customer public IPs, or unredacted connection strings in docs, screenshots, or tickets.

FAQ (this module)

1. Symptom: Index Management shows almost no volume, but collectors look busy �?Cause: Logs are not parsed into the expected index, or they land under another index/tenant view. �?Fix: Check ETL is enabled without ongoing failures; query a short recent window in Security Data Search; align collector targets with ETL output indexes (Data Collection).

2. Symptom: After saving a shorter retention, history vanishes at once—or nothing changes �?Cause: Delete runs asynchronously on a schedule; saving is not an instant full wipe. Or delete is still off, so disk only grows. �?Fix: Re-open the policy and confirm Activate Delete plus days; wait one policy cycle and spot-check boundary ranges; avoid swinging parameters repeatedly.

3. Symptom: Security Data Search keeps getting slower; disk nears alert �?Cause: Online retention too long, query time range too wide, or rollover too rare so one shard of data is huge. �?Fix: Shorten pre-delete retention if compliance allows; shrink search windows; tune rollover size/period slightly. Capacity and path notes: Storage component configuration. This page does not cover cluster JVM/shard tuning.

4. Symptom: Fear of deleting logs still inside the compliance window �?Cause: Delete days set below the compliance floor, or backup never confirmed. �?Fix: Follow “Prevent accidental deletion�?above; retention must be �?compliance days; do not enable aggressive delete without archive while also ignoring disk monitoring.

Verify success

• Index Management opens and shows names plus storage size
• Index Storage Policy saves and reloads with the same values
• Within the retention window, logs appear in Security Data Search
• Clearly outside the window, that range is no longer searchable online (or is archived per your plan)
• Disk use tracks daily growth and retention as expected—no unbounded “only grows�?pattern (when delete is on)
• The change ticket keeps before/after parameters for audit or rollback

Related

• Data Collection
• Parsing rules / ETL
• Security Data Search
• Storage component configuration (config/capacity view; complements this page)

:::caution Docs are browse-only. Source files and bulk downloads are not provided. :::