AiSOC

简介

What's new

VERSION is 7.3.1; everything below is captured under [Unreleased] in CHANGELOG.md and will tag with the v8.0 cut.

Latest — security & stability (May 27–28, 2026) — hardening, dependency, and boot-reliability work merged into main. - Security Audit green — cryptography floor raised to 44.0.1 to clear CVE-2024-12797 and later 42.x advisories across services/connectors and services/osquery-tls; advisories without an upstream fix are time-boxed (90-day expiry) in scripts/security_audit_ignores.txt (#229). - Tenant-isolation fix — detection-loop suggestion lookups are now scoped to the caller's tenant, closing a cross-tenant read path (#221). - Full stack boots clean — the reserved window column is now quoted and pydantic[email] ships in the image, so docker compose comes up end-to-end without manual patching (#227). - OpenAPI auto-export unblocked — the spec-export CI job now has contents: write, so the committed OpenAPI document re-syncs on every merge (#228). - CodeQL quality notes cleared — remaining low-severity CodeQL findings resolved on main (#224). - Dependency refresh — zod 3 → 4.4.3 (#225), recharts 2 → 3.8.1 (#209), plus a Dependabot sweep across fastapi, uvicorn, pydantic, structlog, openai, weasyprint, strawberry-graphql, prometheus-client, go-chi, turbo, and @types/react. - Credits — new Credits section thanking contributors and security researchers (#223).

Console workbenches (v1.5 PR-1 → PR-6) — the SOC operator surface is now a workbench, not a list. - Global time-window selector + topbar context — one selector at the top of the console drives every page (Alerts, Cases, Hunts, Funnel KPIs, Pipeline Health). Persists across reloads, deep-linkable as a URL param. - Tenant switcher + role badge — MSSP operators flip tenants from the topbar; the role badge makes it impossible to confuse a viewer session with an admin session. New endpoint: GET /api/v1/tenants/me/identity. - Critical severity tier — the severity ladder is now info | low | medium | high | critical. Vendor-native criticals (Azure 5-tier, GCP SCC, GitHub critical, ServiceNow priority 1, GuardDuty ≥ 8.0, AuditD identity-destruction, K8s cluster-admin, Tailscale tailnet lockdown) map straight through instead of being collapsed into high. Confidence (alert.confidence, 0–100, band low|medium|high) is now decoupled from severity and emitted by services/fusion ConfidenceScorer. - Operations funnel + pipeline health — new /metrics/funnel and /health/pipeline endpoints feed the FunnelKpiBar (Detected → Triaged → Investigated → Resolved) and an Efficiency Report so SOC leads can answer "where are we losing time?" without a Grafana detour. Docs: apps/docs/docs/console/funnel-kpis.md. - Investigation Rail (W6 / PR-4) — /alerts is now a two-pane workbench with narrative, related entities (pivotPath deep links), 6-event mini-timeline, and structured recommended actions. Fusion writes a deterministic correlation narrative at fuse time. Docs: apps/docs/docs/console/investigation-rail.md. - Investigation Queue workbench (PR-5 / W7) — /queue is the page a Tier-1 analyst lives on: server-anchored SLA countdowns, atomic claim semantics, one-click triage actions. Docs: apps/docs/docs/console/queue.md. - Rule Tuning workbench (PR-6 / W8) — /detection/tuning ranks noisy rules by precision impact and ships one-click suppression + allow-list edits with full audit trail. Docs: apps/docs/docs/console/rule-tuning.md. - Zero-prerequisite installer — install.sh / install.ps1 now bootstrap from a clean machine (Docker, Compose, Node, pnpm, Python) with idempotency and a graduated uninstall.sh. Documented in apps/docs/docs/installation.md, surfaced as Path 0 in the quickstart.

v8.0 wave-1 (architectural foundation, PR #125) — the foundation for the v8.0 line. - Graph at ingest — Neo4j entity graph (17 node labels, 14 edge types) written inline with Kafka consumption. Batched UNWIND upserts + fire-and-forget retry queue keep ingest latency budget intact. Schema doc: apps/docs/docs/architecture/graph-schema.md. - Four-agent rebrand — DetectAgent, TriageAgent, HuntAgent, RespondAgent are now the public façade; back-compat aliases preserve existing imports. Funnel KPI doc: apps/docs/docs/console/funnel-kpis.md. - /hunt natural-language surface — type a hypothesis in English, get ES|QL / SPL / KQL templates back, save and schedule the hunt. HuntAgent never writes raw queries. Saved hunts deep-link into the Investigation Rail via pivotPath. - Sixteen first-party connectors — wave-1 (tines, torq, falco, pagerduty, opsgenie, confluence_audit) and wave-2 fixtures (cloudflare_zt, sysdig, vault, snowflake). Five severity tiers preserved end-to-end. - L0–L4 automation maturity model — apps/docs/docs/concepts/automation-maturity.md plus the marketing surfaces. Ladder: L0 manual → L4 fully autonomous closure with human sign-off. - Public weekly benchmark scoreboard — apps/docs/docs/benchmark-scoreboard.mdx reads apps/docs/static/data/scoreboard.json, refreshed weekly by .github/workflows/wet-eval.yml. Substrate rows are visually separated from wet-eval rows — substrate numbers can never be quoted as live agent performance.

Security & correctness wave — 12 critical/high CVE-class fixes shipped before the v8.0 cut. See apps/docs/docs/operations/security.md for the full inventory. - Rule-engine eval() RCE eliminated — conditions are parsed to a whitelisted AST in services/api/app/services/rules_engine.py (#116). - /hunts and /cases tenant isolation enforced at the query layer (WHERE tenant_id = …), not via RLS alone (#117, #118). - CORS lockdown — a shared cors.py is vendored byte-identical into every Python service and refuses to start with * + credentials in production (#119). - Playbook SSRF guard — every outbound http_request / notify runs through services/agents/app/playbook/ssrf_guard.py with a cloud-metadata block list (#120). - Plugin-manager OCI install hardening — signed manifests verified against an allow-list, image digests pinned and re-verified on every load (#121). - Audit-log integrity (H-4 + M-12) — actor_ip spoofing closed via the new TRUSTED_PROXIES allow-list, secrets stripped from changes, hash-chain tamper-proofing (#122). - /alerts/submit abuse + replay hardening — payload caps (events / per-event bytes / total bytes), Idempotency-Key header, recursive raw_event redaction, timestamp clamping (#123). - Pydantic v1 → v2 settings migration (#124), bounded eval() + playbook timeouts (#126), one-flag dev-mode (AISOC_DEV_MODE — supersedes DEV_MODE / SKIP_AUTH / AISOC_DEMO_MODE, #127), untrusted-enrichment sanitisation before LLM (#128). - Python CodeQL alert count on main driven to zero (#133, #136, #137); enforced as a CI gate going forward. - First community contribution merged: #135 (UEBA env-var alignment, closes #134). Every UEBA variable accepts both unprefixed (DATABASE_URL) and legacy (UEBA_DATABASE_URL) forms; unprefixed wins.

Stage 2 / Stage 3 platform additions — landed alongside v8.0 wave-1. - Wazuh Indexer ingest connector — polls wazuh-alerts-* over HTTPX, paginates time-windowed queries, retries on 5xx; collapses Wazuh severity into the AiSOC ladder. Docs: apps/docs/docs/connectors/wazuh.md. The connector registry now declares 52 first-party connectors. - auditd file_tail connector + aisoc.rules profile — replaces the host-agent dependency for Linux endpoint visibility; 4 new detections pivot on the bundled aisoc_* audit keys. Docs: apps/docs/docs/connectors/auditd.md. - Live Actions dispatcher — generic vendor/capability surface so plugins can register executors against the in-tree taxonomy (isolate_host, disable_user, block_ip, …) without forking. Unknown pairs return a typed LiveActionResult(FAILED, "executor_not_found") — never a 500. Docs: apps/docs/docs/concepts/live-actions.md. - Deterministic NL → ES|QL / KQL / SPL translator — replaces the template fallback in /nl_query with an IR + grammar validator; 50-pair gold eval set scores 100% syntactic, 100% semantic. Air-gapped by default; optional gpt-4o-mini enhancement falls back deterministically. - STIX → MISP push — every STIX 2.1 indicator/bundle published through /api/v1/threatintel/stix/... can now be mirrored into the configured MISP instance. Air-gap gated, with a ?push_to_misp=true query param and a dry-run endpoint for air-gapped audits. Docs: apps/docs/docs/integrations/misp-push.md. - GCP Cloud Run + Cloud SQL Terraform skeleton — serverless-first BYOC equivalent of the existing AWS module. One terraform apply stands AiSOC up on GCP with private-IP networking, Secret Manager, and Artifact Registry. Docs: apps/docs/docs/deployment/gcp.md. - Blameless case post-mortem endpoint — GET /api/v1/cases/{case_id}/postmortem?format=json|html produces a deterministic retrospective covering contributing factors, detection timing/gaps, response phases, blast radius, and action items. Analyst handles are explicitly redacted from the narrative. Docs: apps/docs/docs/operations/case-reports.md. - Per-rule cross-fire FP gate — services/agents/tests/test_detection_fp_rate.py replays every rule's match_when against every other rule's positive fixture; current corpus 816 native rules, worst FPR 0.49% (5% ceiling). Wired into scripts/run_evals.py as suites.detection_fp_rate. - Operator-facing documentation refresh — new pages for notifications, plugin lifecycle, and credentials / vault rotation; v2.2 architecture diagram and the corrected 52-connector count (now including Wazuh Indexer + auditd file_tail) rolled through every surface.

The full inventory (with file paths, env-var changes, and test counts) lives in the [Unreleased] section of CHANGELOG.md.

---

What's in the box

AiSOC bundles the components a SOC normally pieces together from separate vendors:

• Connect data sources in three clicks — a 50-connector click-and-connect catalog spans EDR/XDR (CrowdStrike Falcon, SentinelOne, Microsoft Defender XDR, Palo Alto Cortex XDR, Cortex XSIAM, VMware Carbon Black, Trellix Helix, Trend Vision One), SIEM (Splunk, Microsoft Sentinel, Elastic, Sumo Logic, Datadog Cloud SIEM, Google Chronicle, Rapid7 InsightIDR), cloud + CNAPP (AWS Security Hub, AWS GuardDuty, AWS CloudTrail, AWS VPC Flow Logs, Azure Activity, Azure Defender, GCP Cloud Audit, GCP SCC, Wiz, Lacework, Tenable, Prisma Cloud, Orca), identity (Okta, Microsoft Entra, Auth0, Duo Security, 1Password), SaaS (Microsoft 365 audit, Google Workspace, Cloudflare, Proofpoint, Mimecast, ServiceNow, Jira, Slack audit, Salesforce, Email inbox), VCS (GitHub, Snyk), endpoint fleet (osctrl, FleetDM for fleet-wide osquery), container + orchestration (Kubernetes audit logs via apiserver webhook or audit.log tail), and network (Tailscale, Zscaler, Cisco Umbrella). Each connector renders a schema-driven form, runs a live Test connection round-trip before save, encrypts every secret with the application-layer CredentialVault (Fernet AES-128-CBC + HMAC-SHA256), and starts polling on a per-instance schedule. Walkthrough: docs/connectors. Threat model + key rotation: docs/operations/credentials.
• Own your endpoint telemetry — first-party aisoc-osquery-tls FastAPI service (services/osquery-tls/) and aisoc-direct lightweight agent connector ship a self-hosted osquery TLS plugin, FleetDM-compatible config/log endpoints, and a direct-from-agent ingest path that bypasses third-party SaaS. Built-in file integrity monitoring (FIM) endpoint (services/osquery-tls/app/api/v1/endpoints/fim.py) ingests file_events and synthesizes alerts on writes to /etc/passwd, /etc/shadow, sshd configs, sudoers, and Windows registry hives; bundled osquery packs cover incident response, OSquery-ATT&CK, and FIM out of the box. 16 native osquery detections (detections/endpoint/osquery-*.yaml, IDs det-endpoint-281..296) cover credential access, persistence, lateral movement, defense evasion, and discovery — paired with positive/negative test fixtures (detections/fixtures/osquery_*.json) and CI-gated against the Detection Validation workflow. Live-query playbook step (osquery_live_query) lets responders push allowlisted distributed queries to single hosts or fleet-wide via osctrl/FleetDM with HMAC-signed ChatOps approval. 5 custom Go-based virtual tables (services/osquery-extensions/) extend the agent with aisoc_browser_extensions, aisoc_kernel_modules, aisoc_attck_persistence, aisoc_pending_actions, and aisoc_alert_cache for richer endpoint visibility and bidirectional response. Walkthroughs: docs/connectors/osctrl, docs/connectors/fleetdm.
• Ingest events from any connector into a Kafka spine.
• Correlate them in real time with deduplication, ML scoring, per-alert confidence scoring, and Sigma/YARA detection.
• Roll up signal onto entities — Risk-Based Alerting accumulates time-decayed risk points on the user, host, IP, and domain each alert touches, promotes them to entity-incidents at a tunable threshold, and surfaces an entity-centric queue in the alerts UI. Hits the published 2026 KPI bar of ≥ 50:1 alert-to-incident ratio (CI-gated in services/fusion/tests/test_entity_risk.py).
• Search across SIEMs — Federated Search fans out a single query to connected Splunk, Microsoft Sentinel, and Elastic instances, translating the query into each target's native dialect (SPL, KQL, ES|QL) via pluggable translators in services/connectors/app/federated/.
• Manage detections as code — Detection-as-Code (DAC) provides a propose → review → eval-gate → promote lifecycle for detection rules. Every proposal carries an eval result from the harness; candidates that regress MITRE accuracy cannot be promoted. Endpoints in services/api/app/api/v1/endpoints/detection_proposals.py.
• Run hypothesis-driven hunts on a schedule — Hunt-as-Code YAML definitions in hunts/ declare a hypothesis, MITRE ATT&CK tags, log sources, indicators, and a cron schedule. The hunt engine in services/agents/app/hunt/ loads the corpus at startup, runs hunts on their schedule, and stores findings in the DB.
• Track detection drift — the Purple Team service takes ATT&CK coverage snapshots and diffs them over time, so you can see which techniques gained or lost coverage between releases. Implementation in services/purple-team/app/services/drift.py.
• Verify ChatOps actions — HMAC-signed approval prompts are sent to Slack or Teams before high-impact SOAR actions execute, with a time-limited verification token. Implementation in services/actions/app/executors/chatops.py.
• Benchmark against adversary LLMs — a deterministic attacker-LLM mutator generates adversary incidents to test detection resilience. Script: scripts/generate_adversary_incidents.py; eval: services/agents/tests/test_adversary_eval.py.
• Enrich every signal with threat intelligence from TAXII 2.1, MISP, OTX, and CISA KEV.
• Reason about attacks via a LangGraph multi-agent system grounded in MITRE ATT&CK.
• Detect deviations with UEBA — per-user behavioural baselines and Z-score anomaly scoring.
• Trap adversaries with HMAC-signed honeytokens (URLs, files, AWS credentials, emails).
• Validate coverage with automated Atomic Red Team and Caldera adversary emulation.
• Respond with blast-radius-aware SOAR actions, every step explainable.
• Govern with multi-tenant RLS, granular RBAC, immutable audit logs, and SOC 2 / ISO 27001 / NIST CSF / PCI-DSS / HIPAA / DORA evidence dashboards.
• Manage at scale with an MSSP parent-tenant console — onboard child tenants, delegate actions cross-tenant, and view rollup metrics in one pane.
• Track assets with an asset inventory that auto-correlates vulnerabilities to alerts and surfaces asset blast radius.
• Detect insider threats with user risk profiles, behavioural indicators, and peer-group deviation scoring.
• Gate automation through L0–L4 maturity tiers — each tier unlocks progressively more autonomous remediation, with per-action whitelist and full audit gate log.
• Generate internal threat intelligence — harvest IOCs from alert history, track threat actors and campaigns, subscribe to external STIX/TAXII feeds, all queryable via the REST API.
• Assess cloud posture with a built-in CSPM/KSPM engine that ingests findings, tracks drift between scan runs, and surfaces a per-provider summary with suppress/resolve workflows.
• Correlate through identities with a graph of users, devices, and service accounts; link alerts to identity nodes for blast-radius queries and attack-path reconstruction.
• Automate board reporting — schedule PDF/HTML executive summaries, store artefacts, and deliver via email or webhook.
• Three-tier agent memory — session (in-process LRU), working (Redis-backed, 24 h TTL), and institutional (PostgreSQL + pgvector, permanent). Agents carry context across tool calls, cases, and sessions; institutional knowledge survives restarts.
• Autonomy guardrails — per-action confidence thresholds (e.g. block_ip ≥ 0.90, close_alert ≥ 0.60) gate every autonomous decision. Tenant admins can tighten or loosen thresholds via API; all guard-rail decisions are logged.
• Investigation cost telemetry — every LLM call is tracked by model, prompt tokens, completion tokens, latency, and estimated USD cost. Aggregates are persisted per-run to aisoc_run_costs and surfaced in SOC dashboards.
• SOC metrics dashboard — live MTTD, MTTR, False Positive Rate, alert/case volumes (rolling 7 d), and ATT&CK technique heatmap. Backed by a real-time API endpoint and a polished React component.
• Analyst-override feedback loop with retroactive re-disposition — analysts correct AI verdicts (true_positive, false_positive, benign, escalate) in one click. Corrections persist on the alert, flow into aisoc_institutional_memory keyed by an alert signature (category + connector + primary MITRE technique), and adjust FPR metrics automatically. The API surfaces retroactive candidates — past alerts in the same tenant matching the same signature whose disposition would now flip — for one-click bulk re-disposition.
• Natural-language detection authoring — describe a threat in plain English; the API translates it to Sigma YAML, KQL (Microsoft Sentinel), SPL (Splunk), and ES|QL (Elastic) simultaneously. Falls back to curated templates when no LLM key is configured.
• Closed-loop detection engineering — when an alert is marked as a false positive, the agent drafts a Sigma rule fix using an LLM, then automatically creates a Detection-as-Code proposal routed through the same human-review DAC workflow. CI re-runs evals on approval; regression gates block regressions.
• Natural-language query execution — ask a security question in plain English (POST /nl-query/execute). The API translates it to ES|QL, SPL, and KQL; for Elasticsearch-backed tenants it executes the ES|QL query live and returns structured results, column metadata, and the query text for all three dialects.
• Identity-centric investigation timeline — build a chronological event timeline anchored to any user, device, service account, or IP (POST /identity-timeline/build). The timeline queries alerts and raw events, annotates each event with the relevant ATT&CK technique, computes an entity risk score, and returns a sorted, deduplicated event list for triage.
• Cross-platform detection translation — convert any detection rule bidirectionally between Sigma YAML, Splunk SPL, Microsoft Sentinel KQL, Elastic ES|QL, and Google Chronicle YARA-L2 / UDM Search (POST /translation/translate). An LLM handles complex logic; a regex fallback handles simple field-mapping rules with no external dependency.
• Hypothesis-driven hunt workbench — define a hunt hypothesis in natural language (POST /hunts); the API auto-generates ready-to-execute queries in ES|QL, SPL, and KQL; analysts record findings against any run and the workbench tracks open, completed, and inconclusive hunts.
• Phishing triage workflow — submit raw email text, URLs, attachments, or domain indicators (POST /phishing/submit); the LLM extracts IOCs, assigns a verdict (phishing / benign / spam / malware / unknown), maps to MITRE ATT&CK, and optionally links the submission to an existing case.
• Knowledge-base + RAG — ingest runbooks, policies, SOPs, and wikis (POST /kb/ingest); the API chunks and full-text indexes each document; analysts query with natural language (POST /kb/query) and receive the top matching chunks plus an LLM-synthesised answer with citation, backed by PostgreSQL FTS when no vector store is configured.

Highlights

git clone https://github.com/beenuar/AiSOC.git && cd AiSOC && pnpm aisoc:demo

pnpm aisoc:demo --quick  # 4 cases in 4 minutes

git clone https://github.com/beenuar/AiSOC.git
cd AiSOC
pnpm aisoc:demo

#### 3. Boot