Reference Architecture · Simulated Scenario

Keeping AI Assistants Aligned with Data Platform Architecture

Q: How do we handle schema evolution — rules that change over time?

Update the YAML and set the old decision to status: deprecated. The history is preserved; enforcement is disabled. New rules take effect immediately on the next check.

Enforce warehouse standards, naming conventions, and pipeline constraints — before an AI assistant writes a single query that breaks your data contracts.

Install Mneme HQ View GitHub

The Problem

AI assistants treat your warehouse like a blank slate.

Your data platform has hard-won conventions: raw tables are append-only, reporting views are the only read layer, regional datasets must never be joined cross-region, and column naming follows a strict schema. None of that is visible to an AI assistant generating a dbt model or a Spark job.

The result is pipelines that write directly to raw tables, queries that bypass the semantic layer, and naming violations that break downstream consumers — discovered only after data quality alerts fire.

Without Mneme HQ — assistant suggests:

Prompt: "Add daily active users to the analytics pipeline"

I'll write a Spark job that reads from raw.events and writes the aggregated DAU count directly to raw.user_metrics for simplicity...

With Mneme HQ — pre-flight check:

mneme check --mode strict

✗ FAIL decision/no-writes-to-raw-tables
Rule: Raw tables are append-only via ingestion pipelines only.
Context: Direct writes corrupt audit trail and break CDC.

✗ FAIL decision/use-staging-layer-for-aggregations
Rule: All aggregations must land in staging.*, not raw.*

→ Surfaced 2 violations before code generation.

Why Existing Tools Fall Short

Data contracts and warehouse rules don't live in code.

Approach	Limitation	With Mneme HQ
dbt tests	Catch violations after the pipeline runs; can't prevent bad schema design up front	Decisions enforced before a model is written
Data catalog	Documents what exists; doesn't prevent AI from ignoring it	Catalog knowledge encoded as enforceable decisions
Schema docs	Static; not surfaced at prompt time; AI ignores attached context	Relevant constraints retrieved and checked per prompt
Code review	Catches pipeline mistakes after they're coded; wastes data engineer time	Pre-flight check before the model exists

How Mneme HQ Solves It

Data architecture decisions, enforced at prompt time.

Encode warehouse standards as decisions

Capture rules like layer boundaries, naming conventions, regional constraints, and anti-patterns in structured YAML.

Check before generating pipeline code

Run mneme check against your intended change. Mneme HQ retrieves relevant data architecture decisions and flags violations.

Generate scoped rules for data tools

Mneme HQ can output rules files for tools like Cursor or Claude Code that surface the right constraints when working in dbt, Spark, or SQL files.

Gate pipeline PRs in CI

Add mneme check --mode strict to your data pipeline CI. Schema violations and layer boundary breaks fail before merge.

Technical Implementation

What data platform decisions look like.

decisions/warehouse-layer-rules.yml

id: no-writes-to-raw-tables
title: Raw tables are append-only via ingestion pipelines
status: accepted
rule: No pipeline or model may write directly to raw.* tables.
  All aggregations and transformations must land in staging.* or marts.*
rationale: Direct writes break CDC audit trail and corrupt source-of-truth.
  Enforced after data corruption incident Q2 2024.
enforcement: strict
tags: [data-platform, warehouse, layers, anti-pattern]

decisions/naming-conventions.yml

id: column-naming-snake-case
title: All warehouse columns must use snake_case
status: accepted
rule: Column names must be lowercase snake_case. No camelCase, no PascalCase.
  Event timestamps must end in _at. Dimension keys must end in _id.
rationale: Downstream BI tools and dbt macros depend on consistent naming.
enforcement: strict
tags: [data-platform, naming, conventions]

Terminal — checking a dbt model prompt

$ mneme check "create a DAU model that reads from raw events" --tags data-platform

Checking against 8 data-platform decisions...

✗ FAIL decision/no-writes-to-raw-tables
  Reason: Model reads from raw.* — should use staging.events source.
✗ FAIL decision/use-staging-layer-for-aggregations
  Reason: DAU aggregation should land in marts.*, not raw.*
✓ PASS decision/column-naming-snake-case
✓ PASS decision/regional-dataset-isolation

Result: FAIL (2 violations, strict mode)

Simulated Outcome

What data teams see after enforcement goes live.

100%

of layer boundary violations caught before pipeline code is written

raw table write incidents after decision enforcement

~3 min

to encode a new warehouse standard as an enforceable decision

These figures are based on a simulated reference scenario — not live customer data.

FAQ

Common questions.

Does this work with dbt, Spark, and SQL-based pipelines?

Yes. Mneme HQ operates at the prompt level — it doesn't depend on your specific data tool. Whether you're generating dbt models, Spark jobs, or raw SQL, mneme check runs against the intent before you write any code.

Can we enforce regional data isolation rules?

Yes. Encode regional constraints as tagged decisions (e.g., --tags gdpr,eu-region) and check with mneme check --tags gdpr to scope enforcement to region-specific rules.

How do we handle schema evolution — rules that change over time?

Update the YAML and set the old decision to status: deprecated. The history is preserved; enforcement is disabled. New rules take effect immediately on the next check.

Can data engineers without Python experience manage decisions?

Yes. Decision files are plain YAML in version control. Any engineer comfortable opening a PR can add or update rules — no Python or programming required.