# Portability: SYSTEM
# Last validated: 2026-05-17
# Next review: 2026-11-15
# Resources: [hub/consolidation.py, memory_consolidation table]

MEMORY CONSOLIDATION
====================

STATUS: 2026-05-17

WHAT IS CONSOLIDATION?
----------------------
Consolidation is the active process that creates meaningful structures from
raw data (sessions, lessons, working memory) and transforms them into context.

Analogous to human sleep:
- Experiences are processed, important things retained.
- Repetition strengthens connections (boost), unused things fade (decay).
- Summarization reduces details, retains the essence.

CONSOLIDATION LEVELS (Pipeline)
-------------------------------

  RAW DATA (Sessions, Working Mem) --[Analysis]--> ASM (Metadata)
  ASM (Metadata) --[AI Review]--> ESSENCE (Lessons, Context)
  ESSENCE --[Indexing]--> TRIGGER-DB (Associative Memory)

TABLES (v1.1.80+ Active)
-------------------------
  memory_sessions      Raw data (360+ entries)
  memory_lessons       Cleaned insights (70+ entries)
  memory_consolidation Tracking & scores (350+ entries)
  context_triggers     Associative bridges (900+ triggers)

CLI COMMANDS (bach consolidate)
-------------------------------
  status       Shows statistics and pending consolidations.
  run          Executes all available consolidation steps (weight, archive, index, sync-triggers, forget).
  compress     Compression of sessions to context entries.
               --cleanup: Clean up empty sessions
               --batch: Group sessions by day
               --run: Full compression with rule set
  weight       Update relevance scores (decay/boost).
  archive      Move outdated knowledge to long-term archive.
  index        Reconcile between facts, help and wiki.
  review       Creates review tasks for manual validation.
  init         Initialize tracking for existing entries.
  sync-triggers Update dynamic context triggers (NEW v1.1.80).
  forget       Delete unused entries (weight < threshold).
  reclassify   Correct miscategorized entries (NEW v1.1.81).

DAEMON INTEGRATION
------------------
Consolidation can run as a background job (scheduler_jobs):
- `consolidate-weight`: Daily (decay simulation).
- `consolidate-archive`: Weekly (archive check).
- `consolidate-index`: Update facts index.

(Note: Daemon jobs must be configured - see docs/help/daemon_en.txt)

TRUTHFULNESS:
Every consolidation run generates log entries and creates review tasks
(#category: maintenance) when in doubt.

SEE ALSO
--------
  docs/help/memory.txt          Cognitive memory model
  docs/help/lessons.txt         Best practices & insights
  docs/help/daemon.txt          Daemon jobs
  hub/consolidation.py          Logic implementation
  ARCHITECTURE.md               System architecture (memory model)

SUB-PROCESSES (ALREADY EXISTING)
--------------------------------
These existing tools are part of the consolidation pipeline:

  tools/autolog_analyzer.py     Session analysis (ASM_003)
  tools/context_compressor.py   Compression (ASM_004/005)
  skills/workflows/wiki-*       Wiki authors
  skills/workflows/help-*       Help authors

DATABASE SCHEMA
---------------
memory_consolidation (ACTIVE, 350+ entries):

  id                INTEGER PRIMARY KEY
  source_table      TEXT        -- memory_sessions, memory_lessons, etc.
  source_id         INTEGER     -- ID in source table
  times_accessed    INTEGER     -- Access counter
  last_accessed     TIMESTAMP   -- Last access
  weight            REAL        -- Relevance score (0.0-1.0)
  decay_rate        REAL        -- Decay rate
  threshold         REAL        -- Archiving threshold
  status            TEXT        -- active, archived, deleted
  consolidated_to   INTEGER     -- ID in memory_context (when consolidated)
  created_at        TIMESTAMP
  updated_at        TIMESTAMP

WORKFLOW: SESSION -> CONTEXT
----------------------------
1. Session ends (--shutdown)
2. autolog_analyzer.py extracts activities
3. context_compressor.py creates summary
4. Summary stored in memory_sessions
5. [DAEMON] After X sessions: compression to context
6. [AI REVIEW] Checks structure, corrects
7. Result in memory_context with triggers

WORKFLOW: FACT -> WIKI/HELP
---------------------------
1. New fact is created: bach --memory fact "topic:value"
2. consolidate-index checks: Does help/wiki for "topic" exist?
3. If not: Create task "Create wiki entry for [topic]"
4. Wiki author creates entry
5. Fact is updated with path reference

WORKFLOW: WIKI/HELP -> FACTS INDEX
----------------------------------
1. consolidate-index scans docs/help/*.txt and wiki/*.txt
2. For each entry: Does a fact exist?
3. If not: Create fact as index
   HELP.memory -> "Memory System Documentation"
   WIKI.gemini -> "Google Gemini AI Models"
4. Facts serve as quick lookup

THRESHOLDS (CONFIGURABLE)
--------------------------
  weight_threshold_archive: 0.2   Below this value: archive
  weight_threshold_delete:  0.05  Below this value: delete
  decay_rate_default:       0.95  Daily decay (5%)
  boost_on_access:          0.1   Weight increase on access
  sessions_before_compress: 10    Sessions until compression
  days_before_archive:      90    Days until archiving

NEW OPERATIONS (v1.1.80+)
--------------------------
  sync-triggers    Updates dynamic triggers from:
                   - workflow_trigger_generator.py
                   - lesson_trigger_generator.py
                   - tool_auto_discovery.py
                   - theme_packet_generator.py
                   - trigger_maintainer.py

  forget           Deactivates/deletes entries with weight < 0.05
                   - memory_lessons: is_active = 0
                   - memory_working: is_active = 0
                   - memory_facts: DELETE

  reclassify       Corrects wrong categorizations:
                   - Lesson -> Context (high usage)
                   - Working -> Lesson (lesson pattern detected)
                   - Working -> Fact (key:value format)
                   - Fact without wiki -> create task

                   Manual conversion:
                     bach consolidate reclassify lesson 42 context

                   Automatic analysis:
                     bach consolidate reclassify
                     bach consolidate reclassify --fix
