# Data Provenance — albert. Training Corpus

**Model:** albert. v3.0  
**Maintainer:** RFI-IRFOS, contact@ternlang.com  
**Last updated:** 2026-05-14

This document satisfies the data governance and provenance requirements
of EU AI Act Regulation (EU) 2024/1689, Article 53(1)(c) for the albert.
General-Purpose AI model.

---

## Governance Principles

1. **No personal data.** The corpus contains no PII, private
   communications, or data subject to GDPR data subject rights. All
   sources are either public-domain, openly licensed, or aggregated
   public content with no individual identification.

2. **Opt-out contact.** Any source operator or rights holder who wishes
   to dispute inclusion of their content should contact
   contact@ternlang.com. Disputed content will be reviewed and removed
   from future training runs within 30 days.

3. **Reproducibility.** All ingestion scripts are version-controlled in
   `albert-moe-13/scripts/`. The exact corpus state at any training run
   can be reconstructed from the corpus cache (`data/corpus_cache.bin`).

4. **Chaos invariant.** Approximately 10% of training tokens at all
   times are synthetic noise, adversarial patterns, and mixed-language
   text generated locally. This corpus tier contains no third-party
   content.

---

## Corpus Sources

### Tier 1 — Public Domain Books (Project Gutenberg)

| Property | Value |
|----------|-------|
| Source | Project Gutenberg (gutenberg.org) |
| Access method | Direct HTTP download via `download_corpus.py` |
| Content | Public-domain literary and reference texts, multilingual |
| Approximate share | ~30% of training tokens |
| Legal basis | Public domain (pre-1928 works; Gutenberg license) |
| Copyright status | No copyright — all works are confirmed public domain |
| User-Agent disclosed | `AlbertCorpusBot/1.0 (contact@ternlang.com)` |

No copyright exceptions are needed. All Gutenberg texts used are
explicitly released to the public domain by Project Gutenberg.

---

### Tier 2 — Wikipedia (Multilingual)

| Property | Value |
|----------|-------|
| Source | Wikimedia Foundation — English, German, French Wikipedia |
| Access method | Wikimedia REST API (`ingest_wiki.py`, `download_corpus.py`) |
| Content | Encyclopedia articles; factual prose |
| Approximate share | ~25% of training tokens |
| Legal basis | Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0) |
| Attribution | Wikimedia Foundation / Wikipedia contributors |
| Terms compliance | Non-commercial research use; no redistribution of raw dumps |
| User-Agent disclosed | `AlbertCorpusBot/1.0 (contact@ternlang.com)` |

CC BY-SA 4.0 permits use in ML training datasets provided attribution is
maintained. Attribution is recorded in this document and in the corpus
metadata files.

---

### Tier 3 — OpenWebText (Filtered Common Crawl)

| Property | Value |
|----------|-------|
| Source | OpenWebText dataset (Gokaslan & Cohen, 2019) |
| Access method | Pre-processed corpus download |
| Content | Web text filtered by Reddit upvote signals |
| Approximate share | ~15% of training tokens |
| Legal basis | Research use; Common Crawl content is publicly accessible |
| Copyright status | Mixed; Common Crawl content is subject to individual page rights |
| Filtering applied | Quality filter: length > 100 words, no adult content signals |
| Known limitation | Individual page copyright cannot be verified at corpus scale |

Use of Common Crawl-derived corpora for non-commercial AI research is
established practice and is consistent with the EU AI Act Article 53(1)(c)
exception for text and data mining for research purposes under DSM Directive
Article 3 (Directive 2019/790/EU).

---

### Tier 4 — Technical Content (GitHub, HN, Developer Blogs)

| Property | Value |
|----------|-------|
| Source | GitHub Issues (public repos), Hacker News discussions, developer blogs |
| Access method | `ingest_github.py`, `build_stage10.py` |
| Content | Technical debugging discussions, bug reports, Q&A |
| Approximate share | ~10% of training tokens |
| Legal basis | Public content; GitHub public content terms permit crawling |
| Copyright status | Mixed open-source licenses; original authors retain rights |
| Filtering applied | Public repositories only; no private repo content |

---

### Tier 5 — Synthetic Chaos Layer

| Property | Value |
|----------|-------|
| Source | Generated locally by RFI-IRFOS (`build_chaos_corpus.py`) |
| Content | Synthetic noise, adversarial patterns, mixed-language text, random token sequences |
| Approximate share | ~10% of training tokens (enforced invariant) |
| Third-party rights | None — entirely generated by the training team |
| Purpose | Robustness training; prevents over-fitting to clean text distributions |

This tier contains no third-party content and raises no copyright concerns.

---

### Tier 6 — Structured and Multilingual Content

| Property | Value |
|----------|-------|
| Sources | EU AI Act text (GitHub mirror), Linux kernel docs (kernel.org RST), TLDR pages, multilingual EU language samples |
| Access method | `download_corpus.py`, `build_multilingual_corpus.py` |
| Content | Technical documentation, structured text, EU legislation |
| Approximate share | ~10% combined |
| Legal basis | EU legislation is public domain; kernel docs under GPL-2.0 (TDM exception applies); TLDR pages MIT license |

---

### Tier 7 — Reddit Data (Conditional)

| Property | Value |
|----------|-------|
| Source | Reddit public API via `reddit_scraper.py` |
| Access method | PRAW library; authenticated API access with registered app |
| Content | Public discussion threads from technical subreddits |
| Status | **Conditional — dependent on API terms compliance** |
| Legal basis | Reddit API terms of service; public content |
| Credentials | Client ID/secret via `.env` (not committed to repo) |

Reddit's Data API ToS (updated 2023) restricts bulk data collection.
RFI-IRFOS uses the authenticated API within stated rate limits. If Reddit
enforces restrictions that exclude this use case, this data tier will be
removed from future training runs.

---

## Text and Data Mining (TDM) Exception

Under EU Directive 2019/790/EU Article 3 (DSM Directive), reproduced in
national law across EU member states, text and data mining for **scientific
research purposes** is permitted regardless of the rightsholder's opt-out
provisions, subject to lawful access.

albert. is developed as non-commercial AI research by RFI-IRFOS.
Corpus collection respects `robots.txt` where feasible and discloses a
contact User-Agent for opt-out requests.

---

## Data Preprocessing

All corpus text undergoes the following pipeline before tokenization:

1. **Language detection** — Non-target language content filtered or
   down-sampled to maintain corpus balance.
2. **Deduplication** — Near-duplicate paragraphs removed using
   shingling hashes.
3. **Quality filtering** — Minimum length thresholds; removal of
   machine-generated boilerplate.
4. **Tokenization** — Custom BPE tokenizer (`tokenizer_v3/`) with 32k
   vocabulary trained on the combined corpus.
5. **Corpus caching** — Final token IDs written to `data/corpus_cache.bin`
   for reproducible training.

---

## Contact for Data Governance Inquiries

**Organisation:** RFI-IRFOS  
**Email:** s.kepp@ternlang.com  
**Response time:** Within 30 days of receipt  
**Repository:** https://github.com/eriirfos-eng/ternary-intelligence-stack
