Title: Privileged Bases in the Transformer Residual Stream

URL Source: https://www.anthropic.com/research/privileged-bases-in-the-transformer-residual-stream

Markdown Content:
# Privileged Bases in the Transformer Residual Stream \ Anthropic

[Skip to main content](https://www.anthropic.com/research/privileged-bases-in-the-transformer-residual-stream#main-content)[Skip to footer](https://www.anthropic.com/research/privileged-bases-in-the-transformer-residual-stream#footer)

[](https://www.anthropic.com/)

*   [Research](https://www.anthropic.com/research)
*   [Economic Futures](https://www.anthropic.com/economic-futures)
*   Commitments
*   Learn
*   [News](https://www.anthropic.com/news)

[Try Claude](https://claude.ai/)

Interpretability Research

# Privileged Bases in the Transformer Residual Stream

Mar 16, 2023

[Read Paper](https://transformer-circuits.pub/2023/privileged-basis/index.html)

## Abstract

Our mathematical theories of the Transformer architecture suggest that individual coordinates in the residual stream should have no special significance (that is, the basis directions should be in some sense "arbitrary" and no more likely to encode information than random directions). Recent work has shown that this observation is false in practice. We investigate this phenomenon and provisionally conclude that the per-dimension normalizers in the Adam optimizer are to blame for the effect.

We explore two other obvious sources of basis dependency in a Transformer: Layer normalization, and finite-precision floating-point calculations. We confidently rule these out as being the source of the observed basis-alignment.

[](https://twitter.com/intent/tweet?text=https://www.anthropic.com/research/privileged-bases-in-the-transformer-residual-stream)[](https://www.linkedin.com/shareArticle?mini=true&url=https://www.anthropic.com/research/privileged-bases-in-the-transformer-residual-stream)

## Related content

### Coding agents in the social sciences

Results from a survey of 1,260 social scientists about AI and coding agent use.

[Read more](https://www.anthropic.com/research/coding-agents-social-sciences)

### Project Glasswing: An initial update

An early update on what we've learned from Project Glasswing.

[Read more](https://www.anthropic.com/research/glasswing-initial-update)

### 2028: Two scenarios for global AI leadership

Our views on the AI competition between the US and China.

[Read more](https://www.anthropic.com/research/2028-ai-leadership)

[](https://www.anthropic.com/)

### Products

*   [Claude](https://claude.com/product/overview)
*   [Claude Code](https://claude.com/product/claude-code)
*   [Claude Code Enterprise](https://claude.com/product/claude-code/enterprise)
*   [Claude Cowork](https://claude.com/product/cowork)
*   [Claude Security](https://claude.com/product/claude-security)
*   [Claude for Chrome](https://claude.com/chrome)
*   [Claude for Slack](https://claude.com/claude-for-slack)
*   [Claude for Microsoft 365](https://claude.com/claude-for-microsoft-365)
*   [Skills](https://www.claude.com/skills)
*   [Max plan](https://claude.com/pricing/max)
*   [Team plan](https://claude.com/pricing/team)
*   [Enterprise plan](https://claude.com/pricing/enterprise)
*   [Download app](https://claude.ai/download)
*   [Pricing](https://claude.com/pricing)
*   [Log in to Claude](https://claude.ai/)

### Models

*   [Mythos Preview](https://www.anthropic.com/glasswing)
*   [Opus](https://www.anthropic.com/claude/opus)
*   [Sonnet](https://www.anthropic.com/claude/sonnet)
*   [Haiku](https://www.anthropic.com/claude/haiku)

### Solutions

*   [AI agents](https://claude.com/solutions/agents)
*   [Code modernization](https://claude.com/solutions/code-modernization)
*   [Coding](https://claude.com/solutions/coding)
*   [Customer support](https://claude.com/solutions/customer-support)
*   [Education](https://claude.com/solutions/education)
*   [Financial services](https://claude.com/solutions/financial-services)
*   [Government](https://claude.com/solutions/government)
*   [Healthcare](https://claude.com/solutions/healthcare)
*   [Legal](https://claude.com/solutions/legal)
*   [Life sciences](https://claude.com/solutions/life-sciences)
*   [Nonprofits](https://claude.com/solutions/nonprofits)
*   [Security](https://claude.com/solutions/security)
*   [Small business](https://claude.com/solutions/small-business)

### Claude Platform

*   [Overview](https://claude.com/platform/api)
*   [Developer docs](https://platform.claude.com/docs)
*   [Pricing](https://claude.com/pricing#api)
*   [Marketplace](https://claude.com/platform/marketplace)
*   [Regional compliance](https://claude.com/regional-compliance)
*   [Claude on AWS](https://claude.com/partners/claude-on-aws)
*   [Google Cloud’s Vertex AI](https://claude.com/partners/google-cloud-vertex-ai)
*   [Microsoft Foundry](https://claude.com/partners/microsoft-foundry)
*   [Console login](https://platform.claude.com/)

### Resources

*   [Blog](https://claude.com/blog)
*   [Claude partner network](https://claude.com/partners)
*   [Community](https://claude.com/community)
*   [Connectors](https://claude.com/connectors)
*   [Courses](https://www.anthropic.com/learn)
*   [Customer stories](https://claude.com/customers)
*   [Engineering at Anthropic](https://www.anthropic.com/engineering)
*   [Events](https://www.anthropic.com/events)
*   [Inside Claude Code](https://www.anthropic.com/product/claude-code)
*   [Inside Claude Cowork](https://www.anthropic.com/product/claude-cowork)
*   [Inside Claude Enterprise](https://www.anthropic.com/product/enterprise)
*   [Inside Claude Security](https://www.anthropic.com/product/security)
*   [Plugins](https://claude.com/plugins)
*   [Powered by Claude](https://claude.com/partners/powered-by-claude)
*   [Service partners](https://claude.com/partners/services)
*   [Startups program](https://claude.com/programs/startups)
*   [Tutorials](https://claude.com/resources/tutorials)
*   [Use cases](https://claude.com/resources/use-cases)

### Help and security

*   [Availability](https://www.anthropic.com/supported-countries)
*   [Status](https://status.anthropic.com/)
*   [Support center](https://support.claude.com/en/)

### Company

*   [Anthropic](https://www.anthropic.com/company)
*   [Careers](https://www.anthropic.com/careers)
*   [Economic Futures](https://www.anthropic.com/economic-index)
*   [Research](https://www.anthropic.com/research)
*   [News](https://www.anthropic.com/news)
*   [Claude’s Constitution](https://www.anthropic.com/constitution)
*   [Responsible Scaling Policy](https://www.anthropic.com/news/announcing-our-updated-responsible-scaling-policy)
*   [Security and compliance](https://trust.anthropic.com/)
*   [Transparency](https://www.anthropic.com/transparency)

### Terms and policies

Privacy choices*   [Privacy policy](https://www.anthropic.com/legal/privacy)
*   [Consumer health data privacy policy](https://www.anthropic.com/legal/consumer-health-data-privacy-policy)
*   [Responsible disclosure policy](https://www.anthropic.com/responsible-disclosure-policy)
*   [Terms of service: Commercial](https://www.anthropic.com/legal/commercial-terms)
*   [Terms of service: Consumer](https://www.anthropic.com/legal/consumer-terms)
*   [Usage policy](https://www.anthropic.com/legal/aup)

© 2026 Anthropic PBC
*   [](https://www.linkedin.com/company/anthropicresearch)
*   [](https://x.com/AnthropicAI)
*   [](https://www.youtube.com/@anthropic-ai)
