Class TokenCounter

Universal token counter supporting multiple providers.

Provides accurate token counting via provider APIs (Anthropic, Gemini) or local tiktoken (OpenAI), with fallback to character-based estimation.

Example

const counter = new TokenCounter({
  anthropicApiKey: process.env.ANTHROPIC_API_KEY,
  googleApiKey: process.env.GOOGLE_AI_API_KEY,
});

// Count via Anthropic API
const result = await counter.countAnthropic(messages, 'claude-sonnet-4');

// Count via local tiktoken
const openaiResult = counter.countOpenAI('Hello world', 'gpt-4o');

// Offline estimation
const estimate = counter.estimate('Some text');

Implements

ITokenCounter

Index

Constructors

constructor

Methods

countAnthropic countGemini countOpenAI estimate estimateForProvider clearCache getCacheStats dispose

Constructors

constructor

new TokenCounter(config?: TokenCounterConfig): TokenCounter
Creates a new TokenCounter instance.
Parameters
- config: TokenCounterConfig = {}
  Token counter configuration
Returns TokenCounter
- Defined in packages/nexus-agents/src/context/token-counter.ts:100

Methods

countAnthropic

countAnthropic(
messages: Message[],
model: string,
): Promise<Result<TokenCountResult, TokenCountError>>
Count tokens for Anthropic/Claude models via API.
Parameters
- messages: Message[]
- model: string
Returns Promise<Result<TokenCountResult, TokenCountError>>
Implementation of ITokenCounter.countAnthropic
- Defined in packages/nexus-agents/src/context/token-counter.ts:119

countGemini

countGemini(
content: string,
model: string,
): Promise<Result<TokenCountResult, TokenCountError>>
Count tokens for Gemini models via API.
Parameters
- content: string
- model: string
Returns Promise<Result<TokenCountResult, TokenCountError>>
Implementation of ITokenCounter.countGemini
- Defined in packages/nexus-agents/src/context/token-counter.ts:178

countOpenAI

countOpenAI(
text: string,
model?: string,
): Result<TokenCountResult, TokenCountError>
Count tokens for OpenAI models using local tiktoken.
Parameters
- text: string
- model: string = DEFAULT_TIKTOKEN_MODEL
Returns Result<TokenCountResult, TokenCountError>
Implementation of ITokenCounter.countOpenAI
- Defined in packages/nexus-agents/src/context/token-counter.ts:228

estimate

estimate(text: string): number
Estimate tokens offline using character-based heuristic. Uses ~4 characters per token as a general approximation.
Parameters
- text: string
Returns number
Implementation of ITokenCounter.estimate
- Defined in packages/nexus-agents/src/context/token-counter.ts:268

estimateForProvider

estimateForProvider(text: string, provider: TokenCounterProvider): number
Estimate tokens for a specific provider.
Parameters
- text: string
- provider: TokenCounterProvider
Returns number
- Defined in packages/nexus-agents/src/context/token-counter.ts:278

clearCache

clearCache(): void
Clear the token count cache.

Returns void
Implementation of ITokenCounter.clearCache
- Defined in packages/nexus-agents/src/context/token-counter.ts:289

getCacheStats

getCacheStats(): { size: number; maxSize: number; ttlMs: number }
Get current cache statistics.

Returns { size: number; maxSize: number; ttlMs: number }
Implementation of ITokenCounter.getCacheStats
- Defined in packages/nexus-agents/src/context/token-counter.ts:296

dispose

dispose(): void
Frees resources (tiktoken encoder). Call this when done with the counter.

Returns void
- Defined in packages/nexus-agents/src/context/token-counter.ts:368

Class TokenCounter

Example

Implements

Index

Constructors

Methods

Constructors

constructor

Parameters

Returns TokenCounter

Methods

countAnthropic

Parameters

Returns Promise<Result<TokenCountResult, TokenCountError>>

countGemini

Parameters

Returns Promise<Result<TokenCountResult, TokenCountError>>

countOpenAI

Parameters

Returns Result<TokenCountResult, TokenCountError>

estimate

Parameters

Returns number

estimateForProvider

Parameters

Returns number

clearCache

Returns void

getCacheStats

Returns { size: number; maxSize: number; ttlMs: number }

dispose

Returns void

Settings

On This Page