Model Communication and Privacy

AgentOps measures how efficiently AI models communicate — the Communication Efficiency Score (CES) — and it does so without your transcript ever leaving your machine. This page explains both the methodology and the privacy architecture, with links to the exact code, because "trust us" is not an argument.

The core claim

CES is computed entirely client-side from the local session transcript. Only six integers in the range 0–100 are ever transmitted. The server rejects anything else.

The five CES metrics

CES is a weighted sum of five deterministic heuristics. They are not ground truth — their value is that they are computed identically for every model, which makes them comparable. Source: src/core/ces/cesMetrics.ts.

score = conciseness * 0.25
      + structureRatio * 0.25
      + redundancy * 0.20
      + actionDensity * 0.15
      + iterationRate * 0.15

1. Conciseness — weight 0.25

Action tokens versus prose tokens across all assistant turns. Code (fenced and inline) counts as action; each tool use counts as a flat 40-token action proxy; everything else is prose.

conciseness = 100 * actionTokens / (actionTokens + proseTokens)

2. Structure ratio — weight 0.25

Structured characters (code) versus total characters in assistant text blocks:

structureRatio = 100 * structuredChars / totalChars

3. Redundancy — weight 0.20

Two signals: filler openers and repeated 8-grams in prose.

redundancy = 100 - min(100, fillerCount * 5 + repeatedNgramRatio * 100)

Filler openers come from a fixed, public dictionary ("sure", "certainly", "you're absolutely right", "let me", and so on — the full list is in the source). Counting matches against a fixed dictionary means the counts are safe to report: the phrases come from the dictionary, never from your transcript.

4. Action density — weight 0.15

Acting blocks (tool uses + fenced code blocks) versus narrating blocks (text blocks):

actionDensity = 100 * (toolUses + codeBlocks) / textBlocks

5. Iteration rate — weight 0.15

Consecutive user messages with token-set Jaccard similarity of at least 0.5 count as retries of the same request. One attempt per request scores 100; two attempts average scores 50:

iterationRate = 100 / avgAttemptsPerRequest

What is actually transmitted

When you run agentops submit, the payload contains only:

  • ces — the six integers above (score plus five components), each 0–100
  • score, grade, dimensions — the session score numbers described in How Scoring Works
  • stats — counts and token totals: totalCommands, totalRawTokens, totalSavedTokens, totalCacheHits, totalCacheLookups, durationMs, wastePatterns, estimatedSavedUsd, and a per-family breakdown (family names are classifier labels like test or build, with counts and token totals)
  • identity and context — sessionId, projectName (from your config), optional teamCode, agentKind, modelId
  • projectProfile — primaryLanguage (validated against a closed allowlist of 20 known languages, so it can never become a free-text channel) and fileCount
  • meta — AgentOps version, platform (darwin/linux/win32), timestamp

No transcript text. No source code. No file paths. No command output. The CLI prints the full summary and asks for confirmation before sending anything.

Server-side enforcement

The server does not merely trust the client. Every submission is validated in server/src/validation.ts:

  • every CES field must be a finite number between 0 and 100 — any other type or shape rejects the submission
  • primaryLanguage must be null or one of the 20 allowlisted language identifiers
  • score and stats consistency checks (see How Scoring Works)

A string in a CES field is not logged and ignored — the whole submission is rejected with invalid_ces.

The leak test

The claim "no transcript text leaves the machine" is enforced by an automated test, not a policy document. The test plants unique canary strings in a synthetic transcript (in prose, in code blocks, and in user messages), builds the exact payload the CLI would send, stringifies it, and asserts that not a single canary character survives:

tests/ces-submit.test.ts

The payload builder is a pure function (buildSubmitPayload) exported specifically so this test covers the real production path, not a mock.

Failure behavior

CES is fail-open in the safe direction: if the transcript is missing, unreadable, or deleted between session and submit, CES degrades to null and the submission proceeds without it. A broken transcript can never block a submit, and it can never cause raw text to be sent as a fallback.

The optional data pipeline

Separately from scoring, AgentOps has an opt-out data pipeline that can include recorded command outputs (with secrets automatically stripped) in submissions to improve plugin quality. The CLI tells you explicitly before this happens, shows how many secrets were stripped and by which patterns, and you can disable it by setting telemetry.enabled: false in .agentops/config.yaml. This is the only path by which anything other than numbers is transmitted, and it never includes agent transcripts — only outputs of commands the agent ran.

← How Scoring Works Plugins →