Add embedding-anisotropy-check skill 🤖🤖🤖

[space-bacon]

Adds embedding-anisotropy-check, a skill for checking a cosine similarity over embeddings or transformer hidden states before it is reported.

Most encoders put every vector near one shared direction, so raw cosine is rarely centred on zero: bge-small-en-v1.5 scores an unrelated SciFact half-abstract at 0.60 against 0.86 for the matching half. A model asked to compare embeddings usually reports the raw cosine, and a raw 0.9 or a gap of 0.01 then reads as a result.

The bundled script (numpy only) takes paired rows and reports each side's anisotropy, raw and centred cosine for matched pairs, a permuted floor from the same comparison, and how often the partner ranks first. SKILL.md says when to run it and how to read and report the output; its worked example is reproducible from the first 100 documents of mteb/scifact.

npm run skill:validate passes and npm start regenerated docs/README.skills.md.

I wrote this skill; it comes from the measurement rules we use in our own embedding work at Sunstone North.

[github-actions]

🔒 PR Risk Scan Results

Scanned 3 changed file(s).

Severity	Count
🔴 High	0
🟠 Medium	0
ℹ️ Info	2

Severity	Rule	File	Line	Match
ℹ️	skill-script-touched	skills/embedding-anisotropy-check/scripts/anisotropy_check.py	1	skills/embedding-anisotropy-check/scripts/anisotropy_check.py
ℹ️	skill-script-touched	skills/embedding-anisotropy-check/scripts/example_scifact_halves.py	1	skills/embedding-anisotropy-check/scripts/example_scifact_halves.py

This is an automated soft-gate report. Findings indicate review targets and do not block merge by themselves.

[github-actions]

🟡 Contributor Reputation Check: MEDIUM risk

Check	Risk
Profile	MEDIUM
Credential audit	NONE

Maintainers: please review this contributor before merging.
See the workflow run for full details.
Automated check powered by AGT.

[Copilot]

Copilot review overview

🟡 Changes recommended

The advertised cross-model comparison is mathematically unsafe, and the script has correctness, input-validation, provenance, and scalability issues.

Review effort: Balanced
Findings: 1 · 5 · 1

Open (7)

• Cross-model cosine comparisons lack coordinate-space alignment · New
• Reject zero and non-finite vectors before cosine statistics · New
• Stream corpus blocks to prevent excessive retrieval memory use · New
• Validate --perms as a positive CLI argument · New
• Reuse derangements for raw and centred comparisons · New
• Report fit provenance independently for each embedding side · New
• Pin inputs and commands for reproducible numerical examples · New

What changed in this PR

Adds a skill for diagnosing embedding anisotropy before interpreting cosine similarity.

Changes:

• Documents the analysis workflow and interpretation guidance.
• Adds a NumPy CLI for cosine, floor, anisotropy, and retrieval metrics.
• Registers the skill in generated documentation.

File	Description
skills/​embedding-anisotropy-check/​SKILL.md	Defines usage and interpretation guidance.
skills/​embedding-anisotropy-check/​scripts/​anisotropy_check.py	Implements the analysis CLI.
docs/​README.skills.md	Adds the skill to the catalog.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🔍 Vally Lint Results

✅ All checks passed

Scope	Checked
Skills	1
Agents	0
Total	1

Severity	Count
❌ Errors	0
⚠️ Warnings	0
ℹ️ Advisories	0

Summary

Level	Finding
ℹ️	✅ embedding-anisotropy-check (2/2 checks passed)
ℹ️	✓ [spec-compliance] All 1 skill(s) are spec-compliant.
ℹ️	✓ spec-compliance: All spec checks passed.
ℹ️	✓ [valid-refs] All file references across 1 skill(s) are valid.
ℹ️	✓ valid-refs: All file references resolve to existing files within the skill directory.
ℹ️	1 skill(s) linted, 1 passed

Full linter output

### Linting skills/embedding-anisotropy-check
✅ embedding-anisotropy-check (2/2 checks passed)
    ✓ [spec-compliance] All 1 skill(s) are spec-compliant.
        ✓ spec-compliance: All spec checks passed.
    ✓ [valid-refs] All file references across 1 skill(s) are valid.
        ✓ valid-refs: All file references resolve to existing files within the skill directory.

1 skill(s) linted, 1 passed

[Copilot]

Copilot review overview

🟡 Changes recommended

Centring can produce zero-length vectors and invalid JSON output, and several documentation claims need correction.

Review effort: Balanced
Findings: 1

Open (1)

• Validate centered norms before normalization · New

Resolved since last review (7)

• Cross-model cosine comparisons lack coordinate-space alignment
• Report fit provenance independently for each embedding side
• Reuse derangements for raw and centred comparisons
• Validate --perms as a positive CLI argument
• Stream corpus blocks to prevent excessive retrieval memory use
• Reject zero and non-finite vectors before cosine statistics
• Pin inputs and commands for reproducible numerical examples

Previously missed (2)

In code that hasn't changed since last review

Describe cross-domain means as mis-centring

skills/​embedding-anisotropy-check/​SKILL.md:32

A mean from another domain does not necessarily under-correct: depending on its direction and magnitude, subtracting it can also over-correct or introduce a new direction. Describe this as mis-centring rather than asserting one failure mode.

This issue also appears on line 78 of the same file.

Clarify reporting rule for greatest bias

skills/​embedding-anisotropy-check/​SKILL.md:58

The phrase “and most where” is grammatically incomplete, which makes this reporting rule hard to parse. State that the bias is greatest when the gap is small relative to its noise.

[Copilot]

Copilot review overview

🔵 Needs a closer look

The interpretation overstates the anisotropy metric, and zero-variance floors produce misleading output.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Validate centered norms before normalization

Previously missed (2)

In code that hasn't changed since last review

Clarify that zero floor variance makes the standardized gap undefined

skills/​embedding-anisotropy-check/​scripts/​anisotropy_check.py:132

For valid inputs whose unrelated-pair cosines have zero variance, compare intentionally returns None for gap_over_floor_sd, but this sentence renders that as None floor sd apart. This occurs, for example, with paired orthogonal basis vectors, and leaves users with a misleading interpretation instead of explaining that the standardized gap is undefined.

Avoid overstating mean cosine as a general anisotropy measure

skills/​embedding-anisotropy-check/​SKILL.md:49

This guidance overstates what the reported statistic establishes. The implementation's mean off-diagonal cosine is algebraically (||sum(u)||² - n) / (n(n-1)), so it detects a shared mean direction but not anisotropy generally: for example, a balanced cloud containing only +v and -v has a value approaching zero while remaining confined to one axis and producing raw cosines of ±1. A near-zero value therefore does not make raw cosine directly interpretable; users still need the floor and its spread.

[github-actions]

🚦 Submission status: ⏳ Awaiting automation

Risk tier: merge-risk:high — Privileged execution, automation, or review-policy change
Required to merge: passing submission-gate checks plus 2 approvals from reviewers with write access, including a maintainer with admin or maintain permission.

Why this tier

• skills/embedding-anisotropy-check/scripts/anisotropy_check.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)
• skills/embedding-anisotropy-check/scripts/example_scifact_halves.py is a high-risk path (automation, scripts, MCP config, hooks, or review policy)

Automated checks

Check	Status	Details
Line endings	✅ Passed	Passed · logs
Spelling	✅ Passed	Passed · logs
Generated README consistency	✅ Passed	Passed · logs
Skill validation	⏳ Pending	Waiting for the check to start
Skill lint (vally)	✅ Passed	Passed · logs
Risk scan	✅ Passed	Passed · logs
Contributor reputation	✅ Passed	Passed · logs
Duplicate resource scan	✅ Passed	Passed · logs
PR quality signal	⏭️ Skipped	Skipped by its workflow · logs
Contributor risk signal	🔧 Infrastructure failure	The contributor check succeeded but its result artifact was missing, unreadable, or for another commit · logs

Action needed

• 🔧 Contributor risk signal hit an automation problem that is not caused by your contribution. Comment /rerun-checks to retry; maintainers are notified if it keeps failing.

Review

• Approvals: 0/2
• Assigned reviewer: aaronpowell
• Review target date: not set
• Still needed: 2 more approval(s); an approval from a maintainer with admin or maintain permission
• The core-maintainers pool is not staffed yet; an approver with admin or maintain permission is required instead.

Commands

Command	Who	What it does
/rerun-checks	PR author, maintainers	Re-runs failed or incomplete checks and re-evaluates this gate
/request-review	PR author, maintainers	Asks the review rotation to assign a reviewer (adds needs-reviewer)

_{Updated for 7ced8bc · This comment is maintained automatically — see submission gate docs.}