[Operational] CRD-to-Agent readiness assessment — advisory checks at the CRD-change boundary

A versioned, deterministic, CRD-agnostic ruleset for the published KRM contract of a CustomResourceDefinition, and the checker that runs it: ../tools/readiness.py, with its word lists in ../tools/readiness_vocabulary.py. Scope, rationale and the open decisions: IPCEICIS-11214. It is the advisory-checker layer of the CRD-to-agent readiness framework (IPCEICIS-11223), whose ownership and evidence boundaries it works within.

Each rule asks one question: is this mechanically detectable property of the contract in order? The checker does not review code, score usability with a model or check conformance to ADRs, and it never decides that a feature is agent-usable — that still takes the manual contract review (contract intake, IPCEICIS-11159) and the behavioural tasks (IPCEICIS-10943). A finding here is not an agent FAIL, and a clean run is not an agent PASS.

Advisory. Findings never set the exit code. Domain teams own their CRDs, their controllers and their merge policy; the checker tells them what it can see and says plainly what it cannot.

It is also the one place kind-name collisions are checked. The standalone tools/collisions.py was folded in (see Coverage of the former collision checker); there is no separate command any more.

File	What
ruleset.yaml	The ruleset. Every rule's level, severity, basis, applicability, inputs, guidance and limits. The checker reads it, and refuses to run if a rule has no implementation or an implementation has no rule
fixtures/	Synthetic CRDs, examples, RBAC and suppressions, with hand-written expected.yaml. ../tools/validate.py fails on any disagreement
pilot/	The pilot on the Cluster CRD and one unrelated CRD: pinned inputs with provenance, generated reports, the reviewed findings

Run it

uv run tools/readiness.py --list-rules                          # the ruleset as a table
uv run tools/readiness.py                                       # every pinned CRD in crds/, offline
uv run tools/readiness.py ../repo/config/crd --examples ../repo/config/samples --rbac ../repo/config/rbac
uv run tools/readiness.py crds ../a/config/crd ../b/config/crd # the union, which no single repository holds
uv run tools/readiness.py crds --changed-since origin/main      # only the CRDs a change touched
uv run tools/readiness.py crds --context kind-dev --examples ... # add the server-side dry-run level
uv run tools/readiness.py crds --report out/ --annotations      # CI: readiness.json, readiness.md, annotations
uv run tools/readiness.py crds --json                           # everything, the set analysis included

CRD paths come first; --examples, --rbac and --crd each take several values. Exit 0 whenever it ran, 2 when it could not: an unreadable ruleset, a ruleset that disagrees with the implementation, an invalid suppression file, no CRD found, or an internal error in a rule — in which case the report is still written.

--changed-since REF evaluates the CRDs whose file changed since the merge base with REF, and the kinds whose supplied example changed. Any change to the supplied --rbac inputs — an edit, a new file or a deleted one — re-evaluates every supplied CRD, since a role can grant any kind and a deleted file leaves nothing to match on. So does a change to the ruleset or the checker. Uncommitted and untracked files count as changed.

In a repository that owns CRDs

Self-service, not the agreed integration. The central CI trigger, change detection, artifact destination and pipeline ownership are for Systems and test engineering to agree under IPCEICIS-11223. Until they do, a team may run the checker in its own pipeline, advisory, as below.

Pin this repository at a commit, so the ruleset version cannot move under you, and run it from beside your own checkout. This repository's own job, ../.forgejo/workflows/readiness.yml, is the working reference.

on:
  pull_request:
    paths: ["config/crd/**", "config/samples/**", "config/rbac/**"]
jobs:
  crd-readiness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }          # the merge base decides which CRDs changed
      - name: Fetch the readiness checker at a pinned commit
        run: |
          git clone --quiet https://edp.buildth.ing/DevFW-CICD/k8s-mcp-eval.git .readiness
          git -C .readiness checkout --quiet <commit>
      - name: Install uv
        run: |
          curl -LsSf https://astral.sh/uv/install.sh | sh
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
      - name: CRD readiness (advisory)
        run: >-
          uv run --project .readiness .readiness/tools/readiness.py config/crd
          --examples config/samples --rbac config/rbac
          --changed-since "origin/${{ github.base_ref }}"
          --report readiness-report --annotations
      - uses: actions/upload-artifact@v3
        if: always()
        with: { name: crd-readiness, path: readiness-report/ }

Keep it out of required status checks. Add --suppressions once there is something to suppress, and render any Helm-templated RBAC first. If the runner cannot reach this repository, vendor tools/readiness.py, tools/readiness_vocabulary.py and readiness/ruleset.yaml at one commit, keeping that layout.

Execution levels

Level	Needs	When it runs	Rules
offline	the manifests	every relevant change, in CI	naming, versions, schema, descriptions, validation, examples, status, RBAC, lifecycle, discoverability
server-dry-run	a kube context whose API server has the CRD installed	only with --context; optional	dryrun.crd, dryrun.discovery, dryrun.examples
live	persisted custom resources, running controllers, external infrastructure	never here; pending the Systems decision (IPCEICIS-11223)	reported as not evaluated
manual	a person who knows the domain	the manual contract review (IPCEICIS-11159)	reported as not evaluated
behavioural	an agent driving the API	k8s-mcp-eval golden tasks (IPCEICIS-10943)	reported as not evaluated

The dry-run level sends dryRun=All requests and read-only queries, nothing else: a server-side apply of the CRD manifest, and create --dry-run=server --validate=strict of each example. It never installs a CRD. It evaluates discovery and examples only against an installed CRD whose contract matches the manifest, because a different installed copy would make the server answer for the wrong contract. Contracts are compared as kube-apiserver stores them — defaults applied, so an omitted singular, listKind or conversion matches the server's filled-in value — and a difference in schema, served, storage, subresources or conversion is a mismatch, named in the report. Without --context, or when the server does not answer, the whole level reports not evaluated. Client-side dry-run is never used and never substitutes.

A dry-run environment without Docker or kind

The dry-run level needs an API server, not a cluster: no nodes, no kubelet, so no cgroup v2. The etcd, kube-apiserver and kubectl binaries that Kubernetes controller tests use (envtest, published per platform by kubernetes-sigs/controller-tools, with SHA-512s in its envtest-releases.yaml) are enough. This is how the pilot's dry-run results were produced:

curl -LO https://github.com/kubernetes-sigs/controller-tools/releases/download/envtest-v1.30.3/envtest-v1.30.3-linux-amd64.tar.gz
tar -xzf envtest-v1.30.3-linux-amd64.tar.gz && export PATH="$PWD/controller-tools/envtest:$PATH"  # after checking the SHA-512
openssl genrsa -out sa.key 2048 && openssl rsa -in sa.key -pubout -out sa.pub
echo "$(uuidgen),admin,admin,\"system:masters\"" > tokens.csv    # a throwaway local token
etcd --data-dir etcd --listen-client-urls http://127.0.0.1:12379 --advertise-client-urls http://127.0.0.1:12379 \
  --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 \
  --initial-cluster default=http://127.0.0.1:12380 &
kube-apiserver --etcd-servers http://127.0.0.1:12379 --cert-dir certs --secure-port 16443 --bind-address 127.0.0.1 \
  --advertise-address 127.0.0.1 --service-cluster-ip-range 10.0.0.0/24 --token-auth-file tokens.csv \
  --service-account-issuer https://kubernetes.default.svc --service-account-key-file sa.pub \
  --service-account-signing-key-file sa.key --authorization-mode RBAC --disable-admission-plugins ServiceAccount &
# a kubeconfig for https://127.0.0.1:16443 with that token and insecure-skip-tls-verify, as context readiness-envtest
kubectl apply --server-side -R -f crds/      # the environment installs the CRDs; the checker never does
uv run tools/readiness.py crds --context readiness-envtest --examples examples/*.yaml

It has no webhooks and no controllers, so it answers for the schema, CEL and defaulting only. It has not yet been tried on this repository's CI runner.

What a result licenses you to say

Status	Means	Does not mean
finding	the rule flagged something, at the location reported	that an agent will fail. A heuristic finding can be a false positive
pass	a decided check ran and the property holds, for the inputs supplied	anything about properties other rules own
no_finding	a heuristic ran and matched nothing	that the property holds — heuristic silence is untested, not clean
not_evaluated	an input or the environment was missing, or the property is not mechanical	a pass. Never
not_applicable	the rule's condition does not hold here — no status declared, a single version	—

Basis is what a finding is worth. decided findings are read off the manifest and involve no judgement; heuristic findings pattern-match names or prose and can be wrong in either direction. Severity is the impact if the finding is real: major — likely to make a reader of the published contract use the API wrongly or not at all; minor — degrades discoverability or recovery, a careful reader still succeeds; info — context, not a defect on its own.

Authority is where a rule's expectation comes from. A platform contract is enforced by the API server: it refuses the manifest, or every object the manifest would admit. A convention is Kubernetes practice every client depends on, which the server accepts either way. A consumer need is what a reader — a person or an agent — needs to use the contract. The Definition of Done does not require consumer needs per field, so a finding under one is a quality proposal, not a defect against the DoD.

Inputs

Input	Flag	Absent means
crd — the CRD under evaluation	positional paths	—
crd-set — every CRD in the positional paths	positional paths	cross-CRD rules compare against what was supplied and nothing else
examples — example custom resources	--examples	example rules not evaluated. Supplied, but none for a kind: a finding, never an invented example
rbac — rendered Role and ClusterRole manifests	--rbac	rbac.evidence not evaluated; finalizer evidence not considered
api-server — a kube context	--context	the server-dry-run level not evaluated

• Only apiextensions.k8s.io/v1 CRDs are read. A file that does not parse is listed in the report, not counted — Helm templates must be rendered with helm template first.
• An example is any document whose API group and kind match a supplied CRD. Manifests written to be rejected, such as this harness's examples/reference/, are not examples and must not be passed as such.
• Copies of one CRD whose contract is identical once Kubernetes defaults are applied (a config/crd/ and a chart copy) are evaluated once. Copies that differ anywhere in the contract — schema, served, storage, subresources, conversion — are each evaluated. naming.schema-conflict reports them only when a schema differs; any other difference appears in that rule's context and in the set analysis below.
• A per-repository run cannot see a collision with a kind another repository holds. Pass the union of several repositories for the cross-domain half (../CRD_AUTHORING.md, Cross-domain collisions).

The report

Every report records the ruleset name, version and digest; the checker version and a content digest of tools/readiness.py and tools/readiness_vocabulary.py; the Python, PyYAML and jsonschema versions; per input file, a digest of the documents taken from it; which CRDs were selected and why; and the API server version when one was used. Digests are over parsed content, so a Windows and a Linux checkout agree. There is no timestamp: two runs over the same inputs with the same checker, ruleset and runtime produce byte-identical JSON, and tools/validate.py checks exactly that.

Each finding carries its rule, file, line, field path and version. The Markdown adds, per rule, what it expects, why it matters and a link to the guidance it rests on. --annotations prints one workflow command per open finding for runners that surface them.

Across the supplied set

After the per-CRD results, every report carries set_analysis, rendered as Across the supplied set. It is not a rule: it has no status, is never counted in the summary, never produces an annotation and cannot be suppressed. It lists only entries touching a CRD that was evaluated, so --changed-since narrows it too.

Part	Basis	What it says
copies_that_disagree	decided by the manifests, about the inputs	One group and kind defined more than once with different contracts, and where each copy differs from the first. conflict: one source (one path argument) holds two differing copies — a defect in that source. drift: each source is consistent but they disagree — usually one copy is behind, which says nothing against the API
families	context, not a finding	Kinds sharing a head word. The right way to name variants, provided each first line says which variant it is
hypotheses.unqualified_names	hypothesis — never put to a behavioural run	A bare single-word kind the word list has no entry for. It cannot tell a common noun from a coined one
hypotheses.description_overlap	hypothesis — never put to a behavioural run	Kind pairs whose descriptions share vocabulary once each kind's own name and the set's boilerplate are removed. Shared words, not shared meaning

Established findings live in the rules, graded by basis (decided or heuristic); hypotheses live only here, so nothing that has never been tested against an agent can read as a finding.

Suppressions

A suppression belongs to the team that owns the CRD, lives in that team's repository, and is passed with --suppressions. It is a per-CRD triage decision, which is why it is never part of the ruleset.

suppressions:
  - rule: naming.vocabulary            # a rule id from ruleset.yaml
    crd: widgets.example.com           # one CRD, by metadata.name - no wildcards
    field: spec.colour                 # optional: one field path; omitted, every finding of the rule on this CRD
    reason: A rename is costed separately; the first line of the description disambiguates meanwhile.
    owner: widget domain
    ticket: https://jira.example.com/browse/WID-1   # optional

• rule, crd, reason and owner are required. A missing key, an unknown key or an unknown rule stops the run: a suppression that silently fails to apply is worse than none.
• A suppressed finding stays in the report, struck through with its reason and owner. Suppression is triage, not a pass: it never changes a status, and there is nothing to suppress in a not_evaluated result.
• An entry that matches nothing is listed as stale, so the file cannot quietly outlive the findings it answered.

Versioning

• The ruleset version is semver. Major: a rule removed or renamed, or its trigger or basis changed, so that the same CRD can now receive a different status. Minor: a rule added, or a severity or authority changed. Patch: wording, rationale and links only.
• A code change that changes what a rule flags is a ruleset change and bumps the ruleset version; the checker's own version and digest do not stand in for it.
• The checker refuses to run when the ruleset and the implementation disagree, and tools/validate.py fails in CI for the same reason — so a pinned copy cannot skip a rule unnoticed.
• Reports are comparable only within one ruleset version and one checker digest, as run results are within one grader version (../docs/decisions.md).

Ownership and escalation

What	Owned by
The ruleset, the checker and the fixtures	this repository's maintainers (roles in Jira)
Each CRD, its controller, examples, RBAC and merge policy	the owning domain team
A suppression	the owning domain team, in its own repository
A finding no single domain can fix — one kind served by two groups	the cross-domain owners the naming ADRs name (ADR-0011 with ADR-0001, as established in IPCEICIS-10388)

A finding the owning team disputes is suppressed with its reason, and reported here as an issue carrying the CRD fragment that triggers it. A confirmed false positive is fixed in the checker, guarded by a fixture, and bumps the ruleset version — never worked around by editing the CRD to satisfy the checker. A decided finding one domain cannot resolve goes to the cross-domain owners, and keeps being reported until they decide.

Where each property is decided

Property	Automated here	Manual contract review (IPCEICIS-11159)	Behavioural tasks (IPCEICIS-10943)	Live (pending, IPCEICIS-11223)
Naming	built-in and cross-group collisions, conventions, word list, near-synonyms	domain fit; the cost of a rename	whether agents pick the kind	—
Descriptions	presence, generator boilerplate, restatement, enum values, key vocabularies, patterns	sufficiency; when a caller wants the kind; first-line disambiguation	whether agents find and use them	—
Validation	structural schema, required fields exist, CEL messages exist and name an alternative	whether a message helps	whether agents handle rejection honestly	—
Examples	supplied; conform offline; accepted by a dry-run	whether they are representative	—	—
Status	subresource, conditions shape, observedGeneration	which conditions exist and what they mean	reading and waiting (Tier 2-fake)	reconciliation, failure paths
RBAC	some role grants the kind; status is writable	least privilege, personas	—	behaviour for non-admin callers
Lifecycle	the contract mentions deletion	whether what it says is right	—	cleanup actually happens
Discoverability	categories, printer columns and their paths, reference targets	—	discovery by agents	—

Anything a machine can only guess at is reported as not evaluated, never as passed.

Moving a rule from advisory to enforcing

No rule enforces, and whether any ever does is not this checker's decision: it belongs to Systems and test engineering, with the owning domain's agreement (IPCEICIS-11223). What follows is this ticket's proposal to them. A rule becomes eligible only when all of these hold, and the move is recorded in ../docs/decisions.md with who decided:

1. Its authority is platform-contract and its basis decided. Heuristics never gate anything, and neither do conventions or consumer needs: descriptions, examples and discoverability are quality proposals the DoD does not require, and a missing example is never made merge-blocking.
2. It has produced no false positive across the pinned substrates and at least two domain repositories it was not calibrated on, with the runs recorded.
3. The fix is within the owning domain's reach. A collision with another domain's kind is not.
4. The owning domain agrees for its repositories, and the suppression path is in place.
5. Enforcement is opt-in per repository behind a separate flag, and the ruleset major version records it.

naming.crd-name, naming.valid and schema.structural already meet 1 and 3, and cost little to enforce: kube-apiserver refuses the same manifests (checked against v1.30.3), so enforcing them only moves the failure earlier. validation.required-exists is as cheap for a different reason: the server installs such a CRD, then refuses every object. No collision rule gates anything either: the former tools/collisions.py --strict gate on three collision facts was not carried over. naming.cross-group, naming.schema-conflict and naming.builtin-kind are conventions, so they fail criterion 1, and the first two fail criterion 3 as well — the fix is often another domain's. Nothing makes them gate.

Coverage of the former collision checker

tools/collisions.py was retired and its checks folded in here, so there is one command, one report and one set of fixtures. Nothing was kept as a compatibility shim. Where each part went:

collisions.py	Here	Basis
AMBIGUOUS — one kind name served by two groups	naming.cross-group	decided
CONFLICT — one group serving a kind with two schemas	naming.schema-conflict, and conflict under Across the supplied set	decided
DRIFT — the paths given disagree about a kind	drift under Across the supplied set; it also raises naming.schema-conflict when a schema differs	decided, about the inputs
AMBIGUOUS — a kind shadowing a built-in kind	naming.builtin-kind	decided
SHADOWED — built-in or doubly claimed short names	naming.builtin-short-name, naming.short-name-conflict	decided
rule 1 word list, head and modifier, measured notes, vocabulary denominator	naming.vocabulary, with modifiers, measured notes and the denominator in its context	heuristic
rule 2 near-synonyms	naming.near-synonym	heuristic
families	families under Across the supplied set	context
rules 3, 4, 6, 7, 8 and description coverage	docs.* and validation.cel-*; coverage in docs.field-descriptions' context; conventional patterns in docs.pattern's	decided or heuristic, per rule
hypotheses: dangling references, no printer columns	docs.dangling-reference (heuristic), discoverability.printer-columns (decided, info)	—
hypotheses: bare single-word names, description overlap	hypotheses under Across the supplied set	hypothesis
--strict	not carried over — see above	—

What changed in the move: copies are now compared as contracts, so two copies with one schema but different served, storage, subresources or conversion are no longer deduplicated as identical; and drift is told apart from conflict per source argument, exactly as before. The single-file, PyYAML-only copy is gone: the checker needs this repository's tools/readiness.py, tools/readiness_vocabulary.py and readiness/ruleset.yaml together, plus jsonschema.

Fixtures

Everything under fixtures/ is synthetic, written to exercise the checker, and marked so in each file's header and with a rosa.eval/fixture annotation. It never lives under ../crds/, which is authentic-only and loaded recursively by the harness, and is never installed. expected.yaml is written by hand from the defects each fixture carries; a snapshot of the checker's own output would only test the checker's agreement with itself. The clean fixture carries the shapes that produced false positives during the pilot, so none of them can come back unnoticed.

tools/validate.py also derives variants of one fixture to pin contract comparison — a change to served, storage, subresources or conversion must be a different contract with the same schema, and a stated Kubernetes default must not — and replays --changed-since in a throwaway git repository, where removing a status grant or deleting the RBAC file must produce the same rbac.evidence finding as a full scan. Without git on PATH it warns that the second half did not run.

Limits

• CRD-agnostic is not vocabulary-free. naming.vocabulary reads a hand-written word list (../tools/readiness_vocabulary.py) built from one domain's failures. It is data applied to every kind alike, never a branch on a kind, and its silence is reported as untested.
• Contract comparison applies the defaults kube-apiserver's apiextensions.k8s.io/v1 defaulting sets — singular, listKind, conversion strategy None, webhook service port 443 — and reads absent flags as false and empty subresources, printer columns or names lists as absent. Any other server-side normalisation would show as a mismatch, so the dry-run level reports not evaluated; it never produces a false match.
• Cross-CRD rules and the set analysis see only the CRDs supplied. Drift is judged per path argument, so pass each repository as its own argument; one directory holding both copies reads as a conflict.
• The offline example check is JSON Schema over the OpenAPI subset CRDs use. CEL rules, formats and list uniqueness need the dry-run level; patterns are matched with Python's re, not Go's RE2.
• The heuristics were tuned against the pinned substrate during the pilot, so their figures on that substrate are calibration, not validation (pilot/README.md).
• The dry-run level has run against a real kube-apiserver v1.30.3 with no admission webhooks and no controllers. It has never exercised a webhook, and its answers are that server version's (pilot/README.md).