Skip to content

MAINT: Add assert_never guards to the score dispatches (P311-03, Batch B) - #4551

Open
Mohammed Shoaib Choudry (DevShoaib78) wants to merge 2 commits into
microsoft:mainfrom
DevShoaib78:maint/assert-never-score-batch-b
Open

Mohammed Shoaib Choudry (DevShoaib78) wants to merge 2 commits into
microsoft:mainfrom
DevShoaib78:maint/assert-never-score-batch-b

Conversation

@DevShoaib78

Copy link
Copy Markdown
Contributor

Description

Batch B of #4506: the four closed-set dispatches in pyrit/score now end in a TYPE_CHECKING-only assert_never, with the existing runtime fallback kept, so ty reports a new enum member that is not handled.

  • PlagiarismScorer._plagiarism_score: the metric.value == "..." checks become metric is PlagiarismMetric.X, which ty can narrow. The guard sits before the existing ValueError.
  • _build_unvalidated_score (response_handler.py): REJECT_CONFLICT gets its own branch instead of being the implicit else. A value outside the enum still raises the same ValueError.
  • WildGuardLabel.description: HARMFUL_RESPONSE gets its own case instead of the implicit final return; the fallback still returns "the response is harmful".
  • PackageHallucinationScorer.__init__: JAVASCRIPT gets an explicit no-op case (npm names are matched as written; node: built-ins are already skipped during extraction).
  • The last two use match rather than an if/elif chain: with the final member written out as an elif, ty reports redundant-condition-strict (the condition is always true once the others are excluded). This follows ScorerEvaluator.from_scorer and HumanLabeledDataset in the same package.
  • Tests: each site gets the current-enum / extended-enum ty probe used by the earlier P311-03 PRs, built from the real enum and function source (the plagiarism and package probes stub the helpers and built-in package sets). All four extended-enum probes fail without the guards. Runtime tests cover each PlagiarismMetric branch, every WildGuardLabel.description, the unchanged JavaScript known-package set, and the ValueError for a conflict policy outside the enum.

Batches A, C and D are not part of this PR, so #4506 stays open.

…h B)

PlagiarismMetric, CategoryConflictPolicy, WildGuardLabel.description and PackageEcosystem now end in a TYPE_CHECKING-only assert_never with the existing runtime fallback kept, so ty flags a new member that is not handled. Batch B of microsoft#4506.
Copilot AI balanced review requested due to automatic review settings November 21, 2026 19:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two module-level scorer-construction tests lack the required patch_central_database fixture, leaving them without per-test memory isolation.

2 open issues
What changed in this PR

Implements Batch B of #4506 by making four enum dispatches in PyRIT’s scoring layer statically exhaustive while preserving their runtime fallbacks.

Changes:

  • Add TYPE_CHECKING-only assert_never guards and explicit branches for previously implicit enum members.
  • Use enum identity comparisons for plagiarism metrics and match statements for labels and package ecosystems.
  • Add enum-extension type-check probes and targeted runtime regression tests.
File Description
tests/​unit/​score/​test_wildguard_parser.py Test label descriptions and dispatch exhaustiveness.
tests/​unit/​score/​test_response_handler.py Test unknown-policy rejection and dispatch exhaustiveness.
tests/​unit/​score/​test_plagiarism_scorer.py Exercise every metric and check enum-extension detection.
tests/​unit/​score/​test_package_hallucination_scorer.py Verify JavaScript package preservation and dispatch exhaustiveness.
pyrit/​score/​true_false/​wildguard_parser.py Handle every label explicitly and retain the fallback description.
pyrit/​score/​true_false/​regex/​package_hallucination_scorer.py Make JavaScript’s no-op branch explicit and guard ecosystem dispatch.
pyrit/​score/​response_handler.py Handle conflict rejection explicitly and guard unknown policies.
pyrit/​score/​float_scale/​plagiarism_scorer.py Use enum identity checks and guard unhandled metrics.

Review effort: Balanced

return result.returncode, result.stdout + result.stderr


def test_javascript_keeps_known_packages_as_written() -> None:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9b4a7cc: the test now uses @pytest.mark.usefixtures("patch_central_database").



@pytest.mark.parametrize("metric", list(PlagiarismMetric), ids=lambda m: m.name)
def test_plagiarism_score_handles_every_metric(metric: PlagiarismMetric) -> None:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9b4a7cc: the test now uses @pytest.mark.usefixtures("patch_central_database").

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants