Repository navigation
MAINT: Add assert_never guards to the score dispatches (P311-03, Batch B) - #4551
Open
Mohammed Shoaib Choudry (DevShoaib78) wants to merge 2 commits into
Open
Mohammed Shoaib Choudry (DevShoaib78) wants to merge 2 commits into
Mohammed Shoaib Choudry (DevShoaib78) wants to merge 2 commits into
Conversation
…h B) PlagiarismMetric, CategoryConflictPolicy, WildGuardLabel.description and PackageEcosystem now end in a TYPE_CHECKING-only assert_never with the existing runtime fallback kept, so ty flags a new member that is not handled. Batch B of microsoft#4506.
Copilot started reviewing on behalf of
Mohammed Shoaib Choudry (DevShoaib78)
November 21, 2026 19:05
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Two module-level scorer-construction tests lack the required patch_central_database fixture, leaving them without per-test memory isolation.
2 open issues
What changed in this PR
Implements Batch B of #4506 by making four enum dispatches in PyRIT’s scoring layer statically exhaustive while preserving their runtime fallbacks.
Changes:
- Add
TYPE_CHECKING-onlyassert_neverguards and explicit branches for previously implicit enum members. - Use enum identity comparisons for plagiarism metrics and
matchstatements for labels and package ecosystems. - Add enum-extension type-check probes and targeted runtime regression tests.
| File | Description |
|---|---|
tests/unit/score/test_wildguard_parser.py |
Test label descriptions and dispatch exhaustiveness. |
tests/unit/score/test_response_handler.py |
Test unknown-policy rejection and dispatch exhaustiveness. |
tests/unit/score/test_plagiarism_scorer.py |
Exercise every metric and check enum-extension detection. |
tests/unit/score/test_package_hallucination_scorer.py |
Verify JavaScript package preservation and dispatch exhaustiveness. |
pyrit/score/true_false/wildguard_parser.py |
Handle every label explicitly and retain the fallback description. |
pyrit/score/true_false/regex/package_hallucination_scorer.py |
Make JavaScript’s no-op branch explicit and guard ecosystem dispatch. |
pyrit/score/response_handler.py |
Handle conflict rejection explicitly and guard unknown policies. |
pyrit/score/float_scale/plagiarism_scorer.py |
Use enum identity checks and guard unhandled metrics. |
Review effort: Balanced
| return result.returncode, result.stdout + result.stderr | ||
|
|
||
|
|
||
| def test_javascript_keeps_known_packages_as_written() -> None: |
Contributor
Author
There was a problem hiding this comment.
Fixed in 9b4a7cc: the test now uses @pytest.mark.usefixtures("patch_central_database").
|
|
||
|
|
||
| @pytest.mark.parametrize("metric", list(PlagiarismMetric), ids=lambda m: m.name) | ||
| def test_plagiarism_score_handles_every_metric(metric: PlagiarismMetric) -> None: |
Contributor
Author
There was a problem hiding this comment.
Fixed in 9b4a7cc: the test now uses @pytest.mark.usefixtures("patch_central_database").
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Description
Batch B of #4506: the four closed-set dispatches in
pyrit/scorenow end in aTYPE_CHECKING-onlyassert_never, with the existing runtime fallback kept, sotyreports a new enum member that is not handled.PlagiarismScorer._plagiarism_score: themetric.value == "..."checks becomemetric is PlagiarismMetric.X, whichtycan narrow. The guard sits before the existingValueError._build_unvalidated_score(response_handler.py):REJECT_CONFLICTgets its own branch instead of being the implicitelse. A value outside the enum still raises the sameValueError.WildGuardLabel.description:HARMFUL_RESPONSEgets its own case instead of the implicit final return; the fallback still returns "the response is harmful".PackageHallucinationScorer.__init__:JAVASCRIPTgets an explicit no-op case (npm names are matched as written;node:built-ins are already skipped during extraction).matchrather than anif/elifchain: with the final member written out as anelif,tyreportsredundant-condition-strict(the condition is always true once the others are excluded). This followsScorerEvaluator.from_scorerandHumanLabeledDatasetin the same package.current-enum/extended-enumtyprobe used by the earlier P311-03 PRs, built from the real enum and function source (the plagiarism and package probes stub the helpers and built-in package sets). All fourextended-enumprobes fail without the guards. Runtime tests cover eachPlagiarismMetricbranch, everyWildGuardLabel.description, the unchanged JavaScript known-package set, and theValueErrorfor a conflict policy outside the enum.Batches A, C and D are not part of this PR, so #4506 stays open.