Skip to content

Exercise both modes in the tokenizer test - #2882

Open
j45856021-dev wants to merge 1 commit into
openai:mainfrom
j45856021-dev:test/multilingual-tokenizer-coverage
Open

j45856021-dev wants to merge 1 commit into
openai:mainfrom
j45856021-dev:test/multilingual-tokenizer-coverage

Conversation

@j45856021-dev

Copy link
Copy Markdown

test_tokenizer is parameterized over both tokenizer modes, but always calls get_tokenizer(multilingual=False). Its multilingual case therefore repeats the English-only metadata checks.

Use the parameter and check the decoded default start-of-transcript prefix for each mode: the multilingual tokenizer includes the English language and transcription task markers, while the English-only tokenizer has just the start marker. Keep the existing language-token invariants.

This is a test-only change; model and inference code are unchanged.

Validation

  • The new prefix assertion fails for the multilingual case while the hardcoded False remains.
  • pytest tests/test_tokenizer.py tests/test_normalizer.py -q: 8 passed after the correction.
  • pre-commit run --files tests/test_tokenizer.py: all applicable pinned hooks pass.
  • No model weights were downloaded and no inference evaluation was run.

Use the multilingual parameter instead of always constructing the English-only tokenizer, and verify each mode's default start-of-transcript prefix while retaining the language-token checks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant