Skip to content

gh-140950: Read pushed shlex source after EOF#144560

Open
ThanhNguyxn wants to merge 1 commit intopython:mainfrom
ThanhNguyxn:gh-140950-push-source-eof
Open

gh-140950: Read pushed shlex source after EOF#144560
ThanhNguyxn wants to merge 1 commit intopython:mainfrom
ThanhNguyxn:gh-140950-push-source-eof

Conversation

@ThanhNguyxn
Copy link

Summary

  • Reset lexer state in shlex.push_source() so pushing a new source after EOF can continue tokenization.
  • Add regression test test_push_source_after_eof in Lib/test/test_shlex.py covering both POSIX and non-POSIX modes.
  • Add NEWS entry for gh-140950.

Repro

from shlex import shlex
lexer = shlex('a')
lexer.get_token()  # 'a'
lexer.get_token()  # EOF
lexer.push_source('b')
lexer.get_token()  # now returns 'b'

Verification

  • python -m py_compile Lib/shlex.py Lib/test/test_shlex.py
  • python -c "import importlib.util, sys, unittest; shlex_spec=importlib.util.spec_from_file_location('shlex', r'D:\Code\cpython\Lib\shlex.py'); shlex_mod=importlib.util.module_from_spec(shlex_spec); shlex_spec.loader.exec_module(shlex_mod); sys.modules['shlex']=shlex_mod; test_spec=importlib.util.spec_from_file_location('test_shlex_local', r'D:\Code\cpython\Lib\test\test_shlex.py'); test_mod=importlib.util.module_from_spec(test_spec); test_spec.loader.exec_module(test_mod); suite=unittest.TestSuite([test_mod.ShlexTest('test_push_source_after_eof')]); result=unittest.TextTestRunner(verbosity=2).run(suite); raise SystemExit(0 if result.wasSuccessful() else 1)"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant