Issue #25317: Converted doctests in test_tokenize to unittests.
Made test_tokenize discoverable.