bpo-28595: Allow shlex whitespace_split with punctuation_chars (GH-2071)
diff --git a/Doc/library/shlex.rst b/Doc/library/shlex.rst
index 8c5b023..a8421fd 100644
--- a/Doc/library/shlex.rst
+++ b/Doc/library/shlex.rst
@@ -225,7 +225,8 @@
appear in filename specifications and command line parameters, will also be
included in this attribute, and any characters which appear in
``punctuation_chars`` will be removed from ``wordchars`` if they are present
- there.
+ there. If :attr:`whitespace_split` is set to ``True``, this will have no
+ effect.
.. attribute:: shlex.whitespace
@@ -258,11 +259,13 @@
If ``True``, tokens will only be split in whitespaces. This is useful, for
example, for parsing command lines with :class:`~shlex.shlex`, getting
- tokens in a similar way to shell arguments. If this attribute is ``True``,
- :attr:`punctuation_chars` will have no effect, and splitting will happen
- only on whitespaces. When using :attr:`punctuation_chars`, which is
- intended to provide parsing closer to that implemented by shells, it is
- advisable to leave ``whitespace_split`` as ``False`` (the default value).
+ tokens in a similar way to shell arguments. When used in combination with
+ :attr:`punctuation_chars`, tokens will be split on whitespace in addition to
+ those characters.
+
+ .. versionchanged:: 3.8
+ The :attr:`punctuation_chars` attribute was made compatible with the
+ :attr:`whitespace_split` attribute.
.. attribute:: shlex.infile
@@ -398,12 +401,15 @@
>>> import shlex
>>> text = "a && b; c && d || e; f >'abc'; (def \"ghi\")"
- >>> list(shlex.shlex(text))
- ['a', '&', '&', 'b', ';', 'c', '&', '&', 'd', '|', '|', 'e', ';', 'f', '>',
- "'abc'", ';', '(', 'def', '"ghi"', ')']
- >>> list(shlex.shlex(text, punctuation_chars=True))
- ['a', '&&', 'b', ';', 'c', '&&', 'd', '||', 'e', ';', 'f', '>', "'abc'",
- ';', '(', 'def', '"ghi"', ')']
+ >>> s = shlex.shlex(text, posix=True)
+ >>> s.whitespace_split = True
+ >>> list(s)
+ ['a', '&&', 'b;', 'c', '&&', 'd', '||', 'e;', 'f', '>abc;', '(def', 'ghi)']
+ >>> s = shlex.shlex(text, posix=True, punctuation_chars=True)
+ >>> s.whitespace_split = True
+ >>> list(s)
+ ['a', '&&', 'b', ';', 'c', '&&', 'd', '||', 'e', ';', 'f', '>', 'abc', ';',
+ '(', 'def', 'ghi', ')']
Of course, tokens will be returned which are not valid for shells, and you'll
need to implement your own error checks on the returned tokens.
@@ -428,6 +434,11 @@
>>> list(s)
['~/a', '&&', 'b-c', '--color=auto', '||', 'd', '*.py?']
+ However, to match the shell as closely as possible, it is recommended to
+ always use ``posix`` and :attr:`~shlex.whitespace_split` when using
+ :attr:`~shlex.punctuation_chars`, which will negate
+ :attr:`~shlex.wordchars` entirely.
+
For best effect, ``punctuation_chars`` should be set in conjunction with
``posix=True``. (Note that ``posix=False`` is the default for
:class:`~shlex.shlex`.)