Benjamin Peterson | 90f5ba5 | 2010-03-11 22:53:45 +0000 | [diff] [blame] | 1 | #! /usr/bin/env python3 |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 2 | # -*- coding: iso-8859-1 -*- |
Benjamin Peterson | eaedaec | 2013-12-22 19:45:38 -0600 | [diff] [blame] | 3 | # Originally written by Barry Warsaw <barry@python.org> |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 4 | # |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 5 | # Minimally patched to make it even more xgettext compatible |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 6 | # by Peter Funk <pf@artcom-gmbh.de> |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 7 | # |
| 8 | # 2002-11-22 Jürgen Hermann <jh@web.de> |
| 9 | # Added checks that _() only contains string literals, and |
| 10 | # command line args are resolved to module lists, i.e. you |
| 11 | # can now pass a filename, a module or package name, or a |
| 12 | # directory (including globbing chars, important for Win32). |
| 13 | # Made docstring fit in 80 chars wide displays using pydoc. |
| 14 | # |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 15 | |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 16 | # for selftesting |
| 17 | try: |
| 18 | import fintl |
| 19 | _ = fintl.gettext |
| 20 | except ImportError: |
| 21 | _ = lambda s: s |
| 22 | |
| 23 | __doc__ = _("""pygettext -- Python equivalent of xgettext(1) |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 24 | |
| 25 | Many systems (Solaris, Linux, Gnu) provide extensive tools that ease the |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 26 | internationalization of C programs. Most of these tools are independent of |
| 27 | the programming language and can be used from within Python programs. |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 28 | Martin von Loewis' work[1] helps considerably in this regard. |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 29 | |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 30 | There's one problem though; xgettext is the program that scans source code |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 31 | looking for message strings, but it groks only C (or C++). Python |
| 32 | introduces a few wrinkles, such as dual quoting characters, triple quoted |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 33 | strings, and raw strings. xgettext understands none of this. |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 34 | |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 35 | Enter pygettext, which uses Python's standard tokenize module to scan |
| 36 | Python source code, generating .pot files identical to what GNU xgettext[2] |
| 37 | generates for C and C++ code. From there, the standard GNU tools can be |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 38 | used. |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 39 | |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 40 | A word about marking Python strings as candidates for translation. GNU |
| 41 | xgettext recognizes the following keywords: gettext, dgettext, dcgettext, |
| 42 | and gettext_noop. But those can be a lot of text to include all over your |
| 43 | code. C and C++ have a trick: they use the C preprocessor. Most |
| 44 | internationalized C source includes a #define for gettext() to _() so that |
| 45 | what has to be written in the source is much less. Thus these are both |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 46 | translatable strings: |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 47 | |
| 48 | gettext("Translatable String") |
| 49 | _("Translatable String") |
| 50 | |
| 51 | Python of course has no preprocessor so this doesn't work so well. Thus, |
| 52 | pygettext searches only for _() by default, but see the -k/--keyword flag |
| 53 | below for how to augment this. |
| 54 | |
| 55 | [1] http://www.python.org/workshops/1997-10/proceedings/loewis.html |
| 56 | [2] http://www.gnu.org/software/gettext/gettext.html |
| 57 | |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 58 | NOTE: pygettext attempts to be option and feature compatible with GNU |
| 59 | xgettext where ever possible. However some options are still missing or are |
| 60 | not fully implemented. Also, xgettext's use of command line switches with |
| 61 | option arguments is broken, and in these cases, pygettext just defines |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 62 | additional switches. |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 63 | |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 64 | Usage: pygettext [options] inputfile ... |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 65 | |
| 66 | Options: |
| 67 | |
| 68 | -a |
| 69 | --extract-all |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 70 | Extract all strings. |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 71 | |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 72 | -d name |
| 73 | --default-domain=name |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 74 | Rename the default output file from messages.pot to name.pot. |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 75 | |
| 76 | -E |
| 77 | --escape |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 78 | Replace non-ASCII characters with octal escape sequences. |
| 79 | |
| 80 | -D |
| 81 | --docstrings |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 82 | Extract module, class, method, and function docstrings. These do |
| 83 | not need to be wrapped in _() markers, and in fact cannot be for |
| 84 | Python to consider them docstrings. (See also the -X option). |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 85 | |
| 86 | -h |
| 87 | --help |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 88 | Print this help message and exit. |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 89 | |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 90 | -k word |
| 91 | --keyword=word |
| 92 | Keywords to look for in addition to the default set, which are: |
| 93 | %(DEFAULTKEYWORDS)s |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 94 | |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 95 | You can have multiple -k flags on the command line. |
| 96 | |
| 97 | -K |
| 98 | --no-default-keywords |
| 99 | Disable the default set of keywords (see above). Any keywords |
| 100 | explicitly added with the -k/--keyword option are still recognized. |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 101 | |
| 102 | --no-location |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 103 | Do not write filename/lineno location comments. |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 104 | |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 105 | -n |
| 106 | --add-location |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 107 | Write filename/lineno location comments indicating where each |
| 108 | extracted string is found in the source. These lines appear before |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 109 | each msgid. The style of comments is controlled by the -S/--style |
| 110 | option. This is the default. |
| 111 | |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 112 | -o filename |
| 113 | --output=filename |
| 114 | Rename the default output file from messages.pot to filename. If |
| 115 | filename is `-' then the output is sent to standard out. |
| 116 | |
| 117 | -p dir |
| 118 | --output-dir=dir |
| 119 | Output files will be placed in directory dir. |
| 120 | |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 121 | -S stylename |
| 122 | --style stylename |
| 123 | Specify which style to use for location comments. Two styles are |
| 124 | supported: |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 125 | |
| 126 | Solaris # File: filename, line: line-number |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 127 | GNU #: filename:line |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 128 | |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 129 | The style name is case insensitive. GNU style is the default. |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 130 | |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 131 | -v |
| 132 | --verbose |
| 133 | Print the names of the files being processed. |
| 134 | |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 135 | -V |
| 136 | --version |
| 137 | Print the version of pygettext and exit. |
| 138 | |
| 139 | -w columns |
| 140 | --width=columns |
| 141 | Set width of output to columns. |
| 142 | |
| 143 | -x filename |
| 144 | --exclude-file=filename |
| 145 | Specify a file that contains a list of strings that are not be |
| 146 | extracted from the input files. Each string to be excluded must |
| 147 | appear on a line by itself in the file. |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 148 | |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 149 | -X filename |
| 150 | --no-docstrings=filename |
| 151 | Specify a file that contains a list of files (one per line) that |
| 152 | should not have their docstrings extracted. This is only useful in |
| 153 | conjunction with the -D option above. |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 154 | |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 155 | If `inputfile' is -, standard input is read. |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 156 | """) |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 157 | |
| 158 | import os |
Victor Stinner | 328cb1f | 2016-04-12 18:46:10 +0200 | [diff] [blame] | 159 | import importlib.machinery |
| 160 | import importlib.util |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 161 | import sys |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 162 | import glob |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 163 | import time |
| 164 | import getopt |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 165 | import token |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 166 | import tokenize |
| 167 | |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 168 | __version__ = '1.5' |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 169 | |
| 170 | default_keywords = ['_'] |
| 171 | DEFAULTKEYWORDS = ', '.join(default_keywords) |
| 172 | |
| 173 | EMPTYSTRING = '' |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 174 | |
| 175 | |
| 176 | |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 177 | # The normal pot-file header. msgmerge and Emacs's po-mode work better if it's |
| 178 | # there. |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 179 | pot_header = _('''\ |
| 180 | # SOME DESCRIPTIVE TITLE. |
| 181 | # Copyright (C) YEAR ORGANIZATION |
| 182 | # FIRST AUTHOR <EMAIL@ADDRESS>, YEAR. |
| 183 | # |
| 184 | msgid "" |
| 185 | msgstr "" |
| 186 | "Project-Id-Version: PACKAGE VERSION\\n" |
Martin v. Löwis | 0f6b383 | 2001-03-01 22:56:17 +0000 | [diff] [blame] | 187 | "POT-Creation-Date: %(time)s\\n" |
| 188 | "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\\n" |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 189 | "Last-Translator: FULL NAME <EMAIL@ADDRESS>\\n" |
| 190 | "Language-Team: LANGUAGE <LL@li.org>\\n" |
| 191 | "MIME-Version: 1.0\\n" |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 192 | "Content-Type: text/plain; charset=%(charset)s\\n" |
| 193 | "Content-Transfer-Encoding: %(encoding)s\\n" |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 194 | "Generated-By: pygettext.py %(version)s\\n" |
| 195 | |
| 196 | ''') |
| 197 | |
| 198 | |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 199 | def usage(code, msg=''): |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 200 | print(__doc__ % globals(), file=sys.stderr) |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 201 | if msg: |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 202 | print(msg, file=sys.stderr) |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 203 | sys.exit(code) |
| 204 | |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 205 | |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 206 | |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 207 | def make_escapes(pass_nonascii): |
| 208 | global escapes, escape |
| 209 | if pass_nonascii: |
| 210 | # Allow non-ascii characters to pass through so that e.g. 'msgid |
Barry Warsaw | 7733e12 | 2000-02-27 14:30:48 +0000 | [diff] [blame] | 211 | # "Höhe"' would result not result in 'msgid "H\366he"'. Otherwise we |
| 212 | # escape any character outside the 32..126 range. |
| 213 | mod = 128 |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 214 | escape = escape_ascii |
Barry Warsaw | 7733e12 | 2000-02-27 14:30:48 +0000 | [diff] [blame] | 215 | else: |
| 216 | mod = 256 |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 217 | escape = escape_nonascii |
| 218 | escapes = [r"\%03o" % i for i in range(mod)] |
| 219 | for i in range(32, 127): |
| 220 | escapes[i] = chr(i) |
| 221 | escapes[ord('\\')] = r'\\' |
| 222 | escapes[ord('\t')] = r'\t' |
| 223 | escapes[ord('\r')] = r'\r' |
| 224 | escapes[ord('\n')] = r'\n' |
| 225 | escapes[ord('\"')] = r'\"' |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 226 | |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 227 | |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 228 | def escape_ascii(s, encoding): |
| 229 | return ''.join(escapes[ord(c)] if ord(c) < 128 else c for c in s) |
| 230 | |
| 231 | def escape_nonascii(s, encoding): |
| 232 | return ''.join(escapes[b] for b in s.encode(encoding)) |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 233 | |
| 234 | |
Serhiy Storchaka | 6952482 | 2018-04-19 09:23:03 +0300 | [diff] [blame] | 235 | def is_literal_string(s): |
| 236 | return s[0] in '\'"' or (s[0] in 'rRuU' and s[1] in '\'"') |
| 237 | |
| 238 | |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 239 | def safe_eval(s): |
| 240 | # unwrap quotes, safely |
| 241 | return eval(s, {'__builtins__':{}}, {}) |
| 242 | |
| 243 | |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 244 | def normalize(s, encoding): |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 245 | # This converts the various Python string types into a format that is |
| 246 | # appropriate for .po files, namely much closer to C style. |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 247 | lines = s.split('\n') |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 248 | if len(lines) == 1: |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 249 | s = '"' + escape(s, encoding) + '"' |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 250 | else: |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 251 | if not lines[-1]: |
| 252 | del lines[-1] |
| 253 | lines[-1] = lines[-1] + '\n' |
| 254 | for i in range(len(lines)): |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 255 | lines[i] = escape(lines[i], encoding) |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 256 | lineterm = '\\n"\n"' |
| 257 | s = '""\n"' + lineterm.join(lines) + '"' |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 258 | return s |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 259 | |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 260 | |
| 261 | def containsAny(str, set): |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 262 | """Check whether 'str' contains ANY of the chars in 'set'""" |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 263 | return 1 in [c in str for c in set] |
| 264 | |
| 265 | |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 266 | def getFilesForName(name): |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 267 | """Get a list of module files for a filename, a module or package name, |
| 268 | or a directory. |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 269 | """ |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 270 | if not os.path.exists(name): |
| 271 | # check for glob chars |
| 272 | if containsAny(name, "*?[]"): |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 273 | files = glob.glob(name) |
| 274 | list = [] |
| 275 | for file in files: |
| 276 | list.extend(getFilesForName(file)) |
| 277 | return list |
| 278 | |
| 279 | # try to find module or package |
Victor Stinner | 328cb1f | 2016-04-12 18:46:10 +0200 | [diff] [blame] | 280 | try: |
| 281 | spec = importlib.util.find_spec(name) |
| 282 | name = spec.origin |
| 283 | except ImportError: |
| 284 | name = None |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 285 | if not name: |
| 286 | return [] |
| 287 | |
| 288 | if os.path.isdir(name): |
| 289 | # find all python files in directory |
| 290 | list = [] |
Serhiy Storchaka | c93938b | 2018-04-09 20:09:17 +0300 | [diff] [blame] | 291 | # get extension for python source files |
| 292 | _py_ext = importlib.machinery.SOURCE_SUFFIXES[0] |
| 293 | for root, dirs, files in os.walk(name): |
| 294 | # don't recurse into CVS directories |
| 295 | if 'CVS' in dirs: |
| 296 | dirs.remove('CVS') |
| 297 | # add all *.py files to list |
| 298 | list.extend( |
| 299 | [os.path.join(root, file) for file in files |
| 300 | if os.path.splitext(file)[1] == _py_ext] |
| 301 | ) |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 302 | return list |
| 303 | elif os.path.exists(name): |
| 304 | # a single file |
| 305 | return [name] |
| 306 | |
| 307 | return [] |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 308 | |
| 309 | |
| 310 | class TokenEater: |
| 311 | def __init__(self, options): |
| 312 | self.__options = options |
| 313 | self.__messages = {} |
| 314 | self.__state = self.__waiting |
| 315 | self.__data = [] |
| 316 | self.__lineno = -1 |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 317 | self.__freshmodule = 1 |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 318 | self.__curfile = None |
Tobotimus | eee72d4 | 2018-02-27 09:48:14 +1100 | [diff] [blame] | 319 | self.__enclosurecount = 0 |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 320 | |
| 321 | def __call__(self, ttype, tstring, stup, etup, line): |
| 322 | # dispatch |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 323 | ## import token |
Serhiy Storchaka | 6952482 | 2018-04-19 09:23:03 +0300 | [diff] [blame] | 324 | ## print('ttype:', token.tok_name[ttype], 'tstring:', tstring, |
| 325 | ## file=sys.stderr) |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 326 | self.__state(ttype, tstring, stup[0]) |
| 327 | |
| 328 | def __waiting(self, ttype, tstring, lineno): |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 329 | opts = self.__options |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 330 | # Do docstring extractions, if enabled |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 331 | if opts.docstrings and not opts.nodocstrings.get(self.__curfile): |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 332 | # module docstring? |
| 333 | if self.__freshmodule: |
Serhiy Storchaka | 6952482 | 2018-04-19 09:23:03 +0300 | [diff] [blame] | 334 | if ttype == tokenize.STRING and is_literal_string(tstring): |
Barry Warsaw | 16b62c1 | 2001-05-21 19:51:26 +0000 | [diff] [blame] | 335 | self.__addentry(safe_eval(tstring), lineno, isdocstring=1) |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 336 | self.__freshmodule = 0 |
| 337 | elif ttype not in (tokenize.COMMENT, tokenize.NL): |
| 338 | self.__freshmodule = 0 |
| 339 | return |
Tobotimus | eee72d4 | 2018-02-27 09:48:14 +1100 | [diff] [blame] | 340 | # class or func/method docstring? |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 341 | if ttype == tokenize.NAME and tstring in ('class', 'def'): |
| 342 | self.__state = self.__suiteseen |
| 343 | return |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 344 | if ttype == tokenize.NAME and tstring in opts.keywords: |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 345 | self.__state = self.__keywordseen |
| 346 | |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 347 | def __suiteseen(self, ttype, tstring, lineno): |
Tobotimus | eee72d4 | 2018-02-27 09:48:14 +1100 | [diff] [blame] | 348 | # skip over any enclosure pairs until we see the colon |
| 349 | if ttype == tokenize.OP: |
| 350 | if tstring == ':' and self.__enclosurecount == 0: |
| 351 | # we see a colon and we're not in an enclosure: end of def |
| 352 | self.__state = self.__suitedocstring |
| 353 | elif tstring in '([{': |
| 354 | self.__enclosurecount += 1 |
| 355 | elif tstring in ')]}': |
| 356 | self.__enclosurecount -= 1 |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 357 | |
| 358 | def __suitedocstring(self, ttype, tstring, lineno): |
| 359 | # ignore any intervening noise |
Serhiy Storchaka | 6952482 | 2018-04-19 09:23:03 +0300 | [diff] [blame] | 360 | if ttype == tokenize.STRING and is_literal_string(tstring): |
Barry Warsaw | 16b62c1 | 2001-05-21 19:51:26 +0000 | [diff] [blame] | 361 | self.__addentry(safe_eval(tstring), lineno, isdocstring=1) |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 362 | self.__state = self.__waiting |
| 363 | elif ttype not in (tokenize.NEWLINE, tokenize.INDENT, |
| 364 | tokenize.COMMENT): |
| 365 | # there was no class docstring |
| 366 | self.__state = self.__waiting |
| 367 | |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 368 | def __keywordseen(self, ttype, tstring, lineno): |
| 369 | if ttype == tokenize.OP and tstring == '(': |
| 370 | self.__data = [] |
| 371 | self.__lineno = lineno |
| 372 | self.__state = self.__openseen |
| 373 | else: |
| 374 | self.__state = self.__waiting |
| 375 | |
| 376 | def __openseen(self, ttype, tstring, lineno): |
| 377 | if ttype == tokenize.OP and tstring == ')': |
| 378 | # We've seen the last of the translatable strings. Record the |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 379 | # line number of the first line of the strings and update the list |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 380 | # of messages seen. Reset state for the next batch. If there |
| 381 | # were no strings inside _(), then just ignore this entry. |
| 382 | if self.__data: |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 383 | self.__addentry(EMPTYSTRING.join(self.__data)) |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 384 | self.__state = self.__waiting |
Serhiy Storchaka | 6952482 | 2018-04-19 09:23:03 +0300 | [diff] [blame] | 385 | elif ttype == tokenize.STRING and is_literal_string(tstring): |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 386 | self.__data.append(safe_eval(tstring)) |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 387 | elif ttype not in [tokenize.COMMENT, token.INDENT, token.DEDENT, |
| 388 | token.NEWLINE, tokenize.NL]: |
| 389 | # warn if we see anything else than STRING or whitespace |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 390 | print(_( |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 391 | '*** %(file)s:%(lineno)s: Seen unexpected token "%(token)s"' |
| 392 | ) % { |
| 393 | 'token': tstring, |
| 394 | 'file': self.__curfile, |
| 395 | 'lineno': self.__lineno |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 396 | }, file=sys.stderr) |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 397 | self.__state = self.__waiting |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 398 | |
Barry Warsaw | 16b62c1 | 2001-05-21 19:51:26 +0000 | [diff] [blame] | 399 | def __addentry(self, msg, lineno=None, isdocstring=0): |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 400 | if lineno is None: |
| 401 | lineno = self.__lineno |
| 402 | if not msg in self.__options.toexclude: |
| 403 | entry = (self.__curfile, lineno) |
Barry Warsaw | 16b62c1 | 2001-05-21 19:51:26 +0000 | [diff] [blame] | 404 | self.__messages.setdefault(msg, {})[entry] = isdocstring |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 405 | |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 406 | def set_filename(self, filename): |
| 407 | self.__curfile = filename |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 408 | self.__freshmodule = 1 |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 409 | |
| 410 | def write(self, fp): |
| 411 | options = self.__options |
R David Murray | 2b78129 | 2015-04-16 12:15:09 -0400 | [diff] [blame] | 412 | timestamp = time.strftime('%Y-%m-%d %H:%M%z') |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 413 | encoding = fp.encoding if fp.encoding else 'UTF-8' |
| 414 | print(pot_header % {'time': timestamp, 'version': __version__, |
| 415 | 'charset': encoding, |
| 416 | 'encoding': '8bit'}, file=fp) |
Barry Warsaw | 128c77d | 2001-05-23 16:59:45 +0000 | [diff] [blame] | 417 | # Sort the entries. First sort each particular entry's keys, then |
| 418 | # sort all the entries by their first item. |
| 419 | reverse = {} |
Fred Drake | 33e2c3e | 2000-10-26 03:49:15 +0000 | [diff] [blame] | 420 | for k, v in self.__messages.items(): |
Guido van Rossum | f7bd964 | 2008-01-15 17:41:38 +0000 | [diff] [blame] | 421 | keys = sorted(v.keys()) |
Barry Warsaw | 50cf706 | 2001-05-24 23:06:13 +0000 | [diff] [blame] | 422 | reverse.setdefault(tuple(keys), []).append((k, v)) |
Guido van Rossum | f7bd964 | 2008-01-15 17:41:38 +0000 | [diff] [blame] | 423 | rkeys = sorted(reverse.keys()) |
Barry Warsaw | 128c77d | 2001-05-23 16:59:45 +0000 | [diff] [blame] | 424 | for rkey in rkeys: |
Barry Warsaw | 50cf706 | 2001-05-24 23:06:13 +0000 | [diff] [blame] | 425 | rentries = reverse[rkey] |
| 426 | rentries.sort() |
| 427 | for k, v in rentries: |
| 428 | # If the entry was gleaned out of a docstring, then add a |
| 429 | # comment stating so. This is to aid translators who may wish |
| 430 | # to skip translating some unimportant docstrings. |
Guido van Rossum | 89da5d7 | 2006-08-22 00:21:25 +0000 | [diff] [blame] | 431 | isdocstring = any(v.values()) |
Barry Warsaw | 50cf706 | 2001-05-24 23:06:13 +0000 | [diff] [blame] | 432 | # k is the message string, v is a dictionary-set of (filename, |
| 433 | # lineno) tuples. We want to sort the entries in v first by |
| 434 | # file name and then by line number. |
Guido van Rossum | f7bd964 | 2008-01-15 17:41:38 +0000 | [diff] [blame] | 435 | v = sorted(v.keys()) |
Barry Warsaw | 50cf706 | 2001-05-24 23:06:13 +0000 | [diff] [blame] | 436 | if not options.writelocations: |
| 437 | pass |
| 438 | # location comments are different b/w Solaris and GNU: |
| 439 | elif options.locationstyle == options.SOLARIS: |
| 440 | for filename, lineno in v: |
| 441 | d = {'filename': filename, 'lineno': lineno} |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 442 | print(_( |
| 443 | '# File: %(filename)s, line: %(lineno)d') % d, file=fp) |
Barry Warsaw | 50cf706 | 2001-05-24 23:06:13 +0000 | [diff] [blame] | 444 | elif options.locationstyle == options.GNU: |
| 445 | # fit as many locations on one line, as long as the |
Martin Panter | 69332c1 | 2016-08-04 13:07:31 +0000 | [diff] [blame] | 446 | # resulting line length doesn't exceed 'options.width' |
Barry Warsaw | 50cf706 | 2001-05-24 23:06:13 +0000 | [diff] [blame] | 447 | locline = '#:' |
| 448 | for filename, lineno in v: |
| 449 | d = {'filename': filename, 'lineno': lineno} |
| 450 | s = _(' %(filename)s:%(lineno)d') % d |
| 451 | if len(locline) + len(s) <= options.width: |
| 452 | locline = locline + s |
| 453 | else: |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 454 | print(locline, file=fp) |
Barry Warsaw | 50cf706 | 2001-05-24 23:06:13 +0000 | [diff] [blame] | 455 | locline = "#:" + s |
| 456 | if len(locline) > 2: |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 457 | print(locline, file=fp) |
Barry Warsaw | 5c94ce5 | 2001-06-20 19:41:40 +0000 | [diff] [blame] | 458 | if isdocstring: |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 459 | print('#, docstring', file=fp) |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 460 | print('msgid', normalize(k, encoding), file=fp) |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 461 | print('msgstr ""\n', file=fp) |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 462 | |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 463 | |
| 464 | |
| 465 | def main(): |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 466 | global default_keywords |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 467 | try: |
| 468 | opts, args = getopt.getopt( |
| 469 | sys.argv[1:], |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 470 | 'ad:DEhk:Kno:p:S:Vvw:x:X:', |
Barry Warsaw | 2b63969 | 2001-05-21 19:58:23 +0000 | [diff] [blame] | 471 | ['extract-all', 'default-domain=', 'escape', 'help', |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 472 | 'keyword=', 'no-default-keywords', |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 473 | 'add-location', 'no-location', 'output=', 'output-dir=', |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 474 | 'style=', 'verbose', 'version', 'width=', 'exclude-file=', |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 475 | 'docstrings', 'no-docstrings', |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 476 | ]) |
Guido van Rossum | b940e11 | 2007-01-10 16:19:56 +0000 | [diff] [blame] | 477 | except getopt.error as msg: |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 478 | usage(1, msg) |
| 479 | |
| 480 | # for holding option values |
| 481 | class Options: |
| 482 | # constants |
| 483 | GNU = 1 |
| 484 | SOLARIS = 2 |
| 485 | # defaults |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 486 | extractall = 0 # FIXME: currently this option has no effect at all. |
| 487 | escape = 0 |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 488 | keywords = [] |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 489 | outpath = '' |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 490 | outfile = 'messages.pot' |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 491 | writelocations = 1 |
| 492 | locationstyle = GNU |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 493 | verbose = 0 |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 494 | width = 78 |
| 495 | excludefilename = '' |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 496 | docstrings = 0 |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 497 | nodocstrings = {} |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 498 | |
| 499 | options = Options() |
| 500 | locations = {'gnu' : options.GNU, |
| 501 | 'solaris' : options.SOLARIS, |
| 502 | } |
| 503 | |
| 504 | # parse options |
| 505 | for opt, arg in opts: |
| 506 | if opt in ('-h', '--help'): |
| 507 | usage(0) |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 508 | elif opt in ('-a', '--extract-all'): |
| 509 | options.extractall = 1 |
| 510 | elif opt in ('-d', '--default-domain'): |
| 511 | options.outfile = arg + '.pot' |
| 512 | elif opt in ('-E', '--escape'): |
| 513 | options.escape = 1 |
Barry Warsaw | 08a8a35 | 2000-10-27 04:56:28 +0000 | [diff] [blame] | 514 | elif opt in ('-D', '--docstrings'): |
| 515 | options.docstrings = 1 |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 516 | elif opt in ('-k', '--keyword'): |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 517 | options.keywords.append(arg) |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 518 | elif opt in ('-K', '--no-default-keywords'): |
| 519 | default_keywords = [] |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 520 | elif opt in ('-n', '--add-location'): |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 521 | options.writelocations = 1 |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 522 | elif opt in ('--no-location',): |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 523 | options.writelocations = 0 |
| 524 | elif opt in ('-S', '--style'): |
| 525 | options.locationstyle = locations.get(arg.lower()) |
| 526 | if options.locationstyle is None: |
| 527 | usage(1, _('Invalid value for --style: %s') % arg) |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 528 | elif opt in ('-o', '--output'): |
| 529 | options.outfile = arg |
| 530 | elif opt in ('-p', '--output-dir'): |
| 531 | options.outpath = arg |
Barry Warsaw | 5dbf526 | 1999-11-03 18:47:52 +0000 | [diff] [blame] | 532 | elif opt in ('-v', '--verbose'): |
| 533 | options.verbose = 1 |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 534 | elif opt in ('-V', '--version'): |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 535 | print(_('pygettext.py (xgettext for Python) %s') % __version__) |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 536 | sys.exit(0) |
| 537 | elif opt in ('-w', '--width'): |
| 538 | try: |
| 539 | options.width = int(arg) |
| 540 | except ValueError: |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 541 | usage(1, _('--width argument must be an integer: %s') % arg) |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 542 | elif opt in ('-x', '--exclude-file'): |
| 543 | options.excludefilename = arg |
Barry Warsaw | 63ce5af | 2001-07-27 16:47:18 +0000 | [diff] [blame] | 544 | elif opt in ('-X', '--no-docstrings'): |
| 545 | fp = open(arg) |
| 546 | try: |
| 547 | while 1: |
| 548 | line = fp.readline() |
| 549 | if not line: |
| 550 | break |
| 551 | options.nodocstrings[line[:-1]] = 1 |
| 552 | finally: |
| 553 | fp.close() |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 554 | |
| 555 | # calculate escapes |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 556 | make_escapes(not options.escape) |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 557 | |
| 558 | # calculate all keywords |
| 559 | options.keywords.extend(default_keywords) |
| 560 | |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 561 | # initialize list of strings to exclude |
| 562 | if options.excludefilename: |
| 563 | try: |
Serhiy Storchaka | 172bb39 | 2019-03-30 08:33:02 +0200 | [diff] [blame] | 564 | with open(options.excludefilename) as fp: |
| 565 | options.toexclude = fp.readlines() |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 566 | except IOError: |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 567 | print(_( |
| 568 | "Can't read --exclude-file: %s") % options.excludefilename, file=sys.stderr) |
Barry Warsaw | c8f0892 | 2000-02-26 20:56:47 +0000 | [diff] [blame] | 569 | sys.exit(1) |
| 570 | else: |
| 571 | options.toexclude = [] |
| 572 | |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 573 | # resolve args to module lists |
| 574 | expanded = [] |
| 575 | for arg in args: |
| 576 | if arg == '-': |
| 577 | expanded.append(arg) |
| 578 | else: |
| 579 | expanded.extend(getFilesForName(arg)) |
| 580 | args = expanded |
| 581 | |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 582 | # slurp through all the files |
| 583 | eater = TokenEater(options) |
| 584 | for filename in args: |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 585 | if filename == '-': |
| 586 | if options.verbose: |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 587 | print(_('Reading standard input')) |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 588 | fp = sys.stdin.buffer |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 589 | closep = 0 |
| 590 | else: |
| 591 | if options.verbose: |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 592 | print(_('Working on %s') % filename) |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 593 | fp = open(filename, 'rb') |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 594 | closep = 1 |
| 595 | try: |
| 596 | eater.set_filename(filename) |
Barry Warsaw | 75ee8f5 | 2001-02-26 04:46:53 +0000 | [diff] [blame] | 597 | try: |
Serhiy Storchaka | b6ed173 | 2013-02-09 22:37:22 +0200 | [diff] [blame] | 598 | tokens = tokenize.tokenize(fp.readline) |
Trent Nelson | 428de65 | 2008-03-18 22:41:35 +0000 | [diff] [blame] | 599 | for _token in tokens: |
| 600 | eater(*_token) |
Guido van Rossum | b940e11 | 2007-01-10 16:19:56 +0000 | [diff] [blame] | 601 | except tokenize.TokenError as e: |
Collin Winter | 6afaeb7 | 2007-08-03 17:06:41 +0000 | [diff] [blame] | 602 | print('%s: %s, line %d, column %d' % ( |
Georg Brandl | 6464d47 | 2007-10-22 16:16:13 +0000 | [diff] [blame] | 603 | e.args[0], filename, e.args[1][0], e.args[1][1]), |
| 604 | file=sys.stderr) |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 605 | finally: |
| 606 | if closep: |
| 607 | fp.close() |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 608 | |
Barry Warsaw | a17e0f1 | 2000-03-08 15:18:35 +0000 | [diff] [blame] | 609 | # write the output |
| 610 | if options.outfile == '-': |
| 611 | fp = sys.stdout |
| 612 | closep = 0 |
| 613 | else: |
| 614 | if options.outpath: |
| 615 | options.outfile = os.path.join(options.outpath, options.outfile) |
| 616 | fp = open(options.outfile, 'w') |
| 617 | closep = 1 |
| 618 | try: |
| 619 | eater.write(fp) |
| 620 | finally: |
| 621 | if closep: |
| 622 | fp.close() |
Barry Warsaw | e27db5a | 1999-08-13 20:59:48 +0000 | [diff] [blame] | 623 | |
| 624 | |
| 625 | if __name__ == '__main__': |
| 626 | main() |
Barry Warsaw | 75a6e67 | 2000-05-02 19:28:30 +0000 | [diff] [blame] | 627 | # some more test strings |
Barry Warsaw | e04ee70 | 2003-04-16 18:08:23 +0000 | [diff] [blame] | 628 | # this one creates a warning |
| 629 | _('*** Seen unexpected token "%(token)s"') % {'token': 'test'} |
Martin v. Löwis | 0d1fdea | 2002-11-22 08:36:54 +0000 | [diff] [blame] | 630 | _('more' 'than' 'one' 'string') |