blob: a25a021eb31d28154b709d6a489cecf12c8937b0 [file] [log] [blame]
Guido van Rossum7d5b99d1997-11-21 17:12:59 +00001"""Helper class to quickly write a loop over all standard input files.
2
3Typical use is:
4
5 import fileinput
6 for line in fileinput.input():
7 process(line)
8
9This iterates over the lines of all files listed in sys.argv[1:],
10defaulting to sys.stdin if the list is empty. If a filename is '-' it
11is also replaced by sys.stdin. To specify an alternative list of
12filenames, pass it as the argument to input(). A single file name is
13also allowed.
14
15Functions filename(), lineno() return the filename and cumulative line
16number of the line that has just been read; filelineno() returns its
17line number in the current file; isfirstline() returns true iff the
18line just read is the first line of its file; isstdin() returns true
19iff the line was read from sys.stdin. Function nextfile() closes the
20current file so that the next iteration will read the first line from
21the next file (if any); lines not read from the file will not count
22towards the cumulative line count; the filename is not changed until
23after the first line of the next file has been read. Function close()
24closes the sequence.
25
26Before any lines have been read, filename() returns None and both line
27numbers are zero; nextfile() has no effect. After all lines have been
28read, filename() and the line number functions return the values
29pertaining to the last line read; nextfile() has no effect.
30
Georg Brandlc029f872006-02-19 14:12:34 +000031All files are opened in text mode by default, you can override this by
32setting the mode parameter to input() or FileInput.__init__().
33If an I/O error occurs during opening or reading a file, the IOError
34exception is raised.
Guido van Rossum7d5b99d1997-11-21 17:12:59 +000035
36If sys.stdin is used more than once, the second and further use will
37return no lines, except perhaps for interactive use, or if it has been
38explicitly reset (e.g. using sys.stdin.seek(0)).
39
40Empty files are opened and immediately closed; the only time their
41presence in the list of filenames is noticeable at all is when the
42last file opened is empty.
43
44It is possible that the last line of a file doesn't end in a newline
45character; otherwise lines are returned including the trailing
46newline.
47
48Class FileInput is the implementation; its methods filename(),
49lineno(), fileline(), isfirstline(), isstdin(), nextfile() and close()
50correspond to the functions in the module. In addition it has a
51readline() method which returns the next input line, and a
52__getitem__() method which implements the sequence behavior. The
53sequence must be accessed in strictly sequential order; sequence
54access and readline() cannot be mixed.
55
56Optional in-place filtering: if the keyword argument inplace=1 is
57passed to input() or to the FileInput constructor, the file is moved
58to a backup file and standard output is directed to the input file.
59This makes it possible to write a filter that rewrites its input file
60in place. If the keyword argument backup=".<some extension>" is also
61given, it specifies the extension for the backup file, and the backup
62file remains around; by default, the extension is ".bak" and it is
63deleted when the output file is closed. In-place filtering is
64disabled when standard input is read. XXX The current implementation
65does not work for MS-DOS 8+3 filesystems.
66
Guido van Rossum47955242001-01-05 14:44:39 +000067Performance: this module is unfortunately one of the slower ways of
68processing large numbers of input lines. Nevertheless, a significant
69speed-up has been obtained by using readlines(bufsize) instead of
70readline(). A new keyword argument, bufsize=N, is present on the
71input() function and the FileInput() class to override the default
72buffer size.
73
Guido van Rossum7d5b99d1997-11-21 17:12:59 +000074XXX Possible additions:
75
76- optional getopt argument processing
Guido van Rossum7d5b99d1997-11-21 17:12:59 +000077- isatty()
78- read(), read(size), even readlines()
79
80"""
81
Walter Dörwald294bbf32002-06-06 09:48:13 +000082import sys, os
Guido van Rossum7d5b99d1997-11-21 17:12:59 +000083
Georg Brandlef0a8652009-05-17 12:22:57 +000084__all__ = ["input", "close", "nextfile", "filename", "lineno", "filelineno",
85 "isfirstline", "isstdin", "FileInput"]
Skip Montanaroeccd02a2001-01-20 23:34:12 +000086
Guido van Rossum7d5b99d1997-11-21 17:12:59 +000087_state = None
88
Guido van Rossum47955242001-01-05 14:44:39 +000089DEFAULT_BUFSIZE = 8*1024
90
Georg Brandlef0a8652009-05-17 12:22:57 +000091def input(files=None, inplace=False, backup="", bufsize=0,
Georg Brandlc98eeed2006-02-19 14:57:47 +000092 mode="r", openhook=None):
Georg Brandlef0a8652009-05-17 12:22:57 +000093 """input(files=None, inplace=False, backup="", bufsize=0, \
94mode="r", openhook=None)
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +000095
96 Create an instance of the FileInput class. The instance will be used
97 as global state for the functions of this module, and is also returned
98 to use during iteration. The parameters to this function will be passed
Tim Peters8ac14952002-05-23 15:15:30 +000099 along to the constructor of the FileInput class.
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000100 """
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000101 global _state
102 if _state and _state._file:
Collin Winterce36ad82007-08-30 01:19:48 +0000103 raise RuntimeError("input() already active")
Georg Brandlc98eeed2006-02-19 14:57:47 +0000104 _state = FileInput(files, inplace, backup, bufsize, mode, openhook)
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000105 return _state
106
107def close():
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000108 """Close the sequence."""
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000109 global _state
110 state = _state
111 _state = None
112 if state:
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000113 state.close()
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000114
115def nextfile():
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000116 """
117 Close the current file so that the next iteration will read the first
118 line from the next file (if any); lines not read from the file will
119 not count towards the cumulative line count. The filename is not
120 changed until after the first line of the next file has been read.
121 Before the first line has been read, this function has no effect;
122 it cannot be used to skip the first file. After the last line of the
Tim Peters8ac14952002-05-23 15:15:30 +0000123 last file has been read, this function has no effect.
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000124 """
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000125 if not _state:
Collin Winterce36ad82007-08-30 01:19:48 +0000126 raise RuntimeError("no active input()")
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000127 return _state.nextfile()
128
129def filename():
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000130 """
131 Return the name of the file currently being read.
Tim Peters8ac14952002-05-23 15:15:30 +0000132 Before the first line has been read, returns None.
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000133 """
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000134 if not _state:
Collin Winterce36ad82007-08-30 01:19:48 +0000135 raise RuntimeError("no active input()")
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000136 return _state.filename()
137
138def lineno():
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000139 """
140 Return the cumulative line number of the line that has just been read.
141 Before the first line has been read, returns 0. After the last line
Tim Peters8ac14952002-05-23 15:15:30 +0000142 of the last file has been read, returns the line number of that line.
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000143 """
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000144 if not _state:
Collin Winterce36ad82007-08-30 01:19:48 +0000145 raise RuntimeError("no active input()")
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000146 return _state.lineno()
147
148def filelineno():
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000149 """
150 Return the line number in the current file. Before the first line
151 has been read, returns 0. After the last line of the last file has
Tim Peters8ac14952002-05-23 15:15:30 +0000152 been read, returns the line number of that line within the file.
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000153 """
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000154 if not _state:
Collin Winterce36ad82007-08-30 01:19:48 +0000155 raise RuntimeError("no active input()")
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000156 return _state.filelineno()
157
Georg Brandl67e9fb92006-02-19 13:56:17 +0000158def fileno():
159 """
160 Return the file number of the current file. When no file is currently
161 opened, returns -1.
162 """
163 if not _state:
Collin Winterce36ad82007-08-30 01:19:48 +0000164 raise RuntimeError("no active input()")
Georg Brandl67e9fb92006-02-19 13:56:17 +0000165 return _state.fileno()
166
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000167def isfirstline():
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000168 """
169 Returns true the line just read is the first line of its file,
Tim Peters8ac14952002-05-23 15:15:30 +0000170 otherwise returns false.
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000171 """
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000172 if not _state:
Collin Winterce36ad82007-08-30 01:19:48 +0000173 raise RuntimeError("no active input()")
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000174 return _state.isfirstline()
175
176def isstdin():
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000177 """
178 Returns true if the last line was read from sys.stdin,
Tim Peters8ac14952002-05-23 15:15:30 +0000179 otherwise returns false.
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000180 """
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000181 if not _state:
Collin Winterce36ad82007-08-30 01:19:48 +0000182 raise RuntimeError("no active input()")
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000183 return _state.isstdin()
184
185class FileInput:
Georg Brandlc98eeed2006-02-19 14:57:47 +0000186 """class FileInput([files[, inplace[, backup[, mode[, openhook]]]]])
Tim Peters8ac14952002-05-23 15:15:30 +0000187
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000188 Class FileInput is the implementation of the module; its methods
Georg Brandl67e9fb92006-02-19 13:56:17 +0000189 filename(), lineno(), fileline(), isfirstline(), isstdin(), fileno(),
190 nextfile() and close() correspond to the functions of the same name
191 in the module.
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000192 In addition it has a readline() method which returns the next
193 input line, and a __getitem__() method which implements the
194 sequence behavior. The sequence must be accessed in strictly
Tim Peters8ac14952002-05-23 15:15:30 +0000195 sequential order; random access and readline() cannot be mixed.
Raymond Hettingerd1fa3db2002-05-15 02:56:03 +0000196 """
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000197
Georg Brandlef0a8652009-05-17 12:22:57 +0000198 def __init__(self, files=None, inplace=False, backup="", bufsize=0,
Georg Brandlc98eeed2006-02-19 14:57:47 +0000199 mode="r", openhook=None):
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000200 if isinstance(files, str):
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000201 files = (files,)
202 else:
Guido van Rossum2516b392000-04-10 17:16:12 +0000203 if files is None:
204 files = sys.argv[1:]
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000205 if not files:
Guido van Rossum2516b392000-04-10 17:16:12 +0000206 files = ('-',)
207 else:
208 files = tuple(files)
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000209 self._files = files
210 self._inplace = inplace
211 self._backup = backup
Guido van Rossum47955242001-01-05 14:44:39 +0000212 self._bufsize = bufsize or DEFAULT_BUFSIZE
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000213 self._savestdout = None
214 self._output = None
215 self._filename = None
216 self._lineno = 0
217 self._filelineno = 0
218 self._file = None
Guido van Rossum8ca162f2002-04-07 06:36:23 +0000219 self._isstdin = False
Guido van Rossum0aec9fb1998-07-20 15:49:28 +0000220 self._backupfilename = None
Guido van Rossum47955242001-01-05 14:44:39 +0000221 self._buffer = []
222 self._bufindex = 0
Georg Brandlc029f872006-02-19 14:12:34 +0000223 # restrict mode argument to reading modes
224 if mode not in ('r', 'rU', 'U', 'rb'):
225 raise ValueError("FileInput opening mode must be one of "
226 "'r', 'rU', 'U' and 'rb'")
227 self._mode = mode
Georg Brandlc98eeed2006-02-19 14:57:47 +0000228 if inplace and openhook:
229 raise ValueError("FileInput cannot use an opening hook in inplace mode")
Guido van Rossumd59da4b2007-05-22 18:11:13 +0000230 elif openhook and not hasattr(openhook, '__call__'):
Georg Brandlc98eeed2006-02-19 14:57:47 +0000231 raise ValueError("FileInput openhook must be callable")
232 self._openhook = openhook
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000233
234 def __del__(self):
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000235 self.close()
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000236
237 def close(self):
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000238 self.nextfile()
239 self._files = ()
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000240
Georg Brandl6cb7b652010-07-31 20:08:15 +0000241 def __enter__(self):
242 return self
243
244 def __exit__(self, type, value, traceback):
245 self.close()
246
Neil Schemenauer908632a2002-03-26 20:28:40 +0000247 def __iter__(self):
248 return self
249
Georg Brandla18af4e2007-04-21 15:47:16 +0000250 def __next__(self):
Guido van Rossum47955242001-01-05 14:44:39 +0000251 try:
252 line = self._buffer[self._bufindex]
253 except IndexError:
254 pass
255 else:
256 self._bufindex += 1
257 self._lineno += 1
258 self._filelineno += 1
259 return line
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000260 line = self.readline()
261 if not line:
Neil Schemenauer908632a2002-03-26 20:28:40 +0000262 raise StopIteration
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000263 return line
Tim Peters863ac442002-04-16 01:38:40 +0000264
Neil Schemenauer908632a2002-03-26 20:28:40 +0000265 def __getitem__(self, i):
266 if i != self._lineno:
Collin Winterce36ad82007-08-30 01:19:48 +0000267 raise RuntimeError("accessing lines out of order")
Neil Schemenauer908632a2002-03-26 20:28:40 +0000268 try:
Georg Brandla18af4e2007-04-21 15:47:16 +0000269 return self.__next__()
Neil Schemenauer908632a2002-03-26 20:28:40 +0000270 except StopIteration:
Collin Winterce36ad82007-08-30 01:19:48 +0000271 raise IndexError("end of input reached")
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000272
273 def nextfile(self):
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000274 savestdout = self._savestdout
275 self._savestdout = 0
276 if savestdout:
277 sys.stdout = savestdout
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000278
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000279 output = self._output
280 self._output = 0
281 if output:
282 output.close()
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000283
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000284 file = self._file
285 self._file = 0
286 if file and not self._isstdin:
287 file.close()
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000288
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000289 backupfilename = self._backupfilename
290 self._backupfilename = 0
291 if backupfilename and not self._backup:
292 try: os.unlink(backupfilename)
Skip Montanarocffac662002-08-14 02:58:16 +0000293 except OSError: pass
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000294
Guido van Rossum8ca162f2002-04-07 06:36:23 +0000295 self._isstdin = False
Guido van Rossum47955242001-01-05 14:44:39 +0000296 self._buffer = []
297 self._bufindex = 0
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000298
299 def readline(self):
Guido van Rossum47955242001-01-05 14:44:39 +0000300 try:
301 line = self._buffer[self._bufindex]
302 except IndexError:
303 pass
304 else:
305 self._bufindex += 1
306 self._lineno += 1
307 self._filelineno += 1
308 return line
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000309 if not self._file:
310 if not self._files:
311 return ""
312 self._filename = self._files[0]
313 self._files = self._files[1:]
314 self._filelineno = 0
315 self._file = None
Guido van Rossum8ca162f2002-04-07 06:36:23 +0000316 self._isstdin = False
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000317 self._backupfilename = 0
318 if self._filename == '-':
319 self._filename = '<stdin>'
320 self._file = sys.stdin
Guido van Rossum8ca162f2002-04-07 06:36:23 +0000321 self._isstdin = True
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000322 else:
323 if self._inplace:
324 self._backupfilename = (
Skip Montanaro7a98be22007-08-16 14:35:24 +0000325 self._filename + (self._backup or ".bak"))
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000326 try: os.unlink(self._backupfilename)
327 except os.error: pass
Guido van Rossumdcb85831999-10-18 21:41:43 +0000328 # The next few lines may raise IOError
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000329 os.rename(self._filename, self._backupfilename)
Georg Brandlc029f872006-02-19 14:12:34 +0000330 self._file = open(self._backupfilename, self._mode)
Guido van Rossumdcb85831999-10-18 21:41:43 +0000331 try:
Walter Dörwald294bbf32002-06-06 09:48:13 +0000332 perm = os.fstat(self._file.fileno()).st_mode
Skip Montanarocffac662002-08-14 02:58:16 +0000333 except OSError:
Guido van Rossumdcb85831999-10-18 21:41:43 +0000334 self._output = open(self._filename, "w")
335 else:
Guido van Rossum6203d8f2007-10-29 17:39:59 +0000336 mode = os.O_CREAT | os.O_WRONLY | os.O_TRUNC
337 if hasattr(os, 'O_BINARY'):
338 mode |= os.O_BINARY
339
340 fd = os.open(self._filename, mode, perm)
Guido van Rossumdcb85831999-10-18 21:41:43 +0000341 self._output = os.fdopen(fd, "w")
342 try:
Jack Jansen52941a82003-01-08 16:33:16 +0000343 if hasattr(os, 'chmod'):
344 os.chmod(self._filename, perm)
Skip Montanarocffac662002-08-14 02:58:16 +0000345 except OSError:
Guido van Rossumdcb85831999-10-18 21:41:43 +0000346 pass
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000347 self._savestdout = sys.stdout
348 sys.stdout = self._output
349 else:
350 # This may raise IOError
Georg Brandlc98eeed2006-02-19 14:57:47 +0000351 if self._openhook:
352 self._file = self._openhook(self._filename, self._mode)
353 else:
354 self._file = open(self._filename, self._mode)
Guido van Rossum47955242001-01-05 14:44:39 +0000355 self._buffer = self._file.readlines(self._bufsize)
356 self._bufindex = 0
357 if not self._buffer:
358 self.nextfile()
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000359 # Recursive call
360 return self.readline()
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000361
362 def filename(self):
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000363 return self._filename
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000364
365 def lineno(self):
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000366 return self._lineno
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000367
368 def filelineno(self):
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000369 return self._filelineno
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000370
Georg Brandl67e9fb92006-02-19 13:56:17 +0000371 def fileno(self):
372 if self._file:
373 try:
374 return self._file.fileno()
375 except ValueError:
376 return -1
377 else:
378 return -1
379
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000380 def isfirstline(self):
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000381 return self._filelineno == 1
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000382
383 def isstdin(self):
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000384 return self._isstdin
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000385
Georg Brandlc98eeed2006-02-19 14:57:47 +0000386
387def hook_compressed(filename, mode):
388 ext = os.path.splitext(filename)[1]
389 if ext == '.gz':
390 import gzip
391 return gzip.open(filename, mode)
392 elif ext == '.bz2':
393 import bz2
394 return bz2.BZ2File(filename, mode)
395 else:
396 return open(filename, mode)
397
398
399def hook_encoded(encoding):
400 import codecs
401 def openhook(filename, mode):
402 return codecs.open(filename, mode, encoding)
403 return openhook
404
405
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000406def _test():
407 import getopt
Georg Brandlef0a8652009-05-17 12:22:57 +0000408 inplace = False
409 backup = False
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000410 opts, args = getopt.getopt(sys.argv[1:], "ib:")
411 for o, a in opts:
Georg Brandlef0a8652009-05-17 12:22:57 +0000412 if o == '-i': inplace = True
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000413 if o == '-b': backup = a
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000414 for line in input(args, inplace=inplace, backup=backup):
Guido van Rossum45e2fbc1998-03-26 21:13:24 +0000415 if line[-1:] == '\n': line = line[:-1]
416 if line[-1:] == '\r': line = line[:-1]
Guido van Rossumbe19ed72007-02-09 05:37:30 +0000417 print("%d: %s[%d]%s %s" % (lineno(), filename(), filelineno(),
418 isfirstline() and "*" or "", line))
419 print("%d: %s[%d]" % (lineno(), filename(), filelineno()))
Guido van Rossum7d5b99d1997-11-21 17:12:59 +0000420
421if __name__ == '__main__':
422 _test()