blob: 63ca956a4e020143ff5acc574a9184606aaa14f7 [file] [log] [blame]
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001# subprocess - Subprocesses with accessible I/O streams
2#
Tim Peterse718f612004-10-12 21:51:32 +00003# For more information about this module, see PEP 324.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00004#
Peter Astrand3a708df2005-09-23 17:37:29 +00005# Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00006#
Peter Astrand69bf13f2005-02-14 08:56:32 +00007# Licensed to PSF under a Contributor Agreement.
Peter Astrand3a708df2005-09-23 17:37:29 +00008# See http://www.python.org/2.4/license for licensing details.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00009
Raymond Hettinger837dd932004-10-17 16:36:53 +000010r"""subprocess - Subprocesses with accessible I/O streams
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000011
Fredrik Lundh15aaacc2004-10-17 14:47:05 +000012This module allows you to spawn processes, connect to their
13input/output/error pipes, and obtain their return codes. This module
14intends to replace several other, older modules and functions, like:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000015
16os.system
17os.spawn*
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000018
19Information about how the subprocess module can be used to replace these
20modules and functions can be found below.
21
22
23
24Using the subprocess module
25===========================
26This module defines one class called Popen:
27
28class Popen(args, bufsize=0, executable=None,
29 stdin=None, stdout=None, stderr=None,
30 preexec_fn=None, close_fds=False, shell=False,
31 cwd=None, env=None, universal_newlines=False,
32 startupinfo=None, creationflags=0):
33
34
35Arguments are:
36
37args should be a string, or a sequence of program arguments. The
38program to execute is normally the first item in the args sequence or
39string, but can be explicitly set by using the executable argument.
40
41On UNIX, with shell=False (default): In this case, the Popen class
42uses os.execvp() to execute the child program. args should normally
43be a sequence. A string will be treated as a sequence with the string
44as the only item (the program to execute).
45
46On UNIX, with shell=True: If args is a string, it specifies the
47command string to execute through the shell. If args is a sequence,
48the first item specifies the command string, and any additional items
49will be treated as additional shell arguments.
50
51On Windows: the Popen class uses CreateProcess() to execute the child
52program, which operates on strings. If args is a sequence, it will be
53converted to a string using the list2cmdline method. Please note that
54not all MS Windows applications interpret the command line the same
55way: The list2cmdline is designed for applications using the same
56rules as the MS C runtime.
57
58bufsize, if given, has the same meaning as the corresponding argument
59to the built-in open() function: 0 means unbuffered, 1 means line
60buffered, any other positive value means use a buffer of
61(approximately) that size. A negative bufsize means to use the system
62default, which usually means fully buffered. The default value for
63bufsize is 0 (unbuffered).
64
65stdin, stdout and stderr specify the executed programs' standard
66input, standard output and standard error file handles, respectively.
67Valid values are PIPE, an existing file descriptor (a positive
68integer), an existing file object, and None. PIPE indicates that a
69new pipe to the child should be created. With None, no redirection
70will occur; the child's file handles will be inherited from the
71parent. Additionally, stderr can be STDOUT, which indicates that the
72stderr data from the applications should be captured into the same
73file handle as for stdout.
74
75If preexec_fn is set to a callable object, this object will be called
76in the child process just before the child is executed.
77
78If close_fds is true, all file descriptors except 0, 1 and 2 will be
79closed before the child process is executed.
80
81if shell is true, the specified command will be executed through the
82shell.
83
84If cwd is not None, the current directory will be changed to cwd
85before the child is executed.
86
87If env is not None, it defines the environment variables for the new
88process.
89
90If universal_newlines is true, the file objects stdout and stderr are
91opened as a text files, but lines may be terminated by any of '\n',
92the Unix end-of-line convention, '\r', the Macintosh convention or
93'\r\n', the Windows convention. All of these external representations
94are seen as '\n' by the Python program. Note: This feature is only
95available if Python is built with universal newline support (the
96default). Also, the newlines attribute of the file objects stdout,
97stdin and stderr are not updated by the communicate() method.
98
99The startupinfo and creationflags, if given, will be passed to the
100underlying CreateProcess() function. They can specify things such as
101appearance of the main window and priority for the new process.
102(Windows only)
103
104
Georg Brandlf9734072008-12-07 15:30:06 +0000105This module also defines some shortcut functions:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000106
Peter Astrand5f5e1412004-12-05 20:15:36 +0000107call(*popenargs, **kwargs):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000108 Run command with arguments. Wait for command to complete, then
109 return the returncode attribute.
110
111 The arguments are the same as for the Popen constructor. Example:
112
Florent Xicluna4886d242010-03-08 13:27:26 +0000113 >>> retcode = subprocess.call(["ls", "-l"])
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000114
Peter Astrand454f7672005-01-01 09:36:35 +0000115check_call(*popenargs, **kwargs):
116 Run command with arguments. Wait for command to complete. If the
117 exit code was zero then return, otherwise raise
118 CalledProcessError. The CalledProcessError object will have the
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000119 return code in the returncode attribute.
Peter Astrand454f7672005-01-01 09:36:35 +0000120
121 The arguments are the same as for the Popen constructor. Example:
122
Florent Xicluna4886d242010-03-08 13:27:26 +0000123 >>> subprocess.check_call(["ls", "-l"])
Georg Brandl2708f3a2009-12-20 14:38:23 +0000124 0
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000125
Brett Cannona23810f2008-05-26 19:04:21 +0000126getstatusoutput(cmd):
127 Return (status, output) of executing cmd in a shell.
128
129 Execute the string 'cmd' in a shell with os.popen() and return a 2-tuple
130 (status, output). cmd is actually run as '{ cmd ; } 2>&1', so that the
131 returned output will contain output or error messages. A trailing newline
132 is stripped from the output. The exit status for the command can be
133 interpreted according to the rules for the C function wait(). Example:
134
Brett Cannona23810f2008-05-26 19:04:21 +0000135 >>> subprocess.getstatusoutput('ls /bin/ls')
136 (0, '/bin/ls')
137 >>> subprocess.getstatusoutput('cat /bin/junk')
138 (256, 'cat: /bin/junk: No such file or directory')
139 >>> subprocess.getstatusoutput('/bin/junk')
140 (256, 'sh: /bin/junk: not found')
141
142getoutput(cmd):
143 Return output (stdout or stderr) of executing cmd in a shell.
144
145 Like getstatusoutput(), except the exit status is ignored and the return
146 value is a string containing the command's output. Example:
147
Brett Cannona23810f2008-05-26 19:04:21 +0000148 >>> subprocess.getoutput('ls /bin/ls')
149 '/bin/ls'
150
Georg Brandlf9734072008-12-07 15:30:06 +0000151check_output(*popenargs, **kwargs):
Georg Brandl2708f3a2009-12-20 14:38:23 +0000152 Run command with arguments and return its output as a byte string.
Georg Brandlf9734072008-12-07 15:30:06 +0000153
Georg Brandl2708f3a2009-12-20 14:38:23 +0000154 If the exit code was non-zero it raises a CalledProcessError. The
155 CalledProcessError object will have the return code in the returncode
156 attribute and output in the output attribute.
Georg Brandlf9734072008-12-07 15:30:06 +0000157
Georg Brandl2708f3a2009-12-20 14:38:23 +0000158 The arguments are the same as for the Popen constructor. Example:
Georg Brandlf9734072008-12-07 15:30:06 +0000159
Georg Brandl2708f3a2009-12-20 14:38:23 +0000160 >>> output = subprocess.check_output(["ls", "-l", "/dev/null"])
Georg Brandlf9734072008-12-07 15:30:06 +0000161
Brett Cannona23810f2008-05-26 19:04:21 +0000162
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000163Exceptions
164----------
165Exceptions raised in the child process, before the new program has
166started to execute, will be re-raised in the parent. Additionally,
167the exception object will have one extra attribute called
168'child_traceback', which is a string containing traceback information
169from the childs point of view.
170
171The most common exception raised is OSError. This occurs, for
172example, when trying to execute a non-existent file. Applications
173should prepare for OSErrors.
174
175A ValueError will be raised if Popen is called with invalid arguments.
176
Georg Brandlf9734072008-12-07 15:30:06 +0000177check_call() and check_output() will raise CalledProcessError, if the
178called process returns a non-zero return code.
Peter Astrand454f7672005-01-01 09:36:35 +0000179
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000180
181Security
182--------
183Unlike some other popen functions, this implementation will never call
184/bin/sh implicitly. This means that all characters, including shell
185metacharacters, can safely be passed to child processes.
186
187
188Popen objects
189=============
190Instances of the Popen class have the following methods:
191
192poll()
193 Check if child process has terminated. Returns returncode
194 attribute.
195
196wait()
197 Wait for child process to terminate. Returns returncode attribute.
198
199communicate(input=None)
200 Interact with process: Send data to stdin. Read data from stdout
201 and stderr, until end-of-file is reached. Wait for process to
Thomas Wouters902d6eb2007-01-09 23:18:33 +0000202 terminate. The optional input argument should be a string to be
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000203 sent to the child process, or None, if no data should be sent to
204 the child.
Tim Peterse718f612004-10-12 21:51:32 +0000205
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000206 communicate() returns a tuple (stdout, stderr).
207
208 Note: The data read is buffered in memory, so do not use this
209 method if the data size is large or unlimited.
210
211The following attributes are also available:
212
213stdin
214 If the stdin argument is PIPE, this attribute is a file object
215 that provides input to the child process. Otherwise, it is None.
216
217stdout
218 If the stdout argument is PIPE, this attribute is a file object
219 that provides output from the child process. Otherwise, it is
220 None.
221
222stderr
223 If the stderr argument is PIPE, this attribute is file object that
224 provides error output from the child process. Otherwise, it is
225 None.
226
227pid
228 The process ID of the child process.
229
230returncode
231 The child return code. A None value indicates that the process
232 hasn't terminated yet. A negative value -N indicates that the
233 child was terminated by signal N (UNIX only).
234
235
236Replacing older functions with the subprocess module
237====================================================
238In this section, "a ==> b" means that b can be used as a replacement
239for a.
240
241Note: All functions in this section fail (more or less) silently if
242the executed program cannot be found; this module raises an OSError
243exception.
244
245In the following examples, we assume that the subprocess module is
246imported with "from subprocess import *".
247
248
249Replacing /bin/sh shell backquote
250---------------------------------
251output=`mycmd myarg`
252==>
253output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
254
255
256Replacing shell pipe line
257-------------------------
258output=`dmesg | grep hda`
259==>
260p1 = Popen(["dmesg"], stdout=PIPE)
Peter Astrand6fdf3cb2004-11-30 18:06:42 +0000261p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000262output = p2.communicate()[0]
263
264
265Replacing os.system()
266---------------------
267sts = os.system("mycmd" + " myarg")
268==>
269p = Popen("mycmd" + " myarg", shell=True)
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000270pid, sts = os.waitpid(p.pid, 0)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000271
272Note:
273
274* Calling the program through the shell is usually not required.
275
276* It's easier to look at the returncode attribute than the
277 exitstatus.
278
279A more real-world example would look like this:
280
281try:
282 retcode = call("mycmd" + " myarg", shell=True)
283 if retcode < 0:
Guido van Rossumc2f93dc2007-05-24 00:50:02 +0000284 print("Child was terminated by signal", -retcode, file=sys.stderr)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000285 else:
Guido van Rossumc2f93dc2007-05-24 00:50:02 +0000286 print("Child returned", retcode, file=sys.stderr)
287except OSError as e:
288 print("Execution failed:", e, file=sys.stderr)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000289
290
291Replacing os.spawn*
292-------------------
Tim Peterse718f612004-10-12 21:51:32 +0000293P_NOWAIT example:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000294
295pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
296==>
297pid = Popen(["/bin/mycmd", "myarg"]).pid
298
299
300P_WAIT example:
301
302retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
303==>
304retcode = call(["/bin/mycmd", "myarg"])
305
306
Tim Peterse718f612004-10-12 21:51:32 +0000307Vector example:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000308
309os.spawnvp(os.P_NOWAIT, path, args)
310==>
311Popen([path] + args[1:])
312
313
Tim Peterse718f612004-10-12 21:51:32 +0000314Environment example:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000315
316os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
317==>
318Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000319"""
320
321import sys
322mswindows = (sys.platform == "win32")
323
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000324import io
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000325import os
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000326import traceback
Christian Heimesfdab48e2008-01-20 09:06:41 +0000327import gc
Christian Heimesa342c012008-04-20 21:01:16 +0000328import signal
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000329
Peter Astrand454f7672005-01-01 09:36:35 +0000330# Exception classes used by this module.
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000331class CalledProcessError(Exception):
Georg Brandlf9734072008-12-07 15:30:06 +0000332 """This exception is raised when a process run by check_call() or
333 check_output() returns a non-zero exit status.
334 The exit status will be stored in the returncode attribute;
335 check_output() will also store the output in the output attribute.
336 """
337 def __init__(self, returncode, cmd, output=None):
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000338 self.returncode = returncode
339 self.cmd = cmd
Georg Brandlf9734072008-12-07 15:30:06 +0000340 self.output = output
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000341 def __str__(self):
342 return "Command '%s' returned non-zero exit status %d" % (self.cmd, self.returncode)
343
Peter Astrand454f7672005-01-01 09:36:35 +0000344
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000345if mswindows:
346 import threading
347 import msvcrt
Fredrik Lundh3e73a012004-10-13 18:19:18 +0000348 if 0: # <-- change this to use pywin32 instead of the _subprocess driver
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000349 import pywintypes
Tim Peterse8374a52004-10-13 03:15:00 +0000350 from win32api import GetStdHandle, STD_INPUT_HANDLE, \
351 STD_OUTPUT_HANDLE, STD_ERROR_HANDLE
352 from win32api import GetCurrentProcess, DuplicateHandle, \
353 GetModuleFileName, GetVersion
Peter Astrandc1d65362004-11-07 14:30:34 +0000354 from win32con import DUPLICATE_SAME_ACCESS, SW_HIDE
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000355 from win32pipe import CreatePipe
Tim Peterse8374a52004-10-13 03:15:00 +0000356 from win32process import CreateProcess, STARTUPINFO, \
357 GetExitCodeProcess, STARTF_USESTDHANDLES, \
Peter Astrandc1d65362004-11-07 14:30:34 +0000358 STARTF_USESHOWWINDOW, CREATE_NEW_CONSOLE
Christian Heimesa342c012008-04-20 21:01:16 +0000359 from win32process import TerminateProcess
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000360 from win32event import WaitForSingleObject, INFINITE, WAIT_OBJECT_0
Fredrik Lundh3e73a012004-10-13 18:19:18 +0000361 else:
362 from _subprocess import *
363 class STARTUPINFO:
364 dwFlags = 0
365 hStdInput = None
366 hStdOutput = None
367 hStdError = None
Thomas Wouters73e5a5b2006-06-08 15:35:45 +0000368 wShowWindow = 0
Fredrik Lundh3e73a012004-10-13 18:19:18 +0000369 class pywintypes:
370 error = IOError
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000371else:
372 import select
Gregory P. Smithd06fa472009-07-04 02:46:54 +0000373 _has_poll = hasattr(select, 'poll')
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000374 import errno
375 import fcntl
376 import pickle
377
Amaury Forgeot d'Arcace31022009-07-09 22:44:11 +0000378 # When select or poll has indicated that the file is writable,
379 # we can write up to _PIPE_BUF bytes without risk of blocking.
380 # POSIX defines PIPE_BUF as >= 512.
381 _PIPE_BUF = getattr(select, 'PIPE_BUF', 512)
382
383
Brett Cannona23810f2008-05-26 19:04:21 +0000384__all__ = ["Popen", "PIPE", "STDOUT", "call", "check_call", "getstatusoutput",
Georg Brandlf9734072008-12-07 15:30:06 +0000385 "getoutput", "check_output", "CalledProcessError"]
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000386
387try:
388 MAXFD = os.sysconf("SC_OPEN_MAX")
389except:
390 MAXFD = 256
391
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000392_active = []
393
394def _cleanup():
395 for inst in _active[:]:
Georg Brandl6aa2d1f2008-08-12 08:35:52 +0000396 res = inst._internal_poll(_deadstate=sys.maxsize)
Guido van Rossumb5d47ef2006-08-24 02:27:45 +0000397 if res is not None and res >= 0:
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000398 try:
399 _active.remove(inst)
400 except ValueError:
401 # This can happen if two threads create a new Popen instance.
402 # It's harmless that it was already removed, so ignore.
403 pass
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000404
405PIPE = -1
406STDOUT = -2
407
408
Gregory P. Smitha59c59f2010-03-01 00:17:40 +0000409def _eintr_retry_call(func, *args):
410 while True:
411 try:
412 return func(*args)
413 except OSError as e:
414 if e.errno == errno.EINTR:
415 continue
416 raise
417
418
Peter Astrand5f5e1412004-12-05 20:15:36 +0000419def call(*popenargs, **kwargs):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000420 """Run command with arguments. Wait for command to complete, then
421 return the returncode attribute.
422
423 The arguments are the same as for the Popen constructor. Example:
424
425 retcode = call(["ls", "-l"])
426 """
Peter Astrand5f5e1412004-12-05 20:15:36 +0000427 return Popen(*popenargs, **kwargs).wait()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000428
429
Peter Astrand454f7672005-01-01 09:36:35 +0000430def check_call(*popenargs, **kwargs):
431 """Run command with arguments. Wait for command to complete. If
432 the exit code was zero then return, otherwise raise
433 CalledProcessError. The CalledProcessError object will have the
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000434 return code in the returncode attribute.
Peter Astrand454f7672005-01-01 09:36:35 +0000435
436 The arguments are the same as for the Popen constructor. Example:
437
438 check_call(["ls", "-l"])
439 """
440 retcode = call(*popenargs, **kwargs)
Peter Astrand454f7672005-01-01 09:36:35 +0000441 if retcode:
Georg Brandlf9734072008-12-07 15:30:06 +0000442 cmd = kwargs.get("args")
443 if cmd is None:
444 cmd = popenargs[0]
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000445 raise CalledProcessError(retcode, cmd)
Georg Brandlf9734072008-12-07 15:30:06 +0000446 return 0
447
448
449def check_output(*popenargs, **kwargs):
Georg Brandl2708f3a2009-12-20 14:38:23 +0000450 r"""Run command with arguments and return its output as a byte string.
Georg Brandlf9734072008-12-07 15:30:06 +0000451
452 If the exit code was non-zero it raises a CalledProcessError. The
453 CalledProcessError object will have the return code in the returncode
454 attribute and output in the output attribute.
455
456 The arguments are the same as for the Popen constructor. Example:
457
458 >>> check_output(["ls", "-l", "/dev/null"])
Georg Brandl2708f3a2009-12-20 14:38:23 +0000459 b'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n'
Georg Brandlf9734072008-12-07 15:30:06 +0000460
461 The stdout argument is not allowed as it is used internally.
Georg Brandl127d4702009-12-28 08:10:38 +0000462 To capture standard error in the result, use stderr=STDOUT.
Georg Brandlf9734072008-12-07 15:30:06 +0000463
464 >>> check_output(["/bin/sh", "-c",
Georg Brandl2708f3a2009-12-20 14:38:23 +0000465 ... "ls -l non_existent_file ; exit 0"],
Georg Brandl127d4702009-12-28 08:10:38 +0000466 ... stderr=STDOUT)
Georg Brandl2708f3a2009-12-20 14:38:23 +0000467 b'ls: non_existent_file: No such file or directory\n'
Georg Brandlf9734072008-12-07 15:30:06 +0000468 """
469 if 'stdout' in kwargs:
470 raise ValueError('stdout argument not allowed, it will be overridden.')
471 process = Popen(*popenargs, stdout=PIPE, **kwargs)
472 output, unused_err = process.communicate()
473 retcode = process.poll()
474 if retcode:
475 cmd = kwargs.get("args")
476 if cmd is None:
477 cmd = popenargs[0]
478 raise CalledProcessError(retcode, cmd, output=output)
479 return output
Peter Astrand454f7672005-01-01 09:36:35 +0000480
481
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000482def list2cmdline(seq):
483 """
484 Translate a sequence of arguments into a command line
485 string, using the same rules as the MS C runtime:
486
487 1) Arguments are delimited by white space, which is either a
488 space or a tab.
489
490 2) A string surrounded by double quotation marks is
491 interpreted as a single argument, regardless of white space
Christian Heimesfdab48e2008-01-20 09:06:41 +0000492 or pipe characters contained within. A quoted string can be
493 embedded in an argument.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000494
495 3) A double quotation mark preceded by a backslash is
496 interpreted as a literal double quotation mark.
497
498 4) Backslashes are interpreted literally, unless they
499 immediately precede a double quotation mark.
500
501 5) If backslashes immediately precede a double quotation mark,
502 every pair of backslashes is interpreted as a literal
503 backslash. If the number of backslashes is odd, the last
504 backslash escapes the next double quotation mark as
505 described in rule 3.
506 """
507
508 # See
Eric Smith3c573af2009-11-09 15:23:15 +0000509 # http://msdn.microsoft.com/en-us/library/17w5ykft.aspx
510 # or search http://msdn.microsoft.com for
511 # "Parsing C++ Command-Line Arguments"
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000512 result = []
513 needquote = False
514 for arg in seq:
515 bs_buf = []
516
517 # Add a space to separate this argument from the others
518 if result:
519 result.append(' ')
520
Christian Heimesfdab48e2008-01-20 09:06:41 +0000521 needquote = (" " in arg) or ("\t" in arg) or ("|" in arg) or not arg
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000522 if needquote:
523 result.append('"')
524
525 for c in arg:
526 if c == '\\':
Tim Peterse718f612004-10-12 21:51:32 +0000527 # Don't know if we need to double yet.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000528 bs_buf.append(c)
529 elif c == '"':
Christian Heimesfdab48e2008-01-20 09:06:41 +0000530 # Double backslashes.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000531 result.append('\\' * len(bs_buf)*2)
532 bs_buf = []
533 result.append('\\"')
534 else:
535 # Normal char
536 if bs_buf:
537 result.extend(bs_buf)
538 bs_buf = []
539 result.append(c)
540
Christian Heimesfdab48e2008-01-20 09:06:41 +0000541 # Add remaining backslashes, if any.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000542 if bs_buf:
543 result.extend(bs_buf)
544
545 if needquote:
Peter Astrand7e78ade2005-03-03 21:10:23 +0000546 result.extend(bs_buf)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000547 result.append('"')
548
549 return ''.join(result)
550
551
Brett Cannona23810f2008-05-26 19:04:21 +0000552# Various tools for executing commands and looking at their output and status.
553#
554# NB This only works (and is only relevant) for UNIX.
555
556def getstatusoutput(cmd):
557 """Return (status, output) of executing cmd in a shell.
558
559 Execute the string 'cmd' in a shell with os.popen() and return a 2-tuple
560 (status, output). cmd is actually run as '{ cmd ; } 2>&1', so that the
561 returned output will contain output or error messages. A trailing newline
562 is stripped from the output. The exit status for the command can be
563 interpreted according to the rules for the C function wait(). Example:
564
565 >>> import subprocess
566 >>> subprocess.getstatusoutput('ls /bin/ls')
567 (0, '/bin/ls')
568 >>> subprocess.getstatusoutput('cat /bin/junk')
569 (256, 'cat: /bin/junk: No such file or directory')
570 >>> subprocess.getstatusoutput('/bin/junk')
571 (256, 'sh: /bin/junk: not found')
572 """
573 pipe = os.popen('{ ' + cmd + '; } 2>&1', 'r')
574 text = pipe.read()
575 sts = pipe.close()
576 if sts is None: sts = 0
577 if text[-1:] == '\n': text = text[:-1]
578 return sts, text
579
580
581def getoutput(cmd):
582 """Return output (stdout or stderr) of executing cmd in a shell.
583
584 Like getstatusoutput(), except the exit status is ignored and the return
585 value is a string containing the command's output. Example:
586
587 >>> import subprocess
588 >>> subprocess.getoutput('ls /bin/ls')
589 '/bin/ls'
590 """
591 return getstatusoutput(cmd)[1]
592
593
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000594class Popen(object):
595 def __init__(self, args, bufsize=0, executable=None,
596 stdin=None, stdout=None, stderr=None,
597 preexec_fn=None, close_fds=False, shell=False,
598 cwd=None, env=None, universal_newlines=False,
599 startupinfo=None, creationflags=0):
600 """Create new Popen instance."""
601 _cleanup()
602
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000603 self._child_created = False
Guido van Rossum46a05a72007-06-07 21:56:45 +0000604 if bufsize is None:
605 bufsize = 0 # Restore default
Walter Dörwaldaa97f042007-05-03 21:05:51 +0000606 if not isinstance(bufsize, int):
Peter Astrand738131d2004-11-30 21:04:45 +0000607 raise TypeError("bufsize must be an integer")
608
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000609 if mswindows:
Tim Peterse8374a52004-10-13 03:15:00 +0000610 if preexec_fn is not None:
611 raise ValueError("preexec_fn is not supported on Windows "
612 "platforms")
Guido van Rossume7ba4952007-06-06 23:52:48 +0000613 if close_fds and (stdin is not None or stdout is not None or
614 stderr is not None):
Tim Peterse8374a52004-10-13 03:15:00 +0000615 raise ValueError("close_fds is not supported on Windows "
Guido van Rossume7ba4952007-06-06 23:52:48 +0000616 "platforms if you redirect stdin/stdout/stderr")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000617 else:
618 # POSIX
Tim Peterse8374a52004-10-13 03:15:00 +0000619 if startupinfo is not None:
620 raise ValueError("startupinfo is only supported on Windows "
621 "platforms")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000622 if creationflags != 0:
Tim Peterse8374a52004-10-13 03:15:00 +0000623 raise ValueError("creationflags is only supported on Windows "
624 "platforms")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000625
Tim Peterse718f612004-10-12 21:51:32 +0000626 self.stdin = None
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000627 self.stdout = None
628 self.stderr = None
629 self.pid = None
630 self.returncode = None
631 self.universal_newlines = universal_newlines
632
633 # Input and output objects. The general principle is like
634 # this:
635 #
636 # Parent Child
637 # ------ -----
638 # p2cwrite ---stdin---> p2cread
639 # c2pread <--stdout--- c2pwrite
640 # errread <--stderr--- errwrite
641 #
642 # On POSIX, the child objects are file descriptors. On
643 # Windows, these are Windows file handles. The parent objects
644 # are file descriptors on both platforms. The parent objects
645 # are None when not using PIPEs. The child objects are None
646 # when not redirecting.
Tim Peterse718f612004-10-12 21:51:32 +0000647
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000648 (p2cread, p2cwrite,
649 c2pread, c2pwrite,
650 errread, errwrite) = self._get_handles(stdin, stdout, stderr)
651
652 self._execute_child(args, executable, preexec_fn, close_fds,
653 cwd, env, universal_newlines,
654 startupinfo, creationflags, shell,
655 p2cread, p2cwrite,
656 c2pread, c2pwrite,
657 errread, errwrite)
658
Thomas Wouterscf297e42007-02-23 15:07:44 +0000659 if mswindows:
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000660 if p2cwrite is not None:
661 p2cwrite = msvcrt.open_osfhandle(p2cwrite.Detach(), 0)
662 if c2pread is not None:
663 c2pread = msvcrt.open_osfhandle(c2pread.Detach(), 0)
664 if errread is not None:
665 errread = msvcrt.open_osfhandle(errread.Detach(), 0)
Thomas Wouterscf297e42007-02-23 15:07:44 +0000666
Guido van Rossumc2f93dc2007-05-24 00:50:02 +0000667 if bufsize == 0:
668 bufsize = 1 # Nearly unbuffered (XXX for now)
Thomas Wouterscf297e42007-02-23 15:07:44 +0000669 if p2cwrite is not None:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000670 self.stdin = io.open(p2cwrite, 'wb', bufsize)
671 if self.universal_newlines:
672 self.stdin = io.TextIOWrapper(self.stdin)
Thomas Wouterscf297e42007-02-23 15:07:44 +0000673 if c2pread is not None:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000674 self.stdout = io.open(c2pread, 'rb', bufsize)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000675 if universal_newlines:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000676 self.stdout = io.TextIOWrapper(self.stdout)
Thomas Wouterscf297e42007-02-23 15:07:44 +0000677 if errread is not None:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000678 self.stderr = io.open(errread, 'rb', bufsize)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000679 if universal_newlines:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000680 self.stderr = io.TextIOWrapper(self.stderr)
Tim Peterse718f612004-10-12 21:51:32 +0000681
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000682
Guido van Rossum98297ee2007-11-06 21:34:58 +0000683 def _translate_newlines(self, data, encoding):
684 data = data.replace(b"\r\n", b"\n").replace(b"\r", b"\n")
685 return data.decode(encoding)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000686
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000687
Guido van Rossumd8faa362007-04-27 19:54:29 +0000688 def __del__(self, sys=sys):
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000689 if not self._child_created:
690 # We didn't get to successfully create a child process.
691 return
692 # In case the child hasn't been waited on, check if it's done.
Georg Brandl6aa2d1f2008-08-12 08:35:52 +0000693 self._internal_poll(_deadstate=sys.maxsize)
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000694 if self.returncode is None and _active is not None:
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000695 # Child is still running, keep us alive until we can wait on it.
696 _active.append(self)
697
698
Peter Astrand23109f02005-03-03 20:28:59 +0000699 def communicate(self, input=None):
700 """Interact with process: Send data to stdin. Read data from
701 stdout and stderr, until end-of-file is reached. Wait for
702 process to terminate. The optional input argument should be a
703 string to be sent to the child process, or None, if no data
704 should be sent to the child.
Tim Peterseba28be2005-03-28 01:08:02 +0000705
Peter Astrand23109f02005-03-03 20:28:59 +0000706 communicate() returns a tuple (stdout, stderr)."""
707
708 # Optimization: If we are only using one pipe, or no pipe at
709 # all, using select() or threads is unnecessary.
710 if [self.stdin, self.stdout, self.stderr].count(None) >= 2:
Tim Peterseba28be2005-03-28 01:08:02 +0000711 stdout = None
712 stderr = None
Peter Astrand23109f02005-03-03 20:28:59 +0000713 if self.stdin:
714 if input:
715 self.stdin.write(input)
716 self.stdin.close()
717 elif self.stdout:
718 stdout = self.stdout.read()
Georg Brandlf08a9dd2008-06-10 16:57:31 +0000719 self.stdout.close()
Peter Astrand23109f02005-03-03 20:28:59 +0000720 elif self.stderr:
721 stderr = self.stderr.read()
Georg Brandlf08a9dd2008-06-10 16:57:31 +0000722 self.stderr.close()
Peter Astrand23109f02005-03-03 20:28:59 +0000723 self.wait()
724 return (stdout, stderr)
Tim Peterseba28be2005-03-28 01:08:02 +0000725
Peter Astrand23109f02005-03-03 20:28:59 +0000726 return self._communicate(input)
727
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000728
Georg Brandl6aa2d1f2008-08-12 08:35:52 +0000729 def poll(self):
730 return self._internal_poll()
731
732
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000733 if mswindows:
734 #
735 # Windows methods
736 #
737 def _get_handles(self, stdin, stdout, stderr):
Alexandre Vassalotti711ed4a2009-07-17 10:42:05 +0000738 """Construct and return tuple with IO objects:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000739 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
740 """
Peter Astrandd38ddf42005-02-10 08:32:50 +0000741 if stdin is None and stdout is None and stderr is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000742 return (None, None, None, None, None, None)
Tim Peterse718f612004-10-12 21:51:32 +0000743
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000744 p2cread, p2cwrite = None, None
745 c2pread, c2pwrite = None, None
746 errread, errwrite = None, None
747
Peter Astrandd38ddf42005-02-10 08:32:50 +0000748 if stdin is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000749 p2cread = GetStdHandle(STD_INPUT_HANDLE)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000750 if p2cread is None:
751 p2cread, _ = CreatePipe(None, 0)
752 elif stdin == PIPE:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000753 p2cread, p2cwrite = CreatePipe(None, 0)
Peter Astrandd38ddf42005-02-10 08:32:50 +0000754 elif isinstance(stdin, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000755 p2cread = msvcrt.get_osfhandle(stdin)
756 else:
757 # Assuming file-like object
758 p2cread = msvcrt.get_osfhandle(stdin.fileno())
759 p2cread = self._make_inheritable(p2cread)
760
Peter Astrandd38ddf42005-02-10 08:32:50 +0000761 if stdout is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000762 c2pwrite = GetStdHandle(STD_OUTPUT_HANDLE)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000763 if c2pwrite is None:
764 _, c2pwrite = CreatePipe(None, 0)
765 elif stdout == PIPE:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000766 c2pread, c2pwrite = CreatePipe(None, 0)
Peter Astrandd38ddf42005-02-10 08:32:50 +0000767 elif isinstance(stdout, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000768 c2pwrite = msvcrt.get_osfhandle(stdout)
769 else:
770 # Assuming file-like object
771 c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
772 c2pwrite = self._make_inheritable(c2pwrite)
773
Peter Astrandd38ddf42005-02-10 08:32:50 +0000774 if stderr is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000775 errwrite = GetStdHandle(STD_ERROR_HANDLE)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000776 if errwrite is None:
777 _, errwrite = CreatePipe(None, 0)
778 elif stderr == PIPE:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000779 errread, errwrite = CreatePipe(None, 0)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000780 elif stderr == STDOUT:
781 errwrite = c2pwrite
Peter Astrandd38ddf42005-02-10 08:32:50 +0000782 elif isinstance(stderr, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000783 errwrite = msvcrt.get_osfhandle(stderr)
784 else:
785 # Assuming file-like object
786 errwrite = msvcrt.get_osfhandle(stderr.fileno())
787 errwrite = self._make_inheritable(errwrite)
788
789 return (p2cread, p2cwrite,
790 c2pread, c2pwrite,
791 errread, errwrite)
792
793
794 def _make_inheritable(self, handle):
795 """Return a duplicate of handle, which is inheritable"""
796 return DuplicateHandle(GetCurrentProcess(), handle,
797 GetCurrentProcess(), 0, 1,
798 DUPLICATE_SAME_ACCESS)
799
800
801 def _find_w9xpopen(self):
802 """Find and return absolut path to w9xpopen.exe"""
Tim Peterse8374a52004-10-13 03:15:00 +0000803 w9xpopen = os.path.join(os.path.dirname(GetModuleFileName(0)),
804 "w9xpopen.exe")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000805 if not os.path.exists(w9xpopen):
806 # Eeek - file-not-found - possibly an embedding
807 # situation - see if we can locate it in sys.exec_prefix
Tim Peterse8374a52004-10-13 03:15:00 +0000808 w9xpopen = os.path.join(os.path.dirname(sys.exec_prefix),
809 "w9xpopen.exe")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000810 if not os.path.exists(w9xpopen):
Tim Peterse8374a52004-10-13 03:15:00 +0000811 raise RuntimeError("Cannot locate w9xpopen.exe, which is "
812 "needed for Popen to work with your "
813 "shell or platform.")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000814 return w9xpopen
815
Tim Peterse718f612004-10-12 21:51:32 +0000816
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000817 def _execute_child(self, args, executable, preexec_fn, close_fds,
818 cwd, env, universal_newlines,
819 startupinfo, creationflags, shell,
820 p2cread, p2cwrite,
821 c2pread, c2pwrite,
822 errread, errwrite):
823 """Execute program (MS Windows version)"""
824
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000825 if not isinstance(args, str):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000826 args = list2cmdline(args)
827
Peter Astrandc1d65362004-11-07 14:30:34 +0000828 # Process startup details
Peter Astrandd38ddf42005-02-10 08:32:50 +0000829 if startupinfo is None:
Thomas Wouters73e5a5b2006-06-08 15:35:45 +0000830 startupinfo = STARTUPINFO()
831 if None not in (p2cread, c2pwrite, errwrite):
Peter Astrandc1d65362004-11-07 14:30:34 +0000832 startupinfo.dwFlags |= STARTF_USESTDHANDLES
833 startupinfo.hStdInput = p2cread
834 startupinfo.hStdOutput = c2pwrite
835 startupinfo.hStdError = errwrite
836
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000837 if shell:
Thomas Wouters73e5a5b2006-06-08 15:35:45 +0000838 startupinfo.dwFlags |= STARTF_USESHOWWINDOW
839 startupinfo.wShowWindow = SW_HIDE
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000840 comspec = os.environ.get("COMSPEC", "cmd.exe")
841 args = comspec + " /c " + args
Guido van Rossume2a383d2007-01-15 16:59:06 +0000842 if (GetVersion() >= 0x80000000 or
Tim Peterse8374a52004-10-13 03:15:00 +0000843 os.path.basename(comspec).lower() == "command.com"):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000844 # Win9x, or using command.com on NT. We need to
845 # use the w9xpopen intermediate program. For more
846 # information, see KB Q150956
847 # (http://web.archive.org/web/20011105084002/http://support.microsoft.com/support/kb/articles/Q150/9/56.asp)
848 w9xpopen = self._find_w9xpopen()
849 args = '"%s" %s' % (w9xpopen, args)
850 # Not passing CREATE_NEW_CONSOLE has been known to
851 # cause random failures on win9x. Specifically a
852 # dialog: "Your program accessed mem currently in
853 # use at xxx" and a hopeful warning about the
Mark Dickinson934896d2009-02-21 20:59:32 +0000854 # stability of your system. Cost is Ctrl+C won't
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000855 # kill children.
856 creationflags |= CREATE_NEW_CONSOLE
857
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000858 # Start the process
859 try:
860 hp, ht, pid, tid = CreateProcess(executable, args,
Tim Peterse8374a52004-10-13 03:15:00 +0000861 # no special security
862 None, None,
Guido van Rossume7ba4952007-06-06 23:52:48 +0000863 int(not close_fds),
Tim Peterse8374a52004-10-13 03:15:00 +0000864 creationflags,
865 env,
866 cwd,
867 startupinfo)
Guido van Rossumb940e112007-01-10 16:19:56 +0000868 except pywintypes.error as e:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000869 # Translate pywintypes.error to WindowsError, which is
870 # a subclass of OSError. FIXME: We should really
871 # translate errno using _sys_errlist (or simliar), but
872 # how can this be done from Python?
873 raise WindowsError(*e.args)
874
875 # Retain the process handle, but close the thread handle
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000876 self._child_created = True
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000877 self._handle = hp
878 self.pid = pid
879 ht.Close()
880
Andrew M. Kuchling51ee66e2004-10-12 16:38:42 +0000881 # Child is launched. Close the parent's copy of those pipe
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000882 # handles that only the child should have open. You need
883 # to make sure that no handles to the write end of the
884 # output pipe are maintained in this process or else the
885 # pipe will not close when the child process exits and the
886 # ReadFile will hang.
Peter Astrandd38ddf42005-02-10 08:32:50 +0000887 if p2cread is not None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000888 p2cread.Close()
Peter Astrandd38ddf42005-02-10 08:32:50 +0000889 if c2pwrite is not None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000890 c2pwrite.Close()
Peter Astrandd38ddf42005-02-10 08:32:50 +0000891 if errwrite is not None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000892 errwrite.Close()
893
Tim Peterse718f612004-10-12 21:51:32 +0000894
Georg Brandl6aa2d1f2008-08-12 08:35:52 +0000895 def _internal_poll(self, _deadstate=None):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000896 """Check if child process has terminated. Returns returncode
897 attribute."""
Peter Astrandd38ddf42005-02-10 08:32:50 +0000898 if self.returncode is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000899 if WaitForSingleObject(self._handle, 0) == WAIT_OBJECT_0:
900 self.returncode = GetExitCodeProcess(self._handle)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000901 return self.returncode
902
903
904 def wait(self):
905 """Wait for child process to terminate. Returns returncode
906 attribute."""
Peter Astrandd38ddf42005-02-10 08:32:50 +0000907 if self.returncode is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000908 obj = WaitForSingleObject(self._handle, INFINITE)
909 self.returncode = GetExitCodeProcess(self._handle)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000910 return self.returncode
911
912
913 def _readerthread(self, fh, buffer):
914 buffer.append(fh.read())
915
916
Peter Astrand23109f02005-03-03 20:28:59 +0000917 def _communicate(self, input):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000918 stdout = None # Return
919 stderr = None # Return
920
921 if self.stdout:
922 stdout = []
Tim Peterse8374a52004-10-13 03:15:00 +0000923 stdout_thread = threading.Thread(target=self._readerthread,
924 args=(self.stdout, stdout))
Benjamin Peterson632e0362008-08-18 19:08:51 +0000925 stdout_thread.daemon = True
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000926 stdout_thread.start()
927 if self.stderr:
928 stderr = []
Tim Peterse8374a52004-10-13 03:15:00 +0000929 stderr_thread = threading.Thread(target=self._readerthread,
930 args=(self.stderr, stderr))
Benjamin Peterson632e0362008-08-18 19:08:51 +0000931 stderr_thread.daemon = True
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000932 stderr_thread.start()
933
934 if self.stdin:
Peter Astrandd38ddf42005-02-10 08:32:50 +0000935 if input is not None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000936 self.stdin.write(input)
937 self.stdin.close()
938
939 if self.stdout:
940 stdout_thread.join()
941 if self.stderr:
942 stderr_thread.join()
943
944 # All data exchanged. Translate lists into strings.
Peter Astrandd38ddf42005-02-10 08:32:50 +0000945 if stdout is not None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000946 stdout = stdout[0]
Peter Astrandd38ddf42005-02-10 08:32:50 +0000947 if stderr is not None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000948 stderr = stderr[0]
949
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000950 self.wait()
951 return (stdout, stderr)
952
Christian Heimesa342c012008-04-20 21:01:16 +0000953 def send_signal(self, sig):
954 """Send a signal to the process
955 """
956 if sig == signal.SIGTERM:
957 self.terminate()
958 else:
959 raise ValueError("Only SIGTERM is supported on Windows")
960
961 def terminate(self):
962 """Terminates the process
963 """
964 TerminateProcess(self._handle, 1)
965
966 kill = terminate
967
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000968 else:
969 #
970 # POSIX methods
971 #
972 def _get_handles(self, stdin, stdout, stderr):
Alexandre Vassalotti711ed4a2009-07-17 10:42:05 +0000973 """Construct and return tuple with IO objects:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000974 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
975 """
976 p2cread, p2cwrite = None, None
977 c2pread, c2pwrite = None, None
978 errread, errwrite = None, None
979
Peter Astrandd38ddf42005-02-10 08:32:50 +0000980 if stdin is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000981 pass
982 elif stdin == PIPE:
983 p2cread, p2cwrite = os.pipe()
Peter Astrandd38ddf42005-02-10 08:32:50 +0000984 elif isinstance(stdin, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000985 p2cread = stdin
986 else:
987 # Assuming file-like object
988 p2cread = stdin.fileno()
989
Peter Astrandd38ddf42005-02-10 08:32:50 +0000990 if stdout is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000991 pass
992 elif stdout == PIPE:
993 c2pread, c2pwrite = os.pipe()
Peter Astrandd38ddf42005-02-10 08:32:50 +0000994 elif isinstance(stdout, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000995 c2pwrite = stdout
996 else:
997 # Assuming file-like object
Tim Peterse718f612004-10-12 21:51:32 +0000998 c2pwrite = stdout.fileno()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000999
Peter Astrandd38ddf42005-02-10 08:32:50 +00001000 if stderr is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001001 pass
1002 elif stderr == PIPE:
1003 errread, errwrite = os.pipe()
1004 elif stderr == STDOUT:
1005 errwrite = c2pwrite
Peter Astrandd38ddf42005-02-10 08:32:50 +00001006 elif isinstance(stderr, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001007 errwrite = stderr
1008 else:
1009 # Assuming file-like object
1010 errwrite = stderr.fileno()
1011
1012 return (p2cread, p2cwrite,
1013 c2pread, c2pwrite,
1014 errread, errwrite)
1015
1016
1017 def _set_cloexec_flag(self, fd):
1018 try:
1019 cloexec_flag = fcntl.FD_CLOEXEC
1020 except AttributeError:
1021 cloexec_flag = 1
1022
1023 old = fcntl.fcntl(fd, fcntl.F_GETFD)
1024 fcntl.fcntl(fd, fcntl.F_SETFD, old | cloexec_flag)
1025
1026
1027 def _close_fds(self, but):
Christian Heimesfdab48e2008-01-20 09:06:41 +00001028 os.closerange(3, but)
1029 os.closerange(but + 1, MAXFD)
Tim Peterse718f612004-10-12 21:51:32 +00001030
1031
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001032 def _execute_child(self, args, executable, preexec_fn, close_fds,
1033 cwd, env, universal_newlines,
1034 startupinfo, creationflags, shell,
1035 p2cread, p2cwrite,
1036 c2pread, c2pwrite,
1037 errread, errwrite):
1038 """Execute program (POSIX version)"""
1039
Guido van Rossum3172c5d2007-10-16 18:12:55 +00001040 if isinstance(args, str):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001041 args = [args]
Thomas Wouters89f507f2006-12-13 04:49:30 +00001042 else:
1043 args = list(args)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001044
1045 if shell:
1046 args = ["/bin/sh", "-c"] + args
1047
Peter Astrandd38ddf42005-02-10 08:32:50 +00001048 if executable is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001049 executable = args[0]
1050
1051 # For transferring possible exec failure from child to parent
1052 # The first char specifies the exception type: 0 means
1053 # OSError, 1 means some other error.
1054 errpipe_read, errpipe_write = os.pipe()
Christian Heimesfdab48e2008-01-20 09:06:41 +00001055 try:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001056 try:
Facundo Batista10706e22009-06-19 20:34:30 +00001057 self._set_cloexec_flag(errpipe_write)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001058
Facundo Batista10706e22009-06-19 20:34:30 +00001059 gc_was_enabled = gc.isenabled()
1060 # Disable gc to avoid bug where gc -> file_dealloc ->
1061 # write to stderr -> hang. http://bugs.python.org/issue1336
1062 gc.disable()
1063 try:
1064 self.pid = os.fork()
1065 except:
1066 if gc_was_enabled:
1067 gc.enable()
1068 raise
1069 self._child_created = True
1070 if self.pid == 0:
1071 # Child
1072 try:
1073 # Close parent's pipe ends
1074 if p2cwrite is not None:
1075 os.close(p2cwrite)
1076 if c2pread is not None:
1077 os.close(c2pread)
1078 if errread is not None:
1079 os.close(errread)
1080 os.close(errpipe_read)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001081
Facundo Batista10706e22009-06-19 20:34:30 +00001082 # Dup fds for child
1083 if p2cread is not None:
1084 os.dup2(p2cread, 0)
1085 if c2pwrite is not None:
1086 os.dup2(c2pwrite, 1)
1087 if errwrite is not None:
1088 os.dup2(errwrite, 2)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001089
Facundo Batista10706e22009-06-19 20:34:30 +00001090 # Close pipe fds. Make sure we don't close the
1091 # same fd more than once, or standard fds.
1092 if p2cread is not None and p2cread not in (0,):
1093 os.close(p2cread)
1094 if c2pwrite is not None and \
1095 c2pwrite not in (p2cread, 1):
1096 os.close(c2pwrite)
1097 if (errwrite is not None and
1098 errwrite not in (p2cread, c2pwrite, 2)):
1099 os.close(errwrite)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001100
Facundo Batista10706e22009-06-19 20:34:30 +00001101 # Close all other fds, if asked for
1102 if close_fds:
1103 self._close_fds(but=errpipe_write)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001104
Facundo Batista10706e22009-06-19 20:34:30 +00001105 if cwd is not None:
1106 os.chdir(cwd)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001107
Facundo Batista10706e22009-06-19 20:34:30 +00001108 if preexec_fn:
1109 preexec_fn()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001110
Facundo Batista10706e22009-06-19 20:34:30 +00001111 if env is None:
1112 os.execvp(executable, args)
1113 else:
1114 os.execvpe(executable, args, env)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001115
Facundo Batista10706e22009-06-19 20:34:30 +00001116 except:
1117 exc_type, exc_value, tb = sys.exc_info()
1118 # Save the traceback and attach it to the exception
1119 # object
1120 exc_lines = traceback.format_exception(exc_type,
1121 exc_value,
1122 tb)
1123 exc_value.child_traceback = ''.join(exc_lines)
1124 os.write(errpipe_write, pickle.dumps(exc_value))
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001125
Facundo Batista10706e22009-06-19 20:34:30 +00001126 # This exitcode won't be reported to applications, so
1127 # it really doesn't matter what we return.
1128 os._exit(255)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001129
Facundo Batista10706e22009-06-19 20:34:30 +00001130 # Parent
1131 if gc_was_enabled:
1132 gc.enable()
1133 finally:
1134 # be sure the FD is closed no matter what
1135 os.close(errpipe_write)
1136
1137 if p2cread is not None and p2cwrite is not None:
1138 os.close(p2cread)
1139 if c2pwrite is not None and c2pread is not None:
1140 os.close(c2pwrite)
1141 if errwrite is not None and errread is not None:
1142 os.close(errwrite)
1143
1144 # Wait for exec to fail or succeed; possibly raising an
1145 # exception (limited to 1 MB)
Gregory P. Smitha59c59f2010-03-01 00:17:40 +00001146 data = _eintr_retry_call(os.read, errpipe_read, 1048576)
Facundo Batista10706e22009-06-19 20:34:30 +00001147 finally:
1148 # be sure the FD is closed no matter what
1149 os.close(errpipe_read)
1150
Guido van Rossumaf2362a2007-05-15 22:32:02 +00001151 if data:
Gregory P. Smitha59c59f2010-03-01 00:17:40 +00001152 _eintr_retry_call(os.waitpid, self.pid, 0)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001153 child_exception = pickle.loads(data)
Benjamin Petersond75fcb42009-02-19 04:22:03 +00001154 for fd in (p2cwrite, c2pread, errread):
1155 if fd is not None:
1156 os.close(fd)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001157 raise child_exception
1158
1159
1160 def _handle_exitstatus(self, sts):
1161 if os.WIFSIGNALED(sts):
1162 self.returncode = -os.WTERMSIG(sts)
1163 elif os.WIFEXITED(sts):
1164 self.returncode = os.WEXITSTATUS(sts)
1165 else:
1166 # Should never happen
1167 raise RuntimeError("Unknown child exit status!")
1168
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001169
Georg Brandl6aa2d1f2008-08-12 08:35:52 +00001170 def _internal_poll(self, _deadstate=None):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001171 """Check if child process has terminated. Returns returncode
1172 attribute."""
Peter Astrandd38ddf42005-02-10 08:32:50 +00001173 if self.returncode is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001174 try:
1175 pid, sts = os.waitpid(self.pid, os.WNOHANG)
1176 if pid == self.pid:
1177 self._handle_exitstatus(sts)
1178 except os.error:
Thomas Wouters49fd7fa2006-04-21 10:40:58 +00001179 if _deadstate is not None:
1180 self.returncode = _deadstate
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001181 return self.returncode
1182
1183
1184 def wait(self):
1185 """Wait for child process to terminate. Returns returncode
1186 attribute."""
Peter Astrandd38ddf42005-02-10 08:32:50 +00001187 if self.returncode is None:
Gregory P. Smitha59c59f2010-03-01 00:17:40 +00001188 pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001189 self._handle_exitstatus(sts)
1190 return self.returncode
1191
1192
Peter Astrand23109f02005-03-03 20:28:59 +00001193 def _communicate(self, input):
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001194 if self.stdin:
1195 # Flush stdio buffer. This might block, if the user has
1196 # been writing to .stdin in an uncontrolled fashion.
1197 self.stdin.flush()
1198 if not input:
1199 self.stdin.close()
1200
1201 if _has_poll:
1202 stdout, stderr = self._communicate_with_poll(input)
1203 else:
1204 stdout, stderr = self._communicate_with_select(input)
1205
1206 # All data exchanged. Translate lists into strings.
1207 if stdout is not None:
1208 stdout = b''.join(stdout)
1209 if stderr is not None:
1210 stderr = b''.join(stderr)
1211
1212 # Translate newlines, if requested.
1213 # This also turns bytes into strings.
1214 if self.universal_newlines:
1215 if stdout is not None:
1216 stdout = self._translate_newlines(stdout,
1217 self.stdout.encoding)
1218 if stderr is not None:
1219 stderr = self._translate_newlines(stderr,
1220 self.stderr.encoding)
1221
1222 self.wait()
1223 return (stdout, stderr)
1224
1225
1226 def _communicate_with_poll(self, input):
1227 stdout = None # Return
1228 stderr = None # Return
1229 fd2file = {}
1230 fd2output = {}
1231
1232 poller = select.poll()
1233 def register_and_append(file_obj, eventmask):
1234 poller.register(file_obj.fileno(), eventmask)
1235 fd2file[file_obj.fileno()] = file_obj
1236
1237 def close_unregister_and_remove(fd):
1238 poller.unregister(fd)
1239 fd2file[fd].close()
1240 fd2file.pop(fd)
1241
1242 if self.stdin and input:
1243 register_and_append(self.stdin, select.POLLOUT)
1244
1245 select_POLLIN_POLLPRI = select.POLLIN | select.POLLPRI
1246 if self.stdout:
1247 register_and_append(self.stdout, select_POLLIN_POLLPRI)
1248 fd2output[self.stdout.fileno()] = stdout = []
1249 if self.stderr:
1250 register_and_append(self.stderr, select_POLLIN_POLLPRI)
1251 fd2output[self.stderr.fileno()] = stderr = []
1252
1253 input_offset = 0
1254 while fd2file:
1255 try:
1256 ready = poller.poll()
1257 except select.error as e:
1258 if e.args[0] == errno.EINTR:
1259 continue
1260 raise
1261
1262 # XXX Rewrite these to use non-blocking I/O on the
1263 # file objects; they are no longer using C stdio!
1264
1265 for fd, mode in ready:
1266 if mode & select.POLLOUT:
1267 chunk = input[input_offset : input_offset + _PIPE_BUF]
1268 input_offset += os.write(fd, chunk)
1269 if input_offset >= len(input):
1270 close_unregister_and_remove(fd)
1271 elif mode & select_POLLIN_POLLPRI:
1272 data = os.read(fd, 4096)
1273 if not data:
1274 close_unregister_and_remove(fd)
1275 fd2output[fd].append(data)
1276 else:
1277 # Ignore hang up or errors.
1278 close_unregister_and_remove(fd)
1279
1280 return (stdout, stderr)
1281
1282
1283 def _communicate_with_select(self, input):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001284 read_set = []
1285 write_set = []
1286 stdout = None # Return
1287 stderr = None # Return
1288
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001289 if self.stdin and input:
1290 write_set.append(self.stdin)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001291 if self.stdout:
1292 read_set.append(self.stdout)
Tim Peterse718f612004-10-12 21:51:32 +00001293 stdout = []
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001294 if self.stderr:
1295 read_set.append(self.stderr)
1296 stderr = []
1297
Thomas Wouters902d6eb2007-01-09 23:18:33 +00001298 input_offset = 0
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001299 while read_set or write_set:
Georg Brandl86b2fb92008-07-16 03:43:04 +00001300 try:
1301 rlist, wlist, xlist = select.select(read_set, write_set, [])
1302 except select.error as e:
1303 if e.args[0] == errno.EINTR:
1304 continue
1305 raise
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001306
Guido van Rossum98297ee2007-11-06 21:34:58 +00001307 # XXX Rewrite these to use non-blocking I/O on the
1308 # file objects; they are no longer using C stdio!
1309
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001310 if self.stdin in wlist:
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001311 chunk = input[input_offset : input_offset + _PIPE_BUF]
Guido van Rossumbae07c92007-10-08 02:46:15 +00001312 bytes_written = os.write(self.stdin.fileno(), chunk)
Thomas Wouters9fe394c2007-02-05 01:24:16 +00001313 input_offset += bytes_written
Thomas Wouters902d6eb2007-01-09 23:18:33 +00001314 if input_offset >= len(input):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001315 self.stdin.close()
1316 write_set.remove(self.stdin)
1317
1318 if self.stdout in rlist:
1319 data = os.read(self.stdout.fileno(), 1024)
Guido van Rossumc9e363c2007-05-15 23:18:55 +00001320 if not data:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001321 self.stdout.close()
1322 read_set.remove(self.stdout)
1323 stdout.append(data)
1324
1325 if self.stderr in rlist:
1326 data = os.read(self.stderr.fileno(), 1024)
Guido van Rossumc9e363c2007-05-15 23:18:55 +00001327 if not data:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001328 self.stderr.close()
1329 read_set.remove(self.stderr)
1330 stderr.append(data)
1331
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001332 return (stdout, stderr)
1333
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001334
Christian Heimesa342c012008-04-20 21:01:16 +00001335 def send_signal(self, sig):
1336 """Send a signal to the process
1337 """
1338 os.kill(self.pid, sig)
1339
1340 def terminate(self):
1341 """Terminate the process with SIGTERM
1342 """
1343 self.send_signal(signal.SIGTERM)
1344
1345 def kill(self):
1346 """Kill the process with SIGKILL
1347 """
1348 self.send_signal(signal.SIGKILL)
1349
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001350
1351def _demo_posix():
1352 #
1353 # Example 1: Simple redirection: Get process list
1354 #
1355 plist = Popen(["ps"], stdout=PIPE).communicate()[0]
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001356 print("Process list:")
1357 print(plist)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001358
1359 #
1360 # Example 2: Change uid before executing child
1361 #
1362 if os.getuid() == 0:
1363 p = Popen(["id"], preexec_fn=lambda: os.setuid(100))
1364 p.wait()
1365
1366 #
1367 # Example 3: Connecting several subprocesses
1368 #
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001369 print("Looking for 'hda'...")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001370 p1 = Popen(["dmesg"], stdout=PIPE)
1371 p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001372 print(repr(p2.communicate()[0]))
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001373
1374 #
1375 # Example 4: Catch execution error
1376 #
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001377 print()
1378 print("Trying a weird file...")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001379 try:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001380 print(Popen(["/this/path/does/not/exist"]).communicate())
Guido van Rossumb940e112007-01-10 16:19:56 +00001381 except OSError as e:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001382 if e.errno == errno.ENOENT:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001383 print("The file didn't exist. I thought so...")
1384 print("Child traceback:")
1385 print(e.child_traceback)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001386 else:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001387 print("Error", e.errno)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001388 else:
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001389 print("Gosh. No error.", file=sys.stderr)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001390
1391
1392def _demo_windows():
1393 #
1394 # Example 1: Connecting several subprocesses
1395 #
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001396 print("Looking for 'PROMPT' in set output...")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001397 p1 = Popen("set", stdout=PIPE, shell=True)
1398 p2 = Popen('find "PROMPT"', stdin=p1.stdout, stdout=PIPE)
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001399 print(repr(p2.communicate()[0]))
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001400
1401 #
1402 # Example 2: Simple execution of program
1403 #
Guido van Rossumbe19ed72007-02-09 05:37:30 +00001404 print("Executing calc...")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001405 p = Popen("calc")
1406 p.wait()
1407
1408
1409if __name__ == "__main__":
1410 if mswindows:
1411 _demo_windows()
1412 else:
1413 _demo_posix()