blob: 6e0cf06637ad8a8f264eded72ed87399662df4b4 [file] [log] [blame]
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001# subprocess - Subprocesses with accessible I/O streams
2#
Tim Peterse718f612004-10-12 21:51:32 +00003# For more information about this module, see PEP 324.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00004#
Peter Astrand3a708df2005-09-23 17:37:29 +00005# Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00006#
Peter Astrand69bf13f2005-02-14 08:56:32 +00007# Licensed to PSF under a Contributor Agreement.
Peter Astrand3a708df2005-09-23 17:37:29 +00008# See http://www.python.org/2.4/license for licensing details.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00009
Raymond Hettinger837dd932004-10-17 16:36:53 +000010r"""subprocess - Subprocesses with accessible I/O streams
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000011
Fredrik Lundh15aaacc2004-10-17 14:47:05 +000012This module allows you to spawn processes, connect to their
13input/output/error pipes, and obtain their return codes. This module
14intends to replace several other, older modules and functions, like:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000015
16os.system
17os.spawn*
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000018
19Information about how the subprocess module can be used to replace these
20modules and functions can be found below.
21
22
23
24Using the subprocess module
25===========================
26This module defines one class called Popen:
27
28class Popen(args, bufsize=0, executable=None,
29 stdin=None, stdout=None, stderr=None,
Gregory P. Smith8edd99d2010-12-14 13:43:30 +000030 preexec_fn=None, close_fds=True, shell=False,
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000031 cwd=None, env=None, universal_newlines=False,
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +000032 startupinfo=None, creationflags=0,
Gregory P. Smith8edd99d2010-12-14 13:43:30 +000033 restore_signals=True, start_new_session=False, pass_fds=()):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000034
35
36Arguments are:
37
38args should be a string, or a sequence of program arguments. The
39program to execute is normally the first item in the args sequence or
40string, but can be explicitly set by using the executable argument.
41
Gregory P. Smithf5604852010-12-13 06:45:02 +000042On POSIX, with shell=False (default): In this case, the Popen class
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000043uses os.execvp() to execute the child program. args should normally
44be a sequence. A string will be treated as a sequence with the string
45as the only item (the program to execute).
46
Gregory P. Smithf5604852010-12-13 06:45:02 +000047On POSIX, with shell=True: If args is a string, it specifies the
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000048command string to execute through the shell. If args is a sequence,
49the first item specifies the command string, and any additional items
50will be treated as additional shell arguments.
51
52On Windows: the Popen class uses CreateProcess() to execute the child
53program, which operates on strings. If args is a sequence, it will be
54converted to a string using the list2cmdline method. Please note that
55not all MS Windows applications interpret the command line the same
56way: The list2cmdline is designed for applications using the same
57rules as the MS C runtime.
58
59bufsize, if given, has the same meaning as the corresponding argument
60to the built-in open() function: 0 means unbuffered, 1 means line
61buffered, any other positive value means use a buffer of
62(approximately) that size. A negative bufsize means to use the system
63default, which usually means fully buffered. The default value for
64bufsize is 0 (unbuffered).
65
66stdin, stdout and stderr specify the executed programs' standard
67input, standard output and standard error file handles, respectively.
68Valid values are PIPE, an existing file descriptor (a positive
69integer), an existing file object, and None. PIPE indicates that a
70new pipe to the child should be created. With None, no redirection
71will occur; the child's file handles will be inherited from the
72parent. Additionally, stderr can be STDOUT, which indicates that the
73stderr data from the applications should be captured into the same
74file handle as for stdout.
75
Gregory P. Smithf5604852010-12-13 06:45:02 +000076On POSIX, if preexec_fn is set to a callable object, this object will be
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +000077called in the child process just before the child is executed. The use
78of preexec_fn is not thread safe, using it in the presence of threads
79could lead to a deadlock in the child process before the new executable
80is executed.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000081
82If close_fds is true, all file descriptors except 0, 1 and 2 will be
Gregory P. Smithf5604852010-12-13 06:45:02 +000083closed before the child process is executed. The default for close_fds
Gregory P. Smith8edd99d2010-12-14 13:43:30 +000084varies by platform: Always true on POSIX. True when stdin/stdout/stderr
85are None on Windows, false otherwise.
86
87pass_fds is an optional sequence of file descriptors to keep open between the
88parent and child. Providing any pass_fds implicitly sets close_fds to true.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000089
90if shell is true, the specified command will be executed through the
91shell.
92
93If cwd is not None, the current directory will be changed to cwd
94before the child is executed.
95
Gregory P. Smithf5604852010-12-13 06:45:02 +000096On POSIX, if restore_signals is True all signals that Python sets to
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +000097SIG_IGN are restored to SIG_DFL in the child process before the exec.
98Currently this includes the SIGPIPE, SIGXFZ and SIGXFSZ signals. This
99parameter does nothing on Windows.
100
Gregory P. Smithf5604852010-12-13 06:45:02 +0000101On POSIX, if start_new_session is True, the setsid() system call will be made
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000102in the child process prior to executing the command.
103
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000104If env is not None, it defines the environment variables for the new
105process.
106
107If universal_newlines is true, the file objects stdout and stderr are
108opened as a text files, but lines may be terminated by any of '\n',
Gregory P. Smithf5604852010-12-13 06:45:02 +0000109the Unix end-of-line convention, '\r', the old Macintosh convention or
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000110'\r\n', the Windows convention. All of these external representations
111are seen as '\n' by the Python program. Note: This feature is only
112available if Python is built with universal newline support (the
113default). Also, the newlines attribute of the file objects stdout,
114stdin and stderr are not updated by the communicate() method.
115
116The startupinfo and creationflags, if given, will be passed to the
117underlying CreateProcess() function. They can specify things such as
118appearance of the main window and priority for the new process.
119(Windows only)
120
121
Georg Brandlf9734072008-12-07 15:30:06 +0000122This module also defines some shortcut functions:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000123
Peter Astrand5f5e1412004-12-05 20:15:36 +0000124call(*popenargs, **kwargs):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000125 Run command with arguments. Wait for command to complete, then
126 return the returncode attribute.
127
128 The arguments are the same as for the Popen constructor. Example:
129
Florent Xicluna4886d242010-03-08 13:27:26 +0000130 >>> retcode = subprocess.call(["ls", "-l"])
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000131
Peter Astrand454f7672005-01-01 09:36:35 +0000132check_call(*popenargs, **kwargs):
133 Run command with arguments. Wait for command to complete. If the
134 exit code was zero then return, otherwise raise
135 CalledProcessError. The CalledProcessError object will have the
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000136 return code in the returncode attribute.
Peter Astrand454f7672005-01-01 09:36:35 +0000137
138 The arguments are the same as for the Popen constructor. Example:
139
Florent Xicluna4886d242010-03-08 13:27:26 +0000140 >>> subprocess.check_call(["ls", "-l"])
Georg Brandl2708f3a2009-12-20 14:38:23 +0000141 0
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000142
Brett Cannona23810f2008-05-26 19:04:21 +0000143getstatusoutput(cmd):
144 Return (status, output) of executing cmd in a shell.
145
146 Execute the string 'cmd' in a shell with os.popen() and return a 2-tuple
147 (status, output). cmd is actually run as '{ cmd ; } 2>&1', so that the
148 returned output will contain output or error messages. A trailing newline
149 is stripped from the output. The exit status for the command can be
150 interpreted according to the rules for the C function wait(). Example:
151
Brett Cannona23810f2008-05-26 19:04:21 +0000152 >>> subprocess.getstatusoutput('ls /bin/ls')
153 (0, '/bin/ls')
154 >>> subprocess.getstatusoutput('cat /bin/junk')
155 (256, 'cat: /bin/junk: No such file or directory')
156 >>> subprocess.getstatusoutput('/bin/junk')
157 (256, 'sh: /bin/junk: not found')
158
159getoutput(cmd):
160 Return output (stdout or stderr) of executing cmd in a shell.
161
162 Like getstatusoutput(), except the exit status is ignored and the return
163 value is a string containing the command's output. Example:
164
Brett Cannona23810f2008-05-26 19:04:21 +0000165 >>> subprocess.getoutput('ls /bin/ls')
166 '/bin/ls'
167
Georg Brandlf9734072008-12-07 15:30:06 +0000168check_output(*popenargs, **kwargs):
Georg Brandl2708f3a2009-12-20 14:38:23 +0000169 Run command with arguments and return its output as a byte string.
Georg Brandlf9734072008-12-07 15:30:06 +0000170
Georg Brandl2708f3a2009-12-20 14:38:23 +0000171 If the exit code was non-zero it raises a CalledProcessError. The
172 CalledProcessError object will have the return code in the returncode
173 attribute and output in the output attribute.
Georg Brandlf9734072008-12-07 15:30:06 +0000174
Georg Brandl2708f3a2009-12-20 14:38:23 +0000175 The arguments are the same as for the Popen constructor. Example:
Georg Brandlf9734072008-12-07 15:30:06 +0000176
Georg Brandl2708f3a2009-12-20 14:38:23 +0000177 >>> output = subprocess.check_output(["ls", "-l", "/dev/null"])
Georg Brandlf9734072008-12-07 15:30:06 +0000178
Brett Cannona23810f2008-05-26 19:04:21 +0000179
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000180Exceptions
181----------
182Exceptions raised in the child process, before the new program has
183started to execute, will be re-raised in the parent. Additionally,
184the exception object will have one extra attribute called
185'child_traceback', which is a string containing traceback information
186from the childs point of view.
187
188The most common exception raised is OSError. This occurs, for
189example, when trying to execute a non-existent file. Applications
190should prepare for OSErrors.
191
192A ValueError will be raised if Popen is called with invalid arguments.
193
Gregory P. Smith54d412e2011-03-14 14:08:43 -0400194Exceptions defined within this module inherit from SubprocessError.
195check_call() and check_output() will raise CalledProcessError if the
Gregory P. Smithb4039aa2011-03-14 14:16:20 -0400196called process returns a non-zero return code. TimeoutExpired
Gregory P. Smith54d412e2011-03-14 14:08:43 -0400197be raised if a timeout was specified and expired.
Peter Astrand454f7672005-01-01 09:36:35 +0000198
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000199
200Security
201--------
202Unlike some other popen functions, this implementation will never call
203/bin/sh implicitly. This means that all characters, including shell
204metacharacters, can safely be passed to child processes.
205
206
207Popen objects
208=============
209Instances of the Popen class have the following methods:
210
211poll()
212 Check if child process has terminated. Returns returncode
213 attribute.
214
215wait()
216 Wait for child process to terminate. Returns returncode attribute.
217
218communicate(input=None)
219 Interact with process: Send data to stdin. Read data from stdout
220 and stderr, until end-of-file is reached. Wait for process to
Thomas Wouters902d6eb2007-01-09 23:18:33 +0000221 terminate. The optional input argument should be a string to be
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000222 sent to the child process, or None, if no data should be sent to
223 the child.
Tim Peterse718f612004-10-12 21:51:32 +0000224
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000225 communicate() returns a tuple (stdout, stderr).
226
227 Note: The data read is buffered in memory, so do not use this
228 method if the data size is large or unlimited.
229
230The following attributes are also available:
231
232stdin
233 If the stdin argument is PIPE, this attribute is a file object
234 that provides input to the child process. Otherwise, it is None.
235
236stdout
237 If the stdout argument is PIPE, this attribute is a file object
238 that provides output from the child process. Otherwise, it is
239 None.
240
241stderr
242 If the stderr argument is PIPE, this attribute is file object that
243 provides error output from the child process. Otherwise, it is
244 None.
245
246pid
247 The process ID of the child process.
248
249returncode
250 The child return code. A None value indicates that the process
251 hasn't terminated yet. A negative value -N indicates that the
Gregory P. Smithf5604852010-12-13 06:45:02 +0000252 child was terminated by signal N (POSIX only).
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000253
254
255Replacing older functions with the subprocess module
256====================================================
257In this section, "a ==> b" means that b can be used as a replacement
258for a.
259
260Note: All functions in this section fail (more or less) silently if
261the executed program cannot be found; this module raises an OSError
262exception.
263
264In the following examples, we assume that the subprocess module is
265imported with "from subprocess import *".
266
267
268Replacing /bin/sh shell backquote
269---------------------------------
270output=`mycmd myarg`
271==>
272output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
273
274
275Replacing shell pipe line
276-------------------------
277output=`dmesg | grep hda`
278==>
279p1 = Popen(["dmesg"], stdout=PIPE)
Peter Astrand6fdf3cb2004-11-30 18:06:42 +0000280p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000281output = p2.communicate()[0]
282
283
284Replacing os.system()
285---------------------
286sts = os.system("mycmd" + " myarg")
287==>
288p = Popen("mycmd" + " myarg", shell=True)
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000289pid, sts = os.waitpid(p.pid, 0)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000290
291Note:
292
293* Calling the program through the shell is usually not required.
294
295* It's easier to look at the returncode attribute than the
296 exitstatus.
297
298A more real-world example would look like this:
299
300try:
301 retcode = call("mycmd" + " myarg", shell=True)
302 if retcode < 0:
Guido van Rossumc2f93dc2007-05-24 00:50:02 +0000303 print("Child was terminated by signal", -retcode, file=sys.stderr)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000304 else:
Guido van Rossumc2f93dc2007-05-24 00:50:02 +0000305 print("Child returned", retcode, file=sys.stderr)
306except OSError as e:
307 print("Execution failed:", e, file=sys.stderr)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000308
309
310Replacing os.spawn*
311-------------------
Tim Peterse718f612004-10-12 21:51:32 +0000312P_NOWAIT example:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000313
314pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
315==>
316pid = Popen(["/bin/mycmd", "myarg"]).pid
317
318
319P_WAIT example:
320
321retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
322==>
323retcode = call(["/bin/mycmd", "myarg"])
324
325
Tim Peterse718f612004-10-12 21:51:32 +0000326Vector example:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000327
328os.spawnvp(os.P_NOWAIT, path, args)
329==>
330Popen([path] + args[1:])
331
332
Tim Peterse718f612004-10-12 21:51:32 +0000333Environment example:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000334
335os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
336==>
337Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000338"""
339
340import sys
341mswindows = (sys.platform == "win32")
342
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000343import io
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000344import os
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400345import time
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000346import traceback
Christian Heimesfdab48e2008-01-20 09:06:41 +0000347import gc
Christian Heimesa342c012008-04-20 21:01:16 +0000348import signal
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000349import builtins
Gregory P. Smithd23047b2010-12-04 09:10:44 +0000350import warnings
Ross Lagerwall4f61b022011-04-05 15:34:00 +0200351import errno
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000352
Peter Astrand454f7672005-01-01 09:36:35 +0000353# Exception classes used by this module.
Gregory P. Smith54d412e2011-03-14 14:08:43 -0400354class SubprocessError(Exception): pass
355
356
357class CalledProcessError(SubprocessError):
Georg Brandlf9734072008-12-07 15:30:06 +0000358 """This exception is raised when a process run by check_call() or
359 check_output() returns a non-zero exit status.
360 The exit status will be stored in the returncode attribute;
361 check_output() will also store the output in the output attribute.
362 """
363 def __init__(self, returncode, cmd, output=None):
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000364 self.returncode = returncode
365 self.cmd = cmd
Georg Brandlf9734072008-12-07 15:30:06 +0000366 self.output = output
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000367 def __str__(self):
368 return "Command '%s' returned non-zero exit status %d" % (self.cmd, self.returncode)
369
Peter Astrand454f7672005-01-01 09:36:35 +0000370
Gregory P. Smith54d412e2011-03-14 14:08:43 -0400371class TimeoutExpired(SubprocessError):
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400372 """This exception is raised when the timeout expires while waiting for a
373 child process.
374 """
Reid Kleckner2b228f02011-03-16 16:57:54 -0400375 def __init__(self, cmd, timeout, output=None):
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400376 self.cmd = cmd
Reid Kleckner2b228f02011-03-16 16:57:54 -0400377 self.timeout = timeout
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400378 self.output = output
379
380 def __str__(self):
381 return ("Command '%s' timed out after %s seconds" %
382 (self.cmd, self.timeout))
383
384
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000385if mswindows:
386 import threading
387 import msvcrt
Brian Curtin1ce6b582010-04-24 16:19:22 +0000388 import _subprocess
389 class STARTUPINFO:
390 dwFlags = 0
391 hStdInput = None
392 hStdOutput = None
393 hStdError = None
394 wShowWindow = 0
395 class pywintypes:
396 error = IOError
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000397else:
398 import select
Gregory P. Smithd06fa472009-07-04 02:46:54 +0000399 _has_poll = hasattr(select, 'poll')
Gregory P. Smith59fd1bf2011-05-28 09:32:39 -0700400 import _posixsubprocess
401 _create_pipe = _posixsubprocess.cloexec_pipe
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000402
Amaury Forgeot d'Arcace31022009-07-09 22:44:11 +0000403 # When select or poll has indicated that the file is writable,
404 # we can write up to _PIPE_BUF bytes without risk of blocking.
405 # POSIX defines PIPE_BUF as >= 512.
406 _PIPE_BUF = getattr(select, 'PIPE_BUF', 512)
407
408
Brett Cannona23810f2008-05-26 19:04:21 +0000409__all__ = ["Popen", "PIPE", "STDOUT", "call", "check_call", "getstatusoutput",
Ross Lagerwallba102ec2011-03-16 18:40:25 +0200410 "getoutput", "check_output", "CalledProcessError", "DEVNULL"]
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000411
Brian Curtin1ce6b582010-04-24 16:19:22 +0000412if mswindows:
Brian Curtin08fd8d92011-04-29 16:11:30 -0500413 from _subprocess import (CREATE_NEW_CONSOLE, CREATE_NEW_PROCESS_GROUP,
Brian Curtin8b8e7f42011-04-29 15:48:13 -0500414 STD_INPUT_HANDLE, STD_OUTPUT_HANDLE,
415 STD_ERROR_HANDLE, SW_HIDE,
416 STARTF_USESTDHANDLES, STARTF_USESHOWWINDOW)
Brian Curtin5d9deaa2011-04-29 16:24:07 -0500417
Brian Curtin08fd8d92011-04-29 16:11:30 -0500418 __all__.extend(["CREATE_NEW_CONSOLE", "CREATE_NEW_PROCESS_GROUP",
Brian Curtin8b8e7f42011-04-29 15:48:13 -0500419 "STD_INPUT_HANDLE", "STD_OUTPUT_HANDLE",
420 "STD_ERROR_HANDLE", "SW_HIDE",
421 "STARTF_USESTDHANDLES", "STARTF_USESHOWWINDOW"])
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000422try:
423 MAXFD = os.sysconf("SC_OPEN_MAX")
424except:
425 MAXFD = 256
426
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000427_active = []
428
429def _cleanup():
430 for inst in _active[:]:
Georg Brandl6aa2d1f2008-08-12 08:35:52 +0000431 res = inst._internal_poll(_deadstate=sys.maxsize)
Guido van Rossumb5d47ef2006-08-24 02:27:45 +0000432 if res is not None and res >= 0:
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000433 try:
434 _active.remove(inst)
435 except ValueError:
436 # This can happen if two threads create a new Popen instance.
437 # It's harmless that it was already removed, so ignore.
438 pass
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000439
440PIPE = -1
441STDOUT = -2
Ross Lagerwallba102ec2011-03-16 18:40:25 +0200442DEVNULL = -3
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000443
444
Gregory P. Smitha59c59f2010-03-01 00:17:40 +0000445def _eintr_retry_call(func, *args):
446 while True:
447 try:
448 return func(*args)
449 except OSError as e:
450 if e.errno == errno.EINTR:
451 continue
452 raise
453
454
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400455def call(*popenargs, timeout=None, **kwargs):
456 """Run command with arguments. Wait for command to complete or
457 timeout, then return the returncode attribute.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000458
459 The arguments are the same as for the Popen constructor. Example:
460
461 retcode = call(["ls", "-l"])
462 """
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400463 p = Popen(*popenargs, **kwargs)
464 try:
465 return p.wait(timeout=timeout)
466 except TimeoutExpired:
467 p.kill()
468 p.wait()
469 raise
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000470
471
Peter Astrand454f7672005-01-01 09:36:35 +0000472def check_call(*popenargs, **kwargs):
473 """Run command with arguments. Wait for command to complete. If
474 the exit code was zero then return, otherwise raise
475 CalledProcessError. The CalledProcessError object will have the
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000476 return code in the returncode attribute.
Peter Astrand454f7672005-01-01 09:36:35 +0000477
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400478 The arguments are the same as for the call function. Example:
Peter Astrand454f7672005-01-01 09:36:35 +0000479
480 check_call(["ls", "-l"])
481 """
482 retcode = call(*popenargs, **kwargs)
Peter Astrand454f7672005-01-01 09:36:35 +0000483 if retcode:
Georg Brandlf9734072008-12-07 15:30:06 +0000484 cmd = kwargs.get("args")
485 if cmd is None:
486 cmd = popenargs[0]
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000487 raise CalledProcessError(retcode, cmd)
Georg Brandlf9734072008-12-07 15:30:06 +0000488 return 0
489
490
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400491def check_output(*popenargs, timeout=None, **kwargs):
Georg Brandl2708f3a2009-12-20 14:38:23 +0000492 r"""Run command with arguments and return its output as a byte string.
Georg Brandlf9734072008-12-07 15:30:06 +0000493
494 If the exit code was non-zero it raises a CalledProcessError. The
495 CalledProcessError object will have the return code in the returncode
496 attribute and output in the output attribute.
497
498 The arguments are the same as for the Popen constructor. Example:
499
500 >>> check_output(["ls", "-l", "/dev/null"])
Georg Brandl2708f3a2009-12-20 14:38:23 +0000501 b'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n'
Georg Brandlf9734072008-12-07 15:30:06 +0000502
503 The stdout argument is not allowed as it is used internally.
Georg Brandl127d4702009-12-28 08:10:38 +0000504 To capture standard error in the result, use stderr=STDOUT.
Georg Brandlf9734072008-12-07 15:30:06 +0000505
506 >>> check_output(["/bin/sh", "-c",
Georg Brandl2708f3a2009-12-20 14:38:23 +0000507 ... "ls -l non_existent_file ; exit 0"],
Georg Brandl127d4702009-12-28 08:10:38 +0000508 ... stderr=STDOUT)
Georg Brandl2708f3a2009-12-20 14:38:23 +0000509 b'ls: non_existent_file: No such file or directory\n'
Georg Brandlf9734072008-12-07 15:30:06 +0000510 """
511 if 'stdout' in kwargs:
512 raise ValueError('stdout argument not allowed, it will be overridden.')
513 process = Popen(*popenargs, stdout=PIPE, **kwargs)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400514 try:
515 output, unused_err = process.communicate(timeout=timeout)
516 except TimeoutExpired:
517 process.kill()
518 output, unused_err = process.communicate()
Reid Kleckner2b228f02011-03-16 16:57:54 -0400519 raise TimeoutExpired(process.args, timeout, output=output)
Georg Brandlf9734072008-12-07 15:30:06 +0000520 retcode = process.poll()
521 if retcode:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400522 raise CalledProcessError(retcode, process.args, output=output)
Georg Brandlf9734072008-12-07 15:30:06 +0000523 return output
Peter Astrand454f7672005-01-01 09:36:35 +0000524
525
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000526def list2cmdline(seq):
527 """
528 Translate a sequence of arguments into a command line
529 string, using the same rules as the MS C runtime:
530
531 1) Arguments are delimited by white space, which is either a
532 space or a tab.
533
534 2) A string surrounded by double quotation marks is
535 interpreted as a single argument, regardless of white space
Jean-Paul Calderone1ddd4072010-06-18 20:03:54 +0000536 contained within. A quoted string can be embedded in an
537 argument.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000538
539 3) A double quotation mark preceded by a backslash is
540 interpreted as a literal double quotation mark.
541
542 4) Backslashes are interpreted literally, unless they
543 immediately precede a double quotation mark.
544
545 5) If backslashes immediately precede a double quotation mark,
546 every pair of backslashes is interpreted as a literal
547 backslash. If the number of backslashes is odd, the last
548 backslash escapes the next double quotation mark as
549 described in rule 3.
550 """
551
552 # See
Eric Smith3c573af2009-11-09 15:23:15 +0000553 # http://msdn.microsoft.com/en-us/library/17w5ykft.aspx
554 # or search http://msdn.microsoft.com for
555 # "Parsing C++ Command-Line Arguments"
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000556 result = []
557 needquote = False
558 for arg in seq:
559 bs_buf = []
560
561 # Add a space to separate this argument from the others
562 if result:
563 result.append(' ')
564
Jean-Paul Calderone1ddd4072010-06-18 20:03:54 +0000565 needquote = (" " in arg) or ("\t" in arg) or not arg
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000566 if needquote:
567 result.append('"')
568
569 for c in arg:
570 if c == '\\':
Tim Peterse718f612004-10-12 21:51:32 +0000571 # Don't know if we need to double yet.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000572 bs_buf.append(c)
573 elif c == '"':
Christian Heimesfdab48e2008-01-20 09:06:41 +0000574 # Double backslashes.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000575 result.append('\\' * len(bs_buf)*2)
576 bs_buf = []
577 result.append('\\"')
578 else:
579 # Normal char
580 if bs_buf:
581 result.extend(bs_buf)
582 bs_buf = []
583 result.append(c)
584
Christian Heimesfdab48e2008-01-20 09:06:41 +0000585 # Add remaining backslashes, if any.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000586 if bs_buf:
587 result.extend(bs_buf)
588
589 if needquote:
Peter Astrand7e78ade2005-03-03 21:10:23 +0000590 result.extend(bs_buf)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000591 result.append('"')
592
593 return ''.join(result)
594
595
Brett Cannona23810f2008-05-26 19:04:21 +0000596# Various tools for executing commands and looking at their output and status.
597#
Gregory P. Smithf5604852010-12-13 06:45:02 +0000598# NB This only works (and is only relevant) for POSIX.
Brett Cannona23810f2008-05-26 19:04:21 +0000599
600def getstatusoutput(cmd):
601 """Return (status, output) of executing cmd in a shell.
602
603 Execute the string 'cmd' in a shell with os.popen() and return a 2-tuple
604 (status, output). cmd is actually run as '{ cmd ; } 2>&1', so that the
605 returned output will contain output or error messages. A trailing newline
606 is stripped from the output. The exit status for the command can be
607 interpreted according to the rules for the C function wait(). Example:
608
609 >>> import subprocess
610 >>> subprocess.getstatusoutput('ls /bin/ls')
611 (0, '/bin/ls')
612 >>> subprocess.getstatusoutput('cat /bin/junk')
613 (256, 'cat: /bin/junk: No such file or directory')
614 >>> subprocess.getstatusoutput('/bin/junk')
615 (256, 'sh: /bin/junk: not found')
616 """
617 pipe = os.popen('{ ' + cmd + '; } 2>&1', 'r')
618 text = pipe.read()
619 sts = pipe.close()
620 if sts is None: sts = 0
621 if text[-1:] == '\n': text = text[:-1]
622 return sts, text
623
624
625def getoutput(cmd):
626 """Return output (stdout or stderr) of executing cmd in a shell.
627
628 Like getstatusoutput(), except the exit status is ignored and the return
629 value is a string containing the command's output. Example:
630
631 >>> import subprocess
632 >>> subprocess.getoutput('ls /bin/ls')
633 '/bin/ls'
634 """
635 return getstatusoutput(cmd)[1]
636
637
Gregory P. Smith8edd99d2010-12-14 13:43:30 +0000638_PLATFORM_DEFAULT_CLOSE_FDS = object()
Gregory P. Smithf5604852010-12-13 06:45:02 +0000639
640
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000641class Popen(object):
642 def __init__(self, args, bufsize=0, executable=None,
643 stdin=None, stdout=None, stderr=None,
Gregory P. Smith8edd99d2010-12-14 13:43:30 +0000644 preexec_fn=None, close_fds=_PLATFORM_DEFAULT_CLOSE_FDS,
645 shell=False, cwd=None, env=None, universal_newlines=False,
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000646 startupinfo=None, creationflags=0,
Gregory P. Smithd4cc7bf2010-12-04 11:22:11 +0000647 restore_signals=True, start_new_session=False,
648 pass_fds=()):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000649 """Create new Popen instance."""
650 _cleanup()
651
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000652 self._child_created = False
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400653 self._input = None
654 self._communication_started = False
Guido van Rossum46a05a72007-06-07 21:56:45 +0000655 if bufsize is None:
656 bufsize = 0 # Restore default
Walter Dörwaldaa97f042007-05-03 21:05:51 +0000657 if not isinstance(bufsize, int):
Peter Astrand738131d2004-11-30 21:04:45 +0000658 raise TypeError("bufsize must be an integer")
659
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000660 if mswindows:
Tim Peterse8374a52004-10-13 03:15:00 +0000661 if preexec_fn is not None:
662 raise ValueError("preexec_fn is not supported on Windows "
663 "platforms")
Gregory P. Smith8edd99d2010-12-14 13:43:30 +0000664 any_stdio_set = (stdin is not None or stdout is not None or
665 stderr is not None)
666 if close_fds is _PLATFORM_DEFAULT_CLOSE_FDS:
667 if any_stdio_set:
668 close_fds = False
669 else:
670 close_fds = True
671 elif close_fds and any_stdio_set:
672 raise ValueError(
673 "close_fds is not supported on Windows platforms"
674 " if you redirect stdin/stdout/stderr")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000675 else:
676 # POSIX
Gregory P. Smith8edd99d2010-12-14 13:43:30 +0000677 if close_fds is _PLATFORM_DEFAULT_CLOSE_FDS:
678 close_fds = True
679 if pass_fds and not close_fds:
680 warnings.warn("pass_fds overriding close_fds.", RuntimeWarning)
681 close_fds = True
Tim Peterse8374a52004-10-13 03:15:00 +0000682 if startupinfo is not None:
683 raise ValueError("startupinfo is only supported on Windows "
684 "platforms")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000685 if creationflags != 0:
Tim Peterse8374a52004-10-13 03:15:00 +0000686 raise ValueError("creationflags is only supported on Windows "
687 "platforms")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000688
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400689 self.args = args
Tim Peterse718f612004-10-12 21:51:32 +0000690 self.stdin = None
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000691 self.stdout = None
692 self.stderr = None
693 self.pid = None
694 self.returncode = None
695 self.universal_newlines = universal_newlines
696
697 # Input and output objects. The general principle is like
698 # this:
699 #
700 # Parent Child
701 # ------ -----
702 # p2cwrite ---stdin---> p2cread
703 # c2pread <--stdout--- c2pwrite
704 # errread <--stderr--- errwrite
705 #
706 # On POSIX, the child objects are file descriptors. On
707 # Windows, these are Windows file handles. The parent objects
708 # are file descriptors on both platforms. The parent objects
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000709 # are -1 when not using PIPEs. The child objects are -1
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000710 # when not redirecting.
Tim Peterse718f612004-10-12 21:51:32 +0000711
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000712 (p2cread, p2cwrite,
713 c2pread, c2pwrite,
714 errread, errwrite) = self._get_handles(stdin, stdout, stderr)
715
Antoine Pitrouc9982322011-01-04 19:07:07 +0000716 # We wrap OS handles *before* launching the child, otherwise a
717 # quickly terminating child could make our fds unwrappable
718 # (see #8458).
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000719
Thomas Wouterscf297e42007-02-23 15:07:44 +0000720 if mswindows:
Florent Xicluna3b8bfef2010-03-14 12:31:06 +0000721 if p2cwrite != -1:
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000722 p2cwrite = msvcrt.open_osfhandle(p2cwrite.Detach(), 0)
Florent Xicluna3b8bfef2010-03-14 12:31:06 +0000723 if c2pread != -1:
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000724 c2pread = msvcrt.open_osfhandle(c2pread.Detach(), 0)
Florent Xicluna3b8bfef2010-03-14 12:31:06 +0000725 if errread != -1:
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000726 errread = msvcrt.open_osfhandle(errread.Detach(), 0)
Thomas Wouterscf297e42007-02-23 15:07:44 +0000727
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000728 if p2cwrite != -1:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000729 self.stdin = io.open(p2cwrite, 'wb', bufsize)
730 if self.universal_newlines:
731 self.stdin = io.TextIOWrapper(self.stdin)
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000732 if c2pread != -1:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000733 self.stdout = io.open(c2pread, 'rb', bufsize)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000734 if universal_newlines:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000735 self.stdout = io.TextIOWrapper(self.stdout)
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000736 if errread != -1:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000737 self.stderr = io.open(errread, 'rb', bufsize)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000738 if universal_newlines:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000739 self.stderr = io.TextIOWrapper(self.stderr)
Tim Peterse718f612004-10-12 21:51:32 +0000740
Antoine Pitrouc9982322011-01-04 19:07:07 +0000741 try:
742 self._execute_child(args, executable, preexec_fn, close_fds,
743 pass_fds, cwd, env, universal_newlines,
744 startupinfo, creationflags, shell,
745 p2cread, p2cwrite,
746 c2pread, c2pwrite,
747 errread, errwrite,
748 restore_signals, start_new_session)
749 except:
750 # Cleanup if the child failed starting
751 for f in filter(None, [self.stdin, self.stdout, self.stderr]):
752 try:
753 f.close()
754 except EnvironmentError:
755 # Ignore EBADF or other errors
756 pass
757 raise
758
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000759
Guido van Rossum98297ee2007-11-06 21:34:58 +0000760 def _translate_newlines(self, data, encoding):
761 data = data.replace(b"\r\n", b"\n").replace(b"\r", b"\n")
762 return data.decode(encoding)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000763
Brian Curtin79cdb662010-12-03 02:46:02 +0000764 def __enter__(self):
765 return self
766
767 def __exit__(self, type, value, traceback):
768 if self.stdout:
769 self.stdout.close()
770 if self.stderr:
771 self.stderr.close()
772 if self.stdin:
773 self.stdin.close()
Gregory P. Smith6b657452011-05-11 21:42:08 -0700774 # Wait for the process to terminate, to avoid zombies.
775 self.wait()
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000776
Brett Cannon84df1e62010-05-14 00:33:40 +0000777 def __del__(self, _maxsize=sys.maxsize, _active=_active):
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000778 if not self._child_created:
779 # We didn't get to successfully create a child process.
780 return
781 # In case the child hasn't been waited on, check if it's done.
Brett Cannon84df1e62010-05-14 00:33:40 +0000782 self._internal_poll(_deadstate=_maxsize)
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000783 if self.returncode is None and _active is not None:
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000784 # Child is still running, keep us alive until we can wait on it.
785 _active.append(self)
786
Ross Lagerwallba102ec2011-03-16 18:40:25 +0200787 def _get_devnull(self):
788 if not hasattr(self, '_devnull'):
789 self._devnull = os.open(os.devnull, os.O_RDWR)
790 return self._devnull
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000791
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400792 def communicate(self, input=None, timeout=None):
Peter Astrand23109f02005-03-03 20:28:59 +0000793 """Interact with process: Send data to stdin. Read data from
794 stdout and stderr, until end-of-file is reached. Wait for
Gregory P. Smitha454ef62011-05-22 22:29:49 -0700795 process to terminate. The optional input argument should be
796 bytes to be sent to the child process, or None, if no data
Peter Astrand23109f02005-03-03 20:28:59 +0000797 should be sent to the child.
Tim Peterseba28be2005-03-28 01:08:02 +0000798
Peter Astrand23109f02005-03-03 20:28:59 +0000799 communicate() returns a tuple (stdout, stderr)."""
800
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400801 if self._communication_started and input:
802 raise ValueError("Cannot send input after starting communication")
803
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400804 # Optimization: If we are not worried about timeouts, we haven't
805 # started communicating, and we have one or zero pipes, using select()
806 # or threads is unnecessary.
Victor Stinner7a8d0812011-04-05 13:13:08 +0200807 if (timeout is None and not self._communication_started and
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400808 [self.stdin, self.stdout, self.stderr].count(None) >= 2):
Tim Peterseba28be2005-03-28 01:08:02 +0000809 stdout = None
810 stderr = None
Peter Astrand23109f02005-03-03 20:28:59 +0000811 if self.stdin:
812 if input:
Ross Lagerwall4f61b022011-04-05 15:34:00 +0200813 try:
814 self.stdin.write(input)
815 except IOError as e:
816 if e.errno != errno.EPIPE and e.errno != errno.EINVAL:
817 raise
Peter Astrand23109f02005-03-03 20:28:59 +0000818 self.stdin.close()
819 elif self.stdout:
820 stdout = self.stdout.read()
Georg Brandlf08a9dd2008-06-10 16:57:31 +0000821 self.stdout.close()
Peter Astrand23109f02005-03-03 20:28:59 +0000822 elif self.stderr:
823 stderr = self.stderr.read()
Georg Brandlf08a9dd2008-06-10 16:57:31 +0000824 self.stderr.close()
Peter Astrand23109f02005-03-03 20:28:59 +0000825 self.wait()
Victor Stinner7a8d0812011-04-05 13:13:08 +0200826 else:
827 if timeout is not None:
828 endtime = time.time() + timeout
829 else:
830 endtime = None
Tim Peterseba28be2005-03-28 01:08:02 +0000831
Victor Stinner7a8d0812011-04-05 13:13:08 +0200832 try:
833 stdout, stderr = self._communicate(input, endtime, timeout)
834 finally:
835 self._communication_started = True
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400836
Victor Stinner7a8d0812011-04-05 13:13:08 +0200837 sts = self.wait(timeout=self._remaining_time(endtime))
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400838
839 return (stdout, stderr)
Peter Astrand23109f02005-03-03 20:28:59 +0000840
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000841
Georg Brandl6aa2d1f2008-08-12 08:35:52 +0000842 def poll(self):
843 return self._internal_poll()
844
845
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400846 def _remaining_time(self, endtime):
847 """Convenience for _communicate when computing timeouts."""
848 if endtime is None:
849 return None
850 else:
851 return endtime - time.time()
852
853
Reid Kleckner2b228f02011-03-16 16:57:54 -0400854 def _check_timeout(self, endtime, orig_timeout):
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400855 """Convenience for checking if a timeout has expired."""
856 if endtime is None:
857 return
858 if time.time() > endtime:
Reid Kleckner2b228f02011-03-16 16:57:54 -0400859 raise TimeoutExpired(self.args, orig_timeout)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400860
861
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000862 if mswindows:
863 #
864 # Windows methods
865 #
866 def _get_handles(self, stdin, stdout, stderr):
Alexandre Vassalotti711ed4a2009-07-17 10:42:05 +0000867 """Construct and return tuple with IO objects:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000868 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
869 """
Peter Astrandd38ddf42005-02-10 08:32:50 +0000870 if stdin is None and stdout is None and stderr is None:
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000871 return (-1, -1, -1, -1, -1, -1)
Tim Peterse718f612004-10-12 21:51:32 +0000872
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000873 p2cread, p2cwrite = -1, -1
874 c2pread, c2pwrite = -1, -1
875 errread, errwrite = -1, -1
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000876
Peter Astrandd38ddf42005-02-10 08:32:50 +0000877 if stdin is None:
Brian Curtin1ce6b582010-04-24 16:19:22 +0000878 p2cread = _subprocess.GetStdHandle(_subprocess.STD_INPUT_HANDLE)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000879 if p2cread is None:
Brian Curtin1ce6b582010-04-24 16:19:22 +0000880 p2cread, _ = _subprocess.CreatePipe(None, 0)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000881 elif stdin == PIPE:
Brian Curtin1ce6b582010-04-24 16:19:22 +0000882 p2cread, p2cwrite = _subprocess.CreatePipe(None, 0)
Ross Lagerwallba102ec2011-03-16 18:40:25 +0200883 elif stdin == DEVNULL:
884 p2cread = msvcrt.get_osfhandle(self._get_devnull())
Peter Astrandd38ddf42005-02-10 08:32:50 +0000885 elif isinstance(stdin, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000886 p2cread = msvcrt.get_osfhandle(stdin)
887 else:
888 # Assuming file-like object
889 p2cread = msvcrt.get_osfhandle(stdin.fileno())
890 p2cread = self._make_inheritable(p2cread)
891
Peter Astrandd38ddf42005-02-10 08:32:50 +0000892 if stdout is None:
Brian Curtin1ce6b582010-04-24 16:19:22 +0000893 c2pwrite = _subprocess.GetStdHandle(_subprocess.STD_OUTPUT_HANDLE)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000894 if c2pwrite is None:
Brian Curtin1ce6b582010-04-24 16:19:22 +0000895 _, c2pwrite = _subprocess.CreatePipe(None, 0)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000896 elif stdout == PIPE:
Brian Curtin1ce6b582010-04-24 16:19:22 +0000897 c2pread, c2pwrite = _subprocess.CreatePipe(None, 0)
Ross Lagerwallba102ec2011-03-16 18:40:25 +0200898 elif stdout == DEVNULL:
899 c2pwrite = msvcrt.get_osfhandle(self._get_devnull())
Peter Astrandd38ddf42005-02-10 08:32:50 +0000900 elif isinstance(stdout, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000901 c2pwrite = msvcrt.get_osfhandle(stdout)
902 else:
903 # Assuming file-like object
904 c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
905 c2pwrite = self._make_inheritable(c2pwrite)
906
Peter Astrandd38ddf42005-02-10 08:32:50 +0000907 if stderr is None:
Brian Curtin1ce6b582010-04-24 16:19:22 +0000908 errwrite = _subprocess.GetStdHandle(_subprocess.STD_ERROR_HANDLE)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000909 if errwrite is None:
Brian Curtin1ce6b582010-04-24 16:19:22 +0000910 _, errwrite = _subprocess.CreatePipe(None, 0)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000911 elif stderr == PIPE:
Brian Curtin1ce6b582010-04-24 16:19:22 +0000912 errread, errwrite = _subprocess.CreatePipe(None, 0)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000913 elif stderr == STDOUT:
914 errwrite = c2pwrite
Ross Lagerwallba102ec2011-03-16 18:40:25 +0200915 elif stderr == DEVNULL:
916 errwrite = msvcrt.get_osfhandle(self._get_devnull())
Peter Astrandd38ddf42005-02-10 08:32:50 +0000917 elif isinstance(stderr, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000918 errwrite = msvcrt.get_osfhandle(stderr)
919 else:
920 # Assuming file-like object
921 errwrite = msvcrt.get_osfhandle(stderr.fileno())
922 errwrite = self._make_inheritable(errwrite)
923
924 return (p2cread, p2cwrite,
925 c2pread, c2pwrite,
926 errread, errwrite)
927
928
929 def _make_inheritable(self, handle):
930 """Return a duplicate of handle, which is inheritable"""
Brian Curtin1ce6b582010-04-24 16:19:22 +0000931 return _subprocess.DuplicateHandle(_subprocess.GetCurrentProcess(),
932 handle, _subprocess.GetCurrentProcess(), 0, 1,
933 _subprocess.DUPLICATE_SAME_ACCESS)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000934
935
936 def _find_w9xpopen(self):
937 """Find and return absolut path to w9xpopen.exe"""
Brian Curtin1ce6b582010-04-24 16:19:22 +0000938 w9xpopen = os.path.join(
939 os.path.dirname(_subprocess.GetModuleFileName(0)),
Tim Peterse8374a52004-10-13 03:15:00 +0000940 "w9xpopen.exe")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000941 if not os.path.exists(w9xpopen):
942 # Eeek - file-not-found - possibly an embedding
943 # situation - see if we can locate it in sys.exec_prefix
Tim Peterse8374a52004-10-13 03:15:00 +0000944 w9xpopen = os.path.join(os.path.dirname(sys.exec_prefix),
945 "w9xpopen.exe")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000946 if not os.path.exists(w9xpopen):
Tim Peterse8374a52004-10-13 03:15:00 +0000947 raise RuntimeError("Cannot locate w9xpopen.exe, which is "
948 "needed for Popen to work with your "
949 "shell or platform.")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000950 return w9xpopen
951
Tim Peterse718f612004-10-12 21:51:32 +0000952
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000953 def _execute_child(self, args, executable, preexec_fn, close_fds,
Gregory P. Smithd4cc7bf2010-12-04 11:22:11 +0000954 pass_fds, cwd, env, universal_newlines,
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000955 startupinfo, creationflags, shell,
956 p2cread, p2cwrite,
957 c2pread, c2pwrite,
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000958 errread, errwrite,
959 unused_restore_signals, unused_start_new_session):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000960 """Execute program (MS Windows version)"""
961
Gregory P. Smith8edd99d2010-12-14 13:43:30 +0000962 assert not pass_fds, "pass_fds not supported on Windows."
Gregory P. Smithd4cc7bf2010-12-04 11:22:11 +0000963
Guido van Rossum3172c5d2007-10-16 18:12:55 +0000964 if not isinstance(args, str):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000965 args = list2cmdline(args)
966
Peter Astrandc1d65362004-11-07 14:30:34 +0000967 # Process startup details
Peter Astrandd38ddf42005-02-10 08:32:50 +0000968 if startupinfo is None:
Thomas Wouters73e5a5b2006-06-08 15:35:45 +0000969 startupinfo = STARTUPINFO()
Victor Stinnerb3693582010-05-21 20:13:12 +0000970 if -1 not in (p2cread, c2pwrite, errwrite):
Brian Curtin1ce6b582010-04-24 16:19:22 +0000971 startupinfo.dwFlags |= _subprocess.STARTF_USESTDHANDLES
Peter Astrandc1d65362004-11-07 14:30:34 +0000972 startupinfo.hStdInput = p2cread
973 startupinfo.hStdOutput = c2pwrite
974 startupinfo.hStdError = errwrite
975
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000976 if shell:
Brian Curtin1ce6b582010-04-24 16:19:22 +0000977 startupinfo.dwFlags |= _subprocess.STARTF_USESHOWWINDOW
978 startupinfo.wShowWindow = _subprocess.SW_HIDE
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000979 comspec = os.environ.get("COMSPEC", "cmd.exe")
Tim Golden126c2962010-08-11 14:20:40 +0000980 args = '{} /c "{}"'.format (comspec, args)
Brian Curtin1ce6b582010-04-24 16:19:22 +0000981 if (_subprocess.GetVersion() >= 0x80000000 or
Tim Peterse8374a52004-10-13 03:15:00 +0000982 os.path.basename(comspec).lower() == "command.com"):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000983 # Win9x, or using command.com on NT. We need to
984 # use the w9xpopen intermediate program. For more
985 # information, see KB Q150956
986 # (http://web.archive.org/web/20011105084002/http://support.microsoft.com/support/kb/articles/Q150/9/56.asp)
987 w9xpopen = self._find_w9xpopen()
988 args = '"%s" %s' % (w9xpopen, args)
989 # Not passing CREATE_NEW_CONSOLE has been known to
990 # cause random failures on win9x. Specifically a
991 # dialog: "Your program accessed mem currently in
992 # use at xxx" and a hopeful warning about the
Mark Dickinson934896d2009-02-21 20:59:32 +0000993 # stability of your system. Cost is Ctrl+C won't
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000994 # kill children.
Brian Curtin1ce6b582010-04-24 16:19:22 +0000995 creationflags |= _subprocess.CREATE_NEW_CONSOLE
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000996
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000997 # Start the process
998 try:
Brian Curtin1ce6b582010-04-24 16:19:22 +0000999 hp, ht, pid, tid = _subprocess.CreateProcess(executable, args,
Tim Peterse8374a52004-10-13 03:15:00 +00001000 # no special security
1001 None, None,
Guido van Rossume7ba4952007-06-06 23:52:48 +00001002 int(not close_fds),
Tim Peterse8374a52004-10-13 03:15:00 +00001003 creationflags,
1004 env,
1005 cwd,
1006 startupinfo)
Guido van Rossumb940e112007-01-10 16:19:56 +00001007 except pywintypes.error as e:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001008 # Translate pywintypes.error to WindowsError, which is
1009 # a subclass of OSError. FIXME: We should really
Ezio Melotti13925002011-03-16 11:05:33 +02001010 # translate errno using _sys_errlist (or similar), but
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001011 # how can this be done from Python?
1012 raise WindowsError(*e.args)
Tim Goldenad537f22010-08-08 11:18:16 +00001013 finally:
1014 # Child is launched. Close the parent's copy of those pipe
1015 # handles that only the child should have open. You need
1016 # to make sure that no handles to the write end of the
1017 # output pipe are maintained in this process or else the
1018 # pipe will not close when the child process exits and the
1019 # ReadFile will hang.
1020 if p2cread != -1:
1021 p2cread.Close()
1022 if c2pwrite != -1:
1023 c2pwrite.Close()
1024 if errwrite != -1:
1025 errwrite.Close()
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001026 if hasattr(self, '_devnull'):
1027 os.close(self._devnull)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001028
1029 # Retain the process handle, but close the thread handle
Thomas Wouters49fd7fa2006-04-21 10:40:58 +00001030 self._child_created = True
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001031 self._handle = hp
1032 self.pid = pid
1033 ht.Close()
1034
Brett Cannon84df1e62010-05-14 00:33:40 +00001035 def _internal_poll(self, _deadstate=None,
Victor Stinnerc807a612010-05-14 21:53:45 +00001036 _WaitForSingleObject=_subprocess.WaitForSingleObject,
1037 _WAIT_OBJECT_0=_subprocess.WAIT_OBJECT_0,
1038 _GetExitCodeProcess=_subprocess.GetExitCodeProcess):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001039 """Check if child process has terminated. Returns returncode
Brett Cannon84df1e62010-05-14 00:33:40 +00001040 attribute.
1041
1042 This method is called by __del__, so it can only refer to objects
1043 in its local scope.
1044
1045 """
Peter Astrandd38ddf42005-02-10 08:32:50 +00001046 if self.returncode is None:
Brett Cannon84df1e62010-05-14 00:33:40 +00001047 if _WaitForSingleObject(self._handle, 0) == _WAIT_OBJECT_0:
1048 self.returncode = _GetExitCodeProcess(self._handle)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001049 return self.returncode
1050
1051
Reid Kleckner2b228f02011-03-16 16:57:54 -04001052 def wait(self, timeout=None, endtime=None):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001053 """Wait for child process to terminate. Returns returncode
1054 attribute."""
Reid Kleckner2b228f02011-03-16 16:57:54 -04001055 if endtime is not None:
1056 timeout = self._remaining_time(endtime)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001057 if timeout is None:
Reid Kleckner91156ff2011-03-21 10:06:10 -07001058 timeout_millis = _subprocess.INFINITE
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001059 else:
Reid Kleckner91156ff2011-03-21 10:06:10 -07001060 timeout_millis = int(timeout * 1000)
Peter Astrandd38ddf42005-02-10 08:32:50 +00001061 if self.returncode is None:
Reid Kleckner91156ff2011-03-21 10:06:10 -07001062 result = _subprocess.WaitForSingleObject(self._handle,
1063 timeout_millis)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001064 if result == _subprocess.WAIT_TIMEOUT:
Reid Kleckner2b228f02011-03-16 16:57:54 -04001065 raise TimeoutExpired(self.args, timeout)
Brian Curtin1ce6b582010-04-24 16:19:22 +00001066 self.returncode = _subprocess.GetExitCodeProcess(self._handle)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001067 return self.returncode
1068
1069
1070 def _readerthread(self, fh, buffer):
1071 buffer.append(fh.read())
Victor Stinner667d4b52010-12-25 22:40:32 +00001072 fh.close()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001073
1074
Reid Kleckner2b228f02011-03-16 16:57:54 -04001075 def _communicate(self, input, endtime, orig_timeout):
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001076 # Start reader threads feeding into a list hanging off of this
1077 # object, unless they've already been started.
1078 if self.stdout and not hasattr(self, "_stdout_buff"):
1079 self._stdout_buff = []
1080 self.stdout_thread = \
1081 threading.Thread(target=self._readerthread,
1082 args=(self.stdout, self._stdout_buff))
1083 self.stdout_thread.daemon = True
1084 self.stdout_thread.start()
1085 if self.stderr and not hasattr(self, "_stderr_buff"):
1086 self._stderr_buff = []
1087 self.stderr_thread = \
1088 threading.Thread(target=self._readerthread,
1089 args=(self.stderr, self._stderr_buff))
1090 self.stderr_thread.daemon = True
1091 self.stderr_thread.start()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001092
1093 if self.stdin:
Peter Astrandd38ddf42005-02-10 08:32:50 +00001094 if input is not None:
Ross Lagerwall4f61b022011-04-05 15:34:00 +02001095 try:
1096 self.stdin.write(input)
1097 except IOError as e:
1098 if e.errno != errno.EPIPE:
1099 raise
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001100 self.stdin.close()
1101
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001102 # Wait for the reader threads, or time out. If we time out, the
1103 # threads remain reading and the fds left open in case the user
1104 # calls communicate again.
1105 if self.stdout is not None:
1106 self.stdout_thread.join(self._remaining_time(endtime))
1107 if self.stdout_thread.isAlive():
Reid Kleckner9a67e6c2011-03-20 08:28:07 -07001108 raise TimeoutExpired(self.args, orig_timeout)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001109 if self.stderr is not None:
1110 self.stderr_thread.join(self._remaining_time(endtime))
1111 if self.stderr_thread.isAlive():
Reid Kleckner9a67e6c2011-03-20 08:28:07 -07001112 raise TimeoutExpired(self.args, orig_timeout)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001113
1114 # Collect the output from and close both pipes, now that we know
1115 # both have been read successfully.
1116 stdout = None
1117 stderr = None
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001118 if self.stdout:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001119 stdout = self._stdout_buff
1120 self.stdout.close()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001121 if self.stderr:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001122 stderr = self._stderr_buff
1123 self.stderr.close()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001124
1125 # All data exchanged. Translate lists into strings.
Peter Astrandd38ddf42005-02-10 08:32:50 +00001126 if stdout is not None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001127 stdout = stdout[0]
Peter Astrandd38ddf42005-02-10 08:32:50 +00001128 if stderr is not None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001129 stderr = stderr[0]
1130
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001131 return (stdout, stderr)
1132
Christian Heimesa342c012008-04-20 21:01:16 +00001133 def send_signal(self, sig):
1134 """Send a signal to the process
1135 """
1136 if sig == signal.SIGTERM:
1137 self.terminate()
Brian Curtineb24d742010-04-12 17:16:38 +00001138 elif sig == signal.CTRL_C_EVENT:
1139 os.kill(self.pid, signal.CTRL_C_EVENT)
1140 elif sig == signal.CTRL_BREAK_EVENT:
1141 os.kill(self.pid, signal.CTRL_BREAK_EVENT)
Christian Heimesa342c012008-04-20 21:01:16 +00001142 else:
Brian Curtin19651362010-09-07 13:24:38 +00001143 raise ValueError("Unsupported signal: {}".format(sig))
Christian Heimesa342c012008-04-20 21:01:16 +00001144
1145 def terminate(self):
1146 """Terminates the process
1147 """
Brian Curtin1ce6b582010-04-24 16:19:22 +00001148 _subprocess.TerminateProcess(self._handle, 1)
Christian Heimesa342c012008-04-20 21:01:16 +00001149
1150 kill = terminate
1151
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001152 else:
1153 #
1154 # POSIX methods
1155 #
1156 def _get_handles(self, stdin, stdout, stderr):
Alexandre Vassalotti711ed4a2009-07-17 10:42:05 +00001157 """Construct and return tuple with IO objects:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001158 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
1159 """
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001160 p2cread, p2cwrite = -1, -1
1161 c2pread, c2pwrite = -1, -1
1162 errread, errwrite = -1, -1
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001163
Peter Astrandd38ddf42005-02-10 08:32:50 +00001164 if stdin is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001165 pass
1166 elif stdin == PIPE:
Gregory P. Smith51ee2702010-12-13 07:59:39 +00001167 p2cread, p2cwrite = _create_pipe()
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001168 elif stdin == DEVNULL:
1169 p2cread = self._get_devnull()
Peter Astrandd38ddf42005-02-10 08:32:50 +00001170 elif isinstance(stdin, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001171 p2cread = stdin
1172 else:
1173 # Assuming file-like object
1174 p2cread = stdin.fileno()
1175
Peter Astrandd38ddf42005-02-10 08:32:50 +00001176 if stdout is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001177 pass
1178 elif stdout == PIPE:
Gregory P. Smith51ee2702010-12-13 07:59:39 +00001179 c2pread, c2pwrite = _create_pipe()
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001180 elif stdout == DEVNULL:
1181 c2pwrite = self._get_devnull()
Peter Astrandd38ddf42005-02-10 08:32:50 +00001182 elif isinstance(stdout, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001183 c2pwrite = stdout
1184 else:
1185 # Assuming file-like object
Tim Peterse718f612004-10-12 21:51:32 +00001186 c2pwrite = stdout.fileno()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001187
Peter Astrandd38ddf42005-02-10 08:32:50 +00001188 if stderr is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001189 pass
1190 elif stderr == PIPE:
Gregory P. Smith51ee2702010-12-13 07:59:39 +00001191 errread, errwrite = _create_pipe()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001192 elif stderr == STDOUT:
1193 errwrite = c2pwrite
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001194 elif stderr == DEVNULL:
1195 errwrite = self._get_devnull()
Peter Astrandd38ddf42005-02-10 08:32:50 +00001196 elif isinstance(stderr, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001197 errwrite = stderr
1198 else:
1199 # Assuming file-like object
1200 errwrite = stderr.fileno()
1201
1202 return (p2cread, p2cwrite,
1203 c2pread, c2pwrite,
1204 errread, errwrite)
1205
1206
Antoine Pitrou47f14ba2011-01-03 23:42:01 +00001207 def _close_fds(self, fds_to_keep):
Gregory P. Smithd4cc7bf2010-12-04 11:22:11 +00001208 start_fd = 3
Antoine Pitrou47f14ba2011-01-03 23:42:01 +00001209 for fd in sorted(fds_to_keep):
Gregory P. Smith8edd99d2010-12-14 13:43:30 +00001210 if fd >= start_fd:
Gregory P. Smithd4cc7bf2010-12-04 11:22:11 +00001211 os.closerange(start_fd, fd)
1212 start_fd = fd + 1
1213 if start_fd <= MAXFD:
1214 os.closerange(start_fd, MAXFD)
1215
1216
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001217 def _execute_child(self, args, executable, preexec_fn, close_fds,
Gregory P. Smithd4cc7bf2010-12-04 11:22:11 +00001218 pass_fds, cwd, env, universal_newlines,
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001219 startupinfo, creationflags, shell,
1220 p2cread, p2cwrite,
1221 c2pread, c2pwrite,
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001222 errread, errwrite,
1223 restore_signals, start_new_session):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001224 """Execute program (POSIX version)"""
1225
Victor Stinner7b3b20a2011-03-03 12:54:05 +00001226 if isinstance(args, (str, bytes)):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001227 args = [args]
Thomas Wouters89f507f2006-12-13 04:49:30 +00001228 else:
1229 args = list(args)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001230
1231 if shell:
1232 args = ["/bin/sh", "-c"] + args
Stefan Krah9542cc62010-07-19 14:20:53 +00001233 if executable:
1234 args[0] = executable
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001235
Peter Astrandd38ddf42005-02-10 08:32:50 +00001236 if executable is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001237 executable = args[0]
1238
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001239 # For transferring possible exec failure from child to parent.
1240 # Data format: "exception name:hex errno:description"
1241 # Pickle is not used; it is complex and involves memory allocation.
Gregory P. Smith51ee2702010-12-13 07:59:39 +00001242 errpipe_read, errpipe_write = _create_pipe()
Christian Heimesfdab48e2008-01-20 09:06:41 +00001243 try:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001244 try:
Gregory P. Smith59fd1bf2011-05-28 09:32:39 -07001245 # We must avoid complex work that could involve
1246 # malloc or free in the child process to avoid
1247 # potential deadlocks, thus we do all this here.
1248 # and pass it to fork_exec()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001249
Gregory P. Smith59fd1bf2011-05-28 09:32:39 -07001250 if env:
1251 env_list = [os.fsencode(k) + b'=' + os.fsencode(v)
1252 for k, v in env.items()]
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001253 else:
Gregory P. Smith59fd1bf2011-05-28 09:32:39 -07001254 env_list = None # Use execv instead of execve.
1255 executable = os.fsencode(executable)
1256 if os.path.dirname(executable):
1257 executable_list = (executable,)
1258 else:
1259 # This matches the behavior of os._execvpe().
1260 executable_list = tuple(
1261 os.path.join(os.fsencode(dir), executable)
1262 for dir in os.get_exec_path(env))
1263 fds_to_keep = set(pass_fds)
1264 fds_to_keep.add(errpipe_write)
1265 self.pid = _posixsubprocess.fork_exec(
1266 args, executable_list,
1267 close_fds, sorted(fds_to_keep), cwd, env_list,
1268 p2cread, p2cwrite, c2pread, c2pwrite,
1269 errread, errwrite,
1270 errpipe_read, errpipe_write,
1271 restore_signals, start_new_session, preexec_fn)
Facundo Batista10706e22009-06-19 20:34:30 +00001272 finally:
1273 # be sure the FD is closed no matter what
1274 os.close(errpipe_write)
1275
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001276 if p2cread != -1 and p2cwrite != -1:
Facundo Batista10706e22009-06-19 20:34:30 +00001277 os.close(p2cread)
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001278 if c2pwrite != -1 and c2pread != -1:
Facundo Batista10706e22009-06-19 20:34:30 +00001279 os.close(c2pwrite)
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001280 if errwrite != -1 and errread != -1:
Facundo Batista10706e22009-06-19 20:34:30 +00001281 os.close(errwrite)
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001282 if hasattr(self, '_devnull'):
1283 os.close(self._devnull)
Facundo Batista10706e22009-06-19 20:34:30 +00001284
1285 # Wait for exec to fail or succeed; possibly raising an
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001286 # exception (limited in size)
1287 data = bytearray()
1288 while True:
1289 part = _eintr_retry_call(os.read, errpipe_read, 50000)
1290 data += part
1291 if not part or len(data) > 50000:
1292 break
Facundo Batista10706e22009-06-19 20:34:30 +00001293 finally:
1294 # be sure the FD is closed no matter what
1295 os.close(errpipe_read)
1296
Guido van Rossumaf2362a2007-05-15 22:32:02 +00001297 if data:
Gregory P. Smithe85db2b2010-12-14 14:38:00 +00001298 try:
1299 _eintr_retry_call(os.waitpid, self.pid, 0)
1300 except OSError as e:
1301 if e.errno != errno.ECHILD:
1302 raise
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001303 try:
1304 exception_name, hex_errno, err_msg = data.split(b':', 2)
1305 except ValueError:
1306 print('Bad exception data:', repr(data))
1307 exception_name = b'RuntimeError'
1308 hex_errno = b'0'
1309 err_msg = b'Unknown'
1310 child_exception_type = getattr(
1311 builtins, exception_name.decode('ascii'),
1312 RuntimeError)
Benjamin Petersond75fcb42009-02-19 04:22:03 +00001313 for fd in (p2cwrite, c2pread, errread):
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001314 if fd != -1:
Benjamin Petersond75fcb42009-02-19 04:22:03 +00001315 os.close(fd)
Victor Stinner4d078042010-04-23 19:28:32 +00001316 err_msg = err_msg.decode(errors="surrogatepass")
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001317 if issubclass(child_exception_type, OSError) and hex_errno:
Benjamin Petersonb8bc4392010-11-20 18:24:54 +00001318 errno_num = int(hex_errno, 16)
1319 if errno_num != 0:
1320 err_msg = os.strerror(errno_num)
1321 if errno_num == errno.ENOENT:
Benjamin Peterson5f780402010-11-20 18:07:52 +00001322 err_msg += ': ' + repr(args[0])
Benjamin Petersonb8bc4392010-11-20 18:24:54 +00001323 raise child_exception_type(errno_num, err_msg)
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001324 raise child_exception_type(err_msg)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001325
1326
Brett Cannon84df1e62010-05-14 00:33:40 +00001327 def _handle_exitstatus(self, sts, _WIFSIGNALED=os.WIFSIGNALED,
1328 _WTERMSIG=os.WTERMSIG, _WIFEXITED=os.WIFEXITED,
1329 _WEXITSTATUS=os.WEXITSTATUS):
1330 # This method is called (indirectly) by __del__, so it cannot
1331 # refer to anything outside of its local scope."""
1332 if _WIFSIGNALED(sts):
1333 self.returncode = -_WTERMSIG(sts)
1334 elif _WIFEXITED(sts):
1335 self.returncode = _WEXITSTATUS(sts)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001336 else:
1337 # Should never happen
1338 raise RuntimeError("Unknown child exit status!")
1339
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001340
Brett Cannon84df1e62010-05-14 00:33:40 +00001341 def _internal_poll(self, _deadstate=None, _waitpid=os.waitpid,
1342 _WNOHANG=os.WNOHANG, _os_error=os.error):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001343 """Check if child process has terminated. Returns returncode
Brett Cannon84df1e62010-05-14 00:33:40 +00001344 attribute.
1345
1346 This method is called by __del__, so it cannot reference anything
1347 outside of the local scope (nor can any methods it calls).
1348
1349 """
Peter Astrandd38ddf42005-02-10 08:32:50 +00001350 if self.returncode is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001351 try:
Brett Cannon84df1e62010-05-14 00:33:40 +00001352 pid, sts = _waitpid(self.pid, _WNOHANG)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001353 if pid == self.pid:
1354 self._handle_exitstatus(sts)
Brett Cannon84df1e62010-05-14 00:33:40 +00001355 except _os_error:
Thomas Wouters49fd7fa2006-04-21 10:40:58 +00001356 if _deadstate is not None:
1357 self.returncode = _deadstate
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001358 return self.returncode
1359
1360
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001361 def _try_wait(self, wait_flags):
1362 try:
1363 (pid, sts) = _eintr_retry_call(os.waitpid, self.pid, wait_flags)
1364 except OSError as e:
1365 if e.errno != errno.ECHILD:
1366 raise
1367 # This happens if SIGCLD is set to be ignored or waiting
1368 # for child processes has otherwise been disabled for our
1369 # process. This child is dead, we can't get the status.
1370 pid = self.pid
1371 sts = 0
1372 return (pid, sts)
1373
1374
1375 def wait(self, timeout=None, endtime=None):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001376 """Wait for child process to terminate. Returns returncode
1377 attribute."""
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001378 if self.returncode is not None:
1379 return self.returncode
Reid Kleckner2b228f02011-03-16 16:57:54 -04001380
1381 # endtime is preferred to timeout. timeout is only used for
1382 # printing.
1383 if endtime is not None or timeout is not None:
1384 if endtime is None:
1385 endtime = time.time() + timeout
1386 elif timeout is None:
1387 timeout = self._remaining_time(endtime)
1388
1389 if endtime is not None:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001390 # Enter a busy loop if we have a timeout. This busy loop was
1391 # cribbed from Lib/threading.py in Thread.wait() at r71065.
1392 delay = 0.0005 # 500 us -> initial delay of 1 ms
1393 while True:
1394 (pid, sts) = self._try_wait(os.WNOHANG)
1395 assert pid == self.pid or pid == 0
1396 if pid == self.pid:
1397 self._handle_exitstatus(sts)
1398 break
1399 remaining = self._remaining_time(endtime)
1400 if remaining <= 0:
Reid Kleckner2b228f02011-03-16 16:57:54 -04001401 raise TimeoutExpired(self.args, timeout)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001402 delay = min(delay * 2, remaining, .05)
1403 time.sleep(delay)
1404 elif self.returncode is None:
1405 (pid, sts) = self._try_wait(0)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001406 self._handle_exitstatus(sts)
1407 return self.returncode
1408
1409
Reid Kleckner2b228f02011-03-16 16:57:54 -04001410 def _communicate(self, input, endtime, orig_timeout):
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001411 if self.stdin and not self._communication_started:
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001412 # Flush stdio buffer. This might block, if the user has
1413 # been writing to .stdin in an uncontrolled fashion.
1414 self.stdin.flush()
1415 if not input:
1416 self.stdin.close()
1417
1418 if _has_poll:
Reid Kleckner2b228f02011-03-16 16:57:54 -04001419 stdout, stderr = self._communicate_with_poll(input, endtime,
1420 orig_timeout)
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001421 else:
Reid Kleckner2b228f02011-03-16 16:57:54 -04001422 stdout, stderr = self._communicate_with_select(input, endtime,
1423 orig_timeout)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001424
1425 self.wait(timeout=self._remaining_time(endtime))
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001426
1427 # All data exchanged. Translate lists into strings.
1428 if stdout is not None:
1429 stdout = b''.join(stdout)
1430 if stderr is not None:
1431 stderr = b''.join(stderr)
1432
1433 # Translate newlines, if requested.
1434 # This also turns bytes into strings.
1435 if self.universal_newlines:
1436 if stdout is not None:
1437 stdout = self._translate_newlines(stdout,
1438 self.stdout.encoding)
1439 if stderr is not None:
1440 stderr = self._translate_newlines(stderr,
1441 self.stderr.encoding)
1442
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001443 return (stdout, stderr)
1444
1445
Reid Kleckner2b228f02011-03-16 16:57:54 -04001446 def _communicate_with_poll(self, input, endtime, orig_timeout):
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001447 stdout = None # Return
1448 stderr = None # Return
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001449
1450 if not self._communication_started:
1451 self._fd2file = {}
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001452
1453 poller = select.poll()
1454 def register_and_append(file_obj, eventmask):
1455 poller.register(file_obj.fileno(), eventmask)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001456 self._fd2file[file_obj.fileno()] = file_obj
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001457
1458 def close_unregister_and_remove(fd):
1459 poller.unregister(fd)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001460 self._fd2file[fd].close()
1461 self._fd2file.pop(fd)
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001462
1463 if self.stdin and input:
1464 register_and_append(self.stdin, select.POLLOUT)
1465
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001466 # Only create this mapping if we haven't already.
1467 if not self._communication_started:
1468 self._fd2output = {}
1469 if self.stdout:
1470 self._fd2output[self.stdout.fileno()] = []
1471 if self.stderr:
1472 self._fd2output[self.stderr.fileno()] = []
1473
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001474 select_POLLIN_POLLPRI = select.POLLIN | select.POLLPRI
1475 if self.stdout:
1476 register_and_append(self.stdout, select_POLLIN_POLLPRI)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001477 stdout = self._fd2output[self.stdout.fileno()]
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001478 if self.stderr:
1479 register_and_append(self.stderr, select_POLLIN_POLLPRI)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001480 stderr = self._fd2output[self.stderr.fileno()]
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001481
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001482 # Save the input here so that if we time out while communicating,
1483 # we can continue sending input if we retry.
1484 if self.stdin and self._input is None:
1485 self._input_offset = 0
1486 self._input = input
1487 if self.universal_newlines:
1488 self._input = self._input.encode(self.stdin.encoding)
1489
1490 while self._fd2file:
Victor Stinner7a8d0812011-04-05 13:13:08 +02001491 timeout = self._remaining_time(endtime)
1492 if timeout is not None and timeout < 0:
1493 raise TimeoutExpired(self.args, orig_timeout)
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001494 try:
Victor Stinner7a8d0812011-04-05 13:13:08 +02001495 ready = poller.poll(timeout)
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001496 except select.error as e:
1497 if e.args[0] == errno.EINTR:
1498 continue
1499 raise
Reid Kleckner2b228f02011-03-16 16:57:54 -04001500 self._check_timeout(endtime, orig_timeout)
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001501
1502 # XXX Rewrite these to use non-blocking I/O on the
1503 # file objects; they are no longer using C stdio!
1504
1505 for fd, mode in ready:
1506 if mode & select.POLLOUT:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001507 chunk = self._input[self._input_offset :
1508 self._input_offset + _PIPE_BUF]
Ross Lagerwall4f61b022011-04-05 15:34:00 +02001509 try:
Ross Lagerwall0b9ea932011-04-05 16:07:49 +02001510 self._input_offset += os.write(fd, chunk)
Ross Lagerwall4f61b022011-04-05 15:34:00 +02001511 except OSError as e:
1512 if e.errno == errno.EPIPE:
1513 close_unregister_and_remove(fd)
1514 else:
1515 raise
1516 else:
Ross Lagerwall0b9ea932011-04-05 16:07:49 +02001517 if self._input_offset >= len(self._input):
Ross Lagerwall4f61b022011-04-05 15:34:00 +02001518 close_unregister_and_remove(fd)
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001519 elif mode & select_POLLIN_POLLPRI:
1520 data = os.read(fd, 4096)
1521 if not data:
1522 close_unregister_and_remove(fd)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001523 self._fd2output[fd].append(data)
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001524 else:
1525 # Ignore hang up or errors.
1526 close_unregister_and_remove(fd)
1527
1528 return (stdout, stderr)
1529
1530
Reid Kleckner2b228f02011-03-16 16:57:54 -04001531 def _communicate_with_select(self, input, endtime, orig_timeout):
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001532 if not self._communication_started:
1533 self._read_set = []
1534 self._write_set = []
1535 if self.stdin and input:
1536 self._write_set.append(self.stdin)
1537 if self.stdout:
1538 self._read_set.append(self.stdout)
1539 if self.stderr:
1540 self._read_set.append(self.stderr)
1541
1542 if self.stdin and self._input is None:
1543 self._input_offset = 0
1544 self._input = input
1545 if self.universal_newlines:
1546 self._input = self._input.encode(self.stdin.encoding)
1547
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001548 stdout = None # Return
1549 stderr = None # Return
1550
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001551 if self.stdout:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001552 if not self._communication_started:
1553 self._stdout_buff = []
1554 stdout = self._stdout_buff
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001555 if self.stderr:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001556 if not self._communication_started:
1557 self._stderr_buff = []
1558 stderr = self._stderr_buff
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001559
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001560 while self._read_set or self._write_set:
Victor Stinner7a8d0812011-04-05 13:13:08 +02001561 timeout = self._remaining_time(endtime)
1562 if timeout is not None and timeout < 0:
1563 raise TimeoutExpired(self.args, orig_timeout)
Georg Brandl86b2fb92008-07-16 03:43:04 +00001564 try:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001565 (rlist, wlist, xlist) = \
1566 select.select(self._read_set, self._write_set, [],
Victor Stinner7a8d0812011-04-05 13:13:08 +02001567 timeout)
Georg Brandl86b2fb92008-07-16 03:43:04 +00001568 except select.error as e:
1569 if e.args[0] == errno.EINTR:
1570 continue
1571 raise
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001572
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001573 # According to the docs, returning three empty lists indicates
1574 # that the timeout expired.
1575 if not (rlist or wlist or xlist):
Reid Kleckner2b228f02011-03-16 16:57:54 -04001576 raise TimeoutExpired(self.args, orig_timeout)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001577 # We also check what time it is ourselves for good measure.
Reid Kleckner2b228f02011-03-16 16:57:54 -04001578 self._check_timeout(endtime, orig_timeout)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001579
Guido van Rossum98297ee2007-11-06 21:34:58 +00001580 # XXX Rewrite these to use non-blocking I/O on the
1581 # file objects; they are no longer using C stdio!
1582
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001583 if self.stdin in wlist:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001584 chunk = self._input[self._input_offset :
1585 self._input_offset + _PIPE_BUF]
Ross Lagerwall4f61b022011-04-05 15:34:00 +02001586 try:
1587 bytes_written = os.write(self.stdin.fileno(), chunk)
1588 except OSError as e:
1589 if e.errno == errno.EPIPE:
1590 self.stdin.close()
Ross Lagerwall0b9ea932011-04-05 16:07:49 +02001591 self._write_set.remove(self.stdin)
Ross Lagerwall4f61b022011-04-05 15:34:00 +02001592 else:
1593 raise
1594 else:
Ross Lagerwall0b9ea932011-04-05 16:07:49 +02001595 self._input_offset += bytes_written
1596 if self._input_offset >= len(self._input):
Ross Lagerwall4f61b022011-04-05 15:34:00 +02001597 self.stdin.close()
Ross Lagerwall0b9ea932011-04-05 16:07:49 +02001598 self._write_set.remove(self.stdin)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001599
1600 if self.stdout in rlist:
1601 data = os.read(self.stdout.fileno(), 1024)
Guido van Rossumc9e363c2007-05-15 23:18:55 +00001602 if not data:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001603 self.stdout.close()
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001604 self._read_set.remove(self.stdout)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001605 stdout.append(data)
1606
1607 if self.stderr in rlist:
1608 data = os.read(self.stderr.fileno(), 1024)
Guido van Rossumc9e363c2007-05-15 23:18:55 +00001609 if not data:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001610 self.stderr.close()
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001611 self._read_set.remove(self.stderr)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001612 stderr.append(data)
1613
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001614 return (stdout, stderr)
1615
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001616
Christian Heimesa342c012008-04-20 21:01:16 +00001617 def send_signal(self, sig):
1618 """Send a signal to the process
1619 """
1620 os.kill(self.pid, sig)
1621
1622 def terminate(self):
1623 """Terminate the process with SIGTERM
1624 """
1625 self.send_signal(signal.SIGTERM)
1626
1627 def kill(self):
1628 """Kill the process with SIGKILL
1629 """
1630 self.send_signal(signal.SIGKILL)