blob: 4d316e622659357ad66a57047ea952412da4c34e [file] [log] [blame]
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001# subprocess - Subprocesses with accessible I/O streams
2#
Tim Peterse718f612004-10-12 21:51:32 +00003# For more information about this module, see PEP 324.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00004#
Peter Astrand3a708df2005-09-23 17:37:29 +00005# Copyright (c) 2003-2005 by Peter Astrand <astrand@lysator.liu.se>
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00006#
Peter Astrand69bf13f2005-02-14 08:56:32 +00007# Licensed to PSF under a Contributor Agreement.
Peter Astrand3a708df2005-09-23 17:37:29 +00008# See http://www.python.org/2.4/license for licensing details.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00009
Raymond Hettinger837dd932004-10-17 16:36:53 +000010r"""subprocess - Subprocesses with accessible I/O streams
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000011
Fredrik Lundh15aaacc2004-10-17 14:47:05 +000012This module allows you to spawn processes, connect to their
13input/output/error pipes, and obtain their return codes. This module
Benjamin Peterson5eea8a72014-03-12 21:41:35 -050014intends to replace several older modules and functions:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000015
16os.system
17os.spawn*
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000018
19Information about how the subprocess module can be used to replace these
20modules and functions can be found below.
21
22
23
24Using the subprocess module
25===========================
26This module defines one class called Popen:
27
Gregory P. Smitha1ed5392013-03-23 11:44:25 -070028class Popen(args, bufsize=-1, executable=None,
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000029 stdin=None, stdout=None, stderr=None,
Gregory P. Smith8edd99d2010-12-14 13:43:30 +000030 preexec_fn=None, close_fds=True, shell=False,
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000031 cwd=None, env=None, universal_newlines=False,
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +000032 startupinfo=None, creationflags=0,
Gregory P. Smith8edd99d2010-12-14 13:43:30 +000033 restore_signals=True, start_new_session=False, pass_fds=()):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000034
35
36Arguments are:
37
38args should be a string, or a sequence of program arguments. The
39program to execute is normally the first item in the args sequence or
40string, but can be explicitly set by using the executable argument.
41
Gregory P. Smithf5604852010-12-13 06:45:02 +000042On POSIX, with shell=False (default): In this case, the Popen class
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000043uses os.execvp() to execute the child program. args should normally
44be a sequence. A string will be treated as a sequence with the string
45as the only item (the program to execute).
46
Gregory P. Smithf5604852010-12-13 06:45:02 +000047On POSIX, with shell=True: If args is a string, it specifies the
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000048command string to execute through the shell. If args is a sequence,
49the first item specifies the command string, and any additional items
50will be treated as additional shell arguments.
51
52On Windows: the Popen class uses CreateProcess() to execute the child
53program, which operates on strings. If args is a sequence, it will be
54converted to a string using the list2cmdline method. Please note that
55not all MS Windows applications interpret the command line the same
56way: The list2cmdline is designed for applications using the same
57rules as the MS C runtime.
58
Gregory P. Smitha1ed5392013-03-23 11:44:25 -070059bufsize will be supplied as the corresponding argument to the io.open()
60function when creating the stdin/stdout/stderr pipe file objects:
610 means unbuffered (read & write are one system call and can return short),
621 means line buffered, any other positive value means use a buffer of
63approximately that size. A negative bufsize, the default, means the system
64default of io.DEFAULT_BUFFER_SIZE will be used.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000065
66stdin, stdout and stderr specify the executed programs' standard
67input, standard output and standard error file handles, respectively.
68Valid values are PIPE, an existing file descriptor (a positive
69integer), an existing file object, and None. PIPE indicates that a
70new pipe to the child should be created. With None, no redirection
71will occur; the child's file handles will be inherited from the
72parent. Additionally, stderr can be STDOUT, which indicates that the
73stderr data from the applications should be captured into the same
74file handle as for stdout.
75
Gregory P. Smithf5604852010-12-13 06:45:02 +000076On POSIX, if preexec_fn is set to a callable object, this object will be
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +000077called in the child process just before the child is executed. The use
78of preexec_fn is not thread safe, using it in the presence of threads
79could lead to a deadlock in the child process before the new executable
80is executed.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000081
82If close_fds is true, all file descriptors except 0, 1 and 2 will be
Gregory P. Smithf5604852010-12-13 06:45:02 +000083closed before the child process is executed. The default for close_fds
Gregory P. Smith8edd99d2010-12-14 13:43:30 +000084varies by platform: Always true on POSIX. True when stdin/stdout/stderr
85are None on Windows, false otherwise.
86
87pass_fds is an optional sequence of file descriptors to keep open between the
88parent and child. Providing any pass_fds implicitly sets close_fds to true.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +000089
90if shell is true, the specified command will be executed through the
91shell.
92
93If cwd is not None, the current directory will be changed to cwd
94before the child is executed.
95
Gregory P. Smithf5604852010-12-13 06:45:02 +000096On POSIX, if restore_signals is True all signals that Python sets to
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +000097SIG_IGN are restored to SIG_DFL in the child process before the exec.
98Currently this includes the SIGPIPE, SIGXFZ and SIGXFSZ signals. This
99parameter does nothing on Windows.
100
Gregory P. Smithf5604852010-12-13 06:45:02 +0000101On POSIX, if start_new_session is True, the setsid() system call will be made
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000102in the child process prior to executing the command.
103
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000104If env is not None, it defines the environment variables for the new
105process.
106
Andrew Kuchling4f7b0c32014-04-14 15:08:18 -0400107If universal_newlines is False, the file objects stdin, stdout and stderr
Ronald Oussoren385521c2013-07-07 09:26:45 +0200108are opened as binary files, and no line ending conversion is done.
109
Andrew Kuchling4f7b0c32014-04-14 15:08:18 -0400110If universal_newlines is True, the file objects stdout and stderr are
111opened as a text file, but lines may be terminated by any of '\n',
Gregory P. Smithf5604852010-12-13 06:45:02 +0000112the Unix end-of-line convention, '\r', the old Macintosh convention or
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000113'\r\n', the Windows convention. All of these external representations
Gregory P. Smith1f8a40b2013-03-20 18:32:03 -0700114are seen as '\n' by the Python program. Also, the newlines attribute
115of the file objects stdout, stdin and stderr are not updated by the
116communicate() method.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000117
Andrew Kuchling4f7b0c32014-04-14 15:08:18 -0400118In either case, the process being communicated with should start up
119expecting to receive bytes on its standard input and decode them with
120the same encoding they are sent in.
121
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000122The startupinfo and creationflags, if given, will be passed to the
123underlying CreateProcess() function. They can specify things such as
124appearance of the main window and priority for the new process.
125(Windows only)
126
127
Georg Brandlf9734072008-12-07 15:30:06 +0000128This module also defines some shortcut functions:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000129
Peter Astrand5f5e1412004-12-05 20:15:36 +0000130call(*popenargs, **kwargs):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000131 Run command with arguments. Wait for command to complete, then
132 return the returncode attribute.
133
134 The arguments are the same as for the Popen constructor. Example:
135
Florent Xicluna4886d242010-03-08 13:27:26 +0000136 >>> retcode = subprocess.call(["ls", "-l"])
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000137
Peter Astrand454f7672005-01-01 09:36:35 +0000138check_call(*popenargs, **kwargs):
139 Run command with arguments. Wait for command to complete. If the
140 exit code was zero then return, otherwise raise
141 CalledProcessError. The CalledProcessError object will have the
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000142 return code in the returncode attribute.
Peter Astrand454f7672005-01-01 09:36:35 +0000143
144 The arguments are the same as for the Popen constructor. Example:
145
Florent Xicluna4886d242010-03-08 13:27:26 +0000146 >>> subprocess.check_call(["ls", "-l"])
Georg Brandl2708f3a2009-12-20 14:38:23 +0000147 0
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000148
Brett Cannona23810f2008-05-26 19:04:21 +0000149getstatusoutput(cmd):
150 Return (status, output) of executing cmd in a shell.
151
Tim Golden60798142013-11-05 12:57:25 +0000152 Execute the string 'cmd' in a shell with 'check_output' and
153 return a 2-tuple (status, output). Universal newlines mode is used,
154 meaning that the result with be decoded to a string.
155
156 A trailing newline is stripped from the output.
157 The exit status for the command can be interpreted
158 according to the rules for the function 'wait'. Example:
Brett Cannona23810f2008-05-26 19:04:21 +0000159
Brett Cannona23810f2008-05-26 19:04:21 +0000160 >>> subprocess.getstatusoutput('ls /bin/ls')
161 (0, '/bin/ls')
162 >>> subprocess.getstatusoutput('cat /bin/junk')
163 (256, 'cat: /bin/junk: No such file or directory')
164 >>> subprocess.getstatusoutput('/bin/junk')
165 (256, 'sh: /bin/junk: not found')
166
167getoutput(cmd):
168 Return output (stdout or stderr) of executing cmd in a shell.
169
170 Like getstatusoutput(), except the exit status is ignored and the return
171 value is a string containing the command's output. Example:
172
Brett Cannona23810f2008-05-26 19:04:21 +0000173 >>> subprocess.getoutput('ls /bin/ls')
174 '/bin/ls'
175
Georg Brandlf9734072008-12-07 15:30:06 +0000176check_output(*popenargs, **kwargs):
Gregory P. Smith91110f52013-03-19 23:25:16 -0700177 Run command with arguments and return its output.
Georg Brandlf9734072008-12-07 15:30:06 +0000178
Georg Brandl2708f3a2009-12-20 14:38:23 +0000179 If the exit code was non-zero it raises a CalledProcessError. The
180 CalledProcessError object will have the return code in the returncode
181 attribute and output in the output attribute.
Georg Brandlf9734072008-12-07 15:30:06 +0000182
Georg Brandl2708f3a2009-12-20 14:38:23 +0000183 The arguments are the same as for the Popen constructor. Example:
Georg Brandlf9734072008-12-07 15:30:06 +0000184
Georg Brandl2708f3a2009-12-20 14:38:23 +0000185 >>> output = subprocess.check_output(["ls", "-l", "/dev/null"])
Georg Brandlf9734072008-12-07 15:30:06 +0000186
Serhiy Storchakafcd9f222013-04-22 20:20:54 +0300187 There is an additional optional argument, "input", allowing you to
188 pass a string to the subprocess's stdin. If you use this argument
189 you may not also use the Popen constructor's "stdin" argument.
Brett Cannona23810f2008-05-26 19:04:21 +0000190
Andrew Kuchling4f7b0c32014-04-14 15:08:18 -0400191 If universal_newlines is set to True, the "input" argument must
192 be a string rather than bytes, and the return value will be a string.
193
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000194Exceptions
195----------
196Exceptions raised in the child process, before the new program has
197started to execute, will be re-raised in the parent. Additionally,
198the exception object will have one extra attribute called
199'child_traceback', which is a string containing traceback information
Ezio Melotti30b9d5d2013-08-17 15:50:46 +0300200from the child's point of view.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000201
202The most common exception raised is OSError. This occurs, for
203example, when trying to execute a non-existent file. Applications
204should prepare for OSErrors.
205
206A ValueError will be raised if Popen is called with invalid arguments.
207
Gregory P. Smith54d412e2011-03-14 14:08:43 -0400208Exceptions defined within this module inherit from SubprocessError.
209check_call() and check_output() will raise CalledProcessError if the
Gregory P. Smithb4039aa2011-03-14 14:16:20 -0400210called process returns a non-zero return code. TimeoutExpired
Gregory P. Smith54d412e2011-03-14 14:08:43 -0400211be raised if a timeout was specified and expired.
Peter Astrand454f7672005-01-01 09:36:35 +0000212
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000213
214Security
215--------
216Unlike some other popen functions, this implementation will never call
217/bin/sh implicitly. This means that all characters, including shell
218metacharacters, can safely be passed to child processes.
219
220
221Popen objects
222=============
223Instances of the Popen class have the following methods:
224
225poll()
226 Check if child process has terminated. Returns returncode
227 attribute.
228
229wait()
230 Wait for child process to terminate. Returns returncode attribute.
231
232communicate(input=None)
233 Interact with process: Send data to stdin. Read data from stdout
234 and stderr, until end-of-file is reached. Wait for process to
Andrew Kuchling4f7b0c32014-04-14 15:08:18 -0400235 terminate. The optional input argument should be data to be
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000236 sent to the child process, or None, if no data should be sent to
Andrew Kuchling4f7b0c32014-04-14 15:08:18 -0400237 the child. If the Popen instance was constructed with universal_newlines
238 set to True, the input argument should be a string and will be encoded
239 using the preferred system encoding (see locale.getpreferredencoding);
240 if universal_newlines is False, the input argument should be a
241 byte string.
Tim Peterse718f612004-10-12 21:51:32 +0000242
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000243 communicate() returns a tuple (stdout, stderr).
244
245 Note: The data read is buffered in memory, so do not use this
246 method if the data size is large or unlimited.
247
248The following attributes are also available:
249
250stdin
251 If the stdin argument is PIPE, this attribute is a file object
252 that provides input to the child process. Otherwise, it is None.
253
254stdout
255 If the stdout argument is PIPE, this attribute is a file object
256 that provides output from the child process. Otherwise, it is
257 None.
258
259stderr
260 If the stderr argument is PIPE, this attribute is file object that
261 provides error output from the child process. Otherwise, it is
262 None.
263
264pid
265 The process ID of the child process.
266
267returncode
268 The child return code. A None value indicates that the process
269 hasn't terminated yet. A negative value -N indicates that the
Gregory P. Smithf5604852010-12-13 06:45:02 +0000270 child was terminated by signal N (POSIX only).
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000271
272
273Replacing older functions with the subprocess module
274====================================================
275In this section, "a ==> b" means that b can be used as a replacement
276for a.
277
278Note: All functions in this section fail (more or less) silently if
279the executed program cannot be found; this module raises an OSError
280exception.
281
282In the following examples, we assume that the subprocess module is
283imported with "from subprocess import *".
284
285
286Replacing /bin/sh shell backquote
287---------------------------------
288output=`mycmd myarg`
289==>
290output = Popen(["mycmd", "myarg"], stdout=PIPE).communicate()[0]
291
292
293Replacing shell pipe line
294-------------------------
295output=`dmesg | grep hda`
296==>
297p1 = Popen(["dmesg"], stdout=PIPE)
Peter Astrand6fdf3cb2004-11-30 18:06:42 +0000298p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000299output = p2.communicate()[0]
300
301
302Replacing os.system()
303---------------------
304sts = os.system("mycmd" + " myarg")
305==>
306p = Popen("mycmd" + " myarg", shell=True)
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000307pid, sts = os.waitpid(p.pid, 0)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000308
309Note:
310
311* Calling the program through the shell is usually not required.
312
313* It's easier to look at the returncode attribute than the
314 exitstatus.
315
316A more real-world example would look like this:
317
318try:
319 retcode = call("mycmd" + " myarg", shell=True)
320 if retcode < 0:
Guido van Rossumc2f93dc2007-05-24 00:50:02 +0000321 print("Child was terminated by signal", -retcode, file=sys.stderr)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000322 else:
Guido van Rossumc2f93dc2007-05-24 00:50:02 +0000323 print("Child returned", retcode, file=sys.stderr)
324except OSError as e:
325 print("Execution failed:", e, file=sys.stderr)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000326
327
328Replacing os.spawn*
329-------------------
Tim Peterse718f612004-10-12 21:51:32 +0000330P_NOWAIT example:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000331
332pid = os.spawnlp(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg")
333==>
334pid = Popen(["/bin/mycmd", "myarg"]).pid
335
336
337P_WAIT example:
338
339retcode = os.spawnlp(os.P_WAIT, "/bin/mycmd", "mycmd", "myarg")
340==>
341retcode = call(["/bin/mycmd", "myarg"])
342
343
Tim Peterse718f612004-10-12 21:51:32 +0000344Vector example:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000345
346os.spawnvp(os.P_NOWAIT, path, args)
347==>
348Popen([path] + args[1:])
349
350
Tim Peterse718f612004-10-12 21:51:32 +0000351Environment example:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000352
353os.spawnlpe(os.P_NOWAIT, "/bin/mycmd", "mycmd", "myarg", env)
354==>
355Popen(["/bin/mycmd", "myarg"], env={"PATH": "/usr/bin"})
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000356"""
357
358import sys
Gregory P. Smithcb6fdf22015-04-07 16:11:33 -0700359_mswindows = (sys.platform == "win32")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000360
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000361import io
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000362import os
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400363import time
Christian Heimesa342c012008-04-20 21:01:16 +0000364import signal
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000365import builtins
Gregory P. Smithd23047b2010-12-04 09:10:44 +0000366import warnings
Ross Lagerwall4f61b022011-04-05 15:34:00 +0200367import errno
Victor Stinnerae586492014-09-02 23:18:25 +0200368from time import monotonic as _time
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000369
Peter Astrand454f7672005-01-01 09:36:35 +0000370# Exception classes used by this module.
Gregory P. Smith54d412e2011-03-14 14:08:43 -0400371class SubprocessError(Exception): pass
372
373
374class CalledProcessError(SubprocessError):
Georg Brandlf9734072008-12-07 15:30:06 +0000375 """This exception is raised when a process run by check_call() or
376 check_output() returns a non-zero exit status.
377 The exit status will be stored in the returncode attribute;
378 check_output() will also store the output in the output attribute.
379 """
Gregory P. Smith6e730002015-04-14 16:14:25 -0700380 def __init__(self, returncode, cmd, output=None, stderr=None):
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000381 self.returncode = returncode
382 self.cmd = cmd
Georg Brandlf9734072008-12-07 15:30:06 +0000383 self.output = output
Gregory P. Smith6e730002015-04-14 16:14:25 -0700384 self.stderr = stderr
385
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000386 def __str__(self):
387 return "Command '%s' returned non-zero exit status %d" % (self.cmd, self.returncode)
388
Gregory P. Smith6e730002015-04-14 16:14:25 -0700389 @property
390 def stdout(self):
391 """Alias for output attribute, to match stderr"""
392 return self.output
393
394 @stdout.setter
395 def stdout(self, value):
396 # There's no obvious reason to set this, but allow it anyway so
397 # .stdout is a transparent alias for .output
398 self.output = value
399
Peter Astrand454f7672005-01-01 09:36:35 +0000400
Gregory P. Smith54d412e2011-03-14 14:08:43 -0400401class TimeoutExpired(SubprocessError):
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400402 """This exception is raised when the timeout expires while waiting for a
403 child process.
404 """
Gregory P. Smith6e730002015-04-14 16:14:25 -0700405 def __init__(self, cmd, timeout, output=None, stderr=None):
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400406 self.cmd = cmd
Reid Kleckner2b228f02011-03-16 16:57:54 -0400407 self.timeout = timeout
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400408 self.output = output
Gregory P. Smith6e730002015-04-14 16:14:25 -0700409 self.stderr = stderr
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400410
411 def __str__(self):
412 return ("Command '%s' timed out after %s seconds" %
413 (self.cmd, self.timeout))
414
Gregory P. Smith6e730002015-04-14 16:14:25 -0700415 @property
416 def stdout(self):
417 return self.output
418
419 @stdout.setter
420 def stdout(self, value):
421 # There's no obvious reason to set this, but allow it anyway so
422 # .stdout is a transparent alias for .output
423 self.output = value
424
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400425
Gregory P. Smithcb6fdf22015-04-07 16:11:33 -0700426if _mswindows:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000427 import threading
428 import msvcrt
Antoine Pitrou23bba4c2012-04-18 20:51:15 +0200429 import _winapi
Brian Curtin1ce6b582010-04-24 16:19:22 +0000430 class STARTUPINFO:
431 dwFlags = 0
432 hStdInput = None
433 hStdOutput = None
434 hStdError = None
435 wShowWindow = 0
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000436else:
Gregory P. Smith59fd1bf2011-05-28 09:32:39 -0700437 import _posixsubprocess
Charles-François Natali3a4586a2013-11-08 19:56:59 +0100438 import select
439 import selectors
Gregory P. Smithd65ba512014-04-23 00:27:17 -0700440 try:
441 import threading
442 except ImportError:
443 import dummy_threading as threading
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000444
Amaury Forgeot d'Arcace31022009-07-09 22:44:11 +0000445 # When select or poll has indicated that the file is writable,
446 # we can write up to _PIPE_BUF bytes without risk of blocking.
447 # POSIX defines PIPE_BUF as >= 512.
448 _PIPE_BUF = getattr(select, 'PIPE_BUF', 512)
449
Charles-François Natali3a4586a2013-11-08 19:56:59 +0100450 # poll/select have the advantage of not requiring any extra file
451 # descriptor, contrarily to epoll/kqueue (also, they require a single
452 # syscall).
453 if hasattr(selectors, 'PollSelector'):
454 _PopenSelector = selectors.PollSelector
455 else:
456 _PopenSelector = selectors.SelectSelector
457
Amaury Forgeot d'Arcace31022009-07-09 22:44:11 +0000458
Brett Cannona23810f2008-05-26 19:04:21 +0000459__all__ = ["Popen", "PIPE", "STDOUT", "call", "check_call", "getstatusoutput",
Gregory P. Smith6e730002015-04-14 16:14:25 -0700460 "getoutput", "check_output", "run", "CalledProcessError", "DEVNULL",
461 "SubprocessError", "TimeoutExpired", "CompletedProcess"]
Gregory P. Smithace55862015-04-07 15:57:54 -0700462 # NOTE: We intentionally exclude list2cmdline as it is
463 # considered an internal implementation detail. issue10838.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000464
Gregory P. Smithcb6fdf22015-04-07 16:11:33 -0700465if _mswindows:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +0200466 from _winapi import (CREATE_NEW_CONSOLE, CREATE_NEW_PROCESS_GROUP,
467 STD_INPUT_HANDLE, STD_OUTPUT_HANDLE,
468 STD_ERROR_HANDLE, SW_HIDE,
469 STARTF_USESTDHANDLES, STARTF_USESHOWWINDOW)
Brian Curtin5d9deaa2011-04-29 16:24:07 -0500470
Brian Curtin08fd8d92011-04-29 16:11:30 -0500471 __all__.extend(["CREATE_NEW_CONSOLE", "CREATE_NEW_PROCESS_GROUP",
Brian Curtin8b8e7f42011-04-29 15:48:13 -0500472 "STD_INPUT_HANDLE", "STD_OUTPUT_HANDLE",
473 "STD_ERROR_HANDLE", "SW_HIDE",
474 "STARTF_USESTDHANDLES", "STARTF_USESHOWWINDOW"])
Antoine Pitrou23bba4c2012-04-18 20:51:15 +0200475
476 class Handle(int):
477 closed = False
478
479 def Close(self, CloseHandle=_winapi.CloseHandle):
480 if not self.closed:
481 self.closed = True
482 CloseHandle(self)
483
484 def Detach(self):
485 if not self.closed:
486 self.closed = True
487 return int(self)
488 raise ValueError("already closed")
489
490 def __repr__(self):
Serhiy Storchaka465e60e2014-07-25 23:36:00 +0300491 return "%s(%d)" % (self.__class__.__name__, int(self))
Antoine Pitrou23bba4c2012-04-18 20:51:15 +0200492
493 __del__ = Close
494 __str__ = __repr__
495
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000496
Charles-François Natali134a8ba2011-08-18 18:49:39 +0200497# This lists holds Popen instances for which the underlying process had not
498# exited at the time its __del__ method got called: those processes are wait()ed
499# for synchronously from _cleanup() when a new Popen object is created, to avoid
500# zombie processes.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000501_active = []
502
503def _cleanup():
504 for inst in _active[:]:
Georg Brandl6aa2d1f2008-08-12 08:35:52 +0000505 res = inst._internal_poll(_deadstate=sys.maxsize)
Charles-François Natali134a8ba2011-08-18 18:49:39 +0200506 if res is not None:
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000507 try:
508 _active.remove(inst)
509 except ValueError:
510 # This can happen if two threads create a new Popen instance.
511 # It's harmless that it was already removed, so ignore.
512 pass
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000513
514PIPE = -1
515STDOUT = -2
Ross Lagerwallba102ec2011-03-16 18:40:25 +0200516DEVNULL = -3
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000517
518
Antoine Pitrouebdcd852012-05-18 18:33:07 +0200519# XXX This function is only used by multiprocessing and the test suite,
520# but it's here so that it can be imported when Python is compiled without
521# threads.
522
523def _args_from_interpreter_flags():
524 """Return a list of command-line arguments reproducing the current
525 settings in sys.flags and sys.warnoptions."""
526 flag_opt_map = {
527 'debug': 'd',
528 # 'inspect': 'i',
529 # 'interactive': 'i',
530 'optimize': 'O',
531 'dont_write_bytecode': 'B',
532 'no_user_site': 's',
533 'no_site': 'S',
534 'ignore_environment': 'E',
535 'verbose': 'v',
536 'bytes_warning': 'b',
537 'quiet': 'q',
Antoine Pitrouebdcd852012-05-18 18:33:07 +0200538 }
539 args = []
540 for flag, opt in flag_opt_map.items():
541 v = getattr(sys.flags, flag)
542 if v > 0:
543 args.append('-' + opt * v)
544 for opt in sys.warnoptions:
545 args.append('-W' + opt)
546 return args
547
548
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400549def call(*popenargs, timeout=None, **kwargs):
550 """Run command with arguments. Wait for command to complete or
551 timeout, then return the returncode attribute.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000552
553 The arguments are the same as for the Popen constructor. Example:
554
555 retcode = call(["ls", "-l"])
556 """
Victor Stinnerc15c88c2011-09-01 23:45:04 +0200557 with Popen(*popenargs, **kwargs) as p:
558 try:
559 return p.wait(timeout=timeout)
560 except:
561 p.kill()
562 p.wait()
563 raise
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000564
565
Peter Astrand454f7672005-01-01 09:36:35 +0000566def check_call(*popenargs, **kwargs):
567 """Run command with arguments. Wait for command to complete. If
568 the exit code was zero then return, otherwise raise
569 CalledProcessError. The CalledProcessError object will have the
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000570 return code in the returncode attribute.
Peter Astrand454f7672005-01-01 09:36:35 +0000571
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400572 The arguments are the same as for the call function. Example:
Peter Astrand454f7672005-01-01 09:36:35 +0000573
574 check_call(["ls", "-l"])
575 """
576 retcode = call(*popenargs, **kwargs)
Peter Astrand454f7672005-01-01 09:36:35 +0000577 if retcode:
Georg Brandlf9734072008-12-07 15:30:06 +0000578 cmd = kwargs.get("args")
579 if cmd is None:
580 cmd = popenargs[0]
Thomas Wouters0e3f5912006-08-11 14:57:12 +0000581 raise CalledProcessError(retcode, cmd)
Georg Brandlf9734072008-12-07 15:30:06 +0000582 return 0
583
584
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400585def check_output(*popenargs, timeout=None, **kwargs):
Gregory P. Smith91110f52013-03-19 23:25:16 -0700586 r"""Run command with arguments and return its output.
Georg Brandlf9734072008-12-07 15:30:06 +0000587
588 If the exit code was non-zero it raises a CalledProcessError. The
589 CalledProcessError object will have the return code in the returncode
590 attribute and output in the output attribute.
591
592 The arguments are the same as for the Popen constructor. Example:
593
594 >>> check_output(["ls", "-l", "/dev/null"])
Georg Brandl2708f3a2009-12-20 14:38:23 +0000595 b'crw-rw-rw- 1 root root 1, 3 Oct 18 2007 /dev/null\n'
Georg Brandlf9734072008-12-07 15:30:06 +0000596
597 The stdout argument is not allowed as it is used internally.
Georg Brandl127d4702009-12-28 08:10:38 +0000598 To capture standard error in the result, use stderr=STDOUT.
Georg Brandlf9734072008-12-07 15:30:06 +0000599
600 >>> check_output(["/bin/sh", "-c",
Georg Brandl2708f3a2009-12-20 14:38:23 +0000601 ... "ls -l non_existent_file ; exit 0"],
Georg Brandl127d4702009-12-28 08:10:38 +0000602 ... stderr=STDOUT)
Georg Brandl2708f3a2009-12-20 14:38:23 +0000603 b'ls: non_existent_file: No such file or directory\n'
Gregory P. Smith91110f52013-03-19 23:25:16 -0700604
Serhiy Storchakafcd9f222013-04-22 20:20:54 +0300605 There is an additional optional argument, "input", allowing you to
606 pass a string to the subprocess's stdin. If you use this argument
607 you may not also use the Popen constructor's "stdin" argument, as
608 it too will be used internally. Example:
609
610 >>> check_output(["sed", "-e", "s/foo/bar/"],
611 ... input=b"when in the course of fooman events\n")
612 b'when in the course of barman events\n'
613
Andrew Kuchling4f7b0c32014-04-14 15:08:18 -0400614 If universal_newlines=True is passed, the "input" argument must be a
615 string and the return value will be a string rather than bytes.
Georg Brandlf9734072008-12-07 15:30:06 +0000616 """
617 if 'stdout' in kwargs:
618 raise ValueError('stdout argument not allowed, it will be overridden.')
Gregory P. Smith6e730002015-04-14 16:14:25 -0700619
620 if 'input' in kwargs and kwargs['input'] is None:
621 # Explicitly passing input=None was previously equivalent to passing an
622 # empty string. That is maintained here for backwards compatibility.
623 kwargs['input'] = '' if kwargs.get('universal_newlines', False) else b''
624
625 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
626 **kwargs).stdout
627
628
629class CompletedProcess(object):
630 """A process that has finished running.
631
632 This is returned by run().
633
634 Attributes:
635 args: The list or str args passed to run().
636 returncode: The exit code of the process, negative for signals.
637 stdout: The standard output (None if not captured).
638 stderr: The standard error (None if not captured).
639 """
640 def __init__(self, args, returncode, stdout=None, stderr=None):
641 self.args = args
642 self.returncode = returncode
643 self.stdout = stdout
644 self.stderr = stderr
645
646 def __repr__(self):
647 args = ['args={!r}'.format(self.args),
648 'returncode={!r}'.format(self.returncode)]
649 if self.stdout is not None:
650 args.append('stdout={!r}'.format(self.stdout))
651 if self.stderr is not None:
652 args.append('stderr={!r}'.format(self.stderr))
653 return "{}({})".format(type(self).__name__, ', '.join(args))
654
655 def check_returncode(self):
656 """Raise CalledProcessError if the exit code is non-zero."""
657 if self.returncode:
658 raise CalledProcessError(self.returncode, self.args, self.stdout,
659 self.stderr)
660
661
662def run(*popenargs, input=None, timeout=None, check=False, **kwargs):
663 """Run command with arguments and return a CompletedProcess instance.
664
665 The returned instance will have attributes args, returncode, stdout and
666 stderr. By default, stdout and stderr are not captured, and those attributes
667 will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
668
669 If check is True and the exit code was non-zero, it raises a
670 CalledProcessError. The CalledProcessError object will have the return code
671 in the returncode attribute, and output & stderr attributes if those streams
672 were captured.
673
674 If timeout is given, and the process takes too long, a TimeoutExpired
675 exception will be raised.
676
677 There is an optional argument "input", allowing you to
678 pass a string to the subprocess's stdin. If you use this argument
679 you may not also use the Popen constructor's "stdin" argument, as
680 it will be used internally.
681
682 The other arguments are the same as for the Popen constructor.
683
684 If universal_newlines=True is passed, the "input" argument must be a
685 string and stdout/stderr in the returned object will be strings rather than
686 bytes.
687 """
688 if input is not None:
Serhiy Storchakafcd9f222013-04-22 20:20:54 +0300689 if 'stdin' in kwargs:
690 raise ValueError('stdin and input arguments may not both be used.')
Serhiy Storchakafcd9f222013-04-22 20:20:54 +0300691 kwargs['stdin'] = PIPE
Gregory P. Smith6e730002015-04-14 16:14:25 -0700692
693 with Popen(*popenargs, **kwargs) as process:
Victor Stinnerc15c88c2011-09-01 23:45:04 +0200694 try:
Gregory P. Smith6e730002015-04-14 16:14:25 -0700695 stdout, stderr = process.communicate(input, timeout=timeout)
Victor Stinnerc15c88c2011-09-01 23:45:04 +0200696 except TimeoutExpired:
697 process.kill()
Gregory P. Smith6e730002015-04-14 16:14:25 -0700698 stdout, stderr = process.communicate()
699 raise TimeoutExpired(process.args, timeout, output=stdout,
700 stderr=stderr)
Victor Stinnerc15c88c2011-09-01 23:45:04 +0200701 except:
702 process.kill()
703 process.wait()
704 raise
705 retcode = process.poll()
Gregory P. Smith6e730002015-04-14 16:14:25 -0700706 if check and retcode:
707 raise CalledProcessError(retcode, process.args,
708 output=stdout, stderr=stderr)
709 return CompletedProcess(process.args, retcode, stdout, stderr)
Peter Astrand454f7672005-01-01 09:36:35 +0000710
711
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000712def list2cmdline(seq):
713 """
714 Translate a sequence of arguments into a command line
715 string, using the same rules as the MS C runtime:
716
717 1) Arguments are delimited by white space, which is either a
718 space or a tab.
719
720 2) A string surrounded by double quotation marks is
721 interpreted as a single argument, regardless of white space
Jean-Paul Calderone1ddd4072010-06-18 20:03:54 +0000722 contained within. A quoted string can be embedded in an
723 argument.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000724
725 3) A double quotation mark preceded by a backslash is
726 interpreted as a literal double quotation mark.
727
728 4) Backslashes are interpreted literally, unless they
729 immediately precede a double quotation mark.
730
731 5) If backslashes immediately precede a double quotation mark,
732 every pair of backslashes is interpreted as a literal
733 backslash. If the number of backslashes is odd, the last
734 backslash escapes the next double quotation mark as
735 described in rule 3.
736 """
737
738 # See
Eric Smith3c573af2009-11-09 15:23:15 +0000739 # http://msdn.microsoft.com/en-us/library/17w5ykft.aspx
740 # or search http://msdn.microsoft.com for
741 # "Parsing C++ Command-Line Arguments"
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000742 result = []
743 needquote = False
744 for arg in seq:
745 bs_buf = []
746
747 # Add a space to separate this argument from the others
748 if result:
749 result.append(' ')
750
Jean-Paul Calderone1ddd4072010-06-18 20:03:54 +0000751 needquote = (" " in arg) or ("\t" in arg) or not arg
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000752 if needquote:
753 result.append('"')
754
755 for c in arg:
756 if c == '\\':
Tim Peterse718f612004-10-12 21:51:32 +0000757 # Don't know if we need to double yet.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000758 bs_buf.append(c)
759 elif c == '"':
Christian Heimesfdab48e2008-01-20 09:06:41 +0000760 # Double backslashes.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000761 result.append('\\' * len(bs_buf)*2)
762 bs_buf = []
763 result.append('\\"')
764 else:
765 # Normal char
766 if bs_buf:
767 result.extend(bs_buf)
768 bs_buf = []
769 result.append(c)
770
Christian Heimesfdab48e2008-01-20 09:06:41 +0000771 # Add remaining backslashes, if any.
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000772 if bs_buf:
773 result.extend(bs_buf)
774
775 if needquote:
Peter Astrand7e78ade2005-03-03 21:10:23 +0000776 result.extend(bs_buf)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000777 result.append('"')
778
779 return ''.join(result)
780
781
Brett Cannona23810f2008-05-26 19:04:21 +0000782# Various tools for executing commands and looking at their output and status.
783#
Brett Cannona23810f2008-05-26 19:04:21 +0000784
785def getstatusoutput(cmd):
Tim Golden60798142013-11-05 12:57:25 +0000786 """ Return (status, output) of executing cmd in a shell.
Brett Cannona23810f2008-05-26 19:04:21 +0000787
Tim Golden60798142013-11-05 12:57:25 +0000788 Execute the string 'cmd' in a shell with 'check_output' and
789 return a 2-tuple (status, output). Universal newlines mode is used,
790 meaning that the result with be decoded to a string.
791
792 A trailing newline is stripped from the output.
793 The exit status for the command can be interpreted
794 according to the rules for the function 'wait'. Example:
Brett Cannona23810f2008-05-26 19:04:21 +0000795
796 >>> import subprocess
797 >>> subprocess.getstatusoutput('ls /bin/ls')
798 (0, '/bin/ls')
799 >>> subprocess.getstatusoutput('cat /bin/junk')
800 (256, 'cat: /bin/junk: No such file or directory')
801 >>> subprocess.getstatusoutput('/bin/junk')
802 (256, 'sh: /bin/junk: not found')
803 """
Tim Goldene0041752013-11-03 12:53:17 +0000804 try:
805 data = check_output(cmd, shell=True, universal_newlines=True, stderr=STDOUT)
806 status = 0
807 except CalledProcessError as ex:
808 data = ex.output
809 status = ex.returncode
810 if data[-1:] == '\n':
811 data = data[:-1]
812 return status, data
Brett Cannona23810f2008-05-26 19:04:21 +0000813
814def getoutput(cmd):
815 """Return output (stdout or stderr) of executing cmd in a shell.
816
817 Like getstatusoutput(), except the exit status is ignored and the return
818 value is a string containing the command's output. Example:
819
820 >>> import subprocess
821 >>> subprocess.getoutput('ls /bin/ls')
822 '/bin/ls'
823 """
824 return getstatusoutput(cmd)[1]
825
826
Gregory P. Smith8edd99d2010-12-14 13:43:30 +0000827_PLATFORM_DEFAULT_CLOSE_FDS = object()
Gregory P. Smithf5604852010-12-13 06:45:02 +0000828
829
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000830class Popen(object):
Serhiy Storchaka72e77612014-02-10 19:20:22 +0200831
832 _child_created = False # Set here since __del__ checks it
833
Gregory P. Smitha1ed5392013-03-23 11:44:25 -0700834 def __init__(self, args, bufsize=-1, executable=None,
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000835 stdin=None, stdout=None, stderr=None,
Gregory P. Smith8edd99d2010-12-14 13:43:30 +0000836 preexec_fn=None, close_fds=_PLATFORM_DEFAULT_CLOSE_FDS,
837 shell=False, cwd=None, env=None, universal_newlines=False,
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000838 startupinfo=None, creationflags=0,
Gregory P. Smithd4cc7bf2010-12-04 11:22:11 +0000839 restore_signals=True, start_new_session=False,
840 pass_fds=()):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000841 """Create new Popen instance."""
842 _cleanup()
Gregory P. Smithd65ba512014-04-23 00:27:17 -0700843 # Held while anything is calling waitpid before returncode has been
844 # updated to prevent clobbering returncode if wait() or poll() are
845 # called from multiple threads at once. After acquiring the lock,
846 # code must re-check self.returncode to see if another thread just
847 # finished a waitpid() call.
848 self._waitpid_lock = threading.Lock()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000849
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400850 self._input = None
851 self._communication_started = False
Guido van Rossum46a05a72007-06-07 21:56:45 +0000852 if bufsize is None:
Gregory P. Smitha1ed5392013-03-23 11:44:25 -0700853 bufsize = -1 # Restore default
Walter Dörwaldaa97f042007-05-03 21:05:51 +0000854 if not isinstance(bufsize, int):
Peter Astrand738131d2004-11-30 21:04:45 +0000855 raise TypeError("bufsize must be an integer")
856
Gregory P. Smithcb6fdf22015-04-07 16:11:33 -0700857 if _mswindows:
Tim Peterse8374a52004-10-13 03:15:00 +0000858 if preexec_fn is not None:
859 raise ValueError("preexec_fn is not supported on Windows "
860 "platforms")
Gregory P. Smith8edd99d2010-12-14 13:43:30 +0000861 any_stdio_set = (stdin is not None or stdout is not None or
862 stderr is not None)
863 if close_fds is _PLATFORM_DEFAULT_CLOSE_FDS:
864 if any_stdio_set:
865 close_fds = False
866 else:
867 close_fds = True
868 elif close_fds and any_stdio_set:
869 raise ValueError(
870 "close_fds is not supported on Windows platforms"
871 " if you redirect stdin/stdout/stderr")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000872 else:
873 # POSIX
Gregory P. Smith8edd99d2010-12-14 13:43:30 +0000874 if close_fds is _PLATFORM_DEFAULT_CLOSE_FDS:
875 close_fds = True
876 if pass_fds and not close_fds:
877 warnings.warn("pass_fds overriding close_fds.", RuntimeWarning)
878 close_fds = True
Tim Peterse8374a52004-10-13 03:15:00 +0000879 if startupinfo is not None:
880 raise ValueError("startupinfo is only supported on Windows "
881 "platforms")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000882 if creationflags != 0:
Tim Peterse8374a52004-10-13 03:15:00 +0000883 raise ValueError("creationflags is only supported on Windows "
884 "platforms")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000885
Reid Kleckner31aa7dd2011-03-14 12:02:10 -0400886 self.args = args
Tim Peterse718f612004-10-12 21:51:32 +0000887 self.stdin = None
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000888 self.stdout = None
889 self.stderr = None
890 self.pid = None
891 self.returncode = None
892 self.universal_newlines = universal_newlines
893
894 # Input and output objects. The general principle is like
895 # this:
896 #
897 # Parent Child
898 # ------ -----
899 # p2cwrite ---stdin---> p2cread
900 # c2pread <--stdout--- c2pwrite
901 # errread <--stderr--- errwrite
902 #
903 # On POSIX, the child objects are file descriptors. On
904 # Windows, these are Windows file handles. The parent objects
905 # are file descriptors on both platforms. The parent objects
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000906 # are -1 when not using PIPEs. The child objects are -1
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000907 # when not redirecting.
Tim Peterse718f612004-10-12 21:51:32 +0000908
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000909 (p2cread, p2cwrite,
910 c2pread, c2pwrite,
911 errread, errwrite) = self._get_handles(stdin, stdout, stderr)
912
Antoine Pitrouc9982322011-01-04 19:07:07 +0000913 # We wrap OS handles *before* launching the child, otherwise a
914 # quickly terminating child could make our fds unwrappable
915 # (see #8458).
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000916
Gregory P. Smithcb6fdf22015-04-07 16:11:33 -0700917 if _mswindows:
Florent Xicluna3b8bfef2010-03-14 12:31:06 +0000918 if p2cwrite != -1:
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000919 p2cwrite = msvcrt.open_osfhandle(p2cwrite.Detach(), 0)
Florent Xicluna3b8bfef2010-03-14 12:31:06 +0000920 if c2pread != -1:
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000921 c2pread = msvcrt.open_osfhandle(c2pread.Detach(), 0)
Florent Xicluna3b8bfef2010-03-14 12:31:06 +0000922 if errread != -1:
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +0000923 errread = msvcrt.open_osfhandle(errread.Detach(), 0)
Thomas Wouterscf297e42007-02-23 15:07:44 +0000924
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000925 if p2cwrite != -1:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000926 self.stdin = io.open(p2cwrite, 'wb', bufsize)
Andrew Svetlov592df202012-08-15 17:36:15 +0300927 if universal_newlines:
Antoine Pitrouafe8d062014-09-21 21:10:56 +0200928 self.stdin = io.TextIOWrapper(self.stdin, write_through=True,
929 line_buffering=(bufsize == 1))
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000930 if c2pread != -1:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000931 self.stdout = io.open(c2pread, 'rb', bufsize)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000932 if universal_newlines:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000933 self.stdout = io.TextIOWrapper(self.stdout)
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +0000934 if errread != -1:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000935 self.stderr = io.open(errread, 'rb', bufsize)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000936 if universal_newlines:
Guido van Rossumfa0054a2007-05-24 04:05:35 +0000937 self.stderr = io.TextIOWrapper(self.stderr)
Tim Peterse718f612004-10-12 21:51:32 +0000938
Gregory P. Smithb5461b92013-06-15 18:04:26 -0700939 self._closed_child_pipe_fds = False
Antoine Pitrouc9982322011-01-04 19:07:07 +0000940 try:
941 self._execute_child(args, executable, preexec_fn, close_fds,
Andrew Svetlov592df202012-08-15 17:36:15 +0300942 pass_fds, cwd, env,
Antoine Pitrouc9982322011-01-04 19:07:07 +0000943 startupinfo, creationflags, shell,
944 p2cread, p2cwrite,
945 c2pread, c2pwrite,
946 errread, errwrite,
947 restore_signals, start_new_session)
948 except:
Gregory P. Smith3d8e7762012-11-10 22:32:22 -0800949 # Cleanup if the child failed starting.
950 for f in filter(None, (self.stdin, self.stdout, self.stderr)):
Antoine Pitrouc9982322011-01-04 19:07:07 +0000951 try:
952 f.close()
Andrew Svetlov3438fa42012-12-17 23:35:18 +0200953 except OSError:
Gregory P. Smith3d8e7762012-11-10 22:32:22 -0800954 pass # Ignore EBADF or other errors.
955
Gregory P. Smithb5461b92013-06-15 18:04:26 -0700956 if not self._closed_child_pipe_fds:
957 to_close = []
958 if stdin == PIPE:
959 to_close.append(p2cread)
960 if stdout == PIPE:
961 to_close.append(c2pwrite)
962 if stderr == PIPE:
963 to_close.append(errwrite)
964 if hasattr(self, '_devnull'):
965 to_close.append(self._devnull)
966 for fd in to_close:
967 try:
968 os.close(fd)
Gregory P. Smith22ba31a2013-06-15 18:14:56 -0700969 except OSError:
Gregory P. Smithb5461b92013-06-15 18:04:26 -0700970 pass
Gregory P. Smith3d8e7762012-11-10 22:32:22 -0800971
Antoine Pitrouc9982322011-01-04 19:07:07 +0000972 raise
973
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000974
Guido van Rossum98297ee2007-11-06 21:34:58 +0000975 def _translate_newlines(self, data, encoding):
Andrew Svetlov82860712012-08-19 22:13:41 +0300976 data = data.decode(encoding)
977 return data.replace("\r\n", "\n").replace("\r", "\n")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +0000978
Brian Curtin79cdb662010-12-03 02:46:02 +0000979 def __enter__(self):
980 return self
981
982 def __exit__(self, type, value, traceback):
983 if self.stdout:
984 self.stdout.close()
985 if self.stderr:
986 self.stderr.close()
Serhiy Storchakaab900c22015-02-28 12:43:08 +0200987 try: # Flushing a BufferedWriter may raise an error
988 if self.stdin:
989 self.stdin.close()
990 finally:
991 # Wait for the process to terminate, to avoid zombies.
992 self.wait()
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000993
Serhiy Storchaka72e77612014-02-10 19:20:22 +0200994 def __del__(self, _maxsize=sys.maxsize):
995 if not self._child_created:
Thomas Wouters49fd7fa2006-04-21 10:40:58 +0000996 # We didn't get to successfully create a child process.
997 return
998 # In case the child hasn't been waited on, check if it's done.
Brett Cannon84df1e62010-05-14 00:33:40 +0000999 self._internal_poll(_deadstate=_maxsize)
Thomas Wouters0e3f5912006-08-11 14:57:12 +00001000 if self.returncode is None and _active is not None:
Thomas Wouters49fd7fa2006-04-21 10:40:58 +00001001 # Child is still running, keep us alive until we can wait on it.
1002 _active.append(self)
1003
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001004 def _get_devnull(self):
1005 if not hasattr(self, '_devnull'):
1006 self._devnull = os.open(os.devnull, os.O_RDWR)
1007 return self._devnull
Thomas Wouters49fd7fa2006-04-21 10:40:58 +00001008
Victor Stinnera5e881d2015-01-14 17:07:59 +01001009 def _stdin_write(self, input):
1010 if input:
1011 try:
1012 self.stdin.write(input)
1013 except BrokenPipeError:
1014 # communicate() must ignore broken pipe error
1015 pass
1016 except OSError as e:
1017 if e.errno == errno.EINVAL and self.poll() is not None:
1018 # Issue #19612: On Windows, stdin.write() fails with EINVAL
1019 # if the process already exited before the write
1020 pass
1021 else:
1022 raise
1023 self.stdin.close()
1024
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001025 def communicate(self, input=None, timeout=None):
Peter Astrand23109f02005-03-03 20:28:59 +00001026 """Interact with process: Send data to stdin. Read data from
1027 stdout and stderr, until end-of-file is reached. Wait for
Andrew Kuchling4f7b0c32014-04-14 15:08:18 -04001028 process to terminate.
Tim Peterseba28be2005-03-28 01:08:02 +00001029
Andrew Kuchling4f7b0c32014-04-14 15:08:18 -04001030 The optional "input" argument should be data to be sent to the
1031 child process (if self.universal_newlines is True, this should
1032 be a string; if it is False, "input" should be bytes), or
1033 None, if no data should be sent to the child.
1034
1035 communicate() returns a tuple (stdout, stderr). These will be
1036 bytes or, if self.universal_newlines was True, a string.
1037 """
Peter Astrand23109f02005-03-03 20:28:59 +00001038
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001039 if self._communication_started and input:
1040 raise ValueError("Cannot send input after starting communication")
1041
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001042 # Optimization: If we are not worried about timeouts, we haven't
1043 # started communicating, and we have one or zero pipes, using select()
1044 # or threads is unnecessary.
Victor Stinner7a8d0812011-04-05 13:13:08 +02001045 if (timeout is None and not self._communication_started and
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001046 [self.stdin, self.stdout, self.stderr].count(None) >= 2):
Tim Peterseba28be2005-03-28 01:08:02 +00001047 stdout = None
1048 stderr = None
Peter Astrand23109f02005-03-03 20:28:59 +00001049 if self.stdin:
Victor Stinnera5e881d2015-01-14 17:07:59 +01001050 self._stdin_write(input)
Peter Astrand23109f02005-03-03 20:28:59 +00001051 elif self.stdout:
Charles-François Natali6e6c59b2015-02-07 13:27:50 +00001052 stdout = self.stdout.read()
Georg Brandlf08a9dd2008-06-10 16:57:31 +00001053 self.stdout.close()
Peter Astrand23109f02005-03-03 20:28:59 +00001054 elif self.stderr:
Charles-François Natali6e6c59b2015-02-07 13:27:50 +00001055 stderr = self.stderr.read()
Georg Brandlf08a9dd2008-06-10 16:57:31 +00001056 self.stderr.close()
Peter Astrand23109f02005-03-03 20:28:59 +00001057 self.wait()
Victor Stinner7a8d0812011-04-05 13:13:08 +02001058 else:
1059 if timeout is not None:
Victor Stinner949d8c92012-05-30 13:30:32 +02001060 endtime = _time() + timeout
Victor Stinner7a8d0812011-04-05 13:13:08 +02001061 else:
1062 endtime = None
Tim Peterseba28be2005-03-28 01:08:02 +00001063
Victor Stinner7a8d0812011-04-05 13:13:08 +02001064 try:
1065 stdout, stderr = self._communicate(input, endtime, timeout)
1066 finally:
1067 self._communication_started = True
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001068
Victor Stinner7a8d0812011-04-05 13:13:08 +02001069 sts = self.wait(timeout=self._remaining_time(endtime))
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001070
1071 return (stdout, stderr)
Peter Astrand23109f02005-03-03 20:28:59 +00001072
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001073
Georg Brandl6aa2d1f2008-08-12 08:35:52 +00001074 def poll(self):
1075 return self._internal_poll()
1076
1077
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001078 def _remaining_time(self, endtime):
1079 """Convenience for _communicate when computing timeouts."""
1080 if endtime is None:
1081 return None
1082 else:
Victor Stinner949d8c92012-05-30 13:30:32 +02001083 return endtime - _time()
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001084
1085
Reid Kleckner2b228f02011-03-16 16:57:54 -04001086 def _check_timeout(self, endtime, orig_timeout):
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001087 """Convenience for checking if a timeout has expired."""
1088 if endtime is None:
1089 return
Victor Stinner949d8c92012-05-30 13:30:32 +02001090 if _time() > endtime:
Reid Kleckner2b228f02011-03-16 16:57:54 -04001091 raise TimeoutExpired(self.args, orig_timeout)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001092
1093
Gregory P. Smithcb6fdf22015-04-07 16:11:33 -07001094 if _mswindows:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001095 #
1096 # Windows methods
1097 #
1098 def _get_handles(self, stdin, stdout, stderr):
Alexandre Vassalotti711ed4a2009-07-17 10:42:05 +00001099 """Construct and return tuple with IO objects:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001100 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
1101 """
Peter Astrandd38ddf42005-02-10 08:32:50 +00001102 if stdin is None and stdout is None and stderr is None:
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001103 return (-1, -1, -1, -1, -1, -1)
Tim Peterse718f612004-10-12 21:51:32 +00001104
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001105 p2cread, p2cwrite = -1, -1
1106 c2pread, c2pwrite = -1, -1
1107 errread, errwrite = -1, -1
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001108
Peter Astrandd38ddf42005-02-10 08:32:50 +00001109 if stdin is None:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001110 p2cread = _winapi.GetStdHandle(_winapi.STD_INPUT_HANDLE)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +00001111 if p2cread is None:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001112 p2cread, _ = _winapi.CreatePipe(None, 0)
1113 p2cread = Handle(p2cread)
1114 _winapi.CloseHandle(_)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +00001115 elif stdin == PIPE:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001116 p2cread, p2cwrite = _winapi.CreatePipe(None, 0)
1117 p2cread, p2cwrite = Handle(p2cread), Handle(p2cwrite)
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001118 elif stdin == DEVNULL:
1119 p2cread = msvcrt.get_osfhandle(self._get_devnull())
Peter Astrandd38ddf42005-02-10 08:32:50 +00001120 elif isinstance(stdin, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001121 p2cread = msvcrt.get_osfhandle(stdin)
1122 else:
1123 # Assuming file-like object
1124 p2cread = msvcrt.get_osfhandle(stdin.fileno())
1125 p2cread = self._make_inheritable(p2cread)
1126
Peter Astrandd38ddf42005-02-10 08:32:50 +00001127 if stdout is None:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001128 c2pwrite = _winapi.GetStdHandle(_winapi.STD_OUTPUT_HANDLE)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +00001129 if c2pwrite is None:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001130 _, c2pwrite = _winapi.CreatePipe(None, 0)
1131 c2pwrite = Handle(c2pwrite)
1132 _winapi.CloseHandle(_)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +00001133 elif stdout == PIPE:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001134 c2pread, c2pwrite = _winapi.CreatePipe(None, 0)
1135 c2pread, c2pwrite = Handle(c2pread), Handle(c2pwrite)
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001136 elif stdout == DEVNULL:
1137 c2pwrite = msvcrt.get_osfhandle(self._get_devnull())
Peter Astrandd38ddf42005-02-10 08:32:50 +00001138 elif isinstance(stdout, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001139 c2pwrite = msvcrt.get_osfhandle(stdout)
1140 else:
1141 # Assuming file-like object
1142 c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
1143 c2pwrite = self._make_inheritable(c2pwrite)
1144
Peter Astrandd38ddf42005-02-10 08:32:50 +00001145 if stderr is None:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001146 errwrite = _winapi.GetStdHandle(_winapi.STD_ERROR_HANDLE)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +00001147 if errwrite is None:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001148 _, errwrite = _winapi.CreatePipe(None, 0)
1149 errwrite = Handle(errwrite)
1150 _winapi.CloseHandle(_)
Hirokazu Yamamoto0c988172009-03-03 22:41:26 +00001151 elif stderr == PIPE:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001152 errread, errwrite = _winapi.CreatePipe(None, 0)
1153 errread, errwrite = Handle(errread), Handle(errwrite)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001154 elif stderr == STDOUT:
1155 errwrite = c2pwrite
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001156 elif stderr == DEVNULL:
1157 errwrite = msvcrt.get_osfhandle(self._get_devnull())
Peter Astrandd38ddf42005-02-10 08:32:50 +00001158 elif isinstance(stderr, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001159 errwrite = msvcrt.get_osfhandle(stderr)
1160 else:
1161 # Assuming file-like object
1162 errwrite = msvcrt.get_osfhandle(stderr.fileno())
1163 errwrite = self._make_inheritable(errwrite)
1164
1165 return (p2cread, p2cwrite,
1166 c2pread, c2pwrite,
1167 errread, errwrite)
1168
1169
1170 def _make_inheritable(self, handle):
1171 """Return a duplicate of handle, which is inheritable"""
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001172 h = _winapi.DuplicateHandle(
1173 _winapi.GetCurrentProcess(), handle,
1174 _winapi.GetCurrentProcess(), 0, 1,
1175 _winapi.DUPLICATE_SAME_ACCESS)
1176 return Handle(h)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001177
1178
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001179 def _execute_child(self, args, executable, preexec_fn, close_fds,
Andrew Svetlov592df202012-08-15 17:36:15 +03001180 pass_fds, cwd, env,
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001181 startupinfo, creationflags, shell,
1182 p2cread, p2cwrite,
1183 c2pread, c2pwrite,
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001184 errread, errwrite,
1185 unused_restore_signals, unused_start_new_session):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001186 """Execute program (MS Windows version)"""
1187
Gregory P. Smith8edd99d2010-12-14 13:43:30 +00001188 assert not pass_fds, "pass_fds not supported on Windows."
Gregory P. Smithd4cc7bf2010-12-04 11:22:11 +00001189
Guido van Rossum3172c5d2007-10-16 18:12:55 +00001190 if not isinstance(args, str):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001191 args = list2cmdline(args)
1192
Peter Astrandc1d65362004-11-07 14:30:34 +00001193 # Process startup details
Peter Astrandd38ddf42005-02-10 08:32:50 +00001194 if startupinfo is None:
Thomas Wouters73e5a5b2006-06-08 15:35:45 +00001195 startupinfo = STARTUPINFO()
Victor Stinnerb3693582010-05-21 20:13:12 +00001196 if -1 not in (p2cread, c2pwrite, errwrite):
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001197 startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES
Peter Astrandc1d65362004-11-07 14:30:34 +00001198 startupinfo.hStdInput = p2cread
1199 startupinfo.hStdOutput = c2pwrite
1200 startupinfo.hStdError = errwrite
1201
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001202 if shell:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001203 startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
1204 startupinfo.wShowWindow = _winapi.SW_HIDE
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001205 comspec = os.environ.get("COMSPEC", "cmd.exe")
Tim Golden126c2962010-08-11 14:20:40 +00001206 args = '{} /c "{}"'.format (comspec, args)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001207
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001208 # Start the process
1209 try:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001210 hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
Tim Peterse8374a52004-10-13 03:15:00 +00001211 # no special security
1212 None, None,
Guido van Rossume7ba4952007-06-06 23:52:48 +00001213 int(not close_fds),
Tim Peterse8374a52004-10-13 03:15:00 +00001214 creationflags,
1215 env,
1216 cwd,
1217 startupinfo)
Tim Goldenad537f22010-08-08 11:18:16 +00001218 finally:
1219 # Child is launched. Close the parent's copy of those pipe
1220 # handles that only the child should have open. You need
1221 # to make sure that no handles to the write end of the
1222 # output pipe are maintained in this process or else the
1223 # pipe will not close when the child process exits and the
1224 # ReadFile will hang.
1225 if p2cread != -1:
1226 p2cread.Close()
1227 if c2pwrite != -1:
1228 c2pwrite.Close()
1229 if errwrite != -1:
1230 errwrite.Close()
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001231 if hasattr(self, '_devnull'):
1232 os.close(self._devnull)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001233
1234 # Retain the process handle, but close the thread handle
Thomas Wouters49fd7fa2006-04-21 10:40:58 +00001235 self._child_created = True
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001236 self._handle = Handle(hp)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001237 self.pid = pid
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001238 _winapi.CloseHandle(ht)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001239
Brett Cannon84df1e62010-05-14 00:33:40 +00001240 def _internal_poll(self, _deadstate=None,
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001241 _WaitForSingleObject=_winapi.WaitForSingleObject,
1242 _WAIT_OBJECT_0=_winapi.WAIT_OBJECT_0,
1243 _GetExitCodeProcess=_winapi.GetExitCodeProcess):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001244 """Check if child process has terminated. Returns returncode
Brett Cannon84df1e62010-05-14 00:33:40 +00001245 attribute.
1246
1247 This method is called by __del__, so it can only refer to objects
1248 in its local scope.
1249
1250 """
Peter Astrandd38ddf42005-02-10 08:32:50 +00001251 if self.returncode is None:
Brett Cannon84df1e62010-05-14 00:33:40 +00001252 if _WaitForSingleObject(self._handle, 0) == _WAIT_OBJECT_0:
1253 self.returncode = _GetExitCodeProcess(self._handle)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001254 return self.returncode
1255
1256
Reid Kleckner2b228f02011-03-16 16:57:54 -04001257 def wait(self, timeout=None, endtime=None):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001258 """Wait for child process to terminate. Returns returncode
1259 attribute."""
Reid Kleckner2b228f02011-03-16 16:57:54 -04001260 if endtime is not None:
1261 timeout = self._remaining_time(endtime)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001262 if timeout is None:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001263 timeout_millis = _winapi.INFINITE
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001264 else:
Reid Kleckner91156ff2011-03-21 10:06:10 -07001265 timeout_millis = int(timeout * 1000)
Peter Astrandd38ddf42005-02-10 08:32:50 +00001266 if self.returncode is None:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001267 result = _winapi.WaitForSingleObject(self._handle,
1268 timeout_millis)
1269 if result == _winapi.WAIT_TIMEOUT:
Reid Kleckner2b228f02011-03-16 16:57:54 -04001270 raise TimeoutExpired(self.args, timeout)
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001271 self.returncode = _winapi.GetExitCodeProcess(self._handle)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001272 return self.returncode
1273
1274
1275 def _readerthread(self, fh, buffer):
1276 buffer.append(fh.read())
Victor Stinner667d4b52010-12-25 22:40:32 +00001277 fh.close()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001278
1279
Reid Kleckner2b228f02011-03-16 16:57:54 -04001280 def _communicate(self, input, endtime, orig_timeout):
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001281 # Start reader threads feeding into a list hanging off of this
1282 # object, unless they've already been started.
1283 if self.stdout and not hasattr(self, "_stdout_buff"):
1284 self._stdout_buff = []
1285 self.stdout_thread = \
1286 threading.Thread(target=self._readerthread,
1287 args=(self.stdout, self._stdout_buff))
1288 self.stdout_thread.daemon = True
1289 self.stdout_thread.start()
1290 if self.stderr and not hasattr(self, "_stderr_buff"):
1291 self._stderr_buff = []
1292 self.stderr_thread = \
1293 threading.Thread(target=self._readerthread,
1294 args=(self.stderr, self._stderr_buff))
1295 self.stderr_thread.daemon = True
1296 self.stderr_thread.start()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001297
1298 if self.stdin:
Victor Stinnera5e881d2015-01-14 17:07:59 +01001299 self._stdin_write(input)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001300
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001301 # Wait for the reader threads, or time out. If we time out, the
1302 # threads remain reading and the fds left open in case the user
1303 # calls communicate again.
1304 if self.stdout is not None:
1305 self.stdout_thread.join(self._remaining_time(endtime))
Andrew Svetlov377a1522012-08-19 20:49:39 +03001306 if self.stdout_thread.is_alive():
Reid Kleckner9a67e6c2011-03-20 08:28:07 -07001307 raise TimeoutExpired(self.args, orig_timeout)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001308 if self.stderr is not None:
1309 self.stderr_thread.join(self._remaining_time(endtime))
Andrew Svetlov377a1522012-08-19 20:49:39 +03001310 if self.stderr_thread.is_alive():
Reid Kleckner9a67e6c2011-03-20 08:28:07 -07001311 raise TimeoutExpired(self.args, orig_timeout)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001312
1313 # Collect the output from and close both pipes, now that we know
1314 # both have been read successfully.
1315 stdout = None
1316 stderr = None
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001317 if self.stdout:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001318 stdout = self._stdout_buff
1319 self.stdout.close()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001320 if self.stderr:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001321 stderr = self._stderr_buff
1322 self.stderr.close()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001323
1324 # All data exchanged. Translate lists into strings.
Peter Astrandd38ddf42005-02-10 08:32:50 +00001325 if stdout is not None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001326 stdout = stdout[0]
Peter Astrandd38ddf42005-02-10 08:32:50 +00001327 if stderr is not None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001328 stderr = stderr[0]
1329
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001330 return (stdout, stderr)
1331
Christian Heimesa342c012008-04-20 21:01:16 +00001332 def send_signal(self, sig):
Gregory P. Smitha0c9caa2015-11-15 18:19:10 -08001333 """Send a signal to the process."""
1334 # Don't signal a process that we know has already died.
1335 if self.returncode is not None:
1336 return
Christian Heimesa342c012008-04-20 21:01:16 +00001337 if sig == signal.SIGTERM:
1338 self.terminate()
Brian Curtineb24d742010-04-12 17:16:38 +00001339 elif sig == signal.CTRL_C_EVENT:
1340 os.kill(self.pid, signal.CTRL_C_EVENT)
1341 elif sig == signal.CTRL_BREAK_EVENT:
1342 os.kill(self.pid, signal.CTRL_BREAK_EVENT)
Christian Heimesa342c012008-04-20 21:01:16 +00001343 else:
Brian Curtin19651362010-09-07 13:24:38 +00001344 raise ValueError("Unsupported signal: {}".format(sig))
Christian Heimesa342c012008-04-20 21:01:16 +00001345
1346 def terminate(self):
Gregory P. Smitha0c9caa2015-11-15 18:19:10 -08001347 """Terminates the process."""
1348 # Don't terminate a process that we know has already died.
1349 if self.returncode is not None:
1350 return
Antoine Pitrou1f9a8352012-03-11 19:29:12 +01001351 try:
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001352 _winapi.TerminateProcess(self._handle, 1)
Antoine Pitroub69ef162012-03-11 19:33:29 +01001353 except PermissionError:
Antoine Pitrou1f9a8352012-03-11 19:29:12 +01001354 # ERROR_ACCESS_DENIED (winerror 5) is received when the
1355 # process already died.
Antoine Pitrou23bba4c2012-04-18 20:51:15 +02001356 rc = _winapi.GetExitCodeProcess(self._handle)
1357 if rc == _winapi.STILL_ACTIVE:
Antoine Pitrou1f9a8352012-03-11 19:29:12 +01001358 raise
1359 self.returncode = rc
Christian Heimesa342c012008-04-20 21:01:16 +00001360
1361 kill = terminate
1362
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001363 else:
1364 #
1365 # POSIX methods
1366 #
1367 def _get_handles(self, stdin, stdout, stderr):
Alexandre Vassalotti711ed4a2009-07-17 10:42:05 +00001368 """Construct and return tuple with IO objects:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001369 p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite
1370 """
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001371 p2cread, p2cwrite = -1, -1
1372 c2pread, c2pwrite = -1, -1
1373 errread, errwrite = -1, -1
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001374
Peter Astrandd38ddf42005-02-10 08:32:50 +00001375 if stdin is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001376 pass
1377 elif stdin == PIPE:
Victor Stinnerdaf45552013-08-28 00:53:59 +02001378 p2cread, p2cwrite = os.pipe()
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001379 elif stdin == DEVNULL:
1380 p2cread = self._get_devnull()
Peter Astrandd38ddf42005-02-10 08:32:50 +00001381 elif isinstance(stdin, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001382 p2cread = stdin
1383 else:
1384 # Assuming file-like object
1385 p2cread = stdin.fileno()
1386
Peter Astrandd38ddf42005-02-10 08:32:50 +00001387 if stdout is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001388 pass
1389 elif stdout == PIPE:
Victor Stinnerdaf45552013-08-28 00:53:59 +02001390 c2pread, c2pwrite = os.pipe()
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001391 elif stdout == DEVNULL:
1392 c2pwrite = self._get_devnull()
Peter Astrandd38ddf42005-02-10 08:32:50 +00001393 elif isinstance(stdout, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001394 c2pwrite = stdout
1395 else:
1396 # Assuming file-like object
Tim Peterse718f612004-10-12 21:51:32 +00001397 c2pwrite = stdout.fileno()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001398
Peter Astrandd38ddf42005-02-10 08:32:50 +00001399 if stderr is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001400 pass
1401 elif stderr == PIPE:
Victor Stinnerdaf45552013-08-28 00:53:59 +02001402 errread, errwrite = os.pipe()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001403 elif stderr == STDOUT:
Martin Panterc7635892016-05-13 01:54:44 +00001404 if c2pwrite != -1:
1405 errwrite = c2pwrite
1406 else: # child's stdout is not set, use parent's stdout
1407 errwrite = sys.__stdout__.fileno()
Ross Lagerwallba102ec2011-03-16 18:40:25 +02001408 elif stderr == DEVNULL:
1409 errwrite = self._get_devnull()
Peter Astrandd38ddf42005-02-10 08:32:50 +00001410 elif isinstance(stderr, int):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001411 errwrite = stderr
1412 else:
1413 # Assuming file-like object
1414 errwrite = stderr.fileno()
1415
1416 return (p2cread, p2cwrite,
1417 c2pread, c2pwrite,
1418 errread, errwrite)
1419
1420
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001421 def _execute_child(self, args, executable, preexec_fn, close_fds,
Andrew Svetlov592df202012-08-15 17:36:15 +03001422 pass_fds, cwd, env,
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001423 startupinfo, creationflags, shell,
1424 p2cread, p2cwrite,
1425 c2pread, c2pwrite,
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001426 errread, errwrite,
1427 restore_signals, start_new_session):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001428 """Execute program (POSIX version)"""
1429
Victor Stinner7b3b20a2011-03-03 12:54:05 +00001430 if isinstance(args, (str, bytes)):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001431 args = [args]
Thomas Wouters89f507f2006-12-13 04:49:30 +00001432 else:
1433 args = list(args)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001434
1435 if shell:
1436 args = ["/bin/sh", "-c"] + args
Stefan Krah9542cc62010-07-19 14:20:53 +00001437 if executable:
1438 args[0] = executable
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001439
Peter Astrandd38ddf42005-02-10 08:32:50 +00001440 if executable is None:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001441 executable = args[0]
Gregory P. Smith5591b022012-10-10 03:34:47 -07001442 orig_executable = executable
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001443
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001444 # For transferring possible exec failure from child to parent.
1445 # Data format: "exception name:hex errno:description"
1446 # Pickle is not used; it is complex and involves memory allocation.
Victor Stinnerdaf45552013-08-28 00:53:59 +02001447 errpipe_read, errpipe_write = os.pipe()
Gregory P. Smith53dd8162013-12-01 16:03:24 -08001448 # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
1449 low_fds_to_close = []
1450 while errpipe_write < 3:
1451 low_fds_to_close.append(errpipe_write)
1452 errpipe_write = os.dup(errpipe_write)
1453 for low_fd in low_fds_to_close:
1454 os.close(low_fd)
Christian Heimesfdab48e2008-01-20 09:06:41 +00001455 try:
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001456 try:
Gregory P. Smith59fd1bf2011-05-28 09:32:39 -07001457 # We must avoid complex work that could involve
1458 # malloc or free in the child process to avoid
1459 # potential deadlocks, thus we do all this here.
1460 # and pass it to fork_exec()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001461
Victor Stinner372b8382011-06-21 17:24:21 +02001462 if env is not None:
Gregory P. Smith59fd1bf2011-05-28 09:32:39 -07001463 env_list = [os.fsencode(k) + b'=' + os.fsencode(v)
1464 for k, v in env.items()]
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001465 else:
Gregory P. Smith59fd1bf2011-05-28 09:32:39 -07001466 env_list = None # Use execv instead of execve.
1467 executable = os.fsencode(executable)
1468 if os.path.dirname(executable):
1469 executable_list = (executable,)
1470 else:
1471 # This matches the behavior of os._execvpe().
1472 executable_list = tuple(
1473 os.path.join(os.fsencode(dir), executable)
1474 for dir in os.get_exec_path(env))
Gregory P. Smith361e30c2013-12-01 00:12:24 -08001475 fds_to_keep = set(pass_fds)
Gregory P. Smith59fd1bf2011-05-28 09:32:39 -07001476 fds_to_keep.add(errpipe_write)
1477 self.pid = _posixsubprocess.fork_exec(
1478 args, executable_list,
1479 close_fds, sorted(fds_to_keep), cwd, env_list,
1480 p2cread, p2cwrite, c2pread, c2pwrite,
1481 errread, errwrite,
1482 errpipe_read, errpipe_write,
1483 restore_signals, start_new_session, preexec_fn)
Charles-François Natali558639f2011-08-18 19:11:29 +02001484 self._child_created = True
Facundo Batista10706e22009-06-19 20:34:30 +00001485 finally:
1486 # be sure the FD is closed no matter what
1487 os.close(errpipe_write)
1488
Gregory P. Smithb5461b92013-06-15 18:04:26 -07001489 # self._devnull is not always defined.
1490 devnull_fd = getattr(self, '_devnull', None)
1491 if p2cread != -1 and p2cwrite != -1 and p2cread != devnull_fd:
Facundo Batista10706e22009-06-19 20:34:30 +00001492 os.close(p2cread)
Gregory P. Smithb5461b92013-06-15 18:04:26 -07001493 if c2pwrite != -1 and c2pread != -1 and c2pwrite != devnull_fd:
Facundo Batista10706e22009-06-19 20:34:30 +00001494 os.close(c2pwrite)
Gregory P. Smithb5461b92013-06-15 18:04:26 -07001495 if errwrite != -1 and errread != -1 and errwrite != devnull_fd:
Facundo Batista10706e22009-06-19 20:34:30 +00001496 os.close(errwrite)
Gregory P. Smithb5461b92013-06-15 18:04:26 -07001497 if devnull_fd is not None:
1498 os.close(devnull_fd)
1499 # Prevent a double close of these fds from __init__ on error.
1500 self._closed_child_pipe_fds = True
Facundo Batista10706e22009-06-19 20:34:30 +00001501
1502 # Wait for exec to fail or succeed; possibly raising an
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001503 # exception (limited in size)
Gregory P. Smithf44c9da2012-11-10 23:33:17 -08001504 errpipe_data = bytearray()
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001505 while True:
Charles-François Natali6e6c59b2015-02-07 13:27:50 +00001506 part = os.read(errpipe_read, 50000)
Gregory P. Smithf44c9da2012-11-10 23:33:17 -08001507 errpipe_data += part
1508 if not part or len(errpipe_data) > 50000:
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001509 break
Facundo Batista10706e22009-06-19 20:34:30 +00001510 finally:
1511 # be sure the FD is closed no matter what
1512 os.close(errpipe_read)
1513
Gregory P. Smithf44c9da2012-11-10 23:33:17 -08001514 if errpipe_data:
Gregory P. Smithe85db2b2010-12-14 14:38:00 +00001515 try:
Charles-François Natali6e6c59b2015-02-07 13:27:50 +00001516 os.waitpid(self.pid, 0)
Victor Stinnera5e881d2015-01-14 17:07:59 +01001517 except ChildProcessError:
1518 pass
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001519 try:
Gregory P. Smithf44c9da2012-11-10 23:33:17 -08001520 exception_name, hex_errno, err_msg = (
1521 errpipe_data.split(b':', 2))
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001522 except ValueError:
Gregory P. Smith8d07c262012-11-10 23:53:47 -08001523 exception_name = b'SubprocessError'
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001524 hex_errno = b'0'
Gregory P. Smith3aee2222012-11-11 00:04:13 -08001525 err_msg = (b'Bad exception data from child: ' +
1526 repr(errpipe_data))
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001527 child_exception_type = getattr(
1528 builtins, exception_name.decode('ascii'),
Gregory P. Smith8d07c262012-11-10 23:53:47 -08001529 SubprocessError)
Victor Stinner4d078042010-04-23 19:28:32 +00001530 err_msg = err_msg.decode(errors="surrogatepass")
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001531 if issubclass(child_exception_type, OSError) and hex_errno:
Benjamin Petersonb8bc4392010-11-20 18:24:54 +00001532 errno_num = int(hex_errno, 16)
Gregory P. Smith5591b022012-10-10 03:34:47 -07001533 child_exec_never_called = (err_msg == "noexec")
1534 if child_exec_never_called:
1535 err_msg = ""
Benjamin Petersonb8bc4392010-11-20 18:24:54 +00001536 if errno_num != 0:
1537 err_msg = os.strerror(errno_num)
1538 if errno_num == errno.ENOENT:
Gregory P. Smith5591b022012-10-10 03:34:47 -07001539 if child_exec_never_called:
1540 # The error must be from chdir(cwd).
1541 err_msg += ': ' + repr(cwd)
1542 else:
1543 err_msg += ': ' + repr(orig_executable)
Benjamin Petersonb8bc4392010-11-20 18:24:54 +00001544 raise child_exception_type(errno_num, err_msg)
Gregory P. Smithfb94c5f2010-03-14 06:49:55 +00001545 raise child_exception_type(err_msg)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001546
1547
Brett Cannon84df1e62010-05-14 00:33:40 +00001548 def _handle_exitstatus(self, sts, _WIFSIGNALED=os.WIFSIGNALED,
1549 _WTERMSIG=os.WTERMSIG, _WIFEXITED=os.WIFEXITED,
1550 _WEXITSTATUS=os.WEXITSTATUS):
Gregory P. Smithd65ba512014-04-23 00:27:17 -07001551 """All callers to this function MUST hold self._waitpid_lock."""
Brett Cannon84df1e62010-05-14 00:33:40 +00001552 # This method is called (indirectly) by __del__, so it cannot
Serhiy Storchaka72e77612014-02-10 19:20:22 +02001553 # refer to anything outside of its local scope.
Brett Cannon84df1e62010-05-14 00:33:40 +00001554 if _WIFSIGNALED(sts):
1555 self.returncode = -_WTERMSIG(sts)
1556 elif _WIFEXITED(sts):
1557 self.returncode = _WEXITSTATUS(sts)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001558 else:
1559 # Should never happen
Gregory P. Smith8d07c262012-11-10 23:53:47 -08001560 raise SubprocessError("Unknown child exit status!")
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001561
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001562
Brett Cannon84df1e62010-05-14 00:33:40 +00001563 def _internal_poll(self, _deadstate=None, _waitpid=os.waitpid,
Andrew Svetlov1d960fe2012-12-24 20:08:53 +02001564 _WNOHANG=os.WNOHANG, _ECHILD=errno.ECHILD):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001565 """Check if child process has terminated. Returns returncode
Brett Cannon84df1e62010-05-14 00:33:40 +00001566 attribute.
1567
1568 This method is called by __del__, so it cannot reference anything
1569 outside of the local scope (nor can any methods it calls).
1570
1571 """
Peter Astrandd38ddf42005-02-10 08:32:50 +00001572 if self.returncode is None:
Gregory P. Smithd65ba512014-04-23 00:27:17 -07001573 if not self._waitpid_lock.acquire(False):
1574 # Something else is busy calling waitpid. Don't allow two
1575 # at once. We know nothing yet.
1576 return None
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001577 try:
Gregory P. Smithd65ba512014-04-23 00:27:17 -07001578 if self.returncode is not None:
1579 return self.returncode # Another thread waited.
Brett Cannon84df1e62010-05-14 00:33:40 +00001580 pid, sts = _waitpid(self.pid, _WNOHANG)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001581 if pid == self.pid:
1582 self._handle_exitstatus(sts)
Andrew Svetlovad28c7f2012-12-18 22:02:39 +02001583 except OSError as e:
Thomas Wouters49fd7fa2006-04-21 10:40:58 +00001584 if _deadstate is not None:
1585 self.returncode = _deadstate
Andrew Svetlov08bab072012-12-24 20:06:35 +02001586 elif e.errno == _ECHILD:
Gregory P. Smith39051712012-09-29 11:40:38 -07001587 # This happens if SIGCLD is set to be ignored or
1588 # waiting for child processes has otherwise been
1589 # disabled for our process. This child is dead, we
1590 # can't get the status.
1591 # http://bugs.python.org/issue15756
1592 self.returncode = 0
Gregory P. Smithd65ba512014-04-23 00:27:17 -07001593 finally:
1594 self._waitpid_lock.release()
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001595 return self.returncode
1596
1597
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001598 def _try_wait(self, wait_flags):
Gregory P. Smithd65ba512014-04-23 00:27:17 -07001599 """All callers to this function MUST hold self._waitpid_lock."""
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001600 try:
Charles-François Natali6e6c59b2015-02-07 13:27:50 +00001601 (pid, sts) = os.waitpid(self.pid, wait_flags)
Victor Stinnera5e881d2015-01-14 17:07:59 +01001602 except ChildProcessError:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001603 # This happens if SIGCLD is set to be ignored or waiting
1604 # for child processes has otherwise been disabled for our
1605 # process. This child is dead, we can't get the status.
1606 pid = self.pid
1607 sts = 0
1608 return (pid, sts)
1609
1610
1611 def wait(self, timeout=None, endtime=None):
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001612 """Wait for child process to terminate. Returns returncode
1613 attribute."""
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001614 if self.returncode is not None:
1615 return self.returncode
Reid Kleckner2b228f02011-03-16 16:57:54 -04001616
1617 # endtime is preferred to timeout. timeout is only used for
1618 # printing.
1619 if endtime is not None or timeout is not None:
1620 if endtime is None:
Victor Stinner949d8c92012-05-30 13:30:32 +02001621 endtime = _time() + timeout
Reid Kleckner2b228f02011-03-16 16:57:54 -04001622 elif timeout is None:
1623 timeout = self._remaining_time(endtime)
1624
1625 if endtime is not None:
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001626 # Enter a busy loop if we have a timeout. This busy loop was
1627 # cribbed from Lib/threading.py in Thread.wait() at r71065.
1628 delay = 0.0005 # 500 us -> initial delay of 1 ms
1629 while True:
Gregory P. Smithd65ba512014-04-23 00:27:17 -07001630 if self._waitpid_lock.acquire(False):
1631 try:
1632 if self.returncode is not None:
1633 break # Another thread waited.
1634 (pid, sts) = self._try_wait(os.WNOHANG)
1635 assert pid == self.pid or pid == 0
1636 if pid == self.pid:
1637 self._handle_exitstatus(sts)
1638 break
1639 finally:
1640 self._waitpid_lock.release()
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001641 remaining = self._remaining_time(endtime)
1642 if remaining <= 0:
Reid Kleckner2b228f02011-03-16 16:57:54 -04001643 raise TimeoutExpired(self.args, timeout)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001644 delay = min(delay * 2, remaining, .05)
1645 time.sleep(delay)
Gregory P. Smithf328d792012-11-10 21:06:18 -08001646 else:
1647 while self.returncode is None:
Gregory P. Smithd65ba512014-04-23 00:27:17 -07001648 with self._waitpid_lock:
1649 if self.returncode is not None:
1650 break # Another thread waited.
1651 (pid, sts) = self._try_wait(0)
1652 # Check the pid and loop as waitpid has been known to
1653 # return 0 even without WNOHANG in odd situations.
1654 # http://bugs.python.org/issue14396.
1655 if pid == self.pid:
1656 self._handle_exitstatus(sts)
Fredrik Lundh5b3687d2004-10-12 15:26:28 +00001657 return self.returncode
1658
1659
Reid Kleckner2b228f02011-03-16 16:57:54 -04001660 def _communicate(self, input, endtime, orig_timeout):
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001661 if self.stdin and not self._communication_started:
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001662 # Flush stdio buffer. This might block, if the user has
1663 # been writing to .stdin in an uncontrolled fashion.
1664 self.stdin.flush()
1665 if not input:
1666 self.stdin.close()
1667
Charles-François Natali3a4586a2013-11-08 19:56:59 +01001668 stdout = None
1669 stderr = None
1670
1671 # Only create this mapping if we haven't already.
1672 if not self._communication_started:
1673 self._fileobj2output = {}
1674 if self.stdout:
1675 self._fileobj2output[self.stdout] = []
1676 if self.stderr:
1677 self._fileobj2output[self.stderr] = []
1678
1679 if self.stdout:
1680 stdout = self._fileobj2output[self.stdout]
1681 if self.stderr:
1682 stderr = self._fileobj2output[self.stderr]
1683
1684 self._save_input(input)
1685
Gregory P. Smith5ca129b2013-12-07 19:14:59 -08001686 if self._input:
1687 input_view = memoryview(self._input)
1688
Charles-François Natali3a4586a2013-11-08 19:56:59 +01001689 with _PopenSelector() as selector:
1690 if self.stdin and input:
1691 selector.register(self.stdin, selectors.EVENT_WRITE)
1692 if self.stdout:
1693 selector.register(self.stdout, selectors.EVENT_READ)
1694 if self.stderr:
1695 selector.register(self.stderr, selectors.EVENT_READ)
1696
1697 while selector.get_map():
1698 timeout = self._remaining_time(endtime)
1699 if timeout is not None and timeout < 0:
1700 raise TimeoutExpired(self.args, orig_timeout)
1701
1702 ready = selector.select(timeout)
1703 self._check_timeout(endtime, orig_timeout)
1704
1705 # XXX Rewrite these to use non-blocking I/O on the file
1706 # objects; they are no longer using C stdio!
1707
1708 for key, events in ready:
1709 if key.fileobj is self.stdin:
Gregory P. Smith5ca129b2013-12-07 19:14:59 -08001710 chunk = input_view[self._input_offset :
1711 self._input_offset + _PIPE_BUF]
Charles-François Natali3a4586a2013-11-08 19:56:59 +01001712 try:
1713 self._input_offset += os.write(key.fd, chunk)
Victor Stinnera5e881d2015-01-14 17:07:59 +01001714 except BrokenPipeError:
1715 selector.unregister(key.fileobj)
1716 key.fileobj.close()
Charles-François Natali3a4586a2013-11-08 19:56:59 +01001717 else:
1718 if self._input_offset >= len(self._input):
1719 selector.unregister(key.fileobj)
1720 key.fileobj.close()
1721 elif key.fileobj in (self.stdout, self.stderr):
Gregory P. Smith7b83b182013-12-08 10:58:28 -08001722 data = os.read(key.fd, 32768)
Charles-François Natali3a4586a2013-11-08 19:56:59 +01001723 if not data:
1724 selector.unregister(key.fileobj)
1725 key.fileobj.close()
1726 self._fileobj2output[key.fileobj].append(data)
Reid Kleckner31aa7dd2011-03-14 12:02:10 -04001727
1728 self.wait(timeout=self._remaining_time(endtime))
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001729
1730 # All data exchanged. Translate lists into strings.
1731 if stdout is not None:
1732 stdout = b''.join(stdout)
1733 if stderr is not None:
1734 stderr = b''.join(stderr)
1735
1736 # Translate newlines, if requested.
1737 # This also turns bytes into strings.
1738 if self.universal_newlines:
1739 if stdout is not None:
1740 stdout = self._translate_newlines(stdout,
1741 self.stdout.encoding)
1742 if stderr is not None:
1743 stderr = self._translate_newlines(stderr,
1744 self.stderr.encoding)
1745
Gregory P. Smithd06fa472009-07-04 02:46:54 +00001746 return (stdout, stderr)
1747
1748
Andrew Svetlovaa0dbdc2012-08-14 18:40:21 +03001749 def _save_input(self, input):
1750 # This method is called from the _communicate_with_*() methods
1751 # so that if we time out while communicating, we can continue
1752 # sending input if we retry.
1753 if self.stdin and self._input is None:
1754 self._input_offset = 0
1755 self._input = input
1756 if self.universal_newlines and input is not None:
1757 self._input = self._input.encode(self.stdin.encoding)
1758
1759
Christian Heimesa342c012008-04-20 21:01:16 +00001760 def send_signal(self, sig):
Gregory P. Smitha0c9caa2015-11-15 18:19:10 -08001761 """Send a signal to the process."""
1762 # Skip signalling a process that we know has already died.
1763 if self.returncode is None:
1764 os.kill(self.pid, sig)
Christian Heimesa342c012008-04-20 21:01:16 +00001765
1766 def terminate(self):
1767 """Terminate the process with SIGTERM
1768 """
1769 self.send_signal(signal.SIGTERM)
1770
1771 def kill(self):
1772 """Kill the process with SIGKILL
1773 """
1774 self.send_signal(signal.SIGKILL)