1 .. currentmodule:: asyncio 2 3 .. _asyncio-subprocess: 4 5 ============ 6 Subprocesses 7 ============ 8 9 This section describes high-level async/await asyncio APIs to 10 create and manage subprocesses. 11 12 .. _asyncio_example_subprocess_shell: 13 14 Here's an example of how asyncio can run a shell command and 15 obtain its result:: 16 17 import asyncio 18 19 async def run(cmd): 20 proc = await asyncio.create_subprocess_shell( 21 cmd, 22 stdout=asyncio.subprocess.PIPE, 23 stderr=asyncio.subprocess.PIPE) 24 25 stdout, stderr = await proc.communicate() 26 27 print(f'[{cmd!r} exited with {proc.returncode}]') 28 if stdout: 29 print(f'[stdout]\n{stdout.decode()}') 30 if stderr: 31 print(f'[stderr]\n{stderr.decode()}') 32 33 asyncio.run(run('ls /zzz')) 34 35 will print:: 36 37 ['ls /zzz' exited with 1] 38 [stderr] 39 ls: /zzz: No such file or directory 40 41 Because all asyncio subprocess functions are asynchronous and asyncio 42 provides many tools to work with such functions, it is easy to execute 43 and monitor multiple subprocesses in parallel. It is indeed trivial 44 to modify the above example to run several commands simultaneously:: 45 46 async def main(): 47 await asyncio.gather( 48 run('ls /zzz'), 49 run('sleep 1; echo "hello"')) 50 51 asyncio.run(main()) 52 53 See also the `Examples`_ subsection. 54 55 56 Creating Subprocesses 57 ===================== 58 59 .. coroutinefunction:: create_subprocess_exec(\*args, stdin=None, \ 60 stdout=None, stderr=None, loop=None, \ 61 limit=None, \*\*kwds) 62 63 Create a subprocess. 64 65 The *limit* argument sets the buffer limit for :class:`StreamReader` 66 wrappers for :attr:`Process.stdout` and :attr:`Process.stderr` 67 (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments). 68 69 Return a :class:`~asyncio.subprocess.Process` instance. 70 71 See the documentation of :meth:`loop.subprocess_exec` for other 72 parameters. 73 74 .. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \ 75 stdout=None, stderr=None, loop=None, \ 76 limit=None, \*\*kwds) 77 78 Run the *cmd* shell command. 79 80 The *limit* argument sets the buffer limit for :class:`StreamReader` 81 wrappers for :attr:`Process.stdout` and :attr:`Process.stderr` 82 (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments). 83 84 Return a :class:`~asyncio.subprocess.Process` instance. 85 86 See the documentation of :meth:`loop.subprocess_shell` for other 87 parameters. 88 89 .. important:: 90 91 It is the application's responsibility to ensure that all whitespace and 92 special characters are quoted appropriately to avoid `shell injection 93 <https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_ 94 vulnerabilities. The :func:`shlex.quote` function can be used to properly 95 escape whitespace and special shell characters in strings that are going 96 to be used to construct shell commands. 97 98 .. note:: 99 100 The default asyncio event loop implementation on **Windows** does not 101 support subprocesses. Subprocesses are available for Windows if a 102 :class:`ProactorEventLoop` is used. 103 See :ref:`Subprocess Support on Windows <asyncio-windows-subprocess>` 104 for details. 105 106 .. seealso:: 107 108 asyncio also has the following *low-level* APIs to work with subprocesses: 109 :meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`, 110 :meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`, 111 as well as the :ref:`Subprocess Transports <asyncio-subprocess-transports>` 112 and :ref:`Subprocess Protocols <asyncio-subprocess-protocols>`. 113 114 115 Constants 116 ========= 117 118 .. data:: asyncio.subprocess.PIPE 119 120 Can be passed to the *stdin*, *stdout* or *stderr* parameters. 121 122 If *PIPE* is passed to *stdin* argument, the 123 :attr:`Process.stdin <asyncio.subprocess.Process.stdin>` attribute 124 will point to a :class:`StreamWriter` instance. 125 126 If *PIPE* is passed to *stdout* or *stderr* arguments, the 127 :attr:`Process.stdout <asyncio.subprocess.Process.stdout>` and 128 :attr:`Process.stderr <asyncio.subprocess.Process.stderr>` 129 attributes will point to :class:`StreamReader` instances. 130 131 .. data:: asyncio.subprocess.STDOUT 132 133 Special value that can be used as the *stderr* argument and indicates 134 that standard error should be redirected into standard output. 135 136 .. data:: asyncio.subprocess.DEVNULL 137 138 Special value that can be used as the *stdin*, *stdout* or *stderr* argument 139 to process creation functions. It indicates that the special file 140 :data:`os.devnull` will be used for the corresponding subprocess stream. 141 142 143 Interacting with Subprocesses 144 ============================= 145 146 Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell` 147 functions return instances of the *Process* class. *Process* is a high-level 148 wrapper that allows communicating with subprocesses and watching for 149 their completion. 150 151 .. class:: asyncio.subprocess.Process 152 153 An object that wraps OS processes created by the 154 :func:`create_subprocess_exec` and :func:`create_subprocess_shell` 155 functions. 156 157 This class is designed to have a similar API to the 158 :class:`subprocess.Popen` class, but there are some 159 notable differences: 160 161 * unlike Popen, Process instances do not have an equivalent to 162 the :meth:`~subprocess.Popen.poll` method; 163 164 * the :meth:`~asyncio.subprocess.Process.communicate` and 165 :meth:`~asyncio.subprocess.Process.wait` methods don't have a 166 *timeout* parameter: use the :func:`wait_for` function; 167 168 * the :meth:`Process.wait() <asyncio.subprocess.Process.wait>` method 169 is asynchronous, whereas :meth:`subprocess.Popen.wait` method 170 is implemented as a blocking busy loop; 171 172 * the *universal_newlines* parameter is not supported. 173 174 This class is :ref:`not thread safe <asyncio-multithreading>`. 175 176 See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>` 177 section. 178 179 .. coroutinemethod:: wait() 180 181 Wait for the child process to terminate. 182 183 Set and return the :attr:`returncode` attribute. 184 185 .. note:: 186 187 This method can deadlock when using ``stdout=PIPE`` or 188 ``stderr=PIPE`` and the child process generates so much output 189 that it blocks waiting for the OS pipe buffer to accept 190 more data. Use the :meth:`communicate` method when using pipes 191 to avoid this condition. 192 193 .. coroutinemethod:: communicate(input=None) 194 195 Interact with process: 196 197 1. send data to *stdin* (if *input* is not ``None``); 198 2. read data from *stdout* and *stderr*, until EOF is reached; 199 3. wait for process to terminate. 200 201 The optional *input* argument is the data (:class:`bytes` object) 202 that will be sent to the child process. 203 204 Return a tuple ``(stdout_data, stderr_data)``. 205 206 If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError` 207 exception is raised when writing *input* into *stdin*, the 208 exception is ignored. This condition occurs when the process 209 exits before all data are written into *stdin*. 210 211 If it is desired to send data to the process' *stdin*, 212 the process needs to be created with ``stdin=PIPE``. Similarly, 213 to get anything other than ``None`` in the result tuple, the 214 process has to be created with ``stdout=PIPE`` and/or 215 ``stderr=PIPE`` arguments. 216 217 Note, that the data read is buffered in memory, so do not use 218 this method if the data size is large or unlimited. 219 220 .. method:: send_signal(signal) 221 222 Sends the signal *signal* to the child process. 223 224 .. note:: 225 226 On Windows, :py:data:`SIGTERM` is an alias for :meth:`terminate`. 227 ``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes 228 started with a *creationflags* parameter which includes 229 ``CREATE_NEW_PROCESS_GROUP``. 230 231 .. method:: terminate() 232 233 Stop the child process. 234 235 On POSIX systems this method sends :py:data:`signal.SIGTERM` to the 236 child process. 237 238 On Windows the Win32 API function :c:func:`TerminateProcess` is 239 called to stop the child process. 240 241 .. method:: kill() 242 243 Kill the child. 244 245 On POSIX systems this method sends :py:data:`SIGKILL` to the child 246 process. 247 248 On Windows this method is an alias for :meth:`terminate`. 249 250 .. attribute:: stdin 251 252 Standard input stream (:class:`StreamWriter`) or ``None`` 253 if the process was created with ``stdin=None``. 254 255 .. attribute:: stdout 256 257 Standard output stream (:class:`StreamReader`) or ``None`` 258 if the process was created with ``stdout=None``. 259 260 .. attribute:: stderr 261 262 Standard error stream (:class:`StreamReader`) or ``None`` 263 if the process was created with ``stderr=None``. 264 265 .. warning:: 266 267 Use the :meth:`communicate` method rather than 268 :attr:`process.stdin.write() <stdin>`, 269 :attr:`await process.stdout.read() <stdout>` or 270 :attr:`await process.stderr.read <stderr>`. 271 This avoids deadlocks due to streams pausing reading or writing 272 and blocking the child process. 273 274 .. attribute:: pid 275 276 Process identification number (PID). 277 278 Note that for processes created by the :func:`create_subprocess_shell` 279 function, this attribute is the PID of the spawned shell. 280 281 .. attribute:: returncode 282 283 Return code of the process when it exits. 284 285 A ``None`` value indicates that the process has not terminated yet. 286 287 A negative value ``-N`` indicates that the child was terminated 288 by signal ``N`` (POSIX only). 289 290 291 .. _asyncio-subprocess-threads: 292 293 Subprocess and Threads 294 ---------------------- 295 296 Standard asyncio event loop supports running subprocesses from 297 different threads, but there are limitations: 298 299 * An event loop must run in the main thread. 300 301 * The child watcher must be instantiated in the main thread 302 before executing subprocesses from other threads. Call the 303 :func:`get_child_watcher` function in the main thread to instantiate 304 the child watcher. 305 306 Note that alternative event loop implementations might not share 307 the above limitations; please refer to their documentation. 308 309 .. seealso:: 310 311 The :ref:`Concurrency and multithreading in asyncio 312 <asyncio-multithreading>` section. 313 314 315 Examples 316 -------- 317 318 An example using the :class:`~asyncio.subprocess.Process` class to 319 control a subprocess and the :class:`StreamReader` class to read from 320 its standard output. 321 322 .. _asyncio_example_create_subprocess_exec: 323 324 The subprocess is created by the :func:`create_subprocess_exec` 325 function:: 326 327 import asyncio 328 import sys 329 330 async def get_date(): 331 code = 'import datetime; print(datetime.datetime.now())' 332 333 # Create the subprocess; redirect the standard output 334 # into a pipe. 335 proc = await asyncio.create_subprocess_exec( 336 sys.executable, '-c', code, 337 stdout=asyncio.subprocess.PIPE) 338 339 # Read one line of output. 340 data = await proc.stdout.readline() 341 line = data.decode('ascii').rstrip() 342 343 # Wait for the subprocess exit. 344 await proc.wait() 345 return line 346 347 if sys.platform == "win32": 348 asyncio.set_event_loop_policy( 349 asyncio.WindowsProactorEventLoopPolicy()) 350 351 date = asyncio.run(get_date()) 352 print(f"Current date: {date}") 353 354 355 See also the :ref:`same example <asyncio_example_subprocess_proto>` 356 written using low-level APIs. 357
