显示子过程输出到标准输出并redirect

我正在通过Python的subprocess模块运行一个脚本。 目前我使用:

p = subprocess.Popen('/path/to/script', stdout=subprocess.PIPE, stderr=subprocess.PIPE) result = p.communicate() 

然后我将结果打印到标准输出。 这一切都很好,但由于脚本需要很长时间才能完成,所以我希望从脚本实时输出到stdout。 我输出输出的原因是因为我想parsing它。

将subprocess的stdout保存到一个variables中进行进一步处理, 并在subprocess运行时显示它 :

 #!/usr/bin/env python3 from io import StringIO from subprocess import Popen, PIPE with Popen('/path/to/script', stdout=PIPE, bufsize=1, universal_newlines=True) as p, StringIO() as buf: for line in p.stdout: print(line, end='') buf.write(line) output = buf.getvalue() rc = p.returncode 

要保存这两个subprocess的stdout和stderr更为复杂,因为您应该同时使用两个stream以避免死锁 :

 stdout_buf, stderr_buf = StringIO(), StringIO() rc = teed_call('/path/to/script', stdout=stdout_buf, stderr=stderr_buf, universal_newlines=True) output = stdout_buf.getvalue() ... 

这里定义了teed_call()


更新:这是一个更简单的asyncio版本 。


旧版:

下面是一个基于tulip child_process.py示例的单线程解决scheme:

 import asyncio import sys from asyncio.subprocess import PIPE @asyncio.coroutine def read_and_display(*cmd): """Read cmd's stdout, stderr while displaying them as they arrive.""" # start process process = yield from asyncio.create_subprocess_exec(*cmd, stdout=PIPE, stderr=PIPE) # read child's stdout/stderr concurrently stdout, stderr = [], [] # stderr, stdout buffers tasks = { asyncio.Task(process.stdout.readline()): ( stdout, process.stdout, sys.stdout.buffer), asyncio.Task(process.stderr.readline()): ( stderr, process.stderr, sys.stderr.buffer)} while tasks: done, pending = yield from asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED) assert done for future in done: buf, stream, display = tasks.pop(future) line = future.result() if line: # not EOF buf.append(line) # save for later display.write(line) # display in terminal # schedule to read the next line tasks[asyncio.Task(stream.readline())] = buf, stream, display # wait for the process to exit rc = yield from process.wait() return rc, b''.join(stdout), b''.join(stderr) 

该脚本运行'/path/to/script命令,并行读取stdout和stderr。 这些行相应地打印到父级的stdout / stderr,并保存为字节串,以供将来处理。 要运行read_and_display()协程,我们需要一个事件循环:

 import os if os.name == 'nt': loop = asyncio.ProactorEventLoop() # for subprocess' pipes on Windows asyncio.set_event_loop(loop) else: loop = asyncio.get_event_loop() try: rc, *output = loop.run_until_complete(read_and_display("/path/to/script")) if rc: sys.exit("child failed with '{}' exit code".format(rc)) finally: loop.close() 

p.communicate()等待子p.communicate()完成 ,然后一次返回其整个输出。

你有没有尝试过这样的事情,而不是逐行读取subprocess输出?

 p = subprocess.Popen('/path/to/script', stdout=subprocess.PIPE, stderr=subprocess.PIPE) for line in p.stdout: # do something with this individual line print line 

Popen.communicate doc明确指出:

 Note: The data read is buffered in memory, so do not use this method if the data size is large or unlimited. 

https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate

所以,如果你需要实时输出,你需要使用这样的东西:

 stream_p = subprocess.Popen('/path/to/script', stdout=subprocess.PIPE, stderr=subprocess.PIPE) while stream_line in stream_p: #Parse it the way you want print stream_line