Home | History | Annotate | Download | only in doc
      1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
      2 <html>
      3 <head>
      4 <title>Pexpect - a Pure Python Expect-like module</title>
      5 <link rel="stylesheet" href="clean.css" type="text/css">
      6 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
      7 <meta name="Author" content="Noah Spurrier">
      8 <meta name="Keywords"
      9  content="pexpect, Noah Spurrier, pypect, Python, Libes, TCL, Expect, pipe, popen, pyExpect, expectpy, expect-like, expect-alike, expect like">
     10 <meta name="Description"
     11  content="Pexpect is a pure Python Expect-like module. Pexpect makes Python a better tool for controlling other applications.">
     12 </head>
     13 <body bgcolor="#ffffff" text="#000000">
     14 <div id="Header">
     15 <h1>Pexpect version 2.4<br>
     16 a Pure Python Expect-like module
     17 </h1>
     18 </div>
     19 <div id="Content">
     20 <p>Pexpect makes Python a better tool for controlling other
     21 applications.</p>
     22 <p>Pexpect is a pure Python module for spawning child applications;
     23 controlling them; and responding to expected patterns in their output.
     24 Pexpect works like Don Libes' Expect. Pexpect allows your script to
     25 spawn a child application and control it as if a human were typing
     26 commands.</p>
     27 <p>Pexpect can be used for automating interactive applications such as
     28 ssh, ftp, passwd, telnet, etc. It can be used to a automate setup
     29 scripts for duplicating software package installations on different
     30 servers. It can be used for automated software testing. Pexpect is in
     31 the spirit of Don Libes' Expect, but Pexpect is pure Python. Unlike
     32 other Expect-like modules for Python, Pexpect does not require TCL or
     33 Expect nor does it require C extensions to be compiled. It should work
     34 on any platform that supports the standard Python pty module. The
     35 Pexpect interface was designed to be easy to use.</p>
     36 <table border="0">
     37   <tbody>
     38     <tr>
     39       <td align="right" valign="top">Send questions to:</td>
     40       <td align="left"><a href="http://www.noah.org/email/"><img
     41  src="email.png" alt="Click to send email." border="0" height="16"
     42  width="100"></a></td>
     43     </tr>
     44   </tbody>
     45 </table>
     46 <hr noshade="noshade" size="1">
     47 <h1><a name="license"></a>License: MIT style</h1>
     48 <p>
     49 Free, open source, and all that good stuff.<br>
     50 <br>
     51 Permission is hereby granted, free of charge, to any person obtaining a copy
     52 of this software and associated documentation files (the "Software"), to deal
     53 in the Software without restriction, including without limitation the rights
     54 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
     55 copies of the Software, and to permit persons to whom the Software is
     56 furnished to do so, subject to the following conditions:<br>
     57 <br>
     58 The above copyright notice and this permission notice shall be included in all
     59 copies or substantial portions of the Software.<br>
     60 <br>
     61 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
     62 EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
     63 MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
     64 IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
     65 DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
     66 OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
     67 USE OR OTHER DEALINGS IN THE SOFTWARE.<br>
     68 <br>
     69 Pexpect Copyright (c) 2008 Noah Spurrier<br>
     70 http://pexpect.sourceforge.net/
     71 </p>
     72 
     73 <hr noshade="noshade" size="1">
     74 <h1><a name="download"></a><a
     75  href="http://sourceforge.net/project/showfiles.php?group_id=59762">Download</a></h1>
     76 <p>Download the <a
     77  href="http://sourceforge.net/project/showfiles.php?group_id=59762">
     78 current version here</a> from the SourceForge site. Grab the current Pexpect tarball.
     79 </p>
     80 <h2>Installing Pexpect</h2>
     81 <p>The Pexpect tarball is a standard Python Distutil distribution.</p>
     82 <ol>
     83   <li>download <span class="code">pexpect-2.4.tar.gz</span></li>
     84   <li><span class="code">tar zxf pexpect-2.4.tar.gz</span></li>
     85   <li><span class="code">cd pexpect-2.4</span></li>
     86   <li><span class="code">python setup.py install</span> <i>do this as root</i></li>
     87 </ol>
     88 <h2>Examples</h2>
     89 <p>
     90 Under the <span class="code">pexpect-2.4</span> directory you should find
     91 the <span class="code">examples</span> directory.
     92 This is the best way to learn to use Pexpect.
     93 See the descriptions of <a href="examples.html">Pexpect Examples</a>.
     94 </p>
     95 <h2><a name="doc"></a>API Documentation</h2>
     96 <p>
     97 <blockquote>
     98 <a href="pexpect.html">pexpect</a> This is the main module that you want.<br>
     99 <a href="pxssh.html">pxssh</a> Pexpect SSH is an extension of 'pexpect.spawn' that specializes in SSH.<br>
    100 </blockquote>
    101 the following are experimental extensions to Pexpect<br>
    102 <blockquote>
    103 <a href="fdpexpect.html">fdpexpect</a> fdpexpect extension of 'pexpect.spawn' that uses an open file descriptor.<br>
    104 <a href="screen.html">SCREEN</a> This represents a virtual 'screen'.<br>
    105 <a href="ANSI.html">ANSI</a> This parses ANSI/VT-100 terminal escape codes.<br>
    106 <a href="FSM.html">FSM</a> This is a finite state machine used by ANSI.<br>
    107 </blockquote>
    108 </p>
    109 <hr noshade="noshade" size="1">
    110 <h1><a name="status"></a>Project Status</h1>
    111 <p>Automated pyunit tests reach over 80%
    112 code coverage on pexpect.py. I regularly test on Linux and BSD
    113 platforms. I try to test on Solaris and Irix. 
    114 </p>
    115 <hr noshade="noshade" size="1">
    116 <h1><a name="requirements"></a>Requirements for use of Pexpect</h1>
    117 <h2>Python</h2>
    118 <blockquote>
    119   <p>Pexpect was written and tested with Python 2.4. It should work on
    120 earlier versions that have the <span class="code">pty</span> module. I
    121 sometimes even manually test it with Python 1.5.2, but I can't easily
    122 run the PyUnit test framework against Python 1.5.2, so I have less
    123 confidence in Pexpect on Python 1.5.2.</p>
    124 </blockquote>
    125 <h2>pty module</h2>
    126 <blockquote>
    127   <p>Any POSIX system (UNIX) with a working <span class="code">pty</span>
    128 module should be able to run Pexpect. The <span class="code">pty</span>
    129 module is part of the Standard Python Library, so if you are running on
    130 a POSIX system you should have it. The <span class="code">pty</span>
    131 module does not run the same on all platforms. It should be solid on Linux
    132 and BSD systems. I have taken effort to try to smooth the wrinkles out of the different platforms. To learn more
    133 about the wrinkles see <a href="#bugs">Bugs</a> and <a href="#testing">Testing</a>.</p>
    134 </blockquote>
    135 <p>Pexpect does not currently work on the standard Windows Python (see
    136 the pty requirement); however, it seems to work fine using <a
    137  href="http://www.cygwin.com/">Cygwin</a>. It is possible to build
    138 something like a pty for Windows, but it would have to use a different
    139 technique that I am still investigating. I know it's possible because
    140 Libes' Expect was ported to Windows. <i>If you have any ideas or
    141 skills to contribute in this area then I would really appreciate some
    142 tips on how to approach this problem.</i> </p>
    143 <hr noshade="noshade" size="1">
    144 <h1><a name="overview"></a>Overview</h1>
    145 <p>Pexpect can be used for automating interactive applications such as
    146 ssh, ftp, mencoder, passwd, etc. The Pexpect interface was designed to be
    147 easy to use. Here is an example of Pexpect in action:</p>
    148 <blockquote>
    149   <pre class="code"># This connects to the openbsd ftp site and<br># downloads the recursive directory listing.<br>import pexpect<br>child = pexpect.spawn ('ftp ftp.openbsd.org')<br>child.expect ('Name .*: ')<br>child.sendline ('anonymous')<br>child.expect ('Password:')<br>child.sendline ('noah (a] example.com')<br>child.expect ('ftp&gt; ')<br>child.sendline ('cd pub')<br>child.expect('ftp&gt; ')<br>child.sendline ('get ls-lR.gz')<br>child.expect('ftp&gt; ')<br>child.sendline ('bye')<br></pre>
    150 </blockquote>
    151 <p> Obviously you could write an ftp client using Python's own <span
    152  class="code">ftplib</span> module, but this is just a demonstration.
    153 You can use this technique with any application. This is especially
    154 handy if you are writing automated test tools.</p>
    155 
    156 <p>There are two important methods in Pexpect -- <span class="code"><b>expect()</b></span>
    157 and <span class="code"><b>send()</b></span> (or <span class="code">sendline()</span>
    158 which is like <span class="code">send()</span> with a linefeed). 
    159 The <span class="code">expect()</span> method waits for the child application
    160 to return a given string. The string you specify is a regular expression, so
    161 you can match complicated patterns. The <span class="code"><b>send()</b></span> method
    162 writes a string to the child application. From the child's point of
    163 view it looks just like someone typed the text from a terminal. After
    164 each call to <span class="code"><b>expect()</b></span> the <span
    165  class="code"><b>before</b></span> and <span class="code"><b>after</b></span>
    166 properties will be set to the text printed by child application. The <span
    167  class="code"><b>before</b></span> property will contain all text up to
    168 the expected string pattern. The <span class="code"><b>after</b></span> string
    169 will contain the text that was matched by the expected pattern.
    170 The <span class="code">match</span> property is set to the <span class="code">re MatchObject</span>.
    171 </p>
    172 
    173 <p>An example of Pexpect in action may make things more clear. This example uses
    174 <span class="code">ftp</span> to login to the OpenBSD site; list files
    175 in a directory; and then pass interactive control of the ftp session to
    176 the human user.</p>
    177 <blockquote>
    178   <pre class="code">import pexpect<br>child = pexpect.spawn ('ftp ftp.openbsd.org')<br>child.expect ('Name .*: ')<br>child.sendline ('anonymous')<br>child.expect ('Password:')<br>child.sendline ('noah (a] example.com')<br>child.expect ('ftp&gt; ')<br>child.sendline ('ls /pub/OpenBSD/')<br>child.expect ('ftp&gt; ')<br>print child.before   # Print the result of the ls command.<br>child.interact()     # Give control of the child to the user.<br></pre>
    179 </blockquote>
    180 <h2>Special EOF and TIMEOUT patterns</h2>
    181 <p>
    182 There are two special patterns to match the End Of File or a Timeout condition.
    183 You you can pass these patterns to <span class="code">expect()</span>.
    184 These patterns are not regular expressions. Use them like predefined constants.
    185 </p>
    186 <p>If the child has died and you have read all the child's output then ordinarily
    187 <span class="code">expect()</span> will raise an <span class="code">EOF</span>
    188 exception. You can read everything up to the EOF without generating an
    189 exception by using the EOF pattern <span class="code">expect(pexpect.EOF)</span>.
    190 In this case everything the child has output will be available in the <span
    191  class="code">before</span> property.</p>
    192 <p>The pattern given to <span class="code">expect()</span> may be a
    193 regular expression or it may also be a <b>list</b> of regular expressions.
    194 This allows you to match multiple optional responses. The <span class="code">expect()</span>
    195 method returns the index of the pattern that was matched. For example,
    196 say you wanted to login to a server. After entering a password you
    197 could get various responses from the server -- your password could be
    198 rejected; or you could be allowed in and asked for your terminal type;
    199 or you could be let right in and given a command prompt. The following
    200 code fragment gives an example of this:</p>
    201 <blockquote>
    202   <pre class="code">child.expect('password:')<br>child.sendline (my_secret_password)<br># We expect any of these three patterns...<br>i = child.expect (['Permission denied', 'Terminal type', '[#\$] '])<br>if i==0:<br>    print 'Permission denied on host. Can't login'<br>    child.kill(0)<br>elif i==2:<br>    print 'Login OK... need to send terminal type.'<br>    child.sendline('vt100')<br>    child.expect ('[#\$] ')<br>elif i==3:<br>    print 'Login OK.'<br>    print 'Shell command prompt', child.after</pre>
    203 </blockquote>
    204 <p>If nothing matches an expected pattern then expect will eventually
    205 raise a TIMEOUT exception. The default time is 30 seconds, but you can
    206 change this by passing a timeout argument to expect():</p>
    207 <blockquote>
    208   <pre class="code"># Wait no more than 2 minutes (120 seconds) for password prompt.<br>child.expect('password:', timeout=120)</pre>
    209 </blockquote>
    210 <h2>Find the end of line -- CR/LF conventions<br>
    211 Matching at the end of a line can be tricky<br>
    212 $ regex pattern is useless.<br>
    213 </h2>
    214 <p>Pexpect matches regular expressions a little differently than what
    215 you might be used to.
    216 </p>
    217 <p><i><b>The $ pattern for end of line match is useless</b></i>.
    218 The $ matches the end of string, but Pexpect reads from the child
    219 one character at a time, so each character looks like the end of a line.
    220 Pexpect can't do a look-ahead into the child's output stream.
    221 In general you would have this situation when using regular expressions
    222 with any stream.<br>
    223 <i>Note, pexpect does have an internal buffer, so reads are faster
    224 than one character at a time, but from the user's perspective the regex
    225 patterns test happens one character at a time.</i></p>
    226 <p>The best way to match the end of a line is to look for the
    227 newline: "\r\n" (CR/LF). Yes, that does appear to be DOS-style.
    228 It may surprise some UNIX people to learn that terminal TTY device drivers
    229 (dumb, vt100, ANSI, xterm, etc.) all use the CR/LF combination to signify
    230 the end of line. Pexpect uses a Pseudo-TTY device to talk to the child application, so
    231 when the child app prints "\n" you actually see "\r\n".
    232 </p>
    233 <p><b>UNIX uses just linefeeds to end lines of text, but not when it
    234 comes to TTY devices!</b> TTY devices are more like the Windows world.
    235 Each line of text end with a CR/LF combination. When you intercept data
    236 from a UNIX command from a TTY device you will find that the TTY device
    237 outputs a CR/LF combination. A UNIX command may only write a linefeed
    238 (\n), but the TTY device driver converts it to CR/LF. This means that
    239 your terminal will see lines end with CR/LF (hex&nbsp;<span class="code">0D&nbsp;0A</span>).
    240 Since Pexpect emulates a terminal, to match ends of lines you have to
    241 expect the CR/LF combination.</p>
    242 <blockquote>
    243   <p class="code">child.expect ('\r\n')</p>
    244 </blockquote>
    245 <p>If you just need to skip past a new line then <span class="code">expect
    246 ('\n')</span> by itself will work, but if you are expecting a specific
    247 pattern before the end of line then you need to explicitly look for the
    248 \r. For example the following expects a word at the end of a line:</p>
    249 <blockquote>
    250   <p class="code">child.expect ('\w+\r\n')</p>
    251 </blockquote>
    252 <p>But the following would both fail:</p>
    253 <blockquote>
    254   <p class="code">child.expect ('\w+\n')</p>
    255 </blockquote>
    256 <p>And as explained before, trying to use '$' to match the end of line
    257 would not work either:</p>
    258 <blockquote>
    259   <p class="code">child.expect ('\w+$')</p>
    260 </blockquote>
    261 <p>So if you need to explicitly look for the END OF LINE, you want to
    262 look for the CR/LF combination -- not just the LF and not the $ pattern.</p>
    263 <p>This problem is not limited to Pexpect. This problem happens any
    264 time you try to perform a regular expression match on a stream. Regular
    265 expressions need to look ahead. With a stream it is hard to look ahead
    266 because the process generating the stream may not be finished. There is no
    267 way to know if the process has paused momentarily or is finished and
    268 waiting for you. <font color="#cc0000">Pexpect must implicitly always
    269 do a NON greedy match (minimal) at the end of a input {### already said
    270 this}.</font> </p>
    271 <p>Pexpect compiles all regular expressions with the DOTALL flag. With
    272 the DOTALL flag a "." will match a newline. See the Python <a
    273  href="http://www.python.org/doc/current/lib/node115.html#l2h-733">documentation</a></p>
    274 <h2>Beware of + and * at the end of input.</h2>
    275 <p>Remember that any time you try to match a pattern that needs
    276 look-ahead that you will always get a minimal match (non greedy). For
    277 example, the following will always return just one character:</p>
    278 <blockquote>
    279   <p class="code">child.expect ('.+')</p>
    280 </blockquote>
    281 <p>This example will match successfully, but will always return no
    282 characters:</p>
    283 <blockquote>
    284   <p class="code">child.expect ('.*')</p>
    285 </blockquote>
    286 <p>Generally any star * expression will match as little as possible</p>
    287 <p>One thing you can do is to try to force a non-ambiguous character at
    288 the end of your <span class="code">\d+</span> pattern. Expect that
    289 character to delimit the string. For example, you might try making the
    290 end of your pattrn be <span class="code">\D+</span> instead of <span
    291  class="code">\D*</span>. That means number digits alone would not
    292 satisfy the (<span class="code">\d+</span>) pattern. You would need
    293 some number(s) and at least one <span class="code">\D</span> at the
    294 end. </p>
    295 <h2>Matching groups</h2>
    296 <p>You can group regular expression using parenthesis. After a match,
    297 the <span class="code">match</span> parameter of the spawn object will
    298 contain the Python Match object. </p>
    299 <h2>Examples</h2>
    300 <p>Using "match" and groups...</p>
    301 <h2>Debugging</h2>
    302 <p>If you get the string value of a pexpect.spawn object you will get
    303 lots of useful debugging information. For debugging it's very useful to
    304 use the following pattern:</p>
    305 <p>try:<br>
    306 &nbsp;&nbsp;&nbsp; i = child.expect ([pattern1, pattern2, pattern3,
    307 etc])<br>
    308 except:<br>
    309 &nbsp;&nbsp;&nbsp; print "Exception was thrown"<br>
    310 &nbsp;&nbsp;&nbsp; print "debug information:"<br>
    311 &nbsp;&nbsp;&nbsp; print str(child)<br>
    312 </p>
    313 <p>It is also useful to log the child's input and out to a file or the
    314 screen. The following will turn on logging and send output to stdout
    315 (the screen).<br>
    316 </p>
    317 <p>&nbsp;&nbsp;&nbsp; child = pexpect.spawn (foo)<br>
    318 &nbsp;&nbsp;&nbsp; child.logfile = sys.stdout<br>
    319 <br>
    320 </p>
    321 <hr noshade="noshade" size="1">
    322 <h1>Exceptions</h1>
    323 <p><b>EOF</b></p>
    324 <p>Note that two flavors of EOF Exception may be thrown. They are
    325 virtually identical except for the message string. For practical
    326 purposes you should have no need to distinguish between them, but they
    327 do give a little extra information about what type of platform you are
    328 running. The two messages are:</p>
    329 <blockquote>
    330   <p class="code">End Of File (EOF) in read(). Exception style platform.</p>
    331   <p class="code">End Of File (EOF) in read(). Empty string style
    332 platform.</p>
    333 </blockquote>
    334 <p>Some UNIX platforms will throw an exception when you try to read
    335 from a file descriptor in the EOF state. Other UNIX platforms instead
    336 quietly return an empty string to indicate that the EOF state has been
    337 reached.</p>
    338 <p><b>Expecting EOF</b></p>
    339 <p>If you wish to read up to the end of the child's output without
    340 generating an <span class="code">EOF</span> exception then use the <span
    341  class="code">expect(pexpect.EOF)</span> method.</p>
    342 <p><b>TIMEOUT</b></p>
    343 <p>The <span class="code">expect()</span> and <span class="code">read()</span>
    344 methods will also timeout if the child does not generate any output for
    345 a given amount of time. If this happens they will raise a <span
    346  class="code">TIMEOUT</span> exception. You can have these method
    347 ignore a timeout and block indefinitely by passing None for the timeout
    348 parameter.</p>
    349 <blockquote>
    350   <p class="code">child.expect(pexpect.EOF, timeout=None)</p>
    351 </blockquote>
    352 <hr noshade="noshade" size="1">
    353 <h1><a name="faq"></a>FAQ</h1>
    354 <p><b>Q: Why don't shell pipe and redirect (| and >) work when I
    355 spawn a command?</b></p>
    356 <p>
    357 
    358 A: Remember that Pexpect does NOT interpret shell meta characters such as
    359 redirect, pipe, or wild cards (&gt;, |, or *). That's done by a shell not the
    360 command you are spawning. This is a common mistake. If you want to run a
    361 command and pipe it through another command then you must also start a shell.
    362 For example:
    363 
    364 <pre>
    365     child = pexpect.spawn('/bin/bash -c "ls -l | grep LOG &gt; log_list.txt"')
    366     child.expect(pexpect.EOF)
    367 </pre>
    368 
    369 The second form of spawn (where you pass a list of arguments) is useful in
    370 situations where you wish to spawn a command and pass it its own argument list.
    371 This can make syntax more clear. For example, the following is equivalent to
    372 the previous example:
    373 
    374 <pre>
    375     shell_cmd = 'ls -l | grep LOG &gt; log_list.txt'
    376     child = pexpect.spawn ('/bin/bash', ['-c', shell_cmd])
    377     child.expect (pexpect.EOF)
    378 </pre>
    379 
    380 </p>
    381 <p><b>Q: Isn't there already a Python Expect?</b></p>
    382 <p>A: Yes, there are several of them. They usually require you to
    383 compile C. I wanted something that was pure Python and preferably a
    384 single module that was simple to install. I also wanted something that
    385 was easy to use. This pure Python expect only recently became possible
    386 with the introduction of the pty module in the standard Python library.
    387 Previously C extensions were required.</p>
    388 
    389 <p><strong>Q: The before and after properties sound weird.</strong></p>
    390 <p>Originally I was going to model Pexpect more after Expect, but then
    391 I found that I could never remember how to get the context of the stuff
    392 I was trying to parse. I hate having to read my own documentation. I
    393 decided that it was easier for me to remember what before and after
    394 was. It just so happens that this is how the -B and -A options in grep
    395 works, so that made it even easier for me to remember. Whatever makes
    396 my life easier is what's best.</p>
    397 
    398 <p><b>Q: Why not just use Expect?</b></p>
    399 <p>A: I love it. It's great. I has bailed me out of some real jams, but
    400 I wanted something that would do 90% of what I need from Expect; be 10%
    401 of the size; and allow me to write my code in Python instead of TCL.
    402 Pexpect is not nearly as big as Expect, but Pexpect does everything I
    403 have ever used Expect for.
    404 <!-- :-P If I liked TCL then you wouldn't be reading this. My appologies to Don Libes -- Expect is cool, TK is cool, but TCL is only slightly better than Perl in my book. Hopefully after Expyct is done I will not need to use Expect anymore -- except for that lovely autoexpect tool. Damn, I wish I had that! --> </p>
    405 
    406 <p><b>Q: Why not just use a pipe (popen())?</b></p>
    407 <p>A: A pipe works fine for getting the output to non-interactive
    408 programs. If you just want to get the output from <span class="code">ls</span>,
    409 <span class="code">uname</span>, or <span class="code">ping</span>
    410 then this works. Pipes do not work very well for interactive programs
    411 and pipes will almost certainly fail for most applications that ask for
    412 passwords such as telnet, ftp, or ssh.</p>
    413 <p>There are two reasons for this. </p>
    414 <p>First an application may bypass stdout and print directly to its
    415 controlling TTY. Something like SSH will do this when it asks you for a
    416 password. This is why you cannot redirect the password prompt because
    417 it does not go through stdout or stderr.</p>
    418 <p>The second reason is because most applications are built using the C
    419 Standard IO Library (anything that uses <span class="code">#include
    420 &lt;stdio.h&gt;</span>). One of the features of the stdio library is
    421 that it buffers all input and output. Normally output is <b><i>line
    422 buffered</i></b> when a program is printing to a TTY (your terminal
    423 screen). Every time the program prints a line-feed the currently
    424 buffered data will get printed to your screen. The problem comes when
    425 you connect a pipe. The stdio library is smart and can tell that it is
    426 printing to a pipe instead of a TTY. In that case it switches from line
    427 buffer mode to <i><b>block buffered</b></i>. In this mode the
    428 currently buffered data is flushed when the buffer is full. This causes
    429 most interactive programs to deadlock. Block buffering is more
    430 efficient when writing to disks and pipes. Take the situation where a
    431 program prints a message "Enter your user name:\n" and then waits for
    432 you type type something. In block buffered mode, the stdio library will
    433 not put the message into the pipe even though a linefeed is printed.
    434 The result is that you never receive the message, yet the child
    435 application will sit and wait for you to type a response. Don't confuse
    436 the stdio lib's buffer with the pipe's buffer. The pipe buffer is
    437 another area that can cause problems. You could flush the input side of
    438 a pipe, whereas you have no control over the stdio library buffer. </p>
    439 <p>More information: the Standard IO library has three states for a
    440 FILE *. These are: _IOFBF for block buffered; _IOLBF for line buffered;
    441 and _IONBF for unbuffered. The STDIO lib will use block buffering when
    442 talking to a block file descriptor such as a pipe. This is usually not
    443 helpful for interactive programs. Short of recompiling your program to
    444 include fflush() everywhere or recompiling a custom stdio library there
    445 is not much a controlling application can do about this if talking over
    446 a pipe.</p>
    447 <p> The program may have put data in its output that remains unflushed
    448 because the output buffer is not full; then the program will go and
    449 deadlock while waiting for input -- because you never send it any
    450 because you are still waiting for its output (still stuck in the
    451 STDIO's output buffer).</p>
    452 <p>The answer is to use a pseudo-tty. A TTY device will force <i><b>line</b></i>
    453 buffering (as opposed to block buffering). Line buffering means that
    454 you will get each line when the child program sends a line feed. This
    455 corresponds to the way most interactive programs operate -- send a line
    456 of output then wait for a line of input.</p>
    457 <p>I put "answer" in quotes because it's ugly solution and because
    458 there is no POSIX standard for pseudo-TTY devices (even though they
    459 have a TTY standard...). What would make more sense to me would be to
    460 have some way to set a mode on a file descriptor so that it will tell
    461 the STDIO to be line-buffered. I have investigated, and I don't think
    462 there is a way to set the buffered state of a child process. The STDIO
    463 Library does not maintain any external state in the kernel or whatnot,
    464 so I don't think there is any way for you to alter it. I'm not quite
    465 sure how this line-buffered/block-buffered state change happens
    466 internally in the STDIO library. I think the STDIO lib looks at the
    467 file descriptor and decides to change behavior based on whether it's a
    468 TTY or a block file (see isatty()).</p>
    469 <p>I hope that this qualifies as helpful.</p>
    470 
    471 <h1>Don't use a pipe to control another application...</h1>
    472 <p>Pexpect may seem similar to <span class="code">os.popen()</span> or
    473 <span class="code">commands</span> module. The main difference is that
    474 Pexpect (like Expect) uses a pseudo-TTY to talk to the child
    475 application. Most applications do no work well through the system()
    476 call or through pipes. And probably all applications that ask a user to
    477 type in a password will fail. These applications bypass the stdin and
    478 read directly from the TTY device. Many applications do not explicitly
    479 flush their output buffers. This causes deadlocks if you try to control
    480 an interactive application using a pipe. What happens is that most UNIX
    481 applications use the stdio (#include &lt;stdio.h&gt;) for input and
    482 output. The stdio library behaves differently depending on where the
    483 output is going. There is no way to control this behavior from the
    484 client end.<br>
    485 </p>
    486 
    487 <p><b>Q: Can I do screen scraping with this thing?</b></p>
    488 <p>A: That depends. If your application just does line-oriented output
    489 then this is easy. If it does screen-oriented output then it may work,
    490 but it could be hard. For example, trying to scrape data from the 'top'
    491 command would be hard. The top command repaints the text window. </p>
    492 <p>I am working on an ANSI / VT100 terminal emulator that will have
    493 methods to get characters from an arbitrary X,Y coordinate of the
    494 virtual screen. It works and you can play with it, but I have no
    495 working examples at this time.</p>
    496 <hr noshade="noshade" size="1">
    497 <h1><a name="bugs"></a>Bugs</h1>
    498 <h2>Threads</h2>
    499 <p>On Linux (RH 8) you cannot spawn a child from a different thread and
    500 pass the handle back to a worker thread. The child is successfully
    501 spawned but you can't interact with it. The only way to make it work is
    502 to spawn and interact with the child all in the same thread. [Adam
    503 Kerrison] </p>
    504 <h2><a name="echo_bug"></a>Timing issue with send() and sendline()</h2>
    505 <p>This problem has been addressed and should not effect most users.</p>
    506 <p>It is sometimes possible to read an echo of the string sent with <span
    507  class="code">send()</span> and <span class="code">sendline()</span>.
    508 If you call <span class="code">sendline()</span> and then immediately
    509 call <span class="code">readline()</span> you may get part of your
    510 output echoed back. You may read back what you just wrote even if the
    511 child application does not explicitly echo it. Timing is critical. This
    512 could be a security issue when talking to an application that asks for
    513 a password; otherwise, this does not seem like a big deal. <i>But why
    514 do TTYs do this</i>?</p>
    515 <p>People usually report this when they are trying to control SSH or
    516 some other login. For example, if your code looks something like this: </p>
    517 <pre class="code">child.expect ('[pP]assword:')<br>child.sendline (my_password)</pre>
    518 <p><br>
    519 <blockquote>
    520 1. SSH prints "password:" prompt to the user.<br>
    521 2. SSH turns off echo on the TTY device.<br>
    522 3. SSH waits for user to enter a password.<br>
    523 </blockquote>
    524 When scripting with Pexpect what can happen is that Pexpect will response to the "password:" prompt
    525 before SSH has had time to turn off TTY echo. In other words, Pexpect sends the password between
    526 steps 1. and 2., so the password gets echoed back to the TTY. I would call this an SSH bug.
    527 </p>
    528 <p>
    529 Pexpect now automatically adds a short delay before sending data to a child process.
    530 This more closely mimics what happens in the usual human-to-app interaction.
    531 The delay can be tuned with the 'delaybeforesend' attribute of the spawn class.
    532 In general, this fixes the problem for everyone and so this should not be an issue
    533 for most users. For some applications you might with to turn it off.
    534     child = pexpect.spawn ("ssh user (a] example.com")
    535     child.delaybeforesend = 0
    536 </p>
    537 <p><br>
    538 </p>
    539 <p>Try changing it to look like the following. I know that this fix
    540 does not look correct, but it works. I have not figured out exactly
    541 what is happening. You would think that the sleep should be after the
    542 sendline(). The fact that the sleep helps when it's between the
    543 expect() and the sendline() must be a clue.</p>
    544 <pre class="code">child.expect ('[pP]assword:')<br>child.sendline (my_password)</pre>
    545 <h2>Timing issue with isalive()</h2>
    546 <p>Reading the state of isalive() immediately after a child exits may
    547 sometimes return 1. This is a race condition. The child has closed its
    548 file descriptor, but has not yet fully exited before Pexpect's
    549 isalive() executes. Addings a slight delay before the isalive() will
    550 help. In the following example <span class="code">isalive()</span>
    551 sometimes returns 1:</p>
    552 <blockquote>
    553   <pre class="code">child = pexpect.spawn('ls')<br>child.expect(pexpect.EOF)<br>print child.isalive()</pre>
    554 </blockquote>
    555 <p>But if there is any delay before the call to <span class="code">isalive()</span>
    556 then it will always return 0 as expected.</p>
    557 <blockquote>
    558   <pre class="code">child = pexpect.spawn('ls')<br>child.expect(pexpect.EOF)<br>time.sleep(0.1)<br>print child.isalive()</pre>
    559 </blockquote>
    560 
    561 <h2>Truncated output just before child exits</h2>
    562 <p><i>So far I have seen this only on older versions of <b>Apple's MacOS X</b>.</i>
    563 If the child application quits it may not flush its output buffer. This
    564 means that your Pexpect application will receive an EOF even though it
    565 should have received a little more data before the child died. This is
    566 not generally a problem when talking to interactive child applications.
    567 One example where it is a problem is when trying to read output from a
    568 program like '<span class="code">ls</span>'. You may receive most of
    569 the directory listing, but the last few lines will get lost before you
    570 receive an EOF. The reason for this is that '<span class="code">ls</span>'
    571 runs; completes its task; and then exits. The buffer is not flushed
    572 before exit so the last few lines are lost. The following example
    573 demonstrates the problem:</p>
    574 <p> </p>
    575 <blockquote>
    576   <pre class="code">child = pexpect.spawn ('ls -l')<br>child.expect (pexpect.EOF)<br>print child.before <br>  </pre>
    577 </blockquote>
    578 <p></p>
    579 
    580 <h2>Controlling SSH on Solaris</h2>
    581 <p>Pexpect does not yet work perfectly on Solaris.
    582 One common problem is that SSH sometimes will not allow TTY password
    583 authentication. For example, you may expect SSH to ask you for a
    584 password using code like this:
    585 </p>
    586 <pre class="code">child = pexpect.spawn ('ssh user (a] example.com')<br>child.expect ('assword')<br>child.sendline ('mypassword')<br></pre>
    587 You may see the following error come back from a spawned
    588 child SSH:
    589 <p></p>
    590 <blockquote>Permission denied (publickey,keyboard-interactive). </blockquote>
    591 <p>
    592 This means that SSH thinks it can't access the TTY to ask you for your
    593 password.
    594 The only solution I have found is to use public key authentication with
    595 SSH.
    596 This bypasses the need for a password. I'm not happy with this
    597 solution.
    598 The problem is due to poor support for Solaris Pseudo TTYs in the
    599 Python
    600 Standard Library. </p>
    601 <hr noshade="noshade" size="1">
    602 <h1><a name="changes"></a>CHANGES</h1>
    603 <h2>Current Release</h2>
    604 <p>Fixed OSError exception when a pexpect object is cleaned up.
    605 Previously you might have seen this exception:</p>
    606 <blockquote>
    607   <pre class="code">Exception exceptions.OSError: (10, 'No child processes') <br>in &lt;bound method spawn.__del__ of<br>&lt;pexpect.spawn instance at 0xd248c&gt;&gt; ignored</pre>
    608 </blockquote>
    609 <p>You should not see that anymore. Thanks to Michael Surette.</p>
    610 <p>Added support for buffering reads. This greatly improves speed when
    611 trying to match long output from a child process. When you create an
    612 instance of the spawn object you can then set a buffer size. For now
    613 you MUST do the following to turn on buffering -- it may be on by
    614 default in future version.</p>
    615 <blockquote>
    616   <pre class="code">child = pexpect.spawn ('my_command')<br>child.maxread=1000 # Sets buffer to 1000 characters.</pre>
    617 </blockquote>
    618 <div>
    619 <p>I made a subtle change to the way TIMEOUT and EOF exceptions behave.
    620 Previously you could either expect these states in which case pexpect
    621 will not raise an exception, or you could just let pexpect raise an
    622 exception when these states were encountered. If you expected the
    623 states then the 'before' property was set to everything before the
    624 state was encountered, but if you let pexpect raise the exception then
    625 'before' was not set. Now the 'before' property will get set either way
    626 you choose to handle these states.</p>
    627 <h2><i>Older changes...</i></h2>
    628 <p>The spawn object now provides iterators for a <i>file-like interface</i>.
    629 This makes Pexpect a more complete file-like object. You can now write
    630 code like this:</p>
    631 <blockquote>
    632   <pre class="code">child = pexpect.spawn ('ls -l')<br>for line in child:<br>    print line<br></pre>
    633 </blockquote>
    634 <p>I added the attribute <span class="code">exitstatus</span>. This
    635 will give the exit code returned by the child process. This will be set
    636 to <span class="code">None</span> while the child is still alive. When
    637 <span class="code">isalive()</span> returns 0 then <span class="code">exitstatus</span>
    638 will be set.</p>
    639 <p>I made a few more tweaks to <span class="code">isalive()</span> so
    640 that it will operate more consistently on different platforms. Solaris
    641 is the most difficult to support.</p>
    642 <p>&nbsp;</p>
    643 <p>You can now put <span class="code">TIMEOUT</span> in a list of
    644 expected patterns. This is just like putting <span class="code">EOF</span>
    645 in the pattern list. Expecting for a <span class="code">TIMEOUT</span>
    646 may not be used as often as <span class="code">EOF</span>, but this
    647 makes Pexpect more consitent.</p>
    648 <p>Thanks to a suggestion and sample code from Chad J. Schroeder I
    649 added the ability for Pexpect to operate on a file descriptor that is
    650 already open. This means that Pexpect can be used to control streams
    651 such as those from serial port devices. Now you just pass the integer
    652 file descriptor as the "command" when contsructing a spawn open. For
    653 example on a Linux box with a modem on ttyS1:</p>
    654 <blockquote>
    655   <pre class="code">fd = os.open("/dev/ttyS1", os.O_RDWR|os.O_NONBLOCK|os.O_NOCTTY)<br>m = pexpect.spawn(fd) # Note integer fd is used instead of usual string.<br>m.send("+++") # Escape sequence<br>m.send("ATZ0\r") # Reset modem to profile 0<br>rval = m.expect(["OK", "ERROR"])</pre>
    656 </blockquote>
    657 <h3>Pexpect now tests itself on Compile Farm!</h3>
    658 <p>I wrote a nice script that uses ssh to connect to each machine on
    659 Source Forge's Compile Farm and then run the testall.py script for each
    660 platform. The result of the test is then recorded for each platform.
    661 Now it's easy to run regression tests across multiple platforms.</p>
    662 <h3>Pexpect is a file-like object</h3>
    663 <p>The spawn object now provides a <i>file-like interface</i>. It
    664 supports most of the methods and attributes defined for Python File
    665 Objects. </p>
    666 <p>I changed write and writelines() so that they no longer return a
    667 value. Use send() if you need that functionality. I did this to make
    668 the Spawn object more closely match a file-like object.</p>
    669 <p>read() was renamed to read_nonblocking(). I added a new read()
    670 method that matches file-like object interface. In general, you should
    671 not notice the difference except that read() no longer allows you to
    672 directly set the timeout value. I hope this will not effect any
    673 existing code. Switching to read_nonblocking() should fix existing code.</p>
    674 <p>I changed the name of <span class="code">set_echo()</span> to <span
    675  class="code">setecho()</span>.</p>
    676 <p>I changed the name of <span class="code">send_eof()</span> to <span
    677  class="code">sendeof()</span>.</p>
    678 <p>I modified <span class="code">kill()</span> so that it checks to
    679 make sure the pid isalive().</p>
    680 <p>I modified <span class="code">spawn()</span> (really called from <span
    681  class="code">__spawn()</span>)so that it does not raise an expection
    682 if <span class="code">setwinsize()</span> fails. Some platforms such
    683 as Cygwin do not like setwinsize. This was a constant problem and since
    684 it is not a critical feature I decided to just silence the error.
    685 Normally I don't like to do that, but in this case I'm making an
    686 exception.</p>
    687 <p>Added a method <span class="code">close()</span> that does what you
    688 think. It closes the file descriptor of the child application. It makes
    689 no attempt to actually kill the child or wait for its status. </p>
    690 <p>Add variables <span class="code">__version__</span> and <span
    691  class="code">__revision__</span> (from cvs) to the pexpect modules.
    692 This is mainly helpful to me so that I can make sure that I'm testing
    693 with the right version instead of one already installed.</p>
    694 <h3>Logging changes</h3>
    695 <blockquote>
    696   <p><span class="code">log_open()</span> and <span class="code">log_close()</span>
    697 have been removed. Now use <span class="code">setlog()</span>. The <span
    698  class="code">setlog()</span> method takes a file object. This is far
    699 more flexible than the previous log method. Each time data is written
    700 to the file object it will be flushed. To turn logging off simply call <span
    701  class="code">setlog()</span> with None.</p>
    702 </blockquote>
    703 <h2>isalive changes</h2>
    704 <blockquote>
    705   <p>I renamed the <span class="code">isAlive()</span> method to <span
    706  class="code">isalive()</span> to match the more typical naming style
    707 in Python. Also the technique used to detect child process status has
    708 been drastically modified. Previously I did some funky stuff with
    709 signals which caused indigestion in other Python modules on some
    710 platforms. It's was a big headache. It still is, but I think it works
    711 better now.</p>
    712 </blockquote>
    713 <h3>attribute name changes</h3>
    714 <blockquote>
    715   <p>The names of some attributes have been changed. This effects the
    716 names of the attributes that are set after called the <span
    717  class="code">expect()</span> method.</p>
    718   <table class="pymenu" border="0" cellpadding="5">
    719     <tbody>
    720       <tr>
    721         <th class="pymenu">NEW NAME</th>
    722         <th class="pymenu">OLD NAME</th>
    723       </tr>
    724       <tr>
    725         <td><span class="code">before</span><br>
    726         <i>Everything before the match.</i></td>
    727         <td><span class="code">before</span></td>
    728       </tr>
    729       <tr>
    730         <td><span class="code">after</span><br>
    731         <i>Everything after and including the first character of the
    732 match</i></td>
    733         <td><span class="code">matched</span></td>
    734       </tr>
    735       <tr>
    736         <td><span class="code">match</span><br>
    737         <i>This is the re MatchObject from the match.<br>
    738 You can get groups() from this.<br>
    739 See '<span class="code">uptime.py</span>' in the examples tar ball.</i></td>
    740         <td><i>New -- Did not exist</i></td>
    741       </tr>
    742     </tbody>
    743   </table>
    744 </blockquote>
    745 <h3>EOF changes</h3>
    746 <blockquote>
    747   <p>The <span class="code">expect_eof()</span> method is gone. You
    748 can now simply use the <span class="code">expect()</span> method to
    749 look for EOF.</p>
    750   <p>Was:</p>
    751   <blockquote>
    752     <p><span class="code">p.expect_eof ()</span></p>
    753   </blockquote>
    754   <p>Now:</p>
    755   <blockquote>
    756     <p><span class="code">p.expect (pexpect.EOF)</span></p>
    757   </blockquote>
    758 </blockquote>
    759 <hr noshade="noshade" size="1">
    760 <h1><a name="testing"></a>TESTING</h1>
    761 <p>The following platforms have been tested:</p>
    762 <!--
    763 <table class="pymenu" border="0" cellpadding="5">
    764   <tbody>
    765     <tr>
    766       <th class="pymenu">PLATFORM</th>
    767       <th class="pymenu">RESULTS</th>
    768     </tr>
    769     <tr>
    770       <td>Linux 2.4.9-ac10-rmk2-np1-cerf2<br>
    771 armv4l</td>
    772       <td><b><i>all tests passed</i></b></td>
    773     </tr>
    774     <tr>
    775       <td>Linux 2.4.18 #2<br>
    776 sparc64</td>
    777       <td><b><i>all tests passed</i></b></td>
    778     </tr>
    779     <tr>
    780       <td>MacOS X Darwin Kernel Version 5.5<br>
    781 powerpc</td>
    782       <td>
    783       <p>failed more than one test.</p>
    784       <p>Generally Pexpect works on OS X, but the nature of the quirks
    785 cause a many of the tests to fail. See <a href="#bugs">bugs</a>
    786 (Incomplete Child Output). The problem is more than minor, but Pexpect
    787 is still more than useful for most tasks. The problem is an edge case.</p>
    788       </td>
    789     </tr>
    790     <tr>
    791       <td>Linux 2.2.20<br>
    792 alpha<br>
    793       </td>
    794       <td><b><i>all tests passed</i></b></td>
    795     </tr>
    796     <tr>
    797       <td>Linux 2.4.18-5smp<br>
    798 i686</td>
    799       <td><b><i>all tests passed</i></b></td>
    800     </tr>
    801     <tr>
    802       <td>OpenBSD 2.9 GENERIC#653<br>
    803 i386</td>
    804       <td><b><i>all tests passed</i></b></td>
    805     </tr>
    806     <tr>
    807       <td>Solaris</td>
    808       <td>
    809       <p>failed <span class="code">test_destructor</span></p>
    810       <p>Otherwise, this is working pretty well. The destructor problem
    811 is minor. For some reason, the <i>second</i> time a pty file
    812 descriptor is created and deleted it never gets returned for use. It
    813 does not effect the first time or the third time or any time after
    814 that. It's only the second time. This is weird... This could be a file
    815 descriptor leak, or it could be some peculiarity of how Solaris
    816 recycles them. I thought it was a UNIX requirement for the OS to give
    817 you the lowest available filedescriptor number. In any case, this
    818 should not be a problem unless you create hundreds of pexpect
    819 instances... It may also be a pty module bug. </p>
    820       </td>
    821     </tr>
    822     <tr>
    823       <td>Windows XP Cygwin</td>
    824       <td>failed <span class="code">test_destructor</span>. That it
    825 works at all is amazing to me. Cygwin rules!</td>
    826     </tr>
    827   </tbody>
    828 </table>
    829 -->
    830 <h1>&nbsp;</h1>
    831 <h1><a name="todo">TO DO</a></h1>
    832 <p>Add an option to add a delay after each expect() or before each
    833 read()/readline() call to automatically avoid the <a href="#echo_bug">echo
    834 bug</a>.</p>
    835 <p>&nbsp;</p>
    836 </div>
    837 <hr noshade="noshade" size="1">
    838 <table border="0">
    839   <tbody>
    840     <tr>
    841       <td> <a href="http://www.noah.org/email/"><img src="email.png"
    842  alt="Click to send email." border="0" height="16" width="100"></a> </td>
    843     </tr>
    844   </tbody>
    845 </table>
    846 </div>
    847 <div id="Menu"><b>INDEX</b><br>
    848 <hr noshade="noshade" size="1"> <a href="#license"
    849  title="Python Software Foundation License">License</a><br>
    850 <a href="#download" title="Download and setup instructions">Download</a><br>
    851 <a href="#doc" title="Documentation and overview">Documentation</a><br>
    852 <a href="#status" title="Project Status">Project Status</a><br>
    853 <a href="#requirements" title="System requirements to use Pexpect">Requirements</a><br>
    854 <a href="#overview" title="Overview of what Pexpect does">Overview</a><br>
    855 <a href="#faq" title="FAQ">FAQ</a><br>
    856 <a href="#bugs" title="Bugs and work-arounds">Known Bugs</a><br>
    857 <a href="#changes" title="What's new with Pexpect">Recent Changes</a><br>
    858 <a href="#testing" title="Test results on various platforms">Testing</a><br>
    859 <a href="#todo" title="What to do next">To do</a><br>
    860 <a href="http://pexpect.svn.sourceforge.net/viewvc/pexpect/trunk/pexpect/" title="browse SVN">Browse SVN</a><br>
    861 <br>
    862 <a href="http://sourceforge.net/projects/pexpect/"
    863  title="The Pexpect project page on SourceForge.net"> <img
    864  src="http://sourceforge.net/sflogo.php?group_id=59762&type=5"
    865  alt="The Pexpect project page on SourceForge.net" border="0"
    866  height="31" width="105"> </a> </div>
    867 </body>
    868 </html>
    869