Home | History | Annotate | only in /external/curl/tests
Up to higher level directory
NameDateSize
.gitignore21-Oct-2016257
certs/21-Oct-2016
CMakeLists.txt21-Oct-201674
convsrctest.pl21-Oct-20167.4K
data/21-Oct-2016
directories.pm21-Oct-20167.7K
extern-scan.pl21-Oct-20161.5K
FILEFORMAT21-Oct-201616.2K
ftp.pm21-Oct-20169.7K
ftpserver.pl21-Oct-201686.4K
getpart.pm21-Oct-20167.1K
http_pipe.py21-Oct-201613.7K
httpserver.pl21-Oct-20163.7K
keywords.pl21-Oct-20163.3K
libtest/21-Oct-2016
Makefile.am21-Oct-20162.7K
mem-include-scan.pl21-Oct-20162.6K
memanalyze.pl21-Oct-201611.1K
README21-Oct-20169.6K
rtspserver.pl21-Oct-20162.9K
runtests.121-Oct-20165.4K
runtests.pl21-Oct-2016164.1K
secureserver.pl21-Oct-201610.3K
server/21-Oct-2016
serverhelp.pm21-Oct-20167.7K
sshhelp.pm21-Oct-201611.8K
sshserver.pl21-Oct-201637.5K
stunnel.pem21-Oct-20166.9K
symbol-scan.pl21-Oct-20164.6K
testcurl.121-Oct-20165.1K
testcurl.pl21-Oct-201621.5K
tftpserver.pl21-Oct-20163K
unit/21-Oct-2016
valgrind.pm21-Oct-20163.8K
valgrind.supp21-Oct-20161.7K

README

      1                                   _   _ ____  _
      2                               ___| | | |  _ \| |
      3                              / __| | | | |_) | |
      4                             | (__| |_| |  _ <| |___
      5                              \___|\___/|_| \_\_____|
      6 
      7 The cURL Test Suite
      8 
      9  1. Running
     10   1.1 Requires to run
     11   1.2 Port numbers used by test servers
     12   1.3 Test servers
     13   1.4 Run
     14   1.5 Shell startup scripts
     15   1.6 Memory test
     16   1.7 Debug
     17   1.8 Logs
     18   1.9 Test input files
     19   1.10 Code coverage
     20   1.11 Remote testing
     21 
     22  2. Numbering
     23   2.1 Test case numbering
     24 
     25  3. Write tests
     26   3.1 test data
     27   3.2 curl tests
     28   3.3 libcurl tests
     29   3.4 unit tests
     30 
     31  4. TODO
     32   4.1 More protocols
     33   4.2 SOCKS auth
     34 
     35 ==============================================================================
     36 
     37 1. Running
     38 
     39  1.1 Requires to run
     40 
     41   perl (and a unix-style shell)
     42   python (and a unix-style shell)
     43   diff (when a test fails, a diff is shown)
     44   stunnel (for HTTPS and FTPS tests)
     45   OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
     46 
     47  1.2 Port numbers used by test servers
     48 
     49   - TCP/8990 for HTTP
     50   - TCP/8991 for HTTPS
     51   - TCP/8992 for FTP
     52   - TCP/8993 for FTPS
     53   - TCP/8994 for HTTP IPv6
     54   - TCP/8995 for FTP (2)
     55   - TCP/8996 for FTP IPv6
     56   - UDP/8997 for TFTP
     57   - UDP/8998 for TFTP IPv6
     58   - TCP/8999 for SCP/SFTP
     59   - TCP/9000 for SOCKS
     60   - TCP/9001 for POP3
     61   - TCP/9002 for IMAP
     62   - TCP/9003 for SMTP
     63   - TCP/9004 for SMTP IPv6
     64   - TCP/9005 for RTSP
     65   - TCP/9006 for RTSP IPv6
     66   - TCP/9007 for GOPHER
     67   - TCP/9008 for GOPHER IPv6
     68   - TCP/9008 for HTTPS server with TLS-SRP support
     69 
     70  1.3 Test servers
     71 
     72   The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone
     73   servers on the ports listed above to which it makes requests. For SSL tests,
     74   it runs stunnel to handle encryption to the regular servers. For SSH, it
     75   runs a standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform
     76   the SOCKS functionality and requires a SSH client and server.
     77 
     78   The base port number (8990), which all the individual port numbers are
     79   indexed from, can be set explicitly using runtests.pl' -b option to allow
     80   running more than one instance of the test suite simultaneously on one
     81   machine, or just move the servers in case you have local services on any of
     82   those ports.
     83 
     84   The HTTP server supports listening on a Unix domain socket, the default
     85   location is 'http.sock'.
     86 
     87  1.4 Run
     88 
     89   'make test'. This builds the test suite support code and invokes the
     90   'runtests.pl' perl script to run all the tests. Edit the top variables
     91   of that script in case you have some specific needs, or run the script
     92   manually (after the support code has been built).
     93 
     94   The script breaks on the first test that doesn't do OK. Use -a to prevent
     95   the script from aborting on the first error. Run the script with -v for more
     96   verbose output. Use -d to run the test servers with debug output enabled as
     97   well. Specifying -k keeps all the log files generated by the test intact.
     98 
     99   Use -s for shorter output, or pass test numbers to run specific tests only
    100   (like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case
    101   ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from
    102   3 to 9. Any test numbers starting with ! are disabled, as are any test
    103   numbers found in the files data/DISABLED or data/DISABLED.local (one per
    104   line). The latter is meant for local temporary disables and will be ignored
    105   by git.
    106 
    107   When -s is not present, each successful test will display on one line the
    108   test number and description and on the next line a set of flags, the test
    109   result, current test sequence, total number of tests to be run and an
    110   estimated amount of time to complete the test run. The flags consist of
    111   these letters describing what is checked in this test:
    112 
    113     s stdout
    114     d data
    115     u upload
    116     p protocol
    117     o output
    118     e exit code
    119     m memory
    120     v valgrind
    121 
    122  1.5 Shell startup scripts
    123 
    124   Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
    125   influenced by the output of system wide or user specific shell startup
    126   scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which
    127   output text messages or escape sequences on user login.  When these shell
    128   startup messages or escape sequences are output they might corrupt the
    129   expected stream of data which flows to the sftp-server or from the ssh
    130   client which can result in bad test behaviour or even prevent the test
    131   server from running.
    132 
    133   If the test suite ssh or sftp server fails to start up and logs the message
    134   'Received message too long' then you are certainly suffering the unwanted
    135   output of a shell startup script.  Locate, cleanup or adjust the shell
    136   script.
    137 
    138  1.6 Memory test
    139 
    140   The test script will check that all allocated memory is freed properly IF
    141   curl has been built with the CURLDEBUG define set. The script will
    142   automatically detect if that is the case, and it will use the
    143   'memanalyze.pl' script to analyze the memory debugging output.
    144 
    145   Also, if you run tests on a machine where valgrind is found, the script will
    146   use valgrind to run the test with (unless you use -n) to further verify
    147   correctness.
    148 
    149   runtests.pl's -t option will enable torture testing mode, which runs each
    150   test many times and makes each different memory allocation fail on each
    151   successive run.  This tests the out of memory error handling code to ensure
    152   that memory leaks do not occur even in those situations. It can help to
    153   compile curl with CPPFLAGS=-DMEMDEBUG_LOG_SYNC when using this option, to
    154   ensure that the memory log file is properly written even if curl crashes.
    155 
    156  1.7 Debug
    157 
    158   If a test case fails, you can conveniently get the script to invoke the
    159   debugger (gdb) for you with the server running and the exact same command
    160   line parameters that failed. Just invoke 'runtests.pl <test number> -g' and
    161   then just type 'run' in the debugger to perform the command through the
    162   debugger.
    163 
    164  1.8 Logs
    165 
    166   All logs are generated in the logs/ subdirectory (it is emptied first in the
    167   runtests.pl script). Use runtests.pl -k to force it to keep the temporary
    168   files after the test run since successful runs will clean it up otherwise.
    169 
    170  1.9 Test input files
    171 
    172   All test cases are put in the data/ subdirectory. Each test is stored in the
    173   file named according to the test number.
    174 
    175   See FILEFORMAT for the description of the test case files.
    176 
    177  1.10 Code coverage
    178 
    179   gcc provides a tool that can determine the code coverage figures for
    180   the test suite.  To use it, configure curl with
    181   CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'.  Make sure you run the normal
    182   and torture tests to get more full coverage, i.e. do:
    183 
    184     make test
    185     make test-torture
    186 
    187   The graphical tool ggcov can be used to browse the source and create
    188   coverage reports on *NIX hosts:
    189 
    190     ggcov -r lib src
    191 
    192   The text mode tool gcov may also be used, but it doesn't handle object files
    193   in more than one directory very well.
    194 
    195  1.11 Remote testing
    196 
    197   The runtests.pl script provides some hooks to allow curl to be tested on a
    198   machine where perl can not be run.  The test framework in this case runs on
    199   a workstation where perl is available, while curl itself is run on a remote
    200   system using ssh or some other remote execution method.  See the comments at
    201   the beginning of runtests.pl for details.
    202 
    203 2. Numbering
    204 
    205  2.1 Test case numbering
    206 
    207      1   -  99   HTTP
    208      100 - 199   FTP
    209      200 - 299   FILE
    210      300 - 399   HTTPS
    211      400 - 499   FTPS
    212      500 - 599   libcurl source code tests, not using the curl command tool
    213      600 - 699   SCP/SFTP
    214      700 - 799   SOCKS4 (even numbers) and SOCK5 (odd numbers)
    215      800 - 849   IMAP
    216      850 - 899   POP3
    217      900 - 999   SMTP
    218      1000 - 1299 miscellaneous
    219      1300 - 1399 unit tests
    220      1400 - 1499 miscellaneous
    221      1500 - 1599 libcurl source code tests, not using the curl command tool
    222                  (same as 5xx)
    223      1600 - 1699 unit tests
    224      2000 - x    multiple sequential protocols per test case
    225 
    226   There's nothing in the system that *requires* us to keep within these number
    227   series.
    228 
    229 3. Write tests
    230 
    231   Here's a quick description on writing test cases. We basically have three
    232   kinds of tests: the ones that test the curl tool, the ones that build small
    233   applications and test libcurl directly and the unit tests that test
    234   individual (possibly internal) functions.
    235 
    236  3.1 test data
    237 
    238   Each test has a master file that controls all the test data. What to read,
    239   what the protocol exchange should look like, what exit code to expect and
    240   what command line arguments to use etc.
    241 
    242   These files are tests/data/test[num] where [num] is described in section 2
    243   of this document, and the XML-like file format of them is described in the
    244   separate tests/FILEFORMAT document.
    245 
    246  3.2 curl tests
    247 
    248   A test case that runs the curl tool and verifies that it gets the correct
    249   data, it sends the correct data, it uses the correct protocol primitives
    250   etc.
    251 
    252  3.3 libcurl tests
    253 
    254   The libcurl tests are identical to the curl ones, except that they use a
    255   specific and dedicated custom-built program to run instead of "curl". This
    256   tool is built from source code placed in tests/libtest and if you want to
    257   make a new libcurl test that is where you add your code.
    258 
    259  3.4 unit tests
    260 
    261   Unit tests are tests in the 13xx sequence and they are placed in tests/unit.
    262   There's a tests/unit/README describing the specific set of checks and macros
    263   that may be used when writing tests that verify behaviors of specific
    264   individual functions.
    265 
    266   The unit tests depend on curl being built with debug enabled.
    267 
    268 4. TODO
    269 
    270  4.1 More protocols
    271 
    272   Add tests for TELNET, LDAP, DICT...
    273 
    274  4.2 SOCKS auth
    275 
    276   SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the
    277   test mechanism) doesn't support them
    278