fate.texi 4.22 KB
Newer Older
Luca Barbato's avatar
Luca Barbato committed
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
\input texinfo @c -*- texinfo -*-

@settitle FATE Automated Testing Environment
@titlepage
@center @titlefont{FATE Automated Testing Environment}
@end titlepage

@top

@contents

@chapter Introduction

FATE provides a regression testsuite embedded within the Libav build system.
It can be run locally and optionally configured to send reports to a web
aggregator and viewer @url{http://fate.libav.org}.

It is advised to run FATE before submitting patches to the current codebase
and provide new tests when submitting patches to add additional features.

@chapter Running FATE

@section Samples and References
In order to run, FATE needs a large amount of data (samples and references)
that is provided separately from the actual source distribution.

To inform the build system about the testsuite location, pass
@option{--samples=<path to the samples>} to @command{configure} or set the
29
@var{SAMPLES} Make variable or the @var{LIBAV_SAMPLES} environment variable
Luca Barbato's avatar
Luca Barbato committed
30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77
to a suitable value.

The dataset is available through @command{rsync}, is possible to fetch
the current sample using the straight rsync command or through a specific
@ref{Makefile target}.

@example
# rsync -aL rsync://fate-suite.libav.org/fate-suite/ fate-suite
@end example

@example
# make fate-rsync SAMPLES=fate-suite
@end example


@chapter Manual Run
FATE regression test can be run through @command{make}.
Specific Makefile targets and Makefile variables are available:

@anchor{Makefile target}
@section FATE Makefile targets
@table @option
@item fate-list
List all fate/regression test targets.
@item fate-rsync
Shortcut to download the fate test samples to the specified testsuite location.
@item fate
Run the FATE test suite (requires the fate-suite dataset).
@end table

@section Fate Makefile variables
@table @option
@item V
Verbosity level, can be set to 0, 1 or 2.
@table @option
    @item 0
    show just the test arguments
    @item 1
    show just the command used in the test
    @item 2
    show everything
@end table
@item SAMPLES
Specify or override the path to the FATE samples at make time, it has a
meaning only while running the regression tests.
@item THREADS
Specify how many threads to use while running regression tests, it is
quite useful to detect thread-related regressions.
78 79
@item CPUFLAGS
Specify a mask to be applied to autodetected CPU flags.
Luca Barbato's avatar
Luca Barbato committed
80 81 82
@end table

@example
83
    make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate
Luca Barbato's avatar
Luca Barbato committed
84 85 86 87 88 89
@end example

@chapter Automated Tests
In order to automatically testing specific configurations, e.g. multiple
compilers, @command{tests/fate.sh} is provided.

90 91 92
This shell script builds Libav, runs the regression tests and prepares
a report that can be sent to @url{http://fate.libav.org/} or directly
examined locally.
Luca Barbato's avatar
Luca Barbato committed
93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137

@section Testing Profiles
The configuration file passed to @command{fate.sh} is shell scripts as well.

It must provide at least a @var{slot} identifier, the @var{repo} from
which fetch the sources, the @var{samples} directory, a @var{workdir} with
enough space to build and run all the tests.
Optional submit command @var{fate_recv} and a @var{comment} to describe
the testing profile are available.

Additional optional parameter to tune the Libav building and reporting process
can be passed.

@example
slot=                                   # some unique identifier
repo=git://git.libav.org/libav.git      # the source repository
samples=/path/to/fate/samples
workdir=                                # directory in which to do all the work
fate_recv="ssh -T fate@@fate.libav.org"  # command to submit report
comment=                                # optional description

# the following are optional and map to configure options
arch=
cpu=
cross_prefix=
cc=
target_os=
sysroot=
target_exec=
target_path=
extra_cflags=
extra_ldflags=
extra_libs=
extra_conf=     # extra configure options not covered above

#make=          # name of GNU make if not 'make'
makeopts=       # extra options passed to 'make'
#tar=           # command to create a tar archive from its arguments on
                # stdout, defaults to 'tar c'
@end example

@section Submitting Reports
In order to send reports you need to create an @command{ssh} key and send it
to @email{root@@libav.org}.
The current server fingerprint is @var{a4:99:d7:d3:1c:92:0d:56:d6:d5:61:be:01:ae:7d:e6}