emacs/test/README

Copyright (C) 2008-2022 Free Software Foundation, Inc.
See the end of the file for license conditions.

This directory contains files intended to test various aspects of
Emacs's functionality.  Please help add tests!

See the file file-organization.org for the details of the directory
structure and file-naming conventions.

For tests in the manual/ subdirectory, look there for separate README
files, or look for instructions in the test files themselves.

Emacs uses ERT, Emacs Lisp Regression Testing, for testing.  See (info
"(ert)") or https://www.gnu.org/software/emacs/manual/html_node/ert/
for more information on writing and running tests.

Tests could be tagged by the developer.  In this test directory, the
following tags are recognized:

* :expensive-test
  The test needs a serious amount of time to run.  It is not intended
  to run on a regular basis by users.  Instead, it runs on demand
  only, or during regression tests.

* :nativecomp
  The test runs only if Emacs is configured with Lisp native compiler
  support.

* :unstable
  The test is under development.  It shall run on demand only.

The Makefile sets the environment variable $EMACS_TEST_DIRECTORY,
which points to this directory.  This environment variable does not
exist when the tests are run outside make.  The Makefile supports the
following targets:

* make check
  Run all tests as defined in the directory.  Expensive and unstable
  tests are suppressed.  The result of the tests for <filename>.el is
  stored in <filename>.log.

* make check-maybe
  Like "make check", but run only the tests for files which have
  unresolved prerequisites.

* make check-expensive
  Like "make check", but run also the tests marked as expensive.

* make check-all
  Like "make check", but run all tests.

* make check-<dirname>
  Like "make check", but run only the tests in test/<dirname>/*.el.
  <dirname> is a relative directory path, which has replaced "/" by "-",
  like in "check-src" or "check-lisp-net".

* make <filename>  -or-  make <filename>.log
  Run all tests declared in <filename>.el.  This includes expensive
  tests.  In the former case the output is shown on the terminal, in
  the latter case the output is written to <filename>.log.

<filename> could be either a relative file name like
"lisp/files-tests", or a package name like "files-tests".

ERT offers selectors, which make it possible to filter out which test
cases shall run.  The make variable $(SELECTOR) gives you a simple
mean to use your own selectors.  The ERT manual describes how
selectors are constructed, see (info "(ert)Test Selectors") or
https://www.gnu.org/software/emacs/manual/html_node/ert/Test-Selectors.html

You could use predefined selectors of the Makefile.  "make <filename>
SELECTOR='$(SELECTOR_DEFAULT)'" runs all tests for <filename>.el
except the tests tagged as expensive or unstable.  Other predefined
selectors are $(SELECTOR_EXPENSIVE) (run all tests except unstable
ones) and $(SELECTOR_ALL) (run all tests).

If your test file contains the tests "test-foo", "test2-foo" and
"test-foo-remote", and you want to run only the former two tests, you
could use a selector regexp (note that the "$" needs to be doubled to
protect against "make" variable expansion):

    make <filename> SELECTOR='"foo$$"'

In case you want to use the symbol name of a test as selector, you can
use it directly:

    make <filename> SELECTOR='test-foo-remote'

Note that although the test files are always compiled (unless they set
no-byte-compile), the source files will be run when expensive or
unstable tests are involved, to give nicer backtraces.  To run the
compiled version of a test use

    make TEST_LOAD_EL=no ...

Some tests might take long time to run.  In order to summarize the
<nn> tests with the longest duration, call

    make SUMMARIZE_TESTS=<nn> ...

The backtrace of failing tests are truncated to the default value of
'ert-batch-backtrace-right-margin'.  To see more of the backtrace, use

    make TEST_BACKTRACE_LINE_LENGTH=<nn> ...

The tests are run in batch mode by default; sometimes it's useful to
get precisely the same environment but run in interactive mode for
debugging.  To do that, use

    make TEST_INTERACTIVE=yes ...

By default, ERT test failure summaries are quite brief in batch
mode--only the names of the failed tests are listed.  If the
$EMACS_TEST_VERBOSE environment variable is set and non-empty, the
failure summaries will also include the data from the failing test.

If the $EMACS_TEST_JUNIT_REPORT environment variable is set to a file
name, a JUnit test report is generated under this name.

Some of the tests require a remote temporary directory
(autorevert-tests.el, dnd-tests.el, filenotify-tests.el,
shadowfile-tests.el and tramp-tests.el).  Per default, a mock-up
connection method is used (this might not be possible when running on
MS Windows).  If you want to test a real remote connection, set
$REMOTE_TEMPORARY_FILE_DIRECTORY to a suitable value in order to
overwrite the default value:

    env REMOTE_TEMPORARY_FILE_DIRECTORY=/ssh:host:/tmp make ...


There are also continuous integration tests on
<https://hydra.nixos.org/jobset/gnu/emacs-trunk> (see
admin/notes/hydra) and <https://emba.gnu.org/emacs/emacs> (see
admin/notes/emba).  Both environments provide an environment variable,
which could be used to determine, whether the tests run in one of
these test environments.

$EMACS_HYDRA_CI indicates the hydra environment, and $EMACS_EMBA_CI
indicates the emba environment, respectively.

If tests on these premises take too long, and it is needed to create a
core dump for further analysis, the environment variable
$EMACS_TEST_TIMEOUT could set a limit (in seconds) when this shall
happen.


(Also, see etc/compilation.txt for compilation mode font lock tests
and etc/grep.txt for grep mode font lock tests.)


This file is part of GNU Emacs.

GNU Emacs is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

GNU Emacs is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.