emacs/test/README

Copyright (C) 2008-2019 Free Software Foundation, Inc.
See the end of the file for license conditions.

This directory contains files intended to test various aspects of
Emacs's functionality.  Please help add tests!

See the file file-organization.org for the details of the directory
structure and file-naming conventions.

Emacs uses ERT, Emacs Lisp Regression Testing, for testing.  See (info
"(ert)") or https://www.gnu.org/software/emacs/manual/html_node/ert/
for more information on writing and running tests.

Tests could be tagged by the developer.  In this test directory, the
following tags are recognized:

* :expensive-test
  The test needs a serious amount of time to run.  It is not intended
  to run on a regular basis by users.  Instead, it runs on demand
  only, or during regression tests.

* :unstable
  The test is under development.  It shall run on demand only.

The Makefile in this directory supports the following targets:

* make check
  Run all tests as defined in the directory.  Expensive and unstable
  tests are suppressed.  The result of the tests for <filename>.el is
  stored in <filename>.log.

* make check-maybe
  Like "make check", but run only the tests for files which have
  unresolved prerequisites.

* make check-expensive
  Like "make check", but run also the tests marked as expensive.

* make check-all
  Like "make check", but run all tests.

* make <filename>  or  make <filename>.log
  Run all tests declared in <filename>.el.  This includes expensive
  tests.  In the former case the output is shown on the terminal, in
  the latter case the output is written to <filename>.log.

ERT offers selectors, which make it possible to filter out which test
cases shall run.  The make variable $(SELECTOR) gives you a simple
mean to use your own selectors.  The ERT manual describes how
selectors are constructed, see (info "(ert)Test Selectors") or
https://www.gnu.org/software/emacs/manual/html_node/ert/Test-Selectors.html

You could use predefined selectors of the Makefile.  "make <filename>
SELECTOR='$(SELECTOR_DEFAULT)'" runs all tests for <filename>.el
except the tests tagged as expensive or unstable.

If your test file contains the tests "test-foo", "test2-foo" and
"test-foo-remote", and you want to run only the former two tests, you
could use a selector regexp (note that the "$" needs to be doubled to
protect against "make" variable expansion):

    make <filename> SELECTOR='"foo$$"'

Note that although the test files are always compiled (unless they set
no-byte-compile), the source files will be run when expensive or
unstable tests are involved, to give nicer backtraces.  To run the
compiled version of a test use

    make TEST_LOAD_EL=no ...

Some tests might take long time to run.  In order to summarize the
<nn> tests with the longest duration, call

    make SUMMARIZE_TESTS=<nn> ...

The tests are run in batch mode by default; sometimes it's useful to
get precisely the same environment but run in interactive mode for
debugging.  To do that, use

    make TEST_INTERACTIVE=yes ...

Some of the tests require a remote temporary directory
(filenotify-tests.el, shadowfile-tests.el and tramp-tests.el).  Per
default, a mock-up connection method is used (this might not be
possible when running on MS Windows).  If you want to test a real
remote connection, set $REMOTE_TEMPORARY_FILE_DIRECTORY to a suitable
value in order to overwrite the default value:

  env REMOTE_TEMPORARY_FILE_DIRECTORY=/ssh:host:/tmp make ...


There are also continuous integration tests on
<https://hydra.nixos.org/jobset/gnu/emacs-trunk> (see
admin/notes/hydra) and <https://emba.gnu.org/emacs/emacs>.  Both
environments provide an environment variable, which could be used to
determine, whether the tests run in one of these test environments.

$EMACS_HYDRA_CI indicates the hydra environment, and $EMACS_EMBA_CI
indicates the emba environment, respectively.


(Also, see etc/compilation.txt for compilation mode font lock tests.)


This file is part of GNU Emacs.

GNU Emacs is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

GNU Emacs is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.