2021-01-01 09:13:56 +00:00
|
|
|
|
Copyright (C) 2008-2021 Free Software Foundation, Inc.
|
2016-02-10 11:18:30 +00:00
|
|
|
|
See the end of the file for license conditions.
|
|
|
|
|
|
2008-02-29 04:27:23 +00:00
|
|
|
|
This directory contains files intended to test various aspects of
|
|
|
|
|
Emacs's functionality. Please help add tests!
|
2009-07-26 15:59:37 +00:00
|
|
|
|
|
2016-09-24 10:32:44 +00:00
|
|
|
|
See the file file-organization.org for the details of the directory
|
|
|
|
|
structure and file-naming conventions.
|
|
|
|
|
|
2021-05-20 10:02:29 +00:00
|
|
|
|
For tests in the manual/ subdirectory, look there for separate README
|
|
|
|
|
files, or look for instructions in the test files themselves.
|
|
|
|
|
|
2016-02-10 11:18:30 +00:00
|
|
|
|
Emacs uses ERT, Emacs Lisp Regression Testing, for testing. See (info
|
|
|
|
|
"(ert)") or https://www.gnu.org/software/emacs/manual/html_node/ert/
|
|
|
|
|
for more information on writing and running tests.
|
|
|
|
|
|
2018-03-16 15:19:10 +00:00
|
|
|
|
Tests could be tagged by the developer. In this test directory, the
|
2018-03-20 01:55:14 +00:00
|
|
|
|
following tags are recognized:
|
2018-03-16 15:19:10 +00:00
|
|
|
|
|
|
|
|
|
* :expensive-test
|
|
|
|
|
The test needs a serious amount of time to run. It is not intended
|
|
|
|
|
to run on a regular basis by users. Instead, it runs on demand
|
|
|
|
|
only, or during regression tests.
|
|
|
|
|
|
|
|
|
|
* :unstable
|
|
|
|
|
The test is under development. It shall run on demand only.
|
|
|
|
|
|
2021-03-16 17:40:24 +00:00
|
|
|
|
The Makefile sets the environment variable $EMACS_TEST_DIRECTORY,
|
|
|
|
|
which points to this directory. This environment variable does not
|
|
|
|
|
exist when the tests are run outside make. The Makefile supports the
|
|
|
|
|
following targets:
|
2016-02-10 11:18:30 +00:00
|
|
|
|
|
|
|
|
|
* make check
|
2018-03-16 15:19:10 +00:00
|
|
|
|
Run all tests as defined in the directory. Expensive and unstable
|
|
|
|
|
tests are suppressed. The result of the tests for <filename>.el is
|
|
|
|
|
stored in <filename>.log.
|
2016-02-10 11:18:30 +00:00
|
|
|
|
|
|
|
|
|
* make check-maybe
|
|
|
|
|
Like "make check", but run only the tests for files which have
|
|
|
|
|
unresolved prerequisites.
|
|
|
|
|
|
|
|
|
|
* make check-expensive
|
|
|
|
|
Like "make check", but run also the tests marked as expensive.
|
|
|
|
|
|
2018-03-16 15:19:10 +00:00
|
|
|
|
* make check-all
|
|
|
|
|
Like "make check", but run all tests.
|
|
|
|
|
|
2021-01-17 12:37:58 +00:00
|
|
|
|
* make check-<dirname>
|
|
|
|
|
Like "make check", but run only the tests in test/<dirname>/*.el.
|
|
|
|
|
<dirname> is a relative directory path, which has replaced "/" by "-",
|
|
|
|
|
like in "check-src" or "check-lisp-net".
|
2021-01-14 10:34:09 +00:00
|
|
|
|
|
2019-07-13 11:30:28 +00:00
|
|
|
|
* make <filename> -or- make <filename>.log
|
2016-02-10 11:18:30 +00:00
|
|
|
|
Run all tests declared in <filename>.el. This includes expensive
|
|
|
|
|
tests. In the former case the output is shown on the terminal, in
|
|
|
|
|
the latter case the output is written to <filename>.log.
|
|
|
|
|
|
2019-08-28 18:39:33 +00:00
|
|
|
|
<filename> could be either a relative file name like
|
|
|
|
|
"lisp/files-tests", or a package name like "files-tests".
|
|
|
|
|
|
2016-02-10 11:18:30 +00:00
|
|
|
|
ERT offers selectors, which make it possible to filter out which test
|
|
|
|
|
cases shall run. The make variable $(SELECTOR) gives you a simple
|
|
|
|
|
mean to use your own selectors. The ERT manual describes how
|
|
|
|
|
selectors are constructed, see (info "(ert)Test Selectors") or
|
|
|
|
|
https://www.gnu.org/software/emacs/manual/html_node/ert/Test-Selectors.html
|
|
|
|
|
|
|
|
|
|
You could use predefined selectors of the Makefile. "make <filename>
|
|
|
|
|
SELECTOR='$(SELECTOR_DEFAULT)'" runs all tests for <filename>.el
|
2021-01-19 13:12:22 +00:00
|
|
|
|
except the tests tagged as expensive or unstable. Other predefined
|
|
|
|
|
selectors are $(SELECTOR_EXPENSIVE) (run all tests except unstable
|
|
|
|
|
ones) and $(SELECTOR_ALL) (run all tests).
|
2016-02-10 11:18:30 +00:00
|
|
|
|
|
|
|
|
|
If your test file contains the tests "test-foo", "test2-foo" and
|
|
|
|
|
"test-foo-remote", and you want to run only the former two tests, you
|
2018-06-07 01:25:52 +00:00
|
|
|
|
could use a selector regexp (note that the "$" needs to be doubled to
|
|
|
|
|
protect against "make" variable expansion):
|
|
|
|
|
|
|
|
|
|
make <filename> SELECTOR='"foo$$"'
|
2016-02-10 11:18:30 +00:00
|
|
|
|
|
2020-08-04 12:19:51 +00:00
|
|
|
|
In case you want to use the symbol name of a test as selector, you can
|
|
|
|
|
use it directly:
|
|
|
|
|
|
|
|
|
|
make <filename> SELECTOR='test-foo-remote'
|
|
|
|
|
|
2018-01-04 00:14:05 +00:00
|
|
|
|
Note that although the test files are always compiled (unless they set
|
2018-03-16 15:19:10 +00:00
|
|
|
|
no-byte-compile), the source files will be run when expensive or
|
|
|
|
|
unstable tests are involved, to give nicer backtraces. To run the
|
|
|
|
|
compiled version of a test use
|
2018-01-04 00:14:05 +00:00
|
|
|
|
|
|
|
|
|
make TEST_LOAD_EL=no ...
|
|
|
|
|
|
2018-03-19 11:58:45 +00:00
|
|
|
|
Some tests might take long time to run. In order to summarize the
|
|
|
|
|
<nn> tests with the longest duration, call
|
|
|
|
|
|
|
|
|
|
make SUMMARIZE_TESTS=<nn> ...
|
|
|
|
|
|
2020-10-31 19:26:54 +00:00
|
|
|
|
The backtrace of failing tests are truncated to the default value of
|
|
|
|
|
'ert-batch-backtrace-right-margin'. To see more of the backtrace, use
|
|
|
|
|
|
|
|
|
|
make TEST_BACKTRACE_LINE_LENGTH=<nn> ...
|
|
|
|
|
|
2017-12-17 01:06:11 +00:00
|
|
|
|
The tests are run in batch mode by default; sometimes it's useful to
|
|
|
|
|
get precisely the same environment but run in interactive mode for
|
|
|
|
|
debugging. To do that, use
|
|
|
|
|
|
|
|
|
|
make TEST_INTERACTIVE=yes ...
|
|
|
|
|
|
2021-07-01 11:43:44 +00:00
|
|
|
|
By default, ERT test failure summaries are quite brief in batch
|
|
|
|
|
mode--only the names of the failed tests are listed. If the
|
|
|
|
|
$EMACS_TEST_VERBOSE environment variable is set, the failure summaries
|
|
|
|
|
will also include the data from the failing test.
|
|
|
|
|
|
2018-11-13 08:44:06 +00:00
|
|
|
|
Some of the tests require a remote temporary directory
|
2019-07-13 11:30:28 +00:00
|
|
|
|
(autorevert-tests.el, filenotify-tests.el, shadowfile-tests.el and
|
|
|
|
|
tramp-tests.el). Per default, a mock-up connection method is used
|
|
|
|
|
(this might not be possible when running on MS Windows). If you want
|
|
|
|
|
to test a real remote connection, set $REMOTE_TEMPORARY_FILE_DIRECTORY
|
|
|
|
|
to a suitable value in order to overwrite the default value:
|
2018-11-13 08:44:06 +00:00
|
|
|
|
|
2021-02-19 09:03:20 +00:00
|
|
|
|
env REMOTE_TEMPORARY_FILE_DIRECTORY=/ssh:host:/tmp make ...
|
2018-11-13 08:44:06 +00:00
|
|
|
|
|
Implement NTLM server for ntlm.el testing
* test/Makefile.in
(GNU_ELPA_DIRECTORY, elpa_dependencies, elpa_els, elpa_opts): New
variables.
(EMACSOPT, ert_opts): Add elpa_opts.
* test/README: Document GNU_ELPA_DIRECTORY make variable.
* test/lisp/net/ntlm-tests.el: Fix checkdoc-reported issues.
(ntlm-tests-message, ntlm-server-build-type-2, ntlm-server-hash)
(ntlm-server-check-authorization, ntlm-server-do-token)
(ntlm-server-filter, ntlm-server-handler, ntlm-server-start)
(ntlm-server-stop, ntlm-tests--url-retrieve-internal-around)
(ntlm-tests--authenticate)
(ntlm-tests--start-server-authenticate-stop-server): New
functions.
(ntlm-tests--username-oem, ntlm-tests--username-unicode)
(ntlm-tests--client-supports-unicode, ntlm-tests--challenge)
(ntlm-tests--result-buffer, ntlm-tests--successful-result): New
variables.
(ntlm-authentication)
(ntlm-authentication-old-compatibility-level): New tests.
* test/lisp/net/ntlm-resources/authinfo: New file. (Bug#43566)
2021-02-18 23:05:38 +00:00
|
|
|
|
Some optional tests require packages from GNU ELPA. By default
|
|
|
|
|
../../elpa will be checked for these packages. If GNU ELPA is checked
|
|
|
|
|
out somewhere else, use
|
|
|
|
|
|
|
|
|
|
make GNU_ELPA_DIRECTORY=/path/to/elpa ...
|
2021-03-16 17:40:24 +00:00
|
|
|
|
|
2019-01-03 14:37:53 +00:00
|
|
|
|
|
|
|
|
|
There are also continuous integration tests on
|
|
|
|
|
<https://hydra.nixos.org/jobset/gnu/emacs-trunk> (see
|
2019-07-13 11:30:28 +00:00
|
|
|
|
admin/notes/hydra) and <https://emba.gnu.org/emacs/emacs> (see
|
|
|
|
|
admin/notes/emba). Both environments provide an environment variable,
|
|
|
|
|
which could be used to determine, whether the tests run in one of
|
|
|
|
|
these test environments.
|
2019-01-03 14:37:53 +00:00
|
|
|
|
|
|
|
|
|
$EMACS_HYDRA_CI indicates the hydra environment, and $EMACS_EMBA_CI
|
|
|
|
|
indicates the emba environment, respectively.
|
|
|
|
|
|
2016-02-10 11:18:30 +00:00
|
|
|
|
|
2020-12-09 19:16:03 +00:00
|
|
|
|
(Also, see etc/compilation.txt for compilation mode font lock tests
|
|
|
|
|
and etc/grep.txt for grep mode font lock tests.)
|
2016-02-10 11:18:30 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This file is part of GNU Emacs.
|
|
|
|
|
|
|
|
|
|
GNU Emacs is free software: you can redistribute it and/or modify
|
|
|
|
|
it under the terms of the GNU General Public License as published by
|
|
|
|
|
the Free Software Foundation, either version 3 of the License, or
|
|
|
|
|
(at your option) any later version.
|
|
|
|
|
|
|
|
|
|
GNU Emacs is distributed in the hope that it will be useful,
|
|
|
|
|
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
GNU General Public License for more details.
|
|
|
|
|
|
|
|
|
|
You should have received a copy of the GNU General Public License
|
2017-09-13 22:52:52 +00:00
|
|
|
|
along with GNU Emacs. If not, see <https://www.gnu.org/licenses/>.
|