org-mode/testing

# -*- mode:org -*-
#+TITLE: Org-mode Testing
#+PROPERTY: results silent

* Dependencies

The only dependency is [[http://www.emacswiki.org/emacs/ErtTestLibrary][ERT]] the Emacs testing library which ships with
Emacs24.  If you are running an older version of Emacs and don't
already have ERT installed it can be installed from its old [[https://github.com/ohler/ert][git
repository]].

* Non-interactive batch testing from the command line

The simplest way to run the Org-mode test suite is from the command
line with the following invocation.  Note that the paths below are
relative to the base of the Org-mode directory.

Also note that many of the current tests uses babel evaluation...

#+BEGIN_SRC sh :dir (expand-file-name "..")
  # For Emacs earlier than 24, add -L /path/to/ert
  emacs -Q --batch \
        -L lisp/ -L testing/ -L testing/lisp -l lisp/org.el \
        -l lisp/org-id.el -l testing/org-test.el \
        --eval "(progn (org-reload) (setq org-confirm-babel-evaluate nil) \
        (org-babel-do-load-languages 'org-babel-load-languages \
        '((emacs-lisp . t) (shell . t) (org . t))))" \
        -f org-test-run-batch-tests
#+END_SRC

The options in the above command are explained below.

| -Q      | ignores any personal configuration ensuring a vanilla Emacs instance is used |
| --batch | runs Emacs in "batch" mode with no gui and termination after execution       |
| -l      | loads Org-mode and the org mode test suite defined in testing/org-test.el    |
| --eval  | reloads Org-mode and allows evaluation of code blocks by the tests           |
| -f      | actually runs the tests using the `org-test-run-batch-tests' function        |

* Trigger the tests with 'make'

** Recompile all

Target ~test~ can be used to trigger a test run.  The tests start
after cleaning up and recompilation.

#+BEGIN_SRC sh :dir (expand-file-name "..") :results silent
make test
#+END_SRC

See ../mk/default.mk for details.

** Test dirty

The 'dirty' targets are for recompiling without cleaning and
rebuilding everything.  This usually speeds up the recompilation
considerably.

The 'dirty' target is called test-dirty.

#+BEGIN_SRC sh :dir (expand-file-name "..") :results silent
make test-dirty
#+END_SRC

Note that the outcome may /not/ be in perfect shape.

* Interactive testing from within Emacs

To run the Org-mode test suite from a current Emacs instance simply
load and run the test suite with the following commands.

1) First load the test suite.
   #+BEGIN_SRC emacs-lisp :var here=(buffer-file-name)
     (add-to-list 'load-path (file-name-directory here))
     (require 'org-test)
   #+END_SRC

2) Load required Babel languages
   #+BEGIN_SRC emacs-lisp
     (org-babel-do-load-languages
      'org-babel-load-languages
      (and
       (mapc (lambda (lang) (add-to-list 'org-babel-load-languages (cons lang t)))
             '(emacs-lisp shell org))
       org-babel-load-languages))
   #+END_SRC

3) Then run the test suite.  Babel evaluation confirmation is disabled
   and ~C-c C-c~ is enabled while running the tests.
   #+BEGIN_SRC emacs-lisp
     (let (org-babel-no-eval-on-ctrl-c-ctrl-c
           org-confirm-babel-evaluate)
       (org-test-run-all-tests))
   #+END_SRC

   When a test fails, run it interactively and investigate the problem
   in the ERT results buffer.

   To run one test: Use this as a demo example of a failing test
   #+BEGIN_SRC emacs-lisp
     (ert-deftest test-org/org-link-escape-ascii-character-demo-of-fail ()
       (should (string= "%5B"  ; Expecting %5B is correct.
                        (org-link-escape "[")))
       (should (string= "%5C"  ; Expecting %5C is wrong, %5D correct.
                        (org-link-escape "]"))))
   #+END_SRC
   or evaluate the ~ert-deftest form~ of the test you want to run.
   Then ~M-x ert RET
   test-org/org-link-escape-ascii-character-demo-of-fail RET~.  When
   not visible yet switch to the ERT results buffer named ~*ert*~.
   When a test failed the ERT results buffer shows the details of the
   first ~should~ that failed.  See ~(info "(ert)Running Tests
   Interactively")~ on how to re-run, start the debugger etc.

   To run several tests: ~M-x ert RET "<your regexp here>" RET~.

   To run all tests of a single test file: ~M-x ert-delete-all-tests
   RET~ and confirm.  ~M-x load-file RET testing/lisp/<file>.el RET
   M-x ert RET t RET~.

   Consider to set
   #+BEGIN_SRC emacs-lisp
     (setq pp-escape-newlines nil)
   #+END_SRC
   before running the test when looking at ~should~ in the ERT results
   buffer.  Especially when using ~l~ to look at passed test results
   and possibly missing an appropriate setting of ~pp-escape-newlines~
   made only temporarily for the running time of the test as
   e. g. tests using ~org-test-table-target-expect-tblfm~ do.

* Troubleshooting

- If the variable ~org-babel-no-eval-on-ctrl-c-ctrl-c~ is non-nil then
  it will result in some test failure, as there are tests which rely
  on this behavior.