XTest - Simple Testing with Emacs & ERT

XTest is a simple set of extensions for ERT. XTest speeds up the creation of tests that follow the “one assertion per test” rule of thumb. It also simplifies testing functions that manipulate buffers. XTest aims to do a few things well, instead of being a monolithic library that attempts to solve every conceivable testing need. XTest is designed to be paired with vanilla ERT and other ERT libraries, where the user mixes and matches depending on their needs.

Table of Contents

• Introduction
  • Simple, Concise Testing
  • Simple Buffer Testing
  • What XTest Isn’t
• Install & Setup
  • Install
  • Setup
  • Customization
• Functionality
  • Core
  • Basic Test Group
  • Data Test Group
  • Buffer Test Group
  • Comment Group
• References & Aside
• License

Introduction

Simple, Concise Testing

Suppose we wanted to test a function that adds one to any number passed to it.

(defun one-plus (n)
  "Add one to N."
  (1+ n))

A traditional ERT test to verify one-plus

(ert-deftest one-plus ()
  "Verify increment works."
  (should (= (one-plus 2) 3))
  (should (= (one-plus -1) 0))
  (should-not (= (one-plus 2) 4))
  (should-not (= (one-plus -1) -2)))

XTest evolved over time with the objective of eliminating the redundancies and inconveniences from the code above.

1. Notice the repetition of should assertion, should-not assertion, and the = test.
2. If the first assertion fails we won’t know if any other assertions fail until we fix the first. To know if all assertions pass or fail during the first run, each assertion needs to be wrapped in its own ert-deftest. This increases effort from the developer, increases the verbosity of the code, and decreases comprehensibility of the tests. If only there was an easier way and there is!

XTest verifying one-plus

(defun xt-add (x y)
  (= (one-plus x) y))

(xt-deftest one-plus
  (xt-note "Verify addition works.")
  (xtd-should 'xt-add
              (2 3)
              (-1 0))
  (xtd-should! 'xt-add
               (2 4)
               (-1 -2)))

;;; compiles to 4 ert-deftest

XTest assertions are all contained in a macro called xt-deftest. The positive assertions are wrapped in xtd-should with the function to be applied specified first, and the test case data following. Each list after the function is a test case. xt-add will be applied to each test case and verified as true in a separate ert-deftest.

Notice what XTest does differently than vanilla ERT:

1. DRY (Don’t Repeat Yourself) - No need to repeat the macro should or the function one-plus.
2. One Assertion Per Test - Each test is created as a separate ert-deftest.
3. Grouping - Conceptually similar tests can be grouped. For example, all should assertions can be grouped as xtd-should.

The example xt-deftest above macro expands into the code below. Notice how each test case from above is expanded into a seperate ert-deftest.

(progn
  (progn
    (ert-deftest one-plus-1 nil "" (should (apply 'xt-add '(2 3))))
    (ert-deftest one-plus-2 nil "" (should (apply 'xt-add '(-1 0)))))
  (progn
    (ert-deftest one-plus-3 nil "" (should-not (apply 'xt-add '(2 4))))
    (ert-deftest one-plus-4 nil "" (should-not (apply 'xt-add '(-1 -2))))))

Simple Buffer Testing

XTest simplifies the testing of certain buffer manipulations via two core utilities xtd-setup= and xtd-return=.

Defining a test that verifies the insert procedure.

(xt-deftest insert
  (xt-note "Testing the insert procedure.")
  (xtd-setup= (lambda (_) (insert " Me"))
              ;; "Test-!-" represents a buffer with the
              ;; word ~Test~ and the cursor at the very
              ;; end of the buffer.
              ("Test-!-" "Test Me-!-")  ; Test 1
              ("-!-" " Me-!-")))        ; Test 2

The concept behind xtd-setup= is that buffers can be specified and compared using strings.

What happens in the example above for xtd-setup=, test 1 is described below:

1. Create a temporary buffer.
   1. Insert the setup string Test-!- into the buffer.
   2. Replace the cursor symbol with cursor (e.g. in the example above -!- is replaced with cursor).
2. Execute test-function, in this case (lambda (_) (insert " Me")), in temporary buffer.
3. Convert the buffer into a string replacing the cursor position with the symbol -!-.
4. Assert the buffer string produced is the same as the second string argument “Test Me-!-“.
5. Close the temporary buffer.
6. Repeat starting at step one for test 2.
7. Optional: Third argument for tests is optional and is supplied to the test-function.

The second utility, xtd-return= is similar to xtd-setup= in the fact the first test argument sets up a temporary buffer and the test-function operates on it. Where xtd-return= differs is that it is interested in verifying what the test-function returns when executed in the temporary test buffer. Equality is checked using the equal function.

Defining a test that verifies the buffer-substring function.

(xt-deftest buffer-substring
  (xtd-return= (lambda (_) (let ((point (point)))
                        (buffer-substring point (+ 2 point))))
               ("he-!-llo" "ll")
               ("-!-hidly ho" "hi")
               ;; In the below case, XTest assumes the cursor 
               ;; is at the start of the buffer since it was
               ;; not explicitly specified
               ("hidly ho" "hi")))

What XTest Isn’t

1. Replacement for ERT—in fact one needs to know how to use ERT to be able to use XTest.
2. An exhaustive set of testing utilities.

Install & Setup

Install

Manual

1. Install cl-lib.el (at the minimum version 0.5).
2. Download xtest.el and place it in your path.

Repo

XTest is available to install via MELPA.

Setup

Once installed, add the following at the start of the file you need XTest.

(require 'xtest)

Customization

1. By default the representation or stand in for the cursor by default is -!-. Use the snippet below to change the cursor representation. Also, can be customized via the group xtest.

   ;;; Use '-!-' symbol as the cursor in tests
   (setf xt-cursor "-!-")
       

Functionality

Core

1. xt-deftest - expects a BASE-TEST-NAME and TESTS-GROUPS. BASE-TEST-NAME plus an incrementing number is used to name all the ert-deftest that are created. After the BASE-TEST-NAME, any number of TEST-GROUPS can be specified (for more info on test groups see below). Test groups are the main test mechanism.

Basic Test Group

1. xt-should - asserts all test expressions evaluate to true. Each expression will be expanded into a separate ert-deftest.

   (xt-deftest number-equal
     (xt-should  (= 1 1)   ; Succeeds
                 (= 2 2)   ; Succeeds
                 (= 2 3))  ; Fails
     (xt-should! (= 1 2)   ; Succeeds
                 (= 4 4))  ; Fails
     )
       

2. xt-should! - asserts all test expressions evaluate to nil. Each expression will be expanded into a separate ert-deftest. See example given for xt-should.

Data Test Group

1. xtd-should - asserts when test-function is applied to each test in TESTS this returns true. The test-function must accept as many arguments as each test supplies.

   (xt-deftest data-number-equal
     (xtd-should (lambda (x y) (= x y))
                 (1 1)   ; Success
                 (2 2)   ; Success
                 (2 3))  ; Fails
     (xtd-should! (lambda (x y) (= x y))
                  (1 2)  ; Success
                  (4 4)) ; Fails
     )
       

2. xtd-should! - asserts when test-function is applied to each test in tests this returns nil. The test-function must accept as many arguments as each test supplies.

Buffer Test Group

1. xtd-setup= - test-function is applied to each temporary buffer created by tests. The resulting buffer is turned back into a string with the cursor replaced with xt-cursor. The resulting string is asserted to see if it is equal to the second argument in the tests. Each test in tests must have the form below.

   test = (initial-buffer-setup-string
           final-buffer-string
           optional-argument-for-test-function)
       

   (xt-deftest insert
     (xt-note "Testing the insert procedure.")
     (xtd-setup= (lambda (name) (insert name))
                 ("Hi -!-" "Hi Mustafa-!-" "Mustafa") ; Success
                 ("-!-" "Joey-!-" "Joe")              ; Fails
   ))
       

2. xtd-return= - test-function is applied to each temporary buffer created by tests. The value returned by test-function is asserted to be equal to the second argument in the test list. Equality is checked using the equal function.

   test = (initial-buffer-setup-string
           final-buffer-string
           optional-argument-for-test-function)
       

   (xt-deftest char-after
     (xtd-return= (lambda (_) (char-after (point)))
                  ("he-!-llo" ?l)        ; Success
                  ("-!-hidly ho" ?c)     ; Failure
                  ("hidly ho-!-" nil)))  ; Success
       

Comment Group

1. xt-note - is not processed by XTest and can be used leave comments or comment out other test groups.

References & Aside

1. ERT - (Emacs Regression Testing) documentation: http://www.gnu.org/software/emacs/manual/html_node/ert/.
2. For full rationale of why each test is enclosed in a separate ERT instance see http://blog.jayfields.com/2007/06/testing-one-assertion-per-test.html
3. Emacs Lisp documentation uses the notation -!- as a stand in for the cursor as well, see https://www.gnu.org/software/emacs/manual/html_node/elisp/Buffer-Contents.html#Buffer-Contents for an example.

License

Copyright © 2014 Mustafa Shameem

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.