Reusable tests, test-cases, and test-suites

Questions & Answers
1

Hello fellow Racketeers,
while preparing some lecture notes I noticed that sometimes it is useful to have a multiple implementations of single algorithm prepared. As I am playing with this idea, I started to write identical rackunit tests for each of those implementations - which doesn't make much sense. Is there a way to create a test-suite (or anything else) and then apply it to multiple implementations of given procedure(s)?
I can think of two very simple approaches - parameters and syntax macros. I can do both without virtually any effort, but it makes me wonder whether there isn't already a solution for this problem.

Cheers,
Dominik

2 Likes
2

You can abstract over test cases with a procedure (as long as you're abstracting over values). If your abstraction has too many arguments, you can bundle them up as a unit and use unit linking instead of function application to combine the pieces. (The main db tests work this way: see db/db-tests/tests/db/{config.rkt,all-tests.rkt} for the signatures, helper units, and linking.)

Another approach is to write the tests in a separate non-module file and include them into different environments for the different implementations. I think that would be a good way to test different implementations of the same macro, for example.

4 Likes
3

I tried using signatures and units (and btw, this is the first time they look like they can be useful for me), but failed. I will keep on trying - the db-tests are a the best! :wink:

In the meantime, I opted for something like:

#lang racket/base

(define (my-proc1)
  1)

(define (my-proc2)
  2)

(module+ test
  (require rackunit
           rackunit/text-ui)
  (define impl (make-parameter #f))
  (define tests
    (test-suite
     "Test the whole implementation."
     (test-case "Equality"
       (check-eq? ((impl)) 1))))
  (for ((impl1 (list my-proc1 my-proc2)))
    (parameterize ((impl impl1))
      (run-tests tests))))

Of course this runs two test suites and prints one result - which is far from optimal. The simple case I am preparing for the students is a singly-linked list implementation consisting of the structs and a decent list of operations - some of them implemented in more versions to see the differences. So ideally I want to run one "big" test suite covering the whole module but when there are more implementations, particular test-cases will cover al versions of given procedures.

It is tempting to just write a simple macro for that, but maybe there is a better solution.

1 Like
4

Here's what I meant by using a procedure:

#lang racket/base

(define (my-proc1)
  1)

(define (my-proc2)
  2)

(module+ test
  (require rackunit
           rackunit/text-ui)

  (define (make-tests impl label)
    (test-suite (format "Test implementation (~a)" label)
      (test-case "Equality"
        (check-eq? (impl) 1))))

  (run-tests (make-test-suite
              "Test all implementations"
              (list (make-tests my-proc1 "baseline")
                    (make-tests my-proc2 "my-proc2")))))

Here is an example with units, with several variants of the linking step:

#lang racket/base

(define (my-proc1)
  1)

(define (my-proc2)
  2)

(module+ test
  (require racket/unit
           rackunit
           rackunit/text-ui)

  (define-signature impl^
    (impl ;; -> Number
     label ;; String
     ))

  (define-signature test^
    (suite ;; TestSuite
     ))

  (define-unit test@
    (import impl^)
    (export test^)
    (define suite
      (test-suite (format "Test implementation (~a)" label)
                  (test-case "Equality"
                    (check-eq? (impl) 1)))))

  (define-unit impl1@
    (import)
    (export impl^)
    (define impl my-proc1)
    (define label "baseline"))
  (define-unit impl2@
    (import)
    (export impl^)
    (define impl my-proc2)
    (define label "my-proc2"))

  ;; For dynamic/unknown number of tests:

  ;; Use local environment for linking.
  (run-tests (make-test-suite
              "Test all implementations"
              (for/list ([impl@ (in-list (list impl1@ impl2@))])
                (define-values/invoke-unit impl@
                  (import) (export impl^))
                (define-values/invoke-unit test@
                  (import impl^) (export test^))
                suite)))

  ;; Use explicit linking.
  (run-tests (make-test-suite
              "Test all implementations"
              (for/list ([impl@ (in-list (list impl1@ impl2@))])
                (define-values/invoke-unit
                  (compound-unit
                   (import) (export Test)
                   (link (([Impl : impl^]) impl@)
                         (([Test : test^]) test@ Impl)))
                  (import) (export test^))
                suite)))

  ;; Use inferred linking.
  (run-tests (make-test-suite
              "Test all implementations"
              (for/list ([impl@ (in-list (list impl1@ impl2@))])
                (define-unit-binding impl/sig@ impl@ (import) (export impl^))
                (define-values/invoke-unit/infer
                  (export test^) (link impl/sig@ test@))
                suite)))

  ;; For fixed, known number of tests:

  ;; Link tests with prefixes to avoid name collision.
  (let ()
    (define-values/invoke-unit/infer
      (export (prefix t1: test^)) (link impl1@ test@))
    (define-values/invoke-unit/infer
      (export (prefix t2: test^)) (link impl2@ test@))
    (run-tests (test-suite
                "Test all implementations"
                t1:suite
                t2:suite)))

  ;; Linking with tags to disambiguate imports with same signature.
  ;; (AFAIK, inference can't handle this case: multiple units with
  ;; same export signature, and a unit that imports the same
  ;; signature multiple times.)
  (let ()
    (define-unit all-tests@
      (import (tag t1 (prefix t1: test^)) (tag t2 (prefix t2: test^)))
      (export)
      (run-tests (test-suite
                  "Test all implementations"
                  t1:suite
                  t2:suite)))
    ;; ---- with pre-linked test units:
    (let ()
      (define-compound-unit/infer test1@
        (import) (export test^) (link impl1@ test@))
      (define-compound-unit/infer test2@
        (import) (export test^) (link impl2@ test@))
      (invoke-unit
       (compound-unit
        (import) (export)
        (link (([Test1 : test^]) test1@)
              (([Test2 : test^]) test2@)
              (() all-tests@ (tag t1 Test1) (tag t2 Test2))))))
    ;; ---- with one big linkage clause:
    (invoke-unit
     (compound-unit
      (import)
      (export)
      (link (([Impl1 : impl^]) impl1@)
            (([Impl2 : impl^]) impl2@)
            (([Test1 : test^]) test@ Impl1)
            (([Test2 : test^]) test@ Impl2)
            (() all-tests@ (tag t1 Test1) (tag t2 Test2)))))))
2 Likes
5

Thank you VERY much. As always, the devil is in the detail: define-values/invoke-unit. This is exactly the pattern I wanted to see.

1 Like