I have a test, where the expected results are slightly different when run under an older version of something. I needed to adapt the test to deal with both older and newer expected values.
One bad thing: check-true, when it fails, doesn't say anything very helpful -- whereas check-equal? shows both the actual and expected values. check-equal? is the spirit of what I want to keep using, ideally.
Another bad thing: All the repetition between the two lists -- they're mostly identical. Ideally I'd prefer some kind of "local or" in the middle of the data, to express the variation, that check-equal? cooperates with, somehow?
Starting to get OT, but it would be neat if check-equal? could do something like the sexp-diff library for actual vs diff. (I've bolted that on as a failure message, sometimes.)
Has anyone else dealt with this situation, more elegantly, and if so, how?
Although I have a couple ideas, I'm likely to either over- or under-think it.
Oh, it sounds like I'm describing check-match, maybe?
I seem to recall trying that years ago and it didn't work out as well as I'd hoped. However it may have been pilot error. I'll take a look at that again.
Having said that, still would love to hear any ideas/experience people might have to share!
Based on your description I’d try something like this:
(define version 1)
(define (msg version) (~a "test 1 at version " version))
(check-equal? (action) (gen-expected version) (msg version)
If match is close enough to compare the expected lists, you should be able to generate the different expected results and use regular tests. (This is close to something I have done.)
I think that makes a lot of sense when it's necessary or desirable to assert that specific versions produce specific results.
That's not quite my situation.
I'm usually designing tests that will pass when run against any version of Racket supported by Racket Mode.
And generally I don't need to check specific version numbers of Racket or certain packages. In fact I prefer to avoid it when possible.
My situation is closer to one where there is some small set of similar OK values (where each could be a large-ish compound value, with relatively small differences from the others, and you'd really prefer to DRY/abstract ) and random chooses which one the test happens to get.
After working some more with check-match: Although it has a couple small rough edges, it's pretty good for this -- and definitely much better than my horrible (check-true (or (equal? __ __) ...)).
p.s. I alluded to some rough edges of check-match compared to check-equal?. Specifically:
Most of the check-xxx functions have an optional message argument, but check-match instead has an optional pred argument. This means:
If you don't notice this and supply a string message to check-match, it doesn't complain the string isn't a predicate, instead the test fails via using the string as a predicate. Confusing.
Once you realize this, it's not too big a deal to supply the message instead by wrapping check-match in a with-check-info.
Although check-equal? pretty-prints both the actual and expected values, check-match pretty-prints the actual but displays the pattern on whole line. This makes it very hard to grok -- much less compare/contrast with the actual (like you'd do when check-equal? fails).
Despite those paper cuts I think check-match is good enough, for me, for now.
For better error reporting, you might express the expected value as a contract. You could then just use check-not-exn around a thunk using contract directly, though a check-contract form would be very nice.
For the specific example you gave, you'd probably want to write a contract combinator for record-like hashes. I've written many variants of that with various features over the years. I put a simple one in this example, but, looking at the one use of or in your match pattern, a combinator supporting optional keys might be even better for your use:
@LiberalArtist Oh, that's an interesting idea to use a contract, and get better test failure messages.
Using check-match seems to be a more "natural" step from check-equal?. It's still in the spirit of, "Here's the expected value, which I originally copypasta-d from REPL output" -- just bending that to be more flexible using match's pattern language.
However in reality, in this specific example, I had to modify that original REPL output a lot (such as replacing all the #hasheq literals with hash-table patterns, quoting symbols, etc.).
It is so unlike the original REPL output, in details, that it would hardly be much more work to express it as a contract -- and likely worth the effort to get better test failure messages.
I noticed this, too, when converting your match pattern to a contract, that is was mostly literal data, with or/c as an escape to specify a more complex case. Since contracts are first-class values, you could use a little helper function to mostly keep writing quasiquoted REPL output:
