Mutate: inject bugs into your programs!

Recently added to the package catalog, documented here: Mutate

This is a library for mutating programs, i.e. injecting possible bugs by making small syntactic changes to the program syntax.

There's an example in the docs prologue.

It might be used to create a corpus of known-buggy programs (that's why I wrote it).
Or to evaluate the quality of a test suite through mutation testing.
Or something else?

I'm posting it here in the hope that it can be useful to someone else as well!

Source is here

9 Likes

Thank you @llazarek
This is new to me but I'm finding your documentation very helpful.
best regards
Stephen

This is very cool, but I'm curious about the underlying reasoning. The documentation covers the 'how' but not the 'why'; can you give me a tl;dr so that I don't need to immediately dig into the literature?

The example mutator changes (if c t f) into (if (not c) t f) which is essentially guaranteed to break the software. Is the point simply to verify that you have test cases that will catch this, or is there more to it?

Thanks @spdegabrielle and @dstorrs!

Yeah you're exactly right @dstorrs that the if-swap very likely breaks the program, and that one way to use mutation is to check that a test suite catches such bugs. That's the rationale behind mutation testing: a good test suite should be able to catch bugs injected into the code-to-be-tested.

On the other hand, I can elaborate on my use case as a different example of why one might want mutants. I've been using mutation to analyze the debugging information different tools (like contracts) provide when programs go wrong. To do that, I need a corpus of buggy programs where I know for each one exactly where the bug is. So I use mutation to turn a set of programs that aren't known to be buggy into a huge number of mutants, each with a different potential bug that serves as a different scenario for my analysis.

1 Like

Though... you do need to verify that the change actually introduces a bug, right? Certainly one possible scenario is that you make a change in dead code. I'm sure you've already thought about that.

Yes indeed. And that can be a tricky thing to check! In my use-case we take the simple and conservative choice to filter for mutants that cause the program to crash.