Custom validity and validitybased testing in Haskell
Values of custom types usually have invariants imposed upon them. In this
post I motivate and announce the validity
,
genvalidity
and genvalidityhspec
libraries that
have just come out.
Contrary to what we might like to think, an absence of compilation errors does not imply correctly functioning code. In some cases we expect some invariants to hold about our data, but we don’t necessarily check them rigorously.
This is a situation in which the typesystem may be expensive to use to guarantee correctness. That’s why we will use testing instead.
A running example
I will take a contrived example to keep the blogpost short. Assume we have the following context:
  [...]
 INVARIANT >= 2
newtype GreaterThanOne
= GreaterThanOne Int
  [...]
 INVARIANT >= 2
 INVARIANT must be a prime
newtype Prime
= Prime Int
For now, we just assume that invariants hold. If we now want to write a
safe primeFactorisation
function, it might have the following
type.
primeFactorisation :: GreaterThanOne > Maybe [Prime]
Now we will go on to how we can actually make this safe and test the validity assumptions.
Validity
To be able to express the invariants that we would like to enforce on our data, we can write the following function:
myDataIsValid :: MyData > Bool
We express, in code, exactly what it means for a MyData
to be
valid. That way we can later check whether our MyData
is
valid.
This is the validity package comes in. It is really simple and contains not much more than the following type class, but it opens up some possibilities which I will talk about in the rest of this post.
class Validity a where
isValid :: a > Bool
In the case of the running example, the Validity
instances
could look as follows.
instance Validity GreaterThanOne where
isValid (GreaterThanOne n) = n >= 2
isPrime :: Int > Bool
instance Validity Prime where
isValid (Prime n) = n >= 2 && isPrime n
Genvalidity
Now that we have this concept of Validity
, we can start
writing tests using it. We will ignore the tests which assert that the output
is a valid prime factorisation for now. We can then write the following tests
with respect to validity:
describe "primeFactorisation" $ do
it "fails for invalid GreaterThanOne's" $
forAll ((GreaterThanOne <$> arbitrary) `suchThat` (not . isValid)) $ \i >
primeFactorisation i `shouldBe` Nothing
it "produces valid primes if it succeeds" $
forAll (GreaterThanOne <$> arbitrary) $ \i >
case primeFactorisation i of
Nothing > return ()  Can happen
Just ps > ps `shouldSatisfy` all isValid
This is quite a mouthful, isn’t it? There are also some nonstylistic problems:

someGenerator `suchThat` (not . isValid)
runssomeGenerator
and retries as long asisValid
is satisfied. For types that are mostly valid, this can take a long time and will slow down testing significantly. 
Generators like
GreaterThanOne <$> arbitrary
become quite large (in code) for larger structures. Ideally we would only write them once.
Enter genvalidity.
genvalidity
provides a typeclass called
GenValidity
:
class Validity a => GenValidity a where
genUnchecked :: Gen a
genValid :: Gen a
genValid = genUnchecked `suchThat` isValid
genInvalid :: Gen a
genInvalid = genUnchecked `suchThat` (not . isValid)
When you instantiatie GenValidity
for your custom data type,
tests involving isValid
become much easier to write:
describe "primeFactorisation" $ do
it "fails for invalid GreaterThanOne's" $
forAll genInvalid $ \i >
primeFactorisation i `shouldBe` Nothing
it "produces valid primes if it succeeds" $
forAll genUnchecked $ \i >
case primeFactorisation i of
Nothing > return ()  Can happen
Just ps > ps `shouldSatisfy` isValid
genInvalid
Has a default implementation, but when generating
GreaterThanOne
s, on average 50% of all generations have to be
retried at least once. We can specialize this implementation to run faster by
using an absolute value function:
class GenValidity GreaterThanOne where
genUnchecked = GreaterThanOne <$> arbitrary
genValid = (GreaterThanOne . abs) <$> arbitrary
Standard tests
We can use these new toys to write tests as described above, but we can
also use some of the standard tests that are available via
genvalidityhspec
. For example, the above tests can be rewritten
as follows:
describe "primeFactorisation" $ do
it "fails for invalid GreaterThanOne's" $
failsOnInvalidInput primeFactorisation
it "produces valid primes if it succeeds" $
validIfSucceeds primeFactorisation
Because we have now written a custom implementation of
genValid
for GreaterThanOne
, we should also add the
validitySpec
:
validitySpec (Proxy :: Proxy GreaterThanOne)
This will ensure that genValid
and genInvalid
keep working as intended. However, it cannot check that all possible
valid(/invalid) values can still be generated by
genValid
(/genInvalid
), so be careful and check that
yourself.