Custom validity and validity-based testing in Haskell

Values of custom types usually have invariants imposed upon them. In this post I motivate and announce the validity, genvalidity and genvalidity-hspec libraries that have just come out.

Contrary to what we might like to think, an absence of compilation errors does not imply correctly functioning code. In some cases we expect some invariants to hold about our data, but we don't necessarily check them rigorously.

This is a situation in which the typesystem may be expensive to use to guarantee correctness. That's why we will use testing instead.

A running example

I will take a contrived example to keep the blogpost short. Assume we have the following context:

-- | [...]
-- INVARIANT >= 2
newtype GreaterThanOne
  = GreaterThanOne Int

-- | [...]
-- INVARIANT >= 2
-- INVARIANT must be a prime
newtype Prime    
  = Prime Int

For now, we just assume that invariants hold. If we now want to write a safe primeFactorisation function, it might have the following type.

primeFactorisation :: GreaterThanOne -> Maybe [Prime]

Now we will go on to how we can actually make this safe and test the validity assumptions.

Validity

To be able to express the invariants that we would like to enforce on our data, we can write the following function:

myDataIsValid :: MyData -> Bool

We express, in code, exactly what it means for a MyData to be valid. That way we can later check whether our MyData is valid.

This is the validity package comes in. It is really simple and contains not much more than the following type class, but it opens up some possibilities which I will talk about in the rest of this post.

class Validity a where
    isValid :: a -> Bool

In the case of the running example, the Validity instances could look as follows.

instance Validity GreaterThanOne where
    isValid (GreaterThanOne n) = n >= 2

isPrime :: Int -> Bool

instance Validity Prime where
    isValid (Prime n) = n >= 2 && isPrime n

Genvalidity

Now that we have this concept of Validity, we can start writing tests using it. We will ignore the tests which assert that the output is a valid prime factorisation for now. We can then write the following tests with respect to validity:

describe "primeFactorisation" $ do
    it "fails for invalid GreaterThanOne's" $
        forAll ((GreaterThanOne <$> arbitrary) `suchThat` (not . isValid)) $ \i ->
            primeFactorisation i `shouldBe` Nothing

    it "produces valid primes if it succeeds" $
        forAll (GreaterThanOne <$> arbitrary) $ \i ->
            case primeFactorisation i of
                Nothing -> return () -- Can happen
                Just ps -> ps `shouldSatisfy` all isValid

This is quite a mouthful, isn't it? There are also some non-stylistic problems:

someGenerator `suchThat` (not . isValid) runs someGenerator and retries as long as isValid is satisfied. For types that are mostly valid, this can take a long time and will slow down testing significantly.
Generators like GreaterThanOne <$> arbitrary become quite large (in code) for larger structures. Ideally we would only write them once.

Enter genvalidity. genvalidity provides a typeclass called GenValidity:

class Validity a => GenValidity a where
    genUnchecked :: Gen a

    genValid :: Gen a
    genValid = genUnchecked `suchThat` isValid

    genInvalid :: Gen a
    genInvalid = genUnchecked `suchThat` (not . isValid)

When you instantiatie GenValidity for your custom data type, tests involving isValid become much easier to write:

describe "primeFactorisation" $ do
    it "fails for invalid GreaterThanOne's" $
        forAll genInvalid $ \i ->
            primeFactorisation i `shouldBe` Nothing

    it "produces valid primes if it succeeds" $
        forAll genUnchecked $ \i ->
            case primeFactorisation i of
                Nothing -> return () -- Can happen
                Just ps -> ps `shouldSatisfy` isValid

genInvalid Has a default implementation, but when generating GreaterThanOnes, on average 50% of all generations have to be retried at least once. We can specialize this implementation to run faster by using an absolute value function:

class GenValidity GreaterThanOne where
    genUnchecked = GreaterThanOne <$> arbitrary
    genValid = (GreaterThanOne . abs) <$> arbitrary

Standard tests

We can use these new toys to write tests as described above, but we can also use some of the standard tests that are available via genvalidity-hspec. For example, the above tests can be rewritten as follows:

describe "primeFactorisation" $ do
    it "fails for invalid GreaterThanOne's" $
        failsOnInvalidInput primeFactorisation

    it "produces valid primes if it succeeds" $
        validIfSucceeds primeFactorisation

Because we have now written a custom implementation of genValid for GreaterThanOne, we should also add the validitySpec:

validitySpec (Proxy :: Proxy GreaterThanOne)

This will ensure that genValid and genInvalid keep working as intended. However, it cannot check that all possible valid(/invalid) values can still be generated by genValid(/genInvalid), so be careful and check that yourself.