Announcing genvalidity-1.0.0.0

This post announces the latest 1.0.0.0 release of genvalidity and all its companion packages. It re-introduces validity-based testing and elaborates on the choices that went into the new release.

Quick intro to validity-based testing

If you've ever done any property testing, you will know that writing generators, shrinking functions, and properties is an expensive process. Validity-based testing lets you get the biggest bang for your buck while property testing.

First: You get generators and shrinking functions for free.

data Example = Example
  { exampleText :: !Text
  , exampleInt :: !Int
  } deriving (Show, Eq, Generic)

instance Validity Example
instance GenValid Example

-- Free generator and free (valid!) shrinking function:
genValid :: GenValid a => Gen a

shrinkValid :: a -> [a]

This requires the Validity type-class, which specifies which values are allowed to exist at runtime. It lets you add invariants that you did not specify at the type level (either because you couldn't, or because it wasn't practical). For example, the Ratio Int type has an invariant that the ratio is normalised: 5 :% 5 is not valid, but 1 :% 1 is. These invariants must be maintained during testing, and genvalidity does that for you automatically.

Second: The library companions also provide a broad group of property combinators and test suite combinators:

-- | Property combinator: a function does not produce invalid values.
producesValid
  :: (Show a, Show b, GenValid a, Validity b)
  => (a -> b)
  -> Property

-- | Property combinator to falsify associativity.
associative
  :: (Show a, Eq a, GenValid a)
  => (a -> a -> a)
  -> Property

-- | Test suite combinator to falsify broken 'Ord' instances
ordSpec
  :: forall a. (Show a, Ord a, Typeable a, GenValid a)
  => Spec

Third: The generators are much better at finding failing cases than QuickCheck's arbitrary generators.

Observe:

$ sample (arbitrary :: Gen Int)
0
2
1
5
2
-10
-7
-5
3
7
9

$ sample (genValid :: Gen Int)
-5081479997565653173
0
-5003305438483606793
9223372036854775803
-2939680821367723041
4299164122116245140
-1240527380647485974
5
-9223372036854775807
-4444472881650683405
6

$ sample (arbitrary :: Gen Double)
0.0
0.9385391771900854
1.719756395885873
-2.0417230609407797
-1.9972767267931484
-7.386915861067222
-7.843340116007125
10.106378798872054
-15.709853553311408
-4.501347309863509
8.796620475189744

$ sample (genValid :: Gen Double)
NaN
-2.566131414864011e242
-0.0
4.347709396165567e-296
-1.208938052930469
7.102606943186228e-235
-Infinity
11.476300548223142
-4.921129087884145e37
Infinity
1.6799886742701454e-58

Together, these features get you going with property-based testing much quicker, simpler, and with less ceremony.

Major changes since the 0.0 version

The rest of this blog post will be about the changes in genvalidity and companions that caused the big bump from 0.x to 1.x

GenInvalid

The GenInvalid class has been removed entirely. It became clear rather quickly that was a misfeature.

The GenInvalid class let you generate values for which isValid returns False. This had many issues:

1. Values can be invalid for potentially many different reasons, and you would want separate generators for each of those reasons. In other words; The type-class does not compose.

2. Invalid values are values that can, but should not, exist so they can be nasty, like segfault-nasty. This means tests for invalid values can't even run, and that's not useful.

3. Invalid values should not exist. If they exist you've already made a mistake. It makes more sense testing that you don't produce invalid values than to test what happens if you do.

Relative Validity

The RelativeValidity type-class has been removed. It became clear that it was a misfeature as well.

1. It is a type-class without laws that does not compose.

2. There were 0 instances.

3. Any potential instances should have been expressed as functions instead.

GenValid default implementation

The default implementation of genValid and shrinkValid have been changed:

-- Old
genValid = genUnchecked `suchThat` isValid
shrinkValid = filter isValid . shrinkUnchecked

-- New
genValid = genValidStructurally
shrinkValid = shrinkValidStructurally

The old default was there to let people relax validity in newtypes, but it turns out that that makes no sense in practice. Validity composes nicely and we should use that. The change also means that genValid is now usually much faster by default.

GenUnchecked

The GenUnchecked type-class has been removed. Unchecked values have the same problems as invalid values (because they may be invalid). Now that GenUnchecked is no longer necessary to implement GenValid, it no longer serves any purpose.

Function renames

All property combinators and test suite combinators relating to GenInvalid and GenUnchecked have been removed. The ones relating to valid values have been renamed:

producesValidsOnValids -> producesValid
eqSpecOnValid          -> eqSpec

Migration guide

1. Remove all GenInvalid instances.

2. Remove all GenUnchecked instances.

3. Remove all RelativeValidity an GenRelativeValidity instances. (You probably don't have any.)

4. Rename every forAllUnchecked to forAllValid

   find . -name "*.hs" -exec sed -i 's/forAllUnchecked/forAllValid/g' {} +

5. Rename property combinators and test suite combinators:

   find . -name "*.hs" -exec sed -i 's/sOnValids//g' {} +
   find . -name "*.hs" -exec sed -i 's/OnValid//g' {} +

6. Code-review