After having built a working system that I could use, Now I wanted the Super User Spark to be extremely safe and correct. Types had already helped me a lot. Now I turned to testing, rigorous testing.
Hspec
For the testing framework, I chose hspec. Hspec allows for integration with Quickcheck and HUnit and has you write a friendly DSL to define the tests.
Small-scale testing
Unit tests
When I first tried to implement tests for my code, I noticed that most of the code wasn't very test-friendly. I used a unwieldy monad transformer stack and hardly any code kept out of IO. Piece by piece, I refactored out as much pure code as I could and started to write some unit tests.
eol succeeds for a line feed succeeds for carriage return succeeds for a CRLF fails for the empty string fails for spaces fails for tabs fails for linespace
describe "eol" $ do
it "succeeds for a line feed" $ do
shouldSucceed eol "\n"
it "succeeds for a carriage return" $ do
shouldSucceed eol "\r"
it "succeeds for a CRLF" $ do
shouldSucceed eol "\r\n"
it "fails for the empty string" $ do
shouldFail eol ""
...Property tests
I started with parser tests and I quickly noticed that writing unit tests for parsers is incredibly boring, especially for combinations of parsers. Property tests offered a nice shortcut. Now I could test the linespace parser for an arbitrary number of spaces and tabs instead of just the few unit test cases I would write.
linespace succeeds for spaces succeeds for tabs succeeds for mixtures of spaces and tabs
describe "linespace" $ do
it "succeeds for spaces" $ do
forAll (listOf $ pure ' ') $ shouldSucceed linespace
it "succeeds for tabs" $ do
forAll (listOf $ pure '\t') s shouldSucceed linespace
...Property test failures
One of the nice features of property tests is that the tester, if antropomorfised, becomes obnauxious.
Me: This property should hold for ANY STRING, please test that.
Quickcheck: Okay, let's see.
Quickcheck: ...
Quickcheck: YOUR CODE IS WRONG, THAT PROPERTY DOESN'T HOLD FOR THIS STRING:
"s¨E{¨hUÿ¸§lóFgh[è//XZ["î"
Me: Okay, but they're probably not going to use that as a filepath and ...
Quickcheck: IT ALSO DOESN'T HOLD FOR THIS STRING:
"Ùá³53KìbWn/Ws9u`¸Í:c6îG¢!tG!wåDã4CXy:l19»ê|«FLh{±Bl,M Qu "
Me: Well, that's just ...
Quickcheck: AND THIS ONE:
"òcm_#ne8p®yµöï.c3æïE<½ÞÐ2íOlçúw¡ÿ´+³sjii;MiëԻ_L)N}]N\Ó@*
Ûоúm³Øçq§0u´b7µÙÒuAõLnχbY¥~FH¨`KIîjZ>jòÌúÈѤÝH§ÅF¡%àôށC¼
ƐæÇě1?K°][ºãmÌqjóY¾UùOI7©RÉ#'¬%¦ñúLGõ5iÑÈdzåR¤ɲ|ÚN脪7˘'HÄ
MdûÇÊôeñ«lۈEØøýh¨ûµ´6v®P}ÐXðյ¹ÒZo?¤µ-j§K¾ßâl|Y;ÅH)dâGà
$¾îÌՑ¹UrßÈؽýq1yDÙmיYý`^AYMt9O!¦õy¦Ï)±iú¸Ú@{8ôJA?Q¯yî»É3%"
Me: OKAY! okay, I get it!
Tests for impure functions
The most important part of spark is of course the deployer. In other words: the part that does something. These are arguably the most important parts to test.
I'd rather not test impure functions at all, but because I do have to, I refactored the impure code to pure code that outputs instructions for what to do when IO is available. Then I could test the side effects on a per-instruction basis. For example, here are some snippets from the refactored code:
data Instruction = Instruction FilePath FilePath DeploymentKind
deriving (Show, Eq)
performDeployment :: Instruction -> IO ()
performDeployment (Instruction src dst kind)
= case kind of
LinkDeployment -> link src dst
CopyDeployment -> copy src dst
copy :: FilePath -> FilePath -> IO ()
link :: FilePath -> FilePath -> IO ()Now I could test these two impure functions:
let sandbox = "test_sandbox"
let setup = createDirectoryIfMissing sandbox
let teardown = removeDirectoryRecursive sandbox
beforeAll_ setup $ afterAll_ teardown $ do
describe "copy" $ do
it "succcesfully copies this file" $ do
withCurrentDirectory sandbox $ do
let src = "testfile"
let dst = "testcopy"
writeFile src "This is a file."
diagnoseFp src `shouldReturn` IsFile
diagnoseFp dst `shouldReturn` Nonexistent
copy src dst -- Under test
diagnoseFp src `shouldReturn` IsFile
diagnoseFp dst `shouldReturn` IsFile
dsrc <- diagnose src
ddst <- diagnose dst
diagnosedHashDigest ddst `shouldBe` diagnosedHashDigest dsrc
removeFile src
removeFile dst
diagnoseFp src `shouldReturn` Nonexistent
diagnoseFp dst `shouldReturn` NonexistentRegression tests
I found quite a few bugs while testing spark. I added a regression test for all the bugs I found. Some of these look somewhat funny if you don't know their story.
Somewhere in the deployment process, directories and their files needed to be hashed recursively. I naively implemented this with lazy ByteStrings. As a result, I got this error:
resource exhausted (Too many open files)
Apparently the implementation of lazy ByteString keeps files open until the contents are used. After I replaced the lazy ByteString implementation with strict ByteStrings, all was well. Now there is a test that gives you the following output:
hashFilePath has no problem with hashing a directory of 20000 files
Automatic black-box tests
To lower the bar of entry for people who've found a problem and would like to contribute, I implemented some automatic black-box tests.
Binary black-box tests
For the parser and the compiler, there are some binary tests. These tests pass or fail based on whether the parser/compiler succeeds or fails. They only require you to add a source file to the appropriate directory.
Correct succesful parse examples test_resources/shouldParse/with_quotes.sus test_resources/shouldParse/short_syntax.sus test_resources/shouldParse/littered_with_comments.sus test_resources/shouldParse/empty_card.sus test_resources/shouldCompile/bash.sus test_resources/shouldCompile/complex.sus Correct unsuccesfull parse examples test_resources/shouldNotParse/empty_file.sus test_resources/shouldNotParse/missing_implementation.sus Correct succesful compile examples test_resources/shouldCompile/bash.sus test_resources/shouldCompile/complex.sus Correct unsuccesfull compile examples test_resources/shouldNotParse/empty_file.sus test_resources/shouldNotParse/missing_implementation.sus
Exact black-box tests
For the compiler, there are some black-box tests that require you to add both a card file and the exact desired compilation result to the appropriate directory.
exact tests test_resources/exact_compile_test_src/bash.sus test_resources/exact_compile_test_src/alternatives.sus test_resources/exact_compile_test_src/internal_sparkoff.sus test_resources/exact_compile_test_src/nesting.sus test_resources/exact_compile_test_src/sub.sus test_resources/exact_compile_test_src/sub/subfile.sus
End-to-end tests
To ensure end-to-end correctness, there are also some tests that cover the full spectrum of operation. From argument parsing to interaction with the file system.
This kind of test simply calls the main :: IO () function with withArgs :: [String] -> IO a -> IO a and specific arguments.
EndToEnd
standard bash card test
parses correcty
compiles correctly
checks without exceptions
deploys correctly