Announcing opt-env-conf

This post announces opt-env-conf, a new settings parsing library for Haskell. This library combines the best of optparse-applicative, envparse, and autodocodec, together with the colours from safe-coloured-text to provide all your apps' runtime settings.

Parsing Command-line Arguments, Environment variables, and Configuration files

In just about any application I've worked on, I've always had a very similar module called FooBar.OptParse that does vaguely the same thing every time:

1. Parse command-line arguments into an Arguments or Flags type.

2. Parse environment variables into an Environment type.

3. Use those two to determine out which configuration file to use

4. Parse the configuration file(s) into a Configuration type.

5. Combine the three structures into a Settings or Dispatch type.

I've even put together and sold an Optparse Template on the CS SYD Haskell Templates. (It may have been updated to use opt-env-conf by the time you read this.)

I'd tried and failed many times to turn this whole process into a library instead of a template but about a decade later I think I've finally figured it out.

Settings

The opt-env-conf library uses building blocks called settings and composes them together into a larger Parser. It uses my favourite little EDSL technique: the Free (Selective) Applicative Functor. You will recognise it from the autodocodec library. This technique allows the library to both run and document a parser completely.

The Parser type cannot have a Monad instance because that would make it impossible to generate documentation, but using the ApplicativeDo language extension we can still write code as if it did.

Here is a little worked example:

{-# LANGUAGE ApplicativeDo #-}
{-# LANGUAGE RecordWildCards #-}

module Main where

import Data.Text (Text)
import OptEnvConf
import Paths_opt_env_conf_test (version)

main :: IO ()
main = do
  settings <- runSettingsParser version "Run the foo-bar server"
  runServer settings

runServer :: Settings -> IO ()
runServer = undefined

data Settings = Settings
  { settingPort :: Int,
    settingPaymentSettings :: Maybe PaymentSettings
  }

instance HasParser Settings where
  settingsParser = subEnv "FOO_BAR_" $ withLocalYamlConfig $ do
    settingPort <-
      setting
        [ help "the port to serve web requests on",
          reader auto,
          option,
          long "port",
          env "PORT",
          conf "port",
          metavar "PORT",
          value 8080
        ]
    settingPaymentSettings <- optional $ subSettings "payment"
    pure Settings {..}

data PaymentSettings = PaymentSettings
  { paymentSettingPublicKey :: Text,
    paymentSettingPrivateKey :: Text
  }

instance HasParser PaymentSettings where
  settingsParser = do
    paymentSettingPublicKey <-
      setting
        [ help "public key",
          reader str,
          name "public-key",
          metavar "PUBLIC_KEY"
        ]
    paymentSettingPrivateKey <-
      mapIO readSecretTextFile $
        filePathSetting
          [ help "private key file",
            reader str,
            name "private-key-file",
            metavar "PRIVATE_KEY_FILE"
          ]
    pure PaymentSettings {..}

(Please make sure to look up the more recent worked example if you are reading this in the future.)

You'll notice some interesting things in this example above:

• The runSettingsParser function requires a version so that --version can work automatically.

• The runSettingsParser function requires a program description so that a manpage can be generated.

• You need only one Settings type.

• Settings types can have a HasParser instance for easy composition of Settings' types.

• The subEnv function lets you prefix environment variable names below. Similar functions exist for arguments and configuration values.

• Functions like withLocalYamlConfig lets you define how configuration can be loaded.

• You can define a setting with the setting function.

• The setting function takes a list of setting Builders.

• Within one setting you can define multiple ways to parse it:

  • option: Declare an option

  • long: Declare a name for the option

  • env: Declare an environment variable to parse

  • conf: Declare a configuration value to parse

  • name: All of the above.

  • value: Set a default value

• You can use standard combinators like optional to combine Parsers.

• You can use subSettings to bring in more settings from another type with a HasParser instance.

• You can use mapIO to run IO actions from within a parser to continue parsing.

• There are standard settings like filePathSetting with common settings Parsers.

Linting step

Running a Parser starts with a linting step. In this step, a Parser is inspected for common mistakes.

This linting step seemed fitting because it allowed the library interface to be much more simply typed, and it is done at a phase in a program where exiting is to be expected anyway.

The linting step will warn against issues like these:

• You haven't documented the setting with help.

• The setting does not parse anything because you haven't added any Builders that parse something.

• You have declared a setting as an option but not added a long

• You have declared a setting that reads a configuration variable but not declared any way to load a configuration file.

Each of these errors are shown in colour, and with a reference to the code. Here is an example:

Generated documentation

The opt-env-conf library automatically generates certain options for any parser. We've already discussed --version, but there are others such as --help. The above example generates the following --help page:

As you can see, this is in colour, because why wouldn't this be colourful?

The exact format of this page is still open for improvements (and potentially configuration).

Generated manpage

When an opt-env-conf parser sees --render-man-page, it generates a file that man can render. For example, the example above produces the following man page:

Completion

When using your binary on the command-line, you expect that pressing <tab> auto-completes options and arguments for your program. This is the most "under construction" part of opt-env-conf, because it turns out to be quite complicated and difficult to try out, but all the infrastructure is already in place. You can produce completion scripts with the --bash-completion-script, --zsh-completion-script, and --fish-completion-script options. These can then be sourced from the right places to enable dynamic auto-completion.

Nix integration

To install the man pages and completion scripts in the right places, opt-env-conf provides passthru functions in its overrides.nix. You can use them when packaging up your executable:

opt-env-conf-example =
  self.opt-env-conf.installManpagesAndCompletions [ "opt-env-conf-example" ]
    (self.callPackage ./opt-env-conf-example {});

Testing

The opt-env-conf library comes with a companion library for testing called opt-env-conf-test. It lets you define common tests such as "This Settings parser is Lint-error-free, or golden tests for Settings' documentation.

License

The opt-env-conf library is available freely under the LGPL-3 license with a static linking exception (Just like the OCaml License). In short: You can use opt-env-conf in any of your applications for free, but if you fork opt-env-conf, you (probably) have to make that fork (but not your app) available under the same LGPL license. The static linking exception says that you can also statically link (as is usual in Haskell) against opt-env-conf.

Conclusion

The opt-env-conf library is already used in production. It is available on Hackage and on GitHub, and it is ready to try. Please give it a go and send me your feedback.