README.lhs

# Scientist

[![Hackage](https://img.shields.io/hackage/v/scientist.svg?style=flat)](https://hackage.haskell.org/package/scientist)
[![Stackage Nightly](http://stackage.org/package/scientist/badge/nightly)](http://stackage.org/nightly/package/scientist)
[![Stackage LTS](http://stackage.org/package/scientist/badge/lts)](http://stackage.org/lts/package/scientist)
[![CI](https://github.com/freckle/scientist-hs/actions/workflows/ci.yml/badge.svg)](https://github.com/freckle/scientist-hs/actions/workflows/ci.yml)

Haskell port of
[`github/scientist`](https://github.com/github/scientist#readme).

## Usage

The following extensions are recommended:

```haskell
{-# LANGUAGE OverloadedStrings #-}
```

<!--
```haskell
{-# LANGUAGE DeriveAnyClass #-}
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE DerivingStrategies #-}
{-# LANGUAGE LambdaCase #-}

module Main (module Main) where

import Prelude

import Control.Monad.IO.Class (MonadIO(..))
import Control.Monad.IO.Unlift (MonadUnliftIO)
import Data.Foldable (for_)
import Data.Text (Text)
import UnliftIO.Exception (SomeException, throwIO, displayException)
```
-->

Most usage will only require the top-level library:

```haskell
import Scientist
```

<!--
```haskell
import Text.Markdown.Unlit ()

theOriginalCode :: m a
theOriginalCode = undefined

theExperimentalCode :: m a
theExperimentalCode = undefined
```
-->

1. Define a new `Experiment m c a b` with,

   ```haskell
   ex0 :: Functor m => Experiment m c a b
   ex0 = newExperiment "some name" theOriginalCode
   ```

1. The type variables capture the following details:

   - `m`: Some Monad to operate in, e.g. `IO`.
   - `c`: Any context value you attach with `setExperimentContext`. It will then
     be available in the `Result c a b` you publish.
   - `a`: The result type of your original (aka "control") code, this is what is
     always returned and so is the return type of `experimentRun`.
   - `b`: The result type of your experimental (aka "candidate") code. It
     probably won't differ (and must not to use a comparison like `(==)`), but
     it can (provided you implement a comparison between `a` and `b`).

1. Configure the Experiment as desired

   ```haskell
   ex1 :: (Functor m, Eq a) => Experiment m c a a
   ex1 =
     setExperimentPublish publish0
       $ setExperimentCompare experimentCompareEq
       $ setExperimentTry theExperimentalCode
       $ newExperiment "some name" theOriginalCode

   -- Increment Statsd, Log, store in Redis, whatever
   publish0 :: Result c a b -> m ()
   publish0 = undefined
   ```

1. Run the experiment

   ```haskell
   run0 :: (MonadUnliftIO m, Eq a) => m a
   run0 =
     experimentRun
       $ setExperimentPublish publish0
       $ setExperimentCompare experimentCompareEq
       $ setExperimentTry theExperimentalCode
       $ newExperiment "some name" theOriginalCode
   ```

1. Explore things like `setExperimentIgnore`, `setExperimentEnabled`, etc.

---

The rest of this README matches section-by-section to the ported project and
shows only the differences in syntax for those features. Please follow the
header links for additional details, motivation, etc.

## [How do I science?](https://github.com/github/scientist#how-do-i-science)

<!--
```haskell
data Model = Model

data User = User

userLogin :: User -> Text
userLogin = undefined

userCanRead :: User -> Model -> m Bool
userCanRead = undefined

modelCheckUserValid :: Model -> User -> m Bool
modelCheckUserValid = undefined
```
-->

```haskell
myWidgetAllows :: MonadUnliftIO m => Model -> User -> m Bool
myWidgetAllows model user = do
  experimentRun
    $ setExperimentTry
        (userCanRead user model) -- new way
    $ newExperiment "widget-permissions"
        (modelCheckUserValid model user) -- old way
```

## [Making science useful](https://github.com/github/scientist#making-science-useful)

```haskell
run1 :: MonadUnliftIO m => m a
run1 =
  experimentRun
    $ setExperimentEnabled (pure True)
    $ setExperimentOnException onScientistException
    $ setExperimentPublish (liftIO . putStrLn . formatResult)
    $ setExperimentTry theExperimentalCode
    $ newExperiment "some name" theOriginalCode

onScientistException :: MonadIO m => SomeException -> m ()
onScientistException ex = do
  liftIO $ putStrLn "..."

  -- To re-raise
  throwIO ex

formatResult :: Result c a b -> String
formatResult = undefined
```

### [Controlling comparison](https://github.com/github/scientist#controlling-comparison)

<!--
```haskell
userServiceFetch :: m [User]
userServiceFetch = undefined

fetchAllUsers :: m [User]
fetchAllUsers = undefined
```
--->

```haskell
run2 :: MonadUnliftIO m => m [User]
run2 =
  experimentRun
    $ setExperimentCompare (experimentCompareOn $ map userLogin)
    $ setExperimentTry userServiceFetch
    $ newExperiment "users" fetchAllUsers
```

When using `experimentCompareOn`, `By`, or `Eq`, if a candidate branch raises an
exception, that will never compare equally.

### [Adding context](https://github.com/github/scientist#adding-context)

See `setExperimentContext`.

### [Expensive setup](https://github.com/github/scientist#expensive-setup)

Just do it ahead of time.

<!--
```haskell
data X = X

expensiveSetup :: m X
expensiveSetup = undefined

theOriginalCodeWith :: X -> m a
theOriginalCodeWith = undefined

theExperimentalCodeWith :: X -> m a
theExperimentalCodeWith = undefined
```
-->

```haskell
run3 :: MonadUnliftIO m => m a
run3 = do
  x <- expensiveSetup

  experimentRun
    $ setExperimentTry (theExperimentalCodeWith x)
    $ newExperiment "expensive" (theOriginalCodeWith x)
```

### [Keeping it clean](https://github.com/github/scientist#keeping-it-clean)

Not supported at this time. Format the value(s) as necessary when publishing.

### [Ignoring mismatches](https://github.com/github/scientist#ignoring-mismatches)

See `setExperimentIgnore`.

### [Enabling/disabling experiments](https://github.com/github/scientist#enablingdisabling-experiments)

See `setExperimentRunIf`.

### [Ramping up experiments](https://github.com/github/scientist#ramping-up-experiments)

```haskell
run4 :: MonadUnliftIO m => m a
run4 =
  experimentRun
    $ setExperimentEnabled (experimentEnabledPercent 30)
    $ setExperimentTry theExperimentalCode
    $ newExperiment "some name" theOriginalCode
```

### [Publishing results](https://github.com/github/scientist#publishing-results)

<!--
```haskell
data Value = Value

data Pair = Pair

toJSON :: a -> Value
toJSON = undefined

(.=) :: Text -> a -> Pair
(.=) = undefined

object :: [Pair] -> Value
object = undefined

data MyContext = MyContext

data MyPayload = MyPayload
  { name :: Text
  , context :: Maybe MyContext
  , control :: Value
  , candidate :: Value
  , execution_order :: [Text]
  }

statsdTiming :: Text -> a -> m ()
statsdTiming = undefined

statsdIncrement :: Text -> m ()
statsdIncrement = undefined

redisLpush :: Text -> Value -> m ()
redisLpush = undefined

redisLtrim :: Text -> Int -> Int -> m ()
redisLtrim = undefined
```
-->

```haskell
run5 :: MonadUnliftIO m => m User
run5 =
  experimentRun
    $ setExperimentPublish publish1
    $ setExperimentTry theExperimentalCode
    $ newExperiment "some name" theOriginalCode

publish1 :: MonadIO m => Result MyContext User User -> m ()
publish1 result = do
  -- Details are present unless it's a ResultSkipped, which we'll ignore
  for_ (resultDetails result) $ \details -> do
    let eName = resultDetailsExperimentName details

    -- Store the timing for the control value,
    statsdTiming ("science." <> eName <> ".control")
      $ resultControlDuration
      $ resultDetailsControl details

    -- for the candidate (only the first, see "Breaking the rules" below,
    statsdTiming ("science." <> eName <> ".candidate")
      $ resultCandidateDuration
      $ resultDetailsCandidate details

    -- and counts for match/ignore/mismatch:
    case result of
      ResultSkipped{} -> pure ()
      ResultMatched{} -> do
        statsdIncrement $ "science." <> eName <> ".matched"
      ResultIgnored{} -> do
        statsdIncrement $ "science." <> eName <> ".ignored"
      ResultMismatched{} -> do
        statsdIncrement $ "science." <> eName <> ".mismatched"
        -- Finally, store mismatches in redis so they can be retrieved and
        -- examined later on, for debugging and research.
        storeMismatchData details

storeMismatchData :: Monad m => ResultDetails MyContext User User -> m ()
storeMismatchData details = do
  let
    eName = resultDetailsExperimentName details
    eContext = resultDetailsExperimentContext details

    payload = MyPayload
      { name = eName
      , context = eContext
      , control = controlObservationPayload $ resultDetailsControl details
      , candidate = candidateObservationPayload $ resultDetailsCandidate details
      , execution_order = resultDetailsExecutionOrder details
      }

    key = "science." <> eName <> ".mismatch"

  redisLpush key $ toJSON payload
  redisLtrim key 0 1000

controlObservationPayload :: ResultControl User -> Value
controlObservationPayload rc =
  object ["value" .= cleanValue (resultControlValue rc)]

candidateObservationPayload :: ResultCandidate User -> Value
candidateObservationPayload rc = case resultCandidateValue rc of
  Left ex -> object ["exception" .= displayException ex]
  Right user -> object ["value" .= cleanValue user]

-- See "Keeping it clean" above
cleanValue :: User -> Text
cleanValue = userLogin
```

See `Result`, `ResultDetails`, `ResultControl` and `ResultCandidate` for all the
available data you can publish.

### [Testing](https://github.com/github/scientist#testing)

**TODO**: `raise_on_mismatches`

#### [Custom mismatch errors](https://github.com/github/scientist#custom-mismatch-errors)

**TODO**: `raise_with`

### [Handling errors](https://github.com/github/scientist#handling-errors)

#### [In candidate code](https://github.com/github/scientist#in-candidate-code)

Candidate code is wrapped in `tryAny`, resulting in `Either SomeException`
values in the result candidates list. We use the [safer][blog]
`UnliftIO.Exception` module.

[blog]: https://www.fpcomplete.com/haskell/tutorial/exceptions/

#### [In a Scientist callback](https://github.com/github/scientist#in-a-scientist-callback)

See `setExperimentOnException`.

## [Breaking the rules](https://github.com/github/scientist#breaking-the-rules)

### [Ignoring results entirely](https://github.com/github/scientist#ignoring-results-entirely)

```haskell
nope0 :: Experiment m c a b -> Experiment m c a b
nope0 = setExperimentIgnore (\_ _ -> True)
```

Or, more efficiently:

```haskell
nope1 :: Experiment m c a b -> Experiment m c a b
nope1 = setExperimentCompare (\_ _ -> True)
```

### [Trying more than one thing](https://github.com/github/scientist#trying-more-than-one-thing)

If you call `setExperimentTry` more than once, it will append (not overwrite)
candidate branches. If any candidate is deemed ignored or a mismatch, the
overall result will be.

`setExperimentTryNamed` can be used to give branches explicit names (otherwise,
they are "control", "candidate", "candidate-{n}"). These names are visible in
`ResultControl`, `ResultCandidate`, and `resultDetailsExecutionOrder`.

### [No control, just candidates](https://github.com/github/scientist#no-control-just-candidates)

Not supported.

Supporting the lack of a Control branch in the types would ultimately lead to a
runtime error if you attempt to run such an `Experiment` without having and
naming a Candidate to use instead, or severely complicate the types to account
for that safely. In our opinion, this feature is not worth either of those.

<!--
```haskell
main :: IO ()
main = pure ()
```
-->

---

[LICENSE](./LICENSE) | [CHANGELOG](./CHANGELOG.md)