# The nofib build system

This documents usage of nofib's Shake-based build system. The build system,
`nofib-run`, automates running of the nofib benchmarks and the collection of a
variety of compile- and run-time metrics. These metrics are recorded in a
common semi-structured CSV file format, which can be compared and analysed with
the included `nofib-compare` utility.


## Usage

The `nofib-run` executable is used to run the benchmark suite and accepts a variety of arguments.
By default the results of the run will be placed in `_make/{compiler version}`.
The compiler version component of this path can be overridden with the
`--output` flag.

For instance (run from the root `nofib` directory),
```
$ cabal new-run -- nofib-run --compiler=/path/to/ghc --output=test
```
will produce a number of files in `_make/`:
```
$ ls -R _make

_make:
8.11.0.20200205

_make/8.11.0.20200205:
gc  imaginary  parallel  real  spectral

_make/8.11.0.20200205/gc:
cacheprof  circsim  constraints  fibheaps  gc_bench  happy  hash  lcss  linear  mutstore1  mutstore2  spellcheck  treejoin

_make/8.11.0.20200205/gc/cacheprof:
Arch_x86.hi  Arch_x86.o.result        Arch_x86.o.stats  Generics.hi  Generics.o.result        Generics.o.stats  Main.deps  Main.o         Main.o.results.csv   Main.result
Arch_x86.o   Arch_x86.o.results.csv   config.txt        Generics.o   Generics.o.results.csv   Main              Main.hi    Main.o.result  Main.o.stats         Main.results.csv 

...
```
In the case of the `gc/cacheprof` benchmark we see the following were produced:

 * build artifacts from the build of the benchmark (e.g. `*.o`, `*.hi`)
 * for each object file a `.results.csv` file which contains the metrics for the object size
 * `Main.results.csv`, which aggregates all of the above `*.results.csv` files


A number of classes of metrics are collected:

 * object code size for each module
 * runtime system statistics (e.g. mutator time, GC time, GC counts,
   allocations) from the execution of `ghc` while compiling the testcase
 * runtime system statistics from the execution of the testcase

### Cachegrind

The benchmarks can also be run under valgrind's
[cachegrind](https://valgrind.org/docs/manual/cg-manual.html) tool with the
`--cachegrind`, which simulates a simple cache hierarchy to allow (mostly)
deterministic modelling of instruction counts, memory, and cache effects.
When running with `--cachegrind` tests can be safely parallelised with the
`-j<n>` flag.


### Performance counters with `perf_events`

The benchmarks can also be run under the Linux `perf` tool for collection of
(micro-)architectural event counts. This mode is enabled 

## Data format

The output of `nofib-run` is a simple CSV format (defined by the `Measurements`
type in `src/Measurements.hs`) represents semi-structured string keys (known as
`Label`s) and floating-point values. In particular, labels have a path-like
structure with `/` being the component delimiter.

## Comparing results

```
$ cabal new-run nofib-compare -- approach-b.results.csv approach-a.results.csv
```
