A calculator for distributions, for Fermi estimation

This project is a minimalist, calculator-style DSL for fermi estimation. It can multiply, divide, add and substract scalars, lognormals and beta distributions, and supports variables.

Motivation

Sometimes, Squiggle, simple squiggle or squiggle.c are still too complicated and un-unix-like.

Installation

make build
sudo make install
fermi 

Usage

$ fermi
5000000 12000000
=> 5.0M 12.0M
* beta 1 200
=> 1.9K 123.1K
* 30 180
=> 122.9K 11.7M
/ 48 52
=> 2.5K 234.6K
/ 5 6
=> 448.8 43.0K
/ 6 8
=> 64.5 6.2K
/ 60
=> 1.1 103.7

Perhaps this example is more understandable with comments and better units:

$ fermi
5M 12M           # number of people living in Chicago
beta 1 200       # fraction of people that have a piano
30 180           # minutes it takes to tune a piano, including travel time
/ 48 52          # weeks a year that piano tuners work for
/ 5 6            # days a week in which piano tuners work
/ 6 8            # hours a day in which piano tuners work
/ 60             # minutes to an hour
=: piano_tuners

If you type "help", you can see a small grammar:

$ fermi
help
  Operation | Variable assignment | Special
    Operation:                             operator operand
          operator:                        (empty) | * | / | + | -
          operand:                         scalar | lognormal | beta | variable
            lognormal:                     low high
            beta:                          beta alpha beta
    Variable assignment:                   =: variable_name
    Variable assignment and clear stack:   =. variable_name
    Special:
         Comment:                          # this is a comment
         Clear stack:                      clear | c | .
         Print debug info:                 debug | d
         Print help message:               help  | h
         Start additional stack:           operator (
         Return from additional stack      )
         Exit:                             exit  | e
  Examples:
    + 2
                    # this is a comment
    / 2.5           # this is an operation followed by a comment
    * 1 10          # "low high" is interpreted as lognormal
    + 1 10
    * beta 1 10
    1 10            # multiplication taken as default operation)
    =: x
    .               # return the stack to 1.
    1 100
    + x
    * 1 12 
    * (
    1 10
    + beta 1 100
    )
    =. y            # save to variable and clear stack
    exit

You can see real life examples here, here, here, here, here, here

Tips & tricks

• It's conceptually clearer to have all the multiplications first and then all the divisions
• For things between 0 and 1, consider using a beta distribution

Command line options

You can specify the number of samples to draw when algebraic manipulations are not sufficient:

$ fermi -n 1000000
$ fermi -n 1_000_000

You also run a file with the -f option

$ fermi -f more/piano-tuners.fermi 

Integrations with linux utilities

Because the model reads from standard input, you can a model to it:

$ cat more/piano-tuners.fermi | fermi

You can make a model an executable file by running $ chmod -x model.fermi and then adding the following at the top!

#!/bin/usr/fermi -f

You can save a session to a logfile with tee:

fermi | tee -a fermi.log

Different levels of complexity

The top level f.go file (420 lines) has a bunch of complexity: variables, parenthesis, samples, beta distributions, number of samples, etc. In the simple/ folder:

• f_simple.go (370 lines) strips variables and parenthesis, but keeps beta distributions, samples, and addition and substraction
• f_minimal.go (140 lines) strips everything that isn't lognormal and scalar multiplication and addition, plus a few debug options.

Roadmap

Done:

• Write README
• Add division?
• Read from file?
• Save to file?
• Allow comments?
  • Use a sed filter?
  • Add proper comment processing
• Add show more info version
• Scalar multiplication and division
• Think how to integrate with squiggle.c to draw samples
  • Copy the time to botec go code
  • Define samplers
  • Call those samplers when operating on distributions that can't be operted on algebraically
• Display output more nicely, with K/M/B/T
• Consider the following: make this into a stack-based DSL, with:
  • Variables that can be saved to and then displayed
  • Other types of distributions, particularly beta distributions? => But then this requires moving to bags of samples. It could still be ~instantaneous though.
  • Added bags of samples to support addition and multiplication of betas and lognormals
• Figure out go syntax for
  • Maps
  • Joint types
  • Enums
• Fix correlation problem, by spinning up a new randomness thing every time some serial computation is done.
• Clean up error code. Right now only needed for division
• Maintain both a more complex thing that's more featureful and the more simple multiplication of lognormals thing.
• Allow input with K/M/T
• Document parenthesis syntax
• Specify number of samples as a command line option
• Figure out how to make models executable, by adding a #!/bin/bash-style command at the top?
• Make -n flag work
• Add flag to repeat input lines (useful when reading from files)

To (possibly) do:

• Add functions. Now easier to do with an explicit representation of the stakc
• Think about how to draw a histogram from samples
• Dump samples to file
• Represent samples/statistics in some other way
• Perhaps use qsort rather than full sorting
• Program into a small device, like a calculator?

Discarded:

• Think of some way of calling bc