109 lines
4.8 KiB
Markdown
109 lines
4.8 KiB
Markdown
Models of Bayesian-like updating under constrained compute
|
|
==========================================================
|
|
|
|
This repository contains some implementations of models of bayesian-like updating under constrained compute. The world in which these models operate is the set of sequences from the [Online Encyclopedia of Integer Sequences](https://oeis.org/), which can be downloaded from [here](https://oeis.org/wiki/JSON_Format,_Compressed_Files).
|
|
|
|
|
|
|
|
## Models
|
|
|
|
Given the start of a sequence of integers, what is the probability of possible completions? This post offers various answers:
|
|
|
|
### Unconstrained prediction
|
|
|
|
Just look at all possible sequence, and assign a probability to each continuation based on its frequency across all sequences
|
|
|
|
### Prediction with access to subsequently more hypotheses.
|
|
|
|
Look at 10%, 20%,..., 80%, 90%, 100% of sequences, and give the probability of each continuation based on its frequencies across n% of sequences
|
|
|
|
### Just in time Bayesianism
|
|
|
|
As described [here](https://nunosempere.com/blog/2023/02/04/just-in-time-bayesianism/), Just-in-time bayesianism is this scheme:
|
|
|
|
|
|
|
|
I think that the implementation here provides some value in terms of making fuzzy concepts explicit. For example, instead of "did your past hypotheses do an ok job at predicting this new evidence", I have code which looks at whether past predictions included the correct completion, and which expand the search for hypotheses if not.
|
|
|
|
I later elaborated on this type of scheme on [A computable version of Solomonoff induction](https://nunosempere.com/blog/2023/03/01/computable-solomonoff/).
|
|
|
|
### Infrabayesianism
|
|
|
|
Mini-infrabayesianism part I
|
|
|
|
- Mini-infrabayesianism part II:
|
|
|
|
## Built with
|
|
|
|
- [nim](https://nim-lang.org/)
|
|
- [make](https://www.gnu.org/software/make/) (optional)
|
|
|
|
Why nim? Because it is nice to use and [freaking fast](https://github.com/NunoSempere/time-to-botec#comparison-table).
|
|
|
|
## Getting started
|
|
|
|
### Prerequisites
|
|
|
|
Install [nim](https://nim-lang.org/install.html) and make.
|
|
|
|
### Compilation
|
|
|
|
```
|
|
git clone https://git.nunosempere.com/personal/compute-constrained-bayes.git
|
|
cd compute-constrained-bayes
|
|
cd src
|
|
make deps ## get dependencies
|
|
make fast ## also make, or make build, for compiling it with debug info.
|
|
./compute-constrained-bayes
|
|
```
|
|
|
|
## Contributions
|
|
|
|
Contributions are very welcome, particularly around:
|
|
|
|
- [ ] Making the code more nim-like, using nim's standard styles and libraries
|
|
- [ ] Adding another example which is not logloss minimization for infrabayesianism
|
|
- [ ]
|
|
|
|
## Roadmap
|
|
|
|
- [x] Exploration of OEIS data
|
|
- [x] Subdivide subsequent tasks into steps
|
|
- [x] Simple prediction of the next integer
|
|
- [x] Simple predictions v1
|
|
- [x] Wrangle the return types to something semi-elegant
|
|
- [-] Maybe add some caching, e.g., write continuations to file, and read them next time.
|
|
- [x] JIT Bayesianism:
|
|
- [x] Function to predict with a variable number of hypotheses
|
|
- [x] Function to start predicting with a small number of hypotheses, and get more if the initial ones aren't enough.
|
|
- [x] Add the loop of: start with some small number of sequences, and if these aren't enough, read more.
|
|
- [x] Clean-up
|
|
- [ ] Infrabayesianism
|
|
- [ ] Infrabayesianism x1: Predicting interleaved sequences.
|
|
- Yeah, actually, I think this just captures an implicit assumption of Bayesianism as actually practiced.
|
|
- [x] Infrabayesianism x2: Deterministic game of producing a fixed deterministic prediction, and then the adversary picking whatever minimizes your loss
|
|
- I am actually not sure of what the procedure is exactly for computing that loss. Do you minimize over subsequent rounds of the game, or only for the first round? Look this up.
|
|
- [ ] Also maybe ask for help from e.g., Alex Mennen?
|
|
- [x] Lol, minimizing loss for the case where your utility is the logloss is actually easy.
|
|
|
|
---
|
|
|
|
An implementation of Infrabayesianism over OEIS sequences.
|
|
<https://oeis.org/wiki/JSON_Format,_Compressed_Files>
|
|
|
|
Or "Just-in-Time bayesianism", where getting a new hypothesis = getting a new sequence from OEIS which has the numbers you've seen so far.
|
|
|
|
Implementing Infrabayesianism as a game over OEIS sequences. Two parts:
|
|
1. Prediction over interleaved sequences. I choose two OEIS sequences, and interleave them: a1, b1, a2, b2.
|
|
- Now, you don't have hypothesis over the whole set, but two hypothesis over the
|
|
- I could also have a chemistry like iteration:
|
|
a1
|
|
a2 b1
|
|
a3 b2 c1
|
|
a4 b3 c2 d1
|
|
a5 b4 c3 d2 e1
|
|
.................
|
|
- And then it would just be computationally absurd to have hypotheses over the whole
|
|
|
|
2. Game where: You provide a deterministic procedure for estimating the probability of each OEIS sequence giving a list of trailing examples.
|