nunosempere.com/blog/2023/03/01/computable-solomoff/index.md

A computable version of Solomoff induction

Thinking about Just-in-time Bayesianism a bit more, here is a computable approximation to Solomonoff Induction, which converges to the Turing machine generating your trail of bits in finite time.

The key idea: arrive at the correct hypothesis in finite time

1. Start with a finite set of turing machines, \{T_0, ..., T_n\}
2. If none of the T_i predict your trail bits, (B_0, ..., B_m), compute the first m steps of Turing machine T_{n+1}. If T_{n+1} doesn't predict them either, go to T_{n+2}, and so on^1
3. Observe the next bit, purge the machines from your set which don't predict it. If none predict it, GOTO 2.

Then in finite time, you will arrive at a set which only contains the simplest TM which describes the process generating your train of bits. Proof:

• Let T_j be the simplest machine which describes the process generating your trail of bits. Then by virtue of it being such, all TM machines \{T_1,...,T_{j-1}\} are not the simplest machine generating your trail of bits. Meaning that at some point, they predict something distinct from your trail of bits, meaning that at some point step 2. of the process above discards them. In particular, the step after discarding T_{j-1} is arriving at T_j, and staying there forever.
• Therefore, this process arrives at the simplest TM which describes the process generating your trail of bits in finite time.

QED.

Using the above scheme to arrive at a probability

Now, the problem with the above scheme is that if we use our set of turing machines to output a probability for the next bit

 P\Big(b_{m + 1} = 1 | (B_0, ..., B_m) \Big) := \frac{1}{n} \cdot \sum_0^n \Big(T_i(m+1) = 1\Big)  

then our probabilities are going to be very janky. For example, at step j - 1, that scheme would output a 0% probability to the correct value of the next bit.

To fix this, in step 2, we can require that there be not only one turing machine that predicts all past bits, but multiple of them. What multiple? Well, however many your compute and memory budgets allow. This would make your probabilities less janky.

Interestingly, that scheme also suggests that there is a tradeoff between arriving at the correct hypothesis as fast as possible—in which case we would just implement the first scheme at full speed—and producing accurate probabilities—in which case it seems like we would have to use a tweak

^1: Here we assume that we have an ordering of Turing machines, i.e., that T_i is simpler than T_(i+1)