Blue-Dot Impact AI Safety Fundamentals Course Spring 2024 Final Project

by Raymond Joseph Gabriel Tana

Submitted June 10, 2024

Table of Contents

Introduction to Project

Experimentation

Motivation

Theoretical computer science formalizes the equivalence between the ability to compress and the ability to perform induction for generative AI models. In the limit of stronger compression capabilities, one encounters a theoretically perfect learner: AIXI. What can be said about the compression capabilities of current generative models?

TLDR: Project Takeaway

The main points one should take away from this project are as follows:

1. An understanding of the theoretical connections between compression and induction.
2. A sense that large language models likely do not (yet) qualify as general-compression algorithms; thus are “far” from performing universal induction.
3. A series of jumping-off points for further research in this area, including the measuring of a model’s compression capabilities via latent representation sizes in a manner inspired by Shannon information theory.

How to Navigate this Landing Page

Background and further resources can be found in the Introduction page.

Details on my project’s experiments and opportunities for further research can be found in the Experimentation page.