What do you want to do? How will this make AGI go well?

I want to interpret latent reasoning using NLAs initialized with inferred reasoning.

While latent reasoning is not commonly used today, this might change as high quality pretrain data starts to run out and frontier labs are incentivized to compete in other directions to overtake each other. An additional reason for its likely future adoption is that latent reasoning has a much higher bits per forward pass than a token-based CoT does, by a theoretical 3-4 orders of magnitude.

While many current methods to watch for misalignment in models rely on reading a discrete CoT, latent reasoning is much harder to interpret. If we can interpret reasoning better, it would allow us to detect misalignment in LLMs much easier.

Specifically I want to train NLAs on the (small) latent reasoning model used in lesswrong.com/posts/YGAimivLxycZcqRFR, and then finetune larger open-weights models with CODI and train NLAs for them. I will also train baseline methods to compare this to, including normally initialized NLAs, j-lenses and other methods.

What are this project's goals? How will you achieve them?

To produce human-readable explanations of the reasoning used by LLMs finetuned to reason in latent space. We will do this by training NLAs with different initializations, most of which will be the inferred steps in various multi-hop reasoning problems, especially abstract ones.

How will this funding be used?

Renting GPUs to train NLAs and finetune models to use latent reasoning.

What are the most likely causes and outcomes if this project fails?

This might fail if the NLAs can't generalize well from the somewhat narrow initializations I give them.

What have you done in the past that proves you will be good at doing this? focus on substance, not credentials

I trained a series of BERT models of various sizes (2M, 11M, 100M params) on my local hardware for a project a few months ago in which I quantified how the meanings of words/terms changed from 1850 to 1950.

These are from a while ago but I gained a lot of intuition from them:

8th grade I implemented a word2vec autoencoder and taught myself about L1 and L2 norms by tinkering with the embedding vectors.

7th grade I taught myself about differential equations by making simulations of strange attractors, this gave me intuition about derivatives and integrals

How much money have you raised in the last 12 months, and from where?

None.