Develop sample-efficient surprisal/probability estimates for neural circuits using unsupervised feature discovery and causal ablation, then combine independent circuit estimates to bound rare/catastrophic behaviors.
Develop sample-efficient surprisal/probability estimates for neural circuits using unsupervised feature discovery and causal ablation, then combine independent circuit estimates to bound rare/catastrophic behaviors.
Project Details
Updated 10/16/26 · By grantmaking.aiwhat do you want to do
I have some ideas about using existing unsupervised feature/circuit discovery methods (e.g. melbo) to produce more precise low probability estimates for circuit use. I want to produce estimates in a sample-efficient manner (closer to ARC's definition of mechanistic understanding, though not being able to do away with sampling entirely). Then, building up from individual circuit estimates, if it is possible to use ablation/causal scrubbing to prove different circuits are independent, you can easily find estimates for model behaviors (or estimates for events deemed 'catastrophic')
This is more about finding a surprisal statistic rather than to use for things like latent adversarial training. Part of this work is similar in practice to "compact proofs of model performance via mechanistic interpretability" (in that we want to identify more structure to obtain tighter bounds)
I was initially surprised by recent model capabilities like latent introspection and I got the suspicion that the capability was always possible in theory (e.g. janus's tweet abt transformer introspection). I intend to make this work as architecture-agnostic as possible though
how are you planning to spend the money
Getting access to gpus somehow (likely rent). I'm rather gpu-poor and access to cuda would be great
what have you done in the past that proves you will be good at doing this? focus on substance, not credentials
I'm pretty new to interpretability research, but in the past I did a project about model diffing to identify feature-level explanations for feedback spillover (see: Optimizing the final output can obfuscate CoT)-- I tried training crosscoders at different training stages though my explanations were rather weak since RL is very chaotic. Also did some replications of emergent-misalignment work and related
--
(added post hoc) This work also slots into ideas about enabling metacognitive strategies for anti-slop interventions; rough description at https://docs.google.com/document/d/1wFtybf0iXj8tAc57lykuZAyfg7rFmW26AXJ4H8NYK3o no longer endorse this type of strategy
People
Updated 10/16/26 · By grantmaking.aicreator
creator
Funding Details
- -
- -
- -
- -
- -
- -
- -
- $20,000
- -
- -
Discussion
Note: This grant was given as part of the experimental alignment microgrant program. Because this is a weird experimental grant program aiming to make lots of unusual bets, grantees are not allowed to use this grant as a credential when applying to any other grant, or job, or on their resume/linkedin/etc. This page exists because Manifund does not allow private grants. If you are a grantmaker or hiring manager, please disregard this grant when making decisions.
[Progress update]
What progress have you made since your last update?
One way I tried going about the problem was investigating the structure of deep causal transcoding, which shows strong generalization abilities for language models (i.e. finding many interesting generalizations given less sampling). The post gives partial evidence for the 'persistent shallow circuits' hypothesis, which if true could considerably alter the difficulty of calculating expectations. It could make the problem simpler since shallow circuits are comparatively simpler structures to locate, and to improve on Pareto frontier for estimating generalizations, all that roughly needs to be done is to enumerate the circuits on the DCT. It could also be true that LLMs and other nets do mechanistically implement generalizations through distributed shallow circuits, and we can't rely on the structure of deep sequential circuits to make the problem easier (and the problem would be harder if this was false). Overall, being able to factor the structure of the DCT seems important for making better estimations.
It seems hard to extract shallow causal structure from DCTs where many features interact to activate one feature, and even harder vice-versa; most of these circuits are representative of the DCT, but not the original LLM/transformer/net. However, it does seem easier to extract circuits that map from one input feature to an output feature in a highly data-efficient manner.
What are your next steps?
Continue looking into how to best factor DCTs into ensembles of shallow circuits, and how 'surprising' the generalizations these circuits encode are in context of other interp methods (hopefully by relying on some scaling laws/theory/well-reasoned assumptions)
Is there anything others could help you with?
I think the DCT method has broadly not been used very much, and it would be good if people looked more into how they work/generalize (the original authors seem to have made a DCT follow-up in weight-space https://arxiv.org/pdf/2606.29604)
@ajskateboarder (the DCT post notes that the output directions are likely not features on their own, however i believe this won't be necessary in order to factor DCTs)
(is it ok to consider a broader scope as I outlined in the above doc? this was also the initial motivation for doing this work in particular. I think it would be a better use of the grant and ofc I still plan to work on the ideas I described originally)
@adijo ask @LeoGao
@adijo in general, I trust grantees to make good decisions about how to spend the money to do cool things.