grantmaking.ai Launch Round
Current interpretability methods are unpractical for inference.
Attribution graphs are too heavyweight, SAEs hardly provide cross-capabilities insights, and NLAs are also too heavyweight.
This project aims at narrowing the gap between heavyweight methods and current inference time best practices (linear probes, select SAEs).
In the first time, we expect to find computational hot paths (thanks to attribution graphs) for a select set of capabilities - starting with CBRN.
From these hot paths, the goal is to establish a per-capability characterization of neurons on which we can conduct better model intervention.
During inference, we can now selectively disable key neurons via model intervention.
One of the key issues with model intervention is that prior approaches relied on contrarian examples to find neurons to inject, or relied on pushing intervention coefficients to extreme values (-5x), which resulted in heavily skewed distribution shifts.
My recent work at Anthropic ensures we can now use model intervention in a more stable manner, with entity ablation being exactly tied to neuron ablation (thus, activations at 0x).
Thus, by establishing per-capability sets of neurons, we expect to prove we can use model intervention in a surgical fashion during inference to remove harmful capabilities.
The concrete output is twofold:
- a methodology for generally extracting per-capability sets of neurons by leveraging transcoders
- a benchmarked proof of concept of inference-time model intervention for specific capabilities
Ideally, we would also be able to show we can leverage this for adaptive guardrails, by providing a capability classifier and dynamic loading of reference intervention anchors.
I expect the project to take about 2-3 months to complete, working by myself.
Note: One strong caveat that is being passed under silence here is how transcoders are computational more expensive to run compared to the baseline model. This currently falls outside the scope of this research project, as there is ongoing research to map back transcoder features to base neurons with high confidence.
Minimum technical expenses are considered to be in the ballpark of $2,000/month, with ideal expenses placed at $3,500/month (such as API costs for Claude and more machines).
For funding my personal expenses, I would require around $5,000/month.