I'm trying to use lean to quantify AI safety issues by understanding model internals.
I'm trying to use lean to quantify AI safety issues by understanding model internals.
The usage of this benchmarking system is to quantify how bad the models can get if we alter the model internals. Basically, we want to understand what happens if we change the internal weights, maybe stop, maybe freeze them, maybe use them in training. Is there a quantifiable measure of model deviation from its training data which we can use to analyze how bad a model can get given a little bit of retraining, maybe LoRa or DPO type post training.
It does reduce catastrophic risk because outlier models will be obviously highlighted in this research, and we can quantify how badly outlier models can be an existential risk, and we can also see which companies and which startups are producing these outlier models, and we can flag them.
Team Member
It will be mostly building this system from scratch and using AI to completely make this system really refined. Basically, we want a real mathematical verifiable system which showcases how far away from a distribution a model can get given changes in its modeling internals. So, all of the money will be only. Spend on compute, a little bit of it would be my own salary.
No comments yet. Be the first to share your thoughts.