Competence-Gated AI Oversight

Seattle, WA

Raisingactive1 person

I'm testing how AI verification improves oversight, when correlated errors make it unreliable and when systems should escalate or abstain.

Project Details

I'm developing a replicable research engine and investigating model-based verification within agent systems. My goal is to research efficient small models in order to make useful AI accessible to people around the world, e.g., on second hand phones in developing countries. Before that research, I need to make the safety and governance far more robust. I built a seven model local study and began research into competence boundaries for local model verification capabilities. I found that independent models give the same wrong answers about half the time and began frontier model testing. I continually update my site with my work and the latest autonomous research results, and I'll continue pursuing safety as the risks escalate in AI use. I'll create a frozen protocol, reproducible benchmarks, model-pair scoreboards, and continue/audit/escalation decision rule results.

Theory of Impact

Major risks seem to be undertaken by consumers and AI researchers, and I've been trying to identify the most important ways we can increase legibility and multi agent oversight, especially manipulation resistance in the short term. I think we need to increase safeguarding surrounding AI tools used while the inevitability of recursive development research expands. Most plans for overseeing agentic systems route through agent supervision, but checking models sharing blind-spots with generator models is a major red flag. I'm researching both local and frontier models, and the frontier behavior fails under manipulation in strange ways too. For example, it can flag manipulation when asked and comply with it anyway. These failure modes scale with how autonomous we make agentic systems, and we're making increasingly capable models, increasingly autonomous in agentic systems. I think competence verification is one of the most important things I can continue researching now. In addition to measuring model based oversight capability, I'll study when a model should continue, audit its work or bring it to a human reviewer, and I expect to find compute cost reductions on the way. Model based oversight becoming load bearing before anyone measures where it fails is the most immediate risk I'm working against.

People

Bentley Moon

Team Member

Funding Details

Private

Only visible to verified funders, reviewers, and admins.

Email hi@grantmaking.ai to get verified

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$****

Ideal

$****

How the money will be spent

The minimum would cover a bounded cross vendor pilot on an externally graded hard task family. The fuller funding ask would fund a 12 week study for further research and followup writing, open source findings and writing work. $15k would go to funding the research, $4k for frontier API experimentation, $4k for hosted open model baselines and execution environments and $2k for independent statistical review, artifact preparation and bounded contingency. With the larger budget I would test more task families, same family vs cross family checking and 70b model baselines. My proposed timeline runs August-October and I'll revise the budget if I find support anywhere else. Thank you for your time and consideration.

Discussion

No comments yet. Be the first to share your thoughts.