The latest comments and grant applications on grantmaking.ai.
The latest comments and grant applications on grantmaking.ai.
Hi Gavin - here's a description with more details from the draft paper. I don't yet have a worked example but I could follow up later this week with one if that would still be helpful beyond the description (we will plan to add an example to the paper as an appendix):
Audit Meeting: The audit meeting is similar in nature to an oral examination in a thesis defense or to a design review.…
Hi Valencia,
I really appreciate the focus on lowering the technical barrier for independent researchers while keeping evaluation work systematic and reproducible.
What is the smallest bounded evaluation object that an outside project could bring to EvalLabs today — for example one claim, one system behavior, or one small dataset — and what exact artifacts would EvalLabs return?
I am…
+4 earlier replies
That sounds like a great approach, and I agree that keeping the first exchange intentionally small makes sense.
I also like your suggestion that we each document our observations independently before discussing them. I think that will make for a much cleaner comparison.
A ZIP package works well. I'll create a private shared…
@Valencia Cooper
Hi Valencia,
Thats perfect, thank you :D
Please use: loekverdonk@live.nl
Once the private folder is shared, I’ll upload the fixed evidence package there. It will contain the openly licensed source fragment, license and hashes, the exact ASR run and raw output, visible uncertainty,…
Perfect, thank you! I've just shared the private folder with loekverdonk@live.nl.
Take your time putting the evidence package together. The structure you described sounds ideal for this first comparison, and I like the idea of us documenting our observations independently before sharing…
@Valencia Cooper
Hi Valencia,
The first fixed evidence package is ready and has been uploaded to the private shared folder.
One source-boundary check before you begin: I originally said I would use an openly licensed audio fragment. The source I selected is a LibriVox recording that LibriVox treats as public domain in the United…
Hi Ronald! From just the description, I can't tell what these audits will be like. Do you have a worked example?
When talking to Genevieve, you can clearly see that she is very knowledgeable and passionate about this subject. In the past, we saw a lot of claims about targeted political content and its impact on polarisation and elections being made ahead of sufficient empirical support, so a rigorous evaluation is much needed in this area. Also, WEF's report identified misinformation and disinformation as…
I met Abeer at EA London, and he made a great impression. What he has accomplished in such a short time, despite seemingly limited resources, is amazing. AI Safety Hong Kong clearly addresses an important gap and does so in a highly cost-effective way.
Assessing current and future models on 'hardware hacking' - reverse engineering, focused on computer peripherals and embedded devices
As a Navigator at Lens Academy, I've seen firsthand the impact of its approach to AI safety education. The team is building something thoughtful, scalable, and genuinely valuable, and I'm happy to support their work.
This is a bittersweet update. The good news: Luthien Proxy is now suitable for use as an API-level proxy for Claude Code; it is launched, free and open-source: https://luthien.cc/
With that said, I've decided to step away from the project; going forward Luthien Proxy will be stewarded by Luthien PBC under Luthien co-founder Scott Wofford:…
@jaidhyani I read the Luthien Proxy post-mortem carefully. I don't have the technical background to evaluate the code or the proxy architecture, but one thing stood out to me: the failure wasn't technical it was that the world changed faster than an external safety layer could keep up.
I see a similar pattern in many places: when safety is only an external layer a fence built after the…
I've followed Pedro's work for a while and want to endorse this project.
A novel idea. Interesting in combining fields and will be impacteful.
Is there a reasonable likelihood that this research would also provide negative or dangerous-to-publish outcomes?
AI research designed to reduce X-risk can certainly make things worse, so this is indeed something that one should take very seriously when evaluating a proposed AI research project. But I actually don't think that this is a problem for Alignment Target Analysis.
Some AI safety research can lead to capabilities progress. In addition to that dynamic, it is also true that making it easier to…
Happy to answer any questions or share more info. This project is about AI governance and AI safety, although it isn't reflected in the #codes for some reason.
Happy to answer any questions or share more info 🤓
Build v1 of an open benchmark, dataset, and tool to measure LLM instability in evaluating a person from dialogue under different question framings, across models, temps, and human baselines.
What would be the expected benefits compared to the study already planned by adding additional outpatient and inpatients? Just more data?
Yup, that is right. In particular, we are looking for safety issues that are low prevalence but high impact, as they may only occur in situations where an individual is experiencing heightened symptoms or vulnerability. An inpatient dataset would include people who are having the most serious symptoms and may represent the highest prevalence population (all the currently published clinical case…
Your mechanical enforcement at the tool-call boundary appears to protect the final execution edge. We are testing the preceding continuity problem: whether the authorization arriving at that boundary still means what the human originally approved. Would you be open to a fixture where the tool call is syntactically permitted but its inherited scope has silently expanded upstream?
+6 earlier replies
@Loek Verdonk
Received, unzipped, and read, not just the report, the actual code. I ran your observation and repair scripts against my source and then went and confirmed the two seams in my own harness line by line, because I hold my collaborators to the same standard I hold myself: verify against the source, don't trust the summary. Both of…
@Jamey Kistner
Yeah, same to you!!!
This dance has been valuable in many ways. It is beautiful to see that things can become clearer without either of us having to lose our own dance — simply by dancing with each other :D :D :D
The part that mattered most to me is that you verified the findings against your own source, line by line,…
@Loek Verdonk
Agreed, and it's the right correction to make. Bilateral reproduction with me as source-author is the honest claim, not independent validation, and the production fix is still NOT_RUN. That you drew that line yourself, on your own side, is the whole exchange in miniature.
The two seams are real and they're going straight…
@Jamey Kistner
Well, this invitation was impossible to let pass :D We stepped onto the dance floor and took the full turn :D
We opened the resolver-drift descent, and it returned something more useful than a clean PASS.
The first result held: after resolving a mutable alias, the gate must dispatch the exact explicit effect set that was…
What is the ROI curve like on the gap between $10k and $1m?
I would spend $10k entirely on compute in order to perform experiments in the cloud. I have already run these experiments locally, and the code can be seen here: https://github.com/thenthfool/freetimebench
This funding would hopefully enable me to generate results that are convincing enough to derisk further investment into this project.
If I were to steward a significant amount of…
I buy the same ToC and impact arguments as for other Pause AI chapters.
Thanks!
I just want to add that ergonomic access to big data genuinely has gravity. People just have to extrapolate and understand that IT IS meaningful to have hundreds of terabytes of the most thoughtful writing and work and metadata about it, in a database that agents have low-friction access to. And that people just have to understand that they will experience the benefits of scry and scry links to…
An AI-powered real-time football intelligence platform that transforms live video into actionable tactical and performance insights for coaches using only a single camera.
My app asks AI, what portion of the global GDP a person is worth, and gives accordingly.
I have funded Luc to work on this a few months back, and so far was impressed by what was built and how responsive he was
If we fail to ensure Human Liveness ("humans can do good things"), the role of human intelligence will be crowded out 4000-to-1 by artificial intelligence over the next 20 years.
Even if we ensure AI Safety ("AI won't do bad things"), without Human Liveness, Earth born intelligence will drift toward a babbling equilibrium of inscrutable AI hallucination.
The Human Liveness Project is of…
Does the runtime check bind the invariant set into the approved plan, or re-evaluate it at execution? Asking because in production the scope violations that slip through are the ones where the action passes every count and target check but the payload or the comparison relation drifted after approval, spec-satisfied, still out of scope. Curious whether your eval harness separates that class from…
This seems like an interesting, well-bounded and useful study.
This is a sharp problem framing, and the shutdown-timer example is the crux: the agent isn't adversarial, it's doing what helpfulness rewards, and the oversight mechanism just happens to be in the blast radius. The "third path" point, that sabotage of oversight can be a byproduct of agenticity rather than misaligned goals or adversarial pressure, feels right and under-explored.
Worth pulling…
Getting Australians to events is difficult and expensive, but Australia is well positioned overall to be both receptive to Pause AI arguments and act as a middle power in AI dynamics. so this seems overall worthwhile in terms of expected downstream impact. Given I generally buy Pause AI's ToC and have been impressed with the Australian chapter, easy endorse.
This seems like solid foundational work that has the potential to add a lot of downstream value, although the upsides seem like they will be harder to measure initially outside of things like site traffic.
Testing whether activation probes recover safety signals from reasoning models as their chain-of-thought becomes illegible.
A safe evaluation of whether language models can separate facts, reasonable interpretations, missing information, and their own inventions when answering questions about films and television series
