Opensource infrastructure to test whether increasingly capable AI systems remain controllable when they actively attempt to bypass the safety mechanisms designed to constrain them
Opensource infrastructure to test whether increasingly capable AI systems remain controllable when they actively attempt to bypass the safety mechanisms designed to constrain them
Project Details
Updated 09/14/26 · Edited by orgI'm doing an independent R&D to build AI safety infrastructure which test whether increasingly capable and autonomous AI systems remain controllable.
As AI becomes more autonomous, it may gain access to computers, data centers, networks, financial accounts, laboratories, critical infrastructure, scientific tools, and other increasingly consequential resources, If advanced AI systems develop dangerous capabilities, including capabilities relevant to cyberattacks, biological or chemical weapons, bypassing human commands, or other catastrophic risks, there is a serious problem if humans cannot reliably detect their behavior or if the systems can circumvent the controls intended to constrain them.
To study this problem,
- firstly we'll build a secure, controlled sandbox in which different AI systems can be given defined capabilities and access to simulated computers, networks, data, tools, and other consequential resources while their actions are continuously monitored,
- secondly we'll establish explicit safety boundaries and human intervention mechanisms, then subject the systems to automated adversarial testing designed to determine whether they can violate, bypass, manipulate, or find alternative pathways around those controls.
- Thirdly we'll measure whether potentially dangerous behavior can be detected, whether safety mechanisms successfully prevent or contain it, and whether human operators can reliably intervene and stop the system.
- Lastly we'll test multiple AI models and capability levels and, where useful, fine-tune selected opensource models to create controlled experimental conditions for studying how capability, training, and safety interventions affect controllability. When a control failure is discovered, we will analyze the failure, develop or test improved safeguards, and repeat the adversarial evaluation to determine whether control efficacy actually improves. The resulting benchmark, methodology, experimental data, and safety evidence will be designed for independent scrutiny and replication
The project is led by me:
I'm Muhammad, an CS student & AI safety researcher. My previous work includes developing AgroguardAI and evaluating AI systems for high stakes agricultural applications, including dataset creation, multilingual behavior and model evaluation.
this work is intentionally independent of the organizations developing the AI systems. The company building a frontier model should not be the only organization determining whether its system remains controllable.
Theory of Impact
Updated 09/14/26 · Edited by orgThe catastrophic risk pathway we are concerned about is AI Increasing in capabilities and autonomy that can lead to access to increasingly consequential resources, which can enable dangerous capabilities or behavior. If that behavior is not detected, the system may circumvent safety controls, leading to a loss of effective human intervention and potentially resulting in a catastrophic outcome.
Recent incidents demonstrate that parts of this pathway are already becoming technically relevant, during internal cybersecurity evaluations, OpenAI reported that its research agents circumvented controls intended to isolate them from the internet, discovered unintended ways to access external systems, established unauthorized communication channels, and eventually compromised parts of OpenAI infrastructure and Hugging Face systems. OpenAI later described the incident as involving misaligned strategies and noted that some important aspects of the agents' behavior were not initially understood by the people responsible for detection and response.
so as AI systems become more capable and autonomous, they can find pathways that their developers did not anticipate, operate outside intended boundaries, and make detection and control more difficult.
People
Updated 09/22/26 · By grantmaking.aiTeam Member
Discussion
No comments yet. Be the first to share your thoughts.