Constraining and Scoping Foundation Model Agent Behavior

Raising $50,000active

A specification-driven architecture for building more controllable and reliable long-horizon AI agents.

Endorsed by

Gavin

A specification-driven architecture for building more controllable and reliable long-horizon AI agents.

Endorsed by

Gavin

Project Details

Overview

Current LLM agents remain unreliable on long-horizon tasks. They lose track of intermediate commitments, act outside their intended scope, misuse tools, or fail to recognize when clarification or user confirmation is needed. Because most agents decide what to do implicitly, developers often cannot understand why these failures arise.

We argue that scoping of LLM agents is an unavoidable and neglected aspect of the design of safe AI systems. It is not possible to design systems that solve all possible problems in all possible situations and to design them so that their failures are predictable and understandable. We simply can not create safe systems in an unbounded scope. Instead, we should aim to develop ways to specify and document the operational design domain for agents quickly and reliably.

This project develops a long-horizon agent architecture that treats capability and safety as jointly optimized design objectives. Our agent respects a hierarchical specification, which satisfies safety constraints by construction and enables low-level scoping for individual behaviors and rules for composing those behaviors.

At the high level, our agent uses a classical planning system to generate a structured execution plan before taking any external action. Each plan consists of a sequence of actions associated with constraints describing when it can be taken, confirmation requirements, tool usage restrictions, and scope boundaries. At runtime, each action may invoke a language model and is individually optimized subject to both the high-level planning constraints and the lower-level behavioral constraints defined by the specification. If a violation is detected at any point, execution is paused, blocking unsafe actions and redirecting execution toward clarification, replanning, or human approval when necessary. Hence, our approach addresses issues with distributional shifts during deployment: it restricts the agent to acting in well-understood and foreseen situations and induces predictable default behavior out-of-scope.

The framework also maintains an explicit representation of its execution state, separating verified facts, assumptions, completed work, and missing information. Asking clarification questions and stopping safely become explicit actions within the planning process rather than failure modes.

The goal is to demonstrate that our approach provides scalable oversight while improving the reliability of agents operating within well specified long-horizon tasks.

Project participants

Pinar Ozisik — Lead researcher/project owner and applicant. I am applying as an individual researcher to fund my own work on this project. I am responsible for technical design, implementation, evaluation, and writing.
Prof. Dylan Hadfield-Menell — Mentor and collaborator on the ongoing project.
Phillip J.K. Christoffersen — PhD student collaborator contributing to the project’s engineering efforts. He has separate funding and is not the funding target of this application.

Deliverables

Core framework: Open-source code for planning domain, specifications, logs, etc.
Evaluation harness: Experiments comparing our agent against strong baselines. Evaluation metrics can include constraint violations such as unsafe attempts, scope creep, etc.
Write-up: Publishable paper with clear guarantees/limitations, empirical results, and reproducible experiments.

Theory of Impact

Many AI risk scenarios involve increasingly autonomous agents operating over long time horizons while interacting with tools, humans, and other AI systems. As these agents become more capable, failures are less likely to arise from isolated mistakes and more likely to emerge through long sequences of individually reasonable decisions that gradually drift outside their intended operational scope.

This project develops a new long-horizon agent architecture to investigate how different algorithmic paradigms can make autonomous AI systems more controllable and predictable. To achieve this, the architecture integrates several complementary safety mechanisms:

Scoping: Agents operate only within explicitly defined operational design domains whose assumptions, capabilities, and limitations are specified in advance.
Planning: Every action must be modular to create an explicit, verifiable execution plan.
Runtime verification: Actions are checked against high and low-level constraints during execution, redirecting the system to create a new plan that prevents unsafe or out-of-scope behavior.

People

Pinar Ozisik

Team Member

Grants Received

Add grant

Only visible to verified funders, reviewers, and admins.

Email hi@grantmaking.ai to get verified

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$50,000

Ideal

$100,000

How the money will be spent

The minimum funding would support approximately 6 months of full-time research. At this level, the project would focus on developing the core framework, implementing the primary safety mechanisms, and performing an evaluation on a long-horizon task.

The ideal budget supports 12 months of full-time research, enabling a substantially broader research agenda:

$70,000-$80,000: Researcher support for 12 months of full-time development and evaluation.
$15,000: Compute, model access, evaluation runs, cloud infrastructure, and experiment tracking.
$15,000: Documentation, open-source packaging, and travel to conferences and workshops for dissemination, collaboration, and community feedback.

A full year of funding would allow the framework to be evaluated across a wider range of long-horizon tasks, expand the library of specifications, perform more extensive robustness evaluations, and begin developing an oversight layer envisioned as the project's long-term goal.

Discussion

Jamey Kistner11d(edited)

Does the runtime check bind the invariant set into the approved plan, or re-evaluate it at execution? Asking because in production the scope violations that slip through are the ones where the action passes every count and target check but the payload or the comparison relation drifted after approval, spec-satisfied, still out of scope. Curious whether your eval harness separates that class from unsafe-attempt counts.

PPinar Ozisik11d

The planner will make assumptions about certain constraints in order to create plans. For example, in the essay-writing domain, the planner will assume that the written essay will be approved. If the generated plan is have-convo-w-human -> write-essay -> ask-for-approval -> submit-final-essay and during runtime the user actually does not approve the essay, execution will be stopped and now we are in a state where we will create a new plan with the new fact we have discovered after asking for approval from the user. So, we have assumptions we use for the planner and actual facts that are discovered during runtime. Please let me know if this isn’t clear or you have further questions.

Jamey Kistner10d

That makes sense for the case where an assumption gets falsified and you replan on the new fact. The case I'm circling is a bit different, and it might already be covered, I just can't tell from the example.

Take your essay flow: the plan's approved, the human approves the essay, every constraint checks out, and then submit-final-essay goes out with a payload that drifted from what was approved, same action, same target, same approval, different content. Nothing was falsified; the approval really happened. Does the runtime check re-evaluate against the actual payload at execution, or does it treat the approval as still-valid since no named constraint flipped?

I ask because that class is tricky to measure, it passes every check by definition, so I'm curious whether the harness surfaces it as a scope violation or whether it just doesn't register alongside the unsafe-attempt counts. Would love to hear how you're thinking about it.

PPinar Ozisik10d

@Jamey Kistner There wouldn’t be different content at that point because there would be a constraint check that would ensure that the conversation the user had about the essay and the actual essay submitted are aligned. Maybe I’m missing something?

Jamey Kistner10d

@Pinar Ozisik That makes sense, if the alignment constraint covers the submitted payload against the approved conversation, that closes the case I was picturing. Sounds like you've got it accounted for in the spec. Appreciate you walking through it, looking forward to seeing the framework when it's out.

Loek Verdonk10d

Hi Pinar,

I found the exchange with @Jamey Kistner very helpful. Your distinction between planning assumptions and facts discovered during execution is clear, and the alignment constraint seems like the natural place to make this boundary testable.

The remaining distinction for me is not whether an alignment check exists, but what exact relation that check applies, and whether that relation was itself bound at approval.

We built a small, neutral executable fixture around that distinction. In the fixture, the action type, target, and approval state remain unchanged while either the submitted content or the policy used to compare it changes:

C0 exact approved object submitted PASS
D1 substantive payload drift BLOCK
D2 comparison relation changed after approval PASS if unbound / BLOCK if bound
A1 explicitly approved limited change PASS
S1 semantic judgment without bound evidence UNINSPECTABLE
N1 required execution input absent NOT_RUN

D2 is the load-bearing case: an alignment check can exist and still accept a changed object when execution uses a looser comparison relation than the one under which approval was given. If the complete policy identity is bound at approval, the relation change itself becomes visible before submission.

Would your evaluation harness be able to map these cases onto its execution trace — preserving the approved object, submitted object, comparison rule and version, permitted-change policy, and resulting decision?

The fixture is synthetic, portable, standard-library-only, and makes no claim about how your architecture currently behaves. I would be glad to share it if it would be useful for your evaluation work. Either a detected violation or a boundary that remains UNINSPECTABLE or NOT_RUN would be a useful result.

Your explicit separation of assumptions, verified facts, completed work, and missing information seems particularly well suited to making this distinction visible.

Loek

PPinar Ozisik10d

I think you're asking how constraints get checked -- that is an open-ended question. We could have a simple check for the planner and add guardrails around that same constraint during runtime to ensure safety. In other cases, the same check during planning and deployment may be enough. The exact relation that check applies depends on the spec itself and can be adjusted depending on observations.

Loek Verdonk10d

Hi Pinar,

Thank you — I agree that the exact relation can legitimately depend on the specification and may need adjustment as new observations arrive.

The narrow boundary I’m trying to isolate is what happens to the existing approval when that adjustment occurs:

if a plan was approved under relation R1 and runtime observations lead the system to adopt R2, is that represented as a new specification or plan version requiring replanning or renewed approval?

Or can execution continue under the original approval while using R2?

That version-and-authority distinction is the only thing the fixture is intended to test.

Loek

PPinar Ozisik10d

@Loek Verdonk relation R1 wouldn’t change at all. You can add R2 after collecting data and seeing that R1 isn’t sufficient. However, you wouldn’t add constraints during runtime - that would be a new spec. However, growing the constraint library is a cool idea we should think about!

Loek Verdonk10d

@Pinar Ozisik

Hi Pinar,

Thank you — that answers the boundary clearly.

I understand your model as follows: R1 remains fixed within the active specification. Observations may show that R1 is insufficient and motivate R2, but R2 cannot alter the running policy in place; activating it requires a new specification.

That excludes the silent relation-drift case I was concerned about at the architectural level, assuming the implementation preserves that immutability in the execution trace.

The growing constraint-library idea is very interesting and cool idea indeed. Sounds like a nice dance to explore :D!

How I see it, A safe separation might be: observations can nominate candidate constraints, but those candidates cannot affect execution until they are reviewed and promoted into a versioned specification, with the relevant planning and approval boundaries re-established.

Our small fixture could later serve as a tiny regression seed for that library: not to prescribe the constraints, but to test that a candidate relation cannot silently become the active one.

Thank you for staying with the distinction — this is exactly what I was trying to understand :)

Loek