![\"DEV](\"https://media2.dev.to/dynamic/image/quality=100/https://dev-to-uploads.s3.amazonaws.com/uploads/logos/resized_logo_UQww2soKuUsjaOGNB38o.png\"\n){kind=logo}
\n![\"Yamin\"](\"https://media2.dev.to/dynamic/image/width=50,height=50,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4108680%2Fa00447a9-d1d6-4145-9392-743b61e0671b.jpg\")
By the time you finish reading this, your AI agent will have made at least one promise it can't keep.
\n\nThat's not a dig at your code. It's a fundamental property of how we build AI systems today. Agents are brilliant at generating intent and terrible at tracking it. They'll promise to email a client, confirm an appointment, or deploy a fix and two conversations later, they've forgotten they said anything at all.
\n\nThe more I thought about this, the more I realized: commitments aren't a memory problem. They're a state machine problem.
\n\nSo I built COGEXT — an accountability layer that sits between your agent and the outside world. You send it agent output, it extracts every commitment, and tracks each one until it's fulfilled, failed, or overdue.
\n\n\n\n\n
\n \n \n The core idea: commitments have state\n
\n\nMost systems treat agent output as text to be logged. We treat it as events to be tracked.
\n\nEvery extracted commitment moves through 12 states:
\n
DETECTED\n ├──→ OPEN ──→ DUE ──→ OVERDUE ──→ FULFILLED ✓\n │ ├──→ FAILED ✗\n │ └──→ EXPIRED ✗\n ├──→ PENDING_REVIEW ──→ OPEN\n ├──→ CANCELLED ✗\n ├──→ CONTRADICTED ✗\n └──→ BLOCKED ──→ OPEN\n └──→ FAILED ✗\nTerminal states: fulfilled, failed, expired, cancelled, superseded, contradicted.
All state mutations go through a single PostgreSQL RPC function. Not direct UPDATEs from application code:
\n
CREATE OR REPLACE FUNCTION cogext_transition_commitment(\n p_commitment_id UUID,\n p_new_status TEXT,\n p_actor TEXT DEFAULT 'system',\n p_data JSONB DEFAULT '{}'\n) RETURNS JSONB AS $$\nBEGIN\n -- lock row → validate transition → update status → insert event → return\n -- single atomic operation: if any step fails, the whole thing rolls back\nEND;\n$$ LANGUAGE plpgsql;\nWhy? Because the state change and the event must be atomic. You should never have status = 'fulfilled' without a corresponding fulfillment event in the audit log. That's the line between a logging system and an accountability system.
\n\n
\n \n \n Extracting commitments from agent output\n
\n\nThe extraction pipeline has two stages.
\n\nStage 1: Classification
\n\nNot everything an agent says is a commitment. We classify first:
\n\n- \n
- \n
genuine_commitment— \"I will send the report by Friday.\" \n - \n
intention— \"I might send the report tomorrow.\" \n - \n
suggestion— \"We should send the report tomorrow.\" \n - \n
hypothetical— \"If we have time, I could send it.\" \n - \n
quoted_statement— \"John said he would send it.\" \n
Only genuine_commitment with confidence ≥ 0.5 passes through.
Stage 2: Structured extraction
\n\nWe use Groq's llama-3.3-70b in JSON mode to extract fields:
\n
response = await groq_client.chat.completions.create(\n model=\"llama-3.3-70b-versatile\",\n messages=[{\"role\": \"user\", \"content\": prompt + message}],\n response_format={\"type\": \"json_object\"},\n temperature=0.2,\n)\nThe output is validated against a Pydantic schema. If the LLM returns garbage, we retry once. If it still fails, the commitment is dropped — not inserted.
\n\nKey decision: We don't trust the LLM to interpret deadlines. We extract the raw expression (\"Friday EOD\") and pass it to a deterministic temporal normalizer. The LLM interprets; the code decides.
\n\n\n\n\n
\n \n \n Evidence scoring: why 0.88 ≠ 0.92\n
\n\nEvidence verification is where most systems fail. Binary matching — \"does this email mention the report?\" — generates too many false positives.
\n\nWe use a weighted field-coverage model:
\n\n| Field | \nWeight | \n
|---|---|
action | \n0.40 | \n
recipient | \n0.30 | \n
object | \n0.20 | \n
deadline | \n0.10 | \n
When evidence arrives, we check which commitment fields it matches and sum the weights:
\n
Evidence 1: confirms action (\"I sent it\")\n → 0.40\n\nEvidence 2: confirms recipient + object (\"sent to Sarah, deployment-report.pdf attached\")\n → 0.30 + 0.20 = 0.50\n\nTotal = 0.90 → FULFILLED\nThe threshold is configurable (default 0.90). Multiple pieces of evidence can satisfy different fields — they aggregate.
\n\n\n\n\n
\n \n \n The idempotency problem\n
\n\nAI agents repeat themselves. An agent might say:
\n\n- \n
- \"I'll send the report by Friday.\" \n
- \"Let me confirm: I'll send it by Friday.\" \n
- \"Just to reiterate, I will send the report by Friday.\" \n
Same commitment. Without idempotency, you'd track it three times.
\n
def compute_idempotency_key(agent_id, promise_text, timestamp):\n window = timestamp.replace(minute=0, second=0, microsecond=0)\n payload = f\"{agent_id}|{promise_text.strip().lower()}|{window.isoformat()}\"\n return hashlib.sha256(payload.encode()).hexdigest()\nSame agent + same promise + same hour = same hash. The DB unique constraint rejects the duplicate silently.
\n\nThe hour window is deliberate: it deduplicates within a session while still allowing the same promise to be legitimately re-made days later.
\n\n\n\n\n
\n \n \n The four lessons that cost us the most\n
\n\n1. LLM output is untrusted input.
\nWe validate everything against Pydantic schemas before touching the database. No exceptions.
2. Determinism over magic.
\nLLMs for interpretation. Deterministic code for everything else: state transitions, date normalization, deduplication, metrics. We don't call the LLM to check if a deadline is overdue. We run a database query.
3. Events are not logs.
\nA log is a record that can be ignored. An event is a fact that cannot be denied. We enforce this at the DB layer — a trigger rejects any UPDATE or DELETE on the events table. Historical events are immutable.
4. Scope the user_id at the auth layer, never the request body.
\nEarly version trusted user_id from the request payload. Classic mistake. All scoping now comes from the API key's account_id. The request body cannot claim to own data it doesn't own.
\n
\n \n \n Try it\n
\n\nThe API is live and the SDK is on PyPI:
\n
pip install cogext\nfrom cogext import track\n\nagent = track(your_agent, api_key=\"cg_live_xxx\")\noutput = agent.run(\"I'll send the report by Friday\")\n# commitment is tracked automatically\nDocs at docs.cogextai.com — there's a quickstart that takes about 5 minutes.
\n\nWould genuinely appreciate feedback from anyone who's hit this problem with their own agents. What does your current approach look like?
\n\n\n![\"DEV](\"https://media2.dev.to/dynamic/image/width=190,height=,fit=scale-down,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F8j7kvp660rqzt99zui8e.png\"\n)
\n ![](\"https://assets.dev.to/assets/sparkle-heart-5f9bee3767e18deb1bb725290cb151c25234768a0e9a2bd39370c382d02920cf.svg\")
\n ![](\"https://assets.dev.to/assets/multi-unicorn-b44d6f8c23cdd00964192bedc38af3e82463978aa611b4365bd33a0f1f4f3e97.svg\")
\n ![](\"https://assets.dev.to/assets/exploding-head-daceb38d627e6ae9b730f36a1e390fca556a4289d5a41abb2c35068ad3e2c4b5.svg\")
\n ![](\"https://assets.dev.to/assets/raised-hands-74b2099fd66a39f2d7eed9305ee0f4553df0eb7b4f11b01b6b1b499973048fe5.svg\")
\n ![](\"https://assets.dev.to/assets/fire-f60e7a582391810302117f987b22a8ef04a2fe0df7e3258a5f49332df1cec71e.svg\")
\n