Oblive Docs

Tasks, Schedules, and Actions

Extend durable execution while preserving ownership, retries, human handoffs, and external-effect safety.

Add or Change Task Behavior

  • Keep task input on the task specification rather than duplicating prompt payload in run requests.
  • Allocate immutable organization-local identifiers transactionally from the owning department.
  • Add dependencies only through the cycle-checking service.
  • Keep dependency conditions explicit: success, completion, or artifact.
  • Start only due, allowed, dependency-ready work with no conflicting active execution.
  • Replace a failed required cross-root edge with one bounded recovery review. Keep same-owner recovery in that department and route cross-owner recovery to Operator.
  • Treat an integration task input as required unless it explicitly sets required: false.

A missing required integration does not run the model and does not consume aggregate run budget. The worker creates one idempotent Human Inbox question and finalizes the attempt as waiting_human; answering the blocker resumes through a fresh request. Optional integrations may be absent without blocking execution.

Run Requests and Attempts

run_requests hold task-only mutable queue/retry intent. runs are immutable attempts. Workers create attempts and increment attempt counts. Scheduler and runbeat never create attempts.

tasks.agentRunCount is total immutable attempt history. tasks.consumedAgentRunCount is the aggregate budget counter used at claim time. Claim increments both in the same transaction; successful human_interrupt and action_interrupt finalization refunds only the consumed counter. Failures remain charged. Do not infer either count from Redis delivery.

Do not accept maxCostMinorUnits until the runtime records actual metered cost. A migration removes legacy values instead of enforcing a synthetic zero-cost counter.

Late writes must include and satisfy the relevant lease, state, and version checks. Preserve failed and timed-out attempts.

Public run reads use a dedicated joined projection containing task/profile identity, attempt, mode, status, outcome, error, usage, and timestamps. Never return the execution prompt, hydrated context, working state, worker/lease fields, or workspace/checkpoint/trajectory/log locations through the owner API. Run-history cursors bind the sort key, direction, value, and run ID so changing sort order cannot reuse a stale cursor.

Admission locks the organization and profile rows before counting starting and running attempts. A full organization or profile returns a capacity conflict before attempt insertion. Runbeat closes a stale run, cancels its staged questions/actions, removes its active task reference, and moves the request to backoff in one PostgreSQL transaction. Exhaustion fails the request and task in that transaction. Failed-run repair only transitions claimed requests once; the scheduler owns later due publication. Background pagination must use the same key as its cursor.

Add a Schedule

  1. Define owner, cadence, run mode, and expected task outcome.
  2. Store nextRunAt as the schedule cursor.
  3. Let the scheduler create normal tasks and run requests.
  4. Publish a wakeup only after durable state changes.
  5. Make duplicate scans and wakeups harmless.
  6. Keep healthy polls quiet.
  7. Mark built-in schedules as system; persist the creating profile and exact run/chat provenance for agent schedules.
  8. Coalesce missed intervals into one catch-up firing, then advance to the next future occurrence.
  9. Back off firing failures exponentially. Disable after five consecutive failures and mark the schedule as requiring attention.
  10. Test pause, resume, due calculation, duplicates, backoff, provenance, and restart recovery.

System schedules remain organization-owned and read-only to agent execution capabilities. Department agents may schedule only their own department, Operator may delegate any department, and Chat may schedule work only from an explicit active user request.

Schedule lists accept status, origin, profile, and department filters. Their opaque cursor binds the selected ordering and explicitly preserves PostgreSQL null ordering for optional next/last run times; never combine an ID-only cursor with a different sort column.

Add a Human Question

Stage a typed blocker during an active run. Open it only when run finalization moves the task to waiting-human. The response removes that exact blocker, advances the task version, and queues fresh execution only when no blockers remain.

Cancel Task Work

Cancellation is transactional across the requested task and every active descendant. Cancel their queued requests, active attempts, staged/open questions, and actions that have not started. Publish one best-effort abort signal per active run after commit; durable run state remains authoritative. Preserve terminal descendants and action outcomes that are already executing or uncertain.

Resolve outgoing dependency edges by their declared condition: cancellation satisfies completion, fails success and artifact, and blocks a dependent only when an unresolved edge is required. A later action reconciliation must never reopen a terminal task.

A failed required dependency between separate roots creates one version-bound review task. The failed edge remains as non-required history and a required completion edge points to the recovery review. Completing, failing, or cancelling that bounded review releases the dependent for a fresh decision; it does not recursively create another recovery. Dependencies inside one failing root do not create orphan recovery work because the root closes together.

Automatic replans stop at the task budget, which is at most three. An amendment at that boundary opens one idempotent human question and refunds that waiting run credit. If the human authorizes a final amendment, the next review may apply it without increasing the bounded replan counter and marks the override used. Any later amendment fails the task deterministically instead of opening another question or leaving run finalization in conflict.

Add a Consequential Action

  1. Define typed arguments and result.
  2. Establish stable idempotency.
  3. Snapshot authorization epoch, task version, risk, and policy.
  4. Wait for approval or acquire a fenced execution lease.
  5. Attempt the provider call once.
  6. Store a redacted receipt for confirmed outcomes.
  7. Mark uncertain outcomes unknown.
  8. Reconcile through a human without replaying the effect.

Serialize action admission for each task before enforcing its hard action budget. Separately count organization action volume by the organization’s local calendar day. Reaching maxDailyActions creates one idempotent, non-blocking notification for that day; it does not reject the action. Serialize the daily count so concurrent actions cannot cross the threshold silently.

Recovery Tests

Cover duplicate wakeups, missed wakeups, stale heartbeats, exhausted attempts, late worker writes, refunded waits, output-sync failure, a 100-file output set, a rejected 101-file output set with zero uploads, missing required versus optional integrations, cross-organization capabilities, cross-department/root mutations, multiple blockers, exhausted replans, the single replan override, stale approvals, and abandoned action leases. Also cover cursor reuse under a different sort, nullable schedule-date pagination, and public run projection redaction.