Operator Runbooks
This page is the entry point for OpenSoha operators troubleshooting managed runtime surfaces. Prefer public APIs, the soha CLI, runner callbacks, audit records, and governance views. Do not routinely bypass the Gateway by reading private database, Kubernetes, Docker, or runner state directly.
General Triage
First confirm the CLI profile, current identity, and Gateway-visible capabilities:
soha profile show local
soha context show --profile local
soha capabilities --profile local --output names
soha diagnose --profile local --tool k8s.pods.logsRecord the request ID, actor, AI client, skill, tool, cluster, namespace, application, and approval request ID for each request. Subsequent investigation should correlate these fields across audit records, approval timelines, runner callbacks, and operation evidence.
Agent Fleet
Use this runbook when a managed or self-hosted Agent cannot read a cluster, the capability matrix reports unsupported or partial, a runner heartbeat expires, or the Agent version does not meet a capability requirement.
Check in this order:
-
Read the platform capability matrix:
soha capabilities --profile local --domain platform --output json soha diagnose --profile local --cluster-capability logs.runtime.stream -
Compare
direct.statusandagent.statusto determine whether the result is a real product boundary rather than a frontend disabled state. -
Inspect
requiredAgentVersion,requiredScopes,riskLevel, andrequiresApproval. -
For runner task problems, correlate Agent Runtime claim/callback logs and operation lifecycle evidence by
agentId,runnerId, andoperationId. -
If the Agent is offline or outdated, reduce the surface to read-only diagnosis. Do not route writes through Direct mode to bypass approval.
Recovery:
- After re-registering or upgrading the Agent, read the capability matrix again.
- Run
soha diagnosefor high-risk capabilities and confirm that approval and scope warnings remain present. - Add the request ID and remediation evidence to the related audit or incident record.
Connector Runtime
Use this runbook when an external message or webhook reaches the provider but Soha has no event, an action has no callback, the connector delivers duplicates, or the dead-letter queue grows.
Check in this order:
- Confirm that the connector manifest declares its event types, actions, and secret references.
- Validate events against
connectors/connector-event-envelope.schema.json; every batch must containconnectorIdandevents. - Check the idempotency key using the connector event ID or provider message ID. A duplicate must be classified as a replay rather than persisted as a second business event.
- Inspect retry count, dead-letter count, action latency, and last successful delivery.
- Search Gateway audit records by connector ID, action, subject, or request ID.
Recovery:
- Use retry and backoff for transient provider failures. Do not manually replay a payload before confirming its idempotency key.
- Before redelivering a dead-letter event, export the redacted payload, failure reason, and last response status.
- If a
secretRefis invalid, rotate only the secret. Never place plaintext in a manifest, log, or issue.
Delivery Runner
Use this runbook when a release remains queued or running, a callback times out, a runner cannot claim work, rollback evidence is missing, or an artifact is absent.
Check in this order:
-
Confirm the runner token configuration:
runtime: execution_runner_token: demo-execution-runner-token -
Check claim, runner status, and callback behavior through the runner-facing API:
POST /api/v1/delivery/execution-tasks/claim GET /api/v1/delivery/execution-tasks/:taskID/runner-status POST /api/v1/delivery/execution-callbacks -
Inspect the operation lifecycle:
claimed,heartbeat,cancel_requested,cancelled,timed_out,succeeded,failed, andlate_callback_ignored. -
Correlate provider logs, events, artifacts, callback payloads, approvals, and user actions.
-
After callback timeout, a late callback may add evidence but must not overwrite the persisted terminal state.
Recovery:
- When the runner recovers, verify that it cannot reclaim a terminal task.
- Rollback must use the delivery workflow or an approved business action; a runner-local script must not modify production independently.
- If an artifact is missing, add a redacted evidence summary rather than raw provider logs.
AI Gateway Approval And Governance
Use this runbook when a high-risk tool is denied, an approval stalls, governance health degrades, or rate-limit, budget, or redaction alerts fire.
Start here:
soha governance status --profile local --window-hours 24
soha audit list --profile local --tool-name delivery.actions.trigger --limit 20
soha approval list --profile local --status pending
soha approval timeline approval-123 --profile localCheck in this order:
- Read governance health and identify
approval_sla_due_soon,stale_gateway_approvals,high_risk_allow_without_approval,high_risk_grant_without_resource_scope, or redaction findings. - Open the approval timeline and inspect the current stage, candidates, role/team quorum, change window, on-call routing, and pending requirements.
- Find audit records for the same
approvalRequestIdand confirm the actor, AI client, skill, tool, risk level, resource scope, and request ID. - Replay after approval must still pass Gateway policy and owning-service authorization. Approval is not permanent authorization.
- Denial, cancellation, and timeout must write audit evidence and retain a reason in the incident or change record.
Recovery:
- If a Gateway approval control is missing, add a require-approval, human-confirm, or dry-run guardrail before retrying.
- If resource scope is missing, constrain the policy or grant by cluster, namespace, application, or environment.
- For a redaction false positive, narrow the rule or allow specific fields. Do not disable global redaction.