Agent Runtime
dsh and Hindsight deployment
Core v0.1.13 ships with dsh_runtime and hindsight disabled by default (Core v0.1.12 and earlier used the legacy built-in runtime). Before deploying dsh, Hindsight, or runtime profiles, verify a matching combination of Core (ghcr.io/opensoha/soha:v0.1.13 with its same-commit embedded Web), Contracts v0.2.0, and the dsh runtime image (ghcr.io/opensoha/soha-dsh:v0.2.0, published with agent/v0.2.0). For Kubernetes, use a compatible chart from the Soha Helm repository (check each published chart’s appVersion until chart updates targeting Core v0.1.13 and Agent v0.2.0 are published).
Building the dsh image from source requires Contracts v0.2.0 or a newer compatible release containing agent/dsh-runtime.schema.json (absent from Contracts v0.1.29 and earlier).
- New Chat and Heya require dsh. Deploy the dsh service on a private network, share its service-token file with Core and enable
dsh_runtime.enabled. Enablesecurity.protected_durable_storageonly after all Core workers can read encrypted checkpoints. Keep the installation’s active credential encryption key and protected storage while existing checkpoints need recovery. - Hindsight is optional. Before enabling it, apply Core migrations through
0145. Use its own PostgreSQL database and credentials, the matching protection gateway and a Gateway token scoped to the configured models. - Before using runtime profiles and managed assets, apply Core migrations through
0147and check that dsh reportsruntime-assembly-v1. Mount the registry, Skill files and MCP credentials only in the dsh workload. - Preserve the single-writer dsh journal during upgrades and rollback. If you installed an earlier candidate with an unprotected Helm PVC, first upgrade with dsh enabled and the original claim. Confirm the retention annotation is in the Helm release manifest before disabling dsh. On reinstall, use
dshRuntime.existingClaimto reference the retained claim. - Core
/readyzchecks PostgreSQL. Verify authenticated dsh health and a completed Chat and eligible Heya request with their corresponding Runs before opening traffic. When rotating a shared service token, pause admission, drain affected runs and restart Core replicas and the corresponding runtime together.
Runtime cutover maintenance (operator)
Use the offline runtime-upgrade tool from the same Core revision. Its source build requires the matching released Contracts dependency. Use GOWORK=off go build -o runtime-upgrade ./cmd/runtime-upgrade after verifying that dependency. Run this tool during a maintenance window after draining legacy work and stopping its writers.
Back up the target PostgreSQL database before schema or application changes. The maintenance tool requires schema migrations 0143 and 0144 before dry-run or apply. Apply them using the installation’s migration procedure while the old writers stay stopped.
Use a clean shell without other PG* variables. Prepare an operator-only pg_service.conf with a [soha-upgrade] section specifying the intended host, port, dbname, user and TLS settings; keep credentials in a separate mode 0600 pgpass file. Use these same files and environment for psql, pg_dump and the maintenance tool.
export PGSERVICEFILE=/YOUR_PRIVATE_PATH/pg_service.conf
export PGPASSFILE=/YOUR_PRIVATE_PATH/pgpass
export PGSERVICE=soha-upgrade
export SOHA_UPGRADE_DATABASE_URL="service=$PGSERVICE"
psql -X -v ON_ERROR_STOP=1 -c 'SELECT current_database(), current_user, inet_server_addr(), inet_server_port()'Compare the displayed database, user, server address and port with the intended maintenance target. Stop if they differ.
umask 077
BACKUP_PATH=$(mktemp /YOUR_PRIVATE_PATH/soha-before-dsh.XXXXXX)
pg_dump --format=custom --file "$BACKUP_PATH"
pg_restore --list "$BACKUP_PATH" >/dev/nullAfter applying the prerequisite schema migrations, generate the plan.
./runtime-upgrade -mode=dry-run > /YOUR_PRIVATE_PATH/runtime-upgrade-plan.jsonKeep the service configuration and environment unchanged through backup, dry-run and apply. The tool checks only that the backup is a nonempty regular 0600 file and records its SHA-256; it does not verify its database identity or restorability.
Review the dry-run inventory and its SHA-256 digest before applying. pg_restore --list checks the archive format only; it is not a restore drill. Apply requires the reviewed plan digest and the backup file. If the reviewed inventory contains explicitly selected expired-lease legacy runs, list their exact IDs (up to 1000) using -quarantine-runs; otherwise omit that flag:
./runtime-upgrade -mode=apply -digest REVIEWED_PLAN_SHA256 -backup "$BACKUP_PATH"
./runtime-upgrade -mode=apply -digest REVIEWED_PLAN_SHA256 -backup "$BACKUP_PATH" -quarantine-runs RUN_ID_1,RUN_ID_2The command rechecks that the inventory matches the digest and rejects live leases or other active writers. A quarantined run becomes read-only unknown and fenced for reconciliation; quarantine does not prove it stopped, retry it, or translate a Native checkpoint. If inventory or writers change, stop and produce a fresh dry-run for review.
Rollback is checkpoint-scoped and conditional. It only restores session metadata that has not changed since apply; it does not clear the cutover gate or restore runs, memory, or tombstones:
./runtime-upgrade -mode=rollback -digest APPLIED_CHECKPOINT_SHA256Quarantined memory remains inspectable with its server-owned quarantined and quarantineReason fields, cannot be re-enabled by clients, and may be deleted by its owner. Do not relabel uncertain records as expired or infer a missing owner. This source-candidate procedure is not release or production qualification; dependency, image, and release acceptance remain pending.
With the candidate CLI, for uncertain chat submissions, inspect the original request rather than resending it: soha ai chat request SESSION --request-id ID. native-request is a compatibility alias. A missing lookup is not proof that the request cannot still commit; reconciliation remains server-authoritative.
Positioning
The soha agent process serves two runtime responsibilities:
- Remote cluster agent: provides resource reads and bounded operational actions for Kubernetes clusters configured with
connectionMode=agent. - Control plane runner: claims delivery, Docker, and AI Agent Runtime tasks from the
sohacontrol plane and writes status and results back through callbacks.
AI Agent Runtime is Soha’s unified execution abstraction for the AI workbench, automation policies, and operational diagnostics. Console pages and domain modules target Soha’s capability contracts rather than coupling directly to Hermes, OpenClaw, or any specific agent implementation.
Core abstractions:
AgentProvider: a pluggable executor such asinternalorhermes, extensible to OpenClaw or other providers.AgentCapability: a platform capability such asroot_cause,performance,trace,inspection_review,delivery_failure, ordocker_diagnosis.AgentToolBinding: a binding from a capability to an MCP adapter, a read-only platform tool, or a provider-native tool.AgentSkillBinding: a mapping from a Soha skill to a Hermes skill, a prompt template, or a provider-native skill mechanism.AgentRun: a durable agent execution containing the provider, capability, skills, toolset, scope, status, callback token, tool execution records, and output artifacts.
Hermes Agent is the first external provider. It claims ai_agent_runs records through the Soha agent runner, executes Hermes CLI or Hermes Agent capabilities, and writes the normalized output back as an AnalysisArtifact.
Each AgentRun snapshots toolBindings and skillBindings at creation time. A runner claim receives a self-contained Soha capability contract—including scope, toolset, budget, platform playbook, provider skill mappings, and MCP/internal tool mappings. Adapters for OpenClaw, internal agents, or additional providers consume this snapshot rather than re-deriving Hermes- or OpenClaw-specific parameters inside UI pages or domain modules.
Control Plane Contracts
Authenticated AI workbench catalog endpoints return Agent Runtime directory summaries:
GET /api/v1/copilot/workbench/catalogGET /api/v1/copilot/agent-providersGET /api/v1/copilot/agent-runs
These endpoints serve authorized users and return summaries of providers, capabilities, tool bindings, skill bindings, MCP adapters, data sources, analysis profiles, and the skill registry. Full provider API keys and data source credentials remain managed under Settings > AI.
Runner endpoints for Agent execution authenticate with runtime.execution_runner_token:
POST /api/v1/copilot/agent-runs/claimPOST /api/v1/copilot/agent-runs/callbackPOST /api/v1/copilot/agent-runs/tool-call
tool-call is the governed gateway through which external providers invoke Soha tools. Every request must carry both the runner bearer token and the active AgentRun.callbackToken, and may invoke only the read-only tools snapshotted in toolBindings when that run was created. The control plane currently executes queries for logs, metrics, traces, platform events, delivery releases, delivery builds, execution tasks, platform resource snapshots, Docker operations/services, virtualization operations, alerts, and OnCall route resolution, recording each invocation as a ToolExecution for reuse in the final AnalysisArtifact. Provider adapters must not bypass this gateway to access Soha data sources or credentials directly.
Before running the provider command, the runner’s Hermes/CLI provider POC prefetches up to three eligible read-only tool results—such as events.query, logs.query, metrics.query, traces.query, delivery.releases.list, delivery.builds.list, delivery.execution_tasks.list, platform.resources.snapshot, docker.operations.list, docker.services.list, virtualization.operations.list, alerts.list, and oncall.routes.resolve—and injects them as prefetchedToolResults into both the provider prompt and final output. The traces.query route is retained for compatibility and currently reports capability unavailable; generic logs.query queries persisted cluster logs only. This allows a provider to consume Soha’s governed tool context even before integrating a native tool-call protocol. When Hermes or another agent connects as an MCP client, the runner adapter wraps this same tool-call gateway as an MCP/tool server without exposing Soha data source credentials.
The AgentRun.toolBindings snapshot is filtered at creation time against the runtime permission keys of the initiating user or system principal. Neither the runner token nor the callback token can expand access beyond that filtered snapshot to data sources, tools, or business contexts the caller was not authorized to read. Toolset adapter selection supports both exact adapter IDs (such as logs.v1) and source-kind aliases (such as logs, metrics, and traces), with explicit matching rules backed by test coverage.
All such tools remain read-only and bounded by the AgentRun.toolBindings snapshot. Additional providers or tools should integrate through a reader/adapter into executeAgentToolBindingOutput before joining the runner prefetch allowlist; high-risk write operations must never be exposed directly as chat tools.
ai_agent_runs is the durable queue table for AI Agent Runtime, with states:
queuedrunningcompletedfailedcanceledcallback_timeout
Platform Capability Matrix
Platform runtime capabilities are exposed as a stable contract at GET /api/v1/clusters/capabilities, describing each capability’s availability across Direct and Agent cluster connection modes. Each entry includes:
key,label,category: stable capability identifier, display name, and category.direct.status,agent.status:available,partial, orunsupported.direct.reason,agent.reason,notes: concrete product constraints explaining partial or unsupported states.requiredScopes,riskLevel,requiresApproval: required scopes, risk level, and approval expectations to surface or diagnose before invocation.docsUrl: link to the corresponding operational documentation or limitation notes.
The Web console, CLI, Skills, and external integrations must not infer Agent/Direct parity on their own; they consume this matrix to drive action availability, unsupported states, degradation messaging, and diagnostic output. unsupported and partial are authoritative capability boundaries reported by the control plane rather than frontend fallback states.
The CLI reads the same platform capability matrix directly:
soha capabilities --domain platform --output names
soha capabilities --domain platform --output json
soha diagnose --cluster-capability resource.yaml.applydiagnose --cluster-capability displays Direct/Agent support status, risk level, approval requirements, required scopes, and documentation links for the target capability, explaining why YAML apply, CRD, Helm, logs, exec, port-forward, or Docker runtime actions are available, partially available, or unavailable in a given connection mode.
Analysis Flows
Explicit AI workbench analysis flow:
- A user selects a provider, analysis profile, toolset, and analysis mode in an
/ai-workbenchsession. - The frontend calls
POST /api/v1/copilot/sessions/:sessionID/analyze. - The
internalprovider runs synchronously through Soha’s built-in analysis path. - External providers enqueue an
AgentRun, writing an initialqueuedassistant message. - An agent runner claims the task via the claim endpoint.
- The Hermes provider runner translates scope, toolset, skills, budget, and prompt into a Hermes invocation.
- Whenever the provider needs logs, metrics, traces, events, delivery releases/builds, or alerts, the runner adapter invokes Soha’s governed tools through
agent-runs/tool-call. - The runner reports
running,completed, orfailedvia callback. - The control plane normalizes the provider output into a Soha
AnalysisArtifact, persisting evidence, hypotheses, recommendations, relationship graphs, tool executions, and data source snapshots.
Automation policy flow:
- Soha automation policies remain the primary scheduler for continuous analysis.
- Incoming
alert_webhookalerts are matched, deduplicated, and rate-limited by cooldown windows in the automation policy. - Policies with
agentProviderId=internaluse the built-in analyzer. - Policies with
agentProviderId=hermesor another external provider create anAgentRunfor asynchronous runner execution. - Hermes cron is treated strictly as an experimental provider-native capability rather than Soha’s primary platform scheduler.
Hermes Runner Configuration
Configure the runner token on the server:
runtime:
execution_runner_token: demo-execution-runner-tokenEnable AI Agent Runtime on the agent via configs/agent.config.yaml:
control_plane:
enabled: true
base_url: http://127.0.0.1:8080
bearer_token: demo-execution-runner-token
agent_id: local-agent
poll_interval: 5s
agent_runtime:
enabled: true
worker_id: local-hermes-runner
provider_ids:
- hermes
provider_kinds:
- hermes
hermes_command: hermes
providers:
hermes:
command: hermes
args:
- chat
- -Q
prompt_arg: -q
skill_arg: ""
provider_skill_arg: -s
workspace_root: ./.soha/agent-runtime
poll_interval: 5sStart the agent:
cd ../soha/agent
go run ./cmd/agentStart with a custom configuration file:
SOHA_AGENT_CONFIG_FILE=/abs/path/to/agent.config.yaml go run ./cmd/agentCurrent Hermes provider runner POC conventions:
- The runner claims only
AgentRuntasks whose provider ID or kind matcheshermes. - The default executable is
hermes, overridable viacontrol_plane.agent_runtime.hermes_command. - The runner serializes the Soha run input, tool binding snapshot, and skill binding snapshot into the prompt, passing
AgentSkillBinding.providerSkillRefas the Hermes skill CLI argument first and falling back to the Soha skill ID only when no provider skill ref is set. - The runner provides an
agent-runs/tool-callclient helper so provider adapters can execute authorized read-only tools from the run snapshot; the default Hermes CLI POC prefetches a bounded batch of read-only tool results (logs, metrics, traces, events, delivery releases/builds, and alert context) into the prompt without assuming a proprietary Hermes tool-call protocol. - While the provider command runs, the runner sends periodic
runningheartbeat callbacks to keeplastHeartbeatAtfresh on the control plane; if a callback response returns a terminal state (canceled,callback_timeout,failed, orcompleted), the runner cancels the local provider process and suppresses any subsequentcompletedcallback. soha/agent/internal/agent/runnerincludes smoke test coverage without requiring a live Hermes installation: a fake control plane exercises the full protocol path from claim -> running callback -> tool-call prefetch -> CLI provider -> completed callback ->AnalysisArtifact. Production deployments should still pointhermes_commandat the real Hermes CLI for end-to-end verification.analysisArtifactssupplied in the runner callback take precedence as the final artifact; when omitted, the runner and control plane synthesize a baselineAnalysisArtifactfrom the provider output summary while preserving provider-returnedevidence,hypotheses,recommendations,graph,toolExecutions,dataSourceSnapshot, and the runner’s own tool execution records.
The runner dispatches executors by provider. Hermes is the built-in default executor; additional CLI-based providers can be wired through control_plane.agent_runtime.providers.<providerKind>, for example:
control_plane:
agent_runtime:
provider_ids:
- openclaw
provider_kinds:
- openclaw
providers:
openclaw:
command: openclaw
args:
- run
prompt_arg: --prompt
skill_arg: ""
provider_skill_arg: --skillSuch a provider executor still consumes only the Soha AgentRun contract: the prompt carries the scope, toolset, toolBindings, skillBindings, and output schema, while provider-native skill arguments come from AgentSkillBinding.providerSkillRef. If no matching provider executor is configured, the runner explicitly reports failed via callback rather than masquerading an unknown provider as Hermes.
Extending With a New Provider
Adding a new provider extends only the adapter layer without rewriting the AI workbench or domain analysis workflows:
- Register the
AgentProviderin the copilot service provider catalog. - Declare the
AgentCapabilityentries supported by the provider. - Add
AgentToolBindingandAgentSkillBindingdefinitions for those capabilities. - Implement a provider-specific executor in the runner that translates
AgentRuninputs into provider invocations. - Keep callback outputs formatted as a Soha
AnalysisArtifactor a structure synthesizable into anAnalysisArtifact.
Hard boundaries:
- UI pages, HTTP handlers, and domain modules must never call Hermes directly.
- Budgets, permissions, menus, auditing, data source redaction, and operational boundaries remain governed by Soha.
- Agents operate strictly as pluggable executors and must not bypass Soha’s MCP adapters, scopes, toolsets,
tool-callgateway, or analysis profile contracts. - High-risk write operations must never be attached directly to chat execution; they must flow through durable operations or approval workflows in their owning domain modules.
Remote Cluster Agent And AI Agent Runtime Boundaries
The same soha-agent binary serves both as a remote cluster connector (agent mode) and as a control plane Runner executor, while strictly separating their protocols and authorization boundaries:
- Remote Kubernetes cluster Agent: establishes an outbound persistent reverse connection from a private cluster to Soha (see Multi-Cluster Model and Kubernetes Workbench). Governed by installation presets (
read-only,workbench,legacy), an explicitsecurity.allowed_actionsallowlist, and in-cluster RBAC, it provides resource reads and writes, bounded logs, YAML inspection and apply, events, workload operations, node drain, Pod exec, port forwarding, and optionally enabled Helm and CRD actions. Action readiness can be verified through the authenticatedGET /api/v1/diagnostics/actionsendpoint, while effective availability is defined by the platform capability matrix. - Docker host runtime agent: when the corresponding actions (such as
docker.runtime.logs.v1,docker.runtime.files.v1, anddocker.runtime.terminal.v1) andcontrol_plane.docker.operation_kindsare explicitly configured, provides governed log streaming, safe volume browsing, container terminal sessions, and Runner operation callbacks for registered hosts and projects. - AI Agent Runtime executor: executes governed
AgentRuntasks over the Runner polling and callback protocol. It cannot escalate privileges to invoke unauthorized cluster actions, nor mix the platform cluster Agent channel with the AI Provider execution pipeline.