AI Copilot
Goal
The AI layer in soha has evolved from a single chat page into an AIOps assistant workbench built for platform operations.
Its design centers on two layers:
- A unified entry point at
/ai-workbench - A set of workbench subpages covering investigation, inspection automation, and tool assembly
The AI layer supports:
- Alert-driven root-cause analysis
- Performance jitter and capacity anomaly analysis
- Trace latency bottleneck and error hotspot analysis (planned; external trace-provider integration is not currently available)
- Multi-source evidence correlation across logs, events, audit trails, releases, and builds
- Closed-loop handoff from inspection findings into investigation sessions
- Session-scoped assembly of tools, skills, and data sources
Current Implemented Surface
Implemented frontend and backend routes:
- Overview entry:
/ai-workbench
- Investigation workbench:
/ai-workbench/chat/ai-workbench/root-cause/ai-workbench/performance
- Inspection and automation:
/ai-workbench/inspection
- Tools and skills:
/ai-workbench/tool-settings/ai-workbench/model-settings
Legacy routes retained as compatibility redirects:
/ai-observe/ai-observe/workbench/ai-observe/operations/ai-observe/tools/ai-observe/root-cause/ai-observe/performance/ai-observe/chat/ai-observe/inspection/chat/ai-workbench/investigation/ai-workbench/automation/ai-workbench/tools
Session Model
AI investigations treat sessions as first-class durable objects rather than ephemeral chat transcripts.
Core persistence tables remain:
ai_sessionsai_messages
While ai_sessions.metadata now carries structured workbench metadata:
modestatusscopepinnedContexttoolsetagentProviderIdanalysisRunRefssummarytagsarchivedAt
Within metadata, scope serves as the standard handoff payload from the monitoring workbench into the AI workbench, carrying:
alertIdclusterIdnamespaceworkloadtimeRangeMinutes
And toolset is enforced in the backend execution path:
enabledAdapterIdsrestricts which MCP adapters are available in the sessiondisabledToolNamesblocks specific tool names or qualifiedadapter.toolidentifiersbudgetOverrides.maxEvidenceItemscaps the number of evidence items collected per analysisbudgetOverrides.timeoutSecondsbounds the execution timeout for individual analysis tool runsscopeOverridesmerges over the session scope before root-cause, performance, or trace analysis runs
Session tool assembly reads /api/v1/copilot/workbench/catalog, which returns a safe catalog summary to users holding observe.ai.chat / observe.ai.view / settings.ai.view: adapter metadata, data-source enablement status, analysis profile summaries, skill summaries, agent providers, capabilities, tool bindings, and skill bindings. Full AI Provider configurations and secrets remain restricted to settings.ai.view.
Supported session modes:
generalroot_causeperformancetraceinspection_review
The trace session mode is retained in the session model, but Core currently has no built-in Trace
Explore, trace-provider execution, or service-topology view. Related routes and query DTOs report the
capability as unavailable until a future external provider integration is delivered.
API Surface
Implemented and extended session endpoints:
GET /api/v1/copilot/sessionsGET /api/v1/copilot/sessions/:sessionIDPOST /api/v1/copilot/sessionsPATCH /api/v1/copilot/sessions/:sessionIDDELETE /api/v1/copilot/sessions/:sessionIDGET /api/v1/copilot/sessions/:sessionID/messagesPOST /api/v1/copilot/sessions/:sessionID/messages
Sending a message returns an envelope rather than a bare message list:
messagestoolCallsanalysisArtifactssessionPatch
Analysis and agent-execution endpoints built around the unified artifact model:
GET /api/v1/copilot/root-cause/runsPOST /api/v1/copilot/root-cause/runsGET /api/v1/copilot/root-cause/runs/:runIDPOST /api/v1/copilot/sessions/:sessionID/analyzeGET /api/v1/copilot/agent-providersGET /api/v1/copilot/agent-runsPOST /api/v1/copilot/agent-runs/claimPOST /api/v1/copilot/agent-runs/callback
ai_root_cause_runs currently stores:
kindsession_idtool_executions- Existing root-cause evidence, hypotheses, recommendations, and data-source snapshot fields
Data Sources And MCP
AIOps tool capabilities continue to be exposed through the MCP adapter abstraction.
Registered adapters:
platform-native.v1logs.v1metrics.v1traces.v1delivery.v1
Current status:
platform-native.v1- Reads aggregated platform evidence.
logs.v1- The generic
logs.querycapability currently queries durable cluster logs only.
- The generic
metrics.v1- Uses the Prometheus-backed execution path.
traces.v1- The adapter identifier is retained for compatibility, but trace execution is unavailable in Core.
The control plane has three entry points:
- Settings > AI
- Global provider, data source, analysis profile, and automation policy configuration.
/ai-workbench/tools- Legacy compatibility route that redirects to
/ai-workbench/tool-settings.
- Legacy compatibility route that redirects to
/ai-workbench/tool-settings- Entry point for session-scoped toolset overrides and skill assembly.
The global skill registry uses structured enterprise skill definitions rather than display-only entries:
idnamecategoryownerModulecapabilityRefsblueprintRefsinputSchemaoutputSchemascopeRulesenabled
Agent Runtime
Core v0.1.12 and earlier used the legacy built-in runtime. Starting in Core v0.1.13, dsh_runtime is disabled by default; once enabled alongside the matching soha-dsh:v0.2.0 service, Core sends ordinary Chat and Heya requests directly to the separate Node dsh service. Requests fail explicitly when dsh is disabled or unavailable rather than falling back to Native. The internal and native provider identifiers are compatibility aliases for dsh on new requests.
Core objects:
AgentProvider: executor catalog. With dsh enabled,dshis the built-in default for Chat and Heya. Explicit external Hermes/CLI providers remain available for their configured protocols.AgentCapability: soha platform capabilities, includingroot_cause,performance,trace,inspection_review,delivery_failure,post_deploy_observation,platform_resource_diagnosis,docker_diagnosis,virtualization_diagnosis, andoncall_briefAgentToolBinding: maps a capability to MCP adapters, read-only platform tools, or native provider toolsAgentSkillBinding: maps a soha skill to a Hermes skill, prompt template, or future provider skillAgentRun: durable asynchronous execution record for external providers, storing status, scope, toolset, skills, callback token, tool executions, and output artifacts
Core keeps session and Run facts, authorization, Heya eligibility and environment gates, approvals, budgets, audit, and domain analysis responsibilities. The dsh runtime owns the model/tool loop and calls Core through its authenticated runtime interface and authorized tool bridge. Explicit Hermes or other CLI providers remain separate external integrations; they are not implicit defaults or failure fallbacks for Chat/Heya.
Historical Native runs remain readable as history. In-flight legacy runs must be drained or explicitly quarantined through the offline maintenance procedure before cutover; do not translate Native checkpoints or approvals into a live dsh run. See Agent Runtime operations.
Agent Runtime outputs normalize back into soha AnalysisArtifact records, reusing evidence, hypotheses, recommendations, relationship graphs, tool invocation logs, and data-source snapshots. Permissions, menus, budgets, data-source redaction, auditing, and high-risk operation boundaries remain governed by soha.
Frontend Shape
/ai-workbench
The overview entry provides:
- Assistant welcome view
- Recent investigations
- Recent analyses
- Risk radar
- Quick links to investigation, inspection automation, and tool/skill settings
/ai-workbench/chat
The investigation workbench combines Ant Design X and antd:
- Left pane:
Conversations - Center pane:
Bubble.List+Sender+Prompts - Right pane: context / evidence / hypotheses / recommendations panel
ThoughtChaindrawer displaying the tool invocation chain and analysis steps- When the current session scope carries
alertId, the workbench supports jumping back to the originating monitoring alert detail - Supports switching into trace analysis and inspection review within the same session canvas via
mode=traceandmode=inspection_review - The Session Toolset drawer displays the effective execution policy directly and allows editing adapter selection,
adapter.toolblocklists, budget overrides, and scope overrides - Running an explicit analysis opens a confirmation modal first so the user can choose the provider, analysis profile, and runnable analysis mode, edit the analysis objective for the round, and preview the session scope and toolset source before calling the session analysis API; runnable modes are
root_cause,performance,trace, andinspection_review(generalis rejected by the backend), and upon success the chosen mode andagentProviderIdare written back to session metadata - In the dsh candidate, ordinary Chat and Heya use dsh by default and retain Core-owned authorization, Heya gates, approvals, budgets, audit, and domain analysis. Explicitly selected external Hermes/CLI providers use their own configured protocols; they are not fallback paths.
- All
analysisArtifactsattached to assistant messages in the session aggregate into an Analysis Artifact History view, allowing users to switch graphs, evidence, and recommendations across root-cause, performance, trace, and inspection-review artifacts
/ai-workbench/root-cause and /ai-workbench/performance
Root-cause and performance modes reuse the same session canvas while expressing the active analysis type directly in the route path rather than relying on legacy investigation?mode=* redirect parameters as the primary information architecture.
/ai-workbench/inspection
Unified entry point for inspection tasks, run records, and automation policies.
/ai-workbench/inspection is an interactive workspace rather than a read-only list:
- Supports creating and editing inspection tasks directly inside the AI workbench
- The inspection task form scopes inputs across platform, cluster, and namespace levels
- Inspection tasks can select an
mode=inspectionanalysis profile stored in task metadata asanalysisProfileId, allowing backend execution to override inspection playbooks and data-source constraints - Inspection tasks support deletion; the delete action in the workbench requires
observe.ai.inspection.manage, isolates tasks by creator on the backend, and cascades cleanup to associated inspection runs in PostgreSQL - Creating, editing, immediately executing tasks, and converting an inspection run into an investigation session all happen within the same workspace under
observe.ai.inspection.manage,observe.ai.inspection.run, andobserve.ai.chat; converting an inspection run into an investigation session additionally requiresobserve.ai.view, and generating an inspection task from an investigation session requiresobserve.ai.chat - When generating an inspection task from an investigation session, the workbench attaches the active inspection profile when available so profile-driven check contracts are preserved
- The Automation Policies tab supports creating and editing trigger types, analysis types, analysis profiles, remediation policies, deduplication windows, cooldown intervals, and enablement status
- Reading, creating, editing, and deleting automation policies continue to use
/api/v1/copilot/automation-policiesgated bysettings.ai.manage; without that permission, the workbench does not fetch the policy list proactively and instead renders an explicit permission boundary - The automation policy form selects
analysisProfileIdfrom analysis profile summaries in the safe workbench catalog andagentProviderIdfrom the agent provider catalog; the executor currently supports only thealert_webhooktrigger type, and unimplemented trigger types are not exposed as runnable options - Runnable analysis types for automation policies include
root_cause,performance,trace, andinspection_review; external providers can also extend via Agent Runtime to capabilities such as delivery failure, post-deploy observation, platform resource diagnosis, Docker/virtualization diagnosis, and on-call briefs - Alert severity, status, minimum duration, label matchers, analysis time window, and approval roles are persisted into structured
triggerConditions/approvalPolicyfields so policies created in the workbench are actionable rather than empty shells - The automation executor writes a policy-prefixed
dedupKeyon built-in root-cause, performance, and trace analysis runs, while external providers receive the same deduplication context inAgentRun.input.dedupWindowSecondsdeduplicates identical alert fingerprints, andcooldownSeconds > 0enforces policy-level cooldowns so distinct alerts do not re-trigger repeatedly during the cooldown window
Promoting inspection results into a session initializes rich investigation context rather than an empty chat:
- The backend writes the inspection task’s
clusterId/namespaceinto the session scope - Session
pinnedContextretainsinspectionRunId,inspectionTaskId, severity, and status - The initial assistant message carries an
inspection_reviewanalysis artifact - That artifact converts findings into evidence, recommendations, and a left-to-right relationship graph rendered directly in the right-hand evidence panel
/ai-workbench/tool-settings
Displays:
- MCP adapters
- Mirrored global data-source status
- Session-scoped toolset assembly controls
- Enterprise skill registry
- Safe catalog of Agent Providers / Capabilities / Tool Bindings / Skill Bindings
Tool assembly controls stay aligned with the backend execution contract:
- Disabled tools are stored in qualified
adapter.toolform so identically named tools on other adapters are unaffected - An empty adapter list denotes automatic selection; recommended presets preferentially enable registered adapters backed by active data sources
- Budget overrides persist positive values only so
0is never misinterpreted as an active cap - Scope overrides write structured
clusterId,namespace,workload,service,alertId, andtimeRangeMinutesfields
/ai-workbench/model-settings
Embeds global AI settings inside the AI workbench, continuing to enforce settings.ai.view / settings.ai.manage.
Safety And Execution Model
The AI layer maintains an analysis-and-recommendation-first safety posture.
Current priorities:
- Aggregate operational context
- Invoke read-only tools
- Produce evidence, hypotheses, and recommendations
- Persist tool invocations and analysis artifacts into sessions
- Normalize external agent outputs into soha
AnalysisArtifactrecords
High-risk mutating actions are not wired into unguarded chat auto-execution chains.
Application onboarding spec generation and platform delivery orchestration belong to the application delivery workbench rather than the AI workbench; the AI workbench only exposes MCP/skills capabilities discoverable and invokable by enterprise AI coding clients and Agent Runtime.
Near-Term Expectations
Ongoing AI functionality follows these engineering rules:
- New AI surfaces integrate into the AI workbench rather than adding standalone legacy table pages
- Session-related enhancements extend
metadatafirst before splitting out separate investigation entity tables - Every new data source or tool capability accounts for:
- Global configuration
- Session-scoped assembly
- Durable analysis artifact persistence
- Root-cause, performance, trace, inspection-review, and external agent analyses share the unified artifact model rather than inventing per-page protocols
- Adding a new agent provider extends only the provider adapter, tool bindings, skill bindings, and runner executor; UI pages and domain analysis flows never depend directly on a provider SDK or CLI
- Handoffs from the monitoring workbench into the AI workbench preserve the standard
scopecontract rather than defining ad-hoc per-page redirect parameters
Knowledge source import permissions
External HTTP, Git and Object connectors can store a versioned importPolicy in config to define the users, roles, teams and projects a source may grant. Omission defaults to the stable knowledge-base owner; an explicitly empty private scope grants nobody. Core clips fetched ACLs before storing documents and chunks, so a remote public ACL cannot exceed this ceiling. Shared sources can declare scoped grants. Finite grants intersect within the same identity category, without inferring membership across categories. Inline imports retain their existing behavior.
Source credentials use soha://secrets/<id> (optionally /versions/<n>) and the
SOURCE alias through Gateway secretRefs authorization, pinning and
resolution rather than ordinary plugin input or documents. Runtimes without
per-call credential delivery reject the invocation. Import permission checks do
not establish availability of the three fetch plugins. Existing indexes are not
automatically rewritten; policy editing, revocation of previously indexed data
and real source fetching still require their own acceptance evidence.
Legacy secret: references remain readable configuration. Fetching rejects them
until an authorized edit selects the intended canonical secret reference; it does
not guess a secret identity from a legacy name or path.