SelfHeal Documentation

What SelfHeal does

SelfHeal connects your monitoring system (for example Prometheus Alertmanager) to your servers via SSH. When an alert fires, SelfHeal:

1. Ingests the alert via a webhook or event pipeline.
2. Creates or updates an incident in its internal store.
3. Decides what to do using:
   • deterministic rules you define,
   • AI-powered modes (Advisor / Single action / Loop), or
   • oeno action if policy, mode or license restricts it.
4. Executes or simulates commands/scripts over SSH.
5. Records a detailed trail of decisions, commands and AI reasoning, and presents it as a clear incident timeline in the UI.

High-level components

• Ingest & UI service
  • Receives alerts from your monitoring system via a webhook endpoint.
  • Normalizes alerts and stores a compact incident record.
  • Serves the web UI: Dashboard, Inbox, Actions, Rules, Policy, License, Simulate, Incidents.
  • Streams live incident updates to the browser so operators see changes in real time.
• Action service
  • Consumes normalized alerts from the ingest pipeline.
  • Applies mode logic (rules-only, advisory, single action, or loop).
  • Runs SSH commands or scripts with multiple layers of guardrails.
  • Captures command outputs and AI reasoning for later review.
• State store
  • Backed by a lightweight embedded database on the SelfHeal node.
  • Stores incidents, actions and AI audit data.
  • Designed so operators do not need to manage schema or migrations directly; upgrades take care of internal structure.
• Optional streaming bus
  • In larger deployments you can front SelfHeal with a message bus such as Kafka.
  • Alerts flow through your stream and into SelfHeal, giving you buffering, replay and decoupling from your monitoring system.
• Ganges license portal
  • Runs as a separate portal (this website).
  • Issues cryptographically signed license bundles.
  • SelfHeal validates licenses locally to enforce node limits and which AI capabilities are enabled.
  • License checks are offline; alert data does not leave your cluster.

Alert → Action data flow

1. Alert fires in your monitoring system.
2. Delivery: the monitoring system sends a webhook or event to SelfHeal using the URL and token from your configuration.
3. Ingest & normalization:
   • SelfHeal parses the payload, extracts key labels and metadata, and records an incident.
   • For each alert, SelfHeal creates a normalized internal view that the action engine uses consistently across sources.
4. Decision (safety-first):
   • Is the target instance allowed?
   • Is the instance currently in a maintenance window?
   • Do we have a matching rule for this alert?
   • Is AI available (mode + license) for this incident?
5. Execution or simulation:
   • If a rule applies, SelfHeal runs the associated command or script when it passes policy checks.
   • If an AI mode is active, SelfHeal asks the AI planner to propose a safe command or sequence, then applies the same guardrails.
   • If anything fails a guardrail, SelfHeal records a clear oeblocked reason instead of executing.
6. Observability & UX:
   • Every action and AI step is recorded with timestamps, host, status and (when applicable) stdout/stderr.
   • The incident view shows a single, chronological story of the alert, decisions and actions taken.

Licensing and node caps

SelfHeal reads an offline license file issued from the Ganges portal. This license controls how many nodes you can manage and which AI features are enabled.

• License status and limits are visible under the License page.
• Node caps and AI modes (Advisor, Single action, Loop) are enforced by license.
• Even when a license constraint blocks execution, SelfHeal still records what it would have done so you can see intent and tune settings.

Level 4 AI Loop (Plan → Execute → Observe → Iterate)

• AI can run a short, bounded sequence of commands with feedback:
  • Step 1: run diagnostics (for example, check disk or memory usage).
  • Step 2: choose and run an appropriate fix.
  • Step 3: verify state, and if healthy, stop.
• Loops are limited by a configurable maximum number of steps to avoid unbounded automation.
• Each step records the command, output and AI reasoning and appears in the incident timeline.
• Best suited for advanced users who want a oeself-driving runbook on trusted lab or staging clusters first.

You can keep production clusters at Levels 0 2 indefinitely and only enable Levels 3 4 where your team is comfortable with automated changes.

Simulate

• Central place to send test alerts into SelfHeal without touching your real monitoring system.
• JSON editor prefilled with a realistic alert payload.
• Mode dropdown to simulate rules-only, advisory, single action or loop behaviour.
• Simulated incidents never run real commands, regardless of cluster mode.
• Perfect for demos, regression tests, and verifying rules or policies.

Installation model

SelfHeal is distributed as a single Linux tarball with an installer script. The experience is designed to feel familiar to operators used to CNCF-style tools.

• Download selfheal-1.0.9.tar.gz from the Ganges portal.
• Extract the archive and run the install script as root.
• The installer:
  • Places SelfHeal code in a dedicated installation directory.
  • Creates a configuration directory for configuration and license files.
  • Sets up system services for the UI/ingest and the action engine.
  • Optionally installs and configures nginx as a reverse proxy with HTTPS and basic auth.

For copy-paste commands and concrete examples, refer to the Quickstart page on this portal.

What you get after install

• Two system services:
  • one for the web UI and alert ingest,
  • one for the action engine.
• A main configuration file where you set:
  • cluster mode (rules-only, advisory, single action, loop),
  • authentication token for incoming webhooks,
  • any non-default ports or paths,
  • optional integration with an event stream (like Kafka).
• A place to drop your license file issued from the Ganges portal.
• Log output integrated with your Linux logging system (for example journald), plus structured action and incident history available in the UI.
• An HTTP endpoint (typically behind nginx) where operators access the UI with HTTPS and your chosen authentication method.

The goal is to keep oeday 1 simple: one node, one installer, and a small number of configuration knobs to get from zero to first incident quickly.

Onboarding a new SelfHeal node

1. Install SelfHeal on a small Linux VM or bare-metal host.
2. Open the main configuration file and set:
   • an authentication token for incoming alerts,
   • the initial cluster mode (for example rules-only + Advisor),
   • any HTTPS or proxy-related settings recommended by your organisation.
3. Add one or two lab servers to the Allowlist page.
4. Create a simple rule for a test alert (for example a demo alert that runs a harmless echo command).
5. Use the Simulate page to send a demo alert.
6. Verify Dashboard, Inbox, Actions and Incidents all show the expected story before wiring in your real monitoring system.

Rolling out AI modes safely

1. Keep production clusters in Advisor mode initially so AI only suggests actions.
2. Enable Single action only for a small set of low-risk alerts on trusted hosts (for example simple service restarts in a lab).
3. Monitor actions and AI explanations to build trust with your team.
4. Introduce Loop mode later, starting with staging or non-critical clusters, and with conservative command policies.

Debugging steps

1. Check Inbox for the alert. If it is not there, webhook routing or authentication is likely wrong.
2. Open the Incident view for that alert and read the timeline.
3. Look at the action rows:
   • If an action is marked as Blocked, read the policy reason and adjust allowlist or command policy if appropriate.
   • If actions are marked as Dry-run, switch the cluster to full execution once you are confident.
   • If there are no action rows: SelfHeal either stayed in observation-only mode or had no rule/AI path available for this incident.
4. Check the License page for license validity and node caps.
5. Use your system logs (for example journalctl) for deeper diagnostics if the services themselves are unhealthy.

Is SelfHeal agentless?

Yes. SelfHeal connects to your servers over SSH and does not require any agent daemons on the target machines.

Which OS and workloads are supported?

SelfHeal v1.0.9 focuses on Linux servers and services that can be managed via SSH commands and scripts. Containers, Kubernetes and other platforms can be managed via their CLI tools (for example kubectl) as long as those tools are available on the SelfHeal node.

Can I use AI modes without letting SelfHeal run commands?

Yes. Advisor mode provides AI suggestions only. You can stay in this mode indefinitely and still get value from analysis and recommendations without any automated execution.

How do I keep my secrets safe?

Use your standard Linux practices for SSH keys and other secrets (file permissions, key management, vaults, etc.). SelfHeal reads configuration and license files from its own configuration directory but does not attempt to bypass or replace normal OS security. Root access on the SelfHeal node remains the trust boundary.

Where do I start?

Start with a small lab cluster, one or two alerts, and Level 1 rules plus Advisor mode. Once you 're comfortable with the guardrails and incident timelines, expand to more alerts and consider enabling AI Single action on low-risk scenarios.