The day-2 operations loop

This path is for the people who keep running infrastructure that already exists - an on-call rotation, a platform team, or the one person who gets called when a disk fills up at 3am. It assumes the building is done. Nothing here is about provisioning a new instance or enrolling a new host; it is about the loop that starts the moment something you already run goes wrong.

By the end you will know how an alert becomes an incident, what the ranked root-cause candidates underneath one are and are not, how to write a runbook that actually executes instead of parking itself forever, and how to bind a recurring incident pattern to that runbook without handing an unattended pattern match the keys to a live host. You will also be able to tell apart the several review-and-approval mechanisms in this product, which look alike and behave nothing alike.

It assumes at least one thing is already under management and reporting - an instance Sencai provisioned, or a host with the fleet agent enrolled. Without one of those the first screen in this path is empty and the rest of the loop has no input.

Before you start

Three things have to be true before this path pays off.

• Something is being monitored. Metrics exist only for resources Sencai manages, so you need either a provisioned instance (see Provision & manage instances) or an enrolled agent (see Enrolling a fleet agent, or the whole Run a Linux fleet path if you have not done it yet).
• You know which role you hold. Reading every screen in this path is open to Viewer and Auditor. Resolving an incident or accepting a root-cause candidate needs Member or higher. Creating or triggering a runbook, and creating an auto-remediation rule, need Admin or Owner. See Roles & permissions - and note that Auditor sits below Member, so it is not the role to hand your on-call engineer.
• You have asked about fleet dispatch. Steps 5 and 6 are the only ones that change a real machine, and they depend on a gate that is not self-service.

Caution

Dispatching anything to a fleet host - a runbook, a dry run, or an ad-hoc command - has to be enabled for your environment, and it is not on for every account. Where it is off, a triggered runbook is recorded as failed rather than reaching the agent, and the record looks much like a run that failed on the host. Ask your account team where your environment stands before you plan a response around runbooks. You can still author, version and review them today; what you cannot assume is that triggering one changes anything.

One thing you cannot prepare in advance: the alert thresholds behind the metric cards are not editable from a self-service screen. What you see is what has been configured for your organization, and a custom threshold is an account-team request. Step 2 is about reading thresholds, not tuning them.

The path

The order follows the data dependency, not the sidebar. Each step produces the input the next one consumes: monitoring raises alerts, correlation turns alerts into an incident, the incident carries the candidates you weigh, and a runbook is the thing you trigger once you have decided. Runbooks come before auto-remediation because a rule can only fire a runbook that already exists, and both come after the read-only steps on purpose - they are the two that touch a live host. Change records and notification settings sit at the end because they wrap the loop rather than drive it.

0 of 8 read

1. 1 See the loop in order Monitoring feeds incidents, incidents feed root cause, root cause feeds remediation. Each step needs the one before it. Open →
2. 2 Read the monitoring screens Per-instance metric cards against fixed thresholds, and how to force a collection cycle. Open →
3. 3 Triage an incident Its context bundle, its deduplication key, and the ranked root-cause candidates attached to it. Incidents are created automatically. There is no button to raise one by hand, so you need a real alert before you can practise this. Open →
4. 4 Accept or reject root-cause candidates What the ranking means, and why accepting a wrong candidate is worse than accepting none. Open →
5. 5 Author a runbook The action array, targeting a single agent first, and running a dry run before anything else. Requires an enrolled fleet agent. New runbooks are created with approval required, and that gate is not usable today - turn it off or the runbook waits forever. Open →
6. 6 Bind a pattern to a runbook Match on an incident pattern, set a confidence threshold and a cooldown, and test it before enabling. Open →
7. 7 Record planned changes The Draft to Implemented board, with risk level, rollback plan and evidence. Open →
8. 8 Make sure alerts reach you Categories, email preferences, browser push, and which channel actually delivers. Open →

What to watch out for

You cannot rehearse a triage

There is no New Incident button. Incidents exist only because Sencai correlated real alerts into one, using a dedup key so a recurring problem updates the same record instead of opening a new one. That means you cannot seed a practice incident to walk a new engineer through, and your first real triage will also be your first look at the context bundle and the candidate list. Read Incidents before you need it, not during.

An empty screen often means “not running”, not “nothing found”

Three screens in and around this loop look calm when they are actually inactive, and they fail in the same reassuring way:

• The AI-assisted candidates on Intelligence → Root Cause Analysis (/gravity/operations/rca) are generated once an incident resolves, and that generator is enabled per environment. An empty list can mean it was never switched on for you. The ranked candidates shown on the incident itself are a separate, heuristic list.
• Correlation findings run on a recurring schedule that also has to be enabled.
• Synthetic monitoring stores the HTTP checks you define but nothing currently executes them, so the results table stays empty however correct your check is. Defining checks is still worth doing - the configuration is what your account team needs in order to turn execution on - but do not put an endpoint into your uptime story on the strength of a check that exists here.

Before you conclude that a quiet screen means a healthy system, find out which of the two it is.

Accepting a root-cause candidate is bookkeeping, not a fix

Accept and Reject on the Root Cause Analysis screen update the candidate’s status and are logged. Neither one creates a runbook, opens a change request, or triggers anything else - you still decide and act yourself. That cuts both ways: because nothing downstream depends on your answer, nothing downstream will ever contradict it either. An accepted candidate simply becomes your team’s stored explanation for that incident, and a confidently wrong one is worse than leaving it pending, because it ends the search. Weigh the confidence score (green above 70%, amber between 40% and 70%) and the evidence list, and note whether a model or a simpler rule-based check produced it.

The runbook approval default will park your first run forever

New runbooks are created with Require approval turned on, and approving a pending execution does not work in the app today - the approval step needs a token that is not surfaced in the interface or the API. A runbook you create and trigger without changing that setting sits in pending indefinitely and never reaches a host. Turn it off explicitly on anything you intend to actually run.

Two related traps on the same screen: the Trigger dialog labels the target field Agent ID (optional) even though a trigger with no agent is rejected, and every trigger names exactly one enrolled agent - there is no organization-wide “run everywhere”. A dry run skips the approval gate, but it is still sent to the target agent, so it depends on the same dispatch gate as a real run. The lighter Dry Run on the Runbook Versions screen is the one that involves no agent at all.

Five approval-shaped mechanisms, no two alike

These share vocabulary and almost nothing else. Merging them in your head is how a team ends up believing a change was gated when it was only written down.

Mechanism	Where	What it actually does
Compliance approvals	Compliance & Audit → Approvals (/gravity/approvals)	Genuinely blocks a high-blast-radius action until someone decides. Pending requests expire after four hours, and any member of the organization can approve or reject
Change requests	Operations → Change Requests (/gravity/operations/change-requests)	A Draft to Implemented board that records intent, risk level, rollback plan and evidence. It gates nothing - the underlying change can happen with or without it
Runbook approval	The Require approval switch on each runbook	Parks the execution in pending. Not completable today, so treat it as off-limits rather than as a control
JIT elevation	Security → Access Management (/gravity/access), request and approval screens under /gravity/elevation/	Time-boxes a specific set of capabilities into an organization you manage on someone else’s behalf. Professional plan and above - see Plans
Autonomy level	Fleet → Agent Policies (/gravity/settings/agent-policies)	Sets how far automated and AI-driven action may go by itself, from L0 observe-only to L3 auto-execute, with freeze windows and a maximum blast radius

Only the first row behaves the way people expect an approval to behave: a proposed action waits, and cannot proceed until someone decides. The change-request board blocks nothing at all - it records what you planned. JIT elevation gates access rather than an action, autonomy levels constrain what automated systems may do rather than what you may do, and the runbook switch gates a run it currently has no way to release. If you need Sencai itself to enforce a gate, the approvals queue is the only one that does. Full detail is on Approvals, Change requests, Access management and Agent policies.

Automation is live the moment you save it

Auto-remediation rules are evaluated against every newly created incident automatically. There is no master switch to arm them: an enabled rule with auto-approve on can fire against a live host as soon as you press save. Use Test Pattern on the rule form first - it scores a candidate pattern against a sample incident title in your browser without creating anything - and leave auto-approve off while you are learning what the pattern really matches. With it off, a match creates a change request for review instead of running.

One inconsistency to carry with you: rules created from the Auto-Triggers screen are matched by the same always-on evaluation, but that screen’s severity threshold is not enforced, so a rule you scoped to critical incidents there will still match a low one.

Nothing pages you

This is the finding most likely to hurt an on-call team, so plan around it before you make Sencai your primary alert channel. Escalation policies store your PagerDuty or Opsgenie configuration, but delivery to either provider is not active, and the email fallback is not active either. The outbound webhook event types cover cloud-instance provisioning, drift detection, a budget threshold being reached, and organization-membership changes - not incidents or firing alerts. What does reach you today is the in-app notification center, the email categories you control in Settings → Email Notifications (/gravity/settings/notifications), and browser push if you enable it - all covered in Notifications & escalation. One partial exception: a Slack or Microsoft Teams incoming webhook configured under organization settings does receive a smaller set of notifications that includes firing alerts - useful for a team channel, but it is a channel somebody has to be reading, not a rotation that wakes anybody up.

What is reasonable to automate today

A narrow, well-tested loop: one runbook whose action array you have dry-run against the target agent, bound to a pattern you validated with Test Pattern, with manual review on, a cooldown set, and its recent triggers reviewed after the first few fires. Widen it only when the record shows the rule matching what you expected and nothing else.

What deserves more caution than its screen suggests: auto-approve on a pattern you have not seen match a real incident, and custom probes under Fleet → Custom Probes (/gravity/fleet/probes), which run as root on every enrolled agent in your organization with no per-agent targeting. Everything either one does is recorded in your audit log, which is the honest place to check what your automation has been doing while you were not watching.

What’s next

• Synthetic monitoring - the endpoint checks you can define now, and why their results are not arriving yet
• Deployments - the deploy history that root-cause candidates draw on, fed by your own pipeline
• Approvals - the one mechanism in this product that actually blocks an action
• Agent policies - autonomy levels, freeze windows and blast radius for anything acting on its own
• Audit log - where every trigger, decision and transition in this loop is recorded