Agent policies, secrets, and custom probes

This page covers three related screens under Fleet: Agent Policies, Secret Injections, and Custom Probes. They’re grouped together in the app’s navigation, but each controls something different - read the headings below rather than assuming they’re variations on one setting.

Agent policies

Fleet → Agent Policies (/gravity/settings/agent-policies) controls how much autonomy automated and AI-driven actions are given within your organization. This is different from the per-agent capability grants described elsewhere in this section (those control what a specific fleet agent is allowed to be asked to do at all - patch scanning, remote exec, and so on); agent policies control how much an automated system is allowed to act on its own once something is being considered.

Each policy has:

• Scope - organization-wide, a specific environment (like prod or dev), or a specific action-type pattern
• Autonomy level:
  • L0 - Observe Only: read-only, no writes
  • L1 - Suggest Only: generates recommendations, a human executes them
  • L2 - Execute with Approval: can act, but requires human approval first
  • L3 - Auto-Execute: acts without approval, within the policy’s other limits - the app shows an explicit warning when you choose this level
• Freeze windows - recurring time windows (with days and a timezone) during which automated actions are always blocked, regardless of level, useful for a change freeze or a standing weekend blackout
• Max blast radius - the largest scope a single automated action is allowed to affect: a single resource, a service tier, an availability zone, or a region
• Blocked / allowed action types - an explicit list on top of the level

Safe defaults

Start new policies at L0 or L1 until you’ve reviewed what they’d actually do in practice. If you do use L2 or L3, pair it with a tight blast radius and at least one freeze window, rather than leaving it unconstrained.

Automated cloud provisioning checks these policies before an action proceeds.

Secret injections

Fleet → Secret Injections (/gravity/fleet/secret-injections) delivers secret values to your own infrastructure. You set the value in the app, Sencai stores it encrypted in Sencai Vault, and an agent running on your infrastructure writes it to the target:

• Environment variable and file targets go to one enrolled host: the fleet agent on that host writes the value to a file, or to the environment of a systemd service.
• Kubernetes Secret targets go to a connected cluster: the in-cluster agent writes the value to a Secret in your cluster.

Creating, changing, delivering and revoking a secret injection requires the Owner or Admin role with fleet write access in the organization that owns it.

Before a host can receive secrets

A host can receive environment variable and file targets only when all four of these are true. The Host picker in Add Injection lists every host in your organization and, for a host that can’t receive secrets yet, shows which one is missing.

1. Its fleet agent is active and runs a release that supports secret delivery.
2. The host’s own policy file, /etc/sencai-agent/secret-delivery.yaml, turns delivery on and lists where the agent may write. Only someone with root on the host can change it - nothing sent from Sencai does. See Enabling secret delivery on a host.
3. The agent has the Secret Delivery (secret:deliver) capability, granted on Fleet → Agents.
4. An Owner or Admin has trusted the host for secret delivery.

Trusting a host

On Fleet → Agents, the Secret Delivery column shows where each host stands. Open it to see what the host reports - whether delivery is on, the locations and services it allows, whether systemd is available - when it last reported, and whether it’s trusted.

Trust lets Sencai release secret values to that agent’s current certificate. Only an Owner or Admin of the agent’s organization can trust a host, and only while the host reports delivery as enabled and its agent is running and connected. A host stays trusted until someone clicks Withdraw trust, the agent is disconnected, or the agent is enrolled again - a re-enrolled agent has a new certificate and has to be trusted again.

Caution

The fleet agent runs as root. Anyone with root on a trusted host, or anyone who can run remote commands on it through Remote Exec, can read the values delivered there.

Granting Secret Delivery and Remote Exec

Secret Delivery (secret:deliver) and Remote Exec (exec) are the two capabilities that reach secret values on a host, so they follow stricter rules than the others:

• Only an Owner or Admin of the agent’s organization can grant them, with Grant Capability on Fleet → Agents.
• They can’t be included when you generate an enrollment token. Grant them once the agent is active.
• A grant is authorized only for a short time after you confirm it. If the app reports that granting one of them was refused for a host, try again.
• An agent that isn’t connected receives a granted capability when it reconnects. Until then, its secret delivery details show that the grant hasn’t reached it.

Creating an injection

For a file target:

1. Click Add Injection and choose File as the Injection Method.
2. Enter a Secret name - a label for this injection, never the value - and pick the Host.
3. Enter the File path: an absolute path below a location the host allows, for example /run/sencai-secrets/myapp/db-password. Choose Permissions (default 0600) and the file’s Owner and Group (default root).
4. Click Create. The Set value dialog opens next - enter the value there.

For an environment variable target:

1. Click Add Injection and choose Environment Variable.
2. Enter a Secret name and pick the Host.
3. Choose the Systemd unit - one of the services the host allows, for example myapp.service - and enter the Variable name.
4. Under After delivery, keep Don’t restart — takes effect at the next restart, or choose Restart the service if it is running. The second option is available only if the host’s policy file allows Sencai to restart that service.
5. Click Create, then enter the value in Set value. The value must be text without line breaks, at most 65536 bytes - use a file target for multi-line or binary values.

For a Kubernetes Secret target, pick the Cluster, Namespace, Secret name and Key, then click Create. The Set value dialog opens next - enter the value there.

Variables that change how a program runs, such as PATH, LD_PRELOAD or NODE_OPTIONS, are never delivered, and neither are core system services such as sshd.service.

Delivering

Setting a value doesn’t send it anywhere. To deliver it, click Deliver now on the injection’s row. Sencai doesn’t deliver again on its own, not even when a host reports that a delivered value has gone missing. The row then shows where the delivery stands:

State	What it means
Not delivered yet	No delivery has been requested
Queued	Waiting to be sent to the host
Waiting for host	The host’s agent isn’t connected. The delivery is sent when it reconnects, and expires after 60 minutes if it doesn’t
Delivering…	The host’s agent is writing the value
Applied	The value was written
Missing on host	The host reports that the value is no longer there, for example after a reboot. Deliver it again
Changed on host	The delivered file or environment file was changed on the host. Deliver again to restore it
Failed	The host refused or couldn’t complete the delivery - Delivery history shows each attempt and its result
Expired — not collected in time	The delivery wasn’t completed before it expired
Delivery switched off	See When delivery is switched off

The host checks every delivery against its own policy file again, so a delivery the app accepted can still be refused there - for example when the policy file changed since the host last reported, when a file the agent didn’t create already exists at that path, or when the path runs through a symbolic link.

On the host

• Files. The default location, /run/sencai-secrets/, is in memory, so it’s cleared when the host reboots. After a reboot the injection shows Missing on host, and you deliver it again. Other locations work only if the host’s policy file allows them. The agent writes the file in one step with the permissions, owner and group you chose, never follows a symbolic link, and won’t overwrite a file it didn’t create.
• Environment variables. The agent writes the variable to an environment file it manages for that service, under /run/sencai-agent/env/, adds a systemd drop-in that loads it (50-sencai-agent-secrets.conf in /run/systemd/system/<unit>.d/), and reloads systemd. The service sees the value the next time it starts - straight away only if you chose Restart the service if it is running and the host allows that restart. Both files are in memory too: after a reboot the service starts without the variable until you deliver it again.
• Who can read a delivered value. Root on the host; for a file, also the owner and group you chose, as its permissions allow; for an environment variable, also the service’s own user, through /proc/<pid>/environ; and anyone who can run remote commands on that host through Remote Exec.

Where the value is stored

When Sencai stores a secret value you set, it stores it encrypted at rest in Sencai Vault, a secret store that Sencai operates itself on Google Cloud infrastructure in the EU, as listed in our sub-processor register. Sencai uses the value only to deliver it to your own infrastructure. The terms that apply to these values are part of our Data Processing Agreement.

• A saved value can’t be viewed or exported through the platform, and it’s never shown again after you save it.
• Each delivery is encrypted with a key generated for that delivery alone. For an environment variable or file target, the fleet agent on the host generates that key, fetches the value and writes it to the target; for a Kubernetes Secret target, the in-cluster agent does the same for your Kubernetes Secret.
• A value is released to a host only after the host has been trusted, and only to the certificate its agent had when it was trusted.
• When you replace a value, earlier versions may be kept until the injection is revoked or deleted. After you revoke or delete it, Sencai deletes the value and every kept version from Sencai Vault without undue delay.
• Any copy already delivered to your infrastructure is under your control.

Revoking and deleting

Revoking a delivered environment variable or file injection while its host’s agent is connected tells the agent to remove what it delivered:

• File - the file is removed. Programs that already read it keep the value in memory until they restart.
• Environment variable - the variable is removed from the service’s environment file; when it was the last one Sencai delivered to that service, the environment file and the drop-in are removed too. A running process keeps the old value until it restarts. The agent restarts the service only if the injection was set to restart it and the host still allows that restart.

If the host’s agent isn’t connected, the removal waits for it to reconnect. If a removal failed or expired, you can Retry removal or Close without confirmation. If the host’s agent has been disconnected or deleted, the injection shows Host disconnected and you can only Close without confirmation, because nothing is left that could remove the value. Closing without confirmation deletes the stored value from Sencai Vault, but the copy on the host stays there until you remove it yourself. An injection that was never delivered closes straight away.

Revoking a delivered Kubernetes Secret injection while its cluster is connected tells the in-cluster agent to remove the key from the Secret (or the Secret itself, if that was its last key). Pods that already read the value keep it until they restart. If the removal failed or expired, you can Retry removal; if the cluster is disconnected, you can only Close without confirmation. Where a removal is never confirmed, the value may stay in your cluster - any copy there is under your control.

Delete removes the configuration from Sencai, not anything already written to a target, so revoke first if the value has been delivered.

To see what happens to delivered values when you disconnect a host’s agent, see Uninstalling.

Legacy injections

Environment variable and file injections created before delivery to hosts existed show a Legacy chip and the state Legacy — recreate. They only point at a secret reference: they can’t hold a value and can never be delivered. Close or delete them, and create a new injection with a host target instead. Closing or deleting a legacy injection requires the Owner or Admin role with fleet write access.

When delivery is switched off

Like remote execution, delivering secrets to hosts is gated and can be switched off for the whole platform. While it’s switched off, Secret Injections shows a warning, and:

• You can still create injections, set values and edit notes.
• Deliver now is unavailable and nothing is sent to any host. A delivery or removal that was already waiting shows Delivery switched off.
• Sencai can’t remove a delivered value from a host. Revoke offers to close the injection with acknowledgement instead: Sencai deletes the stored value from Sencai Vault, and the copy already delivered stays on the host until you remove it there.

Custom probes

Fleet → Custom Probes (/gravity/fleet/probes) lets you define a check - an allow-listed binary plus arguments - that runs as root, on every enrolled agent in your organization, on a schedule you choose.

Caution

There is no per-agent targeting. Enabling a probe here means it runs on your entire fleet, not a single host you might have had in mind.

Only a fixed set of binaries can be used: docker, kubectl, systemctl, journalctl, cat (restricted to /etc/os-release), dpkg-query, and rpm. The exact command a probe will run is always shown before you save it - never hidden behind a label.

1. Click New probe.
2. Choose a binary and, if applicable, arguments.
3. Set a timeout and how often it runs.
4. Save.

Results - exit code, output, and when each run was collected - are listed per agent under Recent results, filterable by probe.

Caution

Apart from cat, which is pinned to /etc/os-release, only the binary is restricted - the arguments you give it aren’t. Most real checks with these binaries are read-only (systemctl status, docker inspect, journalctl -u), but nothing stops you from defining one that isn’t (systemctl restart, for example) - and it would then run, as root, on every enrolled agent in your organization on the schedule you set. Review the exact command shown before you save a probe, the same way you’d review a remote command before dispatching it.

Safe defaults

Prefer genuinely read-only commands, and start with a longer interval - a probe with a mutating command run fleet-wide is closer to a scheduled runbook than an inspection check. A probe failing or timing out on one host has no effect on any other host it runs on.

What’s next

• Fleet agent - the capability model these policies sit alongside
• Enrollment - enabling secret delivery on a host, and what disconnecting leaves behind
• Remote execution - commands and runbooks, gated the same way
• Agent releases - cohorts and staged rollouts
• AI governance - the related EU AI Act oversight controls