feat(fuse,packaging): watch the daemon for stalls and answer the systemd watchdog

A supervisor thread ticks every 15 s and does three things: it WARNs
about a worker that has held one job for two minutes, a control request
that never returned, or a lane that has not completed anything while its
queue is non-empty; it samples resident size and the pending-op count
into the log every five minutes; and it answers the systemd watchdog.

The watchdog ping is deliberately not a heartbeat from a thread that
does nothing else — that would keep the unit "healthy" through exactly
the hang it is meant to catch. It goes out only after a real round trip
over the control socket, so it means "a client can still talk to this
daemon", which is the property that failed on 2026-09-18.

`sd_notify` is spoken directly rather than through libsystemd (READY,
WATCHDOG, STOPPING), including the abstract-namespace address; off
systemd every call is a no-op.

The unit gains `NotifyAccess=all` (rather than `Type=notify`, which
would fail the unit on the start timeout when no login is stored yet),
`WatchdogSec=120`, `Restart=always` — a hang is not a failing exit code
— and `MemoryHigh`/`MemoryMax` as a backstop against the 5.2 G growth
the hung run showed.
This commit is contained in:
nvrl committed 2026-09-19 14:45:15 +02:00
1 parent f24ac8a429
commit bee15ee7cf
5 files changed
+370 -1

No files matched your search

+23
View File
@@ -554,6 +554,26 @@ pub fn mount(
}
}
}
// The supervisor watches the pool and answers the systemd watchdog. Started
// after the listener, because its liveness probe is a round trip over that
// socket.
{
let supervisor_core = core.clone();
let socket = control_socket.to_path_buf();
match std::thread::Builder::new()
.name("pdfs-supervisor".into())
.spawn(move || supervisor::run(supervisor_core, socket))
{
Ok(handle) => workers.push(handle),
// Not fatal: the mount works without a supervisor, it just goes back
// to being undiagnosable if it wedges.
Err(error) => warn!(error = %error, "could not start the daemon supervisor"),
}
}
// The mount is live and the control socket answers: this is the moment the
// unit is actually usable, which is what `Type=notify` reports.
systemd::ready();
// Re-establish on-demand mounts only after the primary session is live.
// Keep the worker so shutdown can close publication, wait for any in-flight
// fetch to observe that closure, and then drain a stable registry.
@@ -680,6 +700,9 @@ pub fn mount(
/// expected and correct: a worker part-way through an upload finishes it rather
/// than abandoning the user's bytes mid-flight.
fn stop_workers(core: &Core, control_socket: &Path, workers: Vec<std::thread::JoinHandle<()>>) {
// Say so before the joins: a teardown that takes a while must not be read as
// a missed watchdog ping and turned into a kill.
systemd::stopping();
core.shutdown.stop();
core.wake_drain();
let _ = core.sync_tx.send(sync::SyncMsg::Stop);