mirror of
https://github.com/narrrl/proton-drive-linux.git
synced 2026-11-04 00:47:00 +01:00
feat(fuse,packaging): watch the daemon for stalls and answer the systemd watchdog
A supervisor thread ticks every 15 s and does three things: it WARNs about a worker that has held one job for two minutes, a control request that never returned, or a lane that has not completed anything while its queue is non-empty; it samples resident size and the pending-op count into the log every five minutes; and it answers the systemd watchdog. The watchdog ping is deliberately not a heartbeat from a thread that does nothing else — that would keep the unit "healthy" through exactly the hang it is meant to catch. It goes out only after a real round trip over the control socket, so it means "a client can still talk to this daemon", which is the property that failed on 2026-09-18. `sd_notify` is spoken directly rather than through libsystemd (READY, WATCHDOG, STOPPING), including the abstract-namespace address; off systemd every call is a no-op. The unit gains `NotifyAccess=all` (rather than `Type=notify`, which would fail the unit on the start timeout when no login is stored yet), `WatchdogSec=120`, `Restart=always` — a hang is not a failing exit code — and `MemoryHigh`/`MemoryMax` as a backstop against the 5.2 G growth the hung run showed.
This commit is contained in:
1 parent
f24ac8a429
commit
bee15ee7cf
5 files changed
+370
-1
No files matched your search
@@ -554,6 +554,26 @@ pub fn mount(
|
||||
}
|
||||
}
|
||||
}
|
||||
// The supervisor watches the pool and answers the systemd watchdog. Started
|
||||
// after the listener, because its liveness probe is a round trip over that
|
||||
// socket.
|
||||
{
|
||||
let supervisor_core = core.clone();
|
||||
let socket = control_socket.to_path_buf();
|
||||
match std::thread::Builder::new()
|
||||
.name("pdfs-supervisor".into())
|
||||
.spawn(move || supervisor::run(supervisor_core, socket))
|
||||
{
|
||||
Ok(handle) => workers.push(handle),
|
||||
// Not fatal: the mount works without a supervisor, it just goes back
|
||||
// to being undiagnosable if it wedges.
|
||||
Err(error) => warn!(error = %error, "could not start the daemon supervisor"),
|
||||
}
|
||||
}
|
||||
// The mount is live and the control socket answers: this is the moment the
|
||||
// unit is actually usable, which is what `Type=notify` reports.
|
||||
systemd::ready();
|
||||
|
||||
// Re-establish on-demand mounts only after the primary session is live.
|
||||
// Keep the worker so shutdown can close publication, wait for any in-flight
|
||||
// fetch to observe that closure, and then drain a stable registry.
|
||||
@@ -680,6 +700,9 @@ pub fn mount(
|
||||
/// expected and correct: a worker part-way through an upload finishes it rather
|
||||
/// than abandoning the user's bytes mid-flight.
|
||||
fn stop_workers(core: &Core, control_socket: &Path, workers: Vec<std::thread::JoinHandle<()>>) {
|
||||
// Say so before the joins: a teardown that takes a while must not be read as
|
||||
// a missed watchdog ping and turned into a kill.
|
||||
systemd::stopping();
|
||||
core.shutdown.stop();
|
||||
core.wake_drain();
|
||||
let _ = core.sync_tx.send(sync::SyncMsg::Stop);
|
||||
|
||||
Reference in new issue
Block a user