feat(fuse,packaging): watch the daemon for stalls and answer the systemd watchdog

A supervisor thread ticks every 15 s and does three things: it WARNs
about a worker that has held one job for two minutes, a control request
that never returned, or a lane that has not completed anything while its
queue is non-empty; it samples resident size and the pending-op count
into the log every five minutes; and it answers the systemd watchdog.

The watchdog ping is deliberately not a heartbeat from a thread that
does nothing else — that would keep the unit "healthy" through exactly
the hang it is meant to catch. It goes out only after a real round trip
over the control socket, so it means "a client can still talk to this
daemon", which is the property that failed on 2026-09-18.

`sd_notify` is spoken directly rather than through libsystemd (READY,
WATCHDOG, STOPPING), including the abstract-namespace address; off
systemd every call is a no-op.

The unit gains `NotifyAccess=all` (rather than `Type=notify`, which
would fail the unit on the start timeout when no login is stored yet),
`WatchdogSec=120`, `Restart=always` — a hang is not a failing exit code
— and `MemoryHigh`/`MemoryMax` as a backstop against the 5.2 G growth
the hung run showed.
This commit is contained in:
nvrl committed 2026-09-19 14:45:15 +02:00
1 parent f24ac8a429
commit bee15ee7cf
5 files changed
+370 -1

No files matched your search

+17 -1
View File
@@ -6,12 +6,28 @@ After=graphical-session.target
PartOf=graphical-session.target
[Service]
# Deliberately not Type=notify: the daemon waits for a stored login before it
# mounts anything, and a machine that boots before the user logs in would then
# fail the unit on the start timeout. NotifyAccess=all gets the daemon a
# NOTIFY_SOCKET all the same, which is all the watchdog below needs.
Type=simple
NotifyAccess=all
# The daemon waits for a stored login, mounts ~/ProtonDrive, and keeps it
# mounted. SIGTERM (from `systemctl --user stop`) triggers a clean lazy unmount.
ExecStart=/usr/bin/pdfs daemon
Restart=on-failure
# Always, not on-failure: a daemon that hangs is not a failure by exit code, and
# on 2026-09-18 one had to be restarted by hand after wedging overnight.
Restart=always
RestartSec=3
# The supervisor thread pings this, but only after a round trip over the control
# socket — so a daemon that stops answering its own clients is restarted instead
# of sitting there looking alive. It pings at half this interval.
WatchdogSec=120
# Backstop against unbounded growth: the hung run peaked at 5.2G resident.
# MemoryHigh throttles and reclaims first; MemoryMax is the hard stop, above any
# legitimate working set, so it only fires on a leak.
MemoryHigh=2G
MemoryMax=6G
# Give the lazy unmount time to finish before SIGKILL.
TimeoutStopSec=15