fix(fuse): a stop wakes every drain worker (B180)

This commit is contained in:
nvrl committed 2026-10-04 23:00:07 +02:00
1 parent 4e0fd3c0b1
commit 903452a15b
4 files changed
+68 -13

No files matched your search

+37 -12
View File
@@ -703,12 +703,11 @@ impl Core {
/// Block until there is plausibly something to do: a new op, a reconnect, or /// Block until there is plausibly something to do: a new op, a reconnect, or
/// the shortest outstanding backoff elapsing. /// the shortest outstanding backoff elapsing.
pub(crate) fn wait_for_drain_work(&self) { pub(crate) fn wait_for_drain_work(&self) {
let (lock, cv) = &*self.drain_wake; sleep_for_drain_work(
let mut woken = lock.lock(); &self.drain_wake,
if !*woken { || self.shutdown.is_stopping(),
cv.wait_for(&mut woken, DRAIN_IDLE_POLL); DRAIN_IDLE_POLL,
} );
*woken = false;
} }
/// Like [`wait_for_drain_work`](Self::wait_for_drain_work), but first /// Like [`wait_for_drain_work`](Self::wait_for_drain_work), but first
@@ -722,12 +721,7 @@ impl Core {
} }
_ => DRAIN_IDLE_POLL, _ => DRAIN_IDLE_POLL,
}; };
let (lock, cv) = &*self.drain_wake; sleep_for_drain_work(&self.drain_wake, || self.shutdown.is_stopping(), timeout);
let mut woken = lock.lock();
if !*woken {
cv.wait_for(&mut woken, timeout);
}
*woken = false;
} }
/// Perform one queued op and retire it. /// Perform one queued op and retire it.
@@ -2697,6 +2691,27 @@ fn keeps_landed_upload(pinned: bool, len: u64, budget: u64) -> bool {
} }
/// Where `st` has `uid`: its name and parent. /// Where `st` has `uid`: its name and parent.
/// Sleep on the drain's `wake` for up to `timeout`, unless a wake-up came
/// since a worker last took one, or the mount is `stopping`.
///
/// One wake-up wakes every worker, and the first to wake takes it. A worker
/// that came to sleep after that slept out its timeout, the stop's included,
/// and the stop gave up on it after 10 s (`docs/BUGS.md` B180). The stop sets
/// its flag before it takes the lock, so a worker that finds it unset under
/// the lock is asleep when the wake-up comes.
fn sleep_for_drain_work(
wake: &(Mutex<bool>, parking_lot::Condvar),
stopping: impl Fn() -> bool,
timeout: Duration,
) {
let (lock, cv) = wake;
let mut woken = lock.lock();
if !*woken && !stopping() {
cv.wait_for(&mut woken, timeout);
}
*woken = false;
}
fn placed_in(st: &crate::state::State, uid: &NodeUid) -> Option<(String, Option<NodeUid>)> { fn placed_in(st: &crate::state::State, uid: &NodeUid) -> Option<(String, Option<NodeUid>)> {
let entry = st.by_uid.get(uid).and_then(|ino| st.entries.get(ino))?; let entry = st.by_uid.get(uid).and_then(|ino| st.entries.get(ino))?;
Some((entry.node.name.clone(), entry.node.parent_uid.clone())) Some((entry.node.name.clone(), entry.node.parent_uid.clone()))
@@ -2712,6 +2727,16 @@ mod tests {
use proton_drive_rs::proton_sdk::ids::{LinkId, VolumeId}; use proton_drive_rs::proton_sdk::ids::{LinkId, VolumeId};
use std::cell::Cell; use std::cell::Cell;
#[test]
fn a_drain_worker_that_missed_the_stop_wake_up_does_not_sleep() {
// Another worker took the stop's wake-up, and this one slept out its
// timeout, past the stop's deadline (B180).
let wake = (Mutex::new(false), parking_lot::Condvar::new());
let started = Instant::now();
sleep_for_drain_work(&wake, || true, Duration::from_secs(5));
assert!(started.elapsed() < Duration::from_secs(1));
}
fn pending(kind: &str) -> PendingOp { fn pending(kind: &str) -> PendingOp {
PendingOp { PendingOp {
id: 0, id: 0,
+26
View File
@@ -12,6 +12,32 @@ Conventions:
--- ---
## B180 — A stop waits 10 s for a drain worker that slept through it
**Status:** Fixed.
**Found:** 2026-10-04, by the simulation suite on 0f3c06f: `a_stop_ends_a_mirror_pass_after_the_uploads_it_is_sending`
found "background workers did not finish in time" in the log, left there by two earlier tests
whose stops had given up on a drain worker. On its own,
`a_folder_made_again_while_the_old_one_is_trashed_is_a_new_folder` took 11 s instead of 2 s in
about one run in three.
**Where:** `Core::wait_for_drain_work` and `Core::wait_for_drain_work_or_due` in
`crates/pdfs-fuse/src/drain.rs`.
**Cause.** The drain workers sleep on one flag and one condvar. A stop sets its flag, then
raises the wake-up and wakes them all, and the first worker to wake takes the wake-up. A worker
that was between its stop check and its sleep at that moment found no wake-up and slept its
timeout, up to 30 s. The stop gave up on it after 10 s and shut down around it. With 16 workers
since step 7 this was common.
**Fix.** A worker checks the stop under the wake-up's lock before it sleeps. The stop sets its
flag before it takes that lock, so a worker that finds it unset is asleep when the wake-up comes.
**Test:** `a_drain_worker_that_missed_the_stop_wake_up_does_not_sleep` in
`crates/pdfs-fuse/src/drain.rs`.
---
## B179 — A file read just as its create lands reads empty ## B179 — A file read just as its create lands reads empty
**Status:** Fixed (unverified). **Status:** Fixed (unverified).
+2
View File
@@ -84,6 +84,8 @@ by its local id, in the queue, the stored tree and pins alike.
- **Deleting a folder while sync is paused no longer leaves a queued change behind** for a - **Deleting a folder while sync is paused no longer leaves a queued change behind** for a
file in it that was never uploaded (B178). file in it that was never uploaded (B178).
- **A file read just as it reaches Drive no longer reads empty** (B179). - **A file read just as it reaches Drive no longer reads empty** (B179).
- **Stopping the daemon no longer takes 10 seconds more than it needs** when an idle upload
worker missed the stop (B180).
- **A file renamed just as its upload finished keeps its new name.** The mount no longer showed - **A file renamed just as its upload finished keeps its new name.** The mount no longer showed
the old name, and a second rename no longer failed with "No such file or directory" (B152, the old name, and a second rename no longer failed with "No such file or directory" (B152,
B172). B172).
+3 -1
View File
@@ -9,7 +9,7 @@ Tracks `docs/MILESTONE-3.0.0.md`. Update it with every commit that moves a phase
- **Phase 2**: local ids, inode = local id, one tree for every mount (2.10.0). - **Phase 2**: local ids, inode = local id, one tree for every mount (2.10.0).
- **Phase 3, first part**: mkdir, create, rename, unlink and rmdir on the mount are recorded - **Phase 3, first part**: mkdir, create, rename, unlink and rmdir on the mount are recorded
locally and sent from the queue; `"local_first": false` brings back the old path (5e0f30c). locally and sent from the queue; `"local_first": false` brings back the old path (5e0f30c).
- Fixes from the simulation and account runs since then: B127 and B141 to B179. - Fixes from the simulation and account runs since then: B127 and B141 to B180.
- **Sync issues, first cut** (§6.2): refusals sorted into issues, shown at once in the app, - **Sync issues, first cut** (§6.2): refusals sorted into issues, shown at once in the app,
tray and CLI; Export button and `pdfs sync export`; `pdfs sync issues`. tray and CLI; Export button and `pdfs sync export`; `pdfs sync issues`.
@@ -125,6 +125,8 @@ Tracks `docs/MILESTONE-3.0.0.md`. Update it with every commit that moves a phase
its stand-in and answered `EIO` (B177, fixed). A folder trashed while sync was its stand-in and answered `EIO` (B177, fixed). A folder trashed while sync was
paused kept a create below it as a trash: a paused drain worker had claimed it paused kept a create below it as a trash: a paused drain worker had claimed it
(B178, fixed). A file read as its create landed read empty (B179, fixed). (B178, fixed). A file read as its create landed read empty (B179, fixed).
The suite on 0f3c06f: a stop gave up on a drain worker that slept through its
wake-up (B180, fixed).
5. **Release**: 2.10.0 is tagged; the 2.9.0 tag still waits for the user's go-ahead. 5. **Release**: 2.10.0 is tagged; the 2.9.0 tag still waits for the user's go-ahead.
## After 3.0.0 ## After 3.0.0