perf(fuse): drain with 16 workers, and wake the idle ones when an op lands

crates/pdfs-fuse/src/drain.rs

@@ -363,10 +363,15 @@ impl Core {
             }
             // The access check let this attempt through, so any earlier run of
             // deferrals is over and must not be counted against the next one.
-            if matches!(outcome, Ok(DrainDisposition::Applied))
-                && let Err(error) = self.db.clear_op_access_deferral(op.id)
-            {
-                debug!(uid = %op.uid, %error, "clearing an access-deferral window failed");
+            if matches!(outcome, Ok(DrainDisposition::Applied)) {
+                if let Err(error) = self.db.clear_op_access_deferral(op.id) {
+                    debug!(uid = %op.uid, %error, "clearing an access-deferral window failed");
+                }
+                // What landed may have made other ops claimable: the files
+                // made in a folder that has its uid now, the next change to the
+                // same node. This worker only takes one of them; the others
+                // slept until the next idle poll.
+                self.wake_drain();
             }
             if let Err(e) = outcome {
                 let attempts = op.attempts + 1;

crates/pdfs-fuse/src/lib.rs

@@ -201,10 +201,13 @@ const UPLOAD_TIME_MEMORY: usize = 512;
 ///
 /// One meant a single 10 GiB upload blocked every queued rename, trash and
 /// small write behind it. Ordering only has to hold per node — enforced by the
-/// claim query, not by the thread count — so the rest is throughput. Kept small:
-/// these are uploads, and past a handful they compete for the same uplink and
-/// for the SDK's own per-request concurrency.
-const DRAIN_WORKERS: usize = 3;
+/// claim query, not by the thread count — so the rest is throughput. Most ops
+/// spend their time waiting: a create waits for Drive to list the node it made
+/// before it reads it back. Three workers left a thousand small creates taking
+/// 0.3 s each on a LAN and 2.5 s on Wi-Fi, one after another. Uploads cannot
+/// crowd each other out by number: the SDK caps the blocks in flight across
+/// the client.
+const DRAIN_WORKERS: usize = 16;
 
 /// How long a staged blob that names no node is kept before it is retired.
 ///

docs/ARCHITECTURE.md

@@ -189,7 +189,7 @@ Because Proton Drive does not support partial byte writes, modified files must b
    - Durably publishes the scratch data and authored-range sidecar into `staging` under a `{uid}-{millis}-{counter}` name. Temporary data and metadata are synced before atomic rename, and their directory is synced before the source is removed.
    - Transactionally queues or supersedes a pending database operation (`PendingOp`), so insertion failure cannot erase the previously acknowledged upload. For `OP_REVISION` ops, execution is debounced to give rapid follow-up writes (e.g. `aria2c` preallocation followed by writing) time to supersede the staged blob before network transmission. The debounce is adaptive (`Core::revision_debounce`): it starts at `DRAIN_REVISION_DEBOUNCE = 2s` and, once a node has been uploaded, widens toward how long that upload actually took, bounded by `DRAIN_REVISION_DEBOUNCE_MAX = 60s`. A file saved faster than it can be sent therefore supersedes in the queue rather than mid-upload.
    - Signals `Core::cancel_upload` for the node before touching the queue. A drain worker may already be reading the blob this write supersedes; the flag is read by the upload's `CountingReader`, which refuses the SDK's next block. `queue_trash` and `discard_queued_ops` do the same, because they unlink the blob outright.
-3. **Async Drain Threads (`run_pending_drain`):** `DRAIN_WORKERS = 3` background workers share the operations queue. They pick work up through `Db::claim_next_due_op`, which marks the row `claimed_at` in the same transaction that selects it and excludes any op whose uid another worker already holds — ordering only has to hold *per node*, and that exclusion is what guarantees it. Worker 0 additionally runs the queue's idle chores (LRU touch flush, `recover_fsynced_writes`). A claim is process-local state: the single-writer `flock` means one found at open belongs to a crashed run, and `Db::clear_op_claims` drops the lot so those ops are not invisible forever. Each worker handles revision uploads, resolves conflicts, and cleans up staging files. Upon landing a revision upload (`refresh_after_upload`), it rebaselines both still-queued ops (`rebaseline_pending`) and open write handles targeting the same node to prevent false self-conflict copies on subsequent writes.
+3. **Async Drain Threads (`run_pending_drain`):** `DRAIN_WORKERS = 16` background workers share the operations queue; one that lands an op wakes the idle ones, since it may have made others claimable. They pick work up through `Db::claim_next_due_op`, which marks the row `claimed_at` in the same transaction that selects it and excludes any op whose uid another worker already holds — ordering only has to hold *per node*, and that exclusion is what guarantees it. Worker 0 additionally runs the queue's idle chores (LRU touch flush, `recover_fsynced_writes`). A claim is process-local state: the single-writer `flock` means one found at open belongs to a crashed run, and `Db::clear_op_claims` drops the lot so those ops are not invisible forever. Each worker handles revision uploads, resolves conflicts, and cleans up staging files. Upon landing a revision upload (`refresh_after_upload`), it rebaselines both still-queued ops (`rebaseline_pending`) and open write handles targeting the same node to prevent false self-conflict copies on subsequent writes.
 
 ```mermaid
 sequenceDiagram
@@ -337,7 +337,7 @@ The background daemon relies on the following thread topology:
 2. **FUSE Workers Pool (11 threads, two lanes):** Bounded thread pool handling network operations. Split into 3 threads reserved for metadata (`lookup`, `readdir`) and 8 general threads that serve transfers (block reads) and fall back to metadata when no transfer is waiting. Reserved threads never accept a transfer — that is what keeps a directory listing from queuing behind saturated downloads (audit A6).
 3. **Control Threads (`pdfs-control`):** Listen on the Unix socket and admit at most 64 concurrent, timeout-bound handlers. Long-lived subscriptions hold one handler each, up to 8 (see §6).
 4. **Sync Engine Threads (`pdfs-sync`, `pdfs-sync-poll`):** Serialize sync passes. Wake on debounced inotify changes, the periodic poll, remote events, or a user request.
-5. **Drain Workers (`pdfs-drain-0` … `pdfs-drain-2`):** `DRAIN_WORKERS = 3` threads share the queue of staged writes and changes (`PendingOp`), with per-node ordering (§4), and retry failures with exponential backoff. Worker 0 also runs the queue's idle chores.
+5. **Drain Workers (`pdfs-drain-0` … `pdfs-drain-15`):** `DRAIN_WORKERS = 16` threads share the queue of staged writes and changes (`PendingOp`), with per-node ordering (§4), and retry failures with exponential backoff. Worker 0 also runs the queue's idle chores.
 6. **Conflict Sweep Thread (`pdfs-conflict-sweep`, optional):** Reconciles leftover `(sync-conflict …)` copies — 30 s warmup, then one pass every 5 minutes. A copy proven identical to its live sibling (equal size **and** equal `content_sha1`) is redundant; anything it cannot prove identical is surfaced to the activity feed and left alone. Governed by `AppConfig.conflict_sweep` / `PDFS_CONFLICT_SWEEP` (`SweepMode`): **report-only by default**, `off` skips the thread entirely, and only `enforce` lets it trash. Because it deletes, the enforcing path re-verifies revision id, size, digest, name, parent, queued ops and open handles immediately before acting rather than trusting its own pass-start snapshot. See `docs/BUGS.md` B69 and B71.
 7. **Remote Event Tasks (Tokio):** `run_event_sync` follows Proton's event stream for the Drive volume and applies each change to every mounted inode space, invalidating kernel entries as it goes. The cursor is persisted after every batch, so changes made while the daemon was stopped are applied on the next start. `run_photos_event_sync` follows the photos volume separately; it only drops gallery rows or the timeline's freshness stamp, since photos are not part of the mount.
 8. **Event Sampler (`pdfs-events`):** Publishes status, transfer and sync-progress changes to subscribed front-ends (§6).

docs/CHANGELOG.md

@@ -32,6 +32,8 @@ by its local id, in the queue, the stored tree and pins alike.
   a new file, a rename or a delete is recorded locally and the call returns at once; the queue
   sends it, as it does offline. `"local_first": false` in `config.json` brings back the old way
   for this release.
+- **The queue sends up to 16 changes at once instead of 3.** A thousand new files in a hundred
+  folders reach Drive about three times as fast on a LAN, and four times as fast on Wi-Fi.
 
 ### Fixed
 - **A file made under the name of a file whose upload is still going out keeps its name.** It

docs/MILESTONE-3.0.0-STATUS.md

@@ -55,13 +55,21 @@ Tracks `docs/MILESTONE-3.0.0.md`. Update it with every commit that moves a phase
            the uid Drive knows: ops are matched by lid in the database (6b), `hidden` and
            `creating` only ever hold remote uids, and `landed_row_uid` is the one place at the
            mount's edge where a stand-in a handle still holds becomes the real uid.
-   - [ ] An executor that runs ops in dependency order, many at once.
+   - [x] An executor that runs ops in dependency order, many at once: 16 drain workers instead
+         of 3, and a worker that lands an op wakes the idle ones, which slept through the ops it
+         made claimable. The claim already keeps the order: a create waits for its parent's
+         uid, a node's ops go one at a time. Ops stay blocking calls on worker threads: each
+         one spends its time waiting on Drive, so tasks would buy nothing threads do not.
 3. **Speed (§7)**: run independent ops in parallel, batch trashes and moves; measure 1,000 files
    and 100 folders on Wi-Fi against LAN (target: within 10 %).
    - [x] The measurement: `a_thousand_files_drain_as_fast_on_wifi_as_on_lan` in `sim/daemon.rs`,
          run with `PDFS_SIM_MEASURE=1`. It prints the times and asserts nothing yet.
    - [x] Baseline on eb1c105, three drain threads: the syscalls take 3.7 s on LAN and 4.3 s on
          Wi-Fi; the drain takes 138 s on LAN and 938 s on Wi-Fi, 6.8 times as long.
+   - [x] Sixteen workers: the syscalls take 5.2 s on LAN and 5.0 s on Wi-Fi, which meets the
+         target; the drain takes 47 s on LAN and 247 s on Wi-Fi. Most of a create on Wi-Fi is
+         its read-back waiting for Drive to list the new node (250 ms, then 1 s), so batching
+         (step 8) helps trashes and moves, not creates.
 4. **Done-when checks**:
    - [ ] Simulation passes on the Wi-Fi profile: `one_client_on_a_slow_link` failed in CI on
          seeds 2 and 3. Two bugs (B127, B151) and a harness gap: a settle did not read the log, so