fix(core,fuse): keep replaced and vanished files from failing saves, reads and syncs

Fixes from the account acceptance runs:

- B104: a reader open across a rename-over keeps working; the replaced
  node is unlinked like any other open file instead of forgotten.
- B105: a drain worker holding a revision withdrawn by an unlink no
  longer uploads it as a recovered-… conflict copy (Db::op_exists).
- B106: an open victim's queued ops are withdrawn at trash time, so a
  repeated editor save no longer fails with AlreadyExists; the staged
  blob goes with the last close.
- B107: a remote read gives up after 120 s with EIO instead of hanging
  its caller; the timer is built inside the runtime.
- B108: a mirror upload whose local file was deleted after the scan is
  skipped rather than counted as an error.

The acceptance script reports the failing line for non-assert errors,
ignores queue ops that predate the run, purges with delete-forever
first, and diagnoses a malformed SQLite database against a local replay.
This commit is contained in:
nvrl committed 2026-09-29 16:20:51 +02:00
1 parent 7b67db5ced
commit a2992b3b9d
11 files changed
+579 -34

No files matched your search

+128
View File
@@ -12,6 +12,134 @@ Conventions:
---
## B108 — A file deleted during a mirror pass puts the sync folder in `error`
**Status:** Fixed (unverified).
**Found:** 2026-09-29, by the account run of `scripts/fuse-acceptance.sh --quick`. It stopped with
`sync folder … entered error` before the managed-move phase. At the same time the journal
showed `sync: step failed; continuing error=open …/workloads/tree.tar: No such file or directory`
for eleven files. The contract suite had removed its `workloads` tree while the mirror pass
was still uploading it.
**Where:** `crates/pdfs-fuse/src/sync.rs` (`upload_new`, `upload_revision`, `open_to_upload`).
**Cause.** A pass plans its uploads from a scan and runs them later. When a planned file was
deleted in between, opening it failed with `NotFound`. That counted as a failed step, so the pass
ended in `error` and logged "item(s) failed; will retry" for files nobody wanted any more.
**Fix.** An upload whose local file is gone when it starts is skipped (`Applied::Vanished`) and
does not count as an error. The next pass sees the deletion and handles it like any other.
Other open failures are still errors.
**Also open from the same run.** "application workloads" on My files failed with
`database disk image is malformed` at the `SELECT` after the insert. The daemon logged nothing
for it. The case now reports the step that failed, the database and journal files beside it,
and which pages differ from a local replay of the same statements.
## B107 — One read on a stalled connection hangs the reading program for half an hour
**Status:** Fixed (unverified).
**Found:** 2026-09-29, by the acceptance case "application workloads" on My files, which hit its
180 s timeout. The journal showed `a fuse worker has held the same job for a long time
worker=pdfs-fuse-10 job=read` from `age_secs=124` up to `age_secs=2059`. Event polls in the
same window failed with `HTTP transport error` after 91-103 s.
**Where:** `crates/pdfs-fuse/src/reads.rs` (`Core::read_range_remote`, `READ_FETCH_TIMEOUT`).
**Cause.** The SDK bounds each request (30 s for the API, 300 s for a storage block) and retries
failed ones, but nothing bounds a whole read. A read can need a block table, several blocks and a
repair, each with its own retries. On a connection that stalled without dropping, one `read`
held its FUSE worker for about 34 minutes, and the program that issued it could not be
interrupted for that time.
**Fix.** The network part of a read now has a 120 s deadline. When it passes, the read fails
with `EIO`, the file's revision reader is dropped so the next read starts fresh, and blocks that
already arrived stay cached for the retry. A single read is at most a few 4 MiB blocks, so this
only fails a read on a link slower than about 70 KB/s. A block repair that has to download a
large file whole can also hit the deadline on a slow link; pinning the file avoids that.
The first version of this fix built the timer on the FUSE worker thread, outside the runtime.
Every read that went to the network panicked with `there is no reactor running`, and the kernel
got `EIO`. The next account run failed 11 cases on My files this way. The timer is now built
inside `block_on`, and a unit test runs that pattern from a thread without a runtime.
---
## B106 — Saving over a file that was just read fails with "already exists"
**Status:** Fixed (unverified).
**Found:** 2026-09-29, by the acceptance case "application workloads" on an on-demand folder,
after installing the B104 fix: `OSError: [Errno 5]` from `os.replace` in the editor-save loop,
at the third save. The journal had `rename failed … AlreadyExists (http 422): A file or folder
with name already exists` from `rename_node`.
**Where:** `crates/pdfs-fuse/src/lib.rs` (`Core::remove_replaced`,
`Core::withdraw_queued_ops`), `crates/pdfs-fuse/src/filesystem.rs` (`trash_child`,
`finish_release`), `crates/pdfs-fuse/src/drain.rs` (`Core::drain_revision`).
**Cause.** A regression from B104. The save loop writes a temporary file, renames it over the
document, and reads the document back. `close()` returns before the daemon sees the release, so
at the next save the replaced file still counts as open. B104 then left its queued upload in
the queue until the final close, and the drain could send it after the file had been trashed.
The next rename onto the name was refused. `unlink` of an open file had kept its queued upload
the same way.
**Fix.** A file trashed or replaced while open now loses its queued ops at once, and an upload
already under way is cancelled. Only the staged bytes stay, because the open handle reads them,
and they are deleted at the final close. A drain worker that claimed the op before it was
withdrawn checks that the op still exists before it sends any bytes.
---
## B105 — A file deleted while its upload is in flight comes back as a conflict copy
**Status:** Fixed (unverified).
**Found:** 2026-09-29, in the daemon journal of an acceptance run on the account. During the
SQLite workload two `queued write conflicts; keeping a conflict copy` warnings named files called
`recovered-XrPUSW5W` and `recovered-JbZM_pWK`, with the reason `the file was trashed remotely`.
Each was followed by `pending upload failed; will retry` and `No such file or directory`.
**Where:** `crates/pdfs-fuse/src/drain.rs` (`Core::drain_revision`),
`crates/pdfs-core/src/db/ops.rs` (`Db::op_exists`).
**Cause.** SQLite writes, syncs and deletes its journal file within milliseconds. The sync
queues an upload of the journal. The delete trashes the node and drops that queued upload, but a
drain worker that already holds the op carries on. It then asks the server about the node,
finds it trashed, and treats the trash as another device's change: it tries to upload the bytes
as a conflict copy. The tree has already forgotten the node, so the copy is named `recovered-…`
and placed in the Drive root. Here the blob was already gone and the attempt failed, but with
the blob still on disk the copy would land.
**Fix.** Before keeping a conflict copy, the drain checks that its op is still queued. When the
op is gone, the write was withdrawn by the delete, or superseded, while the worker held it, and
it is dropped.
---
## B104 — Reading a file that was replaced by a rename fails with `EIO`
**Status:** Fixed (unverified).
**Found:** 2026-09-29, by the acceptance case "rename patterns" on My files and on an on-demand
folder: `OSError: [Errno 5] Input/output error` at
`os.pread(reader, 64, 0)` after `os.replace(replacement, target)` with `reader` open on `target`.
The mirror folder, a local disk, passed.
**Where:** `crates/pdfs-fuse/src/lib.rs` (`Core::remove_replaced`).
**Cause.** A rename over an existing file first trashes that file, because Drive has no atomic
replace. That step forgot the replaced inode outright and evicted its cache and reader, even
when a handle was still open on it. The next read through that handle found no inode and
failed. `rename(2)`, like `unlink(2)`, keeps a replaced file readable through the handles
already open on it. The unlink path (`trash_child`) already did this, but the rename path
did not.
**Fix.** `remove_replaced` now handles an open victim as `unlink` does. It keeps the inode as
an unlinked entry, and leaves its cache and reader to the final close. That applies both to a
node the server knows and to one whose create is still queued. Its queued ops at first also
waited for the final close; B106 moved them back to the trash.
---
## B103 — One stuck trash refresh freezes the trash listing until the daemon restarts
**Status:** Fixed (unverified). The hang itself has not been reproduced on purpose.
+15
View File
@@ -31,6 +31,21 @@ scratch (user data in `staging/` and `recovery/` is never touched by this).
- **Removing a folder also cancels the pending uploads of the files inside it.** Before, an
edit that had not reached Drive yet kept retrying against the trash forever and kept its
staged copy on disk. `pdfs rm` now also drops what was queued for what it removes (B102).
- **A program reading a file keeps its data when another program replaces the file.** Saving
over a file by renaming a new copy onto it (what most editors and `os.replace` do) made
reads through handles already open on the old file fail with `Input/output error` (B104).
- **Short-lived files no longer come back as conflict copies.** A file deleted right after it
was written, such as a SQLite journal, could be uploaded as a `recovered-…` conflict copy in
the Drive root (B105).
- **Editors can save over a file several times in a row.** The fix for B104 could leave the
upload of a replaced file queued, and the next save onto the same name failed with
`Input/output error` (B106).
- **Deleting files during a mirror pass no longer marks the folder as failed.** A file removed
after the pass planned its upload was counted as an error, so the folder showed `error` until
a later pass (B108).
- **A read no longer hangs when the connection stalls.** A read that gets no answer from Drive
within two minutes now fails with `Input/output error` instead of blocking the program for
as long as half an hour (B107).
- **The trash no longer freezes after a server call hangs.** Until the daemon restarted, the
trash kept showing the same entries, and a folder just deleted could not be found there to
delete permanently (B103).