fix(core,fuse): un-park abandoned creates, retire orphan staged bytes, serialise refreshes

Four faults found by reading the journal for issues that were never fixed,
tracked as B94-B97 in docs/BUGS.md.

A create parked behind a temp-file rename (B70) had no exit but that rename,
so 43 uploads sat in the queue for weeks holding their staged bytes. The
primary drain worker now sweeps parked creates while the queue is idle: one
whose node is gone is discarded, and one older than PARK_EXPIRY_MS that nobody
holds open is un-parked rather than dropped, since staged bytes are the only
copy of the write. An open node stays parked however old it is.

.tmp and .temp leave TRANSIENT_SUFFIXES. The list served two mechanisms with
different consequences: ignoring a name during mirror reconcile is reversible,
parking a create is not, and a Takeout index.tmp is a final name that nothing
will ever rename.

Staged blobs that match no queued op and name no node were kept forever and
warned about once per blob per mount (95 blobs, ~445 MiB, oldest two months).
Blobs past STAGING_ORPHAN_RETAIN are retired; the rest are one aggregated
warning.

Refresh control requests now run one per scope. Every scope is individually
cheap, but duplicates contend for the database lock the running refresh holds,
which left five in flight for twenty-three minutes. request_kind names the
scope so the report says which view wedged. The photo metadata pass also stops
at shutdown instead of running into the cancelled runtime and logging one
"skipping undecryptable photo" per node.
This commit is contained in:
nvrl committed 2026-09-19 20:29:48 +02:00
1 parent 6cc8f72bab
commit 860d29cfe8
14 files changed
+712 -68

No files matched your search

+2 -2
View File
@@ -56,8 +56,8 @@ pub use local::LocalFileHit;
pub use maintenance::{DbStats, VacuumOutcome};
pub use nodes::{PublishedSharedRoot, SearchHit, StoredNode};
pub use ops::{
AttachedBlob, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, PARK_UNTIL,
PendingCounts, PendingOp, RenameMeta, op_supersedes,
AttachedBlob, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH,
PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts, PendingOp, RenameMeta, op_supersedes,
};
pub use photos::{StoredPhoto, THUMB_HAVE, THUMB_NONE, THUMB_UNKNOWN, TimelineRow};
pub use pins::PinRow;
+53
View File
@@ -84,6 +84,25 @@ pub const LOCAL_VOLUME: &str = "local";
/// stall+resume, fork a `(sync-conflict)` copy (docs/BUGS.md B70).
pub const PARK_UNTIL: i64 = 8_000_000_000_000;
/// How long a create may stay parked before the park is treated as permanent
/// and the bytes are let through anyway (ms).
///
/// The park has exactly one exit: a rename from the transient name to the
/// finished one. Every writer that reaches that rename does so in seconds — an
/// atomic-write scratch file immediately, a download when it completes. A park
/// still standing an hour later belongs to a writer that will never perform
/// that rename: an editor that deletes its swap file instead, a download the
/// user abandoned, or a file that simply *is* named `report.tmp` and is never
/// going to be called anything else.
///
/// Before this existed such a row sat at [`PARK_UNTIL`] for the life of the
/// database, holding the only copy of the file's bytes in `staging/` and
/// counting against the queue while promising progress that could not come. The
/// sweep un-parks it instead of dropping it: bytes the user can see in the mount
/// are bytes the user expects on Drive, and invariant 1 of the drain (a staged
/// blob is dropped only once its op has landed) admits no other answer.
pub const PARK_EXPIRY_MS: i64 = 60 * 60 * 1000;
/// A mutation that has been accepted locally but not yet performed against the
/// API — the durable half of the write-back queue (offline.md Phase 3).
///
@@ -661,6 +680,40 @@ impl Db {
Ok(rows)
}
/// Every parked create/mkdir, oldest first, for the park sweep.
///
/// Parked rows are invisible to [`next_due_op`](Self::next_due_op) by
/// construction, so nothing else in the drain ever looks at them. This is
/// the one query that does.
pub fn parked_create_ops(&self) -> Result<Vec<PendingOp>> {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
attempts, last_error, next_attempt_at
FROM pending_op
WHERE next_attempt_at >= ?1 AND kind IN (?2, ?3)
ORDER BY created_at",
)?;
let rows = stmt
.query_map(params![PARK_UNTIL, OP_CREATE, OP_MKDIR], |r| {
Ok(PendingOp {
id: r.get(0)?,
kind: r.get(1)?,
uid: r.get(2)?,
parent_uid: r.get(3)?,
name: r.get(4)?,
blob_path: r.get(5)?,
meta_json: r.get(6)?,
created_at: r.get(7)?,
attempts: r.get(8)?,
last_error: r.get(9)?,
next_attempt_at: r.get(10)?,
})
})?
.collect::<std::result::Result<Vec<_>, _>>()?;
Ok(rows)
}
/// How many queued ops there are of each kind, for `Response::Status`.
///
/// Split because "3 uploads queued" has to mean three files whose bytes are
+51
View File
@@ -3049,6 +3049,57 @@ fn a_parked_transient_create_stays_off_the_drain_until_finalized() {
assert_eq!(due.blob_path.as_deref(), Some("/staging/blob1"));
}
/// The park sweep's query: parked creates are the rows nothing else in the drain
/// can see, so it has to find them by the sentinel and report how long each has
/// been parked. Ordinary queued work is not its business.
#[test]
fn parked_creates_are_listed_with_the_age_the_sweep_judges_them_by() {
let db = Db::open_in_memory().unwrap();
let parent = uid("root").to_string();
let parked = uid("swap").to_string();
let due = uid("ordinary").to_string();
for (node, name, next_attempt_at, created_at) in [
(&parked, ".notes.txt.swp", PARK_UNTIL, 1_000),
(&due, "notes.txt", 0, 2_000),
] {
db.enqueue_op(&PendingOp {
id: 0,
kind: OP_CREATE.to_string(),
uid: node.clone(),
parent_uid: Some(parent.clone()),
name: Some(name.to_string()),
blob_path: Some(format!("/staging/{name}")),
meta_json: None,
created_at,
attempts: 0,
last_error: None,
next_attempt_at,
})
.unwrap();
}
let rows = db.parked_create_ops().unwrap();
assert_eq!(rows.len(), 1, "only the parked create is swept");
assert_eq!(rows[0].uid, parked);
assert_eq!(rows[0].created_at, 1_000, "the park's age is readable");
assert_eq!(
rows[0].blob_path.as_deref(),
Some("/staging/.notes.txt.swp")
);
// What the sweep does with an expired park: un-park it, and it drains like
// any other create rather than waiting for a rename that is not coming.
assert!(db.set_create_hold(&parked, false).unwrap());
assert!(
db.parked_create_ops().unwrap().is_empty(),
"an un-parked create is no longer the sweep's business"
);
assert_eq!(
db.next_due_op(1_000 + PARK_EXPIRY_MS).unwrap().unwrap().uid,
parked
);
}
/// Backoff still gates: nothing is returned before an op is due.
#[test]
fn next_due_op_respects_backoff() {
+23 -9
View File
@@ -45,18 +45,30 @@ pub const DEFAULT_IGNORE_PATTERNS: &[&str] = &[
"Thumbs.db",
];
/// Suffixes of files that are, by convention, incomplete or throwaway while they
/// wear the name: a download in flight, an editor's swap or backup, an atomic
/// write's scratch copy. On an on-demand mount these should never seal a remote
/// revision — the app renames the finished file to its real name, and only that
/// belongs on Drive. See [`is_transient_name`] and docs/BUGS.md B70.
/// Suffixes that mean a file is incomplete *by construction*: only the program
/// writing it gives it this name, only while the write is in flight, and the
/// finished article always arrives under a different one. On an on-demand mount
/// these should never seal a remote revision — the app renames the finished file
/// to its real name, and only that belongs on Drive. See [`is_transient_name`]
/// and docs/BUGS.md B70.
///
/// `.tmp`/`.temp` are deliberately **not** here, though they are in
/// [`DEFAULT_IGNORE_PATTERNS`]. The two lists answer different questions. Skipping
/// a `report.tmp` during a mirror reconcile costs nothing — the file stays on
/// disk, and a later pass picks it up if the rules change. Parking one on the
/// on-demand mount is irreversible in practice: the only exit is a rename to a
/// finished name, and a file that is simply *called* `index.tmp` — a static-site
/// template, a tool's output — never gets one. It then never reaches Drive at
/// all, silently, while its bytes pin `staging/`. That is what a real Takeout
/// tree of `index.tmp`/`preview.tmp` gallery templates did: 43 creates parked
/// forever. The generic-scratch case they were added for renames within
/// milliseconds, so it never needed the park; the download suffixes below, which
/// stay in flight for minutes, are what B70 was actually about.
const TRANSIENT_SUFFIXES: &[&str] = &[
".crdownload", // Chromium/Brave/Edge
".part", // Firefox, wget, aria2c
".partial", // some downloaders / older IE
".download", // Safari / macOS
".tmp", // generic scratch
".temp", // generic scratch
".swp", // vim swap
".swx", // vim swap (secondary)
];
@@ -319,8 +331,6 @@ mod tests {
"movie.mkv.part",
"big.iso.partial",
"photo.jpg.download",
"export.xml.tmp",
"notes.TEMP",
".notes.txt.swp",
"doc.txt~",
".~lock.report.odt#",
@@ -340,6 +350,10 @@ mod tests {
"archive.partly.zip", // ".partly" is not ".part"
"template.dotx",
"readme",
// A file that is *named* `.tmp` and never renamed: parking it would
// keep it off Drive for good.
"index.tmp",
"notes.TEMP",
] {
assert!(!is_transient_name(name), "{name} should not be transient");
}