Compare commits

...
29 Commits
Author SHA1 Message Date
nvrl f6456829fb chore(release): 2.10.0 2026-10-03 21:57:23 +02:00
nvrl 327f538fed test(core): migrate a database 2.8.2 wrote, with ops in every state
A dump of a database the 2.8.2 build wrote, with queued ops due, parked,
failing, deferred for access and claimed, and creates queued below a
folder made offline. Its migration checks every column of the queue and
the sealed revisions, as the real-database test now does too.
2026-10-03 21:55:15 +02:00
nvrl ad8b225cfd refactor(fuse): one tree for every mount
On-demand sync folders kept their own State, which the drain, the
event applier and the sweeps had to visit one by one (B74). Every mount
now serves from the primary's tree. A session keeps only its root, which
the kernel knows as inode 1, and its notification channel; work that
changes the tree tells every mounted session. Size upgrades become
daemon-wide, and moves compare mounts by root instead of by state.
2026-10-03 16:53:31 +02:00
nvrl ea27f2bcd0 test(core): migrate a copy of a real database
An ignored test takes a copy of the database named by PDFS_MIGRATE_DB
with VACUUM INTO over a read-only connection, migrates it, and checks
that every queued op survives unchanged, every node keeps its row
number as its lid, and no node is lost or doubled.
2026-10-03 16:53:31 +02:00
nvrl e4062cc97b feat(fuse,core): number inodes by local id
A node's inode is now derived from its row's local id instead of a
per-run counter, so the same node keeps the same inode across restarts
and across every place that interns it. A lid handed out before the
row is stored reads as no node.
2026-10-03 16:41:26 +02:00
nvrl c34cc93c45 fix(fuse,core): keep a rename made while its create is on the wire
A rename of a node whose create is still queued rewrites the create's
target, but the drain had already copied the op. When the create
landed, the row was deleted unread and the rename was lost (B136).

finish_create now learns where the create was sent and where it landed,
and queues a rename of the real uid when the row has moved on since.
adopt_real_uid keeps the tree's name and folder while that rename is
queued. A landed folder's children also stop naming its placeholder as
their parent in memory, which put the placeholder back on their rows.
2026-10-03 16:20:00 +02:00
nvrl 7e3df7602d feat(core): give every node row a local id that never changes
First step of phase 2 of docs/MILESTONE-3.0.0.md. Schema 36 rebuilds
`nodes` with `lid INTEGER PRIMARY KEY AUTOINCREMENT`. Each row takes its
old rowid as its lid, so the search index, keyed by rowid, stays valid.
AUTOINCREMENT keeps a deleted row's lid from coming back. `uid` stays
unique and becomes nullable, for nodes that are not on Drive yet.

A landed create's row now takes the real uid in place, keeping its lid.
When a listing had already stored the real node, its content is kept
under the placeholder's lid, which the node was known by first.

Nothing reads the lid yet apart from `Db::lid_of`.
2026-10-03 15:10:51 +02:00
nvrl 61a0d31408 chore(release): 2.9.0 2026-10-03 15:00:38 +02:00
nvrl 644a47eb36 fix(ci): quote the simulation command so the workflow parses
The ': ' in 'sim:: --' read as a YAML mapping, so GitHub rejected
ci.yml and ran no job at all since 2c46d62.
2026-10-03 14:55:13 +02:00
nvrl cd90c566cd fix(fuse,core): renames across folders, replaced files, and writes after a create
Found by the seeded simulation runs (docs/BUGS.md).

- B122: a rename across folders goes to Drive as one move-multiple
  request, so a sibling holding the new name in the old folder no longer
  fails it with EIO. The drain uses the same request.
- B123: a second write open on a file just written no longer fails with
  EIO. An empty file counts as complete, our own renames, moves and
  online creates record their echo, a duplicated event is skipped, a
  rename keeps the file's mtime while its revision stays, and a landed
  upload is cached before open handles are rebased onto it.
- B124: a node put back after a failed replacing rename leaves `hidden`
  and its folder is listed again, so it shows in the mount.
- B131: a landed create is read back with the same short retry as an
  online create, and its bytes are cached under the real uid before an
  open handle is repointed at it.
- B135: an online rename settles the tree through the state guard, so
  the database has both folders unlisted before the reply. A lookup
  right after it no longer rebuilds the destination without the file.

The simulation runs no longer tolerate B122 or B131.
2026-10-03 14:50:12 +02:00
nvrl 1c262eb601 refactor(fuse,core): keep queued-op preconditions in the database
Phase 1 of docs/MILESTONE-3.0.0.md, without a change in behaviour.

- Own sealed revisions live only in the own_sealed_rev table. The
  in-memory map in front of it is gone, so revision_conflict gives the
  same answer before and after a restart.
- A queued rename records the node's original name next to its original
  parent, and keeps both across renames that replace the row. The drain
  logs when Drive has the node neither where the rename found it nor
  where it takes it.

The SDK's NodeUpdated event carries no revision id, so echoes of our own
changes stay recognised by the self-change timer.
2026-10-03 14:49:58 +02:00
nvrl bf73a395e5 docs: record B132-B134 from the simulation runs
- B132: a folder whose mkdir is still queued is gone after a restart.
  The root's listing, served offline, was invalidated when the link
  came back, and the restart listed it on Drive without the queued
  folder.
- B133: a file deleted while its upload is in flight can still come
  back as a recovered copy. B105's check goes by op id, and SQLite gives
  the next op queued the id the delete freed.
- B134: a file deleted online while the drain checks its write is kept
  as a conflict copy. The unlink trashes the file on Drive before it
  drops the queued write, so B105's check still finds the op.
2026-10-03 09:07:56 +02:00
nvrl a5138639a8 test(fuse): delete files as they upload in the simulation runs
The runs did not find B105 with its fix reverted. Drive lists a trash
later than the round trips take, so the drain never saw its own
client's delete while it still held the write.

The FakeDrive can now hold the next read of one node on its way to
Drive. A new profile, one-client-deletes, writes a file Drive holds,
waits for the drain to read the node back, deletes the file, and lets
the read go on once Drive shows the trash. A conflict the drain keeps
because a client's own file was trashed now fails the run as a false
conflict. With B105's fix reverted, seeds 1 and 2 fail on it.

Outside the step, such a conflict is counted as B134 when the copy is
named after the file: the drain looked after an online unlink had
trashed the file on Drive but before it dropped the write. Seed 3 of
three-clients came up with it once.

Half the time the step makes the file again before the read goes on.
That is B133: the new file's write takes the op id the delete freed,
and B105's check finds it. The runs count it.

The runs also count B132, an ENOENT after a restart that restored
queued ops, in a seed that lost the link. Seed 4 of one-client-flaky
failed on it. The known-bug matchers now also see what was logged
before the call.
2026-10-03 09:07:56 +02:00
nvrl 988b78e810 docs: record the bugs the simulation runs found (B122-B131)
The first simulation runs found ten open bugs. They are documented here
and not fixed yet, as phase 0 of docs/MILESTONE-3.0.0.md changes no
behaviour. The runs count each one and end the seed where the mount
leaves the model behind.

- B122: a rename across folders fails with EIO when the source folder
  holds the new name.
- B123: a second write open on a file just written fails with EIO.
- B124: after a replacing rename fails, the file it was replacing is gone
  from the mount.
- B125: a folder made offline fails with EIO once its listing is
  invalidated.
- B126: a file created offline under a name an offline rename freed
  lands as a conflict copy.
- B127: a file created offline lands twice when the answer to its upload
  is lost.
- B128: a file created offline over one deleted offline can be lost.
- B129: a file deleted or replaced while its create is on the wire stays
  on Drive.
- B130: a read of a file that a remote change invalidates can hang the
  daemon. The event sync blocks a runtime worker in inval_inode on the
  read's page lock. Nothing then drives the runtime's timers, which the
  read's fetch is waiting for.
- B131: a write to a file whose create could not be read back lands as a
  conflict copy. The create's op is retired before the read-back, so a
  failed read-back leaves the tree on the placeholder's mtime and no
  revision id.
2026-10-03 07:47:02 +02:00
nvrl 2c46d62429 ci: run the simulation runs outside pull requests
A `simulation` job runs the seeded simulation runs of
docs/MILESTONE-3.0.0.md section 8.2 on pushes to main and manual runs. They
mount FUSE against the in-memory FakeDrive, so they need fuse3 but no
account. Like the acceptance suite they stay off pull requests. The job
has a 30 minute limit, and the runs' own watchdog turns a hang into a
failed seed well inside it.
2026-10-03 07:47:02 +02:00
nvrl 97743f17e3 test(fuse): add seeded simulation runs over the FakeDrive
The second half of the simulation harness in docs/MILESTONE-3.0.0.md
section 8.2. A run starts one or more real FUSE daemons on a shared
FakeDrive. Each client owns a folder `c<n>` that only it writes. A seed
then draws syscalls through the mounts, link flaps, restarts, pauses and
settles. Every syscall is checked against a model of the client's tree as
it happens. After the last step every link comes up and the run checks
no loss, POSIX behaviour, no false conflicts, convergence and liveness.

- sim::model is the POSIX model: the tree, what each call should answer,
  and how it changes the tree.
- sim::run has the profiles: one client on a good link, one on a flaky
  link with flaps and restarts, and three clients. PDFS_SIM_SEEDS sets
  how many seeds each runs, and PDFS_SIM_SEED replays one.
- sim::watchdog catches a call that never returns. It takes every
  thread's stack in the process, since no tracer can stop a thread
  blocked on a FUSE request. It then aborts the run's FUSE connections
  and writes the stacks next to the run's state.
- The daemons' logs at INFO and above are kept, so a known open bug is
  told apart from a new failure. Known bugs are counted, and the seed ends
  where the mount leaves the model behind. PDFS_SIM_KNOWN=fail fails on
  them instead.
- A restart retries opening the database while the last run still holds
  it.
- The FakeDrive restamps a node's modification time on a rename or move,
  as Drive does (B113).

The runs mount FUSE, so they are ignored by default. Run them with
`cargo test -p pdfs-fuse --lib sim:: -- --ignored --test-threads=1`.
Test-only; no behaviour change.
2026-10-03 07:47:01 +02:00
nvrl 3d10338807 test(fuse): run real FUSE daemons against the FakeDrive
mount() becomes a thin wrapper over mount_with(), which takes the
DriveApi to reach Drive through and a Host. The daemon's Host is
unchanged: stopped by SIGTERM or SIGINT, installed at the same point as
before, and indexing the user's home. A simulated daemon is stopped over
a channel and skips the home index, so several can share one test
process without walking the developer's home.

The account quota moves onto DriveApi. It was the only call outside the
trait that marks the link lost, so a simulated daemon would have gone
offline on it; the FakeDrive now also models a full account
(InsufficientQuota).

sim::daemon starts one daemon on its own state directory with a real
FUSE mount, and a smoke test (ignored; needs /dev/fuse) checks a local
write lands, a remote write arrives, and a write made offline lands once
the link is back.
2026-10-03 05:09:02 +02:00
nvrl d53ba1be21 test(fuse): add an in-memory FakeDrive with Drive's known faults
The first half of the simulation harness in docs/MILESTONE-3.0.0.md
section 8.1. FakeDrive is one shared server; each simulated daemon reaches
it through a FakeClient that implements DriveApi and carries its own link
state, latency, faults and an SDK-style entity cache.

The faults are the ones the account runs met: listings that lag (B118,
B48), renames and moves refused for a stale name hash because the cache
is only emptied by events (B111, B121), names that refuse a replace (B13,
B104, B106), events delayed, duplicated, reordered and echoed (B70, B105),
requests that stall, fail, or change Drive and lose their reply (B107,
B121), and non-uniform block sizes (B84, B85, B87).

Test-only; no behaviour change.
2026-10-03 04:51:35 +02:00
nvrl c73d5c9436 refactor(fuse): reach Drive through a DriveApi trait
`Core` held a concrete `ProtonDriveClient`, so no test could make Drive
answer late, answer stale or not answer. This is the first step of
phase 0 in docs/MILESTONE-3.0.0.md.

`DriveApi` (`crates/pdfs-fuse/src/drive.rs`) covers the calls the sync
core makes: listings, events, creates, uploads, downloads, renames,
moves, trashes and restores. `ProtonDriveClient` implements it by
forwarding. `Core::drive` holds it, and every call of those methods now
goes through it. Sharing, Photos, devices, invitations and public links
stay on `Core::client`.

The trait takes readers and writers as trait objects, and
`impl dyn DriveApi` puts the client's generic signatures back, so call
sites read the same. Block reads go through `RevisionRead` instead of
the SDK's `RevisionReader`, which only the SDK can construct.

No behaviour change.
2026-10-03 04:37:37 +02:00
nvrl 8d92c69279 fix(fuse): keep an upload pending until its bytes are cached (B119)
The drain retired an upload's op first, and only then read the node back
and linked the staged blob into the content cache. For that one round
trip, the status reported an empty queue for a file that wasn't cached
yet, so anything that waited for an empty queue before reading the file
could still need the network. The account run of 2026-10-03 hit it over
a slow link: "regression B119" failed in the same second the upload
landed.

`Core::landing_uploads` now counts an upload from just before its op is
retired until its bytes are cached. The status adds it to
`pending_uploads`, and reads the queue before the count, so it always
sees the upload in one of them. A drop guard uncounts it on every exit
path.

The op still goes first. An op restored after a crash would re-send a
revision this daemon hasn't recorded as its own yet, and
`revision_conflict` would keep it as a conflict copy of the user's file.
2026-10-03 03:59:05 +02:00
nvrl e32959e40a fix(fuse): land a move right after a rename, and rename across folders quickly (B121)
The SDK caches each node, including the name hash it sends with the next
rename or move, and only an event evicts it. Renames and moves didn't
update that copy. So after a rename, the next change to the same node
went out with the old name hash, Drive refused it as out of date (422),
and the retry kept sending the stale hash until the event feed came
round, up to 10 seconds later.

- Every remote rename and move goes through `Core::rename_remote` or
  `Core::move_remote`. Both drop the SDK's copy of the node after a
  successful change and before each out-of-date retry, so the retry
  sends the current hash.
- A plain move was not retried at all, so a move right after a rename
  failed with EIO through the mount and with "out of date" from
  `pdfs move`. It now retries like a rename.
- The drain's combined move-and-rename also drops the SDK's copy once
  it lands.
- A rename that also changes folders through the mount replies only
  once both halves have landed. When the move fails, the file keeps
  its new name in the source folder, the tree is told so, and the
  rename reports EIO. A dropped link still queues the change.

B111's cause in docs/BUGS.md is corrected to the same SDK cache. The
acceptance case "regression B121" renames and moves a file at once
through the mount and through `pdfs`, and checks names and bytes.
2026-10-03 03:58:38 +02:00
nvrl 0b27c884a7 test(acceptance): turn B114 to B117 into regressions and ride out a dropped link
New live cases:

- regression B48, B80, B94, B95, B98, B102 and B111 replay the bugs
  their entries describe. B48 and B111 also run against the reference.
- regression B114 to B117 replace the open B114 and B115 cases, and
  fail with `check` when the bug comes back.
- regression B119: files written through the mount stay cached once
  they are uploaded.
- regression B86: after a mirror folder switches to on-demand, the
  mount lists what the mirror uploaded just before.

`--journal-check` also counts three warnings as errors: a conflict
copy (B113), a FUSE job held for a long time (B110) and an authority
re-read because the local tree lacked it (B117, B120).

A timeout no longer ends a target when the mount still answers a stat
and a listing. Cases that make many round trips get twice the limit.

A write is queued when the kernel releases the file, after close(2)
has returned, so cases now wait for the ops they expect
(`Daemon.queued`). "regression B102" saw one revision of two because
it read the queue once.

On flaky Wi-Fi, the run crashed twice on a read that the daemon
correctly failed with EIO once its read budget ran out. The end-of-run
read-back and the mode matrix's file checks now record a result
instead. The mode matrix tries such a read once more after the daemon
is back online, and notes it.
2026-10-03 02:10:49 +02:00
nvrl 37c04f1e69 fix(fuse,core): fix B114 to B120 from the account runs
- B114: `pdfs rm`, `pdfs rename` and `pdfs move` tell each kernel
  session that the old name is gone, so it stops resolving at once
  instead of for up to 30 seconds (`Core::forget_everywhere`,
  `Core::relocate_everywhere`).
- B115: `pdfs ls` and the app's file browser wait up to 10 seconds for
  the listing's size fetch, as `stat` does, so they show a file's real
  size instead of its encrypted one.
- B116: trashing a folder also drops the creates queued in its
  subfolders. A parked one stayed queued with its bytes, and a due one
  would have landed in the root.
- B117: `pdfs move` and `pdfs rename` move the node in each mount
  instead of forgetting it, so its queued upload keeps its path in
  `pdfs sync queue` and is not put off once after the resume.
- B118: when a create or mkdir reads back nothing, the mount asks again
  and then queues it as it does offline, instead of failing with ENOENT
  for something Drive has made.
- B119: a landed upload keeps its staged bytes as the file's cached
  content, unless it is larger than a sixteenth of the cache, so
  reading a file just written does not need the network.
- B120: a landed folder's row passes to its real uid in the same
  transaction, so a file queued inside it is not put off for five
  seconds.

docs/BUGS.md records B117 to B120. The account run of 2026-10-03
verified B114 to B117 and B119 in My files.
2026-10-03 02:10:23 +02:00
nvrl 625776081b test(acceptance): cover B113, B88 and the closed pipe, and report more readably
New live cases:

- regression B113: files renamed and moved while sync is paused upload
  without a conflict copy.
- regression B88: pdfs mkdir, rename and rm by local path, the
  mount-root guard, a path outside every mount, and the refusal in a
  mirrored folder.
- pdfs output into a closed pipe exits 0 and prints nothing on stderr.
- open B114 and open B115 check two bugs that are still open. While
  one reproduces, its case ends as "known", which is listed but does
  not fail the run.

The console output names the pdfs version, the storage and the daemon
state of each target, and prints one aligned line per case, with its
details beneath. A failed case points to its docs/BUGS.md entry. The
run ends with a summary table, the slowest cases, the known issues
and the numbered failures. A queue wait now fails at once on a parked
op instead of waiting out its timeout.

docs/BUGS.md records B114, B115 and B116, which the first live run
found, and marks B113 and B88 as verified live.
2026-10-02 22:16:36 +02:00
nvrl 8cde79b8e3 fix(cli): stop quietly when stdout is closed
`pdfs ls | head` ended in a "Broken pipe" panic once `head` had its
lines, because std's `println!` panics on a failed write.

The CLI now shadows `println!` and `print!` with versions that exit 0 on
a closed pipe and report any other write error, such as a full disk,
with exit 1. Restoring the default SIGPIPE would also stop the panic,
but the same binary runs the daemon, which must not die when a socket's
peer goes away.
2026-10-02 21:16:34 +02:00
nvrl eb53d122cc fix(fuse,cli): let rm, rename and mkdir take paths in on-demand folders
`pdfs rm ~/Downloads/x` answered "is not under the mountpoint" while
`pdfs move` took the same path: the Delete, Rename and CreateFolder
handlers still read paths against the primary mount only (B88).

They now go through `route_change`, the router ResolveConflict already
used, renamed from `route_conflict`. It answers the request in the
inode space of the mount that owns the path.

An on-demand folder's root has a parent on Drive, so `pdfs rm
~/Downloads` would have trashed the whole synced folder from under its
mount. rm and rename refuse a path that names a mount's root.
2026-10-02 19:54:37 +02:00
nvrl c64252193f fix(fuse): keep the revision id of a file created through the mount
The write handle a create opens was given no revision id, so the drain's
conflict check fell back to (mtime, size). Drive advances a node's mtime
on a move or rename, so a new file moved before its first upload drained
landed as a (sync-conflict) copy next to its own empty original. Take the
minted node's revision id, and rebase handles still open on a queued
create onto the node it landed as (B113).
2026-10-02 19:25:33 +02:00
nvrl cc70a21c05 fix(aur): version -bin source filenames
makepkg reuses a cached source file of the same name, so the unversioned
proton-drive.service from an older build failed the 2.8.2 checksum. Carry
$pkgver in every file fetched from the tagged tree, like LICENSE already
did, and bump pkgrel to 2.
2026-10-01 23:50:06 +02:00
Nils Pukropp 644995cc56 chore(aur): update packages to 2.8.2 2026-10-01 19:49:42 +00:00
51 changed files with 10409 additions and 1396 deletions

No files matched your search

+37
View File
@@ -64,6 +64,43 @@ jobs:
# Real Proton credentials and /dev/fuse deliberately do not run on pull # Real Proton credentials and /dev/fuse deliberately do not run on pull
# requests. Follow docs/TESTING.md on a dedicated test-account runner. # requests. Follow docs/TESTING.md on a dedicated test-account runner.
# The seeded simulation runs (docs/MILESTONE-3.0.0.md §8.2): real FUSE
# daemons against an in-memory Drive, so they need /dev/fuse but no account.
simulation:
if: github.event_name != 'pull_request'
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- uses: actions/checkout@v7
- name: Install native dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
pkg-config \
fuse3 \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
- name: Install minimum supported Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
cache: false
- uses: swatinem/rust-cache@v2
with:
shared-key: debug
save-if: false
- name: Simulation runs
run: 'cargo test -p pdfs-fuse --lib --locked sim:: -- --ignored --test-threads=1'
# Keeps the release-profile dependencies cached on main, where the release # Keeps the release-profile dependencies cached on main, where the release
# workflow can restore them; a tag cannot reuse the caches of earlier tags. # workflow can restore them; a tag cannot reuse the caches of earlier tags.
release-cache: release-cache:
Generated
+6 -4
View File
@@ -3474,7 +3474,7 @@ checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4"
[[package]] [[package]]
name = "pdfs-cli" name = "pdfs-cli"
version = "2.8.2" version = "2.10.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"clap", "clap",
@@ -3489,7 +3489,7 @@ dependencies = [
[[package]] [[package]]
name = "pdfs-core" name = "pdfs-core"
version = "2.8.2" version = "2.10.0"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"dbus-secret-service-keyring-store", "dbus-secret-service-keyring-store",
@@ -3513,8 +3513,9 @@ dependencies = [
[[package]] [[package]]
name = "pdfs-fuse" name = "pdfs-fuse"
version = "2.8.2" version = "2.10.0"
dependencies = [ dependencies = [
"async-trait",
"base64 0.23.1", "base64 0.23.1",
"fuser", "fuser",
"futures", "futures",
@@ -3528,12 +3529,13 @@ dependencies = [
"sha1 0.11.0", "sha1 0.11.0",
"tokio", "tokio",
"tracing", "tracing",
"tracing-subscriber",
"zip", "zip",
] ]
[[package]] [[package]]
name = "pdfs-gui" name = "pdfs-gui"
version = "2.8.2" version = "2.10.0"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"async-channel", "async-channel",
+1 -1
View File
@@ -3,7 +3,7 @@ resolver = "2"
members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"] members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"]
[workspace.package] [workspace.package]
version = "2.8.2" version = "2.10.0"
edition = "2024" edition = "2024"
license = "MIT" license = "MIT"
authors = ["Nils Pukropp <contact@narl.io>"] authors = ["Nils Pukropp <contact@narl.io>"]
+39 -3
View File
@@ -20,6 +20,39 @@ use pdfs_core::control::{
use pdfs_core::db::Db; use pdfs_core::db::Db;
use pdfs_core::service; use pdfs_core::service;
/// `println!` for this binary, shadowing std's, which panics when stdout is
/// closed: `pdfs ls | head` ended in a "Broken pipe" panic once `head` had its
/// lines. This one hands the failure to [`stdout_written`] instead.
///
/// Restoring the default `SIGPIPE` would also stop the panic, but the same
/// binary runs the daemon, which must not die when a socket's peer goes away.
macro_rules! println {
($($arg:tt)*) => {{
use std::io::Write as _;
$crate::stdout_written(writeln!(std::io::stdout(), $($arg)*))
}};
}
/// [`println!`] without the newline.
macro_rules! print {
($($arg:tt)*) => {{
use std::io::Write as _;
$crate::stdout_written(write!(std::io::stdout(), $($arg)*))
}};
}
/// Exit on a failed write to stdout. A closed pipe means the reader has all it
/// wanted, so that exit is a success.
fn stdout_written(result: std::io::Result<()>) {
if let Err(error) = result {
if error.kind() == std::io::ErrorKind::BrokenPipe {
std::process::exit(0);
}
eprintln!("Error: writing to stdout: {error}");
std::process::exit(1);
}
}
#[derive(Parser)] #[derive(Parser)]
#[command( #[command(
name = "pdfs", name = "pdfs",
@@ -285,7 +318,8 @@ enum Command {
}, },
/// Rename a file or folder via the running daemon. /// Rename a file or folder via the running daemon.
Rename { Rename {
/// File/folder path, inside the mountpoint or relative to it. /// File/folder path, inside My files or an on-demand synced folder, or
/// relative to the mountpoint.
path: PathBuf, path: PathBuf,
/// New name (a single path component). /// New name (a single path component).
new_name: String, new_name: String,
@@ -304,12 +338,14 @@ enum Command {
}, },
/// Trash a file or folder via the running daemon. /// Trash a file or folder via the running daemon.
Rm { Rm {
/// File/folder path, inside the mountpoint or relative to it. /// File/folder path, inside My files or an on-demand synced folder, or
/// relative to the mountpoint.
path: PathBuf, path: PathBuf,
}, },
/// Create a folder via the running daemon. /// Create a folder via the running daemon.
Mkdir { Mkdir {
/// Parent folder path, inside the mountpoint or relative to it. /// Parent folder path, inside My files or an on-demand synced folder,
/// or relative to the mountpoint.
parent: PathBuf, parent: PathBuf,
/// New folder name (a single path component). /// New folder name (a single path component).
name: String, name: String,
+19
View File
@@ -365,6 +365,14 @@ pub struct StagedWrite {
pub based_on: Option<Baseline>, pub based_on: Option<Baseline>,
} }
impl StagedWrite {
/// Whether `authored` ranges cover a file of `len` bytes, so the staged
/// bytes are its complete new content. An empty file has nothing to cover.
pub fn covers(authored: &[(u64, u64)], len: u64) -> bool {
len == 0 || authored == [(0, len)]
}
}
/// The identity of a remote revision, as far as we can observe it. /// The identity of a remote revision, as far as we can observe it.
/// ///
/// `revision_id` is the authoritative identity when present: the server assigns a /// `revision_id` is the authoritative identity when present: the server assigns a
@@ -2056,6 +2064,17 @@ mod tests {
NodeUid::new(VolumeId::from("vol"), LinkId::from(link)) NodeUid::new(VolumeId::from("vol"), LinkId::from(link))
} }
#[test]
fn an_empty_write_covers_the_file_and_a_gap_does_not() {
assert!(
StagedWrite::covers(&[], 0),
"an empty file has nothing to cover"
);
assert!(StagedWrite::covers(&[(0, 7)], 7));
assert!(!StagedWrite::covers(&[(0, 3), (4, 7)], 7));
assert!(!StagedWrite::covers(&[], 7));
}
/// A unique temp directory removed on drop; avoids a dev-dependency. /// A unique temp directory removed on drop; avoids a dev-dependency.
struct TempDir(PathBuf); struct TempDir(PathBuf);
impl TempDir { impl TempDir {
+249
View File
@@ -0,0 +1,249 @@
/* WARNING: Script requires that SQLITE_DBCONFIG_DEFENSIVE be disabled */
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE sync_state (key TEXT PRIMARY KEY, value TEXT);
INSERT INTO sync_state VALUES('schema_version','35');
CREATE TABLE nodes (
uid TEXT PRIMARY KEY,
parent_uid TEXT,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER,
mtime INTEGER NOT NULL,
revision_hash TEXT,
trashed INTEGER NOT NULL DEFAULT 0
, node_json TEXT, listed INTEGER NOT NULL DEFAULT 0, path TEXT);
INSERT INTO nodes VALUES('vol~root',NULL,'My Files',1,0,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"root"},"parent_uid":null,"kind":"Folder","name":"My Files","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'');
INSERT INTO nodes VALUES('vol~docs','vol~root','Docs',1,0,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"docs"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":"Folder","name":"Docs","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs');
INSERT INTO nodes VALUES('vol~report','vol~docs','report.txt',0,10,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"report"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":20,"active_revision_state":null,"active_revision_id":null,"claimed_size":10,"claimed_modification_time":null,"content_sha1":null}},"name":"report.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/report.txt');
INSERT INTO nodes VALUES('vol~old','vol~docs','old.txt',0,3,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"old"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":13,"active_revision_state":null,"active_revision_id":null,"claimed_size":3,"claimed_modification_time":null,"content_sha1":null}},"name":"old.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/old.txt');
INSERT INTO nodes VALUES('vol~shared','vol~docs','shared.txt',0,4,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"shared"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":14,"active_revision_state":null,"active_revision_id":null,"claimed_size":4,"claimed_modification_time":null,"content_sha1":null}},"name":"shared.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/shared.txt');
INSERT INTO nodes VALUES('local~dir','vol~root','Offline folder',1,0,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"dir"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":"Folder","name":"Offline folder","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',1,'Offline folder');
INSERT INTO nodes VALUES('local~inner','local~dir','inner.txt',0,5,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"inner"},"parent_uid":{"volume_id":"local","link_id":"dir"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":15,"active_revision_state":null,"active_revision_id":null,"claimed_size":5,"claimed_modification_time":null,"content_sha1":null}},"name":"inner.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Offline folder/inner.txt');
INSERT INTO nodes VALUES('local~part','vol~root','movie.part',0,7,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"part"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":17,"active_revision_state":null,"active_revision_id":null,"claimed_size":7,"claimed_modification_time":null,"content_sha1":null}},"name":"movie.part","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'movie.part');
CREATE TABLE cache_entries (
cache_key TEXT PRIMARY KEY,
size_bytes INTEGER,
last_accessed INTEGER,
is_pinned INTEGER NOT NULL DEFAULT 0
, kind TEXT NOT NULL DEFAULT 'blob');
CREATE TABLE pins (
uid TEXT PRIMARY KEY,
path TEXT NOT NULL,
recursive INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE local_files (
id INTEGER PRIMARY KEY,
path TEXT NOT NULL UNIQUE,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER NOT NULL DEFAULT 0,
mtime INTEGER NOT NULL DEFAULT 0,
scan_gen INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE photos (
uid TEXT PRIMARY KEY,
capture_time INTEGER NOT NULL,
name TEXT,
ratio REAL,
thumb_state INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL
, media_type TEXT, kind INTEGER NOT NULL DEFAULT 0, favorite INTEGER NOT NULL DEFAULT 0, content_hash TEXT, main_uid TEXT, group_key TEXT, resolved_at INTEGER, latitude REAL, longitude REAL, place_id INTEGER, similar_hash INTEGER);
CREATE TABLE trash (
uid TEXT PRIMARY KEY,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER NOT NULL DEFAULT 0,
mtime INTEGER NOT NULL DEFAULT 0
, parent_uid TEXT, trashed_at INTEGER);
CREATE TABLE device (
uid TEXT PRIMARY KEY,
share_id TEXT NOT NULL,
root_uid TEXT NOT NULL,
name TEXT NOT NULL,
created INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE sync_folder (
id INTEGER PRIMARY KEY,
local_path TEXT NOT NULL UNIQUE,
remote_uid TEXT NOT NULL,
remote_share_id TEXT NOT NULL,
mode TEXT NOT NULL DEFAULT 'mirror',
state TEXT NOT NULL DEFAULT 'idle',
last_sync INTEGER NOT NULL DEFAULT 0
, pending_mode TEXT);
CREATE TABLE sync_entry (
folder_id INTEGER NOT NULL,
rel_path TEXT NOT NULL,
remote_uid TEXT,
local_mtime INTEGER NOT NULL DEFAULT 0,
local_size INTEGER NOT NULL DEFAULT 0,
remote_hash TEXT,
remote_rev TEXT, local_mtime_ns INTEGER,
PRIMARY KEY (folder_id, rel_path)
);
CREATE TABLE activity (
id INTEGER PRIMARY KEY,
time INTEGER NOT NULL,
kind TEXT NOT NULL,
target TEXT NOT NULL,
detail TEXT NOT NULL DEFAULT '',
ok INTEGER NOT NULL DEFAULT 1
);
CREATE TABLE pending_op (
id INTEGER PRIMARY KEY,
kind TEXT NOT NULL,
uid TEXT NOT NULL,
blob_path TEXT,
meta_json TEXT,
created_at INTEGER NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
next_attempt_at INTEGER NOT NULL DEFAULT 0
, parent_uid TEXT, name TEXT, claimed_at INTEGER NOT NULL DEFAULT 0, access_deferred_since INTEGER NOT NULL DEFAULT 0);
INSERT INTO pending_op VALUES(1,'mkdir','local~dir',NULL,NULL,1700000000000,0,NULL,0,'vol~root','Offline folder',4611686018427387903,0);
INSERT INTO pending_op VALUES(2,'create','local~inner','/blobs/inner','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,0,NULL,0,'local~dir','inner.txt',0,0);
INSERT INTO pending_op VALUES(3,'create','local~part','/blobs/part','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,0,NULL,8000000000000,'vol~root','movie.part',0,0);
INSERT INTO pending_op VALUES(4,'revision','vol~report','/blobs/report','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,2,'network unreachable',1700000900000,NULL,NULL,0,0);
INSERT INTO pending_op VALUES(5,'rename','vol~shared',NULL,'{"original_parent_uid":"vol~docs","original_name":"shared.txt"}',1700000000000,0,NULL,1700000700000,'vol~root','shared-moved.txt',0,1700000100000);
INSERT INTO pending_op VALUES(6,'trash','vol~old',NULL,NULL,1700000000000,0,NULL,0,NULL,NULL,0,0);
CREATE TABLE IF NOT EXISTS 'local_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
INSERT INTO local_fts_data VALUES(1,x'000000');
INSERT INTO local_fts_data VALUES(10,x'00000000000000');
CREATE TABLE IF NOT EXISTS 'local_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'local_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'local_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
INSERT INTO local_fts_config VALUES('version',4);
CREATE TABLE share_access (
root_uid TEXT PRIMARY KEY,
access TEXT NOT NULL
CHECK (access IN ('owner', 'editor', 'viewer', 'unknown'))
);
CREATE TABLE mount (
id INTEGER PRIMARY KEY,
kind TEXT NOT NULL
CHECK (kind IN ('myfiles', 'device', 'shared')),
sync_folder_id INTEGER REFERENCES sync_folder(id) ON DELETE CASCADE,
share_root_uid TEXT,
local_path TEXT NOT NULL DEFAULT '',
root_uid TEXT NOT NULL DEFAULT '',
root_share_id TEXT NOT NULL DEFAULT '',
mode TEXT NOT NULL DEFAULT 'unknown',
access TEXT NOT NULL DEFAULT 'rw'
CHECK (access IN ('rw', 'ro')),
CHECK (
(kind = 'myfiles' AND sync_folder_id IS NULL AND share_root_uid IS NULL)
OR
(kind = 'device' AND sync_folder_id IS NOT NULL AND share_root_uid IS NULL)
OR
(kind = 'shared' AND sync_folder_id IS NULL AND share_root_uid IS NOT NULL)
)
);
CREATE TABLE albums (
uid TEXT PRIMARY KEY,
name TEXT NOT NULL,
photo_count INTEGER NOT NULL DEFAULT 0,
cover_uid TEXT,
last_activity INTEGER,
shared INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL
);
CREATE TABLE album_photos (
album_uid TEXT NOT NULL,
uid TEXT NOT NULL,
capture_time INTEGER NOT NULL,
name TEXT,
media_type TEXT,
kind INTEGER NOT NULL DEFAULT 0,
ratio REAL,
thumb_state INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL,
PRIMARY KEY (album_uid, uid)
);
CREATE TABLE IF NOT EXISTS 'nodes_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
INSERT INTO nodes_fts_data VALUES(1,x'083850');
INSERT INTO nodes_fts_data VALUES(10,x'000000000108080008010101020101030101040101050101060101070101080101');
INSERT INTO nodes_fts_data VALUES(137438953473,x'000000340430206669010204010366696c0102050103696c6501020601036c657301020701036d79200102020103792066010203040808080808');
INSERT INTO nodes_fts_data VALUES(274877906945,x'0000001a0430646f6302080201010201036f6373020803010103040b');
INSERT INTO nodes_fts_data VALUES(412316860417,x'0000008b04302e747803080801010d01032f72650306010106010363732f03060101040103646f630306010102010365706f03080301010801036f637303060101030202727403080501010a0103706f7203080401010901037265700308020101070202742e03080601010b0103732f7203060101050103742e7403080701010c0202787403080901010e040b0a0a0a0b0a0a0b0b0a0a0b');
INSERT INTO nodes_fts_data VALUES(549755813889,x'0000006b04302e747804080501010a01032f6f6c0406010106010363732f04060101040103642e7404080401010902026f63040601010201036c642e04080301010801036f6373040601010302026c640408020101070103732f6f0406010105010374787404080601010b040b0a0a0b090b0a0a0a');
INSERT INTO nodes_fts_data VALUES(687194767361,x'0000008c04302e747805080801010d01032f736805060101060103617265050804010109010363732f05060101040103642e7405080701010c02026f630506010102010365642e05080601010b010368617205080301010801036f63730506010103010372656405080501010a0103732f73050601010502026861050802010107010374787405080901010e040b0a0b0a0b090b0b0a0b0a0a');
INSERT INTO nodes_fts_data VALUES(824633720833,x'00000084043020666f060809010109010364657206080d01010d0103652066060808010108010366666c06080301010302026c6906080401010402026f6c06080a01010a0103696e6506080601010601036c646506080c01010c0202696e06080501010501036e652006080701010701036f666606080201010202026c6406080b01010b040b0b0b0b0a0a0b0b0a0b0b');
INSERT INTO nodes_fts_data VALUES(962072674305,x'000000da043020666f070601010901032e747807080701011601032f696e07060101100103646572070601010d010365206607060101080202722e07080501011403012f070601010e010366666c070601010302026c69070601010402026f6c070601010a0103696e65070601010603016e07080201011101036c6465070601010c0202696e070601010501036e6520070601010703017207080401011302026e6507080301011201036f6666070601010202026c64070601010b0103722e7407080601011502022f69070601010f0103747874070808010117040a0b0a0a0a0a080a09090a090a090a090a0a090b09');
INSERT INTO nodes_fts_data VALUES(1099511627777,x'0000005c04302e706108080701010701036172740808090101090103652e70080806010106010369652e08080501010501036d6f7608080201010201036f766908080301010301037061720808080101080103766965080804010104040b0b0b0b0b0b0b');
CREATE TABLE IF NOT EXISTS 'nodes_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
INSERT INTO nodes_fts_idx VALUES(1,x'',2);
INSERT INTO nodes_fts_idx VALUES(2,x'',2);
INSERT INTO nodes_fts_idx VALUES(3,x'',2);
INSERT INTO nodes_fts_idx VALUES(4,x'',2);
INSERT INTO nodes_fts_idx VALUES(5,x'',2);
INSERT INTO nodes_fts_idx VALUES(6,x'',2);
INSERT INTO nodes_fts_idx VALUES(7,x'',2);
INSERT INTO nodes_fts_idx VALUES(8,x'',2);
CREATE TABLE IF NOT EXISTS 'nodes_fts_content'(id INTEGER PRIMARY KEY, c0, c1);
INSERT INTO nodes_fts_content VALUES(1,'My Files','');
INSERT INTO nodes_fts_content VALUES(2,'Docs','Docs');
INSERT INTO nodes_fts_content VALUES(3,'report.txt','Docs/report.txt');
INSERT INTO nodes_fts_content VALUES(4,'old.txt','Docs/old.txt');
INSERT INTO nodes_fts_content VALUES(5,'shared.txt','Docs/shared.txt');
INSERT INTO nodes_fts_content VALUES(6,'Offline folder','Offline folder');
INSERT INTO nodes_fts_content VALUES(7,'inner.txt','Offline folder/inner.txt');
INSERT INTO nodes_fts_content VALUES(8,'movie.part','movie.part');
CREATE TABLE IF NOT EXISTS 'nodes_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
INSERT INTO nodes_fts_docsize VALUES(1,x'0600');
INSERT INTO nodes_fts_docsize VALUES(2,x'0202');
INSERT INTO nodes_fts_docsize VALUES(3,x'080d');
INSERT INTO nodes_fts_docsize VALUES(4,x'050a');
INSERT INTO nodes_fts_docsize VALUES(5,x'080d');
INSERT INTO nodes_fts_docsize VALUES(6,x'0c0c');
INSERT INTO nodes_fts_docsize VALUES(7,x'0716');
INSERT INTO nodes_fts_docsize VALUES(8,x'0808');
CREATE TABLE IF NOT EXISTS 'nodes_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
INSERT INTO nodes_fts_config VALUES('version',4);
CREATE TABLE own_sealed_rev (
uid TEXT PRIMARY KEY,
revision_id TEXT NOT NULL,
sealed_at INTEGER NOT NULL
);
INSERT INTO own_sealed_rev VALUES('vol~report','rev-1',1700000000000);
PRAGMA writable_schema=ON;
INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)VALUES('table','local_fts','local_fts',0,'CREATE VIRTUAL TABLE local_fts USING fts5(
name, path, content=''local_files'', content_rowid=''id'', tokenize=''trigram''
)');
INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)VALUES('table','nodes_fts','nodes_fts',0,'CREATE VIRTUAL TABLE nodes_fts USING fts5(
name, path, tokenize=''trigram''
)');
CREATE TRIGGER mount_sync_folder_insert
AFTER INSERT ON sync_folder
BEGIN
INSERT OR IGNORE INTO mount (kind, sync_folder_id)
VALUES ('device', NEW.id);
END;
CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_photos_seq ON photos(seq);
CREATE INDEX pending_op_uid ON pending_op(uid);
CREATE INDEX pending_op_parent ON pending_op(parent_uid);
CREATE INDEX cache_entries_lru ON cache_entries(kind, last_accessed);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_local_files_name_nocase
ON local_files(name COLLATE NOCASE);
CREATE UNIQUE INDEX mount_myfiles
ON mount(kind) WHERE kind = 'myfiles';
CREATE UNIQUE INDEX mount_device
ON mount(sync_folder_id) WHERE kind = 'device';
CREATE UNIQUE INDEX mount_shared
ON mount(share_root_uid) WHERE kind = 'shared';
CREATE INDEX idx_album_photos_seq ON album_photos(album_uid, seq);
CREATE INDEX idx_album_photos_uid ON album_photos(uid);
CREATE INDEX idx_photos_favorite ON photos(favorite, seq);
CREATE INDEX idx_sync_entry_remote_uid ON sync_entry(remote_uid);
CREATE INDEX idx_local_files_scan_gen ON local_files(scan_gen);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
CREATE INDEX idx_pending_op_claimed
ON pending_op(uid) WHERE claimed_at <> 0;
CREATE INDEX idx_trash_parent ON trash(parent_uid);
CREATE INDEX idx_photos_group ON photos(group_key);
CREATE INDEX idx_photos_unresolved ON photos(resolved_at);
CREATE INDEX idx_photos_place ON photos(place_id);
PRAGMA writable_schema=OFF;
COMMIT;
+54 -1
View File
@@ -9,7 +9,7 @@ use super::Db;
use crate::Result; use crate::Result;
/// Current schema version. Bump on every forward migration added below. /// Current schema version. Bump on every forward migration added below.
pub(super) const SCHEMA_VERSION: i64 = 35; pub(super) const SCHEMA_VERSION: i64 = 36;
impl Db { impl Db {
pub(super) fn migrate(&self) -> Result<()> { pub(super) fn migrate(&self) -> Result<()> {
@@ -332,6 +332,22 @@ impl Db {
tx.execute_batch(MIGRATION_V35)?; tx.execute_batch(MIGRATION_V35)?;
} }
} }
if current < 36 {
// Same guards as V26-V35.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('nodes') WHERE name = 'lid'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_nodes: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'nodes'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_nodes && !has_column {
tx.execute_batch(MIGRATION_V36)?;
}
}
tx.execute( tx.execute(
"INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1) "INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1)
ON CONFLICT(key) DO UPDATE SET value = excluded.value", ON CONFLICT(key) DO UPDATE SET value = excluded.value",
@@ -1059,3 +1075,40 @@ ALTER TABLE photos ADD COLUMN similar_hash INTEGER;
const MIGRATION_V35: &str = " const MIGRATION_V35: &str = "
ALTER TABLE trash ADD COLUMN trashed_at INTEGER; ALTER TABLE trash ADD COLUMN trashed_at INTEGER;
"; ";
/// Schema v36: every node row has a local id (`lid`), phase 2 of
/// docs/MILESTONE-3.0.0.md.
///
/// The `uid` was the row's key, so a node had no identity until Drive minted
/// one, and a create that landed moved its row to a new key. The `lid` is given
/// when the row is first written and never changes or comes back, which is what
/// `AUTOINCREMENT` adds over a plain rowid. It takes the old rowid, so the
/// search index, keyed by rowid since v15, stays valid. The `uid` becomes
/// nullable for a node that is not on Drive yet.
const MIGRATION_V36: &str = "
CREATE TABLE nodes_v36 (
lid INTEGER PRIMARY KEY AUTOINCREMENT,
uid TEXT UNIQUE,
parent_uid TEXT,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER,
mtime INTEGER NOT NULL,
revision_hash TEXT,
trashed INTEGER NOT NULL DEFAULT 0,
node_json TEXT,
listed INTEGER NOT NULL DEFAULT 0,
path TEXT
);
INSERT INTO nodes_v36
(lid, uid, parent_uid, name, is_dir, size, mtime, revision_hash, trashed, node_json, listed,
path)
SELECT rowid, uid, parent_uid, name, is_dir, size, mtime, revision_hash, trashed, node_json,
listed, path
FROM nodes;
DROP TABLE nodes;
ALTER TABLE nodes_v36 RENAME TO nodes;
CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
";
+3 -2
View File
@@ -56,8 +56,9 @@ pub use local::LocalFileHit;
pub use maintenance::{DbStats, VacuumOutcome}; pub use maintenance::{DbStats, VacuumOutcome};
pub use nodes::{PublishedSharedRoot, SearchHit, StoredNode}; pub use nodes::{PublishedSharedRoot, SearchHit, StoredNode};
pub use ops::{ pub use ops::{
AttachedBlob, FAILING_ATTEMPTS, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, AttachedBlob, CreateLanding, FAILING_ATTEMPTS, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME,
OP_TRASH, PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts, PendingOp, RenameMeta, op_supersedes, OP_REVISION, OP_TRASH, PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts, PendingOp, RenameMeta,
op_supersedes,
}; };
pub use photos::{StoredPhoto, StoredPlace, THUMB_HAVE, THUMB_NONE, THUMB_UNKNOWN, TimelineRow}; pub use photos::{StoredPhoto, StoredPlace, THUMB_HAVE, THUMB_NONE, THUMB_UNKNOWN, TimelineRow};
pub use pins::PinRow; pub use pins::PinRow;
+121 -9
View File
@@ -17,6 +17,8 @@ use super::utils::{
pub struct StoredNode { pub struct StoredNode {
pub node: Node, pub node: Node,
pub listed: bool, pub listed: bool,
/// The row's local id ([`Db::lid_of`]).
pub lid: i64,
} }
/// One full-text search match: the stored [`Node`] plus its mountpoint-relative /// One full-text search match: the stored [`Node`] plus its mountpoint-relative
@@ -289,7 +291,8 @@ impl Db {
let uid_str = parent_uid.to_string(); let uid_str = parent_uid.to_string();
let conn = self.read(); let conn = self.read();
let count: i64 = conn.query_row( let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM nodes WHERE parent_uid = ?1 AND trashed = 0", "SELECT COUNT(*) FROM nodes
WHERE parent_uid = ?1 AND trashed = 0 AND node_json IS NOT NULL",
params![uid_str], params![uid_str],
|row| row.get(0), |row| row.get(0),
)?; )?;
@@ -305,9 +308,11 @@ impl Db {
pub fn node_path(&self, uid: &str) -> Result<Option<String>> { pub fn node_path(&self, uid: &str) -> Result<Option<String>> {
let conn = self.read(); let conn = self.read();
let exists: Option<i64> = conn let exists: Option<i64> = conn
.query_row("SELECT 1 FROM nodes WHERE uid = ?1", params![uid], |r| { .query_row(
r.get(0) "SELECT 1 FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL",
}) params![uid],
|r| r.get(0),
)
.optional()?; .optional()?;
if exists.is_none() { if exists.is_none() {
return Ok(None); return Ok(None);
@@ -511,17 +516,17 @@ impl Db {
pub fn load_all(&self) -> Result<Vec<StoredNode>> { pub fn load_all(&self) -> Result<Vec<StoredNode>> {
let conn = self.read(); let conn = self.read();
let mut stmt = let mut stmt =
conn.prepare("SELECT node_json, listed FROM nodes WHERE node_json IS NOT NULL")?; conn.prepare("SELECT node_json, listed, lid FROM nodes WHERE node_json IS NOT NULL")?;
let rows = stmt.query_map([], |row| { let rows = stmt.query_map([], |row| {
let json: String = row.get(0)?; let json: String = row.get(0)?;
let listed: i64 = row.get(1)?; let listed: i64 = row.get(1)?;
Ok((json, listed != 0)) Ok((json, listed != 0, row.get(2)?))
})?; })?;
let mut out = Vec::new(); let mut out = Vec::new();
for row in rows { for row in rows {
let (json, listed) = row?; let (json, listed, lid) = row?;
let node: Node = serde_json::from_str(&json)?; let node: Node = serde_json::from_str(&json)?;
out.push(StoredNode { node, listed }); out.push(StoredNode { node, listed, lid });
} }
Ok(out) Ok(out)
} }
@@ -544,6 +549,57 @@ impl Db {
} }
} }
/// The local id of the node stored under `uid`, or `None` when it has no
/// row. It stays the same for as long as the row exists, including when a
/// create lands and the row takes the uid Drive gave it.
pub fn lid_of(&self, uid: &str) -> Result<Option<i64>> {
let conn = self.read();
Ok(conn
.query_row("SELECT lid FROM nodes WHERE uid = ?1", params![uid], |r| {
r.get(0)
})
.optional()?)
}
/// The local id of each node, in order, giving a row to each that has none.
///
/// A new row is only a stub: it has no `node_json`, so no listing or search
/// returns it until the node is upserted, which the caller still owes. That
/// keeps this cheap enough to call with the inode lock held, which is
/// where an inode number is first needed.
pub fn lids_for(&self, nodes: &[&Node]) -> Result<Vec<i64>> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let mut lids = Vec::with_capacity(nodes.len());
{
let mut known = tx.prepare_cached("SELECT lid FROM nodes WHERE uid = ?1")?;
let mut stub = tx.prepare_cached(
"INSERT INTO nodes (uid, parent_uid, name, is_dir, mtime, trashed)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
)?;
for node in nodes {
let uid = node.uid.to_string();
let lid = match known.query_row(params![uid], |r| r.get(0)).optional()? {
Some(lid) => lid,
None => {
stub.execute(params![
uid,
node.parent_uid.as_ref().map(ToString::to_string),
node.name,
node.is_folder() as i64,
node.modification_time,
node.trashed as i64,
])?;
tx.last_insert_rowid()
}
};
lids.push(lid);
}
}
tx.commit()?;
Ok(lids)
}
/// Read the persisted incremental-sync cursor (a `DriveEventId`), if any. /// Read the persisted incremental-sync cursor (a `DriveEventId`), if any.
/// The daemon resumes from this on restart instead of reseeding to the /// The daemon resumes from this on restart instead of reseeding to the
/// server head, so changes made while unmounted are still applied (P2). /// server head, so changes made while unmounted are still applied (P2).
@@ -599,6 +655,60 @@ struct PriorRow {
trashed: bool, trashed: bool,
} }
/// Hand a drained placeholder's row, and the rows below it, to the real uid it
/// landed as.
///
/// Dropping the placeholder row and waiting for the server's copy of the real
/// node left a window with no row for either uid. A queued child drained in
/// that window found no authority for its parent, re-read it over the network
/// and was deferred five seconds behind a folder that had just landed
/// (`docs/BUGS.md` B120). The placeholder's own node, readdressed, stands in
/// until the server's copy is upserted over it. A row the real uid already
/// has is newer than the placeholder, so its node is kept, under the
/// placeholder's `lid`: the node was known by that one first.
pub(super) fn adopt_placeholder_row_tx(
tx: &Transaction<'_>,
local: &str,
real: &str,
) -> Result<()> {
tx.execute(
"UPDATE nodes SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real],
)?;
let row: Option<(i64, Option<String>)> = tx
.query_row(
"SELECT lid, node_json FROM nodes WHERE uid = ?1",
params![local],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
let Some((lid, json)) = row else {
return Ok(());
};
let known: Option<(i64, Option<String>)> = tx
.query_row(
"SELECT lid, node_json FROM nodes WHERE uid = ?1",
params![real],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
if let Some((known_lid, _)) = known {
tx.execute("DELETE FROM nodes WHERE lid = ?1", params![known_lid])?;
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![known_lid])?;
}
tx.execute(
"UPDATE nodes SET uid = ?2 WHERE lid = ?1",
params![lid, real],
)?;
let json = known.and_then(|(_, known_json)| known_json).or(json);
let (Some(json), Some(real)) = (json, parse_node_uid(real)) else {
return Ok(());
};
let mut node: Node = serde_json::from_str(&json)?;
node.uid = real;
upsert_node_tx(tx, &node)
}
fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> { fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
let json = serde_json::to_string(node)?; let json = serde_json::to_string(node)?;
let uid = node.uid.to_string(); let uid = node.uid.to_string();
@@ -750,8 +860,10 @@ fn reindex_subtree_tx(
let mut set_path = tx.prepare("UPDATE nodes SET path = ?2 WHERE rowid = ?1")?; let mut set_path = tx.prepare("UPDATE nodes SET path = ?2 WHERE rowid = ?1")?;
let mut unindex = tx.prepare("DELETE FROM nodes_fts WHERE rowid = ?1")?; let mut unindex = tx.prepare("DELETE FROM nodes_fts WHERE rowid = ?1")?;
let mut index = tx.prepare( let mut index = tx.prepare(
// Not a stub row (`Db::lids_for`): it is indexed when its node is
// written.
"INSERT INTO nodes_fts (rowid, name, path) "INSERT INTO nodes_fts (rowid, name, path)
SELECT rowid, name, ?2 FROM nodes WHERE rowid = ?1", SELECT rowid, name, ?2 FROM nodes WHERE rowid = ?1 AND node_json IS NOT NULL",
)?; )?;
for (rowid, path, blocked) in descendants { for (rowid, path, blocked) in descendants {
set_path.execute(params![rowid, path])?; set_path.execute(params![rowid, path])?;
+100 -28
View File
@@ -6,6 +6,7 @@ use rusqlite::{OptionalExtension, params};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use super::Db; use super::Db;
use super::nodes::adopt_placeholder_row_tx;
use crate::Result; use crate::Result;
/// The `kind` of a [`PendingOp`] that uploads a staged file as a new revision. /// The `kind` of a [`PendingOp`] that uploads a staged file as a new revision.
@@ -44,6 +45,12 @@ pub const OP_RENAME: &str = "rename";
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct RenameMeta { pub struct RenameMeta {
pub original_parent_uid: String, pub original_parent_uid: String,
/// The node's name on Drive when the first of its queued renames was made,
/// kept across the renames that replace the row, like the parent. With it
/// the drain knows where the node should still be when the op lands.
/// `None` on a row queued before this field existed.
#[serde(default)]
pub original_name: Option<String>,
} }
/// The `kind` of a [`PendingOp`] that trashes a node the server knows about /// The `kind` of a [`PendingOp`] that trashes a node the server knows about
@@ -75,22 +82,33 @@ pub fn op_supersedes(kind: &str) -> bool {
/// a removed folder: the upload outlived the folder, failed forever against a /// a removed folder: the upload outlived the folder, failed forever against a
/// trashed node, and kept its blob on disk (`docs/BUGS.md` B102). /// trashed node, and kept its blob on disk (`docs/BUGS.md` B102).
/// ///
/// Only revisions are taken from the node tree. A queued rename that moved a /// "Queued beneath it" starts from every folder the node tree places there,
/// node *into* the folder has to land, or the node would stay where it was /// not only from `uid`. A create in a subfolder that already exists on Drive
/// on the server instead of going to the trash with the folder. /// names that subfolder as its parent, so following the queue from `uid` alone
/// missed it: a parked one kept its blob for good, and a due one would have
/// been made in the root once its folder turned out trashed (`docs/BUGS.md`
/// B116).
///
/// Other ops are taken from the node tree only for revisions. A queued rename
/// that moved a node *into* the folder has to land, or the node would stay
/// where it was on the server instead of going to the trash with the folder.
fn drop_doomed_ops(tx: &rusqlite::Transaction<'_>, uid: &str) -> Result<Vec<String>> { fn drop_doomed_ops(tx: &rusqlite::Transaction<'_>, uid: &str) -> Result<Vec<String>> {
const DOOMED: &str = " const DOOMED: &str = "
WITH RECURSIVE WITH RECURSIVE
queued(uid) AS (
SELECT ?1
UNION
SELECT p.uid FROM pending_op p JOIN queued q ON p.parent_uid = q.uid
),
below(uid) AS ( below(uid) AS (
SELECT ?1 SELECT ?1
UNION UNION
SELECT n.uid FROM nodes n JOIN below b ON n.parent_uid = b.uid SELECT n.uid FROM nodes n JOIN below b ON n.parent_uid = b.uid
), ),
queued(uid) AS (
SELECT ?1
UNION
SELECT p.uid FROM pending_op p
WHERE p.kind IN ('create', 'mkdir')
AND p.parent_uid IN (SELECT uid FROM below)
UNION
SELECT p.uid FROM pending_op p JOIN queued q ON p.parent_uid = q.uid
),
doomed(id) AS ( doomed(id) AS (
SELECT id FROM pending_op SELECT id FROM pending_op
WHERE uid IN (SELECT uid FROM queued) WHERE uid IN (SELECT uid FROM queued)
@@ -210,6 +228,28 @@ pub struct PendingCounts {
/// is retrying at the slowest rate it ever will and has been for minutes. /// is retrying at the slowest rate it ever will and has been for minutes.
pub const FAILING_ATTEMPTS: i64 = 6; pub const FAILING_ATTEMPTS: i64 = 6;
/// A queued create that has landed, for [`Db::finish_create`].
#[derive(Debug, Clone, Copy)]
pub struct CreateLanding<'a> {
/// The placeholder the node was known by.
pub local: &'a str,
/// The uid Drive gave it.
pub real: &'a str,
/// The parent and name the op held when the drain took it.
pub sent: (&'a str, &'a str),
/// The parent and name the node has on Drive. They differ from `sent`
/// when the name was taken or the folder was gone.
pub landed: (&'a str, &'a str),
}
struct CreateRow {
blob: Option<String>,
meta: Option<String>,
parent_uid: Option<String>,
name: Option<String>,
created_at: i64,
}
/// The outcome of folding freshly written bytes into a queued create. /// The outcome of folding freshly written bytes into a queued create.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct AttachedBlob { pub struct AttachedBlob {
@@ -383,27 +423,44 @@ impl Db {
/// becomes a revision of `real` instead, with its sidecar passed through /// becomes a revision of `real` instead, with its sidecar passed through
/// `rewrite` so it names the real node. Returns that blob and sidecar, or /// `rewrite` so it names the real node. Returns that blob and sidecar, or
/// `None` when the create is simply done. /// `None` when the create is simply done.
///
/// A rename while the create is on the wire rewrites the row's target
/// ([`Db::rewrite_op_target`]), which the create no longer reads. When the
/// row names another parent or name than `landing.sent`, a rename of `real`
/// to that target is queued in the same transaction, or the move would be
/// lost with the row.
pub fn finish_create( pub fn finish_create(
&self, &self,
id: i64, id: i64,
uploaded: Option<&str>, uploaded: Option<&str>,
local: &str, landing: &CreateLanding<'_>,
real: &str,
rewrite: impl FnOnce(&str) -> Option<String>, rewrite: impl FnOnce(&str) -> Option<String>,
) -> Result<Option<(String, String)>> { ) -> Result<Option<(String, String)>> {
let (local, real) = (landing.local, landing.real);
let mut conn = self.conn.lock(); let mut conn = self.conn.lock();
let tx = conn.transaction()?; let tx = conn.transaction()?;
let row: Option<(Option<String>, Option<String>)> = tx let row: Option<CreateRow> = tx
.query_row( .query_row(
"SELECT blob_path, meta_json FROM pending_op WHERE id = ?1", "SELECT blob_path, meta_json, parent_uid, name, created_at
FROM pending_op WHERE id = ?1",
params![id], params![id],
|r| Ok((r.get(0)?, r.get(1)?)), |r| {
Ok(CreateRow {
blob: r.get(0)?,
meta: r.get(1)?,
parent_uid: r.get(2)?,
name: r.get(3)?,
created_at: r.get(4)?,
})
},
) )
.optional()?; .optional()?;
let newer = match row { let newer = match &row {
Some((Some(blob), Some(meta))) if uploaded != Some(blob.as_str()) => { Some(CreateRow {
rewrite(&meta).map(|meta| (blob, meta)) blob: Some(blob),
} meta: Some(meta),
..
}) if uploaded != Some(blob.as_str()) => rewrite(meta).map(|meta| (blob.clone(), meta)),
_ => None, _ => None,
}; };
match &newer { match &newer {
@@ -420,15 +477,33 @@ impl Db {
tx.execute("DELETE FROM pending_op WHERE id = ?1", params![id])?; tx.execute("DELETE FROM pending_op WHERE id = ?1", params![id])?;
} }
} }
if let Some(CreateRow {
parent_uid: Some(parent),
name: Some(name),
created_at,
..
}) = &row
&& (parent.as_str(), name.as_str()) != landing.sent
{
let meta = serde_json::to_string(&RenameMeta {
original_parent_uid: landing.landed.0.to_string(),
original_name: Some(landing.landed.1.to_string()),
})?;
tx.execute(
"DELETE FROM pending_op WHERE uid = ?1 AND kind = ?2",
params![real, OP_RENAME],
)?;
tx.execute(
"INSERT INTO pending_op (kind, uid, parent_uid, name, meta_json, created_at)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
params![OP_RENAME, real, parent, name, meta, created_at],
)?;
}
tx.execute( tx.execute(
"UPDATE pending_op SET parent_uid = ?2 WHERE parent_uid = ?1", "UPDATE pending_op SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real], params![local, real],
)?; )?;
tx.execute( adopt_placeholder_row_tx(&tx, local, real)?;
"UPDATE nodes SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real],
)?;
tx.execute("DELETE FROM nodes WHERE uid = ?1", params![local])?;
tx.commit()?; tx.commit()?;
Ok(newer) Ok(newer)
} }
@@ -547,7 +622,8 @@ impl Db {
/// Rewrite every queued op that points at a placeholder parent, once that /// Rewrite every queued op that points at a placeholder parent, once that
/// parent has drained and has a real uid. Also moves the node rows whose /// parent has drained and has a real uid. Also moves the node rows whose
/// parent column still names the placeholder, so listings keep resolving. /// parent column still names the placeholder, so listings keep resolving,
/// and readdresses the placeholder's own row.
pub fn remap_local_uid(&self, local: &str, real: &str) -> Result<()> { pub fn remap_local_uid(&self, local: &str, real: &str) -> Result<()> {
let mut conn = self.conn.lock(); let mut conn = self.conn.lock();
let tx = conn.transaction()?; let tx = conn.transaction()?;
@@ -555,11 +631,7 @@ impl Db {
"UPDATE pending_op SET parent_uid = ?2 WHERE parent_uid = ?1", "UPDATE pending_op SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real], params![local, real],
)?; )?;
tx.execute( adopt_placeholder_row_tx(&tx, local, real)?;
"UPDATE nodes SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real],
)?;
tx.execute("DELETE FROM nodes WHERE uid = ?1", params![local])?;
tx.commit()?; tx.commit()?;
Ok(()) Ok(())
} }
+1 -1
View File
@@ -98,7 +98,7 @@ impl Db {
pub fn effective_node_access(&self, uid: &NodeUid) -> Result<Option<Access>> { pub fn effective_node_access(&self, uid: &NodeUid) -> Result<Option<Access>> {
let conn = self.read(); let conn = self.read();
let exists: bool = conn.query_row( let exists: bool = conn.query_row(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE uid = ?1)", "SELECT EXISTS(SELECT 1 FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL)",
[uid.to_string()], [uid.to_string()],
|row| row.get(0), |row| row.get(0),
)?; )?;
+548 -6
View File
@@ -2579,6 +2579,74 @@ fn trashing_a_folder_drops_the_queued_revisions_of_files_below_it() {
); );
} }
/// A create in a subfolder that already exists on Drive names that subfolder,
/// not the folder being trashed, as its parent. It has to go with the folder
/// all the same, along with whatever is queued inside it, or a parked one
/// keeps its bytes for good and a due one is made in the root (B116).
#[test]
fn trashing_a_folder_drops_the_queued_creates_in_its_subfolders() {
let db = Db::open_in_memory().unwrap();
db.upsert_nodes(&[
folder("root", None, "My Files"),
folder("doomed", Some("root"), "doomed"),
folder("sub", Some("doomed"), "sub"),
folder("other", Some("root"), "other"),
])
.unwrap();
let op = |kind: &str, node: &str, parent: &str, blob: Option<&str>| PendingOp {
id: 0,
kind: kind.to_string(),
uid: node.to_string(),
parent_uid: Some(parent.to_string()),
name: Some(format!("{node}.bin")),
blob_path: blob.map(str::to_string),
meta_json: None,
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
};
let sub = uid("sub").to_string();
db.enqueue_op(&op(OP_CREATE, "local~deep", &sub, Some("/staging/deep")))
.unwrap();
db.enqueue_op(&op(OP_MKDIR, "local~dir", &sub, None))
.unwrap();
db.enqueue_op(&op(
OP_CREATE,
"local~deeper",
"local~dir",
Some("/staging/deeper"),
))
.unwrap();
db.enqueue_op(&op(
OP_CREATE,
"local~kept",
&uid("other").to_string(),
Some("/staging/kept"),
))
.unwrap();
let doomed = uid("doomed").to_string();
let (_, mut blobs) = db.replace_ops_with_trash(&doomed, "doomed", 2).unwrap();
blobs.sort();
assert_eq!(blobs, vec!["/staging/deep", "/staging/deeper"]);
let mut left: Vec<(String, String)> = db
.pending_ops()
.unwrap()
.into_iter()
.map(|op| (op.kind, op.uid))
.collect();
left.sort();
assert_eq!(
left,
vec![
(OP_CREATE.to_string(), "local~kept".to_string()),
(OP_TRASH.to_string(), doomed),
]
);
}
/// Deleting a folder that was created offline must take the ops queued /// Deleting a folder that was created offline must take the ops queued
/// underneath it with it: they name a placeholder parent that will now never /// underneath it with it: they name a placeholder parent that will now never
/// become real, so nothing could ever drain them and nothing is left to /// become real, so nothing could ever drain them and nothing is left to
@@ -2785,6 +2853,7 @@ fn a_write_attached_while_its_create_uploads_becomes_a_revision() {
let db = Db::open_in_memory().unwrap(); let db = Db::open_in_memory().unwrap();
let local = "local~abc"; let local = "local~abc";
let real = uid("real").to_string(); let real = uid("real").to_string();
let parent = uid("parent").to_string();
let create = |blob: Option<&str>| PendingOp { let create = |blob: Option<&str>| PendingOp {
id: 0, id: 0,
kind: OP_CREATE.to_string(), kind: OP_CREATE.to_string(),
@@ -2806,9 +2875,12 @@ fn a_write_attached_while_its_create_uploads_becomes_a_revision() {
.expect("create is still queued"); .expect("create is still queued");
let newer = db let newer = db
.finish_create(id, None, local, &real, |json| { .finish_create(
Some(json.replace("local~abc", "vol~real")) id,
}) None,
&landing(local, &real, &parent, "new.txt"),
|json| Some(json.replace("local~abc", "vol~real")),
)
.unwrap(); .unwrap();
assert_eq!( assert_eq!(
@@ -2831,6 +2903,7 @@ fn a_create_whose_blob_landed_is_retired() {
let db = Db::open_in_memory().unwrap(); let db = Db::open_in_memory().unwrap();
let local = "local~abc"; let local = "local~abc";
let real = uid("real").to_string(); let real = uid("real").to_string();
let parent = uid("parent").to_string();
let (id, _) = db let (id, _) = db
.enqueue_op(&PendingOp { .enqueue_op(&PendingOp {
id: 0, id: 0,
@@ -2862,9 +2935,12 @@ fn a_create_whose_blob_landed_is_retired() {
.unwrap(); .unwrap();
let newer = db let newer = db
.finish_create(id, Some("/staging/sent"), local, &real, |_| { .finish_create(
panic!("an unchanged create has no sidecar to rewrite") id,
}) Some("/staging/sent"),
&landing(local, &real, &parent, "new.txt"),
|_| panic!("an unchanged create has no sidecar to rewrite"),
)
.unwrap(); .unwrap();
assert_eq!(newer, None); assert_eq!(newer, None);
@@ -2873,6 +2949,64 @@ fn a_create_whose_blob_landed_is_retired() {
assert_eq!(ops[0].parent_uid.as_deref(), Some(real.as_str())); assert_eq!(ops[0].parent_uid.as_deref(), Some(real.as_str()));
} }
/// The parent and name a create was sent with, landing where it was sent.
fn landing<'a>(local: &'a str, real: &'a str, parent: &'a str, name: &'a str) -> CreateLanding<'a> {
CreateLanding {
local,
real,
sent: (parent, name),
landed: (parent, name),
}
}
#[test]
fn a_rename_made_while_its_create_uploads_is_queued_for_the_landed_node() {
let db = Db::open_in_memory().unwrap();
let local = "local~abc";
let real = uid("real").to_string();
let parent = uid("parent").to_string();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_CREATE.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("parent").to_string()),
name: Some("new.txt".to_string()),
blob_path: Some("/staging/sent".to_string()),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
// The drain took the create; the user moves the file into a folder that
// is itself still queued.
db.rewrite_op_target(local, "local~dir", "moved.txt")
.unwrap();
let mut landed = landing(local, &real, &parent, "new.txt");
landed.landed.1 = "new (conflict).txt";
db.finish_create(id, Some("/staging/sent"), &landed, |_| None)
.unwrap();
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1, "the move outlives the create");
assert_eq!(ops[0].kind, OP_RENAME);
assert_eq!(ops[0].uid, real);
assert_eq!(ops[0].parent_uid.as_deref(), Some("local~dir"));
assert_eq!(ops[0].name.as_deref(), Some("moved.txt"));
let meta: RenameMeta = serde_json::from_str(ops[0].meta_json.as_deref().unwrap()).unwrap();
assert_eq!(meta.original_parent_uid, uid("parent").to_string());
assert_eq!(meta.original_name.as_deref(), Some("new (conflict).txt"));
// The folder lands next and the move follows it.
db.remap_local_uid("local~dir", &uid("dir").to_string())
.unwrap();
let ops = db.pending_ops().unwrap();
assert_eq!(ops[0].parent_uid, Some(uid("dir").to_string()));
}
#[test] #[test]
fn draining_a_folder_repoints_its_queued_children() { fn draining_a_folder_repoints_its_queued_children() {
let db = Db::open_in_memory().unwrap(); let db = Db::open_in_memory().unwrap();
@@ -2902,6 +3036,414 @@ fn draining_a_folder_repoints_its_queued_children() {
assert_eq!(ops[0].parent_uid.as_deref(), Some(real_dir.as_str())); assert_eq!(ops[0].parent_uid.as_deref(), Some(real_dir.as_str()));
} }
/// B120: a child queued inside a folder that has just landed is checked
/// against the folder's real uid before the server's copy of it arrives.
#[test]
fn a_landed_folder_keeps_a_row_until_the_server_copy_replaces_it() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
let local = NodeUid::new(VolumeId::from("local"), LinkId::from("dir"));
let mut placeholder = folder("dir", Some("root"), "New folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
let mut child = file("child", "root", "inside.txt", 1);
child.parent_uid = Some(local.clone());
db.upsert_node(&child).unwrap();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_MKDIR.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("root").to_string()),
name: Some("New folder".to_string()),
blob_path: None,
meta_json: None,
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
db.finish_create(
id,
None,
&landing(
&local.to_string(),
&uid("real").to_string(),
&uid("root").to_string(),
"New folder",
),
|_| None,
)
.unwrap();
assert!(db.node_by_uid(&local.to_string()).unwrap().is_none());
let landed = db
.node_by_uid(&uid("real").to_string())
.unwrap()
.expect("the landed folder has a row before the server's copy is read");
assert_eq!(landed.uid, uid("real"));
assert_eq!(landed.name, "New folder");
assert_eq!(
db.effective_node_access(&uid("child")).unwrap(),
Some(crate::Access::Owner),
"the child's authority resolves without a network re-read"
);
let names: Vec<_> = db
.known_children(&uid("real"))
.unwrap()
.into_iter()
.map(|n| n.name)
.collect();
assert_eq!(names, ["inside.txt"]);
let found = db.search("inside", 10).unwrap();
assert_eq!(found.len(), 1);
assert_eq!(found[0].path, "New folder/inside.txt");
}
#[test]
fn a_landed_placeholder_never_overwrites_the_server_copy() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
let local = NodeUid::new(VolumeId::from("local"), LinkId::from("dir"));
let mut placeholder = folder("dir", Some("root"), "New folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
// The event stream reported the folder before the drain got round to it.
db.upsert_node(&folder("real", Some("root"), "New folder (1)"))
.unwrap();
db.remap_local_uid(&local.to_string(), &uid("real").to_string())
.unwrap();
assert!(db.node_by_uid(&local.to_string()).unwrap().is_none());
assert_eq!(
db.node_by_uid(&uid("real").to_string())
.unwrap()
.map(|n| n.name),
Some("New folder (1)".to_string())
);
}
#[test]
fn a_node_keeps_its_lid_across_updates_and_a_new_one_never_reuses_it() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
db.upsert_node(&file("a", "root", "a.txt", 1)).unwrap();
db.upsert_node(&file("b", "root", "b.txt", 1)).unwrap();
let a = db.lid_of(&uid("a").to_string()).unwrap().unwrap();
let b = db.lid_of(&uid("b").to_string()).unwrap().unwrap();
db.upsert_node(&file("a", "root", "renamed.txt", 2))
.unwrap();
assert_eq!(db.lid_of(&uid("a").to_string()).unwrap(), Some(a));
db.delete_node(&uid("b")).unwrap();
db.upsert_node(&file("c", "root", "c.txt", 1)).unwrap();
let c = db.lid_of(&uid("c").to_string()).unwrap().unwrap();
assert!(c > b, "lid {c} reuses one a deleted row had");
}
#[test]
fn a_lid_handed_out_before_the_node_is_stored_reads_as_no_node() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
db.set_share_access(&uid("root"), crate::Access::Owner)
.unwrap();
let stub = file("stub", "root", "stub.txt", 1);
let [lid] = db.lids_for(&[&stub]).unwrap()[..] else {
panic!("one lid per node");
};
assert_eq!(db.lid_of(&uid("stub").to_string()).unwrap(), Some(lid));
assert_eq!(db.lids_for(&[&stub]).unwrap(), vec![lid]);
assert!(db.node_by_uid(&uid("stub").to_string()).unwrap().is_none());
assert!(db.effective_node_access(&uid("stub")).unwrap().is_none());
assert!(db.node_path(&uid("stub").to_string()).unwrap().is_none());
assert!(!db.has_children(&uid("root")).unwrap());
assert!(db.search("stub", 10).unwrap().is_empty());
assert!(db.load_all().unwrap().iter().all(|n| n.lid != lid));
db.upsert_node(&stub).unwrap();
assert_eq!(db.lid_of(&uid("stub").to_string()).unwrap(), Some(lid));
assert!(db.has_children(&uid("root")).unwrap());
}
#[test]
fn a_landed_placeholder_keeps_its_lid() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
for known in [false, true] {
let local = NodeUid::new(VolumeId::from("local"), LinkId::from(format!("{known}")));
let real = uid(&format!("real-{known}"));
let mut placeholder = folder("dir", Some("root"), "New folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
let lid = db.lid_of(&local.to_string()).unwrap().unwrap();
if known {
// The event stream reported the folder before the drain got round
// to it.
let mut listed = folder("x", Some("root"), "New folder (1)");
listed.uid = real.clone();
db.upsert_node(&listed).unwrap();
}
db.remap_local_uid(&local.to_string(), &real.to_string())
.unwrap();
assert_eq!(db.lid_of(&local.to_string()).unwrap(), None);
assert_eq!(db.lid_of(&real.to_string()).unwrap(), Some(lid));
let name = if known {
"New folder (1)"
} else {
"New folder"
};
assert_eq!(
db.node_by_uid(&real.to_string()).unwrap().map(|n| n.name),
Some(name.to_string())
);
assert_eq!(db.search(name, 10).unwrap().len(), 1, "{name}");
}
}
#[test]
fn migration_v36_gives_every_node_its_rowid_as_lid() {
let path = std::env::temp_dir().join(format!(
"pdfs-db-v35-fixture-{}-{}.db",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
let local = NodeUid::new(VolumeId::from("local"), LinkId::from("dir"));
{
let db = Db::open(&path).unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
let mut placeholder = folder("dir", Some("root"), "Offline folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
db.upsert_node(&file("gone", "root", "gone.txt", 1))
.unwrap();
db.upsert_node(&file("kept", "root", "kept-report.txt", 1))
.unwrap();
db.delete_node(&uid("gone")).unwrap();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_MKDIR.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("root").to_string()),
name: Some("Offline folder".to_string()),
blob_path: None,
meta_json: None,
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
db.record_op_failure(id, "network unreachable", 5_000)
.unwrap();
}
{
// Put the file back in the state a released V35 database was in:
// keyed by uid, with the rowids of the rows it held.
let conn = rusqlite::Connection::open(&path).unwrap();
conn.execute_batch(
"CREATE TABLE nodes_v35 (
uid TEXT PRIMARY KEY, parent_uid TEXT, name TEXT NOT NULL,
is_dir INTEGER NOT NULL, size INTEGER, mtime INTEGER NOT NULL,
revision_hash TEXT, trashed INTEGER NOT NULL DEFAULT 0, node_json TEXT,
listed INTEGER NOT NULL DEFAULT 0, path TEXT
);
INSERT INTO nodes_v35 (rowid, uid, parent_uid, name, is_dir, size, mtime,
revision_hash, trashed, node_json, listed, path)
SELECT lid, uid, parent_uid, name, is_dir, size, mtime, revision_hash,
trashed, node_json, listed, path FROM nodes;
DROP TABLE nodes;
ALTER TABLE nodes_v35 RENAME TO nodes;
CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
DELETE FROM sqlite_sequence WHERE name = 'nodes';
UPDATE sync_state SET value = '35' WHERE key = 'schema_version';",
)
.unwrap();
}
let rowids: Vec<(String, i64)> = {
let conn = rusqlite::Connection::open(&path).unwrap();
let mut stmt = conn
.prepare("SELECT uid, rowid FROM nodes ORDER BY rowid")
.unwrap();
stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))
.unwrap()
.map(|row| row.unwrap())
.collect()
};
assert_eq!(rowids.len(), 3);
let db = Db::open(&path).unwrap();
for (uid, rowid) in &rowids {
assert_eq!(db.lid_of(uid).unwrap(), Some(*rowid), "{uid}");
}
let found = db.search("report", 10).unwrap();
assert_eq!(found.len(), 1, "the search index still finds the node");
assert_eq!(found[0].node.uid, uid("kept"));
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1);
assert_eq!(ops[0].uid, local.to_string());
assert_eq!(ops[0].attempts, 1);
assert_eq!(ops[0].last_error.as_deref(), Some("network unreachable"));
db.upsert_node(&file("new", "root", "new.txt", 1)).unwrap();
let newest = db.lid_of(&uid("new").to_string()).unwrap().unwrap();
assert!(newest > rowids.last().unwrap().1);
drop(db);
let _ = std::fs::remove_file(&path);
}
/// A database as 2.8.2 wrote it (`sqlite3 .dump`, schema 35), with queued ops
/// in every state the drain leaves them in: due, parked, failing, deferred for
/// access, and claimed by a worker when the daemon stopped. The creates below
/// a folder made offline still name their parent by its `local~` placeholder.
#[test]
fn a_2_8_database_migrates_with_every_node_and_op_intact() {
let path = std::env::temp_dir().join(format!(
"pdfs-db-2.8.2-{}-{}.db",
std::process::id(),
now_test_id()
));
let _ = std::fs::remove_file(&path);
rusqlite::Connection::open(&path)
.unwrap()
.execute_batch(include_str!("fixtures/v2.8.2.sql"))
.unwrap();
let (nodes, ops) = migrates_intact(&path);
assert_eq!((nodes, ops), (8, 6));
let db = Db::open(&path).unwrap();
let queued = db.pending_ops().unwrap();
let inner = queued.iter().find(|op| op.uid == "local~inner").unwrap();
assert_eq!(inner.parent_uid.as_deref(), Some("local~dir"));
let parked = db.parked_create_ops().unwrap();
assert_eq!(parked.len(), 1);
assert_eq!(parked[0].name.as_deref(), Some("movie.part"));
drop(db);
let _ = std::fs::remove_file(&path);
}
/// The same migration over a copy of a database a released build wrote, which
/// the fixtures above cannot stand in for: years of rows, every kind of op, and
/// whatever shapes real use left behind. The copy is taken with `VACUUM INTO`
/// over a read-only connection, so the database named is never written.
///
/// `PDFS_MIGRATE_DB=~/.local/state/proton-drive-linux/cache.db cargo test -p
/// pdfs-core a_copy_of_a_real_database -- --ignored`
#[test]
#[ignore = "needs a real database: set PDFS_MIGRATE_DB"]
fn a_copy_of_a_real_database_migrates_with_every_node_and_op_intact() {
let source = std::env::var("PDFS_MIGRATE_DB").expect("set PDFS_MIGRATE_DB");
let path = std::env::temp_dir().join(format!("pdfs-db-real-{}.db", std::process::id()));
let _ = std::fs::remove_file(&path);
rusqlite::Connection::open_with_flags(&source, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY)
.unwrap()
.execute("VACUUM INTO ?1", rusqlite::params![path.to_str().unwrap()])
.unwrap();
let (nodes, ops) = migrates_intact(&path);
eprintln!("{nodes} nodes, {ops} queued ops");
let _ = std::fs::remove_file(&path);
}
/// Opens the database at `path` and checks that the migration kept every node
/// on its row id and every queued op as it was. Returns how many of each.
fn migrates_intact(path: &std::path::Path) -> (usize, usize) {
let (version, nodes, ops, sealed) = {
let conn = rusqlite::Connection::open(path).unwrap();
let version: String = conn
.query_row(
"SELECT value FROM sync_state WHERE key = 'schema_version'",
[],
|r| r.get(0),
)
.unwrap();
let nodes: Vec<(String, i64)> = conn
.prepare("SELECT uid, rowid FROM nodes")
.unwrap()
.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))
.unwrap()
.map(|row| row.unwrap())
.collect();
let ops = rows(&conn, QUEUED);
let sealed = rows(&conn, "SELECT * FROM own_sealed_rev ORDER BY uid");
(version, nodes, ops, sealed)
};
let db = Db::open(path).unwrap();
{
let conn = rusqlite::Connection::open(path).unwrap();
assert_eq!(
rows(&conn, QUEUED),
ops,
"every queued op survives as it was, parked, failing or deferred"
);
// A claim names a drain worker of the daemon that wrote the file.
let claimed: i64 = conn
.query_row(
"SELECT COUNT(*) FROM pending_op WHERE claimed_at <> 0",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(claimed, 0, "a stopped daemon's claims are released");
assert_eq!(
rows(&conn, "SELECT * FROM own_sealed_rev ORDER BY uid"),
sealed
);
}
if version.parse::<i64>().unwrap() < 36 {
for (uid, rowid) in &nodes {
assert_eq!(db.lid_of(uid).unwrap(), Some(*rowid), "{uid}");
}
} else {
for (uid, _) in &nodes {
assert!(db.lid_of(uid).unwrap().is_some(), "{uid}");
}
}
drop(db);
let db = Db::open(path).unwrap();
let conn = rusqlite::Connection::open(path).unwrap();
let count: i64 = conn
.query_row("SELECT COUNT(*) FROM nodes", [], |r| r.get(0))
.unwrap();
assert_eq!(count as usize, nodes.len(), "no node is lost or doubled");
let version: String = conn
.query_row(
"SELECT value FROM sync_state WHERE key = 'schema_version'",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(version, SCHEMA_VERSION.to_string());
drop(db);
(nodes.len(), ops.len())
}
/// Every column of the queue but the drain's claim.
const QUEUED: &str = "SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
attempts, last_error, next_attempt_at, access_deferred_since
FROM pending_op ORDER BY id";
/// Every row `query` returns, every column.
fn rows(conn: &rusqlite::Connection, query: &str) -> Vec<Vec<rusqlite::types::Value>> {
let mut stmt = conn.prepare(query).unwrap();
let columns = stmt.column_count();
stmt.query_map([], |r| (0..columns).map(|i| r.get(i)).collect())
.unwrap()
.map(|row| row.unwrap())
.collect()
}
#[test] #[test]
fn a_failed_op_stays_queued_with_backoff() { fn a_failed_op_stays_queued_with_backoff() {
let db = Db::open_in_memory().unwrap(); let db = Db::open_in_memory().unwrap();
+4
View File
@@ -15,6 +15,7 @@ zip.workspace = true
sha1.workspace = true sha1.workspace = true
proton-drive-rs.workspace = true proton-drive-rs.workspace = true
futures.workspace = true futures.workspace = true
async-trait.workspace = true
tokio.workspace = true tokio.workspace = true
tracing.workspace = true tracing.workspace = true
fuser.workspace = true fuser.workspace = true
@@ -24,3 +25,6 @@ base64.workspace = true
image.workspace = true image.workspace = true
notify = "8" notify = "8"
parking_lot.workspace = true parking_lot.workspace = true
[dev-dependencies]
tracing-subscriber.workspace = true
+95 -12
View File
@@ -97,15 +97,13 @@ fn event_serializes_shared_publication(event: &DriveEvent, own_volume: &VolumeId
/// entry forces the next `lookup`/`readdir` to re-enumerate from the remote, so /// entry forces the next `lookup`/`readdir` to re-enumerate from the remote, so
/// most events only need to invalidate listings rather than re-fetch eagerly. /// most events only need to invalidate listings rather than re-fetch eagerly.
/// ///
/// Applied to **every** mounted inode space, not just the primary one. This task /// Applied to the tree every mount shares, and told to **every** mounted
/// is per-daemon while node state is per-mount, and a `DriveEvent` names a uid, /// session. A `DriveEvent` names a uid, not a mount, so a file trashed from
/// not a mount — so a file trashed from another device has to be withdrawn from /// another device has to be withdrawn from whichever of our sessions is showing
/// whichever of our sessions is showing it, which for anything under a sync /// it — which for anything under a sync folder is a [`Core::fork_state`] fork.
/// folder is a [`Core::fork_state`] fork rather than `core.state`. Reaching for /// Telling the primary session alone left forks serving a deleted file, the
/// the primary state alone left forks serving a deleted file indefinitely, the /// read-side twin of `docs/BUGS.md` B74. [`Core::for_each_mount`] sends what
/// read-side twin of `docs/BUGS.md` B74. Inode numbers are per-mount, so each /// the kernel needs to hear to every session.
/// mount must be notified through its **own** channel; that pairing is what
/// [`Core::for_each_mount`] exists to preserve.
fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdfs_core::Result<()> { fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdfs_core::Result<()> {
let foreign_delete = is_foreign_node_delete(event, &core.primary_root_uid.volume_id); let foreign_delete = is_foreign_node_delete(event, &core.primary_root_uid.volume_id);
let serializes_shared_publication = let serializes_shared_publication =
@@ -233,6 +231,48 @@ fn acknowledge_applied_event(
Ok(id) Ok(id)
} }
/// How many applied node-update ids [`RecentEvents`] remembers. Far more than a
/// replay can reach back: one comes from a batch whose last event was not its
/// newest, so it repeats at most what that batch held.
const RECENT_EVENTS: usize = 4096;
/// The node updates applied lately, by event id, so one that comes round again
/// is skipped rather than applied twice.
///
/// The feed can deliver the same event twice in a batch, and a batch whose
/// events come out of order leaves the cursor short of its newest event, so the
/// next poll replays what lay past it. An update applied twice is not harmless:
/// the first copy of our own change claims the echo [`Core::note_self_change`]
/// recorded, and the second, unclaimed, reads as someone else's edit and evicts
/// the bytes a partial write is about to gap-fill from (docs/BUGS.md B123).
///
/// Only node updates: a delete is idempotent, and a continuity loss names the
/// cursor it lost, which is an id already applied.
#[derive(Default)]
struct RecentEvents {
ids: HashSet<DriveEventId>,
order: VecDeque<DriveEventId>,
}
impl RecentEvents {
/// Record `event` as applied, reporting whether it was already.
fn repeat(&mut self, event: &DriveEvent) -> bool {
let DriveEvent::NodeUpdated { id, .. } = event else {
return false;
};
if !self.ids.insert(id.clone()) {
return true;
}
self.order.push_back(id.clone());
if self.order.len() > RECENT_EVENTS
&& let Some(old) = self.order.pop_front()
{
self.ids.remove(&old);
}
false
}
}
/// The folders one batch of events changed, collected so the listings are /// The folders one batch of events changed, collected so the listings are
/// dropped once at the end of the batch rather than once per event. /// dropped once at the end of the batch rather than once per event.
/// ///
@@ -320,10 +360,10 @@ fn flush_dirty_parents(core: &Core, dirty: &DirtyParents) {
/// only on fatal error. /// only on fatal error.
/// ///
/// Takes the whole `Core` rather than the primary mount's pieces: an event has /// Takes the whole `Core` rather than the primary mount's pieces: an event has
/// to be applied to every mounted inode space, and the registry that enumerates /// to reach every mounted session, and the registry that enumerates them lives
/// them lives on the `Core` (see [`apply_event`]). /// on the `Core` (see [`apply_event`]).
pub(super) async fn run_event_sync( pub(super) async fn run_event_sync(
client: ProtonDriveClient, client: Arc<dyn DriveApi>,
scope: DriveEventScopeId, scope: DriveEventScopeId,
core: Core, core: Core,
) { ) {
@@ -377,6 +417,7 @@ pub(super) async fn run_event_sync(
}; };
info!(?cursor, "event sync started"); info!(?cursor, "event sync started");
let mut recent = RecentEvents::default();
loop { loop {
core.next_poll(POLL_INTERVAL).await; core.next_poll(POLL_INTERVAL).await;
let events = match link::bounded(client.enumerate_events(&scope, cursor.as_ref())).await { let events = match link::bounded(client.enumerate_events(&scope, cursor.as_ref())).await {
@@ -397,6 +438,11 @@ pub(super) async fn run_event_sync(
// did apply. // did apply.
let mut dirty = DirtyParents::default(); let mut dirty = DirtyParents::default();
for event in &events { for event in &events {
if recent.repeat(event) {
debug!(id = %event.id().as_str(), "skipping an event already applied");
cursor = Some(event.id().clone());
continue;
}
// Converge the SDK's own caches (folder keys, entity cache) on the // Converge the SDK's own caches (folder keys, entity cache) on the
// server before applying the event to our tree. Without this, a node // server before applying the event to our tree. Without this, a node
// re-keyed/moved by another client keeps a stale key in the SDK for // re-keyed/moved by another client keeps a stale key in the SDK for
@@ -717,6 +763,43 @@ mod tests {
} }
} }
#[test]
fn a_node_update_seen_again_is_skipped_but_other_events_are_not() {
let update = |id: &str| DriveEvent::NodeUpdated {
id: DriveEventId::from(id),
node_uid: uid("file"),
parent_node_uid: None,
is_trashed: false,
is_shared: false,
};
let mut recent = RecentEvents::default();
assert!(!recent.repeat(&update("e1")));
assert!(!recent.repeat(&update("e2")));
assert!(recent.repeat(&update("e1")), "a duplicate or replay");
let lost = DriveEvent::ContinuityLost {
id: DriveEventId::from("e2"),
};
assert!(!recent.repeat(&lost), "names the cursor, which was applied");
}
#[test]
fn recent_events_forget_the_oldest_past_their_bound() {
let update = |n: usize| DriveEvent::NodeUpdated {
id: DriveEventId::from(format!("e{n}")),
node_uid: uid("file"),
parent_node_uid: None,
is_trashed: false,
is_shared: false,
};
let mut recent = RecentEvents::default();
for n in 0..=RECENT_EVENTS {
assert!(!recent.repeat(&update(n)));
}
assert_eq!(recent.ids.len(), RECENT_EVENTS);
assert!(!recent.repeat(&update(0)), "the oldest was forgotten");
assert!(recent.repeat(&update(RECENT_EVENTS)));
}
#[test] #[test]
fn foreign_delete_invalidates_an_earlier_refresh_generation() { fn foreign_delete_invalidates_an_earlier_refresh_generation() {
let own_volume = VolumeId::from("own"); let own_volume = VolumeId::from("own");
+69 -28
View File
@@ -163,8 +163,8 @@ fn read_request_line(reader: &mut impl BufRead) -> std::io::Result<Option<String
/// Turn a CLI-supplied path into the mount that owns it and the path relative /// Turn a CLI-supplied path into the mount that owns it and the path relative
/// to *that* mount's root. /// to *that* mount's root.
/// ///
/// [`rel_to_mount`] is the older, primary-only reading, and it is what every /// [`rel_to_mount`] is the older, primary-only reading, and it is what most
/// mutating request still uses. It rejects anything outside the primary /// requests still use. It rejects anything outside the primary
/// mountpoint, which leaves a secondary on-demand mount — a synced folder /// mountpoint, which leaves a secondary on-demand mount — a synced folder
/// switched out of mirror mode — with no way to be named at all: `pdfs ls` and /// switched out of mirror mode — with no way to be named at all: `pdfs ls` and
/// `pdfs refresh` both answered "is not under the mountpoint", so a folder /// `pdfs refresh` both answered "is not under the mountpoint", so a folder
@@ -185,11 +185,14 @@ fn route_to_mount(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
rel_to_mount(mountpoint, path).map(|rel| (core.clone(), rel)) rel_to_mount(mountpoint, path).map(|rel| (core.clone(), rel))
} }
/// [`route_to_mount`] for a conflict copy. The listing names a copy inside a /// [`route_to_mount`] for a request that changes what it names: trash, rename,
/// synced folder by its absolute local path (`docs/BUGS.md` B99), so the /// create a folder, resolve a conflict. Each has to run against the on-demand
/// resolve has to run against the on-demand mount serving it. A mirror folder /// mount serving the path, and `pdfs rm ~/Downloads/x` answered "is not under
/// has no mount to run against: its copy is an ordinary local file. /// the mountpoint" while `pdfs move` took the same path. The conflict listing
fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> { /// names a copy in a synced folder by its absolute local path (`docs/BUGS.md`
/// B99), so resolving one needs this too. A mirror folder has no mount to run
/// against: a file there is an ordinary local file.
fn route_change(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
let routed = route_to_mount(core, mountpoint, path).map_err(|e| { let routed = route_to_mount(core, mountpoint, path).map_err(|e| {
let p = Path::new(path); let p = Path::new(path);
let mirrored = core let mirrored = core
@@ -200,7 +203,7 @@ fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
.any(|folder| folder.mode == "mirror" && p.starts_with(&folder.local_path)); .any(|folder| folder.mode == "mirror" && p.starts_with(&folder.local_path));
if mirrored { if mirrored {
CoreError::invalid(format!( CoreError::invalid(format!(
"{path} is in a mirrored folder; rename or delete the local file there" "{path} is in a mirrored folder; change the local file there instead"
)) ))
} else { } else {
e e
@@ -217,6 +220,27 @@ fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
} }
} }
/// [`route_change`] for a request that trashes or renames the node it names.
/// A mount's root is My files or a whole synced folder. An on-demand folder's
/// root has a parent on Drive, so nothing further down would stop `pdfs rm
/// ~/Downloads` from trashing the folder out from under its own mount.
fn route_node(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
let routed = route_change(core, mountpoint, path)?;
if below_root(&routed.1) {
Ok(routed)
} else {
Err(CoreError::invalid(format!(
"{path} is the root of a synced location and cannot be changed"
)))
}
}
/// Whether a mount-relative path names something below the mount's root.
fn below_root(rel: &Path) -> bool {
rel.components()
.any(|component| matches!(component, Component::Normal(_)))
}
/// Turn a CLI-supplied path into a mountpoint-relative path. An absolute path /// Turn a CLI-supplied path into a mountpoint-relative path. An absolute path
/// must live under `mountpoint`; a relative path is taken as already relative to /// must live under `mountpoint`; a relative path is taken as already relative to
/// the mount root. /// the mount root.
@@ -291,6 +315,9 @@ impl Drop for RefreshGuard {
pub(crate) fn status_response(core: &Core, username: &str, mountpoint: &Path) -> CtlResponse { pub(crate) fn status_response(core: &Core, username: &str, mountpoint: &Path) -> CtlResponse {
let pins = core.cache.list_pins(); let pins = core.cache.list_pins();
let queued = core.db.pending_op_counts().unwrap_or_default(); let queued = core.db.pending_op_counts().unwrap_or_default();
// Read after the queue: the drain counts an upload as landing before it
// retires the op, so one of the two always has it (B119).
let landing = core.landing_uploads.load(Ordering::SeqCst);
let staging = core.cache.staging_usage(); let staging = core.cache.staging_usage();
CtlResponse::Status { CtlResponse::Status {
parked_uploads: queued.parked.max(0) as u64, parked_uploads: queued.parked.max(0) as u64,
@@ -307,7 +334,7 @@ pub(crate) fn status_response(core: &Core, username: &str, mountpoint: &Path) ->
budget: core.cache.budget(), budget: core.cache.budget(),
pins, pins,
online: core.online.load(Ordering::Relaxed), online: core.online.load(Ordering::Relaxed),
pending_uploads: queued.uploads.max(0) as u64, pending_uploads: queued.uploads.max(0) as u64 + landing,
pending_changes: queued.changes.max(0) as u64, pending_changes: queued.changes.max(0) as u64,
} }
} }
@@ -959,8 +986,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
}, },
Err(e) => CtlResponse::error(e), Err(e) => CtlResponse::error(e),
}, },
Ok(CtlRequest::Rename { path, new_name }) => match rel_to_mount(mountpoint, &path) { Ok(CtlRequest::Rename { path, new_name }) => match route_node(core, mountpoint, &path) {
Ok(rel) => match core.rename(&rel, &new_name) { Ok((core, rel)) => match core.rename(&rel, &new_name) {
Ok(name) => { Ok(name) => {
core.log_activity(ActivityKind::Rename, &name, format!("was {path}"), true); core.log_activity(ActivityKind::Rename, &name, format!("was {path}"), true);
CtlResponse::Ok { CtlResponse::Ok {
@@ -1017,8 +1044,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
(Err(e), _) | (_, Err(e)) => CtlResponse::error(e), (Err(e), _) | (_, Err(e)) => CtlResponse::error(e),
} }
} }
Ok(CtlRequest::Delete { path }) => match rel_to_mount(mountpoint, &path) { Ok(CtlRequest::Delete { path }) => match route_node(core, mountpoint, &path) {
Ok(rel) => match core.delete(&rel) { Ok((core, rel)) => match core.delete(&rel) {
Ok(name) => { Ok(name) => {
core.log_activity(ActivityKind::Trash, &name, "", true); core.log_activity(ActivityKind::Trash, &name, "", true);
CtlResponse::Ok { CtlResponse::Ok {
@@ -1032,21 +1059,23 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
}, },
Err(e) => CtlResponse::error(e), Err(e) => CtlResponse::error(e),
}, },
Ok(CtlRequest::CreateFolder { parent, name }) => match rel_to_mount(mountpoint, &parent) { Ok(CtlRequest::CreateFolder { parent, name }) => {
Ok(parent_rel) => match core.create_folder(&parent_rel, &name) { match route_change(core, mountpoint, &parent) {
Ok(name) => { Ok((core, parent_rel)) => match core.create_folder(&parent_rel, &name) {
core.log_activity(ActivityKind::CreateFolder, &name, "", true); Ok(name) => {
CtlResponse::Ok { core.log_activity(ActivityKind::CreateFolder, &name, "", true);
message: format!("created folder {name}"), CtlResponse::Ok {
message: format!("created folder {name}"),
}
} }
} Err(e) => {
Err(e) => { core.log_activity(ActivityKind::CreateFolder, &name, &e, false);
core.log_activity(ActivityKind::CreateFolder, &name, &e, false); CtlResponse::error(e)
CtlResponse::error(e) }
} },
}, Err(e) => CtlResponse::error(e),
Err(e) => CtlResponse::error(e), }
}, }
Ok(CtlRequest::UploadPaths { parent, sources }) => { Ok(CtlRequest::UploadPaths { parent, sources }) => {
match rel_to_mount(mountpoint, &parent) { match rel_to_mount(mountpoint, &parent) {
// Ack immediately and upload on a background thread: a directory tree // Ack immediately and upload on a background thread: a directory tree
@@ -1363,7 +1392,7 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
Err(e) => CtlResponse::error(e), Err(e) => CtlResponse::error(e),
}, },
Ok(CtlRequest::ResolveConflict { path, keep }) => { Ok(CtlRequest::ResolveConflict { path, keep }) => {
match route_conflict(core, mountpoint, &path) { match route_change(core, mountpoint, &path) {
Ok((core, rel)) => match core.resolve_conflict(&rel, &keep) { Ok((core, rel)) => match core.resolve_conflict(&rel, &keep) {
Ok(message) => CtlResponse::Ok { message }, Ok(message) => CtlResponse::Ok { message },
Err(e) => CtlResponse::error(e), Err(e) => CtlResponse::error(e),
@@ -1864,6 +1893,18 @@ mod request_limit_tests {
assert!(rel_to_mount(mountpoint, "Photos/../../escape").is_err()); assert!(rel_to_mount(mountpoint, "Photos/../../escape").is_err());
assert!(rel_to_mount(mountpoint, "/mnt/elsewhere").is_err()); assert!(rel_to_mount(mountpoint, "/mnt/elsewhere").is_err());
} }
/// `pdfs rm` and `pdfs rename` reach on-demand mounts, whose root is a whole
/// synced folder with a parent on Drive. Only a path below a root may get
/// through to the trash or rename call.
#[test]
fn only_a_path_below_a_mount_root_names_a_node() {
assert!(below_root(Path::new("Photos")));
assert!(below_root(Path::new("./Photos/2026")));
assert!(!below_root(Path::new("")));
assert!(!below_root(Path::new(".")));
assert!(!below_root(Path::new("./.")));
}
} }
#[cfg(test)] #[cfg(test)]
+14 -24
View File
@@ -27,7 +27,7 @@ use tracing::{info, warn};
use super::sync::{self, base_name}; use super::sync::{self, base_name};
use super::{ use super::{
Core, SecondaryInsertRejection, SecondaryMount, State, SwitchBlocked, clear_stale_mount, Core, SecondaryInsertRejection, SecondaryMount, SwitchBlocked, clear_stale_mount,
device_type_str, dir_is_empty, evict_dir_contents, fuse_connection_id, is_stale_mount, device_type_str, dir_is_empty, evict_dir_contents, fuse_connection_id, is_stale_mount,
now_secs, parse_uid, spawn_session, sync_folder_info, this_hostname, now_secs, parse_uid, spawn_session, sync_folder_info, this_hostname,
}; };
@@ -363,14 +363,14 @@ impl Core {
) -> CoreResult<Option<NodeUid>> { ) -> CoreResult<Option<NodeUid>> {
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(root_uid)) .block_on(self.drive.enumerate_folder_children_node_uids(root_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?; .map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() { if uids.is_empty() {
return Ok(None); return Ok(None);
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes(&uids)) .block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?; .map_err(|e| CoreError::from_api(&e, "resolve device root children"))?;
Ok(nodes Ok(nodes
.into_iter() .into_iter()
@@ -446,10 +446,7 @@ impl Core {
} }
None => self None => self
.rt .rt
.block_on( .block_on(self.drive.create_folder(&root_uid, &name, Some(now_secs())))
self.client
.create_folder(&root_uid, &name, Some(now_secs())),
)
.map_err(|e| CoreError::from_api(&e, &format!("create device folder {name}")))?, .map_err(|e| CoreError::from_api(&e, &format!("create device folder {name}")))?,
}; };
@@ -603,7 +600,7 @@ impl Core {
&& let Some(uid) = parse_uid(&folder.remote_uid) && let Some(uid) = parse_uid(&folder.remote_uid)
&& let Err(e) = self && let Err(e) = self
.rt .rt
.block_on(self.client.trash_nodes(&[uid])) .block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
warn!(id, error = %e, "delete remote device folder failed"); warn!(id, error = %e, "delete remote device folder failed");
@@ -619,23 +616,16 @@ impl Core {
}; };
} }
/// A sibling Core that shares this one's client/rt/cache/db (and transfer, /// A sibling Core for a secondary FUSE session rooted at an `ondemand` sync
/// activity, mount registries) but gets a **fresh, empty `State`** — its own /// folder (devices.md Phase 3). It shares everything with this one,
/// inode space starting at [`ROOT_INO`]. Used to root a secondary FUSE session /// the tree included, except what belongs to a session: its root and its
/// at an `ondemand` sync folder without colliding with the main mount's inodes /// kernel channel, both filled in when the session is built.
/// (devices.md Phase 3).
pub(crate) fn fork_state(&self) -> Core { pub(crate) fn fork_state(&self) -> Core {
let mut fork = self.clone(); let mut fork = self.clone();
fork.primary = false; fork.primary = false;
let share_access = self.state().share_access.clone(); fork.root = Arc::new(OnceLock::new());
fork.state = Arc::new(Mutex::new(State::new(self.db.clone(), share_access, 2)));
// A fresh inode space needs a fresh notification channel: this fork's
// session is the only one that knows these inodes, so it must be the one
// notified about them. Filled in by `spawn_ondemand_mount`.
fork.notifier = Arc::new(OnceLock::new()); fork.notifier = Arc::new(OnceLock::new());
fork.session_live = Arc::new(std::sync::atomic::AtomicBool::new(false)); fork.session_live = Arc::new(std::sync::atomic::AtomicBool::new(false));
// Size upgrades are keyed by inode, which is per-fork too.
fork.size_upgrades = Arc::new(Mutex::new(HashMap::new()));
fork fork
} }
@@ -858,7 +848,7 @@ impl Core {
})?; })?;
let root = self let root = self
.rt .rt
.block_on(self.client.enumerate_nodes(std::slice::from_ref(&root_uid))) .block_on(self.drive.enumerate_nodes(std::slice::from_ref(&root_uid)))
.map_err(|e| SwitchBlocked::Failed(format!("fetch remote root: {e}")))? .map_err(|e| SwitchBlocked::Failed(format!("fetch remote root: {e}")))?
.into_iter() .into_iter()
.next() .next()
@@ -937,8 +927,8 @@ impl Core {
} }
} }
/// Spawn a secondary FUSE session rooted at `root` over `local` on a forked /// Spawn a secondary FUSE session rooted at `root` over `local` from a
/// inode space. Clears any stale kernel mount first (a crashed run can leave /// [`Core::fork_state`] sibling. Clears any stale kernel mount first (a crashed run can leave
/// one, which would fail the fresh mount with EBUSY). /// one, which would fail the fresh mount with EBUSY).
pub(crate) fn spawn_ondemand_mount( pub(crate) fn spawn_ondemand_mount(
&self, &self,
@@ -1017,7 +1007,7 @@ impl Core {
}; };
let root = match self let root = match self
.rt .rt
.block_on(self.client.enumerate_nodes(std::slice::from_ref(&root_uid))) .block_on(self.drive.enumerate_nodes(std::slice::from_ref(&root_uid)))
{ {
Ok(v) => match v.into_iter().next() { Ok(v) => match v.into_iter().next() {
Some(n) => n, Some(n) => n,
+306 -127
View File
@@ -24,28 +24,27 @@ use std::collections::HashMap;
use std::fs::File; use std::fs::File;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::Arc; use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::{Duration, Instant}; use std::time::{Duration, Instant};
use parking_lot::Mutex; use parking_lot::Mutex;
use futures::StreamExt;
use pdfs_core::batch; use pdfs_core::batch;
use pdfs_core::cache::{Baseline, StagedWrite}; use pdfs_core::cache::{Baseline, StagedWrite};
use pdfs_core::control::{ActivityKind, TransferDirection}; use pdfs_core::control::{ActivityKind, TransferDirection};
use pdfs_core::db::{ use pdfs_core::db::{
OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, PARK_EXPIRY_MS, PendingOp, RenameMeta, CreateLanding, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, PARK_EXPIRY_MS,
PendingOp, RenameMeta,
}; };
use proton_drive_rs::proton_sdk::error::ProtonError;
use proton_drive_rs::proton_sdk::ids::NodeUid; use proton_drive_rs::proton_sdk::ids::NodeUid;
use proton_drive_rs::{Node, NodeKind, NodeMoveItem}; use proton_drive_rs::{Node, NodeKind};
use tracing::{debug, error, info, warn}; use tracing::{debug, error, info, warn};
use super::state::{Intervals, PendingRevision}; use super::state::{Intervals, PendingRevision};
use super::transfers::CountingReader; use super::transfers::CountingReader;
use super::{ use super::{
Core, DRAIN_BACKOFF_MAX, DRAIN_BACKOFF_MIN, DRAIN_IDLE_POLL, DRAIN_REVISION_DEBOUNCE, Core, DRAIN_BACKOFF_MAX, DRAIN_BACKOFF_MIN, DRAIN_IDLE_POLL, DRAIN_REVISION_DEBOUNCE,
DRAIN_REVISION_DEBOUNCE_MAX, ROOT_INO, UPLOAD_TIME_MEMORY, WriteAuthority, conflict_name, DRAIN_REVISION_DEBOUNCE_MAX, UPLOAD_TIME_MEMORY, WriteAuthority, conflict_name,
is_already_exists, is_gone, is_local_uid_str, media_type_for, node_revision_id, node_size, is_already_exists, is_gone, is_local_uid_str, media_type_for, node_revision_id, node_size,
now_millis, now_secs, parse_node_uid, now_millis, now_secs, parse_node_uid,
}; };
@@ -113,6 +112,23 @@ impl Drop for UploadCancel {
} }
} }
/// One upload counted in [`Core::landing_uploads`] from its op's retirement
/// until its bytes are cached, and uncounted when dropped, early return or not.
struct LandingUpload(Arc<AtomicU64>);
impl LandingUpload {
fn begin(count: &Arc<AtomicU64>) -> Self {
count.fetch_add(1, Ordering::SeqCst);
Self(count.clone())
}
}
impl Drop for LandingUpload {
fn drop(&mut self) {
self.0.fetch_sub(1, Ordering::SeqCst);
}
}
/// The debounce arithmetic of [`Core::revision_debounce`], without the map: /// The debounce arithmetic of [`Core::revision_debounce`], without the map:
/// the last measured upload time for this node, bounded on both sides. /// the last measured upload time for this node, bounded on both sides.
/// ///
@@ -659,6 +675,15 @@ impl Core {
return Ok(()); return Ok(());
} }
}; };
if let Some(meta) = op.meta_json.as_deref() {
let meta: RenameMeta = serde_json::from_str(meta)?;
if moved_elsewhere(&meta, &node, &parent, &name) {
// Someone moved or renamed it while ours was queued. Ours still
// wins, as the last change does on Drive.
info!(%uid, from = %node.name, to = %name,
"node moved remotely while its rename was queued");
}
}
let landed = if node.parent_uid.as_ref() == Some(&parent) { let landed = if node.parent_uid.as_ref() == Some(&parent) {
self.drain_rename_in_place(&uid, &node.name, &name)? self.drain_rename_in_place(&uid, &node.name, &name)?
} else { } else {
@@ -667,7 +692,7 @@ impl Core {
// no window for the requirements to go stale between two calls — // no window for the requirements to go stale between two calls —
// the race B46 queued this op to get away from. // the race B46 queued this op to get away from.
let target = (node.name != name).then_some(name.as_str()); let target = (node.name != name).then_some(name.as_str());
match self.move_renaming(&uid, &parent, target) { match self.move_rename_remote(&uid, &parent, target) {
Ok(()) => name.clone(), Ok(()) => name.clone(),
// The destination holds that name already. Landing under a // The destination holds that name already. Landing under a
// *different* name is the non-destructive resolution: it neither // *different* name is the non-destructive resolution: it neither
@@ -678,7 +703,7 @@ impl Core {
} }
let alt = conflict_name(&name, now_secs()); let alt = conflict_name(&name, now_secs());
warn!(%uid, name, alt, "destination already holds that name; using a conflict name"); warn!(%uid, name, alt, "destination already holds that name; using a conflict name");
self.move_renaming(&uid, &parent, Some(&alt))?; self.move_rename_remote(&uid, &parent, Some(&alt))?;
self.adopt_drained_name(&uid, &alt); self.adopt_drained_name(&uid, &alt);
self.log_activity( self.log_activity(
ActivityKind::Rename, ActivityKind::Rename,
@@ -715,7 +740,6 @@ impl Core {
} }
}; };
self.db.delete_op(op.id)?; self.db.delete_op(op.id)?;
self.note_self_change(&uid);
info!(%uid, name = %landed, "pending rename landed"); info!(%uid, name = %landed, "pending rename landed");
Ok(()) Ok(())
} }
@@ -754,31 +778,6 @@ impl Core {
} }
} }
/// Move `uid` under `parent` in one request, renaming it to `target_name` on
/// the way when one is given.
fn move_renaming(
&self,
uid: &NodeUid,
parent: &NodeUid,
target_name: Option<&str>,
) -> Result<(), ProtonError> {
let item = NodeMoveItem {
uid: uid.clone(),
target_name: target_name.map(str::to_string),
};
self.rt.block_on(async {
let mut outcomes =
std::pin::pin!(self.client.move_nodes_streaming(vec![item], parent.clone()));
match outcomes.next().await {
Some(Ok((_, outcome))) => outcome,
Some(Err(e)) => Err(e),
None => Err(ProtonError::invalid_operation(format!(
"move of {uid} reported no outcome"
))),
}
})
}
/// Apply a queued trash to the remote. /// Apply a queued trash to the remote.
/// ///
/// A node that is already gone is a success, not a failure: the outcome the /// A node that is already gone is a success, not a failure: the outcome the
@@ -789,7 +788,7 @@ impl Core {
let name = op.name.clone().unwrap_or_else(|| op.uid.clone()); let name = op.name.clone().unwrap_or_else(|| op.uid.clone());
match self match self
.rt .rt
.block_on(self.client.trash_nodes(std::slice::from_ref(&uid))) .block_on(self.drive.trash_nodes(std::slice::from_ref(&uid)))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
Ok(()) => {} Ok(()) => {}
@@ -799,7 +798,6 @@ impl Core {
Err(e) => return Err(e.into()), Err(e) => return Err(e.into()),
} }
self.db.complete_trash_op(op.id, &uid)?; self.db.complete_trash_op(op.id, &uid)?;
self.own_sealed_revs.lock().remove(&uid);
if let Err(e) = self.db.clear_own_sealed_rev(&uid.to_string()) { if let Err(e) = self.db.clear_own_sealed_rev(&uid.to_string()) {
debug!(%uid, error = %e, "clearing the sealed-revision record failed"); debug!(%uid, error = %e, "clearing the sealed-revision record failed");
} }
@@ -854,6 +852,7 @@ impl Core {
// create was queued because nobody heard back, so adopt it. And a node // create was queued because nobody heard back, so adopt it. And a node
// whose trash is still queued is about to let go of it. // whose trash is still queued is about to let go of it.
let mut name = wanted.clone(); let mut name = wanted.clone();
let mut home = parent.clone();
let mut adopted = false; let mut adopted = false;
let mut real = self.create_drained_node(op, &parent, &name); let mut real = self.create_drained_node(op, &parent, &name);
if real.as_ref().is_err_and(|e| is_already_exists(e.as_ref())) if real.as_ref().is_err_and(|e| is_already_exists(e.as_ref()))
@@ -896,6 +895,7 @@ impl Core {
name = conflict_name(&wanted, now_secs()); name = conflict_name(&wanted, now_secs());
real = self.create_drained_node(op, &root, &name); real = self.create_drained_node(op, &root, &name);
} }
home = root;
if real.is_ok() { if real.is_ok() {
self.log_activity( self.log_activity(
ActivityKind::Upload, ActivityKind::Upload,
@@ -913,8 +913,30 @@ impl Core {
// An adopted node was made empty, so a blob this op carries has not been // An adopted node was made empty, so a blob this op carries has not been
// uploaded: it goes on as a revision of the node instead. // uploaded: it goes on as a revision of the node instead.
let uploaded = (!adopted).then_some(op.blob_path.as_deref()).flatten(); let uploaded = (!adopted).then_some(op.blob_path.as_deref()).flatten();
let newer = self.retire_create(op, uploaded, &local, &real)?; let landing = CreateLanding {
self.adopt_real_uid(&local, &real)?; local: &local.to_string(),
real: &real.to_string(),
sent: (parent_str, &wanted),
landed: (&home.to_string(), &name),
};
let newer = self.retire_create(op, uploaded, &landing, &local, &real)?;
// The uploaded blob is the new file's content, so it becomes the cached
// content, as for a revision. A partial write fills its gaps from it;
// without it the next write open was refused (`docs/BUGS.md` B123).
let landed: Option<StagedWrite> = uploaded
.filter(|_| !newer)
.and(op.meta_json.as_deref())
.and_then(|json| serde_json::from_str(json).ok());
self.adopt_real_uid(&local, &real, |node| {
if let (Some(blob), Some(meta)) = (uploaded, &landed)
&& node_size(node) == meta.len
&& keeps_landed_upload(self.cache.is_pinned(&real), meta.len, self.cache.budget())
{
let _ =
self.cache
.store_file(&real, node.modification_time, meta.len, Path::new(blob));
}
})?;
// The feed will report this create back to us; the tree already has it // The feed will report this create back to us; the tree already has it
// under its real uid, so that event is ours to ignore (`Core::self_changes`). // under its real uid, so that event is ours to ignore (`Core::self_changes`).
self.note_self_change(&real); self.note_self_change(&real);
@@ -946,8 +968,8 @@ impl Core {
) -> Result<Option<NodeUid>, Box<dyn std::error::Error>> { ) -> Result<Option<NodeUid>, Box<dyn std::error::Error>> {
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(parent))?; .block_on(self.drive.enumerate_folder_children_node_uids(parent))?;
let children = self.rt.block_on(self.client.enumerate_nodes_light(&uids))?; let children = self.rt.block_on(self.drive.enumerate_nodes_light(&uids))?;
let Some(twin) = children let Some(twin) = children
.into_iter() .into_iter()
.find(|node| node.name == name && !node.trashed) .find(|node| node.name == name && !node.trashed)
@@ -981,7 +1003,7 @@ impl Core {
true => { true => {
Ok(self Ok(self
.rt .rt
.block_on(self.client.create_folder(parent, name, Some(now_secs())))?) .block_on(self.drive.create_folder(parent, name, Some(now_secs())))?)
} }
false => self.upload_created_file(op, parent, name), false => self.upload_created_file(op, parent, name),
} }
@@ -996,7 +1018,7 @@ impl Core {
name: &str, name: &str,
) -> Result<NodeUid, Box<dyn std::error::Error>> { ) -> Result<NodeUid, Box<dyn std::error::Error>> {
let Some(blob) = op.blob_path.as_deref() else { let Some(blob) = op.blob_path.as_deref() else {
return Ok(self.rt.block_on(self.client.upload_file( return Ok(self.rt.block_on(self.drive.upload_file(
parent, parent,
name, name,
media_type_for(name), media_type_for(name),
@@ -1016,7 +1038,7 @@ impl Core {
.transfers .transfers
.begin(name, op.uid.clone(), TransferDirection::Upload, meta.len); .begin(name, op.uid.clone(), TransferDirection::Upload, meta.len);
let reader = CountingReader::new(File::open(blob)?, &guard); let reader = CountingReader::new(File::open(blob)?, &guard);
let uid = self.rt.block_on(self.client.upload_file_from( let uid = self.rt.block_on(self.drive.upload_file_from(
parent, parent,
name, name,
media_type_for(name), media_type_for(name),
@@ -1043,19 +1065,17 @@ impl Core {
&self, &self,
op: &PendingOp, op: &PendingOp,
uploaded: Option<&str>, uploaded: Option<&str>,
landing: &CreateLanding<'_>,
local: &NodeUid, local: &NodeUid,
real: &NodeUid, real: &NodeUid,
) -> Result<bool, Box<dyn std::error::Error>> { ) -> Result<bool, Box<dyn std::error::Error>> {
let real_str = real.to_string();
let newer = { let newer = {
let mut pending = self.pending.lock(); let mut pending = self.pending.lock();
let newer = let newer = self.db.finish_create(op.id, uploaded, landing, |json| {
self.db let mut meta: StagedWrite = serde_json::from_str(json).ok()?;
.finish_create(op.id, uploaded, &local.to_string(), &real_str, |json| { meta.uid = landing.real.to_string();
let mut meta: StagedWrite = serde_json::from_str(json).ok()?; serde_json::to_string(&meta).ok()
meta.uid = real_str.clone(); })?;
serde_json::to_string(&meta).ok()
})?;
pending.remove(local); pending.remove(local);
if let Some((blob, json)) = &newer { if let Some((blob, json)) = &newer {
let meta: StagedWrite = serde_json::from_str(json)?; let meta: StagedWrite = serde_json::from_str(json)?;
@@ -1069,12 +1089,24 @@ impl Core {
} }
newer.is_some() newer.is_some()
}; };
if !newer && let Some(blob) = uploaded {
self.cache_created(op, local, real, blob);
}
self.for_each_state(|st| { self.for_each_state(|st| {
if let Some(ino) = st.by_uid.remove(local) { if let Some(ino) = st.by_uid.remove(local) {
st.by_uid.insert(real.clone(), ino); st.by_uid.insert(real.clone(), ino);
if let Some(e) = st.entries.get_mut(&ino) { if let Some(e) = st.entries.get_mut(&ino) {
e.uid = real.clone(); e.uid = real.clone();
} }
// Its children still name the placeholder as their parent, and
// the next write of one would put it back on the row.
for kid in st.children.get(&ino).cloned().unwrap_or_default() {
if let Some(e) = st.entries.get_mut(&kid)
&& e.node.parent_uid.as_ref() == Some(local)
{
e.node.parent_uid = Some(real.clone());
}
}
} }
for aw in st.active_writes.values_mut() { for aw in st.active_writes.values_mut() {
if aw.uid == *local { if aw.uid == *local {
@@ -1085,28 +1117,69 @@ impl Core {
Ok(newer) Ok(newer)
} }
/// Cache a landed create's uploaded `blob` under `real`, keyed as the tree
/// has the file.
///
/// A write handle open on the file is repointed at `real` right after this,
/// and a release fills its gaps from the cache by the size and mtime it
/// opened over. Drive lists the new node a moment after the create, so
/// `adopt_real_uid` caches the bytes only once it can read the node back.
/// A release in between found nothing to fill from, and the next write open
/// was refused (docs/BUGS.md B131).
fn cache_created(&self, op: &PendingOp, local: &NodeUid, real: &NodeUid, blob: &str) {
let Some(meta) = op
.meta_json
.as_deref()
.and_then(|json| serde_json::from_str::<StagedWrite>(json).ok())
else {
return;
};
let mut mtime = None;
self.for_each_state(|st| {
if let Some(entry) = st.by_uid.get(local).and_then(|ino| st.entries.get(ino)) {
mtime = Some(entry.node.modification_time);
}
});
if let Some(mtime) = mtime
&& keeps_landed_upload(self.cache.is_pinned(real), meta.len, self.cache.budget())
{
let _ = self
.cache
.store_file(real, mtime, meta.len, Path::new(blob));
}
}
/// Swap a placeholder uid for the real one across everything that keyed off /// Swap a placeholder uid for the real one across everything that keyed off
/// it: queued children, the DB, the in-memory tree, and the caches. /// it: queued children, the DB, the in-memory tree, and the caches.
/// ///
/// The inode is deliberately kept, so anything already holding the file open /// The inode is deliberately kept, so anything already holding the file open
/// keeps working across the drain. /// keeps working across the drain.
/// ///
/// Applied to **every** live inode space, not just this Core's. A node /// Applied to the tree every mount shares. A node created inside an
/// created inside an on-demand sync folder lives in that fork's `state`, /// on-demand sync folder once lived in that fork's own state, where the
/// while the drain that lands its create belongs to the primary mount — so /// primary mount's drain never looked, and stayed on its `local~` uid —
/// reaching for `self.state` here found nothing and left the fork's entry on /// an empty file for as long as the daemon ran (`docs/BUGS.md` B74).
/// its `local~` uid, which reads as an empty file for as long as the daemon ///
/// runs (`docs/BUGS.md` B74). A uid is unique across mounts, so at most one /// `adopt` sees the node as Drive made it, before anything is rebased onto
/// of them matches and the others are no-ops. /// it.
pub(crate) fn adopt_real_uid( pub(crate) fn adopt_real_uid(
&self, &self,
local: &NodeUid, local: &NodeUid,
real: &NodeUid, real: &NodeUid,
adopt: impl FnOnce(&Node),
) -> Result<(), Box<dyn std::error::Error>> { ) -> Result<(), Box<dyn std::error::Error>> {
let node = self // Drive answers a create before it lists the new node.
.fetch_node(real) let mut node = self
.read_back(real)
.map_err(|e| self.errno_error(e, "fetch node"))?; .map_err(|e| self.errno_error(e, "fetch node"))?;
// Repoints queued children and node rows, and drops the placeholder row. // A rename made while the create was on the wire is queued against the
// real uid (`Db::finish_create`). Until it lands, the tree keeps the
// name and folder the user gave it.
let renaming = self.db.has_pending_op(&real.to_string(), OP_RENAME)?;
// Before open handles are rebased onto the node, so one released in
// between finds what `adopt` keeps (`docs/BUGS.md` B123).
adopt(&node);
// Repoints queued children and node rows, and readdresses the placeholder row.
// A no-op after `retire_create`, which has already done both. // A no-op after `retire_create`, which has already done both.
self.db self.db
.remap_local_uid(&local.to_string(), &real.to_string())?; .remap_local_uid(&local.to_string(), &real.to_string())?;
@@ -1118,6 +1191,10 @@ impl Core {
let Some(&ino) = st.by_uid.get(real) else { let Some(&ino) = st.by_uid.get(real) else {
return; return;
}; };
if renaming && let Some(e) = st.entries.get(&ino) {
node.name = e.node.name.clone();
node.parent_uid = e.node.parent_uid.clone();
}
let mut landed = node.clone(); let mut landed = node.clone();
st.keep_open_write_size(ino, &mut landed); st.keep_open_write_size(ino, &mut landed);
if let Some(e) = st.entries.get_mut(&ino) { if let Some(e) = st.entries.get_mut(&ino) {
@@ -1151,6 +1228,12 @@ impl Core {
if let Err(e) = self.db.upsert_node(&node) { if let Err(e) = self.db.upsert_node(&node) {
warn!(%real, error = %e, "db upsert_node failed after remap"); warn!(%real, error = %e, "db upsert_node failed after remap");
} }
// A handle still open on the placeholder was based on no revision at
// all: a local-clock mtime and no revision id. Released as it is, its
// write would land as a conflict copy of the empty file this create just
// made (`docs/BUGS.md` B113). Rebased before the restamp below, so a
// release that races this is caught there.
self.rebase_open_writes(real, &node);
// A write carried over from the create was made against no revision at // A write carried over from the create was made against no revision at
// all; the one just created is what it now replaces. // all; the one just created is what it now replaces.
self.rebaseline_pending(real, &node); self.rebaseline_pending(real, &node);
@@ -1202,16 +1285,11 @@ impl Core {
// instead of forking — so the finished bytes win the name rather than // instead of forking — so the finished bytes win the name rather than
// being exiled to a `(sync-conflict)` copy. // being exiled to a `(sync-conflict)` copy.
let remote_rev = node_revision_id(&node); let remote_rev = node_revision_id(&node);
// In memory first, then the table it is written through to: the map is // The table, so the answer survives a restart (`own_sealed_rev`).
// this process's cache of it, and after a restart only the table has let own_rev = self
// the answer (which is the whole reason it is persisted). .db
let own_rev = match self.own_sealed_revs.lock().get(uid).cloned() { .own_sealed_rev(&uid.to_string(), now_millis())
Some(rev) => Some(rev), .unwrap_or_default();
None => self
.db
.own_sealed_rev(&uid.to_string(), now_millis())
.unwrap_or_default(),
};
if is_own_self_supersede(meta.complete, remote_rev.as_deref(), own_rev.as_deref()) { if is_own_self_supersede(meta.complete, remote_rev.as_deref(), own_rev.as_deref()) {
debug!(%uid, ?remote_rev, debug!(%uid, ?remote_rev,
"queued write chains onto our own sealed revision; not a conflict"); "queued write chains onto our own sealed revision; not a conflict");
@@ -1281,7 +1359,7 @@ impl Core {
self.transfers self.transfers
.begin(&alt, meta.uid.clone(), TransferDirection::Upload, meta.len); .begin(&alt, meta.uid.clone(), TransferDirection::Upload, meta.len);
let reader = CountingReader::new(File::open(blob)?, &guard); let reader = CountingReader::new(File::open(blob)?, &guard);
self.rt.block_on(self.client.upload_file_from( self.rt.block_on(self.drive.upload_file_from(
&parent, &parent,
&alt, &alt,
media_type_for(&alt), media_type_for(&alt),
@@ -1478,25 +1556,13 @@ impl Core {
/// A node's parent uid and name, as the tree currently has them. /// A node's parent uid and name, as the tree currently has them.
pub(crate) fn node_place(&self, uid: &NodeUid) -> Option<(NodeUid, String)> { pub(crate) fn node_place(&self, uid: &NodeUid) -> Option<(NodeUid, String)> {
// Across every mount: the drain runs on the primary Core, but a node {
// written inside a sync folder is interned in that fork's inode space, let st = self.state();
// so asking `self.state` alone always missed it and fell through to the
// DB. That still gave the right answer — the fallback below exists for
// exactly this shape of miss — but it went to disk for something already
// in memory, and only *because* of the fallback was it not a bug.
let mut found = None;
self.for_each_state(|st| {
if found.is_some() {
return;
}
if let Some(entry) = st.by_uid.get(uid).and_then(|ino| st.entries.get(ino)) if let Some(entry) = st.by_uid.get(uid).and_then(|ino| st.entries.get(ino))
&& let Some(parent) = entry.node.parent_uid.clone() && let Some(parent) = entry.node.parent_uid.clone()
{ {
found = Some((parent, entry.node.name.clone())); return Some((parent, entry.node.name.clone()));
} }
});
if found.is_some() {
return found;
} }
// The in-memory tree only holds what has been walked to since the daemon // The in-memory tree only holds what has been walked to since the daemon
// started, and the drain routinely runs before anything has walked to // started, and the drain routinely runs before anything has walked to
@@ -1531,8 +1597,8 @@ impl Core {
/// root yet, which is not where the drain runs; the drain must not panic /// root yet, which is not where the drain runs; the drain must not panic
/// over it regardless, since that would stop the queue for good. /// over it regardless, since that would stop the queue for good.
pub(crate) fn root_uid(&self) -> Option<NodeUid> { pub(crate) fn root_uid(&self) -> Option<NodeUid> {
let st = self.state(); let root = self.root_ino();
st.entries.get(&ROOT_INO).map(|e| e.uid.clone()) self.state().entries.get(&root).map(|e| e.uid.clone())
} }
/// Upload a staged revision of a file the server already knows about. /// Upload a staged revision of a file the server already knows about.
@@ -1601,7 +1667,7 @@ impl Core {
} }
let reader = CountingReader::new(File::open(&blob)?, &guard).with_cancel(cancel.flag()); let reader = CountingReader::new(File::open(&blob)?, &guard).with_cancel(cancel.flag());
let started = Instant::now(); let started = Instant::now();
let sent = self.rt.block_on(self.client.upload_new_revision_from( let sent = self.rt.block_on(self.drive.upload_new_revision_from(
&uid, &uid,
reader, reader,
meta.len as i64, meta.len as i64,
@@ -1627,23 +1693,36 @@ impl Core {
// is what the next write's debounce is scaled from. // is what the next write's debounce is scaled from.
self.record_upload_time(&uid, took); self.record_upload_time(&uid, took);
// Still pending to whoever asks until the bytes are cached below, and
// counted before the op goes so the status never reads empty in between.
// An empty queue is how a script or the app knows a written file reads
// without the network; retiring the op alone left the read-back's round
// trip in which it did not (B119). The op itself has to go first: one
// restored after a crash would re-send a revision this daemon has not
// yet recorded as its own, and come back as a conflict copy.
let _landing = LandingUpload::begin(&self.landing_uploads);
// Retire the op before dropping the blob: a crash between the two leaves // Retire the op before dropping the blob: a crash between the two leaves
// an orphaned file (harmless), whereas the reverse would leave a queued // an orphaned file (harmless), whereas the reverse would leave a queued
// op pointing at nothing. // op pointing at nothing.
self.db.delete_op(op.id)?; self.db.delete_op(op.id)?;
self.release_pending(&uid, &blob);
// The staged blob now matches the sealed revision, so a pinned file keeps
// it as its cached content rather than re-downloading what we just sent.
// Adopted by link, not by value: the blob is the whole file, and a pinned
// video read into a `Vec` here is an OOM. `discard_staged` below drops the
// staging name only — the cache keeps the inode.
if self.cache.is_pinned(&uid) {
let _ = self.cache.store_file(&uid, now_secs(), meta.len, &blob);
}
self.cache.discard_staged(&blob);
self.evict_reader(&uid); self.evict_reader(&uid);
self.refresh_after_upload(&uid); // The staged blob now matches the sealed revision, so it becomes the
// file's cached content, stamped like the revision just read back.
// Reading the file again then needs no network, which an editor or a
// build does at once and a flaky link cannot always give (`docs/BUGS.md`
// B119). Adopted by link, not by value: the blob is the whole file, and
// a video read into a `Vec` here is an OOM. `discard_staged` below drops
// the staging name only — the cache keeps the inode.
self.refresh_after_upload(&uid, &blob, |node| {
if node_size(node) == meta.len
&& keeps_landed_upload(self.cache.is_pinned(&uid), meta.len, self.cache.budget())
{
let _ = self
.cache
.store_file(&uid, node.modification_time, meta.len, &blob);
}
});
self.cache.discard_staged(&blob);
self.log_activity(ActivityKind::Upload, &name, "uploaded", true); self.log_activity(ActivityKind::Upload, &name, "uploaded", true);
// Both times, so a slow drain shows whether the upload itself or the // Both times, so a slow drain shows whether the upload itself or the
// checks and gap-fill around it are where the time goes. // checks and gap-fill around it are where the time goes.
@@ -1677,21 +1756,39 @@ impl Core {
/// we just sealed. Skipping both, as this used to, is what produced a file /// we just sealed. Skipping both, as this used to, is what produced a file
/// that conflicted with itself and left a full-size duplicate behind. /// that conflicted with itself and left a full-size duplicate behind.
/// ///
/// `blob` is the revision that landed, which stays the pending one until
/// open write handles are rebased off it, and `adopt` gets the node read
/// back before that, so the uploaded bytes can be cached under its
/// revision first. A handle released in between fills its gaps from one
/// or the other; with the blob let go first and the cache filled last, it
/// found neither, and the next write open was refused (docs/BUGS.md B123).
///
/// Best effort: a failure here costs a spurious conflict copy on the next /// Best effort: a failure here costs a spurious conflict copy on the next
/// write, not this upload, which has already landed. /// write, not this upload, which has already landed. Returns the node as
pub(crate) fn refresh_after_upload(&self, uid: &NodeUid) { /// the server now has it, or `None` when it could not be read back.
let node = match self.fetch_node_remote(uid) { pub(crate) fn refresh_after_upload(
&self,
uid: &NodeUid,
blob: &Path,
adopt: impl FnOnce(&Node),
) -> Option<Node> {
let fetched = self.fetch_node_remote(uid);
if !matches!(fetched, Ok(Some(_))) {
self.release_pending(uid, blob);
}
let node = match fetched {
Ok(Some(node)) => node, Ok(Some(node)) => node,
// Trashed or deleted under us: the tree will hear it from the event // Trashed or deleted under us: the tree will hear it from the event
// sync, which is better placed to unhook the inode than we are. // sync, which is better placed to unhook the inode than we are.
Ok(None) => return, Ok(None) => return None,
Err(e) => { Err(e) => {
warn!(%uid, error = %e, warn!(%uid, error = %e,
"refreshing metadata after an upload failed; \ "refreshing metadata after an upload failed; \
the next write to this file may conflict with itself"); the next write to this file may conflict with itself");
return; return None;
} }
}; };
adopt(&node);
// This function's whole job is to bring the tree level with the revision // This function's whole job is to bring the tree level with the revision
// we just sealed, so the feed's report of that revision has nothing left // we just sealed, so the feed's report of that revision has nothing left
// to tell us. Claimed here rather than at the upload call so it is only // to tell us. Claimed here rather than at the upload call so it is only
@@ -1707,18 +1804,14 @@ impl Core {
// This is the open-handle counterpart of `rebaseline_pending`, which // This is the open-handle counterpart of `rebaseline_pending`, which
// covers the same gap for *queued* ops. // covers the same gap for *queued* ops.
{ {
let sealed_mtime = node.modification_time;
let sealed_size = node_size(&node);
let sealed_rev = node_revision_id(&node);
// Remember it as ours, so a queued write that opened over an earlier // Remember it as ours, so a queued write that opened over an earlier
// revision recognises this as a self-supersede rather than a foreign // revision recognises this as a self-supersede rather than a foreign
// change and does not fork (B70 layer B; consulted in // change and does not fork (B70 layer B; consulted in
// `revision_conflict`). // `revision_conflict`).
if let Some(rev) = sealed_rev.clone() { if let Some(rev) = node_revision_id(&node) {
self.own_sealed_revs.lock().insert(uid.clone(), rev.clone()); // In the table, not in memory: losing it at shutdown reopened
// Write through: losing this at shutdown reopened the fork // the fork window B70 layer B closed, for the first drain
// window B70 layer B closed, for the first drain after a // after a restart.
// restart.
if let Err(e) = self if let Err(e) = self
.db .db
.set_own_sealed_rev(&uid.to_string(), &rev, now_millis()) .set_own_sealed_rev(&uid.to_string(), &rev, now_millis())
@@ -1726,23 +1819,14 @@ impl Core {
warn!(%uid, error = %e, "persisting our sealed revision failed"); warn!(%uid, error = %e, "persisting our sealed revision failed");
} }
} }
self.for_each_state(|st| { self.rebase_open_writes(uid, &node);
for aw in st.active_writes.values_mut() {
if aw.uid == *uid {
aw.base_mtime = sealed_mtime;
aw.base_size = sealed_size;
aw.base_revision_id = sealed_rev.clone();
debug!(%uid, mtime = sealed_mtime, size = sealed_size,
"rebased open write handle onto the revision just uploaded");
}
}
});
} }
self.release_pending(uid, blob);
// Ordered so that a write queued *during* the fetch above is still // Ordered so that a write queued *during* the fetch above is still
// caught: it took its baseline from the node's optimistic stamp, and // caught: it took its baseline from the node's optimistic stamp, and
// this overwrites it with the revision the server actually holds. // this overwrites it with the revision the server actually holds.
if self.rebaseline_pending(uid, &node) { if self.rebaseline_pending(uid, &node) {
return; return Some(node);
} }
self.for_each_state(|st| { self.for_each_state(|st| {
let Some(parent) = st let Some(parent) = st
@@ -1755,6 +1839,21 @@ impl Core {
}; };
st.intern(parent, node.clone()); st.intern(parent, node.clone());
}); });
Some(node)
}
/// Rebase every write handle open on `uid` onto `node`, a revision this
/// daemon has just put there itself (`WriteHandle::rebase_onto`).
fn rebase_open_writes(&self, uid: &NodeUid, node: &Node) {
self.for_each_state(|st| {
for aw in st.active_writes.values_mut() {
if aw.uid == *uid {
aw.rebase_onto(node);
debug!(%uid, mtime = aw.base_mtime, size = aw.base_size,
"rebased open write handle onto the revision just landed");
}
}
});
} }
/// Point a still-queued write at the revision this upload just sealed, and /// Point a still-queued write at the revision this upload just sealed, and
@@ -1845,6 +1944,20 @@ fn is_own_self_supersede(complete: bool, remote_rev: Option<&str>, own_rev: Opti
complete && remote_rev.is_some() && remote_rev == own_rev complete && remote_rev.is_some() && remote_rev == own_rev
} }
/// Whether Drive has `node` neither where its queued rename found it nor where
/// the rename takes it (`parent`, `name`), so someone else moved or renamed it
/// in between. Unknown on a row without the original name.
fn moved_elsewhere(meta: &RenameMeta, node: &Node, parent: &NodeUid, name: &str) -> bool {
let Some(original_name) = meta.original_name.as_deref() else {
return false;
};
let at = |parent: &str, name: &str| {
node.parent_uid.as_ref().map(ToString::to_string).as_deref() == Some(parent)
&& node.name == name
};
!at(&meta.original_parent_uid, original_name) && !at(&parent.to_string(), name)
}
/// What the park sweep does with one parked create whose node still exists. /// What the park sweep does with one parked create whose node still exists.
#[derive(Debug, PartialEq, Eq)] #[derive(Debug, PartialEq, Eq)]
enum ParkVerdict { enum ParkVerdict {
@@ -1888,9 +2001,23 @@ fn adoptable(is_dir: bool, twin: &Node, op_created_ms: i64) -> bool {
} }
} }
/// What share of the cache budget an unpinned upload may take and still be kept
/// as cached content once it lands: a sixteenth, 320 MiB of the default 5 GiB.
/// A larger file would push out much of what the user read lately, for bytes
/// they may never read back.
const LANDED_KEEP_SHARE: u64 = 16;
/// Whether an upload of `len` bytes that just landed stays as the file's cached
/// content. A pinned file always does, and so does any file when the budget is
/// unlimited (`0`).
fn keeps_landed_upload(pinned: bool, len: u64, budget: u64) -> bool {
pinned || budget == 0 || len <= budget / LANDED_KEEP_SHARE
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use crate::state::WriteHandle;
use pdfs_core::Access; use pdfs_core::Access;
use pdfs_core::db::Db; use pdfs_core::db::Db;
use proton_drive_rs::NodeKind; use proton_drive_rs::NodeKind;
@@ -2002,6 +2129,7 @@ mod tests {
op.meta_json = Some( op.meta_json = Some(
serde_json::to_string(&RenameMeta { serde_json::to_string(&RenameMeta {
original_parent_uid: source.uid.to_string(), original_parent_uid: source.uid.to_string(),
original_name: Some(moved.name.clone()),
}) })
.unwrap(), .unwrap(),
); );
@@ -2290,6 +2418,37 @@ mod tests {
)); ));
} }
#[test]
fn a_landed_upload_stays_cached_unless_it_would_crowd_the_cache_out() {
const GIB: u64 = 1 << 30;
let budget = 5 * GIB;
// An edited document or a source file: read back from disk, not Drive.
assert!(keeps_landed_upload(false, 40_000, budget));
assert!(keeps_landed_upload(false, budget / 16, budget));
// A video the user copied in would push out what they read lately.
assert!(!keeps_landed_upload(false, budget / 16 + 1, budget));
// Pinned content is kept whatever its size, and so is everything when
// the cache has no cap.
assert!(keeps_landed_upload(true, 4 * GIB, budget));
assert!(keeps_landed_upload(false, 4 * GIB, 0));
}
#[test]
fn a_landing_upload_counts_until_its_guard_goes() {
let count = Arc::new(AtomicU64::new(0));
let first = LandingUpload::begin(&count);
let landed = || -> Result<(), ()> {
let _second = LandingUpload::begin(&count);
assert_eq!(count.load(Ordering::SeqCst), 2);
Err(())
};
// Uncounted on an early return too, or the queue never reads empty.
assert!(landed().is_err());
assert_eq!(count.load(Ordering::SeqCst), 1);
drop(first);
assert_eq!(count.load(Ordering::SeqCst), 0);
}
fn baseline(mtime: i64, size: u64, rev: Option<&str>) -> Baseline { fn baseline(mtime: i64, size: u64, rev: Option<&str>) -> Baseline {
Baseline { Baseline {
mtime, mtime,
@@ -2317,6 +2476,26 @@ mod tests {
assert!(revision_changed(&base, &remote).is_some()); assert!(revision_changed(&base, &remote).is_some());
} }
/// The reported shape (docs/BUGS.md B113): a downloader creates a file
/// through the mount, and the file is moved before its write drains. The
/// move re-stamps the node's mtime on the server (1790955809 -> 1790956593),
/// but the revision is still the empty one the create minted.
#[test]
fn a_move_before_the_first_write_drains_is_not_a_conflict() {
let minted = file_node(1_790_955_809, 0, Some("empty"));
let path = std::env::temp_dir().join(format!("pdfs-drain-test-{}", std::process::id()));
let file = File::create(&path).unwrap();
let _ = std::fs::remove_file(&path);
let h = WriteHandle::created(2, &minted, file, path);
let base = baseline(h.base_mtime, h.base_size, h.base_revision_id.as_deref());
let moved = file_node(1_790_956_593, 0, Some("empty"));
assert_eq!(revision_changed(&base, &moved), None);
// Another device writing it is still a conflict.
let edited = file_node(1_790_956_593, 0, Some("theirs"));
assert!(revision_changed(&base, &edited).is_some());
}
#[test] #[test]
fn without_ids_it_falls_back_to_mtime_and_size() { fn without_ids_it_falls_back_to_mtime_and_size() {
// Old sidecar (no revision id): the (mtime, size) tuple still governs. // Old sidecar (no revision id): the (mtime, size) tuple still governs.
+466
View File
@@ -0,0 +1,466 @@
//! The calls the sync core makes to Drive, behind a trait.
//!
//! `Core` used to hold a concrete [`ProtonDriveClient`], so no test could make
//! Drive answer late, answer stale, or not answer at all, and the interleavings
//! behind most recent bugs could only be found on a real account over a slow
//! link (`docs/MILESTONE-3.0.0.md` §2, §8.1). [`DriveApi`] is the part of the
//! client the mount, the drain, the event feed and the mirror engine use; the
//! real client implements it by forwarding, and the simulation tests implement
//! it with an in-memory Drive that injects Drive's known faults.
//!
//! Sharing, Photos, devices, invitations and public links stay on the concrete
//! client (`Core::client`) until a test needs them faked.
//!
//! The method names match the client's own, so a call site reads the same
//! whichever it goes through. The client's generic methods (a reader or writer
//! of any type) cannot be part of an object-safe trait, so the trait takes them
//! as trait objects and `impl dyn DriveApi` puts the generic signatures back.
use std::io::{Read, Write};
use std::sync::Arc;
use async_trait::async_trait;
use futures::StreamExt as _;
use futures::stream::BoxStream;
use proton_drive_rs::proton_sdk::account::Quota;
use proton_drive_rs::proton_sdk::error::Result;
use proton_drive_rs::proton_sdk::ids::{DriveEventId, NodeUid};
use proton_drive_rs::{
DriveEvent, DriveEventScopeId, Node, NodeMoveItem, ProtonDriveClient, RevisionReader, Thumbnail,
};
/// One outcome per node of a batch request; see [`ProtonDriveClient::trash_nodes`].
pub(crate) type NodeOutcomes = Vec<(NodeUid, Result<()>)>;
/// The outcomes of a streaming batch request, one item per node as it lands.
pub(crate) type OutcomeStream<'a> = BoxStream<'a, Result<(NodeUid, Result<()>)>>;
/// Drive as the sync core sees it. See the module documentation.
#[async_trait]
pub(crate) trait DriveApi: Send + Sync {
async fn get_my_files_folder(&self) -> Result<Node>;
async fn get_node(&self, uid: &NodeUid) -> Result<Option<Node>>;
async fn enumerate_nodes(&self, uids: &[NodeUid]) -> Result<Vec<Node>>;
/// As [`DriveApi::enumerate_nodes`], but files carry no claimed size or
/// modification time; see [`ProtonDriveClient::enumerate_nodes_light`].
async fn enumerate_nodes_light(&self, uids: &[NodeUid]) -> Result<Vec<Node>>;
async fn enumerate_folder_children_node_uids(
&self,
folder_uid: &NodeUid,
) -> Result<Vec<NodeUid>>;
async fn enumerate_events(
&self,
scope: &DriveEventScopeId,
cursor: Option<&DriveEventId>,
) -> Result<Vec<DriveEvent>>;
async fn invalidate_caches_for_event(&self, event: &DriveEvent) -> Result<()>;
async fn create_folder(
&self,
parent_uid: &NodeUid,
name: &str,
last_modification_time: Option<i64>,
) -> Result<NodeUid>;
async fn upload_file(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
contents: &[u8],
) -> Result<NodeUid>;
#[allow(clippy::too_many_arguments)]
async fn upload_file_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid>;
#[allow(clippy::too_many_arguments)]
async fn upload_file_replacing_draft_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid>;
async fn upload_new_revision_from_dyn(
&self,
file_uid: &NodeUid,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()>;
async fn open_revision(&self, uid: &NodeUid) -> Result<Arc<dyn RevisionRead>>;
async fn download_file_to_dyn(
&self,
uid: &NodeUid,
output: &mut (dyn Write + Send),
) -> Result<()>;
async fn download_revision_to_dyn(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut (dyn Write + Send),
) -> Result<()>;
async fn rename_node(
&self,
uid: &NodeUid,
new_name: &str,
new_media_type: Option<&str>,
) -> Result<()>;
async fn move_node(&self, uid: &NodeUid, new_parent: &NodeUid) -> Result<()>;
fn move_nodes_streaming<'a>(
&'a self,
items: Vec<NodeMoveItem>,
new_parent: NodeUid,
) -> OutcomeStream<'a>;
async fn trash_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes>;
async fn restore_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes>;
fn restore_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a>;
fn delete_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a>;
async fn enumerate_trash_node_uids(&self) -> Result<Vec<NodeUid>>;
/// The account's storage, across every Proton product.
async fn quota(&self) -> Result<Quota>;
}
/// The generic signatures of the client, over the object-safe ones above.
impl dyn DriveApi {
#[allow(clippy::too_many_arguments)]
pub(crate) async fn upload_file_from<R: Read + Send>(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
self.upload_file_from_dyn(
parent_uid,
name,
media_type,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
#[allow(clippy::too_many_arguments)]
pub(crate) async fn upload_file_replacing_draft_from<R: Read + Send>(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
self.upload_file_replacing_draft_from_dyn(
parent_uid,
name,
media_type,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
pub(crate) async fn upload_new_revision_from<R: Read + Send>(
&self,
file_uid: &NodeUid,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()> {
self.upload_new_revision_from_dyn(
file_uid,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
)
.await
}
pub(crate) async fn download_file_to<W: Write + Send>(
&self,
uid: &NodeUid,
output: &mut W,
) -> Result<()> {
self.download_file_to_dyn(uid, output).await
}
pub(crate) async fn download_revision_to<W: Write + Send>(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut W,
) -> Result<()> {
self.download_revision_to_dyn(file_uid, revision_id, writer)
.await
}
}
/// An open revision that block reads are served from; see [`RevisionReader`].
#[async_trait]
pub(crate) trait RevisionRead: Send + Sync {
/// Plaintext size of the revision.
fn size(&self) -> u64;
/// Plaintext size of each block, in block order.
fn block_sizes(&self) -> &[u64];
async fn read_at(&self, offset: u64, length: u64) -> Result<Vec<u8>>;
}
#[async_trait]
impl RevisionRead for RevisionReader {
fn size(&self) -> u64 {
RevisionReader::size(self)
}
fn block_sizes(&self) -> &[u64] {
RevisionReader::block_sizes(self)
}
async fn read_at(&self, offset: u64, length: u64) -> Result<Vec<u8>> {
RevisionReader::read_at(self, offset, length).await
}
}
#[async_trait]
impl DriveApi for ProtonDriveClient {
async fn get_my_files_folder(&self) -> Result<Node> {
ProtonDriveClient::get_my_files_folder(self).await
}
async fn get_node(&self, uid: &NodeUid) -> Result<Option<Node>> {
ProtonDriveClient::get_node(self, uid).await
}
async fn enumerate_nodes(&self, uids: &[NodeUid]) -> Result<Vec<Node>> {
ProtonDriveClient::enumerate_nodes(self, uids).await
}
async fn enumerate_nodes_light(&self, uids: &[NodeUid]) -> Result<Vec<Node>> {
ProtonDriveClient::enumerate_nodes_light(self, uids).await
}
async fn enumerate_folder_children_node_uids(
&self,
folder_uid: &NodeUid,
) -> Result<Vec<NodeUid>> {
ProtonDriveClient::enumerate_folder_children_node_uids(self, folder_uid).await
}
async fn enumerate_events(
&self,
scope: &DriveEventScopeId,
cursor: Option<&DriveEventId>,
) -> Result<Vec<DriveEvent>> {
ProtonDriveClient::enumerate_events(self, scope, cursor).await
}
async fn invalidate_caches_for_event(&self, event: &DriveEvent) -> Result<()> {
ProtonDriveClient::invalidate_caches_for_event(self, event).await
}
async fn create_folder(
&self,
parent_uid: &NodeUid,
name: &str,
last_modification_time: Option<i64>,
) -> Result<NodeUid> {
ProtonDriveClient::create_folder(self, parent_uid, name, last_modification_time).await
}
async fn upload_file(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
contents: &[u8],
) -> Result<NodeUid> {
ProtonDriveClient::upload_file(self, parent_uid, name, media_type, contents).await
}
async fn upload_file_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
ProtonDriveClient::upload_file_from(
self,
parent_uid,
name,
media_type,
reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
async fn upload_file_replacing_draft_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
ProtonDriveClient::upload_file_replacing_draft_from(
self,
parent_uid,
name,
media_type,
reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
async fn upload_new_revision_from_dyn(
&self,
file_uid: &NodeUid,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()> {
ProtonDriveClient::upload_new_revision_from(
self,
file_uid,
reader,
intended_size,
thumbnails,
last_modification_time,
)
.await
}
async fn open_revision(&self, uid: &NodeUid) -> Result<Arc<dyn RevisionRead>> {
let reader = ProtonDriveClient::open_revision(self, uid).await?;
Ok(Arc::new(reader))
}
async fn download_file_to_dyn(
&self,
uid: &NodeUid,
output: &mut (dyn Write + Send),
) -> Result<()> {
ProtonDriveClient::download_file_to(self, uid, &mut &mut *output).await
}
async fn download_revision_to_dyn(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut (dyn Write + Send),
) -> Result<()> {
ProtonDriveClient::download_revision_to(self, file_uid, revision_id, &mut &mut *writer)
.await
}
async fn rename_node(
&self,
uid: &NodeUid,
new_name: &str,
new_media_type: Option<&str>,
) -> Result<()> {
ProtonDriveClient::rename_node(self, uid, new_name, new_media_type).await
}
async fn move_node(&self, uid: &NodeUid, new_parent: &NodeUid) -> Result<()> {
ProtonDriveClient::move_node(self, uid, new_parent).await
}
fn move_nodes_streaming<'a>(
&'a self,
items: Vec<NodeMoveItem>,
new_parent: NodeUid,
) -> OutcomeStream<'a> {
ProtonDriveClient::move_nodes_streaming(self, items, new_parent).boxed()
}
async fn trash_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes> {
ProtonDriveClient::trash_nodes(self, uids).await
}
async fn restore_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes> {
ProtonDriveClient::restore_nodes(self, uids).await
}
fn restore_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a> {
ProtonDriveClient::restore_nodes_streaming(self, uids).boxed()
}
fn delete_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a> {
ProtonDriveClient::delete_nodes_streaming(self, uids).boxed()
}
async fn enumerate_trash_node_uids(&self) -> Result<Vec<NodeUid>> {
ProtonDriveClient::enumerate_trash_node_uids(self).await
}
async fn quota(&self) -> Result<Quota> {
ProtonDriveClient::quota(self).await
}
}
+99 -156
View File
@@ -72,7 +72,7 @@ impl Filesystem for ProtonFs {
} }
fn lookup(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEntry) { fn lookup(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEntry) {
let parent = parent.0; let parent = self.tree(parent).0;
let name = match fuse_name(name) { let name = match fuse_name(name) {
Ok(name) => name, Ok(name) => name,
Err(error) => { Err(error) => {
@@ -95,11 +95,13 @@ impl Filesystem for ProtonFs {
} }
fn forget(&self, _req: &Request, ino: INodeNo, nlookup: u64) { fn forget(&self, _req: &Request, ino: INodeNo, nlookup: u64) {
let ino = self.tree(ino);
let mut st = self.core.state(); let mut st = self.core.state();
st.forget_lookup(ino.0, nlookup); st.forget_lookup(ino.0, nlookup);
} }
fn getattr(&self, _req: &Request, ino: INodeNo, _fh: Option<FileHandle>, reply: ReplyAttr) { fn getattr(&self, _req: &Request, ino: INodeNo, _fh: Option<FileHandle>, reply: ReplyAttr) {
let ino = self.tree(ino);
let (attr, provisional_in) = { let (attr, provisional_in) = {
let st = self.core.state(); let st = self.core.state();
match st.entries.get(&ino.0) { match st.entries.get(&ino.0) {
@@ -175,7 +177,7 @@ impl Filesystem for ProtonFs {
/// Enumerating a cold folder is a remote call, so it goes to a worker for /// Enumerating a cold folder is a remote call, so it goes to a worker for
/// the same reason `lookup` and `readdir` do. /// the same reason `lookup` and `readdir` do.
fn opendir(&self, _req: &Request, ino: INodeNo, _flags: OpenFlags, reply: ReplyOpen) { fn opendir(&self, _req: &Request, ino: INodeNo, _flags: OpenFlags, reply: ReplyOpen) {
let ino = ino.0; let ino = self.tree(ino).0;
if self.core.children_cached(ino) { if self.core.children_cached(ino) {
self.serve_opendir(ino, reply); self.serve_opendir(ino, reply);
return; return;
@@ -194,13 +196,13 @@ impl Filesystem for ProtonFs {
offset: u64, offset: u64,
reply: ReplyDirectory, reply: ReplyDirectory,
) { ) {
let ino = ino.0; let ino = self.tree(ino).0;
// The snapshot is the whole point: served from it, this is a pure // The snapshot is the whole point: served from it, this is a pure
// in-memory walk with no chance of a remote call, so it never needs a // in-memory walk with no chance of a remote call, so it never needs a
// worker. Only a handle whose `opendir` failed to snapshot (or a kernel // worker. Only a handle whose `opendir` failed to snapshot (or a kernel
// that issued `readdir` without one) falls back to the live path. // that issued `readdir` without one) falls back to the live path.
if let Some(listing) = self.core.state().dir_snapshots.get(&fh.0).cloned() { if let Some(listing) = self.core.state().dir_snapshots.get(&fh.0).cloned() {
reply_listing(&listing, offset, reply); reply_listing(&listing, offset, self.root, reply);
return; return;
} }
// Same split as `lookup`: only the cold, remote-enumerating path pays // Same split as `lookup`: only the cold, remote-enumerating path pays
@@ -232,6 +234,7 @@ impl Filesystem for ProtonFs {
/// a blob that is the whole file — so it is served from a worker rather than /// a blob that is the whole file — so it is served from a worker rather than
/// stalling every other request on the mount behind it. /// stalling every other request on the mount behind it.
fn open(&self, _req: &Request, ino: INodeNo, flags: OpenFlags, reply: ReplyOpen) { fn open(&self, _req: &Request, ino: INodeNo, flags: OpenFlags, reply: ReplyOpen) {
let ino = self.tree(ino);
if flags.acc_mode() == OpenAccMode::O_RDONLY { if flags.acc_mode() == OpenAccMode::O_RDONLY {
self.serve_open(ino, flags, reply); self.serve_open(ino, flags, reply);
return; return;
@@ -253,6 +256,7 @@ impl Filesystem for ProtonFs {
_lock_owner: Option<LockOwner>, _lock_owner: Option<LockOwner>,
reply: ReplyData, reply: ReplyData,
) { ) {
let ino = self.tree(ino);
// A file open for writing is served from its handle so reads see the // A file open for writing is served from its handle so reads see the
// in-flight (possibly unsaved) content: authored bytes come from the // in-flight (possibly unsaved) content: authored bytes come from the
// scratch file, untouched bytes from the base. // scratch file, untouched bytes from the base.
@@ -364,7 +368,7 @@ impl Filesystem for ProtonFs {
_flags: i32, _flags: i32,
reply: ReplyCreate, reply: ReplyCreate,
) { ) {
let parent = parent.0; let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) { if let Err(error) = self.core.require_writable(parent) {
reply.error(error); reply.error(error);
return; return;
@@ -400,6 +404,7 @@ impl Filesystem for ProtonFs {
_lock_owner: Option<LockOwner>, _lock_owner: Option<LockOwner>,
reply: ReplyWrite, reply: ReplyWrite,
) { ) {
let ino = self.tree(ino);
if let Err(error) = self.core.require_writable(ino.0) { if let Err(error) = self.core.require_writable(ino.0) {
reply.error(error); reply.error(error);
return; return;
@@ -477,6 +482,7 @@ impl Filesystem for ProtonFs {
_flags: Option<BsdFileFlags>, _flags: Option<BsdFileFlags>,
reply: ReplyAttr, reply: ReplyAttr,
) { ) {
let ino = self.tree(ino);
if let Err(error) = self.core.require_writable(ino.0) { if let Err(error) = self.core.require_writable(ino.0) {
reply.error(error); reply.error(error);
return; return;
@@ -544,6 +550,7 @@ impl Filesystem for ProtonFs {
mode: i32, mode: i32,
reply: ReplyEmpty, reply: ReplyEmpty,
) { ) {
let ino = self.tree(ino);
if let Err(error) = self.core.require_writable(ino.0) { if let Err(error) = self.core.require_writable(ino.0) {
reply.error(error); reply.error(error);
return; return;
@@ -640,16 +647,17 @@ impl Filesystem for ProtonFs {
fn release( fn release(
&self, &self,
_req: &Request, _req: &Request,
_ino: INodeNo, ino: INodeNo,
fh: FileHandle, fh: FileHandle,
_flags: OpenFlags, _flags: OpenFlags,
_lock_owner: Option<LockOwner>, _lock_owner: Option<LockOwner>,
_flush: bool, _flush: bool,
reply: ReplyEmpty, reply: ReplyEmpty,
) { ) {
let ino = self.tree(ino);
let (handle, unlinked_uid) = { let (handle, unlinked_uid) = {
let mut st = self.core.state(); let mut st = self.core.state();
let unlinked_uid = release_unlinked_entry(&mut st, _ino.0); let unlinked_uid = release_unlinked_entry(&mut st, ino.0);
let ino = st.handles.remove(&fh.0); let ino = st.handles.remove(&fh.0);
let h = ino.and_then(|i| { let h = ino.and_then(|i| {
let aw = st.active_writes.get_mut(&i)?; let aw = st.active_writes.get_mut(&i)?;
@@ -680,7 +688,7 @@ impl Filesystem for ProtonFs {
} }
fn unlink(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEmpty) { fn unlink(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEmpty) {
let parent = parent.0; let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) { if let Err(error) = self.core.require_writable(parent) {
reply.error(error); reply.error(error);
return; return;
@@ -700,7 +708,7 @@ impl Filesystem for ProtonFs {
} }
fn rmdir(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEmpty) { fn rmdir(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEmpty) {
let parent = parent.0; let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) { if let Err(error) = self.core.require_writable(parent) {
reply.error(error); reply.error(error);
return; return;
@@ -729,7 +737,7 @@ impl Filesystem for ProtonFs {
_umask: u32, _umask: u32,
reply: ReplyEntry, reply: ReplyEntry,
) { ) {
let parent = parent.0; let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) { if let Err(error) = self.core.require_writable(parent) {
reply.error(error); reply.error(error);
return; return;
@@ -760,8 +768,8 @@ impl Filesystem for ProtonFs {
flags: RenameFlags, flags: RenameFlags,
reply: ReplyEmpty, reply: ReplyEmpty,
) { ) {
let parent = parent.0; let parent = self.tree(parent).0;
let newparent = newparent.0; let newparent = self.tree(newparent).0;
if let Err(error) = self if let Err(error) = self
.core .core
.require_writable(parent) .require_writable(parent)
@@ -794,6 +802,7 @@ impl Filesystem for ProtonFs {
} }
fn access(&self, _req: &Request, ino: INodeNo, mask: AccessFlags, reply: ReplyEmpty) { fn access(&self, _req: &Request, ino: INodeNo, mask: AccessFlags, reply: ReplyEmpty) {
let ino = self.tree(ino);
let state = self.core.state(); let state = self.core.state();
let Some(entry) = state.entries.get(&ino.0) else { let Some(entry) = state.entries.get(&ino.0) else {
reply.error(Errno::ENOENT); reply.error(Errno::ENOENT);
@@ -807,6 +816,7 @@ impl Filesystem for ProtonFs {
} }
fn getxattr(&self, _req: &Request, ino: INodeNo, name: &OsStr, size: u32, reply: ReplyXattr) { fn getxattr(&self, _req: &Request, ino: INodeNo, name: &OsStr, size: u32, reply: ReplyXattr) {
let ino = self.tree(ino);
// Always off the dispatch loop: a miss goes to the wire, and a lister // Always off the dispatch loop: a miss goes to the wire, and a lister
// that stats a directory issues one of these per file per advertised // that stats a directory issues one of these per file per advertised
// name — inline, that serialized the whole mount behind ~186 ms of // name — inline, that serialized the whole mount behind ~186 ms of
@@ -819,6 +829,7 @@ impl Filesystem for ProtonFs {
} }
fn listxattr(&self, _req: &Request, ino: INodeNo, size: u32, reply: ReplyXattr) { fn listxattr(&self, _req: &Request, ino: INodeNo, size: u32, reply: ReplyXattr) {
let ino = self.tree(ino);
self.serve_listxattr(ino, size, reply); self.serve_listxattr(ino, size, reply);
} }
@@ -968,6 +979,9 @@ pub struct ProtonFs {
core: Core, core: Core,
uid: u32, uid: u32,
gid: u32, gid: u32,
/// The tree inode of this session's root, which the kernel knows as
/// [`ROOT_INO`].
root: u64,
} }
/// How many times a write `open` will re-read its base after a `release` /// How many times a write `open` will re-read its base after a `release`
@@ -1008,59 +1022,55 @@ fn build_listing(st: &State, ino: u64) -> DirListing {
/// The cookie handed back is the index of the *next* entry, so a resumed call /// The cookie handed back is the index of the *next* entry, so a resumed call
/// continues exactly where the last one stopped — which is only true because /// continues exactly where the last one stopped — which is only true because
/// the listing behind those indexes is a snapshot (bugs.md B43). /// the listing behind those indexes is a snapshot (bugs.md B43).
fn reply_listing(listing: &DirListing, offset: u64, mut reply: ReplyDirectory) { fn reply_listing(listing: &DirListing, offset: u64, root: u64, mut reply: ReplyDirectory) {
for (i, (ino, is_dir, name)) in listing.iter().enumerate().skip(offset as usize) { for (i, (ino, is_dir, name)) in listing.iter().enumerate().skip(offset as usize) {
let ft = if *is_dir { let ft = if *is_dir {
FileType::Directory FileType::Directory
} else { } else {
FileType::RegularFile FileType::RegularFile
}; };
if reply.add(INodeNo(*ino), (i + 1) as u64, ft, name) { if reply.add(kernel_ino(root, *ino), (i + 1) as u64, ft, name) {
break; break;
} }
} }
reply.ok(); reply.ok();
} }
/// The inode the kernel knows tree inode `ino` by, in a session rooted at
/// `root`. The inverse of [`ProtonFs::tree`].
pub(crate) fn kernel_ino(root: u64, ino: u64) -> INodeNo {
INodeNo(if ino == root { ROOT_INO } else { ino })
}
pub(crate) fn access_allowed(is_dir: bool, access: Access, mask: AccessFlags) -> bool { pub(crate) fn access_allowed(is_dir: bool, access: Access, mask: AccessFlags) -> bool {
(!mask.contains(AccessFlags::W_OK) || access.writable()) (!mask.contains(AccessFlags::W_OK) || access.writable())
&& (!mask.contains(AccessFlags::X_OK) || is_dir) && (!mask.contains(AccessFlags::X_OK) || is_dir)
} }
impl ProtonFs { impl ProtonFs {
/// Build the filesystem rooted at `root` (the user's My Files folder). /// Build the filesystem rooted at `root` (the user's My Files folder, or
/// the folder an on-demand location shows).
pub(super) fn new(core: Core, root: Node) -> Self { pub(super) fn new(core: Core, root: Node) -> Self {
{ let root = core.state().install_root(root);
let mut st = core.state(); let _ = core.root.set(root);
if let Err(e) = st.db.upsert_node(&root) {
warn!(uid = %root.uid, error = %e, "db upsert root failed");
}
st.by_uid.insert(root.uid.clone(), ROOT_INO);
// The filesystem root is always owned — it must never be
// classified as a shared root, even when the API returns
// membership data (which happens when the owner shares the
// folder with someone else). Clean up any stale share_access
// row that a previous downgrade_all_share_access may have
// written for this uid, so hydrate_access cannot pick it up.
let access = Access::Owner;
st.share_access.remove(&root.uid);
let _ = st.db.delete_share_access(&root.uid);
st.entries.insert(
ROOT_INO,
Entry {
uid: root.uid.clone(),
parent: ROOT_INO,
node: root,
access,
lookup_count: 1,
open_count: 0,
unlinked: false,
},
);
}
// SAFETY: geteuid/getegid are infallible and have no preconditions. // SAFETY: geteuid/getegid are infallible and have no preconditions.
let (uid, gid) = unsafe { (libc::geteuid(), libc::getegid()) }; let (uid, gid) = unsafe { (libc::geteuid(), libc::getegid()) };
Self { core, uid, gid } Self {
core,
uid,
gid,
root,
}
}
/// The tree inode the kernel means by `ino`: this session's root for
/// [`ROOT_INO`], and every other inode as it is.
fn tree(&self, ino: INodeNo) -> INodeNo {
if ino.0 == ROOT_INO {
INodeNo(self.root)
} else {
ino
}
} }
/// The body of [`Filesystem::lookup`], on whichever thread ends up serving it. /// The body of [`Filesystem::lookup`], on whichever thread ends up serving it.
@@ -1169,7 +1179,7 @@ impl ProtonFs {
let st = self.core.state(); let st = self.core.state();
let listing = build_listing(&st, ino); let listing = build_listing(&st, ino);
drop(st); drop(st);
reply_listing(&listing, offset, reply); reply_listing(&listing, offset, self.root, reply);
} }
fn attr(&self, ino: u64, node: &Node, access: Access) -> FileAttr { fn attr(&self, ino: u64, node: &Node, access: Access) -> FileAttr {
@@ -1182,7 +1192,7 @@ impl ProtonFs {
let mtime = unix_secs(node.modification_time); let mtime = unix_secs(node.modification_time);
let crtime = unix_secs(node.creation_time); let crtime = unix_secs(node.creation_time);
FileAttr { FileAttr {
ino: INodeNo(ino), ino: kernel_ino(self.root, ino),
size, size,
blocks: size.div_ceil(512), blocks: size.div_ceil(512),
atime: mtime, atime: mtime,
@@ -1435,7 +1445,7 @@ impl ProtonFs {
len, len,
base_size, base_size,
base_mtime, base_mtime,
complete: authored == [(0, len)], complete: StagedWrite::covers(&authored, len),
authored, authored,
based_on: self based_on: self
.core .core
@@ -1511,22 +1521,15 @@ impl ProtonFs {
} }
// Create an empty file on the remote so it has a real uid immediately; // Create an empty file on the remote so it has a real uid immediately;
// written bytes are buffered and sealed as a new revision on close. // written bytes are buffered and sealed as a new revision on close.
match self.core.block_on_bounded(self.core.client.upload_file( match self.core.block_on_bounded(self.core.drive.upload_file(
&parent_uid, &parent_uid,
name, name,
media_type_for(name), media_type_for(name),
b"", b"",
)) { )) {
Ok(new_uid) => match self.core.fetch_node(&new_uid) { // A file that exists remotely but could not be read back is
Ok(n) => Some(n), // queued, and the queued create finds it by name and adopts it.
// The file exists remotely but we could not read it back. Ok(new_uid) => self.core.fetch_minted_node(&new_uid, name),
// The queued create finds it by name and adopts it.
Err(e) if e == Errno::EIO && !self.core.is_online() => None,
Err(e) => {
reply.error(e);
return;
}
},
Err(e) if self.core.lost_link(&e, "create") => None, Err(e) if self.core.lost_link(&e, "create") => None,
Err(e) => { Err(e) => {
error!(%parent_uid, name, error = %e, "create file failed"); error!(%parent_uid, name, error = %e, "create file failed");
@@ -1550,26 +1553,16 @@ impl ProtonFs {
} }
}, },
}; };
let new_uid = node.uid.clone();
// The base this handle writes over is the node as it actually exists —
// the empty file the server just minted — so its modification time comes
// from the node, never from the local clock. `queue_revision` turns this
// into `StagedWrite::based_on` and the drain compares that against the
// remote: a `now_secs()` here differs from the server's stamp by however
// long the create round-trip took, so the first write to a brand-new file
// conflicts with its own create whenever the second happens to tick in
// between.
let base_mtime = node.modification_time;
let (file, path) = match self.core.cache.create_scratch() { let (file, path) = match self.core.cache.create_scratch() {
Ok(x) => x, Ok(x) => x,
Err(e) => { Err(e) => {
error!(%new_uid, error = %e, "create scratch file failed"); error!(uid = %node.uid, error = %e, "create scratch file failed");
reply.error(Errno::EIO); reply.error(Errno::EIO);
return; return;
} }
}; };
let mut st = self.core.state(); let mut st = self.core.state();
let ino = st.intern(parent, node); let ino = st.intern(parent, node.clone());
if let Some(entry) = st.entries.get_mut(&ino) { if let Some(entry) = st.entries.get_mut(&ino) {
entry.open_count = entry.open_count.saturating_add(1); entry.open_count = entry.open_count.saturating_add(1);
} }
@@ -1580,23 +1573,10 @@ impl ProtonFs {
} }
let fh = st.next_fh; let fh = st.next_fh;
st.next_fh += 1; st.next_fh += 1;
let aw = st.active_writes.entry(ino).or_insert_with(|| { let aw = st
// A brand-new file: empty base, everything written is authored. .active_writes
WriteHandle { .entry(ino)
ino, .or_insert_with(|| WriteHandle::created(ino, &node, file, path));
uid: new_uid,
file: Arc::new(file),
path,
written: Intervals::default(),
len: 0,
base_size: 0,
base_mtime,
// A brand-new file has no sealed remote revision to conflict with.
base_revision_id: None,
dirty: false,
open_count: 0,
}
});
aw.open_count += 1; aw.open_count += 1;
st.handles.insert(fh, ino); st.handles.insert(fh, ino);
let entry = st.entries.get(&ino).unwrap(); let entry = st.entries.get(&ino).unwrap();
@@ -1647,16 +1627,9 @@ impl ProtonFs {
} }
match self match self
.core .core
.block_on_bounded(self.core.client.create_folder(&parent_uid, name, now)) .block_on_bounded(self.core.drive.create_folder(&parent_uid, name, now))
{ {
Ok(new_uid) => match self.core.fetch_node(&new_uid) { Ok(new_uid) => self.core.fetch_minted_node(&new_uid, name),
Ok(n) => Some(n),
Err(e) if e == Errno::EIO && !self.core.is_online() => None,
Err(e) => {
reply.error(e);
return;
}
},
Err(e) if self.core.lost_link(&e, "mkdir") => None, Err(e) if self.core.lost_link(&e, "mkdir") => None,
Err(e) => { Err(e) => {
error!(%parent_uid, name, error = %e, "create folder failed"); error!(%parent_uid, name, error = %e, "create folder failed");
@@ -1981,12 +1954,14 @@ impl ProtonFs {
// now and this handle's is stale. A retry resolves it. // now and this handle's is stale. A retry resolves it.
Ok(false) => { Ok(false) => {
warn!(%uid, name, newname, "queued create vanished under a rename"); warn!(%uid, name, newname, "queued create vanished under a rename");
self.core.restore_replaced(victim.as_ref(), newname); self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EBUSY); reply.error(Errno::EBUSY);
} }
Err(e) => { Err(e) => {
error!(%uid, error = %e, "rewriting a queued create's target failed"); error!(%uid, error = %e, "rewriting a queued create's target failed");
self.core.restore_replaced(victim.as_ref(), newname); self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EIO); reply.error(Errno::EIO);
} }
} }
@@ -2029,16 +2004,17 @@ impl ProtonFs {
// and retain an empty destination readdir page. // and retain an empty destination readdir page.
reply.ok(); reply.ok();
if let Some(notifier) = self.core.notifier.get() { if let Some(notifier) = self.core.notifier.get() {
let _ = notifier.inval_entry(INodeNo(parent), OsStr::new(&name)); let _ = notifier.inval_entry(parent, OsStr::new(&name));
let _ = notifier.inval_entry(INodeNo(newparent), OsStr::new(&newname)); let _ = notifier.inval_entry(newparent, OsStr::new(&newname));
let _ = notifier.inval_inode(INodeNo(parent), 0, 0); let _ = notifier.inval_inode(parent);
if newparent != parent { if newparent != parent {
let _ = notifier.inval_inode(INodeNo(newparent), 0, 0); let _ = notifier.inval_inode(newparent);
} }
} }
} }
Err(e) => { Err(e) => {
self.core.restore_replaced(victim.as_ref(), newname); self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(e); reply.error(e);
} }
} }
@@ -2052,8 +2028,6 @@ impl ProtonFs {
queue_it(reply); queue_it(reply);
return; return;
} }
// Rename first if both halves change. Moving first makes the encrypted
// name requirements stale and repeatedly failed with InvalidRequirements.
// A failure past this point has already trashed any node the rename was // A failure past this point has already trashed any node the rename was
// replacing, so put it back rather than leaving the caller with neither // replacing, so put it back rather than leaving the caller with neither
// the source moved nor the destination intact. // the source moved nor the destination intact.
@@ -2063,68 +2037,37 @@ impl ProtonFs {
&old_parent_uid, &old_parent_uid,
&new_parent_uid, &new_parent_uid,
) { ) {
self.core.restore_replaced(victim.as_ref(), newname); self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(error); reply.error(error);
return; return;
} }
if newname != name // A move that also renames is one request, so the name only has to be
&& let Err(e) = self.core.rename_remote(&uid, newname) // free in the destination, and the reply never stands for half of it.
{ let renamed = if newparent != parent {
let target = (newname != name).then_some(newname);
self.core.move_rename_remote(&uid, &new_parent_uid, target)
} else if newname != name {
self.core.rename_remote(&uid, newname)
} else {
Ok(())
};
if let Err(e) = renamed {
if self.core.lost_link(&e, "rename") { if self.core.lost_link(&e, "rename") {
queue_it(reply); queue_it(reply);
return; return;
} }
error!(%uid, error = %e, "rename failed"); error!(%uid, error = %e, "rename failed");
self.core.restore_replaced(victim.as_ref(), newname); self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EIO); reply.error(Errno::EIO);
return; return;
} }
if newparent != parent { // Through the guard, so the database has both folders unlisted before
let mut attempts = 0u32; // the reply. The next lookup rebuilds a folder still marked listed from
let moved = loop { // the database, which did not have the move yet (docs/BUGS.md B135).
match self
.core
.block_on_bounded(self.core.client.move_node(&uid, &new_parent_uid))
{
Ok(()) => break Ok(()),
Err(e)
if newname != name
&& api_code(&e) == Some(ResponseCode::InvalidRequirements)
&& attempts < 20 =>
{
attempts += 1;
// Requirement propagation routinely takes longer than
// three seconds. Keep a bounded ten-second window; a
// successful POSIX reply must mean both remote halves
// landed, not merely that the second half was queued.
std::thread::sleep(Duration::from_millis(500));
}
Err(e) => break Err(e),
}
};
if let Err(e) = moved {
if self.core.lost_link(&e, "move") {
queue_it(reply);
return;
}
error!(%uid, attempts, error = %e, "move after rename failed");
// The rename half landed in the source directory.
let mut state = self.core.state();
let old_parent_uid = state
.entries
.get(&parent)
.map(|entry| entry.uid.clone())
.unwrap_or_else(|| uid.clone());
state.rename_in_place(ino, parent, &old_parent_uid, newname);
drop(state);
self.core.restore_replaced(victim.as_ref(), newname);
reply.error(Errno::EIO);
return;
}
}
self.core self.core
.state .state()
.lock()
.relocate(ino, parent, newparent, &new_parent_uid, newname); .relocate(ino, parent, newparent, &new_parent_uid, newname);
reply.ok(); reply.ok();
} }
@@ -2200,7 +2143,7 @@ impl ProtonFs {
} }
if let Err(e) = self if let Err(e) = self
.core .core
.block_on_bounded(self.core.client.trash_nodes(std::slice::from_ref(&uid))) .block_on_bounded(self.core.drive.trash_nodes(std::slice::from_ref(&uid)))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
if self.core.lost_link(&e, "trash") { if self.core.lost_link(&e, "trash") {
File diff suppressed because it is too large. Load diff
+4 -3
View File
@@ -17,7 +17,7 @@ use proton_drive_rs::proton_sdk::api::ResponseCode;
use proton_drive_rs::proton_sdk::error::{ProtonApiError, ProtonError}; use proton_drive_rs::proton_sdk::error::{ProtonApiError, ProtonError};
use tracing::{debug, info, warn}; use tracing::{debug, info, warn};
use crate::{Core, ONLINE_PROBE_MAX, ONLINE_PROBE_MIN, ROOT_INO}; use crate::{Core, ONLINE_PROBE_MAX, ONLINE_PROBE_MIN};
/// How long a remote call a FUSE handler is waiting on may take before the link /// How long a remote call a FUSE handler is waiting on may take before the link
/// is treated as down. /// is treated as down.
@@ -250,7 +250,7 @@ impl Core {
return false; return false;
} }
let probe = self.rt.block_on(async { let probe = self.rt.block_on(async {
match tokio::time::timeout(PROBE_TIMEOUT, self.client.get_my_files_folder()).await { match tokio::time::timeout(PROBE_TIMEOUT, self.drive.get_my_files_folder()).await {
Ok(result) => result, Ok(result) => result,
Err(_) => Err(no_answer(PROBE_TIMEOUT)), Err(_) => Err(no_answer(PROBE_TIMEOUT)),
} }
@@ -258,8 +258,9 @@ impl Core {
match probe { match probe {
Ok(root) => { Ok(root) => {
{ {
let root_ino = self.root_ino();
let mut st = self.state(); let mut st = self.state();
if let Some(e) = st.entries.get_mut(&ROOT_INO) { if let Some(e) = st.entries.get_mut(&root_ino) {
e.node = root.clone(); e.node = root.clone();
} }
} }
+89 -37
View File
@@ -207,11 +207,11 @@ const ROOT_UID_KEY: &str = "root_uid";
/// means an empty tree, and mounting that would show the user an empty Drive /// means an empty tree, and mounting that would show the user an empty Drive
/// rather than an honest error. /// rather than an honest error.
fn fetch_or_recall_root( fn fetch_or_recall_root(
client: &ProtonDriveClient, drive: &dyn DriveApi,
rt: &tokio::runtime::Handle, rt: &tokio::runtime::Handle,
db: &Db, db: &Db,
) -> std::io::Result<(Node, bool)> { ) -> std::io::Result<(Node, bool)> {
let err = match rt.block_on(client.get_my_files_folder()) { let err = match rt.block_on(drive.get_my_files_folder()) {
Ok(root) => { Ok(root) => {
if let Err(e) = db.set_state_str(ROOT_UID_KEY, &root.uid.to_string()) { if let Err(e) = db.set_state_str(ROOT_UID_KEY, &root.uid.to_string()) {
warn!(error = %e, "persist root uid failed"); warn!(error = %e, "persist root uid failed");
@@ -276,13 +276,14 @@ pub(super) fn spawn_session(
info!(mountpoint = %mountpoint.display(), "mounting Proton Drive location"); info!(mountpoint = %mountpoint.display(), "mounting Proton Drive location");
let fs = ProtonFs::new(core.clone(), root); let fs = ProtonFs::new(core.clone(), root);
if core.primary { if core.primary {
// Only My Files rehydrates the daemon-wide node cache. A secondary // Only My Files rehydrates the daemon-wide node cache, once. A
// inode space is rooted at one device subtree and must not import every // secondary mount shares the tree it fills.
// unrelated persisted node.
core.hydrate(); core.hydrate();
} }
let session = Session::new(fs, mountpoint, &config)?.spawn()?; let session = Session::new(fs, mountpoint, &config)?.spawn()?;
let _ = core.notifier.set(session.notifier()); let _ = core
.notifier
.set(KernelChannel::new(session.notifier(), core.root_ino()));
core.session_live.store(true, Ordering::Release); core.session_live.store(true, Ordering::Release);
Ok(session) Ok(session)
} }
@@ -339,6 +340,45 @@ pub fn mount(
control_socket: &Path, control_socket: &Path,
db: Arc<Db>, db: Arc<Db>,
options: MountOptions, options: MountOptions,
) -> std::io::Result<MountOutcome> {
let drive = Arc::new(client.clone());
mount_with(
client,
drive,
rt,
mountpoint,
cache,
control_socket,
db,
options,
Host::Daemon,
)
}
/// What a mount runs in.
pub(crate) enum Host {
/// The daemon: stopped by SIGTERM or SIGINT, and indexing the user's home
/// for the launcher prompt.
Daemon,
/// One of several simulated daemons in a test process (`sim`): stopped by
/// a message on this channel, and with no home to index.
#[cfg(test)]
Simulation(std::sync::mpsc::Receiver<()>),
}
/// [`mount`], reaching Drive through `drive` and hosted by `host`; `client`
/// serves only what [`DriveApi`] does not cover.
#[allow(clippy::too_many_arguments)]
pub(crate) fn mount_with(
client: ProtonDriveClient,
drive: Arc<dyn DriveApi>,
rt: tokio::runtime::Handle,
mountpoint: &Path,
cache: ContentCache,
control_socket: &Path,
db: Arc<Db>,
options: MountOptions,
host: Host,
) -> std::io::Result<MountOutcome> { ) -> std::io::Result<MountOutcome> {
let MountOptions { let MountOptions {
username, username,
@@ -349,7 +389,7 @@ pub fn mount(
// Start the uptime clock here rather than at the first `pdfs diagnostics`, // Start the uptime clock here rather than at the first `pdfs diagnostics`,
// so the age it reports is the daemon's own. // so the age it reports is the daemon's own.
super::diagnostics::start_clock(); super::diagnostics::start_clock();
let (root, online) = fetch_or_recall_root(&client, &rt, &db)?; let (root, online) = fetch_or_recall_root(drive.as_ref(), &rt, &db)?;
let scope = root.tree_event_scope_id(); let scope = root.tree_event_scope_id();
db.mount_upsert_my_files(&mountpoint.to_string_lossy(), &root.uid.to_string(), None) db.mount_upsert_my_files(&mountpoint.to_string_lossy(), &root.uid.to_string(), None)
.map_err(|error| std::io::Error::other(format!("project My Files mount: {error}")))?; .map_err(|error| std::io::Error::other(format!("project My Files mount: {error}")))?;
@@ -364,12 +404,14 @@ pub fn mount(
let paused_until = pause::load_paused_until(&db); let paused_until = pause::load_paused_until(&db);
let core = Core { let core = Core {
client: client.clone(), drive,
client,
rt: rt.clone(), rt: rt.clone(),
maintenance: Arc::new(Mutex::new(Default::default())), maintenance: Arc::new(Mutex::new(Default::default())),
primary_root_uid: root.uid.clone(), primary_root_uid: root.uid.clone(),
primary: true, primary: true,
state: Arc::new(Mutex::new(State::new(db.clone(), share_access, 2))), state: Arc::new(Mutex::new(State::new(db.clone(), share_access))),
root: Arc::new(OnceLock::new()),
cache: Arc::new(cache), cache: Arc::new(cache),
readers: Arc::new(Mutex::new(HashMap::new())), readers: Arc::new(Mutex::new(HashMap::new())),
block_ring: Arc::new(Mutex::new(BlockRing::default())), block_ring: Arc::new(Mutex::new(BlockRing::default())),
@@ -391,6 +433,7 @@ pub fn mount(
shutdown: Arc::new(crate::shutdown::Shutdown::default()), shutdown: Arc::new(crate::shutdown::Shutdown::default()),
upload_times: Arc::new(Mutex::new(HashMap::new())), upload_times: Arc::new(Mutex::new(HashMap::new())),
upload_cancel: Arc::new(Mutex::new(HashMap::new())), upload_cancel: Arc::new(Mutex::new(HashMap::new())),
landing_uploads: Arc::new(AtomicU64::new(0)),
timeline_refreshing: Arc::new(AtomicBool::new(false)), timeline_refreshing: Arc::new(AtomicBool::new(false)),
timeline_progress: Arc::new(Mutex::new(None)), timeline_progress: Arc::new(Mutex::new(None)),
similar_hashing: Arc::new(AtomicBool::new(false)), similar_hashing: Arc::new(AtomicBool::new(false)),
@@ -399,7 +442,6 @@ pub fn mount(
trash_progress: Arc::new(tokio::sync::Notify::new()), trash_progress: Arc::new(tokio::sync::Notify::new()),
conflict_notified: Arc::new(Mutex::new(HashSet::new())), conflict_notified: Arc::new(Mutex::new(HashSet::new())),
sweep_mode, sweep_mode,
own_sealed_revs: Arc::new(Mutex::new(HashMap::new())),
self_changes: Arc::new(Mutex::new(HashMap::new())), self_changes: Arc::new(Mutex::new(HashMap::new())),
thumb_gen: Arc::new(Mutex::new(HashSet::new())), thumb_gen: Arc::new(Mutex::new(HashSet::new())),
thumb_gen_budget: Arc::new(tokio::sync::Semaphore::new( thumb_gen_budget: Arc::new(tokio::sync::Semaphore::new(
@@ -427,7 +469,7 @@ pub fn mount(
states: Arc::new(StateRegistry::default()), states: Arc::new(StateRegistry::default()),
}; };
// Before anything can queue work against it: the drain thread below reaches // Before anything can queue work against it: the drain thread below reaches
// every mount's inode space through this registry, not through `core.state`. // every mount's session through this registry.
core.register_state(mountpoint); core.register_state(mountpoint);
core.transfers.set_limits(upload_limit, download_limit); core.transfers.set_limits(upload_limit, download_limit);
@@ -500,7 +542,7 @@ pub fn mount(
// Keep the launcher prompt's "This computer" index fresh. Its own thread: // Keep the launcher prompt's "This computer" index fresh. Its own thread:
// the walk is I/O-heavy and must never sit in front of a FUSE callback. // the walk is I/O-heavy and must never sit in front of a FUSE callback.
{ if matches!(host, Host::Daemon) {
let db = core.db.clone(); let db = core.db.clone();
let indexing = core.indexing.clone(); let indexing = core.indexing.clone();
let transfers = core.transfers.clone(); let transfers = core.transfers.clone();
@@ -624,7 +666,7 @@ pub fn mount(
} }
} }
}; };
rt.spawn(run_event_sync(client, scope, core.clone())); rt.spawn(run_event_sync(core.drive.clone(), scope, core.clone()));
// The photos volume has its own event stream; without it the gallery only // The photos volume has its own event stream; without it the gallery only
// learns about a phone-side deletion when the timeline goes stale. // learns about a phone-side deletion when the timeline goes stale.
rt.spawn(run_photos_event_sync(core.clone())); rt.spawn(run_photos_event_sync(core.clone()));
@@ -643,30 +685,11 @@ pub fn mount(
// are delivered onto the async runtime; bridge them onto a sync channel so // are delivered onto the async runtime; bridge them onto a sync channel so
// the loop below can react without blocking a worker thread. A bounded // the loop below can react without blocking a worker thread. A bounded
// channel of 1 is enough — we only need to know that *a* stop arrived. // channel of 1 is enough — we only need to know that *a* stop arrived.
let (sig_tx, sig_rx) = std::sync::mpsc::sync_channel::<()>(1); let sig_rx = match host {
rt.spawn(async move { Host::Daemon => signal_stop(&rt),
let mut sigterm = #[cfg(test)]
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) { Host::Simulation(rx) => rx,
Ok(s) => s, };
Err(e) => {
warn!(error = %e, "install SIGTERM handler failed");
return;
}
};
let mut sigint =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGINT handler failed");
return;
}
};
tokio::select! {
_ = sigterm.recv() => info!("received SIGTERM"),
_ = sigint.recv() => info!("received SIGINT"),
}
let _ = sig_tx.try_send(());
});
// Wait for whichever happens first: a stop signal (→ we unmount ourselves // Wait for whichever happens first: a stop signal (→ we unmount ourselves
// via the lazy MNT_DETACH path, clean even mid-download), or the kernel // via the lazy MNT_DETACH path, clean even mid-download), or the kernel
@@ -723,6 +746,35 @@ pub fn mount(
Ok(outcome) Ok(outcome)
} }
/// A channel that receives once SIGTERM or SIGINT arrives.
fn signal_stop(rt: &tokio::runtime::Handle) -> std::sync::mpsc::Receiver<()> {
let (sig_tx, sig_rx) = std::sync::mpsc::sync_channel::<()>(1);
rt.spawn(async move {
let mut sigterm =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGTERM handler failed");
return;
}
};
let mut sigint =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGINT handler failed");
return;
}
};
tokio::select! {
_ = sigterm.recv() => info!("received SIGTERM"),
_ = sigint.recv() => info!("received SIGINT"),
}
let _ = sig_tx.try_send(());
});
sig_rx
}
/// How long teardown waits for the FUSE worker pool before proceeding without /// How long teardown waits for the FUSE worker pool before proceeding without
/// it. Long enough for an in-flight block to finish, short enough that /// it. Long enough for an in-flight block to finish, short enough that
/// `systemctl --user stop` still returns inside `TimeoutStopSec`. /// `systemctl --user stop` still returns inside `TimeoutStopSec`.
+1 -1
View File
@@ -1038,7 +1038,7 @@ impl Core {
} }
let outcomes = self let outcomes = self
.rt .rt
.block_on(self.client.trash_nodes(uids)) .block_on(self.drive.trash_nodes(uids))
.map_err(|e| CoreError::from_api(&e, "trash"))?; .map_err(|e| CoreError::from_api(&e, "trash"))?;
let (trashed, failed) = pdfs_core::batch::split(outcomes); let (trashed, failed) = pdfs_core::batch::split(outcomes);
+8 -8
View File
@@ -101,7 +101,7 @@ impl Core {
match existing { match existing {
Some(uid) => self Some(uid) => self
.rt .rt
.block_on(self.client.upload_new_revision_from( .block_on(self.drive.upload_new_revision_from(
&uid, &uid,
Cursor::new(bytes), Cursor::new(bytes),
len, len,
@@ -112,7 +112,7 @@ impl Core {
.map_err(|e| CoreError::from_api(&e, "upload profile revision"))?, .map_err(|e| CoreError::from_api(&e, "upload profile revision"))?,
None => { None => {
self.rt self.rt
.block_on(self.client.upload_file_replacing_draft_from( .block_on(self.drive.upload_file_replacing_draft_from(
&dir_uid, &dir_uid,
PROFILE_FILE_NAME, PROFILE_FILE_NAME,
"application/json", "application/json",
@@ -182,14 +182,14 @@ impl Core {
) -> CoreResult<Option<NodeUid>> { ) -> CoreResult<Option<NodeUid>> {
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(parent_uid)) .block_on(self.drive.enumerate_folder_children_node_uids(parent_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?; .map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() { if uids.is_empty() {
return Ok(None); return Ok(None);
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes(&uids)) .block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?; .map_err(|e| CoreError::from_api(&e, "resolve device root children"))?;
Ok(nodes Ok(nodes
.into_iter() .into_iter()
@@ -210,7 +210,7 @@ impl Core {
let uid = self let uid = self
.rt .rt
.block_on( .block_on(
self.client self.drive
.create_folder(root_uid, PROFILE_DIR_NAME, Some(now_secs())), .create_folder(root_uid, PROFILE_DIR_NAME, Some(now_secs())),
) )
.map_err(|e| CoreError::from_api(&e, "create profile folder"))?; .map_err(|e| CoreError::from_api(&e, "create profile folder"))?;
@@ -241,7 +241,7 @@ impl Core {
}; };
let mut buf: Vec<u8> = Vec::new(); let mut buf: Vec<u8> = Vec::new();
self.rt self.rt
.block_on(self.client.download_file_to(&uid, &mut buf)) .block_on(self.drive.download_file_to(&uid, &mut buf))
.map_err(|e| CoreError::from_api(&e, "download profile"))?; .map_err(|e| CoreError::from_api(&e, "download profile"))?;
Profile::parse(&buf).map(Some).map_err(CoreError::invalid) Profile::parse(&buf).map(Some).map_err(CoreError::invalid)
} }
@@ -277,13 +277,13 @@ impl Core {
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(&root_uid)) .block_on(self.drive.enumerate_folder_children_node_uids(&root_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?; .map_err(|e| CoreError::from_api(&e, "list device root"))?;
let nodes = if uids.is_empty() { let nodes = if uids.is_empty() {
Vec::new() Vec::new()
} else { } else {
self.rt self.rt
.block_on(self.client.enumerate_nodes(&uids)) .block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))? .map_err(|e| CoreError::from_api(&e, "resolve device root children"))?
}; };
+11 -11
View File
@@ -70,7 +70,7 @@ const MAX_OPEN_READERS: usize = 64;
/// `(mtime, size)` — the same validity pair the content cache uses (a new /// `(mtime, size)` — the same validity pair the content cache uses (a new
/// revision bumps mtime). On a mismatch the reader is dropped and reopened. /// revision bumps mtime). On a mismatch the reader is dropped and reopened.
pub(super) struct CachedReader { pub(super) struct CachedReader {
reader: Arc<RevisionReader>, reader: Arc<dyn RevisionRead>,
mtime: i64, mtime: i64,
size: u64, size: u64,
/// For LRU eviction. /// For LRU eviction.
@@ -101,7 +101,7 @@ pub(super) struct PendingOpen {
/// must not join this open — it would get a reader for the wrong bytes. /// must not join this open — it would get a reader for the wrong bytes.
mtime: i64, mtime: i64,
size: u64, size: u64,
rx: tokio::sync::watch::Receiver<Option<Result<Arc<RevisionReader>, Errno>>>, rx: tokio::sync::watch::Receiver<Option<Result<Arc<dyn RevisionRead>, Errno>>>,
} }
/// Distinguishes `PendingOpen`s. Only uniqueness matters, so a plain counter /// Distinguishes `PendingOpen`s. Only uniqueness matters, so a plain counter
@@ -313,16 +313,16 @@ impl Core {
uid: &NodeUid, uid: &NodeUid,
mtime: i64, mtime: i64,
fsize: u64, fsize: u64,
) -> Result<Arc<RevisionReader>, Errno> { ) -> Result<Arc<dyn RevisionRead>, Errno> {
loop { loop {
// Decide under the lock, act outside it — network I/O must never run // Decide under the lock, act outside it — network I/O must never run
// under a std `Mutex`, and an await must never hold one. // under a std `Mutex`, and an await must never hold one.
enum Act { enum Act {
Hit(Arc<RevisionReader>), Hit(Arc<dyn RevisionRead>),
Join(tokio::sync::watch::Receiver<Option<Result<Arc<RevisionReader>, Errno>>>), Join(tokio::sync::watch::Receiver<Option<Result<Arc<dyn RevisionRead>, Errno>>>),
Lead( Lead(
u64, u64,
tokio::sync::watch::Sender<Option<Result<Arc<RevisionReader>, Errno>>>, tokio::sync::watch::Sender<Option<Result<Arc<dyn RevisionRead>, Errno>>>,
), ),
} }
let act = { let act = {
@@ -376,8 +376,8 @@ impl Core {
} }
Act::Lead(id, tx) => { Act::Lead(id, tx) => {
debug!(%uid, mtime, fsize, "opening revision"); debug!(%uid, mtime, fsize, "opening revision");
let result = match self.client.open_revision(uid).await { let result = match self.drive.open_revision(uid).await {
Ok(reader) => Ok(Arc::new(reader)), Ok(reader) => Ok(reader),
Err(e) => { Err(e) => {
self.lost_link(&e, "open a revision"); self.lost_link(&e, "open a revision");
warn!(%uid, error = %e, "open_revision failed"); warn!(%uid, error = %e, "open_revision failed");
@@ -568,7 +568,7 @@ impl Core {
/// leader's claim is gone by then, so this cannot recurse. /// leader's claim is gone by then, so this cannot recurse.
async fn fetch_block( async fn fetch_block(
&self, &self,
reader: &Arc<RevisionReader>, reader: &Arc<dyn RevisionRead>,
uid: &NodeUid, uid: &NodeUid,
mtime: i64, mtime: i64,
fsize: u64, fsize: u64,
@@ -634,7 +634,7 @@ impl Core {
/// download, or the read fails. See [`Core::repair_block`] (bugs.md B84). /// download, or the read fails. See [`Core::repair_block`] (bugs.md B84).
async fn read_block( async fn read_block(
&self, &self,
reader: &Arc<RevisionReader>, reader: &Arc<dyn RevisionRead>,
uid: &NodeUid, uid: &NodeUid,
mtime: i64, mtime: i64,
fsize: u64, fsize: u64,
@@ -778,7 +778,7 @@ impl Core {
error!(%uid, %error, "creating a scratch file for a repair download failed"); error!(%uid, %error, "creating a scratch file for a repair download failed");
Errno::EIO Errno::EIO
})?; })?;
let outcome = self.client.download_file_to(uid, &mut file).await; let outcome = self.drive.download_file_to(uid, &mut file).await;
let stored = (|| { let stored = (|| {
outcome.map_err(|error| { outcome.map_err(|error| {
warn!(%uid, %error, "repair download failed"); warn!(%uid, %error, "repair download failed");
+1 -4
View File
@@ -248,10 +248,7 @@ impl Core {
let outcomes = self let outcomes = self
.rt .rt
.block_on( .block_on(self.drive.trash_nodes(std::slice::from_ref(&candidate.uid)))
self.client
.trash_nodes(std::slice::from_ref(&candidate.uid)),
)
.map_err(|e| CoreError::from_api(&e, "trash the mis-dated original"))?; .map_err(|e| CoreError::from_api(&e, "trash the mis-dated original"))?;
if let Some((_, Err(e))) = outcomes.into_iter().next() { if let Some((_, Err(e))) = outcomes.into_iter().next() {
return Err(CoreError::from_api(&e, "trash the mis-dated original")); return Err(CoreError::from_api(&e, "trash the mis-dated original"));
+19 -11
View File
@@ -25,7 +25,6 @@ use proton_drive_rs::NodeKind;
use proton_drive_rs::proton_sdk::ids::NodeUid; use proton_drive_rs::proton_sdk::ids::NodeUid;
use tracing::{info, warn}; use tracing::{info, warn};
use super::background::forget_and_notify;
use super::sync::mirror_subtree_unsynced; use super::sync::mirror_subtree_unsynced;
use super::{Core, is_local_uid, parse_uid}; use super::{Core, is_local_uid, parse_uid};
@@ -36,7 +35,7 @@ const MIRROR_LOCK_WAIT: Duration = Duration::from_secs(5);
/// Where one side of a move lives. /// Where one side of a move lives.
enum Place { enum Place {
/// My files or an on-demand folder: a live inode space. `core` is rooted at /// My files or an on-demand folder: a live FUSE mount. `core` is rooted at
/// that mount and `rel` is relative to its mountpoint. /// that mount and `rel` is relative to its mountpoint.
Mounted { core: Box<Core>, rel: PathBuf }, Mounted { core: Box<Core>, rel: PathBuf },
/// A mirror folder: a local directory kept in step by the sync engine. /// A mirror folder: a local directory kept in step by the sync engine.
@@ -142,7 +141,7 @@ impl Core {
/// are absolute local paths and may be in different locations: My files, /// are absolute local paths and may be in different locations: My files,
/// an on-demand folder, or a mirror folder. Returns the moved name. /// an on-demand folder, or a mirror folder. Returns the moved name.
/// ///
/// Within one inode space this is [`Core::move_to`]. Across locations it /// Within one mount this is [`Core::move_to`]. Across locations it
/// is a server-side move followed by local bookkeeping; see the module /// is a server-side move followed by local bookkeeping; see the module
/// documentation. Anything the move could lose is refused up front. /// documentation. Anything the move could lose is refused up front.
pub(crate) fn move_between(&self, src: &Path, dest_parent: &Path) -> CoreResult<String> { pub(crate) fn move_between(&self, src: &Path, dest_parent: &Path) -> CoreResult<String> {
@@ -155,7 +154,7 @@ impl Core {
rel: dest_rel, rel: dest_rel,
}, },
) = (&source, &dest) ) = (&source, &dest)
&& Arc::ptr_eq(&core.state, &dest_core.state) && Arc::ptr_eq(&core.root, &dest_core.root)
{ {
return core.move_to(rel, dest_rel); return core.move_to(rel, dest_rel);
} }
@@ -255,7 +254,7 @@ impl Core {
))); )));
} }
if let Err(e) = self.rt.block_on(self.client.move_node(&src_uid, &dest_uid)) { if let Err(e) = self.move_remote(&src_uid, &dest_uid) {
if let Some((from, to)) = &renamed if let Some((from, to)) = &renamed
&& let Err(back) = std::fs::rename(to, from) && let Err(back) = std::fs::rename(to, from)
{ {
@@ -265,7 +264,15 @@ impl Core {
return Err(CoreError::from_api(&e, "move")); return Err(CoreError::from_api(&e, "move"));
} }
self.settle_source(&source, &src_uid, renamed.is_some(), &dest, &name, src); self.settle_source(
&source,
&src_uid,
renamed.is_some(),
&dest,
&dest_uid,
&name,
src,
);
self.settle_dest(&dest, &dest_uid, &name); self.settle_dest(&dest, &dest_uid, &name);
drop(guards); drop(guards);
if renamed.is_none() if renamed.is_none()
@@ -429,22 +436,23 @@ impl Core {
} }
/// Bring the source location level with a move Drive has made. /// Bring the source location level with a move Drive has made.
#[allow(clippy::too_many_arguments)]
fn settle_source( fn settle_source(
&self, &self,
source: &Place, source: &Place,
uid: &NodeUid, uid: &NodeUid,
renamed: bool, renamed: bool,
dest: &Place, dest: &Place,
dest_uid: &NodeUid,
name: &str, name: &str,
src: &Path, src: &Path,
) { ) {
match source { match source {
Place::Mounted { core, rel } => { Place::Mounted { core, rel } => {
// Forget it everywhere, so no mount keeps serving it under the // Moved everywhere, so no mount keeps serving it under the old
// old path. The content cache is keyed by uid and stays valid. // path and its row stays (`docs/BUGS.md` B117). The content
self.for_each_mount(|st, notify| { // cache is keyed by uid and stays valid.
forget_and_notify(st, notify, uid); self.relocate_everywhere(uid, dest_uid, name);
});
core.invalidate_parent_listing(rel); core.invalidate_parent_listing(rel);
} }
Place::Mirror { folder, rel } if renamed => { Place::Mirror { folder, rel } if renamed => {
+1 -1
View File
@@ -232,7 +232,7 @@ impl Core {
let mut out = std::io::BufWriter::new(file); let mut out = std::io::BufWriter::new(file);
let written = match self let written = match self
.rt .rt
.block_on(self.client.download_revision_to(uid, revision_id, &mut out)) .block_on(self.drive.download_revision_to(uid, revision_id, &mut out))
.map_err(|e| CoreError::from_api(&e, "download revision")) .map_err(|e| CoreError::from_api(&e, "download revision"))
.and_then(|()| { .and_then(|()| {
use std::io::Write as _; use std::io::Write as _;
+18 -23
View File
@@ -21,7 +21,7 @@ use proton_drive_rs::{MemberRole, NodeKind};
use proton_drive_rs::Node; use proton_drive_rs::Node;
use super::{Core, ROOT_INO, node_size, parse_uid, public_link_info, role_from_str, role_to_str}; use super::{Core, node_size, parse_uid, public_link_info, role_from_str, role_to_str};
/// A node from a *shared* listing as a [`DirEntry`]. /// A node from a *shared* listing as a [`DirEntry`].
/// ///
@@ -463,7 +463,7 @@ impl Core {
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes(&uids)) .block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?; .map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes Ok(nodes
.into_iter() .into_iter()
@@ -481,14 +481,14 @@ impl Core {
parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?; parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?;
let child_uids = self let child_uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(&uid)) .block_on(self.drive.enumerate_folder_children_node_uids(&uid))
.map_err(|e| CoreError::from_api(&e, "enumerate shared children"))?; .map_err(|e| CoreError::from_api(&e, "enumerate shared children"))?;
if child_uids.is_empty() { if child_uids.is_empty() {
return Ok(Vec::new()); return Ok(Vec::new());
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes(&child_uids)) .block_on(self.drive.enumerate_nodes(&child_uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?; .map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes Ok(nodes
.into_iter() .into_iter()
@@ -509,7 +509,7 @@ impl Core {
parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?; parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?;
let node = self let node = self
.rt .rt
.block_on(self.client.get_node(&uid)) .block_on(self.drive.get_node(&uid))
.map_err(|e| CoreError::from_api(&e, "get shared node"))? .map_err(|e| CoreError::from_api(&e, "get shared node"))?
.ok_or_else(|| CoreError::not_found("shared node not found"))?; .ok_or_else(|| CoreError::not_found("shared node not found"))?;
if !node.is_file() { if !node.is_file() {
@@ -610,7 +610,7 @@ impl Core {
/// Proton products (not Drive-only). A remote round-trip; nothing here is /// Proton products (not Drive-only). A remote round-trip; nothing here is
/// cached. /// cached.
pub(crate) fn account_quota(&self) -> CoreResult<(i64, i64)> { pub(crate) fn account_quota(&self) -> CoreResult<(i64, i64)> {
let q = self.block_on_bounded(self.client.quota()).map_err(|e| { let q = self.block_on_bounded(self.drive.quota()).map_err(|e| {
self.lost_link(&e, "account quota"); self.lost_link(&e, "account quota");
CoreError::from_api(&e, "account quota") CoreError::from_api(&e, "account quota")
})?; })?;
@@ -669,7 +669,7 @@ impl Core {
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes(&uids)) .block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?; .map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
let mut items = Vec::with_capacity(nodes.len()); let mut items = Vec::with_capacity(nodes.len());
for n in nodes { for n in nodes {
@@ -714,19 +714,10 @@ impl Core {
/// been seen through the mount (e.g. shared but not browsed to this session) — /// been seen through the mount (e.g. shared but not browsed to this session) —
/// the caller then leaves the path empty. /// the caller then leaves the path empty.
pub(crate) fn rel_path_for_uid(&self, uid: &NodeUid) -> Option<String> { pub(crate) fn rel_path_for_uid(&self, uid: &NodeUid) -> Option<String> {
// Searched across every mount: a node browsed through a sync folder is // Relative to whichever root the node sits under: a node browsed
// interned in that fork's inode space, so the primary state alone had no // through a sync folder is under that fork's root, not My Files
// entry for it and the caller silently showed an empty path // (`docs/BUGS.md` B74).
// (`docs/BUGS.md` B74). The path is relative to whichever mount holds walk_to_root(&self.state(), uid)
// it, which is the one the user reached it through.
let mut path = None;
self.for_each_state(|st| {
if path.is_some() {
return;
}
path = walk_to_root(st, uid);
});
path
} }
} }
@@ -771,14 +762,18 @@ impl StopSharingOutcome {
} }
} }
/// Walk one inode space from `uid` up to its root, building the path. `None` if /// Walk the tree from `uid` up to the root it sits under, building the path.
/// that mount has never interned the node, or if the chain breaks partway — /// `None` if the node has never been interned, or if the chain breaks partway —
/// a broken chain is a missing path, never a partial one. /// a broken chain is a missing path, never a partial one.
fn walk_to_root(st: &super::State, uid: &NodeUid) -> Option<String> { fn walk_to_root(st: &super::State, uid: &NodeUid) -> Option<String> {
let mut ino = *st.by_uid.get(uid)?; let mut ino = *st.by_uid.get(uid)?;
let mut parts = Vec::new(); let mut parts = Vec::new();
while ino != ROOT_INO { while !st.roots.contains(&ino) {
let entry = st.entries.get(&ino)?; let entry = st.entries.get(&ino)?;
// An orphan is its own parent without being a root.
if entry.parent == ino {
return None;
}
parts.push(entry.node.name.clone()); parts.push(entry.node.name.clone());
ino = entry.parent; ino = entry.parent;
} }
+423
View File
@@ -0,0 +1,423 @@
//! One simulated daemon: a real FUSE mount on its own state directory, whose
//! Drive is a [`FakeClient`].
//!
//! Everything the daemon does runs as in production (drain workers, sync
//! engine, event feed, online probe, control socket) except what the host
//! changes (`mount::Host::Simulation`): a channel stops it instead of a signal,
//! and it does not index the developer's home.
use std::io;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::mpsc::SyncSender;
use std::thread::JoinHandle;
use std::time::{Duration, Instant};
use pdfs_core::cache::ContentCache;
use pdfs_core::config::AppConfig;
use pdfs_core::control::{self, PendingOpInfo, Request, Response};
use pdfs_core::db::Db;
use proton_drive_rs::ProtonDriveClient;
use proton_drive_rs::proton_sdk::config::ProtonClientConfiguration;
use proton_drive_rs::proton_sdk::session::{PasswordMode, ProtonApiSession, ResumeParameters};
use super::fake_drive::FakeClient;
use crate::mount::{Host, MountOptions, MountOutcome, mount_with};
/// How long a mount may take to come up.
const MOUNT_DEADLINE: Duration = Duration::from_secs(20);
/// How long a restart may wait for the last run to let go of its database.
const LAST_RUN_DEADLINE: Duration = Duration::from_secs(30);
pub(crate) struct Daemon {
pub(crate) client: FakeClient,
pub(crate) mountpoint: PathBuf,
dir: PathBuf,
socket: PathBuf,
stop: SyncSender<()>,
mount: Option<JoinHandle<io::Result<MountOutcome>>>,
rt: Option<tokio::runtime::Runtime>,
}
impl Daemon {
/// Mount a daemon on the state under `dir`, creating it on the first start.
/// A later start on the same `dir` is a restart: it finds the queue, the
/// cache and the database the last run left.
pub(crate) fn start(dir: &Path, client: FakeClient) -> io::Result<Self> {
log_to_test_output();
let mountpoint = dir.join("mnt");
let socket = dir.join("control.sock");
std::fs::create_dir_all(&mountpoint)?;
// A thread of the last run that is still in a request holds the
// database, and with it the single-writer lock, until it returns. A
// real restart waits for the old process to exit in the same way.
let mut db = Db::open(&dir.join("pdfs.db"));
let deadline = Instant::now() + LAST_RUN_DEADLINE;
while db
.as_ref()
.is_err_and(|error| error.to_string().contains("already using"))
&& Instant::now() < deadline
{
std::thread::sleep(Duration::from_millis(100));
db = Db::open(&dir.join("pdfs.db"));
}
let db = Arc::new(db.map_err(io::Error::other)?);
let cache = ContentCache::open(dir.join("cache"), dir.join("pins.json"), 0, db.clone())
.map_err(io::Error::other)?;
let rt = tokio::runtime::Builder::new_multi_thread()
.worker_threads(4)
.enable_all()
.build()?;
let (stop, stopped) = std::sync::mpsc::sync_channel(1);
let mount = {
let drive = Arc::new(client.clone());
let handle = rt.handle().clone();
let mountpoint = mountpoint.clone();
let socket = socket.clone();
std::thread::Builder::new()
.name("sim-mount".into())
.spawn(move || {
mount_with(
offline_client(),
drive,
handle,
&mountpoint,
cache,
&socket,
db,
MountOptions {
username: "sim".into(),
sweep_mode: AppConfig::default().resolved_conflict_sweep(),
upload_limit: 0,
download_limit: 0,
},
Host::Simulation(stopped),
)
})?
};
let deadline = Instant::now() + MOUNT_DEADLINE;
while !is_mounted(&mountpoint) {
if mount.is_finished() {
return Err(match mount.join() {
Ok(Err(error)) => error,
_ => io::Error::other("mount ended before it came up"),
});
}
if Instant::now() > deadline {
return Err(io::Error::other("mount did not come up"));
}
std::thread::sleep(Duration::from_millis(20));
}
Ok(Self {
client,
mountpoint,
dir: dir.to_path_buf(),
socket,
stop,
mount: Some(mount),
rt: Some(rt),
})
}
/// What the daemon still owes Drive, as `pdfs sync queue` shows it.
pub(crate) fn pending(&self) -> io::Result<Vec<PendingOpInfo>> {
match control::send(&self.socket, &Request::ListPendingOps).map_err(io::Error::other)? {
Response::PendingOps { items } => Ok(items),
other => Err(io::Error::other(format!("unexpected answer {other:?}"))),
}
}
/// Stop the daemon and start it again on the same state and link.
pub(crate) fn restart(mut self) -> io::Result<Self> {
self.shut_down()?;
Self::start(&self.dir.clone(), self.client.clone())
}
/// Stop the daemon the way `systemctl --user stop` does.
pub(crate) fn stop(mut self) -> io::Result<MountOutcome> {
self.shut_down()
}
fn shut_down(&mut self) -> io::Result<MountOutcome> {
let Some(mount) = self.mount.take() else {
return Ok(MountOutcome::Shutdown);
};
let _ = self.stop.try_send(());
let outcome = mount
.join()
.unwrap_or_else(|_| Err(io::Error::other("mount thread panicked")));
if let Some(rt) = self.rt.take() {
rt.shutdown_timeout(Duration::from_secs(5));
}
outcome
}
}
impl Drop for Daemon {
fn drop(&mut self) {
let _ = self.shut_down();
}
}
/// Whether a file system is mounted at `path`: it sits on another device than
/// its parent.
fn is_mounted(path: &Path) -> bool {
use std::os::unix::fs::MetadataExt;
let device = |path: &Path| std::fs::metadata(path).map(|meta| meta.dev()).ok();
let parent = path.parent().and_then(device);
device(path).is_some_and(|dev| Some(dev) != parent)
}
/// The daemons' logs, filtered by `RUST_LOG` and shown with `--nocapture`.
/// What they log at `INFO` and above is also kept for [`take_logged`],
/// whatever the filter.
fn log_to_test_output() {
use tracing_subscriber::prelude::*;
let _ = tracing_subscriber::registry()
.with(
tracing_subscriber::fmt::layer()
.with_test_writer()
.with_filter(tracing_subscriber::EnvFilter::from_default_env()),
)
.with(KeepLogged.with_filter(tracing_subscriber::filter::LevelFilter::INFO))
.try_init();
}
/// What the daemons logged at `INFO` and above since the last call, each as
/// its message followed by its fields: `message key=value ...`.
pub(crate) fn take_logged() -> Vec<String> {
std::mem::take(&mut *LOGGED.lock())
}
static LOGGED: parking_lot::Mutex<Vec<String>> = parking_lot::Mutex::new(Vec::new());
struct KeepLogged;
impl<S: tracing::Subscriber> tracing_subscriber::Layer<S> for KeepLogged {
fn on_event(&self, event: &tracing::Event<'_>, _: tracing_subscriber::layer::Context<'_, S>) {
let mut message = Message(String::new());
event.record(&mut message);
LOGGED.lock().push(message.0);
}
}
struct Message(String);
impl tracing::field::Visit for Message {
fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
if field.name() == "message" {
self.0.insert_str(0, &format!("{value:?}"));
} else {
self.0.push_str(&format!(" {}={value:?}", field.name()));
}
}
}
/// A client for what `DriveApi` does not cover (sharing, Photos, devices),
/// pointed at a closed local port so none of it leaves the machine.
fn offline_client() -> ProtonDriveClient {
let config = ProtonClientConfiguration::new("external-drive-pdfs-sim@0.0.0-dev")
.with_base_url("http://127.0.0.1:9");
let session = ProtonApiSession::resume(
config,
ResumeParameters {
session_id: "sim".to_owned().into(),
username: "sim".into(),
user_id: "sim".to_owned().into(),
access_token: String::new(),
refresh_token: String::new(),
scopes: Vec::new(),
is_waiting_for_second_factor_code: false,
password_mode: PasswordMode::Single,
},
)
.expect("an offline session needs no network");
ProtonDriveClient::new(&session, Vec::new())
}
/// A fresh directory for one simulation, under the system temp dir.
pub(crate) fn scratch(label: &str) -> PathBuf {
use std::sync::atomic::{AtomicU64, Ordering};
static N: AtomicU64 = AtomicU64::new(0);
let dir = std::env::temp_dir().join(format!(
"pdfs-sim-{label}-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed)
));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).expect("create a simulation directory");
dir
}
/// Wait up to `limit` for `done`, polling.
pub(crate) fn wait_until(limit: Duration, mut done: impl FnMut() -> bool) -> bool {
let deadline = Instant::now() + limit;
loop {
if done() {
return true;
}
if Instant::now() > deadline {
return false;
}
std::thread::sleep(Duration::from_millis(50));
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::sim::fake_drive::{Entry, FakeDrive, Faults};
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_daemon_on_the_fake_drive_syncs_both_ways() {
let drive = FakeDrive::new();
let dir = scratch("smoke");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
std::fs::write(daemon.mountpoint.join("hello.txt"), b"hi").unwrap();
assert!(
wait_until(Duration::from_secs(30), || {
drive.tree().get("hello.txt") == Some(&Entry::File(Arc::new(b"hi".to_vec())))
}),
"a local write landed on Drive: {:?}",
drive.tree()
);
drive.device().write("phone.txt", b"from the phone");
let remote = daemon.mountpoint.join("phone.txt");
assert!(
wait_until(Duration::from_secs(30), || {
std::fs::read(&remote).ok().as_deref() == Some(b"from the phone".as_slice())
}),
"a remote write reached the mount"
);
daemon.client.set_online(false);
std::fs::write(daemon.mountpoint.join("offline.txt"), b"queued").unwrap();
std::thread::sleep(Duration::from_secs(2));
assert!(!drive.tree().contains_key("offline.txt"));
daemon.client.set_online(true);
assert!(
wait_until(Duration::from_secs(60), || drive
.tree()
.contains_key("offline.txt")),
"a write made offline landed once the link came back"
);
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
/// Write `bytes` at `offset` into an existing file, without truncating it.
fn write_at(path: &Path, offset: u64, bytes: &[u8]) -> io::Result<()> {
use std::os::unix::fs::FileExt;
std::fs::OpenOptions::new()
.write(true)
.open(path)?
.write_all_at(bytes, offset)
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_file_closed_empty_takes_a_second_write_at_once() {
// The empty revision never counted as complete, so the next write
// open before it drained was refused with EIO (B123).
let drive = FakeDrive::new();
let dir = scratch("empty-rewrite");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
let path = daemon.mountpoint.join("empty.txt");
std::fs::write(&path, b"old").unwrap();
assert!(wait_until(Duration::from_secs(30), || {
drive.tree().contains_key("empty.txt")
}));
std::fs::write(&path, b"").unwrap();
write_at(&path, 0, b"filled").unwrap();
assert!(
wait_until(Duration::from_secs(30), || {
drive.tree().get("empty.txt") == Some(&Entry::File(Arc::new(b"filled".to_vec())))
}),
"the second write landed: {:?}",
drive.tree()
);
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn the_echo_of_our_own_rename_keeps_the_cached_bytes() {
// The echo read as someone else's change and evicted the bytes, so a
// partial write could not gap-fill, and the next write open on the
// file was refused with EIO (B123).
let drive = FakeDrive::new();
let dir = scratch("rename-echo");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
let before = daemon.mountpoint.join("a.txt");
std::fs::write(&before, b"0123456789").unwrap();
assert!(wait_until(Duration::from_secs(30), || {
drive.tree().contains_key("a.txt")
}));
assert_eq!(std::fs::read(&before).unwrap(), b"0123456789");
let after = daemon.mountpoint.join("b.txt");
std::fs::rename(&before, &after).unwrap();
// Long enough for the next event poll to bring the echo.
std::thread::sleep(crate::POLL_INTERVAL + Duration::from_secs(3));
write_at(&after, 3, b"x").unwrap();
write_at(&after, 5, b"y").unwrap();
assert!(
wait_until(Duration::from_secs(30), || {
drive.tree().get("b.txt") == Some(&Entry::File(Arc::new(b"012x4y6789".to_vec())))
}),
"both writes landed: {:?}",
drive.tree()
);
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_rename_across_folders_ignores_the_new_name_in_the_old_folder() {
// The rename went to Drive as a rename in the old folder and then a
// move, so a sibling there with the new name failed it with EIO
// (B122).
let drive = FakeDrive::new();
let dir = scratch("rename-across");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
let mount = &daemon.mountpoint;
std::fs::create_dir(mount.join("x")).unwrap();
std::fs::create_dir(mount.join("z")).unwrap();
std::fs::write(mount.join("x/f.txt"), b"moved").unwrap();
std::fs::write(mount.join("x/d"), b"sibling").unwrap();
std::fs::write(mount.join("z/d"), b"replaced").unwrap();
let file = |bytes: &[u8]| Some(Entry::File(Arc::new(bytes.to_vec())));
assert!(wait_until(Duration::from_secs(30), || {
let tree = drive.tree();
[
("x/f.txt", b"moved".as_slice()),
("x/d", b"sibling"),
("z/d", b"replaced"),
]
.iter()
.all(|(path, bytes)| tree.get(*path).cloned() == file(bytes))
}));
std::fs::rename(mount.join("x/f.txt"), mount.join("z/d")).unwrap();
assert_eq!(std::fs::read(mount.join("z/d")).unwrap(), b"moved");
let tree = drive.tree();
assert_eq!(tree.get("z/d").cloned(), file(b"moved"));
assert_eq!(tree.get("x/d").cloned(), file(b"sibling"));
assert!(!tree.contains_key("x/f.txt"));
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
}
File diff suppressed because it is too large. Load diff
+9
View File
@@ -0,0 +1,9 @@
//! The simulation tests of `docs/MILESTONE-3.0.0.md` §8: a fake Drive with
//! the faults the real one has, and the tools to drive daemons against it.
pub(crate) mod daemon;
pub(crate) mod fake_drive;
pub(crate) mod model;
pub(crate) mod rng;
pub(crate) mod run;
pub(crate) mod watchdog;
+445
View File
@@ -0,0 +1,445 @@
//! The reference file system of the simulation (`docs/MILESTONE-3.0.0.md`
//! §8.2, property 4): what each syscall a simulated client makes should
//! answer, and the tree it should leave, on a POSIX file system.
//!
//! A [`Model`] covers one client's own folder; paths are relative to it. The
//! same type is used for the trees read back from a mount and from Drive.
use std::collections::BTreeMap;
use super::rng::Rng;
/// A tree by path: `None` is a folder, `Some` a file's content.
pub(crate) type Tree = BTreeMap<String, Option<Vec<u8>>>;
/// One file system call, or the few a program makes in a row.
#[derive(Clone, Debug)]
pub(crate) enum FsOp {
/// `open(O_CREAT | O_TRUNC)`, write `data`, close.
Create {
path: String,
data: Vec<u8>,
},
/// Open an existing file, write `data` at `offset` (past the end leaves
/// a hole), close.
Write {
path: String,
offset: u64,
data: Vec<u8>,
},
/// Open an existing file and `ftruncate` it to `len`.
Truncate {
path: String,
len: u64,
},
Mkdir {
path: String,
},
Rename {
from: String,
to: String,
},
Unlink {
path: String,
},
/// Open a file, unlink it, append `data`, read it all back, close.
UnlinkOpen {
path: String,
data: Vec<u8>,
},
Rmdir {
path: String,
},
}
impl FsOp {
/// The paths the op names, for the log.
pub(crate) fn describe(&self) -> String {
match self {
FsOp::Create { path, data } => format!("create {path} ({} bytes)", data.len()),
FsOp::Write { path, offset, data } => {
format!("write {path} at {offset} ({} bytes)", data.len())
}
FsOp::Truncate { path, len } => format!("truncate {path} to {len}"),
FsOp::Mkdir { path } => format!("mkdir {path}"),
FsOp::Rename { from, to } => format!("rename {from} -> {to}"),
FsOp::Unlink { path } => format!("unlink {path}"),
FsOp::UnlinkOpen { path, data } => {
format!("unlink {path} while open, then append {} bytes", data.len())
}
FsOp::Rmdir { path } => format!("rmdir {path}"),
}
}
}
/// The tree one client should have, and what its syscalls should answer.
#[derive(Clone, Debug, Default)]
pub(crate) struct Model {
tree: Tree,
}
/// Names are drawn from small pools, so ops collide: a create over an existing
/// file, a rename over another, a mkdir of a name that is taken.
const FILE_NAMES: [&str; 6] = ["a.txt", "b.txt", "c.bin", "d", "e.md", "f.txt"];
const DIR_NAMES: [&str; 3] = ["x", "y", "z"];
impl Model {
pub(crate) fn tree(&self) -> &Tree {
&self.tree
}
pub(crate) fn content(&self, path: &str) -> Option<&Vec<u8>> {
self.tree.get(path)?.as_ref()
}
/// What `op` should answer on a POSIX file system, applying it on success.
pub(crate) fn apply(&mut self, op: &FsOp) -> Result<(), i32> {
match op {
FsOp::Create { path, data } => {
self.check_parent(path)?;
match self.tree.get(path) {
Some(None) => Err(libc::EISDIR),
_ => {
self.tree.insert(path.clone(), Some(data.clone()));
Ok(())
}
}
}
FsOp::Write { path, offset, data } => {
let content = self.file_mut(path)?;
// Writing nothing changes nothing, even past the end.
if data.is_empty() {
return Ok(());
}
let end = *offset as usize + data.len();
if content.len() < end {
content.resize(end, 0);
}
content[*offset as usize..end].copy_from_slice(data);
Ok(())
}
FsOp::Truncate { path, len } => {
self.file_mut(path)?.resize(*len as usize, 0);
Ok(())
}
FsOp::Mkdir { path } => {
self.check_parent(path)?;
if self.tree.contains_key(path) {
return Err(libc::EEXIST);
}
self.tree.insert(path.clone(), None);
Ok(())
}
FsOp::Rename { from, to } => self.rename(from, to),
FsOp::Unlink { path } | FsOp::UnlinkOpen { path, .. } => {
self.check_parent(path)?;
match self.tree.get(path) {
None => Err(libc::ENOENT),
Some(None) => Err(libc::EISDIR),
Some(Some(_)) => {
self.tree.remove(path);
Ok(())
}
}
}
FsOp::Rmdir { path } => {
self.check_parent(path)?;
match self.tree.get(path) {
None => Err(libc::ENOENT),
Some(Some(_)) => Err(libc::ENOTDIR),
Some(None) if self.has_children(path) => Err(libc::ENOTEMPTY),
Some(None) => {
self.tree.remove(path);
Ok(())
}
}
}
}
}
fn rename(&mut self, from: &str, to: &str) -> Result<(), i32> {
self.check_parent(from)?;
let Some(moving) = self.tree.get(from).cloned() else {
return Err(libc::ENOENT);
};
self.check_parent(to)?;
if from == to {
return Ok(());
}
if is_below(to, from) {
return Err(libc::EINVAL);
}
match (&moving, self.tree.get(to)) {
(None, Some(Some(_))) => return Err(libc::ENOTDIR),
(Some(_), Some(None)) => return Err(libc::EISDIR),
(None, Some(None)) if self.has_children(to) => return Err(libc::ENOTEMPTY),
_ => {}
}
let below: Vec<(String, Option<Vec<u8>>)> = self
.tree
.iter()
.filter(|(path, _)| is_below(path, from))
.map(|(path, entry)| (path.clone(), entry.clone()))
.collect();
self.tree.remove(from);
self.tree.insert(to.to_owned(), moving);
for (path, entry) in below {
self.tree.remove(&path);
self.tree
.insert(format!("{to}{}", &path[from.len()..]), entry);
}
Ok(())
}
fn file_mut(&mut self, path: &str) -> Result<&mut Vec<u8>, i32> {
self.check_parent(path)?;
match self.tree.get_mut(path) {
None => Err(libc::ENOENT),
Some(None) => Err(libc::EISDIR),
Some(Some(content)) => Ok(content),
}
}
fn check_parent(&self, path: &str) -> Result<(), i32> {
let mut at = String::new();
let mut names = path.split('/').peekable();
while let Some(name) = names.next() {
if names.peek().is_none() {
break;
}
if !at.is_empty() {
at.push('/');
}
at.push_str(name);
match self.tree.get(&at) {
None => return Err(libc::ENOENT),
Some(Some(_)) => return Err(libc::ENOTDIR),
Some(None) => {}
}
}
Ok(())
}
fn has_children(&self, path: &str) -> bool {
self.tree.keys().any(|other| is_below(other, path))
}
pub(crate) fn files(&self) -> Vec<&String> {
self.tree
.iter()
.filter(|(_, entry)| entry.is_some())
.map(|(path, _)| path)
.collect()
}
/// Folders, the client's own root (`""`) first.
fn dirs(&self) -> Vec<String> {
std::iter::once(String::new())
.chain(
self.tree
.iter()
.filter(|(_, entry)| entry.is_none())
.map(|(path, _)| path.clone()),
)
.collect()
}
/// A random op, mostly one that should succeed.
pub(crate) fn next_op(&self, rng: &mut Rng) -> FsOp {
let dirs = self.dirs();
let files = self.files();
let in_dir = |rng: &mut Rng, names: &[&str]| {
let parent = rng.pick(&dirs).expect("the root is always there");
join(parent, rng.pick(names).expect("names"))
};
let a_file = |rng: &mut Rng| rng.pick(&files).map(|path| (*path).clone());
let roll = rng.below(100);
let op = match roll {
0..30 => Some(FsOp::Create {
path: in_dir(rng, &FILE_NAMES),
data: data(rng),
}),
30..45 => a_file(rng).map(|path| {
let len = self.content(&path).map_or(0, Vec::len) as u64;
FsOp::Write {
offset: rng.below(len + 8),
path,
data: data(rng),
}
}),
45..50 => a_file(rng).map(|path| FsOp::Truncate {
len: rng.below(32),
path,
}),
50..60 => Some(FsOp::Mkdir {
path: in_dir(rng, &DIR_NAMES),
}),
// A rename in place, over another file as often as not.
60..72 => a_file(rng).map(|from| FsOp::Rename {
to: join(parent_of(&from), rng.pick(&FILE_NAMES).expect("names")),
from,
}),
// A move to another folder, renamed or not.
72..82 => a_file(rng).map(|from| {
let name = if rng.chance(0.5) {
name_of(&from)
} else {
rng.pick(&FILE_NAMES).expect("names")
};
let parent = rng.pick(&dirs).expect("the root is always there");
FsOp::Rename {
to: join(parent, name),
from,
}
}),
// A folder renamed or moved, with what is in it.
82..86 => {
let from = rng.pick(&dirs[1..]).cloned();
from.map(|from| FsOp::Rename {
to: in_dir(rng, &DIR_NAMES),
from,
})
}
86..94 => a_file(rng).map(|path| FsOp::Unlink { path }),
94..97 => a_file(rng).map(|path| FsOp::UnlinkOpen {
path,
data: data(rng),
}),
_ => rng
.pick(&dirs[1..])
.cloned()
.map(|path| FsOp::Rmdir { path }),
};
op.unwrap_or_else(|| FsOp::Create {
path: in_dir(rng, &FILE_NAMES),
data: data(rng),
})
}
}
/// Content for a write: short and recognisable, now and then empty.
pub(crate) fn data(rng: &mut Rng) -> Vec<u8> {
let len = if rng.chance(0.1) {
0
} else {
rng.between(1, 48)
};
let tag = rng.next_u64();
(0..len).map(|i| b'a' + ((tag + i) % 26) as u8).collect()
}
pub(crate) fn join(parent: &str, name: &str) -> String {
if parent.is_empty() {
name.to_owned()
} else {
format!("{parent}/{name}")
}
}
pub(crate) fn parent_of(path: &str) -> &str {
path.rsplit_once('/').map_or("", |(parent, _)| parent)
}
pub(crate) fn name_of(path: &str) -> &str {
path.rsplit_once('/').map_or(path, |(_, name)| name)
}
fn is_below(path: &str, ancestor: &str) -> bool {
path.len() > ancestor.len() + 1
&& path.starts_with(ancestor)
&& path.as_bytes()[ancestor.len()] == b'/'
}
#[cfg(test)]
mod tests {
use super::*;
fn create(path: &str, data: &[u8]) -> FsOp {
FsOp::Create {
path: path.into(),
data: data.to_vec(),
}
}
#[test]
fn a_rename_over_a_file_replaces_it_and_a_folder_moves_with_its_contents() {
let mut model = Model::default();
model.apply(&FsOp::Mkdir { path: "x".into() }).unwrap();
model.apply(&create("x/a", b"1")).unwrap();
model.apply(&create("b", b"2")).unwrap();
model
.apply(&FsOp::Rename {
from: "b".into(),
to: "x/a".into(),
})
.unwrap();
assert_eq!(model.content("x/a"), Some(&b"2".to_vec()));
model
.apply(&FsOp::Rename {
from: "x".into(),
to: "y".into(),
})
.unwrap();
assert_eq!(model.tree().keys().collect::<Vec<_>>(), vec!["y", "y/a"]);
}
#[test]
fn calls_fail_the_way_posix_says() {
let mut model = Model::default();
model.apply(&FsOp::Mkdir { path: "x".into() }).unwrap();
model.apply(&create("x/a", b"1")).unwrap();
assert_eq!(
model.apply(&FsOp::Mkdir { path: "x".into() }),
Err(libc::EEXIST)
);
assert_eq!(model.apply(&create("x", b"")), Err(libc::EISDIR));
assert_eq!(model.apply(&create("x/a/b", b"")), Err(libc::ENOTDIR));
assert_eq!(model.apply(&create("q/a", b"")), Err(libc::ENOENT));
assert_eq!(
model.apply(&FsOp::Rmdir { path: "x".into() }),
Err(libc::ENOTEMPTY)
);
assert_eq!(
model.apply(&FsOp::Unlink { path: "x".into() }),
Err(libc::EISDIR)
);
let into_itself = FsOp::Rename {
from: "x".into(),
to: "x/y".into(),
};
assert_eq!(model.apply(&into_itself), Err(libc::EINVAL));
}
#[test]
fn a_write_past_the_end_leaves_a_hole() {
let mut model = Model::default();
model.apply(&create("a", b"ab")).unwrap();
let write = FsOp::Write {
path: "a".into(),
offset: 4,
data: b"z".to_vec(),
};
model.apply(&write).unwrap();
assert_eq!(model.content("a"), Some(&b"ab\0\0z".to_vec()));
let nothing = FsOp::Write {
path: "a".into(),
offset: 9,
data: Vec::new(),
};
model.apply(&nothing).unwrap();
assert_eq!(model.content("a"), Some(&b"ab\0\0z".to_vec()));
}
#[test]
fn generated_ops_mostly_succeed() {
let mut rng = Rng::new(9);
let mut model = Model::default();
let ok = (0..500)
.filter(|_| {
let op = model.next_op(&mut rng);
model.apply(&op).is_ok()
})
.count();
assert!(ok > 350, "{ok} of 500 succeeded");
}
}
+79
View File
@@ -0,0 +1,79 @@
//! A small seeded random number generator, so a failing simulation is a seed
//! that can be replayed. SplitMix64: fast, good enough for choosing operations
//! and faults, and no dependency.
#[derive(Clone, Debug)]
pub(crate) struct Rng(u64);
impl Rng {
pub(crate) fn new(seed: u64) -> Self {
Self(seed)
}
pub(crate) fn next_u64(&mut self) -> u64 {
self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
let mut z = self.0;
z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
z ^ (z >> 31)
}
/// A value in `0..n`; `0` when `n` is `0`.
pub(crate) fn below(&mut self, n: u64) -> u64 {
if n == 0 { 0 } else { self.next_u64() % n }
}
/// A value in `lo..=hi`.
pub(crate) fn between(&mut self, lo: u64, hi: u64) -> u64 {
lo + self.below(hi.saturating_sub(lo) + 1)
}
/// True with probability `p`.
pub(crate) fn chance(&mut self, p: f64) -> bool {
p > 0.0 && (self.next_u64() >> 11) as f64 / (1u64 << 53) as f64 <= p
}
pub(crate) fn pick<'a, T>(&mut self, items: &'a [T]) -> Option<&'a T> {
items.get(self.below(items.len() as u64) as usize)
}
/// A child generator, so one consumer's draws do not shift another's.
pub(crate) fn fork(&mut self) -> Self {
Self::new(self.next_u64())
}
}
#[cfg(test)]
mod tests {
use super::Rng;
#[test]
fn a_seed_replays_the_same_sequence() {
let (mut a, mut b) = (Rng::new(7), Rng::new(7));
for _ in 0..100 {
assert_eq!(a.next_u64(), b.next_u64());
}
assert_ne!(Rng::new(7).next_u64(), Rng::new(8).next_u64());
}
#[test]
fn bounded_draws_stay_in_range() {
let mut rng = Rng::new(1);
for _ in 0..1000 {
assert!(rng.below(5) < 5);
assert!((3..=9).contains(&rng.between(3, 9)));
}
assert_eq!(rng.below(0), 0);
assert!(!rng.chance(0.0));
assert!(rng.chance(1.0));
assert!([1, 2, 3].contains(rng.pick(&[1, 2, 3]).unwrap()));
assert_eq!(rng.pick::<u8>(&[]), None);
}
#[test]
fn a_fork_draws_apart_from_its_parent() {
let mut parent = Rng::new(3);
let mut child = parent.fork();
assert_ne!(parent.next_u64(), child.next_u64());
}
}
File diff suppressed because it is too large. Load diff
+322
View File
@@ -0,0 +1,322 @@
//! Turns a call through a mount that never returns into a failed seed.
//!
//! A deadlock in a daemon leaves the run's own thread blocked in the kernel,
//! where no budget check is reached, and a CI job then waits for its timeout.
//! The watchdog runs beside the run. When what it was armed for outlives its
//! limit, it aborts the FUSE connections of the run's mounts, so the blocked
//! call fails and the run reports the hang, and writes where every thread was
//! to `stacks.txt`.
//!
//! The stacks are taken in the process: no tracer can stop a thread that
//! waits on a FUSE request, so `gdb` and `eu-stack` wait as long as the hang.
//! Each thread is sent [`stack_signal`] and captures its own backtrace in the
//! handler. A thread in the kernel takes the signal only once it leaves, so
//! the mounts are aborted after the others have answered, and its backtrace
//! is the call it was blocked in. Its kernel wait channel is recorded first.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread::JoinHandle;
use std::time::{Duration, Instant};
use parking_lot::Mutex;
use super::daemon::wait_until;
/// How long the threads that are not blocked in the kernel have to answer,
/// and the blocked ones once the mounts are aborted.
const ANSWER: Duration = Duration::from_secs(5);
pub(crate) struct Watchdog {
shared: Arc<Shared>,
thread: Option<JoinHandle<()>>,
}
struct Shared {
/// The run's directory: its mounts are below it, and the stacks go in it.
dir: PathBuf,
armed: Mutex<Option<Armed>>,
/// What hung, once something has.
fired: Mutex<Option<Hung>>,
stop: AtomicBool,
}
struct Armed {
what: String,
limit: Duration,
since: Instant,
}
/// What hung, and where every thread was then.
#[derive(Clone, Debug, PartialEq)]
pub(crate) struct Hung {
pub(crate) what: String,
pub(crate) stacks: String,
}
/// Disarms the watchdog when dropped.
pub(crate) struct Watching(Arc<Shared>);
impl Watchdog {
pub(crate) fn new(dir: &Path) -> Self {
let shared = Arc::new(Shared {
dir: dir.to_path_buf(),
armed: Mutex::new(None),
fired: Mutex::new(None),
stop: AtomicBool::new(false),
});
let thread = {
let shared = shared.clone();
std::thread::Builder::new()
.name("sim-watchdog".into())
.spawn(move || watch(&shared))
.expect("spawn the simulation watchdog")
};
Self {
shared,
thread: Some(thread),
}
}
/// Watch `what` until the guard is dropped, and abort the mounts if it is
/// still going after `limit`.
pub(crate) fn arm(&self, limit: Duration, what: impl Into<String>) -> Watching {
*self.shared.armed.lock() = Some(Armed {
what: what.into(),
limit,
since: Instant::now(),
});
Watching(self.shared.clone())
}
/// What hung, if something did. Call [`Watchdog::stop`] first, or the
/// stacks may still be on their way.
pub(crate) fn fired(&self) -> Option<Hung> {
self.shared.fired.lock().clone()
}
/// Stop watching, once the stacks of a hang that was caught are taken.
pub(crate) fn stop(&mut self) {
self.shared.stop.store(true, Ordering::Relaxed);
if let Some(thread) = self.thread.take() {
let _ = thread.join();
}
}
}
impl Drop for Watching {
fn drop(&mut self) {
*self.0.armed.lock() = None;
}
}
impl Drop for Watchdog {
fn drop(&mut self) {
self.stop();
}
}
fn watch(shared: &Shared) {
while !shared.stop.load(Ordering::Relaxed) {
std::thread::sleep(Duration::from_millis(200));
let hung = shared
.armed
.lock()
.as_ref()
.filter(|armed| armed.since.elapsed() > armed.limit)
.map(|armed| format!("{} has not returned after {:?}", armed.what, armed.limit));
let Some(hung) = hung else { continue };
*shared.armed.lock() = None;
let path = shared.dir.join("stacks.txt");
// Before anything is let go: the run reports the hang, not whatever
// the aborted call answers.
*shared.fired.lock() = Some(Hung {
what: format!("{hung}; the stacks are in {}", path.display()),
stacks: String::new(),
});
let stacks = take_stacks(|| abort_mounts_below(&shared.dir));
let what = match std::fs::write(&path, &stacks) {
Ok(()) => format!("{hung}; the stacks are in {}", path.display()),
Err(error) => format!("{hung}; the stacks could not be kept: {error}"),
};
*shared.fired.lock() = Some(Hung { what, stacks });
}
}
/// One thread of this process as the kernel sees it.
struct Task {
tid: i32,
name: String,
/// `D` while it waits in the kernel uninterruptibly.
state: String,
wchan: String,
}
/// Where every other thread of this process is, with `release` called once
/// those not blocked in the kernel have answered.
fn take_stacks(release: impl FnOnce()) -> String {
let own = gettid();
let tasks = tasks();
STACKS.lock().clear();
install_stack_handler();
for task in &tasks {
if task.tid != own {
// SAFETY: tgkill only sends a signal, whose handler is installed.
unsafe {
libc::syscall(
libc::SYS_tgkill,
std::process::id() as i32,
task.tid,
stack_signal(),
)
};
}
}
let answered = |blocked: bool| {
let stacks = STACKS.lock();
tasks
.iter()
.filter(|task| task.tid != own && (blocked || task.state != "D"))
.all(|task| stacks.contains_key(&task.tid))
};
wait_until(ANSWER, || answered(false));
release();
wait_until(ANSWER, || answered(true));
let stacks = std::mem::take(&mut *STACKS.lock());
let mut out = String::new();
for task in tasks.iter().filter(|task| task.tid != own) {
out.push_str(&format!(
"thread {} {:?}, state {}, waiting in {}\n",
task.tid, task.name, task.state, task.wchan
));
match stacks.get(&task.tid) {
Some(stack) => out.push_str(&format!("{stack}\n")),
None => out.push_str(" (no answer)\n\n"),
}
}
out
}
fn tasks() -> Vec<Task> {
let Ok(entries) = std::fs::read_dir("/proc/self/task") else {
return Vec::new();
};
let read = |dir: &Path, file: &str| {
std::fs::read_to_string(dir.join(file))
.unwrap_or_default()
.trim()
.to_owned()
};
entries
.flatten()
.filter_map(|entry| {
let tid = entry.file_name().to_str()?.parse().ok()?;
let dir = entry.path();
// The state follows the name in parentheses, which may hold spaces.
let stat = read(&dir, "stat");
let state = stat.rsplit_once(") ")?.1.split(' ').next()?.to_owned();
Some(Task {
tid,
name: read(&dir, "comm"),
state,
wchan: read(&dir, "wchan"),
})
})
.collect()
}
/// The backtraces the threads captured, by thread id.
static STACKS: Mutex<BTreeMap<i32, std::backtrace::Backtrace>> = Mutex::new(BTreeMap::new());
/// The signal that asks a thread for its stack: one of the real-time signals
/// that nothing in the daemon uses.
fn stack_signal() -> i32 {
libc::SIGRTMIN() + 7
}
fn install_stack_handler() {
static INSTALLED: std::sync::Once = std::sync::Once::new();
extern "C" fn handler(_: i32) {
// Not async-signal-safe: the capture allocates. The process is being
// taken down for a hang anyway, and a thread interrupted inside the
// allocator only fails to answer.
let stack = std::backtrace::Backtrace::force_capture();
STACKS.lock().insert(gettid(), stack);
}
INSTALLED.call_once(|| {
// SAFETY: a zeroed sigaction with a handler and SA_RESTART is valid,
// and the handler only touches `STACKS`.
unsafe {
let mut action: libc::sigaction = std::mem::zeroed();
action.sa_sigaction = handler as extern "C" fn(i32) as usize;
action.sa_flags = libc::SA_RESTART;
libc::sigemptyset(&mut action.sa_mask);
libc::sigaction(stack_signal(), &action, std::ptr::null_mut());
}
});
}
fn gettid() -> i32 {
// SAFETY: gettid has no arguments and cannot fail.
unsafe { libc::gettid() }
}
/// Abort the FUSE connection of every mount below `dir`. Its connection is
/// named by the mount's device minor, and the mount's owner may abort it.
fn abort_mounts_below(dir: &Path) {
let Ok(mountinfo) = std::fs::read_to_string("/proc/self/mountinfo") else {
return;
};
for line in mountinfo.lines() {
let fields: Vec<&str> = line.split(' ').collect();
let (Some(device), Some(mountpoint)) = (fields.get(2), fields.get(4)) else {
continue;
};
if !Path::new(mountpoint).starts_with(dir) {
continue;
}
if let Some((_, minor)) = device.split_once(':') {
let _ = std::fs::write(format!("/sys/fs/fuse/connections/{minor}/abort"), "1");
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::sim::daemon::scratch;
#[test]
#[ignore = "signals every thread of the test process: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_call_past_its_limit_is_reported_with_the_stacks() {
let dir = scratch("watchdog");
let mut watchdog = Watchdog::new(&dir);
{
let _watching = watchdog.arm(Duration::from_millis(100), "a nap");
std::thread::sleep(Duration::from_secs(1));
}
watchdog.stop();
let hung = watchdog.fired().expect("the watchdog fired");
assert!(
hung.what.starts_with("a nap has not returned after"),
"{hung:?}"
);
assert!(hung.stacks.contains("a_call_past_its_limit"), "{hung:?}");
let kept = std::fs::read_to_string(dir.join("stacks.txt")).unwrap_or_default();
assert_eq!(kept, hung.stacks);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn a_call_within_its_limit_is_not() {
let dir = scratch("watchdog-quiet");
let watchdog = Watchdog::new(&dir);
drop(watchdog.arm(Duration::from_millis(100), "a quick call"));
std::thread::sleep(Duration::from_millis(500));
assert_eq!(watchdog.fired(), None);
drop(watchdog);
let _ = std::fs::remove_dir_all(&dir);
}
}
+338 -39
View File
@@ -129,6 +129,44 @@ pub(crate) struct WriteHandle {
} }
impl WriteHandle { impl WriteHandle {
/// A handle on a file `create` just made: an empty base, so everything
/// written through it is authored.
///
/// The base is `node` as it actually exists — the empty file the server just
/// minted, or a queued create's placeholder — never the local clock or a
/// blank. `queue_revision` turns it into `StagedWrite::based_on` and the
/// drain compares the remote against that. A `now_secs()` mtime differs from
/// the server's stamp by the create round-trip. A missing revision id drops
/// the check to `(mtime, size)`, and moving or renaming the file advances
/// its mtime on the server. Either way the first write to a brand-new file
/// lands as a conflict copy of its own empty create (`docs/BUGS.md` B113).
pub(crate) fn created(ino: u64, node: &Node, file: File, path: PathBuf) -> Self {
Self {
ino,
uid: node.uid.clone(),
file: Arc::new(file),
path,
written: Intervals::default(),
len: 0,
base_size: 0,
base_mtime: node.modification_time,
base_revision_id: crate::node_revision_id(node),
dirty: false,
open_count: 0,
}
}
/// Write over `node` from now on: a revision this daemon has just put on the
/// server under the handle, by sealing an upload or by landing the queued
/// create the handle was opened on. The base the handle opened over is gone,
/// and keeping it would make the drain read our own revision as another
/// device's edit.
pub(crate) fn rebase_onto(&mut self, node: &Node) {
self.base_mtime = node.modification_time;
self.base_size = crate::node_size(node);
self.base_revision_id = crate::node_revision_id(node);
}
/// Account for `[offset, end)` having been written through this handle. /// Account for `[offset, end)` having been written through this handle.
/// ///
/// A write past the end leaves a hole that reads as zeros, exactly like a /// A write past the end leaves a hole that reads as zeros, exactly like a
@@ -163,8 +201,23 @@ pub(crate) struct PendingRevision {
/// time so this module stays free of `fuser`. /// time so this module stays free of `fuser`.
pub(crate) type DirListing = Vec<(u64, bool, String)>; pub(crate) type DirListing = Vec<(u64, bool, String)>;
/// Where [`State::next_ino`] starts: far above any local id a database hands
/// out, so a fallback inode never names a node that has a row.
pub(crate) const FALLBACK_INO: u64 = 1 << 48;
/// The inode of the node with local id `lid` ([`Db::lids_for`]). One above the
/// lid, so no node takes [`crate::ROOT_INO`], which each session uses for its
/// own root. The same in every mount and after a restart.
pub(crate) fn lid_ino(lid: i64) -> u64 {
lid.unsigned_abs() + crate::ROOT_INO
}
/// Mutable inode bookkeeping, guarded by a mutex because fuser drives the /// Mutable inode bookkeeping, guarded by a mutex because fuser drives the
/// `Filesystem` trait through `&self`. /// `Filesystem` trait through `&self`.
///
/// One tree for every mount of the daemon. Each FUSE session sees it from its
/// own root ([`State::roots`]) and calls that root inode 1; `ProtonFs`
/// translates at the kernel boundary.
pub(crate) struct State { pub(crate) struct State {
/// inode -> node metadata. /// inode -> node metadata.
pub(crate) entries: HashMap<u64, Entry>, pub(crate) entries: HashMap<u64, Entry>,
@@ -173,6 +226,12 @@ pub(crate) struct State {
/// Cached directory listings: parent inode -> child inodes. Presence of a /// Cached directory listings: parent inode -> child inodes. Presence of a
/// key means the directory has been enumerated. /// key means the directory has been enumerated.
pub(crate) children: HashMap<u64, Vec<u64>>, pub(crate) children: HashMap<u64, Vec<u64>>,
/// The inodes some session has been rooted at ([`State::install_root`]).
/// A root is its own parent. Kept after its session unmounts: the node is
/// still one of ours.
pub(crate) roots: HashSet<u64>,
/// The next inode for a node the database could not give a local id, from
/// a range lids never reach ([`FALLBACK_INO`]).
pub(crate) next_ino: u64, pub(crate) next_ino: u64,
/// Shared write state keyed by inode. Concurrent writers share the scratch file. /// Shared write state keyed by inode. Concurrent writers share the scratch file.
pub(crate) active_writes: HashMap<u64, WriteHandle>, pub(crate) active_writes: HashMap<u64, WriteHandle>,
@@ -194,7 +253,7 @@ pub(crate) struct State {
/// intern/root refresh. Core drains this set and notifies the matching /// intern/root refresh. Core drains this set and notifies the matching
/// kernel only after releasing the State lock. /// kernel only after releasing the State lock.
pub(crate) access_changes: HashSet<u64>, pub(crate) access_changes: HashSet<u64>,
/// Persisted shared-root authority, loaded once per inode space and updated /// Persisted shared-root authority, loaded once and updated
/// on every explicit role/tombstone change. Descendant inheritance reads /// on every explicit role/tombstone change. Descendant inheritance reads
/// this map instead of issuing one SQLite query per interned inode. /// this map instead of issuing one SQLite query per interned inode.
pub(crate) share_access: HashMap<NodeUid, Access>, pub(crate) share_access: HashMap<NodeUid, Access>,
@@ -262,15 +321,6 @@ pub(crate) struct StateGuard<'a> {
db: &'a Arc<Db>, db: &'a Arc<Db>,
} }
/// Take a mount's inode lock through a [`StateGuard`], for the loops that walk
/// every live mount rather than this `Core`'s own.
pub(crate) fn lock_state<'a>(
state: &'a parking_lot::Mutex<State>,
db: &'a Arc<Db>,
) -> StateGuard<'a> {
StateGuard::new(state.lock(), db)
}
impl<'a> StateGuard<'a> { impl<'a> StateGuard<'a> {
pub(crate) fn new(guard: parking_lot::MutexGuard<'a, State>, db: &'a Arc<Db>) -> Self { pub(crate) fn new(guard: parking_lot::MutexGuard<'a, State>, db: &'a Arc<Db>) -> Self {
Self { Self {
@@ -311,9 +361,6 @@ impl Drop for StateGuard<'_> {
impl State { impl State {
/// A fresh, empty inode space over `db`. /// A fresh, empty inode space over `db`.
/// ///
/// `next_ino` differs by caller: a real mount hands out [`crate::ROOT_INO`]
/// to its root and starts allocating above it, while a bare state under test
/// has no root and starts at 1.
/// Apply the write-throughs this state owes SQLite, here and now. /// Apply the write-throughs this state owes SQLite, here and now.
/// ///
/// Production code never calls this: [`StateGuard`] releases the inode lock /// Production code never calls this: [`StateGuard`] releases the inode lock
@@ -340,12 +387,15 @@ impl State {
self.db.clone() self.db.clone()
} }
pub(crate) fn new(db: Arc<Db>, share_access: HashMap<NodeUid, Access>, next_ino: u64) -> Self { /// A fresh, empty tree over `db`. A node's inode comes from its local id
/// ([`lid_ino`]).
pub(crate) fn new(db: Arc<Db>, share_access: HashMap<NodeUid, Access>) -> Self {
Self { Self {
entries: HashMap::new(), entries: HashMap::new(),
by_uid: HashMap::new(), by_uid: HashMap::new(),
children: HashMap::new(), children: HashMap::new(),
next_ino, roots: HashSet::new(),
next_ino: FALLBACK_INO,
active_writes: HashMap::new(), active_writes: HashMap::new(),
handles: HashMap::new(), handles: HashMap::new(),
next_fh: 1, next_fh: 1,
@@ -356,18 +406,55 @@ impl State {
outbox: Vec::new(), outbox: Vec::new(),
} }
} }
/// Whether `uid` names an owned node currently reachable from this mount's /// Make `root` the root of a session and return its inode, which comes
/// root. Resident entries can outlive their dentries for open-handle and /// from its local id like any other node's.
/// access-revocation semantics, so `by_uid` membership alone is not proof ///
/// that a control-socket operation may address the node. /// The root is always owned. It must never be classified as a shared root,
pub(crate) fn owns_visible_uid(&self, uid: &NodeUid) -> bool { /// even when the API returns membership data (which happens when the owner
let Some(root) = self.entries.get(&crate::ROOT_INO) else { /// shares the folder with someone else), so a stale share row a downgrade
return false; /// may have left for it is dropped. A node already in the tree keeps its
}; /// inode and its lookups, and stops naming a parent: a session's `..` at
if uid.volume_id != root.uid.volume_id { /// its root is the root.
return false; pub(crate) fn install_root(&mut self, root: Node) -> u64 {
if let Err(e) = self.db.upsert_node(&root) {
warn!(uid = %root.uid, error = %e, "db upsert root failed");
} }
self.share_access.remove(&root.uid);
let _ = self.db.delete_share_access(&root.uid);
let ino = match self.by_uid.get(&root.uid) {
Some(&ino) => ino,
None => self.new_inos(&[&root])[&root.uid],
};
if let Some(old) = self.entries.get(&ino).map(|e| e.parent)
&& old != ino
&& let Some(kids) = self.children.get_mut(&old)
{
kids.retain(|&k| k != ino);
}
self.by_uid.insert(root.uid.clone(), ino);
let lookup_count = self.entries.get(&ino).map_or(1, |e| e.lookup_count.max(1));
self.entries.insert(
ino,
Entry {
uid: root.uid.clone(),
parent: ino,
node: root,
access: Access::Owner,
lookup_count,
open_count: 0,
unlinked: false,
},
);
self.roots.insert(ino);
ino
}
/// Whether `uid` names an owned node currently reachable from one of
/// `roots`, the roots of the sessions that are mounted. Resident entries
/// can outlive their dentries for open-handle and access-revocation
/// semantics, so `by_uid` membership alone is not proof that a
/// control-socket operation may address the node.
pub(crate) fn owns_visible_uid(&self, uid: &NodeUid, roots: &[u64]) -> bool {
let Some(&target) = self.by_uid.get(uid) else { let Some(&target) = self.by_uid.get(uid) else {
return false; return false;
}; };
@@ -380,7 +467,9 @@ impl State {
let Some(entry) = self.entries.get(&current) else { let Some(entry) = self.entries.get(&current) else {
return false; return false;
}; };
if entry.uid.volume_id != root.uid.volume_id // The node's root scopes it to its own volume, excluding foreign
// shared-with-me residents from uid-addressed sharing.
if entry.uid.volume_id != uid.volume_id
|| entry.unlinked || entry.unlinked
|| entry.node.trashed || entry.node.trashed
|| entry.uid != entry.node.uid || entry.uid != entry.node.uid
@@ -388,7 +477,7 @@ impl State {
{ {
return false; return false;
} }
if current == crate::ROOT_INO { if roots.contains(&current) {
return true; return true;
} }
@@ -430,15 +519,17 @@ impl State {
self.access_changes.drain().collect() self.access_changes.drain().collect()
} }
/// Whether `uid` lives on the volume this mount is rooted in. /// Whether `uid` lives on a volume some mount is rooted in.
/// ///
/// Used to decide how a node with no resolvable parent is classified: /// Used to decide how a node with no resolvable parent is classified:
/// our own volume fails open (it is owned content), a foreign volume /// our own volume fails open (it is owned content), a foreign volume
/// fails closed (it can only be there because of a share). /// fails closed (it can only be there because of a share).
pub(crate) fn is_own_volume(&self, uid: &NodeUid) -> bool { pub(crate) fn is_own_volume(&self, uid: &NodeUid) -> bool {
self.entries self.roots.iter().any(|ino| {
.get(&crate::ROOT_INO) self.entries
.is_some_and(|root| root.uid.volume_id == uid.volume_id) .get(ino)
.is_some_and(|root| root.uid.volume_id == uid.volume_id)
})
} }
fn stored_share_access(&self, uid: &NodeUid) -> Option<Access> { fn stored_share_access(&self, uid: &NodeUid) -> Option<Access> {
@@ -787,11 +878,54 @@ impl State {
} }
} }
pub(crate) fn intern(&mut self, parent: u64, node: Node) -> u64 { pub(crate) fn intern(&mut self, parent: u64, mut node: Node) -> u64 {
self.keep_revision_mtime(&mut node);
self.outbox.push(DbWrite::Upsert(vec![node.clone()])); self.outbox.push(DbWrite::Upsert(vec![node.clone()]));
self.intern_mem(parent, node) self.intern_mem(parent, node)
} }
/// Keep what is already known of a file whose content has not changed.
///
/// Drive stamps a link's `ModifyTime` on a rename or a move as well, while
/// POSIX leaves a file's mtime alone there. The content cache is keyed by
/// it, too: taking the new stamp made the cached bytes unreachable, so the
/// next partial write could not fill its gaps (docs/BUGS.md B123). The
/// active revision id says whether the content is still the same, and so
/// whether the size and digest a listing left out are still the ones known.
fn keep_revision_mtime(&self, node: &mut Node) {
let Some(entry) = self
.by_uid
.get(&node.uid)
.and_then(|ino| self.entries.get(ino))
else {
return;
};
if let (
NodeKind::File {
active_revision_id: Some(now),
claimed_size,
content_sha1,
..
},
NodeKind::File {
active_revision_id: Some(was),
claimed_size: size_was,
content_sha1: sha1_was,
..
},
) = (&mut node.kind, &entry.node.kind)
&& now == was
{
node.modification_time = entry.node.modification_time;
if claimed_size.is_none() {
*claimed_size = *size_was;
}
if content_sha1.is_none() {
content_sha1.clone_from(sha1_was);
}
}
}
/// Allocate (or reuse) a stable inode for a node, updating the hot-cache maps /// Allocate (or reuse) a stable inode for a node, updating the hot-cache maps
/// only. Every caller owes the DB a write-through — see the callers below; /// only. Every caller owes the DB a write-through — see the callers below;
/// the split exists so a batch can pay for one transaction instead of `n`. /// the split exists so a batch can pay for one transaction instead of `n`.
@@ -800,7 +934,19 @@ impl State {
self.intern_mem_with_access(parent, node, access) self.intern_mem_with_access(parent, node, access)
} }
fn intern_mem_with_access(&mut self, parent: u64, mut node: Node, access: Access) -> u64 { fn intern_mem_with_access(&mut self, parent: u64, node: Node, access: Access) -> u64 {
self.intern_mem_at(parent, node, access, None)
}
/// [`State::intern_mem_with_access`], with the inode a new entry takes when
/// the caller already has it ([`State::new_inos`]).
fn intern_mem_at(
&mut self,
parent: u64,
mut node: Node,
access: Access,
ino: Option<u64>,
) -> u64 {
if let Some(&ino) = self.by_uid.get(&node.uid) { if let Some(&ino) = self.by_uid.get(&node.uid) {
let changed = self let changed = self
.entries .entries
@@ -819,8 +965,13 @@ impl State {
} }
return ino; return ino;
} }
let ino = self.next_ino; let ino = match ino {
self.next_ino += 1; Some(ino) => ino,
None => self
.new_inos(&[&node])
.remove(&node.uid)
.unwrap_or_default(),
};
self.by_uid.insert(node.uid.clone(), ino); self.by_uid.insert(node.uid.clone(), ino);
self.entries.insert( self.entries.insert(
ino, ino,
@@ -837,6 +988,41 @@ impl State {
ino ino
} }
/// The inodes for nodes that have no entry yet, from their local ids, in
/// one transaction. Should the database fail, they come from
/// [`State::next_ino`] instead, so the mount keeps working, with inodes
/// that last only this run. So do the inodes of virtual folders, which are
/// never stored.
fn new_inos(&mut self, nodes: &[&Node]) -> HashMap<NodeUid, u64> {
let (virtual_nodes, stored): (Vec<&Node>, Vec<&Node>) = nodes
.iter()
.partition(|node| crate::r#virtual::is_virtual_uid(&node.uid));
let mut inos = match self.db.lids_for(&stored) {
Ok(lids) => stored
.iter()
.zip(lids)
.map(|(node, lid)| (node.uid.clone(), lid_ino(lid)))
.collect(),
Err(error) => {
warn!(%error, "no local ids for new inodes; numbering them for this run");
self.run_inos(&stored)
}
};
inos.extend(self.run_inos(&virtual_nodes));
inos
}
fn run_inos(&mut self, nodes: &[&Node]) -> HashMap<NodeUid, u64> {
nodes
.iter()
.map(|node| {
let ino = self.next_ino;
self.next_ino += 1;
(node.uid.clone(), ino)
})
.collect()
}
/// Decrement lookup count for an inode and prune if lookup_count == 0 && open_count == 0 && unlinked. /// Decrement lookup count for an inode and prune if lookup_count == 0 && open_count == 0 && unlinked.
pub(crate) fn forget_lookup(&mut self, ino: u64, nlookup: u64) -> Option<(u64, String)> { pub(crate) fn forget_lookup(&mut self, ino: u64, nlookup: u64) -> Option<(u64, String)> {
if let Some(entry) = self.entries.get_mut(&ino) { if let Some(entry) = self.entries.get_mut(&ino) {
@@ -877,11 +1063,27 @@ impl State {
/// ///
/// This is what keeps `ls` on a large folder quick: one commit for the /// This is what keeps `ls` on a large folder quick: one commit for the
/// listing, rather than one autocommit — and one fsync — per child. /// listing, rather than one autocommit — and one fsync — per child.
pub(crate) fn intern_batch(&mut self, parent: u64, nodes: Vec<Node>) -> Vec<u64> { pub(crate) fn intern_batch(&mut self, parent: u64, mut nodes: Vec<Node>) -> Vec<u64> {
for node in &mut nodes {
self.keep_revision_mtime(node);
}
self.outbox.push(DbWrite::Upsert(nodes.clone())); self.outbox.push(DbWrite::Upsert(nodes.clone()));
let fresh: Vec<&Node> = nodes
.iter()
.filter(|node| !self.by_uid.contains_key(&node.uid))
.collect();
let mut inos = if fresh.is_empty() {
HashMap::new()
} else {
self.new_inos(&fresh)
};
nodes nodes
.into_iter() .into_iter()
.map(|node| self.intern_mem(parent, node)) .map(|node| {
let access = self.access_for_node(parent, &node);
let ino = inos.remove(&node.uid);
self.intern_mem_at(parent, node, access, ino)
})
.collect() .collect()
} }
@@ -1199,7 +1401,7 @@ mod tests {
let dir = TempDir::new(); let dir = TempDir::new();
let db = Db::open(&dir.0.join("cache.db")).unwrap(); let db = Db::open(&dir.0.join("cache.db")).unwrap();
let share_access = db.all_share_access().unwrap(); let share_access = db.all_share_access().unwrap();
let st = State::new(Arc::new(db), share_access, 1); let st = State::new(Arc::new(db), share_access);
(st, dir) (st, dir)
} }
@@ -1217,6 +1419,36 @@ mod tests {
(st, dir, src, dst) (st, dir, src, dst)
} }
/// A node's inode comes from its row's local id, so it is the same in
/// every mount, after the hot cache drops it, and after a restart.
#[test]
fn a_node_keeps_its_inode_across_mounts_evictions_and_restarts() {
let (mut st, _dir) = state();
let inos = st.intern_batch(
0,
vec![
node("a", "root", "a.txt", false),
node("b", "root", "b.txt", false),
],
);
st.flush_outbox();
let db = st.db.clone();
assert_ne!(inos[0], inos[1]);
assert!(!inos.contains(&crate::ROOT_INO));
let mut other = State::new(db.clone(), HashMap::new());
assert_eq!(other.intern(0, node("b", "root", "b.txt", false)), inos[1]);
st.forget_mem(&uid("a"));
assert_eq!(st.intern(0, node("a", "root", "a.txt", false)), inos[0]);
let mut restarted = State::new(db, HashMap::new());
assert_eq!(
restarted.intern_batch(0, vec![node("a", "root", "a.txt", false)]),
[inos[0]]
);
}
/// Audit A5. Moving a file out of a folder leaves that folder's DB row /// Audit A5. Moving a file out of a folder leaves that folder's DB row
/// claiming a complete listing, so once the hot cache drops it, remote /// claiming a complete listing, so once the hot cache drops it, remote
/// changes under it stop being seen. `relocate` has to clear both sides. /// changes under it stop being seen. `relocate` has to clear both sides.
@@ -1282,7 +1514,7 @@ mod tests {
let root = uid("root").to_string(); let root = uid("root").to_string();
let state = parking_lot::Mutex::new(st); let state = parking_lot::Mutex::new(st);
{ {
let mut guard = lock_state(&state, &db); let mut guard = StateGuard::new(state.lock(), &db);
guard.intern(0, node("root", "none", "My Files", true)); guard.intern(0, node("root", "none", "My Files", true));
assert!( assert!(
db.node_by_uid(&root).unwrap().is_none(), db.node_by_uid(&root).unwrap().is_none(),
@@ -1525,6 +1757,7 @@ mod tests {
}) })
.collect(); .collect();
st.next_ino = crate::ROOT_INO + nodes.len() as u64; st.next_ino = crate::ROOT_INO + nodes.len() as u64;
st.roots.insert(crate::ROOT_INO);
// Reverse order so entries are inserted before their parents, matching // Reverse order so entries are inserted before their parents, matching
// the hydration path's non-topological materialization. // the hydration path's non-topological materialization.
for (i, node) in nodes.iter().enumerate().rev() { for (i, node) in nodes.iter().enumerate().rev() {
@@ -1899,6 +2132,51 @@ mod tests {
assert_eq!(st.entries[&shared].access, Access::Viewer); assert_eq!(st.entries[&shared].access, Access::Viewer);
} }
fn at_revision(mut node: Node, revision: &str, mtime: i64) -> Node {
node.modification_time = mtime;
if let NodeKind::File {
active_revision_id, ..
} = &mut node.kind
{
*active_revision_id = Some(revision.into());
}
node
}
#[test]
fn a_renamed_file_keeps_its_mtime_until_its_content_changes() {
let (mut st, _dir) = state();
let root = st.intern(0, node("root", "none", "My Files", true));
let mut file = node("file", "root", "a.txt", false);
if let NodeKind::File {
claimed_size,
total_size_on_storage,
..
} = &mut file.kind
{
*claimed_size = Some(7);
*total_size_on_storage = 519;
}
let ino = st.intern(root, at_revision(file.clone(), "r1", 100));
// Drive stamps the link on a rename; the content is still r1.
let mut renamed = at_revision(file.clone(), "r1", 160);
renamed.name = "b.txt".into();
st.intern_batch(root, vec![renamed]);
assert_eq!(st.entries[&ino].node.modification_time, 100);
assert_eq!(st.entries[&ino].node.name, "b.txt");
// A listing that leaves the size out still describes revision r1.
let mut bare = at_revision(file.clone(), "r1", 170);
assert_eq!(crate::node_size(&st.entries[&ino].node), 7);
if let NodeKind::File { claimed_size, .. } = &mut bare.kind {
*claimed_size = None;
}
st.intern(root, bare);
assert_eq!(st.entries[&ino].node.modification_time, 100);
assert_eq!(crate::node_size(&st.entries[&ino].node), 7);
st.intern(root, at_revision(file, "r2", 200));
assert_eq!(st.entries[&ino].node.modification_time, 200);
}
fn handle(len: u64, base_size: u64) -> WriteHandle { fn handle(len: u64, base_size: u64) -> WriteHandle {
let path = std::env::temp_dir().join(format!("pdfs-handle-test-{}", std::process::id())); let path = std::env::temp_dir().join(format!("pdfs-handle-test-{}", std::process::id()));
let file = File::create(&path).unwrap(); let file = File::create(&path).unwrap();
@@ -1918,6 +2196,27 @@ mod tests {
} }
} }
/// A file created offline is written while its queued create lands. The
/// placeholder had no revision, so the handle takes the empty file the
/// create made as its base (docs/BUGS.md B113).
#[test]
fn a_handle_on_a_placeholder_is_based_on_the_create_that_landed() {
let mut h = handle(0, 0);
assert_eq!(h.base_revision_id, None);
let mut landed = node("real", "root", "f.mkv", false);
landed.modification_time = 160;
if let NodeKind::File {
active_revision_id, ..
} = &mut landed.kind
{
*active_revision_id = Some("empty".into());
}
h.rebase_onto(&landed);
assert_eq!(h.base_mtime, 160);
assert_eq!(h.base_size, 0);
assert_eq!(h.base_revision_id.as_deref(), Some("empty"));
}
#[test] #[test]
fn a_listing_does_not_shrink_a_file_open_for_writing() { fn a_listing_does_not_shrink_a_file_open_for_writing() {
let (mut st, _dir) = state(); let (mut st, _dir) = state();
+4 -5
View File
@@ -165,7 +165,7 @@ impl Core {
} }
match self match self
.rt .rt
.block_on(self.client.trash_nodes(std::slice::from_ref(uid))) .block_on(self.drive.trash_nodes(std::slice::from_ref(uid)))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
Ok(()) => {} Ok(()) => {}
@@ -175,10 +175,9 @@ impl Core {
return; return;
} }
} }
// Every mount, not just the primary: a conflict copy inside a sync // Leaving a conflict copy hooked up in the tree keeps a node we have
// folder lives in that fork's inode space, and leaving it hooked up // just trashed remotely visible and readable in whichever mount shows
// there keeps a node we have just trashed remotely visible and // it (`docs/BUGS.md` B74).
// readable there (`docs/BUGS.md` B74).
self.for_each_state(|st| { self.for_each_state(|st| {
st.forget_or_unlink(uid); st.forget_or_unlink(uid);
}); });
+14 -14
View File
@@ -370,7 +370,7 @@ impl Core {
HashMap::new() HashMap::new()
} else { } else {
self.rt self.rt
.block_on(self.client.enumerate_nodes_light(&check)) .block_on(self.drive.enumerate_nodes_light(&check))
.map_err(|e| format!("resolve changed nodes: {e}"))? .map_err(|e| format!("resolve changed nodes: {e}"))?
.into_iter() .into_iter()
.filter(|n| !n.trashed) .filter(|n| !n.trashed)
@@ -525,7 +525,7 @@ impl Core {
}; };
if let Err(e) = self if let Err(e) = self
.rt .rt
.block_on(self.client.trash_nodes(&[uid])) .block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
warn!(rel, error = %e, "sync: trash remote failed"); warn!(rel, error = %e, "sync: trash remote failed");
@@ -776,7 +776,7 @@ impl Core {
let remote = r.expect("remote deletion requires a remote file"); let remote = r.expect("remote deletion requires a remote file");
if let Err(e) = self if let Err(e) = self
.rt .rt
.block_on(self.client.trash_nodes(std::slice::from_ref(&remote.uid))) .block_on(self.drive.trash_nodes(std::slice::from_ref(&remote.uid)))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
warn!(rel, error = %e, "sync: trash remote failed"); warn!(rel, error = %e, "sync: trash remote failed");
@@ -839,7 +839,7 @@ impl Core {
for (rel, uid) in delete_remote_dirs { for (rel, uid) in delete_remote_dirs {
if let Err(e) = self if let Err(e) = self
.rt .rt
.block_on(self.client.trash_nodes(&[uid])) .block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
warn!(rel, error = %e, "sync: trash remote folder failed"); warn!(rel, error = %e, "sync: trash remote folder failed");
@@ -894,14 +894,14 @@ impl Core {
) -> Result<(), String> { ) -> Result<(), String> {
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(folder)) .block_on(self.drive.enumerate_folder_children_node_uids(folder))
.map_err(|e| format!("enumerate {folder}: {e}"))?; .map_err(|e| format!("enumerate {folder}: {e}"))?;
if uids.is_empty() { if uids.is_empty() {
return Ok(()); return Ok(());
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes_light(&uids)) .block_on(self.drive.enumerate_nodes_light(&uids))
.map_err(|e| format!("resolve nodes: {e}"))?; .map_err(|e| format!("resolve nodes: {e}"))?;
for node in nodes { for node in nodes {
if node.trashed || !node.is_folder() { if node.trashed || !node.is_folder() {
@@ -946,14 +946,14 @@ impl Core {
} }
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(folder)) .block_on(self.drive.enumerate_folder_children_node_uids(folder))
.map_err(|e| format!("enumerate {folder}: {e}"))?; .map_err(|e| format!("enumerate {folder}: {e}"))?;
if uids.is_empty() { if uids.is_empty() {
return Ok(()); return Ok(());
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes_light(&uids)) .block_on(self.drive.enumerate_nodes_light(&uids))
.map_err(|e| format!("resolve nodes: {e}"))?; .map_err(|e| format!("resolve nodes: {e}"))?;
// Files whose recorded mtime no longer matches: their size has to be // Files whose recorded mtime no longer matches: their size has to be
@@ -971,7 +971,7 @@ impl Core {
HashMap::new() HashMap::new()
} else { } else {
self.rt self.rt
.block_on(self.client.enumerate_nodes(&stale)) .block_on(self.drive.enumerate_nodes(&stale))
.map_err(|e| format!("resolve nodes: {e}"))? .map_err(|e| format!("resolve nodes: {e}"))?
.iter() .iter()
.map(|n| (n.uid.to_string(), remote_file_sig(n).1)) .map(|n| (n.uid.to_string(), remote_file_sig(n).1))
@@ -1139,7 +1139,7 @@ impl Core {
match op { match op {
Pending::CreateDir { rel, parent } => { Pending::CreateDir { rel, parent } => {
let uid = self let uid = self
.client .drive
.create_folder(parent, base_name(rel), Some(now_secs())) .create_folder(parent, base_name(rel), Some(now_secs()))
.await .await
.map_err(|e| format!("create remote folder {rel}: {e}"))?; .map_err(|e| format!("create remote folder {rel}: {e}"))?;
@@ -1240,7 +1240,7 @@ impl Core {
.begin(name, "", TransferDirection::Upload, meta.len()), .begin(name, "", TransferDirection::Upload, meta.len()),
); );
let uid = self let uid = self
.client .drive
// Mirror push: the local file is authoritative. If a prior attempt was // Mirror push: the local file is authoritative. If a prior attempt was
// interrupted mid-upload it left an unsealed draft of this name; recover // interrupted mid-upload it left an unsealed draft of this name; recover
// it even across a daemon restart (which rotates our client uid), so the // it even across a daemon restart (which rotates our client uid), so the
@@ -1294,7 +1294,7 @@ impl Core {
meta.len(), meta.len(),
), ),
); );
self.client self.drive
.upload_new_revision_from(uid, reader, meta.len() as i64, thumbnails, Some(mtime)) .upload_new_revision_from(uid, reader, meta.len() as i64, thumbnails, Some(mtime))
.await .await
.map_err(|e| format!("upload revision {rel}: {e}"))?; .map_err(|e| format!("upload revision {rel}: {e}"))?;
@@ -1359,7 +1359,7 @@ impl Core {
size.max(0) as u64, size.max(0) as u64,
); );
let mut out = CountingWriter::new(out, &guard); let mut out = CountingWriter::new(out, &guard);
self.client self.drive
.download_file_to(uid, &mut out) .download_file_to(uid, &mut out)
.await .await
.map_err(|e| { .map_err(|e| {
@@ -1424,7 +1424,7 @@ impl Core {
let (lmtime, lsize) = (local.mtime, local.size); let (lmtime, lsize) = (local.mtime, local.size);
// Re-fetch the node so the stored remote signature is exactly what a walk // Re-fetch the node so the stored remote signature is exactly what a walk
// will report next time. // will report next time.
let (rmtime, rsize) = match self.client.enumerate_nodes(std::slice::from_ref(uid)).await { let (rmtime, rsize) = match self.drive.enumerate_nodes(std::slice::from_ref(uid)).await {
Ok(nodes) => nodes Ok(nodes) => nodes
.first() .first()
.map(remote_file_sig) .map(remote_file_sig)
+4 -7
View File
@@ -73,7 +73,7 @@ async fn run_uploads(
.upload_thumbnails_async(t.path.clone(), t.name.clone()) .upload_thumbnails_async(t.path.clone(), t.name.clone())
.await; .await;
match core match core
.client .drive
.upload_file_from( .upload_file_from(
&t.parent_uid, &t.parent_uid,
&t.name, &t.name,
@@ -238,14 +238,11 @@ impl Core {
.map_err(|error| self.errno_error(error, "upload destination"))?; .map_err(|error| self.errno_error(error, "upload destination"))?;
let new_uid = self let new_uid = self
.rt .rt
.block_on( .block_on(self.drive.create_folder(parent_uid, name, Some(now_secs())))
self.client
.create_folder(parent_uid, name, Some(now_secs())),
)
.map_err(|e| CoreError::from_api(&e, &format!("create folder {name}")))?; .map_err(|e| CoreError::from_api(&e, &format!("create folder {name}")))?;
let node = self let node = self
.fetch_node(&new_uid) .fetch_minted_node(&new_uid, name)
.map_err(|e| self.errno_error(e, "fetch node"))?; .ok_or_else(|| minted_but_unlisted(name))?;
let mut st = self.state(); let mut st = self.state();
let ino = st.intern(pino, node); let ino = st.intern(pino, node);
if let Some(kids) = st.children.get_mut(&pino) if let Some(kids) = st.children.get_mut(&pino)
+746 -6
View File
@@ -12,6 +12,690 @@ Conventions:
--- ---
## B136 — A file renamed while its first upload is on the wire goes back to its old name
**Status:** Fixed (unverified).
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile one-client-flaky, seed 3), at
the end check for lost data. A file made offline was renamed after the drain had taken its create.
The create landed under the old name and the new name was gone. Every replay of the seed.
**Where:** `Db::finish_create` in `crates/pdfs-core/src/db/ops.rs`, `drain_local_node` and
`adopt_real_uid` in `crates/pdfs-fuse/src/drain.rs`.
**Cause.** A rename of a node whose create is still queued rewrites the create's parent and name
(`Db::rewrite_op_target`) instead of queuing a rename. The drain copies the op when it takes it,
so a rewrite while the create was on the wire changed only the row. When the create landed,
`finish_create` deleted the row without looking at it again, and `adopt_real_uid` put the node
back where Drive had it. The move was lost on Drive and in the tree.
The same seed showed a second fault. When a folder made offline landed, the nodes in it still
named the folder's placeholder uid as their parent in memory. The next write of one of them put
the placeholder back on its row.
**Fix.** The drain passes where it sent the create and where it landed (a conflict name, or the
root when the folder was gone) to `finish_create`. When the row names another parent or name than
it was sent with, the same transaction queues a rename of the real uid to it, with the landed
parent and name as the original ones. `adopt_real_uid` keeps the tree's name and parent while
that rename is queued. When a node lands, its children's parent is changed to its real uid.
**Test:** the unit test `a_rename_made_while_its_create_uploads_is_queued_for_the_landed_node`.
Seed 3 of one-client-flaky passes.
## B135 — A file just moved is sometimes not found by its new path
**Status:** Fixed (unverified).
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile three-clients, seed 1), once
B131 was fixed. `rename a.txt -> y/f.txt` succeeded, and the `rename y/f.txt -> y/a.txt` two
milliseconds later failed with `ENOENT`. Once in about four replays of the seed.
**Where:** the online path of `serve_rename` in `crates/pdfs-fuse/src/filesystem.rs`.
**Cause.** After the move lands on Drive, `serve_rename` settles the tree with `State::relocate`.
It took the inode lock directly (`state.lock()`) instead of through `Core::state`, whose guard
writes what a mutation owes the database once the lock is released (DB1). So the writes stayed in
the state's outbox: both folders no longer listed, and the node's new parent. Whichever thread
next released a guard applied them, after it had let go of the lock. The lookup of `y/f.txt`
found `y` unlisted in memory and asked the database, which still marked `y` as listed and placed
the file in the root, so it rebuilt `y` without the file. It took another thread taking the lock
in between, so it happened only now and then.
**Fix.** `serve_rename` relocates through `Core::state`, so the database has the move before the
reply.
**Test:** the unit test `an_online_rename_writes_the_move_through_before_it_answers`. With the
fix, seed 1 of three-clients passes repeatedly with `PDFS_SIM_KNOWN=fail`.
## B134 — A file deleted online while the drain checks its write is kept as a conflict copy
**Status:** Open.
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile three-clients, seed 3).
`c.bin` was written, renamed to `f.txt` and deleted, all online. The drain logged "queued write
conflicts; keeping a conflict copy" for `f.txt`, with "the file was trashed remotely". The copy
was named after the file, not `recovered-…`, so the tree still placed the file when the drain
looked. It did not come up again in five replays with debug logs.
**Where:** `trash_child` in `crates/pdfs-fuse/src/filesystem.rs`, and `Core::drain_revision` in
`crates/pdfs-fuse/src/drain.rs`.
**Cause.** An online `unlink` trashes the node on Drive first, and only then forgets it in the
tree and drops its queued write (`discard_queued_ops`, or `withdraw_queued_ops` for an open
file). A drain worker that has claimed the write and reads the node in between finds it trashed.
B105's check asks whether the op is still queued, and it still is, so the drain keeps the write
as a conflict copy of a file the user deleted. When the copy runs before the unlink discards the
staged blob, it lands next to where the file was.
**Test:** the simulation runs count such a conflict when the copy is named after the file, and
count a copy that lands. With `PDFS_SIM_KNOWN=fail`, seed 3 of three-clients fails on it when it
comes up, which depends on timing.
## B133 — A file deleted while its upload is in flight can still come back as a recovered copy
**Status:** Open.
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile one-client-deletes, seed 3).
`y/e.md` was written, and the drain picked the write up. The step held the drain's read of the
node, deleted the file, and released the read. The next step wrote `a.txt`. The drain then logged
"queued write conflicts; keeping a conflict copy" for `y/e.md` as `recovered-L4`, with "the file
was trashed remotely". It is B105 again, though B105's fix was in place. Here the copy failed
because the blob was already gone.
**Where:** `Core::drain_revision` in `crates/pdfs-fuse/src/drain.rs`, `Db::op_exists` in
`crates/pdfs-core/src/db/ops.rs`, and the `pending_op` table in
`crates/pdfs-core/src/db/migrations.rs`.
**Cause.** B105's fix asks whether the op the worker holds is still queued, by id. `pending_op.id`
is an `INTEGER PRIMARY KEY` without `AUTOINCREMENT`, so SQLite gives a new row the largest id
in the table plus one, or 1 when the table is empty. The delete dropped the newest row, so the
next op queued took the same id. `op_exists` found that other op and said the write was still
wanted. A SQLite workload queues a write to the database right after deleting its journal, so
this is the usual case there, not a rare one.
**Test:** the simulation runs make the file again, or not, while the drain's read is held. When
they did, they count the drain's conflict copy of it. With `PDFS_SIM_KNOWN=fail` one-client-deletes
fails on it in every seed.
## B132 — A folder whose mkdir is still queued is gone after a restart
**Status:** Open.
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile one-client-flaky, seed 4).
The first listing of the Drive root failed with the link down, and the client's folder `c0` was
made offline. The link came back, and the client restarted with 8 ops still queued, `c0`'s
mkdir among them. Right after the restart, a write to `c0/a.txt` failed with `ENOENT`. The kernel's
lookup of `c0` in the root had answered `ENOENT`. A moment later the mkdir landed.
**Where:** `Core::ensure_children` in `crates/pdfs-fuse/src/lib.rs`, with
`Core::relist_offline_listings` and `State::invalidate_listing`.
**Cause.** A listing served offline is invalidated when the link comes back
(`relist_offline_listings`), which also clears its `listed` row. The running daemon does not
notice: the kernel still holds the dentries. After a restart, the first lookup in the folder
lists it on Drive. Drive does not hold a node whose create is still queued, or does not list a
fresh one yet (the lag of B118). `ensure_children` takes Drive's listing as the whole folder and
records it as complete, without the queued node's placeholder. Every path through that node fails
with `ENOENT` until something lists the folder again. The same may happen without a restart
whenever a folder holding a queued create is listed on Drive.
**Test:** the simulation runs count an `ENOENT` after a restart that restored queued ops, in a
seed that lost the link, and end the seed there. With `PDFS_SIM_KNOWN=fail`, seed 4 of
one-client-flaky fails on it when the mkdir is still queued, which depends on timing.
## B131 — A write to a file whose create could not be read back lands as a conflict copy
**Status:** Fixed (unverified).
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile three-clients, seed 1).
`a.txt` was created while the link was down, and the link came back right after. The create
landed, and the drain logged "pending upload failed; will retry" for its placeholder with
"fetch node: no such file or folder". Half a second later a write to `a.txt` queued a revision.
The drain kept it as `a (sync-conflict …).txt`: "expected 19 bytes at mtime 1791005792, found
19 at 1791005793". No other device had touched the file.
**Where:** `Core::drain_local_node` and `Core::adopt_real_uid` in `crates/pdfs-fuse/src/drain.rs`.
**Cause.** `drain_local_node` retires the create's op (`retire_create`) before
`adopt_real_uid` reads the new node back. Drive can answer a create before a lookup finds the
node (B118), and then the read-back fails. By then the op is gone, so the "will retry" never
happens. The tree's entry has the real uid but still holds the placeholder's node: the local
clock's mtime and no revision id. `rebase_open_writes` and `rebaseline_pending` never run. A
write opened afterwards takes that node as its base. Drive stamped the file when the create
landed, and if that was in a later second than the placeholder's mtime, the drain sees the
remote revision changed and keeps the client's own write as a conflict copy.
Once the read-back succeeded, the same seed found a second gap. A write handle open on the file
is repointed at the real uid when the op retires, and its release fills its gaps from the content
cache. `adopt_real_uid` cached the create's bytes only after the read-back, which waits for Drive
to list the node. A release in between found nothing to fill from and queued an incomplete
revision, and the next write open was refused with `EIO` (as in B123).
**Fix.** `adopt_real_uid` reads the node back with `Core::read_back`, which asks again for a moment
while Drive does not list it yet, as the online create does (B118). Before the op retires,
`Core::cache_created` caches the uploaded bytes under the real uid, keyed by the size and mtime
the tree has for the file, which is what an open handle has as its base. `adopt_real_uid` caches
them again, keyed as Drive has the node, before it rebases anything onto it.
**Test:** seed 1 of three-clients passes with `PDFS_SIM_KNOWN=fail`. Before the fix it failed on
the conflict copy when the create landed in a later second than it was made, which depends on
timing, and then on the refused write.
## B130 — A read of a file that a remote change invalidates can hang the daemon
**Status:** Open.
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile one-client-flaky, seed 1).
After the last step, a read of `c0/z/f.txt` never returned. The watchdog's stacks showed the
read's worker in `read_range_remote`, waiting in `block_on` for its fetch, and a runtime worker
running `run_event_sync` blocked in the kernel in `inval_inode`, waiting for a page lock. The
read had started at 05:13:38. Its 120 s timeout, a 20 s `bounded` call and every other timer
of the runtime all fired at 05:17:24, the moment the watchdog aborted the mount.
**Where:** `apply_event` in `crates/pdfs-fuse/src/background.rs`, which runs on the runtime and
flushes its invalidations through `Core::for_each_mount` and `NotifyBatch::flush` in
`crates/pdfs-fuse/src/lib.rs`, and `Core::read_range_remote` in `crates/pdfs-fuse/src/reads.rs`.
**Cause.** The kernel holds the page lock of the pages a FUSE read fills until the daemon
answers. `inval_inode` on the same file waits for that lock, so it waits as long as the read.
The event sync sends it from a runtime worker, which blocks that worker in the kernel. If that
worker was the one parked on the runtime's driver, nothing drives the timers or the I/O any
more: tokio hands the driver on only when a parked worker is woken for new work, and the others
sleep on a condvar. The read's fetch waits on a timer (here the fake's latency, on Drive an
HTTP response), so it never finishes and never lets go of the page lock. The daemon stays stuck
until some other work wakes a worker, and on an idle machine that may never happen.
**Test:** the simulation runs recognise it in the watchdog's stacks (a `tokio-rt-worker` in
state `D` inside `NotifyBatch::flush`), count it and end the seed there. With
`PDFS_SIM_KNOWN=fail`, seed 1 of one-client-flaky fails on it.
## B129 — A file deleted or replaced while its create is on the wire stays on Drive
**Status:** Open.
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile one-client-lan, seed 4).
`a.txt` was created, renamed to `b.txt` and then to `e.md` while its create was still queued.
A moment later `rename f.txt -> e.md` failed with `EIO`, and the daemon logged "rename failed"
with Drive's `AlreadyExists`.
**Where:** `Core::remove_replaced` and `Core::discard_queued_ops` in `crates/pdfs-fuse/src/lib.rs`,
`trash_child` in `crates/pdfs-fuse/src/filesystem.rs`, and `drain_local_node` and
`upload_created_file` in `crates/pdfs-fuse/src/drain.rs`.
**Cause.** A node whose create is still queued has no node on Drive, so deleting it or renaming
another file over it only drops the queued op (`discard_queued_ops`). That cannot stop a drain
worker that has already claimed the op. `cancel_upload` only reaches revisions, and the upload
of a create has no cancel flag. The create lands anyway. `finish_create` finds no op row to
retire, and the tree has already forgotten the local uid, so nothing trashes the new node.
Drive keeps a file that the mount no longer shows. Its name stays taken: an online rename onto
it, as here, fails with `EIO`, and a later create with that name lands as a conflict copy.
**Test:** the simulation runs count a rename it fails and end the seed there. With
`PDFS_SIM_KNOWN=fail`, seed 4 of one-client-lan fails on it when it comes up, which depends
on timing.
## B128 — A file created offline over one deleted offline can be lost
**Status:** Open.
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile three-clients, seed 3).
While the link was down, `f.txt` was deleted and a new `f.txt` was made. Its first version had
not landed yet, so Drive held it empty. When the link came back, the drain logged "the name is
held by our own unanswered create; adopting it" with the old file's uid, then "pending trash
landed" for that uid. The queue drained empty, and Drive held no `f.txt`, only the trashed old
one.
**Where:** `drain_local_node` and `adoptable_twin` in `crates/pdfs-fuse/src/drain.rs`.
**Cause.** Several workers drain at once, so the new file's create can run before the trash of
the old one has landed, and Drive refuses the name. On that refusal the drain first looks for a
node its own unanswered create made: one with that name, made within `ADOPT_WINDOW_MS`, and
empty for a file. The old file matched, because it was made a moment earlier and its bytes were
still queued. The drain adopted it, so the new bytes went to it as a revision, and the queued
trash then trashed it. Only after the adoption check does the drain ask whether a queued trash
holds the name (`has_pending_trash_named`), and the adoption does not exclude a node that has
one.
**Test:** the simulation runs count it and go on. With `PDFS_SIM_KNOWN=fail` the three-client
profile fails on it when it comes up, which depends on timing.
## B127 — A file created offline lands twice when the answer to its upload is lost
**Status:** Open.
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile one-client-flaky, seed 1).
The upload of a queued create timed out after Drive had made the file. On the next attempt the
drain logged "name is taken remotely; creating under a conflict name". Drive ended with the file
and with a conflict copy of the same bytes next to it.
**Where:** `drain_local_node` and `adoptable` in `crates/pdfs-fuse/src/drain.rs`.
**Cause.** When a create finds its name taken, the drain adopts the node only if it looks like
one its own unanswered create made (`adoptable_twin`). It must be made within `ADOPT_WINDOW_MS`
and, for a file, be empty. A queued create uploads its bytes in the same call, though
(`upload_created_file`). So the node a lost answer leaves behind holds those bytes, is not
adopted, and the file is uploaded a second time under a conflict name.
**Test:** the simulation runs count it and go on. With `PDFS_SIM_KNOWN=fail` the flaky profile
fails on it when it comes up, which depends on timing.
## B126 — A file created offline under a name an offline rename freed lands as a conflict copy
**Status:** Open.
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile three-clients, seed 3).
While the link was down, `b.txt` was renamed to `c.bin`, and a new `b.txt` was made. When the link
came back, the drain logged "name is taken remotely; creating under a conflict name" for
`b.txt`. The new file ended up only as `b (sync-conflict …).txt`.
**Where:** `drain_local_node` in `crates/pdfs-fuse/src/drain.rs`, and the claim order in
`Db::claim_next_due_op` in `crates/pdfs-core/src/db/ops.rs`.
**Cause.** The drain serialises the ops of one node, but not ops on different nodes that share a
name. Several workers drain at once, so the create of the new `b.txt` can reach Drive before the
rename that frees the name has landed. Drive then refuses it as taken. `drain_local_node` waits
when the name is held by a node whose trash is queued (`has_pending_trash_named`), but not when a
queued rename is about to move it away. It treats the name as someone else's file and lands the
new one under a conflict name.
**Test:** the simulation runs count it and go on. With `PDFS_SIM_KNOWN=fail` the three-client
profile fails on it when it comes up, which depends on timing.
## B125 — A folder made offline fails with `EIO` once its listing is invalidated
**Status:** Open.
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile one-client-flaky, seeds 3
and 4). Right after a restart, `rmdir y` (seed 3), and a write to a file in the client's folder
(seed 4), failed with `EIO`. The daemon logged "enumerate folder children failed" with Drive's
404 for a `local~` folder uid. Each folder had been made while its create was being queued:
offline, or, as for the client's own folder, through B118's fallback when the read-back missed.
**Where:** `Core::ensure_children` in `crates/pdfs-fuse/src/lib.rs`, and everything that calls
`State::invalidate_listing`.
**Cause.** A folder whose mkdir is still queued exists only here, so `serve_mkdir` records it as
listed and empty, and `ensure_children` never asks Drive about it. `State::invalidate_listing`
clears that flag for any folder, though, and so do its callers: `relocate` after a rename, and
`relist_offline_listings` when the link comes back. The next access asks Drive to list a `local~`
uid. Drive answers 404, and every lookup in the folder fails with `EIO` until its mkdir lands.
Right after it lands, Drive may not list the new folder yet (the lag of B118). Then the listing
fails with a 404 for the real uid too.
**Test:** the simulation runs count it and go on. With `PDFS_SIM_KNOWN=fail` the flaky profile
fails on it.
## B124 — After a replacing rename fails, the file it was replacing is gone from the mount
**Status:** Fixed (unverified).
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile one-client-lan, seed 1). A
rename `x/f.txt -> z/d` over an existing `z/d` failed with `EIO` (B122). Then `create z/d`
failed with `ENOENT`, although `z/d` was back on Drive.
**Where:** `Core::remove_replaced` and `Core::restore_replaced` in `crates/pdfs-fuse/src/lib.rs`.
**Cause.** Before the rename, `remove_replaced` trashes the destination on Drive, drops it from
every mount's state and adds it to `hidden`, which filters listings. When the rename then fails,
`restore_replaced` restores the node on Drive but does not take it out of `hidden` or put it back
in the state, as the ordinary restore does. Until the daemon restarts, the file is on Drive but
missing from the mount, and a create by its name fails.
**Fix.** `restore_replaced` takes the node out of `hidden`, and drops the listing of the folder
it is restored to, and the kernel's entry for its name, in every mount that shows the folder. The
next lookup lists the folder again and finds the node.
**Test:** the unit test `a_replaced_node_put_back_shows_in_the_mount_again`. Since B122 is fixed,
the simulation runs no longer reach it through a rename across folders.
## B123 — A second write open on a file just written fails with `EIO`
**Status:** Fixed (unverified).
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile one-client-lan, seeds 5 and
9). A write at an offset (seed 5) or a create over an existing file (seed 9) failed with `EIO`.
The daemon logged "refusing write over incomplete queued revision". The one-client-echoes profile,
whose events come late, twice and out of order, found it in four more seeds.
**Where:** the write open in `crates/pdfs-fuse/src/filesystem.rs`, the `complete` flag that
`queue_revision` in `crates/pdfs-fuse/src/lib.rs` stamps on a queued revision, and everything that
leaves the content cache without the bytes of the file's base.
**Cause.** A write open over a queued revision that is not complete is refused by design, because
a second partial edit on top of one with gaps cannot be represented. Ordinary cases left a
revision incomplete, though:
- A file closed empty never counted as complete. `queue_revision` compared the authored ranges
with `[(0, len)]`, which for a length of 0 is `[(0, 0)]`, while the authored ranges are empty.
The truncate path already treated a size of 0 as complete.
- The gaps of a partial edit are filled from the content cache, and the cache could be empty or
out of reach then:
- The echo of our own change evicted it, taken for someone else's. `rename_remote`,
`move_remote` and the drain's `move_renaming` did not record an echo, and neither did a
create made online, so the create's echo took the one the upload recorded. The feed can also
deliver an event twice, or replay one after a batch out of order, and the second copy found
the echo already taken.
- Drive stamps a link's `ModifyTime` on a rename or a move too. The listing after it moved the
file's mtime, and the cache, keyed by mtime and size, no longer matched.
- When an upload landed, the drain let go of the staged blob, read the node back, rebased open
write handles onto it, and only then cached the blob. A handle released in between found
neither the blob nor the cached bytes.
Either way, the next write open on the file within the drain debounce (`DRAIN_REVISION_DEBOUNCE`,
2 s) failed, until the revision had landed.
**Fix.** `StagedWrite::covers` counts an empty file as complete, at every site that sets the flag.
`Core::changed_remote` records the echo of every rename and move, and `fetch_minted_node` the echo
of every create made online. The event sync skips a node update whose id it applied lately
(`RecentEvents`). `State::intern` keeps a file's mtime, and the size a listing left out, while its
active revision id stays the same. `refresh_after_upload` caches the uploaded bytes before it
rebases open handles, and lets go of the staged blob only after.
**Test:** the unit tests `an_empty_write_covers_the_file_and_a_gap_does_not`,
`a_node_update_seen_again_is_skipped_but_other_events_are_not`,
`a_renamed_file_keeps_its_mtime_until_its_content_changes` and
`a_rename_or_move_drops_the_sdk_copy_and_expects_its_echo`, and the FUSE tests
`a_file_closed_empty_takes_a_second_write_at_once` and
`the_echo_of_our_own_rename_keeps_the_cached_bytes` in `sim::daemon`. The simulation runs no
longer tolerate it.
## B122 — A rename across folders fails with `EIO` when the source folder holds the new name
**Status:** Fixed (unverified).
**Found:** 2026-10-03, by the simulation runs (`sim::run`, profile one-client-lan, seeds 1, 7
and 10). A `rename(2)` like `x/f.txt -> z/d` failed with `EIO` while `x/d` existed. The daemon
logged "rename failed" with Drive's `AlreadyExists`.
**Where:** `serve_rename` in `crates/pdfs-fuse/src/filesystem.rs`.
**Cause.** Online, a rename that changes both the folder and the name is done in two calls. It
renames in the source folder first, then moves (`rename_remote`, then `move_remote`). A sibling in
the source folder with the new name makes Drive refuse the first call, although the destination
is free. B46's follow-up already notes that this path was not converted to the single
`move-multiple` request the drain uses. If the rename was replacing a file, B124 follows.
**Fix.** `Core::move_rename_remote` moves and renames in one `move-multiple` request, so the name
only has to be free in the destination. `serve_rename` uses it for every rename across folders,
and the drain uses it in place of its own `move_renaming`.
**Test:** the FUSE test `a_rename_across_folders_ignores_the_new_name_in_the_old_folder` in
`sim::daemon`. The simulation runs no longer tolerate it.
## B121 — A move right after a rename fails with `EIO`, and a rename across folders takes seconds
**Status:** Fixed (unverified).
**Found:** 2026-10-03, in the journal of the account run on flaky Wi-Fi. Each change answered
correctly, but slowly. One rename and move through the mount collected 13 `operation failed …
method=PUT status=422` warnings over 8.2 seconds, and another collected 7 over 3.9 seconds. Each
one was a request that Drive refused as out of date and the daemon tried again half a second
later.
**Where:** every remote rename and move: `serve_rename` in `crates/pdfs-fuse/src/filesystem.rs`,
the control `rename` and `move_to` in `crates/pdfs-fuse/src/lib.rs`, `move_between` in
`crates/pdfs-fuse/src/relocate.rs`, and the drain's renames and `move_renaming` in
`crates/pdfs-fuse/src/drain.rs`.
**Cause.** A rename or move sends the node's current name hash as `OriginalHash`. The SDK takes
it from its entity cache before the hash in the link it has just fetched. Its own rename and move
do not update that cache entry. Only the event feed's echo of the change drops it, and the daemon
polls the feed every 10 seconds (`POLL_INTERVAL`). Until then, each rename or move of the same
node sent the old hash. A rename across folders, which is a rename and then a move, was refused
until the next poll, and the retries from B111 hid that as latency. A plain move was not retried
at all. When it came within the poll interval, `mv a b && mv b dir/` through the mount failed with
`EIO`, and `pdfs rename` followed by `pdfs move` failed with Drive's "out of date" error.
**Fix.** `Core::forget_sdk_node` passes the SDK the event its own change will echo, which drops
the cached node. Every rename, and every move but the drain's, goes through `Core::rename_remote`
or `Core::move_remote`. Both drop the entry once the change has landed, and before each retry of a
request Drive called out of date. The drain's `move_renaming` drops it once its move lands.
**Test:** the unit tests `a_rename_or_move_drops_the_sdk_copy_and_expects_its_echo` and
`every_remote_rename_and_move_rides_out_a_stale_hash`. On the account run, "regression B111" and
"regression B113" should show no `status=422` warnings.
## B120 — A file queued in a folder that has just landed is put off by five seconds
**Status:** Fixed (unverified).
**Found:** 2026-10-02, in the journal of the account run on flaky Wi-Fi. A folder made while the
link was down was queued as a mkdir. When it landed, the journal logged `pending create landed …
kind=mkdir`, and right after it `re-interned an authority missing from the local tree; retrying`
and `pending operation deferred … reason="the node was missing locally and has been re-read"` for
a file queued inside it. The file landed on the next recheck. No bytes were lost.
**Where:** `Db::finish_create` and `Db::remap_local_uid` in `crates/pdfs-core/src/db/ops.rs`, and
`adopt_real_uid` in `crates/pdfs-fuse/src/drain.rs`.
**Cause.** When a queued create landed, `finish_create` repointed the queued children at the real
uid and deleted the placeholder's row in one transaction. The real node's row was only written
later, by `adopt_real_uid`, after it had read the node back from Drive. A drain worker that picked
up a child in between found no row for its parent. It read the folder again from Drive and put
the child off by one recheck (`DRAIN_ACCESS_RECHECK`, 5 seconds).
**Fix.** `adopt_placeholder_row_tx` in `crates/pdfs-core/src/db/nodes.rs` hands the placeholder's
row to the real uid in the same transaction, along with the rows below it, and drops the old row
from the search index. `adopt_real_uid` writes the server's copy over it later. A row that the
real uid already has, from the event stream, is newer and is kept.
**Test:** the db tests `a_landed_folder_keeps_a_row_until_the_server_copy_replaces_it` and
`a_landed_placeholder_never_overwrites_the_server_copy`. The suite cannot queue a mkdir on demand,
because a mkdir online is made at once even while sync is paused. Instead, `--journal-check`
now counts `re-interned an authority missing from the local tree` as an error, so a run that goes
offline catches this race, and B117, if either comes back.
## B119 — Reading a file just written through the mount needs the network
**Status:** Fixed (unverified since the second fix below; "regression B119" passed on one account
run of 2026-10-03 and failed on the next).
**Found:** 2026-10-02, by the account run on flaky Wi-Fi, which crashed at its very end. The
run's last step reads back a file it wrote to compare digests. The Wi-Fi had dropped, the read
waited for `open_revision` for 120 seconds and then failed with `EIO`, and the suite stopped with
an `OSError` traceback.
**Where:** the landed-upload path of the drain in `crates/pdfs-fuse/src/drain.rs`.
**Cause.** Once an upload landed, the drain kept the staged bytes as cached content only for a
pinned file. For any other file it discarded them, and then dropped the reader too. The next read
of a file that had just been written therefore downloaded it again, and without a link it could
not.
**Fix.** After the upload, `refresh_after_upload` reads the node back and now returns it. The
staged blob becomes the file's cached content, linked rather than copied, and stamped with the
mtime and size of the revision just read back, so the cache treats it as current. A file larger
than a sixteenth of the cache budget (`LANDED_KEEP_SHARE`, 320 MiB of the default 5 GiB) is still
discarded unless it is pinned, so one large upload cannot push the rest of the cache out. Nothing
is kept when the read-back fails or reports a different size. The suite records a failed end-of-run
read-back as a failed check instead of crashing.
**Found again** on 2026-10-03, by "regression B119" on an account run over a slow link (uploads of
a few bytes took 1.4 seconds). The case failed with "block.bin not cached after the upload" at the
same second the journal logged that upload as landed.
**Cause, second part.** The drain retired the op first, and only then read the node back and
linked the blob into the cache. For that one round trip the status reported an empty queue for a
file that was not yet cached. Anything that waits for an empty queue before reading could still
need the network.
**Fix, second part.** `Core::landing_uploads` counts an upload from just before its op is retired
until its bytes are cached, and the status adds it to `pending_uploads`. The status reads the
queue before the count, so it always sees the upload in one of them. The op still goes first: an
op restored after a crash would re-send a revision this daemon has not yet recorded as its own,
and `revision_conflict` would keep it as a conflict copy.
**Test:** the drain tests `a_landed_upload_stays_cached_unless_it_would_crowd_the_cache_out` and
`a_landing_upload_counts_until_its_guard_goes`, `a_landed_upload_stays_pending_until_its_bytes_are_cached`
in `lib.rs`, and the acceptance case "regression B119", which writes two files, waits for the
queue and expects `pdfs ls` to list both as cached.
## B118 — Creating a file or folder through the mount can fail with `ENOENT` although it was made
**Status:** Fixed (unverified).
**Found:** 2026-10-02, by the account run on flaky Wi-Fi. In the first on-demand folder, case 14
failed in its sqlite step with "unable to open database file", and case 15 failed in `mkdir`
with `FileNotFoundError`. In both, the file or folder had been made on Drive.
**Where:** `serve_create` and `serve_mkdir` in `crates/pdfs-fuse/src/filesystem.rs`.
**Cause.** After a create or mkdir succeeds, the mount reads the new node back to answer the
kernel. Drive can answer the create before its listing returns the new node. The read-back then
came back empty, which `fetch_node` reports as `ENOENT`, and the mount passed that on with nothing
in the log. The program saw its create fail for something that existed.
**Fix.** `Core::fetch_minted_node` asks again after 250 ms and after one more second
(`MINTED_READ_BACK_DELAYS`). If the node still cannot be read, the mount queues the create as it
does offline (`queue_local_node`), and the drain adopts the existing node by its name. The program
gets its file or folder either way. `pdfs mkdir` uses the same read-back, and when it still misses
it says that the folder was made but is not listed yet, instead of reporting that it does not
exist.
**Test:** none that replays it on demand, since the gap depends on Drive. The affected acceptance
cases ("application workloads" and "block boundaries") fail when it comes back.
## B117 — An upload queued for a file `pdfs move` moved is listed by its raw id and put off once
**Status:** Fixed (verified 2026-10-03: "regression B117" passed in My files on the account run,
and its journal had no deferral).
**Found:** 2026-10-02, while verifying B113 live. Three files were moved with `pdfs move` while
their uploads were queued. Until each upload landed, `pdfs sync queue` listed it by its node id
instead of its new path. After the resume, the first attempt of each logged `pending operation
deferred … reason="the node was missing locally and has been re-read"`, and the upload landed on
the next recheck, about 8 seconds later. No bytes were lost.
**Where:** `settle_source` in `crates/pdfs-fuse/src/relocate.rs` and `Core::rename` in
`crates/pdfs-fuse/src/lib.rs`, which forget the node. `pending_op_infos` in
`crates/pdfs-fuse/src/queue.rs`, and `resolve_unknown_authority` in `crates/pdfs-fuse/src/drain.rs`.
**Cause.** `State::forget` deletes the node's row as well as its entry in memory, so the mounts
re-listed it under its new path. Until a listing brought the row back, nothing local knew the
node. `pending_op_infos` then had no path for the queued op and fell back to its name, or to the
raw uid. The drain's access check found no node either. It re-read the node from Drive, stored it
and put the op off to the next recheck instead of going on with it. `pdfs rename` forgot the node
the same way.
**Fix.** `pdfs move` and `pdfs rename` move the node instead of forgetting it
(`Core::relocate_everywhere`). Each mount that has the destination folder moves the node in place
with `State::relocate`, which keeps its inode and writes its row through. A mount without the
destination drops the node from memory only and keeps the row. When no mount had the node, its
row is moved directly. Both names are invalidated in the kernel, as B114 needs.
**Test:** the fuse test `a_control_move_keeps_the_node_and_its_row_and_drops_the_old_name`, and
the acceptance case "regression B117". It pauses sync, rewrites an uploaded file and moves it with
`pdfs move`. It checks the path `pdfs sync queue` shows for the upload, and looks in the journal
for a deferral of the node after the resume.
## B116 — Trashing a folder keeps the queued creates in its subfolders
**Status:** Fixed (verified 2026-10-03: "regression B116" passed in My files on the account run).
**Found:** 2026-10-02, by the first live run of the acceptance case "regression B113". That
version wrote `*.part` files, which park their create until a rename instead of minting the file
(B70). It moved one of them, still named `moved.part`, into the subfolder `b113/moved here` and
then hung, waiting for a queue that held a parked op. The run was interrupted, and its cleanup
trashed the test root with `pdfs rm` and deleted it permanently. The parked create stayed queued
and kept its staged bytes. The cleanup reported it as left over (`queued create #2 for
…/b113/moved here/moved.part`).
**Where:** `drop_doomed_ops` in `crates/pdfs-core/src/db/ops.rs`, which
`Core::discard_queued_ops` runs when a folder is trashed.
**Cause.** The ops removed with a folder are those for the folder and for the nodes queued below
it through `pending_op.parent_uid`, plus the revisions of every file the node tree places below
it (B102). A create for a file inside a *real* subfolder is neither: its parent is not a queued
node, and the tree only contributes revisions. Such a create comes from a transient name, or
from a file created offline in a folder that already exists on Drive. A parked one stays queued
for good, holding its bytes. One that is due would run into a trashed parent, and by the code the
drain then re-homes it to the root (`parent folder is gone remotely; creating in the root
instead`), so the file would reappear in My files after its folder was trashed. Only the parked
case has been seen.
**Fix.** `drop_doomed_ops` also takes the queued creates and mkdirs whose parent is anywhere in
the node tree below the folder, together with everything queued below those.
**Test:** the db test `trashing_a_folder_drops_the_queued_creates_in_its_subfolders`, and the
acceptance case "regression B116", which replays the due case. With sync paused, it writes a
`*.part` file two folders below a new folder and renames it to its final name, so its create is
queued and due. It then trashes the folder with `pdfs rm` and expects the create gone. It restores
the folder before sync resumes, so a create that wrongly stayed lands where the file was written,
and fails if the file shows up in the root anyway.
## B115 — `pdfs ls` shows a file's encrypted size
**Status:** Fixed (verified 2026-10-03: "regression B115" passed in My files on the account run).
**Found:** 2026-10-02, while verifying B88 live. `pdfs ls ~/Downloads/pdfs-b88-test` listed the
6-byte `note.txt` as 57 bytes, and listed `tickets.pdf` as 202316 bytes where the mount's `stat`
said 202257. The acceptance case "open B115" then showed it for every file it wrote, in My files
and in an on-demand folder, right after the upload and again after `pdfs refresh`: 6 bytes listed
as 57, 4097 as 4150, and 4 MiB + 7 bytes (4194311) as 4194421. The mount's `stat` had the real
sizes throughout.
**Where:** `node_size` in `crates/pdfs-fuse/src/lib.rs` and `crates/pdfs-core/src/db/nodes.rs`.
**Cause.** `node_size` reports the node's `claimed_size` and falls back to
`total_size_on_storage` when it has none. That is the size on Drive's storage: encrypted blocks
plus their signatures. A folder is first listed without claimed sizes, to be quick, and the real
sizes are fetched for the whole listing in the background. `stat` through the mount waits for that
fetch. `pdfs ls` and the app's file browser read the listing straight away, so they showed the
encrypted sizes until it landed.
**Fix.** `Core::list_dir`, which serves both, waits for the listing's size fetch first
(`settle_listing_sizes`), for at most `SizeUpgrade::WAIT` (10 seconds), as `stat` does. Offline it
does not wait, since the fetch cannot succeed.
**Test:** the fuse tests `a_listing_for_the_cli_settles_sizes_before_reading_them` and
`a_listing_waits_for_the_whole_size_batch_but_not_forever`, and the acceptance case
"regression B115". It compares the sizes `pdfs ls` reports with the written ones, after the upload
and again after `pdfs refresh`.
## B114 — A name `pdfs rm` or `pdfs rename` took away still resolves for up to 30 seconds
**Status:** Fixed (verified 2026-10-03: "regression B114" passed in My files on the account run).
**Found:** 2026-10-02, while verifying B88 live. After `pdfs rm ~/Downloads/pdfs-b88-test` the
folder was gone from `ls ~/Downloads`, but `ls -d ~/Downloads/pdfs-b88-test` still found it for
up to 30 seconds. `~/ProtonDrive` behaves the same, so B88's routing did not cause it.
**Where:** `Core::rename` and `Core::delete` in `crates/pdfs-fuse/src/lib.rs`, and the `TTL` the
kernel gets for entries in the same file.
**Cause.** Both change Drive, forget the node and drop the parent's listing, but never tell the
kernel. The kernel cached the name's lookup for `TTL`, 30 seconds, and answers from that cache
until it expires. A rename or unlink through the mount has no such gap, because the kernel made
that change itself. `pdfs move` already sends `inval_entry` for its destination
(`settle_dest` in `crates/pdfs-fuse/src/relocate.rs`); its source was not checked.
The account run on flaky Wi-Fi on 2026-10-02 hit it in another place. "regression B111" renamed
a file with `pdfs rename` and straight back, and then failed with `FileNotFoundError`: the
kernel still pointed the first name at the inode the rename had forgotten.
**Fix.** `pdfs rm` forgets the node in every mount and sends each kernel session a delete notice
for the name (`Core::forget_everywhere`). `pdfs rename` and `pdfs move` invalidate the old and the
new name (`Core::relocate_everywhere`, see B117). The notices go out after the mount's lock is
released.
**Test:** the acceptance case "regression B114" removes one file with `pdfs rm` and renames another
with `pdfs rename`, then checks at once that neither old name resolves. It reproduced in My files
and in an on-demand folder before the fix.
## B113 — A new file moved before its upload drains lands as a conflict copy of itself
**Status:** Fixed (verified 2026-10-02).
**Found:** 2026-10-02, on the production account. aria2 downloaded two seasons into the on-demand
`~/Downloads`. The episodes were moved into `Videos/anime/…` while their uploads were still
queued. 45 of 48 landed as `(sync-conflict …)` copies next to a 0-byte original, each with
`the remote revision changed under the queued write (expected 0 bytes at mtime 1790955809, found 0
at 1790956593)`. The three that drained before the move uploaded normally.
**Where:** `serve_create` in `crates/pdfs-fuse/src/filesystem.rs`, `adopt_real_uid` in
`crates/pdfs-fuse/src/drain.rs`.
**Cause.** The handle a create opens is based on the empty file the server mints, but it was
given no revision id. B69 keys the drain's conflict check on the revision id, and without one it
falls back to `(mtime, size)`. The node's mtime is the link's modify time, which the server
advances on a move or rename. So a new file moved before its first write drained looked like
another device had changed it. A file created offline had the same gap in another place: a handle
still open when its queued create landed kept the placeholder's local-clock mtime and no revision
id.
**Fix.** `WriteHandle::created` takes the minted node's revision id along with its mtime. When a
queued create lands, `adopt_real_uid` rebases the handles still open on it onto the new node,
as `refresh_after_upload` already does after a sealed upload. Both go through
`WriteHandle::rebase_onto`.
**Verified live (2026-10-02)** on the installed daemon, in a scratch folder in `~/Downloads`. A
`pdfs move` advanced an empty file's server mtime from 1790962386 to 1790962408, the change the old
check took for another device's edit. With sync paused, three 16 MiB files were written and moved
into a subfolder while their uploads were queued. After the resume all three logged `pending upload
landed`, with no conflict warning or copy, and they read back at their exact sizes. Each upload was
put off once after the move (`the node is missing locally and was re-read`) and landed on the
retry about 8 seconds later. Meanwhile `pdfs sync queue` showed raw ids instead of the new paths.
Both are filed as B117. Not verified live: a file created offline that is still open when its
create lands. Only the unit test covers that half. The acceptance case "regression B113" replays
the online order through the mount, with a rename and a move, and passed in My files and in an
on-demand folder.
## B112 — Folders shared by the same person disappear from Shared with me ## B112 — Folders shared by the same person disappear from Shared with me
**Status:** Fixed (unverified). **Status:** Fixed (unverified).
@@ -45,15 +729,18 @@ InvalidRequirements (http 422): This file or folder was out of date, rename fail
`drain_rename_in_place` in `crates/pdfs-fuse/src/drain.rs`, and the control `rename` in `drain_rename_in_place` in `crates/pdfs-fuse/src/drain.rs`, and the control `rename` in
`crates/pdfs-fuse/src/lib.rs`. `crates/pdfs-fuse/src/lib.rs`.
**Cause.** The SDK's `rename_node` reads the node's details from the server and sends its current **Cause.** The SDK's `rename_node` sends the node's current name hash as `OriginalHash`. Right
name hash as `OriginalHash`. Right after a rename of the same node, that read can still return the after a rename of the same node, it still sent the old one. Drive then refuses the request as out
old hash. Drive then refuses the request as out of date. The daemon reported this at once as of date. The daemon reported this at once as `EIO`, although the same request succeeds a moment
`EIO`, although the same request succeeds a moment later. later. The old hash comes from the SDK's entity cache, not from the server (see B121).
**Fix.** All remote renames go through `Core::rename_remote`. It retries a rename that fails with **Fix.** All remote renames go through `Core::rename_remote`. It retries a rename that fails with
`InvalidRequirements` up to 20 times, half a second apart, the same window the move after a `InvalidRequirements` up to 20 times, half a second apart, the same window the move after a
rename already uses. Any other error still returns at once. rename already uses. Any other error still returns at once.
**Test:** the acceptance case "regression B111" renames a file there and back eight times through
the mount, and three times with `pdfs rename`, and checks its name and content after each round.
## B110 — A read that settles a provisional size deadlocks on its own page lock ## B110 — A read that settles a provisional size deadlocks on its own page lock
**Status:** Fixed (unverified). **Status:** Fixed (unverified).
@@ -194,7 +881,7 @@ the blob still on disk the copy would land.
**Fix.** Before keeping a conflict copy, the drain checks that its op is still queued. When the **Fix.** Before keeping a conflict copy, the drain checks that its op is still queued. When the
op is gone, the write was withdrawn by the delete, or superseded, while the worker held it, and op is gone, the write was withdrawn by the delete, or superseded, while the worker held it, and
it is dropped. it is dropped. The check goes by op id, which the next op queued can take over (B133).
--- ---
@@ -272,6 +959,9 @@ the trash with it. `pdfs rm` now drops the queued ops once the remote trash succ
acceptance janitor lists the daemon queue after cleanup and reports any op still naming one of its acceptance janitor lists the daemon queue after cleanup and reports any op still naming one of its
roots as a leftover. roots as a leftover.
Regression test: `trashing_a_folder_drops_the_queued_revisions_of_files_below_it`. Regression test: `trashing_a_folder_drops_the_queued_revisions_of_files_below_it`.
The acceptance case "regression B102" pauses sync, rewrites a file two folders down in each of
two folders, and removes one folder with `pdfs rm` and the other with `rm -r` through the mount.
Neither revision may stay queued.
--- ---
@@ -419,6 +1109,9 @@ so each new write supersedes the upload of the previous one.
**Fix.** The cancel is reported as `ErrorKind::Other`, which the SDK turns into an error. **Fix.** The cancel is reported as `ErrorKind::Other`, which the SDK turns into an error.
`drain_revision` already recognises a cancelled upload and returns without counting a failure. `drain_revision` already recognises a cancelled upload and returns without counting a failure.
Regression test: `a_cancelled_upload_fails_its_read_instead_of_asking_for_a_retry`. Regression test: `a_cancelled_upload_fails_its_read_instead_of_asking_for_a_retry`.
The acceptance case "regression B98" appends to a 64 MiB file while its upload is on the wire. The
superseded transfer has to end, and the file has to land at its full size. The case skips when the
upload finishes before it is seen.
--- ---
@@ -503,6 +1196,9 @@ forever and the bytes never go up.
rename. A `.tmp` write from a program that does rename is still handled — by the rename itself, rename. A `.tmp` write from a program that does rename is still handled — by the rename itself,
which supersedes the create. which supersedes the create.
**Test:** the acceptance case "regression B95" writes `index.tmp` and `preview.temp` and waits for
the queue, which fails at once on a parked op.
--- ---
## B94 — A parked create has no exit but the rename, so an abandoned one is stuck forever ## B94 — A parked create has no exit but the rename, so an abandoned one is stuck forever
@@ -529,6 +1225,10 @@ invariant 1 says staged bytes are only dropped once their op has landed, so the
bytes through under the transient name instead of destroying them. An open node is left parked bytes through under the transient name instead of destroying them. An open node is left parked
whatever its age: it is still being written. whatever its age: it is still being written.
**Test:** the acceptance case "regression B94" deletes a `*.part` file whose create is parked and
waits for the create to leave the queue. That is the delete exit. The hour-long expiry is not
driven.
--- ---
## B93 — A photo whose node would not resolve lost its name, and with it its tab and its group ## B93 — A photo whose node would not resolve lost its name, and with it its tab and its group
@@ -778,7 +1478,9 @@ in every version — but that was luck, and the next question of this shape may
## B88 — `pdfs pin`, `rename`, `move`, `delete` and sharing still cannot name a path under a secondary mount ## B88 — `pdfs pin`, `rename`, `move`, `delete` and sharing still cannot name a path under a secondary mount
**Status:** Open — B86's fix covers `ls` and `refresh` only. **Status:** Partly fixed. B86's fix covers `ls` and `refresh`. `move` takes paths in any location
through `Core::move_between`. `rm`, `rename` and `mkdir` were fixed and verified live 2026-10-02.
`pin`, `unpin`, opening a file, thumbnails, uploads, sharing and versions are still open.
**Found:** 2026-08-17, while fixing B86. **Found:** 2026-08-17, while fixing B86.
**Where:** `crates/pdfs-fuse/src/control.rs`, every handler still calling `rel_to_mount`. **Where:** `crates/pdfs-fuse/src/control.rs`, every handler still calling `rel_to_mount`.
@@ -804,6 +1506,30 @@ per-mount state assumptions as it goes; the `for_each_state`/uid-keyed helpers a
the ones that touch more than their own inode space. Test: a `pdfs pin` and a `pdfs rename` by the ones that touch more than their own inode space. Test: a `pdfs pin` and a `pdfs rename` by
absolute path inside a secondary on-demand mount, plus a path under no mount still erroring. absolute path inside a secondary on-demand mount, plus a path under no mount still erroring.
**Fix so far (`rm`, `rename`, `mkdir`):** the three handlers go through `control.rs::route_change`,
which is `route_to_mount` plus the mirror-folder hint and escape check that `ResolveConflict`
already had. The audit found nothing primary-only in `Core::delete`, `Core::rename` or
`Core::create_folder`. `Core::rooted_at` swaps in the fork's state, notifier, session flag and size
upgrades, the same per-mount fields `Core::fork_state` makes fresh. So `self.state()` in a routed
call already is the fork's inode space. My files and the on-demand folders do not overlap, so
`delete` forgetting the node in its own space alone is enough. `rename` already forgets it
everywhere (B74).
Routing raised a hazard the primary mount never had. An on-demand folder's root has a parent on
Drive, so `pdfs rm ~/Downloads` would have passed `source_parent_uid` and trashed the whole synced
folder from under its mount. `rm` and `rename` therefore go through `route_node`, which refuses a
path naming a mount's root. Test: `only_a_path_below_a_mount_root_names_a_node`. There is still no
handler-level test, because a `Core` can only be built against a live client.
**Verified live (2026-10-02)** in `~/Downloads`, an on-demand folder. `pdfs mkdir` created a
folder. `pdfs rename` of a file whose upload was still queued reached Drive under the new name, at
6 bytes and with no conflict. `pdfs rm` trashed a file and then the folder. `pdfs rm ~/Downloads`,
`pdfs rename ~/Downloads …` and `pdfs rm ~/ProtonDrive` were refused as the root of a synced
location, and a path outside every mount as not under the mountpoint. `mkdir` and `rm` in
`~/ProtonDrive` still work. Not verified: the refusal in a mirrored folder, because the account has
none. The acceptance case "regression B88" repeats this on every location of a run, and an
`--account` run checks the refusal in the mirror folders it registers.
--- ---
## B87 — A revision's block geometry is learned a read too late ## B87 — A revision's block geometry is learned a read too late
@@ -901,6 +1627,12 @@ Test: `covering_parts_route_a_path_to_the_mount_that_owns_it` covers the routing
mount wins, suffix relative to it, its own inode space and upgrade batches, a path under no mount mount wins, suffix relative to it, its own inode space and upgrade batches, a path under no mount
routes nowhere). The listing-invalidation half is the live repro, which is what is still pending. routes nowhere). The listing-invalidation half is the live repro, which is what is still pending.
**Test:** the managed matrix of an `--account` run replays the repro. Before it switches a mirror
folder to on-demand, it writes `pdfs-mode-switch.bin` there and forces a pass. After the switch it
checks that the mount lists the file with its bytes, as "regression B86" under "mode switches".
Only the third pairing of the full matrix switches a folder that an earlier pairing had
on-demand, which is the stale case. `--quick` does not get there.
--- ---
## B84 — A block read that comes back short truncates the file, silently ## B84 — A block read that comes back short truncates the file, silently
@@ -1315,6 +2047,10 @@ candidates came from `go/pkg/mod`. `SKIP_DIRS` gained `dist`/`build`, a new
`SKIP_PATH_SUFFIXES` covers `go/pkg/mod`, and both candidate functions gained a `SKIP_PATH_SUFFIXES` covers `go/pkg/mod`, and both candidate functions gained a
whole-query prefix lane ahead of the existing single-character one. whole-query prefix lane ahead of the existing single-character one.
**Test:** the acceptance case "regression B80" writes a file with a unique token in its name,
rewrites it, and waits for `pdfs search` to find it. In an on-demand sync folder, that file sits in
the device-folder subtree the index used to lose.
## B79 — Every write in an on-demand device-folder mount fails EACCES ## B79 — Every write in an on-demand device-folder mount fails EACCES
**Status:** Fixed (verified live 2026-07-31, running in a locally rebuilt 1.2.1 **Status:** Fixed (verified live 2026-07-31, running in a locally rebuilt 1.2.1
@@ -3531,6 +4267,10 @@ ordering matters because an invalidation sent before the rename reply is
overwritten by the kernel's response processing. Uids are immutable and never overwritten by the kernel's response processing. Uids are immutable and never
reused, making a session-long tombstone safe. reused, making a session-long tombstone safe.
**Test:** the acceptance case "regression B48" lists a folder of 12 files, moves one more in,
unlinks them all, and on a mount asks Drive for the listing with `pdfs refresh`. The listing has to
be empty, and `rmdir` has to succeed at once.
--- ---
## B49 — Failed Conflict Preservation Still Overwrites the Local File (CRIT-05) ## B49 — Failed Conflict Preservation Still Overwrites the Local File (CRIT-05)
+82
View File
@@ -11,6 +11,86 @@ scratch (user data in `staging/` and `recovery/` is never touched by this).
## [Unreleased] ## [Unreleased]
## [2.10.0] — 2026-10-03
Schema: **36**. Every node row gets a local id that never changes; existing rows keep their
row number as that id. Going back to 2.9 means rebuilding the cache. Requires `proton-drive-rs`
**0.7.2**.
### Added
- **Migration tests on real databases.** A database 2.8.2 wrote, with queued changes due,
parked, failing, waiting for access and in flight, migrates with every change intact.
`PDFS_MIGRATE_DB=<path> cargo test -p pdfs-core a_copy_of_a_real_database -- --ignored` runs
the same check on a read-only copy of any database.
### Changed
- **A file keeps its inode number across daemon restarts.** Inodes are now the node's local id
instead of a counter that started over on every run.
- **Every mount serves from one tree.** An on-demand sync folder used to keep its own copy of the
nodes it showed, which background work had to visit one by one and could miss (B74). It now
shares the tree with My files; only the root it starts at is its own.
### Fixed
- **A file renamed or moved while its first upload is on the wire keeps its new name and folder.**
It landed under the old name and the rename was lost (B136).
## [2.9.0] — 2026-10-03
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.2**.
### Added
- **Seeded simulation runs.** `cargo test -p pdfs-fuse --lib sim:: -- --ignored
--test-threads=1` mounts one or more real FUSE daemons against an in-memory Drive that lags,
drops, reorders and echoes like the real one. Each seed draws syscalls, link flaps and restarts,
checks every answer against a POSIX model, and at the end checks that nothing was lost and the
clients converged. CI runs them on pushes to main. They found B122 to B135.
- **More acceptance cases.** Regressions for B48, B80, B86, B88, B94, B95, B98, B102, B111, B113
and B114 to B121, and a check that `pdfs` output into a closed pipe stays quiet. The console
report has one line per case and a summary table, and `--journal-check` also fails on a
conflict copy, a FUSE job held for long, or an authority re-read.
### Fixed
- **A new file moved or renamed before its first upload no longer lands as a conflict copy**
next to its own empty original (B113).
- **`pdfs rm`, `pdfs rename` and `pdfs mkdir` take paths in on-demand folders.** They answered
"is not under the mountpoint" for a path `pdfs move` took. `rm` and `rename` refuse a path that
names a mount's root (B88).
- **`pdfs ls | head` no longer panics** with "Broken pipe". A closed stdout exits 0; any other
write error exits 1.
- **A file removed, renamed or moved with `pdfs` stops resolving under its old name at once**,
instead of for up to 30 seconds (B114).
- **`pdfs ls` and the app's file browser show a file's real size**, not its encrypted one
(B115).
- **Trashing a folder also drops the creates queued in its subfolders.** One could stay queued
with its bytes, or land in the root (B116).
- **An upload queued for a file `pdfs move` moved keeps its path** in `pdfs sync queue` and is
no longer put off once after the resume (B117).
- **Creating a file or folder through the mount no longer fails with `ENOENT`** when Drive is
slow to list what it made. The mount asks again, and queues the create if it still cannot see
it (B118).
- **A file just written can be read without the network.** Its uploaded bytes stay cached, and
`pdfs status` counts the upload as pending until they are (B119).
- **A file queued inside a folder that just landed is no longer put off** for five seconds
(B120).
- **A move right after a rename lands.** It failed with `Input/output error` through the mount
and with "out of date" from `pdfs move`, and a rename across folders took up to 10 seconds
(B111, B121).
- **A rename across folders no longer fails with `Input/output error`** when the source folder
holds the new name (B122).
- **A second write open on a file just written no longer fails** with `Input/output error`
(B123).
- **After a replacing rename fails, the file it was replacing stays in the mount** (B124).
- **A write to a file whose create could not be read back no longer lands as a conflict copy**
(B131).
- **A file just moved is found by its new path.** A lookup right after the rename could miss it
(B135).
### Notes
- The simulation runs found bugs that are still open: B125 to B130 and B132 to B134 in
`docs/BUGS.md`. The worst are B127 and B128 (a file created offline can land twice, or be lost
when created over one deleted offline), B129 (a file deleted while its create is on the wire
stays on Drive) and B130 (a read that a remote change invalidates can hang the daemon).
## [2.8.2] — 2026-10-01 ## [2.8.2] — 2026-10-01
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.2**. Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.2**.
@@ -1364,6 +1444,8 @@ First stable release: FUSE files-on-demand mount, sync daemon under `proton-driv
- The outstanding FUSE defects tracked in `docs/BUGS.md`, plus a truncate defect, validated - The outstanding FUSE defects tracked in `docs/BUGS.md`, plus a truncate defect, validated
by a new POSIX compliance suite for the filesystem. by a new POSIX compliance suite for the filesystem.
[2.10.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.10.0
[2.9.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.9.0
[2.8.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.2 [2.8.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.2
[2.8.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.1 [2.8.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.1
[2.8.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.0 [2.8.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.0
+216 -10
View File
@@ -37,7 +37,10 @@ An account run does this, in order:
1. Runs the account-free contract as the reference. 1. Runs the account-free contract as the reference.
2. Runs the contract in a fresh `pdfs-acceptance-<id>` folder in My files and 2. Runs the contract in a fresh `pdfs-acceptance-<id>` folder in My files and
diffs it against the reference. diffs it against the reference. After the last case it reads back a file
the run wrote and records its digest. A read that fails there, as one did
with the link down (B119), is a failed result named "a file of this run
reads back at the end", and the cleanup still runs.
3. Registers two new sync folders under `~/.cache/pdfs-acceptance/run-<id>/` 3. Registers two new sync folders under `~/.cache/pdfs-acceptance/run-<id>/`
and runs the contract and the move cases in every on-demand/mirror pairing. and runs the contract and the move cases in every on-demand/mirror pairing.
4. Removes everything it created, locally and remotely, and reports a 4. Removes everything it created, locally and remotely, and reports a
@@ -50,6 +53,17 @@ It only counts a folder as gone once a fresh trash listing no longer shows it.
It then reads `pdfs sync queue` and counts any queued op that still names one It then reads `pdfs sync queue` and counts any queued op that still names one
of its folders as left behind too, since such an op keeps its staged bytes and of its folders as left behind too, since such an op keeps its staged bytes and
retries forever (B102). retries forever (B102).
Ops that were queued before a target starts are not the run's. The run prints
one `NOTE` line naming them, and the queue waits ignore them. They are told
apart by id and queue time together, since SQLite can give a new op the id of
one that has drained.
A write is queued when the kernel releases the file, and `close(2)` returns
without waiting for that. A case that reads `pdfs sync queue` right after
writing therefore waits for the ops it expects (`Daemon.queued`) instead of
reading once. "regression B102" saw one revision of two before it did.
Cleanup also runs on a failure, Ctrl-C, `SIGTERM` or `SIGHUP`. While it runs, Cleanup also runs on a failure, Ctrl-C, `SIGTERM` or `SIGHUP`. While it runs,
two more interrupts are ignored; a third abandons it. two more interrupts are ignored; a third abandons it.
@@ -74,6 +88,43 @@ removes its directory on success, failure, or interruption:
scripts/fuse-acceptance.sh --live /mnt/on-demand-testmount scripts/fuse-acceptance.sh --live /mnt/on-demand-testmount
``` ```
### Reading the output
A run opens with what it is about to test: the mode, the `pdfs` binary and its
version, Python and the kernel, the per-case limit, the options and the start
time. Each target then names its root, the storage behind it, the daemon it
talks to and how many cases it runs, followed by one line per case:
```
==> live FUSE /mnt/testmount
-> root /mnt/testmount/pdfs-acceptance-…
-> storage FUSE mount /mnt/testmount (fuse.protondrive)
-> daemon pdfs 2.8.3, user@proton.me, online, 0 queued ops
-> cases 45 of 45
[ 1/45] creation and open flags .................................. ok 0.41s
…
[21/45] throughput floors ........................................ ok 48.20s
write 31 MiB/s, read 54 MiB/s, 4 KiB writes 12.0 MiB/s, metadata 9 ops/s
…
[43/45] regression B115: pdfs ls reports a file's real size ...... ok 6.12s
-> reference matches the ordinary filesystem in 412 compared observations
-> result 44 passed, 1 skipped in 11m02s
```
A case ends as `ok`, `FAIL`, `TIMEOUT`, `skip` when it does not apply to the
target (no daemon, no share at that role), or `known` when an `open B<n>` case
reproduced a bug that is still open (see below). The dimmed lines under a case
are its details: the reason for a skip, measured rates, a hash. A failed
`regression B<n>` case also points to its entry in `docs/BUGS.md`. When a case
prints something itself, such as a service restart, the case line is printed
again once the case ends.
The run ends with a table of every target's counts and time, the report paths,
the slowest cases, the cases over `--budget`, the known issues that still
reproduce, any errors the daemon logged, and each failure with its target. The
last line is `PASS:` or `FAIL:` and the totals. Colour is used only on a
terminal and never with `NO_COLOR` set.
### Edge cases and performance ### Edge cases and performance
Besides the syscall contract, the suite covers: Besides the syscall contract, the suite covers:
@@ -105,19 +156,20 @@ Besides the syscall contract, the suite covers:
The local reference run is not only a self-test of the runner. Every case The local reference run is not only a self-test of the runner. Every case
records the facts it established, and a live run is compared against those records the facts it established, and a live run is compared against those
recordings, so a case fails when the mount *differs* from an ordinary recordings, so a case fails when the mount *differs* from an ordinary
filesystem — not only when it raises. The report names the exact key: filesystem — not only when it raises. The target's `reference` line names the
exact key:
``` ```
[DIVERGENCE] live FUSE /mnt/testmount does not match the reference filesystem: -> reference 1 observation of 412 differs from the ordinary filesystem:
! creation and open flags.truncated.size: live=6 reference=0 ! creation and open flags.truncated.size: live FUSE /mnt/testmount=6 reference=0
``` ```
Facts are recorded in one of two ways, chosen in the test body: Facts are recorded in one of two ways, chosen in the test body:
- **compared** — a claim about filesystem semantics. The mount must agree. - **compared** — a claim about filesystem semantics. The mount must agree.
- **noted** — context that legitimately differs: inode numbers, block counts, - **noted** — context that legitimately differs: inode numbers, block counts,
reported mode bits, and which optional syscalls exist at all. Reported under reported mode bits, and which optional syscalls exist at all. Listed under
`[capabilities]`, never a failure. the target's `accepted` line, never a failure.
Because the choice is made at each call site, the set of accepted divergences Because the choice is made at each call site, the set of accepted divergences
is visible in the test that accepts it. There is no separate allowlist to drift is visible in the test that accepts it. There is no separate allowlist to drift
@@ -151,6 +203,91 @@ brought back the provisional sizes the bug reported.
different times before writing, so their empty creates land before, during and after the different times before writing, so their empty creates land before, during and after the
bytes arrive, then checks every file after the queue drains. bytes arrive, then checks every file after the queue drains.
**B113** (moving a file before its upload makes no conflict) pauses sync with
`pdfs sync pause --for 10m` and writes three files. While their uploads are held,
it renames one, moves one into a folder, and renames and moves the third. It then
resumes sync, waits for the queue, and checks the bytes and names and that no
conflict copy appeared. It skips when sync was already paused. If the run dies
before the resume, the pause ends by itself after ten minutes.
**B88** (`pdfs mkdir`, `rename` and `rm` by local path) creates, renames and
removes a folder and a file by their paths in the mount. Then it checks the
refusals: `pdfs rename` of the mount's root to its own name, and `pdfs rm` of a
file outside every mount. `pdfs rm` of the root itself is tried only on a sync
folder the run created. In a mirrored sync folder all three commands must be
refused instead, and nothing may change.
**pdfs output into a closed pipe** runs `pdfs ls`, `pdfs --json ls` and
`pdfs status` with stdout on a pipe whose reader is already gone. Each must exit
0 with nothing on stderr, as `pdfs ls | head -1` needs.
**B102** (removing a folder drops the uploads queued inside it) uploads a file
into each of two nested folders, pauses sync and rewrites both. While the
revisions are held, it removes one folder with `pdfs rm` and the other through
the mount. Neither revision may stay queued, and neither folder may come back
after the resume. It skips when sync was already paused.
**B94** (a deleted `.part` file leaves nothing queued) writes a `.part` file,
checks that its create is parked, and unlinks it. The parked op must leave the
queue within 30 seconds. The sweep that lets an hour-old park through is not
driven.
**B95** (a `.tmp` file uploads at once) writes `index.tmp` and `preview.temp`
and waits for the queue, which fails at once on a parked op. `pdfs ls` must
list both, and their bytes must read back.
**B111** (renaming a file straight back works) renames a file there and back
eight times through the mount. On a mount it also does so three times with
`pdfs rename`, after the queue is empty, so the renames race nothing but Drive.
It runs against the reference too.
**B121** (a file renamed and moved at once lands) renames a file, moves it into
a subfolder and renames it back across folders through the mount, with no pause
between the steps. On a mount it then runs `pdfs rename` and `pdfs move` back to
back. The file must end up where it was sent, with its bytes, and the case
reports how long each half took. It runs against the reference too.
**B98** (an upload superseded by a write ends) writes 64 MiB and waits for its
upload to show in `pdfs transfers`, then appends to the file. The queue must
drain, the old upload must leave the transfer list, and the file must hold
every byte with no conflict copy. It skips when the upload is never seen on
the wire, and it gets twice the per-case timeout.
**B80** (`pdfs search` finds a new file after a rewrite) writes a file whose
name carries a unique token, waits for the queue, rewrites it and waits again.
`pdfs search` must find it within 30 seconds.
**B48** (an emptied folder can be removed at once) writes twelve files into a
folder, lists it, and moves one more in from elsewhere. It unlinks every
child, runs `pdfs refresh` on a mount, and checks that nothing is listed and
that `rmdir` works. It runs against the reference too.
**B114** (a name `pdfs rm` or `rename` took away stops resolving) looks up two
files, removes one with `pdfs rm` and renames the other with `pdfs rename`. At
once, neither old name may be listed or resolve.
**B115** (`pdfs ls` reports a file's real size) writes files of 6 bytes, 4097
bytes and one block plus 7, and compares the sizes `pdfs ls` lists with the
written ones, after the upload and again after `pdfs refresh`.
**B116** (trashing a folder drops the creates queued in its subfolders) makes
`b116/sub/deeper` and waits for Drive. With sync paused, it writes a `.part`
file there and renames it to its final name, so its create is due but held.
It then runs `pdfs rm b116`, and the create must be gone from the queue. It
restores the folder with `pdfs restore` before it resumes sync. If the file
ends up in the mount's root anyway, the case fails and moves the file to the
trash.
**B117** (a moved file's queued upload keeps its path) rewrites an uploaded
file with sync paused and moves it with `pdfs move`. `pdfs sync queue` must
list the upload under its new path, and the journal must not show the upload
put off with "pending operation deferred". The bytes must arrive at the new
path with no conflict copy. Without `journalctl`, only the queue is checked.
**B119** (an uploaded file stays cached) writes a small file and one of a block
plus 119 bytes, waits for the queue, and expects `pdfs ls` to list both as
cached, so reading them does not need the network.
B74 was found by exactly this mechanism: the B70 case failed, and narrowing it B74 was found by exactly this mechanism: the B70 case failed, and narrowing it
produced a smaller reproduction that got its own case. The defect is fixed and produced a smaller reproduction that got its own case. The defect is fixed and
live-validated (2026-07-26); the case now guards against its return. live-validated (2026-07-26); the case now guards against its return.
@@ -186,13 +323,61 @@ nothing: the local attribute and content caches will happily return the bytes
that were just written even when nothing reached Drive, which is precisely how that were just written even when nothing reached Drive, which is precisely how
B74 stayed invisible. B74 stayed invisible.
### Bugs that are still open
Cases named `open B<n>` check a bug that `docs/BUGS.md` still lists as open.
While the bug reproduces, the case ends as `known`: it is listed at the end of
the run, but does not fail it. So a known bug does not hide new failures, and
the run notices when a fix lands, because the case then passes. Rename it to
`regression B<n>` then, and fail it with `check` instead of `KnownIssue`. No
bug is open at the moment. B114 to B117 started as such cases.
### What a run verifies
A full account run with `--journal-check` checks these `docs/BUGS.md` entries,
many of them fixed but not yet verified:
- B48, B80, B94, B95, B98, B100, B101, B102, B111, B114, B115, B116, B117, B119
and B121, each with its own `regression B<n>` case.
- B118, with "application workloads" and "block boundaries", which failed on
it. It needs Drive to be slow to list a new node, so a clean run is weak
evidence.
- B120, with `--journal-check`, but only in a run that goes offline while it
writes. A mkdir made online goes to Drive at once, so only then does a
folder land from the queue.
- B86, with the mode-switch check in the full matrix.
- B104, B106, B108 and B109, with the cases that found them: "rename
patterns", "application workloads" and the account run's own setup.
B105, B107 and B110 are races. The suite hit each of them once, so a clean run
is weak evidence for them. `--journal-check` reports B110's stall warning, and
every case that writes checks for B105's conflict copies.
The suite does not drive:
- the GUI and the photo gallery (B81, B92, B93, B97);
- states it cannot set up: a `parent_uid` cycle (B82), staged bytes that name
no node (B96), a stuck trash refresh (B103), conflict copies left from an
earlier start (B99), a foreign FUSE mount in the local-file index (B90),
revisions without 4 MiB blocks (B85), and several shares from one owner
(B112);
- B94's hour-long expiry of a park;
- load and failure injection (B22, B27, B30, B37, B42).
Check those by hand, or with the unit tests their entries name.
### Timeouts, reports, and hung mounts ### Timeouts, reports, and hung mounts
A wedged FUSE operation cannot be interrupted from Python, so each case runs A wedged FUSE operation cannot be interrupted from Python, so each case runs
under a soft alarm (`--timeout`, default 180s) that reports the case and lets under a soft alarm (`--timeout`, default 180s) that reports the case and lets
cleanup run, backed by a hard stop that dumps every thread's stack and exits. cleanup run, backed by a hard stop that dumps every thread's stack and exits.
That backstop is the only way to learn *where* a mount hung. A timeout ends the That backstop is the only way to learn *where* a mount hung. After a timeout the
run: later cases against a wedged mount produce noise, not information. run checks whether the mount still answers a `stat` and a listing within 15
seconds. If it does, the target goes on with its next case, so one slow stretch
of the link does not cost every later case. If it does not, the target's later
cases are not run: against a wedged mount they produce noise, not information.
Cases that make many round trips by design ("application workloads", "unusual
but legal names", "a wide directory" and "regression B98") get twice the limit.
```console ```console
scripts/fuse-acceptance.sh --live /mnt/testmount \ scripts/fuse-acceptance.sh --live /mnt/testmount \
@@ -200,7 +385,8 @@ scripts/fuse-acceptance.sh --live /mnt/testmount \
``` ```
`--report-json` also writes every recorded observation, which is what to attach `--report-json` also writes every recorded observation, which is what to attach
to a bug report. `--budget SECONDS` flags cases that pass but got slower — to a bug report. A known issue has the status `known` there; JUnit has no such
status, so `--report-junit` reports it as skipped. `--budget SECONDS` flags cases that pass but got slower —
B5 was a 186 ms-per-call regression that a pass/fail suite could not see. B5 was a 186 ms-per-call regression that a pass/fail suite could not see.
`--list` prints every case and which targets it runs against, and `--fail-fast` `--list` prints every case and which targets it runs against, and `--fail-fast`
stops at the first failure. stops at the first failure.
@@ -208,6 +394,9 @@ stops at the first failure.
`--journal-check` fails the run if `proton-drive.service` logged any error while `--journal-check` fails the run if `proton-drive.service` logged any error while
it ran. A suite can pass every assertion while the daemon logs failures behind it ran. A suite can pass every assertion while the daemon logs failures behind
it; that has happened, and it was caught only by reading the journal by hand. it; that has happened, and it was caught only by reading the journal by hand.
Two warnings count as errors too: "queued write conflicts; keeping a conflict
copy" (how B113 showed) and "a fuse worker has held the same job for a long
time" (how B110 showed).
With `--live`, abandoned `pdfs-acceptance-*` roots from a crashed run are With `--live`, abandoned `pdfs-acceptance-*` roots from a crashed run are
removed at startup, but only if they match the exact generated name and are removed at startup, but only if they match the exact generated name and are
@@ -253,6 +442,21 @@ The My files case writes a test folder into your real My files and removes it
afterwards. It skips when the daemon reports no My files mount, or when the afterwards. It skips when the daemon reports no My files mount, or when the
locations are on different volumes. locations are on different volumes.
Before a mirror folder switches to on-demand, the runner writes a new file into
it and syncs. Once the folder is mounted, the file must be listed with its
bytes (B86). Each switch reports a `regression B86` result under a separate
`mode switches` target. The bug only shows in a folder that was mounted in an
earlier pairing, because that mount's listing is stale. That happens in the
third pairing, so `--quick` runs the check but does not replay the bug.
In every pairing, each folder must still hold the file the runner wrote into it
at setup, byte for byte. These checks report under `mode switches` as well. A
switch to on-demand frees the local copy, so these reads go to Drive. If the
link drops, the daemon fails such a read with EIO once its two-minute read
budget runs out. That is correct, so the runner waits until the daemon reports
online again, tries once more and notes the retry under the result. Only the
second try decides the result.
### Durability drill ### Durability drill
`--durability` restarts `proton-drive.service` in the middle of the suite and `--durability` restarts `proton-drive.service` in the middle of the suite and
@@ -264,7 +468,9 @@ interrupts a running daemon; `PDFS_ACCEPTANCE_UNIT` selects a different unit.
### Selecting cases ### Selecting cases
During development, `PDFS_ACCEPTANCE_ONLY` runs tests whose descriptive name During development, `PDFS_ACCEPTANCE_ONLY` runs tests whose descriptive name
contains the supplied text (for example `PDFS_ACCEPTANCE_ONLY=namespace`). A contains the supplied text, ignoring case (for example
`PDFS_ACCEPTANCE_ONLY=namespace`). Separate several texts with commas to run
every case that matches any of them (`PDFS_ACCEPTANCE_ONLY=B116,B117`). A
release acceptance run must leave it unset so the complete contract executes. release acceptance run must leave it unset so the complete contract executes.
A filter naming a live-only case (every `regression B<n>` is one) selects nothing A filter naming a live-only case (every `regression B<n>` is one) selects nothing
+1 -1
View File
@@ -9,7 +9,7 @@
# Build + install: cd packaging && makepkg -fi # Build + install: cd packaging && makepkg -fi
pkgname=proton-drive-linux pkgname=proton-drive-linux
pkgver=2.8.2 pkgver=2.10.0
pkgrel=1 pkgrel=1
pkgdesc="Unofficial Proton Drive client for Linux: files-on-demand FUSE mount, CLI, GTK4 app + tray" pkgdesc="Unofficial Proton Drive client for Linux: files-on-demand FUSE mount, CLI, GTK4 app + tray"
arch=('x86_64') arch=('x86_64')
@@ -1,7 +1,7 @@
pkgbase = proton-drive-for-linux-bin pkgbase = proton-drive-for-linux-bin
pkgdesc = Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (binaries) pkgdesc = Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (binaries)
pkgver = 2.8.1 pkgver = 2.8.2
pkgrel = 1 pkgrel = 2
url = https://proton-drive.narl.io url = https://proton-drive.narl.io
arch = x86_64 arch = x86_64
license = MIT license = MIT
@@ -21,21 +21,21 @@ pkgbase = proton-drive-for-linux-bin
optdepends = xdg-utils: "Open folder" action in the tray menu optdepends = xdg-utils: "Open folder" action in the tray menu
optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf) optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf)
optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding
provides = proton-drive-for-linux=2.8.1 provides = proton-drive-for-linux=2.8.2
provides = proton-drive-linux=2.8.1 provides = proton-drive-linux=2.8.2
provides = pdfs provides = pdfs
conflicts = proton-drive-linux conflicts = proton-drive-linux
conflicts = proton-drive-for-linux conflicts = proton-drive-for-linux
conflicts = proton-drive-for-linux-git conflicts = proton-drive-for-linux-git
options = !strip options = !strip
options = !debug options = !debug
source = proton-drive-linux-2.8.1-x86_64.tar.gz::https://github.com/narrrl/proton-drive-linux/releases/download/v2.8.1/proton-drive-linux-2.8.1-x86_64.tar.gz source = proton-drive-linux-2.8.2-x86_64.tar.gz::https://github.com/narrrl/proton-drive-linux/releases/download/v2.8.2/proton-drive-linux-2.8.2-x86_64.tar.gz
source = io.narl.proton-drive-linux.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.1/packaging/io.narl.proton-drive-linux.desktop source = io.narl.proton-drive-linux-2.8.2.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/io.narl.proton-drive-linux.desktop
source = io.narl.proton-drive-linux-tray.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.1/packaging/io.narl.proton-drive-linux-tray.desktop source = io.narl.proton-drive-linux-tray-2.8.2.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/io.narl.proton-drive-linux-tray.desktop
source = io.narl.proton-drive-linux.svg::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.1/packaging/io.narl.proton-drive-linux.svg source = io.narl.proton-drive-linux-2.8.2.svg::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/io.narl.proton-drive-linux.svg
source = proton-drive.service::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.1/packaging/proton-drive.service source = proton-drive-2.8.2.service::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/proton-drive.service
source = LICENSE-2.8.1::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.1/LICENSE source = LICENSE-2.8.2::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/LICENSE
sha256sums = 68cac3de6a4156994d6990a88d6715c05e49da5021a2e18458498364d1ed299a sha256sums = c0af4ff57dc6f587b1b8431f3180b00d45211389f3dc675569af45f64e89cb36
sha256sums = d190d6771fff0b975271fb62fd3fb0fa8bbf17d7278c10853c18557855b0c123 sha256sums = d190d6771fff0b975271fb62fd3fb0fa8bbf17d7278c10853c18557855b0c123
sha256sums = 1d95e250370220e3bb6aead6ce34d1d6cc759f42c4803b75f1667e78ff47c6c0 sha256sums = 1d95e250370220e3bb6aead6ce34d1d6cc759f42c4803b75f1667e78ff47c6c0
sha256sums = e598a93c2715c3a9fdc7fb937902083d4a1135e8fbdfe6fe7e37f0383ebbd828 sha256sums = e598a93c2715c3a9fdc7fb937902083d4a1135e8fbdfe6fe7e37f0383ebbd828
@@ -1,8 +1,8 @@
# Maintainer: Nils Pukropp <contact@narl.io> # Maintainer: Nils Pukropp <contact@narl.io>
pkgname=proton-drive-for-linux-bin pkgname=proton-drive-for-linux-bin
pkgver=2.8.1 pkgver=2.8.2
pkgrel=1 pkgrel=2
pkgdesc="Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (binaries)" pkgdesc="Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (binaries)"
arch=('x86_64') arch=('x86_64')
url="https://proton-drive.narl.io" url="https://proton-drive.narl.io"
@@ -25,14 +25,16 @@ options=('!strip' '!debug')
_tag="v$pkgver" _tag="v$pkgver"
_raw="https://raw.githubusercontent.com/narrrl/proton-drive-linux/$_tag" _raw="https://raw.githubusercontent.com/narrrl/proton-drive-linux/$_tag"
# The release tarball holds the four binaries only, so the desktop integration # The release tarball holds the four binaries only, so the desktop integration
# files come straight from the tagged tree. # files come straight from the tagged tree. Every one carries $pkgver in its
# name: makepkg reuses a cached file of the same name, so an unversioned name
# keeps the old copy after upstream changes it and fails the checksum.
source=("proton-drive-linux-$pkgver-x86_64.tar.gz::$_repo/releases/download/$_tag/proton-drive-linux-$pkgver-x86_64.tar.gz" source=("proton-drive-linux-$pkgver-x86_64.tar.gz::$_repo/releases/download/$_tag/proton-drive-linux-$pkgver-x86_64.tar.gz"
"io.narl.proton-drive-linux.desktop::$_raw/packaging/io.narl.proton-drive-linux.desktop" "io.narl.proton-drive-linux-$pkgver.desktop::$_raw/packaging/io.narl.proton-drive-linux.desktop"
"io.narl.proton-drive-linux-tray.desktop::$_raw/packaging/io.narl.proton-drive-linux-tray.desktop" "io.narl.proton-drive-linux-tray-$pkgver.desktop::$_raw/packaging/io.narl.proton-drive-linux-tray.desktop"
"io.narl.proton-drive-linux.svg::$_raw/packaging/io.narl.proton-drive-linux.svg" "io.narl.proton-drive-linux-$pkgver.svg::$_raw/packaging/io.narl.proton-drive-linux.svg"
"proton-drive.service::$_raw/packaging/proton-drive.service" "proton-drive-$pkgver.service::$_raw/packaging/proton-drive.service"
"LICENSE-$pkgver::$_raw/LICENSE") "LICENSE-$pkgver::$_raw/LICENSE")
sha256sums=('68cac3de6a4156994d6990a88d6715c05e49da5021a2e18458498364d1ed299a' sha256sums=('c0af4ff57dc6f587b1b8431f3180b00d45211389f3dc675569af45f64e89cb36'
'd190d6771fff0b975271fb62fd3fb0fa8bbf17d7278c10853c18557855b0c123' 'd190d6771fff0b975271fb62fd3fb0fa8bbf17d7278c10853c18557855b0c123'
'1d95e250370220e3bb6aead6ce34d1d6cc759f42c4803b75f1667e78ff47c6c0' '1d95e250370220e3bb6aead6ce34d1d6cc759f42c4803b75f1667e78ff47c6c0'
'e598a93c2715c3a9fdc7fb937902083d4a1135e8fbdfe6fe7e37f0383ebbd828' 'e598a93c2715c3a9fdc7fb937902083d4a1135e8fbdfe6fe7e37f0383ebbd828'
@@ -45,15 +47,15 @@ package() {
install -Dm755 "$srcdir/pdfs-app" "$pkgdir/usr/bin/pdfs-app" install -Dm755 "$srcdir/pdfs-app" "$pkgdir/usr/bin/pdfs-app"
install -Dm755 "$srcdir/pdfs-prompt" "$pkgdir/usr/bin/pdfs-prompt" install -Dm755 "$srcdir/pdfs-prompt" "$pkgdir/usr/bin/pdfs-prompt"
install -Dm644 "$srcdir/io.narl.proton-drive-linux.desktop" \ install -Dm644 "$srcdir/io.narl.proton-drive-linux-$pkgver.desktop" \
"$pkgdir/usr/share/applications/io.narl.proton-drive-linux.desktop" "$pkgdir/usr/share/applications/io.narl.proton-drive-linux.desktop"
install -Dm644 "$srcdir/io.narl.proton-drive-linux-tray.desktop" \ install -Dm644 "$srcdir/io.narl.proton-drive-linux-tray-$pkgver.desktop" \
"$pkgdir/etc/xdg/autostart/io.narl.proton-drive-linux-tray.desktop" "$pkgdir/etc/xdg/autostart/io.narl.proton-drive-linux-tray.desktop"
install -Dm644 "$srcdir/io.narl.proton-drive-linux.svg" \ install -Dm644 "$srcdir/io.narl.proton-drive-linux-$pkgver.svg" \
"$pkgdir/usr/share/icons/hicolor/scalable/apps/io.narl.proton-drive-linux.svg" "$pkgdir/usr/share/icons/hicolor/scalable/apps/io.narl.proton-drive-linux.svg"
# systemd user unit for the auto-mount daemon; enabled per user, not by pacman. # systemd user unit for the auto-mount daemon; enabled per user, not by pacman.
install -Dm644 "$srcdir/proton-drive.service" \ install -Dm644 "$srcdir/proton-drive-$pkgver.service" \
"$pkgdir/usr/lib/systemd/user/proton-drive.service" "$pkgdir/usr/lib/systemd/user/proton-drive.service"
# Translations ship precompiled in the release tarball. # Translations ship precompiled in the release tarball.
@@ -1,6 +1,6 @@
pkgbase = proton-drive-for-linux-git pkgbase = proton-drive-for-linux-git
pkgdesc = Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (git) pkgdesc = Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (git)
pkgver = 2.8.1.r0.gba1fdd6 pkgver = 2.8.2.r0.g50483bf
pkgrel = 1 pkgrel = 1
url = https://proton-drive.narl.io url = https://proton-drive.narl.io
arch = x86_64 arch = x86_64
@@ -24,8 +24,8 @@ pkgbase = proton-drive-for-linux-git
optdepends = xdg-utils: "Open folder" action in the tray menu optdepends = xdg-utils: "Open folder" action in the tray menu
optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf) optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf)
optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding
provides = proton-drive-for-linux=2.8.1.r0.gba1fdd6 provides = proton-drive-for-linux=2.8.2.r0.g50483bf
provides = proton-drive-linux=2.8.1.r0.gba1fdd6 provides = proton-drive-linux=2.8.2.r0.g50483bf
provides = pdfs provides = pdfs
conflicts = proton-drive-linux conflicts = proton-drive-linux
conflicts = proton-drive-for-linux conflicts = proton-drive-for-linux
@@ -1,7 +1,7 @@
# Maintainer: Nils Pukropp <contact@narl.io> # Maintainer: Nils Pukropp <contact@narl.io>
pkgname=proton-drive-for-linux-git pkgname=proton-drive-for-linux-git
pkgver=2.8.1.r0.gba1fdd6 pkgver=2.8.2.r0.g50483bf
pkgrel=1 pkgrel=1
pkgdesc="Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (git)" pkgdesc="Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (git)"
arch=('x86_64') arch=('x86_64')
@@ -1,6 +1,6 @@
pkgbase = proton-drive-for-linux pkgbase = proton-drive-for-linux
pkgdesc = Unofficial Proton Drive client: files-on-demand FUSE mount, CLI, GTK4 app and tray pkgdesc = Unofficial Proton Drive client: files-on-demand FUSE mount, CLI, GTK4 app and tray
pkgver = 2.8.1 pkgver = 2.8.2
pkgrel = 1 pkgrel = 1
url = https://proton-drive.narl.io url = https://proton-drive.narl.io
arch = x86_64 arch = x86_64
@@ -23,13 +23,13 @@ pkgbase = proton-drive-for-linux
optdepends = xdg-utils: "Open folder" action in the tray menu optdepends = xdg-utils: "Open folder" action in the tray menu
optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf) optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf)
optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding
provides = proton-drive-linux=2.8.1 provides = proton-drive-linux=2.8.2
provides = pdfs provides = pdfs
conflicts = proton-drive-linux conflicts = proton-drive-linux
conflicts = proton-drive-for-linux-bin conflicts = proton-drive-for-linux-bin
conflicts = proton-drive-for-linux-git conflicts = proton-drive-for-linux-git
options = !lto options = !lto
source = proton-drive-for-linux-2.8.1.tar.gz::https://github.com/narrrl/proton-drive-linux/archive/refs/tags/v2.8.1.tar.gz source = proton-drive-for-linux-2.8.2.tar.gz::https://github.com/narrrl/proton-drive-linux/archive/refs/tags/v2.8.2.tar.gz
sha256sums = ee116a75201ae836921865fe9d2179c30f9e32084c983020120a73720923c1d7 sha256sums = 7f2df1d2748c3d8daaa4c05d242fcd293cb1983572e44bb61a35188dbe5f342c
pkgname = proton-drive-for-linux pkgname = proton-drive-for-linux
@@ -1,7 +1,7 @@
# Maintainer: Nils Pukropp <contact@narl.io> # Maintainer: Nils Pukropp <contact@narl.io>
pkgname=proton-drive-for-linux pkgname=proton-drive-for-linux
pkgver=2.8.1 pkgver=2.8.2
pkgrel=1 pkgrel=1
pkgdesc="Unofficial Proton Drive client: files-on-demand FUSE mount, CLI, GTK4 app and tray" pkgdesc="Unofficial Proton Drive client: files-on-demand FUSE mount, CLI, GTK4 app and tray"
arch=('x86_64') arch=('x86_64')
@@ -24,7 +24,7 @@ conflicts=('proton-drive-linux' 'proton-drive-for-linux-bin' 'proton-drive-for-l
# LTO has broken GTK/Rust links for this workspace; the release profile keeps it off. # LTO has broken GTK/Rust links for this workspace; the release profile keeps it off.
options=('!lto') options=('!lto')
source=("$pkgname-$pkgver.tar.gz::$_repo/archive/refs/tags/v$pkgver.tar.gz") source=("$pkgname-$pkgver.tar.gz::$_repo/archive/refs/tags/v$pkgver.tar.gz")
sha256sums=('ee116a75201ae836921865fe9d2179c30f9e32084c983020120a73720923c1d7') sha256sums=('7f2df1d2748c3d8daaa4c05d242fcd293cb1983572e44bb61a35188dbe5f342c')
_srcname="proton-drive-linux-$pkgver" _srcname="proton-drive-linux-$pkgver"
+1471 -210
View File
File diff suppressed because it is too large. Load diff