Compare commits

...
60 Commits
Author SHA1 Message Date
nvrl f6456829fb chore(release): 2.10.0 2026-10-03 21:57:23 +02:00
nvrl 327f538fed test(core): migrate a database 2.8.2 wrote, with ops in every state
A dump of a database the 2.8.2 build wrote, with queued ops due, parked,
failing, deferred for access and claimed, and creates queued below a
folder made offline. Its migration checks every column of the queue and
the sealed revisions, as the real-database test now does too.
2026-10-03 21:55:15 +02:00
nvrl ad8b225cfd refactor(fuse): one tree for every mount
On-demand sync folders kept their own State, which the drain, the
event applier and the sweeps had to visit one by one (B74). Every mount
now serves from the primary's tree. A session keeps only its root, which
the kernel knows as inode 1, and its notification channel; work that
changes the tree tells every mounted session. Size upgrades become
daemon-wide, and moves compare mounts by root instead of by state.
2026-10-03 16:53:31 +02:00
nvrl ea27f2bcd0 test(core): migrate a copy of a real database
An ignored test takes a copy of the database named by PDFS_MIGRATE_DB
with VACUUM INTO over a read-only connection, migrates it, and checks
that every queued op survives unchanged, every node keeps its row
number as its lid, and no node is lost or doubled.
2026-10-03 16:53:31 +02:00
nvrl e4062cc97b feat(fuse,core): number inodes by local id
A node's inode is now derived from its row's local id instead of a
per-run counter, so the same node keeps the same inode across restarts
and across every place that interns it. A lid handed out before the
row is stored reads as no node.
2026-10-03 16:41:26 +02:00
nvrl c34cc93c45 fix(fuse,core): keep a rename made while its create is on the wire
A rename of a node whose create is still queued rewrites the create's
target, but the drain had already copied the op. When the create
landed, the row was deleted unread and the rename was lost (B136).

finish_create now learns where the create was sent and where it landed,
and queues a rename of the real uid when the row has moved on since.
adopt_real_uid keeps the tree's name and folder while that rename is
queued. A landed folder's children also stop naming its placeholder as
their parent in memory, which put the placeholder back on their rows.
2026-10-03 16:20:00 +02:00
nvrl 7e3df7602d feat(core): give every node row a local id that never changes
First step of phase 2 of docs/MILESTONE-3.0.0.md. Schema 36 rebuilds
`nodes` with `lid INTEGER PRIMARY KEY AUTOINCREMENT`. Each row takes its
old rowid as its lid, so the search index, keyed by rowid, stays valid.
AUTOINCREMENT keeps a deleted row's lid from coming back. `uid` stays
unique and becomes nullable, for nodes that are not on Drive yet.

A landed create's row now takes the real uid in place, keeping its lid.
When a listing had already stored the real node, its content is kept
under the placeholder's lid, which the node was known by first.

Nothing reads the lid yet apart from `Db::lid_of`.
2026-10-03 15:10:51 +02:00
nvrl 61a0d31408 chore(release): 2.9.0 2026-10-03 15:00:38 +02:00
nvrl 644a47eb36 fix(ci): quote the simulation command so the workflow parses
The ': ' in 'sim:: --' read as a YAML mapping, so GitHub rejected
ci.yml and ran no job at all since 2c46d62.
2026-10-03 14:55:13 +02:00
nvrl cd90c566cd fix(fuse,core): renames across folders, replaced files, and writes after a create
Found by the seeded simulation runs (docs/BUGS.md).

- B122: a rename across folders goes to Drive as one move-multiple
  request, so a sibling holding the new name in the old folder no longer
  fails it with EIO. The drain uses the same request.
- B123: a second write open on a file just written no longer fails with
  EIO. An empty file counts as complete, our own renames, moves and
  online creates record their echo, a duplicated event is skipped, a
  rename keeps the file's mtime while its revision stays, and a landed
  upload is cached before open handles are rebased onto it.
- B124: a node put back after a failed replacing rename leaves `hidden`
  and its folder is listed again, so it shows in the mount.
- B131: a landed create is read back with the same short retry as an
  online create, and its bytes are cached under the real uid before an
  open handle is repointed at it.
- B135: an online rename settles the tree through the state guard, so
  the database has both folders unlisted before the reply. A lookup
  right after it no longer rebuilds the destination without the file.

The simulation runs no longer tolerate B122 or B131.
2026-10-03 14:50:12 +02:00
nvrl 1c262eb601 refactor(fuse,core): keep queued-op preconditions in the database
Phase 1 of docs/MILESTONE-3.0.0.md, without a change in behaviour.

- Own sealed revisions live only in the own_sealed_rev table. The
  in-memory map in front of it is gone, so revision_conflict gives the
  same answer before and after a restart.
- A queued rename records the node's original name next to its original
  parent, and keeps both across renames that replace the row. The drain
  logs when Drive has the node neither where the rename found it nor
  where it takes it.

The SDK's NodeUpdated event carries no revision id, so echoes of our own
changes stay recognised by the self-change timer.
2026-10-03 14:49:58 +02:00
nvrl bf73a395e5 docs: record B132-B134 from the simulation runs
- B132: a folder whose mkdir is still queued is gone after a restart.
  The root's listing, served offline, was invalidated when the link
  came back, and the restart listed it on Drive without the queued
  folder.
- B133: a file deleted while its upload is in flight can still come
  back as a recovered copy. B105's check goes by op id, and SQLite gives
  the next op queued the id the delete freed.
- B134: a file deleted online while the drain checks its write is kept
  as a conflict copy. The unlink trashes the file on Drive before it
  drops the queued write, so B105's check still finds the op.
2026-10-03 09:07:56 +02:00
nvrl a5138639a8 test(fuse): delete files as they upload in the simulation runs
The runs did not find B105 with its fix reverted. Drive lists a trash
later than the round trips take, so the drain never saw its own
client's delete while it still held the write.

The FakeDrive can now hold the next read of one node on its way to
Drive. A new profile, one-client-deletes, writes a file Drive holds,
waits for the drain to read the node back, deletes the file, and lets
the read go on once Drive shows the trash. A conflict the drain keeps
because a client's own file was trashed now fails the run as a false
conflict. With B105's fix reverted, seeds 1 and 2 fail on it.

Outside the step, such a conflict is counted as B134 when the copy is
named after the file: the drain looked after an online unlink had
trashed the file on Drive but before it dropped the write. Seed 3 of
three-clients came up with it once.

Half the time the step makes the file again before the read goes on.
That is B133: the new file's write takes the op id the delete freed,
and B105's check finds it. The runs count it.

The runs also count B132, an ENOENT after a restart that restored
queued ops, in a seed that lost the link. Seed 4 of one-client-flaky
failed on it. The known-bug matchers now also see what was logged
before the call.
2026-10-03 09:07:56 +02:00
nvrl 988b78e810 docs: record the bugs the simulation runs found (B122-B131)
The first simulation runs found ten open bugs. They are documented here
and not fixed yet, as phase 0 of docs/MILESTONE-3.0.0.md changes no
behaviour. The runs count each one and end the seed where the mount
leaves the model behind.

- B122: a rename across folders fails with EIO when the source folder
  holds the new name.
- B123: a second write open on a file just written fails with EIO.
- B124: after a replacing rename fails, the file it was replacing is gone
  from the mount.
- B125: a folder made offline fails with EIO once its listing is
  invalidated.
- B126: a file created offline under a name an offline rename freed
  lands as a conflict copy.
- B127: a file created offline lands twice when the answer to its upload
  is lost.
- B128: a file created offline over one deleted offline can be lost.
- B129: a file deleted or replaced while its create is on the wire stays
  on Drive.
- B130: a read of a file that a remote change invalidates can hang the
  daemon. The event sync blocks a runtime worker in inval_inode on the
  read's page lock. Nothing then drives the runtime's timers, which the
  read's fetch is waiting for.
- B131: a write to a file whose create could not be read back lands as a
  conflict copy. The create's op is retired before the read-back, so a
  failed read-back leaves the tree on the placeholder's mtime and no
  revision id.
2026-10-03 07:47:02 +02:00
nvrl 2c46d62429 ci: run the simulation runs outside pull requests
A `simulation` job runs the seeded simulation runs of
docs/MILESTONE-3.0.0.md section 8.2 on pushes to main and manual runs. They
mount FUSE against the in-memory FakeDrive, so they need fuse3 but no
account. Like the acceptance suite they stay off pull requests. The job
has a 30 minute limit, and the runs' own watchdog turns a hang into a
failed seed well inside it.
2026-10-03 07:47:02 +02:00
nvrl 97743f17e3 test(fuse): add seeded simulation runs over the FakeDrive
The second half of the simulation harness in docs/MILESTONE-3.0.0.md
section 8.2. A run starts one or more real FUSE daemons on a shared
FakeDrive. Each client owns a folder `c<n>` that only it writes. A seed
then draws syscalls through the mounts, link flaps, restarts, pauses and
settles. Every syscall is checked against a model of the client's tree as
it happens. After the last step every link comes up and the run checks
no loss, POSIX behaviour, no false conflicts, convergence and liveness.

- sim::model is the POSIX model: the tree, what each call should answer,
  and how it changes the tree.
- sim::run has the profiles: one client on a good link, one on a flaky
  link with flaps and restarts, and three clients. PDFS_SIM_SEEDS sets
  how many seeds each runs, and PDFS_SIM_SEED replays one.
- sim::watchdog catches a call that never returns. It takes every
  thread's stack in the process, since no tracer can stop a thread
  blocked on a FUSE request. It then aborts the run's FUSE connections
  and writes the stacks next to the run's state.
- The daemons' logs at INFO and above are kept, so a known open bug is
  told apart from a new failure. Known bugs are counted, and the seed ends
  where the mount leaves the model behind. PDFS_SIM_KNOWN=fail fails on
  them instead.
- A restart retries opening the database while the last run still holds
  it.
- The FakeDrive restamps a node's modification time on a rename or move,
  as Drive does (B113).

The runs mount FUSE, so they are ignored by default. Run them with
`cargo test -p pdfs-fuse --lib sim:: -- --ignored --test-threads=1`.
Test-only; no behaviour change.
2026-10-03 07:47:01 +02:00
nvrl 3d10338807 test(fuse): run real FUSE daemons against the FakeDrive
mount() becomes a thin wrapper over mount_with(), which takes the
DriveApi to reach Drive through and a Host. The daemon's Host is
unchanged: stopped by SIGTERM or SIGINT, installed at the same point as
before, and indexing the user's home. A simulated daemon is stopped over
a channel and skips the home index, so several can share one test
process without walking the developer's home.

The account quota moves onto DriveApi. It was the only call outside the
trait that marks the link lost, so a simulated daemon would have gone
offline on it; the FakeDrive now also models a full account
(InsufficientQuota).

sim::daemon starts one daemon on its own state directory with a real
FUSE mount, and a smoke test (ignored; needs /dev/fuse) checks a local
write lands, a remote write arrives, and a write made offline lands once
the link is back.
2026-10-03 05:09:02 +02:00
nvrl d53ba1be21 test(fuse): add an in-memory FakeDrive with Drive's known faults
The first half of the simulation harness in docs/MILESTONE-3.0.0.md
section 8.1. FakeDrive is one shared server; each simulated daemon reaches
it through a FakeClient that implements DriveApi and carries its own link
state, latency, faults and an SDK-style entity cache.

The faults are the ones the account runs met: listings that lag (B118,
B48), renames and moves refused for a stale name hash because the cache
is only emptied by events (B111, B121), names that refuse a replace (B13,
B104, B106), events delayed, duplicated, reordered and echoed (B70, B105),
requests that stall, fail, or change Drive and lose their reply (B107,
B121), and non-uniform block sizes (B84, B85, B87).

Test-only; no behaviour change.
2026-10-03 04:51:35 +02:00
nvrl c73d5c9436 refactor(fuse): reach Drive through a DriveApi trait
`Core` held a concrete `ProtonDriveClient`, so no test could make Drive
answer late, answer stale or not answer. This is the first step of
phase 0 in docs/MILESTONE-3.0.0.md.

`DriveApi` (`crates/pdfs-fuse/src/drive.rs`) covers the calls the sync
core makes: listings, events, creates, uploads, downloads, renames,
moves, trashes and restores. `ProtonDriveClient` implements it by
forwarding. `Core::drive` holds it, and every call of those methods now
goes through it. Sharing, Photos, devices, invitations and public links
stay on `Core::client`.

The trait takes readers and writers as trait objects, and
`impl dyn DriveApi` puts the client's generic signatures back, so call
sites read the same. Block reads go through `RevisionRead` instead of
the SDK's `RevisionReader`, which only the SDK can construct.

No behaviour change.
2026-10-03 04:37:37 +02:00
nvrl 8d92c69279 fix(fuse): keep an upload pending until its bytes are cached (B119)
The drain retired an upload's op first, and only then read the node back
and linked the staged blob into the content cache. For that one round
trip, the status reported an empty queue for a file that wasn't cached
yet, so anything that waited for an empty queue before reading the file
could still need the network. The account run of 2026-10-03 hit it over
a slow link: "regression B119" failed in the same second the upload
landed.

`Core::landing_uploads` now counts an upload from just before its op is
retired until its bytes are cached. The status adds it to
`pending_uploads`, and reads the queue before the count, so it always
sees the upload in one of them. A drop guard uncounts it on every exit
path.

The op still goes first. An op restored after a crash would re-send a
revision this daemon hasn't recorded as its own yet, and
`revision_conflict` would keep it as a conflict copy of the user's file.
2026-10-03 03:59:05 +02:00
nvrl e32959e40a fix(fuse): land a move right after a rename, and rename across folders quickly (B121)
The SDK caches each node, including the name hash it sends with the next
rename or move, and only an event evicts it. Renames and moves didn't
update that copy. So after a rename, the next change to the same node
went out with the old name hash, Drive refused it as out of date (422),
and the retry kept sending the stale hash until the event feed came
round, up to 10 seconds later.

- Every remote rename and move goes through `Core::rename_remote` or
  `Core::move_remote`. Both drop the SDK's copy of the node after a
  successful change and before each out-of-date retry, so the retry
  sends the current hash.
- A plain move was not retried at all, so a move right after a rename
  failed with EIO through the mount and with "out of date" from
  `pdfs move`. It now retries like a rename.
- The drain's combined move-and-rename also drops the SDK's copy once
  it lands.
- A rename that also changes folders through the mount replies only
  once both halves have landed. When the move fails, the file keeps
  its new name in the source folder, the tree is told so, and the
  rename reports EIO. A dropped link still queues the change.

B111's cause in docs/BUGS.md is corrected to the same SDK cache. The
acceptance case "regression B121" renames and moves a file at once
through the mount and through `pdfs`, and checks names and bytes.
2026-10-03 03:58:38 +02:00
nvrl 0b27c884a7 test(acceptance): turn B114 to B117 into regressions and ride out a dropped link
New live cases:

- regression B48, B80, B94, B95, B98, B102 and B111 replay the bugs
  their entries describe. B48 and B111 also run against the reference.
- regression B114 to B117 replace the open B114 and B115 cases, and
  fail with `check` when the bug comes back.
- regression B119: files written through the mount stay cached once
  they are uploaded.
- regression B86: after a mirror folder switches to on-demand, the
  mount lists what the mirror uploaded just before.

`--journal-check` also counts three warnings as errors: a conflict
copy (B113), a FUSE job held for a long time (B110) and an authority
re-read because the local tree lacked it (B117, B120).

A timeout no longer ends a target when the mount still answers a stat
and a listing. Cases that make many round trips get twice the limit.

A write is queued when the kernel releases the file, after close(2)
has returned, so cases now wait for the ops they expect
(`Daemon.queued`). "regression B102" saw one revision of two because
it read the queue once.

On flaky Wi-Fi, the run crashed twice on a read that the daemon
correctly failed with EIO once its read budget ran out. The end-of-run
read-back and the mode matrix's file checks now record a result
instead. The mode matrix tries such a read once more after the daemon
is back online, and notes it.
2026-10-03 02:10:49 +02:00
nvrl 37c04f1e69 fix(fuse,core): fix B114 to B120 from the account runs
- B114: `pdfs rm`, `pdfs rename` and `pdfs move` tell each kernel
  session that the old name is gone, so it stops resolving at once
  instead of for up to 30 seconds (`Core::forget_everywhere`,
  `Core::relocate_everywhere`).
- B115: `pdfs ls` and the app's file browser wait up to 10 seconds for
  the listing's size fetch, as `stat` does, so they show a file's real
  size instead of its encrypted one.
- B116: trashing a folder also drops the creates queued in its
  subfolders. A parked one stayed queued with its bytes, and a due one
  would have landed in the root.
- B117: `pdfs move` and `pdfs rename` move the node in each mount
  instead of forgetting it, so its queued upload keeps its path in
  `pdfs sync queue` and is not put off once after the resume.
- B118: when a create or mkdir reads back nothing, the mount asks again
  and then queues it as it does offline, instead of failing with ENOENT
  for something Drive has made.
- B119: a landed upload keeps its staged bytes as the file's cached
  content, unless it is larger than a sixteenth of the cache, so
  reading a file just written does not need the network.
- B120: a landed folder's row passes to its real uid in the same
  transaction, so a file queued inside it is not put off for five
  seconds.

docs/BUGS.md records B117 to B120. The account run of 2026-10-03
verified B114 to B117 and B119 in My files.
2026-10-03 02:10:23 +02:00
nvrl 625776081b test(acceptance): cover B113, B88 and the closed pipe, and report more readably
New live cases:

- regression B113: files renamed and moved while sync is paused upload
  without a conflict copy.
- regression B88: pdfs mkdir, rename and rm by local path, the
  mount-root guard, a path outside every mount, and the refusal in a
  mirrored folder.
- pdfs output into a closed pipe exits 0 and prints nothing on stderr.
- open B114 and open B115 check two bugs that are still open. While
  one reproduces, its case ends as "known", which is listed but does
  not fail the run.

The console output names the pdfs version, the storage and the daemon
state of each target, and prints one aligned line per case, with its
details beneath. A failed case points to its docs/BUGS.md entry. The
run ends with a summary table, the slowest cases, the known issues
and the numbered failures. A queue wait now fails at once on a parked
op instead of waiting out its timeout.

docs/BUGS.md records B114, B115 and B116, which the first live run
found, and marks B113 and B88 as verified live.
2026-10-02 22:16:36 +02:00
nvrl 8cde79b8e3 fix(cli): stop quietly when stdout is closed
`pdfs ls | head` ended in a "Broken pipe" panic once `head` had its
lines, because std's `println!` panics on a failed write.

The CLI now shadows `println!` and `print!` with versions that exit 0 on
a closed pipe and report any other write error, such as a full disk,
with exit 1. Restoring the default SIGPIPE would also stop the panic,
but the same binary runs the daemon, which must not die when a socket's
peer goes away.
2026-10-02 21:16:34 +02:00
nvrl eb53d122cc fix(fuse,cli): let rm, rename and mkdir take paths in on-demand folders
`pdfs rm ~/Downloads/x` answered "is not under the mountpoint" while
`pdfs move` took the same path: the Delete, Rename and CreateFolder
handlers still read paths against the primary mount only (B88).

They now go through `route_change`, the router ResolveConflict already
used, renamed from `route_conflict`. It answers the request in the
inode space of the mount that owns the path.

An on-demand folder's root has a parent on Drive, so `pdfs rm
~/Downloads` would have trashed the whole synced folder from under its
mount. rm and rename refuse a path that names a mount's root.
2026-10-02 19:54:37 +02:00
nvrl c64252193f fix(fuse): keep the revision id of a file created through the mount
The write handle a create opens was given no revision id, so the drain's
conflict check fell back to (mtime, size). Drive advances a node's mtime
on a move or rename, so a new file moved before its first upload drained
landed as a (sync-conflict) copy next to its own empty original. Take the
minted node's revision id, and rebase handles still open on a queued
create onto the node it landed as (B113).
2026-10-02 19:25:33 +02:00
nvrl cc70a21c05 fix(aur): version -bin source filenames
makepkg reuses a cached source file of the same name, so the unversioned
proton-drive.service from an older build failed the 2.8.2 checksum. Carry
$pkgver in every file fetched from the tagged tree, like LICENSE already
did, and bump pkgrel to 2.
2026-10-01 23:50:06 +02:00
Nils Pukropp 644995cc56 chore(aur): update packages to 2.8.2 2026-10-01 19:49:42 +00:00
nvrl 50483bf3e1 chore(release): 2.8.2 2026-10-01 21:39:05 +02:00
nvrl 4caba0a34c chore(deps): require proton SDK 0.7.2
0.7.2 resolves each shared root with its own key. With 0.7.1, when
several accepted shares came from the same owner's volume, only the first
appeared under Shared with me and the rest were skipped with "missing
key" (B112, proton-sdk-rs#2).
2026-10-01 18:17:25 +02:00
Nils Pukropp e8554d6434 chore(aur): update packages to 2.8.1 2026-10-01 14:58:09 +00:00
Claude ba1fdd65be chore(release): 2.8.1 2026-09-30 17:02:31 +00:00
nvrl 0d4163b9bf fix(service): start with the graphical session, not default.target (#24)
default.target is reached before the desktop exports DISPLAY/WAYLAND_DISPLAY
to the user manager, so After=graphical-session.target never applied. The
daemon's first keyring call D-Bus-activated the Secret Service without a
display; on KDE ksecretd aborted, the first start failed, and every login
left a coredump.

Install into graphical-session.target instead, and use reenable when the
front-ends enable the unit so existing default.target.wants links are
replaced rather than kept alongside the new one.

Fixes #23
2026-09-30 19:01:50 +02:00
Nils Pukropp f3c7a809f4 chore(aur): update packages to 2.8.0 2026-09-30 02:15:19 +00:00
nvrl c1b5378045 fix(aur): download sources from GitHub, not the homepage
73e60f1 pointed url at proton-drive.narl.io, and all three AUR PKGBUILDs
built their source URLs from $url, so the release tarball, the binary
archive and the git clone were fetched from the homepage and 404ed. The
sources now come from a separate _repo on GitHub; url stays the homepage.
2026-09-30 04:13:06 +02:00
nvrl 3d6f53c16d chore(release): 2.8.0 2026-09-30 03:56:47 +02:00
nvrl 8ceee5285b chore(fuse): log how long each queued upload takes
"pending upload landed" now carries upload_ms, the time on the wire, and
total_ms, the whole op including the conflict check and gap-fill. A slow
drain then shows where its time goes without a profiler.
2026-09-30 03:55:48 +02:00
nvrl a181f1a5f0 fix(core,fuse): keep working through network drops
The mount only ever went from offline to online: nothing marked it
offline again, so after the Wi-Fi dropped every create, mkdir, delete and
rename called Drive, waited out the SDK's retries for up to two minutes
and answered EIO. Background loops kept polling the dead link.

Connectivity now lives in pdfs-fuse's new link module and goes both
ways. A call that gets no answer (a transport error or a request
timeout) marks the mount offline; a probe thread checks every 2 s,
backing off to 30 s, and on recovery re-lists folders served from the
database, retries failed operations and wakes the drain. Interactive
calls get a 20 s deadline, so a dead link costs one slow call instead of
many.

While offline:
- create, mkdir, unlink, rmdir and rename fall back to the offline
  queue instead of answering EIO;
- folders list from the database even when not marked listed, and are
  re-listed once the link returns;
- reads of uncached data wait up to 30 s for the link, then retry once
  within the 120 s read deadline;
- statfs serves the last quota instead of fetching, and quota refreshes
  are single-flight;
- the event and photo event loops wait for the link instead of polling.

A create whose request timed out may have landed anyway. When the drain
replays it and finds a same-named twin of the same kind, created within
the last two minutes and still empty, it adopts that node instead of
making a conflict copy. The drain also leaves alone a name still held
by a queued trash.
2026-09-30 03:25:00 +02:00
nvrl 7251f00e91 chore(deps): require proton SDK 0.7.1
0.7.1 sends TCP keepalives on its connections, so a link that dies
silently is noticed instead of leaving a request waiting on a dead
socket until its timeout.
2026-09-30 03:25:00 +02:00
nvrl cb38f79aa3 fix(fuse,scripts): log why a create finds no parent and fit the metadata floor to FUSE
A create that answered ENOENT left no trace in the log. It now warns
when the parent folder is no longer known, and when the parent left the
tree while its children were being listed, naming the parent inode and
the new name.

The acceptance script's metadata throughput floor was 20 operations per
second, which a FUSE mount cannot reach: each create and unlink waits for
Drive before it returns, about 350 ms apiece. On a FUSE mount the floor
is now 3 operations per second; other paths keep 20. The mount type is
read from /proc/self/mountinfo.
2026-09-29 21:22:54 +02:00
nvrl 54e9921efb fix(fuse): stop reads deadlocking on size upgrades and retry stale renames
A read that settles a provisional size invalidated the file's page cache
while the kernel held that read's page lock, so the worker waited on
itself forever (B110). Size upgrades now invalidate attributes only.

Renaming a node right after renaming it could be refused by Drive as out
of date, because the name hash read just before still held the old name.
Every remote rename now goes through Core::rename_remote, which retries
InvalidRequirements for up to ten seconds (B111).
2026-09-29 19:47:20 +02:00
nvrl f252b09757 fix(fuse): keep the size of a file open for writing across re-listings
B109: an online create seals an empty file on Drive, and until the last
close the written bytes live only in the open handle. A re-listing of
the parent, a size upgrade or a create landing re-interned the node with
Drive's empty size, the kernel capped reads at that size, and SQLite
reported "database disk image is malformed". State::keep_open_write_size
now carries the handle's size and local mtime over in all three places.

The acceptance script waits for My files to mount after a daemon restart
instead of failing at once.
2026-09-29 16:49:13 +02:00
nvrl a2992b3b9d fix(core,fuse): keep replaced and vanished files from failing saves, reads and syncs
Fixes from the account acceptance runs:

- B104: a reader open across a rename-over keeps working; the replaced
  node is unlinked like any other open file instead of forgotten.
- B105: a drain worker holding a revision withdrawn by an unlink no
  longer uploads it as a recovered-… conflict copy (Db::op_exists).
- B106: an open victim's queued ops are withdrawn at trash time, so a
  repeated editor save no longer fails with AlreadyExists; the staged
  blob goes with the last close.
- B107: a remote read gives up after 120 s with EIO instead of hanging
  its caller; the timer is built inside the runtime.
- B108: a mirror upload whose local file was deleted after the scan is
  skipped rather than counted as an error.

The acceptance script reports the failing line for non-assert errors,
ignores queue ops that predate the run, purges with delete-forever
first, and diagnoses a malformed SQLite database against a local replay.
2026-09-29 16:20:51 +02:00
nvrl 7b67db5ced fix(core,fuse,cli): harden writes, trash and cleanup found by account acceptance runs
The FUSE acceptance suite now runs on the signed-in account by default and
cleans up after itself; its first live runs found three daemon bugs.

- B101: a pwrite past EOF now authors the hole it leaves, so truncated bytes
  no longer come back from the base. A write attached to a create while that
  create uploads becomes a revision of the real uid instead of being dropped.
- B102: removing a folder drops the queued revisions of files below it, found
  through the node tree, with their blobs and in-flight uploads. `pdfs rm`
  now drops queued ops too.
- B103: each server call of a trash refresh times out after 120 s, so one hung
  call can no longer freeze the trash listing until restart.

Acceptance suite: account mode with a manifest, lock and reaper; cleanup by
uid; queued ops naming a test root count as leftovers; new edge-case,
throughput and B100/B101 cases; the wide-directory case is split from the
deep tree and gets a doubled budget. Adds `pdfs trash --json`.
2026-09-29 10:09:44 +02:00
nvrl 84c9ade8e3 fix(fuse): settle provisional sizes before reads and write opens
A file listed by the cheap enumeration reports its ciphertext size until
the size upgrade lands. Since B84 a read on that size fails with EIO, and
a write open started its handle at it. Reads and write opens now fetch
the real size first. A folder past the size-upgrade cap is queued instead
of dropped, so a large copy no longer leaves most folders provisional
(B100).
2026-09-29 07:37:41 +02:00
nvrl 9e26cd0285 chore(release): 2.7.0 2026-09-29 05:30:38 +02:00
nvrl 73e60f1151 chore: point homepage metadata at proton-drive.narl.io
The About dialog's website link, the README, and the package metadata
(Cargo, PKGBUILDs, AUR .SRCINFO, .deb, .rpm) now use the project
homepage. Source code, issue, and release links stay on GitHub.
2026-09-29 05:30:38 +02:00
nvrl 088e8f3b0d test(acceptance): cover moves between locations in the managed matrix
Every managed mode pair now also moves trees between the two folders in both
directions, moves several sources in one command, and passes a tree through
My files and out again. Each case waits for Drive before and after, then
checks the tree is only at the destination, byte for byte, with no conflict
copies. Refusal cases cover a taken name, a folder into itself, a mirror
source holding a symlink, and an on-demand file still open for writing.
2026-09-29 00:38:40 +02:00
nvrl 261d5dbf84 feat(core,fuse,cli): reuse cached content and take several sources in moves
- A mirror download first copies the file out of the content cache when the
  same revision is cached whole or block by block, so moving a file out of an
  on-demand folder into a mirror costs no traffic for what was already read.
- `pdfs move a b c DEST` moves every source, reports each failure and exits
  non-zero if any failed.
- Mirror-to-mirror moves rename with RENAME_NOREPLACE, so a name that
  appeared at the destination after the check is never overwritten.
- The local copy a move out of a mirror drops is checked again just before
  it is deleted; anything written into it meanwhile keeps the copy and is
  uploaded as new by the next pass.
2026-09-29 00:38:40 +02:00
nvrl 5f9d07aff1 feat(fuse,core,cli,gui): move between locations without transferring content
A move request with two absolute paths is resolved against every
location (My files, on-demand and mirror synced folders) and made as a
server-side move_node. The local side is levelled afterwards: mounts
forget and re-list, a mirror folder renames its copy and baseline along
or drops the copy once Drive holds it.

Anything only this disk holds is refused up front: queued ops, open
writes, unsynced or ignored mirror files and symlinks. The GUI Move
dialog gains a top-level list of locations.
2026-09-29 00:26:56 +02:00
nvrl 3fb7b9be6b chore(release): 2.6.2 2026-09-28 23:58:26 +02:00
nvrl 6c3a50ce7f fix(gui): let the Sync page use the full width
The transfer label did not ellipsize, so a long file name set the
column's minimum width and pushed it past the clamp. The page width
changed with every file.

The label now ellipsizes in the middle, and the Sync page drops the
clamp so Overview and Folders fill the space next to the sidebar.
2026-09-28 23:58:26 +02:00
nvrl f8304a0e86 fix(gui): keep Albums and Places selected when returning to Photos
Every visit to the Photos page loaded the timeline, and a finished
timeline load always shows the timeline. Leaving Albums or Places and
coming back therefore showed the timeline under the Albums or Places
toggle.

A visit now reloads the view the toggle selects. An open album or place
still loads its photos as before.
2026-09-28 23:58:26 +02:00
Nils Pukropp e63f39f8c2 chore(aur): update packages to 2.6.1 2026-09-28 21:44:18 +00:00
nvrl 6f100dbe5e chore(images) updated prompt image 2026-09-28 23:35:53 +02:00
nvrl 7c5ef269cd chore(release): 2.6.1 2026-09-28 23:35:14 +02:00
nvrl f13b6fc0ef feat(gui): add pdfs-prompt --preload for session autostart
--preload registers the resident GTK launcher and realizes its window
without showing it, so the first shortcut opens it as fast as later
ones. When a launcher is already running it exits without activating
it.

The application hold now happens in startup, which only the primary
instance runs. Taken before registration, it also kept every remote
invocation alive after it had forwarded its activation.
2026-09-28 23:35:14 +02:00
nvrl 3d61fc637b ci(release): reuse cached builds from main
The release job never hit a cache: its rust-cache key contained the job
name, tags cannot restore caches of earlier tags, and rpmbuild rebuilt
the whole workspace under its own compiler flags.

CI on main now saves shared debug and release caches that the release
workflow restores. Quality gates and the package build run in parallel,
and the GitHub release is created once both pass. The RPM spec skips
its cargo build when the caller passes --define "prebuilt 1".
2026-09-28 23:26:24 +02:00
Nils Pukropp 889b2aa4a9 chore(aur): update packages to 2.6.0 2026-09-28 21:22:30 +00:00
80 changed files with 18765 additions and 4380 deletions

No files matched your search

+77
View File
@@ -32,8 +32,14 @@ jobs:
with:
toolchain: '1.96'
components: rustfmt, clippy
cache: false
# Tags can only restore caches saved on main, so only main saves one. The
# release workflow restores this key for its quality gates.
- uses: swatinem/rust-cache@v2
with:
shared-key: debug
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Formatting
run: cargo fmt --all -- --check
@@ -57,3 +63,74 @@ jobs:
# Real Proton credentials and /dev/fuse deliberately do not run on pull
# requests. Follow docs/TESTING.md on a dedicated test-account runner.
# The seeded simulation runs (docs/MILESTONE-3.0.0.md §8.2): real FUSE
# daemons against an in-memory Drive, so they need /dev/fuse but no account.
simulation:
if: github.event_name != 'pull_request'
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- uses: actions/checkout@v7
- name: Install native dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
pkg-config \
fuse3 \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
- name: Install minimum supported Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
cache: false
- uses: swatinem/rust-cache@v2
with:
shared-key: debug
save-if: false
- name: Simulation runs
run: 'cargo test -p pdfs-fuse --lib --locked sim:: -- --ignored --test-threads=1'
# Keeps the release-profile dependencies cached on main, where the release
# workflow can restore them; a tag cannot reuse the caches of earlier tags.
release-cache:
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
- name: Install native dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
pkg-config \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
- name: Install minimum supported Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
cache: false
- uses: swatinem/rust-cache@v2
with:
shared-key: release
- name: Release build
run: cargo build --release --locked
+73 -22
View File
@@ -10,8 +10,60 @@ permissions:
contents: write
jobs:
checks:
name: Release quality gates
runs-on: ubuntu-24.04
steps:
- name: Checkout repository
uses: actions/checkout@v7
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
pkg-config \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
- name: Set up Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
components: rustfmt, clippy
cache: false
# A tag can read the caches of main but its own caches are useless to the next
# tag, so this only restores what CI saved on main.
- name: Cache Cargo target
uses: swatinem/rust-cache@v2
with:
shared-key: debug
save-if: false
- name: Verify tag and manifest versions
if: startsWith(github.ref, 'refs/tags/')
shell: bash
run: |
tag_version="${GITHUB_REF_NAME#v}"
manifest_version="$(sed -n 's/^version = "\([^"]*\)"/\1/p' Cargo.toml | head -n1)"
pkgbuild_version="$(sed -n 's/^pkgver=//p' packaging/PKGBUILD | head -n1)"
test "$tag_version" = "$manifest_version"
test "$tag_version" = "$pkgbuild_version"
- name: Release quality gates
run: |
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only
build-release:
name: Build and Release
name: Build packages
runs-on: ubuntu-24.04
environment: 'rpm signing'
@@ -38,27 +90,14 @@ jobs:
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
components: rustfmt, clippy
cache: false
# Restores the release-profile dependencies that CI builds on main.
- name: Cache Cargo target
uses: swatinem/rust-cache@v2
- name: Verify tag and manifest versions
if: startsWith(github.ref, 'refs/tags/')
shell: bash
run: |
tag_version="${GITHUB_REF_NAME#v}"
manifest_version="$(sed -n 's/^version = "\([^"]*\)"/\1/p' Cargo.toml | head -n1)"
pkgbuild_version="$(sed -n 's/^pkgver=//p' packaging/PKGBUILD | head -n1)"
test "$tag_version" = "$manifest_version"
test "$tag_version" = "$pkgbuild_version"
- name: Release quality gates
run: |
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only
with:
shared-key: release
save-if: false
- name: Build release binaries
run: |
@@ -127,7 +166,7 @@ jobs:
Maintainer: Nils Pukropp <contact@narl.io>
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4, libimage-exiftool-perl
Installed-Size: ${INSTALLED_SIZE}
Homepage: https://github.com/narrrl/proton-drive-linux
Homepage: https://proton-drive.narl.io
Description: Unofficial Proton Drive client for Linux
Files-on-demand Proton Drive for the Linux desktop, unaffiliated with Proton AG:
- FUSE virtual mount with block-level caching
@@ -152,6 +191,7 @@ jobs:
--define "_topdir $PWD/rpmbuild" \
--define "_sourcedir $PWD" \
--define "git_dir $PWD" \
--define "prebuilt 1" \
--define "version ${RPM_VERSION}" \
--define "_dbpath $PWD/rpmbuild/db" \
packaging/proton-drive-linux.spec
@@ -190,8 +230,19 @@ jobs:
*.rpm
if-no-files-found: error
publish:
name: Create GitHub Release
needs: [checks, build-release]
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-24.04
steps:
- name: Download packages
uses: actions/download-artifact@v8
with:
name: proton-drive-linux-builds
- name: Create GitHub Release
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v3
with:
files: |
@@ -205,7 +256,7 @@ jobs:
aur:
name: Publish AUR packages
needs: build-release
needs: publish
if: startsWith(github.ref, 'refs/tags/')
uses: ./.github/workflows/aur.yml
with:
Generated
+10 -8
View File
@@ -3474,7 +3474,7 @@ checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4"
[[package]]
name = "pdfs-cli"
version = "2.6.0"
version = "2.10.0"
dependencies = [
"anyhow",
"clap",
@@ -3489,7 +3489,7 @@ dependencies = [
[[package]]
name = "pdfs-core"
version = "2.6.0"
version = "2.10.0"
dependencies = [
"async-trait",
"dbus-secret-service-keyring-store",
@@ -3513,8 +3513,9 @@ dependencies = [
[[package]]
name = "pdfs-fuse"
version = "2.6.0"
version = "2.10.0"
dependencies = [
"async-trait",
"base64 0.23.1",
"fuser",
"futures",
@@ -3528,12 +3529,13 @@ dependencies = [
"sha1 0.11.0",
"tokio",
"tracing",
"tracing-subscriber",
"zip",
]
[[package]]
name = "pdfs-gui"
version = "2.6.0"
version = "2.10.0"
dependencies = [
"anyhow",
"async-channel",
@@ -3773,9 +3775,9 @@ dependencies = [
[[package]]
name = "proton-drive-rs"
version = "0.7.0"
version = "0.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0f6996724e51e99f03d2843828de9477afc39baa764403cb69490cabc1c4d93"
checksum = "d6ce113d234474821d34787436ee80e1458e3d8a8f97212219615ed315eb70be"
dependencies = [
"async-trait",
"base64 0.23.1",
@@ -3800,9 +3802,9 @@ dependencies = [
[[package]]
name = "proton-sdk"
version = "0.7.0"
version = "0.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91cc25654133bd5714407a1df0996eaf214a6e8e0bb95d8139bc42e7cee31c34"
checksum = "2a33c67df9694f90ce60e5b34984275956244138511b4640ca0d100e1dac8b3d"
dependencies = [
"aes-gcm 0.11.1",
"async-trait",
+4 -4
View File
@@ -3,20 +3,20 @@ resolver = "2"
members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"]
[workspace.package]
version = "2.6.0"
version = "2.10.0"
edition = "2024"
license = "MIT"
authors = ["Nils Pukropp <contact@narl.io>"]
rust-version = "1.96"
repository = "https://github.com/narrrl/proton-drive-linux"
homepage = "https://github.com/narrrl/proton-drive-linux"
homepage = "https://proton-drive.narl.io"
readme = "README.md"
keywords = ["proton", "drive", "fuse", "sync", "linux"]
categories = ["filesystem", "command-line-utilities"]
[workspace.dependencies]
proton-sdk = { version = "0.7.0" }
proton-drive-rs = { version = "0.7.0" }
proton-sdk = { version = "0.7.2" }
proton-drive-rs = { version = "0.7.2" }
# Internal crates.
pdfs-core = { path = "crates/pdfs-core" }
+2
View File
@@ -8,6 +8,8 @@
local folders, Photos, sharing, and a native GTK4 app with tray and search launcher, plus a
scriptable CLI.
**[proton-drive.narl.io](https://proton-drive.narl.io)**
[![CI](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml/badge.svg)](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/v/release/narrrl/proton-drive-linux?sort=semver)](https://github.com/narrrl/proton-drive-linux/releases/latest)
[![AUR](https://img.shields.io/aur/version/proton-drive-for-linux)](https://aur.archlinux.org/packages/proton-drive-for-linux)
+86 -19
View File
@@ -20,6 +20,39 @@ use pdfs_core::control::{
use pdfs_core::db::Db;
use pdfs_core::service;
/// `println!` for this binary, shadowing std's, which panics when stdout is
/// closed: `pdfs ls | head` ended in a "Broken pipe" panic once `head` had its
/// lines. This one hands the failure to [`stdout_written`] instead.
///
/// Restoring the default `SIGPIPE` would also stop the panic, but the same
/// binary runs the daemon, which must not die when a socket's peer goes away.
macro_rules! println {
($($arg:tt)*) => {{
use std::io::Write as _;
$crate::stdout_written(writeln!(std::io::stdout(), $($arg)*))
}};
}
/// [`println!`] without the newline.
macro_rules! print {
($($arg:tt)*) => {{
use std::io::Write as _;
$crate::stdout_written(write!(std::io::stdout(), $($arg)*))
}};
}
/// Exit on a failed write to stdout. A closed pipe means the reader has all it
/// wanted, so that exit is a success.
fn stdout_written(result: std::io::Result<()>) {
if let Err(error) = result {
if error.kind() == std::io::ErrorKind::BrokenPipe {
std::process::exit(0);
}
eprintln!("Error: writing to stdout: {error}");
std::process::exit(1);
}
}
#[derive(Parser)]
#[command(
name = "pdfs",
@@ -30,7 +63,7 @@ struct Cli {
/// Emit machine-readable JSON instead of formatted text.
///
/// Applies to the query commands (`status`, `ls`, `pins`, `sync list`,
/// `devices list`, `locations`, `transfers`, `activity`, `cache inspect`). Commands that
/// `devices list`, `locations`, `transfers`, `activity`, `trash`, `cache inspect`). Commands that
/// perform an action keep their human output — a script that needs to know
/// whether one succeeded has the exit code.
#[arg(long, global = true)]
@@ -285,26 +318,34 @@ enum Command {
},
/// Rename a file or folder via the running daemon.
Rename {
/// File/folder path, inside the mountpoint or relative to it.
/// File/folder path, inside My files or an on-demand synced folder, or
/// relative to the mountpoint.
path: PathBuf,
/// New name (a single path component).
new_name: String,
},
/// Move a file or folder into another folder via the running daemon.
/// Move files or folders into another folder via the running daemon.
///
/// The paths may be in different locations: My files and any synced
/// folder, on-demand or mirrored. The move happens on Proton Drive, so
/// nothing is downloaded or uploaded again, unlike `mv` between two
/// locations. The last path is the destination folder.
Move {
/// File/folder path, inside the mountpoint or relative to it.
path: PathBuf,
/// Destination folder path, inside the mountpoint or relative to it.
new_parent: PathBuf,
/// Files/folders to move, then the destination folder: each inside a
/// location, or relative to the mountpoint.
#[arg(required = true, num_args = 2.., value_name = "PATH")]
paths: Vec<PathBuf>,
},
/// Trash a file or folder via the running daemon.
Rm {
/// File/folder path, inside the mountpoint or relative to it.
/// File/folder path, inside My files or an on-demand synced folder, or
/// relative to the mountpoint.
path: PathBuf,
},
/// Create a folder via the running daemon.
Mkdir {
/// Parent folder path, inside the mountpoint or relative to it.
/// Parent folder path, inside My files or an on-demand synced folder,
/// or relative to the mountpoint.
parent: PathBuf,
/// New folder name (a single path component).
name: String,
@@ -800,7 +841,10 @@ fn main() -> Result<()> {
scope,
} => cmd_search(query, limit, scope),
Command::Rename { path, new_name } => cmd_rename(path, new_name),
Command::Move { path, new_parent } => cmd_move(path, new_parent),
Command::Move { mut paths } => {
let new_parent = paths.pop().expect("clap requires at least two paths");
cmd_move(paths, new_parent)
}
Command::Rm { path } => cmd_rm(path),
Command::Mkdir { parent, name } => cmd_mkdir(parent, name),
Command::Upload { sources, parent } => cmd_upload(sources, parent),
@@ -2439,14 +2483,33 @@ fn cmd_rename(path: PathBuf, new_name: String) -> Result<()> {
Ok(())
}
fn cmd_move(path: PathBuf, new_parent: PathBuf) -> Result<()> {
match control_request(CtlRequest::Move {
path: path_arg(&path)?,
new_parent: path_arg(&new_parent)?,
})? {
CtlResponse::Ok { message } => println!("{message}"),
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
/// Move each of `sources` into `new_parent`, one request each. A refusal names
/// its source and the rest still move; the command fails if any did not.
fn cmd_move(sources: Vec<PathBuf>, new_parent: PathBuf) -> Result<()> {
let new_parent = path_arg(&new_parent)?;
let mut failed = 0;
for source in &sources {
let outcome = path_arg(source).and_then(|path| {
match control_request(CtlRequest::Move {
path,
new_parent: new_parent.clone(),
})? {
CtlResponse::Ok { message } => Ok(message),
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
}
});
match outcome {
Ok(message) => println!("{message}"),
Err(e) if sources.len() == 1 => return Err(e),
Err(e) => {
eprintln!("{}: {e:#}", source.display());
failed += 1;
}
}
}
if failed > 0 {
bail!("{failed} of {} could not be moved", sources.len());
}
Ok(())
}
@@ -2524,7 +2587,11 @@ fn cmd_diagnostics() -> Result<()> {
}
fn cmd_trash() -> Result<()> {
match control_request(CtlRequest::ListTrash)? {
let response = control_request(CtlRequest::ListTrash)?;
if emit_json(&response)? {
return Ok(());
}
match response {
CtlResponse::Entries { entries } if entries.is_empty() => println!("(trash is empty)"),
CtlResponse::Entries { entries } => {
for e in entries {
+113
View File
@@ -365,6 +365,14 @@ pub struct StagedWrite {
pub based_on: Option<Baseline>,
}
impl StagedWrite {
/// Whether `authored` ranges cover a file of `len` bytes, so the staged
/// bytes are its complete new content. An empty file has nothing to cover.
pub fn covers(authored: &[(u64, u64)], len: u64) -> bool {
len == 0 || authored == [(0, len)]
}
}
/// The identity of a remote revision, as far as we can observe it.
///
/// `revision_id` is the authoritative identity when present: the server assigns a
@@ -730,6 +738,60 @@ impl ContentCache {
Some(buf)
}
/// Write the whole content of `uid` at `(mtime, size)` to `dst` from the
/// cache alone, and report whether that worked.
///
/// A whole-file blob is cloned. Otherwise every block of the revision has to
/// be cached, under its recorded geometry or, failing that, the uniform one
/// the read path would use. Anything short of the full content leaves
/// nothing at `dst`, so a caller falls back to a download with a clean slate.
///
/// This is what lets a mirror folder take a file an on-demand mount has
/// already read, after a move between them, without fetching it again. The
/// trust is the read path's: the same `(mtime, size)` key, and blocks that
/// were validated when they were fetched.
pub fn copy_cached_to(&self, uid: &NodeUid, mtime: i64, size: u64, dst: &Path) -> bool {
if size == 0 {
return false;
}
if let Some(blob) = self.valid_blob(uid, mtime, size) {
if Self::clone_or_copy(&blob, dst).is_ok_and(|len| len == size) {
self.touch(Self::key(uid));
return true;
}
let _ = std::fs::remove_file(dst);
return false;
}
let geometry = self
.block_geometry(uid, mtime, size)
.unwrap_or_else(|| BlockGeometry::uniform(size));
if geometry.size() != size {
return false;
}
let spans = geometry.spans(0, size);
// Checked up front so a file with one block missing costs stats, not a
// copy of every block before it.
if !spans
.iter()
.all(|&span| self.has_block(uid, mtime, size, span))
{
return false;
}
let written = (|| -> Option<()> {
use std::io::Write;
let mut out = std::fs::File::create(dst).ok()?;
for span in spans {
out.write_all(&self.cached_block(uid, mtime, size, span)?)
.ok()?;
}
(out.metadata().ok()?.len() == size).then_some(())
})();
if written.is_none() {
let _ = std::fs::remove_file(dst);
}
written.is_some()
}
/// Store `bytes` as the cached content for `uid`, tagged with `(mtime,
/// size)`. The blob is written to a temp file then renamed so a concurrent
/// reader never sees a partial blob; the meta tag is written last so a
@@ -2002,6 +2064,17 @@ mod tests {
NodeUid::new(VolumeId::from("vol"), LinkId::from(link))
}
#[test]
fn an_empty_write_covers_the_file_and_a_gap_does_not() {
assert!(
StagedWrite::covers(&[], 0),
"an empty file has nothing to cover"
);
assert!(StagedWrite::covers(&[(0, 7)], 7));
assert!(!StagedWrite::covers(&[(0, 3), (4, 7)], 7));
assert!(!StagedWrite::covers(&[], 7));
}
/// A unique temp directory removed on drop; avoids a dev-dependency.
struct TempDir(PathBuf);
impl TempDir {
@@ -2561,6 +2634,46 @@ mod tests {
assert!(c.cached_block(&u, 100, 5000, blk(5000, 0)).is_none());
}
#[test]
fn a_cached_blob_is_copied_out_whole() {
let (c, d) = cache();
let u = uid("blob");
c.store(&u, 7, 5, b"hello").unwrap();
let dst = d.path().join("out");
assert!(c.copy_cached_to(&u, 7, 5, &dst));
assert_eq!(std::fs::read(&dst).unwrap(), b"hello");
// Another revision is not this one.
assert!(!c.copy_cached_to(&u, 8, 5, &d.path().join("stale")));
assert!(!d.path().join("stale").exists());
}
#[test]
fn cached_blocks_are_copied_out_in_order() {
let (c, d) = cache();
let u = uid("blocks");
let size = BLOCK_SIZE + 3;
let first = vec![1_u8; BLOCK_SIZE as usize];
c.store_block(&u, 7, size, blk(size, 0), &first).unwrap();
c.store_block(&u, 7, size, blk(size, 1), b"end").unwrap();
let dst = d.path().join("out");
assert!(c.copy_cached_to(&u, 7, size, &dst));
let bytes = std::fs::read(&dst).unwrap();
assert_eq!(bytes.len() as u64, size);
assert_eq!(&bytes[..BLOCK_SIZE as usize], first.as_slice());
assert_eq!(&bytes[BLOCK_SIZE as usize..], b"end");
}
#[test]
fn a_file_with_a_block_missing_is_not_copied_out() {
let (c, d) = cache();
let u = uid("partial");
let size = BLOCK_SIZE + 3;
c.store_block(&u, 7, size, blk(size, 1), b"end").unwrap();
let dst = d.path().join("out");
assert!(!c.copy_cached_to(&u, 7, size, &dst));
assert!(!dst.exists(), "a partial copy must not be left behind");
}
/// The [`BlockSpan`] the uniform geometry gives block `idx` of a file of
/// `size` bytes — the shape these tests were written against, and the one
/// this client's own uploads still have.
+4 -1
View File
@@ -339,7 +339,10 @@ pub enum Request {
/// single path component (no separators). Replies with [`Response::Ok`].
Rename { path: String, new_name: String },
/// Move a file or folder into a new parent folder. Both `path` and
/// `new_parent` are mountpoint-relative. Replies with [`Response::Ok`].
/// `new_parent` are mountpoint-relative, or both absolute local paths; two
/// absolute paths may lie in different locations (My files, an on-demand
/// or a mirror synced folder), and the move is still made on Proton Drive
/// without transferring content. Replies with [`Response::Ok`].
Move { path: String, new_parent: String },
/// Trash a file or folder. `path` is mountpoint-relative. Replies with
/// [`Response::Ok`].
+249
View File
@@ -0,0 +1,249 @@
/* WARNING: Script requires that SQLITE_DBCONFIG_DEFENSIVE be disabled */
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE sync_state (key TEXT PRIMARY KEY, value TEXT);
INSERT INTO sync_state VALUES('schema_version','35');
CREATE TABLE nodes (
uid TEXT PRIMARY KEY,
parent_uid TEXT,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER,
mtime INTEGER NOT NULL,
revision_hash TEXT,
trashed INTEGER NOT NULL DEFAULT 0
, node_json TEXT, listed INTEGER NOT NULL DEFAULT 0, path TEXT);
INSERT INTO nodes VALUES('vol~root',NULL,'My Files',1,0,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"root"},"parent_uid":null,"kind":"Folder","name":"My Files","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'');
INSERT INTO nodes VALUES('vol~docs','vol~root','Docs',1,0,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"docs"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":"Folder","name":"Docs","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs');
INSERT INTO nodes VALUES('vol~report','vol~docs','report.txt',0,10,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"report"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":20,"active_revision_state":null,"active_revision_id":null,"claimed_size":10,"claimed_modification_time":null,"content_sha1":null}},"name":"report.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/report.txt');
INSERT INTO nodes VALUES('vol~old','vol~docs','old.txt',0,3,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"old"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":13,"active_revision_state":null,"active_revision_id":null,"claimed_size":3,"claimed_modification_time":null,"content_sha1":null}},"name":"old.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/old.txt');
INSERT INTO nodes VALUES('vol~shared','vol~docs','shared.txt',0,4,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"shared"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":14,"active_revision_state":null,"active_revision_id":null,"claimed_size":4,"claimed_modification_time":null,"content_sha1":null}},"name":"shared.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/shared.txt');
INSERT INTO nodes VALUES('local~dir','vol~root','Offline folder',1,0,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"dir"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":"Folder","name":"Offline folder","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',1,'Offline folder');
INSERT INTO nodes VALUES('local~inner','local~dir','inner.txt',0,5,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"inner"},"parent_uid":{"volume_id":"local","link_id":"dir"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":15,"active_revision_state":null,"active_revision_id":null,"claimed_size":5,"claimed_modification_time":null,"content_sha1":null}},"name":"inner.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Offline folder/inner.txt');
INSERT INTO nodes VALUES('local~part','vol~root','movie.part',0,7,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"part"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":17,"active_revision_state":null,"active_revision_id":null,"claimed_size":7,"claimed_modification_time":null,"content_sha1":null}},"name":"movie.part","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'movie.part');
CREATE TABLE cache_entries (
cache_key TEXT PRIMARY KEY,
size_bytes INTEGER,
last_accessed INTEGER,
is_pinned INTEGER NOT NULL DEFAULT 0
, kind TEXT NOT NULL DEFAULT 'blob');
CREATE TABLE pins (
uid TEXT PRIMARY KEY,
path TEXT NOT NULL,
recursive INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE local_files (
id INTEGER PRIMARY KEY,
path TEXT NOT NULL UNIQUE,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER NOT NULL DEFAULT 0,
mtime INTEGER NOT NULL DEFAULT 0,
scan_gen INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE photos (
uid TEXT PRIMARY KEY,
capture_time INTEGER NOT NULL,
name TEXT,
ratio REAL,
thumb_state INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL
, media_type TEXT, kind INTEGER NOT NULL DEFAULT 0, favorite INTEGER NOT NULL DEFAULT 0, content_hash TEXT, main_uid TEXT, group_key TEXT, resolved_at INTEGER, latitude REAL, longitude REAL, place_id INTEGER, similar_hash INTEGER);
CREATE TABLE trash (
uid TEXT PRIMARY KEY,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER NOT NULL DEFAULT 0,
mtime INTEGER NOT NULL DEFAULT 0
, parent_uid TEXT, trashed_at INTEGER);
CREATE TABLE device (
uid TEXT PRIMARY KEY,
share_id TEXT NOT NULL,
root_uid TEXT NOT NULL,
name TEXT NOT NULL,
created INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE sync_folder (
id INTEGER PRIMARY KEY,
local_path TEXT NOT NULL UNIQUE,
remote_uid TEXT NOT NULL,
remote_share_id TEXT NOT NULL,
mode TEXT NOT NULL DEFAULT 'mirror',
state TEXT NOT NULL DEFAULT 'idle',
last_sync INTEGER NOT NULL DEFAULT 0
, pending_mode TEXT);
CREATE TABLE sync_entry (
folder_id INTEGER NOT NULL,
rel_path TEXT NOT NULL,
remote_uid TEXT,
local_mtime INTEGER NOT NULL DEFAULT 0,
local_size INTEGER NOT NULL DEFAULT 0,
remote_hash TEXT,
remote_rev TEXT, local_mtime_ns INTEGER,
PRIMARY KEY (folder_id, rel_path)
);
CREATE TABLE activity (
id INTEGER PRIMARY KEY,
time INTEGER NOT NULL,
kind TEXT NOT NULL,
target TEXT NOT NULL,
detail TEXT NOT NULL DEFAULT '',
ok INTEGER NOT NULL DEFAULT 1
);
CREATE TABLE pending_op (
id INTEGER PRIMARY KEY,
kind TEXT NOT NULL,
uid TEXT NOT NULL,
blob_path TEXT,
meta_json TEXT,
created_at INTEGER NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
next_attempt_at INTEGER NOT NULL DEFAULT 0
, parent_uid TEXT, name TEXT, claimed_at INTEGER NOT NULL DEFAULT 0, access_deferred_since INTEGER NOT NULL DEFAULT 0);
INSERT INTO pending_op VALUES(1,'mkdir','local~dir',NULL,NULL,1700000000000,0,NULL,0,'vol~root','Offline folder',4611686018427387903,0);
INSERT INTO pending_op VALUES(2,'create','local~inner','/blobs/inner','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,0,NULL,0,'local~dir','inner.txt',0,0);
INSERT INTO pending_op VALUES(3,'create','local~part','/blobs/part','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,0,NULL,8000000000000,'vol~root','movie.part',0,0);
INSERT INTO pending_op VALUES(4,'revision','vol~report','/blobs/report','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,2,'network unreachable',1700000900000,NULL,NULL,0,0);
INSERT INTO pending_op VALUES(5,'rename','vol~shared',NULL,'{"original_parent_uid":"vol~docs","original_name":"shared.txt"}',1700000000000,0,NULL,1700000700000,'vol~root','shared-moved.txt',0,1700000100000);
INSERT INTO pending_op VALUES(6,'trash','vol~old',NULL,NULL,1700000000000,0,NULL,0,NULL,NULL,0,0);
CREATE TABLE IF NOT EXISTS 'local_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
INSERT INTO local_fts_data VALUES(1,x'000000');
INSERT INTO local_fts_data VALUES(10,x'00000000000000');
CREATE TABLE IF NOT EXISTS 'local_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'local_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'local_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
INSERT INTO local_fts_config VALUES('version',4);
CREATE TABLE share_access (
root_uid TEXT PRIMARY KEY,
access TEXT NOT NULL
CHECK (access IN ('owner', 'editor', 'viewer', 'unknown'))
);
CREATE TABLE mount (
id INTEGER PRIMARY KEY,
kind TEXT NOT NULL
CHECK (kind IN ('myfiles', 'device', 'shared')),
sync_folder_id INTEGER REFERENCES sync_folder(id) ON DELETE CASCADE,
share_root_uid TEXT,
local_path TEXT NOT NULL DEFAULT '',
root_uid TEXT NOT NULL DEFAULT '',
root_share_id TEXT NOT NULL DEFAULT '',
mode TEXT NOT NULL DEFAULT 'unknown',
access TEXT NOT NULL DEFAULT 'rw'
CHECK (access IN ('rw', 'ro')),
CHECK (
(kind = 'myfiles' AND sync_folder_id IS NULL AND share_root_uid IS NULL)
OR
(kind = 'device' AND sync_folder_id IS NOT NULL AND share_root_uid IS NULL)
OR
(kind = 'shared' AND sync_folder_id IS NULL AND share_root_uid IS NOT NULL)
)
);
CREATE TABLE albums (
uid TEXT PRIMARY KEY,
name TEXT NOT NULL,
photo_count INTEGER NOT NULL DEFAULT 0,
cover_uid TEXT,
last_activity INTEGER,
shared INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL
);
CREATE TABLE album_photos (
album_uid TEXT NOT NULL,
uid TEXT NOT NULL,
capture_time INTEGER NOT NULL,
name TEXT,
media_type TEXT,
kind INTEGER NOT NULL DEFAULT 0,
ratio REAL,
thumb_state INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL,
PRIMARY KEY (album_uid, uid)
);
CREATE TABLE IF NOT EXISTS 'nodes_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
INSERT INTO nodes_fts_data VALUES(1,x'083850');
INSERT INTO nodes_fts_data VALUES(10,x'000000000108080008010101020101030101040101050101060101070101080101');
INSERT INTO nodes_fts_data VALUES(137438953473,x'000000340430206669010204010366696c0102050103696c6501020601036c657301020701036d79200102020103792066010203040808080808');
INSERT INTO nodes_fts_data VALUES(274877906945,x'0000001a0430646f6302080201010201036f6373020803010103040b');
INSERT INTO nodes_fts_data VALUES(412316860417,x'0000008b04302e747803080801010d01032f72650306010106010363732f03060101040103646f630306010102010365706f03080301010801036f637303060101030202727403080501010a0103706f7203080401010901037265700308020101070202742e03080601010b0103732f7203060101050103742e7403080701010c0202787403080901010e040b0a0a0a0b0a0a0b0b0a0a0b');
INSERT INTO nodes_fts_data VALUES(549755813889,x'0000006b04302e747804080501010a01032f6f6c0406010106010363732f04060101040103642e7404080401010902026f63040601010201036c642e04080301010801036f6373040601010302026c640408020101070103732f6f0406010105010374787404080601010b040b0a0a0b090b0a0a0a');
INSERT INTO nodes_fts_data VALUES(687194767361,x'0000008c04302e747805080801010d01032f736805060101060103617265050804010109010363732f05060101040103642e7405080701010c02026f630506010102010365642e05080601010b010368617205080301010801036f63730506010103010372656405080501010a0103732f73050601010502026861050802010107010374787405080901010e040b0a0b0a0b090b0b0a0b0a0a');
INSERT INTO nodes_fts_data VALUES(824633720833,x'00000084043020666f060809010109010364657206080d01010d0103652066060808010108010366666c06080301010302026c6906080401010402026f6c06080a01010a0103696e6506080601010601036c646506080c01010c0202696e06080501010501036e652006080701010701036f666606080201010202026c6406080b01010b040b0b0b0b0a0a0b0b0a0b0b');
INSERT INTO nodes_fts_data VALUES(962072674305,x'000000da043020666f070601010901032e747807080701011601032f696e07060101100103646572070601010d010365206607060101080202722e07080501011403012f070601010e010366666c070601010302026c69070601010402026f6c070601010a0103696e65070601010603016e07080201011101036c6465070601010c0202696e070601010501036e6520070601010703017207080401011302026e6507080301011201036f6666070601010202026c64070601010b0103722e7407080601011502022f69070601010f0103747874070808010117040a0b0a0a0a0a080a09090a090a090a090a0a090b09');
INSERT INTO nodes_fts_data VALUES(1099511627777,x'0000005c04302e706108080701010701036172740808090101090103652e70080806010106010369652e08080501010501036d6f7608080201010201036f766908080301010301037061720808080101080103766965080804010104040b0b0b0b0b0b0b');
CREATE TABLE IF NOT EXISTS 'nodes_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
INSERT INTO nodes_fts_idx VALUES(1,x'',2);
INSERT INTO nodes_fts_idx VALUES(2,x'',2);
INSERT INTO nodes_fts_idx VALUES(3,x'',2);
INSERT INTO nodes_fts_idx VALUES(4,x'',2);
INSERT INTO nodes_fts_idx VALUES(5,x'',2);
INSERT INTO nodes_fts_idx VALUES(6,x'',2);
INSERT INTO nodes_fts_idx VALUES(7,x'',2);
INSERT INTO nodes_fts_idx VALUES(8,x'',2);
CREATE TABLE IF NOT EXISTS 'nodes_fts_content'(id INTEGER PRIMARY KEY, c0, c1);
INSERT INTO nodes_fts_content VALUES(1,'My Files','');
INSERT INTO nodes_fts_content VALUES(2,'Docs','Docs');
INSERT INTO nodes_fts_content VALUES(3,'report.txt','Docs/report.txt');
INSERT INTO nodes_fts_content VALUES(4,'old.txt','Docs/old.txt');
INSERT INTO nodes_fts_content VALUES(5,'shared.txt','Docs/shared.txt');
INSERT INTO nodes_fts_content VALUES(6,'Offline folder','Offline folder');
INSERT INTO nodes_fts_content VALUES(7,'inner.txt','Offline folder/inner.txt');
INSERT INTO nodes_fts_content VALUES(8,'movie.part','movie.part');
CREATE TABLE IF NOT EXISTS 'nodes_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
INSERT INTO nodes_fts_docsize VALUES(1,x'0600');
INSERT INTO nodes_fts_docsize VALUES(2,x'0202');
INSERT INTO nodes_fts_docsize VALUES(3,x'080d');
INSERT INTO nodes_fts_docsize VALUES(4,x'050a');
INSERT INTO nodes_fts_docsize VALUES(5,x'080d');
INSERT INTO nodes_fts_docsize VALUES(6,x'0c0c');
INSERT INTO nodes_fts_docsize VALUES(7,x'0716');
INSERT INTO nodes_fts_docsize VALUES(8,x'0808');
CREATE TABLE IF NOT EXISTS 'nodes_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
INSERT INTO nodes_fts_config VALUES('version',4);
CREATE TABLE own_sealed_rev (
uid TEXT PRIMARY KEY,
revision_id TEXT NOT NULL,
sealed_at INTEGER NOT NULL
);
INSERT INTO own_sealed_rev VALUES('vol~report','rev-1',1700000000000);
PRAGMA writable_schema=ON;
INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)VALUES('table','local_fts','local_fts',0,'CREATE VIRTUAL TABLE local_fts USING fts5(
name, path, content=''local_files'', content_rowid=''id'', tokenize=''trigram''
)');
INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)VALUES('table','nodes_fts','nodes_fts',0,'CREATE VIRTUAL TABLE nodes_fts USING fts5(
name, path, tokenize=''trigram''
)');
CREATE TRIGGER mount_sync_folder_insert
AFTER INSERT ON sync_folder
BEGIN
INSERT OR IGNORE INTO mount (kind, sync_folder_id)
VALUES ('device', NEW.id);
END;
CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_photos_seq ON photos(seq);
CREATE INDEX pending_op_uid ON pending_op(uid);
CREATE INDEX pending_op_parent ON pending_op(parent_uid);
CREATE INDEX cache_entries_lru ON cache_entries(kind, last_accessed);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_local_files_name_nocase
ON local_files(name COLLATE NOCASE);
CREATE UNIQUE INDEX mount_myfiles
ON mount(kind) WHERE kind = 'myfiles';
CREATE UNIQUE INDEX mount_device
ON mount(sync_folder_id) WHERE kind = 'device';
CREATE UNIQUE INDEX mount_shared
ON mount(share_root_uid) WHERE kind = 'shared';
CREATE INDEX idx_album_photos_seq ON album_photos(album_uid, seq);
CREATE INDEX idx_album_photos_uid ON album_photos(uid);
CREATE INDEX idx_photos_favorite ON photos(favorite, seq);
CREATE INDEX idx_sync_entry_remote_uid ON sync_entry(remote_uid);
CREATE INDEX idx_local_files_scan_gen ON local_files(scan_gen);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
CREATE INDEX idx_pending_op_claimed
ON pending_op(uid) WHERE claimed_at <> 0;
CREATE INDEX idx_trash_parent ON trash(parent_uid);
CREATE INDEX idx_photos_group ON photos(group_key);
CREATE INDEX idx_photos_unresolved ON photos(resolved_at);
CREATE INDEX idx_photos_place ON photos(place_id);
PRAGMA writable_schema=OFF;
COMMIT;
+54 -1
View File
@@ -9,7 +9,7 @@ use super::Db;
use crate::Result;
/// Current schema version. Bump on every forward migration added below.
pub(super) const SCHEMA_VERSION: i64 = 35;
pub(super) const SCHEMA_VERSION: i64 = 36;
impl Db {
pub(super) fn migrate(&self) -> Result<()> {
@@ -332,6 +332,22 @@ impl Db {
tx.execute_batch(MIGRATION_V35)?;
}
}
if current < 36 {
// Same guards as V26-V35.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('nodes') WHERE name = 'lid'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_nodes: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'nodes'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_nodes && !has_column {
tx.execute_batch(MIGRATION_V36)?;
}
}
tx.execute(
"INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1)
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
@@ -1059,3 +1075,40 @@ ALTER TABLE photos ADD COLUMN similar_hash INTEGER;
const MIGRATION_V35: &str = "
ALTER TABLE trash ADD COLUMN trashed_at INTEGER;
";
/// Schema v36: every node row has a local id (`lid`), phase 2 of
/// docs/MILESTONE-3.0.0.md.
///
/// The `uid` was the row's key, so a node had no identity until Drive minted
/// one, and a create that landed moved its row to a new key. The `lid` is given
/// when the row is first written and never changes or comes back, which is what
/// `AUTOINCREMENT` adds over a plain rowid. It takes the old rowid, so the
/// search index, keyed by rowid since v15, stays valid. The `uid` becomes
/// nullable for a node that is not on Drive yet.
const MIGRATION_V36: &str = "
CREATE TABLE nodes_v36 (
lid INTEGER PRIMARY KEY AUTOINCREMENT,
uid TEXT UNIQUE,
parent_uid TEXT,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER,
mtime INTEGER NOT NULL,
revision_hash TEXT,
trashed INTEGER NOT NULL DEFAULT 0,
node_json TEXT,
listed INTEGER NOT NULL DEFAULT 0,
path TEXT
);
INSERT INTO nodes_v36
(lid, uid, parent_uid, name, is_dir, size, mtime, revision_hash, trashed, node_json, listed,
path)
SELECT rowid, uid, parent_uid, name, is_dir, size, mtime, revision_hash, trashed, node_json,
listed, path
FROM nodes;
DROP TABLE nodes;
ALTER TABLE nodes_v36 RENAME TO nodes;
CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
";
+3 -2
View File
@@ -56,8 +56,9 @@ pub use local::LocalFileHit;
pub use maintenance::{DbStats, VacuumOutcome};
pub use nodes::{PublishedSharedRoot, SearchHit, StoredNode};
pub use ops::{
AttachedBlob, FAILING_ATTEMPTS, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION,
OP_TRASH, PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts, PendingOp, RenameMeta, op_supersedes,
AttachedBlob, CreateLanding, FAILING_ATTEMPTS, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME,
OP_REVISION, OP_TRASH, PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts, PendingOp, RenameMeta,
op_supersedes,
};
pub use photos::{StoredPhoto, StoredPlace, THUMB_HAVE, THUMB_NONE, THUMB_UNKNOWN, TimelineRow};
pub use pins::PinRow;
+131 -10
View File
@@ -17,6 +17,8 @@ use super::utils::{
pub struct StoredNode {
pub node: Node,
pub listed: bool,
/// The row's local id ([`Db::lid_of`]).
pub lid: i64,
}
/// One full-text search match: the stored [`Node`] plus its mountpoint-relative
@@ -289,7 +291,8 @@ impl Db {
let uid_str = parent_uid.to_string();
let conn = self.read();
let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM nodes WHERE parent_uid = ?1 AND trashed = 0",
"SELECT COUNT(*) FROM nodes
WHERE parent_uid = ?1 AND trashed = 0 AND node_json IS NOT NULL",
params![uid_str],
|row| row.get(0),
)?;
@@ -305,9 +308,11 @@ impl Db {
pub fn node_path(&self, uid: &str) -> Result<Option<String>> {
let conn = self.read();
let exists: Option<i64> = conn
.query_row("SELECT 1 FROM nodes WHERE uid = ?1", params![uid], |r| {
r.get(0)
})
.query_row(
"SELECT 1 FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL",
params![uid],
|r| r.get(0),
)
.optional()?;
if exists.is_none() {
return Ok(None);
@@ -511,17 +516,17 @@ impl Db {
pub fn load_all(&self) -> Result<Vec<StoredNode>> {
let conn = self.read();
let mut stmt =
conn.prepare("SELECT node_json, listed FROM nodes WHERE node_json IS NOT NULL")?;
conn.prepare("SELECT node_json, listed, lid FROM nodes WHERE node_json IS NOT NULL")?;
let rows = stmt.query_map([], |row| {
let json: String = row.get(0)?;
let listed: i64 = row.get(1)?;
Ok((json, listed != 0))
Ok((json, listed != 0, row.get(2)?))
})?;
let mut out = Vec::new();
for row in rows {
let (json, listed) = row?;
let (json, listed, lid) = row?;
let node: Node = serde_json::from_str(&json)?;
out.push(StoredNode { node, listed });
out.push(StoredNode { node, listed, lid });
}
Ok(out)
}
@@ -544,6 +549,57 @@ impl Db {
}
}
/// The local id of the node stored under `uid`, or `None` when it has no
/// row. It stays the same for as long as the row exists, including when a
/// create lands and the row takes the uid Drive gave it.
pub fn lid_of(&self, uid: &str) -> Result<Option<i64>> {
let conn = self.read();
Ok(conn
.query_row("SELECT lid FROM nodes WHERE uid = ?1", params![uid], |r| {
r.get(0)
})
.optional()?)
}
/// The local id of each node, in order, giving a row to each that has none.
///
/// A new row is only a stub: it has no `node_json`, so no listing or search
/// returns it until the node is upserted, which the caller still owes. That
/// keeps this cheap enough to call with the inode lock held, which is
/// where an inode number is first needed.
pub fn lids_for(&self, nodes: &[&Node]) -> Result<Vec<i64>> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let mut lids = Vec::with_capacity(nodes.len());
{
let mut known = tx.prepare_cached("SELECT lid FROM nodes WHERE uid = ?1")?;
let mut stub = tx.prepare_cached(
"INSERT INTO nodes (uid, parent_uid, name, is_dir, mtime, trashed)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
)?;
for node in nodes {
let uid = node.uid.to_string();
let lid = match known.query_row(params![uid], |r| r.get(0)).optional()? {
Some(lid) => lid,
None => {
stub.execute(params![
uid,
node.parent_uid.as_ref().map(ToString::to_string),
node.name,
node.is_folder() as i64,
node.modification_time,
node.trashed as i64,
])?;
tx.last_insert_rowid()
}
};
lids.push(lid);
}
}
tx.commit()?;
Ok(lids)
}
/// Read the persisted incremental-sync cursor (a `DriveEventId`), if any.
/// The daemon resumes from this on restart instead of reseeding to the
/// server head, so changes made while unmounted are still applied (P2).
@@ -559,6 +615,15 @@ impl Db {
if listed != Some(1) {
return Ok(None);
}
drop(conn);
self.known_children(parent).map(Some)
}
/// The children the DB knows under `parent`, whether or not the folder's
/// listing is still marked complete. Only for when nothing better can be
/// had: a listing dropped as stale may miss or keep a child.
pub fn known_children(&self, parent: &NodeUid) -> Result<Vec<Node>> {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT node_json FROM nodes
WHERE parent_uid = ?1 AND node_json IS NOT NULL AND trashed = 0",
@@ -568,7 +633,7 @@ impl Db {
for json in rows {
out.push(serde_json::from_str(&json?)?);
}
Ok(Some(out))
Ok(out)
}
// --- Content-cache LRU index (P4) -------------------------------------
@@ -590,6 +655,60 @@ struct PriorRow {
trashed: bool,
}
/// Hand a drained placeholder's row, and the rows below it, to the real uid it
/// landed as.
///
/// Dropping the placeholder row and waiting for the server's copy of the real
/// node left a window with no row for either uid. A queued child drained in
/// that window found no authority for its parent, re-read it over the network
/// and was deferred five seconds behind a folder that had just landed
/// (`docs/BUGS.md` B120). The placeholder's own node, readdressed, stands in
/// until the server's copy is upserted over it. A row the real uid already
/// has is newer than the placeholder, so its node is kept, under the
/// placeholder's `lid`: the node was known by that one first.
pub(super) fn adopt_placeholder_row_tx(
tx: &Transaction<'_>,
local: &str,
real: &str,
) -> Result<()> {
tx.execute(
"UPDATE nodes SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real],
)?;
let row: Option<(i64, Option<String>)> = tx
.query_row(
"SELECT lid, node_json FROM nodes WHERE uid = ?1",
params![local],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
let Some((lid, json)) = row else {
return Ok(());
};
let known: Option<(i64, Option<String>)> = tx
.query_row(
"SELECT lid, node_json FROM nodes WHERE uid = ?1",
params![real],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
if let Some((known_lid, _)) = known {
tx.execute("DELETE FROM nodes WHERE lid = ?1", params![known_lid])?;
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![known_lid])?;
}
tx.execute(
"UPDATE nodes SET uid = ?2 WHERE lid = ?1",
params![lid, real],
)?;
let json = known.and_then(|(_, known_json)| known_json).or(json);
let (Some(json), Some(real)) = (json, parse_node_uid(real)) else {
return Ok(());
};
let mut node: Node = serde_json::from_str(&json)?;
node.uid = real;
upsert_node_tx(tx, &node)
}
fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
let json = serde_json::to_string(node)?;
let uid = node.uid.to_string();
@@ -741,8 +860,10 @@ fn reindex_subtree_tx(
let mut set_path = tx.prepare("UPDATE nodes SET path = ?2 WHERE rowid = ?1")?;
let mut unindex = tx.prepare("DELETE FROM nodes_fts WHERE rowid = ?1")?;
let mut index = tx.prepare(
// Not a stub row (`Db::lids_for`): it is indexed when its node is
// written.
"INSERT INTO nodes_fts (rowid, name, path)
SELECT rowid, name, ?2 FROM nodes WHERE rowid = ?1",
SELECT rowid, name, ?2 FROM nodes WHERE rowid = ?1 AND node_json IS NOT NULL",
)?;
for (rowid, path, blocked) in descendants {
set_path.execute(params![rowid, path])?;
+220 -45
View File
@@ -6,6 +6,7 @@ use rusqlite::{OptionalExtension, params};
use serde::{Deserialize, Serialize};
use super::Db;
use super::nodes::adopt_placeholder_row_tx;
use crate::Result;
/// The `kind` of a [`PendingOp`] that uploads a staged file as a new revision.
@@ -44,6 +45,12 @@ pub const OP_RENAME: &str = "rename";
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct RenameMeta {
pub original_parent_uid: String,
/// The node's name on Drive when the first of its queued renames was made,
/// kept across the renames that replace the row, like the parent. With it
/// the drain knows where the node should still be when the op lands.
/// `None` on a row queued before this field existed.
#[serde(default)]
pub original_name: Option<String>,
}
/// The `kind` of a [`PendingOp`] that trashes a node the server knows about
@@ -66,6 +73,63 @@ pub fn op_supersedes(kind: &str) -> bool {
matches!(kind, OP_REVISION | OP_RENAME | OP_TRASH)
}
/// Delete the ops that removing `uid` takes with it, returning the staged blobs
/// they held.
///
/// That is every op for `uid` or for anything queued beneath it, plus the
/// revisions of files the node tree places beneath it. A revision op carries
/// no `parent_uid`, so the queue alone cannot tell that its file sits inside
/// a removed folder: the upload outlived the folder, failed forever against a
/// trashed node, and kept its blob on disk (`docs/BUGS.md` B102).
///
/// "Queued beneath it" starts from every folder the node tree places there,
/// not only from `uid`. A create in a subfolder that already exists on Drive
/// names that subfolder as its parent, so following the queue from `uid` alone
/// missed it: a parked one kept its blob for good, and a due one would have
/// been made in the root once its folder turned out trashed (`docs/BUGS.md`
/// B116).
///
/// Other ops are taken from the node tree only for revisions. A queued rename
/// that moved a node *into* the folder has to land, or the node would stay
/// where it was on the server instead of going to the trash with the folder.
fn drop_doomed_ops(tx: &rusqlite::Transaction<'_>, uid: &str) -> Result<Vec<String>> {
const DOOMED: &str = "
WITH RECURSIVE
below(uid) AS (
SELECT ?1
UNION
SELECT n.uid FROM nodes n JOIN below b ON n.parent_uid = b.uid
),
queued(uid) AS (
SELECT ?1
UNION
SELECT p.uid FROM pending_op p
WHERE p.kind IN ('create', 'mkdir')
AND p.parent_uid IN (SELECT uid FROM below)
UNION
SELECT p.uid FROM pending_op p JOIN queued q ON p.parent_uid = q.uid
),
doomed(id) AS (
SELECT id FROM pending_op
WHERE uid IN (SELECT uid FROM queued)
OR (kind = 'revision' AND uid IN (SELECT uid FROM below))
)";
let blobs: Vec<String> = {
let mut stmt = tx.prepare(&format!(
"{DOOMED}
SELECT blob_path FROM pending_op
WHERE id IN (SELECT id FROM doomed) AND blob_path IS NOT NULL"
))?;
let rows = stmt.query_map(params![uid], |r| r.get::<_, String>(0))?;
rows.collect::<std::result::Result<Vec<_>, _>>()?
};
tx.execute(
&format!("{DOOMED} DELETE FROM pending_op WHERE id IN (SELECT id FROM doomed)"),
params![uid],
)?;
Ok(blobs)
}
/// The volume id given to a node that exists only on this machine, so far. A
/// real [`NodeUid`] is `{volume}~{link}`, so a placeholder is `local~<uuid>` and
/// round-trips through the same `Display`/parse path as any other uid.
@@ -164,6 +228,28 @@ pub struct PendingCounts {
/// is retrying at the slowest rate it ever will and has been for minutes.
pub const FAILING_ATTEMPTS: i64 = 6;
/// A queued create that has landed, for [`Db::finish_create`].
#[derive(Debug, Clone, Copy)]
pub struct CreateLanding<'a> {
/// The placeholder the node was known by.
pub local: &'a str,
/// The uid Drive gave it.
pub real: &'a str,
/// The parent and name the op held when the drain took it.
pub sent: (&'a str, &'a str),
/// The parent and name the node has on Drive. They differ from `sent`
/// when the name was taken or the folder was gone.
pub landed: (&'a str, &'a str),
}
struct CreateRow {
blob: Option<String>,
meta: Option<String>,
parent_uid: Option<String>,
name: Option<String>,
created_at: i64,
}
/// The outcome of folding freshly written bytes into a queued create.
#[derive(Debug, Clone)]
pub struct AttachedBlob {
@@ -174,6 +260,24 @@ pub struct AttachedBlob {
}
impl Db {
/// Whether a queued trash still holds `name` somewhere.
///
/// A replacing rename or a delete-then-recreate queues the trash of the old
/// node ahead of the op that wants its name. With several drain workers the
/// later op can run first and find the name still taken; this tells it to
/// wait for the trash rather than land under a conflict name.
pub fn has_pending_trash_named(&self, name: &str) -> Result<bool> {
let conn = self.read();
conn.query_row(
"SELECT EXISTS(
SELECT 1 FROM pending_op WHERE kind = ?1 AND name = ?2
)",
params![OP_TRASH, name],
|row| row.get(0),
)
.map_err(Into::into)
}
/// Whether a specific desired-state operation is still queued for a node.
pub fn has_pending_op(&self, uid: &str, kind: &str) -> Result<bool> {
let conn = self.read();
@@ -252,27 +356,9 @@ impl Db {
name: &str,
created_at: i64,
) -> Result<(i64, Vec<String>)> {
const SUBTREE: &str = "
WITH RECURSIVE doomed(uid) AS (
SELECT ?1
UNION
SELECT p.uid FROM pending_op p JOIN doomed d ON p.parent_uid = d.uid
)";
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let blobs: Vec<String> = {
let mut stmt = tx.prepare(&format!(
"{SUBTREE}
SELECT blob_path FROM pending_op
WHERE uid IN (SELECT uid FROM doomed) AND blob_path IS NOT NULL"
))?;
let rows = stmt.query_map(params![uid], |row| row.get::<_, String>(0))?;
rows.collect::<std::result::Result<Vec<_>, _>>()?
};
tx.execute(
&format!("{SUBTREE} DELETE FROM pending_op WHERE uid IN (SELECT uid FROM doomed)"),
params![uid],
)?;
let blobs = drop_doomed_ops(&tx, uid)?;
tx.execute(
"INSERT INTO pending_op
(kind, uid, parent_uid, name, blob_path, meta_json, created_at, next_attempt_at)
@@ -326,6 +412,102 @@ impl Db {
Ok(Some(AttachedBlob { id, superseded }))
}
/// Retire a queued create that has just landed as `real`, and repoint
/// everything that named its placeholder, in one transaction.
///
/// The drain uploads the blob the create held when it was picked up. A
/// write that closes while that upload is on the wire attaches a newer blob
/// to the *same row* (`attach_blob_to_create`), so deleting the row by id
/// threw that write away: the server kept the older bytes and nothing was
/// left queued to replace them. When the row no longer holds `uploaded`, it
/// becomes a revision of `real` instead, with its sidecar passed through
/// `rewrite` so it names the real node. Returns that blob and sidecar, or
/// `None` when the create is simply done.
///
/// A rename while the create is on the wire rewrites the row's target
/// ([`Db::rewrite_op_target`]), which the create no longer reads. When the
/// row names another parent or name than `landing.sent`, a rename of `real`
/// to that target is queued in the same transaction, or the move would be
/// lost with the row.
pub fn finish_create(
&self,
id: i64,
uploaded: Option<&str>,
landing: &CreateLanding<'_>,
rewrite: impl FnOnce(&str) -> Option<String>,
) -> Result<Option<(String, String)>> {
let (local, real) = (landing.local, landing.real);
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let row: Option<CreateRow> = tx
.query_row(
"SELECT blob_path, meta_json, parent_uid, name, created_at
FROM pending_op WHERE id = ?1",
params![id],
|r| {
Ok(CreateRow {
blob: r.get(0)?,
meta: r.get(1)?,
parent_uid: r.get(2)?,
name: r.get(3)?,
created_at: r.get(4)?,
})
},
)
.optional()?;
let newer = match &row {
Some(CreateRow {
blob: Some(blob),
meta: Some(meta),
..
}) if uploaded != Some(blob.as_str()) => rewrite(meta).map(|meta| (blob.clone(), meta)),
_ => None,
};
match &newer {
Some((_, meta)) => {
tx.execute(
"UPDATE pending_op
SET kind = ?2, uid = ?3, parent_uid = NULL, name = NULL, meta_json = ?4,
attempts = 0, last_error = NULL, next_attempt_at = 0
WHERE id = ?1",
params![id, OP_REVISION, real, meta],
)?;
}
None => {
tx.execute("DELETE FROM pending_op WHERE id = ?1", params![id])?;
}
}
if let Some(CreateRow {
parent_uid: Some(parent),
name: Some(name),
created_at,
..
}) = &row
&& (parent.as_str(), name.as_str()) != landing.sent
{
let meta = serde_json::to_string(&RenameMeta {
original_parent_uid: landing.landed.0.to_string(),
original_name: Some(landing.landed.1.to_string()),
})?;
tx.execute(
"DELETE FROM pending_op WHERE uid = ?1 AND kind = ?2",
params![real, OP_RENAME],
)?;
tx.execute(
"INSERT INTO pending_op (kind, uid, parent_uid, name, meta_json, created_at)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
params![OP_RENAME, real, parent, name, meta, created_at],
)?;
}
tx.execute(
"UPDATE pending_op SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real],
)?;
adopt_placeholder_row_tx(&tx, local, real)?;
tx.commit()?;
Ok(newer)
}
/// Replace the sidecar of a queued op, for a baseline that has moved under
/// it.
///
@@ -428,36 +610,20 @@ impl Db {
/// and in the user's pending count, for the life of the database.
///
/// Only `create`/`mkdir` ops carry a `parent_uid`, so for a file this
/// recursion finds nothing and costs one query.
/// recursion finds nothing and costs one query. Revisions of files the
/// node tree places beneath `uid` go too; see [`drop_doomed_ops`].
pub fn delete_ops_for_uid(&self, uid: &str) -> Result<Vec<String>> {
const SUBTREE: &str = "
WITH RECURSIVE doomed(uid) AS (
SELECT ?1
UNION
SELECT p.uid FROM pending_op p JOIN doomed d ON p.parent_uid = d.uid
)";
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let blobs: Vec<String> = {
let mut stmt = tx.prepare(&format!(
"{SUBTREE}
SELECT blob_path FROM pending_op
WHERE uid IN (SELECT uid FROM doomed) AND blob_path IS NOT NULL"
))?;
let rows = stmt.query_map(params![uid], |r| r.get::<_, String>(0))?;
rows.collect::<std::result::Result<Vec<_>, _>>()?
};
tx.execute(
&format!("{SUBTREE} DELETE FROM pending_op WHERE uid IN (SELECT uid FROM doomed)"),
params![uid],
)?;
let blobs = drop_doomed_ops(&tx, uid)?;
tx.commit()?;
Ok(blobs)
}
/// Rewrite every queued op that points at a placeholder parent, once that
/// parent has drained and has a real uid. Also moves the node rows whose
/// parent column still names the placeholder, so listings keep resolving.
/// parent column still names the placeholder, so listings keep resolving,
/// and readdresses the placeholder's own row.
pub fn remap_local_uid(&self, local: &str, real: &str) -> Result<()> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
@@ -465,11 +631,7 @@ impl Db {
"UPDATE pending_op SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real],
)?;
tx.execute(
"UPDATE nodes SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real],
)?;
tx.execute("DELETE FROM nodes WHERE uid = ?1", params![local])?;
adopt_placeholder_row_tx(&tx, local, real)?;
tx.commit()?;
Ok(())
}
@@ -765,6 +927,19 @@ impl Db {
/// drain replaces it with the uid the server assigns.
pub const LOCAL_VOLUME: &str = "local";
/// Whether op `id` is still queued. A drain worker that claimed it can find
/// it gone: the node was deleted, or a newer write superseded it, while the
/// worker was talking to the server.
pub fn op_exists(&self, id: i64) -> Result<bool> {
let conn = self.read();
let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM pending_op WHERE id = ?1",
params![id],
|row| row.get(0),
)?;
Ok(count > 0)
}
/// Drop a queued op, once its upload has actually landed.
pub fn delete_op(&self, id: i64) -> Result<()> {
let conn = self.conn.lock();
+1 -1
View File
@@ -98,7 +98,7 @@ impl Db {
pub fn effective_node_access(&self, uid: &NodeUid) -> Result<Option<Access>> {
let conn = self.read();
let exists: bool = conn.query_row(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE uid = ?1)",
"SELECT EXISTS(SELECT 1 FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL)",
[uid.to_string()],
|row| row.get(0),
)?;
+82
View File
@@ -260,6 +260,88 @@ impl Db {
Ok(())
}
/// Drop the baseline rows of `rel_path` and everything under it.
///
/// Used when a subtree leaves a mirror folder through a move on Drive: its
/// local copy is about to be removed, and rows left behind would read as
/// "deleted locally" and trash the moved nodes at their new place.
pub fn sync_entries_remove_subtree(&self, folder_id: i64, rel_path: &str) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"DELETE FROM sync_entry
WHERE folder_id = ?1
AND (rel_path = ?2 OR substr(rel_path, 1, length(?2) + 1) = ?2 || '/')",
params![folder_id, rel_path],
)?;
Ok(())
}
/// Move the baseline rows of `from_rel` and everything under it to `to_rel`
/// in folder `to_folder`, in one transaction.
///
/// Used when a move on Drive took a mirror subtree along with its local
/// copy: the rows still describe both sides correctly, only under a new
/// path, so the next pass on either folder has nothing to do. The two
/// folders may be the same one.
pub fn sync_entries_move(
&self,
from_folder: i64,
from_rel: &str,
to_folder: i64,
to_rel: &str,
) -> Result<()> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
{
let mut select = tx.prepare(
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash,
local_mtime_ns
FROM sync_entry
WHERE folder_id = ?1
AND (rel_path = ?2 OR substr(rel_path, 1, length(?2) + 1) = ?2 || '/')",
)?;
let rows = select
.query_map(params![from_folder, from_rel], |r| {
Ok(StoredSyncEntry {
rel_path: r.get(0)?,
remote_uid: r.get(1)?,
local_mtime: r.get(2)?,
local_size: r.get(3)?,
remote_rev: r.get(4)?,
remote_hash: r.get(5)?,
local_mtime_ns: r.get(6)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
let mut delete =
tx.prepare("DELETE FROM sync_entry WHERE folder_id = ?1 AND rel_path = ?2")?;
for row in &rows {
delete.execute(params![from_folder, row.rel_path])?;
}
let mut insert = tx.prepare(
"INSERT OR REPLACE INTO sync_entry
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev,
remote_hash, local_mtime_ns)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
)?;
for row in rows {
let rel = format!("{to_rel}{}", &row.rel_path[from_rel.len()..]);
insert.execute(params![
to_folder,
rel,
row.remote_uid,
row.local_mtime,
row.local_size,
row.remote_rev,
row.remote_hash,
row.local_mtime_ns,
])?;
}
}
tx.commit()?;
Ok(())
}
/// Drop the entire baseline for a folder. Used when flipping ondemand→mirror:
/// the local tree was evicted, so the old baseline is stale and would make the
/// next reconcile mistake "locally deleted" for "must re-download". Clearing it
+829
View File
@@ -788,6 +788,8 @@ fn children_if_listed_gated_on_flag() {
// Not listed yet → unknown, force a re-fetch.
assert!(db.children_if_listed(&uid("root")).unwrap().is_none());
// Still known, for a caller that has nothing better while offline.
assert_eq!(db.known_children(&uid("root")).unwrap().len(), 2);
db.set_listed(&uid("root"), true).unwrap();
let kids = db.children_if_listed(&uid("root")).unwrap().unwrap();
@@ -2458,6 +2460,20 @@ fn atomic_trash_replacement_returns_owned_blobs_after_commit() {
assert!(ops[0].blob_path.is_none());
}
#[test]
fn a_queued_trash_holds_its_name_until_it_lands() {
let db = Db::open_in_memory().unwrap();
let node = uid("held-name").to_string();
assert!(!db.has_pending_trash_named("report.txt").unwrap());
let (trash_id, _) = db.replace_ops_with_trash(&node, "report.txt", 1).unwrap();
assert!(db.has_pending_trash_named("report.txt").unwrap());
assert!(!db.has_pending_trash_named("other.txt").unwrap());
db.delete_op(trash_id).unwrap();
assert!(!db.has_pending_trash_named("report.txt").unwrap());
}
#[test]
fn failed_atomic_trash_insert_keeps_prior_revision_and_blob_ownership() {
let db = Db::open_in_memory().unwrap();
@@ -2498,6 +2514,139 @@ fn failed_atomic_trash_insert_keeps_prior_revision_and_blob_ownership() {
);
}
/// A revision op names no parent, so the queue alone cannot see that its file
/// sits inside a folder being removed. The node tree can: trashing the folder
/// has to take the uploads of the files below it along, or they fail forever
/// against a trashed node and keep their bytes on disk (B102). A rename that
/// moved a node into the folder still has to land.
#[test]
fn trashing_a_folder_drops_the_queued_revisions_of_files_below_it() {
let db = Db::open_in_memory().unwrap();
db.upsert_nodes(&[
folder("root", None, "My Files"),
folder("doomed", Some("root"), "doomed"),
folder("sub", Some("doomed"), "sub"),
file("deep", "sub", "deep.bin", 1),
file("moved", "doomed", "moved.txt", 1),
file("kept", "root", "kept.bin", 1),
])
.unwrap();
let op = |kind: &str, link: &str, blob: Option<&str>| PendingOp {
id: 0,
kind: kind.to_string(),
uid: uid(link).to_string(),
parent_uid: None,
name: None,
blob_path: blob.map(str::to_string),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
};
db.enqueue_op(&op(OP_REVISION, "deep", Some("/staging/deep")))
.unwrap();
db.enqueue_op(&op(OP_RENAME, "moved", None)).unwrap();
db.enqueue_op(&op(OP_REVISION, "kept", Some("/staging/kept")))
.unwrap();
let doomed = uid("doomed").to_string();
let (_, blobs) = db.replace_ops_with_trash(&doomed, "doomed", 2).unwrap();
assert_eq!(blobs, vec!["/staging/deep"]);
let mut left: Vec<(String, String)> = db
.pending_ops()
.unwrap()
.into_iter()
.map(|op| (op.kind, op.uid))
.collect();
left.sort();
assert_eq!(
left,
vec![
(OP_RENAME.to_string(), uid("moved").to_string()),
(OP_REVISION.to_string(), uid("kept").to_string()),
(OP_TRASH.to_string(), doomed.clone()),
]
);
// Deleting outright follows the same tree.
db.enqueue_op(&op(OP_REVISION, "deep", Some("/staging/deep2")))
.unwrap();
assert_eq!(
db.delete_ops_for_uid(&uid("sub").to_string()).unwrap(),
vec!["/staging/deep2"]
);
}
/// A create in a subfolder that already exists on Drive names that subfolder,
/// not the folder being trashed, as its parent. It has to go with the folder
/// all the same, along with whatever is queued inside it, or a parked one
/// keeps its bytes for good and a due one is made in the root (B116).
#[test]
fn trashing_a_folder_drops_the_queued_creates_in_its_subfolders() {
let db = Db::open_in_memory().unwrap();
db.upsert_nodes(&[
folder("root", None, "My Files"),
folder("doomed", Some("root"), "doomed"),
folder("sub", Some("doomed"), "sub"),
folder("other", Some("root"), "other"),
])
.unwrap();
let op = |kind: &str, node: &str, parent: &str, blob: Option<&str>| PendingOp {
id: 0,
kind: kind.to_string(),
uid: node.to_string(),
parent_uid: Some(parent.to_string()),
name: Some(format!("{node}.bin")),
blob_path: blob.map(str::to_string),
meta_json: None,
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
};
let sub = uid("sub").to_string();
db.enqueue_op(&op(OP_CREATE, "local~deep", &sub, Some("/staging/deep")))
.unwrap();
db.enqueue_op(&op(OP_MKDIR, "local~dir", &sub, None))
.unwrap();
db.enqueue_op(&op(
OP_CREATE,
"local~deeper",
"local~dir",
Some("/staging/deeper"),
))
.unwrap();
db.enqueue_op(&op(
OP_CREATE,
"local~kept",
&uid("other").to_string(),
Some("/staging/kept"),
))
.unwrap();
let doomed = uid("doomed").to_string();
let (_, mut blobs) = db.replace_ops_with_trash(&doomed, "doomed", 2).unwrap();
blobs.sort();
assert_eq!(blobs, vec!["/staging/deep", "/staging/deeper"]);
let mut left: Vec<(String, String)> = db
.pending_ops()
.unwrap()
.into_iter()
.map(|op| (op.kind, op.uid))
.collect();
left.sort();
assert_eq!(
left,
vec![
(OP_CREATE.to_string(), "local~kept".to_string()),
(OP_TRASH.to_string(), doomed),
]
);
}
/// Deleting a folder that was created offline must take the ops queued
/// underneath it with it: they name a placeholder parent that will now never
/// become real, so nothing could ever drain them and nothing is left to
@@ -2699,6 +2848,165 @@ fn attaching_to_an_already_drained_create_reports_it_is_gone() {
assert!(out.is_none());
}
#[test]
fn a_write_attached_while_its_create_uploads_becomes_a_revision() {
let db = Db::open_in_memory().unwrap();
let local = "local~abc";
let real = uid("real").to_string();
let parent = uid("parent").to_string();
let create = |blob: Option<&str>| PendingOp {
id: 0,
kind: OP_CREATE.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("parent").to_string()),
name: Some("new.txt".to_string()),
blob_path: blob.map(str::to_string),
meta_json: blob.map(|_| "{\"uid\":\"local~abc\"}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
};
let (id, _) = db.enqueue_op(&create(None)).unwrap();
// The drain picked the create up with no blob; the file's first write
// closes while the empty upload is on the wire.
db.attach_blob_to_create(local, "/staging/written", "{\"uid\":\"local~abc\"}")
.unwrap()
.expect("create is still queued");
let newer = db
.finish_create(
id,
None,
&landing(local, &real, &parent, "new.txt"),
|json| Some(json.replace("local~abc", "vol~real")),
)
.unwrap();
assert_eq!(
newer,
Some((
"/staging/written".to_string(),
"{\"uid\":\"vol~real\"}".to_string()
))
);
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1, "the write is still queued");
assert_eq!(ops[0].kind, OP_REVISION);
assert_eq!(ops[0].uid, real);
assert_eq!(ops[0].blob_path.as_deref(), Some("/staging/written"));
assert_eq!(ops[0].parent_uid, None);
}
#[test]
fn a_create_whose_blob_landed_is_retired() {
let db = Db::open_in_memory().unwrap();
let local = "local~abc";
let real = uid("real").to_string();
let parent = uid("parent").to_string();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_CREATE.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("parent").to_string()),
name: Some("new.txt".to_string()),
blob_path: Some("/staging/sent".to_string()),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
db.enqueue_op(&PendingOp {
id: 0,
kind: OP_CREATE.to_string(),
uid: "local~child".to_string(),
parent_uid: Some(local.to_string()),
name: Some("inside.txt".to_string()),
blob_path: None,
meta_json: None,
created_at: 2,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
let newer = db
.finish_create(
id,
Some("/staging/sent"),
&landing(local, &real, &parent, "new.txt"),
|_| panic!("an unchanged create has no sidecar to rewrite"),
)
.unwrap();
assert_eq!(newer, None);
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1, "only the child is left");
assert_eq!(ops[0].parent_uid.as_deref(), Some(real.as_str()));
}
/// The parent and name a create was sent with, landing where it was sent.
fn landing<'a>(local: &'a str, real: &'a str, parent: &'a str, name: &'a str) -> CreateLanding<'a> {
CreateLanding {
local,
real,
sent: (parent, name),
landed: (parent, name),
}
}
#[test]
fn a_rename_made_while_its_create_uploads_is_queued_for_the_landed_node() {
let db = Db::open_in_memory().unwrap();
let local = "local~abc";
let real = uid("real").to_string();
let parent = uid("parent").to_string();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_CREATE.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("parent").to_string()),
name: Some("new.txt".to_string()),
blob_path: Some("/staging/sent".to_string()),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
// The drain took the create; the user moves the file into a folder that
// is itself still queued.
db.rewrite_op_target(local, "local~dir", "moved.txt")
.unwrap();
let mut landed = landing(local, &real, &parent, "new.txt");
landed.landed.1 = "new (conflict).txt";
db.finish_create(id, Some("/staging/sent"), &landed, |_| None)
.unwrap();
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1, "the move outlives the create");
assert_eq!(ops[0].kind, OP_RENAME);
assert_eq!(ops[0].uid, real);
assert_eq!(ops[0].parent_uid.as_deref(), Some("local~dir"));
assert_eq!(ops[0].name.as_deref(), Some("moved.txt"));
let meta: RenameMeta = serde_json::from_str(ops[0].meta_json.as_deref().unwrap()).unwrap();
assert_eq!(meta.original_parent_uid, uid("parent").to_string());
assert_eq!(meta.original_name.as_deref(), Some("new (conflict).txt"));
// The folder lands next and the move follows it.
db.remap_local_uid("local~dir", &uid("dir").to_string())
.unwrap();
let ops = db.pending_ops().unwrap();
assert_eq!(ops[0].parent_uid, Some(uid("dir").to_string()));
}
#[test]
fn draining_a_folder_repoints_its_queued_children() {
let db = Db::open_in_memory().unwrap();
@@ -2728,6 +3036,414 @@ fn draining_a_folder_repoints_its_queued_children() {
assert_eq!(ops[0].parent_uid.as_deref(), Some(real_dir.as_str()));
}
/// B120: a child queued inside a folder that has just landed is checked
/// against the folder's real uid before the server's copy of it arrives.
#[test]
fn a_landed_folder_keeps_a_row_until_the_server_copy_replaces_it() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
let local = NodeUid::new(VolumeId::from("local"), LinkId::from("dir"));
let mut placeholder = folder("dir", Some("root"), "New folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
let mut child = file("child", "root", "inside.txt", 1);
child.parent_uid = Some(local.clone());
db.upsert_node(&child).unwrap();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_MKDIR.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("root").to_string()),
name: Some("New folder".to_string()),
blob_path: None,
meta_json: None,
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
db.finish_create(
id,
None,
&landing(
&local.to_string(),
&uid("real").to_string(),
&uid("root").to_string(),
"New folder",
),
|_| None,
)
.unwrap();
assert!(db.node_by_uid(&local.to_string()).unwrap().is_none());
let landed = db
.node_by_uid(&uid("real").to_string())
.unwrap()
.expect("the landed folder has a row before the server's copy is read");
assert_eq!(landed.uid, uid("real"));
assert_eq!(landed.name, "New folder");
assert_eq!(
db.effective_node_access(&uid("child")).unwrap(),
Some(crate::Access::Owner),
"the child's authority resolves without a network re-read"
);
let names: Vec<_> = db
.known_children(&uid("real"))
.unwrap()
.into_iter()
.map(|n| n.name)
.collect();
assert_eq!(names, ["inside.txt"]);
let found = db.search("inside", 10).unwrap();
assert_eq!(found.len(), 1);
assert_eq!(found[0].path, "New folder/inside.txt");
}
#[test]
fn a_landed_placeholder_never_overwrites_the_server_copy() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
let local = NodeUid::new(VolumeId::from("local"), LinkId::from("dir"));
let mut placeholder = folder("dir", Some("root"), "New folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
// The event stream reported the folder before the drain got round to it.
db.upsert_node(&folder("real", Some("root"), "New folder (1)"))
.unwrap();
db.remap_local_uid(&local.to_string(), &uid("real").to_string())
.unwrap();
assert!(db.node_by_uid(&local.to_string()).unwrap().is_none());
assert_eq!(
db.node_by_uid(&uid("real").to_string())
.unwrap()
.map(|n| n.name),
Some("New folder (1)".to_string())
);
}
#[test]
fn a_node_keeps_its_lid_across_updates_and_a_new_one_never_reuses_it() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
db.upsert_node(&file("a", "root", "a.txt", 1)).unwrap();
db.upsert_node(&file("b", "root", "b.txt", 1)).unwrap();
let a = db.lid_of(&uid("a").to_string()).unwrap().unwrap();
let b = db.lid_of(&uid("b").to_string()).unwrap().unwrap();
db.upsert_node(&file("a", "root", "renamed.txt", 2))
.unwrap();
assert_eq!(db.lid_of(&uid("a").to_string()).unwrap(), Some(a));
db.delete_node(&uid("b")).unwrap();
db.upsert_node(&file("c", "root", "c.txt", 1)).unwrap();
let c = db.lid_of(&uid("c").to_string()).unwrap().unwrap();
assert!(c > b, "lid {c} reuses one a deleted row had");
}
#[test]
fn a_lid_handed_out_before_the_node_is_stored_reads_as_no_node() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
db.set_share_access(&uid("root"), crate::Access::Owner)
.unwrap();
let stub = file("stub", "root", "stub.txt", 1);
let [lid] = db.lids_for(&[&stub]).unwrap()[..] else {
panic!("one lid per node");
};
assert_eq!(db.lid_of(&uid("stub").to_string()).unwrap(), Some(lid));
assert_eq!(db.lids_for(&[&stub]).unwrap(), vec![lid]);
assert!(db.node_by_uid(&uid("stub").to_string()).unwrap().is_none());
assert!(db.effective_node_access(&uid("stub")).unwrap().is_none());
assert!(db.node_path(&uid("stub").to_string()).unwrap().is_none());
assert!(!db.has_children(&uid("root")).unwrap());
assert!(db.search("stub", 10).unwrap().is_empty());
assert!(db.load_all().unwrap().iter().all(|n| n.lid != lid));
db.upsert_node(&stub).unwrap();
assert_eq!(db.lid_of(&uid("stub").to_string()).unwrap(), Some(lid));
assert!(db.has_children(&uid("root")).unwrap());
}
#[test]
fn a_landed_placeholder_keeps_its_lid() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
for known in [false, true] {
let local = NodeUid::new(VolumeId::from("local"), LinkId::from(format!("{known}")));
let real = uid(&format!("real-{known}"));
let mut placeholder = folder("dir", Some("root"), "New folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
let lid = db.lid_of(&local.to_string()).unwrap().unwrap();
if known {
// The event stream reported the folder before the drain got round
// to it.
let mut listed = folder("x", Some("root"), "New folder (1)");
listed.uid = real.clone();
db.upsert_node(&listed).unwrap();
}
db.remap_local_uid(&local.to_string(), &real.to_string())
.unwrap();
assert_eq!(db.lid_of(&local.to_string()).unwrap(), None);
assert_eq!(db.lid_of(&real.to_string()).unwrap(), Some(lid));
let name = if known {
"New folder (1)"
} else {
"New folder"
};
assert_eq!(
db.node_by_uid(&real.to_string()).unwrap().map(|n| n.name),
Some(name.to_string())
);
assert_eq!(db.search(name, 10).unwrap().len(), 1, "{name}");
}
}
#[test]
fn migration_v36_gives_every_node_its_rowid_as_lid() {
let path = std::env::temp_dir().join(format!(
"pdfs-db-v35-fixture-{}-{}.db",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
let local = NodeUid::new(VolumeId::from("local"), LinkId::from("dir"));
{
let db = Db::open(&path).unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
let mut placeholder = folder("dir", Some("root"), "Offline folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
db.upsert_node(&file("gone", "root", "gone.txt", 1))
.unwrap();
db.upsert_node(&file("kept", "root", "kept-report.txt", 1))
.unwrap();
db.delete_node(&uid("gone")).unwrap();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_MKDIR.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("root").to_string()),
name: Some("Offline folder".to_string()),
blob_path: None,
meta_json: None,
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
db.record_op_failure(id, "network unreachable", 5_000)
.unwrap();
}
{
// Put the file back in the state a released V35 database was in:
// keyed by uid, with the rowids of the rows it held.
let conn = rusqlite::Connection::open(&path).unwrap();
conn.execute_batch(
"CREATE TABLE nodes_v35 (
uid TEXT PRIMARY KEY, parent_uid TEXT, name TEXT NOT NULL,
is_dir INTEGER NOT NULL, size INTEGER, mtime INTEGER NOT NULL,
revision_hash TEXT, trashed INTEGER NOT NULL DEFAULT 0, node_json TEXT,
listed INTEGER NOT NULL DEFAULT 0, path TEXT
);
INSERT INTO nodes_v35 (rowid, uid, parent_uid, name, is_dir, size, mtime,
revision_hash, trashed, node_json, listed, path)
SELECT lid, uid, parent_uid, name, is_dir, size, mtime, revision_hash,
trashed, node_json, listed, path FROM nodes;
DROP TABLE nodes;
ALTER TABLE nodes_v35 RENAME TO nodes;
CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
DELETE FROM sqlite_sequence WHERE name = 'nodes';
UPDATE sync_state SET value = '35' WHERE key = 'schema_version';",
)
.unwrap();
}
let rowids: Vec<(String, i64)> = {
let conn = rusqlite::Connection::open(&path).unwrap();
let mut stmt = conn
.prepare("SELECT uid, rowid FROM nodes ORDER BY rowid")
.unwrap();
stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))
.unwrap()
.map(|row| row.unwrap())
.collect()
};
assert_eq!(rowids.len(), 3);
let db = Db::open(&path).unwrap();
for (uid, rowid) in &rowids {
assert_eq!(db.lid_of(uid).unwrap(), Some(*rowid), "{uid}");
}
let found = db.search("report", 10).unwrap();
assert_eq!(found.len(), 1, "the search index still finds the node");
assert_eq!(found[0].node.uid, uid("kept"));
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1);
assert_eq!(ops[0].uid, local.to_string());
assert_eq!(ops[0].attempts, 1);
assert_eq!(ops[0].last_error.as_deref(), Some("network unreachable"));
db.upsert_node(&file("new", "root", "new.txt", 1)).unwrap();
let newest = db.lid_of(&uid("new").to_string()).unwrap().unwrap();
assert!(newest > rowids.last().unwrap().1);
drop(db);
let _ = std::fs::remove_file(&path);
}
/// A database as 2.8.2 wrote it (`sqlite3 .dump`, schema 35), with queued ops
/// in every state the drain leaves them in: due, parked, failing, deferred for
/// access, and claimed by a worker when the daemon stopped. The creates below
/// a folder made offline still name their parent by its `local~` placeholder.
#[test]
fn a_2_8_database_migrates_with_every_node_and_op_intact() {
let path = std::env::temp_dir().join(format!(
"pdfs-db-2.8.2-{}-{}.db",
std::process::id(),
now_test_id()
));
let _ = std::fs::remove_file(&path);
rusqlite::Connection::open(&path)
.unwrap()
.execute_batch(include_str!("fixtures/v2.8.2.sql"))
.unwrap();
let (nodes, ops) = migrates_intact(&path);
assert_eq!((nodes, ops), (8, 6));
let db = Db::open(&path).unwrap();
let queued = db.pending_ops().unwrap();
let inner = queued.iter().find(|op| op.uid == "local~inner").unwrap();
assert_eq!(inner.parent_uid.as_deref(), Some("local~dir"));
let parked = db.parked_create_ops().unwrap();
assert_eq!(parked.len(), 1);
assert_eq!(parked[0].name.as_deref(), Some("movie.part"));
drop(db);
let _ = std::fs::remove_file(&path);
}
/// The same migration over a copy of a database a released build wrote, which
/// the fixtures above cannot stand in for: years of rows, every kind of op, and
/// whatever shapes real use left behind. The copy is taken with `VACUUM INTO`
/// over a read-only connection, so the database named is never written.
///
/// `PDFS_MIGRATE_DB=~/.local/state/proton-drive-linux/cache.db cargo test -p
/// pdfs-core a_copy_of_a_real_database -- --ignored`
#[test]
#[ignore = "needs a real database: set PDFS_MIGRATE_DB"]
fn a_copy_of_a_real_database_migrates_with_every_node_and_op_intact() {
let source = std::env::var("PDFS_MIGRATE_DB").expect("set PDFS_MIGRATE_DB");
let path = std::env::temp_dir().join(format!("pdfs-db-real-{}.db", std::process::id()));
let _ = std::fs::remove_file(&path);
rusqlite::Connection::open_with_flags(&source, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY)
.unwrap()
.execute("VACUUM INTO ?1", rusqlite::params![path.to_str().unwrap()])
.unwrap();
let (nodes, ops) = migrates_intact(&path);
eprintln!("{nodes} nodes, {ops} queued ops");
let _ = std::fs::remove_file(&path);
}
/// Opens the database at `path` and checks that the migration kept every node
/// on its row id and every queued op as it was. Returns how many of each.
fn migrates_intact(path: &std::path::Path) -> (usize, usize) {
let (version, nodes, ops, sealed) = {
let conn = rusqlite::Connection::open(path).unwrap();
let version: String = conn
.query_row(
"SELECT value FROM sync_state WHERE key = 'schema_version'",
[],
|r| r.get(0),
)
.unwrap();
let nodes: Vec<(String, i64)> = conn
.prepare("SELECT uid, rowid FROM nodes")
.unwrap()
.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))
.unwrap()
.map(|row| row.unwrap())
.collect();
let ops = rows(&conn, QUEUED);
let sealed = rows(&conn, "SELECT * FROM own_sealed_rev ORDER BY uid");
(version, nodes, ops, sealed)
};
let db = Db::open(path).unwrap();
{
let conn = rusqlite::Connection::open(path).unwrap();
assert_eq!(
rows(&conn, QUEUED),
ops,
"every queued op survives as it was, parked, failing or deferred"
);
// A claim names a drain worker of the daemon that wrote the file.
let claimed: i64 = conn
.query_row(
"SELECT COUNT(*) FROM pending_op WHERE claimed_at <> 0",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(claimed, 0, "a stopped daemon's claims are released");
assert_eq!(
rows(&conn, "SELECT * FROM own_sealed_rev ORDER BY uid"),
sealed
);
}
if version.parse::<i64>().unwrap() < 36 {
for (uid, rowid) in &nodes {
assert_eq!(db.lid_of(uid).unwrap(), Some(*rowid), "{uid}");
}
} else {
for (uid, _) in &nodes {
assert!(db.lid_of(uid).unwrap().is_some(), "{uid}");
}
}
drop(db);
let db = Db::open(path).unwrap();
let conn = rusqlite::Connection::open(path).unwrap();
let count: i64 = conn
.query_row("SELECT COUNT(*) FROM nodes", [], |r| r.get(0))
.unwrap();
assert_eq!(count as usize, nodes.len(), "no node is lost or doubled");
let version: String = conn
.query_row(
"SELECT value FROM sync_state WHERE key = 'schema_version'",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(version, SCHEMA_VERSION.to_string());
drop(db);
(nodes.len(), ops.len())
}
/// Every column of the queue but the drain's claim.
const QUEUED: &str = "SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
attempts, last_error, next_attempt_at, access_deferred_since
FROM pending_op ORDER BY id";
/// Every row `query` returns, every column.
fn rows(conn: &rusqlite::Connection, query: &str) -> Vec<Vec<rusqlite::types::Value>> {
let mut stmt = conn.prepare(query).unwrap();
let columns = stmt.column_count();
stmt.query_map([], |r| (0..columns).map(|i| r.get(i)).collect())
.unwrap()
.map(|row| row.unwrap())
.collect()
}
#[test]
fn a_failed_op_stays_queued_with_backoff() {
let db = Db::open_in_memory().unwrap();
@@ -2920,6 +3636,94 @@ fn clearing_a_stale_pending_mode_preserves_the_newer_intent() {
assert_eq!(db.sync_folder_get(id).unwrap().unwrap().pending_mode, None);
}
fn baseline_row(rel: &str) -> StoredSyncEntry {
StoredSyncEntry {
rel_path: rel.into(),
remote_uid: Some(format!("vol~{rel}")),
local_mtime: 1_700_000_000,
local_mtime_ns: Some(1_700_000_000_000_000_001),
local_size: 7,
remote_rev: Some("1700000000".into()),
remote_hash: Some("7".into()),
}
}
#[test]
fn moving_a_baseline_subtree_takes_its_rows_and_leaves_name_prefixed_siblings() {
let db = Db::open_in_memory().unwrap();
let from = db.sync_folder_add("/home/me/A", "vol~a", "share").unwrap();
let to = db.sync_folder_add("/home/me/B", "vol~b", "share").unwrap();
for rel in [
"docs",
"docs/x.txt",
"docs/sub/y.txt",
"docsextra.txt",
"other",
] {
db.sync_entry_upsert(from, &baseline_row(rel)).unwrap();
}
db.sync_entry_upsert(to, &baseline_row("inbox")).unwrap();
db.sync_entries_move(from, "docs", to, "inbox/docs")
.unwrap();
let left = db.sync_entries(from).unwrap();
let mut left: Vec<_> = left.keys().cloned().collect();
left.sort();
assert_eq!(left, ["docsextra.txt", "other"]);
let moved = db.sync_entries(to).unwrap();
let mut keys: Vec<_> = moved.keys().cloned().collect();
keys.sort();
assert_eq!(
keys,
[
"inbox",
"inbox/docs",
"inbox/docs/sub/y.txt",
"inbox/docs/x.txt"
]
);
// Only the path changes: the row still describes the same node on both sides.
assert_eq!(
moved["inbox/docs/x.txt"],
StoredSyncEntry {
rel_path: "inbox/docs/x.txt".into(),
..baseline_row("docs/x.txt")
}
);
}
#[test]
fn moving_a_baseline_subtree_within_one_folder_renames_it() {
let db = Db::open_in_memory().unwrap();
let folder = db.sync_folder_add("/home/me/A", "vol~a", "share").unwrap();
for rel in ["old", "old/f.txt", "new"] {
db.sync_entry_upsert(folder, &baseline_row(rel)).unwrap();
}
db.sync_entries_move(folder, "old", folder, "new/old")
.unwrap();
let mut keys: Vec<_> = db.sync_entries(folder).unwrap().into_keys().collect();
keys.sort();
assert_eq!(keys, ["new", "new/old", "new/old/f.txt"]);
}
#[test]
fn removing_a_baseline_subtree_keeps_name_prefixed_siblings() {
let db = Db::open_in_memory().unwrap();
let folder = db.sync_folder_add("/home/me/A", "vol~a", "share").unwrap();
for rel in ["docs", "docs/x.txt", "docsextra.txt"] {
db.sync_entry_upsert(folder, &baseline_row(rel)).unwrap();
}
db.sync_entries_remove_subtree(folder, "docs").unwrap();
let keys: Vec<_> = db.sync_entries(folder).unwrap().into_keys().collect();
assert_eq!(keys, ["docsextra.txt"]);
}
#[test]
fn mirror_uid_lookup_covers_the_root_and_synced_descendants_only() {
let db = Db::open_in_memory().unwrap();
@@ -4963,3 +5767,28 @@ fn retry_all_touches_only_ops_that_have_failed() {
assert_eq!(due(debounced), 7_000);
assert_eq!(due(parked), ops::PARK_UNTIL);
}
#[test]
fn a_queued_op_stops_existing_once_the_node_is_deleted() {
let db = Db::open_in_memory().unwrap();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_REVISION.to_string(),
uid: uid("journal").to_string(),
parent_uid: None,
name: None,
blob_path: Some("/staging/journal".to_string()),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
assert!(db.op_exists(id).unwrap());
let blobs = db.delete_ops_for_uid(&uid("journal").to_string()).unwrap();
assert_eq!(blobs, vec!["/staging/journal"]);
assert!(!db.op_exists(id).unwrap());
}
+10 -3
View File
@@ -55,9 +55,10 @@ fn systemctl_query(args: &[&str]) -> bool {
/// keyring checks). `--now` is a no-op on an active unit, so the daemon would
/// keep sleeping and the mount would not appear until the next poll fired.
/// Restarting makes the new session take effect immediately, and starts the
/// unit when it is not running at all.
/// unit when it is not running at all. `reenable` for the reason given on
/// [`enable`].
pub fn enable_start() -> bool {
let enabled = systemctl(&["enable"]);
let enabled = systemctl(&["reenable"]);
systemctl(&["restart"]) && enabled
}
@@ -97,8 +98,14 @@ pub fn is_enabled() -> bool {
/// Enable the service so it auto-starts on future logins, without necessarily
/// starting it now beyond what `enable` does. Used by the Settings "Start on
/// login" switch; pairs with [`disable`](disable).
///
/// `reenable` rather than `enable`: older installs were enabled
/// into `default.target.wants`, and a plain `enable` would add the new
/// `graphical-session.target.wants` link while leaving the old one to start the
/// unit too early. `reenable` drops every existing link and recreates them from
/// the current `[Install]` section.
pub fn enable() -> bool {
systemctl(&["enable"])
systemctl(&["reenable"])
}
/// Disable auto-start without stopping the running mount. Used by the Settings
+4
View File
@@ -15,6 +15,7 @@ zip.workspace = true
sha1.workspace = true
proton-drive-rs.workspace = true
futures.workspace = true
async-trait.workspace = true
tokio.workspace = true
tracing.workspace = true
fuser.workspace = true
@@ -24,3 +25,6 @@ base64.workspace = true
image.workspace = true
notify = "8"
parking_lot.workspace = true
[dev-dependencies]
tracing-subscriber.workspace = true
+102 -17
View File
@@ -9,7 +9,7 @@ use crate::shutdown::Shutdown;
///
/// Returns whether this mount actually held the node, so the caller can evict
/// its content blob once if *any* mount did.
fn forget_and_notify(st: &mut State, notify: &mut NotifyBatch, uid: &NodeUid) -> bool {
pub(crate) fn forget_and_notify(st: &mut State, notify: &mut NotifyBatch, uid: &NodeUid) -> bool {
// Capture the inode before `forget` clears the uid mapping.
let child = st.by_uid.get(uid).copied();
let Some((parent, name)) = st.forget(uid) else {
@@ -97,15 +97,13 @@ fn event_serializes_shared_publication(event: &DriveEvent, own_volume: &VolumeId
/// entry forces the next `lookup`/`readdir` to re-enumerate from the remote, so
/// most events only need to invalidate listings rather than re-fetch eagerly.
///
/// Applied to **every** mounted inode space, not just the primary one. This task
/// is per-daemon while node state is per-mount, and a `DriveEvent` names a uid,
/// not a mount — so a file trashed from another device has to be withdrawn from
/// whichever of our sessions is showing it, which for anything under a sync
/// folder is a [`Core::fork_state`] fork rather than `core.state`. Reaching for
/// the primary state alone left forks serving a deleted file indefinitely, the
/// read-side twin of `docs/BUGS.md` B74. Inode numbers are per-mount, so each
/// mount must be notified through its **own** channel; that pairing is what
/// [`Core::for_each_mount`] exists to preserve.
/// Applied to the tree every mount shares, and told to **every** mounted
/// session. A `DriveEvent` names a uid, not a mount, so a file trashed from
/// another device has to be withdrawn from whichever of our sessions is showing
/// it — which for anything under a sync folder is a [`Core::fork_state`] fork.
/// Telling the primary session alone left forks serving a deleted file, the
/// read-side twin of `docs/BUGS.md` B74. [`Core::for_each_mount`] sends what
/// the kernel needs to hear to every session.
fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdfs_core::Result<()> {
let foreign_delete = is_foreign_node_delete(event, &core.primary_root_uid.volume_id);
let serializes_shared_publication =
@@ -233,6 +231,48 @@ fn acknowledge_applied_event(
Ok(id)
}
/// How many applied node-update ids [`RecentEvents`] remembers. Far more than a
/// replay can reach back: one comes from a batch whose last event was not its
/// newest, so it repeats at most what that batch held.
const RECENT_EVENTS: usize = 4096;
/// The node updates applied lately, by event id, so one that comes round again
/// is skipped rather than applied twice.
///
/// The feed can deliver the same event twice in a batch, and a batch whose
/// events come out of order leaves the cursor short of its newest event, so the
/// next poll replays what lay past it. An update applied twice is not harmless:
/// the first copy of our own change claims the echo [`Core::note_self_change`]
/// recorded, and the second, unclaimed, reads as someone else's edit and evicts
/// the bytes a partial write is about to gap-fill from (docs/BUGS.md B123).
///
/// Only node updates: a delete is idempotent, and a continuity loss names the
/// cursor it lost, which is an id already applied.
#[derive(Default)]
struct RecentEvents {
ids: HashSet<DriveEventId>,
order: VecDeque<DriveEventId>,
}
impl RecentEvents {
/// Record `event` as applied, reporting whether it was already.
fn repeat(&mut self, event: &DriveEvent) -> bool {
let DriveEvent::NodeUpdated { id, .. } = event else {
return false;
};
if !self.ids.insert(id.clone()) {
return true;
}
self.order.push_back(id.clone());
if self.order.len() > RECENT_EVENTS
&& let Some(old) = self.order.pop_front()
{
self.ids.remove(&old);
}
false
}
}
/// The folders one batch of events changed, collected so the listings are
/// dropped once at the end of the batch rather than once per event.
///
@@ -320,10 +360,10 @@ fn flush_dirty_parents(core: &Core, dirty: &DirtyParents) {
/// only on fatal error.
///
/// Takes the whole `Core` rather than the primary mount's pieces: an event has
/// to be applied to every mounted inode space, and the registry that enumerates
/// them lives on the `Core` (see [`apply_event`]).
/// to reach every mounted session, and the registry that enumerates them lives
/// on the `Core` (see [`apply_event`]).
pub(super) async fn run_event_sync(
client: ProtonDriveClient,
client: Arc<dyn DriveApi>,
scope: DriveEventScopeId,
core: Core,
) {
@@ -377,10 +417,12 @@ pub(super) async fn run_event_sync(
};
info!(?cursor, "event sync started");
let mut recent = RecentEvents::default();
loop {
tokio::time::sleep(POLL_INTERVAL).await;
let events = match client.enumerate_events(&scope, cursor.as_ref()).await {
core.next_poll(POLL_INTERVAL).await;
let events = match link::bounded(client.enumerate_events(&scope, cursor.as_ref())).await {
Ok(events) => events,
Err(e) if core.lost_link(&e, "poll events") => continue,
Err(e) => {
warn!(error = %e, "event poll failed; retrying after interval");
continue;
@@ -396,6 +438,11 @@ pub(super) async fn run_event_sync(
// did apply.
let mut dirty = DirtyParents::default();
for event in &events {
if recent.repeat(event) {
debug!(id = %event.id().as_str(), "skipping an event already applied");
cursor = Some(event.id().clone());
continue;
}
// Converge the SDK's own caches (folder keys, entity cache) on the
// server before applying the event to our tree. Without this, a node
// re-keyed/moved by another client keeps a stale key in the SDK for
@@ -471,9 +518,10 @@ pub(super) async fn run_photos_event_sync(core: Core) {
info!(?cursor, "photo event sync started");
loop {
tokio::time::sleep(POLL_INTERVAL).await;
let events = match photos.enumerate_events(&scope, cursor.as_ref()).await {
core.next_poll(POLL_INTERVAL).await;
let events = match link::bounded(photos.enumerate_events(&scope, cursor.as_ref())).await {
Ok(events) => events,
Err(error) if core.lost_link(&error, "poll photo events") => continue,
Err(error) => {
warn!(%error, "photo event poll failed; retrying after interval");
continue;
@@ -715,6 +763,43 @@ mod tests {
}
}
#[test]
fn a_node_update_seen_again_is_skipped_but_other_events_are_not() {
let update = |id: &str| DriveEvent::NodeUpdated {
id: DriveEventId::from(id),
node_uid: uid("file"),
parent_node_uid: None,
is_trashed: false,
is_shared: false,
};
let mut recent = RecentEvents::default();
assert!(!recent.repeat(&update("e1")));
assert!(!recent.repeat(&update("e2")));
assert!(recent.repeat(&update("e1")), "a duplicate or replay");
let lost = DriveEvent::ContinuityLost {
id: DriveEventId::from("e2"),
};
assert!(!recent.repeat(&lost), "names the cursor, which was applied");
}
#[test]
fn recent_events_forget_the_oldest_past_their_bound() {
let update = |n: usize| DriveEvent::NodeUpdated {
id: DriveEventId::from(format!("e{n}")),
node_uid: uid("file"),
parent_node_uid: None,
is_trashed: false,
is_shared: false,
};
let mut recent = RecentEvents::default();
for n in 0..=RECENT_EVENTS {
assert!(!recent.repeat(&update(n)));
}
assert_eq!(recent.ids.len(), RECENT_EVENTS);
assert!(!recent.repeat(&update(0)), "the oldest was forgotten");
assert!(recent.repeat(&update(RECENT_EVENTS)));
}
#[test]
fn foreign_delete_invalidates_an_earlier_refresh_generation() {
let own_volume = VolumeId::from("own");
+87 -28
View File
@@ -163,8 +163,8 @@ fn read_request_line(reader: &mut impl BufRead) -> std::io::Result<Option<String
/// Turn a CLI-supplied path into the mount that owns it and the path relative
/// to *that* mount's root.
///
/// [`rel_to_mount`] is the older, primary-only reading, and it is what every
/// mutating request still uses. It rejects anything outside the primary
/// [`rel_to_mount`] is the older, primary-only reading, and it is what most
/// requests still use. It rejects anything outside the primary
/// mountpoint, which leaves a secondary on-demand mount — a synced folder
/// switched out of mirror mode — with no way to be named at all: `pdfs ls` and
/// `pdfs refresh` both answered "is not under the mountpoint", so a folder
@@ -185,11 +185,14 @@ fn route_to_mount(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
rel_to_mount(mountpoint, path).map(|rel| (core.clone(), rel))
}
/// [`route_to_mount`] for a conflict copy. The listing names a copy inside a
/// synced folder by its absolute local path (`docs/BUGS.md` B99), so the
/// resolve has to run against the on-demand mount serving it. A mirror folder
/// has no mount to run against: its copy is an ordinary local file.
fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
/// [`route_to_mount`] for a request that changes what it names: trash, rename,
/// create a folder, resolve a conflict. Each has to run against the on-demand
/// mount serving the path, and `pdfs rm ~/Downloads/x` answered "is not under
/// the mountpoint" while `pdfs move` took the same path. The conflict listing
/// names a copy in a synced folder by its absolute local path (`docs/BUGS.md`
/// B99), so resolving one needs this too. A mirror folder has no mount to run
/// against: a file there is an ordinary local file.
fn route_change(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
let routed = route_to_mount(core, mountpoint, path).map_err(|e| {
let p = Path::new(path);
let mirrored = core
@@ -200,7 +203,7 @@ fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
.any(|folder| folder.mode == "mirror" && p.starts_with(&folder.local_path));
if mirrored {
CoreError::invalid(format!(
"{path} is in a mirrored folder; rename or delete the local file there"
"{path} is in a mirrored folder; change the local file there instead"
))
} else {
e
@@ -217,6 +220,27 @@ fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
}
}
/// [`route_change`] for a request that trashes or renames the node it names.
/// A mount's root is My files or a whole synced folder. An on-demand folder's
/// root has a parent on Drive, so nothing further down would stop `pdfs rm
/// ~/Downloads` from trashing the folder out from under its own mount.
fn route_node(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
let routed = route_change(core, mountpoint, path)?;
if below_root(&routed.1) {
Ok(routed)
} else {
Err(CoreError::invalid(format!(
"{path} is the root of a synced location and cannot be changed"
)))
}
}
/// Whether a mount-relative path names something below the mount's root.
fn below_root(rel: &Path) -> bool {
rel.components()
.any(|component| matches!(component, Component::Normal(_)))
}
/// Turn a CLI-supplied path into a mountpoint-relative path. An absolute path
/// must live under `mountpoint`; a relative path is taken as already relative to
/// the mount root.
@@ -291,6 +315,9 @@ impl Drop for RefreshGuard {
pub(crate) fn status_response(core: &Core, username: &str, mountpoint: &Path) -> CtlResponse {
let pins = core.cache.list_pins();
let queued = core.db.pending_op_counts().unwrap_or_default();
// Read after the queue: the drain counts an upload as landing before it
// retires the op, so one of the two always has it (B119).
let landing = core.landing_uploads.load(Ordering::SeqCst);
let staging = core.cache.staging_usage();
CtlResponse::Status {
parked_uploads: queued.parked.max(0) as u64,
@@ -307,7 +334,7 @@ pub(crate) fn status_response(core: &Core, username: &str, mountpoint: &Path) ->
budget: core.cache.budget(),
pins,
online: core.online.load(Ordering::Relaxed),
pending_uploads: queued.uploads.max(0) as u64,
pending_uploads: queued.uploads.max(0) as u64 + landing,
pending_changes: queued.changes.max(0) as u64,
}
}
@@ -959,8 +986,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
},
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::Rename { path, new_name }) => match rel_to_mount(mountpoint, &path) {
Ok(rel) => match core.rename(&rel, &new_name) {
Ok(CtlRequest::Rename { path, new_name }) => match route_node(core, mountpoint, &path) {
Ok((core, rel)) => match core.rename(&rel, &new_name) {
Ok(name) => {
core.log_activity(ActivityKind::Rename, &name, format!("was {path}"), true);
CtlResponse::Ok {
@@ -974,6 +1001,24 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
},
Err(e) => CtlResponse::error(e),
},
// Two absolute paths may name two different locations, which only
// `move_between` can resolve; a relative pair keeps the old meaning.
Ok(CtlRequest::Move { path, new_parent })
if Path::new(&path).is_absolute() && Path::new(&new_parent).is_absolute() =>
{
match core.move_between(Path::new(&path), Path::new(&new_parent)) {
Ok(name) => {
core.log_activity(ActivityKind::Move, &name, format!("to {new_parent}"), true);
CtlResponse::Ok {
message: format!("moved {name}"),
}
}
Err(e) => {
core.log_activity(ActivityKind::Move, &path, &e, false);
CtlResponse::error(e)
}
}
}
Ok(CtlRequest::Move { path, new_parent }) => {
match (
rel_to_mount(mountpoint, &path),
@@ -999,8 +1044,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
(Err(e), _) | (_, Err(e)) => CtlResponse::error(e),
}
}
Ok(CtlRequest::Delete { path }) => match rel_to_mount(mountpoint, &path) {
Ok(rel) => match core.delete(&rel) {
Ok(CtlRequest::Delete { path }) => match route_node(core, mountpoint, &path) {
Ok((core, rel)) => match core.delete(&rel) {
Ok(name) => {
core.log_activity(ActivityKind::Trash, &name, "", true);
CtlResponse::Ok {
@@ -1014,21 +1059,23 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
},
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::CreateFolder { parent, name }) => match rel_to_mount(mountpoint, &parent) {
Ok(parent_rel) => match core.create_folder(&parent_rel, &name) {
Ok(name) => {
core.log_activity(ActivityKind::CreateFolder, &name, "", true);
CtlResponse::Ok {
message: format!("created folder {name}"),
Ok(CtlRequest::CreateFolder { parent, name }) => {
match route_change(core, mountpoint, &parent) {
Ok((core, parent_rel)) => match core.create_folder(&parent_rel, &name) {
Ok(name) => {
core.log_activity(ActivityKind::CreateFolder, &name, "", true);
CtlResponse::Ok {
message: format!("created folder {name}"),
}
}
}
Err(e) => {
core.log_activity(ActivityKind::CreateFolder, &name, &e, false);
CtlResponse::error(e)
}
},
Err(e) => CtlResponse::error(e),
},
Err(e) => {
core.log_activity(ActivityKind::CreateFolder, &name, &e, false);
CtlResponse::error(e)
}
},
Err(e) => CtlResponse::error(e),
}
}
Ok(CtlRequest::UploadPaths { parent, sources }) => {
match rel_to_mount(mountpoint, &parent) {
// Ack immediately and upload on a background thread: a directory tree
@@ -1345,7 +1392,7 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::ResolveConflict { path, keep }) => {
match route_conflict(core, mountpoint, &path) {
match route_change(core, mountpoint, &path) {
Ok((core, rel)) => match core.resolve_conflict(&rel, &keep) {
Ok(message) => CtlResponse::Ok { message },
Err(e) => CtlResponse::error(e),
@@ -1846,6 +1893,18 @@ mod request_limit_tests {
assert!(rel_to_mount(mountpoint, "Photos/../../escape").is_err());
assert!(rel_to_mount(mountpoint, "/mnt/elsewhere").is_err());
}
/// `pdfs rm` and `pdfs rename` reach on-demand mounts, whose root is a whole
/// synced folder with a parent on Drive. Only a path below a root may get
/// through to the trash or rename call.
#[test]
fn only_a_path_below_a_mount_root_names_a_node() {
assert!(below_root(Path::new("Photos")));
assert!(below_root(Path::new("./Photos/2026")));
assert!(!below_root(Path::new("")));
assert!(!below_root(Path::new(".")));
assert!(!below_root(Path::new("./.")));
}
}
#[cfg(test)]
+14 -24
View File
@@ -27,7 +27,7 @@ use tracing::{info, warn};
use super::sync::{self, base_name};
use super::{
Core, SecondaryInsertRejection, SecondaryMount, State, SwitchBlocked, clear_stale_mount,
Core, SecondaryInsertRejection, SecondaryMount, SwitchBlocked, clear_stale_mount,
device_type_str, dir_is_empty, evict_dir_contents, fuse_connection_id, is_stale_mount,
now_secs, parse_uid, spawn_session, sync_folder_info, this_hostname,
};
@@ -363,14 +363,14 @@ impl Core {
) -> CoreResult<Option<NodeUid>> {
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(root_uid))
.block_on(self.drive.enumerate_folder_children_node_uids(root_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() {
return Ok(None);
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&uids))
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?;
Ok(nodes
.into_iter()
@@ -446,10 +446,7 @@ impl Core {
}
None => self
.rt
.block_on(
self.client
.create_folder(&root_uid, &name, Some(now_secs())),
)
.block_on(self.drive.create_folder(&root_uid, &name, Some(now_secs())))
.map_err(|e| CoreError::from_api(&e, &format!("create device folder {name}")))?,
};
@@ -603,7 +600,7 @@ impl Core {
&& let Some(uid) = parse_uid(&folder.remote_uid)
&& let Err(e) = self
.rt
.block_on(self.client.trash_nodes(&[uid]))
.block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit)
{
warn!(id, error = %e, "delete remote device folder failed");
@@ -619,23 +616,16 @@ impl Core {
};
}
/// A sibling Core that shares this one's client/rt/cache/db (and transfer,
/// activity, mount registries) but gets a **fresh, empty `State`** — its own
/// inode space starting at [`ROOT_INO`]. Used to root a secondary FUSE session
/// at an `ondemand` sync folder without colliding with the main mount's inodes
/// (devices.md Phase 3).
/// A sibling Core for a secondary FUSE session rooted at an `ondemand` sync
/// folder (devices.md Phase 3). It shares everything with this one,
/// the tree included, except what belongs to a session: its root and its
/// kernel channel, both filled in when the session is built.
pub(crate) fn fork_state(&self) -> Core {
let mut fork = self.clone();
fork.primary = false;
let share_access = self.state().share_access.clone();
fork.state = Arc::new(Mutex::new(State::new(self.db.clone(), share_access, 2)));
// A fresh inode space needs a fresh notification channel: this fork's
// session is the only one that knows these inodes, so it must be the one
// notified about them. Filled in by `spawn_ondemand_mount`.
fork.root = Arc::new(OnceLock::new());
fork.notifier = Arc::new(OnceLock::new());
fork.session_live = Arc::new(std::sync::atomic::AtomicBool::new(false));
// Size upgrades are keyed by inode, which is per-fork too.
fork.size_upgrades = Arc::new(Mutex::new(HashMap::new()));
fork
}
@@ -858,7 +848,7 @@ impl Core {
})?;
let root = self
.rt
.block_on(self.client.enumerate_nodes(std::slice::from_ref(&root_uid)))
.block_on(self.drive.enumerate_nodes(std::slice::from_ref(&root_uid)))
.map_err(|e| SwitchBlocked::Failed(format!("fetch remote root: {e}")))?
.into_iter()
.next()
@@ -937,8 +927,8 @@ impl Core {
}
}
/// Spawn a secondary FUSE session rooted at `root` over `local` on a forked
/// inode space. Clears any stale kernel mount first (a crashed run can leave
/// Spawn a secondary FUSE session rooted at `root` over `local` from a
/// [`Core::fork_state`] sibling. Clears any stale kernel mount first (a crashed run can leave
/// one, which would fail the fresh mount with EBUSY).
pub(crate) fn spawn_ondemand_mount(
&self,
@@ -1017,7 +1007,7 @@ impl Core {
};
let root = match self
.rt
.block_on(self.client.enumerate_nodes(std::slice::from_ref(&root_uid)))
.block_on(self.drive.enumerate_nodes(std::slice::from_ref(&root_uid)))
{
Ok(v) => match v.into_iter().next() {
Some(n) => n,
+512 -130
View File
@@ -24,28 +24,27 @@ use std::collections::HashMap;
use std::fs::File;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::{Duration, Instant};
use parking_lot::Mutex;
use futures::StreamExt;
use pdfs_core::batch;
use pdfs_core::cache::{Baseline, StagedWrite};
use pdfs_core::control::{ActivityKind, TransferDirection};
use pdfs_core::db::{
OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, PARK_EXPIRY_MS, PendingOp, RenameMeta,
CreateLanding, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, PARK_EXPIRY_MS,
PendingOp, RenameMeta,
};
use proton_drive_rs::proton_sdk::error::ProtonError;
use proton_drive_rs::proton_sdk::ids::NodeUid;
use proton_drive_rs::{Node, NodeMoveItem};
use proton_drive_rs::{Node, NodeKind};
use tracing::{debug, error, info, warn};
use super::state::Intervals;
use super::state::{Intervals, PendingRevision};
use super::transfers::CountingReader;
use super::{
Core, DRAIN_BACKOFF_MAX, DRAIN_BACKOFF_MIN, DRAIN_IDLE_POLL, DRAIN_REVISION_DEBOUNCE,
DRAIN_REVISION_DEBOUNCE_MAX, ROOT_INO, UPLOAD_TIME_MEMORY, WriteAuthority, conflict_name,
DRAIN_REVISION_DEBOUNCE_MAX, UPLOAD_TIME_MEMORY, WriteAuthority, conflict_name,
is_already_exists, is_gone, is_local_uid_str, media_type_for, node_revision_id, node_size,
now_millis, now_secs, parse_node_uid,
};
@@ -113,6 +112,23 @@ impl Drop for UploadCancel {
}
}
/// One upload counted in [`Core::landing_uploads`] from its op's retirement
/// until its bytes are cached, and uncounted when dropped, early return or not.
struct LandingUpload(Arc<AtomicU64>);
impl LandingUpload {
fn begin(count: &Arc<AtomicU64>) -> Self {
count.fetch_add(1, Ordering::SeqCst);
Self(count.clone())
}
}
impl Drop for LandingUpload {
fn drop(&mut self) {
self.0.fetch_sub(1, Ordering::SeqCst);
}
}
/// The debounce arithmetic of [`Core::revision_debounce`], without the map:
/// the last measured upload time for this node, bounded on both sides.
///
@@ -659,6 +675,15 @@ impl Core {
return Ok(());
}
};
if let Some(meta) = op.meta_json.as_deref() {
let meta: RenameMeta = serde_json::from_str(meta)?;
if moved_elsewhere(&meta, &node, &parent, &name) {
// Someone moved or renamed it while ours was queued. Ours still
// wins, as the last change does on Drive.
info!(%uid, from = %node.name, to = %name,
"node moved remotely while its rename was queued");
}
}
let landed = if node.parent_uid.as_ref() == Some(&parent) {
self.drain_rename_in_place(&uid, &node.name, &name)?
} else {
@@ -667,15 +692,18 @@ impl Core {
// no window for the requirements to go stale between two calls —
// the race B46 queued this op to get away from.
let target = (node.name != name).then_some(name.as_str());
match self.move_renaming(&uid, &parent, target) {
match self.move_rename_remote(&uid, &parent, target) {
Ok(()) => name.clone(),
// The destination holds that name already. Landing under a
// *different* name is the non-destructive resolution: it neither
// clobbers their file nor drops ours, and it is visible.
Err(e) if is_already_exists(&e) => {
if self.db.has_pending_trash_named(&name)? {
return Err(held_by_queued_trash(&name));
}
let alt = conflict_name(&name, now_secs());
warn!(%uid, name, alt, "destination already holds that name; using a conflict name");
self.move_renaming(&uid, &parent, Some(&alt))?;
self.move_rename_remote(&uid, &parent, Some(&alt))?;
self.adopt_drained_name(&uid, &alt);
self.log_activity(
ActivityKind::Rename,
@@ -696,8 +724,8 @@ impl Core {
let landed = match self.drain_rename_in_place(&uid, &node.name, &name) {
Ok(landed) => landed,
// The node went with it; nothing is left to rename.
Err(e) if is_gone(&e) => node.name.clone(),
Err(e) => return Err(e.into()),
Err(e) if is_gone(e.as_ref()) => node.name.clone(),
Err(e) => return Err(e),
};
self.log_activity(
ActivityKind::Rename,
@@ -712,7 +740,6 @@ impl Core {
}
};
self.db.delete_op(op.id)?;
self.note_self_change(&uid);
info!(%uid, name = %landed, "pending rename landed");
Ok(())
}
@@ -725,16 +752,19 @@ impl Core {
uid: &NodeUid,
current: &str,
name: &str,
) -> Result<String, ProtonError> {
) -> Result<String, Box<dyn std::error::Error>> {
if current == name {
return Ok(current.to_string());
}
match self.rt.block_on(self.client.rename_node(uid, name, None)) {
match self.rename_remote(uid, name) {
Ok(()) => Ok(name.to_string()),
Err(e) if is_already_exists(&e) => {
if self.db.has_pending_trash_named(name)? {
return Err(held_by_queued_trash(name));
}
let alt = conflict_name(name, now_secs());
warn!(%uid, name, alt, "rename target name is taken; using a conflict name");
self.rt.block_on(self.client.rename_node(uid, &alt, None))?;
self.rename_remote(uid, &alt)?;
self.adopt_drained_name(uid, &alt);
self.log_activity(
ActivityKind::Rename,
@@ -744,35 +774,10 @@ impl Core {
);
Ok(alt)
}
Err(e) => Err(e),
Err(e) => Err(e.into()),
}
}
/// Move `uid` under `parent` in one request, renaming it to `target_name` on
/// the way when one is given.
fn move_renaming(
&self,
uid: &NodeUid,
parent: &NodeUid,
target_name: Option<&str>,
) -> Result<(), ProtonError> {
let item = NodeMoveItem {
uid: uid.clone(),
target_name: target_name.map(str::to_string),
};
self.rt.block_on(async {
let mut outcomes =
std::pin::pin!(self.client.move_nodes_streaming(vec![item], parent.clone()));
match outcomes.next().await {
Some(Ok((_, outcome))) => outcome,
Some(Err(e)) => Err(e),
None => Err(ProtonError::invalid_operation(format!(
"move of {uid} reported no outcome"
))),
}
})
}
/// Apply a queued trash to the remote.
///
/// A node that is already gone is a success, not a failure: the outcome the
@@ -783,7 +788,7 @@ impl Core {
let name = op.name.clone().unwrap_or_else(|| op.uid.clone());
match self
.rt
.block_on(self.client.trash_nodes(std::slice::from_ref(&uid)))
.block_on(self.drive.trash_nodes(std::slice::from_ref(&uid)))
.and_then(batch::into_unit)
{
Ok(()) => {}
@@ -793,7 +798,6 @@ impl Core {
Err(e) => return Err(e.into()),
}
self.db.complete_trash_op(op.id, &uid)?;
self.own_sealed_revs.lock().remove(&uid);
if let Err(e) = self.db.clear_own_sealed_rev(&uid.to_string()) {
debug!(%uid, error = %e, "clearing the sealed-revision record failed");
}
@@ -842,8 +846,27 @@ impl Core {
// reachable only because the op waited, which is the whole point of the
// queue. Never overwrite theirs and never drop ours: land under a
// conflict name, exactly as the sync engine does.
//
// Two things can hold the name that are not someone else's file. The
// node a create made before the network dropped its answer is ours: the
// create was queued because nobody heard back, so adopt it. And a node
// whose trash is still queued is about to let go of it.
let mut name = wanted.clone();
let mut home = parent.clone();
let mut adopted = false;
let mut real = self.create_drained_node(op, &parent, &name);
if real.as_ref().is_err_and(|e| is_already_exists(e.as_ref()))
&& let Some(twin) = self.adoptable_twin(op, &parent, &wanted)?
{
info!(%local, %twin, wanted, "the name is held by our own unanswered create; adopting it");
real = Ok(twin);
adopted = true;
}
if real.as_ref().is_err_and(|e| is_already_exists(e.as_ref()))
&& self.db.has_pending_trash_named(&wanted)?
{
return Err(held_by_queued_trash(&wanted));
}
if real.as_ref().is_err_and(|e| is_already_exists(e.as_ref())) {
name = conflict_name(&wanted, now_secs());
warn!(%local, wanted, name, "name is taken remotely; creating under a conflict name");
@@ -872,6 +895,7 @@ impl Core {
name = conflict_name(&wanted, now_secs());
real = self.create_drained_node(op, &root, &name);
}
home = root;
if real.is_ok() {
self.log_activity(
ActivityKind::Upload,
@@ -886,20 +910,84 @@ impl Core {
// Retire the op before touching anything else: if we crash here the node
// exists remotely and the local placeholder is reconciled by the event
// sync, whereas a surviving op would create the file a second time.
self.db.delete_op(op.id)?;
self.adopt_real_uid(&local, &real)?;
// An adopted node was made empty, so a blob this op carries has not been
// uploaded: it goes on as a revision of the node instead.
let uploaded = (!adopted).then_some(op.blob_path.as_deref()).flatten();
let landing = CreateLanding {
local: &local.to_string(),
real: &real.to_string(),
sent: (parent_str, &wanted),
landed: (&home.to_string(), &name),
};
let newer = self.retire_create(op, uploaded, &landing, &local, &real)?;
// The uploaded blob is the new file's content, so it becomes the cached
// content, as for a revision. A partial write fills its gaps from it;
// without it the next write open was refused (`docs/BUGS.md` B123).
let landed: Option<StagedWrite> = uploaded
.filter(|_| !newer)
.and(op.meta_json.as_deref())
.and_then(|json| serde_json::from_str(json).ok());
self.adopt_real_uid(&local, &real, |node| {
if let (Some(blob), Some(meta)) = (uploaded, &landed)
&& node_size(node) == meta.len
&& keeps_landed_upload(self.cache.is_pinned(&real), meta.len, self.cache.budget())
{
let _ =
self.cache
.store_file(&real, node.modification_time, meta.len, Path::new(blob));
}
})?;
// The feed will report this create back to us; the tree already has it
// under its real uid, so that event is ours to ignore (`Core::self_changes`).
self.note_self_change(&real);
if let Some(blob) = op.blob_path.as_deref() {
// The uploaded blob is done with either way: a write that replaced it
// while it was on the wire already discarded it when it attached.
if let Some(blob) = uploaded {
self.cache.discard_staged(Path::new(blob));
}
self.pending.lock().remove(&local);
if newer {
info!(%local, %real, name, "a write landed during the create; queued it as a revision");
self.wake_drain();
}
self.log_activity(ActivityKind::Upload, &name, "created", true);
info!(%local, %real, name, kind = %op.kind, "pending create landed");
Ok(())
}
/// The node already holding `name` under `parent`, if it is the one an
/// earlier attempt at this op made before its answer was lost.
///
/// A create that timed out may still have landed; the op was queued because
/// the caller never heard back. Without this, its replay forked a
/// `(sync-conflict)` copy of a file nobody else had touched.
fn adoptable_twin(
&self,
op: &PendingOp,
parent: &NodeUid,
name: &str,
) -> Result<Option<NodeUid>, Box<dyn std::error::Error>> {
let uids = self
.rt
.block_on(self.drive.enumerate_folder_children_node_uids(parent))?;
let children = self.rt.block_on(self.drive.enumerate_nodes_light(&uids))?;
let Some(twin) = children
.into_iter()
.find(|node| node.name == name && !node.trashed)
else {
return Ok(None);
};
// The light listing carries no file size, and an adoptable file is
// exactly an empty one.
let twin = match twin.kind {
NodeKind::Folder => twin,
NodeKind::File { .. } => match self.fetch_node_remote(&twin.uid)? {
Some(node) => node,
None => return Ok(None),
},
};
Ok(adoptable(op.kind == OP_MKDIR, &twin, op.created_at).then_some(twin.uid))
}
/// Make one queued `create`/`mkdir` real under a given name, and hand back
/// the API's own error so the caller can tell a name clash from a failure.
///
@@ -915,7 +1003,7 @@ impl Core {
true => {
Ok(self
.rt
.block_on(self.client.create_folder(parent, name, Some(now_secs())))?)
.block_on(self.drive.create_folder(parent, name, Some(now_secs())))?)
}
false => self.upload_created_file(op, parent, name),
}
@@ -930,7 +1018,7 @@ impl Core {
name: &str,
) -> Result<NodeUid, Box<dyn std::error::Error>> {
let Some(blob) = op.blob_path.as_deref() else {
return Ok(self.rt.block_on(self.client.upload_file(
return Ok(self.rt.block_on(self.drive.upload_file(
parent,
name,
media_type_for(name),
@@ -950,7 +1038,7 @@ impl Core {
.transfers
.begin(name, op.uid.clone(), TransferDirection::Upload, meta.len);
let reader = CountingReader::new(File::open(blob)?, &guard);
let uid = self.rt.block_on(self.client.upload_file_from(
let uid = self.rt.block_on(self.drive.upload_file_from(
parent,
name,
media_type_for(name),
@@ -963,39 +1051,155 @@ impl Core {
Ok(uid)
}
/// Retire a landed create's op and move its queued state to the real uid.
/// True when a newer write attached to the create while it was uploading,
/// which is now queued as a revision of `real`.
///
/// Runs under the `pending` lock, which `Core::enqueue_staged_write` also
/// holds from attaching a blob to a create until it records the pending
/// entry. So a write either attached before this ran and is carried over,
/// or finds the create gone and queues against the real uid itself. The
/// open write handles are repointed right after, for the same reason:
/// a handle released later must not queue against the placeholder.
fn retire_create(
&self,
op: &PendingOp,
uploaded: Option<&str>,
landing: &CreateLanding<'_>,
local: &NodeUid,
real: &NodeUid,
) -> Result<bool, Box<dyn std::error::Error>> {
let newer = {
let mut pending = self.pending.lock();
let newer = self.db.finish_create(op.id, uploaded, landing, |json| {
let mut meta: StagedWrite = serde_json::from_str(json).ok()?;
meta.uid = landing.real.to_string();
serde_json::to_string(&meta).ok()
})?;
pending.remove(local);
if let Some((blob, json)) = &newer {
let meta: StagedWrite = serde_json::from_str(json)?;
pending.insert(
real.clone(),
PendingRevision {
path: PathBuf::from(blob),
meta,
},
);
}
newer.is_some()
};
if !newer && let Some(blob) = uploaded {
self.cache_created(op, local, real, blob);
}
self.for_each_state(|st| {
if let Some(ino) = st.by_uid.remove(local) {
st.by_uid.insert(real.clone(), ino);
if let Some(e) = st.entries.get_mut(&ino) {
e.uid = real.clone();
}
// Its children still name the placeholder as their parent, and
// the next write of one would put it back on the row.
for kid in st.children.get(&ino).cloned().unwrap_or_default() {
if let Some(e) = st.entries.get_mut(&kid)
&& e.node.parent_uid.as_ref() == Some(local)
{
e.node.parent_uid = Some(real.clone());
}
}
}
for aw in st.active_writes.values_mut() {
if aw.uid == *local {
aw.uid = real.clone();
}
}
});
Ok(newer)
}
/// Cache a landed create's uploaded `blob` under `real`, keyed as the tree
/// has the file.
///
/// A write handle open on the file is repointed at `real` right after this,
/// and a release fills its gaps from the cache by the size and mtime it
/// opened over. Drive lists the new node a moment after the create, so
/// `adopt_real_uid` caches the bytes only once it can read the node back.
/// A release in between found nothing to fill from, and the next write open
/// was refused (docs/BUGS.md B131).
fn cache_created(&self, op: &PendingOp, local: &NodeUid, real: &NodeUid, blob: &str) {
let Some(meta) = op
.meta_json
.as_deref()
.and_then(|json| serde_json::from_str::<StagedWrite>(json).ok())
else {
return;
};
let mut mtime = None;
self.for_each_state(|st| {
if let Some(entry) = st.by_uid.get(local).and_then(|ino| st.entries.get(ino)) {
mtime = Some(entry.node.modification_time);
}
});
if let Some(mtime) = mtime
&& keeps_landed_upload(self.cache.is_pinned(real), meta.len, self.cache.budget())
{
let _ = self
.cache
.store_file(real, mtime, meta.len, Path::new(blob));
}
}
/// Swap a placeholder uid for the real one across everything that keyed off
/// it: queued children, the DB, the in-memory tree, and the caches.
///
/// The inode is deliberately kept, so anything already holding the file open
/// keeps working across the drain.
///
/// Applied to **every** live inode space, not just this Core's. A node
/// created inside an on-demand sync folder lives in that fork's `state`,
/// while the drain that lands its create belongs to the primary mount — so
/// reaching for `self.state` here found nothing and left the fork's entry on
/// its `local~` uid, which reads as an empty file for as long as the daemon
/// runs (`docs/BUGS.md` B74). A uid is unique across mounts, so at most one
/// of them matches and the others are no-ops.
/// Applied to the tree every mount shares. A node created inside an
/// on-demand sync folder once lived in that fork's own state, where the
/// primary mount's drain never looked, and stayed on its `local~` uid —
/// an empty file for as long as the daemon ran (`docs/BUGS.md` B74).
///
/// `adopt` sees the node as Drive made it, before anything is rebased onto
/// it.
pub(crate) fn adopt_real_uid(
&self,
local: &NodeUid,
real: &NodeUid,
adopt: impl FnOnce(&Node),
) -> Result<(), Box<dyn std::error::Error>> {
let node = self
.fetch_node(real)
// Drive answers a create before it lists the new node.
let mut node = self
.read_back(real)
.map_err(|e| self.errno_error(e, "fetch node"))?;
// Repoints queued children and node rows, and drops the placeholder row.
// A rename made while the create was on the wire is queued against the
// real uid (`Db::finish_create`). Until it lands, the tree keeps the
// name and folder the user gave it.
let renaming = self.db.has_pending_op(&real.to_string(), OP_RENAME)?;
// Before open handles are rebased onto the node, so one released in
// between finds what `adopt` keeps (`docs/BUGS.md` B123).
adopt(&node);
// Repoints queued children and node rows, and readdresses the placeholder row.
// A no-op after `retire_create`, which has already done both.
self.db
.remap_local_uid(&local.to_string(), &real.to_string())?;
self.for_each_state(|st| {
let Some(ino) = st.by_uid.remove(local) else {
if let Some(ino) = st.by_uid.remove(local) {
st.by_uid.insert(real.clone(), ino);
}
let Some(&ino) = st.by_uid.get(real) else {
return;
};
st.by_uid.insert(real.clone(), ino);
if renaming && let Some(e) = st.entries.get(&ino) {
node.name = e.node.name.clone();
node.parent_uid = e.node.parent_uid.clone();
}
let mut landed = node.clone();
st.keep_open_write_size(ino, &mut landed);
if let Some(e) = st.entries.get_mut(&ino) {
e.uid = real.clone();
e.node = node.clone();
e.node = landed;
}
// Where the op said the node goes and where it actually landed can
// differ — a conflict re-homes it — so the tree follows the parent
@@ -1024,6 +1228,15 @@ impl Core {
if let Err(e) = self.db.upsert_node(&node) {
warn!(%real, error = %e, "db upsert_node failed after remap");
}
// A handle still open on the placeholder was based on no revision at
// all: a local-clock mtime and no revision id. Released as it is, its
// write would land as a conflict copy of the empty file this create just
// made (`docs/BUGS.md` B113). Rebased before the restamp below, so a
// release that races this is caught there.
self.rebase_open_writes(real, &node);
// A write carried over from the create was made against no revision at
// all; the one just created is what it now replaces.
self.rebaseline_pending(real, &node);
if node.is_folder() {
// It was recorded as listed while local (it was empty and had nothing
// to enumerate). That still holds: its queued children re-intern under
@@ -1072,16 +1285,11 @@ impl Core {
// instead of forking — so the finished bytes win the name rather than
// being exiled to a `(sync-conflict)` copy.
let remote_rev = node_revision_id(&node);
// In memory first, then the table it is written through to: the map is
// this process's cache of it, and after a restart only the table has
// the answer (which is the whole reason it is persisted).
let own_rev = match self.own_sealed_revs.lock().get(uid).cloned() {
Some(rev) => Some(rev),
None => self
.db
.own_sealed_rev(&uid.to_string(), now_millis())
.unwrap_or_default(),
};
// The table, so the answer survives a restart (`own_sealed_rev`).
let own_rev = self
.db
.own_sealed_rev(&uid.to_string(), now_millis())
.unwrap_or_default();
if is_own_self_supersede(meta.complete, remote_rev.as_deref(), own_rev.as_deref()) {
debug!(%uid, ?remote_rev,
"queued write chains onto our own sealed revision; not a conflict");
@@ -1151,7 +1359,7 @@ impl Core {
self.transfers
.begin(&alt, meta.uid.clone(), TransferDirection::Upload, meta.len);
let reader = CountingReader::new(File::open(blob)?, &guard);
self.rt.block_on(self.client.upload_file_from(
self.rt.block_on(self.drive.upload_file_from(
&parent,
&alt,
media_type_for(&alt),
@@ -1348,25 +1556,13 @@ impl Core {
/// A node's parent uid and name, as the tree currently has them.
pub(crate) fn node_place(&self, uid: &NodeUid) -> Option<(NodeUid, String)> {
// Across every mount: the drain runs on the primary Core, but a node
// written inside a sync folder is interned in that fork's inode space,
// so asking `self.state` alone always missed it and fell through to the
// DB. That still gave the right answer — the fallback below exists for
// exactly this shape of miss — but it went to disk for something already
// in memory, and only *because* of the fallback was it not a bug.
let mut found = None;
self.for_each_state(|st| {
if found.is_some() {
return;
}
{
let st = self.state();
if let Some(entry) = st.by_uid.get(uid).and_then(|ino| st.entries.get(ino))
&& let Some(parent) = entry.node.parent_uid.clone()
{
found = Some((parent, entry.node.name.clone()));
return Some((parent, entry.node.name.clone()));
}
});
if found.is_some() {
return found;
}
// The in-memory tree only holds what has been walked to since the daemon
// started, and the drain routinely runs before anything has walked to
@@ -1401,12 +1597,13 @@ impl Core {
/// root yet, which is not where the drain runs; the drain must not panic
/// over it regardless, since that would stop the queue for good.
pub(crate) fn root_uid(&self) -> Option<NodeUid> {
let st = self.state();
st.entries.get(&ROOT_INO).map(|e| e.uid.clone())
let root = self.root_ino();
self.state().entries.get(&root).map(|e| e.uid.clone())
}
/// Upload a staged revision of a file the server already knows about.
pub(crate) fn drain_revision(&self, op: &PendingOp) -> Result<(), Box<dyn std::error::Error>> {
let op_started = Instant::now();
let blob = op
.blob_path
.clone()
@@ -1416,6 +1613,16 @@ impl Core {
let uid = parse_node_uid(&meta.uid).ok_or("staged write has an unparseable uid")?;
if let Some(reason) = self.revision_conflict(&uid, &meta)? {
// Trashing the file from the mount drops its queued write, but not
// from under a worker already holding it. That worker then finds the
// node trashed — by us — and would upload the withdrawn bytes as a
// conflict copy (a `recovered-…` file in the root when the tree has
// already forgotten the node). A write nobody queues any more is
// not a conflict.
if !self.db.op_exists(op.id)? {
debug!(%uid, reason, "queued write withdrawn while it drained; dropping it");
return Ok(());
}
return self.keep_as_conflict_copy(op, &blob, &meta, &uid, &reason);
}
@@ -1452,9 +1659,15 @@ impl Core {
// this is on the wire can stop it rather than paying for a revision the
// op it queued will immediately replace.
let cancel = self.begin_cancellable_upload(&uid);
// An unlink that withdrew this op before the flag above existed had no
// upload to cancel; the row is the record that it happened.
if !self.db.op_exists(op.id)? {
debug!(%uid, "queued write withdrawn before it went out; dropping it");
return Ok(());
}
let reader = CountingReader::new(File::open(&blob)?, &guard).with_cancel(cancel.flag());
let started = Instant::now();
let sent = self.rt.block_on(self.client.upload_new_revision_from(
let sent = self.rt.block_on(self.drive.upload_new_revision_from(
&uid,
reader,
meta.len as i64,
@@ -1480,25 +1693,46 @@ impl Core {
// is what the next write's debounce is scaled from.
self.record_upload_time(&uid, took);
// Still pending to whoever asks until the bytes are cached below, and
// counted before the op goes so the status never reads empty in between.
// An empty queue is how a script or the app knows a written file reads
// without the network; retiring the op alone left the read-back's round
// trip in which it did not (B119). The op itself has to go first: one
// restored after a crash would re-send a revision this daemon has not
// yet recorded as its own, and come back as a conflict copy.
let _landing = LandingUpload::begin(&self.landing_uploads);
// Retire the op before dropping the blob: a crash between the two leaves
// an orphaned file (harmless), whereas the reverse would leave a queued
// op pointing at nothing.
self.db.delete_op(op.id)?;
self.release_pending(&uid, &blob);
// The staged blob now matches the sealed revision, so a pinned file keeps
// it as its cached content rather than re-downloading what we just sent.
// Adopted by link, not by value: the blob is the whole file, and a pinned
// video read into a `Vec` here is an OOM. `discard_staged` below drops the
// staging name only — the cache keeps the inode.
if self.cache.is_pinned(&uid) {
let _ = self.cache.store_file(&uid, now_secs(), meta.len, &blob);
}
self.cache.discard_staged(&blob);
self.evict_reader(&uid);
self.refresh_after_upload(&uid);
// The staged blob now matches the sealed revision, so it becomes the
// file's cached content, stamped like the revision just read back.
// Reading the file again then needs no network, which an editor or a
// build does at once and a flaky link cannot always give (`docs/BUGS.md`
// B119). Adopted by link, not by value: the blob is the whole file, and
// a video read into a `Vec` here is an OOM. `discard_staged` below drops
// the staging name only — the cache keeps the inode.
self.refresh_after_upload(&uid, &blob, |node| {
if node_size(node) == meta.len
&& keeps_landed_upload(self.cache.is_pinned(&uid), meta.len, self.cache.budget())
{
let _ = self
.cache
.store_file(&uid, node.modification_time, meta.len, &blob);
}
});
self.cache.discard_staged(&blob);
self.log_activity(ActivityKind::Upload, &name, "uploaded", true);
info!(%uid, len = meta.len, "pending upload landed");
// Both times, so a slow drain shows whether the upload itself or the
// checks and gap-fill around it are where the time goes.
info!(
%uid,
len = meta.len,
upload_ms = took.as_millis() as u64,
total_ms = op_started.elapsed().as_millis() as u64,
"pending upload landed"
);
Ok(())
}
@@ -1522,21 +1756,39 @@ impl Core {
/// we just sealed. Skipping both, as this used to, is what produced a file
/// that conflicted with itself and left a full-size duplicate behind.
///
/// `blob` is the revision that landed, which stays the pending one until
/// open write handles are rebased off it, and `adopt` gets the node read
/// back before that, so the uploaded bytes can be cached under its
/// revision first. A handle released in between fills its gaps from one
/// or the other; with the blob let go first and the cache filled last, it
/// found neither, and the next write open was refused (docs/BUGS.md B123).
///
/// Best effort: a failure here costs a spurious conflict copy on the next
/// write, not this upload, which has already landed.
pub(crate) fn refresh_after_upload(&self, uid: &NodeUid) {
let node = match self.fetch_node_remote(uid) {
/// write, not this upload, which has already landed. Returns the node as
/// the server now has it, or `None` when it could not be read back.
pub(crate) fn refresh_after_upload(
&self,
uid: &NodeUid,
blob: &Path,
adopt: impl FnOnce(&Node),
) -> Option<Node> {
let fetched = self.fetch_node_remote(uid);
if !matches!(fetched, Ok(Some(_))) {
self.release_pending(uid, blob);
}
let node = match fetched {
Ok(Some(node)) => node,
// Trashed or deleted under us: the tree will hear it from the event
// sync, which is better placed to unhook the inode than we are.
Ok(None) => return,
Ok(None) => return None,
Err(e) => {
warn!(%uid, error = %e,
"refreshing metadata after an upload failed; \
the next write to this file may conflict with itself");
return;
return None;
}
};
adopt(&node);
// This function's whole job is to bring the tree level with the revision
// we just sealed, so the feed's report of that revision has nothing left
// to tell us. Claimed here rather than at the upload call so it is only
@@ -1552,18 +1804,14 @@ impl Core {
// This is the open-handle counterpart of `rebaseline_pending`, which
// covers the same gap for *queued* ops.
{
let sealed_mtime = node.modification_time;
let sealed_size = node_size(&node);
let sealed_rev = node_revision_id(&node);
// Remember it as ours, so a queued write that opened over an earlier
// revision recognises this as a self-supersede rather than a foreign
// change and does not fork (B70 layer B; consulted in
// `revision_conflict`).
if let Some(rev) = sealed_rev.clone() {
self.own_sealed_revs.lock().insert(uid.clone(), rev.clone());
// Write through: losing this at shutdown reopened the fork
// window B70 layer B closed, for the first drain after a
// restart.
if let Some(rev) = node_revision_id(&node) {
// In the table, not in memory: losing it at shutdown reopened
// the fork window B70 layer B closed, for the first drain
// after a restart.
if let Err(e) = self
.db
.set_own_sealed_rev(&uid.to_string(), &rev, now_millis())
@@ -1571,23 +1819,14 @@ impl Core {
warn!(%uid, error = %e, "persisting our sealed revision failed");
}
}
self.for_each_state(|st| {
for aw in st.active_writes.values_mut() {
if aw.uid == *uid {
aw.base_mtime = sealed_mtime;
aw.base_size = sealed_size;
aw.base_revision_id = sealed_rev.clone();
debug!(%uid, mtime = sealed_mtime, size = sealed_size,
"rebased open write handle onto the revision just uploaded");
}
}
});
self.rebase_open_writes(uid, &node);
}
self.release_pending(uid, blob);
// Ordered so that a write queued *during* the fetch above is still
// caught: it took its baseline from the node's optimistic stamp, and
// this overwrites it with the revision the server actually holds.
if self.rebaseline_pending(uid, &node) {
return;
return Some(node);
}
self.for_each_state(|st| {
let Some(parent) = st
@@ -1600,6 +1839,21 @@ impl Core {
};
st.intern(parent, node.clone());
});
Some(node)
}
/// Rebase every write handle open on `uid` onto `node`, a revision this
/// daemon has just put there itself (`WriteHandle::rebase_onto`).
fn rebase_open_writes(&self, uid: &NodeUid, node: &Node) {
self.for_each_state(|st| {
for aw in st.active_writes.values_mut() {
if aw.uid == *uid {
aw.rebase_onto(node);
debug!(%uid, mtime = aw.base_mtime, size = aw.base_size,
"rebased open write handle onto the revision just landed");
}
}
});
}
/// Point a still-queued write at the revision this upload just sealed, and
@@ -1690,6 +1944,20 @@ fn is_own_self_supersede(complete: bool, remote_rev: Option<&str>, own_rev: Opti
complete && remote_rev.is_some() && remote_rev == own_rev
}
/// Whether Drive has `node` neither where its queued rename found it nor where
/// the rename takes it (`parent`, `name`), so someone else moved or renamed it
/// in between. Unknown on a row without the original name.
fn moved_elsewhere(meta: &RenameMeta, node: &Node, parent: &NodeUid, name: &str) -> bool {
let Some(original_name) = meta.original_name.as_deref() else {
return false;
};
let at = |parent: &str, name: &str| {
node.parent_uid.as_ref().map(ToString::to_string).as_deref() == Some(parent)
&& node.name == name
};
!at(&meta.original_parent_uid, original_name) && !at(&parent.to_string(), name)
}
/// What the park sweep does with one parked create whose node still exists.
#[derive(Debug, PartialEq, Eq)]
enum ParkVerdict {
@@ -1708,9 +1976,48 @@ fn park_verdict(created_at: i64, now: i64, open: bool) -> ParkVerdict {
ParkVerdict::Release
}
/// The retryable error for an op whose name a queued trash has yet to free.
fn held_by_queued_trash(name: &str) -> Box<dyn std::error::Error> {
format!("{name} is still held by a node whose trash is queued").into()
}
/// How much older than its op a remote node may be and still be the one an
/// unanswered call made. The op is queued the moment the call gives up, so the
/// node is at most one call's deadline older; the rest is clock skew between
/// this machine and Proton.
const ADOPT_WINDOW_MS: i64 = 2 * 60 * 1000;
/// Whether `twin`, found holding the name a queued create wants, is the node an
/// earlier, unanswered attempt at that create made: the same kind, created
/// around when the op was queued, and for a file still empty, as the mount's
/// create leaves it.
fn adoptable(is_dir: bool, twin: &Node, op_created_ms: i64) -> bool {
if twin.creation_time.saturating_mul(1000) < op_created_ms - ADOPT_WINDOW_MS {
return false;
}
match &twin.kind {
NodeKind::Folder => is_dir,
NodeKind::File { claimed_size, .. } => !is_dir && *claimed_size == Some(0),
}
}
/// What share of the cache budget an unpinned upload may take and still be kept
/// as cached content once it lands: a sixteenth, 320 MiB of the default 5 GiB.
/// A larger file would push out much of what the user read lately, for bytes
/// they may never read back.
const LANDED_KEEP_SHARE: u64 = 16;
/// Whether an upload of `len` bytes that just landed stays as the file's cached
/// content. A pinned file always does, and so does any file when the budget is
/// unlimited (`0`).
fn keeps_landed_upload(pinned: bool, len: u64, budget: u64) -> bool {
pinned || budget == 0 || len <= budget / LANDED_KEEP_SHARE
}
#[cfg(test)]
mod tests {
use super::*;
use crate::state::WriteHandle;
use pdfs_core::Access;
use pdfs_core::db::Db;
use proton_drive_rs::NodeKind;
@@ -1822,6 +2129,7 @@ mod tests {
op.meta_json = Some(
serde_json::to_string(&RenameMeta {
original_parent_uid: source.uid.to_string(),
original_name: Some(moved.name.clone()),
})
.unwrap(),
);
@@ -2087,6 +2395,60 @@ mod tests {
}
}
#[test]
fn only_a_fresh_empty_twin_of_the_same_kind_is_adopted() {
// The op was queued at t=1000 s, after a create that got no answer.
let queued = 1_000_000;
let made = |mut node: Node, secs: i64| {
node.creation_time = secs;
node
};
assert!(adoptable(false, &made(file_node(0, 0, None), 990), queued));
assert!(adoptable(true, &made(folder_node("d", None), 990), queued));
// Made long before the op: someone else's file, not our lost create.
assert!(!adoptable(false, &made(file_node(0, 0, None), 800), queued));
// Our create leaves a file empty; one with content is someone else's.
assert!(!adoptable(false, &made(file_node(0, 5, None), 990), queued));
// The kind has to match what the op makes.
assert!(!adoptable(true, &made(file_node(0, 0, None), 990), queued));
assert!(!adoptable(
false,
&made(folder_node("d", None), 990),
queued
));
}
#[test]
fn a_landed_upload_stays_cached_unless_it_would_crowd_the_cache_out() {
const GIB: u64 = 1 << 30;
let budget = 5 * GIB;
// An edited document or a source file: read back from disk, not Drive.
assert!(keeps_landed_upload(false, 40_000, budget));
assert!(keeps_landed_upload(false, budget / 16, budget));
// A video the user copied in would push out what they read lately.
assert!(!keeps_landed_upload(false, budget / 16 + 1, budget));
// Pinned content is kept whatever its size, and so is everything when
// the cache has no cap.
assert!(keeps_landed_upload(true, 4 * GIB, budget));
assert!(keeps_landed_upload(false, 4 * GIB, 0));
}
#[test]
fn a_landing_upload_counts_until_its_guard_goes() {
let count = Arc::new(AtomicU64::new(0));
let first = LandingUpload::begin(&count);
let landed = || -> Result<(), ()> {
let _second = LandingUpload::begin(&count);
assert_eq!(count.load(Ordering::SeqCst), 2);
Err(())
};
// Uncounted on an early return too, or the queue never reads empty.
assert!(landed().is_err());
assert_eq!(count.load(Ordering::SeqCst), 1);
drop(first);
assert_eq!(count.load(Ordering::SeqCst), 0);
}
fn baseline(mtime: i64, size: u64, rev: Option<&str>) -> Baseline {
Baseline {
mtime,
@@ -2114,6 +2476,26 @@ mod tests {
assert!(revision_changed(&base, &remote).is_some());
}
/// The reported shape (docs/BUGS.md B113): a downloader creates a file
/// through the mount, and the file is moved before its write drains. The
/// move re-stamps the node's mtime on the server (1790955809 -> 1790956593),
/// but the revision is still the empty one the create minted.
#[test]
fn a_move_before_the_first_write_drains_is_not_a_conflict() {
let minted = file_node(1_790_955_809, 0, Some("empty"));
let path = std::env::temp_dir().join(format!("pdfs-drain-test-{}", std::process::id()));
let file = File::create(&path).unwrap();
let _ = std::fs::remove_file(&path);
let h = WriteHandle::created(2, &minted, file, path);
let base = baseline(h.base_mtime, h.base_size, h.base_revision_id.as_deref());
let moved = file_node(1_790_956_593, 0, Some("empty"));
assert_eq!(revision_changed(&base, &moved), None);
// Another device writing it is still a conflict.
let edited = file_node(1_790_956_593, 0, Some("theirs"));
assert!(revision_changed(&base, &edited).is_some());
}
#[test]
fn without_ids_it_falls_back_to_mtime_and_size() {
// Old sidecar (no revision id): the (mtime, size) tuple still governs.
+466
View File
@@ -0,0 +1,466 @@
//! The calls the sync core makes to Drive, behind a trait.
//!
//! `Core` used to hold a concrete [`ProtonDriveClient`], so no test could make
//! Drive answer late, answer stale, or not answer at all, and the interleavings
//! behind most recent bugs could only be found on a real account over a slow
//! link (`docs/MILESTONE-3.0.0.md` §2, §8.1). [`DriveApi`] is the part of the
//! client the mount, the drain, the event feed and the mirror engine use; the
//! real client implements it by forwarding, and the simulation tests implement
//! it with an in-memory Drive that injects Drive's known faults.
//!
//! Sharing, Photos, devices, invitations and public links stay on the concrete
//! client (`Core::client`) until a test needs them faked.
//!
//! The method names match the client's own, so a call site reads the same
//! whichever it goes through. The client's generic methods (a reader or writer
//! of any type) cannot be part of an object-safe trait, so the trait takes them
//! as trait objects and `impl dyn DriveApi` puts the generic signatures back.
use std::io::{Read, Write};
use std::sync::Arc;
use async_trait::async_trait;
use futures::StreamExt as _;
use futures::stream::BoxStream;
use proton_drive_rs::proton_sdk::account::Quota;
use proton_drive_rs::proton_sdk::error::Result;
use proton_drive_rs::proton_sdk::ids::{DriveEventId, NodeUid};
use proton_drive_rs::{
DriveEvent, DriveEventScopeId, Node, NodeMoveItem, ProtonDriveClient, RevisionReader, Thumbnail,
};
/// One outcome per node of a batch request; see [`ProtonDriveClient::trash_nodes`].
pub(crate) type NodeOutcomes = Vec<(NodeUid, Result<()>)>;
/// The outcomes of a streaming batch request, one item per node as it lands.
pub(crate) type OutcomeStream<'a> = BoxStream<'a, Result<(NodeUid, Result<()>)>>;
/// Drive as the sync core sees it. See the module documentation.
#[async_trait]
pub(crate) trait DriveApi: Send + Sync {
async fn get_my_files_folder(&self) -> Result<Node>;
async fn get_node(&self, uid: &NodeUid) -> Result<Option<Node>>;
async fn enumerate_nodes(&self, uids: &[NodeUid]) -> Result<Vec<Node>>;
/// As [`DriveApi::enumerate_nodes`], but files carry no claimed size or
/// modification time; see [`ProtonDriveClient::enumerate_nodes_light`].
async fn enumerate_nodes_light(&self, uids: &[NodeUid]) -> Result<Vec<Node>>;
async fn enumerate_folder_children_node_uids(
&self,
folder_uid: &NodeUid,
) -> Result<Vec<NodeUid>>;
async fn enumerate_events(
&self,
scope: &DriveEventScopeId,
cursor: Option<&DriveEventId>,
) -> Result<Vec<DriveEvent>>;
async fn invalidate_caches_for_event(&self, event: &DriveEvent) -> Result<()>;
async fn create_folder(
&self,
parent_uid: &NodeUid,
name: &str,
last_modification_time: Option<i64>,
) -> Result<NodeUid>;
async fn upload_file(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
contents: &[u8],
) -> Result<NodeUid>;
#[allow(clippy::too_many_arguments)]
async fn upload_file_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid>;
#[allow(clippy::too_many_arguments)]
async fn upload_file_replacing_draft_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid>;
async fn upload_new_revision_from_dyn(
&self,
file_uid: &NodeUid,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()>;
async fn open_revision(&self, uid: &NodeUid) -> Result<Arc<dyn RevisionRead>>;
async fn download_file_to_dyn(
&self,
uid: &NodeUid,
output: &mut (dyn Write + Send),
) -> Result<()>;
async fn download_revision_to_dyn(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut (dyn Write + Send),
) -> Result<()>;
async fn rename_node(
&self,
uid: &NodeUid,
new_name: &str,
new_media_type: Option<&str>,
) -> Result<()>;
async fn move_node(&self, uid: &NodeUid, new_parent: &NodeUid) -> Result<()>;
fn move_nodes_streaming<'a>(
&'a self,
items: Vec<NodeMoveItem>,
new_parent: NodeUid,
) -> OutcomeStream<'a>;
async fn trash_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes>;
async fn restore_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes>;
fn restore_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a>;
fn delete_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a>;
async fn enumerate_trash_node_uids(&self) -> Result<Vec<NodeUid>>;
/// The account's storage, across every Proton product.
async fn quota(&self) -> Result<Quota>;
}
/// The generic signatures of the client, over the object-safe ones above.
impl dyn DriveApi {
#[allow(clippy::too_many_arguments)]
pub(crate) async fn upload_file_from<R: Read + Send>(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
self.upload_file_from_dyn(
parent_uid,
name,
media_type,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
#[allow(clippy::too_many_arguments)]
pub(crate) async fn upload_file_replacing_draft_from<R: Read + Send>(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
self.upload_file_replacing_draft_from_dyn(
parent_uid,
name,
media_type,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
pub(crate) async fn upload_new_revision_from<R: Read + Send>(
&self,
file_uid: &NodeUid,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()> {
self.upload_new_revision_from_dyn(
file_uid,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
)
.await
}
pub(crate) async fn download_file_to<W: Write + Send>(
&self,
uid: &NodeUid,
output: &mut W,
) -> Result<()> {
self.download_file_to_dyn(uid, output).await
}
pub(crate) async fn download_revision_to<W: Write + Send>(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut W,
) -> Result<()> {
self.download_revision_to_dyn(file_uid, revision_id, writer)
.await
}
}
/// An open revision that block reads are served from; see [`RevisionReader`].
#[async_trait]
pub(crate) trait RevisionRead: Send + Sync {
/// Plaintext size of the revision.
fn size(&self) -> u64;
/// Plaintext size of each block, in block order.
fn block_sizes(&self) -> &[u64];
async fn read_at(&self, offset: u64, length: u64) -> Result<Vec<u8>>;
}
#[async_trait]
impl RevisionRead for RevisionReader {
fn size(&self) -> u64 {
RevisionReader::size(self)
}
fn block_sizes(&self) -> &[u64] {
RevisionReader::block_sizes(self)
}
async fn read_at(&self, offset: u64, length: u64) -> Result<Vec<u8>> {
RevisionReader::read_at(self, offset, length).await
}
}
#[async_trait]
impl DriveApi for ProtonDriveClient {
async fn get_my_files_folder(&self) -> Result<Node> {
ProtonDriveClient::get_my_files_folder(self).await
}
async fn get_node(&self, uid: &NodeUid) -> Result<Option<Node>> {
ProtonDriveClient::get_node(self, uid).await
}
async fn enumerate_nodes(&self, uids: &[NodeUid]) -> Result<Vec<Node>> {
ProtonDriveClient::enumerate_nodes(self, uids).await
}
async fn enumerate_nodes_light(&self, uids: &[NodeUid]) -> Result<Vec<Node>> {
ProtonDriveClient::enumerate_nodes_light(self, uids).await
}
async fn enumerate_folder_children_node_uids(
&self,
folder_uid: &NodeUid,
) -> Result<Vec<NodeUid>> {
ProtonDriveClient::enumerate_folder_children_node_uids(self, folder_uid).await
}
async fn enumerate_events(
&self,
scope: &DriveEventScopeId,
cursor: Option<&DriveEventId>,
) -> Result<Vec<DriveEvent>> {
ProtonDriveClient::enumerate_events(self, scope, cursor).await
}
async fn invalidate_caches_for_event(&self, event: &DriveEvent) -> Result<()> {
ProtonDriveClient::invalidate_caches_for_event(self, event).await
}
async fn create_folder(
&self,
parent_uid: &NodeUid,
name: &str,
last_modification_time: Option<i64>,
) -> Result<NodeUid> {
ProtonDriveClient::create_folder(self, parent_uid, name, last_modification_time).await
}
async fn upload_file(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
contents: &[u8],
) -> Result<NodeUid> {
ProtonDriveClient::upload_file(self, parent_uid, name, media_type, contents).await
}
async fn upload_file_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
ProtonDriveClient::upload_file_from(
self,
parent_uid,
name,
media_type,
reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
async fn upload_file_replacing_draft_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
ProtonDriveClient::upload_file_replacing_draft_from(
self,
parent_uid,
name,
media_type,
reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
async fn upload_new_revision_from_dyn(
&self,
file_uid: &NodeUid,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()> {
ProtonDriveClient::upload_new_revision_from(
self,
file_uid,
reader,
intended_size,
thumbnails,
last_modification_time,
)
.await
}
async fn open_revision(&self, uid: &NodeUid) -> Result<Arc<dyn RevisionRead>> {
let reader = ProtonDriveClient::open_revision(self, uid).await?;
Ok(Arc::new(reader))
}
async fn download_file_to_dyn(
&self,
uid: &NodeUid,
output: &mut (dyn Write + Send),
) -> Result<()> {
ProtonDriveClient::download_file_to(self, uid, &mut &mut *output).await
}
async fn download_revision_to_dyn(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut (dyn Write + Send),
) -> Result<()> {
ProtonDriveClient::download_revision_to(self, file_uid, revision_id, &mut &mut *writer)
.await
}
async fn rename_node(
&self,
uid: &NodeUid,
new_name: &str,
new_media_type: Option<&str>,
) -> Result<()> {
ProtonDriveClient::rename_node(self, uid, new_name, new_media_type).await
}
async fn move_node(&self, uid: &NodeUid, new_parent: &NodeUid) -> Result<()> {
ProtonDriveClient::move_node(self, uid, new_parent).await
}
fn move_nodes_streaming<'a>(
&'a self,
items: Vec<NodeMoveItem>,
new_parent: NodeUid,
) -> OutcomeStream<'a> {
ProtonDriveClient::move_nodes_streaming(self, items, new_parent).boxed()
}
async fn trash_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes> {
ProtonDriveClient::trash_nodes(self, uids).await
}
async fn restore_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes> {
ProtonDriveClient::restore_nodes(self, uids).await
}
fn restore_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a> {
ProtonDriveClient::restore_nodes_streaming(self, uids).boxed()
}
fn delete_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a> {
ProtonDriveClient::delete_nodes_streaming(self, uids).boxed()
}
async fn enumerate_trash_node_uids(&self) -> Result<Vec<NodeUid>> {
ProtonDriveClient::enumerate_trash_node_uids(self).await
}
async fn quota(&self) -> Result<Quota> {
ProtonDriveClient::quota(self).await
}
}
+257 -222
View File
@@ -72,7 +72,7 @@ impl Filesystem for ProtonFs {
}
fn lookup(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEntry) {
let parent = parent.0;
let parent = self.tree(parent).0;
let name = match fuse_name(name) {
Ok(name) => name,
Err(error) => {
@@ -95,11 +95,13 @@ impl Filesystem for ProtonFs {
}
fn forget(&self, _req: &Request, ino: INodeNo, nlookup: u64) {
let ino = self.tree(ino);
let mut st = self.core.state();
st.forget_lookup(ino.0, nlookup);
}
fn getattr(&self, _req: &Request, ino: INodeNo, _fh: Option<FileHandle>, reply: ReplyAttr) {
let ino = self.tree(ino);
let (attr, provisional_in) = {
let st = self.core.state();
match st.entries.get(&ino.0) {
@@ -175,7 +177,7 @@ impl Filesystem for ProtonFs {
/// Enumerating a cold folder is a remote call, so it goes to a worker for
/// the same reason `lookup` and `readdir` do.
fn opendir(&self, _req: &Request, ino: INodeNo, _flags: OpenFlags, reply: ReplyOpen) {
let ino = ino.0;
let ino = self.tree(ino).0;
if self.core.children_cached(ino) {
self.serve_opendir(ino, reply);
return;
@@ -194,13 +196,13 @@ impl Filesystem for ProtonFs {
offset: u64,
reply: ReplyDirectory,
) {
let ino = ino.0;
let ino = self.tree(ino).0;
// The snapshot is the whole point: served from it, this is a pure
// in-memory walk with no chance of a remote call, so it never needs a
// worker. Only a handle whose `opendir` failed to snapshot (or a kernel
// that issued `readdir` without one) falls back to the live path.
if let Some(listing) = self.core.state().dir_snapshots.get(&fh.0).cloned() {
reply_listing(&listing, offset, reply);
reply_listing(&listing, offset, self.root, reply);
return;
}
// Same split as `lookup`: only the cold, remote-enumerating path pays
@@ -232,6 +234,7 @@ impl Filesystem for ProtonFs {
/// a blob that is the whole file — so it is served from a worker rather than
/// stalling every other request on the mount behind it.
fn open(&self, _req: &Request, ino: INodeNo, flags: OpenFlags, reply: ReplyOpen) {
let ino = self.tree(ino);
if flags.acc_mode() == OpenAccMode::O_RDONLY {
self.serve_open(ino, flags, reply);
return;
@@ -253,6 +256,7 @@ impl Filesystem for ProtonFs {
_lock_owner: Option<LockOwner>,
reply: ReplyData,
) {
let ino = self.tree(ino);
// A file open for writing is served from its handle so reads see the
// in-flight (possibly unsaved) content: authored bytes come from the
// scratch file, untouched bytes from the base.
@@ -290,7 +294,7 @@ impl Filesystem for ProtonFs {
}
return;
}
let (uid, mtime, fsize, is_video) = {
let (uid, mtime, fsize, provisional, is_video) = {
let st = self.core.state();
match st.entries.get(&ino.0) {
Some(e) if e.node.is_file() => {
@@ -300,14 +304,24 @@ impl Filesystem for ProtonFs {
};
let is_video =
PhotoKind::classify(Some(&e.node.name), media_type) == PhotoKind::Video;
let fsize = self
.core
.pending
.lock()
.get(&e.uid)
.map(|p| p.meta.len)
.unwrap_or_else(|| node_size(&e.node));
(e.uid.clone(), e.node.modification_time, fsize, is_video)
let queued = self.core.pending.lock().get(&e.uid).map(|p| p.meta.len);
// Still the cheap enumeration's ciphertext size (B12).
let provisional = queued.is_none()
&& matches!(
&e.node.kind,
NodeKind::File {
claimed_size: None,
..
}
);
let fsize = queued.unwrap_or_else(|| node_size(&e.node));
(
e.uid.clone(),
e.node.modification_time,
fsize,
provisional,
is_video,
)
}
Some(_) => {
reply.error(Errno::EISDIR);
@@ -330,6 +344,13 @@ impl Filesystem for ProtonFs {
// races with nothing. FUSE does not require replies in request order.
let core = self.core.clone();
self.core.workers.run(Lane::Transfer, "read", move || {
// The revision disagrees with a provisional size, and B84 fails
// such a read outright, so settle the size first. On the network,
// hence here and not above (bugs.md B100).
let fsize = match provisional {
true => core.resolve_provisional_size(ino.0, &uid).unwrap_or(fsize),
false => fsize,
};
match core.read_range(&uid, mtime, fsize, offset, size as u64, cache_blocks) {
Ok(bytes) => reply.data(&bytes),
Err(e) => reply.error(e),
@@ -347,7 +368,7 @@ impl Filesystem for ProtonFs {
_flags: i32,
reply: ReplyCreate,
) {
let parent = parent.0;
let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) {
reply.error(error);
return;
@@ -383,6 +404,7 @@ impl Filesystem for ProtonFs {
_lock_owner: Option<LockOwner>,
reply: ReplyWrite,
) {
let ino = self.tree(ino);
if let Err(error) = self.core.require_writable(ino.0) {
reply.error(error);
return;
@@ -425,10 +447,7 @@ impl Filesystem for ProtonFs {
reply.error(Errno::EBADF);
return;
};
let end = offset + data.len() as u64;
aw.written.add(offset, end);
aw.len = aw.len.max(end);
aw.dirty = true;
aw.record_write(offset, offset + data.len() as u64);
let len = aw.len;
st.set_size(ino.0, len);
len
@@ -463,6 +482,7 @@ impl Filesystem for ProtonFs {
_flags: Option<BsdFileFlags>,
reply: ReplyAttr,
) {
let ino = self.tree(ino);
if let Err(error) = self.core.require_writable(ino.0) {
reply.error(error);
return;
@@ -530,6 +550,7 @@ impl Filesystem for ProtonFs {
mode: i32,
reply: ReplyEmpty,
) {
let ino = self.tree(ino);
if let Err(error) = self.core.require_writable(ino.0) {
reply.error(error);
return;
@@ -626,16 +647,17 @@ impl Filesystem for ProtonFs {
fn release(
&self,
_req: &Request,
_ino: INodeNo,
ino: INodeNo,
fh: FileHandle,
_flags: OpenFlags,
_lock_owner: Option<LockOwner>,
_flush: bool,
reply: ReplyEmpty,
) {
let ino = self.tree(ino);
let (handle, unlinked_uid) = {
let mut st = self.core.state();
let unlinked_uid = release_unlinked_entry(&mut st, _ino.0);
let unlinked_uid = release_unlinked_entry(&mut st, ino.0);
let ino = st.handles.remove(&fh.0);
let h = ino.and_then(|i| {
let aw = st.active_writes.get_mut(&i)?;
@@ -666,7 +688,7 @@ impl Filesystem for ProtonFs {
}
fn unlink(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEmpty) {
let parent = parent.0;
let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) {
reply.error(error);
return;
@@ -686,7 +708,7 @@ impl Filesystem for ProtonFs {
}
fn rmdir(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEmpty) {
let parent = parent.0;
let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) {
reply.error(error);
return;
@@ -715,7 +737,7 @@ impl Filesystem for ProtonFs {
_umask: u32,
reply: ReplyEntry,
) {
let parent = parent.0;
let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) {
reply.error(error);
return;
@@ -746,8 +768,8 @@ impl Filesystem for ProtonFs {
flags: RenameFlags,
reply: ReplyEmpty,
) {
let parent = parent.0;
let newparent = newparent.0;
let parent = self.tree(parent).0;
let newparent = self.tree(newparent).0;
if let Err(error) = self
.core
.require_writable(parent)
@@ -780,6 +802,7 @@ impl Filesystem for ProtonFs {
}
fn access(&self, _req: &Request, ino: INodeNo, mask: AccessFlags, reply: ReplyEmpty) {
let ino = self.tree(ino);
let state = self.core.state();
let Some(entry) = state.entries.get(&ino.0) else {
reply.error(Errno::ENOENT);
@@ -793,6 +816,7 @@ impl Filesystem for ProtonFs {
}
fn getxattr(&self, _req: &Request, ino: INodeNo, name: &OsStr, size: u32, reply: ReplyXattr) {
let ino = self.tree(ino);
// Always off the dispatch loop: a miss goes to the wire, and a lister
// that stats a directory issues one of these per file per advertised
// name — inline, that serialized the whole mount behind ~186 ms of
@@ -805,6 +829,7 @@ impl Filesystem for ProtonFs {
}
fn listxattr(&self, _req: &Request, ino: INodeNo, size: u32, reply: ReplyXattr) {
let ino = self.tree(ino);
self.serve_listxattr(ino, size, reply);
}
@@ -954,6 +979,9 @@ pub struct ProtonFs {
core: Core,
uid: u32,
gid: u32,
/// The tree inode of this session's root, which the kernel knows as
/// [`ROOT_INO`].
root: u64,
}
/// How many times a write `open` will re-read its base after a `release`
@@ -994,59 +1022,55 @@ fn build_listing(st: &State, ino: u64) -> DirListing {
/// The cookie handed back is the index of the *next* entry, so a resumed call
/// continues exactly where the last one stopped — which is only true because
/// the listing behind those indexes is a snapshot (bugs.md B43).
fn reply_listing(listing: &DirListing, offset: u64, mut reply: ReplyDirectory) {
fn reply_listing(listing: &DirListing, offset: u64, root: u64, mut reply: ReplyDirectory) {
for (i, (ino, is_dir, name)) in listing.iter().enumerate().skip(offset as usize) {
let ft = if *is_dir {
FileType::Directory
} else {
FileType::RegularFile
};
if reply.add(INodeNo(*ino), (i + 1) as u64, ft, name) {
if reply.add(kernel_ino(root, *ino), (i + 1) as u64, ft, name) {
break;
}
}
reply.ok();
}
/// The inode the kernel knows tree inode `ino` by, in a session rooted at
/// `root`. The inverse of [`ProtonFs::tree`].
pub(crate) fn kernel_ino(root: u64, ino: u64) -> INodeNo {
INodeNo(if ino == root { ROOT_INO } else { ino })
}
pub(crate) fn access_allowed(is_dir: bool, access: Access, mask: AccessFlags) -> bool {
(!mask.contains(AccessFlags::W_OK) || access.writable())
&& (!mask.contains(AccessFlags::X_OK) || is_dir)
}
impl ProtonFs {
/// Build the filesystem rooted at `root` (the user's My Files folder).
/// Build the filesystem rooted at `root` (the user's My Files folder, or
/// the folder an on-demand location shows).
pub(super) fn new(core: Core, root: Node) -> Self {
{
let mut st = core.state();
if let Err(e) = st.db.upsert_node(&root) {
warn!(uid = %root.uid, error = %e, "db upsert root failed");
}
st.by_uid.insert(root.uid.clone(), ROOT_INO);
// The filesystem root is always owned — it must never be
// classified as a shared root, even when the API returns
// membership data (which happens when the owner shares the
// folder with someone else). Clean up any stale share_access
// row that a previous downgrade_all_share_access may have
// written for this uid, so hydrate_access cannot pick it up.
let access = Access::Owner;
st.share_access.remove(&root.uid);
let _ = st.db.delete_share_access(&root.uid);
st.entries.insert(
ROOT_INO,
Entry {
uid: root.uid.clone(),
parent: ROOT_INO,
node: root,
access,
lookup_count: 1,
open_count: 0,
unlinked: false,
},
);
}
let root = core.state().install_root(root);
let _ = core.root.set(root);
// SAFETY: geteuid/getegid are infallible and have no preconditions.
let (uid, gid) = unsafe { (libc::geteuid(), libc::getegid()) };
Self { core, uid, gid }
Self {
core,
uid,
gid,
root,
}
}
/// The tree inode the kernel means by `ino`: this session's root for
/// [`ROOT_INO`], and every other inode as it is.
fn tree(&self, ino: INodeNo) -> INodeNo {
if ino.0 == ROOT_INO {
INodeNo(self.root)
} else {
ino
}
}
/// The body of [`Filesystem::lookup`], on whichever thread ends up serving it.
@@ -1155,7 +1179,7 @@ impl ProtonFs {
let st = self.core.state();
let listing = build_listing(&st, ino);
drop(st);
reply_listing(&listing, offset, reply);
reply_listing(&listing, offset, self.root, reply);
}
fn attr(&self, ino: u64, node: &Node, access: Access) -> FileAttr {
@@ -1168,7 +1192,7 @@ impl ProtonFs {
let mtime = unix_secs(node.modification_time);
let crtime = unix_secs(node.creation_time);
FileAttr {
ino: INodeNo(ino),
ino: kernel_ino(self.root, ino),
size,
blocks: size.div_ceil(512),
atime: mtime,
@@ -1193,7 +1217,7 @@ impl ProtonFs {
/// open.
fn serve_open(&self, ino: INodeNo, flags: OpenFlags, reply: ReplyOpen) {
let write_requested = flags.acc_mode() != OpenAccMode::O_RDONLY;
let (uid, base_mtime, base_size, base_revision_id) = {
let (uid, base_mtime, base_size, provisional, base_revision_id) = {
let mut st = self.core.state();
match st.entries.get_mut(&ino.0) {
Some(e) if e.node.is_file() => {
@@ -1206,6 +1230,14 @@ impl ProtonFs {
e.uid.clone(),
e.node.modification_time,
node_size(&e.node),
// Still the cheap enumeration's ciphertext size (B12).
matches!(
&e.node.kind,
NodeKind::File {
claimed_size: None,
..
}
),
node_revision_id(&e.node),
)
}
@@ -1251,6 +1283,21 @@ impl ProtonFs {
// `reply` is consumed by whichever arm answers, so the loop yields an
// outcome and the answer is sent once, after it.
let outcome = 'install: {
// The handle starts at the base's size. A provisional one is the
// ciphertext size, so the file would grow by the encryption overhead
// and reading the base would fail (bugs.md B100). A queued revision
// stamps a real size, so this only goes to the network for a file
// the listing has not sized yet.
let base_size = match provisional && self.core.pending_blob(&uid).is_none() {
false => base_size,
true => match self.core.resolve_provisional_size(ino.0, &uid) {
Some(size) => size,
None => {
error!(%uid, "could not settle the size of the base for a write open");
break 'install Err(Errno::EIO);
}
},
};
for attempt in 0..OPEN_BASE_ATTEMPTS {
let pending_base = self.core.pending.lock().get(&uid).cloned();
if pending_base.as_ref().is_some_and(|p| !p.meta.complete) {
@@ -1398,7 +1445,7 @@ impl ProtonFs {
len,
base_size,
base_mtime,
complete: authored == [(0, len)],
complete: StagedWrite::covers(&authored, len),
authored,
based_on: self
.core
@@ -1427,6 +1474,12 @@ impl ProtonFs {
return;
}
if let Err(e) = self.core.ensure_children(parent) {
// The parent was looked up a moment ago, so ENOENT here means it
// left the tree in between. Log it: the caller only sees "no such
// file" on a create.
if e == Errno::ENOENT {
warn!(parent, name, "create: the parent folder is no longer known");
}
reply.error(e);
return;
}
@@ -1435,6 +1488,10 @@ impl ProtonFs {
match st.entries.get(&parent) {
Some(e) => e.uid.clone(),
None => {
warn!(
parent,
name, "create: the parent folder left the tree during its listing"
);
reply.error(Errno::ENOENT);
return;
}
@@ -1454,67 +1511,58 @@ impl ProtonFs {
//
// A parent that is itself still queued forces the same path even when we
// are online: the API has no folder to put this in yet.
let node =
if !transient && self.core.online.load(Ordering::Relaxed) && !is_local_uid(&parent_uid)
{
if let Err(error) = self.core.require_uid_writable(&parent_uid) {
reply.error(error);
//
// A create that set out online and lost the network on the way lands in
// the queue too, so a Wi-Fi drop costs the caller a wait, never an EIO.
let minted = if !transient && self.core.is_online() && !is_local_uid(&parent_uid) {
if let Err(error) = self.core.require_uid_writable(&parent_uid) {
reply.error(error);
return;
}
// Create an empty file on the remote so it has a real uid immediately;
// written bytes are buffered and sealed as a new revision on close.
match self.core.block_on_bounded(self.core.drive.upload_file(
&parent_uid,
name,
media_type_for(name),
b"",
)) {
// A file that exists remotely but could not be read back is
// queued, and the queued create finds it by name and adopts it.
Ok(new_uid) => self.core.fetch_minted_node(&new_uid, name),
Err(e) if self.core.lost_link(&e, "create") => None,
Err(e) => {
error!(%parent_uid, name, error = %e, "create file failed");
reply.error(Errno::EIO);
return;
}
// Create an empty file on the remote so it has a real uid immediately;
// written bytes are buffered and sealed as a new revision on close.
let new_uid = match self.core.rt.block_on(self.core.client.upload_file(
&parent_uid,
name,
media_type_for(name),
b"",
)) {
Ok(u) => u,
Err(e) => {
error!(%parent_uid, name, error = %e, "create file failed");
reply.error(Errno::EIO);
return;
}
};
match self.core.fetch_node(&new_uid) {
Ok(n) => n,
Err(e) => {
reply.error(e);
return;
}
}
} else {
None
};
let node = match minted {
Some(node) => node,
None => match self
.core
.queue_local_node(&parent_uid, name, false, transient)
{
Ok(n) => n,
Err(e) => {
reply.error(e);
return;
}
} else {
match self
.core
.queue_local_node(&parent_uid, name, false, transient)
{
Ok(n) => n,
Err(e) => {
reply.error(e);
return;
}
}
};
let new_uid = node.uid.clone();
// The base this handle writes over is the node as it actually exists —
// the empty file the server just minted — so its modification time comes
// from the node, never from the local clock. `queue_revision` turns this
// into `StagedWrite::based_on` and the drain compares that against the
// remote: a `now_secs()` here differs from the server's stamp by however
// long the create round-trip took, so the first write to a brand-new file
// conflicts with its own create whenever the second happens to tick in
// between.
let base_mtime = node.modification_time;
},
};
let (file, path) = match self.core.cache.create_scratch() {
Ok(x) => x,
Err(e) => {
error!(%new_uid, error = %e, "create scratch file failed");
error!(uid = %node.uid, error = %e, "create scratch file failed");
reply.error(Errno::EIO);
return;
}
};
let mut st = self.core.state();
let ino = st.intern(parent, node);
let ino = st.intern(parent, node.clone());
if let Some(entry) = st.entries.get_mut(&ino) {
entry.open_count = entry.open_count.saturating_add(1);
}
@@ -1525,23 +1573,10 @@ impl ProtonFs {
}
let fh = st.next_fh;
st.next_fh += 1;
let aw = st.active_writes.entry(ino).or_insert_with(|| {
// A brand-new file: empty base, everything written is authored.
WriteHandle {
ino,
uid: new_uid,
file: Arc::new(file),
path,
written: Intervals::default(),
len: 0,
base_size: 0,
base_mtime,
// A brand-new file has no sealed remote revision to conflict with.
base_revision_id: None,
dirty: false,
open_count: 0,
}
});
let aw = st
.active_writes
.entry(ino)
.or_insert_with(|| WriteHandle::created(ino, &node, file, path));
aw.open_count += 1;
st.handles.insert(fh, ino);
let entry = st.entries.get(&ino).unwrap();
@@ -1584,39 +1619,36 @@ impl ProtonFs {
// As in `create`: offline — or under a parent that is itself still
// queued — the folder becomes a placeholder that the drain turns into a
// real one (offline.md Phase 3b).
let node = if self.core.online.load(Ordering::Relaxed) && !is_local_uid(&parent_uid) {
// Losing the network on the way queues it the same way.
let minted = if self.core.is_online() && !is_local_uid(&parent_uid) {
if let Err(error) = self.core.require_uid_writable(&parent_uid) {
reply.error(error);
return;
}
let new_uid =
match self
.core
.rt
.block_on(self.core.client.create_folder(&parent_uid, name, now))
{
Ok(u) => u,
Err(e) => {
error!(%parent_uid, name, error = %e, "create folder failed");
reply.error(Errno::EIO);
return;
}
};
match self.core.fetch_node(&new_uid) {
Ok(n) => n,
match self
.core
.block_on_bounded(self.core.drive.create_folder(&parent_uid, name, now))
{
Ok(new_uid) => self.core.fetch_minted_node(&new_uid, name),
Err(e) if self.core.lost_link(&e, "mkdir") => None,
Err(e) => {
reply.error(e);
error!(%parent_uid, name, error = %e, "create folder failed");
reply.error(Errno::EIO);
return;
}
}
} else {
match self.core.queue_local_node(&parent_uid, name, true, false) {
None
};
let node = match minted {
Some(node) => node,
None => match self.core.queue_local_node(&parent_uid, name, true, false) {
Ok(n) => n,
Err(e) => {
reply.error(e);
return;
}
}
},
};
let mut st = self.core.state();
let local = is_local_uid(&node.uid);
@@ -1706,12 +1738,18 @@ impl ProtonFs {
if let Some(uid) = unlinked_uid
&& release_can_discard_unlinked(&self.core.db, &uid)
{
// Ops withdrawn at unlink left the bytes this handle read behind,
// owned by no op any more; they go with the last handle.
let withdrawn = self.core.pending_blob(&uid);
if let Err(error) = self.core.discard_queued_ops(&uid) {
warn!(%uid, ?error, "queued-op cleanup failed; retaining unlinked state");
} else {
if let Err(error) = self.core.db.delete_node(&uid) {
warn!(%uid, %error, "deleting released unlinked node state failed");
}
if let Some(blob) = withdrawn {
self.core.cache.discard_staged(&blob);
}
self.core.cache.evict(&uid);
self.core.evict_reader(&uid);
}
@@ -1916,12 +1954,14 @@ impl ProtonFs {
// now and this handle's is stale. A retry resolves it.
Ok(false) => {
warn!(%uid, name, newname, "queued create vanished under a rename");
self.core.restore_replaced(victim.as_ref(), newname);
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EBUSY);
}
Err(e) => {
error!(%uid, error = %e, "rewriting a queued create's target failed");
self.core.restore_replaced(victim.as_ref(), newname);
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EIO);
}
}
@@ -1932,12 +1972,11 @@ impl ProtonFs {
// take the synchronous path, so a genuine API refusal (permissions, a
// name clash) surfaces to the caller instead of becoming a queued op
// that can only ever conflict.
if rename_needs_queue(
self.core.online.load(Ordering::Relaxed),
is_local_uid(&new_parent_uid),
newparent != parent,
newname != name,
) {
// The queued rename, as a closure: the network going away under the
// synchronous path below lands here too, with whatever half already
// reached Drive. The op records the end state, so the drain finishes
// the rest.
let queue_it = |reply: ReplyEmpty| {
let queued = if replacement_authorized {
self.core.queue_rename_authorized(
ino,
@@ -1965,23 +2004,30 @@ impl ProtonFs {
// and retain an empty destination readdir page.
reply.ok();
if let Some(notifier) = self.core.notifier.get() {
let _ = notifier.inval_entry(INodeNo(parent), OsStr::new(&name));
let _ = notifier.inval_entry(INodeNo(newparent), OsStr::new(&newname));
let _ = notifier.inval_inode(INodeNo(parent), 0, 0);
let _ = notifier.inval_entry(parent, OsStr::new(&name));
let _ = notifier.inval_entry(newparent, OsStr::new(&newname));
let _ = notifier.inval_inode(parent);
if newparent != parent {
let _ = notifier.inval_inode(INodeNo(newparent), 0, 0);
let _ = notifier.inval_inode(newparent);
}
}
}
Err(e) => {
self.core.restore_replaced(victim.as_ref(), newname);
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(e);
}
}
};
if rename_needs_queue(
self.core.is_online(),
is_local_uid(&new_parent_uid),
newparent != parent,
newname != name,
) {
queue_it(reply);
return;
}
// Rename first if both halves change. Moving first makes the encrypted
// name requirements stale and repeatedly failed with InvalidRequirements.
// A failure past this point has already trashed any node the rename was
// replacing, so put it back rather than leaving the caller with neither
// the source moved nor the destination intact.
@@ -1991,64 +2037,37 @@ impl ProtonFs {
&old_parent_uid,
&new_parent_uid,
) {
self.core.restore_replaced(victim.as_ref(), newname);
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(error);
return;
}
if newname != name
&& let Err(e) = self
.core
.rt
.block_on(self.core.client.rename_node(&uid, newname, None))
{
// A move that also renames is one request, so the name only has to be
// free in the destination, and the reply never stands for half of it.
let renamed = if newparent != parent {
let target = (newname != name).then_some(newname);
self.core.move_rename_remote(&uid, &new_parent_uid, target)
} else if newname != name {
self.core.rename_remote(&uid, newname)
} else {
Ok(())
};
if let Err(e) = renamed {
if self.core.lost_link(&e, "rename") {
queue_it(reply);
return;
}
error!(%uid, error = %e, "rename failed");
self.core.restore_replaced(victim.as_ref(), newname);
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EIO);
return;
}
if newparent != parent {
let mut attempts = 0u32;
let moved = loop {
match self
.core
.rt
.block_on(self.core.client.move_node(&uid, &new_parent_uid))
{
Ok(()) => break Ok(()),
Err(e)
if newname != name
&& api_code(&e) == Some(ResponseCode::InvalidRequirements)
&& attempts < 20 =>
{
attempts += 1;
// Requirement propagation routinely takes longer than
// three seconds. Keep a bounded ten-second window; a
// successful POSIX reply must mean both remote halves
// landed, not merely that the second half was queued.
std::thread::sleep(Duration::from_millis(500));
}
Err(e) => break Err(e),
}
};
if let Err(e) = moved {
error!(%uid, attempts, error = %e, "move after rename failed");
// The rename half landed in the source directory.
let mut state = self.core.state();
let old_parent_uid = state
.entries
.get(&parent)
.map(|entry| entry.uid.clone())
.unwrap_or_else(|| uid.clone());
state.rename_in_place(ino, parent, &old_parent_uid, newname);
drop(state);
self.core.restore_replaced(victim.as_ref(), newname);
reply.error(Errno::EIO);
return;
}
}
// Through the guard, so the database has both folders unlisted before
// the reply. The next lookup rebuilds a folder still marked listed from
// the database, which did not have the move yet (docs/BUGS.md B135).
self.core
.state
.lock()
.state()
.relocate(ino, parent, newparent, &new_parent_uid, newname);
reply.ok();
}
@@ -2092,7 +2111,12 @@ impl ProtonFs {
// it just means its queued creation is no longer wanted. This works
// offline, which the remote path below cannot (offline.md Phase 3b).
if is_local_uid(&uid) {
if !open_now && let Err(error) = self.core.discard_queued_ops(&uid) {
let dropped = if open_now {
self.core.withdraw_queued_ops(&uid)
} else {
self.core.discard_queued_ops(&uid)
};
if let Err(error) = dropped {
reply.error(error);
return;
}
@@ -2103,12 +2127,14 @@ impl ProtonFs {
}
// Offline: queue it. Trashing is the one mutation a user expects to work
// regardless — the file is gone from their point of view the moment the
// command returns (offline.md Phase 3b).
if !self.core.online.load(Ordering::Relaxed) {
match self.core.queue_trash(&uid, name) {
Ok(()) => reply.ok(),
Err(e) => reply.error(e),
}
// command returns (offline.md Phase 3b). The same when the network goes
// away under the remote call below.
let queue_it = |reply: ReplyEmpty| match self.core.queue_trash(&uid, name) {
Ok(()) => reply.ok(),
Err(e) => reply.error(e),
};
if !self.core.is_online() {
queue_it(reply);
return;
}
if let Err(e) = self.core.require_uid_writable(&uid) {
@@ -2117,10 +2143,13 @@ impl ProtonFs {
}
if let Err(e) = self
.core
.rt
.block_on(self.core.client.trash_nodes(std::slice::from_ref(&uid)))
.block_on_bounded(self.core.drive.trash_nodes(std::slice::from_ref(&uid)))
.and_then(batch::into_unit)
{
if self.core.lost_link(&e, "trash") {
queue_it(reply);
return;
}
error!(%uid, error = %e, "trash failed");
self.core
.log_activity(ActivityKind::Trash, name, e.to_string(), false);
@@ -2131,7 +2160,13 @@ impl ProtonFs {
// Otherwise an EIO response still made the path disappear locally.
self.core.hidden.lock().insert(uid.clone());
self.core.state().forget_or_unlink(&uid);
if !open_now {
if open_now {
// The open handle keeps the staged bytes; the drain must not send
// them to a node that is already in the trash (docs/BUGS.md B106).
if let Err(error) = self.core.withdraw_queued_ops(&uid) {
warn!(%uid, ?error, "remote trash landed but queued-op cleanup failed");
}
} else {
if let Err(error) = self.core.discard_queued_ops(&uid) {
warn!(%uid, ?error, "remote trash landed but queued-op cleanup failed");
}
+1693 -728
View File
File diff suppressed because it is too large. Load diff
+316
View File
@@ -0,0 +1,316 @@
//! Noticing that the network went away mid-session, and that it came back.
//!
//! The mount used to learn it was offline only at startup: a daemon that had
//! connected once kept `online` true for the rest of its life, so a Wi-Fi drop
//! turned every create, mkdir, delete and rename into a two-minute wait for an
//! `EIO`, while the queue that exists for exactly this sat unused. Now any
//! remote call a handler waits on reports a failure to reach Proton through
//! [`Core::lost_link`], which flips the mount offline; the handler then takes its
//! queued path, and the probe thread here flips it back when Proton answers.
use std::future::Future;
use std::sync::atomic::Ordering;
use std::time::{Duration, Instant};
use parking_lot::{Condvar, Mutex};
use proton_drive_rs::proton_sdk::api::ResponseCode;
use proton_drive_rs::proton_sdk::error::{ProtonApiError, ProtonError};
use tracing::{debug, info, warn};
use crate::{Core, ONLINE_PROBE_MAX, ONLINE_PROBE_MIN};
/// How long a remote call a FUSE handler is waiting on may take before the link
/// is treated as down.
///
/// The SDK's own budget for one request is four attempts of up to 30 seconds
/// each, so a dead link held the caller for about two minutes. A metadata call
/// that has had no answer in 20 seconds is not going to be one the user wants to
/// keep waiting for; queueing the change is the better answer.
pub(crate) const INTERACTIVE_CALL_TIMEOUT: Duration = Duration::from_secs(20);
/// How long one probe may wait for an answer. A probe is a single small request,
/// and a probe stuck in the SDK's retry budget would delay noticing the link is
/// back by as long as it waits.
const PROBE_TIMEOUT: Duration = Duration::from_secs(10);
/// First delay between probes after a drop seen mid-session, and the ceiling it
/// doubles to while the outage is young. A blip should end in seconds, not in
/// the minutes [`ONLINE_PROBE_MAX`] allows.
const PROBE_AFTER_DROP: Duration = Duration::from_secs(2);
const PROBE_YOUNG_MAX: Duration = Duration::from_secs(30);
/// How long an outage counts as young. Past it the probe backs off to
/// [`ONLINE_PROBE_MAX`], because a laptop can sit offline for days and each
/// probe is a real API round trip.
const YOUNG_OUTAGE: Duration = Duration::from_secs(10 * 60);
/// How often the idle probe thread checks for teardown while the link is up.
const IDLE_CHECK: Duration = Duration::from_secs(1);
/// The shared half of the online flag: when the link went down, and a condvar
/// that fires on every change so waiters need not poll.
#[derive(Default)]
pub(crate) struct Link {
lost_at: Mutex<Option<Instant>>,
changed: Condvar,
/// Folders listed from the DB alone while offline, to relist once back.
pub(crate) stale_listings: Mutex<std::collections::HashSet<u64>>,
}
/// Whether a failed call failed because Proton could not be reached, rather
/// than because it answered and refused.
///
/// A transport error is the network. So is a request timeout: the SDK reports
/// Proton's own "try again" codes that way, and [`no_answer`] reports our
/// deadline that way. Anything else Proton actually said, and says again on a
/// retry, so it must reach the caller rather than be queued.
pub(crate) fn is_network_error(e: &ProtonError) -> bool {
match e {
ProtonError::Transport(_) => true,
ProtonError::Api(api) => {
matches!(
api.code,
ResponseCode::RequestTimeout | ResponseCode::Timeout
)
}
_ => false,
}
}
/// The error a call reports when it runs past `after` without an answer.
pub(crate) fn no_answer(after: Duration) -> ProtonError {
ProtonError::Api(ProtonApiError {
code: ResponseCode::RequestTimeout,
http_status: 408,
message: format!("no answer within {}s", after.as_secs()),
details: None,
})
}
/// Run a remote call, giving up after [`INTERACTIVE_CALL_TIMEOUT`].
pub(crate) async fn bounded<T>(
call: impl Future<Output = Result<T, ProtonError>>,
) -> Result<T, ProtonError> {
match tokio::time::timeout(INTERACTIVE_CALL_TIMEOUT, call).await {
Ok(result) => result,
Err(_) => Err(no_answer(INTERACTIVE_CALL_TIMEOUT)),
}
}
/// The next delay between probes, given the current one and how long the
/// outage has lasted.
fn next_probe_delay(delay: Duration, outage: Duration) -> Duration {
let cap = if outage < YOUNG_OUTAGE {
PROBE_YOUNG_MAX
} else {
ONLINE_PROBE_MAX
};
(delay * 2).min(cap)
}
impl Core {
/// Whether the mount currently believes Proton is reachable.
pub(crate) fn is_online(&self) -> bool {
self.online.load(Ordering::Relaxed)
}
/// Run a remote call a caller is waiting on, giving up after
/// [`INTERACTIVE_CALL_TIMEOUT`]. The timer is built inside the runtime: a
/// FUSE worker has no reactor of its own.
pub(crate) fn block_on_bounded<T>(
&self,
call: impl Future<Output = Result<T, ProtonError>>,
) -> Result<T, ProtonError> {
self.rt.block_on(bounded(call))
}
/// Report a failed remote call. Returns whether it failed for want of a
/// network, in which case the mount is now offline and the caller should
/// take its queued path instead of failing.
pub(crate) fn lost_link(&self, e: &ProtonError, what: &str) -> bool {
if !is_network_error(e) {
return false;
}
self.mark_offline(what, e);
true
}
/// Flip the mount offline and wake the probe. Only the first caller of an
/// outage logs it; the rest are the same outage.
pub(crate) fn mark_offline(&self, what: &str, error: &dyn std::fmt::Display) {
let mut lost_at = self.link.lost_at.lock();
if self.online.swap(false, Ordering::Relaxed) {
*lost_at = Some(Instant::now());
warn!(during = what, %error,
"lost the connection to Proton; queueing changes until it returns");
self.link.changed.notify_all();
}
}
/// Flip the mount back online, wake everything waiting for it, and let the
/// queue drain.
fn mark_online(&self) {
{
let mut lost_at = self.link.lost_at.lock();
self.online.store(true, Ordering::Relaxed);
match lost_at.take() {
Some(since) => {
info!(
offline_secs = since.elapsed().as_secs(),
"connection to Proton is back"
)
}
None => info!("back online"),
}
self.link.changed.notify_all();
}
self.relist_offline_listings();
// Anything written while offline is queued and waiting on exactly this,
// including ops that failed on the way down and are sitting out a
// backoff sized for a server that keeps refusing.
if let Err(e) = self.db.retry_failed_ops_now(crate::now_millis()) {
warn!(error = %e, "moving backed-off ops up after a reconnect failed");
}
self.wake_drain();
}
/// Wait up to `budget` for the link to come back. Returns whether it is up.
pub(crate) fn wait_online(&self, budget: Duration) -> bool {
let deadline = Instant::now() + budget;
let mut lost_at = self.link.lost_at.lock();
while !self.is_online() {
if self.shutdown.is_stopping()
|| self
.link
.changed
.wait_until(&mut lost_at, deadline)
.timed_out()
{
return self.is_online();
}
}
true
}
/// Wait out `interval` before a background poll of Proton, and while the
/// link is down, until it is back. Returns as soon as it comes back, so the
/// poll catches up on what changed during the outage at once instead of
/// one interval later. A poll into a dead link would only wait out the
/// SDK's retries and fail.
pub(crate) async fn next_poll(&self, interval: Duration) {
let due = Instant::now() + interval;
let mut was_offline = false;
loop {
if !self.is_online() {
was_offline = true;
} else if was_offline || Instant::now() >= due {
return;
}
tokio::time::sleep(IDLE_CHECK).await;
}
}
/// Watch the link for the life of the mount: sleep while it is up, and
/// probe it back while it is down (offline.md Phase 1).
///
/// A mount that started from the cache is down from the first moment; one
/// that started online goes down when [`Core::mark_offline`] says so.
pub(crate) fn run_online_probe(&self) {
// Only an offline start never had a chance to repair its share id.
let mut repair_share_id = !self.is_online();
loop {
{
let mut lost_at = self.link.lost_at.lock();
while self.is_online() {
if self.shutdown.is_stopping() {
return;
}
self.link.changed.wait_for(&mut lost_at, IDLE_CHECK);
}
}
if !self.probe_until_online() {
return;
}
if std::mem::take(&mut repair_share_id) {
self.repair_primary_share_id(&self.primary_root_uid);
}
}
}
/// Probe until Proton answers, then flip the mount online. Returns `false`
/// if the daemon is stopping.
fn probe_until_online(&self) -> bool {
let started = self.link.lost_at.lock().unwrap_or_else(Instant::now);
let mut delay = if started.elapsed() < YOUNG_OUTAGE {
PROBE_AFTER_DROP
} else {
ONLINE_PROBE_MIN
};
loop {
if !self.shutdown.sleep(delay) {
return false;
}
let probe = self.rt.block_on(async {
match tokio::time::timeout(PROBE_TIMEOUT, self.drive.get_my_files_folder()).await {
Ok(result) => result,
Err(_) => Err(no_answer(PROBE_TIMEOUT)),
}
});
match probe {
Ok(root) => {
{
let root_ino = self.root_ino();
let mut st = self.state();
if let Some(e) = st.entries.get_mut(&root_ino) {
e.node = root.clone();
}
}
if let Err(e) = self.db.upsert_node(&root) {
warn!(error = %e, "refresh root after reconnect failed");
}
self.mark_online();
return true;
}
Err(e) => {
debug!(error = %e, ?delay, "online probe failed; still offline");
delay = next_probe_delay(delay, started.elapsed());
}
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn api(code: ResponseCode) -> ProtonError {
ProtonError::Api(ProtonApiError {
code,
http_status: 422,
message: String::new(),
details: None,
})
}
#[test]
fn only_an_unanswered_call_counts_as_a_lost_link() {
assert!(is_network_error(&no_answer(INTERACTIVE_CALL_TIMEOUT)));
assert!(is_network_error(&api(ResponseCode::Timeout)));
assert!(!is_network_error(&api(ResponseCode::AlreadyExists)));
assert!(!is_network_error(&api(ResponseCode::InvalidRequirements)));
assert!(!is_network_error(&ProtonError::invalid_operation("bug")));
}
#[test]
fn a_young_outage_is_probed_often_and_an_old_one_rarely() {
let young = Duration::from_secs(60);
let old = YOUNG_OUTAGE + Duration::from_secs(1);
assert_eq!(
next_probe_delay(PROBE_AFTER_DROP, young),
2 * PROBE_AFTER_DROP
);
assert_eq!(next_probe_delay(PROBE_YOUNG_MAX, young), PROBE_YOUNG_MAX);
assert_eq!(next_probe_delay(PROBE_YOUNG_MAX, old), 2 * PROBE_YOUNG_MAX);
assert_eq!(next_probe_delay(ONLINE_PROBE_MAX, old), ONLINE_PROBE_MAX);
}
}
+95 -40
View File
@@ -207,11 +207,11 @@ const ROOT_UID_KEY: &str = "root_uid";
/// means an empty tree, and mounting that would show the user an empty Drive
/// rather than an honest error.
fn fetch_or_recall_root(
client: &ProtonDriveClient,
drive: &dyn DriveApi,
rt: &tokio::runtime::Handle,
db: &Db,
) -> std::io::Result<(Node, bool)> {
let err = match rt.block_on(client.get_my_files_folder()) {
let err = match rt.block_on(drive.get_my_files_folder()) {
Ok(root) => {
if let Err(e) = db.set_state_str(ROOT_UID_KEY, &root.uid.to_string()) {
warn!(error = %e, "persist root uid failed");
@@ -276,13 +276,14 @@ pub(super) fn spawn_session(
info!(mountpoint = %mountpoint.display(), "mounting Proton Drive location");
let fs = ProtonFs::new(core.clone(), root);
if core.primary {
// Only My Files rehydrates the daemon-wide node cache. A secondary
// inode space is rooted at one device subtree and must not import every
// unrelated persisted node.
// Only My Files rehydrates the daemon-wide node cache, once. A
// secondary mount shares the tree it fills.
core.hydrate();
}
let session = Session::new(fs, mountpoint, &config)?.spawn()?;
let _ = core.notifier.set(session.notifier());
let _ = core
.notifier
.set(KernelChannel::new(session.notifier(), core.root_ino()));
core.session_live.store(true, Ordering::Release);
Ok(session)
}
@@ -339,6 +340,45 @@ pub fn mount(
control_socket: &Path,
db: Arc<Db>,
options: MountOptions,
) -> std::io::Result<MountOutcome> {
let drive = Arc::new(client.clone());
mount_with(
client,
drive,
rt,
mountpoint,
cache,
control_socket,
db,
options,
Host::Daemon,
)
}
/// What a mount runs in.
pub(crate) enum Host {
/// The daemon: stopped by SIGTERM or SIGINT, and indexing the user's home
/// for the launcher prompt.
Daemon,
/// One of several simulated daemons in a test process (`sim`): stopped by
/// a message on this channel, and with no home to index.
#[cfg(test)]
Simulation(std::sync::mpsc::Receiver<()>),
}
/// [`mount`], reaching Drive through `drive` and hosted by `host`; `client`
/// serves only what [`DriveApi`] does not cover.
#[allow(clippy::too_many_arguments)]
pub(crate) fn mount_with(
client: ProtonDriveClient,
drive: Arc<dyn DriveApi>,
rt: tokio::runtime::Handle,
mountpoint: &Path,
cache: ContentCache,
control_socket: &Path,
db: Arc<Db>,
options: MountOptions,
host: Host,
) -> std::io::Result<MountOutcome> {
let MountOptions {
username,
@@ -349,7 +389,7 @@ pub fn mount(
// Start the uptime clock here rather than at the first `pdfs diagnostics`,
// so the age it reports is the daemon's own.
super::diagnostics::start_clock();
let (root, online) = fetch_or_recall_root(&client, &rt, &db)?;
let (root, online) = fetch_or_recall_root(drive.as_ref(), &rt, &db)?;
let scope = root.tree_event_scope_id();
db.mount_upsert_my_files(&mountpoint.to_string_lossy(), &root.uid.to_string(), None)
.map_err(|error| std::io::Error::other(format!("project My Files mount: {error}")))?;
@@ -364,12 +404,14 @@ pub fn mount(
let paused_until = pause::load_paused_until(&db);
let core = Core {
client: client.clone(),
drive,
client,
rt: rt.clone(),
maintenance: Arc::new(Mutex::new(Default::default())),
primary_root_uid: root.uid.clone(),
primary: true,
state: Arc::new(Mutex::new(State::new(db.clone(), share_access, 2))),
state: Arc::new(Mutex::new(State::new(db.clone(), share_access))),
root: Arc::new(OnceLock::new()),
cache: Arc::new(cache),
readers: Arc::new(Mutex::new(HashMap::new())),
block_ring: Arc::new(Mutex::new(BlockRing::default())),
@@ -383,6 +425,7 @@ pub fn mount(
shared_generation: Arc::new(AtomicU64::new(0)),
shared_refresh_deadlines: Arc::new(Mutex::new(SharedRefreshDeadlines::default())),
online: Arc::new(AtomicBool::new(online)),
link: Arc::new(Default::default()),
sync_paused_until: Arc::new(AtomicI64::new(paused_until)),
pending: Arc::new(Mutex::new(HashMap::new())),
hidden: Arc::new(Mutex::new(HashSet::new())),
@@ -390,6 +433,7 @@ pub fn mount(
shutdown: Arc::new(crate::shutdown::Shutdown::default()),
upload_times: Arc::new(Mutex::new(HashMap::new())),
upload_cancel: Arc::new(Mutex::new(HashMap::new())),
landing_uploads: Arc::new(AtomicU64::new(0)),
timeline_refreshing: Arc::new(AtomicBool::new(false)),
timeline_progress: Arc::new(Mutex::new(None)),
similar_hashing: Arc::new(AtomicBool::new(false)),
@@ -398,7 +442,6 @@ pub fn mount(
trash_progress: Arc::new(tokio::sync::Notify::new()),
conflict_notified: Arc::new(Mutex::new(HashSet::new())),
sweep_mode,
own_sealed_revs: Arc::new(Mutex::new(HashMap::new())),
self_changes: Arc::new(Mutex::new(HashMap::new())),
thumb_gen: Arc::new(Mutex::new(HashSet::new())),
thumb_gen_budget: Arc::new(tokio::sync::Semaphore::new(
@@ -411,6 +454,7 @@ pub fn mount(
thumbnail_build_cancel: Arc::new(tokio::sync::Notify::new()),
thumbnail_misses: Arc::new(Mutex::new(Default::default())),
quota: Arc::new(Mutex::new(None)),
quota_refreshing: Arc::new(AtomicBool::new(false)),
size_upgrades: Arc::new(Mutex::new(HashMap::new())),
size_waiters: Arc::new(Default::default()),
notifier: Arc::new(OnceLock::new()),
@@ -425,7 +469,7 @@ pub fn mount(
states: Arc::new(StateRegistry::default()),
};
// Before anything can queue work against it: the drain thread below reaches
// every mount's inode space through this registry, not through `core.state`.
// every mount's session through this registry.
core.register_state(mountpoint);
core.transfers.set_limits(upload_limit, download_limit);
@@ -484,9 +528,10 @@ pub fn mount(
);
}
// Mounted from the cache: watch for the network coming back so the mount can
// stop being read-only-ish without the user restarting the daemon.
if !online {
// Watch the link for the life of the mount: a mount from the cache waits for
// the network to come back, and an online one can lose it at any moment and
// must notice it coming back just the same.
{
let core = core.clone();
workers.push(
std::thread::Builder::new()
@@ -497,7 +542,7 @@ pub fn mount(
// Keep the launcher prompt's "This computer" index fresh. Its own thread:
// the walk is I/O-heavy and must never sit in front of a FUSE callback.
{
if matches!(host, Host::Daemon) {
let db = core.db.clone();
let indexing = core.indexing.clone();
let transfers = core.transfers.clone();
@@ -621,7 +666,7 @@ pub fn mount(
}
}
};
rt.spawn(run_event_sync(client, scope, core.clone()));
rt.spawn(run_event_sync(core.drive.clone(), scope, core.clone()));
// The photos volume has its own event stream; without it the gallery only
// learns about a phone-side deletion when the timeline goes stale.
rt.spawn(run_photos_event_sync(core.clone()));
@@ -640,30 +685,11 @@ pub fn mount(
// are delivered onto the async runtime; bridge them onto a sync channel so
// the loop below can react without blocking a worker thread. A bounded
// channel of 1 is enough — we only need to know that *a* stop arrived.
let (sig_tx, sig_rx) = std::sync::mpsc::sync_channel::<()>(1);
rt.spawn(async move {
let mut sigterm =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGTERM handler failed");
return;
}
};
let mut sigint =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGINT handler failed");
return;
}
};
tokio::select! {
_ = sigterm.recv() => info!("received SIGTERM"),
_ = sigint.recv() => info!("received SIGINT"),
}
let _ = sig_tx.try_send(());
});
let sig_rx = match host {
Host::Daemon => signal_stop(&rt),
#[cfg(test)]
Host::Simulation(rx) => rx,
};
// Wait for whichever happens first: a stop signal (→ we unmount ourselves
// via the lazy MNT_DETACH path, clean even mid-download), or the kernel
@@ -720,6 +746,35 @@ pub fn mount(
Ok(outcome)
}
/// A channel that receives once SIGTERM or SIGINT arrives.
fn signal_stop(rt: &tokio::runtime::Handle) -> std::sync::mpsc::Receiver<()> {
let (sig_tx, sig_rx) = std::sync::mpsc::sync_channel::<()>(1);
rt.spawn(async move {
let mut sigterm =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGTERM handler failed");
return;
}
};
let mut sigint =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGINT handler failed");
return;
}
};
tokio::select! {
_ = sigterm.recv() => info!("received SIGTERM"),
_ = sigint.recv() => info!("received SIGINT"),
}
let _ = sig_tx.try_send(());
});
sig_rx
}
/// How long teardown waits for the FUSE worker pool before proceeding without
/// it. Long enough for an in-flight block to finish, short enough that
/// `systemctl --user stop` still returns inside `TimeoutStopSec`.
+1 -1
View File
@@ -1038,7 +1038,7 @@ impl Core {
}
let outcomes = self
.rt
.block_on(self.client.trash_nodes(uids))
.block_on(self.drive.trash_nodes(uids))
.map_err(|e| CoreError::from_api(&e, "trash"))?;
let (trashed, failed) = pdfs_core::batch::split(outcomes);
+8 -8
View File
@@ -101,7 +101,7 @@ impl Core {
match existing {
Some(uid) => self
.rt
.block_on(self.client.upload_new_revision_from(
.block_on(self.drive.upload_new_revision_from(
&uid,
Cursor::new(bytes),
len,
@@ -112,7 +112,7 @@ impl Core {
.map_err(|e| CoreError::from_api(&e, "upload profile revision"))?,
None => {
self.rt
.block_on(self.client.upload_file_replacing_draft_from(
.block_on(self.drive.upload_file_replacing_draft_from(
&dir_uid,
PROFILE_FILE_NAME,
"application/json",
@@ -182,14 +182,14 @@ impl Core {
) -> CoreResult<Option<NodeUid>> {
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(parent_uid))
.block_on(self.drive.enumerate_folder_children_node_uids(parent_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() {
return Ok(None);
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&uids))
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?;
Ok(nodes
.into_iter()
@@ -210,7 +210,7 @@ impl Core {
let uid = self
.rt
.block_on(
self.client
self.drive
.create_folder(root_uid, PROFILE_DIR_NAME, Some(now_secs())),
)
.map_err(|e| CoreError::from_api(&e, "create profile folder"))?;
@@ -241,7 +241,7 @@ impl Core {
};
let mut buf: Vec<u8> = Vec::new();
self.rt
.block_on(self.client.download_file_to(&uid, &mut buf))
.block_on(self.drive.download_file_to(&uid, &mut buf))
.map_err(|e| CoreError::from_api(&e, "download profile"))?;
Profile::parse(&buf).map(Some).map_err(CoreError::invalid)
}
@@ -277,13 +277,13 @@ impl Core {
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(&root_uid))
.block_on(self.drive.enumerate_folder_children_node_uids(&root_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?;
let nodes = if uids.is_empty() {
Vec::new()
} else {
self.rt
.block_on(self.client.enumerate_nodes(&uids))
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?
};
+87 -31
View File
@@ -36,6 +36,21 @@ pub(super) const REPAIR_MAX: u64 = 512 * 1024 * 1024;
/// demand read in the SDK's flat block semaphore.
pub(super) const PREFETCH_BUDGET: usize = 8;
/// How long one read may wait on the network before it fails with `EIO`.
///
/// The SDK bounds each request, not a read: a block that times out is retried,
/// and a read can need several blocks, a block table and a repair. On a link
/// that stalled without dropping, one `read` held a FUSE worker for 34 minutes
/// and the application behind it with it (`docs/BUGS.md` B107). An error the
/// caller can retry beats a process stuck in `D` state.
pub(super) const READ_FETCH_TIMEOUT: Duration = Duration::from_secs(120);
/// How long a read of blocks nothing has cached waits for a lost link to come
/// back before failing. Long enough to ride out a Wi-Fi blip, short enough
/// that a program reading while the laptop is really offline gets its `EIO`
/// before the user gives up on it.
const READ_OFFLINE_WAIT: Duration = Duration::from_secs(30);
/// Cap on tracked sequential streams. Only reads in progress matter, so the map
/// is cleared wholesale rather than aged — losing the ramp costs one read.
const MAX_PREFETCH_STREAMS: usize = 256;
@@ -55,7 +70,7 @@ const MAX_OPEN_READERS: usize = 64;
/// `(mtime, size)` — the same validity pair the content cache uses (a new
/// revision bumps mtime). On a mismatch the reader is dropped and reopened.
pub(super) struct CachedReader {
reader: Arc<RevisionReader>,
reader: Arc<dyn RevisionRead>,
mtime: i64,
size: u64,
/// For LRU eviction.
@@ -86,7 +101,7 @@ pub(super) struct PendingOpen {
/// must not join this open — it would get a reader for the wrong bytes.
mtime: i64,
size: u64,
rx: tokio::sync::watch::Receiver<Option<Result<Arc<RevisionReader>, Errno>>>,
rx: tokio::sync::watch::Receiver<Option<Result<Arc<dyn RevisionRead>, Errno>>>,
}
/// Distinguishes `PendingOpen`s. Only uniqueness matters, so a plain counter
@@ -298,16 +313,16 @@ impl Core {
uid: &NodeUid,
mtime: i64,
fsize: u64,
) -> Result<Arc<RevisionReader>, Errno> {
) -> Result<Arc<dyn RevisionRead>, Errno> {
loop {
// Decide under the lock, act outside it — network I/O must never run
// under a std `Mutex`, and an await must never hold one.
enum Act {
Hit(Arc<RevisionReader>),
Join(tokio::sync::watch::Receiver<Option<Result<Arc<RevisionReader>, Errno>>>),
Hit(Arc<dyn RevisionRead>),
Join(tokio::sync::watch::Receiver<Option<Result<Arc<dyn RevisionRead>, Errno>>>),
Lead(
u64,
tokio::sync::watch::Sender<Option<Result<Arc<RevisionReader>, Errno>>>,
tokio::sync::watch::Sender<Option<Result<Arc<dyn RevisionRead>, Errno>>>,
),
}
let act = {
@@ -361,9 +376,10 @@ impl Core {
}
Act::Lead(id, tx) => {
debug!(%uid, mtime, fsize, "opening revision");
let result = match self.client.open_revision(uid).await {
Ok(reader) => Ok(Arc::new(reader)),
let result = match self.drive.open_revision(uid).await {
Ok(reader) => Ok(reader),
Err(e) => {
self.lost_link(&e, "open a revision");
warn!(%uid, error = %e, "open_revision failed");
Err(Errno::EIO)
}
@@ -552,7 +568,7 @@ impl Core {
/// leader's claim is gone by then, so this cannot recurse.
async fn fetch_block(
&self,
reader: &Arc<RevisionReader>,
reader: &Arc<dyn RevisionRead>,
uid: &NodeUid,
mtime: i64,
fsize: u64,
@@ -618,7 +634,7 @@ impl Core {
/// download, or the read fails. See [`Core::repair_block`] (bugs.md B84).
async fn read_block(
&self,
reader: &Arc<RevisionReader>,
reader: &Arc<dyn RevisionRead>,
uid: &NodeUid,
mtime: i64,
fsize: u64,
@@ -649,6 +665,7 @@ impl Core {
.store_block_geometry(uid, mtime, fsize, reader.block_sizes());
}
let bytes = reader.read_at(span.start, span.len).await.map_err(|e| {
self.lost_link(&e, "read a block");
warn!(%uid, start = span.start, len = span.len, error = %e, "block read failed");
Errno::EIO
})?;
@@ -761,7 +778,7 @@ impl Core {
error!(%uid, %error, "creating a scratch file for a repair download failed");
Errno::EIO
})?;
let outcome = self.client.download_file_to(uid, &mut file).await;
let outcome = self.drive.download_file_to(uid, &mut file).await;
let stored = (|| {
outcome.map_err(|error| {
warn!(%uid, %error, "repair download failed");
@@ -939,28 +956,67 @@ impl Core {
}
if !misses.is_empty() {
let fetched = self.rt.block_on(async {
// Resolve the file's keys and block table once, then read every
// missing block through the shared reader. Previously each block
// called `download_range`, which redid that resolution per block.
let reader = self.revision_reader(uid, mtime, fsize).await?;
// One deadline for the whole read, including any wait for a lost
// link. A fetch that failed because the link went down is tried
// once more when it is back, so a blip is a slow read, not an `EIO`.
let deadline = Instant::now() + READ_FETCH_TIMEOUT;
let mut retried = false;
let fetched = loop {
if !self.is_online() {
let wait =
READ_OFFLINE_WAIT.min(deadline.saturating_duration_since(Instant::now()));
if !self.wait_online(wait) {
debug!(%uid, offset, len, "read of uncached blocks while offline; failing it");
return Err(Errno::EIO);
}
}
let fetch = async {
// Resolve the file's keys and block table once, then read every
// missing block through the shared reader. Previously each block
// called `download_range`, which redid that resolution per block.
let reader = self.revision_reader(uid, mtime, fsize).await?;
let mut set = tokio::task::JoinSet::new();
for &span in &misses {
let (core, reader, uid) = (self.clone(), reader.clone(), uid.clone());
set.spawn(async move {
core.fetch_block(&reader, &uid, mtime, fsize, span, cache_blocks)
.await
.map(|bytes| (span.idx, bytes))
});
let mut set = tokio::task::JoinSet::new();
for &span in &misses {
let (core, reader, uid) = (self.clone(), reader.clone(), uid.clone());
set.spawn(async move {
core.fetch_block(&reader, &uid, mtime, fsize, span, cache_blocks)
.await
.map(|bytes| (span.idx, bytes))
});
}
let mut out = Vec::with_capacity(misses.len());
while let Some(joined) = set.join_next().await {
// A join error means the task panicked; surface it as EIO.
out.push(joined.map_err(|_| Errno::EIO)??);
}
Ok::<_, Errno>(out)
};
// Dropping the fetch on the deadline aborts the block tasks with
// it; blocks that already landed stay cached for the retry. The
// timer is built inside the runtime: a FUSE worker has no reactor
// of its own, and building it out here panicked every remote read.
let budget = deadline.saturating_duration_since(Instant::now());
match self
.rt
.block_on(async { tokio::time::timeout(budget, fetch).await })
{
Ok(Ok(fetched)) => break fetched,
// The fetch saw the link go down; the top of the loop waits
// for it within what is left of the deadline.
Ok(Err(_)) if !retried && !self.is_online() => retried = true,
Ok(Err(e)) => return Err(e),
Err(_) => {
warn!(
%uid, offset, len, secs = READ_FETCH_TIMEOUT.as_secs(),
"read got no answer from the network in time; failing it"
);
self.evict_reader(uid);
self.mark_offline("read", &crate::link::no_answer(READ_FETCH_TIMEOUT));
return Err(Errno::EIO);
}
}
let mut out = Vec::with_capacity(misses.len());
while let Some(joined) = set.join_next().await {
// A join error means the task panicked; surface it as EIO.
out.push(joined.map_err(|_| Errno::EIO)??);
}
Ok::<_, Errno>(out)
})?;
};
for (bidx, bytes) in fetched {
blocks[(bidx - first) as usize] = Some(bytes);
}
+1 -4
View File
@@ -248,10 +248,7 @@ impl Core {
let outcomes = self
.rt
.block_on(
self.client
.trash_nodes(std::slice::from_ref(&candidate.uid)),
)
.block_on(self.drive.trash_nodes(std::slice::from_ref(&candidate.uid)))
.map_err(|e| CoreError::from_api(&e, "trash the mis-dated original"))?;
if let Some((_, Err(e))) = outcomes.into_iter().next() {
return Err(CoreError::from_api(&e, "trash the mis-dated original"));
+611
View File
@@ -0,0 +1,611 @@
//! Moving a file or folder between two locations without moving its content.
//!
//! `~/ProtonDrive` and every on-demand synced folder are separate FUSE
//! sessions, and a mirror folder is a plain local directory. The kernel refuses
//! `rename(2)` between two mounts with `EXDEV` before the daemon hears of it, so
//! `mv` falls back to copy and delete: every file is downloaded, uploaded again,
//! and the original is trashed.
//!
//! On Proton Drive the same move is one metadata change, because My files and
//! this computer's device folders all live on the main volume. So a move
//! request that names two absolute paths is resolved here, against every
//! location this daemon knows, and done as a server-side move. The local side
//! is then brought level: a mount forgets the node and re-lists, and a mirror
//! folder either renames its copy along or drops it once Drive has it.
use std::collections::HashMap;
use std::path::{Component, Path, PathBuf};
use std::sync::Arc;
use std::time::Duration;
use parking_lot::Mutex;
use pdfs_core::db::{StoredSyncEntry, StoredSyncFolder};
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::NodeKind;
use proton_drive_rs::proton_sdk::ids::NodeUid;
use tracing::{info, warn};
use super::sync::mirror_subtree_unsynced;
use super::{Core, is_local_uid, parse_uid};
/// How long a move waits for a mirror folder's sync pass to finish before it
/// reports the folder busy. A pass can run for hours on a first sync, so the
/// move does not wait it out.
const MIRROR_LOCK_WAIT: Duration = Duration::from_secs(5);
/// Where one side of a move lives.
enum Place {
/// My files or an on-demand folder: a live FUSE mount. `core` is rooted at
/// that mount and `rel` is relative to its mountpoint.
Mounted { core: Box<Core>, rel: PathBuf },
/// A mirror folder: a local directory kept in step by the sync engine.
/// `rel` is `/`-separated, relative to the folder, as the baseline keys it.
Mirror {
folder: StoredSyncFolder,
rel: String,
},
}
impl Place {
fn mirror_id(&self) -> Option<i64> {
match self {
Place::Mirror { folder, .. } => Some(folder.id),
Place::Mounted { .. } => None,
}
}
fn is_root(&self) -> bool {
match self {
Place::Mounted { rel, .. } => rel.as_os_str().is_empty(),
Place::Mirror { rel, .. } => rel.is_empty(),
}
}
}
/// The mirror folder that holds `abs`, and `abs` relative to it. The deepest
/// folder wins, although folders are not expected to nest.
fn mirror_covering(
folders: Vec<StoredSyncFolder>,
abs: &Path,
) -> Option<(StoredSyncFolder, String)> {
let folder = folders
.into_iter()
.filter(|f| f.mode == "mirror" && abs.starts_with(&f.local_path))
.max_by_key(|f| Path::new(&f.local_path).components().count())?;
let rel = abs.strip_prefix(&folder.local_path).ok()?;
let parts = rel
.components()
.map(|c| match c {
Component::Normal(part) => part.to_str(),
_ => None,
})
.collect::<Option<Vec<_>>>()?;
Some((folder, parts.join("/")))
}
/// `name` inside the folder at `parent`, in baseline form.
fn join_rel(parent: &str, name: &str) -> String {
if parent.is_empty() {
name.to_string()
} else {
format!("{parent}/{name}")
}
}
/// Rename `from` to `to`, failing with `EEXIST` rather than replacing
/// whatever appeared at `to` since it was checked. A filesystem that cannot
/// promise that gets a plain rename after one more look.
fn rename_noreplace(from: &Path, to: &Path) -> std::io::Result<()> {
use std::ffi::CString;
use std::os::unix::ffi::OsStrExt;
let cstr = |p: &Path| {
CString::new(p.as_os_str().as_bytes())
.map_err(|_| std::io::Error::from_raw_os_error(libc::EINVAL))
};
let (c_from, c_to) = (cstr(from)?, cstr(to)?);
// SAFETY: both are valid NUL-terminated paths that outlive the call.
let rc = unsafe {
libc::syscall(
libc::SYS_renameat2,
libc::AT_FDCWD,
c_from.as_ptr(),
libc::AT_FDCWD,
c_to.as_ptr(),
libc::RENAME_NOREPLACE,
)
};
if rc == 0 {
return Ok(());
}
let err = std::io::Error::last_os_error();
match err.raw_os_error() {
Some(libc::EINVAL | libc::ENOSYS) => {
if std::fs::symlink_metadata(to).is_ok() {
return Err(std::io::Error::from_raw_os_error(libc::EEXIST));
}
std::fs::rename(from, to)
}
_ => Err(err),
}
}
/// Whether `uid` is `root` or lies under it, as far as the nodes table knows.
fn under(core: &Core, root: &NodeUid, uid: &str) -> bool {
let root = root.to_string();
uid == root || matches!(core.db.path_relative_to(&root, uid), Ok(Some(_)))
}
impl Core {
/// Move the file or folder at `src` into the folder at `dest_parent`. Both
/// are absolute local paths and may be in different locations: My files,
/// an on-demand folder, or a mirror folder. Returns the moved name.
///
/// Within one mount this is [`Core::move_to`]. Across locations it
/// is a server-side move followed by local bookkeeping; see the module
/// documentation. Anything the move could lose is refused up front.
pub(crate) fn move_between(&self, src: &Path, dest_parent: &Path) -> CoreResult<String> {
let source = self.locate(src)?;
let dest = self.locate(dest_parent)?;
if let (
Place::Mounted { core, rel },
Place::Mounted {
core: dest_core,
rel: dest_rel,
},
) = (&source, &dest)
&& Arc::ptr_eq(&core.root, &dest_core.root)
{
return core.move_to(rel, dest_rel);
}
if source.is_root() {
return Err(CoreError::invalid(format!(
"{} is the root of a synced location and cannot be moved",
src.display()
)));
}
if dest_parent.starts_with(src) {
return Err(CoreError::invalid("cannot move a folder into itself"));
}
let name = src
.file_name()
.and_then(|n| n.to_str())
.ok_or_else(|| CoreError::invalid(format!("{} has no usable name", src.display())))?
.to_string();
// Hold every mirror folder involved, lowest id first, so no sync pass
// sees the move half done and two moves cannot deadlock.
let mut ids: Vec<i64> = [source.mirror_id(), dest.mirror_id()]
.into_iter()
.flatten()
.collect();
ids.sort_unstable();
ids.dedup();
let locks: Vec<Arc<Mutex<()>>> = ids.iter().map(|&id| self.sync_lock(id)).collect();
let mut guards = Vec::with_capacity(locks.len());
for lock in &locks {
guards.push(lock.try_lock_for(MIRROR_LOCK_WAIT).ok_or_else(|| {
CoreError::conflict("a synced folder is busy syncing; try again when it is done")
})?);
}
let mut baselines: HashMap<i64, HashMap<String, StoredSyncEntry>> = HashMap::new();
for &id in &ids {
let baseline = self
.db
.sync_entries(id)
.map_err(|e| CoreError::internal(format!("db: {e}")))?;
baselines.insert(id, baseline);
}
let src_uid = self.source_uid(&source, &baselines)?;
let dest_uid = self.dest_uid(&dest, &baselines, &name)?;
if src_uid.volume_id != dest_uid.volume_id {
return Err(CoreError::invalid(
"the two locations are on different Proton Drive volumes; copy it instead",
));
}
if under(self, &src_uid, &dest_uid.to_string()) {
return Err(CoreError::invalid("cannot move a folder into itself"));
}
// Mirror to mirror renames the local copy along, so it needs nothing
// proven about it. Another filesystem makes that impossible, and the
// pair then drops the source copy like any other move out of a mirror.
let mut renamed = None;
if let (
Place::Mirror { folder, rel },
Place::Mirror {
folder: dest_folder,
rel: dest_rel,
},
) = (&source, &dest)
{
let from = src.to_path_buf();
let to = dest_parent.join(&name);
match rename_noreplace(&from, &to) {
Ok(()) => renamed = Some((from, to)),
Err(e) if e.raw_os_error() == Some(libc::EXDEV) => {
info!(from = %folder.local_path, to = %dest_folder.local_path,
"move: mirror folders are on different filesystems");
}
Err(e) if e.raw_os_error() == Some(libc::EEXIST) => {
return Err(CoreError::conflict(format!(
"{name} already exists in {}",
dest_parent.display()
)));
}
Err(e) => {
return Err(CoreError::internal(format!(
"could not move {} to {}: {e}",
rel,
join_rel(dest_rel, &name)
)));
}
}
}
if renamed.is_none()
&& let Place::Mirror { folder, rel } = &source
&& let Some(reason) =
mirror_subtree_unsynced(Path::new(&folder.local_path), rel, &baselines[&folder.id])
{
return Err(CoreError::conflict(format!(
"{name} is not fully synced yet ({reason}); wait for the sync to finish"
)));
}
if let Err(e) = self.move_remote(&src_uid, &dest_uid) {
if let Some((from, to)) = &renamed
&& let Err(back) = std::fs::rename(to, from)
{
warn!(from = %to.display(), to = %from.display(), error = %back,
"move: could not put the local copy back after the move failed");
}
return Err(CoreError::from_api(&e, "move"));
}
self.settle_source(
&source,
&src_uid,
renamed.is_some(),
&dest,
&dest_uid,
&name,
src,
);
self.settle_dest(&dest, &dest_uid, &name);
drop(guards);
if renamed.is_none()
&& let Place::Mirror { folder, .. } = &dest
{
// Nothing local exists yet; the folder's next pass downloads it.
self.sync_now(Some(folder.id));
}
Ok(name)
}
/// Which location `abs` is in. A mount is looked for first: a mirror folder
/// cannot live inside a FUSE mount, and a mount is what the kernel sees.
fn locate(&self, abs: &Path) -> CoreResult<Place> {
if let Some((core, rel)) = self.rooted_at(abs) {
if !rel
.components()
.all(|c| matches!(c, Component::Normal(_) | Component::CurDir))
{
return Err(CoreError::invalid(format!(
"{} escapes its location",
abs.display()
)));
}
return Ok(Place::Mounted {
core: Box::new(core),
rel,
});
}
let folders = self
.db
.sync_folder_list()
.map_err(|e| CoreError::internal(format!("db: {e}")))?;
mirror_covering(folders, abs)
.map(|(folder, rel)| Place::Mirror { folder, rel })
.ok_or_else(|| {
CoreError::invalid(format!(
"{} is not in a Proton Drive location",
abs.display()
))
})
}
/// The node being moved, once everything that must hold before it leaves
/// its location has been checked.
fn source_uid(
&self,
source: &Place,
baselines: &HashMap<i64, HashMap<String, StoredSyncEntry>>,
) -> CoreResult<NodeUid> {
match source {
Place::Mounted { core, rel } => {
let (ino, uid) = core.resolve(rel)?;
if is_local_uid(&uid) {
return Err(CoreError::conflict(
"it is not uploaded yet; wait for the upload to finish",
));
}
let old_parent = core.source_parent_uid(ino, rel)?;
for authority in [&uid, &old_parent] {
core.require_uid_writable(authority)
.map_err(|e| core.errno_error(e, "move access"))?;
}
if self.has_pending_work_under(&uid)? {
return Err(CoreError::conflict(
"it has changes that are not uploaded yet; wait for the upload to finish",
));
}
Ok(uid)
}
Place::Mirror { folder, rel } => baselines[&folder.id]
.get(rel)
.and_then(|entry| entry.remote_uid.as_deref())
.and_then(parse_uid)
.ok_or_else(|| {
CoreError::conflict(
"it is not on Proton Drive yet; wait for the sync to finish",
)
}),
}
}
/// The folder the node moves into. Refuses one that already holds `name`
/// on this disk; Drive refuses a clash on its side by itself.
fn dest_uid(
&self,
dest: &Place,
baselines: &HashMap<i64, HashMap<String, StoredSyncEntry>>,
name: &str,
) -> CoreResult<NodeUid> {
let uid = match dest {
Place::Mounted { core, rel } => {
let (ino, uid) = core.resolve(rel)?;
let is_folder = core
.state()
.entries
.get(&ino)
.is_some_and(|entry| matches!(entry.node.kind, NodeKind::Folder));
if !is_folder {
return Err(CoreError::invalid("the destination is not a folder"));
}
core.require_uid_writable(&uid)
.map_err(|e| core.errno_error(e, "move access"))?;
uid
}
Place::Mirror { folder, rel } => {
let local = Path::new(&folder.local_path).join(rel);
if !local.is_dir() {
return Err(CoreError::invalid("the destination is not a folder"));
}
if std::fs::symlink_metadata(local.join(name)).is_ok() {
return Err(CoreError::conflict(format!(
"{name} already exists in the destination"
)));
}
let uid = if rel.is_empty() {
Some(folder.remote_uid.as_str())
} else {
baselines[&folder.id]
.get(rel)
.and_then(|entry| entry.remote_uid.as_deref())
};
uid.and_then(parse_uid).ok_or_else(|| {
CoreError::conflict(
"the destination is not on Proton Drive yet; wait for the sync to finish",
)
})?
}
};
if is_local_uid(&uid) {
return Err(CoreError::conflict(
"the destination is not uploaded yet; wait for the upload to finish",
));
}
Ok(uid)
}
/// Whether the queue owes Drive anything for `uid` or a node under it, or a
/// file under it is open for writing. Moving such a node away would strand
/// the pending change against a path that no longer leads to it.
fn has_pending_work_under(&self, uid: &NodeUid) -> CoreResult<bool> {
let ops = self
.db
.pending_ops()
.map_err(|e| CoreError::internal(format!("db: {e}")))?;
if ops.iter().any(|op| {
under(self, uid, &op.uid)
|| op
.parent_uid
.as_deref()
.is_some_and(|p| under(self, uid, p))
}) {
return Ok(true);
}
// Collected first so no State lock is held across the nodes-table reads.
let mut writing = Vec::new();
self.for_each_state(|st| {
writing.extend(st.active_writes.values().map(|w| w.uid.to_string()));
});
Ok(writing.iter().any(|w| under(self, uid, w)))
}
/// Bring the source location level with a move Drive has made.
#[allow(clippy::too_many_arguments)]
fn settle_source(
&self,
source: &Place,
uid: &NodeUid,
renamed: bool,
dest: &Place,
dest_uid: &NodeUid,
name: &str,
src: &Path,
) {
match source {
Place::Mounted { core, rel } => {
// Moved everywhere, so no mount keeps serving it under the old
// path and its row stays (`docs/BUGS.md` B117). The content
// cache is keyed by uid and stays valid.
self.relocate_everywhere(uid, dest_uid, name);
core.invalidate_parent_listing(rel);
}
Place::Mirror { folder, rel } if renamed => {
let Place::Mirror {
folder: dest_folder,
rel: dest_rel,
} = dest
else {
unreachable!("a local rename needs a mirror destination");
};
if let Err(e) = self.db.sync_entries_move(
folder.id,
rel,
dest_folder.id,
&join_rel(dest_rel, name),
) {
// The next passes see a copy with no baseline on one side
// and a baseline with no copy on the other. That uploads
// the files again and trashes the moved node, but loses
// nothing.
warn!(error = ?e, "move: could not carry the sync baseline along");
}
}
Place::Mirror { folder, rel } => {
// Drive had all of it when the move was checked, so the local
// copy is only a stale duplicate now. Something written into it
// since would go with it, so look again first; if anything is
// new, the copy stays and the next pass uploads it as new.
let changed = match self.db.sync_entries(folder.id) {
Ok(baseline) => {
mirror_subtree_unsynced(Path::new(&folder.local_path), rel, &baseline)
}
Err(e) => Some(format!("db: {e}")),
};
if let Err(e) = self.db.sync_entries_remove_subtree(folder.id, rel) {
warn!(error = ?e, "move: could not drop the sync baseline");
}
if let Some(reason) = changed {
warn!(path = %src.display(), reason,
"move: the local copy changed during the move; keeping it");
return;
}
let removed = if src.is_dir() {
std::fs::remove_dir_all(src)
} else {
std::fs::remove_file(src)
};
if let Err(e) = removed {
warn!(path = %src.display(), error = %e,
"move: could not remove the local copy after the move");
}
}
}
}
/// Bring the destination location level with a move Drive has made.
fn settle_dest(&self, dest: &Place, uid: &NodeUid, name: &str) {
if let Place::Mounted { .. } = dest {
self.invalidate_children_of(uid);
// A lookup that missed before the move may be cached as absent.
self.for_each_mount(|st, notify| {
if let Some(&ino) = st.by_uid.get(uid) {
notify.inval_entry(ino, name.to_string());
}
});
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn folder(id: i64, local_path: &str, mode: &str) -> StoredSyncFolder {
StoredSyncFolder {
id,
local_path: local_path.to_string(),
remote_uid: format!("vol~root{id}"),
remote_share_id: String::new(),
mode: mode.to_string(),
pending_mode: None,
state: "idle".to_string(),
last_sync: 0,
}
}
#[test]
fn mirror_covering_finds_the_mirror_folder_and_the_path_inside_it() {
let folders = vec![folder(1, "/home/u/Docs", "mirror")];
let (found, rel) = mirror_covering(folders, Path::new("/home/u/Docs/a/b.txt")).unwrap();
assert_eq!(found.id, 1);
assert_eq!(rel, "a/b.txt");
}
#[test]
fn mirror_covering_gives_an_empty_path_for_the_folder_itself() {
let folders = vec![folder(1, "/home/u/Docs", "mirror")];
let (_, rel) = mirror_covering(folders, Path::new("/home/u/Docs")).unwrap();
assert_eq!(rel, "");
}
#[test]
fn mirror_covering_prefers_the_deepest_folder() {
let folders = vec![
folder(1, "/home/u/Docs", "mirror"),
folder(2, "/home/u/Docs/Work", "mirror"),
];
let (found, rel) = mirror_covering(folders, Path::new("/home/u/Docs/Work/x")).unwrap();
assert_eq!(found.id, 2);
assert_eq!(rel, "x");
}
#[test]
fn mirror_covering_skips_on_demand_folders_and_name_prefixed_siblings() {
let folders = || {
vec![
folder(1, "/home/u/Docs", "mirror"),
folder(2, "/home/u/Media", "ondemand"),
]
};
assert!(mirror_covering(folders(), Path::new("/home/u/Media/x")).is_none());
assert!(mirror_covering(folders(), Path::new("/home/u/Docs2/x")).is_none());
}
#[test]
fn mirror_covering_refuses_a_path_that_climbs_out() {
let folders = vec![folder(1, "/home/u/Docs", "mirror")];
assert!(mirror_covering(folders, Path::new("/home/u/Docs/../etc")).is_none());
}
#[test]
fn join_rel_names_a_child_of_the_folder_root_without_a_slash() {
assert_eq!(join_rel("", "a"), "a");
assert_eq!(join_rel("x/y", "a"), "x/y/a");
}
#[test]
fn a_local_rename_never_replaces_what_is_already_there() {
let dir = std::env::temp_dir().join(format!("pdfs-relocate-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let (a, b, c) = (dir.join("a"), dir.join("b"), dir.join("c"));
std::fs::write(&a, b"moved").unwrap();
std::fs::write(&b, b"kept").unwrap();
let err = rename_noreplace(&a, &b).unwrap_err();
assert_eq!(err.raw_os_error(), Some(libc::EEXIST));
assert_eq!(std::fs::read(&b).unwrap(), b"kept");
assert!(a.exists());
rename_noreplace(&a, &c).unwrap();
assert!(!a.exists());
assert_eq!(std::fs::read(&c).unwrap(), b"moved");
std::fs::remove_dir_all(&dir).unwrap();
}
}
+1 -1
View File
@@ -232,7 +232,7 @@ impl Core {
let mut out = std::io::BufWriter::new(file);
let written = match self
.rt
.block_on(self.client.download_revision_to(uid, revision_id, &mut out))
.block_on(self.drive.download_revision_to(uid, revision_id, &mut out))
.map_err(|e| CoreError::from_api(&e, "download revision"))
.and_then(|()| {
use std::io::Write as _;
+34 -28
View File
@@ -21,7 +21,7 @@ use proton_drive_rs::{MemberRole, NodeKind};
use proton_drive_rs::Node;
use super::{Core, ROOT_INO, node_size, parse_uid, public_link_info, role_from_str, role_to_str};
use super::{Core, node_size, parse_uid, public_link_info, role_from_str, role_to_str};
/// A node from a *shared* listing as a [`DirEntry`].
///
@@ -463,7 +463,7 @@ impl Core {
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&uids))
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes
.into_iter()
@@ -481,14 +481,14 @@ impl Core {
parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?;
let child_uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(&uid))
.block_on(self.drive.enumerate_folder_children_node_uids(&uid))
.map_err(|e| CoreError::from_api(&e, "enumerate shared children"))?;
if child_uids.is_empty() {
return Ok(Vec::new());
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&child_uids))
.block_on(self.drive.enumerate_nodes(&child_uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes
.into_iter()
@@ -509,7 +509,7 @@ impl Core {
parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?;
let node = self
.rt
.block_on(self.client.get_node(&uid))
.block_on(self.drive.get_node(&uid))
.map_err(|e| CoreError::from_api(&e, "get shared node"))?
.ok_or_else(|| CoreError::not_found("shared node not found"))?;
if !node.is_file() {
@@ -610,10 +610,10 @@ impl Core {
/// Proton products (not Drive-only). A remote round-trip; nothing here is
/// cached.
pub(crate) fn account_quota(&self) -> CoreResult<(i64, i64)> {
let q = self
.rt
.block_on(self.client.quota())
.map_err(|e| CoreError::from_api(&e, "account quota"))?;
let q = self.block_on_bounded(self.drive.quota()).map_err(|e| {
self.lost_link(&e, "account quota");
CoreError::from_api(&e, "account quota")
})?;
*self.quota.lock() = Some((std::time::Instant::now(), q.max_space, q.used_space));
Ok((q.max_space, q.used_space))
}
@@ -632,11 +632,22 @@ impl Core {
{
return Some((max_space, used_space));
}
match self.account_quota() {
let last = || self.quota.lock().map(|(_, max, used)| (max, used));
if !self.is_online()
|| self
.quota_refreshing
.swap(true, std::sync::atomic::Ordering::AcqRel)
{
return last();
}
let refreshed = self.account_quota();
self.quota_refreshing
.store(false, std::sync::atomic::Ordering::Release);
match refreshed {
Ok(q) => Some(q),
Err(error) => {
debug!(%error, "account quota refresh failed; serving the last known figures");
self.quota.lock().map(|(_, max, used)| (max, used))
last()
}
}
}
@@ -658,7 +669,7 @@ impl Core {
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&uids))
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
let mut items = Vec::with_capacity(nodes.len());
for n in nodes {
@@ -703,19 +714,10 @@ impl Core {
/// been seen through the mount (e.g. shared but not browsed to this session) —
/// the caller then leaves the path empty.
pub(crate) fn rel_path_for_uid(&self, uid: &NodeUid) -> Option<String> {
// Searched across every mount: a node browsed through a sync folder is
// interned in that fork's inode space, so the primary state alone had no
// entry for it and the caller silently showed an empty path
// (`docs/BUGS.md` B74). The path is relative to whichever mount holds
// it, which is the one the user reached it through.
let mut path = None;
self.for_each_state(|st| {
if path.is_some() {
return;
}
path = walk_to_root(st, uid);
});
path
// Relative to whichever root the node sits under: a node browsed
// through a sync folder is under that fork's root, not My Files
// (`docs/BUGS.md` B74).
walk_to_root(&self.state(), uid)
}
}
@@ -760,14 +762,18 @@ impl StopSharingOutcome {
}
}
/// Walk one inode space from `uid` up to its root, building the path. `None` if
/// that mount has never interned the node, or if the chain breaks partway —
/// Walk the tree from `uid` up to the root it sits under, building the path.
/// `None` if the node has never been interned, or if the chain breaks partway —
/// a broken chain is a missing path, never a partial one.
fn walk_to_root(st: &super::State, uid: &NodeUid) -> Option<String> {
let mut ino = *st.by_uid.get(uid)?;
let mut parts = Vec::new();
while ino != ROOT_INO {
while !st.roots.contains(&ino) {
let entry = st.entries.get(&ino)?;
// An orphan is its own parent without being a root.
if entry.parent == ino {
return None;
}
parts.push(entry.node.name.clone());
ino = entry.parent;
}
+423
View File
@@ -0,0 +1,423 @@
//! One simulated daemon: a real FUSE mount on its own state directory, whose
//! Drive is a [`FakeClient`].
//!
//! Everything the daemon does runs as in production (drain workers, sync
//! engine, event feed, online probe, control socket) except what the host
//! changes (`mount::Host::Simulation`): a channel stops it instead of a signal,
//! and it does not index the developer's home.
use std::io;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::mpsc::SyncSender;
use std::thread::JoinHandle;
use std::time::{Duration, Instant};
use pdfs_core::cache::ContentCache;
use pdfs_core::config::AppConfig;
use pdfs_core::control::{self, PendingOpInfo, Request, Response};
use pdfs_core::db::Db;
use proton_drive_rs::ProtonDriveClient;
use proton_drive_rs::proton_sdk::config::ProtonClientConfiguration;
use proton_drive_rs::proton_sdk::session::{PasswordMode, ProtonApiSession, ResumeParameters};
use super::fake_drive::FakeClient;
use crate::mount::{Host, MountOptions, MountOutcome, mount_with};
/// How long a mount may take to come up.
const MOUNT_DEADLINE: Duration = Duration::from_secs(20);
/// How long a restart may wait for the last run to let go of its database.
const LAST_RUN_DEADLINE: Duration = Duration::from_secs(30);
pub(crate) struct Daemon {
pub(crate) client: FakeClient,
pub(crate) mountpoint: PathBuf,
dir: PathBuf,
socket: PathBuf,
stop: SyncSender<()>,
mount: Option<JoinHandle<io::Result<MountOutcome>>>,
rt: Option<tokio::runtime::Runtime>,
}
impl Daemon {
/// Mount a daemon on the state under `dir`, creating it on the first start.
/// A later start on the same `dir` is a restart: it finds the queue, the
/// cache and the database the last run left.
pub(crate) fn start(dir: &Path, client: FakeClient) -> io::Result<Self> {
log_to_test_output();
let mountpoint = dir.join("mnt");
let socket = dir.join("control.sock");
std::fs::create_dir_all(&mountpoint)?;
// A thread of the last run that is still in a request holds the
// database, and with it the single-writer lock, until it returns. A
// real restart waits for the old process to exit in the same way.
let mut db = Db::open(&dir.join("pdfs.db"));
let deadline = Instant::now() + LAST_RUN_DEADLINE;
while db
.as_ref()
.is_err_and(|error| error.to_string().contains("already using"))
&& Instant::now() < deadline
{
std::thread::sleep(Duration::from_millis(100));
db = Db::open(&dir.join("pdfs.db"));
}
let db = Arc::new(db.map_err(io::Error::other)?);
let cache = ContentCache::open(dir.join("cache"), dir.join("pins.json"), 0, db.clone())
.map_err(io::Error::other)?;
let rt = tokio::runtime::Builder::new_multi_thread()
.worker_threads(4)
.enable_all()
.build()?;
let (stop, stopped) = std::sync::mpsc::sync_channel(1);
let mount = {
let drive = Arc::new(client.clone());
let handle = rt.handle().clone();
let mountpoint = mountpoint.clone();
let socket = socket.clone();
std::thread::Builder::new()
.name("sim-mount".into())
.spawn(move || {
mount_with(
offline_client(),
drive,
handle,
&mountpoint,
cache,
&socket,
db,
MountOptions {
username: "sim".into(),
sweep_mode: AppConfig::default().resolved_conflict_sweep(),
upload_limit: 0,
download_limit: 0,
},
Host::Simulation(stopped),
)
})?
};
let deadline = Instant::now() + MOUNT_DEADLINE;
while !is_mounted(&mountpoint) {
if mount.is_finished() {
return Err(match mount.join() {
Ok(Err(error)) => error,
_ => io::Error::other("mount ended before it came up"),
});
}
if Instant::now() > deadline {
return Err(io::Error::other("mount did not come up"));
}
std::thread::sleep(Duration::from_millis(20));
}
Ok(Self {
client,
mountpoint,
dir: dir.to_path_buf(),
socket,
stop,
mount: Some(mount),
rt: Some(rt),
})
}
/// What the daemon still owes Drive, as `pdfs sync queue` shows it.
pub(crate) fn pending(&self) -> io::Result<Vec<PendingOpInfo>> {
match control::send(&self.socket, &Request::ListPendingOps).map_err(io::Error::other)? {
Response::PendingOps { items } => Ok(items),
other => Err(io::Error::other(format!("unexpected answer {other:?}"))),
}
}
/// Stop the daemon and start it again on the same state and link.
pub(crate) fn restart(mut self) -> io::Result<Self> {
self.shut_down()?;
Self::start(&self.dir.clone(), self.client.clone())
}
/// Stop the daemon the way `systemctl --user stop` does.
pub(crate) fn stop(mut self) -> io::Result<MountOutcome> {
self.shut_down()
}
fn shut_down(&mut self) -> io::Result<MountOutcome> {
let Some(mount) = self.mount.take() else {
return Ok(MountOutcome::Shutdown);
};
let _ = self.stop.try_send(());
let outcome = mount
.join()
.unwrap_or_else(|_| Err(io::Error::other("mount thread panicked")));
if let Some(rt) = self.rt.take() {
rt.shutdown_timeout(Duration::from_secs(5));
}
outcome
}
}
impl Drop for Daemon {
fn drop(&mut self) {
let _ = self.shut_down();
}
}
/// Whether a file system is mounted at `path`: it sits on another device than
/// its parent.
fn is_mounted(path: &Path) -> bool {
use std::os::unix::fs::MetadataExt;
let device = |path: &Path| std::fs::metadata(path).map(|meta| meta.dev()).ok();
let parent = path.parent().and_then(device);
device(path).is_some_and(|dev| Some(dev) != parent)
}
/// The daemons' logs, filtered by `RUST_LOG` and shown with `--nocapture`.
/// What they log at `INFO` and above is also kept for [`take_logged`],
/// whatever the filter.
fn log_to_test_output() {
use tracing_subscriber::prelude::*;
let _ = tracing_subscriber::registry()
.with(
tracing_subscriber::fmt::layer()
.with_test_writer()
.with_filter(tracing_subscriber::EnvFilter::from_default_env()),
)
.with(KeepLogged.with_filter(tracing_subscriber::filter::LevelFilter::INFO))
.try_init();
}
/// What the daemons logged at `INFO` and above since the last call, each as
/// its message followed by its fields: `message key=value ...`.
pub(crate) fn take_logged() -> Vec<String> {
std::mem::take(&mut *LOGGED.lock())
}
static LOGGED: parking_lot::Mutex<Vec<String>> = parking_lot::Mutex::new(Vec::new());
struct KeepLogged;
impl<S: tracing::Subscriber> tracing_subscriber::Layer<S> for KeepLogged {
fn on_event(&self, event: &tracing::Event<'_>, _: tracing_subscriber::layer::Context<'_, S>) {
let mut message = Message(String::new());
event.record(&mut message);
LOGGED.lock().push(message.0);
}
}
struct Message(String);
impl tracing::field::Visit for Message {
fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
if field.name() == "message" {
self.0.insert_str(0, &format!("{value:?}"));
} else {
self.0.push_str(&format!(" {}={value:?}", field.name()));
}
}
}
/// A client for what `DriveApi` does not cover (sharing, Photos, devices),
/// pointed at a closed local port so none of it leaves the machine.
fn offline_client() -> ProtonDriveClient {
let config = ProtonClientConfiguration::new("external-drive-pdfs-sim@0.0.0-dev")
.with_base_url("http://127.0.0.1:9");
let session = ProtonApiSession::resume(
config,
ResumeParameters {
session_id: "sim".to_owned().into(),
username: "sim".into(),
user_id: "sim".to_owned().into(),
access_token: String::new(),
refresh_token: String::new(),
scopes: Vec::new(),
is_waiting_for_second_factor_code: false,
password_mode: PasswordMode::Single,
},
)
.expect("an offline session needs no network");
ProtonDriveClient::new(&session, Vec::new())
}
/// A fresh directory for one simulation, under the system temp dir.
pub(crate) fn scratch(label: &str) -> PathBuf {
use std::sync::atomic::{AtomicU64, Ordering};
static N: AtomicU64 = AtomicU64::new(0);
let dir = std::env::temp_dir().join(format!(
"pdfs-sim-{label}-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed)
));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).expect("create a simulation directory");
dir
}
/// Wait up to `limit` for `done`, polling.
pub(crate) fn wait_until(limit: Duration, mut done: impl FnMut() -> bool) -> bool {
let deadline = Instant::now() + limit;
loop {
if done() {
return true;
}
if Instant::now() > deadline {
return false;
}
std::thread::sleep(Duration::from_millis(50));
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::sim::fake_drive::{Entry, FakeDrive, Faults};
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_daemon_on_the_fake_drive_syncs_both_ways() {
let drive = FakeDrive::new();
let dir = scratch("smoke");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
std::fs::write(daemon.mountpoint.join("hello.txt"), b"hi").unwrap();
assert!(
wait_until(Duration::from_secs(30), || {
drive.tree().get("hello.txt") == Some(&Entry::File(Arc::new(b"hi".to_vec())))
}),
"a local write landed on Drive: {:?}",
drive.tree()
);
drive.device().write("phone.txt", b"from the phone");
let remote = daemon.mountpoint.join("phone.txt");
assert!(
wait_until(Duration::from_secs(30), || {
std::fs::read(&remote).ok().as_deref() == Some(b"from the phone".as_slice())
}),
"a remote write reached the mount"
);
daemon.client.set_online(false);
std::fs::write(daemon.mountpoint.join("offline.txt"), b"queued").unwrap();
std::thread::sleep(Duration::from_secs(2));
assert!(!drive.tree().contains_key("offline.txt"));
daemon.client.set_online(true);
assert!(
wait_until(Duration::from_secs(60), || drive
.tree()
.contains_key("offline.txt")),
"a write made offline landed once the link came back"
);
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
/// Write `bytes` at `offset` into an existing file, without truncating it.
fn write_at(path: &Path, offset: u64, bytes: &[u8]) -> io::Result<()> {
use std::os::unix::fs::FileExt;
std::fs::OpenOptions::new()
.write(true)
.open(path)?
.write_all_at(bytes, offset)
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_file_closed_empty_takes_a_second_write_at_once() {
// The empty revision never counted as complete, so the next write
// open before it drained was refused with EIO (B123).
let drive = FakeDrive::new();
let dir = scratch("empty-rewrite");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
let path = daemon.mountpoint.join("empty.txt");
std::fs::write(&path, b"old").unwrap();
assert!(wait_until(Duration::from_secs(30), || {
drive.tree().contains_key("empty.txt")
}));
std::fs::write(&path, b"").unwrap();
write_at(&path, 0, b"filled").unwrap();
assert!(
wait_until(Duration::from_secs(30), || {
drive.tree().get("empty.txt") == Some(&Entry::File(Arc::new(b"filled".to_vec())))
}),
"the second write landed: {:?}",
drive.tree()
);
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn the_echo_of_our_own_rename_keeps_the_cached_bytes() {
// The echo read as someone else's change and evicted the bytes, so a
// partial write could not gap-fill, and the next write open on the
// file was refused with EIO (B123).
let drive = FakeDrive::new();
let dir = scratch("rename-echo");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
let before = daemon.mountpoint.join("a.txt");
std::fs::write(&before, b"0123456789").unwrap();
assert!(wait_until(Duration::from_secs(30), || {
drive.tree().contains_key("a.txt")
}));
assert_eq!(std::fs::read(&before).unwrap(), b"0123456789");
let after = daemon.mountpoint.join("b.txt");
std::fs::rename(&before, &after).unwrap();
// Long enough for the next event poll to bring the echo.
std::thread::sleep(crate::POLL_INTERVAL + Duration::from_secs(3));
write_at(&after, 3, b"x").unwrap();
write_at(&after, 5, b"y").unwrap();
assert!(
wait_until(Duration::from_secs(30), || {
drive.tree().get("b.txt") == Some(&Entry::File(Arc::new(b"012x4y6789".to_vec())))
}),
"both writes landed: {:?}",
drive.tree()
);
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_rename_across_folders_ignores_the_new_name_in_the_old_folder() {
// The rename went to Drive as a rename in the old folder and then a
// move, so a sibling there with the new name failed it with EIO
// (B122).
let drive = FakeDrive::new();
let dir = scratch("rename-across");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
let mount = &daemon.mountpoint;
std::fs::create_dir(mount.join("x")).unwrap();
std::fs::create_dir(mount.join("z")).unwrap();
std::fs::write(mount.join("x/f.txt"), b"moved").unwrap();
std::fs::write(mount.join("x/d"), b"sibling").unwrap();
std::fs::write(mount.join("z/d"), b"replaced").unwrap();
let file = |bytes: &[u8]| Some(Entry::File(Arc::new(bytes.to_vec())));
assert!(wait_until(Duration::from_secs(30), || {
let tree = drive.tree();
[
("x/f.txt", b"moved".as_slice()),
("x/d", b"sibling"),
("z/d", b"replaced"),
]
.iter()
.all(|(path, bytes)| tree.get(*path).cloned() == file(bytes))
}));
std::fs::rename(mount.join("x/f.txt"), mount.join("z/d")).unwrap();
assert_eq!(std::fs::read(mount.join("z/d")).unwrap(), b"moved");
let tree = drive.tree();
assert_eq!(tree.get("z/d").cloned(), file(b"moved"));
assert_eq!(tree.get("x/d").cloned(), file(b"sibling"));
assert!(!tree.contains_key("x/f.txt"));
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
}
File diff suppressed because it is too large. Load diff
+9
View File
@@ -0,0 +1,9 @@
//! The simulation tests of `docs/MILESTONE-3.0.0.md` §8: a fake Drive with
//! the faults the real one has, and the tools to drive daemons against it.
pub(crate) mod daemon;
pub(crate) mod fake_drive;
pub(crate) mod model;
pub(crate) mod rng;
pub(crate) mod run;
pub(crate) mod watchdog;
+445
View File
@@ -0,0 +1,445 @@
//! The reference file system of the simulation (`docs/MILESTONE-3.0.0.md`
//! §8.2, property 4): what each syscall a simulated client makes should
//! answer, and the tree it should leave, on a POSIX file system.
//!
//! A [`Model`] covers one client's own folder; paths are relative to it. The
//! same type is used for the trees read back from a mount and from Drive.
use std::collections::BTreeMap;
use super::rng::Rng;
/// A tree by path: `None` is a folder, `Some` a file's content.
pub(crate) type Tree = BTreeMap<String, Option<Vec<u8>>>;
/// One file system call, or the few a program makes in a row.
#[derive(Clone, Debug)]
pub(crate) enum FsOp {
/// `open(O_CREAT | O_TRUNC)`, write `data`, close.
Create {
path: String,
data: Vec<u8>,
},
/// Open an existing file, write `data` at `offset` (past the end leaves
/// a hole), close.
Write {
path: String,
offset: u64,
data: Vec<u8>,
},
/// Open an existing file and `ftruncate` it to `len`.
Truncate {
path: String,
len: u64,
},
Mkdir {
path: String,
},
Rename {
from: String,
to: String,
},
Unlink {
path: String,
},
/// Open a file, unlink it, append `data`, read it all back, close.
UnlinkOpen {
path: String,
data: Vec<u8>,
},
Rmdir {
path: String,
},
}
impl FsOp {
/// The paths the op names, for the log.
pub(crate) fn describe(&self) -> String {
match self {
FsOp::Create { path, data } => format!("create {path} ({} bytes)", data.len()),
FsOp::Write { path, offset, data } => {
format!("write {path} at {offset} ({} bytes)", data.len())
}
FsOp::Truncate { path, len } => format!("truncate {path} to {len}"),
FsOp::Mkdir { path } => format!("mkdir {path}"),
FsOp::Rename { from, to } => format!("rename {from} -> {to}"),
FsOp::Unlink { path } => format!("unlink {path}"),
FsOp::UnlinkOpen { path, data } => {
format!("unlink {path} while open, then append {} bytes", data.len())
}
FsOp::Rmdir { path } => format!("rmdir {path}"),
}
}
}
/// The tree one client should have, and what its syscalls should answer.
#[derive(Clone, Debug, Default)]
pub(crate) struct Model {
tree: Tree,
}
/// Names are drawn from small pools, so ops collide: a create over an existing
/// file, a rename over another, a mkdir of a name that is taken.
const FILE_NAMES: [&str; 6] = ["a.txt", "b.txt", "c.bin", "d", "e.md", "f.txt"];
const DIR_NAMES: [&str; 3] = ["x", "y", "z"];
impl Model {
pub(crate) fn tree(&self) -> &Tree {
&self.tree
}
pub(crate) fn content(&self, path: &str) -> Option<&Vec<u8>> {
self.tree.get(path)?.as_ref()
}
/// What `op` should answer on a POSIX file system, applying it on success.
pub(crate) fn apply(&mut self, op: &FsOp) -> Result<(), i32> {
match op {
FsOp::Create { path, data } => {
self.check_parent(path)?;
match self.tree.get(path) {
Some(None) => Err(libc::EISDIR),
_ => {
self.tree.insert(path.clone(), Some(data.clone()));
Ok(())
}
}
}
FsOp::Write { path, offset, data } => {
let content = self.file_mut(path)?;
// Writing nothing changes nothing, even past the end.
if data.is_empty() {
return Ok(());
}
let end = *offset as usize + data.len();
if content.len() < end {
content.resize(end, 0);
}
content[*offset as usize..end].copy_from_slice(data);
Ok(())
}
FsOp::Truncate { path, len } => {
self.file_mut(path)?.resize(*len as usize, 0);
Ok(())
}
FsOp::Mkdir { path } => {
self.check_parent(path)?;
if self.tree.contains_key(path) {
return Err(libc::EEXIST);
}
self.tree.insert(path.clone(), None);
Ok(())
}
FsOp::Rename { from, to } => self.rename(from, to),
FsOp::Unlink { path } | FsOp::UnlinkOpen { path, .. } => {
self.check_parent(path)?;
match self.tree.get(path) {
None => Err(libc::ENOENT),
Some(None) => Err(libc::EISDIR),
Some(Some(_)) => {
self.tree.remove(path);
Ok(())
}
}
}
FsOp::Rmdir { path } => {
self.check_parent(path)?;
match self.tree.get(path) {
None => Err(libc::ENOENT),
Some(Some(_)) => Err(libc::ENOTDIR),
Some(None) if self.has_children(path) => Err(libc::ENOTEMPTY),
Some(None) => {
self.tree.remove(path);
Ok(())
}
}
}
}
}
fn rename(&mut self, from: &str, to: &str) -> Result<(), i32> {
self.check_parent(from)?;
let Some(moving) = self.tree.get(from).cloned() else {
return Err(libc::ENOENT);
};
self.check_parent(to)?;
if from == to {
return Ok(());
}
if is_below(to, from) {
return Err(libc::EINVAL);
}
match (&moving, self.tree.get(to)) {
(None, Some(Some(_))) => return Err(libc::ENOTDIR),
(Some(_), Some(None)) => return Err(libc::EISDIR),
(None, Some(None)) if self.has_children(to) => return Err(libc::ENOTEMPTY),
_ => {}
}
let below: Vec<(String, Option<Vec<u8>>)> = self
.tree
.iter()
.filter(|(path, _)| is_below(path, from))
.map(|(path, entry)| (path.clone(), entry.clone()))
.collect();
self.tree.remove(from);
self.tree.insert(to.to_owned(), moving);
for (path, entry) in below {
self.tree.remove(&path);
self.tree
.insert(format!("{to}{}", &path[from.len()..]), entry);
}
Ok(())
}
fn file_mut(&mut self, path: &str) -> Result<&mut Vec<u8>, i32> {
self.check_parent(path)?;
match self.tree.get_mut(path) {
None => Err(libc::ENOENT),
Some(None) => Err(libc::EISDIR),
Some(Some(content)) => Ok(content),
}
}
fn check_parent(&self, path: &str) -> Result<(), i32> {
let mut at = String::new();
let mut names = path.split('/').peekable();
while let Some(name) = names.next() {
if names.peek().is_none() {
break;
}
if !at.is_empty() {
at.push('/');
}
at.push_str(name);
match self.tree.get(&at) {
None => return Err(libc::ENOENT),
Some(Some(_)) => return Err(libc::ENOTDIR),
Some(None) => {}
}
}
Ok(())
}
fn has_children(&self, path: &str) -> bool {
self.tree.keys().any(|other| is_below(other, path))
}
pub(crate) fn files(&self) -> Vec<&String> {
self.tree
.iter()
.filter(|(_, entry)| entry.is_some())
.map(|(path, _)| path)
.collect()
}
/// Folders, the client's own root (`""`) first.
fn dirs(&self) -> Vec<String> {
std::iter::once(String::new())
.chain(
self.tree
.iter()
.filter(|(_, entry)| entry.is_none())
.map(|(path, _)| path.clone()),
)
.collect()
}
/// A random op, mostly one that should succeed.
pub(crate) fn next_op(&self, rng: &mut Rng) -> FsOp {
let dirs = self.dirs();
let files = self.files();
let in_dir = |rng: &mut Rng, names: &[&str]| {
let parent = rng.pick(&dirs).expect("the root is always there");
join(parent, rng.pick(names).expect("names"))
};
let a_file = |rng: &mut Rng| rng.pick(&files).map(|path| (*path).clone());
let roll = rng.below(100);
let op = match roll {
0..30 => Some(FsOp::Create {
path: in_dir(rng, &FILE_NAMES),
data: data(rng),
}),
30..45 => a_file(rng).map(|path| {
let len = self.content(&path).map_or(0, Vec::len) as u64;
FsOp::Write {
offset: rng.below(len + 8),
path,
data: data(rng),
}
}),
45..50 => a_file(rng).map(|path| FsOp::Truncate {
len: rng.below(32),
path,
}),
50..60 => Some(FsOp::Mkdir {
path: in_dir(rng, &DIR_NAMES),
}),
// A rename in place, over another file as often as not.
60..72 => a_file(rng).map(|from| FsOp::Rename {
to: join(parent_of(&from), rng.pick(&FILE_NAMES).expect("names")),
from,
}),
// A move to another folder, renamed or not.
72..82 => a_file(rng).map(|from| {
let name = if rng.chance(0.5) {
name_of(&from)
} else {
rng.pick(&FILE_NAMES).expect("names")
};
let parent = rng.pick(&dirs).expect("the root is always there");
FsOp::Rename {
to: join(parent, name),
from,
}
}),
// A folder renamed or moved, with what is in it.
82..86 => {
let from = rng.pick(&dirs[1..]).cloned();
from.map(|from| FsOp::Rename {
to: in_dir(rng, &DIR_NAMES),
from,
})
}
86..94 => a_file(rng).map(|path| FsOp::Unlink { path }),
94..97 => a_file(rng).map(|path| FsOp::UnlinkOpen {
path,
data: data(rng),
}),
_ => rng
.pick(&dirs[1..])
.cloned()
.map(|path| FsOp::Rmdir { path }),
};
op.unwrap_or_else(|| FsOp::Create {
path: in_dir(rng, &FILE_NAMES),
data: data(rng),
})
}
}
/// Content for a write: short and recognisable, now and then empty.
pub(crate) fn data(rng: &mut Rng) -> Vec<u8> {
let len = if rng.chance(0.1) {
0
} else {
rng.between(1, 48)
};
let tag = rng.next_u64();
(0..len).map(|i| b'a' + ((tag + i) % 26) as u8).collect()
}
pub(crate) fn join(parent: &str, name: &str) -> String {
if parent.is_empty() {
name.to_owned()
} else {
format!("{parent}/{name}")
}
}
pub(crate) fn parent_of(path: &str) -> &str {
path.rsplit_once('/').map_or("", |(parent, _)| parent)
}
pub(crate) fn name_of(path: &str) -> &str {
path.rsplit_once('/').map_or(path, |(_, name)| name)
}
fn is_below(path: &str, ancestor: &str) -> bool {
path.len() > ancestor.len() + 1
&& path.starts_with(ancestor)
&& path.as_bytes()[ancestor.len()] == b'/'
}
#[cfg(test)]
mod tests {
use super::*;
fn create(path: &str, data: &[u8]) -> FsOp {
FsOp::Create {
path: path.into(),
data: data.to_vec(),
}
}
#[test]
fn a_rename_over_a_file_replaces_it_and_a_folder_moves_with_its_contents() {
let mut model = Model::default();
model.apply(&FsOp::Mkdir { path: "x".into() }).unwrap();
model.apply(&create("x/a", b"1")).unwrap();
model.apply(&create("b", b"2")).unwrap();
model
.apply(&FsOp::Rename {
from: "b".into(),
to: "x/a".into(),
})
.unwrap();
assert_eq!(model.content("x/a"), Some(&b"2".to_vec()));
model
.apply(&FsOp::Rename {
from: "x".into(),
to: "y".into(),
})
.unwrap();
assert_eq!(model.tree().keys().collect::<Vec<_>>(), vec!["y", "y/a"]);
}
#[test]
fn calls_fail_the_way_posix_says() {
let mut model = Model::default();
model.apply(&FsOp::Mkdir { path: "x".into() }).unwrap();
model.apply(&create("x/a", b"1")).unwrap();
assert_eq!(
model.apply(&FsOp::Mkdir { path: "x".into() }),
Err(libc::EEXIST)
);
assert_eq!(model.apply(&create("x", b"")), Err(libc::EISDIR));
assert_eq!(model.apply(&create("x/a/b", b"")), Err(libc::ENOTDIR));
assert_eq!(model.apply(&create("q/a", b"")), Err(libc::ENOENT));
assert_eq!(
model.apply(&FsOp::Rmdir { path: "x".into() }),
Err(libc::ENOTEMPTY)
);
assert_eq!(
model.apply(&FsOp::Unlink { path: "x".into() }),
Err(libc::EISDIR)
);
let into_itself = FsOp::Rename {
from: "x".into(),
to: "x/y".into(),
};
assert_eq!(model.apply(&into_itself), Err(libc::EINVAL));
}
#[test]
fn a_write_past_the_end_leaves_a_hole() {
let mut model = Model::default();
model.apply(&create("a", b"ab")).unwrap();
let write = FsOp::Write {
path: "a".into(),
offset: 4,
data: b"z".to_vec(),
};
model.apply(&write).unwrap();
assert_eq!(model.content("a"), Some(&b"ab\0\0z".to_vec()));
let nothing = FsOp::Write {
path: "a".into(),
offset: 9,
data: Vec::new(),
};
model.apply(&nothing).unwrap();
assert_eq!(model.content("a"), Some(&b"ab\0\0z".to_vec()));
}
#[test]
fn generated_ops_mostly_succeed() {
let mut rng = Rng::new(9);
let mut model = Model::default();
let ok = (0..500)
.filter(|_| {
let op = model.next_op(&mut rng);
model.apply(&op).is_ok()
})
.count();
assert!(ok > 350, "{ok} of 500 succeeded");
}
}
+79
View File
@@ -0,0 +1,79 @@
//! A small seeded random number generator, so a failing simulation is a seed
//! that can be replayed. SplitMix64: fast, good enough for choosing operations
//! and faults, and no dependency.
#[derive(Clone, Debug)]
pub(crate) struct Rng(u64);
impl Rng {
pub(crate) fn new(seed: u64) -> Self {
Self(seed)
}
pub(crate) fn next_u64(&mut self) -> u64 {
self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
let mut z = self.0;
z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
z ^ (z >> 31)
}
/// A value in `0..n`; `0` when `n` is `0`.
pub(crate) fn below(&mut self, n: u64) -> u64 {
if n == 0 { 0 } else { self.next_u64() % n }
}
/// A value in `lo..=hi`.
pub(crate) fn between(&mut self, lo: u64, hi: u64) -> u64 {
lo + self.below(hi.saturating_sub(lo) + 1)
}
/// True with probability `p`.
pub(crate) fn chance(&mut self, p: f64) -> bool {
p > 0.0 && (self.next_u64() >> 11) as f64 / (1u64 << 53) as f64 <= p
}
pub(crate) fn pick<'a, T>(&mut self, items: &'a [T]) -> Option<&'a T> {
items.get(self.below(items.len() as u64) as usize)
}
/// A child generator, so one consumer's draws do not shift another's.
pub(crate) fn fork(&mut self) -> Self {
Self::new(self.next_u64())
}
}
#[cfg(test)]
mod tests {
use super::Rng;
#[test]
fn a_seed_replays_the_same_sequence() {
let (mut a, mut b) = (Rng::new(7), Rng::new(7));
for _ in 0..100 {
assert_eq!(a.next_u64(), b.next_u64());
}
assert_ne!(Rng::new(7).next_u64(), Rng::new(8).next_u64());
}
#[test]
fn bounded_draws_stay_in_range() {
let mut rng = Rng::new(1);
for _ in 0..1000 {
assert!(rng.below(5) < 5);
assert!((3..=9).contains(&rng.between(3, 9)));
}
assert_eq!(rng.below(0), 0);
assert!(!rng.chance(0.0));
assert!(rng.chance(1.0));
assert!([1, 2, 3].contains(rng.pick(&[1, 2, 3]).unwrap()));
assert_eq!(rng.pick::<u8>(&[]), None);
}
#[test]
fn a_fork_draws_apart_from_its_parent() {
let mut parent = Rng::new(3);
let mut child = parent.fork();
assert_ne!(parent.next_u64(), child.next_u64());
}
}
File diff suppressed because it is too large. Load diff
+322
View File
@@ -0,0 +1,322 @@
//! Turns a call through a mount that never returns into a failed seed.
//!
//! A deadlock in a daemon leaves the run's own thread blocked in the kernel,
//! where no budget check is reached, and a CI job then waits for its timeout.
//! The watchdog runs beside the run. When what it was armed for outlives its
//! limit, it aborts the FUSE connections of the run's mounts, so the blocked
//! call fails and the run reports the hang, and writes where every thread was
//! to `stacks.txt`.
//!
//! The stacks are taken in the process: no tracer can stop a thread that
//! waits on a FUSE request, so `gdb` and `eu-stack` wait as long as the hang.
//! Each thread is sent [`stack_signal`] and captures its own backtrace in the
//! handler. A thread in the kernel takes the signal only once it leaves, so
//! the mounts are aborted after the others have answered, and its backtrace
//! is the call it was blocked in. Its kernel wait channel is recorded first.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread::JoinHandle;
use std::time::{Duration, Instant};
use parking_lot::Mutex;
use super::daemon::wait_until;
/// How long the threads that are not blocked in the kernel have to answer,
/// and the blocked ones once the mounts are aborted.
const ANSWER: Duration = Duration::from_secs(5);
pub(crate) struct Watchdog {
shared: Arc<Shared>,
thread: Option<JoinHandle<()>>,
}
struct Shared {
/// The run's directory: its mounts are below it, and the stacks go in it.
dir: PathBuf,
armed: Mutex<Option<Armed>>,
/// What hung, once something has.
fired: Mutex<Option<Hung>>,
stop: AtomicBool,
}
struct Armed {
what: String,
limit: Duration,
since: Instant,
}
/// What hung, and where every thread was then.
#[derive(Clone, Debug, PartialEq)]
pub(crate) struct Hung {
pub(crate) what: String,
pub(crate) stacks: String,
}
/// Disarms the watchdog when dropped.
pub(crate) struct Watching(Arc<Shared>);
impl Watchdog {
pub(crate) fn new(dir: &Path) -> Self {
let shared = Arc::new(Shared {
dir: dir.to_path_buf(),
armed: Mutex::new(None),
fired: Mutex::new(None),
stop: AtomicBool::new(false),
});
let thread = {
let shared = shared.clone();
std::thread::Builder::new()
.name("sim-watchdog".into())
.spawn(move || watch(&shared))
.expect("spawn the simulation watchdog")
};
Self {
shared,
thread: Some(thread),
}
}
/// Watch `what` until the guard is dropped, and abort the mounts if it is
/// still going after `limit`.
pub(crate) fn arm(&self, limit: Duration, what: impl Into<String>) -> Watching {
*self.shared.armed.lock() = Some(Armed {
what: what.into(),
limit,
since: Instant::now(),
});
Watching(self.shared.clone())
}
/// What hung, if something did. Call [`Watchdog::stop`] first, or the
/// stacks may still be on their way.
pub(crate) fn fired(&self) -> Option<Hung> {
self.shared.fired.lock().clone()
}
/// Stop watching, once the stacks of a hang that was caught are taken.
pub(crate) fn stop(&mut self) {
self.shared.stop.store(true, Ordering::Relaxed);
if let Some(thread) = self.thread.take() {
let _ = thread.join();
}
}
}
impl Drop for Watching {
fn drop(&mut self) {
*self.0.armed.lock() = None;
}
}
impl Drop for Watchdog {
fn drop(&mut self) {
self.stop();
}
}
fn watch(shared: &Shared) {
while !shared.stop.load(Ordering::Relaxed) {
std::thread::sleep(Duration::from_millis(200));
let hung = shared
.armed
.lock()
.as_ref()
.filter(|armed| armed.since.elapsed() > armed.limit)
.map(|armed| format!("{} has not returned after {:?}", armed.what, armed.limit));
let Some(hung) = hung else { continue };
*shared.armed.lock() = None;
let path = shared.dir.join("stacks.txt");
// Before anything is let go: the run reports the hang, not whatever
// the aborted call answers.
*shared.fired.lock() = Some(Hung {
what: format!("{hung}; the stacks are in {}", path.display()),
stacks: String::new(),
});
let stacks = take_stacks(|| abort_mounts_below(&shared.dir));
let what = match std::fs::write(&path, &stacks) {
Ok(()) => format!("{hung}; the stacks are in {}", path.display()),
Err(error) => format!("{hung}; the stacks could not be kept: {error}"),
};
*shared.fired.lock() = Some(Hung { what, stacks });
}
}
/// One thread of this process as the kernel sees it.
struct Task {
tid: i32,
name: String,
/// `D` while it waits in the kernel uninterruptibly.
state: String,
wchan: String,
}
/// Where every other thread of this process is, with `release` called once
/// those not blocked in the kernel have answered.
fn take_stacks(release: impl FnOnce()) -> String {
let own = gettid();
let tasks = tasks();
STACKS.lock().clear();
install_stack_handler();
for task in &tasks {
if task.tid != own {
// SAFETY: tgkill only sends a signal, whose handler is installed.
unsafe {
libc::syscall(
libc::SYS_tgkill,
std::process::id() as i32,
task.tid,
stack_signal(),
)
};
}
}
let answered = |blocked: bool| {
let stacks = STACKS.lock();
tasks
.iter()
.filter(|task| task.tid != own && (blocked || task.state != "D"))
.all(|task| stacks.contains_key(&task.tid))
};
wait_until(ANSWER, || answered(false));
release();
wait_until(ANSWER, || answered(true));
let stacks = std::mem::take(&mut *STACKS.lock());
let mut out = String::new();
for task in tasks.iter().filter(|task| task.tid != own) {
out.push_str(&format!(
"thread {} {:?}, state {}, waiting in {}\n",
task.tid, task.name, task.state, task.wchan
));
match stacks.get(&task.tid) {
Some(stack) => out.push_str(&format!("{stack}\n")),
None => out.push_str(" (no answer)\n\n"),
}
}
out
}
fn tasks() -> Vec<Task> {
let Ok(entries) = std::fs::read_dir("/proc/self/task") else {
return Vec::new();
};
let read = |dir: &Path, file: &str| {
std::fs::read_to_string(dir.join(file))
.unwrap_or_default()
.trim()
.to_owned()
};
entries
.flatten()
.filter_map(|entry| {
let tid = entry.file_name().to_str()?.parse().ok()?;
let dir = entry.path();
// The state follows the name in parentheses, which may hold spaces.
let stat = read(&dir, "stat");
let state = stat.rsplit_once(") ")?.1.split(' ').next()?.to_owned();
Some(Task {
tid,
name: read(&dir, "comm"),
state,
wchan: read(&dir, "wchan"),
})
})
.collect()
}
/// The backtraces the threads captured, by thread id.
static STACKS: Mutex<BTreeMap<i32, std::backtrace::Backtrace>> = Mutex::new(BTreeMap::new());
/// The signal that asks a thread for its stack: one of the real-time signals
/// that nothing in the daemon uses.
fn stack_signal() -> i32 {
libc::SIGRTMIN() + 7
}
fn install_stack_handler() {
static INSTALLED: std::sync::Once = std::sync::Once::new();
extern "C" fn handler(_: i32) {
// Not async-signal-safe: the capture allocates. The process is being
// taken down for a hang anyway, and a thread interrupted inside the
// allocator only fails to answer.
let stack = std::backtrace::Backtrace::force_capture();
STACKS.lock().insert(gettid(), stack);
}
INSTALLED.call_once(|| {
// SAFETY: a zeroed sigaction with a handler and SA_RESTART is valid,
// and the handler only touches `STACKS`.
unsafe {
let mut action: libc::sigaction = std::mem::zeroed();
action.sa_sigaction = handler as extern "C" fn(i32) as usize;
action.sa_flags = libc::SA_RESTART;
libc::sigemptyset(&mut action.sa_mask);
libc::sigaction(stack_signal(), &action, std::ptr::null_mut());
}
});
}
fn gettid() -> i32 {
// SAFETY: gettid has no arguments and cannot fail.
unsafe { libc::gettid() }
}
/// Abort the FUSE connection of every mount below `dir`. Its connection is
/// named by the mount's device minor, and the mount's owner may abort it.
fn abort_mounts_below(dir: &Path) {
let Ok(mountinfo) = std::fs::read_to_string("/proc/self/mountinfo") else {
return;
};
for line in mountinfo.lines() {
let fields: Vec<&str> = line.split(' ').collect();
let (Some(device), Some(mountpoint)) = (fields.get(2), fields.get(4)) else {
continue;
};
if !Path::new(mountpoint).starts_with(dir) {
continue;
}
if let Some((_, minor)) = device.split_once(':') {
let _ = std::fs::write(format!("/sys/fs/fuse/connections/{minor}/abort"), "1");
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::sim::daemon::scratch;
#[test]
#[ignore = "signals every thread of the test process: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_call_past_its_limit_is_reported_with_the_stacks() {
let dir = scratch("watchdog");
let mut watchdog = Watchdog::new(&dir);
{
let _watching = watchdog.arm(Duration::from_millis(100), "a nap");
std::thread::sleep(Duration::from_secs(1));
}
watchdog.stop();
let hung = watchdog.fired().expect("the watchdog fired");
assert!(
hung.what.starts_with("a nap has not returned after"),
"{hung:?}"
);
assert!(hung.stacks.contains("a_call_past_its_limit"), "{hung:?}");
let kept = std::fs::read_to_string(dir.join("stacks.txt")).unwrap_or_default();
assert_eq!(kept, hung.stacks);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn a_call_within_its_limit_is_not() {
let dir = scratch("watchdog-quiet");
let watchdog = Watchdog::new(&dir);
drop(watchdog.arm(Duration::from_millis(100), "a quick call"));
std::thread::sleep(Duration::from_millis(500));
assert_eq!(watchdog.fired(), None);
drop(watchdog);
let _ = std::fs::remove_dir_all(&dir);
}
}
+439 -38
View File
@@ -128,6 +128,61 @@ pub(crate) struct WriteHandle {
pub(crate) open_count: usize,
}
impl WriteHandle {
/// A handle on a file `create` just made: an empty base, so everything
/// written through it is authored.
///
/// The base is `node` as it actually exists — the empty file the server just
/// minted, or a queued create's placeholder — never the local clock or a
/// blank. `queue_revision` turns it into `StagedWrite::based_on` and the
/// drain compares the remote against that. A `now_secs()` mtime differs from
/// the server's stamp by the create round-trip. A missing revision id drops
/// the check to `(mtime, size)`, and moving or renaming the file advances
/// its mtime on the server. Either way the first write to a brand-new file
/// lands as a conflict copy of its own empty create (`docs/BUGS.md` B113).
pub(crate) fn created(ino: u64, node: &Node, file: File, path: PathBuf) -> Self {
Self {
ino,
uid: node.uid.clone(),
file: Arc::new(file),
path,
written: Intervals::default(),
len: 0,
base_size: 0,
base_mtime: node.modification_time,
base_revision_id: crate::node_revision_id(node),
dirty: false,
open_count: 0,
}
}
/// Write over `node` from now on: a revision this daemon has just put on the
/// server under the handle, by sealing an upload or by landing the queued
/// create the handle was opened on. The base the handle opened over is gone,
/// and keeping it would make the drain read our own revision as another
/// device's edit.
pub(crate) fn rebase_onto(&mut self, node: &Node) {
self.base_mtime = node.modification_time;
self.base_size = crate::node_size(node);
self.base_revision_id = crate::node_revision_id(node);
}
/// Account for `[offset, end)` having been written through this handle.
///
/// A write past the end leaves a hole that reads as zeros, exactly like a
/// grow through `setattr`, so the hole is authored too. Left unauthored, a
/// hole below the base size was refilled from the base at commit: a file
/// shrunk and then written past its new end got its old bytes back.
pub(crate) fn record_write(&mut self, offset: u64, end: u64) {
if offset > self.len {
self.written.add(self.len, offset);
}
self.written.add(offset, end);
self.len = self.len.max(end);
self.dirty = true;
}
}
/// A released write whose upload has not happened yet (offline.md Phase 3).
///
/// The bytes live in the content cache's staging dir and the intent lives in the
@@ -146,8 +201,23 @@ pub(crate) struct PendingRevision {
/// time so this module stays free of `fuser`.
pub(crate) type DirListing = Vec<(u64, bool, String)>;
/// Where [`State::next_ino`] starts: far above any local id a database hands
/// out, so a fallback inode never names a node that has a row.
pub(crate) const FALLBACK_INO: u64 = 1 << 48;
/// The inode of the node with local id `lid` ([`Db::lids_for`]). One above the
/// lid, so no node takes [`crate::ROOT_INO`], which each session uses for its
/// own root. The same in every mount and after a restart.
pub(crate) fn lid_ino(lid: i64) -> u64 {
lid.unsigned_abs() + crate::ROOT_INO
}
/// Mutable inode bookkeeping, guarded by a mutex because fuser drives the
/// `Filesystem` trait through `&self`.
///
/// One tree for every mount of the daemon. Each FUSE session sees it from its
/// own root ([`State::roots`]) and calls that root inode 1; `ProtonFs`
/// translates at the kernel boundary.
pub(crate) struct State {
/// inode -> node metadata.
pub(crate) entries: HashMap<u64, Entry>,
@@ -156,6 +226,12 @@ pub(crate) struct State {
/// Cached directory listings: parent inode -> child inodes. Presence of a
/// key means the directory has been enumerated.
pub(crate) children: HashMap<u64, Vec<u64>>,
/// The inodes some session has been rooted at ([`State::install_root`]).
/// A root is its own parent. Kept after its session unmounts: the node is
/// still one of ours.
pub(crate) roots: HashSet<u64>,
/// The next inode for a node the database could not give a local id, from
/// a range lids never reach ([`FALLBACK_INO`]).
pub(crate) next_ino: u64,
/// Shared write state keyed by inode. Concurrent writers share the scratch file.
pub(crate) active_writes: HashMap<u64, WriteHandle>,
@@ -177,7 +253,7 @@ pub(crate) struct State {
/// intern/root refresh. Core drains this set and notifies the matching
/// kernel only after releasing the State lock.
pub(crate) access_changes: HashSet<u64>,
/// Persisted shared-root authority, loaded once per inode space and updated
/// Persisted shared-root authority, loaded once and updated
/// on every explicit role/tombstone change. Descendant inheritance reads
/// this map instead of issuing one SQLite query per interned inode.
pub(crate) share_access: HashMap<NodeUid, Access>,
@@ -245,15 +321,6 @@ pub(crate) struct StateGuard<'a> {
db: &'a Arc<Db>,
}
/// Take a mount's inode lock through a [`StateGuard`], for the loops that walk
/// every live mount rather than this `Core`'s own.
pub(crate) fn lock_state<'a>(
state: &'a parking_lot::Mutex<State>,
db: &'a Arc<Db>,
) -> StateGuard<'a> {
StateGuard::new(state.lock(), db)
}
impl<'a> StateGuard<'a> {
pub(crate) fn new(guard: parking_lot::MutexGuard<'a, State>, db: &'a Arc<Db>) -> Self {
Self {
@@ -294,9 +361,6 @@ impl Drop for StateGuard<'_> {
impl State {
/// A fresh, empty inode space over `db`.
///
/// `next_ino` differs by caller: a real mount hands out [`crate::ROOT_INO`]
/// to its root and starts allocating above it, while a bare state under test
/// has no root and starts at 1.
/// Apply the write-throughs this state owes SQLite, here and now.
///
/// Production code never calls this: [`StateGuard`] releases the inode lock
@@ -323,12 +387,15 @@ impl State {
self.db.clone()
}
pub(crate) fn new(db: Arc<Db>, share_access: HashMap<NodeUid, Access>, next_ino: u64) -> Self {
/// A fresh, empty tree over `db`. A node's inode comes from its local id
/// ([`lid_ino`]).
pub(crate) fn new(db: Arc<Db>, share_access: HashMap<NodeUid, Access>) -> Self {
Self {
entries: HashMap::new(),
by_uid: HashMap::new(),
children: HashMap::new(),
next_ino,
roots: HashSet::new(),
next_ino: FALLBACK_INO,
active_writes: HashMap::new(),
handles: HashMap::new(),
next_fh: 1,
@@ -339,18 +406,55 @@ impl State {
outbox: Vec::new(),
}
}
/// Whether `uid` names an owned node currently reachable from this mount's
/// root. Resident entries can outlive their dentries for open-handle and
/// access-revocation semantics, so `by_uid` membership alone is not proof
/// that a control-socket operation may address the node.
pub(crate) fn owns_visible_uid(&self, uid: &NodeUid) -> bool {
let Some(root) = self.entries.get(&crate::ROOT_INO) else {
return false;
};
if uid.volume_id != root.uid.volume_id {
return false;
/// Make `root` the root of a session and return its inode, which comes
/// from its local id like any other node's.
///
/// The root is always owned. It must never be classified as a shared root,
/// even when the API returns membership data (which happens when the owner
/// shares the folder with someone else), so a stale share row a downgrade
/// may have left for it is dropped. A node already in the tree keeps its
/// inode and its lookups, and stops naming a parent: a session's `..` at
/// its root is the root.
pub(crate) fn install_root(&mut self, root: Node) -> u64 {
if let Err(e) = self.db.upsert_node(&root) {
warn!(uid = %root.uid, error = %e, "db upsert root failed");
}
self.share_access.remove(&root.uid);
let _ = self.db.delete_share_access(&root.uid);
let ino = match self.by_uid.get(&root.uid) {
Some(&ino) => ino,
None => self.new_inos(&[&root])[&root.uid],
};
if let Some(old) = self.entries.get(&ino).map(|e| e.parent)
&& old != ino
&& let Some(kids) = self.children.get_mut(&old)
{
kids.retain(|&k| k != ino);
}
self.by_uid.insert(root.uid.clone(), ino);
let lookup_count = self.entries.get(&ino).map_or(1, |e| e.lookup_count.max(1));
self.entries.insert(
ino,
Entry {
uid: root.uid.clone(),
parent: ino,
node: root,
access: Access::Owner,
lookup_count,
open_count: 0,
unlinked: false,
},
);
self.roots.insert(ino);
ino
}
/// Whether `uid` names an owned node currently reachable from one of
/// `roots`, the roots of the sessions that are mounted. Resident entries
/// can outlive their dentries for open-handle and access-revocation
/// semantics, so `by_uid` membership alone is not proof that a
/// control-socket operation may address the node.
pub(crate) fn owns_visible_uid(&self, uid: &NodeUid, roots: &[u64]) -> bool {
let Some(&target) = self.by_uid.get(uid) else {
return false;
};
@@ -363,7 +467,9 @@ impl State {
let Some(entry) = self.entries.get(&current) else {
return false;
};
if entry.uid.volume_id != root.uid.volume_id
// The node's root scopes it to its own volume, excluding foreign
// shared-with-me residents from uid-addressed sharing.
if entry.uid.volume_id != uid.volume_id
|| entry.unlinked
|| entry.node.trashed
|| entry.uid != entry.node.uid
@@ -371,7 +477,7 @@ impl State {
{
return false;
}
if current == crate::ROOT_INO {
if roots.contains(&current) {
return true;
}
@@ -413,15 +519,17 @@ impl State {
self.access_changes.drain().collect()
}
/// Whether `uid` lives on the volume this mount is rooted in.
/// Whether `uid` lives on a volume some mount is rooted in.
///
/// Used to decide how a node with no resolvable parent is classified:
/// our own volume fails open (it is owned content), a foreign volume
/// fails closed (it can only be there because of a share).
pub(crate) fn is_own_volume(&self, uid: &NodeUid) -> bool {
self.entries
.get(&crate::ROOT_INO)
.is_some_and(|root| root.uid.volume_id == uid.volume_id)
self.roots.iter().any(|ino| {
self.entries
.get(ino)
.is_some_and(|root| root.uid.volume_id == uid.volume_id)
})
}
fn stored_share_access(&self, uid: &NodeUid) -> Option<Access> {
@@ -770,11 +878,54 @@ impl State {
}
}
pub(crate) fn intern(&mut self, parent: u64, node: Node) -> u64 {
pub(crate) fn intern(&mut self, parent: u64, mut node: Node) -> u64 {
self.keep_revision_mtime(&mut node);
self.outbox.push(DbWrite::Upsert(vec![node.clone()]));
self.intern_mem(parent, node)
}
/// Keep what is already known of a file whose content has not changed.
///
/// Drive stamps a link's `ModifyTime` on a rename or a move as well, while
/// POSIX leaves a file's mtime alone there. The content cache is keyed by
/// it, too: taking the new stamp made the cached bytes unreachable, so the
/// next partial write could not fill its gaps (docs/BUGS.md B123). The
/// active revision id says whether the content is still the same, and so
/// whether the size and digest a listing left out are still the ones known.
fn keep_revision_mtime(&self, node: &mut Node) {
let Some(entry) = self
.by_uid
.get(&node.uid)
.and_then(|ino| self.entries.get(ino))
else {
return;
};
if let (
NodeKind::File {
active_revision_id: Some(now),
claimed_size,
content_sha1,
..
},
NodeKind::File {
active_revision_id: Some(was),
claimed_size: size_was,
content_sha1: sha1_was,
..
},
) = (&mut node.kind, &entry.node.kind)
&& now == was
{
node.modification_time = entry.node.modification_time;
if claimed_size.is_none() {
*claimed_size = *size_was;
}
if content_sha1.is_none() {
content_sha1.clone_from(sha1_was);
}
}
}
/// Allocate (or reuse) a stable inode for a node, updating the hot-cache maps
/// only. Every caller owes the DB a write-through — see the callers below;
/// the split exists so a batch can pay for one transaction instead of `n`.
@@ -784,11 +935,24 @@ impl State {
}
fn intern_mem_with_access(&mut self, parent: u64, node: Node, access: Access) -> u64 {
self.intern_mem_at(parent, node, access, None)
}
/// [`State::intern_mem_with_access`], with the inode a new entry takes when
/// the caller already has it ([`State::new_inos`]).
fn intern_mem_at(
&mut self,
parent: u64,
mut node: Node,
access: Access,
ino: Option<u64>,
) -> u64 {
if let Some(&ino) = self.by_uid.get(&node.uid) {
let changed = self
.entries
.get(&ino)
.is_some_and(|entry| entry.access != access);
self.keep_open_write_size(ino, &mut node);
if let Some(e) = self.entries.get_mut(&ino) {
e.node = node;
e.parent = parent;
@@ -801,8 +965,13 @@ impl State {
}
return ino;
}
let ino = self.next_ino;
self.next_ino += 1;
let ino = match ino {
Some(ino) => ino,
None => self
.new_inos(&[&node])
.remove(&node.uid)
.unwrap_or_default(),
};
self.by_uid.insert(node.uid.clone(), ino);
self.entries.insert(
ino,
@@ -819,6 +988,41 @@ impl State {
ino
}
/// The inodes for nodes that have no entry yet, from their local ids, in
/// one transaction. Should the database fail, they come from
/// [`State::next_ino`] instead, so the mount keeps working, with inodes
/// that last only this run. So do the inodes of virtual folders, which are
/// never stored.
fn new_inos(&mut self, nodes: &[&Node]) -> HashMap<NodeUid, u64> {
let (virtual_nodes, stored): (Vec<&Node>, Vec<&Node>) = nodes
.iter()
.partition(|node| crate::r#virtual::is_virtual_uid(&node.uid));
let mut inos = match self.db.lids_for(&stored) {
Ok(lids) => stored
.iter()
.zip(lids)
.map(|(node, lid)| (node.uid.clone(), lid_ino(lid)))
.collect(),
Err(error) => {
warn!(%error, "no local ids for new inodes; numbering them for this run");
self.run_inos(&stored)
}
};
inos.extend(self.run_inos(&virtual_nodes));
inos
}
fn run_inos(&mut self, nodes: &[&Node]) -> HashMap<NodeUid, u64> {
nodes
.iter()
.map(|node| {
let ino = self.next_ino;
self.next_ino += 1;
(node.uid.clone(), ino)
})
.collect()
}
/// Decrement lookup count for an inode and prune if lookup_count == 0 && open_count == 0 && unlinked.
pub(crate) fn forget_lookup(&mut self, ino: u64, nlookup: u64) -> Option<(u64, String)> {
if let Some(entry) = self.entries.get_mut(&ino) {
@@ -833,6 +1037,23 @@ impl State {
/// Allocate (or reuse) a stable inode for a node that came *from* the
/// database, which is why nothing is written back.
/// Carry the size and mtime of a file open for writing over onto `node`,
/// which is about to replace its entry.
///
/// The open handle is ahead of anything Drive can report until release
/// queues it. Taking the remote's size shrank a SQLite database to its empty
/// create while SQLite held it open, and the kernel then capped every read
/// at that size (docs/BUGS.md B109).
pub(crate) fn keep_open_write_size(&self, ino: u64, node: &mut Node) {
let (Some(aw), Some(entry)) = (self.active_writes.get(&ino), self.entries.get(&ino)) else {
return;
};
if let NodeKind::File { claimed_size, .. } = &mut node.kind {
*claimed_size = Some(aw.len as i64);
node.modification_time = entry.node.modification_time;
}
}
pub(crate) fn intern_from_db(&mut self, parent: u64, node: Node) -> u64 {
self.intern_mem(parent, node)
}
@@ -842,11 +1063,27 @@ impl State {
///
/// This is what keeps `ls` on a large folder quick: one commit for the
/// listing, rather than one autocommit — and one fsync — per child.
pub(crate) fn intern_batch(&mut self, parent: u64, nodes: Vec<Node>) -> Vec<u64> {
pub(crate) fn intern_batch(&mut self, parent: u64, mut nodes: Vec<Node>) -> Vec<u64> {
for node in &mut nodes {
self.keep_revision_mtime(node);
}
self.outbox.push(DbWrite::Upsert(nodes.clone()));
let fresh: Vec<&Node> = nodes
.iter()
.filter(|node| !self.by_uid.contains_key(&node.uid))
.collect();
let mut inos = if fresh.is_empty() {
HashMap::new()
} else {
self.new_inos(&fresh)
};
nodes
.into_iter()
.map(|node| self.intern_mem(parent, node))
.map(|node| {
let access = self.access_for_node(parent, &node);
let ino = inos.remove(&node.uid);
self.intern_mem_at(parent, node, access, ino)
})
.collect()
}
@@ -1164,7 +1401,7 @@ mod tests {
let dir = TempDir::new();
let db = Db::open(&dir.0.join("cache.db")).unwrap();
let share_access = db.all_share_access().unwrap();
let st = State::new(Arc::new(db), share_access, 1);
let st = State::new(Arc::new(db), share_access);
(st, dir)
}
@@ -1182,6 +1419,36 @@ mod tests {
(st, dir, src, dst)
}
/// A node's inode comes from its row's local id, so it is the same in
/// every mount, after the hot cache drops it, and after a restart.
#[test]
fn a_node_keeps_its_inode_across_mounts_evictions_and_restarts() {
let (mut st, _dir) = state();
let inos = st.intern_batch(
0,
vec![
node("a", "root", "a.txt", false),
node("b", "root", "b.txt", false),
],
);
st.flush_outbox();
let db = st.db.clone();
assert_ne!(inos[0], inos[1]);
assert!(!inos.contains(&crate::ROOT_INO));
let mut other = State::new(db.clone(), HashMap::new());
assert_eq!(other.intern(0, node("b", "root", "b.txt", false)), inos[1]);
st.forget_mem(&uid("a"));
assert_eq!(st.intern(0, node("a", "root", "a.txt", false)), inos[0]);
let mut restarted = State::new(db, HashMap::new());
assert_eq!(
restarted.intern_batch(0, vec![node("a", "root", "a.txt", false)]),
[inos[0]]
);
}
/// Audit A5. Moving a file out of a folder leaves that folder's DB row
/// claiming a complete listing, so once the hot cache drops it, remote
/// changes under it stop being seen. `relocate` has to clear both sides.
@@ -1247,7 +1514,7 @@ mod tests {
let root = uid("root").to_string();
let state = parking_lot::Mutex::new(st);
{
let mut guard = lock_state(&state, &db);
let mut guard = StateGuard::new(state.lock(), &db);
guard.intern(0, node("root", "none", "My Files", true));
assert!(
db.node_by_uid(&root).unwrap().is_none(),
@@ -1490,6 +1757,7 @@ mod tests {
})
.collect();
st.next_ino = crate::ROOT_INO + nodes.len() as u64;
st.roots.insert(crate::ROOT_INO);
// Reverse order so entries are inserted before their parents, matching
// the hydration path's non-topological materialization.
for (i, node) in nodes.iter().enumerate().rev() {
@@ -1863,4 +2131,137 @@ mod tests {
let shared = st.intern(owned_root, shared);
assert_eq!(st.entries[&shared].access, Access::Viewer);
}
fn at_revision(mut node: Node, revision: &str, mtime: i64) -> Node {
node.modification_time = mtime;
if let NodeKind::File {
active_revision_id, ..
} = &mut node.kind
{
*active_revision_id = Some(revision.into());
}
node
}
#[test]
fn a_renamed_file_keeps_its_mtime_until_its_content_changes() {
let (mut st, _dir) = state();
let root = st.intern(0, node("root", "none", "My Files", true));
let mut file = node("file", "root", "a.txt", false);
if let NodeKind::File {
claimed_size,
total_size_on_storage,
..
} = &mut file.kind
{
*claimed_size = Some(7);
*total_size_on_storage = 519;
}
let ino = st.intern(root, at_revision(file.clone(), "r1", 100));
// Drive stamps the link on a rename; the content is still r1.
let mut renamed = at_revision(file.clone(), "r1", 160);
renamed.name = "b.txt".into();
st.intern_batch(root, vec![renamed]);
assert_eq!(st.entries[&ino].node.modification_time, 100);
assert_eq!(st.entries[&ino].node.name, "b.txt");
// A listing that leaves the size out still describes revision r1.
let mut bare = at_revision(file.clone(), "r1", 170);
assert_eq!(crate::node_size(&st.entries[&ino].node), 7);
if let NodeKind::File { claimed_size, .. } = &mut bare.kind {
*claimed_size = None;
}
st.intern(root, bare);
assert_eq!(st.entries[&ino].node.modification_time, 100);
assert_eq!(crate::node_size(&st.entries[&ino].node), 7);
st.intern(root, at_revision(file, "r2", 200));
assert_eq!(st.entries[&ino].node.modification_time, 200);
}
fn handle(len: u64, base_size: u64) -> WriteHandle {
let path = std::env::temp_dir().join(format!("pdfs-handle-test-{}", std::process::id()));
let file = File::create(&path).unwrap();
let _ = std::fs::remove_file(&path);
WriteHandle {
ino: 2,
uid: uid("file"),
file: Arc::new(file),
path,
written: Intervals::default(),
len,
base_size,
base_mtime: 0,
base_revision_id: None,
dirty: false,
open_count: 1,
}
}
/// A file created offline is written while its queued create lands. The
/// placeholder had no revision, so the handle takes the empty file the
/// create made as its base (docs/BUGS.md B113).
#[test]
fn a_handle_on_a_placeholder_is_based_on_the_create_that_landed() {
let mut h = handle(0, 0);
assert_eq!(h.base_revision_id, None);
let mut landed = node("real", "root", "f.mkv", false);
landed.modification_time = 160;
if let NodeKind::File {
active_revision_id, ..
} = &mut landed.kind
{
*active_revision_id = Some("empty".into());
}
h.rebase_onto(&landed);
assert_eq!(h.base_mtime, 160);
assert_eq!(h.base_size, 0);
assert_eq!(h.base_revision_id.as_deref(), Some("empty"));
}
#[test]
fn a_listing_does_not_shrink_a_file_open_for_writing() {
let (mut st, _dir) = state();
let ino = st.intern(0, node("file", "root", "workload.db", false));
let mut h = handle(16384, 0);
h.ino = ino;
st.active_writes.insert(ino, h);
st.set_size(ino, 16384);
st.touch_mtime(ino, 500);
// The remote still holds the empty file the create sealed.
st.intern(0, node("file", "root", "workload.db", false));
assert_eq!(crate::node_size(&st.entries[&ino].node), 16384);
assert_eq!(st.entries[&ino].node.modification_time, 500);
// Once the handle is gone the remote is authoritative again.
st.active_writes.remove(&ino);
st.intern(0, node("file", "root", "workload.db", false));
assert_eq!(crate::node_size(&st.entries[&ino].node), 0);
}
#[test]
fn a_write_past_the_end_authors_the_hole_it_leaves() {
// Opened over a 100-byte base, shrunk to 10, then written at 50: bytes
// 10..50 are zeros now, not the base's bytes 10..50.
let mut h = handle(100, 100);
h.written.clip(10);
h.len = 10;
h.record_write(50, 60);
assert_eq!(h.len, 60);
assert_eq!(
h.written.segments(0, 60),
vec![(0, 10, false), (10, 60, true)],
"only the head still comes from the base"
);
}
#[test]
fn a_write_inside_the_file_leaves_the_rest_to_the_base() {
let mut h = handle(100, 100);
h.record_write(40, 50);
assert_eq!(h.len, 100);
assert_eq!(
h.written.segments(0, 100),
vec![(0, 40, false), (40, 50, true), (50, 100, false)]
);
}
}
+4 -5
View File
@@ -165,7 +165,7 @@ impl Core {
}
match self
.rt
.block_on(self.client.trash_nodes(std::slice::from_ref(uid)))
.block_on(self.drive.trash_nodes(std::slice::from_ref(uid)))
.and_then(batch::into_unit)
{
Ok(()) => {}
@@ -175,10 +175,9 @@ impl Core {
return;
}
}
// Every mount, not just the primary: a conflict copy inside a sync
// folder lives in that fork's inode space, and leaving it hooked up
// there keeps a node we have just trashed remotely visible and
// readable there (`docs/BUGS.md` B74).
// Leaving a conflict copy hooked up in the tree keeps a node we have
// just trashed remotely visible and readable in whichever mount shows
// it (`docs/BUGS.md` B74).
self.for_each_state(|st| {
st.forget_or_unlink(uid);
});
+242 -26
View File
@@ -73,6 +73,9 @@ enum Applied {
Downloaded,
/// A conflict copy was kept.
Conflict,
/// The local file to upload was deleted after the scan planned it. There is
/// nothing left to send, and the next pass sees the deletion (B108).
Vanished,
}
impl Core {
@@ -367,7 +370,7 @@ impl Core {
HashMap::new()
} else {
self.rt
.block_on(self.client.enumerate_nodes_light(&check))
.block_on(self.drive.enumerate_nodes_light(&check))
.map_err(|e| format!("resolve changed nodes: {e}"))?
.into_iter()
.filter(|n| !n.trashed)
@@ -522,7 +525,7 @@ impl Core {
};
if let Err(e) = self
.rt
.block_on(self.client.trash_nodes(&[uid]))
.block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit)
{
warn!(rel, error = %e, "sync: trash remote failed");
@@ -773,7 +776,7 @@ impl Core {
let remote = r.expect("remote deletion requires a remote file");
if let Err(e) = self
.rt
.block_on(self.client.trash_nodes(std::slice::from_ref(&remote.uid)))
.block_on(self.drive.trash_nodes(std::slice::from_ref(&remote.uid)))
.and_then(batch::into_unit)
{
warn!(rel, error = %e, "sync: trash remote failed");
@@ -836,7 +839,7 @@ impl Core {
for (rel, uid) in delete_remote_dirs {
if let Err(e) = self
.rt
.block_on(self.client.trash_nodes(&[uid]))
.block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit)
{
warn!(rel, error = %e, "sync: trash remote folder failed");
@@ -891,14 +894,14 @@ impl Core {
) -> Result<(), String> {
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(folder))
.block_on(self.drive.enumerate_folder_children_node_uids(folder))
.map_err(|e| format!("enumerate {folder}: {e}"))?;
if uids.is_empty() {
return Ok(());
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes_light(&uids))
.block_on(self.drive.enumerate_nodes_light(&uids))
.map_err(|e| format!("resolve nodes: {e}"))?;
for node in nodes {
if node.trashed || !node.is_folder() {
@@ -943,14 +946,14 @@ impl Core {
}
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(folder))
.block_on(self.drive.enumerate_folder_children_node_uids(folder))
.map_err(|e| format!("enumerate {folder}: {e}"))?;
if uids.is_empty() {
return Ok(());
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes_light(&uids))
.block_on(self.drive.enumerate_nodes_light(&uids))
.map_err(|e| format!("resolve nodes: {e}"))?;
// Files whose recorded mtime no longer matches: their size has to be
@@ -968,7 +971,7 @@ impl Core {
HashMap::new()
} else {
self.rt
.block_on(self.client.enumerate_nodes(&stale))
.block_on(self.drive.enumerate_nodes(&stale))
.map_err(|e| format!("resolve nodes: {e}"))?
.iter()
.map(|n| (n.uid.to_string(), remote_file_sig(n).1))
@@ -1136,7 +1139,7 @@ impl Core {
match op {
Pending::CreateDir { rel, parent } => {
let uid = self
.client
.drive
.create_folder(parent, base_name(rel), Some(now_secs()))
.await
.map_err(|e| format!("create remote folder {rel}: {e}"))?;
@@ -1147,13 +1150,10 @@ impl Core {
Ok(Applied::Dir(rel.clone(), uid))
}
Pending::UploadNew { rel, parent } => {
self.upload_new(folder_id, local_root, rel, parent).await?;
Ok(Applied::Uploaded)
self.upload_new(folder_id, local_root, rel, parent).await
}
Pending::UploadRevision { rel, uid } => {
self.upload_revision(folder_id, local_root, rel, uid)
.await?;
Ok(Applied::Uploaded)
self.upload_revision(folder_id, local_root, rel, uid).await
}
Pending::Download {
rel,
@@ -1217,10 +1217,12 @@ impl Core {
local_root: &Path,
rel: &str,
parent: &NodeUid,
) -> Result<(), String> {
) -> Result<Applied, String> {
let path = local_root.join(rel_to_path(rel));
let name = base_name(rel);
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
let Some(file) = open_to_upload(&path, rel)? else {
return Ok(Applied::Vanished);
};
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
let mtime = system_mtime(&meta);
// Taken from the descriptor being streamed, so it names the content that
@@ -1238,7 +1240,7 @@ impl Core {
.begin(name, "", TransferDirection::Upload, meta.len()),
);
let uid = self
.client
.drive
// Mirror push: the local file is authoritative. If a prior attempt was
// interrupted mid-upload it left an unsealed draft of this name; recover
// it even across a daemon restart (which rotates our client uid), so the
@@ -1261,7 +1263,8 @@ impl Core {
self.invalidate_children_of(parent);
warn_if_torn(rel, &path, streamed);
self.record_file_baseline(folder_id, rel, Some(streamed), &uid)
.await
.await?;
Ok(Applied::Uploaded)
}
/// Upload a changed local file as a new revision of an existing remote node.
@@ -1271,9 +1274,11 @@ impl Core {
local_root: &Path,
rel: &str,
uid: &NodeUid,
) -> Result<(), String> {
) -> Result<Applied, String> {
let path = local_root.join(rel_to_path(rel));
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
let Some(file) = open_to_upload(&path, rel)? else {
return Ok(Applied::Vanished);
};
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
let mtime = system_mtime(&meta);
let streamed = LocalSig::from(&meta);
@@ -1289,13 +1294,14 @@ impl Core {
meta.len(),
),
);
self.client
self.drive
.upload_new_revision_from(uid, reader, meta.len() as i64, thumbnails, Some(mtime))
.await
.map_err(|e| format!("upload revision {rel}: {e}"))?;
warn_if_torn(rel, &path, streamed);
self.record_file_baseline(folder_id, rel, Some(streamed), uid)
.await
.await?;
Ok(Applied::Uploaded)
}
/// Download a remote file to its local path (atomically via a temp file),
@@ -1328,7 +1334,21 @@ impl Core {
.map(|d| d.as_nanos())
.unwrap_or(0)
));
{
// A file an on-demand mount already read, say one just moved here from
// it, is in the content cache under the same revision key.
let seeded = size > 0 && self.cache.copy_cached_to(uid, mtime, size as u64, &tmp);
if seeded {
std::fs::File::open(&tmp)
.and_then(|f| f.sync_all())
.map_err(|e| {
let _ = std::fs::remove_file(&tmp);
format!("sync tmp {rel}: {e}")
})?;
debug!(
rel,
"sync: placed from the content cache instead of downloading"
);
} else {
let out = std::fs::File::create(&tmp).map_err(|e| format!("create tmp {rel}: {e}"))?;
// Count the bytes as they land, so the download reports progress like
// an on-demand hydration does.
@@ -1339,7 +1359,7 @@ impl Core {
size.max(0) as u64,
);
let mut out = CountingWriter::new(out, &guard);
self.client
self.drive
.download_file_to(uid, &mut out)
.await
.map_err(|e| {
@@ -1404,7 +1424,7 @@ impl Core {
let (lmtime, lsize) = (local.mtime, local.size);
// Re-fetch the node so the stored remote signature is exactly what a walk
// will report next time.
let (rmtime, rsize) = match self.client.enumerate_nodes(std::slice::from_ref(uid)).await {
let (rmtime, rsize) = match self.drive.enumerate_nodes(std::slice::from_ref(uid)).await {
Ok(nodes) => nodes
.first()
.map(remote_file_sig)
@@ -1431,6 +1451,23 @@ impl Core {
}
}
/// Open a local file for upload, or `None` when it is gone.
///
/// A scan plans uploads from a listing, and the file can be deleted before its
/// upload starts — a build or a test tree cleaned up mid-pass. That is not a
/// failure: the next pass sees the deletion. Counting it as one put the folder
/// in `error` for a file nobody wanted any more (B108).
fn open_to_upload(path: &Path, rel: &str) -> Result<Option<std::fs::File>, String> {
match std::fs::File::open(path) {
Ok(file) => Ok(Some(file)),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
debug!(rel, "sync: file deleted before its upload; skipping");
Ok(None)
}
Err(e) => Err(format!("open {rel}: {e}")),
}
}
/// Say so when a file changed under an upload that was streaming it.
///
/// Purely diagnostic — the correction is already in place, because the baseline
@@ -1602,6 +1639,76 @@ fn walk_local_tree(
Ok(())
}
/// Why the mirror copy at `rel` holds something Proton Drive does not, or
/// `None` when everything under it is on Drive exactly as it is on disk.
///
/// A move out of a mirror folder removes the local copy once Drive has moved
/// the node, so anything only this disk has would be lost with it: a file not
/// uploaded yet, one edited since, a deletion not carried up yet, or anything
/// the engine never uploads. That is why ignored names count here, and why the
/// walk is its own rather than [`walk_local_tree`], which skips symlinks.
pub(crate) fn mirror_subtree_unsynced(
root: &Path,
rel: &str,
baseline: &HashMap<String, StoredSyncEntry>,
) -> Option<String> {
let mut seen = HashSet::new();
if let Some(reason) = unsynced_local(root, rel, baseline, &mut seen) {
return Some(reason);
}
let prefix = format!("{rel}/");
baseline
.keys()
.filter(|r| *r == rel || r.starts_with(&prefix))
.find(|r| !seen.contains(r.as_str()))
.map(|r| format!("{r} was deleted here but not on Proton Drive yet"))
}
fn unsynced_local(
root: &Path,
rel: &str,
baseline: &HashMap<String, StoredSyncEntry>,
seen: &mut HashSet<String>,
) -> Option<String> {
let path = root.join(rel_to_path(rel));
let meta = match std::fs::symlink_metadata(&path) {
Ok(meta) => meta,
Err(e) => return Some(format!("cannot read {rel}: {e}")),
};
let Some(base) = baseline.get(rel) else {
return Some(format!("{rel} is not on Proton Drive yet"));
};
seen.insert(rel.to_string());
if meta.is_file() {
return (!LocalSig::from(&meta).same_content(&LocalSig::from(base)))
.then(|| format!("{rel} has changes that are not on Proton Drive yet"));
}
if !meta.is_dir() {
return Some(format!("{rel} is not a regular file or folder"));
}
let entries = match std::fs::read_dir(&path) {
Ok(entries) => entries,
Err(e) => return Some(format!("cannot read {rel}: {e}")),
};
for entry in entries {
let name = match entry {
Ok(entry) => entry.file_name(),
Err(e) => return Some(format!("cannot read {rel}: {e}")),
};
let Some(name) = name.to_str() else {
return Some(format!("{rel} holds a name that is not valid UTF-8"));
};
// A download the engine left half done; it is the engine's to clean up.
if name.contains(".pdfs-tmp-") {
continue;
}
if let Some(reason) = unsynced_local(root, &format!("{rel}/{name}"), baseline, seen) {
return Some(reason);
}
}
None
}
/// Publish a conflict copy without overwriting an earlier conflict bearing the
/// same timestamp. The original is removed only after the new copy is durable.
fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> {
@@ -2155,6 +2262,27 @@ mod tests {
assert_eq!(remote_sig(&e), None);
}
#[test]
fn a_file_deleted_before_its_upload_is_skipped_not_failed() {
let root = std::env::temp_dir().join(format!("pdfs-b108-{}", std::process::id()));
std::fs::create_dir_all(&root).unwrap();
let gone = root.join("gone.txt");
assert!(open_to_upload(&gone, "gone.txt").unwrap().is_none());
std::fs::write(&gone, b"x").unwrap();
assert!(open_to_upload(&gone, "gone.txt").unwrap().is_some());
// Other open failures still count as errors.
let err = open_to_upload(&gone.join("child"), "gone.txt/child");
assert!(err.unwrap_err().starts_with("open gone.txt/child:"));
let _ = std::fs::remove_dir_all(&root);
let mut o = Outcome::default();
o.record(&Applied::Vanished);
assert!(o.is_empty());
assert_eq!(o.errors, 0);
}
#[test]
fn outcome_summarises_only_what_it_moved() {
let mut o = Outcome::default();
@@ -2346,6 +2474,94 @@ mod tests {
);
}
/// A baseline row recording `path` as it is on disk right now.
fn synced_row(root: &Path, rel: &str) -> (String, StoredSyncEntry) {
let meta = std::fs::symlink_metadata(root.join(rel_to_path(rel))).unwrap();
let sig = LocalSig::from(&meta);
let entry = StoredSyncEntry {
rel_path: rel.to_string(),
remote_uid: Some(format!("vol~{rel}")),
local_mtime: sig.mtime,
local_mtime_ns: sig.mtime_ns,
local_size: sig.size,
remote_rev: None,
remote_hash: None,
};
(rel.to_string(), entry)
}
fn synced_subtree(name: &str) -> (PathBuf, HashMap<String, StoredSyncEntry>) {
let root = sync_test_dir(name);
std::fs::create_dir_all(root.join("dir/sub")).unwrap();
std::fs::write(root.join("dir/a.txt"), b"a").unwrap();
std::fs::write(root.join("dir/sub/b.txt"), b"b").unwrap();
let baseline = ["dir", "dir/sub", "dir/a.txt", "dir/sub/b.txt"]
.into_iter()
.map(|rel| synced_row(&root, rel))
.collect();
(root, baseline)
}
#[test]
fn a_fully_synced_mirror_subtree_may_leave() {
let (root, baseline) = synced_subtree("move-synced");
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert_eq!(reason, None);
}
#[test]
fn a_mirror_subtree_with_a_file_not_uploaded_may_not_leave() {
let (root, baseline) = synced_subtree("move-new-file");
std::fs::write(root.join("dir/sub/new.txt"), b"n").unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/sub/new.txt"));
}
#[test]
fn a_mirror_subtree_with_an_edit_not_uploaded_may_not_leave() {
let (root, baseline) = synced_subtree("move-edit");
std::fs::write(root.join("dir/a.txt"), b"longer now").unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/a.txt"));
}
#[test]
fn a_mirror_subtree_with_a_deletion_not_carried_up_may_not_leave() {
let (root, baseline) = synced_subtree("move-deleted");
std::fs::remove_file(root.join("dir/sub/b.txt")).unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/sub/b.txt"));
}
#[test]
fn a_mirror_subtree_holding_a_symlink_may_not_leave() {
let (root, baseline) = synced_subtree("move-symlink");
std::os::unix::fs::symlink("a.txt", root.join("dir/link")).unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/link"));
}
#[test]
fn a_name_prefixed_sibling_does_not_hold_a_mirror_subtree_back() {
let (root, mut baseline) = synced_subtree("move-sibling");
// Deleted here but not on Drive, yet outside the subtree being moved.
baseline.insert(
"dirt".to_string(),
StoredSyncEntry {
rel_path: "dirt".to_string(),
..baseline["dir/a.txt"].clone()
},
);
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert_eq!(reason, None);
}
/// The same rule one level up: if the sync root itself cannot be read, the
/// pass sees nothing, and "nothing" must never mean "delete everything".
#[test]
+1
View File
@@ -108,6 +108,7 @@ impl Outcome {
Applied::Uploaded => self.uploaded += 1,
Applied::Downloaded => self.downloaded += 1,
Applied::Conflict => self.conflicts += 1,
Applied::Vanished => {}
}
}
+4 -7
View File
@@ -73,7 +73,7 @@ async fn run_uploads(
.upload_thumbnails_async(t.path.clone(), t.name.clone())
.await;
match core
.client
.drive
.upload_file_from(
&t.parent_uid,
&t.name,
@@ -238,14 +238,11 @@ impl Core {
.map_err(|error| self.errno_error(error, "upload destination"))?;
let new_uid = self
.rt
.block_on(
self.client
.create_folder(parent_uid, name, Some(now_secs())),
)
.block_on(self.drive.create_folder(parent_uid, name, Some(now_secs())))
.map_err(|e| CoreError::from_api(&e, &format!("create folder {name}")))?;
let node = self
.fetch_node(&new_uid)
.map_err(|e| self.errno_error(e, "fetch node"))?;
.fetch_minted_node(&new_uid, name)
.ok_or_else(|| minted_but_unlisted(name))?;
let mut st = self.state();
let ino = st.intern(pino, node);
if let Some(kids) = st.children.get_mut(&pino)
+10 -1
View File
@@ -668,6 +668,15 @@ fn build_window(app: &adw::Application) {
sync_sidebar(&ui_nav);
match st.visible_child_name().as_deref() {
Some("browser") => load_browser(&ui_nav),
// The Albums and Places listings are re-read in place. Loading the
// timeline instead would swap it in under the Albums or Places toggle.
Some("gallery") if !ui_nav.gallery.in_collection() => {
match ui_nav.gallery.view_switch.active() {
VIEW_ALBUMS => load_albums(&ui_nav),
VIEW_PLACES => load_places(&ui_nav),
_ => load_gallery(&ui_nav, false),
}
}
Some("gallery") => load_gallery(&ui_nav, false),
// Every visit reads the page again. The rows already on screen stay
// up while it does, so arriving costs no "Loading…" flash.
@@ -1199,7 +1208,7 @@ fn install_window_actions(ui: &Rc<Ui>, window: &adw::ApplicationWindow) {
.application_icon("io.narl.proton-drive-linux")
.version(pdfs_core::config::APP_VERSION)
.developer_name("Nils Pukropp")
.website("https://github.com/narrrl/proton-drive-linux")
.website("https://proton-drive.narl.io")
.issue_url("https://github.com/narrrl/proton-drive-linux/issues")
.copyright("© 2026 Nils Pukropp (contact@narl.io)")
.license_type(gtk4::License::MitX11)
+244 -39
View File
@@ -1920,6 +1920,10 @@ pub(crate) fn prompt_rename(ui: &Rc<Ui>, entry: &DirEntry) {
/// Pick a destination folder by browsing Drive's folders, and move `entries`
/// there through the daemon.
///
/// Besides My files the picker offers every synced folder. A move into one is
/// still made on Proton Drive, so nothing is downloaded or uploaded again; it
/// is sent with absolute paths, which is how the daemon tells locations apart.
pub(crate) fn prompt_move(ui: &Rc<Ui>, entries: Vec<DirEntry>) {
if entries.is_empty() {
return;
@@ -1967,6 +1971,7 @@ pub(crate) fn prompt_move(ui: &Rc<Ui>, entries: Vec<DirEntry>) {
dialog.set_default_response(Some("move"));
dialog.set_close_response("cancel");
let mountpoint = ui.dirs.resolved_mountpoint(&ui.dirs.load_config());
let picker = Rc::new(MovePicker {
ui: ui.clone(),
dialog: dialog.clone(),
@@ -1974,28 +1979,93 @@ pub(crate) fn prompt_move(ui: &Rc<Ui>, entries: Vec<DirEntry>) {
list,
up: up.clone(),
sources: sources.clone(),
folder: RefCell::new(String::new()),
folder: RefCell::new(MoveTarget::Locations),
generation: Cell::new(0),
});
let picker_up = picker.clone();
up.connect_clicked(move |_| {
let folder = picker_up.folder.borrow().clone();
let parent = folder.rfind('/').map(|i| &folder[..i]).unwrap_or_default();
picker_up.open(parent.to_string());
let parent = picker_up.folder.borrow().parent();
picker_up.open(parent);
});
// Start where the browser is: moving is most often one level up or down.
picker.open(ui.browser.path.borrow().clone());
picker.open(MoveTarget::Drive(ui.browser.path.borrow().clone()));
let ui = ui.clone();
let picker_done = picker.clone();
dialog.connect_response(None, move |_, resp| {
if resp == "move" {
run_bulk_move(&ui, sources.clone(), picker_done.folder.borrow().clone());
if resp != "move" {
return;
}
match picker_done.folder.borrow().clone() {
MoveTarget::Locations => {}
MoveTarget::Drive(folder) => run_bulk_move(&ui, sources.clone(), folder),
MoveTarget::Synced { path, .. } => {
let absolute = sources
.iter()
.map(|rel| mountpoint.join(rel).to_string_lossy().into_owned())
.collect();
run_bulk_move(&ui, absolute, path);
}
}
});
dialog.present(ui_window(&picker.ui).as_ref());
}
/// Where the Move dialog is looking, and the destination if the user confirms.
#[derive(Clone, Debug, PartialEq)]
pub(crate) enum MoveTarget {
/// The list of places to move to: My files and each synced folder.
Locations,
/// A folder in My files, mountpoint-relative; empty for My files itself.
Drive(String),
/// A folder inside a synced folder, as an absolute local path. `root` is
/// the synced folder. A mirror folder is a plain directory, read from disk.
Synced {
root: String,
path: String,
mirror: bool,
},
}
impl MoveTarget {
/// One level up. The top of a location leads back to the list of them.
pub(crate) fn parent(&self) -> MoveTarget {
match self {
MoveTarget::Locations => MoveTarget::Locations,
MoveTarget::Drive(folder) if folder.is_empty() => MoveTarget::Locations,
MoveTarget::Drive(folder) => MoveTarget::Drive(
folder
.rfind('/')
.map(|i| folder[..i].to_string())
.unwrap_or_default(),
),
MoveTarget::Synced { root, path, .. } if path == root => MoveTarget::Locations,
MoveTarget::Synced { root, path, mirror } => MoveTarget::Synced {
root: root.clone(),
path: Path::new(path)
.parent()
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|| root.clone()),
mirror: *mirror,
},
}
}
/// `name` inside this folder, in the same form.
fn child(&self, name: &str) -> MoveTarget {
match self {
MoveTarget::Locations => MoveTarget::Locations,
MoveTarget::Drive(folder) if folder.is_empty() => MoveTarget::Drive(name.to_string()),
MoveTarget::Drive(folder) => MoveTarget::Drive(format!("{folder}/{name}")),
MoveTarget::Synced { root, path, mirror } => MoveTarget::Synced {
root: root.clone(),
path: Path::new(path).join(name).to_string_lossy().into_owned(),
mirror: *mirror,
},
}
}
}
/// The Move dialog's folder browser.
struct MovePicker {
ui: Rc<Ui>,
@@ -2006,71 +2076,81 @@ struct MovePicker {
/// What is being moved, mountpoint-relative.
sources: Vec<String>,
/// The folder on show, and the destination if the user confirms now.
folder: RefCell<String>,
folder: RefCell<MoveTarget>,
/// Drops a listing that arrives after the user went elsewhere.
generation: Cell<u64>,
}
impl MovePicker {
fn open(self: &Rc<Self>, folder: String) {
fn open(self: &Rc<Self>, folder: MoveTarget) {
let generation = self.generation.get().wrapping_add(1);
self.generation.set(generation);
self.location.set_label(if folder.is_empty() {
"Proton Drive"
} else {
&folder
});
self.location.set_tooltip_text(Some(&folder));
self.up.set_sensitive(!folder.is_empty());
self.dialog
.set_response_enabled("move", move_target_allowed(&self.sources, &folder));
let (label, tooltip) = match &folder {
MoveTarget::Locations => (gettext("Locations"), String::new()),
MoveTarget::Drive(rel) if rel.is_empty() => (gettext("My files"), String::new()),
MoveTarget::Drive(rel) => (rel.clone(), rel.clone()),
MoveTarget::Synced { path, .. } => (path.clone(), path.clone()),
};
self.location.set_label(&label);
self.location.set_tooltip_text(Some(&tooltip));
self.up
.set_sensitive(!matches!(folder, MoveTarget::Locations));
let allowed = match &folder {
MoveTarget::Locations => false,
MoveTarget::Drive(rel) => move_target_allowed(&self.sources, rel),
// What is moved is in My files, so it cannot be in a synced folder.
MoveTarget::Synced { .. } => true,
};
self.dialog.set_response_enabled("move", allowed);
*self.folder.borrow_mut() = folder.clone();
while let Some(row) = self.list.first_child() {
self.list.remove(&row);
}
self.list.append(&picker_note(&gettext("Loading…")));
let rx = spawn_request(
self.ui.dirs.control_socket(),
Request::ListDir {
path: folder.clone(),
},
);
let picker = self.clone();
glib::spawn_future_local(async move {
let result = rx.recv().await;
let listed = match &folder {
MoveTarget::Locations => picker.locations().await,
MoveTarget::Drive(rel) => picker.drive_folders(&folder, rel.clone()).await,
MoveTarget::Synced {
path, mirror: true, ..
} => local_folders(&folder, path.clone()).await,
MoveTarget::Synced { path, .. } => {
picker.drive_folders(&folder, path.clone()).await
}
};
if picker.generation.get() != generation {
return;
}
while let Some(row) = picker.list.first_child() {
picker.list.remove(&row);
}
let Ok(Ok(Response::Entries { entries })) = result else {
let Some(rows) = listed else {
picker
.list
.append(&picker_note(&gettext("Couldn't read this folder.")));
return;
};
let mut folders: Vec<DirEntry> = entries.into_iter().filter(|e| e.is_dir).collect();
folders.sort_by_key(|e| e.name.to_lowercase());
let mut shown = 0;
for entry in folders {
let path = if folder.is_empty() {
entry.name.clone()
} else {
format!("{folder}/{}", entry.name)
};
for (title, subtitle, target) in rows {
// A folder cannot go inside itself, so do not offer to open it.
if picker.sources.contains(&path) {
if let MoveTarget::Drive(path) = &target
&& picker.sources.contains(path)
{
continue;
}
let row = adw::ActionRow::builder()
.title(glib::markup_escape_text(&entry.name).as_str())
.title(glib::markup_escape_text(&title).as_str())
.activatable(true)
.build();
if let Some(subtitle) = subtitle {
row.set_subtitle(glib::markup_escape_text(&subtitle).as_str());
}
row.add_prefix(&gtk4::Image::from_icon_name("folder-symbolic"));
row.add_suffix(&gtk4::Image::from_icon_name("go-next-symbolic"));
let open = picker.clone();
row.connect_activated(move |_| open.open(path.clone()));
row.connect_activated(move |_| open.open(target.clone()));
picker.list.append(&row);
shown += 1;
}
@@ -2081,6 +2161,92 @@ impl MovePicker {
}
});
}
/// My files and every synced folder, as rows of the top level.
async fn locations(&self) -> Option<Vec<PickerRow>> {
let mut rows = vec![(gettext("My files"), None, MoveTarget::Drive(String::new()))];
let rx = spawn_request(self.ui.dirs.control_socket(), Request::ListSyncFolders);
let Ok(Ok(Response::SyncFolders { mut items })) = rx.recv().await else {
// My files alone is still somewhere to move to.
return Some(rows);
};
items.sort_by_key(|f| f.local_path.to_lowercase());
for folder in items {
let name = Path::new(&folder.local_path)
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_else(|| folder.local_path.clone());
rows.push((
name,
Some(folder.local_path.clone()),
MoveTarget::Synced {
root: folder.local_path.clone(),
path: folder.local_path,
mirror: folder.mode == "mirror",
},
));
}
Some(rows)
}
/// The folders in a mounted folder: `folder` is mountpoint-relative for My
/// files and absolute for an on-demand synced folder.
async fn drive_folders(&self, here: &MoveTarget, folder: String) -> Option<Vec<PickerRow>> {
let rx = spawn_request(
self.ui.dirs.control_socket(),
Request::ListDir {
path: folder.clone(),
},
);
let Ok(Ok(Response::Entries { entries })) = rx.recv().await else {
return None;
};
let mut names: Vec<String> = entries
.into_iter()
.filter(|e| e.is_dir)
.map(|e| e.name)
.collect();
names.sort_by_key(|name| name.to_lowercase());
Some(
names
.into_iter()
.map(|name| {
let target = here.child(&name);
(name, None, target)
})
.collect(),
)
}
}
/// One row of the picker: its title, an optional subtitle, and where it leads.
type PickerRow = (String, Option<String>, MoveTarget);
/// The folders in a mirror folder's directory `path`, read from disk off the
/// main thread. A mirror folder has no mount for `ListDir` to read.
async fn local_folders(here: &MoveTarget, path: String) -> Option<Vec<PickerRow>> {
let names = gio::spawn_blocking(move || {
let mut names: Vec<String> = std::fs::read_dir(&path)
.ok()?
.flatten()
.filter(|e| e.file_type().is_ok_and(|t| t.is_dir()))
.filter_map(|e| e.file_name().into_string().ok())
.filter(|name| !name.contains(".pdfs-tmp-"))
.collect();
names.sort_by_key(|name| name.to_lowercase());
Some(names)
})
.await
.ok()??;
Some(
names
.into_iter()
.map(|name| {
let target = here.child(&name);
(name, None, target)
})
.collect(),
)
}
fn picker_note(text: &str) -> gtk4::ListBoxRow {
@@ -2786,8 +2952,8 @@ fn listing_summary(files: usize, folders: usize, file_bytes: u64) -> String {
#[cfg(test)]
mod tests {
use super::{
hit_location, listing_summary, move_target_allowed, show_thumbnail_build_progress,
sort_entries,
MoveTarget, hit_location, listing_summary, move_target_allowed,
show_thumbnail_build_progress, sort_entries,
};
use pdfs_core::config::{FileSort, FilesView};
use pdfs_core::control::{DirEntry, ThumbnailBuildStatus};
@@ -2880,4 +3046,43 @@ mod tests {
status.running = false;
assert!(!show_thumbnail_build_progress(&status));
}
#[test]
fn move_picker_climbs_from_a_location_top_to_the_list_of_locations() {
assert_eq!(
MoveTarget::Drive(String::new()).parent(),
MoveTarget::Locations
);
let root = MoveTarget::Synced {
root: "/home/u/Docs".into(),
path: "/home/u/Docs".into(),
mirror: true,
};
assert_eq!(root.parent(), MoveTarget::Locations);
}
#[test]
fn move_picker_climbs_one_folder_at_a_time_inside_a_location() {
assert_eq!(
MoveTarget::Drive("a/b".into()).parent(),
MoveTarget::Drive("a".into())
);
assert_eq!(
MoveTarget::Drive("a".into()).parent(),
MoveTarget::Drive(String::new())
);
let inner = MoveTarget::Synced {
root: "/home/u/Docs".into(),
path: "/home/u/Docs/x/y".into(),
mirror: false,
};
assert_eq!(
inner.parent(),
MoveTarget::Synced {
root: "/home/u/Docs".into(),
path: "/home/u/Docs/x".into(),
mirror: false,
}
);
}
}
+9 -7
View File
@@ -262,20 +262,22 @@ pub(crate) fn build_locations_page(history: &gtk4::Widget) -> (gtk4::Widget, Loc
(frame.upcast(), widgets)
}
/// `child` in a clamp that scrolls vertically only, with page margins.
/// `child` across the page's full width, scrolling vertically only, with page
/// margins.
fn scrolled_column(child: &impl IsA<gtk4::Widget>) -> gtk4::ScrolledWindow {
let clamp = adw::Clamp::builder().child(child).build();
clamp.set_margin_top(18);
clamp.set_margin_bottom(18);
clamp.set_margin_start(18);
clamp.set_margin_end(18);
let column = gtk4::Box::new(gtk4::Orientation::Vertical, 0);
column.append(child);
column.set_margin_top(18);
column.set_margin_bottom(18);
column.set_margin_start(18);
column.set_margin_end(18);
// Never scroll sideways: a location's subtitle holds a full path, and letting
// the row grow to fit one pushes the controls at its end off screen.
// Constrained, the path ellipsizes and the controls stay reachable.
gtk4::ScrolledWindow::builder()
.vexpand(true)
.hscrollbar_policy(gtk4::PolicyType::Never)
.child(&clamp)
.child(&column)
.build()
}
+5 -1
View File
@@ -1069,7 +1069,11 @@ pub(crate) fn repaint_transfers(ui: &Rc<Ui>, items: &[TransferItem], jobs: &[Job
let row_box = gtk4::Box::new(gtk4::Orientation::Vertical, 4);
row_box.set_margin_top(8);
row_box.set_margin_bottom(8);
let label = gtk4::Label::builder().halign(gtk4::Align::Start).build();
// Ellipsized so a long file name cannot widen the page.
let label = gtk4::Label::builder()
.halign(gtk4::Align::Start)
.ellipsize(gtk4::pango::EllipsizeMode::Middle)
.build();
label.add_css_class("dim-label");
let bar = gtk4::ProgressBar::new();
row_box.append(&label);
+6
View File
@@ -535,6 +535,12 @@ async fn mounted_target_within(mountpoint: PathBuf, hit: SearchHit) -> Option<Pa
}
impl Ui {
/// `--preload`: create the surface, renderer and style state now, without
/// mapping the window, so the first real summon only has to present it.
pub(crate) fn preload(&self) {
WidgetExt::realize(&self.window);
}
/// Reset and present the resident prompt in response to GApplication
/// activation. A second invocation while a file is materialising merely
/// raises the progress window; it must not make the in-flight open mutable.
+54 -2
View File
@@ -268,6 +268,9 @@ Options:
every keystroke — results appear as you type, unlike
--dmenu. Also settable as \"prompt\": { \"mode\": \"fzf\" }.
--gtk Force the built-in window, overriding that setting.
--preload Start the built-in window hidden and stay resident, so
the first summon opens at once. For session autostart;
does nothing if the prompt is already running.
--menu <COMMAND> Launcher command line for --dmenu, e.g.
--menu 'fuzzel --dmenu --width 60'. Overrides
\"prompt\": { \"menu\": [...] }.
@@ -293,6 +296,8 @@ struct Args {
feed: Option<Option<String>>,
/// `--inner`: we are the process a spawned terminal is running.
inner: bool,
/// `--preload`: become the resident GTK instance without showing it.
preload: bool,
}
fn parse_args(argv: impl Iterator<Item = String>) -> Result<Args, String> {
@@ -302,6 +307,7 @@ fn parse_args(argv: impl Iterator<Item = String>) -> Result<Args, String> {
query: None,
feed: None,
inner: false,
preload: false,
};
let mut argv = argv.peekable();
while let Some(arg) = argv.next() {
@@ -310,6 +316,7 @@ fn parse_args(argv: impl Iterator<Item = String>) -> Result<Args, String> {
"--fzf" => args.mode = Some(PromptMode::Fzf),
"--gtk" => args.mode = Some(PromptMode::Gtk),
"--inner" => args.inner = true,
"--preload" => args.preload = true,
// fzf passes an empty argument for an empty input, so a missing
// value here is that case rather than a user error.
"--feed" => args.feed = Some(argv.next().filter(|text| !text.trim().is_empty())),
@@ -337,6 +344,13 @@ fn parse_args(argv: impl Iterator<Item = String>) -> Result<Args, String> {
other => return Err(format!("unrecognised argument: {other}")),
}
}
// Only the GTK window has a resident process worth warming up.
if args.preload {
if matches!(args.mode, Some(PromptMode::Dmenu | PromptMode::Fzf)) || args.query.is_some() {
return Err("--preload only applies to the built-in window".to_string());
}
args.mode = Some(PromptMode::Gtk);
}
Ok(args)
}
@@ -398,9 +412,31 @@ fn main() -> glib::ExitCode {
// GtkApplication normally remains active while it owns a window, including
// when that window is hidden. Hold it explicitly as well: residency is a
// product requirement here, not an incidental window-lifetime side effect.
let _resident = app.hold();
app.connect_startup(|_| theme::load_resources("/de/nils/protondrivelinux/prompt.css"));
// Only the primary instance runs startup; a hold taken before registration
// would also keep every remote invocation alive after it has forwarded
// its activation.
let resident = Rc::new(RefCell::new(None));
app.connect_startup(move |app| {
*resident.borrow_mut() = Some(app.hold());
theme::load_resources("/de/nils/protondrivelinux/prompt.css");
});
if args.preload {
if let Err(e) = app.register(gtk4::gio::Cancellable::NONE) {
eprintln!("pdfs-prompt: {e}");
return glib::ExitCode::FAILURE;
}
// Another instance is already resident; activating it would show it.
// Dropping a registered application that never ran makes GIO warn
// about the D-Bus registration, so leave it to process exit.
if app.is_remote() {
std::mem::forget(app);
return glib::ExitCode::SUCCESS;
}
}
let prompt: Rc<RefCell<Option<Rc<hud::Ui>>>> = Rc::new(RefCell::new(None));
// The first activation of a preloading instance is its own startup, not a
// summon: build the window but leave it hidden.
let quiet = std::cell::Cell::new(args.preload);
app.connect_activate(move |app| {
let ui = if let Some(ui) = prompt.borrow().clone() {
ui
@@ -411,6 +447,10 @@ fn main() -> glib::ExitCode {
*prompt.borrow_mut() = Some(ui.clone());
ui
};
if quiet.replace(false) {
ui.preload();
return;
}
ui.activate();
});
// GTK must not try to parse our own flags; they were consumed above.
@@ -582,4 +622,16 @@ mod tests {
let args = parse(&["--dmenu", "--query", "tax return 2024"]).unwrap();
assert_eq!(args.query.as_deref(), Some("tax return 2024"));
}
#[test]
fn preload_forces_the_gtk_window_and_rejects_the_launchers() {
let args = parse(&["--preload"]).unwrap();
assert!(args.preload);
assert_eq!(args.mode, Some(PromptMode::Gtk));
assert!(parse(&["--preload", "--dmenu"]).is_err());
assert!(parse(&["--fzf", "--preload"]).is_err());
assert!(parse(&["--preload", "--menu", "fuzzel --dmenu"]).is_err());
assert!(parse(&["--preload", "--query", "tax"]).is_err());
}
}
+13
View File
@@ -539,6 +539,19 @@ A restore is expanded before it is sent (`expand_restore`, `pdfs-fuse/src/lib.rs
The trash view's restore and permanent-delete use the SDK's **streaming** variants (`restore_nodes_streaming` / `delete_nodes_streaming`) and apply local state per node as each batch lands rather than after the last one. For a permanent delete — which is irreversible — that means a daemon interrupted mid-batch has forgotten exactly the nodes the server destroyed, no more and no fewer. Both report per-node failures and count only what actually succeeded; only a batch where nothing succeeded is an error.
### Moves Between Locations
**Files**: [`relocate.rs`](../crates/pdfs-fuse/src/relocate.rs), [`control.rs`](../crates/pdfs-fuse/src/control.rs), [`browser.rs`](../crates/pdfs-gui/src/app/pages/browser.rs)
The primary mount and every on-demand folder are separate FUSE sessions, and a mirror folder has no session at all. The kernel answers `rename(2)` between two mounts with `EXDEV` before the daemon hears of it, so `mv` copies and deletes. On Proton Drive the same move is one `move_node`, because My files and this device's folders are on the main volume.
`Request::Move` with two absolute paths is therefore resolved by `Core::move_between` against every location: a live mount through `rooted_at`, otherwise the mirror folder with the longest `local_path` prefix, whose uids come from the `sync_entry` baseline. Two paths in one inode space still go to `move_to`. Relative paths keep their old meaning.
- **Nothing that exists only on this disk may be lost.** A mounted source is refused while it or anything under it has a queued op or a file open for writing. A mirror source is refused unless every file under it matches its baseline and every baseline row still has its file (`mirror_subtree_unsynced`); ignored names and symlinks count as unsynced, because the local copy is removed after the move.
- **Mirror to mirror renames along.** The local copy is renamed first, with `RENAME_NOREPLACE` so nothing that appeared at the destination meanwhile is overwritten, and its baseline rows are moved with `sync_entries_move`, so neither folder's next pass has anything to do. If `move_node` fails, the rename is undone. `EXDEV` between two filesystems falls back to the check above, then drops the source copy.
- **A dropped mirror copy is checked twice.** The check runs again just before the source copy is deleted. A file written into it during the move keeps the whole copy; its baseline is gone, so the next pass uploads it as new.
- **Sync passes are held off.** Every mirror folder involved is locked through `sync_lock`, by ascending id, with a five-second bound. A folder that is still busy is reported rather than waited on.
- **The local side is levelled afterwards.** A mounted source forgets the node in every inode space and notifies the kernel. A mounted destination has its listing cleared. A mirror destination that received nothing locally gets a reconcile, which downloads the node. That download first asks `ContentCache::copy_cached_to` for the file: a whole cached blob, or every block of it, is copied into place instead of fetched, so a move out of an on-demand folder costs no traffic for what was already opened there.
### Open-for-Write Deferral for Mirror Sync
**File**: [`sync.rs`](../crates/pdfs-fuse/src/sync.rs)
+1090 -1
View File
File diff suppressed because it is too large. Load diff
+227
View File
@@ -11,6 +11,224 @@ scratch (user data in `staging/` and `recovery/` is never touched by this).
## [Unreleased]
## [2.10.0] — 2026-10-03
Schema: **36**. Every node row gets a local id that never changes; existing rows keep their
row number as that id. Going back to 2.9 means rebuilding the cache. Requires `proton-drive-rs`
**0.7.2**.
### Added
- **Migration tests on real databases.** A database 2.8.2 wrote, with queued changes due,
parked, failing, waiting for access and in flight, migrates with every change intact.
`PDFS_MIGRATE_DB=<path> cargo test -p pdfs-core a_copy_of_a_real_database -- --ignored` runs
the same check on a read-only copy of any database.
### Changed
- **A file keeps its inode number across daemon restarts.** Inodes are now the node's local id
instead of a counter that started over on every run.
- **Every mount serves from one tree.** An on-demand sync folder used to keep its own copy of the
nodes it showed, which background work had to visit one by one and could miss (B74). It now
shares the tree with My files; only the root it starts at is its own.
### Fixed
- **A file renamed or moved while its first upload is on the wire keeps its new name and folder.**
It landed under the old name and the rename was lost (B136).
## [2.9.0] — 2026-10-03
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.2**.
### Added
- **Seeded simulation runs.** `cargo test -p pdfs-fuse --lib sim:: -- --ignored
--test-threads=1` mounts one or more real FUSE daemons against an in-memory Drive that lags,
drops, reorders and echoes like the real one. Each seed draws syscalls, link flaps and restarts,
checks every answer against a POSIX model, and at the end checks that nothing was lost and the
clients converged. CI runs them on pushes to main. They found B122 to B135.
- **More acceptance cases.** Regressions for B48, B80, B86, B88, B94, B95, B98, B102, B111, B113
and B114 to B121, and a check that `pdfs` output into a closed pipe stays quiet. The console
report has one line per case and a summary table, and `--journal-check` also fails on a
conflict copy, a FUSE job held for long, or an authority re-read.
### Fixed
- **A new file moved or renamed before its first upload no longer lands as a conflict copy**
next to its own empty original (B113).
- **`pdfs rm`, `pdfs rename` and `pdfs mkdir` take paths in on-demand folders.** They answered
"is not under the mountpoint" for a path `pdfs move` took. `rm` and `rename` refuse a path that
names a mount's root (B88).
- **`pdfs ls | head` no longer panics** with "Broken pipe". A closed stdout exits 0; any other
write error exits 1.
- **A file removed, renamed or moved with `pdfs` stops resolving under its old name at once**,
instead of for up to 30 seconds (B114).
- **`pdfs ls` and the app's file browser show a file's real size**, not its encrypted one
(B115).
- **Trashing a folder also drops the creates queued in its subfolders.** One could stay queued
with its bytes, or land in the root (B116).
- **An upload queued for a file `pdfs move` moved keeps its path** in `pdfs sync queue` and is
no longer put off once after the resume (B117).
- **Creating a file or folder through the mount no longer fails with `ENOENT`** when Drive is
slow to list what it made. The mount asks again, and queues the create if it still cannot see
it (B118).
- **A file just written can be read without the network.** Its uploaded bytes stay cached, and
`pdfs status` counts the upload as pending until they are (B119).
- **A file queued inside a folder that just landed is no longer put off** for five seconds
(B120).
- **A move right after a rename lands.** It failed with `Input/output error` through the mount
and with "out of date" from `pdfs move`, and a rename across folders took up to 10 seconds
(B111, B121).
- **A rename across folders no longer fails with `Input/output error`** when the source folder
holds the new name (B122).
- **A second write open on a file just written no longer fails** with `Input/output error`
(B123).
- **After a replacing rename fails, the file it was replacing stays in the mount** (B124).
- **A write to a file whose create could not be read back no longer lands as a conflict copy**
(B131).
- **A file just moved is found by its new path.** A lookup right after the rename could miss it
(B135).
### Notes
- The simulation runs found bugs that are still open: B125 to B130 and B132 to B134 in
`docs/BUGS.md`. The worst are B127 and B128 (a file created offline can land twice, or be lost
when created over one deleted offline), B129 (a file deleted while its create is on the wire
stays on Drive) and B130 (a read that a remote change invalidates can hang the daemon).
## [2.8.2] — 2026-10-01
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.2**.
### Fixed
- **Folders shared with you no longer go missing when one person shared several.** When
several accepted shares came from the same owner's volume, only the first appeared under
Shared with me, and the rest were skipped with `missing key`. `proton-drive-rs` 0.7.2 resolves
each shared root with its own key (B112, #25).
## [2.8.1] — 2026-09-30
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.1**.
### Fixed
- **The service no longer starts before the desktop session.** `proton-drive.service` was
wanted by `default.target`, which is reached before the desktop hands `DISPLAY` /
`WAYLAND_DISPLAY` to the user's systemd. The first keyring call then started the Secret
Service without a display; on KDE `ksecretd` crashed at every login and the first start of the
service failed. The unit is now wanted by `graphical-session.target`. Existing installs switch
over on the next sign-in or "Start on login" toggle, or with
`systemctl --user reenable proton-drive.service` (#23).
## [2.8.0] — 2026-09-30
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.1**.
### Added
- **`pdfs trash --json`** prints the trash listing as JSON.
- **The FUSE acceptance suite tests the signed-in account by default.**
`scripts/fuse-acceptance.sh` with no arguments creates test folders in My files and new
on-demand and mirror sync folders, runs every case in each, and removes all of it afterwards,
also on a failure or interrupt. A run that dies is cleaned up by the next one. New cases cover
block boundaries, unusual names, a deep tree, a wide directory, rename patterns, handle coherency,
throughput and B100. `--offline-only` runs the account-free suite as before.
### Changed
- **The mount keeps working through a network drop.** When Drive stops answering, the daemon
now marks itself offline within about 20 seconds instead of failing every call for up to two
minutes. While offline, creating, renaming and deleting files and folders is queued instead of
failing with `Input/output error`, folders list from the local cache, and a read that needs
data from Drive waits up to 30 seconds for the connection. The daemon checks every few seconds
for the connection to return, then uploads the queued changes. A file whose creation timed
out but reached Drive anyway is taken over rather than uploaded again as a conflict copy.
- **A dead connection is noticed sooner.** `proton-drive-rs` 0.7.1 sends TCP keepalives, so a
connection that died silently fails instead of waiting for its request timeout.
### Fixed
- **Truncating and then writing past the end no longer brings old bytes back.** The gap
between the new end and the write now reads as zeros, as it should, instead of the file's
earlier content. When that earlier content had not been uploaded yet, the file failed with
`Input/output error` and its upload retried forever (B101).
- **A write made while a new file is first uploading is no longer lost.** Drive could keep an
empty or older copy of a file written right after it was created, and closing a file held open
across its first upload could fail (B101).
- **Removing a folder also cancels the pending uploads of the files inside it.** Before, an
edit that had not reached Drive yet kept retrying against the trash forever and kept its
staged copy on disk. `pdfs rm` now also drops what was queued for what it removes (B102).
- **A program reading a file keeps its data when another program replaces the file.** Saving
over a file by renaming a new copy onto it (what most editors and `os.replace` do) made
reads through handles already open on the old file fail with `Input/output error` (B104).
- **Short-lived files no longer come back as conflict copies.** A file deleted right after it
was written, such as a SQLite journal, could be uploaded as a `recovered-…` conflict copy in
the Drive root (B105).
- **Editors can save over a file several times in a row.** The fix for B104 could leave the
upload of a replaced file queued, and the next save onto the same name failed with
`Input/output error` (B106).
- **Renaming a file back right after renaming it no longer fails.** Drive can briefly call the
second rename out of date, and the daemon now waits it out instead of returning an I/O error
(B111).
- **Reading a freshly listed file no longer hangs.** When a read had to fetch the file's real
size first, the daemon could wait on the read it was serving, and the program froze until
the daemon restarted (B110).
- **Databases and other files kept open while written no longer read back empty.** When the
folder was listed again while a file was still open, its size dropped to what Drive held,
often 0 bytes, and SQLite reported `database disk image is malformed` (B109).
- **Deleting files during a mirror pass no longer marks the folder as failed.** A file removed
after the pass planned its upload was counted as an error, so the folder showed `error` until
a later pass (B108).
- **A read no longer hangs when the connection stalls.** A read that gets no answer from Drive
within two minutes now fails with `Input/output error` instead of blocking the program for
as long as half an hour (B107).
- **The trash no longer freezes after a server call hangs.** Until the daemon restarted, the
trash kept showing the same entries, and a folder just deleted could not be found there to
delete permanently (B103).
- **Freshly listed files read correctly.** A file whose real size had not been fetched yet
reported its encrypted size, about 53 bytes too big, and reading it failed with
`Input/output error`. This happened most often right after copying a large folder. Reads and
write opens now fetch the real size first, and a big folder walk no longer skips size lookups
(B100).
## [2.7.0] — 2026-09-29
Schema: **35** (unchanged).
### Added
- **Moves between locations without transferring content.** `pdfs move` and the app's
**Move to…** now take a file or folder from My files into a synced folder, or between two
synced folders, on-demand or mirrored. The move happens on Proton Drive, so nothing is
downloaded or uploaded again. `mv` between two locations still copies, because the kernel
refuses a rename between two mounts. The app's Move dialog lists every location at its top
level. A move is refused, with the reason, when the source still has changes that are not on
Proton Drive, when the two locations are on different volumes, or when a synced folder stays
busy with a sync pass.
- **`pdfs move` takes several sources.** `pdfs move a b c DEST` moves all three and
reports each one that could not move.
- **Moving into a mirrored folder reuses cached content.** A file already cached by
an online-only folder is copied into place instead of downloaded again.
### Changed
- **New project homepage: [proton-drive.narl.io](https://proton-drive.narl.io).** The About
dialog's website link, the README, and the package metadata (Cargo, PKGBUILDs, `.deb`,
`.rpm`) point there. The source code link stays on GitHub.
## [2.6.2] — 2026-09-28
Schema: **35** (unchanged).
### Fixed
- **Photos remembers the Albums and Places views.** Leaving Photos from Albums or Places and
coming back showed the timeline while the Albums or Places toggle stayed selected. The page
now reopens on the selected view, and Places keeps its Grid or Map layout.
- **The Sync page keeps a steady width.** A long file name in Transfers no longer widens the
page. Overview and Folders now fill the width next to the sidebar.
## [2.6.1] — 2026-09-28
Schema: **35** (unchanged).
### Added
- **`pdfs-prompt --preload`.** Starts the launcher hidden, for session autostart such as
Hyprland's `exec-once`, so the first shortcut opens it as fast as later ones. It does nothing
when the launcher is already running.
### Fixed
- **Launcher invocations no longer pile up.** Each shortcut press after the first used to leave
a `pdfs-prompt` process behind after handing the request to the running launcher. It now exits.
## [2.6.0] — 2026-09-28
The search launcher (`pdfs-prompt`) is rebuilt around a preview pane, and it now looks like the
@@ -1226,6 +1444,15 @@ First stable release: FUSE files-on-demand mount, sync daemon under `proton-driv
- The outstanding FUSE defects tracked in `docs/BUGS.md`, plus a truncate defect, validated
by a new POSIX compliance suite for the filesystem.
[2.10.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.10.0
[2.9.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.9.0
[2.8.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.2
[2.8.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.1
[2.8.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.0
[2.7.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.7.0
[2.6.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.6.2
[2.6.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.6.1
[2.6.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.6.0
[2.5.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.5.0
[2.4.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.4.0
+2 -2
View File
@@ -53,7 +53,7 @@ talks to the running service (`proton-drive.service`); `login`, `logout`, `mount
| `pdfs search QUERY [--in FOLDER] [--limit N]` | Search names in the local index. |
| `pdfs mkdir PARENT NAME` | Create a folder. |
| `pdfs rename PATH NEW_NAME` | Rename a file or folder. |
| `pdfs move PATH NEW_PARENT` | Move into another folder. |
| `pdfs move PATH... NEW_PARENT` | Move one or more files or folders into another folder. The two may be in different locations (My files, an on-demand or a mirrored synced folder); the move happens on Proton Drive, without downloading or uploading anything. |
| `pdfs rm PATH` | Move to the Trash. |
| `pdfs upload SOURCES... [-t FOLDER]` | Upload local files and folders (recursively) in the background. Follow with `pdfs transfers`. |
| `pdfs refresh [TARGET] [--full]` | Forget a cached listing so the next read fetches it again. `TARGET` is a folder, `trash` or `photos`; `--full` with `photos` re-reads every photo's metadata. |
@@ -177,7 +177,7 @@ commands are for scripts and for cases where the mount is not available.
## Scripting with `--json`
The global `--json` flag makes query commands print JSON: `status`, `ls`, `pins`, `sync list`,
`devices list`, `locations`, `transfers`, `activity` and `cache inspect`. Commands that change
`devices list`, `locations`, `transfers`, `activity`, `trash` and `cache inspect`. Commands that change
something keep their human-readable output; check their exit code.
```bash
+8
View File
@@ -215,6 +215,14 @@ starts; to switch at once:
systemctl --user restart proton-drive.service
```
Coming from 2.8.0 or older, re-enable the service once. Those releases hooked it into
`default.target`, which starts it before the desktop session is ready; on KDE that crashes
`ksecretd` at every login. Signing in again or toggling "Start on login" does this as well:
```bash
systemctl --user reenable proton-drive.service
```
Database migrations run automatically and only move forward. A database written by a newer
release refuses to open in an older one, so downgrading means clearing the local database (see
[Troubleshooting](TROUBLESHOOTING.md#start-over-with-a-clean-local-state)).
+319 -19
View File
@@ -25,36 +25,151 @@ Proton:
scripts/fuse-acceptance.sh --offline-only
```
Kernel/FUSE behavior and remote convergence need a real mount. Live testing is
explicitly opt-in and always runs the account-free contract first. It runs
destructive POSIX operations only below a fresh `pdfs-acceptance-*` directory.
It refuses paths that are not FUSE mounts and removes its directory on success,
failure, or interruption.
Kernel/FUSE behavior and remote convergence need a real mount. With no
arguments, the script tests the account the running service is logged in to:
Use a dedicated test account without irreplaceable data:
```console
scripts/fuse-acceptance.sh # every case, every mount kind
scripts/fuse-acceptance.sh --quick # one on-demand/mirror pair instead of four
```
An account run does this, in order:
1. Runs the account-free contract as the reference.
2. Runs the contract in a fresh `pdfs-acceptance-<id>` folder in My files and
diffs it against the reference. After the last case it reads back a file
the run wrote and records its digest. A read that fails there, as one did
with the link down (B119), is a failed result named "a file of this run
reads back at the end", and the cleanup still runs.
3. Registers two new sync folders under `~/.cache/pdfs-acceptance/run-<id>/`
and runs the contract and the move cases in every on-demand/mirror pairing.
4. Removes everything it created, locally and remotely, and reports a
`cleanup` result. The run fails if anything is left behind.
Cleanup identifies remote folders by uid, never by name alone, so nothing of
yours is touched. It removes each sync folder with `sync rm --delete-remote`,
trashes the My files folder, and then deletes both permanently from the trash.
It only counts a folder as gone once a fresh trash listing no longer shows it.
It then reads `pdfs sync queue` and counts any queued op that still names one
of its folders as left behind too, since such an op keeps its staged bytes and
retries forever (B102).
Ops that were queued before a target starts are not the run's. The run prints
one `NOTE` line naming them, and the queue waits ignore them. They are told
apart by id and queue time together, since SQLite can give a new op the id of
one that has drained.
A write is queued when the kernel releases the file, and `close(2)` returns
without waiting for that. A case that reads `pdfs sync queue` right after
writing therefore waits for the ops it expects (`Daemon.queued`) instead of
reading once. "regression B102" saw one revision of two before it did.
Cleanup also runs on a failure, Ctrl-C, `SIGTERM` or `SIGHUP`. While it runs,
two more interrupts are ignored; a third abandons it.
Each run records what it is about to create in
`~/.cache/pdfs-acceptance/run-<id>/manifest.json` before it creates it, and
holds a lock on that directory. If a run dies without cleaning up (`kill -9`, a
hard timeout, a power cut), the next run finds the unlocked manifest and
finishes that cleanup first. It then removes stray `pdfs-acceptance-<32 hex>`
folders that no live run claims: in My files (older than
`PDFS_ACCEPTANCE_REAP_AGE`, default one hour), under the acceptance home, and
in the trash. A run does not start while that reap leaves anything behind.
The run creates and destroys real remote folders, so a dedicated test account
without irreplaceable data is still the safest target.
`--live PATH...` tests mounts you set up yourself and leaves the sync folder
configuration alone. It runs destructive POSIX operations only below a fresh
`pdfs-acceptance-*` directory, refuses paths that are not FUSE mounts, and
removes its directory on success, failure, or interruption:
```console
scripts/fuse-acceptance.sh --live /mnt/on-demand-testmount
```
### Reading the output
A run opens with what it is about to test: the mode, the `pdfs` binary and its
version, Python and the kernel, the per-case limit, the options and the start
time. Each target then names its root, the storage behind it, the daemon it
talks to and how many cases it runs, followed by one line per case:
```
==> live FUSE /mnt/testmount
-> root /mnt/testmount/pdfs-acceptance-…
-> storage FUSE mount /mnt/testmount (fuse.protondrive)
-> daemon pdfs 2.8.3, user@proton.me, online, 0 queued ops
-> cases 45 of 45
[ 1/45] creation and open flags .................................. ok 0.41s
…
[21/45] throughput floors ........................................ ok 48.20s
write 31 MiB/s, read 54 MiB/s, 4 KiB writes 12.0 MiB/s, metadata 9 ops/s
…
[43/45] regression B115: pdfs ls reports a file's real size ...... ok 6.12s
-> reference matches the ordinary filesystem in 412 compared observations
-> result 44 passed, 1 skipped in 11m02s
```
A case ends as `ok`, `FAIL`, `TIMEOUT`, `skip` when it does not apply to the
target (no daemon, no share at that role), or `known` when an `open B<n>` case
reproduced a bug that is still open (see below). The dimmed lines under a case
are its details: the reason for a skip, measured rates, a hash. A failed
`regression B<n>` case also points to its entry in `docs/BUGS.md`. When a case
prints something itself, such as a service restart, the case line is printed
again once the case ends.
The run ends with a table of every target's counts and time, the report paths,
the slowest cases, the cases over `--budget`, the known issues that still
reproduce, any errors the daemon logged, and each failure with its target. The
last line is `PASS:` or `FAIL:` and the totals. Colour is used only on a
terminal and never with `NO_COLOR` set.
### Edge cases and performance
Besides the syscall contract, the suite covers:
- **Block boundaries.** Files of 0 bytes up to two 4 MiB blocks plus 3 bytes,
including every size next to a boundary, then overwrites that grow, shrink
and reshape them.
- **Unusual names.** A 255-byte name, NFC and NFD forms of the same text,
newline, tab, emoji, characters Windows forbids, leading and trailing
spaces, and a case-only rename.
- **A deep tree and a wide directory.** 24 levels of nesting renamed at the
top; 128 entries created concurrently, concurrent unlinks, then `rm -rf`.
Online, each create is a round trip to the server, and the kernel runs
creates in one directory one at a time. So the wide case gets twice the
per-case timeout and notes the time per create as `wide.create_ms`.
- **Rename patterns.** Swaps, chains, log rotation, replacing a file under an
open reader, the `EISDIR`/`ENOTDIR`/`ENOENT` refusals, and a folder moved out
and back.
- **Handle coherency.** Unsynced writes seen by a second handle, two `O_APPEND`
writers, 30 rewrite generations, and a truncate under a reader.
- **Throughput.** 64 MiB written and read sequentially, 1 MiB in 4 KiB writes,
and metadata operations on 100 files. The rates are printed. The case fails
below `PDFS_ACCEPTANCE_MIN_MIBPS` (default 10) or `PDFS_ACCEPTANCE_MIN_OPS`
(default 20 operations per second, or 3 on a FUSE mount, where each create
and unlink waits for Drive before it returns).
### The mount is diffed against an ordinary filesystem
The local reference run is not only a self-test of the runner. Every case
records the facts it established, and a live run is compared against those
recordings, so a case fails when the mount *differs* from an ordinary
filesystem — not only when it raises. The report names the exact key:
filesystem — not only when it raises. The target's `reference` line names the
exact key:
```
[DIVERGENCE] live FUSE /mnt/testmount does not match the reference filesystem:
! creation and open flags.truncated.size: live=6 reference=0
-> reference 1 observation of 412 differs from the ordinary filesystem:
! creation and open flags.truncated.size: live FUSE /mnt/testmount=6 reference=0
```
Facts are recorded in one of two ways, chosen in the test body:
- **compared** — a claim about filesystem semantics. The mount must agree.
- **noted** — context that legitimately differs: inode numbers, block counts,
reported mode bits, and which optional syscalls exist at all. Reported under
`[capabilities]`, never a failure.
reported mode bits, and which optional syscalls exist at all. Listed under
the target's `accepted` line, never a failure.
Because the choice is made at each call site, the set of accepted divergences
is visible in the test that accepts it. There is no separate allowlist to drift
@@ -78,6 +193,101 @@ in-flight `.crdownload` must not be sealed as a revision) and B74 (a file
renamed after close must keep its content) need a real mount and a reachable
daemon; they are skipped, not failed, without one.
**B100** (a copied tree reads back at its exact sizes) copies 48 files of
assorted sizes into the mount at once, as `cp -r` does. On a live mount it
waits for the queue, checks every size and byte, runs `pdfs refresh` on each
folder to drop the cached listing, and checks again. The refresh is what
brought back the provisional sizes the bug reported.
**B101** (a write during its create's upload reaches Drive) holds new files open for
different times before writing, so their empty creates land before, during and after the
bytes arrive, then checks every file after the queue drains.
**B113** (moving a file before its upload makes no conflict) pauses sync with
`pdfs sync pause --for 10m` and writes three files. While their uploads are held,
it renames one, moves one into a folder, and renames and moves the third. It then
resumes sync, waits for the queue, and checks the bytes and names and that no
conflict copy appeared. It skips when sync was already paused. If the run dies
before the resume, the pause ends by itself after ten minutes.
**B88** (`pdfs mkdir`, `rename` and `rm` by local path) creates, renames and
removes a folder and a file by their paths in the mount. Then it checks the
refusals: `pdfs rename` of the mount's root to its own name, and `pdfs rm` of a
file outside every mount. `pdfs rm` of the root itself is tried only on a sync
folder the run created. In a mirrored sync folder all three commands must be
refused instead, and nothing may change.
**pdfs output into a closed pipe** runs `pdfs ls`, `pdfs --json ls` and
`pdfs status` with stdout on a pipe whose reader is already gone. Each must exit
0 with nothing on stderr, as `pdfs ls | head -1` needs.
**B102** (removing a folder drops the uploads queued inside it) uploads a file
into each of two nested folders, pauses sync and rewrites both. While the
revisions are held, it removes one folder with `pdfs rm` and the other through
the mount. Neither revision may stay queued, and neither folder may come back
after the resume. It skips when sync was already paused.
**B94** (a deleted `.part` file leaves nothing queued) writes a `.part` file,
checks that its create is parked, and unlinks it. The parked op must leave the
queue within 30 seconds. The sweep that lets an hour-old park through is not
driven.
**B95** (a `.tmp` file uploads at once) writes `index.tmp` and `preview.temp`
and waits for the queue, which fails at once on a parked op. `pdfs ls` must
list both, and their bytes must read back.
**B111** (renaming a file straight back works) renames a file there and back
eight times through the mount. On a mount it also does so three times with
`pdfs rename`, after the queue is empty, so the renames race nothing but Drive.
It runs against the reference too.
**B121** (a file renamed and moved at once lands) renames a file, moves it into
a subfolder and renames it back across folders through the mount, with no pause
between the steps. On a mount it then runs `pdfs rename` and `pdfs move` back to
back. The file must end up where it was sent, with its bytes, and the case
reports how long each half took. It runs against the reference too.
**B98** (an upload superseded by a write ends) writes 64 MiB and waits for its
upload to show in `pdfs transfers`, then appends to the file. The queue must
drain, the old upload must leave the transfer list, and the file must hold
every byte with no conflict copy. It skips when the upload is never seen on
the wire, and it gets twice the per-case timeout.
**B80** (`pdfs search` finds a new file after a rewrite) writes a file whose
name carries a unique token, waits for the queue, rewrites it and waits again.
`pdfs search` must find it within 30 seconds.
**B48** (an emptied folder can be removed at once) writes twelve files into a
folder, lists it, and moves one more in from elsewhere. It unlinks every
child, runs `pdfs refresh` on a mount, and checks that nothing is listed and
that `rmdir` works. It runs against the reference too.
**B114** (a name `pdfs rm` or `rename` took away stops resolving) looks up two
files, removes one with `pdfs rm` and renames the other with `pdfs rename`. At
once, neither old name may be listed or resolve.
**B115** (`pdfs ls` reports a file's real size) writes files of 6 bytes, 4097
bytes and one block plus 7, and compares the sizes `pdfs ls` lists with the
written ones, after the upload and again after `pdfs refresh`.
**B116** (trashing a folder drops the creates queued in its subfolders) makes
`b116/sub/deeper` and waits for Drive. With sync paused, it writes a `.part`
file there and renames it to its final name, so its create is due but held.
It then runs `pdfs rm b116`, and the create must be gone from the queue. It
restores the folder with `pdfs restore` before it resumes sync. If the file
ends up in the mount's root anyway, the case fails and moves the file to the
trash.
**B117** (a moved file's queued upload keeps its path) rewrites an uploaded
file with sync paused and moves it with `pdfs move`. `pdfs sync queue` must
list the upload under its new path, and the journal must not show the upload
put off with "pending operation deferred". The bytes must arrive at the new
path with no conflict copy. Without `journalctl`, only the queue is checked.
**B119** (an uploaded file stays cached) writes a small file and one of a block
plus 119 bytes, waits for the queue, and expects `pdfs ls` to list both as
cached, so reading them does not need the network.
B74 was found by exactly this mechanism: the B70 case failed, and narrowing it
produced a smaller reproduction that got its own case. The defect is fixed and
live-validated (2026-07-26); the case now guards against its return.
@@ -113,13 +323,61 @@ nothing: the local attribute and content caches will happily return the bytes
that were just written even when nothing reached Drive, which is precisely how
B74 stayed invisible.
### Bugs that are still open
Cases named `open B<n>` check a bug that `docs/BUGS.md` still lists as open.
While the bug reproduces, the case ends as `known`: it is listed at the end of
the run, but does not fail it. So a known bug does not hide new failures, and
the run notices when a fix lands, because the case then passes. Rename it to
`regression B<n>` then, and fail it with `check` instead of `KnownIssue`. No
bug is open at the moment. B114 to B117 started as such cases.
### What a run verifies
A full account run with `--journal-check` checks these `docs/BUGS.md` entries,
many of them fixed but not yet verified:
- B48, B80, B94, B95, B98, B100, B101, B102, B111, B114, B115, B116, B117, B119
and B121, each with its own `regression B<n>` case.
- B118, with "application workloads" and "block boundaries", which failed on
it. It needs Drive to be slow to list a new node, so a clean run is weak
evidence.
- B120, with `--journal-check`, but only in a run that goes offline while it
writes. A mkdir made online goes to Drive at once, so only then does a
folder land from the queue.
- B86, with the mode-switch check in the full matrix.
- B104, B106, B108 and B109, with the cases that found them: "rename
patterns", "application workloads" and the account run's own setup.
B105, B107 and B110 are races. The suite hit each of them once, so a clean run
is weak evidence for them. `--journal-check` reports B110's stall warning, and
every case that writes checks for B105's conflict copies.
The suite does not drive:
- the GUI and the photo gallery (B81, B92, B93, B97);
- states it cannot set up: a `parent_uid` cycle (B82), staged bytes that name
no node (B96), a stuck trash refresh (B103), conflict copies left from an
earlier start (B99), a foreign FUSE mount in the local-file index (B90),
revisions without 4 MiB blocks (B85), and several shares from one owner
(B112);
- B94's hour-long expiry of a park;
- load and failure injection (B22, B27, B30, B37, B42).
Check those by hand, or with the unit tests their entries name.
### Timeouts, reports, and hung mounts
A wedged FUSE operation cannot be interrupted from Python, so each case runs
under a soft alarm (`--timeout`, default 180s) that reports the case and lets
cleanup run, backed by a hard stop that dumps every thread's stack and exits.
That backstop is the only way to learn *where* a mount hung. A timeout ends the
run: later cases against a wedged mount produce noise, not information.
That backstop is the only way to learn *where* a mount hung. After a timeout the
run checks whether the mount still answers a `stat` and a listing within 15
seconds. If it does, the target goes on with its next case, so one slow stretch
of the link does not cost every later case. If it does not, the target's later
cases are not run: against a wedged mount they produce noise, not information.
Cases that make many round trips by design ("application workloads", "unusual
but legal names", "a wide directory" and "regression B98") get twice the limit.
```console
scripts/fuse-acceptance.sh --live /mnt/testmount \
@@ -127,7 +385,8 @@ scripts/fuse-acceptance.sh --live /mnt/testmount \
```
`--report-json` also writes every recorded observation, which is what to attach
to a bug report. `--budget SECONDS` flags cases that pass but got slower —
to a bug report. A known issue has the status `known` there; JUnit has no such
status, so `--report-junit` reports it as skipped. `--budget SECONDS` flags cases that pass but got slower —
B5 was a 186 ms-per-call regression that a pass/fail suite could not see.
`--list` prints every case and which targets it runs against, and `--fail-fast`
stops at the first failure.
@@ -135,14 +394,21 @@ stops at the first failure.
`--journal-check` fails the run if `proton-drive.service` logged any error while
it ran. A suite can pass every assertion while the daemon logs failures behind
it; that has happened, and it was caught only by reading the journal by hand.
Two warnings count as errors too: "queued write conflicts; keeping a conflict
copy" (how B113 showed) and "a fuse worker has held the same job for a long
time" (how B110 showed).
Abandoned `pdfs-acceptance-*` roots from a crashed run are removed at startup,
but only if they match the exact generated name and are over an hour old
(`PDFS_ACCEPTANCE_REAP_AGE`), so a concurrent run is never disturbed.
With `--live`, abandoned `pdfs-acceptance-*` roots from a crashed run are
removed at startup, but only if they match the exact generated name and are
over an hour old (`PDFS_ACCEPTANCE_REAP_AGE`), so a concurrent run is never
disturbed. An account run instead uses the manifest and reaper described
above.
### Automated mode matrix
The managed live runner accepts two existing, empty, unmounted directories. It
An account run drives this matrix itself, in folders it creates. To use
directories of your choice instead, the managed live runner accepts two
existing, empty, unmounted directories. It
registers both as new sync folders, waits for initial synchronization, then
tests all four pairs in order: on-demand/on-demand, on-demand/mirror,
mirror/on-demand, and mirror/mirror. Every transition must become idle, and an
@@ -159,6 +425,38 @@ in `pdfs sync list`. Use `PDFS_ACCEPTANCE_SYNC_TIMEOUT` to extend transition
timeouts and `PDFS_ACCEPTANCE_PDFS` to select a non-installed CLI binary, for
example `target/debug/pdfs`.
After the contract, each mode pair also runs the move cases (`--list` shows
them as `managed`). They move trees between the two folders in both
directions, move several sources in one `pdfs move`, and pass a tree through
My files and out again. Every case waits for Drive before and after the move.
It then checks that the tree is only at the destination, that every byte
arrived, and that neither side made conflict copies. The refusal cases prove
that nothing moves and nothing is lost when:
- the destination already has the name;
- a folder would move into itself;
- a mirror source holds something Drive lacks (a symlink);
- a file in an on-demand source is still open for writing.
The My files case writes a test folder into your real My files and removes it
afterwards. It skips when the daemon reports no My files mount, or when the
locations are on different volumes.
Before a mirror folder switches to on-demand, the runner writes a new file into
it and syncs. Once the folder is mounted, the file must be listed with its
bytes (B86). Each switch reports a `regression B86` result under a separate
`mode switches` target. The bug only shows in a folder that was mounted in an
earlier pairing, because that mount's listing is stale. That happens in the
third pairing, so `--quick` runs the check but does not replay the bug.
In every pairing, each folder must still hold the file the runner wrote into it
at setup, byte for byte. These checks report under `mode switches` as well. A
switch to on-demand frees the local copy, so these reads go to Drive. If the
link drops, the daemon fails such a read with EIO once its two-minute read
budget runs out. That is correct, so the runner waits until the daemon reports
online again, tries once more and notes the retry under the result. Only the
second try decides the result.
### Durability drill
`--durability` restarts `proton-drive.service` in the middle of the suite and
@@ -170,7 +468,9 @@ interrupts a running daemon; `PDFS_ACCEPTANCE_UNIT` selects a different unit.
### Selecting cases
During development, `PDFS_ACCEPTANCE_ONLY` runs tests whose descriptive name
contains the supplied text (for example `PDFS_ACCEPTANCE_ONLY=namespace`). A
contains the supplied text, ignoring case (for example
`PDFS_ACCEPTANCE_ONLY=namespace`). Separate several texts with commas to run
every case that matches any of them (`PDFS_ACCEPTANCE_ONLY=B116,B117`). A
release acceptance run must leave it unset so the complete contract executes.
A filter naming a live-only case (every `regression B<n>` is one) selects nothing
+31
View File
@@ -95,6 +95,28 @@ of the folder, so the files stay visible but take no disk space until opened. Sw
**Synced / Online only** switch on the folder's card, or `pdfs sync mode <id> ondemand` and back
with `pdfs sync mode <id> mirror`. `pdfs locations` lists every local place the service occupies.
### Moving between synced folders and My files
`~/ProtonDrive` and each online-only folder are separate mounts, and a mirrored folder is a plain
folder on disk. `mv`, and dragging in a file manager, cannot rename across them, so they copy
and then delete: every file is downloaded, uploaded again, and the original goes to the Trash.
Use `pdfs move <path> <folder>` or **Move to…** in the app instead. The move happens on Proton
Drive, so it takes seconds whatever the size. For example:
```sh
pdfs move ~/Documents/Scans ~/ProtonDrive/Archive
pdfs move ~/Documents/a.pdf ~/Documents/b.pdf ~/ProtonDrive/Archive
```
The last path is the destination folder; everything before it moves there.
Files already opened in an online-only folder are not downloaded again when
they move into a mirrored one.
A move out of a synced folder waits until its files are on Proton Drive. A file that is still
uploading, has changes that are not uploaded, or is excluded by `.pdfsignore` stops the move,
and the message names it.
Removing a synced folder (**Stop Syncing** or `pdfs sync rm <id>`) keeps the local files and the
copy on Proton Drive. Add `--delete-remote` (or tick **Also delete from Proton Drive**) to delete
the copy on Proton Drive as well.
@@ -235,6 +257,15 @@ The window stays loaded after the first use, so later shortcuts open it instantl
and video open straight from the Proton Drive folder, so players can start before the whole file
downloads. Other Drive files download first, then open.
To make the first shortcut instant as well, start the launcher hidden with your session.
`pdfs-prompt --preload` loads the window without showing it and does nothing if the launcher is
already running. For Hyprland:
```ini
exec-once = pdfs-prompt --preload
bind = SUPER, space, exec, pdfs-prompt
```
### Using your own launcher
`pdfs-prompt --dmenu` shows the results in fuzzel, rofi, wofi, tofi, bemenu or dmenu, whichever is
Binary file not shown.

Before

Width:  |  Height:  |  Size: 35 KiB

After

Width:  |  Height:  |  Size: 114 KiB

+2 -2
View File
@@ -9,11 +9,11 @@
# Build + install: cd packaging && makepkg -fi
pkgname=proton-drive-linux
pkgver=2.6.0
pkgver=2.10.0
pkgrel=1
pkgdesc="Unofficial Proton Drive client for Linux: files-on-demand FUSE mount, CLI, GTK4 app + tray"
arch=('x86_64')
url="https://github.com/narrrl/proton-drive-linux"
url="https://proton-drive.narl.io"
license=('MIT')
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0' 'perl-image-exiftool')
makedepends=('cargo' 'git' 'gettext')
@@ -1,8 +1,8 @@
pkgbase = proton-drive-for-linux-bin
pkgdesc = Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (binaries)
pkgver = 2.5.0
pkgrel = 1
url = https://github.com/narrrl/proton-drive-linux
pkgver = 2.8.2
pkgrel = 2
url = https://proton-drive.narl.io
arch = x86_64
license = MIT
depends = fuse3
@@ -21,25 +21,25 @@ pkgbase = proton-drive-for-linux-bin
optdepends = xdg-utils: "Open folder" action in the tray menu
optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf)
optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding
provides = proton-drive-for-linux=2.5.0
provides = proton-drive-linux=2.5.0
provides = proton-drive-for-linux=2.8.2
provides = proton-drive-linux=2.8.2
provides = pdfs
conflicts = proton-drive-linux
conflicts = proton-drive-for-linux
conflicts = proton-drive-for-linux-git
options = !strip
options = !debug
source = proton-drive-linux-2.5.0-x86_64.tar.gz::https://github.com/narrrl/proton-drive-linux/releases/download/v2.5.0/proton-drive-linux-2.5.0-x86_64.tar.gz
source = io.narl.proton-drive-linux.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.5.0/packaging/io.narl.proton-drive-linux.desktop
source = io.narl.proton-drive-linux-tray.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.5.0/packaging/io.narl.proton-drive-linux-tray.desktop
source = io.narl.proton-drive-linux.svg::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.5.0/packaging/io.narl.proton-drive-linux.svg
source = proton-drive.service::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.5.0/packaging/proton-drive.service
source = LICENSE-2.5.0::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.5.0/LICENSE
sha256sums = 44ee9549dc14260de7e882f810911b253aaf5c83dc70537c0aeeae3769d30ce9
source = proton-drive-linux-2.8.2-x86_64.tar.gz::https://github.com/narrrl/proton-drive-linux/releases/download/v2.8.2/proton-drive-linux-2.8.2-x86_64.tar.gz
source = io.narl.proton-drive-linux-2.8.2.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/io.narl.proton-drive-linux.desktop
source = io.narl.proton-drive-linux-tray-2.8.2.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/io.narl.proton-drive-linux-tray.desktop
source = io.narl.proton-drive-linux-2.8.2.svg::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/io.narl.proton-drive-linux.svg
source = proton-drive-2.8.2.service::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/proton-drive.service
source = LICENSE-2.8.2::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/LICENSE
sha256sums = c0af4ff57dc6f587b1b8431f3180b00d45211389f3dc675569af45f64e89cb36
sha256sums = d190d6771fff0b975271fb62fd3fb0fa8bbf17d7278c10853c18557855b0c123
sha256sums = 1d95e250370220e3bb6aead6ce34d1d6cc759f42c4803b75f1667e78ff47c6c0
sha256sums = e598a93c2715c3a9fdc7fb937902083d4a1135e8fbdfe6fe7e37f0383ebbd828
sha256sums = 25fa3bd9ba52df534be9d8bc66f77d7502fe1423177237d0d14b673dedeb1b60
sha256sums = 77280bff5f693fbc5ffa9f3ccd06a43cfff96d770290aa8d7d0c28e9dde277be
sha256sums = 02c419e7f9af08dda720e4f852f3e57cf5a69e192605440f750e9d244fa2817c
pkgname = proton-drive-for-linux-bin
@@ -1,11 +1,12 @@
# Maintainer: Nils Pukropp <contact@narl.io>
pkgname=proton-drive-for-linux-bin
pkgver=2.5.0
pkgrel=1
pkgver=2.8.2
pkgrel=2
pkgdesc="Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (binaries)"
arch=('x86_64')
url="https://github.com/narrrl/proton-drive-linux"
url="https://proton-drive.narl.io"
_repo="https://github.com/narrrl/proton-drive-linux"
license=('MIT')
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0' 'dbus' 'gcc-libs' 'glibc'
'hicolor-icon-theme')
@@ -24,18 +25,20 @@ options=('!strip' '!debug')
_tag="v$pkgver"
_raw="https://raw.githubusercontent.com/narrrl/proton-drive-linux/$_tag"
# The release tarball holds the four binaries only, so the desktop integration
# files come straight from the tagged tree.
source=("proton-drive-linux-$pkgver-x86_64.tar.gz::$url/releases/download/$_tag/proton-drive-linux-$pkgver-x86_64.tar.gz"
"io.narl.proton-drive-linux.desktop::$_raw/packaging/io.narl.proton-drive-linux.desktop"
"io.narl.proton-drive-linux-tray.desktop::$_raw/packaging/io.narl.proton-drive-linux-tray.desktop"
"io.narl.proton-drive-linux.svg::$_raw/packaging/io.narl.proton-drive-linux.svg"
"proton-drive.service::$_raw/packaging/proton-drive.service"
# files come straight from the tagged tree. Every one carries $pkgver in its
# name: makepkg reuses a cached file of the same name, so an unversioned name
# keeps the old copy after upstream changes it and fails the checksum.
source=("proton-drive-linux-$pkgver-x86_64.tar.gz::$_repo/releases/download/$_tag/proton-drive-linux-$pkgver-x86_64.tar.gz"
"io.narl.proton-drive-linux-$pkgver.desktop::$_raw/packaging/io.narl.proton-drive-linux.desktop"
"io.narl.proton-drive-linux-tray-$pkgver.desktop::$_raw/packaging/io.narl.proton-drive-linux-tray.desktop"
"io.narl.proton-drive-linux-$pkgver.svg::$_raw/packaging/io.narl.proton-drive-linux.svg"
"proton-drive-$pkgver.service::$_raw/packaging/proton-drive.service"
"LICENSE-$pkgver::$_raw/LICENSE")
sha256sums=('44ee9549dc14260de7e882f810911b253aaf5c83dc70537c0aeeae3769d30ce9'
sha256sums=('c0af4ff57dc6f587b1b8431f3180b00d45211389f3dc675569af45f64e89cb36'
'd190d6771fff0b975271fb62fd3fb0fa8bbf17d7278c10853c18557855b0c123'
'1d95e250370220e3bb6aead6ce34d1d6cc759f42c4803b75f1667e78ff47c6c0'
'e598a93c2715c3a9fdc7fb937902083d4a1135e8fbdfe6fe7e37f0383ebbd828'
'25fa3bd9ba52df534be9d8bc66f77d7502fe1423177237d0d14b673dedeb1b60'
'77280bff5f693fbc5ffa9f3ccd06a43cfff96d770290aa8d7d0c28e9dde277be'
'02c419e7f9af08dda720e4f852f3e57cf5a69e192605440f750e9d244fa2817c')
package() {
@@ -44,15 +47,15 @@ package() {
install -Dm755 "$srcdir/pdfs-app" "$pkgdir/usr/bin/pdfs-app"
install -Dm755 "$srcdir/pdfs-prompt" "$pkgdir/usr/bin/pdfs-prompt"
install -Dm644 "$srcdir/io.narl.proton-drive-linux.desktop" \
install -Dm644 "$srcdir/io.narl.proton-drive-linux-$pkgver.desktop" \
"$pkgdir/usr/share/applications/io.narl.proton-drive-linux.desktop"
install -Dm644 "$srcdir/io.narl.proton-drive-linux-tray.desktop" \
install -Dm644 "$srcdir/io.narl.proton-drive-linux-tray-$pkgver.desktop" \
"$pkgdir/etc/xdg/autostart/io.narl.proton-drive-linux-tray.desktop"
install -Dm644 "$srcdir/io.narl.proton-drive-linux.svg" \
install -Dm644 "$srcdir/io.narl.proton-drive-linux-$pkgver.svg" \
"$pkgdir/usr/share/icons/hicolor/scalable/apps/io.narl.proton-drive-linux.svg"
# systemd user unit for the auto-mount daemon; enabled per user, not by pacman.
install -Dm644 "$srcdir/proton-drive.service" \
install -Dm644 "$srcdir/proton-drive-$pkgver.service" \
"$pkgdir/usr/lib/systemd/user/proton-drive.service"
# Translations ship precompiled in the release tarball.
@@ -1,8 +1,8 @@
pkgbase = proton-drive-for-linux-git
pkgdesc = Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (git)
pkgver = 2.5.0.r0.g61266fc
pkgver = 2.8.2.r0.g50483bf
pkgrel = 1
url = https://github.com/narrrl/proton-drive-linux
url = https://proton-drive.narl.io
arch = x86_64
license = MIT
makedepends = cargo
@@ -24,8 +24,8 @@ pkgbase = proton-drive-for-linux-git
optdepends = xdg-utils: "Open folder" action in the tray menu
optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf)
optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding
provides = proton-drive-for-linux=2.5.0.r0.g61266fc
provides = proton-drive-linux=2.5.0.r0.g61266fc
provides = proton-drive-for-linux=2.8.2.r0.g50483bf
provides = proton-drive-linux=2.8.2.r0.g50483bf
provides = pdfs
conflicts = proton-drive-linux
conflicts = proton-drive-for-linux
@@ -1,11 +1,12 @@
# Maintainer: Nils Pukropp <contact@narl.io>
pkgname=proton-drive-for-linux-git
pkgver=2.5.0.r0.g61266fc
pkgver=2.8.2.r0.g50483bf
pkgrel=1
pkgdesc="Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (git)"
arch=('x86_64')
url="https://github.com/narrrl/proton-drive-linux"
url="https://proton-drive.narl.io"
_repo="https://github.com/narrrl/proton-drive-linux"
license=('MIT')
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0' 'dbus' 'gcc-libs' 'glibc'
'hicolor-icon-theme')
@@ -22,7 +23,7 @@ provides=("${pkgname%-git}=$pkgver" "proton-drive-linux=$pkgver" 'pdfs')
conflicts=('proton-drive-linux' 'proton-drive-for-linux' 'proton-drive-for-linux-bin')
# LTO has broken GTK/Rust links for this workspace; the release profile keeps it off.
options=('!lto')
source=("$pkgname::git+$url.git")
source=("$pkgname::git+$_repo.git")
sha256sums=('SKIP')
pkgver() {
@@ -1,8 +1,8 @@
pkgbase = proton-drive-for-linux
pkgdesc = Unofficial Proton Drive client: files-on-demand FUSE mount, CLI, GTK4 app and tray
pkgver = 2.5.0
pkgver = 2.8.2
pkgrel = 1
url = https://github.com/narrrl/proton-drive-linux
url = https://proton-drive.narl.io
arch = x86_64
license = MIT
makedepends = cargo
@@ -23,13 +23,13 @@ pkgbase = proton-drive-for-linux
optdepends = xdg-utils: "Open folder" action in the tray menu
optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf)
optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding
provides = proton-drive-linux=2.5.0
provides = proton-drive-linux=2.8.2
provides = pdfs
conflicts = proton-drive-linux
conflicts = proton-drive-for-linux-bin
conflicts = proton-drive-for-linux-git
options = !lto
source = proton-drive-for-linux-2.5.0.tar.gz::https://github.com/narrrl/proton-drive-linux/archive/refs/tags/v2.5.0.tar.gz
sha256sums = 2872352325204a9d9af31c12e7a793f4546ea0e5426dfb7b4f14a2b17af7c773
source = proton-drive-for-linux-2.8.2.tar.gz::https://github.com/narrrl/proton-drive-linux/archive/refs/tags/v2.8.2.tar.gz
sha256sums = 7f2df1d2748c3d8daaa4c05d242fcd293cb1983572e44bb61a35188dbe5f342c
pkgname = proton-drive-for-linux
@@ -1,11 +1,12 @@
# Maintainer: Nils Pukropp <contact@narl.io>
pkgname=proton-drive-for-linux
pkgver=2.5.0
pkgver=2.8.2
pkgrel=1
pkgdesc="Unofficial Proton Drive client: files-on-demand FUSE mount, CLI, GTK4 app and tray"
arch=('x86_64')
url="https://github.com/narrrl/proton-drive-linux"
url="https://proton-drive.narl.io"
_repo="https://github.com/narrrl/proton-drive-linux"
license=('MIT')
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0' 'dbus' 'gcc-libs' 'glibc'
'hicolor-icon-theme')
@@ -22,8 +23,8 @@ provides=("proton-drive-linux=$pkgver" 'pdfs')
conflicts=('proton-drive-linux' 'proton-drive-for-linux-bin' 'proton-drive-for-linux-git')
# LTO has broken GTK/Rust links for this workspace; the release profile keeps it off.
options=('!lto')
source=("$pkgname-$pkgver.tar.gz::$url/archive/refs/tags/v$pkgver.tar.gz")
sha256sums=('2872352325204a9d9af31c12e7a793f4546ea0e5426dfb7b4f14a2b17af7c773')
source=("$pkgname-$pkgver.tar.gz::$_repo/archive/refs/tags/v$pkgver.tar.gz")
sha256sums=('7f2df1d2748c3d8daaa4c05d242fcd293cb1983572e44bb61a35188dbe5f342c')
_srcname="proton-drive-linux-$pkgver"
+7 -1
View File
@@ -14,7 +14,7 @@ Version: %{version}
Release: 1%{?dist}
Summary: Unofficial Proton Drive client for Linux (FUSE, CLI, GTK4 app + tray)
License: MIT
URL: https://github.com/narrrl/proton-drive-linux
URL: https://proton-drive.narl.io
ExclusiveArch: x86_64
BuildRequires: cargo
@@ -60,12 +60,18 @@ test -n "%{?git_dir}" || (echo 'Pass --define "git_dir $PWD" from the repo root'
test -f %{git_dir}/Cargo.toml
cp -a %{git_dir}/LICENSE .
%build
# The release workflow has already built target/release and passes --define "prebuilt 1".
# rpmbuild exports its own compiler flags, which would make cargo rebuild everything.
%if 0%{?prebuilt}
test -x %{git_dir}/target/release/pdfs
%else
cd %{git_dir}
cargo build --release --locked \
--bin pdfs \
--bin pdfs-tray \
--bin pdfs-app \
--bin pdfs-prompt
%endif
%install
rel=%{git_dir}/target/release
+7 -1
View File
@@ -32,4 +32,10 @@ MemoryMax=6G
TimeoutStopSec=15
[Install]
WantedBy=default.target
# Not default.target: that is reached before the desktop has exported DISPLAY /
# WAYLAND_DISPLAY to the user manager, so After= above never orders anything
# (the two targets are not in the same start transaction). The daemon's first
# keyring call then D-Bus-activates the Secret Service without a display —
# on KDE ksecretd aborts, the start fails, and every login leaves a coredump.
# Being pulled in by the graphical session starts it only once that is up.
WantedBy=graphical-session.target
+298 -292
View File
File diff suppressed because it is too large. Load diff
+298 -292
View File
File diff suppressed because it is too large. Load diff
+298 -292
View File
File diff suppressed because it is too large. Load diff
+298 -292
View File
File diff suppressed because it is too large. Load diff
+298 -292
View File
File diff suppressed because it is too large. Load diff
+298 -292
View File
File diff suppressed because it is too large. Load diff
+299 -293
View File
File diff suppressed because it is too large. Load diff
+298 -292
View File
File diff suppressed because it is too large. Load diff
+298 -292
View File
File diff suppressed because it is too large. Load diff
+2917 -169
View File
File diff suppressed because it is too large. Load diff
+22 -8
View File
@@ -4,20 +4,28 @@ set -Eeuo pipefail
usage() {
cat >&2 <<'EOF'
usage: scripts/fuse-acceptance.sh [--offline-only] [OPTION ...]
usage: scripts/fuse-acceptance.sh [OPTION ...]
scripts/fuse-acceptance.sh --offline-only [OPTION ...]
scripts/fuse-acceptance.sh --live MOUNTPOINT [MOUNTPOINT ...] [OPTION ...]
scripts/fuse-acceptance.sh --managed-live EMPTY_DIR EMPTY_DIR [OPTION ...]
scripts/fuse-acceptance.sh MOUNTPOINT [MOUNTPOINT ...] # compatibility
The local reference suite always runs first and requires no account; the mount
is then diffed against its recorded behaviour. --live runs the same contract on
each mount. Set PDFS_ACCEPTANCE_CONVERGENCE=1 only when all live mountpoints
With no mode, everything runs on the signed-in account: My files, then two
sync folders it registers itself, through every pairing of on-demand and
mirror, with moves and copies between them. Everything it creates is removed
afterwards, on failure and on Ctrl-C too, and permanently deleted from the
trash; a run that was killed is cleaned up by the next one.
--offline-only runs only the local reference suite, which needs no account.
It always runs first; every mount is diffed against its recorded behaviour.
--live runs the same contract on each given mount. Set PDFS_ACCEPTANCE_CONVERGENCE=1 only when all live mountpoints
show the same remote folder.
Options are forwarded to the Python runner. The useful ones:
--list print every case and the targets it runs against
--timeout SECONDS per-case limit (default 180); a hung case dumps stacks
--fail-fast stop at the first failure instead of continuing
--quick one on-demand/mirror pairing instead of all four
--report-json PATH machine-readable results and recorded observations
--report-junit PATH JUnit XML for CI
--budget SECONDS report any case slower than this
@@ -45,8 +53,11 @@ collect_paths() {
case "${1:-}" in
-h|--help) usage ;;
--offline-only|"")
shift || true
"")
exec "${runner[@]}" --account
;;
--offline-only)
shift
exec "${runner[@]}" "$@"
;;
--live)
@@ -60,10 +71,13 @@ case "${1:-}" in
first="$1"; second="$2"; shift 2
exec "${runner[@]}" --managed-live "$first" "$second" "$@"
;;
--*)
# Bare options with no mode: an offline run with those options.
--account)
exec "${runner[@]}" "$@"
;;
--*)
# Bare options with no mode: an account run with those options.
exec "${runner[@]}" --account "$@"
;;
*)
collect_paths "$@"
;;