Compare commits

...
68 Commits
Author SHA1 Message Date
nvrl f6456829fb chore(release): 2.10.0 2026-10-03 21:57:23 +02:00
nvrl 327f538fed test(core): migrate a database 2.8.2 wrote, with ops in every state
A dump of a database the 2.8.2 build wrote, with queued ops due, parked,
failing, deferred for access and claimed, and creates queued below a
folder made offline. Its migration checks every column of the queue and
the sealed revisions, as the real-database test now does too.
2026-10-03 21:55:15 +02:00
nvrl ad8b225cfd refactor(fuse): one tree for every mount
On-demand sync folders kept their own State, which the drain, the
event applier and the sweeps had to visit one by one (B74). Every mount
now serves from the primary's tree. A session keeps only its root, which
the kernel knows as inode 1, and its notification channel; work that
changes the tree tells every mounted session. Size upgrades become
daemon-wide, and moves compare mounts by root instead of by state.
2026-10-03 16:53:31 +02:00
nvrl ea27f2bcd0 test(core): migrate a copy of a real database
An ignored test takes a copy of the database named by PDFS_MIGRATE_DB
with VACUUM INTO over a read-only connection, migrates it, and checks
that every queued op survives unchanged, every node keeps its row
number as its lid, and no node is lost or doubled.
2026-10-03 16:53:31 +02:00
nvrl e4062cc97b feat(fuse,core): number inodes by local id
A node's inode is now derived from its row's local id instead of a
per-run counter, so the same node keeps the same inode across restarts
and across every place that interns it. A lid handed out before the
row is stored reads as no node.
2026-10-03 16:41:26 +02:00
nvrl c34cc93c45 fix(fuse,core): keep a rename made while its create is on the wire
A rename of a node whose create is still queued rewrites the create's
target, but the drain had already copied the op. When the create
landed, the row was deleted unread and the rename was lost (B136).

finish_create now learns where the create was sent and where it landed,
and queues a rename of the real uid when the row has moved on since.
adopt_real_uid keeps the tree's name and folder while that rename is
queued. A landed folder's children also stop naming its placeholder as
their parent in memory, which put the placeholder back on their rows.
2026-10-03 16:20:00 +02:00
nvrl 7e3df7602d feat(core): give every node row a local id that never changes
First step of phase 2 of docs/MILESTONE-3.0.0.md. Schema 36 rebuilds
`nodes` with `lid INTEGER PRIMARY KEY AUTOINCREMENT`. Each row takes its
old rowid as its lid, so the search index, keyed by rowid, stays valid.
AUTOINCREMENT keeps a deleted row's lid from coming back. `uid` stays
unique and becomes nullable, for nodes that are not on Drive yet.

A landed create's row now takes the real uid in place, keeping its lid.
When a listing had already stored the real node, its content is kept
under the placeholder's lid, which the node was known by first.

Nothing reads the lid yet apart from `Db::lid_of`.
2026-10-03 15:10:51 +02:00
nvrl 61a0d31408 chore(release): 2.9.0 2026-10-03 15:00:38 +02:00
nvrl 644a47eb36 fix(ci): quote the simulation command so the workflow parses
The ': ' in 'sim:: --' read as a YAML mapping, so GitHub rejected
ci.yml and ran no job at all since 2c46d62.
2026-10-03 14:55:13 +02:00
nvrl cd90c566cd fix(fuse,core): renames across folders, replaced files, and writes after a create
Found by the seeded simulation runs (docs/BUGS.md).

- B122: a rename across folders goes to Drive as one move-multiple
  request, so a sibling holding the new name in the old folder no longer
  fails it with EIO. The drain uses the same request.
- B123: a second write open on a file just written no longer fails with
  EIO. An empty file counts as complete, our own renames, moves and
  online creates record their echo, a duplicated event is skipped, a
  rename keeps the file's mtime while its revision stays, and a landed
  upload is cached before open handles are rebased onto it.
- B124: a node put back after a failed replacing rename leaves `hidden`
  and its folder is listed again, so it shows in the mount.
- B131: a landed create is read back with the same short retry as an
  online create, and its bytes are cached under the real uid before an
  open handle is repointed at it.
- B135: an online rename settles the tree through the state guard, so
  the database has both folders unlisted before the reply. A lookup
  right after it no longer rebuilds the destination without the file.

The simulation runs no longer tolerate B122 or B131.
2026-10-03 14:50:12 +02:00
nvrl 1c262eb601 refactor(fuse,core): keep queued-op preconditions in the database
Phase 1 of docs/MILESTONE-3.0.0.md, without a change in behaviour.

- Own sealed revisions live only in the own_sealed_rev table. The
  in-memory map in front of it is gone, so revision_conflict gives the
  same answer before and after a restart.
- A queued rename records the node's original name next to its original
  parent, and keeps both across renames that replace the row. The drain
  logs when Drive has the node neither where the rename found it nor
  where it takes it.

The SDK's NodeUpdated event carries no revision id, so echoes of our own
changes stay recognised by the self-change timer.
2026-10-03 14:49:58 +02:00
nvrl bf73a395e5 docs: record B132-B134 from the simulation runs
- B132: a folder whose mkdir is still queued is gone after a restart.
  The root's listing, served offline, was invalidated when the link
  came back, and the restart listed it on Drive without the queued
  folder.
- B133: a file deleted while its upload is in flight can still come
  back as a recovered copy. B105's check goes by op id, and SQLite gives
  the next op queued the id the delete freed.
- B134: a file deleted online while the drain checks its write is kept
  as a conflict copy. The unlink trashes the file on Drive before it
  drops the queued write, so B105's check still finds the op.
2026-10-03 09:07:56 +02:00
nvrl a5138639a8 test(fuse): delete files as they upload in the simulation runs
The runs did not find B105 with its fix reverted. Drive lists a trash
later than the round trips take, so the drain never saw its own
client's delete while it still held the write.

The FakeDrive can now hold the next read of one node on its way to
Drive. A new profile, one-client-deletes, writes a file Drive holds,
waits for the drain to read the node back, deletes the file, and lets
the read go on once Drive shows the trash. A conflict the drain keeps
because a client's own file was trashed now fails the run as a false
conflict. With B105's fix reverted, seeds 1 and 2 fail on it.

Outside the step, such a conflict is counted as B134 when the copy is
named after the file: the drain looked after an online unlink had
trashed the file on Drive but before it dropped the write. Seed 3 of
three-clients came up with it once.

Half the time the step makes the file again before the read goes on.
That is B133: the new file's write takes the op id the delete freed,
and B105's check finds it. The runs count it.

The runs also count B132, an ENOENT after a restart that restored
queued ops, in a seed that lost the link. Seed 4 of one-client-flaky
failed on it. The known-bug matchers now also see what was logged
before the call.
2026-10-03 09:07:56 +02:00
nvrl 988b78e810 docs: record the bugs the simulation runs found (B122-B131)
The first simulation runs found ten open bugs. They are documented here
and not fixed yet, as phase 0 of docs/MILESTONE-3.0.0.md changes no
behaviour. The runs count each one and end the seed where the mount
leaves the model behind.

- B122: a rename across folders fails with EIO when the source folder
  holds the new name.
- B123: a second write open on a file just written fails with EIO.
- B124: after a replacing rename fails, the file it was replacing is gone
  from the mount.
- B125: a folder made offline fails with EIO once its listing is
  invalidated.
- B126: a file created offline under a name an offline rename freed
  lands as a conflict copy.
- B127: a file created offline lands twice when the answer to its upload
  is lost.
- B128: a file created offline over one deleted offline can be lost.
- B129: a file deleted or replaced while its create is on the wire stays
  on Drive.
- B130: a read of a file that a remote change invalidates can hang the
  daemon. The event sync blocks a runtime worker in inval_inode on the
  read's page lock. Nothing then drives the runtime's timers, which the
  read's fetch is waiting for.
- B131: a write to a file whose create could not be read back lands as a
  conflict copy. The create's op is retired before the read-back, so a
  failed read-back leaves the tree on the placeholder's mtime and no
  revision id.
2026-10-03 07:47:02 +02:00
nvrl 2c46d62429 ci: run the simulation runs outside pull requests
A `simulation` job runs the seeded simulation runs of
docs/MILESTONE-3.0.0.md section 8.2 on pushes to main and manual runs. They
mount FUSE against the in-memory FakeDrive, so they need fuse3 but no
account. Like the acceptance suite they stay off pull requests. The job
has a 30 minute limit, and the runs' own watchdog turns a hang into a
failed seed well inside it.
2026-10-03 07:47:02 +02:00
nvrl 97743f17e3 test(fuse): add seeded simulation runs over the FakeDrive
The second half of the simulation harness in docs/MILESTONE-3.0.0.md
section 8.2. A run starts one or more real FUSE daemons on a shared
FakeDrive. Each client owns a folder `c<n>` that only it writes. A seed
then draws syscalls through the mounts, link flaps, restarts, pauses and
settles. Every syscall is checked against a model of the client's tree as
it happens. After the last step every link comes up and the run checks
no loss, POSIX behaviour, no false conflicts, convergence and liveness.

- sim::model is the POSIX model: the tree, what each call should answer,
  and how it changes the tree.
- sim::run has the profiles: one client on a good link, one on a flaky
  link with flaps and restarts, and three clients. PDFS_SIM_SEEDS sets
  how many seeds each runs, and PDFS_SIM_SEED replays one.
- sim::watchdog catches a call that never returns. It takes every
  thread's stack in the process, since no tracer can stop a thread
  blocked on a FUSE request. It then aborts the run's FUSE connections
  and writes the stacks next to the run's state.
- The daemons' logs at INFO and above are kept, so a known open bug is
  told apart from a new failure. Known bugs are counted, and the seed ends
  where the mount leaves the model behind. PDFS_SIM_KNOWN=fail fails on
  them instead.
- A restart retries opening the database while the last run still holds
  it.
- The FakeDrive restamps a node's modification time on a rename or move,
  as Drive does (B113).

The runs mount FUSE, so they are ignored by default. Run them with
`cargo test -p pdfs-fuse --lib sim:: -- --ignored --test-threads=1`.
Test-only; no behaviour change.
2026-10-03 07:47:01 +02:00
nvrl 3d10338807 test(fuse): run real FUSE daemons against the FakeDrive
mount() becomes a thin wrapper over mount_with(), which takes the
DriveApi to reach Drive through and a Host. The daemon's Host is
unchanged: stopped by SIGTERM or SIGINT, installed at the same point as
before, and indexing the user's home. A simulated daemon is stopped over
a channel and skips the home index, so several can share one test
process without walking the developer's home.

The account quota moves onto DriveApi. It was the only call outside the
trait that marks the link lost, so a simulated daemon would have gone
offline on it; the FakeDrive now also models a full account
(InsufficientQuota).

sim::daemon starts one daemon on its own state directory with a real
FUSE mount, and a smoke test (ignored; needs /dev/fuse) checks a local
write lands, a remote write arrives, and a write made offline lands once
the link is back.
2026-10-03 05:09:02 +02:00
nvrl d53ba1be21 test(fuse): add an in-memory FakeDrive with Drive's known faults
The first half of the simulation harness in docs/MILESTONE-3.0.0.md
section 8.1. FakeDrive is one shared server; each simulated daemon reaches
it through a FakeClient that implements DriveApi and carries its own link
state, latency, faults and an SDK-style entity cache.

The faults are the ones the account runs met: listings that lag (B118,
B48), renames and moves refused for a stale name hash because the cache
is only emptied by events (B111, B121), names that refuse a replace (B13,
B104, B106), events delayed, duplicated, reordered and echoed (B70, B105),
requests that stall, fail, or change Drive and lose their reply (B107,
B121), and non-uniform block sizes (B84, B85, B87).

Test-only; no behaviour change.
2026-10-03 04:51:35 +02:00
nvrl c73d5c9436 refactor(fuse): reach Drive through a DriveApi trait
`Core` held a concrete `ProtonDriveClient`, so no test could make Drive
answer late, answer stale or not answer. This is the first step of
phase 0 in docs/MILESTONE-3.0.0.md.

`DriveApi` (`crates/pdfs-fuse/src/drive.rs`) covers the calls the sync
core makes: listings, events, creates, uploads, downloads, renames,
moves, trashes and restores. `ProtonDriveClient` implements it by
forwarding. `Core::drive` holds it, and every call of those methods now
goes through it. Sharing, Photos, devices, invitations and public links
stay on `Core::client`.

The trait takes readers and writers as trait objects, and
`impl dyn DriveApi` puts the client's generic signatures back, so call
sites read the same. Block reads go through `RevisionRead` instead of
the SDK's `RevisionReader`, which only the SDK can construct.

No behaviour change.
2026-10-03 04:37:37 +02:00
nvrl 8d92c69279 fix(fuse): keep an upload pending until its bytes are cached (B119)
The drain retired an upload's op first, and only then read the node back
and linked the staged blob into the content cache. For that one round
trip, the status reported an empty queue for a file that wasn't cached
yet, so anything that waited for an empty queue before reading the file
could still need the network. The account run of 2026-10-03 hit it over
a slow link: "regression B119" failed in the same second the upload
landed.

`Core::landing_uploads` now counts an upload from just before its op is
retired until its bytes are cached. The status adds it to
`pending_uploads`, and reads the queue before the count, so it always
sees the upload in one of them. A drop guard uncounts it on every exit
path.

The op still goes first. An op restored after a crash would re-send a
revision this daemon hasn't recorded as its own yet, and
`revision_conflict` would keep it as a conflict copy of the user's file.
2026-10-03 03:59:05 +02:00
nvrl e32959e40a fix(fuse): land a move right after a rename, and rename across folders quickly (B121)
The SDK caches each node, including the name hash it sends with the next
rename or move, and only an event evicts it. Renames and moves didn't
update that copy. So after a rename, the next change to the same node
went out with the old name hash, Drive refused it as out of date (422),
and the retry kept sending the stale hash until the event feed came
round, up to 10 seconds later.

- Every remote rename and move goes through `Core::rename_remote` or
  `Core::move_remote`. Both drop the SDK's copy of the node after a
  successful change and before each out-of-date retry, so the retry
  sends the current hash.
- A plain move was not retried at all, so a move right after a rename
  failed with EIO through the mount and with "out of date" from
  `pdfs move`. It now retries like a rename.
- The drain's combined move-and-rename also drops the SDK's copy once
  it lands.
- A rename that also changes folders through the mount replies only
  once both halves have landed. When the move fails, the file keeps
  its new name in the source folder, the tree is told so, and the
  rename reports EIO. A dropped link still queues the change.

B111's cause in docs/BUGS.md is corrected to the same SDK cache. The
acceptance case "regression B121" renames and moves a file at once
through the mount and through `pdfs`, and checks names and bytes.
2026-10-03 03:58:38 +02:00
nvrl 0b27c884a7 test(acceptance): turn B114 to B117 into regressions and ride out a dropped link
New live cases:

- regression B48, B80, B94, B95, B98, B102 and B111 replay the bugs
  their entries describe. B48 and B111 also run against the reference.
- regression B114 to B117 replace the open B114 and B115 cases, and
  fail with `check` when the bug comes back.
- regression B119: files written through the mount stay cached once
  they are uploaded.
- regression B86: after a mirror folder switches to on-demand, the
  mount lists what the mirror uploaded just before.

`--journal-check` also counts three warnings as errors: a conflict
copy (B113), a FUSE job held for a long time (B110) and an authority
re-read because the local tree lacked it (B117, B120).

A timeout no longer ends a target when the mount still answers a stat
and a listing. Cases that make many round trips get twice the limit.

A write is queued when the kernel releases the file, after close(2)
has returned, so cases now wait for the ops they expect
(`Daemon.queued`). "regression B102" saw one revision of two because
it read the queue once.

On flaky Wi-Fi, the run crashed twice on a read that the daemon
correctly failed with EIO once its read budget ran out. The end-of-run
read-back and the mode matrix's file checks now record a result
instead. The mode matrix tries such a read once more after the daemon
is back online, and notes it.
2026-10-03 02:10:49 +02:00
nvrl 37c04f1e69 fix(fuse,core): fix B114 to B120 from the account runs
- B114: `pdfs rm`, `pdfs rename` and `pdfs move` tell each kernel
  session that the old name is gone, so it stops resolving at once
  instead of for up to 30 seconds (`Core::forget_everywhere`,
  `Core::relocate_everywhere`).
- B115: `pdfs ls` and the app's file browser wait up to 10 seconds for
  the listing's size fetch, as `stat` does, so they show a file's real
  size instead of its encrypted one.
- B116: trashing a folder also drops the creates queued in its
  subfolders. A parked one stayed queued with its bytes, and a due one
  would have landed in the root.
- B117: `pdfs move` and `pdfs rename` move the node in each mount
  instead of forgetting it, so its queued upload keeps its path in
  `pdfs sync queue` and is not put off once after the resume.
- B118: when a create or mkdir reads back nothing, the mount asks again
  and then queues it as it does offline, instead of failing with ENOENT
  for something Drive has made.
- B119: a landed upload keeps its staged bytes as the file's cached
  content, unless it is larger than a sixteenth of the cache, so
  reading a file just written does not need the network.
- B120: a landed folder's row passes to its real uid in the same
  transaction, so a file queued inside it is not put off for five
  seconds.

docs/BUGS.md records B117 to B120. The account run of 2026-10-03
verified B114 to B117 and B119 in My files.
2026-10-03 02:10:23 +02:00
nvrl 625776081b test(acceptance): cover B113, B88 and the closed pipe, and report more readably
New live cases:

- regression B113: files renamed and moved while sync is paused upload
  without a conflict copy.
- regression B88: pdfs mkdir, rename and rm by local path, the
  mount-root guard, a path outside every mount, and the refusal in a
  mirrored folder.
- pdfs output into a closed pipe exits 0 and prints nothing on stderr.
- open B114 and open B115 check two bugs that are still open. While
  one reproduces, its case ends as "known", which is listed but does
  not fail the run.

The console output names the pdfs version, the storage and the daemon
state of each target, and prints one aligned line per case, with its
details beneath. A failed case points to its docs/BUGS.md entry. The
run ends with a summary table, the slowest cases, the known issues
and the numbered failures. A queue wait now fails at once on a parked
op instead of waiting out its timeout.

docs/BUGS.md records B114, B115 and B116, which the first live run
found, and marks B113 and B88 as verified live.
2026-10-02 22:16:36 +02:00
nvrl 8cde79b8e3 fix(cli): stop quietly when stdout is closed
`pdfs ls | head` ended in a "Broken pipe" panic once `head` had its
lines, because std's `println!` panics on a failed write.

The CLI now shadows `println!` and `print!` with versions that exit 0 on
a closed pipe and report any other write error, such as a full disk,
with exit 1. Restoring the default SIGPIPE would also stop the panic,
but the same binary runs the daemon, which must not die when a socket's
peer goes away.
2026-10-02 21:16:34 +02:00
nvrl eb53d122cc fix(fuse,cli): let rm, rename and mkdir take paths in on-demand folders
`pdfs rm ~/Downloads/x` answered "is not under the mountpoint" while
`pdfs move` took the same path: the Delete, Rename and CreateFolder
handlers still read paths against the primary mount only (B88).

They now go through `route_change`, the router ResolveConflict already
used, renamed from `route_conflict`. It answers the request in the
inode space of the mount that owns the path.

An on-demand folder's root has a parent on Drive, so `pdfs rm
~/Downloads` would have trashed the whole synced folder from under its
mount. rm and rename refuse a path that names a mount's root.
2026-10-02 19:54:37 +02:00
nvrl c64252193f fix(fuse): keep the revision id of a file created through the mount
The write handle a create opens was given no revision id, so the drain's
conflict check fell back to (mtime, size). Drive advances a node's mtime
on a move or rename, so a new file moved before its first upload drained
landed as a (sync-conflict) copy next to its own empty original. Take the
minted node's revision id, and rebase handles still open on a queued
create onto the node it landed as (B113).
2026-10-02 19:25:33 +02:00
nvrl cc70a21c05 fix(aur): version -bin source filenames
makepkg reuses a cached source file of the same name, so the unversioned
proton-drive.service from an older build failed the 2.8.2 checksum. Carry
$pkgver in every file fetched from the tagged tree, like LICENSE already
did, and bump pkgrel to 2.
2026-10-01 23:50:06 +02:00
Nils Pukropp 644995cc56 chore(aur): update packages to 2.8.2 2026-10-01 19:49:42 +00:00
nvrl 50483bf3e1 chore(release): 2.8.2 2026-10-01 21:39:05 +02:00
nvrl 4caba0a34c chore(deps): require proton SDK 0.7.2
0.7.2 resolves each shared root with its own key. With 0.7.1, when
several accepted shares came from the same owner's volume, only the first
appeared under Shared with me and the rest were skipped with "missing
key" (B112, proton-sdk-rs#2).
2026-10-01 18:17:25 +02:00
Nils Pukropp e8554d6434 chore(aur): update packages to 2.8.1 2026-10-01 14:58:09 +00:00
Claude ba1fdd65be chore(release): 2.8.1 2026-09-30 17:02:31 +00:00
nvrl 0d4163b9bf fix(service): start with the graphical session, not default.target (#24)
default.target is reached before the desktop exports DISPLAY/WAYLAND_DISPLAY
to the user manager, so After=graphical-session.target never applied. The
daemon's first keyring call D-Bus-activated the Secret Service without a
display; on KDE ksecretd aborted, the first start failed, and every login
left a coredump.

Install into graphical-session.target instead, and use reenable when the
front-ends enable the unit so existing default.target.wants links are
replaced rather than kept alongside the new one.

Fixes #23
2026-09-30 19:01:50 +02:00
Nils Pukropp f3c7a809f4 chore(aur): update packages to 2.8.0 2026-09-30 02:15:19 +00:00
nvrl c1b5378045 fix(aur): download sources from GitHub, not the homepage
73e60f1 pointed url at proton-drive.narl.io, and all three AUR PKGBUILDs
built their source URLs from $url, so the release tarball, the binary
archive and the git clone were fetched from the homepage and 404ed. The
sources now come from a separate _repo on GitHub; url stays the homepage.
2026-09-30 04:13:06 +02:00
nvrl 3d6f53c16d chore(release): 2.8.0 2026-09-30 03:56:47 +02:00
nvrl 8ceee5285b chore(fuse): log how long each queued upload takes
"pending upload landed" now carries upload_ms, the time on the wire, and
total_ms, the whole op including the conflict check and gap-fill. A slow
drain then shows where its time goes without a profiler.
2026-09-30 03:55:48 +02:00
nvrl a181f1a5f0 fix(core,fuse): keep working through network drops
The mount only ever went from offline to online: nothing marked it
offline again, so after the Wi-Fi dropped every create, mkdir, delete and
rename called Drive, waited out the SDK's retries for up to two minutes
and answered EIO. Background loops kept polling the dead link.

Connectivity now lives in pdfs-fuse's new link module and goes both
ways. A call that gets no answer (a transport error or a request
timeout) marks the mount offline; a probe thread checks every 2 s,
backing off to 30 s, and on recovery re-lists folders served from the
database, retries failed operations and wakes the drain. Interactive
calls get a 20 s deadline, so a dead link costs one slow call instead of
many.

While offline:
- create, mkdir, unlink, rmdir and rename fall back to the offline
  queue instead of answering EIO;
- folders list from the database even when not marked listed, and are
  re-listed once the link returns;
- reads of uncached data wait up to 30 s for the link, then retry once
  within the 120 s read deadline;
- statfs serves the last quota instead of fetching, and quota refreshes
  are single-flight;
- the event and photo event loops wait for the link instead of polling.

A create whose request timed out may have landed anyway. When the drain
replays it and finds a same-named twin of the same kind, created within
the last two minutes and still empty, it adopts that node instead of
making a conflict copy. The drain also leaves alone a name still held
by a queued trash.
2026-09-30 03:25:00 +02:00
nvrl 7251f00e91 chore(deps): require proton SDK 0.7.1
0.7.1 sends TCP keepalives on its connections, so a link that dies
silently is noticed instead of leaving a request waiting on a dead
socket until its timeout.
2026-09-30 03:25:00 +02:00
nvrl cb38f79aa3 fix(fuse,scripts): log why a create finds no parent and fit the metadata floor to FUSE
A create that answered ENOENT left no trace in the log. It now warns
when the parent folder is no longer known, and when the parent left the
tree while its children were being listed, naming the parent inode and
the new name.

The acceptance script's metadata throughput floor was 20 operations per
second, which a FUSE mount cannot reach: each create and unlink waits for
Drive before it returns, about 350 ms apiece. On a FUSE mount the floor
is now 3 operations per second; other paths keep 20. The mount type is
read from /proc/self/mountinfo.
2026-09-29 21:22:54 +02:00
nvrl 54e9921efb fix(fuse): stop reads deadlocking on size upgrades and retry stale renames
A read that settles a provisional size invalidated the file's page cache
while the kernel held that read's page lock, so the worker waited on
itself forever (B110). Size upgrades now invalidate attributes only.

Renaming a node right after renaming it could be refused by Drive as out
of date, because the name hash read just before still held the old name.
Every remote rename now goes through Core::rename_remote, which retries
InvalidRequirements for up to ten seconds (B111).
2026-09-29 19:47:20 +02:00
nvrl f252b09757 fix(fuse): keep the size of a file open for writing across re-listings
B109: an online create seals an empty file on Drive, and until the last
close the written bytes live only in the open handle. A re-listing of
the parent, a size upgrade or a create landing re-interned the node with
Drive's empty size, the kernel capped reads at that size, and SQLite
reported "database disk image is malformed". State::keep_open_write_size
now carries the handle's size and local mtime over in all three places.

The acceptance script waits for My files to mount after a daemon restart
instead of failing at once.
2026-09-29 16:49:13 +02:00
nvrl a2992b3b9d fix(core,fuse): keep replaced and vanished files from failing saves, reads and syncs
Fixes from the account acceptance runs:

- B104: a reader open across a rename-over keeps working; the replaced
  node is unlinked like any other open file instead of forgotten.
- B105: a drain worker holding a revision withdrawn by an unlink no
  longer uploads it as a recovered-… conflict copy (Db::op_exists).
- B106: an open victim's queued ops are withdrawn at trash time, so a
  repeated editor save no longer fails with AlreadyExists; the staged
  blob goes with the last close.
- B107: a remote read gives up after 120 s with EIO instead of hanging
  its caller; the timer is built inside the runtime.
- B108: a mirror upload whose local file was deleted after the scan is
  skipped rather than counted as an error.

The acceptance script reports the failing line for non-assert errors,
ignores queue ops that predate the run, purges with delete-forever
first, and diagnoses a malformed SQLite database against a local replay.
2026-09-29 16:20:51 +02:00
nvrl 7b67db5ced fix(core,fuse,cli): harden writes, trash and cleanup found by account acceptance runs
The FUSE acceptance suite now runs on the signed-in account by default and
cleans up after itself; its first live runs found three daemon bugs.

- B101: a pwrite past EOF now authors the hole it leaves, so truncated bytes
  no longer come back from the base. A write attached to a create while that
  create uploads becomes a revision of the real uid instead of being dropped.
- B102: removing a folder drops the queued revisions of files below it, found
  through the node tree, with their blobs and in-flight uploads. `pdfs rm`
  now drops queued ops too.
- B103: each server call of a trash refresh times out after 120 s, so one hung
  call can no longer freeze the trash listing until restart.

Acceptance suite: account mode with a manifest, lock and reaper; cleanup by
uid; queued ops naming a test root count as leftovers; new edge-case,
throughput and B100/B101 cases; the wide-directory case is split from the
deep tree and gets a doubled budget. Adds `pdfs trash --json`.
2026-09-29 10:09:44 +02:00
nvrl 84c9ade8e3 fix(fuse): settle provisional sizes before reads and write opens
A file listed by the cheap enumeration reports its ciphertext size until
the size upgrade lands. Since B84 a read on that size fails with EIO, and
a write open started its handle at it. Reads and write opens now fetch
the real size first. A folder past the size-upgrade cap is queued instead
of dropped, so a large copy no longer leaves most folders provisional
(B100).
2026-09-29 07:37:41 +02:00
nvrl 9e26cd0285 chore(release): 2.7.0 2026-09-29 05:30:38 +02:00
nvrl 73e60f1151 chore: point homepage metadata at proton-drive.narl.io
The About dialog's website link, the README, and the package metadata
(Cargo, PKGBUILDs, AUR .SRCINFO, .deb, .rpm) now use the project
homepage. Source code, issue, and release links stay on GitHub.
2026-09-29 05:30:38 +02:00
nvrl 088e8f3b0d test(acceptance): cover moves between locations in the managed matrix
Every managed mode pair now also moves trees between the two folders in both
directions, moves several sources in one command, and passes a tree through
My files and out again. Each case waits for Drive before and after, then
checks the tree is only at the destination, byte for byte, with no conflict
copies. Refusal cases cover a taken name, a folder into itself, a mirror
source holding a symlink, and an on-demand file still open for writing.
2026-09-29 00:38:40 +02:00
nvrl 261d5dbf84 feat(core,fuse,cli): reuse cached content and take several sources in moves
- A mirror download first copies the file out of the content cache when the
  same revision is cached whole or block by block, so moving a file out of an
  on-demand folder into a mirror costs no traffic for what was already read.
- `pdfs move a b c DEST` moves every source, reports each failure and exits
  non-zero if any failed.
- Mirror-to-mirror moves rename with RENAME_NOREPLACE, so a name that
  appeared at the destination after the check is never overwritten.
- The local copy a move out of a mirror drops is checked again just before
  it is deleted; anything written into it meanwhile keeps the copy and is
  uploaded as new by the next pass.
2026-09-29 00:38:40 +02:00
nvrl 5f9d07aff1 feat(fuse,core,cli,gui): move between locations without transferring content
A move request with two absolute paths is resolved against every
location (My files, on-demand and mirror synced folders) and made as a
server-side move_node. The local side is levelled afterwards: mounts
forget and re-list, a mirror folder renames its copy and baseline along
or drops the copy once Drive holds it.

Anything only this disk holds is refused up front: queued ops, open
writes, unsynced or ignored mirror files and symlinks. The GUI Move
dialog gains a top-level list of locations.
2026-09-29 00:26:56 +02:00
nvrl 3fb7b9be6b chore(release): 2.6.2 2026-09-28 23:58:26 +02:00
nvrl 6c3a50ce7f fix(gui): let the Sync page use the full width
The transfer label did not ellipsize, so a long file name set the
column's minimum width and pushed it past the clamp. The page width
changed with every file.

The label now ellipsizes in the middle, and the Sync page drops the
clamp so Overview and Folders fill the space next to the sidebar.
2026-09-28 23:58:26 +02:00
nvrl f8304a0e86 fix(gui): keep Albums and Places selected when returning to Photos
Every visit to the Photos page loaded the timeline, and a finished
timeline load always shows the timeline. Leaving Albums or Places and
coming back therefore showed the timeline under the Albums or Places
toggle.

A visit now reloads the view the toggle selects. An open album or place
still loads its photos as before.
2026-09-28 23:58:26 +02:00
Nils Pukropp e63f39f8c2 chore(aur): update packages to 2.6.1 2026-09-28 21:44:18 +00:00
nvrl 6f100dbe5e chore(images) updated prompt image 2026-09-28 23:35:53 +02:00
nvrl 7c5ef269cd chore(release): 2.6.1 2026-09-28 23:35:14 +02:00
nvrl f13b6fc0ef feat(gui): add pdfs-prompt --preload for session autostart
--preload registers the resident GTK launcher and realizes its window
without showing it, so the first shortcut opens it as fast as later
ones. When a launcher is already running it exits without activating
it.

The application hold now happens in startup, which only the primary
instance runs. Taken before registration, it also kept every remote
invocation alive after it had forwarded its activation.
2026-09-28 23:35:14 +02:00
nvrl 3d61fc637b ci(release): reuse cached builds from main
The release job never hit a cache: its rust-cache key contained the job
name, tags cannot restore caches of earlier tags, and rpmbuild rebuilt
the whole workspace under its own compiler flags.

CI on main now saves shared debug and release caches that the release
workflow restores. Quality gates and the package build run in parallel,
and the GitHub release is created once both pass. The RPM spec skips
its cargo build when the caller passes --define "prebuilt 1".
2026-09-28 23:26:24 +02:00
Nils Pukropp 889b2aa4a9 chore(aur): update packages to 2.6.0 2026-09-28 21:22:30 +00:00
nvrl 8bd3867cb3 fix(gui): keep the launcher preview at its fixed width
A wrapping label asks for its whole text on one line as its natural
width, so a long location in the preview pane widened it, squeezed the
results list and stretched the window. Cap the preview labels' natural
width, limit detail values to three lines with a tooltip for the full
location, and keep the pane from taking extra width.
2026-09-28 23:04:48 +02:00
nvrl 9577387c2f chore(release): 2.6.0 2026-09-28 22:57:28 +02:00
nvrl 7a087b8827 feat(gui): rewrite the search launcher with a preview pane
The launcher (pdfs-prompt) now follows the app's theme: the theme code
moved into a shared theme.rs, the prompt's stylesheet lives in
resources/prompt.css on the libadwaita palette, the Proton-purple accent
pin is gone, and every activation re-reads the Proton theme and open_with
settings so changes in the app apply without restarting the launcher.

The window is rebuilt in src/hud/:
- one list with Recent, Pinned and Best matches sections; a click
  selects, a double-click or Enter opens, focus stays in the entry
- rows show a thumbnail, the name with its matched characters in bold,
  location, sync badge, size and age
- a preview pane with a large thumbnail, type, size, modified date,
  location and availability, plus Open (Enter), Show in Folder
  (Ctrl+Enter, via FileManager1) and Copy Path (Ctrl+C / Ctrl+Shift+C)
- thumbnails from the freedesktop cache, local decoding (written back
  to the shared cache) or the daemon, on two worker threads
- a history of opened files that fills the Recent section and adds a
  frecency bonus to search ranking
- the mount check before opening a Drive file runs off the main thread
  with a 2 s timeout, and Escape cancels a slow open
- rows can be dragged out as files
- the search debounce drops to 80 ms

New strings are translated in every catalog.
2026-09-28 22:53:46 +02:00
nvrl 7099106802 docs: show the Places map screenshot
Rename photos_mpa.png to photos_map.png and add it to the README
screenshot table.
2026-09-28 22:10:14 +02:00
nvrl 43385d36ee chore(images) added up to date images 2026-09-28 22:09:07 +02:00
nvrl 4b97f401cb docs: refresh screenshots
Replace the Photos screenshot with one of placeholder stock photos, add
Sync, conflict history and Computers screenshots, and drop the retired
Activity, Settings, Shared and restore-devices images from the README.
2026-09-28 22:08:16 +02:00
nvrl af869944cf docs: restructure and update documentation for 2.5.0
Split the README into a short front page and task-oriented pages under
docs/: installation, user guide, CLI reference, configuration and
troubleshooting, with an index in docs/README.md. Rewrite RECOVERY.md
around Continue This Backup Here and Restore Folders, and DEVELOPMENT.md
around running, debugging and releasing from source.

Bring ARCHITECTURE.md up to date with the read path, push events, the
thread map, conflict copies and the machine profile, and move the feature
design notes there. Collect planned work in ROADMAP.md. Link the threat
model from SECURITY.md and the translation rules from CONTRIBUTING.md.

Add scripts/publish-wiki.sh, which mirrors the user pages to the GitHub
wiki with rewritten links and only pushes with --push.
2026-09-28 20:04:22 +02:00
Nils Pukropp 4b90abf3b5 chore(aur): update packages to 2.5.0 2026-09-28 16:56:58 +00:00
111 changed files with 25205 additions and 7301 deletions

No files matched your search

+77
View File
@@ -32,8 +32,14 @@ jobs:
with:
toolchain: '1.96'
components: rustfmt, clippy
cache: false
# Tags can only restore caches saved on main, so only main saves one. The
# release workflow restores this key for its quality gates.
- uses: swatinem/rust-cache@v2
with:
shared-key: debug
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Formatting
run: cargo fmt --all -- --check
@@ -57,3 +63,74 @@ jobs:
# Real Proton credentials and /dev/fuse deliberately do not run on pull
# requests. Follow docs/TESTING.md on a dedicated test-account runner.
# The seeded simulation runs (docs/MILESTONE-3.0.0.md §8.2): real FUSE
# daemons against an in-memory Drive, so they need /dev/fuse but no account.
simulation:
if: github.event_name != 'pull_request'
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- uses: actions/checkout@v7
- name: Install native dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
pkg-config \
fuse3 \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
- name: Install minimum supported Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
cache: false
- uses: swatinem/rust-cache@v2
with:
shared-key: debug
save-if: false
- name: Simulation runs
run: 'cargo test -p pdfs-fuse --lib --locked sim:: -- --ignored --test-threads=1'
# Keeps the release-profile dependencies cached on main, where the release
# workflow can restore them; a tag cannot reuse the caches of earlier tags.
release-cache:
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
- name: Install native dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
pkg-config \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
- name: Install minimum supported Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
cache: false
- uses: swatinem/rust-cache@v2
with:
shared-key: release
- name: Release build
run: cargo build --release --locked
+73 -22
View File
@@ -10,8 +10,60 @@ permissions:
contents: write
jobs:
checks:
name: Release quality gates
runs-on: ubuntu-24.04
steps:
- name: Checkout repository
uses: actions/checkout@v7
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
pkg-config \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
- name: Set up Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
components: rustfmt, clippy
cache: false
# A tag can read the caches of main but its own caches are useless to the next
# tag, so this only restores what CI saved on main.
- name: Cache Cargo target
uses: swatinem/rust-cache@v2
with:
shared-key: debug
save-if: false
- name: Verify tag and manifest versions
if: startsWith(github.ref, 'refs/tags/')
shell: bash
run: |
tag_version="${GITHUB_REF_NAME#v}"
manifest_version="$(sed -n 's/^version = "\([^"]*\)"/\1/p' Cargo.toml | head -n1)"
pkgbuild_version="$(sed -n 's/^pkgver=//p' packaging/PKGBUILD | head -n1)"
test "$tag_version" = "$manifest_version"
test "$tag_version" = "$pkgbuild_version"
- name: Release quality gates
run: |
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only
build-release:
name: Build and Release
name: Build packages
runs-on: ubuntu-24.04
environment: 'rpm signing'
@@ -38,27 +90,14 @@ jobs:
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
components: rustfmt, clippy
cache: false
# Restores the release-profile dependencies that CI builds on main.
- name: Cache Cargo target
uses: swatinem/rust-cache@v2
- name: Verify tag and manifest versions
if: startsWith(github.ref, 'refs/tags/')
shell: bash
run: |
tag_version="${GITHUB_REF_NAME#v}"
manifest_version="$(sed -n 's/^version = "\([^"]*\)"/\1/p' Cargo.toml | head -n1)"
pkgbuild_version="$(sed -n 's/^pkgver=//p' packaging/PKGBUILD | head -n1)"
test "$tag_version" = "$manifest_version"
test "$tag_version" = "$pkgbuild_version"
- name: Release quality gates
run: |
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only
with:
shared-key: release
save-if: false
- name: Build release binaries
run: |
@@ -127,7 +166,7 @@ jobs:
Maintainer: Nils Pukropp <contact@narl.io>
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4, libimage-exiftool-perl
Installed-Size: ${INSTALLED_SIZE}
Homepage: https://github.com/narrrl/proton-drive-linux
Homepage: https://proton-drive.narl.io
Description: Unofficial Proton Drive client for Linux
Files-on-demand Proton Drive for the Linux desktop, unaffiliated with Proton AG:
- FUSE virtual mount with block-level caching
@@ -152,6 +191,7 @@ jobs:
--define "_topdir $PWD/rpmbuild" \
--define "_sourcedir $PWD" \
--define "git_dir $PWD" \
--define "prebuilt 1" \
--define "version ${RPM_VERSION}" \
--define "_dbpath $PWD/rpmbuild/db" \
packaging/proton-drive-linux.spec
@@ -190,8 +230,19 @@ jobs:
*.rpm
if-no-files-found: error
publish:
name: Create GitHub Release
needs: [checks, build-release]
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-24.04
steps:
- name: Download packages
uses: actions/download-artifact@v8
with:
name: proton-drive-linux-builds
- name: Create GitHub Release
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v3
with:
files: |
@@ -205,7 +256,7 @@ jobs:
aur:
name: Publish AUR packages
needs: build-release
needs: publish
if: startsWith(github.ref, 'refs/tags/')
uses: ./.github/workflows/aur.yml
with:
+14 -5
View File
@@ -6,9 +6,10 @@ deliberately high — everything else is ordinary Rust review.
## Before you start
- Read [`AGENTS.md`](AGENTS.md) for repository conventions and
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for how the daemon, front ends, and SDK fit
together.
- Read [`AGENTS.md`](AGENTS.md) for repository conventions,
[`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) for building, running and debugging from source,
and [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for how the daemon, front ends, and SDK fit
together. [`docs/README.md`](docs/README.md) indexes the rest.
- Check [`docs/BUGS.md`](docs/BUGS.md) — it is the authoritative issue ledger, and your problem may
already be tracked there with context.
- For anything larger than a fix, open an issue first so the design can be discussed before you
@@ -23,8 +24,9 @@ cargo build --workspace
```
GUI and FUSE targets need the GTK4, libadwaita, WebKitGTK 6, libsecret, D-Bus, and FUSE3
development packages listed in the [README](README.md#prerequisites). Minimum supported Rust
version is 1.96.
development packages listed in [`docs/INSTALL.md`](docs/INSTALL.md#build-dependencies). Minimum
supported Rust version is 1.96. [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md#running-from-source)
explains how to run your build next to, or instead of, the installed service.
## Quality gates
@@ -51,6 +53,13 @@ use a dedicated test account, never one holding data you cannot lose.
control-socket work stays off GTK's main thread.
- **`staging/` and `recovery/` may hold the only copy of user data.** Nothing may clear or purge
them opportunistically.
- **The GUI is fully translated.** Wrap every user-visible string in the helpers from
`crates/pdfs-gui/src/i18n.rs`, run `po/update.sh`, and translate the new entries in every
catalog in `po/LINGUAS` in the same change. See [`docs/TRANSLATING.md`](docs/TRANSLATING.md).
The CLI and daemon messages stay in English.
- **Document what users see.** A new command, setting or behaviour belongs in the matching page
under `docs/` (user guide, CLI reference or configuration) and in the `[Unreleased]` section of
[`docs/CHANGELOG.md`](docs/CHANGELOG.md).
## Commits and pull requests
Generated
+10 -8
View File
@@ -3474,7 +3474,7 @@ checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4"
[[package]]
name = "pdfs-cli"
version = "2.5.0"
version = "2.10.0"
dependencies = [
"anyhow",
"clap",
@@ -3489,7 +3489,7 @@ dependencies = [
[[package]]
name = "pdfs-core"
version = "2.5.0"
version = "2.10.0"
dependencies = [
"async-trait",
"dbus-secret-service-keyring-store",
@@ -3513,8 +3513,9 @@ dependencies = [
[[package]]
name = "pdfs-fuse"
version = "2.5.0"
version = "2.10.0"
dependencies = [
"async-trait",
"base64 0.23.1",
"fuser",
"futures",
@@ -3528,12 +3529,13 @@ dependencies = [
"sha1 0.11.0",
"tokio",
"tracing",
"tracing-subscriber",
"zip",
]
[[package]]
name = "pdfs-gui"
version = "2.5.0"
version = "2.10.0"
dependencies = [
"anyhow",
"async-channel",
@@ -3773,9 +3775,9 @@ dependencies = [
[[package]]
name = "proton-drive-rs"
version = "0.7.0"
version = "0.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0f6996724e51e99f03d2843828de9477afc39baa764403cb69490cabc1c4d93"
checksum = "d6ce113d234474821d34787436ee80e1458e3d8a8f97212219615ed315eb70be"
dependencies = [
"async-trait",
"base64 0.23.1",
@@ -3800,9 +3802,9 @@ dependencies = [
[[package]]
name = "proton-sdk"
version = "0.7.0"
version = "0.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91cc25654133bd5714407a1df0996eaf214a6e8e0bb95d8139bc42e7cee31c34"
checksum = "2a33c67df9694f90ce60e5b34984275956244138511b4640ca0d100e1dac8b3d"
dependencies = [
"aes-gcm 0.11.1",
"async-trait",
+4 -4
View File
@@ -3,20 +3,20 @@ resolver = "2"
members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"]
[workspace.package]
version = "2.5.0"
version = "2.10.0"
edition = "2024"
license = "MIT"
authors = ["Nils Pukropp <contact@narl.io>"]
rust-version = "1.96"
repository = "https://github.com/narrrl/proton-drive-linux"
homepage = "https://github.com/narrrl/proton-drive-linux"
homepage = "https://proton-drive.narl.io"
readme = "README.md"
keywords = ["proton", "drive", "fuse", "sync", "linux"]
categories = ["filesystem", "command-line-utilities"]
[workspace.dependencies]
proton-sdk = { version = "0.7.0" }
proton-drive-rs = { version = "0.7.0" }
proton-sdk = { version = "0.7.2" }
proton-drive-rs = { version = "0.7.2" }
# Internal crates.
pdfs-core = { path = "crates/pdfs-core" }
+101 -450
View File
@@ -4,513 +4,164 @@
# Proton Drive for Linux
**Files-on-demand Proton Drive for the Linux desktop** — a FUSE virtual mount with block-level
caching, a scriptable CLI, and a native GTK4 app with tray and search launcher.
**Proton Drive on the Linux desktop.** Your files as a folder that downloads on demand, synced
local folders, Photos, sharing, and a native GTK4 app with tray and search launcher, plus a
scriptable CLI.
**[proton-drive.narl.io](https://proton-drive.narl.io)**
[![CI](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml/badge.svg)](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/v/release/narrrl/proton-drive-linux?sort=semver)](https://github.com/narrrl/proton-drive-linux/releases/latest)
[![AUR](https://img.shields.io/aur/version/proton-drive-for-linux)](https://aur.archlinux.org/packages/proton-drive-for-linux)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust 1.96+](https://img.shields.io/badge/rust-1.96%2B-orange.svg)](https://www.rust-lang.org)
[![Platform: Linux](https://img.shields.io/badge/platform-linux-lightgrey.svg)](#prerequisites)
<img src="images/files.png" alt="The GTK4 file browser" width="820">
<img src="images/files.png" alt="My files in the Proton Drive app" width="820">
</div>
> [!IMPORTANT]
> This is an **unofficial**, community-built client. It is not affiliated with, endorsed by, or
> supported by Proton AG. "Proton" and "Proton Drive" are trademarks of Proton AG, used here only
> to describe what the software talks to. Use a test account first, and read
> [`docs/RECOVERY.md`](docs/RECOVERY.md) before trusting it with data you cannot lose.
## Install
```bash
# Debian / Ubuntu
sudo apt install ./proton-drive-linux_*.deb
# Fedora
sudo dnf install ./proton-drive-linux-*.rpm
# Arch Linux (AUR)
paru -S proton-drive-for-linux # or -git / -bin
```
Packages for each release are attached to the [latest
release](https://github.com/narrrl/proton-drive-linux/releases/latest); see
[Installation & Packages](#installation--packages) for details and
[Building from Source](#building-from-source) to compile it yourself.
```bash
pdfs login # sign in (SRP + 2FA, credentials go to the system keyring)
pdfs status # mount state, queued writes, cache usage
pdfs-app # the desktop app
```
## Contents
- [Features](#features) · [Quick search](#quick-search-and-media-opening) · [Opening files](#choosing-how-files-open)
- [Selective sync](#selective-sync-pdfsignore) · [Diagnostics](#diagnostics--maintenance) · [Scripting](#scripting---json)
- [Performance](#performance--caching) · [Filesystem safety](#filesystem-safety) · [Screenshots](#screenshots)
- [Prerequisites](#prerequisites) · [Building](#building-from-source) · [Packages](#installation--packages) · [Releases](#automated-releases-cicd)
**Documentation:** [Architecture](docs/ARCHITECTURE.md) · [Testing](docs/TESTING.md) ·
[Recovery](docs/RECOVERY.md) · [Development](docs/DEVELOPMENT.md) · [Translating](docs/TRANSLATING.md) ·
[Known issues](docs/BUGS.md) · [Changelog](docs/CHANGELOG.md) ·
[Contributing](CONTRIBUTING.md) · [Security policy](SECURITY.md)
> to describe what the software talks to. Try it with data you have another copy of first, and
> read [what survives a lost machine](docs/RECOVERY.md) before you rely on it.
## Features
- **Files-on-Demand (FUSE)**: Mount your Proton Drive as a virtual filesystem where files are downloaded only when opened, utilizing smart block-level caching and disk-backed writes.
- **Command-Line Interface (CLI)**: Manage your drive, authenticate, and monitor sync status directly from the terminal.
- **Non-Blocking GTK4 Desktop App**: Browse files, manage pins, configure options, and monitor active transfers through a modern, native GUI with a fully non-blocking asynchronous main loop.
- **File Thumbnails**: Browse locally cached image previews in Files, Shared, and Trash, including embedded previews from camera RAW files. Build a whole folder tree from the Files toolbar and resize the grid from the status bar.
- **Shared Links & Invites**: Browse files shared with you by other users, and view/manage your own public shared links directly in the GUI.
- **Local Backup (Computers)**: Sync and back up local directories (like Downloads, Documents, Pictures, etc.) directly to your Proton Drive account.
- **Locations**: One page (and `pdfs locations`) listing every place Proton Drive occupies on this machine — the main mount plus each backed-up folder — with its mode, sync state, and whether it is read-only. Switching a folder between a full local copy and on-demand happens here.
- **System Tray Integration**: Background indicator for status monitoring, quick actions, and fast sync controls.
- **Unified Search Launcher (HUD)**: A resident Google Drive-style launcher (`pdfs-prompt`) that searches Proton Drive and local files together, ranks the best matches, tolerates abbreviations and typos, and is ideal for a system-wide hotkey. It can also present the same search in your own launcher (`--dmenu`) or, searching live as you type, in `fzf` (`--fzf`).
- **Secure Credential Storage**: Integrates with the system Secret Service (GNOME Keyring, KWallet, etc.) with smart in-memory credential caching to avoid UI thread blockages.
- **Proton Photos Support**: Access your Proton Photos timeline, view thumbnails, and download backed-up media natively (available in the GUI as a navigation tab and via the CLI).
- **File Version History**: Every revision Proton Drive still holds for a file, from the browser's details pane (**Versions**) or `pdfs versions list|restore|save|rm`. Restoring is server-side — no re-upload — and an old version can be written out to a local file without touching the live one.
- **Photo Favourites**: Star a photo in the lightbox, filter the gallery to favourites, or use `pdfs favorite <uid>` / `pdfs photos --favorites`.
- **Photo Albums**: Browse your albums — including the ones shared with you — from the Albums view of the Photos page, or with `pdfs albums` / `pdfs album <uid>`. Album contents open in the same gallery as the timeline.
- **Places**: The Places view of the Photos page groups photos by the town they were taken in, using the location stored with each photo, as cards or on a world map. Towns are looked up offline in [GeoNames](https://www.geonames.org/) data (CC BY 4.0), and the map is drawn offline from [Natural Earth](https://www.naturalearthdata.com/) land outlines and country borders (public domain). Each town is marked with one of its photos, and zoomed in, every photo sits where it was taken. A street map from [OpenFreeMap](https://openfreemap.org/) (© [OpenStreetMap](https://www.openstreetmap.org/copyright) contributors) can be turned on under Appearance; it is off by default because the tile server learns which parts of the world you look at. Install `iso-codes` for country names in your language.
- **Human Verification (CAPTCHA) Recovery**: Detects sign-in gates (VPN/new IP challenges) and launches an embedded `WebKitWebView` dialog to safely complete the challenge, transparently retrying authentication with the earned token.
- **Selective Sync (`.pdfsignore`)**: Keep build trees, dependency directories, and editor leftovers out of synced folders using gitignore-style rules.
- **Data-Safe Offline Writes**: Durable scratch/staging files and a transactional pending queue preserve acknowledged writes across network failures and restarts.
**Files**
## Quick Search and Media Opening
- **Your Drive as a folder.** `~/ProtonDrive` works with every program. Files download when they
are opened, a block at a time, so a video starts playing and a large archive can be read
without fetching all of it.
- **Available offline.** Keep chosen files and folders fully downloaded; everything else is
cached within a size limit you set.
- **Synced folders.** Back up folders such as `~/Documents` under your computer in Proton Drive,
as a full two-way copy or online only. Gitignore-style `.pdfsignore` rules, conflict copies
instead of overwrites, pause and bandwidth limits.
- **Safe writes.** Changes are staged on disk and queued before they are acknowledged, so they
survive a lost connection, a crash or a reboot and upload when they can.
`pdfs-prompt` searches Drive metadata and the local home-directory index in one request. Results are ranked together rather than split by source. Matching is case-insensitive and supports exact names, prefixes, parent paths, multiple terms, punctuation-separated words, ordered abbreviations, common typos, and adjacent transpositions.
**Desktop app**
The prompt stays resident after its first launch, so invoking the shortcut again reuses and resets the existing window. Drive folders open from the mount. Audio and video results also open through FUSE, allowing media players to seek and request ranges without downloading the entire file first; ordinary Drive files are materialized into the local cache before opening.
- **Browse** My files, Shared with me, Shared by me, Computers and Trash, with thumbnails
(including camera RAW), drag and drop, details and **version history**.
- **Photos.** Timeline with favorites, albums, and Places on a map; RAW+JPEG pairs shown as one
photo; duplicate finder; slideshow; **import from Google Photos** (Takeout).
- **Sharing.** Invite people as viewer, editor or admin, create public links with password and
expiry, accept invitations.
- **Sync overview.** What is uploading, what is waiting, what failed and why, and a history of
what happened.
- **Tray icon** with status, pause and quick actions, and a **search launcher** (`pdfs-prompt`)
that searches Proton Drive and your home folder together, as its own window, through fuzzel,
rofi or wofi, or live in `fzf`.
- **Restore a computer.** Continue a backup on a new machine and bring its folders back in one
step.
- **Sign-in** with two-factor authentication and CAPTCHA; the session is kept in the system
keyring.
- Available in English, German, Spanish, French, Italian, Dutch, Polish and Brazilian
Portuguese.
### Using your own launcher (fuzzel, rofi, wofi, …)
**Command line**
`pdfs-prompt --dmenu` runs the same search through an external dmenu-style
launcher instead of the built-in window, so the HUD matches the rest of a
tiling-WM desktop:
- About 55 `pdfs` commands cover files, sync, sharing, Photos, Trash, versions and diagnostics.
Query commands print JSON with `--json`.
```bash
pdfs-prompt --dmenu # fuzzel/rofi/wofi/tofi/bemenu/dmenu, whichever is installed
pdfs-prompt --dmenu --menu 'fuzzel --dmenu --width 60'
pdfs-prompt --dmenu --query invoice # skip straight to a search
```
## Install
A launcher filters a fixed list — it cannot ask for a new one per keystroke — so
searching takes two steps, and the prompt says which step you are on:
- `Search Drive ›` lists your pinned files. Type anything that isn't one of them
and press Enter to search for it.
- `Drive: invoice ›` lists the results. Enter opens one; typing something else
and pressing Enter searches again. Escape closes.
`--query` skips the first step. fuzzel and rofi also get file-type icons.
To make an existing keybinding use it without re-binding, and to pin the
launcher command, set them in `config.json`:
```json
{
"prompt": {
"mode": "dmenu",
"menu": ["fuzzel", "--dmenu", "--width", "60"],
"menu_limit": 50
}
}
```
`--gtk` overrides `"mode": "dmenu"` for one invocation. A `{prompt}` token
anywhere in `menu` is replaced by the prompt text; without one, the launcher's
own prompt flag is appended.
### Live launcher search (fzf)
The two-step above exists because no dmenu-style launcher can call back for new
results per keystroke. `fzf` can, so `pdfs-prompt --fzf` searches as you type,
exactly like the built-in window:
```bash
pdfs-prompt --fzf # pinned files, then live results as you type
pdfs-prompt --fzf --query invoice # start on a search
```
Every keystroke re-queries the daemon (debounced, like the GTK prompt) and fzf's
own matcher is switched off, so the ordering you see is the daemon's ranking —
abbreviations and typos included, which a launcher filtering a fixed list cannot
do. Enter opens; Escape closes.
fzf is a terminal program, so when it is started without one — from a WM
keybinding — it spawns a terminal for itself: foot, ghostty, kitty, alacritty,
wezterm or xterm, whichever is installed. Each is launched with the app-id/class
`pdfs-prompt`, so one window rule floats it whichever terminal you have. In
Hyprland:
```
windowrulev2 = float, class:^(pdfs-prompt)$
windowrulev2 = size 900 500, class:^(pdfs-prompt)$
windowrulev2 = center, class:^(pdfs-prompt)$
bind = SUPER, space, exec, pdfs-prompt --fzf
```
Set it as the default front end, and pick the terminal explicitly, in
`config.json`:
```json
{
"prompt": {
"mode": "fzf",
"terminal": ["foot", "--app-id=pdfs-prompt", "--"],
"menu_limit": 50
}
}
```
A `{cmd}` token anywhere in `terminal` is replaced by the command to run;
without one it is appended, which is what every detected terminal expects.
## Choosing How Files Open
By default every result is handed to `xdg-open`. An `open_with` block in
`config.json` overrides that per file type — for instance opening text in
Neovim inside Alacritty, while everything else still goes to the desktop:
```json
{
"open_with": {
"terminal": ["alacritty", "-e"],
"rules": [
{ "match": ["@text"], "command": ["nvim"], "terminal": true },
{ "match": ["*.png", "*.jpg"], "command": ["imv"] }
]
}
}
```
- `match` takes file-name globs (`*.md`, `notes-*.txt`) or the classes `@dir`,
`@text`, `@document`, `@image`, `@media`, `@any`. First matching rule wins.
- `command` is argv, not a shell line. A `{}` token is replaced by the path;
without one the path is appended. `$VAR` tokens expand from the environment,
so `["$EDITOR"]` follows your editor.
- `"terminal": true` wraps the command in `terminal`. When that is unset, the
terminal comes from `$PDFS_TERMINAL`, then `$TERMINAL`, then the first known
emulator on `PATH`; a bare name like `TERMINAL=alacritty` gains the right
"run this" flag automatically.
- `"default": [...]` replaces `xdg-open` for everything unmatched.
The rules apply to the prompt (both front ends) and to the GTK browser, so a
file opens the same way wherever it was picked. Patterns are matched against the
Drive name — a downloaded file is stored in the cache under its content hash,
which says nothing about its type, so matching that would send everything to
`xdg-open`.
## Selective Sync (`.pdfsignore`)
Two-way synced folders skip paths matched by ignore rules, so syncing a project
directory does not upload `node_modules/`, `target/`, or `.git/`.
Rules come from two places, and both apply:
1. **Per folder** — a `.pdfsignore` file at the root of the synced folder
(`.protonignore` also works). Gitignore syntax, including negation:
```gitignore
# everything build-related
build/
*.log
# ...except this one
!important.log
```
2. **Globally** — an `ignore_patterns` list in `config.json`, applied to every
synced folder. When unset, sensible defaults apply: `.git/`, `.hg/`, `.svn/`,
`node_modules/`, `target/`, `.venv/`, `__pycache__/`, `*~`, `*.swp`, `*.tmp`,
`.DS_Store`, and `Thumbs.db`.
```json
{
"ignore_patterns": ["node_modules/", "target/", "*.iso"]
}
```
Set it to `[]` to opt out of the defaults entirely.
Rules are re-read at the start of every sync pass, so edits take effect on the
next pass without restarting the daemon.
**Ignoring is never destructive.** If a rule starts matching a file that was
already synced, its copy on Drive is left untouched — the file simply stops
being tracked. Removing the rule later picks the existing remote file back up
rather than re-uploading it.
## Diagnostics & Maintenance
When something looks wrong, `pdfs diagnose` checks the installation and prints a
report. It runs without a daemon on purpose — the state worth diagnosing is
usually the state where the daemon will not start:
```console
$ pdfs diagnose
Paths
[ok ] state dir: /home/you/.local/state/proton-drive-linux
[ok ] database: /home/you/.local/state/proton-drive-linux/cache.db (170.3 MiB)
Account
[ok ] keyring session: you@proton.me
Daemon
[ok ] daemon responding
[ok ] mounted at: /home/you/ProtonDrive
[ok ] queued writes: none
No problems found.
```
It exits non-zero if any check fails, so it works in a health-check script.
For the local metadata database and content cache:
| Command | What it does |
| Distribution | Package |
|---|---|
| `pdfs cache inspect` | Database size, reclaimable space, per-table row counts, cache usage against budget |
| `pdfs cache inspect --deep` | Also runs SQLite's integrity check — reads every page, so it is slow on a large database |
| `pdfs cache vacuum` | Checkpoints the write-ahead log and compacts the database |
| `pdfs cache clear` | Deletes cached file content, keeping pinned files |
| Debian, Ubuntu 24.04 or newer | `.deb` from the [latest release](https://github.com/narrrl/proton-drive-linux/releases/latest) |
| Fedora 44 or newer | `.rpm` from the [latest release](https://github.com/narrrl/proton-drive-linux/releases/latest) |
| Arch Linux | [`proton-drive-for-linux`](https://aur.archlinux.org/packages/proton-drive-for-linux) (also `-bin` and `-git`) in the AUR |
| Anything else | [Build from source](docs/INSTALL.md#build-from-source) |
`vacuum` takes a write lock for its duration and needs room for a second copy of
the database while it runs, so it is a deliberate operation rather than
something the daemon does on a timer.
Before a release, run the real-kernel acceptance suite against a dedicated test
account and mount. See [`docs/TESTING.md`](docs/TESTING.md); it includes optional
two-mount convergence checks and confines destructive operations to a fresh
test directory.
## Scripting (`--json`)
Query commands accept a global `--json` flag and emit machine-readable output:
```console
$ pdfs --json sync list | jq '.items[] | select(.state != "idle") | .local_path'
$ pdfs --json cache inspect | jq '.db_reclaimable_bytes'
$ pdfs --json status | jq -r '.mount.mountpoint'
```bash
sudo apt install ./proton-drive-linux_*.deb # Debian, Ubuntu
sudo dnf install ./proton-drive-linux-*.rpm # Fedora
paru -S proton-drive-for-linux # Arch
```
Supported on `status`, `ls`, `pins`, `sync list`, `devices list`, `transfers`,
`activity`, and `cache inspect`. Commands that perform an action keep their
human output — a script that needs to know whether one worked has the exit code.
Then open **Proton Drive** from your applications menu and sign in. The app starts the background
service, and your files appear in `~/ProtonDrive`.
Two things worth relying on:
Signing in from a terminal instead? `pdfs login` stores the session but does not start the
service. Enable it once:
- **The payload is unwrapped.** Output is `{"items": […]}`, not the daemon's
internal `{"SyncFolders": {"items": […]}}`, so scripts never name an internal
variant.
- **Errors still fail.** A daemon-side error prints its JSON body (with a
machine-readable `kind`) on stdout *and* exits non-zero, so `set -e` and
`if pdfs …` behave as expected rather than treating a failed lookup as
success.
```bash
pdfs login
systemctl --user enable --now proton-drive.service
pdfs status
```
## Performance & Caching
[docs/INSTALL.md](docs/INSTALL.md) has the requirements, optional extras (tray on GNOME, RAW and
video thumbnails), headless use and uninstalling.
The client includes several optimizations designed for high efficiency, a low memory footprint, and a responsive user experience:
## Documentation
- **On-Demand Block Cache**: Files are read in fixed 4 MiB blocks. For unpinned files, the client fetches and caches only the blocks containing the requested byte ranges. This enables fast sequential and sparse reads (e.g. media streaming or metadata scanning) without downloading entire files.
- **Disk-Backed Writes**: Large file writes are staged on disk in temporary scratch files (rather than fully buffered in RAM) and track modified byte intervals. Only the unedited remote segments are fetched at commit time, keeping memory usage minimal.
- **Non-Blocking GTK4 Loop**: To prevent UI freezes, all synchronous D-Bus credential checks, control socket requests, and cache usage calculations are offloaded to background worker threads or fetched asynchronously.
- **Flicker-Free UI Rendering**: The GTK4 application performs differential rendering of the pins list, only modifying list rows when the list content changes, preserving the user's scroll position and widget focus.
- **Durable Staging**: Scratch metadata and staged writes are synced and atomically published before an upload is acknowledged locally. Rapid revisions supersede pending work transactionally.
## Filesystem Safety
Version 1.0 strengthens the boundaries around local-only data and destructive reconciliation:
- Truncate composes with queued revisions, preserving the authoritative prefix and correct zero-filled growth.
- Combined move-and-rename operations are queued durably and tolerate partially completed remote state.
- Session tombstones prevent successfully unlinked entries from reappearing through eventually consistent listings.
- Names are valid UTF-8 and no longer than Linux's 255-byte component limit.
- Incomplete mirror scans are non-destructive, and total-wipe protection covers every non-empty baseline.
- Failed conflict preservation, local deletion, or staging publication does not settle a new baseline or discard the only local copy.
- State, cache, and config directories must be real, current-user-owned `0700` directories; control sockets are `0600`.
| | |
|---|---|
| [User guide](docs/USER_GUIDE.md) | Everything the app, tray and launcher do |
| [Command-line reference](docs/CLI.md) | Every `pdfs` command and scripting with `--json` |
| [Configuration](docs/CONFIGURATION.md) | `config.json`, file locations, environment variables, the service |
| [Troubleshooting](docs/TROUBLESHOOTING.md) | Diagnosing problems and fixing common ones |
| [Recovering on a new computer](docs/RECOVERY.md) | Getting your synced folders back after a reinstall or a lost machine |
| [Changelog](docs/CHANGELOG.md) | What changed in each release |
For contributors: [development guide](docs/DEVELOPMENT.md), [architecture](docs/ARCHITECTURE.md),
[testing](docs/TESTING.md), [translating](docs/TRANSLATING.md), [roadmap](docs/ROADMAP.md) and the
[bug ledger](docs/BUGS.md). The [documentation index](docs/README.md) lists everything.
## Screenshots
### GUI Application & Launcher
<table>
<tr>
<td align="center" width="50%"><img src="images/login.png" alt="Login Screen" width="100%"><br><sub><b>Login Screen</b></sub></td>
<td align="center" width="50%"><img src="images/files.png" alt="Files Browser" width="100%"><br><sub><b>Files Browser</b></sub></td>
<td align="center" width="50%"><img src="images/files.png" alt="My files" width="100%"><br><sub><b>My files</b></sub></td>
<td align="center" width="50%"><img src="images/photos.png" alt="Photos timeline" width="100%"><br><sub><b>Photos</b></sub></td>
</tr>
<tr>
<td align="center" width="50%"><img src="images/shared.png" alt="Shared Links" width="100%"><br><sub><b>Shared Links</b></sub></td>
<td align="center" width="50%"><img src="images/shared_with_me.png" alt="Shared with me" width="100%"><br><sub><b>Shared with me</b></sub></td>
<td align="center" width="50%"><img src="images/photos_map.png" alt="Photos places map" width="100%"><br><sub><b>Places</b></sub></td>
<td align="center" width="50%"><img src="images/computers.png" alt="Computers" width="100%"><br><sub><b>Computers</b></sub></td>
</tr>
<tr>
<td align="center" width="50%"><img src="images/restore_devices.png" alt="Local Backups (Computers)" width="100%"><br><sub><b>Local Backups (Computers)</b></sub></td>
<td align="center" width="50%"><img src="images/photos.png" alt="Photos Timeline" width="100%"><br><sub><b>Photos Timeline</b></sub></td>
<td align="center" width="50%"><img src="images/sync.png" alt="Sync folders" width="100%"><br><sub><b>Sync</b></sub></td>
<td align="center" width="50%"><img src="images/conflicts.png" alt="Sync history showing resolved conflicts" width="100%"><br><sub><b>Sync history</b></sub></td>
</tr>
<tr>
<td align="center" width="50%"><img src="images/prompt_gtk.png" alt="Search Launcher Prompt (GTK)" width="100%"><br><sub><b>Search Launcher Prompt (GTK)</b></sub></td>
<td align="center" width="50%"><img src="images/prompt_fzf.png" alt="Search Launcher Prompt (fzf)" width="100%"><br><sub><b>Search Launcher Prompt (fzf)</b></sub></td>
<td align="center" width="50%"><img src="images/prompt_gtk.png" alt="Search launcher" width="100%"><br><sub><b>Search launcher</b></sub></td>
<td align="center" width="50%"><img src="images/prompt_fzf.png" alt="Search launcher in fzf" width="100%"><br><sub><b>Search launcher in fzf</b></sub></td>
</tr>
<tr>
<td align="center" width="50%"><img src="images/activity.png" alt="Activity" width="100%"><br><sub><b>Activity</b></sub></td>
<td align="center" width="50%"><img src="images/settings.png" alt="Settings" width="100%"><br><sub><b>Settings</b></sub></td>
<td align="center" width="50%"><img src="images/login.png" alt="Sign in" width="100%"><br><sub><b>Sign in</b></sub></td>
<td></td>
</tr>
</table>
---
## How it works
## Prerequisites
A background service (`proton-drive.service`, the `pdfs daemon` command) holds your session,
mounts `~/ProtonDrive` through FUSE, runs the upload queue and the sync engine, and follows
Proton's event stream for changes made elsewhere. The app, the tray, the launcher and the CLI are
clients of that service over a local socket. Everything is end-to-end encrypted with your Proton
keys before it leaves the machine, through a Rust implementation of Proton's Drive SDK.
To compile the application from source or run the built binaries, ensure you have the following system libraries installed on your distribution:
### Ubuntu / Debian (24.04+)
```bash
sudo apt-get update
sudo apt-get install -y \
pkg-config \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev \
libimage-exiftool-perl
```
### Arch Linux
```bash
sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus webkitgtk-6.0 perl-image-exiftool
```
### Fedora (44+)
```bash
sudo dnf install -y \
pkgconf-pkg-config fuse3-devel gtk4-devel libadwaita-devel \
webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel \
perl-Image-ExifTool cargo rust
```
Runtime extras (pick your desktop):
```bash
# GNOME — keyring + tray (AppIndicator)
sudo dnf install -y gnome-keyring gnome-shell-extension-appindicator xdg-utils
# KDE Plasma — KWallet (Secret Service); tray works via built-in SNI
sudo dnf install -y kwallet xdg-utils
```
---
## Building from Source
Ensure you have Rust and Cargo installed (minimum supported Rust version is 1.96).
1. Clone the repository and navigate into the project directory:
```bash
git clone https://github.com/narrrl/proton-drive-linux.git
cd proton-drive-linux
```
2. Build the workspace in release mode:
```bash
cargo build --release --locked
```
The compiled binaries will be available under `target/release/`:
- `pdfs`: The CLI utility.
- `pdfs-app`: The GTK4 application.
- `pdfs-tray`: The tray status notifier.
- `pdfs-prompt`: The launcher prompt for quick HUD search.
---
## Installation & Packages
### 1. Debian / Ubuntu (.deb)
Install the debian package via `dpkg` or `apt`:
```bash
sudo apt install ./proton-drive-linux_*.deb
```
### 2. Arch Linux (AUR)
Three AUR packages cover the usual choices; they conflict with each other, so
install one:
| Package | Contents |
| --- | --- |
| [`proton-drive-for-linux`](https://aur.archlinux.org/packages/proton-drive-for-linux) | builds the latest tagged release from source |
| [`proton-drive-for-linux-git`](https://aur.archlinux.org/packages/proton-drive-for-linux-git) | builds `main` from git |
| [`proton-drive-for-linux-bin`](https://aur.archlinux.org/packages/proton-drive-for-linux-bin) | installs the release binaries, no compile |
```bash
paru -S proton-drive-for-linux # or any AUR helper
```
The package sources live in `packaging/aur/`; see
[`packaging/aur/README.md`](packaging/aur/README.md) for how they are updated and
published. A local `PKGBUILD` that compiles the working tree in place is still
available for development builds:
```bash
cd packaging && makepkg -fi
```
### 3. Fedora (local RPM)
A local `.spec` is available under `packaging/`. From the repository root:
```bash
sudo dnf install -y rpm-build
rpmbuild -bb packaging/proton-drive-linux.spec \
--define "git_dir $PWD" \
--define "_rpmdir $PWD/packaging/out" \
--define "_builddir $PWD/packaging/build" \
--define "_sourcedir $PWD" \
--define "_specdir $PWD/packaging" \
--define "_srcrpmdir $PWD/packaging/out"
sudo dnf install packaging/out/x86_64/proton-drive-linux-*.rpm
```
---
## Automated Releases (CI/CD)
This project has a GitHub Actions CI workflow configured under `.github/workflows/release.yml`.
### How it works:
1. **Triggers**:
- Pushing a git tag matching `v*` (e.g. `git tag v0.1.0 && git push origin v0.1.0`).
- Manual runs via the **Actions** tab in GitHub (**workflow_dispatch**).
2. **Quality Gates and Build Process**:
- For tagged builds, verifies that the tag, workspace version, and `packaging/PKGBUILD` version agree.
- Runs `cargo fmt`, Clippy with warnings denied, the locked workspace test suite, and the account-free FUSE acceptance contract.
- Spawns an Ubuntu runner and installs GTK4, Libadwaita, FUSE3, and Secret Service packages.
- Sets up the Rust compiler and caches build targets to speed up runs.
- Compiles the workspace members in release mode.
3. **Artifact Packaging**:
- Generates a `.tar.gz` containing the raw binaries (`pdfs`, `pdfs-app`, `pdfs-tray`, `pdfs-prompt`).
- Packs them into Debian (`.deb`) and Fedora (`.rpm`) packages.
- Includes the systemd user service and tray autostart entry in the Debian package.
4. **Publishing**:
- Creates a GitHub Release matching the pushed tag and uploads the `.deb`, `.rpm`, and `.tar.gz` packages as release assets.
- For manual runs, compiles and exposes the packages as workflow run artifacts for testing.
---
Files you open are decrypted into a local cache so programs can read them. What that means for
the security of the machine is spelled out in the
[threat model](docs/ARCHITECTURE.md#8-threat-model-what-this-client-writes-to-disk-in-plaintext).
## Contributing
Bug reports, packaging fixes, and patches are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for
the workflow and the quality gates CI enforces, and [SECURITY.md](SECURITY.md) for reporting a
vulnerability privately.
The app, tray and search prompt are available in English, German, Spanish, French, Italian, Dutch,
Polish and Brazilian Portuguese. [docs/TRANSLATING.md](docs/TRANSLATING.md) explains how to improve
a translation or add a language.
Bug reports, translations and patches are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the
workflow and the checks CI runs, and [SECURITY.md](SECURITY.md) to report a vulnerability
privately.
## License
Released under the [MIT License](LICENSE).
The Places map uses [GeoNames](https://www.geonames.org/) data (CC BY 4.0) and
[Natural Earth](https://www.naturalearthdata.com/) outlines (public domain). The optional street
map comes from [OpenFreeMap](https://openfreemap.org/), © [OpenStreetMap](https://www.openstreetmap.org/copyright)
contributors.
Proton Drive is a service of Proton AG. This project is an independent client and carries no
affiliation with or endorsement from Proton AG.
+5
View File
@@ -30,5 +30,10 @@ the interesting boundaries are:
- **Cryptography.** Key derivation, signature handling, and anything that could publish plaintext
or the wrong key material to Proton.
What the client deliberately keeps in plaintext on disk, and why, is documented in the
[threat model](docs/ARCHITECTURE.md#8-threat-model-what-this-client-writes-to-disk-in-plaintext).
Behaviour described there is known and not a vulnerability by itself; a way to make it worse
than described is.
Out of scope: vulnerabilities in Proton's own service (report those to Proton AG), and issues that
require an attacker who already has your user account on the machine.
+86 -19
View File
@@ -20,6 +20,39 @@ use pdfs_core::control::{
use pdfs_core::db::Db;
use pdfs_core::service;
/// `println!` for this binary, shadowing std's, which panics when stdout is
/// closed: `pdfs ls | head` ended in a "Broken pipe" panic once `head` had its
/// lines. This one hands the failure to [`stdout_written`] instead.
///
/// Restoring the default `SIGPIPE` would also stop the panic, but the same
/// binary runs the daemon, which must not die when a socket's peer goes away.
macro_rules! println {
($($arg:tt)*) => {{
use std::io::Write as _;
$crate::stdout_written(writeln!(std::io::stdout(), $($arg)*))
}};
}
/// [`println!`] without the newline.
macro_rules! print {
($($arg:tt)*) => {{
use std::io::Write as _;
$crate::stdout_written(write!(std::io::stdout(), $($arg)*))
}};
}
/// Exit on a failed write to stdout. A closed pipe means the reader has all it
/// wanted, so that exit is a success.
fn stdout_written(result: std::io::Result<()>) {
if let Err(error) = result {
if error.kind() == std::io::ErrorKind::BrokenPipe {
std::process::exit(0);
}
eprintln!("Error: writing to stdout: {error}");
std::process::exit(1);
}
}
#[derive(Parser)]
#[command(
name = "pdfs",
@@ -30,7 +63,7 @@ struct Cli {
/// Emit machine-readable JSON instead of formatted text.
///
/// Applies to the query commands (`status`, `ls`, `pins`, `sync list`,
/// `devices list`, `locations`, `transfers`, `activity`, `cache inspect`). Commands that
/// `devices list`, `locations`, `transfers`, `activity`, `trash`, `cache inspect`). Commands that
/// perform an action keep their human output — a script that needs to know
/// whether one succeeded has the exit code.
#[arg(long, global = true)]
@@ -285,26 +318,34 @@ enum Command {
},
/// Rename a file or folder via the running daemon.
Rename {
/// File/folder path, inside the mountpoint or relative to it.
/// File/folder path, inside My files or an on-demand synced folder, or
/// relative to the mountpoint.
path: PathBuf,
/// New name (a single path component).
new_name: String,
},
/// Move a file or folder into another folder via the running daemon.
/// Move files or folders into another folder via the running daemon.
///
/// The paths may be in different locations: My files and any synced
/// folder, on-demand or mirrored. The move happens on Proton Drive, so
/// nothing is downloaded or uploaded again, unlike `mv` between two
/// locations. The last path is the destination folder.
Move {
/// File/folder path, inside the mountpoint or relative to it.
path: PathBuf,
/// Destination folder path, inside the mountpoint or relative to it.
new_parent: PathBuf,
/// Files/folders to move, then the destination folder: each inside a
/// location, or relative to the mountpoint.
#[arg(required = true, num_args = 2.., value_name = "PATH")]
paths: Vec<PathBuf>,
},
/// Trash a file or folder via the running daemon.
Rm {
/// File/folder path, inside the mountpoint or relative to it.
/// File/folder path, inside My files or an on-demand synced folder, or
/// relative to the mountpoint.
path: PathBuf,
},
/// Create a folder via the running daemon.
Mkdir {
/// Parent folder path, inside the mountpoint or relative to it.
/// Parent folder path, inside My files or an on-demand synced folder,
/// or relative to the mountpoint.
parent: PathBuf,
/// New folder name (a single path component).
name: String,
@@ -800,7 +841,10 @@ fn main() -> Result<()> {
scope,
} => cmd_search(query, limit, scope),
Command::Rename { path, new_name } => cmd_rename(path, new_name),
Command::Move { path, new_parent } => cmd_move(path, new_parent),
Command::Move { mut paths } => {
let new_parent = paths.pop().expect("clap requires at least two paths");
cmd_move(paths, new_parent)
}
Command::Rm { path } => cmd_rm(path),
Command::Mkdir { parent, name } => cmd_mkdir(parent, name),
Command::Upload { sources, parent } => cmd_upload(sources, parent),
@@ -2439,14 +2483,33 @@ fn cmd_rename(path: PathBuf, new_name: String) -> Result<()> {
Ok(())
}
fn cmd_move(path: PathBuf, new_parent: PathBuf) -> Result<()> {
match control_request(CtlRequest::Move {
path: path_arg(&path)?,
new_parent: path_arg(&new_parent)?,
})? {
CtlResponse::Ok { message } => println!("{message}"),
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
/// Move each of `sources` into `new_parent`, one request each. A refusal names
/// its source and the rest still move; the command fails if any did not.
fn cmd_move(sources: Vec<PathBuf>, new_parent: PathBuf) -> Result<()> {
let new_parent = path_arg(&new_parent)?;
let mut failed = 0;
for source in &sources {
let outcome = path_arg(source).and_then(|path| {
match control_request(CtlRequest::Move {
path,
new_parent: new_parent.clone(),
})? {
CtlResponse::Ok { message } => Ok(message),
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
}
});
match outcome {
Ok(message) => println!("{message}"),
Err(e) if sources.len() == 1 => return Err(e),
Err(e) => {
eprintln!("{}: {e:#}", source.display());
failed += 1;
}
}
}
if failed > 0 {
bail!("{failed} of {} could not be moved", sources.len());
}
Ok(())
}
@@ -2524,7 +2587,11 @@ fn cmd_diagnostics() -> Result<()> {
}
fn cmd_trash() -> Result<()> {
match control_request(CtlRequest::ListTrash)? {
let response = control_request(CtlRequest::ListTrash)?;
if emit_json(&response)? {
return Ok(());
}
match response {
CtlResponse::Entries { entries } if entries.is_empty() => println!("(trash is empty)"),
CtlResponse::Entries { entries } => {
for e in entries {
+113
View File
@@ -365,6 +365,14 @@ pub struct StagedWrite {
pub based_on: Option<Baseline>,
}
impl StagedWrite {
/// Whether `authored` ranges cover a file of `len` bytes, so the staged
/// bytes are its complete new content. An empty file has nothing to cover.
pub fn covers(authored: &[(u64, u64)], len: u64) -> bool {
len == 0 || authored == [(0, len)]
}
}
/// The identity of a remote revision, as far as we can observe it.
///
/// `revision_id` is the authoritative identity when present: the server assigns a
@@ -730,6 +738,60 @@ impl ContentCache {
Some(buf)
}
/// Write the whole content of `uid` at `(mtime, size)` to `dst` from the
/// cache alone, and report whether that worked.
///
/// A whole-file blob is cloned. Otherwise every block of the revision has to
/// be cached, under its recorded geometry or, failing that, the uniform one
/// the read path would use. Anything short of the full content leaves
/// nothing at `dst`, so a caller falls back to a download with a clean slate.
///
/// This is what lets a mirror folder take a file an on-demand mount has
/// already read, after a move between them, without fetching it again. The
/// trust is the read path's: the same `(mtime, size)` key, and blocks that
/// were validated when they were fetched.
pub fn copy_cached_to(&self, uid: &NodeUid, mtime: i64, size: u64, dst: &Path) -> bool {
if size == 0 {
return false;
}
if let Some(blob) = self.valid_blob(uid, mtime, size) {
if Self::clone_or_copy(&blob, dst).is_ok_and(|len| len == size) {
self.touch(Self::key(uid));
return true;
}
let _ = std::fs::remove_file(dst);
return false;
}
let geometry = self
.block_geometry(uid, mtime, size)
.unwrap_or_else(|| BlockGeometry::uniform(size));
if geometry.size() != size {
return false;
}
let spans = geometry.spans(0, size);
// Checked up front so a file with one block missing costs stats, not a
// copy of every block before it.
if !spans
.iter()
.all(|&span| self.has_block(uid, mtime, size, span))
{
return false;
}
let written = (|| -> Option<()> {
use std::io::Write;
let mut out = std::fs::File::create(dst).ok()?;
for span in spans {
out.write_all(&self.cached_block(uid, mtime, size, span)?)
.ok()?;
}
(out.metadata().ok()?.len() == size).then_some(())
})();
if written.is_none() {
let _ = std::fs::remove_file(dst);
}
written.is_some()
}
/// Store `bytes` as the cached content for `uid`, tagged with `(mtime,
/// size)`. The blob is written to a temp file then renamed so a concurrent
/// reader never sees a partial blob; the meta tag is written last so a
@@ -2002,6 +2064,17 @@ mod tests {
NodeUid::new(VolumeId::from("vol"), LinkId::from(link))
}
#[test]
fn an_empty_write_covers_the_file_and_a_gap_does_not() {
assert!(
StagedWrite::covers(&[], 0),
"an empty file has nothing to cover"
);
assert!(StagedWrite::covers(&[(0, 7)], 7));
assert!(!StagedWrite::covers(&[(0, 3), (4, 7)], 7));
assert!(!StagedWrite::covers(&[], 7));
}
/// A unique temp directory removed on drop; avoids a dev-dependency.
struct TempDir(PathBuf);
impl TempDir {
@@ -2561,6 +2634,46 @@ mod tests {
assert!(c.cached_block(&u, 100, 5000, blk(5000, 0)).is_none());
}
#[test]
fn a_cached_blob_is_copied_out_whole() {
let (c, d) = cache();
let u = uid("blob");
c.store(&u, 7, 5, b"hello").unwrap();
let dst = d.path().join("out");
assert!(c.copy_cached_to(&u, 7, 5, &dst));
assert_eq!(std::fs::read(&dst).unwrap(), b"hello");
// Another revision is not this one.
assert!(!c.copy_cached_to(&u, 8, 5, &d.path().join("stale")));
assert!(!d.path().join("stale").exists());
}
#[test]
fn cached_blocks_are_copied_out_in_order() {
let (c, d) = cache();
let u = uid("blocks");
let size = BLOCK_SIZE + 3;
let first = vec![1_u8; BLOCK_SIZE as usize];
c.store_block(&u, 7, size, blk(size, 0), &first).unwrap();
c.store_block(&u, 7, size, blk(size, 1), b"end").unwrap();
let dst = d.path().join("out");
assert!(c.copy_cached_to(&u, 7, size, &dst));
let bytes = std::fs::read(&dst).unwrap();
assert_eq!(bytes.len() as u64, size);
assert_eq!(&bytes[..BLOCK_SIZE as usize], first.as_slice());
assert_eq!(&bytes[BLOCK_SIZE as usize..], b"end");
}
#[test]
fn a_file_with_a_block_missing_is_not_copied_out() {
let (c, d) = cache();
let u = uid("partial");
let size = BLOCK_SIZE + 3;
c.store_block(&u, 7, size, blk(size, 1), b"end").unwrap();
let dst = d.path().join("out");
assert!(!c.copy_cached_to(&u, 7, size, &dst));
assert!(!dst.exists(), "a partial copy must not be left behind");
}
/// The [`BlockSpan`] the uniform geometry gives block `idx` of a file of
/// `size` bytes — the shape these tests were written against, and the one
/// this client's own uploads still have.
+4 -1
View File
@@ -339,7 +339,10 @@ pub enum Request {
/// single path component (no separators). Replies with [`Response::Ok`].
Rename { path: String, new_name: String },
/// Move a file or folder into a new parent folder. Both `path` and
/// `new_parent` are mountpoint-relative. Replies with [`Response::Ok`].
/// `new_parent` are mountpoint-relative, or both absolute local paths; two
/// absolute paths may lie in different locations (My files, an on-demand
/// or a mirror synced folder), and the move is still made on Proton Drive
/// without transferring content. Replies with [`Response::Ok`].
Move { path: String, new_parent: String },
/// Trash a file or folder. `path` is mountpoint-relative. Replies with
/// [`Response::Ok`].
+249
View File
@@ -0,0 +1,249 @@
/* WARNING: Script requires that SQLITE_DBCONFIG_DEFENSIVE be disabled */
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE sync_state (key TEXT PRIMARY KEY, value TEXT);
INSERT INTO sync_state VALUES('schema_version','35');
CREATE TABLE nodes (
uid TEXT PRIMARY KEY,
parent_uid TEXT,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER,
mtime INTEGER NOT NULL,
revision_hash TEXT,
trashed INTEGER NOT NULL DEFAULT 0
, node_json TEXT, listed INTEGER NOT NULL DEFAULT 0, path TEXT);
INSERT INTO nodes VALUES('vol~root',NULL,'My Files',1,0,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"root"},"parent_uid":null,"kind":"Folder","name":"My Files","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'');
INSERT INTO nodes VALUES('vol~docs','vol~root','Docs',1,0,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"docs"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":"Folder","name":"Docs","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs');
INSERT INTO nodes VALUES('vol~report','vol~docs','report.txt',0,10,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"report"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":20,"active_revision_state":null,"active_revision_id":null,"claimed_size":10,"claimed_modification_time":null,"content_sha1":null}},"name":"report.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/report.txt');
INSERT INTO nodes VALUES('vol~old','vol~docs','old.txt',0,3,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"old"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":13,"active_revision_state":null,"active_revision_id":null,"claimed_size":3,"claimed_modification_time":null,"content_sha1":null}},"name":"old.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/old.txt');
INSERT INTO nodes VALUES('vol~shared','vol~docs','shared.txt',0,4,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"shared"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":14,"active_revision_state":null,"active_revision_id":null,"claimed_size":4,"claimed_modification_time":null,"content_sha1":null}},"name":"shared.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/shared.txt');
INSERT INTO nodes VALUES('local~dir','vol~root','Offline folder',1,0,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"dir"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":"Folder","name":"Offline folder","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',1,'Offline folder');
INSERT INTO nodes VALUES('local~inner','local~dir','inner.txt',0,5,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"inner"},"parent_uid":{"volume_id":"local","link_id":"dir"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":15,"active_revision_state":null,"active_revision_id":null,"claimed_size":5,"claimed_modification_time":null,"content_sha1":null}},"name":"inner.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Offline folder/inner.txt');
INSERT INTO nodes VALUES('local~part','vol~root','movie.part',0,7,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"part"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":17,"active_revision_state":null,"active_revision_id":null,"claimed_size":7,"claimed_modification_time":null,"content_sha1":null}},"name":"movie.part","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'movie.part');
CREATE TABLE cache_entries (
cache_key TEXT PRIMARY KEY,
size_bytes INTEGER,
last_accessed INTEGER,
is_pinned INTEGER NOT NULL DEFAULT 0
, kind TEXT NOT NULL DEFAULT 'blob');
CREATE TABLE pins (
uid TEXT PRIMARY KEY,
path TEXT NOT NULL,
recursive INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE local_files (
id INTEGER PRIMARY KEY,
path TEXT NOT NULL UNIQUE,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER NOT NULL DEFAULT 0,
mtime INTEGER NOT NULL DEFAULT 0,
scan_gen INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE photos (
uid TEXT PRIMARY KEY,
capture_time INTEGER NOT NULL,
name TEXT,
ratio REAL,
thumb_state INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL
, media_type TEXT, kind INTEGER NOT NULL DEFAULT 0, favorite INTEGER NOT NULL DEFAULT 0, content_hash TEXT, main_uid TEXT, group_key TEXT, resolved_at INTEGER, latitude REAL, longitude REAL, place_id INTEGER, similar_hash INTEGER);
CREATE TABLE trash (
uid TEXT PRIMARY KEY,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER NOT NULL DEFAULT 0,
mtime INTEGER NOT NULL DEFAULT 0
, parent_uid TEXT, trashed_at INTEGER);
CREATE TABLE device (
uid TEXT PRIMARY KEY,
share_id TEXT NOT NULL,
root_uid TEXT NOT NULL,
name TEXT NOT NULL,
created INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE sync_folder (
id INTEGER PRIMARY KEY,
local_path TEXT NOT NULL UNIQUE,
remote_uid TEXT NOT NULL,
remote_share_id TEXT NOT NULL,
mode TEXT NOT NULL DEFAULT 'mirror',
state TEXT NOT NULL DEFAULT 'idle',
last_sync INTEGER NOT NULL DEFAULT 0
, pending_mode TEXT);
CREATE TABLE sync_entry (
folder_id INTEGER NOT NULL,
rel_path TEXT NOT NULL,
remote_uid TEXT,
local_mtime INTEGER NOT NULL DEFAULT 0,
local_size INTEGER NOT NULL DEFAULT 0,
remote_hash TEXT,
remote_rev TEXT, local_mtime_ns INTEGER,
PRIMARY KEY (folder_id, rel_path)
);
CREATE TABLE activity (
id INTEGER PRIMARY KEY,
time INTEGER NOT NULL,
kind TEXT NOT NULL,
target TEXT NOT NULL,
detail TEXT NOT NULL DEFAULT '',
ok INTEGER NOT NULL DEFAULT 1
);
CREATE TABLE pending_op (
id INTEGER PRIMARY KEY,
kind TEXT NOT NULL,
uid TEXT NOT NULL,
blob_path TEXT,
meta_json TEXT,
created_at INTEGER NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
next_attempt_at INTEGER NOT NULL DEFAULT 0
, parent_uid TEXT, name TEXT, claimed_at INTEGER NOT NULL DEFAULT 0, access_deferred_since INTEGER NOT NULL DEFAULT 0);
INSERT INTO pending_op VALUES(1,'mkdir','local~dir',NULL,NULL,1700000000000,0,NULL,0,'vol~root','Offline folder',4611686018427387903,0);
INSERT INTO pending_op VALUES(2,'create','local~inner','/blobs/inner','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,0,NULL,0,'local~dir','inner.txt',0,0);
INSERT INTO pending_op VALUES(3,'create','local~part','/blobs/part','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,0,NULL,8000000000000,'vol~root','movie.part',0,0);
INSERT INTO pending_op VALUES(4,'revision','vol~report','/blobs/report','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,2,'network unreachable',1700000900000,NULL,NULL,0,0);
INSERT INTO pending_op VALUES(5,'rename','vol~shared',NULL,'{"original_parent_uid":"vol~docs","original_name":"shared.txt"}',1700000000000,0,NULL,1700000700000,'vol~root','shared-moved.txt',0,1700000100000);
INSERT INTO pending_op VALUES(6,'trash','vol~old',NULL,NULL,1700000000000,0,NULL,0,NULL,NULL,0,0);
CREATE TABLE IF NOT EXISTS 'local_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
INSERT INTO local_fts_data VALUES(1,x'000000');
INSERT INTO local_fts_data VALUES(10,x'00000000000000');
CREATE TABLE IF NOT EXISTS 'local_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'local_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'local_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
INSERT INTO local_fts_config VALUES('version',4);
CREATE TABLE share_access (
root_uid TEXT PRIMARY KEY,
access TEXT NOT NULL
CHECK (access IN ('owner', 'editor', 'viewer', 'unknown'))
);
CREATE TABLE mount (
id INTEGER PRIMARY KEY,
kind TEXT NOT NULL
CHECK (kind IN ('myfiles', 'device', 'shared')),
sync_folder_id INTEGER REFERENCES sync_folder(id) ON DELETE CASCADE,
share_root_uid TEXT,
local_path TEXT NOT NULL DEFAULT '',
root_uid TEXT NOT NULL DEFAULT '',
root_share_id TEXT NOT NULL DEFAULT '',
mode TEXT NOT NULL DEFAULT 'unknown',
access TEXT NOT NULL DEFAULT 'rw'
CHECK (access IN ('rw', 'ro')),
CHECK (
(kind = 'myfiles' AND sync_folder_id IS NULL AND share_root_uid IS NULL)
OR
(kind = 'device' AND sync_folder_id IS NOT NULL AND share_root_uid IS NULL)
OR
(kind = 'shared' AND sync_folder_id IS NULL AND share_root_uid IS NOT NULL)
)
);
CREATE TABLE albums (
uid TEXT PRIMARY KEY,
name TEXT NOT NULL,
photo_count INTEGER NOT NULL DEFAULT 0,
cover_uid TEXT,
last_activity INTEGER,
shared INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL
);
CREATE TABLE album_photos (
album_uid TEXT NOT NULL,
uid TEXT NOT NULL,
capture_time INTEGER NOT NULL,
name TEXT,
media_type TEXT,
kind INTEGER NOT NULL DEFAULT 0,
ratio REAL,
thumb_state INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL,
PRIMARY KEY (album_uid, uid)
);
CREATE TABLE IF NOT EXISTS 'nodes_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
INSERT INTO nodes_fts_data VALUES(1,x'083850');
INSERT INTO nodes_fts_data VALUES(10,x'000000000108080008010101020101030101040101050101060101070101080101');
INSERT INTO nodes_fts_data VALUES(137438953473,x'000000340430206669010204010366696c0102050103696c6501020601036c657301020701036d79200102020103792066010203040808080808');
INSERT INTO nodes_fts_data VALUES(274877906945,x'0000001a0430646f6302080201010201036f6373020803010103040b');
INSERT INTO nodes_fts_data VALUES(412316860417,x'0000008b04302e747803080801010d01032f72650306010106010363732f03060101040103646f630306010102010365706f03080301010801036f637303060101030202727403080501010a0103706f7203080401010901037265700308020101070202742e03080601010b0103732f7203060101050103742e7403080701010c0202787403080901010e040b0a0a0a0b0a0a0b0b0a0a0b');
INSERT INTO nodes_fts_data VALUES(549755813889,x'0000006b04302e747804080501010a01032f6f6c0406010106010363732f04060101040103642e7404080401010902026f63040601010201036c642e04080301010801036f6373040601010302026c640408020101070103732f6f0406010105010374787404080601010b040b0a0a0b090b0a0a0a');
INSERT INTO nodes_fts_data VALUES(687194767361,x'0000008c04302e747805080801010d01032f736805060101060103617265050804010109010363732f05060101040103642e7405080701010c02026f630506010102010365642e05080601010b010368617205080301010801036f63730506010103010372656405080501010a0103732f73050601010502026861050802010107010374787405080901010e040b0a0b0a0b090b0b0a0b0a0a');
INSERT INTO nodes_fts_data VALUES(824633720833,x'00000084043020666f060809010109010364657206080d01010d0103652066060808010108010366666c06080301010302026c6906080401010402026f6c06080a01010a0103696e6506080601010601036c646506080c01010c0202696e06080501010501036e652006080701010701036f666606080201010202026c6406080b01010b040b0b0b0b0a0a0b0b0a0b0b');
INSERT INTO nodes_fts_data VALUES(962072674305,x'000000da043020666f070601010901032e747807080701011601032f696e07060101100103646572070601010d010365206607060101080202722e07080501011403012f070601010e010366666c070601010302026c69070601010402026f6c070601010a0103696e65070601010603016e07080201011101036c6465070601010c0202696e070601010501036e6520070601010703017207080401011302026e6507080301011201036f6666070601010202026c64070601010b0103722e7407080601011502022f69070601010f0103747874070808010117040a0b0a0a0a0a080a09090a090a090a090a0a090b09');
INSERT INTO nodes_fts_data VALUES(1099511627777,x'0000005c04302e706108080701010701036172740808090101090103652e70080806010106010369652e08080501010501036d6f7608080201010201036f766908080301010301037061720808080101080103766965080804010104040b0b0b0b0b0b0b');
CREATE TABLE IF NOT EXISTS 'nodes_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
INSERT INTO nodes_fts_idx VALUES(1,x'',2);
INSERT INTO nodes_fts_idx VALUES(2,x'',2);
INSERT INTO nodes_fts_idx VALUES(3,x'',2);
INSERT INTO nodes_fts_idx VALUES(4,x'',2);
INSERT INTO nodes_fts_idx VALUES(5,x'',2);
INSERT INTO nodes_fts_idx VALUES(6,x'',2);
INSERT INTO nodes_fts_idx VALUES(7,x'',2);
INSERT INTO nodes_fts_idx VALUES(8,x'',2);
CREATE TABLE IF NOT EXISTS 'nodes_fts_content'(id INTEGER PRIMARY KEY, c0, c1);
INSERT INTO nodes_fts_content VALUES(1,'My Files','');
INSERT INTO nodes_fts_content VALUES(2,'Docs','Docs');
INSERT INTO nodes_fts_content VALUES(3,'report.txt','Docs/report.txt');
INSERT INTO nodes_fts_content VALUES(4,'old.txt','Docs/old.txt');
INSERT INTO nodes_fts_content VALUES(5,'shared.txt','Docs/shared.txt');
INSERT INTO nodes_fts_content VALUES(6,'Offline folder','Offline folder');
INSERT INTO nodes_fts_content VALUES(7,'inner.txt','Offline folder/inner.txt');
INSERT INTO nodes_fts_content VALUES(8,'movie.part','movie.part');
CREATE TABLE IF NOT EXISTS 'nodes_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
INSERT INTO nodes_fts_docsize VALUES(1,x'0600');
INSERT INTO nodes_fts_docsize VALUES(2,x'0202');
INSERT INTO nodes_fts_docsize VALUES(3,x'080d');
INSERT INTO nodes_fts_docsize VALUES(4,x'050a');
INSERT INTO nodes_fts_docsize VALUES(5,x'080d');
INSERT INTO nodes_fts_docsize VALUES(6,x'0c0c');
INSERT INTO nodes_fts_docsize VALUES(7,x'0716');
INSERT INTO nodes_fts_docsize VALUES(8,x'0808');
CREATE TABLE IF NOT EXISTS 'nodes_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
INSERT INTO nodes_fts_config VALUES('version',4);
CREATE TABLE own_sealed_rev (
uid TEXT PRIMARY KEY,
revision_id TEXT NOT NULL,
sealed_at INTEGER NOT NULL
);
INSERT INTO own_sealed_rev VALUES('vol~report','rev-1',1700000000000);
PRAGMA writable_schema=ON;
INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)VALUES('table','local_fts','local_fts',0,'CREATE VIRTUAL TABLE local_fts USING fts5(
name, path, content=''local_files'', content_rowid=''id'', tokenize=''trigram''
)');
INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)VALUES('table','nodes_fts','nodes_fts',0,'CREATE VIRTUAL TABLE nodes_fts USING fts5(
name, path, tokenize=''trigram''
)');
CREATE TRIGGER mount_sync_folder_insert
AFTER INSERT ON sync_folder
BEGIN
INSERT OR IGNORE INTO mount (kind, sync_folder_id)
VALUES ('device', NEW.id);
END;
CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_photos_seq ON photos(seq);
CREATE INDEX pending_op_uid ON pending_op(uid);
CREATE INDEX pending_op_parent ON pending_op(parent_uid);
CREATE INDEX cache_entries_lru ON cache_entries(kind, last_accessed);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_local_files_name_nocase
ON local_files(name COLLATE NOCASE);
CREATE UNIQUE INDEX mount_myfiles
ON mount(kind) WHERE kind = 'myfiles';
CREATE UNIQUE INDEX mount_device
ON mount(sync_folder_id) WHERE kind = 'device';
CREATE UNIQUE INDEX mount_shared
ON mount(share_root_uid) WHERE kind = 'shared';
CREATE INDEX idx_album_photos_seq ON album_photos(album_uid, seq);
CREATE INDEX idx_album_photos_uid ON album_photos(uid);
CREATE INDEX idx_photos_favorite ON photos(favorite, seq);
CREATE INDEX idx_sync_entry_remote_uid ON sync_entry(remote_uid);
CREATE INDEX idx_local_files_scan_gen ON local_files(scan_gen);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
CREATE INDEX idx_pending_op_claimed
ON pending_op(uid) WHERE claimed_at <> 0;
CREATE INDEX idx_trash_parent ON trash(parent_uid);
CREATE INDEX idx_photos_group ON photos(group_key);
CREATE INDEX idx_photos_unresolved ON photos(resolved_at);
CREATE INDEX idx_photos_place ON photos(place_id);
PRAGMA writable_schema=OFF;
COMMIT;
+54 -1
View File
@@ -9,7 +9,7 @@ use super::Db;
use crate::Result;
/// Current schema version. Bump on every forward migration added below.
pub(super) const SCHEMA_VERSION: i64 = 35;
pub(super) const SCHEMA_VERSION: i64 = 36;
impl Db {
pub(super) fn migrate(&self) -> Result<()> {
@@ -332,6 +332,22 @@ impl Db {
tx.execute_batch(MIGRATION_V35)?;
}
}
if current < 36 {
// Same guards as V26-V35.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('nodes') WHERE name = 'lid'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_nodes: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'nodes'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_nodes && !has_column {
tx.execute_batch(MIGRATION_V36)?;
}
}
tx.execute(
"INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1)
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
@@ -1059,3 +1075,40 @@ ALTER TABLE photos ADD COLUMN similar_hash INTEGER;
const MIGRATION_V35: &str = "
ALTER TABLE trash ADD COLUMN trashed_at INTEGER;
";
/// Schema v36: every node row has a local id (`lid`), phase 2 of
/// docs/MILESTONE-3.0.0.md.
///
/// The `uid` was the row's key, so a node had no identity until Drive minted
/// one, and a create that landed moved its row to a new key. The `lid` is given
/// when the row is first written and never changes or comes back, which is what
/// `AUTOINCREMENT` adds over a plain rowid. It takes the old rowid, so the
/// search index, keyed by rowid since v15, stays valid. The `uid` becomes
/// nullable for a node that is not on Drive yet.
const MIGRATION_V36: &str = "
CREATE TABLE nodes_v36 (
lid INTEGER PRIMARY KEY AUTOINCREMENT,
uid TEXT UNIQUE,
parent_uid TEXT,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER,
mtime INTEGER NOT NULL,
revision_hash TEXT,
trashed INTEGER NOT NULL DEFAULT 0,
node_json TEXT,
listed INTEGER NOT NULL DEFAULT 0,
path TEXT
);
INSERT INTO nodes_v36
(lid, uid, parent_uid, name, is_dir, size, mtime, revision_hash, trashed, node_json, listed,
path)
SELECT rowid, uid, parent_uid, name, is_dir, size, mtime, revision_hash, trashed, node_json,
listed, path
FROM nodes;
DROP TABLE nodes;
ALTER TABLE nodes_v36 RENAME TO nodes;
CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
";
+3 -2
View File
@@ -56,8 +56,9 @@ pub use local::LocalFileHit;
pub use maintenance::{DbStats, VacuumOutcome};
pub use nodes::{PublishedSharedRoot, SearchHit, StoredNode};
pub use ops::{
AttachedBlob, FAILING_ATTEMPTS, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION,
OP_TRASH, PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts, PendingOp, RenameMeta, op_supersedes,
AttachedBlob, CreateLanding, FAILING_ATTEMPTS, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME,
OP_REVISION, OP_TRASH, PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts, PendingOp, RenameMeta,
op_supersedes,
};
pub use photos::{StoredPhoto, StoredPlace, THUMB_HAVE, THUMB_NONE, THUMB_UNKNOWN, TimelineRow};
pub use pins::PinRow;
+131 -10
View File
@@ -17,6 +17,8 @@ use super::utils::{
pub struct StoredNode {
pub node: Node,
pub listed: bool,
/// The row's local id ([`Db::lid_of`]).
pub lid: i64,
}
/// One full-text search match: the stored [`Node`] plus its mountpoint-relative
@@ -289,7 +291,8 @@ impl Db {
let uid_str = parent_uid.to_string();
let conn = self.read();
let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM nodes WHERE parent_uid = ?1 AND trashed = 0",
"SELECT COUNT(*) FROM nodes
WHERE parent_uid = ?1 AND trashed = 0 AND node_json IS NOT NULL",
params![uid_str],
|row| row.get(0),
)?;
@@ -305,9 +308,11 @@ impl Db {
pub fn node_path(&self, uid: &str) -> Result<Option<String>> {
let conn = self.read();
let exists: Option<i64> = conn
.query_row("SELECT 1 FROM nodes WHERE uid = ?1", params![uid], |r| {
r.get(0)
})
.query_row(
"SELECT 1 FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL",
params![uid],
|r| r.get(0),
)
.optional()?;
if exists.is_none() {
return Ok(None);
@@ -511,17 +516,17 @@ impl Db {
pub fn load_all(&self) -> Result<Vec<StoredNode>> {
let conn = self.read();
let mut stmt =
conn.prepare("SELECT node_json, listed FROM nodes WHERE node_json IS NOT NULL")?;
conn.prepare("SELECT node_json, listed, lid FROM nodes WHERE node_json IS NOT NULL")?;
let rows = stmt.query_map([], |row| {
let json: String = row.get(0)?;
let listed: i64 = row.get(1)?;
Ok((json, listed != 0))
Ok((json, listed != 0, row.get(2)?))
})?;
let mut out = Vec::new();
for row in rows {
let (json, listed) = row?;
let (json, listed, lid) = row?;
let node: Node = serde_json::from_str(&json)?;
out.push(StoredNode { node, listed });
out.push(StoredNode { node, listed, lid });
}
Ok(out)
}
@@ -544,6 +549,57 @@ impl Db {
}
}
/// The local id of the node stored under `uid`, or `None` when it has no
/// row. It stays the same for as long as the row exists, including when a
/// create lands and the row takes the uid Drive gave it.
pub fn lid_of(&self, uid: &str) -> Result<Option<i64>> {
let conn = self.read();
Ok(conn
.query_row("SELECT lid FROM nodes WHERE uid = ?1", params![uid], |r| {
r.get(0)
})
.optional()?)
}
/// The local id of each node, in order, giving a row to each that has none.
///
/// A new row is only a stub: it has no `node_json`, so no listing or search
/// returns it until the node is upserted, which the caller still owes. That
/// keeps this cheap enough to call with the inode lock held, which is
/// where an inode number is first needed.
pub fn lids_for(&self, nodes: &[&Node]) -> Result<Vec<i64>> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let mut lids = Vec::with_capacity(nodes.len());
{
let mut known = tx.prepare_cached("SELECT lid FROM nodes WHERE uid = ?1")?;
let mut stub = tx.prepare_cached(
"INSERT INTO nodes (uid, parent_uid, name, is_dir, mtime, trashed)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
)?;
for node in nodes {
let uid = node.uid.to_string();
let lid = match known.query_row(params![uid], |r| r.get(0)).optional()? {
Some(lid) => lid,
None => {
stub.execute(params![
uid,
node.parent_uid.as_ref().map(ToString::to_string),
node.name,
node.is_folder() as i64,
node.modification_time,
node.trashed as i64,
])?;
tx.last_insert_rowid()
}
};
lids.push(lid);
}
}
tx.commit()?;
Ok(lids)
}
/// Read the persisted incremental-sync cursor (a `DriveEventId`), if any.
/// The daemon resumes from this on restart instead of reseeding to the
/// server head, so changes made while unmounted are still applied (P2).
@@ -559,6 +615,15 @@ impl Db {
if listed != Some(1) {
return Ok(None);
}
drop(conn);
self.known_children(parent).map(Some)
}
/// The children the DB knows under `parent`, whether or not the folder's
/// listing is still marked complete. Only for when nothing better can be
/// had: a listing dropped as stale may miss or keep a child.
pub fn known_children(&self, parent: &NodeUid) -> Result<Vec<Node>> {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT node_json FROM nodes
WHERE parent_uid = ?1 AND node_json IS NOT NULL AND trashed = 0",
@@ -568,7 +633,7 @@ impl Db {
for json in rows {
out.push(serde_json::from_str(&json?)?);
}
Ok(Some(out))
Ok(out)
}
// --- Content-cache LRU index (P4) -------------------------------------
@@ -590,6 +655,60 @@ struct PriorRow {
trashed: bool,
}
/// Hand a drained placeholder's row, and the rows below it, to the real uid it
/// landed as.
///
/// Dropping the placeholder row and waiting for the server's copy of the real
/// node left a window with no row for either uid. A queued child drained in
/// that window found no authority for its parent, re-read it over the network
/// and was deferred five seconds behind a folder that had just landed
/// (`docs/BUGS.md` B120). The placeholder's own node, readdressed, stands in
/// until the server's copy is upserted over it. A row the real uid already
/// has is newer than the placeholder, so its node is kept, under the
/// placeholder's `lid`: the node was known by that one first.
pub(super) fn adopt_placeholder_row_tx(
tx: &Transaction<'_>,
local: &str,
real: &str,
) -> Result<()> {
tx.execute(
"UPDATE nodes SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real],
)?;
let row: Option<(i64, Option<String>)> = tx
.query_row(
"SELECT lid, node_json FROM nodes WHERE uid = ?1",
params![local],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
let Some((lid, json)) = row else {
return Ok(());
};
let known: Option<(i64, Option<String>)> = tx
.query_row(
"SELECT lid, node_json FROM nodes WHERE uid = ?1",
params![real],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
if let Some((known_lid, _)) = known {
tx.execute("DELETE FROM nodes WHERE lid = ?1", params![known_lid])?;
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![known_lid])?;
}
tx.execute(
"UPDATE nodes SET uid = ?2 WHERE lid = ?1",
params![lid, real],
)?;
let json = known.and_then(|(_, known_json)| known_json).or(json);
let (Some(json), Some(real)) = (json, parse_node_uid(real)) else {
return Ok(());
};
let mut node: Node = serde_json::from_str(&json)?;
node.uid = real;
upsert_node_tx(tx, &node)
}
fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
let json = serde_json::to_string(node)?;
let uid = node.uid.to_string();
@@ -741,8 +860,10 @@ fn reindex_subtree_tx(
let mut set_path = tx.prepare("UPDATE nodes SET path = ?2 WHERE rowid = ?1")?;
let mut unindex = tx.prepare("DELETE FROM nodes_fts WHERE rowid = ?1")?;
let mut index = tx.prepare(
// Not a stub row (`Db::lids_for`): it is indexed when its node is
// written.
"INSERT INTO nodes_fts (rowid, name, path)
SELECT rowid, name, ?2 FROM nodes WHERE rowid = ?1",
SELECT rowid, name, ?2 FROM nodes WHERE rowid = ?1 AND node_json IS NOT NULL",
)?;
for (rowid, path, blocked) in descendants {
set_path.execute(params![rowid, path])?;
+220 -45
View File
@@ -6,6 +6,7 @@ use rusqlite::{OptionalExtension, params};
use serde::{Deserialize, Serialize};
use super::Db;
use super::nodes::adopt_placeholder_row_tx;
use crate::Result;
/// The `kind` of a [`PendingOp`] that uploads a staged file as a new revision.
@@ -44,6 +45,12 @@ pub const OP_RENAME: &str = "rename";
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct RenameMeta {
pub original_parent_uid: String,
/// The node's name on Drive when the first of its queued renames was made,
/// kept across the renames that replace the row, like the parent. With it
/// the drain knows where the node should still be when the op lands.
/// `None` on a row queued before this field existed.
#[serde(default)]
pub original_name: Option<String>,
}
/// The `kind` of a [`PendingOp`] that trashes a node the server knows about
@@ -66,6 +73,63 @@ pub fn op_supersedes(kind: &str) -> bool {
matches!(kind, OP_REVISION | OP_RENAME | OP_TRASH)
}
/// Delete the ops that removing `uid` takes with it, returning the staged blobs
/// they held.
///
/// That is every op for `uid` or for anything queued beneath it, plus the
/// revisions of files the node tree places beneath it. A revision op carries
/// no `parent_uid`, so the queue alone cannot tell that its file sits inside
/// a removed folder: the upload outlived the folder, failed forever against a
/// trashed node, and kept its blob on disk (`docs/BUGS.md` B102).
///
/// "Queued beneath it" starts from every folder the node tree places there,
/// not only from `uid`. A create in a subfolder that already exists on Drive
/// names that subfolder as its parent, so following the queue from `uid` alone
/// missed it: a parked one kept its blob for good, and a due one would have
/// been made in the root once its folder turned out trashed (`docs/BUGS.md`
/// B116).
///
/// Other ops are taken from the node tree only for revisions. A queued rename
/// that moved a node *into* the folder has to land, or the node would stay
/// where it was on the server instead of going to the trash with the folder.
fn drop_doomed_ops(tx: &rusqlite::Transaction<'_>, uid: &str) -> Result<Vec<String>> {
const DOOMED: &str = "
WITH RECURSIVE
below(uid) AS (
SELECT ?1
UNION
SELECT n.uid FROM nodes n JOIN below b ON n.parent_uid = b.uid
),
queued(uid) AS (
SELECT ?1
UNION
SELECT p.uid FROM pending_op p
WHERE p.kind IN ('create', 'mkdir')
AND p.parent_uid IN (SELECT uid FROM below)
UNION
SELECT p.uid FROM pending_op p JOIN queued q ON p.parent_uid = q.uid
),
doomed(id) AS (
SELECT id FROM pending_op
WHERE uid IN (SELECT uid FROM queued)
OR (kind = 'revision' AND uid IN (SELECT uid FROM below))
)";
let blobs: Vec<String> = {
let mut stmt = tx.prepare(&format!(
"{DOOMED}
SELECT blob_path FROM pending_op
WHERE id IN (SELECT id FROM doomed) AND blob_path IS NOT NULL"
))?;
let rows = stmt.query_map(params![uid], |r| r.get::<_, String>(0))?;
rows.collect::<std::result::Result<Vec<_>, _>>()?
};
tx.execute(
&format!("{DOOMED} DELETE FROM pending_op WHERE id IN (SELECT id FROM doomed)"),
params![uid],
)?;
Ok(blobs)
}
/// The volume id given to a node that exists only on this machine, so far. A
/// real [`NodeUid`] is `{volume}~{link}`, so a placeholder is `local~<uuid>` and
/// round-trips through the same `Display`/parse path as any other uid.
@@ -164,6 +228,28 @@ pub struct PendingCounts {
/// is retrying at the slowest rate it ever will and has been for minutes.
pub const FAILING_ATTEMPTS: i64 = 6;
/// A queued create that has landed, for [`Db::finish_create`].
#[derive(Debug, Clone, Copy)]
pub struct CreateLanding<'a> {
/// The placeholder the node was known by.
pub local: &'a str,
/// The uid Drive gave it.
pub real: &'a str,
/// The parent and name the op held when the drain took it.
pub sent: (&'a str, &'a str),
/// The parent and name the node has on Drive. They differ from `sent`
/// when the name was taken or the folder was gone.
pub landed: (&'a str, &'a str),
}
struct CreateRow {
blob: Option<String>,
meta: Option<String>,
parent_uid: Option<String>,
name: Option<String>,
created_at: i64,
}
/// The outcome of folding freshly written bytes into a queued create.
#[derive(Debug, Clone)]
pub struct AttachedBlob {
@@ -174,6 +260,24 @@ pub struct AttachedBlob {
}
impl Db {
/// Whether a queued trash still holds `name` somewhere.
///
/// A replacing rename or a delete-then-recreate queues the trash of the old
/// node ahead of the op that wants its name. With several drain workers the
/// later op can run first and find the name still taken; this tells it to
/// wait for the trash rather than land under a conflict name.
pub fn has_pending_trash_named(&self, name: &str) -> Result<bool> {
let conn = self.read();
conn.query_row(
"SELECT EXISTS(
SELECT 1 FROM pending_op WHERE kind = ?1 AND name = ?2
)",
params![OP_TRASH, name],
|row| row.get(0),
)
.map_err(Into::into)
}
/// Whether a specific desired-state operation is still queued for a node.
pub fn has_pending_op(&self, uid: &str, kind: &str) -> Result<bool> {
let conn = self.read();
@@ -252,27 +356,9 @@ impl Db {
name: &str,
created_at: i64,
) -> Result<(i64, Vec<String>)> {
const SUBTREE: &str = "
WITH RECURSIVE doomed(uid) AS (
SELECT ?1
UNION
SELECT p.uid FROM pending_op p JOIN doomed d ON p.parent_uid = d.uid
)";
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let blobs: Vec<String> = {
let mut stmt = tx.prepare(&format!(
"{SUBTREE}
SELECT blob_path FROM pending_op
WHERE uid IN (SELECT uid FROM doomed) AND blob_path IS NOT NULL"
))?;
let rows = stmt.query_map(params![uid], |row| row.get::<_, String>(0))?;
rows.collect::<std::result::Result<Vec<_>, _>>()?
};
tx.execute(
&format!("{SUBTREE} DELETE FROM pending_op WHERE uid IN (SELECT uid FROM doomed)"),
params![uid],
)?;
let blobs = drop_doomed_ops(&tx, uid)?;
tx.execute(
"INSERT INTO pending_op
(kind, uid, parent_uid, name, blob_path, meta_json, created_at, next_attempt_at)
@@ -326,6 +412,102 @@ impl Db {
Ok(Some(AttachedBlob { id, superseded }))
}
/// Retire a queued create that has just landed as `real`, and repoint
/// everything that named its placeholder, in one transaction.
///
/// The drain uploads the blob the create held when it was picked up. A
/// write that closes while that upload is on the wire attaches a newer blob
/// to the *same row* (`attach_blob_to_create`), so deleting the row by id
/// threw that write away: the server kept the older bytes and nothing was
/// left queued to replace them. When the row no longer holds `uploaded`, it
/// becomes a revision of `real` instead, with its sidecar passed through
/// `rewrite` so it names the real node. Returns that blob and sidecar, or
/// `None` when the create is simply done.
///
/// A rename while the create is on the wire rewrites the row's target
/// ([`Db::rewrite_op_target`]), which the create no longer reads. When the
/// row names another parent or name than `landing.sent`, a rename of `real`
/// to that target is queued in the same transaction, or the move would be
/// lost with the row.
pub fn finish_create(
&self,
id: i64,
uploaded: Option<&str>,
landing: &CreateLanding<'_>,
rewrite: impl FnOnce(&str) -> Option<String>,
) -> Result<Option<(String, String)>> {
let (local, real) = (landing.local, landing.real);
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let row: Option<CreateRow> = tx
.query_row(
"SELECT blob_path, meta_json, parent_uid, name, created_at
FROM pending_op WHERE id = ?1",
params![id],
|r| {
Ok(CreateRow {
blob: r.get(0)?,
meta: r.get(1)?,
parent_uid: r.get(2)?,
name: r.get(3)?,
created_at: r.get(4)?,
})
},
)
.optional()?;
let newer = match &row {
Some(CreateRow {
blob: Some(blob),
meta: Some(meta),
..
}) if uploaded != Some(blob.as_str()) => rewrite(meta).map(|meta| (blob.clone(), meta)),
_ => None,
};
match &newer {
Some((_, meta)) => {
tx.execute(
"UPDATE pending_op
SET kind = ?2, uid = ?3, parent_uid = NULL, name = NULL, meta_json = ?4,
attempts = 0, last_error = NULL, next_attempt_at = 0
WHERE id = ?1",
params![id, OP_REVISION, real, meta],
)?;
}
None => {
tx.execute("DELETE FROM pending_op WHERE id = ?1", params![id])?;
}
}
if let Some(CreateRow {
parent_uid: Some(parent),
name: Some(name),
created_at,
..
}) = &row
&& (parent.as_str(), name.as_str()) != landing.sent
{
let meta = serde_json::to_string(&RenameMeta {
original_parent_uid: landing.landed.0.to_string(),
original_name: Some(landing.landed.1.to_string()),
})?;
tx.execute(
"DELETE FROM pending_op WHERE uid = ?1 AND kind = ?2",
params![real, OP_RENAME],
)?;
tx.execute(
"INSERT INTO pending_op (kind, uid, parent_uid, name, meta_json, created_at)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
params![OP_RENAME, real, parent, name, meta, created_at],
)?;
}
tx.execute(
"UPDATE pending_op SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real],
)?;
adopt_placeholder_row_tx(&tx, local, real)?;
tx.commit()?;
Ok(newer)
}
/// Replace the sidecar of a queued op, for a baseline that has moved under
/// it.
///
@@ -428,36 +610,20 @@ impl Db {
/// and in the user's pending count, for the life of the database.
///
/// Only `create`/`mkdir` ops carry a `parent_uid`, so for a file this
/// recursion finds nothing and costs one query.
/// recursion finds nothing and costs one query. Revisions of files the
/// node tree places beneath `uid` go too; see [`drop_doomed_ops`].
pub fn delete_ops_for_uid(&self, uid: &str) -> Result<Vec<String>> {
const SUBTREE: &str = "
WITH RECURSIVE doomed(uid) AS (
SELECT ?1
UNION
SELECT p.uid FROM pending_op p JOIN doomed d ON p.parent_uid = d.uid
)";
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let blobs: Vec<String> = {
let mut stmt = tx.prepare(&format!(
"{SUBTREE}
SELECT blob_path FROM pending_op
WHERE uid IN (SELECT uid FROM doomed) AND blob_path IS NOT NULL"
))?;
let rows = stmt.query_map(params![uid], |r| r.get::<_, String>(0))?;
rows.collect::<std::result::Result<Vec<_>, _>>()?
};
tx.execute(
&format!("{SUBTREE} DELETE FROM pending_op WHERE uid IN (SELECT uid FROM doomed)"),
params![uid],
)?;
let blobs = drop_doomed_ops(&tx, uid)?;
tx.commit()?;
Ok(blobs)
}
/// Rewrite every queued op that points at a placeholder parent, once that
/// parent has drained and has a real uid. Also moves the node rows whose
/// parent column still names the placeholder, so listings keep resolving.
/// parent column still names the placeholder, so listings keep resolving,
/// and readdresses the placeholder's own row.
pub fn remap_local_uid(&self, local: &str, real: &str) -> Result<()> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
@@ -465,11 +631,7 @@ impl Db {
"UPDATE pending_op SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real],
)?;
tx.execute(
"UPDATE nodes SET parent_uid = ?2 WHERE parent_uid = ?1",
params![local, real],
)?;
tx.execute("DELETE FROM nodes WHERE uid = ?1", params![local])?;
adopt_placeholder_row_tx(&tx, local, real)?;
tx.commit()?;
Ok(())
}
@@ -765,6 +927,19 @@ impl Db {
/// drain replaces it with the uid the server assigns.
pub const LOCAL_VOLUME: &str = "local";
/// Whether op `id` is still queued. A drain worker that claimed it can find
/// it gone: the node was deleted, or a newer write superseded it, while the
/// worker was talking to the server.
pub fn op_exists(&self, id: i64) -> Result<bool> {
let conn = self.read();
let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM pending_op WHERE id = ?1",
params![id],
|row| row.get(0),
)?;
Ok(count > 0)
}
/// Drop a queued op, once its upload has actually landed.
pub fn delete_op(&self, id: i64) -> Result<()> {
let conn = self.conn.lock();
+1 -1
View File
@@ -98,7 +98,7 @@ impl Db {
pub fn effective_node_access(&self, uid: &NodeUid) -> Result<Option<Access>> {
let conn = self.read();
let exists: bool = conn.query_row(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE uid = ?1)",
"SELECT EXISTS(SELECT 1 FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL)",
[uid.to_string()],
|row| row.get(0),
)?;
+82
View File
@@ -260,6 +260,88 @@ impl Db {
Ok(())
}
/// Drop the baseline rows of `rel_path` and everything under it.
///
/// Used when a subtree leaves a mirror folder through a move on Drive: its
/// local copy is about to be removed, and rows left behind would read as
/// "deleted locally" and trash the moved nodes at their new place.
pub fn sync_entries_remove_subtree(&self, folder_id: i64, rel_path: &str) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"DELETE FROM sync_entry
WHERE folder_id = ?1
AND (rel_path = ?2 OR substr(rel_path, 1, length(?2) + 1) = ?2 || '/')",
params![folder_id, rel_path],
)?;
Ok(())
}
/// Move the baseline rows of `from_rel` and everything under it to `to_rel`
/// in folder `to_folder`, in one transaction.
///
/// Used when a move on Drive took a mirror subtree along with its local
/// copy: the rows still describe both sides correctly, only under a new
/// path, so the next pass on either folder has nothing to do. The two
/// folders may be the same one.
pub fn sync_entries_move(
&self,
from_folder: i64,
from_rel: &str,
to_folder: i64,
to_rel: &str,
) -> Result<()> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
{
let mut select = tx.prepare(
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash,
local_mtime_ns
FROM sync_entry
WHERE folder_id = ?1
AND (rel_path = ?2 OR substr(rel_path, 1, length(?2) + 1) = ?2 || '/')",
)?;
let rows = select
.query_map(params![from_folder, from_rel], |r| {
Ok(StoredSyncEntry {
rel_path: r.get(0)?,
remote_uid: r.get(1)?,
local_mtime: r.get(2)?,
local_size: r.get(3)?,
remote_rev: r.get(4)?,
remote_hash: r.get(5)?,
local_mtime_ns: r.get(6)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
let mut delete =
tx.prepare("DELETE FROM sync_entry WHERE folder_id = ?1 AND rel_path = ?2")?;
for row in &rows {
delete.execute(params![from_folder, row.rel_path])?;
}
let mut insert = tx.prepare(
"INSERT OR REPLACE INTO sync_entry
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev,
remote_hash, local_mtime_ns)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
)?;
for row in rows {
let rel = format!("{to_rel}{}", &row.rel_path[from_rel.len()..]);
insert.execute(params![
to_folder,
rel,
row.remote_uid,
row.local_mtime,
row.local_size,
row.remote_rev,
row.remote_hash,
row.local_mtime_ns,
])?;
}
}
tx.commit()?;
Ok(())
}
/// Drop the entire baseline for a folder. Used when flipping ondemand→mirror:
/// the local tree was evicted, so the old baseline is stale and would make the
/// next reconcile mistake "locally deleted" for "must re-download". Clearing it
+829
View File
@@ -788,6 +788,8 @@ fn children_if_listed_gated_on_flag() {
// Not listed yet → unknown, force a re-fetch.
assert!(db.children_if_listed(&uid("root")).unwrap().is_none());
// Still known, for a caller that has nothing better while offline.
assert_eq!(db.known_children(&uid("root")).unwrap().len(), 2);
db.set_listed(&uid("root"), true).unwrap();
let kids = db.children_if_listed(&uid("root")).unwrap().unwrap();
@@ -2458,6 +2460,20 @@ fn atomic_trash_replacement_returns_owned_blobs_after_commit() {
assert!(ops[0].blob_path.is_none());
}
#[test]
fn a_queued_trash_holds_its_name_until_it_lands() {
let db = Db::open_in_memory().unwrap();
let node = uid("held-name").to_string();
assert!(!db.has_pending_trash_named("report.txt").unwrap());
let (trash_id, _) = db.replace_ops_with_trash(&node, "report.txt", 1).unwrap();
assert!(db.has_pending_trash_named("report.txt").unwrap());
assert!(!db.has_pending_trash_named("other.txt").unwrap());
db.delete_op(trash_id).unwrap();
assert!(!db.has_pending_trash_named("report.txt").unwrap());
}
#[test]
fn failed_atomic_trash_insert_keeps_prior_revision_and_blob_ownership() {
let db = Db::open_in_memory().unwrap();
@@ -2498,6 +2514,139 @@ fn failed_atomic_trash_insert_keeps_prior_revision_and_blob_ownership() {
);
}
/// A revision op names no parent, so the queue alone cannot see that its file
/// sits inside a folder being removed. The node tree can: trashing the folder
/// has to take the uploads of the files below it along, or they fail forever
/// against a trashed node and keep their bytes on disk (B102). A rename that
/// moved a node into the folder still has to land.
#[test]
fn trashing_a_folder_drops_the_queued_revisions_of_files_below_it() {
let db = Db::open_in_memory().unwrap();
db.upsert_nodes(&[
folder("root", None, "My Files"),
folder("doomed", Some("root"), "doomed"),
folder("sub", Some("doomed"), "sub"),
file("deep", "sub", "deep.bin", 1),
file("moved", "doomed", "moved.txt", 1),
file("kept", "root", "kept.bin", 1),
])
.unwrap();
let op = |kind: &str, link: &str, blob: Option<&str>| PendingOp {
id: 0,
kind: kind.to_string(),
uid: uid(link).to_string(),
parent_uid: None,
name: None,
blob_path: blob.map(str::to_string),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
};
db.enqueue_op(&op(OP_REVISION, "deep", Some("/staging/deep")))
.unwrap();
db.enqueue_op(&op(OP_RENAME, "moved", None)).unwrap();
db.enqueue_op(&op(OP_REVISION, "kept", Some("/staging/kept")))
.unwrap();
let doomed = uid("doomed").to_string();
let (_, blobs) = db.replace_ops_with_trash(&doomed, "doomed", 2).unwrap();
assert_eq!(blobs, vec!["/staging/deep"]);
let mut left: Vec<(String, String)> = db
.pending_ops()
.unwrap()
.into_iter()
.map(|op| (op.kind, op.uid))
.collect();
left.sort();
assert_eq!(
left,
vec![
(OP_RENAME.to_string(), uid("moved").to_string()),
(OP_REVISION.to_string(), uid("kept").to_string()),
(OP_TRASH.to_string(), doomed.clone()),
]
);
// Deleting outright follows the same tree.
db.enqueue_op(&op(OP_REVISION, "deep", Some("/staging/deep2")))
.unwrap();
assert_eq!(
db.delete_ops_for_uid(&uid("sub").to_string()).unwrap(),
vec!["/staging/deep2"]
);
}
/// A create in a subfolder that already exists on Drive names that subfolder,
/// not the folder being trashed, as its parent. It has to go with the folder
/// all the same, along with whatever is queued inside it, or a parked one
/// keeps its bytes for good and a due one is made in the root (B116).
#[test]
fn trashing_a_folder_drops_the_queued_creates_in_its_subfolders() {
let db = Db::open_in_memory().unwrap();
db.upsert_nodes(&[
folder("root", None, "My Files"),
folder("doomed", Some("root"), "doomed"),
folder("sub", Some("doomed"), "sub"),
folder("other", Some("root"), "other"),
])
.unwrap();
let op = |kind: &str, node: &str, parent: &str, blob: Option<&str>| PendingOp {
id: 0,
kind: kind.to_string(),
uid: node.to_string(),
parent_uid: Some(parent.to_string()),
name: Some(format!("{node}.bin")),
blob_path: blob.map(str::to_string),
meta_json: None,
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
};
let sub = uid("sub").to_string();
db.enqueue_op(&op(OP_CREATE, "local~deep", &sub, Some("/staging/deep")))
.unwrap();
db.enqueue_op(&op(OP_MKDIR, "local~dir", &sub, None))
.unwrap();
db.enqueue_op(&op(
OP_CREATE,
"local~deeper",
"local~dir",
Some("/staging/deeper"),
))
.unwrap();
db.enqueue_op(&op(
OP_CREATE,
"local~kept",
&uid("other").to_string(),
Some("/staging/kept"),
))
.unwrap();
let doomed = uid("doomed").to_string();
let (_, mut blobs) = db.replace_ops_with_trash(&doomed, "doomed", 2).unwrap();
blobs.sort();
assert_eq!(blobs, vec!["/staging/deep", "/staging/deeper"]);
let mut left: Vec<(String, String)> = db
.pending_ops()
.unwrap()
.into_iter()
.map(|op| (op.kind, op.uid))
.collect();
left.sort();
assert_eq!(
left,
vec![
(OP_CREATE.to_string(), "local~kept".to_string()),
(OP_TRASH.to_string(), doomed),
]
);
}
/// Deleting a folder that was created offline must take the ops queued
/// underneath it with it: they name a placeholder parent that will now never
/// become real, so nothing could ever drain them and nothing is left to
@@ -2699,6 +2848,165 @@ fn attaching_to_an_already_drained_create_reports_it_is_gone() {
assert!(out.is_none());
}
#[test]
fn a_write_attached_while_its_create_uploads_becomes_a_revision() {
let db = Db::open_in_memory().unwrap();
let local = "local~abc";
let real = uid("real").to_string();
let parent = uid("parent").to_string();
let create = |blob: Option<&str>| PendingOp {
id: 0,
kind: OP_CREATE.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("parent").to_string()),
name: Some("new.txt".to_string()),
blob_path: blob.map(str::to_string),
meta_json: blob.map(|_| "{\"uid\":\"local~abc\"}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
};
let (id, _) = db.enqueue_op(&create(None)).unwrap();
// The drain picked the create up with no blob; the file's first write
// closes while the empty upload is on the wire.
db.attach_blob_to_create(local, "/staging/written", "{\"uid\":\"local~abc\"}")
.unwrap()
.expect("create is still queued");
let newer = db
.finish_create(
id,
None,
&landing(local, &real, &parent, "new.txt"),
|json| Some(json.replace("local~abc", "vol~real")),
)
.unwrap();
assert_eq!(
newer,
Some((
"/staging/written".to_string(),
"{\"uid\":\"vol~real\"}".to_string()
))
);
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1, "the write is still queued");
assert_eq!(ops[0].kind, OP_REVISION);
assert_eq!(ops[0].uid, real);
assert_eq!(ops[0].blob_path.as_deref(), Some("/staging/written"));
assert_eq!(ops[0].parent_uid, None);
}
#[test]
fn a_create_whose_blob_landed_is_retired() {
let db = Db::open_in_memory().unwrap();
let local = "local~abc";
let real = uid("real").to_string();
let parent = uid("parent").to_string();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_CREATE.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("parent").to_string()),
name: Some("new.txt".to_string()),
blob_path: Some("/staging/sent".to_string()),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
db.enqueue_op(&PendingOp {
id: 0,
kind: OP_CREATE.to_string(),
uid: "local~child".to_string(),
parent_uid: Some(local.to_string()),
name: Some("inside.txt".to_string()),
blob_path: None,
meta_json: None,
created_at: 2,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
let newer = db
.finish_create(
id,
Some("/staging/sent"),
&landing(local, &real, &parent, "new.txt"),
|_| panic!("an unchanged create has no sidecar to rewrite"),
)
.unwrap();
assert_eq!(newer, None);
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1, "only the child is left");
assert_eq!(ops[0].parent_uid.as_deref(), Some(real.as_str()));
}
/// The parent and name a create was sent with, landing where it was sent.
fn landing<'a>(local: &'a str, real: &'a str, parent: &'a str, name: &'a str) -> CreateLanding<'a> {
CreateLanding {
local,
real,
sent: (parent, name),
landed: (parent, name),
}
}
#[test]
fn a_rename_made_while_its_create_uploads_is_queued_for_the_landed_node() {
let db = Db::open_in_memory().unwrap();
let local = "local~abc";
let real = uid("real").to_string();
let parent = uid("parent").to_string();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_CREATE.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("parent").to_string()),
name: Some("new.txt".to_string()),
blob_path: Some("/staging/sent".to_string()),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
// The drain took the create; the user moves the file into a folder that
// is itself still queued.
db.rewrite_op_target(local, "local~dir", "moved.txt")
.unwrap();
let mut landed = landing(local, &real, &parent, "new.txt");
landed.landed.1 = "new (conflict).txt";
db.finish_create(id, Some("/staging/sent"), &landed, |_| None)
.unwrap();
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1, "the move outlives the create");
assert_eq!(ops[0].kind, OP_RENAME);
assert_eq!(ops[0].uid, real);
assert_eq!(ops[0].parent_uid.as_deref(), Some("local~dir"));
assert_eq!(ops[0].name.as_deref(), Some("moved.txt"));
let meta: RenameMeta = serde_json::from_str(ops[0].meta_json.as_deref().unwrap()).unwrap();
assert_eq!(meta.original_parent_uid, uid("parent").to_string());
assert_eq!(meta.original_name.as_deref(), Some("new (conflict).txt"));
// The folder lands next and the move follows it.
db.remap_local_uid("local~dir", &uid("dir").to_string())
.unwrap();
let ops = db.pending_ops().unwrap();
assert_eq!(ops[0].parent_uid, Some(uid("dir").to_string()));
}
#[test]
fn draining_a_folder_repoints_its_queued_children() {
let db = Db::open_in_memory().unwrap();
@@ -2728,6 +3036,414 @@ fn draining_a_folder_repoints_its_queued_children() {
assert_eq!(ops[0].parent_uid.as_deref(), Some(real_dir.as_str()));
}
/// B120: a child queued inside a folder that has just landed is checked
/// against the folder's real uid before the server's copy of it arrives.
#[test]
fn a_landed_folder_keeps_a_row_until_the_server_copy_replaces_it() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
let local = NodeUid::new(VolumeId::from("local"), LinkId::from("dir"));
let mut placeholder = folder("dir", Some("root"), "New folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
let mut child = file("child", "root", "inside.txt", 1);
child.parent_uid = Some(local.clone());
db.upsert_node(&child).unwrap();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_MKDIR.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("root").to_string()),
name: Some("New folder".to_string()),
blob_path: None,
meta_json: None,
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
db.finish_create(
id,
None,
&landing(
&local.to_string(),
&uid("real").to_string(),
&uid("root").to_string(),
"New folder",
),
|_| None,
)
.unwrap();
assert!(db.node_by_uid(&local.to_string()).unwrap().is_none());
let landed = db
.node_by_uid(&uid("real").to_string())
.unwrap()
.expect("the landed folder has a row before the server's copy is read");
assert_eq!(landed.uid, uid("real"));
assert_eq!(landed.name, "New folder");
assert_eq!(
db.effective_node_access(&uid("child")).unwrap(),
Some(crate::Access::Owner),
"the child's authority resolves without a network re-read"
);
let names: Vec<_> = db
.known_children(&uid("real"))
.unwrap()
.into_iter()
.map(|n| n.name)
.collect();
assert_eq!(names, ["inside.txt"]);
let found = db.search("inside", 10).unwrap();
assert_eq!(found.len(), 1);
assert_eq!(found[0].path, "New folder/inside.txt");
}
#[test]
fn a_landed_placeholder_never_overwrites_the_server_copy() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
let local = NodeUid::new(VolumeId::from("local"), LinkId::from("dir"));
let mut placeholder = folder("dir", Some("root"), "New folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
// The event stream reported the folder before the drain got round to it.
db.upsert_node(&folder("real", Some("root"), "New folder (1)"))
.unwrap();
db.remap_local_uid(&local.to_string(), &uid("real").to_string())
.unwrap();
assert!(db.node_by_uid(&local.to_string()).unwrap().is_none());
assert_eq!(
db.node_by_uid(&uid("real").to_string())
.unwrap()
.map(|n| n.name),
Some("New folder (1)".to_string())
);
}
#[test]
fn a_node_keeps_its_lid_across_updates_and_a_new_one_never_reuses_it() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
db.upsert_node(&file("a", "root", "a.txt", 1)).unwrap();
db.upsert_node(&file("b", "root", "b.txt", 1)).unwrap();
let a = db.lid_of(&uid("a").to_string()).unwrap().unwrap();
let b = db.lid_of(&uid("b").to_string()).unwrap().unwrap();
db.upsert_node(&file("a", "root", "renamed.txt", 2))
.unwrap();
assert_eq!(db.lid_of(&uid("a").to_string()).unwrap(), Some(a));
db.delete_node(&uid("b")).unwrap();
db.upsert_node(&file("c", "root", "c.txt", 1)).unwrap();
let c = db.lid_of(&uid("c").to_string()).unwrap().unwrap();
assert!(c > b, "lid {c} reuses one a deleted row had");
}
#[test]
fn a_lid_handed_out_before_the_node_is_stored_reads_as_no_node() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
db.set_share_access(&uid("root"), crate::Access::Owner)
.unwrap();
let stub = file("stub", "root", "stub.txt", 1);
let [lid] = db.lids_for(&[&stub]).unwrap()[..] else {
panic!("one lid per node");
};
assert_eq!(db.lid_of(&uid("stub").to_string()).unwrap(), Some(lid));
assert_eq!(db.lids_for(&[&stub]).unwrap(), vec![lid]);
assert!(db.node_by_uid(&uid("stub").to_string()).unwrap().is_none());
assert!(db.effective_node_access(&uid("stub")).unwrap().is_none());
assert!(db.node_path(&uid("stub").to_string()).unwrap().is_none());
assert!(!db.has_children(&uid("root")).unwrap());
assert!(db.search("stub", 10).unwrap().is_empty());
assert!(db.load_all().unwrap().iter().all(|n| n.lid != lid));
db.upsert_node(&stub).unwrap();
assert_eq!(db.lid_of(&uid("stub").to_string()).unwrap(), Some(lid));
assert!(db.has_children(&uid("root")).unwrap());
}
#[test]
fn a_landed_placeholder_keeps_its_lid() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
for known in [false, true] {
let local = NodeUid::new(VolumeId::from("local"), LinkId::from(format!("{known}")));
let real = uid(&format!("real-{known}"));
let mut placeholder = folder("dir", Some("root"), "New folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
let lid = db.lid_of(&local.to_string()).unwrap().unwrap();
if known {
// The event stream reported the folder before the drain got round
// to it.
let mut listed = folder("x", Some("root"), "New folder (1)");
listed.uid = real.clone();
db.upsert_node(&listed).unwrap();
}
db.remap_local_uid(&local.to_string(), &real.to_string())
.unwrap();
assert_eq!(db.lid_of(&local.to_string()).unwrap(), None);
assert_eq!(db.lid_of(&real.to_string()).unwrap(), Some(lid));
let name = if known {
"New folder (1)"
} else {
"New folder"
};
assert_eq!(
db.node_by_uid(&real.to_string()).unwrap().map(|n| n.name),
Some(name.to_string())
);
assert_eq!(db.search(name, 10).unwrap().len(), 1, "{name}");
}
}
#[test]
fn migration_v36_gives_every_node_its_rowid_as_lid() {
let path = std::env::temp_dir().join(format!(
"pdfs-db-v35-fixture-{}-{}.db",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
let local = NodeUid::new(VolumeId::from("local"), LinkId::from("dir"));
{
let db = Db::open(&path).unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
let mut placeholder = folder("dir", Some("root"), "Offline folder");
placeholder.uid = local.clone();
db.upsert_node(&placeholder).unwrap();
db.upsert_node(&file("gone", "root", "gone.txt", 1))
.unwrap();
db.upsert_node(&file("kept", "root", "kept-report.txt", 1))
.unwrap();
db.delete_node(&uid("gone")).unwrap();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_MKDIR.to_string(),
uid: local.to_string(),
parent_uid: Some(uid("root").to_string()),
name: Some("Offline folder".to_string()),
blob_path: None,
meta_json: None,
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
db.record_op_failure(id, "network unreachable", 5_000)
.unwrap();
}
{
// Put the file back in the state a released V35 database was in:
// keyed by uid, with the rowids of the rows it held.
let conn = rusqlite::Connection::open(&path).unwrap();
conn.execute_batch(
"CREATE TABLE nodes_v35 (
uid TEXT PRIMARY KEY, parent_uid TEXT, name TEXT NOT NULL,
is_dir INTEGER NOT NULL, size INTEGER, mtime INTEGER NOT NULL,
revision_hash TEXT, trashed INTEGER NOT NULL DEFAULT 0, node_json TEXT,
listed INTEGER NOT NULL DEFAULT 0, path TEXT
);
INSERT INTO nodes_v35 (rowid, uid, parent_uid, name, is_dir, size, mtime,
revision_hash, trashed, node_json, listed, path)
SELECT lid, uid, parent_uid, name, is_dir, size, mtime, revision_hash,
trashed, node_json, listed, path FROM nodes;
DROP TABLE nodes;
ALTER TABLE nodes_v35 RENAME TO nodes;
CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
DELETE FROM sqlite_sequence WHERE name = 'nodes';
UPDATE sync_state SET value = '35' WHERE key = 'schema_version';",
)
.unwrap();
}
let rowids: Vec<(String, i64)> = {
let conn = rusqlite::Connection::open(&path).unwrap();
let mut stmt = conn
.prepare("SELECT uid, rowid FROM nodes ORDER BY rowid")
.unwrap();
stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))
.unwrap()
.map(|row| row.unwrap())
.collect()
};
assert_eq!(rowids.len(), 3);
let db = Db::open(&path).unwrap();
for (uid, rowid) in &rowids {
assert_eq!(db.lid_of(uid).unwrap(), Some(*rowid), "{uid}");
}
let found = db.search("report", 10).unwrap();
assert_eq!(found.len(), 1, "the search index still finds the node");
assert_eq!(found[0].node.uid, uid("kept"));
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1);
assert_eq!(ops[0].uid, local.to_string());
assert_eq!(ops[0].attempts, 1);
assert_eq!(ops[0].last_error.as_deref(), Some("network unreachable"));
db.upsert_node(&file("new", "root", "new.txt", 1)).unwrap();
let newest = db.lid_of(&uid("new").to_string()).unwrap().unwrap();
assert!(newest > rowids.last().unwrap().1);
drop(db);
let _ = std::fs::remove_file(&path);
}
/// A database as 2.8.2 wrote it (`sqlite3 .dump`, schema 35), with queued ops
/// in every state the drain leaves them in: due, parked, failing, deferred for
/// access, and claimed by a worker when the daemon stopped. The creates below
/// a folder made offline still name their parent by its `local~` placeholder.
#[test]
fn a_2_8_database_migrates_with_every_node_and_op_intact() {
let path = std::env::temp_dir().join(format!(
"pdfs-db-2.8.2-{}-{}.db",
std::process::id(),
now_test_id()
));
let _ = std::fs::remove_file(&path);
rusqlite::Connection::open(&path)
.unwrap()
.execute_batch(include_str!("fixtures/v2.8.2.sql"))
.unwrap();
let (nodes, ops) = migrates_intact(&path);
assert_eq!((nodes, ops), (8, 6));
let db = Db::open(&path).unwrap();
let queued = db.pending_ops().unwrap();
let inner = queued.iter().find(|op| op.uid == "local~inner").unwrap();
assert_eq!(inner.parent_uid.as_deref(), Some("local~dir"));
let parked = db.parked_create_ops().unwrap();
assert_eq!(parked.len(), 1);
assert_eq!(parked[0].name.as_deref(), Some("movie.part"));
drop(db);
let _ = std::fs::remove_file(&path);
}
/// The same migration over a copy of a database a released build wrote, which
/// the fixtures above cannot stand in for: years of rows, every kind of op, and
/// whatever shapes real use left behind. The copy is taken with `VACUUM INTO`
/// over a read-only connection, so the database named is never written.
///
/// `PDFS_MIGRATE_DB=~/.local/state/proton-drive-linux/cache.db cargo test -p
/// pdfs-core a_copy_of_a_real_database -- --ignored`
#[test]
#[ignore = "needs a real database: set PDFS_MIGRATE_DB"]
fn a_copy_of_a_real_database_migrates_with_every_node_and_op_intact() {
let source = std::env::var("PDFS_MIGRATE_DB").expect("set PDFS_MIGRATE_DB");
let path = std::env::temp_dir().join(format!("pdfs-db-real-{}.db", std::process::id()));
let _ = std::fs::remove_file(&path);
rusqlite::Connection::open_with_flags(&source, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY)
.unwrap()
.execute("VACUUM INTO ?1", rusqlite::params![path.to_str().unwrap()])
.unwrap();
let (nodes, ops) = migrates_intact(&path);
eprintln!("{nodes} nodes, {ops} queued ops");
let _ = std::fs::remove_file(&path);
}
/// Opens the database at `path` and checks that the migration kept every node
/// on its row id and every queued op as it was. Returns how many of each.
fn migrates_intact(path: &std::path::Path) -> (usize, usize) {
let (version, nodes, ops, sealed) = {
let conn = rusqlite::Connection::open(path).unwrap();
let version: String = conn
.query_row(
"SELECT value FROM sync_state WHERE key = 'schema_version'",
[],
|r| r.get(0),
)
.unwrap();
let nodes: Vec<(String, i64)> = conn
.prepare("SELECT uid, rowid FROM nodes")
.unwrap()
.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))
.unwrap()
.map(|row| row.unwrap())
.collect();
let ops = rows(&conn, QUEUED);
let sealed = rows(&conn, "SELECT * FROM own_sealed_rev ORDER BY uid");
(version, nodes, ops, sealed)
};
let db = Db::open(path).unwrap();
{
let conn = rusqlite::Connection::open(path).unwrap();
assert_eq!(
rows(&conn, QUEUED),
ops,
"every queued op survives as it was, parked, failing or deferred"
);
// A claim names a drain worker of the daemon that wrote the file.
let claimed: i64 = conn
.query_row(
"SELECT COUNT(*) FROM pending_op WHERE claimed_at <> 0",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(claimed, 0, "a stopped daemon's claims are released");
assert_eq!(
rows(&conn, "SELECT * FROM own_sealed_rev ORDER BY uid"),
sealed
);
}
if version.parse::<i64>().unwrap() < 36 {
for (uid, rowid) in &nodes {
assert_eq!(db.lid_of(uid).unwrap(), Some(*rowid), "{uid}");
}
} else {
for (uid, _) in &nodes {
assert!(db.lid_of(uid).unwrap().is_some(), "{uid}");
}
}
drop(db);
let db = Db::open(path).unwrap();
let conn = rusqlite::Connection::open(path).unwrap();
let count: i64 = conn
.query_row("SELECT COUNT(*) FROM nodes", [], |r| r.get(0))
.unwrap();
assert_eq!(count as usize, nodes.len(), "no node is lost or doubled");
let version: String = conn
.query_row(
"SELECT value FROM sync_state WHERE key = 'schema_version'",
[],
|r| r.get(0),
)
.unwrap();
assert_eq!(version, SCHEMA_VERSION.to_string());
drop(db);
(nodes.len(), ops.len())
}
/// Every column of the queue but the drain's claim.
const QUEUED: &str = "SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
attempts, last_error, next_attempt_at, access_deferred_since
FROM pending_op ORDER BY id";
/// Every row `query` returns, every column.
fn rows(conn: &rusqlite::Connection, query: &str) -> Vec<Vec<rusqlite::types::Value>> {
let mut stmt = conn.prepare(query).unwrap();
let columns = stmt.column_count();
stmt.query_map([], |r| (0..columns).map(|i| r.get(i)).collect())
.unwrap()
.map(|row| row.unwrap())
.collect()
}
#[test]
fn a_failed_op_stays_queued_with_backoff() {
let db = Db::open_in_memory().unwrap();
@@ -2920,6 +3636,94 @@ fn clearing_a_stale_pending_mode_preserves_the_newer_intent() {
assert_eq!(db.sync_folder_get(id).unwrap().unwrap().pending_mode, None);
}
fn baseline_row(rel: &str) -> StoredSyncEntry {
StoredSyncEntry {
rel_path: rel.into(),
remote_uid: Some(format!("vol~{rel}")),
local_mtime: 1_700_000_000,
local_mtime_ns: Some(1_700_000_000_000_000_001),
local_size: 7,
remote_rev: Some("1700000000".into()),
remote_hash: Some("7".into()),
}
}
#[test]
fn moving_a_baseline_subtree_takes_its_rows_and_leaves_name_prefixed_siblings() {
let db = Db::open_in_memory().unwrap();
let from = db.sync_folder_add("/home/me/A", "vol~a", "share").unwrap();
let to = db.sync_folder_add("/home/me/B", "vol~b", "share").unwrap();
for rel in [
"docs",
"docs/x.txt",
"docs/sub/y.txt",
"docsextra.txt",
"other",
] {
db.sync_entry_upsert(from, &baseline_row(rel)).unwrap();
}
db.sync_entry_upsert(to, &baseline_row("inbox")).unwrap();
db.sync_entries_move(from, "docs", to, "inbox/docs")
.unwrap();
let left = db.sync_entries(from).unwrap();
let mut left: Vec<_> = left.keys().cloned().collect();
left.sort();
assert_eq!(left, ["docsextra.txt", "other"]);
let moved = db.sync_entries(to).unwrap();
let mut keys: Vec<_> = moved.keys().cloned().collect();
keys.sort();
assert_eq!(
keys,
[
"inbox",
"inbox/docs",
"inbox/docs/sub/y.txt",
"inbox/docs/x.txt"
]
);
// Only the path changes: the row still describes the same node on both sides.
assert_eq!(
moved["inbox/docs/x.txt"],
StoredSyncEntry {
rel_path: "inbox/docs/x.txt".into(),
..baseline_row("docs/x.txt")
}
);
}
#[test]
fn moving_a_baseline_subtree_within_one_folder_renames_it() {
let db = Db::open_in_memory().unwrap();
let folder = db.sync_folder_add("/home/me/A", "vol~a", "share").unwrap();
for rel in ["old", "old/f.txt", "new"] {
db.sync_entry_upsert(folder, &baseline_row(rel)).unwrap();
}
db.sync_entries_move(folder, "old", folder, "new/old")
.unwrap();
let mut keys: Vec<_> = db.sync_entries(folder).unwrap().into_keys().collect();
keys.sort();
assert_eq!(keys, ["new", "new/old", "new/old/f.txt"]);
}
#[test]
fn removing_a_baseline_subtree_keeps_name_prefixed_siblings() {
let db = Db::open_in_memory().unwrap();
let folder = db.sync_folder_add("/home/me/A", "vol~a", "share").unwrap();
for rel in ["docs", "docs/x.txt", "docsextra.txt"] {
db.sync_entry_upsert(folder, &baseline_row(rel)).unwrap();
}
db.sync_entries_remove_subtree(folder, "docs").unwrap();
let keys: Vec<_> = db.sync_entries(folder).unwrap().into_keys().collect();
assert_eq!(keys, ["docsextra.txt"]);
}
#[test]
fn mirror_uid_lookup_covers_the_root_and_synced_descendants_only() {
let db = Db::open_in_memory().unwrap();
@@ -4963,3 +5767,28 @@ fn retry_all_touches_only_ops_that_have_failed() {
assert_eq!(due(debounced), 7_000);
assert_eq!(due(parked), ops::PARK_UNTIL);
}
#[test]
fn a_queued_op_stops_existing_once_the_node_is_deleted() {
let db = Db::open_in_memory().unwrap();
let (id, _) = db
.enqueue_op(&PendingOp {
id: 0,
kind: OP_REVISION.to_string(),
uid: uid("journal").to_string(),
parent_uid: None,
name: None,
blob_path: Some("/staging/journal".to_string()),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
assert!(db.op_exists(id).unwrap());
let blobs = db.delete_ops_for_uid(&uid("journal").to_string()).unwrap();
assert_eq!(blobs, vec!["/staging/journal"]);
assert!(!db.op_exists(id).unwrap());
}
+10 -3
View File
@@ -55,9 +55,10 @@ fn systemctl_query(args: &[&str]) -> bool {
/// keyring checks). `--now` is a no-op on an active unit, so the daemon would
/// keep sleeping and the mount would not appear until the next poll fired.
/// Restarting makes the new session take effect immediately, and starts the
/// unit when it is not running at all.
/// unit when it is not running at all. `reenable` for the reason given on
/// [`enable`].
pub fn enable_start() -> bool {
let enabled = systemctl(&["enable"]);
let enabled = systemctl(&["reenable"]);
systemctl(&["restart"]) && enabled
}
@@ -97,8 +98,14 @@ pub fn is_enabled() -> bool {
/// Enable the service so it auto-starts on future logins, without necessarily
/// starting it now beyond what `enable` does. Used by the Settings "Start on
/// login" switch; pairs with [`disable`](disable).
///
/// `reenable` rather than `enable`: older installs were enabled
/// into `default.target.wants`, and a plain `enable` would add the new
/// `graphical-session.target.wants` link while leaving the old one to start the
/// unit too early. `reenable` drops every existing link and recreates them from
/// the current `[Install]` section.
pub fn enable() -> bool {
systemctl(&["enable"])
systemctl(&["reenable"])
}
/// Disable auto-start without stopping the running mount. Used by the Settings
+4
View File
@@ -15,6 +15,7 @@ zip.workspace = true
sha1.workspace = true
proton-drive-rs.workspace = true
futures.workspace = true
async-trait.workspace = true
tokio.workspace = true
tracing.workspace = true
fuser.workspace = true
@@ -24,3 +25,6 @@ base64.workspace = true
image.workspace = true
notify = "8"
parking_lot.workspace = true
[dev-dependencies]
tracing-subscriber.workspace = true
+102 -17
View File
@@ -9,7 +9,7 @@ use crate::shutdown::Shutdown;
///
/// Returns whether this mount actually held the node, so the caller can evict
/// its content blob once if *any* mount did.
fn forget_and_notify(st: &mut State, notify: &mut NotifyBatch, uid: &NodeUid) -> bool {
pub(crate) fn forget_and_notify(st: &mut State, notify: &mut NotifyBatch, uid: &NodeUid) -> bool {
// Capture the inode before `forget` clears the uid mapping.
let child = st.by_uid.get(uid).copied();
let Some((parent, name)) = st.forget(uid) else {
@@ -97,15 +97,13 @@ fn event_serializes_shared_publication(event: &DriveEvent, own_volume: &VolumeId
/// entry forces the next `lookup`/`readdir` to re-enumerate from the remote, so
/// most events only need to invalidate listings rather than re-fetch eagerly.
///
/// Applied to **every** mounted inode space, not just the primary one. This task
/// is per-daemon while node state is per-mount, and a `DriveEvent` names a uid,
/// not a mount — so a file trashed from another device has to be withdrawn from
/// whichever of our sessions is showing it, which for anything under a sync
/// folder is a [`Core::fork_state`] fork rather than `core.state`. Reaching for
/// the primary state alone left forks serving a deleted file indefinitely, the
/// read-side twin of `docs/BUGS.md` B74. Inode numbers are per-mount, so each
/// mount must be notified through its **own** channel; that pairing is what
/// [`Core::for_each_mount`] exists to preserve.
/// Applied to the tree every mount shares, and told to **every** mounted
/// session. A `DriveEvent` names a uid, not a mount, so a file trashed from
/// another device has to be withdrawn from whichever of our sessions is showing
/// it — which for anything under a sync folder is a [`Core::fork_state`] fork.
/// Telling the primary session alone left forks serving a deleted file, the
/// read-side twin of `docs/BUGS.md` B74. [`Core::for_each_mount`] sends what
/// the kernel needs to hear to every session.
fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdfs_core::Result<()> {
let foreign_delete = is_foreign_node_delete(event, &core.primary_root_uid.volume_id);
let serializes_shared_publication =
@@ -233,6 +231,48 @@ fn acknowledge_applied_event(
Ok(id)
}
/// How many applied node-update ids [`RecentEvents`] remembers. Far more than a
/// replay can reach back: one comes from a batch whose last event was not its
/// newest, so it repeats at most what that batch held.
const RECENT_EVENTS: usize = 4096;
/// The node updates applied lately, by event id, so one that comes round again
/// is skipped rather than applied twice.
///
/// The feed can deliver the same event twice in a batch, and a batch whose
/// events come out of order leaves the cursor short of its newest event, so the
/// next poll replays what lay past it. An update applied twice is not harmless:
/// the first copy of our own change claims the echo [`Core::note_self_change`]
/// recorded, and the second, unclaimed, reads as someone else's edit and evicts
/// the bytes a partial write is about to gap-fill from (docs/BUGS.md B123).
///
/// Only node updates: a delete is idempotent, and a continuity loss names the
/// cursor it lost, which is an id already applied.
#[derive(Default)]
struct RecentEvents {
ids: HashSet<DriveEventId>,
order: VecDeque<DriveEventId>,
}
impl RecentEvents {
/// Record `event` as applied, reporting whether it was already.
fn repeat(&mut self, event: &DriveEvent) -> bool {
let DriveEvent::NodeUpdated { id, .. } = event else {
return false;
};
if !self.ids.insert(id.clone()) {
return true;
}
self.order.push_back(id.clone());
if self.order.len() > RECENT_EVENTS
&& let Some(old) = self.order.pop_front()
{
self.ids.remove(&old);
}
false
}
}
/// The folders one batch of events changed, collected so the listings are
/// dropped once at the end of the batch rather than once per event.
///
@@ -320,10 +360,10 @@ fn flush_dirty_parents(core: &Core, dirty: &DirtyParents) {
/// only on fatal error.
///
/// Takes the whole `Core` rather than the primary mount's pieces: an event has
/// to be applied to every mounted inode space, and the registry that enumerates
/// them lives on the `Core` (see [`apply_event`]).
/// to reach every mounted session, and the registry that enumerates them lives
/// on the `Core` (see [`apply_event`]).
pub(super) async fn run_event_sync(
client: ProtonDriveClient,
client: Arc<dyn DriveApi>,
scope: DriveEventScopeId,
core: Core,
) {
@@ -377,10 +417,12 @@ pub(super) async fn run_event_sync(
};
info!(?cursor, "event sync started");
let mut recent = RecentEvents::default();
loop {
tokio::time::sleep(POLL_INTERVAL).await;
let events = match client.enumerate_events(&scope, cursor.as_ref()).await {
core.next_poll(POLL_INTERVAL).await;
let events = match link::bounded(client.enumerate_events(&scope, cursor.as_ref())).await {
Ok(events) => events,
Err(e) if core.lost_link(&e, "poll events") => continue,
Err(e) => {
warn!(error = %e, "event poll failed; retrying after interval");
continue;
@@ -396,6 +438,11 @@ pub(super) async fn run_event_sync(
// did apply.
let mut dirty = DirtyParents::default();
for event in &events {
if recent.repeat(event) {
debug!(id = %event.id().as_str(), "skipping an event already applied");
cursor = Some(event.id().clone());
continue;
}
// Converge the SDK's own caches (folder keys, entity cache) on the
// server before applying the event to our tree. Without this, a node
// re-keyed/moved by another client keeps a stale key in the SDK for
@@ -471,9 +518,10 @@ pub(super) async fn run_photos_event_sync(core: Core) {
info!(?cursor, "photo event sync started");
loop {
tokio::time::sleep(POLL_INTERVAL).await;
let events = match photos.enumerate_events(&scope, cursor.as_ref()).await {
core.next_poll(POLL_INTERVAL).await;
let events = match link::bounded(photos.enumerate_events(&scope, cursor.as_ref())).await {
Ok(events) => events,
Err(error) if core.lost_link(&error, "poll photo events") => continue,
Err(error) => {
warn!(%error, "photo event poll failed; retrying after interval");
continue;
@@ -715,6 +763,43 @@ mod tests {
}
}
#[test]
fn a_node_update_seen_again_is_skipped_but_other_events_are_not() {
let update = |id: &str| DriveEvent::NodeUpdated {
id: DriveEventId::from(id),
node_uid: uid("file"),
parent_node_uid: None,
is_trashed: false,
is_shared: false,
};
let mut recent = RecentEvents::default();
assert!(!recent.repeat(&update("e1")));
assert!(!recent.repeat(&update("e2")));
assert!(recent.repeat(&update("e1")), "a duplicate or replay");
let lost = DriveEvent::ContinuityLost {
id: DriveEventId::from("e2"),
};
assert!(!recent.repeat(&lost), "names the cursor, which was applied");
}
#[test]
fn recent_events_forget_the_oldest_past_their_bound() {
let update = |n: usize| DriveEvent::NodeUpdated {
id: DriveEventId::from(format!("e{n}")),
node_uid: uid("file"),
parent_node_uid: None,
is_trashed: false,
is_shared: false,
};
let mut recent = RecentEvents::default();
for n in 0..=RECENT_EVENTS {
assert!(!recent.repeat(&update(n)));
}
assert_eq!(recent.ids.len(), RECENT_EVENTS);
assert!(!recent.repeat(&update(0)), "the oldest was forgotten");
assert!(recent.repeat(&update(RECENT_EVENTS)));
}
#[test]
fn foreign_delete_invalidates_an_earlier_refresh_generation() {
let own_volume = VolumeId::from("own");
+87 -28
View File
@@ -163,8 +163,8 @@ fn read_request_line(reader: &mut impl BufRead) -> std::io::Result<Option<String
/// Turn a CLI-supplied path into the mount that owns it and the path relative
/// to *that* mount's root.
///
/// [`rel_to_mount`] is the older, primary-only reading, and it is what every
/// mutating request still uses. It rejects anything outside the primary
/// [`rel_to_mount`] is the older, primary-only reading, and it is what most
/// requests still use. It rejects anything outside the primary
/// mountpoint, which leaves a secondary on-demand mount — a synced folder
/// switched out of mirror mode — with no way to be named at all: `pdfs ls` and
/// `pdfs refresh` both answered "is not under the mountpoint", so a folder
@@ -185,11 +185,14 @@ fn route_to_mount(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
rel_to_mount(mountpoint, path).map(|rel| (core.clone(), rel))
}
/// [`route_to_mount`] for a conflict copy. The listing names a copy inside a
/// synced folder by its absolute local path (`docs/BUGS.md` B99), so the
/// resolve has to run against the on-demand mount serving it. A mirror folder
/// has no mount to run against: its copy is an ordinary local file.
fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
/// [`route_to_mount`] for a request that changes what it names: trash, rename,
/// create a folder, resolve a conflict. Each has to run against the on-demand
/// mount serving the path, and `pdfs rm ~/Downloads/x` answered "is not under
/// the mountpoint" while `pdfs move` took the same path. The conflict listing
/// names a copy in a synced folder by its absolute local path (`docs/BUGS.md`
/// B99), so resolving one needs this too. A mirror folder has no mount to run
/// against: a file there is an ordinary local file.
fn route_change(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
let routed = route_to_mount(core, mountpoint, path).map_err(|e| {
let p = Path::new(path);
let mirrored = core
@@ -200,7 +203,7 @@ fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
.any(|folder| folder.mode == "mirror" && p.starts_with(&folder.local_path));
if mirrored {
CoreError::invalid(format!(
"{path} is in a mirrored folder; rename or delete the local file there"
"{path} is in a mirrored folder; change the local file there instead"
))
} else {
e
@@ -217,6 +220,27 @@ fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
}
}
/// [`route_change`] for a request that trashes or renames the node it names.
/// A mount's root is My files or a whole synced folder. An on-demand folder's
/// root has a parent on Drive, so nothing further down would stop `pdfs rm
/// ~/Downloads` from trashing the folder out from under its own mount.
fn route_node(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
let routed = route_change(core, mountpoint, path)?;
if below_root(&routed.1) {
Ok(routed)
} else {
Err(CoreError::invalid(format!(
"{path} is the root of a synced location and cannot be changed"
)))
}
}
/// Whether a mount-relative path names something below the mount's root.
fn below_root(rel: &Path) -> bool {
rel.components()
.any(|component| matches!(component, Component::Normal(_)))
}
/// Turn a CLI-supplied path into a mountpoint-relative path. An absolute path
/// must live under `mountpoint`; a relative path is taken as already relative to
/// the mount root.
@@ -291,6 +315,9 @@ impl Drop for RefreshGuard {
pub(crate) fn status_response(core: &Core, username: &str, mountpoint: &Path) -> CtlResponse {
let pins = core.cache.list_pins();
let queued = core.db.pending_op_counts().unwrap_or_default();
// Read after the queue: the drain counts an upload as landing before it
// retires the op, so one of the two always has it (B119).
let landing = core.landing_uploads.load(Ordering::SeqCst);
let staging = core.cache.staging_usage();
CtlResponse::Status {
parked_uploads: queued.parked.max(0) as u64,
@@ -307,7 +334,7 @@ pub(crate) fn status_response(core: &Core, username: &str, mountpoint: &Path) ->
budget: core.cache.budget(),
pins,
online: core.online.load(Ordering::Relaxed),
pending_uploads: queued.uploads.max(0) as u64,
pending_uploads: queued.uploads.max(0) as u64 + landing,
pending_changes: queued.changes.max(0) as u64,
}
}
@@ -959,8 +986,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
},
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::Rename { path, new_name }) => match rel_to_mount(mountpoint, &path) {
Ok(rel) => match core.rename(&rel, &new_name) {
Ok(CtlRequest::Rename { path, new_name }) => match route_node(core, mountpoint, &path) {
Ok((core, rel)) => match core.rename(&rel, &new_name) {
Ok(name) => {
core.log_activity(ActivityKind::Rename, &name, format!("was {path}"), true);
CtlResponse::Ok {
@@ -974,6 +1001,24 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
},
Err(e) => CtlResponse::error(e),
},
// Two absolute paths may name two different locations, which only
// `move_between` can resolve; a relative pair keeps the old meaning.
Ok(CtlRequest::Move { path, new_parent })
if Path::new(&path).is_absolute() && Path::new(&new_parent).is_absolute() =>
{
match core.move_between(Path::new(&path), Path::new(&new_parent)) {
Ok(name) => {
core.log_activity(ActivityKind::Move, &name, format!("to {new_parent}"), true);
CtlResponse::Ok {
message: format!("moved {name}"),
}
}
Err(e) => {
core.log_activity(ActivityKind::Move, &path, &e, false);
CtlResponse::error(e)
}
}
}
Ok(CtlRequest::Move { path, new_parent }) => {
match (
rel_to_mount(mountpoint, &path),
@@ -999,8 +1044,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
(Err(e), _) | (_, Err(e)) => CtlResponse::error(e),
}
}
Ok(CtlRequest::Delete { path }) => match rel_to_mount(mountpoint, &path) {
Ok(rel) => match core.delete(&rel) {
Ok(CtlRequest::Delete { path }) => match route_node(core, mountpoint, &path) {
Ok((core, rel)) => match core.delete(&rel) {
Ok(name) => {
core.log_activity(ActivityKind::Trash, &name, "", true);
CtlResponse::Ok {
@@ -1014,21 +1059,23 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
},
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::CreateFolder { parent, name }) => match rel_to_mount(mountpoint, &parent) {
Ok(parent_rel) => match core.create_folder(&parent_rel, &name) {
Ok(name) => {
core.log_activity(ActivityKind::CreateFolder, &name, "", true);
CtlResponse::Ok {
message: format!("created folder {name}"),
Ok(CtlRequest::CreateFolder { parent, name }) => {
match route_change(core, mountpoint, &parent) {
Ok((core, parent_rel)) => match core.create_folder(&parent_rel, &name) {
Ok(name) => {
core.log_activity(ActivityKind::CreateFolder, &name, "", true);
CtlResponse::Ok {
message: format!("created folder {name}"),
}
}
}
Err(e) => {
core.log_activity(ActivityKind::CreateFolder, &name, &e, false);
CtlResponse::error(e)
}
},
Err(e) => CtlResponse::error(e),
},
Err(e) => {
core.log_activity(ActivityKind::CreateFolder, &name, &e, false);
CtlResponse::error(e)
}
},
Err(e) => CtlResponse::error(e),
}
}
Ok(CtlRequest::UploadPaths { parent, sources }) => {
match rel_to_mount(mountpoint, &parent) {
// Ack immediately and upload on a background thread: a directory tree
@@ -1345,7 +1392,7 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::ResolveConflict { path, keep }) => {
match route_conflict(core, mountpoint, &path) {
match route_change(core, mountpoint, &path) {
Ok((core, rel)) => match core.resolve_conflict(&rel, &keep) {
Ok(message) => CtlResponse::Ok { message },
Err(e) => CtlResponse::error(e),
@@ -1846,6 +1893,18 @@ mod request_limit_tests {
assert!(rel_to_mount(mountpoint, "Photos/../../escape").is_err());
assert!(rel_to_mount(mountpoint, "/mnt/elsewhere").is_err());
}
/// `pdfs rm` and `pdfs rename` reach on-demand mounts, whose root is a whole
/// synced folder with a parent on Drive. Only a path below a root may get
/// through to the trash or rename call.
#[test]
fn only_a_path_below_a_mount_root_names_a_node() {
assert!(below_root(Path::new("Photos")));
assert!(below_root(Path::new("./Photos/2026")));
assert!(!below_root(Path::new("")));
assert!(!below_root(Path::new(".")));
assert!(!below_root(Path::new("./.")));
}
}
#[cfg(test)]
+14 -24
View File
@@ -27,7 +27,7 @@ use tracing::{info, warn};
use super::sync::{self, base_name};
use super::{
Core, SecondaryInsertRejection, SecondaryMount, State, SwitchBlocked, clear_stale_mount,
Core, SecondaryInsertRejection, SecondaryMount, SwitchBlocked, clear_stale_mount,
device_type_str, dir_is_empty, evict_dir_contents, fuse_connection_id, is_stale_mount,
now_secs, parse_uid, spawn_session, sync_folder_info, this_hostname,
};
@@ -363,14 +363,14 @@ impl Core {
) -> CoreResult<Option<NodeUid>> {
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(root_uid))
.block_on(self.drive.enumerate_folder_children_node_uids(root_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() {
return Ok(None);
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&uids))
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?;
Ok(nodes
.into_iter()
@@ -446,10 +446,7 @@ impl Core {
}
None => self
.rt
.block_on(
self.client
.create_folder(&root_uid, &name, Some(now_secs())),
)
.block_on(self.drive.create_folder(&root_uid, &name, Some(now_secs())))
.map_err(|e| CoreError::from_api(&e, &format!("create device folder {name}")))?,
};
@@ -603,7 +600,7 @@ impl Core {
&& let Some(uid) = parse_uid(&folder.remote_uid)
&& let Err(e) = self
.rt
.block_on(self.client.trash_nodes(&[uid]))
.block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit)
{
warn!(id, error = %e, "delete remote device folder failed");
@@ -619,23 +616,16 @@ impl Core {
};
}
/// A sibling Core that shares this one's client/rt/cache/db (and transfer,
/// activity, mount registries) but gets a **fresh, empty `State`** — its own
/// inode space starting at [`ROOT_INO`]. Used to root a secondary FUSE session
/// at an `ondemand` sync folder without colliding with the main mount's inodes
/// (devices.md Phase 3).
/// A sibling Core for a secondary FUSE session rooted at an `ondemand` sync
/// folder (devices.md Phase 3). It shares everything with this one,
/// the tree included, except what belongs to a session: its root and its
/// kernel channel, both filled in when the session is built.
pub(crate) fn fork_state(&self) -> Core {
let mut fork = self.clone();
fork.primary = false;
let share_access = self.state().share_access.clone();
fork.state = Arc::new(Mutex::new(State::new(self.db.clone(), share_access, 2)));
// A fresh inode space needs a fresh notification channel: this fork's
// session is the only one that knows these inodes, so it must be the one
// notified about them. Filled in by `spawn_ondemand_mount`.
fork.root = Arc::new(OnceLock::new());
fork.notifier = Arc::new(OnceLock::new());
fork.session_live = Arc::new(std::sync::atomic::AtomicBool::new(false));
// Size upgrades are keyed by inode, which is per-fork too.
fork.size_upgrades = Arc::new(Mutex::new(HashMap::new()));
fork
}
@@ -858,7 +848,7 @@ impl Core {
})?;
let root = self
.rt
.block_on(self.client.enumerate_nodes(std::slice::from_ref(&root_uid)))
.block_on(self.drive.enumerate_nodes(std::slice::from_ref(&root_uid)))
.map_err(|e| SwitchBlocked::Failed(format!("fetch remote root: {e}")))?
.into_iter()
.next()
@@ -937,8 +927,8 @@ impl Core {
}
}
/// Spawn a secondary FUSE session rooted at `root` over `local` on a forked
/// inode space. Clears any stale kernel mount first (a crashed run can leave
/// Spawn a secondary FUSE session rooted at `root` over `local` from a
/// [`Core::fork_state`] sibling. Clears any stale kernel mount first (a crashed run can leave
/// one, which would fail the fresh mount with EBUSY).
pub(crate) fn spawn_ondemand_mount(
&self,
@@ -1017,7 +1007,7 @@ impl Core {
};
let root = match self
.rt
.block_on(self.client.enumerate_nodes(std::slice::from_ref(&root_uid)))
.block_on(self.drive.enumerate_nodes(std::slice::from_ref(&root_uid)))
{
Ok(v) => match v.into_iter().next() {
Some(n) => n,
+512 -130
View File
@@ -24,28 +24,27 @@ use std::collections::HashMap;
use std::fs::File;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::{Duration, Instant};
use parking_lot::Mutex;
use futures::StreamExt;
use pdfs_core::batch;
use pdfs_core::cache::{Baseline, StagedWrite};
use pdfs_core::control::{ActivityKind, TransferDirection};
use pdfs_core::db::{
OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, PARK_EXPIRY_MS, PendingOp, RenameMeta,
CreateLanding, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, PARK_EXPIRY_MS,
PendingOp, RenameMeta,
};
use proton_drive_rs::proton_sdk::error::ProtonError;
use proton_drive_rs::proton_sdk::ids::NodeUid;
use proton_drive_rs::{Node, NodeMoveItem};
use proton_drive_rs::{Node, NodeKind};
use tracing::{debug, error, info, warn};
use super::state::Intervals;
use super::state::{Intervals, PendingRevision};
use super::transfers::CountingReader;
use super::{
Core, DRAIN_BACKOFF_MAX, DRAIN_BACKOFF_MIN, DRAIN_IDLE_POLL, DRAIN_REVISION_DEBOUNCE,
DRAIN_REVISION_DEBOUNCE_MAX, ROOT_INO, UPLOAD_TIME_MEMORY, WriteAuthority, conflict_name,
DRAIN_REVISION_DEBOUNCE_MAX, UPLOAD_TIME_MEMORY, WriteAuthority, conflict_name,
is_already_exists, is_gone, is_local_uid_str, media_type_for, node_revision_id, node_size,
now_millis, now_secs, parse_node_uid,
};
@@ -113,6 +112,23 @@ impl Drop for UploadCancel {
}
}
/// One upload counted in [`Core::landing_uploads`] from its op's retirement
/// until its bytes are cached, and uncounted when dropped, early return or not.
struct LandingUpload(Arc<AtomicU64>);
impl LandingUpload {
fn begin(count: &Arc<AtomicU64>) -> Self {
count.fetch_add(1, Ordering::SeqCst);
Self(count.clone())
}
}
impl Drop for LandingUpload {
fn drop(&mut self) {
self.0.fetch_sub(1, Ordering::SeqCst);
}
}
/// The debounce arithmetic of [`Core::revision_debounce`], without the map:
/// the last measured upload time for this node, bounded on both sides.
///
@@ -659,6 +675,15 @@ impl Core {
return Ok(());
}
};
if let Some(meta) = op.meta_json.as_deref() {
let meta: RenameMeta = serde_json::from_str(meta)?;
if moved_elsewhere(&meta, &node, &parent, &name) {
// Someone moved or renamed it while ours was queued. Ours still
// wins, as the last change does on Drive.
info!(%uid, from = %node.name, to = %name,
"node moved remotely while its rename was queued");
}
}
let landed = if node.parent_uid.as_ref() == Some(&parent) {
self.drain_rename_in_place(&uid, &node.name, &name)?
} else {
@@ -667,15 +692,18 @@ impl Core {
// no window for the requirements to go stale between two calls —
// the race B46 queued this op to get away from.
let target = (node.name != name).then_some(name.as_str());
match self.move_renaming(&uid, &parent, target) {
match self.move_rename_remote(&uid, &parent, target) {
Ok(()) => name.clone(),
// The destination holds that name already. Landing under a
// *different* name is the non-destructive resolution: it neither
// clobbers their file nor drops ours, and it is visible.
Err(e) if is_already_exists(&e) => {
if self.db.has_pending_trash_named(&name)? {
return Err(held_by_queued_trash(&name));
}
let alt = conflict_name(&name, now_secs());
warn!(%uid, name, alt, "destination already holds that name; using a conflict name");
self.move_renaming(&uid, &parent, Some(&alt))?;
self.move_rename_remote(&uid, &parent, Some(&alt))?;
self.adopt_drained_name(&uid, &alt);
self.log_activity(
ActivityKind::Rename,
@@ -696,8 +724,8 @@ impl Core {
let landed = match self.drain_rename_in_place(&uid, &node.name, &name) {
Ok(landed) => landed,
// The node went with it; nothing is left to rename.
Err(e) if is_gone(&e) => node.name.clone(),
Err(e) => return Err(e.into()),
Err(e) if is_gone(e.as_ref()) => node.name.clone(),
Err(e) => return Err(e),
};
self.log_activity(
ActivityKind::Rename,
@@ -712,7 +740,6 @@ impl Core {
}
};
self.db.delete_op(op.id)?;
self.note_self_change(&uid);
info!(%uid, name = %landed, "pending rename landed");
Ok(())
}
@@ -725,16 +752,19 @@ impl Core {
uid: &NodeUid,
current: &str,
name: &str,
) -> Result<String, ProtonError> {
) -> Result<String, Box<dyn std::error::Error>> {
if current == name {
return Ok(current.to_string());
}
match self.rt.block_on(self.client.rename_node(uid, name, None)) {
match self.rename_remote(uid, name) {
Ok(()) => Ok(name.to_string()),
Err(e) if is_already_exists(&e) => {
if self.db.has_pending_trash_named(name)? {
return Err(held_by_queued_trash(name));
}
let alt = conflict_name(name, now_secs());
warn!(%uid, name, alt, "rename target name is taken; using a conflict name");
self.rt.block_on(self.client.rename_node(uid, &alt, None))?;
self.rename_remote(uid, &alt)?;
self.adopt_drained_name(uid, &alt);
self.log_activity(
ActivityKind::Rename,
@@ -744,35 +774,10 @@ impl Core {
);
Ok(alt)
}
Err(e) => Err(e),
Err(e) => Err(e.into()),
}
}
/// Move `uid` under `parent` in one request, renaming it to `target_name` on
/// the way when one is given.
fn move_renaming(
&self,
uid: &NodeUid,
parent: &NodeUid,
target_name: Option<&str>,
) -> Result<(), ProtonError> {
let item = NodeMoveItem {
uid: uid.clone(),
target_name: target_name.map(str::to_string),
};
self.rt.block_on(async {
let mut outcomes =
std::pin::pin!(self.client.move_nodes_streaming(vec![item], parent.clone()));
match outcomes.next().await {
Some(Ok((_, outcome))) => outcome,
Some(Err(e)) => Err(e),
None => Err(ProtonError::invalid_operation(format!(
"move of {uid} reported no outcome"
))),
}
})
}
/// Apply a queued trash to the remote.
///
/// A node that is already gone is a success, not a failure: the outcome the
@@ -783,7 +788,7 @@ impl Core {
let name = op.name.clone().unwrap_or_else(|| op.uid.clone());
match self
.rt
.block_on(self.client.trash_nodes(std::slice::from_ref(&uid)))
.block_on(self.drive.trash_nodes(std::slice::from_ref(&uid)))
.and_then(batch::into_unit)
{
Ok(()) => {}
@@ -793,7 +798,6 @@ impl Core {
Err(e) => return Err(e.into()),
}
self.db.complete_trash_op(op.id, &uid)?;
self.own_sealed_revs.lock().remove(&uid);
if let Err(e) = self.db.clear_own_sealed_rev(&uid.to_string()) {
debug!(%uid, error = %e, "clearing the sealed-revision record failed");
}
@@ -842,8 +846,27 @@ impl Core {
// reachable only because the op waited, which is the whole point of the
// queue. Never overwrite theirs and never drop ours: land under a
// conflict name, exactly as the sync engine does.
//
// Two things can hold the name that are not someone else's file. The
// node a create made before the network dropped its answer is ours: the
// create was queued because nobody heard back, so adopt it. And a node
// whose trash is still queued is about to let go of it.
let mut name = wanted.clone();
let mut home = parent.clone();
let mut adopted = false;
let mut real = self.create_drained_node(op, &parent, &name);
if real.as_ref().is_err_and(|e| is_already_exists(e.as_ref()))
&& let Some(twin) = self.adoptable_twin(op, &parent, &wanted)?
{
info!(%local, %twin, wanted, "the name is held by our own unanswered create; adopting it");
real = Ok(twin);
adopted = true;
}
if real.as_ref().is_err_and(|e| is_already_exists(e.as_ref()))
&& self.db.has_pending_trash_named(&wanted)?
{
return Err(held_by_queued_trash(&wanted));
}
if real.as_ref().is_err_and(|e| is_already_exists(e.as_ref())) {
name = conflict_name(&wanted, now_secs());
warn!(%local, wanted, name, "name is taken remotely; creating under a conflict name");
@@ -872,6 +895,7 @@ impl Core {
name = conflict_name(&wanted, now_secs());
real = self.create_drained_node(op, &root, &name);
}
home = root;
if real.is_ok() {
self.log_activity(
ActivityKind::Upload,
@@ -886,20 +910,84 @@ impl Core {
// Retire the op before touching anything else: if we crash here the node
// exists remotely and the local placeholder is reconciled by the event
// sync, whereas a surviving op would create the file a second time.
self.db.delete_op(op.id)?;
self.adopt_real_uid(&local, &real)?;
// An adopted node was made empty, so a blob this op carries has not been
// uploaded: it goes on as a revision of the node instead.
let uploaded = (!adopted).then_some(op.blob_path.as_deref()).flatten();
let landing = CreateLanding {
local: &local.to_string(),
real: &real.to_string(),
sent: (parent_str, &wanted),
landed: (&home.to_string(), &name),
};
let newer = self.retire_create(op, uploaded, &landing, &local, &real)?;
// The uploaded blob is the new file's content, so it becomes the cached
// content, as for a revision. A partial write fills its gaps from it;
// without it the next write open was refused (`docs/BUGS.md` B123).
let landed: Option<StagedWrite> = uploaded
.filter(|_| !newer)
.and(op.meta_json.as_deref())
.and_then(|json| serde_json::from_str(json).ok());
self.adopt_real_uid(&local, &real, |node| {
if let (Some(blob), Some(meta)) = (uploaded, &landed)
&& node_size(node) == meta.len
&& keeps_landed_upload(self.cache.is_pinned(&real), meta.len, self.cache.budget())
{
let _ =
self.cache
.store_file(&real, node.modification_time, meta.len, Path::new(blob));
}
})?;
// The feed will report this create back to us; the tree already has it
// under its real uid, so that event is ours to ignore (`Core::self_changes`).
self.note_self_change(&real);
if let Some(blob) = op.blob_path.as_deref() {
// The uploaded blob is done with either way: a write that replaced it
// while it was on the wire already discarded it when it attached.
if let Some(blob) = uploaded {
self.cache.discard_staged(Path::new(blob));
}
self.pending.lock().remove(&local);
if newer {
info!(%local, %real, name, "a write landed during the create; queued it as a revision");
self.wake_drain();
}
self.log_activity(ActivityKind::Upload, &name, "created", true);
info!(%local, %real, name, kind = %op.kind, "pending create landed");
Ok(())
}
/// The node already holding `name` under `parent`, if it is the one an
/// earlier attempt at this op made before its answer was lost.
///
/// A create that timed out may still have landed; the op was queued because
/// the caller never heard back. Without this, its replay forked a
/// `(sync-conflict)` copy of a file nobody else had touched.
fn adoptable_twin(
&self,
op: &PendingOp,
parent: &NodeUid,
name: &str,
) -> Result<Option<NodeUid>, Box<dyn std::error::Error>> {
let uids = self
.rt
.block_on(self.drive.enumerate_folder_children_node_uids(parent))?;
let children = self.rt.block_on(self.drive.enumerate_nodes_light(&uids))?;
let Some(twin) = children
.into_iter()
.find(|node| node.name == name && !node.trashed)
else {
return Ok(None);
};
// The light listing carries no file size, and an adoptable file is
// exactly an empty one.
let twin = match twin.kind {
NodeKind::Folder => twin,
NodeKind::File { .. } => match self.fetch_node_remote(&twin.uid)? {
Some(node) => node,
None => return Ok(None),
},
};
Ok(adoptable(op.kind == OP_MKDIR, &twin, op.created_at).then_some(twin.uid))
}
/// Make one queued `create`/`mkdir` real under a given name, and hand back
/// the API's own error so the caller can tell a name clash from a failure.
///
@@ -915,7 +1003,7 @@ impl Core {
true => {
Ok(self
.rt
.block_on(self.client.create_folder(parent, name, Some(now_secs())))?)
.block_on(self.drive.create_folder(parent, name, Some(now_secs())))?)
}
false => self.upload_created_file(op, parent, name),
}
@@ -930,7 +1018,7 @@ impl Core {
name: &str,
) -> Result<NodeUid, Box<dyn std::error::Error>> {
let Some(blob) = op.blob_path.as_deref() else {
return Ok(self.rt.block_on(self.client.upload_file(
return Ok(self.rt.block_on(self.drive.upload_file(
parent,
name,
media_type_for(name),
@@ -950,7 +1038,7 @@ impl Core {
.transfers
.begin(name, op.uid.clone(), TransferDirection::Upload, meta.len);
let reader = CountingReader::new(File::open(blob)?, &guard);
let uid = self.rt.block_on(self.client.upload_file_from(
let uid = self.rt.block_on(self.drive.upload_file_from(
parent,
name,
media_type_for(name),
@@ -963,39 +1051,155 @@ impl Core {
Ok(uid)
}
/// Retire a landed create's op and move its queued state to the real uid.
/// True when a newer write attached to the create while it was uploading,
/// which is now queued as a revision of `real`.
///
/// Runs under the `pending` lock, which `Core::enqueue_staged_write` also
/// holds from attaching a blob to a create until it records the pending
/// entry. So a write either attached before this ran and is carried over,
/// or finds the create gone and queues against the real uid itself. The
/// open write handles are repointed right after, for the same reason:
/// a handle released later must not queue against the placeholder.
fn retire_create(
&self,
op: &PendingOp,
uploaded: Option<&str>,
landing: &CreateLanding<'_>,
local: &NodeUid,
real: &NodeUid,
) -> Result<bool, Box<dyn std::error::Error>> {
let newer = {
let mut pending = self.pending.lock();
let newer = self.db.finish_create(op.id, uploaded, landing, |json| {
let mut meta: StagedWrite = serde_json::from_str(json).ok()?;
meta.uid = landing.real.to_string();
serde_json::to_string(&meta).ok()
})?;
pending.remove(local);
if let Some((blob, json)) = &newer {
let meta: StagedWrite = serde_json::from_str(json)?;
pending.insert(
real.clone(),
PendingRevision {
path: PathBuf::from(blob),
meta,
},
);
}
newer.is_some()
};
if !newer && let Some(blob) = uploaded {
self.cache_created(op, local, real, blob);
}
self.for_each_state(|st| {
if let Some(ino) = st.by_uid.remove(local) {
st.by_uid.insert(real.clone(), ino);
if let Some(e) = st.entries.get_mut(&ino) {
e.uid = real.clone();
}
// Its children still name the placeholder as their parent, and
// the next write of one would put it back on the row.
for kid in st.children.get(&ino).cloned().unwrap_or_default() {
if let Some(e) = st.entries.get_mut(&kid)
&& e.node.parent_uid.as_ref() == Some(local)
{
e.node.parent_uid = Some(real.clone());
}
}
}
for aw in st.active_writes.values_mut() {
if aw.uid == *local {
aw.uid = real.clone();
}
}
});
Ok(newer)
}
/// Cache a landed create's uploaded `blob` under `real`, keyed as the tree
/// has the file.
///
/// A write handle open on the file is repointed at `real` right after this,
/// and a release fills its gaps from the cache by the size and mtime it
/// opened over. Drive lists the new node a moment after the create, so
/// `adopt_real_uid` caches the bytes only once it can read the node back.
/// A release in between found nothing to fill from, and the next write open
/// was refused (docs/BUGS.md B131).
fn cache_created(&self, op: &PendingOp, local: &NodeUid, real: &NodeUid, blob: &str) {
let Some(meta) = op
.meta_json
.as_deref()
.and_then(|json| serde_json::from_str::<StagedWrite>(json).ok())
else {
return;
};
let mut mtime = None;
self.for_each_state(|st| {
if let Some(entry) = st.by_uid.get(local).and_then(|ino| st.entries.get(ino)) {
mtime = Some(entry.node.modification_time);
}
});
if let Some(mtime) = mtime
&& keeps_landed_upload(self.cache.is_pinned(real), meta.len, self.cache.budget())
{
let _ = self
.cache
.store_file(real, mtime, meta.len, Path::new(blob));
}
}
/// Swap a placeholder uid for the real one across everything that keyed off
/// it: queued children, the DB, the in-memory tree, and the caches.
///
/// The inode is deliberately kept, so anything already holding the file open
/// keeps working across the drain.
///
/// Applied to **every** live inode space, not just this Core's. A node
/// created inside an on-demand sync folder lives in that fork's `state`,
/// while the drain that lands its create belongs to the primary mount — so
/// reaching for `self.state` here found nothing and left the fork's entry on
/// its `local~` uid, which reads as an empty file for as long as the daemon
/// runs (`docs/BUGS.md` B74). A uid is unique across mounts, so at most one
/// of them matches and the others are no-ops.
/// Applied to the tree every mount shares. A node created inside an
/// on-demand sync folder once lived in that fork's own state, where the
/// primary mount's drain never looked, and stayed on its `local~` uid —
/// an empty file for as long as the daemon ran (`docs/BUGS.md` B74).
///
/// `adopt` sees the node as Drive made it, before anything is rebased onto
/// it.
pub(crate) fn adopt_real_uid(
&self,
local: &NodeUid,
real: &NodeUid,
adopt: impl FnOnce(&Node),
) -> Result<(), Box<dyn std::error::Error>> {
let node = self
.fetch_node(real)
// Drive answers a create before it lists the new node.
let mut node = self
.read_back(real)
.map_err(|e| self.errno_error(e, "fetch node"))?;
// Repoints queued children and node rows, and drops the placeholder row.
// A rename made while the create was on the wire is queued against the
// real uid (`Db::finish_create`). Until it lands, the tree keeps the
// name and folder the user gave it.
let renaming = self.db.has_pending_op(&real.to_string(), OP_RENAME)?;
// Before open handles are rebased onto the node, so one released in
// between finds what `adopt` keeps (`docs/BUGS.md` B123).
adopt(&node);
// Repoints queued children and node rows, and readdresses the placeholder row.
// A no-op after `retire_create`, which has already done both.
self.db
.remap_local_uid(&local.to_string(), &real.to_string())?;
self.for_each_state(|st| {
let Some(ino) = st.by_uid.remove(local) else {
if let Some(ino) = st.by_uid.remove(local) {
st.by_uid.insert(real.clone(), ino);
}
let Some(&ino) = st.by_uid.get(real) else {
return;
};
st.by_uid.insert(real.clone(), ino);
if renaming && let Some(e) = st.entries.get(&ino) {
node.name = e.node.name.clone();
node.parent_uid = e.node.parent_uid.clone();
}
let mut landed = node.clone();
st.keep_open_write_size(ino, &mut landed);
if let Some(e) = st.entries.get_mut(&ino) {
e.uid = real.clone();
e.node = node.clone();
e.node = landed;
}
// Where the op said the node goes and where it actually landed can
// differ — a conflict re-homes it — so the tree follows the parent
@@ -1024,6 +1228,15 @@ impl Core {
if let Err(e) = self.db.upsert_node(&node) {
warn!(%real, error = %e, "db upsert_node failed after remap");
}
// A handle still open on the placeholder was based on no revision at
// all: a local-clock mtime and no revision id. Released as it is, its
// write would land as a conflict copy of the empty file this create just
// made (`docs/BUGS.md` B113). Rebased before the restamp below, so a
// release that races this is caught there.
self.rebase_open_writes(real, &node);
// A write carried over from the create was made against no revision at
// all; the one just created is what it now replaces.
self.rebaseline_pending(real, &node);
if node.is_folder() {
// It was recorded as listed while local (it was empty and had nothing
// to enumerate). That still holds: its queued children re-intern under
@@ -1072,16 +1285,11 @@ impl Core {
// instead of forking — so the finished bytes win the name rather than
// being exiled to a `(sync-conflict)` copy.
let remote_rev = node_revision_id(&node);
// In memory first, then the table it is written through to: the map is
// this process's cache of it, and after a restart only the table has
// the answer (which is the whole reason it is persisted).
let own_rev = match self.own_sealed_revs.lock().get(uid).cloned() {
Some(rev) => Some(rev),
None => self
.db
.own_sealed_rev(&uid.to_string(), now_millis())
.unwrap_or_default(),
};
// The table, so the answer survives a restart (`own_sealed_rev`).
let own_rev = self
.db
.own_sealed_rev(&uid.to_string(), now_millis())
.unwrap_or_default();
if is_own_self_supersede(meta.complete, remote_rev.as_deref(), own_rev.as_deref()) {
debug!(%uid, ?remote_rev,
"queued write chains onto our own sealed revision; not a conflict");
@@ -1151,7 +1359,7 @@ impl Core {
self.transfers
.begin(&alt, meta.uid.clone(), TransferDirection::Upload, meta.len);
let reader = CountingReader::new(File::open(blob)?, &guard);
self.rt.block_on(self.client.upload_file_from(
self.rt.block_on(self.drive.upload_file_from(
&parent,
&alt,
media_type_for(&alt),
@@ -1348,25 +1556,13 @@ impl Core {
/// A node's parent uid and name, as the tree currently has them.
pub(crate) fn node_place(&self, uid: &NodeUid) -> Option<(NodeUid, String)> {
// Across every mount: the drain runs on the primary Core, but a node
// written inside a sync folder is interned in that fork's inode space,
// so asking `self.state` alone always missed it and fell through to the
// DB. That still gave the right answer — the fallback below exists for
// exactly this shape of miss — but it went to disk for something already
// in memory, and only *because* of the fallback was it not a bug.
let mut found = None;
self.for_each_state(|st| {
if found.is_some() {
return;
}
{
let st = self.state();
if let Some(entry) = st.by_uid.get(uid).and_then(|ino| st.entries.get(ino))
&& let Some(parent) = entry.node.parent_uid.clone()
{
found = Some((parent, entry.node.name.clone()));
return Some((parent, entry.node.name.clone()));
}
});
if found.is_some() {
return found;
}
// The in-memory tree only holds what has been walked to since the daemon
// started, and the drain routinely runs before anything has walked to
@@ -1401,12 +1597,13 @@ impl Core {
/// root yet, which is not where the drain runs; the drain must not panic
/// over it regardless, since that would stop the queue for good.
pub(crate) fn root_uid(&self) -> Option<NodeUid> {
let st = self.state();
st.entries.get(&ROOT_INO).map(|e| e.uid.clone())
let root = self.root_ino();
self.state().entries.get(&root).map(|e| e.uid.clone())
}
/// Upload a staged revision of a file the server already knows about.
pub(crate) fn drain_revision(&self, op: &PendingOp) -> Result<(), Box<dyn std::error::Error>> {
let op_started = Instant::now();
let blob = op
.blob_path
.clone()
@@ -1416,6 +1613,16 @@ impl Core {
let uid = parse_node_uid(&meta.uid).ok_or("staged write has an unparseable uid")?;
if let Some(reason) = self.revision_conflict(&uid, &meta)? {
// Trashing the file from the mount drops its queued write, but not
// from under a worker already holding it. That worker then finds the
// node trashed — by us — and would upload the withdrawn bytes as a
// conflict copy (a `recovered-…` file in the root when the tree has
// already forgotten the node). A write nobody queues any more is
// not a conflict.
if !self.db.op_exists(op.id)? {
debug!(%uid, reason, "queued write withdrawn while it drained; dropping it");
return Ok(());
}
return self.keep_as_conflict_copy(op, &blob, &meta, &uid, &reason);
}
@@ -1452,9 +1659,15 @@ impl Core {
// this is on the wire can stop it rather than paying for a revision the
// op it queued will immediately replace.
let cancel = self.begin_cancellable_upload(&uid);
// An unlink that withdrew this op before the flag above existed had no
// upload to cancel; the row is the record that it happened.
if !self.db.op_exists(op.id)? {
debug!(%uid, "queued write withdrawn before it went out; dropping it");
return Ok(());
}
let reader = CountingReader::new(File::open(&blob)?, &guard).with_cancel(cancel.flag());
let started = Instant::now();
let sent = self.rt.block_on(self.client.upload_new_revision_from(
let sent = self.rt.block_on(self.drive.upload_new_revision_from(
&uid,
reader,
meta.len as i64,
@@ -1480,25 +1693,46 @@ impl Core {
// is what the next write's debounce is scaled from.
self.record_upload_time(&uid, took);
// Still pending to whoever asks until the bytes are cached below, and
// counted before the op goes so the status never reads empty in between.
// An empty queue is how a script or the app knows a written file reads
// without the network; retiring the op alone left the read-back's round
// trip in which it did not (B119). The op itself has to go first: one
// restored after a crash would re-send a revision this daemon has not
// yet recorded as its own, and come back as a conflict copy.
let _landing = LandingUpload::begin(&self.landing_uploads);
// Retire the op before dropping the blob: a crash between the two leaves
// an orphaned file (harmless), whereas the reverse would leave a queued
// op pointing at nothing.
self.db.delete_op(op.id)?;
self.release_pending(&uid, &blob);
// The staged blob now matches the sealed revision, so a pinned file keeps
// it as its cached content rather than re-downloading what we just sent.
// Adopted by link, not by value: the blob is the whole file, and a pinned
// video read into a `Vec` here is an OOM. `discard_staged` below drops the
// staging name only — the cache keeps the inode.
if self.cache.is_pinned(&uid) {
let _ = self.cache.store_file(&uid, now_secs(), meta.len, &blob);
}
self.cache.discard_staged(&blob);
self.evict_reader(&uid);
self.refresh_after_upload(&uid);
// The staged blob now matches the sealed revision, so it becomes the
// file's cached content, stamped like the revision just read back.
// Reading the file again then needs no network, which an editor or a
// build does at once and a flaky link cannot always give (`docs/BUGS.md`
// B119). Adopted by link, not by value: the blob is the whole file, and
// a video read into a `Vec` here is an OOM. `discard_staged` below drops
// the staging name only — the cache keeps the inode.
self.refresh_after_upload(&uid, &blob, |node| {
if node_size(node) == meta.len
&& keeps_landed_upload(self.cache.is_pinned(&uid), meta.len, self.cache.budget())
{
let _ = self
.cache
.store_file(&uid, node.modification_time, meta.len, &blob);
}
});
self.cache.discard_staged(&blob);
self.log_activity(ActivityKind::Upload, &name, "uploaded", true);
info!(%uid, len = meta.len, "pending upload landed");
// Both times, so a slow drain shows whether the upload itself or the
// checks and gap-fill around it are where the time goes.
info!(
%uid,
len = meta.len,
upload_ms = took.as_millis() as u64,
total_ms = op_started.elapsed().as_millis() as u64,
"pending upload landed"
);
Ok(())
}
@@ -1522,21 +1756,39 @@ impl Core {
/// we just sealed. Skipping both, as this used to, is what produced a file
/// that conflicted with itself and left a full-size duplicate behind.
///
/// `blob` is the revision that landed, which stays the pending one until
/// open write handles are rebased off it, and `adopt` gets the node read
/// back before that, so the uploaded bytes can be cached under its
/// revision first. A handle released in between fills its gaps from one
/// or the other; with the blob let go first and the cache filled last, it
/// found neither, and the next write open was refused (docs/BUGS.md B123).
///
/// Best effort: a failure here costs a spurious conflict copy on the next
/// write, not this upload, which has already landed.
pub(crate) fn refresh_after_upload(&self, uid: &NodeUid) {
let node = match self.fetch_node_remote(uid) {
/// write, not this upload, which has already landed. Returns the node as
/// the server now has it, or `None` when it could not be read back.
pub(crate) fn refresh_after_upload(
&self,
uid: &NodeUid,
blob: &Path,
adopt: impl FnOnce(&Node),
) -> Option<Node> {
let fetched = self.fetch_node_remote(uid);
if !matches!(fetched, Ok(Some(_))) {
self.release_pending(uid, blob);
}
let node = match fetched {
Ok(Some(node)) => node,
// Trashed or deleted under us: the tree will hear it from the event
// sync, which is better placed to unhook the inode than we are.
Ok(None) => return,
Ok(None) => return None,
Err(e) => {
warn!(%uid, error = %e,
"refreshing metadata after an upload failed; \
the next write to this file may conflict with itself");
return;
return None;
}
};
adopt(&node);
// This function's whole job is to bring the tree level with the revision
// we just sealed, so the feed's report of that revision has nothing left
// to tell us. Claimed here rather than at the upload call so it is only
@@ -1552,18 +1804,14 @@ impl Core {
// This is the open-handle counterpart of `rebaseline_pending`, which
// covers the same gap for *queued* ops.
{
let sealed_mtime = node.modification_time;
let sealed_size = node_size(&node);
let sealed_rev = node_revision_id(&node);
// Remember it as ours, so a queued write that opened over an earlier
// revision recognises this as a self-supersede rather than a foreign
// change and does not fork (B70 layer B; consulted in
// `revision_conflict`).
if let Some(rev) = sealed_rev.clone() {
self.own_sealed_revs.lock().insert(uid.clone(), rev.clone());
// Write through: losing this at shutdown reopened the fork
// window B70 layer B closed, for the first drain after a
// restart.
if let Some(rev) = node_revision_id(&node) {
// In the table, not in memory: losing it at shutdown reopened
// the fork window B70 layer B closed, for the first drain
// after a restart.
if let Err(e) = self
.db
.set_own_sealed_rev(&uid.to_string(), &rev, now_millis())
@@ -1571,23 +1819,14 @@ impl Core {
warn!(%uid, error = %e, "persisting our sealed revision failed");
}
}
self.for_each_state(|st| {
for aw in st.active_writes.values_mut() {
if aw.uid == *uid {
aw.base_mtime = sealed_mtime;
aw.base_size = sealed_size;
aw.base_revision_id = sealed_rev.clone();
debug!(%uid, mtime = sealed_mtime, size = sealed_size,
"rebased open write handle onto the revision just uploaded");
}
}
});
self.rebase_open_writes(uid, &node);
}
self.release_pending(uid, blob);
// Ordered so that a write queued *during* the fetch above is still
// caught: it took its baseline from the node's optimistic stamp, and
// this overwrites it with the revision the server actually holds.
if self.rebaseline_pending(uid, &node) {
return;
return Some(node);
}
self.for_each_state(|st| {
let Some(parent) = st
@@ -1600,6 +1839,21 @@ impl Core {
};
st.intern(parent, node.clone());
});
Some(node)
}
/// Rebase every write handle open on `uid` onto `node`, a revision this
/// daemon has just put there itself (`WriteHandle::rebase_onto`).
fn rebase_open_writes(&self, uid: &NodeUid, node: &Node) {
self.for_each_state(|st| {
for aw in st.active_writes.values_mut() {
if aw.uid == *uid {
aw.rebase_onto(node);
debug!(%uid, mtime = aw.base_mtime, size = aw.base_size,
"rebased open write handle onto the revision just landed");
}
}
});
}
/// Point a still-queued write at the revision this upload just sealed, and
@@ -1690,6 +1944,20 @@ fn is_own_self_supersede(complete: bool, remote_rev: Option<&str>, own_rev: Opti
complete && remote_rev.is_some() && remote_rev == own_rev
}
/// Whether Drive has `node` neither where its queued rename found it nor where
/// the rename takes it (`parent`, `name`), so someone else moved or renamed it
/// in between. Unknown on a row without the original name.
fn moved_elsewhere(meta: &RenameMeta, node: &Node, parent: &NodeUid, name: &str) -> bool {
let Some(original_name) = meta.original_name.as_deref() else {
return false;
};
let at = |parent: &str, name: &str| {
node.parent_uid.as_ref().map(ToString::to_string).as_deref() == Some(parent)
&& node.name == name
};
!at(&meta.original_parent_uid, original_name) && !at(&parent.to_string(), name)
}
/// What the park sweep does with one parked create whose node still exists.
#[derive(Debug, PartialEq, Eq)]
enum ParkVerdict {
@@ -1708,9 +1976,48 @@ fn park_verdict(created_at: i64, now: i64, open: bool) -> ParkVerdict {
ParkVerdict::Release
}
/// The retryable error for an op whose name a queued trash has yet to free.
fn held_by_queued_trash(name: &str) -> Box<dyn std::error::Error> {
format!("{name} is still held by a node whose trash is queued").into()
}
/// How much older than its op a remote node may be and still be the one an
/// unanswered call made. The op is queued the moment the call gives up, so the
/// node is at most one call's deadline older; the rest is clock skew between
/// this machine and Proton.
const ADOPT_WINDOW_MS: i64 = 2 * 60 * 1000;
/// Whether `twin`, found holding the name a queued create wants, is the node an
/// earlier, unanswered attempt at that create made: the same kind, created
/// around when the op was queued, and for a file still empty, as the mount's
/// create leaves it.
fn adoptable(is_dir: bool, twin: &Node, op_created_ms: i64) -> bool {
if twin.creation_time.saturating_mul(1000) < op_created_ms - ADOPT_WINDOW_MS {
return false;
}
match &twin.kind {
NodeKind::Folder => is_dir,
NodeKind::File { claimed_size, .. } => !is_dir && *claimed_size == Some(0),
}
}
/// What share of the cache budget an unpinned upload may take and still be kept
/// as cached content once it lands: a sixteenth, 320 MiB of the default 5 GiB.
/// A larger file would push out much of what the user read lately, for bytes
/// they may never read back.
const LANDED_KEEP_SHARE: u64 = 16;
/// Whether an upload of `len` bytes that just landed stays as the file's cached
/// content. A pinned file always does, and so does any file when the budget is
/// unlimited (`0`).
fn keeps_landed_upload(pinned: bool, len: u64, budget: u64) -> bool {
pinned || budget == 0 || len <= budget / LANDED_KEEP_SHARE
}
#[cfg(test)]
mod tests {
use super::*;
use crate::state::WriteHandle;
use pdfs_core::Access;
use pdfs_core::db::Db;
use proton_drive_rs::NodeKind;
@@ -1822,6 +2129,7 @@ mod tests {
op.meta_json = Some(
serde_json::to_string(&RenameMeta {
original_parent_uid: source.uid.to_string(),
original_name: Some(moved.name.clone()),
})
.unwrap(),
);
@@ -2087,6 +2395,60 @@ mod tests {
}
}
#[test]
fn only_a_fresh_empty_twin_of_the_same_kind_is_adopted() {
// The op was queued at t=1000 s, after a create that got no answer.
let queued = 1_000_000;
let made = |mut node: Node, secs: i64| {
node.creation_time = secs;
node
};
assert!(adoptable(false, &made(file_node(0, 0, None), 990), queued));
assert!(adoptable(true, &made(folder_node("d", None), 990), queued));
// Made long before the op: someone else's file, not our lost create.
assert!(!adoptable(false, &made(file_node(0, 0, None), 800), queued));
// Our create leaves a file empty; one with content is someone else's.
assert!(!adoptable(false, &made(file_node(0, 5, None), 990), queued));
// The kind has to match what the op makes.
assert!(!adoptable(true, &made(file_node(0, 0, None), 990), queued));
assert!(!adoptable(
false,
&made(folder_node("d", None), 990),
queued
));
}
#[test]
fn a_landed_upload_stays_cached_unless_it_would_crowd_the_cache_out() {
const GIB: u64 = 1 << 30;
let budget = 5 * GIB;
// An edited document or a source file: read back from disk, not Drive.
assert!(keeps_landed_upload(false, 40_000, budget));
assert!(keeps_landed_upload(false, budget / 16, budget));
// A video the user copied in would push out what they read lately.
assert!(!keeps_landed_upload(false, budget / 16 + 1, budget));
// Pinned content is kept whatever its size, and so is everything when
// the cache has no cap.
assert!(keeps_landed_upload(true, 4 * GIB, budget));
assert!(keeps_landed_upload(false, 4 * GIB, 0));
}
#[test]
fn a_landing_upload_counts_until_its_guard_goes() {
let count = Arc::new(AtomicU64::new(0));
let first = LandingUpload::begin(&count);
let landed = || -> Result<(), ()> {
let _second = LandingUpload::begin(&count);
assert_eq!(count.load(Ordering::SeqCst), 2);
Err(())
};
// Uncounted on an early return too, or the queue never reads empty.
assert!(landed().is_err());
assert_eq!(count.load(Ordering::SeqCst), 1);
drop(first);
assert_eq!(count.load(Ordering::SeqCst), 0);
}
fn baseline(mtime: i64, size: u64, rev: Option<&str>) -> Baseline {
Baseline {
mtime,
@@ -2114,6 +2476,26 @@ mod tests {
assert!(revision_changed(&base, &remote).is_some());
}
/// The reported shape (docs/BUGS.md B113): a downloader creates a file
/// through the mount, and the file is moved before its write drains. The
/// move re-stamps the node's mtime on the server (1790955809 -> 1790956593),
/// but the revision is still the empty one the create minted.
#[test]
fn a_move_before_the_first_write_drains_is_not_a_conflict() {
let minted = file_node(1_790_955_809, 0, Some("empty"));
let path = std::env::temp_dir().join(format!("pdfs-drain-test-{}", std::process::id()));
let file = File::create(&path).unwrap();
let _ = std::fs::remove_file(&path);
let h = WriteHandle::created(2, &minted, file, path);
let base = baseline(h.base_mtime, h.base_size, h.base_revision_id.as_deref());
let moved = file_node(1_790_956_593, 0, Some("empty"));
assert_eq!(revision_changed(&base, &moved), None);
// Another device writing it is still a conflict.
let edited = file_node(1_790_956_593, 0, Some("theirs"));
assert!(revision_changed(&base, &edited).is_some());
}
#[test]
fn without_ids_it_falls_back_to_mtime_and_size() {
// Old sidecar (no revision id): the (mtime, size) tuple still governs.
+466
View File
@@ -0,0 +1,466 @@
//! The calls the sync core makes to Drive, behind a trait.
//!
//! `Core` used to hold a concrete [`ProtonDriveClient`], so no test could make
//! Drive answer late, answer stale, or not answer at all, and the interleavings
//! behind most recent bugs could only be found on a real account over a slow
//! link (`docs/MILESTONE-3.0.0.md` §2, §8.1). [`DriveApi`] is the part of the
//! client the mount, the drain, the event feed and the mirror engine use; the
//! real client implements it by forwarding, and the simulation tests implement
//! it with an in-memory Drive that injects Drive's known faults.
//!
//! Sharing, Photos, devices, invitations and public links stay on the concrete
//! client (`Core::client`) until a test needs them faked.
//!
//! The method names match the client's own, so a call site reads the same
//! whichever it goes through. The client's generic methods (a reader or writer
//! of any type) cannot be part of an object-safe trait, so the trait takes them
//! as trait objects and `impl dyn DriveApi` puts the generic signatures back.
use std::io::{Read, Write};
use std::sync::Arc;
use async_trait::async_trait;
use futures::StreamExt as _;
use futures::stream::BoxStream;
use proton_drive_rs::proton_sdk::account::Quota;
use proton_drive_rs::proton_sdk::error::Result;
use proton_drive_rs::proton_sdk::ids::{DriveEventId, NodeUid};
use proton_drive_rs::{
DriveEvent, DriveEventScopeId, Node, NodeMoveItem, ProtonDriveClient, RevisionReader, Thumbnail,
};
/// One outcome per node of a batch request; see [`ProtonDriveClient::trash_nodes`].
pub(crate) type NodeOutcomes = Vec<(NodeUid, Result<()>)>;
/// The outcomes of a streaming batch request, one item per node as it lands.
pub(crate) type OutcomeStream<'a> = BoxStream<'a, Result<(NodeUid, Result<()>)>>;
/// Drive as the sync core sees it. See the module documentation.
#[async_trait]
pub(crate) trait DriveApi: Send + Sync {
async fn get_my_files_folder(&self) -> Result<Node>;
async fn get_node(&self, uid: &NodeUid) -> Result<Option<Node>>;
async fn enumerate_nodes(&self, uids: &[NodeUid]) -> Result<Vec<Node>>;
/// As [`DriveApi::enumerate_nodes`], but files carry no claimed size or
/// modification time; see [`ProtonDriveClient::enumerate_nodes_light`].
async fn enumerate_nodes_light(&self, uids: &[NodeUid]) -> Result<Vec<Node>>;
async fn enumerate_folder_children_node_uids(
&self,
folder_uid: &NodeUid,
) -> Result<Vec<NodeUid>>;
async fn enumerate_events(
&self,
scope: &DriveEventScopeId,
cursor: Option<&DriveEventId>,
) -> Result<Vec<DriveEvent>>;
async fn invalidate_caches_for_event(&self, event: &DriveEvent) -> Result<()>;
async fn create_folder(
&self,
parent_uid: &NodeUid,
name: &str,
last_modification_time: Option<i64>,
) -> Result<NodeUid>;
async fn upload_file(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
contents: &[u8],
) -> Result<NodeUid>;
#[allow(clippy::too_many_arguments)]
async fn upload_file_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid>;
#[allow(clippy::too_many_arguments)]
async fn upload_file_replacing_draft_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid>;
async fn upload_new_revision_from_dyn(
&self,
file_uid: &NodeUid,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()>;
async fn open_revision(&self, uid: &NodeUid) -> Result<Arc<dyn RevisionRead>>;
async fn download_file_to_dyn(
&self,
uid: &NodeUid,
output: &mut (dyn Write + Send),
) -> Result<()>;
async fn download_revision_to_dyn(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut (dyn Write + Send),
) -> Result<()>;
async fn rename_node(
&self,
uid: &NodeUid,
new_name: &str,
new_media_type: Option<&str>,
) -> Result<()>;
async fn move_node(&self, uid: &NodeUid, new_parent: &NodeUid) -> Result<()>;
fn move_nodes_streaming<'a>(
&'a self,
items: Vec<NodeMoveItem>,
new_parent: NodeUid,
) -> OutcomeStream<'a>;
async fn trash_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes>;
async fn restore_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes>;
fn restore_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a>;
fn delete_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a>;
async fn enumerate_trash_node_uids(&self) -> Result<Vec<NodeUid>>;
/// The account's storage, across every Proton product.
async fn quota(&self) -> Result<Quota>;
}
/// The generic signatures of the client, over the object-safe ones above.
impl dyn DriveApi {
#[allow(clippy::too_many_arguments)]
pub(crate) async fn upload_file_from<R: Read + Send>(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
self.upload_file_from_dyn(
parent_uid,
name,
media_type,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
#[allow(clippy::too_many_arguments)]
pub(crate) async fn upload_file_replacing_draft_from<R: Read + Send>(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
self.upload_file_replacing_draft_from_dyn(
parent_uid,
name,
media_type,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
pub(crate) async fn upload_new_revision_from<R: Read + Send>(
&self,
file_uid: &NodeUid,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()> {
self.upload_new_revision_from_dyn(
file_uid,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
)
.await
}
pub(crate) async fn download_file_to<W: Write + Send>(
&self,
uid: &NodeUid,
output: &mut W,
) -> Result<()> {
self.download_file_to_dyn(uid, output).await
}
pub(crate) async fn download_revision_to<W: Write + Send>(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut W,
) -> Result<()> {
self.download_revision_to_dyn(file_uid, revision_id, writer)
.await
}
}
/// An open revision that block reads are served from; see [`RevisionReader`].
#[async_trait]
pub(crate) trait RevisionRead: Send + Sync {
/// Plaintext size of the revision.
fn size(&self) -> u64;
/// Plaintext size of each block, in block order.
fn block_sizes(&self) -> &[u64];
async fn read_at(&self, offset: u64, length: u64) -> Result<Vec<u8>>;
}
#[async_trait]
impl RevisionRead for RevisionReader {
fn size(&self) -> u64 {
RevisionReader::size(self)
}
fn block_sizes(&self) -> &[u64] {
RevisionReader::block_sizes(self)
}
async fn read_at(&self, offset: u64, length: u64) -> Result<Vec<u8>> {
RevisionReader::read_at(self, offset, length).await
}
}
#[async_trait]
impl DriveApi for ProtonDriveClient {
async fn get_my_files_folder(&self) -> Result<Node> {
ProtonDriveClient::get_my_files_folder(self).await
}
async fn get_node(&self, uid: &NodeUid) -> Result<Option<Node>> {
ProtonDriveClient::get_node(self, uid).await
}
async fn enumerate_nodes(&self, uids: &[NodeUid]) -> Result<Vec<Node>> {
ProtonDriveClient::enumerate_nodes(self, uids).await
}
async fn enumerate_nodes_light(&self, uids: &[NodeUid]) -> Result<Vec<Node>> {
ProtonDriveClient::enumerate_nodes_light(self, uids).await
}
async fn enumerate_folder_children_node_uids(
&self,
folder_uid: &NodeUid,
) -> Result<Vec<NodeUid>> {
ProtonDriveClient::enumerate_folder_children_node_uids(self, folder_uid).await
}
async fn enumerate_events(
&self,
scope: &DriveEventScopeId,
cursor: Option<&DriveEventId>,
) -> Result<Vec<DriveEvent>> {
ProtonDriveClient::enumerate_events(self, scope, cursor).await
}
async fn invalidate_caches_for_event(&self, event: &DriveEvent) -> Result<()> {
ProtonDriveClient::invalidate_caches_for_event(self, event).await
}
async fn create_folder(
&self,
parent_uid: &NodeUid,
name: &str,
last_modification_time: Option<i64>,
) -> Result<NodeUid> {
ProtonDriveClient::create_folder(self, parent_uid, name, last_modification_time).await
}
async fn upload_file(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
contents: &[u8],
) -> Result<NodeUid> {
ProtonDriveClient::upload_file(self, parent_uid, name, media_type, contents).await
}
async fn upload_file_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
ProtonDriveClient::upload_file_from(
self,
parent_uid,
name,
media_type,
reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
async fn upload_file_replacing_draft_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
ProtonDriveClient::upload_file_replacing_draft_from(
self,
parent_uid,
name,
media_type,
reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
async fn upload_new_revision_from_dyn(
&self,
file_uid: &NodeUid,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()> {
ProtonDriveClient::upload_new_revision_from(
self,
file_uid,
reader,
intended_size,
thumbnails,
last_modification_time,
)
.await
}
async fn open_revision(&self, uid: &NodeUid) -> Result<Arc<dyn RevisionRead>> {
let reader = ProtonDriveClient::open_revision(self, uid).await?;
Ok(Arc::new(reader))
}
async fn download_file_to_dyn(
&self,
uid: &NodeUid,
output: &mut (dyn Write + Send),
) -> Result<()> {
ProtonDriveClient::download_file_to(self, uid, &mut &mut *output).await
}
async fn download_revision_to_dyn(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut (dyn Write + Send),
) -> Result<()> {
ProtonDriveClient::download_revision_to(self, file_uid, revision_id, &mut &mut *writer)
.await
}
async fn rename_node(
&self,
uid: &NodeUid,
new_name: &str,
new_media_type: Option<&str>,
) -> Result<()> {
ProtonDriveClient::rename_node(self, uid, new_name, new_media_type).await
}
async fn move_node(&self, uid: &NodeUid, new_parent: &NodeUid) -> Result<()> {
ProtonDriveClient::move_node(self, uid, new_parent).await
}
fn move_nodes_streaming<'a>(
&'a self,
items: Vec<NodeMoveItem>,
new_parent: NodeUid,
) -> OutcomeStream<'a> {
ProtonDriveClient::move_nodes_streaming(self, items, new_parent).boxed()
}
async fn trash_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes> {
ProtonDriveClient::trash_nodes(self, uids).await
}
async fn restore_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes> {
ProtonDriveClient::restore_nodes(self, uids).await
}
fn restore_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a> {
ProtonDriveClient::restore_nodes_streaming(self, uids).boxed()
}
fn delete_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a> {
ProtonDriveClient::delete_nodes_streaming(self, uids).boxed()
}
async fn enumerate_trash_node_uids(&self) -> Result<Vec<NodeUid>> {
ProtonDriveClient::enumerate_trash_node_uids(self).await
}
async fn quota(&self) -> Result<Quota> {
ProtonDriveClient::quota(self).await
}
}
+257 -222
View File
@@ -72,7 +72,7 @@ impl Filesystem for ProtonFs {
}
fn lookup(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEntry) {
let parent = parent.0;
let parent = self.tree(parent).0;
let name = match fuse_name(name) {
Ok(name) => name,
Err(error) => {
@@ -95,11 +95,13 @@ impl Filesystem for ProtonFs {
}
fn forget(&self, _req: &Request, ino: INodeNo, nlookup: u64) {
let ino = self.tree(ino);
let mut st = self.core.state();
st.forget_lookup(ino.0, nlookup);
}
fn getattr(&self, _req: &Request, ino: INodeNo, _fh: Option<FileHandle>, reply: ReplyAttr) {
let ino = self.tree(ino);
let (attr, provisional_in) = {
let st = self.core.state();
match st.entries.get(&ino.0) {
@@ -175,7 +177,7 @@ impl Filesystem for ProtonFs {
/// Enumerating a cold folder is a remote call, so it goes to a worker for
/// the same reason `lookup` and `readdir` do.
fn opendir(&self, _req: &Request, ino: INodeNo, _flags: OpenFlags, reply: ReplyOpen) {
let ino = ino.0;
let ino = self.tree(ino).0;
if self.core.children_cached(ino) {
self.serve_opendir(ino, reply);
return;
@@ -194,13 +196,13 @@ impl Filesystem for ProtonFs {
offset: u64,
reply: ReplyDirectory,
) {
let ino = ino.0;
let ino = self.tree(ino).0;
// The snapshot is the whole point: served from it, this is a pure
// in-memory walk with no chance of a remote call, so it never needs a
// worker. Only a handle whose `opendir` failed to snapshot (or a kernel
// that issued `readdir` without one) falls back to the live path.
if let Some(listing) = self.core.state().dir_snapshots.get(&fh.0).cloned() {
reply_listing(&listing, offset, reply);
reply_listing(&listing, offset, self.root, reply);
return;
}
// Same split as `lookup`: only the cold, remote-enumerating path pays
@@ -232,6 +234,7 @@ impl Filesystem for ProtonFs {
/// a blob that is the whole file — so it is served from a worker rather than
/// stalling every other request on the mount behind it.
fn open(&self, _req: &Request, ino: INodeNo, flags: OpenFlags, reply: ReplyOpen) {
let ino = self.tree(ino);
if flags.acc_mode() == OpenAccMode::O_RDONLY {
self.serve_open(ino, flags, reply);
return;
@@ -253,6 +256,7 @@ impl Filesystem for ProtonFs {
_lock_owner: Option<LockOwner>,
reply: ReplyData,
) {
let ino = self.tree(ino);
// A file open for writing is served from its handle so reads see the
// in-flight (possibly unsaved) content: authored bytes come from the
// scratch file, untouched bytes from the base.
@@ -290,7 +294,7 @@ impl Filesystem for ProtonFs {
}
return;
}
let (uid, mtime, fsize, is_video) = {
let (uid, mtime, fsize, provisional, is_video) = {
let st = self.core.state();
match st.entries.get(&ino.0) {
Some(e) if e.node.is_file() => {
@@ -300,14 +304,24 @@ impl Filesystem for ProtonFs {
};
let is_video =
PhotoKind::classify(Some(&e.node.name), media_type) == PhotoKind::Video;
let fsize = self
.core
.pending
.lock()
.get(&e.uid)
.map(|p| p.meta.len)
.unwrap_or_else(|| node_size(&e.node));
(e.uid.clone(), e.node.modification_time, fsize, is_video)
let queued = self.core.pending.lock().get(&e.uid).map(|p| p.meta.len);
// Still the cheap enumeration's ciphertext size (B12).
let provisional = queued.is_none()
&& matches!(
&e.node.kind,
NodeKind::File {
claimed_size: None,
..
}
);
let fsize = queued.unwrap_or_else(|| node_size(&e.node));
(
e.uid.clone(),
e.node.modification_time,
fsize,
provisional,
is_video,
)
}
Some(_) => {
reply.error(Errno::EISDIR);
@@ -330,6 +344,13 @@ impl Filesystem for ProtonFs {
// races with nothing. FUSE does not require replies in request order.
let core = self.core.clone();
self.core.workers.run(Lane::Transfer, "read", move || {
// The revision disagrees with a provisional size, and B84 fails
// such a read outright, so settle the size first. On the network,
// hence here and not above (bugs.md B100).
let fsize = match provisional {
true => core.resolve_provisional_size(ino.0, &uid).unwrap_or(fsize),
false => fsize,
};
match core.read_range(&uid, mtime, fsize, offset, size as u64, cache_blocks) {
Ok(bytes) => reply.data(&bytes),
Err(e) => reply.error(e),
@@ -347,7 +368,7 @@ impl Filesystem for ProtonFs {
_flags: i32,
reply: ReplyCreate,
) {
let parent = parent.0;
let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) {
reply.error(error);
return;
@@ -383,6 +404,7 @@ impl Filesystem for ProtonFs {
_lock_owner: Option<LockOwner>,
reply: ReplyWrite,
) {
let ino = self.tree(ino);
if let Err(error) = self.core.require_writable(ino.0) {
reply.error(error);
return;
@@ -425,10 +447,7 @@ impl Filesystem for ProtonFs {
reply.error(Errno::EBADF);
return;
};
let end = offset + data.len() as u64;
aw.written.add(offset, end);
aw.len = aw.len.max(end);
aw.dirty = true;
aw.record_write(offset, offset + data.len() as u64);
let len = aw.len;
st.set_size(ino.0, len);
len
@@ -463,6 +482,7 @@ impl Filesystem for ProtonFs {
_flags: Option<BsdFileFlags>,
reply: ReplyAttr,
) {
let ino = self.tree(ino);
if let Err(error) = self.core.require_writable(ino.0) {
reply.error(error);
return;
@@ -530,6 +550,7 @@ impl Filesystem for ProtonFs {
mode: i32,
reply: ReplyEmpty,
) {
let ino = self.tree(ino);
if let Err(error) = self.core.require_writable(ino.0) {
reply.error(error);
return;
@@ -626,16 +647,17 @@ impl Filesystem for ProtonFs {
fn release(
&self,
_req: &Request,
_ino: INodeNo,
ino: INodeNo,
fh: FileHandle,
_flags: OpenFlags,
_lock_owner: Option<LockOwner>,
_flush: bool,
reply: ReplyEmpty,
) {
let ino = self.tree(ino);
let (handle, unlinked_uid) = {
let mut st = self.core.state();
let unlinked_uid = release_unlinked_entry(&mut st, _ino.0);
let unlinked_uid = release_unlinked_entry(&mut st, ino.0);
let ino = st.handles.remove(&fh.0);
let h = ino.and_then(|i| {
let aw = st.active_writes.get_mut(&i)?;
@@ -666,7 +688,7 @@ impl Filesystem for ProtonFs {
}
fn unlink(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEmpty) {
let parent = parent.0;
let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) {
reply.error(error);
return;
@@ -686,7 +708,7 @@ impl Filesystem for ProtonFs {
}
fn rmdir(&self, _req: &Request, parent: INodeNo, name: &OsStr, reply: ReplyEmpty) {
let parent = parent.0;
let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) {
reply.error(error);
return;
@@ -715,7 +737,7 @@ impl Filesystem for ProtonFs {
_umask: u32,
reply: ReplyEntry,
) {
let parent = parent.0;
let parent = self.tree(parent).0;
if let Err(error) = self.core.require_writable(parent) {
reply.error(error);
return;
@@ -746,8 +768,8 @@ impl Filesystem for ProtonFs {
flags: RenameFlags,
reply: ReplyEmpty,
) {
let parent = parent.0;
let newparent = newparent.0;
let parent = self.tree(parent).0;
let newparent = self.tree(newparent).0;
if let Err(error) = self
.core
.require_writable(parent)
@@ -780,6 +802,7 @@ impl Filesystem for ProtonFs {
}
fn access(&self, _req: &Request, ino: INodeNo, mask: AccessFlags, reply: ReplyEmpty) {
let ino = self.tree(ino);
let state = self.core.state();
let Some(entry) = state.entries.get(&ino.0) else {
reply.error(Errno::ENOENT);
@@ -793,6 +816,7 @@ impl Filesystem for ProtonFs {
}
fn getxattr(&self, _req: &Request, ino: INodeNo, name: &OsStr, size: u32, reply: ReplyXattr) {
let ino = self.tree(ino);
// Always off the dispatch loop: a miss goes to the wire, and a lister
// that stats a directory issues one of these per file per advertised
// name — inline, that serialized the whole mount behind ~186 ms of
@@ -805,6 +829,7 @@ impl Filesystem for ProtonFs {
}
fn listxattr(&self, _req: &Request, ino: INodeNo, size: u32, reply: ReplyXattr) {
let ino = self.tree(ino);
self.serve_listxattr(ino, size, reply);
}
@@ -954,6 +979,9 @@ pub struct ProtonFs {
core: Core,
uid: u32,
gid: u32,
/// The tree inode of this session's root, which the kernel knows as
/// [`ROOT_INO`].
root: u64,
}
/// How many times a write `open` will re-read its base after a `release`
@@ -994,59 +1022,55 @@ fn build_listing(st: &State, ino: u64) -> DirListing {
/// The cookie handed back is the index of the *next* entry, so a resumed call
/// continues exactly where the last one stopped — which is only true because
/// the listing behind those indexes is a snapshot (bugs.md B43).
fn reply_listing(listing: &DirListing, offset: u64, mut reply: ReplyDirectory) {
fn reply_listing(listing: &DirListing, offset: u64, root: u64, mut reply: ReplyDirectory) {
for (i, (ino, is_dir, name)) in listing.iter().enumerate().skip(offset as usize) {
let ft = if *is_dir {
FileType::Directory
} else {
FileType::RegularFile
};
if reply.add(INodeNo(*ino), (i + 1) as u64, ft, name) {
if reply.add(kernel_ino(root, *ino), (i + 1) as u64, ft, name) {
break;
}
}
reply.ok();
}
/// The inode the kernel knows tree inode `ino` by, in a session rooted at
/// `root`. The inverse of [`ProtonFs::tree`].
pub(crate) fn kernel_ino(root: u64, ino: u64) -> INodeNo {
INodeNo(if ino == root { ROOT_INO } else { ino })
}
pub(crate) fn access_allowed(is_dir: bool, access: Access, mask: AccessFlags) -> bool {
(!mask.contains(AccessFlags::W_OK) || access.writable())
&& (!mask.contains(AccessFlags::X_OK) || is_dir)
}
impl ProtonFs {
/// Build the filesystem rooted at `root` (the user's My Files folder).
/// Build the filesystem rooted at `root` (the user's My Files folder, or
/// the folder an on-demand location shows).
pub(super) fn new(core: Core, root: Node) -> Self {
{
let mut st = core.state();
if let Err(e) = st.db.upsert_node(&root) {
warn!(uid = %root.uid, error = %e, "db upsert root failed");
}
st.by_uid.insert(root.uid.clone(), ROOT_INO);
// The filesystem root is always owned — it must never be
// classified as a shared root, even when the API returns
// membership data (which happens when the owner shares the
// folder with someone else). Clean up any stale share_access
// row that a previous downgrade_all_share_access may have
// written for this uid, so hydrate_access cannot pick it up.
let access = Access::Owner;
st.share_access.remove(&root.uid);
let _ = st.db.delete_share_access(&root.uid);
st.entries.insert(
ROOT_INO,
Entry {
uid: root.uid.clone(),
parent: ROOT_INO,
node: root,
access,
lookup_count: 1,
open_count: 0,
unlinked: false,
},
);
}
let root = core.state().install_root(root);
let _ = core.root.set(root);
// SAFETY: geteuid/getegid are infallible and have no preconditions.
let (uid, gid) = unsafe { (libc::geteuid(), libc::getegid()) };
Self { core, uid, gid }
Self {
core,
uid,
gid,
root,
}
}
/// The tree inode the kernel means by `ino`: this session's root for
/// [`ROOT_INO`], and every other inode as it is.
fn tree(&self, ino: INodeNo) -> INodeNo {
if ino.0 == ROOT_INO {
INodeNo(self.root)
} else {
ino
}
}
/// The body of [`Filesystem::lookup`], on whichever thread ends up serving it.
@@ -1155,7 +1179,7 @@ impl ProtonFs {
let st = self.core.state();
let listing = build_listing(&st, ino);
drop(st);
reply_listing(&listing, offset, reply);
reply_listing(&listing, offset, self.root, reply);
}
fn attr(&self, ino: u64, node: &Node, access: Access) -> FileAttr {
@@ -1168,7 +1192,7 @@ impl ProtonFs {
let mtime = unix_secs(node.modification_time);
let crtime = unix_secs(node.creation_time);
FileAttr {
ino: INodeNo(ino),
ino: kernel_ino(self.root, ino),
size,
blocks: size.div_ceil(512),
atime: mtime,
@@ -1193,7 +1217,7 @@ impl ProtonFs {
/// open.
fn serve_open(&self, ino: INodeNo, flags: OpenFlags, reply: ReplyOpen) {
let write_requested = flags.acc_mode() != OpenAccMode::O_RDONLY;
let (uid, base_mtime, base_size, base_revision_id) = {
let (uid, base_mtime, base_size, provisional, base_revision_id) = {
let mut st = self.core.state();
match st.entries.get_mut(&ino.0) {
Some(e) if e.node.is_file() => {
@@ -1206,6 +1230,14 @@ impl ProtonFs {
e.uid.clone(),
e.node.modification_time,
node_size(&e.node),
// Still the cheap enumeration's ciphertext size (B12).
matches!(
&e.node.kind,
NodeKind::File {
claimed_size: None,
..
}
),
node_revision_id(&e.node),
)
}
@@ -1251,6 +1283,21 @@ impl ProtonFs {
// `reply` is consumed by whichever arm answers, so the loop yields an
// outcome and the answer is sent once, after it.
let outcome = 'install: {
// The handle starts at the base's size. A provisional one is the
// ciphertext size, so the file would grow by the encryption overhead
// and reading the base would fail (bugs.md B100). A queued revision
// stamps a real size, so this only goes to the network for a file
// the listing has not sized yet.
let base_size = match provisional && self.core.pending_blob(&uid).is_none() {
false => base_size,
true => match self.core.resolve_provisional_size(ino.0, &uid) {
Some(size) => size,
None => {
error!(%uid, "could not settle the size of the base for a write open");
break 'install Err(Errno::EIO);
}
},
};
for attempt in 0..OPEN_BASE_ATTEMPTS {
let pending_base = self.core.pending.lock().get(&uid).cloned();
if pending_base.as_ref().is_some_and(|p| !p.meta.complete) {
@@ -1398,7 +1445,7 @@ impl ProtonFs {
len,
base_size,
base_mtime,
complete: authored == [(0, len)],
complete: StagedWrite::covers(&authored, len),
authored,
based_on: self
.core
@@ -1427,6 +1474,12 @@ impl ProtonFs {
return;
}
if let Err(e) = self.core.ensure_children(parent) {
// The parent was looked up a moment ago, so ENOENT here means it
// left the tree in between. Log it: the caller only sees "no such
// file" on a create.
if e == Errno::ENOENT {
warn!(parent, name, "create: the parent folder is no longer known");
}
reply.error(e);
return;
}
@@ -1435,6 +1488,10 @@ impl ProtonFs {
match st.entries.get(&parent) {
Some(e) => e.uid.clone(),
None => {
warn!(
parent,
name, "create: the parent folder left the tree during its listing"
);
reply.error(Errno::ENOENT);
return;
}
@@ -1454,67 +1511,58 @@ impl ProtonFs {
//
// A parent that is itself still queued forces the same path even when we
// are online: the API has no folder to put this in yet.
let node =
if !transient && self.core.online.load(Ordering::Relaxed) && !is_local_uid(&parent_uid)
{
if let Err(error) = self.core.require_uid_writable(&parent_uid) {
reply.error(error);
//
// A create that set out online and lost the network on the way lands in
// the queue too, so a Wi-Fi drop costs the caller a wait, never an EIO.
let minted = if !transient && self.core.is_online() && !is_local_uid(&parent_uid) {
if let Err(error) = self.core.require_uid_writable(&parent_uid) {
reply.error(error);
return;
}
// Create an empty file on the remote so it has a real uid immediately;
// written bytes are buffered and sealed as a new revision on close.
match self.core.block_on_bounded(self.core.drive.upload_file(
&parent_uid,
name,
media_type_for(name),
b"",
)) {
// A file that exists remotely but could not be read back is
// queued, and the queued create finds it by name and adopts it.
Ok(new_uid) => self.core.fetch_minted_node(&new_uid, name),
Err(e) if self.core.lost_link(&e, "create") => None,
Err(e) => {
error!(%parent_uid, name, error = %e, "create file failed");
reply.error(Errno::EIO);
return;
}
// Create an empty file on the remote so it has a real uid immediately;
// written bytes are buffered and sealed as a new revision on close.
let new_uid = match self.core.rt.block_on(self.core.client.upload_file(
&parent_uid,
name,
media_type_for(name),
b"",
)) {
Ok(u) => u,
Err(e) => {
error!(%parent_uid, name, error = %e, "create file failed");
reply.error(Errno::EIO);
return;
}
};
match self.core.fetch_node(&new_uid) {
Ok(n) => n,
Err(e) => {
reply.error(e);
return;
}
}
} else {
None
};
let node = match minted {
Some(node) => node,
None => match self
.core
.queue_local_node(&parent_uid, name, false, transient)
{
Ok(n) => n,
Err(e) => {
reply.error(e);
return;
}
} else {
match self
.core
.queue_local_node(&parent_uid, name, false, transient)
{
Ok(n) => n,
Err(e) => {
reply.error(e);
return;
}
}
};
let new_uid = node.uid.clone();
// The base this handle writes over is the node as it actually exists —
// the empty file the server just minted — so its modification time comes
// from the node, never from the local clock. `queue_revision` turns this
// into `StagedWrite::based_on` and the drain compares that against the
// remote: a `now_secs()` here differs from the server's stamp by however
// long the create round-trip took, so the first write to a brand-new file
// conflicts with its own create whenever the second happens to tick in
// between.
let base_mtime = node.modification_time;
},
};
let (file, path) = match self.core.cache.create_scratch() {
Ok(x) => x,
Err(e) => {
error!(%new_uid, error = %e, "create scratch file failed");
error!(uid = %node.uid, error = %e, "create scratch file failed");
reply.error(Errno::EIO);
return;
}
};
let mut st = self.core.state();
let ino = st.intern(parent, node);
let ino = st.intern(parent, node.clone());
if let Some(entry) = st.entries.get_mut(&ino) {
entry.open_count = entry.open_count.saturating_add(1);
}
@@ -1525,23 +1573,10 @@ impl ProtonFs {
}
let fh = st.next_fh;
st.next_fh += 1;
let aw = st.active_writes.entry(ino).or_insert_with(|| {
// A brand-new file: empty base, everything written is authored.
WriteHandle {
ino,
uid: new_uid,
file: Arc::new(file),
path,
written: Intervals::default(),
len: 0,
base_size: 0,
base_mtime,
// A brand-new file has no sealed remote revision to conflict with.
base_revision_id: None,
dirty: false,
open_count: 0,
}
});
let aw = st
.active_writes
.entry(ino)
.or_insert_with(|| WriteHandle::created(ino, &node, file, path));
aw.open_count += 1;
st.handles.insert(fh, ino);
let entry = st.entries.get(&ino).unwrap();
@@ -1584,39 +1619,36 @@ impl ProtonFs {
// As in `create`: offline — or under a parent that is itself still
// queued — the folder becomes a placeholder that the drain turns into a
// real one (offline.md Phase 3b).
let node = if self.core.online.load(Ordering::Relaxed) && !is_local_uid(&parent_uid) {
// Losing the network on the way queues it the same way.
let minted = if self.core.is_online() && !is_local_uid(&parent_uid) {
if let Err(error) = self.core.require_uid_writable(&parent_uid) {
reply.error(error);
return;
}
let new_uid =
match self
.core
.rt
.block_on(self.core.client.create_folder(&parent_uid, name, now))
{
Ok(u) => u,
Err(e) => {
error!(%parent_uid, name, error = %e, "create folder failed");
reply.error(Errno::EIO);
return;
}
};
match self.core.fetch_node(&new_uid) {
Ok(n) => n,
match self
.core
.block_on_bounded(self.core.drive.create_folder(&parent_uid, name, now))
{
Ok(new_uid) => self.core.fetch_minted_node(&new_uid, name),
Err(e) if self.core.lost_link(&e, "mkdir") => None,
Err(e) => {
reply.error(e);
error!(%parent_uid, name, error = %e, "create folder failed");
reply.error(Errno::EIO);
return;
}
}
} else {
match self.core.queue_local_node(&parent_uid, name, true, false) {
None
};
let node = match minted {
Some(node) => node,
None => match self.core.queue_local_node(&parent_uid, name, true, false) {
Ok(n) => n,
Err(e) => {
reply.error(e);
return;
}
}
},
};
let mut st = self.core.state();
let local = is_local_uid(&node.uid);
@@ -1706,12 +1738,18 @@ impl ProtonFs {
if let Some(uid) = unlinked_uid
&& release_can_discard_unlinked(&self.core.db, &uid)
{
// Ops withdrawn at unlink left the bytes this handle read behind,
// owned by no op any more; they go with the last handle.
let withdrawn = self.core.pending_blob(&uid);
if let Err(error) = self.core.discard_queued_ops(&uid) {
warn!(%uid, ?error, "queued-op cleanup failed; retaining unlinked state");
} else {
if let Err(error) = self.core.db.delete_node(&uid) {
warn!(%uid, %error, "deleting released unlinked node state failed");
}
if let Some(blob) = withdrawn {
self.core.cache.discard_staged(&blob);
}
self.core.cache.evict(&uid);
self.core.evict_reader(&uid);
}
@@ -1916,12 +1954,14 @@ impl ProtonFs {
// now and this handle's is stale. A retry resolves it.
Ok(false) => {
warn!(%uid, name, newname, "queued create vanished under a rename");
self.core.restore_replaced(victim.as_ref(), newname);
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EBUSY);
}
Err(e) => {
error!(%uid, error = %e, "rewriting a queued create's target failed");
self.core.restore_replaced(victim.as_ref(), newname);
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EIO);
}
}
@@ -1932,12 +1972,11 @@ impl ProtonFs {
// take the synchronous path, so a genuine API refusal (permissions, a
// name clash) surfaces to the caller instead of becoming a queued op
// that can only ever conflict.
if rename_needs_queue(
self.core.online.load(Ordering::Relaxed),
is_local_uid(&new_parent_uid),
newparent != parent,
newname != name,
) {
// The queued rename, as a closure: the network going away under the
// synchronous path below lands here too, with whatever half already
// reached Drive. The op records the end state, so the drain finishes
// the rest.
let queue_it = |reply: ReplyEmpty| {
let queued = if replacement_authorized {
self.core.queue_rename_authorized(
ino,
@@ -1965,23 +2004,30 @@ impl ProtonFs {
// and retain an empty destination readdir page.
reply.ok();
if let Some(notifier) = self.core.notifier.get() {
let _ = notifier.inval_entry(INodeNo(parent), OsStr::new(&name));
let _ = notifier.inval_entry(INodeNo(newparent), OsStr::new(&newname));
let _ = notifier.inval_inode(INodeNo(parent), 0, 0);
let _ = notifier.inval_entry(parent, OsStr::new(&name));
let _ = notifier.inval_entry(newparent, OsStr::new(&newname));
let _ = notifier.inval_inode(parent);
if newparent != parent {
let _ = notifier.inval_inode(INodeNo(newparent), 0, 0);
let _ = notifier.inval_inode(newparent);
}
}
}
Err(e) => {
self.core.restore_replaced(victim.as_ref(), newname);
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(e);
}
}
};
if rename_needs_queue(
self.core.is_online(),
is_local_uid(&new_parent_uid),
newparent != parent,
newname != name,
) {
queue_it(reply);
return;
}
// Rename first if both halves change. Moving first makes the encrypted
// name requirements stale and repeatedly failed with InvalidRequirements.
// A failure past this point has already trashed any node the rename was
// replacing, so put it back rather than leaving the caller with neither
// the source moved nor the destination intact.
@@ -1991,64 +2037,37 @@ impl ProtonFs {
&old_parent_uid,
&new_parent_uid,
) {
self.core.restore_replaced(victim.as_ref(), newname);
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(error);
return;
}
if newname != name
&& let Err(e) = self
.core
.rt
.block_on(self.core.client.rename_node(&uid, newname, None))
{
// A move that also renames is one request, so the name only has to be
// free in the destination, and the reply never stands for half of it.
let renamed = if newparent != parent {
let target = (newname != name).then_some(newname);
self.core.move_rename_remote(&uid, &new_parent_uid, target)
} else if newname != name {
self.core.rename_remote(&uid, newname)
} else {
Ok(())
};
if let Err(e) = renamed {
if self.core.lost_link(&e, "rename") {
queue_it(reply);
return;
}
error!(%uid, error = %e, "rename failed");
self.core.restore_replaced(victim.as_ref(), newname);
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EIO);
return;
}
if newparent != parent {
let mut attempts = 0u32;
let moved = loop {
match self
.core
.rt
.block_on(self.core.client.move_node(&uid, &new_parent_uid))
{
Ok(()) => break Ok(()),
Err(e)
if newname != name
&& api_code(&e) == Some(ResponseCode::InvalidRequirements)
&& attempts < 20 =>
{
attempts += 1;
// Requirement propagation routinely takes longer than
// three seconds. Keep a bounded ten-second window; a
// successful POSIX reply must mean both remote halves
// landed, not merely that the second half was queued.
std::thread::sleep(Duration::from_millis(500));
}
Err(e) => break Err(e),
}
};
if let Err(e) = moved {
error!(%uid, attempts, error = %e, "move after rename failed");
// The rename half landed in the source directory.
let mut state = self.core.state();
let old_parent_uid = state
.entries
.get(&parent)
.map(|entry| entry.uid.clone())
.unwrap_or_else(|| uid.clone());
state.rename_in_place(ino, parent, &old_parent_uid, newname);
drop(state);
self.core.restore_replaced(victim.as_ref(), newname);
reply.error(Errno::EIO);
return;
}
}
// Through the guard, so the database has both folders unlisted before
// the reply. The next lookup rebuilds a folder still marked listed from
// the database, which did not have the move yet (docs/BUGS.md B135).
self.core
.state
.lock()
.state()
.relocate(ino, parent, newparent, &new_parent_uid, newname);
reply.ok();
}
@@ -2092,7 +2111,12 @@ impl ProtonFs {
// it just means its queued creation is no longer wanted. This works
// offline, which the remote path below cannot (offline.md Phase 3b).
if is_local_uid(&uid) {
if !open_now && let Err(error) = self.core.discard_queued_ops(&uid) {
let dropped = if open_now {
self.core.withdraw_queued_ops(&uid)
} else {
self.core.discard_queued_ops(&uid)
};
if let Err(error) = dropped {
reply.error(error);
return;
}
@@ -2103,12 +2127,14 @@ impl ProtonFs {
}
// Offline: queue it. Trashing is the one mutation a user expects to work
// regardless — the file is gone from their point of view the moment the
// command returns (offline.md Phase 3b).
if !self.core.online.load(Ordering::Relaxed) {
match self.core.queue_trash(&uid, name) {
Ok(()) => reply.ok(),
Err(e) => reply.error(e),
}
// command returns (offline.md Phase 3b). The same when the network goes
// away under the remote call below.
let queue_it = |reply: ReplyEmpty| match self.core.queue_trash(&uid, name) {
Ok(()) => reply.ok(),
Err(e) => reply.error(e),
};
if !self.core.is_online() {
queue_it(reply);
return;
}
if let Err(e) = self.core.require_uid_writable(&uid) {
@@ -2117,10 +2143,13 @@ impl ProtonFs {
}
if let Err(e) = self
.core
.rt
.block_on(self.core.client.trash_nodes(std::slice::from_ref(&uid)))
.block_on_bounded(self.core.drive.trash_nodes(std::slice::from_ref(&uid)))
.and_then(batch::into_unit)
{
if self.core.lost_link(&e, "trash") {
queue_it(reply);
return;
}
error!(%uid, error = %e, "trash failed");
self.core
.log_activity(ActivityKind::Trash, name, e.to_string(), false);
@@ -2131,7 +2160,13 @@ impl ProtonFs {
// Otherwise an EIO response still made the path disappear locally.
self.core.hidden.lock().insert(uid.clone());
self.core.state().forget_or_unlink(&uid);
if !open_now {
if open_now {
// The open handle keeps the staged bytes; the drain must not send
// them to a node that is already in the trash (docs/BUGS.md B106).
if let Err(error) = self.core.withdraw_queued_ops(&uid) {
warn!(%uid, ?error, "remote trash landed but queued-op cleanup failed");
}
} else {
if let Err(error) = self.core.discard_queued_ops(&uid) {
warn!(%uid, ?error, "remote trash landed but queued-op cleanup failed");
}
+1693 -728
View File
File diff suppressed because it is too large. Load diff
+316
View File
@@ -0,0 +1,316 @@
//! Noticing that the network went away mid-session, and that it came back.
//!
//! The mount used to learn it was offline only at startup: a daemon that had
//! connected once kept `online` true for the rest of its life, so a Wi-Fi drop
//! turned every create, mkdir, delete and rename into a two-minute wait for an
//! `EIO`, while the queue that exists for exactly this sat unused. Now any
//! remote call a handler waits on reports a failure to reach Proton through
//! [`Core::lost_link`], which flips the mount offline; the handler then takes its
//! queued path, and the probe thread here flips it back when Proton answers.
use std::future::Future;
use std::sync::atomic::Ordering;
use std::time::{Duration, Instant};
use parking_lot::{Condvar, Mutex};
use proton_drive_rs::proton_sdk::api::ResponseCode;
use proton_drive_rs::proton_sdk::error::{ProtonApiError, ProtonError};
use tracing::{debug, info, warn};
use crate::{Core, ONLINE_PROBE_MAX, ONLINE_PROBE_MIN};
/// How long a remote call a FUSE handler is waiting on may take before the link
/// is treated as down.
///
/// The SDK's own budget for one request is four attempts of up to 30 seconds
/// each, so a dead link held the caller for about two minutes. A metadata call
/// that has had no answer in 20 seconds is not going to be one the user wants to
/// keep waiting for; queueing the change is the better answer.
pub(crate) const INTERACTIVE_CALL_TIMEOUT: Duration = Duration::from_secs(20);
/// How long one probe may wait for an answer. A probe is a single small request,
/// and a probe stuck in the SDK's retry budget would delay noticing the link is
/// back by as long as it waits.
const PROBE_TIMEOUT: Duration = Duration::from_secs(10);
/// First delay between probes after a drop seen mid-session, and the ceiling it
/// doubles to while the outage is young. A blip should end in seconds, not in
/// the minutes [`ONLINE_PROBE_MAX`] allows.
const PROBE_AFTER_DROP: Duration = Duration::from_secs(2);
const PROBE_YOUNG_MAX: Duration = Duration::from_secs(30);
/// How long an outage counts as young. Past it the probe backs off to
/// [`ONLINE_PROBE_MAX`], because a laptop can sit offline for days and each
/// probe is a real API round trip.
const YOUNG_OUTAGE: Duration = Duration::from_secs(10 * 60);
/// How often the idle probe thread checks for teardown while the link is up.
const IDLE_CHECK: Duration = Duration::from_secs(1);
/// The shared half of the online flag: when the link went down, and a condvar
/// that fires on every change so waiters need not poll.
#[derive(Default)]
pub(crate) struct Link {
lost_at: Mutex<Option<Instant>>,
changed: Condvar,
/// Folders listed from the DB alone while offline, to relist once back.
pub(crate) stale_listings: Mutex<std::collections::HashSet<u64>>,
}
/// Whether a failed call failed because Proton could not be reached, rather
/// than because it answered and refused.
///
/// A transport error is the network. So is a request timeout: the SDK reports
/// Proton's own "try again" codes that way, and [`no_answer`] reports our
/// deadline that way. Anything else Proton actually said, and says again on a
/// retry, so it must reach the caller rather than be queued.
pub(crate) fn is_network_error(e: &ProtonError) -> bool {
match e {
ProtonError::Transport(_) => true,
ProtonError::Api(api) => {
matches!(
api.code,
ResponseCode::RequestTimeout | ResponseCode::Timeout
)
}
_ => false,
}
}
/// The error a call reports when it runs past `after` without an answer.
pub(crate) fn no_answer(after: Duration) -> ProtonError {
ProtonError::Api(ProtonApiError {
code: ResponseCode::RequestTimeout,
http_status: 408,
message: format!("no answer within {}s", after.as_secs()),
details: None,
})
}
/// Run a remote call, giving up after [`INTERACTIVE_CALL_TIMEOUT`].
pub(crate) async fn bounded<T>(
call: impl Future<Output = Result<T, ProtonError>>,
) -> Result<T, ProtonError> {
match tokio::time::timeout(INTERACTIVE_CALL_TIMEOUT, call).await {
Ok(result) => result,
Err(_) => Err(no_answer(INTERACTIVE_CALL_TIMEOUT)),
}
}
/// The next delay between probes, given the current one and how long the
/// outage has lasted.
fn next_probe_delay(delay: Duration, outage: Duration) -> Duration {
let cap = if outage < YOUNG_OUTAGE {
PROBE_YOUNG_MAX
} else {
ONLINE_PROBE_MAX
};
(delay * 2).min(cap)
}
impl Core {
/// Whether the mount currently believes Proton is reachable.
pub(crate) fn is_online(&self) -> bool {
self.online.load(Ordering::Relaxed)
}
/// Run a remote call a caller is waiting on, giving up after
/// [`INTERACTIVE_CALL_TIMEOUT`]. The timer is built inside the runtime: a
/// FUSE worker has no reactor of its own.
pub(crate) fn block_on_bounded<T>(
&self,
call: impl Future<Output = Result<T, ProtonError>>,
) -> Result<T, ProtonError> {
self.rt.block_on(bounded(call))
}
/// Report a failed remote call. Returns whether it failed for want of a
/// network, in which case the mount is now offline and the caller should
/// take its queued path instead of failing.
pub(crate) fn lost_link(&self, e: &ProtonError, what: &str) -> bool {
if !is_network_error(e) {
return false;
}
self.mark_offline(what, e);
true
}
/// Flip the mount offline and wake the probe. Only the first caller of an
/// outage logs it; the rest are the same outage.
pub(crate) fn mark_offline(&self, what: &str, error: &dyn std::fmt::Display) {
let mut lost_at = self.link.lost_at.lock();
if self.online.swap(false, Ordering::Relaxed) {
*lost_at = Some(Instant::now());
warn!(during = what, %error,
"lost the connection to Proton; queueing changes until it returns");
self.link.changed.notify_all();
}
}
/// Flip the mount back online, wake everything waiting for it, and let the
/// queue drain.
fn mark_online(&self) {
{
let mut lost_at = self.link.lost_at.lock();
self.online.store(true, Ordering::Relaxed);
match lost_at.take() {
Some(since) => {
info!(
offline_secs = since.elapsed().as_secs(),
"connection to Proton is back"
)
}
None => info!("back online"),
}
self.link.changed.notify_all();
}
self.relist_offline_listings();
// Anything written while offline is queued and waiting on exactly this,
// including ops that failed on the way down and are sitting out a
// backoff sized for a server that keeps refusing.
if let Err(e) = self.db.retry_failed_ops_now(crate::now_millis()) {
warn!(error = %e, "moving backed-off ops up after a reconnect failed");
}
self.wake_drain();
}
/// Wait up to `budget` for the link to come back. Returns whether it is up.
pub(crate) fn wait_online(&self, budget: Duration) -> bool {
let deadline = Instant::now() + budget;
let mut lost_at = self.link.lost_at.lock();
while !self.is_online() {
if self.shutdown.is_stopping()
|| self
.link
.changed
.wait_until(&mut lost_at, deadline)
.timed_out()
{
return self.is_online();
}
}
true
}
/// Wait out `interval` before a background poll of Proton, and while the
/// link is down, until it is back. Returns as soon as it comes back, so the
/// poll catches up on what changed during the outage at once instead of
/// one interval later. A poll into a dead link would only wait out the
/// SDK's retries and fail.
pub(crate) async fn next_poll(&self, interval: Duration) {
let due = Instant::now() + interval;
let mut was_offline = false;
loop {
if !self.is_online() {
was_offline = true;
} else if was_offline || Instant::now() >= due {
return;
}
tokio::time::sleep(IDLE_CHECK).await;
}
}
/// Watch the link for the life of the mount: sleep while it is up, and
/// probe it back while it is down (offline.md Phase 1).
///
/// A mount that started from the cache is down from the first moment; one
/// that started online goes down when [`Core::mark_offline`] says so.
pub(crate) fn run_online_probe(&self) {
// Only an offline start never had a chance to repair its share id.
let mut repair_share_id = !self.is_online();
loop {
{
let mut lost_at = self.link.lost_at.lock();
while self.is_online() {
if self.shutdown.is_stopping() {
return;
}
self.link.changed.wait_for(&mut lost_at, IDLE_CHECK);
}
}
if !self.probe_until_online() {
return;
}
if std::mem::take(&mut repair_share_id) {
self.repair_primary_share_id(&self.primary_root_uid);
}
}
}
/// Probe until Proton answers, then flip the mount online. Returns `false`
/// if the daemon is stopping.
fn probe_until_online(&self) -> bool {
let started = self.link.lost_at.lock().unwrap_or_else(Instant::now);
let mut delay = if started.elapsed() < YOUNG_OUTAGE {
PROBE_AFTER_DROP
} else {
ONLINE_PROBE_MIN
};
loop {
if !self.shutdown.sleep(delay) {
return false;
}
let probe = self.rt.block_on(async {
match tokio::time::timeout(PROBE_TIMEOUT, self.drive.get_my_files_folder()).await {
Ok(result) => result,
Err(_) => Err(no_answer(PROBE_TIMEOUT)),
}
});
match probe {
Ok(root) => {
{
let root_ino = self.root_ino();
let mut st = self.state();
if let Some(e) = st.entries.get_mut(&root_ino) {
e.node = root.clone();
}
}
if let Err(e) = self.db.upsert_node(&root) {
warn!(error = %e, "refresh root after reconnect failed");
}
self.mark_online();
return true;
}
Err(e) => {
debug!(error = %e, ?delay, "online probe failed; still offline");
delay = next_probe_delay(delay, started.elapsed());
}
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn api(code: ResponseCode) -> ProtonError {
ProtonError::Api(ProtonApiError {
code,
http_status: 422,
message: String::new(),
details: None,
})
}
#[test]
fn only_an_unanswered_call_counts_as_a_lost_link() {
assert!(is_network_error(&no_answer(INTERACTIVE_CALL_TIMEOUT)));
assert!(is_network_error(&api(ResponseCode::Timeout)));
assert!(!is_network_error(&api(ResponseCode::AlreadyExists)));
assert!(!is_network_error(&api(ResponseCode::InvalidRequirements)));
assert!(!is_network_error(&ProtonError::invalid_operation("bug")));
}
#[test]
fn a_young_outage_is_probed_often_and_an_old_one_rarely() {
let young = Duration::from_secs(60);
let old = YOUNG_OUTAGE + Duration::from_secs(1);
assert_eq!(
next_probe_delay(PROBE_AFTER_DROP, young),
2 * PROBE_AFTER_DROP
);
assert_eq!(next_probe_delay(PROBE_YOUNG_MAX, young), PROBE_YOUNG_MAX);
assert_eq!(next_probe_delay(PROBE_YOUNG_MAX, old), 2 * PROBE_YOUNG_MAX);
assert_eq!(next_probe_delay(ONLINE_PROBE_MAX, old), ONLINE_PROBE_MAX);
}
}
+95 -40
View File
@@ -207,11 +207,11 @@ const ROOT_UID_KEY: &str = "root_uid";
/// means an empty tree, and mounting that would show the user an empty Drive
/// rather than an honest error.
fn fetch_or_recall_root(
client: &ProtonDriveClient,
drive: &dyn DriveApi,
rt: &tokio::runtime::Handle,
db: &Db,
) -> std::io::Result<(Node, bool)> {
let err = match rt.block_on(client.get_my_files_folder()) {
let err = match rt.block_on(drive.get_my_files_folder()) {
Ok(root) => {
if let Err(e) = db.set_state_str(ROOT_UID_KEY, &root.uid.to_string()) {
warn!(error = %e, "persist root uid failed");
@@ -276,13 +276,14 @@ pub(super) fn spawn_session(
info!(mountpoint = %mountpoint.display(), "mounting Proton Drive location");
let fs = ProtonFs::new(core.clone(), root);
if core.primary {
// Only My Files rehydrates the daemon-wide node cache. A secondary
// inode space is rooted at one device subtree and must not import every
// unrelated persisted node.
// Only My Files rehydrates the daemon-wide node cache, once. A
// secondary mount shares the tree it fills.
core.hydrate();
}
let session = Session::new(fs, mountpoint, &config)?.spawn()?;
let _ = core.notifier.set(session.notifier());
let _ = core
.notifier
.set(KernelChannel::new(session.notifier(), core.root_ino()));
core.session_live.store(true, Ordering::Release);
Ok(session)
}
@@ -339,6 +340,45 @@ pub fn mount(
control_socket: &Path,
db: Arc<Db>,
options: MountOptions,
) -> std::io::Result<MountOutcome> {
let drive = Arc::new(client.clone());
mount_with(
client,
drive,
rt,
mountpoint,
cache,
control_socket,
db,
options,
Host::Daemon,
)
}
/// What a mount runs in.
pub(crate) enum Host {
/// The daemon: stopped by SIGTERM or SIGINT, and indexing the user's home
/// for the launcher prompt.
Daemon,
/// One of several simulated daemons in a test process (`sim`): stopped by
/// a message on this channel, and with no home to index.
#[cfg(test)]
Simulation(std::sync::mpsc::Receiver<()>),
}
/// [`mount`], reaching Drive through `drive` and hosted by `host`; `client`
/// serves only what [`DriveApi`] does not cover.
#[allow(clippy::too_many_arguments)]
pub(crate) fn mount_with(
client: ProtonDriveClient,
drive: Arc<dyn DriveApi>,
rt: tokio::runtime::Handle,
mountpoint: &Path,
cache: ContentCache,
control_socket: &Path,
db: Arc<Db>,
options: MountOptions,
host: Host,
) -> std::io::Result<MountOutcome> {
let MountOptions {
username,
@@ -349,7 +389,7 @@ pub fn mount(
// Start the uptime clock here rather than at the first `pdfs diagnostics`,
// so the age it reports is the daemon's own.
super::diagnostics::start_clock();
let (root, online) = fetch_or_recall_root(&client, &rt, &db)?;
let (root, online) = fetch_or_recall_root(drive.as_ref(), &rt, &db)?;
let scope = root.tree_event_scope_id();
db.mount_upsert_my_files(&mountpoint.to_string_lossy(), &root.uid.to_string(), None)
.map_err(|error| std::io::Error::other(format!("project My Files mount: {error}")))?;
@@ -364,12 +404,14 @@ pub fn mount(
let paused_until = pause::load_paused_until(&db);
let core = Core {
client: client.clone(),
drive,
client,
rt: rt.clone(),
maintenance: Arc::new(Mutex::new(Default::default())),
primary_root_uid: root.uid.clone(),
primary: true,
state: Arc::new(Mutex::new(State::new(db.clone(), share_access, 2))),
state: Arc::new(Mutex::new(State::new(db.clone(), share_access))),
root: Arc::new(OnceLock::new()),
cache: Arc::new(cache),
readers: Arc::new(Mutex::new(HashMap::new())),
block_ring: Arc::new(Mutex::new(BlockRing::default())),
@@ -383,6 +425,7 @@ pub fn mount(
shared_generation: Arc::new(AtomicU64::new(0)),
shared_refresh_deadlines: Arc::new(Mutex::new(SharedRefreshDeadlines::default())),
online: Arc::new(AtomicBool::new(online)),
link: Arc::new(Default::default()),
sync_paused_until: Arc::new(AtomicI64::new(paused_until)),
pending: Arc::new(Mutex::new(HashMap::new())),
hidden: Arc::new(Mutex::new(HashSet::new())),
@@ -390,6 +433,7 @@ pub fn mount(
shutdown: Arc::new(crate::shutdown::Shutdown::default()),
upload_times: Arc::new(Mutex::new(HashMap::new())),
upload_cancel: Arc::new(Mutex::new(HashMap::new())),
landing_uploads: Arc::new(AtomicU64::new(0)),
timeline_refreshing: Arc::new(AtomicBool::new(false)),
timeline_progress: Arc::new(Mutex::new(None)),
similar_hashing: Arc::new(AtomicBool::new(false)),
@@ -398,7 +442,6 @@ pub fn mount(
trash_progress: Arc::new(tokio::sync::Notify::new()),
conflict_notified: Arc::new(Mutex::new(HashSet::new())),
sweep_mode,
own_sealed_revs: Arc::new(Mutex::new(HashMap::new())),
self_changes: Arc::new(Mutex::new(HashMap::new())),
thumb_gen: Arc::new(Mutex::new(HashSet::new())),
thumb_gen_budget: Arc::new(tokio::sync::Semaphore::new(
@@ -411,6 +454,7 @@ pub fn mount(
thumbnail_build_cancel: Arc::new(tokio::sync::Notify::new()),
thumbnail_misses: Arc::new(Mutex::new(Default::default())),
quota: Arc::new(Mutex::new(None)),
quota_refreshing: Arc::new(AtomicBool::new(false)),
size_upgrades: Arc::new(Mutex::new(HashMap::new())),
size_waiters: Arc::new(Default::default()),
notifier: Arc::new(OnceLock::new()),
@@ -425,7 +469,7 @@ pub fn mount(
states: Arc::new(StateRegistry::default()),
};
// Before anything can queue work against it: the drain thread below reaches
// every mount's inode space through this registry, not through `core.state`.
// every mount's session through this registry.
core.register_state(mountpoint);
core.transfers.set_limits(upload_limit, download_limit);
@@ -484,9 +528,10 @@ pub fn mount(
);
}
// Mounted from the cache: watch for the network coming back so the mount can
// stop being read-only-ish without the user restarting the daemon.
if !online {
// Watch the link for the life of the mount: a mount from the cache waits for
// the network to come back, and an online one can lose it at any moment and
// must notice it coming back just the same.
{
let core = core.clone();
workers.push(
std::thread::Builder::new()
@@ -497,7 +542,7 @@ pub fn mount(
// Keep the launcher prompt's "This computer" index fresh. Its own thread:
// the walk is I/O-heavy and must never sit in front of a FUSE callback.
{
if matches!(host, Host::Daemon) {
let db = core.db.clone();
let indexing = core.indexing.clone();
let transfers = core.transfers.clone();
@@ -621,7 +666,7 @@ pub fn mount(
}
}
};
rt.spawn(run_event_sync(client, scope, core.clone()));
rt.spawn(run_event_sync(core.drive.clone(), scope, core.clone()));
// The photos volume has its own event stream; without it the gallery only
// learns about a phone-side deletion when the timeline goes stale.
rt.spawn(run_photos_event_sync(core.clone()));
@@ -640,30 +685,11 @@ pub fn mount(
// are delivered onto the async runtime; bridge them onto a sync channel so
// the loop below can react without blocking a worker thread. A bounded
// channel of 1 is enough — we only need to know that *a* stop arrived.
let (sig_tx, sig_rx) = std::sync::mpsc::sync_channel::<()>(1);
rt.spawn(async move {
let mut sigterm =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGTERM handler failed");
return;
}
};
let mut sigint =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGINT handler failed");
return;
}
};
tokio::select! {
_ = sigterm.recv() => info!("received SIGTERM"),
_ = sigint.recv() => info!("received SIGINT"),
}
let _ = sig_tx.try_send(());
});
let sig_rx = match host {
Host::Daemon => signal_stop(&rt),
#[cfg(test)]
Host::Simulation(rx) => rx,
};
// Wait for whichever happens first: a stop signal (→ we unmount ourselves
// via the lazy MNT_DETACH path, clean even mid-download), or the kernel
@@ -720,6 +746,35 @@ pub fn mount(
Ok(outcome)
}
/// A channel that receives once SIGTERM or SIGINT arrives.
fn signal_stop(rt: &tokio::runtime::Handle) -> std::sync::mpsc::Receiver<()> {
let (sig_tx, sig_rx) = std::sync::mpsc::sync_channel::<()>(1);
rt.spawn(async move {
let mut sigterm =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGTERM handler failed");
return;
}
};
let mut sigint =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGINT handler failed");
return;
}
};
tokio::select! {
_ = sigterm.recv() => info!("received SIGTERM"),
_ = sigint.recv() => info!("received SIGINT"),
}
let _ = sig_tx.try_send(());
});
sig_rx
}
/// How long teardown waits for the FUSE worker pool before proceeding without
/// it. Long enough for an in-flight block to finish, short enough that
/// `systemctl --user stop` still returns inside `TimeoutStopSec`.
+1 -1
View File
@@ -1038,7 +1038,7 @@ impl Core {
}
let outcomes = self
.rt
.block_on(self.client.trash_nodes(uids))
.block_on(self.drive.trash_nodes(uids))
.map_err(|e| CoreError::from_api(&e, "trash"))?;
let (trashed, failed) = pdfs_core::batch::split(outcomes);
+8 -8
View File
@@ -101,7 +101,7 @@ impl Core {
match existing {
Some(uid) => self
.rt
.block_on(self.client.upload_new_revision_from(
.block_on(self.drive.upload_new_revision_from(
&uid,
Cursor::new(bytes),
len,
@@ -112,7 +112,7 @@ impl Core {
.map_err(|e| CoreError::from_api(&e, "upload profile revision"))?,
None => {
self.rt
.block_on(self.client.upload_file_replacing_draft_from(
.block_on(self.drive.upload_file_replacing_draft_from(
&dir_uid,
PROFILE_FILE_NAME,
"application/json",
@@ -182,14 +182,14 @@ impl Core {
) -> CoreResult<Option<NodeUid>> {
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(parent_uid))
.block_on(self.drive.enumerate_folder_children_node_uids(parent_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() {
return Ok(None);
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&uids))
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?;
Ok(nodes
.into_iter()
@@ -210,7 +210,7 @@ impl Core {
let uid = self
.rt
.block_on(
self.client
self.drive
.create_folder(root_uid, PROFILE_DIR_NAME, Some(now_secs())),
)
.map_err(|e| CoreError::from_api(&e, "create profile folder"))?;
@@ -241,7 +241,7 @@ impl Core {
};
let mut buf: Vec<u8> = Vec::new();
self.rt
.block_on(self.client.download_file_to(&uid, &mut buf))
.block_on(self.drive.download_file_to(&uid, &mut buf))
.map_err(|e| CoreError::from_api(&e, "download profile"))?;
Profile::parse(&buf).map(Some).map_err(CoreError::invalid)
}
@@ -277,13 +277,13 @@ impl Core {
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(&root_uid))
.block_on(self.drive.enumerate_folder_children_node_uids(&root_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?;
let nodes = if uids.is_empty() {
Vec::new()
} else {
self.rt
.block_on(self.client.enumerate_nodes(&uids))
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?
};
+87 -31
View File
@@ -36,6 +36,21 @@ pub(super) const REPAIR_MAX: u64 = 512 * 1024 * 1024;
/// demand read in the SDK's flat block semaphore.
pub(super) const PREFETCH_BUDGET: usize = 8;
/// How long one read may wait on the network before it fails with `EIO`.
///
/// The SDK bounds each request, not a read: a block that times out is retried,
/// and a read can need several blocks, a block table and a repair. On a link
/// that stalled without dropping, one `read` held a FUSE worker for 34 minutes
/// and the application behind it with it (`docs/BUGS.md` B107). An error the
/// caller can retry beats a process stuck in `D` state.
pub(super) const READ_FETCH_TIMEOUT: Duration = Duration::from_secs(120);
/// How long a read of blocks nothing has cached waits for a lost link to come
/// back before failing. Long enough to ride out a Wi-Fi blip, short enough
/// that a program reading while the laptop is really offline gets its `EIO`
/// before the user gives up on it.
const READ_OFFLINE_WAIT: Duration = Duration::from_secs(30);
/// Cap on tracked sequential streams. Only reads in progress matter, so the map
/// is cleared wholesale rather than aged — losing the ramp costs one read.
const MAX_PREFETCH_STREAMS: usize = 256;
@@ -55,7 +70,7 @@ const MAX_OPEN_READERS: usize = 64;
/// `(mtime, size)` — the same validity pair the content cache uses (a new
/// revision bumps mtime). On a mismatch the reader is dropped and reopened.
pub(super) struct CachedReader {
reader: Arc<RevisionReader>,
reader: Arc<dyn RevisionRead>,
mtime: i64,
size: u64,
/// For LRU eviction.
@@ -86,7 +101,7 @@ pub(super) struct PendingOpen {
/// must not join this open — it would get a reader for the wrong bytes.
mtime: i64,
size: u64,
rx: tokio::sync::watch::Receiver<Option<Result<Arc<RevisionReader>, Errno>>>,
rx: tokio::sync::watch::Receiver<Option<Result<Arc<dyn RevisionRead>, Errno>>>,
}
/// Distinguishes `PendingOpen`s. Only uniqueness matters, so a plain counter
@@ -298,16 +313,16 @@ impl Core {
uid: &NodeUid,
mtime: i64,
fsize: u64,
) -> Result<Arc<RevisionReader>, Errno> {
) -> Result<Arc<dyn RevisionRead>, Errno> {
loop {
// Decide under the lock, act outside it — network I/O must never run
// under a std `Mutex`, and an await must never hold one.
enum Act {
Hit(Arc<RevisionReader>),
Join(tokio::sync::watch::Receiver<Option<Result<Arc<RevisionReader>, Errno>>>),
Hit(Arc<dyn RevisionRead>),
Join(tokio::sync::watch::Receiver<Option<Result<Arc<dyn RevisionRead>, Errno>>>),
Lead(
u64,
tokio::sync::watch::Sender<Option<Result<Arc<RevisionReader>, Errno>>>,
tokio::sync::watch::Sender<Option<Result<Arc<dyn RevisionRead>, Errno>>>,
),
}
let act = {
@@ -361,9 +376,10 @@ impl Core {
}
Act::Lead(id, tx) => {
debug!(%uid, mtime, fsize, "opening revision");
let result = match self.client.open_revision(uid).await {
Ok(reader) => Ok(Arc::new(reader)),
let result = match self.drive.open_revision(uid).await {
Ok(reader) => Ok(reader),
Err(e) => {
self.lost_link(&e, "open a revision");
warn!(%uid, error = %e, "open_revision failed");
Err(Errno::EIO)
}
@@ -552,7 +568,7 @@ impl Core {
/// leader's claim is gone by then, so this cannot recurse.
async fn fetch_block(
&self,
reader: &Arc<RevisionReader>,
reader: &Arc<dyn RevisionRead>,
uid: &NodeUid,
mtime: i64,
fsize: u64,
@@ -618,7 +634,7 @@ impl Core {
/// download, or the read fails. See [`Core::repair_block`] (bugs.md B84).
async fn read_block(
&self,
reader: &Arc<RevisionReader>,
reader: &Arc<dyn RevisionRead>,
uid: &NodeUid,
mtime: i64,
fsize: u64,
@@ -649,6 +665,7 @@ impl Core {
.store_block_geometry(uid, mtime, fsize, reader.block_sizes());
}
let bytes = reader.read_at(span.start, span.len).await.map_err(|e| {
self.lost_link(&e, "read a block");
warn!(%uid, start = span.start, len = span.len, error = %e, "block read failed");
Errno::EIO
})?;
@@ -761,7 +778,7 @@ impl Core {
error!(%uid, %error, "creating a scratch file for a repair download failed");
Errno::EIO
})?;
let outcome = self.client.download_file_to(uid, &mut file).await;
let outcome = self.drive.download_file_to(uid, &mut file).await;
let stored = (|| {
outcome.map_err(|error| {
warn!(%uid, %error, "repair download failed");
@@ -939,28 +956,67 @@ impl Core {
}
if !misses.is_empty() {
let fetched = self.rt.block_on(async {
// Resolve the file's keys and block table once, then read every
// missing block through the shared reader. Previously each block
// called `download_range`, which redid that resolution per block.
let reader = self.revision_reader(uid, mtime, fsize).await?;
// One deadline for the whole read, including any wait for a lost
// link. A fetch that failed because the link went down is tried
// once more when it is back, so a blip is a slow read, not an `EIO`.
let deadline = Instant::now() + READ_FETCH_TIMEOUT;
let mut retried = false;
let fetched = loop {
if !self.is_online() {
let wait =
READ_OFFLINE_WAIT.min(deadline.saturating_duration_since(Instant::now()));
if !self.wait_online(wait) {
debug!(%uid, offset, len, "read of uncached blocks while offline; failing it");
return Err(Errno::EIO);
}
}
let fetch = async {
// Resolve the file's keys and block table once, then read every
// missing block through the shared reader. Previously each block
// called `download_range`, which redid that resolution per block.
let reader = self.revision_reader(uid, mtime, fsize).await?;
let mut set = tokio::task::JoinSet::new();
for &span in &misses {
let (core, reader, uid) = (self.clone(), reader.clone(), uid.clone());
set.spawn(async move {
core.fetch_block(&reader, &uid, mtime, fsize, span, cache_blocks)
.await
.map(|bytes| (span.idx, bytes))
});
let mut set = tokio::task::JoinSet::new();
for &span in &misses {
let (core, reader, uid) = (self.clone(), reader.clone(), uid.clone());
set.spawn(async move {
core.fetch_block(&reader, &uid, mtime, fsize, span, cache_blocks)
.await
.map(|bytes| (span.idx, bytes))
});
}
let mut out = Vec::with_capacity(misses.len());
while let Some(joined) = set.join_next().await {
// A join error means the task panicked; surface it as EIO.
out.push(joined.map_err(|_| Errno::EIO)??);
}
Ok::<_, Errno>(out)
};
// Dropping the fetch on the deadline aborts the block tasks with
// it; blocks that already landed stay cached for the retry. The
// timer is built inside the runtime: a FUSE worker has no reactor
// of its own, and building it out here panicked every remote read.
let budget = deadline.saturating_duration_since(Instant::now());
match self
.rt
.block_on(async { tokio::time::timeout(budget, fetch).await })
{
Ok(Ok(fetched)) => break fetched,
// The fetch saw the link go down; the top of the loop waits
// for it within what is left of the deadline.
Ok(Err(_)) if !retried && !self.is_online() => retried = true,
Ok(Err(e)) => return Err(e),
Err(_) => {
warn!(
%uid, offset, len, secs = READ_FETCH_TIMEOUT.as_secs(),
"read got no answer from the network in time; failing it"
);
self.evict_reader(uid);
self.mark_offline("read", &crate::link::no_answer(READ_FETCH_TIMEOUT));
return Err(Errno::EIO);
}
}
let mut out = Vec::with_capacity(misses.len());
while let Some(joined) = set.join_next().await {
// A join error means the task panicked; surface it as EIO.
out.push(joined.map_err(|_| Errno::EIO)??);
}
Ok::<_, Errno>(out)
})?;
};
for (bidx, bytes) in fetched {
blocks[(bidx - first) as usize] = Some(bytes);
}
+1 -4
View File
@@ -248,10 +248,7 @@ impl Core {
let outcomes = self
.rt
.block_on(
self.client
.trash_nodes(std::slice::from_ref(&candidate.uid)),
)
.block_on(self.drive.trash_nodes(std::slice::from_ref(&candidate.uid)))
.map_err(|e| CoreError::from_api(&e, "trash the mis-dated original"))?;
if let Some((_, Err(e))) = outcomes.into_iter().next() {
return Err(CoreError::from_api(&e, "trash the mis-dated original"));
+611
View File
@@ -0,0 +1,611 @@
//! Moving a file or folder between two locations without moving its content.
//!
//! `~/ProtonDrive` and every on-demand synced folder are separate FUSE
//! sessions, and a mirror folder is a plain local directory. The kernel refuses
//! `rename(2)` between two mounts with `EXDEV` before the daemon hears of it, so
//! `mv` falls back to copy and delete: every file is downloaded, uploaded again,
//! and the original is trashed.
//!
//! On Proton Drive the same move is one metadata change, because My files and
//! this computer's device folders all live on the main volume. So a move
//! request that names two absolute paths is resolved here, against every
//! location this daemon knows, and done as a server-side move. The local side
//! is then brought level: a mount forgets the node and re-lists, and a mirror
//! folder either renames its copy along or drops it once Drive has it.
use std::collections::HashMap;
use std::path::{Component, Path, PathBuf};
use std::sync::Arc;
use std::time::Duration;
use parking_lot::Mutex;
use pdfs_core::db::{StoredSyncEntry, StoredSyncFolder};
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::NodeKind;
use proton_drive_rs::proton_sdk::ids::NodeUid;
use tracing::{info, warn};
use super::sync::mirror_subtree_unsynced;
use super::{Core, is_local_uid, parse_uid};
/// How long a move waits for a mirror folder's sync pass to finish before it
/// reports the folder busy. A pass can run for hours on a first sync, so the
/// move does not wait it out.
const MIRROR_LOCK_WAIT: Duration = Duration::from_secs(5);
/// Where one side of a move lives.
enum Place {
/// My files or an on-demand folder: a live FUSE mount. `core` is rooted at
/// that mount and `rel` is relative to its mountpoint.
Mounted { core: Box<Core>, rel: PathBuf },
/// A mirror folder: a local directory kept in step by the sync engine.
/// `rel` is `/`-separated, relative to the folder, as the baseline keys it.
Mirror {
folder: StoredSyncFolder,
rel: String,
},
}
impl Place {
fn mirror_id(&self) -> Option<i64> {
match self {
Place::Mirror { folder, .. } => Some(folder.id),
Place::Mounted { .. } => None,
}
}
fn is_root(&self) -> bool {
match self {
Place::Mounted { rel, .. } => rel.as_os_str().is_empty(),
Place::Mirror { rel, .. } => rel.is_empty(),
}
}
}
/// The mirror folder that holds `abs`, and `abs` relative to it. The deepest
/// folder wins, although folders are not expected to nest.
fn mirror_covering(
folders: Vec<StoredSyncFolder>,
abs: &Path,
) -> Option<(StoredSyncFolder, String)> {
let folder = folders
.into_iter()
.filter(|f| f.mode == "mirror" && abs.starts_with(&f.local_path))
.max_by_key(|f| Path::new(&f.local_path).components().count())?;
let rel = abs.strip_prefix(&folder.local_path).ok()?;
let parts = rel
.components()
.map(|c| match c {
Component::Normal(part) => part.to_str(),
_ => None,
})
.collect::<Option<Vec<_>>>()?;
Some((folder, parts.join("/")))
}
/// `name` inside the folder at `parent`, in baseline form.
fn join_rel(parent: &str, name: &str) -> String {
if parent.is_empty() {
name.to_string()
} else {
format!("{parent}/{name}")
}
}
/// Rename `from` to `to`, failing with `EEXIST` rather than replacing
/// whatever appeared at `to` since it was checked. A filesystem that cannot
/// promise that gets a plain rename after one more look.
fn rename_noreplace(from: &Path, to: &Path) -> std::io::Result<()> {
use std::ffi::CString;
use std::os::unix::ffi::OsStrExt;
let cstr = |p: &Path| {
CString::new(p.as_os_str().as_bytes())
.map_err(|_| std::io::Error::from_raw_os_error(libc::EINVAL))
};
let (c_from, c_to) = (cstr(from)?, cstr(to)?);
// SAFETY: both are valid NUL-terminated paths that outlive the call.
let rc = unsafe {
libc::syscall(
libc::SYS_renameat2,
libc::AT_FDCWD,
c_from.as_ptr(),
libc::AT_FDCWD,
c_to.as_ptr(),
libc::RENAME_NOREPLACE,
)
};
if rc == 0 {
return Ok(());
}
let err = std::io::Error::last_os_error();
match err.raw_os_error() {
Some(libc::EINVAL | libc::ENOSYS) => {
if std::fs::symlink_metadata(to).is_ok() {
return Err(std::io::Error::from_raw_os_error(libc::EEXIST));
}
std::fs::rename(from, to)
}
_ => Err(err),
}
}
/// Whether `uid` is `root` or lies under it, as far as the nodes table knows.
fn under(core: &Core, root: &NodeUid, uid: &str) -> bool {
let root = root.to_string();
uid == root || matches!(core.db.path_relative_to(&root, uid), Ok(Some(_)))
}
impl Core {
/// Move the file or folder at `src` into the folder at `dest_parent`. Both
/// are absolute local paths and may be in different locations: My files,
/// an on-demand folder, or a mirror folder. Returns the moved name.
///
/// Within one mount this is [`Core::move_to`]. Across locations it
/// is a server-side move followed by local bookkeeping; see the module
/// documentation. Anything the move could lose is refused up front.
pub(crate) fn move_between(&self, src: &Path, dest_parent: &Path) -> CoreResult<String> {
let source = self.locate(src)?;
let dest = self.locate(dest_parent)?;
if let (
Place::Mounted { core, rel },
Place::Mounted {
core: dest_core,
rel: dest_rel,
},
) = (&source, &dest)
&& Arc::ptr_eq(&core.root, &dest_core.root)
{
return core.move_to(rel, dest_rel);
}
if source.is_root() {
return Err(CoreError::invalid(format!(
"{} is the root of a synced location and cannot be moved",
src.display()
)));
}
if dest_parent.starts_with(src) {
return Err(CoreError::invalid("cannot move a folder into itself"));
}
let name = src
.file_name()
.and_then(|n| n.to_str())
.ok_or_else(|| CoreError::invalid(format!("{} has no usable name", src.display())))?
.to_string();
// Hold every mirror folder involved, lowest id first, so no sync pass
// sees the move half done and two moves cannot deadlock.
let mut ids: Vec<i64> = [source.mirror_id(), dest.mirror_id()]
.into_iter()
.flatten()
.collect();
ids.sort_unstable();
ids.dedup();
let locks: Vec<Arc<Mutex<()>>> = ids.iter().map(|&id| self.sync_lock(id)).collect();
let mut guards = Vec::with_capacity(locks.len());
for lock in &locks {
guards.push(lock.try_lock_for(MIRROR_LOCK_WAIT).ok_or_else(|| {
CoreError::conflict("a synced folder is busy syncing; try again when it is done")
})?);
}
let mut baselines: HashMap<i64, HashMap<String, StoredSyncEntry>> = HashMap::new();
for &id in &ids {
let baseline = self
.db
.sync_entries(id)
.map_err(|e| CoreError::internal(format!("db: {e}")))?;
baselines.insert(id, baseline);
}
let src_uid = self.source_uid(&source, &baselines)?;
let dest_uid = self.dest_uid(&dest, &baselines, &name)?;
if src_uid.volume_id != dest_uid.volume_id {
return Err(CoreError::invalid(
"the two locations are on different Proton Drive volumes; copy it instead",
));
}
if under(self, &src_uid, &dest_uid.to_string()) {
return Err(CoreError::invalid("cannot move a folder into itself"));
}
// Mirror to mirror renames the local copy along, so it needs nothing
// proven about it. Another filesystem makes that impossible, and the
// pair then drops the source copy like any other move out of a mirror.
let mut renamed = None;
if let (
Place::Mirror { folder, rel },
Place::Mirror {
folder: dest_folder,
rel: dest_rel,
},
) = (&source, &dest)
{
let from = src.to_path_buf();
let to = dest_parent.join(&name);
match rename_noreplace(&from, &to) {
Ok(()) => renamed = Some((from, to)),
Err(e) if e.raw_os_error() == Some(libc::EXDEV) => {
info!(from = %folder.local_path, to = %dest_folder.local_path,
"move: mirror folders are on different filesystems");
}
Err(e) if e.raw_os_error() == Some(libc::EEXIST) => {
return Err(CoreError::conflict(format!(
"{name} already exists in {}",
dest_parent.display()
)));
}
Err(e) => {
return Err(CoreError::internal(format!(
"could not move {} to {}: {e}",
rel,
join_rel(dest_rel, &name)
)));
}
}
}
if renamed.is_none()
&& let Place::Mirror { folder, rel } = &source
&& let Some(reason) =
mirror_subtree_unsynced(Path::new(&folder.local_path), rel, &baselines[&folder.id])
{
return Err(CoreError::conflict(format!(
"{name} is not fully synced yet ({reason}); wait for the sync to finish"
)));
}
if let Err(e) = self.move_remote(&src_uid, &dest_uid) {
if let Some((from, to)) = &renamed
&& let Err(back) = std::fs::rename(to, from)
{
warn!(from = %to.display(), to = %from.display(), error = %back,
"move: could not put the local copy back after the move failed");
}
return Err(CoreError::from_api(&e, "move"));
}
self.settle_source(
&source,
&src_uid,
renamed.is_some(),
&dest,
&dest_uid,
&name,
src,
);
self.settle_dest(&dest, &dest_uid, &name);
drop(guards);
if renamed.is_none()
&& let Place::Mirror { folder, .. } = &dest
{
// Nothing local exists yet; the folder's next pass downloads it.
self.sync_now(Some(folder.id));
}
Ok(name)
}
/// Which location `abs` is in. A mount is looked for first: a mirror folder
/// cannot live inside a FUSE mount, and a mount is what the kernel sees.
fn locate(&self, abs: &Path) -> CoreResult<Place> {
if let Some((core, rel)) = self.rooted_at(abs) {
if !rel
.components()
.all(|c| matches!(c, Component::Normal(_) | Component::CurDir))
{
return Err(CoreError::invalid(format!(
"{} escapes its location",
abs.display()
)));
}
return Ok(Place::Mounted {
core: Box::new(core),
rel,
});
}
let folders = self
.db
.sync_folder_list()
.map_err(|e| CoreError::internal(format!("db: {e}")))?;
mirror_covering(folders, abs)
.map(|(folder, rel)| Place::Mirror { folder, rel })
.ok_or_else(|| {
CoreError::invalid(format!(
"{} is not in a Proton Drive location",
abs.display()
))
})
}
/// The node being moved, once everything that must hold before it leaves
/// its location has been checked.
fn source_uid(
&self,
source: &Place,
baselines: &HashMap<i64, HashMap<String, StoredSyncEntry>>,
) -> CoreResult<NodeUid> {
match source {
Place::Mounted { core, rel } => {
let (ino, uid) = core.resolve(rel)?;
if is_local_uid(&uid) {
return Err(CoreError::conflict(
"it is not uploaded yet; wait for the upload to finish",
));
}
let old_parent = core.source_parent_uid(ino, rel)?;
for authority in [&uid, &old_parent] {
core.require_uid_writable(authority)
.map_err(|e| core.errno_error(e, "move access"))?;
}
if self.has_pending_work_under(&uid)? {
return Err(CoreError::conflict(
"it has changes that are not uploaded yet; wait for the upload to finish",
));
}
Ok(uid)
}
Place::Mirror { folder, rel } => baselines[&folder.id]
.get(rel)
.and_then(|entry| entry.remote_uid.as_deref())
.and_then(parse_uid)
.ok_or_else(|| {
CoreError::conflict(
"it is not on Proton Drive yet; wait for the sync to finish",
)
}),
}
}
/// The folder the node moves into. Refuses one that already holds `name`
/// on this disk; Drive refuses a clash on its side by itself.
fn dest_uid(
&self,
dest: &Place,
baselines: &HashMap<i64, HashMap<String, StoredSyncEntry>>,
name: &str,
) -> CoreResult<NodeUid> {
let uid = match dest {
Place::Mounted { core, rel } => {
let (ino, uid) = core.resolve(rel)?;
let is_folder = core
.state()
.entries
.get(&ino)
.is_some_and(|entry| matches!(entry.node.kind, NodeKind::Folder));
if !is_folder {
return Err(CoreError::invalid("the destination is not a folder"));
}
core.require_uid_writable(&uid)
.map_err(|e| core.errno_error(e, "move access"))?;
uid
}
Place::Mirror { folder, rel } => {
let local = Path::new(&folder.local_path).join(rel);
if !local.is_dir() {
return Err(CoreError::invalid("the destination is not a folder"));
}
if std::fs::symlink_metadata(local.join(name)).is_ok() {
return Err(CoreError::conflict(format!(
"{name} already exists in the destination"
)));
}
let uid = if rel.is_empty() {
Some(folder.remote_uid.as_str())
} else {
baselines[&folder.id]
.get(rel)
.and_then(|entry| entry.remote_uid.as_deref())
};
uid.and_then(parse_uid).ok_or_else(|| {
CoreError::conflict(
"the destination is not on Proton Drive yet; wait for the sync to finish",
)
})?
}
};
if is_local_uid(&uid) {
return Err(CoreError::conflict(
"the destination is not uploaded yet; wait for the upload to finish",
));
}
Ok(uid)
}
/// Whether the queue owes Drive anything for `uid` or a node under it, or a
/// file under it is open for writing. Moving such a node away would strand
/// the pending change against a path that no longer leads to it.
fn has_pending_work_under(&self, uid: &NodeUid) -> CoreResult<bool> {
let ops = self
.db
.pending_ops()
.map_err(|e| CoreError::internal(format!("db: {e}")))?;
if ops.iter().any(|op| {
under(self, uid, &op.uid)
|| op
.parent_uid
.as_deref()
.is_some_and(|p| under(self, uid, p))
}) {
return Ok(true);
}
// Collected first so no State lock is held across the nodes-table reads.
let mut writing = Vec::new();
self.for_each_state(|st| {
writing.extend(st.active_writes.values().map(|w| w.uid.to_string()));
});
Ok(writing.iter().any(|w| under(self, uid, w)))
}
/// Bring the source location level with a move Drive has made.
#[allow(clippy::too_many_arguments)]
fn settle_source(
&self,
source: &Place,
uid: &NodeUid,
renamed: bool,
dest: &Place,
dest_uid: &NodeUid,
name: &str,
src: &Path,
) {
match source {
Place::Mounted { core, rel } => {
// Moved everywhere, so no mount keeps serving it under the old
// path and its row stays (`docs/BUGS.md` B117). The content
// cache is keyed by uid and stays valid.
self.relocate_everywhere(uid, dest_uid, name);
core.invalidate_parent_listing(rel);
}
Place::Mirror { folder, rel } if renamed => {
let Place::Mirror {
folder: dest_folder,
rel: dest_rel,
} = dest
else {
unreachable!("a local rename needs a mirror destination");
};
if let Err(e) = self.db.sync_entries_move(
folder.id,
rel,
dest_folder.id,
&join_rel(dest_rel, name),
) {
// The next passes see a copy with no baseline on one side
// and a baseline with no copy on the other. That uploads
// the files again and trashes the moved node, but loses
// nothing.
warn!(error = ?e, "move: could not carry the sync baseline along");
}
}
Place::Mirror { folder, rel } => {
// Drive had all of it when the move was checked, so the local
// copy is only a stale duplicate now. Something written into it
// since would go with it, so look again first; if anything is
// new, the copy stays and the next pass uploads it as new.
let changed = match self.db.sync_entries(folder.id) {
Ok(baseline) => {
mirror_subtree_unsynced(Path::new(&folder.local_path), rel, &baseline)
}
Err(e) => Some(format!("db: {e}")),
};
if let Err(e) = self.db.sync_entries_remove_subtree(folder.id, rel) {
warn!(error = ?e, "move: could not drop the sync baseline");
}
if let Some(reason) = changed {
warn!(path = %src.display(), reason,
"move: the local copy changed during the move; keeping it");
return;
}
let removed = if src.is_dir() {
std::fs::remove_dir_all(src)
} else {
std::fs::remove_file(src)
};
if let Err(e) = removed {
warn!(path = %src.display(), error = %e,
"move: could not remove the local copy after the move");
}
}
}
}
/// Bring the destination location level with a move Drive has made.
fn settle_dest(&self, dest: &Place, uid: &NodeUid, name: &str) {
if let Place::Mounted { .. } = dest {
self.invalidate_children_of(uid);
// A lookup that missed before the move may be cached as absent.
self.for_each_mount(|st, notify| {
if let Some(&ino) = st.by_uid.get(uid) {
notify.inval_entry(ino, name.to_string());
}
});
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn folder(id: i64, local_path: &str, mode: &str) -> StoredSyncFolder {
StoredSyncFolder {
id,
local_path: local_path.to_string(),
remote_uid: format!("vol~root{id}"),
remote_share_id: String::new(),
mode: mode.to_string(),
pending_mode: None,
state: "idle".to_string(),
last_sync: 0,
}
}
#[test]
fn mirror_covering_finds_the_mirror_folder_and_the_path_inside_it() {
let folders = vec![folder(1, "/home/u/Docs", "mirror")];
let (found, rel) = mirror_covering(folders, Path::new("/home/u/Docs/a/b.txt")).unwrap();
assert_eq!(found.id, 1);
assert_eq!(rel, "a/b.txt");
}
#[test]
fn mirror_covering_gives_an_empty_path_for_the_folder_itself() {
let folders = vec![folder(1, "/home/u/Docs", "mirror")];
let (_, rel) = mirror_covering(folders, Path::new("/home/u/Docs")).unwrap();
assert_eq!(rel, "");
}
#[test]
fn mirror_covering_prefers_the_deepest_folder() {
let folders = vec![
folder(1, "/home/u/Docs", "mirror"),
folder(2, "/home/u/Docs/Work", "mirror"),
];
let (found, rel) = mirror_covering(folders, Path::new("/home/u/Docs/Work/x")).unwrap();
assert_eq!(found.id, 2);
assert_eq!(rel, "x");
}
#[test]
fn mirror_covering_skips_on_demand_folders_and_name_prefixed_siblings() {
let folders = || {
vec![
folder(1, "/home/u/Docs", "mirror"),
folder(2, "/home/u/Media", "ondemand"),
]
};
assert!(mirror_covering(folders(), Path::new("/home/u/Media/x")).is_none());
assert!(mirror_covering(folders(), Path::new("/home/u/Docs2/x")).is_none());
}
#[test]
fn mirror_covering_refuses_a_path_that_climbs_out() {
let folders = vec![folder(1, "/home/u/Docs", "mirror")];
assert!(mirror_covering(folders, Path::new("/home/u/Docs/../etc")).is_none());
}
#[test]
fn join_rel_names_a_child_of_the_folder_root_without_a_slash() {
assert_eq!(join_rel("", "a"), "a");
assert_eq!(join_rel("x/y", "a"), "x/y/a");
}
#[test]
fn a_local_rename_never_replaces_what_is_already_there() {
let dir = std::env::temp_dir().join(format!("pdfs-relocate-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let (a, b, c) = (dir.join("a"), dir.join("b"), dir.join("c"));
std::fs::write(&a, b"moved").unwrap();
std::fs::write(&b, b"kept").unwrap();
let err = rename_noreplace(&a, &b).unwrap_err();
assert_eq!(err.raw_os_error(), Some(libc::EEXIST));
assert_eq!(std::fs::read(&b).unwrap(), b"kept");
assert!(a.exists());
rename_noreplace(&a, &c).unwrap();
assert!(!a.exists());
assert_eq!(std::fs::read(&c).unwrap(), b"moved");
std::fs::remove_dir_all(&dir).unwrap();
}
}
+1 -1
View File
@@ -232,7 +232,7 @@ impl Core {
let mut out = std::io::BufWriter::new(file);
let written = match self
.rt
.block_on(self.client.download_revision_to(uid, revision_id, &mut out))
.block_on(self.drive.download_revision_to(uid, revision_id, &mut out))
.map_err(|e| CoreError::from_api(&e, "download revision"))
.and_then(|()| {
use std::io::Write as _;
+34 -28
View File
@@ -21,7 +21,7 @@ use proton_drive_rs::{MemberRole, NodeKind};
use proton_drive_rs::Node;
use super::{Core, ROOT_INO, node_size, parse_uid, public_link_info, role_from_str, role_to_str};
use super::{Core, node_size, parse_uid, public_link_info, role_from_str, role_to_str};
/// A node from a *shared* listing as a [`DirEntry`].
///
@@ -463,7 +463,7 @@ impl Core {
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&uids))
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes
.into_iter()
@@ -481,14 +481,14 @@ impl Core {
parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?;
let child_uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(&uid))
.block_on(self.drive.enumerate_folder_children_node_uids(&uid))
.map_err(|e| CoreError::from_api(&e, "enumerate shared children"))?;
if child_uids.is_empty() {
return Ok(Vec::new());
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&child_uids))
.block_on(self.drive.enumerate_nodes(&child_uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes
.into_iter()
@@ -509,7 +509,7 @@ impl Core {
parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?;
let node = self
.rt
.block_on(self.client.get_node(&uid))
.block_on(self.drive.get_node(&uid))
.map_err(|e| CoreError::from_api(&e, "get shared node"))?
.ok_or_else(|| CoreError::not_found("shared node not found"))?;
if !node.is_file() {
@@ -610,10 +610,10 @@ impl Core {
/// Proton products (not Drive-only). A remote round-trip; nothing here is
/// cached.
pub(crate) fn account_quota(&self) -> CoreResult<(i64, i64)> {
let q = self
.rt
.block_on(self.client.quota())
.map_err(|e| CoreError::from_api(&e, "account quota"))?;
let q = self.block_on_bounded(self.drive.quota()).map_err(|e| {
self.lost_link(&e, "account quota");
CoreError::from_api(&e, "account quota")
})?;
*self.quota.lock() = Some((std::time::Instant::now(), q.max_space, q.used_space));
Ok((q.max_space, q.used_space))
}
@@ -632,11 +632,22 @@ impl Core {
{
return Some((max_space, used_space));
}
match self.account_quota() {
let last = || self.quota.lock().map(|(_, max, used)| (max, used));
if !self.is_online()
|| self
.quota_refreshing
.swap(true, std::sync::atomic::Ordering::AcqRel)
{
return last();
}
let refreshed = self.account_quota();
self.quota_refreshing
.store(false, std::sync::atomic::Ordering::Release);
match refreshed {
Ok(q) => Some(q),
Err(error) => {
debug!(%error, "account quota refresh failed; serving the last known figures");
self.quota.lock().map(|(_, max, used)| (max, used))
last()
}
}
}
@@ -658,7 +669,7 @@ impl Core {
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&uids))
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
let mut items = Vec::with_capacity(nodes.len());
for n in nodes {
@@ -703,19 +714,10 @@ impl Core {
/// been seen through the mount (e.g. shared but not browsed to this session) —
/// the caller then leaves the path empty.
pub(crate) fn rel_path_for_uid(&self, uid: &NodeUid) -> Option<String> {
// Searched across every mount: a node browsed through a sync folder is
// interned in that fork's inode space, so the primary state alone had no
// entry for it and the caller silently showed an empty path
// (`docs/BUGS.md` B74). The path is relative to whichever mount holds
// it, which is the one the user reached it through.
let mut path = None;
self.for_each_state(|st| {
if path.is_some() {
return;
}
path = walk_to_root(st, uid);
});
path
// Relative to whichever root the node sits under: a node browsed
// through a sync folder is under that fork's root, not My Files
// (`docs/BUGS.md` B74).
walk_to_root(&self.state(), uid)
}
}
@@ -760,14 +762,18 @@ impl StopSharingOutcome {
}
}
/// Walk one inode space from `uid` up to its root, building the path. `None` if
/// that mount has never interned the node, or if the chain breaks partway —
/// Walk the tree from `uid` up to the root it sits under, building the path.
/// `None` if the node has never been interned, or if the chain breaks partway —
/// a broken chain is a missing path, never a partial one.
fn walk_to_root(st: &super::State, uid: &NodeUid) -> Option<String> {
let mut ino = *st.by_uid.get(uid)?;
let mut parts = Vec::new();
while ino != ROOT_INO {
while !st.roots.contains(&ino) {
let entry = st.entries.get(&ino)?;
// An orphan is its own parent without being a root.
if entry.parent == ino {
return None;
}
parts.push(entry.node.name.clone());
ino = entry.parent;
}
+423
View File
@@ -0,0 +1,423 @@
//! One simulated daemon: a real FUSE mount on its own state directory, whose
//! Drive is a [`FakeClient`].
//!
//! Everything the daemon does runs as in production (drain workers, sync
//! engine, event feed, online probe, control socket) except what the host
//! changes (`mount::Host::Simulation`): a channel stops it instead of a signal,
//! and it does not index the developer's home.
use std::io;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::mpsc::SyncSender;
use std::thread::JoinHandle;
use std::time::{Duration, Instant};
use pdfs_core::cache::ContentCache;
use pdfs_core::config::AppConfig;
use pdfs_core::control::{self, PendingOpInfo, Request, Response};
use pdfs_core::db::Db;
use proton_drive_rs::ProtonDriveClient;
use proton_drive_rs::proton_sdk::config::ProtonClientConfiguration;
use proton_drive_rs::proton_sdk::session::{PasswordMode, ProtonApiSession, ResumeParameters};
use super::fake_drive::FakeClient;
use crate::mount::{Host, MountOptions, MountOutcome, mount_with};
/// How long a mount may take to come up.
const MOUNT_DEADLINE: Duration = Duration::from_secs(20);
/// How long a restart may wait for the last run to let go of its database.
const LAST_RUN_DEADLINE: Duration = Duration::from_secs(30);
pub(crate) struct Daemon {
pub(crate) client: FakeClient,
pub(crate) mountpoint: PathBuf,
dir: PathBuf,
socket: PathBuf,
stop: SyncSender<()>,
mount: Option<JoinHandle<io::Result<MountOutcome>>>,
rt: Option<tokio::runtime::Runtime>,
}
impl Daemon {
/// Mount a daemon on the state under `dir`, creating it on the first start.
/// A later start on the same `dir` is a restart: it finds the queue, the
/// cache and the database the last run left.
pub(crate) fn start(dir: &Path, client: FakeClient) -> io::Result<Self> {
log_to_test_output();
let mountpoint = dir.join("mnt");
let socket = dir.join("control.sock");
std::fs::create_dir_all(&mountpoint)?;
// A thread of the last run that is still in a request holds the
// database, and with it the single-writer lock, until it returns. A
// real restart waits for the old process to exit in the same way.
let mut db = Db::open(&dir.join("pdfs.db"));
let deadline = Instant::now() + LAST_RUN_DEADLINE;
while db
.as_ref()
.is_err_and(|error| error.to_string().contains("already using"))
&& Instant::now() < deadline
{
std::thread::sleep(Duration::from_millis(100));
db = Db::open(&dir.join("pdfs.db"));
}
let db = Arc::new(db.map_err(io::Error::other)?);
let cache = ContentCache::open(dir.join("cache"), dir.join("pins.json"), 0, db.clone())
.map_err(io::Error::other)?;
let rt = tokio::runtime::Builder::new_multi_thread()
.worker_threads(4)
.enable_all()
.build()?;
let (stop, stopped) = std::sync::mpsc::sync_channel(1);
let mount = {
let drive = Arc::new(client.clone());
let handle = rt.handle().clone();
let mountpoint = mountpoint.clone();
let socket = socket.clone();
std::thread::Builder::new()
.name("sim-mount".into())
.spawn(move || {
mount_with(
offline_client(),
drive,
handle,
&mountpoint,
cache,
&socket,
db,
MountOptions {
username: "sim".into(),
sweep_mode: AppConfig::default().resolved_conflict_sweep(),
upload_limit: 0,
download_limit: 0,
},
Host::Simulation(stopped),
)
})?
};
let deadline = Instant::now() + MOUNT_DEADLINE;
while !is_mounted(&mountpoint) {
if mount.is_finished() {
return Err(match mount.join() {
Ok(Err(error)) => error,
_ => io::Error::other("mount ended before it came up"),
});
}
if Instant::now() > deadline {
return Err(io::Error::other("mount did not come up"));
}
std::thread::sleep(Duration::from_millis(20));
}
Ok(Self {
client,
mountpoint,
dir: dir.to_path_buf(),
socket,
stop,
mount: Some(mount),
rt: Some(rt),
})
}
/// What the daemon still owes Drive, as `pdfs sync queue` shows it.
pub(crate) fn pending(&self) -> io::Result<Vec<PendingOpInfo>> {
match control::send(&self.socket, &Request::ListPendingOps).map_err(io::Error::other)? {
Response::PendingOps { items } => Ok(items),
other => Err(io::Error::other(format!("unexpected answer {other:?}"))),
}
}
/// Stop the daemon and start it again on the same state and link.
pub(crate) fn restart(mut self) -> io::Result<Self> {
self.shut_down()?;
Self::start(&self.dir.clone(), self.client.clone())
}
/// Stop the daemon the way `systemctl --user stop` does.
pub(crate) fn stop(mut self) -> io::Result<MountOutcome> {
self.shut_down()
}
fn shut_down(&mut self) -> io::Result<MountOutcome> {
let Some(mount) = self.mount.take() else {
return Ok(MountOutcome::Shutdown);
};
let _ = self.stop.try_send(());
let outcome = mount
.join()
.unwrap_or_else(|_| Err(io::Error::other("mount thread panicked")));
if let Some(rt) = self.rt.take() {
rt.shutdown_timeout(Duration::from_secs(5));
}
outcome
}
}
impl Drop for Daemon {
fn drop(&mut self) {
let _ = self.shut_down();
}
}
/// Whether a file system is mounted at `path`: it sits on another device than
/// its parent.
fn is_mounted(path: &Path) -> bool {
use std::os::unix::fs::MetadataExt;
let device = |path: &Path| std::fs::metadata(path).map(|meta| meta.dev()).ok();
let parent = path.parent().and_then(device);
device(path).is_some_and(|dev| Some(dev) != parent)
}
/// The daemons' logs, filtered by `RUST_LOG` and shown with `--nocapture`.
/// What they log at `INFO` and above is also kept for [`take_logged`],
/// whatever the filter.
fn log_to_test_output() {
use tracing_subscriber::prelude::*;
let _ = tracing_subscriber::registry()
.with(
tracing_subscriber::fmt::layer()
.with_test_writer()
.with_filter(tracing_subscriber::EnvFilter::from_default_env()),
)
.with(KeepLogged.with_filter(tracing_subscriber::filter::LevelFilter::INFO))
.try_init();
}
/// What the daemons logged at `INFO` and above since the last call, each as
/// its message followed by its fields: `message key=value ...`.
pub(crate) fn take_logged() -> Vec<String> {
std::mem::take(&mut *LOGGED.lock())
}
static LOGGED: parking_lot::Mutex<Vec<String>> = parking_lot::Mutex::new(Vec::new());
struct KeepLogged;
impl<S: tracing::Subscriber> tracing_subscriber::Layer<S> for KeepLogged {
fn on_event(&self, event: &tracing::Event<'_>, _: tracing_subscriber::layer::Context<'_, S>) {
let mut message = Message(String::new());
event.record(&mut message);
LOGGED.lock().push(message.0);
}
}
struct Message(String);
impl tracing::field::Visit for Message {
fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
if field.name() == "message" {
self.0.insert_str(0, &format!("{value:?}"));
} else {
self.0.push_str(&format!(" {}={value:?}", field.name()));
}
}
}
/// A client for what `DriveApi` does not cover (sharing, Photos, devices),
/// pointed at a closed local port so none of it leaves the machine.
fn offline_client() -> ProtonDriveClient {
let config = ProtonClientConfiguration::new("external-drive-pdfs-sim@0.0.0-dev")
.with_base_url("http://127.0.0.1:9");
let session = ProtonApiSession::resume(
config,
ResumeParameters {
session_id: "sim".to_owned().into(),
username: "sim".into(),
user_id: "sim".to_owned().into(),
access_token: String::new(),
refresh_token: String::new(),
scopes: Vec::new(),
is_waiting_for_second_factor_code: false,
password_mode: PasswordMode::Single,
},
)
.expect("an offline session needs no network");
ProtonDriveClient::new(&session, Vec::new())
}
/// A fresh directory for one simulation, under the system temp dir.
pub(crate) fn scratch(label: &str) -> PathBuf {
use std::sync::atomic::{AtomicU64, Ordering};
static N: AtomicU64 = AtomicU64::new(0);
let dir = std::env::temp_dir().join(format!(
"pdfs-sim-{label}-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed)
));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).expect("create a simulation directory");
dir
}
/// Wait up to `limit` for `done`, polling.
pub(crate) fn wait_until(limit: Duration, mut done: impl FnMut() -> bool) -> bool {
let deadline = Instant::now() + limit;
loop {
if done() {
return true;
}
if Instant::now() > deadline {
return false;
}
std::thread::sleep(Duration::from_millis(50));
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::sim::fake_drive::{Entry, FakeDrive, Faults};
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_daemon_on_the_fake_drive_syncs_both_ways() {
let drive = FakeDrive::new();
let dir = scratch("smoke");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
std::fs::write(daemon.mountpoint.join("hello.txt"), b"hi").unwrap();
assert!(
wait_until(Duration::from_secs(30), || {
drive.tree().get("hello.txt") == Some(&Entry::File(Arc::new(b"hi".to_vec())))
}),
"a local write landed on Drive: {:?}",
drive.tree()
);
drive.device().write("phone.txt", b"from the phone");
let remote = daemon.mountpoint.join("phone.txt");
assert!(
wait_until(Duration::from_secs(30), || {
std::fs::read(&remote).ok().as_deref() == Some(b"from the phone".as_slice())
}),
"a remote write reached the mount"
);
daemon.client.set_online(false);
std::fs::write(daemon.mountpoint.join("offline.txt"), b"queued").unwrap();
std::thread::sleep(Duration::from_secs(2));
assert!(!drive.tree().contains_key("offline.txt"));
daemon.client.set_online(true);
assert!(
wait_until(Duration::from_secs(60), || drive
.tree()
.contains_key("offline.txt")),
"a write made offline landed once the link came back"
);
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
/// Write `bytes` at `offset` into an existing file, without truncating it.
fn write_at(path: &Path, offset: u64, bytes: &[u8]) -> io::Result<()> {
use std::os::unix::fs::FileExt;
std::fs::OpenOptions::new()
.write(true)
.open(path)?
.write_all_at(bytes, offset)
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_file_closed_empty_takes_a_second_write_at_once() {
// The empty revision never counted as complete, so the next write
// open before it drained was refused with EIO (B123).
let drive = FakeDrive::new();
let dir = scratch("empty-rewrite");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
let path = daemon.mountpoint.join("empty.txt");
std::fs::write(&path, b"old").unwrap();
assert!(wait_until(Duration::from_secs(30), || {
drive.tree().contains_key("empty.txt")
}));
std::fs::write(&path, b"").unwrap();
write_at(&path, 0, b"filled").unwrap();
assert!(
wait_until(Duration::from_secs(30), || {
drive.tree().get("empty.txt") == Some(&Entry::File(Arc::new(b"filled".to_vec())))
}),
"the second write landed: {:?}",
drive.tree()
);
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn the_echo_of_our_own_rename_keeps_the_cached_bytes() {
// The echo read as someone else's change and evicted the bytes, so a
// partial write could not gap-fill, and the next write open on the
// file was refused with EIO (B123).
let drive = FakeDrive::new();
let dir = scratch("rename-echo");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
let before = daemon.mountpoint.join("a.txt");
std::fs::write(&before, b"0123456789").unwrap();
assert!(wait_until(Duration::from_secs(30), || {
drive.tree().contains_key("a.txt")
}));
assert_eq!(std::fs::read(&before).unwrap(), b"0123456789");
let after = daemon.mountpoint.join("b.txt");
std::fs::rename(&before, &after).unwrap();
// Long enough for the next event poll to bring the echo.
std::thread::sleep(crate::POLL_INTERVAL + Duration::from_secs(3));
write_at(&after, 3, b"x").unwrap();
write_at(&after, 5, b"y").unwrap();
assert!(
wait_until(Duration::from_secs(30), || {
drive.tree().get("b.txt") == Some(&Entry::File(Arc::new(b"012x4y6789".to_vec())))
}),
"both writes landed: {:?}",
drive.tree()
);
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_rename_across_folders_ignores_the_new_name_in_the_old_folder() {
// The rename went to Drive as a rename in the old folder and then a
// move, so a sibling there with the new name failed it with EIO
// (B122).
let drive = FakeDrive::new();
let dir = scratch("rename-across");
let daemon = Daemon::start(&dir, drive.client(1, Faults::lan())).unwrap();
let mount = &daemon.mountpoint;
std::fs::create_dir(mount.join("x")).unwrap();
std::fs::create_dir(mount.join("z")).unwrap();
std::fs::write(mount.join("x/f.txt"), b"moved").unwrap();
std::fs::write(mount.join("x/d"), b"sibling").unwrap();
std::fs::write(mount.join("z/d"), b"replaced").unwrap();
let file = |bytes: &[u8]| Some(Entry::File(Arc::new(bytes.to_vec())));
assert!(wait_until(Duration::from_secs(30), || {
let tree = drive.tree();
[
("x/f.txt", b"moved".as_slice()),
("x/d", b"sibling"),
("z/d", b"replaced"),
]
.iter()
.all(|(path, bytes)| tree.get(*path).cloned() == file(bytes))
}));
std::fs::rename(mount.join("x/f.txt"), mount.join("z/d")).unwrap();
assert_eq!(std::fs::read(mount.join("z/d")).unwrap(), b"moved");
let tree = drive.tree();
assert_eq!(tree.get("z/d").cloned(), file(b"moved"));
assert_eq!(tree.get("x/d").cloned(), file(b"sibling"));
assert!(!tree.contains_key("x/f.txt"));
assert!(matches!(daemon.stop(), Ok(MountOutcome::Shutdown)));
let _ = std::fs::remove_dir_all(&dir);
}
}
File diff suppressed because it is too large. Load diff
+9
View File
@@ -0,0 +1,9 @@
//! The simulation tests of `docs/MILESTONE-3.0.0.md` §8: a fake Drive with
//! the faults the real one has, and the tools to drive daemons against it.
pub(crate) mod daemon;
pub(crate) mod fake_drive;
pub(crate) mod model;
pub(crate) mod rng;
pub(crate) mod run;
pub(crate) mod watchdog;
+445
View File
@@ -0,0 +1,445 @@
//! The reference file system of the simulation (`docs/MILESTONE-3.0.0.md`
//! §8.2, property 4): what each syscall a simulated client makes should
//! answer, and the tree it should leave, on a POSIX file system.
//!
//! A [`Model`] covers one client's own folder; paths are relative to it. The
//! same type is used for the trees read back from a mount and from Drive.
use std::collections::BTreeMap;
use super::rng::Rng;
/// A tree by path: `None` is a folder, `Some` a file's content.
pub(crate) type Tree = BTreeMap<String, Option<Vec<u8>>>;
/// One file system call, or the few a program makes in a row.
#[derive(Clone, Debug)]
pub(crate) enum FsOp {
/// `open(O_CREAT | O_TRUNC)`, write `data`, close.
Create {
path: String,
data: Vec<u8>,
},
/// Open an existing file, write `data` at `offset` (past the end leaves
/// a hole), close.
Write {
path: String,
offset: u64,
data: Vec<u8>,
},
/// Open an existing file and `ftruncate` it to `len`.
Truncate {
path: String,
len: u64,
},
Mkdir {
path: String,
},
Rename {
from: String,
to: String,
},
Unlink {
path: String,
},
/// Open a file, unlink it, append `data`, read it all back, close.
UnlinkOpen {
path: String,
data: Vec<u8>,
},
Rmdir {
path: String,
},
}
impl FsOp {
/// The paths the op names, for the log.
pub(crate) fn describe(&self) -> String {
match self {
FsOp::Create { path, data } => format!("create {path} ({} bytes)", data.len()),
FsOp::Write { path, offset, data } => {
format!("write {path} at {offset} ({} bytes)", data.len())
}
FsOp::Truncate { path, len } => format!("truncate {path} to {len}"),
FsOp::Mkdir { path } => format!("mkdir {path}"),
FsOp::Rename { from, to } => format!("rename {from} -> {to}"),
FsOp::Unlink { path } => format!("unlink {path}"),
FsOp::UnlinkOpen { path, data } => {
format!("unlink {path} while open, then append {} bytes", data.len())
}
FsOp::Rmdir { path } => format!("rmdir {path}"),
}
}
}
/// The tree one client should have, and what its syscalls should answer.
#[derive(Clone, Debug, Default)]
pub(crate) struct Model {
tree: Tree,
}
/// Names are drawn from small pools, so ops collide: a create over an existing
/// file, a rename over another, a mkdir of a name that is taken.
const FILE_NAMES: [&str; 6] = ["a.txt", "b.txt", "c.bin", "d", "e.md", "f.txt"];
const DIR_NAMES: [&str; 3] = ["x", "y", "z"];
impl Model {
pub(crate) fn tree(&self) -> &Tree {
&self.tree
}
pub(crate) fn content(&self, path: &str) -> Option<&Vec<u8>> {
self.tree.get(path)?.as_ref()
}
/// What `op` should answer on a POSIX file system, applying it on success.
pub(crate) fn apply(&mut self, op: &FsOp) -> Result<(), i32> {
match op {
FsOp::Create { path, data } => {
self.check_parent(path)?;
match self.tree.get(path) {
Some(None) => Err(libc::EISDIR),
_ => {
self.tree.insert(path.clone(), Some(data.clone()));
Ok(())
}
}
}
FsOp::Write { path, offset, data } => {
let content = self.file_mut(path)?;
// Writing nothing changes nothing, even past the end.
if data.is_empty() {
return Ok(());
}
let end = *offset as usize + data.len();
if content.len() < end {
content.resize(end, 0);
}
content[*offset as usize..end].copy_from_slice(data);
Ok(())
}
FsOp::Truncate { path, len } => {
self.file_mut(path)?.resize(*len as usize, 0);
Ok(())
}
FsOp::Mkdir { path } => {
self.check_parent(path)?;
if self.tree.contains_key(path) {
return Err(libc::EEXIST);
}
self.tree.insert(path.clone(), None);
Ok(())
}
FsOp::Rename { from, to } => self.rename(from, to),
FsOp::Unlink { path } | FsOp::UnlinkOpen { path, .. } => {
self.check_parent(path)?;
match self.tree.get(path) {
None => Err(libc::ENOENT),
Some(None) => Err(libc::EISDIR),
Some(Some(_)) => {
self.tree.remove(path);
Ok(())
}
}
}
FsOp::Rmdir { path } => {
self.check_parent(path)?;
match self.tree.get(path) {
None => Err(libc::ENOENT),
Some(Some(_)) => Err(libc::ENOTDIR),
Some(None) if self.has_children(path) => Err(libc::ENOTEMPTY),
Some(None) => {
self.tree.remove(path);
Ok(())
}
}
}
}
}
fn rename(&mut self, from: &str, to: &str) -> Result<(), i32> {
self.check_parent(from)?;
let Some(moving) = self.tree.get(from).cloned() else {
return Err(libc::ENOENT);
};
self.check_parent(to)?;
if from == to {
return Ok(());
}
if is_below(to, from) {
return Err(libc::EINVAL);
}
match (&moving, self.tree.get(to)) {
(None, Some(Some(_))) => return Err(libc::ENOTDIR),
(Some(_), Some(None)) => return Err(libc::EISDIR),
(None, Some(None)) if self.has_children(to) => return Err(libc::ENOTEMPTY),
_ => {}
}
let below: Vec<(String, Option<Vec<u8>>)> = self
.tree
.iter()
.filter(|(path, _)| is_below(path, from))
.map(|(path, entry)| (path.clone(), entry.clone()))
.collect();
self.tree.remove(from);
self.tree.insert(to.to_owned(), moving);
for (path, entry) in below {
self.tree.remove(&path);
self.tree
.insert(format!("{to}{}", &path[from.len()..]), entry);
}
Ok(())
}
fn file_mut(&mut self, path: &str) -> Result<&mut Vec<u8>, i32> {
self.check_parent(path)?;
match self.tree.get_mut(path) {
None => Err(libc::ENOENT),
Some(None) => Err(libc::EISDIR),
Some(Some(content)) => Ok(content),
}
}
fn check_parent(&self, path: &str) -> Result<(), i32> {
let mut at = String::new();
let mut names = path.split('/').peekable();
while let Some(name) = names.next() {
if names.peek().is_none() {
break;
}
if !at.is_empty() {
at.push('/');
}
at.push_str(name);
match self.tree.get(&at) {
None => return Err(libc::ENOENT),
Some(Some(_)) => return Err(libc::ENOTDIR),
Some(None) => {}
}
}
Ok(())
}
fn has_children(&self, path: &str) -> bool {
self.tree.keys().any(|other| is_below(other, path))
}
pub(crate) fn files(&self) -> Vec<&String> {
self.tree
.iter()
.filter(|(_, entry)| entry.is_some())
.map(|(path, _)| path)
.collect()
}
/// Folders, the client's own root (`""`) first.
fn dirs(&self) -> Vec<String> {
std::iter::once(String::new())
.chain(
self.tree
.iter()
.filter(|(_, entry)| entry.is_none())
.map(|(path, _)| path.clone()),
)
.collect()
}
/// A random op, mostly one that should succeed.
pub(crate) fn next_op(&self, rng: &mut Rng) -> FsOp {
let dirs = self.dirs();
let files = self.files();
let in_dir = |rng: &mut Rng, names: &[&str]| {
let parent = rng.pick(&dirs).expect("the root is always there");
join(parent, rng.pick(names).expect("names"))
};
let a_file = |rng: &mut Rng| rng.pick(&files).map(|path| (*path).clone());
let roll = rng.below(100);
let op = match roll {
0..30 => Some(FsOp::Create {
path: in_dir(rng, &FILE_NAMES),
data: data(rng),
}),
30..45 => a_file(rng).map(|path| {
let len = self.content(&path).map_or(0, Vec::len) as u64;
FsOp::Write {
offset: rng.below(len + 8),
path,
data: data(rng),
}
}),
45..50 => a_file(rng).map(|path| FsOp::Truncate {
len: rng.below(32),
path,
}),
50..60 => Some(FsOp::Mkdir {
path: in_dir(rng, &DIR_NAMES),
}),
// A rename in place, over another file as often as not.
60..72 => a_file(rng).map(|from| FsOp::Rename {
to: join(parent_of(&from), rng.pick(&FILE_NAMES).expect("names")),
from,
}),
// A move to another folder, renamed or not.
72..82 => a_file(rng).map(|from| {
let name = if rng.chance(0.5) {
name_of(&from)
} else {
rng.pick(&FILE_NAMES).expect("names")
};
let parent = rng.pick(&dirs).expect("the root is always there");
FsOp::Rename {
to: join(parent, name),
from,
}
}),
// A folder renamed or moved, with what is in it.
82..86 => {
let from = rng.pick(&dirs[1..]).cloned();
from.map(|from| FsOp::Rename {
to: in_dir(rng, &DIR_NAMES),
from,
})
}
86..94 => a_file(rng).map(|path| FsOp::Unlink { path }),
94..97 => a_file(rng).map(|path| FsOp::UnlinkOpen {
path,
data: data(rng),
}),
_ => rng
.pick(&dirs[1..])
.cloned()
.map(|path| FsOp::Rmdir { path }),
};
op.unwrap_or_else(|| FsOp::Create {
path: in_dir(rng, &FILE_NAMES),
data: data(rng),
})
}
}
/// Content for a write: short and recognisable, now and then empty.
pub(crate) fn data(rng: &mut Rng) -> Vec<u8> {
let len = if rng.chance(0.1) {
0
} else {
rng.between(1, 48)
};
let tag = rng.next_u64();
(0..len).map(|i| b'a' + ((tag + i) % 26) as u8).collect()
}
pub(crate) fn join(parent: &str, name: &str) -> String {
if parent.is_empty() {
name.to_owned()
} else {
format!("{parent}/{name}")
}
}
pub(crate) fn parent_of(path: &str) -> &str {
path.rsplit_once('/').map_or("", |(parent, _)| parent)
}
pub(crate) fn name_of(path: &str) -> &str {
path.rsplit_once('/').map_or(path, |(_, name)| name)
}
fn is_below(path: &str, ancestor: &str) -> bool {
path.len() > ancestor.len() + 1
&& path.starts_with(ancestor)
&& path.as_bytes()[ancestor.len()] == b'/'
}
#[cfg(test)]
mod tests {
use super::*;
fn create(path: &str, data: &[u8]) -> FsOp {
FsOp::Create {
path: path.into(),
data: data.to_vec(),
}
}
#[test]
fn a_rename_over_a_file_replaces_it_and_a_folder_moves_with_its_contents() {
let mut model = Model::default();
model.apply(&FsOp::Mkdir { path: "x".into() }).unwrap();
model.apply(&create("x/a", b"1")).unwrap();
model.apply(&create("b", b"2")).unwrap();
model
.apply(&FsOp::Rename {
from: "b".into(),
to: "x/a".into(),
})
.unwrap();
assert_eq!(model.content("x/a"), Some(&b"2".to_vec()));
model
.apply(&FsOp::Rename {
from: "x".into(),
to: "y".into(),
})
.unwrap();
assert_eq!(model.tree().keys().collect::<Vec<_>>(), vec!["y", "y/a"]);
}
#[test]
fn calls_fail_the_way_posix_says() {
let mut model = Model::default();
model.apply(&FsOp::Mkdir { path: "x".into() }).unwrap();
model.apply(&create("x/a", b"1")).unwrap();
assert_eq!(
model.apply(&FsOp::Mkdir { path: "x".into() }),
Err(libc::EEXIST)
);
assert_eq!(model.apply(&create("x", b"")), Err(libc::EISDIR));
assert_eq!(model.apply(&create("x/a/b", b"")), Err(libc::ENOTDIR));
assert_eq!(model.apply(&create("q/a", b"")), Err(libc::ENOENT));
assert_eq!(
model.apply(&FsOp::Rmdir { path: "x".into() }),
Err(libc::ENOTEMPTY)
);
assert_eq!(
model.apply(&FsOp::Unlink { path: "x".into() }),
Err(libc::EISDIR)
);
let into_itself = FsOp::Rename {
from: "x".into(),
to: "x/y".into(),
};
assert_eq!(model.apply(&into_itself), Err(libc::EINVAL));
}
#[test]
fn a_write_past_the_end_leaves_a_hole() {
let mut model = Model::default();
model.apply(&create("a", b"ab")).unwrap();
let write = FsOp::Write {
path: "a".into(),
offset: 4,
data: b"z".to_vec(),
};
model.apply(&write).unwrap();
assert_eq!(model.content("a"), Some(&b"ab\0\0z".to_vec()));
let nothing = FsOp::Write {
path: "a".into(),
offset: 9,
data: Vec::new(),
};
model.apply(&nothing).unwrap();
assert_eq!(model.content("a"), Some(&b"ab\0\0z".to_vec()));
}
#[test]
fn generated_ops_mostly_succeed() {
let mut rng = Rng::new(9);
let mut model = Model::default();
let ok = (0..500)
.filter(|_| {
let op = model.next_op(&mut rng);
model.apply(&op).is_ok()
})
.count();
assert!(ok > 350, "{ok} of 500 succeeded");
}
}
+79
View File
@@ -0,0 +1,79 @@
//! A small seeded random number generator, so a failing simulation is a seed
//! that can be replayed. SplitMix64: fast, good enough for choosing operations
//! and faults, and no dependency.
#[derive(Clone, Debug)]
pub(crate) struct Rng(u64);
impl Rng {
pub(crate) fn new(seed: u64) -> Self {
Self(seed)
}
pub(crate) fn next_u64(&mut self) -> u64 {
self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
let mut z = self.0;
z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
z ^ (z >> 31)
}
/// A value in `0..n`; `0` when `n` is `0`.
pub(crate) fn below(&mut self, n: u64) -> u64 {
if n == 0 { 0 } else { self.next_u64() % n }
}
/// A value in `lo..=hi`.
pub(crate) fn between(&mut self, lo: u64, hi: u64) -> u64 {
lo + self.below(hi.saturating_sub(lo) + 1)
}
/// True with probability `p`.
pub(crate) fn chance(&mut self, p: f64) -> bool {
p > 0.0 && (self.next_u64() >> 11) as f64 / (1u64 << 53) as f64 <= p
}
pub(crate) fn pick<'a, T>(&mut self, items: &'a [T]) -> Option<&'a T> {
items.get(self.below(items.len() as u64) as usize)
}
/// A child generator, so one consumer's draws do not shift another's.
pub(crate) fn fork(&mut self) -> Self {
Self::new(self.next_u64())
}
}
#[cfg(test)]
mod tests {
use super::Rng;
#[test]
fn a_seed_replays_the_same_sequence() {
let (mut a, mut b) = (Rng::new(7), Rng::new(7));
for _ in 0..100 {
assert_eq!(a.next_u64(), b.next_u64());
}
assert_ne!(Rng::new(7).next_u64(), Rng::new(8).next_u64());
}
#[test]
fn bounded_draws_stay_in_range() {
let mut rng = Rng::new(1);
for _ in 0..1000 {
assert!(rng.below(5) < 5);
assert!((3..=9).contains(&rng.between(3, 9)));
}
assert_eq!(rng.below(0), 0);
assert!(!rng.chance(0.0));
assert!(rng.chance(1.0));
assert!([1, 2, 3].contains(rng.pick(&[1, 2, 3]).unwrap()));
assert_eq!(rng.pick::<u8>(&[]), None);
}
#[test]
fn a_fork_draws_apart_from_its_parent() {
let mut parent = Rng::new(3);
let mut child = parent.fork();
assert_ne!(parent.next_u64(), child.next_u64());
}
}
File diff suppressed because it is too large. Load diff
+322
View File
@@ -0,0 +1,322 @@
//! Turns a call through a mount that never returns into a failed seed.
//!
//! A deadlock in a daemon leaves the run's own thread blocked in the kernel,
//! where no budget check is reached, and a CI job then waits for its timeout.
//! The watchdog runs beside the run. When what it was armed for outlives its
//! limit, it aborts the FUSE connections of the run's mounts, so the blocked
//! call fails and the run reports the hang, and writes where every thread was
//! to `stacks.txt`.
//!
//! The stacks are taken in the process: no tracer can stop a thread that
//! waits on a FUSE request, so `gdb` and `eu-stack` wait as long as the hang.
//! Each thread is sent [`stack_signal`] and captures its own backtrace in the
//! handler. A thread in the kernel takes the signal only once it leaves, so
//! the mounts are aborted after the others have answered, and its backtrace
//! is the call it was blocked in. Its kernel wait channel is recorded first.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread::JoinHandle;
use std::time::{Duration, Instant};
use parking_lot::Mutex;
use super::daemon::wait_until;
/// How long the threads that are not blocked in the kernel have to answer,
/// and the blocked ones once the mounts are aborted.
const ANSWER: Duration = Duration::from_secs(5);
pub(crate) struct Watchdog {
shared: Arc<Shared>,
thread: Option<JoinHandle<()>>,
}
struct Shared {
/// The run's directory: its mounts are below it, and the stacks go in it.
dir: PathBuf,
armed: Mutex<Option<Armed>>,
/// What hung, once something has.
fired: Mutex<Option<Hung>>,
stop: AtomicBool,
}
struct Armed {
what: String,
limit: Duration,
since: Instant,
}
/// What hung, and where every thread was then.
#[derive(Clone, Debug, PartialEq)]
pub(crate) struct Hung {
pub(crate) what: String,
pub(crate) stacks: String,
}
/// Disarms the watchdog when dropped.
pub(crate) struct Watching(Arc<Shared>);
impl Watchdog {
pub(crate) fn new(dir: &Path) -> Self {
let shared = Arc::new(Shared {
dir: dir.to_path_buf(),
armed: Mutex::new(None),
fired: Mutex::new(None),
stop: AtomicBool::new(false),
});
let thread = {
let shared = shared.clone();
std::thread::Builder::new()
.name("sim-watchdog".into())
.spawn(move || watch(&shared))
.expect("spawn the simulation watchdog")
};
Self {
shared,
thread: Some(thread),
}
}
/// Watch `what` until the guard is dropped, and abort the mounts if it is
/// still going after `limit`.
pub(crate) fn arm(&self, limit: Duration, what: impl Into<String>) -> Watching {
*self.shared.armed.lock() = Some(Armed {
what: what.into(),
limit,
since: Instant::now(),
});
Watching(self.shared.clone())
}
/// What hung, if something did. Call [`Watchdog::stop`] first, or the
/// stacks may still be on their way.
pub(crate) fn fired(&self) -> Option<Hung> {
self.shared.fired.lock().clone()
}
/// Stop watching, once the stacks of a hang that was caught are taken.
pub(crate) fn stop(&mut self) {
self.shared.stop.store(true, Ordering::Relaxed);
if let Some(thread) = self.thread.take() {
let _ = thread.join();
}
}
}
impl Drop for Watching {
fn drop(&mut self) {
*self.0.armed.lock() = None;
}
}
impl Drop for Watchdog {
fn drop(&mut self) {
self.stop();
}
}
fn watch(shared: &Shared) {
while !shared.stop.load(Ordering::Relaxed) {
std::thread::sleep(Duration::from_millis(200));
let hung = shared
.armed
.lock()
.as_ref()
.filter(|armed| armed.since.elapsed() > armed.limit)
.map(|armed| format!("{} has not returned after {:?}", armed.what, armed.limit));
let Some(hung) = hung else { continue };
*shared.armed.lock() = None;
let path = shared.dir.join("stacks.txt");
// Before anything is let go: the run reports the hang, not whatever
// the aborted call answers.
*shared.fired.lock() = Some(Hung {
what: format!("{hung}; the stacks are in {}", path.display()),
stacks: String::new(),
});
let stacks = take_stacks(|| abort_mounts_below(&shared.dir));
let what = match std::fs::write(&path, &stacks) {
Ok(()) => format!("{hung}; the stacks are in {}", path.display()),
Err(error) => format!("{hung}; the stacks could not be kept: {error}"),
};
*shared.fired.lock() = Some(Hung { what, stacks });
}
}
/// One thread of this process as the kernel sees it.
struct Task {
tid: i32,
name: String,
/// `D` while it waits in the kernel uninterruptibly.
state: String,
wchan: String,
}
/// Where every other thread of this process is, with `release` called once
/// those not blocked in the kernel have answered.
fn take_stacks(release: impl FnOnce()) -> String {
let own = gettid();
let tasks = tasks();
STACKS.lock().clear();
install_stack_handler();
for task in &tasks {
if task.tid != own {
// SAFETY: tgkill only sends a signal, whose handler is installed.
unsafe {
libc::syscall(
libc::SYS_tgkill,
std::process::id() as i32,
task.tid,
stack_signal(),
)
};
}
}
let answered = |blocked: bool| {
let stacks = STACKS.lock();
tasks
.iter()
.filter(|task| task.tid != own && (blocked || task.state != "D"))
.all(|task| stacks.contains_key(&task.tid))
};
wait_until(ANSWER, || answered(false));
release();
wait_until(ANSWER, || answered(true));
let stacks = std::mem::take(&mut *STACKS.lock());
let mut out = String::new();
for task in tasks.iter().filter(|task| task.tid != own) {
out.push_str(&format!(
"thread {} {:?}, state {}, waiting in {}\n",
task.tid, task.name, task.state, task.wchan
));
match stacks.get(&task.tid) {
Some(stack) => out.push_str(&format!("{stack}\n")),
None => out.push_str(" (no answer)\n\n"),
}
}
out
}
fn tasks() -> Vec<Task> {
let Ok(entries) = std::fs::read_dir("/proc/self/task") else {
return Vec::new();
};
let read = |dir: &Path, file: &str| {
std::fs::read_to_string(dir.join(file))
.unwrap_or_default()
.trim()
.to_owned()
};
entries
.flatten()
.filter_map(|entry| {
let tid = entry.file_name().to_str()?.parse().ok()?;
let dir = entry.path();
// The state follows the name in parentheses, which may hold spaces.
let stat = read(&dir, "stat");
let state = stat.rsplit_once(") ")?.1.split(' ').next()?.to_owned();
Some(Task {
tid,
name: read(&dir, "comm"),
state,
wchan: read(&dir, "wchan"),
})
})
.collect()
}
/// The backtraces the threads captured, by thread id.
static STACKS: Mutex<BTreeMap<i32, std::backtrace::Backtrace>> = Mutex::new(BTreeMap::new());
/// The signal that asks a thread for its stack: one of the real-time signals
/// that nothing in the daemon uses.
fn stack_signal() -> i32 {
libc::SIGRTMIN() + 7
}
fn install_stack_handler() {
static INSTALLED: std::sync::Once = std::sync::Once::new();
extern "C" fn handler(_: i32) {
// Not async-signal-safe: the capture allocates. The process is being
// taken down for a hang anyway, and a thread interrupted inside the
// allocator only fails to answer.
let stack = std::backtrace::Backtrace::force_capture();
STACKS.lock().insert(gettid(), stack);
}
INSTALLED.call_once(|| {
// SAFETY: a zeroed sigaction with a handler and SA_RESTART is valid,
// and the handler only touches `STACKS`.
unsafe {
let mut action: libc::sigaction = std::mem::zeroed();
action.sa_sigaction = handler as extern "C" fn(i32) as usize;
action.sa_flags = libc::SA_RESTART;
libc::sigemptyset(&mut action.sa_mask);
libc::sigaction(stack_signal(), &action, std::ptr::null_mut());
}
});
}
fn gettid() -> i32 {
// SAFETY: gettid has no arguments and cannot fail.
unsafe { libc::gettid() }
}
/// Abort the FUSE connection of every mount below `dir`. Its connection is
/// named by the mount's device minor, and the mount's owner may abort it.
fn abort_mounts_below(dir: &Path) {
let Ok(mountinfo) = std::fs::read_to_string("/proc/self/mountinfo") else {
return;
};
for line in mountinfo.lines() {
let fields: Vec<&str> = line.split(' ').collect();
let (Some(device), Some(mountpoint)) = (fields.get(2), fields.get(4)) else {
continue;
};
if !Path::new(mountpoint).starts_with(dir) {
continue;
}
if let Some((_, minor)) = device.split_once(':') {
let _ = std::fs::write(format!("/sys/fs/fuse/connections/{minor}/abort"), "1");
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::sim::daemon::scratch;
#[test]
#[ignore = "signals every thread of the test process: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_call_past_its_limit_is_reported_with_the_stacks() {
let dir = scratch("watchdog");
let mut watchdog = Watchdog::new(&dir);
{
let _watching = watchdog.arm(Duration::from_millis(100), "a nap");
std::thread::sleep(Duration::from_secs(1));
}
watchdog.stop();
let hung = watchdog.fired().expect("the watchdog fired");
assert!(
hung.what.starts_with("a nap has not returned after"),
"{hung:?}"
);
assert!(hung.stacks.contains("a_call_past_its_limit"), "{hung:?}");
let kept = std::fs::read_to_string(dir.join("stacks.txt")).unwrap_or_default();
assert_eq!(kept, hung.stacks);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn a_call_within_its_limit_is_not() {
let dir = scratch("watchdog-quiet");
let watchdog = Watchdog::new(&dir);
drop(watchdog.arm(Duration::from_millis(100), "a quick call"));
std::thread::sleep(Duration::from_millis(500));
assert_eq!(watchdog.fired(), None);
drop(watchdog);
let _ = std::fs::remove_dir_all(&dir);
}
}
+439 -38
View File
@@ -128,6 +128,61 @@ pub(crate) struct WriteHandle {
pub(crate) open_count: usize,
}
impl WriteHandle {
/// A handle on a file `create` just made: an empty base, so everything
/// written through it is authored.
///
/// The base is `node` as it actually exists — the empty file the server just
/// minted, or a queued create's placeholder — never the local clock or a
/// blank. `queue_revision` turns it into `StagedWrite::based_on` and the
/// drain compares the remote against that. A `now_secs()` mtime differs from
/// the server's stamp by the create round-trip. A missing revision id drops
/// the check to `(mtime, size)`, and moving or renaming the file advances
/// its mtime on the server. Either way the first write to a brand-new file
/// lands as a conflict copy of its own empty create (`docs/BUGS.md` B113).
pub(crate) fn created(ino: u64, node: &Node, file: File, path: PathBuf) -> Self {
Self {
ino,
uid: node.uid.clone(),
file: Arc::new(file),
path,
written: Intervals::default(),
len: 0,
base_size: 0,
base_mtime: node.modification_time,
base_revision_id: crate::node_revision_id(node),
dirty: false,
open_count: 0,
}
}
/// Write over `node` from now on: a revision this daemon has just put on the
/// server under the handle, by sealing an upload or by landing the queued
/// create the handle was opened on. The base the handle opened over is gone,
/// and keeping it would make the drain read our own revision as another
/// device's edit.
pub(crate) fn rebase_onto(&mut self, node: &Node) {
self.base_mtime = node.modification_time;
self.base_size = crate::node_size(node);
self.base_revision_id = crate::node_revision_id(node);
}
/// Account for `[offset, end)` having been written through this handle.
///
/// A write past the end leaves a hole that reads as zeros, exactly like a
/// grow through `setattr`, so the hole is authored too. Left unauthored, a
/// hole below the base size was refilled from the base at commit: a file
/// shrunk and then written past its new end got its old bytes back.
pub(crate) fn record_write(&mut self, offset: u64, end: u64) {
if offset > self.len {
self.written.add(self.len, offset);
}
self.written.add(offset, end);
self.len = self.len.max(end);
self.dirty = true;
}
}
/// A released write whose upload has not happened yet (offline.md Phase 3).
///
/// The bytes live in the content cache's staging dir and the intent lives in the
@@ -146,8 +201,23 @@ pub(crate) struct PendingRevision {
/// time so this module stays free of `fuser`.
pub(crate) type DirListing = Vec<(u64, bool, String)>;
/// Where [`State::next_ino`] starts: far above any local id a database hands
/// out, so a fallback inode never names a node that has a row.
pub(crate) const FALLBACK_INO: u64 = 1 << 48;
/// The inode of the node with local id `lid` ([`Db::lids_for`]). One above the
/// lid, so no node takes [`crate::ROOT_INO`], which each session uses for its
/// own root. The same in every mount and after a restart.
pub(crate) fn lid_ino(lid: i64) -> u64 {
lid.unsigned_abs() + crate::ROOT_INO
}
/// Mutable inode bookkeeping, guarded by a mutex because fuser drives the
/// `Filesystem` trait through `&self`.
///
/// One tree for every mount of the daemon. Each FUSE session sees it from its
/// own root ([`State::roots`]) and calls that root inode 1; `ProtonFs`
/// translates at the kernel boundary.
pub(crate) struct State {
/// inode -> node metadata.
pub(crate) entries: HashMap<u64, Entry>,
@@ -156,6 +226,12 @@ pub(crate) struct State {
/// Cached directory listings: parent inode -> child inodes. Presence of a
/// key means the directory has been enumerated.
pub(crate) children: HashMap<u64, Vec<u64>>,
/// The inodes some session has been rooted at ([`State::install_root`]).
/// A root is its own parent. Kept after its session unmounts: the node is
/// still one of ours.
pub(crate) roots: HashSet<u64>,
/// The next inode for a node the database could not give a local id, from
/// a range lids never reach ([`FALLBACK_INO`]).
pub(crate) next_ino: u64,
/// Shared write state keyed by inode. Concurrent writers share the scratch file.
pub(crate) active_writes: HashMap<u64, WriteHandle>,
@@ -177,7 +253,7 @@ pub(crate) struct State {
/// intern/root refresh. Core drains this set and notifies the matching
/// kernel only after releasing the State lock.
pub(crate) access_changes: HashSet<u64>,
/// Persisted shared-root authority, loaded once per inode space and updated
/// Persisted shared-root authority, loaded once and updated
/// on every explicit role/tombstone change. Descendant inheritance reads
/// this map instead of issuing one SQLite query per interned inode.
pub(crate) share_access: HashMap<NodeUid, Access>,
@@ -245,15 +321,6 @@ pub(crate) struct StateGuard<'a> {
db: &'a Arc<Db>,
}
/// Take a mount's inode lock through a [`StateGuard`], for the loops that walk
/// every live mount rather than this `Core`'s own.
pub(crate) fn lock_state<'a>(
state: &'a parking_lot::Mutex<State>,
db: &'a Arc<Db>,
) -> StateGuard<'a> {
StateGuard::new(state.lock(), db)
}
impl<'a> StateGuard<'a> {
pub(crate) fn new(guard: parking_lot::MutexGuard<'a, State>, db: &'a Arc<Db>) -> Self {
Self {
@@ -294,9 +361,6 @@ impl Drop for StateGuard<'_> {
impl State {
/// A fresh, empty inode space over `db`.
///
/// `next_ino` differs by caller: a real mount hands out [`crate::ROOT_INO`]
/// to its root and starts allocating above it, while a bare state under test
/// has no root and starts at 1.
/// Apply the write-throughs this state owes SQLite, here and now.
///
/// Production code never calls this: [`StateGuard`] releases the inode lock
@@ -323,12 +387,15 @@ impl State {
self.db.clone()
}
pub(crate) fn new(db: Arc<Db>, share_access: HashMap<NodeUid, Access>, next_ino: u64) -> Self {
/// A fresh, empty tree over `db`. A node's inode comes from its local id
/// ([`lid_ino`]).
pub(crate) fn new(db: Arc<Db>, share_access: HashMap<NodeUid, Access>) -> Self {
Self {
entries: HashMap::new(),
by_uid: HashMap::new(),
children: HashMap::new(),
next_ino,
roots: HashSet::new(),
next_ino: FALLBACK_INO,
active_writes: HashMap::new(),
handles: HashMap::new(),
next_fh: 1,
@@ -339,18 +406,55 @@ impl State {
outbox: Vec::new(),
}
}
/// Whether `uid` names an owned node currently reachable from this mount's
/// root. Resident entries can outlive their dentries for open-handle and
/// access-revocation semantics, so `by_uid` membership alone is not proof
/// that a control-socket operation may address the node.
pub(crate) fn owns_visible_uid(&self, uid: &NodeUid) -> bool {
let Some(root) = self.entries.get(&crate::ROOT_INO) else {
return false;
};
if uid.volume_id != root.uid.volume_id {
return false;
/// Make `root` the root of a session and return its inode, which comes
/// from its local id like any other node's.
///
/// The root is always owned. It must never be classified as a shared root,
/// even when the API returns membership data (which happens when the owner
/// shares the folder with someone else), so a stale share row a downgrade
/// may have left for it is dropped. A node already in the tree keeps its
/// inode and its lookups, and stops naming a parent: a session's `..` at
/// its root is the root.
pub(crate) fn install_root(&mut self, root: Node) -> u64 {
if let Err(e) = self.db.upsert_node(&root) {
warn!(uid = %root.uid, error = %e, "db upsert root failed");
}
self.share_access.remove(&root.uid);
let _ = self.db.delete_share_access(&root.uid);
let ino = match self.by_uid.get(&root.uid) {
Some(&ino) => ino,
None => self.new_inos(&[&root])[&root.uid],
};
if let Some(old) = self.entries.get(&ino).map(|e| e.parent)
&& old != ino
&& let Some(kids) = self.children.get_mut(&old)
{
kids.retain(|&k| k != ino);
}
self.by_uid.insert(root.uid.clone(), ino);
let lookup_count = self.entries.get(&ino).map_or(1, |e| e.lookup_count.max(1));
self.entries.insert(
ino,
Entry {
uid: root.uid.clone(),
parent: ino,
node: root,
access: Access::Owner,
lookup_count,
open_count: 0,
unlinked: false,
},
);
self.roots.insert(ino);
ino
}
/// Whether `uid` names an owned node currently reachable from one of
/// `roots`, the roots of the sessions that are mounted. Resident entries
/// can outlive their dentries for open-handle and access-revocation
/// semantics, so `by_uid` membership alone is not proof that a
/// control-socket operation may address the node.
pub(crate) fn owns_visible_uid(&self, uid: &NodeUid, roots: &[u64]) -> bool {
let Some(&target) = self.by_uid.get(uid) else {
return false;
};
@@ -363,7 +467,9 @@ impl State {
let Some(entry) = self.entries.get(&current) else {
return false;
};
if entry.uid.volume_id != root.uid.volume_id
// The node's root scopes it to its own volume, excluding foreign
// shared-with-me residents from uid-addressed sharing.
if entry.uid.volume_id != uid.volume_id
|| entry.unlinked
|| entry.node.trashed
|| entry.uid != entry.node.uid
@@ -371,7 +477,7 @@ impl State {
{
return false;
}
if current == crate::ROOT_INO {
if roots.contains(&current) {
return true;
}
@@ -413,15 +519,17 @@ impl State {
self.access_changes.drain().collect()
}
/// Whether `uid` lives on the volume this mount is rooted in.
/// Whether `uid` lives on a volume some mount is rooted in.
///
/// Used to decide how a node with no resolvable parent is classified:
/// our own volume fails open (it is owned content), a foreign volume
/// fails closed (it can only be there because of a share).
pub(crate) fn is_own_volume(&self, uid: &NodeUid) -> bool {
self.entries
.get(&crate::ROOT_INO)
.is_some_and(|root| root.uid.volume_id == uid.volume_id)
self.roots.iter().any(|ino| {
self.entries
.get(ino)
.is_some_and(|root| root.uid.volume_id == uid.volume_id)
})
}
fn stored_share_access(&self, uid: &NodeUid) -> Option<Access> {
@@ -770,11 +878,54 @@ impl State {
}
}
pub(crate) fn intern(&mut self, parent: u64, node: Node) -> u64 {
pub(crate) fn intern(&mut self, parent: u64, mut node: Node) -> u64 {
self.keep_revision_mtime(&mut node);
self.outbox.push(DbWrite::Upsert(vec![node.clone()]));
self.intern_mem(parent, node)
}
/// Keep what is already known of a file whose content has not changed.
///
/// Drive stamps a link's `ModifyTime` on a rename or a move as well, while
/// POSIX leaves a file's mtime alone there. The content cache is keyed by
/// it, too: taking the new stamp made the cached bytes unreachable, so the
/// next partial write could not fill its gaps (docs/BUGS.md B123). The
/// active revision id says whether the content is still the same, and so
/// whether the size and digest a listing left out are still the ones known.
fn keep_revision_mtime(&self, node: &mut Node) {
let Some(entry) = self
.by_uid
.get(&node.uid)
.and_then(|ino| self.entries.get(ino))
else {
return;
};
if let (
NodeKind::File {
active_revision_id: Some(now),
claimed_size,
content_sha1,
..
},
NodeKind::File {
active_revision_id: Some(was),
claimed_size: size_was,
content_sha1: sha1_was,
..
},
) = (&mut node.kind, &entry.node.kind)
&& now == was
{
node.modification_time = entry.node.modification_time;
if claimed_size.is_none() {
*claimed_size = *size_was;
}
if content_sha1.is_none() {
content_sha1.clone_from(sha1_was);
}
}
}
/// Allocate (or reuse) a stable inode for a node, updating the hot-cache maps
/// only. Every caller owes the DB a write-through — see the callers below;
/// the split exists so a batch can pay for one transaction instead of `n`.
@@ -784,11 +935,24 @@ impl State {
}
fn intern_mem_with_access(&mut self, parent: u64, node: Node, access: Access) -> u64 {
self.intern_mem_at(parent, node, access, None)
}
/// [`State::intern_mem_with_access`], with the inode a new entry takes when
/// the caller already has it ([`State::new_inos`]).
fn intern_mem_at(
&mut self,
parent: u64,
mut node: Node,
access: Access,
ino: Option<u64>,
) -> u64 {
if let Some(&ino) = self.by_uid.get(&node.uid) {
let changed = self
.entries
.get(&ino)
.is_some_and(|entry| entry.access != access);
self.keep_open_write_size(ino, &mut node);
if let Some(e) = self.entries.get_mut(&ino) {
e.node = node;
e.parent = parent;
@@ -801,8 +965,13 @@ impl State {
}
return ino;
}
let ino = self.next_ino;
self.next_ino += 1;
let ino = match ino {
Some(ino) => ino,
None => self
.new_inos(&[&node])
.remove(&node.uid)
.unwrap_or_default(),
};
self.by_uid.insert(node.uid.clone(), ino);
self.entries.insert(
ino,
@@ -819,6 +988,41 @@ impl State {
ino
}
/// The inodes for nodes that have no entry yet, from their local ids, in
/// one transaction. Should the database fail, they come from
/// [`State::next_ino`] instead, so the mount keeps working, with inodes
/// that last only this run. So do the inodes of virtual folders, which are
/// never stored.
fn new_inos(&mut self, nodes: &[&Node]) -> HashMap<NodeUid, u64> {
let (virtual_nodes, stored): (Vec<&Node>, Vec<&Node>) = nodes
.iter()
.partition(|node| crate::r#virtual::is_virtual_uid(&node.uid));
let mut inos = match self.db.lids_for(&stored) {
Ok(lids) => stored
.iter()
.zip(lids)
.map(|(node, lid)| (node.uid.clone(), lid_ino(lid)))
.collect(),
Err(error) => {
warn!(%error, "no local ids for new inodes; numbering them for this run");
self.run_inos(&stored)
}
};
inos.extend(self.run_inos(&virtual_nodes));
inos
}
fn run_inos(&mut self, nodes: &[&Node]) -> HashMap<NodeUid, u64> {
nodes
.iter()
.map(|node| {
let ino = self.next_ino;
self.next_ino += 1;
(node.uid.clone(), ino)
})
.collect()
}
/// Decrement lookup count for an inode and prune if lookup_count == 0 && open_count == 0 && unlinked.
pub(crate) fn forget_lookup(&mut self, ino: u64, nlookup: u64) -> Option<(u64, String)> {
if let Some(entry) = self.entries.get_mut(&ino) {
@@ -833,6 +1037,23 @@ impl State {
/// Allocate (or reuse) a stable inode for a node that came *from* the
/// database, which is why nothing is written back.
/// Carry the size and mtime of a file open for writing over onto `node`,
/// which is about to replace its entry.
///
/// The open handle is ahead of anything Drive can report until release
/// queues it. Taking the remote's size shrank a SQLite database to its empty
/// create while SQLite held it open, and the kernel then capped every read
/// at that size (docs/BUGS.md B109).
pub(crate) fn keep_open_write_size(&self, ino: u64, node: &mut Node) {
let (Some(aw), Some(entry)) = (self.active_writes.get(&ino), self.entries.get(&ino)) else {
return;
};
if let NodeKind::File { claimed_size, .. } = &mut node.kind {
*claimed_size = Some(aw.len as i64);
node.modification_time = entry.node.modification_time;
}
}
pub(crate) fn intern_from_db(&mut self, parent: u64, node: Node) -> u64 {
self.intern_mem(parent, node)
}
@@ -842,11 +1063,27 @@ impl State {
///
/// This is what keeps `ls` on a large folder quick: one commit for the
/// listing, rather than one autocommit — and one fsync — per child.
pub(crate) fn intern_batch(&mut self, parent: u64, nodes: Vec<Node>) -> Vec<u64> {
pub(crate) fn intern_batch(&mut self, parent: u64, mut nodes: Vec<Node>) -> Vec<u64> {
for node in &mut nodes {
self.keep_revision_mtime(node);
}
self.outbox.push(DbWrite::Upsert(nodes.clone()));
let fresh: Vec<&Node> = nodes
.iter()
.filter(|node| !self.by_uid.contains_key(&node.uid))
.collect();
let mut inos = if fresh.is_empty() {
HashMap::new()
} else {
self.new_inos(&fresh)
};
nodes
.into_iter()
.map(|node| self.intern_mem(parent, node))
.map(|node| {
let access = self.access_for_node(parent, &node);
let ino = inos.remove(&node.uid);
self.intern_mem_at(parent, node, access, ino)
})
.collect()
}
@@ -1164,7 +1401,7 @@ mod tests {
let dir = TempDir::new();
let db = Db::open(&dir.0.join("cache.db")).unwrap();
let share_access = db.all_share_access().unwrap();
let st = State::new(Arc::new(db), share_access, 1);
let st = State::new(Arc::new(db), share_access);
(st, dir)
}
@@ -1182,6 +1419,36 @@ mod tests {
(st, dir, src, dst)
}
/// A node's inode comes from its row's local id, so it is the same in
/// every mount, after the hot cache drops it, and after a restart.
#[test]
fn a_node_keeps_its_inode_across_mounts_evictions_and_restarts() {
let (mut st, _dir) = state();
let inos = st.intern_batch(
0,
vec![
node("a", "root", "a.txt", false),
node("b", "root", "b.txt", false),
],
);
st.flush_outbox();
let db = st.db.clone();
assert_ne!(inos[0], inos[1]);
assert!(!inos.contains(&crate::ROOT_INO));
let mut other = State::new(db.clone(), HashMap::new());
assert_eq!(other.intern(0, node("b", "root", "b.txt", false)), inos[1]);
st.forget_mem(&uid("a"));
assert_eq!(st.intern(0, node("a", "root", "a.txt", false)), inos[0]);
let mut restarted = State::new(db, HashMap::new());
assert_eq!(
restarted.intern_batch(0, vec![node("a", "root", "a.txt", false)]),
[inos[0]]
);
}
/// Audit A5. Moving a file out of a folder leaves that folder's DB row
/// claiming a complete listing, so once the hot cache drops it, remote
/// changes under it stop being seen. `relocate` has to clear both sides.
@@ -1247,7 +1514,7 @@ mod tests {
let root = uid("root").to_string();
let state = parking_lot::Mutex::new(st);
{
let mut guard = lock_state(&state, &db);
let mut guard = StateGuard::new(state.lock(), &db);
guard.intern(0, node("root", "none", "My Files", true));
assert!(
db.node_by_uid(&root).unwrap().is_none(),
@@ -1490,6 +1757,7 @@ mod tests {
})
.collect();
st.next_ino = crate::ROOT_INO + nodes.len() as u64;
st.roots.insert(crate::ROOT_INO);
// Reverse order so entries are inserted before their parents, matching
// the hydration path's non-topological materialization.
for (i, node) in nodes.iter().enumerate().rev() {
@@ -1863,4 +2131,137 @@ mod tests {
let shared = st.intern(owned_root, shared);
assert_eq!(st.entries[&shared].access, Access::Viewer);
}
fn at_revision(mut node: Node, revision: &str, mtime: i64) -> Node {
node.modification_time = mtime;
if let NodeKind::File {
active_revision_id, ..
} = &mut node.kind
{
*active_revision_id = Some(revision.into());
}
node
}
#[test]
fn a_renamed_file_keeps_its_mtime_until_its_content_changes() {
let (mut st, _dir) = state();
let root = st.intern(0, node("root", "none", "My Files", true));
let mut file = node("file", "root", "a.txt", false);
if let NodeKind::File {
claimed_size,
total_size_on_storage,
..
} = &mut file.kind
{
*claimed_size = Some(7);
*total_size_on_storage = 519;
}
let ino = st.intern(root, at_revision(file.clone(), "r1", 100));
// Drive stamps the link on a rename; the content is still r1.
let mut renamed = at_revision(file.clone(), "r1", 160);
renamed.name = "b.txt".into();
st.intern_batch(root, vec![renamed]);
assert_eq!(st.entries[&ino].node.modification_time, 100);
assert_eq!(st.entries[&ino].node.name, "b.txt");
// A listing that leaves the size out still describes revision r1.
let mut bare = at_revision(file.clone(), "r1", 170);
assert_eq!(crate::node_size(&st.entries[&ino].node), 7);
if let NodeKind::File { claimed_size, .. } = &mut bare.kind {
*claimed_size = None;
}
st.intern(root, bare);
assert_eq!(st.entries[&ino].node.modification_time, 100);
assert_eq!(crate::node_size(&st.entries[&ino].node), 7);
st.intern(root, at_revision(file, "r2", 200));
assert_eq!(st.entries[&ino].node.modification_time, 200);
}
fn handle(len: u64, base_size: u64) -> WriteHandle {
let path = std::env::temp_dir().join(format!("pdfs-handle-test-{}", std::process::id()));
let file = File::create(&path).unwrap();
let _ = std::fs::remove_file(&path);
WriteHandle {
ino: 2,
uid: uid("file"),
file: Arc::new(file),
path,
written: Intervals::default(),
len,
base_size,
base_mtime: 0,
base_revision_id: None,
dirty: false,
open_count: 1,
}
}
/// A file created offline is written while its queued create lands. The
/// placeholder had no revision, so the handle takes the empty file the
/// create made as its base (docs/BUGS.md B113).
#[test]
fn a_handle_on_a_placeholder_is_based_on_the_create_that_landed() {
let mut h = handle(0, 0);
assert_eq!(h.base_revision_id, None);
let mut landed = node("real", "root", "f.mkv", false);
landed.modification_time = 160;
if let NodeKind::File {
active_revision_id, ..
} = &mut landed.kind
{
*active_revision_id = Some("empty".into());
}
h.rebase_onto(&landed);
assert_eq!(h.base_mtime, 160);
assert_eq!(h.base_size, 0);
assert_eq!(h.base_revision_id.as_deref(), Some("empty"));
}
#[test]
fn a_listing_does_not_shrink_a_file_open_for_writing() {
let (mut st, _dir) = state();
let ino = st.intern(0, node("file", "root", "workload.db", false));
let mut h = handle(16384, 0);
h.ino = ino;
st.active_writes.insert(ino, h);
st.set_size(ino, 16384);
st.touch_mtime(ino, 500);
// The remote still holds the empty file the create sealed.
st.intern(0, node("file", "root", "workload.db", false));
assert_eq!(crate::node_size(&st.entries[&ino].node), 16384);
assert_eq!(st.entries[&ino].node.modification_time, 500);
// Once the handle is gone the remote is authoritative again.
st.active_writes.remove(&ino);
st.intern(0, node("file", "root", "workload.db", false));
assert_eq!(crate::node_size(&st.entries[&ino].node), 0);
}
#[test]
fn a_write_past_the_end_authors_the_hole_it_leaves() {
// Opened over a 100-byte base, shrunk to 10, then written at 50: bytes
// 10..50 are zeros now, not the base's bytes 10..50.
let mut h = handle(100, 100);
h.written.clip(10);
h.len = 10;
h.record_write(50, 60);
assert_eq!(h.len, 60);
assert_eq!(
h.written.segments(0, 60),
vec![(0, 10, false), (10, 60, true)],
"only the head still comes from the base"
);
}
#[test]
fn a_write_inside_the_file_leaves_the_rest_to_the_base() {
let mut h = handle(100, 100);
h.record_write(40, 50);
assert_eq!(h.len, 100);
assert_eq!(
h.written.segments(0, 100),
vec![(0, 40, false), (40, 50, true), (50, 100, false)]
);
}
}
+4 -5
View File
@@ -165,7 +165,7 @@ impl Core {
}
match self
.rt
.block_on(self.client.trash_nodes(std::slice::from_ref(uid)))
.block_on(self.drive.trash_nodes(std::slice::from_ref(uid)))
.and_then(batch::into_unit)
{
Ok(()) => {}
@@ -175,10 +175,9 @@ impl Core {
return;
}
}
// Every mount, not just the primary: a conflict copy inside a sync
// folder lives in that fork's inode space, and leaving it hooked up
// there keeps a node we have just trashed remotely visible and
// readable there (`docs/BUGS.md` B74).
// Leaving a conflict copy hooked up in the tree keeps a node we have
// just trashed remotely visible and readable in whichever mount shows
// it (`docs/BUGS.md` B74).
self.for_each_state(|st| {
st.forget_or_unlink(uid);
});
+242 -26
View File
@@ -73,6 +73,9 @@ enum Applied {
Downloaded,
/// A conflict copy was kept.
Conflict,
/// The local file to upload was deleted after the scan planned it. There is
/// nothing left to send, and the next pass sees the deletion (B108).
Vanished,
}
impl Core {
@@ -367,7 +370,7 @@ impl Core {
HashMap::new()
} else {
self.rt
.block_on(self.client.enumerate_nodes_light(&check))
.block_on(self.drive.enumerate_nodes_light(&check))
.map_err(|e| format!("resolve changed nodes: {e}"))?
.into_iter()
.filter(|n| !n.trashed)
@@ -522,7 +525,7 @@ impl Core {
};
if let Err(e) = self
.rt
.block_on(self.client.trash_nodes(&[uid]))
.block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit)
{
warn!(rel, error = %e, "sync: trash remote failed");
@@ -773,7 +776,7 @@ impl Core {
let remote = r.expect("remote deletion requires a remote file");
if let Err(e) = self
.rt
.block_on(self.client.trash_nodes(std::slice::from_ref(&remote.uid)))
.block_on(self.drive.trash_nodes(std::slice::from_ref(&remote.uid)))
.and_then(batch::into_unit)
{
warn!(rel, error = %e, "sync: trash remote failed");
@@ -836,7 +839,7 @@ impl Core {
for (rel, uid) in delete_remote_dirs {
if let Err(e) = self
.rt
.block_on(self.client.trash_nodes(&[uid]))
.block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit)
{
warn!(rel, error = %e, "sync: trash remote folder failed");
@@ -891,14 +894,14 @@ impl Core {
) -> Result<(), String> {
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(folder))
.block_on(self.drive.enumerate_folder_children_node_uids(folder))
.map_err(|e| format!("enumerate {folder}: {e}"))?;
if uids.is_empty() {
return Ok(());
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes_light(&uids))
.block_on(self.drive.enumerate_nodes_light(&uids))
.map_err(|e| format!("resolve nodes: {e}"))?;
for node in nodes {
if node.trashed || !node.is_folder() {
@@ -943,14 +946,14 @@ impl Core {
}
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(folder))
.block_on(self.drive.enumerate_folder_children_node_uids(folder))
.map_err(|e| format!("enumerate {folder}: {e}"))?;
if uids.is_empty() {
return Ok(());
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes_light(&uids))
.block_on(self.drive.enumerate_nodes_light(&uids))
.map_err(|e| format!("resolve nodes: {e}"))?;
// Files whose recorded mtime no longer matches: their size has to be
@@ -968,7 +971,7 @@ impl Core {
HashMap::new()
} else {
self.rt
.block_on(self.client.enumerate_nodes(&stale))
.block_on(self.drive.enumerate_nodes(&stale))
.map_err(|e| format!("resolve nodes: {e}"))?
.iter()
.map(|n| (n.uid.to_string(), remote_file_sig(n).1))
@@ -1136,7 +1139,7 @@ impl Core {
match op {
Pending::CreateDir { rel, parent } => {
let uid = self
.client
.drive
.create_folder(parent, base_name(rel), Some(now_secs()))
.await
.map_err(|e| format!("create remote folder {rel}: {e}"))?;
@@ -1147,13 +1150,10 @@ impl Core {
Ok(Applied::Dir(rel.clone(), uid))
}
Pending::UploadNew { rel, parent } => {
self.upload_new(folder_id, local_root, rel, parent).await?;
Ok(Applied::Uploaded)
self.upload_new(folder_id, local_root, rel, parent).await
}
Pending::UploadRevision { rel, uid } => {
self.upload_revision(folder_id, local_root, rel, uid)
.await?;
Ok(Applied::Uploaded)
self.upload_revision(folder_id, local_root, rel, uid).await
}
Pending::Download {
rel,
@@ -1217,10 +1217,12 @@ impl Core {
local_root: &Path,
rel: &str,
parent: &NodeUid,
) -> Result<(), String> {
) -> Result<Applied, String> {
let path = local_root.join(rel_to_path(rel));
let name = base_name(rel);
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
let Some(file) = open_to_upload(&path, rel)? else {
return Ok(Applied::Vanished);
};
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
let mtime = system_mtime(&meta);
// Taken from the descriptor being streamed, so it names the content that
@@ -1238,7 +1240,7 @@ impl Core {
.begin(name, "", TransferDirection::Upload, meta.len()),
);
let uid = self
.client
.drive
// Mirror push: the local file is authoritative. If a prior attempt was
// interrupted mid-upload it left an unsealed draft of this name; recover
// it even across a daemon restart (which rotates our client uid), so the
@@ -1261,7 +1263,8 @@ impl Core {
self.invalidate_children_of(parent);
warn_if_torn(rel, &path, streamed);
self.record_file_baseline(folder_id, rel, Some(streamed), &uid)
.await
.await?;
Ok(Applied::Uploaded)
}
/// Upload a changed local file as a new revision of an existing remote node.
@@ -1271,9 +1274,11 @@ impl Core {
local_root: &Path,
rel: &str,
uid: &NodeUid,
) -> Result<(), String> {
) -> Result<Applied, String> {
let path = local_root.join(rel_to_path(rel));
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
let Some(file) = open_to_upload(&path, rel)? else {
return Ok(Applied::Vanished);
};
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
let mtime = system_mtime(&meta);
let streamed = LocalSig::from(&meta);
@@ -1289,13 +1294,14 @@ impl Core {
meta.len(),
),
);
self.client
self.drive
.upload_new_revision_from(uid, reader, meta.len() as i64, thumbnails, Some(mtime))
.await
.map_err(|e| format!("upload revision {rel}: {e}"))?;
warn_if_torn(rel, &path, streamed);
self.record_file_baseline(folder_id, rel, Some(streamed), uid)
.await
.await?;
Ok(Applied::Uploaded)
}
/// Download a remote file to its local path (atomically via a temp file),
@@ -1328,7 +1334,21 @@ impl Core {
.map(|d| d.as_nanos())
.unwrap_or(0)
));
{
// A file an on-demand mount already read, say one just moved here from
// it, is in the content cache under the same revision key.
let seeded = size > 0 && self.cache.copy_cached_to(uid, mtime, size as u64, &tmp);
if seeded {
std::fs::File::open(&tmp)
.and_then(|f| f.sync_all())
.map_err(|e| {
let _ = std::fs::remove_file(&tmp);
format!("sync tmp {rel}: {e}")
})?;
debug!(
rel,
"sync: placed from the content cache instead of downloading"
);
} else {
let out = std::fs::File::create(&tmp).map_err(|e| format!("create tmp {rel}: {e}"))?;
// Count the bytes as they land, so the download reports progress like
// an on-demand hydration does.
@@ -1339,7 +1359,7 @@ impl Core {
size.max(0) as u64,
);
let mut out = CountingWriter::new(out, &guard);
self.client
self.drive
.download_file_to(uid, &mut out)
.await
.map_err(|e| {
@@ -1404,7 +1424,7 @@ impl Core {
let (lmtime, lsize) = (local.mtime, local.size);
// Re-fetch the node so the stored remote signature is exactly what a walk
// will report next time.
let (rmtime, rsize) = match self.client.enumerate_nodes(std::slice::from_ref(uid)).await {
let (rmtime, rsize) = match self.drive.enumerate_nodes(std::slice::from_ref(uid)).await {
Ok(nodes) => nodes
.first()
.map(remote_file_sig)
@@ -1431,6 +1451,23 @@ impl Core {
}
}
/// Open a local file for upload, or `None` when it is gone.
///
/// A scan plans uploads from a listing, and the file can be deleted before its
/// upload starts — a build or a test tree cleaned up mid-pass. That is not a
/// failure: the next pass sees the deletion. Counting it as one put the folder
/// in `error` for a file nobody wanted any more (B108).
fn open_to_upload(path: &Path, rel: &str) -> Result<Option<std::fs::File>, String> {
match std::fs::File::open(path) {
Ok(file) => Ok(Some(file)),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
debug!(rel, "sync: file deleted before its upload; skipping");
Ok(None)
}
Err(e) => Err(format!("open {rel}: {e}")),
}
}
/// Say so when a file changed under an upload that was streaming it.
///
/// Purely diagnostic — the correction is already in place, because the baseline
@@ -1602,6 +1639,76 @@ fn walk_local_tree(
Ok(())
}
/// Why the mirror copy at `rel` holds something Proton Drive does not, or
/// `None` when everything under it is on Drive exactly as it is on disk.
///
/// A move out of a mirror folder removes the local copy once Drive has moved
/// the node, so anything only this disk has would be lost with it: a file not
/// uploaded yet, one edited since, a deletion not carried up yet, or anything
/// the engine never uploads. That is why ignored names count here, and why the
/// walk is its own rather than [`walk_local_tree`], which skips symlinks.
pub(crate) fn mirror_subtree_unsynced(
root: &Path,
rel: &str,
baseline: &HashMap<String, StoredSyncEntry>,
) -> Option<String> {
let mut seen = HashSet::new();
if let Some(reason) = unsynced_local(root, rel, baseline, &mut seen) {
return Some(reason);
}
let prefix = format!("{rel}/");
baseline
.keys()
.filter(|r| *r == rel || r.starts_with(&prefix))
.find(|r| !seen.contains(r.as_str()))
.map(|r| format!("{r} was deleted here but not on Proton Drive yet"))
}
fn unsynced_local(
root: &Path,
rel: &str,
baseline: &HashMap<String, StoredSyncEntry>,
seen: &mut HashSet<String>,
) -> Option<String> {
let path = root.join(rel_to_path(rel));
let meta = match std::fs::symlink_metadata(&path) {
Ok(meta) => meta,
Err(e) => return Some(format!("cannot read {rel}: {e}")),
};
let Some(base) = baseline.get(rel) else {
return Some(format!("{rel} is not on Proton Drive yet"));
};
seen.insert(rel.to_string());
if meta.is_file() {
return (!LocalSig::from(&meta).same_content(&LocalSig::from(base)))
.then(|| format!("{rel} has changes that are not on Proton Drive yet"));
}
if !meta.is_dir() {
return Some(format!("{rel} is not a regular file or folder"));
}
let entries = match std::fs::read_dir(&path) {
Ok(entries) => entries,
Err(e) => return Some(format!("cannot read {rel}: {e}")),
};
for entry in entries {
let name = match entry {
Ok(entry) => entry.file_name(),
Err(e) => return Some(format!("cannot read {rel}: {e}")),
};
let Some(name) = name.to_str() else {
return Some(format!("{rel} holds a name that is not valid UTF-8"));
};
// A download the engine left half done; it is the engine's to clean up.
if name.contains(".pdfs-tmp-") {
continue;
}
if let Some(reason) = unsynced_local(root, &format!("{rel}/{name}"), baseline, seen) {
return Some(reason);
}
}
None
}
/// Publish a conflict copy without overwriting an earlier conflict bearing the
/// same timestamp. The original is removed only after the new copy is durable.
fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> {
@@ -2155,6 +2262,27 @@ mod tests {
assert_eq!(remote_sig(&e), None);
}
#[test]
fn a_file_deleted_before_its_upload_is_skipped_not_failed() {
let root = std::env::temp_dir().join(format!("pdfs-b108-{}", std::process::id()));
std::fs::create_dir_all(&root).unwrap();
let gone = root.join("gone.txt");
assert!(open_to_upload(&gone, "gone.txt").unwrap().is_none());
std::fs::write(&gone, b"x").unwrap();
assert!(open_to_upload(&gone, "gone.txt").unwrap().is_some());
// Other open failures still count as errors.
let err = open_to_upload(&gone.join("child"), "gone.txt/child");
assert!(err.unwrap_err().starts_with("open gone.txt/child:"));
let _ = std::fs::remove_dir_all(&root);
let mut o = Outcome::default();
o.record(&Applied::Vanished);
assert!(o.is_empty());
assert_eq!(o.errors, 0);
}
#[test]
fn outcome_summarises_only_what_it_moved() {
let mut o = Outcome::default();
@@ -2346,6 +2474,94 @@ mod tests {
);
}
/// A baseline row recording `path` as it is on disk right now.
fn synced_row(root: &Path, rel: &str) -> (String, StoredSyncEntry) {
let meta = std::fs::symlink_metadata(root.join(rel_to_path(rel))).unwrap();
let sig = LocalSig::from(&meta);
let entry = StoredSyncEntry {
rel_path: rel.to_string(),
remote_uid: Some(format!("vol~{rel}")),
local_mtime: sig.mtime,
local_mtime_ns: sig.mtime_ns,
local_size: sig.size,
remote_rev: None,
remote_hash: None,
};
(rel.to_string(), entry)
}
fn synced_subtree(name: &str) -> (PathBuf, HashMap<String, StoredSyncEntry>) {
let root = sync_test_dir(name);
std::fs::create_dir_all(root.join("dir/sub")).unwrap();
std::fs::write(root.join("dir/a.txt"), b"a").unwrap();
std::fs::write(root.join("dir/sub/b.txt"), b"b").unwrap();
let baseline = ["dir", "dir/sub", "dir/a.txt", "dir/sub/b.txt"]
.into_iter()
.map(|rel| synced_row(&root, rel))
.collect();
(root, baseline)
}
#[test]
fn a_fully_synced_mirror_subtree_may_leave() {
let (root, baseline) = synced_subtree("move-synced");
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert_eq!(reason, None);
}
#[test]
fn a_mirror_subtree_with_a_file_not_uploaded_may_not_leave() {
let (root, baseline) = synced_subtree("move-new-file");
std::fs::write(root.join("dir/sub/new.txt"), b"n").unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/sub/new.txt"));
}
#[test]
fn a_mirror_subtree_with_an_edit_not_uploaded_may_not_leave() {
let (root, baseline) = synced_subtree("move-edit");
std::fs::write(root.join("dir/a.txt"), b"longer now").unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/a.txt"));
}
#[test]
fn a_mirror_subtree_with_a_deletion_not_carried_up_may_not_leave() {
let (root, baseline) = synced_subtree("move-deleted");
std::fs::remove_file(root.join("dir/sub/b.txt")).unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/sub/b.txt"));
}
#[test]
fn a_mirror_subtree_holding_a_symlink_may_not_leave() {
let (root, baseline) = synced_subtree("move-symlink");
std::os::unix::fs::symlink("a.txt", root.join("dir/link")).unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/link"));
}
#[test]
fn a_name_prefixed_sibling_does_not_hold_a_mirror_subtree_back() {
let (root, mut baseline) = synced_subtree("move-sibling");
// Deleted here but not on Drive, yet outside the subtree being moved.
baseline.insert(
"dirt".to_string(),
StoredSyncEntry {
rel_path: "dirt".to_string(),
..baseline["dir/a.txt"].clone()
},
);
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert_eq!(reason, None);
}
/// The same rule one level up: if the sync root itself cannot be read, the
/// pass sees nothing, and "nothing" must never mean "delete everything".
#[test]
+1
View File
@@ -108,6 +108,7 @@ impl Outcome {
Applied::Uploaded => self.uploaded += 1,
Applied::Downloaded => self.downloaded += 1,
Applied::Conflict => self.conflicts += 1,
Applied::Vanished => {}
}
}
+4 -7
View File
@@ -73,7 +73,7 @@ async fn run_uploads(
.upload_thumbnails_async(t.path.clone(), t.name.clone())
.await;
match core
.client
.drive
.upload_file_from(
&t.parent_uid,
&t.name,
@@ -238,14 +238,11 @@ impl Core {
.map_err(|error| self.errno_error(error, "upload destination"))?;
let new_uid = self
.rt
.block_on(
self.client
.create_folder(parent_uid, name, Some(now_secs())),
)
.block_on(self.drive.create_folder(parent_uid, name, Some(now_secs())))
.map_err(|e| CoreError::from_api(&e, &format!("create folder {name}")))?;
let node = self
.fetch_node(&new_uid)
.map_err(|e| self.errno_error(e, "fetch node"))?;
.fetch_minted_node(&new_uid, name)
.ok_or_else(|| minted_but_unlisted(name))?;
let mut st = self.state();
let ino = st.intern(pino, node);
if let Some(kids) = st.children.get_mut(&pino)
+1
View File
@@ -27,5 +27,6 @@ fn main() {
println!("cargo:rerun-if-changed=resources/icons");
println!("cargo:rerun-if-changed=resources/style.css");
println!("cargo:rerun-if-changed=resources/proton-theme.css");
println!("cargo:rerun-if-changed=resources/prompt.css");
println!("cargo:rerun-if-changed=../../packaging/io.narl.proton-drive-linux.svg");
}
@@ -33,5 +33,7 @@
<gresource prefix="/de/nils/protondrivelinux">
<file>style.css</file>
<file>proton-theme.css</file>
<!-- The search launcher's own sheet, over the same palette. -->
<file>prompt.css</file>
</gresource>
</gresources>
+224
View File
@@ -0,0 +1,224 @@
/* The search launcher (pdfs-prompt). Colours come from the libadwaita
palette, so the launcher follows the system accent, light/dark and high
contrast like the app; the opt-in Proton theme recolours both alike. */
window.launcher-window {
background: transparent;
}
.launcher-card {
background-color: var(--view-bg-color);
color: var(--view-fg-color);
border: 1px solid var(--border-color);
border-radius: 14px;
box-shadow: 0 16px 48px rgb(0 0 0 / 35%);
}
.search-bar {
padding: 14px 18px;
border-bottom: 1px solid var(--border-color);
}
.search-icon {
-gtk-icon-size: 20px;
color: color-mix(in srgb, currentColor 55%, transparent);
}
.search-entry {
background: none;
border: none;
box-shadow: none;
outline: none;
padding: 2px 0;
font-size: 1.15rem;
}
.key-hint {
padding: 2px 8px;
border-radius: 6px;
background-color: color-mix(in srgb, currentColor 8%, transparent);
font-size: 0.75rem;
opacity: 0.55;
}
.chip-row {
padding: 10px 14px;
border-bottom: 1px solid var(--border-color);
}
.chip {
padding: 4px 14px;
min-height: 0;
border-radius: 999px;
background: none;
border: 1px solid var(--border-color);
box-shadow: none;
font-size: 0.86rem;
font-weight: 500;
}
.chip:hover {
background-color: color-mix(in srgb, currentColor 6%, transparent);
}
.chip:checked {
background-color: var(--accent-bg-color);
color: var(--accent-fg-color);
border-color: var(--accent-bg-color);
}
/* --- results ------------------------------------------------------------ */
.results {
padding: 2px 8px 10px 8px;
}
.section-header {
padding: 12px 10px 6px 10px;
}
.section-title {
font-size: 0.78rem;
font-weight: 700;
letter-spacing: 0.06em;
text-transform: uppercase;
opacity: 0.55;
}
.section-icon {
-gtk-icon-size: 14px;
opacity: 0.55;
}
.result-list {
background: none;
}
.result-row {
padding: 6px 10px;
border-radius: 10px;
}
.result-row:hover {
background-color: color-mix(in srgb, currentColor 5%, transparent);
}
/* A soft tint rather than a solid accent bar: the text keeps its colour, so
the highlighted match stays readable on the selected row. */
.result-row:selected,
.result-row:selected:hover {
background-color: color-mix(in srgb, var(--accent-bg-color) 18%, transparent);
color: inherit;
}
.result-name {
font-weight: 500;
}
.result-location,
.result-meta {
font-size: 0.8rem;
opacity: 0.55;
}
.result-row:selected .result-location,
.result-row:selected .result-meta {
opacity: 0.7;
}
.thumb {
border-radius: 6px;
}
.result-badge,
.preview-badge {
-gtk-icon-size: 14px;
}
.badge-pinned {
color: var(--success-color);
}
.badge-cached {
color: var(--accent-color);
}
.badge-cloud,
.badge-local {
opacity: 0.55;
}
/* --- preview ------------------------------------------------------------ */
.preview {
padding: 16px;
background-color: color-mix(in srgb, currentColor 3%, transparent);
}
.preview-thumb {
margin-bottom: 12px;
}
.preview-thumb .thumb {
border-radius: 10px;
}
.preview-name {
font-weight: 700;
font-size: 1.05rem;
}
.preview-kind {
font-size: 0.85rem;
opacity: 0.6;
margin-bottom: 12px;
}
.preview-key {
font-size: 0.8rem;
opacity: 0.55;
}
.preview-value {
font-size: 0.85rem;
}
.preview-actions {
margin-top: 12px;
}
.preview-action {
padding: 6px 8px;
border-radius: 8px;
}
.preview-shortcut {
font-size: 0.75rem;
opacity: 0.6;
}
.preview-feedback {
margin-top: 6px;
font-size: 0.8rem;
color: var(--accent-color);
}
.preview-empty {
opacity: 0.2;
}
/* --- footer ------------------------------------------------------------- */
.footer {
padding: 8px 16px;
border-top: 1px solid var(--border-color);
}
.footer-text {
font-size: 0.78rem;
opacity: 0.5;
}
statuspage.compact > scrolledwindow > viewport > box {
margin: 24px 12px;
}
+14 -153
View File
@@ -8,6 +8,8 @@ pub(crate) mod events;
pub(crate) mod i18n;
pub(crate) mod pages;
pub(crate) mod reload;
#[path = "../theme.rs"]
pub(crate) mod theme;
pub(crate) mod widgets;
use compat::*;
@@ -30,6 +32,7 @@ use pages::takeout::*;
use pages::trash::*;
use pages::verify::*;
use reload::{LoadTicket, Loader, replace_items};
use theme::{load_resources, proton_theme_active, set_proton_theme};
use widgets::compat::*;
use widgets::details::*;
use widgets::file_list::*;
@@ -210,7 +213,7 @@ fn main() -> glib::ExitCode {
.flags(gio::ApplicationFlags::HANDLES_COMMAND_LINE)
.build();
app.connect_startup(|_| {
load_proton_theme();
load_resources("/de/nils/protondrivelinux/style.css");
if let Ok(dirs) = AppDirs::new() {
set_proton_theme(dirs.load_config().proton_theme.unwrap_or(false));
}
@@ -287,157 +290,6 @@ pub(crate) fn spawn_tray() {
}
}
/// Register the bundled GResources (custom icons, the stylesheet) and load the
/// app stylesheet. The Proton theme is a separate provider, applied by
/// [`set_proton_theme`] from the saved preference.
fn load_proton_theme() {
let bytes = include_bytes!(concat!(env!("OUT_DIR"), "/pdfs.gresource"));
let resource_data = glib::Bytes::from_static(bytes);
if let Ok(resource) = gio::Resource::from_data(&resource_data) {
gio::resources_register(&resource);
} else {
tracing::error!("failed to load gresource bundle");
}
let Some(display) = gtk4::gdk::Display::default() else {
return;
};
let provider = gtk4::CssProvider::new();
provider.load_from_resource("/de/nils/protondrivelinux/style.css");
// Above the user's own gtk.css, which often imports a whole theme: its
// `button { padding: 6px 10px; min-width: 16px }` would otherwise pad every
// photo tile and push the grid past the window. Every rule in the sheet is
// scoped to the app's own classes, so nothing else the user styled changes.
// One above the Proton theme too, so the app's rules refine it.
gtk4::style_context_add_provider_for_display(
&display,
&provider,
gtk4::STYLE_PROVIDER_PRIORITY_USER + 2,
);
gtk4::IconTheme::for_display(&display).add_resource_path("/de/nils/protondrivelinux/icons");
}
thread_local! {
/// The Proton theme provider while it is installed. Kept so turning the
/// preference off can remove exactly what turning it on added.
static PROTON_THEME: RefCell<Option<gtk4::CssProvider>> = const { RefCell::new(None) };
}
/// Whether the Proton theme is on.
fn proton_theme_active() -> bool {
PROTON_THEME.with(|slot| slot.borrow().is_some())
}
/// libadwaita's own stylesheet, bundled in the library.
const ADWAITA_STYLESHEET: &str = "/org/gnome/Adwaita/styles/gtk.css";
/// The Proton palette laid over it, from our own bundle.
const PROTON_PALETTE: &str = "/de/nils/protondrivelinux/proton-theme.css";
/// Make the theme provider's `@media (prefers-color-scheme)` and
/// `(prefers-contrast)` rules match what libadwaita shows, now and whenever it
/// changes. A provider evaluates those itself, and a new one assumes light.
///
/// The properties are set by name because they only exist since GTK 4.20, and
/// the app still runs on older GTK, whose libadwaita has no such rules anyway.
fn follow_style_manager(provider: &gtk4::CssProvider) {
fn set_enum(provider: &gtk4::CssProvider, property: &str, nick: &str) {
let Some(pspec) = provider.find_property(property) else {
return;
};
if let Some(value) = glib::EnumClass::with_type(pspec.value_type())
.and_then(|class| class.to_value_by_nick(nick))
{
provider.set_property_from_value(property, &value);
}
}
fn apply(provider: &gtk4::CssProvider, manager: &adw::StyleManager) {
let scheme = if manager.is_dark() { "dark" } else { "light" };
set_enum(provider, "prefers-color-scheme", scheme);
let contrast = if manager.is_high_contrast() {
"more"
} else {
"no-preference"
};
set_enum(provider, "prefers-contrast", contrast);
}
let manager = adw::StyleManager::default();
apply(provider, &manager);
// Weak, so a provider the preference removed isn't kept alive; its
// handlers then do nothing.
let weak = provider.downgrade();
let weak_dark = weak.clone();
manager.connect_dark_notify(move |manager| {
if let Some(provider) = weak_dark.upgrade() {
apply(&provider, manager);
}
});
manager.connect_high_contrast_notify(move |manager| {
if let Some(provider) = weak.upgrade() {
apply(&provider, manager);
}
});
}
/// Paint the app in the Proton web apps' colours (`on`) or follow the system
/// theme and accent colour.
///
/// Setting libadwaita's colour variables alone isn't enough: a GTK theme the
/// user's `gtk.css` imports (Catppuccin, Fluent, ...) sits above the app and
/// hard-codes its own accent into hundreds of rules. So the theme provider,
/// one above the user's CSS, first resets every property, then re-applies
/// libadwaita's whole stylesheet and finally the Proton palette, which leaves
/// nothing of the user's theme inside the app.
fn set_proton_theme(on: bool) {
let Some(display) = gtk4::gdk::Display::default() else {
return;
};
PROTON_THEME.with(|slot| {
let mut slot = slot.borrow_mut();
match (on, slot.as_ref()) {
(true, None) => {
let stylesheet = |path| {
gio::resources_lookup_data(path, gio::ResourceLookupFlags::NONE)
.map(|bytes| String::from_utf8_lossy(&bytes).into_owned())
};
let palette = stylesheet(PROTON_PALETTE).unwrap_or_default();
let css = match stylesheet(ADWAITA_STYLESHEET) {
// Its images are named relative to the sheet, which a
// string loaded here doesn't have.
Ok(adwaita) => format!(
"* {{ all: unset; }}\n{}\n{palette}",
adwaita.replace(
"url(\"assets/",
"url(\"resource:///org/gnome/Adwaita/styles/assets/"
)
),
// Still recolours everything that follows libadwaita's
// variables; only a custom theme's own rules show through.
Err(e) => {
tracing::warn!("libadwaita stylesheet not found, only recolouring: {e}");
palette
}
};
let provider = gtk4::CssProvider::new();
follow_style_manager(&provider);
provider.load_from_string(&css);
gtk4::style_context_add_provider_for_display(
&display,
&provider,
gtk4::STYLE_PROVIDER_PRIORITY_USER + 1,
);
*slot = Some(provider);
}
(false, Some(provider)) => {
gtk4::style_context_remove_provider_for_display(&display, provider);
*slot = None;
}
_ => {}
}
});
}
/// Build the application window, wire the two pages, kick off the refresh loop,
/// and present it.
fn build_window(app: &adw::Application) {
@@ -816,6 +668,15 @@ fn build_window(app: &adw::Application) {
sync_sidebar(&ui_nav);
match st.visible_child_name().as_deref() {
Some("browser") => load_browser(&ui_nav),
// The Albums and Places listings are re-read in place. Loading the
// timeline instead would swap it in under the Albums or Places toggle.
Some("gallery") if !ui_nav.gallery.in_collection() => {
match ui_nav.gallery.view_switch.active() {
VIEW_ALBUMS => load_albums(&ui_nav),
VIEW_PLACES => load_places(&ui_nav),
_ => load_gallery(&ui_nav, false),
}
}
Some("gallery") => load_gallery(&ui_nav, false),
// Every visit reads the page again. The rows already on screen stay
// up while it does, so arriving costs no "Loading…" flash.
@@ -1347,7 +1208,7 @@ fn install_window_actions(ui: &Rc<Ui>, window: &adw::ApplicationWindow) {
.application_icon("io.narl.proton-drive-linux")
.version(pdfs_core::config::APP_VERSION)
.developer_name("Nils Pukropp")
.website("https://github.com/narrrl/proton-drive-linux")
.website("https://proton-drive.narl.io")
.issue_url("https://github.com/narrrl/proton-drive-linux/issues")
.copyright("© 2026 Nils Pukropp (contact@narl.io)")
.license_type(gtk4::License::MitX11)
+244 -39
View File
@@ -1920,6 +1920,10 @@ pub(crate) fn prompt_rename(ui: &Rc<Ui>, entry: &DirEntry) {
/// Pick a destination folder by browsing Drive's folders, and move `entries`
/// there through the daemon.
///
/// Besides My files the picker offers every synced folder. A move into one is
/// still made on Proton Drive, so nothing is downloaded or uploaded again; it
/// is sent with absolute paths, which is how the daemon tells locations apart.
pub(crate) fn prompt_move(ui: &Rc<Ui>, entries: Vec<DirEntry>) {
if entries.is_empty() {
return;
@@ -1967,6 +1971,7 @@ pub(crate) fn prompt_move(ui: &Rc<Ui>, entries: Vec<DirEntry>) {
dialog.set_default_response(Some("move"));
dialog.set_close_response("cancel");
let mountpoint = ui.dirs.resolved_mountpoint(&ui.dirs.load_config());
let picker = Rc::new(MovePicker {
ui: ui.clone(),
dialog: dialog.clone(),
@@ -1974,28 +1979,93 @@ pub(crate) fn prompt_move(ui: &Rc<Ui>, entries: Vec<DirEntry>) {
list,
up: up.clone(),
sources: sources.clone(),
folder: RefCell::new(String::new()),
folder: RefCell::new(MoveTarget::Locations),
generation: Cell::new(0),
});
let picker_up = picker.clone();
up.connect_clicked(move |_| {
let folder = picker_up.folder.borrow().clone();
let parent = folder.rfind('/').map(|i| &folder[..i]).unwrap_or_default();
picker_up.open(parent.to_string());
let parent = picker_up.folder.borrow().parent();
picker_up.open(parent);
});
// Start where the browser is: moving is most often one level up or down.
picker.open(ui.browser.path.borrow().clone());
picker.open(MoveTarget::Drive(ui.browser.path.borrow().clone()));
let ui = ui.clone();
let picker_done = picker.clone();
dialog.connect_response(None, move |_, resp| {
if resp == "move" {
run_bulk_move(&ui, sources.clone(), picker_done.folder.borrow().clone());
if resp != "move" {
return;
}
match picker_done.folder.borrow().clone() {
MoveTarget::Locations => {}
MoveTarget::Drive(folder) => run_bulk_move(&ui, sources.clone(), folder),
MoveTarget::Synced { path, .. } => {
let absolute = sources
.iter()
.map(|rel| mountpoint.join(rel).to_string_lossy().into_owned())
.collect();
run_bulk_move(&ui, absolute, path);
}
}
});
dialog.present(ui_window(&picker.ui).as_ref());
}
/// Where the Move dialog is looking, and the destination if the user confirms.
#[derive(Clone, Debug, PartialEq)]
pub(crate) enum MoveTarget {
/// The list of places to move to: My files and each synced folder.
Locations,
/// A folder in My files, mountpoint-relative; empty for My files itself.
Drive(String),
/// A folder inside a synced folder, as an absolute local path. `root` is
/// the synced folder. A mirror folder is a plain directory, read from disk.
Synced {
root: String,
path: String,
mirror: bool,
},
}
impl MoveTarget {
/// One level up. The top of a location leads back to the list of them.
pub(crate) fn parent(&self) -> MoveTarget {
match self {
MoveTarget::Locations => MoveTarget::Locations,
MoveTarget::Drive(folder) if folder.is_empty() => MoveTarget::Locations,
MoveTarget::Drive(folder) => MoveTarget::Drive(
folder
.rfind('/')
.map(|i| folder[..i].to_string())
.unwrap_or_default(),
),
MoveTarget::Synced { root, path, .. } if path == root => MoveTarget::Locations,
MoveTarget::Synced { root, path, mirror } => MoveTarget::Synced {
root: root.clone(),
path: Path::new(path)
.parent()
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|| root.clone()),
mirror: *mirror,
},
}
}
/// `name` inside this folder, in the same form.
fn child(&self, name: &str) -> MoveTarget {
match self {
MoveTarget::Locations => MoveTarget::Locations,
MoveTarget::Drive(folder) if folder.is_empty() => MoveTarget::Drive(name.to_string()),
MoveTarget::Drive(folder) => MoveTarget::Drive(format!("{folder}/{name}")),
MoveTarget::Synced { root, path, mirror } => MoveTarget::Synced {
root: root.clone(),
path: Path::new(path).join(name).to_string_lossy().into_owned(),
mirror: *mirror,
},
}
}
}
/// The Move dialog's folder browser.
struct MovePicker {
ui: Rc<Ui>,
@@ -2006,71 +2076,81 @@ struct MovePicker {
/// What is being moved, mountpoint-relative.
sources: Vec<String>,
/// The folder on show, and the destination if the user confirms now.
folder: RefCell<String>,
folder: RefCell<MoveTarget>,
/// Drops a listing that arrives after the user went elsewhere.
generation: Cell<u64>,
}
impl MovePicker {
fn open(self: &Rc<Self>, folder: String) {
fn open(self: &Rc<Self>, folder: MoveTarget) {
let generation = self.generation.get().wrapping_add(1);
self.generation.set(generation);
self.location.set_label(if folder.is_empty() {
"Proton Drive"
} else {
&folder
});
self.location.set_tooltip_text(Some(&folder));
self.up.set_sensitive(!folder.is_empty());
self.dialog
.set_response_enabled("move", move_target_allowed(&self.sources, &folder));
let (label, tooltip) = match &folder {
MoveTarget::Locations => (gettext("Locations"), String::new()),
MoveTarget::Drive(rel) if rel.is_empty() => (gettext("My files"), String::new()),
MoveTarget::Drive(rel) => (rel.clone(), rel.clone()),
MoveTarget::Synced { path, .. } => (path.clone(), path.clone()),
};
self.location.set_label(&label);
self.location.set_tooltip_text(Some(&tooltip));
self.up
.set_sensitive(!matches!(folder, MoveTarget::Locations));
let allowed = match &folder {
MoveTarget::Locations => false,
MoveTarget::Drive(rel) => move_target_allowed(&self.sources, rel),
// What is moved is in My files, so it cannot be in a synced folder.
MoveTarget::Synced { .. } => true,
};
self.dialog.set_response_enabled("move", allowed);
*self.folder.borrow_mut() = folder.clone();
while let Some(row) = self.list.first_child() {
self.list.remove(&row);
}
self.list.append(&picker_note(&gettext("Loading…")));
let rx = spawn_request(
self.ui.dirs.control_socket(),
Request::ListDir {
path: folder.clone(),
},
);
let picker = self.clone();
glib::spawn_future_local(async move {
let result = rx.recv().await;
let listed = match &folder {
MoveTarget::Locations => picker.locations().await,
MoveTarget::Drive(rel) => picker.drive_folders(&folder, rel.clone()).await,
MoveTarget::Synced {
path, mirror: true, ..
} => local_folders(&folder, path.clone()).await,
MoveTarget::Synced { path, .. } => {
picker.drive_folders(&folder, path.clone()).await
}
};
if picker.generation.get() != generation {
return;
}
while let Some(row) = picker.list.first_child() {
picker.list.remove(&row);
}
let Ok(Ok(Response::Entries { entries })) = result else {
let Some(rows) = listed else {
picker
.list
.append(&picker_note(&gettext("Couldn't read this folder.")));
return;
};
let mut folders: Vec<DirEntry> = entries.into_iter().filter(|e| e.is_dir).collect();
folders.sort_by_key(|e| e.name.to_lowercase());
let mut shown = 0;
for entry in folders {
let path = if folder.is_empty() {
entry.name.clone()
} else {
format!("{folder}/{}", entry.name)
};
for (title, subtitle, target) in rows {
// A folder cannot go inside itself, so do not offer to open it.
if picker.sources.contains(&path) {
if let MoveTarget::Drive(path) = &target
&& picker.sources.contains(path)
{
continue;
}
let row = adw::ActionRow::builder()
.title(glib::markup_escape_text(&entry.name).as_str())
.title(glib::markup_escape_text(&title).as_str())
.activatable(true)
.build();
if let Some(subtitle) = subtitle {
row.set_subtitle(glib::markup_escape_text(&subtitle).as_str());
}
row.add_prefix(&gtk4::Image::from_icon_name("folder-symbolic"));
row.add_suffix(&gtk4::Image::from_icon_name("go-next-symbolic"));
let open = picker.clone();
row.connect_activated(move |_| open.open(path.clone()));
row.connect_activated(move |_| open.open(target.clone()));
picker.list.append(&row);
shown += 1;
}
@@ -2081,6 +2161,92 @@ impl MovePicker {
}
});
}
/// My files and every synced folder, as rows of the top level.
async fn locations(&self) -> Option<Vec<PickerRow>> {
let mut rows = vec![(gettext("My files"), None, MoveTarget::Drive(String::new()))];
let rx = spawn_request(self.ui.dirs.control_socket(), Request::ListSyncFolders);
let Ok(Ok(Response::SyncFolders { mut items })) = rx.recv().await else {
// My files alone is still somewhere to move to.
return Some(rows);
};
items.sort_by_key(|f| f.local_path.to_lowercase());
for folder in items {
let name = Path::new(&folder.local_path)
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_else(|| folder.local_path.clone());
rows.push((
name,
Some(folder.local_path.clone()),
MoveTarget::Synced {
root: folder.local_path.clone(),
path: folder.local_path,
mirror: folder.mode == "mirror",
},
));
}
Some(rows)
}
/// The folders in a mounted folder: `folder` is mountpoint-relative for My
/// files and absolute for an on-demand synced folder.
async fn drive_folders(&self, here: &MoveTarget, folder: String) -> Option<Vec<PickerRow>> {
let rx = spawn_request(
self.ui.dirs.control_socket(),
Request::ListDir {
path: folder.clone(),
},
);
let Ok(Ok(Response::Entries { entries })) = rx.recv().await else {
return None;
};
let mut names: Vec<String> = entries
.into_iter()
.filter(|e| e.is_dir)
.map(|e| e.name)
.collect();
names.sort_by_key(|name| name.to_lowercase());
Some(
names
.into_iter()
.map(|name| {
let target = here.child(&name);
(name, None, target)
})
.collect(),
)
}
}
/// One row of the picker: its title, an optional subtitle, and where it leads.
type PickerRow = (String, Option<String>, MoveTarget);
/// The folders in a mirror folder's directory `path`, read from disk off the
/// main thread. A mirror folder has no mount for `ListDir` to read.
async fn local_folders(here: &MoveTarget, path: String) -> Option<Vec<PickerRow>> {
let names = gio::spawn_blocking(move || {
let mut names: Vec<String> = std::fs::read_dir(&path)
.ok()?
.flatten()
.filter(|e| e.file_type().is_ok_and(|t| t.is_dir()))
.filter_map(|e| e.file_name().into_string().ok())
.filter(|name| !name.contains(".pdfs-tmp-"))
.collect();
names.sort_by_key(|name| name.to_lowercase());
Some(names)
})
.await
.ok()??;
Some(
names
.into_iter()
.map(|name| {
let target = here.child(&name);
(name, None, target)
})
.collect(),
)
}
fn picker_note(text: &str) -> gtk4::ListBoxRow {
@@ -2786,8 +2952,8 @@ fn listing_summary(files: usize, folders: usize, file_bytes: u64) -> String {
#[cfg(test)]
mod tests {
use super::{
hit_location, listing_summary, move_target_allowed, show_thumbnail_build_progress,
sort_entries,
MoveTarget, hit_location, listing_summary, move_target_allowed,
show_thumbnail_build_progress, sort_entries,
};
use pdfs_core::config::{FileSort, FilesView};
use pdfs_core::control::{DirEntry, ThumbnailBuildStatus};
@@ -2880,4 +3046,43 @@ mod tests {
status.running = false;
assert!(!show_thumbnail_build_progress(&status));
}
#[test]
fn move_picker_climbs_from_a_location_top_to_the_list_of_locations() {
assert_eq!(
MoveTarget::Drive(String::new()).parent(),
MoveTarget::Locations
);
let root = MoveTarget::Synced {
root: "/home/u/Docs".into(),
path: "/home/u/Docs".into(),
mirror: true,
};
assert_eq!(root.parent(), MoveTarget::Locations);
}
#[test]
fn move_picker_climbs_one_folder_at_a_time_inside_a_location() {
assert_eq!(
MoveTarget::Drive("a/b".into()).parent(),
MoveTarget::Drive("a".into())
);
assert_eq!(
MoveTarget::Drive("a".into()).parent(),
MoveTarget::Drive(String::new())
);
let inner = MoveTarget::Synced {
root: "/home/u/Docs".into(),
path: "/home/u/Docs/x/y".into(),
mirror: false,
};
assert_eq!(
inner.parent(),
MoveTarget::Synced {
root: "/home/u/Docs".into(),
path: "/home/u/Docs/x".into(),
mirror: false,
}
);
}
}
+9 -7
View File
@@ -262,20 +262,22 @@ pub(crate) fn build_locations_page(history: &gtk4::Widget) -> (gtk4::Widget, Loc
(frame.upcast(), widgets)
}
/// `child` in a clamp that scrolls vertically only, with page margins.
/// `child` across the page's full width, scrolling vertically only, with page
/// margins.
fn scrolled_column(child: &impl IsA<gtk4::Widget>) -> gtk4::ScrolledWindow {
let clamp = adw::Clamp::builder().child(child).build();
clamp.set_margin_top(18);
clamp.set_margin_bottom(18);
clamp.set_margin_start(18);
clamp.set_margin_end(18);
let column = gtk4::Box::new(gtk4::Orientation::Vertical, 0);
column.append(child);
column.set_margin_top(18);
column.set_margin_bottom(18);
column.set_margin_start(18);
column.set_margin_end(18);
// Never scroll sideways: a location's subtitle holds a full path, and letting
// the row grow to fit one pushes the controls at its end off screen.
// Constrained, the path ellipsizes and the controls stay reachable.
gtk4::ScrolledWindow::builder()
.vexpand(true)
.hscrollbar_policy(gtk4::PolicyType::Never)
.child(&clamp)
.child(&column)
.build()
}
+5 -1
View File
@@ -1069,7 +1069,11 @@ pub(crate) fn repaint_transfers(ui: &Rc<Ui>, items: &[TransferItem], jobs: &[Job
let row_box = gtk4::Box::new(gtk4::Orientation::Vertical, 4);
row_box.set_margin_top(8);
row_box.set_margin_bottom(8);
let label = gtk4::Label::builder().halign(gtk4::Align::Start).build();
// Ellipsized so a long file name cannot widen the page.
let label = gtk4::Label::builder()
.halign(gtk4::Align::Start)
.ellipsize(gtk4::pango::EllipsizeMode::Middle)
.build();
label.add_css_class("dim-label");
let bar = gtk4::ProgressBar::new();
row_box.append(&label);
+143
View File
@@ -0,0 +1,143 @@
//! Match highlighting: which characters of a result name the query matched,
//! as Pango markup with those characters in bold.
//!
//! Each query term is looked for as a substring of the name first, and as a
//! subsequence ("rpt" in "report") when that fails, which is roughly how the
//! daemon's ranking treats a term too. Case is folded one character at a
//! time, so every range stays on a character boundary of the original name,
//! whatever the script.
use gtk4::glib;
/// One character of a name: its byte range and its case-folded form.
struct Folded {
start: usize,
end: usize,
folded: String,
}
fn fold(text: &str) -> Vec<Folded> {
text.char_indices()
.map(|(start, c)| Folded {
start,
end: start + c.len_utf8(),
folded: c.to_lowercase().collect(),
})
.collect()
}
/// The byte ranges of `name` that `query` matched, sorted and merged.
pub(crate) fn ranges(name: &str, query: &str) -> Vec<(usize, usize)> {
let chars = fold(name);
let mut found: Vec<(usize, usize)> = Vec::new();
for term in query.split_whitespace() {
let term: Vec<String> = fold(term).into_iter().map(|c| c.folded).collect();
if let Some(at) = substring(&chars, &term) {
found.push((chars[at].start, chars[at + term.len() - 1].end));
} else if let Some(hits) = subsequence(&chars, &term) {
found.extend(hits.into_iter().map(|at| (chars[at].start, chars[at].end)));
}
}
found.sort_unstable();
let mut merged: Vec<(usize, usize)> = Vec::with_capacity(found.len());
for (start, end) in found {
match merged.last_mut() {
Some(last) if start <= last.1 => last.1 = last.1.max(end),
_ => merged.push((start, end)),
}
}
merged
}
/// Where `term` first occurs in `chars`, as a character index.
fn substring(chars: &[Folded], term: &[String]) -> Option<usize> {
if term.is_empty() || term.len() > chars.len() {
return None;
}
(0..=chars.len() - term.len()).find(|&at| {
chars[at..at + term.len()]
.iter()
.zip(term)
.all(|(c, t)| c.folded == *t)
})
}
/// The character indices of the leftmost match of `term` as a subsequence,
/// or `None` when not every character of it can be found in order.
fn subsequence(chars: &[Folded], term: &[String]) -> Option<Vec<usize>> {
if term.is_empty() {
return None;
}
let mut hits = Vec::with_capacity(term.len());
let mut next = 0;
for wanted in term {
let at = (next..chars.len()).find(|&i| chars[i].folded == *wanted)?;
hits.push(at);
next = at + 1;
}
Some(hits)
}
/// `name` as Pango markup, escaped, with the matched characters in bold.
pub(crate) fn markup(name: &str, query: &str) -> String {
let mut out = String::with_capacity(name.len() + 16);
let mut done = 0;
for (start, end) in ranges(name, query) {
out.push_str(&glib::markup_escape_text(&name[done..start]));
out.push_str("<b>");
out.push_str(&glib::markup_escape_text(&name[start..end]));
out.push_str("</b>");
done = end;
}
out.push_str(&glib::markup_escape_text(&name[done..]));
out
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_substring_match_is_one_range_in_any_case() {
assert_eq!(ranges("Tax Report.pdf", "REP"), vec![(4, 7)]);
}
#[test]
fn a_term_that_is_no_substring_falls_back_to_a_subsequence() {
// r, p and t of "report", not the "rt" pair at the end of a substring.
assert_eq!(ranges("report.txt", "rpt"), vec![(0, 1), (2, 3), (5, 6)]);
}
#[test]
fn a_term_with_no_match_highlights_nothing() {
assert!(ranges("report.txt", "xyz").is_empty());
assert!(ranges("report.txt", "").is_empty());
}
#[test]
fn several_terms_merge_into_sorted_ranges() {
assert_eq!(
ranges("tax return 2024", "2024 tax ret"),
vec![(0, 3), (4, 7), (11, 15)]
);
assert_eq!(ranges("abc", "ab bc"), vec![(0, 3)]);
}
#[test]
fn non_ascii_names_keep_ranges_on_character_boundaries() {
assert_eq!(ranges("Über Größe.odt", "über"), vec![(0, 5)]);
assert_eq!(ranges("Über Größe.odt", "GRÖ"), vec![(6, 10)]);
assert_eq!(markup("Über Größe.odt", "größe"), "Über <b>Größe</b>.odt");
assert_eq!(
markup("日本語のメモ.txt", "メモ"),
"日本語の<b>メモ</b>.txt"
);
}
#[test]
fn markup_escapes_the_name_inside_and_outside_matches() {
assert_eq!(markup("a<b>&c", "b"), "a&lt;<b>b</b>&gt;&amp;c");
assert_eq!(markup("R&D.pdf", "r&d"), "<b>R&amp;D</b>.pdf");
assert_eq!(markup("<none>", "zzz"), "&lt;none&gt;");
}
}
+363
View File
@@ -0,0 +1,363 @@
//! Recent files and frecency.
//!
//! The launcher remembers what it opened in `prompt-history.json` under the
//! state directory. With an empty query those files come first, and while
//! searching a file opened often and lately gets a bonus on top of its match
//! score (see [`frecency_bonus`]).
//!
//! One writer thread owns the file: it loads it, applies every recorded visit
//! in order and writes the result back atomically. A visit recorded before the
//! first load finished simply waits in the channel, so it is merged into the
//! loaded list rather than overwriting it. The main thread only ever sees
//! snapshots.
use std::cell::RefCell;
use std::collections::HashMap;
use std::io::Write;
use std::path::{Path, PathBuf};
use std::rc::Rc;
use gtk4::glib;
use crate::Hit;
/// How many files the history keeps. More than the launcher ever shows, so
/// frecency still knows a file that dropped off the recent list.
const MAX_VISITS: usize = 50;
/// The frecency bonus of one visit made just now.
const VISIT_BONUS: f64 = 1500.0;
/// The bonus halves every this many days without a visit.
const HALF_LIFE_DAYS: f64 = 14.0;
/// The largest frecency bonus. A name match is worth up to 20000 points, so
/// history breaks ties between similar matches but cannot beat a much better
/// one.
const MAX_BONUS: f64 = 6000.0;
#[derive(Clone, serde::Serialize, serde::Deserialize)]
pub(crate) struct Visit {
pub(crate) hit: Hit,
/// When the file was last opened, in epoch seconds.
pub(crate) last: i64,
pub(crate) count: u32,
}
/// The history bonus for a file opened `count` times, the last time
/// `age_secs` ago.
pub(crate) fn frecency_bonus(count: u32, age_secs: i64) -> i64 {
let age_days = age_secs.max(0) as f64 / 86_400.0;
let bonus = f64::from(count) * VISIT_BONUS * 0.5f64.powf(age_days / HALF_LIFE_DAYS);
bonus.min(MAX_BONUS) as i64
}
/// Count one more visit of `hit` at `now`, newest first, and keep at most
/// [`MAX_VISITS`]. The stored hit is replaced, so its size, date and sync
/// state are the ones from the latest search.
fn record(visits: &mut Vec<Visit>, hit: Hit, now: i64) {
let key = hit.key();
let count = match visits.iter().position(|visit| visit.hit.key() == key) {
Some(at) => visits.remove(at).count.saturating_add(1),
None => 1,
};
visits.insert(
0,
Visit {
hit,
last: now,
count,
},
);
visits.truncate(MAX_VISITS);
}
/// Drop local files that no longer exist. Drive files stay: checking them
/// would touch the mount, and a stale one only costs a row.
fn prune(visits: &mut Vec<Visit>) -> bool {
let before = visits.len();
visits.retain(|visit| match &visit.hit {
Hit::Local(hit) => Path::new(&hit.path).symlink_metadata().is_ok(),
Hit::Drive(_) => true,
});
visits.len() != before
}
fn load(path: &Path) -> Vec<Visit> {
match std::fs::read(path) {
Ok(bytes) => serde_json::from_slice(&bytes).unwrap_or_else(|e| {
tracing::warn!("ignoring unreadable {}: {e}", path.display());
Vec::new()
}),
Err(_) => Vec::new(),
}
}
/// Write the history through a temporary file and a rename, so a crash
/// mid-write leaves the old file rather than half a new one.
fn save(path: &Path, visits: &[Visit]) -> std::io::Result<()> {
let json = serde_json::to_vec(visits).map_err(std::io::Error::other)?;
if let Some(dir) = path.parent() {
std::fs::create_dir_all(dir)?;
}
let tmp = path.with_extension("json.tmp");
let mut file = std::fs::File::create(&tmp)?;
file.write_all(&json)?;
file.sync_all()?;
std::fs::rename(&tmp, path)
}
enum Command {
Record(Hit, i64),
Prune,
}
/// The writer thread's loop: load, then apply commands until the launcher
/// exits. `publish` receives the list after the load and after every change.
fn serve(path: &Path, commands: &async_channel::Receiver<Command>, publish: impl Fn(Vec<Visit>)) {
let mut visits = load(path);
let pruned = prune(&mut visits);
publish(visits.clone());
if pruned && let Err(e) = save(path, &visits) {
tracing::warn!("cannot write {}: {e}", path.display());
}
while let Ok(command) = commands.recv_blocking() {
let changed = match command {
Command::Record(hit, now) => {
record(&mut visits, hit, now);
true
}
Command::Prune => prune(&mut visits),
};
if changed {
if let Err(e) = save(path, &visits) {
tracing::warn!("cannot write {}: {e}", path.display());
}
publish(visits.clone());
}
}
}
/// The main thread's view of the history.
pub(crate) struct History {
visits: Rc<RefCell<Vec<Visit>>>,
commands: async_channel::Sender<Command>,
}
impl History {
/// Start the writer thread. `changed` runs on the main thread whenever a
/// new snapshot arrives.
pub(crate) fn start(path: PathBuf, changed: impl Fn() + 'static) -> Self {
let (commands, inbox) = async_channel::unbounded();
let (snapshots, updates) = async_channel::unbounded::<Vec<Visit>>();
let spawned = std::thread::Builder::new()
.name("prompt-history".into())
.spawn(move || {
serve(&path, &inbox, |visits| {
let _ = snapshots.send_blocking(visits);
})
});
if let Err(e) = spawned {
tracing::warn!("cannot start the history writer: {e}");
}
let visits = Rc::new(RefCell::new(Vec::new()));
let shared = visits.clone();
glib::spawn_future_local(async move {
while let Ok(snapshot) = updates.recv().await {
*shared.borrow_mut() = snapshot;
changed();
}
});
Self { visits, commands }
}
/// Remember that `hit` was opened just now.
pub(crate) fn record(&self, hit: Hit) {
let _ = self.commands.try_send(Command::Record(hit, now()));
}
/// Forget local files that were deleted since they were opened.
pub(crate) fn prune(&self) {
let _ = self.commands.try_send(Command::Prune);
}
/// The most recently opened files, newest first.
pub(crate) fn recent(&self, limit: usize) -> Vec<Hit> {
self.visits
.borrow()
.iter()
.take(limit)
.map(|visit| visit.hit.clone())
.collect()
}
/// The frecency bonus of every remembered file, by [`Hit::key`].
pub(crate) fn bonuses(&self) -> HashMap<(bool, String), i64> {
let now = now();
self.visits
.borrow()
.iter()
.map(|visit| {
(
visit.hit.key(),
frecency_bonus(visit.count, now - visit.last),
)
})
.collect()
}
}
fn now() -> i64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map_or(0, |d| d.as_secs() as i64)
}
#[cfg(test)]
mod tests {
use super::*;
use pdfs_core::control::{LocalHit, SearchHit};
const DAY: i64 = 86_400;
fn drive(name: &str) -> Hit {
Hit::Drive(SearchHit {
name: name.into(),
path: format!("Docs/{name}"),
is_dir: false,
size: 1,
modified: 1,
pinned: false,
cached: false,
uid: format!("v~{name}"),
mounted_path: None,
score: 0,
})
}
fn scratch_dir(test: &str) -> PathBuf {
let dir = std::env::temp_dir().join(format!(
"pdfs-prompt-history-{test}-{}-{}",
std::process::id(),
now()
));
std::fs::create_dir_all(&dir).unwrap();
dir
}
#[test]
fn the_bonus_halves_every_two_weeks() {
assert_eq!(frecency_bonus(1, 0), 1500);
assert_eq!(frecency_bonus(1, 14 * DAY), 750);
assert_eq!(frecency_bonus(1, 28 * DAY), 375);
assert_eq!(frecency_bonus(2, 14 * DAY), 1500);
// A clock that went backwards counts as "just now", not as a boost.
assert_eq!(frecency_bonus(1, -DAY), 1500);
}
#[test]
fn the_bonus_is_capped() {
assert_eq!(frecency_bonus(4, 0), 6000);
assert_eq!(frecency_bonus(1000, 0), 6000);
assert!(frecency_bonus(1000, 365 * DAY) < 6000);
}
#[test]
fn a_repeat_visit_moves_the_file_to_the_top_and_counts_it() {
let mut visits = Vec::new();
record(&mut visits, drive("a.pdf"), 10);
record(&mut visits, drive("b.pdf"), 20);
record(&mut visits, drive("a.pdf"), 30);
assert_eq!(visits.len(), 2);
assert_eq!(visits[0].hit.name(), "a.pdf");
assert_eq!((visits[0].count, visits[0].last), (2, 30));
assert_eq!(visits[1].count, 1);
}
#[test]
fn the_history_keeps_the_newest_fifty_files() {
let mut visits = Vec::new();
for n in 0..60 {
record(&mut visits, drive(&format!("{n}.pdf")), n);
}
assert_eq!(visits.len(), MAX_VISITS);
assert_eq!(visits[0].hit.name(), "59.pdf");
assert_eq!(visits[MAX_VISITS - 1].hit.name(), "10.pdf");
}
#[test]
fn the_history_file_round_trips() {
let dir = scratch_dir("round-trip");
let path = dir.join("prompt-history.json");
let mut visits = Vec::new();
record(&mut visits, drive("a.pdf"), 10);
record(&mut visits, drive("a.pdf"), 20);
save(&path, &visits).unwrap();
let loaded = load(&path);
assert_eq!(loaded.len(), 1);
assert_eq!(loaded[0].hit.key(), drive("a.pdf").key());
assert_eq!((loaded[0].count, loaded[0].last), (2, 20));
assert!(!path.with_extension("json.tmp").exists());
std::fs::remove_dir_all(dir).unwrap();
}
#[test]
fn visits_recorded_before_the_load_are_merged_into_it() {
let dir = scratch_dir("merge");
let path = dir.join("prompt-history.json");
let mut stored = Vec::new();
record(&mut stored, drive("old.pdf"), 10);
record(&mut stored, drive("old.pdf"), 20);
save(&path, &stored).unwrap();
// Both visits are queued before the writer has read the file.
let (commands, inbox) = async_channel::unbounded();
commands
.send_blocking(Command::Record(drive("old.pdf"), 30))
.unwrap();
commands
.send_blocking(Command::Record(drive("new.pdf"), 40))
.unwrap();
drop(commands);
let published = RefCell::new(Vec::new());
serve(&path, &inbox, |visits| published.borrow_mut().push(visits));
let last = published.borrow().last().cloned().unwrap();
assert_eq!(last.len(), 2);
assert_eq!(last[0].hit.name(), "new.pdf");
assert_eq!((last[1].hit.name(), last[1].count), ("old.pdf", 3));
let on_disk = load(&path);
assert_eq!(on_disk.len(), 2);
assert_eq!(on_disk[1].count, 3);
std::fs::remove_dir_all(dir).unwrap();
}
#[test]
fn pruning_drops_deleted_local_files_only() {
let dir = scratch_dir("prune");
let kept = dir.join("kept.txt");
std::fs::write(&kept, "x").unwrap();
let local = |path: &Path| {
Hit::Local(LocalHit {
name: path.file_name().unwrap().to_string_lossy().into_owned(),
path: path.display().to_string(),
is_dir: false,
size: 1,
modified: 1,
score: 0,
})
};
let mut visits = Vec::new();
record(&mut visits, local(&kept), 1);
record(&mut visits, local(&dir.join("gone.txt")), 2);
record(&mut visits, drive("remote.pdf"), 3);
assert!(prune(&mut visits));
let names: Vec<&str> = visits.iter().map(|visit| visit.hit.name()).collect();
assert_eq!(names, ["remote.pdf", "kept.txt"]);
std::fs::remove_dir_all(dir).unwrap();
}
}
File diff suppressed because it is too large. Load diff
+323
View File
@@ -0,0 +1,323 @@
//! The preview pane beside the results: a thumbnail of the selected hit, what
//! it is, where it is and whether it is on this computer, and the actions for
//! it with their shortcuts.
use std::cell::RefCell;
use std::path::Path;
use std::rc::Rc;
use std::time::Duration;
use adw::prelude::*;
use gtk4::{gio, glib};
use super::row::{fill_slot, thumb_slot};
use super::thumbs::Thumbs;
use crate::{Hit, dates, gettext, human_bytes, pgettext};
/// Width of the pane.
pub(crate) const PREVIEW_WIDTH: i32 = 300;
/// The thumbnail area, inside the pane's padding.
const THUMB_WIDTH: i32 = 268;
const THUMB_HEIGHT: i32 = 180;
/// How long "Path copied" stays up.
const COPIED_FEEDBACK: Duration = Duration::from_millis(1500);
/// One label/value line of the details grid.
struct Detail {
title: gtk4::Label,
value: gtk4::Label,
}
impl Detail {
fn new(grid: &gtk4::Grid, row: i32, title: &str) -> Self {
let title = gtk4::Label::builder()
.label(title)
.xalign(0.0)
.valign(gtk4::Align::Start)
.build();
title.add_css_class("preview-key");
// A wrapping label asks for its whole text on one line as its natural
// width, which would widen the pane and squeeze the results. Capping it
// keeps the pane at its fixed width; long values wrap, then ellipsize.
let value = gtk4::Label::builder()
.xalign(0.0)
.hexpand(true)
.wrap(true)
.wrap_mode(gtk4::pango::WrapMode::WordChar)
.max_width_chars(1)
.lines(3)
.ellipsize(gtk4::pango::EllipsizeMode::Middle)
.build();
value.add_css_class("preview-value");
grid.attach(&title, 0, row, 1, 1);
grid.attach(&value, 1, row, 1, 1);
Self { title, value }
}
/// Show `value`, or hide the line when there is nothing to say.
fn set(&self, value: Option<&str>) {
self.title.set_visible(value.is_some());
self.value.set_visible(value.is_some());
self.value.set_label(value.unwrap_or_default());
}
}
pub(crate) struct Preview {
pub(crate) root: gtk4::Box,
stack: gtk4::Stack,
slot: gtk4::Overlay,
name: gtk4::Label,
kind: gtk4::Label,
size: Detail,
modified: Detail,
location: Detail,
availability: Detail,
availability_icon: gtk4::Image,
pub(crate) open: gtk4::Button,
open_label: gtk4::Label,
pub(crate) reveal: gtk4::Button,
pub(crate) copy: gtk4::Button,
copied: gtk4::Revealer,
copied_timer: Rc<RefCell<Option<glib::SourceId>>>,
}
/// A flat action button: the label, and its shortcut at the far end.
fn action(icon: &str, label: &str, accelerator: &str) -> (gtk4::Button, gtk4::Label) {
let content = gtk4::Box::new(gtk4::Orientation::Horizontal, 10);
content.append(&gtk4::Image::from_icon_name(icon));
let text = gtk4::Label::builder()
.label(label)
.xalign(0.0)
.hexpand(true)
.max_width_chars(1)
.ellipsize(gtk4::pango::EllipsizeMode::End)
.build();
content.append(&text);
let shortcut = gtk4::ShortcutLabel::new(accelerator);
shortcut.add_css_class("preview-shortcut");
content.append(&shortcut);
let button = gtk4::Button::builder()
.child(&content)
.focus_on_click(false)
.focusable(false)
.build();
button.add_css_class("flat");
button.add_css_class("preview-action");
(button, text)
}
impl Preview {
pub(crate) fn new() -> Self {
let details = gtk4::Box::new(gtk4::Orientation::Vertical, 0);
let slot = thumb_slot(THUMB_WIDTH, THUMB_HEIGHT, 96, gtk4::ContentFit::Contain);
slot.add_css_class("preview-thumb");
details.append(&slot);
let name = gtk4::Label::builder()
.xalign(0.0)
.wrap(true)
.wrap_mode(gtk4::pango::WrapMode::WordChar)
.max_width_chars(1)
.lines(3)
.ellipsize(gtk4::pango::EllipsizeMode::End)
.build();
name.add_css_class("preview-name");
details.append(&name);
let kind = gtk4::Label::builder()
.xalign(0.0)
.max_width_chars(1)
.ellipsize(gtk4::pango::EllipsizeMode::End)
.build();
kind.add_css_class("preview-kind");
details.append(&kind);
let grid = gtk4::Grid::builder()
.row_spacing(6)
.column_spacing(12)
.build();
grid.add_css_class("preview-details");
let size = Detail::new(&grid, 0, &pgettext("column", "Size"));
let modified = Detail::new(&grid, 1, &pgettext("column", "Modified"));
let location = Detail::new(&grid, 2, &gettext("Location"));
// Translators: a heading in the launcher's preview pane, followed by
// "Available offline", "Downloaded", "Online only" or "On this computer".
let availability = Detail::new(&grid, 3, &gettext("Availability"));
// The availability value is an icon and a text.
let availability_icon = gtk4::Image::new();
availability_icon.add_css_class("preview-badge");
grid.remove(&availability.value);
let availability_box = gtk4::Box::new(gtk4::Orientation::Horizontal, 6);
availability_box.append(&availability_icon);
availability_box.append(&availability.value);
grid.attach(&availability_box, 1, 3, 1, 1);
details.append(&grid);
let spacer = gtk4::Box::builder().vexpand(true).build();
details.append(&spacer);
let actions = gtk4::Box::new(gtk4::Orientation::Vertical, 2);
actions.add_css_class("preview-actions");
let (open, open_label) = action("document-open-symbolic", "", "Return");
let (reveal, _) = action(
"folder-open-symbolic",
&gettext("Show in Folder"),
"<Control>Return",
);
let (copy, _) = action("edit-copy-symbolic", &gettext("Copy Path"), "<Control>c");
actions.append(&open);
actions.append(&reveal);
actions.append(&copy);
details.append(&actions);
let copied_label = gtk4::Label::new(Some(&gettext("Path copied")));
copied_label.add_css_class("preview-feedback");
let copied = gtk4::Revealer::builder()
.transition_type(gtk4::RevealerTransitionType::Crossfade)
.child(&copied_label)
.build();
details.append(&copied);
let empty = gtk4::Image::from_icon_name("system-search-symbolic");
empty.set_pixel_size(64);
empty.add_css_class("preview-empty");
let stack = gtk4::Stack::builder()
.transition_type(gtk4::StackTransitionType::Crossfade)
.transition_duration(100)
.vexpand(true)
.build();
stack.add_named(&empty, Some("empty"));
stack.add_named(&details, Some("details"));
let root = gtk4::Box::builder()
.orientation(gtk4::Orientation::Vertical)
.width_request(PREVIEW_WIDTH)
.hexpand(false)
.build();
root.add_css_class("preview");
root.append(&stack);
Self {
root,
stack,
slot,
name,
kind,
size,
modified,
location,
availability,
availability_icon,
open,
open_label,
reveal,
copy,
copied,
copied_timer: Rc::new(RefCell::new(None)),
}
}
/// Show `hit`, or the empty pane when nothing is selected.
pub(crate) fn show(&self, hit: Option<&Hit>, thumbs: &Rc<Thumbs>, mountpoint: &Path) {
let Some(hit) = hit else {
self.stack.set_visible_child_name("empty");
return;
};
self.stack.set_visible_child_name("details");
fill_slot(&self.slot, hit, thumbs, mountpoint);
self.name.set_label(hit.name());
self.name.set_tooltip_text(Some(hit.name()));
let content_type = if hit.is_dir() {
glib::GString::from("inode/directory")
} else {
gio::functions::content_type_guess(Some(hit.name()), None).0
};
self.kind
.set_label(&gio::functions::content_type_get_description(&content_type));
let size = (!hit.is_dir() && hit.size() > 0).then(|| human_bytes(hit.size()));
self.size.set(size.as_deref());
let modified = (hit.modified() > 0).then(|| dates::relative(hit.modified()));
self.modified.set(modified.as_deref());
self.modified.value.set_tooltip_text(
(hit.modified() > 0)
.then(|| dates::full(hit.modified()))
.as_deref(),
);
let location = hit.location();
self.location.set(Some(&location));
self.location.value.set_tooltip_text(Some(&location));
match availability(hit) {
Some((icon, class, text)) => {
self.availability.set(Some(&text));
self.availability_icon.set_icon_name(Some(icon));
for other in ["badge-pinned", "badge-cached", "badge-cloud", "badge-local"] {
self.availability_icon.remove_css_class(other);
}
self.availability_icon.add_css_class(class);
self.availability_icon.set_visible(true);
}
None => {
self.availability.set(None);
self.availability_icon.set_visible(false);
}
}
self.open_label.set_label(&if hit.is_dir() {
gettext("Open folder")
} else {
pgettext("verb", "Open")
});
}
/// Say "Path copied" for a moment.
pub(crate) fn flash_copied(&self) {
self.copied.set_reveal_child(true);
if let Some(source) = self.copied_timer.borrow_mut().take() {
source.remove();
}
let copied = self.copied.clone();
let slot = self.copied_timer.clone();
let timer = glib::timeout_add_local_once(COPIED_FEEDBACK, move || {
slot.borrow_mut().take();
copied.set_reveal_child(false);
});
*self.copied_timer.borrow_mut() = Some(timer);
}
}
/// Where the hit's content is: its icon, its style class and its wording.
fn availability(hit: &Hit) -> Option<(&'static str, &'static str, String)> {
match hit {
Hit::Local(_) => Some((
"drive-harddisk-symbolic",
"badge-local",
gettext("On this computer"),
)),
Hit::Drive(drive) if drive.pinned => Some((
"pdfs-offline-symbolic",
"badge-pinned",
gettext("Available offline"),
)),
// A folder is never downloaded as a whole; only pinning says anything.
Hit::Drive(drive) if drive.is_dir => None,
Hit::Drive(drive) if drive.cached => Some((
"pdfs-cached-symbolic",
"badge-cached",
pgettext("file state", "Downloaded"),
)),
Hit::Drive(_) => Some((
"pdfs-online-only-symbolic",
"badge-cloud",
pgettext("file state", "Online only"),
)),
}
}
+187
View File
@@ -0,0 +1,187 @@
//! Result rows: a thumbnail or icon, the name with its matched characters in
//! bold, the location, and on the right the sync state, size and age.
use std::cell::Cell;
use std::path::{Path, PathBuf};
use std::rc::Rc;
use adw::prelude::*;
use gtk4::{gdk, gio};
use super::highlight;
use super::thumbs::Thumbs;
use crate::theme::proton_theme_active;
use crate::{Hit, format_age, human_bytes};
/// Edge of a row's thumbnail.
const ROW_THUMB: i32 = 40;
/// A thumbnail surface: an icon, with a picture laid over it that shows once
/// the thumbnail has loaded. [`Thumbs`] paints the picture.
pub(crate) fn thumb_slot(
width: i32,
height: i32,
icon_size: i32,
fit: gtk4::ContentFit,
) -> gtk4::Overlay {
let fallback = gtk4::Image::builder().pixel_size(icon_size).build();
let picture = gtk4::Picture::builder()
.width_request(width)
.height_request(height)
.content_fit(fit)
.can_shrink(true)
.visible(false)
.build();
picture.set_overflow(gtk4::Overflow::Hidden);
picture.add_css_class("thumb");
let overlay = gtk4::Overlay::new();
overlay.set_size_request(width, height);
overlay.set_halign(gtk4::Align::Center);
overlay.set_valign(gtk4::Align::Center);
overlay.set_child(Some(&fallback));
overlay.add_overlay(&picture);
overlay
}
/// Show `hit` in a slot from [`thumb_slot`]: its icon at once, and its
/// thumbnail when there is one.
pub(crate) fn fill_slot(slot: &gtk4::Overlay, hit: &Hit, thumbs: &Rc<Thumbs>, mountpoint: &Path) {
let Some(fallback) = slot.child().and_downcast::<gtk4::Image>() else {
return;
};
let Some(picture) = fallback.next_sibling().and_downcast::<gtk4::Picture>() else {
return;
};
fallback.set_from_gicon(&icon_for(hit.name(), hit.is_dir()));
fallback.set_visible(true);
picture.set_visible(false);
picture.set_paintable(gdk::Paintable::NONE);
thumbs.want(hit, mountpoint, &picture);
}
/// The desktop's own full-colour icon for a file name, so results look like
/// the user's file manager. Folders are Proton's purple ones while the Proton
/// theme is on, as in the app's browser.
pub(crate) fn icon_for(name: &str, is_dir: bool) -> gio::Icon {
if is_dir {
let folder = if proton_theme_active() {
"pdfs-proton-folder"
} else {
"folder"
};
return gio::ThemedIcon::new(folder).upcast();
}
let (content_type, _uncertain) = gio::functions::content_type_guess(Some(name), None);
gio::functions::content_type_get_icon(&content_type)
}
/// The sync badge of a Drive file, as in the app's browser: pinned, cached,
/// or online only. Folders and local files have none.
pub(crate) fn badge_for(hit: &Hit) -> Option<(&'static str, &'static str)> {
let Hit::Drive(drive) = hit else {
return None;
};
if drive.is_dir {
return None;
}
Some(if drive.pinned {
("pdfs-offline-symbolic", "badge-pinned")
} else if drive.cached {
("pdfs-cached-symbolic", "badge-cached")
} else {
("pdfs-online-only-symbolic", "badge-cloud")
})
}
pub(crate) fn build_row(
hit: &Hit,
query: &str,
thumbs: &Rc<Thumbs>,
mountpoint: &Path,
) -> gtk4::ListBoxRow {
let row = gtk4::ListBoxRow::new();
row.add_css_class("result-row");
// Focus stays in the search entry; the list only shows the cursor.
row.set_focusable(false);
let content = gtk4::Box::new(gtk4::Orientation::Horizontal, 12);
let slot = thumb_slot(ROW_THUMB, ROW_THUMB, 32, gtk4::ContentFit::Cover);
fill_slot(&slot, hit, thumbs, mountpoint);
content.append(&slot);
let text = gtk4::Box::new(gtk4::Orientation::Vertical, 1);
text.set_hexpand(true);
text.set_valign(gtk4::Align::Center);
let name = gtk4::Label::builder()
.label(highlight::markup(hit.name(), query))
.use_markup(true)
.xalign(0.0)
.ellipsize(gtk4::pango::EllipsizeMode::Middle)
.build();
name.add_css_class("result-name");
text.append(&name);
let location = gtk4::Label::builder()
.label(hit.location())
.xalign(0.0)
.ellipsize(gtk4::pango::EllipsizeMode::Start)
.build();
location.add_css_class("result-location");
text.append(&location);
content.append(&text);
let meta = gtk4::Box::new(gtk4::Orientation::Horizontal, 10);
meta.set_valign(gtk4::Align::Center);
if let Some((icon, class)) = badge_for(hit) {
let badge = gtk4::Image::from_icon_name(icon);
badge.add_css_class("result-badge");
badge.add_css_class(class);
meta.append(&badge);
}
if !hit.is_dir() && hit.size() > 0 {
let size = gtk4::Label::new(Some(&human_bytes(hit.size())));
size.add_css_class("result-meta");
meta.append(&size);
}
if hit.modified() > 0 {
let modified = gtk4::Label::new(Some(&format_age(hit.modified())));
modified.add_css_class("result-meta");
meta.append(&modified);
}
content.append(&meta);
row.set_child(Some(&content));
row
}
/// Let a row be dragged into a file manager or an application as a file.
/// `dropped` runs once a drop was accepted; a cancelled drag changes nothing.
pub(crate) fn attach_drag(row: &gtk4::ListBoxRow, path: PathBuf, dropped: impl Fn() + 'static) {
let source = gtk4::DragSource::new();
source.set_actions(gdk::DragAction::COPY);
let files = gdk::FileList::from_array(&[gio::File::for_path(&path)]);
source.set_content(Some(&gdk::ContentProvider::for_value(&files.to_value())));
let cancelled = Rc::new(Cell::new(false));
let begin = cancelled.clone();
source.connect_drag_begin(move |source, _| {
begin.set(false);
if let Some(widget) = source.widget() {
source.set_icon(Some(&gtk4::WidgetPaintable::new(Some(&widget))), 0, 0);
}
});
let cancel = cancelled.clone();
source.connect_drag_cancel(move |_, _, _| {
cancel.set(true);
false
});
source.connect_drag_end(move |_, _, _| {
if !cancelled.get() {
dropped();
}
});
row.add_controller(source);
}
+697
View File
@@ -0,0 +1,697 @@
//! Thumbnails for the launcher's rows and preview pane.
//!
//! The main thread keeps a small LRU of textures, the keys known to have no
//! thumbnail, and the pictures waiting for one (as weak references, so a row
//! that was rebuilt or dropped keeps nothing alive). Everything that touches
//! a file runs on two worker threads, which try in order:
//!
//! 1. the freedesktop thumbnail cache, valid only when its `Thumb::MTime`
//! matches the file's;
//! 2. for local images, decoding the file itself, and writing the result back
//! to that cache for the next time and for every other application;
//! 3. for Drive images, the daemon's `FileThumbs`, batched after a short quiet
//! period.
//!
//! Workers hand back `gdk::Texture`s, never a `Pixbuf`, which may not cross
//! threads. Each job carries the render epoch it was queued in, and workers
//! skip jobs a newer query has made pointless.
use std::cell::{Cell, RefCell};
use std::collections::{HashMap, HashSet, VecDeque};
use std::io::Write;
use std::os::unix::fs::{DirBuilderExt, OpenOptionsExt};
use std::path::{Path, PathBuf};
use std::rc::{Rc, Weak};
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, UNIX_EPOCH};
use gtk4::gdk_pixbuf::Pixbuf;
use gtk4::prelude::*;
use gtk4::{gdk, glib};
use pdfs_core::control::{FileThumbRequest, Request, Response, is_thumbnail_image_name};
use crate::{Hit, spawn_request};
/// Decoded textures kept in memory. A launcher shows a few dozen rows, so
/// this covers several searches back.
const TEXTURE_CACHE_MAX: usize = 128;
const MISSING_MAX: usize = 1024;
const WORKERS: usize = 2;
/// Edge of the thumbnails this loader decodes and writes: the freedesktop
/// "large" size.
const THUMB_SIZE: i32 = 256;
/// Local images above this size are not decoded here; a thumbnail is not
/// worth reading that much.
const MAX_SOURCE_BYTES: u64 = 64 * 1024 * 1024;
/// Quiet period before Drive misses go to the daemon as one batch, so typing
/// a word asks once for the final result set.
const DAEMON_DEBOUNCE: Duration = Duration::from_millis(150);
const DAEMON_RETRY: Duration = Duration::from_secs(1);
const DAEMON_BATCH: usize = 32;
/// Where a job finds its image.
enum Source {
/// A file on this machine.
Local(PathBuf),
/// A Drive file: the thumbnail cache is keyed by its mounted path, and
/// the daemon is asked on a miss.
Drive {
mounted: PathBuf,
request: FileThumbRequest,
},
/// A thumbnail the daemon made, in its own cache.
Daemon(PathBuf),
}
struct Job {
epoch: u64,
key: String,
source: Source,
}
enum Outcome {
Texture(gdk::Texture),
Missing,
/// Not in the thumbnail cache; the daemon has to make it.
Daemon(FileThumbRequest),
/// Queued for an older render; handed back in case it is wanted again.
Skipped(Source),
}
struct Done {
key: String,
outcome: Outcome,
}
pub(crate) struct Thumbs {
socket: PathBuf,
epoch: Arc<AtomicU64>,
jobs: async_channel::Sender<Job>,
textures: RefCell<HashMap<String, gdk::Texture>>,
/// Least recently used first.
order: RefCell<VecDeque<String>>,
missing: RefCell<HashSet<String>>,
missing_order: RefCell<VecDeque<String>>,
waiters: RefCell<HashMap<String, Vec<glib::WeakRef<gtk4::Picture>>>>,
/// Keys with a worker job or a daemon request outstanding.
working: RefCell<HashSet<String>>,
/// Drive misses waiting for the next batch, by key.
queued: RefCell<HashMap<String, FileThumbRequest>>,
/// Drive items the daemon is working on under `generation`, by key.
sent: RefCell<HashMap<String, FileThumbRequest>>,
generation: Cell<u64>,
reserving: Cell<bool>,
inflight: Cell<bool>,
flush: RefCell<Option<glib::SourceId>>,
}
impl Thumbs {
pub(crate) fn new(socket: PathBuf) -> Rc<Self> {
let epoch = Arc::new(AtomicU64::new(0));
let (jobs, inbox) = async_channel::unbounded::<Job>();
let (done, results) = async_channel::unbounded::<Done>();
for n in 0..WORKERS {
let inbox = inbox.clone();
let done = done.clone();
let epoch = epoch.clone();
let spawned = std::thread::Builder::new()
.name(format!("prompt-thumbs-{n}"))
.spawn(move || work(n, &inbox, &done, &epoch));
if let Err(e) = spawned {
tracing::warn!("cannot start a thumbnail worker: {e}");
}
}
let thumbs = Rc::new(Self {
socket,
epoch,
jobs,
textures: RefCell::new(HashMap::new()),
order: RefCell::new(VecDeque::new()),
missing: RefCell::new(HashSet::new()),
missing_order: RefCell::new(VecDeque::new()),
waiters: RefCell::new(HashMap::new()),
working: RefCell::new(HashSet::new()),
queued: RefCell::new(HashMap::new()),
sent: RefCell::new(HashMap::new()),
generation: Cell::new(0),
reserving: Cell::new(false),
inflight: Cell::new(false),
flush: RefCell::new(None),
});
let weak = Rc::downgrade(&thumbs);
glib::spawn_future_local(async move {
while let Ok(done) = results.recv().await {
let Some(thumbs) = weak.upgrade() else {
break;
};
thumbs.finish(done);
}
});
thumbs
}
/// Paint `hit`'s thumbnail into `picture` now if it is known, or as soon
/// as it is. Folders and non-images keep the slot's icon.
pub(crate) fn want(self: &Rc<Self>, hit: &Hit, mountpoint: &Path, picture: &gtk4::Picture) {
let key = hit.thumb_key();
// The widget name binds the picture to one key: a late reply for
// another hit must not paint into it.
picture.set_widget_name(&key);
if hit.is_dir() || !is_thumbnail_image_name(hit.name()) {
return;
}
if matches!(hit, Hit::Drive(drive) if drive.uid.is_empty()) {
return;
}
let cached = self.textures.borrow().get(&key).cloned();
if let Some(texture) = cached {
self.touch(&key);
paint(picture, &key, &texture);
return;
}
if self.missing.borrow().contains(&key) {
return;
}
self.waiters
.borrow_mut()
.entry(key.clone())
.or_default()
.push(picture.downgrade());
if !self.working.borrow_mut().insert(key.clone()) {
return;
}
let source = match hit {
Hit::Local(local) => Source::Local(PathBuf::from(&local.path)),
Hit::Drive(drive) => Source::Drive {
mounted: hit.fs_path(mountpoint),
request: FileThumbRequest {
uid: drive.uid.clone(),
modified: drive.modified,
name: drive.name.clone(),
},
},
};
self.dispatch(key, source);
}
/// The rows are about to be rebuilt: forget their pictures, and let the
/// workers skip what was queued for them.
pub(crate) fn begin_render(&self) {
self.epoch.fetch_add(1, Ordering::SeqCst);
self.waiters.borrow_mut().clear();
}
/// The rows are rebuilt and have asked for their thumbnails. Drive work
/// for hits that are gone is dropped, and cancelled at the daemon when it
/// had already been sent.
pub(crate) fn end_render(self: &Rc<Self>) {
let wanted: HashSet<String> = self.waiters.borrow().keys().cloned().collect();
let mut dropped = Vec::new();
self.queued.borrow_mut().retain(|key, _| {
let keep = wanted.contains(key);
if !keep {
dropped.push(key.clone());
}
keep
});
let stale = self.sent.borrow().keys().any(|key| !wanted.contains(key));
if stale {
self.cancel_daemon();
// Still-wanted items go back into the next batch, under a fresh
// generation.
let sent: Vec<(String, FileThumbRequest)> = self.sent.borrow_mut().drain().collect();
for (key, request) in sent {
if wanted.contains(&key) {
self.queued.borrow_mut().insert(key, request);
} else {
dropped.push(key);
}
}
self.schedule(DAEMON_DEBOUNCE);
}
let mut working = self.working.borrow_mut();
for key in dropped {
working.remove(&key);
}
}
/// The launcher was dismissed: stop everything.
pub(crate) fn cancel_all(&self) {
self.epoch.fetch_add(1, Ordering::SeqCst);
self.waiters.borrow_mut().clear();
self.working.borrow_mut().clear();
self.queued.borrow_mut().clear();
self.sent.borrow_mut().clear();
if let Some(source) = self.flush.borrow_mut().take() {
source.remove();
}
self.cancel_daemon();
}
fn dispatch(&self, key: String, source: Source) {
let job = Job {
epoch: self.epoch.load(Ordering::SeqCst),
key: key.clone(),
source,
};
if self.jobs.try_send(job).is_err() {
self.working.borrow_mut().remove(&key);
}
}
fn wanted(&self, key: &str) -> bool {
self.waiters.borrow().contains_key(key)
}
fn finish(self: &Rc<Self>, done: Done) {
let Done { key, outcome } = done;
match outcome {
Outcome::Texture(texture) => {
self.working.borrow_mut().remove(&key);
self.store_texture(&key, texture.clone());
let waiters = self.waiters.borrow_mut().remove(&key);
for picture in waiters.into_iter().flatten() {
if let Some(picture) = picture.upgrade() {
paint(&picture, &key, &texture);
}
}
}
Outcome::Missing => {
self.working.borrow_mut().remove(&key);
self.store_missing(key.clone());
self.waiters.borrow_mut().remove(&key);
}
Outcome::Daemon(request) => {
if self.wanted(&key) {
self.queued.borrow_mut().insert(key, request);
self.schedule(DAEMON_DEBOUNCE);
} else {
self.working.borrow_mut().remove(&key);
}
}
Outcome::Skipped(source) => {
// A newer render may want the same image again.
if self.wanted(&key) {
self.dispatch(key, source);
} else {
self.working.borrow_mut().remove(&key);
}
}
}
}
fn touch(&self, key: &str) {
let mut order = self.order.borrow_mut();
if let Some(at) = order.iter().position(|held| held == key) {
order.remove(at);
order.push_back(key.to_string());
}
}
fn store_texture(&self, key: &str, texture: gdk::Texture) {
let mut textures = self.textures.borrow_mut();
if textures.insert(key.to_string(), texture).is_none() {
self.order.borrow_mut().push_back(key.to_string());
} else {
drop(textures);
self.touch(key);
return;
}
let mut order = self.order.borrow_mut();
while order.len() > TEXTURE_CACHE_MAX {
if let Some(old) = order.pop_front() {
textures.remove(&old);
}
}
}
fn store_missing(&self, key: String) {
let mut missing = self.missing.borrow_mut();
let mut order = self.missing_order.borrow_mut();
if missing.insert(key.clone()) {
order.push_back(key);
}
while order.len() > MISSING_MAX {
if let Some(old) = order.pop_front() {
missing.remove(&old);
}
}
}
/// Stop the daemon's work for this launcher. The generation is shared
/// with the app, so this is only sent when something was asked for.
fn cancel_daemon(&self) {
let generation = self.generation.replace(0);
if generation != 0 {
// The reply is not needed; `spawn_request` owns the round-trip.
drop(spawn_request(
self.socket.clone(),
Request::CancelFileThumbs {
generation: generation.saturating_add(1),
},
));
}
}
fn schedule(self: &Rc<Self>, delay: Duration) {
if let Some(source) = self.flush.borrow_mut().take() {
source.remove();
}
let weak = Rc::downgrade(self);
let source = glib::timeout_add_local_once(delay, move || {
if let Some(thumbs) = weak.upgrade() {
thumbs.flush.borrow_mut().take();
thumbs.send_batch();
}
});
*self.flush.borrow_mut() = Some(source);
}
fn send_batch(self: &Rc<Self>) {
if self.inflight.get() || self.queued.borrow().is_empty() {
return;
}
if self.generation.get() == 0 {
self.reserve();
return;
}
let batch: Vec<(String, FileThumbRequest)> = {
let mut queued = self.queued.borrow_mut();
let keys: Vec<String> = queued.keys().take(DAEMON_BATCH).cloned().collect();
keys.into_iter()
.filter_map(|key| queued.remove_entry(&key))
.collect()
};
self.sent.borrow_mut().extend(batch.iter().cloned());
let generation = self.generation.get();
self.inflight.set(true);
let rx = spawn_request(
self.socket.clone(),
Request::FileThumbs {
items: batch.iter().map(|(_, request)| request.clone()).collect(),
generation,
},
);
let weak = Rc::downgrade(self);
glib::spawn_future_local(async move {
let reply = rx.recv().await;
let Some(thumbs) = weak.upgrade() else {
return;
};
thumbs.inflight.set(false);
thumbs.receive(generation, batch, reply);
if !thumbs.queued.borrow().is_empty() {
thumbs.schedule(Duration::ZERO);
}
});
}
fn receive(
self: &Rc<Self>,
generation: u64,
batch: Vec<(String, FileThumbRequest)>,
reply: Result<Result<Response, String>, async_channel::RecvError>,
) {
// Cancelled while in flight: `end_render` or `cancel_all` already
// re-queued what is still wanted. A finished thumbnail is still worth
// painting.
let current = generation == self.generation.get();
match reply {
Ok(Ok(Response::Thumbs { items })) => {
let mut by_uid: HashMap<String, (String, FileThumbRequest)> = batch
.into_iter()
.map(|(key, request)| (request.uid.clone(), (key, request)))
.collect();
let mut again = Vec::new();
for item in items {
let Some((key, request)) = by_uid.remove(&item.uid) else {
continue;
};
if let Some(path) = item.path {
if current {
self.sent.borrow_mut().remove(&key);
}
if self.wanted(&key) {
self.working.borrow_mut().insert(key.clone());
self.dispatch(key, Source::Daemon(PathBuf::from(path)));
} else {
self.working.borrow_mut().remove(&key);
}
} else if !current {
continue;
} else if item.pending {
again.push((key, request));
} else {
self.sent.borrow_mut().remove(&key);
self.working.borrow_mut().remove(&key);
self.store_missing(key.clone());
self.waiters.borrow_mut().remove(&key);
}
}
if current {
again.extend(by_uid.into_values());
self.retry(again);
}
}
Ok(Ok(Response::FileThumbsStale)) if current => {
self.generation.set(0);
self.retry(batch);
}
Ok(Ok(Response::Error { message, .. })) if current => {
tracing::debug!("launcher thumbnails failed: {message}");
self.retry(batch);
}
_ if current => self.retry(batch),
_ => {}
}
}
/// Ask again for items the daemon is still making, after a pause.
fn retry(self: &Rc<Self>, items: Vec<(String, FileThumbRequest)>) {
if items.is_empty() {
return;
}
let weak = Rc::downgrade(self);
glib::timeout_add_local_once(DAEMON_RETRY, move || {
let Some(thumbs) = weak.upgrade() else {
return;
};
for (key, request) in items {
// Gone from `sent` means cancelled or already answered.
if thumbs.sent.borrow_mut().remove(&key).is_some() && thumbs.wanted(&key) {
thumbs.queued.borrow_mut().insert(key, request);
} else {
thumbs.working.borrow_mut().remove(&key);
}
}
thumbs.send_batch();
});
}
fn reserve(self: &Rc<Self>) {
if self.reserving.replace(true) {
return;
}
let rx = spawn_request(self.socket.clone(), Request::ReserveFileThumbGeneration);
let weak: Weak<Self> = Rc::downgrade(self);
glib::spawn_future_local(async move {
let reply = rx.recv().await;
let Some(thumbs) = weak.upgrade() else {
return;
};
thumbs.reserving.set(false);
match reply {
Ok(Ok(Response::FileThumbGeneration { generation })) if generation != 0 => {
thumbs.generation.set(generation);
thumbs.send_batch();
}
_ => thumbs.schedule(DAEMON_RETRY),
}
});
}
}
/// Show `texture` in `picture` if the picture still belongs to `key`, and hide
/// the icon underneath it.
fn paint(picture: &gtk4::Picture, key: &str, texture: &gdk::Texture) {
if picture.widget_name() != key {
return;
}
picture.set_paintable(Some(texture));
picture.set_visible(true);
if let Some(overlay) = picture.parent().and_downcast::<gtk4::Overlay>()
&& let Some(fallback) = overlay.child()
{
fallback.set_visible(false);
}
}
fn work(
index: usize,
inbox: &async_channel::Receiver<Job>,
done: &async_channel::Sender<Done>,
epoch: &AtomicU64,
) {
while let Ok(job) = inbox.recv_blocking() {
// A daemon thumbnail is already made; decoding it is cheap and the
// texture is cached for the next render.
let stale =
!matches!(job.source, Source::Daemon(_)) && job.epoch < epoch.load(Ordering::SeqCst);
let outcome = if stale {
Outcome::Skipped(job.source)
} else {
load(index, job.source)
};
if done
.send_blocking(Done {
key: job.key,
outcome,
})
.is_err()
{
break;
}
}
}
fn load(worker: usize, source: Source) -> Outcome {
match source {
Source::Local(path) => {
let Ok(meta) = std::fs::metadata(&path) else {
return Outcome::Missing;
};
let mtime = meta
.modified()
.ok()
.and_then(|at| at.duration_since(UNIX_EPOCH).ok())
.map_or(0, |at| at.as_secs() as i64);
if let Some(pixbuf) = cached(&path, mtime) {
return texture(&pixbuf);
}
if meta.len() > MAX_SOURCE_BYTES {
return Outcome::Missing;
}
let Some((pixbuf, scaled)) = decode(&path) else {
return Outcome::Missing;
};
// An image that already fits is its own thumbnail.
if scaled && let Err(e) = write_cached(worker, &path, mtime, &pixbuf) {
tracing::debug!("cannot cache a thumbnail of {}: {e}", path.display());
}
texture(&pixbuf)
}
Source::Drive { mounted, request } => match cached(&mounted, request.modified) {
Some(pixbuf) => texture(&pixbuf),
None => Outcome::Daemon(request),
},
Source::Daemon(path) => match gdk::Texture::from_filename(&path) {
Ok(texture) => Outcome::Texture(texture),
Err(e) => {
tracing::debug!("cannot decode {}: {e}", path.display());
Outcome::Missing
}
},
}
}
/// `$XDG_CACHE_HOME/thumbnails`.
fn thumbnail_root() -> PathBuf {
glib::user_cache_dir().join("thumbnails")
}
/// The URI and its MD5, which name a file's entry in the thumbnail cache.
fn cache_name(path: &Path) -> Option<(String, String)> {
let uri = glib::filename_to_uri(path, None).ok()?.to_string();
let md5 = glib::compute_checksum_for_data(glib::ChecksumType::Md5, uri.as_bytes())?.to_string();
Some((uri, md5))
}
/// A thumbnail from the shared cache, if one was made for this version of
/// the file.
fn cached(path: &Path, mtime: i64) -> Option<Pixbuf> {
let (_, md5) = cache_name(path)?;
let root = thumbnail_root();
["large", "x-large", "normal"].iter().find_map(|size| {
let pixbuf = Pixbuf::from_file(root.join(size).join(format!("{md5}.png"))).ok()?;
let stamp = pixbuf.option("tEXt::Thumb::MTime")?;
(stamp.parse::<i64>().ok()? == mtime).then_some(pixbuf)
})
}
/// Decode a local image at most [`THUMB_SIZE`] on its longer edge, upright.
/// The bool says whether it had to be scaled down.
fn decode(path: &Path) -> Option<(Pixbuf, bool)> {
let (_, width, height) = Pixbuf::file_info(path)?;
let scaled = width > THUMB_SIZE || height > THUMB_SIZE;
let pixbuf = if scaled {
Pixbuf::from_file_at_scale(path, THUMB_SIZE, THUMB_SIZE, true)
} else {
Pixbuf::from_file(path)
}
.ok()?;
let pixbuf = pixbuf.apply_embedded_orientation().unwrap_or(pixbuf);
Some((pixbuf, scaled))
}
/// Store a thumbnail as the freedesktop spec asks: in `large/`, with
/// `Thumb::URI` and `Thumb::MTime`, readable only by the user, and written
/// through a temporary file so another reader never sees half of it.
fn write_cached(worker: usize, path: &Path, mtime: i64, pixbuf: &Pixbuf) -> std::io::Result<()> {
let (uri, md5) = cache_name(path).ok_or_else(|| std::io::Error::other("no file URI"))?;
let dir = thumbnail_root().join("large");
std::fs::DirBuilder::new()
.recursive(true)
.mode(0o700)
.create(&dir)?;
let mtime = mtime.to_string();
let png = pixbuf
.save_to_bufferv(
"png",
&[
("tEXt::Thumb::URI", uri.as_str()),
("tEXt::Thumb::MTime", mtime.as_str()),
],
)
.map_err(std::io::Error::other)?;
let tmp = dir.join(format!(
"{md5}.png.pdfs-{}-{worker}.tmp",
std::process::id()
));
let written = std::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.mode(0o600)
.open(&tmp)
.and_then(|mut file| file.write_all(&png))
.and_then(|()| std::fs::rename(&tmp, dir.join(format!("{md5}.png"))));
if written.is_err() {
let _ = std::fs::remove_file(&tmp);
}
written
}
/// A `Pixbuf` may not leave its thread; its pixels, as a memory texture, can.
fn texture(pixbuf: &Pixbuf) -> Outcome {
let format = if pixbuf.has_alpha() {
gdk::MemoryFormat::R8g8b8a8
} else {
gdk::MemoryFormat::R8g8b8
};
let texture = gdk::MemoryTexture::new(
pixbuf.width(),
pixbuf.height(),
format,
&pixbuf.read_pixel_bytes(),
pixbuf.rowstride() as usize,
);
Outcome::Texture(texture.upcast())
}
File diff suppressed because it is too large. Load diff
+158
View File
@@ -0,0 +1,158 @@
//! The bundled resources and the optional Proton theme, shared by `pdfs-app`
//! and `pdfs-prompt` so both paint the same way from the same preference.
use std::cell::RefCell;
use adw::prelude::*;
use gtk4::{gio, glib};
/// Register the bundled GResources (custom icons, the stylesheets) and load
/// `stylesheet`, the front end's own sheet. The Proton theme is a separate
/// provider, applied by [`set_proton_theme`] from the saved preference.
pub(crate) fn load_resources(stylesheet: &str) {
let bytes = include_bytes!(concat!(env!("OUT_DIR"), "/pdfs.gresource"));
let resource_data = glib::Bytes::from_static(bytes);
if let Ok(resource) = gio::Resource::from_data(&resource_data) {
gio::resources_register(&resource);
} else {
tracing::error!("failed to load gresource bundle");
}
let Some(display) = gtk4::gdk::Display::default() else {
return;
};
let provider = gtk4::CssProvider::new();
provider.load_from_resource(stylesheet);
// Above the user's own gtk.css, which often imports a whole theme: its
// `button { padding: 6px 10px; min-width: 16px }` would otherwise pad every
// photo tile and push the grid past the window. Every rule in the sheet is
// scoped to the app's own classes, so nothing else the user styled changes.
// One above the Proton theme too, so the app's rules refine it.
gtk4::style_context_add_provider_for_display(
&display,
&provider,
gtk4::STYLE_PROVIDER_PRIORITY_USER + 2,
);
gtk4::IconTheme::for_display(&display).add_resource_path("/de/nils/protondrivelinux/icons");
}
thread_local! {
/// The Proton theme provider while it is installed. Kept so turning the
/// preference off can remove exactly what turning it on added.
static PROTON_THEME: RefCell<Option<gtk4::CssProvider>> = const { RefCell::new(None) };
}
/// Whether the Proton theme is on.
pub(crate) fn proton_theme_active() -> bool {
PROTON_THEME.with(|slot| slot.borrow().is_some())
}
/// libadwaita's own stylesheet, bundled in the library.
const ADWAITA_STYLESHEET: &str = "/org/gnome/Adwaita/styles/gtk.css";
/// The Proton palette laid over it, from our own bundle.
const PROTON_PALETTE: &str = "/de/nils/protondrivelinux/proton-theme.css";
/// Make the theme provider's `@media (prefers-color-scheme)` and
/// `(prefers-contrast)` rules match what libadwaita shows, now and whenever it
/// changes. A provider evaluates those itself, and a new one assumes light.
///
/// The properties are set by name because they only exist since GTK 4.20, and
/// the app still runs on older GTK, whose libadwaita has no such rules anyway.
fn follow_style_manager(provider: &gtk4::CssProvider) {
fn set_enum(provider: &gtk4::CssProvider, property: &str, nick: &str) {
let Some(pspec) = provider.find_property(property) else {
return;
};
if let Some(value) = glib::EnumClass::with_type(pspec.value_type())
.and_then(|class| class.to_value_by_nick(nick))
{
provider.set_property_from_value(property, &value);
}
}
fn apply(provider: &gtk4::CssProvider, manager: &adw::StyleManager) {
let scheme = if manager.is_dark() { "dark" } else { "light" };
set_enum(provider, "prefers-color-scheme", scheme);
let contrast = if manager.is_high_contrast() {
"more"
} else {
"no-preference"
};
set_enum(provider, "prefers-contrast", contrast);
}
let manager = adw::StyleManager::default();
apply(provider, &manager);
// Weak, so a provider the preference removed isn't kept alive; its
// handlers then do nothing.
let weak = provider.downgrade();
let weak_dark = weak.clone();
manager.connect_dark_notify(move |manager| {
if let Some(provider) = weak_dark.upgrade() {
apply(&provider, manager);
}
});
manager.connect_high_contrast_notify(move |manager| {
if let Some(provider) = weak.upgrade() {
apply(&provider, manager);
}
});
}
/// Paint the window in the Proton web apps' colours (`on`) or follow the
/// system theme and accent colour.
///
/// Setting libadwaita's colour variables alone isn't enough: a GTK theme the
/// user's `gtk.css` imports (Catppuccin, Fluent, ...) sits above the app and
/// hard-codes its own accent into hundreds of rules. So the theme provider,
/// one above the user's CSS, first resets every property, then re-applies
/// libadwaita's whole stylesheet and finally the Proton palette, which leaves
/// nothing of the user's theme inside the app.
pub(crate) fn set_proton_theme(on: bool) {
let Some(display) = gtk4::gdk::Display::default() else {
return;
};
PROTON_THEME.with(|slot| {
let mut slot = slot.borrow_mut();
match (on, slot.as_ref()) {
(true, None) => {
let stylesheet = |path| {
gio::resources_lookup_data(path, gio::ResourceLookupFlags::NONE)
.map(|bytes| String::from_utf8_lossy(&bytes).into_owned())
};
let palette = stylesheet(PROTON_PALETTE).unwrap_or_default();
let css = match stylesheet(ADWAITA_STYLESHEET) {
// Its images are named relative to the sheet, which a
// string loaded here doesn't have.
Ok(adwaita) => format!(
"* {{ all: unset; }}\n{}\n{palette}",
adwaita.replace(
"url(\"assets/",
"url(\"resource:///org/gnome/Adwaita/styles/assets/"
)
),
// Still recolours everything that follows libadwaita's
// variables; only a custom theme's own rules show through.
Err(e) => {
tracing::warn!("libadwaita stylesheet not found, only recolouring: {e}");
palette
}
};
let provider = gtk4::CssProvider::new();
follow_style_manager(&provider);
provider.load_from_string(&css);
gtk4::style_context_add_provider_for_display(
&display,
&provider,
gtk4::STYLE_PROVIDER_PRIORITY_USER + 1,
);
*slot = Some(provider);
}
(false, Some(provider)) => {
gtk4::style_context_remove_provider_for_display(&display, provider);
*slot = None;
}
_ => {}
}
});
}
+229 -26
View File
@@ -1,6 +1,13 @@
# Proton Drive Linux Client: Architecture Specification
# Architecture
This document provides a deep, comprehensive architectural description of the Proton Drive Linux Client (`pdfs`). It specifies the design patterns, data flows, thread models, and subsystem dependencies that govern the virtual filesystem (FUSE), database persistence, cache management, and two-way sync engine.
This document describes how the Proton Drive Linux client (`pdfs`) is built: the crates, the
filesystem and its caches, the write queue, the sync engine, the control protocol, the thread
layout, what the client writes to disk in plaintext, and the CAPTCHA flow. It is written for
contributors and for anyone auditing the client. User-facing documentation starts at
[docs/README.md](README.md).
Numbers quoted here (thread counts, timeouts, limits) are the constants in the source at the time
of writing; the source is authoritative when they disagree.
---
@@ -35,6 +42,11 @@ graph TD
| [`pdfs-cli`](../crates/pdfs-cli) | Command Line Interface | Command routing, daemon launcher, IPC client wrapper. | Stateless; communicates with daemon over IPC control socket. |
| [`pdfs-gui`](../crates/pdfs-gui) | Graphical Interface | GTK pages, tray, and the resident quick-search prompt. | Keeps UI state only; all durable state and Drive access remain behind the IPC socket. |
The workspace ships four binaries: `pdfs` (the CLI, and the daemon as `pdfs daemon`) from
`pdfs-cli`, and `pdfs-app`, `pdfs-tray` and `pdfs-prompt` from `pdfs-gui`. The daemon runs as the
systemd user unit `proton-drive.service`. Only the daemon holds the database, the caches and the
API client; every other binary is a client of its control socket.
---
## 2. In-Memory VFS State & File Operations
@@ -105,11 +117,29 @@ A mirror folder is a plain local directory with no FUSE session, so a row can le
## 3. Read Path & Block Caching Pipeline
Read requests are parallelized and served in blocks of size `BLOCK_SIZE` (4 MiB).
Reads are served from the revision's content blocks. `Core::read_range` looks in three places
before the network:
* **Unpinned files (Streaming):** Avoids writing full files to disk to preserve storage. Instead, blocks are kept in a fixed-size `stream_ring` (in-memory ring cache) and evicted immediately.
* **Pinned files (Persistent):** Block downloads are saved directly to `ContentCache` on disk.
* **Read-Ahead:** The reader thread spawns asynchronous tasks to pre-fetch upcoming blocks.
1. **Whole-file blob.** A file kept available offline is downloaded completely into
`ContentCache` and read from there.
2. **Stream ring.** For every other file, a bounded in-memory ring (`RING_BYTES`, 128 MiB) of
recently decrypted blocks is checked first, because the kernel reads in pieces far smaller
than a block.
3. **Block cache.** Blocks are then looked up on disk, under `content/blocks/`. They count
against `cache_budget` and are evicted least-recently-used.
On a miss, only the blocks that overlap the request are fetched, through a `RevisionReader` kept
open per revision (`MAX_OPEN_READERS = 64`, validated by `(mtime, size)`).
* **Block geometry.** Block boundaries come from the revision itself. The first read of a file
assumes uniform `BLOCK_SIZE` (4 MiB) blocks; once a reader is open, the real block sizes are
recorded (`store_block_geometry`) and later reads plan on them. Proton does not guarantee
4 MiB blocks, and assuming it served wrong bytes (`docs/BUGS.md` B85).
* **Read-ahead.** A read that proves sequential prefetches a window of 2 to 8 blocks
(`PREFETCH_MIN`, `PREFETCH_MAX`). Prefetch takes a permit from a global budget of 8 and gives up
instead of queueing, so it never delays a demand read.
* **Large videos.** An unpinned video of 256 MiB or more (`STREAM_BYPASS_MIN`) streams without
persisting its blocks, so watching a film does not evict the rest of the cache.
```mermaid
sequenceDiagram
@@ -121,21 +151,25 @@ sequenceDiagram
participant API as Proton API Client
Kernel->>FUSE: read(ino, fh, offset, size)
FUSE->>FUSE: calculate block indices [first_block..last_block]
loop For each block index
alt Block in disk Cache (Pinned/Cache hit)
FUSE->>Cache: read_block(block_idx)
Cache-->>FUSE: block bytes
else Block in stream ring cache
FUSE->>Ring: get(block_idx)
Ring-->>FUSE: block bytes
else Block Cache Miss
FUSE->>API: download_range(offset, len)
API-->>FUSE: decrypted block bytes
alt caching_enabled
FUSE->>Cache: store_block(block_idx)
else streaming_only
FUSE->>Ring: insert(block_idx)
alt Whole-file blob cached (available offline)
FUSE->>Cache: read blob range
Cache-->>FUSE: bytes
else
FUSE->>FUSE: block_geometry() spans overlapping [offset, offset+size)
loop For each span
alt Span in stream ring
FUSE->>Ring: get(span)
Ring-->>FUSE: block bytes
else Span in block cache
FUSE->>Cache: read_block(span)
Cache-->>FUSE: block bytes
else Miss
FUSE->>API: RevisionReader.read_at(span)
API-->>FUSE: decrypted block bytes
opt Not a large unpinned video
FUSE->>Cache: store_block(span)
end
FUSE->>Ring: insert(span)
end
end
end
@@ -220,6 +254,35 @@ graph TD
5. **Depth-Ascending Batching:** Folders are processed first to ensure hierarchies exist before files are placed. Work is executed concurrently up to a set limit.
6. **Safety Gate and Settle:** Destructive plans are rejected when the scan is incomplete. Total-wipe protection applies to every non-empty baseline, including a single-entry folder. On success, baseline entries are upserted, timestamps updated, and any pending mode switches (e.g. going on-demand) are evaluated. Failed local deletion or conflict preservation retains the previous baseline and prevents remote content from overwriting the local source.
### Conflict Copies
A conflict keeps both versions. The local file is renamed to
`{stem} (sync-conflict {unix_secs}).{ext}` (with `-1`, `-2`, … appended to the timestamp when the
name is taken) and the remote version is downloaded under the original name. The conflict sweep
(§7) later trashes copies proven byte-identical to their sibling when `conflict_sweep` is
`enforce`.
### Devices and the Machine Profile
Synced folders live under a *device*: a per-computer root in Proton Drive. `Core::ensure_device`
decides which device this machine is: the adopted `device_uid` from `config.json` when set,
otherwise a Linux device whose name equals the hostname, otherwise a newly registered one. An
adopted UID that no longer exists is reported, never silently replaced.
Because the local database does not survive a lost machine, the daemon backs up the machine's
arrangement to `<device root>/.proton-drive-linux/profile.json` (`pdfs_core::profile`): each synced
folder's remote UID, local path and mode, the pins, and the ignore patterns. It goes up the normal
upload path and so is end-to-end encrypted like any other file. Mutating requests call
`Core::touch_profile`, which coalesces a burst of changes into one upload. `PROFILE_VERSION`
guards the format: a reader refuses a profile from a newer version instead of applying part of it.
Restore (`ListRestorableFolders`, `RestoreSyncFolders`, and the `Device` variants for another
computer) lists the device root's folders, proposes a local path from the profile when its parent
exists on this machine and `~/<name>` otherwise, and on confirmation adds a `sync_folder` row per
folder bound to the remote UID. The download is an ordinary reconcile against an empty baseline;
on-demand mode is applied through the normal mode-switch request. The user guide is
[RECOVERY.md](RECOVERY.md).
---
## 6. IPC Socket Protocol
@@ -232,7 +295,31 @@ The CLI and GUI front-ends do not access database files or make network calls di
* **Control Protocol:**
* Client sends a single JSON line (`Request`).
* Daemon parses, handles the request, and replies with a single JSON line (`Response`).
* Timeout durations are separated: **2 seconds** for writes (avoids hangs on defunct sockets) and **120 seconds** for reads (accommodates heavy transfers).
* Timeout durations are separated: **2 seconds** for writes (avoids hangs on defunct sockets) and **120 seconds** for reads (accommodates heavy transfers). Status polls use a 5-second read timeout, which is what the app reports as "Not responding".
* **Errors:** A failed request replies with `Response::Error { message, kind }`. The `kind` is machine-readable, and `pdfs --json` passes it through.
* **Compatibility:** After an upgrade, the old daemon keeps running until the service restarts, so front ends must cope with a daemon that does not know a newer request. It answers such a request with an error instead of dropping the connection. New fields are added with `#[serde(default)]`, and new variants get a separate request rather than an optional field an old daemon would ignore.
### Push Events
Since 2.5.0 the front ends do not poll. The app and the tray each send `Request::Subscribe` once
and keep the connection open; the daemon then streams one `Event` per line:
| Event | Sent when | Payload |
|---|---|---|
| `Status` | On subscribe, and whenever any status field changes | A full `Response::Status` |
| `Transfers` | On subscribe, and whenever the transfer snapshot changes | Transfers and background jobs |
| `Changed { topics }` | Something a page shows changed | `Topic`s to re-read: `files`, `photos`, `trash`, `shares`, `devices`, `locations`, `queue`, `conflicts`, `activity` |
| `Heartbeat` | Every 10 seconds of silence | — |
Code that changes state publishes its topics to the `EventHub` (`crates/pdfs-fuse/src/events.rs`);
the control handler maps each mutating request to the topics it touches. Changes within 150 ms
of each other are coalesced into one event. A new subscription is told every topic changed, so it
catches up on anything it missed. At most 8 subscriptions are served at once.
On the client side, `pdfs_core::control::follow` reconnects across daemon restarts with backoff
and treats 25 seconds without a line as a stuck daemon. A daemon older than 2.5.0 answers
`Subscribe` with an error; `follow` then falls back to polling status and transfers, so the front
end sees the same events, only later.
### Unified Search
@@ -246,12 +333,18 @@ The prompt is a single-instance GTK application. It retains and hides its window
The background daemon relies on the following thread topology:
1. **Main Thread / Dispatch Loop:** Blocks on `fuser::Session` loop. Reads kernel FUSE events and hands off network-bound VFS work to the FUSE workers pool.
1. **Main Thread / Dispatch Loop:** Blocks on `fuser::Session` loop. Reads kernel FUSE events and hands off network-bound VFS work to the FUSE workers pool. Each on-demand synced folder is a further FUSE session with its own dispatch loop and its own inode space (`State`).
2. **FUSE Workers Pool (11 threads, two lanes):** Bounded thread pool handling network operations. Split into 3 threads reserved for metadata (`lookup`, `readdir`) and 8 general threads that serve transfers (block reads) and fall back to metadata when no transfer is waiting. Reserved threads never accept a transfer — that is what keeps a directory listing from queuing behind saturated downloads (audit A6).
3. **IPC listener Thread:** Listens on the Unix socket and admits at most 64 concurrent, timeout-bound handlers for front-end status/configuration requests.
4. **Sync Engine Loop Thread:** Serializes sync runs. Wakes on debounced local inotify filesystem changes, remote polling intervals, or manual user requests.
5. **Drain Queue Worker Thread:** Processes staged writes (`PendingOp`) sequentially, uploading revisions and retrying with exponential backoff on failures.
3. **Control Threads (`pdfs-control`):** Listen on the Unix socket and admit at most 64 concurrent, timeout-bound handlers. Long-lived subscriptions hold one handler each, up to 8 (see §6).
4. **Sync Engine Threads (`pdfs-sync`, `pdfs-sync-poll`):** Serialize sync passes. Wake on debounced inotify changes, the periodic poll, remote events, or a user request.
5. **Drain Workers (`pdfs-drain-0` … `pdfs-drain-2`):** `DRAIN_WORKERS = 3` threads share the queue of staged writes and changes (`PendingOp`), with per-node ordering (§4), and retry failures with exponential backoff. Worker 0 also runs the queue's idle chores.
6. **Conflict Sweep Thread (`pdfs-conflict-sweep`, optional):** Reconciles leftover `(sync-conflict …)` copies — 30 s warmup, then one pass every 5 minutes. A copy proven identical to its live sibling (equal size **and** equal `content_sha1`) is redundant; anything it cannot prove identical is surfaced to the activity feed and left alone. Governed by `AppConfig.conflict_sweep` / `PDFS_CONFLICT_SWEEP` (`SweepMode`): **report-only by default**, `off` skips the thread entirely, and only `enforce` lets it trash. Because it deletes, the enforcing path re-verifies revision id, size, digest, name, parent, queued ops and open handles immediately before acting rather than trusting its own pass-start snapshot. See `docs/BUGS.md` B69 and B71.
7. **Remote Event Tasks (Tokio):** `run_event_sync` follows Proton's event stream for the Drive volume and applies each change to every mounted inode space, invalidating kernel entries as it goes. The cursor is persisted after every batch, so changes made while the daemon was stopped are applied on the next start. `run_photos_event_sync` follows the photos volume separately; it only drops gallery rows or the timeline's freshness stamp, since photos are not part of the mount.
8. **Event Sampler (`pdfs-events`):** Publishes status, transfer and sync-progress changes to subscribed front-ends (§6).
9. **Supervisor (`pdfs-supervisor`):** Logs workers that hold one job for minutes, a queue that stops draining and control requests that never return; samples memory; and pings the systemd watchdog only after a real round trip over the control socket, so a hung daemon is restarted (`WatchdogSec=120`). `pdfs diagnostics` reports the same data on demand.
10. **Smaller helpers:** `pdfs-online-probe` (connectivity), `pdfs-localindex` (home-directory index for the search prompt), `pdfs-pause` (ends a timed pause), `pdfs-conflict-sweep`, and `pdfs-similar` (duplicate-photo detection, on request).
Every long-lived loop waits on one shared `Shutdown` signal instead of sleeping, so an in-process remount joins the old generation of threads instead of leaking it (`docs/BUGS.md` B44).
---
@@ -377,3 +470,113 @@ sequenceDiagram
This ensures compatibility with all versions of Proton's client verification scripts.
3. **SRP Handshake Reset:** Because a gated login burns the SRP handshake on the API side, the client cannot simply resume the previous request. Instead, `auth::login_interactive` restarts the SRP process from scratch with the verification credentials attached, keeping the complex handshake details isolated from the front-end.
4. **CLI Fallback:** Since the CLI has no native web browser engine, hitting the CAPTCHA gate fails immediately with a user-friendly message directing the user to sign in once via the GUI (`pdfs-app`) to persist the authenticated session keys to the system keyring.
---
## 10. Feature Design Notes
Notes on individual features whose code is shaped by a constraint that is not obvious from the
code alone.
### File Version History
**Files**: [`revisions.rs`](../crates/pdfs-fuse/src/revisions.rs), [`control.rs`](../crates/pdfs-core/src/control.rs), [`versions_dialog.rs`](../crates/pdfs-gui/src/app/widgets/versions_dialog.rs)
Proton Drive keeps every revision a client committed; the daemon only ever addressed the active one, so a file overwritten by a sync pass could be recovered only from whatever the local `recovery/` directory happened to hold.
The control protocol gained `ListRevisions` / `RestoreRevision` / `DeleteRevision` / `SaveRevisionAs` (each with a `…ByUid` twin for nodes the primary mount cannot name), the CLI gained `pdfs versions list|restore|save|rm`, and the browser's details pane gained a **Versions** button opening a per-file dialog.
Three properties that shape the code:
- **A restore is server-side and asynchronous.** No content crosses the wire and nothing enters the drain queue; the server answers 202 and swaps the active revision in the background, so the daemon evicts the file's cached blocks and open readers rather than describing the new state, and the UI wording promises the request, not the result.
- **The active revision cannot be deleted.** `Core::delete_revision_for_uid` checks that locally so the user gets a sentence rather than an API code, and the dialog gives the current row no delete button.
- **Saving a version never overwrites.** `SaveRevisionAs` refuses an existing destination and removes a partial file if the download fails — a half-written export looks identical to a good one to every tool that opens it afterwards.
### Photo Favorites
**Files**: [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs)
The gallery can mark and filter favorites, through the SDK's photo-tag API (`update_photos`). Schema **v20** adds `photos.favorite`; the flag is learned in the timeline enrichment pass that already resolves each photo's name and media type, and is kept across a refresh that could not resolve a photo — the same learned-and-kept rule as `media_type`. The lightbox carries a star toggle, the Photos header a favorites filter, and the CLI has `pdfs favorite <uid> [--remove]` plus `pdfs photos --favorites`.
Favouriting a photo that is not on this account's own photos volume (shared with us, or album-only) needs it re-encrypted for our timeline root, which the SDK does not implement; the daemon surfaces that as an error rather than silently doing nothing.
### Justified Gallery, Gallery Delete
**Files**: [`photos.rs`](../crates/pdfs-gui/src/app/pages/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs), [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`background.rs`](../crates/pdfs-fuse/src/background.rs)
The gallery lays each day out in justified rows (`justify_rows` / `plan_rows` / `fit_row`): photos are taken in capture order until their summed aspect ratio no longer fits the target row height, and the row is then scaled so it ends exactly on the content width. A tile is therefore its own photo's shape, and `ContentFit::Contain` has nothing left to crop. The last row of a day is left at the target height rather than stretched, because a day holding two photos is a short day, not a layout fault.
- **Ratios that are not known yet.** `PhotoItem::ratio` is persisted, but a photo that has never been decoded has none. Those are laid out square and remembered in `assumed_ratios`; when a decode proves the real shape the day re-flows on the existing debounce timer, so a screenful of decodes costs one re-flow rather than one per photo. Ratios are clamped to 0.4–3.0 so a single panorama cannot decide what a row looks like.
- **Thumbnails survive recycling.** A reply for a tile that has already been re-bound still lands in the texture cache, which is an LRU of 1500 textures keyed by uid, and the rows just outside the realised range are prefetched in the scroll direction.
- **Delete is uid-addressed.** The photos volume is not in the FUSE mount, so the path-based `Request::Delete` cannot reach it. `Request::TrashNodes { uids }` trashes through the SDK and answers `Response::Trashed { trashed, failed }`; the daemon then drops those rows with `Db::photos_delete` (album membership included) so the gallery does not wait for a timeline refresh. The GUI removes the tiles optimistically with an Undo toast and puts back anything the server refused.
- **Remote deletions arrive on their own.** The Drive volume and the photos volume have separate event streams, so `run_photos_event_sync` polls the photos volume with its own cursor (`photos_event_cursor` in the state table). Trash and delete events remove rows; anything else only invalidates freshness, because there is no inode space to converge on this volume. Together with a 60 s `TIMELINE_TTL` and a `RefreshScope::Photos` that now awaits the refresh, a photo deleted on a phone leaves the grid in about ten seconds.
### RAW + JPEG Grouping
**Files**: [`photos.rs`](../crates/pdfs-core/src/db/photos.rs), [`migrations.rs`](../crates/pdfs-core/src/db/migrations.rs), [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs)
Schema **v30** adds `content_hash`, `main_uid` and an indexed `group_key` to `photos`. The first two come from the server's `PhotoProperties` and are filled by the enrichment pass in `refresh_timeline` that already resolves each photo's name, media type and favorite tag; `group_key` is computed in `photos_replace`, where the whole timeline is in hand.
- **Precedence.** The server relation first (`main_photo_uid` and `related_photo_uids`, read from whichever end resolves — the two can land in different enrichment chunks); then same capture day, same case-insensitive name stem, and one member raw while the other is not; otherwise the photo is its own group. The union-find pass is over the timeline in memory, so it costs one pass per refresh rather than a query per photo.
- **Representative.** The non-raw member when there is one — a JPEG decodes in milliseconds and is what the person expects to see. `group_key` holds that member's uid, so "is this the tile" is `uid = group_key`, an index scan.
- **What is grouped and what is not.** `photos_page`, `photos_months` and `photos_counts` count groups, except on the Raw tab, which lists files. Album pages are never grouped: an album is a list someone made. The relation is learned-and-kept like `media_type`, so a refresh that could not resolve a node does not break a group up.
- **Deleting.** `Core::trash_photos` expands each uid to its group before trashing, because the grid shows the shot as one tile — leaving the RAW behind would put the photo back on the Raw tab and nowhere else. The GUI's confirmation names the file count.
- **The other files.** `Request::PhotoGroup { uid }` answers with the group's members as ordinary `PhotoItem`s, which is what the lightbox's switch steps through.
### Persistent SDK Entity Cache
**Files**: [`sdkcache.rs`](../crates/pdfs-core/src/sdkcache.rs), [`auth.rs`](../crates/pdfs-core/src/auth.rs), [`background.rs`](../crates/pdfs-fuse/src/background.rs)
The SDK's Drive entity cache (decrypted node metadata: name, size, parent, signing share) defaulted to memory, so every daemon restart re-fetched and re-decrypted the tree the previous run had already walked. It is now backed by SQLite.
- **Its own file** (`sdk_cache.db` in the state directory), not `cache.db`: the daemon's `Db` is one `Mutex<Connection>` shared by every FUSE thread and the control socket, and this traffic is frequent, small and entirely reconstructible. A separate file also means no schema migration — the store can be deleted at any time.
- **Encrypted at rest** by wrapping it in the SDK's `EncryptedCacheRepository`, keyed by the mailbox password. A password change reads as a cold cache (the SDK treats an undecryptable entry as a miss and clears the store), not as an error.
- **Staleness** is closed by the event loop, which already replays from a persisted cursor and calls `invalidate_caches_for_event` for every event — including those raised while the daemon was down. The one case with no trail is a *seeded* cursor (first-ever mount, or a lost cursor), where the seed path now clears the store instead of trusting it. `auth::logout` removes the file.
Needed a small additive SDK change (`ProtonDriveClient::with_entity_repository`, 0.5.1): `with_entity_cache` is a constructor and so could not be combined with `with_key_salts`, which the daemon requires.
### Per-node Batch Outcomes
**File**: [`batch.rs`](../crates/pdfs-core/src/batch.rs)
SDK 0.5.0 made `trash_nodes` / `restore_nodes` / `delete_nodes` report one outcome per node instead of failing the whole call (mirroring upstream's streamed `NodeActionResult`). `pdfs_core::batch::into_unit` collapses the single-node calls back to a `Result`; `batch::split` handles a collected batch.
A restore is expanded before it is sent (`expand_restore`, `pdfs-fuse/src/lib.rs`): the persisted trash listing carries each row's `parent_uid` (schema 29), so a restore takes the connected piece of the trashed tree — every trashed descendant of what was asked for, and every trashed ancestor above it — and sends it shallowest wave first, because the server cannot put a node back under a parent that is still trashed. A uid the listing does not know about (it is materialised in chunks, and can be stale) is restored on its own rather than dropped.
The trash view's restore and permanent-delete use the SDK's **streaming** variants (`restore_nodes_streaming` / `delete_nodes_streaming`) and apply local state per node as each batch lands rather than after the last one. For a permanent delete — which is irreversible — that means a daemon interrupted mid-batch has forgotten exactly the nodes the server destroyed, no more and no fewer. Both report per-node failures and count only what actually succeeded; only a batch where nothing succeeded is an error.
### Moves Between Locations
**Files**: [`relocate.rs`](../crates/pdfs-fuse/src/relocate.rs), [`control.rs`](../crates/pdfs-fuse/src/control.rs), [`browser.rs`](../crates/pdfs-gui/src/app/pages/browser.rs)
The primary mount and every on-demand folder are separate FUSE sessions, and a mirror folder has no session at all. The kernel answers `rename(2)` between two mounts with `EXDEV` before the daemon hears of it, so `mv` copies and deletes. On Proton Drive the same move is one `move_node`, because My files and this device's folders are on the main volume.
`Request::Move` with two absolute paths is therefore resolved by `Core::move_between` against every location: a live mount through `rooted_at`, otherwise the mirror folder with the longest `local_path` prefix, whose uids come from the `sync_entry` baseline. Two paths in one inode space still go to `move_to`. Relative paths keep their old meaning.
- **Nothing that exists only on this disk may be lost.** A mounted source is refused while it or anything under it has a queued op or a file open for writing. A mirror source is refused unless every file under it matches its baseline and every baseline row still has its file (`mirror_subtree_unsynced`); ignored names and symlinks count as unsynced, because the local copy is removed after the move.
- **Mirror to mirror renames along.** The local copy is renamed first, with `RENAME_NOREPLACE` so nothing that appeared at the destination meanwhile is overwritten, and its baseline rows are moved with `sync_entries_move`, so neither folder's next pass has anything to do. If `move_node` fails, the rename is undone. `EXDEV` between two filesystems falls back to the check above, then drops the source copy.
- **A dropped mirror copy is checked twice.** The check runs again just before the source copy is deleted. A file written into it during the move keeps the whole copy; its baseline is gone, so the next pass uploads it as new.
- **Sync passes are held off.** Every mirror folder involved is locked through `sync_lock`, by ascending id, with a five-second bound. A folder that is still busy is reported rather than waited on.
- **The local side is levelled afterwards.** A mounted source forgets the node in every inode space and notifies the kernel. A mounted destination has its listing cleared. A mirror destination that received nothing locally gets a reconcile, which downloads the node. That download first asks `ContentCache::copy_cached_to` for the file: a whole cached blob, or every block of it, is copied into place instead of fetched, so a move out of an on-demand folder costs no traffic for what was already opened there.
### Open-for-Write Deferral for Mirror Sync
**File**: [`sync.rs`](../crates/pdfs-fuse/src/sync.rs)
The mirror folder sync engine now defers uploading any file that is currently held open for writing by another process, matching the guarantee the FUSE mount path already provides (where uploads are deferred until `close(fd)`).
**Problem**: Previously, the mirror sync path relied solely on a 2-second trailing-edge debounce (with a 30-second ceiling) to avoid uploading files mid-write. This was insufficient for slow continuous writers (e.g., database dumps, large exports, or editors that keep files open for extended periods), which could have their incomplete state uploaded as a real revision.
**Solution**: Before each reconcile pass, the sync engine scans `/proc/*/fd` once to build a set of canonical paths within the sync root that any process holds open for writing (`O_WRONLY` or `O_RDWR`). Files in this set are:
- **Kept in the local walk** — so they are not misclassified as deletions
- **Treated as unchanged** — so no upload is queued for them
- **Counted as `deferred`** — so the activity summary reports them (e.g., "3 uploaded, 1 deferred (open for write)")
- **Picked up on the next pass** — after the writer has closed the file
**How it works**:
1. `open_for_write_set(root)` reads `/proc/*/fd` → `readlink` for each fd → prefix-checks against the sync root → reads `/proc/<pid>/fdinfo/<n>` to check the `flags:` line's low two bits (O_ACCMODE)
2. `walk_local` sets `LocalItem.open_for_write = true` for matching files
3. Both `do_reconcile` and `push_pass` skip upload classification for flagged files
4. `Outcome.deferred` tracks the count for the activity summary
**Performance**: The `/proc` scan completes in low single-digit milliseconds on a typical desktop. Only fds whose `readlink` target falls under the sync root incur the `fdinfo` read.
**Tests added**: 8 unit tests covering `is_write_mode` parsing of various fdinfo flag combinations, and `Outcome` summary formatting with deferred counts.
| Sync Path | Open-for-write protection | Mechanism |
|---|---|---|
| FUSE mount | Yes | Upload deferred until `close(fd)` |
| Mirror folders | Yes | `/proc/*/fd` scan per reconcile pass |
+1090 -1
View File
File diff suppressed because it is too large. Load diff
+280 -3
View File
@@ -11,6 +11,273 @@ scratch (user data in `staging/` and `recovery/` is never touched by this).
## [Unreleased]
## [2.10.0] — 2026-10-03
Schema: **36**. Every node row gets a local id that never changes; existing rows keep their
row number as that id. Going back to 2.9 means rebuilding the cache. Requires `proton-drive-rs`
**0.7.2**.
### Added
- **Migration tests on real databases.** A database 2.8.2 wrote, with queued changes due,
parked, failing, waiting for access and in flight, migrates with every change intact.
`PDFS_MIGRATE_DB=<path> cargo test -p pdfs-core a_copy_of_a_real_database -- --ignored` runs
the same check on a read-only copy of any database.
### Changed
- **A file keeps its inode number across daemon restarts.** Inodes are now the node's local id
instead of a counter that started over on every run.
- **Every mount serves from one tree.** An on-demand sync folder used to keep its own copy of the
nodes it showed, which background work had to visit one by one and could miss (B74). It now
shares the tree with My files; only the root it starts at is its own.
### Fixed
- **A file renamed or moved while its first upload is on the wire keeps its new name and folder.**
It landed under the old name and the rename was lost (B136).
## [2.9.0] — 2026-10-03
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.2**.
### Added
- **Seeded simulation runs.** `cargo test -p pdfs-fuse --lib sim:: -- --ignored
--test-threads=1` mounts one or more real FUSE daemons against an in-memory Drive that lags,
drops, reorders and echoes like the real one. Each seed draws syscalls, link flaps and restarts,
checks every answer against a POSIX model, and at the end checks that nothing was lost and the
clients converged. CI runs them on pushes to main. They found B122 to B135.
- **More acceptance cases.** Regressions for B48, B80, B86, B88, B94, B95, B98, B102, B111, B113
and B114 to B121, and a check that `pdfs` output into a closed pipe stays quiet. The console
report has one line per case and a summary table, and `--journal-check` also fails on a
conflict copy, a FUSE job held for long, or an authority re-read.
### Fixed
- **A new file moved or renamed before its first upload no longer lands as a conflict copy**
next to its own empty original (B113).
- **`pdfs rm`, `pdfs rename` and `pdfs mkdir` take paths in on-demand folders.** They answered
"is not under the mountpoint" for a path `pdfs move` took. `rm` and `rename` refuse a path that
names a mount's root (B88).
- **`pdfs ls | head` no longer panics** with "Broken pipe". A closed stdout exits 0; any other
write error exits 1.
- **A file removed, renamed or moved with `pdfs` stops resolving under its old name at once**,
instead of for up to 30 seconds (B114).
- **`pdfs ls` and the app's file browser show a file's real size**, not its encrypted one
(B115).
- **Trashing a folder also drops the creates queued in its subfolders.** One could stay queued
with its bytes, or land in the root (B116).
- **An upload queued for a file `pdfs move` moved keeps its path** in `pdfs sync queue` and is
no longer put off once after the resume (B117).
- **Creating a file or folder through the mount no longer fails with `ENOENT`** when Drive is
slow to list what it made. The mount asks again, and queues the create if it still cannot see
it (B118).
- **A file just written can be read without the network.** Its uploaded bytes stay cached, and
`pdfs status` counts the upload as pending until they are (B119).
- **A file queued inside a folder that just landed is no longer put off** for five seconds
(B120).
- **A move right after a rename lands.** It failed with `Input/output error` through the mount
and with "out of date" from `pdfs move`, and a rename across folders took up to 10 seconds
(B111, B121).
- **A rename across folders no longer fails with `Input/output error`** when the source folder
holds the new name (B122).
- **A second write open on a file just written no longer fails** with `Input/output error`
(B123).
- **After a replacing rename fails, the file it was replacing stays in the mount** (B124).
- **A write to a file whose create could not be read back no longer lands as a conflict copy**
(B131).
- **A file just moved is found by its new path.** A lookup right after the rename could miss it
(B135).
### Notes
- The simulation runs found bugs that are still open: B125 to B130 and B132 to B134 in
`docs/BUGS.md`. The worst are B127 and B128 (a file created offline can land twice, or be lost
when created over one deleted offline), B129 (a file deleted while its create is on the wire
stays on Drive) and B130 (a read that a remote change invalidates can hang the daemon).
## [2.8.2] — 2026-10-01
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.2**.
### Fixed
- **Folders shared with you no longer go missing when one person shared several.** When
several accepted shares came from the same owner's volume, only the first appeared under
Shared with me, and the rest were skipped with `missing key`. `proton-drive-rs` 0.7.2 resolves
each shared root with its own key (B112, #25).
## [2.8.1] — 2026-09-30
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.1**.
### Fixed
- **The service no longer starts before the desktop session.** `proton-drive.service` was
wanted by `default.target`, which is reached before the desktop hands `DISPLAY` /
`WAYLAND_DISPLAY` to the user's systemd. The first keyring call then started the Secret
Service without a display; on KDE `ksecretd` crashed at every login and the first start of the
service failed. The unit is now wanted by `graphical-session.target`. Existing installs switch
over on the next sign-in or "Start on login" toggle, or with
`systemctl --user reenable proton-drive.service` (#23).
## [2.8.0] — 2026-09-30
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.1**.
### Added
- **`pdfs trash --json`** prints the trash listing as JSON.
- **The FUSE acceptance suite tests the signed-in account by default.**
`scripts/fuse-acceptance.sh` with no arguments creates test folders in My files and new
on-demand and mirror sync folders, runs every case in each, and removes all of it afterwards,
also on a failure or interrupt. A run that dies is cleaned up by the next one. New cases cover
block boundaries, unusual names, a deep tree, a wide directory, rename patterns, handle coherency,
throughput and B100. `--offline-only` runs the account-free suite as before.
### Changed
- **The mount keeps working through a network drop.** When Drive stops answering, the daemon
now marks itself offline within about 20 seconds instead of failing every call for up to two
minutes. While offline, creating, renaming and deleting files and folders is queued instead of
failing with `Input/output error`, folders list from the local cache, and a read that needs
data from Drive waits up to 30 seconds for the connection. The daemon checks every few seconds
for the connection to return, then uploads the queued changes. A file whose creation timed
out but reached Drive anyway is taken over rather than uploaded again as a conflict copy.
- **A dead connection is noticed sooner.** `proton-drive-rs` 0.7.1 sends TCP keepalives, so a
connection that died silently fails instead of waiting for its request timeout.
### Fixed
- **Truncating and then writing past the end no longer brings old bytes back.** The gap
between the new end and the write now reads as zeros, as it should, instead of the file's
earlier content. When that earlier content had not been uploaded yet, the file failed with
`Input/output error` and its upload retried forever (B101).
- **A write made while a new file is first uploading is no longer lost.** Drive could keep an
empty or older copy of a file written right after it was created, and closing a file held open
across its first upload could fail (B101).
- **Removing a folder also cancels the pending uploads of the files inside it.** Before, an
edit that had not reached Drive yet kept retrying against the trash forever and kept its
staged copy on disk. `pdfs rm` now also drops what was queued for what it removes (B102).
- **A program reading a file keeps its data when another program replaces the file.** Saving
over a file by renaming a new copy onto it (what most editors and `os.replace` do) made
reads through handles already open on the old file fail with `Input/output error` (B104).
- **Short-lived files no longer come back as conflict copies.** A file deleted right after it
was written, such as a SQLite journal, could be uploaded as a `recovered-…` conflict copy in
the Drive root (B105).
- **Editors can save over a file several times in a row.** The fix for B104 could leave the
upload of a replaced file queued, and the next save onto the same name failed with
`Input/output error` (B106).
- **Renaming a file back right after renaming it no longer fails.** Drive can briefly call the
second rename out of date, and the daemon now waits it out instead of returning an I/O error
(B111).
- **Reading a freshly listed file no longer hangs.** When a read had to fetch the file's real
size first, the daemon could wait on the read it was serving, and the program froze until
the daemon restarted (B110).
- **Databases and other files kept open while written no longer read back empty.** When the
folder was listed again while a file was still open, its size dropped to what Drive held,
often 0 bytes, and SQLite reported `database disk image is malformed` (B109).
- **Deleting files during a mirror pass no longer marks the folder as failed.** A file removed
after the pass planned its upload was counted as an error, so the folder showed `error` until
a later pass (B108).
- **A read no longer hangs when the connection stalls.** A read that gets no answer from Drive
within two minutes now fails with `Input/output error` instead of blocking the program for
as long as half an hour (B107).
- **The trash no longer freezes after a server call hangs.** Until the daemon restarted, the
trash kept showing the same entries, and a folder just deleted could not be found there to
delete permanently (B103).
- **Freshly listed files read correctly.** A file whose real size had not been fetched yet
reported its encrypted size, about 53 bytes too big, and reading it failed with
`Input/output error`. This happened most often right after copying a large folder. Reads and
write opens now fetch the real size first, and a big folder walk no longer skips size lookups
(B100).
## [2.7.0] — 2026-09-29
Schema: **35** (unchanged).
### Added
- **Moves between locations without transferring content.** `pdfs move` and the app's
**Move to…** now take a file or folder from My files into a synced folder, or between two
synced folders, on-demand or mirrored. The move happens on Proton Drive, so nothing is
downloaded or uploaded again. `mv` between two locations still copies, because the kernel
refuses a rename between two mounts. The app's Move dialog lists every location at its top
level. A move is refused, with the reason, when the source still has changes that are not on
Proton Drive, when the two locations are on different volumes, or when a synced folder stays
busy with a sync pass.
- **`pdfs move` takes several sources.** `pdfs move a b c DEST` moves all three and
reports each one that could not move.
- **Moving into a mirrored folder reuses cached content.** A file already cached by
an online-only folder is copied into place instead of downloaded again.
### Changed
- **New project homepage: [proton-drive.narl.io](https://proton-drive.narl.io).** The About
dialog's website link, the README, and the package metadata (Cargo, PKGBUILDs, `.deb`,
`.rpm`) point there. The source code link stays on GitHub.
## [2.6.2] — 2026-09-28
Schema: **35** (unchanged).
### Fixed
- **Photos remembers the Albums and Places views.** Leaving Photos from Albums or Places and
coming back showed the timeline while the Albums or Places toggle stayed selected. The page
now reopens on the selected view, and Places keeps its Grid or Map layout.
- **The Sync page keeps a steady width.** A long file name in Transfers no longer widens the
page. Overview and Folders now fill the width next to the sidebar.
## [2.6.1] — 2026-09-28
Schema: **35** (unchanged).
### Added
- **`pdfs-prompt --preload`.** Starts the launcher hidden, for session autostart such as
Hyprland's `exec-once`, so the first shortcut opens it as fast as later ones. It does nothing
when the launcher is already running.
### Fixed
- **Launcher invocations no longer pile up.** Each shortcut press after the first used to leave
a `pdfs-prompt` process behind after handing the request to the running launcher. It now exits.
## [2.6.0] — 2026-09-28
The search launcher (`pdfs-prompt`) is rebuilt around a preview pane, and it now looks like the
app it belongs to.
Schema: **35** (unchanged).
### Added
- **Launcher preview pane.** The selected result shows a large thumbnail, its type, size,
modified date, location and whether it is on this computer, with Open (Enter), Show in Folder
(Ctrl+Enter) and Copy Path (Ctrl+C, or Ctrl+Shift+C while text is selected in the search box).
Show in Folder selects the file in the file manager through `org.freedesktop.FileManager1`
and falls back to opening the folder.
- **Recent files.** With an empty search box, the launcher lists the last files opened from it
above the pinned ones. Files opened often and lately also rank higher in search results. The
history lives in `prompt-history.json` in the state directory and forgets local files that
no longer exist.
- **Thumbnails in the launcher.** Images show a thumbnail in the result list and the preview,
from the desktop's thumbnail cache, decoded locally (and written back to that cache), or
fetched by the daemon for Drive files.
- **Drag and drop from the launcher.** A result can be dragged into a file manager or another
application as a file.
### Changed
- **The launcher follows the app's theme.** It uses the system accent colour, light and dark
style, and the opt-in Proton theme, instead of always pinning Proton purple. Changes to the
theme or to `open_with` apply the next time the launcher opens.
- **Launcher layout.** Results are grouped into Recent, Pinned in Proton Drive and Best matches;
matched characters are bold, and each row shows its sync state, size and age. A click selects
a row and a double-click opens it; the search box keeps the keyboard focus throughout.
- **Faster search as you type.** The launcher searches 80 ms after the last keystroke instead
of 120 ms.
### Fixed
- **The launcher no longer freezes on a stuck mount.** Checking whether a Drive file is in the
mount runs off the main thread and gives up after 2 seconds, falling back to the daemon.
Escape cancels a Drive file that is still opening.
### Documentation
- **Rewritten user documentation.** New pages for installation, the user guide, the command-line
reference, configuration and troubleshooting, and a documentation index in `docs/README.md`.
The README is shorter and points to them.
- **Recovery** describes the current restore flow (adopting a computer, `pdfs sync restore`, the
machine profile) instead of the old manual runbook.
- **Architecture** covers push events, remote event streams, the revision block geometry, the
thread layout of 2.5.0 and the machine profile. The development page is now a developer guide,
and open work moved to `docs/ROADMAP.md`.
- `scripts/publish-wiki.sh` publishes the user pages to the GitHub wiki.
## [2.5.0] — 2026-09-28
A polish pass over the desktop app: the daemon pushes changes instead of being polled, pages
@@ -320,7 +587,7 @@ Schema: **31** (unchanged).
opened in between.
- **Photo viewer rewrite.** The lightbox's controls fade out after two seconds without pointer
movement. Photos zoom with Ctrl+scroll, a pinch, `+`/`-`, `1` for 100 % and double-click, and
pan by dragging; `0` fits them again. The details panel docks beside the photo instead of
pan by dragging; `0` fits them again. The details panel docks beside the photo instead of 120 ms
covering it. Stepping past the last loaded photo loads the next page, and the counter shows the
library's total rather than the loaded count. A video plays inline. Save a Copy uses the photo's
real name and reports failures, and Open With… offers the app chooser.
@@ -658,7 +925,7 @@ thumbnail.
slider for the icon grid, and account storage usage. Zoom resizes the realised grid cells rather
than rebuilding the model, so dragging the slider does not tear down the selection; the default
size reproduces the previous fixed 72 px tiles. Capacity stays out of the bar until a real
reading lands, and the quota fetch (60 s TTL) now also runs while Files is on screen instead of
reading lands, and the quota fetch (60 s TTL) now also runs while Files is on screen instead of 120 ms
only in Settings.
- Repository housekeeping: `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, `SECURITY.md`, issue and pull
request templates, a project logo, and workspace crate metadata (`repository`, `homepage`,
@@ -1177,8 +1444,18 @@ First stable release: FUSE files-on-demand mount, sync daemon under `proton-driv
- The outstanding FUSE defects tracked in `docs/BUGS.md`, plus a truncate defect, validated
by a new POSIX compliance suite for the filesystem.
[2.4.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.4.0
[2.10.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.10.0
[2.9.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.9.0
[2.8.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.2
[2.8.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.1
[2.8.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.0
[2.7.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.7.0
[2.6.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.6.2
[2.6.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.6.1
[2.6.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.6.0
[2.5.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.5.0
[2.4.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.4.0
[2.3.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.3.0
[2.2.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.2.2
[2.2.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.2.1
+192
View File
@@ -0,0 +1,192 @@
# Command-line reference
`pdfs` controls the Proton Drive service from a terminal and from scripts. Almost every command
talks to the running service (`proton-drive.service`); `login`, `logout`, `mount`, `daemon` and
`diagnose` work without it.
`pdfs <command> --help` prints the full help of any command. This page groups them by task.
- [Conventions](#conventions)
- [Account and service](#account-and-service)
- [Files](#files)
- [Available offline](#available-offline)
- [Synced folders](#synced-folders)
- [Conflicts](#conflicts)
- [Computers](#computers)
- [Trash](#trash)
- [Versions](#versions)
- [Sharing](#sharing)
- [Photos](#photos)
- [Activity and transfers](#activity-and-transfers)
- [Maintenance and diagnostics](#maintenance-and-diagnostics)
- [Scripting with `--json`](#scripting-with---json)
## Conventions
- **Paths** name items in My files. They may be absolute paths inside the Proton Drive folder
(`~/ProtonDrive/Documents/a.pdf`) or relative to it (`Documents/a.pdf`).
- **UIDs** identify items that have no path in the mount: photos, trashed items, shared items and
computers. The listing commands print them.
- **Exit codes**: `0` on success, non-zero on any failure, including an error reported by the
service.
- **Output** is English and stable enough for `grep`; use `--json` where it is available.
## Account and service
| Command | Description |
|---|---|
| `pdfs login [-u EMAIL]` | Sign in (password, then a 2FA code if the account has one) and store the session in the system keyring. Restarts the service if it is running. |
| `pdfs logout` | Remove the stored session. |
| `pdfs status` | Account, mount, queue and cache state. |
| `pdfs quota` | Storage used across all Proton products. |
| `pdfs locations` | Every local place the service occupies: the Proton Drive folder and each synced folder, with mode and state. |
| `pdfs daemon [MOUNTPOINT]` | Run the service in the foreground: wait for a session, mount, stay mounted. `proton-drive.service` runs this. |
| `pdfs mount [MOUNTPOINT]` | Mount once and block until unmounted. Defaults to `~/ProtonDrive`. |
`pdfs login` cannot show a CAPTCHA. If Proton asks for one, sign in with `pdfs-app` instead.
## Files
| Command | Description |
|---|---|
| `pdfs ls [PATH]` | List a folder. |
| `pdfs search QUERY [--in FOLDER] [--limit N]` | Search names in the local index. |
| `pdfs mkdir PARENT NAME` | Create a folder. |
| `pdfs rename PATH NEW_NAME` | Rename a file or folder. |
| `pdfs move PATH... NEW_PARENT` | Move one or more files or folders into another folder. The two may be in different locations (My files, an on-demand or a mirrored synced folder); the move happens on Proton Drive, without downloading or uploading anything. |
| `pdfs rm PATH` | Move to the Trash. |
| `pdfs upload SOURCES... [-t FOLDER]` | Upload local files and folders (recursively) in the background. Follow with `pdfs transfers`. |
| `pdfs refresh [TARGET] [--full]` | Forget a cached listing so the next read fetches it again. `TARGET` is a folder, `trash` or `photos`; `--full` with `photos` re-reads every photo's metadata. |
Ordinary file tools (`cp`, `mv`, `rsync`, editors) work on the Proton Drive folder directly. These
commands are for scripts and for cases where the mount is not available.
## Available offline
| Command | Description |
|---|---|
| `pdfs pin PATH` | Download a file completely and keep it on this computer. |
| `pdfs unpin PATH` | Release it back to online only. |
| `pdfs pins` | List files kept available offline. |
## Synced folders
| Command | Description |
|---|---|
| `pdfs sync add PATH` | Back up a local folder under this computer and keep it in sync. |
| `pdfs sync list` | List synced folders with their IDs and state. |
| `pdfs sync rm ID [--delete-remote]` | Stop syncing. Local files stay; `--delete-remote` also removes the copy on Proton Drive. |
| `pdfs sync mode ID mirror\|ondemand` | Switch between a full local copy (`mirror`) and online only (`ondemand`). |
| `pdfs sync now [ID]` | Start a sync pass now, for all folders or one. |
| `pdfs sync pause [--for 1h] [--folder ID]` | Pause uploads and synced folders. Reading files keeps working. `--for` ends the pause by itself (`30m`, `1h`, `2d`). |
| `pdfs sync resume [--folder ID]` | Resume after a pause. |
| `pdfs sync limit [--up RATE] [--down RATE]` | Cap bandwidth, for example `500K` or `2M` per second. `0` removes a cap. Without options, prints the current caps. |
| `pdfs sync queue` | List changes that have not reached Proton Drive yet. |
| `pdfs sync retry [ID]` | Retry one queued operation, or every failed one. |
| `pdfs sync restore [--device UID] [--yes]` | Re-attach this computer's backed-up folders to local folders and download them. `--device` restores another computer's folders. See [RECOVERY.md](RECOVERY.md). |
## Conflicts
| Command | Description |
|---|---|
| `pdfs conflicts` | List `(sync-conflict …)` copies. |
| `pdfs conflicts resolve PATH --keep original\|copy\|both [--name NAME]` | `original` trashes the copy; `copy` trashes the original and gives the copy its name; `both` renames the copy to `--name`. |
## Computers
| Command | Description |
|---|---|
| `pdfs devices list` | List computers registered to the account. |
| `pdfs devices rename UID NAME` | Rename a computer. |
| `pdfs devices rm UID` | Remove a computer and its backup from the account. |
| `pdfs devices adopt UID` | Make this machine continue another computer's backup, for example after a reinstall. `--clear` returns to matching by hostname. |
## Trash
| Command | Description |
|---|---|
| `pdfs trash` | List the Trash with the UIDs the next commands take. |
| `pdfs restore UIDS...` | Put items back where they were. |
| `pdfs delete-forever UIDS...` | Delete items permanently. |
| `pdfs empty-trash` | Delete everything in the Trash permanently. |
## Versions
| Command | Description |
|---|---|
| `pdfs versions list PATH` | List a file's versions, newest first. |
| `pdfs versions restore PATH ID` | Make an earlier version current. Runs on the server; nothing is re-uploaded. |
| `pdfs versions save PATH ID DEST` | Write an earlier version to a local file. Refuses to overwrite `DEST`. |
| `pdfs versions rm PATH ID` | Delete an earlier version permanently. The current version cannot be deleted. |
## Sharing
| Command | Description |
|---|---|
| `pdfs share PATH EMAILS... [--role viewer\|editor\|admin] [--message TEXT]` | Invite people to a file or folder. |
| `pdfs members PATH` | List members, pending invitations and the public link. |
| `pdfs share-role PATH ID KIND ROLE` | Change a member's or invitation's role. |
| `pdfs unshare PATH [ID KIND]` | Stop sharing, or remove one member or invitation. |
| `pdfs public-link create PATH [--role] [--password] [--expires EPOCH]` | Create a public link. `--expires` takes Unix seconds. |
| `pdfs public-link remove PATH ID` | Remove a public link. |
| `pdfs shared` | List what you share. |
| `pdfs shared-with-me [UID]` | List what others share with you, or the contents of one shared folder. |
| `pdfs shared-get UID [DEST]` | Download a file shared with you. |
| `pdfs leave UID` | Leave a share. |
| `pdfs invitations list\|accept ID\|reject ID` | Manage invitations addressed to you. |
| `pdfs bookmarks list\|add URL [--password]\|rm TOKEN` | Manage saved public links. |
## Photos
| Command | Description |
|---|---|
| `pdfs photos [--limit N] [--offset N] [--favorites]` | List the timeline, newest first. |
| `pdfs open-photo UID` | Download a photo and print its cached path. |
| `pdfs favorite UID [--remove]` | Mark or unmark a favorite. |
| `pdfs albums` | List albums, including those shared with you. |
| `pdfs album UID` | List an album's photos. |
| `pdfs new-album NAME` | Create an album and print its UID. |
| `pdfs rename-album UID NAME` | Rename an album. |
| `pdfs delete-album UID` | Delete an album. Its photos stay in the timeline. |
| `pdfs add-to-album UID PHOTOS...` | Add photos to an album. |
| `pdfs remove-from-album UID PHOTOS...` | Take photos out of an album. |
| `pdfs import-google-photos ARCHIVES... [--dry-run] [--wait]` | Import a Google Takeout export. Pass every `.zip` of the export. |
| `pdfs import-status` / `pdfs cancel-import` | Follow or stop a running import. |
| `pdfs redate-photos [--dry-run] [--from DATE] [--to DATE] [--limit N] [--wait]` | Fix photos filed under their upload day when the file name carries the real date. |
| `pdfs redate-status` / `pdfs cancel-redate` | Follow or stop a running re-date. |
## Activity and transfers
| Command | Description |
|---|---|
| `pdfs activity [-l N]` | Recent activity, newest first. |
| `pdfs transfers` | Uploads and downloads in progress. |
## Maintenance and diagnostics
| Command | Description |
|---|---|
| `pdfs diagnose` | Check the installation and print a report. Works without the service. Exits non-zero if a check fails. |
| `pdfs diagnostics` | What a running service is doing now: worker threads, queues, requests in flight, memory. For a service that has stopped answering. |
| `pdfs cache inspect [--deep]` | Database size, reclaimable space, row counts, cache use against the limit. `--deep` also runs SQLite's integrity check (slow). |
| `pdfs cache vacuum` | Compact the database. Needs free space for a second copy of it while it runs. |
| `pdfs cache clear` | Delete cached file content. Files kept available offline stay. |
[TROUBLESHOOTING.md](TROUBLESHOOTING.md) explains when to use each of these.
## Scripting with `--json`
The global `--json` flag makes query commands print JSON: `status`, `ls`, `pins`, `sync list`,
`devices list`, `locations`, `transfers`, `activity`, `trash` and `cache inspect`. Commands that change
something keep their human-readable output; check their exit code.
```bash
pdfs --json status | jq -r '.mount.mountpoint'
pdfs --json sync list | jq -r '.items[] | select(.state != "idle") | .local_path'
pdfs --json cache inspect | jq '.db_reclaimable_bytes'
```
- The payload is unwrapped: a list is `{"items": [...]}`, never the service's internal message
name.
- An error from the service is printed as JSON with a machine-readable `kind` and still exits
non-zero, so `set -e` and `if pdfs …` behave as expected.
+201
View File
@@ -0,0 +1,201 @@
# Configuration
Most settings are in the app under **Preferences**. This page documents the configuration file
behind them, the settings that exist only there, the environment variables the programs read,
and where the client keeps its data.
- [Files and directories](#files-and-directories)
- [`config.json`](#configjson)
- [Environment variables](#environment-variables)
- [The systemd service](#the-systemd-service)
## Files and directories
| Path | Contents |
|---|---|
| `~/.config/proton-drive-linux/config.json` | Settings (this page) |
| `~/.local/state/proton-drive-linux/cache.db` | Metadata database: names, folder tree, sync state, upload queue, photo index |
| `~/.local/state/proton-drive-linux/sdk_cache.db` | Encrypted cache of Proton Drive metadata. Safe to delete while the service is stopped |
| `~/.local/state/proton-drive-linux/control.sock` | Socket the app, tray, prompt and CLI use to reach the service |
| `~/.cache/proton-drive-linux/content/` | File content cache, see below |
| `~/ProtonDrive` | The Proton Drive folder (default mountpoint) |
| System keyring | The session. Never written to disk by the client |
The directories honour `XDG_CONFIG_HOME`, `XDG_STATE_HOME` and `XDG_CACHE_HOME`. They must be real
directories owned by you with mode `0700`; the service refuses to start otherwise.
Inside `content/`:
| Directory | Contents | Safe to delete? |
|---|---|---|
| top level, `blocks/`, `thumbs/` | Cached file content and thumbnails | Yes — use `pdfs cache clear` |
| `scratch/` | Files open for writing right now | **No** |
| `staging/` | Closed files waiting to upload | **No** — may be the only copy |
| `recovery/` | Writes rescued after a crash, waiting to be queued | **No** — may be the only copy |
> [!CAUTION]
> Never delete `staging/` or `recovery/` by hand. Check `pdfs sync queue` first. Content there has
> not reached Proton Drive and exists nowhere else.
Everything in these directories except the keyring is **decrypted**. See
[ARCHITECTURE.md §8](ARCHITECTURE.md#8-threat-model-what-this-client-writes-to-disk-in-plaintext).
## `config.json`
The file is created with defaults on first start. Every key is optional; a missing key takes its
default. Edit it while the app is closed, then restart the service
(`systemctl --user restart proton-drive.service`) for service settings to apply.
An unreadable file is never overwritten: the programs log an error and run with defaults until you
fix it.
```json
{
"mountpoint": "/home/you/ProtonDrive",
"cache_budget": 10737418240,
"upload_limit": null,
"download_limit": 5242880,
"ignore_patterns": ["node_modules/", "target/", "*.iso"],
"language": "de",
"proton_theme": true,
"online_map": false,
"tray_hidden": false,
"files_view": { "list": true, "sort": "modified", "descending": true, "folders_first": true },
"open_with": {
"terminal": ["alacritty", "-e"],
"rules": [{ "match": ["@text"], "command": ["nvim"], "terminal": true }]
},
"prompt": { "mode": "fzf", "menu_limit": 50 }
}
```
### Reference
| Key | Type | Default | In the app | Meaning |
|---|---|---|---|---|
| `mountpoint` | path | `~/ProtonDrive` | General → Location | Where My files is mounted |
| `cache_budget` | bytes | 5 GiB | Storage → Cache | Soft limit for cached content. `0` means unlimited. Available-offline files do not count against it |
| `upload_limit` | bytes/s | none | General → Network | Shared cap for all uploads. `0` or `null` means none |
| `download_limit` | bytes/s | none | General → Network | Shared cap for all downloads, reads from the mount included |
| `ignore_patterns` | list | see below | — | Global ignore rules for synced folders |
| `device_uid` | string | none | Computers → Continue This Backup Here… | The computer this machine continues. Unset means "match by hostname" |
| `conflict_sweep` | `off` / `report` / `enforce` | `report` | — | See [`conflict_sweep`](#conflict_sweep) |
| `language` | gettext code | system | General → Appearance | Language of the app, tray and prompt, such as `de` or `pt_BR` |
| `proton_theme` | bool | follow system | General → Appearance | Proton's colours instead of the system theme |
| `online_map` | bool | `false` | General → Appearance | Street map for Places from OpenFreeMap |
| `tray_hidden` | bool | `false` | General → Appearance | Hide the tray icon |
| `files_view` | object | grid, by name | My files view menu | `list`, `sort` (`name`, `size`, `modified`), `descending`, `folders_first` |
| `open_with` | object | `xdg-open` | — | See [`open_with`](#open_with) |
| `prompt` | object | built-in window | — | See [`prompt`](#prompt) |
`app_version` and `user_agent` identify the client to Proton. Leave them alone.
### `ignore_patterns`
Gitignore-style rules applied to every synced folder, in addition to each folder's own
`.pdfsignore`. When the key is absent, these defaults apply:
```json
[".git/", ".hg/", ".svn/", "node_modules/", "target/", ".venv/", "__pycache__/",
"*~", "*.swp", "*.tmp", ".DS_Store", "Thumbs.db"]
```
Set it to `[]` to turn the defaults off. See
[USER_GUIDE.md](USER_GUIDE.md#excluding-files-with-pdfsignore) for how ignoring behaves.
### `conflict_sweep`
A background task looks for `(sync-conflict …)` copies that are byte-identical to the file they
were copied from.
| Value | Behaviour |
|---|---|
| `off` | Do not run. |
| `report` | Log and show what would be removed; remove nothing. **Default.** |
| `enforce` | Move copies proven identical (same size and SHA-1) to the Trash. |
`PDFS_CONFLICT_SWEEP` overrides the setting for one run of the service.
### `open_with`
Decides how files picked in the search launcher or the app are opened. Rules are tried in order;
the first match wins.
```json
{
"open_with": {
"terminal": ["alacritty", "-e"],
"default": ["xdg-open"],
"rules": [
{ "match": ["@text", "*.rs", "*.toml"], "command": ["nvim"], "terminal": true },
{ "match": ["*.png", "*.jpg"], "command": ["imv"] },
{ "match": ["@document"], "command": ["$EDITOR", "{}"], "terminal": true }
]
}
}
```
- `match` takes file-name globs (`*.md`, `notes-*.txt`) or the classes `@dir`, `@text`,
`@document`, `@image`, `@media` and `@any`. Patterns are matched against the name in Drive, not
against the cached file, which is stored under a content hash.
- `command` is an argument list, not a shell line. `{}` is replaced by the path; without it the
path is appended. `$VAR` expands from the environment.
- `"terminal": true` runs the command inside `terminal`. When `terminal` is unset, the client uses
`$PDFS_TERMINAL`, then `$TERMINAL`, then the first known terminal on `PATH`, and adds that
terminal's "run this command" flag for you.
- `default` replaces `xdg-open` for files no rule matches.
### `prompt`
Settings for `pdfs-prompt`.
| Key | Default | Meaning |
|---|---|---|
| `mode` | `gtk` | Front end used without a flag: `gtk`, `dmenu` or `fzf` |
| `menu` | first installed of fuzzel, rofi, wofi, tofi, bemenu, dmenu | Launcher for `dmenu` mode, as an argument list. `{prompt}` is replaced by the prompt text; without it, the launcher's own prompt flag is added |
| `terminal` | first installed of foot, ghostty, kitty, alacritty, wezterm, xterm | Terminal for `fzf` mode. `{cmd}` is replaced by the command; without it, the command is appended |
| `menu_limit` | `50` | Results shown in `dmenu` and `fzf` mode |
| `menu_icons` | `true` | File-type icons in launchers that support them |
```json
{
"prompt": {
"mode": "dmenu",
"menu": ["fuzzel", "--dmenu", "--width", "60"],
"terminal": ["foot", "--app-id=pdfs-prompt", "--"]
}
}
```
Command-line flags win over the file: `--gtk`, `--dmenu`, `--fzf` and `--menu`.
## Environment variables
| Variable | Read by | Effect |
|---|---|---|
| `RUST_LOG` | all | Log filter, for example `RUST_LOG=debug` or `RUST_LOG=pdfs_fuse=debug,info`. Default `info` |
| `PDFS_CONFLICT_SWEEP` | service | Overrides `conflict_sweep` (`off`, `report`, `enforce`) |
| `PDFS_TERMINAL`, `TERMINAL` | app, prompt | Terminal for `open_with` rules with `"terminal": true` |
| `LANGUAGE`, `LC_ALL`, `LC_MESSAGES`, `LANG` | app, tray, prompt | Interface language when `language` is unset |
| `PDFS_LOCALEDIR` | app, tray, prompt | Load translations from another directory (development) |
| `PDFS_STAND_IN_WIDGETS` | app | `1` uses the libadwaita 1.5 fallback widgets on newer systems (development) |
The acceptance suite has its own `PDFS_ACCEPTANCE_*` variables, listed in [TESTING.md](TESTING.md).
## The systemd service
`proton-drive.service` is a user unit. It restarts the service whenever it exits, restarts it when
it stops answering for two minutes (`WatchdogSec=120`), and caps its memory at 6 GiB. To change
the unit without editing the packaged file, use a drop-in:
```bash
systemctl --user edit proton-drive.service
```
```ini
[Service]
Environment=RUST_LOG=pdfs_fuse=debug,info
Environment=PDFS_CONFLICT_SWEEP=off
```
Logs go to the journal: `journalctl --user -u proton-drive.service -f`.
+126 -176
View File
@@ -1,214 +1,164 @@
# Development and Verification Status for 1.0
# Development guide
This document summarizes the work still pending after the 1.0.0 release branch and records the most important recently completed changes. [`BUGS.md`](BUGS.md) is the authoritative issue ledger; [`TESTING.md`](TESTING.md) defines the acceptance commands.
How to build, run, debug and release the client from source. Read
[CONTRIBUTING.md](../CONTRIBUTING.md) for the review rules and [ARCHITECTURE.md](ARCHITECTURE.md)
for how the pieces fit together.
---
- [Setup](#setup)
- [Repository layout](#repository-layout)
- [Running from source](#running-from-source)
- [Quality gates](#quality-gates)
- [Debugging](#debugging)
- [Common changes](#common-changes)
- [Releasing](#releasing)
## 1. Core Feature Roadmap (Horizon Tasks)
Consolidated from the historical feature roadmap; current bug status lives in [`BUGS.md`](BUGS.md).
## Setup
* **Local Cache & Metadata Encryption at Rest (Priority 5 / Horizon 1)**: Encrypt the SQLite database `cache.db` on disk using SQLCipher (`rusqlite` feature `bundled-sqlcipher`), and encrypt raw content cache blocks (`content/blocks/` and `content/scratch/`) using a fast symmetric scheme like AES-GCM or ChaCha20-Poly1305. The encryption key should be derived from the OS Keyring. (The SDK entity cache in `sdk_cache.db` is already encrypted at rest — see *Persistent SDK Entity Cache* below — but `cache.db` and the content cache are not.)
* **Active Bandwidth Throttling & Traffic Shaping (Horizon 1)**: Add speed governors for daemon uploads and downloads. Support configuration variables `max_upload_rate_kbps` and `max_download_rate_kbps` in `config.json` that can be dynamically adjusted over the Unix control socket.
* **Interactive & Policy-Based Conflict Resolution (Horizon 1)**: Implement configurable conflict resolution policies (`rename-local`, `rename-remote`, `prefer-local`, `prefer-remote`, `interactive`). Add a "Conflicts" tab to the GUI to notify the user, block sync, and launch external visual diff tools (like Meld or KDiff3).
* **Multi-Account / Multi-Profile Support (Horizon 3)**: Enable running multiple profiles concurrently. Support separate configuration namespaces (e.g. `profiles/<profile_name>/`), independent SQLite databases, distinct mount paths (e.g. `~/ProtonDrive/Personal` and `~/ProtonDrive/Work`), and multiple daemon processes.
* **Custom FUSE Mount Options (Horizon 1)**: Expose custom mounting parameters in configuration (e.g. `ro` for read-only, `direct_io` to bypass kernel caching, `allow_other`). Auto-generate systemd user mount units upon sync folder configuration.
* **Symbolic Link & Hard Link Virtualization (Horizon 2)**: Add symbolic link virtualization inside the metadata store (`NodeType::Symlink`) by intercepting `readlink(2)` and `symlink(2)`. Targets should sync remotely as small encrypted metadata placeholder files on Drive (`proton-vfs-symlink: <target>`).
* **Desktop File Manager Integration (Horizon 2)**: Develop shell extension plugins for popular Linux file managers (Nautilus Python plugin, Dolphin Service Menus, Thunar custom actions) to offer context-menu shortcuts ("Pin", "Unpin", "Copy Share Link", "View Version History") and file overlay sync badges.
* **Integrated Photo Gallery Enhancements (Horizon 2)**: Album *editing* — rename, cover photo, delete, and removing photos from an album — is still unported in the SDK (TS `updateAlbum` / `deleteAlbum` / `removePhotos`), as is adding a photo that lives on someone else's volume (TS `copyPhoto`). Album **creation** and adding own-volume photos landed with the Google Photos import. Still open: an auto-upload Pictures folder pipeline, and a GUI for creating albums by hand. Exif display, the date-based timeline scrubber and favorites are implemented.
* **Google Photos import follow-ups**: the importer uploads photos one at a time — a large export would finish sooner with a bounded concurrent upload pool. Trash folders are recognized by a hardcoded list of localized names ([`takeout.rs`](../crates/pdfs-core/src/takeout.rs) `TRASH_FOLDERS`); a locale not on it imports its trash as ordinary photos. Motion-photo `.MP`/`.MV` sidecar videos are ignored rather than uploaded as related photos. Live validation against a real export is pending.
* **Random-Access Media Streaming seek support (Horizon 2)**: Intercept out-of-order sparse reads (player seeks) in `pdfs-fuse` and prioritize downloading blocks around the seek offset, aborting/postponing sequential pre-fetches.
* **Dynamic Sync Dashboard & Queue Visualization (Horizon 2)**: GUI real-time transfer list (progress bars, speed, ETA), sync history feed, storage usage breakdown, and global pause/resume sync controls.
* **Interactive Terminal UI (TUI) Mode (Horizon 3)**: Implement `pdfs tui` using the `ratatui` crate to show transfer speed, sync status, active transfer queue with progress bars, and scrolling daemon logs.
* **Shutdown Safety & Write Queue Visibility (Horizon 1 / P4)**: Register systemd inhibitor locks in `pdfs-fuse` when there are outstanding staging writes to prevent shutdown or sleep, pop up warnings when exiting the GUI, and build a blocking `pdfs sync flush` command.
* **Block-Level Deduplication (Horizon 3)**: Transition to a Content-Addressable block cache where cached blocks are stored on disk by their SHA-256 hash. Map logical ranges in the database: `(node_uid, block_idx) -> block_hash` to avoid storing or downloading identical blocks.
* **Pre-emptive Sync Debouncing & File System Events (Horizon 3)**: Group filesystem write events on paths and delay sync queue insertion until a quiet period has elapsed (e.g. 5 seconds of inactivity) to prevent thrashing the Proton API.
You need Rust 1.96 or newer and the native development packages listed under
[Build dependencies](INSTALL.md#build-dependencies) for your distribution. Then:
---
```bash
git clone https://github.com/narrrl/proton-drive-linux.git
cd proton-drive-linux
cargo build --workspace
```
## 2. Performance, Correctness & Robustness Items
Consolidated from the performance and media-streaming audit notes.
The GUI builds against libadwaita 1.5, the version in Ubuntu 24.04, and uses newer widgets at run
time when the installed library has them (`crates/pdfs-gui/src/compat.rs`).
* **B6. Debounce the LRU access touch**: Keep last-touch times in memory and flush them in batches (debounce window of 30-60 seconds) to avoid performing a SQLite `UPDATE` to the `cache_entries` table on every cache hit.
* **E12. Client in-flight cap shape**: Address the lock discrepancy between `MAX_CONCURRENT_BLOCK_DOWNLOADS = 10` (per-file) and `DEFAULT_MAX_INFLIGHT_BLOCKS = 12` (global client-wide). Either raise the global cap, lower the per-file window, or dynamically calculate the window to prevent single files from starving concurrent transfers.
* **Pipeline `enumerate_nodes_detail` (P2.8 / 8)**: Introduce pipelined node enumeration to speed up directory listings.
* **Local ancestor-chain walk (P3.4 / 4)**: Optimize recursive path lookups where full batching is impossible.
* **Name Search for Photos/Videos**: Add search by photo/video name (deferred from Phase 3 gallery implementation).
* **Sequential-Read Prefetch for Media Streaming**: Implement sequential read prefetch for media files to avoid stalls when buffering.
* **Minor sweep items (Phase F)**:
* **F1. `StreamRing.tags` Leak**: Clean up tag entries in `StreamRing.tags` when blocks are evicted. Currently, it only shrinks on `drop_node` (revision mismatch), leading to tag accumulation.
* **F2. Non-saturating bytes subtraction**: Use `saturating_sub` in `lib.rs:382` (`self.bytes -= dropped.len()`) to prevent release mode panics/wrapping if accounting drifts.
* **F3. Short block yields a short read / EOF**: Address the kernel EOF interpretation on short block reads. Either pad the block or fail loudly rather than silently serving short data.
* **F4. `stream_readahead` check-then-spawn race**: Prevent duplicate concurrent readahead tasks for the same block. Check-then-spawn is currently racey.
* **F5. `refresh_blocks` permit lock (SDK)**: Ensure URL refreshing (`refresh_blocks`) does not hold an in-flight block permit, preventing expired URLs from pinning client permits.
* **F7. Document temporary cache file safety**: Add explanatory comment to `cache.rs:362` explaining why `with_extension("tmp")` is safe there but not in `store_thumbnail`.
## Repository layout
---
| Path | Contents |
|---|---|
| `crates/pdfs-core` | Shared library: authentication and keyring (`auth`), configuration (`config`), SQLite schema and queries (`db/`), content cache (`cache`), control protocol (`control`), search scoring, sync ignore rules, Google Takeout parsing, machine profile |
| `crates/pdfs-fuse` | The daemon: FUSE filesystem, read path (`reads`), write queue (`queue`, `drain`, `upload`), sync engine (`sync/`), remote events (`background`), push events for front ends (`events`), control handlers (`control`), photos, devices, supervisor |
| `crates/pdfs-cli` | The `pdfs` binary |
| `crates/pdfs-gui` | `pdfs-app`, `pdfs-tray` and `pdfs-prompt`. Pages in `src/app/pages/`, shared widgets in `src/app/widgets/`, CSS and icons in `resources/` |
| `po/` | Translation template, catalogs and the scripts that maintain them |
| `packaging/` | systemd unit, desktop files, icon, PKGBUILD, RPM spec, AUR recipes |
| `scripts/` | FUSE acceptance suite, map-data generators |
| `docs/` | This documentation |
## 3. Open & Unverified Bugs
Derived from: [`BUGS.md`](BUGS.md)
The GUI talks to the daemon only through the control socket. Anything a front end needs that the
daemon does not offer yet is a new request in `pdfs_core::control`, not a database read.
* **B2. Unattributed Trash Origin**: Investigate why deleted/moved files are occasionally sent to trash instead of vanishing on older rename operations.
* **B10. GLib Critical Warning on HUD Close**: Assertion failure `g_list_store_remove: assertion '!g_sequence_iter_is_end (it)' failed` when prompt window closes during active launch (`xdg-open`). Needs investigation under `G_DEBUG=fatal-criticals`.
* **B15. Empty-but-listed folders and duplicate UIDs**: Investigate folder-identity issues (e.g. duplicate remote folders like `Music` with different UIDs) and how folders obtain `listed = 1` in the database without matching child rows.
* **B8 (CAPTCHA sign-in bridge)**: Verify a real gated sign-in using Webkit CAPTCHA completion.
* **Draft Revision Upload loop**: Verify SDK-side deletion and retry logic when a conflicting draft revision exists under the same client UID.
* **Optimistic Size Loss on Restart**: Verify that pending uploads retain their optimistic sizes across a daemon restart.
* **B48 — stale listing after unlink**: Session tombstones and authoritative empty listings are implemented; run focused live verification against eventually consistent remote enumeration.
* **B49–B55 — data-safety fixes**: Conflict preservation, orphan/staging durability, transactional supersession, one-entry wipe protection, and incomplete-scan handling have regression coverage but still require the fault, power-loss, and live mode-matrix cases recorded in `BUGS.md`.
* **B56–B60 — fail-closed state and IPC fixes**: Future schemas, event cursor advancement, socket limits, private-directory validation, and atomic config saves are implemented. Malformed-schema, database-full, connection-flood, wrong-owner, and concurrent/faulted-save verification remains.
* **B61 — credential rotation persistence**: Surface and retry keyring persistence failures so a rotated single-use refresh token cannot be lost on restart.
* **B62–B65 — release assurance**: Clean-install package verification, complete version/protocol identity, recovery sign-off, advisory/license policy, SBOM, provenance, checksums, and signatures remain.
* **B66–B67 — lifecycle and recovery**: Add queue flush/shutdown protection and explicit device adoption during fresh-state restore.
* **B68/B78 — profile backup**: the destination is fixed and verified live — `profile.json` now lives in `.proton-drive-linux/` under the device root, since the API refuses file creation at the root itself. What remains from B68 is making backup *health* visible (today a failure is a `WARN` in the log and nothing in the UI) and re-running the replacement-machine restore end to end.
* **B79 — cross-mount permission classification**: fixed and verified live; the standing guard is the `regression B79` acceptance case. The second-account cases it neighbours (`regression B34`) still cannot run — see below.
## Running from source
---
The packaged service and a daemon run from source cannot share the same state directory. Stop the
service before you run your own build:
## 4. Testing, Verification & Disaster Recovery Drills
Derived from: [`TESTING.md`](TESTING.md) and [`RECOVERY.md`](RECOVERY.md)
```bash
systemctl --user stop proton-drive.service
RUST_LOG=pdfs_fuse=debug,info cargo run -p pdfs-cli -- daemon
```
* **IPC Memory and Performance Stress Tests (Phase 3)**: Verify memory allocations do not balloon when sending large `UploadPhoto` JSON payloads. Test control socket timeout behaviors under long-running block operations.
* **Fault Injection and Resiliency Tests (Phase 4)**:
* **Network degradation**: Verify that interrupted uploads leave readable staging buffers and result in clean backoff retries.
* **Disk full (ENOSPC)**: Verify that write staging and gap-filling report correct errors to FUSE and preserve staged work.
* **Concurrent reader starvation**: Test that parallel sequential reads do not freeze directory lookups or metadata handlers.
* **Disaster Recovery Drill (P5 / Phase 5)**: Execute a complete manual disaster recovery test:
1. Register a device.
2. Sync a folder and upload data.
3. Wipe the local state directory and SQLite cache.
4. Restart the daemon.
5. Run `pdfs sync restore` and assert the recovered filesystem matches the source byte-for-byte.
* **Second-account sharing run (mount-architecture.md P7)**: The only outstanding verification for the mount/sharing work. One account covers `context_share_id`, the shared-with-me listing, sharing from device folders and the Locations page — all confirmed. A **second account** is required for the shared-tree rendering under a foreign volume, `accept_invitation`, `update_member_role`, the `regression B34` viewer cases (which skip cleanly today), and above all an **editor-role write into a foreign share**, the only thing that validates `membership_address_for`.
* **Managed live mode matrix**: Before release, run `scripts/fuse-acceptance.sh --managed-live <empty-a> <empty-b>` with `PDFS_ACCEPTANCE_ONLY` unset. It validates on-demand/on-demand, on-demand/mirror, mirror/on-demand, and mirror/mirror transitions.
* **Crash-consistency matrix**: Inject failure after scratch/sidecar sync, each staging rename, directory sync, pending-op insert/commit, and source deletion; prove that a complete discoverable copy and either the old or new queue record always survives.
In another terminal:
---
```bash
cargo run -p pdfs-cli -- status
po/build.sh target/locale # compile translations once
PDFS_LOCALEDIR=target/locale cargo run -p pdfs-gui --bin pdfs-app
cargo run -p pdfs-gui --bin pdfs-tray
cargo run -p pdfs-gui --bin pdfs-prompt -- --fzf
```
## 5. Fixed & Resolved Bugs
Derived from: [`BUGS.md`](BUGS.md)
To keep the systemd unit but point it at your build, see
[INSTALL.md](INSTALL.md#using-a-binary-outside-usrbin).
### Completely Fixed & Verified
* **B1 — FUSE rename loses the file (data loss)**: In `rename`, the node's database row was deleted while only updating the in-memory state, causing the file to vanish on next listing sync. Fixed by correctly calling `st.invalidate_listing(newparent)` to refresh SQLite cache states.
* **B4 — `invalidate_listing` silently skipped non-resident folders**: Folders not loaded in memory were being skipped during listing invalidation, leaving stale cache flags in the DB. Fixed by removing the residency check guard.
* **B5 — `ls -l` costs a network round trip per file (thumbnail xattr probes)**: File listings triggered blocking network checks for thumbnails on unsupported files. Fixed by caching negative thumbnail results and restricting xattr advertisement to supported image/media files.
* **B6 — Daemon sets no file modes: `control.sock` is an unguarded authority**: The Unix control socket was created without restricting file permissions, exposing daemon controls to other local users. Fixed by applying proper creation permission masks.
* **B7 — renamed directory reads as missing until the entry TTL expires**: Kernel cache maps remained stale after a FUSE directory re-anchor. Fixed by proactively sending entry invalidation notifications to the kernel.
* **B9 — Enter did not open the selected result in the launcher**: Hooked up keyboard execution (Enter key) in `pdfs-prompt` search HUD to launch the selected item using `xdg-open`.
* **B11 — a file moved while its create is still queued reads as empty**: Reconciling a local rename on a file that had not finished uploading yet uploaded empty blocks. Fixed by carrying over and staging the correct logical file size.
* **B12 — cold enumeration is slow per entry, and goes superlinear past ~500**: S2K decryption and thumbnail resolution loops were serial. Fixed by parallelizing key derivations and caching metadata.
* **B13 — `rename` over an existing destination fails instead of replacing it**: Fixed POSIX rename target overwrite behavior to replace existing destinations.
* **B14 — provisional (ciphertext) sizes make rsync read short and abort the file**: Adjusted size reporting to align logical and ciphertext sizing, preventing premature EOF.
* **B45 — truncate over a queued rewrite**: Path-based and handle-based truncate now compose over the complete pending blob; shrink, growth, and sparse I/O are live-verified.
* **B46 — combined cross-directory move and rename**: The desired end state is durably queued and reconciles partially completed remote operations; replacement and combined cases are live-verified.
* **B47 — overlong/invalid names**: All name-taking callbacks share a validator for `NAME_MAX`, reserved components, and UTF-8; managed-live verification passed.
> [!CAUTION]
> Develop against a test account. The acceptance suite and a buggy daemon can delete or rewrite
> files in Proton Drive.
### Implemented in 1.0 (Further Fault Verification Pending)
> [!TIP]
> A daemon stopped in a debugger or deadlocked freezes every process that touches the mount,
> including your shell if its prompt reads the current directory. Keep your shell outside
> `~/ProtonDrive`, prefix commands that touch it with `timeout`, and unmount a dead mount with
> `fusermount3 -uz ~/ProtonDrive`.
* **Unified search and resident prompt**: `SearchV2` combines Drive and local lookup with shared relevance scoring and content/source filters. The prompt reuses its window and opens streamable media through FUSE.
* **Durable write publication**: Scratch sidecars and staged blobs use synced temporary files, atomic renames, and directory synchronization; orphan rescue retains its source on failure.
* **Transactional pending operations**: Superseding a queued operation cannot delete the old durable record unless the replacement commits.
* **Safer mirror reconciliation**: Conflict-copy failure aborts publication, incomplete scans are non-destructive, and total-wipe protection includes one-entry baselines.
* **Bounded IPC**: Control requests are limited to 1 MiB and 10 seconds, with at most 64 active handlers.
* **Private local state**: Sensitive directories require correct ownership, real-directory identity, and `0700`; sockets use `0600`; config saves are atomic and malformed input is preserved.
* **Release gates**: Tag/workspace/package versions are checked and formatting, lint, tests, and offline FUSE acceptance run before packaging.
## Quality gates
### Code Fixed / Implemented (Pending Verification)
* **B8 — no way to complete a CAPTCHA, so a gated sign-in is unrecoverable**: Implemented a Webkit-based bridge to prompt the user to resolve interactive CAPTCHAs during gated logins. (Needs verification on a real gate request).
CI (`.github/workflows/ci.yml`) runs these on every push to `main` and every pull request. Run them before you
push:
### Closed / Not a Bug
* **B3 — Activity log timestamps written in seconds, read as milliseconds**: Closed as investigator error (timestamps are consistent).
```bash
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only
```
---
For changes to the FUSE layer, the write queue or the sync engine, also run the live acceptance
cases described in [TESTING.md](TESTING.md).
## 6. Recently Implemented Features
For changes to GUI strings, check that every catalog is complete:
### File Version History (Implemented)
**Files**: [`revisions.rs`](../crates/pdfs-fuse/src/revisions.rs), [`control.rs`](../crates/pdfs-core/src/control.rs), [`versions_dialog.rs`](../crates/pdfs-gui/src/app/widgets/versions_dialog.rs)
```bash
for l in $(cat po/LINGUAS); do msgfmt --check --statistics -o /dev/null po/$l.po; done
```
Proton Drive keeps every revision a client committed; the daemon only ever addressed the active one, so a file overwritten by a sync pass could be recovered only from whatever the local `recovery/` directory happened to hold.
## Debugging
The control protocol gained `ListRevisions` / `RestoreRevision` / `DeleteRevision` / `SaveRevisionAs` (each with a `…ByUid` twin for nodes the primary mount cannot name), the CLI gained `pdfs versions list|restore|save|rm`, and the browser's details pane gained a **Versions** button opening a per-file dialog.
| Tool | Use |
|---|---|
| `RUST_LOG` | Log filter for every binary, for example `RUST_LOG=pdfs_fuse::sync=trace,info` |
| `pdfs diagnose` | Static checks: directories, FUSE, keyring, socket, database integrity. Works without the daemon |
| `pdfs diagnostics` | Live state of a running daemon: each worker's current job and its age, queue depth, control requests in flight, memory |
| `pdfs cache inspect --deep` | Database size, row counts and SQLite `integrity_check` |
| `pdfs --json …` | Raw protocol payloads for query commands |
| `GTK_DEBUG=interactive` | The GTK inspector for `pdfs-app` |
| `PDFS_STAND_IN_WIDGETS=1` | Force the libadwaita 1.5 fallback widgets on a newer system |
| `G_DEBUG=fatal-criticals` | Turn GLib criticals into crashes to get a backtrace |
Three properties that shape the code:
The databases are ordinary SQLite files. Open a copy, not the live file:
- **A restore is server-side and asynchronous.** No content crosses the wire and nothing enters the drain queue; the server answers 202 and swaps the active revision in the background, so the daemon evicts the file's cached blocks and open readers rather than describing the new state, and the UI wording promises the request, not the result.
- **The active revision cannot be deleted.** `Core::delete_revision_for_uid` checks that locally so the user gets a sentence rather than an API code, and the dialog gives the current row no delete button.
- **Saving a version never overwrites.** `SaveRevisionAs` refuses an existing destination and removes a partial file if the download fails — a half-written export looks identical to a good one to every tool that opens it afterwards.
```bash
cp ~/.local/state/proton-drive-linux/cache.db /tmp/cache.db
sqlite3 /tmp/cache.db '.tables'
```
### Photo Favorites (Implemented)
**Files**: [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs)
## Common changes
The gallery can mark and filter favorites, through the SDK's photo-tag API (`update_photos`). Schema **v20** adds `photos.favorite`; the flag is learned in the timeline enrichment pass that already resolves each photo's name and media type, and is kept across a refresh that could not resolve a photo — the same learned-and-kept rule as `media_type`. The lightbox carries a star toggle, the Photos header a favorites filter, and the CLI has `pdfs favorite <uid> [--remove]` plus `pdfs photos --favorites`.
**A new control request.** Add the variant to `Request` (and a `Response` if it returns data) in
`crates/pdfs-core/src/control.rs`, handle it in `crates/pdfs-fuse/src/control.rs`, and list the
`Topic`s it changes in the same file so subscribed front ends refresh. A daemon from the previous
release may still be running when a new front end starts, so treat an error reply to a new
request as "not supported".
Favouriting a photo that is not on this account's own photos volume (shared with us, or album-only) needs it re-encrypted for our timeline root, which the SDK does not implement; the daemon surfaces that as an error rather than silently doing nothing.
**A schema change.** Migrations are forward-only. Add a new step to
`crates/pdfs-core/src/db/migrations.rs` and bump `SCHEMA_VERSION`; never edit a shipped step. Add
a test in `crates/pdfs-core/src/db/tests.rs` that migrates from the previous version.
### Justified Gallery, Gallery Delete (Implemented)
**Files**: [`photos.rs`](../crates/pdfs-gui/src/app/pages/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs), [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`background.rs`](../crates/pdfs-fuse/src/background.rs)
**A GUI string.** Wrap it in the helpers from `crates/pdfs-gui/src/i18n.rs`, add any new source
file to `po/POTFILES.in`, run `po/update.sh`, and translate the new entries in every catalog. See
[TRANSLATING.md](TRANSLATING.md). The CLI and daemon messages stay in English.
The gallery lays each day out in justified rows (`justify_rows` / `plan_rows` / `fit_row`): photos are taken in capture order until their summed aspect ratio no longer fits the target row height, and the row is then scaled so it ends exactly on the content width. A tile is therefore its own photo's shape, and `ContentFit::Contain` has nothing left to crop. The last row of a day is left at the target height rather than stretched, because a day holding two photos is a short day, not a layout fault.
**A setting.** Add the field to `AppConfig` in `crates/pdfs-core/src/config.rs` with
`#[serde(default)]`, so older files still load, and document it in
[CONFIGURATION.md](CONFIGURATION.md).
- **Ratios that are not known yet.** `PhotoItem::ratio` is persisted, but a photo that has never been decoded has none. Those are laid out square and remembered in `assumed_ratios`; when a decode proves the real shape the day re-flows on the existing debounce timer, so a screenful of decodes costs one re-flow rather than one per photo. Ratios are clamped to 0.4–3.0 so a single panorama cannot decide what a row looks like.
- **Thumbnails survive recycling.** A reply for a tile that has already been re-bound still lands in the texture cache, which is an LRU of 1500 textures keyed by uid, and the rows just outside the realised range are prefetched in the scroll direction.
- **Delete is uid-addressed.** The photos volume is not in the FUSE mount, so the path-based `Request::Delete` cannot reach it. `Request::TrashNodes { uids }` trashes through the SDK and answers `Response::Trashed { trashed, failed }`; the daemon then drops those rows with `Db::photos_delete` (album membership included) so the gallery does not wait for a timeline refresh. The GUI removes the tiles optimistically with an Undo toast and puts back anything the server refused.
- **Remote deletions arrive on their own.** The Drive volume and the photos volume have separate event streams, so `run_photos_event_sync` polls the photos volume with its own cursor (`photos_event_cursor` in the state table). Trash and delete events remove rows; anything else only invalidates freshness, because there is no inode space to converge on this volume. Together with a 60 s `TIMELINE_TTL` and a `RefreshScope::Photos` that now awaits the refresh, a photo deleted on a phone leaves the grid in about ten seconds.
## Releasing
### RAW + JPEG Grouping (Implemented)
**Files**: [`photos.rs`](../crates/pdfs-core/src/db/photos.rs), [`migrations.rs`](../crates/pdfs-core/src/db/migrations.rs), [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs)
Releases are cut from `main` by pushing a `v*` tag. `.github/workflows/release.yml` then:
Schema **v30** adds `content_hash`, `main_uid` and an indexed `group_key` to `photos`. The first two come from the server's `PhotoProperties` and are filled by the enrichment pass in `refresh_timeline` that already resolves each photo's name, media type and favorite tag; `group_key` is computed in `photos_replace`, where the whole timeline is in hand.
1. checks that the tag, the workspace version in `Cargo.toml` and `pkgver` in
`packaging/PKGBUILD` agree;
2. runs the quality gates;
3. builds the release binaries and packages a `.tar.gz`, a `.deb` and a GPG-signed `.rpm`;
4. publishes them as a GitHub release;
5. calls `.github/workflows/aur.yml`, which updates and pushes the AUR recipes and commits them
back to `main`.
- **Precedence.** The server relation first (`main_photo_uid` and `related_photo_uids`, read from whichever end resolves — the two can land in different enrichment chunks); then same capture day, same case-insensitive name stem, and one member raw while the other is not; otherwise the photo is its own group. The union-find pass is over the timeline in memory, so it costs one pass per refresh rather than a query per photo.
- **Representative.** The non-raw member when there is one — a JPEG decodes in milliseconds and is what the person expects to see. `group_key` holds that member's uid, so "is this the tile" is `uid = group_key`, an index scan.
- **What is grouped and what is not.** `photos_page`, `photos_months` and `photos_counts` count groups, except on the Raw tab, which lists files. Album pages are never grouped: an album is a list someone made. The relation is learned-and-kept like `media_type`, so a refresh that could not resolve a node does not break a group up.
- **Deleting.** `Core::trash_photos` expands each uid to its group before trashing, because the grid shows the shot as one tile — leaving the RAW behind would put the photo back on the Raw tab and nowhere else. The GUI's confirmation names the file count.
- **The other files.** `Request::PhotoGroup { uid }` answers with the group's members as ordinary `PhotoItem`s, which is what the lightbox's switch steps through.
To prepare a release:
### Persistent SDK Entity Cache (Implemented)
**Files**: [`sdkcache.rs`](../crates/pdfs-core/src/sdkcache.rs), [`auth.rs`](../crates/pdfs-core/src/auth.rs), [`background.rs`](../crates/pdfs-fuse/src/background.rs)
```bash
# 1. Bump the version in Cargo.toml and packaging/PKGBUILD, then refresh the lock file.
cargo update --workspace
# 2. Move the [Unreleased] notes in docs/CHANGELOG.md under the new version.
# 3. Commit, tag and push.
git commit -am "chore(release): X.Y.Z"
git tag vX.Y.Z
git push origin main vX.Y.Z
```
The SDK's Drive entity cache (decrypted node metadata: name, size, parent, signing share) defaulted to memory, so every daemon restart re-fetched and re-decrypted the tree the previous run had already walked. It is now backed by SQLite.
- **Its own file** (`sdk_cache.db` in the state directory), not `cache.db`: the daemon's `Db` is one `Mutex<Connection>` shared by every FUSE thread and the control socket, and this traffic is frequent, small and entirely reconstructible. A separate file also means no schema migration — the store can be deleted at any time.
- **Encrypted at rest** by wrapping it in the SDK's `EncryptedCacheRepository`, keyed by the mailbox password. A password change reads as a cold cache (the SDK treats an undecryptable entry as a miss and clears the store), not as an error.
- **Staleness** is closed by the event loop, which already replays from a persisted cursor and calls `invalidate_caches_for_event` for every event — including those raised while the daemon was down. The one case with no trail is a *seeded* cursor (first-ever mount, or a lost cursor), where the seed path now clears the store instead of trusting it. `auth::logout` removes the file.
Needed a small additive SDK change (`ProtonDriveClient::with_entity_repository`, 0.5.1): `with_entity_cache` is a constructor and so could not be combined with `with_key_salts`, which the daemon requires.
### Per-node Batch Outcomes (Implemented)
**File**: [`batch.rs`](../crates/pdfs-core/src/batch.rs)
SDK 0.5.0 made `trash_nodes` / `restore_nodes` / `delete_nodes` report one outcome per node instead of failing the whole call (mirroring upstream's streamed `NodeActionResult`). `pdfs_core::batch::into_unit` collapses the single-node calls back to a `Result`; `batch::split` handles a collected batch.
A restore is expanded before it is sent (`expand_restore`, `pdfs-fuse/src/lib.rs`): the persisted trash listing carries each row's `parent_uid` (schema 29), so a restore takes the connected piece of the trashed tree — every trashed descendant of what was asked for, and every trashed ancestor above it — and sends it shallowest wave first, because the server cannot put a node back under a parent that is still trashed. A uid the listing does not know about (it is materialised in chunks, and can be stale) is restored on its own rather than dropped.
The trash view's restore and permanent-delete use the SDK's **streaming** variants (`restore_nodes_streaming` / `delete_nodes_streaming`) and apply local state per node as each batch lands rather than after the last one. For a permanent delete — which is irreversible — that means a daemon interrupted mid-batch has forgotten exactly the nodes the server destroyed, no more and no fewer. Both report per-node failures and count only what actually succeeded; only a batch where nothing succeeded is an error.
### Open-for-Write Deferral for Mirror Sync (Implemented)
**File**: [`sync.rs`](../crates/pdfs-fuse/src/sync.rs)
The mirror folder sync engine now defers uploading any file that is currently held open for writing by another process, matching the guarantee the FUSE mount path already provides (where uploads are deferred until `close(fd)`).
**Problem**: Previously, the mirror sync path relied solely on a 2-second trailing-edge debounce (with a 30-second ceiling) to avoid uploading files mid-write. This was insufficient for slow continuous writers (e.g., database dumps, large exports, or editors that keep files open for extended periods), which could have their incomplete state uploaded as a real revision.
**Solution**: Before each reconcile pass, the sync engine scans `/proc/*/fd` once to build a set of canonical paths within the sync root that any process holds open for writing (`O_WRONLY` or `O_RDWR`). Files in this set are:
- **Kept in the local walk** — so they are not misclassified as deletions
- **Treated as unchanged** — so no upload is queued for them
- **Counted as `deferred`** — so the activity summary reports them (e.g., "3 uploaded, 1 deferred (open for write)")
- **Picked up on the next pass** — after the writer has closed the file
**How it works**:
1. `open_for_write_set(root)` reads `/proc/*/fd` → `readlink` for each fd → prefix-checks against the sync root → reads `/proc/<pid>/fdinfo/<n>` to check the `flags:` line's low two bits (O_ACCMODE)
2. `walk_local` sets `LocalItem.open_for_write = true` for matching files
3. Both `do_reconcile` and `push_pass` skip upload classification for flagged files
4. `Outcome.deferred` tracks the count for the activity summary
**Performance**: The `/proc` scan completes in low single-digit milliseconds on a typical desktop. Only fds whose `readlink` target falls under the sync root incur the `fdinfo` read.
**Tests added**: 8 unit tests covering `is_write_mode` parsing of various fdinfo flag combinations, and `Outcome` summary formatting with deferred counts.
| Sync Path | Open-for-write protection | Mechanism |
|---|---|---|
| FUSE mount | ✅ Perfect | Upload deferred until `close(fd)` |
| Mirror folders | ✅ Perfect | `/proc/*/fd` scan per reconcile pass |
Before a release that touches the filesystem or sync, run the managed live mode matrix from
[TESTING.md](TESTING.md). Open release-assurance items are listed in [ROADMAP.md](ROADMAP.md).
+251
View File
@@ -0,0 +1,251 @@
# Installation
This page covers installing Proton Drive for Linux from a package or from source, the first
sign-in, running without a desktop, and removing the client again.
- [Requirements](#requirements)
- [Install a package](#install-a-package)
- [Build from source](#build-from-source)
- [First run](#first-run)
- [Headless and server use](#headless-and-server-use)
- [Updating](#updating)
- [Uninstalling](#uninstalling)
## Requirements
| | Minimum |
|---|---|
| Architecture | x86_64 (release packages); other architectures build from source |
| Kernel | FUSE 3 (`fuse3` package, `/dev/fuse` available to your user) |
| GTK / libadwaita | GTK 4.12, libadwaita 1.5 (1.8 or newer gets the newest widgets) |
| WebKitGTK | 6.0 — used only for the CAPTCHA page during sign-in |
| Secret Service | GNOME Keyring, KWallet, KeePassXC, or another provider of `org.freedesktop.secrets` |
| Rust (source builds) | 1.96 |
The session is stored only in the Secret Service. Without a running provider, sign-in succeeds but
cannot be saved, and the service cannot start unattended.
Optional runtime extras:
| Package | Adds |
|---|---|
| `exiftool` (`perl-image-exiftool`, `libimage-exiftool-perl`) | Thumbnails for camera RAW files |
| `ffmpeg` | Video thumbnails |
| `iso-codes` | Country names in your language on the Places map |
| `xdg-utils` | "Open Folder" in the tray, and the default file opener |
| `fzf` plus a terminal (foot, kitty, alacritty, …) | `pdfs-prompt --fzf`, the search-as-you-type launcher |
| AppIndicator extension (GNOME only) | The tray icon. KDE Plasma and most other desktops show it natively |
## Install a package
Every [release](https://github.com/narrrl/proton-drive-linux/releases/latest) ships a `.deb`, a
signed `.rpm` and a `.tar.gz` of the raw binaries. All packages install the same files:
| Path | Purpose |
|---|---|
| `/usr/bin/pdfs` | Command-line client and background service |
| `/usr/bin/pdfs-app` | Desktop app |
| `/usr/bin/pdfs-tray` | Tray icon |
| `/usr/bin/pdfs-prompt` | Search launcher |
| `/usr/lib/systemd/user/proton-drive.service` | User service that keeps the drive mounted |
| `/etc/xdg/autostart/io.narl.proton-drive-linux-tray.desktop` | Starts the tray at login |
| `/usr/share/locale/*/LC_MESSAGES/pdfs.mo` | Translations |
### Debian and Ubuntu (24.04 or newer)
```bash
sudo apt install ./proton-drive-linux_*_amd64.deb
```
### Fedora (44 or newer)
```bash
sudo dnf install ./proton-drive-linux-*.x86_64.rpm
```
For a Fedora package built from your own checkout, see
[Build an RPM locally](#build-an-rpm-locally).
### Arch Linux (AUR)
Three packages cover the usual choices. They conflict with each other, so install one:
| Package | Contents |
|---|---|
| [`proton-drive-for-linux`](https://aur.archlinux.org/packages/proton-drive-for-linux) | Latest tagged release, built from source |
| [`proton-drive-for-linux-bin`](https://aur.archlinux.org/packages/proton-drive-for-linux-bin) | Latest tagged release, prebuilt binaries |
| [`proton-drive-for-linux-git`](https://aur.archlinux.org/packages/proton-drive-for-linux-git) | `main`, built from git |
```bash
paru -S proton-drive-for-linux # or yay, or any other AUR helper
```
The recipes live in [`packaging/aur/`](../packaging/aur/README.md).
### Other distributions
Unpack the `.tar.gz` into a directory on your `PATH`, then install the service unit as shown in
[Using a binary outside `/usr/bin`](#using-a-binary-outside-usrbin).
## Build from source
### Build dependencies
**Debian / Ubuntu 24.04+**
```bash
sudo apt-get install -y pkg-config gettext libfuse3-dev libgtk-4-dev libadwaita-1-dev \
libwebkitgtk-6.0-dev libsecret-1-dev libdbus-1-dev
```
**Fedora 44+**
```bash
sudo dnf install -y cargo rust gettext pkgconf-pkg-config fuse3-devel gtk4-devel \
libadwaita-devel webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel
```
**Arch Linux**
```bash
sudo pacman -S --needed rust gettext pkgconf fuse3 gtk4 libadwaita webkitgtk-6.0 libsecret dbus
```
Install Rust 1.96 or newer with [rustup](https://rustup.rs) if your distribution ships an older
toolchain.
### Compile
```bash
git clone https://github.com/narrrl/proton-drive-linux.git
cd proton-drive-linux
cargo build --release --locked
po/build.sh target/locale # optional: compile the translations
```
The four binaries are in `target/release/`: `pdfs`, `pdfs-app`, `pdfs-tray` and `pdfs-prompt`.
### Build a package from the working tree
**Arch Linux:**
```bash
cd packaging && makepkg -fi
```
<a id="build-an-rpm-locally"></a>**Fedora:**
```bash
sudo dnf install -y rpm-build
rpmbuild -bb packaging/proton-drive-linux.spec \
--define "git_dir $PWD" \
--define "_rpmdir $PWD/packaging/out" \
--define "_builddir $PWD/packaging/build" \
--define "_sourcedir $PWD" \
--define "_specdir $PWD/packaging" \
--define "_srcrpmdir $PWD/packaging/out"
sudo dnf install packaging/out/x86_64/proton-drive-linux-*.rpm
```
### Using a binary outside `/usr/bin`
The service unit starts `/usr/bin/pdfs daemon`. If you installed the binaries somewhere else, for
example `~/.local/bin`, install the unit for your user and point it at your binary:
```bash
mkdir -p ~/.config/systemd/user
cp packaging/proton-drive.service ~/.config/systemd/user/
sed -i "s|^ExecStart=.*|ExecStart=$HOME/.local/bin/pdfs daemon|" \
~/.config/systemd/user/proton-drive.service
systemctl --user daemon-reload
```
To start the tray at login, copy `packaging/io.narl.proton-drive-linux-tray.desktop` to
`~/.config/autostart/`. For the app launcher entry, copy `packaging/io.narl.proton-drive-linux.desktop`
to `~/.local/share/applications/` and the icon `packaging/io.narl.proton-drive-linux.svg` to
`~/.local/share/icons/hicolor/scalable/apps/`.
## First run
### With the desktop app
1. Open **Proton Drive** from your application menu, or run `pdfs-app`.
2. Sign in with your Proton email and password, and your two-factor code if the account has one.
If Proton asks for a CAPTCHA (common on a VPN or a new network), the app shows the challenge in
a window and continues after you solve it.
3. The app enables and starts `proton-drive.service`. Your Drive appears at `~/ProtonDrive`.
The service is tied to your graphical session. It starts after you log in and stops when you log
out. Change the folder location under **Preferences → General → Location**.
### With the command line
`pdfs login` stores the session but does not enable the service, so a setup that runs the daemon
another way stays untouched. Enable the service yourself:
```bash
pdfs login # email, password, 2FA code
systemctl --user enable --now proton-drive.service
pdfs status # account, mount and queue state
```
The CLI cannot show a CAPTCHA. If sign-in reports one, sign in once with `pdfs-app` instead; the
session it stores is used by the CLI and the service as well.
## Headless and server use
The service unit is bound to `graphical-session.target`. On a machine without a desktop session,
run the daemon directly or from your own unit:
```bash
pdfs daemon # waits for a stored session, mounts, stays mounted
pdfs mount /srv/proton # mounts once at a given path, blocks until unmounted
```
Both still need a Secret Service provider. On a server, run
`gnome-keyring-daemon --unlock` or KeePassXC's Secret Service integration inside the user's D-Bus
session.
## Updating
Install the new package over the old one. The service picks up the new binary the next time it
starts; to switch at once:
```bash
systemctl --user restart proton-drive.service
```
Coming from 2.8.0 or older, re-enable the service once. Those releases hooked it into
`default.target`, which starts it before the desktop session is ready; on KDE that crashes
`ksecretd` at every login. Signing in again or toggling "Start on login" does this as well:
```bash
systemctl --user reenable proton-drive.service
```
Database migrations run automatically and only move forward. A database written by a newer
release refuses to open in an older one, so downgrading means clearing the local database (see
[Troubleshooting](TROUBLESHOOTING.md#start-over-with-a-clean-local-state)).
## Uninstalling
> [!WARNING]
> Check that nothing is waiting to upload before you remove local state. `pdfs sync queue` must be
> empty. Queued writes exist only on this computer until they reach Proton Drive.
```bash
pdfs sync queue # must list nothing
systemctl --user disable --now proton-drive.service
pdfs logout # removes the session from the keyring
```
Then remove the package (`sudo apt remove proton-drive-linux`, `sudo dnf remove
proton-drive-linux`, or `sudo pacman -R proton-drive-for-linux`), and, if you want no trace left,
the local data:
```bash
rm -rf ~/.config/proton-drive-linux ~/.local/state/proton-drive-linux ~/.cache/proton-drive-linux
```
Sign out of the device in your Proton account settings as well, and remove the computer from
**Computers** in the Proton Drive web app if you no longer want its backups.
+28
View File
@@ -0,0 +1,28 @@
# Documentation
## For users
| Page | What it covers |
|---|---|
| [Installation](INSTALL.md) | Packages, building from source, first sign-in, updating and uninstalling |
| [User guide](USER_GUIDE.md) | My files, available offline, synced folders, conflicts, Photos, sharing, Trash and versions, Computers, the search launcher, the tray |
| [Command-line reference](CLI.md) | Every `pdfs` command, grouped by task, and scripting with `--json` |
| [Configuration](CONFIGURATION.md) | `config.json`, environment variables, file locations, the systemd service |
| [Troubleshooting](TROUBLESHOOTING.md) | Diagnosing problems, common symptoms, starting over, reporting a bug |
| [Recovering on a new computer](RECOVERY.md) | What survives a lost machine and how to restore your synced folders |
| [Changelog](CHANGELOG.md) | What changed in each release |
## For contributors
| Page | What it covers |
|---|---|
| [Development guide](DEVELOPMENT.md) | Building, running and debugging from source; releasing |
| [Architecture](ARCHITECTURE.md) | Crates, read and write paths, sync engine, control protocol, threads, threat model |
| [Testing](TESTING.md) | Unit tests and the FUSE acceptance suite |
| [Translating](TRANSLATING.md) | Adding and updating translations |
| [Roadmap](ROADMAP.md) | Planned work and verification still owed |
| [Bug ledger](BUGS.md) | Every tracked defect with its analysis and status |
| [UI/UX plan](UI_UX_PLAN.md) | Design audit and notes behind the 2.x desktop app |
Also see [CONTRIBUTING.md](../CONTRIBUTING.md) and [SECURITY.md](../SECURITY.md) in the repository
root.
+119 -213
View File
@@ -1,262 +1,168 @@
# Recovery: Getting a Replacement Machine Back onto Your Device
# Recovering on a new computer
You lose the laptop. You install `pdfs` on a new one. What does it take to get the
synced folders back?
Your laptop is lost, stolen or reinstalled. This page explains what survives, and how to get
your synced folders back on the replacement machine.
This document is two things: **§2 is the runbook that works with the client as it
stands today**, gotchas included, and **§3 is the plan to reduce that runbook to
one command.** §1 is the part neither can fix — what actually died with the
machine.
- [1. What survives](#1-what-survives)
- [2. Restore your synced folders](#2-restore-your-synced-folders)
- [3. How the restore works](#3-how-the-restore-works)
- [4. Known gaps](#4-known-gaps)
---
## 1. What survives
## 1. What survives, and what does not
### On Proton Drive
### Survives (it is on Drive, or derivable)
| Thing | Where it lives |
| What | Where |
|---|---|
| Every file that finished uploading | Proton Drive |
| The Device registration itself | Proton Drive (`enumerate_devices`) |
| The device's folder tree — one folder per synced folder | Proton Drive, under the device root |
| Your account, keys, sharing | Proton |
| The computer's registration and its backup | **Computers** in the app and the web |
| One folder per synced folder, under that computer | Proton Drive |
| The machine profile: which folders were synced, to which local path, in which mode; pins; ignore patterns | `.proton-drive-linux/profile.json` under the computer's backup, end-to-end encrypted like any other file |
| Your account, keys and shares | Proton |
### Does not survive
The service rewrites the profile a short while after you add, remove or switch a synced folder,
or change a pin.
| Thing | Why | Recoverable? |
### Lost with the machine
| What | Why | Recoverable? |
|---|---|---|
| Undrained `staging/` + `recovery/` blobs | Writes the kernel accepted but that never reached Drive. They existed only on that disk. | **No.** |
| Local-only files in a mirror folder that had not uploaded yet | Same. | **No.** |
| `sync_folder` rows — which local path maps to which remote folder, and in which mode | Local SQLite (`cache.db`) | Rebuildable by hand (§2), not automatically (§3) |
| `sync_entry` baseline | Local SQLite | Not needed. An empty baseline reconciles correctly — see §2.4. |
| Pins | Local SQLite | Rebuildable by hand |
| `config.json` — mountpoint, cache budget | Local config dir | Rebuildable by hand |
| `cache.db` node/FTS/photo caches | Local SQLite | Pure cache; rebuilds itself |
| Changes still in `staging/` or `recovery/` | Writes that were accepted locally but never reached Proton Drive | **No** |
| New files in a synced folder that had not uploaded yet | Same | **No** |
| `config.json`: mountpoint, cache limit, bandwidth limits, appearance | Local config directory | Set again in **Preferences** |
| The local database and content cache | Local disk | Not needed; rebuilt from Proton Drive |
The first two rows are the only genuinely unrecoverable ones, and today nothing
warns you they are non-empty before you walk away from a machine. That is tracked
separately from this document as a pending-writes visibility gap.
Only the first two rows are real losses. Before you retire a working machine, check that
`pdfs sync queue` prints "Nothing queued" (or that **Sync → Overview** shows nothing waiting).
### Before anything else: revoke
### Before anything else: revoke the old session
The lost machine holds a decrypted content cache, a plaintext index of every node
name in your Drive (`cache.db`), and a live session in its keyring. Whether that
matters depends on whether its disk was encrypted — but the session does not.
The lost machine holds a decrypted content cache, a plain-text index of your file names and a
live session in its keyring. Whether its disk was encrypted decides how much of that is exposed,
but the session is valid either way.
**Revoke the session from your Proton account settings before restoring.** Nothing
in this client can do that for you; the daemon on the lost machine holds
credentials it will happily keep refreshing.
**Sign the lost machine out from your Proton account's security settings before you restore.**
The client cannot do that for you.
If you have also lost your password, Proton's recovery phrase is what determines
whether the data is reachable at all. That is upstream of this client entirely: if
zero-knowledge encryption worked as advertised, nobody at Proton can help.
If you have also lost your password, account recovery is Proton's matter, not this client's.
Proton Drive is end-to-end encrypted: without the password or a recovery method, the data cannot
be decrypted.
---
## 2. Restore your synced folders
## 2. The runbook that works today
### Step 1: Install and sign in
### 2.1 Install, log in
Install the client as described in [INSTALL.md](INSTALL.md) and sign in with `pdfs-app`. Signing
in from the app also starts the service.
```bash
pdfs login # stores the session in the keyring
systemctl --user start proton-drive # or let autostart do it
```
### Step 2: Continue the old computer's backup
### 2.2 Re-adopt the device — check this before anything else
Tell this machine which computer it replaces, so new changes go into the existing backup instead
of a second, empty one.
`Core::ensure_device` recovers an existing device by **matching a Linux device
whose name equals this machine's hostname**, read from
`/proc/sys/kernel/hostname`. So:
- **In the app:** open **Computers**, open the menu of the old computer and choose
**Continue This Backup Here…**.
- **From a terminal:**
```bash
hostnamectl hostname # what this machine will register as
pdfs devices list # what the account already has
```
```bash
pdfs devices list # find the old computer's UID
pdfs devices adopt UID
```
- **Hostnames match** → the new machine silently re-adopts the old device and its
root folder. This is the happy path, and it is why the rest of the runbook works.
- **Hostnames differ** → `pdfs` registers a **second device**. Nothing errors.
Your old device and all its folders are still there, but this machine is not
attached to them, and every step below will quietly create empty folders under
the new device instead.
Adoption takes effect at once and is stored in `config.json` as `device_uid`. It survives a
change of hostname. `pdfs devices adopt --clear` returns to the default, which is to match a
computer whose name equals this machine's hostname.
> **The single most important step in this document:** if the hostnames differ,
> set the new machine's hostname to match the old device's name *before* adding
> any sync folders.
> ```bash
> sudo hostnamectl set-hostname OLD-DEVICE-NAME
> ```
> There is no supported way to adopt a device by uid yet — see §3.1.
> [!NOTE]
> Without adoption, the client only finds the old computer if this machine has the same
> hostname. Otherwise it registers a new computer, and nothing points at the old backup.
### 2.3 Find out which folders the device had
### Step 3: Restore the folders
```bash
pdfs devices list # confirm the device and note its name
```
- **In the app:** open **Computers**, open the menu of **This computer** and choose
**Restore Folders…**. Confirm or change the local path of each folder.
- **From a terminal:**
The device's synced folders are the folders directly under its root in Drive.
Browse them in the GUI's Computers page, or in the Proton Drive web app under
that device. **Write down their exact names** — you need them in the next step.
```console
$ pdfs sync restore
Documents [mirror] -> /home/you/Documents (Enter to accept, 'n' to skip, or a path)
Projects [ondemand] -> /home/you/Projects (Enter to accept, 'n' to skip, or a path)
```
### 2.4 Re-attach each folder
Press Enter to accept a path, type `n` to skip the folder, or type another path. `--yes`
accepts every proposal without asking.
For each folder name you noted, create the local directory and add it:
For each folder you accept, the service creates the local directory, attaches it to the folder in
the backup, restores its mode (**Mirror** or **Online only**) and downloads it. Follow the
progress on **Sync → Overview** or with `pdfs sync list`.
```bash
mkdir -p ~/Documents
pdfs sync add ~/Documents
```
> [!TIP]
> Restore into empty or missing directories. If a directory already holds files, for example
> from another backup, the client cannot tell which side is newer. It keeps both: your local file
> is renamed to a `(sync-conflict …)` copy and the version from Proton Drive is downloaded. See
> [Conflicts](USER_GUIDE.md#conflicts).
`add_sync_folder` looks for a folder under the device root named after the local
directory's **basename**, and reuses it if found rather than creating a second
one. So `~/Documents` re-attaches to the existing `Documents`. With an empty
baseline and an empty local directory, the first reconcile classifies every remote
path as `(None, Some(remote))` with no baseline → **download**, and the folder
refills from Drive.
### Step 4: Restore the rest by hand
Three things to be careful about:
- **Available offline files.** The profile records them, but the restore does not apply them yet.
Mark them again in the app, or with `pdfs pin PATH`.
- **Preferences.** Set the cache limit, bandwidth limits, mountpoint and appearance again.
- **The basename must match exactly.** Restoring to `~/docs` when the device
folder is `Documents` does not re-attach — it creates a *new*, empty `docs`
folder in Drive and syncs nothing. The failure is silent; you just see an empty
directory. Check with `pdfs sync list` and the Computers page.
- **Add the folder empty, or accept conflict copies.** If the local directory
already has files (restored from a backup, say), there is no baseline to say
which side is newer, so both sides read as changed and the reconcile takes the
conflict arm: your local file is renamed to a `(sync-conflict <ts>)` copy and
the remote version is downloaded. Nothing is lost, but it is noisy. Restoring
into an empty directory avoids it.
- **A wholly empty local tree is now safe.** Before the audit fixes, adding a
folder whose local side was empty against a populated remote could, on a later
pass, be read as "the user deleted everything" and trash the lot. `sync.rs`'s
`guard_local_wipe` now refuses any pass where every baseline path has vanished
locally, which is exactly the restore shape.
### Step 5: Clean up
### 2.5 Restore modes, pins, and settings by hand
```bash
pdfs sync list # note the ids
pdfs sync mode <id> ondemand # for folders that were on-demand
```
Nothing records which folders were `ondemand` before, so this is from memory. Pins
and the cache budget/mountpoint likewise: set them again from the Settings page.
### 2.6 Clean up
If a duplicate device got created before you noticed the hostname mismatch:
If this machine registered a second computer before you adopted the old one, remove it:
```bash
pdfs devices list
pdfs devices rm <uid-of-the-stray-device>
pdfs devices rm UID
```
Check what is under it in the web app first. Deleting a device you actually
attached folders to will take its folders with it.
> [!WARNING]
> Removing a computer deletes its backup from Proton Drive. Check what is under it in
> **Computers → Browse Files** first.
---
### Restoring another computer's folders
## 3. The plan: make this one command
To bring folders from a computer that still exists, for example a desktop whose documents you want
on a laptop, use **Restore to This Computer…** on that computer's row, or:
The pieces all exist — device enumeration, folder-name reuse, an empty-baseline
reconcile that downloads correctly. What is missing is that the restore is
**implicit, name-based, and undiscoverable**. Each phase below removes one of
those properties. They are independent and land in this order by value.
```bash
pdfs sync restore --device UID
```
### P1 — Adopt a device explicitly, by uid
The folders stay part of the other computer's backup and sync in both directions. This machine
does not take over that computer's backup.
**Problem:** device identity is inferred from the hostname. A new laptop with a
new name silently forks a second device, and the failure looks like "my files
didn't come back" rather than an error.
## 3. How the restore works
- Add `Request::AdoptDevice { uid }`: validate the uid against
`enumerate_devices`, write the `StoredDevice` row, and refuse if any
`sync_folder` rows already reference a different device.
- `pdfs devices adopt <uid>`, and an "Adopt this device" action on rows in the
GUI's Computers page that are not the current machine.
- Make the fork *visible* rather than silent: when `ensure_device` is about to
create a device and the account already has other Linux devices, log a warning
naming them, and surface it in `pdfs status`.
This section is for the curious and for anyone debugging a restore.
Keep hostname matching as the default. It is right for the common case; it just
must not be the *only* way.
- **Which computer the machine is.** At start, the service uses `device_uid` from `config.json`
when it is set. When it is not, it looks for a Linux computer in the account whose name equals
the hostname, and registers a new one when there is none. An adopted UID that no longer exists
in the account is reported in the log instead of being replaced silently.
- **What can be restored.** `pdfs sync restore` lists the folders directly under the computer's
backup, except `.proton-drive-linux`. Folders already synced on this machine are skipped.
- **Proposed paths.** When the profile names a local path whose parent directory exists on this
machine, that path is proposed. Otherwise the proposal is `~/<folder name>`. The service never
writes to a path you have not confirmed.
- **Attaching by UID.** The restore binds each local path to the remote folder's UID, not to its
name. Restoring `Documents` into `~/docs` works.
- **Downloading.** A restored folder starts with an empty sync baseline. Against an empty local
directory, every remote file counts as new and is downloaded. Files present on both sides with
no baseline become conflict copies, as described above.
- **The wipe guard.** A sync pass in which every previously synced file has vanished locally is
refused instead of being mirrored to Proton Drive as a mass deletion. A folder that was emptied
by accident therefore does not empty the backup.
- **The profile format.** `profile.json` carries a `version` field. A client refuses a profile
written by a newer, incompatible client instead of applying part of it.
### P2 — Discover and re-attach the device's folders
## 4. Known gaps
**Problem:** §2.3 and §2.4 are "remember what you had, then retype it exactly."
- Add `Request::ListDeviceFolders { device_uid }` — enumerate the device root's
child folders and report name, node uid, child count, and total size. This is
the "what did this machine sync?" answer that today requires the web app.
- Add `Request::AttachSyncFolder { remote_uid, local_path, mode }` — the
re-attach counterpart to `AddSyncFolder`. It binds a local path to a **specific
remote uid** rather than resolving by basename, which removes the silent
`~/docs` vs `Documents` failure entirely.
- CLI: `pdfs sync restore [--device <uid>] [--into <dir>]` — list the device's
folders, propose `<dir>/<name>` for each (default `$HOME`), and attach the ones
the user confirms.
- GUI: a "Restore folders from this device" flow on the Computers page, with a
per-folder local-path picker defaulting to `~/<name>`.
Refuse to attach into a non-empty directory without an explicit
`--allow-nonempty`, so the conflict-copy noise of §2.4 is opt-in rather than a
surprise.
### P3 — Back the profile up to Drive
**Problem:** modes, pins, mountpoint, and cache budget exist only on the dead
machine, so even a perfect P2 restores folders in the wrong mode with no pins.
Write a small JSON document into a hidden folder in the user's own Drive — say
`.proton-drive-linux/profile.json` — containing, per device:
- sync folders: remote uid, the local path last used, and mode
- pinned node uids
- mountpoint and cache budget
It rides on Drive's own encryption, so this introduces no new crypto and no new
place for plaintext to sit. Rewrite it whenever any of those change, debounced.
On a fresh install, after login: *"Found settings from `<device>`, last updated
`<date>`. Restore?"* Node uids are stable across machines, so pins restore
exactly. Local paths need remapping when the username or home differs — show the
old path, default the new one to `$HOME/<basename>`.
Store the last-used local path only as a *hint*. Restoring a profile must never
write to a path the user has not confirmed in this session.
### P4 — Make the unrecoverable visible
Nothing above helps with §1's undrained writes, because by then it is too late.
The mitigation is to make the pending queue impossible to miss *before* the
machine is lost:
- pending count and bytes in `pdfs status`, using the existing `pending` map and
`TransferRegistry`
- a tray icon state and a GUI banner while the queue is non-empty
- a warning on logout or shutdown while it is non-empty
### P5 — Verification
None of the above is trustworthy without exercising the actual shape of the
disaster. The end-to-end test is: register a device, sync a folder, upload
content, **delete the local state directory and cache entirely** (simulating the
lost machine), restart, restore, and assert the tree comes back byte-identical.
That is a two-account or at least two-state-dir integration test against a live
account, not a unit test. Until the mock-backend problem described in
`audit-plan.md` is solved, it stays manual — but it should be run, and its result
recorded here, before P1–P3 are called done.
---
## 4. Current status
| Phase | State |
| Gap | Effect |
|---|---|
| Manual runbook (§2) | Works, with the hostname and basename caveats |
| P1 explicit adoption | Not started |
| P2 discover + attach | Not started |
| P3 profile on Drive | Not started |
| P4 pending visibility | Not started |
| P5 restore drill | Never run |
| Pins are backed up but not restored | Mark files available offline again by hand |
| No warning at shutdown or sign-out while changes are queued | Check `pdfs sync queue` before you retire a machine |
| The full disaster drill (delete all local state, restore, compare byte for byte) has not been run against a live account | The individual steps are tested; the whole sequence is not |
Planned work is tracked in [ROADMAP.md](ROADMAP.md).
+72
View File
@@ -0,0 +1,72 @@
# Roadmap
Planned work and open verification, as of 2.5.0. Individual defects and their status live in
[BUGS.md](BUGS.md), which is the authoritative ledger; this page groups what is left by theme.
Nothing here is a promise of a date.
- [Data safety and recovery](#data-safety-and-recovery)
- [Features](#features)
- [Performance and correctness](#performance-and-correctness)
- [Verification still owed](#verification-still-owed)
- [Release assurance](#release-assurance)
## Data safety and recovery
| Item | State | Notes |
|---|---|---|
| Protect queued changes at shutdown | Not started | A systemd inhibitor lock while `staging/` holds unsent data, a warning when signing out, and a blocking `pdfs sync flush`. Today the queue is visible through `pdfs sync queue`, the tray and **Sync → Overview**, but nothing stops a shutdown (B66) |
| Restore pins from the machine profile | Not started | `profile.json` records pins; `sync restore` does not apply them yet |
| Profile backup health | Not started | A failed profile upload is a `WARN` in the log and nothing in the UI (B68) |
| Rotated credentials survive a keyring failure | Open | Surface and retry keyring write failures so a rotated single-use refresh token is not lost (B61) |
| Encrypt local state at rest | Not started | SQLCipher for `cache.db` and an AEAD for the content cache, keyed from the keyring. `sdk_cache.db` is already encrypted. See [ARCHITECTURE.md §8](ARCHITECTURE.md#8-threat-model-what-this-client-writes-to-disk-in-plaintext) |
## Features
| Item | State | Notes |
|---|---|---|
| File-manager integration | Not started | Context-menu actions (available offline, share link, versions) and sync badges for Nautilus, Dolphin and Thunar |
| Terminal UI | Not started | `pdfs tui` with transfers, sync state and logs |
| Multiple accounts | Not started | Separate config, state and mountpoint per profile, one daemon each |
| Mount options | Not started | Read-only mount, `allow_other` |
| Symbolic links | Not started | Proton Drive has no link type; links would need a placeholder format. Until then, creating one fails with `EOPNOTSUPP` |
| Add someone else's photo to your album | Blocked | Needs `copyPhoto` in the SDK |
| Photo name search | Not started | Photos search by date and filters only |
| Automatic photo upload from a folder | Not started | |
| Faster Google Photos import | Not started | Uploads run one at a time. Trash folders of locales missing from `TRASH_FOLDERS` import as photos; motion-photo `.MP` parts are skipped |
| Content-addressed block cache | Idea | Store blocks by hash to deduplicate identical content |
## Performance and correctness
| Item | Notes |
|---|---|
| Saturating stream-ring accounting | `self.bytes -= …` in `crates/pdfs-fuse/src/reads.rs` should saturate so an accounting drift cannot wrap in release builds |
| Block geometry learned one read late | The first read of a file plans on 4 MiB blocks; accepted as a bounded cost (B87) |
| Path-based commands under secondary mounts | `pin`, `rename`, `move`, `rm` and sharing cannot name a path inside an on-demand synced folder yet (B88) |
| Unicode normalization | NFC and NFD names are distinct, deliberately (B36) |
| Pipelined node enumeration | Faster cold listings of large folders |
## Verification still owed
Fixes marked *unverified* in [BUGS.md](BUGS.md) have unit or offline coverage but have not been
driven against a live account. The larger outstanding runs:
- **Disaster-recovery drill.** Register a device, sync a folder, delete all local state, restore
with `pdfs sync restore` and compare the tree byte for byte. See [RECOVERY.md](RECOVERY.md).
- **Second-account sharing.** Shared trees on a foreign volume, accepting invitations, changing
roles, the viewer cases of `regression B34`, and an editor writing into someone else's share.
- **Fault injection.** Network loss mid-upload, a full disk during staging, and crash points
across the write publication path (after each fsync, rename and queue commit).
- **CAPTCHA sign-in** against a real verification request (B8).
- **Managed live mode matrix** before each release:
`scripts/fuse-acceptance.sh --managed-live <empty-a> <empty-b>`. See [TESTING.md](TESTING.md).
- **The overnight hang** (B91) is not diagnosed yet. The supervisor and `pdfs diagnostics` exist to
catch the next occurrence.
## Release assurance
| Item | State |
|---|---|
| Tag, workspace and PKGBUILD version agreement | Enforced by the release workflow |
| Clean-install test of the `.deb` and `.rpm` in a VM | Pending (B62) |
| Protocol version identity between front ends and daemon | Open (B63) |
| SBOM, provenance and checksums for release artifacts | Open (B65). The RPM is GPG-signed |
+322 -23
View File
@@ -25,36 +25,151 @@ Proton:
scripts/fuse-acceptance.sh --offline-only
```
Kernel/FUSE behavior and remote convergence need a real mount. Live testing is
explicitly opt-in and always runs the account-free contract first. It runs
destructive POSIX operations only below a fresh `pdfs-acceptance-*` directory.
It refuses paths that are not FUSE mounts and removes its directory on success,
failure, or interruption.
Kernel/FUSE behavior and remote convergence need a real mount. With no
arguments, the script tests the account the running service is logged in to:
Use a dedicated test account without irreplaceable data:
```console
scripts/fuse-acceptance.sh # every case, every mount kind
scripts/fuse-acceptance.sh --quick # one on-demand/mirror pair instead of four
```
An account run does this, in order:
1. Runs the account-free contract as the reference.
2. Runs the contract in a fresh `pdfs-acceptance-<id>` folder in My files and
diffs it against the reference. After the last case it reads back a file
the run wrote and records its digest. A read that fails there, as one did
with the link down (B119), is a failed result named "a file of this run
reads back at the end", and the cleanup still runs.
3. Registers two new sync folders under `~/.cache/pdfs-acceptance/run-<id>/`
and runs the contract and the move cases in every on-demand/mirror pairing.
4. Removes everything it created, locally and remotely, and reports a
`cleanup` result. The run fails if anything is left behind.
Cleanup identifies remote folders by uid, never by name alone, so nothing of
yours is touched. It removes each sync folder with `sync rm --delete-remote`,
trashes the My files folder, and then deletes both permanently from the trash.
It only counts a folder as gone once a fresh trash listing no longer shows it.
It then reads `pdfs sync queue` and counts any queued op that still names one
of its folders as left behind too, since such an op keeps its staged bytes and
retries forever (B102).
Ops that were queued before a target starts are not the run's. The run prints
one `NOTE` line naming them, and the queue waits ignore them. They are told
apart by id and queue time together, since SQLite can give a new op the id of
one that has drained.
A write is queued when the kernel releases the file, and `close(2)` returns
without waiting for that. A case that reads `pdfs sync queue` right after
writing therefore waits for the ops it expects (`Daemon.queued`) instead of
reading once. "regression B102" saw one revision of two before it did.
Cleanup also runs on a failure, Ctrl-C, `SIGTERM` or `SIGHUP`. While it runs,
two more interrupts are ignored; a third abandons it.
Each run records what it is about to create in
`~/.cache/pdfs-acceptance/run-<id>/manifest.json` before it creates it, and
holds a lock on that directory. If a run dies without cleaning up (`kill -9`, a
hard timeout, a power cut), the next run finds the unlocked manifest and
finishes that cleanup first. It then removes stray `pdfs-acceptance-<32 hex>`
folders that no live run claims: in My files (older than
`PDFS_ACCEPTANCE_REAP_AGE`, default one hour), under the acceptance home, and
in the trash. A run does not start while that reap leaves anything behind.
The run creates and destroys real remote folders, so a dedicated test account
without irreplaceable data is still the safest target.
`--live PATH...` tests mounts you set up yourself and leaves the sync folder
configuration alone. It runs destructive POSIX operations only below a fresh
`pdfs-acceptance-*` directory, refuses paths that are not FUSE mounts, and
removes its directory on success, failure, or interruption:
```console
scripts/fuse-acceptance.sh --live /mnt/on-demand-testmount
```
### Reading the output
A run opens with what it is about to test: the mode, the `pdfs` binary and its
version, Python and the kernel, the per-case limit, the options and the start
time. Each target then names its root, the storage behind it, the daemon it
talks to and how many cases it runs, followed by one line per case:
```
==> live FUSE /mnt/testmount
-> root /mnt/testmount/pdfs-acceptance-…
-> storage FUSE mount /mnt/testmount (fuse.protondrive)
-> daemon pdfs 2.8.3, user@proton.me, online, 0 queued ops
-> cases 45 of 45
[ 1/45] creation and open flags .................................. ok 0.41s
…
[21/45] throughput floors ........................................ ok 48.20s
write 31 MiB/s, read 54 MiB/s, 4 KiB writes 12.0 MiB/s, metadata 9 ops/s
…
[43/45] regression B115: pdfs ls reports a file's real size ...... ok 6.12s
-> reference matches the ordinary filesystem in 412 compared observations
-> result 44 passed, 1 skipped in 11m02s
```
A case ends as `ok`, `FAIL`, `TIMEOUT`, `skip` when it does not apply to the
target (no daemon, no share at that role), or `known` when an `open B<n>` case
reproduced a bug that is still open (see below). The dimmed lines under a case
are its details: the reason for a skip, measured rates, a hash. A failed
`regression B<n>` case also points to its entry in `docs/BUGS.md`. When a case
prints something itself, such as a service restart, the case line is printed
again once the case ends.
The run ends with a table of every target's counts and time, the report paths,
the slowest cases, the cases over `--budget`, the known issues that still
reproduce, any errors the daemon logged, and each failure with its target. The
last line is `PASS:` or `FAIL:` and the totals. Colour is used only on a
terminal and never with `NO_COLOR` set.
### Edge cases and performance
Besides the syscall contract, the suite covers:
- **Block boundaries.** Files of 0 bytes up to two 4 MiB blocks plus 3 bytes,
including every size next to a boundary, then overwrites that grow, shrink
and reshape them.
- **Unusual names.** A 255-byte name, NFC and NFD forms of the same text,
newline, tab, emoji, characters Windows forbids, leading and trailing
spaces, and a case-only rename.
- **A deep tree and a wide directory.** 24 levels of nesting renamed at the
top; 128 entries created concurrently, concurrent unlinks, then `rm -rf`.
Online, each create is a round trip to the server, and the kernel runs
creates in one directory one at a time. So the wide case gets twice the
per-case timeout and notes the time per create as `wide.create_ms`.
- **Rename patterns.** Swaps, chains, log rotation, replacing a file under an
open reader, the `EISDIR`/`ENOTDIR`/`ENOENT` refusals, and a folder moved out
and back.
- **Handle coherency.** Unsynced writes seen by a second handle, two `O_APPEND`
writers, 30 rewrite generations, and a truncate under a reader.
- **Throughput.** 64 MiB written and read sequentially, 1 MiB in 4 KiB writes,
and metadata operations on 100 files. The rates are printed. The case fails
below `PDFS_ACCEPTANCE_MIN_MIBPS` (default 10) or `PDFS_ACCEPTANCE_MIN_OPS`
(default 20 operations per second, or 3 on a FUSE mount, where each create
and unlink waits for Drive before it returns).
### The mount is diffed against an ordinary filesystem
The local reference run is not only a self-test of the runner. Every case
records the facts it established, and a live run is compared against those
recordings, so a case fails when the mount *differs* from an ordinary
filesystem — not only when it raises. The report names the exact key:
filesystem — not only when it raises. The target's `reference` line names the
exact key:
```
[DIVERGENCE] live FUSE /mnt/testmount does not match the reference filesystem:
! creation and open flags.truncated.size: live=6 reference=0
-> reference 1 observation of 412 differs from the ordinary filesystem:
! creation and open flags.truncated.size: live FUSE /mnt/testmount=6 reference=0
```
Facts are recorded in one of two ways, chosen in the test body:
- **compared** — a claim about filesystem semantics. The mount must agree.
- **noted** — context that legitimately differs: inode numbers, block counts,
reported mode bits, and which optional syscalls exist at all. Reported under
`[capabilities]`, never a failure.
reported mode bits, and which optional syscalls exist at all. Listed under
the target's `accepted` line, never a failure.
Because the choice is made at each call site, the set of accepted divergences
is visible in the test that accepts it. There is no separate allowlist to drift
@@ -78,10 +193,104 @@ in-flight `.crdownload` must not be sealed as a revision) and B74 (a file
renamed after close must keep its content) need a real mount and a reachable
daemon; they are skipped, not failed, without one.
B74 is currently **failing** — it is a live data-loss defect, not a regression
guard, and it was found by exactly this mechanism: the B70 case failed, and
narrowing it produced a smaller reproduction that has its own case now. Expect a
red run against a real mount until it is fixed.
**B100** (a copied tree reads back at its exact sizes) copies 48 files of
assorted sizes into the mount at once, as `cp -r` does. On a live mount it
waits for the queue, checks every size and byte, runs `pdfs refresh` on each
folder to drop the cached listing, and checks again. The refresh is what
brought back the provisional sizes the bug reported.
**B101** (a write during its create's upload reaches Drive) holds new files open for
different times before writing, so their empty creates land before, during and after the
bytes arrive, then checks every file after the queue drains.
**B113** (moving a file before its upload makes no conflict) pauses sync with
`pdfs sync pause --for 10m` and writes three files. While their uploads are held,
it renames one, moves one into a folder, and renames and moves the third. It then
resumes sync, waits for the queue, and checks the bytes and names and that no
conflict copy appeared. It skips when sync was already paused. If the run dies
before the resume, the pause ends by itself after ten minutes.
**B88** (`pdfs mkdir`, `rename` and `rm` by local path) creates, renames and
removes a folder and a file by their paths in the mount. Then it checks the
refusals: `pdfs rename` of the mount's root to its own name, and `pdfs rm` of a
file outside every mount. `pdfs rm` of the root itself is tried only on a sync
folder the run created. In a mirrored sync folder all three commands must be
refused instead, and nothing may change.
**pdfs output into a closed pipe** runs `pdfs ls`, `pdfs --json ls` and
`pdfs status` with stdout on a pipe whose reader is already gone. Each must exit
0 with nothing on stderr, as `pdfs ls | head -1` needs.
**B102** (removing a folder drops the uploads queued inside it) uploads a file
into each of two nested folders, pauses sync and rewrites both. While the
revisions are held, it removes one folder with `pdfs rm` and the other through
the mount. Neither revision may stay queued, and neither folder may come back
after the resume. It skips when sync was already paused.
**B94** (a deleted `.part` file leaves nothing queued) writes a `.part` file,
checks that its create is parked, and unlinks it. The parked op must leave the
queue within 30 seconds. The sweep that lets an hour-old park through is not
driven.
**B95** (a `.tmp` file uploads at once) writes `index.tmp` and `preview.temp`
and waits for the queue, which fails at once on a parked op. `pdfs ls` must
list both, and their bytes must read back.
**B111** (renaming a file straight back works) renames a file there and back
eight times through the mount. On a mount it also does so three times with
`pdfs rename`, after the queue is empty, so the renames race nothing but Drive.
It runs against the reference too.
**B121** (a file renamed and moved at once lands) renames a file, moves it into
a subfolder and renames it back across folders through the mount, with no pause
between the steps. On a mount it then runs `pdfs rename` and `pdfs move` back to
back. The file must end up where it was sent, with its bytes, and the case
reports how long each half took. It runs against the reference too.
**B98** (an upload superseded by a write ends) writes 64 MiB and waits for its
upload to show in `pdfs transfers`, then appends to the file. The queue must
drain, the old upload must leave the transfer list, and the file must hold
every byte with no conflict copy. It skips when the upload is never seen on
the wire, and it gets twice the per-case timeout.
**B80** (`pdfs search` finds a new file after a rewrite) writes a file whose
name carries a unique token, waits for the queue, rewrites it and waits again.
`pdfs search` must find it within 30 seconds.
**B48** (an emptied folder can be removed at once) writes twelve files into a
folder, lists it, and moves one more in from elsewhere. It unlinks every
child, runs `pdfs refresh` on a mount, and checks that nothing is listed and
that `rmdir` works. It runs against the reference too.
**B114** (a name `pdfs rm` or `rename` took away stops resolving) looks up two
files, removes one with `pdfs rm` and renames the other with `pdfs rename`. At
once, neither old name may be listed or resolve.
**B115** (`pdfs ls` reports a file's real size) writes files of 6 bytes, 4097
bytes and one block plus 7, and compares the sizes `pdfs ls` lists with the
written ones, after the upload and again after `pdfs refresh`.
**B116** (trashing a folder drops the creates queued in its subfolders) makes
`b116/sub/deeper` and waits for Drive. With sync paused, it writes a `.part`
file there and renames it to its final name, so its create is due but held.
It then runs `pdfs rm b116`, and the create must be gone from the queue. It
restores the folder with `pdfs restore` before it resumes sync. If the file
ends up in the mount's root anyway, the case fails and moves the file to the
trash.
**B117** (a moved file's queued upload keeps its path) rewrites an uploaded
file with sync paused and moves it with `pdfs move`. `pdfs sync queue` must
list the upload under its new path, and the journal must not show the upload
put off with "pending operation deferred". The bytes must arrive at the new
path with no conflict copy. Without `journalctl`, only the queue is checked.
**B119** (an uploaded file stays cached) writes a small file and one of a block
plus 119 bytes, waits for the queue, and expects `pdfs ls` to list both as
cached, so reading them does not need the network.
B74 was found by exactly this mechanism: the B70 case failed, and narrowing it
produced a smaller reproduction that got its own case. The defect is fixed and
live-validated (2026-07-26); the case now guards against its return.
**B79** (an on-demand device folder accepts writes) discovers its target from
`pdfs locations --json` and skips when this machine has no mounted on-demand
@@ -114,13 +323,61 @@ nothing: the local attribute and content caches will happily return the bytes
that were just written even when nothing reached Drive, which is precisely how
B74 stayed invisible.
### Bugs that are still open
Cases named `open B<n>` check a bug that `docs/BUGS.md` still lists as open.
While the bug reproduces, the case ends as `known`: it is listed at the end of
the run, but does not fail it. So a known bug does not hide new failures, and
the run notices when a fix lands, because the case then passes. Rename it to
`regression B<n>` then, and fail it with `check` instead of `KnownIssue`. No
bug is open at the moment. B114 to B117 started as such cases.
### What a run verifies
A full account run with `--journal-check` checks these `docs/BUGS.md` entries,
many of them fixed but not yet verified:
- B48, B80, B94, B95, B98, B100, B101, B102, B111, B114, B115, B116, B117, B119
and B121, each with its own `regression B<n>` case.
- B118, with "application workloads" and "block boundaries", which failed on
it. It needs Drive to be slow to list a new node, so a clean run is weak
evidence.
- B120, with `--journal-check`, but only in a run that goes offline while it
writes. A mkdir made online goes to Drive at once, so only then does a
folder land from the queue.
- B86, with the mode-switch check in the full matrix.
- B104, B106, B108 and B109, with the cases that found them: "rename
patterns", "application workloads" and the account run's own setup.
B105, B107 and B110 are races. The suite hit each of them once, so a clean run
is weak evidence for them. `--journal-check` reports B110's stall warning, and
every case that writes checks for B105's conflict copies.
The suite does not drive:
- the GUI and the photo gallery (B81, B92, B93, B97);
- states it cannot set up: a `parent_uid` cycle (B82), staged bytes that name
no node (B96), a stuck trash refresh (B103), conflict copies left from an
earlier start (B99), a foreign FUSE mount in the local-file index (B90),
revisions without 4 MiB blocks (B85), and several shares from one owner
(B112);
- B94's hour-long expiry of a park;
- load and failure injection (B22, B27, B30, B37, B42).
Check those by hand, or with the unit tests their entries name.
### Timeouts, reports, and hung mounts
A wedged FUSE operation cannot be interrupted from Python, so each case runs
under a soft alarm (`--timeout`, default 180s) that reports the case and lets
cleanup run, backed by a hard stop that dumps every thread's stack and exits.
That backstop is the only way to learn *where* a mount hung. A timeout ends the
run: later cases against a wedged mount produce noise, not information.
That backstop is the only way to learn *where* a mount hung. After a timeout the
run checks whether the mount still answers a `stat` and a listing within 15
seconds. If it does, the target goes on with its next case, so one slow stretch
of the link does not cost every later case. If it does not, the target's later
cases are not run: against a wedged mount they produce noise, not information.
Cases that make many round trips by design ("application workloads", "unusual
but legal names", "a wide directory" and "regression B98") get twice the limit.
```console
scripts/fuse-acceptance.sh --live /mnt/testmount \
@@ -128,7 +385,8 @@ scripts/fuse-acceptance.sh --live /mnt/testmount \
```
`--report-json` also writes every recorded observation, which is what to attach
to a bug report. `--budget SECONDS` flags cases that pass but got slower —
to a bug report. A known issue has the status `known` there; JUnit has no such
status, so `--report-junit` reports it as skipped. `--budget SECONDS` flags cases that pass but got slower —
B5 was a 186 ms-per-call regression that a pass/fail suite could not see.
`--list` prints every case and which targets it runs against, and `--fail-fast`
stops at the first failure.
@@ -136,14 +394,21 @@ stops at the first failure.
`--journal-check` fails the run if `proton-drive.service` logged any error while
it ran. A suite can pass every assertion while the daemon logs failures behind
it; that has happened, and it was caught only by reading the journal by hand.
Two warnings count as errors too: "queued write conflicts; keeping a conflict
copy" (how B113 showed) and "a fuse worker has held the same job for a long
time" (how B110 showed).
Abandoned `pdfs-acceptance-*` roots from a crashed run are removed at startup,
but only if they match the exact generated name and are over an hour old
(`PDFS_ACCEPTANCE_REAP_AGE`), so a concurrent run is never disturbed.
With `--live`, abandoned `pdfs-acceptance-*` roots from a crashed run are
removed at startup, but only if they match the exact generated name and are
over an hour old (`PDFS_ACCEPTANCE_REAP_AGE`), so a concurrent run is never
disturbed. An account run instead uses the manifest and reaper described
above.
### Automated mode matrix
The managed live runner accepts two existing, empty, unmounted directories. It
An account run drives this matrix itself, in folders it creates. To use
directories of your choice instead, the managed live runner accepts two
existing, empty, unmounted directories. It
registers both as new sync folders, waits for initial synchronization, then
tests all four pairs in order: on-demand/on-demand, on-demand/mirror,
mirror/on-demand, and mirror/mirror. Every transition must become idle, and an
@@ -160,6 +425,38 @@ in `pdfs sync list`. Use `PDFS_ACCEPTANCE_SYNC_TIMEOUT` to extend transition
timeouts and `PDFS_ACCEPTANCE_PDFS` to select a non-installed CLI binary, for
example `target/debug/pdfs`.
After the contract, each mode pair also runs the move cases (`--list` shows
them as `managed`). They move trees between the two folders in both
directions, move several sources in one `pdfs move`, and pass a tree through
My files and out again. Every case waits for Drive before and after the move.
It then checks that the tree is only at the destination, that every byte
arrived, and that neither side made conflict copies. The refusal cases prove
that nothing moves and nothing is lost when:
- the destination already has the name;
- a folder would move into itself;
- a mirror source holds something Drive lacks (a symlink);
- a file in an on-demand source is still open for writing.
The My files case writes a test folder into your real My files and removes it
afterwards. It skips when the daemon reports no My files mount, or when the
locations are on different volumes.
Before a mirror folder switches to on-demand, the runner writes a new file into
it and syncs. Once the folder is mounted, the file must be listed with its
bytes (B86). Each switch reports a `regression B86` result under a separate
`mode switches` target. The bug only shows in a folder that was mounted in an
earlier pairing, because that mount's listing is stale. That happens in the
third pairing, so `--quick` runs the check but does not replay the bug.
In every pairing, each folder must still hold the file the runner wrote into it
at setup, byte for byte. These checks report under `mode switches` as well. A
switch to on-demand frees the local copy, so these reads go to Drive. If the
link drops, the daemon fails such a read with EIO once its two-minute read
budget runs out. That is correct, so the runner waits until the daemon reports
online again, tries once more and notes the retry under the result. Only the
second try decides the result.
### Durability drill
`--durability` restarts `proton-drive.service` in the middle of the suite and
@@ -171,7 +468,9 @@ interrupts a running daemon; `PDFS_ACCEPTANCE_UNIT` selects a different unit.
### Selecting cases
During development, `PDFS_ACCEPTANCE_ONLY` runs tests whose descriptive name
contains the supplied text (for example `PDFS_ACCEPTANCE_ONLY=namespace`). A
contains the supplied text, ignoring case (for example
`PDFS_ACCEPTANCE_ONLY=namespace`). Separate several texts with commas to run
every case that matches any of them (`PDFS_ACCEPTANCE_ONLY=B116,B117`). A
release acceptance run must leave it unset so the complete contract executes.
A filter naming a live-only case (every `regression B<n>` is one) selects nothing
+204
View File
@@ -0,0 +1,204 @@
# Troubleshooting
Start with `pdfs diagnose`, then find your symptom below. If nothing here helps,
[open an issue](https://github.com/narrrl/proton-drive-linux/issues/new/choose) with the
information listed under [Reporting a problem](#reporting-a-problem).
- [First checks](#first-checks)
- [Signing in](#signing-in)
- [The Proton Drive folder](#the-proton-drive-folder)
- [Uploads and sync](#uploads-and-sync)
- [Disk space](#disk-space)
- [Desktop integration](#desktop-integration)
- [Start over with a clean local state](#start-over-with-a-clean-local-state)
- [Reporting a problem](#reporting-a-problem)
## First checks
### `pdfs diagnose`
Checks the installation and prints a report. It does not need the service, because the state
worth diagnosing is often the one where the service will not start.
```console
$ pdfs diagnose
Paths
[ok ] state dir: /home/you/.local/state/proton-drive-linux
[ok ] cache dir: /home/you/.cache/proton-drive-linux
[ok ] state dir writable: write and fsync succeeded
[ok ] database: /home/you/.local/state/proton-drive-linux/cache.db (473.1 MiB)
FUSE
[ok ] /dev/fuse: /dev/fuse
[ok ] unmount helper: fusermount3 or fusermount on PATH
Account
[ok ] keyring session: you@proton.me
Daemon
[ok ] control socket: /home/you/.local/state/proton-drive-linux/control.sock
[ok ] daemon responding
[ok ] active mount: /home/you/ProtonDrive (fuse.protondrive)
[ok ] network: online
[ok ] queued writes: none
[ok ] database integrity: SQLite integrity_check passed
[ok ] configured mountpoint: /home/you/ProtonDrive
No problems found.
```
It exits non-zero when a check fails, so it also works in a health-check script.
### Logs
```bash
journalctl --user -u proton-drive.service -f # follow the service
journalctl --user -u proton-drive.service -b -p warning # warnings since boot
```
For more detail, raise the log level with a drop-in (see
[CONFIGURATION.md](CONFIGURATION.md#the-systemd-service)) and restart the service.
### Service state
```bash
systemctl --user status proton-drive.service
pdfs status
```
## Signing in
**"Proton is asking for a CAPTCHA"** — Proton wants human verification, usually on a VPN or an
unfamiliar network. The CLI cannot show it. Sign in with `pdfs-app`, which displays the challenge;
the stored session then works for the CLI too.
**Sign-in works but is forgotten after a restart** — no Secret Service provider is running, or its
keyring is locked. Install and unlock GNOME Keyring, KWallet or KeePassXC (with Secret Service
enabled). `pdfs diagnose` shows `keyring session` as failing in that case.
## The Proton Drive folder
> [!TIP]
> When the service hangs, any program that touches the Proton Drive folder can hang with it,
> including `ls` and your shell's prompt. Prefix exploratory commands with `timeout 5`.
**`Transport endpoint is not connected`** — the service stopped without unmounting. Unmount and
restart:
```bash
fusermount3 -uz ~/ProtonDrive
systemctl --user restart proton-drive.service
```
**The folder is empty or missing** — check that the service runs and is signed in
(`pdfs status`). The service waits for a stored session before it mounts.
**Everything in the folder hangs** — find out what the service is doing, then restart it:
```bash
timeout 10 pdfs diagnostics
systemctl --user restart proton-drive.service
```
The service watchdog restarts a service that stops answering for two minutes by itself. Attach the
`pdfs diagnostics` output to a bug report; it shows which worker is stuck.
**A change made on another device does not show up** — changes arrive through Proton's event
stream, normally within seconds. To fetch a folder again at once:
```bash
pdfs refresh Documents
```
**"Permission denied" when writing** — the item is in a share where you are a viewer, or under
`Shared with me/`, which is read-only as a folder. Check your role with `pdfs shared-with-me`.
**A program says "Operation not supported"** — Proton Drive cannot store symbolic links, hard
links, device files or FIFOs. The error is intentional. Use a synced folder for
projects that need them, or exclude those paths.
## Uploads and sync
**Something is stuck uploading** — list the queue and retry:
```bash
pdfs sync queue
pdfs sync retry # every failed operation
pdfs sync retry 42 # one operation
```
The app shows the same list on **Sync → Overview**. An operation that keeps failing shows its
error there.
**A file in a synced folder does not upload** — check that:
- it is not matched by `.pdfsignore` or `ignore_patterns`;
- no program still holds it open for writing (such files wait until closed);
- sync is not paused (`pdfs status`, tray);
- the folder has no error on **Sync → Folders**.
`pdfs sync now` starts a pass immediately.
**Files named `(sync-conflict …)` appeared** — the file changed in two places at once. Nothing was
lost; see [Conflicts](USER_GUIDE.md#conflicts) to pick a version.
**Photos show the wrong date after a Google Takeout import** — run
`pdfs redate-photos --dry-run`, see [USER_GUIDE.md](USER_GUIDE.md#importing-from-google-photos).
## Disk space
| What | Check | Reduce |
|---|---|---|
| Content cache | `pdfs cache inspect` | Lower **Preferences → Storage → Cache size limit**, or `pdfs cache clear` |
| Available-offline files | `pdfs pins` | `pdfs unpin PATH` |
| Metadata database | `pdfs cache inspect` (reclaimable) | `pdfs cache vacuum` |
| Synced folders | `pdfs sync list` | Switch large folders to online only: `pdfs sync mode ID ondemand` |
`pdfs cache vacuum` locks the database while it runs and needs free space for a second copy of it.
## Desktop integration
**No tray icon on GNOME** — install and enable the AppIndicator and KStatusNotifierItem Support
extension. Other desktops show the icon natively. If you hid it from its own menu, turn it back on
under **Preferences → General → Appearance**.
**The search launcher does nothing** — `pdfs-prompt` needs the service. Run it from a terminal to
see its error. For `--fzf`, install `fzf` and a supported terminal.
**The app shows "Not responding"** — the service runs but does not answer within five seconds.
See [Everything in the folder hangs](#the-proton-drive-folder).
## Start over with a clean local state
This removes the local database and cache so they are rebuilt from Proton Drive. It is the fix for
a damaged database and the only way to downgrade to an older release.
> [!WARNING]
> Content in `staging/` and `recovery/` has not reached Proton Drive yet. Deleting it loses those
> changes permanently. Wait until `pdfs sync queue` is empty.
```bash
pdfs sync queue # must list nothing
systemctl --user stop proton-drive.service
mv ~/.local/state/proton-drive-linux ~/.local/state/proton-drive-linux.bak
mv ~/.cache/proton-drive-linux ~/.cache/proton-drive-linux.bak
systemctl --user start proton-drive.service
```
Your session is in the keyring and your settings in `~/.config`, so you stay signed in. Synced
folders are not re-attached automatically; restore them with `pdfs sync restore` as described in
[RECOVERY.md](RECOVERY.md#2-restore-your-synced-folders). Delete the `.bak` directories once
everything is back.
## Reporting a problem
Include:
- `pdfs --version`, your distribution and desktop;
- the output of `pdfs diagnose`;
- relevant log lines from `journalctl --user -u proton-drive.service`;
- for a hang, the output of `timeout 10 pdfs diagnostics`;
- the steps that reproduce it.
Logs contain file names. Remove anything private before posting. Report security problems
privately as described in [SECURITY.md](../SECURITY.md).
+325
View File
@@ -0,0 +1,325 @@
# User guide
How Proton Drive for Linux works and how to use it day to day. For installation see
[INSTALL.md](INSTALL.md); every command mentioned here is described in [CLI.md](CLI.md).
- [How it works](#how-it-works)
- [My files](#my-files)
- [Keeping files on this computer](#keeping-files-on-this-computer)
- [Syncing local folders](#syncing-local-folders)
- [Conflicts](#conflicts)
- [Pausing and limiting sync](#pausing-and-limiting-sync)
- [Photos](#photos)
- [Sharing](#sharing)
- [Trash and version history](#trash-and-version-history)
- [Computers](#computers)
- [Search launcher](#search-launcher)
- [Choosing how files open](#choosing-how-files-open)
- [Tray](#tray)
- [Preferences](#preferences)
## How it works
A background service, `proton-drive.service`, runs `pdfs daemon`. It holds your session, talks to
Proton Drive, and keeps your Drive mounted as a folder. Everything else is a front end that asks
the service to do things over a local socket:
| Program | What it is |
|---|---|
| `pdfs-app` | The desktop app: files, photos, sharing, sync, preferences |
| `pdfs-tray` | Tray icon with status, pause and quick actions |
| `pdfs-prompt` | A search launcher for a keyboard shortcut |
| `pdfs` | The command-line client, for everything above and for scripts |
Your Drive, your file names and your photos are end-to-end encrypted by Proton. On this computer,
the service must decrypt what it shows you, and it keeps a decrypted cache and index on disk. Use
full-disk or home-directory encryption. [ARCHITECTURE.md §8](ARCHITECTURE.md#8-threat-model-what-this-client-writes-to-disk-in-plaintext)
lists exactly what is stored.
The app uses these words consistently, and so does this guide:
| Term | Meaning |
|---|---|
| **My files** | The root of your Drive |
| **Proton Drive folder** | Where My files appears on disk, `~/ProtonDrive` by default |
| **Online only** | A file or folder whose content downloads when opened |
| **Available offline** | A single file kept on this computer |
| **Synced folder** | A local folder with a full copy on this computer, backed up to Proton Drive |
| **Proton Drive service** | The background service (`proton-drive.service`) |
## My files
My files is mounted at `~/ProtonDrive`. Every file and folder appears immediately, but content
downloads only when a program reads it:
- **Reads fetch only what is needed.** A video player that seeks, or a tool that reads a file
header, downloads only those parts. Media streams while it plays.
- **Writes are safe offline.** When a program closes a file it wrote, the service stores the new
content on disk and queues the upload. The queue survives network loss, restarts and crashes.
`pdfs sync queue` lists what has not reached Proton Drive yet.
- **Recently read content is cached** on disk up to the cache size limit (5 GiB by default) and
evicted oldest first.
- **Items shared with you** appear under `Shared with me/` inside the Proton Drive folder. Shares
where you are a viewer are read-only.
The **My files** page of the app browses the same tree without going through the mount. It has a
grid and a list view, sorting, thumbnails (including camera RAW previews), multi-select and drag
and drop upload. **Details** (Alt+Enter) shows size, dates, sharing and version history for the
selected item.
Symbolic links, hard links, device files and FIFOs cannot be stored on Proton Drive. Programs that
try to create them get a clean error, not a hang or `EIO`.
## Keeping files on this computer
Turn on **Make available offline** for a file in My files, or run `pdfs pin <path>`. The file is
downloaded completely and never evicted from the cache. `pdfs pins` lists these files, and
**Preferences → Storage → Available offline** shows them in the app.
Turning it off (`pdfs unpin <path>`) frees the space. The file stays on Proton Drive.
## Syncing local folders
A synced folder is an ordinary folder on this computer, for example `~/Documents`, that is
backed up to Proton Drive under this computer's name. It appears on the **Computers** page of the
app and in the web app.
- Add one with **Add Folder to Sync** on the **Computers** page, or `pdfs sync add ~/Documents`.
- Changes flow both ways. Local edits upload a few seconds after the file is closed; remote edits
download on the next pass.
- A file that another program still holds open for writing is not uploaded until it is closed.
- **Sync → Folders** in the app, and `pdfs sync list`, show every folder and its state.
Each synced folder can also be switched to **Online only**. The local copy is replaced by a mount
of the folder, so the files stay visible but take no disk space until opened. Switch with the
**Synced / Online only** switch on the folder's card, or `pdfs sync mode <id> ondemand` and back
with `pdfs sync mode <id> mirror`. `pdfs locations` lists every local place the service occupies.
### Moving between synced folders and My files
`~/ProtonDrive` and each online-only folder are separate mounts, and a mirrored folder is a plain
folder on disk. `mv`, and dragging in a file manager, cannot rename across them, so they copy
and then delete: every file is downloaded, uploaded again, and the original goes to the Trash.
Use `pdfs move <path> <folder>` or **Move to…** in the app instead. The move happens on Proton
Drive, so it takes seconds whatever the size. For example:
```sh
pdfs move ~/Documents/Scans ~/ProtonDrive/Archive
pdfs move ~/Documents/a.pdf ~/Documents/b.pdf ~/ProtonDrive/Archive
```
The last path is the destination folder; everything before it moves there.
Files already opened in an online-only folder are not downloaded again when
they move into a mirrored one.
A move out of a synced folder waits until its files are on Proton Drive. A file that is still
uploading, has changes that are not uploaded, or is excluded by `.pdfsignore` stops the move,
and the message names it.
Removing a synced folder (**Stop Syncing** or `pdfs sync rm <id>`) keeps the local files and the
copy on Proton Drive. Add `--delete-remote` (or tick **Also delete from Proton Drive**) to delete
the copy on Proton Drive as well.
### Excluding files with `.pdfsignore`
Synced folders skip build output, dependency folders and editor leftovers. Rules come from two
places, and both apply:
1. A `.pdfsignore` file at the root of the synced folder (`.protonignore` also works), using
gitignore syntax, including `!` to re-include:
```gitignore
build/
*.log
!important.log
```
2. The global `ignore_patterns` list in `config.json` (see
[CONFIGURATION.md](CONFIGURATION.md#ignore_patterns)). When unset, it defaults to `.git/`,
`.hg/`, `.svn/`, `node_modules/`, `target/`, `.venv/`, `__pycache__/`, `*~`, `*.swp`, `*.tmp`,
`.DS_Store` and `Thumbs.db`.
Rules are re-read on every pass. Ignoring never deletes anything: a file that becomes ignored is
simply no longer tracked, and its copy on Proton Drive stays.
## Conflicts
When the same file changed on this computer and somewhere else since the last sync, neither side
is thrown away. The local version is kept as a copy named like
`report (sync-conflict 1759068180).odt`, where the number is the time of the conflict in seconds
since 1970, and the remote version takes the original name.
Conflicts appear under **Sync → Overview → Needs attention** and in `pdfs conflicts`. For each
one, keep the original, keep the copy, or keep both under a new name:
```bash
pdfs conflicts
pdfs conflicts resolve "Documents/report (sync-conflict 1759068180).odt" --keep copy
```
The version you do not keep goes to the Trash, so a wrong choice can be undone.
## Pausing and limiting sync
- **Pause** from the tray, the Sync page, or `pdfs sync pause`. Uploads and synced folders wait;
reading files keeps working. `pdfs sync resume` continues.
- **Limit bandwidth** under **Preferences → General → Network**, or with
`pdfs sync limit --up 2M --down 10M`. `0` removes a limit.
- **Retry** a failed upload from **Sync → Overview**, or with `pdfs sync retry`.
## Photos
The **Photos** page shows your Proton Photos library. Photos are stored separately from My files
and do not appear in the Proton Drive folder.
- **Timeline** groups photos by day, month or year as you zoom. Drag the scrubber on the right edge
to jump through the years. **On this day** shows photos from the same date in earlier years.
- **Filters** narrow the timeline to images, videos or RAW files, to favorites, or to photos that
are in no album.
- **RAW + JPEG pairs** of the same shot show as one tile. The viewer switches between them, and
moving the tile to the Trash removes both.
- **Albums** lists your albums and albums shared with you. Create, rename and delete albums, and
add or remove photos.
- **Places** groups photos by the town they were taken in, as cards or on a map. Towns come from
bundled [GeoNames](https://www.geonames.org/) data (CC BY 4.0) and the map from
[Natural Earth](https://www.naturalearthdata.com/) (public domain), so nothing is fetched online.
An optional street map from [OpenFreeMap](https://openfreemap.org/) (© OpenStreetMap
contributors) can be enabled under **Preferences → General → Appearance**. It is off by default
because the tile server learns which areas you look at.
- **Viewer**: favorites, EXIF details, show on map, slideshow (F5).
- **Find Duplicates…** in the page menu lists byte-identical copies. The **Similar** switch also
finds resized or re-encoded copies and burst shots by comparing thumbnails.
- **Upload** adds photos. Its menu holds **Import from Google Photos…**.
### Importing from Google Photos
Request a Google Takeout export of Google Photos, then pass every `.zip` of it at once. A photo's
metadata often sits in a different part of the export than the photo itself.
```bash
pdfs import-google-photos --dry-run ~/Downloads/takeout-*.zip # see what would happen
pdfs import-google-photos --wait ~/Downloads/takeout-*.zip
```
Albums are recreated, capture dates and locations are taken from the metadata, and photos already
on the account are skipped, so an interrupted import can be run again. `pdfs import-status` and
`pdfs cancel-import` follow or stop an import that runs in the background.
If photos from an earlier import are filed under the day they were uploaded,
`pdfs redate-photos --dry-run` finds those whose file name carries a different date, and
`pdfs redate-photos` fixes them. Proton cannot change a capture date in place, so each fix
re-uploads the photo and moves the original to the Trash, keeping favorites and albums.
## Sharing
- **Share…** on a file or folder invites people by email as viewer or editor, or creates a public
link with an optional password and expiry. `pdfs share`, `pdfs members`, `pdfs share-role`,
`pdfs unshare` and `pdfs public-link` do the same from a terminal.
- **Shared by me** lists everything you share, filtered by links, people or expired links. Copy a
live link or stop several shares at once.
- **Shared with me** lists what others share with you, with who shared it and your access.
Pending invitations wait in a banner; `pdfs invitations` accepts or rejects them. Saved public
links (`pdfs bookmarks`) have their own view.
## Trash and version history
The **Trash** page lists trashed items with the date they were deleted. Restore puts an item back
where it was, together with trashed folders above it. **Delete Forever** and **Empty Trash** are
permanent. The CLI equivalents are `pdfs trash`, `pdfs restore`, `pdfs delete-forever` and
`pdfs empty-trash`.
Proton Drive keeps earlier versions of every file. **Details → Versions…** in the app, or
`pdfs versions list <path>`, shows them. You can:
- **Restore** a version. This happens on the server; nothing is re-uploaded.
- **Save** a version to a separate local file without touching the current one.
- **Delete** an old version permanently. The current version cannot be deleted.
## Computers
**Computers** lists this computer and every other computer that backs up to your account.
- **This computer** shows its synced folders with their state, and **Restore Folders…** brings
back folders that are backed up under it but not synced here yet.
- **Another computer** can be browsed read-only in the same list or grid as My files, and
**Restore to This Computer…** downloads its folders here.
- **Continue This Backup Here…** makes this computer take over another computer's backup, for
example after reinstalling or replacing a machine. See [RECOVERY.md](RECOVERY.md).
## Search launcher
`pdfs-prompt` searches Proton Drive and your home folder together, in one ranked list. It
tolerates prefixes, abbreviations, typos and swapped letters, and matches on parent folder names
as well. Bind it to a keyboard shortcut in your desktop settings.
The window stays loaded after the first use, so later shortcuts open it instantly. Folders, audio
and video open straight from the Proton Drive folder, so players can start before the whole file
downloads. Other Drive files download first, then open.
To make the first shortcut instant as well, start the launcher hidden with your session.
`pdfs-prompt --preload` loads the window without showing it and does nothing if the launcher is
already running. For Hyprland:
```ini
exec-once = pdfs-prompt --preload
bind = SUPER, space, exec, pdfs-prompt
```
### Using your own launcher
`pdfs-prompt --dmenu` shows the results in fuzzel, rofi, wofi, tofi, bemenu or dmenu, whichever is
installed:
```bash
pdfs-prompt --dmenu
pdfs-prompt --dmenu --menu 'fuzzel --dmenu --width 60'
pdfs-prompt --dmenu --query invoice
```
A dmenu-style launcher filters a fixed list and cannot ask for new results per keystroke, so the
search takes two steps. The first list, `Search Drive ›`, shows your available-offline files; type
a query and press Enter to search. The second, `Drive: invoice ›`, shows the results; Enter opens
one, and typing something else searches again.
### Search as you type with fzf
`pdfs-prompt --fzf` re-queries the service on every keystroke, like the built-in window. When
started from a keybinding without a terminal, it opens one itself (foot, ghostty, kitty,
alacritty, wezterm or xterm) with the app ID `pdfs-prompt`, so one window rule covers all of them.
For Hyprland:
```ini
windowrulev2 = float, class:^(pdfs-prompt)$
windowrulev2 = size 900 500, class:^(pdfs-prompt)$
windowrulev2 = center, class:^(pdfs-prompt)$
bind = SUPER, space, exec, pdfs-prompt --fzf
```
To make `--dmenu` or `--fzf` the default without changing your keybinding, set `prompt.mode` in
`config.json` ([CONFIGURATION.md](CONFIGURATION.md#prompt)). `--gtk` forces the built-in window.
## Choosing how files open
Files open with `xdg-open` by default. The `open_with` section of `config.json` overrides that per
file type, for example text in Neovim inside a terminal. The rules apply to the search launcher
and to the app. See [CONFIGURATION.md](CONFIGURATION.md#open_with).
## Tray
`pdfs-tray` starts at login and shows whether files are up to date, syncing, paused or offline.
Its menu opens the app or the Proton Drive folder, pauses or resumes sync, and stops or connects
the service. **Hide Tray Icon** removes it until you turn it back on under
**Preferences → General → Appearance → Show tray icon**.
GNOME needs the AppIndicator extension to show tray icons.
## Preferences
| Page | Settings |
|---|---|
| **General** | Start on login · Proton Drive folder location · upload and download limits · Proton theme, street map for Places, tray icon · language |
| **Storage** | Cache size limit · clear the cache · files kept available offline |
The account menu shows storage used and holds **Sign Out…**. Settings not exposed in the app are
listed in [CONFIGURATION.md](CONFIGURATION.md).
Binary file not shown.

Before

Width:  |  Height:  |  Size: 197 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 124 KiB

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 92 KiB

After

Width:  |  Height:  |  Size: 52 KiB

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 34 KiB

After

Width:  |  Height:  |  Size: 32 KiB

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 899 KiB

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 35 KiB

After

Width:  |  Height:  |  Size: 114 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 64 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 108 KiB

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 52 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 71 KiB

+2 -2
View File
@@ -9,11 +9,11 @@
# Build + install: cd packaging && makepkg -fi
pkgname=proton-drive-linux
pkgver=2.5.0
pkgver=2.10.0
pkgrel=1
pkgdesc="Unofficial Proton Drive client for Linux: files-on-demand FUSE mount, CLI, GTK4 app + tray"
arch=('x86_64')
url="https://github.com/narrrl/proton-drive-linux"
url="https://proton-drive.narl.io"
license=('MIT')
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0' 'perl-image-exiftool')
makedepends=('cargo' 'git' 'gettext')
@@ -1,8 +1,8 @@
pkgbase = proton-drive-for-linux-bin
pkgdesc = Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (binaries)
pkgver = 2.4.0
pkgrel = 1
url = https://github.com/narrrl/proton-drive-linux
pkgver = 2.8.2
pkgrel = 2
url = https://proton-drive.narl.io
arch = x86_64
license = MIT
depends = fuse3
@@ -21,25 +21,25 @@ pkgbase = proton-drive-for-linux-bin
optdepends = xdg-utils: "Open folder" action in the tray menu
optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf)
optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding
provides = proton-drive-for-linux=2.4.0
provides = proton-drive-linux=2.4.0
provides = proton-drive-for-linux=2.8.2
provides = proton-drive-linux=2.8.2
provides = pdfs
conflicts = proton-drive-linux
conflicts = proton-drive-for-linux
conflicts = proton-drive-for-linux-git
options = !strip
options = !debug
source = proton-drive-linux-2.4.0-x86_64.tar.gz::https://github.com/narrrl/proton-drive-linux/releases/download/v2.4.0/proton-drive-linux-2.4.0-x86_64.tar.gz
source = io.narl.proton-drive-linux.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.4.0/packaging/io.narl.proton-drive-linux.desktop
source = io.narl.proton-drive-linux-tray.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.4.0/packaging/io.narl.proton-drive-linux-tray.desktop
source = io.narl.proton-drive-linux.svg::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.4.0/packaging/io.narl.proton-drive-linux.svg
source = proton-drive.service::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.4.0/packaging/proton-drive.service
source = LICENSE-2.4.0::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.4.0/LICENSE
sha256sums = 00446aa9016466827bb0fe5ea1e1d171f478ab4194845f9005b690a05543fcca
source = proton-drive-linux-2.8.2-x86_64.tar.gz::https://github.com/narrrl/proton-drive-linux/releases/download/v2.8.2/proton-drive-linux-2.8.2-x86_64.tar.gz
source = io.narl.proton-drive-linux-2.8.2.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/io.narl.proton-drive-linux.desktop
source = io.narl.proton-drive-linux-tray-2.8.2.desktop::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/io.narl.proton-drive-linux-tray.desktop
source = io.narl.proton-drive-linux-2.8.2.svg::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/io.narl.proton-drive-linux.svg
source = proton-drive-2.8.2.service::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/packaging/proton-drive.service
source = LICENSE-2.8.2::https://raw.githubusercontent.com/narrrl/proton-drive-linux/v2.8.2/LICENSE
sha256sums = c0af4ff57dc6f587b1b8431f3180b00d45211389f3dc675569af45f64e89cb36
sha256sums = d190d6771fff0b975271fb62fd3fb0fa8bbf17d7278c10853c18557855b0c123
sha256sums = 1d95e250370220e3bb6aead6ce34d1d6cc759f42c4803b75f1667e78ff47c6c0
sha256sums = e598a93c2715c3a9fdc7fb937902083d4a1135e8fbdfe6fe7e37f0383ebbd828
sha256sums = 25fa3bd9ba52df534be9d8bc66f77d7502fe1423177237d0d14b673dedeb1b60
sha256sums = 77280bff5f693fbc5ffa9f3ccd06a43cfff96d770290aa8d7d0c28e9dde277be
sha256sums = 02c419e7f9af08dda720e4f852f3e57cf5a69e192605440f750e9d244fa2817c
pkgname = proton-drive-for-linux-bin
@@ -1,11 +1,12 @@
# Maintainer: Nils Pukropp <contact@narl.io>
pkgname=proton-drive-for-linux-bin
pkgver=2.4.0
pkgrel=1
pkgver=2.8.2
pkgrel=2
pkgdesc="Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (binaries)"
arch=('x86_64')
url="https://github.com/narrrl/proton-drive-linux"
url="https://proton-drive.narl.io"
_repo="https://github.com/narrrl/proton-drive-linux"
license=('MIT')
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0' 'dbus' 'gcc-libs' 'glibc'
'hicolor-icon-theme')
@@ -24,18 +25,20 @@ options=('!strip' '!debug')
_tag="v$pkgver"
_raw="https://raw.githubusercontent.com/narrrl/proton-drive-linux/$_tag"
# The release tarball holds the four binaries only, so the desktop integration
# files come straight from the tagged tree.
source=("proton-drive-linux-$pkgver-x86_64.tar.gz::$url/releases/download/$_tag/proton-drive-linux-$pkgver-x86_64.tar.gz"
"io.narl.proton-drive-linux.desktop::$_raw/packaging/io.narl.proton-drive-linux.desktop"
"io.narl.proton-drive-linux-tray.desktop::$_raw/packaging/io.narl.proton-drive-linux-tray.desktop"
"io.narl.proton-drive-linux.svg::$_raw/packaging/io.narl.proton-drive-linux.svg"
"proton-drive.service::$_raw/packaging/proton-drive.service"
# files come straight from the tagged tree. Every one carries $pkgver in its
# name: makepkg reuses a cached file of the same name, so an unversioned name
# keeps the old copy after upstream changes it and fails the checksum.
source=("proton-drive-linux-$pkgver-x86_64.tar.gz::$_repo/releases/download/$_tag/proton-drive-linux-$pkgver-x86_64.tar.gz"
"io.narl.proton-drive-linux-$pkgver.desktop::$_raw/packaging/io.narl.proton-drive-linux.desktop"
"io.narl.proton-drive-linux-tray-$pkgver.desktop::$_raw/packaging/io.narl.proton-drive-linux-tray.desktop"
"io.narl.proton-drive-linux-$pkgver.svg::$_raw/packaging/io.narl.proton-drive-linux.svg"
"proton-drive-$pkgver.service::$_raw/packaging/proton-drive.service"
"LICENSE-$pkgver::$_raw/LICENSE")
sha256sums=('00446aa9016466827bb0fe5ea1e1d171f478ab4194845f9005b690a05543fcca'
sha256sums=('c0af4ff57dc6f587b1b8431f3180b00d45211389f3dc675569af45f64e89cb36'
'd190d6771fff0b975271fb62fd3fb0fa8bbf17d7278c10853c18557855b0c123'
'1d95e250370220e3bb6aead6ce34d1d6cc759f42c4803b75f1667e78ff47c6c0'
'e598a93c2715c3a9fdc7fb937902083d4a1135e8fbdfe6fe7e37f0383ebbd828'
'25fa3bd9ba52df534be9d8bc66f77d7502fe1423177237d0d14b673dedeb1b60'
'77280bff5f693fbc5ffa9f3ccd06a43cfff96d770290aa8d7d0c28e9dde277be'
'02c419e7f9af08dda720e4f852f3e57cf5a69e192605440f750e9d244fa2817c')
package() {
@@ -44,15 +47,15 @@ package() {
install -Dm755 "$srcdir/pdfs-app" "$pkgdir/usr/bin/pdfs-app"
install -Dm755 "$srcdir/pdfs-prompt" "$pkgdir/usr/bin/pdfs-prompt"
install -Dm644 "$srcdir/io.narl.proton-drive-linux.desktop" \
install -Dm644 "$srcdir/io.narl.proton-drive-linux-$pkgver.desktop" \
"$pkgdir/usr/share/applications/io.narl.proton-drive-linux.desktop"
install -Dm644 "$srcdir/io.narl.proton-drive-linux-tray.desktop" \
install -Dm644 "$srcdir/io.narl.proton-drive-linux-tray-$pkgver.desktop" \
"$pkgdir/etc/xdg/autostart/io.narl.proton-drive-linux-tray.desktop"
install -Dm644 "$srcdir/io.narl.proton-drive-linux.svg" \
install -Dm644 "$srcdir/io.narl.proton-drive-linux-$pkgver.svg" \
"$pkgdir/usr/share/icons/hicolor/scalable/apps/io.narl.proton-drive-linux.svg"
# systemd user unit for the auto-mount daemon; enabled per user, not by pacman.
install -Dm644 "$srcdir/proton-drive.service" \
install -Dm644 "$srcdir/proton-drive-$pkgver.service" \
"$pkgdir/usr/lib/systemd/user/proton-drive.service"
# Translations ship precompiled in the release tarball.
@@ -1,8 +1,8 @@
pkgbase = proton-drive-for-linux-git
pkgdesc = Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (git)
pkgver = 2.4.0.r0.gda35a05
pkgver = 2.8.2.r0.g50483bf
pkgrel = 1
url = https://github.com/narrrl/proton-drive-linux
url = https://proton-drive.narl.io
arch = x86_64
license = MIT
makedepends = cargo
@@ -24,8 +24,8 @@ pkgbase = proton-drive-for-linux-git
optdepends = xdg-utils: "Open folder" action in the tray menu
optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf)
optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding
provides = proton-drive-for-linux=2.4.0.r0.gda35a05
provides = proton-drive-linux=2.4.0.r0.gda35a05
provides = proton-drive-for-linux=2.8.2.r0.g50483bf
provides = proton-drive-linux=2.8.2.r0.g50483bf
provides = pdfs
conflicts = proton-drive-linux
conflicts = proton-drive-for-linux
@@ -1,11 +1,12 @@
# Maintainer: Nils Pukropp <contact@narl.io>
pkgname=proton-drive-for-linux-git
pkgver=2.4.0.r0.gda35a05
pkgver=2.8.2.r0.g50483bf
pkgrel=1
pkgdesc="Unofficial Proton Drive client: FUSE files-on-demand mount, CLI, GTK4 app, tray (git)"
arch=('x86_64')
url="https://github.com/narrrl/proton-drive-linux"
url="https://proton-drive.narl.io"
_repo="https://github.com/narrrl/proton-drive-linux"
license=('MIT')
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0' 'dbus' 'gcc-libs' 'glibc'
'hicolor-icon-theme')
@@ -22,7 +23,7 @@ provides=("${pkgname%-git}=$pkgver" "proton-drive-linux=$pkgver" 'pdfs')
conflicts=('proton-drive-linux' 'proton-drive-for-linux' 'proton-drive-for-linux-bin')
# LTO has broken GTK/Rust links for this workspace; the release profile keeps it off.
options=('!lto')
source=("$pkgname::git+$url.git")
source=("$pkgname::git+$_repo.git")
sha256sums=('SKIP')
pkgver() {
@@ -1,8 +1,8 @@
pkgbase = proton-drive-for-linux
pkgdesc = Unofficial Proton Drive client: files-on-demand FUSE mount, CLI, GTK4 app and tray
pkgver = 2.4.0
pkgver = 2.8.2
pkgrel = 1
url = https://github.com/narrrl/proton-drive-linux
url = https://proton-drive.narl.io
arch = x86_64
license = MIT
makedepends = cargo
@@ -23,13 +23,13 @@ pkgbase = proton-drive-for-linux
optdepends = xdg-utils: "Open folder" action in the tray menu
optdepends = fzf: search-as-you-type launcher (pdfs-prompt --fzf)
optdepends = foot: terminal that hosts pdfs-prompt --fzf when started from a keybinding
provides = proton-drive-linux=2.4.0
provides = proton-drive-linux=2.8.2
provides = pdfs
conflicts = proton-drive-linux
conflicts = proton-drive-for-linux-bin
conflicts = proton-drive-for-linux-git
options = !lto
source = proton-drive-for-linux-2.4.0.tar.gz::https://github.com/narrrl/proton-drive-linux/archive/refs/tags/v2.4.0.tar.gz
sha256sums = ae0f6daec257e9f7ba464ed6385bab7ac99f4247db0176296b2a58f703c24799
source = proton-drive-for-linux-2.8.2.tar.gz::https://github.com/narrrl/proton-drive-linux/archive/refs/tags/v2.8.2.tar.gz
sha256sums = 7f2df1d2748c3d8daaa4c05d242fcd293cb1983572e44bb61a35188dbe5f342c
pkgname = proton-drive-for-linux
@@ -1,11 +1,12 @@
# Maintainer: Nils Pukropp <contact@narl.io>
pkgname=proton-drive-for-linux
pkgver=2.4.0
pkgver=2.8.2
pkgrel=1
pkgdesc="Unofficial Proton Drive client: files-on-demand FUSE mount, CLI, GTK4 app and tray"
arch=('x86_64')
url="https://github.com/narrrl/proton-drive-linux"
url="https://proton-drive.narl.io"
_repo="https://github.com/narrrl/proton-drive-linux"
license=('MIT')
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0' 'dbus' 'gcc-libs' 'glibc'
'hicolor-icon-theme')
@@ -22,8 +23,8 @@ provides=("proton-drive-linux=$pkgver" 'pdfs')
conflicts=('proton-drive-linux' 'proton-drive-for-linux-bin' 'proton-drive-for-linux-git')
# LTO has broken GTK/Rust links for this workspace; the release profile keeps it off.
options=('!lto')
source=("$pkgname-$pkgver.tar.gz::$url/archive/refs/tags/v$pkgver.tar.gz")
sha256sums=('ae0f6daec257e9f7ba464ed6385bab7ac99f4247db0176296b2a58f703c24799')
source=("$pkgname-$pkgver.tar.gz::$_repo/archive/refs/tags/v$pkgver.tar.gz")
sha256sums=('7f2df1d2748c3d8daaa4c05d242fcd293cb1983572e44bb61a35188dbe5f342c')
_srcname="proton-drive-linux-$pkgver"
+7 -1
View File
@@ -14,7 +14,7 @@ Version: %{version}
Release: 1%{?dist}
Summary: Unofficial Proton Drive client for Linux (FUSE, CLI, GTK4 app + tray)
License: MIT
URL: https://github.com/narrrl/proton-drive-linux
URL: https://proton-drive.narl.io
ExclusiveArch: x86_64
BuildRequires: cargo
@@ -60,12 +60,18 @@ test -n "%{?git_dir}" || (echo 'Pass --define "git_dir $PWD" from the repo root'
test -f %{git_dir}/Cargo.toml
cp -a %{git_dir}/LICENSE .
%build
# The release workflow has already built target/release and passes --define "prebuilt 1".
# rpmbuild exports its own compiler flags, which would make cargo rebuild everything.
%if 0%{?prebuilt}
test -x %{git_dir}/target/release/pdfs
%else
cd %{git_dir}
cargo build --release --locked \
--bin pdfs \
--bin pdfs-tray \
--bin pdfs-app \
--bin pdfs-prompt
%endif
%install
rel=%{git_dir}/target/release
+7 -1
View File
@@ -32,4 +32,10 @@ MemoryMax=6G
TimeoutStopSec=15
[Install]
WantedBy=default.target
# Not default.target: that is reached before the desktop has exported DISPLAY /
# WAYLAND_DISPLAY to the user manager, so After= above never orders anything
# (the two targets are not in the same start transaction). The daemon's first
# keyring call then D-Bus-activates the Secret Service without a display —
# on KDE ksecretd aborts, the start fails, and every login leaves a coredump.
# Being pulled in by the graphical session starts it only once that is up.
WantedBy=graphical-session.target
+2
View File
@@ -28,6 +28,8 @@ crates/pdfs-gui/src/app/widgets/thumbnails.rs
crates/pdfs-gui/src/app/widgets/versions_dialog.rs
crates/pdfs-gui/src/dmenu.rs
crates/pdfs-gui/src/fzf.rs
crates/pdfs-gui/src/hud/mod.rs
crates/pdfs-gui/src/hud/preview.rs
crates/pdfs-gui/src/i18n.rs
crates/pdfs-gui/src/main.rs
crates/pdfs-gui/src/prompt.rs
+418 -372
View File
File diff suppressed because it is too large. Load diff
Loaded 100 of 111 files, more files were not shown because too many files have changed in this diff. Show more