Compare commits

...
Author SHA1 Message Date
nvrl d13e74fc12 fix(gui,cli,core): offer the CAPTCHA in the browser when the web view cannot start (B195)
The verification window now runs in its own process, so the app survives
when WebKit cannot start its sandbox on secureblue, but the CAPTCHA still
could not be solved there (#29). WebKit binds the journal socket with
`--ro-bind-try`, bubblewrap only skips a missing path, not an unreadable
one, and WebKit has no switch to leave the socket out.

Proton Mail Bridge never uses a web view: it opens verify.proton.me in the
system browser and retries with the original challenge token, typed with
the offered methods. Do the same as a fallback.

- core: `browser_verification_url` and `browser_verified` build the page
  URL and the retry credential from the challenge
- gui: when the verification window fails, a dialog opens the page in the
  browser and continues the sign-in once the user is done
- cli: `pdfs login` handles a CAPTCHA the same way instead of failing
- i18n: translate the new dialog in every catalog
- docs: B195, TROUBLESHOOTING.md, ARCHITECTURE.md, changelog

Fixes #29
2026-10-08 19:02:29 +02:00
nvrl 07c9b675d3 chore(deps) updated sdk to 0.8.1 2026-10-08 15:25:44 +02:00
nvrl f82343b844 Merge pull request #30 from narrrl/fix/secureblue-captcha
Fix/secureblue captcha
2026-10-08 15:25:05 +02:00
Nils Pukropp d09e10dc7d chore(aur): update packages to 3.1.1 2026-10-08 09:49:25 +00:00
nvrl 8d0351124c chore(release): 3.1.2 2026-10-08 11:31:18 +02:00
nvrl a888e0bb28 fix(gui): run the CAPTCHA page in its own process (B195)
On secureblue, WebKit cannot start the sandbox for the verification page:
bubblewrap is kept from the journal socket, xdg-dbus-proxy fails, and
WebKit ends its process with g_error. The webview lived in pdfs-app, so
the whole app crashed after the sign-in (#29).

- gui: `pdfs-app --human-verification` shows the page in its own window,
  reads the URL on stdin and prints the token on stdout; its logs go to
  stderr
- gui: a child that dies or fails without a token ends the sign-in with a
  message that the page could not be opened, instead of "cancelled"
- i18n: translate the new message in every catalog
- docs: B195 in the bug ledger, the cause and a one-time workaround in
  TROUBLESHOOTING.md, the child process in ARCHITECTURE.md, changelog
2026-10-08 10:25:14 +02:00
nvrl c7fc85e943 fix(gui): create the state dir before the tray and app use it (B194)
On a fresh install only the daemon and the CLI created
~/.local/state/proton-drive-linux. The app starts the tray before the
first sign-in, so the tray failed to bind its single-instance socket and
exited, and the Takeout page could not remember staged archives (#29).

- gui: pdfs-tray and pdfs-app call AppDirs::ensure at start, which also
  makes the directories owner-only
- docs: B194 in the bug ledger, changelog entry
2026-10-08 10:25:04 +02:00
nvrl 5bfae0fee1 docs(changelog): add the 3.1.1 entry
3.1.1 shipped the client identity change without a changelog entry. Also
add the missing release links for 3.1.0 and 3.1.1.
2026-10-08 10:24:07 +02:00
nvrl edd62c5f7a chore(release): 3.1.1 2026-10-07 17:01:32 +02:00
nvrl 5c1530b4b9 fix(core,gui): send the release version as the client identity
Every release sent the fixed `external-drive-linux@0.1.0-alpha`. If Proton
ever retires that version, every release fails sign-in at once and no update
can fix it.

- core: APP_VERSION and USER_AGENT follow CARGO_PKG_VERSION
  (`external-drive-linux@<version>-stable`, `proton-drive-linux/<version>`)
- core: `app_version` and `user_agent` in config.json are optional overrides;
  the fixed values older releases stored there read as unset and leave the
  file on the next save, a hand-set value is kept
- gui: the About dialog shows the release, its debug info the identity
  actually sent
- docs: describe the new default and the migration
2026-10-07 17:01:32 +02:00
Nils Pukropp ce324510de chore(aur): update packages to 3.1.0 2026-10-07 10:53:15 +00:00
nvrl 274bfd0ca2 Merge pull request #27 from narrrl/feature/two-password-mode
Feature/two password mode
2026-10-07 12:47:05 +02:00
nvrl b8cfd30ff3 fix(core): give the kernel-keyring unlock owner read access
A new key grants its owner only "view"; read needs possession, which a
process has only when its session keyring links to the user keyring. A
systemd user service or a container does not guarantee that, so the
daemon could store the unlock and then fail to read it back (EACCES).
Set the permissions explicitly after add_key.
2026-10-07 08:17:50 +02:00
nvrl 9fa88ff397 feat(core,cli,gui): support two-password accounts without storing the mailbox password
A two-password account signed in with its login password as the mailbox
password, so the key chain never unlocked and the daemon retried forever.

Login now asks for the mailbox password and checks it against the account
(proton-sdk 0.8 `unlock`), so a typo is a retry. The password is never
stored: the keyring blob keeps the per-key passphrases instead, and for a
two-password account only when the user opts in (`--remember`, or the
"Keep unlocked on this computer" switch). By default they live in the
kernel keyring for the login session and the account is locked after a
reboot until `pdfs unlock` or the app supplies them again.

- core: kernelkey module, Error::Locked / WrongMailboxPassword, unlock,
  lock, key_state, ready; blobs written by 3.0.x are migrated on first
  start; token refresh updates only the tokens in the stored blob
- cli: `pdfs unlock [--remember]`, `pdfs lock [--forget]`; status and
  diagnose report the key state; the daemon waits while locked and is
  marked non-dumpable
- gui: mailbox password dialog at sign-in and on open, locked tray state
- po: all catalogs translated
- bump to 3.1.0, depend on proton-sdk and proton-drive-rs 0.8.0

Refs #26.
2026-10-07 08:17:50 +02:00
nvrl f105159186 fix(cli,fuse): keep answering the systemd watchdog while the daemon retries
The watchdog is armed when the process starts, but only the mount's supervisor
pinged it. A daemon waiting for a login or backing off after a failed mount
(#26: key unlock failing) was aborted after WatchdogSec and left a coredump
instead of its error. Sleep through idle_sleep in those stretches; the mounted
ping still has to pass the control-socket probe.
2026-10-07 08:17:50 +02:00
nvrl a14a63c94c ci: add an on-demand Fedora test build
Manual workflow that builds the RPM and the bare binaries in a Fedora
container, so a user can try a branch against Fedora's own GTK stack.
Uploads them as an artifact and, on request, as one rolling pre-release
per branch that needs no GitHub login to download.
2026-10-07 04:27:22 +02:00
nvrl ed3477d1b3 docs: point contributors at the simulation runs and the ledger rules
DEVELOPMENT.md lists the daemon's current modules, asks for the
simulation runs on FUSE, queue and sync changes, and adds how to handle
a bug fix and a user-visible change, including the six pages the wiki
publisher copies. The release checklist moves to TESTING.md.
2026-10-06 19:29:02 +02:00
nvrl aa384ae2eb chore: cite docs/BUGS.md and drop references to notes outside the repo
66 comments cited `bugs.md B<n>`, the ledger's old name; they now say
`docs/BUGS.md B<n>`, and every cited number has an entry. About 90 more
cited offline.md, devices.md, features.md, mount-architecture.md,
plan.md and improvements.md, planning notes that were never in the
repository. Those citations are removed; the comments keep their text.
2026-10-06 19:29:02 +02:00
nvrl 10c69b0d64 test(acceptance): explain the metadata floor for local-first mounts
The throughput case said every create and unlink on a Drive mount waits
for Drive. Since 3.0.0 that holds only with local_first off; the floor
of 3 operations a second stays for that case.
2026-10-06 19:29:02 +02:00
nvrl 39be354bd2 docs(bugs): give the ledger an index and one order
The ledger ran B193 down to B78, then B1 up to B75, with T1 to T4 in
the middle and a stray top-level heading inside B14. Status lines used a
dozen wordings for the same few states, so finding what is still open
meant reading 6,600 lines.

Add a header: what the numbers promise (permanent, never reused; B76
and B77 were never assigned), the status vocabulary, and how to add an
entry, with a template. Add an index sorted by status, with what is left
for every open or partly fixed entry. Order all entries newest first
under one Entries section, and move the acceptance-runner findings to
their own section. Titles are in sentence case; anchors are unchanged.

Entry text is kept as written, except B89: the file it described,
proton-dev/CLAUDE.md, no longer exists, so it is closed.
2026-10-06 19:29:02 +02:00
nvrl 2cca8afba9 docs: describe local-first changes and sync issues for users
The user guide said only writes were queued. Since 3.0.0 creating,
renaming and deleting through the mount return at once too, and changes
Drive refuses become sync issues. Add a section on changes waiting for
Proton Drive: retry, export, discard, and the two cases that land under
another name or place instead of failing.

Troubleshooting covers changes that have not reached other devices yet,
`pdfs sync issues` and the local_first fallback, and names the real
error for symlinks and hard links (EPERM, not EOPNOTSUPP). CONFIGURATION
lists MemoryHigh and the test variables; CLI notes that `sync export`
takes a folder.
2026-10-06 19:14:29 +02:00
nvrl 6e9beb4b16 docs: rework contributor docs for the 3.0.0 local-first mount
ARCHITECTURE.md described the 2.x write path, where namespace changes
waited for Drive and every mount had its own inode space. Rewrite it
around the local tree (lid, inode = lid + 1, local~<lid> stand-ins), the
queue and its 16 drain workers, sync issues, and the remaining split
between the mount and the mirror engine. Drop the pins.json claim; pins
moved into the database at schema v5.

TESTING.md gains the simulation runs and loses the per-case narrative
that --list and BUGS.md already carry. ROADMAP.md takes over the
post-3.0.0 items from the milestone status file.

Remove MILESTONE-3.0.0-STATUS.md (a working log) and UI_UX_PLAN.md (the
2.x design audit). Code comments and CI that cited the untracked
docs/MILESTONE-3.0.0.md now point at TESTING.md, ARCHITECTURE.md and
ROADMAP.md. Threat-model links follow the section to §10.
2026-10-06 19:14:29 +02:00
nvrl 63484bf4b6 chore(images): add cover image
1920×1080 cover for link previews, social posts and the GitHub social preview.
The website renders its og.png from cover.svg.
2026-10-06 14:46:56 +02:00
nvrl 37c66b7220 chore(support) addd buy me a coffee badge + link 2026-10-06 10:49:24 +02:00
Nils Pukropp 41b6edbecb chore(aur): update packages to 3.0.0 2026-10-05 05:12:08 +00:00
nvrl 217738318e chore(release): 3.0.0 2026-10-05 05:16:57 +02:00
nvrl 9291580fd6 test(fuse): drop the known-bug matchers the runs confirmed fixed
The confirmation run on 812eb3d (PDFS_SIM_KNOWN=fail, twelve seeds)
passed every profile, so no open bug is tolerated any more: the call,
hang and damage matchers for B125 to B134 go, and any of them now fails
the seed. B126, B128 and B133 were fixed with B138, B162 and B137;
B134 stays open with local_first off only, which the runs do not mount.
2026-10-05 05:15:19 +02:00
nvrl 812eb3d0bc fix(fuse): a deleted file's trash leaves a file of the user's under the name it was sent as (B193) 2026-10-05 04:50:17 +02:00
nvrl 31863a1da1 fix(fuse): a new file closed empty reaches Drive at once (B192) 2026-10-05 04:50:10 +02:00
nvrl 57a825887c fix(core): a file deleted after its failed create was written again leaves Drive (B191) 2026-10-05 04:50:02 +02:00
nvrl 1436c5f37e fix(fuse): a stop queues the writes whose release the kernel never sent (B190) 2026-10-05 03:55:52 +02:00
nvrl f0a43fb923 fix(fuse): changes that failed on a short outage go out once the link is back (B189) 2026-10-05 03:48:13 +02:00
nvrl b47620651f fix(fuse): a restart no longer brings back the write of a file deleted while open (B188) 2026-10-05 03:37:41 +02:00
nvrl 2baad6720d fix(core): leave no temp copy when the cache adopts a blob it already holds (B187) 2026-10-05 02:52:51 +02:00
nvrl c271a1897c fix(fuse): keep what an open file reads when its delete is queued (B186) 2026-10-05 02:51:55 +02:00
nvrl d26dc8ada4 fix(fuse,core): retry a create at once when a write replaced its blob (B185) 2026-10-05 02:34:05 +02:00
nvrl 93f02672e6 docs: describe 3.0.0's local-speed writes and leave batched trashes and moves for later 2026-10-05 02:02:34 +02:00
nvrl 307305282e fix(fuse,core): what a create sent, and where, outlives a restart and a rename (B182, B183, B184)
The known-bug confirmation run failed one_client_on_a_flaky_link seeds 6
and 11 on three bugs around a create whose answer was lost:

- B182: after a restart, a write over a file whose create landed but was
  never read back forked a conflict copy. A revision of a file that has
  no revision of Drive's now adopts the landed node first.
- B183: the SHA-1s a create sent were kept in memory, so after a restart
  its trash or retry no longer knew the node Drive made. They are now on
  the op's row (pending_op.sent_sha1, schema 42).
- B184: a rename rewrote the op's target, and the trash and the retry
  looked for the node under the last name only. Each attempt's parent
  and name are recorded (pending_op.sent_to); the trash looks under all
  of them, and a retry adopts the node an earlier attempt made and moves
  it to the current name.
2026-10-05 02:01:50 +02:00
nvrl a8ed6060af feat(core,fuse,gui): show a sync issue on its node in the file browser
A refused change was only on the Sync page. A folder listing now carries
each node's issue (by local id, so a file not yet on Drive has it too),
and the browser's badge ranks it first, folders included, with the
issue's explanation as its tooltip. The event sampler publishes Files
when the set of issues changes, so the badge comes and goes with it.
2026-10-05 01:06:03 +02:00
nvrl 9dfe4c5450 feat(core,fuse,cli,gui): discard a queued change and undo it here
A change Drive will not take could be exported or retried, never let go
of. The Sync page's Discard button (with a confirmation) and
`pdfs sync discard <id>` drop it: a node made here goes, as if deleted
before it was sent; a node Drive has goes back to what Drive has, under
the same inode, with every change queued on it, while what is queued
below it stays. A change on the wire or an open file is refused.
2026-10-05 00:57:02 +02:00
nvrl da013c29d6 fix(gui): word a queued edit kept as a conflict copy in the user's language (B181)
The daemon logs it as "<reason>; local changes uploaded as <copy>" or
"...kept at <path>", which the Activity page showed as written, in
English. It now reads both and the three reasons, one sentence each.
2026-10-05 00:17:11 +02:00
nvrl 457a9d26f4 ci: keep the stacks of a hung simulation run
CI on f9b1638 failed slow-link seed 2 on a step that hung for 150 s, and
the stacks that would say where were left on the runner.
2026-10-05 00:13:42 +02:00
nvrl a68dae9422 fix(fuse): a kernel notice sent from the runtime hands its worker's tasks on (B130) 2026-10-05 00:02:56 +02:00
nvrl e743cd60cf fix(fuse): a folder whose mkdir is queued is listed from the database (B125) 2026-10-05 00:02:40 +02:00
nvrl f9b1638603 docs: record the 903452a account run and CI in the 3.0.0 status 2026-10-04 23:48:58 +02:00
nvrl 6539c7314b test(acceptance): a queue wait after a restart waits for the daemon to answer
A mirror target's root is plain local storage, so the durability case's
wait for the mount passed at once after the restart. The queue wait then
read `pdfs --json status` before the new daemon listened, took its null
"mount" for an empty queue, and the run's cleanup found no control socket.
2026-10-04 23:47:57 +02:00
nvrl 903452a15b fix(fuse): a stop wakes every drain worker (B180) 2026-10-04 23:00:07 +02:00
nvrl 4e0fd3c0b1 fix(fuse): a file read just as its create lands has its bytes (B179) 2026-10-04 22:52:00 +02:00
nvrl a6f019ebe6 fix(fuse): a folder trashed while sync is paused drops the creates in it (B178) 2026-10-04 22:51:47 +02:00
nvrl 0f3c06f982 fix(core): a folder listed just as its create lands is listed from the database (B177) 2026-10-04 22:41:57 +02:00
nvrl 4f141c78ca fix(fuse): a write closed just as its file's create lands is queued (B176) 2026-10-04 22:31:49 +02:00
nvrl ce60e006ac fix(fuse): a file replaced just as its create lands is trashed on Drive (B175) 2026-10-04 22:24:28 +02:00
nvrl 1612a9d07a fix(core): a file renamed just as its create lands keeps its access (B174) 2026-10-04 22:15:01 +02:00
nvrl 378667fb37 fix(fuse): a file deleted just as its create lands is trashed on Drive (B173) 2026-10-04 21:51:52 +02:00
nvrl 3b81cc5240 test(fuse): a simulated daemon does not ask the real API for its share id 2026-10-04 21:51:36 +02:00
nvrl d850265225 fix(fuse): a file renamed while its upload is read back keeps its new name (B172) 2026-10-04 21:34:20 +02:00
nvrl e32f72bfca fix(core,fuse): a deleted file's trash is not held back by its create (B171)
A file deleted after its create was sent keeps the op as a trash. The op
kept the create's attempts and backoff, and the drain recorded the failure
of a create attempt still on the wire on the row that was by then the trash.
The trash waited out the create's backoff.

The trash starts due, as a create that becomes a revision does, and the
drain records a failed attempt only if the op is still the kind it ran as
(`Db::record_attempt_failure`).
2026-10-04 21:23:10 +02:00
nvrl 2502af6e3c fix(fuse): a write opened as an upload lands takes the landed base (B170)
An upload that landed let go of its blob before the tree had its node. A
write open in between took the base it sampled at its start, the revision
the upload replaced, found none of its bytes cached, and queued an
incomplete revision; the next write open failed with EIO.

The drain keeps the landing node aside until the tree has it, and a write
open with no queued revision takes its base from that node, or from the
tree as it is when the handle goes in. The fake Drive can hold the drain
in that window (`hold_next_landing`).
2026-10-04 21:14:09 +02:00
nvrl 4757b957d2 fix(fuse): end a mirror pass at the stop (B169)
A stop that came while a mirror pass was uploading let the pass go on to
the next level of folders until the stop gave up on it and the runtime's
end cancelled the uploads under it. Once the daemon is stopping, a batch
starts none of its waiting ops, lets the ones sending finish, and ends
the pass; the remote walk and a new pass stop too.
2026-10-04 21:01:38 +02:00
nvrl 8e54ed0517 fix(fuse): do not send a file removed with its folder again (B168)
A delete of a file whose create was on the wire withdraws the create into
a trash. When its folder was removed too and that trash landed first,
Drive refused the create, and the drain re-homed the deleted file in the
root; only its missing bytes stopped it. The withdrawn create's trash
then read the trashed folder back into the cache before finding nothing
to trash.

A withdrawn create is no longer re-homed, and its trash is done once the
folder it was sent to is trashed or gone.
2026-10-04 20:42:04 +02:00
nvrl 9bbc994a4f perf(fuse): rebuild the hydrated listings in one pass (B167)
The hydrate rebuilt each listed folder's listing with a pass over every
entry. With 4,600 listed folders and 37,000 entries that took 1.7 s at
every start before the mount came up; one pass grouping the entries by
parent takes 3 ms.
2026-10-04 20:27:58 +02:00
nvrl ffa0fe698d test(fuse): a listing from before a create commits keeps the landed file (B166)
The account run's 0-byte read of a new file was B166 too: Drive's
listing had the node from before its create was committed, and the
queue no longer held it when read. The fake Drive can now hold a
folder's listing on its way back with one file listed unrevised.
2026-10-04 20:17:49 +02:00
nvrl 25c46cfe0a fix(fuse): list a file whose create lands while its folder is listed (B166)
A listing from Drive was overlaid with the queue as it was after Drive
answered. A create that landed in between was in neither, and a wide
folder on Wi-Fi dropped files that were never unlinked. The queue is now
read before Drive is asked too; a node that left it meanwhile is listed
as the DB has it.
2026-10-04 20:10:14 +02:00
nvrl 573a29e596 fix(fuse,cli): bound the stop's wait for busy threads (B165)
A stop joined every background thread for as long as each took. The sync
engine's mirror pass calls Drive with no deadline, and a stop that waited for
it was killed by systemd at TimeoutStopSec. The stop now waits up to 10 s for
the background threads and FUSE workers together, names any still running, and
the runtime shuts down within 2 s.
2026-10-04 19:12:19 +02:00
nvrl ce2bc0ac84 fix(fuse): list a new file once while its create lands (B164)
A listing from Drive overlays what is still queued for the folder. A create
Drive has made but whose answer is on its way, or which the drain is still
reading back, was in both: Drive's node under its new uid and the queue's
stand-in, under one name. A node queued in the folder now holds its name there.
2026-10-04 19:03:33 +02:00
nvrl b37d5e7928 fix(fuse): do not take a new file Drive lists before its commit as empty (B163) 2026-10-04 18:51:39 +02:00
nvrl 18587124ac fix(fuse): never adopt a node we trashed as an unanswered create's twin (B162)
A folder removed and made again met the old folder while its trash was on
the wire and adopted it, so the new folder's contents went into the old
one and were trashed with it. The twin search now skips hidden nodes, as
listings do.

The fake Drive can hold a trash (hold_next_trash).
2026-10-04 18:26:45 +02:00
nvrl cd18029944 fix(fuse): keep a move that lands while its create is read back (B161)
adopt_real_uid asked whether a move was queued only after reading the
node back. A move that landed in between left no op, and the answer still
had the old name, so the mount showed the file under it. Ask first.

The fake Drive can hold a read's answer (hold_answer_to_next_read) so the
test can land the move while the answer is on its way.
2026-10-04 18:20:56 +02:00
nvrl d6dd41407f fix(fuse): wake the drain when a folder is made (B160)
Queueing a folder's create left the workers asleep until their 30 s idle
poll; every other enqueue woke them. A file's create still waits for the
release that attaches its bytes.
2026-10-04 18:14:00 +02:00
nvrl 60445d3b26 fix(fuse): list again when a listing changed while it was published (B159)
A lookup in the root publishes the "Shared with me" entry only if the root
listing did not change while its plan was read, and answered EAGAIN when it
had. Right after a restart the root is listed again as the first lookups come
in, so a write could fail with "Resource temporarily unavailable".
ensure_children now takes the listing again, up to four times.
2026-10-04 17:50:39 +02:00
nvrl c2f93449ad fix(core): drop a rename replaced while its failure was being recorded (B158)
B141 drops a rename that fails with a newer one queued behind it. The
worker records the failure and releases its claim in two steps, and a
rename queued between them waited behind the old one, which was kept
with a backoff. Released, the old one was not due and the new one was,
so the new one landed first and the old one then put the first name
back.

Releasing the claim now drops the old rename too when a newer one is
queued, and both checks match the node by local id as well as by uid.
2026-10-04 17:23:51 +02:00
nvrl 4676a4e32b fix(fuse): cache an upload whose read-back failed, and know it as ours (B157)
When the read-back after an upload went unanswered, the drain let go of
the staged blob without caching it. A partial write opened next had
nothing to fill its gaps from and queued an incomplete revision, so the
write open after it was refused with EIO.

The bytes are now cached under the size and mtime the tree still has,
which is the base the next write takes. Their SHA-1 stands in for the
revision id the read-back did not bring, so that write chains onto the
upload and does not fork a conflict copy of itself.
2026-10-04 16:20:54 +02:00
nvrl 16fca10e6c perf(fuse): drain with 16 workers, and wake the idle ones when an op lands 2026-10-04 15:05:17 +02:00
nvrl dbc4d9a8c5 fix(core,fuse): a node written under its stand-in after landing updates its landed row 2026-10-04 13:38:54 +02:00
nvrl 9c93860cd8 refactor(core): rows and ops name a node's parent by local id; a landing rewrites nothing below it 2026-10-04 13:26:19 +02:00
nvrl 85ec91d6b3 fix(fuse): adopt a twin holding bytes an earlier attempt of the create sent (B156) 2026-10-04 13:14:29 +02:00
nvrl d5d3b2da3f refactor(core,fuse): find a queued op by its node's local id under either uid 2026-10-04 13:10:19 +02:00
nvrl e5f3f612f8 refactor(core,fuse): a node made here goes by its local id until it lands (schema 41)
A file or folder made on the mount used to get a minted `local~<ms>-<n>`
placeholder. It now gets `local~<lid>`, a function of the row it is given
first (`Db::add_local_row`), so the stand-in never changes and always leads
back to the row, and through it to the real uid once the node lands.
`Db::lid_of` reads it without a lookup.

Migration V41 rewrites the placeholders still stored to their row's
stand-in: the row and its node_json, its children, the ops on it or into
it (uid, parent, sidecar uid, a rename's original parent), and its pins.

Step 6a of the plan for 3.0.0 phase 3.
2026-10-04 12:32:20 +02:00
nvrl 3f74b95b34 fix(fuse): wait for a create on the wire that holds the name (B154); a write whose answer was lost is not a conflict (B155)
B154 is a regression from the B153 fix: after a failure on a held name the
op woke the ops waiting for it, but the database cannot see a create on the
wire, so the op was due again at once and spent its waits in milliseconds.
No wake while another create on the wire holds the name.

B155: a revision upload that reached Drive but lost its answer left the
file at a revision this daemon never heard of, and the retry kept the write
as a conflict copy of itself. A complete write whose bytes the remote holds,
by size and SHA-1, is sent again instead.

The fake Drive gains hold_reply_to_next_create and
lose_reply_to_next_revision for the two new sim tests.
2026-10-04 12:19:06 +02:00
nvrl 7269cb5118 fix(fuse): keep a rename made as its create lands (B152); wake an op once the name it waits for is free (B153)
Adoption asks again under the tree lock whether a rename is queued, and
writes the node row through the lock's outbox, so a rename made while
the create landed is no longer undone in the tree and the row.

An op that finds its name held fails as NameHeld: it backs off by the
minimum without doubling, and wakes the waiters for its name again once
its failure is recorded, so a trash that landed in between is not missed.
2026-10-04 06:04:20 +02:00
nvrl 6d75a56b00 fix(fuse,core): adopt a twin holding the create's bytes (B127); trash the node of a withdrawn create that lost its answer (B151)
A retried create now adopts the file its lost answer made when Drive's
SHA-1 matches the staged blob, not only when the file is empty. A delete
after a sent create keeps the op as a trash that finds and trashes the
node that create may have made; a new create of that name waits for it.
The simulation's settle now reads the log, so known damage is explained.
2026-10-04 05:41:57 +02:00
nvrl eb1c105bbf docs: record the full a6f36d5 account run in the 3.0.0 status 2026-10-04 04:45:48 +02:00
nvrl 85e80d8aac test(acceptance): retry only the sources a busy mirror refused; a mirror-to-mirror move carries an unsynced copy along 2026-10-04 04:45:48 +02:00
nvrl 23dc5c3723 fix(fuse): land a conflict copy in the root when its folder is gone (B150) 2026-10-04 04:45:48 +02:00
nvrl 50ae25bb88 fix(fuse): queue pdfs rename and move behind the mount's changes (B149)
With changes through the mount queued, `pdfs rename` and `pdfs move` went
straight to Drive ahead of a rename the mount still had queued, which then
landed and put its name back: the account run's B121 case on every on-demand
mount. They are now queued whenever the mount would queue, or the node has
anything queued, and rewrite a still-queued create's target.

The B113 acceptance case no longer requires creates to reach Drive at once;
under local-first they are queued and the moves rewrite them.
2026-10-04 04:45:48 +02:00
nvrl 9b2b9f24c5 feat(core): node rows name their parent by local id
Schema 40 adds nodes.parent_lid, backfilled from parent_uid. An upsert
and a new row's stub keep it in step, a folder's new row links the rows
stored before it, and a row merged into a landing folder hands its
children over. A restart places each node under its parent's row as it
is now, so a child whose stored node still names a landed folder's
placeholder lands under the folder's real uid.
2026-10-04 04:45:47 +02:00
nvrl ae6650f7e0 feat(core,fuse): queued ops name their node and folder by local id
Schema 39 adds pending_op.lid and parent_lid, backfilled from nodes. Every
op queued or retargeted gets both, and a create gets its row before it
is queued. The claim reads a parent's uid through parent_lid, and a
landing folder rewrites the ops made inside it by local id instead of by
its placeholder string. Ops queued against a row Drive listed follow it
when it is merged into the landing one.
2026-10-04 04:45:47 +02:00
nvrl f029e652a1 fix(core): keep a folder's pin when the folder lands (B148)
A folder pinned before its mkdir landed kept the placeholder uid in
pins.uid, which named no node once the landing moved the row to the
real uid. The landing now moves the pin with it.
2026-10-04 04:45:47 +02:00
nvrl 9b3651e4e8 fix(fuse): put a deadline on every remote call the drain makes
create_folder, trash_nodes, the twin listing and the uploads waited on
Drive without a limit, so one call that never answered held a drain
thread for good. They now give up after INTERACTIVE_CALL_TIMEOUT, and
uploads after that plus their size at 32 KiB/s; a missed deadline reads
as RequestTimeout, so the op backs off like any lost link.
2026-10-04 04:45:47 +02:00
nvrl 0dc476dfb7 test(sim): a steady Wi-Fi run and a drain-speed measurement
one_client_on_a_slow_link runs the Wi-Fi faults without outages and has
to drain within each settle. a_thousand_files_drain_as_fast_on_wifi_as_on_lan
times 100 folders and 1,000 files on LAN and Wi-Fi; it only prints for
now and runs when PDFS_SIM_MEASURE is set.
2026-10-04 04:45:47 +02:00
nvrl de258cbd60 ci(aur): retry the AUR host key scan
ssh-keyscan sometimes returns no key from the runner, which failed the
AUR publish for 2.8.0 and 2.10.0 at the fingerprint check. Retry the
scan up to five times with a longer timeout before checking the pin.
2026-10-04 03:43:30 +02:00
Nils Pukropp 4ea18e9187 chore(aur): update packages to 2.10.0 2026-10-04 01:42:24 +00:00
nvrl a6f36d56f3 feat(core,fuse,cli,gui): report changes Drive refuses as sync issues
A change Drive refuses because the storage is full, access was taken away,
its folder is gone, a folder limit is reached or the change was rejected
used to look like any other failure until six attempts had passed. The
drain now sorts the refusal into an issue and stores it on the queued op
(schema 38), so it needs attention at once in the app, tray and CLI. The
change stays queued and is retried; nothing is deleted.

The queue listing carries the issue and whether the op has file content.
ExportPendingOp copies that content to a new file: the Sync page offers it
as an Export button next to each issue, and the CLI as
`pdfs sync export <id> <dest>`. `pdfs sync issues` lists what needs
attention, and `pdfs sync queue` shows the issue per change.

docs/MILESTONE-3.0.0-STATUS.md tracks what is left for 3.0.0.
2026-10-04 03:18:57 +02:00
nvrl 77d0f254fc fix(fuse): never adopt a node another queued create is making
A file renamed away from a.txt while its create was on the wire, and a new
a.txt made after it, could both land on the first file's node: the second
create found a fresh, empty a.txt and took it for its own unanswered create.
The first file's rename then carried the second one's bytes away.

A create now leaves alone a node that another create is making under the
same name, and one the tree already knows. It waits for the name like for
any other queued change (docs/BUGS.md B147).
2026-10-04 03:15:50 +02:00
nvrl 4eccdeeda6 test(acceptance): retry a move while a mirror folder is busy syncing 2026-10-04 02:49:33 +02:00
nvrl 473842eefc fix(fuse): keep parked creates, renames and freed names from racing the drain
- B144: when Drive says a name is taken but nothing local holds it any more,
  the rename, move or create is sent once more under the wanted name before
  falling back to a conflict name. The trash that freed it could land between
  Drive's answer and the check.
- B145: the park sweep no longer drops a parked create whose node row is not
  written yet. It waits a minute, and keeps open files, before taking a
  missing row for a deleted file. The account run's B70 check failed with
  EBUSY on this.
- B146: a rename queued while an upload's read-back is on the wire is kept;
  the tree no longer takes back the name Drive still had.
2026-10-04 02:49:33 +02:00
nvrl 5e0f30cc2c feat(fuse,core): record namespace changes locally and send them from the queue
A new folder, a new file, a rename or a delete through the mount no longer
waits for Drive: it is recorded locally, the call returns at once, and the
drain sends it as it does offline. `"local_first": false` in config.json
brings back the old path for this release.

Fixes found by the simulation runs on the way:
- B129: a file deleted while its first upload is on the wire is trashed once
  the upload lands.
- B132, B140: files made in a new folder survive a restart; a landed folder
  hands its queued children over in the stored node too.
- B138: a create waits for the rename or trash that frees its name.
- B139: a file closed while its first upload lands keeps its last write.
- B141: a rename a worker holds stays queued, and the newer one waits behind
  it instead of racing it.
- B142: a rename of a file whose create lands meanwhile goes to the uid Drive
  gave it instead of answering EBUSY.
- B143: a create whose read-back fails as the link drops is adopted before
  the next op for it is sent, so its next write is not a conflict copy.
2026-10-04 01:50:38 +02:00
nvrl b3e33a7002 fix(core): never give out a pending_op id twice
A plain rowid is the largest one plus one, so a trash queued in the same
transaction that dropped the newest op took its id. The drain retires the
op it sent by id, so a rename landing deleted the trash queued over it and
the file stayed on Drive (B137). Schema 37 rebuilds pending_op with
AUTOINCREMENT, keeping every id and index.
2026-10-03 23:44:12 +02:00
nvrl f6456829fb chore(release): 2.10.0 2026-10-03 21:57:23 +02:00
nvrl 327f538fed test(core): migrate a database 2.8.2 wrote, with ops in every state
A dump of a database the 2.8.2 build wrote, with queued ops due, parked,
failing, deferred for access and claimed, and creates queued below a
folder made offline. Its migration checks every column of the queue and
the sealed revisions, as the real-database test now does too.
2026-10-03 21:55:15 +02:00
nvrl ad8b225cfd refactor(fuse): one tree for every mount
On-demand sync folders kept their own State, which the drain, the
event applier and the sweeps had to visit one by one (B74). Every mount
now serves from the primary's tree. A session keeps only its root, which
the kernel knows as inode 1, and its notification channel; work that
changes the tree tells every mounted session. Size upgrades become
daemon-wide, and moves compare mounts by root instead of by state.
2026-10-03 16:53:31 +02:00
nvrl ea27f2bcd0 test(core): migrate a copy of a real database
An ignored test takes a copy of the database named by PDFS_MIGRATE_DB
with VACUUM INTO over a read-only connection, migrates it, and checks
that every queued op survives unchanged, every node keeps its row
number as its lid, and no node is lost or doubled.
2026-10-03 16:53:31 +02:00
nvrl e4062cc97b feat(fuse,core): number inodes by local id
A node's inode is now derived from its row's local id instead of a
per-run counter, so the same node keeps the same inode across restarts
and across every place that interns it. A lid handed out before the
row is stored reads as no node.
2026-10-03 16:41:26 +02:00
nvrl c34cc93c45 fix(fuse,core): keep a rename made while its create is on the wire
A rename of a node whose create is still queued rewrites the create's
target, but the drain had already copied the op. When the create
landed, the row was deleted unread and the rename was lost (B136).

finish_create now learns where the create was sent and where it landed,
and queues a rename of the real uid when the row has moved on since.
adopt_real_uid keeps the tree's name and folder while that rename is
queued. A landed folder's children also stop naming its placeholder as
their parent in memory, which put the placeholder back on their rows.
2026-10-03 16:20:00 +02:00
nvrl 7e3df7602d feat(core): give every node row a local id that never changes
First step of phase 2 of docs/MILESTONE-3.0.0.md. Schema 36 rebuilds
`nodes` with `lid INTEGER PRIMARY KEY AUTOINCREMENT`. Each row takes its
old rowid as its lid, so the search index, keyed by rowid, stays valid.
AUTOINCREMENT keeps a deleted row's lid from coming back. `uid` stays
unique and becomes nullable, for nodes that are not on Drive yet.

A landed create's row now takes the real uid in place, keeping its lid.
When a listing had already stored the real node, its content is kept
under the placeholder's lid, which the node was known by first.

Nothing reads the lid yet apart from `Db::lid_of`.
2026-10-03 15:10:51 +02:00
nvrl 61a0d31408 chore(release): 2.9.0 2026-10-03 15:00:38 +02:00
nvrl 644a47eb36 fix(ci): quote the simulation command so the workflow parses
The ': ' in 'sim:: --' read as a YAML mapping, so GitHub rejected
ci.yml and ran no job at all since 2c46d62.
2026-10-03 14:55:13 +02:00
nvrl cd90c566cd fix(fuse,core): renames across folders, replaced files, and writes after a create
Found by the seeded simulation runs (docs/BUGS.md).

- B122: a rename across folders goes to Drive as one move-multiple
  request, so a sibling holding the new name in the old folder no longer
  fails it with EIO. The drain uses the same request.
- B123: a second write open on a file just written no longer fails with
  EIO. An empty file counts as complete, our own renames, moves and
  online creates record their echo, a duplicated event is skipped, a
  rename keeps the file's mtime while its revision stays, and a landed
  upload is cached before open handles are rebased onto it.
- B124: a node put back after a failed replacing rename leaves `hidden`
  and its folder is listed again, so it shows in the mount.
- B131: a landed create is read back with the same short retry as an
  online create, and its bytes are cached under the real uid before an
  open handle is repointed at it.
- B135: an online rename settles the tree through the state guard, so
  the database has both folders unlisted before the reply. A lookup
  right after it no longer rebuilds the destination without the file.

The simulation runs no longer tolerate B122 or B131.
2026-10-03 14:50:12 +02:00
nvrl 1c262eb601 refactor(fuse,core): keep queued-op preconditions in the database
Phase 1 of docs/MILESTONE-3.0.0.md, without a change in behaviour.

- Own sealed revisions live only in the own_sealed_rev table. The
  in-memory map in front of it is gone, so revision_conflict gives the
  same answer before and after a restart.
- A queued rename records the node's original name next to its original
  parent, and keeps both across renames that replace the row. The drain
  logs when Drive has the node neither where the rename found it nor
  where it takes it.

The SDK's NodeUpdated event carries no revision id, so echoes of our own
changes stay recognised by the self-change timer.
2026-10-03 14:49:58 +02:00
nvrl bf73a395e5 docs: record B132-B134 from the simulation runs
- B132: a folder whose mkdir is still queued is gone after a restart.
  The root's listing, served offline, was invalidated when the link
  came back, and the restart listed it on Drive without the queued
  folder.
- B133: a file deleted while its upload is in flight can still come
  back as a recovered copy. B105's check goes by op id, and SQLite gives
  the next op queued the id the delete freed.
- B134: a file deleted online while the drain checks its write is kept
  as a conflict copy. The unlink trashes the file on Drive before it
  drops the queued write, so B105's check still finds the op.
2026-10-03 09:07:56 +02:00
nvrl a5138639a8 test(fuse): delete files as they upload in the simulation runs
The runs did not find B105 with its fix reverted. Drive lists a trash
later than the round trips take, so the drain never saw its own
client's delete while it still held the write.

The FakeDrive can now hold the next read of one node on its way to
Drive. A new profile, one-client-deletes, writes a file Drive holds,
waits for the drain to read the node back, deletes the file, and lets
the read go on once Drive shows the trash. A conflict the drain keeps
because a client's own file was trashed now fails the run as a false
conflict. With B105's fix reverted, seeds 1 and 2 fail on it.

Outside the step, such a conflict is counted as B134 when the copy is
named after the file: the drain looked after an online unlink had
trashed the file on Drive but before it dropped the write. Seed 3 of
three-clients came up with it once.

Half the time the step makes the file again before the read goes on.
That is B133: the new file's write takes the op id the delete freed,
and B105's check finds it. The runs count it.

The runs also count B132, an ENOENT after a restart that restored
queued ops, in a seed that lost the link. Seed 4 of one-client-flaky
failed on it. The known-bug matchers now also see what was logged
before the call.
2026-10-03 09:07:56 +02:00
nvrl 988b78e810 docs: record the bugs the simulation runs found (B122-B131)
The first simulation runs found ten open bugs. They are documented here
and not fixed yet, as phase 0 of docs/MILESTONE-3.0.0.md changes no
behaviour. The runs count each one and end the seed where the mount
leaves the model behind.

- B122: a rename across folders fails with EIO when the source folder
  holds the new name.
- B123: a second write open on a file just written fails with EIO.
- B124: after a replacing rename fails, the file it was replacing is gone
  from the mount.
- B125: a folder made offline fails with EIO once its listing is
  invalidated.
- B126: a file created offline under a name an offline rename freed
  lands as a conflict copy.
- B127: a file created offline lands twice when the answer to its upload
  is lost.
- B128: a file created offline over one deleted offline can be lost.
- B129: a file deleted or replaced while its create is on the wire stays
  on Drive.
- B130: a read of a file that a remote change invalidates can hang the
  daemon. The event sync blocks a runtime worker in inval_inode on the
  read's page lock. Nothing then drives the runtime's timers, which the
  read's fetch is waiting for.
- B131: a write to a file whose create could not be read back lands as a
  conflict copy. The create's op is retired before the read-back, so a
  failed read-back leaves the tree on the placeholder's mtime and no
  revision id.
2026-10-03 07:47:02 +02:00
nvrl 2c46d62429 ci: run the simulation runs outside pull requests
A `simulation` job runs the seeded simulation runs of
docs/MILESTONE-3.0.0.md section 8.2 on pushes to main and manual runs. They
mount FUSE against the in-memory FakeDrive, so they need fuse3 but no
account. Like the acceptance suite they stay off pull requests. The job
has a 30 minute limit, and the runs' own watchdog turns a hang into a
failed seed well inside it.
2026-10-03 07:47:02 +02:00
nvrl 97743f17e3 test(fuse): add seeded simulation runs over the FakeDrive
The second half of the simulation harness in docs/MILESTONE-3.0.0.md
section 8.2. A run starts one or more real FUSE daemons on a shared
FakeDrive. Each client owns a folder `c<n>` that only it writes. A seed
then draws syscalls through the mounts, link flaps, restarts, pauses and
settles. Every syscall is checked against a model of the client's tree as
it happens. After the last step every link comes up and the run checks
no loss, POSIX behaviour, no false conflicts, convergence and liveness.

- sim::model is the POSIX model: the tree, what each call should answer,
  and how it changes the tree.
- sim::run has the profiles: one client on a good link, one on a flaky
  link with flaps and restarts, and three clients. PDFS_SIM_SEEDS sets
  how many seeds each runs, and PDFS_SIM_SEED replays one.
- sim::watchdog catches a call that never returns. It takes every
  thread's stack in the process, since no tracer can stop a thread
  blocked on a FUSE request. It then aborts the run's FUSE connections
  and writes the stacks next to the run's state.
- The daemons' logs at INFO and above are kept, so a known open bug is
  told apart from a new failure. Known bugs are counted, and the seed ends
  where the mount leaves the model behind. PDFS_SIM_KNOWN=fail fails on
  them instead.
- A restart retries opening the database while the last run still holds
  it.
- The FakeDrive restamps a node's modification time on a rename or move,
  as Drive does (B113).

The runs mount FUSE, so they are ignored by default. Run them with
`cargo test -p pdfs-fuse --lib sim:: -- --ignored --test-threads=1`.
Test-only; no behaviour change.
2026-10-03 07:47:01 +02:00
nvrl 3d10338807 test(fuse): run real FUSE daemons against the FakeDrive
mount() becomes a thin wrapper over mount_with(), which takes the
DriveApi to reach Drive through and a Host. The daemon's Host is
unchanged: stopped by SIGTERM or SIGINT, installed at the same point as
before, and indexing the user's home. A simulated daemon is stopped over
a channel and skips the home index, so several can share one test
process without walking the developer's home.

The account quota moves onto DriveApi. It was the only call outside the
trait that marks the link lost, so a simulated daemon would have gone
offline on it; the FakeDrive now also models a full account
(InsufficientQuota).

sim::daemon starts one daemon on its own state directory with a real
FUSE mount, and a smoke test (ignored; needs /dev/fuse) checks a local
write lands, a remote write arrives, and a write made offline lands once
the link is back.
2026-10-03 05:09:02 +02:00
nvrl d53ba1be21 test(fuse): add an in-memory FakeDrive with Drive's known faults
The first half of the simulation harness in docs/MILESTONE-3.0.0.md
section 8.1. FakeDrive is one shared server; each simulated daemon reaches
it through a FakeClient that implements DriveApi and carries its own link
state, latency, faults and an SDK-style entity cache.

The faults are the ones the account runs met: listings that lag (B118,
B48), renames and moves refused for a stale name hash because the cache
is only emptied by events (B111, B121), names that refuse a replace (B13,
B104, B106), events delayed, duplicated, reordered and echoed (B70, B105),
requests that stall, fail, or change Drive and lose their reply (B107,
B121), and non-uniform block sizes (B84, B85, B87).

Test-only; no behaviour change.
2026-10-03 04:51:35 +02:00
nvrl c73d5c9436 refactor(fuse): reach Drive through a DriveApi trait
`Core` held a concrete `ProtonDriveClient`, so no test could make Drive
answer late, answer stale or not answer. This is the first step of
phase 0 in docs/MILESTONE-3.0.0.md.

`DriveApi` (`crates/pdfs-fuse/src/drive.rs`) covers the calls the sync
core makes: listings, events, creates, uploads, downloads, renames,
moves, trashes and restores. `ProtonDriveClient` implements it by
forwarding. `Core::drive` holds it, and every call of those methods now
goes through it. Sharing, Photos, devices, invitations and public links
stay on `Core::client`.

The trait takes readers and writers as trait objects, and
`impl dyn DriveApi` puts the client's generic signatures back, so call
sites read the same. Block reads go through `RevisionRead` instead of
the SDK's `RevisionReader`, which only the SDK can construct.

No behaviour change.
2026-10-03 04:37:37 +02:00
nvrl 8d92c69279 fix(fuse): keep an upload pending until its bytes are cached (B119)
The drain retired an upload's op first, and only then read the node back
and linked the staged blob into the content cache. For that one round
trip, the status reported an empty queue for a file that wasn't cached
yet, so anything that waited for an empty queue before reading the file
could still need the network. The account run of 2026-10-03 hit it over
a slow link: "regression B119" failed in the same second the upload
landed.

`Core::landing_uploads` now counts an upload from just before its op is
retired until its bytes are cached. The status adds it to
`pending_uploads`, and reads the queue before the count, so it always
sees the upload in one of them. A drop guard uncounts it on every exit
path.

The op still goes first. An op restored after a crash would re-send a
revision this daemon hasn't recorded as its own yet, and
`revision_conflict` would keep it as a conflict copy of the user's file.
2026-10-03 03:59:05 +02:00
nvrl e32959e40a fix(fuse): land a move right after a rename, and rename across folders quickly (B121)
The SDK caches each node, including the name hash it sends with the next
rename or move, and only an event evicts it. Renames and moves didn't
update that copy. So after a rename, the next change to the same node
went out with the old name hash, Drive refused it as out of date (422),
and the retry kept sending the stale hash until the event feed came
round, up to 10 seconds later.

- Every remote rename and move goes through `Core::rename_remote` or
  `Core::move_remote`. Both drop the SDK's copy of the node after a
  successful change and before each out-of-date retry, so the retry
  sends the current hash.
- A plain move was not retried at all, so a move right after a rename
  failed with EIO through the mount and with "out of date" from
  `pdfs move`. It now retries like a rename.
- The drain's combined move-and-rename also drops the SDK's copy once
  it lands.
- A rename that also changes folders through the mount replies only
  once both halves have landed. When the move fails, the file keeps
  its new name in the source folder, the tree is told so, and the
  rename reports EIO. A dropped link still queues the change.

B111's cause in docs/BUGS.md is corrected to the same SDK cache. The
acceptance case "regression B121" renames and moves a file at once
through the mount and through `pdfs`, and checks names and bytes.
2026-10-03 03:58:38 +02:00
nvrl 0b27c884a7 test(acceptance): turn B114 to B117 into regressions and ride out a dropped link
New live cases:

- regression B48, B80, B94, B95, B98, B102 and B111 replay the bugs
  their entries describe. B48 and B111 also run against the reference.
- regression B114 to B117 replace the open B114 and B115 cases, and
  fail with `check` when the bug comes back.
- regression B119: files written through the mount stay cached once
  they are uploaded.
- regression B86: after a mirror folder switches to on-demand, the
  mount lists what the mirror uploaded just before.

`--journal-check` also counts three warnings as errors: a conflict
copy (B113), a FUSE job held for a long time (B110) and an authority
re-read because the local tree lacked it (B117, B120).

A timeout no longer ends a target when the mount still answers a stat
and a listing. Cases that make many round trips get twice the limit.

A write is queued when the kernel releases the file, after close(2)
has returned, so cases now wait for the ops they expect
(`Daemon.queued`). "regression B102" saw one revision of two because
it read the queue once.

On flaky Wi-Fi, the run crashed twice on a read that the daemon
correctly failed with EIO once its read budget ran out. The end-of-run
read-back and the mode matrix's file checks now record a result
instead. The mode matrix tries such a read once more after the daemon
is back online, and notes it.
2026-10-03 02:10:49 +02:00
nvrl 37c04f1e69 fix(fuse,core): fix B114 to B120 from the account runs
- B114: `pdfs rm`, `pdfs rename` and `pdfs move` tell each kernel
  session that the old name is gone, so it stops resolving at once
  instead of for up to 30 seconds (`Core::forget_everywhere`,
  `Core::relocate_everywhere`).
- B115: `pdfs ls` and the app's file browser wait up to 10 seconds for
  the listing's size fetch, as `stat` does, so they show a file's real
  size instead of its encrypted one.
- B116: trashing a folder also drops the creates queued in its
  subfolders. A parked one stayed queued with its bytes, and a due one
  would have landed in the root.
- B117: `pdfs move` and `pdfs rename` move the node in each mount
  instead of forgetting it, so its queued upload keeps its path in
  `pdfs sync queue` and is not put off once after the resume.
- B118: when a create or mkdir reads back nothing, the mount asks again
  and then queues it as it does offline, instead of failing with ENOENT
  for something Drive has made.
- B119: a landed upload keeps its staged bytes as the file's cached
  content, unless it is larger than a sixteenth of the cache, so
  reading a file just written does not need the network.
- B120: a landed folder's row passes to its real uid in the same
  transaction, so a file queued inside it is not put off for five
  seconds.

docs/BUGS.md records B117 to B120. The account run of 2026-10-03
verified B114 to B117 and B119 in My files.
2026-10-03 02:10:23 +02:00
nvrl 625776081b test(acceptance): cover B113, B88 and the closed pipe, and report more readably
New live cases:

- regression B113: files renamed and moved while sync is paused upload
  without a conflict copy.
- regression B88: pdfs mkdir, rename and rm by local path, the
  mount-root guard, a path outside every mount, and the refusal in a
  mirrored folder.
- pdfs output into a closed pipe exits 0 and prints nothing on stderr.
- open B114 and open B115 check two bugs that are still open. While
  one reproduces, its case ends as "known", which is listed but does
  not fail the run.

The console output names the pdfs version, the storage and the daemon
state of each target, and prints one aligned line per case, with its
details beneath. A failed case points to its docs/BUGS.md entry. The
run ends with a summary table, the slowest cases, the known issues
and the numbered failures. A queue wait now fails at once on a parked
op instead of waiting out its timeout.

docs/BUGS.md records B114, B115 and B116, which the first live run
found, and marks B113 and B88 as verified live.
2026-10-02 22:16:36 +02:00
nvrl 8cde79b8e3 fix(cli): stop quietly when stdout is closed
`pdfs ls | head` ended in a "Broken pipe" panic once `head` had its
lines, because std's `println!` panics on a failed write.

The CLI now shadows `println!` and `print!` with versions that exit 0 on
a closed pipe and report any other write error, such as a full disk,
with exit 1. Restoring the default SIGPIPE would also stop the panic,
but the same binary runs the daemon, which must not die when a socket's
peer goes away.
2026-10-02 21:16:34 +02:00
nvrl eb53d122cc fix(fuse,cli): let rm, rename and mkdir take paths in on-demand folders
`pdfs rm ~/Downloads/x` answered "is not under the mountpoint" while
`pdfs move` took the same path: the Delete, Rename and CreateFolder
handlers still read paths against the primary mount only (B88).

They now go through `route_change`, the router ResolveConflict already
used, renamed from `route_conflict`. It answers the request in the
inode space of the mount that owns the path.

An on-demand folder's root has a parent on Drive, so `pdfs rm
~/Downloads` would have trashed the whole synced folder from under its
mount. rm and rename refuse a path that names a mount's root.
2026-10-02 19:54:37 +02:00
nvrl c64252193f fix(fuse): keep the revision id of a file created through the mount
The write handle a create opens was given no revision id, so the drain's
conflict check fell back to (mtime, size). Drive advances a node's mtime
on a move or rename, so a new file moved before its first upload drained
landed as a (sync-conflict) copy next to its own empty original. Take the
minted node's revision id, and rebase handles still open on a queued
create onto the node it landed as (B113).
2026-10-02 19:25:33 +02:00
nvrl cc70a21c05 fix(aur): version -bin source filenames
makepkg reuses a cached source file of the same name, so the unversioned
proton-drive.service from an older build failed the 2.8.2 checksum. Carry
$pkgver in every file fetched from the tagged tree, like LICENSE already
did, and bump pkgrel to 2.
2026-10-01 23:50:06 +02:00
Nils Pukropp 644995cc56 chore(aur): update packages to 2.8.2 2026-10-01 19:49:42 +00:00
nvrl 50483bf3e1 chore(release): 2.8.2 2026-10-01 21:39:05 +02:00
nvrl 4caba0a34c chore(deps): require proton SDK 0.7.2
0.7.2 resolves each shared root with its own key. With 0.7.1, when
several accepted shares came from the same owner's volume, only the first
appeared under Shared with me and the rest were skipped with "missing
key" (B112, proton-sdk-rs#2).
2026-10-01 18:17:25 +02:00
Nils Pukropp e8554d6434 chore(aur): update packages to 2.8.1 2026-10-01 14:58:09 +00:00
Claude ba1fdd65be chore(release): 2.8.1 2026-09-30 17:02:31 +00:00
nvrl 0d4163b9bf fix(service): start with the graphical session, not default.target (#24)
default.target is reached before the desktop exports DISPLAY/WAYLAND_DISPLAY
to the user manager, so After=graphical-session.target never applied. The
daemon's first keyring call D-Bus-activated the Secret Service without a
display; on KDE ksecretd aborted, the first start failed, and every login
left a coredump.

Install into graphical-session.target instead, and use reenable when the
front-ends enable the unit so existing default.target.wants links are
replaced rather than kept alongside the new one.

Fixes #23
2026-09-30 19:01:50 +02:00
Nils Pukropp f3c7a809f4 chore(aur): update packages to 2.8.0 2026-09-30 02:15:19 +00:00
nvrl c1b5378045 fix(aur): download sources from GitHub, not the homepage
73e60f1 pointed url at proton-drive.narl.io, and all three AUR PKGBUILDs
built their source URLs from $url, so the release tarball, the binary
archive and the git clone were fetched from the homepage and 404ed. The
sources now come from a separate _repo on GitHub; url stays the homepage.
2026-09-30 04:13:06 +02:00
nvrl 3d6f53c16d chore(release): 2.8.0 2026-09-30 03:56:47 +02:00
nvrl 8ceee5285b chore(fuse): log how long each queued upload takes
"pending upload landed" now carries upload_ms, the time on the wire, and
total_ms, the whole op including the conflict check and gap-fill. A slow
drain then shows where its time goes without a profiler.
2026-09-30 03:55:48 +02:00
nvrl a181f1a5f0 fix(core,fuse): keep working through network drops
The mount only ever went from offline to online: nothing marked it
offline again, so after the Wi-Fi dropped every create, mkdir, delete and
rename called Drive, waited out the SDK's retries for up to two minutes
and answered EIO. Background loops kept polling the dead link.

Connectivity now lives in pdfs-fuse's new link module and goes both
ways. A call that gets no answer (a transport error or a request
timeout) marks the mount offline; a probe thread checks every 2 s,
backing off to 30 s, and on recovery re-lists folders served from the
database, retries failed operations and wakes the drain. Interactive
calls get a 20 s deadline, so a dead link costs one slow call instead of
many.

While offline:
- create, mkdir, unlink, rmdir and rename fall back to the offline
  queue instead of answering EIO;
- folders list from the database even when not marked listed, and are
  re-listed once the link returns;
- reads of uncached data wait up to 30 s for the link, then retry once
  within the 120 s read deadline;
- statfs serves the last quota instead of fetching, and quota refreshes
  are single-flight;
- the event and photo event loops wait for the link instead of polling.

A create whose request timed out may have landed anyway. When the drain
replays it and finds a same-named twin of the same kind, created within
the last two minutes and still empty, it adopts that node instead of
making a conflict copy. The drain also leaves alone a name still held
by a queued trash.
2026-09-30 03:25:00 +02:00
nvrl 7251f00e91 chore(deps): require proton SDK 0.7.1
0.7.1 sends TCP keepalives on its connections, so a link that dies
silently is noticed instead of leaving a request waiting on a dead
socket until its timeout.
2026-09-30 03:25:00 +02:00
nvrl cb38f79aa3 fix(fuse,scripts): log why a create finds no parent and fit the metadata floor to FUSE
A create that answered ENOENT left no trace in the log. It now warns
when the parent folder is no longer known, and when the parent left the
tree while its children were being listed, naming the parent inode and
the new name.

The acceptance script's metadata throughput floor was 20 operations per
second, which a FUSE mount cannot reach: each create and unlink waits for
Drive before it returns, about 350 ms apiece. On a FUSE mount the floor
is now 3 operations per second; other paths keep 20. The mount type is
read from /proc/self/mountinfo.
2026-09-29 21:22:54 +02:00
nvrl 54e9921efb fix(fuse): stop reads deadlocking on size upgrades and retry stale renames
A read that settles a provisional size invalidated the file's page cache
while the kernel held that read's page lock, so the worker waited on
itself forever (B110). Size upgrades now invalidate attributes only.

Renaming a node right after renaming it could be refused by Drive as out
of date, because the name hash read just before still held the old name.
Every remote rename now goes through Core::rename_remote, which retries
InvalidRequirements for up to ten seconds (B111).
2026-09-29 19:47:20 +02:00
nvrl f252b09757 fix(fuse): keep the size of a file open for writing across re-listings
B109: an online create seals an empty file on Drive, and until the last
close the written bytes live only in the open handle. A re-listing of
the parent, a size upgrade or a create landing re-interned the node with
Drive's empty size, the kernel capped reads at that size, and SQLite
reported "database disk image is malformed". State::keep_open_write_size
now carries the handle's size and local mtime over in all three places.

The acceptance script waits for My files to mount after a daemon restart
instead of failing at once.
2026-09-29 16:49:13 +02:00
nvrl a2992b3b9d fix(core,fuse): keep replaced and vanished files from failing saves, reads and syncs
Fixes from the account acceptance runs:

- B104: a reader open across a rename-over keeps working; the replaced
  node is unlinked like any other open file instead of forgotten.
- B105: a drain worker holding a revision withdrawn by an unlink no
  longer uploads it as a recovered-… conflict copy (Db::op_exists).
- B106: an open victim's queued ops are withdrawn at trash time, so a
  repeated editor save no longer fails with AlreadyExists; the staged
  blob goes with the last close.
- B107: a remote read gives up after 120 s with EIO instead of hanging
  its caller; the timer is built inside the runtime.
- B108: a mirror upload whose local file was deleted after the scan is
  skipped rather than counted as an error.

The acceptance script reports the failing line for non-assert errors,
ignores queue ops that predate the run, purges with delete-forever
first, and diagnoses a malformed SQLite database against a local replay.
2026-09-29 16:20:51 +02:00
nvrl 7b67db5ced fix(core,fuse,cli): harden writes, trash and cleanup found by account acceptance runs
The FUSE acceptance suite now runs on the signed-in account by default and
cleans up after itself; its first live runs found three daemon bugs.

- B101: a pwrite past EOF now authors the hole it leaves, so truncated bytes
  no longer come back from the base. A write attached to a create while that
  create uploads becomes a revision of the real uid instead of being dropped.
- B102: removing a folder drops the queued revisions of files below it, found
  through the node tree, with their blobs and in-flight uploads. `pdfs rm`
  now drops queued ops too.
- B103: each server call of a trash refresh times out after 120 s, so one hung
  call can no longer freeze the trash listing until restart.

Acceptance suite: account mode with a manifest, lock and reaper; cleanup by
uid; queued ops naming a test root count as leftovers; new edge-case,
throughput and B100/B101 cases; the wide-directory case is split from the
deep tree and gets a doubled budget. Adds `pdfs trash --json`.
2026-09-29 10:09:44 +02:00
nvrl 84c9ade8e3 fix(fuse): settle provisional sizes before reads and write opens
A file listed by the cheap enumeration reports its ciphertext size until
the size upgrade lands. Since B84 a read on that size fails with EIO, and
a write open started its handle at it. Reads and write opens now fetch
the real size first. A folder past the size-upgrade cap is queued instead
of dropped, so a large copy no longer leaves most folders provisional
(B100).
2026-09-29 07:37:41 +02:00
nvrl 9e26cd0285 chore(release): 2.7.0 2026-09-29 05:30:38 +02:00
nvrl 73e60f1151 chore: point homepage metadata at proton-drive.narl.io
The About dialog's website link, the README, and the package metadata
(Cargo, PKGBUILDs, AUR .SRCINFO, .deb, .rpm) now use the project
homepage. Source code, issue, and release links stay on GitHub.
2026-09-29 05:30:38 +02:00
nvrl 088e8f3b0d test(acceptance): cover moves between locations in the managed matrix
Every managed mode pair now also moves trees between the two folders in both
directions, moves several sources in one command, and passes a tree through
My files and out again. Each case waits for Drive before and after, then
checks the tree is only at the destination, byte for byte, with no conflict
copies. Refusal cases cover a taken name, a folder into itself, a mirror
source holding a symlink, and an on-demand file still open for writing.
2026-09-29 00:38:40 +02:00
nvrl 261d5dbf84 feat(core,fuse,cli): reuse cached content and take several sources in moves
- A mirror download first copies the file out of the content cache when the
  same revision is cached whole or block by block, so moving a file out of an
  on-demand folder into a mirror costs no traffic for what was already read.
- `pdfs move a b c DEST` moves every source, reports each failure and exits
  non-zero if any failed.
- Mirror-to-mirror moves rename with RENAME_NOREPLACE, so a name that
  appeared at the destination after the check is never overwritten.
- The local copy a move out of a mirror drops is checked again just before
  it is deleted; anything written into it meanwhile keeps the copy and is
  uploaded as new by the next pass.
2026-09-29 00:38:40 +02:00
nvrl 5f9d07aff1 feat(fuse,core,cli,gui): move between locations without transferring content
A move request with two absolute paths is resolved against every
location (My files, on-demand and mirror synced folders) and made as a
server-side move_node. The local side is levelled afterwards: mounts
forget and re-list, a mirror folder renames its copy and baseline along
or drops the copy once Drive holds it.

Anything only this disk holds is refused up front: queued ops, open
writes, unsynced or ignored mirror files and symlinks. The GUI Move
dialog gains a top-level list of locations.
2026-09-29 00:26:56 +02:00
nvrl 3fb7b9be6b chore(release): 2.6.2 2026-09-28 23:58:26 +02:00
nvrl 6c3a50ce7f fix(gui): let the Sync page use the full width
The transfer label did not ellipsize, so a long file name set the
column's minimum width and pushed it past the clamp. The page width
changed with every file.

The label now ellipsizes in the middle, and the Sync page drops the
clamp so Overview and Folders fill the space next to the sidebar.
2026-09-28 23:58:26 +02:00
nvrl f8304a0e86 fix(gui): keep Albums and Places selected when returning to Photos
Every visit to the Photos page loaded the timeline, and a finished
timeline load always shows the timeline. Leaving Albums or Places and
coming back therefore showed the timeline under the Albums or Places
toggle.

A visit now reloads the view the toggle selects. An open album or place
still loads its photos as before.
2026-09-28 23:58:26 +02:00
Nils Pukropp e63f39f8c2 chore(aur): update packages to 2.6.1 2026-09-28 21:44:18 +00:00
nvrl 6f100dbe5e chore(images) updated prompt image 2026-09-28 23:35:53 +02:00
nvrl 7c5ef269cd chore(release): 2.6.1 2026-09-28 23:35:14 +02:00
nvrl f13b6fc0ef feat(gui): add pdfs-prompt --preload for session autostart
--preload registers the resident GTK launcher and realizes its window
without showing it, so the first shortcut opens it as fast as later
ones. When a launcher is already running it exits without activating
it.

The application hold now happens in startup, which only the primary
instance runs. Taken before registration, it also kept every remote
invocation alive after it had forwarded its activation.
2026-09-28 23:35:14 +02:00
nvrl 3d61fc637b ci(release): reuse cached builds from main
The release job never hit a cache: its rust-cache key contained the job
name, tags cannot restore caches of earlier tags, and rpmbuild rebuilt
the whole workspace under its own compiler flags.

CI on main now saves shared debug and release caches that the release
workflow restores. Quality gates and the package build run in parallel,
and the GitHub release is created once both pass. The RPM spec skips
its cargo build when the caller passes --define "prebuilt 1".
2026-09-28 23:26:24 +02:00
Nils Pukropp 889b2aa4a9 chore(aur): update packages to 2.6.0 2026-09-28 21:22:30 +00:00
nvrl 8bd3867cb3 fix(gui): keep the launcher preview at its fixed width
A wrapping label asks for its whole text on one line as its natural
width, so a long location in the preview pane widened it, squeezed the
results list and stretched the window. Cap the preview labels' natural
width, limit detail values to three lines with a tooltip for the full
location, and keep the pane from taking extra width.
2026-09-28 23:04:48 +02:00
nvrl 9577387c2f chore(release): 2.6.0 2026-09-28 22:57:28 +02:00
nvrl 7a087b8827 feat(gui): rewrite the search launcher with a preview pane
The launcher (pdfs-prompt) now follows the app's theme: the theme code
moved into a shared theme.rs, the prompt's stylesheet lives in
resources/prompt.css on the libadwaita palette, the Proton-purple accent
pin is gone, and every activation re-reads the Proton theme and open_with
settings so changes in the app apply without restarting the launcher.

The window is rebuilt in src/hud/:
- one list with Recent, Pinned and Best matches sections; a click
  selects, a double-click or Enter opens, focus stays in the entry
- rows show a thumbnail, the name with its matched characters in bold,
  location, sync badge, size and age
- a preview pane with a large thumbnail, type, size, modified date,
  location and availability, plus Open (Enter), Show in Folder
  (Ctrl+Enter, via FileManager1) and Copy Path (Ctrl+C / Ctrl+Shift+C)
- thumbnails from the freedesktop cache, local decoding (written back
  to the shared cache) or the daemon, on two worker threads
- a history of opened files that fills the Recent section and adds a
  frecency bonus to search ranking
- the mount check before opening a Drive file runs off the main thread
  with a 2 s timeout, and Escape cancels a slow open
- rows can be dragged out as files
- the search debounce drops to 80 ms

New strings are translated in every catalog.
2026-09-28 22:53:46 +02:00
nvrl 7099106802 docs: show the Places map screenshot
Rename photos_mpa.png to photos_map.png and add it to the README
screenshot table.
2026-09-28 22:10:14 +02:00
nvrl 43385d36ee chore(images) added up to date images 2026-09-28 22:09:07 +02:00
nvrl 4b97f401cb docs: refresh screenshots
Replace the Photos screenshot with one of placeholder stock photos, add
Sync, conflict history and Computers screenshots, and drop the retired
Activity, Settings, Shared and restore-devices images from the README.
2026-09-28 22:08:16 +02:00
nvrl af869944cf docs: restructure and update documentation for 2.5.0
Split the README into a short front page and task-oriented pages under
docs/: installation, user guide, CLI reference, configuration and
troubleshooting, with an index in docs/README.md. Rewrite RECOVERY.md
around Continue This Backup Here and Restore Folders, and DEVELOPMENT.md
around running, debugging and releasing from source.

Bring ARCHITECTURE.md up to date with the read path, push events, the
thread map, conflict copies and the machine profile, and move the feature
design notes there. Collect planned work in ROADMAP.md. Link the threat
model from SECURITY.md and the translation rules from CONTRIBUTING.md.

Add scripts/publish-wiki.sh, which mirrors the user pages to the GitHub
wiki with rewritten links and only pushes with --push.
2026-09-28 20:04:22 +02:00
Nils Pukropp 4b90abf3b5 chore(aur): update packages to 2.5.0 2026-09-28 16:56:58 +00:00
144 changed files with 42866 additions and 12768 deletions

No files matched your search

+7 -1
View File
@@ -67,7 +67,13 @@ jobs:
User aur User aur
EOF EOF
# Pin the host key to the fingerprint published on aur.archlinux.org. # Pin the host key to the fingerprint published on aur.archlinux.org.
ssh-keyscan -t ed25519 aur.archlinux.org > "$HOME/.ssh/known_hosts" # The scan sometimes returns nothing from the runner, so retry it.
for attempt in 1 2 3 4 5; do
ssh-keyscan -T 15 -t ed25519 aur.archlinux.org > "$HOME/.ssh/known_hosts" &&
[ -s "$HOME/.ssh/known_hosts" ] && break
echo "ssh-keyscan attempt $attempt returned no key; retrying"
sleep 10
done
ssh-keygen -lf "$HOME/.ssh/known_hosts" | grep -q 'SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4' ssh-keygen -lf "$HOME/.ssh/known_hosts" | grep -q 'SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4'
chown -R builder: "$HOME/.ssh" chown -R builder: "$HOME/.ssh"
chmod 600 "$HOME/.ssh/aur" chmod 600 "$HOME/.ssh/aur"
+87
View File
@@ -32,8 +32,14 @@ jobs:
with: with:
toolchain: '1.96' toolchain: '1.96'
components: rustfmt, clippy components: rustfmt, clippy
cache: false
# Tags can only restore caches saved on main, so only main saves one. The
# release workflow restores this key for its quality gates.
- uses: swatinem/rust-cache@v2 - uses: swatinem/rust-cache@v2
with:
shared-key: debug
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Formatting - name: Formatting
run: cargo fmt --all -- --check run: cargo fmt --all -- --check
@@ -57,3 +63,84 @@ jobs:
# Real Proton credentials and /dev/fuse deliberately do not run on pull # Real Proton credentials and /dev/fuse deliberately do not run on pull
# requests. Follow docs/TESTING.md on a dedicated test-account runner. # requests. Follow docs/TESTING.md on a dedicated test-account runner.
# The seeded simulation runs (docs/TESTING.md): real FUSE
# daemons against an in-memory Drive, so they need /dev/fuse but no account.
simulation:
if: github.event_name != 'pull_request'
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- uses: actions/checkout@v7
- name: Install native dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
pkg-config \
fuse3 \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
- name: Install minimum supported Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
cache: false
- uses: swatinem/rust-cache@v2
with:
shared-key: debug
save-if: false
- name: Simulation runs
run: 'cargo test -p pdfs-fuse --lib --locked sim:: -- --ignored --test-threads=1'
# A hung step writes every daemon thread's stack next to the run's
# state; without them a hang that does not replay cannot be told apart.
- name: Keep the stacks of a hung run
if: failure()
uses: actions/upload-artifact@v7
with:
name: simulation-stacks
path: /tmp/pdfs-sim-*/stacks.txt
if-no-files-found: ignore
# Keeps the release-profile dependencies cached on main, where the release
# workflow can restore them; a tag cannot reuse the caches of earlier tags.
release-cache:
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
- name: Install native dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
pkg-config \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
- name: Install minimum supported Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
cache: false
- uses: swatinem/rust-cache@v2
with:
shared-key: release
- name: Release build
run: cargo build --release --locked
+73 -22
View File
@@ -10,8 +10,60 @@ permissions:
contents: write contents: write
jobs: jobs:
checks:
name: Release quality gates
runs-on: ubuntu-24.04
steps:
- name: Checkout repository
uses: actions/checkout@v7
- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
pkg-config \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
- name: Set up Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
components: rustfmt, clippy
cache: false
# A tag can read the caches of main but its own caches are useless to the next
# tag, so this only restores what CI saved on main.
- name: Cache Cargo target
uses: swatinem/rust-cache@v2
with:
shared-key: debug
save-if: false
- name: Verify tag and manifest versions
if: startsWith(github.ref, 'refs/tags/')
shell: bash
run: |
tag_version="${GITHUB_REF_NAME#v}"
manifest_version="$(sed -n 's/^version = "\([^"]*\)"/\1/p' Cargo.toml | head -n1)"
pkgbuild_version="$(sed -n 's/^pkgver=//p' packaging/PKGBUILD | head -n1)"
test "$tag_version" = "$manifest_version"
test "$tag_version" = "$pkgbuild_version"
- name: Release quality gates
run: |
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only
build-release: build-release:
name: Build and Release name: Build packages
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
environment: 'rpm signing' environment: 'rpm signing'
@@ -38,27 +90,14 @@ jobs:
uses: actions-rust-lang/setup-rust-toolchain@v1 uses: actions-rust-lang/setup-rust-toolchain@v1
with: with:
toolchain: '1.96' toolchain: '1.96'
components: rustfmt, clippy cache: false
# Restores the release-profile dependencies that CI builds on main.
- name: Cache Cargo target - name: Cache Cargo target
uses: swatinem/rust-cache@v2 uses: swatinem/rust-cache@v2
with:
- name: Verify tag and manifest versions shared-key: release
if: startsWith(github.ref, 'refs/tags/') save-if: false
shell: bash
run: |
tag_version="${GITHUB_REF_NAME#v}"
manifest_version="$(sed -n 's/^version = "\([^"]*\)"/\1/p' Cargo.toml | head -n1)"
pkgbuild_version="$(sed -n 's/^pkgver=//p' packaging/PKGBUILD | head -n1)"
test "$tag_version" = "$manifest_version"
test "$tag_version" = "$pkgbuild_version"
- name: Release quality gates
run: |
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only
- name: Build release binaries - name: Build release binaries
run: | run: |
@@ -127,7 +166,7 @@ jobs:
Maintainer: Nils Pukropp <contact@narl.io> Maintainer: Nils Pukropp <contact@narl.io>
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4, libimage-exiftool-perl Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4, libimage-exiftool-perl
Installed-Size: ${INSTALLED_SIZE} Installed-Size: ${INSTALLED_SIZE}
Homepage: https://github.com/narrrl/proton-drive-linux Homepage: https://proton-drive.narl.io
Description: Unofficial Proton Drive client for Linux Description: Unofficial Proton Drive client for Linux
Files-on-demand Proton Drive for the Linux desktop, unaffiliated with Proton AG: Files-on-demand Proton Drive for the Linux desktop, unaffiliated with Proton AG:
- FUSE virtual mount with block-level caching - FUSE virtual mount with block-level caching
@@ -152,6 +191,7 @@ jobs:
--define "_topdir $PWD/rpmbuild" \ --define "_topdir $PWD/rpmbuild" \
--define "_sourcedir $PWD" \ --define "_sourcedir $PWD" \
--define "git_dir $PWD" \ --define "git_dir $PWD" \
--define "prebuilt 1" \
--define "version ${RPM_VERSION}" \ --define "version ${RPM_VERSION}" \
--define "_dbpath $PWD/rpmbuild/db" \ --define "_dbpath $PWD/rpmbuild/db" \
packaging/proton-drive-linux.spec packaging/proton-drive-linux.spec
@@ -190,8 +230,19 @@ jobs:
*.rpm *.rpm
if-no-files-found: error if-no-files-found: error
publish:
name: Create GitHub Release
needs: [checks, build-release]
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-24.04
steps:
- name: Download packages
uses: actions/download-artifact@v8
with:
name: proton-drive-linux-builds
- name: Create GitHub Release - name: Create GitHub Release
if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v3 uses: softprops/action-gh-release@v3
with: with:
files: | files: |
@@ -205,7 +256,7 @@ jobs:
aur: aur:
name: Publish AUR packages name: Publish AUR packages
needs: build-release needs: publish
if: startsWith(github.ref, 'refs/tags/') if: startsWith(github.ref, 'refs/tags/')
uses: ./.github/workflows/aur.yml uses: ./.github/workflows/aur.yml
with: with:
+121
View File
@@ -0,0 +1,121 @@
name: Test build
# On-demand Fedora RPM of any branch, for someone to try a fix before it ships.
# Run it from the Actions tab ("Run workflow", pick the branch). Built inside a
# Fedora container so the binaries link against Fedora's own GTK, libadwaita and
# WebKitGTK, not the Ubuntu ones the release RPM is built against.
#
# The RPM is unsigned and versioned `<Cargo version>-1.test.<sha>`, so it sorts
# below the release it precedes and `dnf` replaces it when that ships.
on:
workflow_dispatch:
inputs:
fedora:
description: 'Fedora release to build on'
required: false
default: '44'
type: string
prerelease:
description: 'Also publish a pre-release anyone can download without a GitHub login'
required: false
default: false
type: boolean
permissions:
contents: read
concurrency:
group: test-build-${{ github.ref }}
cancel-in-progress: true
jobs:
rpm:
name: Fedora ${{ inputs.fedora }} RPM
runs-on: ubuntu-24.04
container: fedora:${{ inputs.fedora }}
steps:
- name: Install build dependencies
run: |
dnf install -y --setopt=install_weak_deps=False \
git rpm-build gcc gcc-c++ make gettext curl \
pkgconf-pkg-config fuse3-devel gtk4-devel libadwaita-devel \
libsecret-devel dbus-devel glib2-devel webkitgtk6.0-devel
- name: Checkout repository
uses: actions/checkout@v7
# Fedora's packaged Rust can trail the workspace's minimum (1.96).
- name: Set up Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
cache: false
- name: Cache Cargo target
uses: swatinem/rust-cache@v2
with:
shared-key: fedora-test-build
save-if: false
- name: Build release binaries
run: cargo build --release --locked
- name: Package RPM
run: |
git config --global --add safe.directory "$GITHUB_WORKSPACE"
VERSION=$(grep -m1 '^version' Cargo.toml | cut -d'"' -f2)
SHORT=${GITHUB_SHA::7}
mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS,BUILDROOT,db}
rpmbuild -bb \
--nodeps \
--define "_topdir $PWD/rpmbuild" \
--define "_sourcedir $PWD" \
--define "git_dir $PWD" \
--define "prebuilt 1" \
--define "version ${VERSION}" \
--define "dist .test.${SHORT}" \
--define "_dbpath $PWD/rpmbuild/db" \
packaging/proton-drive-linux.spec
mkdir -p out
cp rpmbuild/RPMS/x86_64/*.rpm out/
# The bare binaries, for trying one without installing anything.
tar -czf "out/proton-drive-linux-${VERSION}-${SHORT}-fedora${{ inputs.fedora }}-x86_64.tar.gz" \
-C target/release pdfs pdfs-tray pdfs-app pdfs-prompt
ls -l out
- name: Upload artifact
uses: actions/upload-artifact@v7
with:
name: proton-drive-linux-fedora${{ inputs.fedora }}-${{ github.sha }}
path: out/*
retention-days: 14
if-no-files-found: error
prerelease:
name: Publish pre-release
needs: rpm
if: inputs.prerelease
runs-on: ubuntu-24.04
permissions:
contents: write
steps:
- name: Download build
uses: actions/download-artifact@v8
with:
path: out
merge-multiple: true
# One rolling pre-release per branch: a new run replaces the last one, so
# the link a tester was given keeps pointing at the newest build.
- name: Publish
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
run: |
TAG="test-$(echo "${GITHUB_REF_NAME}" | tr -c 'A-Za-z0-9._\n' '-')"
gh release delete "$TAG" --cleanup-tag --yes 2>/dev/null || true
gh release create "$TAG" out/* \
--target "$GITHUB_SHA" \
--prerelease \
--title "Test build of ${GITHUB_REF_NAME} (${GITHUB_SHA::7})" \
--notes "Unsigned test build of branch \`${GITHUB_REF_NAME}\` at ${GITHUB_SHA}. Not a release; replaced by the next test build of this branch."
+17 -8
View File
@@ -6,9 +6,10 @@ deliberately high — everything else is ordinary Rust review.
## Before you start ## Before you start
- Read [`AGENTS.md`](AGENTS.md) for repository conventions and - Read [`AGENTS.md`](AGENTS.md) for repository conventions,
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for how the daemon, front ends, and SDK fit [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md) for building, running and debugging from source,
together. and [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for how the daemon, front ends, and SDK fit
together. [`docs/README.md`](docs/README.md) indexes the rest.
- Check [`docs/BUGS.md`](docs/BUGS.md) — it is the authoritative issue ledger, and your problem may - Check [`docs/BUGS.md`](docs/BUGS.md) — it is the authoritative issue ledger, and your problem may
already be tracked there with context. already be tracked there with context.
- For anything larger than a fix, open an issue first so the design can be discussed before you - For anything larger than a fix, open an issue first so the design can be discussed before you
@@ -23,8 +24,9 @@ cargo build --workspace
``` ```
GUI and FUSE targets need the GTK4, libadwaita, WebKitGTK 6, libsecret, D-Bus, and FUSE3 GUI and FUSE targets need the GTK4, libadwaita, WebKitGTK 6, libsecret, D-Bus, and FUSE3
development packages listed in the [README](README.md#prerequisites). Minimum supported Rust development packages listed in [`docs/INSTALL.md`](docs/INSTALL.md#build-dependencies). Minimum
version is 1.96. supported Rust version is 1.96. [`docs/DEVELOPMENT.md`](docs/DEVELOPMENT.md#running-from-source)
explains how to run your build next to, or instead of, the installed service.
## Quality gates ## Quality gates
@@ -37,9 +39,9 @@ cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only scripts/fuse-acceptance.sh --offline-only
``` ```
Changes to the FUSE layer, sync, or the pending-operation queue should also be exercised against a Changes to the FUSE layer, sync, or the queue should also pass the simulation runs and be
real mount before review — see [`docs/TESTING.md`](docs/TESTING.md). Live runs are destructive: exercised against a real mount before review — see [`docs/TESTING.md`](docs/TESTING.md). Live
use a dedicated test account, never one holding data you cannot lose. runs are destructive: use a dedicated test account, never one holding data you cannot lose.
## Making changes ## Making changes
@@ -51,6 +53,13 @@ use a dedicated test account, never one holding data you cannot lose.
control-socket work stays off GTK's main thread. control-socket work stays off GTK's main thread.
- **`staging/` and `recovery/` may hold the only copy of user data.** Nothing may clear or purge - **`staging/` and `recovery/` may hold the only copy of user data.** Nothing may clear or purge
them opportunistically. them opportunistically.
- **The GUI is fully translated.** Wrap every user-visible string in the helpers from
`crates/pdfs-gui/src/i18n.rs`, run `po/update.sh`, and translate the new entries in every
catalog in `po/LINGUAS` in the same change. See [`docs/TRANSLATING.md`](docs/TRANSLATING.md).
The CLI and daemon messages stay in English.
- **Document what users see.** A new command, setting or behaviour belongs in the matching page
under `docs/` (user guide, CLI reference or configuration) and in the `[Unreleased]` section of
[`docs/CHANGELOG.md`](docs/CHANGELOG.md).
## Commits and pull requests ## Commits and pull requests
Generated
+30 -24
View File
@@ -166,7 +166,7 @@ version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
dependencies = [ dependencies = [
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -177,7 +177,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
dependencies = [ dependencies = [
"anstyle", "anstyle",
"once_cell_polyfill", "once_cell_polyfill",
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -1254,7 +1254,7 @@ dependencies = [
"libc", "libc",
"option-ext", "option-ext",
"redox_users", "redox_users",
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -1433,7 +1433,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [ dependencies = [
"libc", "libc",
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -1896,7 +1896,7 @@ dependencies = [
"gobject-sys", "gobject-sys",
"libc", "libc",
"system-deps 9.0.0", "system-deps 9.0.0",
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -3100,7 +3100,7 @@ version = "0.50.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
dependencies = [ dependencies = [
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -3474,7 +3474,7 @@ checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4"
[[package]] [[package]]
name = "pdfs-cli" name = "pdfs-cli"
version = "2.5.0" version = "3.1.2"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"clap", "clap",
@@ -3485,13 +3485,15 @@ dependencies = [
"tokio", "tokio",
"tracing", "tracing",
"tracing-subscriber", "tracing-subscriber",
"zeroize",
] ]
[[package]] [[package]]
name = "pdfs-core" name = "pdfs-core"
version = "2.5.0" version = "3.1.2"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"base64 0.23.1",
"dbus-secret-service-keyring-store", "dbus-secret-service-keyring-store",
"directories", "directories",
"ignore", "ignore",
@@ -3508,13 +3510,15 @@ dependencies = [
"thiserror 2.0.20", "thiserror 2.0.20",
"tokio", "tokio",
"tracing", "tracing",
"zeroize",
"zip", "zip",
] ]
[[package]] [[package]]
name = "pdfs-fuse" name = "pdfs-fuse"
version = "2.5.0" version = "3.1.2"
dependencies = [ dependencies = [
"async-trait",
"base64 0.23.1", "base64 0.23.1",
"fuser", "fuser",
"futures", "futures",
@@ -3528,12 +3532,13 @@ dependencies = [
"sha1 0.11.0", "sha1 0.11.0",
"tokio", "tokio",
"tracing", "tracing",
"tracing-subscriber",
"zip", "zip",
] ]
[[package]] [[package]]
name = "pdfs-gui" name = "pdfs-gui"
version = "2.5.0" version = "3.1.2"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"async-channel", "async-channel",
@@ -3570,9 +3575,9 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]] [[package]]
name = "pgp" name = "pgp"
version = "0.20.0" version = "0.21.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1cfa4743b28656065ff4c0ba09e46b357a65e8c00fc2341e89084b82f87cbdf1" checksum = "ae70f4d9325a391db30d115d6191a7bb67cec856aa0bba131290f0c4e09a532a"
dependencies = [ dependencies = [
"aead 0.5.2", "aead 0.5.2",
"aes 0.8.4", "aes 0.8.4",
@@ -3773,9 +3778,9 @@ dependencies = [
[[package]] [[package]]
name = "proton-drive-rs" name = "proton-drive-rs"
version = "0.7.0" version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0f6996724e51e99f03d2843828de9477afc39baa764403cb69490cabc1c4d93" checksum = "014165ed0f2d3847a4b8956ca94a82ba16c82f2d5c7fa139c3541c15150dba21"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"base64 0.23.1", "base64 0.23.1",
@@ -3800,9 +3805,9 @@ dependencies = [
[[package]] [[package]]
name = "proton-sdk" name = "proton-sdk"
version = "0.7.0" version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91cc25654133bd5714407a1df0996eaf214a6e8e0bb95d8139bc42e7cee31c34" checksum = "9e16bf5592b7947708dd754defeaee7a58e9fa35988a6ea345b5b9a93765fd81"
dependencies = [ dependencies = [
"aes-gcm 0.11.1", "aes-gcm 0.11.1",
"async-trait", "async-trait",
@@ -3829,6 +3834,7 @@ dependencies = [
"thiserror 2.0.20", "thiserror 2.0.20",
"tokio", "tokio",
"tracing", "tracing",
"zeroize",
] ]
[[package]] [[package]]
@@ -3920,7 +3926,7 @@ dependencies = [
"once_cell", "once_cell",
"socket2", "socket2",
"tracing", "tracing",
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -4368,7 +4374,7 @@ dependencies = [
"errno", "errno",
"libc", "libc",
"linux-raw-sys", "linux-raw-sys",
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -4425,7 +4431,7 @@ dependencies = [
"security-framework", "security-framework",
"security-framework-sys", "security-framework-sys",
"webpki-root-certs", "webpki-root-certs",
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -4758,7 +4764,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4"
dependencies = [ dependencies = [
"libc", "libc",
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -4926,10 +4932,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [ dependencies = [
"fastrand", "fastrand",
"getrandom 0.4.3", "getrandom 0.3.4",
"once_cell", "once_cell",
"rustix", "rustix",
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -5260,7 +5266,7 @@ checksum = "f2f6fb2847f6742cd76af783a2a2c49e9375d0a111c7bef6f71cd9e738c72d6e"
dependencies = [ dependencies = [
"memoffset", "memoffset",
"tempfile", "tempfile",
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
@@ -5584,7 +5590,7 @@ version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [ dependencies = [
"windows-sys 0.61.2", "windows-sys 0.60.2",
] ]
[[package]] [[package]]
+5 -4
View File
@@ -3,20 +3,20 @@ resolver = "2"
members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"] members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"]
[workspace.package] [workspace.package]
version = "2.5.0" version = "3.1.2"
edition = "2024" edition = "2024"
license = "MIT" license = "MIT"
authors = ["Nils Pukropp <contact@narl.io>"] authors = ["Nils Pukropp <contact@narl.io>"]
rust-version = "1.96" rust-version = "1.96"
repository = "https://github.com/narrrl/proton-drive-linux" repository = "https://github.com/narrrl/proton-drive-linux"
homepage = "https://github.com/narrrl/proton-drive-linux" homepage = "https://proton-drive.narl.io"
readme = "README.md" readme = "README.md"
keywords = ["proton", "drive", "fuse", "sync", "linux"] keywords = ["proton", "drive", "fuse", "sync", "linux"]
categories = ["filesystem", "command-line-utilities"] categories = ["filesystem", "command-line-utilities"]
[workspace.dependencies] [workspace.dependencies]
proton-sdk = { version = "0.7.0" } proton-sdk = { version = "0.8.1" }
proton-drive-rs = { version = "0.7.0" } proton-drive-rs = { version = "0.8.1" }
# Internal crates. # Internal crates.
pdfs-core = { path = "crates/pdfs-core" } pdfs-core = { path = "crates/pdfs-core" }
@@ -26,6 +26,7 @@ tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "fs",
serde = { version = "1", features = ["derive"] } serde = { version = "1", features = ["derive"] }
serde_json = "1" serde_json = "1"
base64 = "0.23" base64 = "0.23"
zeroize = "1"
thiserror = "2" thiserror = "2"
anyhow = "1" anyhow = "1"
# Non-poisoning mutexes. A handler that panics while holding a lock must not # Non-poisoning mutexes. A handler that panics while holding a lock must not
+110 -449
View File
@@ -4,513 +4,174 @@
# Proton Drive for Linux # Proton Drive for Linux
**Files-on-demand Proton Drive for the Linux desktop** — a FUSE virtual mount with block-level **Proton Drive on the Linux desktop.** Your files as a folder that downloads on demand, synced
caching, a scriptable CLI, and a native GTK4 app with tray and search launcher. local folders, Photos, sharing, and a native GTK4 app with tray and search launcher, plus a
scriptable CLI.
**[proton-drive.narl.io](https://proton-drive.narl.io)**
[![CI](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml/badge.svg)](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml) [![CI](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml/badge.svg)](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/v/release/narrrl/proton-drive-linux?sort=semver)](https://github.com/narrrl/proton-drive-linux/releases/latest) [![Release](https://img.shields.io/github/v/release/narrrl/proton-drive-linux?sort=semver)](https://github.com/narrrl/proton-drive-linux/releases/latest)
[![AUR](https://img.shields.io/aur/version/proton-drive-for-linux)](https://aur.archlinux.org/packages/proton-drive-for-linux)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust 1.96+](https://img.shields.io/badge/rust-1.96%2B-orange.svg)](https://www.rust-lang.org) [![Rust 1.96+](https://img.shields.io/badge/rust-1.96%2B-orange.svg)](https://www.rust-lang.org)
[![Platform: Linux](https://img.shields.io/badge/platform-linux-lightgrey.svg)](#prerequisites) [![Buy me a coffee](https://img.shields.io/badge/Buy%20me%20a%20coffee-FFDD00?logo=buymeacoffee&logoColor=black)](https://buymeacoffee.com/narl)
<img src="images/files.png" alt="The GTK4 file browser" width="820"> <img src="images/files.png" alt="My files in the Proton Drive app" width="820">
</div> </div>
> [!IMPORTANT] > [!IMPORTANT]
> This is an **unofficial**, community-built client. It is not affiliated with, endorsed by, or > This is an **unofficial**, community-built client. It is not affiliated with, endorsed by, or
> supported by Proton AG. "Proton" and "Proton Drive" are trademarks of Proton AG, used here only > supported by Proton AG. "Proton" and "Proton Drive" are trademarks of Proton AG, used here only
> to describe what the software talks to. Use a test account first, and read > to describe what the software talks to. Try it with data you have another copy of first, and
> [`docs/RECOVERY.md`](docs/RECOVERY.md) before trusting it with data you cannot lose. > read [what survives a lost machine](docs/RECOVERY.md) before you rely on it.
## Install
```bash
# Debian / Ubuntu
sudo apt install ./proton-drive-linux_*.deb
# Fedora
sudo dnf install ./proton-drive-linux-*.rpm
# Arch Linux (AUR)
paru -S proton-drive-for-linux # or -git / -bin
```
Packages for each release are attached to the [latest
release](https://github.com/narrrl/proton-drive-linux/releases/latest); see
[Installation & Packages](#installation--packages) for details and
[Building from Source](#building-from-source) to compile it yourself.
```bash
pdfs login # sign in (SRP + 2FA, credentials go to the system keyring)
pdfs status # mount state, queued writes, cache usage
pdfs-app # the desktop app
```
## Contents
- [Features](#features) · [Quick search](#quick-search-and-media-opening) · [Opening files](#choosing-how-files-open)
- [Selective sync](#selective-sync-pdfsignore) · [Diagnostics](#diagnostics--maintenance) · [Scripting](#scripting---json)
- [Performance](#performance--caching) · [Filesystem safety](#filesystem-safety) · [Screenshots](#screenshots)
- [Prerequisites](#prerequisites) · [Building](#building-from-source) · [Packages](#installation--packages) · [Releases](#automated-releases-cicd)
**Documentation:** [Architecture](docs/ARCHITECTURE.md) · [Testing](docs/TESTING.md) ·
[Recovery](docs/RECOVERY.md) · [Development](docs/DEVELOPMENT.md) · [Translating](docs/TRANSLATING.md) ·
[Known issues](docs/BUGS.md) · [Changelog](docs/CHANGELOG.md) ·
[Contributing](CONTRIBUTING.md) · [Security policy](SECURITY.md)
## Features ## Features
- **Files-on-Demand (FUSE)**: Mount your Proton Drive as a virtual filesystem where files are downloaded only when opened, utilizing smart block-level caching and disk-backed writes. **Files**
- **Command-Line Interface (CLI)**: Manage your drive, authenticate, and monitor sync status directly from the terminal.
- **Non-Blocking GTK4 Desktop App**: Browse files, manage pins, configure options, and monitor active transfers through a modern, native GUI with a fully non-blocking asynchronous main loop.
- **File Thumbnails**: Browse locally cached image previews in Files, Shared, and Trash, including embedded previews from camera RAW files. Build a whole folder tree from the Files toolbar and resize the grid from the status bar.
- **Shared Links & Invites**: Browse files shared with you by other users, and view/manage your own public shared links directly in the GUI.
- **Local Backup (Computers)**: Sync and back up local directories (like Downloads, Documents, Pictures, etc.) directly to your Proton Drive account.
- **Locations**: One page (and `pdfs locations`) listing every place Proton Drive occupies on this machine — the main mount plus each backed-up folder — with its mode, sync state, and whether it is read-only. Switching a folder between a full local copy and on-demand happens here.
- **System Tray Integration**: Background indicator for status monitoring, quick actions, and fast sync controls.
- **Unified Search Launcher (HUD)**: A resident Google Drive-style launcher (`pdfs-prompt`) that searches Proton Drive and local files together, ranks the best matches, tolerates abbreviations and typos, and is ideal for a system-wide hotkey. It can also present the same search in your own launcher (`--dmenu`) or, searching live as you type, in `fzf` (`--fzf`).
- **Secure Credential Storage**: Integrates with the system Secret Service (GNOME Keyring, KWallet, etc.) with smart in-memory credential caching to avoid UI thread blockages.
- **Proton Photos Support**: Access your Proton Photos timeline, view thumbnails, and download backed-up media natively (available in the GUI as a navigation tab and via the CLI).
- **File Version History**: Every revision Proton Drive still holds for a file, from the browser's details pane (**Versions**) or `pdfs versions list|restore|save|rm`. Restoring is server-side — no re-upload — and an old version can be written out to a local file without touching the live one.
- **Photo Favourites**: Star a photo in the lightbox, filter the gallery to favourites, or use `pdfs favorite <uid>` / `pdfs photos --favorites`.
- **Photo Albums**: Browse your albums — including the ones shared with you — from the Albums view of the Photos page, or with `pdfs albums` / `pdfs album <uid>`. Album contents open in the same gallery as the timeline.
- **Places**: The Places view of the Photos page groups photos by the town they were taken in, using the location stored with each photo, as cards or on a world map. Towns are looked up offline in [GeoNames](https://www.geonames.org/) data (CC BY 4.0), and the map is drawn offline from [Natural Earth](https://www.naturalearthdata.com/) land outlines and country borders (public domain). Each town is marked with one of its photos, and zoomed in, every photo sits where it was taken. A street map from [OpenFreeMap](https://openfreemap.org/) (© [OpenStreetMap](https://www.openstreetmap.org/copyright) contributors) can be turned on under Appearance; it is off by default because the tile server learns which parts of the world you look at. Install `iso-codes` for country names in your language.
- **Human Verification (CAPTCHA) Recovery**: Detects sign-in gates (VPN/new IP challenges) and launches an embedded `WebKitWebView` dialog to safely complete the challenge, transparently retrying authentication with the earned token.
- **Selective Sync (`.pdfsignore`)**: Keep build trees, dependency directories, and editor leftovers out of synced folders using gitignore-style rules.
- **Data-Safe Offline Writes**: Durable scratch/staging files and a transactional pending queue preserve acknowledged writes across network failures and restarts.
## Quick Search and Media Opening - **Your Drive as a folder.** `~/ProtonDrive` works with every program. Files download when they
are opened, a block at a time, so a video starts playing and a large archive can be read
without fetching all of it.
- **Available offline.** Keep chosen files and folders fully downloaded; everything else is
cached within a size limit you set.
- **Synced folders.** Back up folders such as `~/Documents` under your computer in Proton Drive,
as a full two-way copy or online only. Gitignore-style `.pdfsignore` rules, conflict copies
instead of overwrites, pause and bandwidth limits.
- **Works at local speed.** Making, renaming and deleting files and folders returns at once, online
or offline. Changes are staged on disk and queued before they are acknowledged, so they survive
a lost connection, a crash or a reboot and reach Drive when they can.
`pdfs-prompt` searches Drive metadata and the local home-directory index in one request. Results are ranked together rather than split by source. Matching is case-insensitive and supports exact names, prefixes, parent paths, multiple terms, punctuation-separated words, ordered abbreviations, common typos, and adjacent transpositions. **Desktop app**
The prompt stays resident after its first launch, so invoking the shortcut again reuses and resets the existing window. Drive folders open from the mount. Audio and video results also open through FUSE, allowing media players to seek and request ranges without downloading the entire file first; ordinary Drive files are materialized into the local cache before opening. - **Browse** My files, Shared with me, Shared by me, Computers and Trash, with thumbnails
(including camera RAW), drag and drop, details and **version history**.
- **Photos.** Timeline with favorites, albums, and Places on a map; RAW+JPEG pairs shown as one
photo; duplicate finder; slideshow; **import from Google Photos** (Takeout).
- **Sharing.** Invite people as viewer, editor or admin, create public links with password and
expiry, accept invitations.
- **Sync overview.** What is uploading, what is waiting, what failed and why, and a history of
what happened. A change Drive refuses (storage full, access taken away) shows at once, on the
file too, and can be exported or discarded.
- **Tray icon** with status, pause and quick actions, and a **search launcher** (`pdfs-prompt`)
that searches Proton Drive and your home folder together, as its own window, through fuzzel,
rofi or wofi, or live in `fzf`.
- **Restore a computer.** Continue a backup on a new machine and bring its folders back in one
step.
- **Sign-in** with two-factor authentication and CAPTCHA; the session is kept in the system
keyring.
- Available in English, German, Spanish, French, Italian, Dutch, Polish and Brazilian
Portuguese.
### Using your own launcher (fuzzel, rofi, wofi, …) **Command line**
`pdfs-prompt --dmenu` runs the same search through an external dmenu-style - About 55 `pdfs` commands cover files, sync, sharing, Photos, Trash, versions and diagnostics.
launcher instead of the built-in window, so the HUD matches the rest of a Query commands print JSON with `--json`.
tiling-WM desktop:
```bash ## Install
pdfs-prompt --dmenu # fuzzel/rofi/wofi/tofi/bemenu/dmenu, whichever is installed
pdfs-prompt --dmenu --menu 'fuzzel --dmenu --width 60'
pdfs-prompt --dmenu --query invoice # skip straight to a search
```
A launcher filters a fixed list — it cannot ask for a new one per keystroke — so | Distribution | Package |
searching takes two steps, and the prompt says which step you are on:
- `Search Drive ›` lists your pinned files. Type anything that isn't one of them
and press Enter to search for it.
- `Drive: invoice ›` lists the results. Enter opens one; typing something else
and pressing Enter searches again. Escape closes.
`--query` skips the first step. fuzzel and rofi also get file-type icons.
To make an existing keybinding use it without re-binding, and to pin the
launcher command, set them in `config.json`:
```json
{
"prompt": {
"mode": "dmenu",
"menu": ["fuzzel", "--dmenu", "--width", "60"],
"menu_limit": 50
}
}
```
`--gtk` overrides `"mode": "dmenu"` for one invocation. A `{prompt}` token
anywhere in `menu` is replaced by the prompt text; without one, the launcher's
own prompt flag is appended.
### Live launcher search (fzf)
The two-step above exists because no dmenu-style launcher can call back for new
results per keystroke. `fzf` can, so `pdfs-prompt --fzf` searches as you type,
exactly like the built-in window:
```bash
pdfs-prompt --fzf # pinned files, then live results as you type
pdfs-prompt --fzf --query invoice # start on a search
```
Every keystroke re-queries the daemon (debounced, like the GTK prompt) and fzf's
own matcher is switched off, so the ordering you see is the daemon's ranking —
abbreviations and typos included, which a launcher filtering a fixed list cannot
do. Enter opens; Escape closes.
fzf is a terminal program, so when it is started without one — from a WM
keybinding — it spawns a terminal for itself: foot, ghostty, kitty, alacritty,
wezterm or xterm, whichever is installed. Each is launched with the app-id/class
`pdfs-prompt`, so one window rule floats it whichever terminal you have. In
Hyprland:
```
windowrulev2 = float, class:^(pdfs-prompt)$
windowrulev2 = size 900 500, class:^(pdfs-prompt)$
windowrulev2 = center, class:^(pdfs-prompt)$
bind = SUPER, space, exec, pdfs-prompt --fzf
```
Set it as the default front end, and pick the terminal explicitly, in
`config.json`:
```json
{
"prompt": {
"mode": "fzf",
"terminal": ["foot", "--app-id=pdfs-prompt", "--"],
"menu_limit": 50
}
}
```
A `{cmd}` token anywhere in `terminal` is replaced by the command to run;
without one it is appended, which is what every detected terminal expects.
## Choosing How Files Open
By default every result is handed to `xdg-open`. An `open_with` block in
`config.json` overrides that per file type — for instance opening text in
Neovim inside Alacritty, while everything else still goes to the desktop:
```json
{
"open_with": {
"terminal": ["alacritty", "-e"],
"rules": [
{ "match": ["@text"], "command": ["nvim"], "terminal": true },
{ "match": ["*.png", "*.jpg"], "command": ["imv"] }
]
}
}
```
- `match` takes file-name globs (`*.md`, `notes-*.txt`) or the classes `@dir`,
`@text`, `@document`, `@image`, `@media`, `@any`. First matching rule wins.
- `command` is argv, not a shell line. A `{}` token is replaced by the path;
without one the path is appended. `$VAR` tokens expand from the environment,
so `["$EDITOR"]` follows your editor.
- `"terminal": true` wraps the command in `terminal`. When that is unset, the
terminal comes from `$PDFS_TERMINAL`, then `$TERMINAL`, then the first known
emulator on `PATH`; a bare name like `TERMINAL=alacritty` gains the right
"run this" flag automatically.
- `"default": [...]` replaces `xdg-open` for everything unmatched.
The rules apply to the prompt (both front ends) and to the GTK browser, so a
file opens the same way wherever it was picked. Patterns are matched against the
Drive name — a downloaded file is stored in the cache under its content hash,
which says nothing about its type, so matching that would send everything to
`xdg-open`.
## Selective Sync (`.pdfsignore`)
Two-way synced folders skip paths matched by ignore rules, so syncing a project
directory does not upload `node_modules/`, `target/`, or `.git/`.
Rules come from two places, and both apply:
1. **Per folder** — a `.pdfsignore` file at the root of the synced folder
(`.protonignore` also works). Gitignore syntax, including negation:
```gitignore
# everything build-related
build/
*.log
# ...except this one
!important.log
```
2. **Globally** — an `ignore_patterns` list in `config.json`, applied to every
synced folder. When unset, sensible defaults apply: `.git/`, `.hg/`, `.svn/`,
`node_modules/`, `target/`, `.venv/`, `__pycache__/`, `*~`, `*.swp`, `*.tmp`,
`.DS_Store`, and `Thumbs.db`.
```json
{
"ignore_patterns": ["node_modules/", "target/", "*.iso"]
}
```
Set it to `[]` to opt out of the defaults entirely.
Rules are re-read at the start of every sync pass, so edits take effect on the
next pass without restarting the daemon.
**Ignoring is never destructive.** If a rule starts matching a file that was
already synced, its copy on Drive is left untouched — the file simply stops
being tracked. Removing the rule later picks the existing remote file back up
rather than re-uploading it.
## Diagnostics & Maintenance
When something looks wrong, `pdfs diagnose` checks the installation and prints a
report. It runs without a daemon on purpose — the state worth diagnosing is
usually the state where the daemon will not start:
```console
$ pdfs diagnose
Paths
[ok ] state dir: /home/you/.local/state/proton-drive-linux
[ok ] database: /home/you/.local/state/proton-drive-linux/cache.db (170.3 MiB)
Account
[ok ] keyring session: you@proton.me
Daemon
[ok ] daemon responding
[ok ] mounted at: /home/you/ProtonDrive
[ok ] queued writes: none
No problems found.
```
It exits non-zero if any check fails, so it works in a health-check script.
For the local metadata database and content cache:
| Command | What it does |
|---|---| |---|---|
| `pdfs cache inspect` | Database size, reclaimable space, per-table row counts, cache usage against budget | | Debian, Ubuntu 24.04 or newer | `.deb` from the [latest release](https://github.com/narrrl/proton-drive-linux/releases/latest) |
| `pdfs cache inspect --deep` | Also runs SQLite's integrity check — reads every page, so it is slow on a large database | | Fedora 44 or newer | `.rpm` from the [latest release](https://github.com/narrrl/proton-drive-linux/releases/latest) |
| `pdfs cache vacuum` | Checkpoints the write-ahead log and compacts the database | | Arch Linux | [`proton-drive-for-linux`](https://aur.archlinux.org/packages/proton-drive-for-linux) (also `-bin` and `-git`) in the AUR |
| `pdfs cache clear` | Deletes cached file content, keeping pinned files | | Anything else | [Build from source](docs/INSTALL.md#build-from-source) |
`vacuum` takes a write lock for its duration and needs room for a second copy of ```bash
the database while it runs, so it is a deliberate operation rather than sudo apt install ./proton-drive-linux_*.deb # Debian, Ubuntu
something the daemon does on a timer. sudo dnf install ./proton-drive-linux-*.rpm # Fedora
paru -S proton-drive-for-linux # Arch
Before a release, run the real-kernel acceptance suite against a dedicated test
account and mount. See [`docs/TESTING.md`](docs/TESTING.md); it includes optional
two-mount convergence checks and confines destructive operations to a fresh
test directory.
## Scripting (`--json`)
Query commands accept a global `--json` flag and emit machine-readable output:
```console
$ pdfs --json sync list | jq '.items[] | select(.state != "idle") | .local_path'
$ pdfs --json cache inspect | jq '.db_reclaimable_bytes'
$ pdfs --json status | jq -r '.mount.mountpoint'
``` ```
Supported on `status`, `ls`, `pins`, `sync list`, `devices list`, `transfers`, Then open **Proton Drive** from your applications menu and sign in. The app starts the background
`activity`, and `cache inspect`. Commands that perform an action keep their service, and your files appear in `~/ProtonDrive`.
human output — a script that needs to know whether one worked has the exit code.
Two things worth relying on: Signing in from a terminal instead? `pdfs login` stores the session but does not start the
service. Enable it once:
- **The payload is unwrapped.** Output is `{"items": […]}`, not the daemon's ```bash
internal `{"SyncFolders": {"items": […]}}`, so scripts never name an internal pdfs login
variant. systemctl --user enable --now proton-drive.service
- **Errors still fail.** A daemon-side error prints its JSON body (with a pdfs status
machine-readable `kind`) on stdout *and* exits non-zero, so `set -e` and ```
`if pdfs …` behave as expected rather than treating a failed lookup as
success.
## Performance & Caching [docs/INSTALL.md](docs/INSTALL.md) has the requirements, optional extras (tray on GNOME, RAW and
video thumbnails), headless use and uninstalling.
The client includes several optimizations designed for high efficiency, a low memory footprint, and a responsive user experience: ## Documentation
- **On-Demand Block Cache**: Files are read in fixed 4 MiB blocks. For unpinned files, the client fetches and caches only the blocks containing the requested byte ranges. This enables fast sequential and sparse reads (e.g. media streaming or metadata scanning) without downloading entire files. | | |
- **Disk-Backed Writes**: Large file writes are staged on disk in temporary scratch files (rather than fully buffered in RAM) and track modified byte intervals. Only the unedited remote segments are fetched at commit time, keeping memory usage minimal. |---|---|
- **Non-Blocking GTK4 Loop**: To prevent UI freezes, all synchronous D-Bus credential checks, control socket requests, and cache usage calculations are offloaded to background worker threads or fetched asynchronously. | [User guide](docs/USER_GUIDE.md) | Everything the app, tray and launcher do |
- **Flicker-Free UI Rendering**: The GTK4 application performs differential rendering of the pins list, only modifying list rows when the list content changes, preserving the user's scroll position and widget focus. | [Command-line reference](docs/CLI.md) | Every `pdfs` command and scripting with `--json` |
- **Durable Staging**: Scratch metadata and staged writes are synced and atomically published before an upload is acknowledged locally. Rapid revisions supersede pending work transactionally. | [Configuration](docs/CONFIGURATION.md) | `config.json`, file locations, environment variables, the service |
| [Troubleshooting](docs/TROUBLESHOOTING.md) | Diagnosing problems and fixing common ones |
## Filesystem Safety | [Recovering on a new computer](docs/RECOVERY.md) | Getting your synced folders back after a reinstall or a lost machine |
| [Changelog](docs/CHANGELOG.md) | What changed in each release |
Version 1.0 strengthens the boundaries around local-only data and destructive reconciliation:
- Truncate composes with queued revisions, preserving the authoritative prefix and correct zero-filled growth.
- Combined move-and-rename operations are queued durably and tolerate partially completed remote state.
- Session tombstones prevent successfully unlinked entries from reappearing through eventually consistent listings.
- Names are valid UTF-8 and no longer than Linux's 255-byte component limit.
- Incomplete mirror scans are non-destructive, and total-wipe protection covers every non-empty baseline.
- Failed conflict preservation, local deletion, or staging publication does not settle a new baseline or discard the only local copy.
- State, cache, and config directories must be real, current-user-owned `0700` directories; control sockets are `0600`.
For contributors: [development guide](docs/DEVELOPMENT.md), [architecture](docs/ARCHITECTURE.md),
[testing](docs/TESTING.md), [translating](docs/TRANSLATING.md), [roadmap](docs/ROADMAP.md) and the
[bug ledger](docs/BUGS.md). The [documentation index](docs/README.md) lists everything.
## Screenshots ## Screenshots
### GUI Application & Launcher
<table> <table>
<tr> <tr>
<td align="center" width="50%"><img src="images/login.png" alt="Login Screen" width="100%"><br><sub><b>Login Screen</b></sub></td> <td align="center" width="50%"><img src="images/files.png" alt="My files" width="100%"><br><sub><b>My files</b></sub></td>
<td align="center" width="50%"><img src="images/files.png" alt="Files Browser" width="100%"><br><sub><b>Files Browser</b></sub></td> <td align="center" width="50%"><img src="images/photos.png" alt="Photos timeline" width="100%"><br><sub><b>Photos</b></sub></td>
</tr> </tr>
<tr> <tr>
<td align="center" width="50%"><img src="images/shared.png" alt="Shared Links" width="100%"><br><sub><b>Shared Links</b></sub></td> <td align="center" width="50%"><img src="images/photos_map.png" alt="Photos places map" width="100%"><br><sub><b>Places</b></sub></td>
<td align="center" width="50%"><img src="images/shared_with_me.png" alt="Shared with me" width="100%"><br><sub><b>Shared with me</b></sub></td> <td align="center" width="50%"><img src="images/computers.png" alt="Computers" width="100%"><br><sub><b>Computers</b></sub></td>
</tr> </tr>
<tr> <tr>
<td align="center" width="50%"><img src="images/restore_devices.png" alt="Local Backups (Computers)" width="100%"><br><sub><b>Local Backups (Computers)</b></sub></td> <td align="center" width="50%"><img src="images/sync.png" alt="Sync folders" width="100%"><br><sub><b>Sync</b></sub></td>
<td align="center" width="50%"><img src="images/photos.png" alt="Photos Timeline" width="100%"><br><sub><b>Photos Timeline</b></sub></td> <td align="center" width="50%"><img src="images/conflicts.png" alt="Sync history showing resolved conflicts" width="100%"><br><sub><b>Sync history</b></sub></td>
</tr> </tr>
<tr> <tr>
<td align="center" width="50%"><img src="images/prompt_gtk.png" alt="Search Launcher Prompt (GTK)" width="100%"><br><sub><b>Search Launcher Prompt (GTK)</b></sub></td> <td align="center" width="50%"><img src="images/prompt_gtk.png" alt="Search launcher" width="100%"><br><sub><b>Search launcher</b></sub></td>
<td align="center" width="50%"><img src="images/prompt_fzf.png" alt="Search Launcher Prompt (fzf)" width="100%"><br><sub><b>Search Launcher Prompt (fzf)</b></sub></td> <td align="center" width="50%"><img src="images/prompt_fzf.png" alt="Search launcher in fzf" width="100%"><br><sub><b>Search launcher in fzf</b></sub></td>
</tr> </tr>
<tr> <tr>
<td align="center" width="50%"><img src="images/activity.png" alt="Activity" width="100%"><br><sub><b>Activity</b></sub></td> <td align="center" width="50%"><img src="images/login.png" alt="Sign in" width="100%"><br><sub><b>Sign in</b></sub></td>
<td align="center" width="50%"><img src="images/settings.png" alt="Settings" width="100%"><br><sub><b>Settings</b></sub></td> <td></td>
</tr> </tr>
</table> </table>
--- ## How it works
## Prerequisites A background service (`proton-drive.service`, the `pdfs daemon` command) holds your session,
mounts `~/ProtonDrive` through FUSE, runs the upload queue and the sync engine, and follows
Proton's event stream for changes made elsewhere. The app, the tray, the launcher and the CLI are
clients of that service over a local socket. Everything is end-to-end encrypted with your Proton
keys before it leaves the machine, through a Rust implementation of Proton's Drive SDK.
To compile the application from source or run the built binaries, ensure you have the following system libraries installed on your distribution: Files you open are decrypted into a local cache so programs can read them. What that means for
the security of the machine is spelled out in the
### Ubuntu / Debian (24.04+) [threat model](docs/ARCHITECTURE.md#10-threat-model-what-this-client-writes-to-disk-in-plaintext).
```bash
sudo apt-get update
sudo apt-get install -y \
pkg-config \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev \
libimage-exiftool-perl
```
### Arch Linux
```bash
sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus webkitgtk-6.0 perl-image-exiftool
```
### Fedora (44+)
```bash
sudo dnf install -y \
pkgconf-pkg-config fuse3-devel gtk4-devel libadwaita-devel \
webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel \
perl-Image-ExifTool cargo rust
```
Runtime extras (pick your desktop):
```bash
# GNOME — keyring + tray (AppIndicator)
sudo dnf install -y gnome-keyring gnome-shell-extension-appindicator xdg-utils
# KDE Plasma — KWallet (Secret Service); tray works via built-in SNI
sudo dnf install -y kwallet xdg-utils
```
---
## Building from Source
Ensure you have Rust and Cargo installed (minimum supported Rust version is 1.96).
1. Clone the repository and navigate into the project directory:
```bash
git clone https://github.com/narrrl/proton-drive-linux.git
cd proton-drive-linux
```
2. Build the workspace in release mode:
```bash
cargo build --release --locked
```
The compiled binaries will be available under `target/release/`:
- `pdfs`: The CLI utility.
- `pdfs-app`: The GTK4 application.
- `pdfs-tray`: The tray status notifier.
- `pdfs-prompt`: The launcher prompt for quick HUD search.
---
## Installation & Packages
### 1. Debian / Ubuntu (.deb)
Install the debian package via `dpkg` or `apt`:
```bash
sudo apt install ./proton-drive-linux_*.deb
```
### 2. Arch Linux (AUR)
Three AUR packages cover the usual choices; they conflict with each other, so
install one:
| Package | Contents |
| --- | --- |
| [`proton-drive-for-linux`](https://aur.archlinux.org/packages/proton-drive-for-linux) | builds the latest tagged release from source |
| [`proton-drive-for-linux-git`](https://aur.archlinux.org/packages/proton-drive-for-linux-git) | builds `main` from git |
| [`proton-drive-for-linux-bin`](https://aur.archlinux.org/packages/proton-drive-for-linux-bin) | installs the release binaries, no compile |
```bash
paru -S proton-drive-for-linux # or any AUR helper
```
The package sources live in `packaging/aur/`; see
[`packaging/aur/README.md`](packaging/aur/README.md) for how they are updated and
published. A local `PKGBUILD` that compiles the working tree in place is still
available for development builds:
```bash
cd packaging && makepkg -fi
```
### 3. Fedora (local RPM)
A local `.spec` is available under `packaging/`. From the repository root:
```bash
sudo dnf install -y rpm-build
rpmbuild -bb packaging/proton-drive-linux.spec \
--define "git_dir $PWD" \
--define "_rpmdir $PWD/packaging/out" \
--define "_builddir $PWD/packaging/build" \
--define "_sourcedir $PWD" \
--define "_specdir $PWD/packaging" \
--define "_srcrpmdir $PWD/packaging/out"
sudo dnf install packaging/out/x86_64/proton-drive-linux-*.rpm
```
---
## Automated Releases (CI/CD)
This project has a GitHub Actions CI workflow configured under `.github/workflows/release.yml`.
### How it works:
1. **Triggers**:
- Pushing a git tag matching `v*` (e.g. `git tag v0.1.0 && git push origin v0.1.0`).
- Manual runs via the **Actions** tab in GitHub (**workflow_dispatch**).
2. **Quality Gates and Build Process**:
- For tagged builds, verifies that the tag, workspace version, and `packaging/PKGBUILD` version agree.
- Runs `cargo fmt`, Clippy with warnings denied, the locked workspace test suite, and the account-free FUSE acceptance contract.
- Spawns an Ubuntu runner and installs GTK4, Libadwaita, FUSE3, and Secret Service packages.
- Sets up the Rust compiler and caches build targets to speed up runs.
- Compiles the workspace members in release mode.
3. **Artifact Packaging**:
- Generates a `.tar.gz` containing the raw binaries (`pdfs`, `pdfs-app`, `pdfs-tray`, `pdfs-prompt`).
- Packs them into Debian (`.deb`) and Fedora (`.rpm`) packages.
- Includes the systemd user service and tray autostart entry in the Debian package.
4. **Publishing**:
- Creates a GitHub Release matching the pushed tag and uploads the `.deb`, `.rpm`, and `.tar.gz` packages as release assets.
- For manual runs, compiles and exposes the packages as workflow run artifacts for testing.
---
## Contributing ## Contributing
Bug reports, packaging fixes, and patches are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for Bug reports, translations and patches are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the
the workflow and the quality gates CI enforces, and [SECURITY.md](SECURITY.md) for reporting a workflow and the checks CI runs, and [SECURITY.md](SECURITY.md) to report a vulnerability
vulnerability privately. privately.
The app, tray and search prompt are available in English, German, Spanish, French, Italian, Dutch, ## Support
Polish and Brazilian Portuguese. [docs/TRANSLATING.md](docs/TRANSLATING.md) explains how to improve
a translation or add a language. Proton Drive Linux is free and built in spare time. If it saves you a few headaches, you can
[buy me a coffee](https://buymeacoffee.com/narl).
<a href="https://buymeacoffee.com/narl"><img src="https://img.shields.io/badge/Buy%20me%20a%20coffee-FFDD00?logo=buymeacoffee&logoColor=black" alt="Buy me a coffee"></a>
## License ## License
Released under the [MIT License](LICENSE). Released under the [MIT License](LICENSE).
The Places map uses [GeoNames](https://www.geonames.org/) data (CC BY 4.0) and
[Natural Earth](https://www.naturalearthdata.com/) outlines (public domain). The optional street
map comes from [OpenFreeMap](https://openfreemap.org/), © [OpenStreetMap](https://www.openstreetmap.org/copyright)
contributors.
Proton Drive is a service of Proton AG. This project is an independent client and carries no Proton Drive is a service of Proton AG. This project is an independent client and carries no
affiliation with or endorsement from Proton AG. affiliation with or endorsement from Proton AG.
+5
View File
@@ -30,5 +30,10 @@ the interesting boundaries are:
- **Cryptography.** Key derivation, signature handling, and anything that could publish plaintext - **Cryptography.** Key derivation, signature handling, and anything that could publish plaintext
or the wrong key material to Proton. or the wrong key material to Proton.
What the client deliberately keeps in plaintext on disk, and why, is documented in the
[threat model](docs/ARCHITECTURE.md#10-threat-model-what-this-client-writes-to-disk-in-plaintext).
Behaviour described there is known and not a vulnerability by itself; a way to make it worse
than described is.
Out of scope: vulnerabilities in Proton's own service (report those to Proton AG), and issues that Out of scope: vulnerabilities in Proton's own service (report those to Proton AG), and issues that
require an attacker who already has your user account on the machine. require an attacker who already has your user account on the machine.
+1
View File
@@ -23,3 +23,4 @@ tracing-subscriber.workspace = true
clap.workspace = true clap.workspace = true
rpassword.workspace = true rpassword.workspace = true
serde_json.workspace = true serde_json.workspace = true
zeroize.workspace = true
+386 -39
View File
@@ -14,12 +14,45 @@ use pdfs_core::cache::ContentCache;
use pdfs_core::config::AppDirs; use pdfs_core::config::AppDirs;
use pdfs_core::control::{ use pdfs_core::control::{
ConflictKeep, ErrorKind, MountKind, RefreshScope, Request as CtlRequest, ConflictKeep, ErrorKind, MountKind, RefreshScope, Request as CtlRequest,
Response as CtlResponse, RestoreItem as CtlRestoreItem, ShareEntryKind, SyncPhase, Response as CtlResponse, RestoreItem as CtlRestoreItem, ShareEntryKind, SyncIssue, SyncPhase,
pending_summary, pending_summary,
}; };
use pdfs_core::db::Db; use pdfs_core::db::Db;
use pdfs_core::service; use pdfs_core::service;
/// `println!` for this binary, shadowing std's, which panics when stdout is
/// closed: `pdfs ls | head` ended in a "Broken pipe" panic once `head` had its
/// lines. This one hands the failure to [`stdout_written`] instead.
///
/// Restoring the default `SIGPIPE` would also stop the panic, but the same
/// binary runs the daemon, which must not die when a socket's peer goes away.
macro_rules! println {
($($arg:tt)*) => {{
use std::io::Write as _;
$crate::stdout_written(writeln!(std::io::stdout(), $($arg)*))
}};
}
/// [`println!`] without the newline.
macro_rules! print {
($($arg:tt)*) => {{
use std::io::Write as _;
$crate::stdout_written(write!(std::io::stdout(), $($arg)*))
}};
}
/// Exit on a failed write to stdout. A closed pipe means the reader has all it
/// wanted, so that exit is a success.
fn stdout_written(result: std::io::Result<()>) {
if let Err(error) = result {
if error.kind() == std::io::ErrorKind::BrokenPipe {
std::process::exit(0);
}
eprintln!("Error: writing to stdout: {error}");
std::process::exit(1);
}
}
#[derive(Parser)] #[derive(Parser)]
#[command( #[command(
name = "pdfs", name = "pdfs",
@@ -30,7 +63,7 @@ struct Cli {
/// Emit machine-readable JSON instead of formatted text. /// Emit machine-readable JSON instead of formatted text.
/// ///
/// Applies to the query commands (`status`, `ls`, `pins`, `sync list`, /// Applies to the query commands (`status`, `ls`, `pins`, `sync list`,
/// `devices list`, `locations`, `transfers`, `activity`, `cache inspect`). Commands that /// `devices list`, `locations`, `transfers`, `activity`, `trash`, `cache inspect`). Commands that
/// perform an action keep their human output — a script that needs to know /// perform an action keep their human output — a script that needs to know
/// whether one succeeded has the exit code. /// whether one succeeded has the exit code.
#[arg(long, global = true)] #[arg(long, global = true)]
@@ -102,6 +135,24 @@ enum Command {
/// Account email; prompted if omitted. /// Account email; prompted if omitted.
#[arg(short, long)] #[arg(short, long)]
username: Option<String>, username: Option<String>,
/// For a two-password account, keep the mailbox password's unlock in the
/// system keyring so the daemon starts without asking after every login.
/// Without it the unlock lives in kernel memory for this session only.
#[arg(long)]
remember: bool,
},
/// Enter the mailbox password of a two-password account so the daemon can
/// unlock your files. Needed once per login session.
Unlock {
/// Keep the unlock in the system keyring instead of kernel memory only.
#[arg(long)]
remember: bool,
},
/// Forget the unlock held in kernel memory; the account is locked again.
Lock {
/// Also remove an unlock kept in the keyring with `--remember`.
#[arg(long)]
forget: bool,
}, },
/// Forget the stored session. /// Forget the stored session.
Logout, Logout,
@@ -285,26 +336,34 @@ enum Command {
}, },
/// Rename a file or folder via the running daemon. /// Rename a file or folder via the running daemon.
Rename { Rename {
/// File/folder path, inside the mountpoint or relative to it. /// File/folder path, inside My files or an on-demand synced folder, or
/// relative to the mountpoint.
path: PathBuf, path: PathBuf,
/// New name (a single path component). /// New name (a single path component).
new_name: String, new_name: String,
}, },
/// Move a file or folder into another folder via the running daemon. /// Move files or folders into another folder via the running daemon.
///
/// The paths may be in different locations: My files and any synced
/// folder, on-demand or mirrored. The move happens on Proton Drive, so
/// nothing is downloaded or uploaded again, unlike `mv` between two
/// locations. The last path is the destination folder.
Move { Move {
/// File/folder path, inside the mountpoint or relative to it. /// Files/folders to move, then the destination folder: each inside a
path: PathBuf, /// location, or relative to the mountpoint.
/// Destination folder path, inside the mountpoint or relative to it. #[arg(required = true, num_args = 2.., value_name = "PATH")]
new_parent: PathBuf, paths: Vec<PathBuf>,
}, },
/// Trash a file or folder via the running daemon. /// Trash a file or folder via the running daemon.
Rm { Rm {
/// File/folder path, inside the mountpoint or relative to it. /// File/folder path, inside My files or an on-demand synced folder, or
/// relative to the mountpoint.
path: PathBuf, path: PathBuf,
}, },
/// Create a folder via the running daemon. /// Create a folder via the running daemon.
Mkdir { Mkdir {
/// Parent folder path, inside the mountpoint or relative to it. /// Parent folder path, inside My files or an on-demand synced folder,
/// or relative to the mountpoint.
parent: PathBuf, parent: PathBuf,
/// New folder name (a single path component). /// New folder name (a single path component).
name: String, name: String,
@@ -636,6 +695,25 @@ enum SyncCmd {
}, },
/// List uploads and changes not yet on Proton Drive. /// List uploads and changes not yet on Proton Drive.
Queue, Queue,
/// List queued changes Proton Drive refused or that keep failing, and what to do.
Issues,
/// Save the content of a queued upload (by id from `sync queue`) to a file.
Export {
/// Queue id.
id: i64,
/// Where to save it: a new file, or an existing folder to save it in.
dest: PathBuf,
},
/// Drop a queued change (by id from `sync queue`) and undo it here: a new file
/// or folder goes, anything else goes back to what Proton Drive has. Its
/// content is lost unless saved with `sync export` first.
Discard {
/// Queue id.
id: i64,
/// Do not ask first.
#[arg(long)]
yes: bool,
},
/// Retry a queued operation now (by id from `sync queue`), or every failed one. /// Retry a queued operation now (by id from `sync queue`), or every failed one.
Retry { Retry {
/// Queue id; omit to retry every failed operation. /// Queue id; omit to retry every failed operation.
@@ -743,7 +821,9 @@ fn main() -> Result<()> {
let cli = Cli::parse(); let cli = Cli::parse();
JSON_OUTPUT.store(cli.json, std::sync::atomic::Ordering::Relaxed); JSON_OUTPUT.store(cli.json, std::sync::atomic::Ordering::Relaxed);
match cli.command { match cli.command {
Command::Login { username } => cmd_login(username), Command::Login { username, remember } => cmd_login(username, remember),
Command::Unlock { remember } => cmd_unlock(remember),
Command::Lock { forget } => cmd_lock(forget),
Command::Logout => cmd_logout(), Command::Logout => cmd_logout(),
Command::Status => cmd_status(), Command::Status => cmd_status(),
Command::Locations => cmd_locations(), Command::Locations => cmd_locations(),
@@ -800,7 +880,10 @@ fn main() -> Result<()> {
scope, scope,
} => cmd_search(query, limit, scope), } => cmd_search(query, limit, scope),
Command::Rename { path, new_name } => cmd_rename(path, new_name), Command::Rename { path, new_name } => cmd_rename(path, new_name),
Command::Move { path, new_parent } => cmd_move(path, new_parent), Command::Move { mut paths } => {
let new_parent = paths.pop().expect("clap requires at least two paths");
cmd_move(paths, new_parent)
}
Command::Rm { path } => cmd_rm(path), Command::Rm { path } => cmd_rm(path),
Command::Mkdir { parent, name } => cmd_mkdir(parent, name), Command::Mkdir { parent, name } => cmd_mkdir(parent, name),
Command::Upload { sources, parent } => cmd_upload(sources, parent), Command::Upload { sources, parent } => cmd_upload(sources, parent),
@@ -984,7 +1067,17 @@ fn cmd_sync(action: SyncCmd) -> Result<()> {
})?)? })?)?
} }
SyncCmd::Queue => return cmd_sync_queue(), SyncCmd::Queue => return cmd_sync_queue(),
SyncCmd::Issues => return cmd_sync_issues(),
SyncCmd::Export { id, dest } => return cmd_sync_export(id, &dest),
SyncCmd::Limit { up, down } => return cmd_sync_limit(up, down), SyncCmd::Limit { up, down } => return cmd_sync_limit(up, down),
SyncCmd::Discard { id, yes } => {
if !yes {
confirm(&format!(
"Discard queued change {id} and undo it here? Content not exported is lost."
))?;
}
ok_or_bail(control_request(CtlRequest::DiscardPendingOp { id })?)?
}
SyncCmd::Retry { id } => ok_or_bail(control_request(CtlRequest::RetryPendingOp { id })?)?, SyncCmd::Retry { id } => ok_or_bail(control_request(CtlRequest::RetryPendingOp { id })?)?,
SyncCmd::Resume { folder: None } => { SyncCmd::Resume { folder: None } => {
ok_or_bail(control_request(CtlRequest::SetSyncPaused { ok_or_bail(control_request(CtlRequest::SetSyncPaused {
@@ -1062,6 +1155,9 @@ fn cmd_sync_queue() -> Result<()> {
(false, _) => "due now".to_string(), (false, _) => "due now".to_string(),
}; };
println!("[{}] {} {} ({state})", op.id, op.kind, op.path); println!("[{}] {} {} ({state})", op.id, op.kind, op.path);
if let Some(issue) = op.issue {
println!(" needs attention: {}", issue_text(issue));
}
if let Some(error) = &op.last_error { if let Some(error) = &op.last_error {
println!(" failed {} time(s): {error}", op.attempts); println!(" failed {} time(s): {error}", op.attempts);
} }
@@ -1073,6 +1169,91 @@ fn cmd_sync_queue() -> Result<()> {
Ok(()) Ok(())
} }
/// `pdfs sync issues`: the queued changes that need the user, and what to do.
fn cmd_sync_issues() -> Result<()> {
let response = control_request(CtlRequest::ListPendingOps)?;
let items = match response {
CtlResponse::PendingOps { items } => items,
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
};
let issues: Vec<_> = items.into_iter().filter(|op| op.failing).collect();
if json_enabled() {
println!("{}", serde_json::to_string_pretty(&issues)?);
return Ok(());
}
if issues.is_empty() {
println!("No sync issues.");
return Ok(());
}
for op in issues {
println!("[{}] {} {}", op.id, op.kind, op.path);
match op.issue {
Some(issue) => println!(" {}", issue_text(issue)),
None => println!(" keeps failing; it is retried."),
}
if let Some(error) = &op.last_error {
println!(" last error: {error}");
}
if op.exportable {
println!(
" save a copy: pdfs sync export {} <file or folder>",
op.id
);
}
}
Ok(())
}
/// What a sync issue means, and what the user can do about it.
fn issue_text(issue: SyncIssue) -> &'static str {
match issue {
SyncIssue::Quota => {
"Proton Drive storage is full. Free up space or upgrade; the change is retried."
}
SyncIssue::Access => {
"you can no longer change this on Proton Drive, for example because a share was \
revoked. Save a copy to keep the content."
}
SyncIssue::Missing => {
"the file or its folder is gone from Proton Drive. Save a copy to keep the content."
}
SyncIssue::Limit => {
"the folder holds too many items or is nested too deep. Move the file to another \
folder."
}
SyncIssue::Rejected => {
"Proton Drive rejected the change, for example the name. Rename the file."
}
SyncIssue::Unknown => "Proton Drive refused the change.",
}
}
/// `pdfs sync export`: save a queued upload's content. A folder as `dest`
/// gets the file under its own name.
fn cmd_sync_export(id: i64, dest: &Path) -> Result<()> {
let mut dest =
std::path::absolute(dest).with_context(|| format!("resolving {}", dest.display()))?;
if dest.is_dir() {
let items = match control_request(CtlRequest::ListPendingOps)? {
CtlResponse::PendingOps { items } => items,
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
};
let Some(op) = items.iter().find(|op| op.id == id) else {
bail!("no queued change {id}; see `pdfs sync queue`");
};
let name = Path::new(&op.path)
.file_name()
.with_context(|| format!("{} has no file name", op.path))?;
dest.push(name);
}
ok_or_bail(control_request(CtlRequest::ExportPendingOp {
id,
dest: dest.display().to_string(),
})?)
}
/// `pdfs sync restore`: list what this machine's device holds, confirm where /// `pdfs sync restore`: list what this machine's device holds, confirm where
/// each folder should land, then hand the mapping to the daemon. /// each folder should land, then hand the mapping to the daemon.
/// ///
@@ -1620,7 +1801,7 @@ fn prompt_line(label: &str) -> Result<String> {
Ok(line.trim().to_owned()) Ok(line.trim().to_owned())
} }
fn cmd_login(username: Option<String>) -> Result<()> { fn cmd_login(username: Option<String>, remember: bool) -> Result<()> {
let username = match username { let username = match username {
Some(u) => u, Some(u) => u,
None => prompt_line("Proton email")?, None => prompt_line("Proton email")?,
@@ -1632,20 +1813,32 @@ fn cmd_login(username: Option<String>) -> Result<()> {
prompt_line("2FA code").map_err(|e| pdfs_core::Error::Other(format!("read 2FA code: {e}"))) prompt_line("2FA code").map_err(|e| pdfs_core::Error::Other(format!("read 2FA code: {e}")))
}; };
// Only a two-password account is ever asked for this.
let get_mailbox = |attempt: u32| prompt_mailbox(attempt, remember);
// Proton may ask for a CAPTCHA first, usually on a VPN. The CLI has no
// browser engine, so the user solves the page in their own browser and says
// when they are done; the retry then carries the challenge token.
let get_hv = |hv: pdfs_core::proton_sdk::api::HumanVerification| {
println!(
"Proton is asking for a CAPTCHA before it will accept this sign-in.\n\
Open this page in your browser and complete it:\n\n {}\n",
auth::browser_verification_url(&hv)
);
prompt_line("Press Enter once the page says you are verified")
.map_err(|e| pdfs_core::Error::Other(format!("read verification answer: {e}")))?;
Ok(auth::browser_verified(&hv))
};
let rt = tokio::runtime::Runtime::new()?; let rt = tokio::runtime::Runtime::new()?;
// Solving a CAPTCHA needs a browser engine, which the CLI has no business rt.block_on(auth::login_interactive(
// carrying. Point at the app that does rather than failing with the raw API &username,
// message, which reads as "your login is broken". &password,
rt.block_on(auth::login(&username, &password, get_totp)) get_totp,
.map_err(|e| match e { get_mailbox,
pdfs_core::Error::HumanVerificationRequired(_) => anyhow::anyhow!( get_hv,
"Proton is asking for a CAPTCHA before it will accept this sign-in.\n\ ))
Run `pdfs-app` and sign in there — it can show the verification page.\n\ .context("login failed")?;
(This is usually triggered by a VPN or an unfamiliar IP; signing in from \
your usual network often avoids it.)"
),
other => anyhow::Error::new(other).context("login failed"),
})?;
// A daemon that is already up is parked in `wait_for_session`, sleeping // A daemon that is already up is parked in `wait_for_session`, sleeping
// between keyring checks; restart it so the mount appears now instead of // between keyring checks; restart it so the mount appears now instead of
@@ -1657,9 +1850,88 @@ fn cmd_login(username: Option<String>) -> Result<()> {
} }
println!("Logged in as {username}. Session stored in the system keyring."); println!("Logged in as {username}. Session stored in the system keyring.");
report_key_state();
Ok(()) Ok(())
} }
/// Ask for the mailbox password of a two-password account, without echo.
fn prompt_mailbox(attempt: u32, remember: bool) -> pdfs_core::Result<auth::Mailbox> {
if attempt == 0 {
eprintln!("This account has a separate mailbox password.");
} else {
eprintln!("That mailbox password did not unlock the account. Try again.");
}
let password = rpassword::prompt_password("Mailbox password: ")
.map_err(|e| pdfs_core::Error::Other(format!("read mailbox password: {e}")))?;
Ok(auth::Mailbox {
password: zeroize::Zeroizing::new(password),
remember,
})
}
/// Say where the account's keys are held, and what that means for the next login.
fn report_key_state() {
match auth::key_state() {
Ok(auth::KeyState::Session) => println!(
"Unlocked for this login session only. After you log out or reboot, run `pdfs unlock`."
),
Ok(auth::KeyState::Locked) => println!("The account is locked. Run `pdfs unlock`."),
_ => {}
}
}
fn cmd_unlock(remember: bool) -> Result<()> {
auth::load().context("not logged in (run `pdfs login` first)")?;
let rt = tokio::runtime::Runtime::new()?;
// Bounded like the login prompt: three wrong answers end the command.
let mut attempt = 0;
loop {
let mailbox = prompt_mailbox(attempt, remember)?;
match rt.block_on(auth::unlock(&mailbox.password, remember)) {
Ok(()) => break,
Err(pdfs_core::Error::WrongMailboxPassword) if attempt < 2 => attempt += 1,
Err(e) => return Err(anyhow::Error::new(e).context("unlock failed")),
}
}
// A daemon parked on the locked account would otherwise notice on its next
// poll; restarting it makes the mount appear now, as after `pdfs login`.
if service::is_active() {
service::restart();
}
println!("Unlocked.");
report_key_state();
Ok(())
}
fn cmd_lock(forget: bool) -> Result<()> {
auth::lock(forget)?;
println!(
"Locked. {}",
if forget {
"Unlock removed from memory and keyring."
} else {
"Run `pdfs unlock` to unlock again."
}
);
// The running daemon keeps its keys until it restarts; say so, since
// "locked" would otherwise suggest the mount stopped.
if service::is_active() {
println!(
"The running daemon keeps its keys until it restarts: `systemctl --user restart proton-drive`."
);
}
Ok(())
}
/// `stored`, `session` or `locked`: where the account's keys are held.
fn key_state_name() -> &'static str {
match auth::key_state() {
Ok(auth::KeyState::Stored) => "stored",
Ok(auth::KeyState::Session) => "session",
_ => "locked",
}
}
fn cmd_logout() -> Result<()> { fn cmd_logout() -> Result<()> {
auth::logout()?; auth::logout()?;
println!("Stored session cleared."); println!("Stored session cleared.");
@@ -1687,6 +1959,7 @@ fn cmd_status() -> Result<()> {
"username": session.as_ref().map(|s| s.username.clone()), "username": session.as_ref().map(|s| s.username.clone()),
"user_id": session.as_ref().map(|s| s.user_id.clone()), "user_id": session.as_ref().map(|s| s.user_id.clone()),
"scopes": session.as_ref().map(|s| s.scopes.clone()), "scopes": session.as_ref().map(|s| s.scopes.clone()),
"keys": session.as_ref().map(|_| key_state_name()),
"mount": mount, "mount": mount,
}); });
println!("{}", serde_json::to_string_pretty(&out)?); println!("{}", serde_json::to_string_pretty(&out)?);
@@ -1697,6 +1970,11 @@ fn cmd_status() -> Result<()> {
Ok(s) => { Ok(s) => {
println!("Logged in as {} (user {})", s.username, s.user_id); println!("Logged in as {} (user {})", s.username, s.user_id);
println!("Scopes: {}", s.scopes.join(", ")); println!("Scopes: {}", s.scopes.join(", "));
match key_state_name() {
"locked" => println!("Keys: locked — run `pdfs unlock`"),
"session" => println!("Keys: unlocked for this login session"),
_ => {}
}
} }
Err(pdfs_core::Error::NotLoggedIn) => { Err(pdfs_core::Error::NotLoggedIn) => {
println!("Not logged in. Run `pdfs login`."); println!("Not logged in. Run `pdfs login`.");
@@ -1828,7 +2106,7 @@ fn login_poll_delay(attempts: u32) -> std::time::Duration {
std::time::Duration::from_secs(BASE_SECS.saturating_mul(factor).min(MAX_SECS)) std::time::Duration::from_secs(BASE_SECS.saturating_mul(factor).min(MAX_SECS))
} }
/// Block until a session is stored, backing off between checks. /// Block until a session is stored and unlocked, backing off between checks.
/// ///
/// `load` and `sleep` are parameters rather than direct calls so the backoff /// `load` and `sleep` are parameters rather than direct calls so the backoff
/// schedule can be tested without a keyring and without real time passing. /// schedule can be tested without a keyring and without real time passing.
@@ -1853,6 +2131,16 @@ where
sleep(delay); sleep(delay);
attempts = attempts.saturating_add(1); attempts = attempts.saturating_add(1);
} }
Err(pdfs_core::Error::Locked) => {
let delay = login_poll_delay(attempts);
if attempts == 0 {
tracing::info!("account is locked; waiting for `pdfs unlock`…");
} else {
tracing::debug!(delay_secs = delay.as_secs(), "account still locked");
}
sleep(delay);
attempts = attempts.saturating_add(1);
}
Err(e) => return Err(e.into()), Err(e) => return Err(e.into()),
} }
} }
@@ -1863,10 +2151,13 @@ where
/// an external unmount triggers a remount; errors back off and retry. This is /// an external unmount triggers a remount; errors back off and retry. This is
/// the entry point for the systemd user service. /// the entry point for the systemd user service.
fn cmd_daemon(mountpoint: Option<PathBuf>) -> Result<()> { fn cmd_daemon(mountpoint: Option<PathBuf>) -> Result<()> {
// The daemon holds the account's key passphrases in memory; keep them out
// of core dumps and away from same-user ptrace.
pdfs_core::kernelkey::forbid_dumps();
loop { loop {
// Wait until a session is stored. The GUI enables this service on login, // Wait until a session is stored. The GUI enables this service on login,
// but the service may also start at boot before the user has logged in. // but the service may also start at boot before the user has logged in.
wait_for_session(|| auth::load().map(|_| ()), std::thread::sleep)?; wait_for_session(auth::ready, pdfs_fuse::idle_sleep)?;
match mount_once(mountpoint.clone()) { match mount_once(mountpoint.clone()) {
Ok(pdfs_fuse::MountOutcome::Shutdown) => { Ok(pdfs_fuse::MountOutcome::Shutdown) => {
@@ -1875,13 +2166,13 @@ fn cmd_daemon(mountpoint: Option<PathBuf>) -> Result<()> {
} }
Ok(pdfs_fuse::MountOutcome::Unmounted) => { Ok(pdfs_fuse::MountOutcome::Unmounted) => {
tracing::warn!("mount ended externally; remounting in 2s"); tracing::warn!("mount ended externally; remounting in 2s");
std::thread::sleep(std::time::Duration::from_secs(2)); pdfs_fuse::idle_sleep(std::time::Duration::from_secs(2));
} }
Err(e) => { Err(e) => {
// `{:#}` so the whole anyhow chain lands in the log — the top // `{:#}` so the whole anyhow chain lands in the log — the top
// context alone ("mount failed") never says *why*. // context alone ("mount failed") never says *why*.
tracing::error!(error = format!("{e:#}"), "mount failed; retrying in 5s"); tracing::error!(error = format!("{e:#}"), "mount failed; retrying in 5s");
std::thread::sleep(std::time::Duration::from_secs(5)); pdfs_fuse::idle_sleep(std::time::Duration::from_secs(5));
} }
} }
} }
@@ -1939,6 +2230,7 @@ fn mount_once(mountpoint: Option<PathBuf>) -> Result<pdfs_fuse::MountOutcome> {
sweep_mode: config.resolved_conflict_sweep(), sweep_mode: config.resolved_conflict_sweep(),
upload_limit: config.upload_limit.unwrap_or(0), upload_limit: config.upload_limit.unwrap_or(0),
download_limit: config.download_limit.unwrap_or(0), download_limit: config.download_limit.unwrap_or(0),
local_first: config.local_first.unwrap_or(true),
}, },
); );
@@ -1946,6 +2238,9 @@ fn mount_once(mountpoint: Option<PathBuf>) -> Result<pdfs_fuse::MountOutcome> {
if let Err(e) = rt.block_on(auth::persist(&session)) { if let Err(e) = rt.block_on(auth::persist(&session)) {
tracing::debug!(error = %e, "no new tokens to persist on shutdown"); tracing::debug!(error = %e, "no new tokens to persist on shutdown");
} }
// Dropping the runtime waits for every blocking task, however long; a call
// still stuck on Drive would hold the stop until systemd kills it (B165).
rt.shutdown_timeout(std::time::Duration::from_secs(2));
let outcome = result.context("mount failed")?; let outcome = result.context("mount failed")?;
Ok(outcome) Ok(outcome)
@@ -2439,14 +2734,33 @@ fn cmd_rename(path: PathBuf, new_name: String) -> Result<()> {
Ok(()) Ok(())
} }
fn cmd_move(path: PathBuf, new_parent: PathBuf) -> Result<()> { /// Move each of `sources` into `new_parent`, one request each. A refusal names
match control_request(CtlRequest::Move { /// its source and the rest still move; the command fails if any did not.
path: path_arg(&path)?, fn cmd_move(sources: Vec<PathBuf>, new_parent: PathBuf) -> Result<()> {
new_parent: path_arg(&new_parent)?, let new_parent = path_arg(&new_parent)?;
})? { let mut failed = 0;
CtlResponse::Ok { message } => println!("{message}"), for source in &sources {
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)), let outcome = path_arg(source).and_then(|path| {
other => bail!("unexpected response: {other:?}"), match control_request(CtlRequest::Move {
path,
new_parent: new_parent.clone(),
})? {
CtlResponse::Ok { message } => Ok(message),
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
}
});
match outcome {
Ok(message) => println!("{message}"),
Err(e) if sources.len() == 1 => return Err(e),
Err(e) => {
eprintln!("{}: {e:#}", source.display());
failed += 1;
}
}
}
if failed > 0 {
bail!("{failed} of {} could not be moved", sources.len());
} }
Ok(()) Ok(())
} }
@@ -2524,7 +2838,11 @@ fn cmd_diagnostics() -> Result<()> {
} }
fn cmd_trash() -> Result<()> { fn cmd_trash() -> Result<()> {
match control_request(CtlRequest::ListTrash)? { let response = control_request(CtlRequest::ListTrash)?;
if emit_json(&response)? {
return Ok(());
}
match response {
CtlResponse::Entries { entries } if entries.is_empty() => println!("(trash is empty)"), CtlResponse::Entries { entries } if entries.is_empty() => println!("(trash is empty)"),
CtlResponse::Entries { entries } => { CtlResponse::Entries { entries } => {
for e in entries { for e in entries {
@@ -3036,6 +3354,19 @@ fn cmd_diagnose() -> Result<()> {
match pdfs_core::auth::load() { match pdfs_core::auth::load() {
Ok(session) => { Ok(session) => {
report.finding(DiagnoseLevel::Ok, " keyring session", session.username); report.finding(DiagnoseLevel::Ok, " keyring session", session.username);
match key_state_name() {
"locked" => report.finding(
DiagnoseLevel::Fail,
" account keys",
"locked — run `pdfs unlock` (the daemon waits for it)",
),
"session" => report.finding(
DiagnoseLevel::Ok,
" account keys",
"unlocked for this login session",
),
_ => report.finding(DiagnoseLevel::Ok, " account keys", "stored in the keyring"),
}
} }
Err(e) => { Err(e) => {
// A locked keyring and an absent login look the same from here, so // A locked keyring and an absent login look the same from here, so
@@ -3334,6 +3665,21 @@ mod login_wait_tests {
); );
} }
#[test]
fn a_locked_account_is_waited_on_until_it_is_unlocked() {
let mut checks = vec![
Ok(()),
Err(pdfs_core::Error::Locked),
Err(pdfs_core::Error::NotLoggedIn),
Err(pdfs_core::Error::Locked),
];
let mut slept = 0;
wait_for_session(|| checks.pop().unwrap(), |_| slept += 1).unwrap();
assert_eq!(slept, 3, "locked and signed-out are both waited out");
}
#[test] #[test]
fn a_stored_session_is_not_waited_for_at_all() { fn a_stored_session_is_not_waited_for_at_all() {
let mut slept = 0; let mut slept = 0;
@@ -3366,6 +3712,7 @@ mod login_wait_tests {
shared_by_unverified: false, shared_by_unverified: false,
trashed_at: 0, trashed_at: 0,
trashed_from: None, trashed_from: None,
issue: None,
}; };
assert_eq!(shared_by_line(&entry), None, "no invitation, no line"); assert_eq!(shared_by_line(&entry), None, "no invitation, no line");
entry.shared_by = "alice@proton.me".into(); entry.shared_by = "alice@proton.me".into();
+2
View File
@@ -28,3 +28,5 @@ dbus-secret-service-keyring-store.workspace = true
parking_lot.workspace = true parking_lot.workspace = true
async-trait.workspace = true async-trait.workspace = true
libc.workspace = true libc.workspace = true
zeroize.workspace = true
base64.workspace = true
+533 -119
View File
@@ -1,22 +1,50 @@
//! Login, session persistence, and Drive-client construction. //! Login, session persistence, and Drive-client construction.
//! //!
//! A successful login is persisted to the OS keyring as a single JSON blob //! A successful login is persisted to the OS keyring as a single JSON blob
//! holding the resumable session tokens plus the mailbox password (needed to //! holding the resumable session tokens, the account's key salts and — unless
//! rebuild the key chain on resume). The daemon resumes from this blob with no //! the user chose otherwise — the per-key *passphrases* derived from the
//! interactive step; the refresh token auto-renews via the HTTP client's 401 //! mailbox password. The password itself is never stored: the passphrases are
//! path, so no fresh 2FA is required until the refresh token itself expires. //! what it exists to produce, and they are enough to rebuild the key chain on
//! resume. The daemon resumes from this blob with no interactive step; the
//! refresh token auto-renews via the HTTP client's 401 path, so no fresh 2FA is
//! required until the refresh token itself expires.
//!
//! # Two-password accounts
//!
//! An account with a separate mailbox password exists precisely so that no one
//! login holds the means to read the data. Storing that password in the keyring
//! would defeat it, so such an account is **locked by default**: its
//! passphrases are kept only in the kernel keyring ([`crate::kernelkey`]), for
//! the length of the user's session, and `pdfs unlock` (or the app) supplies
//! them again after each login. Storing them in the Secret Service as well is
//! an explicit opt-in ([`Mailbox::remember`]). Single-password accounts have no
//! such separation to protect and are stored, as they always were.
use std::collections::BTreeMap;
use base64::Engine as _;
use base64::engine::general_purpose::STANDARD as BASE64;
use keyring_core::Entry; use keyring_core::Entry;
use proton_drive_rs::{KeySalt, ProtonDriveClient}; use proton_drive_rs::{KeySalt, ProtonDriveClient};
use proton_sdk::account::KeyPassphrases;
use proton_sdk::api::{HumanVerification, HumanVerificationCredential}; use proton_sdk::api::{HumanVerification, HumanVerificationCredential};
use proton_sdk::cache::EncryptedCacheRepository; use proton_sdk::cache::EncryptedCacheRepository;
use proton_sdk::config::ProtonClientConfiguration; use proton_sdk::config::ProtonClientConfiguration;
use proton_sdk::error::ProtonError;
use proton_sdk::session::{PasswordMode, ProtonApiSession, ResumeParameters}; use proton_sdk::session::{PasswordMode, ProtonApiSession, ResumeParameters};
use proton_sdk::telemetry::TracingTelemetry; use proton_sdk::telemetry::TracingTelemetry;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use zeroize::Zeroizing;
use crate::config::{APP_VERSION, AppDirs, KEYRING_SERVICE, USER_AGENT}; use crate::config::{APP_VERSION, AppDirs, KEYRING_SERVICE, USER_AGENT};
use crate::error::{Error, Result}; use crate::error::{Error, Result};
use crate::kernelkey;
/// How often a wrong mailbox password is asked for again before the login gives up.
const MAILBOX_ATTEMPTS: u32 = 3;
/// Domain-separation label for the key that encrypts the persistent entity cache.
const ENTITY_CACHE_PURPOSE: &[u8] = b"pdfs entity cache v1";
/// Fixed keyring account name for the single stored session blob. /// Fixed keyring account name for the single stored session blob.
const KEYRING_USER: &str = "session"; const KEYRING_USER: &str = "session";
@@ -32,8 +60,11 @@ pub struct StoredSession {
pub scopes: Vec<String>, pub scopes: Vec<String>,
/// `1` = single password, `2` = dual (Proton wire value). /// `1` = single password, `2` = dual (Proton wire value).
pub password_mode: u8, pub password_mode: u8,
/// Mailbox (data) password — required by `ProtonDriveClient::new` to derive /// The mailbox password of a blob written by 3.0.x and earlier.
/// the key chain. Lives only in the OS keyring, never on disk in cleartext. ///
/// Read only, to migrate: [`resume_client`] derives the passphrases from it
/// once and writes the blob back without it. Never serialized again.
#[serde(default, skip_serializing)]
pub mailbox_password: String, pub mailbox_password: String,
/// The account's key salts, captured at login. /// The account's key salts, captured at login.
/// ///
@@ -49,6 +80,11 @@ pub struct StoredSession {
/// backfilling) them. /// backfilling) them.
#[serde(default)] #[serde(default)]
pub key_salts: Vec<KeySalt>, pub key_salts: Vec<KeySalt>,
/// Per-key passphrases (key id → base64), when the user keeps them in the
/// keyring: always for a single-password account, by opt-in for a
/// two-password one. `None` means they are not stored here.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub key_passphrases: Option<BTreeMap<String, String>>,
} }
impl StoredSession { impl StoredSession {
@@ -73,7 +109,10 @@ fn client_config() -> ProtonClientConfiguration {
let (app_version, user_agent) = match AppDirs::new() { let (app_version, user_agent) = match AppDirs::new() {
Ok(dirs) => { Ok(dirs) => {
let config = dirs.load_config(); let config = dirs.load_config();
(config.app_version, config.user_agent) (
config.resolved_app_version().to_string(),
config.resolved_user_agent().to_string(),
)
} }
Err(_) => (APP_VERSION.to_string(), USER_AGENT.to_string()), Err(_) => (APP_VERSION.to_string(), USER_AGENT.to_string()),
}; };
@@ -89,6 +128,26 @@ fn keyring_entry() -> Result<Entry> {
Ok(Entry::new(KEYRING_SERVICE, KEYRING_USER)?) Ok(Entry::new(KEYRING_SERVICE, KEYRING_USER)?)
} }
/// What the user answered when asked for the mailbox password.
pub struct Mailbox {
pub password: Zeroizing<String>,
/// Keep the derived passphrases in the keyring, so the daemon unlocks by
/// itself after every login. Off by default: it trades the protection a
/// separate mailbox password gives for convenience.
pub remember: bool,
}
/// Where the passphrases that unlock this account's keys currently live.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum KeyState {
/// In the keyring, available after every login.
Stored,
/// In kernel memory for this login session only.
Session,
/// Nowhere: the mailbox password has to be entered (`pdfs unlock`).
Locked,
}
/// Run an interactive SRP + (optional) 2FA login and persist the session. /// Run an interactive SRP + (optional) 2FA login and persist the session.
/// ///
/// `get_totp` is only invoked when the account requires a second factor, so /// `get_totp` is only invoked when the account requires a second factor, so
@@ -96,14 +155,19 @@ fn keyring_entry() -> Result<Entry> {
/// more than once — [`login_interactive`] re-runs the whole login after human /// more than once — [`login_interactive`] re-runs the whole login after human
/// verification, and a 2FA account needs a fresh code on that second attempt. /// verification, and a 2FA account needs a fresh code on that second attempt.
/// ///
/// `get_mailbox` is only invoked for a two-password account. It receives the
/// number of earlier wrong answers (so a prompt can say "try again") and is
/// asked at most [`MAILBOX_ATTEMPTS`] times; an `Err` from it aborts the login.
///
/// A gated login fails with [`Error::HumanVerificationRequired`]; a front-end /// A gated login fails with [`Error::HumanVerificationRequired`]; a front-end
/// that can present the challenge should use [`login_interactive`] instead. /// that can present the challenge should use [`login_interactive`] instead.
pub async fn login( pub async fn login(
username: &str, username: &str,
password: &str, password: &str,
get_totp: impl Fn() -> Result<String>, get_totp: impl Fn() -> Result<String>,
get_mailbox: impl Fn(u32) -> Result<Mailbox>,
) -> Result<()> { ) -> Result<()> {
login_verified(username, password, None, get_totp).await login_verified(username, password, None, get_totp, get_mailbox).await
} }
/// [`login`], able to answer a human-verification gate rather than fail on it. /// [`login`], able to answer a human-verification gate rather than fail on it.
@@ -123,17 +187,63 @@ pub async fn login_interactive(
username: &str, username: &str,
password: &str, password: &str,
get_totp: impl Fn() -> Result<String>, get_totp: impl Fn() -> Result<String>,
get_mailbox: impl Fn(u32) -> Result<Mailbox>,
get_hv: impl FnOnce(HumanVerification) -> Result<HumanVerificationCredential>, get_hv: impl FnOnce(HumanVerification) -> Result<HumanVerificationCredential>,
) -> Result<()> { ) -> Result<()> {
match login_verified(username, password, None, &get_totp).await { match login_verified(username, password, None, &get_totp, &get_mailbox).await {
Err(Error::HumanVerificationRequired(hv)) => { Err(Error::HumanVerificationRequired(hv)) => {
let credential = get_hv(*hv)?; let credential = get_hv(*hv)?;
login_verified(username, password, Some(&credential), &get_totp).await login_verified(
username,
password,
Some(&credential),
&get_totp,
&get_mailbox,
)
.await
} }
other => other, other => other,
} }
} }
/// Proton's verification page for the user's own browser, for a front-end that
/// cannot host it in a webview.
///
/// Unlike [`HumanVerification::verification_url`] this asks for the full page,
/// not the embedded one: a browser tab has no host to post the token to, so the
/// page has to tell the user they are done instead. Proton Mail Bridge opens the
/// same URL.
pub fn browser_verification_url(hv: &HumanVerification) -> String {
let token: String = hv
.token
.bytes()
.map(|b| match b {
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
(b as char).to_string()
}
_ => format!("%{b:02X}"),
})
.collect();
format!(
"https://verify.proton.me/?methods={}&token={token}",
hv.methods.join(","),
)
}
/// The credential to retry with once the user has solved the challenge in
/// their browser.
///
/// The browser keeps the page's own token, so the retry carries the challenge
/// token instead: solving the page marks that token as verified on Proton's
/// side. The type is every offered method, as Proton Mail Bridge sends it, since
/// the user may have picked any of them on the page.
pub fn browser_verified(hv: &HumanVerification) -> HumanVerificationCredential {
HumanVerificationCredential {
token: hv.token.clone(),
method: hv.methods.join(","),
}
}
/// [`login`], replaying a human-verification token the user has already earned. /// [`login`], replaying a human-verification token the user has already earned.
/// ///
/// A gated login fails with [`Error::HumanVerificationRequired`] carrying the /// A gated login fails with [`Error::HumanVerificationRequired`] carrying the
@@ -145,15 +255,18 @@ pub async fn login_verified(
password: &str, password: &str,
verification: Option<&HumanVerificationCredential>, verification: Option<&HumanVerificationCredential>,
get_totp: impl Fn() -> Result<String>, get_totp: impl Fn() -> Result<String>,
get_mailbox: impl Fn(u32) -> Result<Mailbox>,
) -> Result<()> { ) -> Result<()> {
let mut session = ProtonApiSession::begin_verified( let password = Zeroizing::new(password.to_owned());
let session = ProtonApiSession::begin_verified(
client_config(), client_config(),
username, username,
password.as_bytes(), password.as_bytes(),
verification, verification,
) )
.await .await
.map_err(classify_verification_gate)?; .map_err(classify_verification_gate);
let mut session = session?;
if session.is_waiting_for_second_factor() { if session.is_waiting_for_second_factor() {
let code = get_totp()?; let code = get_totp()?;
@@ -163,38 +276,38 @@ pub async fn login_verified(
// Grab the key salts while this access token still has the `locked` scope: // Grab the key salts while this access token still has the `locked` scope:
// after its first refresh it never will again, and without them no later // after its first refresh it never will again, and without them no later
// resume can unlock the key chain. See `StoredSession::key_salts`. // resume can unlock the key chain. See `StoredSession::key_salts`.
let client = ProtonDriveClient::new(&session, password.as_bytes().to_vec()); let key_salts = ProtonDriveClient::new(&session, Vec::new())
let key_salts = client.account().key_salts().await?; .account()
.key_salts()
.await?;
register_refresh_handler(&session, password.to_owned(), key_salts.clone()); // A single-password account's mailbox password is the login password. A
// two-password one has a second secret the user must type, and it is checked
save(&session, password, key_salts).await?; // against the account now — a typo found at login is a retry, found later it
Ok(()) // is a daemon that never mounts.
} let (passphrases, remember) = match session.password_mode() {
PasswordMode::Single => (
/// Lift a human-verification gate out of the generic API error into the typed derive_verified(&session, password.as_bytes(), key_salts.clone()).await?,
/// variant a front-end can act on. true,
/// ),
/// Only a gate that names a solvable method is converted. A `9001` with no PasswordMode::Dual => {
/// `Details`, or one offering only `email`/`sms`, stays a plain API error: let mut attempt = 0;
/// promoting it would send the UI to open a verification page it cannot loop {
/// complete, which reads to the user as a hang rather than a refusal. let mailbox = get_mailbox(attempt)?;
fn classify_verification_gate(e: proton_sdk::error::ProtonError) -> Error { match derive_verified(&session, mailbox.password.as_bytes(), key_salts.clone())
let proton_sdk::error::ProtonError::Api(api) = &e else { .await
return e.into(); {
Ok(passphrases) => break (passphrases, mailbox.remember),
Err(Error::WrongMailboxPassword) if attempt + 1 < MAILBOX_ATTEMPTS => {
attempt += 1;
}
Err(e) => return Err(e),
}
}
}
}; };
match api.human_verification() {
Some(hv) if hv.supports_captcha() => Error::HumanVerificationRequired(Box::new(hv)),
_ => e.into(),
}
}
/// Persist the session's current tokens, mailbox password and key salts. register_refresh_handler(&session);
pub async fn save(
session: &ProtonApiSession,
mailbox_password: &str,
key_salts: Vec<KeySalt>,
) -> Result<()> {
let tokens = session.current_tokens().await; let tokens = session.current_tokens().await;
let stored = StoredSession { let stored = StoredSession {
session_id: session.session_id().as_str().to_owned(), session_id: session.session_id().as_str().to_owned(),
@@ -207,10 +320,103 @@ pub async fn save(
PasswordMode::Single => 1, PasswordMode::Single => 1,
PasswordMode::Dual => 2, PasswordMode::Dual => 2,
}, },
mailbox_password: mailbox_password.to_owned(), mailbox_password: String::new(),
key_salts, key_salts,
key_passphrases: None,
}; };
let json = serde_json::to_string(&stored)?; keep_passphrases(stored, &passphrases, remember)
}
/// Derive the key passphrases from `mailbox` and prove they unlock the account.
///
/// [`Error::WrongMailboxPassword`] means the account answered and the password
/// is wrong; any other error says nothing about the password (no network, an
/// API fault) and must not be reported as a typo.
async fn derive_verified(
session: &ProtonApiSession,
mailbox: &[u8],
key_salts: Vec<KeySalt>,
) -> Result<KeyPassphrases> {
let client = ProtonDriveClient::with_key_salts(session, mailbox.to_vec(), key_salts);
client.account().unlock().await.map_err(|e| match e {
ProtonError::KeysLocked => Error::WrongMailboxPassword,
other => other.into(),
})?;
Ok(client.account().key_passphrases().await?)
}
/// Write `stored` with the passphrases in the place the user chose: the
/// keyring (`remember`) or kernel memory only. Whichever is not chosen is
/// cleared, so a revoked choice never leaves a copy behind.
///
/// The kernel entry is written first. If it cannot be (no keyring support), the
/// secret is *not* quietly stored somewhere weaker: the blob is written locked
/// and the error says so.
fn keep_passphrases(
mut stored: StoredSession,
passphrases: &KeyPassphrases,
remember: bool,
) -> Result<()> {
let outcome = if remember {
stored.key_passphrases = Some(encode(passphrases));
// Best-effort: a system without a kernel keyring has nothing to clear.
let _ = kernelkey::clear();
Ok(())
} else {
stored.key_passphrases = None;
kernelkey::store(&encode_secret(passphrases)).map_err(|e| {
Error::Other(format!(
"the kernel keyring is unavailable, so the unlock cannot be kept for this session: {e}"
))
})
};
write_stored(&stored)?;
outcome
}
fn encode(passphrases: &KeyPassphrases) -> BTreeMap<String, String> {
passphrases
.iter()
.map(|(id, secret)| (id.to_owned(), BASE64.encode(secret)))
.collect()
}
fn decode(encoded: &BTreeMap<String, String>) -> Option<KeyPassphrases> {
let mut passphrases = KeyPassphrases::new();
for (id, secret) in encoded {
passphrases.insert(id.clone(), BASE64.decode(secret).ok()?);
}
(!passphrases.is_empty()).then_some(passphrases)
}
/// The kernel-keyring payload: the same map as the blob's, as JSON.
fn encode_secret(passphrases: &KeyPassphrases) -> Zeroizing<Vec<u8>> {
Zeroizing::new(serde_json::to_vec(&encode(passphrases)).unwrap_or_default())
}
fn decode_secret(secret: &[u8]) -> Option<KeyPassphrases> {
decode(&serde_json::from_slice(secret).ok()?)
}
/// Lift a human-verification gate out of the generic API error into the typed
/// variant a front-end can act on.
///
/// Only a gate that names a solvable method is converted. A `9001` with no
/// `Details`, or one offering only `email`/`sms`, stays a plain API error:
/// promoting it would send the UI to open a verification page it cannot
/// complete, which reads to the user as a hang rather than a refusal.
fn classify_verification_gate(e: ProtonError) -> Error {
let ProtonError::Api(api) = &e else {
return e.into();
};
match api.human_verification() {
Some(hv) if hv.supports_captcha() => Error::HumanVerificationRequired(Box::new(hv)),
_ => e.into(),
}
}
fn write_stored(stored: &StoredSession) -> Result<()> {
let json = Zeroizing::new(serde_json::to_string(stored)?);
keyring_entry()?.set_password(&json)?; keyring_entry()?.set_password(&json)?;
Ok(()) Ok(())
} }
@@ -225,12 +431,93 @@ pub fn load() -> Result<StoredSession> {
} }
} }
/// Where the passphrases for the stored session live right now.
///
/// Reads the kernel keyring and the stored blob only; never the network.
pub fn key_state() -> Result<KeyState> {
Ok(key_state_of(&load()?))
}
fn key_state_of(stored: &StoredSession) -> KeyState {
match find_passphrases(stored) {
Ok(Some((_, origin))) => origin,
// A 3.0.x blob still holds its password; `resume_client` migrates it.
_ if !stored.mailbox_password.is_empty() => KeyState::Stored,
_ => KeyState::Locked,
}
}
/// Whether the stored session can be resumed without asking the user:
/// `Ok` when it can, [`Error::NotLoggedIn`] or [`Error::Locked`] when not.
/// What the daemon waits on before it mounts.
pub fn ready() -> Result<()> {
match key_state_of(&load()?) {
KeyState::Locked => Err(Error::Locked),
_ => Ok(()),
}
}
/// The passphrases the stored session can use without being asked, and which
/// of the two places they came from. Prefers the keyring blob: it is what the
/// user opted into. A corrupt entry is treated as absent, so it reads as
/// locked instead of wedging the daemon.
fn find_passphrases(stored: &StoredSession) -> Result<Option<(KeyPassphrases, KeyState)>> {
if let Some(found) = stored.key_passphrases.as_ref().and_then(decode) {
return Ok(Some((found, KeyState::Stored)));
}
match kernelkey::load() {
Ok(Some(secret)) => Ok(decode_secret(&secret).map(|p| (p, KeyState::Session))),
Ok(None) => Ok(None),
// No keyring on this system: nothing was ever kept there.
Err(e) => {
tracing::debug!(error = %e, "kernel keyring not readable; treating the account as locked");
Ok(None)
}
}
}
/// Supply the mailbox password of a locked account.
///
/// Verifies it against the account, then keeps the derived passphrases in
/// kernel memory for this login session — or, with `remember`, in the keyring.
/// A daemon waiting on [`Error::Locked`] picks them up on its next attempt.
///
/// Fails with [`Error::WrongMailboxPassword`] for a wrong password, which
/// changes nothing, and [`Error::ReloginRequired`] for a session that predates
/// stored key salts.
pub async fn unlock(mailbox: &str, remember: bool) -> Result<()> {
let stored = load()?;
if stored.key_salts.is_empty() {
return Err(Error::ReloginRequired);
}
let mailbox = Zeroizing::new(mailbox.to_owned());
let session = ProtonApiSession::resume(client_config(), stored.to_params())?;
register_refresh_handler(&session);
let passphrases =
derive_verified(&session, mailbox.as_bytes(), stored.key_salts.clone()).await?;
keep_passphrases(stored, &passphrases, remember)
}
/// Drop the passphrases held in kernel memory, so the account is locked again
/// until [`unlock`]. With `forget`, also those in the keyring blob.
pub fn lock(forget: bool) -> Result<()> {
kernelkey::clear()?;
if forget {
let mut stored = load()?;
if stored.key_passphrases.take().is_some() {
write_stored(&stored)?;
}
}
Ok(())
}
/// Forget the persisted session (best-effort; absent entry is not an error). /// Forget the persisted session (best-effort; absent entry is not an error).
pub fn logout() -> Result<()> { pub fn logout() -> Result<()> {
// The persisted entity cache describes the account being logged out of. It // The persisted entity cache describes the account being logged out of. It
// is encrypted under that account's mailbox password, but leaving it behind // is encrypted under a key derived from that account's passphrases, but
// would also mean the next account inherits a store it can only ever read as // leaving it behind would also mean the next account inherits a store it can
// misses. Best-effort: failing to remove a cache must not block the logout. // only ever read as misses. Best-effort: failing to remove a cache must not
// block the logout.
if let Ok(dirs) = AppDirs::new() { if let Ok(dirs) = AppDirs::new() {
let path = dirs.state_dir().join("sdk_cache.db"); let path = dirs.state_dir().join("sdk_cache.db");
if let Some(cache) = crate::sdkcache::SdkCache::opened() if let Some(cache) = crate::sdkcache::SdkCache::opened()
@@ -242,6 +529,9 @@ pub fn logout() -> Result<()> {
let _ = std::fs::remove_file(format!("{}{suffix}", path.display())); let _ = std::fs::remove_file(format!("{}{suffix}", path.display()));
} }
} }
if let Err(e) = kernelkey::clear() {
tracing::warn!(error = %e, "clearing the kernel-keyring unlock on logout failed");
}
match keyring_entry()?.delete_credential() { match keyring_entry()?.delete_credential() {
Ok(()) | Err(keyring_core::Error::NoEntry) => Ok(()), Ok(()) | Err(keyring_core::Error::NoEntry) => Ok(()),
Err(e) => Err(e.into()), Err(e) => Err(e.into()),
@@ -250,44 +540,87 @@ pub fn logout() -> Result<()> {
/// Resume a persisted session and build an authenticated Drive client. /// Resume a persisted session and build an authenticated Drive client.
/// ///
/// Returns `Error::NotLoggedIn` when no session has been saved. The caller is /// Returns `Error::NotLoggedIn` when no session has been saved and
/// responsible for persisting rotated tokens over the session's lifetime via /// [`Error::Locked`] when the account is waiting for its mailbox password. Tokens
/// [`persist`] — Proton refresh tokens are single-use, so a refresh that is not /// the session rotates are written back to the keyring as they happen (see
/// written back leaves the keyring holding a stale (now-invalid) refresh token, /// [`register_refresh_handler`]); Proton refresh tokens are single-use, so a
/// and the next resume fails with `InvalidRefreshToken`. /// refresh that is not written back would leave the keyring holding a stale
/// refresh token, and the next resume would fail with `InvalidRefreshToken`.
///
/// Makes no network call for an account whose passphrases are already known, so
/// a daemon starting offline can still mount from its cache.
pub async fn resume_client() -> Result<(ProtonDriveClient, ProtonApiSession)> { pub async fn resume_client() -> Result<(ProtonDriveClient, ProtonApiSession)> {
let stored = load()?; let stored = load()?;
let session = ProtonApiSession::resume(client_config(), stored.to_params())?; let session = ProtonApiSession::resume(client_config(), stored.to_params())?;
let password = stored.mailbox_password.clone(); register_refresh_handler(&session);
// Blobs written before key salts were persisted have none: fetch them once let passphrases = match find_passphrases(&stored)? {
// more and write them back, which only works while the stored access token Some((passphrases, _)) => passphrases,
// still carries the `locked` scope. If it doesn't, the account must be None if !stored.mailbox_password.is_empty() => migrate_legacy(&session, &stored).await?,
// logged into again — no refresh can restore that scope. None => return Err(Error::Locked),
let key_salts = if stored.key_salts.is_empty() {
register_refresh_handler(&session, password.clone(), Vec::new());
let probe = ProtonDriveClient::new(&session, password.clone().into_bytes());
let salts = probe.account().key_salts().await.map_err(|e| match &e {
proton_sdk::error::ProtonError::Api(api) if api.is_insufficient_scope() => {
Error::ReloginRequired
}
_ => e.into(),
})?;
save(&session, &password, salts.clone()).await?;
salts
} else {
stored.key_salts
}; };
register_refresh_handler(&session, password.clone(), key_salts.clone());
let client = tune( let client = tune(
ProtonDriveClient::with_key_salts(&session, password.clone().into_bytes(), key_salts), ProtonDriveClient::with_key_passphrases(&session, passphrases.clone()),
password.as_bytes(), &passphrases,
); );
Ok((client, session)) Ok((client, session))
} }
/// Turn a blob written by 3.0.x — which holds the mailbox password — into one
/// that holds passphrases, so the password stops being stored.
///
/// The old login saved whatever was typed as the mailbox password, which for a
/// two-password account is the *login* password and unlocks nothing. That is
/// checked here: a password the account rejects leaves the account locked (and
/// the stored copy gone) rather than failing every start forever.
async fn migrate_legacy(
session: &ProtonApiSession,
stored: &StoredSession,
) -> Result<KeyPassphrases> {
let legacy = Zeroizing::new(stored.mailbox_password.clone());
// Blobs from before key salts were persisted have none: fetch them once
// more, which only works while the stored access token still carries the
// `locked` scope. If it doesn't, the account must be logged into again — no
// refresh can restore that scope.
let key_salts = if stored.key_salts.is_empty() {
ProtonDriveClient::new(session, Vec::new())
.account()
.key_salts()
.await
.map_err(|e| match &e {
ProtonError::Api(api) if api.is_insufficient_scope() => Error::ReloginRequired,
_ => e.into(),
})?
} else {
stored.key_salts.clone()
};
let mut migrated = stored.clone();
migrated.mailbox_password = String::new();
migrated.key_salts = key_salts.clone();
match derive_verified(session, legacy.as_bytes(), key_salts).await {
Ok(passphrases) => {
migrated.key_passphrases = Some(encode(&passphrases));
write_stored(&migrated)?;
tracing::info!(
"migrated the stored session from a mailbox password to key passphrases"
);
Ok(passphrases)
}
Err(Error::WrongMailboxPassword) => {
write_stored(&migrated)?;
tracing::warn!(
"the stored password does not unlock the account's keys; unlock required"
);
Err(Error::Locked)
}
Err(e) => Err(e),
}
}
/// The client settings every front-end wants, applied wherever a /// The client settings every front-end wants, applied wherever a
/// [`ProtonDriveClient`] is built for real work. /// [`ProtonDriveClient`] is built for real work.
/// ///
@@ -301,23 +634,24 @@ pub async fn resume_client() -> Result<(ProtonDriveClient, ProtonApiSession)> {
/// failure point per write instead of three. /// failure point per write instead of three.
/// - **A persistent entity cache** ([`crate::sdkcache`]), so a restart does not /// - **A persistent entity cache** ([`crate::sdkcache`]), so a restart does not
/// re-fetch and re-decrypt the tree the previous run already walked. It is /// re-fetch and re-decrypt the tree the previous run already walked. It is
/// wrapped in the SDK's [`EncryptedCacheRepository`] keyed by the mailbox /// wrapped in the SDK's [`EncryptedCacheRepository`] keyed by a secret derived
/// password, so the decrypted node names it holds are not readable from the /// from the key passphrases, so the decrypted node names it holds are not
/// file alone — and a password change simply reads as a cold cache, since the /// readable from the file alone — and a password change simply reads as a cold
/// SDK treats an undecryptable entry as a miss and clears the store. /// cache, since the SDK treats an undecryptable entry as a miss and clears the
/// store.
/// ///
/// Opening it is best-effort: a store this process cannot open costs cache /// Opening it is best-effort: a store this process cannot open costs cache
/// hits, not the session, so the client falls back to the SDK's in-memory /// hits, not the session, so the client falls back to the SDK's in-memory
/// default. /// default.
fn tune(client: ProtonDriveClient, mailbox_password: &[u8]) -> ProtonDriveClient { fn tune(client: ProtonDriveClient, passphrases: &KeyPassphrases) -> ProtonDriveClient {
let client = client let client = client
.with_telemetry(TracingTelemetry::shared()) .with_telemetry(TracingTelemetry::shared())
.with_small_file_upload(true); .with_small_file_upload(true);
match AppDirs::new().and_then(|dirs| crate::sdkcache::SdkCache::shared(&dirs)) { match AppDirs::new().and_then(|dirs| crate::sdkcache::SdkCache::shared(&dirs)) {
Ok(cache) => client.with_entity_repository(EncryptedCacheRepository::shared( Ok(cache) => {
cache, let key = passphrases.derive_secret(ENTITY_CACHE_PURPOSE);
mailbox_password.to_vec(), client.with_entity_repository(EncryptedCacheRepository::shared(cache, key.to_vec()))
)), }
Err(e) => { Err(e) => {
tracing::warn!(error = %e, "persistent SDK entity cache unavailable; using memory"); tracing::warn!(error = %e, "persistent SDK entity cache unavailable; using memory");
client client
@@ -327,49 +661,129 @@ fn tune(client: ProtonDriveClient, mailbox_password: &[u8]) -> ProtonDriveClient
/// Write the session's current tokens back to the keyring. /// Write the session's current tokens back to the keyring.
/// ///
/// The session does not carry the mailbox password (it is only needed to rebuild /// Everything else in the stored blob — salts, passphrases — is preserved, so
/// the key chain on resume), so it is re-read from the existing stored blob and /// this never undoes an [`unlock`] or [`lock`] made since the session started.
/// preserved. Call whenever the session may have rotated its tokens through the /// Call whenever the session may have rotated its tokens so a later
/// 401-refresh path so a later [`resume_client`] presents a live refresh token. /// [`resume_client`] presents a live refresh token.
pub async fn persist(session: &ProtonApiSession) -> Result<()> { pub async fn persist(session: &ProtonApiSession) -> Result<()> {
let stored = load()?; let tokens = session.current_tokens().await;
save(session, &stored.mailbox_password, stored.key_salts).await store_tokens(tokens.access_token, tokens.refresh_token)
} }
fn register_refresh_handler( fn store_tokens(access_token: String, refresh_token: String) -> Result<()> {
session: &ProtonApiSession, let mut stored = load()?;
mailbox_password: String, stored.access_token = access_token;
key_salts: Vec<KeySalt>, stored.refresh_token = refresh_token;
) { write_stored(&stored)
let session_id = session.session_id().as_str().to_owned(); }
let username = session.username().to_owned();
let user_id = session.user_id().as_str().to_owned();
let scopes = session.scopes().to_vec();
let password_mode = match session.password_mode() {
PasswordMode::Single => 1,
PasswordMode::Dual => 2,
};
/// Persist rotated tokens the moment the session obtains them.
///
/// The handler re-reads the stored blob instead of capturing one, so it cannot
/// write back a stale copy over a later unlock or lock — and cannot resurrect a
/// session that was logged out while the daemon still ran.
fn register_refresh_handler(session: &ProtonApiSession) {
session.http().set_on_tokens_refreshed(move |tokens| { session.http().set_on_tokens_refreshed(move |tokens| {
let stored = StoredSession { match store_tokens(tokens.access_token, tokens.refresh_token) {
session_id: session_id.clone(), Ok(()) => tracing::info!("successfully auto-persisted refreshed tokens in keyring"),
username: username.clone(), Err(Error::NotLoggedIn) => {}
user_id: user_id.clone(), Err(e) => {
access_token: tokens.access_token, tracing::warn!(error = %e, "failed to auto-persist refreshed tokens in keyring")
refresh_token: tokens.refresh_token,
scopes: scopes.clone(),
password_mode,
mailbox_password: mailbox_password.clone(),
key_salts: key_salts.clone(),
};
if let Ok(json) = serde_json::to_string(&stored)
&& let Ok(entry) = keyring_entry()
{
if let Err(e) = entry.set_password(&json) {
tracing::warn!(error = %e, "failed to auto-persist refreshed tokens in keyring");
} else {
tracing::info!("successfully auto-persisted refreshed tokens in keyring");
} }
} }
}); });
} }
#[cfg(test)]
mod tests {
use super::*;
fn passphrases() -> KeyPassphrases {
let mut p = KeyPassphrases::new();
p.insert("key-a", b"alpha".to_vec());
p.insert("key-b", vec![0, 255, 7]);
p
}
fn blob(extra: &str) -> String {
format!(
r#"{{"session_id":"s","username":"u","user_id":"i","access_token":"a",
"refresh_token":"r","scopes":[],"password_mode":2{extra}}}"#
)
}
fn challenge() -> HumanVerification {
serde_json::from_str(
r#"{"HumanVerificationToken":"ab+c/d=","HumanVerificationMethods":["captcha","email"]}"#,
)
.unwrap()
}
#[test]
fn the_browser_gets_the_full_verification_page() {
assert_eq!(
browser_verification_url(&challenge()),
"https://verify.proton.me/?methods=captcha,email&token=ab%2Bc%2Fd%3D"
);
}
/// The browser keeps the page's token, so the retry replays the challenge
/// itself, typed with every method the page offered.
#[test]
fn a_browser_verification_retries_with_the_challenge_token() {
let credential = browser_verified(&challenge());
assert_eq!(credential.token, "ab+c/d=");
assert_eq!(credential.method, "captcha,email");
}
#[test]
fn passphrases_survive_the_blob_encoding() {
let decoded = decode(&encode(&passphrases())).expect("decodes");
assert_eq!(decoded, passphrases());
}
#[test]
fn passphrases_survive_the_kernel_payload_encoding() {
let decoded = decode_secret(&encode_secret(&passphrases())).expect("decodes");
assert_eq!(decoded, passphrases());
}
#[test]
fn corrupt_or_empty_secrets_read_as_absent() {
assert!(decode_secret(b"not json").is_none());
assert!(decode(&BTreeMap::new()).is_none());
assert!(decode(&BTreeMap::from([("k".into(), "!!!".into())])).is_none());
}
#[test]
fn legacy_blob_is_read_but_its_password_is_never_written_back() {
let stored: StoredSession =
serde_json::from_str(&blob(r#","mailbox_password":"hunter2""#)).unwrap();
assert_eq!(stored.mailbox_password, "hunter2");
assert!(stored.key_passphrases.is_none());
let written = serde_json::to_string(&stored).unwrap();
assert!(!written.contains("hunter2"));
assert!(!written.contains("mailbox_password"));
}
#[test]
fn locked_blob_has_no_passphrases_field() {
let stored: StoredSession = serde_json::from_str(&blob("")).unwrap();
assert!(
serde_json::to_string(&stored)
.unwrap()
.find("key_passphrases")
.is_none()
);
}
#[test]
fn stored_passphrases_win_over_the_kernel_keyring() {
let mut stored: StoredSession = serde_json::from_str(&blob("")).unwrap();
stored.key_passphrases = Some(encode(&passphrases()));
let (found, origin) = find_passphrases(&stored).unwrap().unwrap();
assert_eq!(origin, KeyState::Stored);
assert_eq!(found, passphrases());
}
}
+147 -15
View File
@@ -91,7 +91,7 @@ const TOUCH_FLUSH_AT: usize = 256;
/// ///
/// Public because it is the *contract* a block has to satisfy, not a detail of /// Public because it is the *contract* a block has to satisfy, not a detail of
/// this cache: the network read path validates against it too, so a block that /// this cache: the network read path validates against it too, so a block that
/// comes back short is refused before it can truncate a file (bugs.md B84). /// comes back short is refused before it can truncate a file (`docs/BUGS.md` B84).
pub fn block_len(size: u64, idx: u64) -> u64 { pub fn block_len(size: u64, idx: u64) -> u64 {
let start = idx.saturating_mul(BLOCK_SIZE); let start = idx.saturating_mul(BLOCK_SIZE);
size.saturating_sub(start).min(BLOCK_SIZE) size.saturating_sub(start).min(BLOCK_SIZE)
@@ -311,8 +311,7 @@ struct PinFile {
pins: BTreeMap<String, Pin>, pins: BTreeMap<String, Pin>,
} }
/// What a staged file actually contains, written beside it as `<name>.json` /// What a staged file actually contains, written beside it as `<name>.json`.
/// (offline.md Phase 2).
/// ///
/// A staged file is **not** necessarily valid whole-file content, and that is /// A staged file is **not** necessarily valid whole-file content, and that is
/// the whole reason this exists. A partial overwrite commits by filling the /// the whole reason this exists. A partial overwrite commits by filling the
@@ -365,6 +364,14 @@ pub struct StagedWrite {
pub based_on: Option<Baseline>, pub based_on: Option<Baseline>,
} }
impl StagedWrite {
/// Whether `authored` ranges cover a file of `len` bytes, so the staged
/// bytes are its complete new content. An empty file has nothing to cover.
pub fn covers(authored: &[(u64, u64)], len: u64) -> bool {
len == 0 || authored == [(0, len)]
}
}
/// The identity of a remote revision, as far as we can observe it. /// The identity of a remote revision, as far as we can observe it.
/// ///
/// `revision_id` is the authoritative identity when present: the server assigns a /// `revision_id` is the authoritative identity when present: the server assigns a
@@ -431,9 +438,9 @@ pub struct ContentCache {
/// Subdirectory for write-handle scratch files (disk-backed write buffers). /// Subdirectory for write-handle scratch files (disk-backed write buffers).
/// Emptied on open so a crashed run leaves no orphans. /// Emptied on open so a crashed run leaves no orphans.
scratch_dir: PathBuf, scratch_dir: PathBuf,
/// Subdirectory holding the bytes of writes that have not been uploaded yet /// Subdirectory holding the bytes of writes that have not been uploaded yet.
/// (offline.md Phase 2/3) — every released write passes through here, since /// Every released write passes through here, since the upload is a queued
/// the upload is a queued op performed later. Unlike `scratch_dir` this is /// op performed later. Unlike `scratch_dir` this is
/// **never** emptied on open: these are the only copy of content the user /// **never** emptied on open: these are the only copy of content the user
/// authored, and the whole point is that they outlive the daemon. /// authored, and the whole point is that they outlive the daemon.
staging_dir: PathBuf, staging_dir: PathBuf,
@@ -468,7 +475,7 @@ pub struct ContentCache {
thumb_bytes: AtomicU64, thumb_bytes: AtomicU64,
/// Unified metadata DB. Its `cache_entries` table is the LRU index: every /// Unified metadata DB. Its `cache_entries` table is the LRU index: every
/// store/read/evict updates it, and the budget enforcers query it instead of /// store/read/evict updates it, and the budget enforcers query it instead of
/// scanning the cache directories (plan.md P4). /// scanning the cache directories.
db: Arc<Db>, db: Arc<Db>,
/// LRU touches not yet written to `cache_entries`, keyed by cache key. /// LRU touches not yet written to `cache_entries`, keyed by cache key.
/// ///
@@ -730,6 +737,60 @@ impl ContentCache {
Some(buf) Some(buf)
} }
/// Write the whole content of `uid` at `(mtime, size)` to `dst` from the
/// cache alone, and report whether that worked.
///
/// A whole-file blob is cloned. Otherwise every block of the revision has to
/// be cached, under its recorded geometry or, failing that, the uniform one
/// the read path would use. Anything short of the full content leaves
/// nothing at `dst`, so a caller falls back to a download with a clean slate.
///
/// This is what lets a mirror folder take a file an on-demand mount has
/// already read, after a move between them, without fetching it again. The
/// trust is the read path's: the same `(mtime, size)` key, and blocks that
/// were validated when they were fetched.
pub fn copy_cached_to(&self, uid: &NodeUid, mtime: i64, size: u64, dst: &Path) -> bool {
if size == 0 {
return false;
}
if let Some(blob) = self.valid_blob(uid, mtime, size) {
if Self::clone_or_copy(&blob, dst).is_ok_and(|len| len == size) {
self.touch(Self::key(uid));
return true;
}
let _ = std::fs::remove_file(dst);
return false;
}
let geometry = self
.block_geometry(uid, mtime, size)
.unwrap_or_else(|| BlockGeometry::uniform(size));
if geometry.size() != size {
return false;
}
let spans = geometry.spans(0, size);
// Checked up front so a file with one block missing costs stats, not a
// copy of every block before it.
if !spans
.iter()
.all(|&span| self.has_block(uid, mtime, size, span))
{
return false;
}
let written = (|| -> Option<()> {
use std::io::Write;
let mut out = std::fs::File::create(dst).ok()?;
for span in spans {
out.write_all(&self.cached_block(uid, mtime, size, span)?)
.ok()?;
}
(out.metadata().ok()?.len() == size).then_some(())
})();
if written.is_none() {
let _ = std::fs::remove_file(dst);
}
written.is_some()
}
/// Store `bytes` as the cached content for `uid`, tagged with `(mtime, /// Store `bytes` as the cached content for `uid`, tagged with `(mtime,
/// size)`. The blob is written to a temp file then renamed so a concurrent /// size)`. The blob is written to a temp file then renamed so a concurrent
/// reader never sees a partial blob; the meta tag is written last so a /// reader never sees a partial blob; the meta tag is written last so a
@@ -783,6 +844,9 @@ impl ContentCache {
} }
let bytes = std::fs::metadata(&tmp)?.len(); let bytes = std::fs::metadata(&tmp)?.len();
std::fs::rename(&tmp, &blob)?; std::fs::rename(&tmp, &blob)?;
// Linked from the blob already cached, the temp name was a second name
// of it, and the rename did nothing (`docs/BUGS.md` B187).
let _ = std::fs::remove_file(&tmp);
let meta = serde_json::to_vec(&Meta { mtime, size })?; let meta = serde_json::to_vec(&Meta { mtime, size })?;
std::fs::write(self.meta_path(uid), meta)?; std::fs::write(self.meta_path(uid), meta)?;
self.db self.db
@@ -924,7 +988,7 @@ impl ContentCache {
/// a miss ([`cached_block`](Self::cached_block) checks), so a bad block /// a miss ([`cached_block`](Self::cached_block) checks), so a bad block
/// cannot escape this cache even though this does not check. Validating a /// cannot escape this cache even though this does not check. Validating a
/// block is the *reader's* job, at the point it is fetched — see /// block is the *reader's* job, at the point it is fetched — see
/// `Core::read_block` (bugs.md B84). /// `Core::read_block` (`docs/BUGS.md` B84).
pub fn store_block( pub fn store_block(
&self, &self,
uid: &NodeUid, uid: &NodeUid,
@@ -1426,9 +1490,8 @@ impl ContentCache {
/// This is what makes a write survive its upload: the caller is releasing a /// This is what makes a write survive its upload: the caller is releasing a
/// write handle and would otherwise delete the file, so until the bytes are /// write handle and would otherwise delete the file, so until the bytes are
/// on the remote, staging holds the only copy. Every dirty handle goes /// on the remote, staging holds the only copy. Every dirty handle goes
/// through here — the upload is a queued op performed later (offline.md /// through here — the upload is a queued op performed later, and a staged
/// Phase 3), and a staged file is also what a human can recover from if the /// file is also what a human can recover from if the queue never drains.
/// queue never drains.
/// ///
/// Falls back to a copy when the rename crosses a filesystem boundary; a /// Falls back to a copy when the rename crosses a filesystem boundary; a
/// failure here means we could not save the bytes at all, so it is reported /// failure here means we could not save the bytes at all, so it is reported
@@ -2002,6 +2065,17 @@ mod tests {
NodeUid::new(VolumeId::from("vol"), LinkId::from(link)) NodeUid::new(VolumeId::from("vol"), LinkId::from(link))
} }
#[test]
fn an_empty_write_covers_the_file_and_a_gap_does_not() {
assert!(
StagedWrite::covers(&[], 0),
"an empty file has nothing to cover"
);
assert!(StagedWrite::covers(&[(0, 7)], 7));
assert!(!StagedWrite::covers(&[(0, 3), (4, 7)], 7));
assert!(!StagedWrite::covers(&[], 7));
}
/// A unique temp directory removed on drop; avoids a dev-dependency. /// A unique temp directory removed on drop; avoids a dev-dependency.
struct TempDir(PathBuf); struct TempDir(PathBuf);
impl TempDir { impl TempDir {
@@ -2124,6 +2198,24 @@ mod tests {
assert_eq!(c.read_range(&u, 200, 5, 0, 5).unwrap(), b"newer"); assert_eq!(c.read_range(&u, 200, 5, 0, 5).unwrap(), b"newer");
} }
/// A landed create and the revision after it can adopt the same staged
/// blob. The link to the temp name is then a second name of the cached
/// blob, and a rename between two names of one file does nothing: the
/// temp name stayed until the next start swept it (`docs/BUGS.md` B187).
#[test]
fn store_file_twice_from_the_same_blob_leaves_no_temp_file() {
let (c, d) = cache();
let u = uid("a");
let src = d.path().join("staged-blob");
std::fs::write(&src, b"staged").unwrap();
c.store_file(&u, 100, 6, &src).unwrap();
c.store_file(&u, 100, 6, &src).unwrap();
assert!(!c.blob_path(&u).with_extension("tmp").exists());
assert_eq!(c.read_range(&u, 100, 6, 0, 6).unwrap(), b"staged");
}
#[test] #[test]
fn reconcile_rebuilds_index_from_disk() { fn reconcile_rebuilds_index_from_disk() {
let dir = TempDir::new(); let dir = TempDir::new();
@@ -2561,6 +2653,46 @@ mod tests {
assert!(c.cached_block(&u, 100, 5000, blk(5000, 0)).is_none()); assert!(c.cached_block(&u, 100, 5000, blk(5000, 0)).is_none());
} }
#[test]
fn a_cached_blob_is_copied_out_whole() {
let (c, d) = cache();
let u = uid("blob");
c.store(&u, 7, 5, b"hello").unwrap();
let dst = d.path().join("out");
assert!(c.copy_cached_to(&u, 7, 5, &dst));
assert_eq!(std::fs::read(&dst).unwrap(), b"hello");
// Another revision is not this one.
assert!(!c.copy_cached_to(&u, 8, 5, &d.path().join("stale")));
assert!(!d.path().join("stale").exists());
}
#[test]
fn cached_blocks_are_copied_out_in_order() {
let (c, d) = cache();
let u = uid("blocks");
let size = BLOCK_SIZE + 3;
let first = vec![1_u8; BLOCK_SIZE as usize];
c.store_block(&u, 7, size, blk(size, 0), &first).unwrap();
c.store_block(&u, 7, size, blk(size, 1), b"end").unwrap();
let dst = d.path().join("out");
assert!(c.copy_cached_to(&u, 7, size, &dst));
let bytes = std::fs::read(&dst).unwrap();
assert_eq!(bytes.len() as u64, size);
assert_eq!(&bytes[..BLOCK_SIZE as usize], first.as_slice());
assert_eq!(&bytes[BLOCK_SIZE as usize..], b"end");
}
#[test]
fn a_file_with_a_block_missing_is_not_copied_out() {
let (c, d) = cache();
let u = uid("partial");
let size = BLOCK_SIZE + 3;
c.store_block(&u, 7, size, blk(size, 1), b"end").unwrap();
let dst = d.path().join("out");
assert!(!c.copy_cached_to(&u, 7, size, &dst));
assert!(!dst.exists(), "a partial copy must not be left behind");
}
/// The [`BlockSpan`] the uniform geometry gives block `idx` of a file of /// The [`BlockSpan`] the uniform geometry gives block `idx` of a file of
/// `size` bytes — the shape these tests were written against, and the one /// `size` bytes — the shape these tests were written against, and the one
/// this client's own uploads still have. /// this client's own uploads still have.
@@ -2573,7 +2705,7 @@ mod tests {
} }
/// The geometry the whole read path validates against, here and over the /// The geometry the whole read path validates against, here and over the
/// network (bugs.md B84): full blocks, then whatever is left. /// network (`docs/BUGS.md` B84): full blocks, then whatever is left.
#[test] #[test]
fn block_len_is_full_blocks_then_the_remainder() { fn block_len_is_full_blocks_then_the_remainder() {
let size = BLOCK_SIZE * 2 + 17; let size = BLOCK_SIZE * 2 + 17;
@@ -2653,7 +2785,7 @@ mod tests {
/// A cached block is addressed by the range it covers, not by its index /// A cached block is addressed by the range it covers, not by its index
/// alone. Re-planning a file onto its real geometry must therefore miss the /// alone. Re-planning a file onto its real geometry must therefore miss the
/// blocks the uniform fallback cached, rather than serve their bytes at the /// blocks the uniform fallback cached, rather than serve their bytes at the
/// wrong offset (bugs.md B85). /// wrong offset (`docs/BUGS.md` B85).
#[test] #[test]
fn a_cached_block_is_not_served_to_a_different_geometry() { fn a_cached_block_is_not_served_to_a_different_geometry() {
let (c, _d) = cache(); let (c, _d) = cache();
@@ -2720,7 +2852,7 @@ mod tests {
/// A block of the wrong length cannot escape the cache even though the /// A block of the wrong length cannot escape the cache even though the
/// store does not check: the read validates it and reports a miss, so the /// store does not check: the read validates it and reports a miss, so the
/// next read refetches rather than serving a hole (bugs.md B84). /// next read refetches rather than serving a hole (`docs/BUGS.md` B84).
#[test] #[test]
fn a_block_of_the_wrong_length_reads_as_a_miss() { fn a_block_of_the_wrong_length_reads_as_a_miss() {
let (c, _d) = cache(); let (c, _d) = cache();
@@ -2785,7 +2917,7 @@ mod tests {
); );
} }
/// A failed upload must never cost the user their bytes (offline.md Phase 2): /// A failed upload must never cost the user their bytes:
/// the scratch file moves to staging intact, and staging survives a reopen /// the scratch file moves to staging intact, and staging survives a reopen
/// the way scratch deliberately does not. /// the way scratch deliberately does not.
#[test] #[test]
+117 -36
View File
@@ -3,7 +3,7 @@
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use directories::ProjectDirs; use directories::ProjectDirs;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Deserializer, Serialize};
use crate::error::{Error, Result}; use crate::error::{Error, Result};
use crate::menu::PromptConfig; use crate::menu::PromptConfig;
@@ -12,8 +12,19 @@ use crate::syncignore::DEFAULT_IGNORE_PATTERNS;
/// Proton requires `external-drive-{name}@{semver}-{channel}` (channel ∈ /// Proton requires `external-drive-{name}@{semver}-{channel}` (channel ∈
/// stable/beta/alpha); a malformed value trips the 422 anti-abuse path. /// stable/beta/alpha); a malformed value trips the 422 anti-abuse path.
pub const APP_VERSION: &str = "external-drive-linux@0.1.0-alpha"; /// Follows the release so Proton can tell releases apart and retire old ones
pub const USER_AGENT: &str = "proton-drive-linux/0.1.0"; /// without retiring every one.
pub const APP_VERSION: &str = concat!(
"external-drive-linux@",
env!("CARGO_PKG_VERSION"),
"-stable"
);
pub const USER_AGENT: &str = concat!("proton-drive-linux/", env!("CARGO_PKG_VERSION"));
/// The fixed identity every release up to 3.1.0 wrote into `config.json`. Read
/// back as "unset" so those configs follow the release instead of pinning it.
const LEGACY_APP_VERSION: &str = "external-drive-linux@0.1.0-alpha";
const LEGACY_USER_AGENT: &str = "proton-drive-linux/0.1.0";
/// Keyring service name; one entry per credential kind keyed by username. /// Keyring service name; one entry per credential kind keyed by username.
pub const KEYRING_SERVICE: &str = "proton-drive-linux"; pub const KEYRING_SERVICE: &str = "proton-drive-linux";
@@ -76,10 +87,24 @@ fn resolve_sweep_mode(env: Option<&str>, stored: Option<SweepMode>) -> SweepMode
} }
/// Configuration structure allowing the user to customize client identification headers. /// Configuration structure allowing the user to customize client identification headers.
#[derive(Serialize, Deserialize, Debug, Clone)] #[derive(Serialize, Deserialize, Debug, Clone, Default)]
pub struct AppConfig { pub struct AppConfig {
pub app_version: String, /// Override for [`APP_VERSION`]. `None` means "use the release's own";
pub user_agent: String, /// the old fixed default reads as `None`, so it leaves the file on the
/// next save.
#[serde(
default,
deserialize_with = "identity_override",
skip_serializing_if = "Option::is_none"
)]
pub app_version: Option<String>,
/// Override for [`USER_AGENT`], handled like `app_version`.
#[serde(
default,
deserialize_with = "identity_override",
skip_serializing_if = "Option::is_none"
)]
pub user_agent: Option<String>,
/// Soft cap on the on-disk content cache, in bytes (`0` = unlimited). /// Soft cap on the on-disk content cache, in bytes (`0` = unlimited).
/// `None` means "use [`DEFAULT_CACHE_BUDGET_BYTES`]"; the Settings page /// `None` means "use [`DEFAULT_CACHE_BUDGET_BYTES`]"; the Settings page
/// writes an explicit value here. Defaulted for configs predating the field. /// writes an explicit value here. Defaulted for configs predating the field.
@@ -154,6 +179,13 @@ pub struct AppConfig {
/// configs predating the field. /// configs predating the field.
#[serde(default)] #[serde(default)]
pub online_map: bool, pub online_map: bool,
/// Record a new folder, a new file, a rename or a delete through the mount
/// locally and send it from the queue, instead of waiting for Drive in the
/// syscall. `None` means on. `false` brings back the old path for one
/// release (docs/ROADMAP.md). Defaulted for configs predating
/// the field.
#[serde(default)]
pub local_first: Option<bool>,
} }
/// The My Files page's view choices, remembered between runs. /// The My Files page's view choices, remembered between runs.
@@ -209,30 +241,28 @@ impl FileSort {
} }
} }
impl Default for AppConfig { /// Reads an `app_version` or `user_agent` override, dropping the fixed
fn default() -> Self { /// identity older releases stored as the default.
Self { fn identity_override<'de, D: Deserializer<'de>>(
app_version: APP_VERSION.to_string(), deserializer: D,
user_agent: USER_AGENT.to_string(), ) -> std::result::Result<Option<String>, D::Error> {
cache_budget: None, let value = Option::<String>::deserialize(deserializer)?;
upload_limit: None, Ok(value.filter(|v| v != LEGACY_APP_VERSION && v != LEGACY_USER_AGENT))
download_limit: None,
mountpoint: None,
ignore_patterns: None,
device_uid: None,
conflict_sweep: None,
open_with: None,
prompt: None,
proton_theme: None,
files_view: FilesView::default(),
tray_hidden: false,
language: None,
online_map: false,
}
}
} }
impl AppConfig { impl AppConfig {
/// The `x-pm-appversion` sent to Proton: the user's override, or
/// [`APP_VERSION`].
pub fn resolved_app_version(&self) -> &str {
self.app_version.as_deref().unwrap_or(APP_VERSION)
}
/// The `User-Agent` sent to Proton: the user's override, or
/// [`USER_AGENT`].
pub fn resolved_user_agent(&self) -> &str {
self.user_agent.as_deref().unwrap_or(USER_AGENT)
}
/// The effective cache budget in bytes: the user's explicit choice, or /// The effective cache budget in bytes: the user's explicit choice, or
/// [`DEFAULT_CACHE_BUDGET_BYTES`] when unset. /// [`DEFAULT_CACHE_BUDGET_BYTES`] when unset.
pub fn resolved_cache_budget(&self) -> u64 { pub fn resolved_cache_budget(&self) -> u64 {
@@ -424,7 +454,7 @@ impl AppDirs {
/// typical umask makes them `0755`. Until this was fixed, the only thing /// typical umask makes them `0755`. Until this was fixed, the only thing
/// keeping another local user out was `~/.cache` and `~/.local/state` /// keeping another local user out was `~/.cache` and `~/.local/state`
/// happening to be `0700`, which is a convention of the user's system and /// happening to be `0700`, which is a convention of the user's system and
/// not something this client established (bugs.md B6). /// not something this client established (`docs/BUGS.md` B6).
/// ///
/// Applied on every start, not just at creation: a directory that already /// Applied on every start, not just at creation: a directory that already
/// exists with a permissive mode — restored from a backup that flattened /// exists with a permissive mode — restored from a backup that flattened
@@ -475,7 +505,7 @@ fn restrict_dir(dir: &Path) -> Result<()> {
/// session — enumerate, read, upload, trash, share — without presenting any /// session — enumerate, read, upload, trash, share — without presenting any
/// credential, since the daemon already holds one. It is an authority boundary, /// credential, since the daemon already holds one. It is an authority boundary,
/// not merely private data, and `UnixListener::bind` applies the umask like any /// not merely private data, and `UnixListener::bind` applies the umask like any
/// other file (bugs.md B6). /// other file (`docs/BUGS.md` B6).
/// ///
/// There is a window between `bind` and this call during which the socket /// There is a window between `bind` and this call during which the socket
/// carries the umask's mode. It is closed in practice by the containing /// carries the umask's mode. It is closed in practice by the containing
@@ -496,7 +526,7 @@ mod tests {
std::fs::metadata(p).unwrap().permissions().mode() & 0o777 std::fs::metadata(p).unwrap().permissions().mode() & 0o777
} }
/// bugs.md B6. A permissive mode here is not a cosmetic problem: these /// `docs/BUGS.md` B6. A permissive mode here is not a cosmetic problem: these
/// directories hold decrypted content, a plaintext index of every node name, /// directories hold decrypted content, a plaintext index of every node name,
/// and the control socket that commands the daemon's session. /// and the control socket that commands the daemon's session.
#[test] #[test]
@@ -563,8 +593,8 @@ mod tests {
#[test] #[test]
fn test_app_config_serialization() { fn test_app_config_serialization() {
let config = AppConfig { let config = AppConfig {
app_version: "external-drive-test-client@1.0.0".to_string(), app_version: Some("external-drive-test-client@1.0.0".to_string()),
user_agent: "test-agent/1.0".to_string(), user_agent: Some("test-agent/1.0".to_string()),
cache_budget: Some(1234), cache_budget: Some(1234),
upload_limit: Some(500_000), upload_limit: Some(500_000),
download_limit: None, download_limit: None,
@@ -584,11 +614,15 @@ mod tests {
tray_hidden: true, tray_hidden: true,
language: Some("de".to_string()), language: Some("de".to_string()),
online_map: true, online_map: true,
local_first: Some(false),
}; };
let json = serde_json::to_string(&config).unwrap(); let json = serde_json::to_string(&config).unwrap();
let decoded: AppConfig = serde_json::from_str(&json).unwrap(); let decoded: AppConfig = serde_json::from_str(&json).unwrap();
assert_eq!(decoded.app_version, "external-drive-test-client@1.0.0"); assert_eq!(
assert_eq!(decoded.user_agent, "test-agent/1.0"); decoded.resolved_app_version(),
"external-drive-test-client@1.0.0"
);
assert_eq!(decoded.resolved_user_agent(), "test-agent/1.0");
assert_eq!(decoded.proton_theme, Some(true)); assert_eq!(decoded.proton_theme, Some(true));
assert!(decoded.tray_hidden); assert!(decoded.tray_hidden);
assert!(decoded.online_map); assert!(decoded.online_map);
@@ -619,8 +653,55 @@ mod tests {
#[test] #[test]
fn test_default_app_config() { fn test_default_app_config() {
let default_config = AppConfig::default(); let default_config = AppConfig::default();
assert_eq!(default_config.app_version, APP_VERSION); assert_eq!(default_config.resolved_app_version(), APP_VERSION);
assert_eq!(default_config.user_agent, USER_AGENT); assert_eq!(default_config.resolved_user_agent(), USER_AGENT);
}
#[test]
fn the_client_identity_follows_the_release() {
let version = env!("CARGO_PKG_VERSION");
assert_eq!(
APP_VERSION,
format!("external-drive-linux@{version}-stable")
);
assert_eq!(USER_AGENT, format!("proton-drive-linux/{version}"));
}
#[test]
fn an_old_config_with_the_fixed_identity_follows_the_release() {
// Every config written up to 3.1.0 holds these two values, so they must
// not pin the identity once Proton retires 0.1.0.
let json = format!(
r#"{{"app_version":"{LEGACY_APP_VERSION}","user_agent":"{LEGACY_USER_AGENT}","tray_hidden":true}}"#
);
let decoded: AppConfig = serde_json::from_str(&json).unwrap();
assert_eq!(decoded.resolved_app_version(), APP_VERSION);
assert_eq!(decoded.resolved_user_agent(), USER_AGENT);
assert!(decoded.tray_hidden, "the rest of the config survives");
let saved = serde_json::to_string(&decoded).unwrap();
assert!(!saved.contains("app_version"), "{saved}");
assert!(!saved.contains("user_agent"), "{saved}");
}
#[test]
fn a_hand_set_client_identity_is_kept() {
let json = r#"{"app_version":"external-drive-linux@9.9.9-beta","user_agent":"custom/1"}"#;
let decoded: AppConfig = serde_json::from_str(json).unwrap();
assert_eq!(
decoded.resolved_app_version(),
"external-drive-linux@9.9.9-beta"
);
assert_eq!(decoded.resolved_user_agent(), "custom/1");
let saved = serde_json::to_string(&decoded).unwrap();
assert!(saved.contains("external-drive-linux@9.9.9-beta"), "{saved}");
}
#[test]
fn a_config_without_a_client_identity_loads() {
let decoded: AppConfig = serde_json::from_str("{}").unwrap();
assert_eq!(decoded.resolved_app_version(), APP_VERSION);
assert_eq!(decoded.resolved_user_agent(), USER_AGENT);
} }
#[test] #[test]
+107 -8
View File
@@ -339,7 +339,10 @@ pub enum Request {
/// single path component (no separators). Replies with [`Response::Ok`]. /// single path component (no separators). Replies with [`Response::Ok`].
Rename { path: String, new_name: String }, Rename { path: String, new_name: String },
/// Move a file or folder into a new parent folder. Both `path` and /// Move a file or folder into a new parent folder. Both `path` and
/// `new_parent` are mountpoint-relative. Replies with [`Response::Ok`]. /// `new_parent` are mountpoint-relative, or both absolute local paths; two
/// absolute paths may lie in different locations (My files, an on-demand
/// or a mirror synced folder), and the move is still made on Proton Drive
/// without transferring content. Replies with [`Response::Ok`].
Move { path: String, new_parent: String }, Move { path: String, new_parent: String },
/// Trash a file or folder. `path` is mountpoint-relative. Replies with /// Trash a file or folder. `path` is mountpoint-relative. Replies with
/// [`Response::Ok`]. /// [`Response::Ok`].
@@ -440,11 +443,11 @@ pub enum Request {
DeleteDevice { uid: String }, DeleteDevice { uid: String },
/// Adopt an existing device as this machine's, pinning its uid in /// Adopt an existing device as this machine's, pinning its uid in
/// `config.json` so a hostname change or reinstall re-attaches to it instead /// `config.json` so a hostname change or reinstall re-attaches to it instead
/// of registering a duplicate (features.md 5.1). `uid: None` clears the pin. /// of registering a duplicate. `uid: None` clears the pin.
/// Replies with [`Response::Ok`]. /// Replies with [`Response::Ok`].
AdoptDevice { uid: Option<String> }, AdoptDevice { uid: Option<String> },
// ---- device folder sync (devices.md) ---------------------------------- // ---- device folder sync ----------------------------------
/// Add a local folder to this machine's device, uploading its tree and /// Add a local folder to this machine's device, uploading its tree and
/// registering the device on first use. Replies with [`Response::Ok`]. /// registering the device on first use. Replies with [`Response::Ok`].
AddSyncFolder { local_path: String }, AddSyncFolder { local_path: String },
@@ -474,6 +477,15 @@ pub enum Request {
/// Retry a backed-off queued op now instead of waiting out its backoff; /// Retry a backed-off queued op now instead of waiting out its backoff;
/// every failed op when `id` is `None`. Replies with [`Response::Ok`]. /// every failed op when `id` is `None`. Replies with [`Response::Ok`].
RetryPendingOp { id: Option<i64> }, RetryPendingOp { id: Option<i64> },
/// Copy the content of a queued upload (`id` from
/// [`Request::ListPendingOps`]) to `dest`, an absolute path that must not
/// exist yet. The op stays queued. Replies with [`Response::Ok`].
ExportPendingOp { id: i64, dest: String },
/// Drop a queued change (`id` from [`Request::ListPendingOps`]) and undo
/// it locally: a node made here goes, a node Drive has goes back to what
/// Drive has, along with every other change queued on it. Its content is
/// lost unless it was exported first. Replies with [`Response::Ok`].
DiscardPendingOp { id: i64 },
/// List the `(sync-conflict …)` copies under My Files and in synced /// List the `(sync-conflict …)` copies under My Files and in synced
/// folders. Replies with /// folders. Replies with
/// [`Response::Conflicts`]. /// [`Response::Conflicts`].
@@ -484,7 +496,7 @@ pub enum Request {
/// [`Response::Ok`]. /// [`Response::Ok`].
ResolveConflict { path: String, keep: ConflictKeep }, ResolveConflict { path: String, keep: ConflictKeep },
/// List the folders under this machine's device that can be synced here, /// List the folders under this machine's device that can be synced here,
/// each with a proposed local path (features.md 5.2). Replies with /// each with a proposed local path. Replies with
/// [`Response::RestorableFolders`]. /// [`Response::RestorableFolders`].
ListRestorableFolders, ListRestorableFolders,
/// Attach the given remote device folders to local paths and sync them down. /// Attach the given remote device folders to local paths and sync them down.
@@ -877,6 +889,59 @@ pub struct PendingOpInfo {
pub parked: bool, pub parked: bool,
/// Failed often enough to count as stuck. /// Failed often enough to count as stuck.
pub failing: bool, pub failing: bool,
/// Why Drive refused it, when it did; such an op needs attention at once.
#[serde(default)]
pub issue: Option<SyncIssue>,
/// Carries file content that [`Request::ExportPendingOp`] can save.
#[serde(default)]
pub exportable: bool,
}
/// Why Drive refused a queued change (in [`PendingOpInfo`]). The change stays
/// queued and keeps being retried; nothing is deleted.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum SyncIssue {
/// The account's storage is full.
Quota,
/// The account may no longer change the file or its folder, for example
/// because a share was revoked.
Access,
/// The file or the folder it goes into is gone from Drive.
Missing,
/// The folder holds too many items, or is nested too deep.
Limit,
/// Drive rejected the change, for example a name it does not accept.
Rejected,
/// An issue from a newer daemon.
#[serde(other)]
Unknown,
}
impl SyncIssue {
/// The name stored in the database and sent on the wire.
pub fn as_str(self) -> &'static str {
match self {
SyncIssue::Quota => "quota",
SyncIssue::Access => "access",
SyncIssue::Missing => "missing",
SyncIssue::Limit => "limit",
SyncIssue::Rejected => "rejected",
SyncIssue::Unknown => "unknown",
}
}
/// The issue stored as `name`.
pub fn parse(name: &str) -> Self {
match name {
"quota" => SyncIssue::Quota,
"access" => SyncIssue::Access,
"missing" => SyncIssue::Missing,
"limit" => SyncIssue::Limit,
"rejected" => SyncIssue::Rejected,
_ => SyncIssue::Unknown,
}
}
} }
/// One synced local folder on this machine's device (in [`Response::SyncFolders`]). /// One synced local folder on this machine's device (in [`Response::SyncFolders`]).
@@ -1231,6 +1296,10 @@ pub struct DirEntry {
/// not known, and for anything not in the trash. /// not known, and for anything not in the trash.
#[serde(default)] #[serde(default)]
pub trashed_from: Option<String>, pub trashed_from: Option<String>,
/// Why Drive refused a change queued on the node, for a [`Request::ListDir`]
/// entry; `None` otherwise.
#[serde(default)]
pub issue: Option<SyncIssue>,
} }
/// One hit in a [`Request::Search`] result. Like [`DirEntry`] but carries the /// One hit in a [`Request::Search`] result. Like [`DirEntry`] but carries the
@@ -1834,17 +1903,17 @@ pub enum Response {
/// The pin registry. /// The pin registry.
pins: Vec<Pin>, pins: Vec<Pin>,
/// False when the daemon is serving the cached tree because the API is /// False when the daemon is serving the cached tree because the API is
/// unreachable (offline.md Phase 1). Cached and pinned content still /// unreachable. Cached and pinned content still
/// reads; anything else fails until the network is back. /// reads; anything else fails until the network is back.
#[serde(default = "default_online")] #[serde(default = "default_online")]
online: bool, online: bool,
/// Writes accepted locally but not yet uploaded (offline.md Phase 3). /// Writes accepted locally but not yet uploaded.
/// Non-zero means the mount is ahead of the remote — either a copy is /// Non-zero means the mount is ahead of the remote — either a copy is
/// still draining, or it cannot drain because we are offline. /// still draining, or it cannot drain because we are offline.
#[serde(default)] #[serde(default)]
pending_uploads: u64, pending_uploads: u64,
/// Queued mutations that carry no bytes: `mkdir`, `rename`, `trash` /// Queued mutations that carry no bytes: `mkdir`, `rename`, `trash`.
/// (offline.md Phase 3b). Counted apart from `pending_uploads` because /// Counted apart from `pending_uploads` because
/// calling a queued `mkdir` an upload is a lie. /// calling a queued `mkdir` an upload is a lie.
#[serde(default)] #[serde(default)]
pending_changes: u64, pending_changes: u64,
@@ -3287,6 +3356,17 @@ mod tests {
}, },
r#"{"SaveRevisionAsByUid":{"uid":"vol~link","revision_id":"rev-3","dest":"/tmp/out.bin"}}"#, r#"{"SaveRevisionAsByUid":{"uid":"vol~link","revision_id":"rev-3","dest":"/tmp/out.bin"}}"#,
), ),
(
Request::ExportPendingOp {
id: 7,
dest: "/tmp/out.bin".into(),
},
r#"{"ExportPendingOp":{"id":7,"dest":"/tmp/out.bin"}}"#,
),
(
Request::DiscardPendingOp { id: 7 },
r#"{"DiscardPendingOp":{"id":7}}"#,
),
]; ];
for (request, wire) in cases { for (request, wire) in cases {
assert_eq!(serde_json::to_string(&request).unwrap(), wire); assert_eq!(serde_json::to_string(&request).unwrap(), wire);
@@ -3295,6 +3375,25 @@ mod tests {
} }
} }
/// A daemon that predates sync issues sends neither field, and a newer one
/// may send an issue this build does not know.
#[test]
fn a_queued_op_from_another_daemon_version_still_decodes() {
let old = r#"{"id":1,"kind":"create","path":"a.txt","attempts":0,"last_error":null,
"queued_at":5,"next_attempt_at":5,"parked":false,"failing":false}"#;
let op: PendingOpInfo = serde_json::from_str(old).unwrap();
assert_eq!(op.issue, None);
assert!(!op.exportable);
let newer = old.replace(r#""failing":false"#, r#""failing":true,"issue":"embargo""#);
let op: PendingOpInfo = serde_json::from_str(&newer).unwrap();
assert_eq!(op.issue, Some(SyncIssue::Unknown));
assert_eq!(
SyncIssue::parse(SyncIssue::Quota.as_str()),
SyncIssue::Quota
);
}
/// A front-end that predates the favorites filter sends no `favorites` /// A front-end that predates the favorites filter sends no `favorites`
/// field, and must keep getting the whole timeline rather than an empty page. /// field, and must keep getting the whole timeline rather than an empty page.
#[test] #[test]
+249
View File
@@ -0,0 +1,249 @@
/* WARNING: Script requires that SQLITE_DBCONFIG_DEFENSIVE be disabled */
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE sync_state (key TEXT PRIMARY KEY, value TEXT);
INSERT INTO sync_state VALUES('schema_version','35');
CREATE TABLE nodes (
uid TEXT PRIMARY KEY,
parent_uid TEXT,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER,
mtime INTEGER NOT NULL,
revision_hash TEXT,
trashed INTEGER NOT NULL DEFAULT 0
, node_json TEXT, listed INTEGER NOT NULL DEFAULT 0, path TEXT);
INSERT INTO nodes VALUES('vol~root',NULL,'My Files',1,0,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"root"},"parent_uid":null,"kind":"Folder","name":"My Files","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'');
INSERT INTO nodes VALUES('vol~docs','vol~root','Docs',1,0,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"docs"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":"Folder","name":"Docs","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs');
INSERT INTO nodes VALUES('vol~report','vol~docs','report.txt',0,10,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"report"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":20,"active_revision_state":null,"active_revision_id":null,"claimed_size":10,"claimed_modification_time":null,"content_sha1":null}},"name":"report.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/report.txt');
INSERT INTO nodes VALUES('vol~old','vol~docs','old.txt',0,3,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"old"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":13,"active_revision_state":null,"active_revision_id":null,"claimed_size":3,"claimed_modification_time":null,"content_sha1":null}},"name":"old.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/old.txt');
INSERT INTO nodes VALUES('vol~shared','vol~docs','shared.txt',0,4,200,NULL,0,'{"uid":{"volume_id":"vol","link_id":"shared"},"parent_uid":{"volume_id":"vol","link_id":"docs"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":14,"active_revision_state":null,"active_revision_id":null,"claimed_size":4,"claimed_modification_time":null,"content_sha1":null}},"name":"shared.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Docs/shared.txt');
INSERT INTO nodes VALUES('local~dir','vol~root','Offline folder',1,0,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"dir"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":"Folder","name":"Offline folder","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',1,'Offline folder');
INSERT INTO nodes VALUES('local~inner','local~dir','inner.txt',0,5,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"inner"},"parent_uid":{"volume_id":"local","link_id":"dir"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":15,"active_revision_state":null,"active_revision_id":null,"claimed_size":5,"claimed_modification_time":null,"content_sha1":null}},"name":"inner.txt","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'Offline folder/inner.txt');
INSERT INTO nodes VALUES('local~part','vol~root','movie.part',0,7,200,NULL,0,'{"uid":{"volume_id":"local","link_id":"part"},"parent_uid":{"volume_id":"vol","link_id":"root"},"kind":{"File":{"media_type":"text/plain","total_size_on_storage":17,"active_revision_state":null,"active_revision_id":null,"claimed_size":7,"claimed_modification_time":null,"content_sha1":null}},"name":"movie.part","creation_time":100,"modification_time":200,"trashed":false,"is_shared":false,"is_shared_publicly":false,"signature_email":null,"membership":null,"direct_role":null,"share_id":null,"photo":null,"album":null,"verification":{"name":"NotSigned","passphrase":"NotSigned","content_key":null,"extended_attributes":null}}',0,'movie.part');
CREATE TABLE cache_entries (
cache_key TEXT PRIMARY KEY,
size_bytes INTEGER,
last_accessed INTEGER,
is_pinned INTEGER NOT NULL DEFAULT 0
, kind TEXT NOT NULL DEFAULT 'blob');
CREATE TABLE pins (
uid TEXT PRIMARY KEY,
path TEXT NOT NULL,
recursive INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE local_files (
id INTEGER PRIMARY KEY,
path TEXT NOT NULL UNIQUE,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER NOT NULL DEFAULT 0,
mtime INTEGER NOT NULL DEFAULT 0,
scan_gen INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE photos (
uid TEXT PRIMARY KEY,
capture_time INTEGER NOT NULL,
name TEXT,
ratio REAL,
thumb_state INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL
, media_type TEXT, kind INTEGER NOT NULL DEFAULT 0, favorite INTEGER NOT NULL DEFAULT 0, content_hash TEXT, main_uid TEXT, group_key TEXT, resolved_at INTEGER, latitude REAL, longitude REAL, place_id INTEGER, similar_hash INTEGER);
CREATE TABLE trash (
uid TEXT PRIMARY KEY,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER NOT NULL DEFAULT 0,
mtime INTEGER NOT NULL DEFAULT 0
, parent_uid TEXT, trashed_at INTEGER);
CREATE TABLE device (
uid TEXT PRIMARY KEY,
share_id TEXT NOT NULL,
root_uid TEXT NOT NULL,
name TEXT NOT NULL,
created INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE sync_folder (
id INTEGER PRIMARY KEY,
local_path TEXT NOT NULL UNIQUE,
remote_uid TEXT NOT NULL,
remote_share_id TEXT NOT NULL,
mode TEXT NOT NULL DEFAULT 'mirror',
state TEXT NOT NULL DEFAULT 'idle',
last_sync INTEGER NOT NULL DEFAULT 0
, pending_mode TEXT);
CREATE TABLE sync_entry (
folder_id INTEGER NOT NULL,
rel_path TEXT NOT NULL,
remote_uid TEXT,
local_mtime INTEGER NOT NULL DEFAULT 0,
local_size INTEGER NOT NULL DEFAULT 0,
remote_hash TEXT,
remote_rev TEXT, local_mtime_ns INTEGER,
PRIMARY KEY (folder_id, rel_path)
);
CREATE TABLE activity (
id INTEGER PRIMARY KEY,
time INTEGER NOT NULL,
kind TEXT NOT NULL,
target TEXT NOT NULL,
detail TEXT NOT NULL DEFAULT '',
ok INTEGER NOT NULL DEFAULT 1
);
CREATE TABLE pending_op (
id INTEGER PRIMARY KEY,
kind TEXT NOT NULL,
uid TEXT NOT NULL,
blob_path TEXT,
meta_json TEXT,
created_at INTEGER NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
next_attempt_at INTEGER NOT NULL DEFAULT 0
, parent_uid TEXT, name TEXT, claimed_at INTEGER NOT NULL DEFAULT 0, access_deferred_since INTEGER NOT NULL DEFAULT 0);
INSERT INTO pending_op VALUES(1,'mkdir','local~dir',NULL,NULL,1700000000000,0,NULL,0,'vol~root','Offline folder',4611686018427387903,0);
INSERT INTO pending_op VALUES(2,'create','local~inner','/blobs/inner','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,0,NULL,0,'local~dir','inner.txt',0,0);
INSERT INTO pending_op VALUES(3,'create','local~part','/blobs/part','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,0,NULL,8000000000000,'vol~root','movie.part',0,0);
INSERT INTO pending_op VALUES(4,'revision','vol~report','/blobs/report','{"len":5,"mtime":1700000000,"based_on":null}',1700000000000,2,'network unreachable',1700000900000,NULL,NULL,0,0);
INSERT INTO pending_op VALUES(5,'rename','vol~shared',NULL,'{"original_parent_uid":"vol~docs","original_name":"shared.txt"}',1700000000000,0,NULL,1700000700000,'vol~root','shared-moved.txt',0,1700000100000);
INSERT INTO pending_op VALUES(6,'trash','vol~old',NULL,NULL,1700000000000,0,NULL,0,NULL,NULL,0,0);
CREATE TABLE IF NOT EXISTS 'local_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
INSERT INTO local_fts_data VALUES(1,x'000000');
INSERT INTO local_fts_data VALUES(10,x'00000000000000');
CREATE TABLE IF NOT EXISTS 'local_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE IF NOT EXISTS 'local_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE IF NOT EXISTS 'local_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
INSERT INTO local_fts_config VALUES('version',4);
CREATE TABLE share_access (
root_uid TEXT PRIMARY KEY,
access TEXT NOT NULL
CHECK (access IN ('owner', 'editor', 'viewer', 'unknown'))
);
CREATE TABLE mount (
id INTEGER PRIMARY KEY,
kind TEXT NOT NULL
CHECK (kind IN ('myfiles', 'device', 'shared')),
sync_folder_id INTEGER REFERENCES sync_folder(id) ON DELETE CASCADE,
share_root_uid TEXT,
local_path TEXT NOT NULL DEFAULT '',
root_uid TEXT NOT NULL DEFAULT '',
root_share_id TEXT NOT NULL DEFAULT '',
mode TEXT NOT NULL DEFAULT 'unknown',
access TEXT NOT NULL DEFAULT 'rw'
CHECK (access IN ('rw', 'ro')),
CHECK (
(kind = 'myfiles' AND sync_folder_id IS NULL AND share_root_uid IS NULL)
OR
(kind = 'device' AND sync_folder_id IS NOT NULL AND share_root_uid IS NULL)
OR
(kind = 'shared' AND sync_folder_id IS NULL AND share_root_uid IS NOT NULL)
)
);
CREATE TABLE albums (
uid TEXT PRIMARY KEY,
name TEXT NOT NULL,
photo_count INTEGER NOT NULL DEFAULT 0,
cover_uid TEXT,
last_activity INTEGER,
shared INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL
);
CREATE TABLE album_photos (
album_uid TEXT NOT NULL,
uid TEXT NOT NULL,
capture_time INTEGER NOT NULL,
name TEXT,
media_type TEXT,
kind INTEGER NOT NULL DEFAULT 0,
ratio REAL,
thumb_state INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL,
PRIMARY KEY (album_uid, uid)
);
CREATE TABLE IF NOT EXISTS 'nodes_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
INSERT INTO nodes_fts_data VALUES(1,x'083850');
INSERT INTO nodes_fts_data VALUES(10,x'000000000108080008010101020101030101040101050101060101070101080101');
INSERT INTO nodes_fts_data VALUES(137438953473,x'000000340430206669010204010366696c0102050103696c6501020601036c657301020701036d79200102020103792066010203040808080808');
INSERT INTO nodes_fts_data VALUES(274877906945,x'0000001a0430646f6302080201010201036f6373020803010103040b');
INSERT INTO nodes_fts_data VALUES(412316860417,x'0000008b04302e747803080801010d01032f72650306010106010363732f03060101040103646f630306010102010365706f03080301010801036f637303060101030202727403080501010a0103706f7203080401010901037265700308020101070202742e03080601010b0103732f7203060101050103742e7403080701010c0202787403080901010e040b0a0a0a0b0a0a0b0b0a0a0b');
INSERT INTO nodes_fts_data VALUES(549755813889,x'0000006b04302e747804080501010a01032f6f6c0406010106010363732f04060101040103642e7404080401010902026f63040601010201036c642e04080301010801036f6373040601010302026c640408020101070103732f6f0406010105010374787404080601010b040b0a0a0b090b0a0a0a');
INSERT INTO nodes_fts_data VALUES(687194767361,x'0000008c04302e747805080801010d01032f736805060101060103617265050804010109010363732f05060101040103642e7405080701010c02026f630506010102010365642e05080601010b010368617205080301010801036f63730506010103010372656405080501010a0103732f73050601010502026861050802010107010374787405080901010e040b0a0b0a0b090b0b0a0b0a0a');
INSERT INTO nodes_fts_data VALUES(824633720833,x'00000084043020666f060809010109010364657206080d01010d0103652066060808010108010366666c06080301010302026c6906080401010402026f6c06080a01010a0103696e6506080601010601036c646506080c01010c0202696e06080501010501036e652006080701010701036f666606080201010202026c6406080b01010b040b0b0b0b0a0a0b0b0a0b0b');
INSERT INTO nodes_fts_data VALUES(962072674305,x'000000da043020666f070601010901032e747807080701011601032f696e07060101100103646572070601010d010365206607060101080202722e07080501011403012f070601010e010366666c070601010302026c69070601010402026f6c070601010a0103696e65070601010603016e07080201011101036c6465070601010c0202696e070601010501036e6520070601010703017207080401011302026e6507080301011201036f6666070601010202026c64070601010b0103722e7407080601011502022f69070601010f0103747874070808010117040a0b0a0a0a0a080a09090a090a090a090a0a090b09');
INSERT INTO nodes_fts_data VALUES(1099511627777,x'0000005c04302e706108080701010701036172740808090101090103652e70080806010106010369652e08080501010501036d6f7608080201010201036f766908080301010301037061720808080101080103766965080804010104040b0b0b0b0b0b0b');
CREATE TABLE IF NOT EXISTS 'nodes_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
INSERT INTO nodes_fts_idx VALUES(1,x'',2);
INSERT INTO nodes_fts_idx VALUES(2,x'',2);
INSERT INTO nodes_fts_idx VALUES(3,x'',2);
INSERT INTO nodes_fts_idx VALUES(4,x'',2);
INSERT INTO nodes_fts_idx VALUES(5,x'',2);
INSERT INTO nodes_fts_idx VALUES(6,x'',2);
INSERT INTO nodes_fts_idx VALUES(7,x'',2);
INSERT INTO nodes_fts_idx VALUES(8,x'',2);
CREATE TABLE IF NOT EXISTS 'nodes_fts_content'(id INTEGER PRIMARY KEY, c0, c1);
INSERT INTO nodes_fts_content VALUES(1,'My Files','');
INSERT INTO nodes_fts_content VALUES(2,'Docs','Docs');
INSERT INTO nodes_fts_content VALUES(3,'report.txt','Docs/report.txt');
INSERT INTO nodes_fts_content VALUES(4,'old.txt','Docs/old.txt');
INSERT INTO nodes_fts_content VALUES(5,'shared.txt','Docs/shared.txt');
INSERT INTO nodes_fts_content VALUES(6,'Offline folder','Offline folder');
INSERT INTO nodes_fts_content VALUES(7,'inner.txt','Offline folder/inner.txt');
INSERT INTO nodes_fts_content VALUES(8,'movie.part','movie.part');
CREATE TABLE IF NOT EXISTS 'nodes_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
INSERT INTO nodes_fts_docsize VALUES(1,x'0600');
INSERT INTO nodes_fts_docsize VALUES(2,x'0202');
INSERT INTO nodes_fts_docsize VALUES(3,x'080d');
INSERT INTO nodes_fts_docsize VALUES(4,x'050a');
INSERT INTO nodes_fts_docsize VALUES(5,x'080d');
INSERT INTO nodes_fts_docsize VALUES(6,x'0c0c');
INSERT INTO nodes_fts_docsize VALUES(7,x'0716');
INSERT INTO nodes_fts_docsize VALUES(8,x'0808');
CREATE TABLE IF NOT EXISTS 'nodes_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
INSERT INTO nodes_fts_config VALUES('version',4);
CREATE TABLE own_sealed_rev (
uid TEXT PRIMARY KEY,
revision_id TEXT NOT NULL,
sealed_at INTEGER NOT NULL
);
INSERT INTO own_sealed_rev VALUES('vol~report','rev-1',1700000000000);
PRAGMA writable_schema=ON;
INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)VALUES('table','local_fts','local_fts',0,'CREATE VIRTUAL TABLE local_fts USING fts5(
name, path, content=''local_files'', content_rowid=''id'', tokenize=''trigram''
)');
INSERT INTO sqlite_schema(type,name,tbl_name,rootpage,sql)VALUES('table','nodes_fts','nodes_fts',0,'CREATE VIRTUAL TABLE nodes_fts USING fts5(
name, path, tokenize=''trigram''
)');
CREATE TRIGGER mount_sync_folder_insert
AFTER INSERT ON sync_folder
BEGIN
INSERT OR IGNORE INTO mount (kind, sync_folder_id)
VALUES ('device', NEW.id);
END;
CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_photos_seq ON photos(seq);
CREATE INDEX pending_op_uid ON pending_op(uid);
CREATE INDEX pending_op_parent ON pending_op(parent_uid);
CREATE INDEX cache_entries_lru ON cache_entries(kind, last_accessed);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_local_files_name_nocase
ON local_files(name COLLATE NOCASE);
CREATE UNIQUE INDEX mount_myfiles
ON mount(kind) WHERE kind = 'myfiles';
CREATE UNIQUE INDEX mount_device
ON mount(sync_folder_id) WHERE kind = 'device';
CREATE UNIQUE INDEX mount_shared
ON mount(share_root_uid) WHERE kind = 'shared';
CREATE INDEX idx_album_photos_seq ON album_photos(album_uid, seq);
CREATE INDEX idx_album_photos_uid ON album_photos(uid);
CREATE INDEX idx_photos_favorite ON photos(favorite, seq);
CREATE INDEX idx_sync_entry_remote_uid ON sync_entry(remote_uid);
CREATE INDEX idx_local_files_scan_gen ON local_files(scan_gen);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
CREATE INDEX idx_pending_op_claimed
ON pending_op(uid) WHERE claimed_at <> 0;
CREATE INDEX idx_trash_parent ON trash(parent_uid);
CREATE INDEX idx_photos_group ON photos(group_key);
CREATE INDEX idx_photos_unresolved ON photos(resolved_at);
CREATE INDEX idx_photos_place ON photos(place_id);
PRAGMA writable_schema=OFF;
COMMIT;
+1 -1
View File
@@ -1,4 +1,4 @@
//! Database health reporting and repair (features.md 4.3). //! Database health reporting and repair.
//! //!
//! Everything here exists for the case where something has already gone wrong: //! Everything here exists for the case where something has already gone wrong:
//! a crash mid-write, a full disk, a cache that has grown out of proportion to //! a crash mid-write, a full disk, a cache that has grown out of proportion to
+277 -6
View File
@@ -5,11 +5,13 @@
//! is already on disk somewhere, so a change here would silently diverge from //! is already on disk somewhere, so a change here would silently diverge from
//! what an existing install actually has. Add a new one instead. //! what an existing install actually has. Add a new one instead.
use rusqlite::OptionalExtension;
use super::Db; use super::Db;
use crate::Result; use crate::Result;
/// Current schema version. Bump on every forward migration added below. /// Current schema version. Bump on every forward migration added below.
pub(super) const SCHEMA_VERSION: i64 = 35; pub(super) const SCHEMA_VERSION: i64 = 42;
impl Db { impl Db {
pub(super) fn migrate(&self) -> Result<()> { pub(super) fn migrate(&self) -> Result<()> {
@@ -332,6 +334,112 @@ impl Db {
tx.execute_batch(MIGRATION_V35)?; tx.execute_batch(MIGRATION_V35)?;
} }
} }
if current < 36 {
// Same guards as V26-V35.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('nodes') WHERE name = 'lid'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_nodes: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'nodes'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_nodes && !has_column {
tx.execute_batch(MIGRATION_V36)?;
}
}
if current < 37 {
// Same guards as V26-V36.
let sql: Option<String> = tx
.query_row(
"SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get(0),
)
.optional()?;
if sql.is_some_and(|sql| !sql.contains("AUTOINCREMENT")) {
tx.execute_batch(MIGRATION_V37)?;
}
}
if current < 38 {
// Same guards as V26-V37.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('pending_op') WHERE name = 'issue'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_ops: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_ops && !has_column {
tx.execute_batch(MIGRATION_V38)?;
}
}
if current < 39 {
// Same guards as V26-V38.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('pending_op') WHERE name = 'lid'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_ops: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_ops && !has_column {
tx.execute_batch(MIGRATION_V39)?;
}
}
if current < 40 {
// Same guards as V26-V39.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('nodes') WHERE name = 'parent_lid'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_nodes: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'nodes'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_nodes && !has_column {
tx.execute_batch(MIGRATION_V40)?;
}
}
if current < 41 {
// Needs both tables; on a database that never had them there is
// nothing to rewrite.
let tables: i64 = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master
WHERE type = 'table' AND name IN ('nodes', 'pending_op', 'pins')",
[],
|row| row.get(0),
)?;
if tables == 3 {
tx.execute_batch(MIGRATION_V41)?;
}
}
if current < 42 {
// Same guards as V26-V40.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('pending_op') WHERE name = 'sent_sha1'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_ops: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_ops && !has_column {
tx.execute_batch(MIGRATION_V42)?;
}
}
tx.execute( tx.execute(
"INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1) "INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1)
ON CONFLICT(key) DO UPDATE SET value = excluded.value", ON CONFLICT(key) DO UPDATE SET value = excluded.value",
@@ -469,7 +577,7 @@ CREATE TABLE trash (
); );
"; ";
/// Schema v8: device sync (devices.md). This machine registers as one Proton Drive /// Schema v8: device sync. This machine registers as one Proton Drive
/// **Device** (a share + root folder on the main volume); `device` is a singleton /// **Device** (a share + root folder on the main volume); `device` is a singleton
/// row cached so we reuse the same device across restarts instead of creating a /// row cached so we reuse the same device across restarts instead of creating a
/// new one each run. `sync_folder` is one row per local folder the user added, /// new one each run. `sync_folder` is one row per local folder the user added,
@@ -530,8 +638,8 @@ ALTER TABLE sync_folder ADD COLUMN pending_mode TEXT;
"; ";
/// Schema v11: writes no longer upload inside the FUSE handler. `release` stages /// Schema v11: writes no longer upload inside the FUSE handler. `release` stages
/// the bytes and records the intended upload here, and a drain worker performs it /// the bytes and records the intended upload here, and a drain worker performs it.
/// (offline.md Phase 3). The row outlives the process, so a write survives both a /// The row outlives the process, so a write survives both a
/// dead network and a restart. /// dead network and a restart.
/// ///
/// `blob_path` points into the content cache's `staging/` dir and `meta_json` is /// `blob_path` points into the content cache's `staging/` dir and `meta_json` is
@@ -558,7 +666,7 @@ CREATE INDEX pending_op_uid ON pending_op(uid);
/// Schema v12: `pending_op` also carries mutations that *create* a node, which a /// Schema v12: `pending_op` also carries mutations that *create* a node, which a
/// revision op never had to describe — it always addressed a node the server had /// revision op never had to describe — it always addressed a node the server had
/// already minted a uid for (offline.md Phase 3b). /// already minted a uid for.
/// ///
/// An offline `create`/`mkdir` cannot get a real uid, so the node is invented /// An offline `create`/`mkdir` cannot get a real uid, so the node is invented
/// locally under a `local~<uuid>` placeholder and the op records where it goes /// locally under a `local~<uuid>` placeholder and the op records where it goes
@@ -948,7 +1056,7 @@ ALTER TABLE pending_op ADD COLUMN access_deferred_since INTEGER NOT NULL DEFAULT
/// as the last sync and left the file the same length — a fixed-size record /// as the last sync and left the file the same length — a fixed-size record
/// rewritten, an image re-exported, a database page updated in place — read as /// rewritten, an image re-exported, a database page updated in place — read as
/// *unchanged* and was never uploaded. The remote copy silently stayed at the /// *unchanged* and was never uploaded. The remote copy silently stayed at the
/// older content until something else about the file moved (bugs.md B25). /// older content until something else about the file moved (`docs/BUGS.md` B25).
/// ///
/// Nullable rather than a converted `local_mtime * 1_000_000_000`: the /// Nullable rather than a converted `local_mtime * 1_000_000_000`: the
/// nanosecond part of an existing baseline is genuinely unknown, and inventing /// nanosecond part of an existing baseline is genuinely unknown, and inventing
@@ -1059,3 +1167,166 @@ ALTER TABLE photos ADD COLUMN similar_hash INTEGER;
const MIGRATION_V35: &str = " const MIGRATION_V35: &str = "
ALTER TABLE trash ADD COLUMN trashed_at INTEGER; ALTER TABLE trash ADD COLUMN trashed_at INTEGER;
"; ";
/// Schema v36: every node row has a local id (`lid`); see
/// docs/ARCHITECTURE.md §2.
///
/// The `uid` was the row's key, so a node had no identity until Drive minted
/// one, and a create that landed moved its row to a new key. The `lid` is given
/// when the row is first written and never changes or comes back, which is what
/// `AUTOINCREMENT` adds over a plain rowid. It takes the old rowid, so the
/// search index, keyed by rowid since v15, stays valid. The `uid` becomes
/// nullable for a node that is not on Drive yet.
const MIGRATION_V36: &str = "
CREATE TABLE nodes_v36 (
lid INTEGER PRIMARY KEY AUTOINCREMENT,
uid TEXT UNIQUE,
parent_uid TEXT,
name TEXT NOT NULL,
is_dir INTEGER NOT NULL,
size INTEGER,
mtime INTEGER NOT NULL,
revision_hash TEXT,
trashed INTEGER NOT NULL DEFAULT 0,
node_json TEXT,
listed INTEGER NOT NULL DEFAULT 0,
path TEXT
);
INSERT INTO nodes_v36
(lid, uid, parent_uid, name, is_dir, size, mtime, revision_hash, trashed, node_json, listed,
path)
SELECT rowid, uid, parent_uid, name, is_dir, size, mtime, revision_hash, trashed, node_json,
listed, path
FROM nodes;
DROP TABLE nodes;
ALTER TABLE nodes_v36 RENAME TO nodes;
CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
";
/// Schema v37: a `pending_op` id is never given out twice.
///
/// A plain rowid is the largest one plus one, so an op queued right after the
/// newest one was dropped took its id. The drain retires the op it is
/// working on by id, and so deleted the new op: a trash queued over a rename
/// on the wire was lost when the rename landed. `AUTOINCREMENT` never reuses
/// an id. Ids are kept, as is every index.
const MIGRATION_V37: &str = "
CREATE TABLE pending_op_v37 (
id INTEGER PRIMARY KEY AUTOINCREMENT,
kind TEXT NOT NULL,
uid TEXT NOT NULL,
blob_path TEXT,
meta_json TEXT,
created_at INTEGER NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
next_attempt_at INTEGER NOT NULL DEFAULT 0,
parent_uid TEXT,
name TEXT,
claimed_at INTEGER NOT NULL DEFAULT 0,
access_deferred_since INTEGER NOT NULL DEFAULT 0
);
INSERT INTO pending_op_v37
(id, kind, uid, blob_path, meta_json, created_at, attempts, last_error, next_attempt_at,
parent_uid, name, claimed_at, access_deferred_since)
SELECT id, kind, uid, blob_path, meta_json, created_at, attempts, last_error,
next_attempt_at, parent_uid, name, claimed_at, access_deferred_since
FROM pending_op;
DROP TABLE pending_op;
ALTER TABLE pending_op_v37 RENAME TO pending_op;
CREATE INDEX pending_op_uid ON pending_op(uid);
CREATE INDEX pending_op_parent ON pending_op(parent_uid);
CREATE INDEX idx_pending_op_claimed
ON pending_op(uid) WHERE claimed_at <> 0;
";
/// Schema v38: why Drive refused a queued op, when it did (a sync issue).
/// `NULL` for an op that has not been refused, which is every op migrated.
const MIGRATION_V38: &str = "
ALTER TABLE pending_op ADD COLUMN issue TEXT;
";
/// Schema v39: a queued op names its node and its parent by local id too.
///
/// A create made inside a folder that had not landed named the folder by its
/// placeholder uid, and every landing had to find and rewrite those strings.
/// The local id stays the same when the folder lands. An op whose node has no
/// row keeps `NULL` and is matched by uid, as before.
const MIGRATION_V39: &str = "
ALTER TABLE pending_op ADD COLUMN lid INTEGER;
ALTER TABLE pending_op ADD COLUMN parent_lid INTEGER;
UPDATE pending_op
SET lid = (SELECT lid FROM nodes WHERE nodes.uid = pending_op.uid),
parent_lid = (SELECT lid FROM nodes WHERE nodes.uid = pending_op.parent_uid);
CREATE INDEX pending_op_lid ON pending_op(lid);
CREATE INDEX pending_op_parent_lid ON pending_op(parent_lid);
";
/// Schema v40: a node row names its parent by local id too.
///
/// The local id stays when a folder made here lands and its row takes the uid
/// Drive gave it, so a child linked by it needs no rewrite. `NULL` while the
/// parent has no row, as `parent_uid` then names nothing either.
const MIGRATION_V40: &str = "
ALTER TABLE nodes ADD COLUMN parent_lid INTEGER;
UPDATE nodes SET parent_lid = (SELECT p.lid FROM nodes p WHERE p.uid = nodes.parent_uid);
CREATE INDEX idx_nodes_parent_lid ON nodes(parent_lid);
";
/// Schema v41: a node made on this machine goes by `local~<lid>` until it
/// lands, a function of its row, where it used to get a `local~<ms>-<n>`
/// placeholder. Every placeholder still stored is rewritten to its row's
/// stand-in: the row itself and its `node_json`, its children, the ops on it
/// or into it, and its pins. A placeholder op with no row keeps its uid and
/// drains as before.
const MIGRATION_V41: &str = "
CREATE TEMP TABLE local_v41 AS
SELECT uid AS old, 'local~' || lid AS new, lid FROM nodes
WHERE uid LIKE 'local~%' AND uid <> 'local~' || lid;
UPDATE pending_op SET uid = (SELECT new FROM local_v41 WHERE old = pending_op.uid)
WHERE uid IN (SELECT old FROM local_v41);
UPDATE pending_op SET parent_uid = (SELECT new FROM local_v41 WHERE old = pending_op.parent_uid)
WHERE parent_uid IN (SELECT old FROM local_v41);
UPDATE pending_op
SET meta_json = json_set(meta_json, '$.uid',
(SELECT new FROM local_v41 WHERE old = json_extract(pending_op.meta_json, '$.uid')))
WHERE json_valid(meta_json)
AND json_extract(meta_json, '$.uid') IN (SELECT old FROM local_v41);
UPDATE pending_op
SET meta_json = json_set(meta_json, '$.original_parent_uid',
(SELECT new FROM local_v41
WHERE old = json_extract(pending_op.meta_json, '$.original_parent_uid')))
WHERE json_valid(meta_json)
AND json_extract(meta_json, '$.original_parent_uid') IN (SELECT old FROM local_v41);
UPDATE pins SET uid = (SELECT new FROM local_v41 WHERE old = pins.uid)
WHERE uid IN (SELECT old FROM local_v41);
UPDATE nodes
SET node_json = json_set(node_json, '$.parent_uid.link_id',
(SELECT CAST(lid AS TEXT) FROM local_v41 WHERE old = nodes.parent_uid))
WHERE parent_uid IN (SELECT old FROM local_v41)
AND json_valid(node_json)
AND json_extract(node_json, '$.parent_uid.volume_id') = 'local';
UPDATE nodes SET parent_uid = (SELECT new FROM local_v41 WHERE old = nodes.parent_uid)
WHERE parent_uid IN (SELECT old FROM local_v41);
UPDATE nodes
SET node_json = json_set(node_json, '$.uid.link_id', CAST(lid AS TEXT))
WHERE lid IN (SELECT lid FROM local_v41)
AND json_valid(node_json)
AND json_extract(node_json, '$.uid.volume_id') = 'local';
UPDATE nodes SET uid = 'local~' || lid WHERE lid IN (SELECT lid FROM local_v41);
DROP TABLE local_v41;
";
/// Schema v42: the SHA-1 of each blob a queued create sent, space-separated.
/// A create whose answer was lost may have made its file with any of them,
/// and the op that finds that file again may run after a restart. `NULL` for
/// an op that has sent nothing, which is every op migrated: what the daemon
/// that wrote it sent is not known. Likewise the parent and name each attempt
/// was sent under, as a JSON list of pairs: a rename since then moves the op
/// but not the file Drive may have made.
const MIGRATION_V42: &str = "
ALTER TABLE pending_op ADD COLUMN sent_sha1 TEXT;
ALTER TABLE pending_op ADD COLUMN sent_to TEXT;
";
+3 -2
View File
@@ -56,8 +56,9 @@ pub use local::LocalFileHit;
pub use maintenance::{DbStats, VacuumOutcome}; pub use maintenance::{DbStats, VacuumOutcome};
pub use nodes::{PublishedSharedRoot, SearchHit, StoredNode}; pub use nodes::{PublishedSharedRoot, SearchHit, StoredNode};
pub use ops::{ pub use ops::{
AttachedBlob, FAILING_ATTEMPTS, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, AttachedBlob, CreateLanding, CreateRetired, FAILING_ATTEMPTS, LOCAL_VOLUME, OP_CREATE,
OP_TRASH, PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts, PendingOp, RenameMeta, op_supersedes, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts,
PendingOp, RenameMeta, local_lid, local_uid, op_supersedes,
}; };
pub use photos::{StoredPhoto, StoredPlace, THUMB_HAVE, THUMB_NONE, THUMB_UNKNOWN, TimelineRow}; pub use photos::{StoredPhoto, StoredPlace, THUMB_HAVE, THUMB_NONE, THUMB_UNKNOWN, TimelineRow};
pub use pins::PinRow; pub use pins::PinRow;
+444 -92
View File
@@ -3,9 +3,11 @@
use std::collections::HashSet; use std::collections::HashSet;
use rusqlite::types::Value;
use rusqlite::{OptionalExtension, Transaction, params}; use rusqlite::{OptionalExtension, Transaction, params};
use super::Db; use super::Db;
use super::ops::{OP_CREATE, OP_MKDIR, OP_RENAME, local_lid, local_uid, wake_ops_waiting_for_tx};
use crate::{Access, Result}; use crate::{Access, Result};
use proton_drive_rs::proton_sdk::ids::{LinkId, NodeUid, VolumeId}; use proton_drive_rs::proton_sdk::ids::{LinkId, NodeUid, VolumeId};
use proton_drive_rs::{Node, NodeKind}; use proton_drive_rs::{Node, NodeKind};
@@ -17,6 +19,8 @@ use super::utils::{
pub struct StoredNode { pub struct StoredNode {
pub node: Node, pub node: Node,
pub listed: bool, pub listed: bool,
/// The row's local id ([`Db::lid_of`]).
pub lid: i64,
} }
/// One full-text search match: the stored [`Node`] plus its mountpoint-relative /// One full-text search match: the stored [`Node`] plus its mountpoint-relative
@@ -62,12 +66,13 @@ impl Db {
/// snapshot while offline, including after a live event expired its TTL. /// snapshot while offline, including after a live event expired its TTL.
pub fn visible_children(&self, parent: &NodeUid) -> Result<Vec<Node>> { pub fn visible_children(&self, parent: &NodeUid) -> Result<Vec<Node>> {
let conn = self.read(); let conn = self.read();
let mut stmt = conn.prepare( let (key, parent) = parent_key(&conn, &parent.to_string())?;
"SELECT node_json FROM nodes let mut stmt = conn.prepare(&format!(
WHERE parent_uid = ?1 AND trashed = 0 AND node_json IS NOT NULL "SELECT {NODE_NOW} FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
ORDER BY name, uid", WHERE n.{key} = ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL
)?; ORDER BY n.name, n.uid"
let rows = stmt.query_map([parent.to_string()], |row| row.get::<_, String>(0))?; ))?;
let rows = stmt.query_map([parent], |row| row.get::<_, String>(0))?;
let mut nodes = Vec::new(); let mut nodes = Vec::new();
for row in rows { for row in rows {
nodes.push(serde_json::from_str(&row?)?); nodes.push(serde_json::from_str(&row?)?);
@@ -264,10 +269,25 @@ impl Db {
/// resolve its shared-tree access. Removing the op first could resurrect /// resolve its shared-tree access. Removing the op first could resurrect
/// that row after a crash; removing the row first could strand a now /// that row after a crash; removing the row first could strand a now
/// unauthorizable op. One transaction closes both windows. /// unauthorizable op. One transaction closes both windows.
pub fn complete_trash_op(&self, op_id: i64, uid: &NodeUid) -> Result<()> { ///
/// An op that found the name still held by this trash failed with a
/// backoff, and the name is free now. It is made due at once
/// ([`Db::wake_ops_waiting_for`]).
///
/// Returns the staged blob of the create the trash withdrew, if it kept
/// one, for the caller to discard.
pub fn complete_trash_op(&self, op_id: i64, uid: &NodeUid) -> Result<Option<String>> {
let uid = uid.to_string(); let uid = uid.to_string();
let mut conn = self.conn.lock(); let mut conn = self.conn.lock();
let tx = conn.transaction()?; let tx = conn.transaction()?;
let (name, blob): (Option<String>, Option<String>) = tx
.query_row(
"SELECT name, blob_path FROM pending_op WHERE id = ?1",
params![op_id],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?
.unwrap_or_default();
let rowid: Option<i64> = tx let rowid: Option<i64> = tx
.query_row( .query_row(
"SELECT rowid FROM nodes WHERE uid = ?1", "SELECT rowid FROM nodes WHERE uid = ?1",
@@ -280,17 +300,23 @@ impl Db {
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![rowid])?; tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![rowid])?;
} }
tx.execute("DELETE FROM pending_op WHERE id = ?1", params![op_id])?; tx.execute("DELETE FROM pending_op WHERE id = ?1", params![op_id])?;
if let Some(name) = name {
wake_ops_waiting_for_tx(&tx, &name)?;
}
tx.commit()?; tx.commit()?;
Ok(()) Ok(blob)
} }
/// Check if a folder node has any non-trashed children in the database. /// Check if a folder node has any non-trashed children in the database.
pub fn has_children(&self, parent_uid: &NodeUid) -> Result<bool> { pub fn has_children(&self, parent_uid: &NodeUid) -> Result<bool> {
let uid_str = parent_uid.to_string();
let conn = self.read(); let conn = self.read();
let (key, parent) = parent_key(&conn, &parent_uid.to_string())?;
let count: i64 = conn.query_row( let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM nodes WHERE parent_uid = ?1 AND trashed = 0", &format!(
params![uid_str], "SELECT COUNT(*) FROM nodes
WHERE {key} = ?1 AND trashed = 0 AND node_json IS NOT NULL"
),
params![parent],
|row| row.get(0), |row| row.get(0),
)?; )?;
Ok(count > 0) Ok(count > 0)
@@ -305,9 +331,11 @@ impl Db {
pub fn node_path(&self, uid: &str) -> Result<Option<String>> { pub fn node_path(&self, uid: &str) -> Result<Option<String>> {
let conn = self.read(); let conn = self.read();
let exists: Option<i64> = conn let exists: Option<i64> = conn
.query_row("SELECT 1 FROM nodes WHERE uid = ?1", params![uid], |r| { .query_row(
r.get(0) "SELECT 1 FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL",
}) params![uid],
|r| r.get(0),
)
.optional()?; .optional()?;
if exists.is_none() { if exists.is_none() {
return Ok(None); return Ok(None);
@@ -325,11 +353,11 @@ impl Db {
pub fn path_relative_to(&self, ancestor_uid: &str, uid: &str) -> Result<Option<String>> { pub fn path_relative_to(&self, ancestor_uid: &str, uid: &str) -> Result<Option<String>> {
let conn = self.read(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"WITH RECURSIVE anc(uid, parent_uid, name, depth) AS ( "WITH RECURSIVE anc(uid, parent_lid, name, depth) AS (
SELECT uid, parent_uid, name, 0 FROM nodes WHERE uid = ?1 SELECT uid, parent_lid, name, 0 FROM nodes WHERE uid = ?1
UNION ALL UNION ALL
SELECT n.uid, n.parent_uid, n.name, anc.depth + 1 SELECT n.uid, n.parent_lid, n.name, anc.depth + 1
FROM nodes n JOIN anc ON n.uid = anc.parent_uid FROM nodes n JOIN anc ON n.lid = anc.parent_lid
WHERE anc.depth < 1024 WHERE anc.depth < 1024
) )
SELECT uid, name FROM anc ORDER BY depth", SELECT uid, name FROM anc ORDER BY depth",
@@ -378,12 +406,13 @@ impl Db {
let conn = self.read(); let conn = self.read();
let rows: Vec<HitRow> = if query.chars().count() < TRIGRAM_MIN { let rows: Vec<HitRow> = if query.chars().count() < TRIGRAM_MIN {
let pat = format!("%{}%", like_escape(query)); let pat = format!("%{}%", like_escape(query));
let mut stmt = conn.prepare( let mut stmt = conn.prepare(&format!(
"SELECT node_json, uid, path FROM nodes "SELECT {NODE_NOW}, n.uid, n.path
WHERE name LIKE ?1 ESCAPE '\\' AND trashed = 0 AND node_json IS NOT NULL FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
AND (?3 IS NULL OR path IS NULL OR path LIKE ?3 ESCAPE '\\') WHERE n.name LIKE ?1 ESCAPE '\\' AND n.trashed = 0 AND n.node_json IS NOT NULL
ORDER BY name LIMIT ?2", AND (?3 IS NULL OR n.path IS NULL OR n.path LIKE ?3 ESCAPE '\\')
)?; ORDER BY n.name LIMIT ?2"
))?;
collect_hits(stmt.query_map(params![pat, limit as i64, scope_pat], hit_row)?)? collect_hits(stmt.query_map(params![pat, limit as i64, scope_pat], hit_row)?)?
} else { } else {
// Escape double quotes and quote each term, then combine with AND so // Escape double quotes and quote each term, then combine with AND so
@@ -393,13 +422,14 @@ impl Db {
.map(|word| format!("\"{}\"", word.replace('"', "\"\""))) .map(|word| format!("\"{}\"", word.replace('"', "\"\"")))
.collect::<Vec<_>>() .collect::<Vec<_>>()
.join(" AND "); .join(" AND ");
let mut stmt = conn.prepare( let mut stmt = conn.prepare(&format!(
"SELECT n.node_json, n.uid, n.path "SELECT {NODE_NOW}, n.uid, n.path
FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid
LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE f.name MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL WHERE f.name MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL
AND (?3 IS NULL OR n.path IS NULL OR n.path LIKE ?3 ESCAPE '\\') AND (?3 IS NULL OR n.path IS NULL OR n.path LIKE ?3 ESCAPE '\\')
ORDER BY f.rank LIMIT ?2", ORDER BY f.rank LIMIT ?2"
)?; ))?;
collect_hits(stmt.query_map(params![phrase, limit as i64, scope_pat], hit_row)?)? collect_hits(stmt.query_map(params![phrase, limit as i64, scope_pat], hit_row)?)?
}; };
@@ -436,20 +466,22 @@ impl Db {
let lane_limit = limit.div_ceil(2); let lane_limit = limit.div_ceil(2);
let mut rows: Vec<HitRow> = if terms.is_empty() { let mut rows: Vec<HitRow> = if terms.is_empty() {
let pat = format!("%{}%", like_escape(query)); let pat = format!("%{}%", like_escape(query));
let mut stmt = conn.prepare( let mut stmt = conn.prepare(&format!(
"SELECT node_json, uid, path FROM nodes "SELECT {NODE_NOW}, n.uid, n.path
WHERE name LIKE ?1 ESCAPE '\\' AND trashed = 0 AND node_json IS NOT NULL FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
ORDER BY name LIMIT ?2", WHERE n.name LIKE ?1 ESCAPE '\\' AND n.trashed = 0 AND n.node_json IS NOT NULL
)?; ORDER BY n.name LIMIT ?2"
))?;
collect_hits(stmt.query_map(params![pat, limit as i64], hit_row)?)? collect_hits(stmt.query_map(params![pat, limit as i64], hit_row)?)?
} else { } else {
let expression = terms.join(" OR "); let expression = terms.join(" OR ");
let mut stmt = conn.prepare( let mut stmt = conn.prepare(&format!(
"SELECT n.node_json, n.uid, n.path "SELECT {NODE_NOW}, n.uid, n.path
FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid
LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE nodes_fts MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL WHERE nodes_fts MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL
ORDER BY f.rank LIMIT ?2", ORDER BY f.rank LIMIT ?2"
)?; ))?;
collect_hits(stmt.query_map(params![expression, lane_limit as i64], hit_row)?)? collect_hits(stmt.query_map(params![expression, lane_limit as i64], hit_row)?)?
}; };
// Trigram rank does not know that a name *starts* with the query, and a // Trigram rank does not know that a name *starts* with the query, and a
@@ -459,12 +491,13 @@ impl Db {
if !terms.is_empty() if !terms.is_empty()
&& let Some(first) = query.chars().next() && let Some(first) = query.chars().next()
{ {
let mut stmt = conn.prepare( let mut stmt = conn.prepare(&format!(
"SELECT node_json, uid, path FROM nodes "SELECT {NODE_NOW}, n.uid, n.path
WHERE name COLLATE NOCASE LIKE ?1 ESCAPE '\\' FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
AND trashed = 0 AND node_json IS NOT NULL WHERE n.name COLLATE NOCASE LIKE ?1 ESCAPE '\\'
ORDER BY name COLLATE NOCASE LIMIT ?2", AND n.trashed = 0 AND n.node_json IS NOT NULL
)?; ORDER BY n.name COLLATE NOCASE LIMIT ?2"
))?;
let mut extra = Vec::new(); let mut extra = Vec::new();
for pattern in [ for pattern in [
format!("{}%", like_escape(query)), format!("{}%", like_escape(query)),
@@ -500,40 +533,50 @@ impl Db {
/// re-enumerates from the remote on next access. /// re-enumerates from the remote on next access.
pub fn set_listed(&self, uid: &NodeUid, listed: bool) -> Result<()> { pub fn set_listed(&self, uid: &NodeUid, listed: bool) -> Result<()> {
let conn = self.conn.lock(); let conn = self.conn.lock();
let (key, uid) = row_key(&uid.to_string());
conn.execute( conn.execute(
"UPDATE nodes SET listed = ?2 WHERE uid = ?1", &format!("UPDATE nodes SET listed = ?2 WHERE {key} = ?1"),
params![uid.to_string(), listed as i64], params![uid, listed as i64],
)?; )?;
Ok(()) Ok(())
} }
/// Load every persisted node for cold-start hydration of the `State` maps. /// Load every persisted node for cold-start hydration of the `State` maps.
///
/// Each node names its parent as the parent's row has it now, found by
/// local id: a folder that landed has its new uid there, while the node
/// stored below it may still hold the placeholder.
pub fn load_all(&self) -> Result<Vec<StoredNode>> { pub fn load_all(&self) -> Result<Vec<StoredNode>> {
let conn = self.read(); let conn = self.read();
let mut stmt = let mut stmt = conn.prepare(&format!(
conn.prepare("SELECT node_json, listed FROM nodes WHERE node_json IS NOT NULL")?; "SELECT {NODE_NOW}, n.listed, n.lid
FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.node_json IS NOT NULL"
))?;
let rows = stmt.query_map([], |row| { let rows = stmt.query_map([], |row| {
let json: String = row.get(0)?; let json: String = row.get(0)?;
let listed: i64 = row.get(1)?; let listed: i64 = row.get(1)?;
Ok((json, listed != 0)) Ok((json, listed != 0, row.get(2)?))
})?; })?;
let mut out = Vec::new(); let mut out = Vec::new();
for row in rows { for row in rows {
let (json, listed) = row?; let (json, listed, lid) = row?;
let node: Node = serde_json::from_str(&json)?; let node: Node = serde_json::from_str(&json)?;
out.push(StoredNode { node, listed }); out.push(StoredNode { node, listed, lid });
} }
Ok(out) Ok(out)
} }
/// Load one persisted node back by uid. Used to recover the My Files root /// Load one persisted node back by uid. Used to recover the My Files root
/// when the API is unreachable, so the mount can still serve the cached tree /// when the API is unreachable, so the mount can still serve the cached tree.
/// (offline.md Phase 1).
pub fn node_by_uid(&self, uid: &str) -> Result<Option<Node>> { pub fn node_by_uid(&self, uid: &str) -> Result<Option<Node>> {
let conn = self.read(); let conn = self.read();
let json: Option<String> = conn let json: Option<String> = conn
.query_row( .query_row(
"SELECT node_json FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL", &format!(
"SELECT {NODE_NOW} FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.uid = ?1 AND n.node_json IS NOT NULL"
),
params![uid], params![uid],
|r| r.get(0), |r| r.get(0),
) )
@@ -544,31 +587,178 @@ impl Db {
} }
} }
/// The uid of the node with local id `lid`: `None` when it has no row, or
/// is not on Drive yet.
pub fn uid_of_lid(&self, lid: i64) -> Result<Option<String>> {
let conn = self.read();
Ok(conn
.query_row("SELECT uid FROM nodes WHERE lid = ?1", params![lid], |r| {
r.get(0)
})
.optional()?
.flatten())
}
/// The local id of the node stored under `uid`, or `None` when it has no
/// row. It stays the same for as long as the row exists, including when a
/// create lands and the row takes the uid Drive gave it: a node made on
/// this machine is found by its stand-in ([`local_uid`]) before and after.
pub fn lid_of(&self, uid: &str) -> Result<Option<i64>> {
node_lid(&self.read(), uid)
}
/// The local id of each node, in order, giving a row to each that has none.
///
/// A new row is only a stub: it has no `node_json`, so no listing or search
/// returns it until the node is upserted, which the caller still owes. That
/// keeps this cheap enough to call with the inode lock held, which is
/// where an inode number is first needed.
pub fn lids_for(&self, nodes: &[&Node]) -> Result<Vec<i64>> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let mut lids = Vec::with_capacity(nodes.len());
{
let mut known = tx.prepare_cached("SELECT lid FROM nodes WHERE uid = ?1")?;
let mut stub = tx.prepare_cached(
"INSERT INTO nodes (uid, parent_uid, name, is_dir, mtime, trashed, parent_lid)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)",
)?;
for node in nodes {
let uid = node.uid.to_string();
let lid = match known.query_row(params![uid], |r| r.get(0)).optional()? {
Some(lid) => lid,
None => {
let parent = node.parent_uid.as_ref().map(ToString::to_string);
let parent_lid = match &parent {
Some(parent) => node_lid(&tx, parent)?,
None => None,
};
stub.execute(params![
uid,
parent,
node.name,
node.is_folder() as i64,
node.modification_time,
node.trashed as i64,
parent_lid,
])?;
let lid = tx.last_insert_rowid();
adopt_children_tx(&tx, &uid, lid)?;
lid
}
};
lids.push(lid);
}
}
tx.commit()?;
Ok(lids)
}
/// Give a node being made on this machine its row under `parent_uid`, and
/// return its local id. The row goes by [`local_uid`] of that id until
/// the node lands.
///
/// Like [`Db::lids_for`], only a stub: the caller upserts the node.
pub fn add_local_row(
&self,
parent_uid: &str,
name: &str,
is_dir: bool,
mtime: i64,
) -> Result<i64> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
tx.execute(
"INSERT INTO nodes (uid, parent_uid, name, is_dir, mtime, trashed, parent_lid)
VALUES (NULL, ?1, ?2, ?3, ?4, 0,
COALESCE(?5, (SELECT lid FROM nodes WHERE uid = ?1)))",
params![
parent_uid,
name,
is_dir as i64,
mtime,
local_lid(parent_uid)
],
)?;
let lid = tx.last_insert_rowid();
tx.execute(
"UPDATE nodes SET uid = ?2 WHERE lid = ?1",
params![lid, local_uid(lid)],
)?;
tx.commit()?;
Ok(lid)
}
/// Read the persisted incremental-sync cursor (a `DriveEventId`), if any. /// Read the persisted incremental-sync cursor (a `DriveEventId`), if any.
/// The daemon resumes from this on restart instead of reseeding to the /// The daemon resumes from this on restart instead of reseeding to the
/// server head, so changes made while unmounted are still applied (P2). /// server head, so changes made while unmounted are still applied (P2).
pub fn children_if_listed(&self, parent: &NodeUid) -> Result<Option<Vec<Node>>> { pub fn children_if_listed(&self, parent: &NodeUid) -> Result<Option<Vec<Node>>> {
let conn = self.read(); let conn = self.read();
let (key, folder) = row_key(&parent.to_string());
let listed: Option<i64> = conn let listed: Option<i64> = conn
.query_row( .query_row(
"SELECT listed FROM nodes WHERE uid = ?1", &format!("SELECT listed FROM nodes WHERE {key} = ?1"),
params![parent.to_string()], params![folder],
|r| r.get(0), |r| r.get(0),
) )
.optional()?; .optional()?;
if listed != Some(1) { if listed != Some(1) {
return Ok(None); return Ok(None);
} }
let mut stmt = conn.prepare( drop(conn);
"SELECT node_json FROM nodes self.known_children(parent).map(Some)
WHERE parent_uid = ?1 AND node_json IS NOT NULL AND trashed = 0", }
)?;
let rows = stmt.query_map(params![parent.to_string()], |r| r.get::<_, String>(0))?; /// The children the DB knows under `parent`, whether or not the folder's
/// listing is still marked complete. Only for when nothing better can be
/// had: a listing dropped as stale may miss or keep a child.
pub fn known_children(&self, parent: &NodeUid) -> Result<Vec<Node>> {
let conn = self.read();
let (key, parent) = parent_key(&conn, &parent.to_string())?;
let mut stmt = conn.prepare(&format!(
"SELECT {NODE_NOW} FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.{key} = ?1 AND n.node_json IS NOT NULL AND n.trashed = 0"
))?;
let rows = stmt.query_map(params![parent], |r| r.get::<_, String>(0))?;
let mut out = Vec::new(); let mut out = Vec::new();
for json in rows { for json in rows {
out.push(serde_json::from_str(&json?)?); out.push(serde_json::from_str(&json?)?);
} }
Ok(Some(out)) Ok(out)
}
/// What a queued change does to `parent`'s listing, which Drive does not
/// show until it lands: the children made or moved in by a queued create,
/// mkdir or rename, as the DB holds them, and the nodes a queued rename
/// moves elsewhere.
///
/// A listing read from Drive is laid over with these, or a folder listed
/// while a create in it is queued, after a restart say, loses the file
/// until the create lands (`docs/BUGS.md` B132).
pub fn queued_children(&self, parent: &NodeUid) -> Result<(Vec<Node>, HashSet<String>)> {
let conn = self.read();
let (key, parent) = parent_key(&conn, &parent.to_string())?;
let mut stmt = conn.prepare(&format!(
"SELECT {NODE_NOW} FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.{key} = ?1 AND n.node_json IS NOT NULL AND n.trashed = 0
AND EXISTS (SELECT 1 FROM pending_op o
WHERE (o.lid = n.lid OR o.uid = n.uid) AND o.kind IN (?2, ?3, ?4))"
))?;
let rows = stmt.query_map(params![parent, OP_CREATE, OP_MKDIR, OP_RENAME], |r| {
r.get::<_, String>(0)
})?;
let mut here = Vec::new();
for json in rows {
here.push(serde_json::from_str(&json?)?);
}
let mut stmt = conn.prepare(&format!(
"SELECT DISTINCT n.uid FROM pending_op p JOIN nodes n ON n.lid = p.lid OR n.uid = p.uid
WHERE p.kind = ?2 AND n.{key} IS NOT ?1"
))?;
let gone = stmt
.query_map(params![parent, OP_RENAME], |r| r.get::<_, String>(0))?
.collect::<std::result::Result<HashSet<_>, _>>()?;
Ok((here, gone))
} }
// --- Content-cache LRU index (P4) ------------------------------------- // --- Content-cache LRU index (P4) -------------------------------------
@@ -590,9 +780,157 @@ struct PriorRow {
trashed: bool, trashed: bool,
} }
/// [`Db::lid_of`] on `conn`, which may be a transaction.
pub(super) fn node_lid(conn: &rusqlite::Connection, uid: &str) -> Result<Option<i64>> {
let row = match local_lid(uid) {
Some(lid) => conn.query_row("SELECT lid FROM nodes WHERE lid = ?1", params![lid], |r| {
r.get(0)
}),
None => conn.query_row("SELECT lid FROM nodes WHERE uid = ?1", params![uid], |r| {
r.get(0)
}),
};
Ok(row.optional()?)
}
/// The column and value that find `uid`'s own row. A stand-in is found by its
/// local id: a create that lands gives the row the uid Drive made before the
/// tree has it, and the tree asks by the stand-in until then (`docs/BUGS.md`
/// B177).
fn row_key(uid: &str) -> (&'static str, Value) {
match local_lid(uid) {
Some(lid) => ("lid", Value::Integer(lid)),
None => ("uid", Value::Text(uid.to_owned())),
}
}
/// The column and value a row names `parent` by: its local id when it has a
/// row, which the children keep when it lands, or its uid when it has none,
/// as a device folder's root does.
fn parent_key(conn: &rusqlite::Connection, parent: &str) -> Result<(&'static str, Value)> {
Ok(match node_lid(conn, parent)? {
Some(lid) => ("parent_lid", Value::Integer(lid)),
None => ("parent_uid", Value::Text(parent.to_owned())),
})
}
/// `n.node_json` naming its parent as the parent's row `p`, joined by
/// `n.parent_lid`, has it now. A folder that lands takes its real uid in its
/// own row only, so a node stored below it still names the stand-in.
const NODE_NOW: &str = "CASE WHEN p.uid IS NULL OR p.uid = n.parent_uid THEN n.node_json
ELSE json_set(n.node_json, '$.parent_uid', json_object(
'volume_id', substr(p.uid, 1, instr(p.uid, '~') - 1),
'link_id', substr(p.uid, instr(p.uid, '~') + 1))) END";
/// Hand a drained placeholder's row to the real uid it landed as.
///
/// Nothing else is rewritten. The rows below it and the ops made inside it
/// name it by its local id, which stays; what reads them for a uid takes the
/// one the row has now ([`NODE_NOW`], `PARENT_NOW` in `ops.rs`).
///
/// Dropping the placeholder row and waiting for the server's copy of the real
/// node left a window with no row for either uid. A queued child drained in
/// that window found no authority for its parent, re-read it over the network
/// and was deferred five seconds behind a folder that had just landed
/// (`docs/BUGS.md` B120). The placeholder's own node, readdressed, stands in
/// until the server's copy is upserted over it. A row the real uid already
/// has is newer than the placeholder, so its node is kept, under the
/// placeholder's `lid`: the node was known by that one first.
pub(super) fn adopt_placeholder_row_tx(
tx: &Transaction<'_>,
local: &str,
real: &str,
) -> Result<()> {
// A folder pinned before it landed stays pinned: the pin keyed by the
// placeholder was dropped with it (`docs/BUGS.md` B148).
tx.execute(
"UPDATE OR IGNORE pins SET uid = ?2 WHERE uid = ?1",
params![local, real],
)?;
tx.execute("DELETE FROM pins WHERE uid = ?1", params![local])?;
let row: Option<(i64, Option<String>)> = tx
.query_row(
"SELECT lid, node_json FROM nodes WHERE uid = ?1",
params![local],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
// Ops queued before schema 39 have no `parent_lid`, nor do ops made
// under a placeholder that never had a row.
tx.execute(
"UPDATE pending_op SET parent_uid = ?2 WHERE parent_lid IS NULL AND parent_uid = ?1",
params![local, real],
)?;
let Some((lid, json)) = row else {
return Ok(());
};
let known: Option<(i64, Option<String>)> = tx
.query_row(
"SELECT lid, node_json FROM nodes WHERE uid = ?1",
params![real],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
if let Some((known_lid, _)) = known {
// Ops queued against the row Drive listed now name the landing one.
tx.execute(
"UPDATE pending_op SET lid = ?2 WHERE lid = ?1",
params![known_lid, lid],
)?;
tx.execute(
"UPDATE pending_op SET parent_lid = ?2 WHERE parent_lid = ?1",
params![known_lid, lid],
)?;
tx.execute(
"UPDATE nodes SET parent_lid = ?2 WHERE parent_lid = ?1",
params![known_lid, lid],
)?;
tx.execute("DELETE FROM nodes WHERE lid = ?1", params![known_lid])?;
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![known_lid])?;
}
tx.execute(
"UPDATE nodes SET uid = ?2 WHERE lid = ?1",
params![lid, real],
)?;
let json = known.and_then(|(_, known_json)| known_json).or(json);
let (Some(json), Some(real)) = (json, parse_node_uid(real)) else {
return Ok(());
};
let mut node: Node = serde_json::from_str(&json)?;
node.uid = real;
upsert_node_tx(tx, &node)
}
/// Link the rows that name `uid` as their parent to its new row `lid`: they
/// were stored before it, or under a row of the same uid since dropped.
fn adopt_children_tx(tx: &Transaction<'_>, uid: &str, lid: i64) -> Result<()> {
tx.execute(
"UPDATE nodes SET parent_lid = ?2 WHERE parent_uid = ?1 AND parent_lid IS NOT ?2",
params![uid, lid],
)?;
Ok(())
}
fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> { fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
let json = serde_json::to_string(node)?;
let uid = node.uid.to_string(); let uid = node.uid.to_string();
// A node written under its stand-in after it landed is its row, which has
// the uid Drive gave it by now: written under the stand-in, it would bring
// the placeholder row back beside it.
if let Some(lid) = local_lid(&uid) {
let landed: Option<String> = tx
.query_row(
"SELECT uid FROM nodes WHERE lid = ?1 AND uid IS NOT ?2",
params![lid, uid],
|row| row.get(0),
)
.optional()?;
if let Some(real) = landed.as_deref().and_then(parse_node_uid) {
let mut node = node.clone();
node.uid = real;
return upsert_node_tx(tx, &node);
}
}
let json = serde_json::to_string(node)?;
let parent_uid = node.parent_uid.as_ref().map(|u| u.to_string()); let parent_uid = node.parent_uid.as_ref().map(|u| u.to_string());
let prior: Option<PriorRow> = tx let prior: Option<PriorRow> = tx
@@ -614,26 +952,34 @@ fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
// where resolving it from scratch is a recursive walk to the root. A node // where resolving it from scratch is a recursive walk to the root. A node
// whose parent is not cached (a device folder's root lives in `device`, never // whose parent is not cached (a device folder's root lives in `device`, never
// in `nodes`) starts a path of its own, which is what the walk did too. // in `nodes`) starts a path of its own, which is what the walk did too.
let path = match &parent_uid { let (parent_lid, path) = match &parent_uid {
None => String::new(), None => (None, String::new()),
Some(parent) => { Some(parent) => {
let parent_path: Option<Option<String>> = tx let parent_row: Option<(i64, Option<String>)> = match node_lid(tx, parent)? {
.query_row( Some(lid) => tx
"SELECT path FROM nodes WHERE uid = ?1", .query_row(
params![parent], "SELECT lid, path FROM nodes WHERE lid = ?1",
|row| row.get(0), params![lid],
) |row| Ok((row.get(0)?, row.get(1)?)),
.optional()?; )
join_path(parent_path.flatten().as_deref().unwrap_or(""), &node.name) .optional()?,
None => None,
};
let (lid, parent_path) = parent_row.unzip();
(
lid,
join_path(parent_path.flatten().as_deref().unwrap_or(""), &node.name),
)
} }
}; };
tx.execute( tx.execute(
"INSERT INTO nodes "INSERT INTO nodes
(uid, parent_uid, name, is_dir, size, mtime, trashed, node_json, path) (uid, parent_uid, name, is_dir, size, mtime, trashed, node_json, path, parent_lid)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10)
ON CONFLICT(uid) DO UPDATE SET ON CONFLICT(uid) DO UPDATE SET
parent_uid = excluded.parent_uid, parent_uid = excluded.parent_uid,
parent_lid = excluded.parent_lid,
name = excluded.name, name = excluded.name,
is_dir = excluded.is_dir, is_dir = excluded.is_dir,
size = excluded.size, size = excluded.size,
@@ -651,6 +997,7 @@ fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
node.trashed as i64, node.trashed as i64,
json, json,
path, path,
parent_lid,
], ],
)?; )?;
let rowid: i64 = tx.query_row( let rowid: i64 = tx.query_row(
@@ -658,6 +1005,9 @@ fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
params![uid], params![uid],
|row| row.get(0), |row| row.get(0),
)?; )?;
if prior.is_none() {
adopt_children_tx(tx, &uid, rowid)?;
}
// FTS5 has no UPSERT, so the node's own row is always replaced. // FTS5 has no UPSERT, so the node's own row is always replaced.
let indexable = node_is_indexable_tx(tx, &uid)?; let indexable = node_is_indexable_tx(tx, &uid)?;
@@ -680,7 +1030,7 @@ fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
|| prior.trashed != node.trashed || prior.trashed != node.trashed
}); });
if node.is_folder() && subtree_moved { if node.is_folder() && subtree_moved {
reindex_subtree_tx(tx, &uid, &path, indexable)?; reindex_subtree_tx(tx, rowid, &path, indexable)?;
} }
Ok(()) Ok(())
} }
@@ -701,22 +1051,22 @@ fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
/// this deep. /// this deep.
fn reindex_subtree_tx( fn reindex_subtree_tx(
tx: &Transaction<'_>, tx: &Transaction<'_>,
folder: &str, folder: i64,
folder_path: &str, folder_path: &str,
folder_indexable: bool, folder_indexable: bool,
) -> Result<()> { ) -> Result<()> {
let descendants: Vec<(i64, String, bool)> = { let descendants: Vec<(i64, String, bool)> = {
let mut stmt = tx.prepare( let mut stmt = tx.prepare(
"WITH RECURSIVE sub(rowid, uid, name, path, blocked, depth) AS ( "WITH RECURSIVE sub(rowid, name, path, blocked, depth) AS (
SELECT n.rowid, n.uid, n.name, SELECT n.rowid, n.name,
CASE WHEN ?2 = '' THEN n.name ELSE ?2 || '/' || n.name END, CASE WHEN ?2 = '' THEN n.name ELSE ?2 || '/' || n.name END,
n.trashed, 0 n.trashed, 0
FROM nodes n WHERE n.parent_uid = ?1 FROM nodes n WHERE n.parent_lid = ?1
UNION ALL UNION ALL
SELECT n.rowid, n.uid, n.name, SELECT n.rowid, n.name,
sub.path || '/' || n.name, sub.path || '/' || n.name,
MAX(sub.blocked, n.trashed), sub.depth + 1 MAX(sub.blocked, n.trashed), sub.depth + 1
FROM nodes n JOIN sub ON n.parent_uid = sub.uid FROM nodes n JOIN sub ON n.parent_lid = sub.rowid
WHERE sub.depth < 256 WHERE sub.depth < 256
) )
SELECT rowid, path, blocked FROM sub", SELECT rowid, path, blocked FROM sub",
@@ -741,8 +1091,10 @@ fn reindex_subtree_tx(
let mut set_path = tx.prepare("UPDATE nodes SET path = ?2 WHERE rowid = ?1")?; let mut set_path = tx.prepare("UPDATE nodes SET path = ?2 WHERE rowid = ?1")?;
let mut unindex = tx.prepare("DELETE FROM nodes_fts WHERE rowid = ?1")?; let mut unindex = tx.prepare("DELETE FROM nodes_fts WHERE rowid = ?1")?;
let mut index = tx.prepare( let mut index = tx.prepare(
// Not a stub row (`Db::lids_for`): it is indexed when its node is
// written.
"INSERT INTO nodes_fts (rowid, name, path) "INSERT INTO nodes_fts (rowid, name, path)
SELECT rowid, name, ?2 FROM nodes WHERE rowid = ?1", SELECT rowid, name, ?2 FROM nodes WHERE rowid = ?1 AND node_json IS NOT NULL",
)?; )?;
for (rowid, path, blocked) in descendants { for (rowid, path, blocked) in descendants {
set_path.execute(params![rowid, path])?; set_path.execute(params![rowid, path])?;
@@ -755,7 +1107,8 @@ fn reindex_subtree_tx(
} }
fn direct_child_uids_tx(tx: &Transaction<'_>, parent: &str) -> Result<Vec<String>> { fn direct_child_uids_tx(tx: &Transaction<'_>, parent: &str) -> Result<Vec<String>> {
let mut stmt = tx.prepare("SELECT uid FROM nodes WHERE parent_uid = ?1")?; let (key, parent) = parent_key(tx, parent)?;
let mut stmt = tx.prepare(&format!("SELECT uid FROM nodes WHERE {key} = ?1"))?;
let rows = stmt.query_map([parent], |row| row.get::<_, String>(0))?; let rows = stmt.query_map([parent], |row| row.get::<_, String>(0))?;
let mut uids = Vec::new(); let mut uids = Vec::new();
for row in rows { for row in rows {
@@ -780,14 +1133,14 @@ fn direct_child_uids_tx(tx: &Transaction<'_>, parent: &str) -> Result<Vec<String
/// ends at one whose parent is a row we have already visited. /// ends at one whose parent is a row we have already visited.
fn node_is_indexable_tx(tx: &Transaction<'_>, uid: &str) -> Result<bool> { fn node_is_indexable_tx(tx: &Transaction<'_>, uid: &str) -> Result<bool> {
let indexable: i64 = tx.query_row( let indexable: i64 = tx.query_row(
"WITH RECURSIVE ancestors(uid, parent_uid, trashed, depth, path) AS ( "WITH RECURSIVE ancestors(lid, parent_lid, trashed, depth, path) AS (
SELECT uid, parent_uid, trashed, 0, char(31) || uid || char(31) SELECT lid, parent_lid, trashed, 0, ',' || lid || ','
FROM nodes WHERE uid = ?1 FROM nodes WHERE uid = ?1
UNION ALL UNION ALL
SELECT n.uid, n.parent_uid, n.trashed, a.depth + 1, SELECT n.lid, n.parent_lid, n.trashed, a.depth + 1,
a.path || n.uid || char(31) a.path || n.lid || ','
FROM ancestors a JOIN nodes n ON n.uid = a.parent_uid FROM ancestors a JOIN nodes n ON n.lid = a.parent_lid
WHERE instr(a.path, char(31) || n.uid || char(31)) = 0 WHERE instr(a.path, ',' || n.lid || ',') = 0
) )
SELECT CASE SELECT CASE
WHEN COUNT(*) = 0 THEN 0 WHEN COUNT(*) = 0 THEN 0
@@ -795,8 +1148,7 @@ fn node_is_indexable_tx(tx: &Transaction<'_>, uid: &str) -> Result<bool> {
WHEN EXISTS ( WHEN EXISTS (
SELECT 1 FROM ancestors a SELECT 1 FROM ancestors a
WHERE a.depth = (SELECT MAX(depth) FROM ancestors) WHERE a.depth = (SELECT MAX(depth) FROM ancestors)
AND a.parent_uid IS NOT NULL AND EXISTS (SELECT 1 FROM nodes p WHERE p.lid = a.parent_lid)
AND EXISTS (SELECT 1 FROM nodes p WHERE p.uid = a.parent_uid)
) THEN 0 ) THEN 0
ELSE 1 ELSE 1
END END
@@ -814,7 +1166,7 @@ fn tombstone_subtree_tx(tx: &Transaction<'_>, root_uid: &str) -> Result<()> {
SELECT rowid, uid, node_json FROM nodes WHERE uid = ?1 SELECT rowid, uid, node_json FROM nodes WHERE uid = ?1
UNION UNION
SELECT n.rowid, n.uid, n.node_json FROM nodes n SELECT n.rowid, n.uid, n.node_json FROM nodes n
JOIN subtree s ON n.parent_uid = s.uid JOIN subtree s ON n.parent_lid = s.rowid
) )
SELECT rowid, uid, node_json FROM subtree", SELECT rowid, uid, node_json FROM subtree",
)?; )?;
File diff suppressed because it is too large. Load diff
+15 -13
View File
@@ -64,11 +64,11 @@ impl Db {
} }
/// Whether `uid` is pinned — directly, or because a strict ancestor folder /// Whether `uid` is pinned — directly, or because a strict ancestor folder
/// is pinned recursively. The ancestor check walks `parent_uid` to the root /// is pinned recursively. The ancestor check walks `parent_lid` to the root
/// via a CTE; a direct pin is honoured even when the node has no `nodes` row /// via a CTE; a direct pin is honoured even when the node has no `nodes` row
/// yet (e.g. a CLI that never hydrates the node cache). /// yet (e.g. a CLI that never hydrates the node cache).
/// ///
/// The walk is depth-capped like [`super::utils::path_of`]: a `parent_uid` /// The walk is depth-capped like [`super::utils::path_of`]: a parent
/// cycle is corrupt data the API can hand us, and `UNION ALL` over one never /// cycle is corrupt data the API can hand us, and `UNION ALL` over one never
/// terminates — while holding the daemon's only SQLite connection, on a path /// terminates — while holding the daemon's only SQLite connection, on a path
/// that runs per cached read. /// that runs per cached read.
@@ -85,11 +85,11 @@ impl Db {
let anc: Option<i64> = conn let anc: Option<i64> = conn
.query_row( .query_row(
&format!( &format!(
"WITH RECURSIVE anc(uid, parent_uid, depth) AS ( "WITH RECURSIVE anc(uid, parent_lid, depth) AS (
SELECT uid, parent_uid, 0 FROM nodes WHERE uid = ?1 SELECT uid, parent_lid, 0 FROM nodes WHERE uid = ?1
UNION ALL UNION ALL
SELECT n.uid, n.parent_uid, anc.depth + 1 SELECT n.uid, n.parent_lid, anc.depth + 1
FROM nodes n JOIN anc ON n.uid = anc.parent_uid FROM nodes n JOIN anc ON n.lid = anc.parent_lid
WHERE anc.depth < {MAX_PATH_DEPTH} WHERE anc.depth < {MAX_PATH_DEPTH}
) )
SELECT 1 FROM anc a JOIN pins p ON p.uid = a.uid SELECT 1 FROM anc a JOIN pins p ON p.uid = a.uid
@@ -109,10 +109,11 @@ impl Db {
pub fn pinned_uids(&self) -> Result<Vec<String>> { pub fn pinned_uids(&self) -> Result<Vec<String>> {
let conn = self.read(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"WITH RECURSIVE sub(uid) AS ( "WITH RECURSIVE sub(uid, lid) AS (
SELECT uid FROM pins WHERE recursive = 1 SELECT p.uid, n.lid FROM pins p LEFT JOIN nodes n ON n.uid = p.uid
WHERE p.recursive = 1
UNION UNION
SELECT n.uid FROM nodes n JOIN sub ON n.parent_uid = sub.uid SELECT n.uid, n.lid FROM nodes n JOIN sub ON n.parent_lid = sub.lid
) )
SELECT uid FROM sub SELECT uid FROM sub
UNION UNION
@@ -126,15 +127,16 @@ impl Db {
Ok(out) Ok(out)
} }
/// Every descendant uid of `folder` (all depths), via a `parent_uid` CTE. /// Every descendant uid of `folder` (all depths), via a `parent_lid` CTE.
/// Used to evict a recursively-pinned subtree's cached blobs on unpin. /// Used to evict a recursively-pinned subtree's cached blobs on unpin.
pub fn descendants(&self, folder: &str) -> Result<Vec<String>> { pub fn descendants(&self, folder: &str) -> Result<Vec<String>> {
let conn = self.read(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"WITH RECURSIVE sub(uid) AS ( "WITH RECURSIVE sub(uid, lid) AS (
SELECT uid FROM nodes WHERE parent_uid = ?1 SELECT c.uid, c.lid FROM nodes c JOIN nodes f ON c.parent_lid = f.lid
WHERE f.uid = ?1
UNION UNION
SELECT n.uid FROM nodes n JOIN sub ON n.parent_uid = sub.uid SELECT n.uid, n.lid FROM nodes n JOIN sub ON n.parent_lid = sub.lid
) )
SELECT uid FROM sub", SELECT uid FROM sub",
)?; )?;
+26 -9
View File
@@ -95,11 +95,27 @@ impl Db {
/// Resolve a persisted node's effective access through its nearest recorded /// Resolve a persisted node's effective access through its nearest recorded
/// share root. `None` means the node itself is not persisted, which callers /// share root. `None` means the node itself is not persisted, which callers
/// must not treat as owned: stale handles need to fail closed. /// must not treat as owned: stale handles need to fail closed.
///
/// A stand-in ([`local_uid`](super::local_uid)) finds its row by local id,
/// so it still does once its create has landed and the row has the uid
/// Drive gave it. The tree learns that uid a moment later, and a rename in
/// between was refused (`docs/BUGS.md` B174).
pub fn effective_node_access(&self, uid: &NodeUid) -> Result<Option<Access>> { pub fn effective_node_access(&self, uid: &NodeUid) -> Result<Option<Access>> {
let conn = self.read(); let conn = self.read();
let key = uid.to_string();
let key = match super::local_lid(&key) {
Some(lid) => conn
.query_row("SELECT uid FROM nodes WHERE lid = ?1", [lid], |row| {
row.get::<_, Option<String>>(0)
})
.optional()?
.flatten()
.unwrap_or(key),
None => key,
};
let exists: bool = conn.query_row( let exists: bool = conn.query_row(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE uid = ?1)", "SELECT EXISTS(SELECT 1 FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL)",
[uid.to_string()], [&key],
|row| row.get(0), |row| row.get(0),
)?; )?;
if !exists { if !exists {
@@ -107,25 +123,26 @@ impl Db {
} }
let value = conn let value = conn
.query_row( .query_row(
"WITH RECURSIVE ancestors(uid, parent_uid, depth, path) AS ( "WITH RECURSIVE ancestors(uid, parent_uid, parent_lid, depth, path) AS (
SELECT uid, parent_uid, 0, char(31) || uid || char(31) SELECT uid, parent_uid, parent_lid, 0, char(31) || uid || char(31)
FROM nodes WHERE uid = ?1 FROM nodes WHERE uid = ?1
UNION ALL UNION ALL
SELECT a.parent_uid, n.parent_uid, a.depth + 1, SELECT COALESCE(n.uid, a.parent_uid), n.parent_uid, n.parent_lid,
a.path || a.parent_uid || char(31) a.depth + 1,
a.path || COALESCE(n.uid, a.parent_uid) || char(31)
FROM ancestors a FROM ancestors a
LEFT JOIN nodes n ON n.uid = a.parent_uid LEFT JOIN nodes n ON n.lid = a.parent_lid
WHERE a.parent_uid IS NOT NULL WHERE a.parent_uid IS NOT NULL
AND instr( AND instr(
a.path, a.path,
char(31) || a.parent_uid || char(31) char(31) || COALESCE(n.uid, a.parent_uid) || char(31)
) = 0 ) = 0
) )
SELECT sa.access SELECT sa.access
FROM ancestors a JOIN share_access sa ON sa.root_uid = a.uid FROM ancestors a JOIN share_access sa ON sa.root_uid = a.uid
ORDER BY a.depth ASC ORDER BY a.depth ASC
LIMIT 1", LIMIT 1",
[uid.to_string()], [&key],
|row| row.get::<_, String>(0), |row| row.get::<_, String>(0),
) )
.optional()?; .optional()?;
+86 -4
View File
@@ -25,8 +25,8 @@ pub struct StoredSyncFolder {
} }
/// One per-file sync baseline row: what a path looked like on both sides at the /// One per-file sync baseline row: what a path looked like on both sides at the
/// last successful sync, so the next reconcile can tell which side changed /// last successful sync, so the next reconcile can tell which side changed.
/// (devices.md Phase 2). `remote_rev`/`remote_hash` hold the remote signature — /// `remote_rev`/`remote_hash` hold the remote signature —
/// its modification time and size as strings — since no cheap content hash is /// its modification time and size as strings — since no cheap content hash is
/// exposed; change detection is `(mtime, size)` on each side. /// exposed; change detection is `(mtime, size)` on each side.
#[derive(Clone, Debug, PartialEq)] #[derive(Clone, Debug, PartialEq)]
@@ -39,7 +39,7 @@ pub struct StoredSyncEntry {
/// ///
/// `local_mtime` alone cannot tell an edit that landed inside the same /// `local_mtime` alone cannot tell an edit that landed inside the same
/// second as the last sync — and left the file the same length — from no /// second as the last sync — and left the file the same length — from no
/// edit at all, so those never uploaded (bugs.md B25). `None` on a row /// edit at all, so those never uploaded (`docs/BUGS.md` B25). `None` on a row
/// written before schema 28, which the comparison reads as "compare whole /// written before schema 28, which the comparison reads as "compare whole
/// seconds", so upgrading does not make an entire mirror look changed. /// seconds", so upgrading does not make an entire mirror look changed.
pub local_mtime_ns: Option<i64>, pub local_mtime_ns: Option<i64>,
@@ -260,10 +260,92 @@ impl Db {
Ok(()) Ok(())
} }
/// Drop the baseline rows of `rel_path` and everything under it.
///
/// Used when a subtree leaves a mirror folder through a move on Drive: its
/// local copy is about to be removed, and rows left behind would read as
/// "deleted locally" and trash the moved nodes at their new place.
pub fn sync_entries_remove_subtree(&self, folder_id: i64, rel_path: &str) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"DELETE FROM sync_entry
WHERE folder_id = ?1
AND (rel_path = ?2 OR substr(rel_path, 1, length(?2) + 1) = ?2 || '/')",
params![folder_id, rel_path],
)?;
Ok(())
}
/// Move the baseline rows of `from_rel` and everything under it to `to_rel`
/// in folder `to_folder`, in one transaction.
///
/// Used when a move on Drive took a mirror subtree along with its local
/// copy: the rows still describe both sides correctly, only under a new
/// path, so the next pass on either folder has nothing to do. The two
/// folders may be the same one.
pub fn sync_entries_move(
&self,
from_folder: i64,
from_rel: &str,
to_folder: i64,
to_rel: &str,
) -> Result<()> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
{
let mut select = tx.prepare(
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash,
local_mtime_ns
FROM sync_entry
WHERE folder_id = ?1
AND (rel_path = ?2 OR substr(rel_path, 1, length(?2) + 1) = ?2 || '/')",
)?;
let rows = select
.query_map(params![from_folder, from_rel], |r| {
Ok(StoredSyncEntry {
rel_path: r.get(0)?,
remote_uid: r.get(1)?,
local_mtime: r.get(2)?,
local_size: r.get(3)?,
remote_rev: r.get(4)?,
remote_hash: r.get(5)?,
local_mtime_ns: r.get(6)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
let mut delete =
tx.prepare("DELETE FROM sync_entry WHERE folder_id = ?1 AND rel_path = ?2")?;
for row in &rows {
delete.execute(params![from_folder, row.rel_path])?;
}
let mut insert = tx.prepare(
"INSERT OR REPLACE INTO sync_entry
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev,
remote_hash, local_mtime_ns)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
)?;
for row in rows {
let rel = format!("{to_rel}{}", &row.rel_path[from_rel.len()..]);
insert.execute(params![
to_folder,
rel,
row.remote_uid,
row.local_mtime,
row.local_size,
row.remote_rev,
row.remote_hash,
row.local_mtime_ns,
])?;
}
}
tx.commit()?;
Ok(())
}
/// Drop the entire baseline for a folder. Used when flipping ondemand→mirror: /// Drop the entire baseline for a folder. Used when flipping ondemand→mirror:
/// the local tree was evicted, so the old baseline is stale and would make the /// the local tree was evicted, so the old baseline is stale and would make the
/// next reconcile mistake "locally deleted" for "must re-download". Clearing it /// next reconcile mistake "locally deleted" for "must re-download". Clearing it
/// leaves an empty baseline + full remote = pure download (devices.md P3). /// leaves an empty baseline + full remote = pure download.
pub fn sync_entries_clear(&self, folder_id: i64) -> Result<()> { pub fn sync_entries_clear(&self, folder_id: i64) -> Result<()> {
let conn = self.conn.lock(); let conn = self.conn.lock();
conn.execute( conn.execute(
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -102,5 +102,5 @@ impl Db {
Ok(seen) Ok(seen)
} }
// ---- device sync (devices.md) ----------------------------------------- // ---- device sync -----------------------------------------
} }
+6 -6
View File
@@ -72,18 +72,18 @@ pub(super) fn path_of(conn: &Connection, uid: &str) -> Result<String> {
} }
} }
/// Resolve a path by walking `parent_uid` to the root via a recursive CTE. /// Resolve a path by walking `parent_lid` to the root via a recursive CTE.
/// ///
/// A cycle in `parent_uid` is corrupt data the API can still hand us, and /// A cycle in the parents is corrupt data the API can still hand us, and
/// `UNION ALL` over one never terminates — the walk is therefore depth-capped /// `UNION ALL` over one never terminates — the walk is therefore depth-capped
/// and returns the truncated path rather than hanging the caller. /// and returns the truncated path rather than hanging the caller.
pub(super) fn walk_path_of(conn: &Connection, uid: &str) -> Result<String> { pub(super) fn walk_path_of(conn: &Connection, uid: &str) -> Result<String> {
let mut stmt = conn.prepare(&format!( let mut stmt = conn.prepare(&format!(
"WITH RECURSIVE anc(uid, parent_uid, name, depth) AS ( "WITH RECURSIVE anc(parent_lid, parent_uid, name, depth) AS (
SELECT uid, parent_uid, name, 0 FROM nodes WHERE uid = ?1 SELECT parent_lid, parent_uid, name, 0 FROM nodes WHERE uid = ?1
UNION ALL UNION ALL
SELECT n.uid, n.parent_uid, n.name, anc.depth + 1 SELECT n.parent_lid, n.parent_uid, n.name, anc.depth + 1
FROM nodes n JOIN anc ON n.uid = anc.parent_uid FROM nodes n JOIN anc ON n.lid = anc.parent_lid
WHERE anc.depth < {MAX_PATH_DEPTH} WHERE anc.depth < {MAX_PATH_DEPTH}
) )
SELECT name, parent_uid FROM anc ORDER BY depth DESC" SELECT name, parent_uid FROM anc ORDER BY depth DESC"
+15 -3
View File
@@ -32,6 +32,16 @@ pub enum Error {
#[error("saved session can no longer unlock your keys — run `pdfs login` again")] #[error("saved session can no longer unlock your keys — run `pdfs login` again")]
ReloginRequired, ReloginRequired,
/// The account has a separate mailbox password that has not been entered
/// since login, so its keys cannot be unlocked. The fix is
/// [`auth::unlock`](crate::auth::unlock), not a new login.
#[error("account is locked — enter the mailbox password with `pdfs unlock`")]
Locked,
/// The mailbox password was checked against the account and is wrong.
#[error("that mailbox password does not unlock your account")]
WrongMailboxPassword,
/// The login was gated behind human verification. Not a failure the user can /// The login was gated behind human verification. Not a failure the user can
/// fix by retrying: the challenge has to be solved and the login restarted /// fix by retrying: the challenge has to be solved and the login restarted
/// with the resulting token (see [`auth::login_verified`](crate::auth::login_verified)). /// with the resulting token (see [`auth::login_verified`](crate::auth::login_verified)).
@@ -199,9 +209,11 @@ impl From<serde_json::Error> for CoreError {
impl From<Error> for CoreError { impl From<Error> for CoreError {
fn from(e: Error) -> Self { fn from(e: Error) -> Self {
let kind = match &e { let kind = match &e {
Error::NotLoggedIn | Error::ReloginRequired | Error::HumanVerificationRequired(_) => { Error::NotLoggedIn
ErrorKind::Denied | Error::ReloginRequired
} | Error::Locked
| Error::WrongMailboxPassword
| Error::HumanVerificationRequired(_) => ErrorKind::Denied,
Error::Proton(proton_sdk::ProtonError::Api(api)) => api_kind(api.code), Error::Proton(proton_sdk::ProtonError::Api(api)) => api_kind(api.code),
// A `ProtonError` that never reached the API is a transport failure. // A `ProtonError` that never reached the API is a transport failure.
Error::Proton(_) => ErrorKind::Offline, Error::Proton(_) => ErrorKind::Offline,
+204
View File
@@ -0,0 +1,204 @@
//! A secret held in the Linux kernel keyring, for as long as the user is logged in.
//!
//! This is where the key passphrases of a two-password account live by default:
//! the mailbox password is never stored, and the secrets derived from it are
//! kept in kernel memory — not on disk, not in the Secret Service — and vanish
//! when the user's last session ends or the machine reboots. Both the daemon and
//! `pdfs unlock` reach the same entry because it hangs off the *user* keyring.
//!
//! Kernel keys are not paged to swap, are invisible to other users, and are
//! wiped by [`clear`] (`KEYCTL_REVOKE`), which makes them a better home than a
//! process's heap. Processes of the same user can still read them, exactly as
//! they could ptrace the daemon: this protects against disk, backup and other
//! users, not against code already running as you.
//!
//! The keyring may be unavailable (a container, a kernel built without
//! `CONFIG_KEYS`); every operation then reports it as an error and the caller
//! falls back to asking again, never to storing the secret somewhere weaker.
use std::io;
use zeroize::Zeroizing;
/// Key description. Namespaced so it cannot collide with another program's key.
#[cfg(not(test))]
const DESCRIPTION: &[u8] = b"pdfs:key-passphrases\0";
/// Tests must never touch a real unlock.
#[cfg(test)]
const DESCRIPTION: &[u8] = b"pdfs:key-passphrases:test\0";
const KEY_TYPE: &[u8] = b"user\0";
const KEY_SPEC_USER_KEYRING: i32 = -4;
const KEYCTL_REVOKE: libc::c_long = 3;
const KEYCTL_SETPERM: libc::c_long = 5;
const KEYCTL_SEARCH: libc::c_long = 10;
const KEYCTL_READ: libc::c_long = 11;
/// Possessor and owner may view, read, write, search, link and set attributes;
/// group and others get nothing.
///
/// A new key's default grants the owner only *view*, and read access to the
/// possessor. A process possesses the user keyring only when its session
/// keyring links to it, which a systemd user service or a container does not
/// guarantee — so with the default, the daemon could store a key it then
/// cannot read back. The owner is the same user that could ptrace the daemon
/// anyway, so granting it read access gives up nothing.
const PERM_OWNER_AND_POSSESSOR: u32 = 0x3f3f_0000;
fn last_error<T>() -> io::Result<T> {
Err(io::Error::last_os_error())
}
fn find() -> io::Result<Option<libc::c_long>> {
// SAFETY: both strings are NUL-terminated and outlive the call.
let id = unsafe {
libc::syscall(
libc::SYS_keyctl,
KEYCTL_SEARCH,
KEY_SPEC_USER_KEYRING,
KEY_TYPE.as_ptr(),
DESCRIPTION.as_ptr(),
0,
)
};
if id >= 0 {
return Ok(Some(id));
}
let err = io::Error::last_os_error();
match err.raw_os_error() {
// ENOKEY: not stored. EKEYREVOKED/EKEYEXPIRED: stored once, now gone.
Some(libc::ENOKEY | libc::EKEYREVOKED | libc::EKEYEXPIRED) => Ok(None),
_ => Err(err),
}
}
/// Stop this process from being core-dumped or ptraced by an unprivileged
/// process of the same user, so the secrets in its memory stay out of crash
/// reports. Call once at start of a long-lived process that holds them.
pub fn forbid_dumps() {
// SAFETY: `PR_SET_DUMPABLE` takes plain integers and only affects this process.
let rc = unsafe { libc::prctl(libc::PR_SET_DUMPABLE, 0, 0, 0, 0) };
if rc != 0 {
tracing::warn!(error = %io::Error::last_os_error(), "could not mark the process non-dumpable");
}
}
/// Store `secret`, replacing any earlier one.
pub fn store(secret: &[u8]) -> io::Result<()> {
// SAFETY: strings are NUL-terminated; the payload pointer/length pair
// describes `secret`, which outlives the call.
let id = unsafe {
libc::syscall(
libc::SYS_add_key,
KEY_TYPE.as_ptr(),
DESCRIPTION.as_ptr(),
secret.as_ptr(),
secret.len(),
KEY_SPEC_USER_KEYRING,
)
};
if id < 0 {
return last_error();
}
// SAFETY: plain integer arguments.
let rc = unsafe {
libc::syscall(
libc::SYS_keyctl,
KEYCTL_SETPERM,
id,
PERM_OWNER_AND_POSSESSOR,
)
};
if rc < 0 {
// A key whose permissions are unknown is not one to leave behind.
let err = io::Error::last_os_error();
let _ = clear();
return Err(err);
}
Ok(())
}
/// Read the stored secret, or `None` when there is none.
pub fn load() -> io::Result<Option<Zeroizing<Vec<u8>>>> {
let Some(id) = find()? else {
return Ok(None);
};
// Size first, then read; retry if the key was replaced in between.
for _ in 0..3 {
// SAFETY: a null buffer asks the kernel only for the payload length.
let len = unsafe {
libc::syscall(
libc::SYS_keyctl,
KEYCTL_READ,
id,
std::ptr::null_mut::<u8>(),
0usize,
)
};
if len < 0 {
return match io::Error::last_os_error().raw_os_error() {
Some(libc::ENOKEY | libc::EKEYREVOKED | libc::EKEYEXPIRED) => Ok(None),
_ => last_error(),
};
}
let mut buf = Zeroizing::new(vec![0u8; len as usize]);
// SAFETY: `buf` is `len` writable bytes.
let read = unsafe {
libc::syscall(
libc::SYS_keyctl,
KEYCTL_READ,
id,
buf.as_mut_ptr(),
buf.len(),
)
};
if read < 0 {
return last_error();
}
if read as usize <= buf.len() {
buf.truncate(read as usize);
return Ok(Some(buf));
}
}
Err(io::Error::other("the key changed while it was being read"))
}
/// Remove the stored secret. Succeeds when there is none.
pub fn clear() -> io::Result<()> {
let Some(id) = find()? else {
return Ok(());
};
// SAFETY: plain integer arguments.
let rc = unsafe { libc::syscall(libc::SYS_keyctl, KEYCTL_REVOKE, id) };
if rc < 0 {
return match io::Error::last_os_error().raw_os_error() {
Some(libc::ENOKEY | libc::EKEYREVOKED) => Ok(()),
_ => last_error(),
};
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
// The kernel keyring is process-wide state and a sandbox may not offer it,
// so one test covers the whole round trip and skips when it is unavailable.
#[test]
fn stored_secret_round_trips_and_clears() {
if store(b"probe").is_err() {
eprintln!("kernel keyring unavailable; skipping");
return;
}
let saved = load().unwrap().expect("stored secret is readable");
assert_eq!(saved.as_slice(), b"probe");
store(b"replaced").unwrap();
assert_eq!(load().unwrap().unwrap().as_slice(), b"replaced");
clear().unwrap();
assert!(load().unwrap().is_none());
clear().unwrap();
}
}
+1
View File
@@ -10,6 +10,7 @@ pub mod config;
pub mod control; pub mod control;
pub mod db; pub mod db;
pub mod error; pub mod error;
pub mod kernelkey;
pub mod localindex; pub mod localindex;
pub mod menu; pub mod menu;
pub mod mounts; pub mod mounts;
+1 -1
View File
@@ -1,4 +1,4 @@
//! The machine profile that travels with a device (features.md 5.3). //! The machine profile that travels with a device.
//! //!
//! Sync folder mappings, pins, and the settings that shape them live in //! Sync folder mappings, pins, and the settings that shape them live in
//! `cache.db` and `config.json` — both on the local disk. When that disk dies, //! `cache.db` and `config.json` — both on the local disk. When that disk dies,
+10 -3
View File
@@ -55,9 +55,10 @@ fn systemctl_query(args: &[&str]) -> bool {
/// keyring checks). `--now` is a no-op on an active unit, so the daemon would /// keyring checks). `--now` is a no-op on an active unit, so the daemon would
/// keep sleeping and the mount would not appear until the next poll fired. /// keep sleeping and the mount would not appear until the next poll fired.
/// Restarting makes the new session take effect immediately, and starts the /// Restarting makes the new session take effect immediately, and starts the
/// unit when it is not running at all. /// unit when it is not running at all. `reenable` for the reason given on
/// [`enable`].
pub fn enable_start() -> bool { pub fn enable_start() -> bool {
let enabled = systemctl(&["enable"]); let enabled = systemctl(&["reenable"]);
systemctl(&["restart"]) && enabled systemctl(&["restart"]) && enabled
} }
@@ -97,8 +98,14 @@ pub fn is_enabled() -> bool {
/// Enable the service so it auto-starts on future logins, without necessarily /// Enable the service so it auto-starts on future logins, without necessarily
/// starting it now beyond what `enable` does. Used by the Settings "Start on /// starting it now beyond what `enable` does. Used by the Settings "Start on
/// login" switch; pairs with [`disable`](disable). /// login" switch; pairs with [`disable`](disable).
///
/// `reenable` rather than `enable`: older installs were enabled
/// into `default.target.wants`, and a plain `enable` would add the new
/// `graphical-session.target.wants` link while leaving the old one to start the
/// unit too early. `reenable` drops every existing link and recreates them from
/// the current `[Install]` section.
pub fn enable() -> bool { pub fn enable() -> bool {
systemctl(&["enable"]) systemctl(&["reenable"])
} }
/// Disable auto-start without stopping the running mount. Used by the Settings /// Disable auto-start without stopping the running mount. Used by the Settings
+1 -1
View File
@@ -1,4 +1,4 @@
//! Selective-sync ignore rules for mirror folders (features.md 1.1). //! Selective-sync ignore rules for mirror folders.
//! //!
//! A mirror folder otherwise uploads every path beneath its root, which for a //! A mirror folder otherwise uploads every path beneath its root, which for a
//! project directory means `node_modules/`, `.git/`, and `target/` — quota //! project directory means `node_modules/`, `.git/`, and `target/` — quota
+4
View File
@@ -15,6 +15,7 @@ zip.workspace = true
sha1.workspace = true sha1.workspace = true
proton-drive-rs.workspace = true proton-drive-rs.workspace = true
futures.workspace = true futures.workspace = true
async-trait.workspace = true
tokio.workspace = true tokio.workspace = true
tracing.workspace = true tracing.workspace = true
fuser.workspace = true fuser.workspace = true
@@ -24,3 +25,6 @@ base64.workspace = true
image.workspace = true image.workspace = true
notify = "8" notify = "8"
parking_lot.workspace = true parking_lot.workspace = true
[dev-dependencies]
tracing-subscriber.workspace = true
+105 -20
View File
@@ -9,7 +9,7 @@ use crate::shutdown::Shutdown;
/// ///
/// Returns whether this mount actually held the node, so the caller can evict /// Returns whether this mount actually held the node, so the caller can evict
/// its content blob once if *any* mount did. /// its content blob once if *any* mount did.
fn forget_and_notify(st: &mut State, notify: &mut NotifyBatch, uid: &NodeUid) -> bool { pub(crate) fn forget_and_notify(st: &mut State, notify: &mut NotifyBatch, uid: &NodeUid) -> bool {
// Capture the inode before `forget` clears the uid mapping. // Capture the inode before `forget` clears the uid mapping.
let child = st.by_uid.get(uid).copied(); let child = st.by_uid.get(uid).copied();
let Some((parent, name)) = st.forget(uid) else { let Some((parent, name)) = st.forget(uid) else {
@@ -97,15 +97,13 @@ fn event_serializes_shared_publication(event: &DriveEvent, own_volume: &VolumeId
/// entry forces the next `lookup`/`readdir` to re-enumerate from the remote, so /// entry forces the next `lookup`/`readdir` to re-enumerate from the remote, so
/// most events only need to invalidate listings rather than re-fetch eagerly. /// most events only need to invalidate listings rather than re-fetch eagerly.
/// ///
/// Applied to **every** mounted inode space, not just the primary one. This task /// Applied to the tree every mount shares, and told to **every** mounted
/// is per-daemon while node state is per-mount, and a `DriveEvent` names a uid, /// session. A `DriveEvent` names a uid, not a mount, so a file trashed from
/// not a mount — so a file trashed from another device has to be withdrawn from /// another device has to be withdrawn from whichever of our sessions is showing
/// whichever of our sessions is showing it, which for anything under a sync /// it — which for anything under a sync folder is a [`Core::fork_state`] fork.
/// folder is a [`Core::fork_state`] fork rather than `core.state`. Reaching for /// Telling the primary session alone left forks serving a deleted file, the
/// the primary state alone left forks serving a deleted file indefinitely, the /// read-side twin of `docs/BUGS.md` B74. [`Core::for_each_mount`] sends what
/// read-side twin of `docs/BUGS.md` B74. Inode numbers are per-mount, so each /// the kernel needs to hear to every session.
/// mount must be notified through its **own** channel; that pairing is what
/// [`Core::for_each_mount`] exists to preserve.
fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdfs_core::Result<()> { fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdfs_core::Result<()> {
let foreign_delete = is_foreign_node_delete(event, &core.primary_root_uid.volume_id); let foreign_delete = is_foreign_node_delete(event, &core.primary_root_uid.volume_id);
let serializes_shared_publication = let serializes_shared_publication =
@@ -145,7 +143,7 @@ fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdf
// create, and re-fetching would replace the size and mtime of the // create, and re-fetching would replace the size and mtime of the
// write we just accepted with the stale revision's — making a file // write we just accepted with the stale revision's — making a file
// that was copied in seconds ago read as empty until its upload // that was copied in seconds ago read as empty until its upload
// lands (offline.md Phase 3). // lands.
// //
// A write that has already drained is the same story one step later: // A write that has already drained is the same story one step later:
// the drain brought the tree level with the revision it sealed, so // the drain brought the tree level with the revision it sealed, so
@@ -233,6 +231,48 @@ fn acknowledge_applied_event(
Ok(id) Ok(id)
} }
/// How many applied node-update ids [`RecentEvents`] remembers. Far more than a
/// replay can reach back: one comes from a batch whose last event was not its
/// newest, so it repeats at most what that batch held.
const RECENT_EVENTS: usize = 4096;
/// The node updates applied lately, by event id, so one that comes round again
/// is skipped rather than applied twice.
///
/// The feed can deliver the same event twice in a batch, and a batch whose
/// events come out of order leaves the cursor short of its newest event, so the
/// next poll replays what lay past it. An update applied twice is not harmless:
/// the first copy of our own change claims the echo [`Core::note_self_change`]
/// recorded, and the second, unclaimed, reads as someone else's edit and evicts
/// the bytes a partial write is about to gap-fill from (docs/BUGS.md B123).
///
/// Only node updates: a delete is idempotent, and a continuity loss names the
/// cursor it lost, which is an id already applied.
#[derive(Default)]
struct RecentEvents {
ids: HashSet<DriveEventId>,
order: VecDeque<DriveEventId>,
}
impl RecentEvents {
/// Record `event` as applied, reporting whether it was already.
fn repeat(&mut self, event: &DriveEvent) -> bool {
let DriveEvent::NodeUpdated { id, .. } = event else {
return false;
};
if !self.ids.insert(id.clone()) {
return true;
}
self.order.push_back(id.clone());
if self.order.len() > RECENT_EVENTS
&& let Some(old) = self.order.pop_front()
{
self.ids.remove(&old);
}
false
}
}
/// The folders one batch of events changed, collected so the listings are /// The folders one batch of events changed, collected so the listings are
/// dropped once at the end of the batch rather than once per event. /// dropped once at the end of the batch rather than once per event.
/// ///
@@ -320,10 +360,10 @@ fn flush_dirty_parents(core: &Core, dirty: &DirtyParents) {
/// only on fatal error. /// only on fatal error.
/// ///
/// Takes the whole `Core` rather than the primary mount's pieces: an event has /// Takes the whole `Core` rather than the primary mount's pieces: an event has
/// to be applied to every mounted inode space, and the registry that enumerates /// to reach every mounted session, and the registry that enumerates them lives
/// them lives on the `Core` (see [`apply_event`]). /// on the `Core` (see [`apply_event`]).
pub(super) async fn run_event_sync( pub(super) async fn run_event_sync(
client: ProtonDriveClient, client: Arc<dyn DriveApi>,
scope: DriveEventScopeId, scope: DriveEventScopeId,
core: Core, core: Core,
) { ) {
@@ -335,7 +375,7 @@ pub(super) async fn run_event_sync(
// server head; persist it so the next restart resumes instead of // server head; persist it so the next restart resumes instead of
// reseeding (which would skip everything that changed offline). // reseeding (which would skip everything that changed offline).
// Seeding needs the network, and this task also runs on mounts that // Seeding needs the network, and this task also runs on mounts that
// started offline (offline.md Phase 1) — so retry rather than giving up, // started offline — so retry rather than giving up,
// which used to disable live sync for the life of the daemon. // which used to disable live sync for the life of the daemon.
Ok(None) => { Ok(None) => {
// Nothing says what changed before this cursor, so a persisted SDK // Nothing says what changed before this cursor, so a persisted SDK
@@ -377,10 +417,12 @@ pub(super) async fn run_event_sync(
}; };
info!(?cursor, "event sync started"); info!(?cursor, "event sync started");
let mut recent = RecentEvents::default();
loop { loop {
tokio::time::sleep(POLL_INTERVAL).await; core.next_poll(POLL_INTERVAL).await;
let events = match client.enumerate_events(&scope, cursor.as_ref()).await { let events = match link::bounded(client.enumerate_events(&scope, cursor.as_ref())).await {
Ok(events) => events, Ok(events) => events,
Err(e) if core.lost_link(&e, "poll events") => continue,
Err(e) => { Err(e) => {
warn!(error = %e, "event poll failed; retrying after interval"); warn!(error = %e, "event poll failed; retrying after interval");
continue; continue;
@@ -396,6 +438,11 @@ pub(super) async fn run_event_sync(
// did apply. // did apply.
let mut dirty = DirtyParents::default(); let mut dirty = DirtyParents::default();
for event in &events { for event in &events {
if recent.repeat(event) {
debug!(id = %event.id().as_str(), "skipping an event already applied");
cursor = Some(event.id().clone());
continue;
}
// Converge the SDK's own caches (folder keys, entity cache) on the // Converge the SDK's own caches (folder keys, entity cache) on the
// server before applying the event to our tree. Without this, a node // server before applying the event to our tree. Without this, a node
// re-keyed/moved by another client keeps a stale key in the SDK for // re-keyed/moved by another client keeps a stale key in the SDK for
@@ -471,9 +518,10 @@ pub(super) async fn run_photos_event_sync(core: Core) {
info!(?cursor, "photo event sync started"); info!(?cursor, "photo event sync started");
loop { loop {
tokio::time::sleep(POLL_INTERVAL).await; core.next_poll(POLL_INTERVAL).await;
let events = match photos.enumerate_events(&scope, cursor.as_ref()).await { let events = match link::bounded(photos.enumerate_events(&scope, cursor.as_ref())).await {
Ok(events) => events, Ok(events) => events,
Err(error) if core.lost_link(&error, "poll photo events") => continue,
Err(error) => { Err(error) => {
warn!(%error, "photo event poll failed; retrying after interval"); warn!(%error, "photo event poll failed; retrying after interval");
continue; continue;
@@ -612,7 +660,7 @@ pub(super) fn run_local_index(
scan_local_once(&db, &indexing, &transfers, &mountpoint); scan_local_once(&db, &indexing, &transfers, &mountpoint);
} }
// Interruptible, so teardown joins this thread instead of leaving it to // Interruptible, so teardown joins this thread instead of leaving it to
// keep rewriting the index for a mount that is gone (bugs.md B44). // keep rewriting the index for a mount that is gone (`docs/BUGS.md` B44).
if !shutdown.sleep(LOCAL_INDEX_CHECK) { if !shutdown.sleep(LOCAL_INDEX_CHECK) {
return; return;
} }
@@ -715,6 +763,43 @@ mod tests {
} }
} }
#[test]
fn a_node_update_seen_again_is_skipped_but_other_events_are_not() {
let update = |id: &str| DriveEvent::NodeUpdated {
id: DriveEventId::from(id),
node_uid: uid("file"),
parent_node_uid: None,
is_trashed: false,
is_shared: false,
};
let mut recent = RecentEvents::default();
assert!(!recent.repeat(&update("e1")));
assert!(!recent.repeat(&update("e2")));
assert!(recent.repeat(&update("e1")), "a duplicate or replay");
let lost = DriveEvent::ContinuityLost {
id: DriveEventId::from("e2"),
};
assert!(!recent.repeat(&lost), "names the cursor, which was applied");
}
#[test]
fn recent_events_forget_the_oldest_past_their_bound() {
let update = |n: usize| DriveEvent::NodeUpdated {
id: DriveEventId::from(format!("e{n}")),
node_uid: uid("file"),
parent_node_uid: None,
is_trashed: false,
is_shared: false,
};
let mut recent = RecentEvents::default();
for n in 0..=RECENT_EVENTS {
assert!(!recent.repeat(&update(n)));
}
assert_eq!(recent.ids.len(), RECENT_EVENTS);
assert!(!recent.repeat(&update(0)), "the oldest was forgotten");
assert!(recent.repeat(&update(RECENT_EVENTS)));
}
#[test] #[test]
fn foreign_delete_invalidates_an_earlier_refresh_generation() { fn foreign_delete_invalidates_an_earlier_refresh_generation() {
let own_volume = VolumeId::from("own"); let own_volume = VolumeId::from("own");
+103 -29
View File
@@ -163,8 +163,8 @@ fn read_request_line(reader: &mut impl BufRead) -> std::io::Result<Option<String
/// Turn a CLI-supplied path into the mount that owns it and the path relative /// Turn a CLI-supplied path into the mount that owns it and the path relative
/// to *that* mount's root. /// to *that* mount's root.
/// ///
/// [`rel_to_mount`] is the older, primary-only reading, and it is what every /// [`rel_to_mount`] is the older, primary-only reading, and it is what most
/// mutating request still uses. It rejects anything outside the primary /// requests still use. It rejects anything outside the primary
/// mountpoint, which leaves a secondary on-demand mount — a synced folder /// mountpoint, which leaves a secondary on-demand mount — a synced folder
/// switched out of mirror mode — with no way to be named at all: `pdfs ls` and /// switched out of mirror mode — with no way to be named at all: `pdfs ls` and
/// `pdfs refresh` both answered "is not under the mountpoint", so a folder /// `pdfs refresh` both answered "is not under the mountpoint", so a folder
@@ -185,11 +185,14 @@ fn route_to_mount(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
rel_to_mount(mountpoint, path).map(|rel| (core.clone(), rel)) rel_to_mount(mountpoint, path).map(|rel| (core.clone(), rel))
} }
/// [`route_to_mount`] for a conflict copy. The listing names a copy inside a /// [`route_to_mount`] for a request that changes what it names: trash, rename,
/// synced folder by its absolute local path (`docs/BUGS.md` B99), so the /// create a folder, resolve a conflict. Each has to run against the on-demand
/// resolve has to run against the on-demand mount serving it. A mirror folder /// mount serving the path, and `pdfs rm ~/Downloads/x` answered "is not under
/// has no mount to run against: its copy is an ordinary local file. /// the mountpoint" while `pdfs move` took the same path. The conflict listing
fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> { /// names a copy in a synced folder by its absolute local path (`docs/BUGS.md`
/// B99), so resolving one needs this too. A mirror folder has no mount to run
/// against: a file there is an ordinary local file.
fn route_change(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
let routed = route_to_mount(core, mountpoint, path).map_err(|e| { let routed = route_to_mount(core, mountpoint, path).map_err(|e| {
let p = Path::new(path); let p = Path::new(path);
let mirrored = core let mirrored = core
@@ -200,7 +203,7 @@ fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
.any(|folder| folder.mode == "mirror" && p.starts_with(&folder.local_path)); .any(|folder| folder.mode == "mirror" && p.starts_with(&folder.local_path));
if mirrored { if mirrored {
CoreError::invalid(format!( CoreError::invalid(format!(
"{path} is in a mirrored folder; rename or delete the local file there" "{path} is in a mirrored folder; change the local file there instead"
)) ))
} else { } else {
e e
@@ -217,6 +220,27 @@ fn route_conflict(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Cor
} }
} }
/// [`route_change`] for a request that trashes or renames the node it names.
/// A mount's root is My files or a whole synced folder. An on-demand folder's
/// root has a parent on Drive, so nothing further down would stop `pdfs rm
/// ~/Downloads` from trashing the folder out from under its own mount.
fn route_node(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
let routed = route_change(core, mountpoint, path)?;
if below_root(&routed.1) {
Ok(routed)
} else {
Err(CoreError::invalid(format!(
"{path} is the root of a synced location and cannot be changed"
)))
}
}
/// Whether a mount-relative path names something below the mount's root.
fn below_root(rel: &Path) -> bool {
rel.components()
.any(|component| matches!(component, Component::Normal(_)))
}
/// Turn a CLI-supplied path into a mountpoint-relative path. An absolute path /// Turn a CLI-supplied path into a mountpoint-relative path. An absolute path
/// must live under `mountpoint`; a relative path is taken as already relative to /// must live under `mountpoint`; a relative path is taken as already relative to
/// the mount root. /// the mount root.
@@ -291,6 +315,9 @@ impl Drop for RefreshGuard {
pub(crate) fn status_response(core: &Core, username: &str, mountpoint: &Path) -> CtlResponse { pub(crate) fn status_response(core: &Core, username: &str, mountpoint: &Path) -> CtlResponse {
let pins = core.cache.list_pins(); let pins = core.cache.list_pins();
let queued = core.db.pending_op_counts().unwrap_or_default(); let queued = core.db.pending_op_counts().unwrap_or_default();
// Read after the queue: the drain counts an upload as landing before it
// retires the op, so one of the two always has it (B119).
let landing = core.landing_uploads.load(Ordering::SeqCst);
let staging = core.cache.staging_usage(); let staging = core.cache.staging_usage();
CtlResponse::Status { CtlResponse::Status {
parked_uploads: queued.parked.max(0) as u64, parked_uploads: queued.parked.max(0) as u64,
@@ -307,7 +334,7 @@ pub(crate) fn status_response(core: &Core, username: &str, mountpoint: &Path) ->
budget: core.cache.budget(), budget: core.cache.budget(),
pins, pins,
online: core.online.load(Ordering::Relaxed), online: core.online.load(Ordering::Relaxed),
pending_uploads: queued.uploads.max(0) as u64, pending_uploads: queued.uploads.max(0) as u64 + landing,
pending_changes: queued.changes.max(0) as u64, pending_changes: queued.changes.max(0) as u64,
} }
} }
@@ -350,6 +377,7 @@ fn touched_topics(request: &CtlRequest) -> &'static [Topic] {
| R::RestoreSyncFolders { .. } | R::RestoreSyncFolders { .. }
| R::RestoreDeviceFolders { .. } => &[Topic::Locations, Topic::Devices], | R::RestoreDeviceFolders { .. } => &[Topic::Locations, Topic::Devices],
R::RetryPendingOp { .. } => &[Topic::Queue], R::RetryPendingOp { .. } => &[Topic::Queue],
R::DiscardPendingOp { .. } => &[Topic::Queue, Topic::Files],
R::ResolveConflict { .. } => &[Topic::Files, Topic::Trash, Topic::Conflicts], R::ResolveConflict { .. } => &[Topic::Files, Topic::Trash, Topic::Conflicts],
R::ShareNode { .. } R::ShareNode { .. }
| R::ShareNodeByUid { .. } | R::ShareNodeByUid { .. }
@@ -959,8 +987,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
}, },
Err(e) => CtlResponse::error(e), Err(e) => CtlResponse::error(e),
}, },
Ok(CtlRequest::Rename { path, new_name }) => match rel_to_mount(mountpoint, &path) { Ok(CtlRequest::Rename { path, new_name }) => match route_node(core, mountpoint, &path) {
Ok(rel) => match core.rename(&rel, &new_name) { Ok((core, rel)) => match core.rename(&rel, &new_name) {
Ok(name) => { Ok(name) => {
core.log_activity(ActivityKind::Rename, &name, format!("was {path}"), true); core.log_activity(ActivityKind::Rename, &name, format!("was {path}"), true);
CtlResponse::Ok { CtlResponse::Ok {
@@ -974,6 +1002,24 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
}, },
Err(e) => CtlResponse::error(e), Err(e) => CtlResponse::error(e),
}, },
// Two absolute paths may name two different locations, which only
// `move_between` can resolve; a relative pair keeps the old meaning.
Ok(CtlRequest::Move { path, new_parent })
if Path::new(&path).is_absolute() && Path::new(&new_parent).is_absolute() =>
{
match core.move_between(Path::new(&path), Path::new(&new_parent)) {
Ok(name) => {
core.log_activity(ActivityKind::Move, &name, format!("to {new_parent}"), true);
CtlResponse::Ok {
message: format!("moved {name}"),
}
}
Err(e) => {
core.log_activity(ActivityKind::Move, &path, &e, false);
CtlResponse::error(e)
}
}
}
Ok(CtlRequest::Move { path, new_parent }) => { Ok(CtlRequest::Move { path, new_parent }) => {
match ( match (
rel_to_mount(mountpoint, &path), rel_to_mount(mountpoint, &path),
@@ -999,8 +1045,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
(Err(e), _) | (_, Err(e)) => CtlResponse::error(e), (Err(e), _) | (_, Err(e)) => CtlResponse::error(e),
} }
} }
Ok(CtlRequest::Delete { path }) => match rel_to_mount(mountpoint, &path) { Ok(CtlRequest::Delete { path }) => match route_node(core, mountpoint, &path) {
Ok(rel) => match core.delete(&rel) { Ok((core, rel)) => match core.delete(&rel) {
Ok(name) => { Ok(name) => {
core.log_activity(ActivityKind::Trash, &name, "", true); core.log_activity(ActivityKind::Trash, &name, "", true);
CtlResponse::Ok { CtlResponse::Ok {
@@ -1014,21 +1060,23 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
}, },
Err(e) => CtlResponse::error(e), Err(e) => CtlResponse::error(e),
}, },
Ok(CtlRequest::CreateFolder { parent, name }) => match rel_to_mount(mountpoint, &parent) { Ok(CtlRequest::CreateFolder { parent, name }) => {
Ok(parent_rel) => match core.create_folder(&parent_rel, &name) { match route_change(core, mountpoint, &parent) {
Ok(name) => { Ok((core, parent_rel)) => match core.create_folder(&parent_rel, &name) {
core.log_activity(ActivityKind::CreateFolder, &name, "", true); Ok(name) => {
CtlResponse::Ok { core.log_activity(ActivityKind::CreateFolder, &name, "", true);
message: format!("created folder {name}"), CtlResponse::Ok {
message: format!("created folder {name}"),
}
} }
} Err(e) => {
Err(e) => { core.log_activity(ActivityKind::CreateFolder, &name, &e, false);
core.log_activity(ActivityKind::CreateFolder, &name, &e, false); CtlResponse::error(e)
CtlResponse::error(e) }
} },
}, Err(e) => CtlResponse::error(e),
Err(e) => CtlResponse::error(e), }
}, }
Ok(CtlRequest::UploadPaths { parent, sources }) => { Ok(CtlRequest::UploadPaths { parent, sources }) => {
match rel_to_mount(mountpoint, &parent) { match rel_to_mount(mountpoint, &parent) {
// Ack immediately and upload on a background thread: a directory tree // Ack immediately and upload on a background thread: a directory tree
@@ -1345,7 +1393,7 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
Err(e) => CtlResponse::error(e), Err(e) => CtlResponse::error(e),
}, },
Ok(CtlRequest::ResolveConflict { path, keep }) => { Ok(CtlRequest::ResolveConflict { path, keep }) => {
match route_conflict(core, mountpoint, &path) { match route_change(core, mountpoint, &path) {
Ok((core, rel)) => match core.resolve_conflict(&rel, &keep) { Ok((core, rel)) => match core.resolve_conflict(&rel, &keep) {
Ok(message) => CtlResponse::Ok { message }, Ok(message) => CtlResponse::Ok { message },
Err(e) => CtlResponse::error(e), Err(e) => CtlResponse::error(e),
@@ -1357,6 +1405,20 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
Ok(items) => CtlResponse::PendingOps { items }, Ok(items) => CtlResponse::PendingOps { items },
Err(e) => CtlResponse::error(e), Err(e) => CtlResponse::error(e),
}, },
Ok(CtlRequest::ExportPendingOp { id, dest }) => {
match core.export_pending_op(id, Path::new(&dest)) {
Ok(written) => CtlResponse::Ok {
message: format!("wrote {} to {dest}", human_bytes(written)),
},
Err(e) => CtlResponse::error(e),
}
}
Ok(CtlRequest::DiscardPendingOp { id }) => match core.discard_pending_op(id) {
Ok(path) => CtlResponse::Ok {
message: format!("discarded the queued change to {path}"),
},
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::RetryPendingOp { id }) => match core.retry_pending_ops(id) { Ok(CtlRequest::RetryPendingOp { id }) => match core.retry_pending_ops(id) {
Ok(0) if id.is_some() => CtlResponse::Ok { Ok(0) if id.is_some() => CtlResponse::Ok {
message: "nothing to retry: the operation is already due or waiting for a rename" message: "nothing to retry: the operation is already due or waiting for a rename"
@@ -1754,7 +1816,7 @@ pub(crate) fn run_control_socket(
info!("control socket listening"); info!("control socket listening");
for conn in listener.incoming() { for conn in listener.incoming() {
// Checked on every accept, including the deliberate poke teardown sends // Checked on every accept, including the deliberate poke teardown sends
// to wake this thread out of a blocking accept (bugs.md B44). // to wake this thread out of a blocking accept (`docs/BUGS.md` B44).
if core.shutdown.is_stopping() { if core.shutdown.is_stopping() {
info!("control socket closing"); info!("control socket closing");
return; return;
@@ -1846,6 +1908,18 @@ mod request_limit_tests {
assert!(rel_to_mount(mountpoint, "Photos/../../escape").is_err()); assert!(rel_to_mount(mountpoint, "Photos/../../escape").is_err());
assert!(rel_to_mount(mountpoint, "/mnt/elsewhere").is_err()); assert!(rel_to_mount(mountpoint, "/mnt/elsewhere").is_err());
} }
/// `pdfs rm` and `pdfs rename` reach on-demand mounts, whose root is a whole
/// synced folder with a parent on Drive. Only a path below a root may get
/// through to the trash or rename call.
#[test]
fn only_a_path_below_a_mount_root_names_a_node() {
assert!(below_root(Path::new("Photos")));
assert!(below_root(Path::new("./Photos/2026")));
assert!(!below_root(Path::new("")));
assert!(!below_root(Path::new(".")));
assert!(!below_root(Path::new("./.")));
}
} }
#[cfg(test)] #[cfg(test)]
+20 -32
View File
@@ -1,5 +1,5 @@
//! This machine as a Proton Drive Device, and the local folders it keeps in //! This machine as a Proton Drive Device, and the local folders it keeps in
//! step with the remote (devices.md). //! step with the remote.
//! //!
//! A sync folder runs in one of two modes. `mirror` keeps a real local tree that //! A sync folder runs in one of two modes. `mirror` keeps a real local tree that
//! the engine in [`super::sync`] reconciles both ways; `ondemand` evicts that //! the engine in [`super::sync`] reconciles both ways; `ondemand` evicts that
@@ -27,7 +27,7 @@ use tracing::{info, warn};
use super::sync::{self, base_name}; use super::sync::{self, base_name};
use super::{ use super::{
Core, SecondaryInsertRejection, SecondaryMount, State, SwitchBlocked, clear_stale_mount, Core, SecondaryInsertRejection, SecondaryMount, SwitchBlocked, clear_stale_mount,
device_type_str, dir_is_empty, evict_dir_contents, fuse_connection_id, is_stale_mount, device_type_str, dir_is_empty, evict_dir_contents, fuse_connection_id, is_stale_mount,
now_secs, parse_uid, spawn_session, sync_folder_info, this_hostname, now_secs, parse_uid, spawn_session, sync_folder_info, this_hostname,
}; };
@@ -184,8 +184,8 @@ impl Core {
} }
/// Adopt an existing device as *this* machine's, pinning it in `config.json` /// Adopt an existing device as *this* machine's, pinning it in `config.json`
/// so hostname changes and reinstalls stop registering duplicates /// so hostname changes and reinstalls stop registering duplicates.
/// (features.md 5.1). `None` clears the pin and returns to hostname matching. /// `None` clears the pin and returns to hostname matching.
/// ///
/// Refuses a uid that is not in the account: the whole point of the pin is /// Refuses a uid that is not in the account: the whole point of the pin is
/// that it is authoritative, so accepting a bad one would defer the failure /// that it is authoritative, so accepting a bad one would defer the failure
@@ -260,7 +260,7 @@ impl Core {
}) })
} }
// ---- device folder sync (devices.md, Phase 1) ------------------------- // ---- device folder sync -------------------------
/// Auto-register (or recover) this machine as a Proton Drive Device, caching /// Auto-register (or recover) this machine as a Proton Drive Device, caching
/// it so restarts reuse the same device. Recovery matches an existing remote /// it so restarts reuse the same device. Recovery matches an existing remote
@@ -270,8 +270,7 @@ impl Core {
/// Resolution order: the *adopted* uid pinned in `config.json`, then the /// Resolution order: the *adopted* uid pinned in `config.json`, then the
/// cached DB row (validated remotely), then a hostname match, then create. /// cached DB row (validated remotely), then a hostname match, then create.
/// The pin exists because the hostname heuristic silently registers a second /// The pin exists because the hostname heuristic silently registers a second
/// device after a rename or reinstall, orphaning the first one's folders /// device after a rename or reinstall, orphaning the first one's folders.
/// (features.md 5.1).
pub(crate) fn ensure_device(&self) -> CoreResult<StoredDevice> { pub(crate) fn ensure_device(&self) -> CoreResult<StoredDevice> {
let name = this_hostname(); let name = this_hostname();
// Enumerate the remote devices once: used to validate the pin, validate // Enumerate the remote devices once: used to validate the pin, validate
@@ -363,14 +362,14 @@ impl Core {
) -> CoreResult<Option<NodeUid>> { ) -> CoreResult<Option<NodeUid>> {
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(root_uid)) .block_on(self.drive.enumerate_folder_children_node_uids(root_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?; .map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() { if uids.is_empty() {
return Ok(None); return Ok(None);
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes(&uids)) .block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?; .map_err(|e| CoreError::from_api(&e, "resolve device root children"))?;
Ok(nodes Ok(nodes
.into_iter() .into_iter()
@@ -446,10 +445,7 @@ impl Core {
} }
None => self None => self
.rt .rt
.block_on( .block_on(self.drive.create_folder(&root_uid, &name, Some(now_secs())))
self.client
.create_folder(&root_uid, &name, Some(now_secs())),
)
.map_err(|e| CoreError::from_api(&e, &format!("create device folder {name}")))?, .map_err(|e| CoreError::from_api(&e, &format!("create device folder {name}")))?,
}; };
@@ -603,7 +599,7 @@ impl Core {
&& let Some(uid) = parse_uid(&folder.remote_uid) && let Some(uid) = parse_uid(&folder.remote_uid)
&& let Err(e) = self && let Err(e) = self
.rt .rt
.block_on(self.client.trash_nodes(&[uid])) .block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
warn!(id, error = %e, "delete remote device folder failed"); warn!(id, error = %e, "delete remote device folder failed");
@@ -619,23 +615,16 @@ impl Core {
}; };
} }
/// A sibling Core that shares this one's client/rt/cache/db (and transfer, /// A sibling Core for a secondary FUSE session rooted at an `ondemand` sync
/// activity, mount registries) but gets a **fresh, empty `State`** — its own /// folder. It shares everything with this one,
/// inode space starting at [`ROOT_INO`]. Used to root a secondary FUSE session /// the tree included, except what belongs to a session: its root and its
/// at an `ondemand` sync folder without colliding with the main mount's inodes /// kernel channel, both filled in when the session is built.
/// (devices.md Phase 3).
pub(crate) fn fork_state(&self) -> Core { pub(crate) fn fork_state(&self) -> Core {
let mut fork = self.clone(); let mut fork = self.clone();
fork.primary = false; fork.primary = false;
let share_access = self.state().share_access.clone(); fork.root = Arc::new(OnceLock::new());
fork.state = Arc::new(Mutex::new(State::new(self.db.clone(), share_access, 2)));
// A fresh inode space needs a fresh notification channel: this fork's
// session is the only one that knows these inodes, so it must be the one
// notified about them. Filled in by `spawn_ondemand_mount`.
fork.notifier = Arc::new(OnceLock::new()); fork.notifier = Arc::new(OnceLock::new());
fork.session_live = Arc::new(std::sync::atomic::AtomicBool::new(false)); fork.session_live = Arc::new(std::sync::atomic::AtomicBool::new(false));
// Size upgrades are keyed by inode, which is per-fork too.
fork.size_upgrades = Arc::new(Mutex::new(HashMap::new()));
fork fork
} }
@@ -858,7 +847,7 @@ impl Core {
})?; })?;
let root = self let root = self
.rt .rt
.block_on(self.client.enumerate_nodes(std::slice::from_ref(&root_uid))) .block_on(self.drive.enumerate_nodes(std::slice::from_ref(&root_uid)))
.map_err(|e| SwitchBlocked::Failed(format!("fetch remote root: {e}")))? .map_err(|e| SwitchBlocked::Failed(format!("fetch remote root: {e}")))?
.into_iter() .into_iter()
.next() .next()
@@ -937,8 +926,8 @@ impl Core {
} }
} }
/// Spawn a secondary FUSE session rooted at `root` over `local` on a forked /// Spawn a secondary FUSE session rooted at `root` over `local` from a
/// inode space. Clears any stale kernel mount first (a crashed run can leave /// [`Core::fork_state`] sibling. Clears any stale kernel mount first (a crashed run can leave
/// one, which would fail the fresh mount with EBUSY). /// one, which would fail the fresh mount with EBUSY).
pub(crate) fn spawn_ondemand_mount( pub(crate) fn spawn_ondemand_mount(
&self, &self,
@@ -996,8 +985,7 @@ impl Core {
/// restart (their local dirs are empty on disk — the files live in the cloud). /// restart (their local dirs are empty on disk — the files live in the cloud).
/// Best-effort per folder: a missing local path or a failed remote fetch marks /// Best-effort per folder: a missing local path or a failed remote fetch marks
/// the folder `error` and moves on rather than aborting the rest. Runs on its /// the folder `error` and moves on rather than aborting the rest. Runs on its
/// own thread from `mount` so the network fetches never block startup /// own thread from `mount` so the network fetches never block startup.
/// (devices.md Phase 4).
pub(crate) fn restore_ondemand_mounts(&self) { pub(crate) fn restore_ondemand_mounts(&self) {
let folders = match self.db.sync_folder_list() { let folders = match self.db.sync_folder_list() {
Ok(f) => f, Ok(f) => f,
@@ -1017,7 +1005,7 @@ impl Core {
}; };
let root = match self let root = match self
.rt .rt
.block_on(self.client.enumerate_nodes(std::slice::from_ref(&root_uid))) .block_on(self.drive.enumerate_nodes(std::slice::from_ref(&root_uid)))
{ {
Ok(v) => match v.into_iter().next() { Ok(v) => match v.into_iter().next() {
Some(n) => n, Some(n) => n,
File diff suppressed because it is too large. Load diff
+473
View File
@@ -0,0 +1,473 @@
//! The calls the sync core makes to Drive, behind a trait.
//!
//! `Core` used to hold a concrete [`ProtonDriveClient`], so no test could make
//! Drive answer late, answer stale, or not answer at all, and the interleavings
//! behind most recent bugs could only be found on a real account over a slow
//! link. [`DriveApi`] is the part of the
//! client the mount, the drain, the event feed and the mirror engine use; the
//! real client implements it by forwarding, and the simulation tests implement
//! it with an in-memory Drive that injects Drive's known faults.
//!
//! Sharing, Photos, devices, invitations and public links stay on the concrete
//! client (`Core::client`) until a test needs them faked.
//!
//! The method names match the client's own, so a call site reads the same
//! whichever it goes through. The client's generic methods (a reader or writer
//! of any type) cannot be part of an object-safe trait, so the trait takes them
//! as trait objects and `impl dyn DriveApi` puts the generic signatures back.
use std::io::{Read, Write};
use std::sync::Arc;
use async_trait::async_trait;
use futures::StreamExt as _;
use futures::stream::BoxStream;
use proton_drive_rs::proton_sdk::account::Quota;
use proton_drive_rs::proton_sdk::error::Result;
use proton_drive_rs::proton_sdk::ids::{DriveEventId, NodeUid};
use proton_drive_rs::{
DriveEvent, DriveEventScopeId, Node, NodeMoveItem, ProtonDriveClient, RevisionReader, Thumbnail,
};
/// One outcome per node of a batch request; see [`ProtonDriveClient::trash_nodes`].
pub(crate) type NodeOutcomes = Vec<(NodeUid, Result<()>)>;
/// The outcomes of a streaming batch request, one item per node as it lands.
pub(crate) type OutcomeStream<'a> = BoxStream<'a, Result<(NodeUid, Result<()>)>>;
/// Drive as the sync core sees it. See the module documentation.
#[async_trait]
pub(crate) trait DriveApi: Send + Sync {
async fn get_my_files_folder(&self) -> Result<Node>;
async fn get_node(&self, uid: &NodeUid) -> Result<Option<Node>>;
async fn enumerate_nodes(&self, uids: &[NodeUid]) -> Result<Vec<Node>>;
/// As [`DriveApi::enumerate_nodes`], but files carry no claimed size or
/// modification time; see [`ProtonDriveClient::enumerate_nodes_light`].
async fn enumerate_nodes_light(&self, uids: &[NodeUid]) -> Result<Vec<Node>>;
async fn enumerate_folder_children_node_uids(
&self,
folder_uid: &NodeUid,
) -> Result<Vec<NodeUid>>;
async fn enumerate_events(
&self,
scope: &DriveEventScopeId,
cursor: Option<&DriveEventId>,
) -> Result<Vec<DriveEvent>>;
async fn invalidate_caches_for_event(&self, event: &DriveEvent) -> Result<()>;
async fn create_folder(
&self,
parent_uid: &NodeUid,
name: &str,
last_modification_time: Option<i64>,
) -> Result<NodeUid>;
async fn upload_file(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
contents: &[u8],
) -> Result<NodeUid>;
#[allow(clippy::too_many_arguments)]
async fn upload_file_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid>;
#[allow(clippy::too_many_arguments)]
async fn upload_file_replacing_draft_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid>;
async fn upload_new_revision_from_dyn(
&self,
file_uid: &NodeUid,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()>;
async fn open_revision(&self, uid: &NodeUid) -> Result<Arc<dyn RevisionRead>>;
async fn download_file_to_dyn(
&self,
uid: &NodeUid,
output: &mut (dyn Write + Send),
) -> Result<()>;
async fn download_revision_to_dyn(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut (dyn Write + Send),
) -> Result<()>;
async fn rename_node(
&self,
uid: &NodeUid,
new_name: &str,
new_media_type: Option<&str>,
) -> Result<()>;
async fn move_node(&self, uid: &NodeUid, new_parent: &NodeUid) -> Result<()>;
fn move_nodes_streaming<'a>(
&'a self,
items: Vec<NodeMoveItem>,
new_parent: NodeUid,
) -> OutcomeStream<'a>;
async fn trash_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes>;
async fn restore_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes>;
fn restore_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a>;
fn delete_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a>;
async fn enumerate_trash_node_uids(&self) -> Result<Vec<NodeUid>>;
/// The account's storage, across every Proton product.
async fn quota(&self) -> Result<Quota>;
/// Called by the drain once it has let go of the blob of an upload to
/// `uid` that landed, or retired the create that made `uid`, before the
/// tree has the node, so a simulation can hold it there. Blocks the drain
/// worker it is called on.
#[cfg(test)]
fn landing(&self, _uid: &NodeUid) {}
}
/// The generic signatures of the client, over the object-safe ones above.
impl dyn DriveApi {
#[allow(clippy::too_many_arguments)]
pub(crate) async fn upload_file_from<R: Read + Send>(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
self.upload_file_from_dyn(
parent_uid,
name,
media_type,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
#[allow(clippy::too_many_arguments)]
pub(crate) async fn upload_file_replacing_draft_from<R: Read + Send>(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
self.upload_file_replacing_draft_from_dyn(
parent_uid,
name,
media_type,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
pub(crate) async fn upload_new_revision_from<R: Read + Send>(
&self,
file_uid: &NodeUid,
mut reader: R,
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()> {
self.upload_new_revision_from_dyn(
file_uid,
&mut reader,
intended_size,
thumbnails,
last_modification_time,
)
.await
}
pub(crate) async fn download_file_to<W: Write + Send>(
&self,
uid: &NodeUid,
output: &mut W,
) -> Result<()> {
self.download_file_to_dyn(uid, output).await
}
pub(crate) async fn download_revision_to<W: Write + Send>(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut W,
) -> Result<()> {
self.download_revision_to_dyn(file_uid, revision_id, writer)
.await
}
}
/// An open revision that block reads are served from; see [`RevisionReader`].
#[async_trait]
pub(crate) trait RevisionRead: Send + Sync {
/// Plaintext size of the revision.
fn size(&self) -> u64;
/// Plaintext size of each block, in block order.
fn block_sizes(&self) -> &[u64];
async fn read_at(&self, offset: u64, length: u64) -> Result<Vec<u8>>;
}
#[async_trait]
impl RevisionRead for RevisionReader {
fn size(&self) -> u64 {
RevisionReader::size(self)
}
fn block_sizes(&self) -> &[u64] {
RevisionReader::block_sizes(self)
}
async fn read_at(&self, offset: u64, length: u64) -> Result<Vec<u8>> {
RevisionReader::read_at(self, offset, length).await
}
}
#[async_trait]
impl DriveApi for ProtonDriveClient {
async fn get_my_files_folder(&self) -> Result<Node> {
ProtonDriveClient::get_my_files_folder(self).await
}
async fn get_node(&self, uid: &NodeUid) -> Result<Option<Node>> {
ProtonDriveClient::get_node(self, uid).await
}
async fn enumerate_nodes(&self, uids: &[NodeUid]) -> Result<Vec<Node>> {
ProtonDriveClient::enumerate_nodes(self, uids).await
}
async fn enumerate_nodes_light(&self, uids: &[NodeUid]) -> Result<Vec<Node>> {
ProtonDriveClient::enumerate_nodes_light(self, uids).await
}
async fn enumerate_folder_children_node_uids(
&self,
folder_uid: &NodeUid,
) -> Result<Vec<NodeUid>> {
ProtonDriveClient::enumerate_folder_children_node_uids(self, folder_uid).await
}
async fn enumerate_events(
&self,
scope: &DriveEventScopeId,
cursor: Option<&DriveEventId>,
) -> Result<Vec<DriveEvent>> {
ProtonDriveClient::enumerate_events(self, scope, cursor).await
}
async fn invalidate_caches_for_event(&self, event: &DriveEvent) -> Result<()> {
ProtonDriveClient::invalidate_caches_for_event(self, event).await
}
async fn create_folder(
&self,
parent_uid: &NodeUid,
name: &str,
last_modification_time: Option<i64>,
) -> Result<NodeUid> {
ProtonDriveClient::create_folder(self, parent_uid, name, last_modification_time).await
}
async fn upload_file(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
contents: &[u8],
) -> Result<NodeUid> {
ProtonDriveClient::upload_file(self, parent_uid, name, media_type, contents).await
}
async fn upload_file_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
ProtonDriveClient::upload_file_from(
self,
parent_uid,
name,
media_type,
reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
async fn upload_file_replacing_draft_from_dyn(
&self,
parent_uid: &NodeUid,
name: &str,
media_type: &str,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
aead: bool,
) -> Result<NodeUid> {
ProtonDriveClient::upload_file_replacing_draft_from(
self,
parent_uid,
name,
media_type,
reader,
intended_size,
thumbnails,
last_modification_time,
aead,
)
.await
}
async fn upload_new_revision_from_dyn(
&self,
file_uid: &NodeUid,
reader: &mut (dyn Read + Send),
intended_size: i64,
thumbnails: Vec<Thumbnail>,
last_modification_time: Option<i64>,
) -> Result<()> {
ProtonDriveClient::upload_new_revision_from(
self,
file_uid,
reader,
intended_size,
thumbnails,
last_modification_time,
)
.await
}
async fn open_revision(&self, uid: &NodeUid) -> Result<Arc<dyn RevisionRead>> {
let reader = ProtonDriveClient::open_revision(self, uid).await?;
Ok(Arc::new(reader))
}
async fn download_file_to_dyn(
&self,
uid: &NodeUid,
output: &mut (dyn Write + Send),
) -> Result<()> {
ProtonDriveClient::download_file_to(self, uid, &mut &mut *output).await
}
async fn download_revision_to_dyn(
&self,
file_uid: &NodeUid,
revision_id: &str,
writer: &mut (dyn Write + Send),
) -> Result<()> {
ProtonDriveClient::download_revision_to(self, file_uid, revision_id, &mut &mut *writer)
.await
}
async fn rename_node(
&self,
uid: &NodeUid,
new_name: &str,
new_media_type: Option<&str>,
) -> Result<()> {
ProtonDriveClient::rename_node(self, uid, new_name, new_media_type).await
}
async fn move_node(&self, uid: &NodeUid, new_parent: &NodeUid) -> Result<()> {
ProtonDriveClient::move_node(self, uid, new_parent).await
}
fn move_nodes_streaming<'a>(
&'a self,
items: Vec<NodeMoveItem>,
new_parent: NodeUid,
) -> OutcomeStream<'a> {
ProtonDriveClient::move_nodes_streaming(self, items, new_parent).boxed()
}
async fn trash_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes> {
ProtonDriveClient::trash_nodes(self, uids).await
}
async fn restore_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes> {
ProtonDriveClient::restore_nodes(self, uids).await
}
fn restore_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a> {
ProtonDriveClient::restore_nodes_streaming(self, uids).boxed()
}
fn delete_nodes_streaming<'a>(&'a self, uids: &[NodeUid]) -> OutcomeStream<'a> {
ProtonDriveClient::delete_nodes_streaming(self, uids).boxed()
}
async fn enumerate_trash_node_uids(&self) -> Result<Vec<NodeUid>> {
ProtonDriveClient::enumerate_trash_node_uids(self).await
}
async fn quota(&self) -> Result<Quota> {
ProtonDriveClient::quota(self).await
}
}
+15 -3
View File
@@ -9,7 +9,7 @@
//! through one place (the status, transfers, sync progress) and publishes them //! through one place (the status, transfers, sync progress) and publishes them
//! when they differ. //! when they differ.
use std::collections::BTreeSet; use std::collections::{BTreeSet, HashMap};
use std::sync::Arc; use std::sync::Arc;
use std::time::Duration; use std::time::Duration;
@@ -153,13 +153,15 @@ pub(crate) fn transfers_line(items: Vec<TransferItem>, jobs: Vec<JobItem>) -> Op
/// Watch what changes without a single place to publish it from, and publish it /// Watch what changes without a single place to publish it from, and publish it
/// when it differs from the last look: the status (queue counts, cache use, /// when it differs from the last look: the status (queue counts, cache use,
/// online, pause), the transfers snapshot, and the synced folders with their /// online, pause), the transfers snapshot, the synced folders with their
/// live progress. Looks only while someone is subscribed. /// live progress, and which nodes have a sync issue. Looks only while someone
/// is subscribed.
pub(crate) fn run_event_sampler(core: Core, username: String, mountpoint: std::path::PathBuf) { pub(crate) fn run_event_sampler(core: Core, username: String, mountpoint: std::path::PathBuf) {
let mut transfers: Option<Arc<str>> = None; let mut transfers: Option<Arc<str>> = None;
let mut status: Option<String> = None; let mut status: Option<String> = None;
let mut queue: Option<String> = None; let mut queue: Option<String> = None;
let mut locations: Option<String> = None; let mut locations: Option<String> = None;
let mut issues: Option<HashMap<i64, String>> = None;
let mut tick = 0u32; let mut tick = 0u32;
while core.shutdown.sleep(SAMPLE_INTERVAL) { while core.shutdown.sleep(SAMPLE_INTERVAL) {
if !core.events.has_subscribers() { if !core.events.has_subscribers() {
@@ -169,6 +171,7 @@ pub(crate) fn run_event_sampler(core: Core, username: String, mountpoint: std::p
status = None; status = None;
queue = None; queue = None;
locations = None; locations = None;
issues = None;
continue; continue;
} }
let line = transfers_line(core.transfers.snapshot(), core.jobs_snapshot()); let line = transfers_line(core.transfers.snapshot(), core.jobs_snapshot());
@@ -196,6 +199,15 @@ pub(crate) fn run_event_sampler(core: Core, username: String, mountpoint: std::p
} }
queue = Some(queue_key); queue = Some(queue_key);
// An issue shows on its node in the file browser: raised, cleared,
// landed or discarded, the listing is out of date.
if let Ok(now) = core.db.node_issues() {
if issues.as_ref().is_some_and(|last| *last != now) {
core.events.publish(&[Topic::Files]);
}
issues = Some(now);
}
if let Ok(now) = core.list_locations() { if let Ok(now) = core.list_locations() {
let key = serde_json::to_string(&now).unwrap_or_default(); let key = serde_json::to_string(&now).unwrap_or_default();
if locations.as_ref().is_some_and(|last| *last != key) { if locations.as_ref().is_some_and(|last| *last != key) {
File diff suppressed because it is too large. Load diff
+2406 -840
View File
File diff suppressed because it is too large. Load diff
+374
View File
@@ -0,0 +1,374 @@
//! Noticing that the network went away mid-session, and that it came back.
//!
//! The mount used to learn it was offline only at startup: a daemon that had
//! connected once kept `online` true for the rest of its life, so a Wi-Fi drop
//! turned every create, mkdir, delete and rename into a two-minute wait for an
//! `EIO`, while the queue that exists for exactly this sat unused. Now any
//! remote call a handler waits on reports a failure to reach Proton through
//! [`Core::lost_link`], which flips the mount offline; the handler then takes its
//! queued path, and the probe thread here flips it back when Proton answers.
use std::future::Future;
use std::sync::atomic::Ordering;
use std::time::{Duration, Instant};
use parking_lot::{Condvar, Mutex};
use proton_drive_rs::proton_sdk::api::ResponseCode;
use proton_drive_rs::proton_sdk::error::{ProtonApiError, ProtonError};
use tracing::{debug, info, warn};
use crate::{Core, ONLINE_PROBE_MAX, ONLINE_PROBE_MIN};
/// How long a remote call a FUSE handler is waiting on may take before the link
/// is treated as down.
///
/// The SDK's own budget for one request is four attempts of up to 30 seconds
/// each, so a dead link held the caller for about two minutes. A metadata call
/// that has had no answer in 20 seconds is not going to be one the user wants to
/// keep waiting for; queueing the change is the better answer.
pub(crate) const INTERACTIVE_CALL_TIMEOUT: Duration = Duration::from_secs(20);
/// How long one probe may wait for an answer. A probe is a single small request,
/// and a probe stuck in the SDK's retry budget would delay noticing the link is
/// back by as long as it waits.
const PROBE_TIMEOUT: Duration = Duration::from_secs(10);
/// First delay between probes after a drop seen mid-session, and the ceiling it
/// doubles to while the outage is young. A blip should end in seconds, not in
/// the minutes [`ONLINE_PROBE_MAX`] allows.
const PROBE_AFTER_DROP: Duration = Duration::from_secs(2);
const PROBE_YOUNG_MAX: Duration = Duration::from_secs(30);
/// How long an outage counts as young. Past it the probe backs off to
/// [`ONLINE_PROBE_MAX`], because a laptop can sit offline for days and each
/// probe is a real API round trip.
const YOUNG_OUTAGE: Duration = Duration::from_secs(10 * 60);
/// How often the idle probe thread checks for teardown while the link is up.
const IDLE_CHECK: Duration = Duration::from_secs(1);
/// The shared half of the online flag: when the link went down, and a condvar
/// that fires on every change so waiters need not poll.
#[derive(Default)]
pub(crate) struct Link {
lost_at: Mutex<Option<Instant>>,
changed: Condvar,
/// Folders listed from the DB alone while offline, to relist once back.
pub(crate) stale_listings: Mutex<std::collections::HashSet<u64>>,
}
/// Whether a failed call failed because Proton could not be reached, rather
/// than because it answered and refused.
///
/// A transport error is the network. So is a request timeout: the SDK reports
/// Proton's own "try again" codes that way, and [`no_answer`] reports our
/// deadline that way. Anything else Proton actually said, and says again on a
/// retry, so it must reach the caller rather than be queued.
pub(crate) fn is_network_error(e: &ProtonError) -> bool {
match e {
ProtonError::Transport(_) => true,
ProtonError::Api(api) => {
matches!(
api.code,
ResponseCode::RequestTimeout | ResponseCode::Timeout
)
}
_ => false,
}
}
/// The error a call reports when it runs past `after` without an answer.
pub(crate) fn no_answer(after: Duration) -> ProtonError {
ProtonError::Api(ProtonApiError {
code: ResponseCode::RequestTimeout,
http_status: 408,
message: format!("no answer within {}s", after.as_secs()),
details: None,
})
}
/// The slowest upload rate an upload's deadline allows for, past the
/// metadata budget it starts with. Far below any link that works, so only a
/// transfer that has stopped moving runs into it.
const SLOWEST_UPLOAD: u64 = 32 * 1024;
/// How long an upload of `len` bytes may take before the link is treated as
/// down: the budget of a metadata call, plus the bytes at [`SLOWEST_UPLOAD`].
pub(crate) fn upload_deadline(len: u64) -> Duration {
INTERACTIVE_CALL_TIMEOUT + Duration::from_secs(len / SLOWEST_UPLOAD)
}
/// Run a remote call, giving up after [`INTERACTIVE_CALL_TIMEOUT`].
pub(crate) async fn bounded<T>(
call: impl Future<Output = Result<T, ProtonError>>,
) -> Result<T, ProtonError> {
bounded_by(INTERACTIVE_CALL_TIMEOUT, call).await
}
/// Run a remote call, giving up after `limit`.
pub(crate) async fn bounded_by<T>(
limit: Duration,
call: impl Future<Output = Result<T, ProtonError>>,
) -> Result<T, ProtonError> {
match tokio::time::timeout(limit, call).await {
Ok(result) => result,
Err(_) => Err(no_answer(limit)),
}
}
/// The next delay between probes, given the current one and how long the
/// outage has lasted.
fn next_probe_delay(delay: Duration, outage: Duration) -> Duration {
let cap = if outage < YOUNG_OUTAGE {
PROBE_YOUNG_MAX
} else {
ONLINE_PROBE_MAX
};
(delay * 2).min(cap)
}
impl Core {
/// Whether the mount currently believes Proton is reachable.
pub(crate) fn is_online(&self) -> bool {
self.online.load(Ordering::Relaxed)
}
/// Whether a namespace change through the mount goes to Drive inside the
/// syscall. Only with [`AppConfig::local_first`] off, and only online;
/// otherwise it is queued and the drain sends it.
///
/// [`AppConfig::local_first`]: pdfs_core::config::AppConfig::local_first
pub(crate) fn sends_inline(&self) -> bool {
!self.local_first && self.is_online()
}
/// Run a remote call a caller is waiting on, giving up after
/// [`INTERACTIVE_CALL_TIMEOUT`]. The timer is built inside the runtime: a
/// FUSE worker has no reactor of its own.
pub(crate) fn block_on_bounded<T>(
&self,
call: impl Future<Output = Result<T, ProtonError>>,
) -> Result<T, ProtonError> {
self.rt.block_on(bounded(call))
}
/// Run a remote call, giving up after `limit`: an upload, whose budget
/// grows with its size ([`upload_deadline`]).
pub(crate) fn block_on_within<T>(
&self,
limit: Duration,
call: impl Future<Output = Result<T, ProtonError>>,
) -> Result<T, ProtonError> {
self.rt.block_on(bounded_by(limit, call))
}
/// Report a failed remote call. Returns whether it failed for want of a
/// network, in which case the mount is now offline and the caller should
/// take its queued path instead of failing.
pub(crate) fn lost_link(&self, e: &ProtonError, what: &str) -> bool {
if !is_network_error(e) {
return false;
}
self.mark_offline(what, e);
true
}
/// Flip the mount offline and wake the probe. Only the first caller of an
/// outage logs it; the rest are the same outage.
pub(crate) fn mark_offline(&self, what: &str, error: &dyn std::fmt::Display) {
let mut lost_at = self.link.lost_at.lock();
if self.online.swap(false, Ordering::Relaxed) {
*lost_at = Some(Instant::now());
warn!(during = what, %error,
"lost the connection to Proton; queueing changes until it returns");
self.link.changed.notify_all();
}
}
/// Flip the mount back online, wake everything waiting for it, and let the
/// queue drain.
fn mark_online(&self) {
{
let mut lost_at = self.link.lost_at.lock();
self.online.store(true, Ordering::Relaxed);
match lost_at.take() {
Some(since) => {
info!(
offline_secs = since.elapsed().as_secs(),
"connection to Proton is back"
)
}
None => info!("back online"),
}
self.link.changed.notify_all();
}
self.relist_offline_listings();
// Anything written while offline is queued and waiting on exactly this,
// including ops that failed on the way down and are sitting out a
// backoff sized for a server that keeps refusing.
if let Err(e) = self.db.retry_failed_ops_now(crate::now_millis()) {
warn!(error = %e, "moving backed-off ops up after a reconnect failed");
}
self.wake_drain();
}
/// Wait up to `budget` for the link to come back. Returns whether it is up.
pub(crate) fn wait_online(&self, budget: Duration) -> bool {
let deadline = Instant::now() + budget;
let mut lost_at = self.link.lost_at.lock();
while !self.is_online() {
if self.shutdown.is_stopping()
|| self
.link
.changed
.wait_until(&mut lost_at, deadline)
.timed_out()
{
return self.is_online();
}
}
true
}
/// Wait out `interval` before a background poll of Proton, and while the
/// link is down, until it is back. Returns as soon as it comes back, so the
/// poll catches up on what changed during the outage at once instead of
/// one interval later. A poll into a dead link would only wait out the
/// SDK's retries and fail.
pub(crate) async fn next_poll(&self, interval: Duration) {
let due = Instant::now() + interval;
let mut was_offline = false;
loop {
if !self.is_online() {
was_offline = true;
} else if was_offline || Instant::now() >= due {
return;
}
tokio::time::sleep(IDLE_CHECK).await;
}
}
/// Watch the link for the life of the mount: sleep while it is up, and
/// probe it back while it is down.
///
/// A mount that started from the cache is down from the first moment; one
/// that started online goes down when [`Core::mark_offline`] says so.
pub(crate) fn run_online_probe(&self) {
// Only an offline start never had a chance to repair its share id.
let mut repair_share_id = !self.is_online();
loop {
{
let mut lost_at = self.link.lost_at.lock();
while self.is_online() {
if self.shutdown.is_stopping() {
return;
}
self.link.changed.wait_for(&mut lost_at, IDLE_CHECK);
}
}
if !self.probe_until_online() {
return;
}
if std::mem::take(&mut repair_share_id) {
self.repair_primary_share_id(&self.primary_root_uid);
}
}
}
/// Probe until Proton answers, then flip the mount online. Returns `false`
/// if the daemon is stopping.
fn probe_until_online(&self) -> bool {
let started = self.link.lost_at.lock().unwrap_or_else(Instant::now);
let mut delay = if started.elapsed() < YOUNG_OUTAGE {
PROBE_AFTER_DROP
} else {
ONLINE_PROBE_MIN
};
loop {
if !self.shutdown.sleep(delay) {
return false;
}
let probe = self.rt.block_on(async {
match tokio::time::timeout(PROBE_TIMEOUT, self.drive.get_my_files_folder()).await {
Ok(result) => result,
Err(_) => Err(no_answer(PROBE_TIMEOUT)),
}
});
match probe {
Ok(root) => {
{
let root_ino = self.root_ino();
let mut st = self.state();
if let Some(e) = st.entries.get_mut(&root_ino) {
e.node = root.clone();
}
}
if let Err(e) = self.db.upsert_node(&root) {
warn!(error = %e, "refresh root after reconnect failed");
}
self.mark_online();
return true;
}
Err(e) => {
debug!(error = %e, ?delay, "online probe failed; still offline");
delay = next_probe_delay(delay, started.elapsed());
}
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn api(code: ResponseCode) -> ProtonError {
ProtonError::Api(ProtonApiError {
code,
http_status: 422,
message: String::new(),
details: None,
})
}
#[test]
fn only_an_unanswered_call_counts_as_a_lost_link() {
assert!(is_network_error(&no_answer(INTERACTIVE_CALL_TIMEOUT)));
assert!(is_network_error(&api(ResponseCode::Timeout)));
assert!(!is_network_error(&api(ResponseCode::AlreadyExists)));
assert!(!is_network_error(&api(ResponseCode::InvalidRequirements)));
assert!(!is_network_error(&ProtonError::invalid_operation("bug")));
}
#[test]
fn an_upload_gets_longer_the_bigger_it_is() {
assert_eq!(upload_deadline(0), INTERACTIVE_CALL_TIMEOUT);
assert_eq!(
upload_deadline(1 << 30),
INTERACTIVE_CALL_TIMEOUT + Duration::from_secs(32_768)
);
}
#[test]
fn a_call_past_its_deadline_reads_as_a_lost_link() {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_time()
.build()
.unwrap();
let limit = Duration::from_millis(20);
let result: Result<(), _> = rt.block_on(bounded_by(limit, std::future::pending()));
assert!(result.is_err_and(|e| is_network_error(&e)));
}
#[test]
fn a_young_outage_is_probed_often_and_an_old_one_rarely() {
let young = Duration::from_secs(60);
let old = YOUNG_OUTAGE + Duration::from_secs(1);
assert_eq!(
next_probe_delay(PROBE_AFTER_DROP, young),
2 * PROBE_AFTER_DROP
);
assert_eq!(next_probe_delay(PROBE_YOUNG_MAX, young), PROBE_YOUNG_MAX);
assert_eq!(next_probe_delay(PROBE_YOUNG_MAX, old), 2 * PROBE_YOUNG_MAX);
assert_eq!(next_probe_delay(ONLINE_PROBE_MAX, old), ONLINE_PROBE_MAX);
}
}
+171 -56
View File
@@ -193,7 +193,7 @@ pub(super) fn clear_stale_mount(mountpoint: &Path) {
} }
/// `sync_state` key holding the uid of the My Files root, so a later run can /// `sync_state` key holding the uid of the My Files root, so a later run can
/// recover it from `nodes` without the API (offline.md Phase 1). /// recover it from `nodes` without the API.
const ROOT_UID_KEY: &str = "root_uid"; const ROOT_UID_KEY: &str = "root_uid";
/// The My Files root, and whether we got it from the API (`true`) or from the /// The My Files root, and whether we got it from the API (`true`) or from the
@@ -207,11 +207,11 @@ const ROOT_UID_KEY: &str = "root_uid";
/// means an empty tree, and mounting that would show the user an empty Drive /// means an empty tree, and mounting that would show the user an empty Drive
/// rather than an honest error. /// rather than an honest error.
fn fetch_or_recall_root( fn fetch_or_recall_root(
client: &ProtonDriveClient, drive: &dyn DriveApi,
rt: &tokio::runtime::Handle, rt: &tokio::runtime::Handle,
db: &Db, db: &Db,
) -> std::io::Result<(Node, bool)> { ) -> std::io::Result<(Node, bool)> {
let err = match rt.block_on(client.get_my_files_folder()) { let err = match rt.block_on(drive.get_my_files_folder()) {
Ok(root) => { Ok(root) => {
if let Err(e) = db.set_state_str(ROOT_UID_KEY, &root.uid.to_string()) { if let Err(e) = db.set_state_str(ROOT_UID_KEY, &root.uid.to_string()) {
warn!(error = %e, "persist root uid failed"); warn!(error = %e, "persist root uid failed");
@@ -244,6 +244,9 @@ pub struct MountOptions {
/// Upload and download caps in bytes per second (`0` = no cap). /// Upload and download caps in bytes per second (`0` = no cap).
pub upload_limit: u64, pub upload_limit: u64,
pub download_limit: u64, pub download_limit: u64,
/// Namespace changes through the mount are queued, not sent in the
/// syscall. See [`pdfs_core::config::AppConfig::local_first`].
pub local_first: bool,
} }
/// Spawn one FUSE session rooted at an arbitrary remote node. /// Spawn one FUSE session rooted at an arbitrary remote node.
@@ -276,13 +279,14 @@ pub(super) fn spawn_session(
info!(mountpoint = %mountpoint.display(), "mounting Proton Drive location"); info!(mountpoint = %mountpoint.display(), "mounting Proton Drive location");
let fs = ProtonFs::new(core.clone(), root); let fs = ProtonFs::new(core.clone(), root);
if core.primary { if core.primary {
// Only My Files rehydrates the daemon-wide node cache. A secondary // Only My Files rehydrates the daemon-wide node cache, once. A
// inode space is rooted at one device subtree and must not import every // secondary mount shares the tree it fills.
// unrelated persisted node.
core.hydrate(); core.hydrate();
} }
let session = Session::new(fs, mountpoint, &config)?.spawn()?; let session = Session::new(fs, mountpoint, &config)?.spawn()?;
let _ = core.notifier.set(session.notifier()); let _ = core
.notifier
.set(KernelChannel::new(session.notifier(), core.root_ino()));
core.session_live.store(true, Ordering::Release); core.session_live.store(true, Ordering::Release);
Ok(session) Ok(session)
} }
@@ -339,17 +343,57 @@ pub fn mount(
control_socket: &Path, control_socket: &Path,
db: Arc<Db>, db: Arc<Db>,
options: MountOptions, options: MountOptions,
) -> std::io::Result<MountOutcome> {
let drive = Arc::new(client.clone());
mount_with(
client,
drive,
rt,
mountpoint,
cache,
control_socket,
db,
options,
Host::Daemon,
)
}
/// What a mount runs in.
pub(crate) enum Host {
/// The daemon: stopped by SIGTERM or SIGINT, and indexing the user's home
/// for the launcher prompt.
Daemon,
/// One of several simulated daemons in a test process (`sim`): stopped by
/// a message on this channel, and with no home to index.
#[cfg(test)]
Simulation(std::sync::mpsc::Receiver<()>),
}
/// [`mount`], reaching Drive through `drive` and hosted by `host`; `client`
/// serves only what [`DriveApi`] does not cover.
#[allow(clippy::too_many_arguments)]
pub(crate) fn mount_with(
client: ProtonDriveClient,
drive: Arc<dyn DriveApi>,
rt: tokio::runtime::Handle,
mountpoint: &Path,
cache: ContentCache,
control_socket: &Path,
db: Arc<Db>,
options: MountOptions,
host: Host,
) -> std::io::Result<MountOutcome> { ) -> std::io::Result<MountOutcome> {
let MountOptions { let MountOptions {
username, username,
sweep_mode, sweep_mode,
upload_limit, upload_limit,
download_limit, download_limit,
local_first,
} = options; } = options;
// Start the uptime clock here rather than at the first `pdfs diagnostics`, // Start the uptime clock here rather than at the first `pdfs diagnostics`,
// so the age it reports is the daemon's own. // so the age it reports is the daemon's own.
super::diagnostics::start_clock(); super::diagnostics::start_clock();
let (root, online) = fetch_or_recall_root(&client, &rt, &db)?; let (root, online) = fetch_or_recall_root(drive.as_ref(), &rt, &db)?;
let scope = root.tree_event_scope_id(); let scope = root.tree_event_scope_id();
db.mount_upsert_my_files(&mountpoint.to_string_lossy(), &root.uid.to_string(), None) db.mount_upsert_my_files(&mountpoint.to_string_lossy(), &root.uid.to_string(), None)
.map_err(|error| std::io::Error::other(format!("project My Files mount: {error}")))?; .map_err(|error| std::io::Error::other(format!("project My Files mount: {error}")))?;
@@ -357,19 +401,21 @@ pub fn mount(
.all_share_access() .all_share_access()
.map_err(|error| std::io::Error::other(format!("load shared access: {error}")))?; .map_err(|error| std::io::Error::other(format!("load shared access: {error}")))?;
// The folder-sync engine (devices.md Phase 2) runs on its own thread and is // The folder-sync engine runs on its own thread and is
// nudged over this channel; the sender lives in Core so control-socket // nudged over this channel; the sender lives in Core so control-socket
// handlers can trigger reconciles. // handlers can trigger reconciles.
let (sync_tx, sync_rx) = std::sync::mpsc::channel::<sync::SyncMsg>(); let (sync_tx, sync_rx) = std::sync::mpsc::channel::<sync::SyncMsg>();
let paused_until = pause::load_paused_until(&db); let paused_until = pause::load_paused_until(&db);
let core = Core { let core = Core {
client: client.clone(), drive,
client,
rt: rt.clone(), rt: rt.clone(),
maintenance: Arc::new(Mutex::new(Default::default())), maintenance: Arc::new(Mutex::new(Default::default())),
primary_root_uid: root.uid.clone(), primary_root_uid: root.uid.clone(),
primary: true, primary: true,
state: Arc::new(Mutex::new(State::new(db.clone(), share_access, 2))), state: Arc::new(Mutex::new(State::new(db.clone(), share_access))),
root: Arc::new(OnceLock::new()),
cache: Arc::new(cache), cache: Arc::new(cache),
readers: Arc::new(Mutex::new(HashMap::new())), readers: Arc::new(Mutex::new(HashMap::new())),
block_ring: Arc::new(Mutex::new(BlockRing::default())), block_ring: Arc::new(Mutex::new(BlockRing::default())),
@@ -383,13 +429,19 @@ pub fn mount(
shared_generation: Arc::new(AtomicU64::new(0)), shared_generation: Arc::new(AtomicU64::new(0)),
shared_refresh_deadlines: Arc::new(Mutex::new(SharedRefreshDeadlines::default())), shared_refresh_deadlines: Arc::new(Mutex::new(SharedRefreshDeadlines::default())),
online: Arc::new(AtomicBool::new(online)), online: Arc::new(AtomicBool::new(online)),
link: Arc::new(Default::default()),
sync_paused_until: Arc::new(AtomicI64::new(paused_until)), sync_paused_until: Arc::new(AtomicI64::new(paused_until)),
pending: Arc::new(Mutex::new(HashMap::new())), pending: Arc::new(Mutex::new(HashMap::new())),
hidden: Arc::new(Mutex::new(HashSet::new())), hidden: Arc::new(Mutex::new(HashSet::new())),
creating: Arc::new(Mutex::new(HashMap::new())),
sealed_unread: Arc::new(Mutex::new(HashMap::new())),
landed: Arc::new(Mutex::new(HashMap::new())),
unadopted: Arc::new(Mutex::new(HashMap::new())),
drain_wake: Arc::new((Mutex::new(false), Condvar::new())), drain_wake: Arc::new((Mutex::new(false), Condvar::new())),
shutdown: Arc::new(crate::shutdown::Shutdown::default()), shutdown: Arc::new(crate::shutdown::Shutdown::default()),
upload_times: Arc::new(Mutex::new(HashMap::new())), upload_times: Arc::new(Mutex::new(HashMap::new())),
upload_cancel: Arc::new(Mutex::new(HashMap::new())), upload_cancel: Arc::new(Mutex::new(HashMap::new())),
landing_uploads: Arc::new(AtomicU64::new(0)),
timeline_refreshing: Arc::new(AtomicBool::new(false)), timeline_refreshing: Arc::new(AtomicBool::new(false)),
timeline_progress: Arc::new(Mutex::new(None)), timeline_progress: Arc::new(Mutex::new(None)),
similar_hashing: Arc::new(AtomicBool::new(false)), similar_hashing: Arc::new(AtomicBool::new(false)),
@@ -398,7 +450,7 @@ pub fn mount(
trash_progress: Arc::new(tokio::sync::Notify::new()), trash_progress: Arc::new(tokio::sync::Notify::new()),
conflict_notified: Arc::new(Mutex::new(HashSet::new())), conflict_notified: Arc::new(Mutex::new(HashSet::new())),
sweep_mode, sweep_mode,
own_sealed_revs: Arc::new(Mutex::new(HashMap::new())), local_first,
self_changes: Arc::new(Mutex::new(HashMap::new())), self_changes: Arc::new(Mutex::new(HashMap::new())),
thumb_gen: Arc::new(Mutex::new(HashSet::new())), thumb_gen: Arc::new(Mutex::new(HashSet::new())),
thumb_gen_budget: Arc::new(tokio::sync::Semaphore::new( thumb_gen_budget: Arc::new(tokio::sync::Semaphore::new(
@@ -411,6 +463,7 @@ pub fn mount(
thumbnail_build_cancel: Arc::new(tokio::sync::Notify::new()), thumbnail_build_cancel: Arc::new(tokio::sync::Notify::new()),
thumbnail_misses: Arc::new(Mutex::new(Default::default())), thumbnail_misses: Arc::new(Mutex::new(Default::default())),
quota: Arc::new(Mutex::new(None)), quota: Arc::new(Mutex::new(None)),
quota_refreshing: Arc::new(AtomicBool::new(false)),
size_upgrades: Arc::new(Mutex::new(HashMap::new())), size_upgrades: Arc::new(Mutex::new(HashMap::new())),
size_waiters: Arc::new(Default::default()), size_waiters: Arc::new(Default::default()),
notifier: Arc::new(OnceLock::new()), notifier: Arc::new(OnceLock::new()),
@@ -425,7 +478,7 @@ pub fn mount(
states: Arc::new(StateRegistry::default()), states: Arc::new(StateRegistry::default()),
}; };
// Before anything can queue work against it: the drain thread below reaches // Before anything can queue work against it: the drain thread below reaches
// every mount's inode space through this registry, not through `core.state`. // every mount's session through this registry.
core.register_state(mountpoint); core.register_state(mountpoint);
core.transfers.set_limits(upload_limit, download_limit); core.transfers.set_limits(upload_limit, download_limit);
@@ -448,7 +501,7 @@ pub fn mount(
// Every long-lived thread started from here on is kept in `workers` and // Every long-lived thread started from here on is kept in `workers` and
// joined at the end of this function. Leaving them running turned an // joined at the end of this function. Leaving them running turned an
// in-process remount into a second generation of drain workers, sync engines // in-process remount into a second generation of drain workers, sync engines
// and sweeps all operating on a mount that no longer exists (bugs.md B44). // and sweeps all operating on a mount that no longer exists (`docs/BUGS.md` B44).
let mut workers: Vec<std::thread::JoinHandle<()>> = Vec::new(); let mut workers: Vec<std::thread::JoinHandle<()>> = Vec::new();
for worker in 0..DRAIN_WORKERS { for worker in 0..DRAIN_WORKERS {
let core = core.clone(); let core = core.clone();
@@ -484,9 +537,10 @@ pub fn mount(
); );
} }
// Mounted from the cache: watch for the network coming back so the mount can // Watch the link for the life of the mount: a mount from the cache waits for
// stop being read-only-ish without the user restarting the daemon. // the network to come back, and an online one can lose it at any moment and
if !online { // must notice it coming back just the same.
{
let core = core.clone(); let core = core.clone();
workers.push( workers.push(
std::thread::Builder::new() std::thread::Builder::new()
@@ -497,7 +551,7 @@ pub fn mount(
// Keep the launcher prompt's "This computer" index fresh. Its own thread: // Keep the launcher prompt's "This computer" index fresh. Its own thread:
// the walk is I/O-heavy and must never sit in front of a FUSE callback. // the walk is I/O-heavy and must never sit in front of a FUSE callback.
{ if matches!(host, Host::Daemon) {
let db = core.db.clone(); let db = core.db.clone();
let indexing = core.indexing.clone(); let indexing = core.indexing.clone();
let transfers = core.transfers.clone(); let transfers = core.transfers.clone();
@@ -621,11 +675,13 @@ pub fn mount(
} }
} }
}; };
rt.spawn(run_event_sync(client, scope, core.clone())); rt.spawn(run_event_sync(core.drive.clone(), scope, core.clone()));
// The photos volume has its own event stream; without it the gallery only // The photos volume has its own event stream; without it the gallery only
// learns about a phone-side deletion when the timeline goes stale. // learns about a phone-side deletion when the timeline goes stale.
rt.spawn(run_photos_event_sync(core.clone())); rt.spawn(run_photos_event_sync(core.clone()));
if online { // Not in a simulation: `client` there reaches no API, and the thread, still
// waiting on it after the stop, held the database a test opens next.
if online && matches!(host, Host::Daemon) {
let repair_core = core.clone(); let repair_core = core.clone();
let root_uid = core.primary_root_uid.clone(); let root_uid = core.primary_root_uid.clone();
if let Err(error) = std::thread::Builder::new() if let Err(error) = std::thread::Builder::new()
@@ -640,30 +696,11 @@ pub fn mount(
// are delivered onto the async runtime; bridge them onto a sync channel so // are delivered onto the async runtime; bridge them onto a sync channel so
// the loop below can react without blocking a worker thread. A bounded // the loop below can react without blocking a worker thread. A bounded
// channel of 1 is enough — we only need to know that *a* stop arrived. // channel of 1 is enough — we only need to know that *a* stop arrived.
let (sig_tx, sig_rx) = std::sync::mpsc::sync_channel::<()>(1); let sig_rx = match host {
rt.spawn(async move { Host::Daemon => signal_stop(&rt),
let mut sigterm = #[cfg(test)]
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) { Host::Simulation(rx) => rx,
Ok(s) => s, };
Err(e) => {
warn!(error = %e, "install SIGTERM handler failed");
return;
}
};
let mut sigint =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGINT handler failed");
return;
}
};
tokio::select! {
_ = sigterm.recv() => info!("received SIGTERM"),
_ = sigint.recv() => info!("received SIGINT"),
}
let _ = sig_tx.try_send(());
});
// Wait for whichever happens first: a stop signal (→ we unmount ourselves // Wait for whichever happens first: a stop signal (→ we unmount ourselves
// via the lazy MNT_DETACH path, clean even mid-download), or the kernel // via the lazy MNT_DETACH path, clean even mid-download), or the kernel
@@ -701,7 +738,7 @@ pub fn mount(
}; };
// Unmount every on-demand sync folder too, or the kernel mounts linger as // Unmount every on-demand sync folder too, or the kernel mounts linger as
// stale and the next start fails with EBUSY (devices.md Phase 3). // stale and the next start fails with EBUSY.
if let Some(worker) = restore_worker if let Some(worker) = restore_worker
&& worker.join().is_err() && worker.join().is_err()
{ {
@@ -713,6 +750,7 @@ pub fn mount(
warn!(id, error = %e, "unmount on-demand folder failed"); warn!(id, error = %e, "unmount on-demand folder failed");
} }
} }
core.queue_unreleased_writes();
// Last, because the workers may legitimately be finishing an upload, and // Last, because the workers may legitimately be finishing an upload, and
// nothing they can still do is harmful once the mounts are down. // nothing they can still do is harmful once the mounts are down.
@@ -720,9 +758,39 @@ pub fn mount(
Ok(outcome) Ok(outcome)
} }
/// How long teardown waits for the FUSE worker pool before proceeding without /// A channel that receives once SIGTERM or SIGINT arrives.
/// it. Long enough for an in-flight block to finish, short enough that fn signal_stop(rt: &tokio::runtime::Handle) -> std::sync::mpsc::Receiver<()> {
/// `systemctl --user stop` still returns inside `TimeoutStopSec`. let (sig_tx, sig_rx) = std::sync::mpsc::sync_channel::<()>(1);
rt.spawn(async move {
let mut sigterm =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGTERM handler failed");
return;
}
};
let mut sigint =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGINT handler failed");
return;
}
};
tokio::select! {
_ = sigterm.recv() => info!("received SIGTERM"),
_ = sigint.recv() => info!("received SIGINT"),
}
let _ = sig_tx.try_send(());
});
sig_rx
}
/// How long teardown waits for the background workers and then the FUSE worker
/// pool before proceeding without them. Long enough for an in-flight block to
/// finish, short enough that `systemctl --user stop` still returns inside
/// `TimeoutStopSec`.
const WORKER_JOIN_DEADLINE: Duration = Duration::from_secs(10); const WORKER_JOIN_DEADLINE: Duration = Duration::from_secs(10);
/// Signal every background loop, wake the ones that are blocked, and join them. /// Signal every background loop, wake the ones that are blocked, and join them.
@@ -730,14 +798,15 @@ const WORKER_JOIN_DEADLINE: Duration = Duration::from_secs(10);
/// The mount used to return with its drain workers, sync engine, sweep, online /// The mount used to return with its drain workers, sync engine, sweep, online
/// probe, indexer and control listener all still running on `Core` clones. The /// probe, indexer and control listener all still running on `Core` clones. The
/// process usually exited straight afterwards and hid it; an in-process remount /// process usually exited straight afterwards and hid it; an in-process remount
/// did not, and started a second full set on top of the first (bugs.md B44). /// did not, and started a second full set on top of the first (`docs/BUGS.md` B44).
/// ///
/// Two of the waits need waking rather than merely flagging: the drain workers /// Two of the waits need waking rather than merely flagging: the drain workers
/// block on their own condvar (`wake_drain`), and the control listener blocks /// block on their own condvar (`wake_drain`), and the control listener blocks
/// inside `accept`, which only returns when a connection arrives — so it gets /// inside `accept`, which only returns when a connection arrives — so it gets
/// one, from us, after the flag is already set. A join that takes a while is /// one, from us, after the flag is already set. A join that takes a while is
/// expected and correct: a worker part-way through an upload finishes it rather /// expected and correct: a worker part-way through an upload finishes it rather
/// than abandoning the user's bytes mid-flight. /// than abandoning the user's bytes mid-flight. Up to [`WORKER_JOIN_DEADLINE`]:
/// past it, systemd kills the daemon anyway, and a queued op is sent again.
fn stop_workers(core: &Core, control_socket: &Path, workers: Vec<std::thread::JoinHandle<()>>) { fn stop_workers(core: &Core, control_socket: &Path, workers: Vec<std::thread::JoinHandle<()>>) {
// Say so before the joins: a teardown that takes a while must not be read as // Say so before the joins: a teardown that takes a while must not be read as
// a missed watchdog ping and turned into a kill. // a missed watchdog ping and turned into a kill.
@@ -751,25 +820,71 @@ fn stop_workers(core: &Core, control_socket: &Path, workers: Vec<std::thread::Jo
// unlink cannot put it back to sleep. // unlink cannot put it back to sleep.
let _ = std::os::unix::net::UnixStream::connect(control_socket); let _ = std::os::unix::net::UnixStream::connect(control_socket);
let _ = std::fs::remove_file(control_socket); let _ = std::fs::remove_file(control_socket);
for worker in workers { let deadline = Instant::now() + WORKER_JOIN_DEADLINE;
if worker.join().is_err() { let running = join_until(workers, deadline);
warn!("a background worker panicked before shutdown"); if !running.is_empty() {
} warn!(
running = running.join(","),
"background workers did not finish in time; shutting down around them"
);
} }
// The FUSE worker pool last, and *before* the caller drops the tokio // The FUSE worker pool last, and *before* the caller drops the tokio
// runtime: a worker still inside a job when the runtime goes away panics on // runtime: a worker still inside a job when the runtime goes away panics on
// its next timer with "A Tokio 1.x context was found, but it is being // its next timer with "A Tokio 1.x context was found, but it is being
// shutdown" (seen 2026-09-18 09:58:35). Bounded, so a worker wedged on the // shutdown" (seen 2026-09-18 09:58:35). Bounded, so a worker wedged on the
// network cannot hold the stop the user asked for. // network cannot hold the stop the user asked for.
core.workers.stop_and_join(WORKER_JOIN_DEADLINE); core.workers
.stop_and_join(deadline.saturating_duration_since(Instant::now()));
debug!("background workers stopped"); debug!("background workers stopped");
} }
/// Join `workers` until `deadline`, and name the ones still running then.
///
/// A stop used to wait for each one however long it took. The sync engine's
/// mirror pass waits on Drive with no deadline, and a stop that waited for it
/// was killed by systemd (`docs/BUGS.md` B165).
fn join_until(workers: Vec<std::thread::JoinHandle<()>>, deadline: Instant) -> Vec<String> {
let mut running = Vec::new();
for worker in workers {
while !worker.is_finished() && Instant::now() < deadline {
std::thread::sleep(Duration::from_millis(10));
}
if !worker.is_finished() {
running.push(worker.thread().name().unwrap_or("unnamed").to_string());
} else if worker.join().is_err() {
warn!("a background worker panicked before shutdown");
}
}
running
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
use super::{SecondaryInsertRejection, SecondaryMountRegistry, teardown_session}; use super::{SecondaryInsertRejection, SecondaryMountRegistry, join_until, teardown_session};
#[test]
fn a_stop_names_the_workers_still_running_at_its_deadline() {
let (release, wait) = std::sync::mpsc::channel::<()>();
let done = std::thread::Builder::new()
.name("done".into())
.spawn(|| {})
.unwrap();
let stuck = std::thread::Builder::new()
.name("stuck".into())
.spawn(move || {
let _ = wait.recv();
})
.unwrap();
let running = join_until(
vec![done, stuck],
Instant::now() + Duration::from_millis(100),
);
assert_eq!(running, ["stuck"]);
drop(release);
}
#[test] #[test]
fn session_is_unroutable_before_teardown_starts() { fn session_is_unroutable_before_teardown_starts() {
+1 -1
View File
@@ -1038,7 +1038,7 @@ impl Core {
} }
let outcomes = self let outcomes = self
.rt .rt
.block_on(self.client.trash_nodes(uids)) .block_on(self.drive.trash_nodes(uids))
.map_err(|e| CoreError::from_api(&e, "trash"))?; .map_err(|e| CoreError::from_api(&e, "trash"))?;
let (trashed, failed) = pdfs_core::batch::split(outcomes); let (trashed, failed) = pdfs_core::batch::split(outcomes);
+10 -10
View File
@@ -1,5 +1,5 @@
//! Reading and writing this machine's `profile.json` in its device root, and //! Reading and writing this machine's `profile.json` in its device root, and
//! the restore that rebuilds a machine from one (features.md 5.2 and 5.3). //! the restore that rebuilds a machine from one.
//! //!
//! The document itself is defined in [`pdfs_core::profile`]; this is the half //! The document itself is defined in [`pdfs_core::profile`]; this is the half
//! that talks to Drive. Saving is best-effort and never blocks a user action: //! that talks to Drive. Saving is best-effort and never blocks a user action:
@@ -101,7 +101,7 @@ impl Core {
match existing { match existing {
Some(uid) => self Some(uid) => self
.rt .rt
.block_on(self.client.upload_new_revision_from( .block_on(self.drive.upload_new_revision_from(
&uid, &uid,
Cursor::new(bytes), Cursor::new(bytes),
len, len,
@@ -112,7 +112,7 @@ impl Core {
.map_err(|e| CoreError::from_api(&e, "upload profile revision"))?, .map_err(|e| CoreError::from_api(&e, "upload profile revision"))?,
None => { None => {
self.rt self.rt
.block_on(self.client.upload_file_replacing_draft_from( .block_on(self.drive.upload_file_replacing_draft_from(
&dir_uid, &dir_uid,
PROFILE_FILE_NAME, PROFILE_FILE_NAME,
"application/json", "application/json",
@@ -182,14 +182,14 @@ impl Core {
) -> CoreResult<Option<NodeUid>> { ) -> CoreResult<Option<NodeUid>> {
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(parent_uid)) .block_on(self.drive.enumerate_folder_children_node_uids(parent_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?; .map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() { if uids.is_empty() {
return Ok(None); return Ok(None);
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes(&uids)) .block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?; .map_err(|e| CoreError::from_api(&e, "resolve device root children"))?;
Ok(nodes Ok(nodes
.into_iter() .into_iter()
@@ -210,7 +210,7 @@ impl Core {
let uid = self let uid = self
.rt .rt
.block_on( .block_on(
self.client self.drive
.create_folder(root_uid, PROFILE_DIR_NAME, Some(now_secs())), .create_folder(root_uid, PROFILE_DIR_NAME, Some(now_secs())),
) )
.map_err(|e| CoreError::from_api(&e, "create profile folder"))?; .map_err(|e| CoreError::from_api(&e, "create profile folder"))?;
@@ -241,12 +241,12 @@ impl Core {
}; };
let mut buf: Vec<u8> = Vec::new(); let mut buf: Vec<u8> = Vec::new();
self.rt self.rt
.block_on(self.client.download_file_to(&uid, &mut buf)) .block_on(self.drive.download_file_to(&uid, &mut buf))
.map_err(|e| CoreError::from_api(&e, "download profile"))?; .map_err(|e| CoreError::from_api(&e, "download profile"))?;
Profile::parse(&buf).map(Some).map_err(CoreError::invalid) Profile::parse(&buf).map(Some).map_err(CoreError::invalid)
} }
// ---- restore (features.md 5.2) ---------------------------------------- // ---- restore ----------------------------------------
/// The folders under a device that could be synced here, each carrying a /// The folders under a device that could be synced here, each carrying a
/// proposed local path. `device` names another registered device by uid; /// proposed local path. `device` names another registered device by uid;
@@ -277,13 +277,13 @@ impl Core {
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(&root_uid)) .block_on(self.drive.enumerate_folder_children_node_uids(&root_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?; .map_err(|e| CoreError::from_api(&e, "list device root"))?;
let nodes = if uids.is_empty() { let nodes = if uids.is_empty() {
Vec::new() Vec::new()
} else { } else {
self.rt self.rt
.block_on(self.client.enumerate_nodes(&uids)) .block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))? .map_err(|e| CoreError::from_api(&e, "resolve device root children"))?
}; };
+247 -5
View File
@@ -1,12 +1,16 @@
//! The user's view of the pending-op queue: what has not reached the remote //! The user's view of the pending-op queue: what has not reached the remote
//! yet, why, and a way to stop waiting out a backoff. //! yet, why, and a way to stop waiting out a backoff.
//! //!
//! Read-only apart from "retry now", which only moves a row's next-attempt time. //! Read-only apart from "retry now", which only moves a row's next-attempt time,
//! Nothing here drops a queued op: its staged blob is the only copy of the //! "export", which copies a staged blob out, and "discard". Only a discard the
//! user's write (see `docs/RECOVERY.md`). //! user asks for drops a queued op: its staged blob is the only copy of the
//! user's write (see `docs/RECOVERY.md`), so it is the user's call, made after
//! an export or for a change with no content.
use super::*; use super::*;
use pdfs_core::control::PendingOpInfo; use std::io::Write;
use pdfs_core::control::{PendingOpInfo, SyncIssue};
use pdfs_core::db::{FAILING_ATTEMPTS, PARK_UNTIL}; use pdfs_core::db::{FAILING_ATTEMPTS, PARK_UNTIL};
impl Core { impl Core {
@@ -16,10 +20,15 @@ impl Core {
.db .db
.pending_ops() .pending_ops()
.map_err(|e| CoreError::internal(format!("listing the queue: {e}")))?; .map_err(|e| CoreError::internal(format!("listing the queue: {e}")))?;
let issues = self
.db
.op_issues()
.map_err(|e| CoreError::internal(format!("listing sync issues: {e}")))?;
Ok(ops Ok(ops
.into_iter() .into_iter()
.map(|op| { .map(|op| {
let parked = op.next_attempt_at >= PARK_UNTIL; let parked = op.next_attempt_at >= PARK_UNTIL;
let exportable = exportable(&op).is_some();
let path = self let path = self
.db .db
.node_path(&op.uid) .node_path(&op.uid)
@@ -37,7 +46,9 @@ impl Core {
queued_at: op.created_at / 1000, queued_at: op.created_at / 1000,
next_attempt_at: (!parked).then_some(op.next_attempt_at / 1000), next_attempt_at: (!parked).then_some(op.next_attempt_at / 1000),
parked, parked,
failing: op.attempts >= FAILING_ATTEMPTS, failing: op.attempts >= FAILING_ATTEMPTS || issues.contains_key(&op.id),
issue: issues.get(&op.id).map(|issue| SyncIssue::parse(issue)),
exportable,
} }
}) })
.collect()) .collect())
@@ -57,4 +68,235 @@ impl Core {
} }
Ok(moved) Ok(moved)
} }
/// Copy the content of queued op `id` to `dest`, which must not exist yet.
/// The op stays queued. Returns the bytes written.
pub(crate) fn export_pending_op(&self, id: i64, dest: &Path) -> CoreResult<u64> {
if !dest.is_absolute() {
return Err(CoreError::invalid(format!(
"destination must be an absolute path: {}",
dest.display()
)));
}
let ops = self
.db
.pending_ops()
.map_err(|e| CoreError::internal(format!("listing the queue: {e}")))?;
let op = ops
.iter()
.find(|op| op.id == id)
.ok_or_else(|| CoreError::not_found(format!("no queued change {id}")))?;
let blob = exportable(op).ok_or_else(|| {
CoreError::invalid(format!("queued change {id} carries no file content"))
})?;
let written = copy_to_new_file(blob, dest)?;
info!(id, dest = %dest.display(), written, "exported a queued change's content");
Ok(written)
}
/// Drop queued op `id` and undo the local change it stood for. Returns the
/// node's path.
///
/// A node only made here goes, as if it had been deleted before it was
/// sent. A node Drive has goes back to what Drive has: every change queued
/// on it goes, since each was made on top of the one before. What is queued
/// below it stays.
pub(crate) fn discard_pending_op(&self, id: i64) -> CoreResult<String> {
let ops = self
.db
.pending_ops()
.map_err(|e| CoreError::internal(format!("listing the queue: {e}")))?;
let op = ops
.into_iter()
.find(|op| op.id == id)
.ok_or_else(|| CoreError::not_found(format!("no queued change {id}")))?;
let uid = parse_node_uid(&op.uid)
.ok_or_else(|| CoreError::internal(format!("queued change {id} names {}", op.uid)))?;
let path = self
.db
.node_path(&op.uid)
.ok()
.flatten()
.filter(|path| !path.is_empty())
.or(op.name)
.unwrap_or_else(|| op.uid.clone());
if self.is_open_anywhere(&uid) {
return Err(CoreError::conflict(format!(
"{path} is open; close it first"
)));
}
if is_local_uid(&uid) {
self.discard_local_node(&uid, &path)?;
} else {
self.discard_remote_change(&uid, &path)?;
}
info!(id, %uid, path, "discarded a queued change");
Ok(path)
}
/// Forget a node made here, and the ops that would have sent it.
fn discard_local_node(&self, uid: &NodeUid, path: &str) -> CoreResult<()> {
// Its children's ops go with its own, and their content with them.
if self
.db
.has_children(uid)
.map_err(|e| CoreError::internal(format!("listing {path}: {e}")))?
{
return Err(CoreError::conflict(format!(
"{path} is not empty; discard what is in it first"
)));
}
self.discard_queued_ops(uid)
.map_err(|_| CoreError::internal(format!("dropping the changes to {path}")))?;
self.for_each_mount(|st, notify| {
background::forget_and_notify(st, notify, uid);
});
self.cache.evict(uid);
self.evict_reader(uid);
Ok(())
}
/// Put a node Drive has back as Drive has it.
fn discard_remote_change(&self, uid: &NodeUid, path: &str) -> CoreResult<()> {
// Asked first: the change is only dropped once what replaces it is known.
let node = self
.fetch_node_remote(uid)
.map_err(|e| CoreError::from_api(&e, "reading the node from Drive"))?
.filter(|node| !node.trashed);
self.cancel_upload(uid);
let blobs = self
.db
.discard_ops_on_node(&uid.to_string())
.map_err(|e| CoreError::internal(format!("dropping the changes to {path}: {e}")))?
.ok_or_else(|| {
CoreError::conflict(format!("{path} is being sent right now; try again"))
})?;
self.release_dropped_ops(uid, blobs);
self.hidden.lock().remove(uid);
match node {
Some(node) => self.for_each_mount(|st, notify| {
put_back(st, notify, node);
}),
None => self.for_each_mount(|st, notify| {
background::forget_and_notify(st, notify, uid);
}),
}
self.cache.evict(uid);
self.evict_reader(uid);
Ok(())
}
}
/// Show `node` as Drive has it, where Drive has it, under the inode it has.
fn put_back(st: &mut State, notify: &mut NotifyBatch, node: Node) {
let parent = node.parent_uid.as_ref().and_then(|parent| {
st.by_uid
.get(parent)
.copied()
.map(|ino| (ino, parent.clone()))
});
let here = st.by_uid.get(&node.uid).and_then(|ino| {
st.entries
.get(ino)
.map(|entry| (*ino, entry.parent, entry.node.name.clone()))
});
match (here, parent) {
(Some((ino, from, name)), Some((to, to_uid))) => {
st.relocate(ino, from, to, &to_uid, &node.name);
st.write_through(node.clone());
st.intern_mem(to, node);
notify.inval_entry(from, name);
notify.inval_inode(ino);
}
(here, parent) => {
if let Some((_, from, name)) = here {
st.forget_mem(&node.uid);
st.invalidate_listing(from);
notify.inval_entry(from, name);
}
if let Some((to, _)) = parent {
st.invalidate_listing(to);
notify.inval_entry(to, node.name.clone());
}
st.write_through(node);
}
}
}
/// Copy `blob` to `dest`, which must not exist: never replace something the
/// user already has.
fn copy_to_new_file(blob: &Path, dest: &Path) -> CoreResult<u64> {
let mut source = File::open(blob)
.map_err(|e| CoreError::internal(format!("open {}: {e}", blob.display())))?;
let mut out = std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.open(dest)
.map_err(|e| match e.kind() {
std::io::ErrorKind::AlreadyExists => {
CoreError::invalid(format!("{} already exists", dest.display()))
}
_ => CoreError::internal(format!("create {}: {e}", dest.display())),
})?;
std::io::copy(&mut source, &mut out)
.and_then(|written| out.flush().and(out.sync_all()).map(|()| written))
.map_err(|e| CoreError::internal(format!("write {}: {e}", dest.display())))
}
/// The staged blob of an op that carries file content.
fn exportable(op: &PendingOp) -> Option<&Path> {
matches!(op.kind.as_str(), OP_CREATE | OP_REVISION)
.then_some(op.blob_path.as_deref())
.flatten()
.map(Path::new)
}
#[cfg(test)]
mod tests {
use super::*;
fn op(kind: &str, blob: Option<&str>) -> PendingOp {
PendingOp {
id: 1,
kind: kind.to_string(),
uid: "vol~a".to_string(),
parent_uid: None,
name: None,
blob_path: blob.map(str::to_string),
meta_json: None,
created_at: 0,
attempts: 0,
last_error: None,
next_attempt_at: 0,
}
}
#[test]
fn only_a_change_that_carries_file_content_can_be_exported() {
assert!(exportable(&op(OP_CREATE, Some("/staging/a"))).is_some());
assert!(exportable(&op(OP_REVISION, Some("/staging/a"))).is_some());
assert!(exportable(&op(OP_CREATE, None)).is_none());
assert!(exportable(&op(OP_RENAME, None)).is_none());
assert!(exportable(&op(OP_TRASH, None)).is_none());
}
#[test]
fn an_export_copies_the_bytes_and_never_replaces_a_file() {
let dir = std::env::temp_dir().join(format!("pdfs-export-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let blob = dir.join("blob");
std::fs::write(&blob, b"queued bytes").unwrap();
let dest = dir.join("copy.txt");
assert_eq!(copy_to_new_file(&blob, &dest).unwrap(), 12);
assert_eq!(std::fs::read(&dest).unwrap(), b"queued bytes");
std::fs::write(&dest, b"the user's own").unwrap();
assert!(copy_to_new_file(&blob, &dest).is_err());
assert_eq!(std::fs::read(&dest).unwrap(), b"the user's own");
// The staged blob is the only copy of the change: an export leaves it.
assert_eq!(std::fs::read(&blob).unwrap(), b"queued bytes");
let _ = std::fs::remove_dir_all(&dir);
}
} }
+102 -41
View File
@@ -36,6 +36,21 @@ pub(super) const REPAIR_MAX: u64 = 512 * 1024 * 1024;
/// demand read in the SDK's flat block semaphore. /// demand read in the SDK's flat block semaphore.
pub(super) const PREFETCH_BUDGET: usize = 8; pub(super) const PREFETCH_BUDGET: usize = 8;
/// How long one read may wait on the network before it fails with `EIO`.
///
/// The SDK bounds each request, not a read: a block that times out is retried,
/// and a read can need several blocks, a block table and a repair. On a link
/// that stalled without dropping, one `read` held a FUSE worker for 34 minutes
/// and the application behind it with it (`docs/BUGS.md` B107). An error the
/// caller can retry beats a process stuck in `D` state.
pub(super) const READ_FETCH_TIMEOUT: Duration = Duration::from_secs(120);
/// How long a read of blocks nothing has cached waits for a lost link to come
/// back before failing. Long enough to ride out a Wi-Fi blip, short enough
/// that a program reading while the laptop is really offline gets its `EIO`
/// before the user gives up on it.
const READ_OFFLINE_WAIT: Duration = Duration::from_secs(30);
/// Cap on tracked sequential streams. Only reads in progress matter, so the map /// Cap on tracked sequential streams. Only reads in progress matter, so the map
/// is cleared wholesale rather than aged — losing the ramp costs one read. /// is cleared wholesale rather than aged — losing the ramp costs one read.
const MAX_PREFETCH_STREAMS: usize = 256; const MAX_PREFETCH_STREAMS: usize = 256;
@@ -55,7 +70,7 @@ const MAX_OPEN_READERS: usize = 64;
/// `(mtime, size)` — the same validity pair the content cache uses (a new /// `(mtime, size)` — the same validity pair the content cache uses (a new
/// revision bumps mtime). On a mismatch the reader is dropped and reopened. /// revision bumps mtime). On a mismatch the reader is dropped and reopened.
pub(super) struct CachedReader { pub(super) struct CachedReader {
reader: Arc<RevisionReader>, reader: Arc<dyn RevisionRead>,
mtime: i64, mtime: i64,
size: u64, size: u64,
/// For LRU eviction. /// For LRU eviction.
@@ -86,7 +101,7 @@ pub(super) struct PendingOpen {
/// must not join this open — it would get a reader for the wrong bytes. /// must not join this open — it would get a reader for the wrong bytes.
mtime: i64, mtime: i64,
size: u64, size: u64,
rx: tokio::sync::watch::Receiver<Option<Result<Arc<RevisionReader>, Errno>>>, rx: tokio::sync::watch::Receiver<Option<Result<Arc<dyn RevisionRead>, Errno>>>,
} }
/// Distinguishes `PendingOpen`s. Only uniqueness matters, so a plain counter /// Distinguishes `PendingOpen`s. Only uniqueness matters, so a plain counter
@@ -298,16 +313,16 @@ impl Core {
uid: &NodeUid, uid: &NodeUid,
mtime: i64, mtime: i64,
fsize: u64, fsize: u64,
) -> Result<Arc<RevisionReader>, Errno> { ) -> Result<Arc<dyn RevisionRead>, Errno> {
loop { loop {
// Decide under the lock, act outside it — network I/O must never run // Decide under the lock, act outside it — network I/O must never run
// under a std `Mutex`, and an await must never hold one. // under a std `Mutex`, and an await must never hold one.
enum Act { enum Act {
Hit(Arc<RevisionReader>), Hit(Arc<dyn RevisionRead>),
Join(tokio::sync::watch::Receiver<Option<Result<Arc<RevisionReader>, Errno>>>), Join(tokio::sync::watch::Receiver<Option<Result<Arc<dyn RevisionRead>, Errno>>>),
Lead( Lead(
u64, u64,
tokio::sync::watch::Sender<Option<Result<Arc<RevisionReader>, Errno>>>, tokio::sync::watch::Sender<Option<Result<Arc<dyn RevisionRead>, Errno>>>,
), ),
} }
let act = { let act = {
@@ -361,9 +376,10 @@ impl Core {
} }
Act::Lead(id, tx) => { Act::Lead(id, tx) => {
debug!(%uid, mtime, fsize, "opening revision"); debug!(%uid, mtime, fsize, "opening revision");
let result = match self.client.open_revision(uid).await { let result = match self.drive.open_revision(uid).await {
Ok(reader) => Ok(Arc::new(reader)), Ok(reader) => Ok(reader),
Err(e) => { Err(e) => {
self.lost_link(&e, "open a revision");
warn!(%uid, error = %e, "open_revision failed"); warn!(%uid, error = %e, "open_revision failed");
Err(Errno::EIO) Err(Errno::EIO)
} }
@@ -448,7 +464,7 @@ impl Core {
/// it, and the [`BLOCK_SIZE`] assumption when it has not — which is every /// it, and the [`BLOCK_SIZE`] assumption when it has not — which is every
/// first read, because learning it means opening a `RevisionReader`, and /// first read, because learning it means opening a `RevisionReader`, and
/// opening one per read is precisely the per-file key derivation the block /// opening one per read is precisely the per-file key derivation the block
/// cache exists to avoid (bugs.md B12, B85). /// cache exists to avoid (`docs/BUGS.md` B12, B85).
/// ///
/// Being wrong here is not a correctness problem: `read_at` plans over the /// Being wrong here is not a correctness problem: `read_at` plans over the
/// real sizes and clamps to what was asked for either way. It costs a /// real sizes and clamps to what was asked for either way. It costs a
@@ -477,15 +493,20 @@ impl Core {
) -> Result<Vec<u8>, Errno> { ) -> Result<Vec<u8>, Errno> {
// A queued write has not reached the remote yet, so the remote's current // A queued write has not reached the remote yet, so the remote's current
// revision is stale and the staged blob is the truth. Serve from it until // revision is stale and the staged blob is the truth. Serve from it until
// the drain worker lands the upload (offline.md Phase 3). // the drain worker lands the upload.
if let Some(pending) = self.pending.lock().get(uid).cloned() { if let Some(pending) = self.pending.lock().get(uid).cloned() {
return self.read_pending(&pending, offset, len); return self.read_pending(&pending, offset, len);
} }
// A node created offline and never written has no blob and no remote: it // A node created offline and never written has no blob and no remote: it
// is an empty file, and asking the API about a `local~` uid would only // is an empty file, and asking the API about a `local~` uid would only
// earn a 404 (offline.md Phase 3b). // earn a 404. Unless its create landed since the
// read looked: the landing moved the queued write to the uid Drive
// gave it, and the row says which (`docs/BUGS.md` B179).
if is_local_uid(uid) { if is_local_uid(uid) {
return Ok(Vec::new()); return match self.landed_stand_in_uid(uid) {
Some(real) => self.read_range(&real, mtime, fsize, offset, len, cache_blocks),
None => Ok(Vec::new()),
};
} }
self.read_range_remote(uid, mtime, fsize, offset, len, cache_blocks) self.read_range_remote(uid, mtime, fsize, offset, len, cache_blocks)
} }
@@ -552,7 +573,7 @@ impl Core {
/// leader's claim is gone by then, so this cannot recurse. /// leader's claim is gone by then, so this cannot recurse.
async fn fetch_block( async fn fetch_block(
&self, &self,
reader: &Arc<RevisionReader>, reader: &Arc<dyn RevisionRead>,
uid: &NodeUid, uid: &NodeUid,
mtime: i64, mtime: i64,
fsize: u64, fsize: u64,
@@ -615,10 +636,10 @@ impl Core {
/// ///
/// A block that comes back shorter than the file's size says it must be is /// A block that comes back shorter than the file's size says it must be is
/// never served, cached or promoted: it is repaired from the whole-file /// never served, cached or promoted: it is repaired from the whole-file
/// download, or the read fails. See [`Core::repair_block`] (bugs.md B84). /// download, or the read fails. See [`Core::repair_block`] (`docs/BUGS.md` B84).
async fn read_block( async fn read_block(
&self, &self,
reader: &Arc<RevisionReader>, reader: &Arc<dyn RevisionRead>,
uid: &NodeUid, uid: &NodeUid,
mtime: i64, mtime: i64,
fsize: u64, fsize: u64,
@@ -630,7 +651,7 @@ impl Core {
// file's boundaries are wrong, and the failure is not confined to short // file's boundaries are wrong, and the failure is not confined to short
// reads: sizes that *overstate* a block shift every block after it and // reads: sizes that *overstate* a block shift every block after it and
// return full-length reads of the wrong bytes. So a disagreement of any // return full-length reads of the wrong bytes. So a disagreement of any
// kind takes the file off the range path entirely (bugs.md B84). // kind takes the file off the range path entirely (`docs/BUGS.md` B84).
let bytes = if reader.size() != fsize { let bytes = if reader.size() != fsize {
warn!( warn!(
%uid, bidx = span.idx, fsize, %uid, bidx = span.idx, fsize,
@@ -643,12 +664,13 @@ impl Core {
// Learned here rather than on a path of its own: this is where a // Learned here rather than on a path of its own: this is where a
// reader is open anyway, and recording the real geometry is what // reader is open anyway, and recording the real geometry is what
// lets the *next* read plan its blocks on the revision's own // lets the *next* read plan its blocks on the revision's own
// boundaries instead of the 4 MiB fallback (bugs.md B85). // boundaries instead of the 4 MiB fallback (`docs/BUGS.md` B85).
if cache_blocks { if cache_blocks {
self.cache self.cache
.store_block_geometry(uid, mtime, fsize, reader.block_sizes()); .store_block_geometry(uid, mtime, fsize, reader.block_sizes());
} }
let bytes = reader.read_at(span.start, span.len).await.map_err(|e| { let bytes = reader.read_at(span.start, span.len).await.map_err(|e| {
self.lost_link(&e, "read a block");
warn!(%uid, start = span.start, len = span.len, error = %e, "block read failed"); warn!(%uid, start = span.start, len = span.len, error = %e, "block read failed");
Errno::EIO Errno::EIO
})?; })?;
@@ -684,7 +706,7 @@ impl Core {
/// The range path derives every block boundary from the revision's recorded /// The range path derives every block boundary from the revision's recorded
/// block sizes. When those understate the file, reads come back short with /// block sizes. When those understate the file, reads come back short with
/// no error anywhere, and the caller hands userspace a truncated file that /// no error anywhere, and the caller hands userspace a truncated file that
/// looks like a clean EOF (bugs.md B84). `download_file_to` does not do that /// looks like a clean EOF (`docs/BUGS.md` B84). `download_file_to` does not do that
/// arithmetic — it streams the block list — so it returns the whole file /// arithmetic — it streams the block list — so it returns the whole file
/// where the range path cannot, which is why pinning was the workaround. /// where the range path cannot, which is why pinning was the workaround.
/// This does the same thing on the user's behalf, once, and the resulting /// This does the same thing on the user's behalf, once, and the resulting
@@ -761,7 +783,7 @@ impl Core {
error!(%uid, %error, "creating a scratch file for a repair download failed"); error!(%uid, %error, "creating a scratch file for a repair download failed");
Errno::EIO Errno::EIO
})?; })?;
let outcome = self.client.download_file_to(uid, &mut file).await; let outcome = self.drive.download_file_to(uid, &mut file).await;
let stored = (|| { let stored = (|| {
outcome.map_err(|error| { outcome.map_err(|error| {
warn!(%uid, %error, "repair download failed"); warn!(%uid, %error, "repair download failed");
@@ -896,7 +918,7 @@ impl Core {
let end = offset.saturating_add(len).min(fsize); let end = offset.saturating_add(len).min(fsize);
let mut out = Vec::with_capacity((end - offset) as usize); let mut out = Vec::with_capacity((end - offset) as usize);
// On the revision's own boundaries where they are known, and on the // On the revision's own boundaries where they are known, and on the
// 4 MiB fallback where they are not (bugs.md B85). // 4 MiB fallback where they are not (`docs/BUGS.md` B85).
let wanted = self.block_geometry(uid, mtime, fsize).spans(offset, end); let wanted = self.block_geometry(uid, mtime, fsize).spans(offset, end);
let (Some(&head), Some(&tail)) = (wanted.first(), wanted.last()) else { let (Some(&head), Some(&tail)) = (wanted.first(), wanted.last()) else {
return Ok(out); return Ok(out);
@@ -939,28 +961,67 @@ impl Core {
} }
if !misses.is_empty() { if !misses.is_empty() {
let fetched = self.rt.block_on(async { // One deadline for the whole read, including any wait for a lost
// Resolve the file's keys and block table once, then read every // link. A fetch that failed because the link went down is tried
// missing block through the shared reader. Previously each block // once more when it is back, so a blip is a slow read, not an `EIO`.
// called `download_range`, which redid that resolution per block. let deadline = Instant::now() + READ_FETCH_TIMEOUT;
let reader = self.revision_reader(uid, mtime, fsize).await?; let mut retried = false;
let fetched = loop {
if !self.is_online() {
let wait =
READ_OFFLINE_WAIT.min(deadline.saturating_duration_since(Instant::now()));
if !self.wait_online(wait) {
debug!(%uid, offset, len, "read of uncached blocks while offline; failing it");
return Err(Errno::EIO);
}
}
let fetch = async {
// Resolve the file's keys and block table once, then read every
// missing block through the shared reader. Previously each block
// called `download_range`, which redid that resolution per block.
let reader = self.revision_reader(uid, mtime, fsize).await?;
let mut set = tokio::task::JoinSet::new(); let mut set = tokio::task::JoinSet::new();
for &span in &misses { for &span in &misses {
let (core, reader, uid) = (self.clone(), reader.clone(), uid.clone()); let (core, reader, uid) = (self.clone(), reader.clone(), uid.clone());
set.spawn(async move { set.spawn(async move {
core.fetch_block(&reader, &uid, mtime, fsize, span, cache_blocks) core.fetch_block(&reader, &uid, mtime, fsize, span, cache_blocks)
.await .await
.map(|bytes| (span.idx, bytes)) .map(|bytes| (span.idx, bytes))
}); });
}
let mut out = Vec::with_capacity(misses.len());
while let Some(joined) = set.join_next().await {
// A join error means the task panicked; surface it as EIO.
out.push(joined.map_err(|_| Errno::EIO)??);
}
Ok::<_, Errno>(out)
};
// Dropping the fetch on the deadline aborts the block tasks with
// it; blocks that already landed stay cached for the retry. The
// timer is built inside the runtime: a FUSE worker has no reactor
// of its own, and building it out here panicked every remote read.
let budget = deadline.saturating_duration_since(Instant::now());
match self
.rt
.block_on(async { tokio::time::timeout(budget, fetch).await })
{
Ok(Ok(fetched)) => break fetched,
// The fetch saw the link go down; the top of the loop waits
// for it within what is left of the deadline.
Ok(Err(_)) if !retried && !self.is_online() => retried = true,
Ok(Err(e)) => return Err(e),
Err(_) => {
warn!(
%uid, offset, len, secs = READ_FETCH_TIMEOUT.as_secs(),
"read got no answer from the network in time; failing it"
);
self.evict_reader(uid);
self.mark_offline("read", &crate::link::no_answer(READ_FETCH_TIMEOUT));
return Err(Errno::EIO);
}
} }
let mut out = Vec::with_capacity(misses.len()); };
while let Some(joined) = set.join_next().await {
// A join error means the task panicked; surface it as EIO.
out.push(joined.map_err(|_| Errno::EIO)??);
}
Ok::<_, Errno>(out)
})?;
for (bidx, bytes) in fetched { for (bidx, bytes) in fetched {
blocks[(bidx - first) as usize] = Some(bytes); blocks[(bidx - first) as usize] = Some(bytes);
} }
@@ -977,7 +1038,7 @@ impl Core {
// — the cache validates it on read, the network path in `read_block` // — the cache validates it on read, the network path in `read_block`
// — so this refuses rather than quietly contributing fewer bytes // — so this refuses rather than quietly contributing fewer bytes
// than the range covers, which is what turned a bad block into a // than the range covers, which is what turned a bad block into a
// truncated file (bugs.md B84). // truncated file (`docs/BUGS.md` B84).
if block.len() as u64 != span.len { if block.len() as u64 != span.len {
error!( error!(
%uid, bidx = span.idx, len = block.len(), expected = span.len, fsize, %uid, bidx = span.idx, len = block.len(), expected = span.len, fsize,
+1 -4
View File
@@ -248,10 +248,7 @@ impl Core {
let outcomes = self let outcomes = self
.rt .rt
.block_on( .block_on(self.drive.trash_nodes(std::slice::from_ref(&candidate.uid)))
self.client
.trash_nodes(std::slice::from_ref(&candidate.uid)),
)
.map_err(|e| CoreError::from_api(&e, "trash the mis-dated original"))?; .map_err(|e| CoreError::from_api(&e, "trash the mis-dated original"))?;
if let Some((_, Err(e))) = outcomes.into_iter().next() { if let Some((_, Err(e))) = outcomes.into_iter().next() {
return Err(CoreError::from_api(&e, "trash the mis-dated original")); return Err(CoreError::from_api(&e, "trash the mis-dated original"));
+611
View File
@@ -0,0 +1,611 @@
//! Moving a file or folder between two locations without moving its content.
//!
//! `~/ProtonDrive` and every on-demand synced folder are separate FUSE
//! sessions, and a mirror folder is a plain local directory. The kernel refuses
//! `rename(2)` between two mounts with `EXDEV` before the daemon hears of it, so
//! `mv` falls back to copy and delete: every file is downloaded, uploaded again,
//! and the original is trashed.
//!
//! On Proton Drive the same move is one metadata change, because My files and
//! this computer's device folders all live on the main volume. So a move
//! request that names two absolute paths is resolved here, against every
//! location this daemon knows, and done as a server-side move. The local side
//! is then brought level: a mount forgets the node and re-lists, and a mirror
//! folder either renames its copy along or drops it once Drive has it.
use std::collections::HashMap;
use std::path::{Component, Path, PathBuf};
use std::sync::Arc;
use std::time::Duration;
use parking_lot::Mutex;
use pdfs_core::db::{StoredSyncEntry, StoredSyncFolder};
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::NodeKind;
use proton_drive_rs::proton_sdk::ids::NodeUid;
use tracing::{info, warn};
use super::sync::mirror_subtree_unsynced;
use super::{Core, is_local_uid, parse_uid};
/// How long a move waits for a mirror folder's sync pass to finish before it
/// reports the folder busy. A pass can run for hours on a first sync, so the
/// move does not wait it out.
const MIRROR_LOCK_WAIT: Duration = Duration::from_secs(5);
/// Where one side of a move lives.
enum Place {
/// My files or an on-demand folder: a live FUSE mount. `core` is rooted at
/// that mount and `rel` is relative to its mountpoint.
Mounted { core: Box<Core>, rel: PathBuf },
/// A mirror folder: a local directory kept in step by the sync engine.
/// `rel` is `/`-separated, relative to the folder, as the baseline keys it.
Mirror {
folder: StoredSyncFolder,
rel: String,
},
}
impl Place {
fn mirror_id(&self) -> Option<i64> {
match self {
Place::Mirror { folder, .. } => Some(folder.id),
Place::Mounted { .. } => None,
}
}
fn is_root(&self) -> bool {
match self {
Place::Mounted { rel, .. } => rel.as_os_str().is_empty(),
Place::Mirror { rel, .. } => rel.is_empty(),
}
}
}
/// The mirror folder that holds `abs`, and `abs` relative to it. The deepest
/// folder wins, although folders are not expected to nest.
fn mirror_covering(
folders: Vec<StoredSyncFolder>,
abs: &Path,
) -> Option<(StoredSyncFolder, String)> {
let folder = folders
.into_iter()
.filter(|f| f.mode == "mirror" && abs.starts_with(&f.local_path))
.max_by_key(|f| Path::new(&f.local_path).components().count())?;
let rel = abs.strip_prefix(&folder.local_path).ok()?;
let parts = rel
.components()
.map(|c| match c {
Component::Normal(part) => part.to_str(),
_ => None,
})
.collect::<Option<Vec<_>>>()?;
Some((folder, parts.join("/")))
}
/// `name` inside the folder at `parent`, in baseline form.
fn join_rel(parent: &str, name: &str) -> String {
if parent.is_empty() {
name.to_string()
} else {
format!("{parent}/{name}")
}
}
/// Rename `from` to `to`, failing with `EEXIST` rather than replacing
/// whatever appeared at `to` since it was checked. A filesystem that cannot
/// promise that gets a plain rename after one more look.
fn rename_noreplace(from: &Path, to: &Path) -> std::io::Result<()> {
use std::ffi::CString;
use std::os::unix::ffi::OsStrExt;
let cstr = |p: &Path| {
CString::new(p.as_os_str().as_bytes())
.map_err(|_| std::io::Error::from_raw_os_error(libc::EINVAL))
};
let (c_from, c_to) = (cstr(from)?, cstr(to)?);
// SAFETY: both are valid NUL-terminated paths that outlive the call.
let rc = unsafe {
libc::syscall(
libc::SYS_renameat2,
libc::AT_FDCWD,
c_from.as_ptr(),
libc::AT_FDCWD,
c_to.as_ptr(),
libc::RENAME_NOREPLACE,
)
};
if rc == 0 {
return Ok(());
}
let err = std::io::Error::last_os_error();
match err.raw_os_error() {
Some(libc::EINVAL | libc::ENOSYS) => {
if std::fs::symlink_metadata(to).is_ok() {
return Err(std::io::Error::from_raw_os_error(libc::EEXIST));
}
std::fs::rename(from, to)
}
_ => Err(err),
}
}
/// Whether `uid` is `root` or lies under it, as far as the nodes table knows.
fn under(core: &Core, root: &NodeUid, uid: &str) -> bool {
let root = root.to_string();
uid == root || matches!(core.db.path_relative_to(&root, uid), Ok(Some(_)))
}
impl Core {
/// Move the file or folder at `src` into the folder at `dest_parent`. Both
/// are absolute local paths and may be in different locations: My files,
/// an on-demand folder, or a mirror folder. Returns the moved name.
///
/// Within one mount this is [`Core::move_to`]. Across locations it
/// is a server-side move followed by local bookkeeping; see the module
/// documentation. Anything the move could lose is refused up front.
pub(crate) fn move_between(&self, src: &Path, dest_parent: &Path) -> CoreResult<String> {
let source = self.locate(src)?;
let dest = self.locate(dest_parent)?;
if let (
Place::Mounted { core, rel },
Place::Mounted {
core: dest_core,
rel: dest_rel,
},
) = (&source, &dest)
&& Arc::ptr_eq(&core.root, &dest_core.root)
{
return core.move_to(rel, dest_rel);
}
if source.is_root() {
return Err(CoreError::invalid(format!(
"{} is the root of a synced location and cannot be moved",
src.display()
)));
}
if dest_parent.starts_with(src) {
return Err(CoreError::invalid("cannot move a folder into itself"));
}
let name = src
.file_name()
.and_then(|n| n.to_str())
.ok_or_else(|| CoreError::invalid(format!("{} has no usable name", src.display())))?
.to_string();
// Hold every mirror folder involved, lowest id first, so no sync pass
// sees the move half done and two moves cannot deadlock.
let mut ids: Vec<i64> = [source.mirror_id(), dest.mirror_id()]
.into_iter()
.flatten()
.collect();
ids.sort_unstable();
ids.dedup();
let locks: Vec<Arc<Mutex<()>>> = ids.iter().map(|&id| self.sync_lock(id)).collect();
let mut guards = Vec::with_capacity(locks.len());
for lock in &locks {
guards.push(lock.try_lock_for(MIRROR_LOCK_WAIT).ok_or_else(|| {
CoreError::conflict("a synced folder is busy syncing; try again when it is done")
})?);
}
let mut baselines: HashMap<i64, HashMap<String, StoredSyncEntry>> = HashMap::new();
for &id in &ids {
let baseline = self
.db
.sync_entries(id)
.map_err(|e| CoreError::internal(format!("db: {e}")))?;
baselines.insert(id, baseline);
}
let src_uid = self.source_uid(&source, &baselines)?;
let dest_uid = self.dest_uid(&dest, &baselines, &name)?;
if src_uid.volume_id != dest_uid.volume_id {
return Err(CoreError::invalid(
"the two locations are on different Proton Drive volumes; copy it instead",
));
}
if under(self, &src_uid, &dest_uid.to_string()) {
return Err(CoreError::invalid("cannot move a folder into itself"));
}
// Mirror to mirror renames the local copy along, so it needs nothing
// proven about it. Another filesystem makes that impossible, and the
// pair then drops the source copy like any other move out of a mirror.
let mut renamed = None;
if let (
Place::Mirror { folder, rel },
Place::Mirror {
folder: dest_folder,
rel: dest_rel,
},
) = (&source, &dest)
{
let from = src.to_path_buf();
let to = dest_parent.join(&name);
match rename_noreplace(&from, &to) {
Ok(()) => renamed = Some((from, to)),
Err(e) if e.raw_os_error() == Some(libc::EXDEV) => {
info!(from = %folder.local_path, to = %dest_folder.local_path,
"move: mirror folders are on different filesystems");
}
Err(e) if e.raw_os_error() == Some(libc::EEXIST) => {
return Err(CoreError::conflict(format!(
"{name} already exists in {}",
dest_parent.display()
)));
}
Err(e) => {
return Err(CoreError::internal(format!(
"could not move {} to {}: {e}",
rel,
join_rel(dest_rel, &name)
)));
}
}
}
if renamed.is_none()
&& let Place::Mirror { folder, rel } = &source
&& let Some(reason) =
mirror_subtree_unsynced(Path::new(&folder.local_path), rel, &baselines[&folder.id])
{
return Err(CoreError::conflict(format!(
"{name} is not fully synced yet ({reason}); wait for the sync to finish"
)));
}
if let Err(e) = self.move_remote(&src_uid, &dest_uid) {
if let Some((from, to)) = &renamed
&& let Err(back) = std::fs::rename(to, from)
{
warn!(from = %to.display(), to = %from.display(), error = %back,
"move: could not put the local copy back after the move failed");
}
return Err(CoreError::from_api(&e, "move"));
}
self.settle_source(
&source,
&src_uid,
renamed.is_some(),
&dest,
&dest_uid,
&name,
src,
);
self.settle_dest(&dest, &dest_uid, &name);
drop(guards);
if renamed.is_none()
&& let Place::Mirror { folder, .. } = &dest
{
// Nothing local exists yet; the folder's next pass downloads it.
self.sync_now(Some(folder.id));
}
Ok(name)
}
/// Which location `abs` is in. A mount is looked for first: a mirror folder
/// cannot live inside a FUSE mount, and a mount is what the kernel sees.
fn locate(&self, abs: &Path) -> CoreResult<Place> {
if let Some((core, rel)) = self.rooted_at(abs) {
if !rel
.components()
.all(|c| matches!(c, Component::Normal(_) | Component::CurDir))
{
return Err(CoreError::invalid(format!(
"{} escapes its location",
abs.display()
)));
}
return Ok(Place::Mounted {
core: Box::new(core),
rel,
});
}
let folders = self
.db
.sync_folder_list()
.map_err(|e| CoreError::internal(format!("db: {e}")))?;
mirror_covering(folders, abs)
.map(|(folder, rel)| Place::Mirror { folder, rel })
.ok_or_else(|| {
CoreError::invalid(format!(
"{} is not in a Proton Drive location",
abs.display()
))
})
}
/// The node being moved, once everything that must hold before it leaves
/// its location has been checked.
fn source_uid(
&self,
source: &Place,
baselines: &HashMap<i64, HashMap<String, StoredSyncEntry>>,
) -> CoreResult<NodeUid> {
match source {
Place::Mounted { core, rel } => {
let (ino, uid) = core.resolve(rel)?;
if is_local_uid(&uid) {
return Err(CoreError::conflict(
"it is not uploaded yet; wait for the upload to finish",
));
}
let old_parent = core.source_parent_uid(ino, rel)?;
for authority in [&uid, &old_parent] {
core.require_uid_writable(authority)
.map_err(|e| core.errno_error(e, "move access"))?;
}
if self.has_pending_work_under(&uid)? {
return Err(CoreError::conflict(
"it has changes that are not uploaded yet; wait for the upload to finish",
));
}
Ok(uid)
}
Place::Mirror { folder, rel } => baselines[&folder.id]
.get(rel)
.and_then(|entry| entry.remote_uid.as_deref())
.and_then(parse_uid)
.ok_or_else(|| {
CoreError::conflict(
"it is not on Proton Drive yet; wait for the sync to finish",
)
}),
}
}
/// The folder the node moves into. Refuses one that already holds `name`
/// on this disk; Drive refuses a clash on its side by itself.
fn dest_uid(
&self,
dest: &Place,
baselines: &HashMap<i64, HashMap<String, StoredSyncEntry>>,
name: &str,
) -> CoreResult<NodeUid> {
let uid = match dest {
Place::Mounted { core, rel } => {
let (ino, uid) = core.resolve(rel)?;
let is_folder = core
.state()
.entries
.get(&ino)
.is_some_and(|entry| matches!(entry.node.kind, NodeKind::Folder));
if !is_folder {
return Err(CoreError::invalid("the destination is not a folder"));
}
core.require_uid_writable(&uid)
.map_err(|e| core.errno_error(e, "move access"))?;
uid
}
Place::Mirror { folder, rel } => {
let local = Path::new(&folder.local_path).join(rel);
if !local.is_dir() {
return Err(CoreError::invalid("the destination is not a folder"));
}
if std::fs::symlink_metadata(local.join(name)).is_ok() {
return Err(CoreError::conflict(format!(
"{name} already exists in the destination"
)));
}
let uid = if rel.is_empty() {
Some(folder.remote_uid.as_str())
} else {
baselines[&folder.id]
.get(rel)
.and_then(|entry| entry.remote_uid.as_deref())
};
uid.and_then(parse_uid).ok_or_else(|| {
CoreError::conflict(
"the destination is not on Proton Drive yet; wait for the sync to finish",
)
})?
}
};
if is_local_uid(&uid) {
return Err(CoreError::conflict(
"the destination is not uploaded yet; wait for the upload to finish",
));
}
Ok(uid)
}
/// Whether the queue owes Drive anything for `uid` or a node under it, or a
/// file under it is open for writing. Moving such a node away would strand
/// the pending change against a path that no longer leads to it.
fn has_pending_work_under(&self, uid: &NodeUid) -> CoreResult<bool> {
let ops = self
.db
.pending_ops()
.map_err(|e| CoreError::internal(format!("db: {e}")))?;
if ops.iter().any(|op| {
under(self, uid, &op.uid)
|| op
.parent_uid
.as_deref()
.is_some_and(|p| under(self, uid, p))
}) {
return Ok(true);
}
// Collected first so no State lock is held across the nodes-table reads.
let mut writing = Vec::new();
self.for_each_state(|st| {
writing.extend(st.active_writes.values().map(|w| w.uid.to_string()));
});
Ok(writing.iter().any(|w| under(self, uid, w)))
}
/// Bring the source location level with a move Drive has made.
#[allow(clippy::too_many_arguments)]
fn settle_source(
&self,
source: &Place,
uid: &NodeUid,
renamed: bool,
dest: &Place,
dest_uid: &NodeUid,
name: &str,
src: &Path,
) {
match source {
Place::Mounted { core, rel } => {
// Moved everywhere, so no mount keeps serving it under the old
// path and its row stays (`docs/BUGS.md` B117). The content
// cache is keyed by uid and stays valid.
self.relocate_everywhere(uid, dest_uid, name);
core.invalidate_parent_listing(rel);
}
Place::Mirror { folder, rel } if renamed => {
let Place::Mirror {
folder: dest_folder,
rel: dest_rel,
} = dest
else {
unreachable!("a local rename needs a mirror destination");
};
if let Err(e) = self.db.sync_entries_move(
folder.id,
rel,
dest_folder.id,
&join_rel(dest_rel, name),
) {
// The next passes see a copy with no baseline on one side
// and a baseline with no copy on the other. That uploads
// the files again and trashes the moved node, but loses
// nothing.
warn!(error = ?e, "move: could not carry the sync baseline along");
}
}
Place::Mirror { folder, rel } => {
// Drive had all of it when the move was checked, so the local
// copy is only a stale duplicate now. Something written into it
// since would go with it, so look again first; if anything is
// new, the copy stays and the next pass uploads it as new.
let changed = match self.db.sync_entries(folder.id) {
Ok(baseline) => {
mirror_subtree_unsynced(Path::new(&folder.local_path), rel, &baseline)
}
Err(e) => Some(format!("db: {e}")),
};
if let Err(e) = self.db.sync_entries_remove_subtree(folder.id, rel) {
warn!(error = ?e, "move: could not drop the sync baseline");
}
if let Some(reason) = changed {
warn!(path = %src.display(), reason,
"move: the local copy changed during the move; keeping it");
return;
}
let removed = if src.is_dir() {
std::fs::remove_dir_all(src)
} else {
std::fs::remove_file(src)
};
if let Err(e) = removed {
warn!(path = %src.display(), error = %e,
"move: could not remove the local copy after the move");
}
}
}
}
/// Bring the destination location level with a move Drive has made.
fn settle_dest(&self, dest: &Place, uid: &NodeUid, name: &str) {
if let Place::Mounted { .. } = dest {
self.invalidate_children_of(uid);
// A lookup that missed before the move may be cached as absent.
self.for_each_mount(|st, notify| {
if let Some(&ino) = st.by_uid.get(uid) {
notify.inval_entry(ino, name.to_string());
}
});
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn folder(id: i64, local_path: &str, mode: &str) -> StoredSyncFolder {
StoredSyncFolder {
id,
local_path: local_path.to_string(),
remote_uid: format!("vol~root{id}"),
remote_share_id: String::new(),
mode: mode.to_string(),
pending_mode: None,
state: "idle".to_string(),
last_sync: 0,
}
}
#[test]
fn mirror_covering_finds_the_mirror_folder_and_the_path_inside_it() {
let folders = vec![folder(1, "/home/u/Docs", "mirror")];
let (found, rel) = mirror_covering(folders, Path::new("/home/u/Docs/a/b.txt")).unwrap();
assert_eq!(found.id, 1);
assert_eq!(rel, "a/b.txt");
}
#[test]
fn mirror_covering_gives_an_empty_path_for_the_folder_itself() {
let folders = vec![folder(1, "/home/u/Docs", "mirror")];
let (_, rel) = mirror_covering(folders, Path::new("/home/u/Docs")).unwrap();
assert_eq!(rel, "");
}
#[test]
fn mirror_covering_prefers_the_deepest_folder() {
let folders = vec![
folder(1, "/home/u/Docs", "mirror"),
folder(2, "/home/u/Docs/Work", "mirror"),
];
let (found, rel) = mirror_covering(folders, Path::new("/home/u/Docs/Work/x")).unwrap();
assert_eq!(found.id, 2);
assert_eq!(rel, "x");
}
#[test]
fn mirror_covering_skips_on_demand_folders_and_name_prefixed_siblings() {
let folders = || {
vec![
folder(1, "/home/u/Docs", "mirror"),
folder(2, "/home/u/Media", "ondemand"),
]
};
assert!(mirror_covering(folders(), Path::new("/home/u/Media/x")).is_none());
assert!(mirror_covering(folders(), Path::new("/home/u/Docs2/x")).is_none());
}
#[test]
fn mirror_covering_refuses_a_path_that_climbs_out() {
let folders = vec![folder(1, "/home/u/Docs", "mirror")];
assert!(mirror_covering(folders, Path::new("/home/u/Docs/../etc")).is_none());
}
#[test]
fn join_rel_names_a_child_of_the_folder_root_without_a_slash() {
assert_eq!(join_rel("", "a"), "a");
assert_eq!(join_rel("x/y", "a"), "x/y/a");
}
#[test]
fn a_local_rename_never_replaces_what_is_already_there() {
let dir = std::env::temp_dir().join(format!("pdfs-relocate-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let (a, b, c) = (dir.join("a"), dir.join("b"), dir.join("c"));
std::fs::write(&a, b"moved").unwrap();
std::fs::write(&b, b"kept").unwrap();
let err = rename_noreplace(&a, &b).unwrap_err();
assert_eq!(err.raw_os_error(), Some(libc::EEXIST));
assert_eq!(std::fs::read(&b).unwrap(), b"kept");
assert!(a.exists());
rename_noreplace(&a, &c).unwrap();
assert!(!a.exists());
assert_eq!(std::fs::read(&c).unwrap(), b"moved");
std::fs::remove_dir_all(&dir).unwrap();
}
}
+1 -1
View File
@@ -232,7 +232,7 @@ impl Core {
let mut out = std::io::BufWriter::new(file); let mut out = std::io::BufWriter::new(file);
let written = match self let written = match self
.rt .rt
.block_on(self.client.download_revision_to(uid, revision_id, &mut out)) .block_on(self.drive.download_revision_to(uid, revision_id, &mut out))
.map_err(|e| CoreError::from_api(&e, "download revision")) .map_err(|e| CoreError::from_api(&e, "download revision"))
.and_then(|()| { .and_then(|()| {
use std::io::Write as _; use std::io::Write as _;
+35 -28
View File
@@ -21,7 +21,7 @@ use proton_drive_rs::{MemberRole, NodeKind};
use proton_drive_rs::Node; use proton_drive_rs::Node;
use super::{Core, ROOT_INO, node_size, parse_uid, public_link_info, role_from_str, role_to_str}; use super::{Core, node_size, parse_uid, public_link_info, role_from_str, role_to_str};
/// A node from a *shared* listing as a [`DirEntry`]. /// A node from a *shared* listing as a [`DirEntry`].
/// ///
@@ -75,6 +75,7 @@ fn shared_entry_of(core: &Core, n: Node) -> DirEntry {
shared_by_unverified, shared_by_unverified,
trashed_at: 0, trashed_at: 0,
trashed_from: None, trashed_from: None,
issue: None,
} }
} }
@@ -463,7 +464,7 @@ impl Core {
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes(&uids)) .block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?; .map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes Ok(nodes
.into_iter() .into_iter()
@@ -481,14 +482,14 @@ impl Core {
parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?; parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?;
let child_uids = self let child_uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(&uid)) .block_on(self.drive.enumerate_folder_children_node_uids(&uid))
.map_err(|e| CoreError::from_api(&e, "enumerate shared children"))?; .map_err(|e| CoreError::from_api(&e, "enumerate shared children"))?;
if child_uids.is_empty() { if child_uids.is_empty() {
return Ok(Vec::new()); return Ok(Vec::new());
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes(&child_uids)) .block_on(self.drive.enumerate_nodes(&child_uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?; .map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes Ok(nodes
.into_iter() .into_iter()
@@ -509,7 +510,7 @@ impl Core {
parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?; parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?;
let node = self let node = self
.rt .rt
.block_on(self.client.get_node(&uid)) .block_on(self.drive.get_node(&uid))
.map_err(|e| CoreError::from_api(&e, "get shared node"))? .map_err(|e| CoreError::from_api(&e, "get shared node"))?
.ok_or_else(|| CoreError::not_found("shared node not found"))?; .ok_or_else(|| CoreError::not_found("shared node not found"))?;
if !node.is_file() { if !node.is_file() {
@@ -610,10 +611,10 @@ impl Core {
/// Proton products (not Drive-only). A remote round-trip; nothing here is /// Proton products (not Drive-only). A remote round-trip; nothing here is
/// cached. /// cached.
pub(crate) fn account_quota(&self) -> CoreResult<(i64, i64)> { pub(crate) fn account_quota(&self) -> CoreResult<(i64, i64)> {
let q = self let q = self.block_on_bounded(self.drive.quota()).map_err(|e| {
.rt self.lost_link(&e, "account quota");
.block_on(self.client.quota()) CoreError::from_api(&e, "account quota")
.map_err(|e| CoreError::from_api(&e, "account quota"))?; })?;
*self.quota.lock() = Some((std::time::Instant::now(), q.max_space, q.used_space)); *self.quota.lock() = Some((std::time::Instant::now(), q.max_space, q.used_space));
Ok((q.max_space, q.used_space)) Ok((q.max_space, q.used_space))
} }
@@ -632,11 +633,22 @@ impl Core {
{ {
return Some((max_space, used_space)); return Some((max_space, used_space));
} }
match self.account_quota() { let last = || self.quota.lock().map(|(_, max, used)| (max, used));
if !self.is_online()
|| self
.quota_refreshing
.swap(true, std::sync::atomic::Ordering::AcqRel)
{
return last();
}
let refreshed = self.account_quota();
self.quota_refreshing
.store(false, std::sync::atomic::Ordering::Release);
match refreshed {
Ok(q) => Some(q), Ok(q) => Some(q),
Err(error) => { Err(error) => {
debug!(%error, "account quota refresh failed; serving the last known figures"); debug!(%error, "account quota refresh failed; serving the last known figures");
self.quota.lock().map(|(_, max, used)| (max, used)) last()
} }
} }
} }
@@ -658,7 +670,7 @@ impl Core {
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes(&uids)) .block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?; .map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
let mut items = Vec::with_capacity(nodes.len()); let mut items = Vec::with_capacity(nodes.len());
for n in nodes { for n in nodes {
@@ -703,19 +715,10 @@ impl Core {
/// been seen through the mount (e.g. shared but not browsed to this session) — /// been seen through the mount (e.g. shared but not browsed to this session) —
/// the caller then leaves the path empty. /// the caller then leaves the path empty.
pub(crate) fn rel_path_for_uid(&self, uid: &NodeUid) -> Option<String> { pub(crate) fn rel_path_for_uid(&self, uid: &NodeUid) -> Option<String> {
// Searched across every mount: a node browsed through a sync folder is // Relative to whichever root the node sits under: a node browsed
// interned in that fork's inode space, so the primary state alone had no // through a sync folder is under that fork's root, not My Files
// entry for it and the caller silently showed an empty path // (`docs/BUGS.md` B74).
// (`docs/BUGS.md` B74). The path is relative to whichever mount holds walk_to_root(&self.state(), uid)
// it, which is the one the user reached it through.
let mut path = None;
self.for_each_state(|st| {
if path.is_some() {
return;
}
path = walk_to_root(st, uid);
});
path
} }
} }
@@ -760,14 +763,18 @@ impl StopSharingOutcome {
} }
} }
/// Walk one inode space from `uid` up to its root, building the path. `None` if /// Walk the tree from `uid` up to the root it sits under, building the path.
/// that mount has never interned the node, or if the chain breaks partway — /// `None` if the node has never been interned, or if the chain breaks partway —
/// a broken chain is a missing path, never a partial one. /// a broken chain is a missing path, never a partial one.
fn walk_to_root(st: &super::State, uid: &NodeUid) -> Option<String> { fn walk_to_root(st: &super::State, uid: &NodeUid) -> Option<String> {
let mut ino = *st.by_uid.get(uid)?; let mut ino = *st.by_uid.get(uid)?;
let mut parts = Vec::new(); let mut parts = Vec::new();
while ino != ROOT_INO { while !st.roots.contains(&ino) {
let entry = st.entries.get(&ino)?; let entry = st.entries.get(&ino)?;
// An orphan is its own parent without being a root.
if entry.parent == ino {
return None;
}
parts.push(entry.node.name.clone()); parts.push(entry.node.name.clone());
ino = entry.parent; ino = entry.parent;
} }
+1 -1
View File
@@ -6,7 +6,7 @@
//! invisible; in an in-process remount — which is how the daemon recovers from a //! invisible; in an in-process remount — which is how the daemon recovers from a
//! mount going away underneath it — it leaks a whole generation of threads that //! mount going away underneath it — it leaks a whole generation of threads that
//! keep writing to the database and calling the API on behalf of a mount that no //! keep writing to the database and calling the API on behalf of a mount that no
//! longer exists (bugs.md B44). //! longer exists (`docs/BUGS.md` B44).
//! //!
//! The primitive is deliberately small: a flag plus a condvar. Every loop that //! The primitive is deliberately small: a flag plus a condvar. Every loop that
//! would otherwise call `thread::sleep` calls [`Shutdown::sleep`] instead, which //! would otherwise call `thread::sleep` calls [`Shutdown::sleep`] instead, which
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+9
View File
@@ -0,0 +1,9 @@
//! The simulation tests of `docs/TESTING.md`: a fake Drive with
//! the faults the real one has, and the tools to drive daemons against it.
pub(crate) mod daemon;
pub(crate) mod fake_drive;
pub(crate) mod model;
pub(crate) mod rng;
pub(crate) mod run;
pub(crate) mod watchdog;
+445
View File
@@ -0,0 +1,445 @@
//! The reference file system of the simulation (`docs/TESTING.md`,
//! "Simulation runs", property 1): what each syscall a simulated client makes should
//! answer, and the tree it should leave, on a POSIX file system.
//!
//! A [`Model`] covers one client's own folder; paths are relative to it. The
//! same type is used for the trees read back from a mount and from Drive.
use std::collections::BTreeMap;
use super::rng::Rng;
/// A tree by path: `None` is a folder, `Some` a file's content.
pub(crate) type Tree = BTreeMap<String, Option<Vec<u8>>>;
/// One file system call, or the few a program makes in a row.
#[derive(Clone, Debug)]
pub(crate) enum FsOp {
/// `open(O_CREAT | O_TRUNC)`, write `data`, close.
Create {
path: String,
data: Vec<u8>,
},
/// Open an existing file, write `data` at `offset` (past the end leaves
/// a hole), close.
Write {
path: String,
offset: u64,
data: Vec<u8>,
},
/// Open an existing file and `ftruncate` it to `len`.
Truncate {
path: String,
len: u64,
},
Mkdir {
path: String,
},
Rename {
from: String,
to: String,
},
Unlink {
path: String,
},
/// Open a file, unlink it, append `data`, read it all back, close.
UnlinkOpen {
path: String,
data: Vec<u8>,
},
Rmdir {
path: String,
},
}
impl FsOp {
/// The paths the op names, for the log.
pub(crate) fn describe(&self) -> String {
match self {
FsOp::Create { path, data } => format!("create {path} ({} bytes)", data.len()),
FsOp::Write { path, offset, data } => {
format!("write {path} at {offset} ({} bytes)", data.len())
}
FsOp::Truncate { path, len } => format!("truncate {path} to {len}"),
FsOp::Mkdir { path } => format!("mkdir {path}"),
FsOp::Rename { from, to } => format!("rename {from} -> {to}"),
FsOp::Unlink { path } => format!("unlink {path}"),
FsOp::UnlinkOpen { path, data } => {
format!("unlink {path} while open, then append {} bytes", data.len())
}
FsOp::Rmdir { path } => format!("rmdir {path}"),
}
}
}
/// The tree one client should have, and what its syscalls should answer.
#[derive(Clone, Debug, Default)]
pub(crate) struct Model {
tree: Tree,
}
/// Names are drawn from small pools, so ops collide: a create over an existing
/// file, a rename over another, a mkdir of a name that is taken.
const FILE_NAMES: [&str; 6] = ["a.txt", "b.txt", "c.bin", "d", "e.md", "f.txt"];
const DIR_NAMES: [&str; 3] = ["x", "y", "z"];
impl Model {
pub(crate) fn tree(&self) -> &Tree {
&self.tree
}
pub(crate) fn content(&self, path: &str) -> Option<&Vec<u8>> {
self.tree.get(path)?.as_ref()
}
/// What `op` should answer on a POSIX file system, applying it on success.
pub(crate) fn apply(&mut self, op: &FsOp) -> Result<(), i32> {
match op {
FsOp::Create { path, data } => {
self.check_parent(path)?;
match self.tree.get(path) {
Some(None) => Err(libc::EISDIR),
_ => {
self.tree.insert(path.clone(), Some(data.clone()));
Ok(())
}
}
}
FsOp::Write { path, offset, data } => {
let content = self.file_mut(path)?;
// Writing nothing changes nothing, even past the end.
if data.is_empty() {
return Ok(());
}
let end = *offset as usize + data.len();
if content.len() < end {
content.resize(end, 0);
}
content[*offset as usize..end].copy_from_slice(data);
Ok(())
}
FsOp::Truncate { path, len } => {
self.file_mut(path)?.resize(*len as usize, 0);
Ok(())
}
FsOp::Mkdir { path } => {
self.check_parent(path)?;
if self.tree.contains_key(path) {
return Err(libc::EEXIST);
}
self.tree.insert(path.clone(), None);
Ok(())
}
FsOp::Rename { from, to } => self.rename(from, to),
FsOp::Unlink { path } | FsOp::UnlinkOpen { path, .. } => {
self.check_parent(path)?;
match self.tree.get(path) {
None => Err(libc::ENOENT),
Some(None) => Err(libc::EISDIR),
Some(Some(_)) => {
self.tree.remove(path);
Ok(())
}
}
}
FsOp::Rmdir { path } => {
self.check_parent(path)?;
match self.tree.get(path) {
None => Err(libc::ENOENT),
Some(Some(_)) => Err(libc::ENOTDIR),
Some(None) if self.has_children(path) => Err(libc::ENOTEMPTY),
Some(None) => {
self.tree.remove(path);
Ok(())
}
}
}
}
}
fn rename(&mut self, from: &str, to: &str) -> Result<(), i32> {
self.check_parent(from)?;
let Some(moving) = self.tree.get(from).cloned() else {
return Err(libc::ENOENT);
};
self.check_parent(to)?;
if from == to {
return Ok(());
}
if is_below(to, from) {
return Err(libc::EINVAL);
}
match (&moving, self.tree.get(to)) {
(None, Some(Some(_))) => return Err(libc::ENOTDIR),
(Some(_), Some(None)) => return Err(libc::EISDIR),
(None, Some(None)) if self.has_children(to) => return Err(libc::ENOTEMPTY),
_ => {}
}
let below: Vec<(String, Option<Vec<u8>>)> = self
.tree
.iter()
.filter(|(path, _)| is_below(path, from))
.map(|(path, entry)| (path.clone(), entry.clone()))
.collect();
self.tree.remove(from);
self.tree.insert(to.to_owned(), moving);
for (path, entry) in below {
self.tree.remove(&path);
self.tree
.insert(format!("{to}{}", &path[from.len()..]), entry);
}
Ok(())
}
fn file_mut(&mut self, path: &str) -> Result<&mut Vec<u8>, i32> {
self.check_parent(path)?;
match self.tree.get_mut(path) {
None => Err(libc::ENOENT),
Some(None) => Err(libc::EISDIR),
Some(Some(content)) => Ok(content),
}
}
fn check_parent(&self, path: &str) -> Result<(), i32> {
let mut at = String::new();
let mut names = path.split('/').peekable();
while let Some(name) = names.next() {
if names.peek().is_none() {
break;
}
if !at.is_empty() {
at.push('/');
}
at.push_str(name);
match self.tree.get(&at) {
None => return Err(libc::ENOENT),
Some(Some(_)) => return Err(libc::ENOTDIR),
Some(None) => {}
}
}
Ok(())
}
fn has_children(&self, path: &str) -> bool {
self.tree.keys().any(|other| is_below(other, path))
}
pub(crate) fn files(&self) -> Vec<&String> {
self.tree
.iter()
.filter(|(_, entry)| entry.is_some())
.map(|(path, _)| path)
.collect()
}
/// Folders, the client's own root (`""`) first.
fn dirs(&self) -> Vec<String> {
std::iter::once(String::new())
.chain(
self.tree
.iter()
.filter(|(_, entry)| entry.is_none())
.map(|(path, _)| path.clone()),
)
.collect()
}
/// A random op, mostly one that should succeed.
pub(crate) fn next_op(&self, rng: &mut Rng) -> FsOp {
let dirs = self.dirs();
let files = self.files();
let in_dir = |rng: &mut Rng, names: &[&str]| {
let parent = rng.pick(&dirs).expect("the root is always there");
join(parent, rng.pick(names).expect("names"))
};
let a_file = |rng: &mut Rng| rng.pick(&files).map(|path| (*path).clone());
let roll = rng.below(100);
let op = match roll {
0..30 => Some(FsOp::Create {
path: in_dir(rng, &FILE_NAMES),
data: data(rng),
}),
30..45 => a_file(rng).map(|path| {
let len = self.content(&path).map_or(0, Vec::len) as u64;
FsOp::Write {
offset: rng.below(len + 8),
path,
data: data(rng),
}
}),
45..50 => a_file(rng).map(|path| FsOp::Truncate {
len: rng.below(32),
path,
}),
50..60 => Some(FsOp::Mkdir {
path: in_dir(rng, &DIR_NAMES),
}),
// A rename in place, over another file as often as not.
60..72 => a_file(rng).map(|from| FsOp::Rename {
to: join(parent_of(&from), rng.pick(&FILE_NAMES).expect("names")),
from,
}),
// A move to another folder, renamed or not.
72..82 => a_file(rng).map(|from| {
let name = if rng.chance(0.5) {
name_of(&from)
} else {
rng.pick(&FILE_NAMES).expect("names")
};
let parent = rng.pick(&dirs).expect("the root is always there");
FsOp::Rename {
to: join(parent, name),
from,
}
}),
// A folder renamed or moved, with what is in it.
82..86 => {
let from = rng.pick(&dirs[1..]).cloned();
from.map(|from| FsOp::Rename {
to: in_dir(rng, &DIR_NAMES),
from,
})
}
86..94 => a_file(rng).map(|path| FsOp::Unlink { path }),
94..97 => a_file(rng).map(|path| FsOp::UnlinkOpen {
path,
data: data(rng),
}),
_ => rng
.pick(&dirs[1..])
.cloned()
.map(|path| FsOp::Rmdir { path }),
};
op.unwrap_or_else(|| FsOp::Create {
path: in_dir(rng, &FILE_NAMES),
data: data(rng),
})
}
}
/// Content for a write: short and recognisable, now and then empty.
pub(crate) fn data(rng: &mut Rng) -> Vec<u8> {
let len = if rng.chance(0.1) {
0
} else {
rng.between(1, 48)
};
let tag = rng.next_u64();
(0..len).map(|i| b'a' + ((tag + i) % 26) as u8).collect()
}
pub(crate) fn join(parent: &str, name: &str) -> String {
if parent.is_empty() {
name.to_owned()
} else {
format!("{parent}/{name}")
}
}
pub(crate) fn parent_of(path: &str) -> &str {
path.rsplit_once('/').map_or("", |(parent, _)| parent)
}
pub(crate) fn name_of(path: &str) -> &str {
path.rsplit_once('/').map_or(path, |(_, name)| name)
}
fn is_below(path: &str, ancestor: &str) -> bool {
path.len() > ancestor.len() + 1
&& path.starts_with(ancestor)
&& path.as_bytes()[ancestor.len()] == b'/'
}
#[cfg(test)]
mod tests {
use super::*;
fn create(path: &str, data: &[u8]) -> FsOp {
FsOp::Create {
path: path.into(),
data: data.to_vec(),
}
}
#[test]
fn a_rename_over_a_file_replaces_it_and_a_folder_moves_with_its_contents() {
let mut model = Model::default();
model.apply(&FsOp::Mkdir { path: "x".into() }).unwrap();
model.apply(&create("x/a", b"1")).unwrap();
model.apply(&create("b", b"2")).unwrap();
model
.apply(&FsOp::Rename {
from: "b".into(),
to: "x/a".into(),
})
.unwrap();
assert_eq!(model.content("x/a"), Some(&b"2".to_vec()));
model
.apply(&FsOp::Rename {
from: "x".into(),
to: "y".into(),
})
.unwrap();
assert_eq!(model.tree().keys().collect::<Vec<_>>(), vec!["y", "y/a"]);
}
#[test]
fn calls_fail_the_way_posix_says() {
let mut model = Model::default();
model.apply(&FsOp::Mkdir { path: "x".into() }).unwrap();
model.apply(&create("x/a", b"1")).unwrap();
assert_eq!(
model.apply(&FsOp::Mkdir { path: "x".into() }),
Err(libc::EEXIST)
);
assert_eq!(model.apply(&create("x", b"")), Err(libc::EISDIR));
assert_eq!(model.apply(&create("x/a/b", b"")), Err(libc::ENOTDIR));
assert_eq!(model.apply(&create("q/a", b"")), Err(libc::ENOENT));
assert_eq!(
model.apply(&FsOp::Rmdir { path: "x".into() }),
Err(libc::ENOTEMPTY)
);
assert_eq!(
model.apply(&FsOp::Unlink { path: "x".into() }),
Err(libc::EISDIR)
);
let into_itself = FsOp::Rename {
from: "x".into(),
to: "x/y".into(),
};
assert_eq!(model.apply(&into_itself), Err(libc::EINVAL));
}
#[test]
fn a_write_past_the_end_leaves_a_hole() {
let mut model = Model::default();
model.apply(&create("a", b"ab")).unwrap();
let write = FsOp::Write {
path: "a".into(),
offset: 4,
data: b"z".to_vec(),
};
model.apply(&write).unwrap();
assert_eq!(model.content("a"), Some(&b"ab\0\0z".to_vec()));
let nothing = FsOp::Write {
path: "a".into(),
offset: 9,
data: Vec::new(),
};
model.apply(&nothing).unwrap();
assert_eq!(model.content("a"), Some(&b"ab\0\0z".to_vec()));
}
#[test]
fn generated_ops_mostly_succeed() {
let mut rng = Rng::new(9);
let mut model = Model::default();
let ok = (0..500)
.filter(|_| {
let op = model.next_op(&mut rng);
model.apply(&op).is_ok()
})
.count();
assert!(ok > 350, "{ok} of 500 succeeded");
}
}
+79
View File
@@ -0,0 +1,79 @@
//! A small seeded random number generator, so a failing simulation is a seed
//! that can be replayed. SplitMix64: fast, good enough for choosing operations
//! and faults, and no dependency.
#[derive(Clone, Debug)]
pub(crate) struct Rng(u64);
impl Rng {
pub(crate) fn new(seed: u64) -> Self {
Self(seed)
}
pub(crate) fn next_u64(&mut self) -> u64 {
self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
let mut z = self.0;
z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
z ^ (z >> 31)
}
/// A value in `0..n`; `0` when `n` is `0`.
pub(crate) fn below(&mut self, n: u64) -> u64 {
if n == 0 { 0 } else { self.next_u64() % n }
}
/// A value in `lo..=hi`.
pub(crate) fn between(&mut self, lo: u64, hi: u64) -> u64 {
lo + self.below(hi.saturating_sub(lo) + 1)
}
/// True with probability `p`.
pub(crate) fn chance(&mut self, p: f64) -> bool {
p > 0.0 && (self.next_u64() >> 11) as f64 / (1u64 << 53) as f64 <= p
}
pub(crate) fn pick<'a, T>(&mut self, items: &'a [T]) -> Option<&'a T> {
items.get(self.below(items.len() as u64) as usize)
}
/// A child generator, so one consumer's draws do not shift another's.
pub(crate) fn fork(&mut self) -> Self {
Self::new(self.next_u64())
}
}
#[cfg(test)]
mod tests {
use super::Rng;
#[test]
fn a_seed_replays_the_same_sequence() {
let (mut a, mut b) = (Rng::new(7), Rng::new(7));
for _ in 0..100 {
assert_eq!(a.next_u64(), b.next_u64());
}
assert_ne!(Rng::new(7).next_u64(), Rng::new(8).next_u64());
}
#[test]
fn bounded_draws_stay_in_range() {
let mut rng = Rng::new(1);
for _ in 0..1000 {
assert!(rng.below(5) < 5);
assert!((3..=9).contains(&rng.between(3, 9)));
}
assert_eq!(rng.below(0), 0);
assert!(!rng.chance(0.0));
assert!(rng.chance(1.0));
assert!([1, 2, 3].contains(rng.pick(&[1, 2, 3]).unwrap()));
assert_eq!(rng.pick::<u8>(&[]), None);
}
#[test]
fn a_fork_draws_apart_from_its_parent() {
let mut parent = Rng::new(3);
let mut child = parent.fork();
assert_ne!(parent.next_u64(), child.next_u64());
}
}
+766
View File
@@ -0,0 +1,766 @@
//! The simulation runs described in `docs/TESTING.md`.
//!
//! A run starts one or more daemons on a shared [`FakeDrive`], each owning a
//! folder `c<n>` that only it writes, so there is one writer per file. A seed
//! draws a sequence of steps: syscalls through a client's mount, the link of a
//! client going down or up, a client restarting, a pause, a settle (every
//! queue must drain within a budget), or a file deleted while the drain holds
//! a write to it. Every syscall is checked against the
//! client's [`Model`] as it happens. After the sequence every link comes up
//! and the run checks:
//!
//! 1. **No loss** and **POSIX**: Drive's copy of each client's folder is what
//! its model says.
//! 2. **No false conflicts**: no conflict copy anywhere, and the drain never
//! takes a client's own delete for another device's.
//! 3. **Convergence**: each mount shows Drive's tree.
//! 4. **Liveness**: no syscall took longer than its budget, settles drained in
//! theirs, and every queue drained once the links were up. A call that
//! never returns is caught by the [`Watchdog`], which keeps the stacks.
//!
//! A restart here is a stop and a start: the next run finds what the last
//! one wrote down, but in-flight work is let finish rather than cut. A kill
//! at an arbitrary point needs the daemon in a process of its own.
//!
//! Any of these fails the seed. No open bug is tolerated: the matchers that
//! counted the ones still open instead (B125 to B134) went once a run on
//! 812eb3d passed every profile without them.
//!
//! The runs are ignored by default because they mount FUSE; run them with
//! `cargo test -p pdfs-fuse --lib sim::run -- --ignored --test-threads=1`.
//! `PDFS_SIM_SEEDS=n` sets how many seeds each runs, `PDFS_SIM_SEED=s` replays
//! one, and `RUST_LOG=pdfs_fuse=debug` with `--nocapture` shows the daemons'
//! logs.
use std::fs::OpenOptions;
use std::io::{self, Read, Seek, SeekFrom, Write};
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant};
use super::daemon::{Daemon, scratch, take_logged, wait_until};
use super::fake_drive::{Entry, FakeDrive, Faults};
use super::model::{FsOp, Model, Tree, data, join};
use super::rng::Rng;
use super::watchdog::Watchdog;
/// What a run looks like.
pub(crate) struct Profile {
pub(crate) name: &'static str,
pub(crate) clients: usize,
pub(crate) steps: usize,
pub(crate) faults: fn() -> Faults,
/// Whether links go down and come back during the sequence.
pub(crate) link_flaps: bool,
pub(crate) restarts: bool,
/// Whether files are deleted while the drain holds a write to them.
pub(crate) deletes_mid_upload: bool,
/// How long a settle may take; `None` leaves settles out.
pub(crate) settle_budget: Option<Duration>,
/// How long one syscall may take.
pub(crate) syscall_budget: Duration,
/// How long the queues may take to drain at the end.
pub(crate) drain_budget: Duration,
}
/// How long the mounts may take to show Drive's tree at the end: one event
/// poll (`POLL_INTERVAL`) and change.
const CONVERGE_BUDGET: Duration = Duration::from_secs(30);
/// How long the drain may take to pick up a write: its debounce, and then
/// some.
const PICKUP_BUDGET: Duration = Duration::from_secs(8);
/// Steps shown before a failure. All of them are written to `steps.log` in
/// the state the failure keeps.
const LOG_TAIL: usize = 40;
/// How long past its own budget a step, or the checks after the last one, may
/// run before the watchdog takes it for hung. Longer than a restart may wait
/// for the last run and the mount.
const STALL: Duration = Duration::from_secs(120);
struct Client {
daemon: Option<Daemon>,
model: Model,
online: bool,
}
impl Client {
fn daemon(&self) -> &Daemon {
self.daemon
.as_ref()
.expect("a client always has a daemon between steps")
}
fn root(&self, index: usize) -> PathBuf {
self.daemon().mountpoint.join(owned_folder(index))
}
}
struct Run {
seed: u64,
profile: &'static Profile,
drive: std::sync::Arc<FakeDrive>,
clients: Vec<Client>,
log: Vec<String>,
started: Instant,
/// What the daemons logged at `INFO` and above so far.
logged: Vec<String>,
watchdog: Watchdog,
}
/// Run `profile` on `seed`. It answers a report of what went wrong, if
/// anything did.
pub(crate) fn run(seed: u64, profile: &'static Profile) -> Result<(), String> {
let dir = scratch(&format!("{}-{seed}", profile.name));
let mut rng = Rng::new(seed);
let drive = FakeDrive::new();
let mut run = Run {
seed,
profile,
drive: drive.clone(),
clients: Vec::new(),
log: Vec::new(),
started: Instant::now(),
logged: Vec::new(),
watchdog: Watchdog::new(&dir),
};
let outcome = run.execute(&dir, &mut rng);
run.watchdog.stop();
// A hang is what failed, whatever the aborted call answered.
let outcome = match run.watchdog.fired() {
Some(hung) => Err(format!("liveness: {}", hung.what)),
None => outcome,
};
for client in &mut run.clients {
if let Some(daemon) = client.daemon.take() {
let _ = daemon.stop();
}
}
match outcome {
Ok(()) => {
let _ = std::fs::remove_dir_all(&dir);
Ok(())
}
Err(failure) => Err(run.report(&failure, &dir)),
}
}
impl Run {
fn execute(&mut self, dir: &Path, rng: &mut Rng) -> Result<(), String> {
for index in 0..self.profile.clients {
let client = self.drive.client(rng.next_u64(), (self.profile.faults)());
let daemon = Daemon::start(&dir.join(format!("client{index}")), client)
.map_err(|error| format!("client {index} did not mount: {error}"))?;
let folder = daemon.mountpoint.join(owned_folder(index));
std::fs::create_dir(&folder)
.map_err(|error| format!("client {index}: mkdir {}: {error}", folder.display()))?;
self.clients.push(Client {
daemon: Some(daemon),
model: Model::default(),
online: true,
});
}
for step in 0..self.profile.steps {
let _watching = self.watchdog.arm(self.step_limit(), format!("step {step}"));
self.step(rng)?;
}
let _watching = self.watchdog.arm(
self.profile.drain_budget + STALL,
"the checks after the last step",
);
self.finish()
}
/// How long a step may run before the watchdog takes it for hung.
fn step_limit(&self) -> Duration {
let budget = self.profile.settle_budget.unwrap_or_default();
budget.max(self.profile.syscall_budget) + STALL
}
fn step(&mut self, rng: &mut Rng) -> Result<(), String> {
let index = rng.below(self.clients.len() as u64) as usize;
let roll = rng.below(100);
match roll {
0..4 if self.profile.link_flaps => {
let client = &mut self.clients[index];
client.online = !client.online;
client.daemon().client.set_online(client.online);
let state = if client.online { "up" } else { "down" };
self.note(format!("client {index}: link {state}"));
}
4..6 if self.profile.restarts => {
self.note(format!("client {index}: restart"));
let daemon = self.clients[index].daemon.take().expect("daemon");
let daemon = daemon
.restart()
.map_err(|error| format!("client {index} did not come back: {error}"))?;
self.clients[index].daemon = Some(daemon);
}
6..12 => {
let ms = rng.below(400);
self.note(format!("pause {ms} ms"));
std::thread::sleep(Duration::from_millis(ms));
}
12..15 => {
if let Some(budget) = self.profile.settle_budget
&& self.clients.iter().all(|client| client.online)
{
self.settle(budget)?;
}
}
15..25 if self.profile.deletes_mid_upload => self.delete_mid_upload(index, rng)?,
_ => {
let op = self.clients[index].model.next_op(rng);
self.syscall(index, op)?;
}
}
Ok(())
}
/// Wait until every queue is empty and Drive holds what the models say.
fn settle(&mut self, budget: Duration) -> Result<(), String> {
self.note("settle".into());
let start = Instant::now();
// What the drain logs meanwhile is what explains a known bug's damage.
if !wait_until(budget, || {
self.logged.extend(take_logged());
self.landed()
}) {
return Err(format!(
"liveness: the queues did not drain within {budget:?}\n{}",
self.difference()
));
}
self.note(format!("settled in {:?}", start.elapsed()));
Ok(())
}
/// Write a file Drive holds and delete it once the drain has picked the
/// write up. The drain's read of the node is held until the delete has
/// landed and a read shows it, so the drain finds the file trashed while
/// it holds a write that is no longer queued (B105). Half the time the
/// file is made again meanwhile, which queues a write under the op id the
/// delete freed (B133).
fn delete_mid_upload(&mut self, index: usize, rng: &mut Rng) -> Result<(), String> {
let file = rng
.pick(&self.clients[index].model.files())
.map(|path| (*path).clone());
let (Some(path), Some(budget), true) = (
file,
self.profile.settle_budget,
self.clients.iter().all(|client| client.online),
) else {
let op = self.clients[index].model.next_op(rng);
return self.syscall(index, op);
};
self.settle(budget)?;
let Some(uid) = self.drive.lookup(&join(&owned_folder(index), &path)) else {
return Err(format!(
"no loss: client {index}'s {path} is not on Drive after a settle"
));
};
let held = self.clients[index].daemon().client.hold_next_read(&uid);
self.note(format!("client {index}: hold the next read of {path}"));
let mut bytes = data(rng);
bytes.push(b'.');
self.syscall(
index,
FsOp::Write {
path: path.clone(),
offset: 0,
data: bytes,
},
)?;
if !wait_until(PICKUP_BUDGET, || held.reached()) {
self.note(format!("client {index}: nothing read {path} back"));
return Ok(());
}
self.syscall(index, FsOp::Unlink { path: path.clone() })?;
if rng.chance(0.5) {
self.syscall(
index,
FsOp::Create {
path: path.clone(),
data: data(rng),
},
)?;
}
let faults = (self.profile.faults)();
let latency = Duration::from_millis(faults.latency_ms.1);
std::thread::sleep(faults.listing_lag + latency);
drop(held);
self.note(format!("client {index}: the read of {path} goes on"));
// Nothing else is queued before the drain has answered.
std::thread::sleep(2 * latency + Duration::from_millis(50));
Ok(())
}
fn syscall(&mut self, index: usize, op: FsOp) -> Result<(), String> {
let client = &mut self.clients[index];
let root = client.root(index);
let before = client.model.content(target(&op)).cloned();
let expected = client.model.apply(&op);
self.logged.extend(take_logged());
let start = Instant::now();
let actual = perform(&root, &op, before.as_deref());
let took = start.elapsed();
self.note(format!(
"client {index}: {} = {} in {took:?}",
op.describe(),
outcome(&actual)
));
let actual = actual?;
self.logged.extend(take_logged());
if actual != expected {
return Err(format!(
"posix: client {index}: {} answered {}, expected {}",
op.describe(),
errno(actual),
errno(expected)
));
}
if took > self.profile.syscall_budget {
return Err(format!(
"liveness: client {index}: {} took {took:?}, over {:?}",
op.describe(),
self.profile.syscall_budget
));
}
Ok(())
}
fn finish(&mut self) -> Result<(), String> {
for client in &mut self.clients {
client.online = true;
client.daemon().client.set_online(true);
}
self.note("every link up; waiting for the queues".into());
if !wait_until(self.profile.drain_budget, || {
self.logged.extend(take_logged());
self.landed()
}) {
return Err(format!(
"no loss: Drive does not hold what the clients wrote, {:?} after the links came up\n{}",
self.profile.drain_budget,
self.difference()
));
}
// Each client's folder is its own, so a file of it gone from Drive was
// deleted by the client.
if let Some(line) = self.logged.iter().find(|line| is_own_delete_kept(line)) {
return Err(format!(
"no false conflicts: the drain took a client's own delete for another device's: {line}"
));
}
let drive = self.drive_tree();
let conflicts: Vec<String> = drive
.into_keys()
.filter(|path| is_conflict_copy(path))
.collect();
if !conflicts.is_empty() {
return Err(format!("no false conflicts: {conflicts:?}"));
}
for index in 0..self.clients.len() {
let mut last = Tree::new();
let converged = wait_until(CONVERGE_BUDGET, || {
last = read_tree(&self.clients[index].daemon().mountpoint);
last == self.drive_tree()
});
if !converged {
return Err(format!(
"convergence: client {index}'s mount differs from Drive\n{}",
diff(&last, &self.drive_tree(), "mount", "Drive")
));
}
}
Ok(())
}
/// Whether every client's queue is empty and Drive holds what its model
/// says.
fn landed(&self) -> bool {
let drive = self.drive_tree();
self.clients.iter().enumerate().all(|(index, client)| {
client
.daemon()
.pending()
.is_ok_and(|items| items.is_empty())
&& subtree(&drive, &owned_folder(index)) == *client.model.tree()
})
}
fn difference(&self) -> String {
let drive = self.drive_tree();
let mut out = String::new();
for (index, client) in self.clients.iter().enumerate() {
let pending = client.daemon().pending().unwrap_or_default();
for item in pending {
out.push_str(&format!(
" client {index} still owes: {} {} (attempts {}, last error {:?})\n",
item.kind, item.path, item.attempts, item.last_error
));
}
out.push_str(&diff(
client.model.tree(),
&subtree(&drive, &owned_folder(index)),
&format!("client {index}'s model"),
"Drive",
));
}
out
}
fn drive_tree(&self) -> Tree {
self.drive
.tree()
.into_iter()
.map(|(path, entry)| {
let content = match entry {
Entry::Folder => None,
Entry::File(content) => Some(content.as_ref().clone()),
};
(path, content)
})
.collect()
}
fn note(&mut self, line: String) {
let line = format!("{:>8.3}s {line}", self.started.elapsed().as_secs_f64());
self.log.push(line);
}
fn report(&self, failure: &str, dir: &Path) -> String {
let _ = std::fs::write(dir.join("steps.log"), self.log.join("\n") + "\n");
let tail = &self.log[self.log.len().saturating_sub(LOG_TAIL)..];
format!(
"{} seed {} failed: {failure}\nreplay with PDFS_SIM_SEED={}; state kept in {}\nlast steps:\n{}",
self.profile.name,
self.seed,
self.seed,
dir.display(),
tail.join("\n")
)
}
}
/// Make `op` under `root`. The outer error is a check that failed on the
/// way (the content an open file reads back), the inner one the errno.
fn perform(root: &Path, op: &FsOp, before: Option<&[u8]>) -> Result<Result<(), i32>, String> {
let at = |path: &str| root.join(path);
let result: io::Result<()> = match op {
FsOp::Create { path, data } => std::fs::write(at(path), data),
FsOp::Write { path, offset, data } => OpenOptions::new()
.write(true)
.open(at(path))
.and_then(|mut file| {
file.seek(SeekFrom::Start(*offset))?;
file.write_all(data)
}),
FsOp::Truncate { path, len } => OpenOptions::new()
.write(true)
.open(at(path))
.and_then(|file| file.set_len(*len)),
FsOp::Mkdir { path } => std::fs::create_dir(at(path)),
FsOp::Rename { from, to } => std::fs::rename(at(from), at(to)),
FsOp::Unlink { path } => std::fs::remove_file(at(path)),
FsOp::Rmdir { path } => std::fs::remove_dir(at(path)),
FsOp::UnlinkOpen { path, data } => {
let mut read = Vec::new();
let result = OpenOptions::new()
.read(true)
.write(true)
.open(at(path))
.and_then(|mut file| {
std::fs::remove_file(at(path))?;
file.seek(SeekFrom::End(0))?;
file.write_all(data)?;
file.seek(SeekFrom::Start(0))?;
file.read_to_end(&mut read).map(|_| ())
});
if result.is_ok() {
let expected = [before.unwrap_or_default(), data].concat();
if read != expected {
return Err(format!(
"posix: {} read back {:?}, expected {:?}",
op.describe(),
String::from_utf8_lossy(&read),
String::from_utf8_lossy(&expected)
));
}
}
result
}
};
Ok(result.map_err(|error| error.raw_os_error().unwrap_or(libc::EIO)))
}
fn target(op: &FsOp) -> &str {
match op {
FsOp::Create { path, .. }
| FsOp::Write { path, .. }
| FsOp::Truncate { path, .. }
| FsOp::Mkdir { path }
| FsOp::Unlink { path }
| FsOp::UnlinkOpen { path, .. }
| FsOp::Rmdir { path } => path,
FsOp::Rename { from, .. } => from,
}
}
/// The tree a mount shows, read through it, without the folders it makes up
/// that are not on Drive.
fn read_tree(mountpoint: &Path) -> Tree {
fn walk(dir: &Path, prefix: &str, tree: &mut Tree) {
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
for entry in entries.flatten() {
let name = entry.file_name().to_string_lossy().into_owned();
if prefix.is_empty() && name == crate::r#virtual::SHARED_WITH_ME_BASE {
continue;
}
let path = super::model::join(prefix, &name);
if entry.file_type().is_ok_and(|kind| kind.is_dir()) {
tree.insert(path.clone(), None);
walk(&entry.path(), &path, tree);
} else {
tree.insert(path, std::fs::read(entry.path()).ok());
}
}
}
let mut tree = Tree::new();
walk(mountpoint, "", &mut tree);
tree
}
/// The part of `tree` below `folder`, relative to it.
fn subtree(tree: &Tree, folder: &str) -> Tree {
let prefix = format!("{folder}/");
tree.iter()
.filter_map(|(path, entry)| Some((path.strip_prefix(&prefix)?.to_owned(), entry.clone())))
.collect()
}
fn owned_folder(index: usize) -> String {
format!("c{index}")
}
/// Whether `line` is the drain keeping a queued write as a conflict copy
/// because Drive no longer holds the file, or holds it in the trash.
fn is_own_delete_kept(line: &str) -> bool {
line.starts_with("queued write conflicts; keeping a conflict copy")
&& (line.contains(" reason=\"the file was trashed remotely\"")
|| line.contains(" reason=\"the file no longer exists remotely\""))
}
fn is_conflict_copy(path: &str) -> bool {
path.split('/')
.any(|name| name.contains("(sync-conflict") || name.starts_with("recovered-"))
}
fn diff(left: &Tree, right: &Tree, left_name: &str, right_name: &str) -> String {
let show = |entry: &Option<Vec<u8>>| match entry {
None => "folder".to_owned(),
Some(content) => format!("{:?}", String::from_utf8_lossy(content)),
};
let mut out = String::new();
for (path, entry) in left {
match right.get(path) {
None => out.push_str(&format!(
" {path}: only in {left_name} ({})\n",
show(entry)
)),
Some(other) if other != entry => out.push_str(&format!(
" {path}: {left_name} has {}, {right_name} has {}\n",
show(entry),
show(other)
)),
Some(_) => {}
}
}
for (path, entry) in right {
if !left.contains_key(path) {
out.push_str(&format!(
" {path}: only in {right_name} ({})\n",
show(entry)
));
}
}
out
}
fn errno(outcome: Result<(), i32>) -> String {
match outcome {
Ok(()) => "ok".into(),
Err(code) => io::Error::from_raw_os_error(code).to_string(),
}
}
fn outcome(actual: &Result<Result<(), i32>, String>) -> String {
match actual {
Ok(result) => errno(*result),
Err(check) => check.clone(),
}
}
/// Run `profile` over the seeds the environment asks for: `PDFS_SIM_SEED`
/// alone, or `PDFS_SIM_SEEDS` of them (default `default_seeds`) from 1.
pub(crate) fn run_seeds(profile: &'static Profile, default_seeds: u64) {
let seeds: Vec<u64> = match std::env::var("PDFS_SIM_SEED") {
Ok(seed) => vec![seed.parse().expect("PDFS_SIM_SEED is a number")],
Err(_) => {
let count = std::env::var("PDFS_SIM_SEEDS")
.ok()
.and_then(|count| count.parse().ok())
.unwrap_or(default_seeds);
(1..=count).collect()
}
};
let failures: Vec<String> = seeds
.into_iter()
.filter_map(|seed| run(seed, profile).err())
.collect();
assert!(failures.is_empty(), "\n{}", failures.join("\n\n"));
}
#[cfg(test)]
mod tests {
use super::*;
static ONE_CLIENT_LAN: Profile = Profile {
name: "one-client-lan",
clients: 1,
steps: 80,
faults: Faults::lan,
link_flaps: false,
restarts: false,
deletes_mid_upload: false,
settle_budget: Some(Duration::from_secs(4)),
syscall_budget: Duration::from_secs(2),
drain_budget: Duration::from_secs(20),
};
static ONE_CLIENT_DELETES: Profile = Profile {
name: "one-client-deletes",
clients: 1,
steps: 30,
faults: Faults::lan,
link_flaps: false,
restarts: false,
deletes_mid_upload: true,
settle_budget: Some(Duration::from_secs(4)),
syscall_budget: Duration::from_secs(2),
drain_budget: Duration::from_secs(20),
};
static ONE_CLIENT_ECHOES: Profile = Profile {
name: "one-client-echoes",
clients: 1,
steps: 60,
faults: Faults::echoes,
link_flaps: false,
restarts: false,
deletes_mid_upload: false,
settle_budget: Some(Duration::from_secs(6)),
syscall_budget: Duration::from_secs(2),
drain_budget: Duration::from_secs(20),
};
static ONE_CLIENT_FLAKY: Profile = Profile {
name: "one-client-flaky",
clients: 1,
steps: 80,
faults: Faults::wifi,
link_flaps: true,
restarts: true,
deletes_mid_upload: false,
settle_budget: None,
syscall_budget: Duration::from_secs(10),
drain_budget: Duration::from_secs(120),
};
/// Wi-Fi's latency, lag and lost replies without the outages: the queue
/// has to drain within a settle, not merely by the end.
static ONE_CLIENT_WIFI: Profile = Profile {
name: "one-client-wifi",
clients: 1,
steps: 80,
faults: Faults::wifi,
link_flaps: false,
restarts: false,
deletes_mid_upload: false,
settle_budget: Some(Duration::from_secs(30)),
syscall_budget: Duration::from_secs(10),
drain_budget: Duration::from_secs(120),
};
static THREE_CLIENTS: Profile = Profile {
name: "three-clients",
clients: 3,
steps: 90,
faults: Faults::lan,
link_flaps: true,
restarts: true,
deletes_mid_upload: false,
settle_budget: Some(Duration::from_secs(4)),
syscall_budget: Duration::from_secs(2),
drain_budget: Duration::from_secs(60),
};
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse --lib sim::run -- --ignored`"]
fn one_client_on_a_good_link() {
run_seeds(&ONE_CLIENT_LAN, 12);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse --lib sim::run -- --ignored`"]
fn one_client_deleting_files_as_they_upload() {
run_seeds(&ONE_CLIENT_DELETES, 3);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse --lib sim::run -- --ignored`"]
fn one_client_with_stale_hashes_and_reordered_echoes() {
run_seeds(&ONE_CLIENT_ECHOES, 6);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse --lib sim::run -- --ignored`"]
fn one_client_on_a_flaky_link() {
run_seeds(&ONE_CLIENT_FLAKY, 4);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse --lib sim::run -- --ignored`"]
fn one_client_on_a_slow_link() {
run_seeds(&ONE_CLIENT_WIFI, 4);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse --lib sim::run -- --ignored`"]
fn three_clients_one_writer_each() {
run_seeds(&THREE_CLIENTS, 3);
}
#[test]
fn conflict_copies_are_recognised_by_name() {
assert!(is_conflict_copy("c0/x/a (sync-conflict 1790000000).txt"));
assert!(is_conflict_copy("recovered-XrPUSW5W"));
assert!(!is_conflict_copy("c0/x/a.txt"));
}
#[test]
fn a_write_kept_because_its_file_is_gone_is_recognised() {
let kept = "queued write conflicts; keeping a conflict copy uid=simvol~L10 \
name=\"f.txt\" reason=\"the file was trashed remotely\"";
assert!(is_own_delete_kept(kept));
assert!(is_own_delete_kept(
&kept.replace("was trashed", "no longer exists")
));
assert!(!is_own_delete_kept(
&kept.replace("the file was trashed remotely", "revision changed")
));
}
}
+322
View File
@@ -0,0 +1,322 @@
//! Turns a call through a mount that never returns into a failed seed.
//!
//! A deadlock in a daemon leaves the run's own thread blocked in the kernel,
//! where no budget check is reached, and a CI job then waits for its timeout.
//! The watchdog runs beside the run. When what it was armed for outlives its
//! limit, it aborts the FUSE connections of the run's mounts, so the blocked
//! call fails and the run reports the hang, and writes where every thread was
//! to `stacks.txt`.
//!
//! The stacks are taken in the process: no tracer can stop a thread that
//! waits on a FUSE request, so `gdb` and `eu-stack` wait as long as the hang.
//! Each thread is sent [`stack_signal`] and captures its own backtrace in the
//! handler. A thread in the kernel takes the signal only once it leaves, so
//! the mounts are aborted after the others have answered, and its backtrace
//! is the call it was blocked in. Its kernel wait channel is recorded first.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread::JoinHandle;
use std::time::{Duration, Instant};
use parking_lot::Mutex;
use super::daemon::wait_until;
/// How long the threads that are not blocked in the kernel have to answer,
/// and the blocked ones once the mounts are aborted.
const ANSWER: Duration = Duration::from_secs(5);
pub(crate) struct Watchdog {
shared: Arc<Shared>,
thread: Option<JoinHandle<()>>,
}
struct Shared {
/// The run's directory: its mounts are below it, and the stacks go in it.
dir: PathBuf,
armed: Mutex<Option<Armed>>,
/// What hung, once something has.
fired: Mutex<Option<Hung>>,
stop: AtomicBool,
}
struct Armed {
what: String,
limit: Duration,
since: Instant,
}
/// What hung, and where every thread was then.
#[derive(Clone, Debug, PartialEq)]
pub(crate) struct Hung {
pub(crate) what: String,
pub(crate) stacks: String,
}
/// Disarms the watchdog when dropped.
pub(crate) struct Watching(Arc<Shared>);
impl Watchdog {
pub(crate) fn new(dir: &Path) -> Self {
let shared = Arc::new(Shared {
dir: dir.to_path_buf(),
armed: Mutex::new(None),
fired: Mutex::new(None),
stop: AtomicBool::new(false),
});
let thread = {
let shared = shared.clone();
std::thread::Builder::new()
.name("sim-watchdog".into())
.spawn(move || watch(&shared))
.expect("spawn the simulation watchdog")
};
Self {
shared,
thread: Some(thread),
}
}
/// Watch `what` until the guard is dropped, and abort the mounts if it is
/// still going after `limit`.
pub(crate) fn arm(&self, limit: Duration, what: impl Into<String>) -> Watching {
*self.shared.armed.lock() = Some(Armed {
what: what.into(),
limit,
since: Instant::now(),
});
Watching(self.shared.clone())
}
/// What hung, if something did. Call [`Watchdog::stop`] first, or the
/// stacks may still be on their way.
pub(crate) fn fired(&self) -> Option<Hung> {
self.shared.fired.lock().clone()
}
/// Stop watching, once the stacks of a hang that was caught are taken.
pub(crate) fn stop(&mut self) {
self.shared.stop.store(true, Ordering::Relaxed);
if let Some(thread) = self.thread.take() {
let _ = thread.join();
}
}
}
impl Drop for Watching {
fn drop(&mut self) {
*self.0.armed.lock() = None;
}
}
impl Drop for Watchdog {
fn drop(&mut self) {
self.stop();
}
}
fn watch(shared: &Shared) {
while !shared.stop.load(Ordering::Relaxed) {
std::thread::sleep(Duration::from_millis(200));
let hung = shared
.armed
.lock()
.as_ref()
.filter(|armed| armed.since.elapsed() > armed.limit)
.map(|armed| format!("{} has not returned after {:?}", armed.what, armed.limit));
let Some(hung) = hung else { continue };
*shared.armed.lock() = None;
let path = shared.dir.join("stacks.txt");
// Before anything is let go: the run reports the hang, not whatever
// the aborted call answers.
*shared.fired.lock() = Some(Hung {
what: format!("{hung}; the stacks are in {}", path.display()),
stacks: String::new(),
});
let stacks = take_stacks(|| abort_mounts_below(&shared.dir));
let what = match std::fs::write(&path, &stacks) {
Ok(()) => format!("{hung}; the stacks are in {}", path.display()),
Err(error) => format!("{hung}; the stacks could not be kept: {error}"),
};
*shared.fired.lock() = Some(Hung { what, stacks });
}
}
/// One thread of this process as the kernel sees it.
struct Task {
tid: i32,
name: String,
/// `D` while it waits in the kernel uninterruptibly.
state: String,
wchan: String,
}
/// Where every other thread of this process is, with `release` called once
/// those not blocked in the kernel have answered.
fn take_stacks(release: impl FnOnce()) -> String {
let own = gettid();
let tasks = tasks();
STACKS.lock().clear();
install_stack_handler();
for task in &tasks {
if task.tid != own {
// SAFETY: tgkill only sends a signal, whose handler is installed.
unsafe {
libc::syscall(
libc::SYS_tgkill,
std::process::id() as i32,
task.tid,
stack_signal(),
)
};
}
}
let answered = |blocked: bool| {
let stacks = STACKS.lock();
tasks
.iter()
.filter(|task| task.tid != own && (blocked || task.state != "D"))
.all(|task| stacks.contains_key(&task.tid))
};
wait_until(ANSWER, || answered(false));
release();
wait_until(ANSWER, || answered(true));
let stacks = std::mem::take(&mut *STACKS.lock());
let mut out = String::new();
for task in tasks.iter().filter(|task| task.tid != own) {
out.push_str(&format!(
"thread {} {:?}, state {}, waiting in {}\n",
task.tid, task.name, task.state, task.wchan
));
match stacks.get(&task.tid) {
Some(stack) => out.push_str(&format!("{stack}\n")),
None => out.push_str(" (no answer)\n\n"),
}
}
out
}
fn tasks() -> Vec<Task> {
let Ok(entries) = std::fs::read_dir("/proc/self/task") else {
return Vec::new();
};
let read = |dir: &Path, file: &str| {
std::fs::read_to_string(dir.join(file))
.unwrap_or_default()
.trim()
.to_owned()
};
entries
.flatten()
.filter_map(|entry| {
let tid = entry.file_name().to_str()?.parse().ok()?;
let dir = entry.path();
// The state follows the name in parentheses, which may hold spaces.
let stat = read(&dir, "stat");
let state = stat.rsplit_once(") ")?.1.split(' ').next()?.to_owned();
Some(Task {
tid,
name: read(&dir, "comm"),
state,
wchan: read(&dir, "wchan"),
})
})
.collect()
}
/// The backtraces the threads captured, by thread id.
static STACKS: Mutex<BTreeMap<i32, std::backtrace::Backtrace>> = Mutex::new(BTreeMap::new());
/// The signal that asks a thread for its stack: one of the real-time signals
/// that nothing in the daemon uses.
fn stack_signal() -> i32 {
libc::SIGRTMIN() + 7
}
fn install_stack_handler() {
static INSTALLED: std::sync::Once = std::sync::Once::new();
extern "C" fn handler(_: i32) {
// Not async-signal-safe: the capture allocates. The process is being
// taken down for a hang anyway, and a thread interrupted inside the
// allocator only fails to answer.
let stack = std::backtrace::Backtrace::force_capture();
STACKS.lock().insert(gettid(), stack);
}
INSTALLED.call_once(|| {
// SAFETY: a zeroed sigaction with a handler and SA_RESTART is valid,
// and the handler only touches `STACKS`.
unsafe {
let mut action: libc::sigaction = std::mem::zeroed();
action.sa_sigaction = handler as extern "C" fn(i32) as usize;
action.sa_flags = libc::SA_RESTART;
libc::sigemptyset(&mut action.sa_mask);
libc::sigaction(stack_signal(), &action, std::ptr::null_mut());
}
});
}
fn gettid() -> i32 {
// SAFETY: gettid has no arguments and cannot fail.
unsafe { libc::gettid() }
}
/// Abort the FUSE connection of every mount below `dir`. Its connection is
/// named by the mount's device minor, and the mount's owner may abort it.
fn abort_mounts_below(dir: &Path) {
let Ok(mountinfo) = std::fs::read_to_string("/proc/self/mountinfo") else {
return;
};
for line in mountinfo.lines() {
let fields: Vec<&str> = line.split(' ').collect();
let (Some(device), Some(mountpoint)) = (fields.get(2), fields.get(4)) else {
continue;
};
if !Path::new(mountpoint).starts_with(dir) {
continue;
}
if let Some((_, minor)) = device.split_once(':') {
let _ = std::fs::write(format!("/sys/fs/fuse/connections/{minor}/abort"), "1");
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::sim::daemon::scratch;
#[test]
#[ignore = "signals every thread of the test process: run with `cargo test -p pdfs-fuse sim:: -- --ignored`"]
fn a_call_past_its_limit_is_reported_with_the_stacks() {
let dir = scratch("watchdog");
let mut watchdog = Watchdog::new(&dir);
{
let _watching = watchdog.arm(Duration::from_millis(100), "a nap");
std::thread::sleep(Duration::from_secs(1));
}
watchdog.stop();
let hung = watchdog.fired().expect("the watchdog fired");
assert!(
hung.what.starts_with("a nap has not returned after"),
"{hung:?}"
);
assert!(hung.stacks.contains("a_call_past_its_limit"), "{hung:?}");
let kept = std::fs::read_to_string(dir.join("stacks.txt")).unwrap_or_default();
assert_eq!(kept, hung.stacks);
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn a_call_within_its_limit_is_not() {
let dir = scratch("watchdog-quiet");
let watchdog = Watchdog::new(&dir);
drop(watchdog.arm(Duration::from_millis(100), "a quick call"));
std::thread::sleep(Duration::from_millis(500));
assert_eq!(watchdog.fired(), None);
drop(watchdog);
let _ = std::fs::remove_dir_all(&dir);
}
}
+461 -42
View File
@@ -128,7 +128,62 @@ pub(crate) struct WriteHandle {
pub(crate) open_count: usize, pub(crate) open_count: usize,
} }
/// A released write whose upload has not happened yet (offline.md Phase 3). impl WriteHandle {
/// A handle on a file `create` just made: an empty base, so everything
/// written through it is authored.
///
/// The base is `node` as it actually exists — the empty file the server just
/// minted, or a queued create's placeholder — never the local clock or a
/// blank. `queue_revision` turns it into `StagedWrite::based_on` and the
/// drain compares the remote against that. A `now_secs()` mtime differs from
/// the server's stamp by the create round-trip. A missing revision id drops
/// the check to `(mtime, size)`, and moving or renaming the file advances
/// its mtime on the server. Either way the first write to a brand-new file
/// lands as a conflict copy of its own empty create (`docs/BUGS.md` B113).
pub(crate) fn created(ino: u64, node: &Node, file: File, path: PathBuf) -> Self {
Self {
ino,
uid: node.uid.clone(),
file: Arc::new(file),
path,
written: Intervals::default(),
len: 0,
base_size: 0,
base_mtime: node.modification_time,
base_revision_id: crate::node_revision_id(node),
dirty: false,
open_count: 0,
}
}
/// Write over `node` from now on: a revision this daemon has just put on the
/// server under the handle, by sealing an upload or by landing the queued
/// create the handle was opened on. The base the handle opened over is gone,
/// and keeping it would make the drain read our own revision as another
/// device's edit.
pub(crate) fn rebase_onto(&mut self, node: &Node) {
self.base_mtime = node.modification_time;
self.base_size = crate::node_size(node);
self.base_revision_id = crate::node_revision_id(node);
}
/// Account for `[offset, end)` having been written through this handle.
///
/// A write past the end leaves a hole that reads as zeros, exactly like a
/// grow through `setattr`, so the hole is authored too. Left unauthored, a
/// hole below the base size was refilled from the base at commit: a file
/// shrunk and then written past its new end got its old bytes back.
pub(crate) fn record_write(&mut self, offset: u64, end: u64) {
if offset > self.len {
self.written.add(self.len, offset);
}
self.written.add(offset, end);
self.len = self.len.max(end);
self.dirty = true;
}
}
/// A released write whose upload has not happened yet.
/// ///
/// The bytes live in the content cache's staging dir and the intent lives in the /// The bytes live in the content cache's staging dir and the intent lives in the
/// `pending_op` table; this pairs them in memory so a read can be served without /// `pending_op` table; this pairs them in memory so a read can be served without
@@ -146,8 +201,29 @@ pub(crate) struct PendingRevision {
/// time so this module stays free of `fuser`. /// time so this module stays free of `fuser`.
pub(crate) type DirListing = Vec<(u64, bool, String)>; pub(crate) type DirListing = Vec<(u64, bool, String)>;
/// Where [`State::next_ino`] starts: far above any local id a database hands
/// out, so a fallback inode never names a node that has a row.
pub(crate) const FALLBACK_INO: u64 = 1 << 48;
/// The inode of the node with local id `lid` ([`Db::lids_for`]). One above the
/// lid, so no node takes [`crate::ROOT_INO`], which each session uses for its
/// own root. The same in every mount and after a restart.
pub(crate) fn lid_ino(lid: i64) -> u64 {
lid.unsigned_abs() + crate::ROOT_INO
}
/// The local id of the node at inode `ino`, the inverse of [`lid_ino`]:
/// `None` for the root and a fallback inode.
pub(crate) fn ino_lid(ino: u64) -> Option<i64> {
(ino > crate::ROOT_INO && ino < FALLBACK_INO).then(|| (ino - crate::ROOT_INO) as i64)
}
/// Mutable inode bookkeeping, guarded by a mutex because fuser drives the /// Mutable inode bookkeeping, guarded by a mutex because fuser drives the
/// `Filesystem` trait through `&self`. /// `Filesystem` trait through `&self`.
///
/// One tree for every mount of the daemon. Each FUSE session sees it from its
/// own root ([`State::roots`]) and calls that root inode 1; `ProtonFs`
/// translates at the kernel boundary.
pub(crate) struct State { pub(crate) struct State {
/// inode -> node metadata. /// inode -> node metadata.
pub(crate) entries: HashMap<u64, Entry>, pub(crate) entries: HashMap<u64, Entry>,
@@ -156,6 +232,12 @@ pub(crate) struct State {
/// Cached directory listings: parent inode -> child inodes. Presence of a /// Cached directory listings: parent inode -> child inodes. Presence of a
/// key means the directory has been enumerated. /// key means the directory has been enumerated.
pub(crate) children: HashMap<u64, Vec<u64>>, pub(crate) children: HashMap<u64, Vec<u64>>,
/// The inodes some session has been rooted at ([`State::install_root`]).
/// A root is its own parent. Kept after its session unmounts: the node is
/// still one of ours.
pub(crate) roots: HashSet<u64>,
/// The next inode for a node the database could not give a local id, from
/// a range lids never reach ([`FALLBACK_INO`]).
pub(crate) next_ino: u64, pub(crate) next_ino: u64,
/// Shared write state keyed by inode. Concurrent writers share the scratch file. /// Shared write state keyed by inode. Concurrent writers share the scratch file.
pub(crate) active_writes: HashMap<u64, WriteHandle>, pub(crate) active_writes: HashMap<u64, WriteHandle>,
@@ -171,13 +253,13 @@ pub(crate) struct State {
/// two pages shifted every later entry, and the caller silently skipped or /// two pages shifted every later entry, and the caller silently skipped or
/// repeated whatever crossed the page boundary. Each entry is /// repeated whatever crossed the page boundary. Each entry is
/// `(ino, is_dir, name)` — the kernel type is mapped at reply time so this /// `(ino, is_dir, name)` — the kernel type is mapped at reply time so this
/// module stays free of `fuser` (bugs.md B43). /// module stays free of `fuser` (`docs/BUGS.md` B43).
pub(crate) dir_snapshots: HashMap<u64, Arc<DirListing>>, pub(crate) dir_snapshots: HashMap<u64, Arc<DirListing>>,
/// Resident inode attributes whose effective access changed during an /// Resident inode attributes whose effective access changed during an
/// intern/root refresh. Core drains this set and notifies the matching /// intern/root refresh. Core drains this set and notifies the matching
/// kernel only after releasing the State lock. /// kernel only after releasing the State lock.
pub(crate) access_changes: HashSet<u64>, pub(crate) access_changes: HashSet<u64>,
/// Persisted shared-root authority, loaded once per inode space and updated /// Persisted shared-root authority, loaded once and updated
/// on every explicit role/tombstone change. Descendant inheritance reads /// on every explicit role/tombstone change. Descendant inheritance reads
/// this map instead of issuing one SQLite query per interned inode. /// this map instead of issuing one SQLite query per interned inode.
pub(crate) share_access: HashMap<NodeUid, Access>, pub(crate) share_access: HashMap<NodeUid, Access>,
@@ -245,15 +327,6 @@ pub(crate) struct StateGuard<'a> {
db: &'a Arc<Db>, db: &'a Arc<Db>,
} }
/// Take a mount's inode lock through a [`StateGuard`], for the loops that walk
/// every live mount rather than this `Core`'s own.
pub(crate) fn lock_state<'a>(
state: &'a parking_lot::Mutex<State>,
db: &'a Arc<Db>,
) -> StateGuard<'a> {
StateGuard::new(state.lock(), db)
}
impl<'a> StateGuard<'a> { impl<'a> StateGuard<'a> {
pub(crate) fn new(guard: parking_lot::MutexGuard<'a, State>, db: &'a Arc<Db>) -> Self { pub(crate) fn new(guard: parking_lot::MutexGuard<'a, State>, db: &'a Arc<Db>) -> Self {
Self { Self {
@@ -294,9 +367,6 @@ impl Drop for StateGuard<'_> {
impl State { impl State {
/// A fresh, empty inode space over `db`. /// A fresh, empty inode space over `db`.
/// ///
/// `next_ino` differs by caller: a real mount hands out [`crate::ROOT_INO`]
/// to its root and starts allocating above it, while a bare state under test
/// has no root and starts at 1.
/// Apply the write-throughs this state owes SQLite, here and now. /// Apply the write-throughs this state owes SQLite, here and now.
/// ///
/// Production code never calls this: [`StateGuard`] releases the inode lock /// Production code never calls this: [`StateGuard`] releases the inode lock
@@ -323,12 +393,15 @@ impl State {
self.db.clone() self.db.clone()
} }
pub(crate) fn new(db: Arc<Db>, share_access: HashMap<NodeUid, Access>, next_ino: u64) -> Self { /// A fresh, empty tree over `db`. A node's inode comes from its local id
/// ([`lid_ino`]).
pub(crate) fn new(db: Arc<Db>, share_access: HashMap<NodeUid, Access>) -> Self {
Self { Self {
entries: HashMap::new(), entries: HashMap::new(),
by_uid: HashMap::new(), by_uid: HashMap::new(),
children: HashMap::new(), children: HashMap::new(),
next_ino, roots: HashSet::new(),
next_ino: FALLBACK_INO,
active_writes: HashMap::new(), active_writes: HashMap::new(),
handles: HashMap::new(), handles: HashMap::new(),
next_fh: 1, next_fh: 1,
@@ -339,18 +412,55 @@ impl State {
outbox: Vec::new(), outbox: Vec::new(),
} }
} }
/// Whether `uid` names an owned node currently reachable from this mount's /// Make `root` the root of a session and return its inode, which comes
/// root. Resident entries can outlive their dentries for open-handle and /// from its local id like any other node's.
/// access-revocation semantics, so `by_uid` membership alone is not proof ///
/// that a control-socket operation may address the node. /// The root is always owned. It must never be classified as a shared root,
pub(crate) fn owns_visible_uid(&self, uid: &NodeUid) -> bool { /// even when the API returns membership data (which happens when the owner
let Some(root) = self.entries.get(&crate::ROOT_INO) else { /// shares the folder with someone else), so a stale share row a downgrade
return false; /// may have left for it is dropped. A node already in the tree keeps its
}; /// inode and its lookups, and stops naming a parent: a session's `..` at
if uid.volume_id != root.uid.volume_id { /// its root is the root.
return false; pub(crate) fn install_root(&mut self, root: Node) -> u64 {
if let Err(e) = self.db.upsert_node(&root) {
warn!(uid = %root.uid, error = %e, "db upsert root failed");
} }
self.share_access.remove(&root.uid);
let _ = self.db.delete_share_access(&root.uid);
let ino = match self.by_uid.get(&root.uid) {
Some(&ino) => ino,
None => self.new_inos(&[&root])[&root.uid],
};
if let Some(old) = self.entries.get(&ino).map(|e| e.parent)
&& old != ino
&& let Some(kids) = self.children.get_mut(&old)
{
kids.retain(|&k| k != ino);
}
self.by_uid.insert(root.uid.clone(), ino);
let lookup_count = self.entries.get(&ino).map_or(1, |e| e.lookup_count.max(1));
self.entries.insert(
ino,
Entry {
uid: root.uid.clone(),
parent: ino,
node: root,
access: Access::Owner,
lookup_count,
open_count: 0,
unlinked: false,
},
);
self.roots.insert(ino);
ino
}
/// Whether `uid` names an owned node currently reachable from one of
/// `roots`, the roots of the sessions that are mounted. Resident entries
/// can outlive their dentries for open-handle and access-revocation
/// semantics, so `by_uid` membership alone is not proof that a
/// control-socket operation may address the node.
pub(crate) fn owns_visible_uid(&self, uid: &NodeUid, roots: &[u64]) -> bool {
let Some(&target) = self.by_uid.get(uid) else { let Some(&target) = self.by_uid.get(uid) else {
return false; return false;
}; };
@@ -363,7 +473,9 @@ impl State {
let Some(entry) = self.entries.get(&current) else { let Some(entry) = self.entries.get(&current) else {
return false; return false;
}; };
if entry.uid.volume_id != root.uid.volume_id // The node's root scopes it to its own volume, excluding foreign
// shared-with-me residents from uid-addressed sharing.
if entry.uid.volume_id != uid.volume_id
|| entry.unlinked || entry.unlinked
|| entry.node.trashed || entry.node.trashed
|| entry.uid != entry.node.uid || entry.uid != entry.node.uid
@@ -371,7 +483,7 @@ impl State {
{ {
return false; return false;
} }
if current == crate::ROOT_INO { if roots.contains(&current) {
return true; return true;
} }
@@ -413,15 +525,17 @@ impl State {
self.access_changes.drain().collect() self.access_changes.drain().collect()
} }
/// Whether `uid` lives on the volume this mount is rooted in. /// Whether `uid` lives on a volume some mount is rooted in.
/// ///
/// Used to decide how a node with no resolvable parent is classified: /// Used to decide how a node with no resolvable parent is classified:
/// our own volume fails open (it is owned content), a foreign volume /// our own volume fails open (it is owned content), a foreign volume
/// fails closed (it can only be there because of a share). /// fails closed (it can only be there because of a share).
pub(crate) fn is_own_volume(&self, uid: &NodeUid) -> bool { pub(crate) fn is_own_volume(&self, uid: &NodeUid) -> bool {
self.entries self.roots.iter().any(|ino| {
.get(&crate::ROOT_INO) self.entries
.is_some_and(|root| root.uid.volume_id == uid.volume_id) .get(ino)
.is_some_and(|root| root.uid.volume_id == uid.volume_id)
})
} }
fn stored_share_access(&self, uid: &NodeUid) -> Option<Access> { fn stored_share_access(&self, uid: &NodeUid) -> Option<Access> {
@@ -770,11 +884,59 @@ impl State {
} }
} }
pub(crate) fn intern(&mut self, parent: u64, node: Node) -> u64 { /// Write `node` through as it is, in order with the tree's other changes.
pub(crate) fn write_through(&mut self, node: Node) {
self.outbox.push(DbWrite::Upsert(vec![node]));
}
pub(crate) fn intern(&mut self, parent: u64, mut node: Node) -> u64 {
self.keep_revision_mtime(&mut node);
self.outbox.push(DbWrite::Upsert(vec![node.clone()])); self.outbox.push(DbWrite::Upsert(vec![node.clone()]));
self.intern_mem(parent, node) self.intern_mem(parent, node)
} }
/// Keep what is already known of a file whose content has not changed.
///
/// Drive stamps a link's `ModifyTime` on a rename or a move as well, while
/// POSIX leaves a file's mtime alone there. The content cache is keyed by
/// it, too: taking the new stamp made the cached bytes unreachable, so the
/// next partial write could not fill its gaps (docs/BUGS.md B123). The
/// active revision id says whether the content is still the same, and so
/// whether the size and digest a listing left out are still the ones known.
fn keep_revision_mtime(&self, node: &mut Node) {
let Some(entry) = self
.by_uid
.get(&node.uid)
.and_then(|ino| self.entries.get(ino))
else {
return;
};
if let (
NodeKind::File {
active_revision_id: Some(now),
claimed_size,
content_sha1,
..
},
NodeKind::File {
active_revision_id: Some(was),
claimed_size: size_was,
content_sha1: sha1_was,
..
},
) = (&mut node.kind, &entry.node.kind)
&& now == was
{
node.modification_time = entry.node.modification_time;
if claimed_size.is_none() {
*claimed_size = *size_was;
}
if content_sha1.is_none() {
content_sha1.clone_from(sha1_was);
}
}
}
/// Allocate (or reuse) a stable inode for a node, updating the hot-cache maps /// Allocate (or reuse) a stable inode for a node, updating the hot-cache maps
/// only. Every caller owes the DB a write-through — see the callers below; /// only. Every caller owes the DB a write-through — see the callers below;
/// the split exists so a batch can pay for one transaction instead of `n`. /// the split exists so a batch can pay for one transaction instead of `n`.
@@ -784,11 +946,24 @@ impl State {
} }
fn intern_mem_with_access(&mut self, parent: u64, node: Node, access: Access) -> u64 { fn intern_mem_with_access(&mut self, parent: u64, node: Node, access: Access) -> u64 {
self.intern_mem_at(parent, node, access, None)
}
/// [`State::intern_mem_with_access`], with the inode a new entry takes when
/// the caller already has it ([`State::new_inos`]).
fn intern_mem_at(
&mut self,
parent: u64,
mut node: Node,
access: Access,
ino: Option<u64>,
) -> u64 {
if let Some(&ino) = self.by_uid.get(&node.uid) { if let Some(&ino) = self.by_uid.get(&node.uid) {
let changed = self let changed = self
.entries .entries
.get(&ino) .get(&ino)
.is_some_and(|entry| entry.access != access); .is_some_and(|entry| entry.access != access);
self.keep_open_write_size(ino, &mut node);
if let Some(e) = self.entries.get_mut(&ino) { if let Some(e) = self.entries.get_mut(&ino) {
e.node = node; e.node = node;
e.parent = parent; e.parent = parent;
@@ -801,8 +976,13 @@ impl State {
} }
return ino; return ino;
} }
let ino = self.next_ino; let ino = match ino {
self.next_ino += 1; Some(ino) => ino,
None => self
.new_inos(&[&node])
.remove(&node.uid)
.unwrap_or_default(),
};
self.by_uid.insert(node.uid.clone(), ino); self.by_uid.insert(node.uid.clone(), ino);
self.entries.insert( self.entries.insert(
ino, ino,
@@ -819,6 +999,41 @@ impl State {
ino ino
} }
/// The inodes for nodes that have no entry yet, from their local ids, in
/// one transaction. Should the database fail, they come from
/// [`State::next_ino`] instead, so the mount keeps working, with inodes
/// that last only this run. So do the inodes of virtual folders, which are
/// never stored.
fn new_inos(&mut self, nodes: &[&Node]) -> HashMap<NodeUid, u64> {
let (virtual_nodes, stored): (Vec<&Node>, Vec<&Node>) = nodes
.iter()
.partition(|node| crate::r#virtual::is_virtual_uid(&node.uid));
let mut inos = match self.db.lids_for(&stored) {
Ok(lids) => stored
.iter()
.zip(lids)
.map(|(node, lid)| (node.uid.clone(), lid_ino(lid)))
.collect(),
Err(error) => {
warn!(%error, "no local ids for new inodes; numbering them for this run");
self.run_inos(&stored)
}
};
inos.extend(self.run_inos(&virtual_nodes));
inos
}
fn run_inos(&mut self, nodes: &[&Node]) -> HashMap<NodeUid, u64> {
nodes
.iter()
.map(|node| {
let ino = self.next_ino;
self.next_ino += 1;
(node.uid.clone(), ino)
})
.collect()
}
/// Decrement lookup count for an inode and prune if lookup_count == 0 && open_count == 0 && unlinked. /// Decrement lookup count for an inode and prune if lookup_count == 0 && open_count == 0 && unlinked.
pub(crate) fn forget_lookup(&mut self, ino: u64, nlookup: u64) -> Option<(u64, String)> { pub(crate) fn forget_lookup(&mut self, ino: u64, nlookup: u64) -> Option<(u64, String)> {
if let Some(entry) = self.entries.get_mut(&ino) { if let Some(entry) = self.entries.get_mut(&ino) {
@@ -833,6 +1048,23 @@ impl State {
/// Allocate (or reuse) a stable inode for a node that came *from* the /// Allocate (or reuse) a stable inode for a node that came *from* the
/// database, which is why nothing is written back. /// database, which is why nothing is written back.
/// Carry the size and mtime of a file open for writing over onto `node`,
/// which is about to replace its entry.
///
/// The open handle is ahead of anything Drive can report until release
/// queues it. Taking the remote's size shrank a SQLite database to its empty
/// create while SQLite held it open, and the kernel then capped every read
/// at that size (docs/BUGS.md B109).
pub(crate) fn keep_open_write_size(&self, ino: u64, node: &mut Node) {
let (Some(aw), Some(entry)) = (self.active_writes.get(&ino), self.entries.get(&ino)) else {
return;
};
if let NodeKind::File { claimed_size, .. } = &mut node.kind {
*claimed_size = Some(aw.len as i64);
node.modification_time = entry.node.modification_time;
}
}
pub(crate) fn intern_from_db(&mut self, parent: u64, node: Node) -> u64 { pub(crate) fn intern_from_db(&mut self, parent: u64, node: Node) -> u64 {
self.intern_mem(parent, node) self.intern_mem(parent, node)
} }
@@ -842,11 +1074,27 @@ impl State {
/// ///
/// This is what keeps `ls` on a large folder quick: one commit for the /// This is what keeps `ls` on a large folder quick: one commit for the
/// listing, rather than one autocommit — and one fsync — per child. /// listing, rather than one autocommit — and one fsync — per child.
pub(crate) fn intern_batch(&mut self, parent: u64, nodes: Vec<Node>) -> Vec<u64> { pub(crate) fn intern_batch(&mut self, parent: u64, mut nodes: Vec<Node>) -> Vec<u64> {
for node in &mut nodes {
self.keep_revision_mtime(node);
}
self.outbox.push(DbWrite::Upsert(nodes.clone())); self.outbox.push(DbWrite::Upsert(nodes.clone()));
let fresh: Vec<&Node> = nodes
.iter()
.filter(|node| !self.by_uid.contains_key(&node.uid))
.collect();
let mut inos = if fresh.is_empty() {
HashMap::new()
} else {
self.new_inos(&fresh)
};
nodes nodes
.into_iter() .into_iter()
.map(|node| self.intern_mem(parent, node)) .map(|node| {
let access = self.access_for_node(parent, &node);
let ino = inos.remove(&node.uid);
self.intern_mem_at(parent, node, access, ino)
})
.collect() .collect()
} }
@@ -950,7 +1198,7 @@ impl State {
/// re-enumerate, which is the cheaper way to stay honest about what the /// re-enumerate, which is the cheaper way to stay honest about what the
/// server did. A queued rename cannot: re-enumerating needs the network, and /// server did. A queued rename cannot: re-enumerating needs the network, and
/// the server has not been told yet in any case — so this *is* the tree's /// the server has not been told yet in any case — so this *is* the tree's
/// new truth until the op drains (offline.md Phase 3b). /// new truth until the op drains.
pub(crate) fn rename_in_place( pub(crate) fn rename_in_place(
&mut self, &mut self,
ino: u64, ino: u64,
@@ -1069,7 +1317,7 @@ impl State {
/// created offline, nothing at all), so this row is the only record that the /// created offline, nothing at all), so this row is the only record that the
/// file is as long as the caller was told it is. Without it a restart serves /// file is as long as the caller was told it is. Without it a restart serves
/// the stale size and the file reads as truncated — or empty — while its /// the stale size and the file reads as truncated — or empty — while its
/// bytes sit safely in staging (offline.md Phase 3). /// bytes sit safely in staging.
/// Returns the node to persist rather than queueing it in the outbox, and is /// Returns the node to persist rather than queueing it in the outbox, and is
/// the one mutation here whose write-through the caller must do itself: /// the one mutation here whose write-through the caller must do itself:
/// failure has to be reported, not logged. Dropping the row silently means /// failure has to be reported, not logged. Dropping the row silently means
@@ -1088,6 +1336,13 @@ mod tests {
use proton_drive_rs::ShareMembership; use proton_drive_rs::ShareMembership;
use proton_drive_rs::proton_sdk::ids::{LinkId, ShareId, ShareMembershipId, VolumeId}; use proton_drive_rs::proton_sdk::ids::{LinkId, ShareId, ShareMembershipId, VolumeId};
#[test]
fn an_inode_gives_back_the_lid_it_was_made_from() {
assert_eq!(ino_lid(lid_ino(7)), Some(7));
assert_eq!(ino_lid(crate::ROOT_INO), None);
assert_eq!(ino_lid(FALLBACK_INO + 3), None);
}
fn uid(link: &str) -> NodeUid { fn uid(link: &str) -> NodeUid {
NodeUid::new(VolumeId::from("vol"), LinkId::from(link)) NodeUid::new(VolumeId::from("vol"), LinkId::from(link))
} }
@@ -1164,7 +1419,7 @@ mod tests {
let dir = TempDir::new(); let dir = TempDir::new();
let db = Db::open(&dir.0.join("cache.db")).unwrap(); let db = Db::open(&dir.0.join("cache.db")).unwrap();
let share_access = db.all_share_access().unwrap(); let share_access = db.all_share_access().unwrap();
let st = State::new(Arc::new(db), share_access, 1); let st = State::new(Arc::new(db), share_access);
(st, dir) (st, dir)
} }
@@ -1182,6 +1437,36 @@ mod tests {
(st, dir, src, dst) (st, dir, src, dst)
} }
/// A node's inode comes from its row's local id, so it is the same in
/// every mount, after the hot cache drops it, and after a restart.
#[test]
fn a_node_keeps_its_inode_across_mounts_evictions_and_restarts() {
let (mut st, _dir) = state();
let inos = st.intern_batch(
0,
vec![
node("a", "root", "a.txt", false),
node("b", "root", "b.txt", false),
],
);
st.flush_outbox();
let db = st.db.clone();
assert_ne!(inos[0], inos[1]);
assert!(!inos.contains(&crate::ROOT_INO));
let mut other = State::new(db.clone(), HashMap::new());
assert_eq!(other.intern(0, node("b", "root", "b.txt", false)), inos[1]);
st.forget_mem(&uid("a"));
assert_eq!(st.intern(0, node("a", "root", "a.txt", false)), inos[0]);
let mut restarted = State::new(db, HashMap::new());
assert_eq!(
restarted.intern_batch(0, vec![node("a", "root", "a.txt", false)]),
[inos[0]]
);
}
/// Audit A5. Moving a file out of a folder leaves that folder's DB row /// Audit A5. Moving a file out of a folder leaves that folder's DB row
/// claiming a complete listing, so once the hot cache drops it, remote /// claiming a complete listing, so once the hot cache drops it, remote
/// changes under it stop being seen. `relocate` has to clear both sides. /// changes under it stop being seen. `relocate` has to clear both sides.
@@ -1247,7 +1532,7 @@ mod tests {
let root = uid("root").to_string(); let root = uid("root").to_string();
let state = parking_lot::Mutex::new(st); let state = parking_lot::Mutex::new(st);
{ {
let mut guard = lock_state(&state, &db); let mut guard = StateGuard::new(state.lock(), &db);
guard.intern(0, node("root", "none", "My Files", true)); guard.intern(0, node("root", "none", "My Files", true));
assert!( assert!(
db.node_by_uid(&root).unwrap().is_none(), db.node_by_uid(&root).unwrap().is_none(),
@@ -1490,6 +1775,7 @@ mod tests {
}) })
.collect(); .collect();
st.next_ino = crate::ROOT_INO + nodes.len() as u64; st.next_ino = crate::ROOT_INO + nodes.len() as u64;
st.roots.insert(crate::ROOT_INO);
// Reverse order so entries are inserted before their parents, matching // Reverse order so entries are inserted before their parents, matching
// the hydration path's non-topological materialization. // the hydration path's non-topological materialization.
for (i, node) in nodes.iter().enumerate().rev() { for (i, node) in nodes.iter().enumerate().rev() {
@@ -1863,4 +2149,137 @@ mod tests {
let shared = st.intern(owned_root, shared); let shared = st.intern(owned_root, shared);
assert_eq!(st.entries[&shared].access, Access::Viewer); assert_eq!(st.entries[&shared].access, Access::Viewer);
} }
fn at_revision(mut node: Node, revision: &str, mtime: i64) -> Node {
node.modification_time = mtime;
if let NodeKind::File {
active_revision_id, ..
} = &mut node.kind
{
*active_revision_id = Some(revision.into());
}
node
}
#[test]
fn a_renamed_file_keeps_its_mtime_until_its_content_changes() {
let (mut st, _dir) = state();
let root = st.intern(0, node("root", "none", "My Files", true));
let mut file = node("file", "root", "a.txt", false);
if let NodeKind::File {
claimed_size,
total_size_on_storage,
..
} = &mut file.kind
{
*claimed_size = Some(7);
*total_size_on_storage = 519;
}
let ino = st.intern(root, at_revision(file.clone(), "r1", 100));
// Drive stamps the link on a rename; the content is still r1.
let mut renamed = at_revision(file.clone(), "r1", 160);
renamed.name = "b.txt".into();
st.intern_batch(root, vec![renamed]);
assert_eq!(st.entries[&ino].node.modification_time, 100);
assert_eq!(st.entries[&ino].node.name, "b.txt");
// A listing that leaves the size out still describes revision r1.
let mut bare = at_revision(file.clone(), "r1", 170);
assert_eq!(crate::node_size(&st.entries[&ino].node), 7);
if let NodeKind::File { claimed_size, .. } = &mut bare.kind {
*claimed_size = None;
}
st.intern(root, bare);
assert_eq!(st.entries[&ino].node.modification_time, 100);
assert_eq!(crate::node_size(&st.entries[&ino].node), 7);
st.intern(root, at_revision(file, "r2", 200));
assert_eq!(st.entries[&ino].node.modification_time, 200);
}
fn handle(len: u64, base_size: u64) -> WriteHandle {
let path = std::env::temp_dir().join(format!("pdfs-handle-test-{}", std::process::id()));
let file = File::create(&path).unwrap();
let _ = std::fs::remove_file(&path);
WriteHandle {
ino: 2,
uid: uid("file"),
file: Arc::new(file),
path,
written: Intervals::default(),
len,
base_size,
base_mtime: 0,
base_revision_id: None,
dirty: false,
open_count: 1,
}
}
/// A file created offline is written while its queued create lands. The
/// placeholder had no revision, so the handle takes the empty file the
/// create made as its base (docs/BUGS.md B113).
#[test]
fn a_handle_on_a_placeholder_is_based_on_the_create_that_landed() {
let mut h = handle(0, 0);
assert_eq!(h.base_revision_id, None);
let mut landed = node("real", "root", "f.mkv", false);
landed.modification_time = 160;
if let NodeKind::File {
active_revision_id, ..
} = &mut landed.kind
{
*active_revision_id = Some("empty".into());
}
h.rebase_onto(&landed);
assert_eq!(h.base_mtime, 160);
assert_eq!(h.base_size, 0);
assert_eq!(h.base_revision_id.as_deref(), Some("empty"));
}
#[test]
fn a_listing_does_not_shrink_a_file_open_for_writing() {
let (mut st, _dir) = state();
let ino = st.intern(0, node("file", "root", "workload.db", false));
let mut h = handle(16384, 0);
h.ino = ino;
st.active_writes.insert(ino, h);
st.set_size(ino, 16384);
st.touch_mtime(ino, 500);
// The remote still holds the empty file the create sealed.
st.intern(0, node("file", "root", "workload.db", false));
assert_eq!(crate::node_size(&st.entries[&ino].node), 16384);
assert_eq!(st.entries[&ino].node.modification_time, 500);
// Once the handle is gone the remote is authoritative again.
st.active_writes.remove(&ino);
st.intern(0, node("file", "root", "workload.db", false));
assert_eq!(crate::node_size(&st.entries[&ino].node), 0);
}
#[test]
fn a_write_past_the_end_authors_the_hole_it_leaves() {
// Opened over a 100-byte base, shrunk to 10, then written at 50: bytes
// 10..50 are zeros now, not the base's bytes 10..50.
let mut h = handle(100, 100);
h.written.clip(10);
h.len = 10;
h.record_write(50, 60);
assert_eq!(h.len, 60);
assert_eq!(
h.written.segments(0, 60),
vec![(0, 10, false), (10, 60, true)],
"only the head still comes from the base"
);
}
#[test]
fn a_write_inside_the_file_leaves_the_rest_to_the_base() {
let mut h = handle(100, 100);
h.record_write(40, 50);
assert_eq!(h.len, 100);
assert_eq!(
h.written.segments(0, 100),
vec![(0, 40, false), (40, 50, true), (50, 100, false)]
);
}
} }
+5 -6
View File
@@ -68,7 +68,7 @@ impl Core {
/// docs for the identical/divergent policy. /// docs for the identical/divergent policy.
pub(crate) fn run_conflict_sweep_loop(&self) { pub(crate) fn run_conflict_sweep_loop(&self) {
// Both waits are interruptible, so teardown does not have to sit out a // Both waits are interruptible, so teardown does not have to sit out a
// five-minute interval to join this thread (bugs.md B44). // five-minute interval to join this thread (`docs/BUGS.md` B44).
if !self.shutdown.sleep(CONFLICT_SWEEP_WARMUP) { if !self.shutdown.sleep(CONFLICT_SWEEP_WARMUP) {
return; return;
} }
@@ -165,7 +165,7 @@ impl Core {
} }
match self match self
.rt .rt
.block_on(self.client.trash_nodes(std::slice::from_ref(uid))) .block_on(self.drive.trash_nodes(std::slice::from_ref(uid)))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
Ok(()) => {} Ok(()) => {}
@@ -175,10 +175,9 @@ impl Core {
return; return;
} }
} }
// Every mount, not just the primary: a conflict copy inside a sync // Leaving a conflict copy hooked up in the tree keeps a node we have
// folder lives in that fork's inode space, and leaving it hooked up // just trashed remotely visible and readable in whichever mount shows
// there keeps a node we have just trashed remotely visible and // it (`docs/BUGS.md` B74).
// readable there (`docs/BUGS.md` B74).
self.for_each_state(|st| { self.for_each_state(|st| {
st.forget_or_unlink(uid); st.forget_or_unlink(uid);
}); });
+295 -48
View File
@@ -1,4 +1,4 @@
//! Two-way folder sync engine (devices.md Phase 2). //! Two-way folder sync engine.
//! //!
//! Each `mirror`-mode synced folder is kept mirrored between a local directory //! Each `mirror`-mode synced folder is kept mirrored between a local directory
//! and a folder under this machine's Proton Drive device. A local filesystem //! and a folder under this machine's Proton Drive device. A local filesystem
@@ -73,6 +73,9 @@ enum Applied {
Downloaded, Downloaded,
/// A conflict copy was kept. /// A conflict copy was kept.
Conflict, Conflict,
/// The local file to upload was deleted after the scan planned it. There is
/// nothing left to send, and the next pass sees the deletion (B108).
Vanished,
} }
impl Core { impl Core {
@@ -99,6 +102,9 @@ impl Core {
/// One reconcile pass over a mirror folder, updating its `state` column. /// One reconcile pass over a mirror folder, updating its `state` column.
fn reconcile_pass(&self, folder: &StoredSyncFolder) { fn reconcile_pass(&self, folder: &StoredSyncFolder) {
if self.shutdown.is_stopping() {
return;
}
// Hold the folder's lock for the whole pass so a mode switch can't evict the // Hold the folder's lock for the whole pass so a mode switch can't evict the
// local tree (and mount FUSE over it) while we walk and upload it. // local tree (and mount FUSE over it) while we walk and upload it.
let lock = self.sync_lock(folder.id); let lock = self.sync_lock(folder.id);
@@ -145,8 +151,6 @@ impl Core {
self.push_pass(folder.id, &local_root, &remote_root, &rules) self.push_pass(folder.id, &local_root, &remote_root, &rules)
} else { } else {
match self.do_reconcile(folder.id, &local_root, &remote_root, &rules) { match self.do_reconcile(folder.id, &local_root, &remote_root, &rules) {
Ok(outcome) => Ok(outcome),
Err(PassAbort::Failed(e)) => Err(e),
// The user asked for on-demand while this pass was running. Rather // The user asked for on-demand while this pass was running. Rather
// than make them wait out a walk-and-download whose results are // than make them wait out a walk-and-download whose results are
// about to be deleted, drop it here and do the only part that still // about to be deleted, drop it here and do the only part that still
@@ -161,6 +165,7 @@ impl Core {
self.progress_begin(folder.id); self.progress_begin(folder.id);
self.push_pass(folder.id, &local_root, &remote_root, &rules) self.push_pass(folder.id, &local_root, &remote_root, &rules)
} }
result => result,
} }
}; };
self.progress_end(folder.id); self.progress_end(folder.id);
@@ -213,7 +218,15 @@ impl Core {
); );
} }
} }
Err(e) => { // A push pass is never interrupted, so only the stop gets here: the
// folder keeps the state it had, which the next start's pass updates.
Err(PassAbort::Stopped | PassAbort::Interrupted) => {
info!(id = folder.id, "sync: pass cut short by the stop");
let _ = self
.db
.sync_folder_set_state(folder.id, &current.state, current.last_sync);
}
Err(PassAbort::Failed(e)) => {
warn!(id = folder.id, error = %e, "sync: reconcile failed"); warn!(id = folder.id, error = %e, "sync: reconcile failed");
let _ = self let _ = self
.db .db
@@ -291,7 +304,7 @@ impl Core {
local_root: &Path, local_root: &Path,
remote_root: &NodeUid, remote_root: &NodeUid,
rules: &IgnoreRules, rules: &IgnoreRules,
) -> Result<Outcome, String> { ) -> Result<Outcome, PassAbort> {
let baseline = self let baseline = self
.db .db
.sync_entries(folder_id) .sync_entries(folder_id)
@@ -367,7 +380,7 @@ impl Core {
HashMap::new() HashMap::new()
} else { } else {
self.rt self.rt
.block_on(self.client.enumerate_nodes_light(&check)) .block_on(self.drive.enumerate_nodes_light(&check))
.map_err(|e| format!("resolve changed nodes: {e}"))? .map_err(|e| format!("resolve changed nodes: {e}"))?
.into_iter() .into_iter()
.filter(|n| !n.trashed) .filter(|n| !n.trashed)
@@ -397,7 +410,7 @@ impl Core {
&mut remote_dirs, &mut remote_dirs,
std::mem::take(&mut batch), std::mem::take(&mut batch),
&mut outcome, &mut outcome,
); )?;
batch_depth = depth; batch_depth = depth;
} }
let item = &local[rel]; let item = &local[rel];
@@ -495,7 +508,7 @@ impl Core {
&mut remote_dirs, &mut remote_dirs,
std::mem::take(&mut batch), std::mem::take(&mut batch),
&mut outcome, &mut outcome,
); )?;
// Paths the baseline knows that are no longer on disk: the user deleted them // Paths the baseline knows that are no longer on disk: the user deleted them
// locally, and the deletion has to reach Drive before the mount starts // locally, and the deletion has to reach Drive before the mount starts
@@ -522,7 +535,7 @@ impl Core {
}; };
if let Err(e) = self if let Err(e) = self
.rt .rt
.block_on(self.client.trash_nodes(&[uid])) .block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
warn!(rel, error = %e, "sync: trash remote failed"); warn!(rel, error = %e, "sync: trash remote failed");
@@ -622,7 +635,7 @@ impl Core {
&mut remote_dirs, &mut remote_dirs,
std::mem::take(&mut batch), std::mem::take(&mut batch),
&mut outcome, &mut outcome,
); )?;
batch_depth = depth; batch_depth = depth;
// A depth boundary is the pass's natural checkpoint — everything // A depth boundary is the pass's natural checkpoint — everything
// queued has been applied, so stopping here leaves nothing half done. // queued has been applied, so stopping here leaves nothing half done.
@@ -773,7 +786,7 @@ impl Core {
let remote = r.expect("remote deletion requires a remote file"); let remote = r.expect("remote deletion requires a remote file");
if let Err(e) = self if let Err(e) = self
.rt .rt
.block_on(self.client.trash_nodes(std::slice::from_ref(&remote.uid))) .block_on(self.drive.trash_nodes(std::slice::from_ref(&remote.uid)))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
warn!(rel, error = %e, "sync: trash remote failed"); warn!(rel, error = %e, "sync: trash remote failed");
@@ -810,7 +823,7 @@ impl Core {
&mut remote_dirs, &mut remote_dirs,
std::mem::take(&mut batch), std::mem::take(&mut batch),
&mut outcome, &mut outcome,
); )?;
// Deferred folder deletions, deepest first. // Deferred folder deletions, deepest first.
delete_local_dirs.sort_by_key(|p| std::cmp::Reverse(p.matches('/').count())); delete_local_dirs.sort_by_key(|p| std::cmp::Reverse(p.matches('/').count()));
@@ -836,7 +849,7 @@ impl Core {
for (rel, uid) in delete_remote_dirs { for (rel, uid) in delete_remote_dirs {
if let Err(e) = self if let Err(e) = self
.rt .rt
.block_on(self.client.trash_nodes(&[uid])) .block_on(self.drive.trash_nodes(&[uid]))
.and_then(batch::into_unit) .and_then(batch::into_unit)
{ {
warn!(rel, error = %e, "sync: trash remote folder failed"); warn!(rel, error = %e, "sync: trash remote folder failed");
@@ -891,14 +904,14 @@ impl Core {
) -> Result<(), String> { ) -> Result<(), String> {
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(folder)) .block_on(self.drive.enumerate_folder_children_node_uids(folder))
.map_err(|e| format!("enumerate {folder}: {e}"))?; .map_err(|e| format!("enumerate {folder}: {e}"))?;
if uids.is_empty() { if uids.is_empty() {
return Ok(()); return Ok(());
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes_light(&uids)) .block_on(self.drive.enumerate_nodes_light(&uids))
.map_err(|e| format!("resolve nodes: {e}"))?; .map_err(|e| format!("resolve nodes: {e}"))?;
for node in nodes { for node in nodes {
if node.trashed || !node.is_folder() { if node.trashed || !node.is_folder() {
@@ -938,19 +951,22 @@ impl Core {
// is safe precisely because the walk has applied nothing — a *partial* // is safe precisely because the walk has applied nothing — a *partial*
// remote map, on the other hand, would be read as "the remote deleted // remote map, on the other hand, would be read as "the remote deleted
// everything we haven't reached yet", so it must never reach the diff. // everything we haven't reached yet", so it must never reach the diff.
if self.shutdown.is_stopping() {
return Err(PassAbort::Stopped);
}
if self.ondemand_queued(folder_id) { if self.ondemand_queued(folder_id) {
return Err(PassAbort::Interrupted); return Err(PassAbort::Interrupted);
} }
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(folder)) .block_on(self.drive.enumerate_folder_children_node_uids(folder))
.map_err(|e| format!("enumerate {folder}: {e}"))?; .map_err(|e| format!("enumerate {folder}: {e}"))?;
if uids.is_empty() { if uids.is_empty() {
return Ok(()); return Ok(());
} }
let nodes = self let nodes = self
.rt .rt
.block_on(self.client.enumerate_nodes_light(&uids)) .block_on(self.drive.enumerate_nodes_light(&uids))
.map_err(|e| format!("resolve nodes: {e}"))?; .map_err(|e| format!("resolve nodes: {e}"))?;
// Files whose recorded mtime no longer matches: their size has to be // Files whose recorded mtime no longer matches: their size has to be
@@ -968,7 +984,7 @@ impl Core {
HashMap::new() HashMap::new()
} else { } else {
self.rt self.rt
.block_on(self.client.enumerate_nodes(&stale)) .block_on(self.drive.enumerate_nodes(&stale))
.map_err(|e| format!("resolve nodes: {e}"))? .map_err(|e| format!("resolve nodes: {e}"))?
.iter() .iter()
.map(|n| (n.uid.to_string(), remote_file_sig(n).1)) .map(|n| (n.uid.to_string(), remote_file_sig(n).1))
@@ -1048,6 +1064,9 @@ impl Core {
/// `remote_dirs` (so deeper paths resolve their parent), and what each op did /// `remote_dirs` (so deeper paths resolve their parent), and what each op did
/// is tallied into `outcome`. Driven by `block_on` from the (non-runtime) sync /// is tallied into `outcome`. Driven by `block_on` from the (non-runtime) sync
/// engine thread, spawning the tasks onto the shared runtime. /// engine thread, spawning the tasks onto the shared runtime.
///
/// Fails with [`PassAbort::Stopped`] once the daemon is stopping, after the
/// ops already sending have finished.
fn flush_batch( fn flush_batch(
&self, &self,
folder_id: i64, folder_id: i64,
@@ -1055,9 +1074,9 @@ impl Core {
remote_dirs: &mut HashMap<String, NodeUid>, remote_dirs: &mut HashMap<String, NodeUid>,
batch: Vec<Pending>, batch: Vec<Pending>,
outcome: &mut Outcome, outcome: &mut Outcome,
) { ) -> Result<(), PassAbort> {
if batch.is_empty() { if batch.is_empty() {
return; return Ok(());
} }
self.progress_queued(folder_id, batch.len()); self.progress_queued(folder_id, batch.len());
let core = self.clone(); let core = self.clone();
@@ -1071,14 +1090,22 @@ impl Core {
let root = root.clone(); let root = root.clone();
set.spawn(async move { set.spawn(async move {
let _permit = sem.acquire_owned().await.expect("semaphore not closed"); let _permit = sem.acquire_owned().await.expect("semaphore not closed");
core.apply_pending(folder_id, &root, op).await // An op still waiting for its turn when the stop comes is
// not started; the ones already sending finish.
if core.shutdown.is_stopping() {
return None;
}
Some(core.apply_pending(folder_id, &root, op).await)
}); });
} }
let mut out = Vec::new(); let mut out = Vec::new();
let mut join_errors = 0usize; let mut join_errors = 0usize;
while let Some(joined) = set.join_next().await { while let Some(joined) = set.join_next().await {
match joined { match joined {
Ok(result) => out.push(result), Ok(Some(result)) => out.push(result),
Ok(None) => {}
// The runtime ends with the daemon and cancels what is left.
Err(e) if e.is_cancelled() => {}
Err(e) => { Err(e) => {
join_errors += 1; join_errors += 1;
warn!(error = %e, "sync: task panicked"); warn!(error = %e, "sync: task panicked");
@@ -1103,6 +1130,13 @@ impl Core {
} }
} }
} }
// The pass goes no further: the next depth's paths may need a folder
// this batch did not get to make, and went on to fail one by one
// (`docs/BUGS.md` B169).
if self.shutdown.is_stopping() {
return Err(PassAbort::Stopped);
}
Ok(())
} }
/// Apply one [`Pending`] op (async, so it can run concurrently in a batch), /// Apply one [`Pending`] op (async, so it can run concurrently in a batch),
@@ -1136,7 +1170,7 @@ impl Core {
match op { match op {
Pending::CreateDir { rel, parent } => { Pending::CreateDir { rel, parent } => {
let uid = self let uid = self
.client .drive
.create_folder(parent, base_name(rel), Some(now_secs())) .create_folder(parent, base_name(rel), Some(now_secs()))
.await .await
.map_err(|e| format!("create remote folder {rel}: {e}"))?; .map_err(|e| format!("create remote folder {rel}: {e}"))?;
@@ -1147,13 +1181,10 @@ impl Core {
Ok(Applied::Dir(rel.clone(), uid)) Ok(Applied::Dir(rel.clone(), uid))
} }
Pending::UploadNew { rel, parent } => { Pending::UploadNew { rel, parent } => {
self.upload_new(folder_id, local_root, rel, parent).await?; self.upload_new(folder_id, local_root, rel, parent).await
Ok(Applied::Uploaded)
} }
Pending::UploadRevision { rel, uid } => { Pending::UploadRevision { rel, uid } => {
self.upload_revision(folder_id, local_root, rel, uid) self.upload_revision(folder_id, local_root, rel, uid).await
.await?;
Ok(Applied::Uploaded)
} }
Pending::Download { Pending::Download {
rel, rel,
@@ -1217,10 +1248,12 @@ impl Core {
local_root: &Path, local_root: &Path,
rel: &str, rel: &str,
parent: &NodeUid, parent: &NodeUid,
) -> Result<(), String> { ) -> Result<Applied, String> {
let path = local_root.join(rel_to_path(rel)); let path = local_root.join(rel_to_path(rel));
let name = base_name(rel); let name = base_name(rel);
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?; let Some(file) = open_to_upload(&path, rel)? else {
return Ok(Applied::Vanished);
};
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?; let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
let mtime = system_mtime(&meta); let mtime = system_mtime(&meta);
// Taken from the descriptor being streamed, so it names the content that // Taken from the descriptor being streamed, so it names the content that
@@ -1238,7 +1271,7 @@ impl Core {
.begin(name, "", TransferDirection::Upload, meta.len()), .begin(name, "", TransferDirection::Upload, meta.len()),
); );
let uid = self let uid = self
.client .drive
// Mirror push: the local file is authoritative. If a prior attempt was // Mirror push: the local file is authoritative. If a prior attempt was
// interrupted mid-upload it left an unsealed draft of this name; recover // interrupted mid-upload it left an unsealed draft of this name; recover
// it even across a daemon restart (which rotates our client uid), so the // it even across a daemon restart (which rotates our client uid), so the
@@ -1261,7 +1294,8 @@ impl Core {
self.invalidate_children_of(parent); self.invalidate_children_of(parent);
warn_if_torn(rel, &path, streamed); warn_if_torn(rel, &path, streamed);
self.record_file_baseline(folder_id, rel, Some(streamed), &uid) self.record_file_baseline(folder_id, rel, Some(streamed), &uid)
.await .await?;
Ok(Applied::Uploaded)
} }
/// Upload a changed local file as a new revision of an existing remote node. /// Upload a changed local file as a new revision of an existing remote node.
@@ -1271,9 +1305,11 @@ impl Core {
local_root: &Path, local_root: &Path,
rel: &str, rel: &str,
uid: &NodeUid, uid: &NodeUid,
) -> Result<(), String> { ) -> Result<Applied, String> {
let path = local_root.join(rel_to_path(rel)); let path = local_root.join(rel_to_path(rel));
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?; let Some(file) = open_to_upload(&path, rel)? else {
return Ok(Applied::Vanished);
};
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?; let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
let mtime = system_mtime(&meta); let mtime = system_mtime(&meta);
let streamed = LocalSig::from(&meta); let streamed = LocalSig::from(&meta);
@@ -1289,13 +1325,14 @@ impl Core {
meta.len(), meta.len(),
), ),
); );
self.client self.drive
.upload_new_revision_from(uid, reader, meta.len() as i64, thumbnails, Some(mtime)) .upload_new_revision_from(uid, reader, meta.len() as i64, thumbnails, Some(mtime))
.await .await
.map_err(|e| format!("upload revision {rel}: {e}"))?; .map_err(|e| format!("upload revision {rel}: {e}"))?;
warn_if_torn(rel, &path, streamed); warn_if_torn(rel, &path, streamed);
self.record_file_baseline(folder_id, rel, Some(streamed), uid) self.record_file_baseline(folder_id, rel, Some(streamed), uid)
.await .await?;
Ok(Applied::Uploaded)
} }
/// Download a remote file to its local path (atomically via a temp file), /// Download a remote file to its local path (atomically via a temp file),
@@ -1305,7 +1342,7 @@ impl Core {
/// rename below is the destructive step, and the download in between can run /// rename below is the destructive step, and the download in between can run
/// for minutes, so the destination is re-examined right before it: a local /// for minutes, so the destination is re-examined right before it: a local
/// file that no longer matches the plan is set aside as a conflict copy /// file that no longer matches the plan is set aside as a conflict copy
/// instead of being replaced (bugs.md B39). /// instead of being replaced (`docs/BUGS.md` B39).
#[allow(clippy::too_many_arguments)] #[allow(clippy::too_many_arguments)]
async fn download_file( async fn download_file(
&self, &self,
@@ -1328,7 +1365,21 @@ impl Core {
.map(|d| d.as_nanos()) .map(|d| d.as_nanos())
.unwrap_or(0) .unwrap_or(0)
)); ));
{ // A file an on-demand mount already read, say one just moved here from
// it, is in the content cache under the same revision key.
let seeded = size > 0 && self.cache.copy_cached_to(uid, mtime, size as u64, &tmp);
if seeded {
std::fs::File::open(&tmp)
.and_then(|f| f.sync_all())
.map_err(|e| {
let _ = std::fs::remove_file(&tmp);
format!("sync tmp {rel}: {e}")
})?;
debug!(
rel,
"sync: placed from the content cache instead of downloading"
);
} else {
let out = std::fs::File::create(&tmp).map_err(|e| format!("create tmp {rel}: {e}"))?; let out = std::fs::File::create(&tmp).map_err(|e| format!("create tmp {rel}: {e}"))?;
// Count the bytes as they land, so the download reports progress like // Count the bytes as they land, so the download reports progress like
// an on-demand hydration does. // an on-demand hydration does.
@@ -1339,7 +1390,7 @@ impl Core {
size.max(0) as u64, size.max(0) as u64,
); );
let mut out = CountingWriter::new(out, &guard); let mut out = CountingWriter::new(out, &guard);
self.client self.drive
.download_file_to(uid, &mut out) .download_file_to(uid, &mut out)
.await .await
.map_err(|e| { .map_err(|e| {
@@ -1390,7 +1441,7 @@ impl Core {
/// the pre-upload identity is what makes the next pass see the path as /// the pre-upload identity is what makes the next pass see the path as
/// locally changed and upload a whole, clean revision over it. Recording the /// locally changed and upload a whole, clean revision over it. Recording the
/// post-upload stat instead declared the torn revision synchronized and left /// post-upload stat instead declared the torn revision synchronized and left
/// it as the file's content on Drive (bugs.md B40). /// it as the file's content on Drive (`docs/BUGS.md` B40).
async fn record_file_baseline( async fn record_file_baseline(
&self, &self,
folder_id: i64, folder_id: i64,
@@ -1404,7 +1455,7 @@ impl Core {
let (lmtime, lsize) = (local.mtime, local.size); let (lmtime, lsize) = (local.mtime, local.size);
// Re-fetch the node so the stored remote signature is exactly what a walk // Re-fetch the node so the stored remote signature is exactly what a walk
// will report next time. // will report next time.
let (rmtime, rsize) = match self.client.enumerate_nodes(std::slice::from_ref(uid)).await { let (rmtime, rsize) = match self.drive.enumerate_nodes(std::slice::from_ref(uid)).await {
Ok(nodes) => nodes Ok(nodes) => nodes
.first() .first()
.map(remote_file_sig) .map(remote_file_sig)
@@ -1431,13 +1482,30 @@ impl Core {
} }
} }
/// Open a local file for upload, or `None` when it is gone.
///
/// A scan plans uploads from a listing, and the file can be deleted before its
/// upload starts — a build or a test tree cleaned up mid-pass. That is not a
/// failure: the next pass sees the deletion. Counting it as one put the folder
/// in `error` for a file nobody wanted any more (B108).
fn open_to_upload(path: &Path, rel: &str) -> Result<Option<std::fs::File>, String> {
match std::fs::File::open(path) {
Ok(file) => Ok(Some(file)),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
debug!(rel, "sync: file deleted before its upload; skipping");
Ok(None)
}
Err(e) => Err(format!("open {rel}: {e}")),
}
}
/// Say so when a file changed under an upload that was streaming it. /// Say so when a file changed under an upload that was streaming it.
/// ///
/// Purely diagnostic — the correction is already in place, because the baseline /// Purely diagnostic — the correction is already in place, because the baseline
/// records the identity the upload *read* rather than the one on disk now, so /// records the identity the upload *read* rather than the one on disk now, so
/// the next pass classifies the path as locally changed and replaces the torn /// the next pass classifies the path as locally changed and replaces the torn
/// revision. Without the log there is nothing in the journal to explain the /// revision. Without the log there is nothing in the journal to explain the
/// extra revision (bugs.md B40). /// extra revision (`docs/BUGS.md` B40).
fn warn_if_torn(rel: &str, path: &Path, streamed: LocalSig) { fn warn_if_torn(rel: &str, path: &Path, streamed: LocalSig) {
let now = local_sig(path); let now = local_sig(path);
if !now.is_some_and(|now| now.same_content(&streamed)) { if !now.is_some_and(|now| now.same_content(&streamed)) {
@@ -1469,7 +1537,7 @@ fn local_sig(path: &Path) -> Option<LocalSig> {
/// transfer started, and a download of a large file can run for minutes. Without /// transfer started, and a download of a large file can run for minutes. Without
/// this, a local edit completed inside that window was replaced by the remote /// this, a local edit completed inside that window was replaced by the remote
/// version with no copy of it kept anywhere — the one shape of loss a sync /// version with no copy of it kept anywhere — the one shape of loss a sync
/// engine must not have (bugs.md B39). Keeping the copy makes it re-upload as a /// engine must not have (`docs/BUGS.md` B39). Keeping the copy makes it re-upload as a
/// new file on the next pass, which is the same resolution a both-sides-changed /// new file on the next pass, which is the same resolution a both-sides-changed
/// conflict already gets. /// conflict already gets.
/// ///
@@ -1505,7 +1573,7 @@ fn keep_racing_local_edit(path: &Path, expected: Option<LocalSig>) -> std::io::R
/// exists, so the same removal was re-attempted and re-reported as an error on /// exists, so the same removal was re-attempted and re-reported as an error on
/// every pass, forever. Every other error is still an error, and keeping the /// every pass, forever. Every other error is still an error, and keeping the
/// baseline is exactly right there: the surviving file must not be mistaken for /// baseline is exactly right there: the surviving file must not be mistaken for
/// a new local creation on the next pass and uploaded back (bugs.md B42). /// a new local creation on the next pass and uploaded back (`docs/BUGS.md` B42).
fn removed_locally(result: std::io::Result<()>) -> std::io::Result<()> { fn removed_locally(result: std::io::Result<()>) -> std::io::Result<()> {
match result { match result {
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
@@ -1602,6 +1670,76 @@ fn walk_local_tree(
Ok(()) Ok(())
} }
/// Why the mirror copy at `rel` holds something Proton Drive does not, or
/// `None` when everything under it is on Drive exactly as it is on disk.
///
/// A move out of a mirror folder removes the local copy once Drive has moved
/// the node, so anything only this disk has would be lost with it: a file not
/// uploaded yet, one edited since, a deletion not carried up yet, or anything
/// the engine never uploads. That is why ignored names count here, and why the
/// walk is its own rather than [`walk_local_tree`], which skips symlinks.
pub(crate) fn mirror_subtree_unsynced(
root: &Path,
rel: &str,
baseline: &HashMap<String, StoredSyncEntry>,
) -> Option<String> {
let mut seen = HashSet::new();
if let Some(reason) = unsynced_local(root, rel, baseline, &mut seen) {
return Some(reason);
}
let prefix = format!("{rel}/");
baseline
.keys()
.filter(|r| *r == rel || r.starts_with(&prefix))
.find(|r| !seen.contains(r.as_str()))
.map(|r| format!("{r} was deleted here but not on Proton Drive yet"))
}
fn unsynced_local(
root: &Path,
rel: &str,
baseline: &HashMap<String, StoredSyncEntry>,
seen: &mut HashSet<String>,
) -> Option<String> {
let path = root.join(rel_to_path(rel));
let meta = match std::fs::symlink_metadata(&path) {
Ok(meta) => meta,
Err(e) => return Some(format!("cannot read {rel}: {e}")),
};
let Some(base) = baseline.get(rel) else {
return Some(format!("{rel} is not on Proton Drive yet"));
};
seen.insert(rel.to_string());
if meta.is_file() {
return (!LocalSig::from(&meta).same_content(&LocalSig::from(base)))
.then(|| format!("{rel} has changes that are not on Proton Drive yet"));
}
if !meta.is_dir() {
return Some(format!("{rel} is not a regular file or folder"));
}
let entries = match std::fs::read_dir(&path) {
Ok(entries) => entries,
Err(e) => return Some(format!("cannot read {rel}: {e}")),
};
for entry in entries {
let name = match entry {
Ok(entry) => entry.file_name(),
Err(e) => return Some(format!("cannot read {rel}: {e}")),
};
let Some(name) = name.to_str() else {
return Some(format!("{rel} holds a name that is not valid UTF-8"));
};
// A download the engine left half done; it is the engine's to clean up.
if name.contains(".pdfs-tmp-") {
continue;
}
if let Some(reason) = unsynced_local(root, &format!("{rel}/{name}"), baseline, seen) {
return Some(reason);
}
}
None
}
/// Publish a conflict copy without overwriting an earlier conflict bearing the /// Publish a conflict copy without overwriting an earlier conflict bearing the
/// same timestamp. The original is removed only after the new copy is durable. /// same timestamp. The original is removed only after the new copy is durable.
fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> { fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> {
@@ -1692,7 +1830,7 @@ fn system_mtime(meta: &std::fs::Metadata) -> i64 {
/// ///
/// Whole seconds cannot distinguish an edit made in the same second as the last /// Whole seconds cannot distinguish an edit made in the same second as the last
/// sync from no edit at all, and if the size did not change either, that file /// sync from no edit at all, and if the size did not change either, that file
/// was never uploaded (bugs.md B25). `None` for a clock or filesystem that /// was never uploaded (`docs/BUGS.md` B25). `None` for a clock or filesystem that
/// cannot answer, which falls the comparison back to seconds rather than /// cannot answer, which falls the comparison back to seconds rather than
/// asserting an mtime of zero. /// asserting an mtime of zero.
fn system_mtime_ns(meta: &std::fs::Metadata) -> Option<i64> { fn system_mtime_ns(meta: &std::fs::Metadata) -> Option<i64> {
@@ -1707,7 +1845,7 @@ mod tests {
/// A removal of something already gone has produced the state the sync /// A removal of something already gone has produced the state the sync
/// engine wanted, so its baseline is dropped rather than retried forever. /// engine wanted, so its baseline is dropped rather than retried forever.
/// Anything else stays an error and keeps the baseline (bugs.md B42). /// Anything else stays an error and keeps the baseline (`docs/BUGS.md` B42).
#[test] #[test]
fn an_already_absent_path_counts_as_removed() { fn an_already_absent_path_counts_as_removed() {
let root = std::env::temp_dir().join(format!("pdfs-b42-{}", std::process::id())); let root = std::env::temp_dir().join(format!("pdfs-b42-{}", std::process::id()));
@@ -1724,7 +1862,7 @@ mod tests {
} }
/// The download guard: a destination that still matches the plan is /// The download guard: a destination that still matches the plan is
/// replaced silently, one that has moved on since is kept (bugs.md B39). /// replaced silently, one that has moved on since is kept (`docs/BUGS.md` B39).
#[test] #[test]
fn a_download_keeps_a_local_edit_it_did_not_plan_for() { fn a_download_keeps_a_local_edit_it_did_not_plan_for() {
let root = std::env::temp_dir().join(format!("pdfs-b39-{}", std::process::id())); let root = std::env::temp_dir().join(format!("pdfs-b39-{}", std::process::id()));
@@ -2039,7 +2177,7 @@ mod tests {
std::fs::remove_dir_all(&root).unwrap(); std::fs::remove_dir_all(&root).unwrap();
} }
/// A restored folder (features.md 5.2) starts with an empty baseline against /// A restored folder starts with an empty baseline against
/// an empty local directory and a full remote. That must reconcile as /// an empty local directory and a full remote. That must reconcile as
/// "download everything" rather than tripping the wipe guard — otherwise /// "download everything" rather than tripping the wipe guard — otherwise
/// every restore would wedge its folder on the first pass. /// every restore would wedge its folder on the first pass.
@@ -2155,6 +2293,27 @@ mod tests {
assert_eq!(remote_sig(&e), None); assert_eq!(remote_sig(&e), None);
} }
#[test]
fn a_file_deleted_before_its_upload_is_skipped_not_failed() {
let root = std::env::temp_dir().join(format!("pdfs-b108-{}", std::process::id()));
std::fs::create_dir_all(&root).unwrap();
let gone = root.join("gone.txt");
assert!(open_to_upload(&gone, "gone.txt").unwrap().is_none());
std::fs::write(&gone, b"x").unwrap();
assert!(open_to_upload(&gone, "gone.txt").unwrap().is_some());
// Other open failures still count as errors.
let err = open_to_upload(&gone.join("child"), "gone.txt/child");
assert!(err.unwrap_err().starts_with("open gone.txt/child:"));
let _ = std::fs::remove_dir_all(&root);
let mut o = Outcome::default();
o.record(&Applied::Vanished);
assert!(o.is_empty());
assert_eq!(o.errors, 0);
}
#[test] #[test]
fn outcome_summarises_only_what_it_moved() { fn outcome_summarises_only_what_it_moved() {
let mut o = Outcome::default(); let mut o = Outcome::default();
@@ -2346,6 +2505,94 @@ mod tests {
); );
} }
/// A baseline row recording `path` as it is on disk right now.
fn synced_row(root: &Path, rel: &str) -> (String, StoredSyncEntry) {
let meta = std::fs::symlink_metadata(root.join(rel_to_path(rel))).unwrap();
let sig = LocalSig::from(&meta);
let entry = StoredSyncEntry {
rel_path: rel.to_string(),
remote_uid: Some(format!("vol~{rel}")),
local_mtime: sig.mtime,
local_mtime_ns: sig.mtime_ns,
local_size: sig.size,
remote_rev: None,
remote_hash: None,
};
(rel.to_string(), entry)
}
fn synced_subtree(name: &str) -> (PathBuf, HashMap<String, StoredSyncEntry>) {
let root = sync_test_dir(name);
std::fs::create_dir_all(root.join("dir/sub")).unwrap();
std::fs::write(root.join("dir/a.txt"), b"a").unwrap();
std::fs::write(root.join("dir/sub/b.txt"), b"b").unwrap();
let baseline = ["dir", "dir/sub", "dir/a.txt", "dir/sub/b.txt"]
.into_iter()
.map(|rel| synced_row(&root, rel))
.collect();
(root, baseline)
}
#[test]
fn a_fully_synced_mirror_subtree_may_leave() {
let (root, baseline) = synced_subtree("move-synced");
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert_eq!(reason, None);
}
#[test]
fn a_mirror_subtree_with_a_file_not_uploaded_may_not_leave() {
let (root, baseline) = synced_subtree("move-new-file");
std::fs::write(root.join("dir/sub/new.txt"), b"n").unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/sub/new.txt"));
}
#[test]
fn a_mirror_subtree_with_an_edit_not_uploaded_may_not_leave() {
let (root, baseline) = synced_subtree("move-edit");
std::fs::write(root.join("dir/a.txt"), b"longer now").unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/a.txt"));
}
#[test]
fn a_mirror_subtree_with_a_deletion_not_carried_up_may_not_leave() {
let (root, baseline) = synced_subtree("move-deleted");
std::fs::remove_file(root.join("dir/sub/b.txt")).unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/sub/b.txt"));
}
#[test]
fn a_mirror_subtree_holding_a_symlink_may_not_leave() {
let (root, baseline) = synced_subtree("move-symlink");
std::os::unix::fs::symlink("a.txt", root.join("dir/link")).unwrap();
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert!(reason.unwrap().contains("dir/link"));
}
#[test]
fn a_name_prefixed_sibling_does_not_hold_a_mirror_subtree_back() {
let (root, mut baseline) = synced_subtree("move-sibling");
// Deleted here but not on Drive, yet outside the subtree being moved.
baseline.insert(
"dirt".to_string(),
StoredSyncEntry {
rel_path: "dirt".to_string(),
..baseline["dir/a.txt"].clone()
},
);
let reason = mirror_subtree_unsynced(&root, "dir", &baseline);
std::fs::remove_dir_all(&root).unwrap();
assert_eq!(reason, None);
}
/// The same rule one level up: if the sync root itself cannot be read, the /// The same rule one level up: if the sync root itself cannot be read, the
/// pass sees nothing, and "nothing" must never mean "delete everything". /// pass sees nothing, and "nothing" must never mean "delete everything".
#[test] #[test]
+1 -1
View File
@@ -33,7 +33,7 @@ pub(crate) enum SyncMsg {
/// Re-read the folder set and adjust the filesystem watches (after add/remove). /// Re-read the folder set and adjust the filesystem watches (after add/remove).
Rewatch, Rewatch,
/// The daemon is tearing down: finish the current pass and end the engine /// The daemon is tearing down: finish the current pass and end the engine
/// thread, so the mount can join it (bugs.md B44). /// thread, so the mount can join it (`docs/BUGS.md` B44).
Stop, Stop,
} }
+7 -2
View File
@@ -20,7 +20,7 @@ impl LocalSig {
/// Nanoseconds decide it when both sides have them; otherwise the comparison /// Nanoseconds decide it when both sides have them; otherwise the comparison
/// falls back to whole seconds, which is what a baseline row written before /// falls back to whole seconds, which is what a baseline row written before
/// schema 28 — or a filesystem that reports no sub-second time — can support /// schema 28 — or a filesystem that reports no sub-second time — can support
/// (bugs.md B25). /// (`docs/BUGS.md` B25).
pub(super) fn same_content(&self, other: &Self) -> bool { pub(super) fn same_content(&self, other: &Self) -> bool {
if self.size != other.size { if self.size != other.size {
return false; return false;
@@ -108,6 +108,7 @@ impl Outcome {
Applied::Uploaded => self.uploaded += 1, Applied::Uploaded => self.uploaded += 1,
Applied::Downloaded => self.downloaded += 1, Applied::Downloaded => self.downloaded += 1,
Applied::Conflict => self.conflicts += 1, Applied::Conflict => self.conflicts += 1,
Applied::Vanished => {}
} }
} }
@@ -160,7 +161,7 @@ pub(super) enum Pending {
/// was there. A download can run for minutes, so the apply step revalidates /// was there. A download can run for minutes, so the apply step revalidates
/// against it immediately before it renames over the destination: a writer /// against it immediately before it renames over the destination: a writer
/// that finished an edit in the meantime must not be overwritten without a /// that finished an edit in the meantime must not be overwritten without a
/// conflict copy (bugs.md B39). /// conflict copy (`docs/BUGS.md` B39).
Download { Download {
rel: String, rel: String,
uid: NodeUid, uid: NodeUid,
@@ -226,6 +227,10 @@ pub(super) enum PassAbort {
Interrupted, Interrupted,
/// The pass could not establish its diff (a walk or the baseline load failed). /// The pass could not establish its diff (a walk or the baseline load failed).
Failed(String), Failed(String),
/// The daemon is stopping. What the pass sent before the stop has landed or
/// will be cancelled with the runtime; nothing more is sent, and the next
/// start's pass picks up from the baseline.
Stopped,
} }
impl From<String> for PassAbort { impl From<String> for PassAbort {
+2 -2
View File
@@ -218,7 +218,7 @@ pub(super) fn plan_file(
} }
let local_changed = local.is_some_and(|item| { let local_changed = local.is_some_and(|item| {
// Nanoseconds where both sides have them (bugs.md B25): whole seconds // Nanoseconds where both sides have them (`docs/BUGS.md` B25): whole seconds
// could not see an edit made in the same second as the last sync that // could not see an edit made in the same second as the last sync that
// left the file the same length, so it was never uploaded. // left the file the same length, so it was never uploaded.
baseline.is_none_or(|base| !LocalSig::from(item).same_content(&LocalSig::from(base))) baseline.is_none_or(|base| !LocalSig::from(item).same_content(&LocalSig::from(base)))
@@ -291,7 +291,7 @@ mod file_plan_tests {
/// Whole seconds cannot see an edit made inside the same second that leaves the /// Whole seconds cannot see an edit made inside the same second that leaves the
/// file the same length, so a mirror silently stopped uploading it. With a /// file the same length, so a mirror silently stopped uploading it. With a
/// nanosecond time on both sides it is an ordinary local change (bugs.md B25). /// nanosecond time on both sides it is an ordinary local change (`docs/BUGS.md` B25).
#[test] #[test]
fn a_same_second_same_size_edit_is_still_a_local_change() { fn a_same_second_same_size_edit_is_still_a_local_change() {
let mut base = baseline(10, 20, 30, 40); let mut base = baseline(10, 20, 30, 40);
+20
View File
@@ -74,6 +74,26 @@ pub(crate) fn watchdog_ping() {
let _ = notify("WATCHDOG=1\n"); let _ = notify("WATCHDOG=1\n");
} }
/// Sleep for `duration`, pinging the watchdog on the way.
///
/// For the stretches where the daemon is alive but has no mount, and so no
/// supervisor: waiting for a login, or backing off after a failed mount. The
/// manager arms the watchdog when the process starts, so without these pings a
/// daemon that is merely retrying is aborted after `WatchdogSec` and leaves a
/// coredump instead of the error it was logging. Never use this while mounted:
/// there the ping must prove the control socket answers.
pub fn idle_sleep(duration: Duration) {
const CHUNK: Duration = Duration::from_secs(10);
let mut left = duration;
watchdog_ping();
while !left.is_zero() {
let step = left.min(CHUNK);
std::thread::sleep(step);
left -= step;
watchdog_ping();
}
}
/// Tell the manager the daemon is stopping, so a slow teardown is not mistaken /// Tell the manager the daemon is stopping, so a slow teardown is not mistaken
/// for a hang by the watchdog it is racing. /// for a hang by the watchdog it is racing.
pub(crate) fn stopping() { pub(crate) fn stopping() {
+1 -1
View File
@@ -530,7 +530,7 @@ fn open_archive(
/// The lowercase-hex SHA-1 of everything `reader` yields — the digest Proton's /// The lowercase-hex SHA-1 of everything `reader` yields — the digest Proton's
/// photo duplicate detection is keyed on. /// photo duplicate detection is keyed on.
fn sha1_of(reader: &mut (dyn Read + Send)) -> CoreResult<String> { pub(crate) fn sha1_of(reader: &mut (dyn Read + Send)) -> CoreResult<String> {
let mut hasher = Sha1::new(); let mut hasher = Sha1::new();
let mut buffer = vec![0u8; 256 * 1024]; let mut buffer = vec![0u8; 256 * 1024];
loop { loop {
+4 -7
View File
@@ -73,7 +73,7 @@ async fn run_uploads(
.upload_thumbnails_async(t.path.clone(), t.name.clone()) .upload_thumbnails_async(t.path.clone(), t.name.clone())
.await; .await;
match core match core
.client .drive
.upload_file_from( .upload_file_from(
&t.parent_uid, &t.parent_uid,
&t.name, &t.name,
@@ -238,14 +238,11 @@ impl Core {
.map_err(|error| self.errno_error(error, "upload destination"))?; .map_err(|error| self.errno_error(error, "upload destination"))?;
let new_uid = self let new_uid = self
.rt .rt
.block_on( .block_on(self.drive.create_folder(parent_uid, name, Some(now_secs())))
self.client
.create_folder(parent_uid, name, Some(now_secs())),
)
.map_err(|e| CoreError::from_api(&e, &format!("create folder {name}")))?; .map_err(|e| CoreError::from_api(&e, &format!("create folder {name}")))?;
let node = self let node = self
.fetch_node(&new_uid) .fetch_minted_node(&new_uid, name)
.map_err(|e| self.errno_error(e, "fetch node"))?; .ok_or_else(|| minted_but_unlisted(name))?;
let mut st = self.state(); let mut st = self.state();
let ino = st.intern(pino, node); let ino = st.intern(pino, node);
if let Some(kids) = st.children.get_mut(&pino) if let Some(kids) = st.children.get_mut(&pino)
+1
View File
@@ -27,5 +27,6 @@ fn main() {
println!("cargo:rerun-if-changed=resources/icons"); println!("cargo:rerun-if-changed=resources/icons");
println!("cargo:rerun-if-changed=resources/style.css"); println!("cargo:rerun-if-changed=resources/style.css");
println!("cargo:rerun-if-changed=resources/proton-theme.css"); println!("cargo:rerun-if-changed=resources/proton-theme.css");
println!("cargo:rerun-if-changed=resources/prompt.css");
println!("cargo:rerun-if-changed=../../packaging/io.narl.proton-drive-linux.svg"); println!("cargo:rerun-if-changed=../../packaging/io.narl.proton-drive-linux.svg");
} }
@@ -33,5 +33,7 @@
<gresource prefix="/de/nils/protondrivelinux"> <gresource prefix="/de/nils/protondrivelinux">
<file>style.css</file> <file>style.css</file>
<file>proton-theme.css</file> <file>proton-theme.css</file>
<!-- The search launcher's own sheet, over the same palette. -->
<file>prompt.css</file>
</gresource> </gresource>
</gresources> </gresources>
+224
View File
@@ -0,0 +1,224 @@
/* The search launcher (pdfs-prompt). Colours come from the libadwaita
palette, so the launcher follows the system accent, light/dark and high
contrast like the app; the opt-in Proton theme recolours both alike. */
window.launcher-window {
background: transparent;
}
.launcher-card {
background-color: var(--view-bg-color);
color: var(--view-fg-color);
border: 1px solid var(--border-color);
border-radius: 14px;
box-shadow: 0 16px 48px rgb(0 0 0 / 35%);
}
.search-bar {
padding: 14px 18px;
border-bottom: 1px solid var(--border-color);
}
.search-icon {
-gtk-icon-size: 20px;
color: color-mix(in srgb, currentColor 55%, transparent);
}
.search-entry {
background: none;
border: none;
box-shadow: none;
outline: none;
padding: 2px 0;
font-size: 1.15rem;
}
.key-hint {
padding: 2px 8px;
border-radius: 6px;
background-color: color-mix(in srgb, currentColor 8%, transparent);
font-size: 0.75rem;
opacity: 0.55;
}
.chip-row {
padding: 10px 14px;
border-bottom: 1px solid var(--border-color);
}
.chip {
padding: 4px 14px;
min-height: 0;
border-radius: 999px;
background: none;
border: 1px solid var(--border-color);
box-shadow: none;
font-size: 0.86rem;
font-weight: 500;
}
.chip:hover {
background-color: color-mix(in srgb, currentColor 6%, transparent);
}
.chip:checked {
background-color: var(--accent-bg-color);
color: var(--accent-fg-color);
border-color: var(--accent-bg-color);
}
/* --- results ------------------------------------------------------------ */
.results {
padding: 2px 8px 10px 8px;
}
.section-header {
padding: 12px 10px 6px 10px;
}
.section-title {
font-size: 0.78rem;
font-weight: 700;
letter-spacing: 0.06em;
text-transform: uppercase;
opacity: 0.55;
}
.section-icon {
-gtk-icon-size: 14px;
opacity: 0.55;
}
.result-list {
background: none;
}
.result-row {
padding: 6px 10px;
border-radius: 10px;
}
.result-row:hover {
background-color: color-mix(in srgb, currentColor 5%, transparent);
}
/* A soft tint rather than a solid accent bar: the text keeps its colour, so
the highlighted match stays readable on the selected row. */
.result-row:selected,
.result-row:selected:hover {
background-color: color-mix(in srgb, var(--accent-bg-color) 18%, transparent);
color: inherit;
}
.result-name {
font-weight: 500;
}
.result-location,
.result-meta {
font-size: 0.8rem;
opacity: 0.55;
}
.result-row:selected .result-location,
.result-row:selected .result-meta {
opacity: 0.7;
}
.thumb {
border-radius: 6px;
}
.result-badge,
.preview-badge {
-gtk-icon-size: 14px;
}
.badge-pinned {
color: var(--success-color);
}
.badge-cached {
color: var(--accent-color);
}
.badge-cloud,
.badge-local {
opacity: 0.55;
}
/* --- preview ------------------------------------------------------------ */
.preview {
padding: 16px;
background-color: color-mix(in srgb, currentColor 3%, transparent);
}
.preview-thumb {
margin-bottom: 12px;
}
.preview-thumb .thumb {
border-radius: 10px;
}
.preview-name {
font-weight: 700;
font-size: 1.05rem;
}
.preview-kind {
font-size: 0.85rem;
opacity: 0.6;
margin-bottom: 12px;
}
.preview-key {
font-size: 0.8rem;
opacity: 0.55;
}
.preview-value {
font-size: 0.85rem;
}
.preview-actions {
margin-top: 12px;
}
.preview-action {
padding: 6px 8px;
border-radius: 8px;
}
.preview-shortcut {
font-size: 0.75rem;
opacity: 0.6;
}
.preview-feedback {
margin-top: 6px;
font-size: 0.8rem;
color: var(--accent-color);
}
.preview-empty {
opacity: 0.2;
}
/* --- footer ------------------------------------------------------------- */
.footer {
padding: 8px 16px;
border-top: 1px solid var(--border-color);
}
.footer-text {
font-size: 0.78rem;
opacity: 0.5;
}
statuspage.compact > scrolledwindow > viewport > box {
margin: 24px 12px;
}
+1
View File
@@ -46,6 +46,7 @@
.badge-pinned { color: var(--success-color); } .badge-pinned { color: var(--success-color); }
.badge-cached { color: var(--success-color); } .badge-cached { color: var(--success-color); }
.badge-cloud { color: color-mix(in srgb, currentColor 55%, transparent); } .badge-cloud { color: color-mix(in srgb, currentColor 55%, transparent); }
.badge-issue { color: var(--error-color); }
.browser-statusbar { background-color: alpha(currentColor, 0.025); } .browser-statusbar { background-color: alpha(currentColor, 0.025); }
scale.browser-status-meter trough { min-width: 104px; min-height: 6px; } scale.browser-status-meter trough { min-width: 104px; min-height: 6px; }
.bulk-bar { padding: 6px 8px; border-radius: 12px; background: color-mix(in srgb, var(--accent-bg-color) 12%, transparent); } .bulk-bar { padding: 6px 8px; border-radius: 12px; background: color-mix(in srgb, var(--accent-bg-color) 12%, transparent); }
+41 -163
View File
@@ -8,6 +8,8 @@ pub(crate) mod events;
pub(crate) mod i18n; pub(crate) mod i18n;
pub(crate) mod pages; pub(crate) mod pages;
pub(crate) mod reload; pub(crate) mod reload;
#[path = "../theme.rs"]
pub(crate) mod theme;
pub(crate) mod widgets; pub(crate) mod widgets;
use compat::*; use compat::*;
@@ -30,6 +32,7 @@ use pages::takeout::*;
use pages::trash::*; use pages::trash::*;
use pages::verify::*; use pages::verify::*;
use reload::{LoadTicket, Loader, replace_items}; use reload::{LoadTicket, Loader, replace_items};
use theme::{load_resources, proton_theme_active, set_proton_theme};
use widgets::compat::*; use widgets::compat::*;
use widgets::details::*; use widgets::details::*;
use widgets::file_list::*; use widgets::file_list::*;
@@ -72,8 +75,8 @@ use pdfs_core::control::{
DirEntry, ErrorKind, ImportSummary, InvitationInfo, JobItem, MappingProgress, PendingOpInfo, DirEntry, ErrorKind, ImportSummary, InvitationInfo, JobItem, MappingProgress, PendingOpInfo,
PhotoItem, PhotoKind, PhotoMonth, PlaceInfo, PublicLinkInfo, RefreshScope, Request, Response, PhotoItem, PhotoKind, PhotoMonth, PlaceInfo, PublicLinkInfo, RefreshScope, Request, Response,
RestorableFolder, RestoreItem, RevisionInfo, SearchHit, ShareEntry, ShareEntryKind, SharedItem, RestorableFolder, RestoreItem, RevisionInfo, SearchHit, ShareEntry, ShareEntryKind, SharedItem,
SyncFolderInfo, SyncPhase, SyncProgress, ThumbnailBuildStatus, Topic, TransferDirection, SyncFolderInfo, SyncIssue, SyncPhase, SyncProgress, ThumbnailBuildStatus, Topic,
TransferItem, send, TransferDirection, TransferItem, send,
}; };
use pdfs_core::mounts::{MountAccess, MountKind, MountMode, MountSpec}; use pdfs_core::mounts::{MountAccess, MountKind, MountMode, MountSpec};
@@ -116,6 +119,9 @@ struct Ui {
/// keyring (a DBus round-trip). Populated at startup and updated only on /// keyring (a DBus round-trip). Populated at startup and updated only on
/// login / logout. `None` = signed out. /// login / logout. `None` = signed out.
session: RefCell<Option<auth::StoredSession>>, session: RefCell<Option<auth::StoredSession>>,
/// Whether this run already asked for the mailbox password of a locked
/// account, so a refresh never re-opens a dialog the user cancelled.
unlock_offered: Cell<bool>,
/// Whether the last refresh saw a live mount daemon. Gates the unpin buttons /// Whether the last refresh saw a live mount daemon. Gates the unpin buttons
/// (which need the daemon to evict + re-hydrate) and every mutating action. /// (which need the daemon to evict + re-hydrate) and every mutating action.
mounted: RefCell<bool>, mounted: RefCell<bool>,
@@ -196,12 +202,20 @@ impl Ui {
} }
fn main() -> glib::ExitCode { fn main() -> glib::ExitCode {
tracing_subscriber::fmt() let verify = std::env::args().nth(1).as_deref() == Some(VERIFY_ARG);
.with_env_filter( let logs = tracing_subscriber::fmt().with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env().unwrap_or_else(|_| "info".into()), tracing_subscriber::EnvFilter::try_from_default_env().unwrap_or_else(|_| "info".into()),
) );
.init(); // The verification window's stdout carries its token back to the app.
if verify {
logs.with_writer(std::io::stderr).init();
} else {
logs.init();
}
i18n::init(); i18n::init();
if verify {
return run_verification_window();
}
// The tray launches the app with arguments ("--page locations", // The tray launches the app with arguments ("--page locations",
// "--confirm-stop"), which a running instance must receive too. // "--confirm-stop"), which a running instance must receive too.
@@ -210,7 +224,7 @@ fn main() -> glib::ExitCode {
.flags(gio::ApplicationFlags::HANDLES_COMMAND_LINE) .flags(gio::ApplicationFlags::HANDLES_COMMAND_LINE)
.build(); .build();
app.connect_startup(|_| { app.connect_startup(|_| {
load_proton_theme(); load_resources("/de/nils/protondrivelinux/style.css");
if let Ok(dirs) = AppDirs::new() { if let Ok(dirs) = AppDirs::new() {
set_proton_theme(dirs.load_config().proton_theme.unwrap_or(false)); set_proton_theme(dirs.load_config().proton_theme.unwrap_or(false));
} }
@@ -287,157 +301,6 @@ pub(crate) fn spawn_tray() {
} }
} }
/// Register the bundled GResources (custom icons, the stylesheet) and load the
/// app stylesheet. The Proton theme is a separate provider, applied by
/// [`set_proton_theme`] from the saved preference.
fn load_proton_theme() {
let bytes = include_bytes!(concat!(env!("OUT_DIR"), "/pdfs.gresource"));
let resource_data = glib::Bytes::from_static(bytes);
if let Ok(resource) = gio::Resource::from_data(&resource_data) {
gio::resources_register(&resource);
} else {
tracing::error!("failed to load gresource bundle");
}
let Some(display) = gtk4::gdk::Display::default() else {
return;
};
let provider = gtk4::CssProvider::new();
provider.load_from_resource("/de/nils/protondrivelinux/style.css");
// Above the user's own gtk.css, which often imports a whole theme: its
// `button { padding: 6px 10px; min-width: 16px }` would otherwise pad every
// photo tile and push the grid past the window. Every rule in the sheet is
// scoped to the app's own classes, so nothing else the user styled changes.
// One above the Proton theme too, so the app's rules refine it.
gtk4::style_context_add_provider_for_display(
&display,
&provider,
gtk4::STYLE_PROVIDER_PRIORITY_USER + 2,
);
gtk4::IconTheme::for_display(&display).add_resource_path("/de/nils/protondrivelinux/icons");
}
thread_local! {
/// The Proton theme provider while it is installed. Kept so turning the
/// preference off can remove exactly what turning it on added.
static PROTON_THEME: RefCell<Option<gtk4::CssProvider>> = const { RefCell::new(None) };
}
/// Whether the Proton theme is on.
fn proton_theme_active() -> bool {
PROTON_THEME.with(|slot| slot.borrow().is_some())
}
/// libadwaita's own stylesheet, bundled in the library.
const ADWAITA_STYLESHEET: &str = "/org/gnome/Adwaita/styles/gtk.css";
/// The Proton palette laid over it, from our own bundle.
const PROTON_PALETTE: &str = "/de/nils/protondrivelinux/proton-theme.css";
/// Make the theme provider's `@media (prefers-color-scheme)` and
/// `(prefers-contrast)` rules match what libadwaita shows, now and whenever it
/// changes. A provider evaluates those itself, and a new one assumes light.
///
/// The properties are set by name because they only exist since GTK 4.20, and
/// the app still runs on older GTK, whose libadwaita has no such rules anyway.
fn follow_style_manager(provider: &gtk4::CssProvider) {
fn set_enum(provider: &gtk4::CssProvider, property: &str, nick: &str) {
let Some(pspec) = provider.find_property(property) else {
return;
};
if let Some(value) = glib::EnumClass::with_type(pspec.value_type())
.and_then(|class| class.to_value_by_nick(nick))
{
provider.set_property_from_value(property, &value);
}
}
fn apply(provider: &gtk4::CssProvider, manager: &adw::StyleManager) {
let scheme = if manager.is_dark() { "dark" } else { "light" };
set_enum(provider, "prefers-color-scheme", scheme);
let contrast = if manager.is_high_contrast() {
"more"
} else {
"no-preference"
};
set_enum(provider, "prefers-contrast", contrast);
}
let manager = adw::StyleManager::default();
apply(provider, &manager);
// Weak, so a provider the preference removed isn't kept alive; its
// handlers then do nothing.
let weak = provider.downgrade();
let weak_dark = weak.clone();
manager.connect_dark_notify(move |manager| {
if let Some(provider) = weak_dark.upgrade() {
apply(&provider, manager);
}
});
manager.connect_high_contrast_notify(move |manager| {
if let Some(provider) = weak.upgrade() {
apply(&provider, manager);
}
});
}
/// Paint the app in the Proton web apps' colours (`on`) or follow the system
/// theme and accent colour.
///
/// Setting libadwaita's colour variables alone isn't enough: a GTK theme the
/// user's `gtk.css` imports (Catppuccin, Fluent, ...) sits above the app and
/// hard-codes its own accent into hundreds of rules. So the theme provider,
/// one above the user's CSS, first resets every property, then re-applies
/// libadwaita's whole stylesheet and finally the Proton palette, which leaves
/// nothing of the user's theme inside the app.
fn set_proton_theme(on: bool) {
let Some(display) = gtk4::gdk::Display::default() else {
return;
};
PROTON_THEME.with(|slot| {
let mut slot = slot.borrow_mut();
match (on, slot.as_ref()) {
(true, None) => {
let stylesheet = |path| {
gio::resources_lookup_data(path, gio::ResourceLookupFlags::NONE)
.map(|bytes| String::from_utf8_lossy(&bytes).into_owned())
};
let palette = stylesheet(PROTON_PALETTE).unwrap_or_default();
let css = match stylesheet(ADWAITA_STYLESHEET) {
// Its images are named relative to the sheet, which a
// string loaded here doesn't have.
Ok(adwaita) => format!(
"* {{ all: unset; }}\n{}\n{palette}",
adwaita.replace(
"url(\"assets/",
"url(\"resource:///org/gnome/Adwaita/styles/assets/"
)
),
// Still recolours everything that follows libadwaita's
// variables; only a custom theme's own rules show through.
Err(e) => {
tracing::warn!("libadwaita stylesheet not found, only recolouring: {e}");
palette
}
};
let provider = gtk4::CssProvider::new();
follow_style_manager(&provider);
provider.load_from_string(&css);
gtk4::style_context_add_provider_for_display(
&display,
&provider,
gtk4::STYLE_PROVIDER_PRIORITY_USER + 1,
);
*slot = Some(provider);
}
(false, Some(provider)) => {
gtk4::style_context_remove_provider_for_display(&display, provider);
*slot = None;
}
_ => {}
}
});
}
/// Build the application window, wire the two pages, kick off the refresh loop, /// Build the application window, wire the two pages, kick off the refresh loop,
/// and present it. /// and present it.
fn build_window(app: &adw::Application) { fn build_window(app: &adw::Application) {
@@ -448,6 +311,11 @@ fn build_window(app: &adw::Application) {
return; return;
} }
}; };
// On a fresh install the daemon has not run yet to make these, and the
// Takeout page keeps its staged archives in the state dir (B194).
if let Err(e) = dirs.ensure() {
tracing::warn!("cannot create the app's directories: {e}");
}
let stack = adw::ViewStack::new(); let stack = adw::ViewStack::new();
let (login_page, login_widgets) = build_login_page(); let (login_page, login_widgets) = build_login_page();
@@ -510,6 +378,7 @@ fn build_window(app: &adw::Application) {
busy: Cell::new(0), busy: Cell::new(0),
opening: RefCell::new(HashSet::new()), opening: RefCell::new(HashSet::new()),
session: RefCell::new(auth::load().ok()), session: RefCell::new(auth::load().ok()),
unlock_offered: Cell::new(false),
mounted: RefCell::new(false), mounted: RefCell::new(false),
sidebar: sidebar_list.clone(), sidebar: sidebar_list.clone(),
nav: split.clone(), nav: split.clone(),
@@ -816,6 +685,15 @@ fn build_window(app: &adw::Application) {
sync_sidebar(&ui_nav); sync_sidebar(&ui_nav);
match st.visible_child_name().as_deref() { match st.visible_child_name().as_deref() {
Some("browser") => load_browser(&ui_nav), Some("browser") => load_browser(&ui_nav),
// The Albums and Places listings are re-read in place. Loading the
// timeline instead would swap it in under the Albums or Places toggle.
Some("gallery") if !ui_nav.gallery.in_collection() => {
match ui_nav.gallery.view_switch.active() {
VIEW_ALBUMS => load_albums(&ui_nav),
VIEW_PLACES => load_places(&ui_nav),
_ => load_gallery(&ui_nav, false),
}
}
Some("gallery") => load_gallery(&ui_nav, false), Some("gallery") => load_gallery(&ui_nav, false),
// Every visit reads the page again. The rows already on screen stay // Every visit reads the page again. The rows already on screen stay
// up while it does, so arriving costs no "Loading…" flash. // up while it does, so arriving costs no "Loading…" flash.
@@ -1345,9 +1223,9 @@ fn install_window_actions(ui: &Rc<Ui>, window: &adw::ApplicationWindow) {
let dialog = adw::AboutDialog::builder() let dialog = adw::AboutDialog::builder()
.application_name(gettext("Proton Drive for Linux")) .application_name(gettext("Proton Drive for Linux"))
.application_icon("io.narl.proton-drive-linux") .application_icon("io.narl.proton-drive-linux")
.version(pdfs_core::config::APP_VERSION) .version(env!("CARGO_PKG_VERSION"))
.developer_name("Nils Pukropp") .developer_name("Nils Pukropp")
.website("https://github.com/narrrl/proton-drive-linux") .website("https://proton-drive.narl.io")
.issue_url("https://github.com/narrrl/proton-drive-linux/issues") .issue_url("https://github.com/narrrl/proton-drive-linux/issues")
.copyright("© 2026 Nils Pukropp (contact@narl.io)") .copyright("© 2026 Nils Pukropp (contact@narl.io)")
.license_type(gtk4::License::MitX11) .license_type(gtk4::License::MitX11)
@@ -1485,8 +1363,8 @@ fn debug_info(ui: &Rc<Ui>) -> String {
format!( format!(
"App version: {}\nUser agent: {}\nlibadwaita: {}.{}.{}\nGTK: {}.{}.{}\n\ "App version: {}\nUser agent: {}\nlibadwaita: {}.{}.{}\nGTK: {}.{}.{}\n\
Mountpoint: {}\nControl socket: {}\nMount service running: {}\n", Mountpoint: {}\nControl socket: {}\nMount service running: {}\n",
pdfs_core::config::APP_VERSION, config.resolved_app_version(),
pdfs_core::config::USER_AGENT, config.resolved_user_agent(),
adw::major_version(), adw::major_version(),
adw::minor_version(), adw::minor_version(),
adw::micro_version(), adw::micro_version(),
@@ -341,6 +341,59 @@ fn detail_label(entry: &ActivityEntry) -> Option<String> {
&[("name", alt)], &[("name", alt)],
)); ));
} }
if let Some((reason, copy)) = detail.split_once("; local changes uploaded as ") {
let args = [("name", copy)];
return Some(match edit_conflict(reason) {
// Translators: {name} is the copy the changes made on this
// computer were uploaded as.
EditConflict::Gone => gettext_f(
"It no longer exists on Proton Drive, so your changes were uploaded as {name}",
&args,
),
// Translators: {name} is the copy the changes made on this
// computer were uploaded as.
EditConflict::Trashed => gettext_f(
"It was moved to the Trash on Proton Drive, so your changes were uploaded as {name}",
&args,
),
// Translators: {name} is the copy the changes made on this
// computer were uploaded as.
EditConflict::Changed => gettext_f(
"It was changed on Proton Drive, so your changes were uploaded as {name}",
&args,
),
// Translators: {name} is the copy the changes made on this
// computer were uploaded as.
EditConflict::Other => gettext_f("Your changes were uploaded as {name}", &args),
});
}
if let Some((reason, path)) = detail.split_once("; local changes kept at ") {
let path = tilde_path(path);
let args = [("path", path.as_str())];
return Some(match edit_conflict(reason) {
// Translators: {path} is the file on this computer that holds the
// changes which could not be uploaded.
EditConflict::Gone => gettext_f(
"It no longer exists on Proton Drive, so your changes were kept in {path}",
&args,
),
// Translators: {path} is the file on this computer that holds the
// changes which could not be uploaded.
EditConflict::Trashed => gettext_f(
"It was moved to the Trash on Proton Drive, so your changes were kept in {path}",
&args,
),
// Translators: {path} is the file on this computer that holds the
// changes which could not be uploaded.
EditConflict::Changed => gettext_f(
"It was changed on Proton Drive, so your changes were kept in {path}",
&args,
),
// Translators: {path} is the file on this computer that holds the
// changes which could not be uploaded.
EditConflict::Other => gettext_f("Your changes were kept in {path}", &args),
});
}
if let Some(base) = detail.strip_prefix("differs from ") { if let Some(base) = detail.strip_prefix("differs from ") {
// Translators: {name} is the file the conflicting copy was made from. // Translators: {name} is the file the conflicting copy was made from.
return Some(gettext_f("Differs from {name}", &[("name", base)])); return Some(gettext_f("Differs from {name}", &[("name", base)]));
@@ -388,6 +441,26 @@ fn detail_label(entry: &ActivityEntry) -> Option<String> {
Some(capitalize(detail)) Some(capitalize(detail))
} }
/// Why a queued edit could not land on its file, from the reason the daemon
/// gives before "; local changes uploaded as" or "; local changes kept at".
enum EditConflict {
Gone,
Trashed,
Changed,
Other,
}
fn edit_conflict(reason: &str) -> EditConflict {
match reason {
"the file no longer exists remotely" => EditConflict::Gone,
"the file was trashed remotely" => EditConflict::Trashed,
r if r.starts_with("the remote revision changed under the queued write") => {
EditConflict::Changed
}
_ => EditConflict::Other,
}
}
/// A sync pass summary such as "2 uploaded, 1 folder(s) created", as a /// A sync pass summary such as "2 uploaded, 1 folder(s) created", as a
/// list of counts. `None` when a part is not one the daemon writes. /// list of counts. `None` when a part is not one the daemon writes.
fn sync_summary(detail: &str) -> Option<String> { fn sync_summary(detail: &str) -> Option<String> {
@@ -596,6 +669,38 @@ mod tests {
assert_eq!(d.details, vec!["1 person can view"]); assert_eq!(d.details, vec!["1 person can view"]);
} }
#[test]
fn an_edit_kept_as_a_copy_says_why_in_words() {
let d = describe(&entry(
ActivityKind::Upload,
"notes.txt",
"the remote revision changed under the queued write \
(based on revision r1, remote now at r2); \
local changes uploaded as notes (sync-conflict 1700000000).txt",
false,
));
assert_eq!(
d.details,
vec![
"It was changed on Proton Drive, so your changes were uploaded as \
notes (sync-conflict 1700000000).txt"
]
);
let d = describe(&entry(
ActivityKind::Upload,
"notes.txt",
"the file was trashed remotely; local changes kept at /var/staging/7",
false,
));
assert_eq!(
d.details,
vec![
"It was moved to the Trash on Proton Drive, so your changes were kept in \
/var/staging/7"
]
);
}
#[test] #[test]
fn unknown_details_show_as_written() { fn unknown_details_show_as_written() {
let d = describe(&entry( let d = describe(&entry(
+271 -44
View File
@@ -832,8 +832,8 @@ fn show_location_column(ui: &Rc<Ui>, visible: bool) {
} }
} }
/// The menu for one entry, in the order of `docs/UI_UX_PLAN.md` §6: open, /// The menu for one entry, in a fixed order: open, offline, sharing,
/// offline, sharing, organising, and the destructive item last on its own. /// organising, and the destructive item last on its own.
pub(crate) fn entry_context_menu(ui: &Rc<Ui>, entry: &DirEntry) -> ActionMenu { pub(crate) fn entry_context_menu(ui: &Rc<Ui>, entry: &DirEntry) -> ActionMenu {
let mut menu = ActionMenu::new(); let mut menu = ActionMenu::new();
let (ui_c, entry_c) = (ui.clone(), entry.clone()); let (ui_c, entry_c) = (ui.clone(), entry.clone());
@@ -1920,6 +1920,10 @@ pub(crate) fn prompt_rename(ui: &Rc<Ui>, entry: &DirEntry) {
/// Pick a destination folder by browsing Drive's folders, and move `entries` /// Pick a destination folder by browsing Drive's folders, and move `entries`
/// there through the daemon. /// there through the daemon.
///
/// Besides My files the picker offers every synced folder. A move into one is
/// still made on Proton Drive, so nothing is downloaded or uploaded again; it
/// is sent with absolute paths, which is how the daemon tells locations apart.
pub(crate) fn prompt_move(ui: &Rc<Ui>, entries: Vec<DirEntry>) { pub(crate) fn prompt_move(ui: &Rc<Ui>, entries: Vec<DirEntry>) {
if entries.is_empty() { if entries.is_empty() {
return; return;
@@ -1967,6 +1971,7 @@ pub(crate) fn prompt_move(ui: &Rc<Ui>, entries: Vec<DirEntry>) {
dialog.set_default_response(Some("move")); dialog.set_default_response(Some("move"));
dialog.set_close_response("cancel"); dialog.set_close_response("cancel");
let mountpoint = ui.dirs.resolved_mountpoint(&ui.dirs.load_config());
let picker = Rc::new(MovePicker { let picker = Rc::new(MovePicker {
ui: ui.clone(), ui: ui.clone(),
dialog: dialog.clone(), dialog: dialog.clone(),
@@ -1974,28 +1979,93 @@ pub(crate) fn prompt_move(ui: &Rc<Ui>, entries: Vec<DirEntry>) {
list, list,
up: up.clone(), up: up.clone(),
sources: sources.clone(), sources: sources.clone(),
folder: RefCell::new(String::new()), folder: RefCell::new(MoveTarget::Locations),
generation: Cell::new(0), generation: Cell::new(0),
}); });
let picker_up = picker.clone(); let picker_up = picker.clone();
up.connect_clicked(move |_| { up.connect_clicked(move |_| {
let folder = picker_up.folder.borrow().clone(); let parent = picker_up.folder.borrow().parent();
let parent = folder.rfind('/').map(|i| &folder[..i]).unwrap_or_default(); picker_up.open(parent);
picker_up.open(parent.to_string());
}); });
// Start where the browser is: moving is most often one level up or down. // Start where the browser is: moving is most often one level up or down.
picker.open(ui.browser.path.borrow().clone()); picker.open(MoveTarget::Drive(ui.browser.path.borrow().clone()));
let ui = ui.clone(); let ui = ui.clone();
let picker_done = picker.clone(); let picker_done = picker.clone();
dialog.connect_response(None, move |_, resp| { dialog.connect_response(None, move |_, resp| {
if resp == "move" { if resp != "move" {
run_bulk_move(&ui, sources.clone(), picker_done.folder.borrow().clone()); return;
}
match picker_done.folder.borrow().clone() {
MoveTarget::Locations => {}
MoveTarget::Drive(folder) => run_bulk_move(&ui, sources.clone(), folder),
MoveTarget::Synced { path, .. } => {
let absolute = sources
.iter()
.map(|rel| mountpoint.join(rel).to_string_lossy().into_owned())
.collect();
run_bulk_move(&ui, absolute, path);
}
} }
}); });
dialog.present(ui_window(&picker.ui).as_ref()); dialog.present(ui_window(&picker.ui).as_ref());
} }
/// Where the Move dialog is looking, and the destination if the user confirms.
#[derive(Clone, Debug, PartialEq)]
pub(crate) enum MoveTarget {
/// The list of places to move to: My files and each synced folder.
Locations,
/// A folder in My files, mountpoint-relative; empty for My files itself.
Drive(String),
/// A folder inside a synced folder, as an absolute local path. `root` is
/// the synced folder. A mirror folder is a plain directory, read from disk.
Synced {
root: String,
path: String,
mirror: bool,
},
}
impl MoveTarget {
/// One level up. The top of a location leads back to the list of them.
pub(crate) fn parent(&self) -> MoveTarget {
match self {
MoveTarget::Locations => MoveTarget::Locations,
MoveTarget::Drive(folder) if folder.is_empty() => MoveTarget::Locations,
MoveTarget::Drive(folder) => MoveTarget::Drive(
folder
.rfind('/')
.map(|i| folder[..i].to_string())
.unwrap_or_default(),
),
MoveTarget::Synced { root, path, .. } if path == root => MoveTarget::Locations,
MoveTarget::Synced { root, path, mirror } => MoveTarget::Synced {
root: root.clone(),
path: Path::new(path)
.parent()
.map(|p| p.to_string_lossy().into_owned())
.unwrap_or_else(|| root.clone()),
mirror: *mirror,
},
}
}
/// `name` inside this folder, in the same form.
fn child(&self, name: &str) -> MoveTarget {
match self {
MoveTarget::Locations => MoveTarget::Locations,
MoveTarget::Drive(folder) if folder.is_empty() => MoveTarget::Drive(name.to_string()),
MoveTarget::Drive(folder) => MoveTarget::Drive(format!("{folder}/{name}")),
MoveTarget::Synced { root, path, mirror } => MoveTarget::Synced {
root: root.clone(),
path: Path::new(path).join(name).to_string_lossy().into_owned(),
mirror: *mirror,
},
}
}
}
/// The Move dialog's folder browser. /// The Move dialog's folder browser.
struct MovePicker { struct MovePicker {
ui: Rc<Ui>, ui: Rc<Ui>,
@@ -2006,71 +2076,81 @@ struct MovePicker {
/// What is being moved, mountpoint-relative. /// What is being moved, mountpoint-relative.
sources: Vec<String>, sources: Vec<String>,
/// The folder on show, and the destination if the user confirms now. /// The folder on show, and the destination if the user confirms now.
folder: RefCell<String>, folder: RefCell<MoveTarget>,
/// Drops a listing that arrives after the user went elsewhere. /// Drops a listing that arrives after the user went elsewhere.
generation: Cell<u64>, generation: Cell<u64>,
} }
impl MovePicker { impl MovePicker {
fn open(self: &Rc<Self>, folder: String) { fn open(self: &Rc<Self>, folder: MoveTarget) {
let generation = self.generation.get().wrapping_add(1); let generation = self.generation.get().wrapping_add(1);
self.generation.set(generation); self.generation.set(generation);
self.location.set_label(if folder.is_empty() { let (label, tooltip) = match &folder {
"Proton Drive" MoveTarget::Locations => (gettext("Locations"), String::new()),
} else { MoveTarget::Drive(rel) if rel.is_empty() => (gettext("My files"), String::new()),
&folder MoveTarget::Drive(rel) => (rel.clone(), rel.clone()),
}); MoveTarget::Synced { path, .. } => (path.clone(), path.clone()),
self.location.set_tooltip_text(Some(&folder)); };
self.up.set_sensitive(!folder.is_empty()); self.location.set_label(&label);
self.dialog self.location.set_tooltip_text(Some(&tooltip));
.set_response_enabled("move", move_target_allowed(&self.sources, &folder)); self.up
.set_sensitive(!matches!(folder, MoveTarget::Locations));
let allowed = match &folder {
MoveTarget::Locations => false,
MoveTarget::Drive(rel) => move_target_allowed(&self.sources, rel),
// What is moved is in My files, so it cannot be in a synced folder.
MoveTarget::Synced { .. } => true,
};
self.dialog.set_response_enabled("move", allowed);
*self.folder.borrow_mut() = folder.clone(); *self.folder.borrow_mut() = folder.clone();
while let Some(row) = self.list.first_child() { while let Some(row) = self.list.first_child() {
self.list.remove(&row); self.list.remove(&row);
} }
self.list.append(&picker_note(&gettext("Loading…"))); self.list.append(&picker_note(&gettext("Loading…")));
let rx = spawn_request(
self.ui.dirs.control_socket(),
Request::ListDir {
path: folder.clone(),
},
);
let picker = self.clone(); let picker = self.clone();
glib::spawn_future_local(async move { glib::spawn_future_local(async move {
let result = rx.recv().await; let listed = match &folder {
MoveTarget::Locations => picker.locations().await,
MoveTarget::Drive(rel) => picker.drive_folders(&folder, rel.clone()).await,
MoveTarget::Synced {
path, mirror: true, ..
} => local_folders(&folder, path.clone()).await,
MoveTarget::Synced { path, .. } => {
picker.drive_folders(&folder, path.clone()).await
}
};
if picker.generation.get() != generation { if picker.generation.get() != generation {
return; return;
} }
while let Some(row) = picker.list.first_child() { while let Some(row) = picker.list.first_child() {
picker.list.remove(&row); picker.list.remove(&row);
} }
let Ok(Ok(Response::Entries { entries })) = result else { let Some(rows) = listed else {
picker picker
.list .list
.append(&picker_note(&gettext("Couldn't read this folder."))); .append(&picker_note(&gettext("Couldn't read this folder.")));
return; return;
}; };
let mut folders: Vec<DirEntry> = entries.into_iter().filter(|e| e.is_dir).collect();
folders.sort_by_key(|e| e.name.to_lowercase());
let mut shown = 0; let mut shown = 0;
for entry in folders { for (title, subtitle, target) in rows {
let path = if folder.is_empty() {
entry.name.clone()
} else {
format!("{folder}/{}", entry.name)
};
// A folder cannot go inside itself, so do not offer to open it. // A folder cannot go inside itself, so do not offer to open it.
if picker.sources.contains(&path) { if let MoveTarget::Drive(path) = &target
&& picker.sources.contains(path)
{
continue; continue;
} }
let row = adw::ActionRow::builder() let row = adw::ActionRow::builder()
.title(glib::markup_escape_text(&entry.name).as_str()) .title(glib::markup_escape_text(&title).as_str())
.activatable(true) .activatable(true)
.build(); .build();
if let Some(subtitle) = subtitle {
row.set_subtitle(glib::markup_escape_text(&subtitle).as_str());
}
row.add_prefix(&gtk4::Image::from_icon_name("folder-symbolic")); row.add_prefix(&gtk4::Image::from_icon_name("folder-symbolic"));
row.add_suffix(&gtk4::Image::from_icon_name("go-next-symbolic")); row.add_suffix(&gtk4::Image::from_icon_name("go-next-symbolic"));
let open = picker.clone(); let open = picker.clone();
row.connect_activated(move |_| open.open(path.clone())); row.connect_activated(move |_| open.open(target.clone()));
picker.list.append(&row); picker.list.append(&row);
shown += 1; shown += 1;
} }
@@ -2081,6 +2161,92 @@ impl MovePicker {
} }
}); });
} }
/// My files and every synced folder, as rows of the top level.
async fn locations(&self) -> Option<Vec<PickerRow>> {
let mut rows = vec![(gettext("My files"), None, MoveTarget::Drive(String::new()))];
let rx = spawn_request(self.ui.dirs.control_socket(), Request::ListSyncFolders);
let Ok(Ok(Response::SyncFolders { mut items })) = rx.recv().await else {
// My files alone is still somewhere to move to.
return Some(rows);
};
items.sort_by_key(|f| f.local_path.to_lowercase());
for folder in items {
let name = Path::new(&folder.local_path)
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_else(|| folder.local_path.clone());
rows.push((
name,
Some(folder.local_path.clone()),
MoveTarget::Synced {
root: folder.local_path.clone(),
path: folder.local_path,
mirror: folder.mode == "mirror",
},
));
}
Some(rows)
}
/// The folders in a mounted folder: `folder` is mountpoint-relative for My
/// files and absolute for an on-demand synced folder.
async fn drive_folders(&self, here: &MoveTarget, folder: String) -> Option<Vec<PickerRow>> {
let rx = spawn_request(
self.ui.dirs.control_socket(),
Request::ListDir {
path: folder.clone(),
},
);
let Ok(Ok(Response::Entries { entries })) = rx.recv().await else {
return None;
};
let mut names: Vec<String> = entries
.into_iter()
.filter(|e| e.is_dir)
.map(|e| e.name)
.collect();
names.sort_by_key(|name| name.to_lowercase());
Some(
names
.into_iter()
.map(|name| {
let target = here.child(&name);
(name, None, target)
})
.collect(),
)
}
}
/// One row of the picker: its title, an optional subtitle, and where it leads.
type PickerRow = (String, Option<String>, MoveTarget);
/// The folders in a mirror folder's directory `path`, read from disk off the
/// main thread. A mirror folder has no mount for `ListDir` to read.
async fn local_folders(here: &MoveTarget, path: String) -> Option<Vec<PickerRow>> {
let names = gio::spawn_blocking(move || {
let mut names: Vec<String> = std::fs::read_dir(&path)
.ok()?
.flatten()
.filter(|e| e.file_type().is_ok_and(|t| t.is_dir()))
.filter_map(|e| e.file_name().into_string().ok())
.filter(|name| !name.contains(".pdfs-tmp-"))
.collect();
names.sort_by_key(|name| name.to_lowercase());
Some(names)
})
.await
.ok()??;
Some(
names
.into_iter()
.map(|name| {
let target = here.child(&name);
(name, None, target)
})
.collect(),
)
} }
fn picker_note(text: &str) -> gtk4::ListBoxRow { fn picker_note(text: &str) -> gtk4::ListBoxRow {
@@ -2329,9 +2495,13 @@ pub(crate) fn start_upload(ui: &Rc<Ui>, sources: Vec<String>) {
} }
/// The sync-state badge for an entry: `(icon, css-class)`, or `None` for folders /// The sync-state badge for an entry: `(icon, css-class)`, or `None` for folders
/// (which carry no per-file cache state). Pinned (kept offline) ranks above merely /// (which carry no per-file cache state). A sync issue ranks first, on folders
/// too: the change waits for the user. Pinned (kept offline) ranks above merely
/// cached (downloaded, evictable); everything else is online-only. /// cached (downloaded, evictable); everything else is online-only.
pub(crate) fn badge_for(entry: &DirEntry) -> Option<(&'static str, &'static str)> { pub(crate) fn badge_for(entry: &DirEntry) -> Option<(&'static str, &'static str)> {
if entry.issue.is_some() {
return Some(("dialog-warning-symbolic", "badge-issue"));
}
if entry.is_dir { if entry.is_dir {
return None; return None;
} }
@@ -2347,9 +2517,11 @@ pub(crate) fn badge_for(entry: &DirEntry) -> Option<(&'static str, &'static str)
/// Paint `badge` to reflect the entry's sync state (see [`badge_for`]). Clears any /// Paint `badge` to reflect the entry's sync state (see [`badge_for`]). Clears any
/// prior colour class first, since list factories recycle cells. /// prior colour class first, since list factories recycle cells.
pub(crate) fn apply_badge(badge: &gtk4::Image, entry: &DirEntry) { pub(crate) fn apply_badge(badge: &gtk4::Image, entry: &DirEntry) {
for class in ["badge-pinned", "badge-cached", "badge-cloud"] { for class in ["badge-pinned", "badge-cached", "badge-cloud", "badge-issue"] {
badge.remove_css_class(class); badge.remove_css_class(class);
} }
let issue = entry.issue.map(super::locations::issue_text);
badge.set_tooltip_text(issue.as_deref());
match badge_for(entry) { match badge_for(entry) {
Some((icon, class)) => { Some((icon, class)) => {
badge.set_icon_name(Some(icon)); badge.set_icon_name(Some(icon));
@@ -2749,6 +2921,7 @@ pub(crate) fn repaint_search(ui: &Rc<Ui>, hits: &[SearchHit]) {
shared_by_unverified: false, shared_by_unverified: false,
trashed_at: 0, trashed_at: 0,
trashed_from: None, trashed_from: None,
issue: None,
}) })
.collect(); .collect();
sort_entries(&mut entries, ui.browser.view.get()); sort_entries(&mut entries, ui.browser.view.get());
@@ -2786,11 +2959,11 @@ fn listing_summary(files: usize, folders: usize, file_bytes: u64) -> String {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::{ use super::{
hit_location, listing_summary, move_target_allowed, show_thumbnail_build_progress, MoveTarget, badge_for, hit_location, listing_summary, move_target_allowed,
sort_entries, show_thumbnail_build_progress, sort_entries,
}; };
use pdfs_core::config::{FileSort, FilesView}; use pdfs_core::config::{FileSort, FilesView};
use pdfs_core::control::{DirEntry, ThumbnailBuildStatus}; use pdfs_core::control::{DirEntry, SyncIssue, ThumbnailBuildStatus};
fn entry(name: &str, is_dir: bool, size: u64, modified: i64) -> DirEntry { fn entry(name: &str, is_dir: bool, size: u64, modified: i64) -> DirEntry {
DirEntry { DirEntry {
@@ -2808,6 +2981,7 @@ mod tests {
shared_by_unverified: false, shared_by_unverified: false,
trashed_at: 0, trashed_at: 0,
trashed_from: None, trashed_from: None,
issue: None,
} }
} }
@@ -2815,6 +2989,20 @@ mod tests {
entries.iter().map(|e| e.name.as_str()).collect() entries.iter().map(|e| e.name.as_str()).collect()
} }
#[test]
fn a_sync_issue_shows_on_the_node_before_its_cache_state() {
let mut file = entry("a.txt", false, 1, 0);
file.pinned = true;
assert_eq!(badge_for(&file).unwrap().1, "badge-pinned");
file.issue = Some(SyncIssue::Quota);
assert_eq!(badge_for(&file).unwrap().1, "badge-issue");
let mut folder = entry("d", true, 0, 0);
assert_eq!(badge_for(&folder), None);
folder.issue = Some(SyncIssue::Missing);
assert_eq!(badge_for(&folder).unwrap().1, "badge-issue");
}
#[test] #[test]
fn a_search_hit_location_is_its_parent_folder() { fn a_search_hit_location_is_its_parent_folder() {
assert_eq!(hit_location("Work/Old/a.txt"), "Work/Old"); assert_eq!(hit_location("Work/Old/a.txt"), "Work/Old");
@@ -2880,4 +3068,43 @@ mod tests {
status.running = false; status.running = false;
assert!(!show_thumbnail_build_progress(&status)); assert!(!show_thumbnail_build_progress(&status));
} }
#[test]
fn move_picker_climbs_from_a_location_top_to_the_list_of_locations() {
assert_eq!(
MoveTarget::Drive(String::new()).parent(),
MoveTarget::Locations
);
let root = MoveTarget::Synced {
root: "/home/u/Docs".into(),
path: "/home/u/Docs".into(),
mirror: true,
};
assert_eq!(root.parent(), MoveTarget::Locations);
}
#[test]
fn move_picker_climbs_one_folder_at_a_time_inside_a_location() {
assert_eq!(
MoveTarget::Drive("a/b".into()).parent(),
MoveTarget::Drive("a".into())
);
assert_eq!(
MoveTarget::Drive("a".into()).parent(),
MoveTarget::Drive(String::new())
);
let inner = MoveTarget::Synced {
root: "/home/u/Docs".into(),
path: "/home/u/Docs/x/y".into(),
mirror: false,
};
assert_eq!(
inner.parent(),
MoveTarget::Synced {
root: "/home/u/Docs".into(),
path: "/home/u/Docs/x".into(),
mirror: false,
}
);
}
} }
+167 -12
View File
@@ -7,7 +7,7 @@
//! `~/ProtonDrive` mount, plus each folder this computer backs up, whether it is //! `~/ProtonDrive` mount, plus each folder this computer backs up, whether it is
//! a mirrored local directory or an on-demand FUSE session. The page is called //! a mirrored local directory or an on-demand FUSE session. The page is called
//! *Locations* rather than *Mounts* because a mirror folder is a plain directory //! *Locations* rather than *Mounts* because a mirror folder is a plain directory
//! with no FUSE session behind it (mount-architecture.md §4). //! with no FUSE session behind it.
//! //!
//! The device rows drive the same control requests the Computers page used to: //! The device rows drive the same control requests the Computers page used to:
//! mode switch, sync now, remove. What is new here is that the primary mount is //! mode switch, sync now, remove. What is new here is that the primary mount is
@@ -70,8 +70,9 @@ pub(crate) struct LocationRow {
progress: gtk4::ProgressBar, progress: gtk4::ProgressBar,
} }
/// What a queue row shows that can change: id, attempts, next attempt, parked. /// What a queue row shows that can change: id, attempts, next attempt, parked,
pub(crate) type QueueKey = (i64, i64, Option<i64>, bool); /// issue.
pub(crate) type QueueKey = (i64, i64, Option<i64>, bool, Option<SyncIssue>);
/// The "Waiting to Upload" list: the daemon's pending-op queue. /// The "Waiting to Upload" list: the daemon's pending-op queue.
pub(crate) struct QueueState { pub(crate) struct QueueState {
@@ -262,20 +263,22 @@ pub(crate) fn build_locations_page(history: &gtk4::Widget) -> (gtk4::Widget, Loc
(frame.upcast(), widgets) (frame.upcast(), widgets)
} }
/// `child` in a clamp that scrolls vertically only, with page margins. /// `child` across the page's full width, scrolling vertically only, with page
/// margins.
fn scrolled_column(child: &impl IsA<gtk4::Widget>) -> gtk4::ScrolledWindow { fn scrolled_column(child: &impl IsA<gtk4::Widget>) -> gtk4::ScrolledWindow {
let clamp = adw::Clamp::builder().child(child).build(); let column = gtk4::Box::new(gtk4::Orientation::Vertical, 0);
clamp.set_margin_top(18); column.append(child);
clamp.set_margin_bottom(18); column.set_margin_top(18);
clamp.set_margin_start(18); column.set_margin_bottom(18);
clamp.set_margin_end(18); column.set_margin_start(18);
column.set_margin_end(18);
// Never scroll sideways: a location's subtitle holds a full path, and letting // Never scroll sideways: a location's subtitle holds a full path, and letting
// the row grow to fit one pushes the controls at its end off screen. // the row grow to fit one pushes the controls at its end off screen.
// Constrained, the path ellipsizes and the controls stay reachable. // Constrained, the path ellipsizes and the controls stay reachable.
gtk4::ScrolledWindow::builder() gtk4::ScrolledWindow::builder()
.vexpand(true) .vexpand(true)
.hscrollbar_policy(gtk4::PolicyType::Never) .hscrollbar_policy(gtk4::PolicyType::Never)
.child(&clamp) .child(&column)
.build() .build()
} }
@@ -589,7 +592,7 @@ fn repaint_queue(ui: &Rc<Ui>, items: &[PendingOpInfo]) {
items.sort_by_key(|op| (!op.failing, op.attempts == 0, op.id)); items.sort_by_key(|op| (!op.failing, op.attempts == 0, op.id));
let key: Vec<QueueKey> = items let key: Vec<QueueKey> = items
.iter() .iter()
.map(|op| (op.id, op.attempts, op.next_attempt_at, op.parked)) .map(|op| (op.id, op.attempts, op.next_attempt_at, op.parked, op.issue))
.collect(); .collect();
if *queue.painted.borrow() == key { if *queue.painted.borrow() == key {
return; return;
@@ -646,6 +649,10 @@ fn queue_row(ui: &Rc<Ui>, op: &PendingOpInfo, now: i64) -> adw::ActionRow {
"{action} · {state}", "{action} · {state}",
&[("action", &action), ("state", &state)], &[("action", &action), ("state", &state)],
); );
if let Some(issue) = op.issue {
subtitle.push('\n');
subtitle.push_str(&issue_text(issue));
}
if let Some(error) = &op.last_error { if let Some(error) = &op.last_error {
subtitle.push('\n'); subtitle.push('\n');
// Translators: {n} is how many attempts failed, {error} the last error message. // Translators: {n} is how many attempts failed, {error} the last error message.
@@ -660,7 +667,7 @@ fn queue_row(ui: &Rc<Ui>, op: &PendingOpInfo, now: i64) -> adw::ActionRow {
.title(glib::markup_escape_text(&op.path).as_str()) .title(glib::markup_escape_text(&op.path).as_str())
.subtitle(glib::markup_escape_text(&subtitle).as_str()) .subtitle(glib::markup_escape_text(&subtitle).as_str())
.title_lines(1) .title_lines(1)
.subtitle_lines(3) .subtitle_lines(if op.issue.is_some() { 5 } else { 3 })
.tooltip_text(&op.path) .tooltip_text(&op.path)
.build(); .build();
let image = gtk4::Image::from_icon_name(if op.failing { let image = gtk4::Image::from_icon_name(if op.failing {
@@ -685,9 +692,157 @@ fn queue_row(ui: &Rc<Ui>, op: &PendingOpInfo, now: i64) -> adw::ActionRow {
retry.connect_clicked(move |_| retry_queued(&ui, Some(id))); retry.connect_clicked(move |_| retry_queued(&ui, Some(id)));
row.add_suffix(&retry); row.add_suffix(&retry);
} }
if op.failing && op.exportable {
let export = gtk4::Button::builder()
.label(gettext("Export"))
.tooltip_text(gettext("Save a copy of this file's content elsewhere"))
.valign(gtk4::Align::Center)
.build();
export.add_css_class("flat");
let ui = ui.clone();
let id = op.id;
let name = Path::new(&op.path)
.file_name()
.map(|name| name.to_string_lossy().into_owned())
.unwrap_or_default();
export.connect_clicked(move |_| prompt_export_queued(&ui, id, &name));
row.add_suffix(&export);
}
if op.failing {
let discard = gtk4::Button::builder()
.label(gettext("Discard"))
.tooltip_text(gettext("Undo this change here instead of sending it"))
.valign(gtk4::Align::Center)
.build();
discard.add_css_class("flat");
let ui = ui.clone();
let id = op.id;
let path = op.path.clone();
let exportable = op.exportable;
discard.connect_clicked(move |_| prompt_discard_queued(&ui, id, &path, exportable));
row.add_suffix(&discard);
}
row row
} }
/// What a sync issue means and what the user can do about it. The change stays
/// queued either way.
pub(crate) fn issue_text(issue: SyncIssue) -> String {
match issue {
SyncIssue::Quota => gettext(
"Your Proton Drive storage is full. Free up space or upgrade your plan, and this change uploads on its own.",
),
SyncIssue::Access => gettext(
"You can no longer change this on Proton Drive. Ask the owner for edit access, or export the file to keep your version.",
),
SyncIssue::Missing => gettext(
"The folder this change belongs in is gone from Proton Drive. Export the file to keep your version.",
),
SyncIssue::Limit => gettext(
"Proton Drive's limit on items in a folder or on folder depth is reached. Make room in the folder, then retry.",
),
SyncIssue::Rejected | SyncIssue::Unknown => {
gettext("Proton Drive refused this change. It stays queued and is tried again.")
}
}
}
/// Ask where to save a copy of a queued file's content, then have the daemon
/// write it there. The change itself stays queued.
fn prompt_export_queued(ui: &Rc<Ui>, id: i64, name: &str) {
let window = ui_window(ui);
let dialog = gtk4::FileDialog::builder()
.title(gettext("Export File"))
.initial_name(name)
.build();
let ui = ui.clone();
dialog.save(window.as_ref(), gio::Cancellable::NONE, move |res| {
let Ok(file) = res else { return };
let Some(dest) = file.path().and_then(|p| p.to_str().map(str::to_string)) else {
toast_error(
&ui,
&gettext("Couldn't export the file"),
&gettext("That location isn't a local file."),
);
return;
};
let rx = spawn_request(
ui.dirs.control_socket(),
Request::ExportPendingOp { id, dest },
);
let ui = ui.clone();
glib::spawn_future_local(async move {
match rx.recv().await {
Ok(Ok(Response::Ok { .. })) => toast(&ui, &gettext("File exported")),
Ok(Ok(Response::Error { message, kind })) => {
toast_failure(&ui, &gettext("Couldn't export the file"), &message, kind)
}
_ => toast_error(
&ui,
&gettext("Couldn't export the file"),
&gettext("The Proton Drive service didn't respond."),
),
}
});
});
}
/// Confirm, then have the daemon drop a queued change and undo it here.
fn prompt_discard_queued(ui: &Rc<Ui>, id: i64, path: &str, exportable: bool) {
let body = if exportable {
// Translators: {path} is the file whose queued change would be dropped.
gettext_f(
"Your changes to {path} are undone here and never reach Proton Drive. Export the file first to keep your version.",
&[("path", path)],
)
} else {
// Translators: {path} is the file or folder whose queued change would be dropped.
gettext_f(
"The change to {path} is undone here and never reaches Proton Drive.",
&[("path", path)],
)
};
let dialog = adw::AlertDialog::builder()
.heading(gettext("Discard Change?"))
.body(body)
.build();
dialog.add_response("cancel", &gettext("Cancel"));
dialog.add_response("discard", &gettext("Discard"));
dialog.set_response_appearance("discard", adw::ResponseAppearance::Destructive);
dialog.set_default_response(Some("cancel"));
dialog.set_close_response("cancel");
let window = ui_window(ui);
let ui = ui.clone();
dialog.connect_response(None, move |_, resp| {
if resp == "discard" {
discard_queued(&ui, id);
}
});
dialog.present(window.as_ref());
}
fn discard_queued(ui: &Rc<Ui>, id: i64) {
let rx = spawn_request(ui.dirs.control_socket(), Request::DiscardPendingOp { id });
let ui = ui.clone();
glib::spawn_future_local(async move {
match rx.recv().await {
Ok(Ok(Response::Ok { .. })) => {
toast(&ui, &gettext("Change discarded"));
ui.locations.queue.painted.borrow_mut().clear();
refresh_queue(&ui);
}
Ok(Ok(Response::Error { message, kind })) => {
toast_failure(&ui, &gettext("Couldn't discard the change"), &message, kind)
}
_ => toast_error(
&ui,
&gettext("Couldn't discard the change"),
&gettext("The Proton Drive service didn't respond."),
),
}
});
}
/// Ask the daemon to stop waiting out one op's backoff, or every failed op's. /// Ask the daemon to stop waiting out one op's backoff, or every failed op's.
fn retry_queued(ui: &Rc<Ui>, id: Option<i64>) { fn retry_queued(ui: &Rc<Ui>, id: Option<i64>) {
let rx = spawn_request(ui.dirs.control_socket(), Request::RetryPendingOp { id }); let rx = spawn_request(ui.dirs.control_socket(), Request::RetryPendingOp { id });
+286 -32
View File
@@ -1,5 +1,5 @@
use crate::*; use crate::*;
use pdfs_core::proton_sdk::api::HumanVerificationCredential; use pdfs_core::proton_sdk::api::{HumanVerification, HumanVerificationCredential};
pub(crate) struct LoginState { pub(crate) struct LoginState {
// Login page. // Login page.
@@ -128,7 +128,12 @@ pub(crate) fn wire_login(ui: &Rc<Ui>) {
set_signing_in(&ui, true); set_signing_in(&ui, true);
ui.login.login_status.set_text(&gettext("Signing in…")); ui.login.login_status.set_text(&gettext("Signing in…"));
let (rx, totp_req_rx, hv_req_rx) = spawn_login(username, password); let LoginChannels {
result: rx,
totp: totp_req_rx,
hv: hv_req_rx,
mailbox: mailbox_req_rx,
} = spawn_login(username, password);
// Surface the 2FA dialog only if the SDK actually asks for a code (i.e. // Surface the 2FA dialog only if the SDK actually asks for a code (i.e.
// the account has 2FA enabled). The worker blocks until the dialog feeds // the account has 2FA enabled). The worker blocks until the dialog feeds
@@ -145,15 +150,40 @@ pub(crate) fn wire_login(ui: &Rc<Ui>) {
}); });
// Same shape for human verification: only fires when the API gates the // Same shape for human verification: only fires when the API gates the
// sign-in, and the worker is blocked on the token until it does. // sign-in, and the worker is blocked on the credential until it does.
let ui_hv = ui.clone(); let ui_hv = ui.clone();
glib::spawn_future_local(async move { glib::spawn_future_local(async move {
if let Ok((url, token_tx)) = hv_req_rx.recv().await { if let Ok((hv, credential_tx)) = hv_req_rx.recv().await {
ui_hv ui_hv
.login .login
.login_status .login_status
.set_text(&gettext("Complete the verification to continue…")); .set_text(&gettext("Complete the verification to continue…"));
prompt_human_verification(&ui_hv, &url, token_tx); // Anything but a credential drops the sender, which the worker
// reads as a cancelled sign-in.
match verify_human(hv.verification_url()).await {
Verification::Solved(token) => {
let _ = credential_tx.send(HumanVerificationCredential::captcha(token));
}
Verification::Cancelled => {}
// The web view could not start, most likely its sandbox
// (B195): the user's browser can show the page instead.
Verification::Failed => verify_in_browser(&ui_hv, &hv, credential_tx),
}
}
});
// And for the mailbox password, which only a two-password account has.
// The worker asks again after a wrong answer, so this loops too.
let ui_mailbox = ui.clone();
glib::spawn_future_local(async move {
while let Ok((attempt, answer_tx)) = mailbox_req_rx.recv().await {
ask_mailbox(&ui_mailbox, attempt > 0, move |answer| {
// A cancelled dialog drops the sender, which the worker
// reads as a cancelled login.
if let Some(answer) = answer {
let _ = answer_tx.send(answer);
}
});
} }
}); });
@@ -209,6 +239,12 @@ pub(crate) fn login_error_message(error: &pdfs_core::Error) -> String {
pdfs_core::Error::Other(message) if message.contains("two-factor") => { pdfs_core::Error::Other(message) if message.contains("two-factor") => {
gettext("Sign-in cancelled: no two-factor code was entered.") gettext("Sign-in cancelled: no two-factor code was entered.")
} }
pdfs_core::Error::Other(message) if message.contains("mailbox") => {
gettext("Sign-in cancelled: no mailbox password was entered.")
}
pdfs_core::Error::WrongMailboxPassword => {
gettext("That mailbox password didn't unlock your account.")
}
pdfs_core::Error::Other(message) if message.contains("verification") => gettext( pdfs_core::Error::Other(message) if message.contains("verification") => gettext(
"Sign-in cancelled: the verification wasn't completed. Sign in again to get a new one.", "Sign-in cancelled: the verification wasn't completed. Sign in again to get a new one.",
), ),
@@ -220,26 +256,34 @@ pub(crate) fn login_error_message(error: &pdfs_core::Error) -> String {
} }
} }
/// What the login worker asks the UI for, and how it reports back.
pub(crate) struct LoginChannels {
/// The final outcome, once.
pub(crate) result: async_channel::Receiver<Result<(), String>>,
/// Fires only if the account needs a 2FA code; carries the sender for it.
pub(crate) totp: async_channel::Receiver<std::sync::mpsc::Sender<String>>,
/// Fires only if the sign-in is gated: the challenge, and the sender for the
/// credential the user earns by solving it.
pub(crate) hv: async_channel::Receiver<(
HumanVerification,
std::sync::mpsc::Sender<HumanVerificationCredential>,
)>,
/// Fires only for a two-password account: how many answers were already
/// wrong, and the sender for the password and the "remember" choice.
pub(crate) mailbox: async_channel::Receiver<(u32, std::sync::mpsc::Sender<(String, bool)>)>,
}
/// Run the async SRP + optional 2FA login on a dedicated current-thread Tokio /// Run the async SRP + optional 2FA login on a dedicated current-thread Tokio
/// runtime. Returns two channels: the first yields the final login result once; /// runtime. The returned [`LoginChannels`] carry the final result and the lazy
/// the second fires *only if* the account needs a 2FA code, carrying a /// prompts. Each login closure blocks the worker on a sender the UI answers, so
/// [`std::sync::mpsc::Sender`] the UI uses to feed the code back. The login /// a code is requested lazily and can't expire before the password proof.
/// closure blocks the worker on that sender until the dialog answers, so the pub(crate) fn spawn_login(username: String, password: String) -> LoginChannels {
/// code is requested lazily and can't expire before the password proof.
#[allow(clippy::type_complexity)]
pub(crate) fn spawn_login(
username: String,
password: String,
) -> (
async_channel::Receiver<Result<(), String>>,
async_channel::Receiver<std::sync::mpsc::Sender<String>>,
async_channel::Receiver<(String, std::sync::mpsc::Sender<String>)>,
) {
let (tx, rx) = async_channel::bounded(1); let (tx, rx) = async_channel::bounded(1);
// Two, not one: a CAPTCHA-gated login restarts, so a 2FA account is asked // Two, not one: a CAPTCHA-gated login restarts, so a 2FA account is asked
// for a code on each attempt and a single slot would deadlock the worker. // for a code on each attempt and a single slot would deadlock the worker.
let (totp_req_tx, totp_req_rx) = async_channel::bounded(2); let (totp_req_tx, totp_req_rx) = async_channel::bounded(2);
let (hv_req_tx, hv_req_rx) = async_channel::bounded(1); let (hv_req_tx, hv_req_rx) = async_channel::bounded(1);
let (mailbox_req_tx, mailbox_req_rx) = async_channel::bounded(1);
std::thread::spawn(move || { std::thread::spawn(move || {
let rt = match tokio::runtime::Builder::new_current_thread() let rt = match tokio::runtime::Builder::new_current_thread()
.enable_all() .enable_all()
@@ -267,19 +311,30 @@ pub(crate) fn spawn_login(
.recv() .recv()
.map_err(|_| pdfs_core::Error::Other("two-factor entry cancelled".into())) .map_err(|_| pdfs_core::Error::Other("two-factor entry cancelled".into()))
}, },
|hv| { |attempt| {
// Gated: hand the UI the page to show, block on the token the let (answer_tx, answer_rx) = std::sync::mpsc::channel::<(String, bool)>();
// user earns by solving it. mailbox_req_tx
let (token_tx, token_rx) = std::sync::mpsc::channel::<String>(); .send_blocking((attempt, answer_tx))
hv_req_tx .map_err(|_| pdfs_core::Error::Other("mailbox prompt closed".into()))?;
.send_blocking((hv.verification_url(), token_tx)) let (password, remember) = answer_rx
.map_err(|_| {
pdfs_core::Error::Other("verification prompt closed".into())
})?;
let token = token_rx
.recv() .recv()
.map_err(|_| pdfs_core::Error::Other("verification cancelled".into()))?; .map_err(|_| pdfs_core::Error::Other("mailbox entry cancelled".into()))?;
Ok(HumanVerificationCredential::captcha(token)) Ok(auth::Mailbox {
password: password.into(),
remember,
})
},
|hv| {
// Gated: hand the UI the challenge, block on the credential
// the user earns by solving it.
let (credential_tx, credential_rx) =
std::sync::mpsc::channel::<HumanVerificationCredential>();
hv_req_tx.send_blocking((hv, credential_tx)).map_err(|_| {
pdfs_core::Error::Other("verification prompt closed".into())
})?;
credential_rx
.recv()
.map_err(|_| pdfs_core::Error::Other("verification cancelled".into()))
}, },
) )
.await .await
@@ -287,7 +342,143 @@ pub(crate) fn spawn_login(
}); });
let _ = tx.send_blocking(result); let _ = tx.send_blocking(result);
}); });
(rx, totp_req_rx, hv_req_rx) LoginChannels {
result: rx,
totp: totp_req_rx,
hv: hv_req_rx,
mailbox: mailbox_req_rx,
}
}
/// Ask for the mailbox password of a two-password account.
///
/// `done` gets the password and the "keep unlocked" choice, or `None` when the
/// dialog was cancelled. `retry` says the last answer was wrong. The choice
/// defaults to off: storing the mailbox unlock in the keyring trades away what
/// a separate mailbox password is for, so it is something to opt into.
pub(crate) fn ask_mailbox(
ui: &Rc<Ui>,
retry: bool,
done: impl FnOnce(Option<(String, bool)>) + 'static,
) {
let body = if retry {
gettext("That mailbox password didn't unlock your account. Try again.")
} else {
gettext("This account has a separate mailbox password. Enter it to unlock your files.")
};
let dialog = adw::AlertDialog::builder()
.heading(gettext("Mailbox Password"))
.body(body)
.build();
let group = adw::PreferencesGroup::new();
let entry = adw::PasswordEntryRow::builder()
.title(gettext("Mailbox password"))
.activates_default(true)
.build();
let remember = adw::SwitchRow::builder()
.title(gettext("Keep unlocked on this computer"))
.subtitle(gettext(
"Stores the unlock in the system keyring, so Proton Drive opens without asking after every login. Otherwise you enter the password once per session.",
))
.build();
group.add(&entry);
group.add(&remember);
dialog.set_extra_child(Some(&group));
dialog.add_response("cancel", &gettext("Cancel"));
dialog.add_response("unlock", &gettext("Unlock"));
dialog.set_response_appearance("unlock", adw::ResponseAppearance::Suggested);
dialog.set_default_response(Some("unlock"));
dialog.set_close_response("cancel");
let done = RefCell::new(Some(done));
dialog.connect_response(None, move |_, resp| {
let Some(done) = done.borrow_mut().take() else {
return;
};
let password = entry.text().to_string();
if resp == "unlock" && !password.is_empty() {
done(Some((password, remember.is_active())));
} else {
done(None);
}
});
let parent = ui_window(ui).map(|w| w.upcast::<gtk4::Window>());
dialog.present(parent.as_ref());
}
/// If the signed-in account is locked, ask for its mailbox password — once per
/// run; after a cancel, a toast keeps an Unlock button within reach.
///
/// The key state is a keyring and kernel-keyring read, so it runs on a worker.
pub(crate) fn offer_unlock(ui: &Rc<Ui>) {
if ui.unlock_offered.get() {
return;
}
let ui = ui.clone();
glib::spawn_future_local(async move {
let state = gio::spawn_blocking(|| auth::key_state().ok()).await;
if matches!(state, Ok(Some(auth::KeyState::Locked))) && !ui.unlock_offered.replace(true) {
prompt_unlock(&ui, false);
}
});
}
enum UnlockOutcome {
Unlocked,
Wrong,
Failed(String),
}
/// Ask for the mailbox password and unlock the account with it.
fn prompt_unlock(ui: &Rc<Ui>, retry: bool) {
let ui_done = ui.clone();
ask_mailbox(ui, retry, move |answer| {
let Some((password, remember)) = answer else {
toast_action(
&ui_done,
&gettext("Proton Drive is locked. Enter your mailbox password to unlock it."),
&gettext("Unlock"),
|ui| prompt_unlock(ui, false),
);
return;
};
glib::spawn_future_local(async move {
// Verifying talks to Proton and the restart talks to systemd, so
// both run on a worker, never on the main loop.
let outcome = gio::spawn_blocking(move || {
let rt = match tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
{
Ok(rt) => rt,
Err(e) => return UnlockOutcome::Failed(e.to_string()),
};
match rt.block_on(auth::unlock(&password, remember)) {
Ok(()) => {
service::restart();
UnlockOutcome::Unlocked
}
Err(pdfs_core::Error::WrongMailboxPassword) => UnlockOutcome::Wrong,
Err(e) => UnlockOutcome::Failed(login_error_message(&e)),
}
})
.await;
match outcome {
Ok(UnlockOutcome::Unlocked) => {
toast(&ui_done, &gettext("Unlocked."));
refresh(&ui_done);
}
Ok(UnlockOutcome::Wrong) => prompt_unlock(&ui_done, true),
Ok(UnlockOutcome::Failed(message)) => {
toast_error(&ui_done, &gettext("Couldn't unlock"), &message)
}
Err(_) => {}
}
});
});
} }
/// Show the lazy two-factor dialog and feed the entered code back to the waiting /// Show the lazy two-factor dialog and feed the entered code back to the waiting
@@ -330,6 +521,63 @@ pub(crate) fn prompt_2fa(ui: &Rc<Ui>, code_tx: std::sync::mpsc::Sender<String>)
dialog.present(parent.as_ref()); dialog.present(parent.as_ref());
} }
/// Let the user solve the challenge in their own browser, for when the
/// verification window could not come up. The page cannot hand its token back
/// to us, so the user says when they are done, and the retry carries the
/// challenge token, which the solved page made good (see
/// [`auth::browser_verified`]).
fn verify_in_browser(
ui: &Rc<Ui>,
hv: &HumanVerification,
credential_tx: std::sync::mpsc::Sender<HumanVerificationCredential>,
) {
let dialog = adw::AlertDialog::builder()
.heading(gettext("Verify in Your Browser"))
.body(gettext(
"The verification page can't be shown in the app on this system. Open it in your browser, complete it there, then come back and choose Continue.",
))
.build();
let url = auth::browser_verification_url(hv);
let open = gtk4::Button::builder()
.label(gettext("Open Verification Page"))
.halign(gtk4::Align::Center)
.build();
open.add_css_class("pill");
dialog.set_extra_child(Some(&open));
dialog.add_response("cancel", &gettext("Cancel"));
dialog.add_response("continue", &gettext("Continue"));
dialog.set_response_appearance("continue", adw::ResponseAppearance::Suggested);
// Continuing before the page was even opened would only fail the retry.
dialog.set_response_enabled("continue", false);
dialog.set_default_response(Some("continue"));
dialog.set_close_response("cancel");
let dialog_weak = dialog.downgrade();
open.connect_clicked(move |_| {
open_uri(&url);
if let Some(dialog) = dialog_weak.upgrade() {
dialog.set_response_enabled("continue", true);
}
});
let credential = auth::browser_verified(hv);
let credential_tx = RefCell::new(Some(credential_tx));
dialog.connect_response(None, move |_, resp| {
// On cancel/close we take + drop `tx` without sending, so the worker's
// recv errors out and the login is reported as cancelled.
if let Some(tx) = credential_tx.borrow_mut().take()
&& resp == "continue"
{
let _ = tx.send(credential.clone());
}
});
let parent = ui.login.login_button.root().and_downcast::<gtk4::Window>();
dialog.present(parent.as_ref());
}
/// Sign out, after a confirmation: disable+stop the mount service (so the /// Sign out, after a confirmation: disable+stop the mount service (so the
/// daemon isn't left running without credentials), forget the stored session, /// daemon isn't left running without credentials), forget the stored session,
/// and drop back to the login page. /// and drop back to the login page.
@@ -374,6 +622,12 @@ mod tests {
assert!(login_error_message(&hv).contains("verification wasn't completed")); assert!(login_error_message(&hv).contains("verification wasn't completed"));
} }
#[test]
fn a_cancelled_mailbox_prompt_is_reported_as_such() {
let error = pdfs_core::Error::Other("mailbox entry cancelled".into());
assert!(login_error_message(&error).contains("no mailbox password"));
}
#[test] #[test]
fn an_unmapped_error_keeps_its_text() { fn an_unmapped_error_keeps_its_text() {
let error = pdfs_core::Error::Other("something odd".into()); let error = pdfs_core::Error::Other("something odd".into());
@@ -618,6 +618,7 @@ pub(crate) fn shared_item_as_entry(item: &SharedItem) -> DirEntry {
shared_by_unverified: false, shared_by_unverified: false,
trashed_at: 0, trashed_at: 0,
trashed_from: None, trashed_from: None,
issue: None,
} }
} }
+6 -1
View File
@@ -860,6 +860,7 @@ pub(crate) fn refresh(ui: &Rc<Ui>) {
} }
} }
offer_unlock(ui);
refresh_status(ui); refresh_status(ui);
refresh_transfers(ui); refresh_transfers(ui);
} }
@@ -1069,7 +1070,11 @@ pub(crate) fn repaint_transfers(ui: &Rc<Ui>, items: &[TransferItem], jobs: &[Job
let row_box = gtk4::Box::new(gtk4::Orientation::Vertical, 4); let row_box = gtk4::Box::new(gtk4::Orientation::Vertical, 4);
row_box.set_margin_top(8); row_box.set_margin_top(8);
row_box.set_margin_bottom(8); row_box.set_margin_bottom(8);
let label = gtk4::Label::builder().halign(gtk4::Align::Start).build(); // Ellipsized so a long file name cannot widen the page.
let label = gtk4::Label::builder()
.halign(gtk4::Align::Start)
.ellipsize(gtk4::pango::EllipsizeMode::Middle)
.build();
label.add_css_class("dim-label"); label.add_css_class("dim-label");
let bar = gtk4::ProgressBar::new(); let bar = gtk4::ProgressBar::new();
row_box.append(&label); row_box.append(&label);
+1
View File
@@ -533,6 +533,7 @@ mod tests {
shared_by_unverified: false, shared_by_unverified: false,
trashed_at, trashed_at,
trashed_from: from.map(str::to_string), trashed_from: from.map(str::to_string),
issue: None,
} }
} }
+168 -40
View File
@@ -1,4 +1,4 @@
//! The human-verification (CAPTCHA) dialog. //! The human-verification (CAPTCHA) window.
//! //!
//! Proton gates logins it does not recognise — a new IP, a VPN exit — behind a //! Proton gates logins it does not recognise — a new IP, a VPN exit — behind a
//! CAPTCHA. The API answers the login with a challenge instead of a session, and //! CAPTCHA. The API answers the login with a challenge instead of a session, and
@@ -8,10 +8,26 @@
//! The page is hosted in a `WebKitWebView` rather than handed to the system //! The page is hosted in a `WebKitWebView` rather than handed to the system
//! browser because it reports completion by posting a message to its host, not //! browser because it reports completion by posting a message to its host, not
//! by redirecting: a browser has nowhere to post it back to. //! by redirecting: a browser has nowhere to post it back to.
//!
//! The webview runs in a child `pdfs-app` started with [`VERIFY_ARG`], not in the
//! app itself. WebKit aborts its whole process when its sandbox cannot start, as
//! on hardened systems that keep bubblewrap from the journal socket
//! (`docs/BUGS.md` B195); in a child that costs the verification, not the app.
//! The URL goes to the child on stdin, so it never shows in a process list, and
//! the token comes back on stdout.
use crate::*; use crate::*;
use std::io::{BufRead, BufReader, Write as _};
use std::process::{ExitStatus, Stdio};
use webkit6::prelude::*; use webkit6::prelude::*;
/// The argument that makes `pdfs-app` the verification window instead of the app.
pub(crate) const VERIFY_ARG: &str = "--human-verification";
/// Marks the token among whatever else lands on the child's stdout: GLib and
/// WebKit write debug output there.
const TOKEN_PREFIX: &str = "hv-token ";
/// Bridge the verification page's `postMessage` to a handler this side can read. /// Bridge the verification page's `postMessage` to a handler this side can read.
/// ///
/// The page targets its embedding host (`window.parent`), which in a webview is /// The page targets its embedding host (`window.parent`), which in a webview is
@@ -26,15 +42,100 @@ window.addEventListener('message', function (event) {
}); });
"#; "#;
/// Present the challenge and hand the solved token back through `token_tx`. /// How the verification window ended.
/// #[derive(Debug, PartialEq, Eq)]
/// Closing the dialog without solving drops the sender, which the waiting login pub(crate) enum Verification {
/// worker reads as a cancelled sign-in — the same contract as the 2FA prompt. Solved(String),
pub(crate) fn prompt_human_verification( /// Closed without solving.
ui: &Rc<Ui>, Cancelled,
url: &str, /// Never came up, or died: most likely WebKit could not start its sandbox.
token_tx: std::sync::mpsc::Sender<String>, Failed,
) { }
/// Show the challenge in the verification window and wait until it closes.
pub(crate) async fn verify_human(url: String) -> Verification {
match gio::spawn_blocking(move || run_verification_child(&url)).await {
Ok(Ok(verification)) => verification,
Ok(Err(e)) => {
tracing::error!("cannot run the verification window: {e}");
Verification::Failed
}
Err(_) => Verification::Failed,
}
}
fn run_verification_child(url: &str) -> std::io::Result<Verification> {
let mut child = Command::new(own_exe())
.arg(VERIFY_ARG)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn()?;
if let Some(mut stdin) = child.stdin.take() {
// A failed write means the child is already gone; its exit status says
// how. Dropping `stdin` closes it.
let _ = writeln!(stdin, "{url}");
}
// Stops reading at the token rather than at EOF: WebKit's helper processes
// inherit the pipe and may hold it open a moment after the child quits.
let token = child
.stdout
.take()
.and_then(|out| find_token(BufReader::new(out)));
let status = child.wait()?;
Ok(outcome(token, status))
}
/// This very binary, so a development build opens its own window. After an
/// upgrade replaced the file, `current_exe` names a deleted path, and the
/// installed `pdfs-app` is the next best.
fn own_exe() -> PathBuf {
std::env::current_exe()
.ok()
.filter(|path| path.is_file())
.unwrap_or_else(|| PathBuf::from("pdfs-app"))
}
fn find_token(out: impl BufRead) -> Option<String> {
out.lines()
.map_while(Result::ok)
.find_map(|line| line.strip_prefix(TOKEN_PREFIX).map(str::to_owned))
}
/// A token wins over the exit status: the user solved the page, even if
/// WebKit's teardown crashed after it.
fn outcome(token: Option<String>, status: ExitStatus) -> Verification {
match token {
Some(token) => Verification::Solved(token),
None if status.success() => Verification::Cancelled,
None => {
tracing::error!(%status, "the verification window failed");
Verification::Failed
}
}
}
/// Run as the verification window: read the page's URL from stdin, show it, and
/// print the token once the user has solved it. Closing the window exits
/// without a token, which the app reads as a cancelled sign-in.
pub(crate) fn run_verification_window() -> glib::ExitCode {
let mut url = String::new();
if std::io::stdin().lock().read_line(&mut url).is_err() || url.trim().is_empty() {
tracing::error!("{VERIFY_ARG} expects the verification URL on stdin");
return glib::ExitCode::FAILURE;
}
let url = url.trim().to_owned();
// Not unique: the app that started it already holds APP_ID.
let app = adw::Application::builder()
.flags(gio::ApplicationFlags::NON_UNIQUE)
.build();
app.connect_activate(move |app| build_verification_window(app, &url));
// GApplication would refuse VERIFY_ARG as an unknown option.
let argv0 = std::env::args().next().unwrap_or_default();
app.run_with_args(&[argv0])
}
fn build_verification_window(app: &adw::Application, url: &str) {
let content = webkit6::UserContentManager::new(); let content = webkit6::UserContentManager::new();
content.add_script(&webkit6::UserScript::new( content.add_script(&webkit6::UserScript::new(
BRIDGE, BRIDGE,
@@ -53,24 +154,19 @@ pub(crate) fn prompt_human_verification(
.hexpand(true) .hexpand(true)
.build(); .build();
let dialog = adw::Dialog::builder() let body = gtk4::Box::new(gtk4::Orientation::Vertical, 0);
body.append(&adw::HeaderBar::new());
body.append(&webview);
let window = adw::ApplicationWindow::builder()
.application(app)
.title(gettext("Verification")) .title(gettext("Verification"))
.content_width(420) .icon_name(APP_ID)
.content_height(560) .default_width(420)
.default_height(560)
.content(&body)
.build(); .build();
let header = adw::HeaderBar::new(); let app = app.downgrade();
let body = gtk4::Box::new(gtk4::Orientation::Vertical, 0);
body.append(&header);
body.append(&webview);
dialog.set_child(Some(&body));
// Taken on the first success *or* on close, so the token is sent at most
// once and a cancelled dialog always drops the sender.
let token_tx = Rc::new(RefCell::new(Some(token_tx)));
let tx = token_tx.clone();
let dlg_weak = dialog.downgrade();
content.connect_script_message_received(Some("hv"), move |_, value| { content.connect_script_message_received(Some("hv"), move |_, value| {
let Some(token) = extract_token(&value.to_str()) else { let Some(token) = extract_token(&value.to_str()) else {
// Every `postMessage` on the page reaches here, most of them the // Every `postMessage` on the page reaches here, most of them the
@@ -78,25 +174,17 @@ pub(crate) fn prompt_human_verification(
// is simply not ours. // is simply not ours.
return; return;
}; };
if let Some(tx) = tx.borrow_mut().take() { let mut stdout = std::io::stdout().lock();
let _ = tx.send(token); if let Err(e) = writeln!(stdout, "{TOKEN_PREFIX}{token}").and_then(|()| stdout.flush()) {
tracing::error!("cannot hand the verification token back: {e}");
} }
if let Some(dlg) = dlg_weak.upgrade() { if let Some(app) = app.upgrade() {
dlg.close(); app.quit();
} }
}); });
// Covers the close button, Escape, and the user giving up.
dialog.connect_closed(move |_| {
token_tx.borrow_mut().take();
});
webview.load_uri(url); webview.load_uri(url);
window.present();
let parent = ui.login.login_button.root().and_downcast::<gtk4::Window>();
// Returns immediately; the login worker stays blocked on the channel until
// the handler above sends, or the dialog closes and drops the sender.
dialog.present(parent.as_ref());
} }
/// Pull the verification token out of a message posted by the verification page. /// Pull the verification token out of a message posted by the verification page.
@@ -133,7 +221,12 @@ fn extract_token(raw: &str) -> Option<String> {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::extract_token; use super::{Verification, extract_token, find_token, outcome};
use std::os::unix::process::ExitStatusExt as _;
use std::process::ExitStatus;
/// The wait status of a process killed by SIGABRT.
const ABORTED: i32 = 6;
#[test] #[test]
fn a_completion_message_yields_its_token() { fn a_completion_message_yields_its_token() {
@@ -163,4 +256,39 @@ mod tests {
assert!(extract_token(raw).is_none(), "accepted: {raw}"); assert!(extract_token(raw).is_none(), "accepted: {raw}");
} }
} }
#[test]
fn the_token_is_found_among_other_output() {
let out = "Gtk-Message: something\nhv-token tok-1\nmore\n";
assert_eq!(find_token(out.as_bytes()).as_deref(), Some("tok-1"));
assert_eq!(find_token("only chatter\n".as_bytes()), None);
}
#[test]
fn a_window_closed_without_a_token_cancels() {
assert_eq!(
outcome(None, ExitStatus::from_raw(0)),
Verification::Cancelled
);
}
/// WebKit aborts when its sandbox cannot start (B195). That must read as a
/// failure the app explains, not as the user giving up.
#[test]
fn a_window_that_died_fails() {
assert_eq!(
outcome(None, ExitStatus::from_raw(ABORTED)),
Verification::Failed
);
let exited_1 = ExitStatus::from_raw(1 << 8);
assert_eq!(outcome(None, exited_1), Verification::Failed);
}
#[test]
fn a_token_counts_even_if_the_window_died_after_it() {
assert_eq!(
outcome(Some("tok-1".into()), ExitStatus::from_raw(ABORTED)),
Verification::Solved("tok-1".into())
);
}
} }
@@ -133,7 +133,7 @@ impl Drop for Busy {
/// from Shared or Shared-with-me may not: it can live under a device folder's /// from Shared or Shared-with-me may not: it can live under a device folder's
/// own inode space, or on someone else's volume, neither of which the primary /// own inode space, or on someone else's volume, neither of which the primary
/// mount's path space can name. Those carry a uid instead, and the daemon /// mount's path space can name. Those carry a uid instead, and the daemon
/// resolves it with `Core::resolve_anywhere` (mount-architecture.md P4). /// resolves it with `Core::resolve_anywhere`.
/// ///
/// The two are separate request variants rather than one request with an /// The two are separate request variants rather than one request with an
/// optional field on purpose: an older daemon that ignored `uid` would resolve /// optional field on purpose: an older daemon that ignored `uid` would resolve
+143
View File
@@ -0,0 +1,143 @@
//! Match highlighting: which characters of a result name the query matched,
//! as Pango markup with those characters in bold.
//!
//! Each query term is looked for as a substring of the name first, and as a
//! subsequence ("rpt" in "report") when that fails, which is roughly how the
//! daemon's ranking treats a term too. Case is folded one character at a
//! time, so every range stays on a character boundary of the original name,
//! whatever the script.
use gtk4::glib;
/// One character of a name: its byte range and its case-folded form.
struct Folded {
start: usize,
end: usize,
folded: String,
}
fn fold(text: &str) -> Vec<Folded> {
text.char_indices()
.map(|(start, c)| Folded {
start,
end: start + c.len_utf8(),
folded: c.to_lowercase().collect(),
})
.collect()
}
/// The byte ranges of `name` that `query` matched, sorted and merged.
pub(crate) fn ranges(name: &str, query: &str) -> Vec<(usize, usize)> {
let chars = fold(name);
let mut found: Vec<(usize, usize)> = Vec::new();
for term in query.split_whitespace() {
let term: Vec<String> = fold(term).into_iter().map(|c| c.folded).collect();
if let Some(at) = substring(&chars, &term) {
found.push((chars[at].start, chars[at + term.len() - 1].end));
} else if let Some(hits) = subsequence(&chars, &term) {
found.extend(hits.into_iter().map(|at| (chars[at].start, chars[at].end)));
}
}
found.sort_unstable();
let mut merged: Vec<(usize, usize)> = Vec::with_capacity(found.len());
for (start, end) in found {
match merged.last_mut() {
Some(last) if start <= last.1 => last.1 = last.1.max(end),
_ => merged.push((start, end)),
}
}
merged
}
/// Where `term` first occurs in `chars`, as a character index.
fn substring(chars: &[Folded], term: &[String]) -> Option<usize> {
if term.is_empty() || term.len() > chars.len() {
return None;
}
(0..=chars.len() - term.len()).find(|&at| {
chars[at..at + term.len()]
.iter()
.zip(term)
.all(|(c, t)| c.folded == *t)
})
}
/// The character indices of the leftmost match of `term` as a subsequence,
/// or `None` when not every character of it can be found in order.
fn subsequence(chars: &[Folded], term: &[String]) -> Option<Vec<usize>> {
if term.is_empty() {
return None;
}
let mut hits = Vec::with_capacity(term.len());
let mut next = 0;
for wanted in term {
let at = (next..chars.len()).find(|&i| chars[i].folded == *wanted)?;
hits.push(at);
next = at + 1;
}
Some(hits)
}
/// `name` as Pango markup, escaped, with the matched characters in bold.
pub(crate) fn markup(name: &str, query: &str) -> String {
let mut out = String::with_capacity(name.len() + 16);
let mut done = 0;
for (start, end) in ranges(name, query) {
out.push_str(&glib::markup_escape_text(&name[done..start]));
out.push_str("<b>");
out.push_str(&glib::markup_escape_text(&name[start..end]));
out.push_str("</b>");
done = end;
}
out.push_str(&glib::markup_escape_text(&name[done..]));
out
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_substring_match_is_one_range_in_any_case() {
assert_eq!(ranges("Tax Report.pdf", "REP"), vec![(4, 7)]);
}
#[test]
fn a_term_that_is_no_substring_falls_back_to_a_subsequence() {
// r, p and t of "report", not the "rt" pair at the end of a substring.
assert_eq!(ranges("report.txt", "rpt"), vec![(0, 1), (2, 3), (5, 6)]);
}
#[test]
fn a_term_with_no_match_highlights_nothing() {
assert!(ranges("report.txt", "xyz").is_empty());
assert!(ranges("report.txt", "").is_empty());
}
#[test]
fn several_terms_merge_into_sorted_ranges() {
assert_eq!(
ranges("tax return 2024", "2024 tax ret"),
vec![(0, 3), (4, 7), (11, 15)]
);
assert_eq!(ranges("abc", "ab bc"), vec![(0, 3)]);
}
#[test]
fn non_ascii_names_keep_ranges_on_character_boundaries() {
assert_eq!(ranges("Über Größe.odt", "über"), vec![(0, 5)]);
assert_eq!(ranges("Über Größe.odt", "GRÖ"), vec![(6, 10)]);
assert_eq!(markup("Über Größe.odt", "größe"), "Über <b>Größe</b>.odt");
assert_eq!(
markup("日本語のメモ.txt", "メモ"),
"日本語の<b>メモ</b>.txt"
);
}
#[test]
fn markup_escapes_the_name_inside_and_outside_matches() {
assert_eq!(markup("a<b>&c", "b"), "a&lt;<b>b</b>&gt;&amp;c");
assert_eq!(markup("R&D.pdf", "r&d"), "<b>R&amp;D</b>.pdf");
assert_eq!(markup("<none>", "zzz"), "&lt;none&gt;");
}
}
+363
View File
@@ -0,0 +1,363 @@
//! Recent files and frecency.
//!
//! The launcher remembers what it opened in `prompt-history.json` under the
//! state directory. With an empty query those files come first, and while
//! searching a file opened often and lately gets a bonus on top of its match
//! score (see [`frecency_bonus`]).
//!
//! One writer thread owns the file: it loads it, applies every recorded visit
//! in order and writes the result back atomically. A visit recorded before the
//! first load finished simply waits in the channel, so it is merged into the
//! loaded list rather than overwriting it. The main thread only ever sees
//! snapshots.
use std::cell::RefCell;
use std::collections::HashMap;
use std::io::Write;
use std::path::{Path, PathBuf};
use std::rc::Rc;
use gtk4::glib;
use crate::Hit;
/// How many files the history keeps. More than the launcher ever shows, so
/// frecency still knows a file that dropped off the recent list.
const MAX_VISITS: usize = 50;
/// The frecency bonus of one visit made just now.
const VISIT_BONUS: f64 = 1500.0;
/// The bonus halves every this many days without a visit.
const HALF_LIFE_DAYS: f64 = 14.0;
/// The largest frecency bonus. A name match is worth up to 20000 points, so
/// history breaks ties between similar matches but cannot beat a much better
/// one.
const MAX_BONUS: f64 = 6000.0;
#[derive(Clone, serde::Serialize, serde::Deserialize)]
pub(crate) struct Visit {
pub(crate) hit: Hit,
/// When the file was last opened, in epoch seconds.
pub(crate) last: i64,
pub(crate) count: u32,
}
/// The history bonus for a file opened `count` times, the last time
/// `age_secs` ago.
pub(crate) fn frecency_bonus(count: u32, age_secs: i64) -> i64 {
let age_days = age_secs.max(0) as f64 / 86_400.0;
let bonus = f64::from(count) * VISIT_BONUS * 0.5f64.powf(age_days / HALF_LIFE_DAYS);
bonus.min(MAX_BONUS) as i64
}
/// Count one more visit of `hit` at `now`, newest first, and keep at most
/// [`MAX_VISITS`]. The stored hit is replaced, so its size, date and sync
/// state are the ones from the latest search.
fn record(visits: &mut Vec<Visit>, hit: Hit, now: i64) {
let key = hit.key();
let count = match visits.iter().position(|visit| visit.hit.key() == key) {
Some(at) => visits.remove(at).count.saturating_add(1),
None => 1,
};
visits.insert(
0,
Visit {
hit,
last: now,
count,
},
);
visits.truncate(MAX_VISITS);
}
/// Drop local files that no longer exist. Drive files stay: checking them
/// would touch the mount, and a stale one only costs a row.
fn prune(visits: &mut Vec<Visit>) -> bool {
let before = visits.len();
visits.retain(|visit| match &visit.hit {
Hit::Local(hit) => Path::new(&hit.path).symlink_metadata().is_ok(),
Hit::Drive(_) => true,
});
visits.len() != before
}
fn load(path: &Path) -> Vec<Visit> {
match std::fs::read(path) {
Ok(bytes) => serde_json::from_slice(&bytes).unwrap_or_else(|e| {
tracing::warn!("ignoring unreadable {}: {e}", path.display());
Vec::new()
}),
Err(_) => Vec::new(),
}
}
/// Write the history through a temporary file and a rename, so a crash
/// mid-write leaves the old file rather than half a new one.
fn save(path: &Path, visits: &[Visit]) -> std::io::Result<()> {
let json = serde_json::to_vec(visits).map_err(std::io::Error::other)?;
if let Some(dir) = path.parent() {
std::fs::create_dir_all(dir)?;
}
let tmp = path.with_extension("json.tmp");
let mut file = std::fs::File::create(&tmp)?;
file.write_all(&json)?;
file.sync_all()?;
std::fs::rename(&tmp, path)
}
enum Command {
Record(Hit, i64),
Prune,
}
/// The writer thread's loop: load, then apply commands until the launcher
/// exits. `publish` receives the list after the load and after every change.
fn serve(path: &Path, commands: &async_channel::Receiver<Command>, publish: impl Fn(Vec<Visit>)) {
let mut visits = load(path);
let pruned = prune(&mut visits);
publish(visits.clone());
if pruned && let Err(e) = save(path, &visits) {
tracing::warn!("cannot write {}: {e}", path.display());
}
while let Ok(command) = commands.recv_blocking() {
let changed = match command {
Command::Record(hit, now) => {
record(&mut visits, hit, now);
true
}
Command::Prune => prune(&mut visits),
};
if changed {
if let Err(e) = save(path, &visits) {
tracing::warn!("cannot write {}: {e}", path.display());
}
publish(visits.clone());
}
}
}
/// The main thread's view of the history.
pub(crate) struct History {
visits: Rc<RefCell<Vec<Visit>>>,
commands: async_channel::Sender<Command>,
}
impl History {
/// Start the writer thread. `changed` runs on the main thread whenever a
/// new snapshot arrives.
pub(crate) fn start(path: PathBuf, changed: impl Fn() + 'static) -> Self {
let (commands, inbox) = async_channel::unbounded();
let (snapshots, updates) = async_channel::unbounded::<Vec<Visit>>();
let spawned = std::thread::Builder::new()
.name("prompt-history".into())
.spawn(move || {
serve(&path, &inbox, |visits| {
let _ = snapshots.send_blocking(visits);
})
});
if let Err(e) = spawned {
tracing::warn!("cannot start the history writer: {e}");
}
let visits = Rc::new(RefCell::new(Vec::new()));
let shared = visits.clone();
glib::spawn_future_local(async move {
while let Ok(snapshot) = updates.recv().await {
*shared.borrow_mut() = snapshot;
changed();
}
});
Self { visits, commands }
}
/// Remember that `hit` was opened just now.
pub(crate) fn record(&self, hit: Hit) {
let _ = self.commands.try_send(Command::Record(hit, now()));
}
/// Forget local files that were deleted since they were opened.
pub(crate) fn prune(&self) {
let _ = self.commands.try_send(Command::Prune);
}
/// The most recently opened files, newest first.
pub(crate) fn recent(&self, limit: usize) -> Vec<Hit> {
self.visits
.borrow()
.iter()
.take(limit)
.map(|visit| visit.hit.clone())
.collect()
}
/// The frecency bonus of every remembered file, by [`Hit::key`].
pub(crate) fn bonuses(&self) -> HashMap<(bool, String), i64> {
let now = now();
self.visits
.borrow()
.iter()
.map(|visit| {
(
visit.hit.key(),
frecency_bonus(visit.count, now - visit.last),
)
})
.collect()
}
}
fn now() -> i64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map_or(0, |d| d.as_secs() as i64)
}
#[cfg(test)]
mod tests {
use super::*;
use pdfs_core::control::{LocalHit, SearchHit};
const DAY: i64 = 86_400;
fn drive(name: &str) -> Hit {
Hit::Drive(SearchHit {
name: name.into(),
path: format!("Docs/{name}"),
is_dir: false,
size: 1,
modified: 1,
pinned: false,
cached: false,
uid: format!("v~{name}"),
mounted_path: None,
score: 0,
})
}
fn scratch_dir(test: &str) -> PathBuf {
let dir = std::env::temp_dir().join(format!(
"pdfs-prompt-history-{test}-{}-{}",
std::process::id(),
now()
));
std::fs::create_dir_all(&dir).unwrap();
dir
}
#[test]
fn the_bonus_halves_every_two_weeks() {
assert_eq!(frecency_bonus(1, 0), 1500);
assert_eq!(frecency_bonus(1, 14 * DAY), 750);
assert_eq!(frecency_bonus(1, 28 * DAY), 375);
assert_eq!(frecency_bonus(2, 14 * DAY), 1500);
// A clock that went backwards counts as "just now", not as a boost.
assert_eq!(frecency_bonus(1, -DAY), 1500);
}
#[test]
fn the_bonus_is_capped() {
assert_eq!(frecency_bonus(4, 0), 6000);
assert_eq!(frecency_bonus(1000, 0), 6000);
assert!(frecency_bonus(1000, 365 * DAY) < 6000);
}
#[test]
fn a_repeat_visit_moves_the_file_to_the_top_and_counts_it() {
let mut visits = Vec::new();
record(&mut visits, drive("a.pdf"), 10);
record(&mut visits, drive("b.pdf"), 20);
record(&mut visits, drive("a.pdf"), 30);
assert_eq!(visits.len(), 2);
assert_eq!(visits[0].hit.name(), "a.pdf");
assert_eq!((visits[0].count, visits[0].last), (2, 30));
assert_eq!(visits[1].count, 1);
}
#[test]
fn the_history_keeps_the_newest_fifty_files() {
let mut visits = Vec::new();
for n in 0..60 {
record(&mut visits, drive(&format!("{n}.pdf")), n);
}
assert_eq!(visits.len(), MAX_VISITS);
assert_eq!(visits[0].hit.name(), "59.pdf");
assert_eq!(visits[MAX_VISITS - 1].hit.name(), "10.pdf");
}
#[test]
fn the_history_file_round_trips() {
let dir = scratch_dir("round-trip");
let path = dir.join("prompt-history.json");
let mut visits = Vec::new();
record(&mut visits, drive("a.pdf"), 10);
record(&mut visits, drive("a.pdf"), 20);
save(&path, &visits).unwrap();
let loaded = load(&path);
assert_eq!(loaded.len(), 1);
assert_eq!(loaded[0].hit.key(), drive("a.pdf").key());
assert_eq!((loaded[0].count, loaded[0].last), (2, 20));
assert!(!path.with_extension("json.tmp").exists());
std::fs::remove_dir_all(dir).unwrap();
}
#[test]
fn visits_recorded_before_the_load_are_merged_into_it() {
let dir = scratch_dir("merge");
let path = dir.join("prompt-history.json");
let mut stored = Vec::new();
record(&mut stored, drive("old.pdf"), 10);
record(&mut stored, drive("old.pdf"), 20);
save(&path, &stored).unwrap();
// Both visits are queued before the writer has read the file.
let (commands, inbox) = async_channel::unbounded();
commands
.send_blocking(Command::Record(drive("old.pdf"), 30))
.unwrap();
commands
.send_blocking(Command::Record(drive("new.pdf"), 40))
.unwrap();
drop(commands);
let published = RefCell::new(Vec::new());
serve(&path, &inbox, |visits| published.borrow_mut().push(visits));
let last = published.borrow().last().cloned().unwrap();
assert_eq!(last.len(), 2);
assert_eq!(last[0].hit.name(), "new.pdf");
assert_eq!((last[1].hit.name(), last[1].count), ("old.pdf", 3));
let on_disk = load(&path);
assert_eq!(on_disk.len(), 2);
assert_eq!(on_disk[1].count, 3);
std::fs::remove_dir_all(dir).unwrap();
}
#[test]
fn pruning_drops_deleted_local_files_only() {
let dir = scratch_dir("prune");
let kept = dir.join("kept.txt");
std::fs::write(&kept, "x").unwrap();
let local = |path: &Path| {
Hit::Local(LocalHit {
name: path.file_name().unwrap().to_string_lossy().into_owned(),
path: path.display().to_string(),
is_dir: false,
size: 1,
modified: 1,
score: 0,
})
};
let mut visits = Vec::new();
record(&mut visits, local(&kept), 1);
record(&mut visits, local(&dir.join("gone.txt")), 2);
record(&mut visits, drive("remote.pdf"), 3);
assert!(prune(&mut visits));
let names: Vec<&str> = visits.iter().map(|visit| visit.hit.name()).collect();
assert_eq!(names, ["remote.pdf", "kept.txt"]);
std::fs::remove_dir_all(dir).unwrap();
}
}
File diff suppressed because it is too large. Load diff
+323
View File
@@ -0,0 +1,323 @@
//! The preview pane beside the results: a thumbnail of the selected hit, what
//! it is, where it is and whether it is on this computer, and the actions for
//! it with their shortcuts.
use std::cell::RefCell;
use std::path::Path;
use std::rc::Rc;
use std::time::Duration;
use adw::prelude::*;
use gtk4::{gio, glib};
use super::row::{fill_slot, thumb_slot};
use super::thumbs::Thumbs;
use crate::{Hit, dates, gettext, human_bytes, pgettext};
/// Width of the pane.
pub(crate) const PREVIEW_WIDTH: i32 = 300;
/// The thumbnail area, inside the pane's padding.
const THUMB_WIDTH: i32 = 268;
const THUMB_HEIGHT: i32 = 180;
/// How long "Path copied" stays up.
const COPIED_FEEDBACK: Duration = Duration::from_millis(1500);
/// One label/value line of the details grid.
struct Detail {
title: gtk4::Label,
value: gtk4::Label,
}
impl Detail {
fn new(grid: &gtk4::Grid, row: i32, title: &str) -> Self {
let title = gtk4::Label::builder()
.label(title)
.xalign(0.0)
.valign(gtk4::Align::Start)
.build();
title.add_css_class("preview-key");
// A wrapping label asks for its whole text on one line as its natural
// width, which would widen the pane and squeeze the results. Capping it
// keeps the pane at its fixed width; long values wrap, then ellipsize.
let value = gtk4::Label::builder()
.xalign(0.0)
.hexpand(true)
.wrap(true)
.wrap_mode(gtk4::pango::WrapMode::WordChar)
.max_width_chars(1)
.lines(3)
.ellipsize(gtk4::pango::EllipsizeMode::Middle)
.build();
value.add_css_class("preview-value");
grid.attach(&title, 0, row, 1, 1);
grid.attach(&value, 1, row, 1, 1);
Self { title, value }
}
/// Show `value`, or hide the line when there is nothing to say.
fn set(&self, value: Option<&str>) {
self.title.set_visible(value.is_some());
self.value.set_visible(value.is_some());
self.value.set_label(value.unwrap_or_default());
}
}
pub(crate) struct Preview {
pub(crate) root: gtk4::Box,
stack: gtk4::Stack,
slot: gtk4::Overlay,
name: gtk4::Label,
kind: gtk4::Label,
size: Detail,
modified: Detail,
location: Detail,
availability: Detail,
availability_icon: gtk4::Image,
pub(crate) open: gtk4::Button,
open_label: gtk4::Label,
pub(crate) reveal: gtk4::Button,
pub(crate) copy: gtk4::Button,
copied: gtk4::Revealer,
copied_timer: Rc<RefCell<Option<glib::SourceId>>>,
}
/// A flat action button: the label, and its shortcut at the far end.
fn action(icon: &str, label: &str, accelerator: &str) -> (gtk4::Button, gtk4::Label) {
let content = gtk4::Box::new(gtk4::Orientation::Horizontal, 10);
content.append(&gtk4::Image::from_icon_name(icon));
let text = gtk4::Label::builder()
.label(label)
.xalign(0.0)
.hexpand(true)
.max_width_chars(1)
.ellipsize(gtk4::pango::EllipsizeMode::End)
.build();
content.append(&text);
let shortcut = gtk4::ShortcutLabel::new(accelerator);
shortcut.add_css_class("preview-shortcut");
content.append(&shortcut);
let button = gtk4::Button::builder()
.child(&content)
.focus_on_click(false)
.focusable(false)
.build();
button.add_css_class("flat");
button.add_css_class("preview-action");
(button, text)
}
impl Preview {
pub(crate) fn new() -> Self {
let details = gtk4::Box::new(gtk4::Orientation::Vertical, 0);
let slot = thumb_slot(THUMB_WIDTH, THUMB_HEIGHT, 96, gtk4::ContentFit::Contain);
slot.add_css_class("preview-thumb");
details.append(&slot);
let name = gtk4::Label::builder()
.xalign(0.0)
.wrap(true)
.wrap_mode(gtk4::pango::WrapMode::WordChar)
.max_width_chars(1)
.lines(3)
.ellipsize(gtk4::pango::EllipsizeMode::End)
.build();
name.add_css_class("preview-name");
details.append(&name);
let kind = gtk4::Label::builder()
.xalign(0.0)
.max_width_chars(1)
.ellipsize(gtk4::pango::EllipsizeMode::End)
.build();
kind.add_css_class("preview-kind");
details.append(&kind);
let grid = gtk4::Grid::builder()
.row_spacing(6)
.column_spacing(12)
.build();
grid.add_css_class("preview-details");
let size = Detail::new(&grid, 0, &pgettext("column", "Size"));
let modified = Detail::new(&grid, 1, &pgettext("column", "Modified"));
let location = Detail::new(&grid, 2, &gettext("Location"));
// Translators: a heading in the launcher's preview pane, followed by
// "Available offline", "Downloaded", "Online only" or "On this computer".
let availability = Detail::new(&grid, 3, &gettext("Availability"));
// The availability value is an icon and a text.
let availability_icon = gtk4::Image::new();
availability_icon.add_css_class("preview-badge");
grid.remove(&availability.value);
let availability_box = gtk4::Box::new(gtk4::Orientation::Horizontal, 6);
availability_box.append(&availability_icon);
availability_box.append(&availability.value);
grid.attach(&availability_box, 1, 3, 1, 1);
details.append(&grid);
let spacer = gtk4::Box::builder().vexpand(true).build();
details.append(&spacer);
let actions = gtk4::Box::new(gtk4::Orientation::Vertical, 2);
actions.add_css_class("preview-actions");
let (open, open_label) = action("document-open-symbolic", "", "Return");
let (reveal, _) = action(
"folder-open-symbolic",
&gettext("Show in Folder"),
"<Control>Return",
);
let (copy, _) = action("edit-copy-symbolic", &gettext("Copy Path"), "<Control>c");
actions.append(&open);
actions.append(&reveal);
actions.append(&copy);
details.append(&actions);
let copied_label = gtk4::Label::new(Some(&gettext("Path copied")));
copied_label.add_css_class("preview-feedback");
let copied = gtk4::Revealer::builder()
.transition_type(gtk4::RevealerTransitionType::Crossfade)
.child(&copied_label)
.build();
details.append(&copied);
let empty = gtk4::Image::from_icon_name("system-search-symbolic");
empty.set_pixel_size(64);
empty.add_css_class("preview-empty");
let stack = gtk4::Stack::builder()
.transition_type(gtk4::StackTransitionType::Crossfade)
.transition_duration(100)
.vexpand(true)
.build();
stack.add_named(&empty, Some("empty"));
stack.add_named(&details, Some("details"));
let root = gtk4::Box::builder()
.orientation(gtk4::Orientation::Vertical)
.width_request(PREVIEW_WIDTH)
.hexpand(false)
.build();
root.add_css_class("preview");
root.append(&stack);
Self {
root,
stack,
slot,
name,
kind,
size,
modified,
location,
availability,
availability_icon,
open,
open_label,
reveal,
copy,
copied,
copied_timer: Rc::new(RefCell::new(None)),
}
}
/// Show `hit`, or the empty pane when nothing is selected.
pub(crate) fn show(&self, hit: Option<&Hit>, thumbs: &Rc<Thumbs>, mountpoint: &Path) {
let Some(hit) = hit else {
self.stack.set_visible_child_name("empty");
return;
};
self.stack.set_visible_child_name("details");
fill_slot(&self.slot, hit, thumbs, mountpoint);
self.name.set_label(hit.name());
self.name.set_tooltip_text(Some(hit.name()));
let content_type = if hit.is_dir() {
glib::GString::from("inode/directory")
} else {
gio::functions::content_type_guess(Some(hit.name()), None).0
};
self.kind
.set_label(&gio::functions::content_type_get_description(&content_type));
let size = (!hit.is_dir() && hit.size() > 0).then(|| human_bytes(hit.size()));
self.size.set(size.as_deref());
let modified = (hit.modified() > 0).then(|| dates::relative(hit.modified()));
self.modified.set(modified.as_deref());
self.modified.value.set_tooltip_text(
(hit.modified() > 0)
.then(|| dates::full(hit.modified()))
.as_deref(),
);
let location = hit.location();
self.location.set(Some(&location));
self.location.value.set_tooltip_text(Some(&location));
match availability(hit) {
Some((icon, class, text)) => {
self.availability.set(Some(&text));
self.availability_icon.set_icon_name(Some(icon));
for other in ["badge-pinned", "badge-cached", "badge-cloud", "badge-local"] {
self.availability_icon.remove_css_class(other);
}
self.availability_icon.add_css_class(class);
self.availability_icon.set_visible(true);
}
None => {
self.availability.set(None);
self.availability_icon.set_visible(false);
}
}
self.open_label.set_label(&if hit.is_dir() {
gettext("Open folder")
} else {
pgettext("verb", "Open")
});
}
/// Say "Path copied" for a moment.
pub(crate) fn flash_copied(&self) {
self.copied.set_reveal_child(true);
if let Some(source) = self.copied_timer.borrow_mut().take() {
source.remove();
}
let copied = self.copied.clone();
let slot = self.copied_timer.clone();
let timer = glib::timeout_add_local_once(COPIED_FEEDBACK, move || {
slot.borrow_mut().take();
copied.set_reveal_child(false);
});
*self.copied_timer.borrow_mut() = Some(timer);
}
}
/// Where the hit's content is: its icon, its style class and its wording.
fn availability(hit: &Hit) -> Option<(&'static str, &'static str, String)> {
match hit {
Hit::Local(_) => Some((
"drive-harddisk-symbolic",
"badge-local",
gettext("On this computer"),
)),
Hit::Drive(drive) if drive.pinned => Some((
"pdfs-offline-symbolic",
"badge-pinned",
gettext("Available offline"),
)),
// A folder is never downloaded as a whole; only pinning says anything.
Hit::Drive(drive) if drive.is_dir => None,
Hit::Drive(drive) if drive.cached => Some((
"pdfs-cached-symbolic",
"badge-cached",
pgettext("file state", "Downloaded"),
)),
Hit::Drive(_) => Some((
"pdfs-online-only-symbolic",
"badge-cloud",
pgettext("file state", "Online only"),
)),
}
}
+187
View File
@@ -0,0 +1,187 @@
//! Result rows: a thumbnail or icon, the name with its matched characters in
//! bold, the location, and on the right the sync state, size and age.
use std::cell::Cell;
use std::path::{Path, PathBuf};
use std::rc::Rc;
use adw::prelude::*;
use gtk4::{gdk, gio};
use super::highlight;
use super::thumbs::Thumbs;
use crate::theme::proton_theme_active;
use crate::{Hit, format_age, human_bytes};
/// Edge of a row's thumbnail.
const ROW_THUMB: i32 = 40;
/// A thumbnail surface: an icon, with a picture laid over it that shows once
/// the thumbnail has loaded. [`Thumbs`] paints the picture.
pub(crate) fn thumb_slot(
width: i32,
height: i32,
icon_size: i32,
fit: gtk4::ContentFit,
) -> gtk4::Overlay {
let fallback = gtk4::Image::builder().pixel_size(icon_size).build();
let picture = gtk4::Picture::builder()
.width_request(width)
.height_request(height)
.content_fit(fit)
.can_shrink(true)
.visible(false)
.build();
picture.set_overflow(gtk4::Overflow::Hidden);
picture.add_css_class("thumb");
let overlay = gtk4::Overlay::new();
overlay.set_size_request(width, height);
overlay.set_halign(gtk4::Align::Center);
overlay.set_valign(gtk4::Align::Center);
overlay.set_child(Some(&fallback));
overlay.add_overlay(&picture);
overlay
}
/// Show `hit` in a slot from [`thumb_slot`]: its icon at once, and its
/// thumbnail when there is one.
pub(crate) fn fill_slot(slot: &gtk4::Overlay, hit: &Hit, thumbs: &Rc<Thumbs>, mountpoint: &Path) {
let Some(fallback) = slot.child().and_downcast::<gtk4::Image>() else {
return;
};
let Some(picture) = fallback.next_sibling().and_downcast::<gtk4::Picture>() else {
return;
};
fallback.set_from_gicon(&icon_for(hit.name(), hit.is_dir()));
fallback.set_visible(true);
picture.set_visible(false);
picture.set_paintable(gdk::Paintable::NONE);
thumbs.want(hit, mountpoint, &picture);
}
/// The desktop's own full-colour icon for a file name, so results look like
/// the user's file manager. Folders are Proton's purple ones while the Proton
/// theme is on, as in the app's browser.
pub(crate) fn icon_for(name: &str, is_dir: bool) -> gio::Icon {
if is_dir {
let folder = if proton_theme_active() {
"pdfs-proton-folder"
} else {
"folder"
};
return gio::ThemedIcon::new(folder).upcast();
}
let (content_type, _uncertain) = gio::functions::content_type_guess(Some(name), None);
gio::functions::content_type_get_icon(&content_type)
}
/// The sync badge of a Drive file, as in the app's browser: pinned, cached,
/// or online only. Folders and local files have none.
pub(crate) fn badge_for(hit: &Hit) -> Option<(&'static str, &'static str)> {
let Hit::Drive(drive) = hit else {
return None;
};
if drive.is_dir {
return None;
}
Some(if drive.pinned {
("pdfs-offline-symbolic", "badge-pinned")
} else if drive.cached {
("pdfs-cached-symbolic", "badge-cached")
} else {
("pdfs-online-only-symbolic", "badge-cloud")
})
}
pub(crate) fn build_row(
hit: &Hit,
query: &str,
thumbs: &Rc<Thumbs>,
mountpoint: &Path,
) -> gtk4::ListBoxRow {
let row = gtk4::ListBoxRow::new();
row.add_css_class("result-row");
// Focus stays in the search entry; the list only shows the cursor.
row.set_focusable(false);
let content = gtk4::Box::new(gtk4::Orientation::Horizontal, 12);
let slot = thumb_slot(ROW_THUMB, ROW_THUMB, 32, gtk4::ContentFit::Cover);
fill_slot(&slot, hit, thumbs, mountpoint);
content.append(&slot);
let text = gtk4::Box::new(gtk4::Orientation::Vertical, 1);
text.set_hexpand(true);
text.set_valign(gtk4::Align::Center);
let name = gtk4::Label::builder()
.label(highlight::markup(hit.name(), query))
.use_markup(true)
.xalign(0.0)
.ellipsize(gtk4::pango::EllipsizeMode::Middle)
.build();
name.add_css_class("result-name");
text.append(&name);
let location = gtk4::Label::builder()
.label(hit.location())
.xalign(0.0)
.ellipsize(gtk4::pango::EllipsizeMode::Start)
.build();
location.add_css_class("result-location");
text.append(&location);
content.append(&text);
let meta = gtk4::Box::new(gtk4::Orientation::Horizontal, 10);
meta.set_valign(gtk4::Align::Center);
if let Some((icon, class)) = badge_for(hit) {
let badge = gtk4::Image::from_icon_name(icon);
badge.add_css_class("result-badge");
badge.add_css_class(class);
meta.append(&badge);
}
if !hit.is_dir() && hit.size() > 0 {
let size = gtk4::Label::new(Some(&human_bytes(hit.size())));
size.add_css_class("result-meta");
meta.append(&size);
}
if hit.modified() > 0 {
let modified = gtk4::Label::new(Some(&format_age(hit.modified())));
modified.add_css_class("result-meta");
meta.append(&modified);
}
content.append(&meta);
row.set_child(Some(&content));
row
}
/// Let a row be dragged into a file manager or an application as a file.
/// `dropped` runs once a drop was accepted; a cancelled drag changes nothing.
pub(crate) fn attach_drag(row: &gtk4::ListBoxRow, path: PathBuf, dropped: impl Fn() + 'static) {
let source = gtk4::DragSource::new();
source.set_actions(gdk::DragAction::COPY);
let files = gdk::FileList::from_array(&[gio::File::for_path(&path)]);
source.set_content(Some(&gdk::ContentProvider::for_value(&files.to_value())));
let cancelled = Rc::new(Cell::new(false));
let begin = cancelled.clone();
source.connect_drag_begin(move |source, _| {
begin.set(false);
if let Some(widget) = source.widget() {
source.set_icon(Some(&gtk4::WidgetPaintable::new(Some(&widget))), 0, 0);
}
});
let cancel = cancelled.clone();
source.connect_drag_cancel(move |_, _, _| {
cancel.set(true);
false
});
source.connect_drag_end(move |_, _, _| {
if !cancelled.get() {
dropped();
}
});
row.add_controller(source);
}
+697
View File
@@ -0,0 +1,697 @@
//! Thumbnails for the launcher's rows and preview pane.
//!
//! The main thread keeps a small LRU of textures, the keys known to have no
//! thumbnail, and the pictures waiting for one (as weak references, so a row
//! that was rebuilt or dropped keeps nothing alive). Everything that touches
//! a file runs on two worker threads, which try in order:
//!
//! 1. the freedesktop thumbnail cache, valid only when its `Thumb::MTime`
//! matches the file's;
//! 2. for local images, decoding the file itself, and writing the result back
//! to that cache for the next time and for every other application;
//! 3. for Drive images, the daemon's `FileThumbs`, batched after a short quiet
//! period.
//!
//! Workers hand back `gdk::Texture`s, never a `Pixbuf`, which may not cross
//! threads. Each job carries the render epoch it was queued in, and workers
//! skip jobs a newer query has made pointless.
use std::cell::{Cell, RefCell};
use std::collections::{HashMap, HashSet, VecDeque};
use std::io::Write;
use std::os::unix::fs::{DirBuilderExt, OpenOptionsExt};
use std::path::{Path, PathBuf};
use std::rc::{Rc, Weak};
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, UNIX_EPOCH};
use gtk4::gdk_pixbuf::Pixbuf;
use gtk4::prelude::*;
use gtk4::{gdk, glib};
use pdfs_core::control::{FileThumbRequest, Request, Response, is_thumbnail_image_name};
use crate::{Hit, spawn_request};
/// Decoded textures kept in memory. A launcher shows a few dozen rows, so
/// this covers several searches back.
const TEXTURE_CACHE_MAX: usize = 128;
const MISSING_MAX: usize = 1024;
const WORKERS: usize = 2;
/// Edge of the thumbnails this loader decodes and writes: the freedesktop
/// "large" size.
const THUMB_SIZE: i32 = 256;
/// Local images above this size are not decoded here; a thumbnail is not
/// worth reading that much.
const MAX_SOURCE_BYTES: u64 = 64 * 1024 * 1024;
/// Quiet period before Drive misses go to the daemon as one batch, so typing
/// a word asks once for the final result set.
const DAEMON_DEBOUNCE: Duration = Duration::from_millis(150);
const DAEMON_RETRY: Duration = Duration::from_secs(1);
const DAEMON_BATCH: usize = 32;
/// Where a job finds its image.
enum Source {
/// A file on this machine.
Local(PathBuf),
/// A Drive file: the thumbnail cache is keyed by its mounted path, and
/// the daemon is asked on a miss.
Drive {
mounted: PathBuf,
request: FileThumbRequest,
},
/// A thumbnail the daemon made, in its own cache.
Daemon(PathBuf),
}
struct Job {
epoch: u64,
key: String,
source: Source,
}
enum Outcome {
Texture(gdk::Texture),
Missing,
/// Not in the thumbnail cache; the daemon has to make it.
Daemon(FileThumbRequest),
/// Queued for an older render; handed back in case it is wanted again.
Skipped(Source),
}
struct Done {
key: String,
outcome: Outcome,
}
pub(crate) struct Thumbs {
socket: PathBuf,
epoch: Arc<AtomicU64>,
jobs: async_channel::Sender<Job>,
textures: RefCell<HashMap<String, gdk::Texture>>,
/// Least recently used first.
order: RefCell<VecDeque<String>>,
missing: RefCell<HashSet<String>>,
missing_order: RefCell<VecDeque<String>>,
waiters: RefCell<HashMap<String, Vec<glib::WeakRef<gtk4::Picture>>>>,
/// Keys with a worker job or a daemon request outstanding.
working: RefCell<HashSet<String>>,
/// Drive misses waiting for the next batch, by key.
queued: RefCell<HashMap<String, FileThumbRequest>>,
/// Drive items the daemon is working on under `generation`, by key.
sent: RefCell<HashMap<String, FileThumbRequest>>,
generation: Cell<u64>,
reserving: Cell<bool>,
inflight: Cell<bool>,
flush: RefCell<Option<glib::SourceId>>,
}
impl Thumbs {
pub(crate) fn new(socket: PathBuf) -> Rc<Self> {
let epoch = Arc::new(AtomicU64::new(0));
let (jobs, inbox) = async_channel::unbounded::<Job>();
let (done, results) = async_channel::unbounded::<Done>();
for n in 0..WORKERS {
let inbox = inbox.clone();
let done = done.clone();
let epoch = epoch.clone();
let spawned = std::thread::Builder::new()
.name(format!("prompt-thumbs-{n}"))
.spawn(move || work(n, &inbox, &done, &epoch));
if let Err(e) = spawned {
tracing::warn!("cannot start a thumbnail worker: {e}");
}
}
let thumbs = Rc::new(Self {
socket,
epoch,
jobs,
textures: RefCell::new(HashMap::new()),
order: RefCell::new(VecDeque::new()),
missing: RefCell::new(HashSet::new()),
missing_order: RefCell::new(VecDeque::new()),
waiters: RefCell::new(HashMap::new()),
working: RefCell::new(HashSet::new()),
queued: RefCell::new(HashMap::new()),
sent: RefCell::new(HashMap::new()),
generation: Cell::new(0),
reserving: Cell::new(false),
inflight: Cell::new(false),
flush: RefCell::new(None),
});
let weak = Rc::downgrade(&thumbs);
glib::spawn_future_local(async move {
while let Ok(done) = results.recv().await {
let Some(thumbs) = weak.upgrade() else {
break;
};
thumbs.finish(done);
}
});
thumbs
}
/// Paint `hit`'s thumbnail into `picture` now if it is known, or as soon
/// as it is. Folders and non-images keep the slot's icon.
pub(crate) fn want(self: &Rc<Self>, hit: &Hit, mountpoint: &Path, picture: &gtk4::Picture) {
let key = hit.thumb_key();
// The widget name binds the picture to one key: a late reply for
// another hit must not paint into it.
picture.set_widget_name(&key);
if hit.is_dir() || !is_thumbnail_image_name(hit.name()) {
return;
}
if matches!(hit, Hit::Drive(drive) if drive.uid.is_empty()) {
return;
}
let cached = self.textures.borrow().get(&key).cloned();
if let Some(texture) = cached {
self.touch(&key);
paint(picture, &key, &texture);
return;
}
if self.missing.borrow().contains(&key) {
return;
}
self.waiters
.borrow_mut()
.entry(key.clone())
.or_default()
.push(picture.downgrade());
if !self.working.borrow_mut().insert(key.clone()) {
return;
}
let source = match hit {
Hit::Local(local) => Source::Local(PathBuf::from(&local.path)),
Hit::Drive(drive) => Source::Drive {
mounted: hit.fs_path(mountpoint),
request: FileThumbRequest {
uid: drive.uid.clone(),
modified: drive.modified,
name: drive.name.clone(),
},
},
};
self.dispatch(key, source);
}
/// The rows are about to be rebuilt: forget their pictures, and let the
/// workers skip what was queued for them.
pub(crate) fn begin_render(&self) {
self.epoch.fetch_add(1, Ordering::SeqCst);
self.waiters.borrow_mut().clear();
}
/// The rows are rebuilt and have asked for their thumbnails. Drive work
/// for hits that are gone is dropped, and cancelled at the daemon when it
/// had already been sent.
pub(crate) fn end_render(self: &Rc<Self>) {
let wanted: HashSet<String> = self.waiters.borrow().keys().cloned().collect();
let mut dropped = Vec::new();
self.queued.borrow_mut().retain(|key, _| {
let keep = wanted.contains(key);
if !keep {
dropped.push(key.clone());
}
keep
});
let stale = self.sent.borrow().keys().any(|key| !wanted.contains(key));
if stale {
self.cancel_daemon();
// Still-wanted items go back into the next batch, under a fresh
// generation.
let sent: Vec<(String, FileThumbRequest)> = self.sent.borrow_mut().drain().collect();
for (key, request) in sent {
if wanted.contains(&key) {
self.queued.borrow_mut().insert(key, request);
} else {
dropped.push(key);
}
}
self.schedule(DAEMON_DEBOUNCE);
}
let mut working = self.working.borrow_mut();
for key in dropped {
working.remove(&key);
}
}
/// The launcher was dismissed: stop everything.
pub(crate) fn cancel_all(&self) {
self.epoch.fetch_add(1, Ordering::SeqCst);
self.waiters.borrow_mut().clear();
self.working.borrow_mut().clear();
self.queued.borrow_mut().clear();
self.sent.borrow_mut().clear();
if let Some(source) = self.flush.borrow_mut().take() {
source.remove();
}
self.cancel_daemon();
}
fn dispatch(&self, key: String, source: Source) {
let job = Job {
epoch: self.epoch.load(Ordering::SeqCst),
key: key.clone(),
source,
};
if self.jobs.try_send(job).is_err() {
self.working.borrow_mut().remove(&key);
}
}
fn wanted(&self, key: &str) -> bool {
self.waiters.borrow().contains_key(key)
}
fn finish(self: &Rc<Self>, done: Done) {
let Done { key, outcome } = done;
match outcome {
Outcome::Texture(texture) => {
self.working.borrow_mut().remove(&key);
self.store_texture(&key, texture.clone());
let waiters = self.waiters.borrow_mut().remove(&key);
for picture in waiters.into_iter().flatten() {
if let Some(picture) = picture.upgrade() {
paint(&picture, &key, &texture);
}
}
}
Outcome::Missing => {
self.working.borrow_mut().remove(&key);
self.store_missing(key.clone());
self.waiters.borrow_mut().remove(&key);
}
Outcome::Daemon(request) => {
if self.wanted(&key) {
self.queued.borrow_mut().insert(key, request);
self.schedule(DAEMON_DEBOUNCE);
} else {
self.working.borrow_mut().remove(&key);
}
}
Outcome::Skipped(source) => {
// A newer render may want the same image again.
if self.wanted(&key) {
self.dispatch(key, source);
} else {
self.working.borrow_mut().remove(&key);
}
}
}
}
fn touch(&self, key: &str) {
let mut order = self.order.borrow_mut();
if let Some(at) = order.iter().position(|held| held == key) {
order.remove(at);
order.push_back(key.to_string());
}
}
fn store_texture(&self, key: &str, texture: gdk::Texture) {
let mut textures = self.textures.borrow_mut();
if textures.insert(key.to_string(), texture).is_none() {
self.order.borrow_mut().push_back(key.to_string());
} else {
drop(textures);
self.touch(key);
return;
}
let mut order = self.order.borrow_mut();
while order.len() > TEXTURE_CACHE_MAX {
if let Some(old) = order.pop_front() {
textures.remove(&old);
}
}
}
fn store_missing(&self, key: String) {
let mut missing = self.missing.borrow_mut();
let mut order = self.missing_order.borrow_mut();
if missing.insert(key.clone()) {
order.push_back(key);
}
while order.len() > MISSING_MAX {
if let Some(old) = order.pop_front() {
missing.remove(&old);
}
}
}
/// Stop the daemon's work for this launcher. The generation is shared
/// with the app, so this is only sent when something was asked for.
fn cancel_daemon(&self) {
let generation = self.generation.replace(0);
if generation != 0 {
// The reply is not needed; `spawn_request` owns the round-trip.
drop(spawn_request(
self.socket.clone(),
Request::CancelFileThumbs {
generation: generation.saturating_add(1),
},
));
}
}
fn schedule(self: &Rc<Self>, delay: Duration) {
if let Some(source) = self.flush.borrow_mut().take() {
source.remove();
}
let weak = Rc::downgrade(self);
let source = glib::timeout_add_local_once(delay, move || {
if let Some(thumbs) = weak.upgrade() {
thumbs.flush.borrow_mut().take();
thumbs.send_batch();
}
});
*self.flush.borrow_mut() = Some(source);
}
fn send_batch(self: &Rc<Self>) {
if self.inflight.get() || self.queued.borrow().is_empty() {
return;
}
if self.generation.get() == 0 {
self.reserve();
return;
}
let batch: Vec<(String, FileThumbRequest)> = {
let mut queued = self.queued.borrow_mut();
let keys: Vec<String> = queued.keys().take(DAEMON_BATCH).cloned().collect();
keys.into_iter()
.filter_map(|key| queued.remove_entry(&key))
.collect()
};
self.sent.borrow_mut().extend(batch.iter().cloned());
let generation = self.generation.get();
self.inflight.set(true);
let rx = spawn_request(
self.socket.clone(),
Request::FileThumbs {
items: batch.iter().map(|(_, request)| request.clone()).collect(),
generation,
},
);
let weak = Rc::downgrade(self);
glib::spawn_future_local(async move {
let reply = rx.recv().await;
let Some(thumbs) = weak.upgrade() else {
return;
};
thumbs.inflight.set(false);
thumbs.receive(generation, batch, reply);
if !thumbs.queued.borrow().is_empty() {
thumbs.schedule(Duration::ZERO);
}
});
}
fn receive(
self: &Rc<Self>,
generation: u64,
batch: Vec<(String, FileThumbRequest)>,
reply: Result<Result<Response, String>, async_channel::RecvError>,
) {
// Cancelled while in flight: `end_render` or `cancel_all` already
// re-queued what is still wanted. A finished thumbnail is still worth
// painting.
let current = generation == self.generation.get();
match reply {
Ok(Ok(Response::Thumbs { items })) => {
let mut by_uid: HashMap<String, (String, FileThumbRequest)> = batch
.into_iter()
.map(|(key, request)| (request.uid.clone(), (key, request)))
.collect();
let mut again = Vec::new();
for item in items {
let Some((key, request)) = by_uid.remove(&item.uid) else {
continue;
};
if let Some(path) = item.path {
if current {
self.sent.borrow_mut().remove(&key);
}
if self.wanted(&key) {
self.working.borrow_mut().insert(key.clone());
self.dispatch(key, Source::Daemon(PathBuf::from(path)));
} else {
self.working.borrow_mut().remove(&key);
}
} else if !current {
continue;
} else if item.pending {
again.push((key, request));
} else {
self.sent.borrow_mut().remove(&key);
self.working.borrow_mut().remove(&key);
self.store_missing(key.clone());
self.waiters.borrow_mut().remove(&key);
}
}
if current {
again.extend(by_uid.into_values());
self.retry(again);
}
}
Ok(Ok(Response::FileThumbsStale)) if current => {
self.generation.set(0);
self.retry(batch);
}
Ok(Ok(Response::Error { message, .. })) if current => {
tracing::debug!("launcher thumbnails failed: {message}");
self.retry(batch);
}
_ if current => self.retry(batch),
_ => {}
}
}
/// Ask again for items the daemon is still making, after a pause.
fn retry(self: &Rc<Self>, items: Vec<(String, FileThumbRequest)>) {
if items.is_empty() {
return;
}
let weak = Rc::downgrade(self);
glib::timeout_add_local_once(DAEMON_RETRY, move || {
let Some(thumbs) = weak.upgrade() else {
return;
};
for (key, request) in items {
// Gone from `sent` means cancelled or already answered.
if thumbs.sent.borrow_mut().remove(&key).is_some() && thumbs.wanted(&key) {
thumbs.queued.borrow_mut().insert(key, request);
} else {
thumbs.working.borrow_mut().remove(&key);
}
}
thumbs.send_batch();
});
}
fn reserve(self: &Rc<Self>) {
if self.reserving.replace(true) {
return;
}
let rx = spawn_request(self.socket.clone(), Request::ReserveFileThumbGeneration);
let weak: Weak<Self> = Rc::downgrade(self);
glib::spawn_future_local(async move {
let reply = rx.recv().await;
let Some(thumbs) = weak.upgrade() else {
return;
};
thumbs.reserving.set(false);
match reply {
Ok(Ok(Response::FileThumbGeneration { generation })) if generation != 0 => {
thumbs.generation.set(generation);
thumbs.send_batch();
}
_ => thumbs.schedule(DAEMON_RETRY),
}
});
}
}
/// Show `texture` in `picture` if the picture still belongs to `key`, and hide
/// the icon underneath it.
fn paint(picture: &gtk4::Picture, key: &str, texture: &gdk::Texture) {
if picture.widget_name() != key {
return;
}
picture.set_paintable(Some(texture));
picture.set_visible(true);
if let Some(overlay) = picture.parent().and_downcast::<gtk4::Overlay>()
&& let Some(fallback) = overlay.child()
{
fallback.set_visible(false);
}
}
fn work(
index: usize,
inbox: &async_channel::Receiver<Job>,
done: &async_channel::Sender<Done>,
epoch: &AtomicU64,
) {
while let Ok(job) = inbox.recv_blocking() {
// A daemon thumbnail is already made; decoding it is cheap and the
// texture is cached for the next render.
let stale =
!matches!(job.source, Source::Daemon(_)) && job.epoch < epoch.load(Ordering::SeqCst);
let outcome = if stale {
Outcome::Skipped(job.source)
} else {
load(index, job.source)
};
if done
.send_blocking(Done {
key: job.key,
outcome,
})
.is_err()
{
break;
}
}
}
fn load(worker: usize, source: Source) -> Outcome {
match source {
Source::Local(path) => {
let Ok(meta) = std::fs::metadata(&path) else {
return Outcome::Missing;
};
let mtime = meta
.modified()
.ok()
.and_then(|at| at.duration_since(UNIX_EPOCH).ok())
.map_or(0, |at| at.as_secs() as i64);
if let Some(pixbuf) = cached(&path, mtime) {
return texture(&pixbuf);
}
if meta.len() > MAX_SOURCE_BYTES {
return Outcome::Missing;
}
let Some((pixbuf, scaled)) = decode(&path) else {
return Outcome::Missing;
};
// An image that already fits is its own thumbnail.
if scaled && let Err(e) = write_cached(worker, &path, mtime, &pixbuf) {
tracing::debug!("cannot cache a thumbnail of {}: {e}", path.display());
}
texture(&pixbuf)
}
Source::Drive { mounted, request } => match cached(&mounted, request.modified) {
Some(pixbuf) => texture(&pixbuf),
None => Outcome::Daemon(request),
},
Source::Daemon(path) => match gdk::Texture::from_filename(&path) {
Ok(texture) => Outcome::Texture(texture),
Err(e) => {
tracing::debug!("cannot decode {}: {e}", path.display());
Outcome::Missing
}
},
}
}
/// `$XDG_CACHE_HOME/thumbnails`.
fn thumbnail_root() -> PathBuf {
glib::user_cache_dir().join("thumbnails")
}
/// The URI and its MD5, which name a file's entry in the thumbnail cache.
fn cache_name(path: &Path) -> Option<(String, String)> {
let uri = glib::filename_to_uri(path, None).ok()?.to_string();
let md5 = glib::compute_checksum_for_data(glib::ChecksumType::Md5, uri.as_bytes())?.to_string();
Some((uri, md5))
}
/// A thumbnail from the shared cache, if one was made for this version of
/// the file.
fn cached(path: &Path, mtime: i64) -> Option<Pixbuf> {
let (_, md5) = cache_name(path)?;
let root = thumbnail_root();
["large", "x-large", "normal"].iter().find_map(|size| {
let pixbuf = Pixbuf::from_file(root.join(size).join(format!("{md5}.png"))).ok()?;
let stamp = pixbuf.option("tEXt::Thumb::MTime")?;
(stamp.parse::<i64>().ok()? == mtime).then_some(pixbuf)
})
}
/// Decode a local image at most [`THUMB_SIZE`] on its longer edge, upright.
/// The bool says whether it had to be scaled down.
fn decode(path: &Path) -> Option<(Pixbuf, bool)> {
let (_, width, height) = Pixbuf::file_info(path)?;
let scaled = width > THUMB_SIZE || height > THUMB_SIZE;
let pixbuf = if scaled {
Pixbuf::from_file_at_scale(path, THUMB_SIZE, THUMB_SIZE, true)
} else {
Pixbuf::from_file(path)
}
.ok()?;
let pixbuf = pixbuf.apply_embedded_orientation().unwrap_or(pixbuf);
Some((pixbuf, scaled))
}
/// Store a thumbnail as the freedesktop spec asks: in `large/`, with
/// `Thumb::URI` and `Thumb::MTime`, readable only by the user, and written
/// through a temporary file so another reader never sees half of it.
fn write_cached(worker: usize, path: &Path, mtime: i64, pixbuf: &Pixbuf) -> std::io::Result<()> {
let (uri, md5) = cache_name(path).ok_or_else(|| std::io::Error::other("no file URI"))?;
let dir = thumbnail_root().join("large");
std::fs::DirBuilder::new()
.recursive(true)
.mode(0o700)
.create(&dir)?;
let mtime = mtime.to_string();
let png = pixbuf
.save_to_bufferv(
"png",
&[
("tEXt::Thumb::URI", uri.as_str()),
("tEXt::Thumb::MTime", mtime.as_str()),
],
)
.map_err(std::io::Error::other)?;
let tmp = dir.join(format!(
"{md5}.png.pdfs-{}-{worker}.tmp",
std::process::id()
));
let written = std::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.mode(0o600)
.open(&tmp)
.and_then(|mut file| file.write_all(&png))
.and_then(|()| std::fs::rename(&tmp, dir.join(format!("{md5}.png"))));
if written.is_err() {
let _ = std::fs::remove_file(&tmp);
}
written
}
/// A `Pixbuf` may not leave its thread; its pixels, as a memory texture, can.
fn texture(pixbuf: &Pixbuf) -> Outcome {
let format = if pixbuf.has_alpha() {
gdk::MemoryFormat::R8g8b8a8
} else {
gdk::MemoryFormat::R8g8b8
};
let texture = gdk::MemoryTexture::new(
pixbuf.width(),
pixbuf.height(),
format,
&pixbuf.read_pixel_bytes(),
pixbuf.rowstride() as usize,
);
Outcome::Texture(texture.upcast())
}
+13
View File
@@ -166,6 +166,15 @@ fn poll_state(socket: &Path, default_mountpoint: &Path) -> DriveState {
// No daemon: describe login state instead so the menu is still useful. // No daemon: describe login state instead so the menu is still useful.
Err(_) => { Err(_) => {
let (line, phase) = match auth::load() { let (line, phase) = match auth::load() {
Ok(s) if matches!(auth::key_state(), Ok(auth::KeyState::Locked)) => (
// Translators: {username} is the Proton account name. The
// account has a mailbox password that has to be entered.
gettext_f(
"Signed in as {username} — locked",
&[("username", &s.username)],
),
Phase::Attention,
),
Ok(s) => ( Ok(s) => (
// Translators: {username} is the Proton account name. // Translators: {username} is the Proton account name.
gettext_f( gettext_f(
@@ -476,6 +485,10 @@ fn main() -> Result<()> {
return Ok(()); return Ok(());
} }
// The socket lives in the state dir, which nothing has made yet when the app
// starts the tray on a fresh install (B194).
dirs.ensure().context("create the app's directories")?;
// Ensure only one instance of the tray runs. // Ensure only one instance of the tray runs.
let tray_sock = dirs.tray_socket(); let tray_sock = dirs.tray_socket();
if std::os::unix::net::UnixStream::connect(&tray_sock).is_ok() { if std::os::unix::net::UnixStream::connect(&tray_sock).is_ok() {
File diff suppressed because it is too large. Load diff
+158
View File
@@ -0,0 +1,158 @@
//! The bundled resources and the optional Proton theme, shared by `pdfs-app`
//! and `pdfs-prompt` so both paint the same way from the same preference.
use std::cell::RefCell;
use adw::prelude::*;
use gtk4::{gio, glib};
/// Register the bundled GResources (custom icons, the stylesheets) and load
/// `stylesheet`, the front end's own sheet. The Proton theme is a separate
/// provider, applied by [`set_proton_theme`] from the saved preference.
pub(crate) fn load_resources(stylesheet: &str) {
let bytes = include_bytes!(concat!(env!("OUT_DIR"), "/pdfs.gresource"));
let resource_data = glib::Bytes::from_static(bytes);
if let Ok(resource) = gio::Resource::from_data(&resource_data) {
gio::resources_register(&resource);
} else {
tracing::error!("failed to load gresource bundle");
}
let Some(display) = gtk4::gdk::Display::default() else {
return;
};
let provider = gtk4::CssProvider::new();
provider.load_from_resource(stylesheet);
// Above the user's own gtk.css, which often imports a whole theme: its
// `button { padding: 6px 10px; min-width: 16px }` would otherwise pad every
// photo tile and push the grid past the window. Every rule in the sheet is
// scoped to the app's own classes, so nothing else the user styled changes.
// One above the Proton theme too, so the app's rules refine it.
gtk4::style_context_add_provider_for_display(
&display,
&provider,
gtk4::STYLE_PROVIDER_PRIORITY_USER + 2,
);
gtk4::IconTheme::for_display(&display).add_resource_path("/de/nils/protondrivelinux/icons");
}
thread_local! {
/// The Proton theme provider while it is installed. Kept so turning the
/// preference off can remove exactly what turning it on added.
static PROTON_THEME: RefCell<Option<gtk4::CssProvider>> = const { RefCell::new(None) };
}
/// Whether the Proton theme is on.
pub(crate) fn proton_theme_active() -> bool {
PROTON_THEME.with(|slot| slot.borrow().is_some())
}
/// libadwaita's own stylesheet, bundled in the library.
const ADWAITA_STYLESHEET: &str = "/org/gnome/Adwaita/styles/gtk.css";
/// The Proton palette laid over it, from our own bundle.
const PROTON_PALETTE: &str = "/de/nils/protondrivelinux/proton-theme.css";
/// Make the theme provider's `@media (prefers-color-scheme)` and
/// `(prefers-contrast)` rules match what libadwaita shows, now and whenever it
/// changes. A provider evaluates those itself, and a new one assumes light.
///
/// The properties are set by name because they only exist since GTK 4.20, and
/// the app still runs on older GTK, whose libadwaita has no such rules anyway.
fn follow_style_manager(provider: &gtk4::CssProvider) {
fn set_enum(provider: &gtk4::CssProvider, property: &str, nick: &str) {
let Some(pspec) = provider.find_property(property) else {
return;
};
if let Some(value) = glib::EnumClass::with_type(pspec.value_type())
.and_then(|class| class.to_value_by_nick(nick))
{
provider.set_property_from_value(property, &value);
}
}
fn apply(provider: &gtk4::CssProvider, manager: &adw::StyleManager) {
let scheme = if manager.is_dark() { "dark" } else { "light" };
set_enum(provider, "prefers-color-scheme", scheme);
let contrast = if manager.is_high_contrast() {
"more"
} else {
"no-preference"
};
set_enum(provider, "prefers-contrast", contrast);
}
let manager = adw::StyleManager::default();
apply(provider, &manager);
// Weak, so a provider the preference removed isn't kept alive; its
// handlers then do nothing.
let weak = provider.downgrade();
let weak_dark = weak.clone();
manager.connect_dark_notify(move |manager| {
if let Some(provider) = weak_dark.upgrade() {
apply(&provider, manager);
}
});
manager.connect_high_contrast_notify(move |manager| {
if let Some(provider) = weak.upgrade() {
apply(&provider, manager);
}
});
}
/// Paint the window in the Proton web apps' colours (`on`) or follow the
/// system theme and accent colour.
///
/// Setting libadwaita's colour variables alone isn't enough: a GTK theme the
/// user's `gtk.css` imports (Catppuccin, Fluent, ...) sits above the app and
/// hard-codes its own accent into hundreds of rules. So the theme provider,
/// one above the user's CSS, first resets every property, then re-applies
/// libadwaita's whole stylesheet and finally the Proton palette, which leaves
/// nothing of the user's theme inside the app.
pub(crate) fn set_proton_theme(on: bool) {
let Some(display) = gtk4::gdk::Display::default() else {
return;
};
PROTON_THEME.with(|slot| {
let mut slot = slot.borrow_mut();
match (on, slot.as_ref()) {
(true, None) => {
let stylesheet = |path| {
gio::resources_lookup_data(path, gio::ResourceLookupFlags::NONE)
.map(|bytes| String::from_utf8_lossy(&bytes).into_owned())
};
let palette = stylesheet(PROTON_PALETTE).unwrap_or_default();
let css = match stylesheet(ADWAITA_STYLESHEET) {
// Its images are named relative to the sheet, which a
// string loaded here doesn't have.
Ok(adwaita) => format!(
"* {{ all: unset; }}\n{}\n{palette}",
adwaita.replace(
"url(\"assets/",
"url(\"resource:///org/gnome/Adwaita/styles/assets/"
)
),
// Still recolours everything that follows libadwaita's
// variables; only a custom theme's own rules show through.
Err(e) => {
tracing::warn!("libadwaita stylesheet not found, only recolouring: {e}");
palette
}
};
let provider = gtk4::CssProvider::new();
follow_style_manager(&provider);
provider.load_from_string(&css);
gtk4::style_context_add_provider_for_display(
&display,
&provider,
gtk4::STYLE_PROVIDER_PRIORITY_USER + 1,
);
*slot = Some(provider);
}
(false, Some(provider)) => {
gtk4::style_context_remove_provider_for_display(&display, provider);
*slot = None;
}
_ => {}
}
});
}
+614 -351
View File
@@ -1,379 +1,642 @@
# Proton Drive Linux Client: Architecture Specification # Architecture
This document provides a deep, comprehensive architectural description of the Proton Drive Linux Client (`pdfs`). It specifies the design patterns, data flows, thread models, and subsystem dependencies that govern the virtual filesystem (FUSE), database persistence, cache management, and two-way sync engine. How the Proton Drive Linux client is built: the crates, the local tree behind the mount, the read
and write paths, the queue that sends changes to Drive, the sync engine for synced folders, the
control protocol, the threads, and what the client writes to disk in plaintext. It is written for
contributors and for anyone auditing the client. User documentation starts at
[docs/README.md](README.md).
Numbers quoted here (thread counts, timeouts, limits) are constants in the source. When they
disagree, the source is right.
- [1. Crates and processes](#1-crates-and-processes)
- [2. The local tree](#2-the-local-tree)
- [3. Read path](#3-read-path)
- [4. Write path](#4-write-path)
- [5. The queue and the drain](#5-the-queue-and-the-drain)
- [6. Remote changes](#6-remote-changes)
- [7. Synced folders](#7-synced-folders)
- [8. Control protocol](#8-control-protocol)
- [9. Threads](#9-threads)
- [10. Threat model: what this client writes to disk in plaintext](#10-threat-model-what-this-client-writes-to-disk-in-plaintext)
- [11. Sign-in and CAPTCHA](#11-sign-in-and-captcha)
- [12. Feature notes](#12-feature-notes)
- [13. Where it is going](#13-where-it-is-going)
--- ---
## 1. Subsystem Overview & Crate Topology ## 1. Crates and processes
The application is modularized into four workspace crates, dividing core library logic, filesystem mounting, control-socket IPC, and front-ends.
```mermaid ```mermaid
graph TD graph TD
%% Crates CLI["pdfs-cli: pdfs, pdfs daemon"]
CLI["crates/pdfs-cli (CLI & Daemon Entrypoint)"] GUI["pdfs-gui: pdfs-app, pdfs-tray, pdfs-prompt"]
GUI["crates/pdfs-gui (GTK Front-end)"] FUSE["pdfs-fuse: the daemon"]
FUSE["crates/pdfs-fuse (FUSE VFS & Sync Loop)"] CORE["pdfs-core: database, cache, config, control protocol"]
CORE["crates/pdfs-core (DB, Cache & IPC Protocol)"] SDK["proton-drive-rs: Drive API and cryptography"]
SDK["proton-sdk-rs (Proton Drive API & Cryptography)"]
%% Dependencies
CLI --> FUSE CLI --> FUSE
CLI --> CORE
GUI --> CORE GUI --> CORE
FUSE --> CORE FUSE --> CORE
FUSE --> SDK
CORE --> SDK CORE --> SDK
CLI -.->|Unix Socket IPC| FUSE CLI -.->|control socket| FUSE
GUI -.->|Unix Socket IPC| FUSE GUI -.->|control socket| FUSE
``` ```
### Crate Division & Responsibility Matrix | Crate | Holds |
| Crate | Primary Role | Key Components | State Management |
|---|---|---|---|
| [`pdfs-core`](../crates/pdfs-core) | Core Infrastructure & Services | Cache bookkeeping, database migrations/schemas, IPC protocol payloads, and shared search relevance scoring. | Holds the unified SQLite DB (`Db`) connection and the on-disk cache metadata (`ContentCache`). |
| [`pdfs-fuse`](../crates/pdfs-fuse) | VFS Layer & Reconciliation | FUSE callbacks, background upload queue (`drain`), two-way sync runner. | Manages in-memory inode maps (`State`), active descriptors (`WriteHandle`), and background task threads. |
| [`pdfs-cli`](../crates/pdfs-cli) | Command Line Interface | Command routing, daemon launcher, IPC client wrapper. | Stateless; communicates with daemon over IPC control socket. |
| [`pdfs-gui`](../crates/pdfs-gui) | Graphical Interface | GTK pages, tray, and the resident quick-search prompt. | Keeps UI state only; all durable state and Drive access remain behind the IPC socket. |
---
## 2. In-Memory VFS State & File Operations
The VFS layer implements FUSE via the `fuser` crate. Because the remote storage contains base64-encoded file keys and requires cryptographic envelope parsing, raw listings and inodes are virtualized and stored in a local state directory.
### Inode and Path Resolution
* **In-Memory Cache (`State`):** Maps FUSE `u64` inodes to Proton Drive `NodeUid`s.
* **Database Row Mapping (`StoredNode`):** Stores directories, sizes, and timestamps.
* **On-Demand Loading (`ensure_children`):** If a directory is accessed, the daemon checks its database `listed` flag. If `listed = 0`, it triggers an API call to fetch remote nodes, populates the DB and in-memory caches, and returns.
```mermaid
sequenceDiagram
autonumber
actor User as Kernel (VFS Call)
participant FUSE as pdfs-fuse VFS
participant ST as State (In-Memory)
participant DB as Db (SQLite)
participant API as Proton API Client
User->>FUSE: lookup(parent_ino, "report.pdf")
FUSE->>ST: children.get(&parent_ino)
alt Parent listing is resident in-memory
ST-->>FUSE: returns child_ino
else Listing missing in-memory
FUSE->>DB: children_if_listed(parent_uid)
alt Parent marked listed in DB
DB-->>FUSE: returns child node metadata list
FUSE->>ST: intern_from_db() and populate children cache
else Parent not listed in DB
FUSE->>API: enumerate_folder_children_node_uids()
API-->>FUSE: list of UIDs
FUSE->>API: enumerate_nodes(uids)
API-->>FUSE: list of decrypted Nodes
FUSE->>DB: upsert_nodes() & set_listed(true)
FUSE->>ST: intern_batch() and populate children cache
end
end
FUSE-->>User: returns child inode metadata (attributes, TTL)
```
### Access Classification and Enforcement
Every `Entry` carries an `Access` (`Owner | Editor | Viewer | Unknown`), inherited from its parent at intern time rather than resolved per node — a child is always interned from its parent's listing, so one edge lookup answers it. A known share root takes its access from the persisted `share_access` table instead, which is also what makes the classification correct offline.
Two rules decide the cases inheritance cannot:
* **Not under a share, no role → `Owner` (fail open).** My Files and device folders are owned content; regressing this denies ordinary writes.
* **Under a share with no usable role → `Viewer` (fail closed).** An unrecognised permission mask is never degraded into a guess.
A node whose parent is not resident is the awkward case: a device folder's parent is the device root, which is never persisted as a node, so the whole subtree hydrates parentless. It resolves the way the persisted authority (`Db::effective_node_access`) does — a recorded share row above it wins, otherwise fail open on the mount's own volume and closed on a foreign one (`docs/BUGS.md` B79).
Enforcement is three layers, and only the third closes B34:
1. **Mode bits.** `attr()` returns `0o555`/`0o444` for a non-writable entry, and both mount paths set `MountOption::DefaultPermissions`, so the *kernel* refuses `open(O_WRONLY)`, `access(W_OK)` and namespace operations for any non-root process.
2. **Handler gates.** `EACCES` in `create`/`mkdir`/`unlink`/`rmdir`/`rename` (parent-writable; both parents for a rename) and in `open`-for-write/`write`/`setattr`/`fallocate`. Covers root and stale attribute TTLs.
3. **Queue guards.** `Core::require_uid_writable` admits a mutation only when the persisted authority *and* every live inode space agree the uid is writable. Nothing reaches `pending_op` otherwise, so the perpetual failing drain that B34 describes cannot occur even if a handler check is missed. The intersection is across mounts because a uid may be resident in more than one inode space.
`EACCES`, not `EROFS`, for a read-only subtree inside a read-write mount: `EROFS` means "read-only filesystem" and misleads the heuristics in `cp`, `rsync` and `git`.
### Local Locations
`mount` is a presentation table: one row per local place this client occupies — the primary My Files session, and each device folder in mirror or on-demand mode. `sync_folder` remains the sync engine's own table (`sync_entry` is FK'd to it) and `MountKind::Device { sync_folder_id }` is the join. `Request::ListLocations` serves the table to `pdfs locations` and the GUI's Locations page.
A mirror folder is a plain local directory with no FUSE session, so a row can legitimately describe a location that is not mounted — which is why the page is called *Locations* rather than *Mounts*. The primary mountpoint stays in `AppConfig`, with its `mount` row written at daemon start as a projection of it, so there are not two sources of truth for that path.
---
## 3. Read Path & Block Caching Pipeline
Read requests are parallelized and served in blocks of size `BLOCK_SIZE` (4 MiB).
* **Unpinned files (Streaming):** Avoids writing full files to disk to preserve storage. Instead, blocks are kept in a fixed-size `stream_ring` (in-memory ring cache) and evicted immediately.
* **Pinned files (Persistent):** Block downloads are saved directly to `ContentCache` on disk.
* **Read-Ahead:** The reader thread spawns asynchronous tasks to pre-fetch upcoming blocks.
```mermaid
sequenceDiagram
autonumber
actor Kernel as Kernel Read (offset, size)
participant FUSE as pdfs-fuse VFS
participant Cache as ContentCache (Local Disk)
participant Ring as StreamRing (In-Memory)
participant API as Proton API Client
Kernel->>FUSE: read(ino, fh, offset, size)
FUSE->>FUSE: calculate block indices [first_block..last_block]
loop For each block index
alt Block in disk Cache (Pinned/Cache hit)
FUSE->>Cache: read_block(block_idx)
Cache-->>FUSE: block bytes
else Block in stream ring cache
FUSE->>Ring: get(block_idx)
Ring-->>FUSE: block bytes
else Block Cache Miss
FUSE->>API: download_range(offset, len)
API-->>FUSE: decrypted block bytes
alt caching_enabled
FUSE->>Cache: store_block(block_idx)
else streaming_only
FUSE->>Ring: insert(block_idx)
end
end
end
FUSE->>FUSE: stitch blocks and slice to offset/size
FUSE-->>Kernel: return data buffer
```
---
## 4. Write Path & Staging/Draining Pipeline
Because Proton Drive does not support partial byte writes, modified files must be uploaded as whole new revisions.
1. **Staging writes (`WriteHandle`):** Writes are stored locally in a `scratch` file. The daemon tracks modified regions using `Intervals` (which holds ranges of edited bytes).
2. **Close/Release (`queue_revision`):** When the application closes the file descriptor, the daemon:
- Fetches any untouched gaps from the remote base file to compile the full file.
- Durably publishes the scratch data and authored-range sidecar into `staging` under a `{uid}-{millis}-{counter}` name. Temporary data and metadata are synced before atomic rename, and their directory is synced before the source is removed.
- Transactionally queues or supersedes a pending database operation (`PendingOp`), so insertion failure cannot erase the previously acknowledged upload. For `OP_REVISION` ops, execution is debounced to give rapid follow-up writes (e.g. `aria2c` preallocation followed by writing) time to supersede the staged blob before network transmission. The debounce is adaptive (`Core::revision_debounce`): it starts at `DRAIN_REVISION_DEBOUNCE = 2s` and, once a node has been uploaded, widens toward how long that upload actually took, bounded by `DRAIN_REVISION_DEBOUNCE_MAX = 60s`. A file saved faster than it can be sent therefore supersedes in the queue rather than mid-upload.
- Signals `Core::cancel_upload` for the node before touching the queue. A drain worker may already be reading the blob this write supersedes; the flag is read by the upload's `CountingReader`, which refuses the SDK's next block. `queue_trash` and `discard_queued_ops` do the same, because they unlink the blob outright.
3. **Async Drain Threads (`run_pending_drain`):** `DRAIN_WORKERS = 3` background workers share the operations queue. They pick work up through `Db::claim_next_due_op`, which marks the row `claimed_at` in the same transaction that selects it and excludes any op whose uid another worker already holds — ordering only has to hold *per node*, and that exclusion is what guarantees it. Worker 0 additionally runs the queue's idle chores (LRU touch flush, `recover_fsynced_writes`). A claim is process-local state: the single-writer `flock` means one found at open belongs to a crashed run, and `Db::clear_op_claims` drops the lot so those ops are not invisible forever. Each worker handles revision uploads, resolves conflicts, and cleans up staging files. Upon landing a revision upload (`refresh_after_upload`), it rebaselines both still-queued ops (`rebaseline_pending`) and open write handles targeting the same node to prevent false self-conflict copies on subsequent writes.
```mermaid
sequenceDiagram
autonumber
actor Kernel as Kernel Write (fh, offset, data)
participant FUSE as pdfs-fuse VFS
participant WH as WriteHandle (Scratch File)
participant DB as Db (SQLite Queue)
participant DR as Drain Thread
participant API as Proton API Client
Kernel->>FUSE: write(ino, fh, offset, data)
FUSE->>WH: write_at(offset, data)
FUSE->>WH: update written intervals
FUSE-->>Kernel: return bytes_written
Note over Kernel, FUSE: Application closes file (close(2))
Kernel->>FUSE: release(fh)
FUSE->>FUSE: fill_gaps() (fetch untouched remote ranges)
FUSE->>FUSE: move scratch file to staging directory
FUSE->>DB: enqueue_op(OP_REVISION, staged_path, meta)
FUSE->>FUSE: record_pending_write() (update size/mtime in memory & DB)
FUSE-->>Kernel: return success (async release)
Note over DB, DR: Background Queue Processing
DR->>DB: next_due_op()
DB-->>DR: return OP_REVISION
DR->>API: upload_new_revision_from(staged_path)
API-->>DR: return new node revision metadata
DR->>DB: delete_op()
DR->>FUSE: refresh_after_upload() (sync local metadata with server time)
```
---
## 5. Sync Engine (Two-Way Reconciliation)
The sync engine handles offline-capable, bidirectional synchronization between the local disk and Proton Drive for directories marked in `mirror` mode.
### Lifecycle of a Sync Pass
1. **Walk Local:** Walks the local directory tree recursively, scanning sizes and modification times while carrying a completeness result. A `readdir`, metadata, permission, or transient I/O failure makes the pass non-destructive instead of turning omitted paths into deletions.
2. **Walk Remote:** Walks the remote database representation. If remote file modification times are updated, it calls the API to decrypt their sizes.
3. **Load Baseline:** Loads the `sync_entry` database table, which contains the snapshot of both sides during the *last successful sync*.
4. **Permutation Diffing:** The loop compares the three states (`local`, `remote`, `baseline`) to classify items:
```mermaid
graph TD
%% States
Classify{"Classify (Local, Remote, Baseline)"}
%% Logic Rules
Classify -->|Both Sides Match| Match["No-Op (In Sync)"]
Classify -->|Local Changed, Remote Untouched| Upload["Upload Revision"]
Classify -->|Remote Changed, Local Untouched| Download["Download Revision"]
Classify -->|Both Sides Changed| Conflict["Conflict Copy (Local renamed to 'sync-conflict', remote downloaded)"]
Classify -->|Local Deleted, Remote Untouched| RemoteDelete["Trash Remote Node"]
Classify -->|Remote Deleted, Local Untouched| LocalDelete["Delete Local File"]
Classify -->|New Local File, No Remote/Baseline| UploadNew["Upload New Node"]
Classify -->|New Remote File, No Local/Baseline| DownloadNew["Download New Node"]
```
5. **Depth-Ascending Batching:** Folders are processed first to ensure hierarchies exist before files are placed. Work is executed concurrently up to a set limit.
6. **Safety Gate and Settle:** Destructive plans are rejected when the scan is incomplete. Total-wipe protection applies to every non-empty baseline, including a single-entry folder. On success, baseline entries are upserted, timestamps updated, and any pending mode switches (e.g. going on-demand) are evaluated. Failed local deletion or conflict preservation retains the previous baseline and prevents remote content from overwriting the local source.
---
## 6. IPC Socket Protocol
The CLI and GUI front-ends do not access database files or make network calls directly. They communicate with the background daemon process over a Unix domain socket.
* **Transport:** IPC over Unix Stream Socket.
* **Framing:** Line-delimited JSON payloads.
* **Resource limits:** Requests are limited to 1 MiB, must end with a newline, and must arrive within 10 seconds. At most 64 connection handlers are active at once.
* **Control Protocol:**
* Client sends a single JSON line (`Request`).
* Daemon parses, handles the request, and replies with a single JSON line (`Response`).
* Timeout durations are separated: **2 seconds** for writes (avoids hangs on defunct sockets) and **120 seconds** for reads (accommodates heavy transfers).
### Unified Search
`SearchV2` is the shared search boundary for the resident prompt. One request carries the query, result limit, requested sources (`Drive`, `Local`), and content kind. The daemon queries Drive metadata and the local home-directory index, then applies [`pdfs_core::search::relevance_score`](../crates/pdfs-core/src/search.rs) to both result sets. Exact and prefix matches rank above substring, abbreviation, and bounded typo matches; all query terms must match either the basename or parent path. Returning scores on one scale lets the GUI merge both sources into a deterministic **Best matches** list.
The prompt is a single-instance GTK application. It retains and hides its window between activations, resets its query state when summoned, and ignores stale asynchronous replies. Folder and streamable audio/video results open through the FUSE mount so applications can issue range reads; ordinary Drive files use the daemon's materialize-then-open path.
---
## 7. Subsystem Interaction & Thread Map
The background daemon relies on the following thread topology:
1. **Main Thread / Dispatch Loop:** Blocks on `fuser::Session` loop. Reads kernel FUSE events and hands off network-bound VFS work to the FUSE workers pool.
2. **FUSE Workers Pool (11 threads, two lanes):** Bounded thread pool handling network operations. Split into 3 threads reserved for metadata (`lookup`, `readdir`) and 8 general threads that serve transfers (block reads) and fall back to metadata when no transfer is waiting. Reserved threads never accept a transfer — that is what keeps a directory listing from queuing behind saturated downloads (audit A6).
3. **IPC listener Thread:** Listens on the Unix socket and admits at most 64 concurrent, timeout-bound handlers for front-end status/configuration requests.
4. **Sync Engine Loop Thread:** Serializes sync runs. Wakes on debounced local inotify filesystem changes, remote polling intervals, or manual user requests.
5. **Drain Queue Worker Thread:** Processes staged writes (`PendingOp`) sequentially, uploading revisions and retrying with exponential backoff on failures.
6. **Conflict Sweep Thread (`pdfs-conflict-sweep`, optional):** Reconciles leftover `(sync-conflict …)` copies — 30 s warmup, then one pass every 5 minutes. A copy proven identical to its live sibling (equal size **and** equal `content_sha1`) is redundant; anything it cannot prove identical is surfaced to the activity feed and left alone. Governed by `AppConfig.conflict_sweep` / `PDFS_CONFLICT_SWEEP` (`SweepMode`): **report-only by default**, `off` skips the thread entirely, and only `enforce` lets it trash. Because it deletes, the enforcing path re-verifies revision id, size, digest, name, parent, queued ops and open handles immediately before acting rather than trusting its own pass-start snapshot. See `docs/BUGS.md` B69 and B71.
---
## 8. Threat Model: What This Client Writes to Disk in Plaintext
Proton Drive is zero-knowledge: the server never holds the keys to your content. That property ends at this daemon. Serving a remote file through a POSIX filesystem means producing plaintext, and serving it *quickly* means keeping some of that plaintext around. This section states exactly what lands on disk, because the guarantee users infer from "zero-knowledge" is stronger than the one a files-on-demand client can offer locally.
**The short version: the cache and state directories hold decrypted content and decrypted metadata, and this client assumes the disk underneath them is encrypted (LUKS, or an encrypted home).** On an unencrypted disk, an attacker with the powered-off machine can read cached file content and the full name/structure of your Drive without ever touching your password.
### 8.1 Decrypted content
Everything under `$XDG_CACHE_HOME/<app>/content/` is plaintext:
| Path | Holds | Lifetime |
|---|---|---|
| `<uid>` blobs | Whole decrypted files (pinned files, opened files) | Until LRU eviction or budget purge |
| `blocks/` | Decrypted 4 MiB block ranges of partially-read files | Until LRU eviction |
| `thumbs/` | Decrypted thumbnails and previews | Until LRU eviction |
| `scratch/` | In-progress writes from open file handles | Until `release`, or rescued at next open |
| `staging/` | Released writes awaiting upload | **Until the upload lands** |
| `recovery/` | `fsync`ed writes rescued from an unclean shutdown | **Until replayed into `staging/`** |
`staging/` and `recovery/` deserve separate attention: unlike the cache directories, they are not a copy of something the server already has. They hold user-authored content that may exist **nowhere else yet**, which is why they are deliberately never cleared on startup (§4, and audit A2). They are simultaneously the most sensitive thing on disk and the thing that must not be deleted to reclaim space.
### 8.2 Decrypted metadata
`$XDG_STATE_HOME/<app>/cache.db` is a plain SQLite database containing **decrypted node names**, the folder hierarchy, sizes, timestamps, a trigram full-text index over those names, the activity log, and the photos timeline. It is not evictable and not budgeted — it is the persistence layer the in-memory tree rehydrates from (§2).
Filename and directory-structure confidentiality is an explicit part of Proton Drive's model (each folder's manifest is encrypted server-side). This database is where that property is spent locally: it is a queryable, plaintext index of your entire Drive, and it survives cache purges. A `PurgeCache` clears content, not this.
### 8.3 What is *not* written in plaintext
Credentials. The session blob — access and refresh tokens, and the key material needed to resume unattended — lives only in the OS keyring via libsecret (`auth.rs`), never on disk in cleartext. `config.json` and `pins.json` hold settings and node uids, no secrets.
Note that the *control socket* is a credential of a different kind: anything that can connect to `control.sock` can drive the daemon — list and read paths, upload, trash, create share links — without touching the keyring at all. See §8.5.
### 8.4 Memory, swap, and the page cache
Two exposures this client does **not** currently mitigate, stated plainly rather than left implied:
- **Swap.** Content keys, session keys, and decrypted buffers live in ordinary heap memory. Nothing calls `mlock(2)`, so under memory pressure they may be paged out. Raising `LimitMEMLOCK` in the systemd unit would *not* change this — there is no locking to permit. The effective mitigation is encrypted swap (dm-crypt / `systemd-cryptsetup`), which is standard on a LUKS install.
- **Kernel page cache.** Plaintext returned through FUSE is cached by the kernel like any other file data, and is likewise swappable. Defeating this would mean `direct_io` on every read, forfeiting the readahead and caching that make the mount usable. The trade is taken deliberately in favour of performance.
### 8.5 Enforced ownership and file modes
On every start, `AppDirs::ensure` verifies that the state, cache, and configuration paths are real directories owned by the effective user, rejects symlink substitution or the wrong owner, and enforces mode `0700`. Both Unix sockets are changed to `0600` immediately after binding; failure is fatal rather than falling back to an unguarded daemon.
These checks protect the artifacts even when the surrounding home or XDG parent is traversable:
| Artifact | Why access is restricted |
|---|---| |---|---|
| `content/` | Another local user can read cached plaintext file content | | [`pdfs-core`](../crates/pdfs-core) | Authentication and keyring (`auth`), configuration and directories (`config`), the SQLite schema and queries (`db/`), the content cache (`cache`), the control protocol (`control`), search scoring, ignore rules, Google Takeout parsing, the machine profile |
| `cache.db` | Another local user can read the full decrypted name/structure index | | [`pdfs-fuse`](../crates/pdfs-fuse) | The daemon: FUSE handlers (`filesystem`), the tree (`state`), reads (`reads`), the queue and the drain (`queue`, `drain`, `upload`), the link state (`link`), remote events (`background`), the sync engine (`sync`, `sync/`), moves between locations (`relocate`), control handlers (`control`), push events (`events`), photos, sharing, devices, the supervisor, and the simulation tests (`sim/`) |
| `control.sock` | **Another local user can drive the daemon**: enumerate, read, upload, trash, create public share links | | [`pdfs-cli`](../crates/pdfs-cli) | The `pdfs` binary. `pdfs daemon` runs the daemon in-process; every other command is a control-socket client |
| [`pdfs-gui`](../crates/pdfs-gui) | `pdfs-app`, `pdfs-tray` and `pdfs-prompt`. Pages in `src/app/pages/`, widgets in `src/app/widgets/`, CSS and icons in `resources/` |
The socket is the sharpest of the three, because it is an authority boundary rather than a data one — connecting to it confers the daemon's authenticated session without any credential. The daemon runs as the systemd user unit `proton-drive.service`. Only the daemon holds the
database, the caches and the API client. Every other program is a client of its control socket
Configuration publication follows the same fail-closed model: a restricted temporary file is written and synced, atomically renamed, and followed by a directory sync. A malformed existing configuration is reported and preserved rather than overwritten with defaults. (§8) and never opens the database or calls Drive itself.
See [RECOVERY.md](RECOVERY.md) for what a lost machine means for the plaintext described in this section, and what to revoke.
### 8.6 Implications for deployment
- Treat the cache and state directories as being as sensitive as the Drive contents themselves.
- Restored or manually copied profiles must remain owned by the user. The daemon refuses to start against wrong-owner or symlinked sensitive directories.
- Purging the cache (`pdfs` settings, or `PurgeCache` over IPC) removes content but **not** `cache.db`, and deliberately never removes undrained `staging/` or `recovery/` blobs.
--- ---
## 9. Human Verification (CAPTCHA) Flow ## 2. The local tree
When logging in from an unfamiliar IP address or VPN, the Proton API may gate the sign-in with a human verification challenge (CAPTCHA). This client handles this asynchronously and interactively. ### Nodes and local ids
### 9.1 Sequence of Verification and Re-Authentication Every node the daemon knows is a row in the `nodes` table of `cache.db`. Each row has a **local
id** (`lid`), given when the row is made and never changed. Rows link to their parent by
`parent_lid`, so a folder's children, its subtree and its ancestors are found by local id.
- **Inode = local id.** A node's inode is `lid + 1` (`lid_ino` in `state.rs`). It is the same in
every mount and after a restart, so the kernel's cached entries never point at an inode the
daemon has forgotten.
- **Nodes made here.** A node created on this machine has no Drive uid until its create lands. It
goes by the stand-in `local~<lid>` (`db::local_uid`), a function of its row. When the create
lands, the row gets the real uid in the same transaction.
- **One tree for every mount.** The daemon holds one in-memory `State`, a cache of the `nodes`
table. `~/ProtonDrive` and each online-only synced folder are FUSE sessions rooted at different
nodes of that tree; each session calls its own root inode 1 and translates at the kernel
boundary.
### Listings on demand
A folder is listed from Drive the first time it is opened. `ensure_children` checks the folder's
`listed` flag. If it is set, the children come from the database. If not, the daemon enumerates
the folder from Drive, writes the children to the database, sets the flag, and interns them.
### Access
Every entry carries an `Access` (`Owner`, `Editor`, `Viewer` or `Unknown`). It is inherited from
the parent when the entry is interned. A share root takes its access from the persisted
`share_access` table, which also keeps the classification right offline. Two rules cover what
inheritance cannot:
- **Not under a share, no role: `Owner`.** My files and device folders are the user's own.
- **Under a share, no usable role: `Viewer`.** An unknown permission mask is never guessed upward.
A node whose parent is not resident resolves as `Db::effective_node_access` does: a recorded share
above it wins; otherwise it is writable on the mount's own volume and read-only on a foreign one
(`docs/BUGS.md` B79).
Writes are refused in three layers:
1. **Mode bits.** A non-writable entry reports `0o555` or `0o444`, and every mount sets
`DefaultPermissions`, so the kernel refuses writes from any non-root process.
2. **Handler checks.** `create`, `mkdir`, `unlink`, `rmdir`, `rename` (both parents), and opens
for writing, `write`, `setattr` and `fallocate` answer `EACCES`. This covers root and stale
attribute caches.
3. **Queue guard.** `Core::require_uid_writable` admits a change only when the database and the
live tree both agree the node is writable. Nothing reaches the queue otherwise, so a refused
write cannot become an op that fails forever (`docs/BUGS.md` B34).
`EACCES`, not `EROFS`: a read-only subtree inside a read-write mount is not a read-only
filesystem, and `EROFS` misleads `cp`, `rsync` and `git`.
### Locations
The `mount` table has one row per local place the daemon occupies: My files and each synced
folder, in either mode. It serves `pdfs locations` and the app. A mirrored folder is an ordinary
directory with no FUSE session, which is why the table speaks of locations, not mounts. The My
files mountpoint stays in `config.json`; its row is written at start as a copy of it.
---
## 3. Read path
`Core::read_range` serves a read from the revision's content blocks. It looks in three places
before the network:
1. **Whole-file blob.** A file kept available offline is downloaded completely into the content
cache and read from there.
2. **Stream ring.** An in-memory ring of recently decrypted blocks (`RING_BYTES`, 128 MiB). The
kernel reads in pieces far smaller than a block.
3. **Block cache.** Blocks on disk under `content/blocks/`. They count against `cache_budget` and
are evicted least recently used first.
On a miss, only the blocks that overlap the request are fetched, through a `RevisionReader` kept
open per revision (`MAX_OPEN_READERS = 64`). A fetch that cannot finish within
`READ_FETCH_TIMEOUT` (120 s) fails the read with `EIO`.
- **Block geometry.** The first read of a file plans on uniform 4 MiB blocks (`BLOCK_SIZE`). Once
a reader is open, the real block sizes are stored (`store_block_geometry`) and later reads plan
on them. Proton does not guarantee 4 MiB blocks (`docs/BUGS.md` B85, B87).
- **Read-ahead.** A sequential reader gets 2 to 8 blocks of prefetch (`PREFETCH_MIN`,
`PREFETCH_MAX`). Prefetch takes a permit from a budget of 8 (`PREFETCH_BUDGET`) and gives up
instead of waiting, so it never delays a read someone asked for.
- **Large videos.** An unpinned video of 256 MiB or more (`STREAM_BYPASS_MIN`) streams without
storing its blocks, so watching a film does not evict the rest of the cache.
A file open for writing is read from its handle: written bytes from the scratch file, the rest
from its base.
---
## 4. Write path
Since 3.0.0 the mount is **local-first**. A syscall changes the local tree and the queue, and
returns. It does not wait for Drive, except to read content that is not cached. Being offline
only means more work is queued.
### Namespace changes
`mkdir`, `create`, `rename`, `unlink` and `rmdir` through the mount:
| Call | What happens locally | Queued op |
|---|---|---|
| `mkdir`, `create` | A new row under `local~<lid>` (`queue_local_node`) | `mkdir` or `create` |
| `rename` | The node moves in the tree at once (`queue_rename`) | `rename`, holding the end state and the node's original parent and name |
| `unlink`, `rmdir` | The node leaves the tree; ops queued for it are dropped (`queue_trash`) | `trash`, or nothing for a node Drive never saw |
A rename or move asked for over the control socket (`pdfs rename`, `pdfs move`, the app) is queued
the same way, and always when the node already has something queued, so it cannot overtake it
(`docs/BUGS.md` B149).
`"local_first": false` in `config.json` brings back the old path for this release: while online,
these calls go to Drive inside the syscall. It is removed in the next release (§13).
### Content
1. **Open for writing.** A `WriteHandle` with a scratch file under `content/scratch/`. An interval
set records which bytes were written; reads of the rest go to the base.
2. **Release** (`close(2)` of the last handle). Gaps the program did not write are filled from the
base, from the cache when it has them. The scratch file and its metadata are published into
`content/staging/` (fsync, atomic rename, directory fsync), and a `revision` op is queued in
the same transaction that supersedes an older one. A new file's bytes ride on its `create` op
instead.
3. **Debounce.** A `revision` op waits `DRAIN_REVISION_DEBOUNCE` (2 s) before it is sent, so a
quick follow-up write replaces it in the queue. Once a file has been uploaded, the wait widens
toward how long that upload took, up to `DRAIN_REVISION_DEBOUNCE_MAX` (60 s)
(`Core::revision_debounce`).
4. **Cancel.** Before the queue is touched, `Core::cancel_upload` stops an upload already reading
the blob that is being replaced.
`fsync` means "durable locally". Writes that were fsynced but not yet released when the daemon
died are found at the next start (`recover_fsynced_writes`), kept in `content/recovery/`, and
queued.
### Transient names
A file under a scratch name, such as a browser's `.crdownload` or `.part`, or an editor's `.swp`,
is created locally and its op is **parked** (`PARK_UNTIL`). The rename to the finished name
releases it, so only the finished file reaches Drive (`docs/BUGS.md` B70). A park still standing
after an hour (`PARK_EXPIRY_MS`) is released anyway: bytes the user can see are bytes the user
expects on Drive.
---
## 5. The queue and the drain
### Ops
The queue is the `pending_op` table. An op is a change accepted locally that Drive has not seen
yet.
| Kind | Does | Carries |
|---|---|---|
| `create` | Makes a file | Parent and name; the staged bytes |
| `mkdir` | Makes a folder | Parent and name |
| `revision` | Uploads new content | The staged blob and the revision it was based on |
| `rename` | Moves and renames | The end state (parent, name), and the original parent and name |
| `trash` | Trashes a node | Its name |
Each op names its node and parent by local id (`lid`, `parent_lid`) as well as by uid.
`revision`, `rename` and `trash` describe an end state, so a newer one replaces an older one for
the same node (`op_supersedes`). `create` and `mkdir` are never replaced.
### Two invariants
From `drain.rs`:
1. **A staged blob is the only copy of the user's bytes.** It is deleted only after its op has
provably landed, never on a path that may be retried.
2. **A failure never stops the queue.** A failed op gets a later `next_attempt_at` (exponential
backoff), so one stuck file cannot hold up the others.
### The drain
`DRAIN_WORKERS = 16` threads share the queue. A worker takes work with
`Db::claim_next_due_op`, which selects and marks an op in one transaction. It skips an op when:
- it is not due yet (debounce, backoff, or parked);
- another worker holds an op for the same node, so each node's ops land in the order they were
made;
- its parent is still a `local~` stand-in, so a child's create waits for its folder's create.
A worker that lands an op wakes the idle ones, because the landing may make other ops claimable.
Claims are process-local: the database is single-writer (a `flock`), so a claim found at start
belongs to a crashed run and `Db::clear_op_claims` drops it.
A landing create records the real uid on the node's row. Ops inside a landed folder need no
rewrite: they find the parent's uid through `parent_lid`.
### Conflicts
A `revision` op stores the revision it was based on. If Drive's current revision differs and is
not one this daemon sealed itself (`own_sealed_rev` table), the queued bytes are kept as a
conflict copy (`Core::revision_conflict`) next to the remote version. A name taken on Drive by a
node this machine has not seen lands under a conflict name too. Both go to the activity log.
Conflict copies are named `{stem} (sync-conflict {unix_secs}).{ext}`.
### Failures and sync issues
An op that has failed `FAILING_ATTEMPTS` (6) times counts as failing. An op that Drive refused
gets a **sync issue** at once, stored on the op:
| Issue | Meaning |
|---|---|
| `quota` | The account's storage is full |
| `access` | The account may no longer change the node, for example a revoked share |
| `missing` | The node or its folder is gone from Drive |
| `limit` | Too many items in the folder, or nested too deep |
| `rejected` | Drive refused the change, for example a name it does not accept |
The op stays queued and keeps retrying; nothing is deleted. The user sees the issue in
`pdfs sync issues`, the tray, **Sync → Overview**, and as a badge on the node in the file browser.
They can save the content (`pdfs sync export`) or drop the change and undo it locally
(`pdfs sync discard`). Only a discard the user asks for removes an op.
### Link state
`link.rs` tracks whether Drive is reachable. A network error marks the daemon offline, and a probe
thread (`pdfs-online-probe`) backs off until a request succeeds. Remote calls a caller waits on
have a deadline (`INTERACTIVE_CALL_TIMEOUT`, 20 s); uploads get one that grows with their size
(`link::upload_deadline`).
---
## 6. Remote changes
`run_event_sync` follows Proton's event stream for the Drive volume. Each event updates the
database and the tree, and the kernel is told about entries that changed. The cursor is stored as
events are applied, so changes made while the daemon was stopped arrive at the next start. An
event for a revision in `own_sealed_rev` is our own echo, not a change from elsewhere.
`run_photos_event_sync` follows the photos volume with its own cursor. Photos are not in the mount,
so it only drops gallery rows or marks the timeline stale (`TIMELINE_TTL`, 60 s).
`pdfs refresh` drops a cached listing so the next read fetches it again.
---
## 7. Synced folders
A synced folder in **mirror** mode is a plain local directory kept in step with a folder under
this computer's device in Drive. It has its own engine (`sync.rs`, `sync/`); it does not use the
queue of §5 yet (§13). A folder in **online-only** mode is a FUSE session on the tree (§2), and
its changes go through the queue like My files.
### A sync pass
A filesystem watcher (`notify`) and a remote poll every 120 s feed a debounced pass (2 s, at most
30 s). Passes are serialized per daemon. A pass:
1. **Walks the local tree.** Any `readdir`, metadata or permission failure marks the scan
incomplete, and an incomplete scan cannot delete anything.
2. **Walks the remote tree.**
3. **Loads the baseline**, the `sync_entry` rows: both sides as of the last successful pass.
4. **Classifies each path** by comparing local, remote and baseline, with `(mtime, size)` as the
change signal:
| Local | Remote | Action |
|---|---|---|
| changed | unchanged | upload |
| unchanged | changed | download |
| changed | changed | conflict: the local file becomes a conflict copy, the remote one is downloaded |
| deleted | unchanged | trash on Drive |
| unchanged | deleted | delete locally |
| new | — | upload as new |
| — | new | download as new |
5. **Applies** the actions, folders before their contents, a few at a time.
6. **Settles.** The baseline is updated from what actually happened. A failed local delete or
conflict copy keeps the old baseline.
Safety rules:
- **Wipe guard.** A pass in which every previously synced file has vanished locally is refused
instead of trashing the whole folder on Drive. It applies to any non-empty baseline.
- **Open for writing.** Files that some process holds open for writing (`/proc/*/fd`,
`open_for_write_set`) are left out of the pass and picked up after they are closed.
- **Ignore rules.** `.pdfsignore` (or `.protonignore`) at the folder root plus `ignore_patterns`
from `config.json`, in gitignore syntax. Ignoring never deletes.
### Moves between locations
My files and each online-only folder are separate FUSE sessions, and a mirrored folder has none,
so the kernel answers `rename(2)` between them with `EXDEV` and `mv` copies and deletes. On Drive
the same move is one `move_node`. `Request::Move` with two absolute paths (`pdfs move`, the app)
is resolved against every location by `Core::move_between` (`relocate.rs`):
- **Nothing that exists only on this disk may be lost.** A mounted source is refused while it or
anything under it is queued or open for writing. A mirrored source is refused unless every file
under it matches its baseline (`mirror_subtree_unsynced`); ignored files and symlinks count as
unsynced.
- **Mirror to mirror** renames the local copy with `RENAME_NOREPLACE` and moves its baseline rows
(`sync_entries_move`), so neither folder's next pass has work. A failed `move_node` undoes the
rename.
- **Passes are held off.** Every mirrored folder involved is locked by ascending id, for at most
five seconds.
- **No downloads for what is cached.** A mirrored destination that has to download the node first
copies cached blocks into place (`ContentCache::copy_cached_to`).
### Devices and the machine profile
Synced folders live under a **device**, a per-computer root in Drive. `Core::ensure_device` picks
this machine's device: the adopted `device_uid` from `config.json` when set, else a Linux device
named like the hostname, else a new one. An adopted uid that no longer exists is reported, never
replaced silently.
The local database does not survive a lost machine, so the daemon backs up its arrangement to
`<device root>/.proton-drive-linux/profile.json` (`pdfs_core::profile`): each synced folder's
remote uid, local path and mode, the pins, and the ignore patterns. It is uploaded like any other
file, so it is end-to-end encrypted. `Core::touch_profile` coalesces bursts of changes into one
upload. `PROFILE_VERSION` guards the format: a profile from a newer client is refused, not applied
in part.
Restore lists the device's folders, proposes a local path from the profile, and on confirmation
adds a synced folder bound to the remote uid. The download is an ordinary pass against an empty
baseline. The user side is in [RECOVERY.md](RECOVERY.md).
---
## 8. Control protocol
Front ends talk to the daemon over a Unix socket, `control.sock` in the state directory
(`pdfs_core::control`).
- **Framing.** One JSON `Request` per line, one JSON `Response` per line.
- **Limits.** A request is at most 1 MiB, must end with a newline, and must arrive within 10 s. At
most 64 handlers run at once.
- **Client timeouts.** 2 s to write a request; 120 s to read the answer; 5 s for a status poll,
which is what the app shows as "Not responding".
- **Errors.** `Response::Error { message, kind }`. The `kind` is machine-readable, and
`pdfs --json` passes it through.
- **Compatibility.** After an upgrade the old daemon runs until the service restarts, so a front
end must cope with a daemon that does not know a newer request. The daemon answers such a
request with an error instead of closing the connection. New fields get `#[serde(default)]`;
new behaviour gets a new request rather than an optional field an old daemon would ignore.
### Push events
The app and the tray send `Request::Subscribe` once and keep the connection open. The daemon then
streams one `Event` per line:
| Event | Sent when |
|---|---|
| `Status` | On subscribe, and whenever a status field changes |
| `Transfers` | On subscribe, and whenever the transfer list changes |
| `Changed { topics }` | Something a page shows changed. Topics: `files`, `photos`, `trash`, `shares`, `devices`, `locations`, `queue`, `conflicts`, `activity` |
| `Heartbeat` | After 10 s of silence |
Code that changes state publishes topics to the `EventHub` (`events.rs`); the control handler
lists the topics each mutating request touches. Changes within 150 ms are coalesced. A new
subscriber is told every topic changed. At most 8 subscriptions are served at once.
`pdfs_core::control::follow` reconnects across daemon restarts with backoff and treats 25 s
without a line as a stuck daemon.
### Search
`SearchV2` carries the query, a limit, the sources (`Drive`, `Local`) and a content kind. The
daemon searches Drive metadata and its index of the home directory (`pdfs-localindex`), and scores
both with `pdfs_core::search::relevance_score`, so the prompt can merge them into one list.
---
## 9. Threads
| Thread | Does |
|---|---|
| FUSE dispatch loop, one per session | Reads kernel requests. Cheap ones are answered inline; the rest go to the worker pool |
| FUSE workers (`FUSE_WORKERS = 11`) | Handlers that may touch the network. 3 serve metadata only (`lookup`, `readdir`, namespace changes), so a listing never waits behind block downloads; 8 prefer transfers and help with metadata when idle |
| `pdfs-drain-0` … `pdfs-drain-15` | The drain (§5). Worker 0 also runs the queue's idle chores |
| `pdfs-control` | Accepts control connections; at most 64 handlers |
| `pdfs-events` | Publishes status, transfer and topic changes to subscribers |
| `pdfs-sync`, `pdfs-sync-poll` | Mirror passes and the remote poll (§7) |
| Tokio runtime | Remote calls, and the Drive and photos event streams (§6) |
| `pdfs-online-probe` | Probes Drive while offline |
| `pdfs-supervisor` | Logs workers that hold one job for minutes, a queue that stops draining, and control requests that never return; samples memory; pings the systemd watchdog only after a round trip over the control socket succeeds (`WatchdogSec=120`) |
| `pdfs-conflict-sweep` | Optional. After 30 s, then every 5 minutes, finds `(sync-conflict …)` copies identical (size and SHA-1) to their original. `conflict_sweep` decides: `off`, `report` (default) or `enforce`, which trashes them after re-checking each one |
| Smaller helpers | `pdfs-localindex` (home index for search), `pdfs-pause` (ends a timed pause), `pdfs-similar` (similar photos, on request), `pdfs-diagnostics` |
Long-running loops wait on one shared `Shutdown` signal instead of sleeping, so a stop joins them
instead of leaving them behind (`docs/BUGS.md` B44). `pdfs diagnostics` reports what each worker
is doing.
---
## 10. Threat model: what this client writes to disk in plaintext
Proton Drive is end-to-end encrypted: the server never holds the keys to your content. That
property ends at this daemon. Serving a file through a POSIX filesystem means producing
plaintext, and serving it quickly means keeping some of it. This section states exactly what lands
on disk.
**The short version: the cache and state directories hold decrypted content and decrypted
metadata. This client assumes the disk under them is encrypted (LUKS, or an encrypted home).** On
an unencrypted disk, someone with the powered-off machine can read cached files and the names and
structure of your whole Drive without your password.
### 10.1 Decrypted content
Everything under `$XDG_CACHE_HOME/proton-drive-linux/content/` is plaintext:
| Path | Holds | Kept until |
|---|---|---|
| top level | Whole files: available offline, or opened whole | Evicted, or unpinned |
| `blocks/` | Blocks of partly read files | Evicted |
| `thumbs/` | Thumbnails and previews | Evicted |
| `scratch/` | Files open for writing | Released, or rescued at the next start |
| `staging/` | Released writes waiting to upload | **The upload lands** |
| `recovery/` | Fsynced writes rescued after an unclean stop | **Queued into `staging/`** |
`staging/` and `recovery/` are not copies of something Drive has. They hold content that may exist
nowhere else yet, so they are never cleared at start or by a cache purge. They are both the most
sensitive data on disk and the data that must not be deleted to free space.
### 10.2 Decrypted metadata
`$XDG_STATE_HOME/proton-drive-linux/cache.db` is a plain SQLite database. It holds decrypted node
names, the folder tree, sizes, timestamps, a trigram index over names, the activity log, the
photos timeline and the queue. It is not evicted and not budgeted. Clearing the cache does not
touch it.
`sdk_cache.db` holds the SDK's decrypted node metadata, encrypted at rest with a key from the
mailbox password (§12).
### 10.3 What is not written in plaintext
Credentials. The session (access and refresh tokens and the key material needed to resume) is
stored only in the system keyring through the Secret Service (`auth.rs`). `config.json` holds
settings and uids, no secrets.
The control socket is a credential of another kind: anything that can connect to `control.sock`
can drive the daemon (list and read, upload, trash, create public links) without the keyring. See
§10.5.
### 10.4 Memory, swap and the page cache
Two exposures this client does not mitigate:
- **Swap.** Keys and decrypted buffers live in ordinary heap memory. Nothing calls `mlock(2)`, so
they can be paged out. Use encrypted swap.
- **Kernel page cache.** Plaintext returned through FUSE is cached by the kernel like any file
data, and can be swapped. Avoiding that would mean `direct_io` on every read and losing the
kernel's read-ahead and caching. The client takes the performance.
### 10.5 Ownership and file modes
At every start, `AppDirs::ensure` checks that the config, state and cache directories are real
directories owned by the user, refuses symlinks and other owners, and sets mode `0700`. The control
socket is set to `0600` right after it is bound (`restrict_socket`); if that fails, the daemon
does not serve.
| Artifact | Why it is restricted |
|---|---|
| `content/` | Another local user could read cached file content |
| `cache.db` | Another local user could read the names and structure of your Drive |
| `control.sock` | **Another local user could drive the daemon** with your session |
`config.json` is written to a restricted temporary file, synced, renamed into place, and the
directory synced. A config file that cannot be parsed is reported and left alone, never replaced
with defaults.
### 10.6 What follows
- Treat the cache and state directories as being as sensitive as your Drive.
- Restored or copied state must stay owned by the user, or the daemon refuses to start.
- `pdfs cache clear` removes cached content, not `cache.db`, and never removes unsent `staging/`
or `recovery/` content.
[RECOVERY.md](RECOVERY.md) covers what a lost machine means for this data and what to revoke.
---
## 11. Sign-in and CAPTCHA
Proton may answer a sign-in from an unfamiliar network or a VPN with a human verification
challenge (API error 9001).
```mermaid ```mermaid
sequenceDiagram sequenceDiagram
autonumber actor User
actor User as User (GUI) participant UI as pdfs-app
participant Core as pdfs-core (Auth) participant Auth as pdfs-core auth
participant UI as pdfs-gui (Login Page) participant Web as pdfs-app --human-verification
participant Web as WebKitWebView Dialog participant API as Proton API
participant API as Proton API Server
User->>UI: Enter credentials & click Sign In User->>UI: email and password
UI->>Core: login_interactive() UI->>Auth: login_interactive()
Core->>API: auth/v4 (Initial SRP Handshake) Auth->>API: SRP handshake
API-->>Core: HTTP 422 (Error 9001: CAPTCHA Challenge URL) API-->>Auth: 9001, challenge URL
Core->>UI: Error::HumanVerificationRequired(hv) Auth-->>UI: HumanVerificationRequired
UI->>UI: Block login thread, dispatch to GTK main loop UI->>Web: challenge URL (stdin)
UI->>Web: Create dialog & load verification URL User->>Web: solve it
Note over User, Web: User completes CAPTCHA in embedded WebView Web-->>UI: token (stdout)
Web->>UI: window.postMessage(HUMAN_VERIFICATION_SUCCESS) UI->>Auth: login_verified(token)
UI->>UI: Extract token (with double-serialization safety) Auth->>API: new SRP handshake with the token
UI->>UI: Close dialog & send token to blocked login thread API-->>Auth: session
UI->>Core: login_verified(with verification token)
Core->>API: auth/v4 (SRP retry with x-pm-human-verification-token)
API-->>Core: Returns session tokens
Core-->>UI: Login Successful
``` ```
### 9.2 Key Technical Design Decisions - The challenge used up the first SRP handshake, so the retry starts a new one with the token
attached.
- The page may post the token as an object or as a JSON string; both are accepted.
- The page runs in a WebKitGTK view inside a child `pdfs-app`. WebKit aborts its process when its
sandbox cannot start, as on hardened systems such as secureblue (B195). In the child, that ends
the web view, not the app.
- When the child fails, the app offers the page in the user's own browser instead, as Proton Mail
Bridge does. A browser tab cannot post the token back, so the user says when they are done, and
the retry carries the challenge token itself, which the solved page made good. Its type header
lists every offered method (`auth::browser_verified`).
- The CLI has no web view, so it always takes the browser path: it prints the page's URL and waits
for Enter.
1. **Weak Reference UI Binding:** To prevent memory leaks and strong reference cycles between the parent dialog, the child `WebKitWebView`, the script message manager, and the connection callback, the dialog is downgraded to a `WeakRef` inside the callback: ---
```rust
let dlg_weak = dialog.downgrade(); ## 12. Feature notes
content.connect_script_message_received(Some("hv"), move |_, value| {
// ... Features whose code is shaped by a constraint that the code alone does not explain.
if let Some(dlg) = dlg_weak.upgrade() {
dlg.close(); ### Version history
}
}); `revisions.rs`, `versions_dialog.rs`. `ListRevisions`, `RestoreRevision`, `DeleteRevision` and
``` `SaveRevisionAs`, each with a `…ByUid` twin for nodes no mount can name.
2. **Double-Serialization Tolerance:** The JavaScript message listener forwards event data as a JSON string to the native handler. Since the underlying page may post either JS objects or pre-serialized JSON strings, the Rust side performs dual-phase parsing:
```rust - **A restore runs on the server, later.** Drive answers 202 and swaps the active revision in the
let mut value = serde_json::from_str(raw).ok()?; background. No content is sent and nothing is queued; the daemon evicts the file's cached
if let Some(inner) = value.as_str() { blocks and readers, and the app promises the request, not the result.
if let Ok(parsed) = serde_json::from_str(inner) { - **The active revision cannot be deleted.** Checked locally, so the user gets a sentence instead
value = parsed; of an API code.
} - **Saving a version never overwrites.** An existing destination is refused, and a partial file is
} removed when the download fails.
```
This ensures compatibility with all versions of Proton's client verification scripts. ### Photos
3. **SRP Handshake Reset:** Because a gated login burns the SRP handshake on the API side, the client cannot simply resume the previous request. Instead, `auth::login_interactive` restarts the SRP process from scratch with the verification credentials attached, keeping the complex handshake details isolated from the front-end.
4. **CLI Fallback:** Since the CLI has no native web browser engine, hitting the CAPTCHA gate fails immediately with a user-friendly message directing the user to sign in once via the GUI (`pdfs-app`) to persist the authenticated session keys to the system keyring. `pdfs-fuse/src/photos.rs`, `pdfs-core/src/db/photos.rs`, `pdfs-gui/src/app/pages/photos.rs`.
- **Favorites** use the SDK's photo tags. Favoriting a photo that is not on the account's own
photos volume (shared, or only in an album) is refused: it would need re-encryption the SDK does
not offer.
- **RAW + JPEG** pairs share a `group_key`. The server's relation (`main_photo_uid`,
`related_photo_uids`) comes first; otherwise the same capture day and name stem with one member
raw. The tile shows the non-raw member. Trashing a tile trashes the whole group. Albums are
never grouped.
- **The gallery** lays each day out in justified rows (`justify_rows`), so every tile has its
photo's shape.
- **Delete** is by uid (`Request::TrashNodes`), because the photos volume is not in the mount. The
rows are dropped at once, and the app offers Undo.
### SDK entity cache
`sdkcache.rs`. The SDK's cache of decrypted node metadata is stored in `sdk_cache.db`, so a restart
does not fetch and decrypt the tree again.
- **Its own file**, not `cache.db`: the traffic is frequent, small and can be rebuilt, and it needs
no migrations. It may be deleted while the daemon is stopped.
- **Encrypted at rest** with the SDK's `EncryptedCacheRepository`, keyed by the mailbox password.
After a password change it reads as empty.
- **Staleness** is handled by the event stream (§6). When the cursor has to be seeded (first
mount, or a lost cursor), the store is cleared. Signing out deletes it.
### Batched calls
`batch.rs`. `trash_nodes`, `restore_nodes` and `delete_nodes` report one outcome per node;
`batch::into_unit` turns a single-node call back into a `Result`. A restore first adds every
trashed ancestor and descendant of what was asked for, from the trash listing (`expand_restore`),
and sends them shallowest first, because Drive cannot restore a node into a parent that is still
trashed. Restore and permanent delete from the Trash page use the streaming variants and update
local state per node as each batch lands.
---
## 13. Where it is going
3.0.0 made the mount local-first. What is still split:
- **The old online path** in `filesystem.rs` and the `local_first` switch go in the next release.
- **Remote changes** are applied by the event code and by several invalidation helpers. One
applier for everything that comes from Drive is the next step.
- **Mirrored folders** still have their own engine (§7). They move onto the local tree and the
queue after that.
[ROADMAP.md](ROADMAP.md) tracks these and the rest of the planned work.
+6016 -3086
View File
File diff suppressed because it is too large. Load diff
+515 -3
View File
@@ -11,6 +11,500 @@ scratch (user data in `staging/` and `recovery/` is never touched by this).
## [Unreleased] ## [Unreleased]
### Fixed
- **A CAPTCHA could not be solved on hardened systems.** When the verification page cannot open in
the app, as on secureblue, the app now offers it in your browser and continues the sign-in
once you are done. `pdfs login` now handles a CAPTCHA the same way instead of failing.
([#29](https://github.com/narrrl/proton-drive-linux/issues/29))
## [3.1.2] — 2026-10-08
Schema: **42**, unchanged.
### Fixed
- **The app crashed when Proton asked for a CAPTCHA on hardened systems.** On secureblue, WebKit
cannot start the sandbox for the verification page and ends its process, which took the app
down with it. The page now opens in a separate window process: if that process fails, the
sign-in stops with a message that says why, and TROUBLESHOOTING.md has a workaround.
([#29](https://github.com/narrrl/proton-drive-linux/issues/29))
- **On a fresh install the tray icon did not start.** The tray and the app now create their state
directory themselves instead of waiting for the service to do it. This also lets the Takeout
page remember staged archives before the first mount.
([#29](https://github.com/narrrl/proton-drive-linux/issues/29))
## [3.1.1] — 2026-10-07
Schema: **42**, unchanged.
### Changed
- **The client identifies itself with its release.** Sign-in and every API call used to send the
fixed `external-drive-linux@0.1.0-alpha` and `proton-drive-linux/0.1.0`. If Proton retired that
version, every release would fail to sign in at once and no update could fix it. The client now
sends `external-drive-linux@<version>-stable` and `proton-drive-linux/<version>`. `app_version`
and `user_agent` in `config.json` are optional overrides now. The fixed values that releases up
to 3.1.0 wrote there count as unset and are removed on the next save. A value you set by hand is
kept.
- The About dialog shows the release version, and its debug info shows the identity that is
actually sent.
## [3.1.0] — 2026-10-07
Schema: **42**, unchanged.
### Added
- **Two-password accounts.** An account with a separate mailbox password now signs in: the app
and `pdfs login` ask for it after the login password and check it against the account, so a
typo is a retry and not a daemon that never mounts. The mailbox password is never stored.
What it unlocks is kept in kernel memory for the login session, so it is asked for once per
session and is gone after a reboot or when you log out of the machine.
- **`pdfs unlock` and `pdfs lock`.** `pdfs unlock` enters the mailbox password of a locked
account and starts the daemon's mount; `pdfs lock` forgets it again. `--remember` on `login`
and `unlock` (and the "Keep unlocked on this computer" switch in the app) keeps the unlock in
the system keyring instead, so the daemon starts without asking; `pdfs lock --forget` removes
it. Off by default: it gives up the separation a second password is for.
- **Locked state.** `pdfs status`, `pdfs diagnose` and the tray say when the account is waiting
for its mailbox password, and the app asks for it when it opens.
### Changed
- **The keyring no longer holds a password.** The session blob keeps the keys' passphrases
instead of the mailbox password, and a blob written by 3.0.x is converted on the first start.
A stored password the account rejects (a two-password account that was signed in with only
its login password) leaves the account locked instead of failing every start.
- **The daemon is not dumpable.** It holds the unlock in memory, so core dumps and same-user
`ptrace` are off for it.
- **The offline cache is re-keyed** from the new secret, so the first start after the update
reads it as cold and fetches the tree again once.
### Fixed
- **systemd killed a daemon that was only retrying.** `WatchdogSec` was answered only once a
mount existed, so a daemon waiting for a login or retrying a failed mount was aborted after
two minutes and restarted in a loop, hiding the error it logged. It now answers the watchdog
while it waits. ([#26](https://github.com/narrrl/proton-drive-linux/issues/26))
- **A refreshed token could overwrite a newer unlock or revive a signed-out session.** Token
rotation now updates only the tokens in the stored blob.
## [3.0.0] — 2026-10-05
Schema: **42**. The queue table is rebuilt so an id is never given out twice; queued changes
are kept. Queued changes get a column for the reason Drive refused them, and two for the
content a new file was sent with and where, and name their file and folder by local id as well
as by Drive's id. Stored files and folders name their parent
folder by local id too. A file or folder made on this machine that Drive does not have yet goes
by its local id, in the queue, the stored tree and pins alike.
### Added
- **Sync issues.** A change Drive refuses because the storage is full, access was taken away,
its folder is gone, a folder limit is reached or Drive rejected it shows up at once as needing
attention, with what it means and what to do, instead of after six failed attempts. The
change stays queued and is still retried.
- **Export a queued file.** The Sync page's Export button and `pdfs sync export <id> <dest>`
save a copy of a queued upload's content. The upload stays queued.
- **`pdfs sync issues`** lists the changes that need attention; `pdfs sync queue` shows the
issue next to each change.
- **Discard a queued change.** The Sync page's Discard button and `pdfs sync discard <id>` drop
a change Drive will not take and undo it here: a new file or folder goes, anything else goes
back to what Drive has. Export a file first to keep its content.
- **Sync issues show on the file.** A file or folder with a change Drive refused carries a
warning badge in the file browser, and its tooltip says what it means and what to do.
### Changed
- **Making, renaming and deleting through the mount no longer waits for Drive.** A new folder,
a new file, a rename or a delete is recorded locally and the call returns at once; the queue
sends it, as it does offline. `"local_first": false` in `config.json` brings back the old way
for this release.
- **The queue sends up to 16 changes at once instead of 3.** A thousand new files in a hundred
folders reach Drive about three times as fast on a LAN, and four times as fast on Wi-Fi.
### Fixed
- **A file made under the name of a file whose upload is still going out keeps its name.** It
no longer lands as a conflict copy, and the mount no longer shows that copy's name (B154).
- **A change to a file no longer lands as a conflict copy of itself when the link loses Drive's
answer to its upload** (B155).
- **A file written again while its upload was being retried no longer lands as a conflict copy
of itself when the link lost Drive's answer to the first try** (B156).
- **Writing to a file twice in a row no longer fails with "Input/output error" when the link
lost Drive's answer to reading back the file's last upload** (B157).
- **A file renamed twice in a row, while the first name was still taken, keeps the second name**
(B158).
- **Opening a file right after the daemon restarts no longer fails with "Resource temporarily
unavailable"** (B159).
- **A new folder, or a new empty file, reaches Drive at once**, not up to half a minute later
(B160, B192).
- **A file moved while it was still being uploaded keeps its new name in the mount** on a slow
link (B161).
- **A folder removed and made again at once keeps what goes into it.** It used to be created
inside the removed folder and was trashed along with it (B162).
- **A new file no longer reads as empty just after it is uploaded.** Drive can list it a moment
before its contents are committed, and the mount took that answer (B163).
- **A folder no longer lists a new file twice while it is uploaded.** A listing taken as the
upload landed showed it both as Drive had it and as the queue did (B164).
- **Stopping the daemon no longer hangs until systemd kills it.** A stop waits up to 10 s for
work in progress, then exits and logs what was still running (B165).
- **A folder no longer drops or empties a new file while others are uploaded.** A file whose
upload landed while the folder was being listed was missing from that listing, or listed as
empty (B166).
- **The mount comes up faster on a large Drive.** Rebuilding the folder listings from the cache
took close to 2 s with 37,000 files at every start (B167).
- **Files deleted with their folder while uploading stay deleted.** The mount no longer tries
to upload them to the top of your Drive instead (B168).
- **Stopping the daemon during a folder sync no longer waits for the sync.** The uploads in
progress finish, and the rest waits for the next start (B169).
- **Writing to a file again just as its upload finishes no longer fails.** A write right after
it failed with "Input/output error" (B170).
- **A file deleted while its upload was retrying is removed from Drive right away.** It no
longer waits for the upload's next retry (B171).
- **A file deleted just as it reached Drive is removed from Drive too.** It no longer stayed
there, and the next file with that name no longer became a conflict copy (B173).
- **Renaming a file just as it reaches Drive no longer fails with "Permission denied"**
(B174).
- **A file replaced by a rename just as it reached Drive is removed from Drive too.** It no
longer stayed there, and the next rename of its name no longer moved it instead (B175).
- **A write closed just as its file reached Drive is no longer lost** (B176).
- **Opening a folder just as it reaches Drive no longer fails with "Input/output error"**
(B177).
- **Deleting a folder while sync is paused no longer leaves a queued change behind** for a
file in it that was never uploaded (B178).
- **A file read just as it reaches Drive no longer reads empty** (B179).
- **Stopping the daemon no longer takes 10 seconds more than it needs** when an idle upload
worker missed the stop (B180).
- **A folder made while offline or with sync paused no longer fails with "Input/output
error"** once it is refreshed or the connection comes back (B125).
- **The daemon no longer hangs when a file being read changes on Drive** (B130).
- **The Activity page says in your language why an edit was kept as a conflicting copy**,
instead of the daemon's English (B181).
- **A file written after a restart no longer lands as a conflict copy of itself** when the
connection dropped just as it first reached Drive (B182).
- **A file deleted after a restart is removed from Drive too** when the connection lost Drive's
answer to its upload and the file was written again meanwhile. It no longer stayed there and
turned the next file under its name into a conflict copy (B183).
- **A file renamed while the connection lost Drive's answer to its upload lands once.** It no
longer stayed on Drive under its old name too, or, when deleted, under its old name alone
(B184).
- **A new file written again just as it starts uploading no longer waits out a retry** before
it reaches Drive (B185).
- **A file deleted while open can still be read and written until it is closed**, offline too.
A write into it no longer failed with "Input/output error" (B186).
- **An uploaded file no longer leaves a copy of itself in the cache** until the next start
(B187).
- **Changes that failed during a short outage go out soon after the connection is back**, not
after a retry delay of up to five minutes (B189).
- **A file saved just before the daemon stops reaches Drive** with what was written. It could
keep its old content, or stay empty (B190).
- **A file renamed just as its upload finished keeps its new name.** The mount no longer showed
the old name, and a second rename no longer failed with "No such file or directory" (B152,
B172).
- **A change that waits for a name to be freed no longer waits seconds after it is.** A rename or
upload over a file being deleted goes out as soon as the delete lands (B153).
- **A file deleted right after it was made no longer stays on Drive when the link lost Drive's
answer.** The node Drive made is trashed (B151).
- **A new file no longer lands twice when the link loses Drive's answer to its upload.** The
file Drive made is taken as it is, instead of a second upload as a conflict copy (B127).
- **A change to a file deleted elsewhere, together with its folder, no longer retries forever.**
The conflict copy that keeps it now lands in the location's root (B150).
- **`pdfs rename` and `pdfs move` right after a change through the mount are no longer undone.**
They are queued behind it, and they work on a file that has not reached Drive yet (B149).
- **A folder kept offline before it reached Drive stays kept offline.** Its pin was lost when the
folder landed (B148).
- **A queued change no longer holds up the queue forever on a link that stops answering.**
Sending a new folder, a delete or a file waits at most 20 seconds for Drive, plus time for
the file's size, and then the queue treats the link as down and tries again.
- **A file deleted while its first upload is on the wire no longer stays on Drive.** The node
the upload made is trashed as soon as it lands, so its name is free again (B129).
- **A delete queued while a rename is sent is no longer lost.** The rename landing could remove
the delete from the queue, leaving the file on Drive (B137).
- **A file made offline over one deleted offline is no longer lost.** Its upload could take
over the deleted file, and the delete then took the new bytes along with it (B128).
- **A file deleted while its upload is on the wire no longer comes back as a `recovered-…`
copy** when another file is made right after (B133).
- **A file made where another one is leaving no longer lands as a conflict copy.** Its upload
waits for the rename or trash that frees the name (B126, B138).
- **A file closed just as its first upload lands keeps its last write.** The write was refused
and left in staging (B139).
- **Files made in a new folder are still there after a restart.** A folder that landed as the
daemon stopped, or one listed from Drive while files in it were queued, came back empty
(B132, B140).
- **A file renamed twice in quick succession ends up under the second name.** The first rename
could land after the second and undo it (B141).
- **Renaming a new file just as its upload lands no longer fails with "Device or resource
busy".** The rename goes to the file Drive made (B142).
- **A file whose upload landed just as the connection dropped no longer gets a conflict copy at
its next save.** The daemon reads the file back once the connection returns (B143).
- **A rename whose target name was just freed no longer lands as a conflict copy.** The rename
is sent once more under the name it asked for (B144).
- **Saving a download under its final name no longer fails with "Device or resource busy".**
A file made a moment before was taken for deleted and dropped from the queue (B145).
- **A file renamed just as its upload lands keeps its new name.** The tree could take back the
name Drive still had (B146).
- **A new file made under a name another file just left no longer takes that file's place on
Drive.** Both could land on one node, and one file's content was lost (B147).
## [2.10.0] — 2026-10-03
Schema: **36**. Every node row gets a local id that never changes; existing rows keep their
row number as that id. Going back to 2.9 means rebuilding the cache. Requires `proton-drive-rs`
**0.7.2**.
### Added
- **Migration tests on real databases.** A database 2.8.2 wrote, with queued changes due,
parked, failing, waiting for access and in flight, migrates with every change intact.
`PDFS_MIGRATE_DB=<path> cargo test -p pdfs-core a_copy_of_a_real_database -- --ignored` runs
the same check on a read-only copy of any database.
### Changed
- **A file keeps its inode number across daemon restarts.** Inodes are now the node's local id
instead of a counter that started over on every run.
- **Every mount serves from one tree.** An on-demand sync folder used to keep its own copy of the
nodes it showed, which background work had to visit one by one and could miss (B74). It now
shares the tree with My files; only the root it starts at is its own.
### Fixed
- **A file renamed or moved while its first upload is on the wire keeps its new name and folder.**
It landed under the old name and the rename was lost (B136).
## [2.9.0] — 2026-10-03
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.2**.
### Added
- **Seeded simulation runs.** `cargo test -p pdfs-fuse --lib sim:: -- --ignored
--test-threads=1` mounts one or more real FUSE daemons against an in-memory Drive that lags,
drops, reorders and echoes like the real one. Each seed draws syscalls, link flaps and restarts,
checks every answer against a POSIX model, and at the end checks that nothing was lost and the
clients converged. CI runs them on pushes to main. They found B122 to B135.
- **More acceptance cases.** Regressions for B48, B80, B86, B88, B94, B95, B98, B102, B111, B113
and B114 to B121, and a check that `pdfs` output into a closed pipe stays quiet. The console
report has one line per case and a summary table, and `--journal-check` also fails on a
conflict copy, a FUSE job held for long, or an authority re-read.
### Fixed
- **A new file moved or renamed before its first upload no longer lands as a conflict copy**
next to its own empty original (B113).
- **`pdfs rm`, `pdfs rename` and `pdfs mkdir` take paths in on-demand folders.** They answered
"is not under the mountpoint" for a path `pdfs move` took. `rm` and `rename` refuse a path that
names a mount's root (B88).
- **`pdfs ls | head` no longer panics** with "Broken pipe". A closed stdout exits 0; any other
write error exits 1.
- **A file removed, renamed or moved with `pdfs` stops resolving under its old name at once**,
instead of for up to 30 seconds (B114).
- **`pdfs ls` and the app's file browser show a file's real size**, not its encrypted one
(B115).
- **Trashing a folder also drops the creates queued in its subfolders.** One could stay queued
with its bytes, or land in the root (B116).
- **An upload queued for a file `pdfs move` moved keeps its path** in `pdfs sync queue` and is
no longer put off once after the resume (B117).
- **Creating a file or folder through the mount no longer fails with `ENOENT`** when Drive is
slow to list what it made. The mount asks again, and queues the create if it still cannot see
it (B118).
- **A file just written can be read without the network.** Its uploaded bytes stay cached, and
`pdfs status` counts the upload as pending until they are (B119).
- **A file queued inside a folder that just landed is no longer put off** for five seconds
(B120).
- **A move right after a rename lands.** It failed with `Input/output error` through the mount
and with "out of date" from `pdfs move`, and a rename across folders took up to 10 seconds
(B111, B121).
- **A rename across folders no longer fails with `Input/output error`** when the source folder
holds the new name (B122).
- **A second write open on a file just written no longer fails** with `Input/output error`
(B123).
- **After a replacing rename fails, the file it was replacing stays in the mount** (B124).
- **A write to a file whose create could not be read back no longer lands as a conflict copy**
(B131).
- **A file just moved is found by its new path.** A lookup right after the rename could miss it
(B135).
### Notes
- The simulation runs found bugs that are still open: B125 to B130 and B132 to B134 in
`docs/BUGS.md`. The worst are B127 and B128 (a file created offline can land twice, or be lost
when created over one deleted offline), B129 (a file deleted while its create is on the wire
stays on Drive) and B130 (a read that a remote change invalidates can hang the daemon).
## [2.8.2] — 2026-10-01
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.2**.
### Fixed
- **Folders shared with you no longer go missing when one person shared several.** When
several accepted shares came from the same owner's volume, only the first appeared under
Shared with me, and the rest were skipped with `missing key`. `proton-drive-rs` 0.7.2 resolves
each shared root with its own key (B112, #25).
## [2.8.1] — 2026-09-30
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.1**.
### Fixed
- **The service no longer starts before the desktop session.** `proton-drive.service` was
wanted by `default.target`, which is reached before the desktop hands `DISPLAY` /
`WAYLAND_DISPLAY` to the user's systemd. The first keyring call then started the Secret
Service without a display; on KDE `ksecretd` crashed at every login and the first start of the
service failed. The unit is now wanted by `graphical-session.target`. Existing installs switch
over on the next sign-in or "Start on login" toggle, or with
`systemctl --user reenable proton-drive.service` (#23).
## [2.8.0] — 2026-09-30
Schema: **35** (unchanged). Requires `proton-drive-rs` **0.7.1**.
### Added
- **`pdfs trash --json`** prints the trash listing as JSON.
- **The FUSE acceptance suite tests the signed-in account by default.**
`scripts/fuse-acceptance.sh` with no arguments creates test folders in My files and new
on-demand and mirror sync folders, runs every case in each, and removes all of it afterwards,
also on a failure or interrupt. A run that dies is cleaned up by the next one. New cases cover
block boundaries, unusual names, a deep tree, a wide directory, rename patterns, handle coherency,
throughput and B100. `--offline-only` runs the account-free suite as before.
### Changed
- **The mount keeps working through a network drop.** When Drive stops answering, the daemon
now marks itself offline within about 20 seconds instead of failing every call for up to two
minutes. While offline, creating, renaming and deleting files and folders is queued instead of
failing with `Input/output error`, folders list from the local cache, and a read that needs
data from Drive waits up to 30 seconds for the connection. The daemon checks every few seconds
for the connection to return, then uploads the queued changes. A file whose creation timed
out but reached Drive anyway is taken over rather than uploaded again as a conflict copy.
- **A dead connection is noticed sooner.** `proton-drive-rs` 0.7.1 sends TCP keepalives, so a
connection that died silently fails instead of waiting for its request timeout.
### Fixed
- **Truncating and then writing past the end no longer brings old bytes back.** The gap
between the new end and the write now reads as zeros, as it should, instead of the file's
earlier content. When that earlier content had not been uploaded yet, the file failed with
`Input/output error` and its upload retried forever (B101).
- **A write made while a new file is first uploading is no longer lost.** Drive could keep an
empty or older copy of a file written right after it was created, and closing a file held open
across its first upload could fail (B101).
- **Removing a folder also cancels the pending uploads of the files inside it.** Before, an
edit that had not reached Drive yet kept retrying against the trash forever and kept its
staged copy on disk. `pdfs rm` now also drops what was queued for what it removes (B102).
- **A program reading a file keeps its data when another program replaces the file.** Saving
over a file by renaming a new copy onto it (what most editors and `os.replace` do) made
reads through handles already open on the old file fail with `Input/output error` (B104).
- **Short-lived files no longer come back as conflict copies.** A file deleted right after it
was written, such as a SQLite journal, could be uploaded as a `recovered-…` conflict copy in
the Drive root (B105).
- **Editors can save over a file several times in a row.** The fix for B104 could leave the
upload of a replaced file queued, and the next save onto the same name failed with
`Input/output error` (B106).
- **Renaming a file back right after renaming it no longer fails.** Drive can briefly call the
second rename out of date, and the daemon now waits it out instead of returning an I/O error
(B111).
- **Reading a freshly listed file no longer hangs.** When a read had to fetch the file's real
size first, the daemon could wait on the read it was serving, and the program froze until
the daemon restarted (B110).
- **Databases and other files kept open while written no longer read back empty.** When the
folder was listed again while a file was still open, its size dropped to what Drive held,
often 0 bytes, and SQLite reported `database disk image is malformed` (B109).
- **Deleting files during a mirror pass no longer marks the folder as failed.** A file removed
after the pass planned its upload was counted as an error, so the folder showed `error` until
a later pass (B108).
- **A read no longer hangs when the connection stalls.** A read that gets no answer from Drive
within two minutes now fails with `Input/output error` instead of blocking the program for
as long as half an hour (B107).
- **The trash no longer freezes after a server call hangs.** Until the daemon restarted, the
trash kept showing the same entries, and a folder just deleted could not be found there to
delete permanently (B103).
- **Freshly listed files read correctly.** A file whose real size had not been fetched yet
reported its encrypted size, about 53 bytes too big, and reading it failed with
`Input/output error`. This happened most often right after copying a large folder. Reads and
write opens now fetch the real size first, and a big folder walk no longer skips size lookups
(B100).
## [2.7.0] — 2026-09-29
Schema: **35** (unchanged).
### Added
- **Moves between locations without transferring content.** `pdfs move` and the app's
**Move to…** now take a file or folder from My files into a synced folder, or between two
synced folders, on-demand or mirrored. The move happens on Proton Drive, so nothing is
downloaded or uploaded again. `mv` between two locations still copies, because the kernel
refuses a rename between two mounts. The app's Move dialog lists every location at its top
level. A move is refused, with the reason, when the source still has changes that are not on
Proton Drive, when the two locations are on different volumes, or when a synced folder stays
busy with a sync pass.
- **`pdfs move` takes several sources.** `pdfs move a b c DEST` moves all three and
reports each one that could not move.
- **Moving into a mirrored folder reuses cached content.** A file already cached by
an online-only folder is copied into place instead of downloaded again.
### Changed
- **New project homepage: [proton-drive.narl.io](https://proton-drive.narl.io).** The About
dialog's website link, the README, and the package metadata (Cargo, PKGBUILDs, `.deb`,
`.rpm`) point there. The source code link stays on GitHub.
## [2.6.2] — 2026-09-28
Schema: **35** (unchanged).
### Fixed
- **Photos remembers the Albums and Places views.** Leaving Photos from Albums or Places and
coming back showed the timeline while the Albums or Places toggle stayed selected. The page
now reopens on the selected view, and Places keeps its Grid or Map layout.
- **The Sync page keeps a steady width.** A long file name in Transfers no longer widens the
page. Overview and Folders now fill the width next to the sidebar.
## [2.6.1] — 2026-09-28
Schema: **35** (unchanged).
### Added
- **`pdfs-prompt --preload`.** Starts the launcher hidden, for session autostart such as
Hyprland's `exec-once`, so the first shortcut opens it as fast as later ones. It does nothing
when the launcher is already running.
### Fixed
- **Launcher invocations no longer pile up.** Each shortcut press after the first used to leave
a `pdfs-prompt` process behind after handing the request to the running launcher. It now exits.
## [2.6.0] — 2026-09-28
The search launcher (`pdfs-prompt`) is rebuilt around a preview pane, and it now looks like the
app it belongs to.
Schema: **35** (unchanged).
### Added
- **Launcher preview pane.** The selected result shows a large thumbnail, its type, size,
modified date, location and whether it is on this computer, with Open (Enter), Show in Folder
(Ctrl+Enter) and Copy Path (Ctrl+C, or Ctrl+Shift+C while text is selected in the search box).
Show in Folder selects the file in the file manager through `org.freedesktop.FileManager1`
and falls back to opening the folder.
- **Recent files.** With an empty search box, the launcher lists the last files opened from it
above the pinned ones. Files opened often and lately also rank higher in search results. The
history lives in `prompt-history.json` in the state directory and forgets local files that
no longer exist.
- **Thumbnails in the launcher.** Images show a thumbnail in the result list and the preview,
from the desktop's thumbnail cache, decoded locally (and written back to that cache), or
fetched by the daemon for Drive files.
- **Drag and drop from the launcher.** A result can be dragged into a file manager or another
application as a file.
### Changed
- **The launcher follows the app's theme.** It uses the system accent colour, light and dark
style, and the opt-in Proton theme, instead of always pinning Proton purple. Changes to the
theme or to `open_with` apply the next time the launcher opens.
- **Launcher layout.** Results are grouped into Recent, Pinned in Proton Drive and Best matches;
matched characters are bold, and each row shows its sync state, size and age. A click selects
a row and a double-click opens it; the search box keeps the keyboard focus throughout.
- **Faster search as you type.** The launcher searches 80 ms after the last keystroke instead
of 120 ms.
### Fixed
- **The launcher no longer freezes on a stuck mount.** Checking whether a Drive file is in the
mount runs off the main thread and gives up after 2 seconds, falling back to the daemon.
Escape cancels a Drive file that is still opening.
### Documentation
- **Rewritten user documentation.** New pages for installation, the user guide, the command-line
reference, configuration and troubleshooting, and a documentation index in `docs/README.md`.
The README is shorter and points to them.
- **Recovery** describes the current restore flow (adopting a computer, `pdfs sync restore`, the
machine profile) instead of the old manual runbook.
- **Architecture** covers push events, remote event streams, the revision block geometry, the
thread layout of 2.5.0 and the machine profile. The development page is now a developer guide,
and open work moved to `docs/ROADMAP.md`.
- `scripts/publish-wiki.sh` publishes the user pages to the GitHub wiki.
## [2.5.0] — 2026-09-28 ## [2.5.0] — 2026-09-28
A polish pass over the desktop app: the daemon pushes changes instead of being polled, pages A polish pass over the desktop app: the daemon pushes changes instead of being polled, pages
@@ -320,7 +814,7 @@ Schema: **31** (unchanged).
opened in between. opened in between.
- **Photo viewer rewrite.** The lightbox's controls fade out after two seconds without pointer - **Photo viewer rewrite.** The lightbox's controls fade out after two seconds without pointer
movement. Photos zoom with Ctrl+scroll, a pinch, `+`/`-`, `1` for 100 % and double-click, and movement. Photos zoom with Ctrl+scroll, a pinch, `+`/`-`, `1` for 100 % and double-click, and
pan by dragging; `0` fits them again. The details panel docks beside the photo instead of pan by dragging; `0` fits them again. The details panel docks beside the photo instead of 120 ms
covering it. Stepping past the last loaded photo loads the next page, and the counter shows the covering it. Stepping past the last loaded photo loads the next page, and the counter shows the
library's total rather than the loaded count. A video plays inline. Save a Copy uses the photo's library's total rather than the loaded count. A video plays inline. Save a Copy uses the photo's
real name and reports failures, and Open With… offers the app chooser. real name and reports failures, and Open With… offers the app chooser.
@@ -658,7 +1152,7 @@ thumbnail.
slider for the icon grid, and account storage usage. Zoom resizes the realised grid cells rather slider for the icon grid, and account storage usage. Zoom resizes the realised grid cells rather
than rebuilding the model, so dragging the slider does not tear down the selection; the default than rebuilding the model, so dragging the slider does not tear down the selection; the default
size reproduces the previous fixed 72 px tiles. Capacity stays out of the bar until a real size reproduces the previous fixed 72 px tiles. Capacity stays out of the bar until a real
reading lands, and the quota fetch (60 s TTL) now also runs while Files is on screen instead of reading lands, and the quota fetch (60 s TTL) now also runs while Files is on screen instead of 120 ms
only in Settings. only in Settings.
- Repository housekeeping: `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, `SECURITY.md`, issue and pull - Repository housekeeping: `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, `SECURITY.md`, issue and pull
request templates, a project logo, and workspace crate metadata (`repository`, `homepage`, request templates, a project logo, and workspace crate metadata (`repository`, `homepage`,
@@ -1177,8 +1671,26 @@ First stable release: FUSE files-on-demand mount, sync daemon under `proton-driv
- The outstanding FUSE defects tracked in `docs/BUGS.md`, plus a truncate defect, validated - The outstanding FUSE defects tracked in `docs/BUGS.md`, plus a truncate defect, validated
by a new POSIX compliance suite for the filesystem. by a new POSIX compliance suite for the filesystem.
[2.4.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.4.0 [3.1.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v3.1.2
[3.1.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v3.1.1
[3.1.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v3.1.0
[3.0.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v3.0.0
[2.10.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.10.0
[2.9.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.9.0
[2.8.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.2
[2.8.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.1
[2.8.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.8.0
[2.7.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.7.0
[2.6.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.6.2
[2.6.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.6.1
[2.6.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.6.0
[2.5.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.5.0
[2.4.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.4.0
[2.3.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.3.0 [2.3.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.3.0
[2.2.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.2.2 [2.2.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.2.2
[2.2.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.2.1 [2.2.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v2.2.1
+195
View File
@@ -0,0 +1,195 @@
# Command-line reference
`pdfs` controls the Proton Drive service from a terminal and from scripts. Almost every command
talks to the running service (`proton-drive.service`); `login`, `logout`, `mount`, `daemon` and
`diagnose` work without it.
`pdfs <command> --help` prints the full help of any command. This page groups them by task.
- [Conventions](#conventions)
- [Account and service](#account-and-service)
- [Files](#files)
- [Available offline](#available-offline)
- [Synced folders](#synced-folders)
- [Conflicts](#conflicts)
- [Computers](#computers)
- [Trash](#trash)
- [Versions](#versions)
- [Sharing](#sharing)
- [Photos](#photos)
- [Activity and transfers](#activity-and-transfers)
- [Maintenance and diagnostics](#maintenance-and-diagnostics)
- [Scripting with `--json`](#scripting-with---json)
## Conventions
- **Paths** name items in My files. They may be absolute paths inside the Proton Drive folder
(`~/ProtonDrive/Documents/a.pdf`) or relative to it (`Documents/a.pdf`).
- **UIDs** identify items that have no path in the mount: photos, trashed items, shared items and
computers. The listing commands print them.
- **Exit codes**: `0` on success, non-zero on any failure, including an error reported by the
service.
- **Output** is English and stable enough for `grep`; use `--json` where it is available.
## Account and service
| Command | Description |
|---|---|
| `pdfs login [-u EMAIL]` | Sign in (password, then a 2FA code if the account has one) and store the session in the system keyring. Restarts the service if it is running. |
| `pdfs logout` | Remove the stored session. |
| `pdfs status` | Account, mount, queue and cache state. |
| `pdfs quota` | Storage used across all Proton products. |
| `pdfs locations` | Every local place the service occupies: the Proton Drive folder and each synced folder, with mode and state. |
| `pdfs daemon [MOUNTPOINT]` | Run the service in the foreground: wait for a session, mount, stay mounted. `proton-drive.service` runs this. |
| `pdfs mount [MOUNTPOINT]` | Mount once and block until unmounted. Defaults to `~/ProtonDrive`. |
`pdfs login` cannot show a CAPTCHA. If Proton asks for one, sign in with `pdfs-app` instead.
## Files
| Command | Description |
|---|---|
| `pdfs ls [PATH]` | List a folder. |
| `pdfs search QUERY [--in FOLDER] [--limit N]` | Search names in the local index. |
| `pdfs mkdir PARENT NAME` | Create a folder. |
| `pdfs rename PATH NEW_NAME` | Rename a file or folder. |
| `pdfs move PATH... NEW_PARENT` | Move one or more files or folders into another folder. The two may be in different locations (My files, an on-demand or a mirrored synced folder); the move happens on Proton Drive, without downloading or uploading anything. |
| `pdfs rm PATH` | Move to the Trash. |
| `pdfs upload SOURCES... [-t FOLDER]` | Upload local files and folders (recursively) in the background. Follow with `pdfs transfers`. |
| `pdfs refresh [TARGET] [--full]` | Forget a cached listing so the next read fetches it again. `TARGET` is a folder, `trash` or `photos`; `--full` with `photos` re-reads every photo's metadata. |
Ordinary file tools (`cp`, `mv`, `rsync`, editors) work on the Proton Drive folder directly. These
commands are for scripts and for cases where the mount is not available.
## Available offline
| Command | Description |
|---|---|
| `pdfs pin PATH` | Download a file completely and keep it on this computer. |
| `pdfs unpin PATH` | Release it back to online only. |
| `pdfs pins` | List files kept available offline. |
## Synced folders
| Command | Description |
|---|---|
| `pdfs sync add PATH` | Back up a local folder under this computer and keep it in sync. |
| `pdfs sync list` | List synced folders with their IDs and state. |
| `pdfs sync rm ID [--delete-remote]` | Stop syncing. Local files stay; `--delete-remote` also removes the copy on Proton Drive. |
| `pdfs sync mode ID mirror\|ondemand` | Switch between a full local copy (`mirror`) and online only (`ondemand`). |
| `pdfs sync now [ID]` | Start a sync pass now, for all folders or one. |
| `pdfs sync pause [--for 1h] [--folder ID]` | Pause uploads and synced folders. Reading files keeps working. `--for` ends the pause by itself (`30m`, `1h`, `2d`). |
| `pdfs sync resume [--folder ID]` | Resume after a pause. |
| `pdfs sync limit [--up RATE] [--down RATE]` | Cap bandwidth, for example `500K` or `2M` per second. `0` removes a cap. Without options, prints the current caps. |
| `pdfs sync queue` | List changes that have not reached Proton Drive yet. |
| `pdfs sync issues` | List queued changes Proton Drive refused or that keep failing, and what to do. |
| `pdfs sync export ID DEST` | Save a copy of a queued upload's content. `DEST` may be a folder; an existing file is never overwritten. The upload stays queued. |
| `pdfs sync discard ID [--yes]` | Drop a queued change and undo it here: a new file or folder goes, anything else goes back to what Proton Drive has. Content not exported first is lost. |
| `pdfs sync retry [ID]` | Retry one queued operation, or every failed one. |
| `pdfs sync restore [--device UID] [--yes]` | Re-attach this computer's backed-up folders to local folders and download them. `--device` restores another computer's folders. See [RECOVERY.md](RECOVERY.md). |
## Conflicts
| Command | Description |
|---|---|
| `pdfs conflicts` | List `(sync-conflict …)` copies. |
| `pdfs conflicts resolve PATH --keep original\|copy\|both [--name NAME]` | `original` trashes the copy; `copy` trashes the original and gives the copy its name; `both` renames the copy to `--name`. |
## Computers
| Command | Description |
|---|---|
| `pdfs devices list` | List computers registered to the account. |
| `pdfs devices rename UID NAME` | Rename a computer. |
| `pdfs devices rm UID` | Remove a computer and its backup from the account. |
| `pdfs devices adopt UID` | Make this machine continue another computer's backup, for example after a reinstall. `--clear` returns to matching by hostname. |
## Trash
| Command | Description |
|---|---|
| `pdfs trash` | List the Trash with the UIDs the next commands take. |
| `pdfs restore UIDS...` | Put items back where they were. |
| `pdfs delete-forever UIDS...` | Delete items permanently. |
| `pdfs empty-trash` | Delete everything in the Trash permanently. |
## Versions
| Command | Description |
|---|---|
| `pdfs versions list PATH` | List a file's versions, newest first. |
| `pdfs versions restore PATH ID` | Make an earlier version current. Runs on the server; nothing is re-uploaded. |
| `pdfs versions save PATH ID DEST` | Write an earlier version to a local file. Refuses to overwrite `DEST`. |
| `pdfs versions rm PATH ID` | Delete an earlier version permanently. The current version cannot be deleted. |
## Sharing
| Command | Description |
|---|---|
| `pdfs share PATH EMAILS... [--role viewer\|editor\|admin] [--message TEXT]` | Invite people to a file or folder. |
| `pdfs members PATH` | List members, pending invitations and the public link. |
| `pdfs share-role PATH ID KIND ROLE` | Change a member's or invitation's role. |
| `pdfs unshare PATH [ID KIND]` | Stop sharing, or remove one member or invitation. |
| `pdfs public-link create PATH [--role] [--password] [--expires EPOCH]` | Create a public link. `--expires` takes Unix seconds. |
| `pdfs public-link remove PATH ID` | Remove a public link. |
| `pdfs shared` | List what you share. |
| `pdfs shared-with-me [UID]` | List what others share with you, or the contents of one shared folder. |
| `pdfs shared-get UID [DEST]` | Download a file shared with you. |
| `pdfs leave UID` | Leave a share. |
| `pdfs invitations list\|accept ID\|reject ID` | Manage invitations addressed to you. |
| `pdfs bookmarks list\|add URL [--password]\|rm TOKEN` | Manage saved public links. |
## Photos
| Command | Description |
|---|---|
| `pdfs photos [--limit N] [--offset N] [--favorites]` | List the timeline, newest first. |
| `pdfs open-photo UID` | Download a photo and print its cached path. |
| `pdfs favorite UID [--remove]` | Mark or unmark a favorite. |
| `pdfs albums` | List albums, including those shared with you. |
| `pdfs album UID` | List an album's photos. |
| `pdfs new-album NAME` | Create an album and print its UID. |
| `pdfs rename-album UID NAME` | Rename an album. |
| `pdfs delete-album UID` | Delete an album. Its photos stay in the timeline. |
| `pdfs add-to-album UID PHOTOS...` | Add photos to an album. |
| `pdfs remove-from-album UID PHOTOS...` | Take photos out of an album. |
| `pdfs import-google-photos ARCHIVES... [--dry-run] [--wait]` | Import a Google Takeout export. Pass every `.zip` of the export. |
| `pdfs import-status` / `pdfs cancel-import` | Follow or stop a running import. |
| `pdfs redate-photos [--dry-run] [--from DATE] [--to DATE] [--limit N] [--wait]` | Fix photos filed under their upload day when the file name carries the real date. |
| `pdfs redate-status` / `pdfs cancel-redate` | Follow or stop a running re-date. |
## Activity and transfers
| Command | Description |
|---|---|
| `pdfs activity [-l N]` | Recent activity, newest first. |
| `pdfs transfers` | Uploads and downloads in progress. |
## Maintenance and diagnostics
| Command | Description |
|---|---|
| `pdfs diagnose` | Check the installation and print a report. Works without the service. Exits non-zero if a check fails. |
| `pdfs diagnostics` | What a running service is doing now: worker threads, queues, requests in flight, memory. For a service that has stopped answering. |
| `pdfs cache inspect [--deep]` | Database size, reclaimable space, row counts, cache use against the limit. `--deep` also runs SQLite's integrity check (slow). |
| `pdfs cache vacuum` | Compact the database. Needs free space for a second copy of it while it runs. |
| `pdfs cache clear` | Delete cached file content. Files kept available offline stay. |
[TROUBLESHOOTING.md](TROUBLESHOOTING.md) explains when to use each of these.
## Scripting with `--json`
The global `--json` flag makes query commands print JSON: `status`, `ls`, `pins`, `sync list`,
`devices list`, `locations`, `transfers`, `activity`, `trash` and `cache inspect`. Commands that change
something keep their human-readable output; check their exit code.
```bash
pdfs --json status | jq -r '.mount.mountpoint'
pdfs --json sync list | jq -r '.items[] | select(.state != "idle") | .local_path'
pdfs --json cache inspect | jq '.db_reclaimable_bytes'
```
- The payload is unwrapped: a list is `{"items": [...]}`, never the service's internal message
name.
- An error from the service is printed as JSON with a machine-readable `kind` and still exits
non-zero, so `set -e` and `if pdfs …` behave as expected.
+218
View File
@@ -0,0 +1,218 @@
# Configuration
Most settings are in the app under **Preferences**. This page documents the configuration file
behind them, the settings that exist only there, the environment variables the programs read,
and where the client keeps its data.
- [Files and directories](#files-and-directories)
- [`config.json`](#configjson)
- [Environment variables](#environment-variables)
- [The systemd service](#the-systemd-service)
## Files and directories
| Path | Contents |
|---|---|
| `~/.config/proton-drive-linux/config.json` | Settings (this page) |
| `~/.local/state/proton-drive-linux/cache.db` | Metadata database: names, folder tree, sync state, upload queue, photo index |
| `~/.local/state/proton-drive-linux/sdk_cache.db` | Encrypted cache of Proton Drive metadata. Safe to delete while the service is stopped |
| `~/.local/state/proton-drive-linux/control.sock` | Socket the app, tray, prompt and CLI use to reach the service |
| `~/.cache/proton-drive-linux/content/` | File content cache, see below |
| `~/ProtonDrive` | The Proton Drive folder (default mountpoint) |
| System keyring | The session. Never written to disk by the client |
The directories honour `XDG_CONFIG_HOME`, `XDG_STATE_HOME` and `XDG_CACHE_HOME`. They must be real
directories owned by you with mode `0700`; the service refuses to start otherwise.
Inside `content/`:
| Directory | Contents | Safe to delete? |
|---|---|---|
| top level, `blocks/`, `thumbs/` | Cached file content and thumbnails | Yes — use `pdfs cache clear` |
| `scratch/` | Files open for writing right now | **No** |
| `staging/` | Closed files waiting to upload | **No** — may be the only copy |
| `recovery/` | Writes rescued after a crash, waiting to be queued | **No** — may be the only copy |
> [!CAUTION]
> Never delete `staging/` or `recovery/` by hand. Check `pdfs sync queue` first. Content there has
> not reached Proton Drive and exists nowhere else.
Everything in these directories except the keyring is **decrypted**. See
[ARCHITECTURE.md §10](ARCHITECTURE.md#10-threat-model-what-this-client-writes-to-disk-in-plaintext).
## `config.json`
The file is created with defaults on first start. Every key is optional; a missing key takes its
default. Edit it while the app is closed, then restart the service
(`systemctl --user restart proton-drive.service`) for service settings to apply.
An unreadable file is never overwritten: the programs log an error and run with defaults until you
fix it.
```json
{
"mountpoint": "/home/you/ProtonDrive",
"cache_budget": 10737418240,
"upload_limit": null,
"download_limit": 5242880,
"ignore_patterns": ["node_modules/", "target/", "*.iso"],
"language": "de",
"proton_theme": true,
"online_map": false,
"tray_hidden": false,
"files_view": { "list": true, "sort": "modified", "descending": true, "folders_first": true },
"open_with": {
"terminal": ["alacritty", "-e"],
"rules": [{ "match": ["@text"], "command": ["nvim"], "terminal": true }]
},
"prompt": { "mode": "fzf", "menu_limit": 50 }
}
```
### Reference
| Key | Type | Default | In the app | Meaning |
|---|---|---|---|---|
| `mountpoint` | path | `~/ProtonDrive` | General → Location | Where My files is mounted |
| `cache_budget` | bytes | 5 GiB | Storage → Cache | Soft limit for cached content. `0` means unlimited. Available-offline files do not count against it |
| `upload_limit` | bytes/s | none | General → Network | Shared cap for all uploads. `0` or `null` means none |
| `download_limit` | bytes/s | none | General → Network | Shared cap for all downloads, reads from the mount included |
| `ignore_patterns` | list | see below | — | Global ignore rules for synced folders |
| `device_uid` | string | none | Computers → Continue This Backup Here… | The computer this machine continues. Unset means "match by hostname" |
| `conflict_sweep` | `off` / `report` / `enforce` | `report` | — | See [`conflict_sweep`](#conflict_sweep) |
| `local_first` | bool | `true` | — | See [`local_first`](#local_first) |
| `language` | gettext code | system | General → Appearance | Language of the app, tray and prompt, such as `de` or `pt_BR` |
| `proton_theme` | bool | follow system | General → Appearance | Proton's colours instead of the system theme |
| `online_map` | bool | `false` | General → Appearance | Street map for Places from OpenFreeMap |
| `tray_hidden` | bool | `false` | General → Appearance | Hide the tray icon |
| `files_view` | object | grid, by name | My files view menu | `list`, `sort` (`name`, `size`, `modified`), `descending`, `folders_first` |
| `open_with` | object | `xdg-open` | — | See [`open_with`](#open_with) |
| `prompt` | object | built-in window | — | See [`prompt`](#prompt) |
`app_version` and `user_agent` identify the client to Proton. When the keys are absent, the
client sends its own release (`external-drive-linux@<version>-stable`, `proton-drive-linux/<version>`).
Leave them out unless a support case asks for a different value. The fixed
`external-drive-linux@0.1.0-alpha` and `proton-drive-linux/0.1.0` that releases up to 3.1.0 wrote
here are treated as absent and removed on the next save.
### `ignore_patterns`
Gitignore-style rules applied to every synced folder, in addition to each folder's own
`.pdfsignore`. When the key is absent, these defaults apply:
```json
[".git/", ".hg/", ".svn/", "node_modules/", "target/", ".venv/", "__pycache__/",
"*~", "*.swp", "*.tmp", ".DS_Store", "Thumbs.db"]
```
Set it to `[]` to turn the defaults off. See
[USER_GUIDE.md](USER_GUIDE.md#excluding-files-with-pdfsignore) for how ignoring behaves.
### `conflict_sweep`
A background task looks for `(sync-conflict …)` copies that are byte-identical to the file they
were copied from.
| Value | Behaviour |
|---|---|
| `off` | Do not run. |
| `report` | Log and show what would be removed; remove nothing. **Default.** |
| `enforce` | Move copies proven identical (same size and SHA-1) to the Trash. |
`PDFS_CONFLICT_SWEEP` overrides the setting for one run of the service.
### `local_first`
A new folder, a new file, a rename or a delete made through the mount is recorded locally and the
call returns at once. The change goes to Drive from the queue, the same way it does offline, and
`pdfs status` counts it until it lands.
Set it to `false` to have these calls wait for Drive again, as before 3.0.0. The switch is there
for one release, in case the queued path misbehaves for you; please report it if it does.
### `open_with`
Decides how files picked in the search launcher or the app are opened. Rules are tried in order;
the first match wins.
```json
{
"open_with": {
"terminal": ["alacritty", "-e"],
"default": ["xdg-open"],
"rules": [
{ "match": ["@text", "*.rs", "*.toml"], "command": ["nvim"], "terminal": true },
{ "match": ["*.png", "*.jpg"], "command": ["imv"] },
{ "match": ["@document"], "command": ["$EDITOR", "{}"], "terminal": true }
]
}
}
```
- `match` takes file-name globs (`*.md`, `notes-*.txt`) or the classes `@dir`, `@text`,
`@document`, `@image`, `@media` and `@any`. Patterns are matched against the name in Drive, not
against the cached file, which is stored under a content hash.
- `command` is an argument list, not a shell line. `{}` is replaced by the path; without it the
path is appended. `$VAR` expands from the environment.
- `"terminal": true` runs the command inside `terminal`. When `terminal` is unset, the client uses
`$PDFS_TERMINAL`, then `$TERMINAL`, then the first known terminal on `PATH`, and adds that
terminal's "run this command" flag for you.
- `default` replaces `xdg-open` for files no rule matches.
### `prompt`
Settings for `pdfs-prompt`.
| Key | Default | Meaning |
|---|---|---|
| `mode` | `gtk` | Front end used without a flag: `gtk`, `dmenu` or `fzf` |
| `menu` | first installed of fuzzel, rofi, wofi, tofi, bemenu, dmenu | Launcher for `dmenu` mode, as an argument list. `{prompt}` is replaced by the prompt text; without it, the launcher's own prompt flag is added |
| `terminal` | first installed of foot, ghostty, kitty, alacritty, wezterm, xterm | Terminal for `fzf` mode. `{cmd}` is replaced by the command; without it, the command is appended |
| `menu_limit` | `50` | Results shown in `dmenu` and `fzf` mode |
| `menu_icons` | `true` | File-type icons in launchers that support them |
```json
{
"prompt": {
"mode": "dmenu",
"menu": ["fuzzel", "--dmenu", "--width", "60"],
"terminal": ["foot", "--app-id=pdfs-prompt", "--"]
}
}
```
Command-line flags win over the file: `--gtk`, `--dmenu`, `--fzf` and `--menu`.
## Environment variables
| Variable | Read by | Effect |
|---|---|---|
| `RUST_LOG` | all | Log filter, for example `RUST_LOG=debug` or `RUST_LOG=pdfs_fuse=debug,info`. Default `info` |
| `PDFS_CONFLICT_SWEEP` | service | Overrides `conflict_sweep` (`off`, `report`, `enforce`) |
| `PDFS_TERMINAL`, `TERMINAL` | app, prompt | Terminal for `open_with` rules with `"terminal": true` |
| `LANGUAGE`, `LC_ALL`, `LC_MESSAGES`, `LANG` | app, tray, prompt | Interface language when `language` is unset |
| `PDFS_LOCALEDIR` | app, tray, prompt | Load translations from another directory (development) |
| `PDFS_STAND_IN_WIDGETS` | app | `1` uses the libadwaita 1.5 fallback widgets on newer systems (development) |
The tests have their own variables (`PDFS_ACCEPTANCE_*`, `PDFS_SIM_*`, `PDFS_MIGRATE_DB`),
listed in [TESTING.md](TESTING.md).
## The systemd service
`proton-drive.service` is a user unit started with your graphical session
(`WantedBy=graphical-session.target`). It restarts the service whenever it exits, restarts it
when it stops answering for two minutes (`WatchdogSec=120`), throttles it above 2 GiB of memory
(`MemoryHigh`) and stops it at 6 GiB (`MemoryMax`). To change the unit without editing the
packaged file, use a drop-in:
```bash
systemctl --user edit proton-drive.service
```
```ini
[Service]
Environment=RUST_LOG=pdfs_fuse=debug,info
Environment=PDFS_CONFLICT_SWEEP=off
```
Logs go to the journal: `journalctl --user -u proton-drive.service -f`.
+134 -174
View File
@@ -1,214 +1,174 @@
# Development and Verification Status for 1.0 # Development guide
This document summarizes the work still pending after the 1.0.0 release branch and records the most important recently completed changes. [`BUGS.md`](BUGS.md) is the authoritative issue ledger; [`TESTING.md`](TESTING.md) defines the acceptance commands. How to build, run, debug and release the client from source. Read
[CONTRIBUTING.md](../CONTRIBUTING.md) for the review rules and [ARCHITECTURE.md](ARCHITECTURE.md)
for how the pieces fit together.
--- - [Setup](#setup)
- [Repository layout](#repository-layout)
- [Running from source](#running-from-source)
- [Quality gates](#quality-gates)
- [Debugging](#debugging)
- [Common changes](#common-changes)
- [Releasing](#releasing)
## 1. Core Feature Roadmap (Horizon Tasks) ## Setup
Consolidated from the historical feature roadmap; current bug status lives in [`BUGS.md`](BUGS.md).
* **Local Cache & Metadata Encryption at Rest (Priority 5 / Horizon 1)**: Encrypt the SQLite database `cache.db` on disk using SQLCipher (`rusqlite` feature `bundled-sqlcipher`), and encrypt raw content cache blocks (`content/blocks/` and `content/scratch/`) using a fast symmetric scheme like AES-GCM or ChaCha20-Poly1305. The encryption key should be derived from the OS Keyring. (The SDK entity cache in `sdk_cache.db` is already encrypted at rest — see *Persistent SDK Entity Cache* below — but `cache.db` and the content cache are not.) You need Rust 1.96 or newer and the native development packages listed under
* **Active Bandwidth Throttling & Traffic Shaping (Horizon 1)**: Add speed governors for daemon uploads and downloads. Support configuration variables `max_upload_rate_kbps` and `max_download_rate_kbps` in `config.json` that can be dynamically adjusted over the Unix control socket. [Build dependencies](INSTALL.md#build-dependencies) for your distribution. Then:
* **Interactive & Policy-Based Conflict Resolution (Horizon 1)**: Implement configurable conflict resolution policies (`rename-local`, `rename-remote`, `prefer-local`, `prefer-remote`, `interactive`). Add a "Conflicts" tab to the GUI to notify the user, block sync, and launch external visual diff tools (like Meld or KDiff3).
* **Multi-Account / Multi-Profile Support (Horizon 3)**: Enable running multiple profiles concurrently. Support separate configuration namespaces (e.g. `profiles/<profile_name>/`), independent SQLite databases, distinct mount paths (e.g. `~/ProtonDrive/Personal` and `~/ProtonDrive/Work`), and multiple daemon processes.
* **Custom FUSE Mount Options (Horizon 1)**: Expose custom mounting parameters in configuration (e.g. `ro` for read-only, `direct_io` to bypass kernel caching, `allow_other`). Auto-generate systemd user mount units upon sync folder configuration.
* **Symbolic Link & Hard Link Virtualization (Horizon 2)**: Add symbolic link virtualization inside the metadata store (`NodeType::Symlink`) by intercepting `readlink(2)` and `symlink(2)`. Targets should sync remotely as small encrypted metadata placeholder files on Drive (`proton-vfs-symlink: <target>`).
* **Desktop File Manager Integration (Horizon 2)**: Develop shell extension plugins for popular Linux file managers (Nautilus Python plugin, Dolphin Service Menus, Thunar custom actions) to offer context-menu shortcuts ("Pin", "Unpin", "Copy Share Link", "View Version History") and file overlay sync badges.
* **Integrated Photo Gallery Enhancements (Horizon 2)**: Album *editing* — rename, cover photo, delete, and removing photos from an album — is still unported in the SDK (TS `updateAlbum` / `deleteAlbum` / `removePhotos`), as is adding a photo that lives on someone else's volume (TS `copyPhoto`). Album **creation** and adding own-volume photos landed with the Google Photos import. Still open: an auto-upload Pictures folder pipeline, and a GUI for creating albums by hand. Exif display, the date-based timeline scrubber and favorites are implemented.
* **Google Photos import follow-ups**: the importer uploads photos one at a time — a large export would finish sooner with a bounded concurrent upload pool. Trash folders are recognized by a hardcoded list of localized names ([`takeout.rs`](../crates/pdfs-core/src/takeout.rs) `TRASH_FOLDERS`); a locale not on it imports its trash as ordinary photos. Motion-photo `.MP`/`.MV` sidecar videos are ignored rather than uploaded as related photos. Live validation against a real export is pending.
* **Random-Access Media Streaming seek support (Horizon 2)**: Intercept out-of-order sparse reads (player seeks) in `pdfs-fuse` and prioritize downloading blocks around the seek offset, aborting/postponing sequential pre-fetches.
* **Dynamic Sync Dashboard & Queue Visualization (Horizon 2)**: GUI real-time transfer list (progress bars, speed, ETA), sync history feed, storage usage breakdown, and global pause/resume sync controls.
* **Interactive Terminal UI (TUI) Mode (Horizon 3)**: Implement `pdfs tui` using the `ratatui` crate to show transfer speed, sync status, active transfer queue with progress bars, and scrolling daemon logs.
* **Shutdown Safety & Write Queue Visibility (Horizon 1 / P4)**: Register systemd inhibitor locks in `pdfs-fuse` when there are outstanding staging writes to prevent shutdown or sleep, pop up warnings when exiting the GUI, and build a blocking `pdfs sync flush` command.
* **Block-Level Deduplication (Horizon 3)**: Transition to a Content-Addressable block cache where cached blocks are stored on disk by their SHA-256 hash. Map logical ranges in the database: `(node_uid, block_idx) -> block_hash` to avoid storing or downloading identical blocks.
* **Pre-emptive Sync Debouncing & File System Events (Horizon 3)**: Group filesystem write events on paths and delay sync queue insertion until a quiet period has elapsed (e.g. 5 seconds of inactivity) to prevent thrashing the Proton API.
--- ```bash
git clone https://github.com/narrrl/proton-drive-linux.git
cd proton-drive-linux
cargo build --workspace
```
## 2. Performance, Correctness & Robustness Items The GUI builds against libadwaita 1.5, the version in Ubuntu 24.04, and uses newer widgets at run
Consolidated from the performance and media-streaming audit notes. time when the installed library has them (`crates/pdfs-gui/src/compat.rs`).
* **B6. Debounce the LRU access touch**: Keep last-touch times in memory and flush them in batches (debounce window of 30-60 seconds) to avoid performing a SQLite `UPDATE` to the `cache_entries` table on every cache hit. ## Repository layout
* **E12. Client in-flight cap shape**: Address the lock discrepancy between `MAX_CONCURRENT_BLOCK_DOWNLOADS = 10` (per-file) and `DEFAULT_MAX_INFLIGHT_BLOCKS = 12` (global client-wide). Either raise the global cap, lower the per-file window, or dynamically calculate the window to prevent single files from starving concurrent transfers.
* **Pipeline `enumerate_nodes_detail` (P2.8 / 8)**: Introduce pipelined node enumeration to speed up directory listings.
* **Local ancestor-chain walk (P3.4 / 4)**: Optimize recursive path lookups where full batching is impossible.
* **Name Search for Photos/Videos**: Add search by photo/video name (deferred from Phase 3 gallery implementation).
* **Sequential-Read Prefetch for Media Streaming**: Implement sequential read prefetch for media files to avoid stalls when buffering.
* **Minor sweep items (Phase F)**:
* **F1. `StreamRing.tags` Leak**: Clean up tag entries in `StreamRing.tags` when blocks are evicted. Currently, it only shrinks on `drop_node` (revision mismatch), leading to tag accumulation.
* **F2. Non-saturating bytes subtraction**: Use `saturating_sub` in `lib.rs:382` (`self.bytes -= dropped.len()`) to prevent release mode panics/wrapping if accounting drifts.
* **F3. Short block yields a short read / EOF**: Address the kernel EOF interpretation on short block reads. Either pad the block or fail loudly rather than silently serving short data.
* **F4. `stream_readahead` check-then-spawn race**: Prevent duplicate concurrent readahead tasks for the same block. Check-then-spawn is currently racey.
* **F5. `refresh_blocks` permit lock (SDK)**: Ensure URL refreshing (`refresh_blocks`) does not hold an in-flight block permit, preventing expired URLs from pinning client permits.
* **F7. Document temporary cache file safety**: Add explanatory comment to `cache.rs:362` explaining why `with_extension("tmp")` is safe there but not in `store_thumbnail`.
--- | Path | Contents |
|---|---|
| `crates/pdfs-core` | Shared library: authentication and keyring (`auth`), configuration (`config`), SQLite schema and queries (`db/`), content cache (`cache`), control protocol (`control`), search scoring, sync ignore rules, Google Takeout parsing, machine profile |
| `crates/pdfs-fuse` | The daemon: FUSE handlers (`filesystem`), the tree (`state`), reads (`reads`), the queue and the drain (`queue`, `drain`, `upload`), link state (`link`), the sync engine (`sync`, `sync/`), moves between locations (`relocate`), remote events (`background`), push events (`events`), control handlers (`control`), photos, devices, the supervisor, and the simulation tests (`sim/`) |
| `crates/pdfs-cli` | The `pdfs` binary |
| `crates/pdfs-gui` | `pdfs-app`, `pdfs-tray` and `pdfs-prompt`. Pages in `src/app/pages/`, shared widgets in `src/app/widgets/`, CSS and icons in `resources/` |
| `po/` | Translation template, catalogs and the scripts that maintain them |
| `packaging/` | systemd unit, desktop files, icon, PKGBUILD, RPM spec, AUR recipes |
| `scripts/` | FUSE acceptance suite, map-data generators, the wiki publisher |
| `docs/` | This documentation |
## 3. Open & Unverified Bugs The GUI talks to the daemon only through the control socket. Anything a front end needs that the
Derived from: [`BUGS.md`](BUGS.md) daemon does not offer yet is a new request in `pdfs_core::control`, not a database read.
* **B2. Unattributed Trash Origin**: Investigate why deleted/moved files are occasionally sent to trash instead of vanishing on older rename operations. ## Running from source
* **B10. GLib Critical Warning on HUD Close**: Assertion failure `g_list_store_remove: assertion '!g_sequence_iter_is_end (it)' failed` when prompt window closes during active launch (`xdg-open`). Needs investigation under `G_DEBUG=fatal-criticals`.
* **B15. Empty-but-listed folders and duplicate UIDs**: Investigate folder-identity issues (e.g. duplicate remote folders like `Music` with different UIDs) and how folders obtain `listed = 1` in the database without matching child rows.
* **B8 (CAPTCHA sign-in bridge)**: Verify a real gated sign-in using Webkit CAPTCHA completion.
* **Draft Revision Upload loop**: Verify SDK-side deletion and retry logic when a conflicting draft revision exists under the same client UID.
* **Optimistic Size Loss on Restart**: Verify that pending uploads retain their optimistic sizes across a daemon restart.
* **B48 — stale listing after unlink**: Session tombstones and authoritative empty listings are implemented; run focused live verification against eventually consistent remote enumeration.
* **B49–B55 — data-safety fixes**: Conflict preservation, orphan/staging durability, transactional supersession, one-entry wipe protection, and incomplete-scan handling have regression coverage but still require the fault, power-loss, and live mode-matrix cases recorded in `BUGS.md`.
* **B56–B60 — fail-closed state and IPC fixes**: Future schemas, event cursor advancement, socket limits, private-directory validation, and atomic config saves are implemented. Malformed-schema, database-full, connection-flood, wrong-owner, and concurrent/faulted-save verification remains.
* **B61 — credential rotation persistence**: Surface and retry keyring persistence failures so a rotated single-use refresh token cannot be lost on restart.
* **B62–B65 — release assurance**: Clean-install package verification, complete version/protocol identity, recovery sign-off, advisory/license policy, SBOM, provenance, checksums, and signatures remain.
* **B66–B67 — lifecycle and recovery**: Add queue flush/shutdown protection and explicit device adoption during fresh-state restore.
* **B68/B78 — profile backup**: the destination is fixed and verified live — `profile.json` now lives in `.proton-drive-linux/` under the device root, since the API refuses file creation at the root itself. What remains from B68 is making backup *health* visible (today a failure is a `WARN` in the log and nothing in the UI) and re-running the replacement-machine restore end to end.
* **B79 — cross-mount permission classification**: fixed and verified live; the standing guard is the `regression B79` acceptance case. The second-account cases it neighbours (`regression B34`) still cannot run — see below.
--- The packaged service and a daemon run from source cannot share the same state directory. Stop the
service before you run your own build:
## 4. Testing, Verification & Disaster Recovery Drills ```bash
Derived from: [`TESTING.md`](TESTING.md) and [`RECOVERY.md`](RECOVERY.md) systemctl --user stop proton-drive.service
RUST_LOG=pdfs_fuse=debug,info cargo run -p pdfs-cli -- daemon
```
* **IPC Memory and Performance Stress Tests (Phase 3)**: Verify memory allocations do not balloon when sending large `UploadPhoto` JSON payloads. Test control socket timeout behaviors under long-running block operations. In another terminal:
* **Fault Injection and Resiliency Tests (Phase 4)**:
* **Network degradation**: Verify that interrupted uploads leave readable staging buffers and result in clean backoff retries.
* **Disk full (ENOSPC)**: Verify that write staging and gap-filling report correct errors to FUSE and preserve staged work.
* **Concurrent reader starvation**: Test that parallel sequential reads do not freeze directory lookups or metadata handlers.
* **Disaster Recovery Drill (P5 / Phase 5)**: Execute a complete manual disaster recovery test:
1. Register a device.
2. Sync a folder and upload data.
3. Wipe the local state directory and SQLite cache.
4. Restart the daemon.
5. Run `pdfs sync restore` and assert the recovered filesystem matches the source byte-for-byte.
* **Second-account sharing run (mount-architecture.md P7)**: The only outstanding verification for the mount/sharing work. One account covers `context_share_id`, the shared-with-me listing, sharing from device folders and the Locations page — all confirmed. A **second account** is required for the shared-tree rendering under a foreign volume, `accept_invitation`, `update_member_role`, the `regression B34` viewer cases (which skip cleanly today), and above all an **editor-role write into a foreign share**, the only thing that validates `membership_address_for`.
* **Managed live mode matrix**: Before release, run `scripts/fuse-acceptance.sh --managed-live <empty-a> <empty-b>` with `PDFS_ACCEPTANCE_ONLY` unset. It validates on-demand/on-demand, on-demand/mirror, mirror/on-demand, and mirror/mirror transitions.
* **Crash-consistency matrix**: Inject failure after scratch/sidecar sync, each staging rename, directory sync, pending-op insert/commit, and source deletion; prove that a complete discoverable copy and either the old or new queue record always survives.
--- ```bash
cargo run -p pdfs-cli -- status
po/build.sh target/locale # compile translations once
PDFS_LOCALEDIR=target/locale cargo run -p pdfs-gui --bin pdfs-app
cargo run -p pdfs-gui --bin pdfs-tray
cargo run -p pdfs-gui --bin pdfs-prompt -- --fzf
```
## 5. Fixed & Resolved Bugs To keep the systemd unit but point it at your build, see
Derived from: [`BUGS.md`](BUGS.md) [INSTALL.md](INSTALL.md#using-a-binary-outside-usrbin).
### Completely Fixed & Verified > [!CAUTION]
* **B1 — FUSE rename loses the file (data loss)**: In `rename`, the node's database row was deleted while only updating the in-memory state, causing the file to vanish on next listing sync. Fixed by correctly calling `st.invalidate_listing(newparent)` to refresh SQLite cache states. > Develop against a test account. The acceptance suite and a buggy daemon can delete or rewrite
* **B4 — `invalidate_listing` silently skipped non-resident folders**: Folders not loaded in memory were being skipped during listing invalidation, leaving stale cache flags in the DB. Fixed by removing the residency check guard. > files in Proton Drive.
* **B5 — `ls -l` costs a network round trip per file (thumbnail xattr probes)**: File listings triggered blocking network checks for thumbnails on unsupported files. Fixed by caching negative thumbnail results and restricting xattr advertisement to supported image/media files.
* **B6 — Daemon sets no file modes: `control.sock` is an unguarded authority**: The Unix control socket was created without restricting file permissions, exposing daemon controls to other local users. Fixed by applying proper creation permission masks.
* **B7 — renamed directory reads as missing until the entry TTL expires**: Kernel cache maps remained stale after a FUSE directory re-anchor. Fixed by proactively sending entry invalidation notifications to the kernel.
* **B9 — Enter did not open the selected result in the launcher**: Hooked up keyboard execution (Enter key) in `pdfs-prompt` search HUD to launch the selected item using `xdg-open`.
* **B11 — a file moved while its create is still queued reads as empty**: Reconciling a local rename on a file that had not finished uploading yet uploaded empty blocks. Fixed by carrying over and staging the correct logical file size.
* **B12 — cold enumeration is slow per entry, and goes superlinear past ~500**: S2K decryption and thumbnail resolution loops were serial. Fixed by parallelizing key derivations and caching metadata.
* **B13 — `rename` over an existing destination fails instead of replacing it**: Fixed POSIX rename target overwrite behavior to replace existing destinations.
* **B14 — provisional (ciphertext) sizes make rsync read short and abort the file**: Adjusted size reporting to align logical and ciphertext sizing, preventing premature EOF.
* **B45 — truncate over a queued rewrite**: Path-based and handle-based truncate now compose over the complete pending blob; shrink, growth, and sparse I/O are live-verified.
* **B46 — combined cross-directory move and rename**: The desired end state is durably queued and reconciles partially completed remote operations; replacement and combined cases are live-verified.
* **B47 — overlong/invalid names**: All name-taking callbacks share a validator for `NAME_MAX`, reserved components, and UTF-8; managed-live verification passed.
### Implemented in 1.0 (Further Fault Verification Pending) > [!TIP]
> A daemon stopped in a debugger or deadlocked freezes every process that touches the mount,
> including your shell if its prompt reads the current directory. Keep your shell outside
> `~/ProtonDrive`, prefix commands that touch it with `timeout`, and unmount a dead mount with
> `fusermount3 -uz ~/ProtonDrive`.
* **Unified search and resident prompt**: `SearchV2` combines Drive and local lookup with shared relevance scoring and content/source filters. The prompt reuses its window and opens streamable media through FUSE. ## Quality gates
* **Durable write publication**: Scratch sidecars and staged blobs use synced temporary files, atomic renames, and directory synchronization; orphan rescue retains its source on failure.
* **Transactional pending operations**: Superseding a queued operation cannot delete the old durable record unless the replacement commits.
* **Safer mirror reconciliation**: Conflict-copy failure aborts publication, incomplete scans are non-destructive, and total-wipe protection includes one-entry baselines.
* **Bounded IPC**: Control requests are limited to 1 MiB and 10 seconds, with at most 64 active handlers.
* **Private local state**: Sensitive directories require correct ownership, real-directory identity, and `0700`; sockets use `0600`; config saves are atomic and malformed input is preserved.
* **Release gates**: Tag/workspace/package versions are checked and formatting, lint, tests, and offline FUSE acceptance run before packaging.
### Code Fixed / Implemented (Pending Verification) CI (`.github/workflows/ci.yml`) runs these on every push to `main` and every pull request. Run them before you
* **B8 — no way to complete a CAPTCHA, so a gated sign-in is unrecoverable**: Implemented a Webkit-based bridge to prompt the user to resolve interactive CAPTCHAs during gated logins. (Needs verification on a real gate request). push:
### Closed / Not a Bug ```bash
* **B3 — Activity log timestamps written in seconds, read as milliseconds**: Closed as investigator error (timestamps are consistent). cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only
```
--- For changes to the FUSE layer, the queue or the sync engine, also run the
[simulation runs](TESTING.md#simulation-runs), which CI runs only after a push to `main`, and the
[acceptance suite](TESTING.md#acceptance-suite) on a test account.
## 6. Recently Implemented Features For changes to GUI strings, check that every catalog is complete:
### File Version History (Implemented) ```bash
**Files**: [`revisions.rs`](../crates/pdfs-fuse/src/revisions.rs), [`control.rs`](../crates/pdfs-core/src/control.rs), [`versions_dialog.rs`](../crates/pdfs-gui/src/app/widgets/versions_dialog.rs) for l in $(cat po/LINGUAS); do msgfmt --check --statistics -o /dev/null po/$l.po; done
```
Proton Drive keeps every revision a client committed; the daemon only ever addressed the active one, so a file overwritten by a sync pass could be recovered only from whatever the local `recovery/` directory happened to hold. ## Debugging
The control protocol gained `ListRevisions` / `RestoreRevision` / `DeleteRevision` / `SaveRevisionAs` (each with a `…ByUid` twin for nodes the primary mount cannot name), the CLI gained `pdfs versions list|restore|save|rm`, and the browser's details pane gained a **Versions** button opening a per-file dialog. | Tool | Use |
|---|---|
| `RUST_LOG` | Log filter for every binary, for example `RUST_LOG=pdfs_fuse::sync=trace,info` |
| `pdfs diagnose` | Static checks: directories, FUSE, keyring, socket, database integrity. Works without the daemon |
| `pdfs diagnostics` | Live state of a running daemon: each worker's current job and its age, queue depth, control requests in flight, memory |
| `pdfs cache inspect --deep` | Database size, row counts and SQLite `integrity_check` |
| `pdfs --json …` | Raw protocol payloads for query commands |
| `GTK_DEBUG=interactive` | The GTK inspector for `pdfs-app` |
| `PDFS_STAND_IN_WIDGETS=1` | Force the libadwaita 1.5 fallback widgets on a newer system |
| `G_DEBUG=fatal-criticals` | Turn GLib criticals into crashes to get a backtrace |
Three properties that shape the code: The databases are ordinary SQLite files. Open a copy, not the live file:
- **A restore is server-side and asynchronous.** No content crosses the wire and nothing enters the drain queue; the server answers 202 and swaps the active revision in the background, so the daemon evicts the file's cached blocks and open readers rather than describing the new state, and the UI wording promises the request, not the result. ```bash
- **The active revision cannot be deleted.** `Core::delete_revision_for_uid` checks that locally so the user gets a sentence rather than an API code, and the dialog gives the current row no delete button. cp ~/.local/state/proton-drive-linux/cache.db /tmp/cache.db
- **Saving a version never overwrites.** `SaveRevisionAs` refuses an existing destination and removes a partial file if the download fails — a half-written export looks identical to a good one to every tool that opens it afterwards. sqlite3 /tmp/cache.db '.tables'
```
### Photo Favorites (Implemented) ## Common changes
**Files**: [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs)
The gallery can mark and filter favorites, through the SDK's photo-tag API (`update_photos`). Schema **v20** adds `photos.favorite`; the flag is learned in the timeline enrichment pass that already resolves each photo's name and media type, and is kept across a refresh that could not resolve a photo — the same learned-and-kept rule as `media_type`. The lightbox carries a star toggle, the Photos header a favorites filter, and the CLI has `pdfs favorite <uid> [--remove]` plus `pdfs photos --favorites`. **A new control request.** Add the variant to `Request` (and a `Response` if it returns data) in
`crates/pdfs-core/src/control.rs`, handle it in `crates/pdfs-fuse/src/control.rs`, and list the
`Topic`s it changes in the same file so subscribed front ends refresh. A daemon from the previous
release may still be running when a new front end starts, so treat an error reply to a new
request as "not supported".
Favouriting a photo that is not on this account's own photos volume (shared with us, or album-only) needs it re-encrypted for our timeline root, which the SDK does not implement; the daemon surfaces that as an error rather than silently doing nothing. **A schema change.** Migrations are forward-only. Add a new step to
`crates/pdfs-core/src/db/migrations.rs` and bump `SCHEMA_VERSION`; never edit a shipped step. Add
a test in `crates/pdfs-core/src/db/tests.rs` that migrates from the previous version.
### Justified Gallery, Gallery Delete (Implemented) **A GUI string.** Wrap it in the helpers from `crates/pdfs-gui/src/i18n.rs`, add any new source
**Files**: [`photos.rs`](../crates/pdfs-gui/src/app/pages/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs), [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`background.rs`](../crates/pdfs-fuse/src/background.rs) file to `po/POTFILES.in`, run `po/update.sh`, and translate the new entries in every catalog. See
[TRANSLATING.md](TRANSLATING.md). The CLI and daemon messages stay in English.
The gallery lays each day out in justified rows (`justify_rows` / `plan_rows` / `fit_row`): photos are taken in capture order until their summed aspect ratio no longer fits the target row height, and the row is then scaled so it ends exactly on the content width. A tile is therefore its own photo's shape, and `ContentFit::Contain` has nothing left to crop. The last row of a day is left at the target height rather than stretched, because a day holding two photos is a short day, not a layout fault. **A setting.** Add the field to `AppConfig` in `crates/pdfs-core/src/config.rs` with
`#[serde(default)]`, so older files still load, and document it in
[CONFIGURATION.md](CONFIGURATION.md).
- **Ratios that are not known yet.** `PhotoItem::ratio` is persisted, but a photo that has never been decoded has none. Those are laid out square and remembered in `assumed_ratios`; when a decode proves the real shape the day re-flows on the existing debounce timer, so a screenful of decodes costs one re-flow rather than one per photo. Ratios are clamped to 0.4–3.0 so a single panorama cannot decide what a row looks like. **A bug fix.** Add or update the entry in [BUGS.md](BUGS.md#adding-an-entry), cover the fix with a
- **Thumbnails survive recycling.** A reply for a tile that has already been re-bound still lands in the texture cache, which is an LRU of 1500 textures keyed by uid, and the rows just outside the realised range are prefetched in the scroll direction. unit test, a simulation scenario or a `regression B<n>` acceptance case, and cite the entry in the
- **Delete is uid-addressed.** The photos volume is not in the FUSE mount, so the path-based `Request::Delete` cannot reach it. `Request::TrashNodes { uids }` trashes through the SDK and answers `Response::Trashed { trashed, failed }`; the daemon then drops those rows with `Db::photos_delete` (album membership included) so the gallery does not wait for a timeline refresh. The GUI removes the tiles optimistically with an Undo toast and puts back anything the server refused. code as `docs/BUGS.md B<n>`. Do not cite planning notes that are not in the repository.
- **Remote deletions arrive on their own.** The Drive volume and the photos volume have separate event streams, so `run_photos_event_sync` polls the photos volume with its own cursor (`photos_event_cursor` in the state table). Trash and delete events remove rows; anything else only invalidates freshness, because there is no inode space to converge on this volume. Together with a 60 s `TIMELINE_TTL` and a `RefreshScope::Photos` that now awaits the refresh, a photo deleted on a phone leaves the grid in about ten seconds.
### RAW + JPEG Grouping (Implemented) **A user-visible change.** Update the matching page (user guide, CLI reference, configuration)
**Files**: [`photos.rs`](../crates/pdfs-core/src/db/photos.rs), [`migrations.rs`](../crates/pdfs-core/src/db/migrations.rs), [`photos.rs`](../crates/pdfs-fuse/src/photos.rs), [`photo_viewer.rs`](../crates/pdfs-gui/src/app/pages/photo_viewer.rs) and the `[Unreleased]` section of [CHANGELOG.md](CHANGELOG.md). `scripts/publish-wiki.sh` copies
the six user pages (`INSTALL`, `USER_GUIDE`, `CLI`, `CONFIGURATION`, `TROUBLESHOOTING`, `RECOVERY`)
to the GitHub wiki, so keep their file names.
Schema **v30** adds `content_hash`, `main_uid` and an indexed `group_key` to `photos`. The first two come from the server's `PhotoProperties` and are filled by the enrichment pass in `refresh_timeline` that already resolves each photo's name, media type and favorite tag; `group_key` is computed in `photos_replace`, where the whole timeline is in hand. ## Releasing
- **Precedence.** The server relation first (`main_photo_uid` and `related_photo_uids`, read from whichever end resolves — the two can land in different enrichment chunks); then same capture day, same case-insensitive name stem, and one member raw while the other is not; otherwise the photo is its own group. The union-find pass is over the timeline in memory, so it costs one pass per refresh rather than a query per photo. Releases are cut from `main` by pushing a `v*` tag. `.github/workflows/release.yml` then:
- **Representative.** The non-raw member when there is one — a JPEG decodes in milliseconds and is what the person expects to see. `group_key` holds that member's uid, so "is this the tile" is `uid = group_key`, an index scan.
- **What is grouped and what is not.** `photos_page`, `photos_months` and `photos_counts` count groups, except on the Raw tab, which lists files. Album pages are never grouped: an album is a list someone made. The relation is learned-and-kept like `media_type`, so a refresh that could not resolve a node does not break a group up.
- **Deleting.** `Core::trash_photos` expands each uid to its group before trashing, because the grid shows the shot as one tile — leaving the RAW behind would put the photo back on the Raw tab and nowhere else. The GUI's confirmation names the file count.
- **The other files.** `Request::PhotoGroup { uid }` answers with the group's members as ordinary `PhotoItem`s, which is what the lightbox's switch steps through.
### Persistent SDK Entity Cache (Implemented) 1. checks that the tag, the workspace version in `Cargo.toml` and `pkgver` in
**Files**: [`sdkcache.rs`](../crates/pdfs-core/src/sdkcache.rs), [`auth.rs`](../crates/pdfs-core/src/auth.rs), [`background.rs`](../crates/pdfs-fuse/src/background.rs) `packaging/PKGBUILD` agree;
2. runs the quality gates;
3. builds the release binaries and packages a `.tar.gz`, a `.deb` and a GPG-signed `.rpm`;
4. publishes them as a GitHub release;
5. calls `.github/workflows/aur.yml`, which updates and pushes the AUR recipes and commits them
back to `main`.
The SDK's Drive entity cache (decrypted node metadata: name, size, parent, signing share) defaulted to memory, so every daemon restart re-fetched and re-decrypted the tree the previous run had already walked. It is now backed by SQLite. To prepare a release:
- **Its own file** (`sdk_cache.db` in the state directory), not `cache.db`: the daemon's `Db` is one `Mutex<Connection>` shared by every FUSE thread and the control socket, and this traffic is frequent, small and entirely reconstructible. A separate file also means no schema migration — the store can be deleted at any time. ```bash
- **Encrypted at rest** by wrapping it in the SDK's `EncryptedCacheRepository`, keyed by the mailbox password. A password change reads as a cold cache (the SDK treats an undecryptable entry as a miss and clears the store), not as an error. # 1. Bump the version in Cargo.toml and packaging/PKGBUILD, then refresh the lock file.
- **Staleness** is closed by the event loop, which already replays from a persisted cursor and calls `invalidate_caches_for_event` for every event — including those raised while the daemon was down. The one case with no trail is a *seeded* cursor (first-ever mount, or a lost cursor), where the seed path now clears the store instead of trusting it. `auth::logout` removes the file. cargo update --workspace
# 2. Move the [Unreleased] notes in docs/CHANGELOG.md under the new version.
# 3. Commit, tag and push.
git commit -am "chore(release): X.Y.Z"
git tag vX.Y.Z
git push origin main vX.Y.Z
```
Needed a small additive SDK change (`ProtonDriveClient::with_entity_repository`, 0.5.1): `with_entity_cache` is a constructor and so could not be combined with `with_key_salts`, which the daemon requires. Before a release, work through [Before a release](TESTING.md#before-a-release). Open
release-assurance items are listed in [ROADMAP.md](ROADMAP.md#release-assurance).
### Per-node Batch Outcomes (Implemented)
**File**: [`batch.rs`](../crates/pdfs-core/src/batch.rs)
SDK 0.5.0 made `trash_nodes` / `restore_nodes` / `delete_nodes` report one outcome per node instead of failing the whole call (mirroring upstream's streamed `NodeActionResult`). `pdfs_core::batch::into_unit` collapses the single-node calls back to a `Result`; `batch::split` handles a collected batch.
A restore is expanded before it is sent (`expand_restore`, `pdfs-fuse/src/lib.rs`): the persisted trash listing carries each row's `parent_uid` (schema 29), so a restore takes the connected piece of the trashed tree — every trashed descendant of what was asked for, and every trashed ancestor above it — and sends it shallowest wave first, because the server cannot put a node back under a parent that is still trashed. A uid the listing does not know about (it is materialised in chunks, and can be stale) is restored on its own rather than dropped.
The trash view's restore and permanent-delete use the SDK's **streaming** variants (`restore_nodes_streaming` / `delete_nodes_streaming`) and apply local state per node as each batch lands rather than after the last one. For a permanent delete — which is irreversible — that means a daemon interrupted mid-batch has forgotten exactly the nodes the server destroyed, no more and no fewer. Both report per-node failures and count only what actually succeeded; only a batch where nothing succeeded is an error.
### Open-for-Write Deferral for Mirror Sync (Implemented)
**File**: [`sync.rs`](../crates/pdfs-fuse/src/sync.rs)
The mirror folder sync engine now defers uploading any file that is currently held open for writing by another process, matching the guarantee the FUSE mount path already provides (where uploads are deferred until `close(fd)`).
**Problem**: Previously, the mirror sync path relied solely on a 2-second trailing-edge debounce (with a 30-second ceiling) to avoid uploading files mid-write. This was insufficient for slow continuous writers (e.g., database dumps, large exports, or editors that keep files open for extended periods), which could have their incomplete state uploaded as a real revision.
**Solution**: Before each reconcile pass, the sync engine scans `/proc/*/fd` once to build a set of canonical paths within the sync root that any process holds open for writing (`O_WRONLY` or `O_RDWR`). Files in this set are:
- **Kept in the local walk** — so they are not misclassified as deletions
- **Treated as unchanged** — so no upload is queued for them
- **Counted as `deferred`** — so the activity summary reports them (e.g., "3 uploaded, 1 deferred (open for write)")
- **Picked up on the next pass** — after the writer has closed the file
**How it works**:
1. `open_for_write_set(root)` reads `/proc/*/fd` → `readlink` for each fd → prefix-checks against the sync root → reads `/proc/<pid>/fdinfo/<n>` to check the `flags:` line's low two bits (O_ACCMODE)
2. `walk_local` sets `LocalItem.open_for_write = true` for matching files
3. Both `do_reconcile` and `push_pass` skip upload classification for flagged files
4. `Outcome.deferred` tracks the count for the activity summary
**Performance**: The `/proc` scan completes in low single-digit milliseconds on a typical desktop. Only fds whose `readlink` target falls under the sync root incur the `fdinfo` read.
**Tests added**: 8 unit tests covering `is_write_mode` parsing of various fdinfo flag combinations, and `Outcome` summary formatting with deferred counts.
| Sync Path | Open-for-write protection | Mechanism |
|---|---|---|
| FUSE mount | ✅ Perfect | Upload deferred until `close(fd)` |
| Mirror folders | ✅ Perfect | `/proc/*/fd` scan per reconcile pass |
+251
View File
@@ -0,0 +1,251 @@
# Installation
This page covers installing Proton Drive for Linux from a package or from source, the first
sign-in, running without a desktop, and removing the client again.
- [Requirements](#requirements)
- [Install a package](#install-a-package)
- [Build from source](#build-from-source)
- [First run](#first-run)
- [Headless and server use](#headless-and-server-use)
- [Updating](#updating)
- [Uninstalling](#uninstalling)
## Requirements
| | Minimum |
|---|---|
| Architecture | x86_64 (release packages); other architectures build from source |
| Kernel | FUSE 3 (`fuse3` package, `/dev/fuse` available to your user) |
| GTK / libadwaita | GTK 4.12, libadwaita 1.5 (1.8 or newer gets the newest widgets) |
| WebKitGTK | 6.0 — used only for the CAPTCHA page during sign-in |
| Secret Service | GNOME Keyring, KWallet, KeePassXC, or another provider of `org.freedesktop.secrets` |
| Rust (source builds) | 1.96 |
The session is stored only in the Secret Service. Without a running provider, sign-in succeeds but
cannot be saved, and the service cannot start unattended.
Optional runtime extras:
| Package | Adds |
|---|---|
| `exiftool` (`perl-image-exiftool`, `libimage-exiftool-perl`) | Thumbnails for camera RAW files |
| `ffmpeg` | Video thumbnails |
| `iso-codes` | Country names in your language on the Places map |
| `xdg-utils` | "Open Folder" in the tray, and the default file opener |
| `fzf` plus a terminal (foot, kitty, alacritty, …) | `pdfs-prompt --fzf`, the search-as-you-type launcher |
| AppIndicator extension (GNOME only) | The tray icon. KDE Plasma and most other desktops show it natively |
## Install a package
Every [release](https://github.com/narrrl/proton-drive-linux/releases/latest) ships a `.deb`, a
signed `.rpm` and a `.tar.gz` of the raw binaries. All packages install the same files:
| Path | Purpose |
|---|---|
| `/usr/bin/pdfs` | Command-line client and background service |
| `/usr/bin/pdfs-app` | Desktop app |
| `/usr/bin/pdfs-tray` | Tray icon |
| `/usr/bin/pdfs-prompt` | Search launcher |
| `/usr/lib/systemd/user/proton-drive.service` | User service that keeps the drive mounted |
| `/etc/xdg/autostart/io.narl.proton-drive-linux-tray.desktop` | Starts the tray at login |
| `/usr/share/locale/*/LC_MESSAGES/pdfs.mo` | Translations |
### Debian and Ubuntu (24.04 or newer)
```bash
sudo apt install ./proton-drive-linux_*_amd64.deb
```
### Fedora (44 or newer)
```bash
sudo dnf install ./proton-drive-linux-*.x86_64.rpm
```
For a Fedora package built from your own checkout, see
[Build an RPM locally](#build-an-rpm-locally).
### Arch Linux (AUR)
Three packages cover the usual choices. They conflict with each other, so install one:
| Package | Contents |
|---|---|
| [`proton-drive-for-linux`](https://aur.archlinux.org/packages/proton-drive-for-linux) | Latest tagged release, built from source |
| [`proton-drive-for-linux-bin`](https://aur.archlinux.org/packages/proton-drive-for-linux-bin) | Latest tagged release, prebuilt binaries |
| [`proton-drive-for-linux-git`](https://aur.archlinux.org/packages/proton-drive-for-linux-git) | `main`, built from git |
```bash
paru -S proton-drive-for-linux # or yay, or any other AUR helper
```
The recipes live in [`packaging/aur/`](../packaging/aur/README.md).
### Other distributions
Unpack the `.tar.gz` into a directory on your `PATH`, then install the service unit as shown in
[Using a binary outside `/usr/bin`](#using-a-binary-outside-usrbin).
## Build from source
### Build dependencies
**Debian / Ubuntu 24.04+**
```bash
sudo apt-get install -y pkg-config gettext libfuse3-dev libgtk-4-dev libadwaita-1-dev \
libwebkitgtk-6.0-dev libsecret-1-dev libdbus-1-dev
```
**Fedora 44+**
```bash
sudo dnf install -y cargo rust gettext pkgconf-pkg-config fuse3-devel gtk4-devel \
libadwaita-devel webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel
```
**Arch Linux**
```bash
sudo pacman -S --needed rust gettext pkgconf fuse3 gtk4 libadwaita webkitgtk-6.0 libsecret dbus
```
Install Rust 1.96 or newer with [rustup](https://rustup.rs) if your distribution ships an older
toolchain.
### Compile
```bash
git clone https://github.com/narrrl/proton-drive-linux.git
cd proton-drive-linux
cargo build --release --locked
po/build.sh target/locale # optional: compile the translations
```
The four binaries are in `target/release/`: `pdfs`, `pdfs-app`, `pdfs-tray` and `pdfs-prompt`.
### Build a package from the working tree
**Arch Linux:**
```bash
cd packaging && makepkg -fi
```
<a id="build-an-rpm-locally"></a>**Fedora:**
```bash
sudo dnf install -y rpm-build
rpmbuild -bb packaging/proton-drive-linux.spec \
--define "git_dir $PWD" \
--define "_rpmdir $PWD/packaging/out" \
--define "_builddir $PWD/packaging/build" \
--define "_sourcedir $PWD" \
--define "_specdir $PWD/packaging" \
--define "_srcrpmdir $PWD/packaging/out"
sudo dnf install packaging/out/x86_64/proton-drive-linux-*.rpm
```
### Using a binary outside `/usr/bin`
The service unit starts `/usr/bin/pdfs daemon`. If you installed the binaries somewhere else, for
example `~/.local/bin`, install the unit for your user and point it at your binary:
```bash
mkdir -p ~/.config/systemd/user
cp packaging/proton-drive.service ~/.config/systemd/user/
sed -i "s|^ExecStart=.*|ExecStart=$HOME/.local/bin/pdfs daemon|" \
~/.config/systemd/user/proton-drive.service
systemctl --user daemon-reload
```
To start the tray at login, copy `packaging/io.narl.proton-drive-linux-tray.desktop` to
`~/.config/autostart/`. For the app launcher entry, copy `packaging/io.narl.proton-drive-linux.desktop`
to `~/.local/share/applications/` and the icon `packaging/io.narl.proton-drive-linux.svg` to
`~/.local/share/icons/hicolor/scalable/apps/`.
## First run
### With the desktop app
1. Open **Proton Drive** from your application menu, or run `pdfs-app`.
2. Sign in with your Proton email and password, and your two-factor code if the account has one.
If Proton asks for a CAPTCHA (common on a VPN or a new network), the app shows the challenge in
a window and continues after you solve it.
3. The app enables and starts `proton-drive.service`. Your Drive appears at `~/ProtonDrive`.
The service is tied to your graphical session. It starts after you log in and stops when you log
out. Change the folder location under **Preferences → General → Location**.
### With the command line
`pdfs login` stores the session but does not enable the service, so a setup that runs the daemon
another way stays untouched. Enable the service yourself:
```bash
pdfs login # email, password, 2FA code
systemctl --user enable --now proton-drive.service
pdfs status # account, mount and queue state
```
The CLI cannot show a CAPTCHA. If sign-in reports one, sign in once with `pdfs-app` instead; the
session it stores is used by the CLI and the service as well.
## Headless and server use
The service unit is bound to `graphical-session.target`. On a machine without a desktop session,
run the daemon directly or from your own unit:
```bash
pdfs daemon # waits for a stored session, mounts, stays mounted
pdfs mount /srv/proton # mounts once at a given path, blocks until unmounted
```
Both still need a Secret Service provider. On a server, run
`gnome-keyring-daemon --unlock` or KeePassXC's Secret Service integration inside the user's D-Bus
session.
## Updating
Install the new package over the old one. The service picks up the new binary the next time it
starts; to switch at once:
```bash
systemctl --user restart proton-drive.service
```
Coming from 2.8.0 or older, re-enable the service once. Those releases hooked it into
`default.target`, which starts it before the desktop session is ready; on KDE that crashes
`ksecretd` at every login. Signing in again or toggling "Start on login" does this as well:
```bash
systemctl --user reenable proton-drive.service
```
Database migrations run automatically and only move forward. A database written by a newer
release refuses to open in an older one, so downgrading means clearing the local database (see
[Troubleshooting](TROUBLESHOOTING.md#start-over-with-a-clean-local-state)).
## Uninstalling
> [!WARNING]
> Check that nothing is waiting to upload before you remove local state. `pdfs sync queue` must be
> empty. Queued writes exist only on this computer until they reach Proton Drive.
```bash
pdfs sync queue # must list nothing
systemctl --user disable --now proton-drive.service
pdfs logout # removes the session from the keyring
```
Then remove the package (`sudo apt remove proton-drive-linux`, `sudo dnf remove
proton-drive-linux`, or `sudo pacman -R proton-drive-for-linux`), and, if you want no trace left,
the local data:
```bash
rm -rf ~/.config/proton-drive-linux ~/.local/state/proton-drive-linux ~/.cache/proton-drive-linux
```
Sign out of the device in your Proton account settings as well, and remove the computer from
**Computers** in the Proton Drive web app if you no longer want its backups.
Loaded 100 of 144 files, more files were not shown because too many files have changed in this diff. Show more