Compare commits

...
204 Commits
Author SHA1 Message Date
nvrl f717a692ff fix(packaging): COPR project belongs to narl-rs
The COPR project lives under the FAS account narl-rs, not narrrl.

Refs #42
2026-10-17 12:38:09 +02:00
nvrl e66308498b feat(gui,core): opt-in check for new releases
Preferences gets a switch that asks GitHub once a day for the latest
release and shows a banner over the sidebar when it is newer. Off by
default. Packages whose package manager brings updates install the
marker share/proton-drive-linux/managed-updates, which hides the switch.

Refs #42
2026-10-17 11:45:09 +02:00
nvrl 29b87ece72 feat(packaging): COPR repository for Fedora
The spec builds from the tag archive and a vendored crates tarball when
git_dir is not defined, so COPR can build it without network access.
release.yml publishes the vendor tarball and submits the build to COPR
after the release is published.

Refs #42
2026-10-17 11:44:33 +02:00
nvrl eb25c16fe6 Merge pull request #41 from narrrl/release/3.5.0
Release/3.5.0
2026-10-16 19:18:13 +02:00
nvrl 8f771fdcc4 fix(fuse): read the size again for a baseline row without a revision id
A mirror upload that Drive did not list yet when it was read back left a
baseline row with no revision id. The next walk reused the recorded size
whenever the modification time matched, and a row without an id never
counts as moved. So a revision another computer saved in the same second
looked unchanged and never arrived (B254). The B250 simulation hit this
about one run in five on CI.

A row without an id now reuses its size only while Drive gives no id
either. Otherwise the walk reads the size, and an unchanged file gets
Drive's revision id written into its row, so each row costs one extra
read. Rows from before schema 44 are read once after upgrading.
2026-10-16 18:44:51 +02:00
nvrl c1911ea1de docs: window size, keyring banner and the mailbox password per session
Add the window size and the narrow layout to the 3.5.0 changelog with
B253, document the `window` config field, and explain in the
troubleshooting guide why a two-password account asks for its mailbox
password after every reboot, and how to keep it unlocked.
2026-10-16 17:21:46 +02:00
nvrl 179b6555c4 fix(gui): keep a saved sign-in when the keyring can't be read
The app read the session with `auth::load().ok()`, so a locked keyring, a
Secret Service that was not up yet or any D-Bus error showed the sign-in
form as if nobody were signed in. The session was still saved (B253).

Keyring errors are now kept apart from "not signed in". The login page
shows a banner saying the saved sign-in could not be read, with a Retry
button. The app tries again after 2, 5, 10 and 30 seconds and opens on
Files once the keyring answers. The login page's header also loses the
back button the collapsed split gave it.

Translated into every shipped language.
2026-10-16 17:21:46 +02:00
nvrl e6971dd702 fix(gui): let the main window shrink to fit small screens
While signed in, the sidebar never collapsed. The window could not get
narrower than the sidebar plus the widest page, about 950 pixels, so on a
small or scaled screen the minimize, maximize and close buttons ended up
off-screen.

- The page stack is no longer homogeneous, so only the page on screen sets
  the minimum width.
- The selection bars of My files and Photos scroll sideways instead of
  holding the window at their full width, even while hidden.
- Below 760sp the sidebar and the page take turns, with a back button,
  and the My files search entry narrows. Picking a sidebar row opens the
  page.

Every page, in English and in German, now fits in a window 480 pixels
wide with its window buttons visible.
2026-10-16 17:21:46 +02:00
nvrl 770d4fda38 feat(core,gui): remember the main window's size
The app always opened at 980x680. It now keeps the unmaximized size and
whether the window was maximized in config.json under `window`, written
when the window closes and when the app quits with it still open. A
missing field, or a saved size below 480x400, opens at the old default.
2026-10-16 17:21:46 +02:00
nvrl 11d03091a7 docs: syncing several computers into one device; changelog for 3.5.0
Add B248 to B252 to the bug ledger, the shared-device feature and schema 44
to the 3.5.0 changelog, and describe the join flow, per-machine profiles,
conflict copy names and the mirror's multi-writer rules in the user guide,
recovery guide, CLI and configuration references and the architecture notes.
2026-10-16 16:18:14 +02:00
nvrl 7a0f1ebd6f feat(cli,gui): sync with another computer's device
Two or more computers can now sync the same folders through one device in
Proton Drive. `pdfs devices join UID` and "Sync With This Computer…" on
another computer's row adopt that device as this machine's and then offer its
folders for syncing here, like a restore. Unlike "Continue This Backup Here",
the other computer goes on syncing, and the confirmation says that changes
and deletions on one reach the other.

Translated into every shipped language.
2026-10-16 16:04:26 +02:00
nvrl 950056b48d feat(core,fuse): one profile per machine on a shared device
Every computer that synced into one device wrote the same profile.json, so
each save replaced the other computers' folders with its own (B252). Each
machine now writes profile-<machine id>.json, where the id is random, made on
first use and kept in machine-id next to config.json. No host identifier goes
to Drive.

Restore reads every profile in .proton-drive-linux, including the shared one
older clients wrote. A folder's proposed path and mode come from this
computer's own profile, otherwise from the newest one that has the folder.
The legacy file is left as it is, so a client before 3.5.0 still finds it.
2026-10-16 14:55:13 +02:00
nvrl 05b84bf796 fix(fuse,core): keep a mirrored folder safe when another computer syncs it
A mirrored folder assumed it was the only client writing to its folder on
Drive. With a second computer syncing the same folder, edits could be lost:

- The first sync took a same-size file at the same path for the same file.
  It now records the baseline only if the local file's SHA-1 matches Drive's
  digest, and otherwise keeps a conflict copy (B248).
- An upload replaced a revision another client had committed since the pass
  looked. The upload now checks Drive's active revision first, and an edit
  that read back with someone else's digest is left for the next pass to keep
  as a conflict copy (B249).
- A download recorded a revision committed while it ran as synced, so that
  edit never arrived. The baseline now records the revision the pass planned
  to download (B250).
- Conflict copies now carry the computer's hostname, so two computers do not
  make the same name on Drive (B251).

The baseline gains the revision id it was synced at (schema 44), and a new
revision counts as a remote change even at the same mtime and size.
2026-10-16 14:48:32 +02:00
nvrl 7cd516a164 chore(release): 3.5.0 2026-10-16 12:58:06 +02:00
nvrl c3c536892e docs(changelog): add B245 to B247 2026-10-16 12:31:51 +02:00
nvrl cc9db51b78 fix(gui): translate the tray's transfer status lines (B247) 2026-10-16 12:31:40 +02:00
nvrl 7384d8e97d fix(gui): read the new session off the main thread after sign-in (B246) 2026-10-16 12:31:40 +02:00
nvrl 4bf176a631 fix(packaging): accept watchdog pings from the daemon only (B245) 2026-10-16 12:17:53 +02:00
nvrl aa0c60f5a2 docs(changelog): list the second round of 3.5.0 fixes 2026-10-16 12:14:17 +02:00
nvrl 47934b7ddb fix(core,fuse,gui): date photos by their EXIF when nothing else does
A Takeout photo with no sidecar and no date in its name was filed under
the day of the import, and the app dated uploads by modification time.
A new `exiftime` module reads DateTimeOriginal (then DateTimeDigitized)
with its offset. Takeout uses it for undated photos it copies out for a
thumbnail, and the daemon uses it for app uploads before falling back to
the modification time (B244).
2026-10-16 12:12:41 +02:00
nvrl 48c8910e7b fix(gui,core): ask the mount off the main thread when opening, reap children
Opening a file or a location stat'ed the mount on the GTK main thread, so
a hung mount hung the app. The browser now uses the launcher's bounded
worker check (`present_within`, 2 s), and opening a location runs on a
thread. Every child the GUI starts is also waited for on a thread, so it
no longer stays a zombie (B243).
2026-10-16 12:07:48 +02:00
nvrl 6ec8043934 fix(gui): decode viewer photos and read their EXIF off the main thread
The viewer decoded each full-size photo, read its EXIF and stat'ed it on
the GTK main thread, freezing the app for a large photo. That now runs in
gio::spawn_blocking, and a result for a photo the user has left is
dropped (B242).
2026-10-16 12:04:05 +02:00
nvrl 99707c48fa fix(gui): free the photo viewer and the share dialog when they close
The viewer's handlers held strong references to the viewer and its
window, which own them, so every closed viewer stayed in memory with its
full-size photo. The share dialog's row handlers had the same cycle with
the dialog state. The handlers now hold weak references, and the viewer's
one strong handle is dropped on close with the photo (B241).
2026-10-16 12:03:02 +02:00
nvrl 5557aa66d5 ci: run the simulations on pull requests and lint with --locked 2026-10-16 11:58:56 +02:00
nvrl 6456f503f1 fix(cli,core): keep secret-holding commands out of core dumps
Only `pdfs daemon` marked itself non-dumpable, though `mount`, `login`,
`unlock` and `logout` hold the same passphrases or tokens. All five now
do. The file-manager script also passes each path after `--`, so a file
named like an option stays a path (B240).
2026-10-16 11:58:28 +02:00
nvrl 4836c5a84a fix(cli,fuse,gui): make --wait fail when an import or re-date did
`import-google-photos --wait` and `redate-photos --wait` exited 0 however
the run ended, and a run that failed as a whole left no summary, so the
CLI said no import had run and the app said it finished. Failed runs now
leave a summary with the reason, `--wait` exits 1 on failure, cancel or
failed photos, and the app shows the reason (B239).
2026-10-16 11:56:22 +02:00
nvrl 937a0a6824 perf(core): index queued ops by kind and name (schema 43)
The name checks the drain makes before each create, rename and trash,
and the wake of ops waiting for a name, read the whole queue, which made
a large offline batch quadratic to drain. Schema 43 adds an index on
pending_op(kind, name) (B238).
2026-10-16 11:50:56 +02:00
nvrl fcbb15cb4b fix(core,fuse): back failed uploads off properly and spread their retries
A reconnect made every failed op due at once, quota and access refusals
included, and ops that failed together retried in the same second. The
doubling also stopped at 128 s, short of the 300 s maximum. Reconnects
now move up only unrefused ops, and the backoff reaches its maximum with
a per-op spread of up to a fifth (B237).
2026-10-16 11:48:15 +02:00
nvrl 8aca7836f9 fix(fuse): upload thumbnails with imported, re-dated and app photos
Takeout imports, re-dates and the app's photo upload all sent an empty
thumbnail list, so the photos showed blank in Proton's apps. Re-dates and
app uploads now make thumbnails from their file. A Takeout photo that can
have one is copied out of the archive to the private staging folder,
thumbnailed and uploaded from the copy (B236).
2026-10-16 11:46:17 +02:00
nvrl 60a5a98174 perf(core): find Takeout sidecars by prefix lookup, not by ranking all
Each photo was ranked against every sidecar in its folder, so a folder
of 10,000 photos took about 30 s to scan. The sidecars are now indexed
by stem and each photo is ranked only against the stems a few range
lookups return: 0.1 s for the same folder, with the same matches (B235).
2026-10-16 11:41:02 +02:00
nvrl 51800c77d7 fix(core,fuse): refuse sync roots that overlap home, the mount or app data
Nothing stopped `/`, the home folder, the Proton Drive mount, the app's
own state or cache, or a folder nested in another root from becoming a
sync root, and a restore took a path from another machine's profile as
it was. One validator, `syncroot::problem`, now runs where roots are
added and restored, on paths resolved through symlinks with `..`
refused (B234).
2026-10-16 11:30:40 +02:00
nvrl 518763c5c9 fix(core,cli): end the session on Proton when signing out
Logout only deleted the keyring entry, so the session's tokens stayed
valid on Proton's side. It now ends the session first (DELETE auth/v4,
bounded to 10 s) and signs out locally either way. `pdfs logout` says
when Proton could not be reached (B233).
2026-10-16 11:26:29 +02:00
nvrl ef0f046939 fix(gui): show remote names as text, not Pango markup
Rows, toasts and status page descriptions parse their text as markup by
default. Shared item names, inviter addresses, device names, link URLs,
paths, EXIF values and daemon errors went in unescaped, so a name with
`&` showed as an empty row and other names were drawn as markup. Those
rows and toasts now use plain text, and status page descriptions are
escaped (B232).
2026-10-16 11:24:43 +02:00
nvrl 519ba0591a fix(fuse): give uploads under a low cap the time their share needs
The upload deadline assumed at least 32 KiB/s. Under an upload cap shared
by every drain worker and sync task, a large file ran out of time on every
attempt, took the mount offline and restarted from byte 0. The deadline
now counts the upload's share of the cap as the slowest rate when that
share is lower (B231).
2026-10-16 11:21:57 +02:00
nvrl 42b2f69fa0 fix(core,fuse): keep queued writes durable and their retries their own
- A gap-filled revision blob is recorded as complete once filled, so a
  retry after a lost upload answer recognises its own bytes on Drive
  instead of forking a conflict copy (B228).
- Commits that accept a change into the queue run at synchronous=FULL,
  so an op `close()` reported as done survives a power cut (B229).
- A queued create's adoption window starts at its first send, not when
  it was queued, so a create queued offline no longer takes another
  device's empty node of the same name. A send record that does not
  parse is kept instead of overwritten (B230).
2026-10-16 11:21:48 +02:00
nvrl 0204f69e9e docs(changelog): list the 3.5.0 fixes under Unreleased 2026-10-16 11:07:23 +02:00
nvrl d4ef9116c8 fix(fuse): download the base of a file open for writing on a worker
A read on a write handle was answered on the dispatch loop to keep its
order with the handle's writes, and when the bytes it asked for were
neither written nor cached it downloaded them there, holding every request
of the mount for up to the read deadline. The loop now answers from the
scratch file and the local caches only, and hands a read that needs a
download to a transfer worker with the handle as it was when the read came
in (B227).
2026-10-16 10:47:58 +02:00
nvrl f8c60c9753 build: add a release profile that strips the binaries
No package ships a separate debug package, and the unstripped binaries were
43, 24 and 12 MB. LTO stays off, as the packaging comments already said the
release profile kept it, and unwinding stays on for the workers that catch
panics.
2026-10-16 10:32:05 +02:00
nvrl 607972cf1e fix(fuse): turn generated thumbnails the way their EXIF orientation says
Thumbnails were decoded without the EXIF orientation, so a portrait photo
stored as landscape pixels plus a tag got a sideways tile, and the drain
uploaded that thumbnail to Drive for every client to show (B226).
2026-10-16 10:32:05 +02:00
nvrl 96b70600f0 fix(core): match a Takeout sidecar by prefix only where Google cut its name
The prefix rule for sidecars whose names Google cut at 51 characters also
matched names that were never cut, so IMG_1.jpg's sidecar gave its title
and date to an IMG_12.jpg that had none. It now applies only to a sidecar
name at the cap. Edited copies, which have no sidecar of their own, take
their original's date by a rule of their own and keep their own name
(B225).
2026-10-16 10:32:05 +02:00
nvrl b0ff6ee441 fix(core): never write defaults over a config file that does not parse
load_config gives defaults for a file it cannot read, and every setter
loads, changes one field and saves. The first settings change after the file
was damaged therefore wrote defaults over the mountpoint, the adopted device
and the ignore rules. save_config now refuses while the file on disk does
not parse, and the first-run default is written atomically with mode 0600
(B223).
2026-10-16 10:32:05 +02:00
nvrl bca22056b0 fix(fuse): report write errors at close, and take fsync and fallocate as POSIX does
flush answered ok at once and release synced the scratch file, but close(2)
only returns what flush answers, so a full disk at close was reported to
nobody. flush now syncs a handle with writes on a worker and returns ENOSPC,
EDQUOT or EIO; release then finds the pages on disk instead of writing a
whole file back on the dispatch loop. fsync on a descriptor opened for
reading no longer fails with EBADF (B222).

fallocate passed any mode through and tracked only a punched hole or
growth: a zeroed range came back from the remote base at commit, and a
collapse or insert corrupted the file. Unsupported modes now get EOPNOTSUPP,
zeroed ranges count as written, the real errno is returned, and the state
lock is not held across the syscall (B224).
2026-10-16 10:32:05 +02:00
nvrl 830cfd9faa fix(fuse): allocate a listing's inodes before taking the state lock
A cold listing allocated the local id of every new child in a write
transaction while it held the global state lock, then marked the folder
listed, still under it. Every inline handler of every mount waited on that
transaction, and the transaction on the write connection the drain shares.
The ids are now allocated before the lock is taken, and the folder is marked
listed through the outbox after its children's rows, so a crash in between
no longer leaves it recorded as complete and empty (B221).
2026-10-16 10:05:58 +02:00
nvrl 77dac97afe fix(fuse): keep slow remote calls off the dispatch loop and the metadata workers
A shared folder's listing is fetched again once it is a minute old, and that
fetch ran on the dispatch loop whenever the old listing was still resident,
so every request on the mount waited for Drive (B219).

Thumbnail downloads, size upgrades and the Shared with me listing had no
deadline of their own. On a stalled link each held a mount thread for the
SDK's retry budget, and a folder of photos filled the three metadata workers
that every cold lookup and rename needs. Offline, an uncached thumbnail is
now answered as absent, and these calls give up after 20 or 60 seconds
(B220).
2026-10-16 10:02:35 +02:00
nvrl 2bea1f7672 fix(core): check and count the database on a read connection
PRAGMA integrity_check and the row counts behind pdfs cache inspect read
the whole file, 34 s on a 567 MiB database, and they held the write
connection while they did. Every write in the daemon waited behind them.
They now read a WAL snapshot on a pooled read connection (B218).
2026-10-16 09:56:25 +02:00
nvrl 0f279c7536 fix(core,fuse): compact the search indexes and stop pruning after partial scans
FTS5 keeps deleted entries until their segments are merged, which with
these indexes is close to never. On one machine local_fts held 317 MiB for
40,000 paths, against 20 MiB rebuilt, and search ran four times slower. The
indexes are now compacted once a day after a local scan. A scan that hit the
entry cap or lost a batch no longer prunes the paths it never reached, which
the next scan used to add back (B217).
2026-10-16 09:56:13 +02:00
nvrl d6bc921a4b fix(core): keep the upload queue when a damaged database is rebuilt
A database SQLite reported as corrupt at open was moved aside and replaced
by an empty one, on the grounds that everything in it is fetched again. The
queue of writes Drive has not received, the mirror baselines and the pins
are not, so they were lost. They are now copied out of the damaged file,
read only and table by table, into the new one (B216).
2026-10-16 09:50:53 +02:00
nvrl 3221e4aa97 fix(fuse): mirror sync no longer deletes what Drive or this disk never had
Five ways a mirror pass could lose files:

- A folder deleted on Drive was removed here with everything in it, including
  new files and edits whose uploads had just failed for want of a parent, and
  ignored content such as .git. Such a folder now stays and is created on
  Drive again (B211).
- A node the SDK could not decrypt was missing from the remote walk and so
  read as deleted on Drive. A short listing or an unreadable size now leaves
  the affected paths out of the pass, and a pass in which no synced path is
  left on Drive is refused (B212).
- A file Drive deleted was removed here even when it had been edited since
  the scan. It is now checked again first, and kept (B213).
- A folder deleted here was trashed on Drive although another device had just
  put new files in it. It now stays on Drive (B214).
- A decrypted name holding / or .. became a path outside the sync folder, and
  failed the whole listing in the mount. Such names are left out (B215).
2026-10-16 09:47:08 +02:00
Nils Pukropp 3d96cdbab1 chore(aur): update packages to 3.4.1 2026-10-15 23:47:24 +00:00
nvrl 18cc2ea498 chore(release): 3.4.1 2026-10-16 01:32:59 +02:00
nvrl a9cb389f04 docs: a locked volume after a password reset (B210)
Issue #40: after a password reset the daemon failed every mount with
"failed to decrypt: missing key", and the app waited on "Connecting…"
because control.sock was never created. proton-drive-rs 0.9.1 now
creates a new My Files volume when no active regular volume is left,
as the web client does.

- BUGS.md: B210 with cause, fix and the SDK tests that cover it
- TROUBLESHOOTING.md: what the log line means and how to restore the
  files of the locked volume from the web
2026-10-16 01:32:59 +02:00
nvrl 923c47acb8 chore(deps) updated sdk to 0.9.1
0.9.1 replaces a locked My Files volume, the state a password reset leaves behind, instead of failing every mount with "missing key" (#40).
2026-10-15 23:02:36 +02:00
nvrl 478a0e6cdd fix(core,gui,cli): a blocked sign-in no longer asks to update the app (B209)
Proton answers a sign-in it blocks for abuse or rate limits with code
2028 and the text "This version of the app is no longer supported". The
SDK read 2028 as Unknown, so the app passed that text through and told
users to update a current release (issues #28, #39).

- deps: proton-sdk and proton-drive-rs 0.8.3, which name 2028 Blocked
- core: error::sign_in_blocked detects it
- gui: the login page says Proton is refusing sign-in for now and that
  waiting helps, translated in every catalog
- cli: pdfs login adds the same hint to the error
- cli: pdfs diagnose warns instead of failing when cache.db does not
  exist yet; the daemon creates it after the first sign-in
- docs: BUGS.md B209, changelog, troubleshooting entry
2026-10-13 20:29:23 +02:00
Nils Pukropp 2633b4ed72 chore(aur): update packages to 3.4.0 2026-10-12 16:21:56 +00:00
nvrl b46abfc07c chore(release): 3.4.0 2026-10-12 18:12:00 +02:00
nvrl fd78f088ec test(fuse): let a held create carry its bytes in the simulation
FUSE delivers `release` after `close()` returns, and the release is what
attaches the written bytes to the queued create. A drain worker that is
free can claim the create first and send the file empty, with the bytes
following as a revision. Tests that hold a create, its reply, or lose
its reply then act on the empty create, and their assertions about the
file's content race the revision. CI saw this as an occasional failure
of a_listing_taken_before_a_create_commits_keeps_the_landed_file.

Add Daemon::write_new, which pauses sync, writes the file, waits until
the pending create carries the bytes, and resumes. Use it for every
write that directly follows a create hold.
2026-10-12 18:05:45 +02:00
nvrl 005f7ded1f fix(fuse): a file landing while its folder is pinned stays available offline (B208)
A folder pin reads each file's modification time and size before it
downloads the file. If the file's create landed during the download, the
landing set Drive's time, and the pin then stored the blob under the old
time. The file showed as kept offline but read as stale.

After each download, the pin now compares the file's time and size with
the tree's. If they changed, it downloads the file again under the new
values, at most three times.

The simulation gains FakeClient::hold_next_download to force that order.
2026-10-12 15:40:56 +02:00
nvrl 3e4886f965 fix(fuse): refuse a move between locations under a just-opened file (B207)
The account run of 3.3.0 failed "move refuses a file still being
written" on the on-demand to mirror pairing: pdfs move moved the folder
while open.txt in it was open for writing, and the write landed after
the move.

retire_create re-keys a landed create's nodes row to the real uid first
and repoints the tree and the open write handles after. A file opened in
between holds a handle with the placeholder uid. has_pending_work_under
placed each open handle by looking its uid up in the nodes table, which
no longer has the placeholder, so it saw nothing open under the folder,
and with the queue already empty the move went ahead. Open handles are
now placed in the in-memory tree by inode, which a landing create does
not change.

The simulation test holds the drain at that point with the new
FakeClient::hold_next_landing_of_any, and fails without the fix.
2026-10-12 00:57:10 +02:00
nvrl aac53e3139 test(fuse): let the simulation sync local folders
A simulated daemon could not add a synced folder, so nothing that moves
between a mount, an on-demand folder and a mirror folder could be tested
without a real account.

Listing and registering devices now go through DriveApi, and the fake
Drive keeps the account's computers, each with a root folder outside My
files. The simulated daemons also read their config.json from a scratch
XDG directory instead of the developer's own: an adopted device of the
real account made every sync folder fail to register against the fake
Drive.
2026-10-12 00:57:04 +02:00
nvrl c2bf87596f test(acceptance): remove the local-only symlink the mirror-copy case leaves
"move never loses a mirror copy Drive lacks" puts a symlink in a mirror
folder to prove a move does not lose what sync never uploads, and left it
there. The next mode pairing switches that folder to on-demand, which the
daemon refuses because the link exists only on this computer, so the
account run stopped after pairing 2 of 4. The case now removes the link
wherever the move left it, whether it passes or fails.
2026-10-12 00:24:40 +02:00
nvrl 69d4feaa07 docs: add a security model page for users
Readers asked whether the client can be trusted with their files. The
threat model in ARCHITECTURE.md is written for contributors, so
docs/SECURITY_MODEL.md summarises it for users: encryption comes from
the SDK and rPGP, sign-in uses SRP, what is stored where and how it is
protected, who can reach the daemon, every network connection, what the
client does not protect against, and how each package is built and
signed. It says that there has been no independent audit.

It is linked from the README, the documentation index and SECURITY.md.
2026-10-12 00:07:18 +02:00
nvrl d481dcc15c feat(fuse,core,cli,gui): make folders available offline from the app
The daemon has kept recursive folder pins for a long time, but the app
offered "Available offline" for files only. The context menu, the
details pane and the selection bar now offer it for folders too, and a
pinned folder carries the offline badge in the browser and the launcher.

A folder pin used to download the whole subtree inside the control
request, so a large folder outlasted the client's 120 s read timeout,
and every pinned file was read into memory before it was cached. The
daemon now records the pin, answers at once and downloads on a
background thread as a job the tray and the app show, stopping when the
folder is unpinned or the daemon stops. Pins of files and folders both
go through fetch_content, which streams to disk and skips fresh blobs.
A folder pin is refused while offline instead of looking kept.

A file kept only because a folder above it is pinned has no pin of its
own, and unpinning it silently did nothing. ListDir now reports
pinned_by_folder; the app shows such an entry as available offline with
the toggle greyed out, and the daemon refuses the unpin with a message
naming the folder. Unpinning a folder no longer evicts files that are
pinned on their own.

Requested on Reddit.
2026-10-12 00:07:18 +02:00
nvrl e0442c56b9 fix(packaging): stop the service before the package is removed
Removing the package left the user service running from the deleted
binary, with ~/ProtonDrive still mounted. A user who then deleted the
folder to clean up had every file in it moved to the Proton Drive trash,
as any delete through the mount is.

The .deb (prerm on remove), the .rpm (%preun on erase) and the Arch
packages (pre_remove) now stop proton-drive.service for every user with
a running user manager, through runuser with that user's
XDG_RUNTIME_DIR, so no systemd-machined is needed. Upgrades are left
alone. Queued writes stay in the state directory for a later install.

The pacman install script lives once in packaging/ and the AUR
directories link it; push.sh copies its content into each AUR
repository. The uninstall guide now warns not to delete the mount point
while the service runs.

Reported on Reddit by a user who restored the files from the trash.
2026-10-12 00:07:07 +02:00
nvrl ee76f4ac60 fix(packaging): recommend each keyring only on its own desktop
The RPM recommended both gnome-keyring and kwallet unconditionally, and
dnf installs weak dependencies by default, so a GNOME install on Fedora
pulled in KWallet and KDE Frameworks with it. The recommendations are
now rich dependencies: KWallet only with plasma-workspace, GNOME Keyring
only without it, and the AppIndicator extension only with gnome-shell.
RPM does not accept `unless` in a weak dependency, so the keyring uses
the equivalent `(gnome-keyring or plasma-workspace)`.

Reported on Reddit by a Fedora GNOME user.
2026-10-12 00:06:51 +02:00
nvrl f6c651b06b ci(nix): push Nix builds to a Cachix binary cache
Every Nix build of main and of each release tag now goes to
proton-drive-linux.cachix.org, so installing the flake downloads the
client instead of compiling it. The flake names the cache in nixConfig
for nix run and nix shell, and the install guide shows the nix.settings
a NixOS system build needs, since it does not read an input's
nixConfig. A fork's pull request has no token and only pulls.
2026-10-11 23:02:22 +02:00
Nils Pukropp 8b7d7ae87f chore(aur): update packages to 3.3.0 2026-10-11 20:08:31 +00:00
nvrl 160626b29d chore(release): 3.3.0 2026-10-11 21:53:52 +02:00
nvrl f1f1f8ad4d ci(release): build the Nix flake at every release tag
A flake input pinned to vX.Y.Z is the Nix release, so the release
workflow now builds the flake at the tag and checks its version before
the GitHub release is published. The install guide shows how to pin the
input to a release tag, and the release checklist names the files that
carry the version.
2026-10-11 21:53:52 +02:00
nvrl bb28a18f49 Merge branch 'feat/issue-37-nix' 2026-10-11 21:48:28 +02:00
nvrl a3c0c9ee9b ci(release): take the release notes from the changelog
The release workflow created each GitHub release without a body, so 3.1.2,
3.2.0 and 3.2.1 shipped with empty notes. It now copies the version's
section of docs/CHANGELOG.md into the release, and fails when the section
is missing.
2026-10-11 21:24:39 +02:00
nvrl 91fa1614ed fix(core,fuse,cli,gui): say when an upgrade left the old daemon running
A package upgrade replaces the binaries but does not restart the user
service, so the daemon keeps the old release and its bugs until the next
login. On 3.2.0 the reporter of #33 still saw "Couldn't sync Documents:
read .../snapshots/37: Permission denied", which only a daemon older than
3.2.0 can produce: since then the scan leaves an unreadable subfolder out
and only an unreadable sync root fails the pass (B206).

The daemon now reports its release in Status; one from before the field
omits it, which reads as older. When the daemon is older than the
front-end, the tray says "Update not finished - restart Proton Drive" and
offers a restart, the sync card shows "Restart to finish updating" with a
Restart button, and pdfs status and pdfs diagnose print the systemctl
command. A newer daemon is not flagged, since then the front-end is the
stale one.

Refs #33
2026-10-11 21:20:51 +02:00
nvrl 87361a4573 fix(core,fuse,gui): a photo upload the SDK refuses no longer says offline
Every SDK error that was not an API reply was classified as Offline.
Only a transport failure is. An upload on an account without a photo
library fails in the SDK before any request, and the app said "You're
offline" while every other call worked (issue #38).

Classify by variant: Transport is Offline, InvalidOperation is Invalid,
Serialization is Remote, Crypto and KeysLocked are Internal. The daemon
logs a failed photo upload, and the Photos page greys out Upload when
the account has no photo library.

docs/BUGS.md B205.
2026-10-11 19:26:53 +02:00
Nils Pukropp 6d0b6ad553 chore(aur): update packages to 3.2.1 2026-10-11 17:24:37 +00:00
nvrl 170c3f01ff chore(release): 3.2.1 2026-10-11 19:19:36 +02:00
nvrl 976ad2aa04 docs(changelog): list the issue #35 fixes under Unreleased
The CPU spin while offline, the expired session shown as offline, the
lost refresh tokens and the keyring-refused rotation were fixed without
a changelog entry.
2026-10-11 18:56:55 +02:00
nvrl b73cd087aa test(core): time the B12 index write as the best of five runs
A single timed run lost the CPU to the rest of the suite running
alongside it and read as the full-scan regression (171 ms against
701 ms). The fastest of five runs still catches the 6x scaling of the
old path.
2026-10-11 18:56:55 +02:00
nvrl e402981708 fix(packaging): ship the Adwaita icons with the Nix package
GTK ships only a few stock icons itself. On a NixOS session without
Adwaita installed, such as Hyprland, the Photos and Computers sidebar
entries and about twenty other icons showed as missing (#37).
adwaita-icon-theme in buildInputs puts it on the wrapper's XDG_DATA_DIRS.
2026-10-11 18:55:00 +02:00
Claude 453c439b74 fix(packaging): give the Nix package TLS for WebKitGTK
On NixOS the CAPTCHA window failed with "TLS support is not available":
WebKitGTK's GIO had no TLS backend. With glib-networking in buildInputs,
wrapGAppsHook4 puts its GIO modules on GIO_EXTRA_MODULES in every
wrapper.

Refs #37

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuyxByBpK3fSWtyPkugeHz
2026-10-10 08:23:21 +00:00
Claude ceba0f5849 feat(packaging): add a Nix flake and NixOS module
Builds the four binaries with rustPlatform.buildRustPackage from
Cargo.lock, installs the desktop entries, icon, user service and
translations as the other packages do, and wraps the binaries with
exiftool and ffmpeg on PATH. The NixOS module installs the package,
pulls proton-drive.service into the graphical session with the
fusermount3 wrapper on its PATH, and enables FUSE and GNOME Keyring.

A Nix workflow builds the flake when its recipe, the lock files or the
code change.

Refs #37

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuyxByBpK3fSWtyPkugeHz
2026-10-09 22:07:20 +00:00
Claude 3e0db8cf46 fix(fuse): fill an incomplete queued revision's gaps on a write open
A partial write whose base was not in the content cache queued an
incomplete revision, and every write open over it was refused with EIO
until the drain uploaded it. CI's one-client-lan simulation hit this on
seed 1: a create over a file whose partial edit had not drained yet.

The write open now copies the blob and fills its gaps from the remote
base with Core::fill_gaps, as the drain does, and fails only when that
fill fails (B204).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EuyxByBpK3fSWtyPkugeHz
2026-10-09 11:36:46 +00:00
nvrl 2a86830e72 Merge branch 'fix/issue-35-dead-session'
Issue #35: a refused session showed as offline forever, the drain
workers spun while offline, and refreshed tokens could be lost or
overwritten (B200-B203, B61). Needs proton-sdk 0.8.2.
2026-10-09 12:34:38 +02:00
nvrl 43031ecf99 chore(deps) updated sdk to 0.8.2 2026-10-09 12:34:02 +02:00
nvrl 2b6c37f543 fix(fuse,core,cli,gui): say when the keyring refuses the refreshed sign-in
A rotation the keyring refused was retried and kept in the kernel
keyring, but only the log said so, and a reboot before the keyring took
it still signed the user out.

`Response::Status` carries `tokens_unsaved` while one waits. The tray
says "Sign-in not saved — unlock your keyring" and needs attention, the
app's status says the same, `pdfs status` prints a Keyring line and
`pdfs diagnose` warns. New strings are translated in every catalog
(B61).
2026-10-09 12:22:02 +02:00
nvrl 3458e2e72d fix(core): keep refused token rotations in the kernel keyring
A rotation the Secret Service refused lived only in the daemon's
memory, so a crash or SIGKILL before the retry landed lost the only live
refresh token.

Keep it in the kernel keyring as well until it is written. The next
resume, by the daemon or `pdfs unlock`, puts it back in when the stored
blob still holds the refresh token it spent, and writes it; one left
over from an older login is dropped. `kernelkey` now holds named keys,
the passphrases and the unsaved tokens (B61).
2026-10-09 12:14:25 +02:00
nvrl 0ac5928138 fix(fuse,core,cli): resume a session another process refreshed
`pdfs unlock` resumes the daemon's session and may refresh it. The
daemon then holds a spent refresh token, and since B202 a refusal of it
read as an expired session although the keyring held a live one.

Before calling a refused session expired, compare the keyring with what
this process last stored. When it holds a newer rotation or another
login, end the mount with `MountOutcome::ResumeStored` and mount again
from the keyring. A rotation another process already wrote counts as
this session's, so the daemon keeps writing its later ones (B203).
2026-10-09 12:08:17 +02:00
nvrl beb3b2e4e4 fix(core): retry rotated tokens the keyring refused
A failed keyring write after a refresh was only a warning, and the one
live refresh token then lived in memory only (B61). The sink now keeps
the rotation as unsaved and retries it in the background after 1 s, 5 s,
30 s and then every 5 minutes, until it is written or the blob has moved
on. The refusal is logged as an error naming `pdfs login`.

docs/BUGS.md B61.
2026-10-09 11:29:31 +02:00
nvrl bf43476e88 fix(fuse,core,cli,gui): show a refused session as expired, not offline
A refresh token Proton refused, or a deleted or disabled account, was
treated as a lost connection: the mount came up "offline", the probe
retried forever and logged only at debug, and nothing told the user that
only signing in again helps (issue #35).

session_revoked() names those errors. The root fetch, probe, drain,
event poll, event seed and photos root now set session_expired, log one
warning naming `pdfs login`, and stop retrying. Cached files stay
readable and changes stay queued.

Response::Status carries session_expired (serde default false). The
tray shows "Session expired — sign in again" with a Sign In Again…
item that opens the app's sign-in page; the app's status and Sync card
say the same with a button. `pdfs status` and `pdfs diagnose` report it.
New strings are translated in every catalog.

docs/BUGS.md B202.
2026-10-09 11:27:09 +02:00
nvrl 6363125fec fix(core,cli): write rotated tokens only into their own session's blob
Proton refresh tokens are single-use. store_tokens replaced the tokens of
whatever blob was stored, so after `pdfs login` restarted the daemon, the
old daemon's shutdown persist wrote its spent tokens into the new
session's blob, and the new daemon failed with InvalidRefreshToken
(issue #35). `pdfs unlock` and the legacy migration had the same shape.

Each session now writes through a TokenSink that only touches a blob
with its session id that still holds the refresh token it last left
there. Shutdown only flushes a rotation the keyring refused. unlock and
migrate_legacy re-read the blob after verifying.

docs/BUGS.md B201.
2026-10-09 11:08:40 +02:00
nvrl 51d114a8da fix(fuse,core): idle drain workers no longer spin while offline
Offline or paused, a drain worker claims nothing and then slept until the
earliest queued op fell due. Every queued op was already due and nothing
attempted it, so the wait was zero and all 16 workers spun on the queue
for the whole outage (issue #35: 17 min of CPU in 2 min 16 s).

A worker that claims nothing now sleeps the full idle poll; a reconnect
or resume wakes it. earliest_due_at also uses the claim's exclusions, so
an op held behind a claimed op on the same node no longer counts as due.

docs/BUGS.md B200.
2026-10-09 11:05:12 +02:00
Nils Pukropp 31be27f30e chore(aur): update packages to 3.2.0 2026-10-09 01:14:52 +00:00
nvrl 4c498d559c Merge pull request #32 from narrrl/fix/captcha-in-browser
fix(gui,cli,core): offer the CAPTCHA in the browser when the web view…
2026-10-09 03:04:10 +02:00
nvrl 7fd3439b60 Merge remote-tracking branch 'origin/main' into pr32-merge
# Conflicts:
#	docs/CHANGELOG.md
#	po/de.po
#	po/en_GB.po
#	po/es.po
#	po/fr.po
#	po/it.po
#	po/nl.po
#	po/pdfs.pot
#	po/pl.po
#	po/pt_BR.po
2026-10-09 03:03:31 +02:00
nvrl c4847c4d52 chore(release): 3.2.0 2026-10-09 02:56:26 +02:00
nvrl ad49e32de9 fix(gui): context menus size to fit instead of scrolling
A popover is sized and placed again when its parent's layout manager
allocates it. Context menus were parented to the outermost list view,
and the keyboard menu to the ScrolledWindow; neither has a layout
manager, so a menu kept the size it was first given and often opened
too short, with a scrollbar.

The host is now the first widget from the list view up that has a
layout manager, which still keeps menus out of list items and the theme
rules that style their insides.
2026-10-09 02:44:00 +02:00
nvrl 19854b2d76 fix(gui): tidy the Share dialog
Send Invitations and Create Public Link sat alone in boxed-list rows,
which read as broken settings. They now sit under their section at its
right edge. The role picker moves into the address row it applies to,
people get avatars, a member needs no subtitle, and invitations say they
are pending. The People and Public link sections show a loading row
until the share is read, and one line instead of nothing when it can't
be. The page is an AdwPreferencesPage, for the standard spacing.

Translated in every catalog.
2026-10-09 02:41:07 +02:00
nvrl 085335ca14 feat(gui): one main menu, on the sidebar's account row
The sidebar header had a hamburger beside the brand, and the account row
in the footer had its own ⋮ menu; both offered Preferences. The header
button sat next to the page's own tools and read as one of them.

The account row is now the main menu's button, as in Proton's own apps:
Preferences, Keyboard Shortcuts and About, then Sign Out. The whole row
opens it, upwards, and F10 still does as the primary menu.
2026-10-09 02:37:25 +02:00
nvrl 9fb7e84d41 fix(fuse,gui): a stopped share leaves Shared by me at once (B199)
Stop Sharing removes the link, invitations and members but cannot delete
the share itself, and Drive keeps listing the empty share until its
backend removes it, minutes later.

The daemon now leaves a share with no member, invitation or link out of
the listing (unless a lookup failed, so a live share is never hidden),
and the app drops stopped items from the page as soon as the requests
succeed instead of waiting for the slow reload.
2026-10-09 02:34:03 +02:00
nvrl 7ebe8e5210 fix(fuse,core,gui): routine mirror scans no longer show as syncing
Every mirror pass, the two-minute poll included, reported a "Checking
{folder}" job, and the tray took any job for syncing, so its icon spun
every two minutes over folders with nothing to do.

JobItem gains a background flag (absent from older daemons, so false)
for housekeeping that changes nothing in the user's files: a pass still
scanning, the home index rebuild and the cache integrity check. The tray
keeps them in its menu line but only turns busy for transfers or a job
that is not background; a pass that queues work becomes a Syncing job
and still spins the icon.

The watcher also reconciles only the folder an event lies in, instead of
walking every mirror folder over one edit.
2026-10-09 02:27:50 +02:00
nvrl 922399b9a2 fix(gui): even breadcrumb spacing, no invented hyphens in tile names
The current folder in the breadcrumb trail was a bare label, so the "›"
before it hugged it while the ones between buttons sat in their padding.
One helper now builds the separator and the current segment for My Files,
Shared with me and Computers, with a flat button's side padding.

Grid tile names broke mid-word with a hyphen that is not in the name
("SteamSetup.e-xe"); turn Pango's inserted hyphens off.
2026-10-09 02:18:47 +02:00
nvrl 3b80e9bc8f test(core): make under-budget cache check deterministic
The timing ratio between a near-empty and a 2500-entry index measured the
CI runner's load and flaked on main. Plant an oversized index row behind
the running total instead: any budget check that consults the index would
evict and reseed, so the test fails without depending on wall-clock time.
2026-10-08 21:26:56 +02:00
nvrl dc8eb27c99 Merge pull request #34 from narrrl/fix/sync-unreadable-subfolder
fix(fuse): skip unreadable folders in sync, never evict local-only files
2026-10-08 20:59:55 +02:00
nvrl 07d4ee42d9 Merge branch 'main' into fix/sync-unreadable-subfolder 2026-10-08 20:59:21 +02:00
nvrl 0ac5e2e4fe Merge pull request #31 from narrrl/feature/windows-migration
Feature/windows migration
2026-10-08 20:57:31 +02:00
nvrl 7eb6aa0640 fix(fuse): skip unreadable folders in sync, never evict local-only files
An unreadable subfolder, such as a container's root-owned cache, failed
every pass of its synced folder, so nothing else in the tree reached
Drive (#33, B196). The scan now leaves such a path out, together with an
entry that vanished mid-scan and a name that is not UTF-8, and the pass
drops everything under it from classification, the wipe guard and the
push pass's deletions. It is never read as deleted and never downloaded
into. An unreadable or absent root still fails the pass. The activity
feed names unreadable paths once each time the set changes.

Two related bugs on the way to online-only:

- The push pass read deletions from the raw baseline, so a synced path
  that later became ignored was trashed on Drive (B197).
- The switch evicted everything once the folder was idle, including
  ignored paths such as .git/, symlinks, unreadable folders and files
  changed after the pass, none of which are on Drive (B198). It now
  refuses and names the path.

Closes #33
2026-10-08 20:45:54 +02:00
nvrl d13e74fc12 fix(gui,cli,core): offer the CAPTCHA in the browser when the web view cannot start (B195)
The verification window now runs in its own process, so the app survives
when WebKit cannot start its sandbox on secureblue, but the CAPTCHA still
could not be solved there (#29). WebKit binds the journal socket with
`--ro-bind-try`, bubblewrap only skips a missing path, not an unreadable
one, and WebKit has no switch to leave the socket out.

Proton Mail Bridge never uses a web view: it opens verify.proton.me in the
system browser and retries with the original challenge token, typed with
the offered methods. Do the same as a fallback.

- core: `browser_verification_url` and `browser_verified` build the page
  URL and the retry credential from the challenge
- gui: when the verification window fails, a dialog opens the page in the
  browser and continues the sign-in once the user is done
- cli: `pdfs login` handles a CAPTCHA the same way instead of failing
- i18n: translate the new dialog in every catalog
- docs: B195, TROUBLESHOOTING.md, ARCHITECTURE.md, changelog

Fixes #29
2026-10-08 19:02:29 +02:00
Nils Pukropp c0d211238b chore(aur): update packages to 3.1.2 2026-10-08 13:55:27 +00:00
nvrl 07c9b675d3 chore(deps) updated sdk to 0.8.1 2026-10-08 15:25:44 +02:00
nvrl f82343b844 Merge pull request #30 from narrrl/fix/secureblue-captcha
Fix/secureblue captcha
2026-10-08 15:25:05 +02:00
nvrl 3e3dd7db41 feat(fuse,core,cli,gui): move another computer's backup to this one
A user switching from a Windows PC to Linux had two half answers:
adopting the old device makes this machine back up as it, and restoring
its folders keeps them in the old device's tree. Neither puts the data
under this machine's own device.

Device roots are separate shares on the main volume, so each top-level
folder of the other device can move under this device's root in one
server-side move. Nothing is uploaded again, and the other device stays
registered, without those folders.

- core: `Request::MigrateDevice { uid }` and
  `Db::sync_folder_set_share`
- fuse: `Core::migrate_device` moves every folder, suffixes a name
  already taken here with the other device's name, keeps going past a
  failed folder, and re-points sync folders that already tracked a
  moved folder at the new share
- cli: `pdfs devices migrate UID [--yes]` confirms, moves, then runs
  the ordinary restore picker
- gui: "Move to This Computer…" on another computer's row, followed by
  the restore picker
- i18n: translate the new strings in every catalog
- docs: CLI.md, USER_GUIDE.md, RECOVERY.md, ARCHITECTURE.md, changelog
2026-10-08 13:34:28 +02:00
Nils Pukropp d09e10dc7d chore(aur): update packages to 3.1.1 2026-10-08 09:49:25 +00:00
nvrl 8d0351124c chore(release): 3.1.2 2026-10-08 11:31:18 +02:00
nvrl a888e0bb28 fix(gui): run the CAPTCHA page in its own process (B195)
On secureblue, WebKit cannot start the sandbox for the verification page:
bubblewrap is kept from the journal socket, xdg-dbus-proxy fails, and
WebKit ends its process with g_error. The webview lived in pdfs-app, so
the whole app crashed after the sign-in (#29).

- gui: `pdfs-app --human-verification` shows the page in its own window,
  reads the URL on stdin and prints the token on stdout; its logs go to
  stderr
- gui: a child that dies or fails without a token ends the sign-in with a
  message that the page could not be opened, instead of "cancelled"
- i18n: translate the new message in every catalog
- docs: B195 in the bug ledger, the cause and a one-time workaround in
  TROUBLESHOOTING.md, the child process in ARCHITECTURE.md, changelog
2026-10-08 10:25:14 +02:00
nvrl c7fc85e943 fix(gui): create the state dir before the tray and app use it (B194)
On a fresh install only the daemon and the CLI created
~/.local/state/proton-drive-linux. The app starts the tray before the
first sign-in, so the tray failed to bind its single-instance socket and
exited, and the Takeout page could not remember staged archives (#29).

- gui: pdfs-tray and pdfs-app call AppDirs::ensure at start, which also
  makes the directories owner-only
- docs: B194 in the bug ledger, changelog entry
2026-10-08 10:25:04 +02:00
nvrl 5bfae0fee1 docs(changelog): add the 3.1.1 entry
3.1.1 shipped the client identity change without a changelog entry. Also
add the missing release links for 3.1.0 and 3.1.1.
2026-10-08 10:24:07 +02:00
nvrl edd62c5f7a chore(release): 3.1.1 2026-10-07 17:01:32 +02:00
nvrl 5c1530b4b9 fix(core,gui): send the release version as the client identity
Every release sent the fixed `external-drive-linux@0.1.0-alpha`. If Proton
ever retires that version, every release fails sign-in at once and no update
can fix it.

- core: APP_VERSION and USER_AGENT follow CARGO_PKG_VERSION
  (`external-drive-linux@<version>-stable`, `proton-drive-linux/<version>`)
- core: `app_version` and `user_agent` in config.json are optional overrides;
  the fixed values older releases stored there read as unset and leave the
  file on the next save, a hand-set value is kept
- gui: the About dialog shows the release, its debug info the identity
  actually sent
- docs: describe the new default and the migration
2026-10-07 17:01:32 +02:00
Nils Pukropp ce324510de chore(aur): update packages to 3.1.0 2026-10-07 10:53:15 +00:00
nvrl 274bfd0ca2 Merge pull request #27 from narrrl/feature/two-password-mode
Feature/two password mode
2026-10-07 12:47:05 +02:00
nvrl b8cfd30ff3 fix(core): give the kernel-keyring unlock owner read access
A new key grants its owner only "view"; read needs possession, which a
process has only when its session keyring links to the user keyring. A
systemd user service or a container does not guarantee that, so the
daemon could store the unlock and then fail to read it back (EACCES).
Set the permissions explicitly after add_key.
2026-10-07 08:17:50 +02:00
nvrl 9fa88ff397 feat(core,cli,gui): support two-password accounts without storing the mailbox password
A two-password account signed in with its login password as the mailbox
password, so the key chain never unlocked and the daemon retried forever.

Login now asks for the mailbox password and checks it against the account
(proton-sdk 0.8 `unlock`), so a typo is a retry. The password is never
stored: the keyring blob keeps the per-key passphrases instead, and for a
two-password account only when the user opts in (`--remember`, or the
"Keep unlocked on this computer" switch). By default they live in the
kernel keyring for the login session and the account is locked after a
reboot until `pdfs unlock` or the app supplies them again.

- core: kernelkey module, Error::Locked / WrongMailboxPassword, unlock,
  lock, key_state, ready; blobs written by 3.0.x are migrated on first
  start; token refresh updates only the tokens in the stored blob
- cli: `pdfs unlock [--remember]`, `pdfs lock [--forget]`; status and
  diagnose report the key state; the daemon waits while locked and is
  marked non-dumpable
- gui: mailbox password dialog at sign-in and on open, locked tray state
- po: all catalogs translated
- bump to 3.1.0, depend on proton-sdk and proton-drive-rs 0.8.0

Refs #26.
2026-10-07 08:17:50 +02:00
nvrl f105159186 fix(cli,fuse): keep answering the systemd watchdog while the daemon retries
The watchdog is armed when the process starts, but only the mount's supervisor
pinged it. A daemon waiting for a login or backing off after a failed mount
(#26: key unlock failing) was aborted after WatchdogSec and left a coredump
instead of its error. Sleep through idle_sleep in those stretches; the mounted
ping still has to pass the control-socket probe.
2026-10-07 08:17:50 +02:00
nvrl a14a63c94c ci: add an on-demand Fedora test build
Manual workflow that builds the RPM and the bare binaries in a Fedora
container, so a user can try a branch against Fedora's own GTK stack.
Uploads them as an artifact and, on request, as one rolling pre-release
per branch that needs no GitHub login to download.
2026-10-07 04:27:22 +02:00
nvrl ed3477d1b3 docs: point contributors at the simulation runs and the ledger rules
DEVELOPMENT.md lists the daemon's current modules, asks for the
simulation runs on FUSE, queue and sync changes, and adds how to handle
a bug fix and a user-visible change, including the six pages the wiki
publisher copies. The release checklist moves to TESTING.md.
2026-10-06 19:29:02 +02:00
nvrl aa384ae2eb chore: cite docs/BUGS.md and drop references to notes outside the repo
66 comments cited `bugs.md B<n>`, the ledger's old name; they now say
`docs/BUGS.md B<n>`, and every cited number has an entry. About 90 more
cited offline.md, devices.md, features.md, mount-architecture.md,
plan.md and improvements.md, planning notes that were never in the
repository. Those citations are removed; the comments keep their text.
2026-10-06 19:29:02 +02:00
nvrl 10c69b0d64 test(acceptance): explain the metadata floor for local-first mounts
The throughput case said every create and unlink on a Drive mount waits
for Drive. Since 3.0.0 that holds only with local_first off; the floor
of 3 operations a second stays for that case.
2026-10-06 19:29:02 +02:00
nvrl 39be354bd2 docs(bugs): give the ledger an index and one order
The ledger ran B193 down to B78, then B1 up to B75, with T1 to T4 in
the middle and a stray top-level heading inside B14. Status lines used a
dozen wordings for the same few states, so finding what is still open
meant reading 6,600 lines.

Add a header: what the numbers promise (permanent, never reused; B76
and B77 were never assigned), the status vocabulary, and how to add an
entry, with a template. Add an index sorted by status, with what is left
for every open or partly fixed entry. Order all entries newest first
under one Entries section, and move the acceptance-runner findings to
their own section. Titles are in sentence case; anchors are unchanged.

Entry text is kept as written, except B89: the file it described,
proton-dev/CLAUDE.md, no longer exists, so it is closed.
2026-10-06 19:29:02 +02:00
nvrl 2cca8afba9 docs: describe local-first changes and sync issues for users
The user guide said only writes were queued. Since 3.0.0 creating,
renaming and deleting through the mount return at once too, and changes
Drive refuses become sync issues. Add a section on changes waiting for
Proton Drive: retry, export, discard, and the two cases that land under
another name or place instead of failing.

Troubleshooting covers changes that have not reached other devices yet,
`pdfs sync issues` and the local_first fallback, and names the real
error for symlinks and hard links (EPERM, not EOPNOTSUPP). CONFIGURATION
lists MemoryHigh and the test variables; CLI notes that `sync export`
takes a folder.
2026-10-06 19:14:29 +02:00
nvrl 6e9beb4b16 docs: rework contributor docs for the 3.0.0 local-first mount
ARCHITECTURE.md described the 2.x write path, where namespace changes
waited for Drive and every mount had its own inode space. Rewrite it
around the local tree (lid, inode = lid + 1, local~<lid> stand-ins), the
queue and its 16 drain workers, sync issues, and the remaining split
between the mount and the mirror engine. Drop the pins.json claim; pins
moved into the database at schema v5.

TESTING.md gains the simulation runs and loses the per-case narrative
that --list and BUGS.md already carry. ROADMAP.md takes over the
post-3.0.0 items from the milestone status file.

Remove MILESTONE-3.0.0-STATUS.md (a working log) and UI_UX_PLAN.md (the
2.x design audit). Code comments and CI that cited the untracked
docs/MILESTONE-3.0.0.md now point at TESTING.md, ARCHITECTURE.md and
ROADMAP.md. Threat-model links follow the section to §10.
2026-10-06 19:14:29 +02:00
nvrl 63484bf4b6 chore(images): add cover image
1920×1080 cover for link previews, social posts and the GitHub social preview.
The website renders its og.png from cover.svg.
2026-10-06 14:46:56 +02:00
nvrl 37c66b7220 chore(support) addd buy me a coffee badge + link 2026-10-06 10:49:24 +02:00
Nils Pukropp 41b6edbecb chore(aur): update packages to 3.0.0 2026-10-05 05:12:08 +00:00
nvrl 217738318e chore(release): 3.0.0 2026-10-05 05:16:57 +02:00
nvrl 9291580fd6 test(fuse): drop the known-bug matchers the runs confirmed fixed
The confirmation run on 812eb3d (PDFS_SIM_KNOWN=fail, twelve seeds)
passed every profile, so no open bug is tolerated any more: the call,
hang and damage matchers for B125 to B134 go, and any of them now fails
the seed. B126, B128 and B133 were fixed with B138, B162 and B137;
B134 stays open with local_first off only, which the runs do not mount.
2026-10-05 05:15:19 +02:00
nvrl 812eb3d0bc fix(fuse): a deleted file's trash leaves a file of the user's under the name it was sent as (B193) 2026-10-05 04:50:17 +02:00
nvrl 31863a1da1 fix(fuse): a new file closed empty reaches Drive at once (B192) 2026-10-05 04:50:10 +02:00
nvrl 57a825887c fix(core): a file deleted after its failed create was written again leaves Drive (B191) 2026-10-05 04:50:02 +02:00
nvrl 1436c5f37e fix(fuse): a stop queues the writes whose release the kernel never sent (B190) 2026-10-05 03:55:52 +02:00
nvrl f0a43fb923 fix(fuse): changes that failed on a short outage go out once the link is back (B189) 2026-10-05 03:48:13 +02:00
nvrl b47620651f fix(fuse): a restart no longer brings back the write of a file deleted while open (B188) 2026-10-05 03:37:41 +02:00
nvrl 2baad6720d fix(core): leave no temp copy when the cache adopts a blob it already holds (B187) 2026-10-05 02:52:51 +02:00
nvrl c271a1897c fix(fuse): keep what an open file reads when its delete is queued (B186) 2026-10-05 02:51:55 +02:00
nvrl d26dc8ada4 fix(fuse,core): retry a create at once when a write replaced its blob (B185) 2026-10-05 02:34:05 +02:00
nvrl 93f02672e6 docs: describe 3.0.0's local-speed writes and leave batched trashes and moves for later 2026-10-05 02:02:34 +02:00
nvrl 307305282e fix(fuse,core): what a create sent, and where, outlives a restart and a rename (B182, B183, B184)
The known-bug confirmation run failed one_client_on_a_flaky_link seeds 6
and 11 on three bugs around a create whose answer was lost:

- B182: after a restart, a write over a file whose create landed but was
  never read back forked a conflict copy. A revision of a file that has
  no revision of Drive's now adopts the landed node first.
- B183: the SHA-1s a create sent were kept in memory, so after a restart
  its trash or retry no longer knew the node Drive made. They are now on
  the op's row (pending_op.sent_sha1, schema 42).
- B184: a rename rewrote the op's target, and the trash and the retry
  looked for the node under the last name only. Each attempt's parent
  and name are recorded (pending_op.sent_to); the trash looks under all
  of them, and a retry adopts the node an earlier attempt made and moves
  it to the current name.
2026-10-05 02:01:50 +02:00
nvrl a8ed6060af feat(core,fuse,gui): show a sync issue on its node in the file browser
A refused change was only on the Sync page. A folder listing now carries
each node's issue (by local id, so a file not yet on Drive has it too),
and the browser's badge ranks it first, folders included, with the
issue's explanation as its tooltip. The event sampler publishes Files
when the set of issues changes, so the badge comes and goes with it.
2026-10-05 01:06:03 +02:00
nvrl 9dfe4c5450 feat(core,fuse,cli,gui): discard a queued change and undo it here
A change Drive will not take could be exported or retried, never let go
of. The Sync page's Discard button (with a confirmation) and
`pdfs sync discard <id>` drop it: a node made here goes, as if deleted
before it was sent; a node Drive has goes back to what Drive has, under
the same inode, with every change queued on it, while what is queued
below it stays. A change on the wire or an open file is refused.
2026-10-05 00:57:02 +02:00
nvrl da013c29d6 fix(gui): word a queued edit kept as a conflict copy in the user's language (B181)
The daemon logs it as "<reason>; local changes uploaded as <copy>" or
"...kept at <path>", which the Activity page showed as written, in
English. It now reads both and the three reasons, one sentence each.
2026-10-05 00:17:11 +02:00
nvrl 457a9d26f4 ci: keep the stacks of a hung simulation run
CI on f9b1638 failed slow-link seed 2 on a step that hung for 150 s, and
the stacks that would say where were left on the runner.
2026-10-05 00:13:42 +02:00
nvrl a68dae9422 fix(fuse): a kernel notice sent from the runtime hands its worker's tasks on (B130) 2026-10-05 00:02:56 +02:00
nvrl e743cd60cf fix(fuse): a folder whose mkdir is queued is listed from the database (B125) 2026-10-05 00:02:40 +02:00
nvrl f9b1638603 docs: record the 903452a account run and CI in the 3.0.0 status 2026-10-04 23:48:58 +02:00
nvrl 6539c7314b test(acceptance): a queue wait after a restart waits for the daemon to answer
A mirror target's root is plain local storage, so the durability case's
wait for the mount passed at once after the restart. The queue wait then
read `pdfs --json status` before the new daemon listened, took its null
"mount" for an empty queue, and the run's cleanup found no control socket.
2026-10-04 23:47:57 +02:00
nvrl 903452a15b fix(fuse): a stop wakes every drain worker (B180) 2026-10-04 23:00:07 +02:00
nvrl 4e0fd3c0b1 fix(fuse): a file read just as its create lands has its bytes (B179) 2026-10-04 22:52:00 +02:00
nvrl a6f019ebe6 fix(fuse): a folder trashed while sync is paused drops the creates in it (B178) 2026-10-04 22:51:47 +02:00
nvrl 0f3c06f982 fix(core): a folder listed just as its create lands is listed from the database (B177) 2026-10-04 22:41:57 +02:00
nvrl 4f141c78ca fix(fuse): a write closed just as its file's create lands is queued (B176) 2026-10-04 22:31:49 +02:00
nvrl ce60e006ac fix(fuse): a file replaced just as its create lands is trashed on Drive (B175) 2026-10-04 22:24:28 +02:00
nvrl 1612a9d07a fix(core): a file renamed just as its create lands keeps its access (B174) 2026-10-04 22:15:01 +02:00
nvrl 378667fb37 fix(fuse): a file deleted just as its create lands is trashed on Drive (B173) 2026-10-04 21:51:52 +02:00
nvrl 3b81cc5240 test(fuse): a simulated daemon does not ask the real API for its share id 2026-10-04 21:51:36 +02:00
nvrl d850265225 fix(fuse): a file renamed while its upload is read back keeps its new name (B172) 2026-10-04 21:34:20 +02:00
nvrl e32f72bfca fix(core,fuse): a deleted file's trash is not held back by its create (B171)
A file deleted after its create was sent keeps the op as a trash. The op
kept the create's attempts and backoff, and the drain recorded the failure
of a create attempt still on the wire on the row that was by then the trash.
The trash waited out the create's backoff.

The trash starts due, as a create that becomes a revision does, and the
drain records a failed attempt only if the op is still the kind it ran as
(`Db::record_attempt_failure`).
2026-10-04 21:23:10 +02:00
nvrl 2502af6e3c fix(fuse): a write opened as an upload lands takes the landed base (B170)
An upload that landed let go of its blob before the tree had its node. A
write open in between took the base it sampled at its start, the revision
the upload replaced, found none of its bytes cached, and queued an
incomplete revision; the next write open failed with EIO.

The drain keeps the landing node aside until the tree has it, and a write
open with no queued revision takes its base from that node, or from the
tree as it is when the handle goes in. The fake Drive can hold the drain
in that window (`hold_next_landing`).
2026-10-04 21:14:09 +02:00
nvrl 4757b957d2 fix(fuse): end a mirror pass at the stop (B169)
A stop that came while a mirror pass was uploading let the pass go on to
the next level of folders until the stop gave up on it and the runtime's
end cancelled the uploads under it. Once the daemon is stopping, a batch
starts none of its waiting ops, lets the ones sending finish, and ends
the pass; the remote walk and a new pass stop too.
2026-10-04 21:01:38 +02:00
nvrl 8e54ed0517 fix(fuse): do not send a file removed with its folder again (B168)
A delete of a file whose create was on the wire withdraws the create into
a trash. When its folder was removed too and that trash landed first,
Drive refused the create, and the drain re-homed the deleted file in the
root; only its missing bytes stopped it. The withdrawn create's trash
then read the trashed folder back into the cache before finding nothing
to trash.

A withdrawn create is no longer re-homed, and its trash is done once the
folder it was sent to is trashed or gone.
2026-10-04 20:42:04 +02:00
nvrl 9bbc994a4f perf(fuse): rebuild the hydrated listings in one pass (B167)
The hydrate rebuilt each listed folder's listing with a pass over every
entry. With 4,600 listed folders and 37,000 entries that took 1.7 s at
every start before the mount came up; one pass grouping the entries by
parent takes 3 ms.
2026-10-04 20:27:58 +02:00
nvrl ffa0fe698d test(fuse): a listing from before a create commits keeps the landed file (B166)
The account run's 0-byte read of a new file was B166 too: Drive's
listing had the node from before its create was committed, and the
queue no longer held it when read. The fake Drive can now hold a
folder's listing on its way back with one file listed unrevised.
2026-10-04 20:17:49 +02:00
nvrl 25c46cfe0a fix(fuse): list a file whose create lands while its folder is listed (B166)
A listing from Drive was overlaid with the queue as it was after Drive
answered. A create that landed in between was in neither, and a wide
folder on Wi-Fi dropped files that were never unlinked. The queue is now
read before Drive is asked too; a node that left it meanwhile is listed
as the DB has it.
2026-10-04 20:10:14 +02:00
nvrl 573a29e596 fix(fuse,cli): bound the stop's wait for busy threads (B165)
A stop joined every background thread for as long as each took. The sync
engine's mirror pass calls Drive with no deadline, and a stop that waited for
it was killed by systemd at TimeoutStopSec. The stop now waits up to 10 s for
the background threads and FUSE workers together, names any still running, and
the runtime shuts down within 2 s.
2026-10-04 19:12:19 +02:00
nvrl ce2bc0ac84 fix(fuse): list a new file once while its create lands (B164)
A listing from Drive overlays what is still queued for the folder. A create
Drive has made but whose answer is on its way, or which the drain is still
reading back, was in both: Drive's node under its new uid and the queue's
stand-in, under one name. A node queued in the folder now holds its name there.
2026-10-04 19:03:33 +02:00
nvrl b37d5e7928 fix(fuse): do not take a new file Drive lists before its commit as empty (B163) 2026-10-04 18:51:39 +02:00
nvrl 18587124ac fix(fuse): never adopt a node we trashed as an unanswered create's twin (B162)
A folder removed and made again met the old folder while its trash was on
the wire and adopted it, so the new folder's contents went into the old
one and were trashed with it. The twin search now skips hidden nodes, as
listings do.

The fake Drive can hold a trash (hold_next_trash).
2026-10-04 18:26:45 +02:00
nvrl cd18029944 fix(fuse): keep a move that lands while its create is read back (B161)
adopt_real_uid asked whether a move was queued only after reading the
node back. A move that landed in between left no op, and the answer still
had the old name, so the mount showed the file under it. Ask first.

The fake Drive can hold a read's answer (hold_answer_to_next_read) so the
test can land the move while the answer is on its way.
2026-10-04 18:20:56 +02:00
nvrl d6dd41407f fix(fuse): wake the drain when a folder is made (B160)
Queueing a folder's create left the workers asleep until their 30 s idle
poll; every other enqueue woke them. A file's create still waits for the
release that attaches its bytes.
2026-10-04 18:14:00 +02:00
nvrl 60445d3b26 fix(fuse): list again when a listing changed while it was published (B159)
A lookup in the root publishes the "Shared with me" entry only if the root
listing did not change while its plan was read, and answered EAGAIN when it
had. Right after a restart the root is listed again as the first lookups come
in, so a write could fail with "Resource temporarily unavailable".
ensure_children now takes the listing again, up to four times.
2026-10-04 17:50:39 +02:00
nvrl c2f93449ad fix(core): drop a rename replaced while its failure was being recorded (B158)
B141 drops a rename that fails with a newer one queued behind it. The
worker records the failure and releases its claim in two steps, and a
rename queued between them waited behind the old one, which was kept
with a backoff. Released, the old one was not due and the new one was,
so the new one landed first and the old one then put the first name
back.

Releasing the claim now drops the old rename too when a newer one is
queued, and both checks match the node by local id as well as by uid.
2026-10-04 17:23:51 +02:00
nvrl 4676a4e32b fix(fuse): cache an upload whose read-back failed, and know it as ours (B157)
When the read-back after an upload went unanswered, the drain let go of
the staged blob without caching it. A partial write opened next had
nothing to fill its gaps from and queued an incomplete revision, so the
write open after it was refused with EIO.

The bytes are now cached under the size and mtime the tree still has,
which is the base the next write takes. Their SHA-1 stands in for the
revision id the read-back did not bring, so that write chains onto the
upload and does not fork a conflict copy of itself.
2026-10-04 16:20:54 +02:00
nvrl 16fca10e6c perf(fuse): drain with 16 workers, and wake the idle ones when an op lands 2026-10-04 15:05:17 +02:00
nvrl dbc4d9a8c5 fix(core,fuse): a node written under its stand-in after landing updates its landed row 2026-10-04 13:38:54 +02:00
nvrl 9c93860cd8 refactor(core): rows and ops name a node's parent by local id; a landing rewrites nothing below it 2026-10-04 13:26:19 +02:00
nvrl 85ec91d6b3 fix(fuse): adopt a twin holding bytes an earlier attempt of the create sent (B156) 2026-10-04 13:14:29 +02:00
nvrl d5d3b2da3f refactor(core,fuse): find a queued op by its node's local id under either uid 2026-10-04 13:10:19 +02:00
nvrl e5f3f612f8 refactor(core,fuse): a node made here goes by its local id until it lands (schema 41)
A file or folder made on the mount used to get a minted `local~<ms>-<n>`
placeholder. It now gets `local~<lid>`, a function of the row it is given
first (`Db::add_local_row`), so the stand-in never changes and always leads
back to the row, and through it to the real uid once the node lands.
`Db::lid_of` reads it without a lookup.

Migration V41 rewrites the placeholders still stored to their row's
stand-in: the row and its node_json, its children, the ops on it or into
it (uid, parent, sidecar uid, a rename's original parent), and its pins.

Step 6a of the plan for 3.0.0 phase 3.
2026-10-04 12:32:20 +02:00
nvrl 3f74b95b34 fix(fuse): wait for a create on the wire that holds the name (B154); a write whose answer was lost is not a conflict (B155)
B154 is a regression from the B153 fix: after a failure on a held name the
op woke the ops waiting for it, but the database cannot see a create on the
wire, so the op was due again at once and spent its waits in milliseconds.
No wake while another create on the wire holds the name.

B155: a revision upload that reached Drive but lost its answer left the
file at a revision this daemon never heard of, and the retry kept the write
as a conflict copy of itself. A complete write whose bytes the remote holds,
by size and SHA-1, is sent again instead.

The fake Drive gains hold_reply_to_next_create and
lose_reply_to_next_revision for the two new sim tests.
2026-10-04 12:19:06 +02:00
nvrl 7269cb5118 fix(fuse): keep a rename made as its create lands (B152); wake an op once the name it waits for is free (B153)
Adoption asks again under the tree lock whether a rename is queued, and
writes the node row through the lock's outbox, so a rename made while
the create landed is no longer undone in the tree and the row.

An op that finds its name held fails as NameHeld: it backs off by the
minimum without doubling, and wakes the waiters for its name again once
its failure is recorded, so a trash that landed in between is not missed.
2026-10-04 06:04:20 +02:00
nvrl 6d75a56b00 fix(fuse,core): adopt a twin holding the create's bytes (B127); trash the node of a withdrawn create that lost its answer (B151)
A retried create now adopts the file its lost answer made when Drive's
SHA-1 matches the staged blob, not only when the file is empty. A delete
after a sent create keeps the op as a trash that finds and trashes the
node that create may have made; a new create of that name waits for it.
The simulation's settle now reads the log, so known damage is explained.
2026-10-04 05:41:57 +02:00
nvrl eb1c105bbf docs: record the full a6f36d5 account run in the 3.0.0 status 2026-10-04 04:45:48 +02:00
nvrl 85e80d8aac test(acceptance): retry only the sources a busy mirror refused; a mirror-to-mirror move carries an unsynced copy along 2026-10-04 04:45:48 +02:00
nvrl 23dc5c3723 fix(fuse): land a conflict copy in the root when its folder is gone (B150) 2026-10-04 04:45:48 +02:00
nvrl 50ae25bb88 fix(fuse): queue pdfs rename and move behind the mount's changes (B149)
With changes through the mount queued, `pdfs rename` and `pdfs move` went
straight to Drive ahead of a rename the mount still had queued, which then
landed and put its name back: the account run's B121 case on every on-demand
mount. They are now queued whenever the mount would queue, or the node has
anything queued, and rewrite a still-queued create's target.

The B113 acceptance case no longer requires creates to reach Drive at once;
under local-first they are queued and the moves rewrite them.
2026-10-04 04:45:48 +02:00
nvrl 9b2b9f24c5 feat(core): node rows name their parent by local id
Schema 40 adds nodes.parent_lid, backfilled from parent_uid. An upsert
and a new row's stub keep it in step, a folder's new row links the rows
stored before it, and a row merged into a landing folder hands its
children over. A restart places each node under its parent's row as it
is now, so a child whose stored node still names a landed folder's
placeholder lands under the folder's real uid.
2026-10-04 04:45:47 +02:00
nvrl ae6650f7e0 feat(core,fuse): queued ops name their node and folder by local id
Schema 39 adds pending_op.lid and parent_lid, backfilled from nodes. Every
op queued or retargeted gets both, and a create gets its row before it
is queued. The claim reads a parent's uid through parent_lid, and a
landing folder rewrites the ops made inside it by local id instead of by
its placeholder string. Ops queued against a row Drive listed follow it
when it is merged into the landing one.
2026-10-04 04:45:47 +02:00
nvrl f029e652a1 fix(core): keep a folder's pin when the folder lands (B148)
A folder pinned before its mkdir landed kept the placeholder uid in
pins.uid, which named no node once the landing moved the row to the
real uid. The landing now moves the pin with it.
2026-10-04 04:45:47 +02:00
nvrl 9b3651e4e8 fix(fuse): put a deadline on every remote call the drain makes
create_folder, trash_nodes, the twin listing and the uploads waited on
Drive without a limit, so one call that never answered held a drain
thread for good. They now give up after INTERACTIVE_CALL_TIMEOUT, and
uploads after that plus their size at 32 KiB/s; a missed deadline reads
as RequestTimeout, so the op backs off like any lost link.
2026-10-04 04:45:47 +02:00
nvrl 0dc476dfb7 test(sim): a steady Wi-Fi run and a drain-speed measurement
one_client_on_a_slow_link runs the Wi-Fi faults without outages and has
to drain within each settle. a_thousand_files_drain_as_fast_on_wifi_as_on_lan
times 100 folders and 1,000 files on LAN and Wi-Fi; it only prints for
now and runs when PDFS_SIM_MEASURE is set.
2026-10-04 04:45:47 +02:00
nvrl de258cbd60 ci(aur): retry the AUR host key scan
ssh-keyscan sometimes returns no key from the runner, which failed the
AUR publish for 2.8.0 and 2.10.0 at the fingerprint check. Retry the
scan up to five times with a longer timeout before checking the pin.
2026-10-04 03:43:30 +02:00
Nils Pukropp 4ea18e9187 chore(aur): update packages to 2.10.0 2026-10-04 01:42:24 +00:00
nvrl a6f36d56f3 feat(core,fuse,cli,gui): report changes Drive refuses as sync issues
A change Drive refuses because the storage is full, access was taken away,
its folder is gone, a folder limit is reached or the change was rejected
used to look like any other failure until six attempts had passed. The
drain now sorts the refusal into an issue and stores it on the queued op
(schema 38), so it needs attention at once in the app, tray and CLI. The
change stays queued and is retried; nothing is deleted.

The queue listing carries the issue and whether the op has file content.
ExportPendingOp copies that content to a new file: the Sync page offers it
as an Export button next to each issue, and the CLI as
`pdfs sync export <id> <dest>`. `pdfs sync issues` lists what needs
attention, and `pdfs sync queue` shows the issue per change.

docs/MILESTONE-3.0.0-STATUS.md tracks what is left for 3.0.0.
2026-10-04 03:18:57 +02:00
nvrl 77d0f254fc fix(fuse): never adopt a node another queued create is making
A file renamed away from a.txt while its create was on the wire, and a new
a.txt made after it, could both land on the first file's node: the second
create found a fresh, empty a.txt and took it for its own unanswered create.
The first file's rename then carried the second one's bytes away.

A create now leaves alone a node that another create is making under the
same name, and one the tree already knows. It waits for the name like for
any other queued change (docs/BUGS.md B147).
2026-10-04 03:15:50 +02:00
nvrl 4eccdeeda6 test(acceptance): retry a move while a mirror folder is busy syncing 2026-10-04 02:49:33 +02:00
nvrl 473842eefc fix(fuse): keep parked creates, renames and freed names from racing the drain
- B144: when Drive says a name is taken but nothing local holds it any more,
  the rename, move or create is sent once more under the wanted name before
  falling back to a conflict name. The trash that freed it could land between
  Drive's answer and the check.
- B145: the park sweep no longer drops a parked create whose node row is not
  written yet. It waits a minute, and keeps open files, before taking a
  missing row for a deleted file. The account run's B70 check failed with
  EBUSY on this.
- B146: a rename queued while an upload's read-back is on the wire is kept;
  the tree no longer takes back the name Drive still had.
2026-10-04 02:49:33 +02:00
nvrl 5e0f30cc2c feat(fuse,core): record namespace changes locally and send them from the queue
A new folder, a new file, a rename or a delete through the mount no longer
waits for Drive: it is recorded locally, the call returns at once, and the
drain sends it as it does offline. `"local_first": false` in config.json
brings back the old path for this release.

Fixes found by the simulation runs on the way:
- B129: a file deleted while its first upload is on the wire is trashed once
  the upload lands.
- B132, B140: files made in a new folder survive a restart; a landed folder
  hands its queued children over in the stored node too.
- B138: a create waits for the rename or trash that frees its name.
- B139: a file closed while its first upload lands keeps its last write.
- B141: a rename a worker holds stays queued, and the newer one waits behind
  it instead of racing it.
- B142: a rename of a file whose create lands meanwhile goes to the uid Drive
  gave it instead of answering EBUSY.
- B143: a create whose read-back fails as the link drops is adopted before
  the next op for it is sent, so its next write is not a conflict copy.
2026-10-04 01:50:38 +02:00
nvrl b3e33a7002 fix(core): never give out a pending_op id twice
A plain rowid is the largest one plus one, so a trash queued in the same
transaction that dropped the newest op took its id. The drain retires the
op it sent by id, so a rename landing deleted the trash queued over it and
the file stayed on Drive (B137). Schema 37 rebuilds pending_op with
AUTOINCREMENT, keeping every id and index.
2026-10-03 23:44:12 +02:00
161 changed files with 42154 additions and 17244 deletions

No files matched your search

+7 -1
View File
@@ -67,7 +67,13 @@ jobs:
User aur
EOF
# Pin the host key to the fingerprint published on aur.archlinux.org.
ssh-keyscan -t ed25519 aur.archlinux.org > "$HOME/.ssh/known_hosts"
# The scan sometimes returns nothing from the runner, so retry it.
for attempt in 1 2 3 4 5; do
ssh-keyscan -T 15 -t ed25519 aur.archlinux.org > "$HOME/.ssh/known_hosts" &&
[ -s "$HOME/.ssh/known_hosts" ] && break
echo "ssh-keyscan attempt $attempt returned no key; retrying"
sleep 10
done
ssh-keygen -lf "$HOME/.ssh/known_hosts" | grep -q 'SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4'
chown -R builder: "$HOME/.ssh"
chmod 600 "$HOME/.ssh/aur"
+14 -3
View File
@@ -45,7 +45,7 @@ jobs:
run: cargo fmt --all -- --check
- name: Lints
run: cargo clippy --workspace --all-targets -- -D warnings
run: cargo clippy --workspace --all-targets --locked -- -D warnings
- name: Unit tests
run: cargo test --workspace --locked
@@ -64,10 +64,11 @@ jobs:
# Real Proton credentials and /dev/fuse deliberately do not run on pull
# requests. Follow docs/TESTING.md on a dedicated test-account runner.
# The seeded simulation runs (docs/MILESTONE-3.0.0.md §8.2): real FUSE
# The seeded simulation runs (docs/TESTING.md): real FUSE
# daemons against an in-memory Drive, so they need /dev/fuse but no account.
# They run on pull requests too: most data-loss fixes are only exercised
# here, and a change that breaks one should not reach main first.
simulation:
if: github.event_name != 'pull_request'
runs-on: ubuntu-24.04
timeout-minutes: 30
@@ -101,6 +102,16 @@ jobs:
- name: Simulation runs
run: 'cargo test -p pdfs-fuse --lib --locked sim:: -- --ignored --test-threads=1'
# A hung step writes every daemon thread's stack next to the run's
# state; without them a hang that does not replay cannot be told apart.
- name: Keep the stacks of a hung run
if: failure()
uses: actions/upload-artifact@v7
with:
name: simulation-stacks
path: /tmp/pdfs-sim-*/stacks.txt
if-no-files-found: ignore
# Keeps the release-profile dependencies cached on main, where the release
# workflow can restore them; a tag cannot reuse the caches of earlier tags.
release-cache:
+66
View File
@@ -0,0 +1,66 @@
name: COPR
on:
# Called by release.yml once the GitHub release and its assets exist: COPR
# downloads the tag archive and the vendored crates from the release.
workflow_call:
inputs:
version:
required: true
type: string
secrets:
COPR_CONFIG:
required: true
workflow_dispatch:
inputs:
version:
description: 'Release to build, such as 3.5.0. It must already have its vendor tarball.'
required: true
type: string
permissions:
contents: read
concurrency:
group: copr
cancel-in-progress: false
jobs:
build:
name: Build in COPR
runs-on: ubuntu-24.04
container: fedora:latest
env:
VERSION: ${{ inputs.version }}
COPR_PROJECT: narl-rs/proton-drive-linux
steps:
- name: Install packaging tools
run: dnf -y install copr-cli rpm-build rpmdevtools
- name: Checkout repository
uses: actions/checkout@v7
- name: Configure COPR access
env:
COPR_CONFIG: ${{ secrets.COPR_CONFIG }}
run: |
install -d -m700 ~/.config
printf '%s\n' "$COPR_CONFIG" > ~/.config/copr
chmod 600 ~/.config/copr
# COPR rebuilds the source package without our --define, so the version
# has to be written into the spec itself.
- name: Build the source package
run: |
version="${VERSION#v}"
spec=packaging/proton-drive-linux.spec
sed -i -E "s/^(%\{!\?version: %global version )[^}]*\}/\1${version}}/" "$spec"
grep -qF "%global version ${version}}" "$spec"
mkdir -p rpmbuild/SOURCES
spectool -g -C rpmbuild/SOURCES "$spec"
rpmbuild -bs --define "_topdir $PWD/rpmbuild" "$spec"
# Waits for the build, so a failed COPR build fails this job.
- name: Submit to COPR
run: copr-cli build "$COPR_PROJECT" rpmbuild/SRPMS/*.src.rpm
+59
View File
@@ -0,0 +1,59 @@
name: Nix
# Builds the flake (docs/INSTALL.md#nixos-flake) whenever its recipe, the lock
# files or the code change, so a NixOS user's `nix build` keeps working. Run it
# from the Actions tab for any branch to get a build to try.
on:
push:
paths:
- 'flake.nix'
- 'flake.lock'
- 'packaging/nix/**'
- 'Cargo.lock'
- 'Cargo.toml'
- 'crates/**'
- 'po/**'
- '.github/workflows/nix.yml'
pull_request:
paths:
- 'flake.nix'
- 'flake.lock'
- 'packaging/nix/**'
- 'Cargo.lock'
- '.github/workflows/nix.yml'
workflow_dispatch:
permissions:
contents: read
concurrency:
group: nix-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
- uses: cachix/install-nix-action@v31
# Pulls from the project's binary cache, and pushes what this run built
# when the token is there; a fork's pull request has none and only pulls.
- uses: cachix/cachix-action@v17
with:
name: proton-drive-linux
authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}
- name: Check the flake
run: nix flake check --print-build-logs --no-build
- name: Build the package
run: nix build --print-build-logs .#proton-drive-linux
- name: Smoke test
run: |
./result/bin/pdfs --version
for bin in pdfs-app pdfs-tray pdfs-prompt; do test -x "result/bin/$bin"; done
grep -q "ExecStart=$(readlink -f result)/bin/pdfs" result/lib/systemd/user/proton-drive.service
ls result/share/locale/*/LC_MESSAGES/pdfs.mo
+70 -1
View File
@@ -52,8 +52,10 @@ jobs:
tag_version="${GITHUB_REF_NAME#v}"
manifest_version="$(sed -n 's/^version = "\([^"]*\)"/\1/p' Cargo.toml | head -n1)"
pkgbuild_version="$(sed -n 's/^pkgver=//p' packaging/PKGBUILD | head -n1)"
spec_version="$(sed -n 's/^%{!?version: %global version \([^}]*\)}/\1/p' packaging/proton-drive-linux.spec)"
test "$tag_version" = "$manifest_version"
test "$tag_version" = "$pkgbuild_version"
test "$tag_version" = "$spec_version"
- name: Release quality gates
run: |
@@ -129,6 +131,12 @@ jobs:
# Create tarball
tar -czf proton-drive-linux-${{ env.VERSION }}-x86_64.tar.gz -C dist/binaries .
# Every crate the build needs, so COPR can build the spec without network.
- name: Package vendored crates (.tar.xz)
run: |
cargo vendor --locked dist/vendor > /dev/null
tar -cJf proton-drive-linux-vendor-${{ env.VERSION }}.tar.xz -C dist vendor
- name: Package Debian package (.deb)
run: |
mkdir -p deb/DEBIAN
@@ -153,6 +161,9 @@ jobs:
cp LICENSE deb/usr/share/licenses/proton-drive-linux/
po/build.sh deb/usr/share/locale
# Stops the running daemon before removal, so the mount stops mirroring deletes
install -m755 packaging/deb/prerm deb/DEBIAN/prerm
# Calculate Installed-Size in KB for the control file
INSTALLED_SIZE=$(du -sk deb | cut -f1)
@@ -227,27 +238,75 @@ jobs:
path: |
*.deb
*.tar.gz
*.tar.xz
*.rpm
if-no-files-found: error
# The flake is the Nix release: a tag is what `github:narrrl/proton-drive-linux/vX.Y.Z`
# pins, so it has to build before the release is announced.
nix:
name: Build the Nix package
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
- uses: cachix/install-nix-action@v31
# Pulls from the project's binary cache, and pushes what this run built
# when the token is there; a fork's pull request has none and only pulls.
- uses: cachix/cachix-action@v17
with:
name: proton-drive-linux
authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}
- name: Build the flake
run: nix build --print-build-logs .#proton-drive-linux
- name: Check the version
run: |
built="$(./result/bin/pdfs --version)"
echo "$built"
case "$GITHUB_REF" in
refs/tags/v*) grep -qF "${GITHUB_REF_NAME#v}" <<< "$built" ;;
esac
publish:
name: Create GitHub Release
needs: [checks, build-release]
needs: [checks, build-release, nix]
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v7
- name: Download packages
uses: actions/download-artifact@v8
with:
name: proton-drive-linux-builds
# The release notes are this version's section of the changelog, from its
# heading up to the next one.
- name: Extract release notes
run: |
version="${GITHUB_REF_NAME#v}"
awk -v heading="## [$version]" \
'index($0, heading) == 1 { found = 1; print; next }
found && /^## \[/ { exit }
found' docs/CHANGELOG.md > release-notes.md
if [ ! -s release-notes.md ]; then
echo "::error::docs/CHANGELOG.md has no section for $version"
exit 1
fi
- name: Create GitHub Release
uses: softprops/action-gh-release@v3
with:
body_path: release-notes.md
files: |
*.deb
*.tar.gz
*.tar.xz
*.rpm
draft: false
prerelease: false
@@ -263,3 +322,13 @@ jobs:
version: ${{ github.ref_name }}
secrets:
AUR_SSH_PRIVATE_KEY: ${{ secrets.AUR_SSH_PRIVATE_KEY }}
copr:
name: Build COPR packages
needs: publish
if: startsWith(github.ref, 'refs/tags/')
uses: ./.github/workflows/copr.yml
with:
version: ${{ github.ref_name }}
secrets:
COPR_CONFIG: ${{ secrets.COPR_CONFIG }}
+121
View File
@@ -0,0 +1,121 @@
name: Test build
# On-demand Fedora RPM of any branch, for someone to try a fix before it ships.
# Run it from the Actions tab ("Run workflow", pick the branch). Built inside a
# Fedora container so the binaries link against Fedora's own GTK, libadwaita and
# WebKitGTK, not the Ubuntu ones the release RPM is built against.
#
# The RPM is unsigned and versioned `<Cargo version>-1.test.<sha>`, so it sorts
# below the release it precedes and `dnf` replaces it when that ships.
on:
workflow_dispatch:
inputs:
fedora:
description: 'Fedora release to build on'
required: false
default: '44'
type: string
prerelease:
description: 'Also publish a pre-release anyone can download without a GitHub login'
required: false
default: false
type: boolean
permissions:
contents: read
concurrency:
group: test-build-${{ github.ref }}
cancel-in-progress: true
jobs:
rpm:
name: Fedora ${{ inputs.fedora }} RPM
runs-on: ubuntu-24.04
container: fedora:${{ inputs.fedora }}
steps:
- name: Install build dependencies
run: |
dnf install -y --setopt=install_weak_deps=False \
git rpm-build gcc gcc-c++ make gettext curl \
pkgconf-pkg-config fuse3-devel gtk4-devel libadwaita-devel \
libsecret-devel dbus-devel glib2-devel webkitgtk6.0-devel
- name: Checkout repository
uses: actions/checkout@v7
# Fedora's packaged Rust can trail the workspace's minimum (1.96).
- name: Set up Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
cache: false
- name: Cache Cargo target
uses: swatinem/rust-cache@v2
with:
shared-key: fedora-test-build
save-if: false
- name: Build release binaries
run: cargo build --release --locked
- name: Package RPM
run: |
git config --global --add safe.directory "$GITHUB_WORKSPACE"
VERSION=$(grep -m1 '^version' Cargo.toml | cut -d'"' -f2)
SHORT=${GITHUB_SHA::7}
mkdir -p rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS,BUILDROOT,db}
rpmbuild -bb \
--nodeps \
--define "_topdir $PWD/rpmbuild" \
--define "_sourcedir $PWD" \
--define "git_dir $PWD" \
--define "prebuilt 1" \
--define "version ${VERSION}" \
--define "dist .test.${SHORT}" \
--define "_dbpath $PWD/rpmbuild/db" \
packaging/proton-drive-linux.spec
mkdir -p out
cp rpmbuild/RPMS/x86_64/*.rpm out/
# The bare binaries, for trying one without installing anything.
tar -czf "out/proton-drive-linux-${VERSION}-${SHORT}-fedora${{ inputs.fedora }}-x86_64.tar.gz" \
-C target/release pdfs pdfs-tray pdfs-app pdfs-prompt
ls -l out
- name: Upload artifact
uses: actions/upload-artifact@v7
with:
name: proton-drive-linux-fedora${{ inputs.fedora }}-${{ github.sha }}
path: out/*
retention-days: 14
if-no-files-found: error
prerelease:
name: Publish pre-release
needs: rpm
if: inputs.prerelease
runs-on: ubuntu-24.04
permissions:
contents: write
steps:
- name: Download build
uses: actions/download-artifact@v8
with:
path: out
merge-multiple: true
# One rolling pre-release per branch: a new run replaces the last one, so
# the link a tester was given keeps pointing at the newest build.
- name: Publish
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
run: |
TAG="test-$(echo "${GITHUB_REF_NAME}" | tr -c 'A-Za-z0-9._\n' '-')"
gh release delete "$TAG" --cleanup-tag --yes 2>/dev/null || true
gh release create "$TAG" out/* \
--target "$GITHUB_SHA" \
--prerelease \
--title "Test build of ${GITHUB_REF_NAME} (${GITHUB_SHA::7})" \
--notes "Unsigned test build of branch \`${GITHUB_REF_NAME}\` at ${GITHUB_SHA}. Not a release; replaced by the next test build of this branch."
+2
View File
@@ -3,3 +3,5 @@
*.pdb
.env
/rpmbuild
/result
/result-*
+3 -3
View File
@@ -39,9 +39,9 @@ cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only
```
Changes to the FUSE layer, sync, or the pending-operation queue should also be exercised against a
real mount before review — see [`docs/TESTING.md`](docs/TESTING.md). Live runs are destructive:
use a dedicated test account, never one holding data you cannot lose.
Changes to the FUSE layer, sync, or the queue should also pass the simulation runs and be
exercised against a real mount before review — see [`docs/TESTING.md`](docs/TESTING.md). Live
runs are destructive: use a dedicated test account, never one holding data you cannot lose.
## Making changes
Generated
+15 -10
View File
@@ -3474,7 +3474,7 @@ checksum = "2ee67f1008b1ba2321834326597b8e186293b049a023cdef258527550b9935b4"
[[package]]
name = "pdfs-cli"
version = "2.10.0"
version = "3.5.0"
dependencies = [
"anyhow",
"clap",
@@ -3485,16 +3485,19 @@ dependencies = [
"tokio",
"tracing",
"tracing-subscriber",
"zeroize",
]
[[package]]
name = "pdfs-core"
version = "2.10.0"
version = "3.5.0"
dependencies = [
"async-trait",
"base64 0.23.1",
"dbus-secret-service-keyring-store",
"directories",
"ignore",
"kamadak-exif",
"keyring-core",
"libc",
"parking_lot",
@@ -3508,12 +3511,13 @@ dependencies = [
"thiserror 2.0.20",
"tokio",
"tracing",
"zeroize",
"zip",
]
[[package]]
name = "pdfs-fuse"
version = "2.10.0"
version = "3.5.0"
dependencies = [
"async-trait",
"base64 0.23.1",
@@ -3535,7 +3539,7 @@ dependencies = [
[[package]]
name = "pdfs-gui"
version = "2.10.0"
version = "3.5.0"
dependencies = [
"anyhow",
"async-channel",
@@ -3572,9 +3576,9 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]]
name = "pgp"
version = "0.20.0"
version = "0.21.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1cfa4743b28656065ff4c0ba09e46b357a65e8c00fc2341e89084b82f87cbdf1"
checksum = "ae70f4d9325a391db30d115d6191a7bb67cec856aa0bba131290f0c4e09a532a"
dependencies = [
"aead 0.5.2",
"aes 0.8.4",
@@ -3775,9 +3779,9 @@ dependencies = [
[[package]]
name = "proton-drive-rs"
version = "0.7.2"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6ce113d234474821d34787436ee80e1458e3d8a8f97212219615ed315eb70be"
checksum = "9e6b87f78711e8fc72737c8eb24ebda156079308d79ab37ca2b1b32de6812880"
dependencies = [
"async-trait",
"base64 0.23.1",
@@ -3802,9 +3806,9 @@ dependencies = [
[[package]]
name = "proton-sdk"
version = "0.7.2"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a33c67df9694f90ce60e5b34984275956244138511b4640ca0d100e1dac8b3d"
checksum = "19f1936755c603b98de1c82a5b343c6232e9fea8adaa77139f7cb8fcddcda697"
dependencies = [
"aes-gcm 0.11.1",
"async-trait",
@@ -3831,6 +3835,7 @@ dependencies = [
"thiserror 2.0.20",
"tokio",
"tracing",
"zeroize",
]
[[package]]
+16 -3
View File
@@ -3,7 +3,7 @@ resolver = "2"
members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"]
[workspace.package]
version = "2.10.0"
version = "3.5.0"
edition = "2024"
license = "MIT"
authors = ["Nils Pukropp <contact@narl.io>"]
@@ -15,8 +15,8 @@ keywords = ["proton", "drive", "fuse", "sync", "linux"]
categories = ["filesystem", "command-line-utilities"]
[workspace.dependencies]
proton-sdk = { version = "0.7.2" }
proton-drive-rs = { version = "0.7.2" }
proton-sdk = { version = "0.9.1" }
proton-drive-rs = { version = "0.9.1" }
# Internal crates.
pdfs-core = { path = "crates/pdfs-core" }
@@ -26,6 +26,7 @@ tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "fs",
serde = { version = "1", features = ["derive"] }
serde_json = "1"
base64 = "0.23"
zeroize = "1"
thiserror = "2"
anyhow = "1"
# Non-poisoning mutexes. A handler that panics while holding a lock must not
@@ -60,3 +61,15 @@ async-trait = "0.1"
futures = "0.3"
fuser = "0.18"
libc = "0.2"
[profile.release]
# No package ships a separate debug package (the PKGBUILD sets `!debug`, the
# RPM spec sets `debug_package` to nil), and the unstripped binaries weighed
# 43 MB (`pdfs`), 24 MB (`pdfs-app`) and 12 MB (`pdfs-tray`).
strip = "symbols"
# Off on purpose: LTO has broken GTK links for this workspace, which is why
# the packages turn it off as well.
lto = false
# `Workers` and the thumbnail build catch panics to keep the daemon alive, so
# unwinding must stay on.
panic = "unwind"
+19 -6
View File
@@ -15,6 +15,7 @@ scriptable CLI.
[![AUR](https://img.shields.io/aur/version/proton-drive-for-linux)](https://aur.archlinux.org/packages/proton-drive-for-linux)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust 1.96+](https://img.shields.io/badge/rust-1.96%2B-orange.svg)](https://www.rust-lang.org)
[![Buy me a coffee](https://img.shields.io/badge/Buy%20me%20a%20coffee-FFDD00?logo=buymeacoffee&logoColor=black)](https://buymeacoffee.com/narl)
<img src="images/files.png" alt="My files in the Proton Drive app" width="820">
@@ -38,8 +39,9 @@ scriptable CLI.
- **Synced folders.** Back up folders such as `~/Documents` under your computer in Proton Drive,
as a full two-way copy or online only. Gitignore-style `.pdfsignore` rules, conflict copies
instead of overwrites, pause and bandwidth limits.
- **Safe writes.** Changes are staged on disk and queued before they are acknowledged, so they
survive a lost connection, a crash or a reboot and upload when they can.
- **Works at local speed.** Making, renaming and deleting files and folders returns at once, online
or offline. Changes are staged on disk and queued before they are acknowledged, so they survive
a lost connection, a crash or a reboot and reach Drive when they can.
**Desktop app**
@@ -50,7 +52,8 @@ scriptable CLI.
- **Sharing.** Invite people as viewer, editor or admin, create public links with password and
expiry, accept invitations.
- **Sync overview.** What is uploading, what is waiting, what failed and why, and a history of
what happened.
what happened. A change Drive refuses (storage full, access taken away) shows at once, on the
file too, and can be exported or discarded.
- **Tray icon** with status, pause and quick actions, and a **search launcher** (`pdfs-prompt`)
that searches Proton Drive and your home folder together, as its own window, through fuzzel,
rofi or wofi, or live in `fzf`.
@@ -71,13 +74,15 @@ scriptable CLI.
| Distribution | Package |
|---|---|
| Debian, Ubuntu 24.04 or newer | `.deb` from the [latest release](https://github.com/narrrl/proton-drive-linux/releases/latest) |
| Fedora 44 or newer | `.rpm` from the [latest release](https://github.com/narrrl/proton-drive-linux/releases/latest) |
| Fedora 44 or newer | [`narl-rs/proton-drive-linux`](https://copr.fedorainfracloud.org/coprs/narl-rs/proton-drive-linux/) in COPR, or the `.rpm` from the [latest release](https://github.com/narrrl/proton-drive-linux/releases/latest) |
| Arch Linux | [`proton-drive-for-linux`](https://aur.archlinux.org/packages/proton-drive-for-linux) (also `-bin` and `-git`) in the AUR |
| NixOS | The flake in this repository: see [Install → NixOS](docs/INSTALL.md#nixos-flake) |
| Anything else | [Build from source](docs/INSTALL.md#build-from-source) |
```bash
sudo apt install ./proton-drive-linux_*.deb # Debian, Ubuntu
sudo dnf install ./proton-drive-linux-*.rpm # Fedora
sudo dnf copr enable narl-rs/proton-drive-linux # Fedora
sudo dnf install proton-drive-linux
paru -S proton-drive-for-linux # Arch
```
@@ -105,6 +110,7 @@ video thumbnails), headless use and uninstalling.
| [Configuration](docs/CONFIGURATION.md) | `config.json`, file locations, environment variables, the service |
| [Troubleshooting](docs/TROUBLESHOOTING.md) | Diagnosing problems and fixing common ones |
| [Recovering on a new computer](docs/RECOVERY.md) | Getting your synced folders back after a reinstall or a lost machine |
| [Security model](docs/SECURITY_MODEL.md) | What protects your files, what stays on this computer, what the client connects to |
| [Changelog](docs/CHANGELOG.md) | What changed in each release |
For contributors: [development guide](docs/DEVELOPMENT.md), [architecture](docs/ARCHITECTURE.md),
@@ -146,7 +152,7 @@ keys before it leaves the machine, through a Rust implementation of Proton's Dri
Files you open are decrypted into a local cache so programs can read them. What that means for
the security of the machine is spelled out in the
[threat model](docs/ARCHITECTURE.md#8-threat-model-what-this-client-writes-to-disk-in-plaintext).
[threat model](docs/ARCHITECTURE.md#10-threat-model-what-this-client-writes-to-disk-in-plaintext).
## Contributing
@@ -154,6 +160,13 @@ Bug reports, translations and patches are welcome. See [CONTRIBUTING.md](CONTRIB
workflow and the checks CI runs, and [SECURITY.md](SECURITY.md) to report a vulnerability
privately.
## Support
Proton Drive Linux is free and built in spare time. If it saves you a few headaches, you can
[buy me a coffee](https://buymeacoffee.com/narl).
<a href="https://buymeacoffee.com/narl"><img src="https://img.shields.io/badge/Buy%20me%20a%20coffee-FFDD00?logo=buymeacoffee&logoColor=black" alt="Buy me a coffee"></a>
## License
Released under the [MIT License](LICENSE).
+4 -1
View File
@@ -4,6 +4,9 @@
Fixes land on the latest release. Please reproduce on the most recent version before reporting.
For what the client protects and what it does not, written for users, see the
[security model](docs/SECURITY_MODEL.md).
## Reporting a vulnerability
**Do not open a public issue.** Report privately through GitHub's
@@ -31,7 +34,7 @@ the interesting boundaries are:
or the wrong key material to Proton.
What the client deliberately keeps in plaintext on disk, and why, is documented in the
[threat model](docs/ARCHITECTURE.md#8-threat-model-what-this-client-writes-to-disk-in-plaintext).
[threat model](docs/ARCHITECTURE.md#10-threat-model-what-this-client-writes-to-disk-in-plaintext).
Behaviour described there is known and not a vulnerability by itself; a way to make it worse
than described is.
+1
View File
@@ -23,3 +23,4 @@ tracing-subscriber.workspace = true
clap.workspace = true
rpassword.workspace = true
serde_json.workspace = true
zeroize.workspace = true
+552 -52
View File
@@ -14,8 +14,8 @@ use pdfs_core::cache::ContentCache;
use pdfs_core::config::AppDirs;
use pdfs_core::control::{
ConflictKeep, ErrorKind, MountKind, RefreshScope, Request as CtlRequest,
Response as CtlResponse, RestoreItem as CtlRestoreItem, ShareEntryKind, SyncPhase,
pending_summary,
Response as CtlResponse, RestoreItem as CtlRestoreItem, ShareEntryKind, SyncIssue, SyncPhase,
daemon_outdated, pending_summary,
};
use pdfs_core::db::Db;
use pdfs_core::service;
@@ -135,6 +135,24 @@ enum Command {
/// Account email; prompted if omitted.
#[arg(short, long)]
username: Option<String>,
/// For a two-password account, keep the mailbox password's unlock in the
/// system keyring so the daemon starts without asking after every login.
/// Without it the unlock lives in kernel memory for this session only.
#[arg(long)]
remember: bool,
},
/// Enter the mailbox password of a two-password account so the daemon can
/// unlock your files. Needed once per login session.
Unlock {
/// Keep the unlock in the system keyring instead of kernel memory only.
#[arg(long)]
remember: bool,
},
/// Forget the unlock held in kernel memory; the account is locked again.
Lock {
/// Also remove an unlock kept in the keyring with `--remember`.
#[arg(long)]
forget: bool,
},
/// Forget the stored session.
Logout,
@@ -154,17 +172,19 @@ enum Command {
/// Mountpoint; defaults to ~/ProtonDrive.
mountpoint: Option<PathBuf>,
},
/// Keep a file on this device (download + cache it). Needs a running mount.
/// Keep a file or folder on this device (download + cache it). A folder's
/// files download in the background; `pdfs transfers` shows the progress.
/// Needs a running mount.
Pin {
/// File path, inside the mountpoint or relative to it.
/// File or folder path, inside the mountpoint or relative to it.
path: PathBuf,
},
/// Stop keeping a file on this device, evicting its cached content.
/// Stop keeping a file or folder on this device, evicting its cached content.
Unpin {
/// File path, inside the mountpoint or relative to it.
/// File or folder path, inside the mountpoint or relative to it.
path: PathBuf,
},
/// List files pinned to this device.
/// List files and folders pinned to this device.
Pins,
/// List a directory via the running daemon (in-app browser backend).
Ls {
@@ -610,6 +630,25 @@ enum DeviceCmd {
#[arg(long, conflicts_with = "uid")]
clear: bool,
},
/// Sync the same folders as another computer: this machine continues that
/// computer's device alongside it, and its folders are offered for syncing
/// here. Both computers keep them in step.
Join {
/// Device uid (from `devices list`).
uid: String,
/// Skip the confirmation and accept every proposed local path.
#[arg(long)]
yes: bool,
},
/// Move another computer's folders into this computer's backup and sync
/// them here. The other computer stays registered, without those folders.
Migrate {
/// Device uid (from `devices list`).
uid: String,
/// Skip the confirmation and accept every proposed local path.
#[arg(long)]
yes: bool,
},
}
#[derive(Subcommand)]
@@ -677,6 +716,25 @@ enum SyncCmd {
},
/// List uploads and changes not yet on Proton Drive.
Queue,
/// List queued changes Proton Drive refused or that keep failing, and what to do.
Issues,
/// Save the content of a queued upload (by id from `sync queue`) to a file.
Export {
/// Queue id.
id: i64,
/// Where to save it: a new file, or an existing folder to save it in.
dest: PathBuf,
},
/// Drop a queued change (by id from `sync queue`) and undo it here: a new file
/// or folder goes, anything else goes back to what Proton Drive has. Its
/// content is lost unless saved with `sync export` first.
Discard {
/// Queue id.
id: i64,
/// Do not ask first.
#[arg(long)]
yes: bool,
},
/// Retry a queued operation now (by id from `sync queue`), or every failed one.
Retry {
/// Queue id; omit to retry every failed operation.
@@ -783,8 +841,24 @@ fn main() -> Result<()> {
let cli = Cli::parse();
JSON_OUTPUT.store(cli.json, std::sync::atomic::Ordering::Relaxed);
// The commands that hold the password, the key passphrases or the session
// tokens in memory: keep them out of core dumps and away from same-user
// ptrace. Only the daemon did, though `mount` holds the same keys for as
// long as it runs.
if matches!(
cli.command,
Command::Login { .. }
| Command::Unlock { .. }
| Command::Logout
| Command::Mount { .. }
| Command::Daemon { .. }
) {
pdfs_core::kernelkey::forbid_dumps();
}
match cli.command {
Command::Login { username } => cmd_login(username),
Command::Login { username, remember } => cmd_login(username, remember),
Command::Unlock { remember } => cmd_unlock(remember),
Command::Lock { forget } => cmd_lock(forget),
Command::Logout => cmd_logout(),
Command::Status => cmd_status(),
Command::Locations => cmd_locations(),
@@ -928,6 +1002,44 @@ fn cmd_devices(action: DeviceCmd) -> Result<()> {
let uid = if clear { None } else { uid };
ok_or_bail(control_request(CtlRequest::AdoptDevice { uid })?)?
}
DeviceCmd::Join { uid, yes } => {
if !yes {
println!(
"This computer syncs into that device alongside it. Its folders are kept\n\
in step on both computers: an edit or a deletion on one reaches the other.\n\
New synced folders here go under that device too. Folders already synced\n\
here stay where they are."
);
confirm("Sync with it?")?;
}
let response = control_request(CtlRequest::AdoptDevice { uid: Some(uid) })?;
// A script gets the one reply and runs `sync restore` itself.
if emit_json(&response)? {
return Ok(());
}
ok_or_bail(response)?;
// The device is this machine's now, so the plain restore offers its
// folders.
cmd_sync_restore(yes, None)?
}
DeviceCmd::Migrate { uid, yes } => {
if !yes {
println!(
"Every folder that device backs up moves into this computer's backup.\n\
The device stays registered, without those folders. Stop Proton Drive\n\
on that computer first, or it sees its folders disappear."
);
confirm("Move them here?")?;
}
let response = control_request(CtlRequest::MigrateDevice { uid })?;
// A script gets the one reply and runs `sync restore` itself.
if emit_json(&response)? {
return Ok(());
}
ok_or_bail(response)?;
// The folders are this device's now, so the plain restore offers them.
cmd_sync_restore(yes, None)?
}
}
Ok(())
}
@@ -1028,7 +1140,17 @@ fn cmd_sync(action: SyncCmd) -> Result<()> {
})?)?
}
SyncCmd::Queue => return cmd_sync_queue(),
SyncCmd::Issues => return cmd_sync_issues(),
SyncCmd::Export { id, dest } => return cmd_sync_export(id, &dest),
SyncCmd::Limit { up, down } => return cmd_sync_limit(up, down),
SyncCmd::Discard { id, yes } => {
if !yes {
confirm(&format!(
"Discard queued change {id} and undo it here? Content not exported is lost."
))?;
}
ok_or_bail(control_request(CtlRequest::DiscardPendingOp { id })?)?
}
SyncCmd::Retry { id } => ok_or_bail(control_request(CtlRequest::RetryPendingOp { id })?)?,
SyncCmd::Resume { folder: None } => {
ok_or_bail(control_request(CtlRequest::SetSyncPaused {
@@ -1106,6 +1228,9 @@ fn cmd_sync_queue() -> Result<()> {
(false, _) => "due now".to_string(),
};
println!("[{}] {} {} ({state})", op.id, op.kind, op.path);
if let Some(issue) = op.issue {
println!(" needs attention: {}", issue_text(issue));
}
if let Some(error) = &op.last_error {
println!(" failed {} time(s): {error}", op.attempts);
}
@@ -1117,6 +1242,91 @@ fn cmd_sync_queue() -> Result<()> {
Ok(())
}
/// `pdfs sync issues`: the queued changes that need the user, and what to do.
fn cmd_sync_issues() -> Result<()> {
let response = control_request(CtlRequest::ListPendingOps)?;
let items = match response {
CtlResponse::PendingOps { items } => items,
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
};
let issues: Vec<_> = items.into_iter().filter(|op| op.failing).collect();
if json_enabled() {
println!("{}", serde_json::to_string_pretty(&issues)?);
return Ok(());
}
if issues.is_empty() {
println!("No sync issues.");
return Ok(());
}
for op in issues {
println!("[{}] {} {}", op.id, op.kind, op.path);
match op.issue {
Some(issue) => println!(" {}", issue_text(issue)),
None => println!(" keeps failing; it is retried."),
}
if let Some(error) = &op.last_error {
println!(" last error: {error}");
}
if op.exportable {
println!(
" save a copy: pdfs sync export {} <file or folder>",
op.id
);
}
}
Ok(())
}
/// What a sync issue means, and what the user can do about it.
fn issue_text(issue: SyncIssue) -> &'static str {
match issue {
SyncIssue::Quota => {
"Proton Drive storage is full. Free up space or upgrade; the change is retried."
}
SyncIssue::Access => {
"you can no longer change this on Proton Drive, for example because a share was \
revoked. Save a copy to keep the content."
}
SyncIssue::Missing => {
"the file or its folder is gone from Proton Drive. Save a copy to keep the content."
}
SyncIssue::Limit => {
"the folder holds too many items or is nested too deep. Move the file to another \
folder."
}
SyncIssue::Rejected => {
"Proton Drive rejected the change, for example the name. Rename the file."
}
SyncIssue::Unknown => "Proton Drive refused the change.",
}
}
/// `pdfs sync export`: save a queued upload's content. A folder as `dest`
/// gets the file under its own name.
fn cmd_sync_export(id: i64, dest: &Path) -> Result<()> {
let mut dest =
std::path::absolute(dest).with_context(|| format!("resolving {}", dest.display()))?;
if dest.is_dir() {
let items = match control_request(CtlRequest::ListPendingOps)? {
CtlResponse::PendingOps { items } => items,
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
};
let Some(op) = items.iter().find(|op| op.id == id) else {
bail!("no queued change {id}; see `pdfs sync queue`");
};
let name = Path::new(&op.path)
.file_name()
.with_context(|| format!("{} has no file name", op.path))?;
dest.push(name);
}
ok_or_bail(control_request(CtlRequest::ExportPendingOp {
id,
dest: dest.display().to_string(),
})?)
}
/// `pdfs sync restore`: list what this machine's device holds, confirm where
/// each folder should land, then hand the mapping to the daemon.
///
@@ -1664,7 +1874,7 @@ fn prompt_line(label: &str) -> Result<String> {
Ok(line.trim().to_owned())
}
fn cmd_login(username: Option<String>) -> Result<()> {
fn cmd_login(username: Option<String>, remember: bool) -> Result<()> {
let username = match username {
Some(u) => u,
None => prompt_line("Proton email")?,
@@ -1676,20 +1886,47 @@ fn cmd_login(username: Option<String>) -> Result<()> {
prompt_line("2FA code").map_err(|e| pdfs_core::Error::Other(format!("read 2FA code: {e}")))
};
// Only a two-password account is ever asked for this.
let get_mailbox = |attempt: u32| prompt_mailbox(attempt, remember);
// Proton may ask for a CAPTCHA first, usually on a VPN. The CLI has no
// browser engine, so the user solves the page in their own browser and says
// when they are done; the retry then carries the challenge token.
let get_hv = |hv: pdfs_core::proton_sdk::api::HumanVerification| {
println!(
"Proton is asking for a CAPTCHA before it will accept this sign-in.\n\
Open this page in your browser and complete it:\n\n {}\n",
auth::browser_verification_url(&hv)
);
prompt_line("Press Enter once the page says you are verified")
.map_err(|e| pdfs_core::Error::Other(format!("read verification answer: {e}")))?;
Ok(auth::browser_verified(&hv))
};
let rt = tokio::runtime::Runtime::new()?;
// Solving a CAPTCHA needs a browser engine, which the CLI has no business
// carrying. Point at the app that does rather than failing with the raw API
// message, which reads as "your login is broken".
rt.block_on(auth::login(&username, &password, get_totp))
.map_err(|e| match e {
pdfs_core::Error::HumanVerificationRequired(_) => anyhow::anyhow!(
"Proton is asking for a CAPTCHA before it will accept this sign-in.\n\
Run `pdfs-app` and sign in there — it can show the verification page.\n\
(This is usually triggered by a VPN or an unfamiliar IP; signing in from \
your usual network often avoids it.)"
),
other => anyhow::Error::new(other).context("login failed"),
})?;
rt.block_on(auth::login_interactive(
&username,
&password,
get_totp,
get_mailbox,
get_hv,
))
.map_err(|e| {
// Proton words this block as "update the app", which the user can't
// act on: the release is current and only waiting helps.
let blocked =
matches!(&e, pdfs_core::Error::Proton(p) if pdfs_core::error::sign_in_blocked(p));
let e = anyhow::Error::new(e).context("login failed");
if blocked {
e.context(
"Proton is temporarily refusing sign-in from this device (code 2028). \
Your version is not the problem. Wait several hours, up to a day, \
without trying again, then sign in.",
)
} else {
e
}
})?;
// A daemon that is already up is parked in `wait_for_session`, sleeping
// between keyring checks; restart it so the mount appears now instead of
@@ -1701,12 +1938,97 @@ fn cmd_login(username: Option<String>) -> Result<()> {
}
println!("Logged in as {username}. Session stored in the system keyring.");
report_key_state();
Ok(())
}
/// Ask for the mailbox password of a two-password account, without echo.
fn prompt_mailbox(attempt: u32, remember: bool) -> pdfs_core::Result<auth::Mailbox> {
if attempt == 0 {
eprintln!("This account has a separate mailbox password.");
} else {
eprintln!("That mailbox password did not unlock the account. Try again.");
}
let password = rpassword::prompt_password("Mailbox password: ")
.map_err(|e| pdfs_core::Error::Other(format!("read mailbox password: {e}")))?;
Ok(auth::Mailbox {
password: zeroize::Zeroizing::new(password),
remember,
})
}
/// Say where the account's keys are held, and what that means for the next login.
fn report_key_state() {
match auth::key_state() {
Ok(auth::KeyState::Session) => println!(
"Unlocked for this login session only. After you log out or reboot, run `pdfs unlock`."
),
Ok(auth::KeyState::Locked) => println!("The account is locked. Run `pdfs unlock`."),
_ => {}
}
}
fn cmd_unlock(remember: bool) -> Result<()> {
auth::load().context("not logged in (run `pdfs login` first)")?;
let rt = tokio::runtime::Runtime::new()?;
// Bounded like the login prompt: three wrong answers end the command.
let mut attempt = 0;
loop {
let mailbox = prompt_mailbox(attempt, remember)?;
match rt.block_on(auth::unlock(&mailbox.password, remember)) {
Ok(()) => break,
Err(pdfs_core::Error::WrongMailboxPassword) if attempt < 2 => attempt += 1,
Err(e) => return Err(anyhow::Error::new(e).context("unlock failed")),
}
}
// A daemon parked on the locked account would otherwise notice on its next
// poll; restarting it makes the mount appear now, as after `pdfs login`.
if service::is_active() {
service::restart();
}
println!("Unlocked.");
report_key_state();
Ok(())
}
fn cmd_lock(forget: bool) -> Result<()> {
auth::lock(forget)?;
println!(
"Locked. {}",
if forget {
"Unlock removed from memory and keyring."
} else {
"Run `pdfs unlock` to unlock again."
}
);
// The running daemon keeps its keys until it restarts; say so, since
// "locked" would otherwise suggest the mount stopped.
if service::is_active() {
println!(
"The running daemon keeps its keys until it restarts: `systemctl --user restart proton-drive`."
);
}
Ok(())
}
/// `stored`, `session` or `locked`: where the account's keys are held.
fn key_state_name() -> &'static str {
match auth::key_state() {
Ok(auth::KeyState::Stored) => "stored",
Ok(auth::KeyState::Session) => "session",
_ => "locked",
}
}
fn cmd_logout() -> Result<()> {
auth::logout()?;
println!("Stored session cleared.");
if auth::logout()? {
println!("Signed out: the session is ended on Proton and cleared here.");
} else {
println!(
"Stored session cleared. Proton could not be reached to end the session; \
it stays valid until it expires or is signed out in your account's security settings."
);
}
Ok(())
}
@@ -1731,6 +2053,7 @@ fn cmd_status() -> Result<()> {
"username": session.as_ref().map(|s| s.username.clone()),
"user_id": session.as_ref().map(|s| s.user_id.clone()),
"scopes": session.as_ref().map(|s| s.scopes.clone()),
"keys": session.as_ref().map(|_| key_state_name()),
"mount": mount,
});
println!("{}", serde_json::to_string_pretty(&out)?);
@@ -1741,6 +2064,11 @@ fn cmd_status() -> Result<()> {
Ok(s) => {
println!("Logged in as {} (user {})", s.username, s.user_id);
println!("Scopes: {}", s.scopes.join(", "));
match key_state_name() {
"locked" => println!("Keys: locked — run `pdfs unlock`"),
"session" => println!("Keys: unlocked for this login session"),
_ => {}
}
}
Err(pdfs_core::Error::NotLoggedIn) => {
println!("Not logged in. Run `pdfs login`.");
@@ -1763,9 +2091,13 @@ fn cmd_status() -> Result<()> {
staged_oldest_secs,
paused,
paused_until,
session_expired,
tokens_unsaved,
version,
..
}) => {
let state = match (online, paused) {
_ if session_expired => ", session expired: run `pdfs login`",
(_, true) => ", sync paused",
(false, false) => ", offline",
(true, false) => "",
@@ -1798,6 +2130,14 @@ fn cmd_status() -> Result<()> {
failing_error.as_deref().unwrap_or("no error recorded"),
);
}
if tokens_unsaved {
println!(
"Keyring refuses the refreshed sign-in; unlock it, or a reboot needs `pdfs login`"
);
}
if daemon_outdated(&version) {
println!("Daemon {}", outdated_daemon_note(&version));
}
}
Ok(other) => println!("Mount: unexpected response {other:?}"),
Err(_) => println!("Mount: not running."),
@@ -1805,6 +2145,20 @@ fn cmd_status() -> Result<()> {
Ok(())
}
/// What `pdfs status` and `pdfs diagnose` say about a daemon still running
/// an older release than this CLI.
fn outdated_daemon_note(version: &str) -> String {
let running = if version.is_empty() {
"an older release".to_string()
} else {
version.to_string()
};
format!(
"still runs {running}, this is {}; run `systemctl --user restart proton-drive` to finish the update",
env!("CARGO_PKG_VERSION"),
)
}
fn cmd_locations() -> Result<()> {
let response = control_request(CtlRequest::ListLocations)?;
if emit_json(&response)? {
@@ -1872,7 +2226,7 @@ fn login_poll_delay(attempts: u32) -> std::time::Duration {
std::time::Duration::from_secs(BASE_SECS.saturating_mul(factor).min(MAX_SECS))
}
/// Block until a session is stored, backing off between checks.
/// Block until a session is stored and unlocked, backing off between checks.
///
/// `load` and `sleep` are parameters rather than direct calls so the backoff
/// schedule can be tested without a keyring and without real time passing.
@@ -1897,6 +2251,16 @@ where
sleep(delay);
attempts = attempts.saturating_add(1);
}
Err(pdfs_core::Error::Locked) => {
let delay = login_poll_delay(attempts);
if attempts == 0 {
tracing::info!("account is locked; waiting for `pdfs unlock`…");
} else {
tracing::debug!(delay_secs = delay.as_secs(), "account still locked");
}
sleep(delay);
attempts = attempts.saturating_add(1);
}
Err(e) => return Err(e.into()),
}
}
@@ -1910,22 +2274,25 @@ fn cmd_daemon(mountpoint: Option<PathBuf>) -> Result<()> {
loop {
// Wait until a session is stored. The GUI enables this service on login,
// but the service may also start at boot before the user has logged in.
wait_for_session(|| auth::load().map(|_| ()), std::thread::sleep)?;
wait_for_session(auth::ready, pdfs_fuse::idle_sleep)?;
match mount_once(mountpoint.clone()) {
Ok(pdfs_fuse::MountOutcome::Shutdown) => {
tracing::info!("daemon stopping");
return Ok(());
}
Ok(pdfs_fuse::MountOutcome::ResumeStored) => {
tracing::info!("resuming the session stored in the keyring");
}
Ok(pdfs_fuse::MountOutcome::Unmounted) => {
tracing::warn!("mount ended externally; remounting in 2s");
std::thread::sleep(std::time::Duration::from_secs(2));
pdfs_fuse::idle_sleep(std::time::Duration::from_secs(2));
}
Err(e) => {
// `{:#}` so the whole anyhow chain lands in the log — the top
// context alone ("mount failed") never says *why*.
tracing::error!(error = format!("{e:#}"), "mount failed; retrying in 5s");
std::thread::sleep(std::time::Duration::from_secs(5));
pdfs_fuse::idle_sleep(std::time::Duration::from_secs(5));
}
}
}
@@ -1983,13 +2350,22 @@ fn mount_once(mountpoint: Option<PathBuf>) -> Result<pdfs_fuse::MountOutcome> {
sweep_mode: config.resolved_conflict_sweep(),
upload_limit: config.upload_limit.unwrap_or(0),
download_limit: config.download_limit.unwrap_or(0),
local_first: config.local_first.unwrap_or(true),
},
);
// Clean unmount check (as a best-effort final backup).
if let Err(e) = rt.block_on(auth::persist(&session)) {
tracing::debug!(error = %e, "no new tokens to persist on shutdown");
// Last chance for a rotation the keyring refused when it happened. Only
// that: by now the keyring may hold a newer login, the one this stop was
// restarted for, and the tokens of this session would overwrite it.
if let Err(e) = auth::persist(&session) {
tracing::error!(
error = %e,
"could not save the refreshed session tokens; the next start may need `pdfs login`"
);
}
// Dropping the runtime waits for every blocking task, however long; a call
// still stuck on Drive would hold the stop until systemd kills it (B165).
rt.shutdown_timeout(std::time::Duration::from_secs(2));
let outcome = result.context("mount failed")?;
Ok(outcome)
@@ -2263,7 +2639,10 @@ fn cmd_import_google_photos(archives: Vec<PathBuf>, dry_run: bool, wait: bool) -
} => continue,
CtlResponse::ImportStatus { summary, .. } => {
print_import_summary(summary.as_ref());
return Ok(());
return finished_run(
"the import",
summary.map(|s| (s.failed, s.cancelled, s.error)),
);
}
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
@@ -2306,7 +2685,10 @@ fn cmd_redate_photos(
CtlResponse::RedateStatus { running: true, .. } => continue,
CtlResponse::RedateStatus { summary, .. } => {
print_redate_summary(summary.as_ref());
return Ok(());
return finished_run(
"the re-date",
summary.map(|s| (s.failed, s.cancelled, s.error)),
);
}
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
@@ -2314,6 +2696,22 @@ fn cmd_redate_photos(
}
}
/// How a run `--wait` followed ended, as the exit status: an error when it
/// failed, was cancelled, or left photos failed, so a script can tell.
/// `outcome` is `(failed, cancelled, error)`; `None` when the daemon has no
/// result, which happens when it restarted during the run.
fn finished_run(what: &str, outcome: Option<(usize, bool, Option<String>)>) -> Result<()> {
match outcome {
None => bail!("{what} ended without a result; the service may have restarted"),
Some((_, _, Some(error))) => bail!("{what} failed: {error}"),
Some((_, true, None)) => bail!("{what} was cancelled"),
Some((failed, false, None)) if failed > 0 => {
bail!("{what} finished, but {failed} photo(s) failed")
}
Some(_) => Ok(()),
}
}
/// A `YYYY-MM-DD` day as epoch seconds at UTC midnight. Reuses the Takeout
/// scanner's date reader, which already accepts exactly this spelling.
fn parse_day(day: &str) -> Result<i64> {
@@ -2349,6 +2747,10 @@ fn print_redate_summary(summary: Option<&pdfs_core::control::RedateSummary>) {
println!("No re-date has run yet.");
return;
};
if let Some(error) = &summary.error {
println!("Failed : {error}");
return;
}
println!("Photos examined : {}", summary.examined);
println!("Mis-dated : {}", summary.candidates);
if summary.dry_run {
@@ -2408,6 +2810,10 @@ fn print_import_summary(summary: Option<&pdfs_core::control::ImportSummary>) {
println!("No import has run yet.");
return;
};
if let Some(error) = &summary.error {
println!("Failed : {error}");
return;
}
println!("Photos in export : {}", summary.found);
println!("Uploaded : {}", summary.uploaded);
println!("Already on Proton: {}", summary.duplicates);
@@ -3082,12 +3488,25 @@ fn cmd_diagnose() -> Result<()> {
}
let db_path = dirs.db_path();
let db_size = std::fs::metadata(&db_path).map(|m| m.len()).unwrap_or(0);
report.check(
" database",
db_path.is_file(),
format!("{} ({})", db_path.display(), human_bytes(db_size)),
);
if db_path.exists() {
let db_size = std::fs::metadata(&db_path).map(|m| m.len()).unwrap_or(0);
report.check(
" database",
db_path.is_file(),
format!("{} ({})", db_path.display(), human_bytes(db_size)),
);
} else {
// The daemon creates it on its first start, which waits for a sign-in,
// so a fresh install has none yet (issue #39 read this as the fault).
report.finding(
DiagnoseLevel::Warn,
" database",
format!(
"{} not created yet (the daemon creates it on first start)",
db_path.display()
),
);
}
println!("\nFUSE");
let fuse = Path::new("/dev/fuse");
@@ -3103,6 +3522,19 @@ fn cmd_diagnose() -> Result<()> {
match pdfs_core::auth::load() {
Ok(session) => {
report.finding(DiagnoseLevel::Ok, " keyring session", session.username);
match key_state_name() {
"locked" => report.finding(
DiagnoseLevel::Fail,
" account keys",
"locked — run `pdfs unlock` (the daemon waits for it)",
),
"session" => report.finding(
DiagnoseLevel::Ok,
" account keys",
"unlocked for this login session",
),
_ => report.finding(DiagnoseLevel::Ok, " account keys", "stored in the keyring"),
}
}
Err(e) => {
// A locked keyring and an absent login look the same from here, so
@@ -3133,11 +3565,21 @@ fn cmd_diagnose() -> Result<()> {
Ok(CtlResponse::Status {
mountpoint,
online,
session_expired,
tokens_unsaved,
pending_uploads,
pending_changes,
version,
..
}) => {
report.finding(DiagnoseLevel::Ok, " daemon responding", "");
if daemon_outdated(&version) {
report.finding(
DiagnoseLevel::Warn,
" daemon version",
outdated_daemon_note(&version),
);
}
let live_type = mounted_fs_type(Path::new(&mountpoint));
let is_fuse = live_type
.as_deref()
@@ -3154,19 +3596,38 @@ fn cmd_diagnose() -> Result<()> {
|kind| format!("{mountpoint} ({kind})"),
),
);
report.finding(
if online {
DiagnoseLevel::Ok
} else {
DiagnoseLevel::Warn
},
" network",
if online {
"online"
} else {
"offline; cached data remains available"
},
);
// The keyring check above read the blob, which still holds the
// tokens the daemon could not replace.
if tokens_unsaved {
report.finding(
DiagnoseLevel::Warn,
" session tokens",
"the keyring refuses the refreshed ones; unlock it, or a reboot needs `pdfs login`",
);
}
// A refused session reads as offline too, but no connection will
// bring it back, and the keyring check above cannot tell.
if session_expired {
report.finding(
DiagnoseLevel::Fail,
" session",
"Proton refused it (expired or revoked) — run `pdfs login`",
);
} else {
report.finding(
if online {
DiagnoseLevel::Ok
} else {
DiagnoseLevel::Warn
},
" network",
if online {
"online"
} else {
"offline; cached data remains available"
},
);
}
let pending = pending_uploads + pending_changes;
report.finding(
if pending == 0 {
@@ -3367,6 +3828,28 @@ mod login_wait_tests {
assert_eq!(login_poll_delay(u32::MAX), Duration::from_secs(15));
}
/// The file-manager script passes each selected path after `--`, so a
/// file named like an option is still the path.
#[test]
fn a_path_after_a_double_dash_is_a_path_even_when_it_looks_like_a_flag() {
let cli = Cli::try_parse_from(["pdfs", "pin", "--", "--json"]).unwrap();
assert!(!cli.json);
match cli.command {
Command::Pin { path } => assert_eq!(path, PathBuf::from("--json")),
_ => panic!("not a pin"),
}
}
#[test]
fn a_waited_for_run_that_did_not_fully_succeed_exits_with_an_error() {
assert!(finished_run("the import", Some((0, false, None))).is_ok());
assert!(finished_run("the import", Some((3, false, None))).is_err());
assert!(finished_run("the import", Some((0, true, None))).is_err());
let failed = finished_run("the import", Some((0, false, Some("not a zip".into()))));
assert!(failed.unwrap_err().to_string().contains("not a zip"));
assert!(finished_run("the import", None).is_err());
}
#[test]
fn waiting_for_a_session_stops_hammering_the_keyring() {
let mut checks_left = 20;
@@ -3401,6 +3884,21 @@ mod login_wait_tests {
);
}
#[test]
fn a_locked_account_is_waited_on_until_it_is_unlocked() {
let mut checks = vec![
Ok(()),
Err(pdfs_core::Error::Locked),
Err(pdfs_core::Error::NotLoggedIn),
Err(pdfs_core::Error::Locked),
];
let mut slept = 0;
wait_for_session(|| checks.pop().unwrap(), |_| slept += 1).unwrap();
assert_eq!(slept, 3, "locked and signed-out are both waited out");
}
#[test]
fn a_stored_session_is_not_waited_for_at_all() {
let mut slept = 0;
@@ -3424,6 +3922,7 @@ mod login_wait_tests {
size: 1,
modified: 0,
pinned: false,
pinned_by_folder: false,
cached: false,
uid: "vol~link".into(),
path: String::new(),
@@ -3433,6 +3932,7 @@ mod login_wait_tests {
shared_by_unverified: false,
trashed_at: 0,
trashed_from: None,
issue: None,
};
assert_eq!(shared_by_line(&entry), None, "no invitation, no line");
entry.shared_by = "alice@proton.me".into();
+3
View File
@@ -28,3 +28,6 @@ dbus-secret-service-keyring-store.workspace = true
parking_lot.workspace = true
async-trait.workspace = true
libc.workspace = true
zeroize.workspace = true
base64.workspace = true
exif = { version = "0.6", package = "kamadak-exif" }
+1072 -123
View File
File diff suppressed because it is too large. Load diff
+46 -43
View File
@@ -91,7 +91,7 @@ const TOUCH_FLUSH_AT: usize = 256;
///
/// Public because it is the *contract* a block has to satisfy, not a detail of
/// this cache: the network read path validates against it too, so a block that
/// comes back short is refused before it can truncate a file (bugs.md B84).
/// comes back short is refused before it can truncate a file (`docs/BUGS.md` B84).
pub fn block_len(size: u64, idx: u64) -> u64 {
let start = idx.saturating_mul(BLOCK_SIZE);
size.saturating_sub(start).min(BLOCK_SIZE)
@@ -311,8 +311,7 @@ struct PinFile {
pins: BTreeMap<String, Pin>,
}
/// What a staged file actually contains, written beside it as `<name>.json`
/// (offline.md Phase 2).
/// What a staged file actually contains, written beside it as `<name>.json`.
///
/// A staged file is **not** necessarily valid whole-file content, and that is
/// the whole reason this exists. A partial overwrite commits by filling the
@@ -439,9 +438,9 @@ pub struct ContentCache {
/// Subdirectory for write-handle scratch files (disk-backed write buffers).
/// Emptied on open so a crashed run leaves no orphans.
scratch_dir: PathBuf,
/// Subdirectory holding the bytes of writes that have not been uploaded yet
/// (offline.md Phase 2/3) — every released write passes through here, since
/// the upload is a queued op performed later. Unlike `scratch_dir` this is
/// Subdirectory holding the bytes of writes that have not been uploaded yet.
/// Every released write passes through here, since the upload is a queued
/// op performed later. Unlike `scratch_dir` this is
/// **never** emptied on open: these are the only copy of content the user
/// authored, and the whole point is that they outlive the daemon.
staging_dir: PathBuf,
@@ -476,7 +475,7 @@ pub struct ContentCache {
thumb_bytes: AtomicU64,
/// Unified metadata DB. Its `cache_entries` table is the LRU index: every
/// store/read/evict updates it, and the budget enforcers query it instead of
/// scanning the cache directories (plan.md P4).
/// scanning the cache directories.
db: Arc<Db>,
/// LRU touches not yet written to `cache_entries`, keyed by cache key.
///
@@ -845,6 +844,9 @@ impl ContentCache {
}
let bytes = std::fs::metadata(&tmp)?.len();
std::fs::rename(&tmp, &blob)?;
// Linked from the blob already cached, the temp name was a second name
// of it, and the rename did nothing (`docs/BUGS.md` B187).
let _ = std::fs::remove_file(&tmp);
let meta = serde_json::to_vec(&Meta { mtime, size })?;
std::fs::write(self.meta_path(uid), meta)?;
self.db
@@ -986,7 +988,7 @@ impl ContentCache {
/// a miss ([`cached_block`](Self::cached_block) checks), so a bad block
/// cannot escape this cache even though this does not check. Validating a
/// block is the *reader's* job, at the point it is fetched — see
/// `Core::read_block` (bugs.md B84).
/// `Core::read_block` (`docs/BUGS.md` B84).
pub fn store_block(
&self,
uid: &NodeUid,
@@ -1488,9 +1490,8 @@ impl ContentCache {
/// This is what makes a write survive its upload: the caller is releasing a
/// write handle and would otherwise delete the file, so until the bytes are
/// on the remote, staging holds the only copy. Every dirty handle goes
/// through here — the upload is a queued op performed later (offline.md
/// Phase 3), and a staged file is also what a human can recover from if the
/// queue never drains.
/// through here — the upload is a queued op performed later, and a staged
/// file is also what a human can recover from if the queue never drains.
///
/// Falls back to a copy when the rename crosses a filesystem boundary; a
/// failure here means we could not save the bytes at all, so it is reported
@@ -2197,6 +2198,24 @@ mod tests {
assert_eq!(c.read_range(&u, 200, 5, 0, 5).unwrap(), b"newer");
}
/// A landed create and the revision after it can adopt the same staged
/// blob. The link to the temp name is then a second name of the cached
/// blob, and a rename between two names of one file does nothing: the
/// temp name stayed until the next start swept it (`docs/BUGS.md` B187).
#[test]
fn store_file_twice_from_the_same_blob_leaves_no_temp_file() {
let (c, d) = cache();
let u = uid("a");
let src = d.path().join("staged-blob");
std::fs::write(&src, b"staged").unwrap();
c.store_file(&u, 100, 6, &src).unwrap();
c.store_file(&u, 100, 6, &src).unwrap();
assert!(!c.blob_path(&u).with_extension("tmp").exists());
assert_eq!(c.read_range(&u, 100, 6, 0, 6).unwrap(), b"staged");
}
#[test]
fn reconcile_rebuilds_index_from_disk() {
let dir = TempDir::new();
@@ -2290,42 +2309,26 @@ mod tests {
/// call also needs.
#[test]
fn budget_check_is_free_when_under_budget() {
use std::time::Instant;
// Cap far above what we store, so every store stays under budget.
let (c, _d) = cache_capped(1 << 30);
let payload = vec![0u8; 4096];
// Cost of a store against a near-empty index...
let t = Instant::now();
for i in 0..500u64 {
for i in 0..8u64 {
c.store_block(&uid("a"), 1, 1 << 30, blk(1 << 30, i), &payload)
.unwrap();
}
let empty = t.elapsed() / 500;
// ...and against one with a few thousand entries in it.
for i in 500..2500u64 {
c.store_block(&uid("a"), 1, 1 << 30, blk(1 << 30, i), &payload)
.unwrap();
}
let t = Instant::now();
for i in 2500..3000u64 {
c.store_block(&uid("a"), 1, 1 << 30, blk(1 << 30, i), &payload)
.unwrap();
}
let full = t.elapsed() / 500;
// Plant an index row far over the cap behind the running total's back,
// and older than every real block. Any budget check that consults the
// index would see the pool over budget, evict, and reseed the total.
c.db.cache_touch("planted", KIND_BLOCK, 1 << 40, 0).unwrap();
println!("B4: store_block under budget — {empty:?} at ~0 entries, {full:?} at 2500");
// A ratio, not a wall-clock bound: the claim is that the under-budget
// path does not grow with the index, and only a ratio tests that. An
// absolute threshold measures the machine's load instead, and this test
// runs alongside the rest of the suite.
assert!(
full < empty * 3,
"under-budget store should not scale with the index: \
{empty:?} at ~0 entries vs {full:?} at 2500"
);
c.store_block(&uid("a"), 1, 1 << 30, blk(1 << 30, 8), &payload)
.unwrap();
// Not a timing ratio: that measured the CI runner's load and flaked.
assert_eq!(c.block_bytes.load(Ordering::Relaxed), 9 * 4096);
let entries = c.db.cache_entries_by_kind(KIND_BLOCK).unwrap();
assert_eq!(entries.len(), 10, "under-budget store evicted: {entries:?}");
}
/// The running totals are a fast path, not a second source of truth: after
@@ -2686,7 +2689,7 @@ mod tests {
}
/// The geometry the whole read path validates against, here and over the
/// network (bugs.md B84): full blocks, then whatever is left.
/// network (`docs/BUGS.md` B84): full blocks, then whatever is left.
#[test]
fn block_len_is_full_blocks_then_the_remainder() {
let size = BLOCK_SIZE * 2 + 17;
@@ -2766,7 +2769,7 @@ mod tests {
/// A cached block is addressed by the range it covers, not by its index
/// alone. Re-planning a file onto its real geometry must therefore miss the
/// blocks the uniform fallback cached, rather than serve their bytes at the
/// wrong offset (bugs.md B85).
/// wrong offset (`docs/BUGS.md` B85).
#[test]
fn a_cached_block_is_not_served_to_a_different_geometry() {
let (c, _d) = cache();
@@ -2833,7 +2836,7 @@ mod tests {
/// A block of the wrong length cannot escape the cache even though the
/// store does not check: the read validates it and reports a miss, so the
/// next read refetches rather than serving a hole (bugs.md B84).
/// next read refetches rather than serving a hole (`docs/BUGS.md` B84).
#[test]
fn a_block_of_the_wrong_length_reads_as_a_miss() {
let (c, _d) = cache();
@@ -2898,7 +2901,7 @@ mod tests {
);
}
/// A failed upload must never cost the user their bytes (offline.md Phase 2):
/// A failed upload must never cost the user their bytes:
/// the scratch file moves to staging intact, and staging survives a reopen
/// the way scratch deliberately does not.
#[test]
+324 -39
View File
@@ -3,7 +3,7 @@
use std::path::{Path, PathBuf};
use directories::ProjectDirs;
use serde::{Deserialize, Serialize};
use serde::{Deserialize, Deserializer, Serialize};
use crate::error::{Error, Result};
use crate::menu::PromptConfig;
@@ -12,8 +12,19 @@ use crate::syncignore::DEFAULT_IGNORE_PATTERNS;
/// Proton requires `external-drive-{name}@{semver}-{channel}` (channel ∈
/// stable/beta/alpha); a malformed value trips the 422 anti-abuse path.
pub const APP_VERSION: &str = "external-drive-linux@0.1.0-alpha";
pub const USER_AGENT: &str = "proton-drive-linux/0.1.0";
/// Follows the release so Proton can tell releases apart and retire old ones
/// without retiring every one.
pub const APP_VERSION: &str = concat!(
"external-drive-linux@",
env!("CARGO_PKG_VERSION"),
"-stable"
);
pub const USER_AGENT: &str = concat!("proton-drive-linux/", env!("CARGO_PKG_VERSION"));
/// The fixed identity every release up to 3.1.0 wrote into `config.json`. Read
/// back as "unset" so those configs follow the release instead of pinning it.
const LEGACY_APP_VERSION: &str = "external-drive-linux@0.1.0-alpha";
const LEGACY_USER_AGENT: &str = "proton-drive-linux/0.1.0";
/// Keyring service name; one entry per credential kind keyed by username.
pub const KEYRING_SERVICE: &str = "proton-drive-linux";
@@ -76,10 +87,24 @@ fn resolve_sweep_mode(env: Option<&str>, stored: Option<SweepMode>) -> SweepMode
}
/// Configuration structure allowing the user to customize client identification headers.
#[derive(Serialize, Deserialize, Debug, Clone)]
#[derive(Serialize, Deserialize, Debug, Clone, Default)]
pub struct AppConfig {
pub app_version: String,
pub user_agent: String,
/// Override for [`APP_VERSION`]. `None` means "use the release's own";
/// the old fixed default reads as `None`, so it leaves the file on the
/// next save.
#[serde(
default,
deserialize_with = "identity_override",
skip_serializing_if = "Option::is_none"
)]
pub app_version: Option<String>,
/// Override for [`USER_AGENT`], handled like `app_version`.
#[serde(
default,
deserialize_with = "identity_override",
skip_serializing_if = "Option::is_none"
)]
pub user_agent: Option<String>,
/// Soft cap on the on-disk content cache, in bytes (`0` = unlimited).
/// `None` means "use [`DEFAULT_CACHE_BUDGET_BYTES`]"; the Settings page
/// writes an explicit value here. Defaulted for configs predating the field.
@@ -138,6 +163,10 @@ pub struct AppConfig {
/// Defaulted for configs predating the field.
#[serde(default)]
pub files_view: FilesView,
/// The desktop app's main window size, remembered between runs. Defaulted
/// for configs predating the field.
#[serde(default)]
pub window: WindowState,
/// The user hid the tray icon from its own menu. The tray exits at start
/// while this is set, and the app does not spawn it; Preferences turns it
/// back on. Defaulted for configs predating the field.
@@ -154,6 +183,31 @@ pub struct AppConfig {
/// configs predating the field.
#[serde(default)]
pub online_map: bool,
/// Record a new folder, a new file, a rename or a delete through the mount
/// locally and send it from the queue, instead of waiting for Drive in the
/// syscall. `None` means on. `false` brings back the old path for one
/// release (docs/ROADMAP.md). Defaulted for configs predating
/// the field.
#[serde(default)]
pub local_first: Option<bool>,
/// Ask GitHub once a day whether a newer release is out, and say so in the
/// desktop app. Off by default: GitHub then sees the address and version
/// asking. Never used where a package manager installs updates (see
/// [`crate::update::updates_managed`]). Defaulted for configs predating the
/// field.
#[serde(default)]
pub update_check: bool,
/// When the update check last asked GitHub, in Unix seconds, so a restart
/// does not ask again within [`crate::update::CHECK_INTERVAL`]. Defaulted
/// for configs predating the field.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub update_last_checked: Option<i64>,
/// The latest release that check found, such as `3.6.0`, so the app can
/// say so again after a restart without asking. Older than or equal to the
/// running version once the update is installed. Defaulted for configs
/// predating the field.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub update_latest: Option<String>,
}
/// The My Files page's view choices, remembered between runs.
@@ -179,6 +233,44 @@ impl Default for FilesView {
}
}
/// The desktop app's main window, as it was when it last closed.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
#[serde(default)]
pub struct WindowState {
/// The unmaximised size, in logical pixels.
pub width: i32,
pub height: i32,
pub maximized: bool,
}
impl WindowState {
/// Below this the saved size is taken for damage and the default is used.
pub const MIN_WIDTH: i32 = 480;
pub const MIN_HEIGHT: i32 = 400;
/// This state, with a size too small to use replaced by the default.
pub fn usable(self) -> Self {
if self.width < Self::MIN_WIDTH || self.height < Self::MIN_HEIGHT {
Self {
maximized: self.maximized,
..Self::default()
}
} else {
self
}
}
}
impl Default for WindowState {
fn default() -> Self {
Self {
width: 980,
height: 680,
maximized: false,
}
}
}
/// The key a My Files listing is ordered by.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq, Default)]
#[serde(rename_all = "lowercase")]
@@ -209,30 +301,28 @@ impl FileSort {
}
}
impl Default for AppConfig {
fn default() -> Self {
Self {
app_version: APP_VERSION.to_string(),
user_agent: USER_AGENT.to_string(),
cache_budget: None,
upload_limit: None,
download_limit: None,
mountpoint: None,
ignore_patterns: None,
device_uid: None,
conflict_sweep: None,
open_with: None,
prompt: None,
proton_theme: None,
files_view: FilesView::default(),
tray_hidden: false,
language: None,
online_map: false,
}
}
/// Reads an `app_version` or `user_agent` override, dropping the fixed
/// identity older releases stored as the default.
fn identity_override<'de, D: Deserializer<'de>>(
deserializer: D,
) -> std::result::Result<Option<String>, D::Error> {
let value = Option::<String>::deserialize(deserializer)?;
Ok(value.filter(|v| v != LEGACY_APP_VERSION && v != LEGACY_USER_AGENT))
}
impl AppConfig {
/// The `x-pm-appversion` sent to Proton: the user's override, or
/// [`APP_VERSION`].
pub fn resolved_app_version(&self) -> &str {
self.app_version.as_deref().unwrap_or(APP_VERSION)
}
/// The `User-Agent` sent to Proton: the user's override, or
/// [`USER_AGENT`].
pub fn resolved_user_agent(&self) -> &str {
self.user_agent.as_deref().unwrap_or(USER_AGENT)
}
/// The effective cache budget in bytes: the user's explicit choice, or
/// [`DEFAULT_CACHE_BUDGET_BYTES`] when unset.
pub fn resolved_cache_budget(&self) -> u64 {
@@ -280,6 +370,22 @@ impl AppConfig {
}
}
/// Fail when the config file at `path` exists but does not parse, so that
/// [`AppDirs::save_config`] never writes over settings it could not read.
fn refuse_to_replace_unreadable(path: &Path) -> Result<()> {
match std::fs::read_to_string(path) {
Ok(content) => match serde_json::from_str::<AppConfig>(&content) {
Ok(_) => Ok(()),
Err(e) => Err(Error::Other(format!(
"{} cannot be read ({e}); fix it or move it aside before changing settings",
path.display()
))),
},
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(e) => Err(e.into()),
}
}
/// Resolved XDG locations for state, cache, and the default mountpoint.
pub struct AppDirs {
dirs: ProjectDirs,
@@ -311,6 +417,7 @@ impl AppDirs {
// Preserve the original bytes for diagnosis/recovery. In
// particular, never replace a truncated config with defaults
// and silently lose mount, device, or ignore settings.
// `save_config` refuses to write over it for the same reason.
tracing::error!(path = %path.display(), error = %e, "config is unreadable; using in-memory defaults without overwriting it");
return AppConfig::default();
}
@@ -318,9 +425,8 @@ impl AppDirs {
}
let default_config = AppConfig::default();
if let Ok(content) = serde_json::to_string_pretty(&default_config) {
let _ = std::fs::create_dir_all(self.dirs.config_dir());
let _ = std::fs::write(&path, content);
if let Err(e) = self.save_config(&default_config) {
tracing::warn!(path = %path.display(), error = %e, "could not write the default config");
}
default_config
}
@@ -328,8 +434,15 @@ impl AppDirs {
/// Persist `config` to [`config_path`](Self::config_path), creating the
/// config dir if missing. The Settings page calls this after editing the
/// cache budget or mountpoint; the daemon re-reads the file on its next mount.
///
/// Refuses while the file on disk does not parse. Every caller loads,
/// changes one field and saves, and [`load_config`](Self::load_config)
/// gives defaults for a file it cannot read, so the first change after
/// the file was damaged wrote defaults over the mountpoint, the device and
/// the ignore rules the user had (`docs/BUGS.md` B223).
pub fn save_config(&self, config: &AppConfig) -> Result<()> {
let path = self.config_path();
refuse_to_replace_unreadable(&path)?;
let parent = path
.parent()
.ok_or_else(|| Error::Other("configuration path has no parent".into()))?;
@@ -353,6 +466,46 @@ impl AppDirs {
Ok(())
}
/// This machine's id (e.g. ~/.config/proton-drive-linux/machine-id). See
/// [`machine_id`](Self::machine_id).
pub fn machine_id_path(&self) -> PathBuf {
self.dirs.config_dir().join("machine-id")
}
/// A random id for this machine, made on first use and kept in
/// [`machine_id_path`](Self::machine_id_path).
///
/// It names this machine's profile in a device it may share with other
/// computers. Random rather than `/etc/machine-id`, so nothing that
/// identifies the host goes to Drive. In a file of its own rather than
/// `config.json`, because the app saves the config from a copy it loaded
/// earlier, and that would drop an id the daemon had just added.
pub fn machine_id(&self) -> Result<String> {
let path = self.machine_id_path();
match std::fs::read_to_string(&path) {
Ok(id) if is_machine_id(id.trim()) => return Ok(id.trim().to_string()),
Ok(_) => {
tracing::warn!(path = %path.display(), "machine id is unusable; making a new one");
}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => return Err(e.into()),
}
let mut bytes = [0u8; 16];
use std::io::Read as _;
std::fs::File::open("/dev/urandom")?.read_exact(&mut bytes)?;
let id: String = bytes.iter().map(|b| format!("{b:02x}")).collect();
let parent = path
.parent()
.ok_or_else(|| Error::Other("machine id path has no parent".into()))?;
std::fs::create_dir_all(parent)?;
let tmp = path.with_extension(format!("tmp-{}", std::process::id()));
std::fs::write(&tmp, format!("{id}\n"))?;
std::fs::File::open(&tmp)?.sync_all()?;
std::fs::rename(&tmp, &path)?;
std::fs::File::open(parent)?.sync_all()?;
Ok(id)
}
/// Persistent state: inode map / cache index DB lives here.
pub fn state_dir(&self) -> PathBuf {
// `state_dir` is Linux-only in `directories`; fall back to data dir.
@@ -424,7 +577,7 @@ impl AppDirs {
/// typical umask makes them `0755`. Until this was fixed, the only thing
/// keeping another local user out was `~/.cache` and `~/.local/state`
/// happening to be `0700`, which is a convention of the user's system and
/// not something this client established (bugs.md B6).
/// not something this client established (`docs/BUGS.md` B6).
///
/// Applied on every start, not just at creation: a directory that already
/// exists with a permissive mode — restored from a backup that flattened
@@ -441,6 +594,12 @@ impl AppDirs {
}
}
/// Whether `id` is what [`AppDirs::machine_id`] writes: 32 lowercase hex
/// digits.
fn is_machine_id(id: &str) -> bool {
id.len() == 32 && id.bytes().all(|b| matches!(b, b'0'..=b'9' | b'a'..=b'f'))
}
/// Make `dir` owner-only (`0700`). These directories contain decrypted bytes,
/// plaintext names, and an authenticated control socket, so failure is fatal.
fn restrict_dir(dir: &Path) -> Result<()> {
@@ -475,7 +634,7 @@ fn restrict_dir(dir: &Path) -> Result<()> {
/// session — enumerate, read, upload, trash, share — without presenting any
/// credential, since the daemon already holds one. It is an authority boundary,
/// not merely private data, and `UnixListener::bind` applies the umask like any
/// other file (bugs.md B6).
/// other file (`docs/BUGS.md` B6).
///
/// There is a window between `bind` and this call during which the socket
/// carries the umask's mode. It is closed in practice by the containing
@@ -496,7 +655,29 @@ mod tests {
std::fs::metadata(p).unwrap().permissions().mode() & 0o777
}
/// bugs.md B6. A permissive mode here is not a cosmetic problem: these
/// A damaged config must not be replaced by defaults the next time a
/// setting changes (`docs/BUGS.md` B223).
#[test]
fn an_unreadable_config_is_never_written_over() {
let dir = std::env::temp_dir().join(format!(
"pdfs-cfg-guard-{}-{:?}",
std::process::id(),
std::thread::current().id()
));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("config.json");
assert!(refuse_to_replace_unreadable(&path).is_ok(), "no file yet");
std::fs::write(&path, serde_json::to_vec(&AppConfig::default()).unwrap()).unwrap();
assert!(refuse_to_replace_unreadable(&path).is_ok());
std::fs::write(&path, br#"{"mountpoint": "/home/me/Pro"#).unwrap();
assert!(refuse_to_replace_unreadable(&path).is_err(), "truncated");
let _ = std::fs::remove_dir_all(&dir);
}
/// `docs/BUGS.md` B6. A permissive mode here is not a cosmetic problem: these
/// directories hold decrypted content, a plaintext index of every node name,
/// and the control socket that commands the daemon's session.
#[test]
@@ -563,8 +744,8 @@ mod tests {
#[test]
fn test_app_config_serialization() {
let config = AppConfig {
app_version: "external-drive-test-client@1.0.0".to_string(),
user_agent: "test-agent/1.0".to_string(),
app_version: Some("external-drive-test-client@1.0.0".to_string()),
user_agent: Some("test-agent/1.0".to_string()),
cache_budget: Some(1234),
upload_limit: Some(500_000),
download_limit: None,
@@ -581,21 +762,69 @@ mod tests {
descending: true,
folders_first: false,
},
window: WindowState {
width: 1200,
height: 800,
maximized: true,
},
tray_hidden: true,
language: Some("de".to_string()),
online_map: true,
local_first: Some(false),
update_check: true,
update_last_checked: Some(1_760_000_000),
update_latest: Some("3.6.0".to_string()),
};
let json = serde_json::to_string(&config).unwrap();
let decoded: AppConfig = serde_json::from_str(&json).unwrap();
assert_eq!(decoded.app_version, "external-drive-test-client@1.0.0");
assert_eq!(decoded.user_agent, "test-agent/1.0");
assert_eq!(
decoded.resolved_app_version(),
"external-drive-test-client@1.0.0"
);
assert_eq!(decoded.resolved_user_agent(), "test-agent/1.0");
assert_eq!(decoded.proton_theme, Some(true));
assert!(decoded.tray_hidden);
assert!(decoded.online_map);
assert!(decoded.update_check);
assert_eq!(decoded.update_last_checked, Some(1_760_000_000));
assert_eq!(decoded.update_latest.as_deref(), Some("3.6.0"));
assert_eq!(decoded.language.as_deref(), Some("de"));
assert_eq!(decoded.upload_limit, Some(500_000));
assert_eq!(decoded.download_limit, None);
assert_eq!(decoded.files_view, config.files_view);
assert_eq!(decoded.window, config.window);
}
#[test]
fn a_config_without_a_window_opens_at_the_default_size() {
let decoded: AppConfig =
serde_json::from_str(r#"{"app_version":"a","user_agent":"b"}"#).unwrap();
assert_eq!(decoded.window, WindowState::default());
let partial: WindowState = serde_json::from_str(r#"{"maximized":true}"#).unwrap();
assert_eq!((partial.width, partial.height), (980, 680));
assert!(partial.maximized);
}
#[test]
fn a_window_saved_too_small_opens_at_the_default_size() {
let tiny = WindowState {
width: 0,
height: 500,
maximized: true,
};
assert_eq!(
tiny.usable(),
WindowState {
maximized: true,
..WindowState::default()
}
);
let fine = WindowState {
width: 640,
height: 480,
maximized: false,
};
assert_eq!(fine.usable(), fine);
}
#[test]
@@ -619,8 +848,55 @@ mod tests {
#[test]
fn test_default_app_config() {
let default_config = AppConfig::default();
assert_eq!(default_config.app_version, APP_VERSION);
assert_eq!(default_config.user_agent, USER_AGENT);
assert_eq!(default_config.resolved_app_version(), APP_VERSION);
assert_eq!(default_config.resolved_user_agent(), USER_AGENT);
}
#[test]
fn the_client_identity_follows_the_release() {
let version = env!("CARGO_PKG_VERSION");
assert_eq!(
APP_VERSION,
format!("external-drive-linux@{version}-stable")
);
assert_eq!(USER_AGENT, format!("proton-drive-linux/{version}"));
}
#[test]
fn an_old_config_with_the_fixed_identity_follows_the_release() {
// Every config written up to 3.1.0 holds these two values, so they must
// not pin the identity once Proton retires 0.1.0.
let json = format!(
r#"{{"app_version":"{LEGACY_APP_VERSION}","user_agent":"{LEGACY_USER_AGENT}","tray_hidden":true}}"#
);
let decoded: AppConfig = serde_json::from_str(&json).unwrap();
assert_eq!(decoded.resolved_app_version(), APP_VERSION);
assert_eq!(decoded.resolved_user_agent(), USER_AGENT);
assert!(decoded.tray_hidden, "the rest of the config survives");
let saved = serde_json::to_string(&decoded).unwrap();
assert!(!saved.contains("app_version"), "{saved}");
assert!(!saved.contains("user_agent"), "{saved}");
}
#[test]
fn a_hand_set_client_identity_is_kept() {
let json = r#"{"app_version":"external-drive-linux@9.9.9-beta","user_agent":"custom/1"}"#;
let decoded: AppConfig = serde_json::from_str(json).unwrap();
assert_eq!(
decoded.resolved_app_version(),
"external-drive-linux@9.9.9-beta"
);
assert_eq!(decoded.resolved_user_agent(), "custom/1");
let saved = serde_json::to_string(&decoded).unwrap();
assert!(saved.contains("external-drive-linux@9.9.9-beta"), "{saved}");
}
#[test]
fn a_config_without_a_client_identity_loads() {
let decoded: AppConfig = serde_json::from_str("{}").unwrap();
assert_eq!(decoded.resolved_app_version(), APP_VERSION);
assert_eq!(decoded.resolved_user_agent(), USER_AGENT);
}
#[test]
@@ -679,6 +955,15 @@ mod tests {
assert_eq!(decoded.conflict_sweep, Some(SweepMode::Enforce));
}
#[test]
fn a_machine_id_is_32_lowercase_hex_digits() {
assert!(is_machine_id("0123456789abcdef0123456789abcdef"));
assert!(!is_machine_id("0123456789ABCDEF0123456789ABCDEF"));
assert!(!is_machine_id("0123456789abcdef"));
assert!(!is_machine_id("../../../../etc/passwd/0123456789"));
assert!(!is_machine_id(""));
}
#[test]
fn a_config_predating_ignore_patterns_still_loads() {
// Every installed config on disk looks like this; the field must not be
+192 -8
View File
@@ -443,11 +443,18 @@ pub enum Request {
DeleteDevice { uid: String },
/// Adopt an existing device as this machine's, pinning its uid in
/// `config.json` so a hostname change or reinstall re-attaches to it instead
/// of registering a duplicate (features.md 5.1). `uid: None` clears the pin.
/// of registering a duplicate. `uid: None` clears the pin.
/// Replies with [`Response::Ok`].
AdoptDevice { uid: Option<String> },
/// Move every folder another device backs up into this machine's own
/// device, as server-side moves on the main volume. The other device stays
/// registered, without those folders. Sync folders here that already track
/// one of them follow it to its new share. Nothing is downloaded: the moved
/// folders are then offered by [`Request::ListRestorableFolders`].
/// Replies with [`Response::Ok`].
MigrateDevice { uid: String },
// ---- device folder sync (devices.md) ----------------------------------
// ---- device folder sync ----------------------------------
/// Add a local folder to this machine's device, uploading its tree and
/// registering the device on first use. Replies with [`Response::Ok`].
AddSyncFolder { local_path: String },
@@ -477,6 +484,15 @@ pub enum Request {
/// Retry a backed-off queued op now instead of waiting out its backoff;
/// every failed op when `id` is `None`. Replies with [`Response::Ok`].
RetryPendingOp { id: Option<i64> },
/// Copy the content of a queued upload (`id` from
/// [`Request::ListPendingOps`]) to `dest`, an absolute path that must not
/// exist yet. The op stays queued. Replies with [`Response::Ok`].
ExportPendingOp { id: i64, dest: String },
/// Drop a queued change (`id` from [`Request::ListPendingOps`]) and undo
/// it locally: a node made here goes, a node Drive has goes back to what
/// Drive has, along with every other change queued on it. Its content is
/// lost unless it was exported first. Replies with [`Response::Ok`].
DiscardPendingOp { id: i64 },
/// List the `(sync-conflict …)` copies under My Files and in synced
/// folders. Replies with
/// [`Response::Conflicts`].
@@ -487,7 +503,7 @@ pub enum Request {
/// [`Response::Ok`].
ResolveConflict { path: String, keep: ConflictKeep },
/// List the folders under this machine's device that can be synced here,
/// each with a proposed local path (features.md 5.2). Replies with
/// each with a proposed local path. Replies with
/// [`Response::RestorableFolders`].
ListRestorableFolders,
/// Attach the given remote device folders to local paths and sync them down.
@@ -880,6 +896,59 @@ pub struct PendingOpInfo {
pub parked: bool,
/// Failed often enough to count as stuck.
pub failing: bool,
/// Why Drive refused it, when it did; such an op needs attention at once.
#[serde(default)]
pub issue: Option<SyncIssue>,
/// Carries file content that [`Request::ExportPendingOp`] can save.
#[serde(default)]
pub exportable: bool,
}
/// Why Drive refused a queued change (in [`PendingOpInfo`]). The change stays
/// queued and keeps being retried; nothing is deleted.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum SyncIssue {
/// The account's storage is full.
Quota,
/// The account may no longer change the file or its folder, for example
/// because a share was revoked.
Access,
/// The file or the folder it goes into is gone from Drive.
Missing,
/// The folder holds too many items, or is nested too deep.
Limit,
/// Drive rejected the change, for example a name it does not accept.
Rejected,
/// An issue from a newer daemon.
#[serde(other)]
Unknown,
}
impl SyncIssue {
/// The name stored in the database and sent on the wire.
pub fn as_str(self) -> &'static str {
match self {
SyncIssue::Quota => "quota",
SyncIssue::Access => "access",
SyncIssue::Missing => "missing",
SyncIssue::Limit => "limit",
SyncIssue::Rejected => "rejected",
SyncIssue::Unknown => "unknown",
}
}
/// The issue stored as `name`.
pub fn parse(name: &str) -> Self {
match name {
"quota" => SyncIssue::Quota,
"access" => SyncIssue::Access,
"missing" => SyncIssue::Missing,
"limit" => SyncIssue::Limit,
"rejected" => SyncIssue::Rejected,
_ => SyncIssue::Unknown,
}
}
}
/// One synced local folder on this machine's device (in [`Response::SyncFolders`]).
@@ -1156,6 +1225,12 @@ pub struct JobItem {
/// Steps known to need doing, or `0` when unknown (indeterminate progress).
/// May *grow* mid-job as more work is discovered.
pub total: u64,
/// Housekeeping that changes nothing in the user's files: a mirror folder
/// scan that finds nothing to do yet, an index rebuild, a cache check. Front
/// ends may show it, but should not report the drive as busy over it. Absent
/// from older daemons, which never set it.
#[serde(default)]
pub background: bool,
}
/// One in-flight transfer in a [`Response::Transfers`] snapshot.
@@ -1185,8 +1260,15 @@ pub struct DirEntry {
pub size: u64,
/// Modification time, epoch seconds.
pub modified: i64,
/// Whether the file is pinned to this device.
/// Whether the node is pinned to this device, on its own or through a
/// pinned folder above it.
pub pinned: bool,
/// Whether the node is kept on this device because a folder above it is
/// pinned. Such a node cannot be unpinned on its own. Set in
/// [`Request::ListDir`] listings only. Defaulted for wire-compat with
/// daemons predating the field.
#[serde(default)]
pub pinned_by_folder: bool,
/// Whether the file's full content is present in the local cache (a current,
/// non-stale blob). Always false for folders. Defaulted for wire-compat with
/// clients/daemons predating the field.
@@ -1234,6 +1316,10 @@ pub struct DirEntry {
/// not known, and for anything not in the trash.
#[serde(default)]
pub trashed_from: Option<String>,
/// Why Drive refused a change queued on the node, for a [`Request::ListDir`]
/// entry; `None` otherwise.
#[serde(default)]
pub issue: Option<SyncIssue>,
}
/// One hit in a [`Request::Search`] result. Like [`DirEntry`] but carries the
@@ -1505,6 +1591,10 @@ pub struct RedateSummary {
/// name implies)` — what a dry run is read for.
#[serde(default)]
pub samples: Vec<(String, i64, i64)>,
/// Why the run as a whole failed, when it did. The counts are then
/// empty: the run never got to the photos.
#[serde(default)]
pub error: Option<String>,
}
/// How a Google Photos Takeout import is going, or how the last one ended
@@ -1534,6 +1624,10 @@ pub struct ImportSummary {
pub bytes: u64,
/// True when the run stopped early because it was cancelled.
pub cancelled: bool,
/// Why the run as a whole failed, when it did, such as an archive that
/// is not a zip. The counts are then empty.
#[serde(default)]
pub error: Option<String>,
}
/// One album in a [`Response::Albums`] listing.
@@ -1771,6 +1865,27 @@ pub fn pending_summary(uploads: u64, changes: u64) -> Option<String> {
}
}
/// Whether the daemon that reported `version` in a [`Response::Status`] is an
/// older release than this build. Installing a package replaces the binaries
/// but leaves the running user service on the old one, which keeps its bugs
/// until it restarts, while the app and the CLI already describe the new
/// release (#33). Empty means a daemon from before the field, so older too.
///
/// Only older counts. A newer daemon means this front-end is the one still
/// running from before the upgrade, and restarting the service would not help.
pub fn daemon_outdated(version: &str) -> bool {
fn parse(v: &str) -> Option<(u64, u64, u64)> {
let mut parts = v.split('.').map(|p| p.parse().ok());
let parsed = (parts.next()??, parts.next()??, parts.next()??);
parts.next().is_none().then_some(parsed)
}
match (parse(version), parse(env!("CARGO_PKG_VERSION"))) {
(Some(running), Some(this)) => running < this,
(None, _) => version.is_empty(),
(Some(_), None) => false,
}
}
/// One worker thread in a [`Diagnostics`] report.
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct WorkerState {
@@ -1837,17 +1952,17 @@ pub enum Response {
/// The pin registry.
pins: Vec<Pin>,
/// False when the daemon is serving the cached tree because the API is
/// unreachable (offline.md Phase 1). Cached and pinned content still
/// unreachable. Cached and pinned content still
/// reads; anything else fails until the network is back.
#[serde(default = "default_online")]
online: bool,
/// Writes accepted locally but not yet uploaded (offline.md Phase 3).
/// Writes accepted locally but not yet uploaded.
/// Non-zero means the mount is ahead of the remote — either a copy is
/// still draining, or it cannot drain because we are offline.
#[serde(default)]
pending_uploads: u64,
/// Queued mutations that carry no bytes: `mkdir`, `rename`, `trash`
/// (offline.md Phase 3b). Counted apart from `pending_uploads` because
/// Queued mutations that carry no bytes: `mkdir`, `rename`, `trash`.
/// Counted apart from `pending_uploads` because
/// calling a queued `mkdir` an upload is a lie.
#[serde(default)]
pending_changes: u64,
@@ -1880,6 +1995,21 @@ pub enum Response {
/// resumed, or not paused.
#[serde(default)]
paused_until: Option<i64>,
/// True when Proton refused the session itself (refresh token refused,
/// account deleted or disabled). `online` is false then too, but no
/// connection brings it back: only signing in again does.
#[serde(default)]
session_expired: bool,
/// True while the keyring refuses the session's refreshed tokens. The
/// daemon keeps retrying; until it succeeds, a reboot signs the user
/// out.
#[serde(default)]
tokens_unsaved: bool,
/// The release of the daemon that answered. A package upgrade does not
/// restart the user service, so it can be older than the front-end
/// asking; daemons before 3.3.0 omit it, which reads as empty.
#[serde(default)]
version: String,
},
/// A human-readable success message.
Ok { message: String },
@@ -2523,6 +2653,27 @@ mod tests {
);
}
/// #33: a daemon left running over an upgrade is named as older, also when
/// it predates the version field, but a newer one is not, since restarting
/// the service cannot bring it closer to this build.
#[test]
fn only_an_older_daemon_is_outdated() {
assert!(!daemon_outdated(env!("CARGO_PKG_VERSION")));
assert!(daemon_outdated(""));
assert!(daemon_outdated("0.9.9"));
assert!(!daemon_outdated("999.0.0"));
assert!(!daemon_outdated("not-a-version"));
let old: Response = serde_json::from_str(
r#"{"Status":{"username":"u","mountpoint":"/m","pinned":0,
"used":0,"budget":0,"pins":[]}}"#,
)
.unwrap();
let Response::Status { version, .. } = old else {
panic!("not a status: {old:?}");
};
assert!(daemon_outdated(&version));
}
/// The mutation requests must survive a line-delimited JSON round-trip, since
/// that is exactly how they cross the control socket.
#[test]
@@ -3016,6 +3167,9 @@ mod tests {
uid: Some("dev-1".into()),
},
Request::AdoptDevice { uid: None },
Request::MigrateDevice {
uid: "dev-2".into(),
},
Request::ListRestorableFolders,
Request::RestoreSyncFolders {
items: vec![RestoreItem {
@@ -3290,6 +3444,17 @@ mod tests {
},
r#"{"SaveRevisionAsByUid":{"uid":"vol~link","revision_id":"rev-3","dest":"/tmp/out.bin"}}"#,
),
(
Request::ExportPendingOp {
id: 7,
dest: "/tmp/out.bin".into(),
},
r#"{"ExportPendingOp":{"id":7,"dest":"/tmp/out.bin"}}"#,
),
(
Request::DiscardPendingOp { id: 7 },
r#"{"DiscardPendingOp":{"id":7}}"#,
),
];
for (request, wire) in cases {
assert_eq!(serde_json::to_string(&request).unwrap(), wire);
@@ -3298,6 +3463,25 @@ mod tests {
}
}
/// A daemon that predates sync issues sends neither field, and a newer one
/// may send an issue this build does not know.
#[test]
fn a_queued_op_from_another_daemon_version_still_decodes() {
let old = r#"{"id":1,"kind":"create","path":"a.txt","attempts":0,"last_error":null,
"queued_at":5,"next_attempt_at":5,"parked":false,"failing":false}"#;
let op: PendingOpInfo = serde_json::from_str(old).unwrap();
assert_eq!(op.issue, None);
assert!(!op.exportable);
let newer = old.replace(r#""failing":false"#, r#""failing":true,"issue":"embargo""#);
let op: PendingOpInfo = serde_json::from_str(&newer).unwrap();
assert_eq!(op.issue, Some(SyncIssue::Unknown));
assert_eq!(
SyncIssue::parse(SyncIssue::Quota.as_str()),
SyncIssue::Quota
);
}
/// A front-end that predates the favorites filter sends no `favorites`
/// field, and must keep getting the whole timeline rather than an empty page.
#[test]
+71
View File
@@ -111,9 +111,80 @@ impl Db {
)?;
let count: i64 = tx.query_row("SELECT COUNT(*) FROM local_files", [], |r| r.get(0))?;
tx.commit()?;
drop(conn);
self.compact_search_indexes_if_due(finished_at)?;
Ok(count)
}
/// Close a scan that did not see everything: it stopped at the entry cap,
/// or a batch failed to write. Nothing is pruned, since a path it missed is
/// not a path that is gone; the next complete scan prunes. Returns the
/// number of indexed entries.
///
/// Pruning after such a scan dropped every entry it had not reached, and
/// the next scan put them back. That churn is most of what grew
/// `local_fts` to 16 times its size (`docs/BUGS.md` B217).
pub fn local_finish_partial_scan(&self, finished_at: i64) -> Result<i64> {
let conn = self.conn.lock();
conn.execute(
"INSERT INTO sync_state (key, value) VALUES ('local_indexed_at', ?1)
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
[finished_at.to_string()],
)?;
let count: i64 = conn.query_row("SELECT COUNT(*) FROM local_files", [], |r| r.get(0))?;
Ok(count)
}
/// Compact both full-text indexes, at most once a day.
///
/// FTS5 keeps a deleted entry's data until the segments holding it are
/// merged, and with the few large segments these indexes settle into that
/// is close to never. Every file the local scan sees disappear or appear
/// leaves some behind, and every node row the tree replaces does too: on
/// one machine `local_fts` held 317 MiB for 40,000 paths, against 20 MiB
/// rebuilt, and a search took four times as long (`docs/BUGS.md` B217).
///
/// `local_fts` is rebuilt from `local_files`, which also drops anything
/// left over, and `nodes_fts`, which has no content table to rebuild from,
/// is merged into one segment. Each runs in its own transaction on the
/// write connection, about half a second on an index that size; once a
/// day keeps the leftovers small without paying that at every scan.
pub fn compact_search_indexes_if_due(&self, now: i64) -> Result<()> {
const EVERY_SECS: i64 = 24 * 60 * 60;
let last: i64 = self
.read()
.query_row(
"SELECT value FROM sync_state WHERE key = 'fts_compacted_at'",
[],
|row| row.get::<_, String>(0),
)
.optional()?
.and_then(|v| v.parse().ok())
.unwrap_or(0);
if now - last < EVERY_SECS {
return Ok(());
}
let started = std::time::Instant::now();
{
let conn = self.conn.lock();
conn.execute("INSERT INTO local_fts(local_fts) VALUES('rebuild')", [])?;
}
{
let conn = self.conn.lock();
conn.execute("INSERT INTO nodes_fts(nodes_fts) VALUES('optimize')", [])?;
conn.execute(
"INSERT INTO sync_state (key, value) VALUES ('fts_compacted_at', ?1)
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
[now.to_string()],
)?;
}
tracing::info!(
elapsed_ms = started.elapsed().as_millis() as u64,
"compacted the search indexes"
);
Ok(())
}
/// When the last local scan completed (epoch seconds), or `None` if the index
/// has never been built. The daemon uses this to decide whether a fresh mount
/// needs an immediate rescan or can serve the existing index.
+45 -4
View File
@@ -1,4 +1,4 @@
//! Database health reporting and repair (features.md 4.3).
//! Database health reporting and repair.
//!
//! Everything here exists for the case where something has already gone wrong:
//! a crash mid-write, a full disk, a cache that has grown out of proportion to
@@ -91,8 +91,13 @@ impl Db {
/// Returns an empty vector when the database is sound; otherwise each entry
/// is one problem SQLite found. Read-only and safe against a live daemon,
/// though it reads every page, so it is not free on a large file.
///
/// Runs on a read connection: it reads every page, which took 34 s on a
/// 567 MiB file, and on the write connection every write in the daemon
/// waited that long behind it (`docs/BUGS.md` B218). WAL lets it read a
/// consistent snapshot while the writer goes on.
pub fn integrity_check(&self) -> Result<Vec<String>> {
let conn = self.conn.lock();
let conn = self.read();
let mut stmt = conn.prepare("PRAGMA integrity_check")?;
let rows = stmt.query_map([], |r| r.get::<_, String>(0))?;
let mut problems = Vec::new();
@@ -107,9 +112,11 @@ impl Db {
Ok(problems)
}
/// Size and row-count snapshot for `pdfs cache inspect`.
/// Size and row-count snapshot for `pdfs cache inspect`. On a read
/// connection, for the reason [`integrity_check`](Self::integrity_check)
/// gives: counting the rows of every table reads most of the file.
pub fn stats(&self) -> Result<DbStats> {
let conn = self.conn.lock();
let conn = self.read();
let page_count: i64 = conn.query_row("PRAGMA page_count", [], |r| r.get(0))?;
let free_pages: i64 = conn.query_row("PRAGMA freelist_count", [], |r| r.get(0))?;
let page_size: i64 = conn.query_row("PRAGMA page_size", [], |r| r.get(0))?;
@@ -218,6 +225,40 @@ mod tests {
assert!(db.integrity_check().unwrap().is_empty());
}
/// `pdfs diagnose` and `pdfs cache inspect` read the whole file. Behind the
/// write lock they froze every write in the daemon for as long as that
/// took (`docs/BUGS.md` B218).
#[test]
fn the_integrity_check_and_stats_do_not_wait_for_the_writer() {
let path = std::env::temp_dir().join(format!(
"pdfs-check-{}-{}.db",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
let db = std::sync::Arc::new(Db::open(&path).unwrap());
let writer = db.conn.lock();
let (tx, rx) = std::sync::mpsc::channel();
let reader = std::sync::Arc::clone(&db);
std::thread::spawn(move || {
let checked = reader.integrity_check().map(|p| p.is_empty());
let counted = reader.stats().map(|s| s.page_count > 0);
let _ = tx.send((checked.ok(), counted.ok()));
});
let answer = rx.recv_timeout(std::time::Duration::from_secs(10));
drop(writer);
assert_eq!(answer.ok(), Some((Some(true), Some(true))));
drop(db);
for suffix in ["", "-wal", "-shm"] {
let mut file = path.as_os_str().to_owned();
file.push(suffix);
let _ = std::fs::remove_file(std::path::PathBuf::from(file));
}
let _ = std::fs::remove_file(path.with_extension("db.lock"));
}
#[test]
fn stats_report_the_schema_and_the_known_tables() {
let db = Db::open_in_memory().unwrap();
+278 -8
View File
@@ -5,11 +5,13 @@
//! is already on disk somewhere, so a change here would silently diverge from
//! what an existing install actually has. Add a new one instead.
use rusqlite::OptionalExtension;
use super::Db;
use crate::Result;
/// Current schema version. Bump on every forward migration added below.
pub(super) const SCHEMA_VERSION: i64 = 36;
pub(super) const SCHEMA_VERSION: i64 = 44;
impl Db {
pub(super) fn migrate(&self) -> Result<()> {
@@ -348,6 +350,123 @@ impl Db {
tx.execute_batch(MIGRATION_V36)?;
}
}
if current < 37 {
// Same guards as V26-V36.
let sql: Option<String> = tx
.query_row(
"SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get(0),
)
.optional()?;
if sql.is_some_and(|sql| !sql.contains("AUTOINCREMENT")) {
tx.execute_batch(MIGRATION_V37)?;
}
}
if current < 38 {
// Same guards as V26-V37.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('pending_op') WHERE name = 'issue'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_ops: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_ops && !has_column {
tx.execute_batch(MIGRATION_V38)?;
}
}
if current < 39 {
// Same guards as V26-V38.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('pending_op') WHERE name = 'lid'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_ops: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_ops && !has_column {
tx.execute_batch(MIGRATION_V39)?;
}
}
if current < 40 {
// Same guards as V26-V39.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('nodes') WHERE name = 'parent_lid'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_nodes: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'nodes'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_nodes && !has_column {
tx.execute_batch(MIGRATION_V40)?;
}
}
if current < 41 {
// Needs both tables; on a database that never had them there is
// nothing to rewrite.
let tables: i64 = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master
WHERE type = 'table' AND name IN ('nodes', 'pending_op', 'pins')",
[],
|row| row.get(0),
)?;
if tables == 3 {
tx.execute_batch(MIGRATION_V41)?;
}
}
if current < 42 {
// Same guards as V26-V40.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('pending_op') WHERE name = 'sent_sha1'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_ops: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_ops && !has_column {
tx.execute_batch(MIGRATION_V42)?;
}
}
if current < 43 {
let has_ops: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_ops {
tx.execute_batch(MIGRATION_V43)?;
}
}
if current < 44 {
// Same guards as V26-V42.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('sync_entry') \
WHERE name = 'remote_revision_id'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_entries: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'sync_entry'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_entries && !has_column {
tx.execute_batch(MIGRATION_V44)?;
}
}
tx.execute(
"INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1)
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
@@ -485,7 +604,7 @@ CREATE TABLE trash (
);
";
/// Schema v8: device sync (devices.md). This machine registers as one Proton Drive
/// Schema v8: device sync. This machine registers as one Proton Drive
/// **Device** (a share + root folder on the main volume); `device` is a singleton
/// row cached so we reuse the same device across restarts instead of creating a
/// new one each run. `sync_folder` is one row per local folder the user added,
@@ -546,8 +665,8 @@ ALTER TABLE sync_folder ADD COLUMN pending_mode TEXT;
";
/// Schema v11: writes no longer upload inside the FUSE handler. `release` stages
/// the bytes and records the intended upload here, and a drain worker performs it
/// (offline.md Phase 3). The row outlives the process, so a write survives both a
/// the bytes and records the intended upload here, and a drain worker performs it.
/// The row outlives the process, so a write survives both a
/// dead network and a restart.
///
/// `blob_path` points into the content cache's `staging/` dir and `meta_json` is
@@ -574,7 +693,7 @@ CREATE INDEX pending_op_uid ON pending_op(uid);
/// Schema v12: `pending_op` also carries mutations that *create* a node, which a
/// revision op never had to describe — it always addressed a node the server had
/// already minted a uid for (offline.md Phase 3b).
/// already minted a uid for.
///
/// An offline `create`/`mkdir` cannot get a real uid, so the node is invented
/// locally under a `local~<uuid>` placeholder and the op records where it goes
@@ -964,7 +1083,7 @@ ALTER TABLE pending_op ADD COLUMN access_deferred_since INTEGER NOT NULL DEFAULT
/// as the last sync and left the file the same length — a fixed-size record
/// rewritten, an image re-exported, a database page updated in place — read as
/// *unchanged* and was never uploaded. The remote copy silently stayed at the
/// older content until something else about the file moved (bugs.md B25).
/// older content until something else about the file moved (`docs/BUGS.md` B25).
///
/// Nullable rather than a converted `local_mtime * 1_000_000_000`: the
/// nanosecond part of an existing baseline is genuinely unknown, and inventing
@@ -1076,8 +1195,8 @@ const MIGRATION_V35: &str = "
ALTER TABLE trash ADD COLUMN trashed_at INTEGER;
";
/// Schema v36: every node row has a local id (`lid`), phase 2 of
/// docs/MILESTONE-3.0.0.md.
/// Schema v36: every node row has a local id (`lid`); see
/// docs/ARCHITECTURE.md §2.
///
/// The `uid` was the row's key, so a node had no identity until Drive minted
/// one, and a create that landed moved its row to a new key. The `lid` is given
@@ -1112,3 +1231,154 @@ CREATE INDEX idx_nodes_parent ON nodes(parent_uid);
CREATE INDEX idx_nodes_name_nocase ON nodes(name COLLATE NOCASE);
CREATE INDEX idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;
";
/// Schema v37: a `pending_op` id is never given out twice.
///
/// A plain rowid is the largest one plus one, so an op queued right after the
/// newest one was dropped took its id. The drain retires the op it is
/// working on by id, and so deleted the new op: a trash queued over a rename
/// on the wire was lost when the rename landed. `AUTOINCREMENT` never reuses
/// an id. Ids are kept, as is every index.
const MIGRATION_V37: &str = "
CREATE TABLE pending_op_v37 (
id INTEGER PRIMARY KEY AUTOINCREMENT,
kind TEXT NOT NULL,
uid TEXT NOT NULL,
blob_path TEXT,
meta_json TEXT,
created_at INTEGER NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
next_attempt_at INTEGER NOT NULL DEFAULT 0,
parent_uid TEXT,
name TEXT,
claimed_at INTEGER NOT NULL DEFAULT 0,
access_deferred_since INTEGER NOT NULL DEFAULT 0
);
INSERT INTO pending_op_v37
(id, kind, uid, blob_path, meta_json, created_at, attempts, last_error, next_attempt_at,
parent_uid, name, claimed_at, access_deferred_since)
SELECT id, kind, uid, blob_path, meta_json, created_at, attempts, last_error,
next_attempt_at, parent_uid, name, claimed_at, access_deferred_since
FROM pending_op;
DROP TABLE pending_op;
ALTER TABLE pending_op_v37 RENAME TO pending_op;
CREATE INDEX pending_op_uid ON pending_op(uid);
CREATE INDEX pending_op_parent ON pending_op(parent_uid);
CREATE INDEX idx_pending_op_claimed
ON pending_op(uid) WHERE claimed_at <> 0;
";
/// Schema v38: why Drive refused a queued op, when it did (a sync issue).
/// `NULL` for an op that has not been refused, which is every op migrated.
const MIGRATION_V38: &str = "
ALTER TABLE pending_op ADD COLUMN issue TEXT;
";
/// Schema v39: a queued op names its node and its parent by local id too.
///
/// A create made inside a folder that had not landed named the folder by its
/// placeholder uid, and every landing had to find and rewrite those strings.
/// The local id stays the same when the folder lands. An op whose node has no
/// row keeps `NULL` and is matched by uid, as before.
const MIGRATION_V39: &str = "
ALTER TABLE pending_op ADD COLUMN lid INTEGER;
ALTER TABLE pending_op ADD COLUMN parent_lid INTEGER;
UPDATE pending_op
SET lid = (SELECT lid FROM nodes WHERE nodes.uid = pending_op.uid),
parent_lid = (SELECT lid FROM nodes WHERE nodes.uid = pending_op.parent_uid);
CREATE INDEX pending_op_lid ON pending_op(lid);
CREATE INDEX pending_op_parent_lid ON pending_op(parent_lid);
";
/// Schema v40: a node row names its parent by local id too.
///
/// The local id stays when a folder made here lands and its row takes the uid
/// Drive gave it, so a child linked by it needs no rewrite. `NULL` while the
/// parent has no row, as `parent_uid` then names nothing either.
const MIGRATION_V40: &str = "
ALTER TABLE nodes ADD COLUMN parent_lid INTEGER;
UPDATE nodes SET parent_lid = (SELECT p.lid FROM nodes p WHERE p.uid = nodes.parent_uid);
CREATE INDEX idx_nodes_parent_lid ON nodes(parent_lid);
";
/// Schema v41: a node made on this machine goes by `local~<lid>` until it
/// lands, a function of its row, where it used to get a `local~<ms>-<n>`
/// placeholder. Every placeholder still stored is rewritten to its row's
/// stand-in: the row itself and its `node_json`, its children, the ops on it
/// or into it, and its pins. A placeholder op with no row keeps its uid and
/// drains as before.
const MIGRATION_V41: &str = "
CREATE TEMP TABLE local_v41 AS
SELECT uid AS old, 'local~' || lid AS new, lid FROM nodes
WHERE uid LIKE 'local~%' AND uid <> 'local~' || lid;
UPDATE pending_op SET uid = (SELECT new FROM local_v41 WHERE old = pending_op.uid)
WHERE uid IN (SELECT old FROM local_v41);
UPDATE pending_op SET parent_uid = (SELECT new FROM local_v41 WHERE old = pending_op.parent_uid)
WHERE parent_uid IN (SELECT old FROM local_v41);
UPDATE pending_op
SET meta_json = json_set(meta_json, '$.uid',
(SELECT new FROM local_v41 WHERE old = json_extract(pending_op.meta_json, '$.uid')))
WHERE json_valid(meta_json)
AND json_extract(meta_json, '$.uid') IN (SELECT old FROM local_v41);
UPDATE pending_op
SET meta_json = json_set(meta_json, '$.original_parent_uid',
(SELECT new FROM local_v41
WHERE old = json_extract(pending_op.meta_json, '$.original_parent_uid')))
WHERE json_valid(meta_json)
AND json_extract(meta_json, '$.original_parent_uid') IN (SELECT old FROM local_v41);
UPDATE pins SET uid = (SELECT new FROM local_v41 WHERE old = pins.uid)
WHERE uid IN (SELECT old FROM local_v41);
UPDATE nodes
SET node_json = json_set(node_json, '$.parent_uid.link_id',
(SELECT CAST(lid AS TEXT) FROM local_v41 WHERE old = nodes.parent_uid))
WHERE parent_uid IN (SELECT old FROM local_v41)
AND json_valid(node_json)
AND json_extract(node_json, '$.parent_uid.volume_id') = 'local';
UPDATE nodes SET parent_uid = (SELECT new FROM local_v41 WHERE old = nodes.parent_uid)
WHERE parent_uid IN (SELECT old FROM local_v41);
UPDATE nodes
SET node_json = json_set(node_json, '$.uid.link_id', CAST(lid AS TEXT))
WHERE lid IN (SELECT lid FROM local_v41)
AND json_valid(node_json)
AND json_extract(node_json, '$.uid.volume_id') = 'local';
UPDATE nodes SET uid = 'local~' || lid WHERE lid IN (SELECT lid FROM local_v41);
DROP TABLE local_v41;
";
/// Schema v42: the SHA-1 of each blob a queued create sent, space-separated.
/// A create whose answer was lost may have made its file with any of them,
/// and the op that finds that file again may run after a restart. `NULL` for
/// an op that has sent nothing, which is every op migrated: what the daemon
/// that wrote it sent is not known. Likewise the parent and name each attempt
/// was sent under, as a JSON list of pairs: a rename since then moves the op
/// but not the file Drive may have made.
const MIGRATION_V42: &str = "
ALTER TABLE pending_op ADD COLUMN sent_sha1 TEXT;
ALTER TABLE pending_op ADD COLUMN sent_to TEXT;
";
/// Schema v43: queued ops by kind and name.
///
/// Before each create, rename and trash the drain asks whether a queued trash
/// or rename holds the name it wants, and a released name wakes the ops
/// waiting for it. Each of those scanned the whole queue, so a large offline
/// batch made draining it quadratic. The kind leads, so a lookup of every
/// queued rename uses the index too.
const MIGRATION_V43: &str = "
CREATE INDEX IF NOT EXISTS pending_op_kind_name ON pending_op(kind, name);
";
/// Schema v44: a synced file's baseline remembers the revision it was synced
/// at.
///
/// A mirrored file edited here is uploaded as a new revision of whatever is
/// current on Drive, and another computer syncing the same folder may have
/// committed one since this pass looked. The revision id is what the upload
/// checks first, so that edit becomes a conflict copy instead of being
/// replaced (B249). `NULL` means "this row predates the column", and the check
/// falls back to the modification time and size; each row gains a real value
/// the next time its path is synced.
const MIGRATION_V44: &str = "
ALTER TABLE sync_entry ADD COLUMN remote_revision_id TEXT;
";
+166 -18
View File
@@ -56,9 +56,9 @@ pub use local::LocalFileHit;
pub use maintenance::{DbStats, VacuumOutcome};
pub use nodes::{PublishedSharedRoot, SearchHit, StoredNode};
pub use ops::{
AttachedBlob, CreateLanding, FAILING_ATTEMPTS, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME,
OP_REVISION, OP_TRASH, PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts, PendingOp, RenameMeta,
op_supersedes,
AttachedBlob, CreateLanding, CreateRetired, FAILING_ATTEMPTS, LOCAL_VOLUME, OP_CREATE,
OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts,
PendingOp, RenameMeta, local_lid, local_uid, op_supersedes,
};
pub use photos::{StoredPhoto, StoredPlace, THUMB_HAVE, THUMB_NONE, THUMB_UNKNOWN, TimelineRow};
pub use pins::PinRow;
@@ -179,6 +179,45 @@ impl Drop for Reader<'_> {
}
}
/// The write connection with `synchronous=FULL`, back to NORMAL on drop.
///
/// NORMAL in WAL mode keeps a commit across a crash of the process but not
/// across a power loss: the WAL is synced only at checkpoints. That is fine for
/// metadata the next listing re-fetches, and not for the queue. A staged blob
/// is synced before its op is queued, and the op row is what makes it
/// reachable; losing that row loses a write `close()` already reported as
/// done. A FULL commit syncs the WAL, which covers every NORMAL commit before
/// it as well.
pub(super) struct Durable<'a>(parking_lot::MutexGuard<'a, Connection>);
impl<'a> Durable<'a> {
pub(super) fn new(conn: parking_lot::MutexGuard<'a, Connection>) -> Result<Self> {
conn.pragma_update(None, "synchronous", "FULL")?;
Ok(Self(conn))
}
}
impl std::ops::Deref for Durable<'_> {
type Target = Connection;
fn deref(&self) -> &Connection {
&self.0
}
}
impl std::ops::DerefMut for Durable<'_> {
fn deref_mut(&mut self) -> &mut Connection {
&mut self.0
}
}
impl Drop for Durable<'_> {
fn drop(&mut self) {
// A failure leaves the writer at FULL: slower, never less safe.
let _ = self.0.pragma_update(None, "synchronous", "NORMAL");
}
}
/// Open a second connection to the same file for reads only.
///
/// `SQLITE_OPEN_READ_ONLY` plus `query_only` is belt and braces: the flag stops
@@ -216,18 +255,34 @@ impl Db {
let conn = match Self::open_configured(path) {
Ok(conn) => conn,
Err(e) if is_corrupt(&e) => {
// Every byte in here is derived from the account or from the
// content cache directory, so the recovery is to start over —
// loudly, and keeping the damaged file for a post-mortem rather
// than deleting evidence.
// Most of what is in here is derived from the account and is
// fetched again. The queue is not: it holds writes accepted
// here that Drive does not have yet, and the mirror baselines
// and pins are this machine's alone. So the damaged file is
// moved aside, kept as evidence, and those tables are copied
// out of it into the new one as far as SQLite can still read
// them (`docs/BUGS.md` B216).
tracing::error!(
db = %path.display(),
error = %e,
"cache database is corrupt — moving it aside and rebuilding \
(metadata will be re-fetched; no user data is stored here)"
"cache database is corrupt; moving it aside, rebuilding, and \
copying the upload queue, sync baselines and pins out of it"
);
quarantine_corrupt_db(path)?;
Self::open_configured(path)?
let damaged = quarantine_corrupt_db(path)?;
let conn = Self::open_configured(path)?;
for (table, result) in salvage(&conn, &damaged) {
match result {
Ok(rows) => {
tracing::warn!(table, rows, "salvaged from the damaged database")
}
Err(error) => tracing::error!(
table,
%error,
"could not salvage from the damaged database"
),
}
}
conn
}
Err(e) => return Err(e),
};
@@ -254,6 +309,12 @@ impl Db {
Ok(db)
}
/// The write connection for a commit that accepts a user's change into
/// the queue. See [`Durable`].
pub(super) fn durable(&self) -> Result<Durable<'_>> {
Durable::new(self.conn.lock())
}
/// A connection for a `SELECT`-only method.
///
/// Reads see the last committed state, which is the same thing they saw
@@ -281,7 +342,8 @@ impl Db {
fn open_configured(path: &Path) -> Result<Connection> {
let conn = Connection::open(path)?;
// WAL: readers never block the single writer. NORMAL sync is the
// standard durability/throughput tradeoff for WAL.
// standard durability/throughput tradeoff for WAL; the queue commits
// that need more raise it to FULL for themselves (`Durable`).
conn.pragma_update(None, "journal_mode", "WAL")?;
conn.pragma_update(None, "synchronous", "NORMAL")?;
conn.pragma_update(None, "foreign_keys", "ON")?;
@@ -428,12 +490,96 @@ fn is_corrupt(e: &Error) -> bool {
)
}
/// Move a damaged database (and its WAL sidecars) out of the way so the next
/// open starts from empty.
/// The tables [`salvage`] copies out of a damaged database, in order, each
/// with the rows it keeps.
///
/// Renamed rather than deleted: this is a cache, but it is also the only
/// evidence of what went wrong, and it costs one file the user can remove.
fn quarantine_corrupt_db(path: &Path) -> Result<()> {
/// The queue goes with the node rows it names by local id: a create made here
/// has no other record of its node, and a real node keeps its local id when its
/// folder is listed again. Those rows come back unlisted, so their folders are
/// fetched again rather than trusted. Claims are cleared by [`Db::open`] as
/// after any other crash.
const SALVAGE: &[(&str, &str)] = &[
(
"nodes",
"lid IN (SELECT lid FROM old.pending_op WHERE lid IS NOT NULL
UNION SELECT parent_lid FROM old.pending_op WHERE parent_lid IS NOT NULL)
OR uid LIKE 'local~%'",
),
("pending_op", "1"),
("sync_folder", "1"),
("sync_entry", "1"),
("pins", "1"),
("own_sealed_rev", "1"),
];
/// Copy what only this machine has out of the damaged database at `damaged`
/// into the fresh one `conn` just created. Returns how each table went.
///
/// Each table is copied in its own transaction, so a table SQLite cannot read
/// to the end is left out whole rather than half copied, and does not stop
/// the others. Only the columns both files have are copied, which covers a
/// file that broke in the middle of an upgrade. The damaged file is opened read
/// only and left as it was.
fn salvage(conn: &Connection, damaged: &Path) -> Vec<(&'static str, Result<usize>)> {
let uri = format!(
"file:{}?mode=ro",
damaged
.to_string_lossy()
.replace('%', "%25")
.replace('?', "%3f")
.replace('#', "%23")
);
if let Err(e) = conn.execute("ATTACH DATABASE ?1 AS old", [&uri]) {
let e = Error::Db(e);
tracing::error!(error = %e, "could not open the damaged database to salvage it");
return Vec::new();
}
let results = SALVAGE
.iter()
.map(|&(table, rows)| (table, salvage_table(conn, table, rows)))
.collect();
let _ = conn.execute("UPDATE nodes SET listed = 0", []);
if let Err(e) = conn.execute("DETACH DATABASE old", []) {
tracing::warn!(error = %e, "could not detach the damaged database");
}
results
}
fn salvage_table(conn: &Connection, table: &str, rows: &str) -> Result<usize> {
let columns = |schema: &str| -> Result<Vec<String>> {
let mut stmt = conn.prepare(&format!("PRAGMA {schema}.table_info({table})"))?;
let names = stmt
.query_map([], |r| r.get::<_, String>(1))?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(names)
};
let new = columns("main")?;
let shared: Vec<String> = columns("old")?
.into_iter()
.filter(|c| new.contains(c))
.map(|c| format!("\"{c}\""))
.collect();
if shared.is_empty() {
return Ok(0);
}
let shared = shared.join(", ");
let tx = conn.unchecked_transaction()?;
let copied = tx.execute(
&format!(
"INSERT OR IGNORE INTO main.{table} ({shared}) SELECT {shared} FROM old.{table} WHERE {rows}"
),
[],
)?;
tx.commit()?;
Ok(copied)
}
/// Move a damaged database (and its WAL sidecars) out of the way so the next
/// open starts from empty. Returns where the database itself went.
///
/// Renamed rather than deleted: it is the only evidence of what went wrong,
/// and [`salvage`] reads what only this machine has back out of it.
fn quarantine_corrupt_db(path: &Path) -> Result<PathBuf> {
let stamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
@@ -457,7 +603,9 @@ fn quarantine_corrupt_db(path: &Path) -> Result<()> {
Err(e) => tracing::warn!(file = %from.display(), error = %e, "could not quarantine"),
}
}
Ok(())
let mut kept = path.as_os_str().to_owned();
kept.push(format!(".corrupt-{stamp}"));
Ok(PathBuf::from(kept))
}
#[cfg(test)]
+329 -98
View File
@@ -3,9 +3,11 @@
use std::collections::HashSet;
use rusqlite::types::Value;
use rusqlite::{OptionalExtension, Transaction, params};
use super::Db;
use super::ops::{OP_CREATE, OP_MKDIR, OP_RENAME, local_lid, local_uid, wake_ops_waiting_for_tx};
use crate::{Access, Result};
use proton_drive_rs::proton_sdk::ids::{LinkId, NodeUid, VolumeId};
use proton_drive_rs::{Node, NodeKind};
@@ -64,12 +66,13 @@ impl Db {
/// snapshot while offline, including after a live event expired its TTL.
pub fn visible_children(&self, parent: &NodeUid) -> Result<Vec<Node>> {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT node_json FROM nodes
WHERE parent_uid = ?1 AND trashed = 0 AND node_json IS NOT NULL
ORDER BY name, uid",
)?;
let rows = stmt.query_map([parent.to_string()], |row| row.get::<_, String>(0))?;
let (key, parent) = parent_key(&conn, &parent.to_string())?;
let mut stmt = conn.prepare(&format!(
"SELECT {NODE_NOW} FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.{key} = ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL
ORDER BY n.name, n.uid"
))?;
let rows = stmt.query_map([parent], |row| row.get::<_, String>(0))?;
let mut nodes = Vec::new();
for row in rows {
nodes.push(serde_json::from_str(&row?)?);
@@ -266,10 +269,25 @@ impl Db {
/// resolve its shared-tree access. Removing the op first could resurrect
/// that row after a crash; removing the row first could strand a now
/// unauthorizable op. One transaction closes both windows.
pub fn complete_trash_op(&self, op_id: i64, uid: &NodeUid) -> Result<()> {
///
/// An op that found the name still held by this trash failed with a
/// backoff, and the name is free now. It is made due at once
/// ([`Db::wake_ops_waiting_for`]).
///
/// Returns the staged blob of the create the trash withdrew, if it kept
/// one, for the caller to discard.
pub fn complete_trash_op(&self, op_id: i64, uid: &NodeUid) -> Result<Option<String>> {
let uid = uid.to_string();
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let (name, blob): (Option<String>, Option<String>) = tx
.query_row(
"SELECT name, blob_path FROM pending_op WHERE id = ?1",
params![op_id],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?
.unwrap_or_default();
let rowid: Option<i64> = tx
.query_row(
"SELECT rowid FROM nodes WHERE uid = ?1",
@@ -282,18 +300,23 @@ impl Db {
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![rowid])?;
}
tx.execute("DELETE FROM pending_op WHERE id = ?1", params![op_id])?;
if let Some(name) = name {
wake_ops_waiting_for_tx(&tx, &name)?;
}
tx.commit()?;
Ok(())
Ok(blob)
}
/// Check if a folder node has any non-trashed children in the database.
pub fn has_children(&self, parent_uid: &NodeUid) -> Result<bool> {
let uid_str = parent_uid.to_string();
let conn = self.read();
let (key, parent) = parent_key(&conn, &parent_uid.to_string())?;
let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM nodes
WHERE parent_uid = ?1 AND trashed = 0 AND node_json IS NOT NULL",
params![uid_str],
&format!(
"SELECT COUNT(*) FROM nodes
WHERE {key} = ?1 AND trashed = 0 AND node_json IS NOT NULL"
),
params![parent],
|row| row.get(0),
)?;
Ok(count > 0)
@@ -330,11 +353,11 @@ impl Db {
pub fn path_relative_to(&self, ancestor_uid: &str, uid: &str) -> Result<Option<String>> {
let conn = self.read();
let mut stmt = conn.prepare(
"WITH RECURSIVE anc(uid, parent_uid, name, depth) AS (
SELECT uid, parent_uid, name, 0 FROM nodes WHERE uid = ?1
"WITH RECURSIVE anc(uid, parent_lid, name, depth) AS (
SELECT uid, parent_lid, name, 0 FROM nodes WHERE uid = ?1
UNION ALL
SELECT n.uid, n.parent_uid, n.name, anc.depth + 1
FROM nodes n JOIN anc ON n.uid = anc.parent_uid
SELECT n.uid, n.parent_lid, n.name, anc.depth + 1
FROM nodes n JOIN anc ON n.lid = anc.parent_lid
WHERE anc.depth < 1024
)
SELECT uid, name FROM anc ORDER BY depth",
@@ -383,12 +406,13 @@ impl Db {
let conn = self.read();
let rows: Vec<HitRow> = if query.chars().count() < TRIGRAM_MIN {
let pat = format!("%{}%", like_escape(query));
let mut stmt = conn.prepare(
"SELECT node_json, uid, path FROM nodes
WHERE name LIKE ?1 ESCAPE '\\' AND trashed = 0 AND node_json IS NOT NULL
AND (?3 IS NULL OR path IS NULL OR path LIKE ?3 ESCAPE '\\')
ORDER BY name LIMIT ?2",
)?;
let mut stmt = conn.prepare(&format!(
"SELECT {NODE_NOW}, n.uid, n.path
FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.name LIKE ?1 ESCAPE '\\' AND n.trashed = 0 AND n.node_json IS NOT NULL
AND (?3 IS NULL OR n.path IS NULL OR n.path LIKE ?3 ESCAPE '\\')
ORDER BY n.name LIMIT ?2"
))?;
collect_hits(stmt.query_map(params![pat, limit as i64, scope_pat], hit_row)?)?
} else {
// Escape double quotes and quote each term, then combine with AND so
@@ -398,13 +422,14 @@ impl Db {
.map(|word| format!("\"{}\"", word.replace('"', "\"\"")))
.collect::<Vec<_>>()
.join(" AND ");
let mut stmt = conn.prepare(
"SELECT n.node_json, n.uid, n.path
let mut stmt = conn.prepare(&format!(
"SELECT {NODE_NOW}, n.uid, n.path
FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid
LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE f.name MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL
AND (?3 IS NULL OR n.path IS NULL OR n.path LIKE ?3 ESCAPE '\\')
ORDER BY f.rank LIMIT ?2",
)?;
ORDER BY f.rank LIMIT ?2"
))?;
collect_hits(stmt.query_map(params![phrase, limit as i64, scope_pat], hit_row)?)?
};
@@ -441,20 +466,22 @@ impl Db {
let lane_limit = limit.div_ceil(2);
let mut rows: Vec<HitRow> = if terms.is_empty() {
let pat = format!("%{}%", like_escape(query));
let mut stmt = conn.prepare(
"SELECT node_json, uid, path FROM nodes
WHERE name LIKE ?1 ESCAPE '\\' AND trashed = 0 AND node_json IS NOT NULL
ORDER BY name LIMIT ?2",
)?;
let mut stmt = conn.prepare(&format!(
"SELECT {NODE_NOW}, n.uid, n.path
FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.name LIKE ?1 ESCAPE '\\' AND n.trashed = 0 AND n.node_json IS NOT NULL
ORDER BY n.name LIMIT ?2"
))?;
collect_hits(stmt.query_map(params![pat, limit as i64], hit_row)?)?
} else {
let expression = terms.join(" OR ");
let mut stmt = conn.prepare(
"SELECT n.node_json, n.uid, n.path
let mut stmt = conn.prepare(&format!(
"SELECT {NODE_NOW}, n.uid, n.path
FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid
LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE nodes_fts MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL
ORDER BY f.rank LIMIT ?2",
)?;
ORDER BY f.rank LIMIT ?2"
))?;
collect_hits(stmt.query_map(params![expression, lane_limit as i64], hit_row)?)?
};
// Trigram rank does not know that a name *starts* with the query, and a
@@ -464,12 +491,13 @@ impl Db {
if !terms.is_empty()
&& let Some(first) = query.chars().next()
{
let mut stmt = conn.prepare(
"SELECT node_json, uid, path FROM nodes
WHERE name COLLATE NOCASE LIKE ?1 ESCAPE '\\'
AND trashed = 0 AND node_json IS NOT NULL
ORDER BY name COLLATE NOCASE LIMIT ?2",
)?;
let mut stmt = conn.prepare(&format!(
"SELECT {NODE_NOW}, n.uid, n.path
FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.name COLLATE NOCASE LIKE ?1 ESCAPE '\\'
AND n.trashed = 0 AND n.node_json IS NOT NULL
ORDER BY n.name COLLATE NOCASE LIMIT ?2"
))?;
let mut extra = Vec::new();
for pattern in [
format!("{}%", like_escape(query)),
@@ -505,18 +533,26 @@ impl Db {
/// re-enumerates from the remote on next access.
pub fn set_listed(&self, uid: &NodeUid, listed: bool) -> Result<()> {
let conn = self.conn.lock();
let (key, uid) = row_key(&uid.to_string());
conn.execute(
"UPDATE nodes SET listed = ?2 WHERE uid = ?1",
params![uid.to_string(), listed as i64],
&format!("UPDATE nodes SET listed = ?2 WHERE {key} = ?1"),
params![uid, listed as i64],
)?;
Ok(())
}
/// Load every persisted node for cold-start hydration of the `State` maps.
///
/// Each node names its parent as the parent's row has it now, found by
/// local id: a folder that landed has its new uid there, while the node
/// stored below it may still hold the placeholder.
pub fn load_all(&self) -> Result<Vec<StoredNode>> {
let conn = self.read();
let mut stmt =
conn.prepare("SELECT node_json, listed, lid FROM nodes WHERE node_json IS NOT NULL")?;
let mut stmt = conn.prepare(&format!(
"SELECT {NODE_NOW}, n.listed, n.lid
FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.node_json IS NOT NULL"
))?;
let rows = stmt.query_map([], |row| {
let json: String = row.get(0)?;
let listed: i64 = row.get(1)?;
@@ -532,13 +568,15 @@ impl Db {
}
/// Load one persisted node back by uid. Used to recover the My Files root
/// when the API is unreachable, so the mount can still serve the cached tree
/// (offline.md Phase 1).
/// when the API is unreachable, so the mount can still serve the cached tree.
pub fn node_by_uid(&self, uid: &str) -> Result<Option<Node>> {
let conn = self.read();
let json: Option<String> = conn
.query_row(
"SELECT node_json FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL",
&format!(
"SELECT {NODE_NOW} FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.uid = ?1 AND n.node_json IS NOT NULL"
),
params![uid],
|r| r.get(0),
)
@@ -549,16 +587,24 @@ impl Db {
}
}
/// The local id of the node stored under `uid`, or `None` when it has no
/// row. It stays the same for as long as the row exists, including when a
/// create lands and the row takes the uid Drive gave it.
pub fn lid_of(&self, uid: &str) -> Result<Option<i64>> {
/// The uid of the node with local id `lid`: `None` when it has no row, or
/// is not on Drive yet.
pub fn uid_of_lid(&self, lid: i64) -> Result<Option<String>> {
let conn = self.read();
Ok(conn
.query_row("SELECT lid FROM nodes WHERE uid = ?1", params![uid], |r| {
.query_row("SELECT uid FROM nodes WHERE lid = ?1", params![lid], |r| {
r.get(0)
})
.optional()?)
.optional()?
.flatten())
}
/// The local id of the node stored under `uid`, or `None` when it has no
/// row. It stays the same for as long as the row exists, including when a
/// create lands and the row takes the uid Drive gave it: a node made on
/// this machine is found by its stand-in ([`local_uid`]) before and after.
pub fn lid_of(&self, uid: &str) -> Result<Option<i64>> {
node_lid(&self.read(), uid)
}
/// The local id of each node, in order, giving a row to each that has none.
@@ -574,23 +620,31 @@ impl Db {
{
let mut known = tx.prepare_cached("SELECT lid FROM nodes WHERE uid = ?1")?;
let mut stub = tx.prepare_cached(
"INSERT INTO nodes (uid, parent_uid, name, is_dir, mtime, trashed)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
"INSERT INTO nodes (uid, parent_uid, name, is_dir, mtime, trashed, parent_lid)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)",
)?;
for node in nodes {
let uid = node.uid.to_string();
let lid = match known.query_row(params![uid], |r| r.get(0)).optional()? {
Some(lid) => lid,
None => {
let parent = node.parent_uid.as_ref().map(ToString::to_string);
let parent_lid = match &parent {
Some(parent) => node_lid(&tx, parent)?,
None => None,
};
stub.execute(params![
uid,
node.parent_uid.as_ref().map(ToString::to_string),
parent,
node.name,
node.is_folder() as i64,
node.modification_time,
node.trashed as i64,
parent_lid,
])?;
tx.last_insert_rowid()
let lid = tx.last_insert_rowid();
adopt_children_tx(&tx, &uid, lid)?;
lid
}
};
lids.push(lid);
@@ -600,15 +654,51 @@ impl Db {
Ok(lids)
}
/// Give a node being made on this machine its row under `parent_uid`, and
/// return its local id. The row goes by [`local_uid`] of that id until
/// the node lands.
///
/// Like [`Db::lids_for`], only a stub: the caller upserts the node.
pub fn add_local_row(
&self,
parent_uid: &str,
name: &str,
is_dir: bool,
mtime: i64,
) -> Result<i64> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
tx.execute(
"INSERT INTO nodes (uid, parent_uid, name, is_dir, mtime, trashed, parent_lid)
VALUES (NULL, ?1, ?2, ?3, ?4, 0,
COALESCE(?5, (SELECT lid FROM nodes WHERE uid = ?1)))",
params![
parent_uid,
name,
is_dir as i64,
mtime,
local_lid(parent_uid)
],
)?;
let lid = tx.last_insert_rowid();
tx.execute(
"UPDATE nodes SET uid = ?2 WHERE lid = ?1",
params![lid, local_uid(lid)],
)?;
tx.commit()?;
Ok(lid)
}
/// Read the persisted incremental-sync cursor (a `DriveEventId`), if any.
/// The daemon resumes from this on restart instead of reseeding to the
/// server head, so changes made while unmounted are still applied (P2).
pub fn children_if_listed(&self, parent: &NodeUid) -> Result<Option<Vec<Node>>> {
let conn = self.read();
let (key, folder) = row_key(&parent.to_string());
let listed: Option<i64> = conn
.query_row(
"SELECT listed FROM nodes WHERE uid = ?1",
params![parent.to_string()],
&format!("SELECT listed FROM nodes WHERE {key} = ?1"),
params![folder],
|r| r.get(0),
)
.optional()?;
@@ -624,11 +714,12 @@ impl Db {
/// had: a listing dropped as stale may miss or keep a child.
pub fn known_children(&self, parent: &NodeUid) -> Result<Vec<Node>> {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT node_json FROM nodes
WHERE parent_uid = ?1 AND node_json IS NOT NULL AND trashed = 0",
)?;
let rows = stmt.query_map(params![parent.to_string()], |r| r.get::<_, String>(0))?;
let (key, parent) = parent_key(&conn, &parent.to_string())?;
let mut stmt = conn.prepare(&format!(
"SELECT {NODE_NOW} FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.{key} = ?1 AND n.node_json IS NOT NULL AND n.trashed = 0"
))?;
let rows = stmt.query_map(params![parent], |r| r.get::<_, String>(0))?;
let mut out = Vec::new();
for json in rows {
out.push(serde_json::from_str(&json?)?);
@@ -636,6 +727,40 @@ impl Db {
Ok(out)
}
/// What a queued change does to `parent`'s listing, which Drive does not
/// show until it lands: the children made or moved in by a queued create,
/// mkdir or rename, as the DB holds them, and the nodes a queued rename
/// moves elsewhere.
///
/// A listing read from Drive is laid over with these, or a folder listed
/// while a create in it is queued, after a restart say, loses the file
/// until the create lands (`docs/BUGS.md` B132).
pub fn queued_children(&self, parent: &NodeUid) -> Result<(Vec<Node>, HashSet<String>)> {
let conn = self.read();
let (key, parent) = parent_key(&conn, &parent.to_string())?;
let mut stmt = conn.prepare(&format!(
"SELECT {NODE_NOW} FROM nodes n LEFT JOIN nodes p ON p.lid = n.parent_lid
WHERE n.{key} = ?1 AND n.node_json IS NOT NULL AND n.trashed = 0
AND EXISTS (SELECT 1 FROM pending_op o
WHERE (o.lid = n.lid OR o.uid = n.uid) AND o.kind IN (?2, ?3, ?4))"
))?;
let rows = stmt.query_map(params![parent, OP_CREATE, OP_MKDIR, OP_RENAME], |r| {
r.get::<_, String>(0)
})?;
let mut here = Vec::new();
for json in rows {
here.push(serde_json::from_str(&json?)?);
}
let mut stmt = conn.prepare(&format!(
"SELECT DISTINCT n.uid FROM pending_op p JOIN nodes n ON n.lid = p.lid OR n.uid = p.uid
WHERE p.kind = ?2 AND n.{key} IS NOT ?1"
))?;
let gone = stmt
.query_map(params![parent, OP_RENAME], |r| r.get::<_, String>(0))?
.collect::<std::result::Result<HashSet<_>, _>>()?;
Ok((here, gone))
}
// --- Content-cache LRU index (P4) -------------------------------------
//
// Replaces the per-eviction `read_dir` scans in `ContentCache`. Each cached
@@ -655,8 +780,53 @@ struct PriorRow {
trashed: bool,
}
/// Hand a drained placeholder's row, and the rows below it, to the real uid it
/// landed as.
/// [`Db::lid_of`] on `conn`, which may be a transaction.
pub(super) fn node_lid(conn: &rusqlite::Connection, uid: &str) -> Result<Option<i64>> {
let row = match local_lid(uid) {
Some(lid) => conn.query_row("SELECT lid FROM nodes WHERE lid = ?1", params![lid], |r| {
r.get(0)
}),
None => conn.query_row("SELECT lid FROM nodes WHERE uid = ?1", params![uid], |r| {
r.get(0)
}),
};
Ok(row.optional()?)
}
/// The column and value that find `uid`'s own row. A stand-in is found by its
/// local id: a create that lands gives the row the uid Drive made before the
/// tree has it, and the tree asks by the stand-in until then (`docs/BUGS.md`
/// B177).
fn row_key(uid: &str) -> (&'static str, Value) {
match local_lid(uid) {
Some(lid) => ("lid", Value::Integer(lid)),
None => ("uid", Value::Text(uid.to_owned())),
}
}
/// The column and value a row names `parent` by: its local id when it has a
/// row, which the children keep when it lands, or its uid when it has none,
/// as a device folder's root does.
fn parent_key(conn: &rusqlite::Connection, parent: &str) -> Result<(&'static str, Value)> {
Ok(match node_lid(conn, parent)? {
Some(lid) => ("parent_lid", Value::Integer(lid)),
None => ("parent_uid", Value::Text(parent.to_owned())),
})
}
/// `n.node_json` naming its parent as the parent's row `p`, joined by
/// `n.parent_lid`, has it now. A folder that lands takes its real uid in its
/// own row only, so a node stored below it still names the stand-in.
const NODE_NOW: &str = "CASE WHEN p.uid IS NULL OR p.uid = n.parent_uid THEN n.node_json
ELSE json_set(n.node_json, '$.parent_uid', json_object(
'volume_id', substr(p.uid, 1, instr(p.uid, '~') - 1),
'link_id', substr(p.uid, instr(p.uid, '~') + 1))) END";
/// Hand a drained placeholder's row to the real uid it landed as.
///
/// Nothing else is rewritten. The rows below it and the ops made inside it
/// name it by its local id, which stays; what reads them for a uid takes the
/// one the row has now ([`NODE_NOW`], `PARENT_NOW` in `ops.rs`).
///
/// Dropping the placeholder row and waiting for the server's copy of the real
/// node left a window with no row for either uid. A queued child drained in
@@ -671,10 +841,13 @@ pub(super) fn adopt_placeholder_row_tx(
local: &str,
real: &str,
) -> Result<()> {
// A folder pinned before it landed stays pinned: the pin keyed by the
// placeholder was dropped with it (`docs/BUGS.md` B148).
tx.execute(
"UPDATE nodes SET parent_uid = ?2 WHERE parent_uid = ?1",
"UPDATE OR IGNORE pins SET uid = ?2 WHERE uid = ?1",
params![local, real],
)?;
tx.execute("DELETE FROM pins WHERE uid = ?1", params![local])?;
let row: Option<(i64, Option<String>)> = tx
.query_row(
"SELECT lid, node_json FROM nodes WHERE uid = ?1",
@@ -682,6 +855,12 @@ pub(super) fn adopt_placeholder_row_tx(
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
// Ops queued before schema 39 have no `parent_lid`, nor do ops made
// under a placeholder that never had a row.
tx.execute(
"UPDATE pending_op SET parent_uid = ?2 WHERE parent_lid IS NULL AND parent_uid = ?1",
params![local, real],
)?;
let Some((lid, json)) = row else {
return Ok(());
};
@@ -693,6 +872,19 @@ pub(super) fn adopt_placeholder_row_tx(
)
.optional()?;
if let Some((known_lid, _)) = known {
// Ops queued against the row Drive listed now name the landing one.
tx.execute(
"UPDATE pending_op SET lid = ?2 WHERE lid = ?1",
params![known_lid, lid],
)?;
tx.execute(
"UPDATE pending_op SET parent_lid = ?2 WHERE parent_lid = ?1",
params![known_lid, lid],
)?;
tx.execute(
"UPDATE nodes SET parent_lid = ?2 WHERE parent_lid = ?1",
params![known_lid, lid],
)?;
tx.execute("DELETE FROM nodes WHERE lid = ?1", params![known_lid])?;
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![known_lid])?;
}
@@ -709,9 +901,36 @@ pub(super) fn adopt_placeholder_row_tx(
upsert_node_tx(tx, &node)
}
/// Link the rows that name `uid` as their parent to its new row `lid`: they
/// were stored before it, or under a row of the same uid since dropped.
fn adopt_children_tx(tx: &Transaction<'_>, uid: &str, lid: i64) -> Result<()> {
tx.execute(
"UPDATE nodes SET parent_lid = ?2 WHERE parent_uid = ?1 AND parent_lid IS NOT ?2",
params![uid, lid],
)?;
Ok(())
}
fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
let json = serde_json::to_string(node)?;
let uid = node.uid.to_string();
// A node written under its stand-in after it landed is its row, which has
// the uid Drive gave it by now: written under the stand-in, it would bring
// the placeholder row back beside it.
if let Some(lid) = local_lid(&uid) {
let landed: Option<String> = tx
.query_row(
"SELECT uid FROM nodes WHERE lid = ?1 AND uid IS NOT ?2",
params![lid, uid],
|row| row.get(0),
)
.optional()?;
if let Some(real) = landed.as_deref().and_then(parse_node_uid) {
let mut node = node.clone();
node.uid = real;
return upsert_node_tx(tx, &node);
}
}
let json = serde_json::to_string(node)?;
let parent_uid = node.parent_uid.as_ref().map(|u| u.to_string());
let prior: Option<PriorRow> = tx
@@ -733,26 +952,34 @@ fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
// where resolving it from scratch is a recursive walk to the root. A node
// whose parent is not cached (a device folder's root lives in `device`, never
// in `nodes`) starts a path of its own, which is what the walk did too.
let path = match &parent_uid {
None => String::new(),
let (parent_lid, path) = match &parent_uid {
None => (None, String::new()),
Some(parent) => {
let parent_path: Option<Option<String>> = tx
.query_row(
"SELECT path FROM nodes WHERE uid = ?1",
params![parent],
|row| row.get(0),
)
.optional()?;
join_path(parent_path.flatten().as_deref().unwrap_or(""), &node.name)
let parent_row: Option<(i64, Option<String>)> = match node_lid(tx, parent)? {
Some(lid) => tx
.query_row(
"SELECT lid, path FROM nodes WHERE lid = ?1",
params![lid],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?,
None => None,
};
let (lid, parent_path) = parent_row.unzip();
(
lid,
join_path(parent_path.flatten().as_deref().unwrap_or(""), &node.name),
)
}
};
tx.execute(
"INSERT INTO nodes
(uid, parent_uid, name, is_dir, size, mtime, trashed, node_json, path)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)
(uid, parent_uid, name, is_dir, size, mtime, trashed, node_json, path, parent_lid)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10)
ON CONFLICT(uid) DO UPDATE SET
parent_uid = excluded.parent_uid,
parent_lid = excluded.parent_lid,
name = excluded.name,
is_dir = excluded.is_dir,
size = excluded.size,
@@ -770,6 +997,7 @@ fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
node.trashed as i64,
json,
path,
parent_lid,
],
)?;
let rowid: i64 = tx.query_row(
@@ -777,6 +1005,9 @@ fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
params![uid],
|row| row.get(0),
)?;
if prior.is_none() {
adopt_children_tx(tx, &uid, rowid)?;
}
// FTS5 has no UPSERT, so the node's own row is always replaced.
let indexable = node_is_indexable_tx(tx, &uid)?;
@@ -799,7 +1030,7 @@ fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
|| prior.trashed != node.trashed
});
if node.is_folder() && subtree_moved {
reindex_subtree_tx(tx, &uid, &path, indexable)?;
reindex_subtree_tx(tx, rowid, &path, indexable)?;
}
Ok(())
}
@@ -820,22 +1051,22 @@ fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
/// this deep.
fn reindex_subtree_tx(
tx: &Transaction<'_>,
folder: &str,
folder: i64,
folder_path: &str,
folder_indexable: bool,
) -> Result<()> {
let descendants: Vec<(i64, String, bool)> = {
let mut stmt = tx.prepare(
"WITH RECURSIVE sub(rowid, uid, name, path, blocked, depth) AS (
SELECT n.rowid, n.uid, n.name,
"WITH RECURSIVE sub(rowid, name, path, blocked, depth) AS (
SELECT n.rowid, n.name,
CASE WHEN ?2 = '' THEN n.name ELSE ?2 || '/' || n.name END,
n.trashed, 0
FROM nodes n WHERE n.parent_uid = ?1
FROM nodes n WHERE n.parent_lid = ?1
UNION ALL
SELECT n.rowid, n.uid, n.name,
SELECT n.rowid, n.name,
sub.path || '/' || n.name,
MAX(sub.blocked, n.trashed), sub.depth + 1
FROM nodes n JOIN sub ON n.parent_uid = sub.uid
FROM nodes n JOIN sub ON n.parent_lid = sub.rowid
WHERE sub.depth < 256
)
SELECT rowid, path, blocked FROM sub",
@@ -876,7 +1107,8 @@ fn reindex_subtree_tx(
}
fn direct_child_uids_tx(tx: &Transaction<'_>, parent: &str) -> Result<Vec<String>> {
let mut stmt = tx.prepare("SELECT uid FROM nodes WHERE parent_uid = ?1")?;
let (key, parent) = parent_key(tx, parent)?;
let mut stmt = tx.prepare(&format!("SELECT uid FROM nodes WHERE {key} = ?1"))?;
let rows = stmt.query_map([parent], |row| row.get::<_, String>(0))?;
let mut uids = Vec::new();
for row in rows {
@@ -901,14 +1133,14 @@ fn direct_child_uids_tx(tx: &Transaction<'_>, parent: &str) -> Result<Vec<String
/// ends at one whose parent is a row we have already visited.
fn node_is_indexable_tx(tx: &Transaction<'_>, uid: &str) -> Result<bool> {
let indexable: i64 = tx.query_row(
"WITH RECURSIVE ancestors(uid, parent_uid, trashed, depth, path) AS (
SELECT uid, parent_uid, trashed, 0, char(31) || uid || char(31)
"WITH RECURSIVE ancestors(lid, parent_lid, trashed, depth, path) AS (
SELECT lid, parent_lid, trashed, 0, ',' || lid || ','
FROM nodes WHERE uid = ?1
UNION ALL
SELECT n.uid, n.parent_uid, n.trashed, a.depth + 1,
a.path || n.uid || char(31)
FROM ancestors a JOIN nodes n ON n.uid = a.parent_uid
WHERE instr(a.path, char(31) || n.uid || char(31)) = 0
SELECT n.lid, n.parent_lid, n.trashed, a.depth + 1,
a.path || n.lid || ','
FROM ancestors a JOIN nodes n ON n.lid = a.parent_lid
WHERE instr(a.path, ',' || n.lid || ',') = 0
)
SELECT CASE
WHEN COUNT(*) = 0 THEN 0
@@ -916,8 +1148,7 @@ fn node_is_indexable_tx(tx: &Transaction<'_>, uid: &str) -> Result<bool> {
WHEN EXISTS (
SELECT 1 FROM ancestors a
WHERE a.depth = (SELECT MAX(depth) FROM ancestors)
AND a.parent_uid IS NOT NULL
AND EXISTS (SELECT 1 FROM nodes p WHERE p.uid = a.parent_uid)
AND EXISTS (SELECT 1 FROM nodes p WHERE p.lid = a.parent_lid)
) THEN 0
ELSE 1
END
@@ -935,7 +1166,7 @@ fn tombstone_subtree_tx(tx: &Transaction<'_>, root_uid: &str) -> Result<()> {
SELECT rowid, uid, node_json FROM nodes WHERE uid = ?1
UNION
SELECT n.rowid, n.uid, n.node_json FROM nodes n
JOIN subtree s ON n.parent_uid = s.uid
JOIN subtree s ON n.parent_lid = s.rowid
)
SELECT rowid, uid, node_json FROM subtree",
)?;
File diff suppressed because it is too large. Load diff
+15 -13
View File
@@ -64,11 +64,11 @@ impl Db {
}
/// Whether `uid` is pinned — directly, or because a strict ancestor folder
/// is pinned recursively. The ancestor check walks `parent_uid` to the root
/// is pinned recursively. The ancestor check walks `parent_lid` to the root
/// via a CTE; a direct pin is honoured even when the node has no `nodes` row
/// yet (e.g. a CLI that never hydrates the node cache).
///
/// The walk is depth-capped like [`super::utils::path_of`]: a `parent_uid`
/// The walk is depth-capped like [`super::utils::path_of`]: a parent
/// cycle is corrupt data the API can hand us, and `UNION ALL` over one never
/// terminates — while holding the daemon's only SQLite connection, on a path
/// that runs per cached read.
@@ -85,11 +85,11 @@ impl Db {
let anc: Option<i64> = conn
.query_row(
&format!(
"WITH RECURSIVE anc(uid, parent_uid, depth) AS (
SELECT uid, parent_uid, 0 FROM nodes WHERE uid = ?1
"WITH RECURSIVE anc(uid, parent_lid, depth) AS (
SELECT uid, parent_lid, 0 FROM nodes WHERE uid = ?1
UNION ALL
SELECT n.uid, n.parent_uid, anc.depth + 1
FROM nodes n JOIN anc ON n.uid = anc.parent_uid
SELECT n.uid, n.parent_lid, anc.depth + 1
FROM nodes n JOIN anc ON n.lid = anc.parent_lid
WHERE anc.depth < {MAX_PATH_DEPTH}
)
SELECT 1 FROM anc a JOIN pins p ON p.uid = a.uid
@@ -109,10 +109,11 @@ impl Db {
pub fn pinned_uids(&self) -> Result<Vec<String>> {
let conn = self.read();
let mut stmt = conn.prepare(
"WITH RECURSIVE sub(uid) AS (
SELECT uid FROM pins WHERE recursive = 1
"WITH RECURSIVE sub(uid, lid) AS (
SELECT p.uid, n.lid FROM pins p LEFT JOIN nodes n ON n.uid = p.uid
WHERE p.recursive = 1
UNION
SELECT n.uid FROM nodes n JOIN sub ON n.parent_uid = sub.uid
SELECT n.uid, n.lid FROM nodes n JOIN sub ON n.parent_lid = sub.lid
)
SELECT uid FROM sub
UNION
@@ -126,15 +127,16 @@ impl Db {
Ok(out)
}
/// Every descendant uid of `folder` (all depths), via a `parent_uid` CTE.
/// Every descendant uid of `folder` (all depths), via a `parent_lid` CTE.
/// Used to evict a recursively-pinned subtree's cached blobs on unpin.
pub fn descendants(&self, folder: &str) -> Result<Vec<String>> {
let conn = self.read();
let mut stmt = conn.prepare(
"WITH RECURSIVE sub(uid) AS (
SELECT uid FROM nodes WHERE parent_uid = ?1
"WITH RECURSIVE sub(uid, lid) AS (
SELECT c.uid, c.lid FROM nodes c JOIN nodes f ON c.parent_lid = f.lid
WHERE f.uid = ?1
UNION
SELECT n.uid FROM nodes n JOIN sub ON n.parent_uid = sub.uid
SELECT n.uid, n.lid FROM nodes n JOIN sub ON n.parent_lid = sub.lid
)
SELECT uid FROM sub",
)?;
+25 -8
View File
@@ -95,11 +95,27 @@ impl Db {
/// Resolve a persisted node's effective access through its nearest recorded
/// share root. `None` means the node itself is not persisted, which callers
/// must not treat as owned: stale handles need to fail closed.
///
/// A stand-in ([`local_uid`](super::local_uid)) finds its row by local id,
/// so it still does once its create has landed and the row has the uid
/// Drive gave it. The tree learns that uid a moment later, and a rename in
/// between was refused (`docs/BUGS.md` B174).
pub fn effective_node_access(&self, uid: &NodeUid) -> Result<Option<Access>> {
let conn = self.read();
let key = uid.to_string();
let key = match super::local_lid(&key) {
Some(lid) => conn
.query_row("SELECT uid FROM nodes WHERE lid = ?1", [lid], |row| {
row.get::<_, Option<String>>(0)
})
.optional()?
.flatten()
.unwrap_or(key),
None => key,
};
let exists: bool = conn.query_row(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL)",
[uid.to_string()],
[&key],
|row| row.get(0),
)?;
if !exists {
@@ -107,25 +123,26 @@ impl Db {
}
let value = conn
.query_row(
"WITH RECURSIVE ancestors(uid, parent_uid, depth, path) AS (
SELECT uid, parent_uid, 0, char(31) || uid || char(31)
"WITH RECURSIVE ancestors(uid, parent_uid, parent_lid, depth, path) AS (
SELECT uid, parent_uid, parent_lid, 0, char(31) || uid || char(31)
FROM nodes WHERE uid = ?1
UNION ALL
SELECT a.parent_uid, n.parent_uid, a.depth + 1,
a.path || a.parent_uid || char(31)
SELECT COALESCE(n.uid, a.parent_uid), n.parent_uid, n.parent_lid,
a.depth + 1,
a.path || COALESCE(n.uid, a.parent_uid) || char(31)
FROM ancestors a
LEFT JOIN nodes n ON n.uid = a.parent_uid
LEFT JOIN nodes n ON n.lid = a.parent_lid
WHERE a.parent_uid IS NOT NULL
AND instr(
a.path,
char(31) || a.parent_uid || char(31)
char(31) || COALESCE(n.uid, a.parent_uid) || char(31)
) = 0
)
SELECT sa.access
FROM ancestors a JOIN share_access sa ON sa.root_uid = a.uid
ORDER BY a.depth ASC
LIMIT 1",
[uid.to_string()],
[&key],
|row| row.get::<_, String>(0),
)
.optional()?;
+38 -16
View File
@@ -25,8 +25,8 @@ pub struct StoredSyncFolder {
}
/// One per-file sync baseline row: what a path looked like on both sides at the
/// last successful sync, so the next reconcile can tell which side changed
/// (devices.md Phase 2). `remote_rev`/`remote_hash` hold the remote signature —
/// last successful sync, so the next reconcile can tell which side changed.
/// `remote_rev`/`remote_hash` hold the remote signature —
/// its modification time and size as strings — since no cheap content hash is
/// exposed; change detection is `(mtime, size)` on each side.
#[derive(Clone, Debug, PartialEq)]
@@ -39,12 +39,18 @@ pub struct StoredSyncEntry {
///
/// `local_mtime` alone cannot tell an edit that landed inside the same
/// second as the last sync — and left the file the same length — from no
/// edit at all, so those never uploaded (bugs.md B25). `None` on a row
/// edit at all, so those never uploaded (`docs/BUGS.md` B25). `None` on a row
/// written before schema 28, which the comparison reads as "compare whole
/// seconds", so upgrading does not make an entire mirror look changed.
pub local_mtime_ns: Option<i64>,
pub remote_rev: Option<String>,
pub remote_hash: Option<String>,
/// The id of the remote revision this row was synced at. A mirror upload
/// checks it before it supersedes what Drive holds, because another client
/// may have committed a revision since (`docs/BUGS.md` B249). `None` on a
/// row written before schema 44, or by a pass that did not learn it; the
/// check then falls back to `remote_rev` and `remote_hash`.
pub remote_revision_id: Option<String>,
}
impl Db {
@@ -139,6 +145,17 @@ impl Db {
Ok(())
}
/// Re-point a synced folder at the share that now holds its remote folder,
/// after the folder moved to another device without changing its uid.
pub fn sync_folder_set_share(&self, id: i64, share_id: &str) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"UPDATE sync_folder SET remote_share_id = ?2 WHERE id = ?1",
params![id, share_id],
)?;
Ok(())
}
/// Queue (or, with `None`, withdraw) a mode the folder should move to once it
/// is safe to switch.
pub fn sync_folder_set_pending_mode(&self, id: i64, mode: Option<&str>) -> Result<()> {
@@ -177,7 +194,7 @@ impl Db {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash,
local_mtime_ns
local_mtime_ns, remote_revision_id
FROM sync_entry WHERE folder_id = ?1",
)?;
let rows = stmt.query_map(params![folder_id], |r| {
@@ -189,6 +206,7 @@ impl Db {
remote_rev: r.get(4)?,
remote_hash: r.get(5)?,
local_mtime_ns: r.get(6)?,
remote_revision_id: r.get(7)?,
})
})?;
let mut map = HashMap::new();
@@ -227,15 +245,16 @@ impl Db {
conn.execute(
"INSERT INTO sync_entry
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev,
remote_hash, local_mtime_ns)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)
remote_hash, local_mtime_ns, remote_revision_id)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)
ON CONFLICT(folder_id, rel_path) DO UPDATE SET
remote_uid = excluded.remote_uid,
local_mtime = excluded.local_mtime,
local_size = excluded.local_size,
remote_rev = excluded.remote_rev,
remote_hash = excluded.remote_hash,
local_mtime_ns = excluded.local_mtime_ns",
remote_uid = excluded.remote_uid,
local_mtime = excluded.local_mtime,
local_size = excluded.local_size,
remote_rev = excluded.remote_rev,
remote_hash = excluded.remote_hash,
local_mtime_ns = excluded.local_mtime_ns,
remote_revision_id = excluded.remote_revision_id",
params![
folder_id,
e.rel_path,
@@ -245,6 +264,7 @@ impl Db {
e.remote_rev,
e.remote_hash,
e.local_mtime_ns,
e.remote_revision_id,
],
)?;
Ok(())
@@ -295,7 +315,7 @@ impl Db {
{
let mut select = tx.prepare(
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash,
local_mtime_ns
local_mtime_ns, remote_revision_id
FROM sync_entry
WHERE folder_id = ?1
AND (rel_path = ?2 OR substr(rel_path, 1, length(?2) + 1) = ?2 || '/')",
@@ -310,6 +330,7 @@ impl Db {
remote_rev: r.get(4)?,
remote_hash: r.get(5)?,
local_mtime_ns: r.get(6)?,
remote_revision_id: r.get(7)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
@@ -321,8 +342,8 @@ impl Db {
let mut insert = tx.prepare(
"INSERT OR REPLACE INTO sync_entry
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev,
remote_hash, local_mtime_ns)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
remote_hash, local_mtime_ns, remote_revision_id)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)",
)?;
for row in rows {
let rel = format!("{to_rel}{}", &row.rel_path[from_rel.len()..]);
@@ -335,6 +356,7 @@ impl Db {
row.remote_rev,
row.remote_hash,
row.local_mtime_ns,
row.remote_revision_id,
])?;
}
}
@@ -345,7 +367,7 @@ impl Db {
/// Drop the entire baseline for a folder. Used when flipping ondemand→mirror:
/// the local tree was evicted, so the old baseline is stale and would make the
/// next reconcile mistake "locally deleted" for "must re-download". Clearing it
/// leaves an empty baseline + full remote = pure download (devices.md P3).
/// leaves an empty baseline + full remote = pure download.
pub fn sync_entries_clear(&self, folder_id: i64) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -102,5 +102,5 @@ impl Db {
Ok(seen)
}
// ---- device sync (devices.md) -----------------------------------------
// ---- device sync -----------------------------------------
}
+6 -6
View File
@@ -72,18 +72,18 @@ pub(super) fn path_of(conn: &Connection, uid: &str) -> Result<String> {
}
}
/// Resolve a path by walking `parent_uid` to the root via a recursive CTE.
/// Resolve a path by walking `parent_lid` to the root via a recursive CTE.
///
/// A cycle in `parent_uid` is corrupt data the API can still hand us, and
/// A cycle in the parents is corrupt data the API can still hand us, and
/// `UNION ALL` over one never terminates — the walk is therefore depth-capped
/// and returns the truncated path rather than hanging the caller.
pub(super) fn walk_path_of(conn: &Connection, uid: &str) -> Result<String> {
let mut stmt = conn.prepare(&format!(
"WITH RECURSIVE anc(uid, parent_uid, name, depth) AS (
SELECT uid, parent_uid, name, 0 FROM nodes WHERE uid = ?1
"WITH RECURSIVE anc(parent_lid, parent_uid, name, depth) AS (
SELECT parent_lid, parent_uid, name, 0 FROM nodes WHERE uid = ?1
UNION ALL
SELECT n.uid, n.parent_uid, n.name, anc.depth + 1
FROM nodes n JOIN anc ON n.uid = anc.parent_uid
SELECT n.parent_lid, n.parent_uid, n.name, anc.depth + 1
FROM nodes n JOIN anc ON n.lid = anc.parent_lid
WHERE anc.depth < {MAX_PATH_DEPTH}
)
SELECT name, parent_uid FROM anc ORDER BY depth DESC"
+102 -15
View File
@@ -32,6 +32,16 @@ pub enum Error {
#[error("saved session can no longer unlock your keys — run `pdfs login` again")]
ReloginRequired,
/// The account has a separate mailbox password that has not been entered
/// since login, so its keys cannot be unlocked. The fix is
/// [`auth::unlock`](crate::auth::unlock), not a new login.
#[error("account is locked — enter the mailbox password with `pdfs unlock`")]
Locked,
/// The mailbox password was checked against the account and is wrong.
#[error("that mailbox password does not unlock your account")]
WrongMailboxPassword,
/// The login was gated behind human verification. Not a failure the user can
/// fix by retrying: the challenge has to be solved and the login restarted
/// with the resulting token (see [`auth::login_verified`](crate::auth::login_verified)).
@@ -144,17 +154,17 @@ impl CoreError {
/// through the boxes the drain deals in, where the concrete type survives
/// but the static one does not.
///
/// A failure that is not an API response at all — a transport error, a
/// timeout, a TLS failure — is what being offline looks like from here, so
/// it classifies as [`ErrorKind::Offline`] rather than
/// [`ErrorKind::Remote`]: the request never reached Proton, and telling the
/// user their connection is down beats telling them the server refused.
/// A transport error, a timeout or a TLS failure is what being offline
/// looks like from here, so it classifies as [`ErrorKind::Offline`] rather
/// than [`ErrorKind::Remote`]: the request never reached Proton, and
/// telling the user their connection is down beats telling them the
/// server refused. The SDK's other failures say what went wrong instead
/// (see [`proton_kind`]).
pub fn from_api(e: &(dyn std::error::Error + 'static), what: &str) -> Self {
let kind = match e.downcast_ref::<proton_sdk::ProtonError>() {
Some(proton_sdk::ProtonError::Api(api)) => api_kind(api.code),
// A `ProtonError` that is not an `Api` variant never reached the
// API; so does anything that is not a `ProtonError` at all.
_ => ErrorKind::Offline,
Some(proton) => proton_kind(proton),
// Not the SDK's at all: a timeout around the call, or the socket.
None => ErrorKind::Offline,
};
Self::new(kind, format!("{what}: {e}"))
}
@@ -199,18 +209,38 @@ impl From<serde_json::Error> for CoreError {
impl From<Error> for CoreError {
fn from(e: Error) -> Self {
let kind = match &e {
Error::NotLoggedIn | Error::ReloginRequired | Error::HumanVerificationRequired(_) => {
ErrorKind::Denied
}
Error::Proton(proton_sdk::ProtonError::Api(api)) => api_kind(api.code),
// A `ProtonError` that never reached the API is a transport failure.
Error::Proton(_) => ErrorKind::Offline,
Error::NotLoggedIn
| Error::ReloginRequired
| Error::Locked
| Error::WrongMailboxPassword
| Error::HumanVerificationRequired(_) => ErrorKind::Denied,
Error::Proton(proton) => proton_kind(proton),
_ => ErrorKind::Internal,
};
Self::new(kind, e.to_string())
}
}
/// What a failed SDK call means to a front-end.
///
/// Only a transport failure is the connection. The rest happened on this
/// machine, and calling them "offline" sent the user to check a network that
/// worked (issue #38: a photo upload on an account without a photo library
/// read "You're offline").
fn proton_kind(e: &proton_sdk::ProtonError) -> ErrorKind {
use proton_sdk::ProtonError as P;
match e {
P::Api(api) => api_kind(api.code),
P::Transport(_) => ErrorKind::Offline,
// Proton answered, but not with anything this client can read.
P::Serialization(_) => ErrorKind::Remote,
// The SDK refused the call before sending it, such as an upload to a
// photo library the account does not have.
P::InvalidOperation(_) => ErrorKind::Invalid,
P::Crypto(_) | P::KeysLocked => ErrorKind::Internal,
}
}
/// What an API response code means to a front-end.
///
/// The grouping is by *what the user can do about it*, which is what
@@ -266,6 +296,32 @@ fn api_kind(code: proton_sdk::api::ResponseCode) -> ErrorKind {
}
}
/// Whether `e` says the session itself is gone: its refresh token was refused,
/// or the account was deleted or disabled.
///
/// Nothing but a new login helps then, and no retry will ever succeed, so a
/// caller must not take it for a lost connection and wait for it to come back
/// (issue #35: a revoked session was shown as "offline" for good).
pub fn session_revoked(e: &proton_sdk::ProtonError) -> bool {
use proton_sdk::api::ResponseCode as C;
match e {
proton_sdk::ProtonError::Api(api) => {
api.is_invalid_refresh_token()
|| matches!(api.code, C::AccountDeleted | C::AccountDisabled)
}
_ => false,
}
}
/// Whether `e` is Proton's temporary block on a sign-in (code 2028).
///
/// Its message says the app version is no longer supported, which is not what
/// happened: the release is current and only waiting helps (issues #28, #39).
/// A front-end must say that instead of passing the message through.
pub fn sign_in_blocked(e: &proton_sdk::ProtonError) -> bool {
matches!(e, proton_sdk::ProtonError::Api(api) if api.code == proton_sdk::api::ResponseCode::Blocked)
}
/// The result every request-serving method in the daemon returns.
pub type CoreResult<T> = std::result::Result<T, CoreError>;
@@ -297,6 +353,25 @@ mod tests {
assert_eq!(api_kind(C::ProtonDriveUnknown), ErrorKind::Remote);
}
#[test]
fn only_a_refused_refresh_or_a_closed_account_revokes_the_session() {
assert!(session_revoked(&api(C::InvalidRefreshToken)));
assert!(session_revoked(&api(C::AccountDeleted)));
assert!(session_revoked(&api(C::AccountDisabled)));
// One file the account may not touch says nothing about the session.
assert!(!session_revoked(&api(C::Forbidden)));
assert!(!session_revoked(&api(C::Timeout)));
assert!(!session_revoked(&ProtonError::KeysLocked));
}
#[test]
fn only_code_2028_is_a_blocked_sign_in() {
assert!(sign_in_blocked(&api(C::Blocked)));
assert!(!sign_in_blocked(&api(C::OutdatedApp)));
assert!(!sign_in_blocked(&api(C::TooManyRequests)));
assert!(!sign_in_blocked(&ProtonError::KeysLocked));
}
/// The regression this whole pass exists to prevent: before it, every SDK
/// failure was hardcoded to `Remote`, so a 403 asked the user to retry a
/// request that could never succeed.
@@ -327,6 +402,18 @@ mod tests {
assert_eq!(CoreError::from_api(&io, "list").kind, ErrorKind::Offline);
}
/// Issue #38: the SDK refused a photo upload before sending it, and the
/// app said "You're offline" while every other call went through.
#[test]
fn a_call_the_sdk_refuses_to_send_is_not_offline() {
let refused = ProtonError::invalid_operation("account has no photos volume");
let e = CoreError::from_api(&refused, "upload photo failed");
assert_eq!(e.kind, ErrorKind::Invalid);
assert!(e.message.ends_with("account has no photos volume"));
let e: CoreError = Error::Proton(ProtonError::KeysLocked).into();
assert_ne!(e.kind, ErrorKind::Offline);
}
/// `Error` is the plumbing type; converting it must classify as sharply as
/// the direct path, or the two disagree about the same failure.
#[test]
+133
View File
@@ -0,0 +1,133 @@
//! When a photo was taken, as its own EXIF tags say.
//!
//! The fallback for a photo nothing else dates: a Takeout photo without a
//! sidecar or a date in its name, and a photo uploaded from the app. Without
//! it the first was filed under the moment of the import and the second under
//! the file's modification time, which a copy resets.
use std::io::{BufRead, Seek};
use std::path::Path;
use crate::takeout::days_from_civil;
/// The capture time in the EXIF of the file at `path`, in epoch seconds.
pub fn capture_time_of(path: &Path) -> Option<i64> {
let file = std::fs::File::open(path).ok()?;
capture_time(&mut std::io::BufReader::new(file))
}
/// The capture time in the EXIF `reader` holds: `DateTimeOriginal`, then
/// `DateTimeDigitized`, each adjusted by its `OffsetTime…` tag when the camera
/// wrote one. Without an offset the time is read as UTC, as a date in a file
/// name is (`takeout::capture_time_from_name`).
pub fn capture_time(reader: &mut (impl BufRead + Seek)) -> Option<i64> {
let exif = exif::Reader::new().read_from_container(reader).ok()?;
let ascii = |tag: exif::Tag| -> Option<String> {
let field = exif.get_field(tag, exif::In::PRIMARY)?;
match &field.value {
exif::Value::Ascii(parts) => parts.first().map(|part| {
String::from_utf8_lossy(part)
.trim_matches('\0')
.trim()
.to_string()
}),
_ => None,
}
};
[
(exif::Tag::DateTimeOriginal, exif::Tag::OffsetTimeOriginal),
(exif::Tag::DateTimeDigitized, exif::Tag::OffsetTimeDigitized),
]
.into_iter()
.find_map(|(time, offset)| {
let local = parse_exif_time(&ascii(time)?)?;
let offset = ascii(offset).and_then(|o| parse_offset(&o)).unwrap_or(0);
Some(local - offset)
})
}
/// `YYYY:MM:DD HH:MM:SS` as epoch seconds, read as UTC. `None` for the
/// all-zero or blank value cameras write when they do not know.
fn parse_exif_time(text: &str) -> Option<i64> {
let bytes = text.as_bytes();
if bytes.len() < 19 || bytes[4] != b':' || bytes[7] != b':' || bytes[13] != b':' {
return None;
}
let number = |range: std::ops::Range<usize>| text.get(range)?.parse::<i64>().ok();
let (year, month, day) = (number(0..4)?, number(5..7)?, number(8..10)?);
let (hour, minute, second) = (number(11..13)?, number(14..16)?, number(17..19)?);
if !(1800..=9999).contains(&year)
|| !(1..=12).contains(&month)
|| !(1..=31).contains(&day)
|| hour > 23
|| minute > 59
|| second > 60
{
return None;
}
Some(days_from_civil(year, month, day) * 86_400 + hour * 3600 + minute * 60 + second)
}
/// `+HH:MM` or `-HH:MM` as seconds east of UTC.
fn parse_offset(text: &str) -> Option<i64> {
let sign = match text.as_bytes().first()? {
b'+' => 1,
b'-' => -1,
_ => return None,
};
let (hours, minutes) = text.get(1..)?.split_once(':')?;
let (hours, minutes) = (hours.parse::<i64>().ok()?, minutes.parse::<i64>().ok()?);
(hours <= 14 && minutes < 60).then_some(sign * (hours * 3600 + minutes * 60))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn an_exif_time_reads_as_utc_and_its_offset_moves_it() {
// 2023-06-01 12:00:00 UTC.
let noon = 1_685_620_800;
assert_eq!(parse_exif_time("2023:06:01 12:00:00"), Some(noon));
assert_eq!(parse_offset("+02:00"), Some(7200));
assert_eq!(parse_offset("-05:30"), Some(-19_800));
assert_eq!(
parse_exif_time("2023:06:01 14:00:00").map(|t| t - parse_offset("+02:00").unwrap()),
Some(noon)
);
}
#[test]
fn an_unknown_exif_time_is_no_time() {
assert_eq!(parse_exif_time("0000:00:00 00:00:00"), None);
assert_eq!(parse_exif_time(" : : : : "), None);
assert_eq!(parse_exif_time(""), None);
assert_eq!(parse_offset("02:00"), None);
}
#[test]
fn a_photo_is_dated_by_its_original_time_and_offset() {
let ascii = |tag, text: &str| exif::Field {
tag,
ifd_num: exif::In::PRIMARY,
value: exif::Value::Ascii(vec![text.as_bytes().to_vec()]),
};
let taken = ascii(exif::Tag::DateTimeOriginal, "2023:06:01 14:00:00");
let offset = ascii(exif::Tag::OffsetTimeOriginal, "+02:00");
let digitized = ascii(exif::Tag::DateTimeDigitized, "2024:01:01 00:00:00");
let mut writer = exif::experimental::Writer::new();
writer.push_field(&taken);
writer.push_field(&offset);
writer.push_field(&digitized);
let mut tiff = std::io::Cursor::new(Vec::new());
writer.write(&mut tiff, false).unwrap();
tiff.set_position(0);
assert_eq!(capture_time(&mut tiff), Some(1_685_620_800));
}
#[test]
fn a_file_without_exif_has_no_capture_time() {
let mut png = std::io::Cursor::new(b"\x89PNG\r\n\x1a\n".to_vec());
assert_eq!(capture_time(&mut png), None);
}
}
+233
View File
@@ -0,0 +1,233 @@
//! A secret held in the Linux kernel keyring, for as long as the user is logged in.
//!
//! This is where the key passphrases of a two-password account live by default:
//! the mailbox password is never stored, and the secrets derived from it are
//! kept in kernel memory — not on disk, not in the Secret Service — and vanish
//! when the user's last session ends or the machine reboots. Both the daemon and
//! `pdfs unlock` reach the same entry because it hangs off the *user* keyring.
//! It is also where session tokens wait while the Secret Service refuses them
//! ([`UNSAVED_TOKENS`]), so they outlive the process that holds them.
//!
//! Kernel keys are not paged to swap, are invisible to other users, and are
//! wiped by [`clear`] (`KEYCTL_REVOKE`), which makes them a better home than a
//! process's heap. Processes of the same user can still read them, exactly as
//! they could ptrace the daemon: this protects against disk, backup and other
//! users, not against code already running as you.
//!
//! The keyring may be unavailable (a container, a kernel built without
//! `CONFIG_KEYS`); every operation then reports it as an error and the caller
//! falls back to asking again, never to storing the secret somewhere weaker.
use std::io;
use zeroize::Zeroizing;
/// One secret in the user keyring, by its description. Descriptions are
/// namespaced so they cannot collide with another program's keys.
pub struct Key {
description: &'static [u8],
}
/// The account's key passphrases (see the module docs).
#[cfg(not(test))]
pub const PASSPHRASES: Key = Key {
description: b"pdfs:key-passphrases\0",
};
/// Tests must never touch a real unlock.
#[cfg(test)]
pub const PASSPHRASES: Key = Key {
description: b"pdfs:key-passphrases:test\0",
};
/// Session tokens the Secret Service refused to store, until it takes them.
/// A refresh token held only in a process's memory dies with the process.
#[cfg(not(test))]
pub const UNSAVED_TOKENS: Key = Key {
description: b"pdfs:unsaved-tokens\0",
};
#[cfg(test)]
pub const UNSAVED_TOKENS: Key = Key {
description: b"pdfs:unsaved-tokens:test\0",
};
const KEY_TYPE: &[u8] = b"user\0";
const KEY_SPEC_USER_KEYRING: i32 = -4;
const KEYCTL_REVOKE: libc::c_long = 3;
const KEYCTL_SETPERM: libc::c_long = 5;
const KEYCTL_SEARCH: libc::c_long = 10;
const KEYCTL_READ: libc::c_long = 11;
/// Possessor and owner may view, read, write, search, link and set attributes;
/// group and others get nothing.
///
/// A new key's default grants the owner only *view*, and read access to the
/// possessor. A process possesses the user keyring only when its session
/// keyring links to it, which a systemd user service or a container does not
/// guarantee — so with the default, the daemon could store a key it then
/// cannot read back. The owner is the same user that could ptrace the daemon
/// anyway, so granting it read access gives up nothing.
const PERM_OWNER_AND_POSSESSOR: u32 = 0x3f3f_0000;
fn last_error<T>() -> io::Result<T> {
Err(io::Error::last_os_error())
}
fn find(description: &[u8]) -> io::Result<Option<libc::c_long>> {
// SAFETY: both strings are NUL-terminated and outlive the call.
let id = unsafe {
libc::syscall(
libc::SYS_keyctl,
KEYCTL_SEARCH,
KEY_SPEC_USER_KEYRING,
KEY_TYPE.as_ptr(),
description.as_ptr(),
0,
)
};
if id >= 0 {
return Ok(Some(id));
}
let err = io::Error::last_os_error();
match err.raw_os_error() {
// ENOKEY: not stored. EKEYREVOKED/EKEYEXPIRED: stored once, now gone.
Some(libc::ENOKEY | libc::EKEYREVOKED | libc::EKEYEXPIRED) => Ok(None),
_ => Err(err),
}
}
/// Stop this process from being core-dumped or ptraced by an unprivileged
/// process of the same user, so the secrets in its memory stay out of crash
/// reports. Call once at start of a long-lived process that holds them.
pub fn forbid_dumps() {
// SAFETY: `PR_SET_DUMPABLE` takes plain integers and only affects this process.
let rc = unsafe { libc::prctl(libc::PR_SET_DUMPABLE, 0, 0, 0, 0) };
if rc != 0 {
tracing::warn!(error = %io::Error::last_os_error(), "could not mark the process non-dumpable");
}
}
impl Key {
/// Store `secret`, replacing any earlier one.
pub fn store(&self, secret: &[u8]) -> io::Result<()> {
// SAFETY: strings are NUL-terminated; the payload pointer/length pair
// describes `secret`, which outlives the call.
let id = unsafe {
libc::syscall(
libc::SYS_add_key,
KEY_TYPE.as_ptr(),
self.description.as_ptr(),
secret.as_ptr(),
secret.len(),
KEY_SPEC_USER_KEYRING,
)
};
if id < 0 {
return last_error();
}
// SAFETY: plain integer arguments.
let rc = unsafe {
libc::syscall(
libc::SYS_keyctl,
KEYCTL_SETPERM,
id,
PERM_OWNER_AND_POSSESSOR,
)
};
if rc < 0 {
// A key whose permissions are unknown is not one to leave behind.
let err = io::Error::last_os_error();
let _ = self.clear();
return Err(err);
}
Ok(())
}
/// Read the stored secret, or `None` when there is none.
pub fn load(&self) -> io::Result<Option<Zeroizing<Vec<u8>>>> {
let Some(id) = find(self.description)? else {
return Ok(None);
};
// Size first, then read; retry if the key was replaced in between.
for _ in 0..3 {
// SAFETY: a null buffer asks the kernel only for the payload length.
let len = unsafe {
libc::syscall(
libc::SYS_keyctl,
KEYCTL_READ,
id,
std::ptr::null_mut::<u8>(),
0usize,
)
};
if len < 0 {
return match io::Error::last_os_error().raw_os_error() {
Some(libc::ENOKEY | libc::EKEYREVOKED | libc::EKEYEXPIRED) => Ok(None),
_ => last_error(),
};
}
let mut buf = Zeroizing::new(vec![0u8; len as usize]);
// SAFETY: `buf` is `len` writable bytes.
let read = unsafe {
libc::syscall(
libc::SYS_keyctl,
KEYCTL_READ,
id,
buf.as_mut_ptr(),
buf.len(),
)
};
if read < 0 {
return last_error();
}
if read as usize <= buf.len() {
buf.truncate(read as usize);
return Ok(Some(buf));
}
}
Err(io::Error::other("the key changed while it was being read"))
}
/// Remove the stored secret. Succeeds when there is none.
pub fn clear(&self) -> io::Result<()> {
let Some(id) = find(self.description)? else {
return Ok(());
};
// SAFETY: plain integer arguments.
let rc = unsafe { libc::syscall(libc::SYS_keyctl, KEYCTL_REVOKE, id) };
if rc < 0 {
return match io::Error::last_os_error().raw_os_error() {
Some(libc::ENOKEY | libc::EKEYREVOKED) => Ok(()),
_ => last_error(),
};
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
// The kernel keyring is process-wide state and a sandbox may not offer it,
// so one test covers the whole round trip and skips when it is unavailable.
#[test]
fn stored_secret_round_trips_and_clears() {
if PASSPHRASES.store(b"probe").is_err() {
eprintln!("kernel keyring unavailable; skipping");
return;
}
let saved = PASSPHRASES
.load()
.unwrap()
.expect("stored secret is readable");
assert_eq!(saved.as_slice(), b"probe");
PASSPHRASES.store(b"replaced").unwrap();
assert_eq!(PASSPHRASES.load().unwrap().unwrap().as_slice(), b"replaced");
PASSPHRASES.clear().unwrap();
assert!(PASSPHRASES.load().unwrap().is_none());
PASSPHRASES.clear().unwrap();
}
}
+4
View File
@@ -10,6 +10,8 @@ pub mod config;
pub mod control;
pub mod db;
pub mod error;
pub mod exiftime;
pub mod kernelkey;
pub mod localindex;
pub mod menu;
pub mod mounts;
@@ -22,8 +24,10 @@ pub mod service;
pub mod shell;
pub mod similar;
pub mod syncignore;
pub mod syncroot;
pub mod takeout;
pub mod tray;
pub mod update;
pub use access::{Access, access_for, perm_bits};
pub use error::{CoreError, CoreResult, Error, Result};
+1 -1
View File
@@ -42,7 +42,7 @@ const SKIP_PATH_SUFFIXES: &[&str] = &["go/pkg/mod"];
/// Hard cap on indexed entries, so a pathological home directory cannot grow the
/// database without bound. Reaching it stops the walk early.
const MAX_ENTRIES: usize = 400_000;
pub const MAX_ENTRIES: usize = 400_000;
/// Rows handed to the sink at a time. Large enough that the per-transaction cost
/// amortises, small enough that the writer lock is released often (FUSE
+27 -2
View File
@@ -189,11 +189,21 @@ fn spawn(argv: &[String], path: &Path) {
.stderr(Stdio::null())
.process_group(0)
.spawn();
if let Err(e) = spawned {
tracing::error!(command = ?argv, "open failed: {e}");
match spawned {
Ok(child) => reap(child),
Err(e) => tracing::error!(command = ?argv, "open failed: {e}"),
}
}
/// Wait for `child` on a thread of its own. A child nobody waits for stays a
/// zombie for as long as its parent lives, which for the app and the tray is
/// the whole session.
pub fn reap(mut child: std::process::Child) {
std::thread::spawn(move || {
let _ = child.wait();
});
}
/// [`open`] with the policy from the user's `config.json`, read once per
/// process. For callers that open the occasional path and have no reason to
/// carry a policy around; long-lived windows should hold their own copy.
@@ -472,6 +482,21 @@ fn glob_match(pattern: &str, name: &str) -> bool {
mod tests {
use super::*;
#[test]
fn a_reaped_child_does_not_stay_a_zombie() {
let child = std::process::Command::new("true").spawn().unwrap();
let proc_entry = std::path::PathBuf::from(format!("/proc/{}", child.id()));
reap(child);
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5);
while proc_entry.exists() && std::time::Instant::now() < deadline {
std::thread::sleep(std::time::Duration::from_millis(10));
}
assert!(
!proc_entry.exists(),
"the child is still in the process table"
);
}
fn rule(patterns: &[&str], command: &[&str], terminal: bool) -> OpenRule {
OpenRule {
patterns: patterns.iter().map(|p| (*p).to_string()).collect(),
+94 -2
View File
@@ -1,4 +1,4 @@
//! The machine profile that travels with a device (features.md 5.3).
//! The machine profile that travels with a device.
//!
//! Sync folder mappings, pins, and the settings that shape them live in
//! `cache.db` and `config.json` — both on the local disk. When that disk dies,
@@ -19,9 +19,31 @@ use serde::{Deserialize, Serialize};
use crate::mounts::MountMode;
/// File name of the profile.
/// File name of the profile clients before 3.5.0 wrote, one per device.
///
/// Read, never written: two computers sharing a device each overwrote the
/// other's. See [`profile_file_name`].
pub const PROFILE_FILE_NAME: &str = "profile.json";
/// File name of one machine's profile: `profile-<machine id>.json`.
///
/// One per machine, because several computers may sync into one device, and
/// each describes its own arrangement. A single shared file went to whichever
/// computer saved last.
pub fn profile_file_name(machine_id: &str) -> String {
format!("profile-{machine_id}.json")
}
/// Whether `name` is a profile: [`PROFILE_FILE_NAME`] or one machine's
/// [`profile_file_name`].
pub fn is_profile_file_name(name: &str) -> bool {
name == PROFILE_FILE_NAME
|| name
.strip_prefix("profile-")
.and_then(|rest| rest.strip_suffix(".json"))
.is_some_and(|id| !id.is_empty() && id.bytes().all(|b| b.is_ascii_alphanumeric()))
}
/// Folder, directly under the device root, that holds the profile.
///
/// The API refuses `Cannot create file at the root of a device` (HTTP 422), so
@@ -74,6 +96,10 @@ pub struct Profile {
/// Hostname of the machine that wrote it, so a restore prompt can say
/// *"restore settings from laptop-xyz?"* rather than quoting a uid.
pub hostname: String,
/// The writing machine's id, as in its [`profile_file_name`]. `None` in a
/// profile written before 3.5.0.
#[serde(default)]
pub machine_id: Option<String>,
/// When it was written, epoch seconds.
pub saved_at: i64,
#[serde(default)]
@@ -92,6 +118,25 @@ pub struct Profile {
pub mountpoint: Option<String>,
}
/// The saved arrangement of the folder with uid `remote_uid`, from the
/// profiles of every computer that syncs the device.
///
/// The profile of the machine named `hostname` wins, since it describes this
/// computer; otherwise the most recently saved one that has the folder.
pub fn saved_folder<'a>(
profiles: &'a [Profile],
remote_uid: &str,
hostname: &str,
) -> Option<&'a ProfileFolder> {
let holding = |p: &&'a Profile| p.folders.iter().any(|f| f.remote_uid == remote_uid);
let profile = profiles
.iter()
.filter(holding)
.find(|p| p.hostname == hostname)
.or_else(|| profiles.iter().filter(holding).max_by_key(|p| p.saved_at))?;
profile.folders.iter().find(|f| f.remote_uid == remote_uid)
}
impl Profile {
/// Parse a profile document, refusing one written by a newer client.
pub fn parse(bytes: &[u8]) -> Result<Self, String> {
@@ -122,6 +167,7 @@ mod tests {
version: PROFILE_VERSION,
device_uid: "dev-1".to_string(),
hostname: "laptop-xyz".to_string(),
machine_id: Some("0123456789abcdef0123456789abcdef".to_string()),
saved_at: 1_700_000_000,
folders: vec![ProfileFolder {
remote_uid: "vol~link".to_string(),
@@ -219,6 +265,52 @@ mod tests {
assert!(p.ignore_patterns.is_none());
}
#[test]
fn a_profile_name_is_the_legacy_one_or_one_per_machine() {
assert!(is_profile_file_name(PROFILE_FILE_NAME));
let name = profile_file_name("0123456789abcdef0123456789abcdef");
assert_eq!(name, "profile-0123456789abcdef0123456789abcdef.json");
assert!(is_profile_file_name(&name));
assert!(!is_profile_file_name("profile-.json"));
assert!(!is_profile_file_name("profile-a/b.json"));
assert!(!is_profile_file_name("profile (1).json"));
assert!(!is_profile_file_name("notes.json"));
}
/// A profile written before 3.5.0 has no machine id and still parses.
#[test]
fn a_profile_without_a_machine_id_parses() {
let json = br#"{"version":1,"device_uid":"d","hostname":"h","saved_at":0}"#;
assert_eq!(Profile::parse(json).unwrap().machine_id, None);
}
/// Several computers sync one device, each with its own profile. This
/// computer's own wins, then the newest that has the folder.
#[test]
fn the_saved_folder_comes_from_this_computer_then_the_newest_profile() {
let profile = |hostname: &str, saved_at: i64, path: &str| Profile {
hostname: hostname.to_string(),
saved_at,
folders: vec![ProfileFolder {
remote_uid: "vol~docs".to_string(),
local_path: path.to_string(),
mode: MountMode::Mirror,
}],
..sample()
};
let profiles = vec![
profile("desktop", 10, "/home/a/Docs"),
profile("laptop", 5, "/home/b/Docs"),
profile("old", 1, "/home/c/Docs"),
];
let path =
|host: &str| saved_folder(&profiles, "vol~docs", host).map(|f| f.local_path.as_str());
assert_eq!(path("laptop"), Some("/home/b/Docs"));
assert_eq!(path("new-machine"), Some("/home/a/Docs"));
assert_eq!(saved_folder(&profiles, "vol~other", "laptop"), None);
assert_eq!(saved_folder(&[], "vol~docs", "laptop"), None);
}
#[test]
fn future_version_is_refused() {
let mut p = sample();
+16 -1
View File
@@ -23,6 +23,11 @@ const SCRIPTS: [(&str, &str); 2] = [
/// `NAUTILUS_SCRIPT_SELECTED_FILE_PATHS`; the positional args carry the same
/// selection when the script runs on a local (non-URI) folder, so we prefer the
/// environment and fall back to `"$@"` for the Nemo/Caja variants.
///
/// The `--` keeps a file named like an option, such as `--json`, a path. A
/// name with a newline in it still arrives split in two from Nautilus, whose
/// list is newline-separated; `pdfs` then fails on both halves, which touches
/// nothing.
fn script_body(subcommand: &str) -> String {
format!(
"#!/bin/sh\n\
@@ -41,7 +46,7 @@ fn script_body(subcommand: &str) -> String {
\n\
printf '%s' \"$paths\" | while IFS= read -r path; do\n\
\x20 [ -n \"$path\" ] || continue\n\
\x20 pdfs {subcommand} \"$path\" || exit 1\n\
\x20 pdfs {subcommand} -- \"$path\" || exit 1\n\
done\n"
)
}
@@ -95,3 +100,13 @@ fn write_scripts(dir: &PathBuf) -> io::Result<()> {
tracing::debug!("installed file-manager scripts in {dir:?}");
Ok(())
}
#[cfg(test)]
mod tests {
use super::script_body;
#[test]
fn the_script_ends_options_before_each_path() {
assert!(script_body("pin").contains(r#"pdfs pin -- "$path""#));
}
}
+1 -1
View File
@@ -1,4 +1,4 @@
//! Selective-sync ignore rules for mirror folders (features.md 1.1).
//! Selective-sync ignore rules for mirror folders.
//!
//! A mirror folder otherwise uploads every path beneath its root, which for a
//! project directory means `node_modules/`, `.git/`, and `target/` — quota
+209
View File
@@ -0,0 +1,209 @@
//! Which local folders may become a sync root.
//!
//! A mirror uploads everything under its root and deletes what Drive deleted,
//! so a root that holds more than the user meant is dangerous. `$HOME` takes
//! the decrypted content cache, `cache.db` and `~/.ssh` up to Drive, and walks
//! into the mount, which downloads the whole Drive only to upload it again.
//! Every place a root is added or restored goes through [`problem`].
use std::path::{Component, Path, PathBuf};
use crate::config::{AppConfig, AppDirs};
/// The paths a sync root must stay clear of.
#[derive(Debug, Clone, Default)]
pub struct Reserved {
/// The user's home folder. A root may sit inside it, not be it or above it.
pub home: Option<PathBuf>,
/// The Proton Drive mount.
pub mountpoint: Option<PathBuf>,
/// This app's own configuration, state and cache folders.
pub app_dirs: Vec<PathBuf>,
}
impl Reserved {
/// The reserved paths of this user and `config`, resolved like a root.
pub fn current(config: &AppConfig) -> Self {
let Ok(dirs) = AppDirs::new() else {
return Self::default();
};
let resolved = |path: PathBuf| resolve(&path).unwrap_or(path);
let mut app_dirs = vec![dirs.state_dir(), dirs.cache_dir()];
if let Some(config_dir) = dirs.config_path().parent() {
app_dirs.push(config_dir.to_path_buf());
}
Self {
home: dirs.home_dir().map(resolved),
mountpoint: Some(resolved(dirs.resolved_mountpoint(config))),
app_dirs: app_dirs.into_iter().map(resolved).collect(),
}
}
}
/// `path` as a root is compared: absolute, with symlinks resolved as far as the
/// path exists. A folder a restore is about to create does not exist yet, so
/// the part past the deepest existing folder is kept as written.
///
/// A `..` is refused rather than resolved: before a symlink it means something
/// else than after it, and a path recorded on another machine has no business
/// holding one.
pub fn resolve(path: &Path) -> Result<PathBuf, String> {
if !path.is_absolute() {
return Err(format!("{} is not an absolute path", path.display()));
}
if path.components().any(|c| c == Component::ParentDir) {
return Err(format!("{} goes up with `..`", path.display()));
}
let mut existing = path.to_path_buf();
let mut rest = Vec::new();
loop {
match existing.canonicalize() {
Ok(canonical) => {
return Ok(rest
.into_iter()
.rev()
.fold(canonical, |path: PathBuf, part| path.join(part)));
}
Err(_) => {
let Some(name) = existing.file_name().map(|n| n.to_os_string()) else {
return Ok(path.to_path_buf());
};
rest.push(name);
if !existing.pop() {
return Ok(path.to_path_buf());
}
}
}
}
}
/// Why `root` cannot be synced, or `None` when it can. `root` is resolved
/// ([`resolve`]); `others` are the roots already synced.
pub fn problem(root: &Path, others: &[PathBuf], reserved: &Reserved) -> Option<String> {
let shown = root.display();
if root.parent().is_none() {
return Some(format!("{shown} is the root of the file system"));
}
if let Some(home) = &reserved.home {
if root == home {
return Some(format!(
"{shown} is your home folder; pick a folder inside it"
));
}
if home.starts_with(root) {
return Some(format!("{shown} holds your home folder"));
}
}
if let Some(mountpoint) = &reserved.mountpoint
&& overlaps(root, mountpoint)
{
return Some(format!(
"{shown} overlaps the Proton Drive folder {}",
mountpoint.display()
));
}
if let Some(dir) = reserved.app_dirs.iter().find(|dir| overlaps(root, dir)) {
return Some(format!(
"{shown} overlaps this app's own data in {}",
dir.display()
));
}
if others.iter().any(|other| other == root) {
return Some(format!("{shown} is already synced"));
}
if let Some(other) = others.iter().find(|other| overlaps(root, other)) {
return Some(format!(
"{shown} overlaps the synced folder {}",
other.display()
));
}
None
}
/// Whether one of `a` and `b` is the other or inside it.
fn overlaps(a: &Path, b: &Path) -> bool {
a.starts_with(b) || b.starts_with(a)
}
#[cfg(test)]
mod tests {
use super::*;
fn reserved() -> Reserved {
Reserved {
home: Some("/home/u".into()),
mountpoint: Some("/home/u/ProtonDrive".into()),
app_dirs: vec![
"/home/u/.local/state/proton-drive-linux".into(),
"/home/u/.cache/proton-drive-linux".into(),
"/home/u/.config/proton-drive-linux".into(),
],
}
}
fn refused(root: &str, others: &[&str]) -> bool {
let others: Vec<PathBuf> = others.iter().map(PathBuf::from).collect();
problem(Path::new(root), &others, &reserved()).is_some()
}
#[test]
fn a_folder_inside_home_can_be_synced() {
assert!(!refused("/home/u/Documents", &[]));
assert!(!refused("/home/u/Pictures", &["/home/u/Documents"]));
assert!(!refused("/mnt/data/photos", &[]));
}
#[test]
fn the_file_system_root_home_and_its_parents_are_refused() {
assert!(refused("/", &[]));
assert!(refused("/home/u", &[]));
assert!(refused("/home", &[]));
}
#[test]
fn the_mount_and_the_apps_own_data_are_refused_either_way_round() {
assert!(refused("/home/u/ProtonDrive", &[]));
assert!(refused("/home/u/ProtonDrive/Documents", &[]));
assert!(refused("/home/u/.cache", &[]));
assert!(refused(
"/home/u/.local/state/proton-drive-linux/staging",
&[]
));
assert!(refused("/home/u/.config", &[]));
}
#[test]
fn a_root_inside_or_around_another_root_is_refused() {
assert!(refused("/home/u/Documents", &["/home/u/Documents"]));
assert!(refused("/home/u/Documents/Work", &["/home/u/Documents"]));
assert!(refused("/home/u/stuff", &["/home/u/stuff/Documents"]));
// A shared prefix of characters is not a shared folder.
assert!(!refused("/home/u/Documents2", &["/home/u/Documents"]));
}
#[test]
fn a_path_that_goes_up_or_is_relative_does_not_resolve() {
assert!(resolve(Path::new("/home/u/../../etc")).is_err());
assert!(resolve(Path::new("Documents")).is_err());
}
#[test]
fn a_folder_still_to_be_created_resolves_under_its_existing_parent() {
let base = std::env::temp_dir().join(format!("pdfs-syncroot-{}", std::process::id()));
std::fs::create_dir_all(&base).unwrap();
let canonical = base.canonicalize().unwrap();
assert_eq!(
resolve(&base.join("new/inner")).unwrap(),
canonical.join("new/inner")
);
let link = base.join("link");
let _ = std::fs::remove_file(&link);
std::os::unix::fs::symlink(&canonical, &link).unwrap();
assert_eq!(
resolve(&link.join("later")).unwrap(),
canonical.join("later"),
"a symlink is followed to where it points"
);
std::fs::remove_dir_all(&base).unwrap();
}
}
+251 -25
View File
@@ -346,6 +346,11 @@ fn assemble(entries: Vec<RawEntry>, sidecars: &HashMap<String, Vec<u8>>) -> Take
.insert(file.to_string(), body);
}
let indexes: HashMap<&str, SidecarIndex<'_>> = per_folder_sidecars
.iter()
.map(|(folder, in_folder)| (folder.as_str(), SidecarIndex::new(in_folder)))
.collect();
let mut scan = TakeoutScan::default();
let mut seen_albums: HashMap<String, ()> = HashMap::new();
@@ -364,9 +369,13 @@ fn assemble(entries: Vec<RawEntry>, sidecars: &HashMap<String, Vec<u8>>) -> Take
continue;
};
let sidecar = per_folder_sidecars
.get(&folder)
.and_then(|in_folder| find_sidecar(&file_name, in_folder))
let found = indexes
.get(folder.as_str())
.and_then(|index| find_sidecar(&file_name, index));
// An edited copy keeps its own name: the title is its original's.
let title_is_ours = found.is_some_and(|((rule, _), _)| rule != EDITED_COPY);
let sidecar = found
.map(|(_, body)| body)
.and_then(|body| serde_json::from_slice::<PhotoSidecar>(body).ok())
.unwrap_or_default();
@@ -388,6 +397,7 @@ fn assemble(entries: Vec<RawEntry>, sidecars: &HashMap<String, Vec<u8>>) -> Take
let name = sidecar
.title
.as_deref()
.filter(|_| title_is_ours)
.map(str::trim)
.filter(|title| !title.is_empty() && same_extension(title, &file_name))
.unwrap_or(&file_name)
@@ -598,7 +608,7 @@ pub fn capture_time_from_name(name: &str) -> Option<i64> {
/// Days between 1970-01-01 and a proleptic-Gregorian date (Howard Hinnant's
/// `days_from_civil`). Avoids pulling a date crate in for one conversion.
fn days_from_civil(year: i64, month: i64, day: i64) -> i64 {
pub(crate) fn days_from_civil(year: i64, month: i64, day: i64) -> i64 {
let year = year - i64::from(month <= 2);
let era = if year >= 0 { year } else { year - 399 } / 400;
let year_of_era = year - era * 400;
@@ -627,17 +637,106 @@ fn swap_duplicate_marker(file_name: &str) -> Option<String> {
Some(format!("{}{}{}", &file_name[..marker_start], tail, marker))
}
/// Find the sidecar belonging to `file_name` among the JSON files of its folder.
/// How long Google lets a sidecar's file name get, `.json` included. A longer
/// one is cut, into its suffix first and then into the photo name.
const SIDECAR_NAME_CAP: usize = 51;
/// The rank [`sidecar_rank`] gives an edited copy that shares its original's
/// sidecar. Its date is the original's, but its name is its own.
const EDITED_COPY: u8 = 4;
/// The sidecars of one folder by their `.json`-stripped stem, so a photo is
/// ranked only against those that can match it.
///
/// Every rule in [`sidecar_rank`] needs the stem to start with a prefix of the
/// photo name that the name itself determines, and a sorted map finds the
/// stems with a given prefix as one range. Ranking every photo against every
/// sidecar instead took 27 s for a folder of 10,000 photos.
struct SidecarIndex<'a> {
by_stem: BTreeMap<&'a str, Vec<(&'a str, &'a Vec<u8>)>>,
}
impl<'a> SidecarIndex<'a> {
fn new(in_folder: &'a BTreeMap<String, &'a Vec<u8>>) -> Self {
let mut by_stem: BTreeMap<&'a str, Vec<(&'a str, &'a Vec<u8>)>> = BTreeMap::new();
for (name, body) in in_folder {
by_stem
.entry(strip_json_suffix(name))
.or_default()
.push((name.as_str(), *body));
}
Self { by_stem }
}
/// Add every sidecar whose stem starts with `prefix` to `out`.
fn starting_with(&self, prefix: &str, out: &mut BTreeMap<&'a str, &'a Vec<u8>>) {
let from = (
std::ops::Bound::Included(prefix),
std::ops::Bound::Unbounded,
);
for (stem, sidecars) in self.by_stem.range::<str, _>(from) {
if !stem.starts_with(prefix) {
break;
}
out.extend(sidecars.iter().copied());
}
}
/// Add every sidecar whose stem is `head`, or starts with `head` and a `.`:
/// those that have `head` as a boundary ([`suffix_boundaries`]).
fn bounded_by(&self, head: &str, out: &mut BTreeMap<&'a str, &'a Vec<u8>>) {
if let Some(sidecars) = self.by_stem.get(head) {
out.extend(sidecars.iter().copied());
}
self.starting_with(&format!("{head}."), out);
}
/// Every sidecar [`sidecar_rank`] could match `file_name` to, by name. A
/// superset: each one is still ranked.
fn candidates(&self, file_name: &str) -> BTreeMap<&'a str, &'a Vec<u8>> {
let mut out = BTreeMap::new();
// Rules 0 and 2: the stem starts with the name, or with its swapped form.
self.starting_with(file_name, &mut out);
if let Some(swapped) = swap_duplicate_marker(file_name) {
self.starting_with(&swapped, &mut out);
}
// Rule 1: the stem of `A(N).B` starts with `A.B(N)`, so with `A.`.
for (paren, _) in file_name.match_indices('(') {
self.starting_with(&format!("{}.", &file_name[..paren]), &mut out);
}
// Rule 3: a boundary of the stem is a prefix of the name.
for end in file_name
.char_indices()
.map(|(index, _)| index)
.skip(1)
.chain([file_name.len()])
{
self.bounded_by(&file_name[..end], &mut out);
}
// Rule 4: `<original>-<word>.<ext>` has `<original>.<ext>` as a boundary.
if let Some((_, extension)) = file_name.rsplit_once('.') {
for (dash, _) in file_name.match_indices('-') {
self.bounded_by(&format!("{}.{extension}", &file_name[..dash]), &mut out);
}
}
out
}
}
/// Find the sidecar belonging to `file_name` among the JSON files of its folder,
/// with the rank it matched at.
///
/// Tried in order of confidence: the exact name, the `(N)`-swapped name, an
/// unknown trailing suffix, then a prefix match for the case where truncation
/// cut into the photo name itself. The best-ranked candidate in the folder wins.
/// unknown trailing suffix, a prefix match for the case where truncation cut
/// into the photo name itself, and last an edited copy's original. The
/// best-ranked candidate in the folder wins; of equal ranks, the first by
/// sidecar name.
fn find_sidecar<'a>(
file_name: &str,
in_folder: &BTreeMap<String, &'a Vec<u8>>,
) -> Option<&'a Vec<u8>> {
index: &SidecarIndex<'a>,
) -> Option<((u8, usize), &'a Vec<u8>)> {
let mut best: Option<((u8, usize), &'a Vec<u8>)> = None;
for (sidecar_name, body) in in_folder {
for (sidecar_name, body) in index.candidates(file_name) {
let Some(rank) = sidecar_rank(strip_json_suffix(sidecar_name), file_name) else {
continue;
};
@@ -645,7 +744,7 @@ fn find_sidecar<'a>(
best = Some((rank, body));
}
}
best.map(|(_, body)| body)
best
}
/// How well a sidecar's `.json`-stripped `stem` matches `file_name`, as a sort
@@ -683,19 +782,44 @@ fn sidecar_rank(stem: &str, file_name: &str) -> Option<(u8, usize)> {
}
// 3 — truncation ate into the photo name itself, leaving only a prefix of
// it. Every suffix boundary in the stem is tried, so this holds for a
// localized suffix too; the longest surviving prefix wins, because a folder
// with `IMG_1.jpg` and `IMG_12.jpg` must not let the shorter stem claim the
// longer photo.
let mut longest: Option<usize> = None;
for head in suffix_boundaries(stem) {
if !head.is_empty()
&& file_name.starts_with(head)
&& longest.is_none_or(|len| head.len() > len)
{
longest = Some(head.len());
// localized suffix too; the longest surviving prefix wins.
//
// Only for a sidecar whose name is as long as Google lets one get: a name
// that short of the cap was not cut, so a prefix of it is the name of a
// different photo. Without this, `IMG_1.jpg`'s sidecar gave its name and
// date to an `IMG_12.jpg` that had none (`docs/BUGS.md` B225).
if stem.chars().count() + ".json".len() >= SIDECAR_NAME_CAP - 4 {
let mut longest: Option<usize> = None;
for head in suffix_boundaries(stem) {
if !head.is_empty()
&& file_name.starts_with(head)
&& longest.is_none_or(|len| head.len() > len)
{
longest = Some(head.len());
}
}
if let Some(len) = longest {
return Some((3, usize::MAX - len));
}
}
longest.map(|len| (3, usize::MAX - len))
// 4 — an edited copy, `IMG_1-edited.jpg` (the word is localized), which
// Google exports without a sidecar of its own next to `IMG_1.jpg`'s.
suffix_boundaries(stem)
.filter_map(|head| head.rsplit_once('.'))
.any(|(original, extension)| is_edited_copy_of(file_name, original, extension))
.then_some((EDITED_COPY, 0))
}
/// Whether `file_name` is `<original>-<word>.<extension>`, the name Google
/// gives an edited copy of `<original>.<extension>`.
fn is_edited_copy_of(file_name: &str, original: &str, extension: &str) -> bool {
!original.is_empty()
&& file_name
.strip_prefix(original)
.and_then(|rest| rest.strip_prefix('-'))
.and_then(|rest| rest.strip_suffix(extension))
.and_then(|rest| rest.strip_suffix('.'))
.is_some_and(|word| !word.is_empty() && word.chars().all(char::is_alphabetic))
}
/// The stem plus each of its prefixes ending before a `.`, longest first — the
@@ -994,21 +1118,25 @@ mod tests {
#[test]
fn longest_prefix_wins_when_the_photo_name_itself_was_truncated() {
// Both sidecar names are at Google's cap, so both were cut into the
// photo name; the longer surviving prefix is the better claim.
let short = "Takeout/Google Photos/Photos from 2019/PXL_20190101_120000000.PORTRAIT_ORIGINAL_E.json";
let long = "Takeout/Google Photos/Photos from 2019/PXL_20190101_120000000.PORTRAIT_ORIGINAL_EXPO.json";
let mut sidecars = HashMap::new();
sidecar(
&mut sidecars,
"Takeout/Google Photos/Photos from 2019/IMG_1.json",
short,
r#"{"photoTakenTime":{"timestamp":"1"}}"#,
);
sidecar(
&mut sidecars,
"Takeout/Google Photos/Photos from 2019/IMG_12.json",
long,
r#"{"photoTakenTime":{"timestamp":"2"}}"#,
);
let scan = assemble(
vec![entry(
0,
"Takeout/Google Photos/Photos from 2019/IMG_12345.jpg",
"Takeout/Google Photos/Photos from 2019/PXL_20190101_120000000.PORTRAIT_ORIGINAL_EXPOSURE.jpg",
10,
)],
&sidecars,
@@ -1017,6 +1145,104 @@ mod tests {
assert_eq!(scan.photos[0].capture_time, Some(2));
}
/// A short sidecar name was not cut, so a prefix of it names another photo:
/// `IMG_1.jpg`'s sidecar gave its title and date to `IMG_12.jpg`
/// (`docs/BUGS.md` B225).
#[test]
fn a_photo_without_a_sidecar_does_not_take_a_shorter_named_siblings() {
let mut sidecars = HashMap::new();
sidecar(
&mut sidecars,
"Takeout/Google Photos/Photos from 2019/IMG_1.jpg.supplemental-metadata.json",
r#"{"title":"IMG_1.jpg","photoTakenTime":{"timestamp":"1"}}"#,
);
let scan = assemble(
vec![
entry(0, "Takeout/Google Photos/Photos from 2019/IMG_1.jpg", 10),
entry(0, "Takeout/Google Photos/Photos from 2019/IMG_12.jpg", 10),
],
&sidecars,
);
assert_eq!(scan.photos[0].capture_time, Some(1));
assert_eq!(scan.photos[1].name, "IMG_12.jpg");
assert_eq!(scan.photos[1].capture_time, None);
}
/// An edited copy has no sidecar of its own. It takes its original's date,
/// but keeps its own name rather than the original's title.
#[test]
fn an_edited_copy_takes_its_originals_date_and_keeps_its_name() {
let mut sidecars = HashMap::new();
sidecar(
&mut sidecars,
"Takeout/Google Fotos/Urlaub/IMG_2.jpg.ergänzende-Metadaten.json",
r#"{"title":"Holiday.jpg","photoTakenTime":{"timestamp":"7"}}"#,
);
let scan = assemble(
vec![
entry(0, "Takeout/Google Fotos/Urlaub/IMG_2.jpg", 10),
entry(0, "Takeout/Google Fotos/Urlaub/IMG_2-bearbeitet.jpg", 10),
],
&sidecars,
);
assert_eq!(scan.photos[0].name, "Holiday.jpg");
assert_eq!(scan.photos[1].name, "IMG_2-bearbeitet.jpg");
assert_eq!(scan.photos[1].capture_time, Some(7));
}
/// The index looks only at the sidecars that can match, and must find the
/// same one that ranking every sidecar in the folder finds.
#[test]
fn the_sidecar_index_finds_what_ranking_every_sidecar_finds() {
let long = "A_very_long_holiday_photo_name_from_the_beach_2019";
let names = [
"IMG_1.jpg".to_string(),
"IMG_12.jpg".to_string(),
"IMG_1(1).jpg".to_string(),
"IMG_1-edited.jpg".to_string(),
"IMG_1-bearbeitet.jpg".to_string(),
"my.photo(2).jpg".to_string(),
"x(1).jpg".to_string(),
"Ölbild.png".to_string(),
"PXL_20230101_101010.mp4".to_string(),
format!("{long}.jpg"),
format!("{long}_2.jpg"),
"nothing.heic".to_string(),
];
let sidecar_names = [
"IMG_1.jpg.supplemental-metadata.json".to_string(),
"IMG_1.jpg(1).supplemental-metadata.json".to_string(),
"IMG_12.jpg.ergänzende-Metadaten.json".to_string(),
"my.photo.jpg(2).json".to_string(),
"x.jpg(1).json".to_string(),
"Ölbild.png.suppl.json".to_string(),
"PXL_20230101_101010.mp4.JSON".to_string(),
format!("{}.json", &long[..44]),
format!("{long}.jp.json"),
"IMG.json".to_string(),
"unrelated.json".to_string(),
];
let bodies: Vec<Vec<u8>> = (0..sidecar_names.len())
.map(|i| i.to_string().into_bytes())
.collect();
let in_folder: BTreeMap<String, &Vec<u8>> =
sidecar_names.iter().cloned().zip(bodies.iter()).collect();
let index = SidecarIndex::new(&in_folder);
for name in &names {
let mut every: Option<((u8, usize), &Vec<u8>)> = None;
for (sidecar_name, body) in &in_folder {
if let Some(rank) = sidecar_rank(strip_json_suffix(sidecar_name), name)
&& every.as_ref().is_none_or(|(current, _)| rank < *current)
{
every = Some((rank, *body));
}
}
assert_eq!(find_sidecar(name, &index), every, "{name}");
}
}
#[test]
fn sidecar_title_replaces_a_mangled_on_disk_name() {
let mut sidecars = HashMap::new();
+178
View File
@@ -0,0 +1,178 @@
//! The opt-in check for a newer release ([`crate::config::AppConfig::update_check`]).
//!
//! Only the decisions live here: what GitHub's answer says, which version is
//! newer, when to ask again, and whether a package manager already owns
//! updates. The desktop app does the request itself.
use std::path::Path;
use std::time::Duration;
/// GitHub's latest published release. Drafts and pre-releases never show up
/// here, so a tagged release candidate does not reach anyone.
pub const RELEASES_LATEST_API: &str =
"https://api.github.com/repos/narrrl/proton-drive-linux/releases/latest";
/// How often the app asks, across restarts.
pub const CHECK_INTERVAL: Duration = Duration::from_secs(24 * 60 * 60);
/// Installed next to the binaries' `share/` by packages whose package manager
/// brings new releases (COPR, the AUR, Nix). The release `.deb`, `.rpm` and
/// `.tar.gz` do not ship it, since nothing updates them.
pub const MANAGED_MARKER: &str = "share/proton-drive-linux/managed-updates";
/// The version of the release in a `/releases/latest` answer, without the
/// tag's `v`. `None` for an answer that does not parse or a tag that is not
/// `X.Y.Z`, so a changed API can never claim an update.
pub fn latest_version(body: &[u8]) -> Option<String> {
let json: serde_json::Value = serde_json::from_slice(body).ok()?;
let tag = json.get("tag_name")?.as_str()?;
let version = tag.strip_prefix('v').unwrap_or(tag);
parse(version).map(|_| version.to_owned())
}
/// Whether `version` is a later release than `current`. Anything that is not
/// `X.Y.Z` on either side is not.
pub fn is_newer(version: &str, current: &str) -> bool {
matches!((parse(version), parse(current)), (Some(v), Some(c)) if v > c)
}
/// The release page of `version` on GitHub.
pub fn release_url(version: &str) -> String {
format!("https://github.com/narrrl/proton-drive-linux/releases/tag/v{version}")
}
/// `X.Y.Z` as numbers, so `3.10.0` sorts after `3.9.0`.
fn parse(version: &str) -> Option<(u64, u64, u64)> {
let mut parts = version.split('.').map(|p| p.parse::<u64>().ok());
let triple = (parts.next()??, parts.next()??, parts.next()??);
parts.next().is_none().then_some(triple)
}
/// How long until the next check is due, given the last one in Unix seconds.
/// Zero when it is due now. A clock set back past the last check counts as
/// due, so a wrong clock cannot stop checks.
pub fn next_check_in(last_checked: Option<i64>, now: i64) -> Duration {
let interval = CHECK_INTERVAL.as_secs() as i64;
match last_checked {
Some(last) if (0..interval).contains(&(now - last)) => {
Duration::from_secs((interval - (now - last)) as u64)
}
_ => Duration::ZERO,
}
}
/// Whether the binary at `exe` was installed by a package manager that updates
/// it, which [`MANAGED_MARKER`] next to its `bin/` says.
pub fn updates_managed(exe: &Path) -> bool {
exe.parent()
.and_then(Path::parent)
.is_some_and(|prefix| prefix.join(MANAGED_MARKER).exists())
}
#[cfg(test)]
mod tests {
use super::*;
fn latest(tag: &str) -> Vec<u8> {
format!(
r#"{{"tag_name":"{tag}","html_url":"https://github.com/narrrl/proton-drive-linux/releases/tag/{tag}","draft":false}}"#
)
.into_bytes()
}
#[test]
fn the_latest_release_is_read_without_its_v() {
assert_eq!(latest_version(&latest("v3.6.0")).as_deref(), Some("3.6.0"));
assert_eq!(latest_version(&latest("3.6.0")).as_deref(), Some("3.6.0"));
}
#[test]
fn an_answer_that_does_not_parse_has_no_version() {
assert_eq!(latest_version(b"not json"), None);
assert_eq!(latest_version(b"{}"), None);
assert_eq!(latest_version(br#"{"tag_name":7}"#), None);
assert_eq!(latest_version(&latest("nightly")), None);
assert_eq!(latest_version(&latest("v4.0.0-rc1")), None);
assert_eq!(latest_version(&latest("v4.0")), None);
assert_eq!(latest_version(&latest("v4.0.0.1")), None);
}
#[test]
fn only_a_later_release_is_newer() {
assert!(is_newer("3.6.0", "3.5.0"));
assert!(!is_newer("3.5.0", "3.5.0"));
assert!(!is_newer("3.4.1", "3.5.0"));
assert!(!is_newer("2.99.99", "3.0.0"));
}
#[test]
fn versions_compare_as_numbers() {
assert!(is_newer("3.10.0", "3.9.0"));
assert!(is_newer("3.5.10", "3.5.9"));
assert!(is_newer("4.0.0", "3.99.99"));
}
#[test]
fn a_version_that_is_not_x_y_z_is_never_newer() {
assert!(!is_newer("4.0.0", "3.5.0-dev"));
assert!(!is_newer("garbage", "3.5.0"));
}
#[test]
fn the_release_page_uses_the_tag() {
assert_eq!(
release_url("3.6.0"),
"https://github.com/narrrl/proton-drive-linux/releases/tag/v3.6.0"
);
}
#[test]
fn a_check_is_due_once_a_day() {
let day = CHECK_INTERVAL.as_secs() as i64;
assert_eq!(next_check_in(None, 1_000), Duration::ZERO);
assert_eq!(next_check_in(Some(1_000), 1_000), CHECK_INTERVAL);
assert_eq!(
next_check_in(Some(1_000), 1_000 + day - 1),
Duration::from_secs(1)
);
assert_eq!(next_check_in(Some(1_000), 1_000 + day), Duration::ZERO);
assert_eq!(
next_check_in(Some(1_000), 999),
Duration::ZERO,
"a clock set back must not stop checks"
);
}
/// A unique temp directory removed on drop; avoids a dev-dependency, as in
/// [`crate::syncignore`]'s tests.
struct TempDir(std::path::PathBuf);
impl TempDir {
fn new() -> Self {
use std::sync::atomic::{AtomicU64, Ordering};
static N: AtomicU64 = AtomicU64::new(0);
let p = std::env::temp_dir().join(format!(
"pdfs-update-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed)
));
std::fs::create_dir_all(&p).unwrap();
TempDir(p)
}
}
impl Drop for TempDir {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
#[test]
fn the_marker_next_to_bin_means_a_package_manager_updates() {
let prefix = TempDir::new();
let exe = prefix.0.join("bin/pdfs-app");
assert!(!updates_managed(&exe));
let marker = prefix.0.join(MANAGED_MARKER);
std::fs::create_dir_all(marker.parent().unwrap()).unwrap();
std::fs::write(&marker, "").unwrap();
assert!(updates_managed(&exe));
}
}
+26 -7
View File
@@ -2,6 +2,7 @@
use super::*;
use crate::shutdown::Shutdown;
use pdfs_core::error::session_revoked;
/// Drop `uid` from one mount's tree and tell that mount's kernel session the
/// entry is gone. Shared by the trash and delete paths, which differ only in
@@ -143,7 +144,7 @@ fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdf
// create, and re-fetching would replace the size and mtime of the
// write we just accepted with the stale revision's — making a file
// that was copied in seconds ago read as empty until its upload
// lands (offline.md Phase 3).
// lands.
//
// A write that has already drained is the same story one step later:
// the drain brought the tree level with the revision it sealed, so
@@ -375,7 +376,7 @@ pub(super) async fn run_event_sync(
// server head; persist it so the next restart resumes instead of
// reseeding (which would skip everything that changed offline).
// Seeding needs the network, and this task also runs on mounts that
// started offline (offline.md Phase 1) — so retry rather than giving up,
// started offline — so retry rather than giving up,
// which used to disable live sync for the life of the daemon.
Ok(None) => {
// Nothing says what changed before this cursor, so a persisted SDK
@@ -402,6 +403,12 @@ pub(super) async fn run_event_sync(
}
break Some(c.clone());
}
// Refused for good: retrying would be refused the same way
// for the life of the daemon. The next login restarts it.
Err(e) if session_revoked(&e) => {
core.mark_session_expired("seed event cursor", &e);
return;
}
Err(e) => {
warn!(error = %e, ?delay, "seed event cursor failed; retrying");
tokio::time::sleep(delay).await;
@@ -502,6 +509,10 @@ pub(super) async fn run_photos_event_sync(core: Core) {
debug!("no photos volume; photo event sync not started");
return;
}
Err(error) if session_revoked(&error) => {
core.mark_session_expired("read the photos root", &error);
return;
}
Err(error) => {
warn!(%error, "reading the photos root failed; retrying");
tokio::time::sleep(ONLINE_PROBE_MAX).await;
@@ -660,7 +671,7 @@ pub(super) fn run_local_index(
scan_local_once(&db, &indexing, &transfers, &mountpoint);
}
// Interruptible, so teardown joins this thread instead of leaving it to
// keep rewriting the index for a mount that is gone (bugs.md B44).
// keep rewriting the index for a mount that is gone (`docs/BUGS.md` B44).
if !shutdown.sleep(LOCAL_INDEX_CHECK) {
return;
}
@@ -710,18 +721,26 @@ fn scan_local_once(
// The walk has no idea how many files it will find, so the job counts what it
// has seen and stays indeterminate.
let job = transfers.begin_job("Indexing this computer");
let job = transfers.begin_background_job("Indexing this computer");
job.detail("Scanning your files");
let mut failed = false;
let walked = localindex::scan(&[home], &excludes, |batch| {
if let Err(e) = db.local_upsert_batch(generation, &batch) {
warn!(error = %e, "local index: batch write failed");
failed = true;
}
});
// Prune what this scan did not see and rebuild the FTS index over the rest,
// even if some batches failed — a partial index still beats none.
// Prune what this scan did not see, unless it did not see everything:
// a path it never reached, or one whose batch failed to write, is not a
// path that is gone (`docs/BUGS.md` B217).
job.detail("Building the search index");
match db.local_finish_scan(generation, now_secs()) {
let finished = if failed || walked >= localindex::MAX_ENTRIES {
db.local_finish_partial_scan(now_secs())
} else {
db.local_finish_scan(generation, now_secs())
};
match finished {
Ok(indexed) => info!(
walked,
indexed,
+40 -3
View File
@@ -334,6 +334,9 @@ pub(crate) fn status_response(core: &Core, username: &str, mountpoint: &Path) ->
budget: core.cache.budget(),
pins,
online: core.online.load(Ordering::Relaxed),
session_expired: core.session_expired(),
tokens_unsaved: pdfs_core::auth::tokens_unsaved(),
version: env!("CARGO_PKG_VERSION").to_string(),
pending_uploads: queued.uploads.max(0) as u64 + landing,
pending_changes: queued.changes.max(0) as u64,
}
@@ -369,7 +372,7 @@ fn touched_topics(request: &CtlRequest) -> &'static [Topic] {
RefreshScope::Photos { .. } => &[],
},
R::RenameDevice { .. } | R::DeleteDevice { .. } => &[Topic::Devices],
R::AdoptDevice { .. } => &[Topic::Devices, Topic::Locations],
R::AdoptDevice { .. } | R::MigrateDevice { .. } => &[Topic::Devices, Topic::Locations],
R::AddSyncFolder { .. }
| R::RemoveSyncFolder { .. }
| R::SetSyncFolderMode { .. }
@@ -377,6 +380,7 @@ fn touched_topics(request: &CtlRequest) -> &'static [Topic] {
| R::RestoreSyncFolders { .. }
| R::RestoreDeviceFolders { .. } => &[Topic::Locations, Topic::Devices],
R::RetryPendingOp { .. } => &[Topic::Queue],
R::DiscardPendingOp { .. } => &[Topic::Queue, Topic::Files],
R::ResolveConflict { .. } => &[Topic::Files, Topic::Trash, Topic::Conflicts],
R::ShareNode { .. }
| R::ShareNodeByUid { .. }
@@ -907,10 +911,22 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
))),
Ok((file, len)) => {
let photos = core.photos();
// Unless the caller knows better: the photo's own EXIF,
// then the file's modification time, which a copy resets.
let capture_time = capture_time.or_else(|| {
let path = Path::new(&source_path);
pdfs_core::exiftime::capture_time_of(path).or_else(|| {
let modified =
std::fs::metadata(path).and_then(|m| m.modified()).ok()?;
let since = modified.duration_since(std::time::UNIX_EPOCH).ok()?;
Some(since.as_secs() as i64)
})
});
let metadata = proton_drive_rs::PhotoUploadMetadata {
capture_time,
..Default::default()
};
let thumbnails = core.upload_thumbnails(Path::new(&source_path), &name);
let guard =
core.transfers
.begin(name.clone(), "", TransferDirection::Upload, len);
@@ -920,7 +936,7 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
&media_type,
reader,
len as i64,
Vec::new(),
thumbnails,
metadata,
false,
)) {
@@ -937,6 +953,9 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
}
}
Err(e) => {
// The reply goes to the app's toast only; without
// this the journal shows a failure and no reason.
warn!(error = %e, "photo upload failed");
CtlResponse::error(CoreError::from_api(&e, "upload photo failed"))
}
}
@@ -1296,6 +1315,10 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
Ok(message) => CtlResponse::Ok { message },
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::MigrateDevice { uid }) => match core.migrate_device(&uid) {
Ok(message) => CtlResponse::Ok { message },
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::AddSyncFolder { local_path }) => {
// Registering the device and uploading a folder tree far outlasts the
// socket read timeout, so ack immediately and work on a background
@@ -1404,6 +1427,20 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
Ok(items) => CtlResponse::PendingOps { items },
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::ExportPendingOp { id, dest }) => {
match core.export_pending_op(id, Path::new(&dest)) {
Ok(written) => CtlResponse::Ok {
message: format!("wrote {} to {dest}", human_bytes(written)),
},
Err(e) => CtlResponse::error(e),
}
}
Ok(CtlRequest::DiscardPendingOp { id }) => match core.discard_pending_op(id) {
Ok(path) => CtlResponse::Ok {
message: format!("discarded the queued change to {path}"),
},
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::RetryPendingOp { id }) => match core.retry_pending_ops(id) {
Ok(0) if id.is_some() => CtlResponse::Ok {
message: "nothing to retry: the operation is already due or waiting for a rename"
@@ -1801,7 +1838,7 @@ pub(crate) fn run_control_socket(
info!("control socket listening");
for conn in listener.incoming() {
// Checked on every accept, including the deliberate poke teardown sends
// to wake this thread out of a blocking accept (bugs.md B44).
// to wake this thread out of a blocking accept (`docs/BUGS.md` B44).
if core.shutdown.is_stopping() {
info!("control socket closing");
return;
+211 -31
View File
@@ -1,5 +1,5 @@
//! This machine as a Proton Drive Device, and the local folders it keeps in
//! step with the remote (devices.md).
//! step with the remote.
//!
//! A sync folder runs in one of two modes. `mirror` keeps a real local tree that
//! the engine in [`super::sync`] reconciles both ways; `ondemand` evicts that
@@ -19,6 +19,7 @@ use pdfs_core::control::{
};
use pdfs_core::db::{StoredDevice, StoredSyncFolder};
use pdfs_core::mounts::MountMode;
use pdfs_core::profile::PROFILE_DIR_NAME;
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::proton_sdk::ids::{DeviceUid, NodeUid};
use proton_drive_rs::{DeviceType, Node};
@@ -104,6 +105,15 @@ pub(crate) fn adopted_device_uid() -> Option<String> {
(!uid.is_empty()).then_some(uid)
}
/// The paths no sync root may overlap, with the mountpoint as the config says
/// now.
pub(crate) fn sync_root_reserved() -> pdfs_core::syncroot::Reserved {
let config = pdfs_core::config::AppDirs::new()
.map(|dirs| dirs.load_config())
.unwrap_or_default();
pdfs_core::syncroot::Reserved::current(&config)
}
/// Pin (or, with `None`, unpin) the adopted device uid in `config.json`.
pub(crate) fn set_adopted_device_uid(uid: Option<&str>) -> CoreResult<()> {
let dirs = pdfs_core::config::AppDirs::new()
@@ -114,6 +124,22 @@ pub(crate) fn set_adopted_device_uid(uid: Option<&str>) -> CoreResult<()> {
.map_err(|e| CoreError::internal(format!("write config: {e}")))
}
/// The name a folder moved in from `device` takes when `name` is already
/// taken under this device's root, or `None` when it can keep its own.
fn migrated_name(name: &str, device: &str, taken: &[String]) -> Option<String> {
let free = |candidate: &str| !taken.iter().any(|t| t == candidate);
if free(name) {
return None;
}
let base = format!("{name} ({device})");
if free(&base) {
return Some(base);
}
(2..)
.map(|n| format!("{base} {n}"))
.find(|candidate| free(candidate))
}
impl Core {
// ---- devices ----------------------------------------------------------
@@ -140,7 +166,7 @@ impl Core {
pub(crate) fn list_devices(&self) -> CoreResult<Vec<DeviceInfo>> {
let devices = self
.rt
.block_on(self.client.enumerate_devices())
.block_on(self.drive.enumerate_devices())
.map_err(|e| CoreError::from_api(&e, "list devices"))?;
// No cached device row yet means this machine syncs nothing, so none of
// the listed devices is ours.
@@ -184,8 +210,8 @@ impl Core {
}
/// Adopt an existing device as *this* machine's, pinning it in `config.json`
/// so hostname changes and reinstalls stop registering duplicates
/// (features.md 5.1). `None` clears the pin and returns to hostname matching.
/// so hostname changes and reinstalls stop registering duplicates.
/// `None` clears the pin and returns to hostname matching.
///
/// Refuses a uid that is not in the account: the whole point of the pin is
/// that it is authoritative, so accepting a bad one would defer the failure
@@ -197,7 +223,7 @@ impl Core {
};
let remote = self
.rt
.block_on(self.client.enumerate_devices())
.block_on(self.drive.enumerate_devices())
.map_err(|e| CoreError::from_api(&e, "enumerate devices"))?;
let dev = remote
.into_iter()
@@ -240,7 +266,7 @@ impl Core {
};
let remote = self
.rt
.block_on(self.client.enumerate_devices())
.block_on(self.drive.enumerate_devices())
.map_err(|e| CoreError::from_api(&e, "enumerate devices"))?;
let d = remote
.into_iter()
@@ -260,7 +286,108 @@ impl Core {
})
}
// ---- device folder sync (devices.md, Phase 1) -------------------------
/// Move every folder another device backs up into this machine's own
/// device, so this machine carries on backing them up as itself. The other
/// device stays registered, without those folders.
///
/// Device roots all live on the main volume, so each folder is one
/// server-side move: nothing is downloaded or uploaded, and no quota is
/// used twice. A folder whose name is taken here keeps its content under
/// "<name> (<device>)". A failed folder does not stop the others, and a
/// second run picks up whatever is still left on the other device.
///
/// Nothing is synced to this disk here. The moved folders are now this
/// device's, so the ordinary restore offers them. Only folders that were
/// already synced here from the other device are touched: their uid
/// survives the move but their share does not.
pub(crate) fn migrate_device(&self, uid: &str) -> CoreResult<String> {
let source = self.restore_source(Some(uid))?;
let target = self.ensure_device()?;
if source.uid == target.uid {
return Err(CoreError::invalid(format!(
"{} is this computer's own device",
source.name
)));
}
let bad_root = |d: &StoredDevice| {
parse_uid(&d.root_uid)
.ok_or_else(|| CoreError::internal(format!("bad device root uid: {}", d.root_uid)))
};
let (source_root, target_root) = (bad_root(&source)?, bad_root(&target)?);
if source_root.volume_id != target_root.volume_id {
return Err(CoreError::invalid(
"the two devices are on different Proton Drive volumes; copy it instead",
));
}
let mut taken: Vec<String> = self
.device_root_nodes(&target_root)?
.into_iter()
.filter(|n| !n.trashed)
.map(|n| n.name)
.collect();
let synced = self
.db
.sync_folder_list()
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?;
let mut moved = 0usize;
let mut renamed: Vec<String> = Vec::new();
let mut left: Vec<String> = Vec::new();
let mut failed: Vec<String> = Vec::new();
for node in self.device_root_nodes(&source_root)? {
if node.trashed || node.name == PROFILE_DIR_NAME {
continue;
}
// A device root only ever holds folders (B68); anything else was
// put there by hand and is left for the user to deal with.
if !node.is_folder() {
left.push(node.name);
continue;
}
let new_name = migrated_name(&node.name, &source.name, &taken);
if let Err(e) = self.move_rename_remote(&node.uid, &target_root, new_name.as_deref()) {
warn!(name = %node.name, error = %e, "migrate: could not move folder");
failed.push(format!("{}: {e}", node.name));
continue;
}
let landed = new_name.clone().unwrap_or_else(|| node.name.clone());
if new_name.is_some() {
renamed.push(format!("{} -> {landed}", node.name));
}
taken.push(landed);
moved += 1;
let uid = node.uid.to_string();
for folder in synced.iter().filter(|f| f.remote_uid == uid) {
if let Err(e) = self.db.sync_folder_set_share(folder.id, &target.share_id) {
warn!(id = folder.id, error = ?e, "migrate: could not re-point sync folder");
}
self.sync_now(Some(folder.id));
}
}
self.invalidate_children_of(&source_root);
self.invalidate_children_of(&target_root);
let mut message = format!("moved {moved} folder(s) from {}", source.name);
if !renamed.is_empty() {
message.push_str(&format!("; renamed: {}", renamed.join(", ")));
}
if !left.is_empty() {
message.push_str(&format!("; left behind: {}", left.join(", ")));
}
if !failed.is_empty() {
message.push_str(&format!("; failed: {}", failed.join(", ")));
}
info!(from = %source.name, to = %target.name, moved, failed = failed.len(),
"device migrated");
if moved == 0 && !failed.is_empty() {
return Err(CoreError::conflict(message));
}
Ok(message)
}
// ---- device folder sync -------------------------
/// Auto-register (or recover) this machine as a Proton Drive Device, caching
/// it so restarts reuse the same device. Recovery matches an existing remote
@@ -270,15 +397,14 @@ impl Core {
/// Resolution order: the *adopted* uid pinned in `config.json`, then the
/// cached DB row (validated remotely), then a hostname match, then create.
/// The pin exists because the hostname heuristic silently registers a second
/// device after a rename or reinstall, orphaning the first one's folders
/// (features.md 5.1).
/// device after a rename or reinstall, orphaning the first one's folders.
pub(crate) fn ensure_device(&self) -> CoreResult<StoredDevice> {
let name = this_hostname();
// Enumerate the remote devices once: used to validate the pin, validate
// any cached record, and recover an existing device by name.
let remote = self
.rt
.block_on(self.client.enumerate_devices())
.block_on(self.drive.enumerate_devices())
.map_err(|e| CoreError::from_api(&e, "enumerate devices"))?;
// An adopted uid is an explicit instruction, so it outranks everything
@@ -337,7 +463,7 @@ impl Core {
None => {
let d = self
.rt
.block_on(self.client.create_device(&name, DeviceType::Linux))
.block_on(self.drive.create_device(&name, DeviceType::Linux))
.map_err(|e| CoreError::from_api(&e, "create device"))?;
StoredDevice {
uid: d.uid.to_string(),
@@ -361,21 +487,25 @@ impl Core {
root_uid: &NodeUid,
name: &str,
) -> CoreResult<Option<NodeUid>> {
Ok(self
.device_root_nodes(root_uid)?
.into_iter()
.find(|n| n.is_folder() && !n.trashed && n.name == name)
.map(|n| n.uid))
}
/// Everything directly under a device root, trashed nodes included.
fn device_root_nodes(&self, root_uid: &NodeUid) -> CoreResult<Vec<Node>> {
let uids = self
.rt
.block_on(self.drive.enumerate_folder_children_node_uids(root_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() {
return Ok(None);
return Ok(Vec::new());
}
let nodes = self
.rt
self.rt
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?;
Ok(nodes
.into_iter()
.find(|n| n.is_folder() && !n.trashed && n.name == name)
.map(|n| n.uid))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))
}
/// Add a local folder to this device's sync set: register the device if
@@ -396,18 +526,20 @@ impl Core {
.map_err(|e| CoreError::internal(format!("canonicalize {}: {e}", local.display())))?;
let local_str = local.to_string_lossy().to_string();
// Reject duplicates up front for a clear error (UNIQUE would also catch it).
if self
// `/`, the home folder, the mount, this app's own data, and a folder
// inside or around another root are refused (`syncroot::problem`). A
// duplicate is one of those, refused here for a clear error before
// the UNIQUE constraint would catch it.
let others = self
.db
.sync_folder_list()
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?
.iter()
.any(|f| f.local_path == local_str)
{
return Err(CoreError::invalid(format!(
"{} is already synced",
local.display()
)));
.map(|f| PathBuf::from(&f.local_path))
.collect::<Vec<_>>();
let reserved = sync_root_reserved();
if let Some(problem) = pdfs_core::syncroot::problem(&local, &others, &reserved) {
return Err(CoreError::invalid(problem));
}
let name = local
@@ -536,12 +668,16 @@ impl Core {
} else {
p.total.max(p.done) as u64
},
// Most passes find nothing; only one that queues work is
// the drive syncing.
background: true,
},
SyncPhase::Applying => JobItem {
title: format!("Syncing {folder}"),
detail: p.current.clone(),
done: p.done as u64,
total: p.total.max(p.done) as u64,
background: false,
},
});
}
@@ -617,7 +753,7 @@ impl Core {
}
/// A sibling Core for a secondary FUSE session rooted at an `ondemand` sync
/// folder (devices.md Phase 3). It shares everything with this one,
/// folder. It shares everything with this one,
/// the tree included, except what belongs to a session: its root and its
/// kernel channel, both filled in when the session is built.
pub(crate) fn fork_state(&self) -> Core {
@@ -843,6 +979,23 @@ impl Core {
if folder.state != "idle" {
return Err(SwitchBlocked::NotNow);
}
// `idle` only says the last pass applied everything it tried.
// It never tries an ignored path, a symlink or a folder it
// cannot read, it defers a file open for writing, and a file can
// change after it ends. The eviction below deletes all of those
// for good, so it waits until nothing here is missing from Drive
// (B198).
let baseline = self
.db
.sync_entries(id)
.map_err(|e| SwitchBlocked::Failed(format!("db: {e:?}")))?;
if let Some(reason) =
sync::mirror_tree_unsynced(&local, &baseline, &self.ignore_rules(&local))
{
return Err(SwitchBlocked::Failed(format!(
"going online-only would delete what is only on this computer: {reason}"
)));
}
let root_uid = parse_uid(&folder.remote_uid).ok_or_else(|| {
SwitchBlocked::Failed(format!("bad remote uid: {}", folder.remote_uid))
})?;
@@ -986,8 +1139,7 @@ impl Core {
/// restart (their local dirs are empty on disk — the files live in the cloud).
/// Best-effort per folder: a missing local path or a failed remote fetch marks
/// the folder `error` and moves on rather than aborting the rest. Runs on its
/// own thread from `mount` so the network fetches never block startup
/// (devices.md Phase 4).
/// own thread from `mount` so the network fetches never block startup.
pub(crate) fn restore_ondemand_mounts(&self) {
let folders = match self.db.sync_folder_list() {
Ok(f) => f,
@@ -1109,10 +1261,38 @@ mod tests {
use pdfs_core::db::StoredSyncFolder;
use super::{
ApplicableMountMode, commit_after_teardown, pending_intent_matches,
ApplicableMountMode, commit_after_teardown, migrated_name, pending_intent_matches,
restore_snapshot_matches, select_mode_transition,
};
#[test]
fn migrated_folder_keeps_its_name_when_it_is_free() {
let taken = vec!["Pictures".to_string()];
assert_eq!(migrated_name("Documents", "DESKTOP-1", &taken), None);
}
#[test]
fn migrated_folder_is_named_after_its_device_on_a_clash() {
let taken = vec!["Documents".to_string()];
assert_eq!(
migrated_name("Documents", "DESKTOP-1", &taken).as_deref(),
Some("Documents (DESKTOP-1)")
);
}
#[test]
fn migrated_folder_is_numbered_when_the_device_name_is_taken_too() {
let taken = vec![
"Documents".to_string(),
"Documents (DESKTOP-1)".to_string(),
"Documents (DESKTOP-1) 2".to_string(),
];
assert_eq!(
migrated_name("Documents", "DESKTOP-1", &taken).as_deref(),
Some("Documents (DESKTOP-1) 3")
);
}
#[test]
fn unknown_mode_cannot_select_a_destructive_transition() {
let mut unmounts = 0;
File diff suppressed because it is too large. Load diff
+28 -4
View File
@@ -3,13 +3,15 @@
//! `Core` used to hold a concrete [`ProtonDriveClient`], so no test could make
//! Drive answer late, answer stale, or not answer at all, and the interleavings
//! behind most recent bugs could only be found on a real account over a slow
//! link (`docs/MILESTONE-3.0.0.md` §2, §8.1). [`DriveApi`] is the part of the
//! link. [`DriveApi`] is the part of the
//! client the mount, the drain, the event feed and the mirror engine use; the
//! real client implements it by forwarding, and the simulation tests implement
//! it with an in-memory Drive that injects Drive's known faults.
//!
//! Sharing, Photos, devices, invitations and public links stay on the concrete
//! client (`Core::client`) until a test needs them faked.
//! Sharing, Photos, invitations, public links and the rest of the device calls
//! stay on the concrete client (`Core::client`) until a test needs them faked.
//! Listing and registering devices are here, so a simulated daemon can sync a
//! local folder.
//!
//! The method names match the client's own, so a call site reads the same
//! whichever it goes through. The client's generic methods (a reader or writer
@@ -26,7 +28,8 @@ use proton_drive_rs::proton_sdk::account::Quota;
use proton_drive_rs::proton_sdk::error::Result;
use proton_drive_rs::proton_sdk::ids::{DriveEventId, NodeUid};
use proton_drive_rs::{
DriveEvent, DriveEventScopeId, Node, NodeMoveItem, ProtonDriveClient, RevisionReader, Thumbnail,
Device, DeviceType, DriveEvent, DriveEventScopeId, Node, NodeMoveItem, ProtonDriveClient,
RevisionReader, Thumbnail,
};
/// One outcome per node of a batch request; see [`ProtonDriveClient::trash_nodes`].
@@ -153,6 +156,19 @@ pub(crate) trait DriveApi: Send + Sync {
/// The account's storage, across every Proton product.
async fn quota(&self) -> Result<Quota>;
/// Every computer that backs up to the account.
async fn enumerate_devices(&self) -> Result<Vec<Device>>;
/// Register a computer, with a root folder of its own named `name`.
async fn create_device(&self, name: &str, device_type: DeviceType) -> Result<Device>;
/// Called by the drain once it has let go of the blob of an upload to
/// `uid` that landed, or retired the create that made `uid`, before the
/// tree has the node, so a simulation can hold it there. Blocks the drain
/// worker it is called on.
#[cfg(test)]
fn landing(&self, _uid: &NodeUid) {}
}
/// The generic signatures of the client, over the object-safe ones above.
@@ -463,4 +479,12 @@ impl DriveApi for ProtonDriveClient {
async fn quota(&self) -> Result<Quota> {
ProtonDriveClient::quota(self).await
}
async fn enumerate_devices(&self) -> Result<Vec<Device>> {
ProtonDriveClient::enumerate_devices(self).await
}
async fn create_device(&self, name: &str, device_type: DeviceType) -> Result<Device> {
ProtonDriveClient::create_device(self, name, device_type).await
}
}
+21 -4
View File
@@ -9,7 +9,7 @@
//! through one place (the status, transfers, sync progress) and publishes them
//! when they differ.
use std::collections::BTreeSet;
use std::collections::{BTreeSet, HashMap};
use std::sync::Arc;
use std::time::Duration;
@@ -153,13 +153,15 @@ pub(crate) fn transfers_line(items: Vec<TransferItem>, jobs: Vec<JobItem>) -> Op
/// Watch what changes without a single place to publish it from, and publish it
/// when it differs from the last look: the status (queue counts, cache use,
/// online, pause), the transfers snapshot, and the synced folders with their
/// live progress. Looks only while someone is subscribed.
/// online, pause), the transfers snapshot, the synced folders with their
/// live progress, and which nodes have a sync issue. Looks only while someone
/// is subscribed.
pub(crate) fn run_event_sampler(core: Core, username: String, mountpoint: std::path::PathBuf) {
let mut transfers: Option<Arc<str>> = None;
let mut status: Option<String> = None;
let mut queue: Option<String> = None;
let mut locations: Option<String> = None;
let mut issues: Option<HashMap<i64, String>> = None;
let mut tick = 0u32;
while core.shutdown.sleep(SAMPLE_INTERVAL) {
if !core.events.has_subscribers() {
@@ -169,6 +171,7 @@ pub(crate) fn run_event_sampler(core: Core, username: String, mountpoint: std::p
status = None;
queue = None;
locations = None;
issues = None;
continue;
}
let line = transfers_line(core.transfers.snapshot(), core.jobs_snapshot());
@@ -196,6 +199,15 @@ pub(crate) fn run_event_sampler(core: Core, username: String, mountpoint: std::p
}
queue = Some(queue_key);
// An issue shows on its node in the file browser: raised, cleared,
// landed or discarded, the listing is out of date.
if let Ok(now) = core.db.node_issues() {
if issues.as_ref().is_some_and(|last| *last != now) {
core.events.publish(&[Topic::Files]);
}
issues = Some(now);
}
if let Ok(now) = core.list_locations() {
let key = serde_json::to_string(&now).unwrap_or_default();
if locations.as_ref().is_some_and(|last| *last != key) {
@@ -223,6 +235,8 @@ fn status_keys(status: &CtlResponse) -> (String, String) {
used,
budget,
online,
session_expired,
tokens_unsaved,
pending_uploads,
pending_changes,
..
@@ -235,7 +249,7 @@ fn status_keys(status: &CtlResponse) -> (String, String) {
);
let key = format!(
"{queue}/{staged_bytes}/{paused}/{paused_until:?}/{username}/{mountpoint}/{pinned}/\
{used}/{budget}/{online}"
{used}/{budget}/{online}/{session_expired}/{tokens_unsaved}"
);
(key, queue)
}
@@ -313,6 +327,9 @@ mod tests {
budget: 0,
pins: Vec::new(),
online: true,
session_expired: false,
tokens_unsaved: false,
version: "3.3.0".into(),
pending_uploads: 1,
pending_changes: 0,
};
+369 -118
View File
@@ -132,7 +132,7 @@ impl Filesystem for ProtonFs {
// A provisional size is the *ciphertext* size, which is larger than the
// file. Publishing it makes every reader that trusts `st_size` — rsync,
// mmap, sendfile, a sized read loop — run off the end of the file and
// fail. So resolve it before answering rather than after (bugs.md B14).
// fail. So resolve it before answering rather than after (`docs/BUGS.md` B14).
//
// The cost is one batched round trip for the whole folder, not one per
// file: `ls -l` is one `getattr` per entry and they collapse onto a
@@ -172,7 +172,7 @@ impl Filesystem for ProtonFs {
/// here — once per `opendir`, which the kernel pairs exactly with one
/// `releasedir` — is what makes the offsets stable, so a create or a trash
/// during an enumeration can no longer make the caller skip or repeat an
/// entry (bugs.md B43).
/// entry (`docs/BUGS.md` B43).
///
/// Enumerating a cold folder is a remote call, so it goes to a worker for
/// the same reason `lookup` and `readdir` do.
@@ -261,10 +261,13 @@ impl Filesystem for ProtonFs {
// in-flight (possibly unsaved) content: authored bytes come from the
// scratch file, untouched bytes from the base.
//
// This path stays on the dispatch loop. `write` runs there too, so
// keeping its reads there as well preserves the ordering the write path
// has always had between a write and a read of the same handle; a mostly
// local read is not worth reasoning about that for.
// Answered on the dispatch loop when everything it needs is on this
// machine, which keeps the order `write` has with reads of the same
// handle. A base range that has to be downloaded goes to a worker
// instead, with the handle as it was when the read came in: the kernel
// issued the read before any write that comes after it, so that is
// the content it asked for. On the loop, the download held every
// request of the mount (`docs/BUGS.md` B227).
let handle = {
let st = self.core.state();
st.active_writes.get(&ino.0).map(|h| {
@@ -279,7 +282,7 @@ impl Filesystem for ProtonFs {
})
};
if let Some((file, len, uid, base_mtime, base_size, written)) = handle {
match self.core.serve_open_read(
match self.core.serve_open_read_local(
&file,
len,
&uid,
@@ -289,7 +292,25 @@ impl Filesystem for ProtonFs {
offset,
size as u64,
) {
Ok(bytes) => reply.data(&bytes),
Ok(Some(bytes)) => reply.data(&bytes),
Ok(None) => {
let fs = self.clone();
self.core.workers.run(Lane::Transfer, "read", move || {
match fs.core.serve_open_read(
&file,
len,
&uid,
base_mtime,
base_size,
&written,
offset,
size as u64,
) {
Ok(bytes) => reply.data(&bytes),
Err(e) => reply.error(e),
}
});
}
Err(e) => reply.error(e),
}
return;
@@ -346,7 +367,7 @@ impl Filesystem for ProtonFs {
self.core.workers.run(Lane::Transfer, "read", move || {
// The revision disagrees with a provisional size, and B84 fails
// such a read outright, so settle the size first. On the network,
// hence here and not above (bugs.md B100).
// hence here and not above (`docs/BUGS.md` B100).
let fsize = match provisional {
true => core.resolve_provisional_size(ino.0, &uid).unwrap_or(fsize),
false => fsize,
@@ -517,7 +538,7 @@ impl Filesystem for ProtonFs {
// `> file`. This is the second write path into the API, and
// queueing it is what lets a redirect work offline at all: it
// never reaches `release`, so without this it failed before any
// byte was written (offline.md Phase 2/3b).
// byte was written.
//
// The one branch of `setattr` that can reach the network:
// shrinking keeps the prefix, which `queue_truncate` gap-fills
@@ -556,48 +577,62 @@ impl Filesystem for ProtonFs {
return;
}
let new_end = offset.saturating_add(length);
let keep_size = (mode & 1) != 0; // FALLOC_FL_KEEP_SIZE
let is_punch_hole = (mode & 0x02) != 0; // FALLOC_FL_PUNCH_HOLE
let res = {
// Only the modes whose effect on the scratch file this handle can
// record. Anything else went through to the scratch file untracked:
// a zeroed range was refilled from the base at commit, and a collapse
// or insert shifted bytes that `len` and `written` did not follow
// (`docs/BUGS.md` B224). `EOPNOTSUPP` is also what makes glibc's
// `posix_fallocate` fall back to writing zeros.
let Some(kind) = falloc_kind(mode) else {
reply.error(Errno::EOPNOTSUPP);
return;
};
let file = {
let st = self.core.state();
match st.handles.get(&_fh.0).and_then(|i| st.active_writes.get(i)) {
Some(aw) => aw.file.clone(),
None => {
reply.error(if st.entries.contains_key(&ino.0) {
Errno::EBADF
} else {
Errno::ENOENT
});
return;
}
}
};
// The syscall can take a while on a large range; the state lock is
// not held across it, as `write` does not hold it across its I/O.
let ret = {
use std::os::unix::io::AsRawFd;
unsafe { libc::fallocate(file.as_raw_fd(), mode, offset as i64, length as i64) }
};
let res = if ret != 0 {
Err(Errno::from(std::io::Error::last_os_error()))
} else {
let mut st = self.core.state();
match st
if let Some(aw) = st
.handles
.get(&_fh.0)
.copied()
.and_then(|i| st.active_writes.get_mut(&i))
{
Some(aw) => {
use std::os::unix::io::AsRawFd;
let fd = aw.file.as_raw_fd();
let ret = unsafe { libc::fallocate(fd, mode, offset as i64, length as i64) };
if ret == 0 {
if is_punch_hole {
// A punched range reads as zero. Mark it authored so
// queue_revision does not refill it from the remote
// baseline and silently undo the hole at commit.
aw.written.add(offset, new_end.min(aw.len));
aw.dirty = true;
} else if !keep_size && new_end > aw.len {
aw.written.add(aw.len, new_end);
aw.len = new_end;
aw.dirty = true;
st.set_size(ino.0, new_end);
}
Ok(())
} else {
Err(Errno::EIO)
}
let len = aw.len;
let (authored, grown) = kind.effect(offset, new_end, len);
if let Some((from, to)) = authored {
// Zeroed or punched bytes read as zero. Marked authored so
// queue_revision does not refill them from the remote
// baseline and silently undo them at commit.
aw.written.add(from, to);
aw.dirty = true;
}
None => {
if st.entries.contains_key(&ino.0) {
Err(Errno::EBADF)
} else {
Err(Errno::ENOENT)
}
if let Some(end) = grown {
aw.len = end;
aw.dirty = true;
st.set_size(ino.0, end);
}
}
Ok(())
};
match res {
@@ -606,18 +641,47 @@ impl Filesystem for ProtonFs {
}
}
/// `close(2)` calls flush before release. The upload is queued in `release`
/// and performed in the background (offline.md Phase 3), so there is nothing
/// to push here — the written bytes are already in the scratch file.
/// `close(2)` calls flush before release, and flush is the only one of the
/// two whose answer reaches the program: the kernel drops `release`'s. So
/// the scratch file of a handle with writes on it is synced here, on a
/// worker, and a full disk or a failing one is what `close` returns.
///
/// Before, flush answered `ok` at once and `release` synced the file, so an
/// `ENOSPC` that delayed allocation held back until then was reported to
/// nobody, and the program took the save as done (`docs/BUGS.md` B222).
/// It also means `release`, which stays on the dispatch loop for ordering,
/// finds the pages already on disk instead of writing back a whole file
/// while every other request on the mount waits.
fn flush(
&self,
_req: &Request,
_ino: INodeNo,
_fh: FileHandle,
fh: FileHandle,
_lock_owner: LockOwner,
reply: ReplyEmpty,
) {
reply.ok();
let file = {
let st = self.core.state();
st.handles
.get(&fh.0)
.and_then(|ino| st.active_writes.get(ino))
.filter(|aw| aw.dirty)
.map(|aw| aw.file.clone())
};
let Some(file) = file else {
reply.ok();
return;
};
let fs = self.clone();
self.core
.workers
.run(Lane::Transfer, "flush", move || match file.sync_data() {
Ok(()) => reply.ok(),
Err(e) => {
error!(fh = fh.0, error = %e, "flush of scratch file failed");
reply.error(fs.scratch_sync_errno(&e));
}
});
}
/// Durability here means the scratch file, not the remote: a caller asking
@@ -985,7 +1049,7 @@ pub struct ProtonFs {
}
/// How many times a write `open` will re-read its base after a `release`
/// published a newer queued revision underneath it (bugs.md B32).
/// published a newer queued revision underneath it (`docs/BUGS.md` B32).
///
/// One retry covers the race, which needs a close and an open of the same file
/// to overlap on different threads. More than that is not contention, it is a
@@ -1021,7 +1085,53 @@ fn build_listing(st: &State, ino: u64) -> DirListing {
///
/// The cookie handed back is the index of the *next* entry, so a resumed call
/// continues exactly where the last one stopped — which is only true because
/// the listing behind those indexes is a snapshot (bugs.md B43).
/// the listing behind those indexes is a snapshot (`docs/BUGS.md` B43).
/// What an `fallocate` mode does to a write handle, for the modes it can
/// record (`docs/BUGS.md` B224).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum FallocKind {
/// Reserve space; past the end it grows the file unless `keep_size`.
Allocate { keep_size: bool },
/// Deallocate a range inside the file, which then reads as zero.
Punch,
/// Zero a range; past the end it grows the file unless `keep_size`.
Zero { keep_size: bool },
}
fn falloc_kind(mode: i32) -> Option<FallocKind> {
const KEEP: i32 = libc::FALLOC_FL_KEEP_SIZE;
const PUNCH: i32 = libc::FALLOC_FL_PUNCH_HOLE;
const ZERO: i32 = libc::FALLOC_FL_ZERO_RANGE;
match mode {
0 => Some(FallocKind::Allocate { keep_size: false }),
KEEP => Some(FallocKind::Allocate { keep_size: true }),
m if m == PUNCH | KEEP => Some(FallocKind::Punch),
ZERO => Some(FallocKind::Zero { keep_size: false }),
m if m == ZERO | KEEP => Some(FallocKind::Zero { keep_size: true }),
_ => None,
}
}
impl FallocKind {
/// For `[offset, end)` on a file `len` bytes long: the range that now
/// reads as zero and must count as written, and the new length if it grew.
fn effect(self, offset: u64, end: u64, len: u64) -> (Option<(u64, u64)>, Option<u64>) {
let grow = |keep_size: bool| (!keep_size && end > len).then_some(end);
let zeroed = |to: u64| (offset < to).then_some((offset, to));
match self {
// Space past the old end reads as zero, and nothing before it moved.
Self::Allocate { keep_size } => {
(grow(keep_size).map(|end| (len, end)), grow(keep_size))
}
Self::Punch => (zeroed(end.min(len)), None),
Self::Zero { keep_size } => {
let to = if keep_size { end.min(len) } else { end };
(zeroed(to), grow(keep_size))
}
}
}
}
fn reply_listing(listing: &DirListing, offset: u64, root: u64, mut reply: ReplyDirectory) {
for (i, (ino, is_dir, name)) in listing.iter().enumerate().skip(offset as usize) {
let ft = if *is_dir {
@@ -1094,7 +1204,7 @@ impl ProtonFs {
// `getattr` would, so `ls -l` — which is one `lookup`
// per entry and no `getattr` at all — takes its sizes
// from here. Resolving only in `getattr` left the whole
// listing provisional (bugs.md B14).
// listing provisional (`docs/BUGS.md` B14).
let provisional = matches!(
&e.node.kind,
NodeKind::File {
@@ -1217,7 +1327,7 @@ impl ProtonFs {
/// open.
fn serve_open(&self, ino: INodeNo, flags: OpenFlags, reply: ReplyOpen) {
let write_requested = flags.acc_mode() != OpenAccMode::O_RDONLY;
let (uid, base_mtime, base_size, provisional, base_revision_id) = {
let (uid, mut base_mtime, mut base_size, mut provisional, mut base_revision_id) = {
let mut st = self.core.state();
match st.entries.get_mut(&ino.0) {
Some(e) if e.node.is_file() => {
@@ -1267,8 +1377,12 @@ impl ProtonFs {
//
// Complete pending blobs are the common case and can be copied without
// any network access. An incomplete blob still has gaps referring to
// the remote base; stacking another write on it cannot be represented
// safely by WriteHandle, so fail the open rather than risk corruption.
// the remote base, and stacking another write on it cannot be
// represented by WriteHandle, so the copy's gaps are filled from that
// base first, as the drain fills them before an upload. Only when that
// fails (offline, or the base is gone) does the open fail: refusing
// every such open failed a create over a file whose partial edit had
// not drained yet (`docs/BUGS.md` B204).
//
// Reading that blob and installing the handle cannot be done under one
// lock — the copy is the size of the file — so the base is re-checked at
@@ -1276,34 +1390,32 @@ impl ProtonFs {
// revision in between sends this around again. Without the re-check the
// new handle carried the *older* base, and its own release published
// content that silently discarded the revision closed a moment earlier
// (bugs.md B32). A second attempt is already unlikely and a third would
// (`docs/BUGS.md` B32). A second attempt is already unlikely and a third would
// mean something is republishing continuously, so give up rather than
// spin: the caller's `open` fails instead of its data being lost.
//
// `reply` is consumed by whichever arm answers, so the loop yields an
// outcome and the answer is sent once, after it.
let outcome = 'install: {
// The handle starts at the base's size. A provisional one is the
// ciphertext size, so the file would grow by the encryption overhead
// and reading the base would fail (bugs.md B100). A queued revision
// stamps a real size, so this only goes to the network for a file
// the listing has not sized yet.
let base_size = match provisional && self.core.pending_blob(&uid).is_none() {
false => base_size,
true => match self.core.resolve_provisional_size(ino.0, &uid) {
Some(size) => size,
None => {
error!(%uid, "could not settle the size of the base for a write open");
break 'install Err(Errno::EIO);
}
},
};
for attempt in 0..OPEN_BASE_ATTEMPTS {
let pending_base = self.core.pending.lock().get(&uid).cloned();
if pending_base.as_ref().is_some_and(|p| !p.meta.complete) {
error!(%uid, "refusing write over incomplete queued revision");
break 'install Err(Errno::EIO);
// The handle starts at the base's size. A provisional one is the
// ciphertext size, so the file would grow by the encryption
// overhead and reading the base would fail (`docs/BUGS.md` B100). A
// queued revision stamps a real size, so this only goes to the
// network for a file the listing has not sized yet.
if provisional && self.core.pending_blob(&uid).is_none() {
match self.core.resolve_provisional_size(ino.0, &uid) {
Some(size) => {
base_size = size;
provisional = false;
}
None => {
error!(%uid, "could not settle the size of the base for a write open");
break 'install Err(Errno::EIO);
}
}
}
let pending_base = self.core.pending.lock().get(&uid).cloned();
let (file, path) = match self.core.cache.create_scratch() {
Ok(x) => x,
Err(e) => {
@@ -1323,6 +1435,25 @@ impl ProtonFs {
"copy queued revision into write scratch failed");
break 'install Err(Errno::EIO);
}
if !pending.meta.complete {
let mut authored = Intervals::default();
for &(s, e) in &pending.meta.authored {
authored.add(s, e);
}
if let Err(error) = self.core.fill_gaps(
&uid,
&file,
pending.meta.len,
pending.meta.base_mtime,
pending.meta.base_size,
&authored,
) {
let _ = std::fs::remove_file(&path);
error!(%uid, ?error, "cannot complete the queued revision a write open is based on");
break 'install Err(error);
}
debug!(%uid, "write open completed an incomplete queued revision");
}
initial_written.add(0, pending.meta.len);
}
// Sampled *before* the state lock, never under it: no site in
@@ -1332,17 +1463,49 @@ impl ProtonFs {
// a lock-tight comparison — against the multi-gigabyte copy
// above, which is the window that actually loses data.
let current = self.core.pending_blob(&uid);
let landed = self.core.landed.lock().get(&uid).cloned();
let mut st = self.core.state();
// Only a handle this open is about to *create* can carry a stale
// base; if one already exists, that handle is authoritative and
// the scratch prepared above is discarded either way.
let creating = !st.active_writes.contains_key(&ino.0);
if creating && current != pending_base.map(|p| p.path) {
if creating && current.as_ref() != pending_base.as_ref().map(|p| &p.path) {
drop(st);
let _ = std::fs::remove_file(&path);
warn!(%uid, attempt, "queued revision changed while opening for write; retrying");
continue;
}
// With no queued revision, the base's bytes come from the cache,
// keyed by the base's mtime and size. The ones sampled above can
// name a revision this daemon has since replaced: an upload that
// lands lets go of its blob before the tree has its node, and
// the cache holds its bytes under the node only. So the base is
// the landing node while there is one, and the tree's otherwise
// (docs/BUGS.md B170).
if creating && pending_base.is_none() {
let now = landed.or_else(|| st.entries.get(&ino.0).map(|e| e.node.clone()));
if let Some(node) = now
&& (node.modification_time != base_mtime
|| node_revision_id(&node) != base_revision_id)
{
base_mtime = node.modification_time;
base_size = node_size(&node);
base_revision_id = node_revision_id(&node);
provisional = matches!(
&node.kind,
NodeKind::File {
claimed_size: None,
..
}
);
debug!(%uid, base_mtime, base_size, "write open took the base the file has now");
if provisional {
drop(st);
let _ = std::fs::remove_file(&path);
continue;
}
}
}
let fh = st.next_fh;
st.next_fh += 1;
let aw = st.active_writes.entry(ino.0).or_insert_with(|| {
@@ -1382,6 +1545,20 @@ impl ProtonFs {
}
/// The body of [`Filesystem::fsync`], run off the dispatch loop.
/// The errno to give a program for a failed sync of a scratch file.
/// Delayed allocation means a full disk usually surfaces at the sync
/// rather than at `write(2)`, and `ENOSPC` is what a program checks for.
fn scratch_sync_errno(&self, e: &std::io::Error) -> Errno {
match e.raw_os_error() {
Some(libc::ENOSPC) => {
self.core.cache.emergency_evict();
Errno::ENOSPC
}
Some(libc::EDQUOT) => Errno::EDQUOT,
_ => Errno::EIO,
}
}
fn serve_fsync(&self, fh: FileHandle, reply: ReplyEmpty) {
// Everything the durability sidecar needs, copied out so no lock is held
// across the sync.
@@ -1402,22 +1579,18 @@ impl ProtonFs {
)
})
};
// A handle opened for reading has nothing of its own to make durable,
// and POSIX allows `fsync` on it; `EBADF` failed `sync FILE` and
// backup tools that sync what they read (`docs/BUGS.md` B222).
let Some((f, path, uid, dirty, written, len, base_size, base_mtime, base_revision_id)) =
handle
else {
reply.error(Errno::EBADF);
reply.ok();
return;
};
if let Err(e) = f.sync_all() {
error!(fh = fh.0, error = %e, "fsync of scratch file failed");
// Delayed allocation means a full disk usually surfaces here rather
// than at `write(2)`; `ENOSPC` is what an application checks for.
if e.raw_os_error() == Some(libc::ENOSPC) {
self.core.cache.emergency_evict();
reply.error(Errno::ENOSPC);
} else {
reply.error(Errno::EIO);
}
reply.error(self.scratch_sync_errno(&e));
return;
}
// A clean handle has nothing on it that the remote does not already
@@ -1507,14 +1680,14 @@ impl ProtonFs {
let transient = is_transient_name(name);
// Offline the server cannot mint a uid, so invent one and queue the
// create. The file is real to the caller either way; only its identity is
// provisional until the drain (offline.md Phase 3b).
// provisional until the drain.
//
// A parent that is itself still queued forces the same path even when we
// are online: the API has no folder to put this in yet.
//
// A create that set out online and lost the network on the way lands in
// the queue too, so a Wi-Fi drop costs the caller a wait, never an EIO.
let minted = if !transient && self.core.is_online() && !is_local_uid(&parent_uid) {
let minted = if !transient && self.core.sends_inline() && !is_local_uid(&parent_uid) {
if let Err(error) = self.core.require_uid_writable(&parent_uid) {
reply.error(error);
return;
@@ -1618,9 +1791,9 @@ impl ProtonFs {
.ok();
// As in `create`: offline — or under a parent that is itself still
// queued — the folder becomes a placeholder that the drain turns into a
// real one (offline.md Phase 3b).
// real one.
// Losing the network on the way queues it the same way.
let minted = if self.core.is_online() && !is_local_uid(&parent_uid) {
let minted = if self.core.sends_inline() && !is_local_uid(&parent_uid) {
if let Err(error) = self.core.require_uid_writable(&parent_uid) {
reply.error(error);
return;
@@ -1664,9 +1837,7 @@ impl ProtonFs {
// empty, which it is, so reads stay offline-clean across a restart too.
if local {
st.children.insert(ino, Vec::new());
if let Err(e) = self.core.db.set_listed(&uid, true) {
warn!(%uid, error = %e, "db set_listed(true) failed for local folder");
}
st.record_listed(uid);
}
let entry = st.entries.get(&ino).unwrap();
let attr = self.attr(ino, &entry.node, entry.access);
@@ -1770,10 +1941,13 @@ impl ProtonFs {
self.core.cache.clear_scratch_durable(&h.path);
reply.ok();
}
(Some(h), false) => match self.core.queue_revision(&h) {
Ok(()) => reply.ok(),
Err(e) => reply.error(e),
},
(Some(mut h), false) => {
self.core.follow_landed_create(&mut h);
match self.core.queue_revision(&mut h) {
Ok(()) => reply.ok(),
Err(e) => reply.error(e),
}
}
(None, _) => reply.ok(),
}
}
@@ -1796,7 +1970,7 @@ impl ProtonFs {
reply.error(error);
return;
}
let (ino, uid) = match self.core.lookup_child(parent, name) {
let (ino, mut uid) = match self.core.lookup_child(parent, name) {
Ok(x) => x,
Err(e) => {
reply.error(e);
@@ -1837,7 +2011,7 @@ impl ProtonFs {
// rename, so the victim has to be removed first and the operation stops
// being atomic. Without this the 422 surfaced as a blanket EIO and every
// write-to-temp-then-rename tool — rsync, atomic editor saves — failed
// at the very end of its transfer (bugs.md B13).
// at the very end of its transfer (`docs/BUGS.md` B13).
//
// `RENAME_EXCHANGE` has no Proton primitive and cannot be emulated
// without a window in which one of the two names does not exist.
@@ -1909,7 +2083,7 @@ impl ProtonFs {
reply.error(error);
return;
}
if let Err(e) = self.core.remove_replaced(victim_uid, newname) {
if let Err(e) = self.core.remove_replaced(*victim_ino, victim_uid, newname) {
reply.error(e);
return;
}
@@ -1919,8 +2093,9 @@ impl ProtonFs {
};
// A node whose own creation is still queued has no server-side identity
// to rename: the queued op *is* the node, so rewriting its target is the
// whole rename. Nothing reaches the API, which is why this works offline
// (offline.md Phase 3b).
// whole rename. Nothing reaches the API, which is why this works offline.
// A create that lands meanwhile leaves no op to rewrite; the node is
// then renamed by the uid Drive gave it (docs/BUGS.md B142).
if is_local_uid(&uid) {
match self.core.db.rewrite_op_target(
&uid.to_string(),
@@ -1931,6 +2106,8 @@ impl ProtonFs {
self.core
.state()
.rename_in_place(ino, newparent, &new_parent_uid, newname);
// A create that failed on its old name is due again.
self.core.wake_drain();
// Finalize: a transient scratch file (its create parked, bytes
// held back) renamed to a finished name is the moment the
// completed file is meant to reach Drive. Un-park the create so
@@ -1949,23 +2126,29 @@ impl ProtonFs {
}
debug!(%uid, newname, "renamed a node whose create is still queued");
reply.ok();
return;
}
// The create drained underneath us, so the node has a real uid
// now and this handle's is stale. A retry resolves it.
Ok(false) => {
warn!(%uid, name, newname, "queued create vanished under a rename");
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EBUSY);
}
Ok(false) => match self.core.landed_row_uid(ino) {
Some(real) => {
debug!(local = %uid, %real, newname, "create landed under a rename");
uid = real;
}
None => {
warn!(%uid, name, newname, "queued create vanished under a rename");
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EBUSY);
return;
}
},
Err(e) => {
error!(%uid, error = %e, "rewriting a queued create's target failed");
self.core
.restore_replaced(victim.as_ref(), &new_parent_uid, newname);
reply.error(Errno::EIO);
return;
}
}
return;
}
// Offline, or into a folder that does not exist remotely yet: queue the
// rename rather than 404 or fail. Online moves into a real parent still
@@ -2020,7 +2203,7 @@ impl ProtonFs {
}
};
if rename_needs_queue(
self.core.is_online(),
self.core.sends_inline(),
is_local_uid(&new_parent_uid),
newparent != parent,
newname != name,
@@ -2093,7 +2276,7 @@ impl ProtonFs {
reply.error(error);
return;
}
let (ino, uid) = match self.core.lookup_child(parent, name) {
let (ino, mut uid) = match self.core.lookup_child(parent, name) {
Ok(x) => x,
Err(e) => {
reply.error(e);
@@ -2109,7 +2292,7 @@ impl ProtonFs {
.is_some_and(|e| e.open_count > 0);
// A node the server has never heard of cannot be trashed there; deleting
// it just means its queued creation is no longer wanted. This works
// offline, which the remote path below cannot (offline.md Phase 3b).
// offline, which the remote path below cannot.
if is_local_uid(&uid) {
let dropped = if open_now {
self.core.withdraw_queued_ops(&uid)
@@ -2121,19 +2304,30 @@ impl ProtonFs {
return;
}
self.core.state().forget_or_unlink(&uid);
debug!(%uid, name, "deleted a node that had not been created remotely yet");
reply.ok();
return;
// A create that landed before the ops were dropped left none: Drive
// has the node, and the tree had not taken its uid yet. It is
// trashed there like any other (docs/BUGS.md B173).
match self.core.landed_row_uid(ino) {
Some(real) => {
debug!(local = %uid, %real, name, "create landed under a delete");
uid = real;
}
None => {
debug!(%uid, name, "deleted a node that had not been created remotely yet");
reply.ok();
return;
}
}
}
// Offline: queue it. Trashing is the one mutation a user expects to work
// regardless — the file is gone from their point of view the moment the
// command returns (offline.md Phase 3b). The same when the network goes
// command returns. The same when the network goes
// away under the remote call below.
let queue_it = |reply: ReplyEmpty| match self.core.queue_trash(&uid, name) {
Ok(()) => reply.ok(),
Err(e) => reply.error(e),
};
if !self.core.is_online() {
if !self.core.sends_inline() {
queue_it(reply);
return;
}
@@ -2176,7 +2370,7 @@ impl ProtonFs {
self.core.invalidate_trash();
// Every other trash site records itself; this one did not, which made a
// file found in the trash impossible to attribute after the fact — the
// activity log was the only record and it showed nothing (bugs.md B2).
// activity log was the only record and it showed nothing (`docs/BUGS.md` B2).
self.core
.log_activity(ActivityKind::Trash, name, "trashed from the mount", true);
reply.ok();
@@ -2261,3 +2455,60 @@ impl ProtonFs {
}
}
}
#[cfg(test)]
mod falloc_tests {
use super::*;
#[test]
fn only_the_modes_a_handle_can_record_are_accepted() {
let keep = libc::FALLOC_FL_KEEP_SIZE;
assert_eq!(
falloc_kind(0),
Some(FallocKind::Allocate { keep_size: false })
);
assert_eq!(
falloc_kind(keep),
Some(FallocKind::Allocate { keep_size: true })
);
assert_eq!(
falloc_kind(libc::FALLOC_FL_PUNCH_HOLE | keep),
Some(FallocKind::Punch)
);
assert_eq!(
falloc_kind(libc::FALLOC_FL_ZERO_RANGE),
Some(FallocKind::Zero { keep_size: false })
);
for mode in [
libc::FALLOC_FL_PUNCH_HOLE,
libc::FALLOC_FL_COLLAPSE_RANGE,
libc::FALLOC_FL_INSERT_RANGE,
libc::FALLOC_FL_UNSHARE_RANGE,
] {
assert_eq!(falloc_kind(mode), None, "{mode:#x}");
}
}
/// A zeroed range was left unwritten, so the commit filled it back in
/// from the remote base (`docs/BUGS.md` B224).
#[test]
fn a_zeroed_or_punched_range_counts_as_written() {
let zero = FallocKind::Zero { keep_size: false };
assert_eq!(zero.effect(10, 20, 100), (Some((10, 20)), None));
assert_eq!(zero.effect(90, 120, 100), (Some((90, 120)), Some(120)));
let zero_keep = FallocKind::Zero { keep_size: true };
assert_eq!(zero_keep.effect(90, 120, 100), (Some((90, 100)), None));
assert_eq!(
FallocKind::Punch.effect(90, 120, 100),
(Some((90, 100)), None)
);
assert_eq!(FallocKind::Punch.effect(120, 130, 100), (None, None));
let alloc = FallocKind::Allocate { keep_size: false };
assert_eq!(alloc.effect(0, 50, 100), (None, None));
assert_eq!(alloc.effect(90, 120, 100), (Some((100, 120)), Some(120)));
assert_eq!(
FallocKind::Allocate { keep_size: true }.effect(90, 120, 100),
(None, None)
);
}
}
+1084 -223
View File
File diff suppressed because it is too large. Load diff
+169 -8
View File
@@ -9,10 +9,11 @@
//! queued path, and the probe thread here flips it back when Proton answers.
use std::future::Future;
use std::sync::atomic::Ordering;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
use parking_lot::{Condvar, Mutex};
use pdfs_core::error::session_revoked;
use proton_drive_rs::proton_sdk::api::ResponseCode;
use proton_drive_rs::proton_sdk::error::{ProtonApiError, ProtonError};
use tracing::{debug, info, warn};
@@ -28,6 +29,13 @@ use crate::{Core, ONLINE_PROBE_MAX, ONLINE_PROBE_MIN};
/// keep waiting for; queueing the change is the better answer.
pub(crate) const INTERACTIVE_CALL_TIMEOUT: Duration = Duration::from_secs(20);
/// How long a mount thread waits on a remote read that answers a batch: a
/// chunk of file sizes, or the list of shares. Longer than an interactive call,
/// since its answer grows with the batch, but well short of the SDK's own retry
/// budget, which held a worker for minutes on a stalled link
/// (`docs/BUGS.md` B220).
pub(crate) const BATCH_CALL_TIMEOUT: Duration = Duration::from_secs(60);
/// How long one probe may wait for an answer. A probe is a single small request,
/// and a probe stuck in the SDK's retry budget would delay noticing the link is
/// back by as long as it waits.
@@ -54,6 +62,26 @@ pub(crate) struct Link {
changed: Condvar,
/// Folders listed from the DB alone while offline, to relist once back.
pub(crate) stale_listings: Mutex<std::collections::HashSet<u64>>,
/// Proton refused the session itself ([`session_revoked`]). The mount is
/// offline then, but no probe brings it back: only a new login does, and
/// that restarts the daemon. Read by front-ends through `Response::Status`,
/// so they can ask the user to sign in again rather than say "offline".
pub(crate) session_expired: AtomicBool,
/// Proton refused the session, but only because another process rotated
/// it (`pdfs unlock`): the keyring holds the live tokens. The mount loop
/// ends the mount for the daemon to mount again from the keyring.
pub(crate) resume_stored: AtomicBool,
}
impl Link {
/// The link of a mount that comes up with Proton refusing its session, or
/// not.
pub(crate) fn starting(session_expired: bool) -> Self {
Self {
session_expired: AtomicBool::new(session_expired),
..Default::default()
}
}
}
/// Whether a failed call failed because Proton could not be reached, rather
@@ -86,13 +114,47 @@ pub(crate) fn no_answer(after: Duration) -> ProtonError {
})
}
/// The slowest upload rate an upload's deadline allows for, past the
/// metadata budget it starts with. Far below any link that works, so only a
/// transfer that has stopped moving runs into it.
const SLOWEST_UPLOAD: u64 = 32 * 1024;
/// How many uploads can share the upload cap at once: one per drain worker
/// and one per mirror sync task.
const UPLOAD_SLOTS: u64 = (crate::DRAIN_WORKERS + crate::sync::SYNC_CONCURRENCY) as u64;
/// How long an upload of `len` bytes may take before the link is treated as
/// down: the budget of a metadata call, plus the bytes at [`SLOWEST_UPLOAD`].
///
/// An upload cap of `cap` bytes per second (`0` for none) is shared by every
/// upload running, so each may get as little as its share of it. Below
/// [`SLOWEST_UPLOAD`] a share counts as the slowest rate instead: measured
/// against the fixed rate, a large file under a low cap ran out of time on
/// every attempt, took the mount offline each time, and started over from
/// its first byte.
pub(crate) fn upload_deadline(len: u64, cap: u64) -> Duration {
let slowest = match cap {
0 => SLOWEST_UPLOAD,
cap => SLOWEST_UPLOAD.min((cap / UPLOAD_SLOTS).max(1)),
};
INTERACTIVE_CALL_TIMEOUT + Duration::from_secs(len / slowest)
}
/// Run a remote call, giving up after [`INTERACTIVE_CALL_TIMEOUT`].
pub(crate) async fn bounded<T>(
call: impl Future<Output = Result<T, ProtonError>>,
) -> Result<T, ProtonError> {
match tokio::time::timeout(INTERACTIVE_CALL_TIMEOUT, call).await {
bounded_by(INTERACTIVE_CALL_TIMEOUT, call).await
}
/// Run a remote call, giving up after `limit`.
pub(crate) async fn bounded_by<T>(
limit: Duration,
call: impl Future<Output = Result<T, ProtonError>>,
) -> Result<T, ProtonError> {
match tokio::time::timeout(limit, call).await {
Ok(result) => result,
Err(_) => Err(no_answer(INTERACTIVE_CALL_TIMEOUT)),
Err(_) => Err(no_answer(limit)),
}
}
@@ -113,6 +175,15 @@ impl Core {
self.online.load(Ordering::Relaxed)
}
/// Whether a namespace change through the mount goes to Drive inside the
/// syscall. Only with [`AppConfig::local_first`] off, and only online;
/// otherwise it is queued and the drain sends it.
///
/// [`AppConfig::local_first`]: pdfs_core::config::AppConfig::local_first
pub(crate) fn sends_inline(&self) -> bool {
!self.local_first && self.is_online()
}
/// Run a remote call a caller is waiting on, giving up after
/// [`INTERACTIVE_CALL_TIMEOUT`]. The timer is built inside the runtime: a
/// FUSE worker has no reactor of its own.
@@ -123,10 +194,30 @@ impl Core {
self.rt.block_on(bounded(call))
}
/// Run a remote call, giving up after `limit`: an upload, whose budget
/// grows with its size ([`upload_deadline`]).
pub(crate) fn block_on_within<T>(
&self,
limit: Duration,
call: impl Future<Output = Result<T, ProtonError>>,
) -> Result<T, ProtonError> {
self.rt.block_on(bounded_by(limit, call))
}
/// Whether Proton refused the session itself, so nothing reaches it until
/// the user signs in again.
pub(crate) fn session_expired(&self) -> bool {
self.link.session_expired.load(Ordering::Relaxed)
}
/// Report a failed remote call. Returns whether it failed for want of a
/// network, in which case the mount is now offline and the caller should
/// take its queued path instead of failing.
/// network, or of a session, in which case the mount is now offline and the
/// caller should take its queued path instead of failing.
pub(crate) fn lost_link(&self, e: &ProtonError, what: &str) -> bool {
if session_revoked(e) {
self.mark_session_expired(what, e);
return true;
}
if !is_network_error(e) {
return false;
}
@@ -134,6 +225,29 @@ impl Core {
true
}
/// Flip the mount offline for good: Proton refused the session, and only a
/// new login brings it back. Changes stay queued for the daemon that login
/// starts. Only the first caller logs it.
///
/// Unless the keyring holds a newer session than the refused one: then
/// the mount ends, for the daemon to mount again on that one.
pub(crate) fn mark_session_expired(&self, what: &str, error: &dyn std::fmt::Display) {
if self.link.resume_stored.load(Ordering::Relaxed) {
return self.mark_offline(what, error);
}
if pdfs_core::auth::stored_session_is_newer() {
info!(during = what, %error,
"Proton refused a session another process has since refreshed; resuming it");
self.link.resume_stored.store(true, Ordering::Relaxed);
return self.mark_offline(what, error);
}
if !self.link.session_expired.swap(true, Ordering::Relaxed) {
warn!(during = what, %error,
"Proton refused the session; run `pdfs login` to sign in again");
}
self.mark_offline(what, error);
}
/// Flip the mount offline and wake the probe. Only the first caller of an
/// outage logs it; the rest are the same outage.
pub(crate) fn mark_offline(&self, what: &str, error: &dyn std::fmt::Display) {
@@ -166,8 +280,9 @@ impl Core {
self.relist_offline_listings();
// Anything written while offline is queued and waiting on exactly this,
// including ops that failed on the way down and are sitting out a
// backoff sized for a server that keeps refusing.
if let Err(e) = self.db.retry_failed_ops_now(crate::now_millis()) {
// backoff sized for a server that keeps refusing. Ops Drive refused
// keep theirs: the link was not why.
if let Err(e) = self.db.retry_unrefused_ops_now(crate::now_millis()) {
warn!(error = %e, "moving backed-off ops up after a reconnect failed");
}
self.wake_drain();
@@ -210,7 +325,7 @@ impl Core {
}
/// Watch the link for the life of the mount: sleep while it is up, and
/// probe it back while it is down (offline.md Phase 1).
/// probe it back while it is down.
///
/// A mount that started from the cache is down from the first moment; one
/// that started online goes down when [`Core::mark_offline`] says so.
@@ -238,6 +353,9 @@ impl Core {
/// Probe until Proton answers, then flip the mount online. Returns `false`
/// if the daemon is stopping.
///
/// A refused session is not probed: every probe would be refused the same
/// way, forever (issue #35). The thread waits for the stop instead.
fn probe_until_online(&self) -> bool {
let started = self.link.lost_at.lock().unwrap_or_else(Instant::now);
let mut delay = if started.elapsed() < YOUNG_OUTAGE {
@@ -249,6 +367,10 @@ impl Core {
if !self.shutdown.sleep(delay) {
return false;
}
if self.session_expired() {
delay = ONLINE_PROBE_MAX;
continue;
}
let probe = self.rt.block_on(async {
match tokio::time::timeout(PROBE_TIMEOUT, self.drive.get_my_files_folder()).await {
Ok(result) => result,
@@ -270,6 +392,7 @@ impl Core {
self.mark_online();
return true;
}
Err(e) if session_revoked(&e) => self.mark_session_expired("online probe", &e),
Err(e) => {
debug!(error = %e, ?delay, "online probe failed; still offline");
delay = next_probe_delay(delay, started.elapsed());
@@ -301,6 +424,44 @@ mod tests {
assert!(!is_network_error(&ProtonError::invalid_operation("bug")));
}
#[test]
fn an_upload_gets_longer_the_bigger_it_is() {
assert_eq!(upload_deadline(0, 0), INTERACTIVE_CALL_TIMEOUT);
assert_eq!(
upload_deadline(1 << 30, 0),
INTERACTIVE_CALL_TIMEOUT + Duration::from_secs(32_768)
);
}
#[test]
fn an_upload_under_a_low_cap_gets_the_time_its_share_of_the_cap_needs() {
// A generous cap leaves the deadline as it was.
assert_eq!(
upload_deadline(1 << 30, 100 << 20),
upload_deadline(1 << 30, 0)
);
// 512 KiB/s shared by every upload slot is below the slowest rate.
let share = (512 << 10) / UPLOAD_SLOTS;
assert!(share < SLOWEST_UPLOAD);
assert_eq!(
upload_deadline(1 << 30, 512 << 10),
INTERACTIVE_CALL_TIMEOUT + Duration::from_secs((1 << 30) / share)
);
// A cap below one byte per slot still gives a finite deadline.
assert!(upload_deadline(1 << 20, 1) > upload_deadline(1 << 20, 512 << 10));
}
#[test]
fn a_call_past_its_deadline_reads_as_a_lost_link() {
let rt = tokio::runtime::Builder::new_current_thread()
.enable_time()
.build()
.unwrap();
let limit = Duration::from_millis(20);
let result: Result<(), _> = rt.block_on(bounded_by(limit, std::future::pending()));
assert!(result.is_err_and(|e| is_network_error(&e)));
}
#[test]
fn a_young_outage_is_probed_often_and_an_old_one_rarely() {
let young = Duration::from_secs(60);
+129 -21
View File
@@ -12,6 +12,10 @@ pub enum MountOutcome {
/// The kernel mount ended on its own (e.g. an external `fusermount -u`).
/// The caller may want to remount.
Unmounted,
/// Proton refused the session, and the keyring holds a newer one: another
/// process (`pdfs unlock`) refreshed it, or a new login replaced it. We
/// unmounted ourselves; the caller should mount again, which resumes it.
ResumeStored,
}
/// Whether `path` is a mountpoint whose FUSE connection is dead — the state a
@@ -193,7 +197,7 @@ pub(super) fn clear_stale_mount(mountpoint: &Path) {
}
/// `sync_state` key holding the uid of the My Files root, so a later run can
/// recover it from `nodes` without the API (offline.md Phase 1).
/// recover it from `nodes` without the API.
const ROOT_UID_KEY: &str = "root_uid";
/// The My Files root, and whether we got it from the API (`true`) or from the
@@ -210,30 +214,60 @@ fn fetch_or_recall_root(
drive: &dyn DriveApi,
rt: &tokio::runtime::Handle,
db: &Db,
) -> std::io::Result<(Node, bool)> {
) -> std::io::Result<(Node, Start)> {
let err = match rt.block_on(drive.get_my_files_folder()) {
Ok(root) => {
if let Err(e) = db.set_state_str(ROOT_UID_KEY, &root.uid.to_string()) {
warn!(error = %e, "persist root uid failed");
}
return Ok((root, true));
return Ok((root, Start::Online));
}
Err(e) => e,
};
// Still mounted from the cache, so cached files stay readable and changes
// queue for the daemon the next login starts.
let start = if pdfs_core::error::session_revoked(&err) {
Start::SessionExpired
} else {
Start::Offline
};
if start == Start::SessionExpired && pdfs_core::auth::stored_session_is_newer() {
return Err(std::io::Error::other(format!(
"Proton refused a session another process has since refreshed ({err}); resuming it"
)));
}
let cached = db
.state_str(ROOT_UID_KEY)
.ok()
.flatten()
.and_then(|uid| db.node_by_uid(&uid).ok().flatten());
match cached {
Some(root) if start == Start::SessionExpired => {
warn!(error = %err,
"Proton refused the session; mounting from cache until `pdfs login`");
Ok((root, start))
}
Some(root) => {
warn!(error = %err, "fetch My Files root failed; mounting from cache (offline)");
Ok((root, false))
Ok((root, start))
}
None if start == Start::SessionExpired => Err(std::io::Error::other(format!(
"Proton refused the session ({err}); run `pdfs login`"
))),
None => Err(std::io::Error::other(format!("fetch My Files root: {err}"))),
}
}
/// How a mount comes up, from what fetching its root said.
#[derive(Clone, Copy, PartialEq, Eq)]
enum Start {
Online,
/// Proton could not be reached: serve the cache and probe for the link.
Offline,
/// Proton refused the session: serve the cache until a new login.
SessionExpired,
}
/// Per-mount settings resolved from [`pdfs_core::config::AppConfig`] by the
/// caller, so `mount` itself never reads config or the environment.
pub struct MountOptions {
@@ -244,6 +278,9 @@ pub struct MountOptions {
/// Upload and download caps in bytes per second (`0` = no cap).
pub upload_limit: u64,
pub download_limit: u64,
/// Namespace changes through the mount are queued, not sent in the
/// syscall. See [`pdfs_core::config::AppConfig::local_first`].
pub local_first: bool,
}
/// Spawn one FUSE session rooted at an arbitrary remote node.
@@ -385,11 +422,13 @@ pub(crate) fn mount_with(
sweep_mode,
upload_limit,
download_limit,
local_first,
} = options;
// Start the uptime clock here rather than at the first `pdfs diagnostics`,
// so the age it reports is the daemon's own.
super::diagnostics::start_clock();
let (root, online) = fetch_or_recall_root(drive.as_ref(), &rt, &db)?;
let (root, start) = fetch_or_recall_root(drive.as_ref(), &rt, &db)?;
let online = start == Start::Online;
let scope = root.tree_event_scope_id();
db.mount_upsert_my_files(&mountpoint.to_string_lossy(), &root.uid.to_string(), None)
.map_err(|error| std::io::Error::other(format!("project My Files mount: {error}")))?;
@@ -397,7 +436,7 @@ pub(crate) fn mount_with(
.all_share_access()
.map_err(|error| std::io::Error::other(format!("load shared access: {error}")))?;
// The folder-sync engine (devices.md Phase 2) runs on its own thread and is
// The folder-sync engine runs on its own thread and is
// nudged over this channel; the sender lives in Core so control-socket
// handlers can trigger reconciles.
let (sync_tx, sync_rx) = std::sync::mpsc::channel::<sync::SyncMsg>();
@@ -425,10 +464,14 @@ pub(crate) fn mount_with(
shared_generation: Arc::new(AtomicU64::new(0)),
shared_refresh_deadlines: Arc::new(Mutex::new(SharedRefreshDeadlines::default())),
online: Arc::new(AtomicBool::new(online)),
link: Arc::new(Default::default()),
link: Arc::new(link::Link::starting(start == Start::SessionExpired)),
sync_paused_until: Arc::new(AtomicI64::new(paused_until)),
pending: Arc::new(Mutex::new(HashMap::new())),
hidden: Arc::new(Mutex::new(HashSet::new())),
creating: Arc::new(Mutex::new(HashMap::new())),
sealed_unread: Arc::new(Mutex::new(HashMap::new())),
landed: Arc::new(Mutex::new(HashMap::new())),
unadopted: Arc::new(Mutex::new(HashMap::new())),
drain_wake: Arc::new((Mutex::new(false), Condvar::new())),
shutdown: Arc::new(crate::shutdown::Shutdown::default()),
upload_times: Arc::new(Mutex::new(HashMap::new())),
@@ -442,6 +485,7 @@ pub(crate) fn mount_with(
trash_progress: Arc::new(tokio::sync::Notify::new()),
conflict_notified: Arc::new(Mutex::new(HashSet::new())),
sweep_mode,
local_first,
self_changes: Arc::new(Mutex::new(HashMap::new())),
thumb_gen: Arc::new(Mutex::new(HashSet::new())),
thumb_gen_budget: Arc::new(tokio::sync::Semaphore::new(
@@ -466,6 +510,7 @@ pub(crate) fn mount_with(
sync_tx,
mounts: Arc::new(Mutex::new(SecondaryMountRegistry::default())),
sync_locks: Arc::new(Mutex::new(HashMap::new())),
sync_unscanned: Arc::new(Mutex::new(HashMap::new())),
states: Arc::new(StateRegistry::default()),
};
// Before anything can queue work against it: the drain thread below reaches
@@ -492,7 +537,7 @@ pub(crate) fn mount_with(
// Every long-lived thread started from here on is kept in `workers` and
// joined at the end of this function. Leaving them running turned an
// in-process remount into a second generation of drain workers, sync engines
// and sweeps all operating on a mount that no longer exists (bugs.md B44).
// and sweeps all operating on a mount that no longer exists (`docs/BUGS.md` B44).
let mut workers: Vec<std::thread::JoinHandle<()>> = Vec::new();
for worker in 0..DRAIN_WORKERS {
let core = core.clone();
@@ -670,7 +715,9 @@ pub(crate) fn mount_with(
// The photos volume has its own event stream; without it the gallery only
// learns about a phone-side deletion when the timeline goes stale.
rt.spawn(run_photos_event_sync(core.clone()));
if online {
// Not in a simulation: `client` there reaches no API, and the thread, still
// waiting on it after the stop, held the database a test opens next.
if online && matches!(host, Host::Daemon) {
let repair_core = core.clone();
let root_uid = core.primary_root_uid.clone();
if let Err(error) = std::thread::Builder::new()
@@ -707,6 +754,18 @@ pub(crate) fn mount_with(
}
break MountOutcome::Shutdown;
}
Err(std::sync::mpsc::RecvTimeoutError::Timeout)
if core.link.resume_stored.load(Ordering::Relaxed) =>
{
info!("unmounting to resume the stored session");
core.mounts.lock().close();
if let Err(e) = teardown_session(&core.session_live, || {
umount_session_unblocked(bg, main_conn)
}) {
warn!(error = %e, "umount_and_join failed");
}
break MountOutcome::ResumeStored;
}
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {
if bg.guard.is_finished() {
info!("mount ended externally");
@@ -727,7 +786,7 @@ pub(crate) fn mount_with(
};
// Unmount every on-demand sync folder too, or the kernel mounts linger as
// stale and the next start fails with EBUSY (devices.md Phase 3).
// stale and the next start fails with EBUSY.
if let Some(worker) = restore_worker
&& worker.join().is_err()
{
@@ -739,6 +798,7 @@ pub(crate) fn mount_with(
warn!(id, error = %e, "unmount on-demand folder failed");
}
}
core.queue_unreleased_writes();
// Last, because the workers may legitimately be finishing an upload, and
// nothing they can still do is harmful once the mounts are down.
@@ -775,9 +835,10 @@ fn signal_stop(rt: &tokio::runtime::Handle) -> std::sync::mpsc::Receiver<()> {
sig_rx
}
/// How long teardown waits for the FUSE worker pool before proceeding without
/// it. Long enough for an in-flight block to finish, short enough that
/// `systemctl --user stop` still returns inside `TimeoutStopSec`.
/// How long teardown waits for the background workers and then the FUSE worker
/// pool before proceeding without them. Long enough for an in-flight block to
/// finish, short enough that `systemctl --user stop` still returns inside
/// `TimeoutStopSec`.
const WORKER_JOIN_DEADLINE: Duration = Duration::from_secs(10);
/// Signal every background loop, wake the ones that are blocked, and join them.
@@ -785,14 +846,15 @@ const WORKER_JOIN_DEADLINE: Duration = Duration::from_secs(10);
/// The mount used to return with its drain workers, sync engine, sweep, online
/// probe, indexer and control listener all still running on `Core` clones. The
/// process usually exited straight afterwards and hid it; an in-process remount
/// did not, and started a second full set on top of the first (bugs.md B44).
/// did not, and started a second full set on top of the first (`docs/BUGS.md` B44).
///
/// Two of the waits need waking rather than merely flagging: the drain workers
/// block on their own condvar (`wake_drain`), and the control listener blocks
/// inside `accept`, which only returns when a connection arrives — so it gets
/// one, from us, after the flag is already set. A join that takes a while is
/// expected and correct: a worker part-way through an upload finishes it rather
/// than abandoning the user's bytes mid-flight.
/// than abandoning the user's bytes mid-flight. Up to [`WORKER_JOIN_DEADLINE`]:
/// past it, systemd kills the daemon anyway, and a queued op is sent again.
fn stop_workers(core: &Core, control_socket: &Path, workers: Vec<std::thread::JoinHandle<()>>) {
// Say so before the joins: a teardown that takes a while must not be read as
// a missed watchdog ping and turned into a kill.
@@ -806,25 +868,71 @@ fn stop_workers(core: &Core, control_socket: &Path, workers: Vec<std::thread::Jo
// unlink cannot put it back to sleep.
let _ = std::os::unix::net::UnixStream::connect(control_socket);
let _ = std::fs::remove_file(control_socket);
for worker in workers {
if worker.join().is_err() {
warn!("a background worker panicked before shutdown");
}
let deadline = Instant::now() + WORKER_JOIN_DEADLINE;
let running = join_until(workers, deadline);
if !running.is_empty() {
warn!(
running = running.join(","),
"background workers did not finish in time; shutting down around them"
);
}
// The FUSE worker pool last, and *before* the caller drops the tokio
// runtime: a worker still inside a job when the runtime goes away panics on
// its next timer with "A Tokio 1.x context was found, but it is being
// shutdown" (seen 2026-09-18 09:58:35). Bounded, so a worker wedged on the
// network cannot hold the stop the user asked for.
core.workers.stop_and_join(WORKER_JOIN_DEADLINE);
core.workers
.stop_and_join(deadline.saturating_duration_since(Instant::now()));
debug!("background workers stopped");
}
/// Join `workers` until `deadline`, and name the ones still running then.
///
/// A stop used to wait for each one however long it took. The sync engine's
/// mirror pass waits on Drive with no deadline, and a stop that waited for it
/// was killed by systemd (`docs/BUGS.md` B165).
fn join_until(workers: Vec<std::thread::JoinHandle<()>>, deadline: Instant) -> Vec<String> {
let mut running = Vec::new();
for worker in workers {
while !worker.is_finished() && Instant::now() < deadline {
std::thread::sleep(Duration::from_millis(10));
}
if !worker.is_finished() {
running.push(worker.thread().name().unwrap_or("unnamed").to_string());
} else if worker.join().is_err() {
warn!("a background worker panicked before shutdown");
}
}
running
}
#[cfg(test)]
mod tests {
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
use super::{SecondaryInsertRejection, SecondaryMountRegistry, teardown_session};
use super::{SecondaryInsertRejection, SecondaryMountRegistry, join_until, teardown_session};
#[test]
fn a_stop_names_the_workers_still_running_at_its_deadline() {
let (release, wait) = std::sync::mpsc::channel::<()>();
let done = std::thread::Builder::new()
.name("done".into())
.spawn(|| {})
.unwrap();
let stuck = std::thread::Builder::new()
.name("stuck".into())
.spawn(move || {
let _ = wait.recv();
})
.unwrap();
let running = join_until(
vec![done, stuck],
Instant::now() + Duration::from_millis(100),
);
assert_eq!(running, ["stuck"]);
drop(release);
}
#[test]
fn session_is_unroutable_before_teardown_starts() {
+94 -8
View File
@@ -226,11 +226,7 @@ fn scale_thumbnail_with_exiftool(
exiftool: &OsStr,
timeout: Duration,
) -> ScaleAttempt {
let direct = image::ImageReader::new(std::io::Cursor::new(bytes))
.with_guessed_format()
.ok()
.and_then(|reader| reader.decode().ok());
let image = match direct {
let image = match decode_upright(bytes) {
Some(image) => image,
None if is_raw_image_name(name) => {
let (preview, orientation) =
@@ -256,6 +252,27 @@ fn scale_thumbnail_with_exiftool(
encode_thumb(&image)
}
/// Decode `bytes` and turn the picture the way its EXIF orientation says.
///
/// `ImageReader::decode` does not: a phone stores a portrait photo as landscape
/// pixels plus an orientation tag, so every such thumbnail came out on its
/// side, and the drain uploaded it to Drive that way for every client to show
/// (`docs/BUGS.md` B226).
fn decode_upright(bytes: &[u8]) -> Option<image::DynamicImage> {
use image::ImageDecoder as _;
let mut decoder = image::ImageReader::new(std::io::Cursor::new(bytes))
.with_guessed_format()
.ok()?
.into_decoder()
.ok()?;
let orientation = decoder
.orientation()
.unwrap_or(image::metadata::Orientation::NoTransforms);
let mut image = image::DynamicImage::from_decoder(decoder).ok()?;
image.apply_orientation(orientation);
Some(image)
}
/// Shrink a decoded image to a [`THUMB_EDGE`] JPEG, keeping its aspect ratio.
fn encode_thumb(image: &image::DynamicImage) -> ScaleAttempt {
let (width, height) = (image.width(), image.height());
@@ -417,6 +434,16 @@ fn exiftool_query(
serde_json::from_slice(&output).map_err(|_| RawPreviewUnavailable)
}
/// Whether [`upload_thumbnails`] can make a thumbnail for a file `name` of
/// `size` bytes, asked before the file is on disk: an image small enough to
/// decode, or a video while ffmpeg is installed.
pub(crate) fn can_make_thumbnail(name: &str, size: u64) -> bool {
if is_video_name(name) {
return ffmpeg_available();
}
is_thumbnail_image_name(name) && size <= UPLOAD_IMAGE_MAX_BYTES
}
/// Whether `name` is a video, whose thumbnail is a frame ffmpeg grabs rather
/// than a picture the image decoder can scale.
fn is_video_name(name: &str) -> bool {
@@ -2092,9 +2119,10 @@ impl Core {
#[cfg(test)]
mod thumb_tests {
use super::{
RawTempFile, ScaleAttempt, THUMB_EDGE, UPLOAD_THUMB_MAX_BYTES, exiftool_binary,
fit_upload_thumb, is_video_name, ratio_of, scale_thumbnail, scale_thumbnail_with_exiftool,
similar_hashes, thumbnail_build_may_start, upload_thumbnails_with, video_thumbnail,
RawTempFile, ScaleAttempt, THUMB_EDGE, UPLOAD_IMAGE_MAX_BYTES, UPLOAD_THUMB_MAX_BYTES,
can_make_thumbnail, exiftool_binary, ffmpeg_available, fit_upload_thumb, is_video_name,
ratio_of, scale_thumbnail, scale_thumbnail_with_exiftool, similar_hashes,
thumbnail_build_may_start, upload_thumbnails_with, video_thumbnail,
};
use pdfs_core::control::{ThumbnailBuildStatus, is_thumbnail_image_name};
use proton_drive_rs::ThumbnailType;
@@ -2149,6 +2177,64 @@ mod thumb_tests {
assert_eq!(ratio_of(&thumb.bytes).map(|r| r < 1.0), Some(true));
}
/// `jpeg` with an EXIF APP1 segment that carries only `orientation`.
fn jpeg_with_orientation(width: u32, height: u32, orientation: u16) -> Vec<u8> {
let plain = jpeg(width, height);
let mut tiff = Vec::new();
tiff.extend_from_slice(b"II*\0");
tiff.extend_from_slice(&8u32.to_le_bytes());
tiff.extend_from_slice(&1u16.to_le_bytes());
tiff.extend_from_slice(&0x0112u16.to_le_bytes());
tiff.extend_from_slice(&3u16.to_le_bytes());
tiff.extend_from_slice(&1u32.to_le_bytes());
tiff.extend_from_slice(&orientation.to_le_bytes());
tiff.extend_from_slice(&[0, 0]);
tiff.extend_from_slice(&0u32.to_le_bytes());
let mut app1 = b"Exif\0\0".to_vec();
app1.extend_from_slice(&tiff);
let mut out = plain[..2].to_vec();
out.extend_from_slice(&[0xFF, 0xE1]);
out.extend_from_slice(&u16::try_from(app1.len() + 2).unwrap().to_be_bytes());
out.extend_from_slice(&app1);
out.extend_from_slice(&plain[2..]);
out
}
/// A phone stores a portrait photo as landscape pixels and an EXIF
/// orientation; its thumbnail came out on its side (`docs/BUGS.md` B226).
/// A Takeout photo is copied out of its archive only when a thumbnail can
/// be made from the copy.
#[test]
fn only_a_photo_that_can_have_a_thumbnail_is_worth_copying_out() {
assert!(can_make_thumbnail("IMG_1.jpg", 4 << 20));
assert!(can_make_thumbnail("Screenshot.png", 4 << 20));
assert!(!can_make_thumbnail("IMG_1.jpg", UPLOAD_IMAGE_MAX_BYTES + 1));
assert!(!can_make_thumbnail("notes.txt", 10));
assert_eq!(can_make_thumbnail("clip.mp4", 1 << 30), ffmpeg_available());
}
#[test]
fn a_thumbnail_is_turned_the_way_its_exif_says() {
let ScaleAttempt::Made(thumb) = scale_thumbnail(
&jpeg_with_orientation(2000, 1000, 6),
"portrait.jpg",
&std::env::temp_dir(),
) else {
panic!("a JPEG scales");
};
assert!(thumb.ratio < 1.0, "turned upright: {}", thumb.ratio);
assert_eq!(ratio_of(&thumb.bytes).map(|r| r < 1.0), Some(true));
let ScaleAttempt::Made(level) = scale_thumbnail(
&jpeg_with_orientation(2000, 1000, 1),
"landscape.jpg",
&std::env::temp_dir(),
) else {
panic!("a JPEG scales");
};
assert!(level.ratio > 1.0);
}
#[test]
fn undecodable_bytes_are_not_a_thumbnail() {
// What a photo in a format this build has no decoder for looks like: the
+115 -49
View File
@@ -1,5 +1,5 @@
//! Reading and writing this machine's `profile.json` in its device root, and
//! the restore that rebuilds a machine from one (features.md 5.2 and 5.3).
//! the restore that rebuilds a machine from one.
//!
//! The document itself is defined in [`pdfs_core::profile`]; this is the half
//! that talks to Drive. Saving is best-effort and never blocks a user action:
@@ -15,6 +15,7 @@ use pdfs_core::control::RestorableFolder;
use pdfs_core::mounts::MountMode;
use pdfs_core::profile::{
PROFILE_DIR_NAME, PROFILE_FILE_NAME, PROFILE_VERSION, Profile, ProfileFolder, ProfilePin,
is_profile_file_name, profile_file_name, saved_folder,
};
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::proton_sdk::ids::NodeUid;
@@ -66,9 +67,14 @@ impl Core {
});
}
/// Serialize this machine's arrangement and upload it to `profile.json`
/// inside the device's `.proton-drive-linux` folder, replacing any existing
/// one.
/// Serialize this machine's arrangement and upload it to its own
/// `profile-<machine id>.json` inside the device's `.proton-drive-linux`
/// folder, replacing any existing one.
///
/// Each machine writes its own file because several computers may sync
/// into one device. They used to share `profile.json`, and each save
/// replaced the other computers' folders with its own (B252). That file is
/// left as it is, so a client before 3.5.0 still finds it.
///
/// Does nothing when no device is registered: a machine that has never
/// synced a folder has nothing to describe, and registering a device just to
@@ -85,7 +91,11 @@ impl Core {
CoreError::internal(format!("bad device root uid: {}", device.root_uid))
})?;
let profile = self.build_profile(&device.uid)?;
let machine_id = pdfs_core::config::AppDirs::new()
.and_then(|dirs| dirs.machine_id())
.map_err(|e| CoreError::internal(format!("machine id: {e}")))?;
let file_name = profile_file_name(&machine_id);
let profile = self.build_profile(&device.uid, &machine_id)?;
let bytes = profile.to_bytes().map_err(CoreError::internal)?;
let len = bytes.len() as i64;
@@ -97,7 +107,7 @@ impl Core {
// same name: the profile is a single mutable record, and a folder with
// `profile.json` and `profile (1).json` in it teaches the restore
// nothing about which to trust.
let existing = self.find_device_child_file(&dir_uid, PROFILE_FILE_NAME)?;
let existing = self.find_device_child_file(&dir_uid, &file_name)?;
match existing {
Some(uid) => self
.rt
@@ -114,7 +124,7 @@ impl Core {
self.rt
.block_on(self.drive.upload_file_replacing_draft_from(
&dir_uid,
PROFILE_FILE_NAME,
&file_name,
"application/json",
Cursor::new(bytes),
len,
@@ -130,7 +140,7 @@ impl Core {
}
/// Snapshot the local arrangement into a [`Profile`].
fn build_profile(&self, device_uid: &str) -> CoreResult<Profile> {
fn build_profile(&self, device_uid: &str, machine_id: &str) -> CoreResult<Profile> {
let folders = self
.db
.sync_folder_list()
@@ -162,6 +172,7 @@ impl Core {
version: PROFILE_VERSION,
device_uid: device_uid.to_string(),
hostname: this_hostname(),
machine_id: Some(machine_id.to_string()),
saved_at: now_secs(),
folders,
pins,
@@ -197,6 +208,31 @@ impl Core {
.map(|n| n.uid))
}
/// The untrashed files directly under `parent_uid` whose name passes
/// `keep`.
fn device_child_files(
&self,
parent_uid: &NodeUid,
keep: impl Fn(&str) -> bool,
) -> CoreResult<Vec<NodeUid>> {
let uids = self
.rt
.block_on(self.drive.enumerate_folder_children_node_uids(parent_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() {
return Ok(Vec::new());
}
let nodes = self
.rt
.block_on(self.drive.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?;
Ok(nodes
.into_iter()
.filter(|n| !n.is_folder() && !n.trashed && keep(&n.name))
.map(|n| n.uid)
.collect())
}
/// The device's `.proton-drive-linux` folder, creating it if this is the
/// first save on this device.
///
@@ -218,35 +254,46 @@ impl Core {
Ok(uid)
}
/// Download and parse this device's profile, if there is one.
/// Download and parse the profiles of every computer that syncs this
/// device.
///
/// Looks in `.proton-drive-linux/` first and falls back to the device root,
/// where clients before the B78 fix wrote it. The fallback is read-only: the
/// next save writes to the folder, and the legacy document is left alone
/// rather than trashed, so an older client on another machine keeps working.
fn load_profile(&self, root_uid: &NodeUid) -> CoreResult<Option<Profile>> {
let current = match self.find_device_child_folder(root_uid, PROFILE_DIR_NAME)? {
Some(dir_uid) => self.find_device_child_file(&dir_uid, PROFILE_FILE_NAME)?,
None => None,
/// Reads every profile in `.proton-drive-linux/`: one per machine, and the
/// shared `profile.json` of clients before 3.5.0. Falls back to the device
/// root, where clients before the B78 fix wrote it. The fallback is
/// read-only: the next save writes to the folder, and the legacy document
/// is left alone rather than trashed, so an older client on another
/// machine keeps working. A profile that cannot be read is left out and
/// logged; the others still count.
fn load_profiles(&self, root_uid: &NodeUid) -> CoreResult<Vec<Profile>> {
let mut uids = match self.find_device_child_folder(root_uid, PROFILE_DIR_NAME)? {
Some(dir_uid) => self.device_child_files(&dir_uid, is_profile_file_name)?,
None => Vec::new(),
};
let uid = match current {
Some(uid) => uid,
None => match self.find_device_child_file(root_uid, PROFILE_FILE_NAME)? {
Some(uid) => {
info!("reading legacy profile from the device root");
uid
if uids.is_empty() {
uids = self.device_child_files(root_uid, |name| name == PROFILE_FILE_NAME)?;
if !uids.is_empty() {
info!("reading legacy profile from the device root");
}
}
let mut profiles = Vec::new();
for uid in uids {
let mut buf: Vec<u8> = Vec::new();
let parsed = self
.rt
.block_on(self.drive.download_file_to(&uid, &mut buf))
.map_err(|e| CoreError::from_api(&e, "download profile"))
.and_then(|()| Profile::parse(&buf).map_err(CoreError::invalid));
match parsed {
Ok(profile) => profiles.push(profile),
Err(e) => {
warn!(%uid, error = %e.message, "could not read a profile; leaving it out")
}
None => return Ok(None),
},
};
let mut buf: Vec<u8> = Vec::new();
self.rt
.block_on(self.drive.download_file_to(&uid, &mut buf))
.map_err(|e| CoreError::from_api(&e, "download profile"))?;
Profile::parse(&buf).map(Some).map_err(CoreError::invalid)
}
}
Ok(profiles)
}
// ---- restore (features.md 5.2) ----------------------------------------
// ---- restore ----------------------------------------
/// The folders under a device that could be synced here, each carrying a
/// proposed local path. `device` names another registered device by uid;
@@ -265,15 +312,16 @@ impl Core {
CoreError::internal(format!("bad device root uid: {}", device.root_uid))
})?;
let profile = match self.load_profile(&root_uid) {
let profiles = match self.load_profiles(&root_uid) {
Ok(p) => p,
// A missing or unreadable profile is not fatal: the remote tree
// alone is enough to offer a restore, just with plainer defaults.
Err(e) => {
warn!(error = %e.message, "could not read profile; proposing defaults");
None
warn!(error = %e.message, "could not read profiles; proposing defaults");
Vec::new()
}
};
let hostname = this_hostname();
let uids = self
.rt
@@ -304,11 +352,12 @@ impl Core {
// offering it as a restorable folder would sync the bookkeeping
// back onto the machine it describes.
.filter(|n| n.is_folder() && !n.trashed && n.name != PROFILE_DIR_NAME)
// The name becomes a folder under the home folder: one with a `/`
// or a `..` in it would put the restore somewhere else.
.filter(|n| crate::remote_name_is_safe(&n.name))
.map(|n| {
let uid = n.uid.to_string();
let saved = profile
.as_ref()
.and_then(|p| p.folders.iter().find(|f| f.remote_uid == uid));
let saved = saved_folder(&profiles, &uid, &hostname);
let proposed = saved
.map(|f| PathBuf::from(&f.local_path))
.filter(|p| path_is_usable_here(p))
@@ -349,23 +398,29 @@ impl Core {
.sync_folder_list()
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?;
let reserved = crate::devices::sync_root_reserved();
let mut roots: Vec<PathBuf> = existing
.iter()
.map(|f| PathBuf::from(&f.local_path))
.collect();
let mut restored = 0usize;
let mut skipped: Vec<String> = Vec::new();
for item in items {
let local = PathBuf::from(&item.local_path);
if !local.is_absolute() {
skipped.push(format!("{}: local path must be absolute", item.local_path));
continue;
}
// The path is proposed from the profile another machine wrote, and
// may hold a folder name from Drive: checked like any new root.
let local = match pdfs_core::syncroot::resolve(Path::new(&item.local_path)) {
Ok(local) => local,
Err(e) => {
skipped.push(format!("{}: {e}", item.local_path));
continue;
}
};
if existing.iter().any(|f| f.remote_uid == item.remote_uid) {
skipped.push(format!("{}: already synced", item.local_path));
continue;
}
if existing
.iter()
.any(|f| Path::new(&f.local_path) == local.as_path())
{
skipped.push(format!("{}: path already in use", item.local_path));
if let Some(problem) = pdfs_core::syncroot::problem(&local, &roots, &reserved) {
skipped.push(format!("{}: {problem}", item.local_path));
continue;
}
if let Err(e) = std::fs::create_dir_all(&local) {
@@ -390,6 +445,7 @@ impl Core {
let _ = self.request_sync_folder_mode(id, "ondemand");
}
let _ = self.sync_tx.send(sync::SyncMsg::Reconcile(id));
roots.push(local);
restored += 1;
}
let _ = self.sync_tx.send(sync::SyncMsg::Rewatch);
@@ -482,7 +538,7 @@ mod tests {
#[test]
fn loading_falls_back_to_a_legacy_device_root_profile() {
let source = include_str!("profile.rs");
let load = function_source(source, "fn load_profile(");
let load = function_source(source, "fn load_profiles(");
assert!(load.contains("PROFILE_DIR_NAME"), "prefers the folder");
assert!(
load.find("PROFILE_DIR_NAME").unwrap()
@@ -491,6 +547,16 @@ mod tests {
);
}
/// Several computers may share a device, so each saves to a file named
/// after its own machine id and never to the shared `profile.json`
/// (B252).
#[test]
fn each_machine_saves_its_own_profile() {
let save = function_source(include_str!("profile.rs"), "fn save_profile(");
assert!(save.contains("profile_file_name(&machine_id)"));
assert!(!save.contains("PROFILE_FILE_NAME"));
}
/// The profile folder is the client's own bookkeeping: neither the restore
/// picker nor `add_sync_folder` may treat it as a user folder.
#[test]
+247 -5
View File
@@ -1,12 +1,16 @@
//! The user's view of the pending-op queue: what has not reached the remote
//! yet, why, and a way to stop waiting out a backoff.
//!
//! Read-only apart from "retry now", which only moves a row's next-attempt time.
//! Nothing here drops a queued op: its staged blob is the only copy of the
//! user's write (see `docs/RECOVERY.md`).
//! Read-only apart from "retry now", which only moves a row's next-attempt time,
//! "export", which copies a staged blob out, and "discard". Only a discard the
//! user asks for drops a queued op: its staged blob is the only copy of the
//! user's write (see `docs/RECOVERY.md`), so it is the user's call, made after
//! an export or for a change with no content.
use super::*;
use pdfs_core::control::PendingOpInfo;
use std::io::Write;
use pdfs_core::control::{PendingOpInfo, SyncIssue};
use pdfs_core::db::{FAILING_ATTEMPTS, PARK_UNTIL};
impl Core {
@@ -16,10 +20,15 @@ impl Core {
.db
.pending_ops()
.map_err(|e| CoreError::internal(format!("listing the queue: {e}")))?;
let issues = self
.db
.op_issues()
.map_err(|e| CoreError::internal(format!("listing sync issues: {e}")))?;
Ok(ops
.into_iter()
.map(|op| {
let parked = op.next_attempt_at >= PARK_UNTIL;
let exportable = exportable(&op).is_some();
let path = self
.db
.node_path(&op.uid)
@@ -37,7 +46,9 @@ impl Core {
queued_at: op.created_at / 1000,
next_attempt_at: (!parked).then_some(op.next_attempt_at / 1000),
parked,
failing: op.attempts >= FAILING_ATTEMPTS,
failing: op.attempts >= FAILING_ATTEMPTS || issues.contains_key(&op.id),
issue: issues.get(&op.id).map(|issue| SyncIssue::parse(issue)),
exportable,
}
})
.collect())
@@ -57,4 +68,235 @@ impl Core {
}
Ok(moved)
}
/// Copy the content of queued op `id` to `dest`, which must not exist yet.
/// The op stays queued. Returns the bytes written.
pub(crate) fn export_pending_op(&self, id: i64, dest: &Path) -> CoreResult<u64> {
if !dest.is_absolute() {
return Err(CoreError::invalid(format!(
"destination must be an absolute path: {}",
dest.display()
)));
}
let ops = self
.db
.pending_ops()
.map_err(|e| CoreError::internal(format!("listing the queue: {e}")))?;
let op = ops
.iter()
.find(|op| op.id == id)
.ok_or_else(|| CoreError::not_found(format!("no queued change {id}")))?;
let blob = exportable(op).ok_or_else(|| {
CoreError::invalid(format!("queued change {id} carries no file content"))
})?;
let written = copy_to_new_file(blob, dest)?;
info!(id, dest = %dest.display(), written, "exported a queued change's content");
Ok(written)
}
/// Drop queued op `id` and undo the local change it stood for. Returns the
/// node's path.
///
/// A node only made here goes, as if it had been deleted before it was
/// sent. A node Drive has goes back to what Drive has: every change queued
/// on it goes, since each was made on top of the one before. What is queued
/// below it stays.
pub(crate) fn discard_pending_op(&self, id: i64) -> CoreResult<String> {
let ops = self
.db
.pending_ops()
.map_err(|e| CoreError::internal(format!("listing the queue: {e}")))?;
let op = ops
.into_iter()
.find(|op| op.id == id)
.ok_or_else(|| CoreError::not_found(format!("no queued change {id}")))?;
let uid = parse_node_uid(&op.uid)
.ok_or_else(|| CoreError::internal(format!("queued change {id} names {}", op.uid)))?;
let path = self
.db
.node_path(&op.uid)
.ok()
.flatten()
.filter(|path| !path.is_empty())
.or(op.name)
.unwrap_or_else(|| op.uid.clone());
if self.is_open_anywhere(&uid) {
return Err(CoreError::conflict(format!(
"{path} is open; close it first"
)));
}
if is_local_uid(&uid) {
self.discard_local_node(&uid, &path)?;
} else {
self.discard_remote_change(&uid, &path)?;
}
info!(id, %uid, path, "discarded a queued change");
Ok(path)
}
/// Forget a node made here, and the ops that would have sent it.
fn discard_local_node(&self, uid: &NodeUid, path: &str) -> CoreResult<()> {
// Its children's ops go with its own, and their content with them.
if self
.db
.has_children(uid)
.map_err(|e| CoreError::internal(format!("listing {path}: {e}")))?
{
return Err(CoreError::conflict(format!(
"{path} is not empty; discard what is in it first"
)));
}
self.discard_queued_ops(uid)
.map_err(|_| CoreError::internal(format!("dropping the changes to {path}")))?;
self.for_each_mount(|st, notify| {
background::forget_and_notify(st, notify, uid);
});
self.cache.evict(uid);
self.evict_reader(uid);
Ok(())
}
/// Put a node Drive has back as Drive has it.
fn discard_remote_change(&self, uid: &NodeUid, path: &str) -> CoreResult<()> {
// Asked first: the change is only dropped once what replaces it is known.
let node = self
.fetch_node_remote(uid)
.map_err(|e| CoreError::from_api(&e, "reading the node from Drive"))?
.filter(|node| !node.trashed);
self.cancel_upload(uid);
let blobs = self
.db
.discard_ops_on_node(&uid.to_string())
.map_err(|e| CoreError::internal(format!("dropping the changes to {path}: {e}")))?
.ok_or_else(|| {
CoreError::conflict(format!("{path} is being sent right now; try again"))
})?;
self.release_dropped_ops(uid, blobs);
self.hidden.lock().remove(uid);
match node {
Some(node) => self.for_each_mount(|st, notify| {
put_back(st, notify, node);
}),
None => self.for_each_mount(|st, notify| {
background::forget_and_notify(st, notify, uid);
}),
}
self.cache.evict(uid);
self.evict_reader(uid);
Ok(())
}
}
/// Show `node` as Drive has it, where Drive has it, under the inode it has.
fn put_back(st: &mut State, notify: &mut NotifyBatch, node: Node) {
let parent = node.parent_uid.as_ref().and_then(|parent| {
st.by_uid
.get(parent)
.copied()
.map(|ino| (ino, parent.clone()))
});
let here = st.by_uid.get(&node.uid).and_then(|ino| {
st.entries
.get(ino)
.map(|entry| (*ino, entry.parent, entry.node.name.clone()))
});
match (here, parent) {
(Some((ino, from, name)), Some((to, to_uid))) => {
st.relocate(ino, from, to, &to_uid, &node.name);
st.write_through(node.clone());
st.intern_mem(to, node);
notify.inval_entry(from, name);
notify.inval_inode(ino);
}
(here, parent) => {
if let Some((_, from, name)) = here {
st.forget_mem(&node.uid);
st.invalidate_listing(from);
notify.inval_entry(from, name);
}
if let Some((to, _)) = parent {
st.invalidate_listing(to);
notify.inval_entry(to, node.name.clone());
}
st.write_through(node);
}
}
}
/// Copy `blob` to `dest`, which must not exist: never replace something the
/// user already has.
fn copy_to_new_file(blob: &Path, dest: &Path) -> CoreResult<u64> {
let mut source = File::open(blob)
.map_err(|e| CoreError::internal(format!("open {}: {e}", blob.display())))?;
let mut out = std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.open(dest)
.map_err(|e| match e.kind() {
std::io::ErrorKind::AlreadyExists => {
CoreError::invalid(format!("{} already exists", dest.display()))
}
_ => CoreError::internal(format!("create {}: {e}", dest.display())),
})?;
std::io::copy(&mut source, &mut out)
.and_then(|written| out.flush().and(out.sync_all()).map(|()| written))
.map_err(|e| CoreError::internal(format!("write {}: {e}", dest.display())))
}
/// The staged blob of an op that carries file content.
fn exportable(op: &PendingOp) -> Option<&Path> {
matches!(op.kind.as_str(), OP_CREATE | OP_REVISION)
.then_some(op.blob_path.as_deref())
.flatten()
.map(Path::new)
}
#[cfg(test)]
mod tests {
use super::*;
fn op(kind: &str, blob: Option<&str>) -> PendingOp {
PendingOp {
id: 1,
kind: kind.to_string(),
uid: "vol~a".to_string(),
parent_uid: None,
name: None,
blob_path: blob.map(str::to_string),
meta_json: None,
created_at: 0,
attempts: 0,
last_error: None,
next_attempt_at: 0,
}
}
#[test]
fn only_a_change_that_carries_file_content_can_be_exported() {
assert!(exportable(&op(OP_CREATE, Some("/staging/a"))).is_some());
assert!(exportable(&op(OP_REVISION, Some("/staging/a"))).is_some());
assert!(exportable(&op(OP_CREATE, None)).is_none());
assert!(exportable(&op(OP_RENAME, None)).is_none());
assert!(exportable(&op(OP_TRASH, None)).is_none());
}
#[test]
fn an_export_copies_the_bytes_and_never_replaces_a_file() {
let dir = std::env::temp_dir().join(format!("pdfs-export-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let blob = dir.join("blob");
std::fs::write(&blob, b"queued bytes").unwrap();
let dest = dir.join("copy.txt");
assert_eq!(copy_to_new_file(&blob, &dest).unwrap(), 12);
assert_eq!(std::fs::read(&dest).unwrap(), b"queued bytes");
std::fs::write(&dest, b"the user's own").unwrap();
assert!(copy_to_new_file(&blob, &dest).is_err());
assert_eq!(std::fs::read(&dest).unwrap(), b"the user's own");
// The staged blob is the only copy of the change: an export leaves it.
assert_eq!(std::fs::read(&blob).unwrap(), b"queued bytes");
let _ = std::fs::remove_dir_all(&dir);
}
}
+138 -41
View File
@@ -464,7 +464,7 @@ impl Core {
/// it, and the [`BLOCK_SIZE`] assumption when it has not — which is every
/// first read, because learning it means opening a `RevisionReader`, and
/// opening one per read is precisely the per-file key derivation the block
/// cache exists to avoid (bugs.md B12, B85).
/// cache exists to avoid (`docs/BUGS.md` B12, B85).
///
/// Being wrong here is not a correctness problem: `read_at` plans over the
/// real sizes and clamps to what was asked for either way. It costs a
@@ -493,15 +493,20 @@ impl Core {
) -> Result<Vec<u8>, Errno> {
// A queued write has not reached the remote yet, so the remote's current
// revision is stale and the staged blob is the truth. Serve from it until
// the drain worker lands the upload (offline.md Phase 3).
// the drain worker lands the upload.
if let Some(pending) = self.pending.lock().get(uid).cloned() {
return self.read_pending(&pending, offset, len);
}
// A node created offline and never written has no blob and no remote: it
// is an empty file, and asking the API about a `local~` uid would only
// earn a 404 (offline.md Phase 3b).
// earn a 404. Unless its create landed since the
// read looked: the landing moved the queued write to the uid Drive
// gave it, and the row says which (`docs/BUGS.md` B179).
if is_local_uid(uid) {
return Ok(Vec::new());
return match self.landed_stand_in_uid(uid) {
Some(real) => self.read_range(&real, mtime, fsize, offset, len, cache_blocks),
None => Ok(Vec::new()),
};
}
self.read_range_remote(uid, mtime, fsize, offset, len, cache_blocks)
}
@@ -631,7 +636,7 @@ impl Core {
///
/// A block that comes back shorter than the file's size says it must be is
/// never served, cached or promoted: it is repaired from the whole-file
/// download, or the read fails. See [`Core::repair_block`] (bugs.md B84).
/// download, or the read fails. See [`Core::repair_block`] (`docs/BUGS.md` B84).
async fn read_block(
&self,
reader: &Arc<dyn RevisionRead>,
@@ -646,7 +651,7 @@ impl Core {
// file's boundaries are wrong, and the failure is not confined to short
// reads: sizes that *overstate* a block shift every block after it and
// return full-length reads of the wrong bytes. So a disagreement of any
// kind takes the file off the range path entirely (bugs.md B84).
// kind takes the file off the range path entirely (`docs/BUGS.md` B84).
let bytes = if reader.size() != fsize {
warn!(
%uid, bidx = span.idx, fsize,
@@ -659,7 +664,7 @@ impl Core {
// Learned here rather than on a path of its own: this is where a
// reader is open anyway, and recording the real geometry is what
// lets the *next* read plan its blocks on the revision's own
// boundaries instead of the 4 MiB fallback (bugs.md B85).
// boundaries instead of the 4 MiB fallback (`docs/BUGS.md` B85).
if cache_blocks {
self.cache
.store_block_geometry(uid, mtime, fsize, reader.block_sizes());
@@ -701,7 +706,7 @@ impl Core {
/// The range path derives every block boundary from the revision's recorded
/// block sizes. When those understate the file, reads come back short with
/// no error anywhere, and the caller hands userspace a truncated file that
/// looks like a clean EOF (bugs.md B84). `download_file_to` does not do that
/// looks like a clean EOF (`docs/BUGS.md` B84). `download_file_to` does not do that
/// arithmetic — it streams the block list — so it returns the whole file
/// where the range path cannot, which is why pinning was the workaround.
/// This does the same thing on the user's behalf, once, and the resulting
@@ -913,7 +918,7 @@ impl Core {
let end = offset.saturating_add(len).min(fsize);
let mut out = Vec::with_capacity((end - offset) as usize);
// On the revision's own boundaries where they are known, and on the
// 4 MiB fallback where they are not (bugs.md B85).
// 4 MiB fallback where they are not (`docs/BUGS.md` B85).
let wanted = self.block_geometry(uid, mtime, fsize).spans(offset, end);
let (Some(&head), Some(&tail)) = (wanted.first(), wanted.last()) else {
return Ok(out);
@@ -1033,7 +1038,7 @@ impl Core {
// — the cache validates it on read, the network path in `read_block`
// — so this refuses rather than quietly contributing fewer bytes
// than the range covers, which is what turned a bad block into a
// truncated file (bugs.md B84).
// truncated file (`docs/BUGS.md` B84).
if block.len() as u64 != span.len {
error!(
%uid, bidx = span.idx, len = block.len(), expected = span.len, fsize,
@@ -1065,37 +1070,35 @@ impl Core {
offset: u64,
size: u64,
) -> Result<Vec<u8>, Errno> {
if offset >= len || size == 0 {
return Ok(Vec::new());
}
let end = offset.saturating_add(size).min(len);
let mut out = Vec::with_capacity((end - offset) as usize);
for (s, e, authored) in written.segments(offset, end) {
if authored {
let mut buf = vec![0u8; (e - s) as usize];
file.read_exact_at(&mut buf, s).map_err(|err| {
warn!(%uid, error = %err, "scratch read failed");
Errno::EIO
})?;
out.extend_from_slice(&buf);
} else {
let bend = e.min(base_size);
if s < bend {
out.extend_from_slice(&self.read_range(
uid,
base_mtime,
base_size,
s,
bend - s,
true,
)?);
}
// Anything past the base is a hole: zero-fill.
let zeros = e.saturating_sub(s.max(base_size));
out.resize(out.len() + zeros as usize, 0);
}
}
Ok(out)
let read = assemble_open_read(file, len, uid, base_size, written, offset, size, |s, n| {
self.read_range(uid, base_mtime, base_size, s, n, true)
.map(Some)
})?;
Ok(read.unwrap_or_default())
}
/// [`Core::serve_open_read`] from bytes already on this machine only:
/// `Ok(None)` the moment a range of the base would have to be downloaded.
///
/// The dispatch loop answers a read on a write handle with this, and hands
/// it to a worker only when it says `None`. Fetching the base there held
/// every request of the mount for as long as the download took, up to the
/// read deadline (`docs/BUGS.md` B227).
#[allow(clippy::too_many_arguments)]
pub(super) fn serve_open_read_local(
&self,
file: &Arc<File>,
len: u64,
uid: &NodeUid,
base_mtime: i64,
base_size: u64,
written: &Intervals,
offset: u64,
size: u64,
) -> Result<Option<Vec<u8>>, Errno> {
assemble_open_read(file, len, uid, base_size, written, offset, size, |s, n| {
Ok(self.cached_range(uid, base_mtime, base_size, s, n))
})
}
/// The largest amount [`Core::fill_gaps_cached`] will copy out of the local
@@ -1237,6 +1240,49 @@ impl Core {
}
}
/// The bytes `[offset, offset + size)` of a file open for writing: authored
/// ranges from its scratch `file`, the rest from its base through `base`,
/// and zeros past the base. `Ok(None)` when `base` says it cannot answer.
#[allow(clippy::too_many_arguments)]
fn assemble_open_read(
file: &File,
len: u64,
uid: &NodeUid,
base_size: u64,
written: &Intervals,
offset: u64,
size: u64,
mut base: impl FnMut(u64, u64) -> Result<Option<Vec<u8>>, Errno>,
) -> Result<Option<Vec<u8>>, Errno> {
if offset >= len || size == 0 {
return Ok(Some(Vec::new()));
}
let end = offset.saturating_add(size).min(len);
let mut out = Vec::with_capacity((end - offset) as usize);
for (s, e, authored) in written.segments(offset, end) {
if authored {
let mut buf = vec![0u8; (e - s) as usize];
file.read_exact_at(&mut buf, s).map_err(|err| {
warn!(%uid, error = %err, "scratch read failed");
Errno::EIO
})?;
out.extend_from_slice(&buf);
} else {
let bend = e.min(base_size);
if s < bend {
match base(s, bend - s)? {
Some(bytes) => out.extend_from_slice(&bytes),
None => return Ok(None),
}
}
// Anything past the base is a hole: zero-fill.
let zeros = e.saturating_sub(s.max(base_size));
out.resize(out.len() + zeros as usize, 0);
}
}
Ok(Some(out))
}
#[cfg(test)]
mod block_ring_tests {
use super::*;
@@ -1389,3 +1435,54 @@ mod prefetch_tests {
assert!(prefetch.streams.lock().len() <= MAX_PREFETCH_STREAMS + 1);
}
}
#[cfg(test)]
mod open_read_tests {
use super::*;
use proton_drive_rs::proton_sdk::ids::{LinkId, VolumeId};
fn scratch(bytes: &[u8]) -> (File, PathBuf) {
let path = std::env::temp_dir().join(format!(
"pdfs-open-read-{}-{}",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
let mut file = File::options()
.create_new(true)
.read(true)
.write(true)
.open(&path)
.unwrap();
file.write_all(bytes).unwrap();
(file, path)
}
/// The dispatch loop answers from what is on this machine and says so
/// when the base would have to be downloaded, for a worker to do it
/// (`docs/BUGS.md` B227).
#[test]
fn a_read_needing_the_base_says_so_instead_of_fetching() {
let uid = NodeUid::new(VolumeId::from("v"), LinkId::from("f"));
let (file, path) = scratch(b"ABCDEFGH");
let mut written = Intervals::default();
written.add(0, 4);
let authored = assemble_open_read(&file, 8, &uid, 8, &written, 0, 4, |_, _| {
panic!("an authored range never asks the base")
});
assert_eq!(authored.unwrap().as_deref(), Some(&b"ABCD"[..]));
let missing = assemble_open_read(&file, 8, &uid, 8, &written, 2, 6, |_, _| Ok(None));
assert_eq!(missing.unwrap(), None);
let base = assemble_open_read(&file, 10, &uid, 8, &written, 2, 8, |s, n| {
assert_eq!((s, n), (4, 4));
Ok(Some(b"efgh".to_vec()))
});
assert_eq!(base.unwrap().as_deref(), Some(&b"CDefgh\0\0"[..]));
let _ = std::fs::remove_file(path);
}
}
+10 -2
View File
@@ -190,6 +190,7 @@ impl Core {
.map_err(|e| CoreError::internal(format!("stat {}: {e}", path.display())))?
.len();
let media_type = media_type_of(&candidate.name).unwrap_or("application/octet-stream");
let thumbnails = self.upload_thumbnails(&path, &candidate.name);
let guard =
self.transfers
@@ -206,7 +207,7 @@ impl Core {
media_type,
reader,
size as i64,
Vec::new(),
thumbnails,
metadata,
false,
));
@@ -291,7 +292,14 @@ impl Core {
let mut state = REDATE.lock();
state.running = false;
state.cancel = None;
state.last = outcome.as_ref().ok().cloned();
state.last = Some(match &outcome {
Ok(summary) => summary.clone(),
Err(e) => RedateSummary {
error: Some(e.to_string()),
dry_run,
..Default::default()
},
});
match &outcome {
Ok(summary) => info!(
candidates = summary.candidates,
+35 -9
View File
@@ -25,6 +25,7 @@ use proton_drive_rs::NodeKind;
use proton_drive_rs::proton_sdk::ids::NodeUid;
use tracing::{info, warn};
use super::state::State;
use super::sync::mirror_subtree_unsynced;
use super::{Core, is_local_uid, parse_uid};
@@ -130,6 +131,24 @@ fn rename_noreplace(from: &Path, to: &Path) -> std::io::Result<()> {
}
}
/// Whether inode `ino` is `root` or lies under it in the tree. Bounded, so a
/// parent cycle in corrupt data cannot hang the caller.
fn ino_is_under(st: &State, mut ino: u64, root: u64) -> bool {
for _ in 0..MAX_TREE_DEPTH {
if ino == root {
return true;
}
match st.entries.get(&ino) {
Some(entry) if entry.parent != ino => ino = entry.parent,
_ => return false,
}
}
false
}
/// How deep [`ino_is_under`] walks before it gives up.
const MAX_TREE_DEPTH: usize = 1024;
/// Whether `uid` is `root` or lies under it, as far as the nodes table knows.
fn under(core: &Core, root: &NodeUid, uid: &str) -> bool {
let root = root.to_string();
@@ -336,7 +355,7 @@ impl Core {
core.require_uid_writable(authority)
.map_err(|e| core.errno_error(e, "move access"))?;
}
if self.has_pending_work_under(&uid)? {
if self.has_pending_work_under(ino, &uid)? {
return Err(CoreError::conflict(
"it has changes that are not uploaded yet; wait for the upload to finish",
));
@@ -410,10 +429,11 @@ impl Core {
Ok(uid)
}
/// Whether the queue owes Drive anything for `uid` or a node under it, or a
/// file under it is open for writing. Moving such a node away would strand
/// the pending change against a path that no longer leads to it.
fn has_pending_work_under(&self, uid: &NodeUid) -> CoreResult<bool> {
/// Whether the queue owes Drive anything for `uid` (inode `ino`) or a node
/// under it, or a file under it is open for writing. Moving such a node away
/// would strand the pending change against a path that no longer leads to
/// it.
fn has_pending_work_under(&self, ino: u64, uid: &NodeUid) -> CoreResult<bool> {
let ops = self
.db
.pending_ops()
@@ -427,12 +447,18 @@ impl Core {
}) {
return Ok(true);
}
// Collected first so no State lock is held across the nodes-table reads.
let mut writing = Vec::new();
// Open handles are placed in the tree by inode, not by uid in the nodes
// table. A file opened just after its create landed holds the
// placeholder uid until the tree catches up, and the nodes table no
// longer has that uid, so the move went ahead under an open write.
let mut writing = false;
self.for_each_state(|st| {
writing.extend(st.active_writes.values().map(|w| w.uid.to_string()));
writing = st
.active_writes
.keys()
.any(|&open| ino_is_under(st, open, ino));
});
Ok(writing.iter().any(|w| under(self, uid, w)))
Ok(writing)
}
/// Bring the source location level with a move Drive has made.
+22 -12
View File
@@ -66,6 +66,7 @@ fn shared_entry_of(core: &Core, n: Node) -> DirEntry {
size,
modified: n.modification_time,
pinned: false,
pinned_by_folder: false,
cached: false,
path: core.rel_path_for_uid(&n.uid).unwrap_or_default(),
uid: n.uid.to_string(),
@@ -75,6 +76,7 @@ fn shared_entry_of(core: &Core, n: Node) -> DirEntry {
shared_by_unverified,
trashed_at: 0,
trashed_from: None,
issue: None,
}
}
@@ -659,6 +661,12 @@ impl Core {
/// the shared uids; the per-node detail is then gathered best-effort — a single
/// node racing with an unshare drops from the list rather than failing the whole
/// request.
///
/// A share with no member, no invitation and no link is left out: it is
/// shared with no one. Stop Sharing leaves exactly that behind, since the SDK
/// cannot delete a share, and Drive keeps listing it until its backend gets
/// round to removing it, minutes later. A share whose details could not all
/// be read stays listed, so a failed lookup never hides a live share.
pub(crate) fn list_shared_by_me(&self) -> CoreResult<Vec<SharedItem>> {
let uids = self
.rt
@@ -677,24 +685,26 @@ impl Core {
let members = self
.rt
.block_on(self.client.list_share_members(&uid))
.map(|m| m.len())
.unwrap_or(0);
.map(|m| m.len());
let proton_invites = self
.rt
.block_on(self.client.list_share_invitations(&uid))
.map(|i| i.len())
.unwrap_or(0);
.map(|i| i.len());
let external_invites = self
.rt
.block_on(self.client.list_external_invitations(&uid))
.map(|i| i.len())
.unwrap_or(0);
let link = self
.rt
.block_on(self.client.get_public_link(&uid))
.ok()
.flatten()
.map(public_link_info);
.map(|i| i.len());
let link = self.rt.block_on(self.client.get_public_link(&uid));
if let (Ok(0), Ok(0), Ok(0), Ok(None)) =
(&members, &proton_invites, &external_invites, &link)
{
debug!(%uid, "shared by me: skipping an emptied share");
continue;
}
let members = members.unwrap_or(0);
let proton_invites = proton_invites.unwrap_or(0);
let external_invites = external_invites.unwrap_or(0);
let link = link.ok().flatten().map(public_link_info);
items.push(SharedItem {
uid: uid.to_string(),
is_dir: n.is_folder(),
+1 -1
View File
@@ -6,7 +6,7 @@
//! invisible; in an in-process remount — which is how the daemon recovers from a
//! mount going away underneath it — it leaks a whole generation of threads that
//! keep writing to the database and calling the API on behalf of a mount that no
//! longer exists (bugs.md B44).
//! longer exists (`docs/BUGS.md` B44).
//!
//! The primitive is deliberately small: a flag plus a condvar. Every loop that
//! would otherwise call `thread::sleep` calls [`Shutdown::sleep`] instead, which
File diff suppressed because it is too large. Load diff
+425 -32
View File
@@ -1,4 +1,4 @@
//! An in-memory Drive for the simulation tests (`docs/MILESTONE-3.0.0.md` §8.1).
//! An in-memory Drive for the simulation tests (`docs/TESTING.md`).
//!
//! [`FakeDrive`] is the server: one tree of nodes with uids, revisions and
//! names, and an event log. Each simulated daemon reaches it through its own
@@ -31,9 +31,18 @@
//! - **A full account** ([`FakeDrive::set_quota`]) refusing uploads with
//! `InsufficientQuota`.
//!
//! It also holds the account's computers, each with a root folder outside My
//! files, so a simulated daemon can register its own and sync a local folder.
//!
//! A test can also hold the next read of one node on its way to Drive
//! ([`FakeClient::hold_next_read`]), to make a change while the daemon waits
//! for the answer.
//! for the answer, or on its way back ([`FakeClient::hold_answer_to_next_read`]),
//! so the daemon gets an answer a change made meanwhile has overtaken. It can
//! also have a new file's next reads answered as Drive lists it before its
//! create is committed ([`FakeClient::answer_reads_unrevised`]; B163), and
//! hold a folder's listing on its way back with one of its files listed that
//! way ([`FakeClient::hold_answer_to_next_listing`]), or hold a new file's
//! create on its way to Drive ([`FakeClient::hold_next_create`]).
use std::collections::{BTreeMap, HashMap};
use std::io::{Read, Write};
@@ -48,9 +57,12 @@ use parking_lot::Mutex;
use proton_drive_rs::proton_sdk::account::Quota;
use proton_drive_rs::proton_sdk::api::ResponseCode;
use proton_drive_rs::proton_sdk::error::{ProtonApiError, ProtonError, Result};
use proton_drive_rs::proton_sdk::ids::{DriveEventId, LinkId, NodeUid, VolumeId};
use proton_drive_rs::proton_sdk::ids::{
DeviceUid, DriveEventId, LinkId, NodeUid, ShareId, VolumeId,
};
use proton_drive_rs::{
DriveEvent, DriveEventScopeId, Node, NodeKind, NodeMoveItem, RevisionState, Thumbnail,
Device as Computer, DeviceType, DriveEvent, DriveEventScopeId, Node, NodeKind, NodeMoveItem,
RevisionState, Thumbnail,
};
use sha1::{Digest as _, Sha1};
@@ -105,6 +117,9 @@ struct Server {
block_pattern: Vec<u64>,
/// The account's storage limit in bytes.
max_space: i64,
/// The account's computers, oldest first, as `(name, type, root folder)`.
/// [`Computer`] is not `Clone`, so each answer builds them afresh.
devices: Vec<(String, DeviceType, NodeUid)>,
}
impl Server {
@@ -156,6 +171,20 @@ impl Server {
false
}
/// The `n`th computer of the account, as Drive lists it.
fn computer(&self, n: usize) -> Computer {
let (name, device_type, root) = &self.devices[n];
Computer {
uid: DeviceUid::new(format!("device{}", n + 1)),
device_type: *device_type,
name: Ok(name.clone()),
root_folder_uid: root.clone(),
creation_time: self.current(root).map_or(0, |remote| remote.created),
last_sync_time: None,
share_id: ShareId::new(format!("device-share{}", n + 1)),
}
}
fn mint_uid(&mut self) -> NodeUid {
self.next_link += 1;
uid(&format!("L{}", self.next_link))
@@ -299,6 +328,7 @@ impl FakeDrive {
events: Vec::new(),
block_pattern: vec![pdfs_core::cache::BLOCK_SIZE],
max_space: 1 << 40,
devices: Vec::new(),
};
let now = now_secs();
server.nodes.insert(
@@ -330,6 +360,20 @@ impl FakeDrive {
online: AtomicBool::new(true),
requests: AtomicU64::new(0),
hold: Mutex::new(None),
hold_answer: Mutex::new(None),
hold_trash: Mutex::new(None),
hold_download: Mutex::new(None),
hold_any_landing: AtomicBool::new(false),
hold_landing: Mutex::new(None),
drop_after_create: AtomicBool::new(false),
lose_create_reply: AtomicBool::new(false),
hold_create_reply: Mutex::new(None),
hold_create: Mutex::new(None),
lose_revision_reply: AtomicBool::new(false),
lose_revision_read_back: AtomicBool::new(false),
lose_read: Mutex::new(None),
unrevised_reads: Mutex::new(None),
hold_listing: Mutex::new(None),
}))
}
@@ -390,6 +434,23 @@ impl FakeDrive {
tree
}
/// Every name a node has had in the root, trashed or not, as far back as
/// Drive keeps its versions.
pub(crate) fn names_ever_in_root(&self) -> Vec<String> {
let server = self.server.lock();
let mut names: Vec<String> = server
.nodes
.values()
.flatten()
.filter_map(|(_, remote)| remote.as_ref())
.filter(|remote| remote.parent.as_ref() == Some(&server.root))
.map(|remote| remote.name.clone())
.collect();
names.sort();
names.dedup();
names
}
/// The uid at `path` below the root, as Drive has it now.
pub(crate) fn lookup(&self, path: &str) -> Option<NodeUid> {
let server = self.server.lock();
@@ -577,6 +638,37 @@ struct ClientInner {
requests: AtomicU64,
/// The read [`FakeClient::hold_next_read`] is waiting for.
hold: Mutex<Option<Arc<HoldState>>>,
/// The read [`FakeClient::hold_answer_to_next_read`] is waiting for.
hold_answer: Mutex<Option<Arc<HoldState>>>,
/// The trash [`FakeClient::hold_next_trash`] is waiting for.
hold_trash: Mutex<Option<Arc<HoldState>>>,
/// The download [`FakeClient::hold_next_download`] is waiting for.
hold_download: Mutex<Option<Arc<HoldState>>>,
/// The landing [`FakeClient::hold_next_landing`] is waiting for.
hold_landing: Mutex<Option<Arc<HoldState>>>,
/// Set by [`FakeClient::hold_next_landing_of_any`]: hold whichever node
/// lands next.
hold_any_landing: AtomicBool,
/// Set by [`FakeClient::drop_link_after_next_create`].
drop_after_create: AtomicBool,
/// Set by [`FakeClient::lose_reply_to_next_create`].
lose_create_reply: AtomicBool,
/// The create [`FakeClient::hold_reply_to_next_create`] is waiting for.
hold_create_reply: Mutex<Option<Arc<HoldState>>>,
/// The create [`FakeClient::hold_next_create`] is waiting for.
hold_create: Mutex<Option<Arc<HoldState>>>,
/// Set by [`FakeClient::lose_reply_to_next_revision`].
lose_revision_reply: AtomicBool,
/// Set by [`FakeClient::lose_read_back_of_next_revision`].
lose_revision_read_back: AtomicBool,
/// The file whose next read goes unanswered, once its revision is made.
lose_read: Mutex<Option<NodeUid>>,
/// Set by [`FakeClient::answer_reads_unrevised`]: the file and how many
/// more of its reads are answered without an active revision.
unrevised_reads: Mutex<Option<(NodeUid, usize)>>,
/// The listing [`FakeClient::hold_answer_to_next_listing`] is waiting
/// for, and the file it lists without an active revision.
hold_listing: Mutex<Option<(Arc<HoldState>, NodeUid)>>,
}
struct HoldState {
@@ -585,8 +677,37 @@ struct HoldState {
released: AtomicBool,
}
/// A read held on its way to Drive, from [`FakeClient::hold_next_read`]. It
/// goes on when this is dropped.
/// Wait while `state` holds a call.
/// `node` as Drive lists a file whose create it has not committed yet: no
/// active revision, no size.
fn unrevise(node: &mut Node) {
if let NodeKind::File {
total_size_on_storage,
active_revision_state,
active_revision_id,
claimed_size,
content_sha1,
..
} = &mut node.kind
{
*total_size_on_storage = 0;
*active_revision_state = None;
*active_revision_id = None;
*claimed_size = None;
*content_sha1 = None;
}
}
async fn wait_until_released(state: &HoldState) {
state.reached.store(true, Ordering::SeqCst);
while !state.released.load(Ordering::SeqCst) {
tokio::time::sleep(Duration::from_millis(5)).await;
}
}
/// A call held on its way to or from Drive, from [`FakeClient::hold_next_read`]
/// or [`FakeClient::hold_reply_to_next_create`]. It goes on when this is
/// dropped.
pub(crate) struct Held(Arc<HoldState>);
impl Held {
@@ -616,6 +737,18 @@ impl FakeClient {
*self.0.faults.lock() = faults;
}
/// Take the link down as soon as the next file create has been answered,
/// so whatever the daemon asks next goes unanswered.
pub(crate) fn drop_link_after_next_create(&self) {
self.0.drop_after_create.store(true, Ordering::SeqCst);
}
/// Make the next file create on Drive and lose its answer, as a link that
/// drops it on the way back does.
pub(crate) fn lose_reply_to_next_create(&self) {
self.0.lose_create_reply.store(true, Ordering::SeqCst);
}
/// Hold the next read of `uid` alone, before it reaches Drive, until the
/// answer is dropped. Only one read is held at a time.
pub(crate) fn hold_next_read(&self, uid: &NodeUid) -> Held {
@@ -628,20 +761,135 @@ impl FakeClient {
Held(state)
}
/// Wait here if `uids` is the read a [`Held`] is for.
async fn wait_if_held(&self, uids: &[NodeUid]) {
/// Answer the next read of `uid` alone with what Drive holds now, then
/// hold the answer until it is dropped, as a slow link does. Only one
/// answer is held at a time.
pub(crate) fn hold_answer_to_next_read(&self, uid: &NodeUid) -> Held {
let state = Arc::new(HoldState {
uid: uid.clone(),
reached: AtomicBool::new(false),
released: AtomicBool::new(false),
});
*self.0.hold_answer.lock() = Some(state.clone());
Held(state)
}
/// Hold the next trash of `uid` alone, before it reaches Drive, until the
/// answer is dropped.
pub(crate) fn hold_next_trash(&self, uid: &NodeUid) -> Held {
let state = Arc::new(HoldState {
uid: uid.clone(),
reached: AtomicBool::new(false),
released: AtomicBool::new(false),
});
*self.0.hold_trash.lock() = Some(state.clone());
Held(state)
}
/// Hold the next download of `uid`'s content once Drive has answered it,
/// before the bytes reach the daemon, until the answer is dropped.
pub(crate) fn hold_next_download(&self, uid: &NodeUid) -> Held {
let state = Arc::new(HoldState {
uid: uid.clone(),
reached: AtomicBool::new(false),
released: AtomicBool::new(false),
});
*self.0.hold_download.lock() = Some(state.clone());
Held(state)
}
/// Hold the drain once the next upload to `uid` has landed and its blob
/// is let go of, before the tree has the node (`DriveApi::landing`), until
/// the answer is dropped.
pub(crate) fn hold_next_landing(&self, uid: &NodeUid) -> Held {
let state = Arc::new(HoldState {
uid: uid.clone(),
reached: AtomicBool::new(false),
released: AtomicBool::new(false),
});
*self.0.hold_landing.lock() = Some(state.clone());
Held(state)
}
/// [`FakeClient::hold_next_landing`] for whichever node lands next, such
/// as a new file, whose uid Drive has not minted yet.
pub(crate) fn hold_next_landing_of_any(&self) -> Held {
let held = self.hold_next_landing(&uid("*"));
self.0.hold_any_landing.store(true, Ordering::SeqCst);
held
}
/// Make the next file create on Drive, then hold its answer until the
/// answer is dropped, as a slow link does.
pub(crate) fn hold_reply_to_next_create(&self) -> Held {
let state = Arc::new(HoldState {
uid: self.0.drive.root(),
reached: AtomicBool::new(false),
released: AtomicBool::new(false),
});
*self.0.hold_create_reply.lock() = Some(state.clone());
Held(state)
}
/// Hold the next file create before it reaches Drive, until the answer
/// is dropped.
pub(crate) fn hold_next_create(&self) -> Held {
let state = Arc::new(HoldState {
uid: self.0.drive.root(),
reached: AtomicBool::new(false),
released: AtomicBool::new(false),
});
*self.0.hold_create.lock() = Some(state.clone());
Held(state)
}
/// Make the next revision of a file on Drive and lose its answer.
pub(crate) fn lose_reply_to_next_revision(&self) {
self.0.lose_revision_reply.store(true, Ordering::SeqCst);
}
/// Make the next revision of a file on Drive and answer it, then lose the
/// answer to the next read of that file alone, as the read-back after an
/// upload meets on a link that drops it.
pub(crate) fn lose_read_back_of_next_revision(&self) {
self.0.lose_revision_read_back.store(true, Ordering::SeqCst);
}
/// Answer the next `reads` reads of file `uid` alone that find it as Drive
/// lists a file whose create it has not committed yet: no active revision,
/// no size.
pub(crate) fn answer_reads_unrevised(&self, uid: &NodeUid, reads: usize) {
*self.0.unrevised_reads.lock() = Some((uid.clone(), reads));
}
/// Answer the next listing of `folder`'s nodes with what Drive holds now,
/// `unrevised` in it as Drive lists a file whose create it has not
/// committed yet, then hold the answer until it is dropped.
pub(crate) fn hold_answer_to_next_listing(
&self,
folder: &NodeUid,
unrevised: &NodeUid,
) -> Held {
let state = Arc::new(HoldState {
uid: folder.clone(),
reached: AtomicBool::new(false),
released: AtomicBool::new(false),
});
*self.0.hold_listing.lock() = Some((state.clone(), unrevised.clone()));
Held(state)
}
/// Wait here if `uids` is the read a [`Held`] in `slot` is for.
async fn wait_if_held(&self, slot: &Mutex<Option<Arc<HoldState>>>, uids: &[NodeUid]) {
let state = {
let mut hold = self.0.hold.lock();
let mut hold = slot.lock();
match hold.as_ref() {
Some(state) if uids == std::slice::from_ref(&state.uid) => hold.take(),
_ => None,
}
};
let Some(state) = state else { return };
state.reached.store(true, Ordering::SeqCst);
while !state.released.load(Ordering::SeqCst) {
tokio::time::sleep(Duration::from_millis(5)).await;
}
wait_until_released(&state).await;
}
/// Requests made through this client so far.
@@ -868,7 +1116,17 @@ impl DriveApi for FakeClient {
}
async fn enumerate_nodes(&self, uids: &[NodeUid]) -> Result<Vec<Node>> {
self.wait_if_held(uids).await;
self.wait_if_held(&self.0.hold, uids).await;
{
let mut lose = self.0.lose_read.lock();
if lose
.as_ref()
.is_some_and(|uid| uids == std::slice::from_ref(uid))
{
*lose = None;
return Err(no_answer());
}
}
let lag = self.lag();
let nodes: Vec<Node> = self
.request(false, |server| {
@@ -879,20 +1137,49 @@ impl DriveApi for FakeClient {
.collect())
})
.await?;
self.wait_if_held(&self.0.hold_answer, uids).await;
let mut nodes = nodes;
{
let mut unrevised = self.0.unrevised_reads.lock();
if let Some((uid, left)) = unrevised.as_mut()
&& uids == std::slice::from_ref(uid)
&& !nodes.is_empty()
{
*left -= 1;
if *left == 0 {
*unrevised = None;
}
nodes.iter_mut().for_each(unrevise);
}
}
self.cache(&nodes);
Ok(nodes)
}
async fn enumerate_nodes_light(&self, uids: &[NodeUid]) -> Result<Vec<Node>> {
let lag = self.lag();
self.request(false, |server| {
let now = Instant::now();
Ok(uids
let mut nodes: Vec<Node> = self
.request(false, |server| {
let now = Instant::now();
Ok(uids
.iter()
.filter_map(|uid| Some(to_node(uid, &server.visible(uid, lag, now)?, true)))
.collect())
})
.await?;
let held = self.0.hold_listing.lock().take_if(|(state, _)| {
nodes
.iter()
.filter_map(|uid| Some(to_node(uid, &server.visible(uid, lag, now)?, true)))
.collect())
})
.await
.any(|node| node.parent_uid.as_ref() == Some(&state.uid))
});
if let Some((state, unrevised)) = held {
nodes
.iter_mut()
.filter(|node| node.uid == unrevised)
.for_each(unrevise);
wait_until_released(&state).await;
}
Ok(nodes)
}
async fn enumerate_folder_children_node_uids(
@@ -1038,16 +1325,34 @@ impl DriveApi for FakeClient {
_aead: bool,
) -> Result<NodeUid> {
let content = read_all(reader)?;
self.request(true, |server| {
server.new_file(
parent_uid,
name,
media_type,
content,
last_modification_time,
)
})
.await
let held = self.0.hold_create.lock().take();
if let Some(state) = held {
wait_until_released(&state).await;
}
let made = self
.request(true, |server| {
server.new_file(
parent_uid,
name,
media_type,
content,
last_modification_time,
)
})
.await;
if made.is_ok() && self.0.drop_after_create.swap(false, Ordering::SeqCst) {
self.set_online(false);
}
let held = self.0.hold_create_reply.lock().take();
if made.is_ok()
&& let Some(state) = held
{
wait_until_released(&state).await;
}
if made.is_ok() && self.0.lose_create_reply.swap(false, Ordering::SeqCst) {
return Err(no_answer());
}
made
}
async fn upload_file_replacing_draft_from_dyn(
@@ -1103,7 +1408,14 @@ impl DriveApi for FakeClient {
server.put(file_uid, Some(remote));
Ok(())
})
.await
.await?;
if self.0.lose_revision_reply.swap(false, Ordering::SeqCst) {
return Err(no_answer());
}
if self.0.lose_revision_read_back.swap(false, Ordering::SeqCst) {
*self.0.lose_read.lock() = Some(file_uid.clone());
}
Ok(())
}
async fn open_revision(&self, uid: &NodeUid) -> Result<Arc<dyn RevisionRead>> {
@@ -1135,6 +1447,8 @@ impl DriveApi for FakeClient {
.ok_or_else(|| api(ResponseCode::DoesNotExist, "file not found"))
})
.await?;
self.wait_if_held(&self.0.hold_download, std::slice::from_ref(uid))
.await;
output.write_all(&content).map_err(io_error)
}
@@ -1204,6 +1518,7 @@ impl DriveApi for FakeClient {
}
async fn trash_nodes(&self, uids: &[NodeUid]) -> Result<NodeOutcomes> {
self.wait_if_held(&self.0.hold_trash, uids).await;
self.batch(uids, trash_one).await
}
@@ -1219,6 +1534,25 @@ impl DriveApi for FakeClient {
outcome_stream(self.batch(uids, delete_one))
}
fn landing(&self, uid: &NodeUid) {
let state = {
let mut hold = self.0.hold_landing.lock();
let any = self.0.hold_any_landing.load(Ordering::SeqCst);
match hold.as_ref() {
Some(state) if any || state.uid == *uid => {
self.0.hold_any_landing.store(false, Ordering::SeqCst);
hold.take()
}
_ => None,
}
};
let Some(state) = state else { return };
state.reached.store(true, Ordering::SeqCst);
while !state.released.load(Ordering::SeqCst) {
std::thread::sleep(Duration::from_millis(5));
}
}
async fn quota(&self) -> Result<Quota> {
self.request(false, |server| {
Ok(Quota {
@@ -1229,6 +1563,42 @@ impl DriveApi for FakeClient {
.await
}
async fn enumerate_devices(&self) -> Result<Vec<Computer>> {
self.request(false, |server| {
Ok((0..server.devices.len())
.map(|n| server.computer(n))
.collect())
})
.await
}
async fn create_device(&self, name: &str, device_type: DeviceType) -> Result<Computer> {
self.request(true, |server| {
// A device root has no parent: like on Drive, it is not in My files.
// It is listed at once, as the My files root is: the listing lag is
// about children appearing, and the daemon lists the new root next.
let root = server.mint_uid();
let now = now_secs();
server.nodes.insert(
root.clone(),
vec![(
Instant::now() - HISTORY,
Some(Remote {
parent: None,
name: name.to_owned(),
trashed: false,
created: now,
modified: now,
file: None,
}),
)],
);
server.devices.push((name.to_owned(), device_type, root));
Ok(server.computer(server.devices.len() - 1))
})
.await
}
async fn enumerate_trash_node_uids(&self) -> Result<Vec<NodeUid>> {
self.request(false, |server| {
Ok(server
@@ -1554,6 +1924,29 @@ mod tests {
});
}
#[test]
fn a_held_answer_is_what_drive_held_when_read() {
rt().block_on(async {
let drive = FakeDrive::new();
let client = drive.client(1, Faults::none());
let root = drive.root();
let file = client.upload_file(&root, "a", "", b"1").await.unwrap();
let held = client.hold_answer_to_next_read(&file);
let read = tokio::spawn({
let client = client.clone();
let file = file.clone();
async move { client.enumerate_nodes(std::slice::from_ref(&file)).await }
});
tokio::time::sleep(Duration::from_millis(50)).await;
assert!(held.reached());
assert!(!read.is_finished());
drive.device().trash("a");
drop(held);
let nodes = read.await.unwrap().unwrap();
assert!(!nodes[0].trashed);
});
}
#[test]
fn a_rename_lags_only_by_the_update_lag() {
rt().block_on(async {
+1 -1
View File
@@ -1,4 +1,4 @@
//! The simulation tests of `docs/MILESTONE-3.0.0.md` §8: a fake Drive with
//! The simulation tests of `docs/TESTING.md`: a fake Drive with
//! the faults the real one has, and the tools to drive daemons against it.
pub(crate) mod daemon;
+2 -2
View File
@@ -1,5 +1,5 @@
//! The reference file system of the simulation (`docs/MILESTONE-3.0.0.md`
//! §8.2, property 4): what each syscall a simulated client makes should
//! The reference file system of the simulation (`docs/TESTING.md`,
//! "Simulation runs", property 1): what each syscall a simulated client makes should
//! answer, and the tree it should leave, on a POSIX file system.
//!
//! A [`Model`] covers one client's own folder; paths are relative to it. The
+49 -403
View File
@@ -1,4 +1,4 @@
//! The simulation runs of `docs/MILESTONE-3.0.0.md` §8.2.
//! The simulation runs described in `docs/TESTING.md`.
//!
//! A run starts one or more daemons on a shared [`FakeDrive`], each owning a
//! folder `c<n>` that only it writes, so there is one writer per file. A seed
@@ -22,14 +22,9 @@
//! one wrote down, but in-flight work is let finish rather than cut. A kill
//! at an arbitrary point needs the daemon in a process of its own.
//!
//! A call that answers otherwise than the model says fails the run, unless it
//! is one of the [`KNOWN`] open bugs. Those are counted instead and the model
//! takes the call back, or the seed ends there when the bug leaves the mount
//! where the model cannot follow, so the runs stay a regression net until
//! they are fixed. A file gone, or a conflict copy the drain made of one of
//! the client's own, is counted too when a known bug explains it
//! (`Run::known_damage`), and so is a hang the stacks show is one of the
//! [`KNOWN_HANGS`]. `PDFS_SIM_KNOWN=fail` fails on all of them.
//! Any of these fails the seed. No open bug is tolerated: the matchers that
//! counted the ones still open instead (B125 to B134) went once a run on
//! 812eb3d passed every profile without them.
//!
//! The runs are ignored by default because they mount FUSE; run them with
//! `cargo test -p pdfs-fuse --lib sim::run -- --ignored --test-threads=1`.
@@ -42,105 +37,11 @@ use std::io::{self, Read, Seek, SeekFrom, Write};
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant};
use std::collections::BTreeMap;
use proton_drive_rs::proton_sdk::ids::NodeUid;
use super::daemon::{Daemon, scratch, take_logged, wait_until};
use super::fake_drive::{Entry, FakeDrive, Faults};
use super::model::{FsOp, Model, Tree, data, join, name_of, parent_of};
use super::model::{FsOp, Model, Tree, data, join};
use super::rng::Rng;
use super::watchdog::{Hung, Watchdog};
/// An open bug in `docs/BUGS.md` the runs tolerate.
struct Known {
bug: &'static str,
/// Whether a call that answered `errno` against the model's word, with
/// lines `logged` at `INFO` and above meanwhile and `before` it, is this
/// bug. `model` is from before the call.
is: fn(model: &Model, op: &FsOp, errno: i32, logged: &[String], before: &[String]) -> bool,
/// Whether it also left the mount where the model cannot follow, so the
/// seed ends there.
ends_seed: fn(model: &Model, op: &FsOp) -> bool,
}
const KNOWN: &[Known] = &[
// A replacing rename trashes what it replaces first, so Drive refuses the
// name only when it holds a node the mount has forgotten: the create of
// the replaced file landed after its op was dropped.
Known {
bug: "B129",
is: |model, op, errno, logged, _| {
let FsOp::Rename { to, .. } = op else {
return false;
};
errno == libc::EIO
&& model.tree().contains_key(to)
&& logged
.iter()
.any(|line| line.starts_with("rename failed") && line.contains("AlreadyExists"))
},
// The model made the rename, and Drive holds a node the mount does
// not know.
ends_seed: |_, _| true,
},
// A folder made offline is listed on Drive once its listing has been
// invalidated, and Drive does not know it, or does not list it yet. Only
// the client writes its folder, so a folder of its own is never really
// gone.
Known {
bug: "B125",
is: |_, _, errno, logged, _| {
matches!(errno, libc::EIO | libc::ENOENT)
&& logged.iter().any(|line| {
line.starts_with("enumerate folder children failed")
&& line.contains("DoesNotExist")
})
},
// The folder keeps failing to list, and what is in it can come back
// as gone, so the mount no longer follows the model.
ends_seed: |_, _| true,
},
// A restart forgets the listings served offline, so the first lookup
// lists a folder on Drive, which does not hold what is still queued for
// it.
Known {
bug: "B132",
is: |_, _, errno, _, before| {
errno == libc::ENOENT
&& before
.iter()
.any(|line| line.starts_with("lost the connection to Proton"))
&& before
.iter()
.any(|line| line.starts_with("restored pending ops"))
},
// The listing stays without it until something relists the folder.
ends_seed: |_, _| true,
},
];
/// An open bug in `docs/BUGS.md` that hangs a call, which the runs tolerate.
/// The seed ends there, since the watchdog aborted the mounts.
struct KnownHang {
bug: &'static str,
/// Whether the stacks the watchdog kept show this bug.
is: fn(stacks: &str) -> bool,
}
const KNOWN_HANGS: &[KnownHang] = &[
// A runtime worker invalidating a file's pages waits for the page lock
// of a read, whose fetch waits for a timer that no worker drives.
KnownHang {
bug: "B130",
is: |stacks| {
stacks.split("\nthread ").any(|thread| {
thread.contains("\"tokio-rt-worker\", state D")
&& thread.contains("NotifyBatch>::flush")
})
},
},
];
use super::watchdog::Watchdog;
/// What a run looks like.
pub(crate) struct Profile {
@@ -178,16 +79,6 @@ const LOG_TAIL: usize = 40;
/// for the last run and the mount.
const STALL: Duration = Duration::from_secs(120);
/// What a known bug left on Drive in place of a file, and how the model
/// takes it in.
struct KnownDamage {
bug: &'static str,
path: String,
/// The conflict copy it made, if it made one.
copy: Option<String>,
change: FsOp,
}
struct Client {
daemon: Option<Daemon>,
model: Model,
@@ -213,25 +104,14 @@ struct Run {
clients: Vec<Client>,
log: Vec<String>,
started: Instant,
/// How often each [`KNOWN`] bug turned up.
known: BTreeMap<&'static str, usize>,
/// The known bug that ended the seed early, if one did.
ended_by: Option<&'static str>,
/// What the daemons logged at `INFO` and above so far.
logged: Vec<String>,
/// Files deleted while the drain held a write to them.
held: Vec<NodeUid>,
/// Those of them made again before it went on.
remade: Vec<NodeUid>,
watchdog: Watchdog,
}
/// Run `profile` on `seed`. It answers how often each known bug turned up,
/// or a report of what went wrong.
pub(crate) fn run(
seed: u64,
profile: &'static Profile,
) -> Result<BTreeMap<&'static str, usize>, String> {
/// Run `profile` on `seed`. It answers a report of what went wrong, if
/// anything did.
pub(crate) fn run(seed: u64, profile: &'static Profile) -> Result<(), String> {
let dir = scratch(&format!("{}-{seed}", profile.name));
let mut rng = Rng::new(seed);
let drive = FakeDrive::new();
@@ -242,18 +122,14 @@ pub(crate) fn run(
clients: Vec::new(),
log: Vec::new(),
started: Instant::now(),
known: BTreeMap::new(),
ended_by: None,
logged: Vec::new(),
held: Vec::new(),
remade: Vec::new(),
watchdog: Watchdog::new(&dir),
};
let outcome = run.execute(&dir, &mut rng);
run.watchdog.stop();
// A hang is what failed, whatever the aborted call answered.
let outcome = match run.watchdog.fired() {
Some(hung) => run.known_hang(&hung),
Some(hung) => Err(format!("liveness: {}", hung.what)),
None => outcome,
};
for client in &mut run.clients {
@@ -264,27 +140,13 @@ pub(crate) fn run(
match outcome {
Ok(()) => {
let _ = std::fs::remove_dir_all(&dir);
Ok(run.known)
Ok(())
}
Err(failure) => Err(run.report(&failure, &dir)),
}
}
impl Run {
/// Count `hung` if it is one of the [`KNOWN_HANGS`], or fail on it.
fn known_hang(&mut self, hung: &Hung) -> Result<(), String> {
let known = KNOWN_HANGS.iter().find(|known| (known.is)(&hung.stacks));
match known {
Some(known) if tolerate_known() => {
*self.known.entry(known.bug).or_default() += 1;
self.note(format!("known bug {}; the seed ends here", known.bug));
self.ended_by = Some(known.bug);
Ok(())
}
_ => Err(format!("liveness: {}", hung.what)),
}
}
fn execute(&mut self, dir: &Path, rng: &mut Rng) -> Result<(), String> {
for index in 0..self.profile.clients {
let client = self.drive.client(rng.next_u64(), (self.profile.faults)());
@@ -302,9 +164,6 @@ impl Run {
for step in 0..self.profile.steps {
let _watching = self.watchdog.arm(self.step_limit(), format!("step {step}"));
self.step(rng)?;
if self.ended_by.is_some() {
return Ok(());
}
}
let _watching = self.watchdog.arm(
self.profile.drain_budget + STALL,
@@ -363,7 +222,11 @@ impl Run {
fn settle(&mut self, budget: Duration) -> Result<(), String> {
self.note("settle".into());
let start = Instant::now();
if !wait_until(budget, || self.landed()) {
// What the drain logs meanwhile is what explains a known bug's damage.
if !wait_until(budget, || {
self.logged.extend(take_logged());
self.landed()
}) {
return Err(format!(
"liveness: the queues did not drain within {budget:?}\n{}",
self.difference()
@@ -409,18 +272,11 @@ impl Run {
data: bytes,
},
)?;
if self.ended_by.is_some() {
return Ok(());
}
if !wait_until(PICKUP_BUDGET, || held.reached()) {
self.note(format!("client {index}: nothing read {path} back"));
return Ok(());
}
self.syscall(index, FsOp::Unlink { path: path.clone() })?;
if self.ended_by.is_some() {
return Ok(());
}
self.held.push(uid.clone());
if rng.chance(0.5) {
self.syscall(
index,
@@ -429,7 +285,6 @@ impl Run {
data: data(rng),
},
)?;
self.remade.push(uid);
}
let faults = (self.profile.faults)();
let latency = Duration::from_millis(faults.latency_ms.1);
@@ -444,11 +299,9 @@ impl Run {
fn syscall(&mut self, index: usize, op: FsOp) -> Result<(), String> {
let client = &mut self.clients[index];
let root = client.root(index);
let model = client.model.clone();
let before = client.model.content(target(&op)).cloned();
let expected = client.model.apply(&op);
self.logged.extend(take_logged());
let logged_before = self.logged.len();
let start = Instant::now();
let actual = perform(&root, &op, before.as_deref());
let took = start.elapsed();
@@ -458,25 +311,7 @@ impl Run {
outcome(&actual)
));
let actual = actual?;
let logged = take_logged();
self.logged.extend(logged.iter().cloned());
if actual != expected
&& let Err(errno) = actual
&& tolerate_known()
&& let Some(known) = KNOWN.iter().find(|known| {
(known.is)(&model, &op, errno, &logged, &self.logged[..logged_before])
})
{
*self.known.entry(known.bug).or_default() += 1;
if (known.ends_seed)(&model, &op) {
self.note(format!("known bug {}; the seed ends here", known.bug));
self.ended_by = Some(known.bug);
} else {
self.note(format!("known bug {}; taken back", known.bug));
}
self.clients[index].model = model;
return Ok(());
}
self.logged.extend(take_logged());
if actual != expected {
return Err(format!(
"posix: client {index}: {} answered {}, expected {}",
@@ -513,61 +348,15 @@ impl Run {
}
// Each client's folder is its own, so a file of it gone from Drive was
// deleted by the client.
let own_deletes: Vec<String> = self
.logged
.iter()
.filter(|line| is_own_delete_kept(line))
.cloned()
.collect();
for line in own_deletes {
let bug = if self
.remade
.iter()
.any(|uid| line.contains(&format!(" uid={uid} ")))
{
// Made again while the drain held its write: the new file's
// write took the freed op id.
"B133"
} else if !self
.held
.iter()
.any(|uid| line.contains(&format!(" uid={uid} ")))
&& !line.contains(" name=\"recovered-")
{
// The tree still placed the file, so the drain looked before
// the unlink had dropped its write.
"B134"
} else {
return Err(format!(
"no false conflicts: the drain took a client's own delete for another device's: {line}"
));
};
if !tolerate_known() {
return Err(format!(
"no false conflicts: the drain took a client's own delete for another device's: {line}"
));
}
*self.known.entry(bug).or_default() += 1;
self.note(format!("known bug {bug}: {line}"));
if let Some(line) = self.logged.iter().find(|line| is_own_delete_kept(line)) {
return Err(format!(
"no false conflicts: the drain took a client's own delete for another device's: {line}"
));
}
let drive = self.drive_tree();
let mut explained = Vec::new();
for index in 0..self.clients.len() {
for damage in self.known_damage(index, &drive) {
*self.known.entry(damage.bug).or_default() += 1;
self.note(format!(
"known bug {}: client {index}'s {} landed as {:?}",
damage.bug, damage.path, damage.copy
));
let _ = self.clients[index].model.apply(&damage.change);
if let Some(copy) = damage.copy {
explained.push(join(&owned_folder(index), &copy));
}
}
}
let conflicts: Vec<String> = drive
.into_keys()
.filter(|path| is_conflict_copy(path) && !explained.contains(path))
.filter(|path| is_conflict_copy(path))
.collect();
if !conflicts.is_empty() {
return Err(format!("no false conflicts: {conflicts:?}"));
@@ -589,155 +378,18 @@ impl Run {
}
/// Whether every client's queue is empty and Drive holds what its model
/// says, but for what known bugs explain.
/// says.
fn landed(&self) -> bool {
let drive = self.drive_tree();
self.clients.iter().enumerate().all(|(index, client)| {
let mut model = client.model.clone();
for damage in self.known_damage(index, &drive) {
let _ = model.apply(&damage.change);
}
client
.daemon()
.pending()
.is_ok_and(|items| items.is_empty())
&& subtree(&drive, &owned_folder(index)) == *model.tree()
&& subtree(&drive, &owned_folder(index)) == *client.model.tree()
})
}
/// The files of client `index` that a known bug left otherwise on Drive
/// than the model says. A conflict copy takes Drive holding it with the
/// model's bytes, and the drain saying it made that copy for that name:
///
/// - B126: the file's queued create ran into its name while a queued
/// rename was about to free it, so it is there only as the copy.
/// - B127: the file's queued create landed but its answer was lost, so
/// the retry found it and made the copy next to it.
///
/// A lost file takes the drain saying it adopted a node for that name
/// whose queued trash then landed:
///
/// - B128: the file's queued create ran into the name of the file it
/// replaced and took that over, trash and all.
///
/// A conflict copy of a file the client deleted takes the drain saying it
/// kept that file's write as the copy, because Drive had the file in the
/// trash while the tree still placed it:
///
/// - B134: the unlink trashed the file on Drive before it dropped the
/// write.
fn known_damage(&self, index: usize, drive: &Tree) -> Vec<KnownDamage> {
if !tolerate_known() {
return Vec::new();
}
let drive = subtree(drive, &owned_folder(index));
let model = self.clients[index].model.tree();
let made_copy = |path: &str, copy: &str| {
let wanted = format!("wanted={:?}", name_of(path));
let name = format!("name={:?}", name_of(copy));
self.logged.iter().any(|line| {
line.starts_with("name is taken remotely; creating under a conflict name")
&& line.contains(&wanted)
&& line.contains(&name)
})
};
let adopted_trashed = |path: &str| {
let wanted = format!("wanted={:?}", name_of(path));
self.logged.iter().any(|line| {
line.starts_with("the name is held by our own unanswered create; adopting it")
&& line.contains(&wanted)
&& line.split(' ').any(|field| {
field.strip_prefix("twin=").is_some_and(|twin| {
let uid = format!("uid={twin}");
self.logged.iter().any(|line| {
line.starts_with("pending trash landed")
&& line.split(' ').any(|field| field == uid)
})
})
})
})
};
let kept_deleted = |copy: &str| {
let alt = format!(" alt={:?}", name_of(copy));
self.logged.iter().any(|line| {
line.starts_with("queued write landed as a conflict copy")
&& line.ends_with(&alt)
&& line.split(' ').any(|field| {
field.starts_with("uid=")
&& self.logged.iter().any(|kept| {
is_own_delete_kept(kept)
&& !kept.contains(" name=\"recovered-")
&& kept.split(' ').any(|other| other == field)
})
})
})
};
let mut damage = Vec::new();
for (copy, content) in &drive {
let Some(content) = content else { continue };
if !is_conflict_copy(copy) || model.contains_key(copy) {
continue;
}
if !kept_deleted(copy) {
continue;
}
damage.push(KnownDamage {
bug: "B134",
path: copy.clone(),
copy: Some(copy.clone()),
change: FsOp::Create {
path: copy.clone(),
data: content.clone(),
},
});
}
for (path, content) in model {
let Some(content) = content else { continue };
let copy = drive.iter().find(|(copy, other)| {
is_conflict_copy(copy)
&& !model.contains_key(*copy)
&& parent_of(copy) == parent_of(path)
&& other.as_ref() == Some(content)
&& made_copy(path, copy)
});
let Some((copy, _)) = copy else {
if !drive.contains_key(path) && adopted_trashed(path) {
damage.push(KnownDamage {
bug: "B128",
path: path.clone(),
copy: None,
change: FsOp::Unlink { path: path.clone() },
});
}
continue;
};
let (bug, change) = match drive.get(path) {
None => (
"B126",
FsOp::Rename {
from: path.clone(),
to: copy.clone(),
},
),
Some(Some(landed)) if landed == content => (
"B127",
FsOp::Create {
path: copy.clone(),
data: content.clone(),
},
),
Some(_) => continue,
};
damage.push(KnownDamage {
bug,
path: path.clone(),
copy: Some(copy.clone()),
change,
});
}
damage
}
fn difference(&self) -> String {
let drive = self.drive_tree();
let mut out = String::new();
@@ -890,11 +542,6 @@ fn subtree(tree: &Tree, folder: &str) -> Tree {
.collect()
}
/// Whether a known bug is counted rather than failing the run.
fn tolerate_known() -> bool {
std::env::var("PDFS_SIM_KNOWN").as_deref() != Ok("fail")
}
fn owned_folder(index: usize) -> String {
format!("c{index}")
}
@@ -970,21 +617,10 @@ pub(crate) fn run_seeds(profile: &'static Profile, default_seeds: u64) {
(1..=count).collect()
}
};
let mut failures = Vec::new();
let mut known = BTreeMap::<&str, usize>::new();
for seed in seeds {
match run(seed, profile) {
Ok(hits) => {
for (bug, count) in hits {
*known.entry(bug).or_default() += count;
}
}
Err(failure) => failures.push(failure),
}
}
if !known.is_empty() {
eprintln!("{}: known bugs turned up: {known:?}", profile.name);
}
let failures: Vec<String> = seeds
.into_iter()
.filter_map(|seed| run(seed, profile).err())
.collect();
assert!(failures.is_empty(), "\n{}", failures.join("\n\n"));
}
@@ -1044,6 +680,21 @@ mod tests {
drain_budget: Duration::from_secs(120),
};
/// Wi-Fi's latency, lag and lost replies without the outages: the queue
/// has to drain within a settle, not merely by the end.
static ONE_CLIENT_WIFI: Profile = Profile {
name: "one-client-wifi",
clients: 1,
steps: 80,
faults: Faults::wifi,
link_flaps: false,
restarts: false,
deletes_mid_upload: false,
settle_budget: Some(Duration::from_secs(30)),
syscall_budget: Duration::from_secs(10),
drain_budget: Duration::from_secs(120),
};
static THREE_CLIENTS: Profile = Profile {
name: "three-clients",
clients: 3,
@@ -1081,6 +732,12 @@ mod tests {
run_seeds(&ONE_CLIENT_FLAKY, 4);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse --lib sim::run -- --ignored`"]
fn one_client_on_a_slow_link() {
run_seeds(&ONE_CLIENT_WIFI, 4);
}
#[test]
#[ignore = "mounts FUSE: run with `cargo test -p pdfs-fuse --lib sim::run -- --ignored`"]
fn three_clients_one_writer_each() {
@@ -1106,15 +763,4 @@ mod tests {
&kept.replace("the file was trashed remotely", "revision changed")
));
}
#[test]
fn an_invalidation_stuck_on_a_page_lock_is_b130() {
let b130 = |stacks: &str| KNOWN_HANGS.iter().any(|known| (known.is)(stacks));
let stuck = "thread 1 \"fuser-0\", state S, waiting in fuse_dev_do_read\n\
\nthread 2 \"tokio-rt-worker\", state D, waiting in __folio_lock\n\
\x20 11: <pdfs_fuse::NotifyBatch>::flush\n";
assert!(b130(stuck));
assert!(!b130(&stuck.replace("state D", "state S")));
assert!(!b130(&stuck.replace("NotifyBatch", "Other")));
}
}
+113 -11
View File
@@ -183,7 +183,7 @@ impl WriteHandle {
}
}
/// A released write whose upload has not happened yet (offline.md Phase 3).
/// A released write whose upload has not happened yet.
///
/// The bytes live in the content cache's staging dir and the intent lives in the
/// `pending_op` table; this pairs them in memory so a read can be served without
@@ -212,6 +212,37 @@ pub(crate) fn lid_ino(lid: i64) -> u64 {
lid.unsigned_abs() + crate::ROOT_INO
}
/// The inodes of the stored nodes among `nodes`, from their local ids, for
/// [`State::intern_batch_with_inos`]. Called before the state lock is taken;
/// a node that already has a row keeps its id. Empty when the database fails,
/// which leaves the allocation to the lock holder, as before.
pub(crate) fn inos_for(db: &Db, nodes: &[Node]) -> HashMap<NodeUid, u64> {
let stored: Vec<&Node> = nodes
.iter()
.filter(|node| !crate::r#virtual::is_virtual_uid(&node.uid))
.collect();
if stored.is_empty() {
return HashMap::new();
}
match db.lids_for(&stored) {
Ok(lids) => stored
.iter()
.zip(lids)
.map(|(node, lid)| (node.uid.clone(), lid_ino(lid)))
.collect(),
Err(error) => {
warn!(%error, "no local ids before the listing; allocating them under the lock");
HashMap::new()
}
}
}
/// The local id of the node at inode `ino`, the inverse of [`lid_ino`]:
/// `None` for the root and a fallback inode.
pub(crate) fn ino_lid(ino: u64) -> Option<i64> {
(ino > crate::ROOT_INO && ino < FALLBACK_INO).then(|| (ino - crate::ROOT_INO) as i64)
}
/// Mutable inode bookkeeping, guarded by a mutex because fuser drives the
/// `Filesystem` trait through `&self`.
///
@@ -247,7 +278,7 @@ pub(crate) struct State {
/// two pages shifted every later entry, and the caller silently skipped or
/// repeated whatever crossed the page boundary. Each entry is
/// `(ino, is_dir, name)` — the kernel type is mapped at reply time so this
/// module stays free of `fuser` (bugs.md B43).
/// module stays free of `fuser` (`docs/BUGS.md` B43).
pub(crate) dir_snapshots: HashMap<u64, Arc<DirListing>>,
/// Resident inode attributes whose effective access changed during an
/// intern/root refresh. Core drains this set and notifies the matching
@@ -878,6 +909,11 @@ impl State {
}
}
/// Write `node` through as it is, in order with the tree's other changes.
pub(crate) fn write_through(&mut self, node: Node) {
self.outbox.push(DbWrite::Upsert(vec![node]));
}
pub(crate) fn intern(&mut self, parent: u64, mut node: Node) -> u64 {
self.keep_revision_mtime(&mut node);
self.outbox.push(DbWrite::Upsert(vec![node.clone()]));
@@ -1063,20 +1099,34 @@ impl State {
///
/// This is what keeps `ls` on a large folder quick: one commit for the
/// listing, rather than one autocommit — and one fsync — per child.
pub(crate) fn intern_batch(&mut self, parent: u64, mut nodes: Vec<Node>) -> Vec<u64> {
#[cfg(test)]
pub(crate) fn intern_batch(&mut self, parent: u64, nodes: Vec<Node>) -> Vec<u64> {
self.intern_batch_with_inos(parent, nodes, HashMap::new())
}
/// [`State::intern_batch`], with inodes for new entries that the caller
/// allocated before it took the lock, from [`inos_for`]. Allocating them
/// is a write transaction, and under the lock every inline handler of
/// every mount waited for it (`docs/BUGS.md` B221). A node missing from
/// `inos` still gets one here.
pub(crate) fn intern_batch_with_inos(
&mut self,
parent: u64,
mut nodes: Vec<Node>,
mut inos: HashMap<NodeUid, u64>,
) -> Vec<u64> {
for node in &mut nodes {
self.keep_revision_mtime(node);
}
self.outbox.push(DbWrite::Upsert(nodes.clone()));
let fresh: Vec<&Node> = nodes
.iter()
.filter(|node| !self.by_uid.contains_key(&node.uid))
.filter(|node| !self.by_uid.contains_key(&node.uid) && !inos.contains_key(&node.uid))
.collect();
let mut inos = if fresh.is_empty() {
HashMap::new()
} else {
self.new_inos(&fresh)
};
if !fresh.is_empty() {
let allocated = self.new_inos(&fresh);
inos.extend(allocated);
}
nodes
.into_iter()
.map(|node| {
@@ -1087,6 +1137,13 @@ impl State {
.collect()
}
/// Record `uid`'s listing as complete once the lock is released, after the
/// rows of its children: written the other way round, a crash between the
/// two left a folder marked complete with no children to show.
pub(crate) fn record_listed(&mut self, uid: NodeUid) {
self.outbox.push(DbWrite::SetListed(uid, true));
}
/// Check if a directory inode has any child nodes in memory or in the database.
pub(crate) fn has_children(&self, parent: u64) -> bool {
// A resident listing is authoritative even when empty. Falling through
@@ -1187,7 +1244,7 @@ impl State {
/// re-enumerate, which is the cheaper way to stay honest about what the
/// server did. A queued rename cannot: re-enumerating needs the network, and
/// the server has not been told yet in any case — so this *is* the tree's
/// new truth until the op drains (offline.md Phase 3b).
/// new truth until the op drains.
pub(crate) fn rename_in_place(
&mut self,
ino: u64,
@@ -1306,7 +1363,7 @@ impl State {
/// created offline, nothing at all), so this row is the only record that the
/// file is as long as the caller was told it is. Without it a restart serves
/// the stale size and the file reads as truncated — or empty — while its
/// bytes sit safely in staging (offline.md Phase 3).
/// bytes sit safely in staging.
/// Returns the node to persist rather than queueing it in the outbox, and is
/// the one mutation here whose write-through the caller must do itself:
/// failure has to be reported, not logged. Dropping the row silently means
@@ -1325,6 +1382,13 @@ mod tests {
use proton_drive_rs::ShareMembership;
use proton_drive_rs::proton_sdk::ids::{LinkId, ShareId, ShareMembershipId, VolumeId};
#[test]
fn an_inode_gives_back_the_lid_it_was_made_from() {
assert_eq!(ino_lid(lid_ino(7)), Some(7));
assert_eq!(ino_lid(crate::ROOT_INO), None);
assert_eq!(ino_lid(FALLBACK_INO + 3), None);
}
fn uid(link: &str) -> NodeUid {
NodeUid::new(VolumeId::from("vol"), LinkId::from(link))
}
@@ -1449,6 +1513,44 @@ mod tests {
);
}
/// A listing's inodes are allocated before the lock is taken, and are the
/// ones a node would get under it (`docs/BUGS.md` B221).
#[test]
fn inodes_allocated_before_the_lock_are_the_ones_the_node_keeps() {
let (mut st, _dir) = state();
let db = st.db.clone();
let listing = vec![
node("a", "root", "a.txt", false),
node("b", "root", "b.txt", false),
];
let early = inos_for(&db, &listing);
assert_eq!(early.len(), 2);
let inos = st.intern_batch_with_inos(0, listing, early.clone());
assert_eq!(inos, [early[&uid("a")], early[&uid("b")]]);
st.flush_outbox();
let mut restarted = State::new(db, HashMap::new());
assert_eq!(
restarted.intern(0, node("b", "root", "b.txt", false)),
early[&uid("b")]
);
}
/// The listing is recorded complete with its children's rows, when the
/// lock goes, never ahead of them.
#[test]
fn a_listing_is_recorded_complete_after_its_children() {
let (mut st, _dir) = state();
let root = st.intern(0, node("root", "none", "root", true));
st.flush_outbox();
st.intern_batch(root, vec![node("a", "root", "a.txt", false)]);
st.record_listed(uid("root"));
assert!(st.db.children_if_listed(&uid("root")).unwrap().is_none());
st.flush_outbox();
let kids = st.db.children_if_listed(&uid("root")).unwrap().unwrap();
assert_eq!(kids.len(), 1);
}
/// Audit A5. Moving a file out of a folder leaves that folder's DB row
/// claiming a complete listing, so once the hot cache drops it, remote
/// changes under it stop being seen. `relocate` has to clear both sides.
+39 -5
View File
@@ -37,10 +37,12 @@ const CONFLICT_SWEEP_INTERVAL: Duration = Duration::from_secs(300);
/// fresh mount does at startup (hydrate, recover, initial enumeration).
const CONFLICT_SWEEP_WARMUP: Duration = Duration::from_secs(30);
/// Strip a ` (sync-conflict <stamp>)` (optionally `-<n>`) marker from `name`,
/// returning the original file name it was a copy of. `None` when `name` is not a
/// conflict copy. Inverse of the `conflict_name` / `conflict_path_with_suffix`
/// formats, so the two halves of the product agree on what a conflict is.
/// Strip a ` (sync-conflict <label> <stamp>)` (optionally `-<n>`) marker from
/// `name`, returning the original file name it was a copy of. The label names
/// the computer that made the copy (B251); copies made before it have the bare
/// stamp. `None` when `name` is not a conflict copy. Inverse of the
/// `conflict_name` / `conflict_path_with_suffix` formats, so the two halves of
/// the product agree on what a conflict is.
pub(crate) fn conflict_base_name(name: &str) -> Option<String> {
const MARKER: &str = " (sync-conflict ";
let start = name.find(MARKER)?;
@@ -48,6 +50,20 @@ pub(crate) fn conflict_base_name(name: &str) -> Option<String> {
let close_rel = name[after..].find(')')?;
let close = after + close_rel;
let inner = &name[after..close];
// A label is one token of the characters `conflict_label_of` keeps.
let inner = match inner.rsplit_once(' ') {
Some((label, token)) => {
let label_ok = !label.is_empty()
&& label
.chars()
.all(|c| c.is_ascii_alphanumeric() || matches!(c, '-' | '_' | '.'));
if !label_ok {
return None;
}
token
}
None => inner,
};
// The token is `<digits>` or `<digits>-<digits>`; reject anything else so a
// file that merely happens to contain the phrase is not mistaken for a copy.
let mut parts = inner.splitn(2, '-');
@@ -68,7 +84,7 @@ impl Core {
/// docs for the identical/divergent policy.
pub(crate) fn run_conflict_sweep_loop(&self) {
// Both waits are interruptible, so teardown does not have to sit out a
// five-minute interval to join this thread (bugs.md B44).
// five-minute interval to join this thread (`docs/BUGS.md` B44).
if !self.shutdown.sleep(CONFLICT_SWEEP_WARMUP) {
return;
}
@@ -486,6 +502,24 @@ mod tests {
conflict_base_name("notes (sync-conflict 1700-3).txt").as_deref(),
Some("notes.txt")
);
// Named after the computer that made it (B251).
assert_eq!(
conflict_base_name("notes (sync-conflict laptop 1700).txt").as_deref(),
Some("notes.txt")
);
assert_eq!(
conflict_base_name("notes (sync-conflict my-pc.local 1700-2).txt").as_deref(),
Some("notes.txt")
);
assert_eq!(
conflict_base_name(&crate::conflict_name("notes.txt", 1700)).as_deref(),
Some("notes.txt")
);
assert_eq!(
conflict_base_name("x (sync-conflict two words 1).txt"),
None
);
assert_eq!(conflict_base_name("x (sync-conflict 1).txt"), None);
// Not a conflict copy.
assert_eq!(conflict_base_name("report.xml"), None);
// Looks close but the token is not a stamp.
+1226 -260
View File
File diff suppressed because it is too large. Load diff
+31 -5
View File
@@ -22,7 +22,7 @@ const POLL_INTERVAL: Duration = Duration::from_secs(120);
/// per-item work is a network round-trip, so a folder with thousands of files is
/// unusable done one-at-a-time; this bounds concurrency like the bulk-upload
/// engine does.
pub(super) const SYNC_CONCURRENCY: usize = 8;
pub(crate) const SYNC_CONCURRENCY: usize = 8;
/// A message to the sync engine's control thread.
pub(crate) enum SyncMsg {
@@ -33,7 +33,7 @@ pub(crate) enum SyncMsg {
/// Re-read the folder set and adjust the filesystem watches (after add/remove).
Rewatch,
/// The daemon is tearing down: finish the current pass and end the engine
/// thread, so the mount can join it (bugs.md B44).
/// thread, so the mount can join it (`docs/BUGS.md` B44).
Stop,
}
@@ -57,9 +57,10 @@ pub(crate) fn spawn(core: Core, rx: Receiver<SyncMsg>) -> Option<std::thread::Jo
fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
// Paths the watcher currently covers, mapped to their folder id, so an event
// path can be resolved back to the folder it belongs to.
let watched: Mutex<Vec<(PathBuf, i64)>> = Mutex::new(Vec::new());
let watched: Arc<Mutex<Vec<(PathBuf, i64)>>> = Arc::new(Mutex::new(Vec::new()));
let tx_events = core.sync_tx.clone();
let watched_events = watched.clone();
let mut watcher =
match notify::recommended_watcher(move |res: notify::Result<notify::Event>| {
let Ok(event) = res else { return };
@@ -67,7 +68,8 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
if matches!(event.kind, notify::EventKind::Access(_)) {
return;
}
let _ = tx_events.send(SyncMsg::ReconcileAll);
let msg = event_target(&watched_events.lock(), &event.paths);
let _ = tx_events.send(msg);
}) {
Ok(w) => w,
Err(e) => {
@@ -256,8 +258,31 @@ fn reconcile_all(core: &Core) {
}
}
/// The reconcile a watcher event asks for: the one mirror folder all its paths
/// lie in, so an edit in one folder does not walk every other one too. An event
/// that spans folders, or that no watched folder claims (a watch list a moment
/// out of date), falls back to every folder.
pub(super) fn event_target(watched: &[(PathBuf, i64)], paths: &[PathBuf]) -> SyncMsg {
let mut ids = paths.iter().map(|path| {
watched
.iter()
.filter(|(root, _)| path.starts_with(root))
// The deepest root, should one mirror folder sit inside another.
.max_by_key(|(root, _)| root.components().count())
.map(|(_, id)| *id)
});
match ids.next() {
Some(Some(first)) if ids.all(|id| id == Some(first)) => SyncMsg::Reconcile(first),
_ => SyncMsg::ReconcileAll,
}
}
/// Bring the filesystem watches in line with the current mirror-folder set:
/// watch newly-added folders, drop removed ones.
///
/// The watch calls run on a copy, outside the lock: the watcher's event
/// callback takes the same lock, and a backend that hands a watch request to
/// its event thread and waits for the answer would deadlock against it.
fn rewatch(
core: &Core,
watcher: &mut notify::RecommendedWatcher,
@@ -274,7 +299,7 @@ fn rewatch(
return;
}
};
let mut have = watched.lock();
let mut have = watched.lock().clone();
// Drop watches no longer wanted.
have.retain(|(path, _)| {
if want.iter().any(|(p, _)| p == path) {
@@ -294,6 +319,7 @@ fn rewatch(
Err(e) => warn!(path = %path.display(), error = %e, "sync: watch failed"),
}
}
*watched.lock() = have;
}
// ---- reconcile ------------------------------------------------------------
+81 -12
View File
@@ -20,7 +20,7 @@ impl LocalSig {
/// Nanoseconds decide it when both sides have them; otherwise the comparison
/// falls back to whole seconds, which is what a baseline row written before
/// schema 28 — or a filesystem that reports no sub-second time — can support
/// (bugs.md B25).
/// (`docs/BUGS.md` B25).
pub(super) fn same_content(&self, other: &Self) -> bool {
if self.size != other.size {
return false;
@@ -83,6 +83,46 @@ pub(super) struct RemoteItem {
pub(super) is_dir: bool,
pub(super) mtime: i64,
pub(super) size: i64,
/// The active revision's id. The light walk carries it for every file, so
/// it is the signal that tells another client's new revision from a link
/// merely re-stamped.
pub(super) revision: Option<String>,
/// The active revision's content SHA-1, read only for files the walk
/// enumerated in full (no baseline, or a moved mtime). `None` is never
/// proof of anything.
pub(super) sha1: Option<String>,
}
/// What a pass saw of a remote file: the signature its baseline records once
/// the file is synced, and what an upload checks Drive still holds before it
/// supersedes it.
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) struct RemoteSeen {
pub(super) mtime: i64,
pub(super) size: i64,
pub(super) revision: Option<String>,
}
impl RemoteSeen {
/// Whether `node` is still the revision this was taken from. The revision
/// id decides it when both sides have one; otherwise the modification time
/// and size, which is all a baseline row from before schema 44 holds.
pub(super) fn still_current(&self, node: &Node) -> bool {
match (&self.revision, crate::node_revision_id(node)) {
(Some(seen), Some(now)) => *seen == now,
_ => (self.mtime, self.size) == remote_file_sig(node),
}
}
}
impl From<&RemoteItem> for RemoteSeen {
fn from(item: &RemoteItem) -> Self {
Self {
mtime: item.mtime,
size: item.size,
revision: item.revision.clone(),
}
}
}
/// The result of a reconcile pass: what it moved, how many paths were kept as
@@ -98,6 +138,11 @@ pub(super) struct Outcome {
pub(super) errors: usize,
/// Files skipped because another process held them open for writing.
pub(super) deferred: usize,
/// Paths the local scan could not read, left out of the pass. Not part of
/// [`is_empty`](Self::is_empty) or the summary: an unreadable folder stays
/// unreadable from one poll to the next, so the pass reports it once, when
/// the set changes, rather than on every poll.
pub(super) unscanned: Vec<String>,
}
impl Outcome {
@@ -108,7 +153,7 @@ impl Outcome {
Applied::Uploaded => self.uploaded += 1,
Applied::Downloaded => self.downloaded += 1,
Applied::Conflict => self.conflicts += 1,
Applied::Vanished => {}
Applied::Vanished | Applied::Adopted => {}
}
}
@@ -152,29 +197,45 @@ pub(super) enum Pending {
CreateDir { rel: String, parent: NodeUid },
/// Upload a brand-new local file into `parent`.
UploadNew { rel: String, parent: NodeUid },
/// Upload a changed local file as a new revision of `uid`.
UploadRevision { rel: String, uid: NodeUid },
/// Download remote `uid` to the local path, stamping `mtime`. `size` is the
/// remote's reported size, used as the transfer's expected total.
/// Upload a changed local file as a new revision of `uid`, if Drive still
/// holds the revision `seen` describes (B249).
UploadRevision {
rel: String,
uid: NodeUid,
seen: RemoteSeen,
},
/// Download remote `uid` to the local path, stamping `seen.mtime`.
/// `seen.size` is the remote's reported size, used as the transfer's
/// expected total.
///
/// `local` is what the planning walk saw at this path, or `None` if nothing
/// was there. A download can run for minutes, so the apply step revalidates
/// against it immediately before it renames over the destination: a writer
/// that finished an edit in the meantime must not be overwritten without a
/// conflict copy (bugs.md B39).
/// conflict copy (`docs/BUGS.md` B39).
Download {
rel: String,
uid: NodeUid,
mtime: i64,
size: i64,
seen: RemoteSeen,
local: Option<LocalSig>,
},
/// First sync of a path present on both sides at the same size, with a
/// digest on Drive: hash the local file, and record the baseline if it holds
/// the same bytes. A different file of the same size goes the way of
/// [`Pending::Conflict`]: two machines syncing one folder must not read as
/// in step while they hold different content.
Adopt {
rel: String,
uid: NodeUid,
seen: RemoteSeen,
sha1: String,
local: LocalSig,
},
/// Both sides changed: set the local copy aside, then download remote `uid`.
Conflict {
rel: String,
uid: NodeUid,
mtime: i64,
size: i64,
seen: RemoteSeen,
},
/// Both sides changed during a [`push pass`](Core::push_pass): upload the local
/// copy into `parent` under a conflict name, leaving the remote file as it is.
@@ -189,6 +250,7 @@ impl Pending {
| Pending::UploadNew { rel, .. }
| Pending::UploadRevision { rel, .. }
| Pending::Download { rel, .. }
| Pending::Adopt { rel, .. }
| Pending::Conflict { rel, .. }
| Pending::PushConflict { rel, .. } => rel,
}
@@ -201,7 +263,9 @@ impl Pending {
Pending::UploadNew { .. }
| Pending::UploadRevision { .. }
| Pending::PushConflict { .. } => ActivityKind::Upload,
Pending::Download { .. } | Pending::Conflict { .. } => ActivityKind::Download,
Pending::Download { .. } | Pending::Adopt { .. } | Pending::Conflict { .. } => {
ActivityKind::Download
}
}
}
@@ -212,6 +276,7 @@ impl Pending {
Pending::UploadNew { .. } => "new file",
Pending::UploadRevision { .. } => "new version",
Pending::Download { .. } => "from Drive",
Pending::Adopt { .. } => "already the same on Drive",
Pending::Conflict { .. } => "local changes kept as a conflict copy",
Pending::PushConflict { .. } => "local changes uploaded as a conflict copy",
}
@@ -227,6 +292,10 @@ pub(super) enum PassAbort {
Interrupted,
/// The pass could not establish its diff (a walk or the baseline load failed).
Failed(String),
/// The daemon is stopping. What the pass sent before the stop has landed or
/// will be cancelled with the runtime; nothing more is sent, and the next
/// start's pass picks up from the baseline.
Stopped,
}
impl From<String> for PassAbort {
+460 -22
View File
@@ -90,6 +90,96 @@ pub(super) fn filter_baseline<'a, B>(
.collect()
}
/// The paths a local scan left out because it could not read them: a folder
/// it has no permission for, an entry that vanished between the listing and
/// its `stat`, a name that is not UTF-8.
///
/// A left-out path is not a deleted one. A pass drops everything an entry
/// covers from classification, exactly as it drops ignored paths, so a
/// baseline row under an unreadable folder is neither read as a local deletion
/// nor answered with a download into a folder the pass cannot write. Failing
/// the whole pass instead is what B55 did, and it left one root-owned folder
/// stopping every other file in the tree from syncing (B196).
#[derive(Default)]
pub(super) struct Unscanned {
covered: HashSet<String>,
/// The subset worth telling the user about, with why: an entry that only
/// vanished mid-scan is not, since the next pass sees it settled.
unreadable: Vec<String>,
}
impl Unscanned {
/// Leave out `rel`, which vanished while the scan looked at it.
pub(super) fn vanished(&mut self, rel: String) {
self.covered.insert(rel);
}
/// Leave out `rel`, which cannot be read for `reason`.
pub(super) fn unreadable(&mut self, rel: String, reason: &str) {
self.unreadable.push(format!("{rel} ({reason})"));
self.covered.insert(rel);
}
/// Whether `rel` is a left-out path or lies under one.
pub(super) fn covers(&self, rel: &str) -> bool {
if self.covered.is_empty() {
return false;
}
let mut at = rel;
loop {
if self.covered.contains(at) {
return true;
}
match at.rfind('/') {
Some(i) => at = &at[..i],
None => return false,
}
}
}
/// The unreadable paths with their reasons, sorted, for the activity feed.
pub(super) fn report(&self) -> Vec<String> {
let mut lines = self.unreadable.clone();
lines.sort_unstable();
lines
}
}
/// The baseline paths a pass can see: [`filter_baseline`], minus everything
/// under a path the scan could not read. This is the set [`guard_local_wipe`]
/// checks and the set a push pass reads deletions from, since a path outside
/// it is absent from the local walk by rule or by failure, never by loss.
pub(super) fn tracked_baseline<'a, B>(
baseline: &'a HashMap<String, B>,
rules: &IgnoreRules,
unscanned: &Unscanned,
) -> HashMap<String, &'a B> {
let mut tracked = filter_baseline(baseline, rules);
tracked.retain(|rel, _| !unscanned.covers(rel));
tracked
}
/// The baseline paths a push pass has to trash on Drive: tracked, and gone
/// from the local walk. Shallowest first, so a trashed folder takes its
/// children with it.
///
/// Built from [`tracked_baseline`] rather than from the raw baseline. Reading
/// it raw trashed every synced path that later became ignored, because the walk
/// skips those, and the switch to on-demand that the push pass runs for then
/// deleted the local copy too (B197).
pub(super) fn local_deletions<B, L>(
tracked: &HashMap<String, &B>,
local: &HashMap<String, L>,
) -> Vec<String> {
let mut missing: Vec<String> = tracked
.keys()
.filter(|rel| !local.contains_key(*rel))
.cloned()
.collect();
missing.sort_by_key(|p| p.matches('/').count());
missing
}
/// Refuse to run a pass whose local side has vanished in its entirety.
///
/// When every baseline path is absent locally, the likely cause is an unavailable
@@ -109,14 +199,160 @@ pub(super) fn guard_local_wipe<B, L>(
Ok(())
}
/// Refuse to run a pass whose remote side has vanished in its entirety: the
/// remote twin of [`guard_local_wipe`].
///
/// An empty or truncated remote walk reads to the planner as "Drive deleted
/// everything", which a pass answers by deleting the local copies. When not
/// one synced path is left on Drive, a failed listing is far likelier than a
/// deliberate wipe of the whole folder, so the pass stops instead.
pub(super) fn guard_remote_wipe<B, R>(
baseline: &HashMap<String, B>,
remote: &HashMap<String, R>,
) -> Result<(), String> {
if !baseline.is_empty() && baseline.keys().all(|rel| !remote.contains_key(rel)) {
return Err(format!(
"every one of the {} synced paths is missing on Proton Drive; refusing to \
delete them here. If they were deleted on purpose, remove the folder from sync.",
baseline.len()
));
}
Ok(())
}
/// What a remote walk saw but could not read: a node the SDK left out because
/// it would not decrypt, or a file whose size could not be read.
///
/// A path missing from the walk is read as deleted on Drive, so a pass must
/// not classify one it only failed to read. Since a node that failed to decrypt
/// has no name, all a walk can say is that a folder's listing came back short;
/// every path under that folder that the walk did not return is then left out,
/// as [`Unscanned`] leaves out what the local scan could not read.
#[derive(Default)]
pub(super) struct RemoteGaps {
/// Folders whose listing came back with fewer nodes than it named.
short: HashSet<String>,
/// Paths seen, but whose entry could not be read.
unread: HashSet<String>,
}
impl RemoteGaps {
/// The listing of `folder` named nodes that did not come back.
pub(super) fn short_listing(&mut self, folder: &str) {
self.short.insert(folder.to_string());
}
/// `rel` is on Drive, but the walk could not read what it needs of it.
pub(super) fn unread(&mut self, rel: String) {
self.unread.insert(rel);
}
pub(super) fn is_empty(&self) -> bool {
self.short.is_empty() && self.unread.is_empty()
}
/// Whether `rel` may be missing from `remote` only because the walk could
/// not read it. A path the walk returned is never hidden.
pub(super) fn hides<R>(
&self,
rel: &str,
remote: &HashMap<String, R>,
remote_dirs: &HashMap<String, NodeUid>,
) -> bool {
if self.is_empty() || remote.contains_key(rel) {
return false;
}
if self.unread.contains(rel) {
return true;
}
// The deepest folder the walk did list decides: under a short listing,
// anything not returned may be one of the nodes it could not read.
let mut at = parent_rel(rel);
loop {
if remote_dirs.contains_key(at) {
return self.short.contains(at);
}
if at.is_empty() {
return false;
}
at = parent_rel(at);
}
}
/// The folders to name to the user, sorted.
pub(super) fn report(&self) -> Vec<String> {
let mut lines: Vec<String> = self
.short
.iter()
.map(|rel| {
let shown = if rel.is_empty() { "the folder" } else { rel };
format!("{shown} (items on Proton Drive that cannot be read)")
})
.chain(
self.unread
.iter()
.map(|rel| format!("{rel} (cannot be read on Proton Drive)")),
)
.collect();
lines.sort_unstable();
lines
}
}
/// Why the remote folder at `rel` must not be trashed although it was deleted
/// here: something under it is new or changed on Drive since the last sync,
/// so it is another device's work that this computer has never had.
///
/// Trashing the folder would take that work with it, and the next pass would
/// then delete the copy this pass just downloaded. `None` when everything under
/// it is exactly what the baseline recorded.
pub(super) fn remote_subtree_changed(
rel: &str,
remote: &HashMap<String, RemoteItem>,
baseline: &HashMap<String, StoredSyncEntry>,
) -> Option<String> {
let prefix = format!("{rel}/");
let mut changed: Vec<&String> = remote
.iter()
.filter(|(path, _)| path.starts_with(&prefix))
.filter(|(path, item)| match baseline.get(*path) {
None => true,
Some(_) if item.is_dir => false,
Some(base) => {
remote_sig(base) != Some((item.mtime, item.size))
|| revision_moved(Some(base), item.revision.as_deref())
}
})
.map(|(path, _)| path)
.collect();
changed.sort_unstable();
changed
.first()
.map(|path| format!("{path} is new or changed on Proton Drive"))
}
/// The size the baseline recorded for `rel`, if its remote signature's mtime
/// still matches `mtime`.
/// still matches `mtime` and its revision is still `revision`.
///
/// A row without a revision id, while Drive reports one, cannot vouch for the
/// revision, so the size is read for real. Such a row comes from before
/// schema 44, or from an upload Drive did not list yet when it was read back.
/// Another computer's revision in the same second as that upload kept the
/// modification time, and only the size could show it (B254). The pass then
/// records the id on the row, so this costs one read per row.
pub(super) fn unchanged_remote_size(
baseline: &HashMap<String, StoredSyncEntry>,
rel: &str,
mtime: i64,
revision: Option<&str>,
) -> Option<i64> {
match baseline.get(rel).and_then(remote_sig) {
let base = baseline.get(rel)?;
if revision_moved(Some(base), revision)
|| (base.remote_revision_id.is_none() && revision.is_some())
{
return None;
}
match remote_sig(base) {
Some((recorded, size)) if recorded == mtime => Some(size),
_ => None,
}
@@ -131,6 +367,17 @@ pub(super) fn join_rel(prefix: &str, name: &str) -> String {
}
}
/// Whether Drive's active revision `now` is not the one the baseline row was
/// synced at. Only a revision id on both sides can say so; a row from before
/// schema 44, or a node read without one, leaves it to the modification time
/// and size.
pub(super) fn revision_moved(base: Option<&StoredSyncEntry>, now: Option<&str>) -> bool {
match (base.and_then(|b| b.remote_revision_id.as_deref()), now) {
(Some(recorded), Some(now)) => recorded != now,
_ => false,
}
}
/// The stored remote signature of a baseline row, if it has one.
pub(super) fn remote_sig(e: &StoredSyncEntry) -> Option<(i64, i64)> {
match (&e.remote_rev, &e.remote_hash) {
@@ -161,7 +408,8 @@ pub(super) fn rel_to_path(rel: &str) -> PathBuf {
rel.split('/').collect()
}
/// The name for a conflict copy of `path`, e.g. `notes (sync-conflict 1700000000).txt`.
/// The name for a conflict copy of `path`, e.g.
/// `notes (sync-conflict laptop 1700000000).txt`.
#[cfg(test)]
pub(super) fn conflict_path(path: &Path, stamp: i64) -> PathBuf {
conflict_path_with_suffix(path, stamp, 0)
@@ -170,16 +418,17 @@ pub(super) fn conflict_path(path: &Path, stamp: i64) -> PathBuf {
/// As [`conflict_path`], with a deterministic suffix used when that name already
/// exists. This keeps conflict preservation from ever replacing an older copy.
pub(super) fn conflict_path_with_suffix(path: &Path, stamp: i64, suffix: u32) -> PathBuf {
conflict_path_as(path, crate::conflict_label(), stamp, suffix)
}
/// [`conflict_path_with_suffix`] with the machine's label given.
pub(super) fn conflict_path_as(path: &Path, label: &str, stamp: i64, suffix: u32) -> PathBuf {
let stem = path.file_stem().and_then(|s| s.to_str()).unwrap_or("file");
let ext = path.extension().and_then(|s| s.to_str());
let suffix = if suffix == 0 {
String::new()
} else {
format!("-{suffix}")
};
let tag = crate::conflict_tag(label, stamp, suffix);
let name = match ext {
Some(ext) => format!("{stem} (sync-conflict {stamp}{suffix}).{ext}"),
None => format!("{stem} (sync-conflict {stamp}{suffix})"),
Some(ext) => format!("{stem} (sync-conflict {tag}).{ext}"),
None => format!("{stem} (sync-conflict {tag})"),
};
match path.parent() {
Some(dir) => dir.join(name),
@@ -200,11 +449,14 @@ pub(super) enum FilePlan {
DeleteLocal,
DeleteRemote,
ForgetBaseline,
/// First sync of a file that already exists on both sides with identical
/// content: record the baseline without transferring anything. Without this,
/// First sync of a file that exists on both sides at the same size, with a
/// digest on Drive: record the baseline without transferring anything if
/// the local file hashes the same, else keep a conflict copy. Without this,
/// a folder synced for the first time cuts a `(sync-conflict)` copy of every
/// pre-existing file, because a missing baseline makes both sides read as
/// "changed" (see the mass first-sync conflict storm in docs/BUGS.md).
/// "changed" (see the mass first-sync conflict storm in docs/BUGS.md). Size
/// alone is not enough: a second machine joining the folder would read a
/// different file of the same size as in step (B248).
AdoptBaseline,
}
@@ -218,13 +470,18 @@ pub(super) fn plan_file(
}
let local_changed = local.is_some_and(|item| {
// Nanoseconds where both sides have them (bugs.md B25): whole seconds
// Nanoseconds where both sides have them (`docs/BUGS.md` B25): whole seconds
// could not see an edit made in the same second as the last sync that
// left the file the same length, so it was never uploaded.
baseline.is_none_or(|base| !LocalSig::from(item).same_content(&LocalSig::from(base)))
});
// Another client's new revision shows in its id even when the link's
// modification time and the size come out the same.
let remote_changed = remote.is_some_and(|item| {
baseline.is_none_or(|base| remote_sig(base) != Some((item.mtime, item.size)))
baseline.is_none_or(|base| {
remote_sig(base) != Some((item.mtime, item.size))
|| revision_moved(Some(base), item.revision.as_deref())
})
});
match (local, remote) {
@@ -234,10 +491,13 @@ pub(super) fn plan_file(
(false, true) => FilePlan::Download,
// With a baseline, both-changed is a genuine concurrent edit and must
// be preserved as a conflict. Without one, "both changed" only means
// this is the first sync and there was nothing to compare to: adopt
// the baseline when the sizes already match rather than conflict-copy
// an already-identical file. Divergent sizes stay a real conflict.
(true, true) if baseline.is_none() && l.size == r.size => FilePlan::AdoptBaseline,
// this is the first sync and there was nothing to compare to: when the
// sizes match and Drive has a digest to compare against, the apply
// step hashes the local file before adopting it. Divergent sizes, or
// no digest, stay a real conflict: a missing digest proves nothing.
(true, true) if baseline.is_none() && l.size == r.size && r.sha1.is_some() => {
FilePlan::AdoptBaseline
}
(true, true) => FilePlan::Conflict,
},
(Some(_), None) if baseline.is_none() || local_changed => FilePlan::UploadNew,
@@ -269,6 +529,8 @@ mod file_plan_tests {
is_dir: false,
mtime,
size,
revision: None,
sha1: None,
}
}
@@ -286,12 +548,13 @@ mod file_plan_tests {
local_size,
remote_rev: Some(remote_mtime.to_string()),
remote_hash: Some(remote_size.to_string()),
remote_revision_id: None,
}
}
/// Whole seconds cannot see an edit made inside the same second that leaves the
/// file the same length, so a mirror silently stopped uploading it. With a
/// nanosecond time on both sides it is an ordinary local change (bugs.md B25).
/// nanosecond time on both sides it is an ordinary local change (`docs/BUGS.md` B25).
#[test]
fn a_same_second_same_size_edit_is_still_a_local_change() {
let mut base = baseline(10, 20, 30, 40);
@@ -407,12 +670,28 @@ mod file_plan_tests {
fn first_sync_of_an_identical_file_adopts_the_baseline_not_a_conflict() {
// No baseline yet (first sync). Same size on both sides is a pre-existing
// identical file, not a concurrent edit: adopt it rather than cut a
// conflict copy of every file in the folder.
// conflict copy of every file in the folder. The apply step still
// hashes the local file before it records anything.
let local = local(10, 40);
let remote = RemoteItem {
sha1: Some("da39a3ee".into()),
..remote(30, 40)
};
assert_eq!(
plan_file(Some(&local), Some(&remote), None),
FilePlan::AdoptBaseline
);
}
/// A file on Drive without a digest cannot be compared without a download,
/// and the same size is not the same content (B248).
#[test]
fn first_sync_without_a_remote_digest_is_a_conflict() {
let local = local(10, 40);
let remote = remote(30, 40);
assert_eq!(
plan_file(Some(&local), Some(&remote), None),
FilePlan::AdoptBaseline
FilePlan::Conflict
);
}
@@ -428,6 +707,73 @@ mod file_plan_tests {
);
}
/// Another computer's new revision can leave the modification time and
/// the size as they were. The revision id still tells it apart (B249).
#[test]
fn a_new_remote_revision_is_a_change_even_at_the_same_mtime_and_size() {
let mut base = baseline(10, 20, 30, 40);
base.remote_revision_id = Some("r1".into());
let unedited = local(10, 20);
let same = RemoteItem {
revision: Some("r1".into()),
..remote(30, 40)
};
let newer = RemoteItem {
revision: Some("r2".into()),
..remote(30, 40)
};
assert_eq!(
plan_file(Some(&unedited), Some(&same), Some(&base)),
FilePlan::Unchanged
);
assert_eq!(
plan_file(Some(&unedited), Some(&newer), Some(&base)),
FilePlan::Download
);
let edited = local(11, 20);
assert_eq!(
plan_file(Some(&edited), Some(&newer), Some(&base)),
FilePlan::Conflict
);
// A row from before schema 44 has no id to compare, and a node read
// without one says nothing: both fall back to the signature.
let old = baseline(10, 20, 30, 40);
assert_eq!(
plan_file(Some(&unedited), Some(&newer), Some(&old)),
FilePlan::Unchanged
);
assert_eq!(
plan_file(Some(&unedited), Some(&remote(30, 40)), Some(&base)),
FilePlan::Unchanged
);
}
/// The walk reuses the recorded size only while the revision is the one
/// the baseline holds.
#[test]
fn a_moved_revision_makes_the_walk_read_the_size_again() {
let mut base = baseline(10, 20, 30, 40);
base.remote_revision_id = Some("r1".into());
let rows = HashMap::from([("file".to_string(), base)]);
assert_eq!(
unchanged_remote_size(&rows, "file", 30, Some("r1")),
Some(40)
);
assert_eq!(unchanged_remote_size(&rows, "file", 30, None), Some(40));
assert_eq!(unchanged_remote_size(&rows, "file", 30, Some("r2")), None);
assert_eq!(unchanged_remote_size(&rows, "file", 31, Some("r1")), None);
}
/// A row without a revision id reuses its size only while Drive gives no
/// id either (B254).
#[test]
fn a_row_without_a_revision_makes_the_walk_read_the_size_again() {
let rows = HashMap::from([("file".to_string(), baseline(10, 20, 30, 40))]);
assert_eq!(unchanged_remote_size(&rows, "file", 30, Some("r1")), None);
assert_eq!(unchanged_remote_size(&rows, "file", 30, None), Some(40));
}
#[test]
fn a_missing_remote_signature_is_treated_as_changed() {
let local = local(10, 20);
@@ -440,4 +786,96 @@ mod file_plan_tests {
FilePlan::Download
);
}
fn uid(id: &str) -> NodeUid {
NodeUid::new(VolumeId::from("v"), LinkId::from(id))
}
fn folder() -> RemoteItem {
RemoteItem {
is_dir: true,
mtime: 0,
size: 0,
..remote(0, 0)
}
}
/// An empty or failed remote listing must not read as "Drive deleted
/// everything" and delete the local copies.
#[test]
fn the_remote_wipe_guard_blocks_only_a_total_disappearance() {
let base = HashMap::from([("a".to_string(), ()), ("b".to_string(), ())]);
let none: HashMap<String, RemoteItem> = HashMap::new();
assert!(guard_remote_wipe(&base, &none).is_err());
let one = HashMap::from([("b".to_string(), remote(1, 1))]);
assert!(guard_remote_wipe(&base, &one).is_ok());
let empty: HashMap<String, ()> = HashMap::new();
assert!(guard_remote_wipe(&empty, &none).is_ok());
}
/// A node that failed to decrypt has no name, so everything a short
/// listing did not return is left out, and nothing else.
#[test]
fn a_short_remote_listing_hides_what_it_did_not_return() {
let remote = HashMap::from([
("docs".to_string(), folder()),
("docs/a.txt".to_string(), remote(1, 1)),
("pics".to_string(), folder()),
]);
let dirs = HashMap::from([
(String::new(), uid("root")),
("docs".to_string(), uid("docs")),
("pics".to_string(), uid("pics")),
]);
let mut gaps = RemoteGaps::default();
assert!(!gaps.hides("docs/b.txt", &remote, &dirs));
gaps.short_listing("docs");
// Returned paths stand; unreturned ones under the short folder hide,
// and so does anything under a subfolder it failed to return.
assert!(!gaps.hides("docs/a.txt", &remote, &dirs));
assert!(gaps.hides("docs/b.txt", &remote, &dirs));
assert!(gaps.hides("docs/sub/deep.txt", &remote, &dirs));
// A complete sibling listing still reads deletions as deletions.
assert!(!gaps.hides("pics/gone.jpg", &remote, &dirs));
assert!(!gaps.hides("top.txt", &remote, &dirs));
gaps.short_listing("");
assert!(gaps.hides("top.txt", &remote, &dirs));
}
#[test]
fn a_file_whose_size_could_not_be_read_is_hidden() {
let remote: HashMap<String, RemoteItem> = HashMap::new();
let dirs = HashMap::from([(String::new(), uid("root"))]);
let mut gaps = RemoteGaps::default();
gaps.unread("a.txt".to_string());
assert!(gaps.hides("a.txt", &remote, &dirs));
assert!(!gaps.hides("b.txt", &remote, &dirs));
assert_eq!(gaps.report(), ["a.txt (cannot be read on Proton Drive)"]);
}
/// A folder deleted here holds another device's new work on Drive, so
/// trashing it would lose that work.
#[test]
fn a_remote_folder_with_new_or_changed_files_is_not_trashed() {
let mut base = HashMap::from([
("x".to_string(), baseline(0, 0, 0, 0)),
("x/old.txt".to_string(), baseline(1, 1, 5, 6)),
]);
let mut remote_tree = HashMap::from([
("x".to_string(), folder()),
("x/old.txt".to_string(), remote(5, 6)),
("xy/new.txt".to_string(), remote(1, 1)),
]);
// Unchanged under x; a name-prefixed sibling does not count.
assert_eq!(remote_subtree_changed("x", &remote_tree, &base), None);
remote_tree.insert("x/old.txt".to_string(), remote(9, 6));
assert!(remote_subtree_changed("x", &remote_tree, &base).is_some());
remote_tree.insert("x/old.txt".to_string(), remote(5, 6));
remote_tree.insert("x/sub".to_string(), folder());
assert!(remote_subtree_changed("x", &remote_tree, &base).is_some());
base.insert("x/sub".to_string(), baseline(0, 0, 0, 0));
assert_eq!(remote_subtree_changed("x", &remote_tree, &base), None);
}
}
+20
View File
@@ -74,6 +74,26 @@ pub(crate) fn watchdog_ping() {
let _ = notify("WATCHDOG=1\n");
}
/// Sleep for `duration`, pinging the watchdog on the way.
///
/// For the stretches where the daemon is alive but has no mount, and so no
/// supervisor: waiting for a login, or backing off after a failed mount. The
/// manager arms the watchdog when the process starts, so without these pings a
/// daemon that is merely retrying is aborted after `WatchdogSec` and leaves a
/// coredump instead of the error it was logging. Never use this while mounted:
/// there the ping must prove the control socket answers.
pub fn idle_sleep(duration: Duration) {
const CHUNK: Duration = Duration::from_secs(10);
let mut left = duration;
watchdog_ping();
while !left.is_zero() {
let step = left.min(CHUNK);
std::thread::sleep(step);
left -= step;
watchdog_ping();
}
}
/// Tell the manager the daemon is stopping, so a slow teardown is not mistaken
/// for a hang by the watchdog it is racing.
pub(crate) fn stopping() {
+96 -16
View File
@@ -25,7 +25,7 @@ use std::collections::{BTreeMap, HashMap};
use std::io::Read;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use pdfs_core::control::{ActivityKind, ImportSummary, TransferDirection};
use pdfs_core::takeout::{Bucket, TakeoutPhoto, TakeoutScan, scan_archives};
@@ -36,6 +36,7 @@ use sha1::{Digest, Sha1};
use tracing::{info, warn};
use super::Core;
use crate::photos::can_make_thumbnail;
use crate::transfers::CountingReader;
/// The one import this daemon may be running, and how the last one ended.
@@ -136,6 +137,30 @@ fn read_entry<R>(
}
impl Core {
/// Copy archive entry `entry` into the thumbnail staging folder and return
/// its path. The caller removes it.
fn stage_takeout_entry(
&self,
archive: &mut zip::ZipArchive<std::io::BufReader<std::fs::File>>,
entry: &str,
) -> CoreResult<PathBuf> {
static STAGED: AtomicU64 = AtomicU64::new(0);
let path = self.cache.raw_thumbnail_staging_dir().join(format!(
"takeout-{}-{}",
std::process::id(),
STAGED.fetch_add(1, Ordering::Relaxed)
));
let copied = read_entry(archive, entry, |reader| {
let mut file = std::fs::File::create(&path)?;
std::io::copy(reader, &mut file)
})?;
if let Err(e) = copied {
let _ = std::fs::remove_file(&path);
return Err(CoreError::internal(format!("copy {entry}: {e}")));
}
Ok(path)
}
/// Import Google Photos Takeout archives. Blocking; run on its own thread.
///
/// `cancel` stops the run at the next photo boundary — an upload already on
@@ -321,18 +346,65 @@ impl Core {
capture_time: photo.capture_time,
..Default::default()
};
let uploaded = read_entry(&mut archive, &photo.entry, |reader| {
let counting = CountingReader::new(reader, &guard);
self.rt.block_on(photos.upload_photo_from(
&photo.name,
&photo.media_type,
counting,
photo.size as i64,
Vec::new(),
metadata,
false,
))
});
// A thumbnail is made from a file, so a photo that can have one
// is copied out of the archive first and uploaded from the copy.
// Without one, Proton's apps show a blank tile until they make
// their own.
let staged = can_make_thumbnail(&photo.name, photo.size)
.then(|| self.stage_takeout_entry(&mut archive, &photo.entry))
.and_then(|staged| {
staged
.inspect_err(|e| {
warn!(name = photo.name, error = %e,
"copying a takeout photo out failed; uploading it without a thumbnail")
})
.ok()
});
let uploaded = match &staged {
Some(path) => {
let thumbnails = self.upload_thumbnails(path, &photo.name);
// No sidecar and no date in the name: the photo's own
// EXIF, before Drive files it under today.
let metadata = PhotoUploadMetadata {
capture_time: metadata
.capture_time
.or_else(|| pdfs_core::exiftime::capture_time_of(path)),
..metadata
};
std::fs::File::open(path)
.map_err(|e| {
CoreError::internal(format!("open {}: {e}", path.display()))
})
.map(|file| {
let counting =
CountingReader::new(std::io::BufReader::new(file), &guard);
self.rt.block_on(photos.upload_photo_from(
&photo.name,
&photo.media_type,
counting,
photo.size as i64,
thumbnails,
metadata,
false,
))
})
}
None => read_entry(&mut archive, &photo.entry, |reader| {
let counting = CountingReader::new(reader, &guard);
self.rt.block_on(photos.upload_photo_from(
&photo.name,
&photo.media_type,
counting,
photo.size as i64,
Vec::new(),
metadata,
false,
))
}),
};
if let Some(path) = staged {
let _ = std::fs::remove_file(path);
}
drop(guard);
job.step();
@@ -480,7 +552,15 @@ impl Core {
let mut state = IMPORT.lock();
state.running = false;
state.cancel = None;
state.last = outcome.as_ref().ok().cloned();
// A failed run is reported too: without it, `--wait` read
// "no import has run" and exited as if all was well.
state.last = Some(match &outcome {
Ok(summary) => summary.clone(),
Err(e) => ImportSummary {
error: Some(e.to_string()),
..Default::default()
},
});
}
match outcome {
Ok(summary) => {
@@ -530,7 +610,7 @@ fn open_archive(
/// The lowercase-hex SHA-1 of everything `reader` yields — the digest Proton's
/// photo duplicate detection is keyed on.
fn sha1_of(reader: &mut (dyn Read + Send)) -> CoreResult<String> {
pub(crate) fn sha1_of(reader: &mut (dyn Read + Send)) -> CoreResult<String> {
let mut hasher = Sha1::new();
let mut buffer = vec![0u8; 256 * 1024];
loop {
@@ -545,7 +625,7 @@ fn sha1_of(reader: &mut (dyn Read + Send)) -> CoreResult<String> {
Ok(hex_lower(&hasher.finalize()))
}
fn hex_lower(bytes: &[u8]) -> String {
pub(crate) fn hex_lower(bytes: &[u8]) -> String {
let mut out = String::with_capacity(bytes.len() * 2);
for byte in bytes {
out.push_str(&format!("{byte:02x}"));
+22 -1
View File
@@ -46,6 +46,8 @@ struct JobEntry {
detail: Mutex<String>,
done: AtomicU64,
total: AtomicU64,
/// Housekeeping that moves nothing; see [`JobItem::background`].
background: bool,
}
/// The set of transfers and jobs currently in flight. Cloned `Arc`-style across
@@ -123,6 +125,11 @@ impl TransferRegistry {
self.download_limit.set(download);
}
/// The upload cap in bytes per second, `0` when there is none.
pub fn upload_cap(&self) -> u64 {
self.upload_limit.rate.load(Ordering::Relaxed)
}
fn limit(&self, direction: TransferDirection) -> &RateLimit {
match direction {
TransferDirection::Upload => &self.upload_limit,
@@ -160,12 +167,25 @@ impl TransferRegistry {
/// that carries its progress and deregisters the job when dropped. The job
/// starts indeterminate; give it a `total` once one is known.
pub fn begin_job(self: &Arc<Self>, title: impl Into<String>) -> JobGuard {
self.register_job(title.into(), false)
}
/// [`TransferRegistry::begin_job`] for housekeeping the user did not ask
/// for and that changes nothing in their files, such as an index rebuild or
/// a cache check. It is reported, but front ends don't call the drive busy
/// over it.
pub fn begin_background_job(self: &Arc<Self>, title: impl Into<String>) -> JobGuard {
self.register_job(title.into(), true)
}
fn register_job(self: &Arc<Self>, title: String, background: bool) -> JobGuard {
let id = self.next.fetch_add(1, Ordering::Relaxed);
let entry = Arc::new(JobEntry {
title: title.into(),
title,
detail: Mutex::new(String::new()),
done: AtomicU64::new(0),
total: AtomicU64::new(0),
background,
});
self.jobs.lock().insert(id, entry.clone());
JobGuard {
@@ -191,6 +211,7 @@ impl TransferRegistry {
detail: e.detail.lock().clone(),
done: e.done.load(Ordering::Relaxed),
total: e.total.load(Ordering::Relaxed),
background: e.background,
}
})
.collect()
@@ -1,5 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16">
<!-- Main menu: three bars. Bundled because themes such as Papirus draw
open-menu-symbolic as three dots, the same as every row's "more" menu. -->
<path fill="#222222" d="M 2,2.5 H 14 A 1,1 0 0 1 14,4.5 H 2 A 1,1 0 0 1 2,2.5 Z M 2,7 H 14 A 1,1 0 0 1 14,9 H 2 A 1,1 0 0 1 2,7 Z M 2,11.5 H 14 A 1,1 0 0 1 14,13.5 H 2 A 1,1 0 0 1 2,11.5 Z"/>
</svg>

Before

Width:  |  Height:  |  Size: 441 B

@@ -14,7 +14,6 @@
<file preprocess="xml-stripblanks">scalable/actions/pdfs-details-pane-symbolic.svg</file>
<file preprocess="xml-stripblanks">scalable/actions/pdfs-link-symbolic.svg</file>
<file preprocess="xml-stripblanks">scalable/actions/pdfs-map-symbolic.svg</file>
<file preprocess="xml-stripblanks">scalable/actions/pdfs-menu-symbolic.svg</file>
<file preprocess="xml-stripblanks">scalable/actions/pdfs-people-symbolic.svg</file>
<file preprocess="xml-stripblanks">scalable/actions/pdfs-share-symbolic.svg</file>
<file preprocess="xml-stripblanks">scalable/actions/pdfs-sync-symbolic.svg</file>
+7 -1
View File
@@ -18,7 +18,9 @@
.sidebar-status image.warning { color: var(--warning-color); }
.sidebar-status image.error { color: var(--error-color); }
.sidebar-quota progressbar trough, .sidebar-quota progressbar progress { min-height: 4px; }
.sidebar-account { padding: 4px 4px 4px 8px; }
/* The account row is the main menu's button: a row, not a pushbutton, so
* the name keeps the text weight. */
button.sidebar-account { padding: 4px 8px; font-weight: normal; }
/* A count of what waits on a page, at the end of its sidebar row */
.sidebar-badge {
min-width: 10px;
@@ -46,7 +48,11 @@
.badge-pinned { color: var(--success-color); }
.badge-cached { color: var(--success-color); }
.badge-cloud { color: color-mix(in srgb, currentColor 55%, transparent); }
.badge-issue { color: var(--error-color); }
.browser-statusbar { background-color: alpha(currentColor, 0.025); }
/* The folder on screen, last in the breadcrumb trail: a flat button's side
* padding, so the "›" before it is spaced like the ones between buttons. */
.crumb-current { padding: 0 10px; }
scale.browser-status-meter trough { min-width: 104px; min-height: 6px; }
.bulk-bar { padding: 6px 8px; border-radius: 12px; background: color-mix(in srgb, var(--accent-bg-color) 12%, transparent); }
.dropzone { border: 2px dashed alpha(currentColor, 0.25); border-radius: 16px; padding: 28px 18px; transition: background 160ms ease, border-color 160ms ease; }
+27 -8
View File
@@ -10,9 +10,14 @@
//! `dead_code` allowances below are for — not code nobody uses.
use std::path::{Path, PathBuf};
use std::time::Duration;
use pdfs_core::control::{PhotoKind, SearchHit};
/// How long a `stat` of the mount may take before the file counts as not
/// there. A frozen mount never answers, and every caller has a fallback.
const STAT_TIMEOUT: Duration = Duration::from_secs(2);
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum DriveActivation {
Folder,
@@ -55,18 +60,32 @@ pub(crate) fn mounted_target(mountpoint: &Path, hit: &SearchHit) -> Option<PathB
present(mounted_or_relative(mountpoint, hit))
}
/// [`mounted_target`] for callers holding a mountpoint-relative path rather
/// than a search hit — the browser's listing entries, which are by definition
/// paths in the primary mount's tree.
#[allow(dead_code)] // browser only
pub(crate) fn mounted_target_rel(mountpoint: &Path, relative_path: &str) -> Option<PathBuf> {
present(mounted_path(mountpoint, relative_path))
}
fn present(path: PathBuf) -> Option<PathBuf> {
path.symlink_metadata().is_ok().then_some(path)
}
/// Whether anything is at `path`, asked on a thread of its own and given up
/// on after [`STAT_TIMEOUT`]. `None` means "not there" or "no answer".
///
/// For the GTK binaries, whose main thread must not `stat` the mount: a
/// mount that hangs hung the whole window with it. The thread of a `stat`
/// that never returns is left behind; there is no way to cancel one.
#[allow(dead_code)] // GTK binaries only
pub(crate) async fn present_within(path: PathBuf) -> Option<PathBuf> {
let (tx, rx) = async_channel::bounded(1);
std::thread::spawn(move || {
let _ = tx.send_blocking(present(path));
});
match gtk4::glib::future_with_timeout(STAT_TIMEOUT, rx.recv()).await {
Ok(Ok(target)) => target,
Ok(Err(_)) => None,
Err(_) => {
tracing::warn!("the mount did not answer in time");
None
}
}
}
/// The mount path for a hit, whether or not anything is there — for callers
/// that have no fallback (a folder cannot be materialised) or already know the
/// entry came from a mount listing.
+186 -45
View File
@@ -10,6 +10,7 @@ pub(crate) mod pages;
pub(crate) mod reload;
#[path = "../theme.rs"]
pub(crate) mod theme;
pub(crate) mod update;
pub(crate) mod widgets;
use compat::*;
@@ -33,7 +34,9 @@ use pages::trash::*;
use pages::verify::*;
use reload::{LoadTicket, Loader, replace_items};
use theme::{load_resources, proton_theme_active, set_proton_theme};
use update::*;
use widgets::compat::*;
use widgets::crumb::*;
use widgets::details::*;
use widgets::file_list::*;
use widgets::menu::*;
@@ -68,15 +71,15 @@ use gtk4::glib::BoxedAnyObject;
use pdfs_core::auth;
use pdfs_core::config::{AppDirs, FileSort, FilesView};
use pdfs_core::config::{AppDirs, FileSort, FilesView, WindowState};
use pdfs_core::control::{
ActivityEntry, ActivityKind, AlbumInfo, BookmarkInfo, ConflictInfo, ConflictKeep, DeviceInfo,
DirEntry, ErrorKind, ImportSummary, InvitationInfo, JobItem, MappingProgress, PendingOpInfo,
PhotoItem, PhotoKind, PhotoMonth, PlaceInfo, PublicLinkInfo, RefreshScope, Request, Response,
RestorableFolder, RestoreItem, RevisionInfo, SearchHit, ShareEntry, ShareEntryKind, SharedItem,
SyncFolderInfo, SyncPhase, SyncProgress, ThumbnailBuildStatus, Topic, TransferDirection,
TransferItem, send,
SyncFolderInfo, SyncIssue, SyncPhase, SyncProgress, ThumbnailBuildStatus, Topic,
TransferDirection, TransferItem, send,
};
use pdfs_core::mounts::{MountAccess, MountKind, MountMode, MountSpec};
@@ -119,6 +122,9 @@ struct Ui {
/// keyring (a DBus round-trip). Populated at startup and updated only on
/// login / logout. `None` = signed out.
session: RefCell<Option<auth::StoredSession>>,
/// Whether this run already asked for the mailbox password of a locked
/// account, so a refresh never re-opens a dialog the user cancelled.
unlock_offered: Cell<bool>,
/// Whether the last refresh saw a live mount daemon. Gates the unpin buttons
/// (which need the daemon to evict + re-hydrate) and every mutating action.
mounted: RefCell<bool>,
@@ -126,8 +132,12 @@ struct Ui {
/// the page stack; [`sync_sidebar`] mirrors navigation that starts elsewhere.
sidebar: gtk4::ListBox,
/// The sidebar/content split. Collapsed while signed out, so the login page
/// owns the whole window and no destination is reachable without a session.
/// owns the whole window and no destination is reachable without a session,
/// and while the window is [`narrow`](Self::narrow).
nav: adw::NavigationSplitView,
/// The window is too narrow for the sidebar beside a page, so the two take
/// turns. Set by the narrow breakpoint; [`refresh`] reads it.
narrow: Cell<bool>,
/// Shared thumbnails for ordinary image files outside the Photos gallery.
/// One cache and request queue serves Files, search, Shared and Trash, so
/// the same image is downloaded and decoded only once.
@@ -147,6 +157,7 @@ struct Ui {
pub(crate) locations: LocationsState,
pub(crate) activity: ActivityState,
pub(crate) takeout: TakeoutState,
pub(crate) update: UpdateState,
}
impl Ui {
@@ -199,12 +210,20 @@ impl Ui {
}
fn main() -> glib::ExitCode {
tracing_subscriber::fmt()
.with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env().unwrap_or_else(|_| "info".into()),
)
.init();
let verify = std::env::args().nth(1).as_deref() == Some(VERIFY_ARG);
let logs = tracing_subscriber::fmt().with_env_filter(
tracing_subscriber::EnvFilter::try_from_default_env().unwrap_or_else(|_| "info".into()),
);
// The verification window's stdout carries its token back to the app.
if verify {
logs.with_writer(std::io::stderr).init();
} else {
logs.init();
}
i18n::init();
if verify {
return run_verification_window();
}
// The tray launches the app with arguments ("--page locations",
// "--confirm-stop"), which a running instance must receive too.
@@ -285,7 +304,10 @@ fn present_window(app: &adw::Application) {
/// Spawn the tray icon process in the background.
pub(crate) fn spawn_tray() {
match Command::new("pdfs-tray").spawn() {
Ok(_) => tracing::info!("spawned `pdfs-tray`"),
Ok(child) => {
tracing::info!("spawned `pdfs-tray`");
pdfs_core::opener::reap(child);
}
Err(e) => tracing::error!("failed to spawn `pdfs-tray`: {e}"),
}
}
@@ -300,8 +322,15 @@ fn build_window(app: &adw::Application) {
return;
}
};
// On a fresh install the daemon has not run yet to make these, and the
// Takeout page keeps its staged archives in the state dir (B194).
if let Err(e) = dirs.ensure() {
tracing::warn!("cannot create the app's directories: {e}");
}
let stack = adw::ViewStack::new();
// Not homogeneous: each page sets the window's minimum width only while it
// is shown, so the widest page does not hold every other one wide.
let stack = adw::ViewStack::builder().hhomogeneous(false).build();
let (login_page, login_widgets) = build_login_page();
let main_widgets = build_main_page();
let (browser_page, browser_widgets) = build_browser_page();
@@ -319,8 +348,12 @@ fn build_window(app: &adw::Application) {
login_frame.add_top_bar(&{
let header = adw::HeaderBar::new();
header.set_show_title(false);
// The split is collapsed while signed out; going "back" would only
// uncover a sidebar with nowhere to go.
header.set_show_back_button(false);
header
});
login_frame.add_top_bar(&login_widgets.keyring_banner);
login_frame.set_content(Some(&login_page));
stack.add_named(&login_frame, Some("login"));
stack.add_named(&browser_page, Some("browser"));
@@ -335,7 +368,8 @@ fn build_window(app: &adw::Application) {
// Sidebar: the signed-in destinations. Selecting a row swaps the page stack;
// `sync_sidebar` pushes the other way when navigation happens elsewhere (e.g.
// login lands on Files).
let (sidebar_page, sidebar_list) = build_sidebar(&main_widgets.footer);
let update = UpdateState::new();
let (sidebar_page, sidebar_list) = build_sidebar(&main_widgets.footer, &update.banner);
// Every page brings its own header bar (see `page_frame`), so the content
// side is the bare stack.
@@ -348,6 +382,8 @@ fn build_window(app: &adw::Application) {
.content(&content_page)
.min_sidebar_width(200.0)
.max_sidebar_width(240.0)
// A narrow window opens on the page, not the sidebar.
.show_content(true)
.build();
// Toasts float over everything, so a report from a background action reaches
@@ -355,16 +391,22 @@ fn build_window(app: &adw::Application) {
let toasts = adw::ToastOverlay::new();
toasts.set_child(Some(&split));
let (startup_session, keyring_error) = match read_session() {
Ok(session) => (session, None),
Err(e) => (None, Some(e)),
};
let ui = Rc::new(Ui {
dirs,
stack: stack.clone(),
toasts: toasts.clone(),
busy: Cell::new(0),
opening: RefCell::new(HashSet::new()),
session: RefCell::new(auth::load().ok()),
session: RefCell::new(startup_session),
unlock_offered: Cell::new(false),
mounted: RefCell::new(false),
sidebar: sidebar_list.clone(),
nav: split.clone(),
narrow: Cell::new(false),
file_thumbs: FileThumbnailState::new(),
login: login_widgets,
status: StatusState {
@@ -394,6 +436,7 @@ fn build_window(app: &adw::Application) {
theme_row: main_widgets.theme_row.clone(),
online_map_row: main_widgets.online_map_row.clone(),
tray_row: main_widgets.tray_row.clone(),
update_row: main_widgets.update_row.clone(),
language_row: main_widgets.language_row.clone(),
settings_suppress: Cell::new(false),
budget_source: RefCell::new(None),
@@ -520,6 +563,7 @@ fn build_window(app: &adw::Application) {
reload_pending: Cell::new(false),
loading: Cell::new(false),
has_more: Cell::new(false),
library: Cell::new(true),
page_waiters: RefCell::new(Vec::new()),
width: Cell::new(0),
photo_tex: RefCell::new(HashMap::new()),
@@ -561,6 +605,8 @@ fn build_window(app: &adw::Application) {
detail: locations_widgets.card_detail.clone(),
pause: locations_widgets.pause.clone(),
paused: Cell::new(false),
sign_in: locations_widgets.card_sign_in.clone(),
restart: locations_widgets.card_restart.clone(),
},
queue: QueueState {
group: locations_widgets.queue_group.clone(),
@@ -613,8 +659,12 @@ fn build_window(app: &adw::Application) {
running: Cell::new(false),
dry_run: Cell::new(false),
},
update,
});
wire_login(&ui);
if let Some(e) = keyring_error {
keyring_unreadable(&ui, &e);
}
let ui_status = ui.clone();
main_widgets
.status_button
@@ -656,6 +706,7 @@ fn build_window(app: &adw::Application) {
wire_activity(&ui, &activity_widgets);
wire_takeout(&ui, &takeout_widgets);
wire_retry(&ui);
wire_update(&ui);
// Lazily load the Files / Photos pages the first time they're shown, so the
// network round-trip only happens on demand rather than on every refresh.
@@ -692,13 +743,18 @@ fn build_window(app: &adw::Application) {
}
});
let saved = ui.dirs.load_config().window.usable();
let window = adw::ApplicationWindow::builder()
.application(app)
.title("Proton Drive")
.default_width(980)
.default_height(680)
.default_width(saved.width)
.default_height(saved.height)
.maximized(saved.maximized)
.width_request(WindowState::MIN_WIDTH)
.height_request(WindowState::MIN_HEIGHT)
.content(&toasts)
.build();
remember_window_state(&ui, app, &window);
// A wide window docks the details pane beside the files instead of laying it
// over them.
let wide = adw::Breakpoint::new(
@@ -706,6 +762,18 @@ fn build_window(app: &adw::Application) {
);
wide.add_setter(&ui.browser.split, "collapsed", Some(&false.to_value()));
window.add_breakpoint(wide);
// A narrow window shows the sidebar and the page in turn, so every header
// bar keeps room for the window buttons. Signals rather than a setter:
// `refresh` decides the collapse while signed out, and would undo one.
let narrow = adw::Breakpoint::new(
adw::BreakpointCondition::parse("max-width: 760sp").expect("static breakpoint"),
);
narrow.add_setter(&ui.browser.search, "width-chars", Some(&8.to_value()));
let ui_narrow = ui.clone();
narrow.connect_apply(move |_| set_narrow(&ui_narrow, true));
let ui_wide = ui.clone();
narrow.connect_unapply(move |_| set_narrow(&ui_wide, false));
window.add_breakpoint(narrow);
install_shortcuts(&ui, &window);
install_window_actions(&ui, &window);
install_launch_actions(&ui, app, &window);
@@ -719,6 +787,51 @@ fn build_window(app: &adw::Application) {
window.present();
}
/// Follow the narrow breakpoint. While signed out the login page keeps the
/// split collapsed whatever the width.
fn set_narrow(ui: &Ui, narrow: bool) {
ui.narrow.set(narrow);
if ui.session.borrow().is_some() {
ui.nav.set_collapsed(narrow);
}
}
/// Save the window's size when it closes, and when the app quits with the
/// window still open (Ctrl+Q, the tray's Quit), which never asks it to close.
fn remember_window_state(ui: &Rc<Ui>, app: &adw::Application, window: &adw::ApplicationWindow) {
let ui_close = ui.clone();
window.connect_close_request(move |window| {
save_window_state(&ui_close, window);
glib::Propagation::Proceed
});
let ui_quit = ui.clone();
let window_quit = window.downgrade();
app.connect_shutdown(move |_| {
if let Some(window) = window_quit.upgrade() {
save_window_state(&ui_quit, &window);
}
});
}
/// Write the window's unmaximised size and maximised state to the config, if
/// either changed. Runs as the window goes away, so a failure is only logged.
fn save_window_state(ui: &Ui, window: &adw::ApplicationWindow) {
let (width, height) = window.default_size();
let state = WindowState {
width,
height,
maximized: window.is_maximized(),
};
let mut config = ui.dirs.load_config();
if config.window == state {
return;
}
config.window = state;
if let Err(e) = ui.dirs.save_config(&config) {
tracing::warn!("cannot save the window size: {e}");
}
}
/// Whether `window` is minimised or otherwise out of sight (another workspace,
/// fully covered), as far as the compositor tells us.
fn window_hidden(window: &adw::ApplicationWindow) -> bool {
@@ -765,10 +878,13 @@ const DESTINATIONS: [(&str, &str, &str); 7] = [
/// Index of the first row in the sidebar's sync section.
const SIDEBAR_SYNC_SECTION: i32 = 6;
/// The navigation sidebar: the destinations over a footer carrying the sync
/// status, the account quota and the account menu. Returns the page (for the
/// The navigation sidebar: the destinations under the update banner and over a
/// footer carrying the sync status, the account quota and the account menu. Returns the page (for the
/// split view) and the list (to drive + reflect the current page).
fn build_sidebar(footer: &gtk4::Box) -> (adw::NavigationPage, gtk4::ListBox) {
fn build_sidebar(
footer: &gtk4::Box,
update_banner: &adw::Banner,
) -> (adw::NavigationPage, gtk4::ListBox) {
let list = gtk4::ListBox::new();
list.set_selection_mode(gtk4::SelectionMode::Single);
list.add_css_class("navigation-sidebar");
@@ -813,7 +929,6 @@ fn build_sidebar(footer: &gtk4::Box) -> (adw::NavigationPage, gtk4::ListBox) {
let header = adw::HeaderBar::new();
header.set_title_widget(Some(&brand));
header.pack_end(&build_primary_menu());
let scroll = gtk4::ScrolledWindow::builder()
.hscrollbar_policy(gtk4::PolicyType::Never)
@@ -822,6 +937,7 @@ fn build_sidebar(footer: &gtk4::Box) -> (adw::NavigationPage, gtk4::ListBox) {
.build();
let toolbar = adw::ToolbarView::new();
toolbar.add_top_bar(&header);
toolbar.add_top_bar(update_banner);
toolbar.set_content(Some(&scroll));
toolbar.add_bottom_bar(footer);
@@ -847,6 +963,18 @@ fn set_busy_spinners(visible: bool) {
});
}
/// `row` in a frame that scrolls sideways once the window is too narrow for it,
/// so a row of labelled buttons never sets the window's minimum width.
fn scroll_sideways(row: &impl IsA<gtk4::Widget>) -> gtk4::ScrolledWindow {
gtk4::ScrolledWindow::builder()
.hscrollbar_policy(gtk4::PolicyType::Automatic)
.vscrollbar_policy(gtk4::PolicyType::Never)
.propagate_natural_width(true)
.propagate_natural_height(true)
.child(row)
.build()
}
/// A destination page's frame: its own header bar over the page content, the
/// way GNOME apps lay a split view out. The title sits in the middle with the
/// busy spinner beside it; the page packs its actions into the returned header
@@ -910,6 +1038,14 @@ fn wire_sidebar(ui: &Rc<Ui>) {
ui_row.stack.set_visible_child_name(page);
}
});
// In a narrow window the sidebar covers the page, so picking a row, even
// the selected one, has to bring the page forward.
let ui_open = ui.clone();
ui.sidebar.connect_row_activated(move |_, _| {
if ui_open.nav.is_collapsed() {
ui_open.nav.set_show_content(true);
}
});
}
/// Re-fetch the visible page from the server, bypassing every layer of cache
@@ -1045,8 +1181,17 @@ pub(crate) fn set_sidebar_badge(ui: &Ui, page: &str, count: u64) {
/// Show a transient toast. Non-blocking by design: an action's outcome is
/// reported without stealing focus or forcing a click, so the user can keep
/// working while a slow upload lands.
///
/// A toast's title is Pango markup unless told otherwise. Messages carry file
/// names and daemon errors, and one with a `&` or `<` in it showed as an empty
/// toast, so every toast here is plain text.
fn toast(ui: &Rc<Ui>, message: &str) {
ui.toasts.add_toast(adw::Toast::new(message));
ui.toasts.add_toast(
adw::Toast::builder()
.title(message)
.use_markup(false)
.build(),
);
}
/// Show a toast that offers one follow-up action — an Undo, or a way to the page
@@ -1057,6 +1202,7 @@ fn toast(ui: &Rc<Ui>, message: &str) {
fn toast_action(ui: &Rc<Ui>, message: &str, label: &str, action: impl Fn(&Rc<Ui>) + 'static) {
let toast = adw::Toast::builder()
.title(message)
.use_markup(false)
.button_label(label)
.timeout(8)
.build();
@@ -1079,7 +1225,11 @@ fn toast_error(ui: &Rc<Ui>, what: &str, detail: &str) {
gettext_f("{what}: {detail}", &[("what", what), ("detail", detail)])
};
tracing::warn!("{message}");
let toast = adw::Toast::builder().title(&message).timeout(6).build();
let toast = adw::Toast::builder()
.title(&message)
.use_markup(false)
.timeout(6)
.build();
ui.toasts.add_toast(toast);
}
@@ -1136,25 +1286,7 @@ fn toast_failure(ui: &Rc<Ui>, what: &str, message: &str, kind: ErrorKind) {
}
}
/// The sidebar's primary (hamburger) menu: the app-level entries that don't
/// belong on any one page.
fn build_primary_menu() -> gtk4::MenuButton {
let menu = gio::Menu::new();
menu.append(Some(&gettext("Preferences")), Some("win.preferences"));
menu.append(Some(&gettext("Keyboard Shortcuts")), Some("win.shortcuts"));
menu.append(
Some(&gettext("About Proton Drive for Linux")),
Some("win.about"),
);
gtk4::MenuButton::builder()
.icon_name("pdfs-menu-symbolic")
.tooltip_text(gettext("Main menu"))
.primary(true)
.menu_model(&menu)
.build()
}
/// Back the primary and account menus' entries with window actions.
/// Back the main menu's entries with window actions.
fn install_window_actions(ui: &Rc<Ui>, window: &adw::ApplicationWindow) {
let preferences = gio::SimpleAction::new("preferences", None);
let ui_prefs = ui.clone();
@@ -1206,7 +1338,7 @@ fn install_window_actions(ui: &Rc<Ui>, window: &adw::ApplicationWindow) {
let dialog = adw::AboutDialog::builder()
.application_name(gettext("Proton Drive for Linux"))
.application_icon("io.narl.proton-drive-linux")
.version(pdfs_core::config::APP_VERSION)
.version(env!("CARGO_PKG_VERSION"))
.developer_name("Nils Pukropp")
.website("https://proton-drive.narl.io")
.issue_url("https://github.com/narrrl/proton-drive-linux/issues")
@@ -1291,6 +1423,12 @@ fn install_launch_actions(ui: &Rc<Ui>, app: &adw::Application, window: &adw::App
if ui_page.session.borrow().is_none() {
return;
}
// Signed in, but Proton refused the session (the tray's "Sign In
// Again…"): the login page, with the rest of the app still reachable.
if page == "login" {
show_login(&ui_page);
return;
}
// Activity lives on as the Sync page's History view.
if page == "activity" {
ui_page.locations.views.set_visible_child_name("history");
@@ -1346,8 +1484,8 @@ fn debug_info(ui: &Rc<Ui>) -> String {
format!(
"App version: {}\nUser agent: {}\nlibadwaita: {}.{}.{}\nGTK: {}.{}.{}\n\
Mountpoint: {}\nControl socket: {}\nMount service running: {}\n",
pdfs_core::config::APP_VERSION,
pdfs_core::config::USER_AGENT,
config.resolved_app_version(),
config.resolved_user_agent(),
adw::major_version(),
adw::minor_version(),
adw::micro_version(),
@@ -1691,9 +1829,12 @@ pub(crate) fn flash_copied(button: &gtk4::Button, icon: &'static str) {
/// Open a local path with the handler the user configured — `xdg-open` unless
/// `open_with` in `config.json` overrides it for this kind of file.
///
/// On a thread of its own: telling a folder from a file is a `stat`, and the
/// path is often in the mount, which may hang.
fn open_path(path: &str) {
let path = Path::new(path);
pdfs_core::opener::open_default(path, path.is_dir());
let path = PathBuf::from(path);
std::thread::spawn(move || pdfs_core::opener::open_default(&path, path.is_dir()));
}
/// [`open_path`] for a file the daemon materialised into the content cache,
+3 -1
View File
@@ -378,7 +378,9 @@ pub(crate) fn activity_status(
) {
ui.activity.status.set_icon_name(Some(icon));
ui.activity.status.set_title(title);
ui.activity.status.set_description(Some(description));
ui.activity
.status
.set_description(Some(&gtk4::glib::markup_escape_text(description)));
ui.activity.retry.set_visible(retry);
ui.activity.content.set_visible_child_name("status");
}
@@ -341,6 +341,59 @@ fn detail_label(entry: &ActivityEntry) -> Option<String> {
&[("name", alt)],
));
}
if let Some((reason, copy)) = detail.split_once("; local changes uploaded as ") {
let args = [("name", copy)];
return Some(match edit_conflict(reason) {
// Translators: {name} is the copy the changes made on this
// computer were uploaded as.
EditConflict::Gone => gettext_f(
"It no longer exists on Proton Drive, so your changes were uploaded as {name}",
&args,
),
// Translators: {name} is the copy the changes made on this
// computer were uploaded as.
EditConflict::Trashed => gettext_f(
"It was moved to the Trash on Proton Drive, so your changes were uploaded as {name}",
&args,
),
// Translators: {name} is the copy the changes made on this
// computer were uploaded as.
EditConflict::Changed => gettext_f(
"It was changed on Proton Drive, so your changes were uploaded as {name}",
&args,
),
// Translators: {name} is the copy the changes made on this
// computer were uploaded as.
EditConflict::Other => gettext_f("Your changes were uploaded as {name}", &args),
});
}
if let Some((reason, path)) = detail.split_once("; local changes kept at ") {
let path = tilde_path(path);
let args = [("path", path.as_str())];
return Some(match edit_conflict(reason) {
// Translators: {path} is the file on this computer that holds the
// changes which could not be uploaded.
EditConflict::Gone => gettext_f(
"It no longer exists on Proton Drive, so your changes were kept in {path}",
&args,
),
// Translators: {path} is the file on this computer that holds the
// changes which could not be uploaded.
EditConflict::Trashed => gettext_f(
"It was moved to the Trash on Proton Drive, so your changes were kept in {path}",
&args,
),
// Translators: {path} is the file on this computer that holds the
// changes which could not be uploaded.
EditConflict::Changed => gettext_f(
"It was changed on Proton Drive, so your changes were kept in {path}",
&args,
),
// Translators: {path} is the file on this computer that holds the
// changes which could not be uploaded.
EditConflict::Other => gettext_f("Your changes were kept in {path}", &args),
});
}
if let Some(base) = detail.strip_prefix("differs from ") {
// Translators: {name} is the file the conflicting copy was made from.
return Some(gettext_f("Differs from {name}", &[("name", base)]));
@@ -388,6 +441,26 @@ fn detail_label(entry: &ActivityEntry) -> Option<String> {
Some(capitalize(detail))
}
/// Why a queued edit could not land on its file, from the reason the daemon
/// gives before "; local changes uploaded as" or "; local changes kept at".
enum EditConflict {
Gone,
Trashed,
Changed,
Other,
}
fn edit_conflict(reason: &str) -> EditConflict {
match reason {
"the file no longer exists remotely" => EditConflict::Gone,
"the file was trashed remotely" => EditConflict::Trashed,
r if r.starts_with("the remote revision changed under the queued write") => {
EditConflict::Changed
}
_ => EditConflict::Other,
}
}
/// A sync pass summary such as "2 uploaded, 1 folder(s) created", as a
/// list of counts. `None` when a part is not one the daemon writes.
fn sync_summary(detail: &str) -> Option<String> {
@@ -596,6 +669,38 @@ mod tests {
assert_eq!(d.details, vec!["1 person can view"]);
}
#[test]
fn an_edit_kept_as_a_copy_says_why_in_words() {
let d = describe(&entry(
ActivityKind::Upload,
"notes.txt",
"the remote revision changed under the queued write \
(based on revision r1, remote now at r2); \
local changes uploaded as notes (sync-conflict 1700000000).txt",
false,
));
assert_eq!(
d.details,
vec![
"It was changed on Proton Drive, so your changes were uploaded as \
notes (sync-conflict 1700000000).txt"
]
);
let d = describe(&entry(
ActivityKind::Upload,
"notes.txt",
"the file was trashed remotely; local changes kept at /var/staging/7",
false,
));
assert_eq!(
d.details,
vec![
"It was moved to the Trash on Proton Drive, so your changes were kept in \
/var/staging/7"
]
);
}
#[test]
fn unknown_details_show_as_written() {
let d = describe(&entry(
+3 -1
View File
@@ -645,7 +645,9 @@ pub(crate) fn close_album(ui: &Rc<Ui>) {
fn albums_status(ui: &Rc<Ui>, icon: &str, title: &str, description: &str) {
ui.gallery.albums_status.set_icon_name(Some(icon));
ui.gallery.albums_status.set_title(title);
ui.gallery.albums_status.set_description(Some(description));
ui.gallery
.albums_status
.set_description(Some(&gtk4::glib::markup_escape_text(description)));
ui.gallery.albums_stack.set_visible_child_name("status");
}
+119 -75
View File
@@ -1,4 +1,4 @@
use crate::activation::{DriveActivation, drive_activation, mounted_path, mounted_target_rel};
use crate::activation::{DriveActivation, drive_activation, mounted_path, present_within};
use crate::*;
pub(crate) struct BrowserState {
@@ -376,7 +376,7 @@ pub(crate) fn build_browser_page() -> (gtk4::Widget, BrowserWidgets) {
bulk_box.append(&bulk_clear);
let bulk = gtk4::Revealer::builder()
.transition_type(gtk4::RevealerTransitionType::SlideDown)
.child(&bulk_box)
.child(&scroll_sideways(&bulk_box))
.build();
// The status bar: contextual text (stretch 1), then the icon-size slider.
@@ -500,7 +500,9 @@ pub(crate) fn build_browser_page() -> (gtk4::Widget, BrowserWidgets) {
pub(crate) fn browser_status(ui: &Rc<Ui>, icon: &str, title: &str, description: &str, retry: bool) {
ui.browser.status.set_icon_name(Some(icon));
ui.browser.status.set_title(title);
ui.browser.status.set_description(Some(description));
ui.browser
.status
.set_description(Some(&gtk4::glib::markup_escape_text(description)));
ui.browser.retry.set_visible(retry);
// Only the empty-folder state offers a way to fill the folder; every other
// status (loading, offline, error) turns them back off.
@@ -520,9 +522,9 @@ pub(crate) fn clear_selection(ui: &Rc<Ui>) {
}
/// Reflect the current selection in the bulk bar: how many are selected, and
/// which bulk actions apply to that mix. Offline state is a file-only notion, so
/// a folder anywhere in the selection disables the two pin buttons rather than
/// half-failing once pressed.
/// which bulk actions apply to that mix. An entry kept offline through a pinned
/// folder above it cannot be made online only on its own, so it does not count
/// towards Make online only.
pub(crate) fn sync_bulk_bar(ui: &Rc<Ui>) {
let entries = selected_entries(ui);
// One selected entry is the details pane's job; the bar is for a batch.
@@ -530,8 +532,7 @@ pub(crate) fn sync_bulk_bar(ui: &Rc<Ui>) {
ui.browser.bulk.set_reveal_child(false);
return;
}
let files_only = entries.iter().all(|e| !e.is_dir);
let any_pinned = entries.iter().any(|e| e.pinned);
let any_pinned = entries.iter().any(|e| e.pinned && !e.pinned_by_folder);
let any_unpinned = entries.iter().any(|e| !e.pinned);
let mounted = *ui.mounted.borrow();
ui.browser.bulk_label.set_label(&ngettext_f(
@@ -541,12 +542,8 @@ pub(crate) fn sync_bulk_bar(ui: &Rc<Ui>) {
&[],
));
ui.browser.bulk_trash.set_sensitive(mounted);
ui.browser
.bulk_pin
.set_sensitive(mounted && files_only && any_unpinned);
ui.browser
.bulk_unpin
.set_sensitive(mounted && files_only && any_pinned);
ui.browser.bulk_pin.set_sensitive(mounted && any_unpinned);
ui.browser.bulk_unpin.set_sensitive(mounted && any_pinned);
ui.browser.bulk.set_reveal_child(true);
}
@@ -652,16 +649,19 @@ pub(crate) fn restore_uids(ui: &Rc<Ui>, uids: Vec<String>, count: usize) {
);
}
/// Pin or unpin every selected file, one request at a time.
/// Pin or unpin every selected entry, one request at a time. A folder's files
/// download in the background, so its request returns before they are there.
pub(crate) fn run_bulk_pin(ui: &Rc<Ui>, entries: Vec<DirEntry>, pin: bool) {
let socket = ui.dirs.control_socket();
// Only the entries that need changing: re-pinning an already-pinned file is
// a wasted round-trip, and it would inflate the count the toast reports.
let paths: Vec<String> = entries
// a wasted round-trip, and it would inflate the count the toast reports. One
// kept through a pinned folder above it has no pin of its own to remove.
let changing: Vec<&DirEntry> = entries
.iter()
.filter(|e| !e.is_dir && e.pinned != pin)
.map(|e| entry_rel(ui, e))
.filter(|e| !e.pinned_by_folder && e.pinned != pin)
.collect();
let any_folder = changing.iter().any(|e| e.is_dir);
let paths: Vec<String> = changing.iter().map(|e| entry_rel(ui, e)).collect();
if paths.is_empty() {
return;
}
@@ -698,13 +698,27 @@ pub(crate) fn run_bulk_pin(ui: &Rc<Ui>, entries: Vec<DirEntry>, pin: bool) {
(0, None) => {}
(n, _) => toast(
&ui,
&if pin {
&if pin && any_folder {
ngettext_f(
"Making {n} item available offline…",
"Making {n} items available offline…",
n as u64,
&[],
)
} else if pin {
ngettext_f(
"{n} file is now available offline",
"{n} files are now available offline",
n as u64,
&[],
)
} else if any_folder {
ngettext_f(
"{n} item is now online only",
"{n} items are now online only",
n as u64,
&[],
)
} else {
ngettext_f(
"{n} file is now online only",
@@ -832,8 +846,8 @@ fn show_location_column(ui: &Rc<Ui>, visible: bool) {
}
}
/// The menu for one entry, in the order of `docs/UI_UX_PLAN.md` §6: open,
/// offline, sharing, organising, and the destructive item last on its own.
/// The menu for one entry, in a fixed order: open, offline, sharing,
/// organising, and the destructive item last on its own.
pub(crate) fn entry_context_menu(ui: &Rc<Ui>, entry: &DirEntry) -> ActionMenu {
let mut menu = ActionMenu::new();
let (ui_c, entry_c) = (ui.clone(), entry.clone());
@@ -860,13 +874,17 @@ pub(crate) fn entry_context_menu(ui: &Rc<Ui>, entry: &DirEntry) -> ActionMenu {
}
menu.section();
if !entry.is_dir {
// Kept through a pinned folder above it, the entry has no pin of its own
// to remove: the check shows that state without offering to change it.
if entry.pinned_by_folder {
menu.locked_toggle(&gettext("Available offline"), true);
} else {
let (ui_c, entry_c) = (ui.clone(), entry.clone());
menu.toggle(&gettext("Available offline"), entry.pinned, move |_| {
toggle_pin(&ui_c, &entry_c)
});
menu.section();
}
menu.section();
let (ui_c, entry_c) = (ui.clone(), entry.clone());
menu.item(&gettext("Share…"), move || {
@@ -908,9 +926,8 @@ pub(crate) fn entry_context_menu(ui: &Rc<Ui>, entry: &DirEntry) -> ActionMenu {
/// simply absent here rather than offered and then refused.
pub(crate) fn bulk_context_menu(ui: &Rc<Ui>, entries: Vec<DirEntry>) -> ActionMenu {
let mut menu = ActionMenu::new();
// Offline state is a file-only notion; the check shows whether the whole
// batch already is.
if entries.iter().all(|e| !e.is_dir) {
// The check shows whether the whole batch already is offline.
{
let (ui_c, batch) = (ui.clone(), entries.clone());
let all_pinned = entries.iter().all(|e| e.pinned);
menu.toggle(&gettext("Available offline"), all_pinned, move |pin| {
@@ -966,28 +983,27 @@ pub(crate) fn background_context_menu(ui: &Rc<Ui>) -> ActionMenu {
pub(crate) fn open_entry_with(ui: &Rc<Ui>, entry: &DirEntry) {
let rel = entry_rel(ui, entry);
let mountpoint = ui.dirs.resolved_mountpoint(&ui.dirs.load_config());
let Some(path) = mounted_target_rel(&mountpoint, &rel) else {
toast_error(
ui,
&gettext("Couldn't open file"),
&gettext("Open With needs the Proton Drive service to be running."),
);
return;
};
let launcher = gtk4::FileLauncher::new(Some(&gio::File::for_path(path)));
launcher.set_always_ask(true);
let ui = ui.clone();
launcher.launch(
ui_window(&ui).as_ref(),
gio::Cancellable::NONE,
move |result| {
glib::spawn_future_local(async move {
let Some(path) = present_within(mounted_path(&mountpoint, &rel)).await else {
toast_error(
&ui,
&gettext("Couldn't open file"),
&gettext("Open With needs the Proton Drive service to be running."),
);
return;
};
let launcher = gtk4::FileLauncher::new(Some(&gio::File::for_path(path)));
launcher.set_always_ask(true);
let window = ui_window(&ui);
launcher.launch(window.as_ref(), gio::Cancellable::NONE, move |result| {
if let Err(e) = result
&& !e.matches(gtk4::DialogError::Dismissed)
{
toast_error(&ui, &gettext("Couldn't open file"), &e.to_string());
}
},
);
});
});
}
/// Copy the entry's public link, when it has one this client can read the
@@ -1095,27 +1111,29 @@ pub(crate) fn activate_entry(ui: &Rc<Ui>, entry: &DirEntry) {
/// [`mounted_target`](crate::activation::mounted_target).
pub(crate) fn download_and_open(ui: &Rc<Ui>, entry: &DirEntry) {
let rel = entry_rel(ui, entry);
let mountpoint = ui.dirs.resolved_mountpoint(&ui.dirs.load_config());
if let Some(path) = mounted_target_rel(&mountpoint, &rel) {
open_path(&path.to_string_lossy());
return;
}
// Ignore a repeat activation of a file already downloading, so an impatient
// double-click doesn't kick off a second round-trip.
// Ignore a repeat activation of a file already opening, so an impatient
// double-click doesn't open it twice or kick off a second round-trip.
if !ui.opening.borrow_mut().insert(rel.clone()) {
return;
}
ui.busy_begin();
let name = entry.name.clone();
let rx = spawn_request(
ui.dirs.control_socket(),
Request::OpenFile {
path: rel.clone(),
uid: Some(entry.uid.clone()),
},
);
let mountpoint = ui.dirs.resolved_mountpoint(&ui.dirs.load_config());
let (name, uid) = (entry.name.clone(), entry.uid.clone());
let ui = ui.clone();
glib::spawn_future_local(async move {
// Asked off the main thread: a mount that hangs used to hang the app.
if let Some(path) = present_within(mounted_path(&mountpoint, &rel)).await {
ui.opening.borrow_mut().remove(&rel);
open_path(&path.to_string_lossy());
return;
}
ui.busy_begin();
let rx = spawn_request(
ui.dirs.control_socket(),
Request::OpenFile {
path: rel.clone(),
uid: Some(uid),
},
);
let result = rx.recv().await;
ui.busy_end();
ui.opening.borrow_mut().remove(&rel);
@@ -1147,6 +1165,7 @@ pub(crate) fn toggle_pin(ui: &Rc<Ui>, entry: &DirEntry) {
let rx = spawn_request(ui.dirs.control_socket(), req);
let name = entry.name.clone();
let pinned = entry.pinned;
let is_dir = entry.is_dir;
let ui = ui.clone();
glib::spawn_future_local(async move {
match rx.recv().await {
@@ -1166,6 +1185,9 @@ pub(crate) fn toggle_pin(ui: &Rc<Ui>, entry: &DirEntry) {
&ui,
&if pinned {
gettext_f("“{name}” is now online only", &[("name", &name)])
} else if is_dir {
// The daemon downloads a folder's files in the background.
gettext_f("Making “{name}” available offline…", &[("name", &name)])
} else {
gettext_f("“{name}” is now available offline", &[("name", &name)])
},
@@ -1220,9 +1242,7 @@ pub(crate) fn repaint_crumb(ui: &Rc<Ui>, path: &str) {
));
let mut acc = String::new();
for (i, seg) in segments.iter().enumerate() {
let sep = gtk4::Label::new(Some("›"));
sep.add_css_class("dim-label");
ui.browser.crumb.append(&sep);
ui.browser.crumb.append(&crumb_separator());
acc = if acc.is_empty() {
seg.to_string()
} else {
@@ -1237,12 +1257,7 @@ pub(crate) fn repaint_crumb(ui: &Rc<Ui>, path: &str) {
/// flat button that navigates to `target` (clearing any active search first).
pub(crate) fn crumb_node(ui: &Rc<Ui>, label: &str, target: &str, current: bool) -> gtk4::Widget {
if current {
let l = gtk4::Label::builder()
.label(label)
.ellipsize(gtk4::pango::EllipsizeMode::Start)
.build();
l.add_css_class("heading");
return l.upcast();
return crumb_current(label);
}
let button = gtk4::Button::builder().label(label).build();
button.add_css_class("flat");
@@ -2494,12 +2509,19 @@ pub(crate) fn start_upload(ui: &Rc<Ui>, sources: Vec<String>) {
});
}
/// The sync-state badge for an entry: `(icon, css-class)`, or `None` for folders
/// (which carry no per-file cache state). Pinned (kept offline) ranks above merely
/// cached (downloaded, evictable); everything else is online-only.
/// The sync-state badge for an entry: `(icon, css-class)`. A folder carries no
/// cache state of its own, so it has one only when kept offline. A sync issue
/// ranks first, on folders too: the change waits for the user. Pinned (kept
/// offline) ranks above merely cached (downloaded, evictable); everything else
/// is online-only.
pub(crate) fn badge_for(entry: &DirEntry) -> Option<(&'static str, &'static str)> {
if entry.issue.is_some() {
return Some(("dialog-warning-symbolic", "badge-issue"));
}
if entry.is_dir {
return None;
return entry
.pinned
.then_some(("pdfs-offline-symbolic", "badge-pinned"));
}
Some(if entry.pinned {
("pdfs-offline-symbolic", "badge-pinned")
@@ -2513,9 +2535,11 @@ pub(crate) fn badge_for(entry: &DirEntry) -> Option<(&'static str, &'static str)
/// Paint `badge` to reflect the entry's sync state (see [`badge_for`]). Clears any
/// prior colour class first, since list factories recycle cells.
pub(crate) fn apply_badge(badge: &gtk4::Image, entry: &DirEntry) {
for class in ["badge-pinned", "badge-cached", "badge-cloud"] {
for class in ["badge-pinned", "badge-cached", "badge-cloud", "badge-issue"] {
badge.remove_css_class(class);
}
let issue = entry.issue.map(super::locations::issue_text);
badge.set_tooltip_text(issue.as_deref());
match badge_for(entry) {
Some((icon, class)) => {
badge.set_icon_name(Some(icon));
@@ -2906,6 +2930,7 @@ pub(crate) fn repaint_search(ui: &Rc<Ui>, hits: &[SearchHit]) {
size: h.size,
modified: h.modified,
pinned: h.pinned,
pinned_by_folder: false,
cached: h.cached,
uid: h.uid.clone(),
path: h.path.clone(),
@@ -2915,6 +2940,7 @@ pub(crate) fn repaint_search(ui: &Rc<Ui>, hits: &[SearchHit]) {
shared_by_unverified: false,
trashed_at: 0,
trashed_from: None,
issue: None,
})
.collect();
sort_entries(&mut entries, ui.browser.view.get());
@@ -2952,11 +2978,11 @@ fn listing_summary(files: usize, folders: usize, file_bytes: u64) -> String {
#[cfg(test)]
mod tests {
use super::{
MoveTarget, hit_location, listing_summary, move_target_allowed,
MoveTarget, badge_for, hit_location, listing_summary, move_target_allowed,
show_thumbnail_build_progress, sort_entries,
};
use pdfs_core::config::{FileSort, FilesView};
use pdfs_core::control::{DirEntry, ThumbnailBuildStatus};
use pdfs_core::control::{DirEntry, SyncIssue, ThumbnailBuildStatus};
fn entry(name: &str, is_dir: bool, size: u64, modified: i64) -> DirEntry {
DirEntry {
@@ -2965,6 +2991,7 @@ mod tests {
size,
modified,
pinned: false,
pinned_by_folder: false,
cached: false,
uid: String::new(),
path: String::new(),
@@ -2974,6 +3001,7 @@ mod tests {
shared_by_unverified: false,
trashed_at: 0,
trashed_from: None,
issue: None,
}
}
@@ -2981,6 +3009,22 @@ mod tests {
entries.iter().map(|e| e.name.as_str()).collect()
}
#[test]
fn a_sync_issue_shows_on_the_node_before_its_cache_state() {
let mut file = entry("a.txt", false, 1, 0);
file.pinned = true;
assert_eq!(badge_for(&file).unwrap().1, "badge-pinned");
file.issue = Some(SyncIssue::Quota);
assert_eq!(badge_for(&file).unwrap().1, "badge-issue");
let mut folder = entry("d", true, 0, 0);
assert_eq!(badge_for(&folder), None);
folder.pinned = true;
assert_eq!(badge_for(&folder).unwrap().1, "badge-pinned");
folder.issue = Some(SyncIssue::Missing);
assert_eq!(badge_for(&folder).unwrap().1, "badge-issue");
}
#[test]
fn a_search_hit_location_is_its_parent_folder() {
assert_eq!(hit_location("Work/Old/a.txt"), "Work/Old");
+143 -11
View File
@@ -204,7 +204,9 @@ pub(crate) fn wire_devices(ui: &Rc<Ui>, widgets: &DevicesWidgets) {
pub(crate) fn devices_status(ui: &Rc<Ui>, icon: &str, title: &str, description: &str, retry: bool) {
ui.devices.status.set_icon_name(Some(icon));
ui.devices.status.set_title(title);
ui.devices.status.set_description(Some(description));
ui.devices
.status
.set_description(Some(&gtk4::glib::markup_escape_text(description)));
ui.devices.retry.set_visible(retry);
ui.devices.content.set_visible_child_name("status");
}
@@ -390,6 +392,7 @@ fn this_computer_row(
let row = adw::ActionRow::builder()
.title(me.map_or_else(|| gettext("This computer"), |d| d.name.clone()))
.subtitle(this_computer_subtitle(folders))
.use_markup(false)
.activatable(true)
.build();
row.add_prefix(&gtk4::Image::from_icon_name("computer-symbolic"));
@@ -426,6 +429,7 @@ fn other_computer_row(ui: &Rc<Ui>, dev: &DeviceInfo) -> adw::ActionRow {
let row = adw::ActionRow::builder()
.title(&dev.name)
.subtitle(device_subtitle(dev))
.use_markup(false)
.activatable(true)
.build();
row.add_prefix(&gtk4::Image::from_icon_name("computer-symbolic"));
@@ -439,6 +443,14 @@ fn other_computer_row(ui: &Rc<Ui>, dev: &DeviceInfo) -> adw::ActionRow {
menu.item(&gettext("Restore to This Computer…"), move || {
prompt_restore_folders(&ui_c, Some((uid_c.clone(), name_c.clone())))
});
let (ui_c, uid_c, name_c) = (ui.clone(), uid.clone(), name.clone());
menu.item(&gettext("Sync With This Computer…"), move || {
prompt_join_device(&ui_c, &uid_c, &name_c)
});
let (ui_c, uid_c, name_c) = (ui.clone(), uid.clone(), name.clone());
menu.item(&gettext("Move to This Computer…"), move || {
prompt_migrate_device(&ui_c, &uid_c, &name_c)
});
menu.section();
let (ui_c, uid_c, name_c) = (ui.clone(), uid.clone(), name.clone());
menu.item(&gettext("Rename…"), move || {
@@ -483,9 +495,7 @@ fn repaint_device_crumb(ui: &Rc<Ui>, nav: &[(String, String)]) {
nav.is_empty(),
));
for (i, (_, name)) in nav.iter().enumerate() {
let sep = gtk4::Label::new(Some("›"));
sep.add_css_class("dim-label");
crumb.append(&sep);
crumb.append(&crumb_separator());
crumb.append(&device_crumb_node(ui, name, i + 1, i + 1 == nav.len()));
}
}
@@ -494,12 +504,7 @@ fn repaint_device_crumb(ui: &Rc<Ui>, nav: &[(String, String)]) {
/// that goes back up to `depth` levels below the device list.
fn device_crumb_node(ui: &Rc<Ui>, label: &str, depth: usize, current: bool) -> gtk4::Widget {
if current {
let l = gtk4::Label::builder()
.label(label)
.ellipsize(gtk4::pango::EllipsizeMode::Start)
.build();
l.add_css_class("heading");
return l.upcast();
return crumb_current(label);
}
let button = gtk4::Button::builder().label(label).build();
button.add_css_class("flat");
@@ -766,7 +771,10 @@ fn show_restore_picker(
.active(true)
.valign(gtk4::Align::Center)
.build();
let row = adw::EntryRow::builder().title(&f.name).build();
let row = adw::EntryRow::builder()
.title(&f.name)
.use_markup(false)
.build();
row.set_text(&f.local_path);
row.add_prefix(&check);
group.add(&row);
@@ -1096,6 +1104,130 @@ pub(crate) fn prompt_adopt_device(ui: &Rc<Ui>, uid: &str, name: &str) {
dialog.present(win.as_ref());
}
/// Ask before syncing into another computer's device alongside it, then offer
/// its folders for syncing here.
///
/// The same adoption as "Continue This Backup Here", but the other computer
/// goes on syncing, so the dialog says what that means: the two keep the same
/// folders in step, deletions included.
pub(crate) fn prompt_join_device(ui: &Rc<Ui>, uid: &str, name: &str) {
let win = ui_window(ui);
let dialog = adw::AlertDialog::builder()
.heading(gettext("Sync With This Computer?"))
// Translators: {name} is the name of the other computer this one syncs with.
.body(gettext_f(
"This computer and “{name}” keep the same folders in step. A change or a deletion on one reaches the other.\n\nNew synced folders on this computer go under “{name}” in Proton Drive. Folders already synced here stay where they are. Next you choose which of its folders to sync here.",
&[("name", name)],
))
.build();
dialog.add_response("cancel", &gettext("Cancel"));
dialog.add_response("join", &gettext("Sync Here"));
dialog.set_response_appearance("join", adw::ResponseAppearance::Suggested);
dialog.set_default_response(Some("cancel"));
dialog.set_close_response("cancel");
let ui = ui.clone();
let uid = uid.to_string();
// Translators: {name} is the name of the other computer this one now syncs with.
let done = gettext_f("Now syncing with “{name}”", &[("name", name)]);
dialog.connect_response(None, move |_, resp| {
if resp != "join" {
return;
}
ui.busy_begin();
let rx = spawn_request(
ui.dirs.control_socket(),
Request::AdoptDevice {
uid: Some(uid.clone()),
},
);
let (ui, done) = (ui.clone(), done.clone());
glib::spawn_future_local(async move {
let result = rx.recv().await;
ui.busy_end();
let failed = gettext("Couldn't sync with that computer");
match result {
Ok(Ok(Response::Ok { .. })) => {
reload_sync_pages(&ui);
toast(&ui, &done);
// The device is this computer's now, so its own restore
// picker offers the folders.
prompt_restore_folders(&ui, None);
}
Ok(Ok(Response::Error { message, kind })) => {
toast_failure(&ui, &failed, &message, kind)
}
_ => toast_error(
&ui,
&failed,
&gettext("The Proton Drive service didn't respond."),
),
}
});
});
dialog.present(win.as_ref());
}
/// Ask before moving another computer's folders into this computer's backup,
/// then offer them for syncing here.
///
/// Destructive-looking on purpose: the other computer keeps its device but
/// loses its folders, and a copy of Proton Drive still running there would
/// watch them vanish.
pub(crate) fn prompt_migrate_device(ui: &Rc<Ui>, uid: &str, name: &str) {
let win = ui_window(ui);
let dialog = adw::AlertDialog::builder()
.heading(gettext("Move Backup to This Computer?"))
// Translators: {name} is the name of the other computer whose folders move to this one.
.body(gettext_f(
"The folders backed up by “{name}” move into this computer's backup in Proton Drive, and you choose where they go on this computer. “{name}” stays in your account, without those folders.\n\nStop Proton Drive on “{name}” first. Otherwise it sees its folders disappear.",
&[("name", name)],
))
.build();
dialog.add_response("cancel", &gettext("Cancel"));
dialog.add_response("move", &gettext("Move Here"));
dialog.set_response_appearance("move", adw::ResponseAppearance::Destructive);
dialog.set_default_response(Some("cancel"));
dialog.set_close_response("cancel");
let ui = ui.clone();
let uid = uid.to_string();
// Translators: {name} is the name of the computer whose folders moved to this one.
let done = gettext_f("Moved the backup of “{name}”", &[("name", name)]);
dialog.connect_response(None, move |_, resp| {
if resp != "move" {
return;
}
ui.busy_begin();
let rx = spawn_request(
ui.dirs.control_socket(),
Request::MigrateDevice { uid: uid.clone() },
);
let (ui, done) = (ui.clone(), done.clone());
glib::spawn_future_local(async move {
let result = rx.recv().await;
ui.busy_end();
let failed = gettext("Couldn't move that backup");
match result {
Ok(Ok(Response::Ok { .. })) => {
reload_sync_pages(&ui);
toast(&ui, &done);
// The folders are this computer's now, so its own restore
// picker is where the user chooses their local paths.
prompt_restore_folders(&ui, None);
}
Ok(Ok(Response::Error { message, kind })) => {
toast_failure(&ui, &failed, &message, kind)
}
_ => toast_error(
&ui,
&failed,
&gettext("The Proton Drive service didn't respond."),
),
}
});
});
dialog.present(win.as_ref());
}
/// Run a mutation raised from Computers or Locations and reload the page on success.
pub(crate) fn run_devices_mutation(ui: &Rc<Ui>, req: Request, done: &str, failed: &str) {
ui.busy_begin();
+2 -1
View File
@@ -209,7 +209,8 @@ fn ask(view: &Rc<FinderView>, finder: Finder, generation: u32) {
Ok(Ok(Response::Error { message, .. })) => {
view.status.set_icon_name(Some("dialog-warning-symbolic"));
view.status.set_title(&failed);
view.status.set_description(Some(&message));
view.status
.set_description(Some(&gtk4::glib::markup_escape_text(&message)));
view.progress.set_visible(false);
return;
}
+207 -11
View File
@@ -7,7 +7,7 @@
//! `~/ProtonDrive` mount, plus each folder this computer backs up, whether it is
//! a mirrored local directory or an on-demand FUSE session. The page is called
//! *Locations* rather than *Mounts* because a mirror folder is a plain directory
//! with no FUSE session behind it (mount-architecture.md §4).
//! with no FUSE session behind it.
//!
//! The device rows drive the same control requests the Computers page used to:
//! mode switch, sync now, remove. What is new here is that the primary mount is
@@ -46,6 +46,10 @@ pub(crate) struct SyncCard {
pub(crate) pause: adw::SplitButton,
/// Whether the last status said paused, so the button knows which way to go.
pub(crate) paused: Cell<bool>,
/// Shown only while Proton refuses the session; opens the sign-in page.
pub(crate) sign_in: gtk4::Button,
/// Shown only while the daemon is older than the app; restarts it.
pub(crate) restart: gtk4::Button,
}
/// What a folder row's controls are built from. The subtitle and progress bar
@@ -70,8 +74,9 @@ pub(crate) struct LocationRow {
progress: gtk4::ProgressBar,
}
/// What a queue row shows that can change: id, attempts, next attempt, parked.
pub(crate) type QueueKey = (i64, i64, Option<i64>, bool);
/// What a queue row shows that can change: id, attempts, next attempt, parked,
/// issue.
pub(crate) type QueueKey = (i64, i64, Option<i64>, bool, Option<SyncIssue>);
/// The "Waiting to Upload" list: the daemon's pending-op queue.
pub(crate) struct QueueState {
@@ -107,6 +112,8 @@ pub(crate) struct LocationsWidgets {
pub(crate) card_icon: gtk4::Image,
pub(crate) card_title: gtk4::Label,
pub(crate) card_detail: gtk4::Label,
pub(crate) card_sign_in: gtk4::Button,
pub(crate) card_restart: gtk4::Button,
pub(crate) pause: adw::SplitButton,
pub(crate) queue_group: adw::PreferencesGroup,
pub(crate) retry_all: gtk4::Button,
@@ -133,7 +140,7 @@ pub(crate) fn build_locations_page(history: &gtk4::Widget) -> (gtk4::Widget, Loc
.build();
add_folder.add_css_class("flat");
let transfers_group = build_transfers_group();
let (card, card_icon, card_title, card_detail) = build_sync_card();
let (card, card_icon, card_title, card_detail, card_sign_in, card_restart) = build_sync_card();
let pause = build_pause_button();
let (queue_group, retry_all) = build_queue_group();
let conflicts_group = adw::PreferencesGroup::builder()
@@ -252,6 +259,8 @@ pub(crate) fn build_locations_page(history: &gtk4::Widget) -> (gtk4::Widget, Loc
card_icon,
card_title,
card_detail,
card_sign_in,
card_restart,
pause,
queue_group,
retry_all,
@@ -289,6 +298,16 @@ pub(crate) fn wire_locations(ui: &Rc<Ui>, retry: &gtk4::Button, add_folder: &gtk
let ui_add = ui.clone();
add_folder.connect_clicked(move |_| prompt_add_sync_folder(&ui_add));
wire_pause(ui);
let ui_sign_in = ui.clone();
ui.locations
.card
.sign_in
.connect_clicked(move |_| show_login(&ui_sign_in));
let ui_restart = ui.clone();
ui.locations
.card
.restart
.connect_clicked(move |_| restart_service_then(&ui_restart, |_| {}));
let ui_retry = ui.clone();
ui.locations
.queue
@@ -378,9 +397,16 @@ pub(crate) fn paint_sync_banner(ui: &Rc<Ui>) {
state.banner.set_revealed(away && title.is_some());
}
/// The status hero: a large state icon, the state in words, and what it
/// means.
fn build_sync_card() -> (gtk4::Box, gtk4::Image, gtk4::Label, gtk4::Label) {
/// The status hero: a large state icon, the state in words, what it means,
/// and the way out when there is one only the user can take.
fn build_sync_card() -> (
gtk4::Box,
gtk4::Image,
gtk4::Label,
gtk4::Label,
gtk4::Button,
gtk4::Button,
) {
let icon = gtk4::Image::builder()
.icon_name("pdfs-sync-symbolic")
.pixel_size(48)
@@ -407,9 +433,25 @@ fn build_sync_card() -> (gtk4::Box, gtk4::Image, gtk4::Label, gtk4::Label) {
let card = gtk4::Box::new(gtk4::Orientation::Horizontal, 18);
card.add_css_class("card");
card.add_css_class("sync-card");
let sign_in = gtk4::Button::builder()
.label(gettext("Sign In Again"))
.valign(gtk4::Align::Center)
.visible(false)
.build();
sign_in.add_css_class("pill");
sign_in.add_css_class("suggested-action");
let restart = gtk4::Button::builder()
.label(gettext("Restart"))
.valign(gtk4::Align::Center)
.visible(false)
.build();
restart.add_css_class("pill");
restart.add_css_class("suggested-action");
card.append(&icon);
card.append(&text);
(card, icon, title, detail)
card.append(&sign_in);
card.append(&restart);
(card, icon, title, detail, sign_in, restart)
}
/// Pause/Resume, with timed pauses in its menu.
@@ -591,7 +633,7 @@ fn repaint_queue(ui: &Rc<Ui>, items: &[PendingOpInfo]) {
items.sort_by_key(|op| (!op.failing, op.attempts == 0, op.id));
let key: Vec<QueueKey> = items
.iter()
.map(|op| (op.id, op.attempts, op.next_attempt_at, op.parked))
.map(|op| (op.id, op.attempts, op.next_attempt_at, op.parked, op.issue))
.collect();
if *queue.painted.borrow() == key {
return;
@@ -648,6 +690,10 @@ fn queue_row(ui: &Rc<Ui>, op: &PendingOpInfo, now: i64) -> adw::ActionRow {
"{action} · {state}",
&[("action", &action), ("state", &state)],
);
if let Some(issue) = op.issue {
subtitle.push('\n');
subtitle.push_str(&issue_text(issue));
}
if let Some(error) = &op.last_error {
subtitle.push('\n');
// Translators: {n} is how many attempts failed, {error} the last error message.
@@ -662,7 +708,7 @@ fn queue_row(ui: &Rc<Ui>, op: &PendingOpInfo, now: i64) -> adw::ActionRow {
.title(glib::markup_escape_text(&op.path).as_str())
.subtitle(glib::markup_escape_text(&subtitle).as_str())
.title_lines(1)
.subtitle_lines(3)
.subtitle_lines(if op.issue.is_some() { 5 } else { 3 })
.tooltip_text(&op.path)
.build();
let image = gtk4::Image::from_icon_name(if op.failing {
@@ -687,9 +733,157 @@ fn queue_row(ui: &Rc<Ui>, op: &PendingOpInfo, now: i64) -> adw::ActionRow {
retry.connect_clicked(move |_| retry_queued(&ui, Some(id)));
row.add_suffix(&retry);
}
if op.failing && op.exportable {
let export = gtk4::Button::builder()
.label(gettext("Export"))
.tooltip_text(gettext("Save a copy of this file's content elsewhere"))
.valign(gtk4::Align::Center)
.build();
export.add_css_class("flat");
let ui = ui.clone();
let id = op.id;
let name = Path::new(&op.path)
.file_name()
.map(|name| name.to_string_lossy().into_owned())
.unwrap_or_default();
export.connect_clicked(move |_| prompt_export_queued(&ui, id, &name));
row.add_suffix(&export);
}
if op.failing {
let discard = gtk4::Button::builder()
.label(gettext("Discard"))
.tooltip_text(gettext("Undo this change here instead of sending it"))
.valign(gtk4::Align::Center)
.build();
discard.add_css_class("flat");
let ui = ui.clone();
let id = op.id;
let path = op.path.clone();
let exportable = op.exportable;
discard.connect_clicked(move |_| prompt_discard_queued(&ui, id, &path, exportable));
row.add_suffix(&discard);
}
row
}
/// What a sync issue means and what the user can do about it. The change stays
/// queued either way.
pub(crate) fn issue_text(issue: SyncIssue) -> String {
match issue {
SyncIssue::Quota => gettext(
"Your Proton Drive storage is full. Free up space or upgrade your plan, and this change uploads on its own.",
),
SyncIssue::Access => gettext(
"You can no longer change this on Proton Drive. Ask the owner for edit access, or export the file to keep your version.",
),
SyncIssue::Missing => gettext(
"The folder this change belongs in is gone from Proton Drive. Export the file to keep your version.",
),
SyncIssue::Limit => gettext(
"Proton Drive's limit on items in a folder or on folder depth is reached. Make room in the folder, then retry.",
),
SyncIssue::Rejected | SyncIssue::Unknown => {
gettext("Proton Drive refused this change. It stays queued and is tried again.")
}
}
}
/// Ask where to save a copy of a queued file's content, then have the daemon
/// write it there. The change itself stays queued.
fn prompt_export_queued(ui: &Rc<Ui>, id: i64, name: &str) {
let window = ui_window(ui);
let dialog = gtk4::FileDialog::builder()
.title(gettext("Export File"))
.initial_name(name)
.build();
let ui = ui.clone();
dialog.save(window.as_ref(), gio::Cancellable::NONE, move |res| {
let Ok(file) = res else { return };
let Some(dest) = file.path().and_then(|p| p.to_str().map(str::to_string)) else {
toast_error(
&ui,
&gettext("Couldn't export the file"),
&gettext("That location isn't a local file."),
);
return;
};
let rx = spawn_request(
ui.dirs.control_socket(),
Request::ExportPendingOp { id, dest },
);
let ui = ui.clone();
glib::spawn_future_local(async move {
match rx.recv().await {
Ok(Ok(Response::Ok { .. })) => toast(&ui, &gettext("File exported")),
Ok(Ok(Response::Error { message, kind })) => {
toast_failure(&ui, &gettext("Couldn't export the file"), &message, kind)
}
_ => toast_error(
&ui,
&gettext("Couldn't export the file"),
&gettext("The Proton Drive service didn't respond."),
),
}
});
});
}
/// Confirm, then have the daemon drop a queued change and undo it here.
fn prompt_discard_queued(ui: &Rc<Ui>, id: i64, path: &str, exportable: bool) {
let body = if exportable {
// Translators: {path} is the file whose queued change would be dropped.
gettext_f(
"Your changes to {path} are undone here and never reach Proton Drive. Export the file first to keep your version.",
&[("path", path)],
)
} else {
// Translators: {path} is the file or folder whose queued change would be dropped.
gettext_f(
"The change to {path} is undone here and never reaches Proton Drive.",
&[("path", path)],
)
};
let dialog = adw::AlertDialog::builder()
.heading(gettext("Discard Change?"))
.body(body)
.build();
dialog.add_response("cancel", &gettext("Cancel"));
dialog.add_response("discard", &gettext("Discard"));
dialog.set_response_appearance("discard", adw::ResponseAppearance::Destructive);
dialog.set_default_response(Some("cancel"));
dialog.set_close_response("cancel");
let window = ui_window(ui);
let ui = ui.clone();
dialog.connect_response(None, move |_, resp| {
if resp == "discard" {
discard_queued(&ui, id);
}
});
dialog.present(window.as_ref());
}
fn discard_queued(ui: &Rc<Ui>, id: i64) {
let rx = spawn_request(ui.dirs.control_socket(), Request::DiscardPendingOp { id });
let ui = ui.clone();
glib::spawn_future_local(async move {
match rx.recv().await {
Ok(Ok(Response::Ok { .. })) => {
toast(&ui, &gettext("Change discarded"));
ui.locations.queue.painted.borrow_mut().clear();
refresh_queue(&ui);
}
Ok(Ok(Response::Error { message, kind })) => {
toast_failure(&ui, &gettext("Couldn't discard the change"), &message, kind)
}
_ => toast_error(
&ui,
&gettext("Couldn't discard the change"),
&gettext("The Proton Drive service didn't respond."),
),
}
});
}
/// Ask the daemon to stop waiting out one op's backoff, or every failed op's.
fn retry_queued(ui: &Rc<Ui>, id: Option<i64>) {
let rx = spawn_request(ui.dirs.control_socket(), Request::RetryPendingOp { id });
@@ -951,7 +1145,9 @@ pub(crate) fn locations_status(
) {
ui.locations.status.set_icon_name(Some(icon));
ui.locations.status.set_title(title);
ui.locations.status.set_description(Some(description));
ui.locations
.status
.set_description(Some(&gtk4::glib::markup_escape_text(description)));
ui.locations.retry.set_visible(retry);
ui.locations.content.set_visible_child_name("status");
}
+394 -34
View File
@@ -1,5 +1,5 @@
use crate::*;
use pdfs_core::proton_sdk::api::HumanVerificationCredential;
use pdfs_core::proton_sdk::api::{HumanVerification, HumanVerificationCredential};
pub(crate) struct LoginState {
// Login page.
@@ -9,6 +9,9 @@ pub(crate) struct LoginState {
/// Spins inside the Sign in button while a sign-in is running.
pub(crate) login_spinner: Spinner,
pub(crate) login_status: gtk4::Label,
/// Shown over the form while a saved sign-in exists but the keyring would
/// not hand it over; see [`keyring_unreadable`].
pub(crate) keyring_banner: adw::Banner,
}
/// Where the login page's account links go.
@@ -87,6 +90,12 @@ pub(crate) fn build_login_page() -> (gtk4::Widget, LoginState) {
.child(&inner)
.build();
let scroll = gtk4::ScrolledWindow::builder().child(&clamp).build();
let keyring_banner = adw::Banner::builder()
.title(gettext(
"Couldn't read your saved sign-in. Unlock the system keyring, then try again.",
))
.button_label(gettext("Retry"))
.build();
(
scroll.upcast(),
@@ -96,10 +105,80 @@ pub(crate) fn build_login_page() -> (gtk4::Widget, LoginState) {
login_button,
login_spinner,
login_status,
keyring_banner,
},
)
}
/// The stored session, `None` when there is none, or the keyring's error when
/// it could not be read. A locked or absent keyring is not a sign-out: the
/// session is still saved, and signing in again would only replace it.
pub(crate) fn read_session() -> pdfs_core::Result<Option<auth::StoredSession>> {
match auth::load() {
Ok(session) => Ok(Some(session)),
Err(pdfs_core::Error::NotLoggedIn) => Ok(None),
Err(e) => Err(e),
}
}
/// How long to wait before each automatic retry of an unreadable keyring. A
/// keyring daemon started with the session is usually up within seconds.
const KEYRING_RETRIES: [u32; 4] = [2, 5, 10, 30];
/// The keyring refused to hand over the saved session. Keep the login page up,
/// but say why, and pick the session up as soon as the keyring answers: on its
/// own a few times, and whenever the user presses Retry.
pub(crate) fn keyring_unreadable(ui: &Rc<Ui>, error: &pdfs_core::Error) {
tracing::warn!("cannot read the saved session: {error}");
let banner = &ui.login.keyring_banner;
if banner.is_revealed() {
return;
}
banner.set_revealed(true);
let ui_retry = ui.clone();
let handler = banner.connect_button_clicked(move |_| retry_session(&ui_retry));
// Disconnected once the banner goes, so a later failure wires it afresh.
let handler = RefCell::new(Some(handler));
banner.connect_revealed_notify(move |banner| {
if !banner.is_revealed()
&& let Some(handler) = handler.borrow_mut().take()
{
banner.disconnect(handler);
}
});
let ui = ui.clone();
glib::spawn_future_local(async move {
for seconds in KEYRING_RETRIES {
glib::timeout_future_seconds(seconds).await;
if !ui.login.keyring_banner.is_revealed() {
return;
}
retry_session(&ui);
}
});
}
/// Read the saved session again after [`keyring_unreadable`]. Found: show the
/// signed-in window. Gone: the plain login page. Still unreadable: keep waiting.
fn retry_session(ui: &Rc<Ui>) {
let ui = ui.clone();
glib::spawn_future_local(async move {
let Ok(result) = gio::spawn_blocking(read_session).await else {
return;
};
match result {
Ok(session) => {
ui.login.keyring_banner.set_revealed(false);
if session.is_some() && ui.session.borrow().is_none() {
*ui.session.borrow_mut() = session;
refresh(&ui);
}
}
Err(e) => tracing::debug!("the keyring still cannot be read: {e}"),
}
});
}
/// Connect the sign-in button: read the fields, run [`auth::login`] on a worker
/// thread, and report the outcome back on the main loop.
pub(crate) fn wire_login(ui: &Rc<Ui>) {
@@ -128,7 +207,12 @@ pub(crate) fn wire_login(ui: &Rc<Ui>) {
set_signing_in(&ui, true);
ui.login.login_status.set_text(&gettext("Signing in…"));
let (rx, totp_req_rx, hv_req_rx) = spawn_login(username, password);
let LoginChannels {
result: rx,
totp: totp_req_rx,
hv: hv_req_rx,
mailbox: mailbox_req_rx,
} = spawn_login(username, password);
// Surface the 2FA dialog only if the SDK actually asks for a code (i.e.
// the account has 2FA enabled). The worker blocks until the dialog feeds
@@ -145,15 +229,40 @@ pub(crate) fn wire_login(ui: &Rc<Ui>) {
});
// Same shape for human verification: only fires when the API gates the
// sign-in, and the worker is blocked on the token until it does.
// sign-in, and the worker is blocked on the credential until it does.
let ui_hv = ui.clone();
glib::spawn_future_local(async move {
if let Ok((url, token_tx)) = hv_req_rx.recv().await {
if let Ok((hv, credential_tx)) = hv_req_rx.recv().await {
ui_hv
.login
.login_status
.set_text(&gettext("Complete the verification to continue…"));
prompt_human_verification(&ui_hv, &url, token_tx);
// Anything but a credential drops the sender, which the worker
// reads as a cancelled sign-in.
match verify_human(hv.verification_url()).await {
Verification::Solved(token) => {
let _ = credential_tx.send(HumanVerificationCredential::captcha(token));
}
Verification::Cancelled => {}
// The web view could not start, most likely its sandbox
// (B195): the user's browser can show the page instead.
Verification::Failed => verify_in_browser(&ui_hv, &hv, credential_tx),
}
}
});
// And for the mailbox password, which only a two-password account has.
// The worker asks again after a wrong answer, so this loops too.
let ui_mailbox = ui.clone();
glib::spawn_future_local(async move {
while let Ok((attempt, answer_tx)) = mailbox_req_rx.recv().await {
ask_mailbox(&ui_mailbox, attempt > 0, move |answer| {
// A cancelled dialog drops the sender, which the worker
// reads as a cancelled login.
if let Some(answer) = answer {
let _ = answer_tx.send(answer);
}
});
}
});
@@ -169,7 +278,14 @@ pub(crate) fn wire_login(ui: &Rc<Ui>) {
ui.login.login_status.set_text("");
ui.login.password.set_text("");
// Cache the new identity so `refresh` never hits the keyring.
*ui.session.borrow_mut() = auth::load().ok();
// Read on a worker: a keyring that has to be unlocked first
// answers only once the user has, and the window must not
// freeze until then.
match gio::spawn_blocking(read_session).await {
Ok(Ok(session)) => *ui.session.borrow_mut() = session,
Ok(Err(e)) => keyring_unreadable(&ui, &e),
Err(_) => {}
}
// Enable+start the mount service now that we have a session.
let _ = gio::spawn_blocking(service::enable_start).await;
refresh(&ui);
@@ -197,6 +313,11 @@ pub(crate) fn login_error_message(error: &pdfs_core::Error) -> String {
pdfs_core::Error::Proton(ProtonError::Api(api)) if api.http_status == 429 => {
gettext("Too many sign-in attempts. Wait a few minutes and try again.")
}
// Proton's text for this one asks the user to update an app that is
// already current.
pdfs_core::Error::Proton(e) if pdfs_core::error::sign_in_blocked(e) => gettext(
"Proton is temporarily refusing sign-in from this device. Your app version is not the problem. Wait several hours, up to a day, without trying again, then sign in.",
),
pdfs_core::Error::Proton(ProtonError::Api(api)) if !api.message.is_empty() => {
api.message.clone()
}
@@ -209,6 +330,12 @@ pub(crate) fn login_error_message(error: &pdfs_core::Error) -> String {
pdfs_core::Error::Other(message) if message.contains("two-factor") => {
gettext("Sign-in cancelled: no two-factor code was entered.")
}
pdfs_core::Error::Other(message) if message.contains("mailbox") => {
gettext("Sign-in cancelled: no mailbox password was entered.")
}
pdfs_core::Error::WrongMailboxPassword => {
gettext("That mailbox password didn't unlock your account.")
}
pdfs_core::Error::Other(message) if message.contains("verification") => gettext(
"Sign-in cancelled: the verification wasn't completed. Sign in again to get a new one.",
),
@@ -220,26 +347,34 @@ pub(crate) fn login_error_message(error: &pdfs_core::Error) -> String {
}
}
/// What the login worker asks the UI for, and how it reports back.
pub(crate) struct LoginChannels {
/// The final outcome, once.
pub(crate) result: async_channel::Receiver<Result<(), String>>,
/// Fires only if the account needs a 2FA code; carries the sender for it.
pub(crate) totp: async_channel::Receiver<std::sync::mpsc::Sender<String>>,
/// Fires only if the sign-in is gated: the challenge, and the sender for the
/// credential the user earns by solving it.
pub(crate) hv: async_channel::Receiver<(
HumanVerification,
std::sync::mpsc::Sender<HumanVerificationCredential>,
)>,
/// Fires only for a two-password account: how many answers were already
/// wrong, and the sender for the password and the "remember" choice.
pub(crate) mailbox: async_channel::Receiver<(u32, std::sync::mpsc::Sender<(String, bool)>)>,
}
/// Run the async SRP + optional 2FA login on a dedicated current-thread Tokio
/// runtime. Returns two channels: the first yields the final login result once;
/// the second fires *only if* the account needs a 2FA code, carrying a
/// [`std::sync::mpsc::Sender`] the UI uses to feed the code back. The login
/// closure blocks the worker on that sender until the dialog answers, so the
/// code is requested lazily and can't expire before the password proof.
#[allow(clippy::type_complexity)]
pub(crate) fn spawn_login(
username: String,
password: String,
) -> (
async_channel::Receiver<Result<(), String>>,
async_channel::Receiver<std::sync::mpsc::Sender<String>>,
async_channel::Receiver<(String, std::sync::mpsc::Sender<String>)>,
) {
/// runtime. The returned [`LoginChannels`] carry the final result and the lazy
/// prompts. Each login closure blocks the worker on a sender the UI answers, so
/// a code is requested lazily and can't expire before the password proof.
pub(crate) fn spawn_login(username: String, password: String) -> LoginChannels {
let (tx, rx) = async_channel::bounded(1);
// Two, not one: a CAPTCHA-gated login restarts, so a 2FA account is asked
// for a code on each attempt and a single slot would deadlock the worker.
let (totp_req_tx, totp_req_rx) = async_channel::bounded(2);
let (hv_req_tx, hv_req_rx) = async_channel::bounded(1);
let (mailbox_req_tx, mailbox_req_rx) = async_channel::bounded(1);
std::thread::spawn(move || {
let rt = match tokio::runtime::Builder::new_current_thread()
.enable_all()
@@ -267,19 +402,30 @@ pub(crate) fn spawn_login(
.recv()
.map_err(|_| pdfs_core::Error::Other("two-factor entry cancelled".into()))
},
|hv| {
// Gated: hand the UI the page to show, block on the token the
// user earns by solving it.
let (token_tx, token_rx) = std::sync::mpsc::channel::<String>();
hv_req_tx
.send_blocking((hv.verification_url(), token_tx))
.map_err(|_| {
pdfs_core::Error::Other("verification prompt closed".into())
})?;
let token = token_rx
|attempt| {
let (answer_tx, answer_rx) = std::sync::mpsc::channel::<(String, bool)>();
mailbox_req_tx
.send_blocking((attempt, answer_tx))
.map_err(|_| pdfs_core::Error::Other("mailbox prompt closed".into()))?;
let (password, remember) = answer_rx
.recv()
.map_err(|_| pdfs_core::Error::Other("verification cancelled".into()))?;
Ok(HumanVerificationCredential::captcha(token))
.map_err(|_| pdfs_core::Error::Other("mailbox entry cancelled".into()))?;
Ok(auth::Mailbox {
password: password.into(),
remember,
})
},
|hv| {
// Gated: hand the UI the challenge, block on the credential
// the user earns by solving it.
let (credential_tx, credential_rx) =
std::sync::mpsc::channel::<HumanVerificationCredential>();
hv_req_tx.send_blocking((hv, credential_tx)).map_err(|_| {
pdfs_core::Error::Other("verification prompt closed".into())
})?;
credential_rx
.recv()
.map_err(|_| pdfs_core::Error::Other("verification cancelled".into()))
},
)
.await
@@ -287,7 +433,143 @@ pub(crate) fn spawn_login(
});
let _ = tx.send_blocking(result);
});
(rx, totp_req_rx, hv_req_rx)
LoginChannels {
result: rx,
totp: totp_req_rx,
hv: hv_req_rx,
mailbox: mailbox_req_rx,
}
}
/// Ask for the mailbox password of a two-password account.
///
/// `done` gets the password and the "keep unlocked" choice, or `None` when the
/// dialog was cancelled. `retry` says the last answer was wrong. The choice
/// defaults to off: storing the mailbox unlock in the keyring trades away what
/// a separate mailbox password is for, so it is something to opt into.
pub(crate) fn ask_mailbox(
ui: &Rc<Ui>,
retry: bool,
done: impl FnOnce(Option<(String, bool)>) + 'static,
) {
let body = if retry {
gettext("That mailbox password didn't unlock your account. Try again.")
} else {
gettext("This account has a separate mailbox password. Enter it to unlock your files.")
};
let dialog = adw::AlertDialog::builder()
.heading(gettext("Mailbox Password"))
.body(body)
.build();
let group = adw::PreferencesGroup::new();
let entry = adw::PasswordEntryRow::builder()
.title(gettext("Mailbox password"))
.activates_default(true)
.build();
let remember = adw::SwitchRow::builder()
.title(gettext("Keep unlocked on this computer"))
.subtitle(gettext(
"Stores the unlock in the system keyring, so Proton Drive opens without asking after every login. Otherwise you enter the password once per session.",
))
.build();
group.add(&entry);
group.add(&remember);
dialog.set_extra_child(Some(&group));
dialog.add_response("cancel", &gettext("Cancel"));
dialog.add_response("unlock", &gettext("Unlock"));
dialog.set_response_appearance("unlock", adw::ResponseAppearance::Suggested);
dialog.set_default_response(Some("unlock"));
dialog.set_close_response("cancel");
let done = RefCell::new(Some(done));
dialog.connect_response(None, move |_, resp| {
let Some(done) = done.borrow_mut().take() else {
return;
};
let password = entry.text().to_string();
if resp == "unlock" && !password.is_empty() {
done(Some((password, remember.is_active())));
} else {
done(None);
}
});
let parent = ui_window(ui).map(|w| w.upcast::<gtk4::Window>());
dialog.present(parent.as_ref());
}
/// If the signed-in account is locked, ask for its mailbox password — once per
/// run; after a cancel, a toast keeps an Unlock button within reach.
///
/// The key state is a keyring and kernel-keyring read, so it runs on a worker.
pub(crate) fn offer_unlock(ui: &Rc<Ui>) {
if ui.unlock_offered.get() {
return;
}
let ui = ui.clone();
glib::spawn_future_local(async move {
let state = gio::spawn_blocking(|| auth::key_state().ok()).await;
if matches!(state, Ok(Some(auth::KeyState::Locked))) && !ui.unlock_offered.replace(true) {
prompt_unlock(&ui, false);
}
});
}
enum UnlockOutcome {
Unlocked,
Wrong,
Failed(String),
}
/// Ask for the mailbox password and unlock the account with it.
fn prompt_unlock(ui: &Rc<Ui>, retry: bool) {
let ui_done = ui.clone();
ask_mailbox(ui, retry, move |answer| {
let Some((password, remember)) = answer else {
toast_action(
&ui_done,
&gettext("Proton Drive is locked. Enter your mailbox password to unlock it."),
&gettext("Unlock"),
|ui| prompt_unlock(ui, false),
);
return;
};
glib::spawn_future_local(async move {
// Verifying talks to Proton and the restart talks to systemd, so
// both run on a worker, never on the main loop.
let outcome = gio::spawn_blocking(move || {
let rt = match tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
{
Ok(rt) => rt,
Err(e) => return UnlockOutcome::Failed(e.to_string()),
};
match rt.block_on(auth::unlock(&password, remember)) {
Ok(()) => {
service::restart();
UnlockOutcome::Unlocked
}
Err(pdfs_core::Error::WrongMailboxPassword) => UnlockOutcome::Wrong,
Err(e) => UnlockOutcome::Failed(login_error_message(&e)),
}
})
.await;
match outcome {
Ok(UnlockOutcome::Unlocked) => {
toast(&ui_done, &gettext("Unlocked."));
refresh(&ui_done);
}
Ok(UnlockOutcome::Wrong) => prompt_unlock(&ui_done, true),
Ok(UnlockOutcome::Failed(message)) => {
toast_error(&ui_done, &gettext("Couldn't unlock"), &message)
}
Err(_) => {}
}
});
});
}
/// Show the lazy two-factor dialog and feed the entered code back to the waiting
@@ -330,6 +612,63 @@ pub(crate) fn prompt_2fa(ui: &Rc<Ui>, code_tx: std::sync::mpsc::Sender<String>)
dialog.present(parent.as_ref());
}
/// Let the user solve the challenge in their own browser, for when the
/// verification window could not come up. The page cannot hand its token back
/// to us, so the user says when they are done, and the retry carries the
/// challenge token, which the solved page made good (see
/// [`auth::browser_verified`]).
fn verify_in_browser(
ui: &Rc<Ui>,
hv: &HumanVerification,
credential_tx: std::sync::mpsc::Sender<HumanVerificationCredential>,
) {
let dialog = adw::AlertDialog::builder()
.heading(gettext("Verify in Your Browser"))
.body(gettext(
"The verification page can't be shown in the app on this system. Open it in your browser, complete it there, then come back and choose Continue.",
))
.build();
let url = auth::browser_verification_url(hv);
let open = gtk4::Button::builder()
.label(gettext("Open Verification Page"))
.halign(gtk4::Align::Center)
.build();
open.add_css_class("pill");
dialog.set_extra_child(Some(&open));
dialog.add_response("cancel", &gettext("Cancel"));
dialog.add_response("continue", &gettext("Continue"));
dialog.set_response_appearance("continue", adw::ResponseAppearance::Suggested);
// Continuing before the page was even opened would only fail the retry.
dialog.set_response_enabled("continue", false);
dialog.set_default_response(Some("continue"));
dialog.set_close_response("cancel");
let dialog_weak = dialog.downgrade();
open.connect_clicked(move |_| {
open_uri(&url);
if let Some(dialog) = dialog_weak.upgrade() {
dialog.set_response_enabled("continue", true);
}
});
let credential = auth::browser_verified(hv);
let credential_tx = RefCell::new(Some(credential_tx));
dialog.connect_response(None, move |_, resp| {
// On cancel/close we take + drop `tx` without sending, so the worker's
// recv errors out and the login is reported as cancelled.
if let Some(tx) = credential_tx.borrow_mut().take()
&& resp == "continue"
{
let _ = tx.send(credential.clone());
}
});
let parent = ui.login.login_button.root().and_downcast::<gtk4::Window>();
dialog.present(parent.as_ref());
}
/// Sign out, after a confirmation: disable+stop the mount service (so the
/// daemon isn't left running without credentials), forget the stored session,
/// and drop back to the login page.
@@ -349,7 +688,7 @@ pub(crate) fn sign_out(ui: &Rc<Ui>) {
glib::spawn_future_local(async move {
let result = gio::spawn_blocking(|| {
service::disable_stop();
auth::logout().map_err(|e| e.to_string())
auth::logout().map(|_| ()).map_err(|e| e.to_string())
})
.await;
if let Ok(Err(e)) = result {
@@ -374,6 +713,27 @@ mod tests {
assert!(login_error_message(&hv).contains("verification wasn't completed"));
}
#[test]
fn a_cancelled_mailbox_prompt_is_reported_as_such() {
let error = pdfs_core::Error::Other("mailbox entry cancelled".into());
assert!(login_error_message(&error).contains("no mailbox password"));
}
#[test]
fn a_blocked_sign_in_does_not_ask_for_an_update() {
use pdfs_core::proton_sdk::api::ResponseCode;
use pdfs_core::proton_sdk::{ProtonApiError, ProtonError};
let error = pdfs_core::Error::Proton(ProtonError::Api(ProtonApiError {
code: ResponseCode::Blocked,
http_status: 422,
message: "This version of the app is no longer supported, please update to continue using the app".into(),
details: None,
}));
let message = login_error_message(&error);
assert!(message.contains("temporarily refusing"), "{message}");
assert!(!message.contains("update"), "{message}");
}
#[test]
fn an_unmapped_error_keeps_its_text() {
let error = pdfs_core::Error::Other("something odd".into());
+168 -47
View File
@@ -275,14 +275,42 @@ pub(crate) fn load_photo(ui: &Rc<Ui>, viewer: &Rc<Viewer>, uid: String) {
viewer.status.set_label(message);
viewer.status.set_visible(true);
};
match result {
Ok(Ok(Response::FilePath { path })) if is_video => {
let path = match result {
Ok(Ok(Response::FilePath { path })) => path,
Ok(Ok(Response::Error { message, .. })) => return fail(&message),
Ok(Ok(_)) => return fail(&gettext("Unexpected reply from the Proton Drive service.")),
Ok(Err(_)) | Err(_) => return fail(&gettext("Couldn't reach Proton Drive.")),
};
// Decoding a full-size photo and reading its EXIF take a good part of
// a second for a large one, which froze the whole app on the main
// thread. Done on a worker; a reply for a photo the user has since
// left is dropped like the request's own.
let loaded = {
let path = path.clone();
gio::spawn_blocking(move || {
let size = std::fs::metadata(&path).ok().map(|meta| meta.len());
if is_video {
return (size, None, ExifInfo::default());
}
let texture = gtk4::gdk::Texture::from_filename(&path);
(size, Some(texture), read_exif(&path))
})
.await
};
if *viewer.uid.borrow() != uid {
return;
}
let Ok((size, texture, info)) = loaded else {
return fail(&gettext("Couldn't render this photo."));
};
match texture {
None => {
viewer.video.set_filename(Some(&path));
viewer.media.set_visible_child_name("video");
*viewer.path.borrow_mut() = Some(path.clone());
show_info(&viewer, &path, ExifInfo::default());
show_info(&viewer, size, ExifInfo::default());
}
Ok(Ok(Response::FilePath { path })) => match gtk4::gdk::Texture::from_filename(&path) {
Some(texture) => match texture {
Ok(texture) => {
// Zoomed in on the thumbnail: keep the photo the size it is
// on screen, now drawn from the full-size pixels.
@@ -297,16 +325,13 @@ pub(crate) fn load_photo(ui: &Rc<Ui>, viewer: &Rc<Viewer>, uid: String) {
zoom_to(&viewer, Some(zoom), None);
}
*viewer.path.borrow_mut() = Some(path.clone());
show_info(&viewer, &path, read_exif(&path));
show_info(&viewer, size, info);
}
Err(e) => {
tracing::error!("Failed to load texture for {path}: {e}");
fail(&gettext("Couldn't render this photo."));
}
},
Ok(Ok(Response::Error { message, .. })) => fail(&message),
Ok(Ok(_)) => fail(&gettext("Unexpected reply from the Proton Drive service.")),
Ok(Err(_)) | Err(_) => fail(&gettext("Couldn't reach Proton Drive.")),
}
});
}
@@ -452,16 +477,16 @@ pub(crate) fn clear_info(viewer: &Rc<Viewer>) {
}
/// Fill the details drawer for the photo now on screen: its own file facts
/// (size, dimensions) plus whatever EXIF it carries.
pub(crate) fn show_info(viewer: &Rc<Viewer>, path: &str, info: ExifInfo) {
/// (`size`, dimensions) plus whatever EXIF it carries.
pub(crate) fn show_info(viewer: &Rc<Viewer>, size: Option<u64>, info: ExifInfo) {
while let Some(row) = viewer.info_rows.first_child() {
viewer.info_rows.remove(&row);
}
let mut fields: Vec<(String, String)> = Vec::new();
if let Ok(meta) = std::fs::metadata(path) {
if let Some(size) = size {
// Translators: label of a photo's file size in the details panel.
fields.push((pgettext("property", "Size"), human_bytes(meta.len())));
fields.push((pgettext("property", "Size"), human_bytes(size)));
}
fields.extend(info.fields);
@@ -481,6 +506,7 @@ pub(crate) fn show_info(viewer: &Rc<Viewer>, path: &str, info: ExifInfo) {
let row = adw::ActionRow::builder()
.title(label.as_str())
.subtitle(value)
.use_markup(false)
.subtitle_selectable(true)
.build();
row.add_css_class("property");
@@ -1259,8 +1285,11 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
// Any real pointer movement brings the controls back; resting over one of
// them keeps them up.
let motion = gtk4::EventControllerMotion::new();
let viewer_motion = viewer.clone();
let viewer_motion = Rc::downgrade(&viewer);
motion.connect_motion(move |_, x, y| {
let Some(viewer_motion) = viewer_motion.upgrade() else {
return;
};
let (last_x, last_y) = viewer_motion.pointer.get();
viewer_motion.pointer.set((x, y));
if (x - last_x).abs() > 1.0 || (y - last_y).abs() > 1.0 {
@@ -1270,10 +1299,17 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
body.add_controller(motion);
for revealer in [&top_reveal, &prev_reveal, &next_reveal] {
let hover = gtk4::EventControllerMotion::new();
let viewer_enter = viewer.clone();
hover.connect_enter(move |_, _, _| viewer_enter.chrome_hover.set(true));
let viewer_leave = viewer.clone();
let viewer_enter = Rc::downgrade(&viewer);
hover.connect_enter(move |_, _, _| {
if let Some(viewer) = viewer_enter.upgrade() {
viewer.chrome_hover.set(true);
}
});
let viewer_leave = Rc::downgrade(&viewer);
hover.connect_leave(move |_| {
let Some(viewer_leave) = viewer_leave.upgrade() else {
return;
};
viewer_leave.chrome_hover.set(false);
show_chrome(&viewer_leave);
});
@@ -1283,8 +1319,11 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
// Ctrl+scroll zooms around the pointer; a plain scroll pans a zoomed photo.
let wheel = gtk4::EventControllerScroll::new(gtk4::EventControllerScrollFlags::VERTICAL);
wheel.set_propagation_phase(gtk4::PropagationPhase::Capture);
let viewer_wheel = viewer.clone();
let viewer_wheel = Rc::downgrade(&viewer);
wheel.connect_scroll(move |controller, _, dy| {
let Some(viewer_wheel) = viewer_wheel.upgrade() else {
return glib::Propagation::Proceed;
};
if !controller
.current_event_state()
.contains(gtk4::gdk::ModifierType::CONTROL_MASK)
@@ -1304,14 +1343,20 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
scroller.add_controller(wheel);
let pinch = gtk4::GestureZoom::new();
let viewer_pinch = viewer.clone();
let viewer_pinch = Rc::downgrade(&viewer);
pinch.connect_begin(move |_, _| {
let Some(viewer_pinch) = viewer_pinch.upgrade() else {
return;
};
viewer_pinch
.pinch_start
.set(current_scale(&viewer_pinch).unwrap_or(1.0));
});
let viewer_pinch = viewer.clone();
let viewer_pinch = Rc::downgrade(&viewer);
pinch.connect_scale_changed(move |gesture, delta| {
let Some(viewer_pinch) = viewer_pinch.upgrade() else {
return;
};
let start = viewer_pinch.pinch_start.get();
zoom_to(
&viewer_pinch,
@@ -1323,8 +1368,11 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
// Dragging a zoomed photo pans it.
let drag = gtk4::GestureDrag::new();
let viewer_drag = viewer.clone();
let viewer_drag = Rc::downgrade(&viewer);
drag.connect_drag_begin(move |_, _, _| {
let Some(viewer_drag) = viewer_drag.upgrade() else {
return;
};
let scroller = &viewer_drag.scroller;
viewer_drag.drag_start.set((
scroller.hadjustment().value(),
@@ -1334,8 +1382,11 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
viewer_drag.picture.set_cursor_from_name(Some("grabbing"));
}
});
let viewer_drag = viewer.clone();
let viewer_drag = Rc::downgrade(&viewer);
drag.connect_drag_update(move |_, dx, dy| {
let Some(viewer_drag) = viewer_drag.upgrade() else {
return;
};
if viewer_drag.zoom.get().is_none() {
return;
}
@@ -1343,39 +1394,58 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
viewer_drag.scroller.hadjustment().set_value(x - dx);
viewer_drag.scroller.vadjustment().set_value(y - dy);
});
let viewer_drag = viewer.clone();
let viewer_drag = Rc::downgrade(&viewer);
drag.connect_drag_end(move |_, _, _| {
let Some(viewer_drag) = viewer_drag.upgrade() else {
return;
};
if viewer_drag.zoom.get().is_some() {
viewer_drag.picture.set_cursor_from_name(Some("grab"));
}
});
scroller.add_controller(drag);
let w_close = window.clone();
let w_close = window.downgrade();
close_btn.connect_clicked(move |_| {
let Some(w_close) = w_close.upgrade() else {
return;
};
w_close.close();
});
let viewer_closing = viewer.clone();
// The one strong handle on the viewer. Every handler holds a weak one: a
// strong one in a handler of a widget the viewer holds is a cycle, which
// kept every closed lightbox, its window and its full-size photo in memory
// for the life of the app.
let keep = RefCell::new(Some(viewer.clone()));
window.connect_close_request(move |_| {
stop_video(&viewer_closing);
disarm_slide(&viewer_closing);
if let Some(source) = viewer_closing.chrome_source.borrow_mut().take() {
source.remove();
if let Some(viewer) = keep.borrow_mut().take() {
stop_video(&viewer);
disarm_slide(&viewer);
if let Some(source) = viewer.chrome_source.borrow_mut().take() {
source.remove();
}
viewer.picture.set_paintable(gtk4::gdk::Paintable::NONE);
}
glib::Propagation::Proceed
});
let viewer_info = viewer.clone();
let viewer_info = Rc::downgrade(&viewer);
info_toggle.connect_toggled(move |toggle| {
let Some(viewer_info) = viewer_info.upgrade() else {
return;
};
viewer_info
.info_revealer
.set_reveal_child(toggle.is_active());
});
let ui_slides = ui.clone();
let viewer_slides = viewer.clone();
let viewer_slides = Rc::downgrade(&viewer);
slideshow_btn.connect_toggled(move |toggle| {
let Some(viewer_slides) = viewer_slides.upgrade() else {
return;
};
if toggle.is_active() {
start_slideshow(&ui_slides, &viewer_slides);
} else {
@@ -1387,9 +1457,15 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
info_close.connect_clicked(move |_| toggle_off.set_active(false));
let ui_map = ui.clone();
let viewer_map = viewer.clone();
let w_map = window.clone();
let viewer_map = Rc::downgrade(&viewer);
let w_map = window.downgrade();
info_map.connect_clicked(move |_| {
let Some(w_map) = w_map.upgrade() else {
return;
};
let Some(viewer_map) = viewer_map.upgrade() else {
return;
};
let coords = *viewer_map.coords.borrow();
if let Some((lat, lon)) = coords {
w_map.close();
@@ -1398,8 +1474,11 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
});
let ui_fav = ui.clone();
let viewer_fav = viewer.clone();
let viewer_fav = Rc::downgrade(&viewer);
favorite_btn.connect_toggled(move |btn| {
let Some(viewer_fav) = viewer_fav.upgrade() else {
return;
};
if viewer_fav.favorite_suppress.get() {
return;
}
@@ -1443,17 +1522,26 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
// gone, and leaving it open on a trashed photo invites a second Delete on
// something the user cannot see.
let ui_delete = ui.clone();
let viewer_delete = viewer.clone();
let w_delete = window.clone();
let viewer_delete = Rc::downgrade(&viewer);
let w_delete = window.downgrade();
delete_btn.connect_clicked(move |_| {
let Some(w_delete) = w_delete.upgrade() else {
return;
};
let Some(viewer_delete) = viewer_delete.upgrade() else {
return;
};
let uid = viewer_delete.uid.borrow().clone();
w_delete.close();
trash_photos(&ui_delete, vec![uid], false);
});
let ui_group = ui.clone();
let viewer_group = viewer.clone();
let viewer_group = Rc::downgrade(&viewer);
group_btn.connect_clicked(move |_| {
let Some(viewer_group) = viewer_group.upgrade() else {
return;
};
let uid = viewer_group.uid.borrow().clone();
let next =
next_group_member(&viewer_group.group.borrow(), &uid).map(|item| item.uid.clone());
@@ -1464,9 +1552,15 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
});
let ui_download = ui.clone();
let w_download = window.clone();
let viewer_download = viewer.clone();
let w_download = window.downgrade();
let viewer_download = Rc::downgrade(&viewer);
download_btn.connect_clicked(move |_| {
let Some(w_download) = w_download.upgrade() else {
return;
};
let Some(viewer_download) = viewer_download.upgrade() else {
return;
};
if let Some(path) = viewer_download.path.borrow().as_deref() {
let name = viewer_download.name.borrow();
save_photo_to_disk(
@@ -1478,23 +1572,35 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
}
});
let w_ext = window.clone();
let viewer_ext = viewer.clone();
let w_ext = window.downgrade();
let viewer_ext = Rc::downgrade(&viewer);
open_ext_btn.connect_clicked(move |_| {
let Some(w_ext) = w_ext.upgrade() else {
return;
};
let Some(viewer_ext) = viewer_ext.upgrade() else {
return;
};
if let Some(path) = viewer_ext.path.borrow().as_deref() {
open_with(&w_ext, path, viewer_ext.name.borrow().as_deref());
}
});
let ui_prev = ui.clone();
let viewer_prev = viewer.clone();
let viewer_prev = Rc::downgrade(&viewer);
prev_btn.connect_clicked(move |_| {
let Some(viewer_prev) = viewer_prev.upgrade() else {
return;
};
navigate_photo(&ui_prev, &viewer_prev, -1);
});
let ui_next = ui.clone();
let viewer_next = viewer.clone();
let viewer_next = Rc::downgrade(&viewer);
next_btn.connect_clicked(move |_| {
let Some(viewer_next) = viewer_next.upgrade() else {
return;
};
navigate_photo(&ui_next, &viewer_next, 1);
});
@@ -1503,8 +1609,11 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
// reaching for the image doesn't fling the window shut; a zoomed photo has
// no backdrop to speak of. Double-click toggles between fitted and 100 %.
let backdrop = gtk4::GestureClick::new();
let viewer_click = viewer.clone();
let viewer_click = Rc::downgrade(&viewer);
backdrop.connect_pressed(move |_, n_press, x, y| {
let Some(viewer_click) = viewer_click.upgrade() else {
return;
};
if n_press != 2 {
return;
}
@@ -1521,9 +1630,15 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
zoom_to(&viewer_click, Some((fit * 2.0).max(1.0)), Some(anchor));
}
});
let viewer_click = viewer.clone();
let w_click = window.clone();
let viewer_click = Rc::downgrade(&viewer);
let w_click = window.downgrade();
backdrop.connect_released(move |_, n_press, x, y| {
let Some(w_click) = w_click.upgrade() else {
return;
};
let Some(viewer_click) = viewer_click.upgrade() else {
return;
};
if n_press == 1
&& viewer_click.zoom.get().is_none()
&& !over_photo(&viewer_click.picture, x, y)
@@ -1535,9 +1650,15 @@ pub(crate) fn open_photo_viewer(ui: &Rc<Ui>, initial_uid: String) {
let key_controller = gtk4::EventControllerKey::new();
let ui_key = ui.clone();
let viewer_key = viewer.clone();
let w_key = window.clone();
let viewer_key = Rc::downgrade(&viewer);
let w_key = window.downgrade();
key_controller.connect_key_pressed(move |_, key, _keycode, state| {
let Some(w_key) = w_key.upgrade() else {
return glib::Propagation::Proceed;
};
let Some(viewer_key) = viewer_key.upgrade() else {
return glib::Propagation::Proceed;
};
let ctrl = state.contains(gtk4::gdk::ModifierType::CONTROL_MASK);
let playing_video = viewer_key.media.visible_child_name().as_deref() == Some("video");
match key.name().as_deref() {
+19 -10
View File
@@ -140,6 +140,9 @@ pub(crate) struct GalleryState {
pub(crate) loading: Cell<bool>,
/// Whether the last page came back full, so there may be more to load.
pub(crate) has_more: Cell<bool>,
/// Whether the account has a photo library. Until the daemon says it has
/// none, assume it does, so Upload is not greyed out while the page loads.
pub(crate) library: Cell<bool>,
/// Run once the page in flight has landed in [`Self::model`] — how the
/// lightbox steps past the last loaded photo.
pub(crate) page_waiters: RefCell<Vec<Box<dyn FnOnce()>>>,
@@ -606,7 +609,7 @@ pub(crate) fn build_gallery_page() -> (gtk4::Widget, GalleryWidgets) {
select_box.append(&select_done);
let select_bar = gtk4::Revealer::builder()
.transition_type(gtk4::RevealerTransitionType::SlideDown)
.child(&select_box)
.child(&scroll_sideways(&select_box))
.build();
// All / Photos / Videos / Raw filter. Linked toggles acting as one segmented
@@ -1376,11 +1379,9 @@ pub(crate) fn wire_gallery(
"tiff" | "tif" => "image/tiff",
_ => "image/jpeg",
};
let capture_time = std::fs::metadata(&path)
.and_then(|m| m.modified())
.ok()
.and_then(|t| t.duration_since(std::time::SystemTime::UNIX_EPOCH).ok())
.map(|d| d.as_secs() as i64);
// The daemon dates the photo: by its EXIF, or by the file's
// modification time when it has none.
let capture_time = None;
ui.busy_begin();
let rx = spawn_request(
@@ -3341,6 +3342,12 @@ pub(crate) fn load_gallery(ui: &Rc<Ui>, append: bool) {
items,
counts,
})) => {
// Nothing can go into a library the account does not have; the
// SDK refuses the upload before sending it (issue #38).
ui.gallery.library.set(available);
ui.gallery
.upload
.set_sensitive(available && *ui.mounted.borrow());
if !available {
gallery_status(
&ui,
@@ -3450,7 +3457,9 @@ pub(crate) fn load_gallery(ui: &Rc<Ui>, append: bool) {
pub(crate) fn gallery_status(ui: &Rc<Ui>, icon: &str, title: &str, description: &str, retry: bool) {
ui.gallery.status.set_icon_name(Some(icon));
ui.gallery.status.set_title(title);
ui.gallery.status.set_description(Some(description));
ui.gallery
.status
.set_description(Some(&gtk4::glib::markup_escape_text(description)));
ui.gallery.retry.set_visible(retry);
// Only the empty timeline offers a way to fill it; every other status turns
// those buttons back off.
@@ -3470,10 +3479,10 @@ pub(crate) fn gallery_unreachable(ui: &Rc<Ui>) {
/// right tool for the HEVC `.mkv`s this is aimed at — and falls back to the
/// user's default handler when mpv isn't installed.
pub(crate) fn play_external(path: &str) {
if Command::new("mpv").arg(path).spawn().is_ok() {
return;
match Command::new("mpv").arg(path).spawn() {
Ok(child) => pdfs_core::opener::reap(child),
Err(_) => open_path(path),
}
open_path(path);
}
#[cfg(test)]
+3 -1
View File
@@ -448,7 +448,9 @@ pub(crate) fn close_place(ui: &Rc<Ui>) {
fn places_status(ui: &Rc<Ui>, icon: &str, title: &str, description: &str) {
ui.gallery.places_status.set_icon_name(Some(icon));
ui.gallery.places_status.set_title(title);
ui.gallery.places_status.set_description(Some(description));
ui.gallery
.places_status
.set_description(Some(&gtk4::glib::markup_escape_text(description)));
ui.gallery.places_stack.set_visible_child_name("status");
}
+7 -10
View File
@@ -311,7 +311,9 @@ pub(crate) fn wire_shared(ui: &Rc<Ui>, widgets: &SharedWidgets) {
pub(crate) fn shared_status(ui: &Rc<Ui>, icon: &str, title: &str, description: &str, retry: bool) {
ui.shared.status.set_icon_name(Some(icon));
ui.shared.status.set_title(title);
ui.shared.status.set_description(Some(description));
ui.shared
.status
.set_description(Some(&gtk4::glib::markup_escape_text(description)));
ui.shared.retry.set_visible(retry);
ui.shared.content.set_visible_child_name("status");
}
@@ -484,9 +486,7 @@ fn repaint_shared_crumb(ui: &Rc<Ui>, nav: &[(String, String)]) {
nav.is_empty(),
));
for (i, (_, name)) in nav.iter().enumerate() {
let sep = gtk4::Label::new(Some("›"));
sep.add_css_class("dim-label");
crumb.append(&sep);
crumb.append(&crumb_separator());
crumb.append(&shared_crumb_node(ui, name, i + 1, i + 1 == nav.len()));
}
}
@@ -495,12 +495,7 @@ fn repaint_shared_crumb(ui: &Rc<Ui>, nav: &[(String, String)]) {
/// that goes back up to `depth` folders below the top level.
fn shared_crumb_node(ui: &Rc<Ui>, label: &str, depth: usize, current: bool) -> gtk4::Widget {
if current {
let l = gtk4::Label::builder()
.label(label)
.ellipsize(gtk4::pango::EllipsizeMode::Start)
.build();
l.add_css_class("heading");
return l.upcast();
return crumb_current(label);
}
let button = gtk4::Button::builder().label(label).build();
button.add_css_class("flat");
@@ -825,6 +820,7 @@ fn invitation_row(inv: &InvitationInfo) -> adw::ActionRow {
.title(&item)
// Translators: {email} is the address of the person who sent the invitation.
.subtitle(gettext_f("From {email}", &[("email", &inv.inviter_email)]))
.use_markup(false)
.build();
row.add_prefix(&gtk4::Image::from_icon_name(if inv.is_dir {
"folder-symbolic"
@@ -869,6 +865,7 @@ fn repaint_links(ui: &Rc<Ui>, bookmarks: &[BookmarkInfo]) {
let row = adw::ActionRow::builder()
.title(&title)
.subtitle(&bm.url)
.use_markup(false)
.subtitle_lines(1)
.activatable(true)
.tooltip_text(gettext("Open in browser"))
+19 -1
View File
@@ -329,7 +329,9 @@ pub(crate) fn shared_by_me_status(
) {
ui.shared_by_me.status.set_icon_name(Some(icon));
ui.shared_by_me.status.set_title(title);
ui.shared_by_me.status.set_description(Some(description));
ui.shared_by_me
.status
.set_description(Some(&gtk4::glib::markup_escape_text(description)));
ui.shared_by_me.retry.set_visible(retry);
ui.shared_by_me.content.set_visible_child_name("status");
ui.shared_by_me.subtitle.set_subtitle("");
@@ -565,6 +567,7 @@ pub(crate) fn prompt_stop_sharing(ui: &Rc<Ui>, entries: &[DirEntry]) {
let uids: Vec<String> = entries.iter().map(|e| e.uid.clone()).collect();
dialog.connect_response(Some("stop"), move |_, _| {
let done = done.clone();
let stopped = uids.clone();
run_requests(
&ui_c,
uids.iter()
@@ -572,6 +575,7 @@ pub(crate) fn prompt_stop_sharing(ui: &Rc<Ui>, entries: &[DirEntry]) {
.collect(),
gettext_noop("Couldn't stop sharing"),
move |ui| {
forget_shared_by_me(ui, &stopped);
reload_listing(ui);
toast(ui, &done);
},
@@ -580,6 +584,18 @@ pub(crate) fn prompt_stop_sharing(ui: &Rc<Ui>, entries: &[DirEntry]) {
dialog.present(ui_window(ui).as_ref());
}
/// Take items whose sharing just stopped off the page now, rather than when
/// the reload that confirms it comes back: that reload asks after every share
/// in turn and takes a while with many of them.
fn forget_shared_by_me(ui: &Rc<Ui>, uids: &[String]) {
let items = {
let mut items = ui.shared_by_me.items.borrow_mut();
items.retain(|item| !uids.contains(&item.uid));
items.clone()
};
repaint_shared_by_me(ui, &items);
}
/// Open a shared item: a folder in My Files, a file the way My Files would.
fn open_shared_by_me(ui: &Rc<Ui>, entry: &DirEntry) {
if entry.is_dir {
@@ -608,6 +624,7 @@ pub(crate) fn shared_item_as_entry(item: &SharedItem) -> DirEntry {
size: 0,
modified: item.modified,
pinned: false,
pinned_by_folder: false,
cached: false,
uid: item.uid.clone(),
path: item.path.clone(),
@@ -618,6 +635,7 @@ pub(crate) fn shared_item_as_entry(item: &SharedItem) -> DirEntry {
shared_by_unverified: false,
trashed_at: 0,
trashed_from: None,
issue: None,
}
}
+142 -17
View File
@@ -52,6 +52,9 @@ pub(crate) struct StatusState {
pub(crate) online_map_row: adw::SwitchRow,
/// "Show tray icon" toggle. Guarded by [`Self::settings_suppress`].
pub(crate) tray_row: adw::SwitchRow,
/// "Check for updates" toggle, not shown where a package manager brings
/// updates. Guarded by [`Self::settings_suppress`].
pub(crate) update_row: adw::SwitchRow,
/// Language picker: "System Default", then [`i18n::LANGUAGES`]. Guarded by
/// [`Self::settings_suppress`].
pub(crate) language_row: adw::ComboRow,
@@ -164,6 +167,7 @@ pub(crate) struct MainWidgets {
pub(crate) theme_row: adw::SwitchRow,
pub(crate) online_map_row: adw::SwitchRow,
pub(crate) tray_row: adw::SwitchRow,
pub(crate) update_row: adw::SwitchRow,
pub(crate) language_row: adw::ComboRow,
}
@@ -250,6 +254,17 @@ pub(crate) fn build_main_page() -> MainWidgets {
let language_row = build_language_row();
appearance_group.add(&language_row);
let updates_group = adw::PreferencesGroup::builder()
.title(gettext("Updates"))
.build();
let update_row = adw::SwitchRow::builder()
.title(gettext("Check for updates"))
.subtitle(gettext(
"Ask GitHub once a day whether a new version is out. GitHub learns your IP address and which version you use.",
))
.build();
updates_group.add(&update_row);
let general = adw::PreferencesPage::builder()
.title(gettext("General"))
.icon_name("preferences-system-symbolic")
@@ -258,6 +273,10 @@ pub(crate) fn build_main_page() -> MainWidgets {
general.add(&location_group);
general.add(&network_group);
general.add(&appearance_group);
// A package manager brings new versions, so there is nothing to tell.
if !updates_managed_here() {
general.add(&updates_group);
}
// ---- Preferences: Storage
let storage_group = adw::PreferencesGroup::builder()
@@ -381,23 +400,34 @@ pub(crate) fn build_main_page() -> MainWidgets {
.hexpand(true)
.ellipsize(gtk4::pango::EllipsizeMode::Middle)
.build();
// The app's one menu, on the account row the way Proton's own apps keep it:
// app-wide entries first, the account's last. The whole row opens it, and
// F10 too, as the primary menu.
let app_section = gio::Menu::new();
app_section.append(Some(&gettext("Preferences")), Some("win.preferences"));
app_section.append(Some(&gettext("Keyboard Shortcuts")), Some("win.shortcuts"));
app_section.append(
Some(&gettext("About Proton Drive for Linux")),
Some("win.about"),
);
let account_section = gio::Menu::new();
account_section.append(Some(&gettext("Sign Out…")), Some("win.sign-out"));
let account_menu = gio::Menu::new();
account_menu.append(Some(&gettext("Preferences")), Some("win.preferences"));
let sign_out = gio::Menu::new();
sign_out.append(Some(&gettext("Sign Out…")), Some("win.sign-out"));
account_menu.append_section(None, &sign_out);
let account_button = gtk4::MenuButton::builder()
.icon_name("view-more-symbolic")
.tooltip_text(gettext("Account"))
account_menu.append_section(None, &app_section);
account_menu.append_section(None, &account_section);
let row = gtk4::Box::new(gtk4::Orientation::Horizontal, 10);
row.append(&avatar);
row.append(&account_name);
row.append(&gtk4::Image::from_icon_name("view-more-symbolic"));
let account = gtk4::MenuButton::builder()
.child(&row)
.tooltip_text(gettext("Main menu"))
.menu_model(&account_menu)
.valign(gtk4::Align::Center)
.direction(gtk4::ArrowType::Up)
.primary(true)
.build();
account_button.add_css_class("flat");
let account = gtk4::Box::new(gtk4::Orientation::Horizontal, 10);
account.add_css_class("flat");
account.add_css_class("sidebar-account");
account.append(&avatar);
account.append(&account_name);
account.append(&account_button);
let footer = gtk4::Box::new(gtk4::Orientation::Vertical, 10);
footer.add_css_class("sidebar-footer");
@@ -431,6 +461,7 @@ pub(crate) fn build_main_page() -> MainWidgets {
theme_row,
online_map_row,
tray_row,
update_row,
language_row,
}
}
@@ -530,6 +561,7 @@ pub(crate) fn wire_settings(
.set_active(config.proton_theme.unwrap_or(false));
ui.status.online_map_row.set_active(config.online_map);
ui.status.tray_row.set_active(!config.tray_hidden);
ui.status.update_row.set_active(config.update_check);
ui.status
.language_row
.set_selected(language_index(config.language.as_deref()));
@@ -679,6 +711,26 @@ pub(crate) fn wire_settings(
}
});
// Update check: saved, and the first check runs right away if one is due.
let ui_update = ui.clone();
ui.status.update_row.connect_active_notify(move |row| {
if ui_update.status.settings_suppress.get() {
return;
}
let mut config = ui_update.dirs.load_config();
config.update_check = row.is_active();
if let Err(e) = ui_update.dirs.save_config(&config) {
toast_error(
&ui_update,
&gettext("Couldn't save the update setting"),
&e.to_string(),
);
return;
}
show_latest(&ui_update);
start_update_checks(&ui_update);
});
// Tray icon: the tray reads the same flag at start, so hiding it here also
// keeps the login autostart from bringing it back.
let ui_tray = ui.clone();
@@ -825,6 +877,13 @@ pub(crate) fn wire_retry(ui: &Rc<Ui>) {
.connect_clicked(move |_| restart_service_then(&ui_trash, load_trash));
}
/// Show the sign-in page while a session is still stored, because Proton has
/// refused it. The sidebar stays, so the user can go back to cached files; a
/// login moves on to the files as from any sign-in ([`refresh`]).
pub(crate) fn show_login(ui: &Rc<Ui>) {
ui.stack.set_visible_child_name("login");
}
/// Repaint the window from the cached login identity, then ask the daemon for
/// its status and transfers. Runs at startup, after a login and after a service
/// restart; the daemon's event feed keeps both current in between. The identity
@@ -842,7 +901,7 @@ pub(crate) fn refresh(ui: &Rc<Ui>) {
if ui.stack.visible_child_name().as_deref() == Some("login") {
ui.stack.set_visible_child_name("browser");
}
ui.nav.set_collapsed(false);
ui.nav.set_collapsed(ui.narrow.get());
if ui.status.account_name.label() != s.username {
ui.status.account_name.set_label(&s.username);
ui.status.account_name.set_tooltip_text(Some(&s.username));
@@ -860,6 +919,7 @@ pub(crate) fn refresh(ui: &Rc<Ui>) {
}
}
offer_unlock(ui);
refresh_status(ui);
refresh_transfers(ui);
}
@@ -939,7 +999,9 @@ fn quota_display(max_space: i64, used_space: i64) -> (f64, String) {
pub(crate) fn set_mounted(ui: &Rc<Ui>, mounted: bool) {
*ui.mounted.borrow_mut() = mounted;
sync_mounted_actions(ui, mounted);
ui.gallery.upload.set_sensitive(mounted);
ui.gallery
.upload
.set_sensitive(mounted && ui.gallery.library.get());
ui.details.details.pin_row.set_sensitive(mounted);
ui.details.details.rename_button.set_sensitive(mounted);
ui.details.details.trash_button.set_sensitive(mounted);
@@ -1209,6 +1271,9 @@ pub(crate) fn paint_status(ui: &Rc<Ui>, status: Response) {
failing_error,
paused,
paused_until,
session_expired,
tokens_unsaved,
version,
..
} = status
else {
@@ -1219,7 +1284,13 @@ pub(crate) fn paint_status(ui: &Rc<Ui>, status: Response) {
// in it: it is why a file that looks saved is not on the remote
// yet, and offline is usually the reason it is still queued.
let queued = i18n::pending_summary(pending_uploads, pending_changes);
let state = if paused {
let state = if session_expired {
SyncState::SessionExpired { queued }
} else if tokens_unsaved {
SyncState::TokensUnsaved
} else if pdfs_core::control::daemon_outdated(&version) {
SyncState::Outdated { running: version }
} else if paused {
SyncState::Paused {
until: paused_until,
queued,
@@ -1282,6 +1353,20 @@ pub(crate) fn paint_disconnected(ui: &Rc<Ui>) {
/// What the sidebar's status strip reports, most urgent first.
pub(crate) enum SyncState {
/// Proton refused the session: nothing reaches it until the user signs in
/// again. Outranks a pause, which it makes moot.
SessionExpired {
queued: Option<String>,
},
/// The keyring refuses the session's refreshed tokens. Sync goes on, but
/// a reboot before the keyring takes them signs the user out.
TokensUnsaved,
/// The daemon is an older release than this window: an upgrade replaced
/// the files but left the service running, with the old release's bugs,
/// until it restarts (#33). `running` is empty when it is too old to say.
Outdated {
running: String,
},
/// The user paused syncing; nothing goes up until it resumes. Outranks
/// everything else, because it is the reason for everything else.
Paused {
@@ -1316,7 +1401,44 @@ pub(crate) enum SyncState {
fn paint_sync_status(ui: &Rc<Ui>, state: SyncState) {
let paused = matches!(state, SyncState::Paused { .. });
let connected = !matches!(state, SyncState::Disconnected | SyncState::NotResponding);
let expired = matches!(state, SyncState::SessionExpired { .. });
let outdated = matches!(state, SyncState::Outdated { .. });
let (icon, class, title, detail) = match state {
SyncState::SessionExpired { queued } => (
"dialog-password-symbolic",
Some("error"),
gettext("Session expired"),
Some(match queued {
// Translators: {queued} says what is waiting, such as "3 uploads
// queued".
Some(queued) => {
gettext_f("{queued} · sign in again to sync", &[("queued", &queued)])
}
None => gettext("Sign in again to sync"),
}),
),
SyncState::TokensUnsaved => (
"dialog-warning-symbolic",
Some("warning"),
gettext("Sign-in not saved"),
Some(gettext(
"Unlock your keyring, or the next reboot signs you out",
)),
),
SyncState::Outdated { running } => (
"dialog-warning-symbolic",
Some("warning"),
gettext("Restart to finish updating"),
Some(if running.is_empty() {
gettext("An older version of Proton Drive is still running")
} else {
// Translators: {version} is a release number such as "3.2.0".
gettext_f(
"Proton Drive {version} is still running",
&[("version", &running)],
)
}),
),
SyncState::Paused { until, queued } => (
"media-playback-pause-symbolic",
Some("warning"),
@@ -1387,6 +1509,8 @@ fn paint_sync_status(ui: &Rc<Ui>, state: SyncState) {
if let Some(class) = class {
image.add_css_class(class);
}
ui.locations.card.sign_in.set_visible(expired);
ui.locations.card.restart.set_visible(outdated);
ui.status.status_title.set_label(&title);
ui.status.status_detail.set_visible(detail.is_some());
ui.status
@@ -1455,7 +1579,7 @@ pub(crate) fn repaint_pins(ui: &Rc<Ui>, pins: &[pdfs_core::cache::Pin], mounted:
let row = adw::ActionRow::builder()
.title(gettext("No files kept offline"))
.subtitle(gettext(
"Right-click a file and choose “Make available offline”.",
"Right-click a file or folder and choose “Available offline”.",
))
.build();
ui.status.pins_group.add(&row);
@@ -1483,6 +1607,7 @@ pub(crate) fn repaint_pins(ui: &Rc<Ui>, pins: &[pdfs_core::cache::Pin], mounted:
let row = adw::ActionRow::builder()
.title(&name)
.subtitle(&pin.path)
.use_markup(false)
.build();
let icon = gtk4::Image::from_icon_name("pdfs-offline-symbolic");
row.add_prefix(&icon);
+19 -2
View File
@@ -510,6 +510,7 @@ pub(crate) fn repaint_archives(ui: &Rc<Ui>) {
.unwrap_or_else(|| path.display().to_string()),
)
.subtitle(human_bytes(size))
.use_markup(false)
.build();
let remove = gtk4::Button::builder()
.icon_name("edit-delete-symbolic")
@@ -645,7 +646,12 @@ pub(crate) fn refresh_takeout(ui: &Rc<Ui>) {
// An import runs for hours, so its ending is announced rather than
// left to be discovered: a toast for whoever is looking at the
// window, and a desktop notification for whoever is not.
if was_running {
let failed = summary.as_ref().and_then(|s| s.error.clone());
if was_running && let Some(error) = &failed {
let headline = gettext("Couldn't import from Google Photos");
toast_error(&ui, &headline, error);
notify("takeout-import", &headline, error);
} else if was_running {
let dry = ui.takeout.dry_run.get();
let headline = if dry {
gettext("Scan finished")
@@ -701,7 +707,18 @@ fn repaint_summary(ui: &Rc<Ui>, summary: &ImportSummary) {
} else {
gettext("Last import")
});
ui.takeout.summary_group.set_description(Some(&if dry {
ui.takeout.summary_group.set_description(Some(&if let Some(error) = &summary.error {
// The reason is the service's own text, which is not translated. A
// group's description is markup, and the reason may name a path.
glib::markup_escape_text(&gettext_f(
"{what}: {detail}",
&[
("what", &gettext("Couldn't import from Google Photos")),
("detail", error),
],
))
.to_string()
} else if dry {
gettext("Nothing was uploaded — this is what an import would do.")
} else if summary.cancelled {
gettext("Stopped early. Everything uploaded so far is in your timeline; run it again to continue where it left off.")
+5 -1
View File
@@ -330,7 +330,9 @@ pub(crate) fn trash_unreachable(ui: &Rc<Ui>) {
pub(crate) fn trash_status(ui: &Rc<Ui>, icon: &str, title: &str, description: &str, retry: bool) {
ui.trash.status.set_icon_name(Some(icon));
ui.trash.status.set_title(title);
ui.trash.status.set_description(Some(description));
ui.trash
.status
.set_description(Some(&gtk4::glib::markup_escape_text(description)));
ui.trash.retry.set_visible(retry);
ui.trash.content.set_visible_child_name("status");
ui.trash.subtitle.set_subtitle("");
@@ -524,6 +526,7 @@ mod tests {
size: 0,
modified,
pinned: false,
pinned_by_folder: false,
cached: false,
uid: name.into(),
path: String::new(),
@@ -533,6 +536,7 @@ mod tests {
shared_by_unverified: false,
trashed_at,
trashed_from: from.map(str::to_string),
issue: None,
}
}
+168 -40
View File
@@ -1,4 +1,4 @@
//! The human-verification (CAPTCHA) dialog.
//! The human-verification (CAPTCHA) window.
//!
//! Proton gates logins it does not recognise — a new IP, a VPN exit — behind a
//! CAPTCHA. The API answers the login with a challenge instead of a session, and
@@ -8,10 +8,26 @@
//! The page is hosted in a `WebKitWebView` rather than handed to the system
//! browser because it reports completion by posting a message to its host, not
//! by redirecting: a browser has nowhere to post it back to.
//!
//! The webview runs in a child `pdfs-app` started with [`VERIFY_ARG`], not in the
//! app itself. WebKit aborts its whole process when its sandbox cannot start, as
//! on hardened systems that keep bubblewrap from the journal socket
//! (`docs/BUGS.md` B195); in a child that costs the verification, not the app.
//! The URL goes to the child on stdin, so it never shows in a process list, and
//! the token comes back on stdout.
use crate::*;
use std::io::{BufRead, BufReader, Write as _};
use std::process::{ExitStatus, Stdio};
use webkit6::prelude::*;
/// The argument that makes `pdfs-app` the verification window instead of the app.
pub(crate) const VERIFY_ARG: &str = "--human-verification";
/// Marks the token among whatever else lands on the child's stdout: GLib and
/// WebKit write debug output there.
const TOKEN_PREFIX: &str = "hv-token ";
/// Bridge the verification page's `postMessage` to a handler this side can read.
///
/// The page targets its embedding host (`window.parent`), which in a webview is
@@ -26,15 +42,100 @@ window.addEventListener('message', function (event) {
});
"#;
/// Present the challenge and hand the solved token back through `token_tx`.
///
/// Closing the dialog without solving drops the sender, which the waiting login
/// worker reads as a cancelled sign-in — the same contract as the 2FA prompt.
pub(crate) fn prompt_human_verification(
ui: &Rc<Ui>,
url: &str,
token_tx: std::sync::mpsc::Sender<String>,
) {
/// How the verification window ended.
#[derive(Debug, PartialEq, Eq)]
pub(crate) enum Verification {
Solved(String),
/// Closed without solving.
Cancelled,
/// Never came up, or died: most likely WebKit could not start its sandbox.
Failed,
}
/// Show the challenge in the verification window and wait until it closes.
pub(crate) async fn verify_human(url: String) -> Verification {
match gio::spawn_blocking(move || run_verification_child(&url)).await {
Ok(Ok(verification)) => verification,
Ok(Err(e)) => {
tracing::error!("cannot run the verification window: {e}");
Verification::Failed
}
Err(_) => Verification::Failed,
}
}
fn run_verification_child(url: &str) -> std::io::Result<Verification> {
let mut child = Command::new(own_exe())
.arg(VERIFY_ARG)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn()?;
if let Some(mut stdin) = child.stdin.take() {
// A failed write means the child is already gone; its exit status says
// how. Dropping `stdin` closes it.
let _ = writeln!(stdin, "{url}");
}
// Stops reading at the token rather than at EOF: WebKit's helper processes
// inherit the pipe and may hold it open a moment after the child quits.
let token = child
.stdout
.take()
.and_then(|out| find_token(BufReader::new(out)));
let status = child.wait()?;
Ok(outcome(token, status))
}
/// This very binary, so a development build opens its own window. After an
/// upgrade replaced the file, `current_exe` names a deleted path, and the
/// installed `pdfs-app` is the next best.
fn own_exe() -> PathBuf {
std::env::current_exe()
.ok()
.filter(|path| path.is_file())
.unwrap_or_else(|| PathBuf::from("pdfs-app"))
}
fn find_token(out: impl BufRead) -> Option<String> {
out.lines()
.map_while(Result::ok)
.find_map(|line| line.strip_prefix(TOKEN_PREFIX).map(str::to_owned))
}
/// A token wins over the exit status: the user solved the page, even if
/// WebKit's teardown crashed after it.
fn outcome(token: Option<String>, status: ExitStatus) -> Verification {
match token {
Some(token) => Verification::Solved(token),
None if status.success() => Verification::Cancelled,
None => {
tracing::error!(%status, "the verification window failed");
Verification::Failed
}
}
}
/// Run as the verification window: read the page's URL from stdin, show it, and
/// print the token once the user has solved it. Closing the window exits
/// without a token, which the app reads as a cancelled sign-in.
pub(crate) fn run_verification_window() -> glib::ExitCode {
let mut url = String::new();
if std::io::stdin().lock().read_line(&mut url).is_err() || url.trim().is_empty() {
tracing::error!("{VERIFY_ARG} expects the verification URL on stdin");
return glib::ExitCode::FAILURE;
}
let url = url.trim().to_owned();
// Not unique: the app that started it already holds APP_ID.
let app = adw::Application::builder()
.flags(gio::ApplicationFlags::NON_UNIQUE)
.build();
app.connect_activate(move |app| build_verification_window(app, &url));
// GApplication would refuse VERIFY_ARG as an unknown option.
let argv0 = std::env::args().next().unwrap_or_default();
app.run_with_args(&[argv0])
}
fn build_verification_window(app: &adw::Application, url: &str) {
let content = webkit6::UserContentManager::new();
content.add_script(&webkit6::UserScript::new(
BRIDGE,
@@ -53,24 +154,19 @@ pub(crate) fn prompt_human_verification(
.hexpand(true)
.build();
let dialog = adw::Dialog::builder()
let body = gtk4::Box::new(gtk4::Orientation::Vertical, 0);
body.append(&adw::HeaderBar::new());
body.append(&webview);
let window = adw::ApplicationWindow::builder()
.application(app)
.title(gettext("Verification"))
.content_width(420)
.content_height(560)
.icon_name(APP_ID)
.default_width(420)
.default_height(560)
.content(&body)
.build();
let header = adw::HeaderBar::new();
let body = gtk4::Box::new(gtk4::Orientation::Vertical, 0);
body.append(&header);
body.append(&webview);
dialog.set_child(Some(&body));
// Taken on the first success *or* on close, so the token is sent at most
// once and a cancelled dialog always drops the sender.
let token_tx = Rc::new(RefCell::new(Some(token_tx)));
let tx = token_tx.clone();
let dlg_weak = dialog.downgrade();
let app = app.downgrade();
content.connect_script_message_received(Some("hv"), move |_, value| {
let Some(token) = extract_token(&value.to_str()) else {
// Every `postMessage` on the page reaches here, most of them the
@@ -78,25 +174,17 @@ pub(crate) fn prompt_human_verification(
// is simply not ours.
return;
};
if let Some(tx) = tx.borrow_mut().take() {
let _ = tx.send(token);
let mut stdout = std::io::stdout().lock();
if let Err(e) = writeln!(stdout, "{TOKEN_PREFIX}{token}").and_then(|()| stdout.flush()) {
tracing::error!("cannot hand the verification token back: {e}");
}
if let Some(dlg) = dlg_weak.upgrade() {
dlg.close();
if let Some(app) = app.upgrade() {
app.quit();
}
});
// Covers the close button, Escape, and the user giving up.
dialog.connect_closed(move |_| {
token_tx.borrow_mut().take();
});
webview.load_uri(url);
let parent = ui.login.login_button.root().and_downcast::<gtk4::Window>();
// Returns immediately; the login worker stays blocked on the channel until
// the handler above sends, or the dialog closes and drops the sender.
dialog.present(parent.as_ref());
window.present();
}
/// Pull the verification token out of a message posted by the verification page.
@@ -133,7 +221,12 @@ fn extract_token(raw: &str) -> Option<String> {
#[cfg(test)]
mod tests {
use super::extract_token;
use super::{Verification, extract_token, find_token, outcome};
use std::os::unix::process::ExitStatusExt as _;
use std::process::ExitStatus;
/// The wait status of a process killed by SIGABRT.
const ABORTED: i32 = 6;
#[test]
fn a_completion_message_yields_its_token() {
@@ -163,4 +256,39 @@ mod tests {
assert!(extract_token(raw).is_none(), "accepted: {raw}");
}
}
#[test]
fn the_token_is_found_among_other_output() {
let out = "Gtk-Message: something\nhv-token tok-1\nmore\n";
assert_eq!(find_token(out.as_bytes()).as_deref(), Some("tok-1"));
assert_eq!(find_token("only chatter\n".as_bytes()), None);
}
#[test]
fn a_window_closed_without_a_token_cancels() {
assert_eq!(
outcome(None, ExitStatus::from_raw(0)),
Verification::Cancelled
);
}
/// WebKit aborts when its sandbox cannot start (B195). That must read as a
/// failure the app explains, not as the user giving up.
#[test]
fn a_window_that_died_fails() {
assert_eq!(
outcome(None, ExitStatus::from_raw(ABORTED)),
Verification::Failed
);
let exited_1 = ExitStatus::from_raw(1 << 8);
assert_eq!(outcome(None, exited_1), Verification::Failed);
}
#[test]
fn a_token_counts_even_if_the_window_died_after_it() {
assert_eq!(
outcome(Some("tok-1".into()), ExitStatus::from_raw(ABORTED)),
Verification::Solved("tok-1".into())
);
}
}
+168
View File
@@ -0,0 +1,168 @@
//! Tells the user when a newer release is out, if they turned on Preferences →
//! Updates. The decisions are [`pdfs_core::update`]'s; this asks GitHub and
//! shows the banner.
use crate::*;
use std::time::{SystemTime, UNIX_EPOCH};
use pdfs_core::update::{
RELEASES_LATEST_API, is_newer, latest_version, next_check_in, release_url, updates_managed,
};
/// How long one request to GitHub may take.
const FETCH_TIMEOUT: Duration = Duration::from_secs(10);
/// How long to wait after a failed check, such as one made offline, before
/// trying again. Not saved, so a restart tries again straight away.
const RETRY_AFTER: Duration = Duration::from_secs(60 * 60);
pub(crate) struct UpdateState {
/// Over the sidebar, so it shows whichever page is open.
pub(crate) banner: adw::Banner,
/// The check loop is running. Turning the setting off and on again while it
/// sleeps must not start a second one.
running: Cell<bool>,
/// The release page the banner's button opens.
url: RefCell<String>,
}
impl UpdateState {
pub(crate) fn new() -> Self {
Self {
banner: adw::Banner::builder()
.button_label(gettext("Download"))
.build(),
running: Cell::new(false),
url: RefCell::new(String::new()),
}
}
}
/// Whether a package manager brings new releases to this install. The app then
/// never asks, and Preferences does not offer to.
pub(crate) fn updates_managed_here() -> bool {
std::env::current_exe().is_ok_and(|exe| updates_managed(&exe))
}
/// Wire the banner, show a release an earlier check found, and start checking
/// if the user asked for it.
pub(crate) fn wire_update(ui: &Rc<Ui>) {
let ui_open = ui.clone();
ui.update
.banner
.connect_button_clicked(move |_| open_uri(&ui_open.update.url.borrow()));
show_latest(ui);
start_update_checks(ui);
}
/// Start the daily check, unless it is off, managed elsewhere, or running.
pub(crate) fn start_update_checks(ui: &Rc<Ui>) {
if updates_managed_here()
|| !ui.dirs.load_config().update_check
|| ui.update.running.replace(true)
{
return;
}
let ui = ui.clone();
glib::spawn_future_local(async move {
// Also kept here, so a config that cannot be saved does not make every
// pass ask GitHub again.
let mut checked = None;
loop {
let config = ui.dirs.load_config();
if !config.update_check {
break;
}
let wait = next_check_in(config.update_last_checked.max(checked), unix_now());
if !wait.is_zero() {
glib::timeout_future(wait).await;
continue;
}
match fetch_latest().await {
Ok(latest) => {
checked = Some(unix_now());
let mut config = ui.dirs.load_config();
config.update_last_checked = checked;
config.update_latest = latest;
if let Err(e) = ui.dirs.save_config(&config) {
tracing::warn!("update check: cannot save the result: {e}");
}
show_latest(&ui);
}
Err(e) => {
tracing::warn!("update check: {e}");
glib::timeout_future(RETRY_AFTER).await;
}
}
}
ui.update.running.set(false);
});
}
/// Reveal the banner if the last check found a release newer than this one,
/// and hide it otherwise.
pub(crate) fn show_latest(ui: &Ui) {
let config = ui.dirs.load_config();
let newer = config
.update_latest
.filter(|latest| config.update_check && is_newer(latest, env!("CARGO_PKG_VERSION")));
match newer {
Some(version) => {
ui.update.banner.set_title(&gettext_f(
"Proton Drive {version} is available",
&[("version", &version)],
));
*ui.update.url.borrow_mut() = release_url(&version);
ui.update.banner.set_revealed(!updates_managed_here());
}
None => ui.update.banner.set_revealed(false),
}
}
fn unix_now() -> i64 {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map_or(0, |d| d.as_secs() as i64)
}
/// The latest release's version, asked on a worker thread so the window never
/// waits on GitHub. `Ok(None)` when the answer names no `X.Y.Z` release.
async fn fetch_latest() -> Result<Option<String>, String> {
let (tx, rx) = async_channel::bounded(1);
std::thread::spawn(move || {
let _ = tx.send_blocking(fetch_latest_blocking());
});
rx.recv().await.map_err(|e| e.to_string())?
}
fn fetch_latest_blocking() -> Result<Option<String>, String> {
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.map_err(|e| format!("no runtime: {e}"))?;
let client = reqwest::Client::builder()
.user_agent(concat!(
"proton-drive-linux/",
env!("CARGO_PKG_VERSION"),
" (",
env!("CARGO_PKG_REPOSITORY"),
")"
))
.timeout(FETCH_TIMEOUT)
.build()
.map_err(|e| format!("no HTTP client: {e}"))?;
runtime.block_on(async {
let reply = client
.get(RELEASES_LATEST_API)
.header("Accept", "application/vnd.github+json")
.send()
.await
.map_err(|e| e.to_string())?;
if !reply.status().is_success() {
return Err(format!("GitHub answered {}", reply.status()));
}
let body = reply.bytes().await.map_err(|e| e.to_string())?;
Ok(latest_version(&body))
})
}
+21
View File
@@ -0,0 +1,21 @@
use crate::*;
/// The "›" between two breadcrumb segments.
pub(crate) fn crumb_separator() -> gtk4::Label {
let sep = gtk4::Label::new(Some("›"));
sep.add_css_class("dim-label");
sep
}
/// The last breadcrumb segment, the folder on screen: a heading rather than a
/// button. It carries a flat button's side padding (`.crumb-current`), so the
/// "›" before it sits as far from it as from the button before.
pub(crate) fn crumb_current(label: &str) -> gtk4::Widget {
let l = gtk4::Label::builder()
.label(label)
.ellipsize(gtk4::pango::EllipsizeMode::Start)
.build();
l.add_css_class("heading");
l.add_css_class("crumb-current");
l.upcast()
}
+15 -4
View File
@@ -98,6 +98,8 @@ pub(crate) fn build_details_pane() -> (gtk4::Widget, DetailsWidgets) {
.title(gettext("Location"))
.subtitle("—")
.subtitle_lines(3)
// A folder path, which may hold `&` or `<`.
.use_markup(false)
.build();
path_row.add_css_class("property");
props.add(&size_row);
@@ -384,11 +386,20 @@ pub(crate) fn show_details(ui: &Rc<Ui>, entry: &DirEntry) {
} else {
parent.to_string()
});
// Pinning is per file: the context menu and the selection bar offer it for
// files only, so the pane must not offer it for folders either.
// Kept through a pinned folder above it, the entry has no pin of its own
// to remove, so the switch shows that state without offering to change it.
d.pin_row.set_active(entry.pinned);
d.pin_row.set_visible(!entry.is_dir);
d.pin_row.set_sensitive(*ui.mounted.borrow());
d.pin_row.set_subtitle(&if entry.pinned_by_folder {
gettext("Kept on this computer because a folder it is in is available offline.")
} else if entry.is_dir {
gettext(
"Keep a copy of everything in this folder on this computer, even without a connection.",
)
} else {
gettext("Keep a copy on this computer, even without a connection.")
});
d.pin_row
.set_sensitive(*ui.mounted.borrow() && !entry.pinned_by_folder);
d.open_button.set_label(&if entry.is_dir {
gettext("Open folder")
} else {
Loaded 100 of 161 files, more files were not shown because too many files have changed in this diff. Show more