Compare commits

...
163 Commits
Author SHA1 Message Date
nvrl 61266fc9b1 chore(release): 2.5.0 2026-09-28 16:37:23 +02:00
nvrl d18655538e feat(core,fuse,gui): push daemon events, polish Photos, adopt newer widgets
The app and the tray stop polling. pdfs-core adds Request::Subscribe: the
connection stays open and carries Event lines (status, transfers, and
Changed with a Topic), with a heartbeat while quiet. In pdfs-fuse, an
EventHub gives each subscriber a mailbox, code that changes something
publishes its topic, and a sampler publishes the status, transfers and sync
progress when they differ. The app follows the feed on one thread, paints
status and transfers as they arrive, and reloads the page on screen when its
topic changes, including after changes made with the pdfs CLI. A hidden
window keeps the changes and catches up when focused. Front-ends fall back
to polling a daemon that predates subscriptions. Refresh buttons leave the
page headers; F5 and Ctrl+R still reload from the server.

Photos gets a scrubber overlaid on the right edge of the timeline in place
of the slider and scrollbar. It shows the years, appears when the pointer
comes near or the timeline scrolls, names the month under the pointer and
jumps to it on drag. Upload becomes a split button whose menu holds the
Google Photos import. Tiles are pooled per row and rebound instead of
rebuilt, hover scales the thumbnail to 1.02 without a shadow, and tiles
lose their tooltip.

On libadwaita 1.8 or newer the app uses AdwSpinner, AdwToggleGroup,
AdwButtonRow and AdwShortcutsDialog, and keeps look-alike stand-ins on
older releases (src/compat.rs, widgets/compat.rs). It still builds against
libadwaita 1.5 and picks at run time, creating the newer widgets by class
name through GtkBuilder, so every package gets them. PDFS_STAND_IN_WIDGETS
forces the stand-ins.

The Sync and share-link icons draw in full again: GTK drops group
transforms from symbolic icons, so their coordinates are baked into the
paths. All catalogs are updated and complete.
2026-09-28 16:37:23 +02:00
nvrl da7ea0ec56 feat(core,gui): split Sync into views, tidy Activity, rework Computers
The Sync page gets Overview, Folders and History views under a switcher in
the header. Overview holds the status card, conflicts, transfers and the
queue; Folders holds the folder list, with Add Folder and a popover that
explains the sync modes; History is the former Activity page, which leaves
the sidebar. A banner in Folders and History points back to Overview when
changes fail or conflict.

Activity hides save-by-rename temporaries (pdfs-core gains
syncignore::is_scratch_name), with a toggle to show them again. Runs of
similar events fold into one expandable row, "Show Older" pages further back
through what the daemon keeps, and titles and details are translated
sentences built from the daemon's targets instead of its raw text.
Conflict copies say when they were made.

Computers lists every device in one group, this computer first with a
"This computer" pill; its row opens Sync's folder list. Another computer's
row browses its backup read-only in the shared file list, and its menu
offers Browse Files, Restore, Rename, "Continue This Backup Here…" (the old
identity adoption, now explained in its dialog) and Remove. Subtitles read
"last backup" or "no backups yet", and an empty list links to the download
page. DirEntry derives Default for the browse view's synthesized entries.

The sidebar shows badges for sync problems and waiting invitations, polled
every five minutes away from their pages. All catalogs are updated and
complete.
2026-09-28 15:14:59 +02:00
nvrl 127b71efd0 feat(core,fuse,gui): rebuild Trash and shared pages on one file list
The list and grid view from My files moves into widgets/file_list.rs as
FileList, so Trash, Shared with me and Shared by me share its columns,
selection, context menus, keyboard handling and list/grid switch instead of
each drawing its own rows.

Trash shows where each item came from and when it was deleted. Proton does
not expose the trash time through the SDK yet, so the daemon records when it
first sees a node in the trash (migration V35, trash_seen) and reports that;
the date becomes exact once proton-sdk-rs exposes Node::trash_time. Restoring
offers a Show action, and opening a trashed item explains that it needs to be
restored first.

Shared with me splits files and saved links into two views. Files show who
shared them, with a warning pill for unverified senders, the access they give
and when they were shared. Pending invitations sit in a banner that opens a
dialog to accept or decline each one. Several items can be left at once.
Saved links get their own list with open, copy and remove actions.

Shared by me gains a filter (all, links, people, expired links) and columns
for location, people and link state, with a copy button for live links.
Several items can stop being shared at once, the page refreshes after the
share dialog closes, and a reload rebinds only the rows that changed so the
selection survives.

All catalogs are updated and complete.
2026-09-28 14:35:37 +02:00
nvrl 2339b17d97 feat(core,gui): keep pages up while they reload
Pages no longer swap their content for a "Loading…" status on every load.
A new Loader keeps the last rows on screen and only shows the loading
state if the reply takes longer than 300 ms; a newer load always
supersedes an older one, so a late reply is dropped instead of painted.
Reloading what is on screen keeps it usable; loading something else greys
the old rows out and clears the selection, since their paths would
resolve against the new folder.

My files, search results and Trash now swap only the rows that changed,
so selection, scroll position and thumbnails survive a refresh. Activity
is one sectioned ListView: open conflicts first, then a section per day,
with the filter bar pinned above the list.

Polls give up after 5 s instead of 120 s. A daemon that is running but
doesn't answer now reads as "Not responding" in the app and the tray
rather than as disconnected.
2026-09-28 13:51:28 +02:00
nvrl bd4f797d1f feat(gui): polish pass over wording, icons and responsiveness
The Sync page gives each folder one card with its state, a Synced /
Online only switch and a menu for the rest; switching either way asks
first and the download prompt shows the free space on the disk.

Enabling, disabling and restarting the Proton Drive service ran
systemctl on GTK's main thread and froze the window. They now run in
the background, and every Retry button goes through one shared restart
helper.

Ten icons are bundled under resources/icons for glyphs no stock theme
has or that read ambiguously (sync, share, people, link, menu, map and
so on). A new test fails when the app or search prompt asks for a
symbolic icon that is neither bundled nor on a list checked against
Adwaita 50, libadwaita and GTK, or when a bundled icon is missing from
the resource manifest. The tray's sync icon, which Adwaita 50 no longer
ships, is now view-refresh.

Wording follows one glossary, recorded in docs/TRANSLATING.md: "Proton
Drive service" instead of "mount service" or "daemon", "Online only"
instead of "On-demand", "Proton Drive folder" instead of "mountpoint".
Dialog headings and buttons use title case and page titles match the
sidebar. Dates go through one formatter (app/dates.rs) and sizes
through i18n::human_bytes, and the cache label is translated.

The status tick stops while the window is hidden or minimized and runs
every fifth tick while it is unfocused, refreshing as soon as it gains
focus. "Sync complete" is announced only for large batches and only
when the window isn't focused. Smaller fixes: Empty Trash hides when
there is nothing to empty, copy buttons confirm with a check mark, the
sidebar status spins until the first report, and Photos moves Import
and Find Duplicates into a menu beside a labelled Select toggle.

All catalogs are updated and complete.
2026-09-28 13:26:59 +02:00
nvrl e872cad058 ci: move to Node 24 versions of checkout, upload-artifact and action-gh-release
actions/checkout@v4, actions/upload-artifact@v4 and
softprops/action-gh-release@v2 still target Node.js 20, which the runners
now force onto Node.js 24 with a deprecation warning. Their current majors
(checkout v7, upload-artifact v7, action-gh-release v3) run on Node.js 24.
2026-09-28 12:16:54 +02:00
Nils Pukropp 081c8fb889 chore(aur): update packages to 2.4.0 2026-09-28 07:33:52 +00:00
nvrl da35a05b84 chore(release): 2.4.0 2026-09-28 09:16:54 +02:00
nvrl a90eeecda9 feat(core,fuse,gui): find photos that look alike
The duplicate finder gains an Exact / Similar switch. Similar asks the
daemon for PhotoSimilar, which starts a background pass that hashes the
thumbnail of every shot not hashed yet and replies at once with the sets
found so far and how far hashing has got. The dialog shows a progress bar
and asks again until the pass is done, then lists the sets.

The pass takes thumbnails from the server only: a photo without one is
skipped for the run instead of downloaded whole, and the dialog says how
many photos could not be compared. Decoding and hashing are spread over
the machine's cores.

Every similar photo starts ticked Keep, since photos that look alike can
be different shots. A similar photo stands for its whole shot, so one not
kept goes to Trash with the rest of its shot.
2026-09-28 09:16:54 +02:00
nvrl c98ed0d83b feat(core): perceptual hashes for finding similar photos
A 64-bit difference hash of each thumbnail, stored in the new
photos.similar_hash column (schema 34), and the queries the similar-photo
finder needs: what is left to hash, progress, and the sets of shots that
look alike. Sets made only of byte-identical copies are left to the
duplicate finder. A refresh keeps a hash unless the photo's bytes changed.

The daemon pass, the control request and the GUI are not built yet; see
PLAN.md.
2026-09-28 09:16:54 +02:00
Nils Pukropp c46b2c1082 chore(aur): update packages to 2.3.0 2026-09-28 01:42:04 +00:00
nvrl fde9c2d271 feat(core,fuse,gui): Proton theme, photos on the map and an opt-in street map
- The Proton accent preference becomes a whole Proton theme that also
  holds against a GTK theme imported from gtk.css; folders get Proton's
  purple folder icon while it is on.
- Places map markers show one of each town's photos with a count, and
  zoomed in on a town every photo sits where it was taken
  (Request::PlaceLocations). Photos taken on one spot are listed in a
  popover. "Show on map" in the photo viewer goes to the Places map.
- The built-in map draws country borders, sea and land in their own
  colours.
- An opt-in street map from OpenFreeMap (vector tiles drawn with cairo,
  cached on disk up to 256 MB), with the OpenStreetMap credit on the map
  and in About.
- Translations for all catalogs; changelog folded into 2.3.0.
2026-09-28 03:12:57 +02:00
nvrl 4fd14eb745 chore(release): 2.3.0 2026-09-28 02:12:59 +02:00
nvrl b3c03a7162 feat(core,fuse,gui): places on a world map
The Places view gets a Grid/Map toggle. The map shows a bubble per town,
sized by its number of photos, over the outline of the world's land. Towns
whose bubbles would overlap merge into one that counts all their photos, and
clicking it zooms in until they part; clicking a single town opens it like a
card does. Drag to pan, scroll or pinch to zoom, and the buttons zoom in,
zoom out or frame every place again. Hovering names the town, or the first
few towns of a merged bubble.

No tile server is involved: none that is free has terms allowing an app like
this one to use it. The map is drawn from Natural Earth's 1:50m land
polygons (public domain), trimmed and simplified by
scripts/natural-earth-land.py into crates/pdfs-gui/data/land.txt, in Web
Mercator. The About dialog credits Natural Earth.

PlaceInfo carries the town's latitude and longitude; a daemon older than the
map sends none, and its towns are left off the map.
2026-09-28 02:12:59 +02:00
nvrl 8749d6c2c6 feat(core,fuse,gui): show how far photo locations have been read
The first timeline refresh after the Places update reads the metadata of
the whole library, and until it is done the Places view has nothing to show
and no sign that anything is happening.

The refresh now records how many photos it has resolved out of how many,
counting photos read by an earlier pass as done. Response::Places carries
it as mapping (absent from older daemons, which parse as None). The Places
view shows "Mapping photos: 120 of 480" with a bar while it is set, asks
again every two seconds, and loads the new places once the pass ends. An
empty listing during the pass says the photos are being mapped rather than
that there are no places.
2026-09-28 02:12:59 +02:00
nvrl 4ee2aed29b fix(gui): scroll the "On this day" strip with the photos timeline
The strip sat above the timeline's list and was hidden as soon as the list
left its top. Hiding it grew the viewport, which moved the list back to its
top and brought the strip back, so scrolling up near the top shook the whole
page between the two heights.

The strip is now the first row of the list (GalleryRow::Memories), so it
scrolls away with the grid and never resizes the viewport. It is still only
there atop the whole, unfiltered timeline.
2026-09-28 02:12:59 +02:00
nvrl 753056418c feat(core,fuse,gui): places view from photo locations
Photos now know where they were taken. proton-drive-rs 0.7.0 reads the
Location from a photo's extended attributes, and the timeline refresh stores
it with the photo (schema v33). The migration marks every photo unresolved
once, so the next refresh backfills locations from node metadata without
downloading anything.

Towns are looked up offline: pdfs-core bundles GeoNames' cities15000 table,
trimmed by scripts/geonames-cities.py, and files each photo under the
nearest town within 50 km. The new PhotoPlaces and PlacePhotos requests list
the towns and page one town's photos.

The Photos page gets a third view, Places, with a card per town (newest
photo, town, country, count). A place opens in the gallery like an album.
Country names come translated from the iso-codes catalogs, now an optional
dependency. The About dialog credits GeoNames (CC BY 4.0).
2026-09-28 02:12:59 +02:00
nvrl 2757977fd6 feat(gui): one Filters popover for the photos timeline
Favorites, "not in an album" and the month move from separate pills and a
dropdown into a Filters popover next to the kind tabs. The kind tabs stay
in the row because their counts are worth a glance. The button carries a
badge with the number of active filters, kind included, and the popover
has a "Clear filters" action that resets everything in one reload.

Flipping a filter while a page was still loading used to drop the reload,
leaving the timeline answering the old filter. The request is now queued
and the stale page is discarded.
2026-09-28 02:12:59 +02:00
nvrl 2f14c1688f feat(core,fuse,gui): find and trash duplicate photos
A "Find duplicates" button in the Photos header opens a review of every
set of files that share a content hash. Each set shows its copies side by
side with a Keep toggle; the copy in the most albums, then in the larger
RAW+JPEG group, then the oldest, is kept by default, and a set can't lose
its last kept copy. The copies not kept move to Trash one file at a time.

- core: Db::photo_duplicates, Request::PhotoDuplicates, and a files_only
  flag on TrashNodes (defaults to false, so old clients still trash whole
  groups).
- core: deleting a group's representative now clears the group key of its
  remaining files instead of leaving them pointing at a missing uid.
- fuse: handlers for both requests.
2026-09-28 02:12:59 +02:00
nvrl 7dbf818d83 feat(core,fuse,gui): "On this day" strip above the photos timeline
Add a PhotosOnThisDay control request, served from the stored timeline:
photos whose local capture date matches today's in earlier years, one
row per group. The GUI shows a card per year ("3 years ago") with the
year's newest photo as its cover; clicking a card scrolls the timeline
to that day. The strip is revealed only at the top of the whole,
unfiltered timeline, so it gives the grid its room back once you
scroll or filter.
2026-09-28 02:12:59 +02:00
nvrl c3ec82392a fix(gui): keep the photos timeline at its top across a relayout
Anchoring on the first photo row after a relayout scrolled the heading
above it out of view, so the timeline opened a heading's height down.
2026-09-28 02:12:58 +02:00
nvrl ce4656e46a fix(gui): square photo grid laid out to the viewport
Lay the timeline out as a grid of square tiles that spans the
scrolled window's width exactly, instead of justified rows sized from
learned aspect ratios. Track the width from the horizontal adjustment
so the widest row can no longer push the window wider, and put the
scrubber beside the list rather than over it.

Load the app stylesheet above user priority: a theme imported from the
user's gtk.css padded every button, tiles included. Keep the top photo
at its screen offset across a relayout, and only drop thumbnail
requests the unbound row still owns.
2026-09-28 02:12:58 +02:00
nvrl 557eeb5f1d feat(gui): slideshow in the photo viewer
A new top-bar button, or F5, plays the loaded photos as a slideshow: it goes
fullscreen, hides the controls and moves on every five seconds. A video plays
to its end first, and the slideshow waits for each photo to load before it
starts counting. Stepping by hand restarts the count. Escape stops the
slideshow instead of closing the viewer, and fullscreen is left only if the
slideshow entered it. The slideshow ends on the last photo of the timeline.
2026-09-28 02:12:58 +02:00
nvrl b47e6c91a0 feat(gui): group the photos timeline by zoom level, select ranges
Zooming out now changes how the timeline is sectioned: by day at the default
row height, by month further out and by year at the smallest tiles, which
gain one more zoom step. Under a month or year heading the tile caption shows
the day instead of the clock time.

Shift+click selects every photo from the last one clicked, and dragging
across the timeline selects the photos swept over, scrolling at the top and
bottom edges. Sweeping back gives photos up again.
2026-09-28 02:12:58 +02:00
nvrl 5c1da9d7a5 feat(core,fuse,gui,cli): filter the photos timeline to photos in no album
A new pill next to the favorites filter narrows the timeline to the photos
no album of ours holds yet, so what is left to file is easy to find. A
RAW+JPEG pair counts as filed when any of its files is in an album.

PhotosTimeline gains an optional not_in_album flag; older front-ends omit it
and get the whole timeline. Album contents are fetched lazily, so the first
filtered page fetches the contents of any album not yet listed. Photos added
to an album while the filter is on leave the timeline at once.
2026-09-28 02:12:58 +02:00
Nils Pukropp 0135c29a4f chore(aur): update packages to 2.2.2 2026-09-27 21:44:59 +00:00
nvrl 0b81b76cb7 feat(gui): branded app icon in the sidebar header and login
The sidebar header and the login page used the generic folder-remote-symbolic
icon. Ship the app icon from packaging/ in the GUI resources and use it in
both places, so the in-app brand matches the launcher even when the app runs
uninstalled.
2026-09-27 23:28:00 +02:00
nvrl 7470bd89f3 chore(release): 2.2.2 2026-09-27 23:28:00 +02:00
nvrl e6cb52339d fix(fuse,core): list and resolve conflict copies in synced folders, log each once (B99)
The conflict sweep flagged divergent copies anywhere in the tree but kept its
dedupe set in memory, so every daemon start logged them again. The conflict
listing only covered My Files, so copies in on-demand sync folders were never
listed, and the Activity page showed them as resolved while they were still on disk.

List copies in synced folders by their absolute local path, route resolves
to the mount that serves them, flag only what the listing can show, and
consult the activity log before flagging a copy again.
2026-09-27 23:28:00 +02:00
Nils Pukropp 912acc760a chore(aur): update packages to 2.2.1 2026-09-24 18:08:08 +00:00
nvrl 5109427c32 chore(release): 2.2.1 2026-09-24 19:49:42 +02:00
nvrl e2c4dac8df feat(gui): legal links and disclaimer in the About dialog
Add a copyright line with contact address, links to the source code and
to Proton's terms of service and privacy policy, a Legal section with the
full non-affiliation, trademark and no-warranty disclaimer, and a license
credit for proton-sdk-rs. Translated into every shipped language.
2026-09-24 19:49:42 +02:00
nvrl ebeb9cdbef chore(aur): refresh -git pkgver in update.sh and bump to 2.2.0.r1.g82e9654
makepkg --printsrcinfo does not run pkgver(), so the -git .SRCINFO kept
the version from when the package was created and push.sh never had a
change to publish. update.sh now runs makepkg -od --noprepare first so
pkgver() writes the current git describe version into the PKGBUILD.
2026-09-24 19:49:42 +02:00
Nils Pukropp 82e96543ac chore(aur): update packages to 2.2.0 2026-09-23 18:48:21 +00:00
nvrl 96c77e96cb chore(release): 2.2.0 2026-09-23 20:29:15 +02:00
nvrl 90e110e893 feat(fuse,core): thumbnails for videos and for our own uploads
Videos without a server thumbnail fell through to the local fallback,
which downloaded the whole file, failed to decode it as an image and
wrote the video off for good. Grab one frame with ffmpeg instead, a
second in or from the start for shorter clips. Probe for ffmpeg before
downloading, bound every run, and skip videos over 256 MiB.
Migration V32 gives the videos written off so far another try.

Those videos had no server thumbnail because our uploads never sent one.
Every upload path (bulk upload, drained mount writes, conflict copies,
new revisions and folder sync) now attaches a thumbnail rendered from
the local file, kept under the 60 KiB Proton's own clients use. Files
that are not images or videos, or cannot be decoded, upload bare as
before; a .ts file only counts as video with an MPEG-TS sync byte.

ffmpeg is a new optional dependency in the PKGBUILDs.
2026-09-23 20:29:15 +02:00
nvrl 0b7236bd4a fix(gui): steady the photo gallery and fill the details panel
The month scrubber sat under the overlay scrollbar, which widens under
the pointer, so the two collided. Keep the scrubber clear of it.

The gallery jumped when thumbnails loaded, on zoom and on tab switches.
Relayout anchored on the first bound row, but ListView keeps off-screen
rows bound with stale allocations, and scroll_to on an off-screen row
aligns it to the top. Anchor on the first mapped row that is actually
visible instead.

The viewer's details panel opened empty: the title's hexpand propagated
up to the revealer, which then took all the width. Set hexpand on the
revealer explicitly.

EXIF details read like a camera: Make, Model and Lens lose their EXIF
quotes, a model that already names its maker is not repeated, shutter
speeds are rounded (1/19 s), and ISO is labelled.
2026-09-23 20:29:15 +02:00
Nils Pukropp 2fa5eaa75b chore(aur): update packages to 2.1.1 2026-09-23 16:44:58 +00:00
Nils Pukropp dc9ea9f541 chore(aur): update packages to 2.1.0 2026-09-23 16:29:57 +00:00
Nils Pukropp b13d9d9f38 chore(aur): update packages to 2.1.1 2026-09-23 16:17:09 +00:00
nvrl b696adefc2 chore(release): 2.1.1 2026-09-23 18:00:03 +02:00
nvrl 519ec1af07 fix(fuse): stop a superseded upload from spinning its drain thread
A write that lands while a revision is on the wire cancels that upload
through the flag CountingReader checks before each read. The cancel was
reported as ErrorKind::Interrupted, which every read loop retries,
including the SDK's read_full_block. The flag never clears, so the drain
thread spun at 100% CPU forever and the transfer stayed listed as
uploading at 0 bytes until the daemon restarted. A growing browser
download (.crdownload) triggers it naturally.

Report the cancel as ErrorKind::Other so the SDK fails the upload;
drain_revision already treats a cancelled upload as abandoned, not
failed. Tracked as B98.
2026-09-23 17:58:06 +02:00
Nils Pukropp 4820ea1236 chore(aur): update packages to 2.1.0 2026-09-23 09:23:32 +00:00
nvrl a9d6e0280d ci(aur): publish the AUR packages from CI
A release tag now updates and pushes all three AUR packages once the
GitHub release exists, then commits the refreshed recipes back to main.
A push to main that changes the -git recipe publishes that package
alone; plain upstream commits need no AUR push because its pkgver()
follows git at build time.

Needs the AUR_SSH_PRIVATE_KEY repository secret.
2026-09-23 11:02:30 +02:00
nvrl 3e4c756361 docs: refresh the README screenshots
Retake the login, files, photos and computers screenshots, split the
search launcher into its GTK and fzf front ends, and add the Activity
page.
2026-09-23 10:57:48 +02:00
nvrl 38126ff2b3 feat(gui): translate the Photos filter tabs, subtitle and filtered empty states
The All/Photos/Videos/Raw tabs, the "12 photos" subtitle and the
"No favorite videos in June 2024" heading were still built from English
literals. Each heading is now a whole translatable sentence with a
{month} placeholder, and the subtitle uses plural forms.

Also fill in the French "Documents" search chip, which was left
untranslated because it matches the English word.
2026-09-23 10:41:30 +02:00
nvrl 9529f2124b feat(gui): translate the Photos empty states
The empty timeline and empty favorites texts go through gettext and are
translated in every catalog.
2026-09-23 10:37:19 +02:00
nvrl 77b8fd65be chore(release): 2.1.0 2026-09-23 08:08:16 +02:00
nvrl d9064ef880 feat(core,gui): gettext translations and language preference
The app, the tray and the search prompt route every user-visible string
through GNU gettext in the `pdfs` text domain, with named placeholders and
plural forms. po/ holds the template, update and build scripts, and catalogs
for de, en_GB, es, fr, it, nl, pl and pt_BR. Preferences gains a Language row
that overrides the system language from the next start (config `language`).

Packages compile the catalogs into /usr/share/locale and the desktop files
carry translated descriptions and keywords. docs/TRANSLATING.md describes
the workflow.
2026-09-23 08:07:50 +02:00
nvrl 66ab1f5b16 chore(aur): update packages to 2.0.0 2026-09-23 04:31:13 +02:00
nvrl 56161f649e docs: use the new app icon in the README 2026-09-23 04:22:16 +02:00
nvrl 69b5ccb28b chore(version) bump 2026-09-23 04:19:56 +02:00
nvrl af84ce2168 feat(gui): redesign the Shared with Me page
Show only sections with content, invitations first, and an empty state
when nothing is shared. Rows drop the mount path from the subtitle, show
access as a tinted pill (Can view / Can edit / Can manage) and move
Open, Show in My Files and Leave into a menu. The invitations banner is
gone now that invitations lead the page.
2026-09-23 04:18:41 +02:00
nvrl 52651caf8e feat(core,fuse,gui,cli): restore to this computer per device
ListDeviceRestorableFolders and RestoreDeviceFolders read another
registered device's folders and attach them with that device's share,
without changing this machine's cached device. The Computers page moves
restore into each computer's menu, and pdfs sync restore takes --device.
2026-09-23 04:15:17 +02:00
nvrl 0b63a65661 feat(gui): shared with me path header and invitations banner
The header subtitle shows the path inside a shared folder and a header
back button replaces the Back row. An adw::Banner counts pending
invitations and scrolls to them.
2026-09-23 04:11:48 +02:00
nvrl 7f84ff7dea feat(packaging): new app icon
A white cloud with the download arrow as negative space, on a rounded
purple tile with a darker lower edge, following the GNOME app-icon grid.
2026-09-23 04:10:35 +02:00
nvrl 15985f8684 feat(core,fuse,gui,cli): stop sharing and share dialog polish
StopSharing / StopSharingByUid remove the public link, external
invitations, invitations and members, trying every step and reporting
what is still shared. Shared by Me gets a Stop Sharing… item and
pdfs unshare <path> works without an id.

The Share dialog validates addresses inline, shows spinners on buttons
while requests run, offers a link expiry and shows an existing link's
expiry.
2026-09-23 04:09:32 +02:00
nvrl dfc48375e2 fix(gui): parent context menus to the view, not the item
Themes style the insides of list items with descendant selectors, and
those reached through a popover parented to a grid tile, spacing out
every menu section. Menus now live under the outermost list view.
2026-09-23 03:58:43 +02:00
nvrl 2dde69525e fix(gui): keep the details pane under the toggle's control
Pin the split view's sidebar so the wide breakpoint no longer opens the
pane by itself, and show a "No item selected" state instead of closing
it when a navigation clears the entry. The pane follows the header
toggle alone, which also tracks an overlay closed by a click outside.
2026-09-23 03:58:43 +02:00
nvrl f6136ce344 feat(core,gui): tray states and menu
The tray icon follows the drive's phase (synced, syncing, paused,
offline, needs attention) and reports NeedsAttention while changes keep
failing. The menu gains an issues item that opens the Sync page, Sign In
when signed out, Hide Tray Icon (persisted as tray_hidden) and Stop
Proton Drive, which pdfs-app confirms via --confirm-stop. A second
pdfs-app launch raises the running window.
2026-09-23 03:55:51 +02:00
nvrl 4fa61a14c3 fix(gui): clear the selection on an empty-space click
A click between the files in the grid or list drops the selection, as
in a file manager, unless Ctrl or Shift is held. Backspace goes back
while the focus is on the files.
2026-09-23 03:48:57 +02:00
nvrl fe8fb8cdcf feat(gui): activity log redesign
Group the feed by day with a time column and filter chips, fold a
conflict logged on every daemon run into one row with a count, and put
the conflicts still open in a Needs attention section. Conflict rows
open the Sync page's resolve dialog and read as resolved once the copy
is gone.
2026-09-23 03:44:21 +02:00
nvrl f457fa68fc fix(gui): open the details pane on request, not on selection
Revealing the pane on the first click of a double-click reflowed the files under the pointer, so the second click missed and the folder did not open. The pane now follows a header toggle, Alt+Enter and a Details menu item.
2026-09-23 03:38:48 +02:00
nvrl be8464f4a0 feat(gui): photos infinite scroll and month scrubber
The timeline pages itself in 1.5 screens before the end instead of offering a Load more button, and a year-marked month scrubber jumps to any month, loading long pages until it is reached.
2026-09-23 03:37:08 +02:00
nvrl a5d29969dc feat(gui): photo viewer rewrite
Auto-hiding controls, zoom and pan, a docked details panel, paging past the loaded photos with the library total in the counter, inline video playback, Save a Copy under the real name with an error toast, and Open With through the app chooser.
2026-09-23 03:34:29 +02:00
nvrl 43d1afe705 feat(core,fuse,gui,cli): album management
New requests CreateAlbum, RenameAlbum, DeleteAlbum, AddToAlbum and
RemoveFromAlbum, backed by proton-drive-rs 0.6.6. The Albums view gains a
New Album button and a card menu with Rename and Delete; photos gain Add
to Album (menu and selection bar) and Remove from Album inside an album.
CLI: pdfs new-album, rename-album, delete-album, add-to-album,
remove-from-album.

Also fixes the album view state: the timeline filters are hidden on the
album grid, the header counts albums, Back keeps the grid and its scroll
position, and the timeline is only reloaded when an album replaced it.
2026-09-23 03:25:16 +02:00
nvrl 29e5769dec feat(core,fuse,gui,cli): search scope and location column
Request::Search takes an optional folder scope (serde default, so older
front-ends keep searching everywhere). The GUI offers a This folder toggle
beside the search box and shows a Location column for hits; the CLI gains
pdfs search --in. Pins report whether a pinned file is cached, so pin-derived
search hits badge correctly.
2026-09-23 03:12:18 +02:00
nvrl 0fbc6421ba feat(gui): docked details pane on wide windows
An adw::Breakpoint at 1000sp uncollapses the details OverlaySplitView, so
the pane sits beside the files instead of over them. The pane gains
Sharing and Versions sections, filled from ListShare and ListRevisions
when an entry is selected, with Share…, Copy Link and Versions… buttons.
2026-09-23 03:06:32 +02:00
nvrl abf76cd5e8 feat(gui): shared selection and rubberband in My Files
The grid and column view wrap one MultiSelection instead of one each, so
a selection survives switching layout, and both views enable rubberband
selection.
2026-09-23 03:05:07 +02:00
nvrl 8d0a575a7c feat(gui): menu-model context menus with keyboard and background menu
Context menus are PopoverMenus built from gio::Menu through a small
ActionMenu helper, so they are keyboard navigable and read as menus. My
Files follows the item order of UI_UX_PLAN.md §6, adds Open With…, Copy
Link and Versions…, and gains a background menu on empty space and the
Menu key / Shift+F10. The photo tile menu and the ⋮ row menus on Shared
by Me and Computers use the same helper.
2026-09-23 03:04:32 +02:00
nvrl 91a5f21340 docs(core,fuse,gui,cli): US spelling and correct schema lines
User-visible strings, doc comments and docs now say "Favorite" and
"color", matching the `pdfs favorite` command; UI_UX_PLAN.md §8 records
the choice. The changelog's 1.11.x and Unreleased entries said schema 30,
but V31 (`photos.resolved_at`) shipped in 1.11.0. Adds the missing
release links for 1.10.0 to 1.11.1.
2026-09-23 03:01:27 +02:00
nvrl 8351382ef2 feat(core,fuse,gui,cli): upload and download bandwidth limits (phase 6)
Every transfer charges the bytes it moves to its direction's rate limit,
a virtual clock shared by all transfers with a one-second burst, and sleeps
off any overshoot. On runtime workers the sleep goes through block_in_place
so FUSE reads keep their threads.

The caps are set live with SetBandwidthLimits, persisted to config and
applied at mount start. Preferences gains a Network group with debounced
MiB/s editors, and the CLI gains `pdfs sync limit --up/--down`.
2026-09-23 02:43:42 +02:00
nvrl b58b60010e feat(core,fuse,gui,cli): pause a single mirrored folder (phase 6) 2026-09-23 02:36:45 +02:00
nvrl a86c3e586d feat(gui): trash multi-select, shared and computer row menus, sign-in errors (phase 6) 2026-09-23 02:31:13 +02:00
nvrl fdda35918f feat(gui): photos context menu, filter empty states and import banner (phase 5) 2026-09-23 02:26:42 +02:00
nvrl 8bc52edeb4 feat(core,gui): file browser menus, sort, history and folder-picker move (phase 4)
The My Files header gets a New menu, a grid/list split button whose menu
sorts by name, size or date (persisted as files_view in the config), and a
folder menu for Build Thumbnails and Open in File Manager. Navigation
keeps a back/forward history (Alt+Left/Right, mouse buttons, Alt+Up).
Move to… browses Drive folders and handles a whole selection, rename
preselects the stem, and files dropped from a file manager upload into
the folder on screen.
2026-09-23 02:21:39 +02:00
nvrl 792eee75a4 feat(core,fuse,gui,cli): list and resolve sync conflicts (phase 3)
Conflict copies under My Files are paired with their original and listed
over the control socket. The Sync page shows them with a resolve dialog
(keep original, keep copy, keep both under a new name), and the CLI gains
`pdfs conflicts` and `pdfs conflicts resolve`. Every resolution goes
through the Proton trash, and files that are open, being written or still
owed an upload are refused.
2026-09-23 02:14:56 +02:00
nvrl 067561564e feat(core,fuse,gui,cli): pause/resume sync and a visible upload queue (phase 2)
Backend (docs/UI_UX_PLAN.md B-1, B-2):
- Request::SetSyncPaused { paused, until } pauses the drain worker and the
  mirror reconcile. FUSE reads, write staging, the watcher and the poll keep
  running. The pause is persisted in the `state` table (`sync_paused_until`),
  so it survives a restart; a timed pause ends by itself through a timer
  thread, re-armed on mount. Resuming wakes the drain and runs a full
  reconcile. Response::Status gains `paused` and `paused_until`.
- Request::ListPendingOps lists the queue with paths, attempts, last error,
  next attempt and parked/failing flags. Request::RetryPendingOp { id } makes
  one backed-off op (or every failed one) due now. Parked ops are never
  un-parked and nothing is dropped from the queue.

Front ends:
- Sync page: a status card with Pause/Resume (split button with 1 h, 8 h and
  24 h pauses) and a "Waiting to Upload" list with Retry now and Retry All.
- The sidebar strip and the tray show the paused state; the tray gains Pause
  Sync / Resume Sync and names failing changes.
- CLI: `pdfs sync pause [--for 30m|1h|2d]`, `pdfs sync resume`,
  `pdfs sync queue`, `pdfs sync retry [id]`; `pdfs status` reports a pause.

Not in this phase: merging Activity into the Sync page, discarding a queued
op (needs a recovery copy first), and per-folder pause.
2026-09-23 02:06:41 +02:00
nvrl cf9e1665ca feat(core,gui): phase 1 shell — per-page headers, sidebar footer, Preferences dialog
Restructure the app shell per docs/UI_UX_PLAN.md phase 1:

- Every destination page gets its own AdwHeaderBar via page_frame(), with
  page actions packed into it and a busy spinner beside the title.
- The sidebar is regrouped (files, then a Sync section) with a primary menu
  (Preferences, Keyboard Shortcuts, About) and a footer showing sync state,
  storage quota and the account with Sign Out.
- The Settings page is gone: startup, folder location, appearance and cache
  options move into an AdwPreferencesDialog (Ctrl+,); transfers move to the
  Sync page; developer details move into About's debug info.
- Inline CSS moves to a gresource stylesheet using libadwaita colour
  variables. The system accent is the default; the Proton purple accent is
  an opt-in preference (new `proton_accent` config key, defaults off).
- Add a keyboard shortcuts dialog (Ctrl+?).

libadwaita requirement stays at 1.5, so Ubuntu 24.04 remains supported.
2026-09-23 01:50:37 +02:00
nvrl cc308ac25a fix(core,fuse,gui): UI/UX quick wins from the audit (phase 0)
- Move Google Takeout import out of Settings; Photos owns it, and the
  Takeout page's back button returns there.
- Trash from the file browser is Undo-only (no confirm); F2 on a
  multi-selection explains why it does nothing.
- Ask before sign-out, rejecting an invitation, removing someone's access,
  removing a public link, and switching a folder to on-demand.
- Clearer offline wording ("Make available offline" / "Make online only")
  with dedicated offline, online-only and upload symbolic icons; the pin row
  is hidden for folders.
- Search hits report whether a file is cached, so search results show the
  offline badge.
- Activity log: readable Empty Trash / link entries, distinct icons, and
  error/warning styling for failures and conflicts.
- Proper singular/plural counts instead of "item(s)" / "archive(s)".
- Name earlier versions by save time instead of revision id; capitalise
  the public link role; stop-syncing uses a stop icon, not a trash can.
- Label the Photos Upload and "Show on map" buttons; toast when a
  thumbnail build finishes; the versions button follows the mount state.
- Mode switches and device mutations reload both sync pages.
2026-09-23 01:34:35 +02:00
nvrl 690b91deab docs: add UI/UX audit and improvement plan
Audit of every GUI page, the photo viewer, the tray and the backend
capabilities a sync UI needs. Lists found bugs, a glossary, a new
information architecture (Sync hub, Preferences dialog) and a phased
roadmap including pause/resume and conflict resolution.
2026-09-23 01:24:41 +02:00
nvrl 9cd8a196aa chore(packaging): add AUR package sources for source, git, and binary builds
packaging/aur/ now holds the full content of three AUR repositories:
proton-drive-for-linux builds the latest tag from source,
proton-drive-for-linux-git builds main, and proton-drive-for-linux-bin
installs the binaries attached to the GitHub release. Each carries a
PKGBUILD, a generated .SRCINFO, an ignore-everything .gitignore, and the
0BSD license the AUR expects for package sources.

update.sh bumps the release-based packages and regenerates the .SRCINFO
files; push.sh publishes each package to its AUR remote. The README now
points users at the AUR packages and keeps the in-tree PKGBUILD as the
development build.
2026-09-21 12:58:13 +02:00
nvrl 860d29cfe8 fix(core,fuse): un-park abandoned creates, retire orphan staged bytes, serialise refreshes
Four faults found by reading the journal for issues that were never fixed,
tracked as B94-B97 in docs/BUGS.md.

A create parked behind a temp-file rename (B70) had no exit but that rename,
so 43 uploads sat in the queue for weeks holding their staged bytes. The
primary drain worker now sweeps parked creates while the queue is idle: one
whose node is gone is discarded, and one older than PARK_EXPIRY_MS that nobody
holds open is un-parked rather than dropped, since staged bytes are the only
copy of the write. An open node stays parked however old it is.

.tmp and .temp leave TRANSIENT_SUFFIXES. The list served two mechanisms with
different consequences: ignoring a name during mirror reconcile is reversible,
parking a create is not, and a Takeout index.tmp is a final name that nothing
will ever rename.

Staged blobs that match no queued op and name no node were kept forever and
warned about once per blob per mount (95 blobs, ~445 MiB, oldest two months).
Blobs past STAGING_ORPHAN_RETAIN are retired; the rest are one aggregated
warning.

Refresh control requests now run one per scope. Every scope is individually
cheap, but duplicates contend for the database lock the running refresh holds,
which left five in flight for twenty-three minutes. request_kind names the
scope so the report says which view wedged. The photo metadata pass also stops
at shutdown instead of running into the cancelled runtime and logging one
"skipping undecryptable photo" per node.
2026-09-19 20:29:48 +02:00
nvrl 6cc8f72bab fix(core,fuse,gui,cli): read only the photos a refresh has not read
A timeline refresh resolved every photo's node to learn its name, media
type, favourite tag, content hash and photo relation — 63 round-trips for
a 12,445-photo library, minutes of work, repeated every time the timeline
went stale sixty seconds later. None of that metadata changes on its own,
and when it does change the event feed names the node it changed.

`photos.resolved_at` (schema v31) records that a photo's node was read.
The refresh resolves the photos without that mark: the ones it has never
read, and the ones a remote event marked stale. An existing timeline is
backfilled as resolved, so upgrading does not re-read the library once.

Remote changes now name what they changed instead of asking for
everything. `classify_photo_events` turns a `NodeUpdated` into
`photos_mark_unresolved(uid)`, so a favourite toggled on a phone costs one
node read rather than the whole library; a local toggle is claimed as our
own echo, because the feed replays it and a `NodeUpdated` carries nothing
that says what changed. A continuity loss and `refresh photos --full` drop
every mark, which is the way back to a whole-library read.

Along the way: `photos_replace` never kept a photo's name, so a photo the
SDK could not decrypt was renamed to nothing on every refresh — and the
tab split and the RAW+JPEG grouping are both derived from the name. It is
kept now, and a row that *was* resolved is authoritative instead, so a
photo unlinked from its main elsewhere finally loses its `main_uid`.

`TIMELINE_ENRICH_CHUNK` is 150, matching the SDK's own `MAX_BATCH_COUNT`:
a chunk of 200 was split there into a full request and a 50-uid tail.
2026-09-19 17:39:49 +02:00
nvrl 376f3edb28 fix(core,fuse,gui,cli): pair a Pixel raw with its JPEG, and stop refresh timing out
Three things the 12,445-photo library showed that the tests did not.

A Pixel raw capture is two files named after the role each plays —
`PXL_….RAW-01.COVER.jpg` and `PXL_….RAW-02.ORIGINAL.dng` — so their stems differ
and the pairing rule never saw one shot. The marker is now cut off before names
are compared, narrowly: digits, then one role word, so an ordinary name that
happens to contain `.raw-` keeps its stem. Fed the real library's first 40 rows
through the grouping, they collapse to 21 tiles.

A raw that has a JPEG beside it is now painted with that JPEG's thumbnail, which
the server usually already holds. The Raw tab costs no raw download and no raw
decode for those tiles. Raws are found in one batched read, so an ordinary page
pays nothing for the lookup.

Photos could not be moved to Trash at all: they live on the photo volume, which
the mount never holds, so no `nodes` row exists, every authority answered
`Unknown`, and the write check refused the whole library with "trash access: not
allowed". Membership of our own timeline is the authority that does exist for a
photo, and it now answers. An album photo on someone else's volume still fails
closed.

Refreshing the gallery then reported `io: Resource temporarily unavailable (os
error 11)` (B92). The daemon held the control reply until the whole timeline had
been re-read — 63 batches of 200 nodes for this library, minutes of work against
a 120 s read bound. The daemon now answers as soon as the refresh is running,
`PhotosRefreshStatus` reports when it is done, and both front ends follow it: the
gallery reloads the page when it clears, and `pdfs refresh photos` blocks so the
command still means what it says. Each refresh logs its photo count, batch count
and duration.

Released as 1.11.0.
2026-09-19 17:08:33 +02:00
nvrl 09c0684a5e fix(core): keep pre-v30 photos on the page until a refresh groups them
A database written before the grouping column opens with `group_key` NULL on
every row, and NULL matches nothing: the join dropped the whole timeline until
the first refresh ran. `COALESCE(group_key, uid)` is what makes such a row its
own group, which is how the gallery behaved before the column existed.

Covered by a migration test that rewinds a fixture to v29.
2026-09-19 15:41:24 +02:00
nvrl c1df3b6cc2 feat(gui): show a grouped shot as one tile, and reach its other files
The grouping landed in the daemon; this is what it looks like. A tile whose shot
was stored as more than one file badges that in the corner — "RAW" when one of
the members is a raw file, since that is the word the person is looking for, and
a plain count otherwise (a live photo, a burst). The lightbox gains a control
that steps through the group's files, which is the only way to reach the RAW now
that the grid shows the shot once.

Deleting says what it will do: the daemon takes the whole group, so the
confirmation names the file count when a selection stands for more files than
tiles.
2026-09-19 15:39:14 +02:00
nvrl afd71713d5 feat(core,fuse): group the RAW and the JPEG of one shot
A camera that writes both puts two files on the volume, and the timeline showed
two tiles of the same photo. They are one shot to the person who took it.

Schema v30 records what ties them together: the server's own duplicate hash and
main-photo relation, plus a `group_key` every member of a shot carries. The key
is computed when the timeline is replaced, in three steps — the server relation
first, because it knows about live photos and bursts and we do not have to guess
at them; then same capture day, same file-name stem and one member raw while the
other is not, which is the pair a camera writes; otherwise the photo is its own
group. The member the grid shows is the non-raw one, since a JPEG decodes in
milliseconds and is what the person expects to see.

The gallery therefore pages by group and carries `group_size` and `has_raw` on
each entry, and the date scrubber counts what the page shows. The Raw tab is the
exception and still lists every raw file, because that tab exists to answer
"which raws do I have". Album pages are not grouped at all: an album is a list
someone made, and it shows exactly what they put in it.

`TimelineRow` is a struct now. It had grown to five positional fields and was
about to grow to seven, at which point nobody can read a call site.
2026-09-19 15:34:17 +02:00
nvrl c6bdba024f feat(gui,fuse): delete photos from the gallery and notice remote deletions
The gallery could show photos and nothing else: a photo deleted on the phone
stayed on the page for minutes, and a photo you wanted gone had to be deleted
on another device. Both are now handled here.

Deleting needs a uid-addressed request, because `Request::Delete` takes a
mountpoint path and the photos volume is not in the FUSE mount. `TrashNodes`
trashes through the SDK and answers per-uid outcomes, so a partial failure is
reportable; the daemon then drops those rows with `Db::photos_delete`, album
membership included, rather than leaving the gallery to wait for the next
timeline refresh. The GUI removes the tiles immediately with an Undo toast and
puts back anything the server refused — the photos are in Proton's trash and
stay recoverable there either way. Selection is Ctrl/Shift+click, the Select
toggle or Delete in the lightbox.

Remote deletions arrive on their own now. The Drive volume and the photos
volume have separate event streams, so the photos volume gets its own poller
and its own cursor. Trash and delete events remove rows; anything else only
marks the timeline stale, because there is no inode space to converge on this
volume. With the timeline TTL down to 60 s and an explicit Refresh that waits
for the server instead of only clearing the freshness stamp, a photo deleted
elsewhere leaves the grid in about ten seconds.
2026-09-19 15:25:04 +02:00
nvrl 5a8e73657c feat(gui): justify the gallery rows and keep the thumbnails on screen
The Photos grid laid every photo out as a centre-cropped square. That is
why portraits showed black bars and why a panorama was unrecognisable at
tile size: the tile's shape had nothing to do with the photo's.

Rows are now justified. Photos are taken in capture order until they no
longer fit the target height, and the row is then scaled to land exactly
on the right margin, so each tile is its own photo's shape and nothing is
cropped or letterboxed. The last row of a day stays at the target height
rather than being stretched — a day holding two photos should look like
a short day, not like a layout bug. Ratios come from the daemon where it
has one; where it has not, the tile is laid out square and the day
re-flows once a decoded thumbnail proves the real shape, on the existing
debounce so a screenful of decodes costs one re-flow. Extreme ratios are
clamped, so one strip of film cannot decide what a row looks like.

Ctrl+scroll and Ctrl+± now move the target row height, and the grid is
tighter to match: 2 px gaps, smaller heading margins, a 6 px radius and
no card behind each tile, so the page reads as photographs rather than
as a deck of cards.

Thumbnails also stopped disappearing on scroll. The texture cache was
evicting in insertion order — which is precisely what a scroll back up
asks for again — so it is now least-recently-used and holds 1500 rather
than 600, which at the new density is several screens instead of under
two. A batch tops itself up with the photos just past the rows on
screen, in the direction of travel, and once the queue is short the
near-miss tiles are fetched rather than dropped.
2026-09-19 15:05:30 +02:00
nvrl cf0d495f8c fix(fuse): join the worker pool before the runtime goes away, and log deep queues
Teardown flagged the FUSE worker pool closed but never joined it, so the
process could drop the tokio runtime while a worker was still inside a
job — which is where 2026-09-18's "A Tokio 1.x context was found, but it
is being shutdown" panic came from. `Workers::stop_and_join` now waits
for the jobs in flight, bounded at 10 s so a worker wedged on the network
cannot hold the stop the user asked for; what is still stuck is named in
the log and shutdown continues around it.

The queues stay unbounded — the only caller that could feel backpressure
is fuser's dispatch loop, and blocking that is the stall the pool exists
to avoid — but a lane now logs its depth every 256 jobs, so a queue that
keeps climbing is visible before it is reported as a slow mount.

Documents the hang as B91, with the journal evidence, what made it
undiagnosable, and the candidates the evidence does not yet separate.
2026-09-19 14:49:31 +02:00
nvrl bee15ee7cf feat(fuse,packaging): watch the daemon for stalls and answer the systemd watchdog
A supervisor thread ticks every 15 s and does three things: it WARNs
about a worker that has held one job for two minutes, a control request
that never returned, or a lane that has not completed anything while its
queue is non-empty; it samples resident size and the pending-op count
into the log every five minutes; and it answers the systemd watchdog.

The watchdog ping is deliberately not a heartbeat from a thread that
does nothing else — that would keep the unit "healthy" through exactly
the hang it is meant to catch. It goes out only after a real round trip
over the control socket, so it means "a client can still talk to this
daemon", which is the property that failed on 2026-09-18.

`sd_notify` is spoken directly rather than through libsystemd (READY,
WATCHDOG, STOPPING), including the abstract-namespace address; off
systemd every call is a no-op.

The unit gains `NotifyAccess=all` (rather than `Type=notify`, which
would fail the unit on the start timeout when no login is stored yet),
`WatchdogSec=120`, `Restart=always` — a hang is not a failing exit code
— and `MemoryHigh`/`MemoryMax` as a backstop against the 5.2 G growth
the hung run showed.
2026-09-19 14:45:15 +02:00
nvrl f24ac8a429 feat(fuse,core,cli): report what the daemon is doing, for the next hang
On 2026-09-18 the daemon hung: 43 minutes of CPU over 20 hours, 5.2 GiB
resident, silent from 17:10 until a manual restart the next day, and no
panic. Nothing in the journal said what it had been waiting on, and
nothing in the daemon could be asked.

- The worker pool publishes per-thread state: whether the thread is busy,
  how long it has been in that state, and the label of the job in hand.
  Every FUSE handler that hands work to the pool now names its job
  (`read`, `lookup`, `rename`), so a stuck worker is named rather than
  merely counted. Queue depths and per-lane completion counts are
  mirrored outside the queue lock.
- `Request::Diagnostics` / `pdfs diagnostics` reports those, plus
  resident size, the control handler count against its limit, the
  in-flight control requests with their ages, and the pending-op count.
- Built to answer while the daemon is wedged: atomics and `try_lock`
  only, and the one field that needs the database reads it on a throwaway
  thread with a 500 ms deadline.
2026-09-19 14:40:26 +02:00
nvrl f78f663740 feat(core,fuse): restore the whole trashed tree, and bump the SDK to 0.6.5
The trash is a flat list of uids, but a delete has a shape: a folder, and
often items inside it that were trashed separately beforehand. Restoring
one of those in isolation is not what it means — a folder came back
without the contents deleted with it, and a file came back inside a
folder that is still trashed, which is to say nowhere the user can see.

- Schema 29: `trash.parent_uid`, filled by the trash refresh from the
  node's parent, plus an index for the walk.
- `expand_restore` takes the connected piece of the trashed tree around
  what was asked for (every trashed descendant, every trashed ancestor)
  and orders it into waves, shallowest first: the server cannot put a
  node back under a parent that is still trashed. A uid the listing does
  not know about is restored on its own, because the listing is
  materialised in chunks and can be stale.
- `Core::restore` restores wave by wave, still streaming per-node
  outcomes, so one node the server refuses leaves the rest restored.

The SDK bump brings the 6cbf2f44 port: permanent API errors are no
longer retried on block transfers, expired block URLs are re-minted on
the whole-file and public-link download paths, and a server-supplied
Retry-After is capped at 120 s.
2026-09-19 14:31:40 +02:00
nvrl 470948c191 chore(version) bumped to 1.10.2 2026-09-16 23:05:09 +02:00
nvrl b095b9f6b0 fix(core,fuse): never stat a foreign mount from the local-file indexer
The "This computer" indexer walks $HOME, and `default_excludes` only kept it
out of the Drive mountpoint and our own state/cache dirs. A home directory
that also holds third-party FUSE mounts (sshfs, rclone) got walked into.

`scan` does ask for `same_file_system(true)`, which looks sufficient and is
not: `ignore` implements it as `is_same_file_system` -> `device_num(path)` ->
`path.metadata()`, run after the directory is queued. Learning that a mount is
foreign requires stat'ing it first, and against a mount whose server is gone
that stat never returns.

The consequence is worse than a stalled scan. The blocked thread sits in
uninterruptible FUSE wait, so it ignores SIGKILL: the daemon can never exit,
systemd's stop times out, and the unreapable process keeps the cache.db lock
the next start needs, turning one dead mount into a permanent restart loop.
No userspace recovery exists once a thread is in that state, so the fix has to
be to never enter it.

Exclude every mountpoint strictly below the scan root, read by path from
/proc/self/mounts (octal-unescaped, so a path with a space still matches).
`filter_entry` runs before the device stat, so the walker never touches them.
Re-read per scan rather than cached at startup: a mount that appeared since
the daemon started is the likeliest to be half-alive.

This does not change what is indexed — a nested mount is a different device,
so it was already going to be skipped. Only the stat in front of that decision
goes away.

Refs: docs/BUGS.md B90
2026-09-16 23:05:06 +02:00
nvrl f4d6580cd3 chore(version) bumped to 1.10.1 2026-09-15 15:13:22 +02:00
nvrl b2f62f70e1 chore(deps,core,gui): upgrade dependencies to latest
- keyring 3 -> keyring-core 1 + dbus-secret-service-keyring-store 1
  (crypto-rust). Same libdbus Secret Service backend; with no target the
  store matches items keyring 3 wrote, so stored sessions survive.
- ksni 0.2 -> 0.3 via the blocking API, with assume_sni_available so a
  tray that starts before the SNI watcher waits instead of failing. Drops
  the clap 2 / atty chain (Dependabot atty advisory).
- gtk4 0.11, libadwaita 0.9, webkit6 0.6, rusqlite 0.40, fuser 0.18,
  directories 6, sha1/sha2 0.11, base64 0.23.
2026-09-15 15:07:00 +02:00
nvrl dcb2ab7fac chore(version) bumped to 1.10.0 2026-09-15 13:59:24 +02:00
nvrl eb56f4a1e0 feat(fuse,core,gui,cli): one-request move+rename drain, show who shared an item
- A queued rename that also changes folder now lands as one `move-multiple`
  request carrying the target name instead of rename-then-move, so there is
  no half-applied state and no second call for stale name requirements to
  hit (B46 follow-up). A destination name collision moves under a conflict
  name in one step; same-folder renames still use the rename endpoint.
- Shared with me shows the inviter and invite date (GUI subtitle, `pdfs
  shared-with-me`), with a warning when the invitation signature does not
  verify against the claimed inviter's keys. `DirEntry` gains `shared_by`,
  `shared_at` and `shared_by_unverified`, all serde-defaulted.
- Bump `proton-sdk` / `proton-drive-rs` to 0.6.3 (anonymous-node moves,
  `TrashTime` trashed detection).
2026-09-15 13:59:13 +02:00
nvrl c28803b037 chore(changelog) updated changelog for 1.9.2 and 1.9.3 2026-09-08 14:58:48 +02:00
nvrl cd8474954b chore(version) bumped to 1.9.3 2026-09-01 17:00:43 +02:00
nvrl d7e07d042d Merge pull request #22 from kacperpaczos/fix/login-poll-backoff
fix(cli): back off while waiting for a stored session
2026-09-01 16:38:29 +02:00
nvrl 45852082c0 fix(cli,core): wake the parked login-wait daemon on sign-in
The login backoff had no wake-up path. A unit started at boot sits in
wait_for_session and reaches the ceiling within ~90s; the user then signs
in and `enable --now` is a no-op on an already-active unit, so the daemon
kept sleeping and the mount appeared up to a full interval late.

- service::enable_start now runs `enable` + `restart` instead of
  `enable --now`, so the GUI login and the tray's Connect interrupt the
  sleep immediately and still start a unit that is down.
- `pdfs login` restarts the unit when it is already active. It stays out
  of `enable` on purpose, so a headless setup driving `pdfs daemon` by
  hand is left alone.
- Lower the backoff ceiling from 60s to 15s. It is still an 80% cut in
  keyring/D-Bus wakeups (28,800/day -> 5,760/day) and it bounds the
  worst-case latency for paths that never touch systemd.

Also corrects the flat-3s-poll figure in the doc comment: 86400/3 is
28,800 checks a day, not 48,000.
2026-09-01 16:28:25 +02:00
nvrl 99ad3838e1 fix(GALLERY) fixed takeout date conversion + repair command 2026-08-28 13:06:50 +02:00
Kacper Paczos 637b512f6a fix(cli): back off while waiting for a stored session
The daemon's wait-for-login loop slept a flat 3 seconds between checks and
never gave up, so an enabled service with no saved session polled the keyring
28,800 times a day. Each check is more than a sleep: auth::load() builds a
fresh keyring::Entry, so every iteration opens a new D-Bus connection to the
Secret Service and negotiates a session on it.

Back off from 3s to a 60s ceiling and say "not logged in" once at INFO rather
than on every iteration, which also stops the journal filling with tens of
thousands of identical lines.

The wait is extracted into wait_for_session() taking the load and sleep calls
as parameters, so the schedule is testable without a keyring and without real
time passing.

Refs #21
2026-08-25 13:24:46 +02:00
nvrl 16ad6548c8 chore(DOCS) updated changelog 2026-08-21 23:32:06 +02:00
nvrl 847ece17d7 version bumped to 1.9.1 2026-08-21 22:48:28 +02:00
nvrl 50d2acdaba Merge pull request #20 from emptyname-org/feature/files-thumbnails-statusbar
feat(gui): add file thumbnails and a browser status bar
2026-08-21 22:44:58 +02:00
emptyname 53caf63ad4 Fix thumbnail lifecycle failures 2026-08-21 12:32:31 +01:00
nvrl 5176dd91e1 fixed icon 2026-08-21 04:13:11 +02:00
nvrl e88903d5ff chore(branding) added branding, best practice to repo 2026-08-21 04:09:58 +02:00
emptyname b3e6af1ae5 docs: document thumbnail runtime support 2026-08-18 13:06:40 +01:00
emptyname b0f9513915 feat(gui): show file thumbnails and browser status bar 2026-08-18 13:06:36 +01:00
emptyname 91b0864fce feat(thumbnails): generate cached previews for Drive files 2026-08-18 13:06:27 +01:00
nvrl 8bd9481902 fix(ui) improved ux 2026-08-18 12:50:49 +02:00
nvrl b2d5870834 fix(ui) improved ux 2026-08-18 12:06:28 +02:00
nvrl 078774f6a1 release(1.8.2) 2026-08-17 00:49:24 +02:00
nvrl 326a9b4e92 chore(docs) updated changelog + bug states 2026-08-17 00:04:54 +02:00
nvrl 78dc72a34d release(1.8.1) 2026-08-16 23:33:43 +02:00
nvrl 438b2b463e chore(docs) updated changelog 2026-08-16 22:03:55 +02:00
nvrl 4cd49168c4 fix(drain) failed attempts 2026-08-16 22:02:27 +02:00
nvrl 481c3fe0c7 version bump + bug docs 2026-08-16 21:34:55 +02:00
nvrl 098da2e97a fix(log) log unreachable nodes 2026-08-16 21:34:12 +02:00
nvrl d03d3eac3f fix(drive) locks + 1.7.1 2026-08-16 20:41:03 +02:00
nvrl 433e206203 chore(changelog) 2026-08-16 00:54:55 +02:00
nvrl 35418eea3f fix(perf) read-only connection pool, States doesn't write to db, search optimized 2026-08-16 00:33:27 +02:00
nvrl e48c4bb4b2 fix(fs) block fetch, sequential prefetch, in memory LRU 2026-08-15 23:44:13 +02:00
nvrl 7ad9231858 fix(fs) locks, vacuum, recover fsynced writes 2026-08-15 19:17:56 +02:00
nvrl e396c6b4d6 fix(fs) performance, locks 2026-08-15 18:49:07 +02:00
nvrl b37ab78451 version bump 2026-08-12 16:16:16 +02:00
nvrl 6e3d2c335e improved search prompt 2026-08-12 16:15:27 +02:00
nvrl 14b6df7bce fix(prompt) cached file open instead of actualy files 2026-08-11 14:22:13 +02:00
nvrl 4d76462e62 chore(versio) bumped to 1.4.0 2026-08-10 15:35:59 +02:00
nvrl a5e03ad8da persistent entity cache, streaming batche 2026-08-10 15:22:04 +02:00
nvrl 2336ad9e87 added album to photos, renamed gallery view 2026-07-31 20:31:25 +02:00
nvrl d988236bfa wired new locations into gui, fuse acceptance with new permission based mounts 2026-07-31 19:37:12 +02:00
nvrl d78ddeab8e cargo fmt 2026-07-31 10:18:30 +02:00
nvrl b04cce3a23 fixed B78 2026-07-31 10:15:13 +02:00
nvrl 7e8403936b fixed permission error on on-demand mounts 2026-07-31 09:46:51 +02:00
nvrl 866a174c7a fixed rpm signing 2026-07-30 02:55:15 +02:00
nvrl 352117e806 fixed rpm signing 2026-07-30 02:23:49 +02:00
nvrl 4b5666eacb added .deb signing + version bump 2026-07-30 02:13:07 +02:00
nvrl 9bbd8eb982 fixed a migration bug where ProtonDrive becomse unwritable 2026-07-30 02:11:23 +02:00
nvrl 77198ed261 version bumped to 1.2 2026-07-29 21:36:24 +02:00
nvrl 0d79c46729 version bumped to 1.2 2026-07-29 21:30:10 +02:00
nvrl a8210ce8fd implemented shared with me + verification 2026-07-29 21:29:07 +02:00
nvrl 551e14b644 implemented shared id permissions 2026-07-29 19:26:57 +02:00
nvrl 7de667b386 implemented primary mount shared with me 2026-07-29 17:04:42 +02:00
nvrl 295bbeadac implemented share_access, POSIX read-only mode 2026-07-29 14:57:56 +02:00
nvrlandClaude Opus 5 32817870a5 chore(packaging): bump PKGBUILD pkgver to 1.1.1
Version bump in b93e41e updated Cargo.toml only, so the release
workflow's tag/manifest/PKGBUILD agreement check failed on v1.1.1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BEWKS8smdPfLMZacgDXjAR
2026-07-27 03:25:56 +02:00
nvrl b93e41efc5 version bump to 1.1.1 2026-07-26 20:18:45 +02:00
nvrl 082ef8522a fixed refresh pull bug for trash 2026-07-26 20:17:59 +02:00
nvrl 2e95655244 Merge pull request #19 from narrrl/release/1.1.0
release 1.1.0
2026-07-26 19:03:00 +02:00
nvrl 942cb9f583 fixed concurrency bugs + performance 2026-07-26 19:01:48 +02:00
nvrl 8470662c54 fixed race conditions + extendet test suite 2026-07-26 15:56:58 +02:00
nvrl a091c2f34f implemented sweep mode and better conflict handling 2026-07-26 02:06:58 +02:00
nvrlandClaude Opus 5 bca0fe34bc fix(fuse,core): key conflict detection on revision id, stop transient-name seals (B69, B70)
B69 — spurious `(sync-conflict)` copies came from `(mtime, size)` being used as a
revision identity. The server re-stamps a sealed revision with its own commit
time, so the drain diverted a write into a conflict copy of its own base; and
`plan_file` classified a no-baseline (local, remote) pair as `Conflict`, forking
every pre-existing file on first reconcile.

- `Baseline` carries `revision_id` (from the SDK's new `active_revision_id`),
  captured at write-open and preserved across supersede/rebaseline.
  `revision_changed` trusts the id and only falls back to `(mtime, size)` for
  old sidecars.
- Planner gains `FilePlan::AdoptBaseline`: equal-size no-baseline pairs are
  recorded, not conflicted. Divergent sizes stay a real conflict.
- New `sweep.rs` (`pdfs-conflict-sweep`, 5-min cadence) removes conflict copies
  proven identical by size + `content_sha1` and surfaces divergent ones once as
  an `ActivityKind::Conflict` entry. Never removes what it cannot prove.

B70 — browsers downloading into an on-demand mount had their in-flight temps
sealed as canonical revisions, exiling the finished file to a conflict copy.
`syncignore::is_transient_name` now recognises download/scratch temps, and the
on-demand `release`/`rename` paths hold them (`set_create_hold`) instead of
sealing until the name is finalized.

Workspace deps bumped to proton-sdk/proton-drive-rs 0.2.2 (registry, no patch).

Live validation against a real account still pending for both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8P42asymU158CPnhjRwri
2026-07-25 15:24:31 +02:00
nvrl 6df757006a updated documentation + readme 2026-07-22 15:29:34 +02:00
nvrl 6f70eca78e Merge pull request #18 from narrrl/release/1.0.0
release/1.0.0
2026-07-22 15:18:00 +02:00
nvrl d5165b36a2 Merge pull request #17 from narrrl/feature/testing
Feature/testing
2026-07-22 14:29:39 +02:00
nvrl e9009dd47c Merge pull request #16 from narrrl/fix/prompt
Fix/prompt
2026-07-22 11:51:43 +02:00
212 changed files with 158573 additions and 7594 deletions

No files matched your search

+76
View File
@@ -0,0 +1,76 @@
name: Bug report
description: Something in the client misbehaves
labels: ["bug"]
body:
- type: markdown
attributes:
value: |
Please check [`docs/BUGS.md`](../blob/main/docs/BUGS.md) first — the issue may already be
tracked there with context.
**Do not report a security vulnerability here.** See [SECURITY.md](../blob/main/SECURITY.md).
- type: textarea
id: what
attributes:
label: What happened
description: What you did, what you expected, and what happened instead.
validations:
required: true
- type: textarea
id: repro
attributes:
label: Steps to reproduce
placeholder: |
1. …
2. …
validations:
required: true
- type: input
id: version
attributes:
label: Version
description: Output of `pdfs --version`, and how you installed it (deb / rpm / AUR / source).
validations:
required: true
- type: input
id: environment
attributes:
label: Distribution and desktop
placeholder: "Fedora 44, GNOME 48 (Wayland)"
validations:
required: true
- type: dropdown
id: surface
attributes:
label: Where does it happen
multiple: true
options:
- FUSE mount / file operations
- Sync or backed-up folders (Computers)
- CLI (pdfs)
- Desktop app (pdfs-app)
- Tray (pdfs-tray)
- Search launcher (pdfs-prompt)
- Login / keyring
- Photos
- Packaging / installation
validations:
required: true
- type: textarea
id: diagnose
attributes:
label: "`pdfs diagnose` output"
description: Redact paths or the account address if you prefer.
render: console
- type: textarea
id: logs
attributes:
label: Daemon logs
description: "`journalctl --user -u proton-drive.service -n 200 --no-pager`"
render: console
- type: checkboxes
id: dataloss
attributes:
label: Data
options:
- label: This bug involved missing, truncated, or overwritten file content.
+11
View File
@@ -0,0 +1,11 @@
blank_issues_enabled: false
contact_links:
- name: Security vulnerability
url: https://github.com/narrrl/proton-drive-linux/security/advisories/new
about: Report privately — never in a public issue.
- name: Known issues
url: https://github.com/narrrl/proton-drive-linux/blob/main/docs/BUGS.md
about: The authoritative ledger of tracked bugs and their status.
- name: Proton Drive service problems
url: https://proton.me/support
about: This is an unofficial client. Account or service issues go to Proton AG.
@@ -0,0 +1,37 @@
name: Feature request
description: Suggest a capability or an improvement
labels: ["enhancement"]
body:
- type: textarea
id: problem
attributes:
label: The problem
description: What are you trying to do that the client makes hard or impossible today?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: What you'd like to see
validations:
required: true
- type: dropdown
id: surface
attributes:
label: Which part of the client
multiple: true
options:
- FUSE mount
- Sync / Computers
- CLI (pdfs)
- Desktop app (pdfs-app)
- Tray (pdfs-tray)
- Search launcher (pdfs-prompt)
- Photos
- Packaging
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives you considered
+26
View File
@@ -0,0 +1,26 @@
## What this changes
<!-- The user-visible effect, in a sentence or two. -->
**Crates touched:**
**Related `docs/BUGS.md` entries:**
## Verification
```bash
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
```
<!-- Plus whatever else you ran: acceptance suite, a live mount, manual steps. -->
## Implications
- [ ] Adds a SQLite migration (new `MIGRATION_V*`, `SCHEMA_VERSION` bumped — no shipped migration edited)
- [ ] Touches recovery, staging, or anything that can be the only copy of user data
- [ ] Changes packaging, the systemd unit, or desktop entries
- [ ] Changes the control protocol (daemon and front ends updated together)
- [ ] None of the above
<!-- Screenshots for GTK changes. -->
+101
View File
@@ -0,0 +1,101 @@
name: AUR
on:
# The -git package computes its pkgver from git at build time, so a new commit
# upstream needs no AUR push. Only a changed -git recipe is published.
push:
branches: [main]
paths:
- 'packaging/aur/proton-drive-for-linux-git/**'
# Called by release.yml once the GitHub release and its assets exist.
workflow_call:
inputs:
version:
required: true
type: string
secrets:
AUR_SSH_PRIVATE_KEY:
required: true
workflow_dispatch:
inputs:
version:
description: 'Release to publish, such as 2.1.0. Leave empty to publish only the -git package.'
required: false
type: string
permissions:
contents: write
concurrency:
group: aur
cancel-in-progress: false
jobs:
publish:
name: Publish AUR packages
runs-on: ubuntu-24.04
container: archlinux:base-devel
env:
# makepkg refuses to run as root, so the packaging steps run as `builder`.
HOME: /home/builder
VERSION: ${{ inputs.version }}
steps:
- name: Install packaging tools
run: pacman -Syu --noconfirm --needed git openssh pacman-contrib
- name: Checkout repository
uses: actions/checkout@v7
with:
# The updated recipes are committed back to main, not to the tag.
ref: main
- name: Create build user
run: |
useradd --create-home builder
chown -R builder: "$GITHUB_WORKSPACE"
- name: Configure AUR access
env:
AUR_SSH_PRIVATE_KEY: ${{ secrets.AUR_SSH_PRIVATE_KEY }}
run: |
install -d -m700 -o builder -g builder "$HOME/.ssh"
printf '%s\n' "$AUR_SSH_PRIVATE_KEY" > "$HOME/.ssh/aur"
cat > "$HOME/.ssh/config" <<'EOF'
Host aur.archlinux.org
IdentityFile ~/.ssh/aur
User aur
EOF
# Pin the host key to the fingerprint published on aur.archlinux.org.
ssh-keyscan -t ed25519 aur.archlinux.org > "$HOME/.ssh/known_hosts"
ssh-keygen -lf "$HOME/.ssh/known_hosts" | grep -q 'SHA256:RFzBCUItH9LZS0cKB5UE6ceAYhBD5C8GeOBip8Z11+4'
chown -R builder: "$HOME/.ssh"
chmod 600 "$HOME/.ssh/aur"
runuser -u builder -- git config --global user.name 'Nils Pukropp'
runuser -u builder -- git config --global user.email 'contact@narl.io'
- name: Update release packages
if: env.VERSION != ''
run: runuser -u builder -- packaging/aur/update.sh "${VERSION#v}"
- name: Refresh the -git metadata
if: env.VERSION == ''
working-directory: packaging/aur/proton-drive-for-linux-git
run: runuser -u builder -- sh -c 'makepkg --printsrcinfo > .SRCINFO'
- name: Publish to the AUR
run: |
if [ -n "$VERSION" ]; then
runuser -u builder -- packaging/aur/push.sh
else
runuser -u builder -- packaging/aur/push.sh proton-drive-for-linux-git
fi
- name: Commit the updated recipes to main
if: env.VERSION != ''
run: |
runuser -u builder -- git add packaging/aur
if ! runuser -u builder -- git diff --cached --quiet; then
runuser -u builder -- git commit -m "chore(aur): update packages to ${VERSION#v}"
runuser -u builder -- git push origin HEAD:main
fi
+1 -1
View File
@@ -13,7 +13,7 @@ jobs:
rust: rust:
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
steps: steps:
- uses: actions/checkout@v4 - uses: actions/checkout@v7
- name: Install native dependencies - name: Install native dependencies
run: | run: |
+44 -9
View File
@@ -13,10 +13,11 @@ jobs:
build-release: build-release:
name: Build and Release name: Build and Release
runs-on: ubuntu-24.04 runs-on: ubuntu-24.04
environment: 'rpm signing'
steps: steps:
- name: Checkout repository - name: Checkout repository
uses: actions/checkout@v4 uses: actions/checkout@v7
- name: Install system dependencies - name: Install system dependencies
run: | run: |
@@ -30,6 +31,7 @@ jobs:
libsecret-1-dev \ libsecret-1-dev \
libdbus-1-dev \ libdbus-1-dev \
desktop-file-utils \ desktop-file-utils \
gettext \
rpm rpm
- name: Set up Rust - name: Set up Rust
@@ -83,6 +85,7 @@ jobs:
cp target/release/pdfs-tray dist/binaries/ cp target/release/pdfs-tray dist/binaries/
cp target/release/pdfs-app dist/binaries/ cp target/release/pdfs-app dist/binaries/
cp target/release/pdfs-prompt dist/binaries/ cp target/release/pdfs-prompt dist/binaries/
po/build.sh dist/binaries/locale
# Create tarball # Create tarball
tar -czf proton-drive-linux-${{ env.VERSION }}-x86_64.tar.gz -C dist/binaries . tar -czf proton-drive-linux-${{ env.VERSION }}-x86_64.tar.gz -C dist/binaries .
@@ -109,6 +112,7 @@ jobs:
cp packaging/io.narl.proton-drive-linux.svg deb/usr/share/icons/hicolor/scalable/apps/ cp packaging/io.narl.proton-drive-linux.svg deb/usr/share/icons/hicolor/scalable/apps/
cp packaging/proton-drive.service deb/usr/lib/systemd/user/ cp packaging/proton-drive.service deb/usr/lib/systemd/user/
cp LICENSE deb/usr/share/licenses/proton-drive-linux/ cp LICENSE deb/usr/share/licenses/proton-drive-linux/
po/build.sh deb/usr/share/locale
# Calculate Installed-Size in KB for the control file # Calculate Installed-Size in KB for the control file
INSTALLED_SIZE=$(du -sk deb | cut -f1) INSTALLED_SIZE=$(du -sk deb | cut -f1)
@@ -121,13 +125,14 @@ jobs:
Priority: optional Priority: optional
Architecture: amd64 Architecture: amd64
Maintainer: Nils Pukropp <contact@narl.io> Maintainer: Nils Pukropp <contact@narl.io>
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4 Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4, libimage-exiftool-perl
Installed-Size: ${INSTALLED_SIZE} Installed-Size: ${INSTALLED_SIZE}
Description: Proton Drive client for Linux Homepage: https://github.com/narrrl/proton-drive-linux
Proton Drive client for Linux featuring: Description: Unofficial Proton Drive client for Linux
- FUSE files-on-demand mount Files-on-demand Proton Drive for the Linux desktop, unaffiliated with Proton AG:
- Command-line interface (CLI) - FUSE virtual mount with block-level caching
- GTK4 system tray, desktop application, and search launcher - Scriptable command-line interface (pdfs)
- GTK4 desktop application, system tray, and search launcher
EOF EOF
# Build Debian package # Build Debian package
@@ -154,9 +159,29 @@ jobs:
# Copy built RPM to root directory # Copy built RPM to root directory
cp rpmbuild/RPMS/x86_64/*.rpm . cp rpmbuild/RPMS/x86_64/*.rpm .
- name: Sign Fedora package (.rpm)
env:
GPG_PRIVATE_KEY: ${{ secrets.GPG_PRIVATE_KEY }}
GPG_KEY_NAME: ${{ secrets.GPG_KEY_NAME }}
GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }}
run: |
if [ -n "$GPG_PRIVATE_KEY" ] && [ -n "$GPG_KEY_NAME" ]; then
echo "$GPG_PRIVATE_KEY" | gpg --batch --pinentry-mode loopback --passphrase "$GPG_PASSPHRASE" --import
cat <<EOF > ~/.rpmmacros
%_signature gpg
%_gpg_name ${GPG_KEY_NAME}
%__gpg_sign_cmd %{__gpg} gpg --batch --verbose --no-armor --pinentry-mode loopback --passphrase %{_gpg_passphrase} --no-secmem-warning -u "%{_gpg_name}" -sbo %{__signature_filename} --digest-algo sha256 %{__plaintext_filename}
EOF
rpm --addsign --define "_gpg_passphrase $GPG_PASSPHRASE" *.rpm
else
echo "Skipping RPM signing: GPG_PRIVATE_KEY or GPG_KEY_NAME secret is not set."
fi
- name: Upload Workflow Artifacts - name: Upload Workflow Artifacts
uses: actions/upload-artifact@v4 uses: actions/upload-artifact@v7
with: with:
name: proton-drive-linux-builds name: proton-drive-linux-builds
path: | path: |
@@ -167,7 +192,7 @@ jobs:
- name: Create GitHub Release - name: Create GitHub Release
if: startsWith(github.ref, 'refs/tags/') if: startsWith(github.ref, 'refs/tags/')
uses: softprops/action-gh-release@v2 uses: softprops/action-gh-release@v3
with: with:
files: | files: |
*.deb *.deb
@@ -177,3 +202,13 @@ jobs:
prerelease: false prerelease: false
env: env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
aur:
name: Publish AUR packages
needs: build-release
if: startsWith(github.ref, 'refs/tags/')
uses: ./.github/workflows/aur.yml
with:
version: ${{ github.ref_name }}
secrets:
AUR_SSH_PRIVATE_KEY: ${{ secrets.AUR_SSH_PRIVATE_KEY }}
+32
View File
@@ -0,0 +1,32 @@
# Code of Conduct
## Our pledge
We want participation in this project to be a harassment-free experience for everyone, regardless
of age, body size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socioeconomic status, nationality, personal appearance, race,
religion, or sexual identity and orientation.
## Standards
Examples of behavior that helps:
- being respectful of differing viewpoints and experience levels,
- giving and accepting constructive feedback gracefully,
- focusing on what is best for the project and its users.
Examples of unacceptable behavior:
- sexualized language or imagery, and unwelcome sexual attention,
- trolling, insulting or derogatory comments, and personal or political attacks,
- public or private harassment,
- publishing others' private information without explicit permission.
## Scope and enforcement
This applies in all project spaces — issues, pull requests, discussions — and when representing the
project in public. Report unacceptable behavior to <contact@narl.io>. Reports are handled
confidentially. Maintainers may edit, remove, or reject contributions that violate this document,
and may temporarily or permanently ban any contributor for behavior they deem inappropriate.
Adapted from the [Contributor Covenant](https://www.contributor-covenant.org), version 2.1.
+70
View File
@@ -0,0 +1,70 @@
# Contributing
Thanks for taking the time. This is an unofficial Proton Drive client that people trust with real
files, so the bar for changes that touch the filesystem, the sync engine, or the database is
deliberately high — everything else is ordinary Rust review.
## Before you start
- Read [`AGENTS.md`](AGENTS.md) for repository conventions and
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for how the daemon, front ends, and SDK fit
together.
- Check [`docs/BUGS.md`](docs/BUGS.md) — it is the authoritative issue ledger, and your problem may
already be tracked there with context.
- For anything larger than a fix, open an issue first so the design can be discussed before you
spend the time.
## Development setup
```bash
git clone https://github.com/narrrl/proton-drive-linux.git
cd proton-drive-linux
cargo build --workspace
```
GUI and FUSE targets need the GTK4, libadwaita, WebKitGTK 6, libsecret, D-Bus, and FUSE3
development packages listed in the [README](README.md#prerequisites). Minimum supported Rust
version is 1.96.
## Quality gates
CI runs exactly these, and they must pass with zero warnings:
```bash
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only
```
Changes to the FUSE layer, sync, or the pending-operation queue should also be exercised against a
real mount before review — see [`docs/TESTING.md`](docs/TESTING.md). Live runs are destructive:
use a dedicated test account, never one holding data you cannot lose.
## Making changes
- **Keep shared logic in `pdfs-core`.** The CLI and GUI never touch the database or the network;
they speak the control protocol to the daemon.
- **The SQLite schema is forward-only.** Never edit a shipped migration — add a new one and bump
`SCHEMA_VERSION` in `crates/pdfs-core/src/db/migrations.rs`.
- **Typed errors and `?` over panics** on runtime paths. Blocking filesystem, keyring, or
control-socket work stays off GTK's main thread.
- **`staging/` and `recovery/` may hold the only copy of user data.** Nothing may clear or purge
them opportunistically.
## Commits and pull requests
Commits follow scoped [Conventional Commits](https://www.conventionalcommits.org), e.g.
`fix(fuse,core): keep truncate composable with queued revisions`.
A pull request should state:
- the user-visible effect and which crates it touches,
- the `docs/BUGS.md` entries it fixes or affects,
- the commands you ran to verify it,
- screenshots for GTK changes,
- and, explicitly, any migration, recovery, data-loss, or packaging implications.
## Security
Do not open a public issue for a vulnerability — see [SECURITY.md](SECURITY.md).
Generated
+715 -811
View File
File diff suppressed because it is too large. Load diff
+26 -8
View File
@@ -3,15 +3,20 @@ resolver = "2"
members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"] members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"]
[workspace.package] [workspace.package]
version = "1.0.0" version = "2.5.0"
edition = "2024" edition = "2024"
license = "MIT" license = "MIT"
authors = ["Nils Pukropp <contact@narl.io>"] authors = ["Nils Pukropp <contact@narl.io>"]
rust-version = "1.96" rust-version = "1.96"
repository = "https://github.com/narrrl/proton-drive-linux"
homepage = "https://github.com/narrrl/proton-drive-linux"
readme = "README.md"
keywords = ["proton", "drive", "fuse", "sync", "linux"]
categories = ["filesystem", "command-line-utilities"]
[workspace.dependencies] [workspace.dependencies]
proton-sdk = { version = "0.1" } proton-sdk = { version = "0.7.0" }
proton-drive-rs = { version = "0.1" } proton-drive-rs = { version = "0.7.0" }
# Internal crates. # Internal crates.
pdfs-core = { path = "crates/pdfs-core" } pdfs-core = { path = "crates/pdfs-core" }
@@ -20,6 +25,7 @@ pdfs-fuse = { path = "crates/pdfs-fuse" }
tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "fs", "io-util", "time", "signal"] } tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "fs", "io-util", "time", "signal"] }
serde = { version = "1", features = ["derive"] } serde = { version = "1", features = ["derive"] }
serde_json = "1" serde_json = "1"
base64 = "0.23"
thiserror = "2" thiserror = "2"
anyhow = "1" anyhow = "1"
# Non-poisoning mutexes. A handler that panics while holding a lock must not # Non-poisoning mutexes. A handler that panics while holding a lock must not
@@ -29,16 +35,28 @@ anyhow = "1"
parking_lot = "0.12" parking_lot = "0.12"
tracing = "0.1" tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] } tracing-subscriber = { version = "0.3", features = ["env-filter"] }
directories = "5" directories = "6"
ignore = "0.4" ignore = "0.4"
sha2 = "0.10" sha2 = "0.11"
# Google Takeout exports are zip sets; the importer reads their central
# directories and streams entries out without extracting to disk first.
zip = { version = "8", default-features = false, features = ["deflate"] }
# Proton's photo duplicate detection is keyed on the plaintext SHA-1, so an
# import has to speak SHA-1 whatever its age.
sha1 = "0.11"
# Thumbnail generation for photos the server has none for. Decoders only for the # Thumbnail generation for photos the server has none for. Decoders only for the
# formats a camera or phone actually produces — the rest of the crate's codecs # formats a camera or phone actually produces — the rest of the crate's codecs
# would be dead weight in the daemon. # would be dead weight in the daemon.
image = { version = "0.25", default-features = false, features = ["jpeg", "png", "webp", "gif", "tiff", "bmp", "rayon"] } image = { version = "0.25", default-features = false, features = ["jpeg", "png", "webp", "gif", "tiff", "bmp", "rayon"] }
rusqlite = { version = "0.32", features = ["bundled"] } rusqlite = { version = "0.40", features = ["bundled"] }
keyring = { version = "3", features = ["sync-secret-service", "crypto-rust"] } # keyring 4 split into `keyring-core` plus per-platform stores. The libdbus
# Secret Service store is the one keyring 3's `sync-secret-service` wrapped, and
# with no `target` it matches items v3 wrote, so existing logins survive.
keyring-core = "1"
dbus-secret-service-keyring-store = { version = "1", features = ["crypto-rust"] }
clap = { version = "4", features = ["derive"] } clap = { version = "4", features = ["derive"] }
rpassword = "7" rpassword = "7"
fuser = "0.17" async-trait = "0.1"
futures = "0.3"
fuser = "0.18"
libc = "0.2" libc = "0.2"
+255 -15
View File
@@ -1,20 +1,209 @@
# Proton Drive client for Linux (unofficial) <div align="center">
A fast, unofficial Proton Drive client for Linux. This client features an advanced files-on-demand FUSE virtual mount with block-level caching, a command-line interface (CLI), and a fully non-blocking GTK4 desktop application with system tray integration. <img src="images/logo.svg" alt="" width="112" height="112">
# Proton Drive for Linux
**Files-on-demand Proton Drive for the Linux desktop** — a FUSE virtual mount with block-level
caching, a scriptable CLI, and a native GTK4 app with tray and search launcher.
[![CI](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml/badge.svg)](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/v/release/narrrl/proton-drive-linux?sort=semver)](https://github.com/narrrl/proton-drive-linux/releases/latest)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust 1.96+](https://img.shields.io/badge/rust-1.96%2B-orange.svg)](https://www.rust-lang.org)
[![Platform: Linux](https://img.shields.io/badge/platform-linux-lightgrey.svg)](#prerequisites)
<img src="images/files.png" alt="The GTK4 file browser" width="820">
</div>
> [!IMPORTANT]
> This is an **unofficial**, community-built client. It is not affiliated with, endorsed by, or
> supported by Proton AG. "Proton" and "Proton Drive" are trademarks of Proton AG, used here only
> to describe what the software talks to. Use a test account first, and read
> [`docs/RECOVERY.md`](docs/RECOVERY.md) before trusting it with data you cannot lose.
## Install
```bash
# Debian / Ubuntu
sudo apt install ./proton-drive-linux_*.deb
# Fedora
sudo dnf install ./proton-drive-linux-*.rpm
# Arch Linux (AUR)
paru -S proton-drive-for-linux # or -git / -bin
```
Packages for each release are attached to the [latest
release](https://github.com/narrrl/proton-drive-linux/releases/latest); see
[Installation & Packages](#installation--packages) for details and
[Building from Source](#building-from-source) to compile it yourself.
```bash
pdfs login # sign in (SRP + 2FA, credentials go to the system keyring)
pdfs status # mount state, queued writes, cache usage
pdfs-app # the desktop app
```
## Contents
- [Features](#features) · [Quick search](#quick-search-and-media-opening) · [Opening files](#choosing-how-files-open)
- [Selective sync](#selective-sync-pdfsignore) · [Diagnostics](#diagnostics--maintenance) · [Scripting](#scripting---json)
- [Performance](#performance--caching) · [Filesystem safety](#filesystem-safety) · [Screenshots](#screenshots)
- [Prerequisites](#prerequisites) · [Building](#building-from-source) · [Packages](#installation--packages) · [Releases](#automated-releases-cicd)
**Documentation:** [Architecture](docs/ARCHITECTURE.md) · [Testing](docs/TESTING.md) ·
[Recovery](docs/RECOVERY.md) · [Development](docs/DEVELOPMENT.md) · [Translating](docs/TRANSLATING.md) ·
[Known issues](docs/BUGS.md) · [Changelog](docs/CHANGELOG.md) ·
[Contributing](CONTRIBUTING.md) · [Security policy](SECURITY.md)
## Features ## Features
- **Files-on-Demand (FUSE)**: Mount your Proton Drive as a virtual filesystem where files are downloaded only when opened, utilizing smart block-level caching and disk-backed writes. - **Files-on-Demand (FUSE)**: Mount your Proton Drive as a virtual filesystem where files are downloaded only when opened, utilizing smart block-level caching and disk-backed writes.
- **Command-Line Interface (CLI)**: Manage your drive, authenticate, and monitor sync status directly from the terminal. - **Command-Line Interface (CLI)**: Manage your drive, authenticate, and monitor sync status directly from the terminal.
- **Non-Blocking GTK4 Desktop App**: Browse files, manage pins, configure options, and monitor active transfers through a modern, native GUI with a fully non-blocking asynchronous main loop. - **Non-Blocking GTK4 Desktop App**: Browse files, manage pins, configure options, and monitor active transfers through a modern, native GUI with a fully non-blocking asynchronous main loop.
- **File Thumbnails**: Browse locally cached image previews in Files, Shared, and Trash, including embedded previews from camera RAW files. Build a whole folder tree from the Files toolbar and resize the grid from the status bar.
- **Shared Links & Invites**: Browse files shared with you by other users, and view/manage your own public shared links directly in the GUI. - **Shared Links & Invites**: Browse files shared with you by other users, and view/manage your own public shared links directly in the GUI.
- **Local Backup (Computers)**: Sync and back up local directories (like Downloads, Documents, Pictures, etc.) directly to your Proton Drive account. - **Local Backup (Computers)**: Sync and back up local directories (like Downloads, Documents, Pictures, etc.) directly to your Proton Drive account.
- **Locations**: One page (and `pdfs locations`) listing every place Proton Drive occupies on this machine — the main mount plus each backed-up folder — with its mode, sync state, and whether it is read-only. Switching a folder between a full local copy and on-demand happens here.
- **System Tray Integration**: Background indicator for status monitoring, quick actions, and fast sync controls. - **System Tray Integration**: Background indicator for status monitoring, quick actions, and fast sync controls.
- **Fuzzy Search Launcher Prompt (HUD)**: A Google Drive-style search launcher (`pdfs-prompt`) for finding and opening files or folders instantly, ideal for binding to a system-wide hotkey. - **Unified Search Launcher (HUD)**: A resident Google Drive-style launcher (`pdfs-prompt`) that searches Proton Drive and local files together, ranks the best matches, tolerates abbreviations and typos, and is ideal for a system-wide hotkey. It can also present the same search in your own launcher (`--dmenu`) or, searching live as you type, in `fzf` (`--fzf`).
- **Secure Credential Storage**: Integrates with the system Secret Service (GNOME Keyring, KWallet, etc.) with smart in-memory credential caching to avoid UI thread blockages. - **Secure Credential Storage**: Integrates with the system Secret Service (GNOME Keyring, KWallet, etc.) with smart in-memory credential caching to avoid UI thread blockages.
- **Proton Photos Support**: Access your Proton Photos timeline, view thumbnails, and download backed-up media natively (available in the GUI as a navigation tab and via the CLI). - **Proton Photos Support**: Access your Proton Photos timeline, view thumbnails, and download backed-up media natively (available in the GUI as a navigation tab and via the CLI).
- **File Version History**: Every revision Proton Drive still holds for a file, from the browser's details pane (**Versions**) or `pdfs versions list|restore|save|rm`. Restoring is server-side — no re-upload — and an old version can be written out to a local file without touching the live one.
- **Photo Favourites**: Star a photo in the lightbox, filter the gallery to favourites, or use `pdfs favorite <uid>` / `pdfs photos --favorites`.
- **Photo Albums**: Browse your albums — including the ones shared with you — from the Albums view of the Photos page, or with `pdfs albums` / `pdfs album <uid>`. Album contents open in the same gallery as the timeline.
- **Places**: The Places view of the Photos page groups photos by the town they were taken in, using the location stored with each photo, as cards or on a world map. Towns are looked up offline in [GeoNames](https://www.geonames.org/) data (CC BY 4.0), and the map is drawn offline from [Natural Earth](https://www.naturalearthdata.com/) land outlines and country borders (public domain). Each town is marked with one of its photos, and zoomed in, every photo sits where it was taken. A street map from [OpenFreeMap](https://openfreemap.org/) (© [OpenStreetMap](https://www.openstreetmap.org/copyright) contributors) can be turned on under Appearance; it is off by default because the tile server learns which parts of the world you look at. Install `iso-codes` for country names in your language.
- **Human Verification (CAPTCHA) Recovery**: Detects sign-in gates (VPN/new IP challenges) and launches an embedded `WebKitWebView` dialog to safely complete the challenge, transparently retrying authentication with the earned token. - **Human Verification (CAPTCHA) Recovery**: Detects sign-in gates (VPN/new IP challenges) and launches an embedded `WebKitWebView` dialog to safely complete the challenge, transparently retrying authentication with the earned token.
- **Selective Sync (`.pdfsignore`)**: Keep build trees, dependency directories, and editor leftovers out of synced folders using gitignore-style rules. - **Selective Sync (`.pdfsignore`)**: Keep build trees, dependency directories, and editor leftovers out of synced folders using gitignore-style rules.
- **Data-Safe Offline Writes**: Durable scratch/staging files and a transactional pending queue preserve acknowledged writes across network failures and restarts.
## Quick Search and Media Opening
`pdfs-prompt` searches Drive metadata and the local home-directory index in one request. Results are ranked together rather than split by source. Matching is case-insensitive and supports exact names, prefixes, parent paths, multiple terms, punctuation-separated words, ordered abbreviations, common typos, and adjacent transpositions.
The prompt stays resident after its first launch, so invoking the shortcut again reuses and resets the existing window. Drive folders open from the mount. Audio and video results also open through FUSE, allowing media players to seek and request ranges without downloading the entire file first; ordinary Drive files are materialized into the local cache before opening.
### Using your own launcher (fuzzel, rofi, wofi, …)
`pdfs-prompt --dmenu` runs the same search through an external dmenu-style
launcher instead of the built-in window, so the HUD matches the rest of a
tiling-WM desktop:
```bash
pdfs-prompt --dmenu # fuzzel/rofi/wofi/tofi/bemenu/dmenu, whichever is installed
pdfs-prompt --dmenu --menu 'fuzzel --dmenu --width 60'
pdfs-prompt --dmenu --query invoice # skip straight to a search
```
A launcher filters a fixed list — it cannot ask for a new one per keystroke — so
searching takes two steps, and the prompt says which step you are on:
- `Search Drive ›` lists your pinned files. Type anything that isn't one of them
and press Enter to search for it.
- `Drive: invoice ›` lists the results. Enter opens one; typing something else
and pressing Enter searches again. Escape closes.
`--query` skips the first step. fuzzel and rofi also get file-type icons.
To make an existing keybinding use it without re-binding, and to pin the
launcher command, set them in `config.json`:
```json
{
"prompt": {
"mode": "dmenu",
"menu": ["fuzzel", "--dmenu", "--width", "60"],
"menu_limit": 50
}
}
```
`--gtk` overrides `"mode": "dmenu"` for one invocation. A `{prompt}` token
anywhere in `menu` is replaced by the prompt text; without one, the launcher's
own prompt flag is appended.
### Live launcher search (fzf)
The two-step above exists because no dmenu-style launcher can call back for new
results per keystroke. `fzf` can, so `pdfs-prompt --fzf` searches as you type,
exactly like the built-in window:
```bash
pdfs-prompt --fzf # pinned files, then live results as you type
pdfs-prompt --fzf --query invoice # start on a search
```
Every keystroke re-queries the daemon (debounced, like the GTK prompt) and fzf's
own matcher is switched off, so the ordering you see is the daemon's ranking —
abbreviations and typos included, which a launcher filtering a fixed list cannot
do. Enter opens; Escape closes.
fzf is a terminal program, so when it is started without one — from a WM
keybinding — it spawns a terminal for itself: foot, ghostty, kitty, alacritty,
wezterm or xterm, whichever is installed. Each is launched with the app-id/class
`pdfs-prompt`, so one window rule floats it whichever terminal you have. In
Hyprland:
```
windowrulev2 = float, class:^(pdfs-prompt)$
windowrulev2 = size 900 500, class:^(pdfs-prompt)$
windowrulev2 = center, class:^(pdfs-prompt)$
bind = SUPER, space, exec, pdfs-prompt --fzf
```
Set it as the default front end, and pick the terminal explicitly, in
`config.json`:
```json
{
"prompt": {
"mode": "fzf",
"terminal": ["foot", "--app-id=pdfs-prompt", "--"],
"menu_limit": 50
}
}
```
A `{cmd}` token anywhere in `terminal` is replaced by the command to run;
without one it is appended, which is what every detected terminal expects.
## Choosing How Files Open
By default every result is handed to `xdg-open`. An `open_with` block in
`config.json` overrides that per file type — for instance opening text in
Neovim inside Alacritty, while everything else still goes to the desktop:
```json
{
"open_with": {
"terminal": ["alacritty", "-e"],
"rules": [
{ "match": ["@text"], "command": ["nvim"], "terminal": true },
{ "match": ["*.png", "*.jpg"], "command": ["imv"] }
]
}
}
```
- `match` takes file-name globs (`*.md`, `notes-*.txt`) or the classes `@dir`,
`@text`, `@document`, `@image`, `@media`, `@any`. First matching rule wins.
- `command` is argv, not a shell line. A `{}` token is replaced by the path;
without one the path is appended. `$VAR` tokens expand from the environment,
so `["$EDITOR"]` follows your editor.
- `"terminal": true` wraps the command in `terminal`. When that is unset, the
terminal comes from `$PDFS_TERMINAL`, then `$TERMINAL`, then the first known
emulator on `PATH`; a bare name like `TERMINAL=alacritty` gains the right
"run this" flag automatically.
- `"default": [...]` replaces `xdg-open` for everything unmatched.
The rules apply to the prompt (both front ends) and to the GTK browser, so a
file opens the same way wherever it was picked. Patterns are matched against the
Drive name — a downloaded file is stored in the cache under its content hash,
which says nothing about its type, so matching that would send everything to
`xdg-open`.
## Selective Sync (`.pdfsignore`) ## Selective Sync (`.pdfsignore`)
@@ -131,6 +320,19 @@ The client includes several optimizations designed for high efficiency, a low me
- **Disk-Backed Writes**: Large file writes are staged on disk in temporary scratch files (rather than fully buffered in RAM) and track modified byte intervals. Only the unedited remote segments are fetched at commit time, keeping memory usage minimal. - **Disk-Backed Writes**: Large file writes are staged on disk in temporary scratch files (rather than fully buffered in RAM) and track modified byte intervals. Only the unedited remote segments are fetched at commit time, keeping memory usage minimal.
- **Non-Blocking GTK4 Loop**: To prevent UI freezes, all synchronous D-Bus credential checks, control socket requests, and cache usage calculations are offloaded to background worker threads or fetched asynchronously. - **Non-Blocking GTK4 Loop**: To prevent UI freezes, all synchronous D-Bus credential checks, control socket requests, and cache usage calculations are offloaded to background worker threads or fetched asynchronously.
- **Flicker-Free UI Rendering**: The GTK4 application performs differential rendering of the pins list, only modifying list rows when the list content changes, preserving the user's scroll position and widget focus. - **Flicker-Free UI Rendering**: The GTK4 application performs differential rendering of the pins list, only modifying list rows when the list content changes, preserving the user's scroll position and widget focus.
- **Durable Staging**: Scratch metadata and staged writes are synced and atomically published before an upload is acknowledged locally. Rapid revisions supersede pending work transactionally.
## Filesystem Safety
Version 1.0 strengthens the boundaries around local-only data and destructive reconciliation:
- Truncate composes with queued revisions, preserving the authoritative prefix and correct zero-filled growth.
- Combined move-and-rename operations are queued durably and tolerate partially completed remote state.
- Session tombstones prevent successfully unlinked entries from reappearing through eventually consistent listings.
- Names are valid UTF-8 and no longer than Linux's 255-byte component limit.
- Incomplete mirror scans are non-destructive, and total-wipe protection covers every non-empty baseline.
- Failed conflict preservation, local deletion, or staging publication does not settle a new baseline or discard the only local copy.
- State, cache, and config directories must be real, current-user-owned `0700` directories; control sockets are `0600`.
## Screenshots ## Screenshots
@@ -147,11 +349,15 @@ The client includes several optimizations designed for high efficiency, a low me
<td align="center" width="50%"><img src="images/shared_with_me.png" alt="Shared with me" width="100%"><br><sub><b>Shared with me</b></sub></td> <td align="center" width="50%"><img src="images/shared_with_me.png" alt="Shared with me" width="100%"><br><sub><b>Shared with me</b></sub></td>
</tr> </tr>
<tr> <tr>
<td align="center" width="50%"><img src="images/computers.png" alt="Local Backups (Computers)" width="100%"><br><sub><b>Local Backups (Computers)</b></sub></td> <td align="center" width="50%"><img src="images/restore_devices.png" alt="Local Backups (Computers)" width="100%"><br><sub><b>Local Backups (Computers)</b></sub></td>
<td align="center" width="50%"><img src="images/photos.png" alt="Photos Timeline" width="100%"><br><sub><b>Photos Timeline</b></sub></td> <td align="center" width="50%"><img src="images/photos.png" alt="Photos Timeline" width="100%"><br><sub><b>Photos Timeline</b></sub></td>
</tr> </tr>
<tr> <tr>
<td align="center" width="50%"><img src="images/prompt.png" alt="Search Launcher Prompt" width="100%"><br><sub><b>Search Launcher Prompt</b></sub></td> <td align="center" width="50%"><img src="images/prompt_gtk.png" alt="Search Launcher Prompt (GTK)" width="100%"><br><sub><b>Search Launcher Prompt (GTK)</b></sub></td>
<td align="center" width="50%"><img src="images/prompt_fzf.png" alt="Search Launcher Prompt (fzf)" width="100%"><br><sub><b>Search Launcher Prompt (fzf)</b></sub></td>
</tr>
<tr>
<td align="center" width="50%"><img src="images/activity.png" alt="Activity" width="100%"><br><sub><b>Activity</b></sub></td>
<td align="center" width="50%"><img src="images/settings.png" alt="Settings" width="100%"><br><sub><b>Settings</b></sub></td> <td align="center" width="50%"><img src="images/settings.png" alt="Settings" width="100%"><br><sub><b>Settings</b></sub></td>
</tr> </tr>
</table> </table>
@@ -172,19 +378,21 @@ sudo apt-get install -y \
libadwaita-1-dev \ libadwaita-1-dev \
libwebkitgtk-6.0-dev \ libwebkitgtk-6.0-dev \
libsecret-1-dev \ libsecret-1-dev \
libdbus-1-dev libdbus-1-dev \
libimage-exiftool-perl
``` ```
### Arch Linux ### Arch Linux
```bash ```bash
sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus webkitgtk-6.0 sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus webkitgtk-6.0 perl-image-exiftool
``` ```
### Fedora (44+) ### Fedora (44+)
```bash ```bash
sudo dnf install -y \ sudo dnf install -y \
pkgconf-pkg-config fuse3-devel gtk4-devel libadwaita-devel \ pkgconf-pkg-config fuse3-devel gtk4-devel libadwaita-devel \
webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel cargo rust webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel \
perl-Image-ExifTool cargo rust
``` ```
Runtime extras (pick your desktop): Runtime extras (pick your desktop):
@@ -204,7 +412,7 @@ Ensure you have Rust and Cargo installed (minimum supported Rust version is 1.96
1. Clone the repository and navigate into the project directory: 1. Clone the repository and navigate into the project directory:
```bash ```bash
git clone https://github.com/narl/proton-drive-linux.git git clone https://github.com/narrrl/proton-drive-linux.git
cd proton-drive-linux cd proton-drive-linux
``` ```
2. Build the workspace in release mode: 2. Build the workspace in release mode:
@@ -228,8 +436,24 @@ Install the debian package via `dpkg` or `apt`:
sudo apt install ./proton-drive-linux_*.deb sudo apt install ./proton-drive-linux_*.deb
``` ```
### 2. Arch Linux ### 2. Arch Linux (AUR)
A local `PKGBUILD` is available under the `packaging/` directory. You can build and install it using: Three AUR packages cover the usual choices; they conflict with each other, so
install one:
| Package | Contents |
| --- | --- |
| [`proton-drive-for-linux`](https://aur.archlinux.org/packages/proton-drive-for-linux) | builds the latest tagged release from source |
| [`proton-drive-for-linux-git`](https://aur.archlinux.org/packages/proton-drive-for-linux-git) | builds `main` from git |
| [`proton-drive-for-linux-bin`](https://aur.archlinux.org/packages/proton-drive-for-linux-bin) | installs the release binaries, no compile |
```bash
paru -S proton-drive-for-linux # or any AUR helper
```
The package sources live in `packaging/aur/`; see
[`packaging/aur/README.md`](packaging/aur/README.md) for how they are updated and
published. A local `PKGBUILD` that compiles the working tree in place is still
available for development builds:
```bash ```bash
cd packaging && makepkg -fi cd packaging && makepkg -fi
``` ```
@@ -258,19 +482,35 @@ This project has a GitHub Actions CI workflow configured under `.github/workflow
1. **Triggers**: 1. **Triggers**:
- Pushing a git tag matching `v*` (e.g. `git tag v0.1.0 && git push origin v0.1.0`). - Pushing a git tag matching `v*` (e.g. `git tag v0.1.0 && git push origin v0.1.0`).
- Manual runs via the **Actions** tab in GitHub (**workflow_dispatch**). - Manual runs via the **Actions** tab in GitHub (**workflow_dispatch**).
2. **Build Process**: 2. **Quality Gates and Build Process**:
- For tagged builds, verifies that the tag, workspace version, and `packaging/PKGBUILD` version agree.
- Runs `cargo fmt`, Clippy with warnings denied, the locked workspace test suite, and the account-free FUSE acceptance contract.
- Spawns an Ubuntu runner and installs GTK4, Libadwaita, FUSE3, and Secret Service packages. - Spawns an Ubuntu runner and installs GTK4, Libadwaita, FUSE3, and Secret Service packages.
- Sets up the Rust compiler and caches build targets to speed up runs. - Sets up the Rust compiler and caches build targets to speed up runs.
- Compiles the workspace members in release mode. - Compiles the workspace members in release mode.
3. **Artifact Packaging**: 3. **Artifact Packaging**:
- Generates a `.tar.gz` containing the raw binaries (`pdfs`, `pdfs-app`, `pdfs-tray`, `pdfs-prompt`). - Generates a `.tar.gz` containing the raw binaries (`pdfs`, `pdfs-app`, `pdfs-tray`, `pdfs-prompt`).
- Packs them into a Debian package (`.deb`). - Packs them into Debian (`.deb`) and Fedora (`.rpm`) packages.
- Includes the systemd user service and tray autostart entry in the Debian package.
4. **Publishing**: 4. **Publishing**:
- Creates a GitHub Release matching the pushed tag and uploads the `.deb` and `.tar.gz` packages as release assets. - Creates a GitHub Release matching the pushed tag and uploads the `.deb`, `.rpm`, and `.tar.gz` packages as release assets.
- For manual runs, compiles and exposes the packages as workflow run artifacts for testing. - For manual runs, compiles and exposes the packages as workflow run artifacts for testing.
--- ---
## Contributing
Bug reports, packaging fixes, and patches are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for
the workflow and the quality gates CI enforces, and [SECURITY.md](SECURITY.md) for reporting a
vulnerability privately.
The app, tray and search prompt are available in English, German, Spanish, French, Italian, Dutch,
Polish and Brazilian Portuguese. [docs/TRANSLATING.md](docs/TRANSLATING.md) explains how to improve
a translation or add a language.
## License ## License
This project is licensed under the [MIT License](LICENSE). Released under the [MIT License](LICENSE).
Proton Drive is a service of Proton AG. This project is an independent client and carries no
affiliation with or endorsement from Proton AG.
+34
View File
@@ -0,0 +1,34 @@
# Security Policy
## Supported versions
Fixes land on the latest release. Please reproduce on the most recent version before reporting.
## Reporting a vulnerability
**Do not open a public issue.** Report privately through GitHub's
[private vulnerability reporting](https://github.com/narrrl/proton-drive-linux/security/advisories/new),
or by email to <contact@narl.io>.
Please include the version (`pdfs --version`), your distribution and desktop environment, what an
attacker gains, and a reproduction if you have one. You can expect an acknowledgement within a few
days; a fix ships in the next release, and you are credited in the advisory unless you ask
otherwise.
## Scope
This client holds Proton account credentials and decrypted file content on the local machine, so
the interesting boundaries are:
- **Credentials.** Sessions and the mailbox password live in the system Secret Service. Anything
that writes them elsewhere — logs, temp files, crash dumps — is a vulnerability.
- **The control socket.** Anything that can connect to `control.sock` has the daemon's
authenticated session. It is `0600`, and the state, cache, and config directories must be real,
current-user-owned `0700` directories; the daemon fails closed otherwise. Reports that defeat
those checks are in scope.
- **Local cache and staging.** Decrypted block content and staged writes on disk.
- **Cryptography.** Key derivation, signature handling, and anything that could publish plaintext
or the wrong key material to Proton.
Out of scope: vulnerabilities in Proton's own service (report those to Proton AG), and issues that
require an attacker who already has your user account on the machine.
+2
View File
@@ -5,6 +5,8 @@ edition.workspace = true
license.workspace = true license.workspace = true
authors.workspace = true authors.workspace = true
rust-version.workspace = true rust-version.workspace = true
repository.workspace = true
homepage.workspace = true
description = "CLI for the Proton Drive Linux client" description = "CLI for the Proton Drive Linux client"
[[bin]] [[bin]]
File diff suppressed because it is too large. Load diff
+7 -1
View File
@@ -5,6 +5,8 @@ edition.workspace = true
license.workspace = true license.workspace = true
authors.workspace = true authors.workspace = true
rust-version.workspace = true rust-version.workspace = true
repository.workspace = true
homepage.workspace = true
description = "Core auth/session/config for the Proton Drive Linux client" description = "Core auth/session/config for the Proton Drive Linux client"
[dependencies] [dependencies]
@@ -18,7 +20,11 @@ tracing.workspace = true
directories.workspace = true directories.workspace = true
ignore.workspace = true ignore.workspace = true
sha2.workspace = true sha2.workspace = true
sha1.workspace = true
zip.workspace = true
rusqlite.workspace = true rusqlite.workspace = true
keyring.workspace = true keyring-core.workspace = true
dbus-secret-service-keyring-store.workspace = true
parking_lot.workspace = true parking_lot.workspace = true
async-trait.workspace = true
libc.workspace = true libc.workspace = true
File diff suppressed because it is too large. Load diff
+250
View File
@@ -0,0 +1,250 @@
AD Andorra
AE United Arab Emirates
AF Afghanistan
AG Antigua and Barbuda
AI Anguilla
AL Albania
AM Armenia
AO Angola
AQ Antarctica
AR Argentina
AS American Samoa
AT Austria
AU Australia
AW Aruba
AX Åland Islands
AZ Azerbaijan
BA Bosnia and Herzegovina
BB Barbados
BD Bangladesh
BE Belgium
BF Burkina Faso
BG Bulgaria
BH Bahrain
BI Burundi
BJ Benin
BL Saint Barthélemy
BM Bermuda
BN Brunei Darussalam
BO Bolivia, Plurinational State of
BQ Bonaire, Sint Eustatius and Saba
BR Brazil
BS Bahamas
BT Bhutan
BV Bouvet Island
BW Botswana
BY Belarus
BZ Belize
CA Canada
CC Cocos (Keeling) Islands
CD Congo, The Democratic Republic of the
CF Central African Republic
CG Congo
CH Switzerland
CI Côte d'Ivoire
CK Cook Islands
CL Chile
CM Cameroon
CN China
CO Colombia
CR Costa Rica
CU Cuba
CV Cabo Verde
CW Curaçao
CX Christmas Island
CY Cyprus
CZ Czechia
DE Germany
DJ Djibouti
DK Denmark
DM Dominica
DO Dominican Republic
DZ Algeria
EC Ecuador
EE Estonia
EG Egypt
EH Western Sahara
ER Eritrea
ES Spain
ET Ethiopia
FI Finland
FJ Fiji
FK Falkland Islands (Malvinas)
FM Micronesia, Federated States of
FO Faroe Islands
FR France
GA Gabon
GB United Kingdom
GD Grenada
GE Georgia
GF French Guiana
GG Guernsey
GH Ghana
GI Gibraltar
GL Greenland
GM Gambia
GN Guinea
GP Guadeloupe
GQ Equatorial Guinea
GR Greece
GS South Georgia and the South Sandwich Islands
GT Guatemala
GU Guam
GW Guinea-Bissau
GY Guyana
HK Hong Kong
HM Heard Island and McDonald Islands
HN Honduras
HR Croatia
HT Haiti
HU Hungary
ID Indonesia
IE Ireland
IL Israel
IM Isle of Man
IN India
IO British Indian Ocean Territory
IQ Iraq
IR Iran, Islamic Republic of
IS Iceland
IT Italy
JE Jersey
JM Jamaica
JO Jordan
JP Japan
KE Kenya
KG Kyrgyzstan
KH Cambodia
KI Kiribati
KM Comoros
KN Saint Kitts and Nevis
KP Korea, Democratic People's Republic of
KR Korea, Republic of
KW Kuwait
KY Cayman Islands
KZ Kazakhstan
LA Lao People's Democratic Republic
LB Lebanon
LC Saint Lucia
LI Liechtenstein
LK Sri Lanka
LR Liberia
LS Lesotho
LT Lithuania
LU Luxembourg
LV Latvia
LY Libya
MA Morocco
MC Monaco
MD Moldova, Republic of
ME Montenegro
MF Saint Martin (French part)
MG Madagascar
MH Marshall Islands
MK North Macedonia
ML Mali
MM Myanmar
MN Mongolia
MO Macao
MP Northern Mariana Islands
MQ Martinique
MR Mauritania
MS Montserrat
MT Malta
MU Mauritius
MV Maldives
MW Malawi
MX Mexico
MY Malaysia
MZ Mozambique
NA Namibia
NC New Caledonia
NE Niger
NF Norfolk Island
NG Nigeria
NI Nicaragua
NL Netherlands
NO Norway
NP Nepal
NR Nauru
NU Niue
NZ New Zealand
OM Oman
PA Panama
PE Peru
PF French Polynesia
PG Papua New Guinea
PH Philippines
PK Pakistan
PL Poland
PM Saint Pierre and Miquelon
PN Pitcairn
PR Puerto Rico
PS Palestine, State of
PT Portugal
PW Palau
PY Paraguay
QA Qatar
RE Réunion
RO Romania
RS Serbia
RU Russian Federation
RW Rwanda
SA Saudi Arabia
SB Solomon Islands
SC Seychelles
SD Sudan
SE Sweden
SG Singapore
SH Saint Helena, Ascension and Tristan da Cunha
SI Slovenia
SJ Svalbard and Jan Mayen
SK Slovakia
SL Sierra Leone
SM San Marino
SN Senegal
SO Somalia
SR Suriname
SS South Sudan
ST Sao Tome and Principe
SV El Salvador
SX Sint Maarten (Dutch part)
SY Syrian Arab Republic
SZ Eswatini
TC Turks and Caicos Islands
TD Chad
TF French Southern Territories
TG Togo
TH Thailand
TJ Tajikistan
TK Tokelau
TL Timor-Leste
TM Turkmenistan
TN Tunisia
TO Tonga
TR Türkiye
TT Trinidad and Tobago
TV Tuvalu
TW Taiwan, Province of China
TZ Tanzania, United Republic of
UA Ukraine
UG Uganda
UM United States Minor Outlying Islands
US United States
UY Uruguay
UZ Uzbekistan
VA Holy See (Vatican City State)
VC Saint Vincent and the Grenadines
VE Venezuela, Bolivarian Republic of
VG Virgin Islands, British
VI Virgin Islands, U.S.
VN Viet Nam
VU Vanuatu
WF Wallis and Futuna
WS Samoa
XK Kosovo
YE Yemen
YT Mayotte
ZA South Africa
ZM Zambia
ZW Zimbabwe
1 AD Andorra
2 AE United Arab Emirates
3 AF Afghanistan
4 AG Antigua and Barbuda
5 AI Anguilla
6 AL Albania
7 AM Armenia
8 AO Angola
9 AQ Antarctica
10 AR Argentina
11 AS American Samoa
12 AT Austria
13 AU Australia
14 AW Aruba
15 AX Åland Islands
16 AZ Azerbaijan
17 BA Bosnia and Herzegovina
18 BB Barbados
19 BD Bangladesh
20 BE Belgium
21 BF Burkina Faso
22 BG Bulgaria
23 BH Bahrain
24 BI Burundi
25 BJ Benin
26 BL Saint Barthélemy
27 BM Bermuda
28 BN Brunei Darussalam
29 BO Bolivia, Plurinational State of
30 BQ Bonaire, Sint Eustatius and Saba
31 BR Brazil
32 BS Bahamas
33 BT Bhutan
34 BV Bouvet Island
35 BW Botswana
36 BY Belarus
37 BZ Belize
38 CA Canada
39 CC Cocos (Keeling) Islands
40 CD Congo, The Democratic Republic of the
41 CF Central African Republic
42 CG Congo
43 CH Switzerland
44 CI Côte d'Ivoire
45 CK Cook Islands
46 CL Chile
47 CM Cameroon
48 CN China
49 CO Colombia
50 CR Costa Rica
51 CU Cuba
52 CV Cabo Verde
53 CW Curaçao
54 CX Christmas Island
55 CY Cyprus
56 CZ Czechia
57 DE Germany
58 DJ Djibouti
59 DK Denmark
60 DM Dominica
61 DO Dominican Republic
62 DZ Algeria
63 EC Ecuador
64 EE Estonia
65 EG Egypt
66 EH Western Sahara
67 ER Eritrea
68 ES Spain
69 ET Ethiopia
70 FI Finland
71 FJ Fiji
72 FK Falkland Islands (Malvinas)
73 FM Micronesia, Federated States of
74 FO Faroe Islands
75 FR France
76 GA Gabon
77 GB United Kingdom
78 GD Grenada
79 GE Georgia
80 GF French Guiana
81 GG Guernsey
82 GH Ghana
83 GI Gibraltar
84 GL Greenland
85 GM Gambia
86 GN Guinea
87 GP Guadeloupe
88 GQ Equatorial Guinea
89 GR Greece
90 GS South Georgia and the South Sandwich Islands
91 GT Guatemala
92 GU Guam
93 GW Guinea-Bissau
94 GY Guyana
95 HK Hong Kong
96 HM Heard Island and McDonald Islands
97 HN Honduras
98 HR Croatia
99 HT Haiti
100 HU Hungary
101 ID Indonesia
102 IE Ireland
103 IL Israel
104 IM Isle of Man
105 IN India
106 IO British Indian Ocean Territory
107 IQ Iraq
108 IR Iran, Islamic Republic of
109 IS Iceland
110 IT Italy
111 JE Jersey
112 JM Jamaica
113 JO Jordan
114 JP Japan
115 KE Kenya
116 KG Kyrgyzstan
117 KH Cambodia
118 KI Kiribati
119 KM Comoros
120 KN Saint Kitts and Nevis
121 KP Korea, Democratic People's Republic of
122 KR Korea, Republic of
123 KW Kuwait
124 KY Cayman Islands
125 KZ Kazakhstan
126 LA Lao People's Democratic Republic
127 LB Lebanon
128 LC Saint Lucia
129 LI Liechtenstein
130 LK Sri Lanka
131 LR Liberia
132 LS Lesotho
133 LT Lithuania
134 LU Luxembourg
135 LV Latvia
136 LY Libya
137 MA Morocco
138 MC Monaco
139 MD Moldova, Republic of
140 ME Montenegro
141 MF Saint Martin (French part)
142 MG Madagascar
143 MH Marshall Islands
144 MK North Macedonia
145 ML Mali
146 MM Myanmar
147 MN Mongolia
148 MO Macao
149 MP Northern Mariana Islands
150 MQ Martinique
151 MR Mauritania
152 MS Montserrat
153 MT Malta
154 MU Mauritius
155 MV Maldives
156 MW Malawi
157 MX Mexico
158 MY Malaysia
159 MZ Mozambique
160 NA Namibia
161 NC New Caledonia
162 NE Niger
163 NF Norfolk Island
164 NG Nigeria
165 NI Nicaragua
166 NL Netherlands
167 NO Norway
168 NP Nepal
169 NR Nauru
170 NU Niue
171 NZ New Zealand
172 OM Oman
173 PA Panama
174 PE Peru
175 PF French Polynesia
176 PG Papua New Guinea
177 PH Philippines
178 PK Pakistan
179 PL Poland
180 PM Saint Pierre and Miquelon
181 PN Pitcairn
182 PR Puerto Rico
183 PS Palestine, State of
184 PT Portugal
185 PW Palau
186 PY Paraguay
187 QA Qatar
188 RE Réunion
189 RO Romania
190 RS Serbia
191 RU Russian Federation
192 RW Rwanda
193 SA Saudi Arabia
194 SB Solomon Islands
195 SC Seychelles
196 SD Sudan
197 SE Sweden
198 SG Singapore
199 SH Saint Helena, Ascension and Tristan da Cunha
200 SI Slovenia
201 SJ Svalbard and Jan Mayen
202 SK Slovakia
203 SL Sierra Leone
204 SM San Marino
205 SN Senegal
206 SO Somalia
207 SR Suriname
208 SS South Sudan
209 ST Sao Tome and Principe
210 SV El Salvador
211 SX Sint Maarten (Dutch part)
212 SY Syrian Arab Republic
213 SZ Eswatini
214 TC Turks and Caicos Islands
215 TD Chad
216 TF French Southern Territories
217 TG Togo
218 TH Thailand
219 TJ Tajikistan
220 TK Tokelau
221 TL Timor-Leste
222 TM Turkmenistan
223 TN Tunisia
224 TO Tonga
225 TR Türkiye
226 TT Trinidad and Tobago
227 TV Tuvalu
228 TW Taiwan, Province of China
229 TZ Tanzania, United Republic of
230 UA Ukraine
231 UG Uganda
232 UM United States Minor Outlying Islands
233 US United States
234 UY Uruguay
235 UZ Uzbekistan
236 VA Holy See (Vatican City State)
237 VC Saint Vincent and the Grenadines
238 VE Venezuela, Bolivarian Republic of
239 VG Virgin Islands, British
240 VI Virgin Islands, U.S.
241 VN Viet Nam
242 VU Vanuatu
243 WF Wallis and Futuna
244 WS Samoa
245 XK Kosovo
246 YE Yemen
247 YT Mayotte
248 ZA South Africa
249 ZM Zambia
250 ZW Zimbabwe
+108
View File
@@ -0,0 +1,108 @@
//! Effective access carried by mounted Drive nodes.
use proton_drive_rs::MemberRole;
/// Effective write access for a node in a mounted tree.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Access {
Owner,
Editor,
Viewer,
Unknown,
}
impl Access {
pub fn writable(self) -> bool {
matches!(self, Self::Owner | Self::Editor)
}
pub(crate) fn as_db_str(self) -> &'static str {
match self {
Self::Owner => "owner",
Self::Editor => "editor",
Self::Viewer => "viewer",
Self::Unknown => "unknown",
}
}
pub(crate) fn from_db_str(value: &str) -> Option<Self> {
match value {
"owner" => Some(Self::Owner),
"editor" => Some(Self::Editor),
"viewer" => Some(Self::Viewer),
"unknown" => Some(Self::Unknown),
_ => None,
}
}
}
/// Resolve a share root's effective access from its exact membership role.
///
/// Owned trees fail open. A share whose role is absent or unrecognised fails
/// closed. Descendants normally inherit directly at intern time; `Inherited`
/// exists for the explicit wire role and therefore returns `parent`.
pub fn access_for(role: Option<MemberRole>, parent: Access, under_share: bool) -> Access {
match role {
Some(MemberRole::Viewer) => Access::Viewer,
Some(MemberRole::Editor | MemberRole::Admin) => Access::Editor,
Some(MemberRole::Inherited) => parent,
None if under_share => Access::Viewer,
None => Access::Owner,
}
}
/// POSIX permission bits advertised for a mounted node.
pub fn perm_bits(is_dir: bool, access: Access) -> u16 {
match (is_dir, access.writable()) {
(true, true) => 0o755,
(false, true) => 0o644,
(true, false) => 0o555,
(false, false) => 0o444,
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn access_for_covers_the_fail_open_and_fail_closed_table() {
for parent in [
Access::Owner,
Access::Editor,
Access::Viewer,
Access::Unknown,
] {
assert_eq!(access_for(None, parent, false), Access::Owner);
assert_eq!(access_for(None, parent, true), Access::Viewer);
assert_eq!(
access_for(Some(MemberRole::Viewer), parent, true),
Access::Viewer
);
assert_eq!(
access_for(Some(MemberRole::Editor), parent, true),
Access::Editor
);
assert_eq!(
access_for(Some(MemberRole::Admin), parent, true),
Access::Editor
);
assert_eq!(
access_for(Some(MemberRole::Inherited), parent, true),
parent
);
}
}
#[test]
fn permission_bits_follow_effective_access() {
for writable in [Access::Owner, Access::Editor] {
assert_eq!(perm_bits(true, writable), 0o755);
assert_eq!(perm_bits(false, writable), 0o644);
}
for read_only in [Access::Viewer, Access::Unknown] {
assert_eq!(perm_bits(true, read_only), 0o555);
assert_eq!(perm_bits(false, read_only), 0o444);
}
}
}
+66 -4
View File
@@ -6,11 +6,13 @@
//! interactive step; the refresh token auto-renews via the HTTP client's 401 //! interactive step; the refresh token auto-renews via the HTTP client's 401
//! path, so no fresh 2FA is required until the refresh token itself expires. //! path, so no fresh 2FA is required until the refresh token itself expires.
use keyring::Entry; use keyring_core::Entry;
use proton_drive_rs::{KeySalt, ProtonDriveClient}; use proton_drive_rs::{KeySalt, ProtonDriveClient};
use proton_sdk::api::{HumanVerification, HumanVerificationCredential}; use proton_sdk::api::{HumanVerification, HumanVerificationCredential};
use proton_sdk::cache::EncryptedCacheRepository;
use proton_sdk::config::ProtonClientConfiguration; use proton_sdk::config::ProtonClientConfiguration;
use proton_sdk::session::{PasswordMode, ProtonApiSession, ResumeParameters}; use proton_sdk::session::{PasswordMode, ProtonApiSession, ResumeParameters};
use proton_sdk::telemetry::TracingTelemetry;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use crate::config::{APP_VERSION, AppDirs, KEYRING_SERVICE, USER_AGENT}; use crate::config::{APP_VERSION, AppDirs, KEYRING_SERVICE, USER_AGENT};
@@ -79,6 +81,11 @@ fn client_config() -> ProtonClientConfiguration {
} }
fn keyring_entry() -> Result<Entry> { fn keyring_entry() -> Result<Entry> {
// Installed lazily rather than once at startup so a session bus that was not
// up yet on the first attempt is retried on the next one.
if keyring_core::get_default_store().is_none() {
keyring_core::set_default_store(dbus_secret_service_keyring_store::Store::new()?);
}
Ok(Entry::new(KEYRING_SERVICE, KEYRING_USER)?) Ok(Entry::new(KEYRING_SERVICE, KEYRING_USER)?)
} }
@@ -213,15 +220,30 @@ pub fn load() -> Result<StoredSession> {
let entry = keyring_entry()?; let entry = keyring_entry()?;
match entry.get_password() { match entry.get_password() {
Ok(json) => Ok(serde_json::from_str(&json)?), Ok(json) => Ok(serde_json::from_str(&json)?),
Err(keyring::Error::NoEntry) => Err(Error::NotLoggedIn), Err(keyring_core::Error::NoEntry) => Err(Error::NotLoggedIn),
Err(e) => Err(e.into()), Err(e) => Err(e.into()),
} }
} }
/// Forget the persisted session (best-effort; absent entry is not an error). /// Forget the persisted session (best-effort; absent entry is not an error).
pub fn logout() -> Result<()> { pub fn logout() -> Result<()> {
// The persisted entity cache describes the account being logged out of. It
// is encrypted under that account's mailbox password, but leaving it behind
// would also mean the next account inherits a store it can only ever read as
// misses. Best-effort: failing to remove a cache must not block the logout.
if let Ok(dirs) = AppDirs::new() {
let path = dirs.state_dir().join("sdk_cache.db");
if let Some(cache) = crate::sdkcache::SdkCache::opened()
&& let Err(e) = cache.clear_now()
{
tracing::warn!(error = %e, "clearing the SDK entity cache on logout failed");
}
for suffix in ["", "-wal", "-shm"] {
let _ = std::fs::remove_file(format!("{}{suffix}", path.display()));
}
}
match keyring_entry()?.delete_credential() { match keyring_entry()?.delete_credential() {
Ok(()) | Err(keyring::Error::NoEntry) => Ok(()), Ok(()) | Err(keyring_core::Error::NoEntry) => Ok(()),
Err(e) => Err(e.into()), Err(e) => Err(e.into()),
} }
} }
@@ -259,10 +281,50 @@ pub async fn resume_client() -> Result<(ProtonDriveClient, ProtonApiSession)> {
register_refresh_handler(&session, password.clone(), key_salts.clone()); register_refresh_handler(&session, password.clone(), key_salts.clone());
let client = ProtonDriveClient::with_key_salts(&session, password.into_bytes(), key_salts); let client = tune(
ProtonDriveClient::with_key_salts(&session, password.clone().into_bytes(), key_salts),
password.as_bytes(),
);
Ok((client, session)) Ok((client, session))
} }
/// The client settings every front-end wants, applied wherever a
/// [`ProtonDriveClient`] is built for real work.
///
/// - **Telemetry** into `tracing`, so the SDK's own spans (transfers, block
/// storage, per-request timings) land in the daemon's log next to the
/// daemon's, instead of being dropped by the default no-op sink.
/// - **Small-file uploads**: a file that fits in one block is uploaded as a
/// single atomic request instead of the multi-step draft/block/commit dance.
/// The SDK makes this opt-in because it has no remote feature-flag provider;
/// the drain queue writes plenty of small files, and the shorter path is one
/// failure point per write instead of three.
/// - **A persistent entity cache** ([`crate::sdkcache`]), so a restart does not
/// re-fetch and re-decrypt the tree the previous run already walked. It is
/// wrapped in the SDK's [`EncryptedCacheRepository`] keyed by the mailbox
/// password, so the decrypted node names it holds are not readable from the
/// file alone — and a password change simply reads as a cold cache, since the
/// SDK treats an undecryptable entry as a miss and clears the store.
///
/// Opening it is best-effort: a store this process cannot open costs cache
/// hits, not the session, so the client falls back to the SDK's in-memory
/// default.
fn tune(client: ProtonDriveClient, mailbox_password: &[u8]) -> ProtonDriveClient {
let client = client
.with_telemetry(TracingTelemetry::shared())
.with_small_file_upload(true);
match AppDirs::new().and_then(|dirs| crate::sdkcache::SdkCache::shared(&dirs)) {
Ok(cache) => client.with_entity_repository(EncryptedCacheRepository::shared(
cache,
mailbox_password.to_vec(),
)),
Err(e) => {
tracing::warn!(error = %e, "persistent SDK entity cache unavailable; using memory");
client
}
}
}
/// Write the session's current tokens back to the keyring. /// Write the session's current tokens back to the keyring.
/// ///
/// The session does not carry the mailbox password (it is only needed to rebuild /// The session does not carry the mailbox password (it is only needed to rebuild
+84
View File
@@ -0,0 +1,84 @@
//! Per-node outcomes of the SDK's batch node operations.
//!
//! `trash_nodes`, `restore_nodes`, `delete_nodes` and `move_nodes` report one
//! outcome per node instead of failing the whole call: a node the server
//! rejects leaves the rest of the batch applied. The daemon's callers fall into
//! two shapes, and both are here so neither has to re-derive the fold:
//!
//! - one node in, one answer out — [`into_unit`], which collapses the single
//! outcome back to a `Result`;
//! - a user-supplied list — [`split`], which keeps the successes and hands back
//! the failures to report.
use proton_sdk::error::ProtonError;
use proton_sdk::ids::NodeUid;
/// One node's outcome, as the SDK batch operations report it.
pub type Outcome = (NodeUid, Result<(), ProtonError>);
/// Collapse a batch result to the first per-node failure, for the callers that
/// pass exactly one node and want the old all-or-nothing answer.
///
/// An empty outcome list is `Ok`: the server accepted the request and reported
/// on nothing, which is not an error the caller can act on.
pub fn into_unit(outcomes: Vec<Outcome>) -> Result<(), ProtonError> {
for (_, outcome) in outcomes {
outcome?;
}
Ok(())
}
/// Split a batch result into the nodes that succeeded and the ones that failed.
pub fn split(outcomes: Vec<Outcome>) -> (Vec<NodeUid>, Vec<(NodeUid, ProtonError)>) {
let mut done = Vec::with_capacity(outcomes.len());
let mut failed = Vec::new();
for (uid, outcome) in outcomes {
match outcome {
Ok(()) => done.push(uid),
Err(e) => failed.push((uid, e)),
}
}
(done, failed)
}
#[cfg(test)]
mod tests {
use super::*;
use proton_sdk::ids::{LinkId, VolumeId};
fn uid(link: &str) -> NodeUid {
NodeUid::new(VolumeId::new("vol"), LinkId::new(link))
}
fn failure() -> ProtonError {
ProtonError::invalid_operation("nope")
}
#[test]
fn into_unit_is_ok_when_every_node_succeeded() {
assert!(into_unit(vec![(uid("a"), Ok(())), (uid("b"), Ok(()))]).is_ok());
assert!(
into_unit(Vec::new()).is_ok(),
"nothing reported is not a failure"
);
}
#[test]
fn into_unit_surfaces_a_failed_node() {
let outcomes = vec![(uid("a"), Ok(())), (uid("b"), Err(failure()))];
assert!(into_unit(outcomes).is_err());
}
#[test]
fn split_keeps_successes_and_failures_apart() {
let outcomes = vec![
(uid("a"), Ok(())),
(uid("b"), Err(failure())),
(uid("c"), Ok(())),
];
let (done, failed) = split(outcomes);
assert_eq!(done, vec![uid("a"), uid("c")]);
assert_eq!(failed.len(), 1);
assert_eq!(failed[0].0, uid("b"));
}
}
File diff suppressed because it is too large. Load diff
+296
View File
@@ -6,6 +6,8 @@ use directories::ProjectDirs;
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use crate::error::{Error, Result}; use crate::error::{Error, Result};
use crate::menu::PromptConfig;
use crate::opener::OpenWith;
use crate::syncignore::DEFAULT_IGNORE_PATTERNS; use crate::syncignore::DEFAULT_IGNORE_PATTERNS;
/// Proton requires `external-drive-{name}@{semver}-{channel}` (channel ∈ /// Proton requires `external-drive-{name}@{semver}-{channel}` (channel ∈
@@ -20,6 +22,59 @@ pub const KEYRING_SERVICE: &str = "proton-drive-linux";
/// blobs back under this; pinned files are exempt. See [`crate::cache`]. /// blobs back under this; pinned files are exempt. See [`crate::cache`].
pub const DEFAULT_CACHE_BUDGET_BYTES: u64 = 5 * 1024 * 1024 * 1024; pub const DEFAULT_CACHE_BUDGET_BYTES: u64 = 5 * 1024 * 1024 * 1024;
/// Environment override for [`AppConfig::resolved_conflict_sweep`]. Exists so a
/// support case can disable the sweep without editing the config file or
/// reinstalling — it wins over the stored setting.
pub const CONFLICT_SWEEP_ENV: &str = "PDFS_CONFLICT_SWEEP";
/// What the background conflict sweep (`pdfs-fuse`'s `pdfs-conflict-sweep`
/// thread) is allowed to do about a `(sync-conflict …)` copy it believes is a
/// redundant duplicate.
///
/// Defaults to [`SweepMode::Report`]: the sweep's whole reason to exist is that
/// the *previous* revision-identity check was wrong (B69), so the replacement is
/// not trusted with deletion until it has a release cycle of field evidence
/// behind it. See `docs/BUGS.md` B71.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq, Default)]
#[serde(rename_all = "lowercase")]
pub enum SweepMode {
/// Do not run the sweep at all — the thread is never spawned.
Off,
/// Run, and log what it *would* remove, but never trash anything. Divergent
/// and orphaned copies are still surfaced to the activity feed as usual.
#[default]
Report,
/// Trash copies proven identical to their live sibling.
Enforce,
}
impl SweepMode {
/// Parse an operator-supplied value ([`CONFLICT_SWEEP_ENV`] or a CLI flag).
/// `None` for anything unrecognised, so a typo falls back to the stored
/// setting rather than silently picking a mode the operator did not ask for.
pub fn parse(value: &str) -> Option<Self> {
match value.trim().to_ascii_lowercase().as_str() {
"off" | "no" | "false" | "0" | "disabled" => Some(Self::Off),
"report" | "dry-run" | "dryrun" | "observe" => Some(Self::Report),
"enforce" | "on" | "yes" | "true" | "1" => Some(Self::Enforce),
_ => None,
}
}
/// Whether this mode is allowed to trash a duplicate.
pub fn is_enforcing(self) -> bool {
matches!(self, Self::Enforce)
}
}
/// Precedence for [`AppConfig::resolved_conflict_sweep`], split out from the
/// environment read so it can be unit-tested without mutating process state.
fn resolve_sweep_mode(env: Option<&str>, stored: Option<SweepMode>) -> SweepMode {
env.and_then(SweepMode::parse)
.or(stored)
.unwrap_or_default()
}
/// Configuration structure allowing the user to customize client identification headers. /// Configuration structure allowing the user to customize client identification headers.
#[derive(Serialize, Deserialize, Debug, Clone)] #[derive(Serialize, Deserialize, Debug, Clone)]
pub struct AppConfig { pub struct AppConfig {
@@ -30,6 +85,15 @@ pub struct AppConfig {
/// writes an explicit value here. Defaulted for configs predating the field. /// writes an explicit value here. Defaulted for configs predating the field.
#[serde(default)] #[serde(default)]
pub cache_budget: Option<u64>, pub cache_budget: Option<u64>,
/// Upload cap in bytes per second shared by every upload. `None` or `0`
/// means no cap. Defaulted for configs predating the field.
#[serde(default)]
pub upload_limit: Option<u64>,
/// Download cap in bytes per second shared by every download, FUSE reads
/// included. `None` or `0` means no cap. Defaulted for configs predating
/// the field.
#[serde(default)]
pub download_limit: Option<u64>,
/// Mountpoint the daemon mounts at. `None` means /// Mountpoint the daemon mounts at. `None` means
/// [`AppDirs::default_mountpoint`]; the Settings page writes an explicit /// [`AppDirs::default_mountpoint`]; the Settings page writes an explicit
/// path here. Defaulted for configs predating the field. /// path here. Defaulted for configs predating the field.
@@ -48,6 +112,101 @@ pub struct AppConfig {
/// by hostname". Defaulted for configs predating the field. /// by hostname". Defaulted for configs predating the field.
#[serde(default)] #[serde(default)]
pub device_uid: Option<String>, pub device_uid: Option<String>,
/// What the background conflict sweep may do. `None` means "use
/// [`SweepMode::default`]" (report-only). Defaulted for configs predating
/// the field, and overridable at runtime via [`CONFLICT_SWEEP_ENV`].
#[serde(default)]
pub conflict_sweep: Option<SweepMode>,
/// How files opened from the prompt/browser are launched. `None` means
/// "hand everything to `xdg-open`", which is what the client did before
/// this existed. See [`crate::opener`]. Defaulted for configs predating the
/// field.
#[serde(default)]
pub open_with: Option<OpenWith>,
/// `pdfs-prompt` behaviour, including the external launcher used by
/// `--dmenu`. See [`crate::menu`]. Defaulted for configs predating the
/// field.
#[serde(default)]
pub prompt: Option<PromptConfig>,
/// Paint the desktop app in Proton's colours instead of the system theme
/// and accent colour. `None` means "follow the system". Defaulted for
/// configs predating the field; also read from `proton_accent`, its name
/// while it only changed the accent.
#[serde(default, alias = "proton_accent")]
pub proton_theme: Option<bool>,
/// How the desktop app's My Files page lays out and orders a folder.
/// Defaulted for configs predating the field.
#[serde(default)]
pub files_view: FilesView,
/// The user hid the tray icon from its own menu. The tray exits at start
/// while this is set, and the app does not spawn it; Preferences turns it
/// back on. Defaulted for configs predating the field.
#[serde(default)]
pub tray_hidden: bool,
/// Language for the desktop app, tray and search prompt, as a gettext code
/// such as `de` or `pt_BR`. `None` means "follow the system locale".
/// Defaulted for configs predating the field.
#[serde(default)]
pub language: Option<String>,
/// Draw the Places map from OpenFreeMap's street map instead of the
/// built-in outline. Off by default: the tiles come from a third party,
/// which then learns which parts of the world are looked at. Defaulted for
/// configs predating the field.
#[serde(default)]
pub online_map: bool,
}
/// The My Files page's view choices, remembered between runs.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
#[serde(default)]
pub struct FilesView {
/// Columns instead of the icon grid.
pub list: bool,
pub sort: FileSort,
pub descending: bool,
/// Folders ahead of files, whatever the sort.
pub folders_first: bool,
}
impl Default for FilesView {
fn default() -> Self {
Self {
list: false,
sort: FileSort::Name,
descending: false,
folders_first: true,
}
}
}
/// The key a My Files listing is ordered by.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq, Default)]
#[serde(rename_all = "lowercase")]
pub enum FileSort {
#[default]
Name,
Size,
Modified,
}
impl FileSort {
/// The name used in the desktop app's actions and the config file.
pub fn as_str(self) -> &'static str {
match self {
FileSort::Name => "name",
FileSort::Size => "size",
FileSort::Modified => "modified",
}
}
pub fn parse(value: &str) -> Option<Self> {
match value {
"name" => Some(FileSort::Name),
"size" => Some(FileSort::Size),
"modified" => Some(FileSort::Modified),
_ => None,
}
}
} }
impl Default for AppConfig { impl Default for AppConfig {
@@ -56,9 +215,19 @@ impl Default for AppConfig {
app_version: APP_VERSION.to_string(), app_version: APP_VERSION.to_string(),
user_agent: USER_AGENT.to_string(), user_agent: USER_AGENT.to_string(),
cache_budget: None, cache_budget: None,
upload_limit: None,
download_limit: None,
mountpoint: None, mountpoint: None,
ignore_patterns: None, ignore_patterns: None,
device_uid: None, device_uid: None,
conflict_sweep: None,
open_with: None,
prompt: None,
proton_theme: None,
files_view: FilesView::default(),
tray_hidden: false,
language: None,
online_map: false,
} }
} }
} }
@@ -70,6 +239,32 @@ impl AppConfig {
self.cache_budget.unwrap_or(DEFAULT_CACHE_BUDGET_BYTES) self.cache_budget.unwrap_or(DEFAULT_CACHE_BUDGET_BYTES)
} }
/// The effective opener policy: the user's rules, or the all-`xdg-open`
/// default. Cloned rather than borrowed because callers hold it for the
/// life of a window, long after the config value is gone.
pub fn resolved_open_with(&self) -> OpenWith {
self.open_with.clone().unwrap_or_default()
}
/// The effective prompt settings, defaulted when the section is absent.
pub fn resolved_prompt(&self) -> PromptConfig {
self.prompt.clone().unwrap_or_default()
}
/// The effective conflict-sweep mode: the [`CONFLICT_SWEEP_ENV`] override if
/// it parses, then the user's stored choice, then [`SweepMode::default`].
///
/// The environment wins so a support case can stop a destructive background
/// loop without editing config or reinstalling (`docs/BUGS.md` B71). An
/// unparseable value is ignored rather than treated as "off", so a typo
/// cannot silently change behaviour in either direction.
pub fn resolved_conflict_sweep(&self) -> SweepMode {
resolve_sweep_mode(
std::env::var(CONFLICT_SWEEP_ENV).ok().as_deref(),
self.conflict_sweep,
)
}
/// The effective global sync ignore patterns: the user's explicit list, or /// The effective global sync ignore patterns: the user's explicit list, or
/// [`DEFAULT_IGNORE_PATTERNS`] when unset. An explicit empty list is /// [`DEFAULT_IGNORE_PATTERNS`] when unset. An explicit empty list is
/// honoured as "no global patterns", which is why this is not a /// honoured as "no global patterns", which is why this is not a
@@ -371,14 +566,54 @@ mod tests {
app_version: "external-drive-test-client@1.0.0".to_string(), app_version: "external-drive-test-client@1.0.0".to_string(),
user_agent: "test-agent/1.0".to_string(), user_agent: "test-agent/1.0".to_string(),
cache_budget: Some(1234), cache_budget: Some(1234),
upload_limit: Some(500_000),
download_limit: None,
mountpoint: Some("/tmp/x".to_string()), mountpoint: Some("/tmp/x".to_string()),
ignore_patterns: Some(vec!["build/".to_string()]), ignore_patterns: Some(vec!["build/".to_string()]),
device_uid: Some("dev-uid".to_string()), device_uid: Some("dev-uid".to_string()),
conflict_sweep: Some(SweepMode::Enforce),
open_with: None,
prompt: None,
proton_theme: Some(true),
files_view: FilesView {
list: true,
sort: FileSort::Modified,
descending: true,
folders_first: false,
},
tray_hidden: true,
language: Some("de".to_string()),
online_map: true,
}; };
let json = serde_json::to_string(&config).unwrap(); let json = serde_json::to_string(&config).unwrap();
let decoded: AppConfig = serde_json::from_str(&json).unwrap(); let decoded: AppConfig = serde_json::from_str(&json).unwrap();
assert_eq!(decoded.app_version, "external-drive-test-client@1.0.0"); assert_eq!(decoded.app_version, "external-drive-test-client@1.0.0");
assert_eq!(decoded.user_agent, "test-agent/1.0"); assert_eq!(decoded.user_agent, "test-agent/1.0");
assert_eq!(decoded.proton_theme, Some(true));
assert!(decoded.tray_hidden);
assert!(decoded.online_map);
assert_eq!(decoded.language.as_deref(), Some("de"));
assert_eq!(decoded.upload_limit, Some(500_000));
assert_eq!(decoded.download_limit, None);
assert_eq!(decoded.files_view, config.files_view);
}
#[test]
fn a_config_without_a_files_view_gets_the_default() {
let decoded: AppConfig =
serde_json::from_str(r#"{"app_version":"a","user_agent":"b"}"#).unwrap();
assert_eq!(decoded.files_view, FilesView::default());
let partial: FilesView = serde_json::from_str(r#"{"sort":"size"}"#).unwrap();
assert_eq!(partial.sort, FileSort::Size);
assert!(partial.folders_first);
}
#[test]
fn the_old_proton_accent_setting_turns_on_the_proton_theme() {
let decoded: AppConfig =
serde_json::from_str(r#"{"app_version":"a","user_agent":"b","proton_accent":true}"#)
.unwrap();
assert_eq!(decoded.proton_theme, Some(true));
} }
#[test] #[test]
@@ -388,6 +623,62 @@ mod tests {
assert_eq!(default_config.user_agent, USER_AGENT); assert_eq!(default_config.user_agent, USER_AGENT);
} }
#[test]
fn sweep_mode_defaults_to_report_only() {
// The sweep trashes user files, so an installed config that has never
// heard of the setting must not get the enforcing behaviour.
assert_eq!(SweepMode::default(), SweepMode::Report);
assert!(!SweepMode::default().is_enforcing());
assert_eq!(resolve_sweep_mode(None, None), SweepMode::Report);
}
#[test]
fn sweep_mode_parses_operator_spellings() {
for s in ["off", "OFF", " no ", "false", "0", "disabled"] {
assert_eq!(SweepMode::parse(s), Some(SweepMode::Off), "{s}");
}
for s in ["report", "dry-run", "dryrun", "observe"] {
assert_eq!(SweepMode::parse(s), Some(SweepMode::Report), "{s}");
}
for s in ["enforce", "on", "yes", "true", "1"] {
assert_eq!(SweepMode::parse(s), Some(SweepMode::Enforce), "{s}");
}
// A typo must not resolve to anything, so it falls through to the config.
assert_eq!(SweepMode::parse("offf"), None);
assert_eq!(SweepMode::parse(""), None);
}
#[test]
fn sweep_mode_env_overrides_config_but_a_typo_does_not() {
let stored = Some(SweepMode::Enforce);
// The point of the override: kill a destructive loop without editing config.
assert_eq!(resolve_sweep_mode(Some("off"), stored), SweepMode::Off);
// Unrecognised value falls back to the stored choice rather than guessing.
assert_eq!(resolve_sweep_mode(Some("nope"), stored), SweepMode::Enforce);
assert_eq!(resolve_sweep_mode(None, stored), SweepMode::Enforce);
// No env, no stored value: report-only.
assert_eq!(resolve_sweep_mode(Some("nope"), None), SweepMode::Report);
}
#[test]
fn a_config_predating_conflict_sweep_still_loads() {
let json = r#"{"app_version":"v","user_agent":"u"}"#;
let decoded: AppConfig = serde_json::from_str(json).unwrap();
assert!(decoded.conflict_sweep.is_none());
}
#[test]
fn sweep_mode_round_trips_through_the_config_file() {
let config = AppConfig {
conflict_sweep: Some(SweepMode::Enforce),
..Default::default()
};
let json = serde_json::to_string(&config).unwrap();
assert!(json.contains(r#""conflict_sweep":"enforce""#), "{json}");
let decoded: AppConfig = serde_json::from_str(&json).unwrap();
assert_eq!(decoded.conflict_sweep, Some(SweepMode::Enforce));
}
#[test] #[test]
fn a_config_predating_ignore_patterns_still_loads() { fn a_config_predating_ignore_patterns_still_loads() {
// Every installed config on disk looks like this; the field must not be // Every installed config on disk looks like this; the field must not be
@@ -395,6 +686,11 @@ mod tests {
let json = r#"{"app_version":"v","user_agent":"u"}"#; let json = r#"{"app_version":"v","user_agent":"u"}"#;
let decoded: AppConfig = serde_json::from_str(json).unwrap(); let decoded: AppConfig = serde_json::from_str(json).unwrap();
assert!(decoded.ignore_patterns.is_none()); assert!(decoded.ignore_patterns.is_none());
assert!(!decoded.tray_hidden, "an old config keeps the tray icon");
assert!(
decoded.language.is_none(),
"an old config follows the system"
);
assert_eq!( assert_eq!(
decoded.resolved_ignore_patterns().len(), decoded.resolved_ignore_patterns().len(),
DEFAULT_IGNORE_PATTERNS.len(), DEFAULT_IGNORE_PATTERNS.len(),
File diff suppressed because it is too large. Load diff
+12 -1
View File
@@ -29,11 +29,22 @@ impl Db {
Ok(()) Ok(())
} }
/// Whether the kept log already holds an entry of `kind` for `target`. Lets
/// the daemon report something once across restarts rather than once per
/// run, without keeping a second table in step with the log.
pub fn activity_has(&self, kind: ActivityKind, target: &str) -> Result<bool> {
let kind = serde_json::to_string(&kind)?;
let conn = self.read();
let mut stmt =
conn.prepare("SELECT 1 FROM activity WHERE kind = ?1 AND target = ?2 LIMIT 1")?;
Ok(stmt.exists(params![kind, target])?)
}
/// The most recent activity, newest first, capped at `limit` entries. Rows /// The most recent activity, newest first, capped at `limit` entries. Rows
/// whose stored `kind` no longer parses (written by an older build) are /// whose stored `kind` no longer parses (written by an older build) are
/// skipped rather than failing the whole read. /// skipped rather than failing the whole read.
pub fn activity_list(&self, limit: usize) -> Result<Vec<ActivityEntry>> { pub fn activity_list(&self, limit: usize) -> Result<Vec<ActivityEntry>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT time, kind, target, detail, ok FROM activity "SELECT time, kind, target, detail, ok FROM activity
ORDER BY id DESC LIMIT ?1", ORDER BY id DESC LIMIT ?1",
+267
View File
@@ -0,0 +1,267 @@
//! Photo albums: the album listing and each album's contents, persisted like
//! the timeline so the Albums view opens from disk instead of re-enumerating the
//! photos volume on every launch.
//!
//! An album's photos are stored per album rather than as a join against
//! `photos`: an album shared with us lives on the sharer's volume, so its photos
//! never appear in our own timeline and have nowhere else to record what a
//! thumbnail attempt learned about them.
use rusqlite::params;
use std::collections::HashMap;
use super::{Db, StoredPhoto, THUMB_UNKNOWN};
use crate::Result;
/// One album of the persisted listing, ordered newest-activity-first in `seq`.
#[derive(Clone, Debug, PartialEq)]
pub struct StoredAlbum {
pub uid: String,
pub name: String,
pub photo_count: usize,
/// The photo shown as the album's cover, when the server named one.
pub cover_uid: Option<String>,
/// Epoch seconds of the last change to the album's contents.
pub last_activity: Option<i64>,
/// True when the album lives on someone else's photos volume.
pub shared: bool,
}
impl Db {
/// Replace the album listing wholesale, in the given order.
pub fn albums_replace(&self, albums: &[StoredAlbum]) -> Result<()> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
// Contents of an album that no longer exists (unshared, deleted) go with
// it — otherwise they would linger forever, keyed by an album uid no
// listing mentions again.
{
let kept: std::collections::HashSet<&str> =
albums.iter().map(|a| a.uid.as_str()).collect();
let stale: Vec<String> = {
let mut stmt = tx.prepare("SELECT DISTINCT album_uid FROM album_photos")?;
let rows = stmt.query_map([], |r| r.get::<_, String>(0))?;
rows.filter_map(|r| r.ok())
.filter(|uid| !kept.contains(uid.as_str()))
.collect()
};
let mut stmt = tx.prepare("DELETE FROM album_photos WHERE album_uid = ?1")?;
for uid in stale {
stmt.execute([uid])?;
}
}
tx.execute("DELETE FROM albums", [])?;
{
let mut stmt = tx.prepare(
"INSERT INTO albums
(uid, name, photo_count, cover_uid, last_activity, shared, seq)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)",
)?;
for (seq, album) in albums.iter().enumerate() {
stmt.execute(params![
album.uid,
album.name,
album.photo_count as i64,
album.cover_uid,
album.last_activity,
album.shared as i64,
seq as i64,
])?;
}
}
tx.commit()?;
Ok(())
}
/// The persisted album listing, in stored order.
pub fn albums_list(&self) -> Result<Vec<StoredAlbum>> {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT uid, name, photo_count, cover_uid, last_activity, shared
FROM albums ORDER BY seq",
)?;
let rows = stmt
.query_map([], |r| {
Ok(StoredAlbum {
uid: r.get(0)?,
name: r.get(1)?,
photo_count: r.get::<_, i64>(2)?.max(0) as usize,
cover_uid: r.get(3)?,
last_activity: r.get(4)?,
shared: r.get::<_, i64>(5)? != 0,
})
})?
.collect::<rusqlite::Result<_>>()?;
Ok(rows)
}
/// Number of albums in the persisted listing.
pub fn albums_count(&self) -> Result<usize> {
let conn = self.read();
let n: i64 = conn.query_row("SELECT COUNT(*) FROM albums", [], |r| r.get(0))?;
Ok(n.max(0) as usize)
}
/// Replace one album's contents, in the given (newest-capture-first) order.
/// Learned aspect ratios and thumbnail verdicts survive the replacement, the
/// same way [`Db::photos_replace`] keeps them for the timeline.
pub fn album_photos_replace(
&self,
album_uid: &str,
items: &[(String, i64, Option<String>, Option<String>)],
) -> Result<()> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let learned: HashMap<String, (Option<f64>, i64, Option<String>)> = {
let mut stmt = tx.prepare(
"SELECT uid, ratio, thumb_state, media_type FROM album_photos
WHERE album_uid = ?1",
)?;
let rows = stmt.query_map([album_uid], |r| {
Ok((r.get::<_, String>(0)?, (r.get(1)?, r.get(2)?, r.get(3)?)))
})?;
rows.collect::<rusqlite::Result<_>>()?
};
tx.execute("DELETE FROM album_photos WHERE album_uid = ?1", [album_uid])?;
{
let mut stmt = tx.prepare(
"INSERT INTO album_photos
(album_uid, uid, capture_time, name, media_type, kind, ratio, thumb_state, seq)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)",
)?;
for (seq, (uid, capture_time, name, media_type)) in items.iter().enumerate() {
let (ratio, thumb_state, learned_media) =
learned
.get(uid)
.cloned()
.unwrap_or((None, THUMB_UNKNOWN, None));
let media_type = media_type.clone().or(learned_media);
let kind =
crate::control::PhotoKind::classify(name.as_deref(), media_type.as_deref());
stmt.execute(params![
album_uid,
uid,
capture_time,
name,
media_type,
kind.as_i64(),
ratio,
thumb_state,
seq as i64,
])?;
}
}
tx.commit()?;
Ok(())
}
/// One page of an album's photos, in stored order.
pub fn album_photos_page(
&self,
album_uid: &str,
offset: usize,
limit: usize,
) -> Result<Vec<StoredPhoto>> {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT uid, capture_time, name, ratio, thumb_state, kind FROM album_photos
WHERE album_uid = ?1 ORDER BY seq LIMIT ?2 OFFSET ?3",
)?;
let rows = stmt
.query_map(params![album_uid, limit as i64, offset as i64], |r| {
Ok(StoredPhoto {
uid: r.get(0)?,
capture_time: r.get(1)?,
name: r.get(2)?,
ratio: r.get(3)?,
thumb_state: r.get(4)?,
kind: crate::control::PhotoKind::from_i64(r.get(5)?),
// Album rows have no favorite state of their own: an album
// photo on someone else's volume is not in our timeline, and
// favouriting one is not supported yet anyway.
favorite: false,
// Nor are album rows grouped: an album is a list the user
// made, and it shows exactly what they put in it.
group_size: 1,
has_raw: false,
})
})?
.collect::<rusqlite::Result<_>>()?;
Ok(rows)
}
/// Which albums hold `uid`, by album node uid. Read by the photo re-date,
/// which has to put the re-uploaded copy back into the albums the original
/// was in.
pub fn albums_of_photo(&self, uid: &str) -> Result<Vec<String>> {
let conn = self.read();
let mut stmt =
conn.prepare("SELECT DISTINCT album_uid FROM album_photos WHERE uid = ?1")?;
let rows = stmt
.query_map([uid], |r| r.get(0))?
.collect::<rusqlite::Result<_>>()?;
Ok(rows)
}
/// How many photos of `album_uid` are persisted.
pub fn album_photos_count(&self, album_uid: &str) -> Result<usize> {
let conn = self.read();
let n: i64 = conn.query_row(
"SELECT COUNT(*) FROM album_photos WHERE album_uid = ?1",
[album_uid],
|r| r.get(0),
)?;
Ok(n.max(0) as usize)
}
/// The album-held photos among `uids`, one row per uid (a photo in two albums
/// is returned once). Backs the thumbnail path for photos that are in an
/// album but not in our own timeline — everything in a shared album, and an
/// album cover whose photo the current timeline page hasn't reached.
pub fn album_photos_by_uid(&self, uids: &[String]) -> Result<Vec<StoredPhoto>> {
if uids.is_empty() {
return Ok(Vec::new());
}
let conn = self.read();
let placeholders = vec!["?"; uids.len()].join(",");
let mut stmt = conn.prepare(&format!(
"SELECT uid, capture_time, name, ratio, thumb_state, kind FROM album_photos
WHERE uid IN ({placeholders}) GROUP BY uid"
))?;
let rows = stmt
.query_map(rusqlite::params_from_iter(uids), |r| {
Ok(StoredPhoto {
uid: r.get(0)?,
capture_time: r.get(1)?,
name: r.get(2)?,
ratio: r.get(3)?,
thumb_state: r.get(4)?,
kind: crate::control::PhotoKind::from_i64(r.get(5)?),
// Album rows have no favorite state of their own: an album
// photo on someone else's volume is not in our timeline, and
// favouriting one is not supported yet anyway.
favorite: false,
// Nor are album rows grouped: an album is a list the user
// made, and it shows exactly what they put in it.
group_size: 1,
has_raw: false,
})
})?
.collect::<rusqlite::Result<_>>()?;
Ok(rows)
}
/// Record a thumbnail verdict against every album row for `uid`, so what one
/// attempt learned is not re-learned per album the photo sits in. The
/// timeline's own copy is updated separately by [`Db::photo_set_thumb`].
pub fn album_photo_set_thumb(&self, uid: &str, state: i64, ratio: Option<f64>) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"UPDATE album_photos SET thumb_state = ?2, ratio = COALESCE(?3, ratio) WHERE uid = ?1",
params![uid, state, ratio],
)?;
Ok(())
}
}
+27 -3
View File
@@ -42,6 +42,30 @@ impl Db {
Ok(()) Ok(())
} }
/// Bump `last_accessed` for many keys in one transaction.
///
/// The read path touches a key per 128 KiB the kernel asks for — 32 times
/// per 4 MiB block — and each of those was its own write transaction under
/// the single connection mutex, contending with every `lookup` and
/// `readdir`. Callers accumulate touches in memory and flush them here.
pub fn cache_accessed_batch(&self, touches: &[(String, i64)]) -> Result<()> {
if touches.is_empty() {
return Ok(());
}
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
{
let mut stmt = tx.prepare_cached(
"UPDATE cache_entries SET last_accessed = ?2 WHERE cache_key = ?1",
)?;
for (key, at) in touches {
stmt.execute(params![key, at])?;
}
}
tx.commit()?;
Ok(())
}
/// Drop a single cache-entry row (one evicted blob or block). /// Drop a single cache-entry row (one evicted blob or block).
pub fn cache_remove(&self, key: &str) -> Result<()> { pub fn cache_remove(&self, key: &str) -> Result<()> {
let conn = self.conn.lock(); let conn = self.conn.lock();
@@ -75,7 +99,7 @@ impl Db {
/// because `enforce_block_budget` runs on every cached block — once per /// because `enforce_block_budget` runs on every cached block — once per
/// 4 MiB of every cold read — while holding the shared connection. /// 4 MiB of every cold read — while holding the shared connection.
pub fn cache_total_bytes(&self, kind: &str) -> Result<u64> { pub fn cache_total_bytes(&self, kind: &str) -> Result<u64> {
let conn = self.conn.lock(); let conn = self.read();
let total: i64 = conn.query_row( let total: i64 = conn.query_row(
"SELECT COALESCE(SUM(size_bytes), 0) FROM cache_entries WHERE kind = ?1", "SELECT COALESCE(SUM(size_bytes), 0) FROM cache_entries WHERE kind = ?1",
params![kind], params![kind],
@@ -95,7 +119,7 @@ impl Db {
kind: &str, kind: &str,
limit: usize, limit: usize,
) -> Result<Vec<(String, u64)>> { ) -> Result<Vec<(String, u64)>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare_cached( let mut stmt = conn.prepare_cached(
"SELECT cache_key, size_bytes FROM cache_entries "SELECT cache_key, size_bytes FROM cache_entries
WHERE kind = ?1 ORDER BY last_accessed ASC LIMIT ?2", WHERE kind = ?1 ORDER BY last_accessed ASC LIMIT ?2",
@@ -115,7 +139,7 @@ impl Db {
/// (`clear_unpinned`); incremental eviction wants /// (`clear_unpinned`); incremental eviction wants
/// [`cache_eviction_candidates`](Self::cache_eviction_candidates). /// [`cache_eviction_candidates`](Self::cache_eviction_candidates).
pub fn cache_entries_by_kind(&self, kind: &str) -> Result<Vec<(String, u64)>> { pub fn cache_entries_by_kind(&self, kind: &str) -> Result<Vec<(String, u64)>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT cache_key, size_bytes FROM cache_entries "SELECT cache_key, size_bytes FROM cache_entries
WHERE kind = ?1 ORDER BY last_accessed ASC", WHERE kind = ?1 ORDER BY last_accessed ASC",
+1 -1
View File
@@ -15,7 +15,7 @@ pub struct StoredDevice {
impl Db { impl Db {
pub fn device_get(&self) -> Result<Option<StoredDevice>> { pub fn device_get(&self) -> Result<Option<StoredDevice>> {
let conn = self.conn.lock(); let conn = self.read();
conn.query_row( conn.query_row(
"SELECT uid, share_id, root_uid, name, created FROM device LIMIT 1", "SELECT uid, share_id, root_uid, name, created FROM device LIMIT 1",
[], [],
+76 -36
View File
@@ -41,32 +41,43 @@ impl Db {
Ok(next) Ok(next)
} }
/// Write one batch of walked entries under scan generation `generation`. The /// Write one batch of walked entries under scan generation `generation`,
/// FTS index is *not* touched here — it is rebuilt once in /// keeping the FTS index in step with them.
/// [`local_finish_scan`](Self::local_finish_scan). ///
/// The index is maintained incrementally rather than rebuilt at the end of
/// the scan, because a rebuild reads and re-tokenises every row in the table
/// — trigrams over a few hundred thousand paths — inside one transaction on
/// the single shared connection, which stalls the whole mount for as long as
/// it takes. Incremental maintenance is possible here for a reason specific
/// to this table: `local_fts` covers `name` and `path`, `path` is the
/// table's key, and `name` is derived from it, so an existing row's indexed
/// content can never change. Only insertions and deletions touch the index;
/// a re-scan that finds the same files does no index work at all.
///
/// Hence the update-then-insert shape below rather than an upsert: the
/// distinction between "already indexed" and "new row" is exactly what
/// decides whether the index needs a write.
pub fn local_upsert_batch(&self, generation: i64, entries: &[LocalEntry]) -> Result<()> { pub fn local_upsert_batch(&self, generation: i64, entries: &[LocalEntry]) -> Result<()> {
let mut conn = self.conn.lock(); let mut conn = self.conn.lock();
let tx = conn.transaction()?; let tx = conn.transaction()?;
{ {
let mut stmt = tx.prepare_cached( let mut update = tx.prepare_cached(
"INSERT INTO local_files (path, name, is_dir, size, mtime, scan_gen) "UPDATE local_files
VALUES (?1, ?2, ?3, ?4, ?5, ?6) SET name = ?2, is_dir = ?3, size = ?4, mtime = ?5, scan_gen = ?6
ON CONFLICT(path) DO UPDATE SET WHERE path = ?1",
name = excluded.name,
is_dir = excluded.is_dir,
size = excluded.size,
mtime = excluded.mtime,
scan_gen = excluded.scan_gen",
)?; )?;
let mut insert = tx.prepare_cached(
"INSERT INTO local_files (path, name, is_dir, size, mtime, scan_gen)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
)?;
let mut index =
tx.prepare_cached("INSERT INTO local_fts (rowid, name, path) VALUES (?1, ?2, ?3)")?;
for e in entries { for e in entries {
stmt.execute(params![ let row = params![e.path, e.name, e.is_dir as i64, e.size, e.mtime, generation];
e.path, if update.execute(row)? == 0 {
e.name, insert.execute(row)?;
e.is_dir as i64, index.execute(params![tx.last_insert_rowid(), e.name, e.path])?;
e.size, }
e.mtime,
generation
])?;
} }
} }
tx.commit()?; tx.commit()?;
@@ -74,16 +85,25 @@ impl Db {
} }
/// Close scan generation `generation`: drop every row an older scan wrote /// Close scan generation `generation`: drop every row an older scan wrote
/// (those paths are gone from disk), rebuild the FTS index over what remains, /// (those paths are gone from disk), drop their FTS entries, and stamp the
/// and stamp the completion time. Returns the number of indexed entries. /// completion time. Returns the number of indexed entries.
///
/// The FTS deletes name the old values explicitly, which is how an
/// external-content FTS5 table is told to retract a row — see
/// [`local_upsert_batch`](Self::local_upsert_batch) for why this is done
/// per-row instead of by rebuilding the index.
pub fn local_finish_scan(&self, generation: i64, finished_at: i64) -> Result<i64> { pub fn local_finish_scan(&self, generation: i64, finished_at: i64) -> Result<i64> {
let mut conn = self.conn.lock(); let mut conn = self.conn.lock();
let tx = conn.transaction()?; let tx = conn.transaction()?;
tx.execute(
"INSERT INTO local_fts (local_fts, rowid, name, path)
SELECT 'delete', id, name, path FROM local_files WHERE scan_gen != ?1",
params![generation],
)?;
tx.execute( tx.execute(
"DELETE FROM local_files WHERE scan_gen != ?1", "DELETE FROM local_files WHERE scan_gen != ?1",
params![generation], params![generation],
)?; )?;
tx.execute_batch("INSERT INTO local_fts(local_fts) VALUES('rebuild');")?;
tx.execute( tx.execute(
"INSERT INTO sync_state (key, value) VALUES ('local_indexed_at', ?1) "INSERT INTO sync_state (key, value) VALUES ('local_indexed_at', ?1)
ON CONFLICT(key) DO UPDATE SET value = excluded.value", ON CONFLICT(key) DO UPDATE SET value = excluded.value",
@@ -98,7 +118,7 @@ impl Db {
/// has never been built. The daemon uses this to decide whether a fresh mount /// has never been built. The daemon uses this to decide whether a fresh mount
/// needs an immediate rescan or can serve the existing index. /// needs an immediate rescan or can serve the existing index.
pub fn local_indexed_at(&self) -> Result<Option<i64>> { pub fn local_indexed_at(&self) -> Result<Option<i64>> {
let conn = self.conn.lock(); let conn = self.read();
Ok(conn Ok(conn
.query_row( .query_row(
"SELECT value FROM sync_state WHERE key = 'local_indexed_at'", "SELECT value FROM sync_state WHERE key = 'local_indexed_at'",
@@ -118,7 +138,7 @@ impl Db {
if query.is_empty() { if query.is_empty() {
return Ok(Vec::new()); return Ok(Vec::new());
} }
let conn = self.conn.lock(); let conn = self.read();
let mut stmt; let mut stmt;
let rows = if query.chars().count() < TRIGRAM_MIN { let rows = if query.chars().count() < TRIGRAM_MIN {
let pat = format!("%{}%", like_escape(query)); let pat = format!("%{}%", like_escape(query));
@@ -157,7 +177,7 @@ impl Db {
if query.is_empty() || limit == 0 { if query.is_empty() || limit == 0 {
return Ok(Vec::new()); return Ok(Vec::new());
} }
let conn = self.conn.lock(); let conn = self.read();
let terms = candidate_trigrams(query); let terms = candidate_trigrams(query);
let lane_limit = limit.div_ceil(2); let lane_limit = limit.div_ceil(2);
let mut rows: Vec<LocalFileHit> = if terms.is_empty() { let mut rows: Vec<LocalFileHit> = if terms.is_empty() {
@@ -182,16 +202,36 @@ impl Db {
if !terms.is_empty() if !terms.is_empty()
&& let Some(first) = query.chars().next() && let Some(first) = query.chars().next()
{ {
let prefix = format!("{}%", like_escape(&first.to_string())); // Trigram rank says nothing about how well a name *starts* with the
let mut stmt = conn.prepare( // query, so on a home directory full of dependency trees the pool
"SELECT path, name, is_dir, size, mtime FROM local_files // fills with vendored files before the obvious answer is reached
WHERE name COLLATE NOCASE LIKE ?1 ESCAPE '\\' // (measured: 73% of the 500 best-ranked candidates for "test" came
ORDER BY name COLLATE NOCASE LIMIT ?2", // from one Go module cache). Give whole-query prefix matches their
)?; // own lane, newest first, and keep the single-character lane —
let fallback = stmt // which recovers a typo that destroyed every trigram — behind it.
.query_map(params![prefix, lane_limit as i64], local_hit)? let mut lanes = vec![(
.collect::<std::result::Result<Vec<_>, _>>()?; format!("{}%", like_escape(query)),
for hit in fallback { "ORDER BY mtime DESC LIMIT ?2",
)];
lanes.push((
format!("{}%", like_escape(&first.to_string())),
"ORDER BY name COLLATE NOCASE LIMIT ?2",
));
let mut extra = Vec::new();
for (pattern, order) in lanes {
let mut stmt = conn.prepare(&format!(
"SELECT path, name, is_dir, size, mtime FROM local_files
WHERE name COLLATE NOCASE LIKE ?1 ESCAPE '\\' {order}"
))?;
extra.extend(
stmt.query_map(params![pattern, lane_limit as i64], local_hit)?
.collect::<std::result::Result<Vec<_>, _>>()?,
);
}
// Prefix hits go in front: `rows` is truncated to `limit` below, and
// the lane exists precisely because trigram rank was burying them.
let trigram = std::mem::take(&mut rows);
for hit in extra.into_iter().chain(trigram) {
if !rows.iter().any(|existing| existing.path == hit.path) { if !rows.iter().any(|existing| existing.path == hit.path) {
rows.push(hit); rows.push(hit);
} }
+2
View File
@@ -77,10 +77,12 @@ const REPORTED_TABLES: &[&str] = &[
"trash", "trash",
"device", "device",
"sync_folder", "sync_folder",
"mount",
"sync_entry", "sync_entry",
"pending_op", "pending_op",
"activity", "activity",
"local_files", "local_files",
"share_access",
]; ];
impl Db { impl Db {
+649 -1
View File
@@ -9,7 +9,7 @@ use super::Db;
use crate::Result; use crate::Result;
/// Current schema version. Bump on every forward migration added below. /// Current schema version. Bump on every forward migration added below.
pub(super) const SCHEMA_VERSION: i64 = 16; pub(super) const SCHEMA_VERSION: i64 = 35;
impl Db { impl Db {
pub(super) fn migrate(&self) -> Result<()> { pub(super) fn migrate(&self) -> Result<()> {
@@ -89,6 +89,249 @@ impl Db {
if current < 16 { if current < 16 {
tx.execute_batch(MIGRATION_V16)?; tx.execute_batch(MIGRATION_V16)?;
} }
if current < 17 {
tx.execute_batch(MIGRATION_V17)?;
}
if current < 18 {
tx.execute_batch(MIGRATION_V18)?;
}
if current < 19 {
tx.execute_batch(MIGRATION_V19)?;
}
if current < 20 {
tx.execute_batch(MIGRATION_V20)?;
}
if current < 21 {
// A V17-and-older fixture may carry only the tables its own test
// cares about; there is nothing to reindex without `nodes`.
let has_nodes: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'nodes'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_nodes {
tx.execute_batch(MIGRATION_V21)?;
}
}
if current < 22 {
// Older fixtures (and a database that never grew a table because
// its feature was never used) may be missing any of these, and an
// index on an absent table is a hard error rather than a no-op.
for (table, sql) in MIGRATION_V22 {
let present: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?1",
[table],
|row| row.get::<_, i64>(0),
)? > 0;
if present {
tx.execute_batch(sql)?;
}
}
// `activity_time` indexes `time DESC` while both activity queries
// order by `id`: it has only ever cost writes.
tx.execute_batch("DROP INDEX IF EXISTS activity_time;")?;
}
if current < 23 {
// One last full rebuild of the local index, because from here on the
// scan maintains it incrementally (`local_upsert_batch`) and so can
// no longer repair a `local_fts` that drifted from `local_files`.
// Whatever it costs, it costs once, at startup, before the mount
// serves anything.
let present: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'local_fts'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if present {
tx.execute_batch("INSERT INTO local_fts(local_fts) VALUES('rebuild');")?;
}
}
if current < 24 {
tx.execute_batch(MIGRATION_V24)?;
}
if current < 25 {
// Same guard as V21: an old fixture may not have `nodes` at all,
// and there is no path to materialise without it.
let has_nodes: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'nodes'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_nodes {
tx.execute_batch(MIGRATION_V25)?;
}
}
if current < 26 {
// Same guard as V22: a fixture that never exercised the queue may
// not have `pending_op`, and altering an absent table is an error
// rather than a no-op.
// A fixture built from the current schema and then rewound to an
// older `schema_version` already has the column, and `ADD COLUMN`
// has no `IF NOT EXISTS`.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('pending_op') WHERE name = 'claimed_at'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_ops: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_ops && !has_column {
tx.execute_batch(MIGRATION_V26)?;
}
}
if current < 27 {
// Same guards as V26, for the same two reasons: the table may not
// exist in an old fixture, and a fixture rewound from the current
// schema already has the column.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('pending_op') \
WHERE name = 'access_deferred_since'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_ops: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_ops && !has_column {
tx.execute_batch(MIGRATION_V27)?;
}
}
if current < 28 {
// Same guards as V26/V27, for the same two reasons.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('sync_entry') \
WHERE name = 'local_mtime_ns'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_entries: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'sync_entry'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_entries && !has_column {
tx.execute_batch(MIGRATION_V28)?;
}
}
if current < 29 {
// Same guards as V26/V27/V28: an old fixture may not have `trash`,
// and a fixture rewound from the current schema already has the
// column.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('trash') WHERE name = 'parent_uid'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_trash: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'trash'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_trash && !has_column {
tx.execute_batch(MIGRATION_V29)?;
}
}
if current < 30 {
// Same guards as V26-V29: the column may already be there on a
// fixture rewound from the current schema, and a very old fixture
// may not have `photos` at all.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('photos') WHERE name = 'group_key'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_photos: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'photos'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_photos && !has_column {
tx.execute_batch(MIGRATION_V30)?;
}
}
if current < 31 {
// Same guards as V26-V30.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('photos') WHERE name = 'resolved_at'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_photos: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'photos'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_photos && !has_column {
tx.execute_batch(MIGRATION_V31)?;
}
}
if current < 32 {
// Data only, so safe to repeat; each table is guarded for a very
// old fixture that never had it.
for table in ["photos", "album_photos"] {
let has_table: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?1",
[table],
|row| row.get::<_, i64>(0),
)? > 0;
if has_table {
tx.execute(&MIGRATION_V32.replace("{table}", table), [])?;
}
}
}
if current < 33 {
// Same guards as V26-V31.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('photos') WHERE name = 'place_id'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_photos: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'photos'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_photos && !has_column {
tx.execute_batch(MIGRATION_V33)?;
}
}
if current < 34 {
// Same guards as V26-V33.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('photos') WHERE name = 'similar_hash'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_photos: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'photos'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_photos && !has_column {
tx.execute_batch(MIGRATION_V34)?;
}
}
if current < 35 {
// Same guards as V26-V34.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('trash') WHERE name = 'trashed_at'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_trash: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'trash'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_trash && !has_column {
tx.execute_batch(MIGRATION_V35)?;
}
}
tx.execute( tx.execute(
"INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1) "INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1)
ON CONFLICT(key) DO UPDATE SET value = excluded.value", ON CONFLICT(key) DO UPDATE SET value = excluded.value",
@@ -411,3 +654,408 @@ CREATE INDEX IF NOT EXISTS idx_local_files_name_nocase
ON local_files(name COLLATE NOCASE); ON local_files(name COLLATE NOCASE);
INSERT INTO local_fts(local_fts) VALUES('rebuild'); INSERT INTO local_fts(local_fts) VALUES('rebuild');
"; ";
/// Schema v17: effective access for roots shared with this account. The role is
/// persisted separately from `nodes.node_json` so a cold/offline mount can
/// enforce the last known permission even when the SDK cannot refresh metadata.
const MIGRATION_V17: &str = "
CREATE TABLE share_access (
root_uid TEXT PRIMARY KEY,
access TEXT NOT NULL
CHECK (access IN ('owner', 'editor', 'viewer', 'unknown'))
);
";
/// Schema v18: unified location presentation. Device rows carry only their
/// `sync_folder` identity; reads join all device state from that authoritative
/// table. The trigger projects future folders, while the foreign key removes a
/// presentation row with its sync folder.
const MIGRATION_V18: &str = "
CREATE TABLE mount (
id INTEGER PRIMARY KEY,
kind TEXT NOT NULL
CHECK (kind IN ('myfiles', 'device', 'shared')),
sync_folder_id INTEGER REFERENCES sync_folder(id) ON DELETE CASCADE,
share_root_uid TEXT,
local_path TEXT NOT NULL DEFAULT '',
root_uid TEXT NOT NULL DEFAULT '',
root_share_id TEXT NOT NULL DEFAULT '',
mode TEXT NOT NULL DEFAULT 'unknown',
access TEXT NOT NULL DEFAULT 'rw'
CHECK (access IN ('rw', 'ro')),
CHECK (
(kind = 'myfiles' AND sync_folder_id IS NULL AND share_root_uid IS NULL)
OR
(kind = 'device' AND sync_folder_id IS NOT NULL AND share_root_uid IS NULL)
OR
(kind = 'shared' AND sync_folder_id IS NULL AND share_root_uid IS NOT NULL)
)
);
CREATE UNIQUE INDEX mount_myfiles
ON mount(kind) WHERE kind = 'myfiles';
CREATE UNIQUE INDEX mount_device
ON mount(sync_folder_id) WHERE kind = 'device';
CREATE UNIQUE INDEX mount_shared
ON mount(share_root_uid) WHERE kind = 'shared';
INSERT INTO mount (kind, sync_folder_id)
SELECT 'device', id FROM sync_folder;
CREATE TRIGGER mount_sync_folder_insert
AFTER INSERT ON sync_folder
BEGIN
INSERT OR IGNORE INTO mount (kind, sync_folder_id)
VALUES ('device', NEW.id);
END;
";
/// Schema v19: photo albums and their contents. Both are projections of the
/// server's listings — like `photos`, they are replaced wholesale on refresh —
/// so the Albums view opens from disk instead of re-enumerating every launch.
///
/// `album_photos` carries its own `ratio` / `thumb_state` rather than joining
/// `photos`: an album shared with us lives on the sharer's volume, so its
/// photos are not in our timeline at all and would otherwise have nowhere to
/// remember what a thumbnail attempt learned.
const MIGRATION_V19: &str = "
CREATE TABLE albums (
uid TEXT PRIMARY KEY,
name TEXT NOT NULL,
photo_count INTEGER NOT NULL DEFAULT 0,
cover_uid TEXT,
last_activity INTEGER,
shared INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL
);
CREATE TABLE album_photos (
album_uid TEXT NOT NULL,
uid TEXT NOT NULL,
capture_time INTEGER NOT NULL,
name TEXT,
media_type TEXT,
kind INTEGER NOT NULL DEFAULT 0,
ratio REAL,
thumb_state INTEGER NOT NULL DEFAULT 0,
seq INTEGER NOT NULL,
PRIMARY KEY (album_uid, uid)
);
CREATE INDEX idx_album_photos_seq ON album_photos(album_uid, seq);
CREATE INDEX idx_album_photos_uid ON album_photos(uid);
";
/// V20: whether a photo is a favorite.
///
/// A column rather than a tag table: `Favorite` is the only Proton photo tag a
/// user sets by hand (the rest — Video, Screenshot, Selfie, Raw… — are the
/// server's own classification, and the Photos page already derives its tabs
/// from the media type). Defaulting to 0 means an existing timeline stays
/// intact and each row learns its real value on the next refresh, the same way
/// `media_type` did.
const MIGRATION_V20: &str = "
ALTER TABLE photos ADD COLUMN favorite INTEGER NOT NULL DEFAULT 0;
CREATE INDEX IF NOT EXISTS idx_photos_favorite ON photos(favorite, seq);
";
/// V21: rebuild the node search index after fixing what counts as indexable.
///
/// The v16 backfill rooted its path walk at any node whose parent is null *or*
/// not cached, but the write path's `node_is_indexable_tx` demanded a
/// null-parent ancestor. Only the My Files root has one — a device folder's
/// root is stored in `device`/`sync_folder`, never in `nodes` — so every node
/// under a device folder was dropped from `nodes_fts` the first time it was
/// upserted after the backfill, and was unfindable from the prompt (2,705 of
/// 9,252 nodes on the account this was diagnosed against).
///
/// The rule now matches the backfill, so this replays it: same recursive walk
/// as v16, over the current `nodes`.
/// Two details the v16 backfill did not carry, both cheap here: a node under a
/// trashed folder stays out (the write path has always excluded it, and
/// `tombstone_subtree_tx` only marks a subtree it was told about), and the walk
/// is depth-capped so a parent cycle in a corrupted database cannot spin the
/// migration forever.
const MIGRATION_V21: &str = "
DROP TABLE IF EXISTS nodes_fts;
CREATE VIRTUAL TABLE nodes_fts USING fts5(
name, path, tokenize='trigram'
);
INSERT INTO nodes_fts (rowid, name, path)
WITH RECURSIVE paths(rowid, uid, path, trashed, depth) AS (
-- Only a true root (no parent at all) contributes nothing to the path: its
-- name is the mount. A row whose parent is merely uncached is an ordinary
-- folder and keeps its name, which is what `path_of` does at runtime.
SELECT n.rowid, n.uid,
CASE WHEN n.parent_uid IS NULL THEN '' ELSE n.name END, n.trashed, 0
FROM nodes n
WHERE n.parent_uid IS NULL
OR NOT EXISTS (SELECT 1 FROM nodes p WHERE p.uid = n.parent_uid)
UNION ALL
SELECT n.rowid, n.uid,
CASE WHEN paths.path = '' THEN n.name ELSE paths.path || '/' || n.name END,
MAX(paths.trashed, n.trashed), paths.depth + 1
FROM nodes n JOIN paths ON n.parent_uid = paths.uid
WHERE paths.depth < 256
)
SELECT n.rowid, n.name, paths.path
FROM nodes n JOIN paths ON paths.rowid = n.rowid
WHERE paths.trashed = 0;
";
/// V22: indexes for the lookups that were scanning, each paired with the table
/// it needs to exist for.
///
/// Each backs a query on a hot path that had no index to use:
///
/// - `sync_entry(remote_uid)` — the reconcile resolves a remote uid to its
/// local entry once per node, per pass.
/// - `local_files(scan_gen)` — the sweep that retires the previous generation
/// after a local index scan.
/// - `nodes(parent_uid) WHERE trashed = 0` — every listing, with the partial
/// index keeping it to the live rows a listing actually wants.
///
/// Deliberately *not* here: an index on `pending_op(next_attempt_at, id)`.
/// `next_due_op` orders by `id`, which is the rowid, so it already stops at the
/// first due row; offering the planner a `next_attempt_at` index makes it scan
/// that index and then sort the matches by `id` instead —
/// `next_due_op_does_not_scale_with_queue_length` measures the regression.
const MIGRATION_V22: [(&str, &str); 3] = [
(
"sync_entry",
"CREATE INDEX IF NOT EXISTS idx_sync_entry_remote_uid ON sync_entry(remote_uid);",
),
(
"local_files",
"CREATE INDEX IF NOT EXISTS idx_local_files_scan_gen ON local_files(scan_gen);",
),
(
"nodes",
"CREATE INDEX IF NOT EXISTS idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;",
),
];
/// V24: the revisions this daemon sealed itself, so restarting does not reopen
/// the conflict hole B70 layer B closed.
///
/// `Core::own_sealed_revs` held these in memory: a queued write whose baseline
/// names an earlier revision is a conflict *unless* the remote sits at a
/// revision we sealed, in which case it is one writer stalling and resuming and
/// must supersede rather than fork a `(sync-conflict)` copy. Losing the map at
/// shutdown meant the first drain after a restart could not tell the two apart.
///
/// `sealed_at` is what bounds the table: entries older than the window in which
/// a stall→resume is plausible are pruned on write.
const MIGRATION_V24: &str = "
CREATE TABLE IF NOT EXISTS own_sealed_rev (
uid TEXT PRIMARY KEY,
revision_id TEXT NOT NULL,
sealed_at INTEGER NOT NULL
);
";
/// V25: `nodes.path` — a node's mountpoint-relative path, stored rather than
/// walked.
///
/// `path_of` was a recursive CTE per node, and search ran it once per candidate
/// (up to a thousand per keystroke) on top of the one the write path already ran
/// per descendant of every folder upsert. The value only changes when a node's
/// name or parent changes, which is exactly when the write path is already
/// walking the subtree — so it belongs in a column.
///
/// The backfill is the same walk V21 used for `nodes_fts`, minus the trashed
/// filter: a trashed node stays out of the index but still has a path. Roots get
/// `''` (their name is the mount), and a node whose parent is merely uncached
/// keeps its own name — both matching what `path_of` returned at runtime.
///
/// Materialised into a temp table rather than written by a correlated subquery,
/// so the recursive walk runs once instead of once per row.
const MIGRATION_V25: &str = "
ALTER TABLE nodes ADD COLUMN path TEXT;
CREATE TEMP TABLE node_paths (uid TEXT PRIMARY KEY, path TEXT NOT NULL);
INSERT INTO node_paths (uid, path)
WITH RECURSIVE paths(uid, path, depth) AS (
SELECT n.uid, CASE WHEN n.parent_uid IS NULL THEN '' ELSE n.name END, 0
FROM nodes n
WHERE n.parent_uid IS NULL
OR NOT EXISTS (SELECT 1 FROM nodes p WHERE p.uid = n.parent_uid)
UNION ALL
SELECT n.uid,
CASE WHEN paths.path = '' THEN n.name ELSE paths.path || '/' || n.name END,
paths.depth + 1
FROM nodes n JOIN paths ON n.parent_uid = paths.uid
WHERE paths.depth < 256
)
SELECT uid, path FROM paths;
UPDATE nodes
SET path = (SELECT p.path FROM node_paths p WHERE p.uid = nodes.uid)
WHERE EXISTS (SELECT 1 FROM node_paths p WHERE p.uid = nodes.uid);
DROP TABLE node_paths;
";
/// V26: `pending_op.claimed_at` — the drain's claim column.
///
/// The drain was one thread taking one op at a time, so a 10 GiB upload held
/// every queued rename and small write behind it for as long as it ran. Several
/// workers can share the queue only if a claim is visible to the others: this
/// column is what makes "already being drained by someone" a fact about the row
/// rather than about one thread's local variable.
///
/// `0` is unclaimed; anything else is the ms at which a worker took it. A
/// claim is process-local state that outlives nothing — the single-writer lock
/// means a non-zero value found at open belongs to a crashed run, and
/// [`Db::clear_op_claims`](super::Db::clear_op_claims) drops the lot.
///
/// The index is partial so it costs nothing on the ordinary row: only the
/// handful of claimed rows are in it, which is exactly the set the claim query
/// has to subtract to keep two workers off one node.
const MIGRATION_V26: &str = "
ALTER TABLE pending_op ADD COLUMN claimed_at INTEGER NOT NULL DEFAULT 0;
CREATE INDEX IF NOT EXISTS idx_pending_op_claimed
ON pending_op(uid) WHERE claimed_at <> 0;
";
/// Schema v27: when an op was first deferred by the local access check.
///
/// An access deferral consumes no attempt and records no error — the op has not
/// failed remotely, it has not been sent at all. That is right for a permission
/// change the user is about to undo, and wrong for a condition that never
/// clears: the row is re-deferred every
/// [`DRAIN_ACCESS_RECHECK`](../../../pdfs_fuse/drain/constant.DRAIN_ACCESS_RECHECK.html)
/// forever while `attempts`, `last_error`, `failing` and the logs all stay
/// empty, so a queue that can never drain is indistinguishable from a busy one.
/// Found in the field as 28 revision ops holding 18 GiB of accepted writes for
/// 30 days, none of them visible anywhere.
///
/// `0` means "not currently access-deferred"; anything else is the ms at which
/// the first consecutive deferral happened, which is what
/// [`Db::defer_op_for_access`](super::Db::defer_op_for_access) measures the
/// deferral against.
const MIGRATION_V27: &str = "
ALTER TABLE pending_op ADD COLUMN access_deferred_since INTEGER NOT NULL DEFAULT 0;
";
/// Schema v28: sub-second precision for a mirror folder's local baseline.
///
/// A reconcile pass decides whether the local side of a path changed by
/// comparing `(local_mtime, local_size)` against the baseline. Both sides of
/// that comparison were whole seconds, so an edit that landed in the same second
/// as the last sync and left the file the same length — a fixed-size record
/// rewritten, an image re-exported, a database page updated in place — read as
/// *unchanged* and was never uploaded. The remote copy silently stayed at the
/// older content until something else about the file moved (bugs.md B25).
///
/// Nullable rather than a converted `local_mtime * 1_000_000_000`: the
/// nanosecond part of an existing baseline is genuinely unknown, and inventing
/// zero for it would make every already-synced file compare as changed and
/// re-upload an entire mirror on first start. `NULL` means "this row predates
/// the column", which the comparison reads as "fall back to whole seconds" —
/// exactly the behaviour that row was written under. Each row gains a real value
/// the next time its path is synced.
const MIGRATION_V28: &str = "
ALTER TABLE sync_entry ADD COLUMN local_mtime_ns INTEGER;
";
/// Schema v29: the trash listing remembers each trashed node's parent, so a
/// restore can reason about the shape of what it is putting back.
///
/// Trash is a flat list of uids on the wire, but the user deletes a *tree*: a
/// folder and, often, items inside it that were trashed separately. Restoring
/// one of those in isolation is what the parent link fixes — a child restored
/// under a still-trashed parent lands somewhere invisible, and a folder restored
/// without its separately-trashed contents comes back empty.
///
/// `NULL` means "this row predates the column" (or the node is a volume root);
/// the next trash refresh fills it in, and until then the restore behaves as it
/// did before.
const MIGRATION_V29: &str = "
ALTER TABLE trash ADD COLUMN parent_uid TEXT;
CREATE INDEX IF NOT EXISTS idx_trash_parent ON trash(parent_uid);
";
/// Schema v30: a RAW and the JPEG of the same shot are one photo to the person
/// who took it, so the timeline records what ties them together.
///
/// `content_hash` and `main_uid` come from the server's own photo properties —
/// the duplicate-detection hash, and the main photo a related one belongs to.
/// `group_key` is what the gallery pages by: every member of one shot carries
/// the same key, and the member the grid shows is the one whose `main_uid` is
/// its own uid.
///
/// All three are `NULL` until the next timeline refresh resolves the photo's
/// node, and a row without a `group_key` is its own group — which is exactly how
/// the gallery behaved before this column existed.
const MIGRATION_V30: &str = "
ALTER TABLE photos ADD COLUMN content_hash TEXT;
ALTER TABLE photos ADD COLUMN main_uid TEXT;
ALTER TABLE photos ADD COLUMN group_key TEXT;
CREATE INDEX IF NOT EXISTS idx_photos_group ON photos(group_key);
";
/// Schema v31: which photos a refresh still has to read from the server.
///
/// A refresh used to resolve every photo's node to learn its name, media type,
/// favorite tag, content hash and photo relation — minutes of round-trips for a
/// library of any size, repeated every time the timeline went stale. `resolved_at`
/// records that the work was already done, so a refresh only reads the photos it
/// has never read and the ones a remote event marked stale (`resolved_at = NULL`).
///
/// The backfill treats an existing timeline as resolved, so an upgrade does not
/// re-read the whole library once. `WHERE name IS NOT NULL` keeps the photos
/// whose earlier resolve had failed eligible: a nameless row is one nothing was
/// ever learned about.
const MIGRATION_V31: &str = "
ALTER TABLE photos ADD COLUMN resolved_at INTEGER;
CREATE INDEX IF NOT EXISTS idx_photos_unresolved ON photos(resolved_at);
UPDATE photos SET resolved_at = strftime('%s','now') WHERE name IS NOT NULL;
";
/// V32: give videos written off as un-thumbnailable another chance.
///
/// Before local video thumbnails, the fallback fed a video's bytes to the image
/// decoder, which could only fail, and that failure was recorded as the
/// permanent "no thumbnail" verdict. Run once per table in `{table}`.
const MIGRATION_V32: &str = "UPDATE {table} SET thumb_state = 0 WHERE kind = 1 AND thumb_state = 2";
/// Schema v33: where each photo was taken.
///
/// `latitude` and `longitude` come from the location other clients write into
/// a photo's extended attributes; `place_id` is the GeoNames id of the nearest
/// town (see `crate::places`), so the Places view is a plain `GROUP BY`.
///
/// Every photo is marked unresolved, so the next refresh reads the location of
/// the whole library once. That is the backfill: the metadata is on the node
/// the refresh reads anyway, and nothing has to be downloaded.
const MIGRATION_V33: &str = "
ALTER TABLE photos ADD COLUMN latitude REAL;
ALTER TABLE photos ADD COLUMN longitude REAL;
ALTER TABLE photos ADD COLUMN place_id INTEGER;
CREATE INDEX IF NOT EXISTS idx_photos_place ON photos(place_id);
UPDATE photos SET resolved_at = NULL;
";
/// Schema v34: what each photo looks like, for finding similar ones.
///
/// `similar_hash` is a 64-bit difference hash of the photo's thumbnail (see
/// `crate::similar`): resized and re-encoded copies of one picture land a few
/// bits apart, where the server's `content_hash` only matches byte-identical
/// files. `NULL` until the daemon has hashed the thumbnail, which it does as
/// thumbnails arrive and when the similar-photo finder asks for the rest.
const MIGRATION_V34: &str = "
ALTER TABLE photos ADD COLUMN similar_hash INTEGER;
";
/// Schema v35: when each trashed node went to the trash.
///
/// The API does not tell this client, so `trashed_at` is when the daemon first
/// saw the node in the trash, in epoch seconds. `NULL` when it cannot say: rows
/// from before this version, and nodes found by a refresh that did not follow a
/// complete one, which may have been in the trash for a long time already.
const MIGRATION_V35: &str = "
ALTER TABLE trash ADD COLUMN trashed_at INTEGER;
";
+345 -10
View File
@@ -2,9 +2,17 @@
//! inode bookkeeping, full-text search, content-cache LRU tracking, and pins. //! inode bookkeeping, full-text search, content-cache LRU tracking, and pins.
//! //!
//! Only the daemon (`pdfs-fuse`) opens this for writes; the GUI and CLI reach //! Only the daemon (`pdfs-fuse`) opens this for writes; the GUI and CLI reach
//! the same data through the control socket. The connection is wrapped in a //! the same data through the control socket. There is one write connection
//! `Mutex` because the FUSE callbacks are synchronous and already serialize //! behind a `Mutex` — SQLite allows exactly one writer, and the single-instance
//! behind the `State` lock, so a connection pool would be overkill. //! `flock` makes sure it is ours — and a small pool of read-only connections
//! behind [`Db::read`], which every `SELECT`-only method uses.
//!
//! The pool is what makes WAL worth having. The original justification for a
//! single connection was that "FUSE callbacks already serialize behind the
//! `State` lock"; that stopped being true when reads moved onto an 11-thread
//! worker pool, and a lookup then queued behind whatever listing happened to be
//! committing. Under WAL a reader never blocks the writer and vice versa, so the
//! only thing left serialising them was this process's own mutex.
//! //!
//! This module is the P0 foundation: it opens the database, enables WAL, and //! This module is the P0 foundation: it opens the database, enables WAL, and
//! applies the forward-only schema migrations. Write-through of nodes, the //! applies the forward-only schema migrations. Write-through of nodes, the
@@ -18,34 +26,41 @@
use parking_lot::Mutex; use parking_lot::Mutex;
use rusqlite::Connection; use rusqlite::Connection;
use crate::Result; use crate::{Error, Result};
use std::path::Path; use std::fs::File;
use std::os::fd::AsRawFd;
use std::path::{Path, PathBuf};
mod activity; mod activity;
mod albums;
mod cache; mod cache;
mod devices; mod devices;
mod local; mod local;
mod maintenance; mod maintenance;
mod migrations; mod migrations;
mod mounts;
mod nodes; mod nodes;
mod ops; mod ops;
mod photos; mod photos;
mod pins; mod pins;
mod share_access;
mod state; mod state;
mod sync; mod sync;
mod trash; mod trash;
mod utils; mod utils;
pub use albums::StoredAlbum;
pub use cache::CacheEntryInput; pub use cache::CacheEntryInput;
pub use devices::StoredDevice; pub use devices::StoredDevice;
pub use local::LocalFileHit; pub use local::LocalFileHit;
pub use maintenance::{DbStats, VacuumOutcome}; pub use maintenance::{DbStats, VacuumOutcome};
pub use nodes::{SearchHit, StoredNode}; pub use nodes::{PublishedSharedRoot, SearchHit, StoredNode};
pub use ops::{ pub use ops::{
AttachedBlob, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, AttachedBlob, FAILING_ATTEMPTS, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION,
PendingCounts, PendingOp, op_supersedes, OP_TRASH, PARK_EXPIRY_MS, PARK_UNTIL, PendingCounts, PendingOp, RenameMeta, op_supersedes,
}; };
pub use photos::{StoredPhoto, THUMB_HAVE, THUMB_NONE, THUMB_UNKNOWN}; pub use photos::{StoredPhoto, StoredPlace, THUMB_HAVE, THUMB_NONE, THUMB_UNKNOWN, TimelineRow};
pub use pins::PinRow;
pub use sync::{StoredSyncEntry, StoredSyncFolder}; pub use sync::{StoredSyncEntry, StoredSyncFolder};
pub use trash::StoredTrash; pub use trash::StoredTrash;
@@ -61,18 +76,208 @@ const WAL_SIZE_LIMIT: i64 = 64 * 1024 * 1024;
/// genuinely stuck holder surfaces as an error rather than an apparent hang. /// genuinely stuck holder surfaces as an error rather than an apparent hang.
const BUSY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5); const BUSY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
/// Page cache, in KiB (applied negated, which is how SQLite reads a size in KiB
/// rather than in pages). 32 MiB against a database that is typically tens of
/// MiB: large enough to hold the working set of a listing-heavy session, small
/// enough to be unremarkable in a desktop daemon's RSS.
const CACHE_SIZE_KIB: i64 = 32 * 1024;
/// Ceiling on the memory-mapped window over the database file. 256 MiB covers
/// any realistic index while keeping the mapping bounded.
const MMAP_SIZE: i64 = 256 * 1024 * 1024;
/// WAL pages between automatic checkpoints (default 1000, i.e. ~4 MiB). Halved
/// so the synchronous checkpoint a committing thread inherits stays short.
const WAL_AUTOCHECKPOINT_PAGES: i64 = 500;
/// Handle to the unified metadata database. /// Handle to the unified metadata database.
/// ///
/// Cheap to wrap in an `Arc`; clone the `Arc`, not this. All access goes through /// Cheap to wrap in an `Arc`; clone the `Arc`, not this. All access goes through
/// the inner `Mutex<Connection>`. /// the inner `Mutex<Connection>`.
pub struct Db { pub struct Db {
conn: Mutex<Connection>, conn: Mutex<Connection>,
/// Read-only connections for `SELECT`-only methods. `None` for an in-memory
/// database, where a second connection would open a different, empty one —
/// those fall back to the write connection.
readers: Option<ReadPool>,
/// The single-writer lock, held open for as long as this handle lives.
///
/// Never read; dropping the `File` is what releases the `flock`, so it has
/// to be owned here rather than by `open`'s stack frame. `None` for
/// in-memory databases, which nothing else can reach.
_single_writer: Option<File>,
}
/// Idle read-only connections, opened on demand and kept for reuse.
///
/// Not a bounded pool: the number in flight is bounded by the number of threads
/// that can be inside a read at once (the FUSE worker lanes plus the control
/// handlers), and blocking one of those on a permit would reintroduce exactly
/// the queueing this exists to remove. Only the *idle* set is capped, so a burst
/// does not leave a connection per thread parked forever.
struct ReadPool {
path: PathBuf,
idle: Mutex<Vec<Connection>>,
}
/// How many idle read connections to keep. Enough that the steady state never
/// reopens, small enough to be unremarkable.
const MAX_IDLE_READERS: usize = 4;
impl ReadPool {
fn take(&self) -> Option<Connection> {
if let Some(conn) = self.idle.lock().pop() {
return Some(conn);
}
match open_read_only(&self.path) {
Ok(conn) => Some(conn),
Err(error) => {
// Falling back to the write connection is correct, just slower,
// so this is a warning and not a failed read.
tracing::warn!(%error, "could not open a read-only connection");
None
}
}
}
fn put(&self, conn: Connection) {
let mut idle = self.idle.lock();
if idle.len() < MAX_IDLE_READERS {
idle.push(conn);
}
}
}
/// A connection to read through: one borrowed from [`ReadPool`], or the write
/// connection when there is no pool (in-memory) or opening one failed.
///
/// Derefs to `Connection`, so a method converts by swapping `self.conn.lock()`
/// for `self.read()` and nothing else.
pub(super) struct Reader<'a> {
pooled: Option<Connection>,
writer: Option<parking_lot::MutexGuard<'a, Connection>>,
pool: Option<&'a ReadPool>,
}
impl std::ops::Deref for Reader<'_> {
type Target = Connection;
fn deref(&self) -> &Connection {
self.pooled
.as_ref()
.or(self.writer.as_deref())
.expect("a Reader always holds one of the two")
}
}
impl Drop for Reader<'_> {
fn drop(&mut self) {
if let (Some(conn), Some(pool)) = (self.pooled.take(), self.pool) {
pool.put(conn);
}
}
}
/// Open a second connection to the same file for reads only.
///
/// `SQLITE_OPEN_READ_ONLY` plus `query_only` is belt and braces: the flag stops
/// a routing mistake at the file, the pragma stops it at the statement. The rest
/// mirror [`Db::open_configured`] — a read connection has its own page cache,
/// its own temp store and its own busy timeout, and gets nothing from the
/// writer's.
///
/// `journal_mode` is deliberately absent: it is a property of the database file,
/// which the writer already set to WAL, and a read-only connection cannot change
/// it anyway.
fn open_read_only(path: &Path) -> Result<Connection> {
use rusqlite::OpenFlags;
let conn = Connection::open_with_flags(
path,
OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX,
)?;
conn.busy_timeout(BUSY_TIMEOUT)?;
conn.pragma_update(None, "cache_size", -CACHE_SIZE_KIB)?;
conn.pragma_update(None, "temp_store", "MEMORY")?;
conn.pragma_update(None, "mmap_size", MMAP_SIZE)?;
conn.pragma_update(None, "query_only", true)?;
Ok(conn)
} }
impl Db { impl Db {
/// Open (creating if absent) the database at `path`, enable WAL, and bring /// Open (creating if absent) the database at `path`, enable WAL, and bring
/// the schema up to [`SCHEMA_VERSION`]. /// the schema up to [`SCHEMA_VERSION`].
///
/// Takes the single-instance lock first (see
/// [`acquire_single_writer_lock`]), and rebuilds from empty if what is on
/// disk turns out not to be a readable database (see [`is_corrupt`]).
pub fn open(path: &Path) -> Result<Self> { pub fn open(path: &Path) -> Result<Self> {
let single_writer = acquire_single_writer_lock(path)?;
let conn = match Self::open_configured(path) {
Ok(conn) => conn,
Err(e) if is_corrupt(&e) => {
// Every byte in here is derived from the account or from the
// content cache directory, so the recovery is to start over —
// loudly, and keeping the damaged file for a post-mortem rather
// than deleting evidence.
tracing::error!(
db = %path.display(),
error = %e,
"cache database is corrupt — moving it aside and rebuilding \
(metadata will be re-fetched; no user data is stored here)"
);
quarantine_corrupt_db(path)?;
Self::open_configured(path)?
}
Err(e) => return Err(e),
};
let db = Self {
conn: Mutex::new(conn),
// No pool for `:memory:`. A second connection to that name is a
// second, empty database rather than another view of this one, so
// every routed read would find a schema that does not exist.
readers: (!is_in_memory(path)).then(|| ReadPool {
path: path.to_path_buf(),
idle: Mutex::new(Vec::new()),
}),
_single_writer: single_writer,
};
// A drain claim names a worker thread in the process that took it, and
// the single-writer lock above says that process is not running. Any
// claim still on disk is therefore a crashed run's, and leaving it would
// hide those ops from every worker for good.
match db.clear_op_claims() {
Ok(0) => {}
Ok(n) => tracing::info!(ops = n, "released drain claims left by a previous run"),
Err(e) => tracing::warn!(error = %e, "clearing stale drain claims failed"),
}
Ok(db)
}
/// A connection for a `SELECT`-only method.
///
/// Reads see the last committed state, which is the same thing they saw
/// through the write connection: every method here commits before it
/// returns, and nothing holds a transaction open across calls.
pub(super) fn read(&self) -> Reader<'_> {
if let Some(pool) = &self.readers
&& let Some(conn) = pool.take()
{
return Reader {
pooled: Some(conn),
writer: None,
pool: Some(pool),
};
}
Reader {
pooled: None,
writer: Some(self.conn.lock()),
pool: None,
}
}
/// Open the file, apply every PRAGMA, and migrate. Split out of
/// [`open`](Self::open) so the corruption path can retry it verbatim.
fn open_configured(path: &Path) -> Result<Connection> {
let conn = Connection::open(path)?; let conn = Connection::open(path)?;
// WAL: readers never block the single writer. NORMAL sync is the // WAL: readers never block the single writer. NORMAL sync is the
// standard durability/throughput tradeoff for WAL. // standard durability/throughput tradeoff for WAL.
@@ -97,20 +302,53 @@ impl Db {
// (Bare SQLite does default to 0, which is where the belief that this // (Bare SQLite does default to 0, which is where the belief that this
// was unset came from. That default is not what we get.) // was unset came from. That default is not what we get.)
conn.busy_timeout(BUSY_TIMEOUT)?; conn.busy_timeout(BUSY_TIMEOUT)?;
// Page cache. The default is 2 MiB, against a schema whose hot table
// stores a JSON blob per node — a listing of a few thousand files
// evicts itself while it is being read.
conn.pragma_update(None, "cache_size", -CACHE_SIZE_KIB)?;
// Sorts and recursive CTEs materialise their intermediate results.
// Every `ORDER BY … COLLATE NOCASE` listing and every ancestor walk
// spills those to a file in `/tmp` by default; they are small and
// short-lived, so memory is both faster and less surprising.
conn.pragma_update(None, "temp_store", "MEMORY")?;
// Read the database through the page cache of the OS rather than
// `pread` per page. Bounded rather than unlimited so the daemon's RSS
// does not track the database size.
conn.pragma_update(None, "mmap_size", MMAP_SIZE)?;
// Checkpointing is synchronous in whichever thread happens to commit
// the transaction that crosses the threshold — often a FUSE callback.
// A smaller autocheckpoint makes that debt smaller and more frequent
// rather than rare and multi-second.
conn.pragma_update(None, "wal_autocheckpoint", WAL_AUTOCHECKPOINT_PAGES)?;
let db = Self { let db = Self {
conn: Mutex::new(conn), conn: Mutex::new(conn),
readers: None,
_single_writer: None,
}; };
db.migrate()?; db.migrate()?;
Ok(db) Ok(db.into_conn())
}
/// Unwrap the connection back out of a temporary handle. Only
/// [`open_configured`](Self::open_configured) uses it, to run `migrate`
/// (which is written against `&self`) before the real handle exists.
fn into_conn(self) -> Connection {
self.conn.into_inner()
} }
/// Open an in-memory database. For tests. /// Open an in-memory database. For tests.
#[cfg(test)] #[cfg(test)]
pub fn open_in_memory() -> Result<Self> { pub fn open_in_memory() -> Result<Self> {
let conn = Connection::open_in_memory()?; let conn = Connection::open_in_memory()?;
// Keep relational behavior identical to `open`: V18 relies on the
// `mount.sync_folder_id` cascade, and SQLite disables foreign keys per
// connection unless explicitly enabled.
conn.pragma_update(None, "foreign_keys", "ON")?;
let db = Self { let db = Self {
conn: Mutex::new(conn), conn: Mutex::new(conn),
readers: None,
_single_writer: None,
}; };
db.migrate()?; db.migrate()?;
Ok(db) Ok(db)
@@ -124,5 +362,102 @@ impl Db {
} }
} }
/// Whether `path` names the in-memory database rather than a file. Both
/// spellings SQLite accepts are checked, because neither has a lock file or a
/// corruption story.
fn is_in_memory(path: &Path) -> bool {
let p = path.as_os_str();
p == ":memory:" || p.is_empty()
}
/// Take the single-instance write lock for the database at `path`.
///
/// The daemon is the only writer by design — the front-ends go through the
/// control socket — but nothing stopped a hand-run `pdfs mount` from starting
/// alongside the systemd unit, at which point two processes own the same inode
/// space, the same content cache, and the same drain queue. SQLite's own
/// locking makes each *statement* safe and does nothing about that.
///
/// `flock` on a sibling `.lock` file rather than on the database itself:
/// SQLite uses POSIX record locks, so the two never interact, and a lock file
/// left behind carries no state — the lock lives in the kernel and dies with
/// the process, however it exits.
fn acquire_single_writer_lock(path: &Path) -> Result<Option<File>> {
if is_in_memory(path) {
return Ok(None);
}
let lock_path = lock_path_for(path);
let file = std::fs::OpenOptions::new()
.create(true)
.read(true)
.truncate(false)
.write(true)
.open(&lock_path)?;
// SAFETY: `file` owns the descriptor and outlives the call.
let rc = unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX | libc::LOCK_NB) };
if rc != 0 {
let e = std::io::Error::last_os_error();
if e.raw_os_error() == Some(libc::EWOULDBLOCK) {
return Err(Error::Other(format!(
"another Proton Drive daemon is already using {} — \
stop it first (`systemctl --user stop proton-drive.service`)",
path.display()
)));
}
return Err(Error::Io(e));
}
Ok(Some(file))
}
fn lock_path_for(path: &Path) -> PathBuf {
let mut name = path.as_os_str().to_owned();
name.push(".lock");
PathBuf::from(name)
}
/// Whether `e` says the file is not a usable database, as opposed to any other
/// failure. These are the only two conditions rebuilding fixes; a disk error, a
/// permission problem or a newer-than-supported schema must all still fail.
fn is_corrupt(e: &Error) -> bool {
matches!(
e,
Error::Db(rusqlite::Error::SqliteFailure(inner, _))
if inner.code == rusqlite::ErrorCode::DatabaseCorrupt
|| inner.code == rusqlite::ErrorCode::NotADatabase
)
}
/// Move a damaged database (and its WAL sidecars) out of the way so the next
/// open starts from empty.
///
/// Renamed rather than deleted: this is a cache, but it is also the only
/// evidence of what went wrong, and it costs one file the user can remove.
fn quarantine_corrupt_db(path: &Path) -> Result<()> {
let stamp = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_secs())
.unwrap_or(0);
for suffix in ["", "-wal", "-shm"] {
let mut from = path.as_os_str().to_owned();
from.push(suffix);
let from = PathBuf::from(from);
if !from.exists() {
continue;
}
let mut to = path.as_os_str().to_owned();
to.push(format!(".corrupt-{stamp}{suffix}"));
let to = PathBuf::from(to);
// A failure to rename the sidecars is survivable — SQLite discards a
// WAL whose database header no longer matches — but failing to move the
// database itself is not, because the retry would hit the same file.
match std::fs::rename(&from, &to) {
Ok(()) => tracing::warn!(from = %from.display(), to = %to.display(), "quarantined"),
Err(e) if suffix.is_empty() => return Err(Error::Io(e)),
Err(e) => tracing::warn!(file = %from.display(), error = %e, "could not quarantine"),
}
}
Ok(())
}
#[cfg(test)] #[cfg(test)]
mod tests; mod tests;
+179
View File
@@ -0,0 +1,179 @@
//! Unified mount/location presentation rows.
//!
//! A device row stores only its `sync_folder` identity. Its path, remote root,
//! mode, state, pending mode, and last-sync timestamp are selected from
//! `sync_folder`, which remains the authoritative sync-engine table.
use rusqlite::params;
use super::Db;
use crate::mounts::{MountAccess, MountKind, MountMode, MountSpec};
use crate::{Error, Result};
impl Db {
/// Project the configured primary mount into the presentation table.
///
/// `root_share_id` is optional because an offline startup may only have the
/// cached root node. A previously learned share id is retained only when it
/// belongs to the same root uid.
pub fn mount_upsert_my_files(
&self,
local_path: &str,
root_uid: &str,
root_share_id: Option<&str>,
) -> Result<i64> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let updated = tx.execute(
"UPDATE mount
SET local_path = ?1,
root_share_id = CASE
WHEN root_uid = ?2
THEN COALESCE(NULLIF(?3, ''), root_share_id)
ELSE COALESCE(?3, '')
END,
root_uid = ?2,
mode = 'ondemand',
access = 'rw'
WHERE kind = 'myfiles'",
params![local_path, root_uid, root_share_id],
)?;
let id = if updated == 0 {
tx.execute(
"INSERT INTO mount
(kind, local_path, root_uid, root_share_id, mode, access)
VALUES ('myfiles', ?1, ?2, COALESCE(?3, ''), 'ondemand', 'rw')",
params![local_path, root_uid, root_share_id],
)?;
tx.last_insert_rowid()
} else {
tx.query_row("SELECT id FROM mount WHERE kind = 'myfiles'", [], |row| {
row.get(0)
})?
};
tx.commit()?;
Ok(id)
}
/// Store a share id learned asynchronously, but only if the projection still
/// names the root it was resolved for.
pub fn mount_repair_my_files_share_id(
&self,
root_uid: &str,
root_share_id: &str,
) -> Result<bool> {
let conn = self.conn.lock();
Ok(conn.execute(
"UPDATE mount
SET root_share_id = ?2
WHERE kind = 'myfiles'
AND root_uid = ?1",
params![root_uid, root_share_id],
)? > 0)
}
/// Every local location, with device fields joined from `sync_folder`.
pub fn mount_list(&self) -> Result<Vec<MountSpec>> {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT
m.id,
m.kind,
m.sync_folder_id,
m.share_root_uid,
CASE WHEN m.kind = 'device' THEN sf.local_path ELSE m.local_path END,
CASE WHEN m.kind = 'device' THEN sf.remote_uid ELSE m.root_uid END,
CASE WHEN m.kind = 'device' THEN sf.remote_share_id ELSE m.root_share_id END,
CASE WHEN m.kind = 'device' THEN sf.mode ELSE m.mode END,
m.access,
CASE WHEN m.kind = 'device' THEN sf.state ELSE 'idle' END,
CASE WHEN m.kind = 'device' THEN sf.last_sync ELSE 0 END,
CASE WHEN m.kind = 'device' THEN sf.pending_mode ELSE NULL END
FROM mount AS m
LEFT JOIN sync_folder AS sf ON sf.id = m.sync_folder_id
ORDER BY CASE m.kind
WHEN 'myfiles' THEN 0
WHEN 'device' THEN 1
ELSE 2
END,
m.id",
)?;
let rows = stmt.query_map([], |row| {
Ok((
row.get::<_, i64>(0)?,
row.get::<_, String>(1)?,
row.get::<_, Option<i64>>(2)?,
row.get::<_, Option<String>>(3)?,
row.get::<_, String>(4)?,
row.get::<_, String>(5)?,
row.get::<_, String>(6)?,
row.get::<_, String>(7)?,
row.get::<_, String>(8)?,
row.get::<_, String>(9)?,
row.get::<_, i64>(10)?,
row.get::<_, Option<String>>(11)?,
))
})?;
let mut mounts = Vec::new();
for row in rows {
let (
id,
kind,
sync_folder_id,
share_root_uid,
local_path,
root_uid,
root_share_id,
mode,
access,
state,
last_sync,
pending_mode,
) = row?;
let kind = match kind.as_str() {
"myfiles" => MountKind::MyFiles,
"device" => MountKind::Device {
sync_folder_id: sync_folder_id.ok_or_else(|| {
Error::Other(format!("mount row {id} has no sync_folder_id"))
})?,
},
"shared" => MountKind::Shared {
share_root_uid: share_root_uid.ok_or_else(|| {
Error::Other(format!("mount row {id} has no share_root_uid"))
})?,
},
other => {
return Err(Error::Other(format!(
"mount row {id} has unknown kind {other:?}"
)));
}
};
let access = match access.as_str() {
"rw" => MountAccess::Rw,
"ro" => MountAccess::Ro,
other => {
return Err(Error::Other(format!(
"mount row {id} has unknown access {other:?}"
)));
}
};
mounts.push(MountSpec {
id,
kind,
local_path,
root_uid,
root_share_id,
mode: MountMode::from(mode.as_str()),
access,
state,
last_sync,
pending_mode: pending_mode.as_deref().map(MountMode::from),
mounted: false,
progress: None,
paused: false,
});
}
Ok(mounts)
}
}
+586 -110
View File
@@ -1,14 +1,18 @@
//! Node rows: the persisted mirror of the remote tree, plus the trigram //! Node rows: the persisted mirror of the remote tree, plus the trigram
//! full-text index over names and paths that backs `Request::Search`. //! full-text index over names and paths that backs `Request::Search`.
use rusqlite::{OptionalExtension, params}; use std::collections::HashSet;
use rusqlite::{OptionalExtension, Transaction, params};
use super::Db; use super::Db;
use crate::Result; use crate::{Access, Result};
use proton_drive_rs::proton_sdk::ids::NodeUid; use proton_drive_rs::proton_sdk::ids::{LinkId, NodeUid, VolumeId};
use proton_drive_rs::{Node, NodeKind}; use proton_drive_rs::{Node, NodeKind};
use super::utils::{TRIGRAM_MIN, collect_pairs, like_escape, pair, path_of}; use super::utils::{
HitRow, TRIGRAM_MIN, collect_hits, hit_row, join_path, like_escape, path_of, walk_path_of,
};
pub struct StoredNode { pub struct StoredNode {
pub node: Node, pub node: Node,
@@ -22,6 +26,11 @@ pub struct SearchHit {
pub path: String, pub path: String,
} }
pub struct PublishedSharedRoot {
pub node: Node,
pub access: Access,
}
impl Db { impl Db {
pub fn upsert_node(&self, node: &Node) -> Result<()> { pub fn upsert_node(&self, node: &Node) -> Result<()> {
self.upsert_nodes(std::slice::from_ref(node)) self.upsert_nodes(std::slice::from_ref(node))
@@ -42,87 +51,188 @@ impl Db {
let mut conn = self.conn.lock(); let mut conn = self.conn.lock();
let tx = conn.transaction()?; let tx = conn.transaction()?;
for node in nodes { for node in nodes {
let json = serde_json::to_string(node)?; upsert_node_tx(&tx, node)?;
let uid = node.uid.to_string();
tx.execute(
"INSERT INTO nodes
(uid, parent_uid, name, is_dir, size, mtime, trashed, node_json)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)
ON CONFLICT(uid) DO UPDATE SET
parent_uid = excluded.parent_uid,
name = excluded.name,
is_dir = excluded.is_dir,
size = excluded.size,
mtime = excluded.mtime,
trashed = excluded.trashed,
node_json = excluded.node_json",
params![
uid,
node.parent_uid.as_ref().map(|u| u.to_string()),
node.name,
node.is_folder() as i64,
node_size(node),
node.modification_time,
node.trashed as i64,
json,
],
)?;
// FTS5 has no UPSERT, so refresh the row by delete-then-insert.
// Trashed nodes are kept out of the index entirely so they never
// surface in search results.
//
// Keyed by the node's rowid, never by its uid: the uid is not an
// indexed FTS column, so deleting by it scans the entire index once
// per node written (B12). The rowid is stable across the upsert
// above — `ON CONFLICT DO UPDATE` keeps the existing row — so it is
// the same key before and after.
let rowid: i64 = tx.query_row(
"SELECT rowid FROM nodes WHERE uid = ?1",
params![uid],
|row| row.get(0),
)?;
// A folder rename/move changes every descendant's searchable path.
// Refresh the affected subtree, while retaining efficient rowid
// deletes for the overwhelmingly common one-file case.
let mut affected = vec![rowid];
if node.is_folder() {
let mut stmt = tx.prepare(
"WITH RECURSIVE descendants(rowid, uid) AS (
SELECT rowid, uid FROM nodes WHERE parent_uid = ?1
UNION ALL
SELECT n.rowid, n.uid FROM nodes n
JOIN descendants d ON n.parent_uid = d.uid
) SELECT rowid FROM descendants",
)?;
for descendant in stmt.query_map(params![uid], |row| row.get(0))? {
affected.push(descendant?);
}
}
for affected_rowid in affected {
tx.execute(
"DELETE FROM nodes_fts WHERE rowid = ?1",
params![affected_rowid],
)?;
let indexed: Option<(String, String)> = tx
.query_row(
"SELECT name, uid FROM nodes WHERE rowid = ?1 AND trashed = 0",
params![affected_rowid],
|row| Ok((row.get(0)?, row.get(1)?)),
)
.optional()?;
if let Some((name, indexed_uid)) = indexed {
let path = path_of(&tx, &indexed_uid)?;
tx.execute(
"INSERT INTO nodes_fts (rowid, name, path) VALUES (?1, ?2, ?3)",
params![affected_rowid, name, path],
)?;
}
}
} }
tx.commit()?; tx.commit()?;
Ok(()) Ok(())
} }
/// Return visible direct children without consulting the parent's `listed`
/// flag. Synthetic shared listings use this to serve the last completed
/// snapshot while offline, including after a live event expired its TTL.
pub fn visible_children(&self, parent: &NodeUid) -> Result<Vec<Node>> {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT node_json FROM nodes
WHERE parent_uid = ?1 AND trashed = 0 AND node_json IS NOT NULL
ORDER BY name, uid",
)?;
let rows = stmt.query_map([parent.to_string()], |row| row.get::<_, String>(0))?;
let mut nodes = Vec::new();
for row in rows {
nodes.push(serde_json::from_str(&row?)?);
}
Ok(nodes)
}
/// Atomically publish one completed `Shared with me` root listing.
///
/// Roots absent from `accepted` and every persisted descendant are
/// tombstoned, not deleted. Accepted roots omitted from materialization keep
/// their visible snapshot but are downgraded to Viewer until membership is
/// verified again. Both cases preserve queued operations and staged writes.
pub fn publish_shared_roots(
&self,
parent: &NodeUid,
accepted: &[NodeUid],
roots: &[PublishedSharedRoot],
) -> Result<Vec<NodeUid>> {
let parent = parent.to_string();
let present: HashSet<String> = accepted.iter().map(NodeUid::to_string).collect();
let materialized: HashSet<String> = roots
.iter()
.map(|published| published.node.uid.to_string())
.collect();
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let existing = direct_child_uids_tx(&tx, &parent)?;
let removed: Vec<String> = existing
.into_iter()
.filter(|uid| !present.contains(uid))
.collect();
for published in roots {
if !present.contains(&published.node.uid.to_string()) {
continue;
}
upsert_node_tx(&tx, &published.node)?;
upsert_share_access_tx(&tx, &published.node.uid.to_string(), published.access)?;
}
for uid in present.difference(&materialized) {
upsert_share_access_tx(&tx, uid, Access::Viewer)?;
}
for root in &removed {
tombstone_subtree_tx(&tx, root)?;
upsert_share_access_tx(&tx, root, Access::Viewer)?;
}
tx.execute(
"UPDATE nodes SET listed = 1 WHERE uid = ?1",
params![parent],
)?;
tx.commit()?;
Ok(removed
.into_iter()
.filter_map(|uid| parse_node_uid(&uid))
.collect())
}
/// Atomically withdraw a deleted foreign subtree and deny stale handles.
///
/// Node rows and queued operations remain so retries and staged writes keep
/// their references. FTS and completed-listing state are withdrawn for the
/// whole subtree, and the deleted UID becomes a fail-closed authority.
pub fn tombstone_foreign_subtree(&self, uid: &NodeUid) -> Result<()> {
let uid = uid.to_string();
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
tombstone_subtree_tx(&tx, &uid)?;
upsert_share_access_tx(&tx, &uid, Access::Viewer)?;
tx.commit()?;
Ok(())
}
/// Publish a completed foreign-folder listing from its authoritative UID
/// list and the subset that materialized successfully. Accepted-but-omitted
/// children retain their previous snapshot; only absent UIDs are tombstoned.
pub fn publish_foreign_children(
&self,
parent: &NodeUid,
accepted: &[NodeUid],
materialized: &[Node],
) -> Result<Vec<NodeUid>> {
let parent = parent.to_string();
let present: HashSet<String> = accepted.iter().map(NodeUid::to_string).collect();
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let removed: Vec<String> = direct_child_uids_tx(&tx, &parent)?
.into_iter()
.filter(|uid| !present.contains(uid))
.collect();
for node in materialized {
if present.contains(&node.uid.to_string()) {
upsert_node_tx(&tx, node)?;
}
}
for uid in &removed {
tombstone_subtree_tx(&tx, uid)?;
}
tx.execute(
"UPDATE nodes SET listed = 1 WHERE uid = ?1",
params![parent],
)?;
tx.commit()?;
Ok(removed
.into_iter()
.filter_map(|uid| parse_node_uid(&uid))
.collect())
}
/// Atomically publish the synthetic root's pinned name, node/FTS state, and
/// Viewer authority.
///
/// The pin is inserted last so any failure there rolls back the node and
/// its descendant FTS updates as well as the access row. Returning `false`
/// guarantees no write occurred, which keeps cached root lookups O(1).
pub fn publish_virtual_root(&self, pinned_name_key: &str, node: &Node) -> Result<bool> {
let uid = node.uid.to_string();
let desired_parent = node.parent_uid.as_ref().map(NodeUid::to_string);
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let pinned_name: Option<String> = tx
.query_row(
"SELECT value FROM sync_state WHERE key = ?1",
params![pinned_name_key],
|row| row.get(0),
)
.optional()?;
let stored: Option<(Option<String>, String, i64, i64)> = tx
.query_row(
"SELECT parent_uid, name, is_dir, trashed FROM nodes WHERE uid = ?1",
params![uid],
|row| Ok((row.get(0)?, row.get(1)?, row.get(2)?, row.get(3)?)),
)
.optional()?;
let access: Option<String> = tx
.query_row(
"SELECT access FROM share_access WHERE root_uid = ?1",
params![uid],
|row| row.get(0),
)
.optional()?;
let unchanged = stored.is_some_and(|(parent, name, is_dir, trashed)| {
parent == desired_parent
&& name == node.name
&& is_dir == 1
&& trashed == node.trashed as i64
}) && access.as_deref() == Some(Access::Viewer.as_db_str())
&& pinned_name.is_some();
if unchanged {
tx.commit()?;
return Ok(false);
}
upsert_share_access_tx(&tx, &uid, Access::Viewer)?;
upsert_node_tx(&tx, node)?;
tx.execute(
"INSERT INTO sync_state (key, value) VALUES (?1, ?2)
ON CONFLICT(key) DO NOTHING",
params![pinned_name_key, node.name],
)?;
tx.commit()?;
Ok(true)
}
/// Drop a node row (delete or trash from the hot cache). Children rows are /// Drop a node row (delete or trash from the hot cache). Children rows are
/// not cascaded here; the daemon forgets a whole subtree node-by-node. /// not cascaded here; the daemon forgets a whole subtree node-by-node.
pub fn delete_node(&self, uid: &NodeUid) -> Result<()> { pub fn delete_node(&self, uid: &NodeUid) -> Result<()> {
@@ -148,10 +258,36 @@ impl Db {
Ok(()) Ok(())
} }
/// Atomically retire a landed trash operation and its retained authority.
///
/// Queued trash keeps the node row so drain-time permission checks can
/// resolve its shared-tree access. Removing the op first could resurrect
/// that row after a crash; removing the row first could strand a now
/// unauthorizable op. One transaction closes both windows.
pub fn complete_trash_op(&self, op_id: i64, uid: &NodeUid) -> Result<()> {
let uid = uid.to_string();
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let rowid: Option<i64> = tx
.query_row(
"SELECT rowid FROM nodes WHERE uid = ?1",
params![uid],
|row| row.get(0),
)
.optional()?;
tx.execute("DELETE FROM nodes WHERE uid = ?1", params![uid])?;
if let Some(rowid) = rowid {
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![rowid])?;
}
tx.execute("DELETE FROM pending_op WHERE id = ?1", params![op_id])?;
tx.commit()?;
Ok(())
}
/// Check if a folder node has any non-trashed children in the database. /// Check if a folder node has any non-trashed children in the database.
pub fn has_children(&self, parent_uid: &NodeUid) -> Result<bool> { pub fn has_children(&self, parent_uid: &NodeUid) -> Result<bool> {
let uid_str = parent_uid.to_string(); let uid_str = parent_uid.to_string();
let conn = self.conn.lock(); let conn = self.read();
let count: i64 = conn.query_row( let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM nodes WHERE parent_uid = ?1 AND trashed = 0", "SELECT COUNT(*) FROM nodes WHERE parent_uid = ?1 AND trashed = 0",
params![uid_str], params![uid_str],
@@ -160,6 +296,25 @@ impl Db {
Ok(count > 0) Ok(count > 0)
} }
/// A node's mountpoint-relative path, or `None` when it is not cached.
///
/// One indexed read of the stored `path`. Callers that need to relate many
/// nodes to a handful of roots — search decorating each hit with its
/// mountpoint — resolve the roots once through this and do the rest as
/// string arithmetic, instead of a `path_relative_to` walk per pair.
pub fn node_path(&self, uid: &str) -> Result<Option<String>> {
let conn = self.read();
let exists: Option<i64> = conn
.query_row("SELECT 1 FROM nodes WHERE uid = ?1", params![uid], |r| {
r.get(0)
})
.optional()?;
if exists.is_none() {
return Ok(None);
}
Ok(Some(path_of(&conn, uid)?))
}
/// Resolve `uid` relative to an ancestor node. /// Resolve `uid` relative to an ancestor node.
/// ///
/// This is used when a remote subtree has its own local sync mount: the /// This is used when a remote subtree has its own local sync mount: the
@@ -168,7 +323,7 @@ impl Db {
/// chain keeps callers from accidentally joining a Drive-wide path onto the /// chain keeps callers from accidentally joining a Drive-wide path onto the
/// wrong mountpoint. The ancestor itself resolves to the empty path. /// wrong mountpoint. The ancestor itself resolves to the empty path.
pub fn path_relative_to(&self, ancestor_uid: &str, uid: &str) -> Result<Option<String>> { pub fn path_relative_to(&self, ancestor_uid: &str, uid: &str) -> Result<Option<String>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"WITH RECURSIVE anc(uid, parent_uid, name, depth) AS ( "WITH RECURSIVE anc(uid, parent_uid, name, depth) AS (
SELECT uid, parent_uid, name, 0 FROM nodes WHERE uid = ?1 SELECT uid, parent_uid, name, 0 FROM nodes WHERE uid = ?1
@@ -200,19 +355,36 @@ impl Db {
/// than [`TRIGRAM_MIN`] fall back to a `LIKE` scan since trigram indexes /// than [`TRIGRAM_MIN`] fall back to a `LIKE` scan since trigram indexes
/// nothing below 3 chars. /// nothing below 3 chars.
pub fn search(&self, query: &str, limit: usize) -> Result<Vec<SearchHit>> { pub fn search(&self, query: &str, limit: usize) -> Result<Vec<SearchHit>> {
self.search_in(query, limit, None)
}
/// [`search`](Self::search) restricted to the subtree under the
/// mountpoint-relative folder `scope`. `None` or an empty scope searches
/// everything.
pub fn search_in(
&self,
query: &str,
limit: usize,
scope: Option<&str>,
) -> Result<Vec<SearchHit>> {
let scope = scope.map(|s| s.trim_matches('/')).filter(|s| !s.is_empty());
// Rows whose path is not persisted yet pass the SQL filter and are
// checked once their path is walked below.
let scope_pat = scope.map(|s| format!("{}/%", like_escape(s)));
let query = query.trim(); let query = query.trim();
if query.is_empty() { if query.is_empty() {
return Ok(Vec::new()); return Ok(Vec::new());
} }
let conn = self.conn.lock(); let conn = self.read();
let rows: Vec<(String, String)> = if query.chars().count() < TRIGRAM_MIN { let rows: Vec<HitRow> = if query.chars().count() < TRIGRAM_MIN {
let pat = format!("%{}%", like_escape(query)); let pat = format!("%{}%", like_escape(query));
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT node_json, uid FROM nodes "SELECT node_json, uid, path FROM nodes
WHERE name LIKE ?1 ESCAPE '\\' AND trashed = 0 AND node_json IS NOT NULL WHERE name LIKE ?1 ESCAPE '\\' AND trashed = 0 AND node_json IS NOT NULL
AND (?3 IS NULL OR path IS NULL OR path LIKE ?3 ESCAPE '\\')
ORDER BY name LIMIT ?2", ORDER BY name LIMIT ?2",
)?; )?;
collect_pairs(stmt.query_map(params![pat, limit as i64], pair)?)? collect_hits(stmt.query_map(params![pat, limit as i64, scope_pat], hit_row)?)?
} else { } else {
// Escape double quotes and quote each term, then combine with AND so // Escape double quotes and quote each term, then combine with AND so
// all terms must match but can appear in any order or position. // all terms must match but can appear in any order or position.
@@ -222,18 +394,29 @@ impl Db {
.collect::<Vec<_>>() .collect::<Vec<_>>()
.join(" AND "); .join(" AND ");
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT n.node_json, n.uid "SELECT n.node_json, n.uid, n.path
FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid
WHERE f.name MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL WHERE f.name MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL
AND (?3 IS NULL OR n.path IS NULL OR n.path LIKE ?3 ESCAPE '\\')
ORDER BY f.rank LIMIT ?2", ORDER BY f.rank LIMIT ?2",
)?; )?;
collect_pairs(stmt.query_map(params![phrase, limit as i64], pair)?)? collect_hits(stmt.query_map(params![phrase, limit as i64, scope_pat], hit_row)?)?
}; };
let mut hits = Vec::with_capacity(rows.len()); let mut hits = Vec::with_capacity(rows.len());
for (json, uid) in rows { for (json, uid, path) in rows {
let node: Node = serde_json::from_str(&json)?; let node: Node = serde_json::from_str(&json)?;
let path = path_of(&conn, &uid)?; let path = match path {
Some(path) => path,
None => walk_path_of(&conn, &uid)?,
};
if let Some(scope) = scope
&& !path
.strip_prefix(scope)
.is_some_and(|rest| rest.starts_with('/'))
{
continue;
}
hits.push(SearchHit { node, path }); hits.push(SearchHit { node, path });
} }
Ok(hits) Ok(hits)
@@ -248,54 +431,65 @@ impl Db {
if query.is_empty() || limit == 0 { if query.is_empty() || limit == 0 {
return Ok(Vec::new()); return Ok(Vec::new());
} }
let conn = self.conn.lock(); let conn = self.read();
let terms = candidate_trigrams(query); let terms = candidate_trigrams(query);
let lane_limit = limit.div_ceil(2); let lane_limit = limit.div_ceil(2);
let mut rows: Vec<(String, String)> = if terms.is_empty() { let mut rows: Vec<HitRow> = if terms.is_empty() {
let pat = format!("%{}%", like_escape(query)); let pat = format!("%{}%", like_escape(query));
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT node_json, uid FROM nodes "SELECT node_json, uid, path FROM nodes
WHERE name LIKE ?1 ESCAPE '\\' AND trashed = 0 AND node_json IS NOT NULL WHERE name LIKE ?1 ESCAPE '\\' AND trashed = 0 AND node_json IS NOT NULL
ORDER BY name LIMIT ?2", ORDER BY name LIMIT ?2",
)?; )?;
collect_pairs(stmt.query_map(params![pat, limit as i64], pair)?)? collect_hits(stmt.query_map(params![pat, limit as i64], hit_row)?)?
} else { } else {
let expression = terms.join(" OR "); let expression = terms.join(" OR ");
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT n.node_json, n.uid "SELECT n.node_json, n.uid, n.path
FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid
WHERE nodes_fts MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL WHERE nodes_fts MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL
ORDER BY f.rank LIMIT ?2", ORDER BY f.rank LIMIT ?2",
)?; )?;
collect_pairs(stmt.query_map(params![expression, lane_limit as i64], pair)?)? collect_hits(stmt.query_map(params![expression, lane_limit as i64], hit_row)?)?
}; };
// A short substitution can destroy every trigram (`vedio` vs `video`). // Trigram rank does not know that a name *starts* with the query, and a
// Feed the scorer an additional indexed same-initial lane so it can // short substitution can destroy every trigram outright (`vedio` vs
// recover those candidates without scanning the full node table. // `video`). Two indexed lanes cover both without scanning the node
// table: whole-query prefix first, then same-initial.
if !terms.is_empty() if !terms.is_empty()
&& let Some(first) = query.chars().next() && let Some(first) = query.chars().next()
{ {
let prefix = format!("{}%", like_escape(&first.to_string()));
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT node_json, uid FROM nodes "SELECT node_json, uid, path FROM nodes
WHERE name COLLATE NOCASE LIKE ?1 ESCAPE '\\' WHERE name COLLATE NOCASE LIKE ?1 ESCAPE '\\'
AND trashed = 0 AND node_json IS NOT NULL AND trashed = 0 AND node_json IS NOT NULL
ORDER BY name COLLATE NOCASE LIMIT ?2", ORDER BY name COLLATE NOCASE LIMIT ?2",
)?; )?;
let fallback = let mut extra = Vec::new();
collect_pairs(stmt.query_map(params![prefix, lane_limit as i64], pair)?)?; for pattern in [
for row in fallback { format!("{}%", like_escape(query)),
if !rows.iter().any(|(_, uid)| uid == &row.1) { format!("{}%", like_escape(&first.to_string())),
] {
extra.extend(collect_hits(
stmt.query_map(params![pattern, lane_limit as i64], hit_row)?,
)?);
}
let trigram = std::mem::take(&mut rows);
for row in extra.into_iter().chain(trigram) {
if !rows.iter().any(|(_, uid, _)| uid == &row.1) {
rows.push(row); rows.push(row);
} }
} }
rows.truncate(limit); rows.truncate(limit);
} }
rows.into_iter() rows.into_iter()
.map(|(json, uid)| { .map(|(json, uid, path)| {
Ok(SearchHit { Ok(SearchHit {
node: serde_json::from_str(&json)?, node: serde_json::from_str(&json)?,
path: path_of(&conn, &uid)?, path: match path {
Some(path) => path,
None => walk_path_of(&conn, &uid)?,
},
}) })
}) })
.collect() .collect()
@@ -315,7 +509,7 @@ impl Db {
/// Load every persisted node for cold-start hydration of the `State` maps. /// Load every persisted node for cold-start hydration of the `State` maps.
pub fn load_all(&self) -> Result<Vec<StoredNode>> { pub fn load_all(&self) -> Result<Vec<StoredNode>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = let mut stmt =
conn.prepare("SELECT node_json, listed FROM nodes WHERE node_json IS NOT NULL")?; conn.prepare("SELECT node_json, listed FROM nodes WHERE node_json IS NOT NULL")?;
let rows = stmt.query_map([], |row| { let rows = stmt.query_map([], |row| {
@@ -336,7 +530,7 @@ impl Db {
/// when the API is unreachable, so the mount can still serve the cached tree /// when the API is unreachable, so the mount can still serve the cached tree
/// (offline.md Phase 1). /// (offline.md Phase 1).
pub fn node_by_uid(&self, uid: &str) -> Result<Option<Node>> { pub fn node_by_uid(&self, uid: &str) -> Result<Option<Node>> {
let conn = self.conn.lock(); let conn = self.read();
let json: Option<String> = conn let json: Option<String> = conn
.query_row( .query_row(
"SELECT node_json FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL", "SELECT node_json FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL",
@@ -354,7 +548,7 @@ impl Db {
/// The daemon resumes from this on restart instead of reseeding to the /// The daemon resumes from this on restart instead of reseeding to the
/// server head, so changes made while unmounted are still applied (P2). /// server head, so changes made while unmounted are still applied (P2).
pub fn children_if_listed(&self, parent: &NodeUid) -> Result<Option<Vec<Node>>> { pub fn children_if_listed(&self, parent: &NodeUid) -> Result<Option<Vec<Node>>> {
let conn = self.conn.lock(); let conn = self.read();
let listed: Option<i64> = conn let listed: Option<i64> = conn
.query_row( .query_row(
"SELECT listed FROM nodes WHERE uid = ?1", "SELECT listed FROM nodes WHERE uid = ?1",
@@ -387,6 +581,288 @@ impl Db {
// in sync on every store/read/evict, so it is authoritative for eviction. // in sync on every store/read/evict, so it is authoritative for eviction.
} }
/// What a node's row looked like before this upsert, for the fields that decide
/// whether the subtree below it has to be touched at all.
struct PriorRow {
parent_uid: Option<String>,
name: String,
path: Option<String>,
trashed: bool,
}
fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
let json = serde_json::to_string(node)?;
let uid = node.uid.to_string();
let parent_uid = node.parent_uid.as_ref().map(|u| u.to_string());
let prior: Option<PriorRow> = tx
.query_row(
"SELECT parent_uid, name, path, trashed FROM nodes WHERE uid = ?1",
params![uid],
|row| {
Ok(PriorRow {
parent_uid: row.get(0)?,
name: row.get(1)?,
path: row.get(2)?,
trashed: row.get::<_, i64>(3)? != 0,
})
},
)
.optional()?;
// A node's path is its parent's path plus its own name — one indexed lookup,
// where resolving it from scratch is a recursive walk to the root. A node
// whose parent is not cached (a device folder's root lives in `device`, never
// in `nodes`) starts a path of its own, which is what the walk did too.
let path = match &parent_uid {
None => String::new(),
Some(parent) => {
let parent_path: Option<Option<String>> = tx
.query_row(
"SELECT path FROM nodes WHERE uid = ?1",
params![parent],
|row| row.get(0),
)
.optional()?;
join_path(parent_path.flatten().as_deref().unwrap_or(""), &node.name)
}
};
tx.execute(
"INSERT INTO nodes
(uid, parent_uid, name, is_dir, size, mtime, trashed, node_json, path)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)
ON CONFLICT(uid) DO UPDATE SET
parent_uid = excluded.parent_uid,
name = excluded.name,
is_dir = excluded.is_dir,
size = excluded.size,
mtime = excluded.mtime,
trashed = excluded.trashed,
node_json = excluded.node_json,
path = excluded.path",
params![
uid,
parent_uid,
node.name,
node.is_folder() as i64,
node_size(node),
node.modification_time,
node.trashed as i64,
json,
path,
],
)?;
let rowid: i64 = tx.query_row(
"SELECT rowid FROM nodes WHERE uid = ?1",
params![uid],
|row| row.get(0),
)?;
// FTS5 has no UPSERT, so the node's own row is always replaced.
let indexable = node_is_indexable_tx(tx, &uid)?;
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![rowid])?;
if indexable {
tx.execute(
"INSERT INTO nodes_fts (rowid, name, path) VALUES (?1, ?2, ?3)",
params![rowid, node.name, path],
)?;
}
// Everything below only matters when the subtree's paths or reachability
// actually moved. Re-listing a folder upserts it unchanged, and paying for a
// full descendant walk on each of those was most of what a large tree spent
// its time on.
let subtree_moved = prior.is_none_or(|prior| {
prior.parent_uid != parent_uid
|| prior.name != node.name
|| prior.path.as_deref() != Some(path.as_str())
|| prior.trashed != node.trashed
});
if node.is_folder() && subtree_moved {
reindex_subtree_tx(tx, &uid, &path, indexable)?;
}
Ok(())
}
/// Rewrite every descendant's stored path and search-index row after `folder`
/// moved, was renamed, or changed reachability.
///
/// One recursive walk carries both the new path and whether anything on the way
/// down is trashed, so each descendant costs two trivial statements instead of a
/// `path_of` walk plus an ancestor walk of its own. `folder_indexable` is the
/// answer `node_is_indexable_tx` gave for the folder itself: what is above the
/// subtree is the same for every node in it.
///
/// The depth cap is a liveness guard, not a policy: `UNION ALL` over a parent
/// cycle (a's parent is b, b's parent is a — corrupt data, but the API can hand
/// it to us) never terminates, and this runs inside the write transaction that
/// holds the daemon's only SQLite connection. Real Drive trees are nowhere near
/// this deep.
fn reindex_subtree_tx(
tx: &Transaction<'_>,
folder: &str,
folder_path: &str,
folder_indexable: bool,
) -> Result<()> {
let descendants: Vec<(i64, String, bool)> = {
let mut stmt = tx.prepare(
"WITH RECURSIVE sub(rowid, uid, name, path, blocked, depth) AS (
SELECT n.rowid, n.uid, n.name,
CASE WHEN ?2 = '' THEN n.name ELSE ?2 || '/' || n.name END,
n.trashed, 0
FROM nodes n WHERE n.parent_uid = ?1
UNION ALL
SELECT n.rowid, n.uid, n.name,
sub.path || '/' || n.name,
MAX(sub.blocked, n.trashed), sub.depth + 1
FROM nodes n JOIN sub ON n.parent_uid = sub.uid
WHERE sub.depth < 256
)
SELECT rowid, path, blocked FROM sub",
)?;
let rows = stmt.query_map(params![folder, folder_path], |row| {
Ok((
row.get::<_, i64>(0)?,
row.get::<_, String>(1)?,
row.get::<_, i64>(2)? != 0,
))
})?;
let mut out = Vec::new();
for row in rows {
out.push(row?);
}
out
};
if descendants.is_empty() {
return Ok(());
}
let mut set_path = tx.prepare("UPDATE nodes SET path = ?2 WHERE rowid = ?1")?;
let mut unindex = tx.prepare("DELETE FROM nodes_fts WHERE rowid = ?1")?;
let mut index = tx.prepare(
"INSERT INTO nodes_fts (rowid, name, path)
SELECT rowid, name, ?2 FROM nodes WHERE rowid = ?1",
)?;
for (rowid, path, blocked) in descendants {
set_path.execute(params![rowid, path])?;
unindex.execute(params![rowid])?;
if folder_indexable && !blocked {
index.execute(params![rowid, path])?;
}
}
Ok(())
}
fn direct_child_uids_tx(tx: &Transaction<'_>, parent: &str) -> Result<Vec<String>> {
let mut stmt = tx.prepare("SELECT uid FROM nodes WHERE parent_uid = ?1")?;
let rows = stmt.query_map([parent], |row| row.get::<_, String>(0))?;
let mut uids = Vec::new();
for row in rows {
uids.push(row?);
}
Ok(uids)
}
/// Whether a node belongs in the search index: it exists, and nothing on the
/// way up to the top of its cached chain is trashed.
///
/// The chain does *not* have to reach a `parent_uid IS NULL` row. Only the My
/// Files root is stored that way; a device folder's root lives in `device` /
/// `sync_folder` and never gets a `nodes` row, so requiring a null-parent
/// ancestor silently excluded every device-folder subtree from search — 29% of
/// the nodes on the account this was found on. A subtree whose topmost cached
/// row has an uncached parent is ordinary, reachable data.
///
/// What is still excluded is a *cycle*: the walk's own guard stops it, but
/// [`path_of`] has no such guard, so an indexed cycle would hang a search. A
/// clean chain ends at a row whose parent is null or simply not cached; a cycle
/// ends at one whose parent is a row we have already visited.
fn node_is_indexable_tx(tx: &Transaction<'_>, uid: &str) -> Result<bool> {
let indexable: i64 = tx.query_row(
"WITH RECURSIVE ancestors(uid, parent_uid, trashed, depth, path) AS (
SELECT uid, parent_uid, trashed, 0, char(31) || uid || char(31)
FROM nodes WHERE uid = ?1
UNION ALL
SELECT n.uid, n.parent_uid, n.trashed, a.depth + 1,
a.path || n.uid || char(31)
FROM ancestors a JOIN nodes n ON n.uid = a.parent_uid
WHERE instr(a.path, char(31) || n.uid || char(31)) = 0
)
SELECT CASE
WHEN COUNT(*) = 0 THEN 0
WHEN MAX(trashed) != 0 THEN 0
WHEN EXISTS (
SELECT 1 FROM ancestors a
WHERE a.depth = (SELECT MAX(depth) FROM ancestors)
AND a.parent_uid IS NOT NULL
AND EXISTS (SELECT 1 FROM nodes p WHERE p.uid = a.parent_uid)
) THEN 0
ELSE 1
END
FROM ancestors",
[uid],
|row| row.get(0),
)?;
Ok(indexable == 1)
}
fn tombstone_subtree_tx(tx: &Transaction<'_>, root_uid: &str) -> Result<()> {
let rows = {
let mut stmt = tx.prepare(
"WITH RECURSIVE subtree(rowid, uid, node_json) AS (
SELECT rowid, uid, node_json FROM nodes WHERE uid = ?1
UNION
SELECT n.rowid, n.uid, n.node_json FROM nodes n
JOIN subtree s ON n.parent_uid = s.uid
)
SELECT rowid, uid, node_json FROM subtree",
)?;
let mapped = stmt.query_map([root_uid], |row| {
Ok((
row.get::<_, i64>(0)?,
row.get::<_, String>(1)?,
row.get::<_, Option<String>>(2)?,
))
})?;
let mut rows = Vec::new();
for row in mapped {
rows.push(row?);
}
rows
};
for (rowid, uid, json) in rows {
let json = match json {
Some(json) => {
let mut node: Node = serde_json::from_str(&json)?;
node.trashed = true;
Some(serde_json::to_string(&node)?)
}
None => None,
};
tx.execute(
"UPDATE nodes SET trashed = 1, listed = 0, node_json = ?2 WHERE uid = ?1",
params![uid, json],
)?;
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", [rowid])?;
}
Ok(())
}
fn upsert_share_access_tx(tx: &Transaction<'_>, uid: &str, access: Access) -> Result<()> {
tx.execute(
"INSERT INTO share_access (root_uid, access) VALUES (?1, ?2)
ON CONFLICT(root_uid) DO UPDATE SET access = excluded.access",
params![uid, access.as_db_str()],
)?;
Ok(())
}
fn parse_node_uid(value: &str) -> Option<NodeUid> {
let (volume, link) = value.split_once('~')?;
Some(NodeUid::new(VolumeId::from(volume), LinkId::from(link)))
}
/// Unique quoted character trigrams suitable for an FTS5 OR expression. /// Unique quoted character trigrams suitable for an FTS5 OR expression.
pub(super) fn candidate_trigrams(query: &str) -> Vec<String> { pub(super) fn candidate_trigrams(query: &str) -> Vec<String> {
let chars: Vec<char> = query.to_lowercase().chars().collect(); let chars: Vec<char> = query.to_lowercase().chars().collect();
+515 -11
View File
@@ -3,6 +3,7 @@
//! order, so a child never drains before the parent that gives it a real uid. //! order, so a child never drains before the parent that gives it a real uid.
use rusqlite::{OptionalExtension, params}; use rusqlite::{OptionalExtension, params};
use serde::{Deserialize, Serialize};
use super::Db; use super::Db;
use crate::Result; use crate::Result;
@@ -35,6 +36,16 @@ pub const OP_MKDIR: &str = "mkdir";
/// that folder's own drain, exactly as for [`OP_CREATE`]. /// that folder's own drain, exactly as for [`OP_CREATE`].
pub const OP_RENAME: &str = "rename"; pub const OP_RENAME: &str = "rename";
/// Drain-time authority retained by a queued rename.
///
/// The optimistic local move rewrites the node's persisted ancestry to the
/// destination immediately. Keeping the original parent here prevents that
/// rewrite from erasing the shared tree that admitted the move.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct RenameMeta {
pub original_parent_uid: String,
}
/// The `kind` of a [`PendingOp`] that trashes a node the server knows about /// The `kind` of a [`PendingOp`] that trashes a node the server knows about
/// (offline.md Phase 3b). /// (offline.md Phase 3b).
/// ///
@@ -63,6 +74,35 @@ pub fn op_supersedes(kind: &str) -> bool {
/// drain replaces it with the uid the server assigns. /// drain replaces it with the uid the server assigns.
pub const LOCAL_VOLUME: &str = "local"; pub const LOCAL_VOLUME: &str = "local";
/// A `next_attempt_at` far enough in the future that [`Db::next_due_op`] never
/// selects the op (ms since epoch, ≈ year 2223). Used to *park* a queued create
/// for a transient file — a browser's in-flight `*.crdownload`/`*.part`, an
/// editor's `*.swp` — so its bytes never upload while it wears that name. A
/// rename to the finished name un-parks it (see [`Db::set_create_hold`]), which
/// is the only moment the completed file is meant to reach Drive. Without the
/// park, every partial and every abandoned temp would upload and, on a
/// stall+resume, fork a `(sync-conflict)` copy (docs/BUGS.md B70).
pub const PARK_UNTIL: i64 = 8_000_000_000_000;
/// How long a create may stay parked before the park is treated as permanent
/// and the bytes are let through anyway (ms).
///
/// The park has exactly one exit: a rename from the transient name to the
/// finished one. Every writer that reaches that rename does so in seconds — an
/// atomic-write scratch file immediately, a download when it completes. A park
/// still standing an hour later belongs to a writer that will never perform
/// that rename: an editor that deletes its swap file instead, a download the
/// user abandoned, or a file that simply *is* named `report.tmp` and is never
/// going to be called anything else.
///
/// Before this existed such a row sat at [`PARK_UNTIL`] for the life of the
/// database, holding the only copy of the file's bytes in `staging/` and
/// counting against the queue while promising progress that could not come. The
/// sweep un-parks it instead of dropping it: bytes the user can see in the mount
/// are bytes the user expects on Drive, and invariant 1 of the drain (a staged
/// blob is dropped only once its op has landed) admits no other answer.
pub const PARK_EXPIRY_MS: i64 = 60 * 60 * 1000;
/// A mutation that has been accepted locally but not yet performed against the /// A mutation that has been accepted locally but not yet performed against the
/// API — the durable half of the write-back queue (offline.md Phase 3). /// API — the durable half of the write-back queue (offline.md Phase 3).
/// ///
@@ -98,15 +138,32 @@ pub struct PendingOp {
} }
/// How much work the queue owes the server, by whether it carries bytes. /// How much work the queue owes the server, by whether it carries bytes.
#[derive(Debug, Clone, Copy, Default)] #[derive(Debug, Clone, Default)]
pub struct PendingCounts { pub struct PendingCounts {
/// Queued `create`/`revision` ops: files whose content is not on the remote. /// Queued `create`/`revision` ops: files whose content is not on the remote.
pub uploads: i64, pub uploads: i64,
/// Queued `mkdir`/`rename`/`trash` ops: metadata the remote has not been /// Queued `mkdir`/`rename`/`trash` ops: metadata the remote has not been
/// told about. /// told about.
pub changes: i64, pub changes: i64,
/// Ops parked at [`PARK_UNTIL`]: a transient file's bytes, held back until
/// the rename that gives it its finished name. They are queued but will
/// never drain on their own, so counting them as "waiting to upload" would
/// have the front-end promise progress that is not coming.
pub parked: i64,
/// Ops that have failed at least [`FAILING_ATTEMPTS`] times. A permanently
/// failing op never wedges the queue — the backoff sees to that — which is
/// exactly why it needs surfacing: it retries forever, invisibly.
pub failing: i64,
/// The most recent error text from a failing op, for the front-end to show
/// instead of the bare count.
pub last_error: Option<String>,
} }
/// Failed attempts after which an op is reported as failing rather than merely
/// retrying. Six is where the backoff has reached its ceiling, so an op past it
/// is retrying at the slowest rate it ever will and has been for minutes.
pub const FAILING_ATTEMPTS: i64 = 6;
/// The outcome of folding freshly written bytes into a queued create. /// The outcome of folding freshly written bytes into a queued create.
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct AttachedBlob { pub struct AttachedBlob {
@@ -117,6 +174,32 @@ pub struct AttachedBlob {
} }
impl Db { impl Db {
/// Whether a specific desired-state operation is still queued for a node.
pub fn has_pending_op(&self, uid: &str, kind: &str) -> Result<bool> {
let conn = self.read();
conn.query_row(
"SELECT EXISTS(
SELECT 1 FROM pending_op WHERE uid = ?1 AND kind = ?2
)",
params![uid, kind],
|row| row.get(0),
)
.map_err(Into::into)
}
/// Read an existing operation's metadata before a superseding enqueue.
pub fn pending_op_meta(&self, uid: &str, kind: &str) -> Result<Option<String>> {
let conn = self.read();
conn.query_row(
"SELECT meta_json FROM pending_op WHERE uid = ?1 AND kind = ?2",
params![uid, kind],
|row| row.get(0),
)
.optional()
.map(|value| value.flatten())
.map_err(Into::into)
}
pub fn enqueue_op(&self, op: &PendingOp) -> Result<(i64, Option<String>)> { pub fn enqueue_op(&self, op: &PendingOp) -> Result<(i64, Option<String>)> {
let mut conn = self.conn.lock(); let mut conn = self.conn.lock();
let tx = conn.transaction()?; let tx = conn.transaction()?;
@@ -157,6 +240,50 @@ impl Db {
Ok((id, superseded)) Ok((id, superseded))
} }
/// Atomically replace every queued operation for `uid` and its queued
/// descendants with one remote trash intent.
///
/// Staged blob paths are returned only after the transaction commits. A
/// failed trash insert therefore rolls the deletions back and leaves every
/// prior operation owning its bytes.
pub fn replace_ops_with_trash(
&self,
uid: &str,
name: &str,
created_at: i64,
) -> Result<(i64, Vec<String>)> {
const SUBTREE: &str = "
WITH RECURSIVE doomed(uid) AS (
SELECT ?1
UNION
SELECT p.uid FROM pending_op p JOIN doomed d ON p.parent_uid = d.uid
)";
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let blobs: Vec<String> = {
let mut stmt = tx.prepare(&format!(
"{SUBTREE}
SELECT blob_path FROM pending_op
WHERE uid IN (SELECT uid FROM doomed) AND blob_path IS NOT NULL"
))?;
let rows = stmt.query_map(params![uid], |row| row.get::<_, String>(0))?;
rows.collect::<std::result::Result<Vec<_>, _>>()?
};
tx.execute(
&format!("{SUBTREE} DELETE FROM pending_op WHERE uid IN (SELECT uid FROM doomed)"),
params![uid],
)?;
tx.execute(
"INSERT INTO pending_op
(kind, uid, parent_uid, name, blob_path, meta_json, created_at, next_attempt_at)
VALUES (?1, ?2, NULL, ?3, NULL, NULL, ?4, 0)",
params![OP_TRASH, uid, name, created_at],
)?;
let id = tx.last_insert_rowid();
tx.commit()?;
Ok((id, blobs))
}
/// Point a queued create at the bytes that were just written to it, returning /// Point a queued create at the bytes that were just written to it, returning
/// any blob it previously held so the caller can discard it. /// any blob it previously held so the caller can discard it.
/// ///
@@ -183,11 +310,18 @@ impl Db {
let Some((id, superseded)) = existing else { let Some((id, superseded)) = existing else {
return Ok(None); return Ok(None);
}; };
// Clearing the backoff lets the fresh bytes be tried promptly — except
// when the create is *parked* (a transient file, next_attempt_at at the
// PARK_UNTIL sentinel). Attaching a new revision to a still-growing
// download must not wake it; only the finalize rename does. So preserve a
// park and reset an ordinary backoff.
conn.execute( conn.execute(
"UPDATE pending_op "UPDATE pending_op
SET blob_path = ?2, meta_json = ?3, attempts = 0, next_attempt_at = 0 SET blob_path = ?2, meta_json = ?3, attempts = 0,
next_attempt_at = CASE WHEN next_attempt_at >= ?4
THEN next_attempt_at ELSE 0 END
WHERE id = ?1", WHERE id = ?1",
params![id, blob_path, meta_json], params![id, blob_path, meta_json, PARK_UNTIL],
)?; )?;
Ok(Some(AttachedBlob { id, superseded })) Ok(Some(AttachedBlob { id, superseded }))
} }
@@ -233,9 +367,28 @@ impl Db {
Ok(n > 0) Ok(n > 0)
} }
/// Park or un-park a queued create/mkdir for `uid`, returning whether a row
/// was touched.
///
/// Parking sets `next_attempt_at` to [`PARK_UNTIL`] so [`Db::next_due_op`]
/// skips it indefinitely; un-parking sets it due now. Used to keep a
/// transient file's bytes off Drive until it is renamed to its finished name
/// (docs/BUGS.md B70). Un-parking a create that was never parked is harmless:
/// it just makes the create due immediately, which for a normal local node it
/// already is.
pub fn set_create_hold(&self, uid: &str, held: bool) -> Result<bool> {
let conn = self.conn.lock();
let n = conn.execute(
"UPDATE pending_op SET next_attempt_at = ?2
WHERE uid = ?1 AND kind IN (?3, ?4)",
params![uid, if held { PARK_UNTIL } else { 0 }, OP_CREATE, OP_MKDIR],
)?;
Ok(n > 0)
}
/// Check if a queued create or mkdir op exists for `uid`. /// Check if a queued create or mkdir op exists for `uid`.
pub fn has_create_op(&self, uid: &str) -> Result<bool> { pub fn has_create_op(&self, uid: &str) -> Result<bool> {
let conn = self.conn.lock(); let conn = self.read();
let count: i64 = conn.query_row( let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM pending_op WHERE uid = ?1 AND kind IN (?2, ?3)", "SELECT COUNT(*) FROM pending_op WHERE uid = ?1 AND kind IN (?2, ?3)",
params![uid, OP_CREATE, OP_MKDIR], params![uid, OP_CREATE, OP_MKDIR],
@@ -244,6 +397,22 @@ impl Db {
Ok(count > 0) Ok(count > 0)
} }
/// Check whether *any* op of any kind is queued against `uid`.
///
/// Used as a safety interlock by callers that are about to remove a node
/// they only know about from a stale snapshot: a queued op means the user's
/// bytes are still owed an upload, and dropping it would discard the staged
/// blob that holds them (`docs/BUGS.md` B71).
pub fn has_any_op(&self, uid: &str) -> Result<bool> {
let conn = self.read();
let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM pending_op WHERE uid = ?1",
params![uid],
|row| row.get(0),
)?;
Ok(count > 0)
}
/// Drop every op targeting a node **or anything queued beneath it**, /// Drop every op targeting a node **or anything queued beneath it**,
/// returning the staged blobs they held so the caller can delete them. /// returning the staged blobs they held so the caller can delete them.
/// ///
@@ -331,7 +500,7 @@ impl Db {
/// sort, and no separate index to maintain. The common case (something is /// sort, and no separate index to maintain. The common case (something is
/// due) exits on the first row. /// due) exits on the first row.
pub fn next_due_op(&self, now: i64) -> Result<Option<PendingOp>> { pub fn next_due_op(&self, now: i64) -> Result<Option<PendingOp>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare_cached(&format!( let mut stmt = conn.prepare_cached(&format!(
"SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at, "SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
attempts, last_error, next_attempt_at attempts, last_error, next_attempt_at
@@ -362,17 +531,115 @@ impl Db {
Ok(op) Ok(op)
} }
/// Take the oldest due op for this worker, marking it claimed so no other
/// drain worker takes it, or `None` when there is nothing to do.
///
/// The drain is several workers over one queue, which is only safe because
/// the claim is a column: a row is offered to exactly one worker, and the
/// worker that took it is the only one that may retire, defer or fail it.
///
/// Two exclusions on top of [`next_due_op`](Self::next_due_op)'s:
///
/// * `claimed_at = 0` — not already in flight somewhere else.
/// * no *other* claimed op shares this `uid`. Ordering only has to hold per
/// node, but there it has to hold absolutely: a queued rename and a queued
/// revision of one file must land in the order they were made, and two
/// workers on one node would also race over its staged blob and its
/// pending entry. `ORDER BY id` plus this exclusion gives per-uid serial
/// order for free — the older op is claimed first and the newer one is
/// invisible until it retires.
///
/// Select and mark are one transaction, so two workers cannot both see the
/// row as unclaimed.
pub fn claim_next_due_op(&self, now: i64) -> Result<Option<PendingOp>> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let op = {
let mut stmt = tx.prepare_cached(&format!(
"SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
attempts, last_error, next_attempt_at
FROM pending_op
WHERE next_attempt_at <= ?1
AND claimed_at = 0
AND (parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~')
AND uid NOT IN (SELECT uid FROM pending_op WHERE claimed_at <> 0)
ORDER BY id LIMIT 1",
v = LOCAL_VOLUME,
n = LOCAL_VOLUME.len() + 1,
))?;
stmt.query_row(params![now], |r| {
Ok(PendingOp {
id: r.get(0)?,
kind: r.get(1)?,
uid: r.get(2)?,
parent_uid: r.get(3)?,
name: r.get(4)?,
blob_path: r.get(5)?,
meta_json: r.get(6)?,
created_at: r.get(7)?,
attempts: r.get(8)?,
last_error: r.get(9)?,
next_attempt_at: r.get(10)?,
})
})
.optional()?
};
if let Some(op) = op.as_ref() {
tx.execute(
"UPDATE pending_op SET claimed_at = ?2 WHERE id = ?1",
params![op.id, now.max(1)],
)?;
}
tx.commit()?;
Ok(op)
}
/// Give a claimed op back to the queue, whatever became of the attempt.
///
/// Called on every path out of a drain attempt rather than only the failing
/// ones: a handler that retires its own row leaves nothing to release (the
/// `UPDATE` matches nothing), and a handler that returns without retiring
/// *and* without failing would otherwise leave the row claimed by a worker
/// that has moved on — invisible to every worker, forever.
pub fn release_op_claim(&self, id: i64) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"UPDATE pending_op SET claimed_at = 0 WHERE id = ?1",
params![id],
)?;
Ok(())
}
/// Drop every claim in the table.
///
/// A claim describes a worker in *this* process, so one surviving on disk
/// means the previous run died holding it. The single-writer lock
/// ([`Db::open`]) is what makes that inference safe: nobody else can be
/// draining this queue while we are opening it.
pub fn clear_op_claims(&self) -> Result<usize> {
let conn = self.conn.lock();
Ok(conn.execute(
"UPDATE pending_op SET claimed_at = 0 WHERE claimed_at <> 0",
[],
)?)
}
/// The earliest `next_attempt_at` among all queued ops, or `None` if the /// The earliest `next_attempt_at` among all queued ops, or `None` if the
/// queue is empty. Used by the drain loop to sleep exactly until the next /// queue is empty. Used by the drain loop to sleep exactly until the next
/// debounced or backed-off op becomes eligible rather than waiting the full /// debounced or backed-off op becomes eligible rather than waiting the full
/// idle-poll interval. /// idle-poll interval.
///
/// Claimed ops are excluded: another worker is already on them, so they are
/// not work this one is waiting for, and counting them would have an idle
/// worker spin on a row it cannot have.
pub fn earliest_due_at(&self) -> Result<Option<i64>> { pub fn earliest_due_at(&self) -> Result<Option<i64>> {
let conn = self.conn.lock(); let conn = self.read();
let ts: Option<i64> = conn let ts: Option<i64> = conn
.query_row( .query_row(
&format!( &format!(
"SELECT MIN(next_attempt_at) FROM pending_op \ "SELECT MIN(next_attempt_at) FROM pending_op \
WHERE parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~'", WHERE claimed_at = 0 \
AND (parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~')",
v = LOCAL_VOLUME, v = LOCAL_VOLUME,
n = LOCAL_VOLUME.len() + 1, n = LOCAL_VOLUME.len() + 1,
), ),
@@ -387,7 +654,7 @@ impl Db {
/// Every queued op, oldest first. For status read-outs and the CLI — the /// Every queued op, oldest first. For status read-outs and the CLI — the
/// drain wants [`next_due_op`](Self::next_due_op) instead. /// drain wants [`next_due_op`](Self::next_due_op) instead.
pub fn pending_ops(&self) -> Result<Vec<PendingOp>> { pub fn pending_ops(&self) -> Result<Vec<PendingOp>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at, "SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
attempts, last_error, next_attempt_at attempts, last_error, next_attempt_at
@@ -413,6 +680,40 @@ impl Db {
Ok(rows) Ok(rows)
} }
/// Every parked create/mkdir, oldest first, for the park sweep.
///
/// Parked rows are invisible to [`next_due_op`](Self::next_due_op) by
/// construction, so nothing else in the drain ever looks at them. This is
/// the one query that does.
pub fn parked_create_ops(&self) -> Result<Vec<PendingOp>> {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
attempts, last_error, next_attempt_at
FROM pending_op
WHERE next_attempt_at >= ?1 AND kind IN (?2, ?3)
ORDER BY created_at",
)?;
let rows = stmt
.query_map(params![PARK_UNTIL, OP_CREATE, OP_MKDIR], |r| {
Ok(PendingOp {
id: r.get(0)?,
kind: r.get(1)?,
uid: r.get(2)?,
parent_uid: r.get(3)?,
name: r.get(4)?,
blob_path: r.get(5)?,
meta_json: r.get(6)?,
created_at: r.get(7)?,
attempts: r.get(8)?,
last_error: r.get(9)?,
next_attempt_at: r.get(10)?,
})
})?
.collect::<std::result::Result<Vec<_>, _>>()?;
Ok(rows)
}
/// How many queued ops there are of each kind, for `Response::Status`. /// How many queued ops there are of each kind, for `Response::Status`.
/// ///
/// Split because "3 uploads queued" has to mean three files whose bytes are /// Split because "3 uploads queued" has to mean three files whose bytes are
@@ -420,7 +721,7 @@ impl Db {
/// mount owes the server, but it carries no bytes and reporting it as an /// mount owes the server, but it carries no bytes and reporting it as an
/// upload is simply untrue. /// upload is simply untrue.
pub fn pending_op_counts(&self) -> Result<PendingCounts> { pub fn pending_op_counts(&self) -> Result<PendingCounts> {
let conn = self.conn.lock(); let conn = self.read();
let uploads = conn.query_row( let uploads = conn.query_row(
"SELECT COUNT(*) FROM pending_op WHERE kind IN (?1, ?2)", "SELECT COUNT(*) FROM pending_op WHERE kind IN (?1, ?2)",
params![OP_REVISION, OP_CREATE], params![OP_REVISION, OP_CREATE],
@@ -431,7 +732,33 @@ impl Db {
params![OP_REVISION, OP_CREATE], params![OP_REVISION, OP_CREATE],
|r| r.get(0), |r| r.get(0),
)?; )?;
Ok(PendingCounts { uploads, changes }) let parked = conn.query_row(
"SELECT COUNT(*) FROM pending_op WHERE next_attempt_at >= ?1",
params![PARK_UNTIL],
|r| r.get(0),
)?;
let failing = conn.query_row(
"SELECT COUNT(*) FROM pending_op WHERE attempts >= ?1",
params![FAILING_ATTEMPTS],
|r| r.get(0),
)?;
let last_error = conn
.query_row(
"SELECT last_error FROM pending_op
WHERE attempts >= ?1 AND last_error IS NOT NULL
ORDER BY attempts DESC LIMIT 1",
params![FAILING_ATTEMPTS],
|r| r.get::<_, Option<String>>(0),
)
.optional()?
.flatten();
Ok(PendingCounts {
uploads,
changes,
parked,
failing,
last_error,
})
} }
/// Nothing bearing this volume may be handed to the API — it would 404. The /// Nothing bearing this volume may be handed to the API — it would 404. The
@@ -445,16 +772,193 @@ impl Db {
Ok(()) Ok(())
} }
/// Make one backed-off op due now, for the user's "Retry now". Parked ops are
/// left alone: they wait for a rename, not for time, and un-parking one would
/// upload a file its writer has not finished. Returns whether a row changed.
pub fn retry_op_now(&self, id: i64, now: i64) -> Result<bool> {
let conn = self.conn.lock();
let changed = conn.execute(
"UPDATE pending_op SET next_attempt_at = ?2
WHERE id = ?1 AND next_attempt_at > ?2 AND next_attempt_at < ?3",
params![id, now, PARK_UNTIL],
)?;
Ok(changed > 0)
}
/// [`retry_op_now`](Self::retry_op_now) for every op that has failed at
/// least once. Returns how many rows changed.
pub fn retry_failed_ops_now(&self, now: i64) -> Result<usize> {
let conn = self.conn.lock();
let changed = conn.execute(
"UPDATE pending_op SET next_attempt_at = ?1
WHERE attempts > 0 AND next_attempt_at > ?1 AND next_attempt_at < ?2",
params![now, PARK_UNTIL],
)?;
Ok(changed)
}
/// Record a failed attempt and when to next try. Leaves the row in place — /// Record a failed attempt and when to next try. Leaves the row in place —
/// the staged bytes are still the only copy of the user's write. /// the staged bytes are still the only copy of the user's write.
///
/// Clears any access-deferral window: whatever the op was waiting for has
/// now been reported, so the next uncleared deferral starts its own window
/// rather than escalating again on the following recheck.
pub fn record_op_failure(&self, id: i64, error: &str, next_attempt_at: i64) -> Result<()> { pub fn record_op_failure(&self, id: i64, error: &str, next_attempt_at: i64) -> Result<()> {
let conn = self.conn.lock(); let conn = self.conn.lock();
conn.execute( conn.execute(
"UPDATE pending_op "UPDATE pending_op
SET attempts = attempts + 1, last_error = ?2, next_attempt_at = ?3 SET attempts = attempts + 1, last_error = ?2, next_attempt_at = ?3,
access_deferred_since = 0
WHERE id = ?1", WHERE id = ?1",
params![id, error, next_attempt_at], params![id, error, next_attempt_at],
)?; )?;
Ok(()) Ok(())
} }
/// Defer a due operation without recording a failed remote attempt.
///
/// Access checks happen before the drain calls Proton, so a locally denied
/// operation has not failed remotely and must not consume a retry. Preserve
/// the long-lived transient-create sentinel if this is ever called for one.
pub fn defer_op_without_attempt(&self, id: i64, next_attempt_at: i64) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"UPDATE pending_op
SET next_attempt_at = CASE
WHEN next_attempt_at >= ?3 THEN next_attempt_at
ELSE ?2
END
WHERE id = ?1",
params![id, next_attempt_at, PARK_UNTIL],
)?;
Ok(())
}
/// Defer an op the local access check refused, and report how long it has
/// been refused for.
///
/// Identical to [`defer_op_without_attempt`](Self::defer_op_without_attempt)
/// on the retry clock, plus the one thing that was missing: the first
/// deferral of a run stamps `access_deferred_since`, and every deferral
/// returns that stamp. A caller can therefore tell a deferral that is one
/// recheck old from one that has been repeating unchanged for days, which is
/// the difference between a permission the user is mid-way through changing
/// and a queue that will never drain.
///
/// Returns the ms at which the current run of deferrals began — `now` on the
/// first one.
pub fn defer_op_for_access(&self, id: i64, now: i64, next_attempt_at: i64) -> Result<i64> {
let conn = self.conn.lock();
conn.execute(
"UPDATE pending_op
SET next_attempt_at = CASE
WHEN next_attempt_at >= ?4 THEN next_attempt_at
ELSE ?3
END,
access_deferred_since = CASE
WHEN access_deferred_since = 0 THEN ?2
ELSE access_deferred_since
END
WHERE id = ?1",
params![id, now, next_attempt_at, PARK_UNTIL],
)?;
let since = conn
.query_row(
"SELECT access_deferred_since FROM pending_op WHERE id = ?1",
params![id],
|r| r.get::<_, i64>(0),
)
.optional()?;
// A row that drained out from under this call leaves nothing to measure;
// reporting `now` says "this run just started", which is the reading that
// never escalates something that no longer exists.
Ok(since.filter(|since| *since != 0).unwrap_or(now))
}
/// Forget an op's access-deferral window, after an attempt the access check
/// let through. The op is no longer blocked, so a later deferral is the
/// start of a new run rather than the continuation of an old one.
pub fn clear_op_access_deferral(&self, id: i64) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"UPDATE pending_op SET access_deferred_since = 0
WHERE id = ?1 AND access_deferred_since <> 0",
params![id],
)?;
Ok(())
}
/// Every staged blob the queue still refers to.
///
/// The startup staging reconcile subtracts this from what is actually in
/// `staging/`: the difference is bytes the kernel was told were written and
/// that nothing is going to upload.
pub fn op_blob_paths(&self) -> Result<std::collections::HashSet<String>> {
let conn = self.read();
let mut stmt =
conn.prepare("SELECT blob_path FROM pending_op WHERE blob_path IS NOT NULL")?;
let rows = stmt.query_map([], |r| r.get::<_, String>(0))?;
let mut out = std::collections::HashSet::new();
for row in rows {
out.insert(row?);
}
Ok(out)
}
/// Record that this daemon sealed `revision_id` on `uid`, and prune entries
/// past [`OWN_SEALED_TTL_MS`].
///
/// One row per node, overwritten as the node advances. See
/// [`own_sealed_rev`](Self::own_sealed_rev) for what reads it.
pub fn set_own_sealed_rev(&self, uid: &str, revision_id: &str, now_ms: i64) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"INSERT INTO own_sealed_rev (uid, revision_id, sealed_at)
VALUES (?1, ?2, ?3)
ON CONFLICT(uid) DO UPDATE SET
revision_id = excluded.revision_id,
sealed_at = excluded.sealed_at",
params![uid, revision_id, now_ms],
)?;
conn.execute(
"DELETE FROM own_sealed_rev WHERE sealed_at < ?1",
params![now_ms - OWN_SEALED_TTL_MS],
)?;
Ok(())
}
/// The last revision of `uid` this daemon sealed itself, if it is still
/// within [`OWN_SEALED_TTL_MS`].
///
/// The drain asks before treating a moved-on remote as a conflict: a
/// revision we sealed means one writer stalled and resumed, which
/// supersedes rather than forking a `(sync-conflict)` copy (docs/BUGS.md
/// B70 layer B).
pub fn own_sealed_rev(&self, uid: &str, now_ms: i64) -> Result<Option<String>> {
let conn = self.read();
Ok(conn
.query_row(
"SELECT revision_id FROM own_sealed_rev
WHERE uid = ?1 AND sealed_at >= ?2",
params![uid, now_ms - OWN_SEALED_TTL_MS],
|r| r.get(0),
)
.optional()?)
}
/// Forget what we sealed on `uid` — it has been trashed, so there is no
/// revision left to chain onto.
pub fn clear_own_sealed_rev(&self, uid: &str) -> Result<()> {
let conn = self.conn.lock();
conn.execute("DELETE FROM own_sealed_rev WHERE uid = ?1", params![uid])?;
Ok(())
}
} }
/// How long a revision this daemon sealed stays recognisable as its own.
///
/// Bounds a table that would otherwise grow with every write for the life of
/// the install. A week is far longer than any stall→resume — a browser download
/// that pauses and continues, an editor that reopens its handle — and short
/// enough that the table tracks recent work rather than accumulating history.
pub const OWN_SEALED_TTL_MS: i64 = 7 * 24 * 60 * 60 * 1000;
+779 -71
View File
@@ -3,7 +3,7 @@
use rusqlite::params; use rusqlite::params;
use std::collections::HashMap; use std::collections::{HashMap, HashSet};
use super::Db; use super::Db;
use crate::Result; use crate::Result;
@@ -32,56 +32,382 @@ pub struct StoredPhoto {
/// Which Photos-page tab this entry belongs to, derived from its name and /// Which Photos-page tab this entry belongs to, derived from its name and
/// media type when the timeline was last replaced. /// media type when the timeline was last replaced.
pub kind: crate::control::PhotoKind, pub kind: crate::control::PhotoKind,
/// Whether the photo carries Proton's `Favorite` tag.
pub favorite: bool,
/// How many photos this photo's group holds — one shot taken as RAW+JPEG is
/// two files and one group. `1` for a photo that stands on its own.
pub group_size: usize,
/// Whether any member of the group is a camera raw file. The grid badges
/// that, because the JPEG it shows is not the whole of what is stored.
pub has_raw: bool,
}
/// A town photos were taken in: its GeoNames id (see [`crate::places`]), how
/// many photos it holds, and the newest of them.
#[derive(Clone, Debug, PartialEq)]
pub struct StoredPlace {
pub id: u32,
pub count: usize,
pub cover: StoredPhoto,
}
/// One row of a timeline replacement. Everything but the uid and the capture
/// time is "what this refresh learned": a `None` keeps whatever is already
/// stored rather than clearing it, because the timeline DTO carries only the uid
/// and the capture time and resolving the node can fail.
#[derive(Clone, Debug, Default, PartialEq)]
pub struct TimelineRow {
pub uid: String,
pub capture_time: i64,
pub name: Option<String>,
pub media_type: Option<String>,
pub favorite: Option<bool>,
/// The server's duplicate-detection hash for the plaintext.
pub content_hash: Option<String>,
/// The main photo this one is *related* to (a live-photo video, a burst
/// sibling), when the server says it belongs to one.
pub main_uid: Option<String>,
/// Where the photo was taken, as `(latitude, longitude)` in decimal degrees.
pub location: Option<(f64, f64)>,
/// When this refresh resolved the photo's node, if it did at all. `Some`
/// makes every other field authoritative — including a `None` that means
/// "the server no longer reports one" — while `None` means the refresh
/// skipped this photo and whatever is stored must be kept.
pub resolved_at: Option<i64>,
}
impl TimelineRow {
/// A row carrying only what the timeline listing itself gives.
pub fn new(uid: impl Into<String>, capture_time: i64) -> Self {
Self {
uid: uid.into(),
capture_time,
..Self::default()
}
}
}
/// What a previous refresh learned about a photo and a new one must not lose.
#[derive(Clone, Debug, Default)]
struct Learned {
name: Option<String>,
ratio: Option<f64>,
thumb_state: i64,
media_type: Option<String>,
favorite: bool,
content_hash: Option<String>,
main_uid: Option<String>,
location: Option<(f64, f64)>,
/// What its thumbnail looks like (see [`crate::similar`]).
similar_hash: Option<i64>,
/// When this photo's node was last resolved, kept so a refresh that skips it
/// does not make it look unresolved again.
resolved_at: Option<i64>,
}
/// The local calendar day a capture time falls on, as a `YYYY-MM-DD` string.
/// Grouping by day rather than by exact second is deliberate: a camera writes
/// the RAW and the JPEG of one shot a moment apart, and two shots of the same
/// subject on different days are not one photo.
fn capture_day(conn: &rusqlite::Connection, capture_time: i64) -> Option<String> {
conn.query_row(
"SELECT strftime('%Y-%m-%d', ?1, 'unixepoch', 'localtime')",
params![capture_time],
|r| r.get::<_, Option<String>>(0),
)
.ok()
.flatten()
}
/// The file name without its extension, lowercased — `IMG_1234.CR2` and
/// `IMG_1234.JPG` share `img_1234` — with the camera's own part-of-a-shot
/// marker removed, so the two files a raw capture writes share a stem too.
fn name_stem(name: Option<&str>) -> Option<String> {
let name = name?;
let stem = match name.rsplit_once('.') {
Some((stem, _)) if !stem.is_empty() => stem,
_ => name,
};
let stem = stem.to_ascii_lowercase();
Some(strip_shot_marker(&stem).to_string())
}
/// Drop a `.RAW-NN.ROLE` marker from a stem.
///
/// Pixel stores one raw capture as `PXL_20260919_000625670.RAW-01.COVER.jpg`
/// and `PXL_20260919_000625670.RAW-02.ORIGINAL.dng`: one shot, two files, two
/// stems that differ in exactly the part naming which file of the shot this is.
/// Cutting the marker off is what lets rule 2 see the pair. The check is narrow
/// on purpose — digits, then a single role word — so an ordinary name that
/// happens to contain `.raw-` keeps its stem.
fn strip_shot_marker(stem: &str) -> &str {
let Some(at) = stem.rfind(".raw-") else {
return stem;
};
let rest = &stem[at + ".raw-".len()..];
let Some((index, role)) = rest.split_once('.') else {
return stem;
};
let marked = !index.is_empty()
&& index.bytes().all(|b| b.is_ascii_digit())
&& !role.is_empty()
&& role.bytes().all(|b| b.is_ascii_alphabetic());
if marked { &stem[..at] } else { stem }
}
/// One entry of the grouping pass: what rule 2 needs, plus what choosing a
/// representative needs.
struct Grouped {
uid: String,
kind: crate::control::PhotoKind,
main_uid: Option<String>,
day: Option<String>,
stem: Option<String>,
}
/// Which group each photo belongs to, and which member of it the gallery shows.
///
/// Precedence, as agreed:
///
/// 1. The server's own relation — a photo with a `main_photo_uid` joins that
/// photo's group. The server knows about live photos and bursts; we do not
/// have to guess at them.
/// 2. Otherwise, same capture day **and** same file-name stem **and** one of the
/// two is a camera raw while the other is not. That is the RAW+JPEG pair a
/// camera writes, and the differing-kind clause keeps two JPEGs that happen
/// to share a name apart.
/// 3. Otherwise the photo is its own group.
///
/// The representative is the non-raw member when the group has one — a JPEG
/// decodes in milliseconds and is what the person expects to see — and the
/// earliest member in server order otherwise.
fn group_photos(entries: &[Grouped]) -> Vec<String> {
let mut parent: Vec<usize> = (0..entries.len()).collect();
fn find(parent: &mut [usize], mut i: usize) -> usize {
while parent[i] != i {
parent[i] = parent[parent[i]];
i = parent[i];
}
i
}
let union = |parent: &mut [usize], a: usize, b: usize| {
let (a, b) = (find(parent, a), find(parent, b));
if a != b {
parent[b] = a;
}
};
let index: HashMap<&str, usize> = entries
.iter()
.enumerate()
.map(|(i, e)| (e.uid.as_str(), i))
.collect();
for (i, entry) in entries.iter().enumerate() {
if let Some(main) = entry.main_uid.as_deref()
&& let Some(&j) = index.get(main)
{
union(&mut parent, j, i);
}
}
// Rule 2 needs only one pass: everything sharing a day and a stem lands in
// the same bucket, and the raw / non-raw split is checked pairwise there.
let mut by_name: HashMap<(&str, &str), Vec<usize>> = HashMap::new();
for (i, entry) in entries.iter().enumerate() {
if let (Some(day), Some(stem)) = (entry.day.as_deref(), entry.stem.as_deref()) {
by_name.entry((day, stem)).or_default().push(i);
}
}
let is_raw = |i: usize| entries[i].kind == crate::control::PhotoKind::Raw;
for bucket in by_name.values() {
for (n, &i) in bucket.iter().enumerate() {
for &j in &bucket[n + 1..] {
if is_raw(i) != is_raw(j) {
union(&mut parent, i, j);
}
}
}
}
// Representative per root, then the key every member carries.
let mut representative: HashMap<usize, usize> = HashMap::new();
for i in 0..entries.len() {
let root = find(&mut parent, i);
match representative.get(&root).copied() {
Some(current) if !is_raw(current) || is_raw(i) => {}
_ => {
representative.insert(root, i);
}
}
}
(0..entries.len())
.map(|i| {
let root = find(&mut parent, i);
entries[representative[&root]].uid.clone()
})
.collect()
}
/// The columns every photo read selects, with the group aggregates joined in.
/// `g.n` is how many photos the group holds and `g.raw` whether any of them is a
/// camera raw file — one query rather than a second lookup per tile.
///
/// `COALESCE(group_key, uid)` is what makes a row written before schema v30 its
/// own group: the column is `NULL` until the next refresh computes it, and a
/// `NULL` would otherwise match nothing and drop the photo off the page.
const PHOTO_SELECT: &str = "SELECT p.uid, p.capture_time, p.name, p.ratio, p.thumb_state, \
p.kind, p.favorite, g.n, g.raw \
FROM photos p JOIN (SELECT COALESCE(group_key, uid) AS k, COUNT(*) AS n, \
MAX(kind = 2) AS raw FROM photos GROUP BY k) g ON g.k = COALESCE(p.group_key, p.uid)";
/// Read one row of [`PHOTO_SELECT`].
fn stored_photo(r: &rusqlite::Row<'_>) -> rusqlite::Result<StoredPhoto> {
Ok(StoredPhoto {
uid: r.get(0)?,
capture_time: r.get(1)?,
name: r.get(2)?,
ratio: r.get(3)?,
thumb_state: r.get(4)?,
kind: crate::control::PhotoKind::from_i64(r.get(5)?),
favorite: r.get::<_, i64>(6)? != 0,
group_size: r.get::<_, i64>(7)?.max(1) as usize,
has_raw: r.get::<_, i64>(8)? != 0,
})
} }
impl Db { impl Db {
pub fn photos_replace( /// Replace the timeline wholesale. A row whose `resolved_at` is `Some` was
&self, /// read from the server in this pass, so its fields are taken as they are —
items: &[(String, i64, Option<String>, Option<String>)], /// a `None` there means the server no longer reports that value. A row whose
) -> Result<()> { /// `resolved_at` is `None` was skipped or failed to resolve, and keeps
/// whatever is already stored rather than having it silently cleared.
pub fn photos_replace(&self, items: &[TimelineRow]) -> Result<()> {
let mut conn = self.conn.lock(); let mut conn = self.conn.lock();
let tx = conn.transaction()?; let tx = conn.transaction()?;
// `media_type` is learned-and-kept like the ratio and thumb verdict: the // `name` and `media_type` are learned-and-kept like the ratio and thumb
// timeline DTO carries only the uid and capture time, so the daemon may // verdict: the timeline DTO carries only the uid and capture time, so the
// not know a photo's media type yet when it replaces the timeline. Keep // daemon may not know them when it replaces the timeline — because the
// any previously learned value so the Photos/Videos/Raw split survives a // refresh skipped an already-resolved photo, or because resolving it
// refresh instead of collapsing back to name-extension guesses. // failed. Keep any previously learned value so the tab split and the
let learned: HashMap<String, (Option<f64>, i64, Option<String>)> = { // RAW+JPEG grouping survive a refresh instead of collapsing to nothing:
let mut stmt = tx.prepare("SELECT uid, ratio, thumb_state, media_type FROM photos")?; // both are derived from the name. The photo relation is kept for the same
// reason — losing it would break a group up until the next resolve.
let learned: HashMap<String, Learned> = {
let mut stmt = tx.prepare(
"SELECT uid, name, ratio, thumb_state, media_type, favorite, content_hash, \
main_uid, resolved_at, latitude, longitude, similar_hash FROM photos",
)?;
let rows = stmt.query_map([], |r| { let rows = stmt.query_map([], |r| {
Ok((r.get::<_, String>(0)?, (r.get(1)?, r.get(2)?, r.get(3)?))) Ok((
r.get::<_, String>(0)?,
Learned {
name: r.get(1)?,
ratio: r.get(2)?,
thumb_state: r.get(3)?,
media_type: r.get(4)?,
favorite: r.get::<_, i64>(5)? != 0,
content_hash: r.get(6)?,
main_uid: r.get(7)?,
resolved_at: r.get(8)?,
location: r
.get::<_, Option<f64>>(9)?
.zip(r.get::<_, Option<f64>>(10)?),
similar_hash: r.get(11)?,
},
))
})?; })?;
rows.collect::<rusqlite::Result<_>>()? rows.collect::<rusqlite::Result<_>>()?
}; };
// Everything the insert and the grouping need, resolved once per photo.
let resolved: Vec<(&TimelineRow, Learned, crate::control::PhotoKind)> = items
.iter()
.map(|row| {
let mut carried = learned.get(&row.uid).cloned().unwrap_or(Learned {
thumb_state: THUMB_UNKNOWN,
..Learned::default()
});
if row.resolved_at.is_some() {
// This pass read the node itself, so what it says is the
// truth — including what it no longer says. A photo unlinked
// from its main on another device loses its `main_uid` here,
// which a carry could never express.
carried.name = row.name.clone();
carried.media_type = row.media_type.clone();
carried.favorite = row.favorite.unwrap_or(false);
// New bytes may show another picture: hash it again.
if row.content_hash != carried.content_hash {
carried.similar_hash = None;
}
carried.content_hash = row.content_hash.clone();
carried.main_uid = row.main_uid.clone();
carried.location = row.location;
} else {
carried.name = row.name.clone().or(carried.name);
carried.media_type = row.media_type.clone().or(carried.media_type);
carried.favorite = row.favorite.unwrap_or(carried.favorite);
carried.content_hash = row.content_hash.clone().or(carried.content_hash);
carried.main_uid = row.main_uid.clone().or(carried.main_uid);
carried.location = row.location.or(carried.location);
}
carried.resolved_at = row.resolved_at.or(carried.resolved_at);
// The tab this photo lands in is derived here, once, so a page or
// count query is a plain indexed `WHERE kind = ?` rather than a
// reclassification of every row.
let kind = crate::control::PhotoKind::classify(
carried.name.as_deref(),
carried.media_type.as_deref(),
);
(row, carried, kind)
})
.collect();
let entries: Vec<Grouped> = resolved
.iter()
.map(|(row, carried, kind)| Grouped {
uid: row.uid.clone(),
kind: *kind,
main_uid: carried.main_uid.clone(),
day: capture_day(&tx, row.capture_time),
stem: name_stem(carried.name.as_deref()),
})
.collect();
let group_keys = group_photos(&entries);
tx.execute("DELETE FROM photos", [])?; tx.execute("DELETE FROM photos", [])?;
{ {
let mut stmt = tx.prepare( let mut stmt = tx.prepare(
"INSERT INTO photos "INSERT INTO photos
(uid, capture_time, name, ratio, thumb_state, seq, media_type, kind) (uid, capture_time, name, ratio, thumb_state, seq, media_type, kind, favorite,
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)", content_hash, main_uid, group_key, resolved_at, latitude, longitude,
place_id, similar_hash)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, ?15,
?16, ?17)",
)?; )?;
for (seq, (uid, capture_time, name, media_type)) in items.iter().enumerate() { for (seq, ((row, carried, kind), group_key)) in
let (ratio, thumb_state, learned_media) = resolved.iter().zip(group_keys.iter()).enumerate()
learned {
.get(uid)
.cloned()
.unwrap_or((None, THUMB_UNKNOWN, None));
let media_type = media_type.clone().or(learned_media);
// The tab this photo lands in is derived here, once, so a page or
// count query is a plain indexed `WHERE kind = ?` rather than a
// reclassification of every row.
let kind =
crate::control::PhotoKind::classify(name.as_deref(), media_type.as_deref());
stmt.execute(params![ stmt.execute(params![
uid, row.uid,
capture_time, row.capture_time,
name, carried.name,
ratio, carried.ratio,
thumb_state, carried.thumb_state,
seq as i64, seq as i64,
media_type, carried.media_type,
kind.as_i64(), kind.as_i64(),
carried.favorite as i64,
carried.content_hash,
carried.main_uid,
group_key,
carried.resolved_at,
carried.location.map(|(lat, _)| lat),
carried.location.map(|(_, lon)| lon),
carried
.location
.and_then(|(lat, lon)| crate::places::nearest(lat, lon))
.map(|city| city.id),
carried.similar_hash,
])?; ])?;
} }
} }
@@ -89,33 +415,132 @@ impl Db {
Ok(()) Ok(())
} }
/// The uids whose node metadata is already resolved, so a refresh can skip
/// reading their nodes again. Everything else — a photo never seen before, or
/// one a remote event marked stale — is resolved. That is what keeps a
/// 12k-photo library off 63 round-trips of `enumerate_nodes` every refresh.
pub fn photos_resolved(&self) -> Result<Vec<String>> {
let conn = self.read();
let mut stmt = conn.prepare("SELECT uid FROM photos WHERE resolved_at IS NOT NULL")?;
let rows = stmt.query_map([], |r| r.get::<_, String>(0))?;
let mut uids = Vec::new();
for row in rows {
uids.push(row?);
}
Ok(uids)
}
/// Mark photos as needing a fresh metadata resolve, and report how many rows
/// that matched. This is how a remote change is recorded: the event feed says
/// *which* node changed but never *what* changed about it, so the uid is
/// written down and the next refresh reads the node.
pub fn photos_mark_unresolved(&self, uids: &[String]) -> Result<usize> {
if uids.is_empty() {
return Ok(0);
}
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let mut marked = 0;
{
let mut stmt = tx.prepare("UPDATE photos SET resolved_at = NULL WHERE uid = ?1")?;
for uid in uids {
marked += stmt.execute(params![uid])?;
}
}
tx.commit()?;
Ok(marked)
}
/// The same for the whole timeline: every photo is resolved again on the next
/// refresh. For a continuity loss, where the daemon cannot know what it
/// missed, and for an explicit `refresh photos --full`.
pub fn photos_mark_all_unresolved(&self) -> Result<usize> {
let conn = self.conn.lock();
Ok(conn.execute("UPDATE photos SET resolved_at = NULL", [])?)
}
/// Forget photos that have been trashed, so the gallery does not show them
/// until the next timeline refresh would have caught up.
///
/// Album membership goes with them: a trashed photo is gone from every album
/// that held it, and leaving the rows behind would keep it on the album
/// pages and in the album covers.
pub fn photos_delete(&self, uids: &[String]) -> Result<usize> {
if uids.is_empty() {
return Ok(0);
}
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let mut removed = 0;
{
let mut photos = tx.prepare("DELETE FROM photos WHERE uid = ?1")?;
let mut albums = tx.prepare("DELETE FROM album_photos WHERE uid = ?1")?;
for uid in uids {
removed += photos.execute([uid])?;
albums.execute([uid])?;
}
}
// A group whose representative went is left pointing at nothing, which
// drops its other files off every page. Make them stand alone until the
// next refresh groups them again.
tx.execute(
"UPDATE photos SET group_key = NULL WHERE group_key IS NOT NULL \
AND group_key NOT IN (SELECT uid FROM photos)",
[],
)?;
tx.commit()?;
Ok(removed)
}
/// One page of the persisted timeline, newest first. `kind`, when set, /// One page of the persisted timeline, newest first. `kind`, when set,
/// restricts the page to one tab (Photos / Videos / Raw); `range`, when set, /// restricts the page to one tab (Photos / Videos / Raw); `range`, when set,
/// restricts it to a `[from, to)` capture-time window (epoch seconds) — the /// restricts it to a `[from, to)` capture-time window (epoch seconds) — the
/// date scrubber's jump. `offset` is relative to whatever the filters leave. /// date scrubber's jump. `offset` is relative to whatever the filters leave.
/// `favorites` restricts the page to favorited photos, and `not_in_album`
/// to the shots no album holds any file of.
///
/// The page holds one row per *group*: the RAW and the JPEG of one shot are
/// one tile. The Raw tab is the exception and lists every raw file, because
/// that tab exists to answer "which raws do I have".
pub fn photos_page( pub fn photos_page(
&self, &self,
offset: usize, offset: usize,
limit: usize, limit: usize,
kind: Option<crate::control::PhotoKind>, kind: Option<crate::control::PhotoKind>,
range: Option<(i64, i64)>, range: Option<(i64, i64)>,
favorites: bool,
not_in_album: bool,
) -> Result<Vec<StoredPhoto>> { ) -> Result<Vec<StoredPhoto>> {
let conn = self.conn.lock(); let conn = self.read();
// Built up so any combination of the optional filters is one indexed // Built up so any combination of the optional filters is one indexed
// query rather than a statement per case. // query rather than a statement per case.
let mut sql = let mut sql = String::from(PHOTO_SELECT);
String::from("SELECT uid, capture_time, name, ratio, thumb_state, kind FROM photos");
let mut binds: Vec<i64> = Vec::new(); let mut binds: Vec<i64> = Vec::new();
let mut conds: Vec<String> = Vec::new(); let mut conds: Vec<String> = Vec::new();
if kind != Some(crate::control::PhotoKind::Raw) {
conds.push("p.uid = COALESCE(p.group_key, p.uid)".to_string());
}
if let Some(k) = kind { if let Some(k) = kind {
binds.push(k.as_i64()); binds.push(k.as_i64());
conds.push(format!("kind = ?{}", binds.len())); conds.push(format!("p.kind = ?{}", binds.len()));
} }
if let Some((from, to)) = range { if let Some((from, to)) = range {
binds.push(from); binds.push(from);
conds.push(format!("capture_time >= ?{}", binds.len())); conds.push(format!("p.capture_time >= ?{}", binds.len()));
binds.push(to); binds.push(to);
conds.push(format!("capture_time < ?{}", binds.len())); conds.push(format!("p.capture_time < ?{}", binds.len()));
}
if favorites {
conds.push("p.favorite = 1".to_string());
}
// By group, not by file: a shot filed as its JPEG is filed, even though
// its RAW was never added anywhere.
if not_in_album {
conds.push(
"NOT EXISTS (SELECT 1 FROM album_photos a JOIN photos m ON m.uid = a.uid \
WHERE COALESCE(m.group_key, m.uid) = COALESCE(p.group_key, p.uid))"
.to_string(),
);
} }
if !conds.is_empty() { if !conds.is_empty() {
sql.push_str(" WHERE "); sql.push_str(" WHERE ");
@@ -126,21 +551,41 @@ impl Db {
binds.push(offset as i64); binds.push(offset as i64);
let offset_pos = binds.len(); let offset_pos = binds.len();
sql.push_str(&format!( sql.push_str(&format!(
" ORDER BY seq LIMIT ?{limit_pos} OFFSET ?{offset_pos}" " ORDER BY p.seq LIMIT ?{limit_pos} OFFSET ?{offset_pos}"
)); ));
let mut stmt = conn.prepare(&sql)?; let mut stmt = conn.prepare(&sql)?;
let rows = stmt let rows = stmt
.query_map(rusqlite::params_from_iter(binds), |r| { .query_map(rusqlite::params_from_iter(binds), stored_photo)?
Ok(StoredPhoto { .collect::<rusqlite::Result<_>>()?;
uid: r.get(0)?, Ok(rows)
capture_time: r.get(1)?, }
name: r.get(2)?,
ratio: r.get(3)?, /// Every photo of one photo's group, in server order — the RAW and the JPEG
thumb_state: r.get(4)?, /// of a shot, or just the photo itself when it stands alone. The lightbox
kind: crate::control::PhotoKind::from_i64(r.get(5)?), /// switches between these, and deleting a tile trashes all of them.
}) pub fn photos_group(&self, uid: &str) -> Result<Vec<StoredPhoto>> {
})? let conn = self.read();
let sql = format!(
"{PHOTO_SELECT} WHERE COALESCE(p.group_key, p.uid) = \
(SELECT COALESCE(group_key, uid) FROM photos WHERE uid = ?1) ORDER BY p.seq"
);
let mut stmt = conn.prepare(&sql)?;
let rows = stmt
.query_map(params![uid], stored_photo)?
.collect::<rusqlite::Result<_>>()?;
Ok(rows)
}
/// The whole persisted timeline, in stored order — every file, not one row
/// per group. Only for whole-library passes (the photo re-date); the gallery
/// pages with [`Db::photos_page`].
pub fn photos_all(&self) -> Result<Vec<StoredPhoto>> {
let conn = self.read();
let sql = format!("{PHOTO_SELECT} ORDER BY p.seq");
let mut stmt = conn.prepare(&sql)?;
let rows = stmt
.query_map([], stored_photo)?
.collect::<rusqlite::Result<_>>()?; .collect::<rusqlite::Result<_>>()?;
Ok(rows) Ok(rows)
} }
@@ -154,10 +599,16 @@ impl Db {
&self, &self,
kind: Option<crate::control::PhotoKind>, kind: Option<crate::control::PhotoKind>,
) -> Result<Vec<(i32, i32, usize)>> { ) -> Result<Vec<(i32, i32, usize)>> {
let conn = self.conn.lock(); let conn = self.read();
// The scrubber has to agree with the page it scrubs, so it counts what
// the page shows: groups everywhere except on the Raw tab.
let (filter, binds): (&str, Vec<i64>) = match kind { let (filter, binds): (&str, Vec<i64>) = match kind {
Some(k) => (" WHERE kind = ?1", vec![k.as_i64()]), Some(crate::control::PhotoKind::Raw) => (" WHERE kind = 2", Vec::new()),
None => ("", Vec::new()), Some(k) => (
" WHERE uid = COALESCE(group_key, uid) AND kind = ?1",
vec![k.as_i64()],
),
None => (" WHERE uid = COALESCE(group_key, uid)", Vec::new()),
}; };
let sql = format!( let sql = format!(
"SELECT CAST(strftime('%Y', capture_time, 'unixepoch', 'localtime') AS INTEGER) AS y, \ "SELECT CAST(strftime('%Y', capture_time, 'unixepoch', 'localtime') AS INTEGER) AS y, \
@@ -178,11 +629,267 @@ impl Db {
Ok(rows) Ok(rows)
} }
/// Photos taken on `today`'s calendar day in earlier years, newest first —
/// the "On this day" strip. `today` is epoch seconds: its local month and day
/// pick the date, and its own year is left out, because today's photos are
/// what the timeline already opens on. One row per group, like the timeline.
pub fn photos_on_this_day(&self, today: i64, limit: usize) -> Result<Vec<StoredPhoto>> {
let conn = self.read();
let sql = format!(
"{PHOTO_SELECT} WHERE p.uid = COALESCE(p.group_key, p.uid) \
AND strftime('%m-%d', p.capture_time, 'unixepoch', 'localtime') \
= strftime('%m-%d', ?1, 'unixepoch', 'localtime') \
AND strftime('%Y', p.capture_time, 'unixepoch', 'localtime') \
< strftime('%Y', ?1, 'unixepoch', 'localtime') \
ORDER BY p.seq LIMIT ?2"
);
let mut stmt = conn.prepare(&sql)?;
let rows = stmt
.query_map(params![today, limit as i64], stored_photo)?
.collect::<rusqlite::Result<_>>()?;
Ok(rows)
}
/// Every town photos were taken in, with how many and the newest one as its
/// cover, most photographed first. Counted per group, like the timeline;
/// photos without a location, or taken far from any town, are in none.
pub fn photo_places(&self) -> Result<Vec<StoredPlace>> {
let rows: Vec<(u32, usize, String)> = {
let conn = self.read();
// SQLite fills the bare `uid` from the row `MIN(seq)` picked, which
// is the newest photo of the place.
let mut stmt = conn.prepare(
"SELECT place_id, COUNT(*), uid, MIN(seq) FROM photos \
WHERE uid = COALESCE(group_key, uid) AND place_id IS NOT NULL \
GROUP BY place_id ORDER BY COUNT(*) DESC, MIN(seq)",
)?;
stmt.query_map([], |r| {
Ok((r.get(0)?, r.get::<_, i64>(1)? as usize, r.get(2)?))
})?
.collect::<rusqlite::Result<_>>()?
};
let uids: Vec<String> = rows.iter().map(|(_, _, uid)| uid.clone()).collect();
let mut covers: HashMap<String, StoredPhoto> = self
.photos_by_uid(&uids)?
.into_iter()
.map(|photo| (photo.uid.clone(), photo))
.collect();
Ok(rows
.into_iter()
.filter_map(|(id, count, uid)| {
Some(StoredPlace {
id,
count,
cover: covers.remove(&uid)?,
})
})
.collect())
}
/// One page of the photos taken in a town, newest first, one per group.
pub fn photos_at_place(
&self,
id: u32,
offset: usize,
limit: usize,
) -> Result<Vec<StoredPhoto>> {
let conn = self.read();
let sql = format!(
"{PHOTO_SELECT} WHERE p.uid = COALESCE(p.group_key, p.uid) AND p.place_id = ?1 \
ORDER BY p.seq LIMIT ?2 OFFSET ?3"
);
let mut stmt = conn.prepare(&sql)?;
let rows = stmt
.query_map(params![id, limit as i64, offset as i64], stored_photo)?
.collect::<rusqlite::Result<_>>()?;
Ok(rows)
}
/// Every photo taken in a town, one per group as in [`Self::photos_at_place`],
/// with where exactly it was taken. A group's own location is that of the
/// photo standing for it.
pub fn photo_locations_at_place(&self, id: u32) -> Result<Vec<(StoredPhoto, (f64, f64))>> {
let photos = self.photos_at_place(id, 0, usize::MAX >> 1)?;
let located: HashMap<String, (f64, f64)> = {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT uid, latitude, longitude FROM photos \
WHERE place_id = ?1 AND latitude IS NOT NULL AND longitude IS NOT NULL",
)?;
stmt.query_map(params![id], |r| Ok((r.get(0)?, (r.get(1)?, r.get(2)?))))?
.collect::<rusqlite::Result<_>>()?
};
Ok(photos
.into_iter()
.filter_map(|photo| {
let at = *located.get(&photo.uid)?;
Some((photo, at))
})
.collect())
}
/// Sets of photos with the same content hash, each set ordered with the
/// copy worth keeping first, and the sets newest first.
///
/// The copy to keep is the one in the most albums, then the one that is part
/// of a larger group (the JPEG's RAW is filed with it, a stray re-upload is
/// not), then the earliest in server order. At most one file per group is
/// listed: the files of one RAW+JPEG shot are not copies of each other, and a
/// set left with one member is no set.
pub fn photo_duplicates(&self) -> Result<Vec<Vec<StoredPhoto>>> {
let rows: Vec<(String, String, String)> = {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT p.uid, p.content_hash, COALESCE(p.group_key, p.uid) AS k FROM photos p \
WHERE p.content_hash IN (SELECT content_hash FROM photos \
WHERE content_hash IS NOT NULL GROUP BY content_hash HAVING COUNT(*) > 1) \
ORDER BY p.content_hash, \
(SELECT COUNT(*) FROM album_photos a WHERE a.uid = p.uid) DESC, \
(SELECT COUNT(*) FROM photos m WHERE COALESCE(m.group_key, m.uid) = COALESCE(p.group_key, p.uid)) DESC, \
p.seq",
)?;
stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))?
.collect::<rusqlite::Result<_>>()?
};
let uids: Vec<String> = rows.iter().map(|(uid, _, _)| uid.clone()).collect();
let mut stored: HashMap<String, StoredPhoto> = self
.photos_by_uid(&uids)?
.into_iter()
.map(|photo| (photo.uid.clone(), photo))
.collect();
let mut sets: Vec<Vec<StoredPhoto>> = Vec::new();
let mut hash_of_set: Option<&str> = None;
let mut groups_in_set: HashSet<&str> = HashSet::new();
for (uid, hash, group) in &rows {
if hash_of_set != Some(hash.as_str()) {
hash_of_set = Some(hash);
groups_in_set.clear();
sets.push(Vec::new());
}
if !groups_in_set.insert(group) {
continue;
}
if let (Some(set), Some(photo)) = (sets.last_mut(), stored.remove(uid)) {
set.push(photo);
}
}
sets.retain(|set| set.len() > 1);
sets.sort_by_key(|set| std::cmp::Reverse(set.iter().map(|p| p.capture_time).max()));
Ok(sets)
}
/// Shots whose look hasn't been hashed yet, by the uid of the photo the
/// grid shows for each, newest first, at most `limit`. Videos, and photos
/// with no thumbnail to hash, are left out. A shot counts as hashed once
/// any of its files is, since they all show one picture.
pub fn photos_unhashed(&self, limit: usize) -> Result<Vec<StoredPhoto>> {
let uids: Vec<String> = {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT COALESCE(group_key, uid) AS k FROM photos \
GROUP BY k \
HAVING MAX(similar_hash) IS NULL \
AND MAX(kind != ?1 AND thumb_state != ?2) \
ORDER BY MAX(capture_time) DESC LIMIT ?3",
)?;
stmt.query_map(
params![
crate::control::PhotoKind::Video.as_i64(),
THUMB_NONE,
limit as i64
],
|r| r.get(0),
)?
.collect::<rusqlite::Result<_>>()?
};
self.photos_by_uid(&uids)
}
/// Record the look of photo `uid`: its difference hash, or
/// [`crate::similar::NO_HASH`] when its thumbnail gave none.
pub fn photo_set_similar_hash(&self, uid: &str, hash: u64) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"UPDATE photos SET similar_hash = ?2 WHERE uid = ?1",
params![uid, hash as i64],
)?;
Ok(())
}
/// How many shots have been hashed, out of how many could be.
pub fn similar_progress(&self) -> Result<(usize, usize)> {
let conn = self.read();
let (hashed, total): (i64, i64) = conn.query_row(
"SELECT COUNT(*) FILTER (WHERE hashed), COUNT(*) FROM \
(SELECT MAX(similar_hash) IS NOT NULL AS hashed FROM photos \
GROUP BY COALESCE(group_key, uid) \
HAVING MAX(similar_hash) IS NOT NULL \
OR MAX(kind != ?1 AND thumb_state != ?2))",
params![crate::control::PhotoKind::Video.as_i64(), THUMB_NONE],
|r| Ok((r.get(0)?, r.get(1)?)),
)?;
Ok((hashed.max(0) as usize, total.max(0) as usize))
}
/// Sets of shots that look alike (see [`crate::similar`]), one photo per
/// shot in capture order, and the sets newest first. A set whose photos
/// are all byte-identical copies is left to [`Self::photo_duplicates`].
pub fn photo_similar(&self) -> Result<Vec<Vec<StoredPhoto>>> {
let rows: Vec<(String, u64, Option<String>)> = {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT COALESCE(group_key, uid) AS k, MAX(similar_hash), \
MAX(content_hash) FILTER (WHERE uid = COALESCE(group_key, uid)) \
FROM photos GROUP BY k \
HAVING MAX(similar_hash) IS NOT NULL AND MAX(similar_hash) != ?1",
)?;
stmt.query_map(params![crate::similar::NO_HASH as i64], |r| {
Ok((r.get(0)?, r.get::<_, i64>(1)? as u64, r.get(2)?))
})?
.collect::<rusqlite::Result<_>>()?
};
let hashes: Vec<u64> = rows.iter().map(|(_, hash, _)| *hash).collect();
let found: Vec<Vec<usize>> = crate::similar::sets(&hashes)
.into_iter()
.filter(|set| {
let first = &rows[set[0]].2;
first.is_none() || set.iter().any(|&i| rows[i].2 != *first)
})
.collect();
let uids: Vec<String> = found.iter().flatten().map(|&i| rows[i].0.clone()).collect();
let mut stored: HashMap<String, StoredPhoto> = self
.photos_by_uid(&uids)?
.into_iter()
.map(|photo| (photo.uid.clone(), photo))
.collect();
let mut sets: Vec<Vec<StoredPhoto>> = found
.iter()
.map(|set| {
let mut photos: Vec<StoredPhoto> = set
.iter()
.filter_map(|&i| stored.remove(&rows[i].0))
.collect();
photos.sort_by_key(|p| p.capture_time);
photos
})
.filter(|set| set.len() > 1)
.collect();
sets.sort_by_key(|set| std::cmp::Reverse(set.iter().map(|p| p.capture_time).max()));
Ok(sets)
}
/// Per-tab counts for the Photos page subtitle: `(photos, videos, raw)`. /// Per-tab counts for the Photos page subtitle: `(photos, videos, raw)`.
pub fn photos_counts(&self) -> Result<(usize, usize, usize)> { pub fn photos_counts(&self) -> Result<(usize, usize, usize)> {
use crate::control::PhotoKind; use crate::control::PhotoKind;
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare("SELECT kind, COUNT(*) FROM photos GROUP BY kind")?; // Photos and Videos count groups — one shot is one entry, however many
// files it was stored as. Raw counts files, because the Raw tab lists
// them individually.
let mut stmt = conn.prepare(
"SELECT kind, COUNT(*) FROM photos \
WHERE uid = COALESCE(group_key, uid) OR kind = 2 GROUP BY kind",
)?;
let rows = stmt.query_map([], |r| Ok((r.get::<_, i64>(0)?, r.get::<_, i64>(1)?)))?; let rows = stmt.query_map([], |r| Ok((r.get::<_, i64>(0)?, r.get::<_, i64>(1)?)))?;
let (mut photos, mut videos, mut raw) = (0usize, 0usize, 0usize); let (mut photos, mut videos, mut raw) = (0usize, 0usize, 0usize);
for row in rows { for row in rows {
@@ -203,22 +910,11 @@ impl Db {
if uids.is_empty() { if uids.is_empty() {
return Ok(Vec::new()); return Ok(Vec::new());
} }
let conn = self.conn.lock(); let conn = self.read();
let placeholders = vec!["?"; uids.len()].join(","); let placeholders = vec!["?"; uids.len()].join(",");
let mut stmt = conn.prepare(&format!( let sql = format!("{PHOTO_SELECT} WHERE p.uid IN ({placeholders})");
"SELECT uid, capture_time, name, ratio, thumb_state, kind FROM photos let mut stmt = conn.prepare(&sql)?;
WHERE uid IN ({placeholders})" let rows = stmt.query_map(rusqlite::params_from_iter(uids), stored_photo)?;
))?;
let rows = stmt.query_map(rusqlite::params_from_iter(uids), |r| {
Ok(StoredPhoto {
uid: r.get(0)?,
capture_time: r.get(1)?,
name: r.get(2)?,
ratio: r.get(3)?,
thumb_state: r.get(4)?,
kind: crate::control::PhotoKind::from_i64(r.get(5)?),
})
})?;
let mut photos = Vec::new(); let mut photos = Vec::new();
for row in rows { for row in rows {
photos.push(row?); photos.push(row?);
@@ -226,6 +922,18 @@ impl Db {
Ok(photos) Ok(photos)
} }
/// Record a photo's favorite flag locally, after the server accepted the
/// change. A uid the timeline does not hold is a no-op — an album photo on
/// someone else's volume is never in our own `photos` table.
pub fn photos_set_favorite(&self, uid: &str, favorite: bool) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"UPDATE photos SET favorite = ?2 WHERE uid = ?1",
params![uid, favorite as i64],
)?;
Ok(())
}
/// Record what a thumbnail attempt learned: whether the photo now has one /// Record what a thumbnail attempt learned: whether the photo now has one
/// ([`THUMB_HAVE`] / [`THUMB_NONE`]), and its aspect ratio if the pixels were /// ([`THUMB_HAVE`] / [`THUMB_NONE`]), and its aspect ratio if the pixels were
/// seen. A `None` ratio leaves any previously learned one alone. /// seen. A `None` ratio leaves any previously learned one alone.
@@ -240,7 +948,7 @@ impl Db {
/// Number of photos in the persisted timeline. /// Number of photos in the persisted timeline.
pub fn photos_count(&self) -> Result<usize> { pub fn photos_count(&self) -> Result<usize> {
let conn = self.conn.lock(); let conn = self.read();
let n: i64 = conn.query_row("SELECT COUNT(*) FROM photos", [], |r| r.get(0))?; let n: i64 = conn.query_row("SELECT COUNT(*) FROM photos", [], |r| r.get(0))?;
Ok(n.max(0) as usize) Ok(n.max(0) as usize)
} }
+49 -20
View File
@@ -4,8 +4,21 @@
use rusqlite::{OptionalExtension, params}; use rusqlite::{OptionalExtension, params};
use super::Db; use super::Db;
use super::utils::MAX_PATH_DEPTH;
use crate::Result; use crate::Result;
/// One row of [`Db::pin_list`]. `recursive` is pin *policy* — whether the whole
/// subtree is kept — while `is_dir` is the node's kind as the node cache knows
/// it, or `None` for a pin whose node is not cached. Reading the former as the
/// latter is what made a non-recursively pinned folder render as a file.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PinRow {
pub uid: String,
pub path: String,
pub recursive: bool,
pub is_dir: Option<bool>,
}
impl Db { impl Db {
pub fn pin_add(&self, uid: &str, path: &str, recursive: bool) -> Result<()> { pub fn pin_add(&self, uid: &str, path: &str, recursive: bool) -> Result<()> {
let conn = self.conn.lock(); let conn = self.conn.lock();
@@ -25,16 +38,23 @@ impl Db {
Ok(n > 0) Ok(n > 0)
} }
/// Every directly-pinned entry `(uid, path, recursive)`, ordered by uid. /// Every directly-pinned entry, ordered by uid.
pub fn pin_list(&self) -> Result<Vec<(String, String, bool)>> { pub fn pin_list(&self) -> Result<Vec<PinRow>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare("SELECT uid, path, recursive FROM pins ORDER BY uid")?; // `recursive` is pin policy; the node's kind comes from `nodes`, and is
// NULL for a pin whose node was never cached.
let mut stmt = conn.prepare(
"SELECT p.uid, p.path, p.recursive, n.is_dir
FROM pins p LEFT JOIN nodes n ON n.uid = p.uid
ORDER BY p.uid",
)?;
let rows = stmt.query_map([], |r| { let rows = stmt.query_map([], |r| {
Ok(( Ok(PinRow {
r.get::<_, String>(0)?, uid: r.get(0)?,
r.get::<_, String>(1)?, path: r.get(1)?,
r.get::<_, i64>(2)? != 0, recursive: r.get::<_, i64>(2)? != 0,
)) is_dir: r.get::<_, Option<i64>>(3)?.map(|is_dir| is_dir != 0),
})
})?; })?;
let mut out = Vec::new(); let mut out = Vec::new();
for r in rows { for r in rows {
@@ -47,8 +67,13 @@ impl Db {
/// is pinned recursively. The ancestor check walks `parent_uid` to the root /// is pinned recursively. The ancestor check walks `parent_uid` to the root
/// via a CTE; a direct pin is honoured even when the node has no `nodes` row /// via a CTE; a direct pin is honoured even when the node has no `nodes` row
/// yet (e.g. a CLI that never hydrates the node cache). /// yet (e.g. a CLI that never hydrates the node cache).
///
/// The walk is depth-capped like [`super::utils::path_of`]: a `parent_uid`
/// cycle is corrupt data the API can hand us, and `UNION ALL` over one never
/// terminates — while holding the daemon's only SQLite connection, on a path
/// that runs per cached read.
pub fn is_pinned(&self, uid: &str) -> Result<bool> { pub fn is_pinned(&self, uid: &str) -> Result<bool> {
let conn = self.conn.lock(); let conn = self.read();
let direct: Option<i64> = conn let direct: Option<i64> = conn
.query_row("SELECT 1 FROM pins WHERE uid = ?1", params![uid], |r| { .query_row("SELECT 1 FROM pins WHERE uid = ?1", params![uid], |r| {
r.get(0) r.get(0)
@@ -59,14 +84,18 @@ impl Db {
} }
let anc: Option<i64> = conn let anc: Option<i64> = conn
.query_row( .query_row(
"WITH RECURSIVE anc(uid, parent_uid) AS ( &format!(
SELECT uid, parent_uid FROM nodes WHERE uid = ?1 "WITH RECURSIVE anc(uid, parent_uid, depth) AS (
UNION ALL SELECT uid, parent_uid, 0 FROM nodes WHERE uid = ?1
SELECT n.uid, n.parent_uid FROM nodes n JOIN anc ON n.uid = anc.parent_uid UNION ALL
) SELECT n.uid, n.parent_uid, anc.depth + 1
SELECT 1 FROM anc a JOIN pins p ON p.uid = a.uid FROM nodes n JOIN anc ON n.uid = anc.parent_uid
WHERE a.uid != ?1 AND p.recursive = 1 WHERE anc.depth < {MAX_PATH_DEPTH}
LIMIT 1", )
SELECT 1 FROM anc a JOIN pins p ON p.uid = a.uid
WHERE a.uid != ?1 AND p.recursive = 1
LIMIT 1"
),
params![uid], params![uid],
|r| r.get(0), |r| r.get(0),
) )
@@ -78,7 +107,7 @@ impl Db {
/// every descendant of a recursively-pinned folder. Used to build the /// every descendant of a recursively-pinned folder. Used to build the
/// eviction-exempt set (the budget enforcer hashes these into cache keys). /// eviction-exempt set (the budget enforcer hashes these into cache keys).
pub fn pinned_uids(&self) -> Result<Vec<String>> { pub fn pinned_uids(&self) -> Result<Vec<String>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"WITH RECURSIVE sub(uid) AS ( "WITH RECURSIVE sub(uid) AS (
SELECT uid FROM pins WHERE recursive = 1 SELECT uid FROM pins WHERE recursive = 1
@@ -100,7 +129,7 @@ impl Db {
/// Every descendant uid of `folder` (all depths), via a `parent_uid` CTE. /// Every descendant uid of `folder` (all depths), via a `parent_uid` CTE.
/// Used to evict a recursively-pinned subtree's cached blobs on unpin. /// Used to evict a recursively-pinned subtree's cached blobs on unpin.
pub fn descendants(&self, folder: &str) -> Result<Vec<String>> { pub fn descendants(&self, folder: &str) -> Result<Vec<String>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"WITH RECURSIVE sub(uid) AS ( "WITH RECURSIVE sub(uid) AS (
SELECT uid FROM nodes WHERE parent_uid = ?1 SELECT uid FROM nodes WHERE parent_uid = ?1
+143
View File
@@ -0,0 +1,143 @@
//! Persisted effective access for roots shared with this account.
use std::collections::HashMap;
use proton_drive_rs::proton_sdk::ids::{LinkId, NodeUid, VolumeId};
use rusqlite::{OptionalExtension, params};
use super::Db;
use crate::{Access, Error, Result};
impl Db {
/// Load the complete, deliberately small shared-root authority map.
///
/// Mount state keeps this resident so ordinary node interning and subtree
/// propagation never turn into one SQLite query per inode. The database
/// remains the restart/offline authority and every explicit role change is
/// written through before this cache is updated.
pub fn all_share_access(&self) -> Result<HashMap<NodeUid, Access>> {
let conn = self.read();
let mut stmt = conn.prepare("SELECT root_uid, access FROM share_access")?;
let rows = stmt.query_map([], |row| {
Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?))
})?;
let mut access = HashMap::new();
for row in rows {
let (uid, value) = row?;
let (volume, link) = uid.split_once('~').ok_or_else(|| {
Error::Other(format!(
"invalid shared-root uid {uid:?} stored in database"
))
})?;
let access_value = Access::from_db_str(&value).ok_or_else(|| {
Error::Other(format!("invalid share access {value:?} stored for {uid}"))
})?;
access.insert(
NodeUid::new(VolumeId::from(volume), LinkId::from(link)),
access_value,
);
}
Ok(access)
}
/// Read the effective access recorded for a shared-tree root.
pub fn share_access(&self, uid: &NodeUid) -> Result<Option<Access>> {
let conn = self.read();
let value = conn
.query_row(
"SELECT access FROM share_access WHERE root_uid = ?1",
[uid.to_string()],
|row| row.get::<_, String>(0),
)
.optional()?;
value
.map(|value| {
Access::from_db_str(&value).ok_or_else(|| {
Error::Other(format!("invalid share access {value:?} stored for {uid}"))
})
})
.transpose()
}
/// Insert or replace the effective access for a shared-tree root.
pub fn set_share_access(&self, uid: &NodeUid, access: Access) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"INSERT INTO share_access (root_uid, access) VALUES (?1, ?2)
ON CONFLICT(root_uid) DO UPDATE SET access = excluded.access",
params![uid.to_string(), access.as_db_str()],
)?;
Ok(())
}
/// Forget a shared-tree root's effective access.
pub fn delete_share_access(&self, uid: &NodeUid) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"DELETE FROM share_access WHERE root_uid = ?1",
[uid.to_string()],
)?;
Ok(())
}
/// Fail closed every persisted shared-tree authority.
///
/// Owned and device trees have no row in `share_access`, so this only
/// affects roots already identified as shared with this account.
pub fn downgrade_all_share_access(&self) -> Result<usize> {
let conn = self.conn.lock();
Ok(conn.execute(
"UPDATE share_access SET access = 'viewer' WHERE access != 'viewer'",
[],
)?)
}
/// Resolve a persisted node's effective access through its nearest recorded
/// share root. `None` means the node itself is not persisted, which callers
/// must not treat as owned: stale handles need to fail closed.
pub fn effective_node_access(&self, uid: &NodeUid) -> Result<Option<Access>> {
let conn = self.read();
let exists: bool = conn.query_row(
"SELECT EXISTS(SELECT 1 FROM nodes WHERE uid = ?1)",
[uid.to_string()],
|row| row.get(0),
)?;
if !exists {
return Ok(None);
}
let value = conn
.query_row(
"WITH RECURSIVE ancestors(uid, parent_uid, depth, path) AS (
SELECT uid, parent_uid, 0, char(31) || uid || char(31)
FROM nodes WHERE uid = ?1
UNION ALL
SELECT a.parent_uid, n.parent_uid, a.depth + 1,
a.path || a.parent_uid || char(31)
FROM ancestors a
LEFT JOIN nodes n ON n.uid = a.parent_uid
WHERE a.parent_uid IS NOT NULL
AND instr(
a.path,
char(31) || a.parent_uid || char(31)
) = 0
)
SELECT sa.access
FROM ancestors a JOIN share_access sa ON sa.root_uid = a.uid
ORDER BY a.depth ASC
LIMIT 1",
[uid.to_string()],
|row| row.get::<_, String>(0),
)
.optional()?;
value
.map(|value| {
Access::from_db_str(&value).ok_or_else(|| {
Error::Other(format!(
"invalid effective share access {value:?} for {uid}"
))
})
})
.transpose()
.map(|access| Some(access.unwrap_or(Access::Owner)))
}
}
+25 -3
View File
@@ -8,7 +8,7 @@ use crate::Result;
impl Db { impl Db {
pub fn get_event_cursor(&self) -> Result<Option<String>> { pub fn get_event_cursor(&self) -> Result<Option<String>> {
let conn = self.conn.lock(); let conn = self.read();
let v = conn let v = conn
.query_row( .query_row(
"SELECT value FROM sync_state WHERE key = 'event_cursor'", "SELECT value FROM sync_state WHERE key = 'event_cursor'",
@@ -32,7 +32,7 @@ impl Db {
/// Read a `sync_state` value as a string. /// Read a `sync_state` value as a string.
pub fn state_str(&self, key: &str) -> Result<Option<String>> { pub fn state_str(&self, key: &str) -> Result<Option<String>> {
let conn = self.conn.lock(); let conn = self.read();
let v = conn let v = conn
.query_row( .query_row(
"SELECT value FROM sync_state WHERE key = ?1", "SELECT value FROM sync_state WHERE key = ?1",
@@ -66,7 +66,7 @@ impl Db {
/// dropping it would leave a file that exists nowhere but this machine with /// dropping it would leave a file that exists nowhere but this machine with
/// nothing left to create it. (Writes to a node that is itself still queued /// nothing left to create it. (Writes to a node that is itself still queued
pub fn state_i64(&self, key: &str) -> Result<Option<i64>> { pub fn state_i64(&self, key: &str) -> Result<Option<i64>> {
let conn = self.conn.lock(); let conn = self.read();
let v: Option<String> = conn let v: Option<String> = conn
.query_row( .query_row(
"SELECT value FROM sync_state WHERE key = ?1", "SELECT value FROM sync_state WHERE key = ?1",
@@ -94,4 +94,26 @@ impl Db {
conn.execute("DELETE FROM sync_state WHERE key = ?1", params![key])?; conn.execute("DELETE FROM sync_state WHERE key = ?1", params![key])?;
Ok(()) Ok(())
} }
/// Drop every `sync_state` key starting with `prefix` — the per-entity
/// stamps (one album, one shared folder) that are keyed by uid and so have
/// no fixed name to clear individually.
pub fn clear_state_prefix(&self, prefix: &str) -> Result<()> {
let conn = self.conn.lock();
// `\` escapes the LIKE wildcards, so a prefix containing `%` or `_`
// (a uid never does today, but this is not the place to rely on that)
// matches literally instead of sweeping unrelated keys.
let pattern = format!(
"{}%",
prefix
.replace('\\', "\\\\")
.replace('%', "\\%")
.replace('_', "\\_")
);
conn.execute(
"DELETE FROM sync_state WHERE key LIKE ?1 ESCAPE '\\'",
params![pattern],
)?;
Ok(())
}
} }
+66 -16
View File
@@ -35,6 +35,14 @@ pub struct StoredSyncEntry {
pub remote_uid: Option<String>, pub remote_uid: Option<String>,
pub local_mtime: i64, pub local_mtime: i64,
pub local_size: i64, pub local_size: i64,
/// The same modification time to nanosecond precision, when it is known.
///
/// `local_mtime` alone cannot tell an edit that landed inside the same
/// second as the last sync — and left the file the same length — from no
/// edit at all, so those never uploaded (bugs.md B25). `None` on a row
/// written before schema 28, which the comparison reads as "compare whole
/// seconds", so upgrading does not make an entire mirror look changed.
pub local_mtime_ns: Option<i64>,
pub remote_rev: Option<String>, pub remote_rev: Option<String>,
pub remote_hash: Option<String>, pub remote_hash: Option<String>,
} }
@@ -57,7 +65,7 @@ impl Db {
/// Every synced folder, oldest first. /// Every synced folder, oldest first.
pub fn sync_folder_list(&self) -> Result<Vec<StoredSyncFolder>> { pub fn sync_folder_list(&self) -> Result<Vec<StoredSyncFolder>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT id, local_path, remote_uid, remote_share_id, mode, pending_mode, state, last_sync "SELECT id, local_path, remote_uid, remote_share_id, mode, pending_mode, state, last_sync
FROM sync_folder ORDER BY id", FROM sync_folder ORDER BY id",
@@ -83,7 +91,7 @@ impl Db {
/// Look up one synced folder by id. /// Look up one synced folder by id.
pub fn sync_folder_get(&self, id: i64) -> Result<Option<StoredSyncFolder>> { pub fn sync_folder_get(&self, id: i64) -> Result<Option<StoredSyncFolder>> {
let conn = self.conn.lock(); let conn = self.read();
conn.query_row( conn.query_row(
"SELECT id, local_path, remote_uid, remote_share_id, mode, pending_mode, state, last_sync "SELECT id, local_path, remote_uid, remote_share_id, mode, pending_mode, state, last_sync
FROM sync_folder WHERE id = ?1", FROM sync_folder WHERE id = ?1",
@@ -115,14 +123,17 @@ impl Db {
Ok(n > 0) Ok(n > 0)
} }
/// Update a synced folder's mode (`mirror`/`ondemand`). The folder has /// Update a synced folder's mode (`mirror`/`ondemand`). A queued request for
/// reached the mode it was asked for, so any queued request is satisfied and /// that mode is satisfied and cleared in the same write. An opposite pending
/// cleared in the same write — a `pending_mode` outliving the switch it asked /// mode arrived after this transition started, so preserve it as the newer
/// for would have the engine try to apply it again on the next pass. /// last-request intent.
pub fn sync_folder_set_mode(&self, id: i64, mode: &str) -> Result<()> { pub fn sync_folder_set_mode(&self, id: i64, mode: &str) -> Result<()> {
let conn = self.conn.lock(); let conn = self.conn.lock();
conn.execute( conn.execute(
"UPDATE sync_folder SET mode = ?2, pending_mode = NULL WHERE id = ?1", "UPDATE sync_folder
SET mode = ?2,
pending_mode = CASE WHEN pending_mode = ?2 THEN NULL ELSE pending_mode END
WHERE id = ?1",
params![id, mode], params![id, mode],
)?; )?;
Ok(()) Ok(())
@@ -139,6 +150,18 @@ impl Db {
Ok(()) Ok(())
} }
/// Withdraw a queued mode only if it is still the intent the caller
/// inspected. A newer request must survive an older settlement attempt.
pub fn sync_folder_clear_pending_mode_if(&self, id: i64, expected: &str) -> Result<bool> {
let conn = self.conn.lock();
let changed = conn.execute(
"UPDATE sync_folder SET pending_mode = NULL
WHERE id = ?1 AND pending_mode = ?2",
params![id, expected],
)?;
Ok(changed > 0)
}
/// Update a synced folder's state and stamp `last_sync` to now. /// Update a synced folder's state and stamp `last_sync` to now.
pub fn sync_folder_set_state(&self, id: i64, state: &str, last_sync: i64) -> Result<()> { pub fn sync_folder_set_state(&self, id: i64, state: &str, last_sync: i64) -> Result<()> {
let conn = self.conn.lock(); let conn = self.conn.lock();
@@ -151,9 +174,10 @@ impl Db {
/// The whole per-file sync baseline for a folder, keyed by relative path. /// The whole per-file sync baseline for a folder, keyed by relative path.
pub fn sync_entries(&self, folder_id: i64) -> Result<HashMap<String, StoredSyncEntry>> { pub fn sync_entries(&self, folder_id: i64) -> Result<HashMap<String, StoredSyncEntry>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash "SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash,
local_mtime_ns
FROM sync_entry WHERE folder_id = ?1", FROM sync_entry WHERE folder_id = ?1",
)?; )?;
let rows = stmt.query_map(params![folder_id], |r| { let rows = stmt.query_map(params![folder_id], |r| {
@@ -164,6 +188,7 @@ impl Db {
local_size: r.get(3)?, local_size: r.get(3)?,
remote_rev: r.get(4)?, remote_rev: r.get(4)?,
remote_hash: r.get(5)?, remote_hash: r.get(5)?,
local_mtime_ns: r.get(6)?,
}) })
})?; })?;
let mut map = HashMap::new(); let mut map = HashMap::new();
@@ -174,19 +199,43 @@ impl Db {
Ok(map) Ok(map)
} }
/// Whether `uid` is represented by a mirror location.
///
/// An on-demand location proves residency through its live inode state.
/// Mirror locations have no FUSE state, so their folder root comes from
/// `sync_folder` and descendants come from the last successful
/// `sync_entry` baseline.
pub fn mirror_contains_uid(&self, uid: &str) -> Result<bool> {
let conn = self.read();
conn.query_row(
"SELECT EXISTS(
SELECT 1
FROM sync_folder AS folder
LEFT JOIN sync_entry AS entry ON entry.folder_id = folder.id
WHERE folder.mode = 'mirror'
AND (folder.remote_uid = ?1 OR entry.remote_uid = ?1)
)",
params![uid],
|row| row.get(0),
)
.map_err(Into::into)
}
/// Insert or replace one baseline row. /// Insert or replace one baseline row.
pub fn sync_entry_upsert(&self, folder_id: i64, e: &StoredSyncEntry) -> Result<()> { pub fn sync_entry_upsert(&self, folder_id: i64, e: &StoredSyncEntry) -> Result<()> {
let conn = self.conn.lock(); let conn = self.conn.lock();
conn.execute( conn.execute(
"INSERT INTO sync_entry "INSERT INTO sync_entry
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash) (folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev,
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7) remote_hash, local_mtime_ns)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)
ON CONFLICT(folder_id, rel_path) DO UPDATE SET ON CONFLICT(folder_id, rel_path) DO UPDATE SET
remote_uid = excluded.remote_uid, remote_uid = excluded.remote_uid,
local_mtime = excluded.local_mtime, local_mtime = excluded.local_mtime,
local_size = excluded.local_size, local_size = excluded.local_size,
remote_rev = excluded.remote_rev, remote_rev = excluded.remote_rev,
remote_hash = excluded.remote_hash", remote_hash = excluded.remote_hash,
local_mtime_ns = excluded.local_mtime_ns",
params![ params![
folder_id, folder_id,
e.rel_path, e.rel_path,
@@ -195,6 +244,7 @@ impl Db {
e.local_size, e.local_size,
e.remote_rev, e.remote_rev,
e.remote_hash, e.remote_hash,
e.local_mtime_ns,
], ],
)?; )?;
Ok(()) Ok(())
File diff suppressed because it is too large. Load diff
+46 -4
View File
@@ -1,5 +1,7 @@
//! Trashed nodes, persisted so the trash view opens without a round-trip. //! Trashed nodes, persisted so the trash view opens without a round-trip.
use std::collections::HashMap;
use rusqlite::params; use rusqlite::params;
use super::Db; use super::Db;
@@ -11,6 +13,13 @@ pub struct StoredTrash {
pub is_dir: bool, pub is_dir: bool,
pub size: i64, pub size: i64,
pub mtime: i64, pub mtime: i64,
/// The folder this node was trashed from, when it is known. `None` for a
/// row written before schema v29, and for a node whose parent the server
/// did not report.
pub parent_uid: Option<String>,
/// When the daemon first saw the node in the trash, epoch seconds. `None`
/// when it cannot say (see schema v35).
pub trashed_at: Option<i64>,
} }
impl Db { impl Db {
@@ -20,7 +29,8 @@ impl Db {
tx.execute("DELETE FROM trash", [])?; tx.execute("DELETE FROM trash", [])?;
{ {
let mut stmt = tx.prepare( let mut stmt = tx.prepare(
"INSERT INTO trash (uid, name, is_dir, size, mtime) VALUES (?1, ?2, ?3, ?4, ?5)", "INSERT INTO trash (uid, name, is_dir, size, mtime, parent_uid, trashed_at) \
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)",
)?; )?;
for item in items { for item in items {
stmt.execute(params![ stmt.execute(params![
@@ -28,7 +38,9 @@ impl Db {
item.name, item.name,
item.is_dir as i64, item.is_dir as i64,
item.size, item.size,
item.mtime item.mtime,
item.parent_uid,
item.trashed_at
])?; ])?;
} }
} }
@@ -39,9 +51,9 @@ impl Db {
/// The persisted trash listing, folders first then by name — the order the /// The persisted trash listing, folders first then by name — the order the
/// Trash page shows it in. /// Trash page shows it in.
pub fn trash_list(&self) -> Result<Vec<StoredTrash>> { pub fn trash_list(&self) -> Result<Vec<StoredTrash>> {
let conn = self.conn.lock(); let conn = self.read();
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
"SELECT uid, name, is_dir, size, mtime FROM trash "SELECT uid, name, is_dir, size, mtime, parent_uid, trashed_at FROM trash
ORDER BY is_dir DESC, name COLLATE NOCASE", ORDER BY is_dir DESC, name COLLATE NOCASE",
)?; )?;
let rows = stmt.query_map([], |r| { let rows = stmt.query_map([], |r| {
@@ -51,6 +63,8 @@ impl Db {
is_dir: r.get::<_, i64>(2)? != 0, is_dir: r.get::<_, i64>(2)? != 0,
size: r.get(3)?, size: r.get(3)?,
mtime: r.get(4)?, mtime: r.get(4)?,
parent_uid: r.get(5)?,
trashed_at: r.get(6)?,
}) })
})?; })?;
let mut items = Vec::new(); let mut items = Vec::new();
@@ -60,5 +74,33 @@ impl Db {
Ok(items) Ok(items)
} }
/// Every trashed node paired with the folder it was trashed from, for the
/// restore's tree walk. Kept separate from [`Db::trash_list`] because the
/// restore wants only the links, not the names, sizes or ordering.
pub fn trash_parents(&self) -> Result<Vec<(String, Option<String>)>> {
let conn = self.read();
let mut stmt = conn.prepare("SELECT uid, parent_uid FROM trash")?;
let rows = stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))?;
let mut pairs = Vec::new();
for row in rows {
pairs.push(row?);
}
Ok(pairs)
}
/// When each node now in the trash was first seen there, keyed by uid, so
/// a refresh can carry the times over into the listing it writes.
pub fn trash_seen(&self) -> Result<HashMap<String, Option<i64>>> {
let conn = self.read();
let mut stmt = conn.prepare("SELECT uid, trashed_at FROM trash")?;
let rows = stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))?;
let mut seen = HashMap::new();
for row in rows {
let (uid, at) = row?;
seen.insert(uid, at);
}
Ok(seen)
}
// ---- device sync (devices.md) ----------------------------------------- // ---- device sync (devices.md) -----------------------------------------
} }
+57 -13
View File
@@ -1,6 +1,6 @@
//! Small query helpers shared across the table modules. //! Small query helpers shared across the table modules.
use rusqlite::{Connection, params}; use rusqlite::{Connection, OptionalExtension, params};
use crate::Result; use crate::Result;
@@ -8,14 +8,19 @@ use crate::Result;
/// grams), so short queries fall back to a `LIKE` scan over `nodes.name`. /// grams), so short queries fall back to a `LIKE` scan over `nodes.name`.
pub(super) const TRIGRAM_MIN: usize = 3; pub(super) const TRIGRAM_MIN: usize = 3;
pub(super) fn pair(row: &rusqlite::Row) -> rusqlite::Result<(String, String)> { /// One raw search row: the node's JSON, its uid, and its stored path — `None`
Ok((row.get(0)?, row.get(1)?)) /// only for a row written before the `path` column existed, which
/// [`path_of`] repairs by walking.
pub(super) type HitRow = (String, String, Option<String>);
pub(super) fn hit_row(row: &rusqlite::Row) -> rusqlite::Result<HitRow> {
Ok((row.get(0)?, row.get(1)?, row.get(2)?))
} }
/// Drain a `query_map` of [`pair`] rows into a `Vec`, propagating row errors. /// Drain a `query_map` of [`hit_row`] rows into a `Vec`, propagating row errors.
pub(super) fn collect_pairs( pub(super) fn collect_hits(
rows: impl Iterator<Item = rusqlite::Result<(String, String)>>, rows: impl Iterator<Item = rusqlite::Result<HitRow>>,
) -> Result<Vec<(String, String)>> { ) -> Result<Vec<HitRow>> {
let mut out = Vec::new(); let mut out = Vec::new();
for r in rows { for r in rows {
out.push(r?); out.push(r?);
@@ -31,19 +36,58 @@ pub(super) fn like_escape(s: &str) -> String {
.replace('_', "\\_") .replace('_', "\\_")
} }
/// Resolve a node's mountpoint-relative path by walking `parent_uid` to the /// How deep the parent walk may go before it gives up. Real Drive trees are
/// root via a recursive CTE. The root (the node with no parent) is excluded, so /// nowhere near this; a parent *cycle* is unbounded, and this query is on the
/// a top-level file `report.pdf` yields `"report.pdf"`, not `"My Files/report.pdf"`. /// path of every search result, including the `LIKE` lane that reads `nodes`
/// directly and so never passed through the index's cycle check.
pub(super) const MAX_PATH_DEPTH: usize = 256;
/// Append `name` to a parent's stored path. An empty parent path is the root,
/// whose name is the mount and so contributes nothing.
pub(super) fn join_path(parent_path: &str, name: &str) -> String {
if parent_path.is_empty() {
name.to_string()
} else {
format!("{parent_path}/{name}")
}
}
/// Resolve a node's mountpoint-relative path. The root (the node with no
/// parent) is excluded, so a top-level file `report.pdf` yields
/// `"report.pdf"`, not `"My Files/report.pdf"`.
///
/// Reads the stored `nodes.path`, which the write path maintains (see
/// `upsert_node_tx`). Falls back to [`walk_path_of`] for a row that predates the
/// column or was never written through — a search serving a keystroke used to
/// run that walk once per candidate, which is the reason the column exists.
pub(super) fn path_of(conn: &Connection, uid: &str) -> Result<String> { pub(super) fn path_of(conn: &Connection, uid: &str) -> Result<String> {
let mut stmt = conn.prepare( let stored: Option<Option<String>> = conn
.query_row("SELECT path FROM nodes WHERE uid = ?1", params![uid], |r| {
r.get(0)
})
.optional()?;
match stored.flatten() {
Some(path) => Ok(path),
None => walk_path_of(conn, uid),
}
}
/// Resolve a path by walking `parent_uid` to the root via a recursive CTE.
///
/// A cycle in `parent_uid` is corrupt data the API can still hand us, and
/// `UNION ALL` over one never terminates — the walk is therefore depth-capped
/// and returns the truncated path rather than hanging the caller.
pub(super) fn walk_path_of(conn: &Connection, uid: &str) -> Result<String> {
let mut stmt = conn.prepare(&format!(
"WITH RECURSIVE anc(uid, parent_uid, name, depth) AS ( "WITH RECURSIVE anc(uid, parent_uid, name, depth) AS (
SELECT uid, parent_uid, name, 0 FROM nodes WHERE uid = ?1 SELECT uid, parent_uid, name, 0 FROM nodes WHERE uid = ?1
UNION ALL UNION ALL
SELECT n.uid, n.parent_uid, n.name, anc.depth + 1 SELECT n.uid, n.parent_uid, n.name, anc.depth + 1
FROM nodes n JOIN anc ON n.uid = anc.parent_uid FROM nodes n JOIN anc ON n.uid = anc.parent_uid
WHERE anc.depth < {MAX_PATH_DEPTH}
) )
SELECT name, parent_uid FROM anc ORDER BY depth DESC", SELECT name, parent_uid FROM anc ORDER BY depth DESC"
)?; ))?;
let rows = stmt.query_map(params![uid], |r| { let rows = stmt.query_map(params![uid], |r| {
Ok((r.get::<_, String>(0)?, r.get::<_, Option<String>>(1)?)) Ok((r.get::<_, String>(0)?, r.get::<_, Option<String>>(1)?))
})?; })?;
+36 -1
View File
@@ -11,7 +11,7 @@ pub enum Error {
Proton(#[from] proton_sdk::ProtonError), Proton(#[from] proton_sdk::ProtonError),
#[error("keyring: {0}")] #[error("keyring: {0}")]
Keyring(#[from] keyring::Error), Keyring(#[from] keyring_core::Error),
#[error("serialization: {0}")] #[error("serialization: {0}")]
Serde(#[from] serde_json::Error), Serde(#[from] serde_json::Error),
@@ -45,6 +45,41 @@ pub enum Error {
Other(String), Other(String),
} }
impl Error {
/// Whether this is the local disk being full (`ENOSPC`), as opposed to any
/// other I/O failure.
///
/// Worth distinguishing because it is the one storage failure the user can
/// act on, and because the caller's response differs: a full disk is met by
/// evicting cached blobs and answering `ENOSPC`, where `EIO` would read as a
/// broken file. `std::io::ErrorKind::StorageFull` is still unstable, so this
/// compares the raw code.
pub fn is_disk_full(&self) -> bool {
match self {
Error::Io(e) => e.raw_os_error() == Some(libc::ENOSPC),
// SQLite reports it as its own code rather than surfacing the errno.
Error::Db(rusqlite::Error::SqliteFailure(e, _)) => {
e.code == rusqlite::ErrorCode::DiskFull
}
_ => false,
}
}
}
impl Error {
/// Whether a control-socket round-trip gave up waiting for the reply: the
/// daemon is running but stuck, as opposed to not running at all.
pub fn is_timeout(&self) -> bool {
matches!(
self,
Error::Io(e) if matches!(
e.kind(),
std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut
)
)
}
}
pub type Result<T> = std::result::Result<T, Error>; pub type Result<T> = std::result::Result<T, Error>;
/// A failure on its way to a front-end: prose for the user, plus the /// A failure on its way to a front-end: prose for the user, plus the
+11
View File
@@ -2,19 +2,30 @@
//! identity, XDG paths, and the login/session/keyring layer that every //! identity, XDG paths, and the login/session/keyring layer that every
//! front-end (CLI today, daemon + GUI later) shares. //! front-end (CLI today, daemon + GUI later) shares.
pub mod access;
pub mod auth; pub mod auth;
pub mod batch;
pub mod cache; pub mod cache;
pub mod config; pub mod config;
pub mod control; pub mod control;
pub mod db; pub mod db;
pub mod error; pub mod error;
pub mod localindex; pub mod localindex;
pub mod menu;
pub mod mounts;
pub mod opener;
pub mod places;
pub mod profile; pub mod profile;
pub mod sdkcache;
pub mod search; pub mod search;
pub mod service; pub mod service;
pub mod shell; pub mod shell;
pub mod similar;
pub mod syncignore; pub mod syncignore;
pub mod takeout;
pub mod tray;
pub use access::{Access, access_for, perm_bits};
pub use error::{CoreError, CoreResult, Error, Result}; pub use error::{CoreError, CoreResult, Error, Result};
// Re-export the SDK surface downstream crates need, so they depend on // Re-export the SDK surface downstream crates need, so they depend on
+149 -1
View File
@@ -26,10 +26,20 @@ const SKIP_DIRS: &[&str] = &[
"venv", "venv",
".venv", ".venv",
"vendor", "vendor",
"dist",
"dist-newstyle", "dist-newstyle",
"build",
"Trash", "Trash",
]; ];
/// Directory *paths* skipped when they end with one of these, for caches whose
/// own name is too generic to blacklist (`mod`). The Go module cache was 63,399
/// of the 110,187 entries indexed on the machine this was measured on — 58% of
/// the index, crowding real files out of the bounded candidate pool the fuzzy
/// scorer draws from. The other big dependency caches (`.cargo`, `.rustup`,
/// `.gradle`, `.m2`) are already skipped by the hidden-file filter.
const SKIP_PATH_SUFFIXES: &[&str] = &["go/pkg/mod"];
/// Hard cap on indexed entries, so a pathological home directory cannot grow the /// Hard cap on indexed entries, so a pathological home directory cannot grow the
/// database without bound. Reaching it stops the walk early. /// database without bound. Reaching it stops the walk early.
const MAX_ENTRIES: usize = 400_000; const MAX_ENTRIES: usize = 400_000;
@@ -90,7 +100,10 @@ pub fn scan(
if excludes.iter().any(|e| path.starts_with(e)) { if excludes.iter().any(|e| path.starts_with(e)) {
return false; return false;
} }
!matches!(entry.file_name().to_str(), Some(name) if SKIP_DIRS.contains(&name)) if matches!(entry.file_name().to_str(), Some(name) if SKIP_DIRS.contains(&name)) {
return false;
}
!is_skipped_path(path)
}); });
let state = parking_lot::Mutex::new(SinkState { let state = parking_lot::Mutex::new(SinkState {
@@ -150,6 +163,17 @@ impl<F: FnMut(Vec<LocalEntry>)> SinkState<F> {
} }
} }
/// Whether `path` is (or is inside) one of the [`SKIP_PATH_SUFFIXES`] caches.
/// Matched on the path's own components so a directory merely *named* `mod`
/// keeps being indexed.
fn is_skipped_path(path: &Path) -> bool {
let path = path.to_string_lossy();
SKIP_PATH_SUFFIXES.iter().any(|suffix| {
path.strip_suffix(suffix)
.is_some_and(|head| head.is_empty() || head.ends_with('/'))
})
}
/// Convert a walker entry into a [`LocalEntry`], reusing the metadata the walker /// Convert a walker entry into a [`LocalEntry`], reusing the metadata the walker
/// already fetched. Entries with non-UTF-8 paths are dropped: the index (and the /// already fetched. Entries with non-UTF-8 paths are dropped: the index (and the
/// JSON wire format) is UTF-8 only. /// JSON wire format) is UTF-8 only.
@@ -175,6 +199,9 @@ fn to_entry(entry: &ignore::DirEntry) -> Option<LocalEntry> {
/// The paths a scan should never descend into: the Drive mountpoint (walking it /// The paths a scan should never descend into: the Drive mountpoint (walking it
/// would fault every remote node in through FUSE) plus our own state and cache /// would fault every remote node in through FUSE) plus our own state and cache
/// dirs, which hold blobs no user searches for by name. /// dirs, which hold blobs no user searches for by name.
///
/// Callers should extend this with [`nested_mount_points`] for each scan root —
/// see that function for why `same_file_system(true)` is not enough on its own.
pub fn default_excludes(mountpoint: &Path, state_dir: &Path, cache_dir: &Path) -> Vec<PathBuf> { pub fn default_excludes(mountpoint: &Path, state_dir: &Path, cache_dir: &Path) -> Vec<PathBuf> {
vec![ vec![
mountpoint.to_path_buf(), mountpoint.to_path_buf(),
@@ -183,6 +210,66 @@ pub fn default_excludes(mountpoint: &Path, state_dir: &Path, cache_dir: &Path) -
] ]
} }
/// Every mountpoint strictly below `root`, read from `/proc/self/mounts`.
///
/// [`scan`] already asks the walker to stay on one filesystem, but that check is
/// a *stat of the mountpoint* (`ignore`'s `is_same_file_system`), performed after
/// the directory is queued — so the walker has to touch a foreign mount to learn
/// it should skip it. On a dead network FUSE mount (an sshfs whose server is
/// gone, an unresponsive rclone) that stat blocks in `request_wait_answer` with
/// nobody left to answer it, and blocks *uninterruptibly*: the walker thread then
/// ignores SIGKILL, so the daemon can never exit, systemd's stop times out, and
/// the half-dead process keeps the cache.db lock that the next start needs
/// (docs/BUGS.md B90).
///
/// Excluding the mountpoints by path avoids that stat entirely. It does not
/// change what ends up indexed: a nested mount is a different device, so
/// `same_file_system(true)` was already going to skip it.
pub fn nested_mount_points(root: &Path) -> Vec<PathBuf> {
let Ok(mounts) = std::fs::read_to_string("/proc/self/mounts") else {
return Vec::new();
};
parse_nested_mount_points(&mounts, root)
}
/// The mountpoint-field parse behind [`nested_mount_points`], split out so it can
/// be tested without a real `/proc`.
fn parse_nested_mount_points(mounts: &str, root: &Path) -> Vec<PathBuf> {
mounts
.lines()
.filter_map(|line| line.split_whitespace().nth(1))
.map(|field| PathBuf::from(unescape_mount_field(field)))
// Strictly below: `root` itself being a mountpoint must not exclude the
// whole scan.
.filter(|path| path.starts_with(root) && path != root)
.collect()
}
/// Undo the octal escaping `/proc/self/mounts` applies to space, tab, newline and
/// backslash in path fields. An un-unescaped `\040` would simply fail to match a
/// real path, silently putting a mount with a space in its name back in the walk.
fn unescape_mount_field(field: &str) -> String {
let mut out = String::with_capacity(field.len());
let mut rest = field;
while let Some(at) = rest.find('\\') {
out.push_str(&rest[..at]);
let escape = rest[at + 1..].get(..3);
match escape.and_then(|digits| u8::from_str_radix(digits, 8).ok()) {
Some(byte) => {
out.push(byte as char);
rest = &rest[at + 4..];
}
// Not an octal escape after all — keep the backslash verbatim.
None => {
out.push('\\');
rest = &rest[at + 1..];
}
}
}
out.push_str(rest);
out
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
@@ -221,4 +308,65 @@ mod tests {
std::fs::remove_dir_all(&tmp).unwrap(); std::fs::remove_dir_all(&tmp).unwrap();
} }
/// The Go module cache is the single largest source of index noise on a
/// developer's machine (58% of the entries on the one this was measured
/// on), and it cannot be pruned by directory name: `mod` is far too
/// common a name to blacklist outright.
#[test]
fn the_go_module_cache_is_pruned_but_a_plain_mod_dir_is_not() {
let tmp = std::env::temp_dir().join(format!("pdfs-scan-gomod-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&tmp);
std::fs::create_dir_all(tmp.join("go/pkg/mod/example.com/lib")).unwrap();
std::fs::create_dir_all(tmp.join("kernel/mod")).unwrap();
std::fs::write(tmp.join("go/pkg/mod/example.com/lib/vendored.go"), b"x").unwrap();
std::fs::write(tmp.join("kernel/mod/mine.conf"), b"x").unwrap();
let mut got = Vec::new();
scan(std::slice::from_ref(&tmp), &[], |batch| got.extend(batch));
let names: Vec<&str> = got.iter().map(|e| e.name.as_str()).collect();
assert!(!names.contains(&"vendored.go"), "{names:?}");
assert!(names.contains(&"mine.conf"), "{names:?}");
std::fs::remove_dir_all(&tmp).unwrap();
}
/// A dead network FUSE mount under the scan root must be skipped by *path*,
/// before anything stats it — see `nested_mount_points`. The scan root itself
/// being a mountpoint must not exclude the entire walk.
#[test]
fn nested_mount_points_are_excluded_but_the_root_mount_is_not() {
let mounts = "\
/dev/sda2 /home ext4 rw 0 0
narl@narl.io:/opt /home/narl/remote/narl.io fuse.sshfs rw 0 0
google_drive: /home/narl/remote/gdrive fuse.rclone rw 0 0
tmpfs /run/user/1000 tmpfs rw 0 0
";
let nested = parse_nested_mount_points(mounts, Path::new("/home/narl"));
assert_eq!(
nested,
vec![
PathBuf::from("/home/narl/remote/narl.io"),
PathBuf::from("/home/narl/remote/gdrive"),
]
);
// `/home` is an ancestor, not a descendant: excluding it would empty the
// scan. `/run/user/1000` is unrelated.
assert!(parse_nested_mount_points(mounts, Path::new("/home")).len() == 2);
}
/// `/proc/self/mounts` octal-escapes a space; leaving it escaped would put a
/// mount whose path contains one back into the walk.
#[test]
fn mount_paths_are_unescaped() {
let mounts = "srv:/x /home/narl/My\\040Cloud fuse.sshfs rw 0 0\n";
assert_eq!(
parse_nested_mount_points(mounts, Path::new("/home/narl")),
vec![PathBuf::from("/home/narl/My Cloud")]
);
assert_eq!(unescape_mount_field("/a\\134b"), "/a\\b");
assert_eq!(unescape_mount_field("/plain/path"), "/plain/path");
}
} }
+529
View File
@@ -0,0 +1,529 @@
//! Driving an external dmenu-style launcher (fuzzel, rofi, wofi, …).
//!
//! `pdfs-prompt` ships its own GTK HUD, but on a tiling-WM desktop the user
//! already has a launcher bound to a key, themed to match everything else. This
//! module lets the prompt borrow it: entries go out on the child's stdin, the
//! chosen line comes back on stdout.
//!
//! Two things vary between launchers and are the only reasons this is not four
//! lines of `Command`:
//!
//! * the flag that turns on filter mode and the one that sets the prompt text;
//! * icon support, which fuzzel and rofi both spell `text\0icon\x1fNAME` and
//! everything else would render as literal garbage.
//!
//! Selection is matched back by label rather than by index, because the one
//! feature no launcher agrees on is index output. A line that matches nothing is
//! the user's own typing — [`MenuChoice::Custom`] — which is what makes
//! type-then-search work at all.
use std::io::{BufRead, BufReader, Write};
use std::path::Path;
use std::process::{Command, Stdio};
use serde::{Deserialize, Serialize};
use crate::error::{Error, Result};
/// Separates the label from its icon name in the fuzzel/rofi protocol.
const ICON_SEPARATOR: &str = "\0icon\x1f";
/// Launchers we know how to invoke, most-preferred first:
/// `(program, filter-mode args, prompt flag, icon protocol)`.
///
/// The prompt flag is written as it is passed: fuzzel wants `--prompt=TEXT` as
/// one argument, the rest take the text as a separate one.
const KNOWN_MENUS: [MenuFlavor; 6] = [
MenuFlavor {
program: "fuzzel",
mode: &["--dmenu"],
prompt: PromptFlag::Joined("--prompt="),
icons: true,
},
MenuFlavor {
program: "rofi",
mode: &["-dmenu"],
prompt: PromptFlag::Separate("-p"),
icons: true,
},
MenuFlavor {
program: "wofi",
mode: &["--dmenu"],
prompt: PromptFlag::Separate("--prompt"),
icons: false,
},
MenuFlavor {
program: "tofi",
mode: &[],
prompt: PromptFlag::Joined("--prompt-text="),
icons: false,
},
MenuFlavor {
program: "bemenu",
mode: &[],
prompt: PromptFlag::Separate("-p"),
icons: false,
},
MenuFlavor {
program: "dmenu",
mode: &[],
prompt: PromptFlag::Separate("-p"),
icons: false,
},
];
/// Terminals that can host `pdfs-prompt --fzf`, most-preferred first.
///
/// The fzf front end is a terminal program, so a keybinding that has no tty
/// needs one spawned for it. Each entry sets a stable app-id/class
/// ([`TERMINAL_APP_ID`]) so a tiling WM can be told to float exactly this
/// window, and ends with the flag after which the command to run follows.
const KNOWN_TERMINALS: [&[&str]; 6] = [
&["foot", "--app-id=pdfs-prompt", "--"],
&["ghostty", "--class=pdfs-prompt", "-e"],
&["kitty", "--class", "pdfs-prompt"],
&["alacritty", "--class", "pdfs-prompt", "-e"],
&["wezterm", "start", "--class", "pdfs-prompt", "--"],
&["xterm", "-class", "pdfs-prompt", "-e"],
];
/// The app-id/class every [`KNOWN_TERMINALS`] entry sets, so the window rule a
/// user writes for one terminal keeps working if they switch.
pub const TERMINAL_APP_ID: &str = "pdfs-prompt";
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum PromptFlag {
/// `--prompt=TEXT`
Joined(&'static str),
/// `-p TEXT`
Separate(&'static str),
}
#[derive(Debug, Clone, Copy)]
struct MenuFlavor {
program: &'static str,
mode: &'static [&'static str],
prompt: PromptFlag,
icons: bool,
}
/// Prompt behaviour from `config.json`, under `prompt`.
///
/// ```json
/// "prompt": { "menu": ["fuzzel", "--dmenu", "--width=60"], "menu_limit": 40 }
/// ```
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct PromptConfig {
/// Which front end `pdfs-prompt` uses when invoked with no mode flag, so a
/// desktop shortcut does not have to be re-bound to switch. `None` is
/// [`PromptMode::Gtk`], the built-in HUD.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub mode: Option<PromptMode>,
/// Launcher argv for `pdfs-prompt --dmenu`. Unset means "find one" — see
/// [`resolve_menu`]. A `{prompt}` token anywhere in the argv is replaced by
/// the prompt text; without one the launcher's own prompt flag is appended.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub menu: Option<Vec<String>>,
/// Terminal argv for `pdfs-prompt --fzf` when it is started without a tty.
/// Unset means "find one" — see [`resolve_terminal`]. A `{cmd}` token
/// anywhere in the argv is replaced by the command to run; without one the
/// command is appended, which is what every [`KNOWN_TERMINALS`] entry
/// expects.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub terminal: Option<Vec<String>>,
/// How many results to feed the launcher. Unset means
/// [`DEFAULT_MENU_LIMIT`].
#[serde(default, skip_serializing_if = "Option::is_none")]
pub menu_limit: Option<usize>,
/// Show icons when the launcher understands them. Defaults to true; set
/// false for a launcher that advertises support but is themed without them.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub menu_icons: Option<bool>,
}
/// Which front end the prompt presents.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum PromptMode {
/// The built-in GTK HUD.
#[default]
Gtk,
/// The user's own launcher, driven over stdin/stdout.
Dmenu,
/// `fzf` in a terminal, re-querying the daemon on every keystroke.
Fzf,
}
/// Rows an external launcher can show without becoming a scroll exercise. Larger
/// than the GTK prompt's cap because filtering happens launcher-side.
pub const DEFAULT_MENU_LIMIT: usize = 50;
impl PromptConfig {
pub fn resolved_mode(&self) -> PromptMode {
self.mode.unwrap_or_default()
}
pub fn resolved_menu_limit(&self) -> usize {
self.menu_limit.unwrap_or(DEFAULT_MENU_LIMIT)
}
pub fn resolved_menu_icons(&self) -> bool {
self.menu_icons.unwrap_or(true)
}
}
/// One line offered to the launcher.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MenuItem {
/// What the user sees, and what identifies the choice on the way back.
pub label: String,
/// Icon theme name, used only by launchers that support the protocol.
pub icon: Option<String>,
}
impl MenuItem {
pub fn new(label: impl Into<String>, icon: Option<String>) -> Self {
Self {
label: label.into(),
icon,
}
}
}
/// What came back from the launcher.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum MenuChoice {
/// Index into the items that were passed in.
Item(usize),
/// Text the user typed that matched no item — treat it as a query.
Custom(String),
/// Escape, or an empty selection.
Cancelled,
}
/// The launcher to run: the configured one, or the first [`KNOWN_MENUS`] entry
/// on `PATH`. Returns the argv as it should be spawned, filter-mode flags
/// included.
pub fn resolve_menu(configured: Option<&Vec<String>>) -> Option<Vec<String>> {
if let Some(argv) = configured.filter(|argv| !argv.is_empty()) {
return Some(argv.clone());
}
let flavor = KNOWN_MENUS.iter().find(|f| on_path(f.program))?;
let mut argv = vec![flavor.program.to_string()];
argv.extend(flavor.mode.iter().map(|f| (*f).to_string()));
Some(argv)
}
/// The terminal to host `pdfs-prompt --fzf` in: the configured one, or the
/// first [`KNOWN_TERMINALS`] entry on `PATH`. Returns the argv up to and
/// including the "run this command" flag — the caller appends the command, or
/// substitutes it for a `{cmd}` token, via [`with_command`].
pub fn resolve_terminal(configured: Option<&Vec<String>>) -> Option<Vec<String>> {
if let Some(argv) = configured.filter(|argv| !argv.is_empty()) {
return Some(argv.clone());
}
let flavor = KNOWN_TERMINALS
.iter()
.find(|argv| argv.first().is_some_and(|program| on_path(program)))?;
Some(flavor.iter().map(|arg| (*arg).to_string()).collect())
}
/// Place `command` into a terminal argv: substituted for a `{cmd}` token if the
/// user wrote one, otherwise appended.
///
/// The token exists for terminals that want the command in the middle of their
/// own arguments; appending is right for every [`KNOWN_TERMINALS`] entry, whose
/// last element is the flag the command follows.
pub fn with_command(argv: &[String], command: &[String]) -> Vec<String> {
if !argv.iter().any(|arg| arg == "{cmd}") {
let mut argv = argv.to_vec();
argv.extend_from_slice(command);
return argv;
}
let mut out = Vec::with_capacity(argv.len() + command.len());
for arg in argv {
if arg == "{cmd}" {
out.extend_from_slice(command);
} else {
out.push(arg.clone());
}
}
out
}
/// Show `items` in the launcher named by `argv` and wait for a choice.
///
/// Blocking by design: the caller is a one-shot CLI invocation whose entire job
/// is this round-trip.
pub fn run(argv: &[String], prompt: &str, items: &[MenuItem]) -> Result<MenuChoice> {
let argv = with_prompt(argv, prompt);
let Some((program, args)) = argv.split_first() else {
return Err(Error::Other("no launcher command configured".into()));
};
let icons = supports_icons(program);
let mut child = Command::new(program)
.args(args)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn()
.map_err(|e| Error::Other(format!("cannot run launcher {program}: {e}")))?;
// Write on a worker: a launcher that starts reading only after it maps its
// window would otherwise deadlock against a full pipe buffer.
let payload = render(items, icons);
let mut stdin = child
.stdin
.take()
.ok_or_else(|| Error::Other("launcher stdin unavailable".into()))?;
let writer = std::thread::spawn(move || {
let _ = stdin.write_all(payload.as_bytes());
let _ = stdin.flush();
});
let mut selected = String::new();
if let Some(stdout) = child.stdout.take() {
// Only the first line matters; multi-select is not a thing here, and a
// launcher that emits more must not leave us blocked on a full pipe.
let mut reader = BufReader::new(stdout);
let mut line = String::new();
while reader.read_line(&mut line).unwrap_or(0) > 0 {
if selected.is_empty() {
selected = line.trim_end_matches('\n').to_string();
}
line.clear();
}
}
let _ = child.wait();
let _ = writer.join();
Ok(interpret(&selected, items))
}
/// Map the launcher's output back to a choice. Exposed for tests; the ambiguity
/// it resolves (label vs. free text) is the whole contract with the launcher.
fn interpret(selected: &str, items: &[MenuItem]) -> MenuChoice {
// Some launchers echo the icon protocol back verbatim.
let selected = selected
.split(ICON_SEPARATOR)
.next()
.unwrap_or(selected)
.trim_end();
if selected.is_empty() {
return MenuChoice::Cancelled;
}
match items.iter().position(|item| item.label == selected) {
Some(index) => MenuChoice::Item(index),
None => MenuChoice::Custom(selected.to_string()),
}
}
/// The stdin payload. Labels are stripped of newlines rather than escaped —
/// there is no escape in this protocol, and a file name with a newline in it
/// would otherwise become two entries.
fn render(items: &[MenuItem], icons: bool) -> String {
let mut out = String::new();
for item in items {
let label = item.label.replace(['\n', '\r'], " ");
out.push_str(&label);
if icons && let Some(icon) = &item.icon {
out.push_str(ICON_SEPARATOR);
out.push_str(icon);
}
out.push('\n');
}
out
}
/// Add the prompt text: substituted for `{prompt}` if the user placed one,
/// otherwise appended using the launcher's own flag. An unrecognised launcher
/// gets no prompt rather than a guessed flag that might not parse.
fn with_prompt(argv: &[String], prompt: &str) -> Vec<String> {
if argv.iter().any(|arg| arg.contains("{prompt}")) {
return argv
.iter()
.map(|arg| arg.replace("{prompt}", prompt))
.collect();
}
let mut argv = argv.to_vec();
if prompt.is_empty() {
return argv;
}
if let Some(flavor) = argv.first().and_then(|program| flavor_of(program)) {
match flavor.prompt {
PromptFlag::Joined(flag) => argv.push(format!("{flag}{prompt}")),
PromptFlag::Separate(flag) => {
argv.push(flag.to_string());
argv.push(prompt.to_string());
}
}
}
argv
}
fn supports_icons(program: &str) -> bool {
flavor_of(program).is_some_and(|flavor| flavor.icons)
}
fn flavor_of(program: &str) -> Option<&'static MenuFlavor> {
let base = Path::new(program).file_name()?.to_str()?;
KNOWN_MENUS.iter().find(|flavor| flavor.program == base)
}
/// Whether `program` is an executable file on `PATH`.
pub fn on_path(program: &str) -> bool {
let Some(path) = std::env::var_os("PATH") else {
return false;
};
std::env::split_paths(&path).any(|dir| dir.join(program).is_file())
}
#[cfg(test)]
mod tests {
use super::*;
fn items() -> Vec<MenuItem> {
vec![
MenuItem::new(
"notes.md · My files / Docs",
Some("text-x-generic".into()),
),
MenuItem::new("photo.png · Home / Pictures", None),
]
}
#[test]
fn a_matching_line_is_an_item_and_anything_else_is_a_query() {
let items = items();
assert_eq!(
interpret("notes.md · My files / Docs", &items),
MenuChoice::Item(0)
);
assert_eq!(
interpret("invoice", &items),
MenuChoice::Custom("invoice".into())
);
assert_eq!(interpret("", &items), MenuChoice::Cancelled);
}
#[test]
fn an_echoed_icon_field_does_not_break_the_match() {
let items = items();
assert_eq!(
interpret(
"notes.md · My files / Docs\0icon\x1ftext-x-generic",
&items
),
MenuChoice::Item(0)
);
}
#[test]
fn icons_are_only_rendered_for_launchers_that_understand_them() {
let items = items();
assert_eq!(
render(&items, true),
"notes.md · My files / Docs\0icon\x1ftext-x-generic\nphoto.png · Home / Pictures\n"
);
assert_eq!(
render(&items, false),
"notes.md · My files / Docs\nphoto.png · Home / Pictures\n"
);
assert!(supports_icons("fuzzel"));
assert!(supports_icons("/usr/bin/rofi"));
assert!(!supports_icons("wofi"));
assert!(!supports_icons("mystery-menu"));
}
#[test]
fn a_label_with_a_newline_stays_one_entry() {
let sneaky = vec![MenuItem::new("two\nlines.txt", None)];
assert_eq!(render(&sneaky, false), "two lines.txt\n");
}
#[test]
fn the_prompt_uses_each_launchers_own_flag() {
assert_eq!(
with_prompt(&["fuzzel".into(), "--dmenu".into()], "Drive"),
vec!["fuzzel", "--dmenu", "--prompt=Drive"]
);
assert_eq!(
with_prompt(&["rofi".into(), "-dmenu".into()], "Drive"),
vec!["rofi", "-dmenu", "-p", "Drive"]
);
// Unknown launcher: no guessed flag.
assert_eq!(
with_prompt(&["mystery-menu".into()], "Drive"),
vec!["mystery-menu"]
);
}
#[test]
fn an_explicit_placeholder_wins_over_the_flag() {
assert_eq!(
with_prompt(
&[
"rofi".into(),
"-dmenu".into(),
"-p".into(),
"{prompt} >".into()
],
"Drive"
),
vec!["rofi", "-dmenu", "-p", "Drive >"]
);
}
#[test]
fn a_configured_terminal_is_used_verbatim() {
let configured = vec!["my-term".to_string(), "-e".to_string()];
assert_eq!(
resolve_terminal(Some(&configured)),
Some(configured.clone())
);
// An empty list is "not configured", not "run nothing".
assert_eq!(
resolve_terminal(Some(&Vec::new())),
resolve_terminal(None),
"an empty terminal list must fall back to detection"
);
}
#[test]
fn the_command_is_appended_unless_the_user_placed_it() {
let command = vec!["pdfs-prompt".to_string(), "--fzf".to_string()];
assert_eq!(
with_command(&["foot".into(), "--".into()], &command),
vec!["foot", "--", "pdfs-prompt", "--fzf"]
);
assert_eq!(
with_command(
&["term".into(), "-e".into(), "{cmd}".into(), "--hold".into()],
&command
),
vec!["term", "-e", "pdfs-prompt", "--fzf", "--hold"]
);
}
#[test]
fn every_known_terminal_sets_the_same_app_id() {
for argv in KNOWN_TERMINALS {
assert!(
argv.iter().any(|arg| arg.contains(TERMINAL_APP_ID)),
"{argv:?} must be floatable by one window rule"
);
}
}
#[test]
fn a_configured_launcher_is_used_verbatim() {
let configured = vec!["my-menu".to_string(), "--pick".to_string()];
assert_eq!(resolve_menu(Some(&configured)), Some(configured.clone()));
// An empty list is "not configured", not "run nothing".
assert_eq!(
resolve_menu(Some(&Vec::new())),
resolve_menu(None),
"an empty menu list must fall back to detection"
);
}
}
+149
View File
@@ -0,0 +1,149 @@
//! Unified presentation model for every local Proton Drive location.
//!
//! Device sync state remains owned by `sync_folder`; these types are the stable
//! shape exposed to front-ends after that state is joined with the `mount`
//! presentation table.
use std::fmt;
use serde::{Deserialize, Serialize};
use crate::control::SyncProgress;
/// How a location makes remote content available locally.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq, Default)]
#[serde(rename_all = "lowercase")]
pub enum MountMode {
/// A full local copy reconciled in both directions.
Mirror,
/// A FUSE session that fetches content as it is accessed.
OnDemand,
/// A mode written by a newer client.
#[default]
#[serde(other)]
Unknown,
}
impl MountMode {
pub const fn as_str(self) -> &'static str {
match self {
Self::Mirror => "mirror",
Self::OnDemand => "ondemand",
Self::Unknown => "unknown",
}
}
}
impl From<&str> for MountMode {
fn from(value: &str) -> Self {
match value {
"mirror" => Self::Mirror,
"ondemand" => Self::OnDemand,
_ => Self::Unknown,
}
}
}
impl fmt::Display for MountMode {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(self.as_str())
}
}
/// Whether the location itself was declared writable.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum MountAccess {
Rw,
Ro,
}
impl MountAccess {
pub const fn as_str(self) -> &'static str {
match self {
Self::Rw => "rw",
Self::Ro => "ro",
}
}
}
impl fmt::Display for MountAccess {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(self.as_str())
}
}
/// What remote root a local location presents.
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum MountKind {
/// The account's primary My Files mount.
MyFiles,
/// A folder owned by this machine's Proton Drive device.
Device { sync_folder_id: i64 },
/// A future standalone mount rooted at a node shared with this account.
Shared { share_root_uid: String },
}
/// One local Proton Drive location as exposed over the control protocol.
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct MountSpec {
/// Presentation-row id. Device actions use the `sync_folder_id` carried by
/// [`MountKind::Device`], not this id.
pub id: i64,
pub kind: MountKind,
pub local_path: String,
pub root_uid: String,
pub root_share_id: String,
pub mode: MountMode,
pub access: MountAccess,
/// `idle` | `syncing` | `error` | `conflict` for device locations.
pub state: String,
/// Last successful device sync, epoch seconds; `0` when not applicable.
pub last_sync: i64,
/// A device mode switch accepted but not yet applied.
#[serde(default)]
pub pending_mode: Option<MountMode>,
/// Whether a FUSE session currently owns `local_path`.
#[serde(default)]
pub mounted: bool,
/// Live device sync progress, absent when no pass is running.
#[serde(default)]
pub progress: Option<SyncProgress>,
/// A synced folder paused on its own. Live daemon state, not a stored column.
#[serde(default)]
pub paused: bool,
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn mount_mode_keeps_legacy_wire_strings() {
assert_eq!(
serde_json::to_string(&MountMode::Mirror).unwrap(),
r#""mirror""#
);
assert_eq!(
serde_json::to_string(&MountMode::OnDemand).unwrap(),
r#""ondemand""#
);
assert_eq!(
serde_json::from_str::<MountMode>(r#""mirror""#).unwrap(),
MountMode::Mirror
);
assert_eq!(
serde_json::from_str::<MountMode>(r#""ondemand""#).unwrap(),
MountMode::OnDemand
);
}
#[test]
fn mount_mode_tolerates_unknown_values() {
assert_eq!(
serde_json::from_str::<MountMode>(r#""from-the-future""#).unwrap(),
MountMode::Unknown
);
}
}
+618
View File
@@ -0,0 +1,618 @@
//! How a file chosen in the prompt (or the browser) is actually opened.
//!
//! The default is `xdg-open`, which routes through the desktop's MIME
//! associations. That is the right answer for a GNOME session and the wrong one
//! for a tiling-WM setup where "open a text file" means "run `nvim` inside my
//! terminal" — there is no desktop file for that, and pointing `xdg-open` at a
//! terminal editor globally breaks every graphical caller.
//!
//! So the rules live in `config.json` under `open_with`: an ordered list of
//! name/class patterns mapped to argv, each optionally wrapped in a terminal.
//! First match wins; anything unmatched falls through to `default` (itself
//! `xdg-open` unless overridden). Nothing here is required — an absent
//! `open_with` reproduces the previous behaviour exactly.
//!
//! ```json
//! "open_with": {
//! "terminal": ["alacritty", "-e"],
//! "rules": [
//! { "match": ["@text", "*.rs", "*.toml"], "command": ["nvim"], "terminal": true },
//! { "match": ["*.png"], "command": ["imv"] }
//! ]
//! }
//! ```
use std::ffi::OsStr;
use std::os::unix::process::CommandExt;
use std::path::Path;
use std::process::{Command, Stdio};
use serde::{Deserialize, Serialize};
/// Environment override for the terminal used by `"terminal": true` rules.
/// Checked before `$TERMINAL` so a support case can redirect it without editing
/// config.
pub const TERMINAL_ENV: &str = "PDFS_TERMINAL";
/// Terminals we know how to hand a command line to, most-preferred first, with
/// the argument that means "run this, don't parse it as options". `kitty` and
/// `wezterm start --` take the command positionally.
const KNOWN_TERMINALS: [(&str, &[&str]); 11] = [
("alacritty", &["-e"]),
("foot", &["-e"]),
("kitty", &[]),
("ghostty", &["-e"]),
("wezterm", &["start", "--"]),
("gnome-terminal", &["--"]),
("konsole", &["-e"]),
("xfce4-terminal", &["-x"]),
("urxvt", &["-e"]),
("st", &["-e"]),
("xterm", &["-e"]),
];
/// The opener policy as it appears in `config.json`.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct OpenWith {
/// Terminal argv used by rules with `"terminal": true`, e.g.
/// `["alacritty", "-e"]`. Unset means "resolve one" — see
/// [`resolve_terminal`].
#[serde(default, skip_serializing_if = "Option::is_none")]
pub terminal: Option<Vec<String>>,
/// What to run when no rule matches. Unset means `xdg-open`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub default: Option<Vec<String>>,
/// Ordered; first match wins.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub rules: Vec<OpenRule>,
}
/// One `pattern set -> command` mapping.
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct OpenRule {
/// Patterns this rule answers to. Either a shell-style glob against the file
/// name (`*.md`, `notes-*.txt`) or one of the classes `@dir`, `@text`,
/// `@document`, `@image`, `@media`, `@any`. Matching is case-insensitive.
#[serde(rename = "match", default)]
pub patterns: Vec<String>,
/// argv to run. A `{}` token is replaced by the path; without one the path
/// is appended. `$VAR`/`${VAR}` tokens expand from the environment, so
/// `["$EDITOR"]` follows the user's editor.
#[serde(default)]
pub command: Vec<String>,
/// Wrap [`command`](Self::command) in the terminal emulator.
#[serde(default)]
pub terminal: bool,
}
impl OpenWith {
/// The argv that opens `path`, terminal wrapping included. Never fails: an
/// unmatched path — or a rule whose terminal cannot be resolved — degrades
/// to the default opener rather than doing nothing.
pub fn command_for(&self, path: &Path, is_dir: bool) -> Vec<String> {
let name = path
.file_name()
.and_then(OsStr::to_str)
.unwrap_or_default()
.to_string();
self.command_for_named(path, &name, is_dir)
}
/// [`command_for`](Self::command_for) with the rules matched against `name`
/// instead of the last component of `path`.
///
/// A materialised Drive file lives in the content cache under its content
/// hash — no extension, no original name — so matching the on-disk path
/// classifies every download as "unknown" and hands it to `xdg-open`, which
/// sniffs the bytes and picks whatever claims `text/plain` (LibreOffice, for
/// a Markdown file). The rules are about *what the user opened*, so the
/// caller passes that name and the argv still gets the real path.
pub fn command_for_named(&self, path: &Path, name: &str, is_dir: bool) -> Vec<String> {
for rule in &self.rules {
if rule.command.is_empty() {
continue;
}
if !rule.patterns.iter().any(|p| matches(p, name, is_dir)) {
continue;
}
let argv = expand(&rule.command, path);
if !rule.terminal {
return argv;
}
match self.terminal_argv() {
Some(mut terminal) => {
terminal.extend(argv);
return terminal;
}
// A terminal rule with no terminal to run it in would silently
// launch a TUI with no tty. Fall through to the desktop instead.
None => {
tracing::warn!(
"no terminal found for a terminal open rule; using the default opener"
);
break;
}
}
}
let fallback = self
.default
.clone()
.unwrap_or_else(|| vec!["xdg-open".to_string()]);
expand(&fallback, path)
}
/// The configured terminal, or a resolved one.
fn terminal_argv(&self) -> Option<Vec<String>> {
match &self.terminal {
Some(argv) if !argv.is_empty() => Some(argv.clone()),
_ => resolve_terminal(),
}
}
}
/// Launch `path` under `policy`. Fire-and-forget: the child outlives us, and a
/// spawn failure is logged rather than propagated — every caller is a UI action
/// with nowhere useful to report to.
pub fn open(policy: &OpenWith, path: &Path, is_dir: bool) {
let argv = policy.command_for(path, is_dir);
spawn(&argv, path);
}
/// [`open`] for a path whose own file name says nothing about its type — see
/// [`OpenWith::command_for_named`].
pub fn open_named(policy: &OpenWith, path: &Path, name: &str, is_dir: bool) {
let argv = policy.command_for_named(path, name, is_dir);
spawn(&argv, path);
}
/// Launch `argv`, detached from whatever is opening it.
///
/// The child gets its own process group and no inherited stdio, because the
/// caller may be about to exit and take its terminal with it: `pdfs-prompt
/// --fzf` runs inside a terminal it spawned for itself, and that terminal
/// closes the moment the selection is made. A child left in the terminal's
/// foreground process group is sent `SIGHUP` when the pty hangs up, so
/// "open this folder in a terminal" would open a window that dies immediately
/// — which is exactly what it did. Nothing here needs the caller's stdio in the
/// first place: every `argv` is another application, and a terminal one is
/// wrapped in its own emulator by the `terminal` rule flag.
fn spawn(argv: &[String], path: &Path) {
let Some((program, args)) = argv.split_first() else {
return;
};
tracing::info!(path = %path.display(), command = ?argv, "opening");
let spawned = Command::new(program)
.args(args)
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null())
.process_group(0)
.spawn();
if let Err(e) = spawned {
tracing::error!(command = ?argv, "open failed: {e}");
}
}
/// [`open`] with the policy from the user's `config.json`, read once per
/// process. For callers that open the occasional path and have no reason to
/// carry a policy around; long-lived windows should hold their own copy.
pub fn open_default(path: &Path, is_dir: bool) {
open(default_policy(), path, is_dir);
}
/// [`open_default`] matching the rules against `name` — see
/// [`OpenWith::command_for_named`].
pub fn open_default_named(path: &Path, name: &str, is_dir: bool) {
open_named(default_policy(), path, name, is_dir);
}
fn default_policy() -> &'static OpenWith {
static POLICY: std::sync::OnceLock<OpenWith> = std::sync::OnceLock::new();
POLICY.get_or_init(|| {
crate::config::AppDirs::new()
.map(|dirs| dirs.load_config().resolved_open_with())
.unwrap_or_default()
})
}
/// `$PDFS_TERMINAL`, then `$TERMINAL`, then the first [`KNOWN_TERMINALS`] entry
/// on `PATH`. An env value may be a full command line (`"foot -e"`), in which
/// case it is used verbatim; a bare program name gains that terminal's "run
/// this" argument so `TERMINAL=alacritty` does the obvious thing.
pub fn resolve_terminal() -> Option<Vec<String>> {
for key in [TERMINAL_ENV, "TERMINAL"] {
let Ok(value) = std::env::var(key) else {
continue;
};
let argv: Vec<String> = value.split_whitespace().map(str::to_string).collect();
match argv.len() {
0 => continue,
1 => return Some(with_exec_flag(&argv[0])),
_ => return Some(argv),
}
}
KNOWN_TERMINALS
.iter()
.find(|(program, _)| on_path(program))
.map(|(program, _)| with_exec_flag(program))
}
/// A bare terminal name plus whatever it needs before a command.
fn with_exec_flag(program: &str) -> Vec<String> {
let base = Path::new(program)
.file_name()
.and_then(OsStr::to_str)
.unwrap_or(program);
let mut argv = vec![program.to_string()];
let flags = KNOWN_TERMINALS
.iter()
.find(|(known, _)| *known == base)
.map_or(&["-e"][..], |(_, flags)| *flags);
argv.extend(flags.iter().map(|f| (*f).to_string()));
argv
}
fn on_path(program: &str) -> bool {
let Some(path) = std::env::var_os("PATH") else {
return false;
};
std::env::split_paths(&path).any(|dir| dir.join(program).is_file())
}
/// Substitute `{}` (or append) the path, and expand `$VAR` tokens. Tokens that
/// expand to nothing are dropped, so `["$EDITOR"]` with no `EDITOR` set leaves
/// an empty argv and falls back at the call site.
fn expand(argv: &[String], path: &Path) -> Vec<String> {
let target = path.display().to_string();
let mut out = Vec::with_capacity(argv.len() + 1);
let mut substituted = false;
for token in argv {
if token.contains("{}") {
out.push(token.replace("{}", &target));
substituted = true;
continue;
}
match expand_env(token) {
Some(value) => out.push(value),
None => continue,
}
}
if out.is_empty() {
return vec!["xdg-open".to_string(), target];
}
if !substituted {
out.push(target);
}
out
}
/// `$VAR` / `${VAR}` for a whole token, `None` when the variable is unset or
/// empty so the token drops out. A token that is not a variable reference is
/// returned unchanged; this is deliberately not a shell — no word splitting, no
/// quoting rules.
fn expand_env(token: &str) -> Option<String> {
let Some(name) = token
.strip_prefix("${")
.and_then(|rest| rest.strip_suffix('}'))
.or_else(|| token.strip_prefix('$'))
else {
return Some(token.to_string());
};
if name.is_empty() {
return Some(token.to_string());
}
// Follow `$EDITOR`'s convention of preferring the graphical-capable one.
let value = if name == "EDITOR" {
std::env::var("VISUAL")
.ok()
.filter(|v| !v.is_empty())
.or_else(|| std::env::var("EDITOR").ok())
} else {
std::env::var(name).ok()
};
value.filter(|v| !v.is_empty())
}
/// Whether `pattern` — a glob or an `@class` — matches this name.
fn matches(pattern: &str, name: &str, is_dir: bool) -> bool {
let name_lower = name.to_lowercase();
match pattern.to_lowercase().as_str() {
"@any" => true,
"@dir" => is_dir,
"@text" => !is_dir && class_text(&name_lower),
"@document" => !is_dir && class_document(&name_lower),
"@image" => !is_dir && class_image(&name_lower),
"@media" => !is_dir && class_media(&name_lower),
other if other.starts_with('@') => false,
other => glob_match(other, &name_lower),
}
}
fn extension(name: &str) -> &str {
Path::new(name)
.extension()
.and_then(OsStr::to_str)
.unwrap_or_default()
}
/// Plain-text-ish files a terminal editor is a sensible default for. Extensions
/// only: the daemon's hits may name files that are not on this machine yet, so
/// there is nothing to sniff.
fn class_text(name: &str) -> bool {
matches!(
extension(name),
"txt"
| "md"
| "markdown"
| "rst"
| "org"
| "log"
| "conf"
| "cfg"
| "ini"
| "toml"
| "yaml"
| "yml"
| "json"
| "xml"
| "csv"
| "tsv"
| "env"
| "sh"
| "bash"
| "zsh"
| "fish"
| "rs"
| "py"
| "go"
| "c"
| "h"
| "cpp"
| "hpp"
| "js"
| "ts"
| "tsx"
| "jsx"
| "css"
| "html"
| "sql"
| "lua"
| "vim"
| "nix"
| "patch"
| "diff"
) || name == "makefile"
|| name == "dockerfile"
}
fn class_document(name: &str) -> bool {
matches!(
extension(name),
"pdf"
| "doc"
| "docx"
| "odt"
| "rtf"
| "xls"
| "xlsx"
| "ods"
| "ppt"
| "pptx"
| "odp"
| "epub"
) || class_text(name)
}
fn class_image(name: &str) -> bool {
matches!(
extension(name),
"png" | "jpg" | "jpeg" | "webp" | "gif" | "bmp" | "svg" | "avif" | "heic" | "tiff"
)
}
fn class_media(name: &str) -> bool {
matches!(
extension(name),
"mp4"
| "mkv"
| "webm"
| "mov"
| "avi"
| "mp3"
| "flac"
| "wav"
| "ogg"
| "opus"
| "m4a"
| "aac"
)
}
/// `*` and `?` globbing, no character classes and no path semantics — patterns
/// are matched against a bare file name. Iterative with backtracking so a
/// pathological pattern cannot blow the stack.
fn glob_match(pattern: &str, name: &str) -> bool {
let pattern: Vec<char> = pattern.chars().collect();
let name: Vec<char> = name.chars().collect();
let (mut p, mut n) = (0usize, 0usize);
// Where to resume if the current `*` turns out to have matched too little.
let (mut star, mut resume) = (None, 0usize);
while n < name.len() {
match pattern.get(p) {
Some('*') => {
star = Some(p);
resume = n;
p += 1;
}
Some('?') => {
p += 1;
n += 1;
}
Some(c) if *c == name[n] => {
p += 1;
n += 1;
}
_ => match star {
Some(s) => {
p = s + 1;
resume += 1;
n = resume;
}
None => return false,
},
}
}
pattern[p..].iter().all(|c| *c == '*')
}
#[cfg(test)]
mod tests {
use super::*;
fn rule(patterns: &[&str], command: &[&str], terminal: bool) -> OpenRule {
OpenRule {
patterns: patterns.iter().map(|p| (*p).to_string()).collect(),
command: command.iter().map(|c| (*c).to_string()).collect(),
terminal,
}
}
#[test]
fn a_cache_blob_is_classified_by_the_name_it_was_opened_as() {
let policy = OpenWith {
terminal: Some(vec!["alacritty".into(), "-e".into()]),
default: None,
rules: vec![rule(&["@text"], &["nvim"], true)],
};
// What the daemon hands back: content-hash name, no extension. Matched
// on its own it is "unknown" and would land in xdg-open's lap.
let blob = Path::new("/home/me/.cache/proton-drive-linux/content/364d1d8cc8e6");
assert_eq!(
policy.command_for(blob, false),
vec![
"xdg-open",
"/home/me/.cache/proton-drive-linux/content/364d1d8cc8e6"
]
);
assert_eq!(
policy.command_for_named(blob, "ONBOARDING.md", false),
vec![
"alacritty",
"-e",
"nvim",
"/home/me/.cache/proton-drive-linux/content/364d1d8cc8e6"
]
);
}
#[test]
fn an_empty_policy_is_xdg_open() {
let policy = OpenWith::default();
assert_eq!(
policy.command_for(Path::new("/tmp/a.md"), false),
vec!["xdg-open".to_string(), "/tmp/a.md".to_string()]
);
}
#[test]
fn a_terminal_rule_wraps_the_command() {
let policy = OpenWith {
terminal: Some(vec!["alacritty".into(), "-e".into()]),
default: None,
rules: vec![rule(&["@text"], &["nvim"], true)],
};
assert_eq!(
policy.command_for(Path::new("/tmp/notes.md"), false),
vec!["alacritty", "-e", "nvim", "/tmp/notes.md"]
);
}
#[test]
fn the_first_matching_rule_wins_and_globs_are_case_insensitive() {
let policy = OpenWith {
terminal: None,
default: None,
rules: vec![
rule(&["*.PNG"], &["imv"], false),
rule(&["@image"], &["eog"], false),
],
};
assert_eq!(
policy.command_for(Path::new("/tmp/shot.png"), false),
vec!["imv", "/tmp/shot.png"]
);
}
#[test]
fn a_placeholder_positions_the_path_instead_of_appending_it() {
let policy = OpenWith {
terminal: None,
default: None,
rules: vec![rule(&["@any"], &["sh", "-c", "cat {} | less"], false)],
};
assert_eq!(
policy.command_for(Path::new("/tmp/a.log"), false),
vec!["sh", "-c", "cat /tmp/a.log | less"]
);
}
#[test]
fn directories_only_match_the_dir_class() {
let policy = OpenWith {
terminal: None,
default: None,
rules: vec![
rule(&["@text"], &["nvim"], false),
rule(&["@dir"], &["nautilus"], false),
],
};
assert_eq!(
policy.command_for(Path::new("/tmp/notes.md"), true),
vec!["nautilus", "/tmp/notes.md"]
);
}
#[test]
fn a_rule_that_cannot_find_a_terminal_falls_back_to_the_default() {
// A terminal that is spelled out in config is always honoured; this is
// the other path — resolution must not invent one from the test host.
let policy = OpenWith {
terminal: Some(Vec::new()),
default: Some(vec!["gio".into(), "open".into()]),
rules: vec![rule(&["@text"], &["nvim"], true)],
};
let argv = policy.command_for(Path::new("/tmp/a.md"), false);
// Either a terminal exists on this machine (wrapped nvim) or it does
// not (the default). Both are correct; what must not happen is a bare
// `nvim` with no tty.
assert!(
argv == vec!["gio", "open", "/tmp/a.md"]
|| argv.ends_with(&["nvim".to_string(), "/tmp/a.md".to_string()])
);
}
#[test]
fn globs_backtrack() {
assert!(glob_match("*.md", "notes.md"));
assert!(glob_match("*notes*.md", "my-notes-2024.md"));
assert!(glob_match("a?c.txt", "abc.txt"));
assert!(!glob_match("*.md", "notes.mdx"));
assert!(!glob_match("a?c.txt", "ac.txt"));
assert!(glob_match("*", "anything"));
}
#[test]
fn a_bare_terminal_name_gains_its_exec_flag() {
assert_eq!(with_exec_flag("alacritty"), vec!["alacritty", "-e"]);
assert_eq!(with_exec_flag("kitty"), vec!["kitty"]);
assert_eq!(
with_exec_flag("/usr/bin/wezterm"),
vec!["/usr/bin/wezterm", "start", "--"]
);
// Unknown terminals get the near-universal `-e`.
assert_eq!(with_exec_flag("mystery-term"), vec!["mystery-term", "-e"]);
}
}
+185
View File
@@ -0,0 +1,185 @@
//! Offline reverse geocoding: the town a photo was taken in, from its GPS
//! position, without asking any service.
//!
//! The towns are GeoNames' `cities15000` dump (every place with more than
//! 15,000 people), trimmed to id, name, position and country in
//! `data/cities15000.tsv` by `scripts/geonames-cities.py`. The script drops the
//! parts of a city, such as Kreuzberg or "Paris 15 Vaugirard", so that a photo
//! lands in the town people would name. GeoNames publishes the dump under
//! CC BY 4.0, which is why the About dialog credits it.
//!
//! `data/countries.tsv` maps each country code to its ISO 3166-1 English name,
//! as the `iso-codes` project spells it, so a front end can translate it
//! through that project's `iso_3166-1` gettext domain.
use std::collections::HashMap;
use std::sync::OnceLock;
/// How far from the nearest town a photo may be and still count as taken
/// there, in km. Farther out (at sea, in the wilderness) it has no place.
pub const MAX_DISTANCE_KM: f64 = 50.0;
/// Mean Earth radius in km, for the haversine distance.
const EARTH_RADIUS_KM: f64 = 6371.0;
const CITIES: &str = include_str!("../data/cities15000.tsv");
const COUNTRIES: &str = include_str!("../data/countries.tsv");
/// One town of the table.
#[derive(Debug, Clone, PartialEq)]
pub struct City {
/// The GeoNames id, stable across dumps.
pub id: u32,
pub name: &'static str,
pub latitude: f64,
pub longitude: f64,
/// ISO 3166-1 alpha-2 country code.
pub country: &'static str,
}
/// The towns, and an index of them by whole-degree cell.
struct Table {
cities: Vec<City>,
cells: HashMap<(i32, i32), Vec<usize>>,
}
fn table() -> &'static Table {
static TABLE: OnceLock<Table> = OnceLock::new();
TABLE.get_or_init(|| {
let cities: Vec<City> = CITIES.lines().filter_map(parse_city).collect();
let mut cells: HashMap<(i32, i32), Vec<usize>> = HashMap::new();
for (index, city) in cities.iter().enumerate() {
cells
.entry(cell(city.latitude, city.longitude))
.or_default()
.push(index);
}
Table { cities, cells }
})
}
fn parse_city(line: &'static str) -> Option<City> {
let mut fields = line.split('\t');
Some(City {
id: fields.next()?.parse().ok()?,
name: fields.next()?,
latitude: fields.next()?.parse().ok()?,
longitude: fields.next()?.parse().ok()?,
country: fields.next()?,
})
}
fn cell(latitude: f64, longitude: f64) -> (i32, i32) {
(latitude.floor() as i32, longitude.floor() as i32)
}
/// Great-circle distance between two positions, in km.
pub fn distance_km(a: (f64, f64), b: (f64, f64)) -> f64 {
let (lat1, lat2) = (a.0.to_radians(), b.0.to_radians());
let dlat = lat2 - lat1;
let dlon = (b.1 - a.1).to_radians();
let h = (dlat / 2.0).sin().powi(2) + lat1.cos() * lat2.cos() * (dlon / 2.0).sin().powi(2);
2.0 * EARTH_RADIUS_KM * h.sqrt().min(1.0).asin()
}
/// The town nearest to a position, if one lies within [`MAX_DISTANCE_KM`].
pub fn nearest(latitude: f64, longitude: f64) -> Option<&'static City> {
if !latitude.is_finite() || !longitude.is_finite() {
return None;
}
let table = table();
// A degree of latitude is about 111 km everywhere; a degree of longitude
// shrinks towards the poles, so more cells are needed to cover the radius.
let lat_span = (MAX_DISTANCE_KM / 111.0).ceil() as i32;
let lon_span = {
let scale = latitude.to_radians().cos().abs().max(0.01);
((MAX_DISTANCE_KM / (111.0 * scale)).ceil() as i32).min(180)
};
let (row, column) = cell(latitude, longitude);
let mut best: Option<(f64, &City)> = None;
for dlat in -lat_span..=lat_span {
for dlon in -lon_span..=lon_span {
// Longitude wraps at the antimeridian: -181 is 179.
let lon_cell = (column + dlon + 180).rem_euclid(360) - 180;
let Some(indices) = table.cells.get(&(row + dlat, lon_cell)) else {
continue;
};
for &index in indices {
let city = &table.cities[index];
let d = distance_km((latitude, longitude), (city.latitude, city.longitude));
if d <= MAX_DISTANCE_KM && best.is_none_or(|(bd, _)| d < bd) {
best = Some((d, city));
}
}
}
}
best.map(|(_, city)| city)
}
/// The town with this GeoNames id.
pub fn city(id: u32) -> Option<&'static City> {
let cities = &table().cities;
// The table is sorted by id.
cities
.binary_search_by_key(&id, |city| city.id)
.ok()
.map(|index| &cities[index])
}
/// The English ISO 3166-1 name of a country code, such as "Germany" for "DE".
pub fn country_name(code: &str) -> Option<&'static str> {
COUNTRIES.lines().find_map(|line| {
let (c, name) = line.split_once('\t')?;
(c == code).then_some(name)
})
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_position_in_a_town_resolves_to_that_town() {
let berlin = nearest(52.52, 13.405).expect("Berlin is a town");
assert_eq!(berlin.name, "Berlin");
assert_eq!(berlin.country, "DE");
assert_eq!(city(berlin.id), Some(berlin));
}
#[test]
fn the_open_sea_has_no_town() {
assert_eq!(nearest(0.0, -30.0), None);
assert_eq!(nearest(f64::NAN, 0.0), None);
}
#[test]
fn a_town_across_the_antimeridian_is_still_found() {
// Just east of 180°, the nearest towns in Fiji are west of it.
let town = nearest(-16.8, 179.99);
assert!(town.is_none_or(|t| t.country == "FJ"));
let suva = nearest(-18.14, 178.44).expect("Suva is a town");
assert_eq!(suva.country, "FJ");
}
#[test]
fn every_town_has_a_named_country() {
for city in &table().cities {
assert!(
country_name(city.country).is_some(),
"{} has no country name for {}",
city.name,
city.country
);
}
assert_eq!(country_name("DE"), Some("Germany"));
}
#[test]
fn distances_are_great_circle_km() {
let d = distance_km((52.52, 13.405), (48.137, 11.575));
assert!(
(d - 504.0).abs() < 5.0,
"Berlin to Munich is about 504 km, got {d}"
);
}
}
+62 -4
View File
@@ -17,9 +17,20 @@
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
/// File name of the profile inside the device root folder. use crate::mounts::MountMode;
/// File name of the profile.
pub const PROFILE_FILE_NAME: &str = "profile.json"; pub const PROFILE_FILE_NAME: &str = "profile.json";
/// Folder, directly under the device root, that holds the profile.
///
/// The API refuses `Cannot create file at the root of a device` (HTTP 422), so
/// the profile cannot live at the device root itself even though that is where
/// it logically belongs (docs/BUGS.md B68/B78). A dotted name keeps it out of
/// the way in the web UI, and the restore ignores it when listing the device's
/// restorable folders.
pub const PROFILE_DIR_NAME: &str = ".proton-drive-linux";
/// Format version of the profile document. /// Format version of the profile document.
/// ///
/// Bumped only for changes a *reader* cannot survive. Adding fields does not /// Bumped only for changes a *reader* cannot survive. Adding fields does not
@@ -39,8 +50,9 @@ pub struct ProfileFolder {
/// Absolute local path it was synced to. A *suggestion* on restore: another /// Absolute local path it was synced to. A *suggestion* on restore: another
/// machine may have a different username, home, or mount layout. /// machine may have a different username, home, or mount layout.
pub local_path: String, pub local_path: String,
/// `mirror` or `ondemand`. /// `mirror` or `ondemand`. Unknown values written by a newer client do not
pub mode: String, /// invalidate the rest of the profile.
pub mode: MountMode,
} }
/// One pinned node (P5 pins), as `pins` stores it. /// One pinned node (P5 pins), as `pins` stores it.
@@ -114,7 +126,7 @@ mod tests {
folders: vec![ProfileFolder { folders: vec![ProfileFolder {
remote_uid: "vol~link".to_string(), remote_uid: "vol~link".to_string(),
local_path: "/home/nils/Documents".to_string(), local_path: "/home/nils/Documents".to_string(),
mode: "mirror".to_string(), mode: MountMode::Mirror,
}], }],
pins: vec![ProfilePin { pins: vec![ProfilePin {
uid: "vol~pin".to_string(), uid: "vol~pin".to_string(),
@@ -133,6 +145,52 @@ mod tests {
assert_eq!(Profile::parse(&p.to_bytes().unwrap()).unwrap(), p); assert_eq!(Profile::parse(&p.to_bytes().unwrap()).unwrap(), p);
} }
#[test]
fn profile_keeps_legacy_mode_bytes_and_parses_old_documents() {
let bytes = sample().to_bytes().unwrap();
let json = String::from_utf8(bytes).unwrap();
assert!(json.contains(r#""mode": "mirror""#));
let old = br#"{
"version": 1,
"device_uid": "dev-1",
"hostname": "old-client",
"saved_at": 1,
"folders": [
{
"remote_uid": "vol~mirror",
"local_path": "/home/me/Mirror",
"mode": "mirror"
},
{
"remote_uid": "vol~ondemand",
"local_path": "/home/me/Cloud",
"mode": "ondemand"
}
]
}"#;
let profile = Profile::parse(old).unwrap();
assert_eq!(profile.folders[0].mode, MountMode::Mirror);
assert_eq!(profile.folders[1].mode, MountMode::OnDemand);
}
#[test]
fn unknown_folder_mode_does_not_reject_profile() {
let json = br#"{
"version": 1,
"device_uid": "dev-1",
"hostname": "future-client",
"saved_at": 1,
"folders": [{
"remote_uid": "vol~future",
"local_path": "/home/me/Future",
"mode": "streaming"
}]
}"#;
let profile = Profile::parse(json).unwrap();
assert_eq!(profile.folders[0].mode, MountMode::Unknown);
}
/// A field added by a later client must not break this one's restore: the /// A field added by a later client must not break this one's restore: the
/// common subset is still usable, which is the whole reason `version` is /// common subset is still usable, which is the whole reason `version` is
/// reserved for breaking changes only. /// reserved for breaking changes only.
+283
View File
@@ -0,0 +1,283 @@
//! A persistent [`CacheRepository`] for the SDK's Drive entity cache.
//!
//! The SDK caches decrypted node metadata (name, size, parent, the share whose
//! membership signs operations on it) so navigating a tree does not re-fetch and
//! re-decrypt what it already resolved. That cache defaults to memory, which
//! means every daemon restart pays for the whole walk again. This backs it with
//! SQLite so it survives one.
//!
//! **Its own database file, not `cache.db`.** The daemon's `Db` is a single
//! `Mutex<Connection>` shared by every FUSE thread and the control socket; SDK
//! cache traffic is frequent, small, and entirely reconstructible, so putting it
//! on that mutex would add contention to the daemon's hottest lock (and to a
//! write-ahead log whose growth is already sensitive to how long that lock is
//! held) for no durability benefit. A separate file also means no schema
//! migration: this store can be deleted and rebuilt at any time.
//!
//! **What it holds.** Decrypted metadata, never key material — the SDK keeps
//! node keys and hash keys in a separate in-memory secret cache that nothing
//! here can reach. The same names and sizes are already persisted in `cache.db`
//! (`nodes`), so this file widens no exposure that the state directory did not
//! already have; `AppDirs::ensure` is what keeps that directory private.
//!
//! **Staleness.** A persisted cache outlives the process, so an entry can
//! describe a node another client has since changed. The daemon's event loop is
//! what closes that: it resumes from a persisted cursor and calls the SDK's
//! `invalidate_caches_for_event` for every event it replays, including the ones
//! that happened while the daemon was down. The one case that leaves no trail is
//! a *seeded* cursor — a first-ever mount, or a cursor the daemon lost — where
//! nothing says what changed, so the caller [`clear`](SdkCache::clear)s this
//! store instead of trusting it.
use std::path::Path;
use std::sync::{Arc, OnceLock};
use async_trait::async_trait;
use parking_lot::Mutex;
use proton_sdk::cache::CacheRepository;
use proton_sdk::error::{ProtonError, Result as SdkResult};
use rusqlite::{Connection, params};
use crate::config::AppDirs;
use crate::error::Result;
/// Process-wide handle, so the daemon's event loop can clear the same store the
/// client is writing through without threading it down every call.
static SHARED: OnceLock<Arc<SdkCache>> = OnceLock::new();
/// SQLite-backed entity cache for the SDK.
pub struct SdkCache {
conn: Mutex<Connection>,
}
impl SdkCache {
/// Open (creating if needed) the entity cache under the app's state
/// directory. The first call wins; later calls return the same handle, which
/// is what lets the event loop clear the store the Drive client is using.
pub fn shared(dirs: &AppDirs) -> Result<Arc<Self>> {
if let Some(existing) = SHARED.get() {
return Ok(existing.clone());
}
let cache = Arc::new(Self::open(&dirs.state_dir().join("sdk_cache.db"))?);
Ok(SHARED.get_or_init(|| cache).clone())
}
/// The already-open shared cache, if one was opened in this process.
pub fn opened() -> Option<Arc<Self>> {
SHARED.get().cloned()
}
/// Open the store at `path`, creating its schema.
pub fn open(path: &Path) -> Result<Self> {
let conn = Connection::open(path)?;
conn.pragma_update(None, "journal_mode", "WAL")?;
conn.pragma_update(None, "synchronous", "NORMAL")?;
conn.pragma_update(None, "foreign_keys", "ON")?;
// Tags cascade with their entry, so a `set` that replaces a value cannot
// leave the old value's tags behind pointing at the new one.
conn.execute_batch(
"CREATE TABLE IF NOT EXISTS entry (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS tag (
key TEXT NOT NULL REFERENCES entry(key) ON DELETE CASCADE,
tag TEXT NOT NULL,
PRIMARY KEY (key, tag)
);
CREATE INDEX IF NOT EXISTS idx_tag_tag ON tag(tag);",
)?;
Ok(Self {
conn: Mutex::new(conn),
})
}
/// Drop every entry. Used when the daemon cannot prove the cache is current
/// — a seeded event cursor — and on logout.
pub fn clear_now(&self) -> Result<()> {
let conn = self.conn.lock();
conn.execute_batch("DELETE FROM tag; DELETE FROM entry;")?;
Ok(())
}
fn set_now(&self, key: &str, value: &str, tags: &[String]) -> Result<()> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
tx.execute(
"INSERT INTO entry (key, value) VALUES (?1, ?2)
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
params![key, value],
)?;
tx.execute("DELETE FROM tag WHERE key = ?1", params![key])?;
{
let mut stmt = tx.prepare("INSERT INTO tag (key, tag) VALUES (?1, ?2)")?;
for tag in tags {
stmt.execute(params![key, tag])?;
}
}
tx.commit()?;
Ok(())
}
fn get_now(&self, key: &str) -> Result<Option<String>> {
let conn = self.conn.lock();
let mut stmt = conn.prepare("SELECT value FROM entry WHERE key = ?1")?;
let mut rows = stmt.query(params![key])?;
match rows.next()? {
Some(row) => Ok(Some(row.get(0)?)),
None => Ok(None),
}
}
fn remove_now(&self, key: &str) -> Result<()> {
let conn = self.conn.lock();
conn.execute("DELETE FROM entry WHERE key = ?1", params![key])?;
Ok(())
}
fn remove_by_tag_now(&self, tag: &str) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"DELETE FROM entry WHERE key IN (SELECT key FROM tag WHERE tag = ?1)",
params![tag],
)?;
Ok(())
}
/// Entries carrying **all** of `tags` — the set intersection the SDK's
/// `get_by_tags` contract specifies, not a union.
fn get_by_tags_now(&self, tags: &[String]) -> Result<Vec<(String, String)>> {
if tags.is_empty() {
return Ok(Vec::new());
}
let conn = self.conn.lock();
let placeholders = vec!["?"; tags.len()].join(",");
let sql = format!(
"SELECT e.key, e.value FROM entry e
JOIN tag t ON t.key = e.key
WHERE t.tag IN ({placeholders})
GROUP BY e.key
HAVING COUNT(DISTINCT t.tag) = {}",
tags.len()
);
let mut stmt = conn.prepare(&sql)?;
let rows = stmt.query_map(rusqlite::params_from_iter(tags), |r| {
Ok((r.get(0)?, r.get(1)?))
})?;
let mut out = Vec::new();
for row in rows {
out.push(row?);
}
Ok(out)
}
}
/// A cache failure is never fatal to the operation that triggered it — the SDK
/// treats this repository as best-effort — but it still has to come back as the
/// SDK's error type.
fn sdk_error(e: crate::error::Error) -> ProtonError {
ProtonError::invalid_operation(format!("sdk entity cache: {e}"))
}
// Every method is a local SQLite statement on its own connection: microseconds,
// no network, no shared lock with the daemon's database. Running them inline on
// the caller's task is cheaper than the thread hop that moving them off would
// cost.
#[async_trait]
impl CacheRepository for SdkCache {
async fn set(&self, key: &str, value: &str, tags: &[String]) -> SdkResult<()> {
self.set_now(key, value, tags).map_err(sdk_error)
}
async fn get(&self, key: &str) -> SdkResult<Option<String>> {
self.get_now(key).map_err(sdk_error)
}
async fn remove(&self, key: &str) -> SdkResult<()> {
self.remove_now(key).map_err(sdk_error)
}
async fn remove_by_tag(&self, tag: &str) -> SdkResult<()> {
self.remove_by_tag_now(tag).map_err(sdk_error)
}
async fn clear(&self) -> SdkResult<()> {
self.clear_now().map_err(sdk_error)
}
async fn get_by_tags(&self, tags: &[String]) -> SdkResult<Vec<(String, String)>> {
self.get_by_tags_now(tags).map_err(sdk_error)
}
}
#[cfg(test)]
mod tests {
use super::*;
fn cache() -> SdkCache {
let conn = Connection::open_in_memory().unwrap();
conn.pragma_update(None, "foreign_keys", "ON").unwrap();
conn.execute_batch(
"CREATE TABLE entry (key TEXT PRIMARY KEY, value TEXT NOT NULL);
CREATE TABLE tag (
key TEXT NOT NULL REFERENCES entry(key) ON DELETE CASCADE,
tag TEXT NOT NULL,
PRIMARY KEY (key, tag)
);",
)
.unwrap();
SdkCache {
conn: Mutex::new(conn),
}
}
fn tags(values: &[&str]) -> Vec<String> {
values.iter().map(|t| t.to_string()).collect()
}
#[test]
fn set_replaces_the_value_and_its_tags() {
let cache = cache();
cache.set_now("k", "one", &tags(&["a", "b"])).unwrap();
cache.set_now("k", "two", &tags(&["b"])).unwrap();
assert_eq!(cache.get_now("k").unwrap().as_deref(), Some("two"));
// The tag dropped by the second write must not still select the entry —
// that is how a stale invalidation would miss it.
assert!(cache.get_by_tags_now(&tags(&["a"])).unwrap().is_empty());
assert_eq!(cache.get_by_tags_now(&tags(&["b"])).unwrap().len(), 1);
}
#[test]
fn get_by_tags_intersects_rather_than_unions() {
let cache = cache();
cache.set_now("both", "1", &tags(&["x", "y"])).unwrap();
cache.set_now("one", "2", &tags(&["x"])).unwrap();
let hits = cache.get_by_tags_now(&tags(&["x", "y"])).unwrap();
assert_eq!(hits, vec![("both".to_string(), "1".to_string())]);
// An empty tag list selects nothing, matching the SDK contract.
assert!(cache.get_by_tags_now(&[]).unwrap().is_empty());
}
#[test]
fn removing_by_tag_takes_the_entry_and_its_tags() {
let cache = cache();
cache.set_now("k", "v", &tags(&["gone"])).unwrap();
cache.remove_by_tag_now("gone").unwrap();
assert!(cache.get_now("k").unwrap().is_none());
assert!(cache.get_by_tags_now(&tags(&["gone"])).unwrap().is_empty());
}
#[test]
fn clear_empties_the_store() {
let cache = cache();
cache.set_now("a", "1", &tags(&["t"])).unwrap();
cache.set_now("b", "2", &[]).unwrap();
cache.clear_now().unwrap();
assert!(cache.get_now("a").unwrap().is_none());
assert!(cache.get_now("b").unwrap().is_none());
}
}
+11 -3
View File
@@ -47,10 +47,18 @@ fn systemctl_query(args: &[&str]) -> bool {
.unwrap_or(false) .unwrap_or(false)
} }
/// Enable the service so it survives reboots/logins and start it now. Called on /// Enable the service so it survives reboots/logins, and bring it up now.
/// successful login. /// Called on successful login.
///
/// `restart` rather than `enable --now`: the unit may already be active but
/// parked in the daemon's login-wait backoff (`pdfs daemon` sleeps between
/// keyring checks). `--now` is a no-op on an active unit, so the daemon would
/// keep sleeping and the mount would not appear until the next poll fired.
/// Restarting makes the new session take effect immediately, and starts the
/// unit when it is not running at all.
pub fn enable_start() -> bool { pub fn enable_start() -> bool {
systemctl(&["enable", "--now"]) let enabled = systemctl(&["enable"]);
systemctl(&["restart"]) && enabled
} }
/// Stop the running mount without disabling it. Called by the tray's /// Stop the running mount without disabling it. Called by the tray's
+193
View File
@@ -0,0 +1,193 @@
//! Telling photos apart by what they show rather than by their bytes.
//!
//! The server's content hash only matches byte-identical files. A photo that
//! was resized, re-encoded or sent through a messenger is another file, so the
//! similar-photo finder hashes each thumbnail instead: a 64-bit difference
//! hash ("dHash"), which compares neighbouring brightness in a 9×8 grid. Two
//! copies of one picture land a few bits apart, and different pictures about
//! half of the bits apart.
/// Hashes this many bits apart or fewer show the same picture.
pub const SIMILAR_BITS: u32 = 6;
/// Stored for a photo whose thumbnail was looked at and gave no usable hash
/// (it couldn't be decoded, or is one flat colour), so it isn't looked at
/// again. A real hash is never 0: a picture that would hash to 0 has no
/// brightness changes, which [`difference_hash`] turns down as flat.
pub const NO_HASH: u64 = 0;
/// Samples of the grid that differ by less than this in brightness, all of
/// them, make a flat picture — a black frame, a blank page — whose hash is
/// noise and would match every other flat picture.
const FLAT: u8 = 8;
/// The difference hash of an 8-bit greyscale image `width` px wide, rows
/// stored one after another in `luma`. `None` for an empty or flat image.
pub fn difference_hash(width: usize, height: usize, luma: &[u8]) -> Option<u64> {
if width == 0 || height == 0 || luma.len() < width * height {
return None;
}
// Box-average down to 9×8, so every source pixel counts once and the hash
// doesn't depend on which pixels a nearest-neighbour pick happens to hit.
let mut grid = [[0u8; 9]; 8];
for (gy, row) in grid.iter_mut().enumerate() {
let (y0, y1) = span(gy, 8, height);
for (gx, cell) in row.iter_mut().enumerate() {
let (x0, x1) = span(gx, 9, width);
let mut sum = 0u64;
for y in y0..y1 {
sum += luma[y * width + x0..y * width + x1]
.iter()
.map(|&v| u64::from(v))
.sum::<u64>();
}
*cell = (sum / ((y1 - y0) * (x1 - x0)) as u64) as u8;
}
}
let (min, max) = grid
.iter()
.flatten()
.fold((u8::MAX, 0), |(min, max), &v| (min.min(v), max.max(v)));
if max - min < FLAT {
return None;
}
let mut hash = 0u64;
for row in &grid {
for pair in row.windows(2) {
hash = hash << 1 | u64::from(pair[0] < pair[1]);
}
}
(hash != NO_HASH).then_some(hash)
}
/// The source pixels `from..to` that cell `i` of `cells` covers along an edge
/// of `len` px; never empty, even for an edge shorter than the grid.
fn span(i: usize, cells: usize, len: usize) -> (usize, usize) {
let from = (i * len / cells).min(len - 1);
let to = ((i + 1) * len / cells).max(from + 1);
(from, to)
}
/// Whether two hashes show the same picture.
pub fn alike(a: u64, b: u64) -> bool {
(a ^ b).count_ones() <= SIMILAR_BITS
}
/// Group `hashes` into sets of alike ones, by index, each set in index order
/// and the sets by their first index. A hash alike to none other is in no
/// set. Alike is taken as catching: a burst of shots each a little different
/// from the last is one set.
pub fn sets(hashes: &[u64]) -> Vec<Vec<usize>> {
// Two hashes at most SIMILAR_BITS apart agree on at least one of their
// eight bytes, as long as SIMILAR_BITS < 8. So only hashes sharing some
// byte are compared, not every pair of the library.
const _: () = assert!(SIMILAR_BITS < 8);
let mut parent: Vec<usize> = (0..hashes.len()).collect();
fn root(parent: &mut [usize], mut i: usize) -> usize {
while parent[i] != i {
parent[i] = parent[parent[i]];
i = parent[i];
}
i
}
for byte in 0..8 {
let mut buckets: Vec<Vec<usize>> = vec![Vec::new(); 256];
for (i, &hash) in hashes.iter().enumerate() {
buckets[(hash >> (byte * 8)) as u8 as usize].push(i);
}
for bucket in &buckets {
for (n, &a) in bucket.iter().enumerate() {
for &b in &bucket[n + 1..] {
if alike(hashes[a], hashes[b]) {
let (ra, rb) = (root(&mut parent, a), root(&mut parent, b));
if ra != rb {
parent[ra.max(rb)] = ra.min(rb);
}
}
}
}
}
}
let mut by_root: std::collections::BTreeMap<usize, Vec<usize>> = Default::default();
for i in 0..hashes.len() {
let r = root(&mut parent, i);
by_root.entry(r).or_default().push(i);
}
by_root.into_values().filter(|set| set.len() > 1).collect()
}
#[cfg(test)]
mod tests {
use super::*;
/// A 90×80 picture: a bright disc on a gradient.
fn picture(width: usize, height: usize) -> Vec<u8> {
let mut luma = Vec::with_capacity(width * height);
for y in 0..height {
for x in 0..width {
let (fx, fy) = (x as f64 / width as f64, y as f64 / height as f64);
let disc = ((fx - 0.3).powi(2) + (fy - 0.6).powi(2)).sqrt() < 0.2;
let v = if disc { 240.0 } else { 40.0 + 120.0 * fx * fy };
luma.push(v as u8);
}
}
luma
}
#[test]
fn a_resized_copy_hashes_alike() {
let big = difference_hash(900, 800, &picture(900, 800)).unwrap();
let small = difference_hash(91, 79, &picture(91, 79)).unwrap();
assert!(
alike(big, small),
"{} bits apart",
(big ^ small).count_ones()
);
}
#[test]
fn a_different_picture_hashes_apart() {
let one = difference_hash(90, 80, &picture(90, 80)).unwrap();
let mirrored: Vec<u8> = picture(90, 80)
.chunks(90)
.flat_map(|row| row.iter().rev().copied().collect::<Vec<_>>())
.collect();
let other = difference_hash(90, 80, &mirrored).unwrap();
assert!(!alike(one, other));
}
#[test]
fn a_flat_picture_has_no_hash() {
assert_eq!(difference_hash(64, 64, &[17; 64 * 64]), None);
assert_eq!(difference_hash(0, 0, &[]), None);
assert_eq!(difference_hash(4, 4, &[0; 3]), None);
}
#[test]
fn a_picture_smaller_than_the_grid_still_hashes() {
let luma: Vec<u8> = (0..4 * 3).map(|i| (i * 20) as u8).collect();
assert!(difference_hash(4, 3, &luma).is_some());
}
#[test]
fn alike_hashes_form_sets_and_catch_on() {
let a = 0x0123_4567_89ab_cdef_u64;
let hashes = [
a,
!a,
a ^ 0b111,
a ^ 0b111 ^ (0b1111 << 40),
0x5555_0000_ffff_aaaa,
];
assert_eq!(sets(&hashes), vec![vec![0, 2, 3]]);
}
#[test]
fn hashes_apart_in_most_bytes_are_still_found() {
// One bit apart in each of six bytes: six bits in all, and only the
// top two bytes agree.
let a = 1u64 << 63;
let b = (0..6).fold(a, |h, byte| h | 1 << (byte * 8));
assert_eq!(sets(&[a, b]), vec![vec![0, 1]]);
}
}
+125
View File
@@ -45,6 +45,77 @@ pub const DEFAULT_IGNORE_PATTERNS: &[&str] = &[
"Thumbs.db", "Thumbs.db",
]; ];
/// Suffixes that mean a file is incomplete *by construction*: only the program
/// writing it gives it this name, only while the write is in flight, and the
/// finished article always arrives under a different one. On an on-demand mount
/// these should never seal a remote revision — the app renames the finished file
/// to its real name, and only that belongs on Drive. See [`is_transient_name`]
/// and docs/BUGS.md B70.
///
/// `.tmp`/`.temp` are deliberately **not** here, though they are in
/// [`DEFAULT_IGNORE_PATTERNS`]. The two lists answer different questions. Skipping
/// a `report.tmp` during a mirror reconcile costs nothing — the file stays on
/// disk, and a later pass picks it up if the rules change. Parking one on the
/// on-demand mount is irreversible in practice: the only exit is a rename to a
/// finished name, and a file that is simply *called* `index.tmp` — a static-site
/// template, a tool's output — never gets one. It then never reaches Drive at
/// all, silently, while its bytes pin `staging/`. That is what a real Takeout
/// tree of `index.tmp`/`preview.tmp` gallery templates did: 43 creates parked
/// forever. The generic-scratch case they were added for renames within
/// milliseconds, so it never needed the park; the download suffixes below, which
/// stay in flight for minutes, are what B70 was actually about.
const TRANSIENT_SUFFIXES: &[&str] = &[
".crdownload", // Chromium/Brave/Edge
".part", // Firefox, wget, aria2c
".partial", // some downloaders / older IE
".download", // Safari / macOS
".swp", // vim swap
".swx", // vim swap (secondary)
];
/// Whether `name` is a transient scratch/in-flight file whose bytes should not
/// be uploaded on an on-demand mount until it is renamed to its finished name.
///
/// Recognises browser download temps, editor swap/backup files, and lock files.
/// Case-insensitive on the suffix so `FOO.CRDOWNLOAD` counts too. This is the
/// on-demand-write counterpart to [`DEFAULT_IGNORE_PATTERNS`] (which only covers
/// mirror-folder reconcile); keep the two conceptually in step.
pub fn is_transient_name(name: &str) -> bool {
let lower = name.to_ascii_lowercase();
if TRANSIENT_SUFFIXES.iter().any(|s| lower.ends_with(s)) {
return true;
}
// Emacs/gedit/kate backup copies.
if name.ends_with('~') {
return true;
}
// LibreOffice lock files (`.~lock.<doc>#`) and MS Office temps (`~$doc`).
if name.starts_with(".~lock.") || name.starts_with("~$") {
return true;
}
false
}
/// Whether `name` is scratch another program writes and throws away: the
/// [`is_transient_name`] set, plus the save-by-rename temporaries of GTK, Qt,
/// Chromium and Electron, generic `.tmp` files and Emacs lock files. None of
/// them is a file the user made, so a front-end can hide them from a history
/// of what happened to the user's files.
pub fn is_scratch_name(name: &str) -> bool {
if is_transient_name(name) {
return true;
}
let lower = name.to_ascii_lowercase();
lower.ends_with(".tmp")
|| lower.ends_with(".temp")
|| name.starts_with(".goutputstream-")
|| name.starts_with(".org.chromium.")
|| name.starts_with(".com.google.chrome.")
|| name.starts_with(".#")
|| (name.starts_with('#') && name.ends_with('#') && name.len() > 1)
|| (name.starts_with('.') && name.contains(".tmp."))
}
/// Filenames accepted as a folder's ignore file, in precedence order. /// Filenames accepted as a folder's ignore file, in precedence order.
const IGNORE_FILE_NAMES: &[&str] = &[".pdfsignore", ".protonignore"]; const IGNORE_FILE_NAMES: &[&str] = &[".pdfsignore", ".protonignore"];
@@ -272,4 +343,58 @@ mod tests {
// have nothing to reconcile against. // have nothing to reconcile against.
assert!(!rules.is_ignored("", true)); assert!(!rules.is_ignored("", true));
} }
#[test]
fn transient_download_and_editor_names_are_recognised() {
for name in [
"Unconfirmed 540477.crdownload",
"movie.mkv.part",
"big.iso.partial",
"photo.jpg.download",
".notes.txt.swp",
"doc.txt~",
".~lock.report.odt#",
"~$budget.xlsx",
"FOO.CRDOWNLOAD",
] {
assert!(is_transient_name(name), "{name} should be transient");
}
}
#[test]
fn save_by_rename_temporaries_are_scratch() {
for name in [
".goutputstream-4XK2B2",
".org.chromium.Chromium.aBc123",
"~$report.docx",
".~lock.budget.ods#",
".#notes.org",
"#notes.org#",
"cache.tmp",
"page.html.crdownload",
] {
assert!(is_scratch_name(name), "{name}");
}
for name in ["report.docx", "notes.org", "#1 hits.txt", ".bashrc"] {
assert!(!is_scratch_name(name), "{name}");
}
}
#[test]
fn finished_names_are_not_transient() {
for name in [
"teamspeak.tar.gz",
"nils.zip",
"report.xml",
"archive.partly.zip", // ".partly" is not ".part"
"template.dotx",
"readme",
// A file that is *named* `.tmp` and never renamed: parking it would
// keep it off Drive for good.
"index.tmp",
"notes.TEMP",
] {
assert!(!is_transient_name(name), "{name} should not be transient");
}
}
} }
File diff suppressed because it is too large. Load diff
+20
View File
@@ -0,0 +1,20 @@
//! Talking to a running `pdfs-tray`.
//!
//! The tray binds [`AppDirs::tray_socket`] to keep a single instance. The same
//! socket takes one command, [`QUIT`], so the desktop app can take the icon
//! away when the user hides it or stops Proton Drive.
use std::io::Write;
use std::os::unix::net::UnixStream;
use crate::config::AppDirs;
/// The line that asks the tray to exit.
pub const QUIT: &str = "quit";
/// Ask a running tray to exit. False when no tray answers.
pub fn quit(dirs: &AppDirs) -> bool {
UnixStream::connect(dirs.tray_socket())
.and_then(|mut stream| stream.write_all(format!("{QUIT}\n").as_bytes()))
.is_ok()
}
+6
View File
@@ -5,16 +5,22 @@ edition.workspace = true
license.workspace = true license.workspace = true
authors.workspace = true authors.workspace = true
rust-version.workspace = true rust-version.workspace = true
repository.workspace = true
homepage.workspace = true
description = "FUSE on-demand filesystem for Proton Drive" description = "FUSE on-demand filesystem for Proton Drive"
[dependencies] [dependencies]
pdfs-core.workspace = true pdfs-core.workspace = true
zip.workspace = true
sha1.workspace = true
proton-drive-rs.workspace = true proton-drive-rs.workspace = true
futures.workspace = true
tokio.workspace = true tokio.workspace = true
tracing.workspace = true tracing.workspace = true
fuser.workspace = true fuser.workspace = true
libc.workspace = true libc.workspace = true
serde_json.workspace = true serde_json.workspace = true
base64.workspace = true
image.workspace = true image.workspace = true
notify = "8" notify = "8"
parking_lot.workspace = true parking_lot.workspace = true
+373
View File
@@ -0,0 +1,373 @@
//! Photo albums: the album listing, and one album's contents.
//!
//! Both are persisted projections of what the photos volume reports, refreshed
//! on the same stale-while-serving rule as the timeline
//! ([`crate::photos`]): a listing already on disk is served immediately and
//! refreshed behind the caller, and only an empty one waits on the network.
//!
//! An album is a folder node carrying album properties, so it has a name like
//! any other node; the properties supply the photo count and the cover. Albums
//! other people share with us come from a second listing and live on *their*
//! photos volume — which is why an album's photos are persisted per album rather
//! than looked up in our own timeline, where a shared album's photos never
//! appear.
use std::collections::{HashMap, HashSet};
use std::sync::atomic::Ordering;
use pdfs_core::control::{AlbumInfo, PhotoItem, Topic};
use pdfs_core::db::StoredAlbum;
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::NodeKind;
use proton_drive_rs::proton_sdk::ids::NodeUid;
use tracing::warn;
use super::{
ALBUM_SYNCED_PREFIX, ALBUMS_SYNCED_MS, Core, PHOTOS_AVAILABLE, PHOTOS_SYNCED_MS,
TIMELINE_ENRICH_CHUNK, TIMELINE_TTL, now_ms, parse_uid,
};
impl Core {
/// The album listing, served from disk and refreshed behind the caller when
/// stale. `None` means the account has no photos volume — the same signal
/// [`Core::photos_timeline`] gives.
pub(crate) fn albums(&self) -> CoreResult<Option<Vec<AlbumInfo>>> {
let count = self.db.albums_count().map_err(CoreError::from)?;
if count == 0 {
// Nothing on disk, so this one request waits for the fetch — unless
// the account is already known to have no photos volume.
let known_empty = self.db.state_i64(PHOTOS_AVAILABLE).ok().flatten() == Some(0);
if known_empty && !self.listing_stale(ALBUMS_SYNCED_MS, TIMELINE_TTL) {
return Ok(None);
}
if !self.rt.block_on(self.refresh_albums())? {
return Ok(None);
}
} else if self.listing_stale(ALBUMS_SYNCED_MS, TIMELINE_TTL) {
self.spawn_albums_refresh();
}
let albums = self.db.albums_list().map_err(CoreError::from)?;
Ok(Some(albums.into_iter().map(album_info).collect()))
}
/// Re-enumerate the albums — ours and the ones shared with us — and replace
/// the persisted listing. Returns false when the account has no photos
/// volume.
pub(crate) async fn refresh_albums(&self) -> CoreResult<bool> {
let photos = self.photos();
if photos
.get_photos_root()
.await
.map_err(|e| CoreError::from_api(&e, "photos root"))?
.is_none()
{
let _ = self.db.set_state_i64(PHOTOS_AVAILABLE, 0);
let _ = self.db.set_state_i64(PHOTOS_SYNCED_MS, now_ms());
let _ = self.db.set_state_i64(ALBUMS_SYNCED_MS, now_ms());
return Ok(false);
}
let mut uids = photos
.enumerate_album_node_uids()
.await
.map_err(|e| CoreError::from_api(&e, "albums"))?;
// Shared-with-me albums are a separate listing on the sharer's volume.
// Best-effort: an account whose sharing listing fails still gets its own
// albums rather than an error page.
let shared: HashSet<String> = match photos.enumerate_shared_with_me_album_uids().await {
Ok(shared) => {
let keys = shared.iter().map(|u| u.to_string()).collect();
uids.extend(shared);
keys
}
Err(e) => {
warn!(error = %e, "enumerating shared-with-me albums failed");
HashSet::new()
}
};
// The listings carry uids only; the name, photo count and cover come from
// the album nodes themselves. A chunk that fails to resolve drops those
// albums from this refresh rather than failing the whole listing.
let mut albums: Vec<StoredAlbum> = Vec::with_capacity(uids.len());
for chunk in uids.chunks(TIMELINE_ENRICH_CHUNK) {
match photos.enumerate_nodes(chunk).await {
Ok(nodes) => {
for node in nodes {
let key = node.uid.to_string();
let props = node.album.as_ref();
albums.push(StoredAlbum {
shared: shared.contains(&key),
uid: key,
name: node.name,
photo_count: props.map_or(0, |a| a.photo_count.max(0) as usize),
cover_uid: props
.and_then(|a| a.cover_photo_uid.as_ref())
.map(|u| u.to_string()),
last_activity: props.and_then(|a| a.last_activity_time),
});
}
}
Err(e) => warn!(error = %e, "resolving an album chunk failed"),
}
}
// Newest activity first, and albums the server gave no activity time for
// last — an album nobody has touched belongs below the ones in use. Ties
// fall back to the name so the order is stable between refreshes.
albums.sort_by(|a, b| {
b.last_activity
.cmp(&a.last_activity)
.then_with(|| a.name.to_lowercase().cmp(&b.name.to_lowercase()))
});
self.db.albums_replace(&albums).map_err(CoreError::from)?;
let _ = self.db.set_state_i64(PHOTOS_AVAILABLE, 1);
let _ = self.db.set_state_i64(ALBUMS_SYNCED_MS, now_ms());
Ok(true)
}
/// Refresh the album listing off the request path. At most one runs at a time.
pub(crate) fn spawn_albums_refresh(&self) {
if self.albums_refreshing.swap(true, Ordering::SeqCst) {
return;
}
let core = self.clone();
self.rt.spawn(async move {
let refreshed = core.refresh_albums().await;
core.albums_refreshing.store(false, Ordering::SeqCst);
match refreshed {
Ok(_) => core.events.publish(&[Topic::Photos]),
Err(e) => warn!(error = %e, "background album refresh failed"),
}
});
}
/// A page of one album's photos, newest capture first. The album's contents
/// are fetched on first open and refreshed behind the caller afterwards, so
/// paging never waits on the network twice.
pub(crate) fn album_photos(
&self,
album: &NodeUid,
offset: usize,
limit: usize,
) -> CoreResult<Vec<PhotoItem>> {
let key = album.to_string();
let stamp = format!("{ALBUM_SYNCED_PREFIX}{key}");
let count = self.db.album_photos_count(&key).map_err(CoreError::from)?;
if count == 0 {
self.rt.block_on(self.refresh_album(album))?;
} else if self.listing_stale(&stamp, TIMELINE_TTL) {
let core = self.clone();
let album = album.clone();
self.rt.spawn(async move {
if let Err(e) = core.refresh_album(&album).await {
warn!(%album, error = %e, "background album refresh failed");
}
});
}
let page = self
.db
.album_photos_page(&key, offset, limit)
.map_err(CoreError::from)?;
Ok(page.into_iter().map(|p| self.photo_item(p)).collect())
}
/// Make sure every album of ours has its contents on disk, for a filter
/// that asks which photos no album holds. Contents are otherwise fetched
/// the first time an album is opened, so an album never opened would read
/// as empty and its photos as unfiled.
///
/// Shared-with-me albums are skipped: their photos live on the sharer's
/// volume and are never in our timeline. The freshness stamps are no guide
/// here — any album change clears all of them — so only an album whose
/// server count disagrees with an empty local copy is waited for.
pub(crate) fn ensure_album_contents(&self) -> CoreResult<()> {
if self.db.albums_count().map_err(CoreError::from)? == 0
&& !self.rt.block_on(self.refresh_albums())?
{
return Ok(());
}
let albums = self.db.albums_list().map_err(CoreError::from)?;
for album in albums.iter().filter(|a| !a.shared && a.photo_count > 0) {
let stored = self
.db
.album_photos_count(&album.uid)
.map_err(CoreError::from)?;
if stored > 0 {
continue;
}
let Some(uid) = parse_uid(&album.uid) else {
continue;
};
if let Err(error) = self.rt.block_on(self.refresh_album(&uid)) {
warn!(album = %album.uid, %error, "fetching an album's contents failed");
}
}
Ok(())
}
/// Re-enumerate one album's photos and replace its persisted contents.
pub(crate) async fn refresh_album(&self, album: &NodeUid) -> CoreResult<()> {
let photos = self.photos();
let items = photos
.enumerate_album(album)
.await
.map_err(|e| CoreError::from_api(&e, "album"))?;
// Same enrichment as the timeline: the listing carries a uid and a
// capture time, and the gallery needs a name and media type to classify
// each entry. Best-effort per chunk.
let uids: Vec<NodeUid> = items.iter().map(|it| it.uid.clone()).collect();
let mut meta: HashMap<String, (Option<String>, Option<String>)> = HashMap::new();
for chunk in uids.chunks(TIMELINE_ENRICH_CHUNK) {
match photos.enumerate_nodes(chunk).await {
Ok(nodes) => {
for node in nodes {
let media_type = match &node.kind {
NodeKind::File { media_type, .. } => Some(media_type.clone()),
NodeKind::Folder => None,
};
meta.insert(node.uid.to_string(), (Some(node.name), media_type));
}
}
Err(e) => warn!(error = %e, "resolving album photo metadata failed"),
}
}
let rows: Vec<(String, i64, Option<String>, Option<String>)> = items
.iter()
.map(|it| {
let key = it.uid.to_string();
let (name, media_type) = meta.get(&key).cloned().unwrap_or((None, None));
(key, it.capture_time, name, media_type)
})
.collect();
let key = album.to_string();
self.db
.album_photos_replace(&key, &rows)
.map_err(CoreError::from)?;
let _ = self
.db
.set_state_i64(&format!("{ALBUM_SYNCED_PREFIX}{key}"), now_ms());
Ok(())
}
/// Create an album and return its uid. The listing is refreshed before
/// answering, so the next [`Core::albums`] already shows it.
pub(crate) fn create_album(&self, name: &str) -> CoreResult<String> {
let uid = self
.rt
.block_on(self.photos().create_album(name))
.map_err(|e| CoreError::from_api(&e, "create album"))?;
self.after_album_change(None);
Ok(uid.to_string())
}
pub(crate) fn rename_album(&self, album: &NodeUid, name: &str) -> CoreResult<()> {
self.rt
.block_on(self.photos().rename_album(album, name))
.map_err(|e| CoreError::from_api(&e, "rename album"))?;
self.after_album_change(None);
Ok(())
}
/// Delete an album. Its photos stay in the timeline unless `delete_photos`,
/// which only matters for photos that exist in the album alone.
pub(crate) fn delete_album(&self, album: &NodeUid, delete_photos: bool) -> CoreResult<()> {
self.rt
.block_on(self.photos().delete_album(album, delete_photos))
.map_err(|e| CoreError::from_api(&e, "delete album"))?;
let key = album.to_string();
if let Err(error) = self.db.album_photos_replace(&key, &[]) {
warn!(%album, %error, "a deleted album's photos could not be dropped");
}
self.after_album_change(None);
Ok(())
}
/// Add photos to an album, per photo: `(added, failed)`.
pub(crate) fn add_to_album(
&self,
album: &NodeUid,
photos: &[NodeUid],
) -> CoreResult<AlbumOutcome> {
let outcomes = self
.rt
.block_on(self.photos().add_photos_to_album(album, photos))
.map_err(|e| CoreError::from_api(&e, "add to album"))?;
self.after_album_change(Some(album));
Ok(album_outcome(outcomes))
}
/// Take photos out of an album, per photo: `(removed, failed)`. The photos
/// stay in the timeline.
pub(crate) fn remove_from_album(
&self,
album: &NodeUid,
photos: &[NodeUid],
) -> CoreResult<AlbumOutcome> {
let outcomes = self
.rt
.block_on(self.photos().remove_photos_from_album(album, photos))
.map_err(|e| CoreError::from_api(&e, "remove from album"))?;
self.after_album_change(Some(album));
Ok(album_outcome(outcomes))
}
/// Bring the persisted listing, and `album`'s contents when given, up to
/// date after a change made here, so the reply is followed by a listing
/// that shows it. A failed refresh only leaves them stale: the change
/// itself went through.
fn after_album_change(&self, album: Option<&NodeUid>) {
self.invalidate_albums();
if let Err(error) = self.rt.block_on(self.refresh_albums()) {
warn!(%error, "album listing refresh after a change failed");
}
if let Some(album) = album
&& let Err(error) = self.rt.block_on(self.refresh_album(album))
{
warn!(%album, %error, "album refresh after a change failed");
}
}
/// Drop the album listing's and every album's freshness stamp, so the next
/// request re-enumerates. Paired with [`Core::invalidate_photos`] — a change
/// to the timeline can just as easily be a change to an album.
pub(crate) fn invalidate_albums(&self) {
let _ = self.db.clear_state(ALBUMS_SYNCED_MS);
let _ = self.db.clear_state_prefix(ALBUM_SYNCED_PREFIX);
}
}
/// Photos an album change applied to, and the ones it failed for with why.
pub(crate) type AlbumOutcome = (Vec<String>, Vec<(String, String)>);
fn album_outcome(outcomes: Vec<pdfs_core::batch::Outcome>) -> AlbumOutcome {
let (done, failed) = pdfs_core::batch::split(outcomes);
(
done.iter().map(|uid| uid.to_string()).collect(),
failed
.into_iter()
.map(|(uid, error)| (uid.to_string(), error.to_string()))
.collect(),
)
}
/// Project a persisted album into the wire item a front-end paints.
fn album_info(album: StoredAlbum) -> AlbumInfo {
AlbumInfo {
uid: album.uid,
name: album.name,
photo_count: album.photo_count,
// A cover uid the daemon cannot parse is no cover: the front-end would
// only ask for a thumbnail that can never be resolved.
cover_uid: album.cover_uid.filter(|uid| parse_uid(uid).is_some()),
last_activity: album.last_activity,
shared: album.shared,
}
}
File diff suppressed because it is too large. Load diff
+179
View File
@@ -0,0 +1,179 @@
//! Listing and resolving `(sync-conflict …)` copies for the user.
//!
//! The sweep ([`crate::sweep`]) removes copies it can prove identical and flags
//! the rest once in the activity feed. That leaves divergent copies sitting in
//! folders with nothing but a name to find them by. This module lists them with
//! the file they are a copy of, and resolves one the way the user chooses:
//!
//! - **Keep the original**: trash the copy.
//! - **Keep the copy**: trash the original, then give the copy its name.
//! - **Keep both**: rename the copy to a name of the user's choosing.
//!
//! Every resolution goes through the Proton trash, never a delete, so a wrong
//! choice is undone by restoring from Trash. Resolution refuses to touch a file
//! that is open, being written, or still owed an upload, for the same reason the
//! sweep does: the node is about to change under the decision.
//!
//! Copies under My Files are named by their mountpoint-relative path. Copies
//! inside a synced folder are named by their absolute local path, so a resolve
//! request routes to the on-demand mount that serves them. A copy inside a
//! mirror folder is listed too, but it is a local file first: the sync engine
//! owns it, so it is resolved by editing the folder, not through here.
use super::*;
use pdfs_core::control::{ConflictInfo, ConflictKeep};
use sweep::conflict_base_name;
impl Core {
/// Where the user finds a conflict copy: its mountpoint-relative path under
/// My Files, or its absolute local path inside a synced folder. `None` for a
/// node no location of this machine shows, such as one in another
/// computer's device tree.
///
/// The sweep only flags what this can name, and the listing uses it too. A
/// flagged copy the listing cannot see reads as resolved in the activity
/// feed while it still sits in the folder (`docs/BUGS.md` B99).
pub(crate) fn conflict_path(&self, roots: &SearchRoots, uid: &NodeUid) -> Option<String> {
let uid = uid.to_string();
if let Ok(Some(path)) = self
.db
.path_relative_to(&self.primary_root_uid.to_string(), &uid)
{
return Some(path);
}
let drive_path = self.db.node_path(&uid).ok().flatten()?;
roots.resolve(&drive_path)
}
/// Every conflict copy this machine shows, with the file it is a copy of.
pub(crate) fn list_conflicts(&self) -> CoreResult<Vec<ConflictInfo>> {
let stored = self
.db
.load_all()
.map_err(|e| CoreError::internal(format!("listing conflicts: {e:?}")))?;
let nodes: Vec<Node> = stored
.into_iter()
.map(|stored| stored.node)
.filter(|node| !node.trashed && !node.is_folder() && self.is_own_or_virtual(&node.uid))
.collect();
let mut by_parent_name: HashMap<(Option<&NodeUid>, &str), &Node> = HashMap::new();
for node in &nodes {
by_parent_name.insert((node.parent_uid.as_ref(), node.name.as_str()), node);
}
let roots = self.search_roots();
let mut items = Vec::new();
for node in &nodes {
let Some(base) = conflict_base_name(&node.name) else {
continue;
};
let Some(path) = self.conflict_path(&roots, &node.uid) else {
continue;
};
let original = by_parent_name
.get(&(node.parent_uid.as_ref(), base.as_str()))
.copied();
items.push(ConflictInfo {
original_path: sibling_path(&path, &base),
original_exists: original.is_some(),
size: node_size(node),
modified: node.modification_time,
original_size: original.map(node_size),
original_modified: original.map(|o| o.modification_time),
identical: original.is_some_and(|o| {
node_size(o) == node_size(node)
&& node_content_sha1(node)
.is_some_and(|sha| node_content_sha1(o).as_deref() == Some(&sha))
}),
path,
});
}
items.sort_by(|a, b| a.path.cmp(&b.path));
Ok(items)
}
/// Resolve the conflict copy at mountpoint-relative `copy`. Returns a line
/// for the user saying what happened.
pub(crate) fn resolve_conflict(&self, copy: &Path, keep: &ConflictKeep) -> CoreResult<String> {
let copy_name = copy
.file_name()
.and_then(|n| n.to_str())
.ok_or_else(|| CoreError::invalid("the conflict copy has no file name"))?;
let base = conflict_base_name(copy_name)
.ok_or_else(|| CoreError::invalid(format!("{copy_name} is not a conflict copy")))?;
let original = copy.with_file_name(&base);
self.require_settled(copy)?;
let message = match keep {
ConflictKeep::Original => {
self.delete(copy)?;
self.log_activity(ActivityKind::Trash, copy_name, format!("kept {base}"), true);
format!("kept {base}; the copy is in Trash")
}
ConflictKeep::Copy => {
self.require_settled(&original)?;
self.delete(&original)?;
// The original is already in Trash. A failed rename leaves the
// copy under its conflict name, which loses nothing — say so
// rather than leave the user guessing where their file went.
self.rename(copy, &base).map_err(|e| {
e.context(&format!(
"{base} was moved to Trash, but the copy kept its conflict name"
))
})?;
self.log_activity(
ActivityKind::Rename,
&base,
format!("kept the conflict copy {copy_name}"),
true,
);
format!("kept the copy as {base}; the previous version is in Trash")
}
ConflictKeep::Both { name } => {
let name = name.trim();
if name == base || name == copy_name {
return Err(CoreError::invalid("choose a name that no other file has"));
}
self.rename(copy, name)?;
self.log_activity(ActivityKind::Rename, name, format!("was {copy_name}"), true);
format!("kept both; the copy is now {name}")
}
};
Ok(message)
}
/// Refuse to resolve around a file that is open, being written, or still
/// owed an upload: the decision would be about content that is changing.
fn require_settled(&self, rel: &Path) -> CoreResult<()> {
let (_, uid) = self.resolve(rel)?;
let queued = self.db.has_any_op(&uid.to_string()).unwrap_or(true);
if queued || self.is_busy(&uid) {
return Err(CoreError::conflict(format!(
"{} is still being written or uploaded; try again once it has synced",
rel.display()
)));
}
self.conflict_notified.lock().remove(&uid);
Ok(())
}
}
/// The path of `name` next to the file at `path`.
fn sibling_path(path: &str, name: &str) -> String {
match path.rsplit_once('/') {
Some((dir, _)) => format!("{dir}/{name}"),
None => name.to_string(),
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_original_sits_next_to_its_copy() {
assert_eq!(
sibling_path("a/b/f (sync-conflict 1).txt", "f.txt"),
"a/b/f.txt"
);
assert_eq!(sibling_path("f (sync-conflict 1).txt", "f.txt"), "f.txt");
}
}
File diff suppressed because it is too large. Load diff
+492 -130
View File
@@ -11,12 +11,14 @@ use std::collections::HashMap;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::Arc; use std::sync::Arc;
use fuser::{Config, MountOption, Session};
use parking_lot::Mutex; use parking_lot::Mutex;
use pdfs_core::batch;
use pdfs_core::control::{ use pdfs_core::control::{
ActivityKind, DeviceInfo, JobItem, SyncFolderInfo, SyncPhase, SyncProgress, ActivityKind, DeviceInfo, JobItem, MountKind, MountSpec, SyncFolderInfo, SyncPhase,
SyncProgress,
}; };
use pdfs_core::db::{StoredDevice, StoredSyncFolder}; use pdfs_core::db::{StoredDevice, StoredSyncFolder};
use pdfs_core::mounts::MountMode;
use pdfs_core::{CoreError, CoreResult}; use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::proton_sdk::ids::{DeviceUid, NodeUid}; use proton_drive_rs::proton_sdk::ids::{DeviceUid, NodeUid};
use proton_drive_rs::{DeviceType, Node}; use proton_drive_rs::{DeviceType, Node};
@@ -25,11 +27,69 @@ use tracing::{info, warn};
use super::sync::{self, base_name}; use super::sync::{self, base_name};
use super::{ use super::{
BackgroundSession, Core, ProtonFs, State, SwitchBlocked, clear_stale_mount, device_type_str, Core, SecondaryInsertRejection, SecondaryMount, State, SwitchBlocked, clear_stale_mount,
dir_is_empty, evict_dir_contents, fuse_connection_id, is_stale_mount, now_secs, parse_uid, device_type_str, dir_is_empty, evict_dir_contents, fuse_connection_id, is_stale_mount,
sync_folder_info, this_hostname, umount_session_unblocked, now_secs, parse_uid, spawn_session, sync_folder_info, this_hostname,
}; };
/// A mode that this daemon is allowed to apply.
///
/// Persisted and control-protocol values first pass through [`MountMode`] so
/// forward-compatible unknown values never reach either destructive transition.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum ApplicableMountMode {
Mirror,
OnDemand,
}
impl ApplicableMountMode {
fn parse(value: &str) -> Result<Self, String> {
match MountMode::from(value) {
MountMode::Mirror => Ok(Self::Mirror),
MountMode::OnDemand => Ok(Self::OnDemand),
MountMode::Unknown => Err(format!("unknown mode {value:?} (want mirror|ondemand)")),
}
}
const fn as_str(self) -> &'static str {
match self {
Self::Mirror => "mirror",
Self::OnDemand => "ondemand",
}
}
}
fn select_mode_transition(
current: &str,
target: &str,
) -> Result<Option<ApplicableMountMode>, String> {
let current = ApplicableMountMode::parse(current)
.map_err(|error| format!("unsupported current mode: {error}"))?;
let target = ApplicableMountMode::parse(target)
.map_err(|error| format!("unsupported target mode: {error}"))?;
Ok((current != target).then_some(target))
}
fn restore_snapshot_matches(snapshot: &StoredSyncFolder, current: &StoredSyncFolder) -> bool {
current.id == snapshot.id
&& current.local_path == snapshot.local_path
&& current.remote_uid == snapshot.remote_uid
&& current.remote_share_id == snapshot.remote_share_id
&& current.mode == "ondemand"
}
fn pending_intent_matches(pending: Option<&str>, expected: Option<&str>) -> bool {
expected.is_none_or(|expected| pending == Some(expected))
}
fn commit_after_teardown<T, E>(
teardown: Result<(), E>,
commit: impl FnOnce() -> T,
) -> Result<T, E> {
teardown?;
Ok(commit())
}
/// The device uid this machine has been adopted into, from `config.json`. /// The device uid this machine has been adopted into, from `config.json`.
/// ///
/// Read fresh each time rather than cached on [`Core`]: adoption rewrites the /// Read fresh each time rather than cached on [`Core`]: adoption rewrites the
@@ -57,6 +117,23 @@ pub(crate) fn set_adopted_device_uid(uid: Option<&str>) -> CoreResult<()> {
impl Core { impl Core {
// ---- devices ---------------------------------------------------------- // ---- devices ----------------------------------------------------------
/// Unified local-location listing for the control protocol.
pub(crate) fn list_locations(&self) -> CoreResult<Vec<MountSpec>> {
let progress = self.sync_progress.lock();
let mut locations = self
.db
.mount_list()
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?;
for location in &mut locations {
location.mounted = self.states.is_mounted_at(Path::new(&location.local_path));
if let MountKind::Device { sync_folder_id } = &location.kind {
location.progress = progress.get(sync_folder_id).cloned();
location.paused = self.sync_folder_paused(*sync_folder_id);
}
}
Ok(locations)
}
/// List the account's registered devices, flagging the one *this* machine /// List the account's registered devices, flagging the one *this* machine
/// syncs to so a front-end can treat it as more than another computer in the /// syncs to so a front-end can treat it as more than another computer in the
/// list — deleting it takes this machine's synced folders down with it. /// list — deleting it takes this machine's synced folders down with it.
@@ -150,6 +227,39 @@ impl Core {
Ok(format!("adopted device {name}")) Ok(format!("adopted device {name}"))
} }
/// The device a restore reads from: this machine's own for `None`, else
/// the registered device with that uid.
///
/// Another device is looked up without touching the cached device row —
/// restoring its folders here must not make this machine back up as it.
/// Its folders are then attached with that device's share id, so they sync
/// against the other computer's tree.
pub(crate) fn restore_source(&self, device: Option<&str>) -> CoreResult<StoredDevice> {
let Some(uid) = device else {
return self.ensure_device();
};
let remote = self
.rt
.block_on(self.client.enumerate_devices())
.map_err(|e| CoreError::from_api(&e, "enumerate devices"))?;
let d = remote
.into_iter()
.find(|d| d.uid.to_string() == uid)
.ok_or_else(|| CoreError::not_found(format!("no device {uid} in this account")))?;
Ok(StoredDevice {
uid: d.uid.to_string(),
share_id: d.share_id.to_string(),
root_uid: d.root_folder_uid.to_string(),
name: d
.name
.as_deref()
.ok()
.unwrap_or("(unnamed device)")
.to_string(),
created: d.creation_time,
})
}
// ---- device folder sync (devices.md, Phase 1) ------------------------- // ---- device folder sync (devices.md, Phase 1) -------------------------
/// Auto-register (or recover) this machine as a Proton Drive Device, caching /// Auto-register (or recover) this machine as a Proton Drive Device, caching
@@ -308,6 +418,15 @@ impl Core {
})? })?
.to_string(); .to_string();
// Syncing a local `.proton-drive-linux` would land on the folder holding
// this device's own profile (`find_device_child_folder` reuses by name),
// and the sync engine would then delete or overwrite it.
if name == pdfs_core::profile::PROFILE_DIR_NAME {
return Err(CoreError::invalid(format!(
"{name} is reserved for this device's profile backup"
)));
}
let device = self.ensure_device()?; let device = self.ensure_device()?;
let root_uid = parse_uid(&device.root_uid).ok_or_else(|| { let root_uid = parse_uid(&device.root_uid).ok_or_else(|| {
CoreError::internal(format!("bad device root uid: {}", device.root_uid)) CoreError::internal(format!("bad device root uid: {}", device.root_uid))
@@ -363,7 +482,8 @@ impl Core {
.into_iter() .into_iter()
.map(|f| { .map(|f| {
let live = progress.get(&f.id).cloned(); let live = progress.get(&f.id).cloned();
sync_folder_info(f, live) let paused = self.sync_folder_paused(f.id);
sync_folder_info(f, live, paused)
}) })
.collect()) .collect())
} }
@@ -439,6 +559,8 @@ impl Core {
/// Remove a synced folder from the sync set. `delete_remote` also deletes its /// Remove a synced folder from the sync set. `delete_remote` also deletes its
/// folder under the device root; otherwise the cloud copy is left in place. /// folder under the device root; otherwise the cloud copy is left in place.
pub(crate) fn remove_sync_folder(&self, id: i64, delete_remote: bool) -> CoreResult<()> { pub(crate) fn remove_sync_folder(&self, id: i64, delete_remote: bool) -> CoreResult<()> {
let lock = self.sync_lock(id);
let _guard = lock.lock();
let folder = self let folder = self
.db .db
.sync_folder_get(id) .sync_folder_get(id)
@@ -449,19 +571,23 @@ impl Core {
// folder the daemon no longer tracks, and nothing would ever unmount it. // folder the daemon no longer tracks, and nothing would ever unmount it.
// Tear it down first — including before trashing the remote tree it // Tear it down first — including before trashing the remote tree it
// serves, which would otherwise leave it answering for deleted nodes. // serves, which would otherwise leave it answering for deleted nodes.
if let Some((session, conn)) = self.mounts.lock().remove(&id) { // Taken in its own statement so the registry lock is released before the
if let Err(e) = umount_session_unblocked(session, conn) { // teardown: an edition-2024 `if let` chain holds the guard for the whole
warn!(id, error = %e, "unmount on-demand folder failed"); // body, and `teardown` unmounts and joins the session thread — which can
} else { // be inside a multi-second transfer. The shutdown path takes the same
info!(id, path = %folder.local_path, "unmounted on-demand folder"); // lock, so holding it here risks systemd's stop timeout landing mid-
} // unmount and stranding the endpoint.
} let taken = self.mounts.lock().remove(&id);
if delete_remote if let Some(mount) = taken
&& let Some(uid) = parse_uid(&folder.remote_uid) && let Err(error) = commit_after_teardown(mount.teardown(), || ())
&& let Err(e) = self.rt.block_on(self.client.trash_nodes(&[uid]))
{ {
warn!(id, error = %e, "delete remote device folder failed"); let _ = self.db.sync_folder_set_state(id, "error", now_secs());
return Err(CoreError::internal(format!(
"cannot remove synced folder: unmount {} failed: {error}",
folder.local_path
)));
} }
info!(id, path = %folder.local_path, "unmounted on-demand folder");
if !self if !self
.db .db
.sync_folder_remove(id) .sync_folder_remove(id)
@@ -469,8 +595,19 @@ impl Core {
{ {
return Err(CoreError::invalid(format!("no synced folder with id {id}"))); return Err(CoreError::invalid(format!("no synced folder with id {id}")));
} }
drop(_guard);
self.clear_sync_folder_paused(id);
// Stop watching the folder we just dropped. // Stop watching the folder we just dropped.
let _ = self.sync_tx.send(sync::SyncMsg::Rewatch); let _ = self.sync_tx.send(sync::SyncMsg::Rewatch);
if delete_remote
&& let Some(uid) = parse_uid(&folder.remote_uid)
&& let Err(e) = self
.rt
.block_on(self.client.trash_nodes(&[uid]))
.and_then(batch::into_unit)
{
warn!(id, error = %e, "delete remote device folder failed");
}
Ok(()) Ok(())
} }
@@ -489,20 +626,14 @@ impl Core {
/// (devices.md Phase 3). /// (devices.md Phase 3).
pub(crate) fn fork_state(&self) -> Core { pub(crate) fn fork_state(&self) -> Core {
let mut fork = self.clone(); let mut fork = self.clone();
fork.state = Arc::new(Mutex::new(State { fork.primary = false;
entries: HashMap::new(), let share_access = self.state().share_access.clone();
by_uid: HashMap::new(), fork.state = Arc::new(Mutex::new(State::new(self.db.clone(), share_access, 2)));
children: HashMap::new(),
next_ino: 2,
active_writes: HashMap::new(),
handles: HashMap::new(),
next_fh: 1,
db: self.db.clone(),
}));
// A fresh inode space needs a fresh notification channel: this fork's // A fresh inode space needs a fresh notification channel: this fork's
// session is the only one that knows these inodes, so it must be the one // session is the only one that knows these inodes, so it must be the one
// notified about them. Filled in by `spawn_ondemand_mount`. // notified about them. Filled in by `spawn_ondemand_mount`.
fork.notifier = Arc::new(OnceLock::new()); fork.notifier = Arc::new(OnceLock::new());
fork.session_live = Arc::new(std::sync::atomic::AtomicBool::new(false));
// Size upgrades are keyed by inode, which is per-fork too. // Size upgrades are keyed by inode, which is per-fork too.
fork.size_upgrades = Arc::new(Mutex::new(HashMap::new())); fork.size_upgrades = Arc::new(Mutex::new(HashMap::new()));
fork fork
@@ -520,49 +651,91 @@ impl Core {
/// running — which, seeing a queued `ondemand`, also stops pulling down files /// running — which, seeing a queued `ondemand`, also stops pulling down files
/// it is about to evict ([`Core::push_pass`]). /// it is about to evict ([`Core::push_pass`]).
pub(crate) fn request_sync_folder_mode(&self, id: i64, mode: &str) -> CoreResult<String> { pub(crate) fn request_sync_folder_mode(&self, id: i64, mode: &str) -> CoreResult<String> {
if mode != "mirror" && mode != "ondemand" { // Preserve the existing control behavior: reject an unsupported request
return Err(CoreError::invalid(format!( // before looking up the folder, then validate the persisted source too.
"unknown mode {mode:?} (want mirror|ondemand)" let target_mode = ApplicableMountMode::parse(mode).map_err(CoreError::invalid)?;
)));
}
let folder = self let folder = self
.db .db
.sync_folder_get(id) .sync_folder_get(id)
.map_err(|e| CoreError::internal(format!("db: {e:?}")))? .map_err(|e| CoreError::internal(format!("db: {e:?}")))?
.ok_or_else(|| CoreError::not_found(format!("no synced folder with id {id}")))?; .ok_or_else(|| CoreError::not_found(format!("no synced folder with id {id}")))?;
if folder.mode == mode { let current_mode = ApplicableMountMode::parse(&folder.mode)
// Already there. A queued switch the other way is the user changing .map_err(|error| CoreError::invalid(format!("unsupported current mode: {error}")))?;
// their mind back before it landed, so withdraw it. // A switch waits on the folder's reconcile, which a pause skips; refuse
if folder.pending_mode.is_some() { // it up front rather than queue it indefinitely.
self.db if current_mode != target_mode && self.sync_folder_paused(id) {
.sync_folder_set_pending_mode(id, None) return Err(CoreError::invalid(
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?; "this folder is paused; resume it before changing its mode",
let _ = self.sync_tx.send(sync::SyncMsg::Rewatch); ));
return Ok(format!("staying {mode}"));
}
return Ok(format!("already {mode}"));
} }
if current_mode == target_mode {
let mode_str = target_mode.as_str();
let lock = self.sync_lock(id);
match lock.try_lock() {
Some(_guard) => {
let current = self
.db
.sync_folder_get(id)
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?
.ok_or_else(|| {
CoreError::not_found(format!("no synced folder with id {id}"))
})?;
let locked_mode =
ApplicableMountMode::parse(&current.mode).map_err(|error| {
CoreError::invalid(format!("unsupported current mode: {error}"))
})?;
if locked_mode == target_mode {
self.db
.sync_folder_set_pending_mode(id, None)
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?;
let _ = self.sync_tx.send(sync::SyncMsg::Rewatch);
return Ok(if current.pending_mode.is_some() {
format!("staying {mode_str}")
} else {
format!("already {mode_str}")
});
}
}
None => {
// A transition or reconcile owns the folder lock. Record
// this request instead of clearing the old target: if a
// queued switch already passed its recheck, its mode write
// preserves this opposite marker and the engine switches
// back next.
self.db
.sync_folder_set_pending_mode(id, Some(mode_str))
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?;
let _ = self.sync_tx.send(sync::SyncMsg::Reconcile(id));
return Ok(format!("staying {mode_str}"));
}
}
}
let mode = target_mode;
let mode_str = target_mode.as_str();
match self.apply_sync_folder_mode(id, mode) { match self.apply_sync_folder_mode(id, mode, None) {
Ok(message) => Ok(message), Ok(message) => Ok(message),
// Not switchable this instant — remember the intent instead of making // Not switchable this instant — remember the intent instead of making
// the user retry, and kick a pass to clear whatever is in the way. // the user retry, and kick a pass to clear whatever is in the way.
Err(SwitchBlocked::NotNow) => { Err(SwitchBlocked::NotNow) => {
self.db self.db
.sync_folder_set_pending_mode(id, Some(mode)) .sync_folder_set_pending_mode(id, Some(mode_str))
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?; .map_err(|e| CoreError::internal(format!("db: {e:?}")))?;
let _ = self.sync_tx.send(sync::SyncMsg::Reconcile(id)); let _ = self.sync_tx.send(sync::SyncMsg::Reconcile(id));
Ok(match mode { Ok(match mode {
"ondemand" => format!( ApplicableMountMode::OnDemand => format!(
"{} will go on-demand once its local changes are uploaded", "{} will go on-demand once its local changes are uploaded",
base_name(&folder.local_path) base_name(&folder.local_path)
), ),
_ => format!( ApplicableMountMode::Mirror => format!(
"{} will start mirroring once the current sync finishes", "{} will start mirroring once the current sync finishes",
base_name(&folder.local_path) base_name(&folder.local_path)
), ),
}) })
} }
Err(SwitchBlocked::Superseded) => Err(CoreError::internal(
"direct mode request was unexpectedly superseded",
)),
Err(SwitchBlocked::Failed(e)) => Err(CoreError::internal(e)), Err(SwitchBlocked::Failed(e)) => Err(CoreError::internal(e)),
} }
} }
@@ -576,27 +749,45 @@ impl Core {
let Ok(Some(folder)) = self.db.sync_folder_get(id) else { let Ok(Some(folder)) = self.db.sync_folder_get(id) else {
return; return;
}; };
let Some(mode) = folder.pending_mode.as_deref() else { let Some(expected_mode) = folder.pending_mode.as_deref() else {
return; return;
}; };
if folder.mode == mode { let mode = match select_mode_transition(&folder.mode, expected_mode) {
let _ = self.db.sync_folder_set_pending_mode(id, None); Ok(Some(mode)) => mode,
return; Ok(None) => {
} let _ = self.db.sync_folder_clear_pending_mode_if(id, expected_mode);
match self.apply_sync_folder_mode(id, mode) { return;
}
Err(error) => {
warn!(
id,
current_mode = %folder.mode,
pending_mode = %expected_mode,
%error,
"withdrawing unsafe queued mode transition"
);
let _ = self.db.sync_folder_clear_pending_mode_if(id, expected_mode);
return;
}
};
let mode_str = mode.as_str();
match self.apply_sync_folder_mode(id, mode, Some(expected_mode)) {
Ok(message) => { Ok(message) => {
info!(id, mode, "applied queued mode switch"); info!(id, mode = mode_str, "applied queued mode switch");
self.log_activity(ActivityKind::Sync, &message, "", true); self.log_activity(ActivityKind::Sync, &message, "", true);
} }
// Still blocked: the pass could not get everything up, so the local copy // Still blocked: the pass could not get everything up, so the local copy
// is not safe to evict yet. Leave the request standing — the next pass // is not safe to evict yet. Leave the request standing — the next pass
// (poll, or the retry the engine schedules) tries again. // (poll, or the retry the engine schedules) tries again.
Err(SwitchBlocked::NotNow) => { Err(SwitchBlocked::NotNow) => {
info!(id, mode, "queued mode switch still waiting"); info!(id, mode = mode_str, "queued mode switch still waiting");
}
Err(SwitchBlocked::Superseded) => {
info!(id, mode = mode_str, "queued mode switch was replaced");
} }
Err(SwitchBlocked::Failed(e)) => { Err(SwitchBlocked::Failed(e)) => {
warn!(id, mode, error = %e, "queued mode switch failed; withdrawing"); warn!(id, mode = mode_str, error = %e, "queued mode switch failed; withdrawing");
let _ = self.db.sync_folder_set_pending_mode(id, None); let _ = self.db.sync_folder_clear_pending_mode_if(id, expected_mode);
self.log_activity( self.log_activity(
ActivityKind::Sync, ActivityKind::Sync,
format!("couldn't switch {}", base_name(&folder.local_path)), format!("couldn't switch {}", base_name(&folder.local_path)),
@@ -622,8 +813,10 @@ impl Core {
pub(crate) fn apply_sync_folder_mode( pub(crate) fn apply_sync_folder_mode(
&self, &self,
id: i64, id: i64,
mode: &str, mode: ApplicableMountMode,
expected_pending: Option<&str>,
) -> Result<String, SwitchBlocked> { ) -> Result<String, SwitchBlocked> {
let mode_str = mode.as_str();
// Hold the folder's lock for the whole switch so no reconcile pass can be // Hold the folder's lock for the whole switch so no reconcile pass can be
// running over the tree we are about to evict (or start while we mount over // running over the tree we are about to evict (or start while we mount over
// it). A pass in flight holds the lock for its full duration, so `try_lock` // it). A pass in flight holds the lock for its full duration, so `try_lock`
@@ -641,13 +834,18 @@ impl Core {
.sync_folder_get(id) .sync_folder_get(id)
.map_err(|e| SwitchBlocked::Failed(format!("db: {e:?}")))? .map_err(|e| SwitchBlocked::Failed(format!("db: {e:?}")))?
.ok_or_else(|| SwitchBlocked::Failed(format!("no synced folder with id {id}")))?; .ok_or_else(|| SwitchBlocked::Failed(format!("no synced folder with id {id}")))?;
if folder.mode == mode { if !pending_intent_matches(folder.pending_mode.as_deref(), expected_pending) {
return Ok(format!("already {mode}")); return Err(SwitchBlocked::Superseded);
}
let current_mode = ApplicableMountMode::parse(&folder.mode)
.map_err(|e| SwitchBlocked::Failed(format!("unsupported current mode: {e}")))?;
if current_mode == mode {
return Ok(format!("already {mode_str}"));
} }
let local = PathBuf::from(&folder.local_path); let local = PathBuf::from(&folder.local_path);
match mode { match mode {
"ondemand" => { ApplicableMountMode::OnDemand => {
// Only flip a folder that is fully in sync — a failed reconcile means // Only flip a folder that is fully in sync — a failed reconcile means
// local edits could still be un-uploaded, and we are about to delete // local edits could still be un-uploaded, and we are about to delete
// the local copy. Not an error: a pass makes this true, and the queued // the local copy. Not an error: a pass makes this true, and the queued
@@ -694,25 +892,31 @@ impl Core {
if let Err(e) = evict_dir_contents(&local) { if let Err(e) = evict_dir_contents(&local) {
warn!(id, path = %local.display(), error = %e, "evict local dir contents failed"); warn!(id, path = %local.display(), error = %e, "evict local dir contents failed");
} }
let session = self.spawn_ondemand_mount(&local, root).map_err(|e| { let mount = self.spawn_ondemand_mount(&local, root).map_err(|e| {
SwitchBlocked::Failed(format!( SwitchBlocked::Failed(format!(
"mount ondemand failed after local cache eviction; switch back to mirror to restore it: {e}" "mount ondemand failed after local cache eviction; switch back to mirror to restore it: {e}"
)) ))
})?; })?;
self.mounts self.publish_secondary_mount(id, mount)
.lock() .map_err(SwitchBlocked::Failed)?;
.insert(id, (session, fuse_connection_id(&local)));
let _ = self.sync_tx.send(sync::SyncMsg::Rewatch); let _ = self.sync_tx.send(sync::SyncMsg::Rewatch);
self.db.sync_folder_set_state(id, "idle", now_secs()).ok(); self.db.sync_folder_set_state(id, "idle", now_secs()).ok();
info!(id, path = %local.display(), "mounted sync folder on-demand"); info!(id, path = %local.display(), "mounted sync folder on-demand");
Ok(format!("{} is now on-demand", local.display())) Ok(format!("{} is now on-demand", local.display()))
} }
_ => { ApplicableMountMode::Mirror => {
// ondemand→mirror: tear down the secondary mount first. // ondemand→mirror: tear down the secondary mount first. Taken
if let Some((session, conn)) = self.mounts.lock().remove(&id) // before the teardown so the registry lock is not held across
&& let Err(e) = umount_session_unblocked(session, conn) // the unmount and thread join (see `remove_sync_folder`).
let taken = self.mounts.lock().remove(&id);
if let Some(mount) = taken
&& let Err(error) = commit_after_teardown(mount.teardown(), || ())
{ {
warn!(id, error = %e, "unmount on-demand folder failed"); let _ = self.db.sync_folder_set_state(id, "error", now_secs());
return Err(SwitchBlocked::Failed(format!(
"unmount {} failed; folder remains on-demand: {error}",
local.display()
)));
} }
// The evicted local tree makes the old baseline lie ("everything // The evicted local tree makes the old baseline lie ("everything
// deleted locally"); clear it so the reconcile is a pure download. // deleted locally"); clear it so the reconcile is a pure download.
@@ -740,23 +944,52 @@ impl Core {
&self, &self,
local: &Path, local: &Path,
root: Node, root: Node,
) -> CoreResult<BackgroundSession> { ) -> CoreResult<SecondaryMount> {
clear_stale_mount(local);
let mut config = Config::default();
config.mount_options = vec![
MountOption::FSName("protondrive".to_string()),
MountOption::Subtype("protondrive".to_string()),
MountOption::DefaultPermissions,
];
let core = self.fork_state(); let core = self.fork_state();
let fs = ProtonFs::new(core.clone(), root); let session_live = core.session_live.clone();
let session = Session::new(fs, local, &config) // The fork owns this state/path registration. `spawn_session` only
.and_then(|s| s.spawn()) // marks it live after FUSE has spawned successfully.
core.register_state(local);
let session = spawn_session(&core, local, root)
.map_err(|e| CoreError::internal(format!("mount {}: {e}", local.display())))?; .map_err(|e| CoreError::internal(format!("mount {}: {e}", local.display())))?;
// Hand this fork its own session's notification channel, so a background Ok(SecondaryMount::new(
// size upgrade can drop the kernel's cached attrs for these inodes. session,
let _ = core.notifier.set(session.notifier()); fuse_connection_id(local),
Ok(session) session_live,
))
}
/// Publish one newly spawned secondary or explicitly tear it down when
/// shutdown has closed publication or another session already owns the id.
fn publish_secondary_mount(&self, id: i64, mount: SecondaryMount) -> Result<(), String> {
let rejected = self.mounts.lock().insert(id, mount);
let Err((reason, mount)) = rejected else {
return Ok(());
};
let reason = match reason {
SecondaryInsertRejection::Closed => "daemon is shutting down",
SecondaryInsertRejection::Duplicate => "folder already has a live mount",
};
match mount.teardown() {
Ok(()) => Err(reason.to_string()),
Err(error) => Err(format!("{reason}; rejected mount teardown failed: {error}")),
}
}
fn mark_restore_error_if_current(&self, snapshot: &StoredSyncFolder) {
let lock = self.sync_lock(snapshot.id);
let _guard = lock.lock();
if self
.db
.sync_folder_get(snapshot.id)
.ok()
.flatten()
.is_some_and(|current| restore_snapshot_matches(snapshot, &current))
{
let _ = self
.db
.sync_folder_set_state(snapshot.id, "error", now_secs());
}
} }
/// Re-establish FUSE mounts for folders left in `ondemand` mode across a daemon /// Re-establish FUSE mounts for folders left in `ondemand` mode across a daemon
@@ -777,14 +1010,57 @@ impl Core {
if folder.mode != "ondemand" { if folder.mode != "ondemand" {
continue; continue;
} }
let local = PathBuf::from(&folder.local_path); let Some(root_uid) = parse_uid(&folder.remote_uid) else {
// A daemon killed before it could unmount (systemd's stop timeout, warn!(id = folder.id, "restore on-demand: bad remote uid");
// say) leaves this path attached to a dead FUSE connection: it still self.mark_restore_error_if_current(&folder);
// exists, but every stat on it fails with ENOTCONN — so `is_dir()` continue;
// reads it as *missing* and the folder gets parked in `error` on };
// every subsequent start, with no way back but a manual let root = match self
// `fusermount -u`. Detach the corpse first; the directory .rt
// underneath then restores normally. .block_on(self.client.enumerate_nodes(std::slice::from_ref(&root_uid)))
{
Ok(v) => match v.into_iter().next() {
Some(n) => n,
None => {
warn!(id = folder.id, "restore on-demand: remote folder gone");
self.mark_restore_error_if_current(&folder);
continue;
}
},
Err(e) => {
warn!(id = folder.id, error = %e, "restore on-demand: fetch remote failed");
self.mark_restore_error_if_current(&folder);
continue;
}
};
let lock = self.sync_lock(folder.id);
let _guard = lock.lock();
let current = match self.db.sync_folder_get(folder.id) {
Ok(Some(current)) if restore_snapshot_matches(&folder, &current) => current,
Ok(_) => {
info!(
id = folder.id,
"restore on-demand: folder changed while fetching"
);
continue;
}
Err(error) => {
warn!(
id = folder.id,
?error,
"restore on-demand: cannot re-read folder"
);
continue;
}
};
if !self.mounts.lock().is_accepting() {
info!(id = folder.id, "restore on-demand: shutdown in progress");
continue;
}
let local = PathBuf::from(&current.local_path);
// A daemon killed before it could unmount can leave a dead FUSE
// connection over the path. Clear it only after the row identity
// and mode have been revalidated under the folder lock.
if is_stale_mount(&local) { if is_stale_mount(&local) {
warn!(id = folder.id, path = %local.display(), "restore on-demand: clearing stale mount"); warn!(id = folder.id, path = %local.display(), "restore on-demand: clearing stale mount");
clear_stale_mount(&local); clear_stale_mount(&local);
@@ -796,11 +1072,6 @@ impl Core {
.sync_folder_set_state(folder.id, "error", now_secs()); .sync_folder_set_state(folder.id, "error", now_secs());
continue; continue;
} }
// An `ondemand` folder's local dir is empty by construction — the switch
// evicts it. Finding files there means the row is lying (a switch that
// died between persisting the mode and evicting), and mounting over them
// would hide real local data behind the remote tree. Leave the files
// alone and let the user resolve it.
match dir_is_empty(&local) { match dir_is_empty(&local) {
Ok(true) => {} Ok(true) => {}
Ok(false) => { Ok(false) => {
@@ -822,40 +1093,16 @@ impl Core {
continue; continue;
} }
} }
let Some(root_uid) = parse_uid(&folder.remote_uid) else { match self.spawn_ondemand_mount(&local, root) {
warn!(id = folder.id, "restore on-demand: bad remote uid"); Ok(mount) => match self.publish_secondary_mount(folder.id, mount) {
continue; Ok(()) => {
}; let _ = self.db.sync_folder_set_state(folder.id, "idle", now_secs());
let root = match self info!(id = folder.id, path = %local.display(), "remounted on-demand folder");
.rt }
.block_on(self.client.enumerate_nodes(std::slice::from_ref(&root_uid))) Err(error) => {
{ warn!(id = folder.id, %error, "restore on-demand: mount rejected");
Ok(v) => match v.into_iter().next() {
Some(n) => n,
None => {
warn!(id = folder.id, "restore on-demand: remote folder gone");
let _ = self
.db
.sync_folder_set_state(folder.id, "error", now_secs());
continue;
} }
}, },
Err(e) => {
warn!(id = folder.id, error = %e, "restore on-demand: fetch remote failed");
let _ = self
.db
.sync_folder_set_state(folder.id, "error", now_secs());
continue;
}
};
match self.spawn_ondemand_mount(&local, root) {
Ok(session) => {
self.mounts
.lock()
.insert(folder.id, (session, fuse_connection_id(&local)));
let _ = self.db.sync_folder_set_state(folder.id, "idle", now_secs());
info!(id = folder.id, path = %local.display(), "remounted on-demand folder");
}
Err(e) => { Err(e) => {
warn!(id = folder.id, error = %e, "restore on-demand: mount failed"); warn!(id = folder.id, error = %e, "restore on-demand: mount failed");
let _ = self let _ = self
@@ -866,3 +1113,118 @@ impl Core {
} }
} }
} }
#[cfg(test)]
mod tests {
use pdfs_core::db::StoredSyncFolder;
use super::{
ApplicableMountMode, commit_after_teardown, pending_intent_matches,
restore_snapshot_matches, select_mode_transition,
};
#[test]
fn unknown_mode_cannot_select_a_destructive_transition() {
let mut unmounts = 0;
let mut baseline_clears = 0;
let mut mode_writes = 0;
if let Ok(mode) = ApplicableMountMode::parse("streaming") {
match mode {
ApplicableMountMode::Mirror => {
unmounts += 1;
baseline_clears += 1;
mode_writes += 1;
}
ApplicableMountMode::OnDemand => {
mode_writes += 1;
}
}
}
assert_eq!((unmounts, baseline_clears, mode_writes), (0, 0, 0));
assert_eq!(
ApplicableMountMode::parse("streaming"),
Err("unknown mode \"streaming\" (want mirror|ondemand)".to_string())
);
assert_eq!(
ApplicableMountMode::parse("mirror"),
Ok(ApplicableMountMode::Mirror)
);
assert_eq!(
ApplicableMountMode::parse("ondemand"),
Ok(ApplicableMountMode::OnDemand)
);
}
#[test]
fn unknown_source_mode_with_pending_mirror_cannot_select_a_transition() {
let mut destructive_helper_reached = false;
let selected = select_mode_transition("streaming", "mirror");
if let Ok(Some(_)) = selected {
destructive_helper_reached = true;
}
assert!(!destructive_helper_reached);
assert_eq!(
selected,
Err(
"unsupported current mode: unknown mode \"streaming\" (want mirror|ondemand)"
.to_string()
)
);
}
fn folder(local_path: &str, remote_uid: &str, mode: &str) -> StoredSyncFolder {
StoredSyncFolder {
id: 7,
local_path: local_path.to_string(),
remote_uid: remote_uid.to_string(),
remote_share_id: "share".to_string(),
mode: mode.to_string(),
pending_mode: None,
state: "idle".to_string(),
last_sync: 0,
}
}
#[test]
fn stale_restore_identity_or_mode_is_rejected() {
let snapshot = folder("/mnt/folder", "volume~node", "ondemand");
assert!(restore_snapshot_matches(
&snapshot,
&folder("/mnt/folder", "volume~node", "ondemand")
));
assert!(!restore_snapshot_matches(
&snapshot,
&folder("/mnt/replaced", "volume~node", "ondemand")
));
assert!(!restore_snapshot_matches(
&snapshot,
&folder("/mnt/folder", "volume~other", "ondemand")
));
assert!(!restore_snapshot_matches(
&snapshot,
&folder("/mnt/folder", "volume~node", "mirror")
));
}
#[test]
fn teardown_failure_prevents_follow_up_state_changes() {
let mut commits = 0;
let result = commit_after_teardown::<_, &str>(Err("busy"), || {
commits += 1;
});
assert_eq!(result, Err("busy"));
assert_eq!(commits, 0);
commit_after_teardown::<_, &str>(Ok(()), || commits += 1).unwrap();
assert_eq!(commits, 1);
}
#[test]
fn canceled_or_replaced_queued_intent_cannot_transition() {
assert!(pending_intent_matches(Some("ondemand"), Some("ondemand")));
assert!(!pending_intent_matches(None, Some("ondemand")));
assert!(!pending_intent_matches(Some("mirror"), Some("ondemand")));
assert!(pending_intent_matches(Some("mirror"), None));
}
}
+254
View File
@@ -0,0 +1,254 @@
//! What the daemon is doing right now, for the case where it has stopped doing
//! anything at all.
//!
//! On 2026-09-18 the daemon hung: it consumed 43 minutes of CPU over 20 hours,
//! peaked at 5.2 GiB resident, went silent at 17:10, and only came back after a
//! manual `systemctl --user restart`. There was no panic, so nothing in the
//! journal said what it had been waiting on — and nothing in the daemon could be
//! asked, either. This module is the answer to "what was it stuck on?".
//!
//! Everything here is built to work *while* the daemon is wedged, which rules
//! out the obvious implementations: no lock that a stuck job might hold, no
//! database query on the calling thread, no waiting on the worker pool. A
//! diagnostic that blocks on the hang it is diagnosing reports nothing at
//! exactly the moment it matters.
use std::sync::LazyLock;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Instant;
use parking_lot::Mutex;
use pdfs_core::control::{Diagnostics, InflightRequest, WorkerState};
use super::Core;
/// When this process started. Read at the first touch, which
/// [`super::mount::mount`] makes happen during startup so the uptime is the
/// daemon's, not the first diagnostics request's.
static STARTED: LazyLock<Instant> = LazyLock::new(Instant::now);
/// Control requests currently being served, keyed by a monotonic ticket.
///
/// A `Vec` rather than a map because it holds at most `MAX_CONTROL_HANDLERS`
/// entries, and the only operations are push, remove-by-ticket and read-all.
static INFLIGHT: Mutex<Vec<(u64, String, Instant)>> = Mutex::new(Vec::new());
static NEXT_TICKET: AtomicU64 = AtomicU64::new(0);
/// Start the daemon's uptime clock. Called once, from the mount path.
pub(crate) fn start_clock() {
LazyLock::force(&STARTED);
}
/// Registers one in-flight control request for as long as it is held.
pub(crate) struct InflightGuard(u64);
impl InflightGuard {
pub(crate) fn new(kind: String) -> Self {
let ticket = NEXT_TICKET.fetch_add(1, Ordering::Relaxed);
INFLIGHT.lock().push((ticket, kind, Instant::now()));
Self(ticket)
}
}
impl Drop for InflightGuard {
fn drop(&mut self) {
INFLIGHT.lock().retain(|(ticket, _, _)| *ticket != self.0);
}
}
/// The variant name of a serialised [`pdfs_core::control::Request`], without
/// parsing it into one.
///
/// serde's external tagging makes this a property of the text: a unit variant is
/// the bare string `"Status"`, anything else is an object whose single key is
/// the variant name. Reading the name off the line rather than matching on 80
/// variants keeps this cheap, and keeps it correct when a variant is added.
pub(crate) fn request_kind(line: &str) -> String {
let line = line.trim();
let name = if let Some(rest) = line.strip_prefix('{') {
rest.trim_start()
.strip_prefix('"')
.and_then(|rest| rest.split('"').next())
} else {
line.strip_prefix('"')
.and_then(|rest| rest.split('"').next())
};
let name = name.unwrap_or("unknown");
// A bare `Refresh` says nothing about which view wedged, and the three
// scopes cost wildly different amounts of work: a directory refresh is two
// cheap calls, a photos refresh can pull a whole library. The scope is one
// more peek into the same line.
if name == "Refresh"
&& let Some(scope) = tagged_name_after(line, "\"scope\"")
{
return format!("Refresh({scope})");
}
name.to_string()
}
/// The variant name serde wrote for the value of `key`, whether that value is a
/// bare string (a unit variant) or an object whose single key is the name.
fn tagged_name_after(line: &str, key: &str) -> Option<String> {
let rest = line.split_once(key)?.1.trim_start();
let rest = rest.strip_prefix(':')?.trim_start();
let rest = rest.strip_prefix('{').map_or(rest, str::trim_start);
rest.strip_prefix('"')?
.split('"')
.next()
.map(str::to_string)
}
/// Resident set size in bytes, or `0` when `/proc` did not answer.
///
/// From `VmRSS` in `/proc/self/status`, which is already in kB — `statm` would
/// need the page size, and this needs no dependency at all.
pub(crate) fn rss_bytes() -> u64 {
let Ok(status) = std::fs::read_to_string("/proc/self/status") else {
return 0;
};
parse_vm_rss(&status)
}
/// The `VmRSS` line of a `/proc/<pid>/status`, in bytes.
fn parse_vm_rss(status: &str) -> u64 {
status
.lines()
.find_map(|line| line.strip_prefix("VmRSS:"))
.and_then(|rest| rest.split_whitespace().next())
.and_then(|kb| kb.parse::<u64>().ok())
.map_or(0, |kb| kb.saturating_mul(1024))
}
impl Core {
/// Assemble a [`Diagnostics`] report.
///
/// The pending-op count is the one field that needs the database, so it is
/// read on a throwaway thread with a short deadline: a daemon wedged *on*
/// the database must still be able to report everything else about itself.
pub(crate) fn diagnostics(&self) -> Diagnostics {
let pool = self.workers.snapshot();
let now = Instant::now();
let mut inflight: Vec<InflightRequest> = INFLIGHT
.try_lock()
.map(|entries| {
entries
.iter()
.map(|(_, kind, started)| InflightRequest {
kind: kind.clone(),
age_secs: now.duration_since(*started).as_secs(),
})
.collect()
})
.unwrap_or_default();
inflight.sort_by_key(|request| std::cmp::Reverse(request.age_secs));
Diagnostics {
uptime_secs: STARTED.elapsed().as_secs(),
rss_bytes: rss_bytes(),
workers: pool
.workers
.into_iter()
.map(|worker| WorkerState {
name: worker.name,
busy: worker.busy,
label: worker.label,
age_secs: worker.age_ms / 1000,
})
.collect(),
meta_queued: pool.meta_queued,
transfer_queued: pool.transfer_queued,
meta_completed: pool.meta_completed,
transfer_completed: pool.transfer_completed,
inflight,
control_handlers: super::control::active_handlers(),
control_handler_limit: super::control::handler_limit(),
pending_ops: self.pending_ops_quickly(),
}
}
/// Queued mutations, or `0` if the database did not answer within
/// [`PENDING_COUNT_DEADLINE`].
fn pending_ops_quickly(&self) -> u64 {
let db = self.db.clone();
let (tx, rx) = std::sync::mpsc::channel();
if std::thread::Builder::new()
.name("pdfs-diagnostics".into())
.spawn(move || {
let counts = db.pending_op_counts().unwrap_or_default();
let total = counts.uploads.max(0) as u64 + counts.changes.max(0) as u64;
let _ = tx.send(total);
})
.is_err()
{
return 0;
}
rx.recv_timeout(PENDING_COUNT_DEADLINE).unwrap_or(0)
}
}
/// How long the report waits for the database before reporting without it.
const PENDING_COUNT_DEADLINE: std::time::Duration = std::time::Duration::from_millis(500);
#[cfg(test)]
mod tests {
use super::*;
/// The request name is read off the wire format, so it must match both
/// shapes serde produces.
#[test]
fn the_request_kind_comes_off_both_wire_shapes() {
assert_eq!(request_kind("\"Status\""), "Status");
assert_eq!(request_kind("{\"Pin\":{\"path\":\"a\"}}"), "Pin");
assert_eq!(request_kind(" { \"Restore\" : {} } "), "Restore");
assert_eq!(request_kind("garbage"), "unknown");
}
/// A stuck `Refresh` is only actionable if the report says which view it
/// was refreshing, so the scope rides along with the name.
#[test]
fn a_refresh_reports_its_scope() {
assert_eq!(
request_kind("{\"Refresh\":{\"scope\":{\"Photos\":{\"full\":true}}}}"),
"Refresh(Photos)"
);
assert_eq!(
request_kind("{\"Refresh\":{\"scope\":{\"Dir\":{\"path\":\"a\"}}}}"),
"Refresh(Dir)"
);
assert_eq!(
request_kind("{\"Refresh\":{\"scope\":\"Trash\"}}"),
"Refresh(Trash)"
);
// A shape this does not recognise still names the request.
assert_eq!(request_kind("{\"Refresh\":{}}"), "Refresh");
}
/// A guard registers for its lifetime and nothing longer: a handler that
/// returns must not leave a permanent "in flight" entry behind.
#[test]
fn a_guard_registers_only_while_it_lives() {
let before = INFLIGHT.lock().len();
{
let _guard = InflightGuard::new("OpenFile".into());
assert_eq!(INFLIGHT.lock().len(), before + 1);
}
assert_eq!(INFLIGHT.lock().len(), before);
}
/// Whatever else it reports, the resident size has to be a real reading.
#[test]
fn rss_is_readable() {
assert!(rss_bytes() > 0);
}
/// `VmRSS` is in kB, and a status without one must read as unknown rather
/// than as a daemon using no memory.
#[test]
fn vm_rss_is_kilobytes_and_absence_is_zero() {
assert_eq!(
parse_vm_rss("VmPeak:\t 100 kB\nVmRSS:\t 2048 kB\n"),
2048 * 1024
);
assert_eq!(parse_vm_rss("VmPeak:\t 100 kB\n"), 0);
}
}
File diff suppressed because it is too large. Load diff
+321
View File
@@ -0,0 +1,321 @@
//! Push notifications for front-ends: the daemon's side of
//! [`Request::Subscribe`](pdfs_core::control::Request::Subscribe).
//!
//! Front-ends used to poll. The app asked for the status every two seconds, the
//! tray every three, and pages re-read themselves on a timer, whether or not
//! anything had changed. Now each subscription owns a [`Mailbox`] in the
//! [`EventHub`]. Code that changes something publishes a [`Topic`], and
//! [`run_event_sampler`] watches the few things that change without passing
//! through one place (the status, transfers, sync progress) and publishes them
//! when they differ.
use std::collections::BTreeSet;
use std::sync::Arc;
use std::time::Duration;
use parking_lot::{Condvar, Mutex, MutexGuard};
use pdfs_core::control::{Event, JobItem, Response as CtlResponse, Topic, TransferItem};
use tracing::warn;
use super::Core;
/// Most subscriptions served at once. Each holds a control handler for as long
/// as it lasts; the app and the tray need one apiece, and the cap keeps a
/// misbehaving client from spending the handlers that requests need.
const MAX_SUBSCRIBERS: usize = 8;
/// How long a subscription waits after a change before it sends, so a burst
/// (a folder of files trashed at once) goes out as one event.
const COALESCE: Duration = Duration::from_millis(150);
/// How often [`run_event_sampler`] looks at transfers. Status and sync
/// progress are looked at every other time.
const SAMPLE_INTERVAL: Duration = Duration::from_millis(500);
/// Every open subscription's mailbox. Shared by every clone of [`Core`],
/// on-demand forks included, so a change in any mount reaches every client.
#[derive(Default)]
pub(crate) struct EventHub {
mailboxes: Mutex<Vec<Arc<Mailbox>>>,
}
/// What one subscription has yet to send.
#[derive(Default)]
pub(crate) struct Mailbox {
inbox: Mutex<Inbox>,
ready: Condvar,
}
#[derive(Default)]
struct Inbox {
topics: BTreeSet<Topic>,
/// The newest [`Event::Status`] line. A newer one replaces it unsent: a
/// client wants the current status, not every status in between.
status: Option<Arc<str>>,
/// The newest [`Event::Transfers`] line, replaced the same way.
transfers: Option<Arc<str>>,
}
impl Inbox {
fn is_empty(&self) -> bool {
self.topics.is_empty() && self.status.is_none() && self.transfers.is_none()
}
}
/// `event` as one line of the wire format, newline included.
pub(crate) fn event_line(event: &Event) -> Option<Arc<str>> {
match serde_json::to_string(event) {
Ok(mut line) => {
line.push('\n');
Some(line.into())
}
Err(e) => {
warn!(error = %e, "events: serialize event failed");
None
}
}
}
impl EventHub {
/// Tell every subscriber that `topics` changed.
pub(crate) fn publish(&self, topics: &[Topic]) {
self.deliver(|inbox| inbox.topics.extend(topics.iter().copied()));
}
/// Hand every subscriber a fresh status.
fn publish_status(&self, line: Arc<str>) {
self.deliver(|inbox| inbox.status = Some(line.clone()));
}
/// Hand every subscriber a fresh transfers snapshot.
fn publish_transfers(&self, line: Arc<str>) {
self.deliver(|inbox| inbox.transfers = Some(line.clone()));
}
fn deliver(&self, put: impl Fn(&mut Inbox)) {
for mailbox in self.mailboxes.lock().iter() {
put(&mut mailbox.inbox.lock());
mailbox.ready.notify_one();
}
}
/// Open a mailbox, or `None` at [`MAX_SUBSCRIBERS`].
pub(crate) fn subscribe(&self) -> Option<Arc<Mailbox>> {
let mut mailboxes = self.mailboxes.lock();
if mailboxes.len() >= MAX_SUBSCRIBERS {
return None;
}
let mailbox = Arc::new(Mailbox::default());
mailboxes.push(mailbox.clone());
Some(mailbox)
}
pub(crate) fn unsubscribe(&self, mailbox: &Arc<Mailbox>) {
self.mailboxes
.lock()
.retain(|other| !Arc::ptr_eq(other, mailbox));
}
fn has_subscribers(&self) -> bool {
!self.mailboxes.lock().is_empty()
}
}
impl Mailbox {
/// Wait up to `timeout` for something to send, then take it as wire lines:
/// the status first, then transfers, then the changed topics. Empty when
/// nothing arrived in time.
pub(crate) fn take(&self, timeout: Duration) -> Vec<Arc<str>> {
let mut inbox = self.inbox.lock();
if inbox.is_empty() {
self.ready.wait_for(&mut inbox, timeout);
if inbox.is_empty() {
return Vec::new();
}
// Let the rest of a burst arrive before sending the first of it.
MutexGuard::unlocked(&mut inbox, || std::thread::sleep(COALESCE));
}
let mut lines = Vec::new();
lines.extend(inbox.status.take());
lines.extend(inbox.transfers.take());
if !inbox.topics.is_empty() {
let topics = std::mem::take(&mut inbox.topics).into_iter().collect();
lines.extend(event_line(&Event::Changed { topics }));
}
lines
}
}
/// The [`Event::Transfers`] line for a snapshot.
pub(crate) fn transfers_line(items: Vec<TransferItem>, jobs: Vec<JobItem>) -> Option<Arc<str>> {
event_line(&Event::Transfers { items, jobs })
}
/// Watch what changes without a single place to publish it from, and publish it
/// when it differs from the last look: the status (queue counts, cache use,
/// online, pause), the transfers snapshot, and the synced folders with their
/// live progress. Looks only while someone is subscribed.
pub(crate) fn run_event_sampler(core: Core, username: String, mountpoint: std::path::PathBuf) {
let mut transfers: Option<Arc<str>> = None;
let mut status: Option<String> = None;
let mut queue: Option<String> = None;
let mut locations: Option<String> = None;
let mut tick = 0u32;
while core.shutdown.sleep(SAMPLE_INTERVAL) {
if !core.events.has_subscribers() {
// A new subscription starts from a snapshot of its own, so there is
// nothing to compare against until then.
transfers = None;
status = None;
queue = None;
locations = None;
continue;
}
let line = transfers_line(core.transfers.snapshot(), core.jobs_snapshot());
if line.is_some() && line != transfers {
transfers = line.clone();
if let Some(line) = line {
core.events.publish_transfers(line);
}
}
tick = tick.wrapping_add(1);
if !tick.is_multiple_of(2) {
continue;
}
let now = super::control::status_response(&core, &username, &mountpoint);
let (key, queue_key) = status_keys(&now);
if status.as_ref().is_some_and(|last| *last != key)
&& let Some(line) = event_line(&Event::Status(Box::new(now)))
{
core.events.publish_status(line);
}
status = Some(key);
if queue.as_ref().is_some_and(|last| *last != queue_key) {
core.events.publish(&[Topic::Queue]);
}
queue = Some(queue_key);
if let Ok(now) = core.list_locations() {
let key = serde_json::to_string(&now).unwrap_or_default();
if locations.as_ref().is_some_and(|last| *last != key) {
core.events.publish(&[Topic::Locations]);
}
locations = Some(key);
}
}
}
/// What makes two statuses different enough to send, and what makes their
/// queues different. The age of the oldest staged write is left out of both:
/// it grows every second while anything is staged, which is not news.
fn status_keys(status: &CtlResponse) -> (String, String) {
let CtlResponse::Status {
parked_uploads,
failing_ops,
failing_error,
staged_bytes,
paused,
paused_until,
username,
mountpoint,
pinned,
used,
budget,
online,
pending_uploads,
pending_changes,
..
} = status
else {
return (String::new(), String::new());
};
let queue = format!(
"{pending_uploads}/{pending_changes}/{parked_uploads}/{failing_ops}/{failing_error:?}"
);
let key = format!(
"{queue}/{staged_bytes}/{paused}/{paused_until:?}/{username}/{mountpoint}/{pinned}/\
{used}/{budget}/{online}"
);
(key, queue)
}
#[cfg(test)]
mod tests {
use super::*;
fn changed(line: &str) -> Vec<Topic> {
match serde_json::from_str::<Event>(line.trim()) {
Ok(Event::Changed { topics }) => topics,
other => panic!("not a change event: {other:?}"),
}
}
#[test]
fn a_burst_of_changes_arrives_as_one_event() {
let hub = EventHub::default();
let mailbox = hub.subscribe().expect("room for a subscriber");
hub.publish(&[Topic::Trash, Topic::Files]);
hub.publish(&[Topic::Files]);
let lines = mailbox.take(Duration::from_secs(1));
assert_eq!(lines.len(), 1);
assert_eq!(changed(&lines[0]), vec![Topic::Files, Topic::Trash]);
assert!(mailbox.take(Duration::from_millis(10)).is_empty());
}
#[test]
fn a_newer_status_replaces_one_not_yet_sent() {
let hub = EventHub::default();
let mailbox = hub.subscribe().expect("room for a subscriber");
hub.publish_status("first\n".into());
hub.publish_status("second\n".into());
assert_eq!(
mailbox.take(Duration::from_secs(1)),
vec![Arc::<str>::from("second\n")]
);
}
#[test]
fn a_closed_subscription_gets_nothing_more() {
let hub = EventHub::default();
let mailbox = hub.subscribe().expect("room for a subscriber");
hub.unsubscribe(&mailbox);
hub.publish(&[Topic::Files]);
assert!(mailbox.take(Duration::from_millis(10)).is_empty());
assert!(!hub.has_subscribers());
}
#[test]
fn subscriptions_are_capped() {
let hub = EventHub::default();
let open: Vec<_> = (0..MAX_SUBSCRIBERS)
.map(|_| hub.subscribe().expect("room for a subscriber"))
.collect();
assert!(hub.subscribe().is_none());
hub.unsubscribe(&open[0]);
assert!(hub.subscribe().is_some());
}
#[test]
fn the_age_of_a_staged_write_is_not_a_change() {
let status = |staged_oldest_secs| CtlResponse::Status {
parked_uploads: 0,
failing_ops: 0,
failing_error: None,
staged_bytes: 10,
staged_oldest_secs,
paused: false,
paused_until: None,
username: "u".into(),
mountpoint: "/m".into(),
pinned: 0,
used: 0,
budget: 0,
pins: Vec::new(),
online: true,
pending_uploads: 1,
pending_changes: 0,
};
assert_eq!(status_keys(&status(5)), status_keys(&status(6)));
}
}
File diff suppressed because it is too large. Load diff
+4842 -355
View File
File diff suppressed because it is too large. Load diff
+500 -89
View File
@@ -28,10 +28,101 @@ pub(crate) fn is_stale_mount(path: &Path) -> bool {
) )
} }
/// A secondary (on-demand sync folder) FUSE session, paired with its FUSE /// A secondary (on-demand sync folder) FUSE session and the exact liveness flag
/// connection id so teardown can abort a mid-transfer connection rather than /// published with its forked inode state.
/// block on it. See [`Core::mounts`] and [`umount_session_unblocked`]. ///
pub(super) type SecondaryMount = (BackgroundSession, Option<u32>); /// Keeping these together makes every teardown mark the fork unroutable before
/// unmounting, including paths that can block on an in-flight FUSE request.
pub(super) struct SecondaryMount {
session: BackgroundSession,
conn: Option<u32>,
session_live: Arc<AtomicBool>,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(super) enum SecondaryInsertRejection {
Closed,
Duplicate,
}
/// Secondary sessions and the daemon-wide gate controlling whether newly
/// spawned sessions may become visible. The gate and map share one lock so
/// shutdown cannot race an insertion between checking and publishing.
pub(super) struct SecondaryMountRegistry<T> {
accepting: bool,
entries: HashMap<i64, T>,
}
impl<T> Default for SecondaryMountRegistry<T> {
fn default() -> Self {
Self {
accepting: false,
entries: HashMap::new(),
}
}
}
impl<T> SecondaryMountRegistry<T> {
pub(super) fn open(&mut self) {
self.accepting = true;
}
pub(super) fn is_accepting(&self) -> bool {
self.accepting
}
pub(super) fn close(&mut self) {
self.accepting = false;
}
pub(super) fn insert(
&mut self,
id: i64,
mount: T,
) -> Result<(), (SecondaryInsertRejection, T)> {
if !self.accepting {
return Err((SecondaryInsertRejection::Closed, mount));
}
if self.entries.contains_key(&id) {
return Err((SecondaryInsertRejection::Duplicate, mount));
}
self.entries.insert(id, mount);
Ok(())
}
pub(super) fn remove(&mut self, id: &i64) -> Option<T> {
self.entries.remove(id)
}
pub(super) fn contains_key(&self, id: &i64) -> bool {
self.entries.contains_key(id)
}
pub(super) fn close_and_drain(&mut self) -> Vec<(i64, T)> {
self.close();
self.entries.drain().collect()
}
}
impl SecondaryMount {
pub(super) fn new(
session: BackgroundSession,
conn: Option<u32>,
session_live: Arc<AtomicBool>,
) -> Self {
Self {
session,
conn,
session_live,
}
}
pub(super) fn teardown(self) -> std::io::Result<()> {
teardown_session(&self.session_live, || {
umount_session_unblocked(self.session, self.conn)
})
}
}
/// The kernel's id for the FUSE connection backing the mount at `mountpoint` — /// The kernel's id for the FUSE connection backing the mount at `mountpoint` —
/// the directory name under `/sys/fs/fuse/connections`, which is the minor /// the directory name under `/sys/fs/fuse/connections`, which is the minor
@@ -143,6 +234,91 @@ fn fetch_or_recall_root(
} }
} }
/// Per-mount settings resolved from [`pdfs_core::config::AppConfig`] by the
/// caller, so `mount` itself never reads config or the environment.
pub struct MountOptions {
/// Proton account the session belongs to, for display and per-user paths.
pub username: String,
/// What the background conflict sweep may do. See `docs/BUGS.md` B71.
pub sweep_mode: SweepMode,
/// Upload and download caps in bytes per second (`0` = no cap).
pub upload_limit: u64,
pub download_limit: u64,
}
/// Spawn one FUSE session rooted at an arbitrary remote node.
///
/// Primary My Files and secondary on-demand device locations use this same
/// construction path so mount options, stale-mount recovery, registry
/// liveness, and notifier ownership cannot drift apart. Each caller owns its
/// state/path registration and performs it exactly once before calling here.
pub(super) fn spawn_session(
core: &Core,
mountpoint: &Path,
root: Node,
) -> std::io::Result<BackgroundSession> {
let mut config = Config::default();
// `n_threads` stays at fuser's default of 1 (with `clone_fd` off) on
// purpose, for now. A concurrent dispatch loop is worth having, but
// `release` currently relies on being serialised against the `open`+`read`
// an application issues right after `close(2)` — see the comment on
// `Filesystem::release`. Raising it before that ordering is made explicit
// (a per-node staging-in-flight barrier) reintroduces the acceptance
// suite's "concurrent file 1 mismatch". Kernel-side caps that do *not*
// depend on that ordering are negotiated in `ProtonFs::init`.
config.mount_options = vec![
MountOption::FSName("protondrive".to_string()),
MountOption::Subtype("protondrive".to_string()),
MountOption::DefaultPermissions,
];
clear_stale_mount(mountpoint);
info!(mountpoint = %mountpoint.display(), "mounting Proton Drive location");
let fs = ProtonFs::new(core.clone(), root);
if core.primary {
// Only My Files rehydrates the daemon-wide node cache. A secondary
// inode space is rooted at one device subtree and must not import every
// unrelated persisted node.
core.hydrate();
}
let session = Session::new(fs, mountpoint, &config)?.spawn()?;
let _ = core.notifier.set(session.notifier());
core.session_live.store(true, Ordering::Release);
Ok(session)
}
/// Make a session unroutable before beginning any teardown that may block.
fn teardown_session<T>(session_live: &AtomicBool, teardown: impl FnOnce() -> T) -> T {
session_live.store(false, Ordering::Release);
teardown()
}
impl Core {
/// Resolve the primary share id away from startup and publish it only if the
/// projected root has not changed while the request was in flight.
pub(crate) fn repair_primary_share_id(&self, root_uid: &NodeUid) {
match self.rt.block_on(self.client.context_share_id(root_uid)) {
Ok(share_id) => {
match self
.db
.mount_repair_my_files_share_id(&root_uid.to_string(), &share_id.to_string())
{
Ok(true) => debug!(%root_uid, "repaired My Files share id"),
Ok(false) => {
debug!(%root_uid, "discarded stale My Files share-id result");
}
Err(error) => {
warn!(%root_uid, error = %error, "persist My Files share id failed");
}
}
}
Err(error) => {
debug!(%root_uid, error = %error, "resolve My Files share id failed");
}
}
}
}
/// Mount the filesystem at `mountpoint` and block until it is unmounted or the /// Mount the filesystem at `mountpoint` and block until it is unmounted or the
/// daemon is asked to stop. /// daemon is asked to stop.
/// ///
@@ -162,51 +338,96 @@ pub fn mount(
cache: ContentCache, cache: ContentCache,
control_socket: &Path, control_socket: &Path,
db: Arc<Db>, db: Arc<Db>,
username: String, options: MountOptions,
) -> std::io::Result<MountOutcome> { ) -> std::io::Result<MountOutcome> {
let MountOptions {
username,
sweep_mode,
upload_limit,
download_limit,
} = options;
// Start the uptime clock here rather than at the first `pdfs diagnostics`,
// so the age it reports is the daemon's own.
super::diagnostics::start_clock();
let (root, online) = fetch_or_recall_root(&client, &rt, &db)?; let (root, online) = fetch_or_recall_root(&client, &rt, &db)?;
let scope = root.tree_event_scope_id(); let scope = root.tree_event_scope_id();
db.mount_upsert_my_files(&mountpoint.to_string_lossy(), &root.uid.to_string(), None)
.map_err(|error| std::io::Error::other(format!("project My Files mount: {error}")))?;
let share_access = db
.all_share_access()
.map_err(|error| std::io::Error::other(format!("load shared access: {error}")))?;
// The folder-sync engine (devices.md Phase 2) runs on its own thread and is // The folder-sync engine (devices.md Phase 2) runs on its own thread and is
// nudged over this channel; the sender lives in Core so control-socket // nudged over this channel; the sender lives in Core so control-socket
// handlers can trigger reconciles. // handlers can trigger reconciles.
let (sync_tx, sync_rx) = std::sync::mpsc::channel::<sync::SyncMsg>(); let (sync_tx, sync_rx) = std::sync::mpsc::channel::<sync::SyncMsg>();
let paused_until = pause::load_paused_until(&db);
let core = Core { let core = Core {
client: client.clone(), client: client.clone(),
rt: rt.clone(), rt: rt.clone(),
state: Arc::new(Mutex::new(State { maintenance: Arc::new(Mutex::new(Default::default())),
entries: HashMap::new(), primary_root_uid: root.uid.clone(),
by_uid: HashMap::new(), primary: true,
children: HashMap::new(), state: Arc::new(Mutex::new(State::new(db.clone(), share_access, 2))),
next_ino: 2,
active_writes: HashMap::new(),
handles: HashMap::new(),
next_fh: 1,
db: db.clone(),
})),
cache: Arc::new(cache), cache: Arc::new(cache),
readers: Arc::new(Mutex::new(HashMap::new())), readers: Arc::new(Mutex::new(HashMap::new())),
stream_ring: Arc::new(Mutex::new(StreamRing::default())), block_ring: Arc::new(Mutex::new(BlockRing::default())),
block_flight: Arc::new(BlockFlight::default()),
content_repairs: Arc::new(tokio::sync::Mutex::new(HashMap::new())),
prefetch: Arc::new(Prefetch::default()),
prefetch_budget: Arc::new(tokio::sync::Semaphore::new(PREFETCH_BUDGET)),
workers: Arc::new(Workers::new(FUSE_WORKERS)?), workers: Arc::new(Workers::new(FUSE_WORKERS)?),
db, db,
shared_publication: Arc::new(Mutex::new(())),
shared_generation: Arc::new(AtomicU64::new(0)),
shared_refresh_deadlines: Arc::new(Mutex::new(SharedRefreshDeadlines::default())),
online: Arc::new(AtomicBool::new(online)), online: Arc::new(AtomicBool::new(online)),
sync_paused_until: Arc::new(AtomicI64::new(paused_until)),
pending: Arc::new(Mutex::new(HashMap::new())), pending: Arc::new(Mutex::new(HashMap::new())),
hidden: Arc::new(Mutex::new(HashSet::new())), hidden: Arc::new(Mutex::new(HashSet::new())),
drain_wake: Arc::new((Mutex::new(false), Condvar::new())), drain_wake: Arc::new((Mutex::new(false), Condvar::new())),
shutdown: Arc::new(crate::shutdown::Shutdown::default()),
upload_times: Arc::new(Mutex::new(HashMap::new())),
upload_cancel: Arc::new(Mutex::new(HashMap::new())),
timeline_refreshing: Arc::new(AtomicBool::new(false)), timeline_refreshing: Arc::new(AtomicBool::new(false)),
timeline_progress: Arc::new(Mutex::new(None)),
similar_hashing: Arc::new(AtomicBool::new(false)),
albums_refreshing: Arc::new(AtomicBool::new(false)),
trash_refreshing: Arc::new(AtomicBool::new(false)), trash_refreshing: Arc::new(AtomicBool::new(false)),
trash_progress: Arc::new(tokio::sync::Notify::new()),
conflict_notified: Arc::new(Mutex::new(HashSet::new())),
sweep_mode,
own_sealed_revs: Arc::new(Mutex::new(HashMap::new())),
self_changes: Arc::new(Mutex::new(HashMap::new())),
thumb_gen: Arc::new(Mutex::new(HashSet::new())), thumb_gen: Arc::new(Mutex::new(HashSet::new())),
no_thumbnail: Arc::new(Mutex::new(HashMap::new())), thumb_gen_budget: Arc::new(tokio::sync::Semaphore::new(
super::photos::THUMB_GEN_CONCURRENCY,
)),
file_thumb_generation: Arc::new(AtomicU64::new(0)),
file_thumb_cancel: Arc::new(tokio::sync::Notify::new()),
thumbnail_build: Arc::new(Mutex::new(Default::default())),
thumbnail_build_cancelled: Arc::new(AtomicBool::new(false)),
thumbnail_build_cancel: Arc::new(tokio::sync::Notify::new()),
thumbnail_misses: Arc::new(Mutex::new(Default::default())),
quota: Arc::new(Mutex::new(None)),
size_upgrades: Arc::new(Mutex::new(HashMap::new())), size_upgrades: Arc::new(Mutex::new(HashMap::new())),
size_waiters: Arc::new(Default::default()),
notifier: Arc::new(OnceLock::new()), notifier: Arc::new(OnceLock::new()),
session_live: Arc::new(AtomicBool::new(false)),
transfers: TransferRegistry::new(), transfers: TransferRegistry::new(),
events: Arc::new(Default::default()),
indexing: Arc::new(AtomicBool::new(false)), indexing: Arc::new(AtomicBool::new(false)),
sync_progress: Arc::new(Mutex::new(HashMap::new())), sync_progress: Arc::new(Mutex::new(HashMap::new())),
sync_tx, sync_tx,
mounts: Arc::new(Mutex::new(HashMap::new())), mounts: Arc::new(Mutex::new(SecondaryMountRegistry::default())),
sync_locks: Arc::new(Mutex::new(HashMap::new())), sync_locks: Arc::new(Mutex::new(HashMap::new())),
states: Arc::new(StateRegistry::default()),
}; };
// Before anything can queue work against it: the drain thread below reaches
// every mount's inode space through this registry, not through `core.state`.
core.register_state(mountpoint);
core.transfers.set_limits(upload_limit, download_limit);
// Writes queued by a previous run (or left behind by a crash) are still owed // Writes queued by a previous run (or left behind by a crash) are still owed
// an upload, and reads must be served from their staged blobs until they land. // an upload, and reads must be served from their staged blobs until they land.
@@ -215,34 +436,63 @@ pub fn mount(
// aside at open. After `hydrate_pending`, so a recovered partial write can // aside at open. After `hydrate_pending`, so a recovered partial write can
// see an already-queued write to the same node. // see an already-queued write to the same node.
core.recover_fsynced_writes(); core.recover_fsynced_writes();
{ // And finally the staged blobs neither of those accounts for: bytes a
// failed release left behind with nothing pointing at them.
core.reconcile_staging();
// Several drain workers over one queue. One thread meant a single large
// upload held every queued rename, trash and small write behind it for as
// long as it ran; the claim column (`Db::claim_next_due_op`) keeps the
// workers off each other's nodes, which is the only ordering the queue
// actually needs. Worker 0 additionally runs the queue's idle chores.
//
// Every long-lived thread started from here on is kept in `workers` and
// joined at the end of this function. Leaving them running turned an
// in-process remount into a second generation of drain workers, sync engines
// and sweeps all operating on a mount that no longer exists (bugs.md B44).
let mut workers: Vec<std::thread::JoinHandle<()>> = Vec::new();
for worker in 0..DRAIN_WORKERS {
let core = core.clone(); let core = core.clone();
std::thread::Builder::new() workers.push(
.name("pdfs-drain".into()) std::thread::Builder::new()
.spawn(move || core.run_pending_drain())?; .name(format!("pdfs-drain-{worker}"))
.spawn(move || core.run_pending_drain(worker == 0))?,
);
} }
// Start the folder-sync engine. It watches every mirror folder, polls the // Start the folder-sync engine. It watches every mirror folder, polls the
// remotes, and reconciles on its own thread — never in front of a FUSE call. // remotes, and reconciles on its own thread — never in front of a FUSE call.
sync::spawn(core.clone(), sync_rx); workers.extend(sync::spawn(core.clone(), sync_rx));
// A timed pause persisted by an earlier run still has to end on time.
if pause::paused_at(paused_until, now_secs()) && paused_until != pause::PAUSED_INDEFINITELY {
core.arm_pause_timer(paused_until);
}
// Reconcile leftover `(sync-conflict …)` copies: drop the ones that turned
// out identical to the live file, surface the ones that genuinely diverge.
// Its own thread — it enumerates and trashes, and must never block a FUSE call.
// `Off` skips the thread entirely rather than starting an idle one, so the
// setting is verifiable from the outside (`ls /proc/<pid>/task/*/comm`).
if sweep_mode == SweepMode::Off {
info!("conflict sweep disabled by configuration");
} else {
info!(mode = ?sweep_mode, "conflict sweep enabled");
let core = core.clone();
workers.push(
std::thread::Builder::new()
.name("pdfs-conflict-sweep".into())
.spawn(move || core.run_conflict_sweep_loop())?,
);
}
// Mounted from the cache: watch for the network coming back so the mount can // Mounted from the cache: watch for the network coming back so the mount can
// stop being read-only-ish without the user restarting the daemon. // stop being read-only-ish without the user restarting the daemon.
if !online { if !online {
let core = core.clone(); let core = core.clone();
std::thread::Builder::new() workers.push(
.name("pdfs-online-probe".into()) std::thread::Builder::new()
.spawn(move || core.run_online_probe())?; .name("pdfs-online-probe".into())
} .spawn(move || core.run_online_probe())?,
);
// Re-establish on-demand mounts left over from a previous run (devices.md
// Phase 4). On its own thread: each remount fetches a remote node, and we
// must not block the main mount below on the network.
{
let core = core.clone();
std::thread::Builder::new()
.name("pdfs-restore-ondemand".into())
.spawn(move || core.restore_ondemand_mounts())?;
} }
// Keep the launcher prompt's "This computer" index fresh. Its own thread: // Keep the launcher prompt's "This computer" index fresh. Its own thread:
@@ -252,23 +502,14 @@ pub fn mount(
let indexing = core.indexing.clone(); let indexing = core.indexing.clone();
let transfers = core.transfers.clone(); let transfers = core.transfers.clone();
let mountpoint = mountpoint.to_path_buf(); let mountpoint = mountpoint.to_path_buf();
std::thread::Builder::new() let shutdown = core.shutdown.clone();
.name("pdfs-localindex".into()) workers.push(
.spawn(move || run_local_index(db, indexing, transfers, mountpoint))?; std::thread::Builder::new()
.name("pdfs-localindex".into())
.spawn(move || run_local_index(db, indexing, transfers, mountpoint, shutdown))?,
);
} }
let mut config = Config::default();
config.mount_options = vec![
MountOption::FSName("protondrive".to_string()),
MountOption::Subtype("protondrive".to_string()),
MountOption::DefaultPermissions,
];
// A crashed previous run can leave the kernel mount in place, which makes
// the fresh mount below fail with EBUSY. Lazily detach any leftover first.
clear_stale_mount(mountpoint);
info!(mountpoint = %mountpoint.display(), "mounting Proton Drive");
// Bind the control socket before the FUSE session takes over the thread. A // Bind the control socket before the FUSE session takes over the thread. A
// stale socket file from a previous run would block the bind, so clear it. // stale socket file from a previous run would block the bind, so clear it.
let _ = std::fs::remove_file(control_socket); let _ = std::fs::remove_file(control_socket);
@@ -285,40 +526,115 @@ pub fn mount(
"control socket permissions: {e}" "control socket permissions: {e}"
))); )));
} }
{ // Do not accept control requests until `spawn_session` has transitioned the
let core = core.clone(); // primary registration to live. The listener may queue a connection during
let username = username.clone(); // this short window, but its handler cannot observe `mounted = false`.
let mountpoint = mountpoint.to_path_buf(); let bg = match spawn_session(&core, mountpoint, root) {
std::thread::Builder::new() Ok(session) => session,
.name("pdfs-control".into()) Err(error) => {
.spawn(move || run_control_socket(core, username, mountpoint, listener))?; let _ = std::fs::remove_file(control_socket);
} return Err(error);
}
let fs = ProtonFs::new(core.clone(), root); };
// Warm the in-memory maps from the DB so a cold start serves previously core.mounts.lock().open();
// seen metadata without re-hitting the API (plan.md P1).
core.hydrate();
// Build the session explicitly (not `mount2`) so we can grab a `Notifier`
// for the event task. `spawn` runs the session loop on its own background
// thread; we then wait here for either a stop signal or the mount ending.
let bg = Session::new(fs, mountpoint, &config)?.spawn()?;
// The connection id, captured now while the mount is live, so a stop signal // The connection id, captured now while the mount is live, so a stop signal
// mid-transfer can abort it rather than block `join` (see `abort_fuse_connection`). // mid-transfer can abort it rather than block `join` (see `abort_fuse_connection`).
let main_conn = fuse_connection_id(mountpoint); let main_conn = fuse_connection_id(mountpoint);
let notifier = bg.notifier(); {
// Same channel, kept on the `Core` so background work (a size upgrade, say) let control_core = core.clone();
// can invalidate kernel-cached metadata without threading a handle through. let username = username.clone();
let _ = core.notifier.set(notifier.clone()); let mountpoint = mountpoint.to_path_buf();
rt.spawn(run_event_sync( match std::thread::Builder::new()
client, .name("pdfs-control".into())
scope, .spawn(move || run_control_socket(control_core, username, mountpoint, listener))
core.state, {
core.cache, Ok(handle) => workers.push(handle),
core.db, Err(error) => {
core.pending, // A failure here abandons the mount, so the background threads
notifier, // started above have to be told to stop with it.
)); stop_workers(&core, control_socket, workers);
let secondaries = core.mounts.lock().close_and_drain();
for (id, mount) in secondaries {
if let Err(unmount_error) = mount.teardown() {
warn!(id, error = %unmount_error, "secondary teardown after control startup failure failed");
}
}
if let Err(unmount_error) = teardown_session(&core.session_live, || {
umount_session_unblocked(bg, main_conn)
}) {
warn!(error = %unmount_error, "unmount after control startup failure failed");
}
let _ = std::fs::remove_file(control_socket);
return Err(error);
}
}
}
// Publishes what changes without one place to publish it from (status,
// transfers, sync progress) to the front-ends subscribed on that socket.
{
let sampler_core = core.clone();
let username = username.clone();
let mountpoint = mountpoint.to_path_buf();
match std::thread::Builder::new()
.name("pdfs-events".into())
.spawn(move || events::run_event_sampler(sampler_core, username, mountpoint))
{
Ok(handle) => workers.push(handle),
// Not fatal: front-ends still get every published change, only
// not the sampled ones.
Err(error) => warn!(error = %error, "could not start the event sampler"),
}
}
// The supervisor watches the pool and answers the systemd watchdog. Started
// after the listener, because its liveness probe is a round trip over that
// socket.
{
let supervisor_core = core.clone();
let socket = control_socket.to_path_buf();
match std::thread::Builder::new()
.name("pdfs-supervisor".into())
.spawn(move || supervisor::run(supervisor_core, socket))
{
Ok(handle) => workers.push(handle),
// Not fatal: the mount works without a supervisor, it just goes back
// to being undiagnosable if it wedges.
Err(error) => warn!(error = %error, "could not start the daemon supervisor"),
}
}
// The mount is live and the control socket answers: this is the moment the
// unit is actually usable, which is what `Type=notify` reports.
systemd::ready();
// Re-establish on-demand mounts only after the primary session is live.
// Keep the worker so shutdown can close publication, wait for any in-flight
// fetch to observe that closure, and then drain a stable registry.
let restore_worker = {
let restore_core = core.clone();
match std::thread::Builder::new()
.name("pdfs-restore-ondemand".into())
.spawn(move || restore_core.restore_ondemand_mounts())
{
Ok(worker) => Some(worker),
Err(error) => {
warn!(%error, "start on-demand restore worker failed");
None
}
}
};
rt.spawn(run_event_sync(client, scope, core.clone()));
// The photos volume has its own event stream; without it the gallery only
// learns about a phone-side deletion when the timeline goes stale.
rt.spawn(run_photos_event_sync(core.clone()));
if online {
let repair_core = core.clone();
let root_uid = core.primary_root_uid.clone();
if let Err(error) = std::thread::Builder::new()
.name("pdfs-share-id".into())
.spawn(move || repair_core.repair_primary_share_id(&root_uid))
{
warn!(%error, "start My Files share-id repair failed");
}
}
// Stop signals (SIGTERM from `systemctl --user stop`, SIGINT from Ctrl-C) // Stop signals (SIGTERM from `systemctl --user stop`, SIGINT from Ctrl-C)
// are delivered onto the async runtime; bridge them onto a sync channel so // are delivered onto the async runtime; bridge them onto a sync channel so
@@ -357,7 +673,10 @@ pub fn mount(
match sig_rx.recv_timeout(Duration::from_millis(500)) { match sig_rx.recv_timeout(Duration::from_millis(500)) {
Ok(()) => { Ok(()) => {
info!("stop requested; unmounting"); info!("stop requested; unmounting");
if let Err(e) = umount_session_unblocked(bg, main_conn) { core.mounts.lock().close();
if let Err(e) = teardown_session(&core.session_live, || {
umount_session_unblocked(bg, main_conn)
}) {
warn!(error = %e, "umount_and_join failed"); warn!(error = %e, "umount_and_join failed");
} }
break MountOutcome::Shutdown; break MountOutcome::Shutdown;
@@ -365,7 +684,8 @@ pub fn mount(
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => { Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {
if bg.guard.is_finished() { if bg.guard.is_finished() {
info!("mount ended externally"); info!("mount ended externally");
if let Err(e) = bg.join() { core.mounts.lock().close();
if let Err(e) = teardown_session(&core.session_live, || bg.join()) {
warn!(error = %e, "session join failed"); warn!(error = %e, "session join failed");
} }
break MountOutcome::Unmounted; break MountOutcome::Unmounted;
@@ -373,7 +693,8 @@ pub fn mount(
} }
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => { Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {
// Signal task gone (failed to install); fall back to join. // Signal task gone (failed to install); fall back to join.
let _ = bg.join(); core.mounts.lock().close();
let _ = teardown_session(&core.session_live, || bg.join());
break MountOutcome::Unmounted; break MountOutcome::Unmounted;
} }
} }
@@ -381,13 +702,103 @@ pub fn mount(
// Unmount every on-demand sync folder too, or the kernel mounts linger as // Unmount every on-demand sync folder too, or the kernel mounts linger as
// stale and the next start fails with EBUSY (devices.md Phase 3). // stale and the next start fails with EBUSY (devices.md Phase 3).
let secondaries: Vec<_> = core.mounts.lock().drain().collect(); if let Some(worker) = restore_worker
for (id, (session, conn)) in secondaries { && worker.join().is_err()
if let Err(e) = umount_session_unblocked(session, conn) { {
warn!("on-demand restore worker panicked");
}
let secondaries = core.mounts.lock().close_and_drain();
for (id, mount) in secondaries {
if let Err(e) = mount.teardown() {
warn!(id, error = %e, "unmount on-demand folder failed"); warn!(id, error = %e, "unmount on-demand folder failed");
} }
} }
let _ = std::fs::remove_file(control_socket); // Last, because the workers may legitimately be finishing an upload, and
// nothing they can still do is harmful once the mounts are down.
stop_workers(&core, control_socket, workers);
Ok(outcome) Ok(outcome)
} }
/// How long teardown waits for the FUSE worker pool before proceeding without
/// it. Long enough for an in-flight block to finish, short enough that
/// `systemctl --user stop` still returns inside `TimeoutStopSec`.
const WORKER_JOIN_DEADLINE: Duration = Duration::from_secs(10);
/// Signal every background loop, wake the ones that are blocked, and join them.
///
/// The mount used to return with its drain workers, sync engine, sweep, online
/// probe, indexer and control listener all still running on `Core` clones. The
/// process usually exited straight afterwards and hid it; an in-process remount
/// did not, and started a second full set on top of the first (bugs.md B44).
///
/// Two of the waits need waking rather than merely flagging: the drain workers
/// block on their own condvar (`wake_drain`), and the control listener blocks
/// inside `accept`, which only returns when a connection arrives — so it gets
/// one, from us, after the flag is already set. A join that takes a while is
/// expected and correct: a worker part-way through an upload finishes it rather
/// than abandoning the user's bytes mid-flight.
fn stop_workers(core: &Core, control_socket: &Path, workers: Vec<std::thread::JoinHandle<()>>) {
// Say so before the joins: a teardown that takes a while must not be read as
// a missed watchdog ping and turned into a kill.
systemd::stopping();
core.shutdown.stop();
core.wake_drain();
let _ = core.sync_tx.send(sync::SyncMsg::Stop);
// Connect while the socket still exists — that is the whole point of the
// poke — then unlink it so nothing else can arrive behind us. The listener
// re-checks the flag after every accept, so even a connection that races the
// unlink cannot put it back to sleep.
let _ = std::os::unix::net::UnixStream::connect(control_socket);
let _ = std::fs::remove_file(control_socket);
for worker in workers {
if worker.join().is_err() {
warn!("a background worker panicked before shutdown");
}
}
// The FUSE worker pool last, and *before* the caller drops the tokio
// runtime: a worker still inside a job when the runtime goes away panics on
// its next timer with "A Tokio 1.x context was found, but it is being
// shutdown" (seen 2026-09-18 09:58:35). Bounded, so a worker wedged on the
// network cannot hold the stop the user asked for.
core.workers.stop_and_join(WORKER_JOIN_DEADLINE);
debug!("background workers stopped");
}
#[cfg(test)]
mod tests {
use std::sync::atomic::{AtomicBool, Ordering};
use super::{SecondaryInsertRejection, SecondaryMountRegistry, teardown_session};
#[test]
fn session_is_unroutable_before_teardown_starts() {
let live = AtomicBool::new(true);
teardown_session(&live, || {
assert!(
!live.load(Ordering::Acquire),
"teardown must never run while the registry still reports mounted"
);
});
}
#[test]
fn secondary_registry_rejects_closed_and_duplicate_insertions() {
let mut mounts = SecondaryMountRegistry::default();
assert_eq!(
mounts.insert(7, "closed"),
Err((SecondaryInsertRejection::Closed, "closed"))
);
mounts.open();
assert_eq!(mounts.insert(7, "first"), Ok(()));
assert_eq!(
mounts.insert(7, "duplicate"),
Err((SecondaryInsertRejection::Duplicate, "duplicate"))
);
assert!(mounts.contains_key(&7));
assert_eq!(mounts.close_and_drain(), vec![(7, "first")]);
assert!(!mounts.is_accepting());
}
}
+235
View File
@@ -0,0 +1,235 @@
//! Pausing sync: the user's "stop uploading for a while".
//!
//! A pause holds back the two things that push local state to the remote — the
//! drain worker's queued uploads and changes, and the mirror-folder reconcile.
//! Nothing else stops: reads through the mount keep hydrating on demand (a read
//! is not a sync), writes keep being accepted and staged, and the watcher and
//! poll keep noticing changes. Resuming lets all of it catch up at once.
//!
//! The pause is persisted in the database, so a restart does not quietly undo
//! it. A timed pause ends by itself; [`Core::set_sync_paused`] arms a timer for
//! that, and [`Core::sync_paused`] compares against the clock anyway, so a
//! deadline that passed while the daemon was down is already over on mount.
//!
//! A single mirror folder can be paused on its own as well: its reconcile is
//! skipped, and with it any mode switch queued behind that reconcile, until the
//! folder is resumed. On-demand folders cannot be paused alone — their writes
//! go through the shared queue, which only the global pause holds back.
use super::*;
/// The `state` key holding the resume time, in Unix seconds.
const PAUSED_UNTIL_KEY: &str = "sync_paused_until";
/// The `state` key prefix marking one synced folder paused; its id follows.
const FOLDER_PAUSED_PREFIX: &str = "sync_folder_paused:";
fn folder_paused_key(id: i64) -> String {
format!("{FOLDER_PAUSED_PREFIX}{id}")
}
/// Whether synced folder `id` is paused on its own. A row that cannot be read
/// counts as running, like the global pause.
fn load_folder_paused(db: &Db, id: i64) -> bool {
match db.state_i64(&folder_paused_key(id)) {
Ok(value) => value.is_some_and(|v| v != 0),
Err(error) => {
warn!(id, %error, "reading a folder pause failed; treating it as running");
false
}
}
}
/// Persist synced folder `id`'s own pause, or clear it.
fn store_folder_paused(db: &Db, id: i64, paused: bool) -> pdfs_core::Result<()> {
match paused {
true => db.set_state_i64(&folder_paused_key(id), 1),
false => db.clear_state(&folder_paused_key(id)),
}
}
/// A pause with no end time: it lasts until the user resumes.
pub(crate) const PAUSED_INDEFINITELY: i64 = i64::MAX;
/// The persisted pause, or `0` when there is none (or it cannot be read — a
/// broken state row must not keep sync paused forever).
pub(crate) fn load_paused_until(db: &Db) -> i64 {
match db.state_i64(PAUSED_UNTIL_KEY) {
Ok(Some(until)) => until,
Ok(None) => 0,
Err(error) => {
warn!(%error, "reading the sync pause failed; treating sync as running");
0
}
}
}
/// Whether a pause lasting until `until` is still in force at `now` (both Unix
/// seconds). `0` is "not paused".
pub(crate) fn paused_at(until: i64, now: i64) -> bool {
until != 0 && now < until
}
impl Core {
/// True while the user has sync paused.
pub(crate) fn sync_paused(&self) -> bool {
paused_at(self.sync_paused_until.load(Ordering::Relaxed), now_secs())
}
/// The pause's end, for status reporting: `None` when not paused,
/// `Some(None)` when paused until resumed, `Some(Some(t))` when it ends at
/// Unix second `t`.
pub(crate) fn sync_pause(&self) -> Option<Option<i64>> {
let until = self.sync_paused_until.load(Ordering::Relaxed);
match paused_at(until, now_secs()) {
false => None,
true if until == PAUSED_INDEFINITELY => Some(None),
true => Some(Some(until)),
}
}
/// Pause sync until Unix second `until` (`None`: until resumed), or resume
/// it when `paused` is false.
pub(crate) fn set_sync_paused(
&self,
paused: bool,
until: Option<i64>,
) -> Result<(), CoreError> {
let value = match (paused, until) {
(false, _) => 0,
(true, None) => PAUSED_INDEFINITELY,
(true, Some(until)) if until <= now_secs() => {
return Err(CoreError::invalid("the pause would already be over"));
}
(true, Some(until)) => until,
};
let persisted = match value {
0 => self.db.clear_state(PAUSED_UNTIL_KEY),
value => self.db.set_state_i64(PAUSED_UNTIL_KEY, value),
};
if let Err(error) = persisted {
return Err(CoreError::internal(format!(
"saving the sync pause: {error}"
)));
}
self.sync_paused_until.store(value, Ordering::Relaxed);
match value {
0 => {
info!("sync resumed");
self.resume_sync_work();
}
PAUSED_INDEFINITELY => info!("sync paused until resumed"),
until => {
info!(until, "sync paused");
self.arm_pause_timer(until);
}
}
Ok(())
}
/// True while the user has this one synced folder paused. A row that cannot
/// be read counts as running, like the global pause.
pub(crate) fn sync_folder_paused(&self, id: i64) -> bool {
load_folder_paused(&self.db, id)
}
/// Pause or resume one mirror folder's reconcile. Resuming reconciles it
/// straight away to catch up on what the watcher saw meanwhile.
pub(crate) fn set_sync_folder_paused(&self, id: i64, paused: bool) -> Result<(), CoreError> {
let folder = self
.db
.sync_folder_get(id)
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?
.ok_or_else(|| CoreError::not_found(format!("no synced folder with id {id}")))?;
if paused && folder.mode != "mirror" {
return Err(CoreError::invalid(
"only a mirrored folder can be paused on its own; an on-demand folder uploads \
through the shared queue, so pause all sync instead",
));
}
if let Err(error) = store_folder_paused(&self.db, id, paused) {
return Err(CoreError::internal(format!(
"saving the folder pause: {error}"
)));
}
if paused {
info!(id, path = %folder.local_path, "folder sync paused");
} else {
info!(id, path = %folder.local_path, "folder sync resumed");
self.sync_now(Some(id));
}
Ok(())
}
/// Forget a removed folder's pause, so a later folder reusing its id does
/// not start out paused.
pub(crate) fn clear_sync_folder_paused(&self, id: i64) {
if let Err(error) = store_folder_paused(&self.db, id, false) {
warn!(id, %error, "clearing a removed folder's pause failed");
}
}
/// Wake everything a pause held back: the drain worker, and a full
/// reconcile to pick up whatever the watcher saw meanwhile.
fn resume_sync_work(&self) {
self.wake_drain();
self.sync_now(None);
}
/// End a timed pause when its time comes. A later pause or resume replaces
/// the stored deadline, which the timer checks before acting, so a stale
/// timer does nothing.
pub(crate) fn arm_pause_timer(&self, until: i64) {
let core = self.clone();
let wait = Duration::from_secs(until.saturating_sub(now_secs()).max(0) as u64);
let spawned = std::thread::Builder::new()
.name("pdfs-pause".into())
.spawn(move || {
if !core.shutdown.sleep(wait) {
return;
}
if core
.sync_paused_until
.compare_exchange(until, 0, Ordering::Relaxed, Ordering::Relaxed)
.is_ok()
{
if let Err(error) = core.db.clear_state(PAUSED_UNTIL_KEY) {
warn!(%error, "clearing an expired sync pause failed");
}
info!("sync pause ended");
core.resume_sync_work();
}
});
if let Err(error) = spawned {
// The drain and the reconcile still read the clock, so the pause
// still ends — just at the next idle poll instead of on time.
warn!(%error, "starting the sync pause timer failed");
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_pause_is_in_force_until_its_deadline() {
assert!(!paused_at(0, 100));
assert!(paused_at(200, 100));
assert!(!paused_at(200, 200));
assert!(!paused_at(200, 300));
assert!(paused_at(PAUSED_INDEFINITELY, i64::MAX - 1));
}
#[test]
fn a_folder_pause_is_kept_per_folder_until_cleared() {
let db = Db::open(std::path::Path::new(":memory:")).unwrap();
assert!(!load_folder_paused(&db, 3));
store_folder_paused(&db, 3, true).unwrap();
assert!(load_folder_paused(&db, 3));
assert!(!load_folder_paused(&db, 4));
assert!(!load_folder_paused(&db, 33));
store_folder_paused(&db, 3, false).unwrap();
assert!(!load_folder_paused(&db, 3));
}
}
File diff suppressed because it is too large. Load diff
+191 -30
View File
@@ -12,7 +12,10 @@ use std::sync::OnceLock;
use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::atomic::{AtomicBool, Ordering};
use pdfs_core::control::RestorableFolder; use pdfs_core::control::RestorableFolder;
use pdfs_core::profile::{PROFILE_FILE_NAME, PROFILE_VERSION, Profile, ProfileFolder, ProfilePin}; use pdfs_core::mounts::MountMode;
use pdfs_core::profile::{
PROFILE_DIR_NAME, PROFILE_FILE_NAME, PROFILE_VERSION, Profile, ProfileFolder, ProfilePin,
};
use pdfs_core::{CoreError, CoreResult}; use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::proton_sdk::ids::NodeUid; use proton_drive_rs::proton_sdk::ids::NodeUid;
use tracing::{info, warn}; use tracing::{info, warn};
@@ -63,8 +66,9 @@ impl Core {
}); });
} }
/// Serialize this machine's arrangement and upload it to `profile.json` in /// Serialize this machine's arrangement and upload it to `profile.json`
/// the device root, replacing any existing one. /// inside the device's `.proton-drive-linux` folder, replacing any existing
/// one.
/// ///
/// Does nothing when no device is registered: a machine that has never /// Does nothing when no device is registered: a machine that has never
/// synced a folder has nothing to describe, and registering a device just to /// synced a folder has nothing to describe, and registering a device just to
@@ -85,11 +89,15 @@ impl Core {
let bytes = profile.to_bytes().map_err(CoreError::internal)?; let bytes = profile.to_bytes().map_err(CoreError::internal)?;
let len = bytes.len() as i64; let len = bytes.len() as i64;
// The device root itself rejects file creation (B78), so the profile
// lives one level down, in a folder created on first save.
let dir_uid = self.ensure_profile_dir(&root_uid)?;
// Replace the existing document rather than adding a second file of the // Replace the existing document rather than adding a second file of the
// same name: the profile is a single mutable record, and a device root // same name: the profile is a single mutable record, and a folder with
// with `profile.json` and `profile (1).json` in it teaches the restore // `profile.json` and `profile (1).json` in it teaches the restore
// nothing about which to trust. // nothing about which to trust.
let existing = self.find_device_child_file(&root_uid, PROFILE_FILE_NAME)?; let existing = self.find_device_child_file(&dir_uid, PROFILE_FILE_NAME)?;
match existing { match existing {
Some(uid) => self Some(uid) => self
.rt .rt
@@ -105,7 +113,7 @@ impl Core {
None => { None => {
self.rt self.rt
.block_on(self.client.upload_file_replacing_draft_from( .block_on(self.client.upload_file_replacing_draft_from(
&root_uid, &dir_uid,
PROFILE_FILE_NAME, PROFILE_FILE_NAME,
"application/json", "application/json",
Cursor::new(bytes), Cursor::new(bytes),
@@ -133,7 +141,7 @@ impl Core {
local_path: f.local_path, local_path: f.local_path,
// A queued switch is where the user asked the folder to be, so // A queued switch is where the user asked the folder to be, so
// that is what the next machine should set up. // that is what the next machine should set up.
mode: f.pending_mode.unwrap_or(f.mode), mode: MountMode::from(f.pending_mode.unwrap_or(f.mode).as_str()),
}) })
.collect(); .collect();
let pins = self let pins = self
@@ -141,10 +149,10 @@ impl Core {
.pin_list() .pin_list()
.map_err(|e| CoreError::internal(format!("db: {e:?}")))? .map_err(|e| CoreError::internal(format!("db: {e:?}")))?
.into_iter() .into_iter()
.map(|(uid, path, recursive)| ProfilePin { .map(|row| ProfilePin {
uid, uid: row.uid,
path, path: row.path,
recursive, recursive: row.recursive,
}) })
.collect(); .collect();
let cfg = pdfs_core::config::AppDirs::new() let cfg = pdfs_core::config::AppDirs::new()
@@ -163,18 +171,18 @@ impl Core {
}) })
} }
/// An untrashed *file* named `name` directly under the device root. /// An untrashed *file* named `name` directly under `parent_uid`.
/// ///
/// The sibling of `find_device_child_folder`; kept separate because a folder /// The sibling of `find_device_child_folder`; kept separate because a folder
/// named `profile.json` must not be mistaken for the profile. /// named `profile.json` must not be mistaken for the profile.
fn find_device_child_file( fn find_device_child_file(
&self, &self,
root_uid: &NodeUid, parent_uid: &NodeUid,
name: &str, name: &str,
) -> CoreResult<Option<NodeUid>> { ) -> CoreResult<Option<NodeUid>> {
let uids = self let uids = self
.rt .rt
.block_on(self.client.enumerate_folder_children_node_uids(root_uid)) .block_on(self.client.enumerate_folder_children_node_uids(parent_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?; .map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() { if uids.is_empty() {
return Ok(None); return Ok(None);
@@ -189,10 +197,47 @@ impl Core {
.map(|n| n.uid)) .map(|n| n.uid))
} }
/// Download and parse the profile in the device root, if there is one. /// The device's `.proton-drive-linux` folder, creating it if this is the
/// first save on this device.
///
/// Reused by stable identity rather than recreated: a second folder of the
/// same name would split the profile across two documents, and the restore
/// would have no way to tell which is current.
fn ensure_profile_dir(&self, root_uid: &NodeUid) -> CoreResult<NodeUid> {
if let Some(uid) = self.find_device_child_folder(root_uid, PROFILE_DIR_NAME)? {
return Ok(uid);
}
let uid = self
.rt
.block_on(
self.client
.create_folder(root_uid, PROFILE_DIR_NAME, Some(now_secs())),
)
.map_err(|e| CoreError::from_api(&e, "create profile folder"))?;
info!(name = PROFILE_DIR_NAME, "created device profile folder");
Ok(uid)
}
/// Download and parse this device's profile, if there is one.
///
/// Looks in `.proton-drive-linux/` first and falls back to the device root,
/// where clients before the B78 fix wrote it. The fallback is read-only: the
/// next save writes to the folder, and the legacy document is left alone
/// rather than trashed, so an older client on another machine keeps working.
fn load_profile(&self, root_uid: &NodeUid) -> CoreResult<Option<Profile>> { fn load_profile(&self, root_uid: &NodeUid) -> CoreResult<Option<Profile>> {
let Some(uid) = self.find_device_child_file(root_uid, PROFILE_FILE_NAME)? else { let current = match self.find_device_child_folder(root_uid, PROFILE_DIR_NAME)? {
return Ok(None); Some(dir_uid) => self.find_device_child_file(&dir_uid, PROFILE_FILE_NAME)?,
None => None,
};
let uid = match current {
Some(uid) => uid,
None => match self.find_device_child_file(root_uid, PROFILE_FILE_NAME)? {
Some(uid) => {
info!("reading legacy profile from the device root");
uid
}
None => return Ok(None),
},
}; };
let mut buf: Vec<u8> = Vec::new(); let mut buf: Vec<u8> = Vec::new();
self.rt self.rt
@@ -203,15 +248,19 @@ impl Core {
// ---- restore (features.md 5.2) ---------------------------------------- // ---- restore (features.md 5.2) ----------------------------------------
/// The folders under this machine's device that could be synced here, each /// The folders under a device that could be synced here, each carrying a
/// carrying a proposed local path. /// proposed local path. `device` names another registered device by uid;
/// `None` is this machine's own.
/// ///
/// The proposal comes from the profile when one exists and its path is /// The proposal comes from the profile when one exists and its path is
/// usable on this machine, and from `~/<name>` otherwise — a restored /// usable on this machine, and from `~/<name>` otherwise — a restored
/// `/home/other-user/Documents` would be wrong on a machine where that user /// `/home/other-user/Documents` would be wrong on a machine where that user
/// does not exist, and silently creating it would be worse. /// does not exist, and silently creating it would be worse.
pub(crate) fn list_restorable_folders(&self) -> CoreResult<Vec<RestorableFolder>> { pub(crate) fn list_restorable_folders(
let device = self.ensure_device()?; &self,
device: Option<&str>,
) -> CoreResult<Vec<RestorableFolder>> {
let device = self.restore_source(device)?;
let root_uid = parse_uid(&device.root_uid).ok_or_else(|| { let root_uid = parse_uid(&device.root_uid).ok_or_else(|| {
CoreError::internal(format!("bad device root uid: {}", device.root_uid)) CoreError::internal(format!("bad device root uid: {}", device.root_uid))
})?; })?;
@@ -251,7 +300,10 @@ impl Core {
Ok(nodes Ok(nodes
.into_iter() .into_iter()
.filter(|n| n.is_folder() && !n.trashed) // `.proton-drive-linux` holds the profile itself, not user data:
// offering it as a restorable folder would sync the bookkeeping
// back onto the machine it describes.
.filter(|n| n.is_folder() && !n.trashed && n.name != PROFILE_DIR_NAME)
.map(|n| { .map(|n| {
let uid = n.uid.to_string(); let uid = n.uid.to_string();
let saved = profile let saved = profile
@@ -267,7 +319,7 @@ impl Core {
name: n.name, name: n.name,
local_path: proposed.to_string_lossy().to_string(), local_path: proposed.to_string_lossy().to_string(),
mode: saved mode: saved
.map(|f| f.mode.clone()) .map(|f| f.mode.to_string())
.unwrap_or_else(|| "mirror".into()), .unwrap_or_else(|| "mirror".into()),
} }
}) })
@@ -286,11 +338,12 @@ impl Core {
pub(crate) fn restore_sync_folders( pub(crate) fn restore_sync_folders(
&self, &self,
items: &[pdfs_core::control::RestoreItem], items: &[pdfs_core::control::RestoreItem],
device: Option<&str>,
) -> CoreResult<String> { ) -> CoreResult<String> {
if items.is_empty() { if items.is_empty() {
return Err(CoreError::invalid("nothing to restore")); return Err(CoreError::invalid("nothing to restore"));
} }
let device = self.ensure_device()?; let device = self.restore_source(device)?;
let existing = self let existing = self
.db .db
.sync_folder_list() .sync_folder_list()
@@ -319,10 +372,10 @@ impl Core {
skipped.push(format!("{}: {e}", item.local_path)); skipped.push(format!("{}: {e}", item.local_path));
continue; continue;
} }
let mode = match item.mode.as_str() { let mode = match restore_mode(&item.mode) {
"mirror" | "ondemand" => item.mode.as_str(), Some(mode) => mode,
other => { None => {
skipped.push(format!("{}: unknown mode {other}", item.local_path)); skipped.push(format!("{}: unknown mode {}", item.local_path, item.mode));
continue; continue;
} }
}; };
@@ -333,7 +386,7 @@ impl Core {
// `ondemand` is applied through the normal request path, which // `ondemand` is applied through the normal request path, which
// mounts the FUSE session and handles the folder being busy — the // mounts the FUSE session and handles the folder being busy — the
// restore does not get its own copy of that logic. // restore does not get its own copy of that logic.
if mode == "ondemand" { if mode == MountMode::OnDemand {
let _ = self.request_sync_folder_mode(id, "ondemand"); let _ = self.request_sync_folder_mode(id, "ondemand");
} }
let _ = self.sync_tx.send(sync::SyncMsg::Reconcile(id)); let _ = self.sync_tx.send(sync::SyncMsg::Reconcile(id));
@@ -350,6 +403,17 @@ impl Core {
} }
} }
/// Accept only modes this build can safely restore. `MountMode` itself is
/// tolerant so a future value does not reject the whole profile; the folder
/// carrying that value is skipped here with the message assembled above.
fn restore_mode(value: &str) -> Option<MountMode> {
match MountMode::from(value) {
MountMode::Mirror => Some(MountMode::Mirror),
MountMode::OnDemand => Some(MountMode::OnDemand),
MountMode::Unknown => None,
}
}
/// Whether a path recorded on another machine can be reused verbatim here. /// Whether a path recorded on another machine can be reused verbatim here.
/// ///
/// Requires the *parent* to exist: `~/Documents` on a machine with the same /// Requires the *parent* to exist: `~/Documents` on a machine with the same
@@ -382,6 +446,103 @@ mod tests {
assert!(!path_is_usable_here(Path::new("relative/path"))); assert!(!path_is_usable_here(Path::new("relative/path")));
} }
#[test]
fn unknown_profile_mode_is_skipped_without_rejecting_known_modes() {
assert_eq!(restore_mode("mirror"), Some(MountMode::Mirror));
assert_eq!(restore_mode("ondemand"), Some(MountMode::OnDemand));
assert_eq!(restore_mode("streaming"), None);
assert_eq!(restore_mode("unknown"), None);
}
/// The device root refuses file creation (B78), so every upload in
/// `save_profile` must be addressed to the profile folder. Checked against
/// the source because both branches are pure network calls.
#[test]
fn the_profile_is_never_uploaded_to_the_device_root() {
let source = include_str!("profile.rs");
let save = function_source(source, "fn save_profile(");
assert!(
save.contains("ensure_profile_dir(&root_uid)"),
"save_profile resolves the profile folder"
);
for upload in [
"upload_new_revision_from",
"upload_file_replacing_draft_from",
] {
let call = save.find(upload).expect("upload call exists");
let args = &save[call..];
let root = args.find("&root_uid").unwrap_or(usize::MAX);
let dir = args.find("&dir_uid").unwrap_or(usize::MAX);
assert!(dir < root, "{upload} must target the profile folder");
}
}
/// A read has to keep finding profiles written before the fix, which sat in
/// the device root.
#[test]
fn loading_falls_back_to_a_legacy_device_root_profile() {
let source = include_str!("profile.rs");
let load = function_source(source, "fn load_profile(");
assert!(load.contains("PROFILE_DIR_NAME"), "prefers the folder");
assert!(
load.find("PROFILE_DIR_NAME").unwrap()
< load.find("legacy profile").expect("legacy fallback exists"),
"the folder is consulted before the device root"
);
}
/// The profile folder is the client's own bookkeeping: neither the restore
/// picker nor `add_sync_folder` may treat it as a user folder.
#[test]
fn the_profile_folder_is_not_offered_as_user_data() {
let restorable = function_source(include_str!("profile.rs"), "fn list_restorable_folders(");
assert!(
restorable.contains("n.name != PROFILE_DIR_NAME"),
"restorable folders exclude the profile folder"
);
let add = function_source(include_str!("devices.rs"), "fn add_sync_folder(");
assert_before(add, "PROFILE_DIR_NAME", "ensure_device()");
}
/// Restoring another computer's folders reads that device, but must not
/// rewrite the cached device row: this machine would then back up as the
/// other computer.
#[test]
fn restoring_from_another_device_keeps_this_machines_identity() {
let source = function_source(include_str!("devices.rs"), "fn restore_source(");
assert!(source.contains("ensure_device()"), "None is this machine");
assert!(
!source.contains("device_set"),
"the cached row is untouched"
);
}
fn function_source<'a>(source: &'a str, signature: &str) -> &'a str {
let start = source.find(signature).expect("function signature exists");
let body = &source[start..];
let open = body.find('{').expect("function body starts");
let mut depth = 0usize;
for (offset, byte) in body.as_bytes()[open..].iter().enumerate() {
match byte {
b'{' => depth += 1,
b'}' => {
depth -= 1;
if depth == 0 {
return &body[..open + offset + 1];
}
}
_ => {}
}
}
panic!("function body closes")
}
fn assert_before(source: &str, first: &str, second: &str) {
let first_at = source.find(first).expect("first operation exists");
let second_at = source.find(second).expect("second operation exists");
assert!(first_at < second_at, "{first} must precede {second}");
}
/// A path whose parent exists but which does not itself is exactly the /// A path whose parent exists but which does not itself is exactly the
/// restore case — the directory is about to be created. /// restore case — the directory is about to be created.
#[test] #[test]
+60
View File
@@ -0,0 +1,60 @@
//! The user's view of the pending-op queue: what has not reached the remote
//! yet, why, and a way to stop waiting out a backoff.
//!
//! Read-only apart from "retry now", which only moves a row's next-attempt time.
//! Nothing here drops a queued op: its staged blob is the only copy of the
//! user's write (see `docs/RECOVERY.md`).
use super::*;
use pdfs_core::control::PendingOpInfo;
use pdfs_core::db::{FAILING_ATTEMPTS, PARK_UNTIL};
impl Core {
/// Every queued op, oldest first, with a path a person can read.
pub(crate) fn pending_op_infos(&self) -> CoreResult<Vec<PendingOpInfo>> {
let ops = self
.db
.pending_ops()
.map_err(|e| CoreError::internal(format!("listing the queue: {e}")))?;
Ok(ops
.into_iter()
.map(|op| {
let parked = op.next_attempt_at >= PARK_UNTIL;
let path = self
.db
.node_path(&op.uid)
.ok()
.flatten()
.filter(|path| !path.is_empty())
.or_else(|| op.name.clone())
.unwrap_or_else(|| op.uid.clone());
PendingOpInfo {
id: op.id,
kind: op.kind,
path,
attempts: op.attempts,
last_error: op.last_error,
queued_at: op.created_at / 1000,
next_attempt_at: (!parked).then_some(op.next_attempt_at / 1000),
parked,
failing: op.attempts >= FAILING_ATTEMPTS,
}
})
.collect())
}
/// Retry one backed-off op now, or every failed one when `id` is `None`.
/// Returns how many ops were moved up.
pub(crate) fn retry_pending_ops(&self, id: Option<i64>) -> CoreResult<usize> {
let now = now_millis();
let moved = match id {
Some(id) => self.db.retry_op_now(id, now).map(usize::from),
None => self.db.retry_failed_ops_now(now),
}
.map_err(|e| CoreError::internal(format!("retrying the queue: {e}")))?;
if moved > 0 {
self.wake_drain();
}
Ok(moved)
}
}
File diff suppressed because it is too large. Load diff
+332
View File
@@ -0,0 +1,332 @@
//! Repairing photos whose capture time is when they were *imported* rather than
//! when they were taken.
//!
//! A Google Photos Takeout whose metadata sidecars did not match — a localized
//! `.supplemental-metadata` spelling, say — imports with no capture time at all,
//! and the SDK then seals "now" into every revision. A decade of photos lands on
//! one day.
//!
//! Proton has no API for editing a sealed capture time, so the repair is a
//! rewrite: download the photo, upload it again with the capture time its file
//! name implies, put the new copy back into the albums and favorites the old
//! one was in, then **trash** (not delete) the original. Trash rather than
//! delete so a bad run is recoverable from Proton's own trash for 30 days.
//!
//! A photo is a candidate only when its name carries a date that disagrees with
//! its stored capture time by more than [`DRIFT`]. That is deliberately narrow:
//! a photo whose name says nothing about its date is left alone, because there
//! is nothing better to set it to.
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use pdfs_core::control::{RedateSummary, TransferDirection};
use pdfs_core::takeout::{capture_time_from_name, media_type_of};
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::proton_sdk::ids::NodeUid;
use proton_drive_rs::{PhotoTag, PhotoTagsUpdate, PhotoUploadMetadata};
use tracing::{info, warn};
use super::Core;
use crate::parse_uid as parse_node_uid;
use crate::transfers::CountingReader;
/// How far a stored capture time may sit from the one the file name implies
/// before the photo counts as mis-dated. A day, because a name usually carries
/// only the date and the stored time carries a clock — and because a timezone
/// can legitimately move a photo by hours.
const DRIFT: i64 = 86_400;
/// How many examples a report carries.
const SAMPLES: usize = 10;
/// The one re-date this daemon may be running, and how the last one ended.
/// Process-global for the same reasons as the Takeout import's state: one run
/// per daemon however many `Core` clones exist, and a front-end asking after it
/// finished must still find the report.
static REDATE: std::sync::LazyLock<parking_lot::Mutex<RedateState>> =
std::sync::LazyLock::new(|| parking_lot::Mutex::new(RedateState::default()));
#[derive(Default)]
struct RedateState {
running: bool,
cancel: Option<Arc<AtomicBool>>,
last: Option<RedateSummary>,
}
/// Whether a re-date is running, and the last one's report.
pub(crate) fn redate_status() -> (bool, Option<RedateSummary>) {
let state = REDATE.lock();
(state.running, state.last.clone())
}
/// Ask the running re-date to stop. False when none is running.
pub(crate) fn cancel_redate() -> bool {
match &REDATE.lock().cancel {
Some(flag) => {
flag.store(true, Ordering::Relaxed);
true
}
None => false,
}
}
/// One photo to repair.
struct Candidate {
uid: NodeUid,
name: String,
/// The capture time it carries now.
stored: i64,
/// The capture time its name implies.
wanted: i64,
favorite: bool,
/// Album node uids the original is filed under.
albums: Vec<String>,
}
impl Core {
/// Re-date mis-dated photos. Blocking; run on its own thread.
///
/// `cancel` stops the run at the next photo boundary. A photo is only ever
/// trashed after its replacement has uploaded, so a cancel — or a crash —
/// can leave a duplicate, never a hole.
pub(crate) fn redate_photos(
&self,
dry_run: bool,
limit: Option<usize>,
range: Option<(i64, i64)>,
cancel: Arc<AtomicBool>,
) -> CoreResult<RedateSummary> {
let job = self.transfers.begin_job("Fixing photo dates");
job.detail("looking for mis-dated photos");
let stored = self
.db
.photos_all()
.map_err(|e| CoreError::internal(format!("read timeline: {e}")))?;
let mut summary = RedateSummary {
examined: stored.len(),
dry_run,
..Default::default()
};
let mut candidates = Vec::new();
for photo in stored {
if let Some((from, to)) = range
&& !(from..to).contains(&photo.capture_time)
{
continue;
}
let Some(name) = photo.name.filter(|n| !n.is_empty()) else {
continue;
};
let Some(wanted) = capture_time_from_name(&name) else {
continue;
};
if (wanted - photo.capture_time).abs() <= DRIFT {
continue;
}
let Some(uid) = parse_node_uid(&photo.uid) else {
continue;
};
candidates.push(Candidate {
uid,
name,
stored: photo.capture_time,
wanted,
favorite: photo.favorite,
albums: self.db.albums_of_photo(&photo.uid).unwrap_or_default(),
});
}
if let Some(limit) = limit {
candidates.truncate(limit);
}
summary.candidates = candidates.len();
// Sampled after the limit, so the examples are of photos the run would
// actually touch.
summary.samples = candidates
.iter()
.take(SAMPLES)
.map(|c| (c.name.clone(), c.stored, c.wanted))
.collect();
job.set_total(candidates.len() as u64);
info!(
examined = summary.examined,
candidates = summary.candidates,
dry_run,
"photo re-date scan complete"
);
if dry_run || candidates.is_empty() {
return Ok(summary);
}
for candidate in candidates {
if cancel.load(Ordering::Relaxed) || self.shutdown.is_stopping() {
break;
}
job.detail(candidate.name.clone());
if let Err(e) = self.redate_one(&candidate, &mut summary) {
summary.failed += 1;
warn!(name = candidate.name, error = %e, "cannot re-date photo");
}
job.step();
}
self.spawn_timeline_refresh();
summary.cancelled = cancel.load(Ordering::Relaxed);
Ok(summary)
}
/// Rewrite one photo at the capture time its name implies.
///
/// Order is load-bearing: upload first, file the new copy, and only then
/// trash the original. Every earlier step failing leaves the account exactly
/// as it was.
fn redate_one(&self, candidate: &Candidate, summary: &mut RedateSummary) -> CoreResult<()> {
let photos = self.photos();
let path = self.open_photo(&candidate.uid)?;
let size = std::fs::metadata(&path)
.map_err(|e| CoreError::internal(format!("stat {}: {e}", path.display())))?
.len();
let media_type = media_type_of(&candidate.name).unwrap_or("application/octet-stream");
let guard =
self.transfers
.begin(candidate.name.clone(), "", TransferDirection::Upload, size);
let file = std::fs::File::open(&path)
.map_err(|e| CoreError::internal(format!("open {}: {e}", path.display())))?;
let reader = CountingReader::new(std::io::BufReader::new(file), &guard);
let metadata = PhotoUploadMetadata {
capture_time: Some(candidate.wanted),
..Default::default()
};
let uploaded = self.rt.block_on(photos.upload_photo_from(
&candidate.name,
media_type,
reader,
size as i64,
Vec::new(),
metadata,
false,
));
drop(guard);
let new_uid = uploaded.map_err(|e| CoreError::from_api(&e, "re-upload photo"))?;
// Favorite and albums before the trash: a failure here leaves both
// copies on the account, which is recoverable by hand. Losing the
// original first would not be.
if candidate.favorite {
let update = PhotoTagsUpdate {
node_uid: new_uid.clone(),
tags_to_add: vec![PhotoTag::Favorite],
tags_to_remove: Vec::new(),
};
match self.rt.block_on(photos.update_photos(&[update])) {
// As in `set_photo_favorite`: claim our own echo, so the tag we
// just set does not come back as a remote change.
Ok(_) => self.note_self_change(&new_uid),
Err(e) => {
warn!(name = candidate.name, error = %e, "cannot re-favorite re-dated photo")
}
}
}
for album in &candidate.albums {
let Some(album_uid) = parse_node_uid(album) else {
continue;
};
match self
.rt
.block_on(photos.add_photos_to_album(&album_uid, std::slice::from_ref(&new_uid)))
{
Ok(_) => summary.album_links += 1,
Err(e) => {
warn!(name = candidate.name, error = %e, "cannot re-file re-dated photo")
}
}
}
let outcomes = self
.rt
.block_on(
self.client
.trash_nodes(std::slice::from_ref(&candidate.uid)),
)
.map_err(|e| CoreError::from_api(&e, "trash the mis-dated original"))?;
if let Some((_, Err(e))) = outcomes.into_iter().next() {
return Err(CoreError::from_api(&e, "trash the mis-dated original"));
}
summary.redated += 1;
summary.bytes += size;
info!(
name = candidate.name,
from = candidate.stored,
to = candidate.wanted,
"photo re-dated"
);
Ok(())
}
/// Start a re-date on its own thread. Errors when one is already running.
pub(crate) fn spawn_redate(
&self,
dry_run: bool,
limit: Option<usize>,
range: Option<(i64, i64)>,
) -> CoreResult<()> {
let cancel = Arc::new(AtomicBool::new(false));
{
let mut state = REDATE.lock();
if state.running {
return Err(CoreError::conflict(
"a photo re-date is already running; wait for it to finish or cancel it",
));
}
state.running = true;
state.cancel = Some(cancel.clone());
state.last = None;
}
let core = self.clone();
std::thread::spawn(move || {
let outcome = core.redate_photos(dry_run, limit, range, cancel);
let mut state = REDATE.lock();
state.running = false;
state.cancel = None;
state.last = outcome.as_ref().ok().cloned();
match &outcome {
Ok(summary) => info!(
candidates = summary.candidates,
redated = summary.redated,
failed = summary.failed,
"photo re-date finished"
),
Err(e) => warn!(error = %e, "photo re-date failed"),
}
});
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_uid_without_both_halves_is_not_a_node() {
assert!(parse_node_uid("vol~link").is_some());
assert!(parse_node_uid("vol~").is_none());
assert!(parse_node_uid("~link").is_none());
assert!(parse_node_uid("nothing").is_none());
}
#[test]
fn only_a_disagreement_larger_than_a_day_counts() {
// The name says 2023-02-19; a stored time inside that day is fine.
let wanted = capture_time_from_name("IMG-20230219-WA0001.jpg").unwrap();
assert!((wanted - (wanted + 3_600)).abs() <= DRIFT);
// An import three years later is not.
assert!((wanted - (wanted + 3 * 365 * 86_400)).abs() > DRIFT);
}
}
+264
View File
@@ -0,0 +1,264 @@
//! File version history: listing a file's revisions, restoring one, deleting
//! one, and writing an old one out to a local path.
//!
//! Proton Drive keeps every revision a client ever committed, and until now the
//! daemon only ever addressed the active one — a file that a sync pass or a
//! `(sync-conflict)` fork overwrote could only be recovered from whatever the
//! local `recovery/` directory happened to hold. This is the remote equivalent,
//! and it is the reason a *restore* here is not a download-then-upload: the
//! server re-points the file at an existing revision, so no content crosses the
//! wire and nothing lands in the drain queue.
//!
//! Two things about the SDK surface leak into every method here:
//!
//! - **A restore is asynchronous.** The server answers 202 and swaps the active
//! revision in the background, so the daemon drops its cached content for the
//! file rather than trying to describe the new one. The next read repopulates
//! from whatever is live by then.
//! - **The active revision cannot be deleted**, and the server — not this code —
//! is what enforces that. [`Core::delete_revision_for_uid`] refuses locally
//! too, so the user gets a sentence instead of an API code.
use std::path::Path;
use pdfs_core::control::{ActivityKind, RevisionInfo};
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::Revision;
use proton_drive_rs::proton_sdk::ids::NodeUid;
use tracing::info;
use super::Core;
/// A [`Revision`] as the control protocol reports it.
fn revision_info(revision: Revision) -> RevisionInfo {
RevisionInfo {
is_active: revision.is_active(),
id: revision.revision_id,
created: revision.creation_time,
size_on_storage: revision.size_on_storage,
claimed_size: revision.claimed_size,
claimed_modified: revision.claimed_modification_time,
// An empty signer means the node key signed it anonymously, which is not
// an address a front-end can show.
signed_by: revision.signature_email.filter(|email| !email.is_empty()),
has_thumbnails: revision.has_thumbnails,
}
}
impl Core {
/// The version history of the file at mountpoint-relative `rel`.
pub(crate) fn list_revisions(&self, rel: &Path) -> CoreResult<Vec<RevisionInfo>> {
let (_ino, uid) = self.resolve(rel)?;
self.list_revisions_for_uid(&uid)
}
/// [`Core::list_revisions`] for a node addressed through any daemon location.
pub(crate) fn list_revisions_by_uid(&self, uid: &str) -> CoreResult<Vec<RevisionInfo>> {
let uid = self.resolve_anywhere(uid)?;
self.list_revisions_for_uid(&uid)
}
fn list_revisions_for_uid(&self, uid: &NodeUid) -> CoreResult<Vec<RevisionInfo>> {
let revisions = self
.rt
.block_on(self.client.enumerate_revisions(uid))
.map_err(|e| CoreError::from_api(&e, "list revisions"))?;
Ok(revisions.into_iter().map(revision_info).collect())
}
/// Make `revision_id` the current content of the file at `rel`.
pub(crate) fn restore_revision(&self, rel: &Path, revision_id: &str) -> CoreResult<()> {
let (_ino, uid) = self.resolve(rel)?;
let name = rel
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_else(|| uid.to_string());
self.restore_revision_for_uid(&uid, revision_id, &name)?;
// The size and mtime a listing shows describe the revision that was
// active a moment ago.
self.invalidate_parent_listing(rel);
Ok(())
}
/// [`Core::restore_revision`] for a node addressed through any daemon
/// location.
pub(crate) fn restore_revision_by_uid(&self, uid: &str, revision_id: &str) -> CoreResult<()> {
let uid = self.resolve_anywhere(uid)?;
let name = uid.to_string();
self.restore_revision_for_uid(&uid, revision_id, &name)
}
fn restore_revision_for_uid(
&self,
uid: &NodeUid,
revision_id: &str,
name: &str,
) -> CoreResult<()> {
// Restoring rewrites the file's content: the same authority a write
// needs, not the one a read needs.
self.require_uid_writable(uid)
.map_err(|error| self.errno_error(error, "restore revision"))?;
if let Err(e) = self
.rt
.block_on(self.client.restore_revision(uid, revision_id))
{
let error = CoreError::from_api(&e, "restore revision");
self.log_activity(ActivityKind::Restore, name, &error, false);
return Err(error);
}
// The active revision is changing behind our back — anything describing
// the old one (cached blocks, an open reader, the size in a listing) is
// stale from here on, and the server may not have applied the swap yet.
self.cache.evict(uid);
self.evict_reader(uid);
self.log_activity(
ActivityKind::Restore,
name,
format!("restored version {revision_id}"),
true,
);
info!(%uid, revision_id, name, "restored an earlier revision");
Ok(())
}
/// Permanently delete `revision_id` from the history of the file at `rel`.
pub(crate) fn delete_revision(&self, rel: &Path, revision_id: &str) -> CoreResult<()> {
let (_ino, uid) = self.resolve(rel)?;
let name = rel
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_else(|| uid.to_string());
self.delete_revision_for_uid(&uid, revision_id, &name)
}
/// [`Core::delete_revision`] for a node addressed through any daemon
/// location.
pub(crate) fn delete_revision_by_uid(&self, uid: &str, revision_id: &str) -> CoreResult<()> {
let uid = self.resolve_anywhere(uid)?;
let name = uid.to_string();
self.delete_revision_for_uid(&uid, revision_id, &name)
}
fn delete_revision_for_uid(
&self,
uid: &NodeUid,
revision_id: &str,
name: &str,
) -> CoreResult<()> {
self.require_uid_writable(uid)
.map_err(|error| self.errno_error(error, "delete revision"))?;
// The server rejects deleting the active revision; saying so here is a
// sentence the user can act on instead of an API code, and it costs one
// listing the front-end already made to show the version list.
let revisions = self
.rt
.block_on(self.client.enumerate_revisions(uid))
.map_err(|e| CoreError::from_api(&e, "list revisions"))?;
let target = revisions
.iter()
.find(|r| r.revision_id == revision_id)
.ok_or_else(|| CoreError::not_found(format!("no version {revision_id} on {name}")))?;
if target.is_active() {
return Err(CoreError::invalid(format!(
"version {revision_id} is the current content of {name}; restore another version first"
)));
}
self.rt
.block_on(self.client.delete_revision(uid, revision_id))
.map_err(|e| CoreError::from_api(&e, "delete revision"))?;
self.log_activity(
ActivityKind::DeleteForever,
name,
format!("deleted version {revision_id}"),
true,
);
info!(%uid, revision_id, name, "deleted a revision");
Ok(())
}
/// Write `revision_id` of the file at `rel` to the absolute local path
/// `dest`, leaving the file itself untouched.
pub(crate) fn save_revision_as(
&self,
rel: &Path,
revision_id: &str,
dest: &Path,
) -> CoreResult<u64> {
let (_ino, uid) = self.resolve(rel)?;
let name = rel
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_else(|| uid.to_string());
self.save_revision_for_uid(&uid, revision_id, &name, dest)
}
/// [`Core::save_revision_as`] for a node addressed through any daemon
/// location.
pub(crate) fn save_revision_as_by_uid(
&self,
uid: &str,
revision_id: &str,
dest: &Path,
) -> CoreResult<u64> {
let uid = self.resolve_anywhere(uid)?;
let name = uid.to_string();
self.save_revision_for_uid(&uid, revision_id, &name, dest)
}
fn save_revision_for_uid(
&self,
uid: &NodeUid,
revision_id: &str,
name: &str,
dest: &Path,
) -> CoreResult<u64> {
if !dest.is_absolute() {
return Err(CoreError::invalid(format!(
"destination must be an absolute path: {}",
dest.display()
)));
}
// Never silently replace something the user already has: an old version
// written over the wrong file is exactly the loss this feature exists to
// undo.
if dest.exists() {
return Err(CoreError::invalid(format!(
"{} already exists",
dest.display()
)));
}
let file = std::fs::File::create(dest)
.map_err(|e| CoreError::internal(format!("create {}: {e}", dest.display())))?;
let mut out = std::io::BufWriter::new(file);
let written = match self
.rt
.block_on(self.client.download_revision_to(uid, revision_id, &mut out))
.map_err(|e| CoreError::from_api(&e, "download revision"))
.and_then(|()| {
use std::io::Write as _;
out.flush()
.map_err(|e| CoreError::internal(format!("write {}: {e}", dest.display())))?;
let written = std::fs::metadata(dest)
.map(|m| m.len())
.map_err(|e| CoreError::internal(format!("stat {}: {e}", dest.display())))?;
Ok(written)
}) {
Ok(written) => written,
Err(error) => {
// A half-written file looks like a successful export to every
// tool that opens it afterwards.
let _ = std::fs::remove_file(dest);
self.log_activity(ActivityKind::Download, name, &error, false);
return Err(error);
}
};
self.log_activity(
ActivityKind::Download,
name,
format!("saved version {revision_id} to {}", dest.display()),
true,
);
info!(%uid, revision_id, dest = %dest.display(), "wrote an earlier revision to disk");
Ok(written)
}
}
+361 -31
View File
@@ -12,6 +12,10 @@ use pdfs_core::control::{
BookmarkInfo, DirEntry, InvitationInfo, PublicLinkInfo, ShareEntry, ShareEntryKind, SharedItem, BookmarkInfo, DirEntry, InvitationInfo, PublicLinkInfo, ShareEntry, ShareEntryKind, SharedItem,
}; };
use pdfs_core::{CoreError, CoreResult}; use pdfs_core::{CoreError, CoreResult};
use tracing::debug;
use crate::QUOTA_TTL;
use proton_drive_rs::proton_sdk::crypto::VerificationStatus;
use proton_drive_rs::proton_sdk::ids::NodeUid; use proton_drive_rs::proton_sdk::ids::NodeUid;
use proton_drive_rs::{MemberRole, NodeKind}; use proton_drive_rs::{MemberRole, NodeKind};
@@ -19,10 +23,15 @@ use proton_drive_rs::Node;
use super::{Core, ROOT_INO, node_size, parse_uid, public_link_info, role_from_str, role_to_str}; use super::{Core, ROOT_INO, node_size, parse_uid, public_link_info, role_from_str, role_to_str};
/// A node from a *shared* listing as a [`DirEntry`]. Shared nodes live outside /// A node from a *shared* listing as a [`DirEntry`].
/// the mount, so the entry is uid-addressed: `path` is empty, and the local-state ///
/// flags (`pinned`, `cached`) are false — they describe my own tree, not this one. /// Still uid-addressed — that is the only handle a foreign-volume node is
fn shared_entry(n: Node) -> DirEntry { /// guaranteed to have — but `path` is filled in when the node is resident under
/// the synthetic `Shared with me/` directory, which is what lets a front-end
/// offer path-keyed actions on it. The local-state flags (`pinned`, `cached`)
/// stay false: they describe content this account owns a local copy policy for,
/// which a share on someone else's volume is not.
fn shared_entry_of(core: &Core, n: Node) -> DirEntry {
let is_dir = n.is_folder(); let is_dir = n.is_folder();
let size = match &n.kind { let size = match &n.kind {
NodeKind::File { NodeKind::File {
@@ -32,6 +41,25 @@ fn shared_entry(n: Node) -> DirEntry {
} => claimed_size.unwrap_or(*total_size_on_storage).max(0) as u64, } => claimed_size.unwrap_or(*total_size_on_storage).max(0) as u64,
NodeKind::Folder => 0, NodeKind::Folder => 0,
}; };
let role = n
.membership
.as_ref()
.and_then(|m| m.role_exact())
.map(role_to_str)
.unwrap_or_default()
.to_string();
let invitation = n.membership.as_ref();
let shared_by = invitation
.and_then(|m| m.inviter_email.clone())
.unwrap_or_default();
let shared_at = invitation.and_then(|m| m.invite_time).unwrap_or(0);
// Only a signature that is present and wrong is a warning. An invitation
// that could not be checked — unsigned, or no keys for the claimed inviter —
// is not evidence of forgery, and flagging every such share would teach the
// user to ignore the flag.
let shared_by_unverified = invitation
.and_then(|m| m.inviter_verification)
.is_some_and(|status| status == VerificationStatus::Failed);
DirEntry { DirEntry {
name: n.name, name: n.name,
is_dir, is_dir,
@@ -39,8 +67,14 @@ fn shared_entry(n: Node) -> DirEntry {
modified: n.modification_time, modified: n.modification_time,
pinned: false, pinned: false,
cached: false, cached: false,
path: core.rel_path_for_uid(&n.uid).unwrap_or_default(),
uid: n.uid.to_string(), uid: n.uid.to_string(),
path: String::new(), role,
shared_by,
shared_at,
shared_by_unverified,
trashed_at: 0,
trashed_from: None,
} }
} }
@@ -57,11 +91,33 @@ impl Core {
message: Option<&str>, message: Option<&str>,
) -> CoreResult<(usize, usize)> { ) -> CoreResult<(usize, usize)> {
let (_ino, uid) = self.resolve(rel)?; let (_ino, uid) = self.resolve(rel)?;
self.share_node_for_uid(&uid, emails, role, message)
}
/// [`Core::share_node`] for a node addressed through any daemon location.
pub(crate) fn share_node_by_uid(
&self,
uid: &str,
emails: &[String],
role: &str,
message: Option<&str>,
) -> CoreResult<(usize, usize)> {
let uid = self.resolve_anywhere(uid)?;
self.share_node_for_uid(&uid, emails, role, message)
}
fn share_node_for_uid(
&self,
uid: &NodeUid,
emails: &[String],
role: &str,
message: Option<&str>,
) -> CoreResult<(usize, usize)> {
let role = role_from_str(role)?; let role = role_from_str(role)?;
let invitees: Vec<(String, MemberRole)> = let invitees: Vec<(String, MemberRole)> =
emails.iter().map(|e| (e.clone(), role)).collect(); emails.iter().map(|e| (e.clone(), role)).collect();
self.rt self.rt
.block_on(self.client.invite_users(&uid, &invitees, message)) .block_on(self.client.invite_users(uid, &invitees, message))
.map_err(|e| CoreError::from_api(&e, "share")) .map_err(|e| CoreError::from_api(&e, "share"))
} }
@@ -71,11 +127,26 @@ impl Core {
rel: &Path, rel: &Path,
) -> CoreResult<(Vec<ShareEntry>, Option<PublicLinkInfo>)> { ) -> CoreResult<(Vec<ShareEntry>, Option<PublicLinkInfo>)> {
let (_ino, uid) = self.resolve(rel)?; let (_ino, uid) = self.resolve(rel)?;
self.list_share_for_uid(&uid)
}
/// [`Core::list_share`] for a node addressed through any daemon location.
pub(crate) fn list_share_by_uid(
&self,
uid: &str,
) -> CoreResult<(Vec<ShareEntry>, Option<PublicLinkInfo>)> {
let uid = self.resolve_anywhere(uid)?;
self.list_share_for_uid(&uid)
}
fn list_share_for_uid(
&self,
uid: &NodeUid,
) -> CoreResult<(Vec<ShareEntry>, Option<PublicLinkInfo>)> {
let mut entries = Vec::new(); let mut entries = Vec::new();
for m in self for m in self
.rt .rt
.block_on(self.client.list_share_members(&uid)) .block_on(self.client.list_share_members(uid))
.map_err(|e| CoreError::from_api(&e, "list members"))? .map_err(|e| CoreError::from_api(&e, "list members"))?
{ {
entries.push(ShareEntry { entries.push(ShareEntry {
@@ -87,7 +158,7 @@ impl Core {
} }
for inv in self for inv in self
.rt .rt
.block_on(self.client.list_share_invitations(&uid)) .block_on(self.client.list_share_invitations(uid))
.map_err(|e| CoreError::from_api(&e, "list invitations"))? .map_err(|e| CoreError::from_api(&e, "list invitations"))?
{ {
entries.push(ShareEntry { entries.push(ShareEntry {
@@ -99,7 +170,7 @@ impl Core {
} }
for ext in self for ext in self
.rt .rt
.block_on(self.client.list_external_invitations(&uid)) .block_on(self.client.list_external_invitations(uid))
.map_err(|e| CoreError::from_api(&e, "list external invitations"))? .map_err(|e| CoreError::from_api(&e, "list external invitations"))?
{ {
entries.push(ShareEntry { entries.push(ShareEntry {
@@ -112,7 +183,7 @@ impl Core {
let link = self let link = self
.rt .rt
.block_on(self.client.get_public_link(&uid)) .block_on(self.client.get_public_link(uid))
.map_err(|e| CoreError::from_api(&e, "get public link"))? .map_err(|e| CoreError::from_api(&e, "get public link"))?
.map(public_link_info); .map(public_link_info);
@@ -129,12 +200,35 @@ impl Core {
role: &str, role: &str,
) -> CoreResult<()> { ) -> CoreResult<()> {
let (_ino, uid) = self.resolve(rel)?; let (_ino, uid) = self.resolve(rel)?;
self.update_share_role_for_uid(&uid, id, kind, role)
}
/// [`Core::update_share_role`] for a node addressed through any daemon
/// location.
pub(crate) fn update_share_role_by_uid(
&self,
uid: &str,
id: &str,
kind: ShareEntryKind,
role: &str,
) -> CoreResult<()> {
let uid = self.resolve_anywhere(uid)?;
self.update_share_role_for_uid(&uid, id, kind, role)
}
fn update_share_role_for_uid(
&self,
uid: &NodeUid,
id: &str,
kind: ShareEntryKind,
role: &str,
) -> CoreResult<()> {
let role = role_from_str(role)?; let role = role_from_str(role)?;
match kind { match kind {
ShareEntryKind::Member => { ShareEntryKind::Member => {
let member = self let member = self
.rt .rt
.block_on(self.client.list_share_members(&uid)) .block_on(self.client.list_share_members(uid))
.map_err(|e| CoreError::from_api(&e, "list members"))? .map_err(|e| CoreError::from_api(&e, "list members"))?
.into_iter() .into_iter()
.find(|m| m.membership_id.to_string() == id) .find(|m| m.membership_id.to_string() == id)
@@ -146,7 +240,7 @@ impl Core {
ShareEntryKind::ProtonInvite => { ShareEntryKind::ProtonInvite => {
let inv = self let inv = self
.rt .rt
.block_on(self.client.list_share_invitations(&uid)) .block_on(self.client.list_share_invitations(uid))
.map_err(|e| CoreError::from_api(&e, "list invitations"))? .map_err(|e| CoreError::from_api(&e, "list invitations"))?
.into_iter() .into_iter()
.find(|i| i.invitation_id == id) .find(|i| i.invitation_id == id)
@@ -170,11 +264,32 @@ impl Core {
kind: ShareEntryKind, kind: ShareEntryKind,
) -> CoreResult<()> { ) -> CoreResult<()> {
let (_ino, uid) = self.resolve(rel)?; let (_ino, uid) = self.resolve(rel)?;
self.remove_share_entry_for_uid(&uid, id, kind)
}
/// [`Core::remove_share_entry`] for a node addressed through any daemon
/// location.
pub(crate) fn remove_share_entry_by_uid(
&self,
uid: &str,
id: &str,
kind: ShareEntryKind,
) -> CoreResult<()> {
let uid = self.resolve_anywhere(uid)?;
self.remove_share_entry_for_uid(&uid, id, kind)
}
fn remove_share_entry_for_uid(
&self,
uid: &NodeUid,
id: &str,
kind: ShareEntryKind,
) -> CoreResult<()> {
match kind { match kind {
ShareEntryKind::Member => { ShareEntryKind::Member => {
let member = self let member = self
.rt .rt
.block_on(self.client.list_share_members(&uid)) .block_on(self.client.list_share_members(uid))
.map_err(|e| CoreError::from_api(&e, "list members"))? .map_err(|e| CoreError::from_api(&e, "list members"))?
.into_iter() .into_iter()
.find(|m| m.membership_id.to_string() == id) .find(|m| m.membership_id.to_string() == id)
@@ -186,7 +301,7 @@ impl Core {
ShareEntryKind::ProtonInvite => { ShareEntryKind::ProtonInvite => {
let inv = self let inv = self
.rt .rt
.block_on(self.client.list_share_invitations(&uid)) .block_on(self.client.list_share_invitations(uid))
.map_err(|e| CoreError::from_api(&e, "list invitations"))? .map_err(|e| CoreError::from_api(&e, "list invitations"))?
.into_iter() .into_iter()
.find(|i| i.invitation_id == id) .find(|i| i.invitation_id == id)
@@ -198,7 +313,7 @@ impl Core {
ShareEntryKind::ExternalInvite => { ShareEntryKind::ExternalInvite => {
let ext = self let ext = self
.rt .rt
.block_on(self.client.list_external_invitations(&uid)) .block_on(self.client.list_external_invitations(uid))
.map_err(|e| CoreError::from_api(&e, "list external invitations"))? .map_err(|e| CoreError::from_api(&e, "list external invitations"))?
.into_iter() .into_iter()
.find(|i| i.invitation_id == id) .find(|i| i.invitation_id == id)
@@ -219,13 +334,33 @@ impl Core {
expires: Option<i64>, expires: Option<i64>,
) -> CoreResult<PublicLinkInfo> { ) -> CoreResult<PublicLinkInfo> {
let (_ino, uid) = self.resolve(rel)?; let (_ino, uid) = self.resolve(rel)?;
self.create_public_link_for_uid(&uid, role, password, expires)
}
/// [`Core::create_public_link`] for a node addressed through any daemon
/// location.
pub(crate) fn create_public_link_by_uid(
&self,
uid: &str,
role: &str,
password: Option<&str>,
expires: Option<i64>,
) -> CoreResult<PublicLinkInfo> {
let uid = self.resolve_anywhere(uid)?;
self.create_public_link_for_uid(&uid, role, password, expires)
}
fn create_public_link_for_uid(
&self,
uid: &NodeUid,
role: &str,
password: Option<&str>,
expires: Option<i64>,
) -> CoreResult<PublicLinkInfo> {
let role = role_from_str(role)?; let role = role_from_str(role)?;
let link = self let link = self
.rt .rt
.block_on( .block_on(self.client.create_public_link(uid, role, password, expires))
self.client
.create_public_link(&uid, role, password, expires),
)
.map_err(|e| CoreError::from_api(&e, "create public link"))?; .map_err(|e| CoreError::from_api(&e, "create public link"))?;
Ok(public_link_info(link)) Ok(public_link_info(link))
} }
@@ -233,9 +368,20 @@ impl Core {
/// Remove the public link `id` from the node at `rel`. /// Remove the public link `id` from the node at `rel`.
pub(crate) fn remove_public_link(&self, rel: &Path, id: &str) -> CoreResult<()> { pub(crate) fn remove_public_link(&self, rel: &Path, id: &str) -> CoreResult<()> {
let (_ino, uid) = self.resolve(rel)?; let (_ino, uid) = self.resolve(rel)?;
self.remove_public_link_for_uid(&uid, id)
}
/// [`Core::remove_public_link`] for a node addressed through any daemon
/// location.
pub(crate) fn remove_public_link_by_uid(&self, uid: &str, id: &str) -> CoreResult<()> {
let uid = self.resolve_anywhere(uid)?;
self.remove_public_link_for_uid(&uid, id)
}
fn remove_public_link_for_uid(&self, uid: &NodeUid, id: &str) -> CoreResult<()> {
let link = self let link = self
.rt .rt
.block_on(self.client.get_public_link(&uid)) .block_on(self.client.get_public_link(uid))
.map_err(|e| CoreError::from_api(&e, "get public link"))? .map_err(|e| CoreError::from_api(&e, "get public link"))?
.filter(|l| l.public_link_id == id) .filter(|l| l.public_link_id == id)
.ok_or_else(|| CoreError::not_found("public link not found"))?; .ok_or_else(|| CoreError::not_found("public link not found"))?;
@@ -244,6 +390,66 @@ impl Core {
.map_err(|e| CoreError::from_api(&e, "remove public link")) .map_err(|e| CoreError::from_api(&e, "remove public link"))
} }
/// Stop sharing the node at `rel`: remove its public link, then every
/// external invitation, Proton invitation and member. The SDK has no call
/// that deletes a share outright, so this composes the per-entry ones.
pub(crate) fn stop_sharing(&self, rel: &Path) -> CoreResult<StopSharingOutcome> {
let (_ino, uid) = self.resolve(rel)?;
self.stop_sharing_for_uid(&uid)
}
/// [`Core::stop_sharing`] for a node addressed through any daemon location.
pub(crate) fn stop_sharing_by_uid(&self, uid: &str) -> CoreResult<StopSharingOutcome> {
let uid = self.resolve_anywhere(uid)?;
self.stop_sharing_for_uid(&uid)
}
/// Each step runs even when an earlier one failed, so a single stuck
/// member does not leave the public link up. The link goes first: it is the
/// widest exposure. Only a failure to list what is there fails the whole
/// request; a failed removal is counted in the outcome.
fn stop_sharing_for_uid(&self, uid: &NodeUid) -> CoreResult<StopSharingOutcome> {
let mut outcome = StopSharingOutcome::default();
let link = self
.rt
.block_on(self.client.get_public_link(uid))
.map_err(|e| CoreError::from_api(&e, "get public link"))?;
if let Some(link) = link {
let result = self.rt.block_on(self.client.remove_public_link(&link));
outcome.record("the public link", result);
}
let external = self
.rt
.block_on(self.client.list_external_invitations(uid))
.map_err(|e| CoreError::from_api(&e, "list external invitations"))?;
for ext in external {
let result = self
.rt
.block_on(self.client.delete_external_invitation(&ext));
outcome.record(&ext.invitee_email, result);
}
let invitations = self
.rt
.block_on(self.client.list_share_invitations(uid))
.map_err(|e| CoreError::from_api(&e, "list invitations"))?;
for inv in invitations {
let result = self.rt.block_on(self.client.delete_invitation(&inv));
outcome.record(&inv.invitee_email, result);
}
let members = self
.rt
.block_on(self.client.list_share_members(uid))
.map_err(|e| CoreError::from_api(&e, "list members"))?;
for member in members {
let result = self.rt.block_on(self.client.remove_member(&member));
outcome.record(&member.email, result);
}
if !outcome.failed.is_empty() {
debug!(%uid, failed = ?outcome.failed, "stop sharing: some entries remain");
}
Ok(outcome)
}
// ---- shared with me --------------------------------------------------- // ---- shared with me ---------------------------------------------------
/// List nodes shared with me that I have accepted. /// List nodes shared with me that I have accepted.
@@ -259,7 +465,10 @@ impl Core {
.rt .rt
.block_on(self.client.enumerate_nodes(&uids)) .block_on(self.client.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?; .map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes.into_iter().map(shared_entry).collect()) Ok(nodes
.into_iter()
.map(|node| shared_entry_of(self, node))
.collect())
} }
/// List the children of a folder shared with me, addressed by uid. /// List the children of a folder shared with me, addressed by uid.
@@ -281,7 +490,10 @@ impl Core {
.rt .rt
.block_on(self.client.enumerate_nodes(&child_uids)) .block_on(self.client.enumerate_nodes(&child_uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?; .map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes.into_iter().map(shared_entry).collect()) Ok(nodes
.into_iter()
.map(|node| shared_entry_of(self, node))
.collect())
} }
/// Download a file shared with me into the content cache, returning its /// Download a file shared with me into the content cache, returning its
@@ -402,9 +614,33 @@ impl Core {
.rt .rt
.block_on(self.client.quota()) .block_on(self.client.quota())
.map_err(|e| CoreError::from_api(&e, "account quota"))?; .map_err(|e| CoreError::from_api(&e, "account quota"))?;
*self.quota.lock() = Some((std::time::Instant::now(), q.max_space, q.used_space));
Ok((q.max_space, q.used_space)) Ok((q.max_space, q.used_space))
} }
/// The account quota as `statfs(2)` needs it: cached for [`QUOTA_TTL`],
/// refetched after that, and `None` when there is no usable answer at all —
/// offline with nothing cached.
///
/// Never blocks on the network for a *fresh* value when a recent one exists,
/// because the callers are `df` and every "is there room?" preflight, and a
/// stale byte count is worth more to them than a stalled syscall. Offline,
/// the last known figures keep being served rather than degrading to zero.
pub(crate) fn account_quota_cached(&self) -> Option<(i64, i64)> {
if let Some((at, max_space, used_space)) = *self.quota.lock()
&& at.elapsed() < QUOTA_TTL
{
return Some((max_space, used_space));
}
match self.account_quota() {
Ok(q) => Some(q),
Err(error) => {
debug!(%error, "account quota refresh failed; serving the last known figures");
self.quota.lock().map(|(_, max, used)| (max, used))
}
}
}
// ---- shared by me ----------------------------------------------------- // ---- shared by me -----------------------------------------------------
/// List the nodes I have shared with others, each with a summary of its share /// List the nodes I have shared with others, each with a summary of its share
@@ -451,6 +687,7 @@ impl Core {
items.push(SharedItem { items.push(SharedItem {
uid: uid.to_string(), uid: uid.to_string(),
is_dir: n.is_folder(), is_dir: n.is_folder(),
modified: n.modification_time,
name: n.name, name: n.name,
path: self.rel_path_for_uid(&uid).unwrap_or_default(), path: self.rel_path_for_uid(&uid).unwrap_or_default(),
member_count: members, member_count: members,
@@ -466,15 +703,108 @@ impl Core {
/// been seen through the mount (e.g. shared but not browsed to this session) — /// been seen through the mount (e.g. shared but not browsed to this session) —
/// the caller then leaves the path empty. /// the caller then leaves the path empty.
pub(crate) fn rel_path_for_uid(&self, uid: &NodeUid) -> Option<String> { pub(crate) fn rel_path_for_uid(&self, uid: &NodeUid) -> Option<String> {
let st = self.state.lock(); // Searched across every mount: a node browsed through a sync folder is
let mut ino = *st.by_uid.get(uid)?; // interned in that fork's inode space, so the primary state alone had no
let mut parts = Vec::new(); // entry for it and the caller silently showed an empty path
while ino != ROOT_INO { // (`docs/BUGS.md` B74). The path is relative to whichever mount holds
let entry = st.entries.get(&ino)?; // it, which is the one the user reached it through.
parts.push(entry.node.name.clone()); let mut path = None;
ino = entry.parent; self.for_each_state(|st| {
if path.is_some() {
return;
}
path = walk_to_root(st, uid);
});
path
}
}
/// What [`Core::stop_sharing`] managed to remove.
#[derive(Debug, Default, PartialEq, Eq)]
pub(crate) struct StopSharingOutcome {
/// Entries removed (the link, invitations and members alike).
pub(crate) removed: usize,
/// What could not be removed, by email (or "the public link"), with why.
pub(crate) failed: Vec<(String, String)>,
}
impl StopSharingOutcome {
fn record<E: std::fmt::Display>(&mut self, what: &str, result: Result<(), E>) {
match result {
Ok(()) => self.removed += 1,
Err(e) => self.failed.push((what.to_string(), e.to_string())),
} }
parts.reverse(); }
Some(parts.join("/"))
/// The reply for the control socket: a sentence for a full success, an
/// error naming what is left otherwise.
pub(crate) fn into_result(self) -> Result<String, String> {
if self.failed.is_empty() {
return Ok(match self.removed {
0 => "it was not shared".to_string(),
1 => "stopped sharing: removed 1 entry".to_string(),
n => format!("stopped sharing: removed {n} entries"),
});
}
let left = self
.failed
.iter()
.map(|(what, why)| format!("{what} ({why})"))
.collect::<Vec<_>>()
.join(", ");
Err(format!(
"removed {} of {}; still shared: {left}",
self.removed,
self.removed + self.failed.len()
))
}
}
/// Walk one inode space from `uid` up to its root, building the path. `None` if
/// that mount has never interned the node, or if the chain breaks partway —
/// a broken chain is a missing path, never a partial one.
fn walk_to_root(st: &super::State, uid: &NodeUid) -> Option<String> {
let mut ino = *st.by_uid.get(uid)?;
let mut parts = Vec::new();
while ino != ROOT_INO {
let entry = st.entries.get(&ino)?;
parts.push(entry.node.name.clone());
ino = entry.parent;
}
parts.reverse();
Some(parts.join("/"))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn stopping_a_share_that_is_gone_says_so() {
let outcome = StopSharingOutcome::default();
assert_eq!(outcome.into_result().unwrap(), "it was not shared");
}
#[test]
fn a_failed_removal_names_what_is_still_shared() {
let mut outcome = StopSharingOutcome::default();
outcome.record::<String>("the public link", Ok(()));
outcome.record("bob@example.com", Err("forbidden"));
outcome.record::<String>("carol@example.com", Ok(()));
assert_eq!(
outcome.into_result().unwrap_err(),
"removed 2 of 3; still shared: bob@example.com (forbidden)"
);
}
#[test]
fn a_full_removal_counts_the_entries() {
let mut outcome = StopSharingOutcome::default();
outcome.record::<String>("the public link", Ok(()));
outcome.record::<String>("bob@example.com", Ok(()));
assert_eq!(
outcome.into_result().unwrap(),
"stopped sharing: removed 2 entries"
);
} }
} }
+96
View File
@@ -0,0 +1,96 @@
//! One shutdown signal shared by every long-lived worker in the daemon.
//!
//! The mount used to end by unmounting and returning, leaving the drain
//! workers, the sync engine, the conflict sweep, the online probe and the local
//! indexer running on `Core` clones. In a process that exits immediately that is
//! invisible; in an in-process remount — which is how the daemon recovers from a
//! mount going away underneath it — it leaks a whole generation of threads that
//! keep writing to the database and calling the API on behalf of a mount that no
//! longer exists (bugs.md B44).
//!
//! The primitive is deliberately small: a flag plus a condvar. Every loop that
//! would otherwise call `thread::sleep` calls [`Shutdown::sleep`] instead, which
//! returns early — and reports `false` — the moment [`Shutdown::stop`] is
//! called, so teardown does not have to wait out a five-minute sweep interval to
//! join a thread.
use std::time::Duration;
use parking_lot::{Condvar, Mutex};
/// The daemon's stop signal. Set once, never cleared: a `Core` that has been
/// told to stop belongs to a mount that is going away.
#[derive(Default)]
pub(crate) struct Shutdown {
stopping: Mutex<bool>,
changed: Condvar,
}
impl Shutdown {
/// Whether teardown has begun. Loops check this where they would otherwise
/// start another unit of work.
pub(crate) fn is_stopping(&self) -> bool {
*self.stopping.lock()
}
/// Begin teardown and wake everything waiting in [`sleep`](Self::sleep).
/// Idempotent, because more than one path can reach teardown (a signal, the
/// kernel mount ending, a failed startup).
pub(crate) fn stop(&self) {
let mut stopping = self.stopping.lock();
*stopping = true;
self.changed.notify_all();
}
/// Wait up to `duration`, or until teardown starts.
///
/// Returns `false` if the daemon is stopping — including when it was already
/// stopping on entry, so `while shutdown.sleep(interval)` is a complete and
/// correct loop condition and no separate check is needed at the top.
pub(crate) fn sleep(&self, duration: Duration) -> bool {
let mut stopping = self.stopping.lock();
if !*stopping {
self.changed.wait_for(&mut stopping, duration);
}
!*stopping
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
use std::time::Instant;
#[test]
fn sleeping_returns_early_once_stopped() {
let shutdown = Arc::new(Shutdown::default());
assert!(!shutdown.is_stopping());
let waker = shutdown.clone();
std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(20));
waker.stop();
});
let started = Instant::now();
// An hour's worth of interval, cut short by the stop.
assert!(!shutdown.sleep(Duration::from_secs(3600)));
assert!(
started.elapsed() < Duration::from_secs(5),
"stop must interrupt the wait, not wait it out"
);
assert!(shutdown.is_stopping());
// Already stopping: the next wait does not sleep at all.
let started = Instant::now();
assert!(!shutdown.sleep(Duration::from_secs(3600)));
assert!(started.elapsed() < Duration::from_secs(5));
}
#[test]
fn sleeping_runs_to_the_timeout_while_running() {
let shutdown = Shutdown::default();
assert!(shutdown.sleep(Duration::from_millis(10)));
assert!(!shutdown.is_stopping());
}
}
File diff suppressed because it is too large. Load diff
+229
View File
@@ -0,0 +1,229 @@
//! The thread that watches the daemon for the thing the daemon cannot report
//! about itself: that it has stopped making progress.
//!
//! Three jobs, all cheap, all on one timer:
//!
//! 1. **Say so in the journal.** A worker that has held one job for minutes, a
//! queue that is not draining, a control request that never returned — each
//! gets a WARN naming it. The 2026-09-18 hang left nothing in the journal at
//! all; this is what turns the next one into a one-line diagnosis.
//! 2. **Ping the systemd watchdog**, but only after a real round trip over the
//! control socket. Pinging unconditionally from a thread that does nothing
//! else would keep the unit "healthy" through exactly the hang it is meant to
//! catch: the probe is what makes the ping mean "a client can still talk to
//! this daemon".
//! 3. **Sample memory.** That hang peaked at 5.2 GiB resident, and nobody knew
//! until it was over.
use std::io::{BufRead, BufReader, Write as _};
use std::os::unix::net::UnixStream;
use std::path::{Path, PathBuf};
use std::time::{Duration, Instant};
use pdfs_core::control::Request as CtlRequest;
use tracing::{info, warn};
use super::Core;
use super::systemd;
/// How often the supervisor looks. Fast enough that a stall is named while the
/// user is still watching it, slow enough to cost nothing.
const TICK: Duration = Duration::from_secs(15);
/// How long one job may hold a worker, or one control request may run, before it
/// is called out. Generous: a large upload chunk or a cold folder listing on a
/// slow link is minutes of legitimate work, and a false WARN every few minutes
/// would train everyone to ignore the real one.
const STALL_AFTER: Duration = Duration::from_secs(120);
/// How often resident size goes into the log.
const RSS_EVERY: Duration = Duration::from_secs(300);
/// How long the liveness probe waits for the daemon to answer its own control
/// socket. Well under the watchdog period, so a probe that times out still
/// leaves time for the manager to hear nothing and act.
const PROBE_TIMEOUT: Duration = Duration::from_secs(10);
/// Run until shutdown. Spawned once per mount, from [`super::mount::mount`].
pub(crate) fn run(core: Core, control_socket: PathBuf) {
let watchdog = systemd::watchdog_period();
if let Some(period) = watchdog {
info!(period_secs = period.as_secs(), "systemd watchdog enabled");
}
let mut last_rss = Instant::now();
let mut last_ping = Instant::now();
// Per-lane completion counts from the previous tick, to tell a queue that is
// being worked through from one that is not moving at all.
let mut last_completed = (0_u64, 0_u64);
while core.shutdown.sleep(TICK) {
let report = core.diagnostics();
let stalled = report_stalls(&report, last_completed);
last_completed = (report.meta_completed, report.transfer_completed);
if last_rss.elapsed() >= RSS_EVERY {
info!(
rss_bytes = report.rss_bytes,
pending_ops = report.pending_ops,
meta_queued = report.meta_queued,
transfer_queued = report.transfer_queued,
stalled,
"daemon resource sample"
);
last_rss = Instant::now();
}
// A stalled lane is deliberately *not* a reason to withhold the ping:
// the mount may still be answering everything else, and a restart would
// throw away in-flight uploads. The probe decides that; a stall only
// gets said out loud.
if let Some(period) = watchdog
&& last_ping.elapsed() >= period
{
if probe(&control_socket) {
systemd::watchdog_ping();
} else {
warn!(
"daemon did not answer its own control socket; withholding the watchdog ping"
);
}
last_ping = Instant::now();
}
}
}
/// WARN about anything that has not moved for [`STALL_AFTER`]. Returns whether
/// anything was reported, so the caller can weigh it.
fn report_stalls(report: &pdfs_core::control::Diagnostics, last_completed: (u64, u64)) -> bool {
let threshold = STALL_AFTER.as_secs();
let mut stalled = false;
for worker in &report.workers {
if worker.busy && worker.age_secs >= threshold {
stalled = true;
warn!(
worker = %worker.name,
job = %worker.label,
age_secs = worker.age_secs,
"a fuse worker has held the same job for a long time"
);
}
}
if let Some(oldest) = report.inflight.first()
&& oldest.age_secs >= threshold
{
stalled = true;
warn!(
request = %oldest.kind,
age_secs = oldest.age_secs,
in_flight = report.inflight.len(),
"a control request has been running for a long time"
);
}
// A backed-up lane is normal; a backed-up lane that finished nothing since
// the last tick is not.
if report.meta_queued > 0 && report.meta_completed == last_completed.0 {
stalled = true;
warn!(
queued = report.meta_queued,
"the metadata lane has not completed a job since the last check"
);
}
if report.transfer_queued > 0 && report.transfer_completed == last_completed.1 {
stalled = true;
warn!(
queued = report.transfer_queued,
"the transfer lane has not completed a job since the last check"
);
}
stalled
}
/// One round trip over the control socket: connect, ask for the cheapest
/// request there is, read the answer.
///
/// This is the liveness the watchdog ping stands for. It covers the accept loop,
/// the handler pool and the reply path — everything a `pdfs` command or the GUI
/// depends on — without touching the network or the database.
fn probe(control_socket: &Path) -> bool {
let Ok(stream) = UnixStream::connect(control_socket) else {
return false;
};
if stream.set_read_timeout(Some(PROBE_TIMEOUT)).is_err()
|| stream.set_write_timeout(Some(PROBE_TIMEOUT)).is_err()
{
return false;
}
let Ok(request) = serde_json::to_string(&CtlRequest::Diagnostics) else {
return false;
};
let mut stream = stream;
if stream.write_all(format!("{request}\n").as_bytes()).is_err() {
return false;
}
let mut line = String::new();
BufReader::new(stream).read_line(&mut line).is_ok() && !line.trim().is_empty()
}
#[cfg(test)]
mod tests {
use super::*;
use pdfs_core::control::{Diagnostics, InflightRequest, WorkerState};
fn busy_worker(age_secs: u64) -> WorkerState {
WorkerState {
name: "pdfs-fuse-3".into(),
busy: true,
label: "read".into(),
age_secs,
}
}
/// A worker doing normal work must not be reported: a WARN every tick is a
/// WARN nobody reads.
#[test]
fn ordinary_work_is_not_a_stall() {
let report = Diagnostics {
workers: vec![busy_worker(5)],
..Default::default()
};
assert!(!report_stalls(&report, (0, 0)));
}
/// A worker on the same job past the threshold is the shape of the hang.
#[test]
fn a_long_held_job_is_a_stall() {
let report = Diagnostics {
workers: vec![busy_worker(STALL_AFTER.as_secs() + 1)],
..Default::default()
};
assert!(report_stalls(&report, (0, 0)));
}
/// So is a control request that never returned.
#[test]
fn a_long_running_request_is_a_stall() {
let report = Diagnostics {
inflight: vec![InflightRequest {
kind: "OpenFile".into(),
age_secs: STALL_AFTER.as_secs() + 1,
}],
..Default::default()
};
assert!(report_stalls(&report, (0, 0)));
}
/// A queue that is draining is busy, not stalled — that is what comparing
/// the completion counts is for.
#[test]
fn a_draining_queue_is_not_a_stall() {
let report = Diagnostics {
transfer_queued: 12,
transfer_completed: 40,
..Default::default()
};
assert!(!report_stalls(&report, (0, 30)));
assert!(report_stalls(&report, (0, 40)));
}
}
+503
View File
@@ -0,0 +1,503 @@
//! Background conflict sweep.
//!
//! A `(sync-conflict <ts>)` copy is only ever a fallback: the drain and the sync
//! engine cut one when they cannot prove a queued write is safe to apply, so the
//! user never loses bytes. Most of those copies turn out to be byte-identical to
//! the live file — a spurious conflict from a re-stamped revision (B16/B25) or a
//! first-sync of an already-identical file. This sweep reconciles them after the
//! fact:
//!
//! - **Identical** to the live sibling (same size *and* same content SHA-1): the
//! copy is redundant, so it is trashed and logged as a [`ActivityKind::Trash`].
//! - **Divergent** (different bytes) or **orphaned** (no live sibling): left in
//! place and surfaced once as an [`ActivityKind::Conflict`], because only the
//! user can decide which version wins.
//!
//! Identity is proved without downloading anything: the SHA-1 comes from the
//! revision's decrypted extended attributes, which enumeration already reads. A
//! copy whose sibling has no recorded SHA-1 is treated as divergent — the sweep
//! never removes a file it cannot prove is a duplicate.
use std::collections::HashMap;
use std::time::Duration;
use pdfs_core::batch;
use pdfs_core::config::SweepMode;
use pdfs_core::control::ActivityKind;
use proton_drive_rs::proton_sdk::ids::NodeUid;
use tracing::{debug, info, warn};
use super::{Core, node_content_sha1, node_revision_id, node_size};
/// Time between sweeps. Conflicts are rare and non-urgent; a slow cadence keeps
/// the sweep off the hot path while still cleaning up within a few minutes.
const CONFLICT_SWEEP_INTERVAL: Duration = Duration::from_secs(300);
/// Delay before the first sweep, so it does not compete with the burst of work a
/// fresh mount does at startup (hydrate, recover, initial enumeration).
const CONFLICT_SWEEP_WARMUP: Duration = Duration::from_secs(30);
/// Strip a ` (sync-conflict <stamp>)` (optionally `-<n>`) marker from `name`,
/// returning the original file name it was a copy of. `None` when `name` is not a
/// conflict copy. Inverse of the `conflict_name` / `conflict_path_with_suffix`
/// formats, so the two halves of the product agree on what a conflict is.
pub(crate) fn conflict_base_name(name: &str) -> Option<String> {
const MARKER: &str = " (sync-conflict ";
let start = name.find(MARKER)?;
let after = start + MARKER.len();
let close_rel = name[after..].find(')')?;
let close = after + close_rel;
let inner = &name[after..close];
// The token is `<digits>` or `<digits>-<digits>`; reject anything else so a
// file that merely happens to contain the phrase is not mistaken for a copy.
let mut parts = inner.splitn(2, '-');
let stamp = parts.next().unwrap_or_default();
let suffix = parts.next();
let is_digits = |s: &str| !s.is_empty() && s.bytes().all(|b| b.is_ascii_digit());
if !is_digits(stamp) || suffix.is_some_and(|s| !is_digits(s)) {
return None;
}
let mut base = String::with_capacity(start + name.len() - close);
base.push_str(&name[..start]);
base.push_str(&name[close + 1..]);
Some(base)
}
impl Core {
/// Sweep for stale conflict copies forever, on its own thread. See the module
/// docs for the identical/divergent policy.
pub(crate) fn run_conflict_sweep_loop(&self) {
// Both waits are interruptible, so teardown does not have to sit out a
// five-minute interval to join this thread (bugs.md B44).
if !self.shutdown.sleep(CONFLICT_SWEEP_WARMUP) {
return;
}
loop {
self.sweep_conflicts_once();
if !self.shutdown.sleep(CONFLICT_SWEEP_INTERVAL) {
return;
}
}
}
/// One sweep pass over every node the daemon currently knows.
fn sweep_conflicts_once(&self) {
let stored = match self.db.load_all() {
Ok(s) => s,
Err(e) => {
warn!(error = ?e, "conflict sweep: load nodes failed");
return;
}
};
let nodes: Vec<_> = stored
.into_iter()
.map(|stored| stored.node)
.filter(|node| self.is_own_or_virtual(&node.uid))
.collect();
// Index live (non-trashed) files by (parent, name) for sibling lookup.
let mut by_parent_name: HashMap<(String, &str), &_> = HashMap::new();
for node in &nodes {
if node.trashed || node.is_folder() {
continue;
}
let parent = node
.parent_uid
.as_ref()
.map(|u| u.to_string())
.unwrap_or_default();
by_parent_name.insert((parent, node.name.as_str()), node);
}
let roots = self.search_roots();
let online = self.online.load(std::sync::atomic::Ordering::Relaxed);
for node in &nodes {
if node.trashed || node.is_folder() {
continue;
}
let Some(base) = conflict_base_name(&node.name) else {
continue;
};
let parent = node
.parent_uid
.as_ref()
.map(|u| u.to_string())
.unwrap_or_default();
let sibling = by_parent_name.get(&(parent, base.as_str())).copied();
match decide(node, sibling, &base, self.sweep_mode) {
SweepAction::Remove => {
// Only while online — the remote trash is the point. Offline,
// it waits for a later pass.
if online {
self.remove_conflict_copy(node, &base);
}
}
// Only flag what the conflict listing can show. Anything else
// would sit in the feed as "resolved" with no way to act on
// it (`docs/BUGS.md` B99).
SweepAction::Flag(detail) => {
if self.conflict_path(&roots, &node.uid).is_some() {
self.flag_conflict(&node.uid, &node.name, detail);
} else {
debug!(uid = %node.uid, name = %node.name, "conflict sweep: copy is outside every location; not flagged");
}
}
}
}
}
/// Trash a conflict copy proven identical to its live sibling, and unhook it
/// locally — the same sequence the interactive unlink path uses after a trash.
///
/// `snapshot` is the node as the pass saw it, which by now is stale: the pass
/// enumerated every node up front and has been doing network I/O since. Before
/// touching anything remote this re-reads the node and refuses to act unless
/// it is *still* the same revision, still untrashed, still unwritten and still
/// unopened (`docs/BUGS.md` B71). Nothing on this path discards a queued write
/// or its staged blob: a write that arrives after that check is left for the
/// drain to resolve against the now-trashed node.
fn remove_conflict_copy(&self, snapshot: &proton_drive_rs::Node, base: &str) {
let uid = &snapshot.uid;
let name = snapshot.name.as_str();
if !self.duplicate_still_removable(snapshot) {
return;
}
match self
.rt
.block_on(self.client.trash_nodes(std::slice::from_ref(uid)))
.and_then(batch::into_unit)
{
Ok(()) => {}
Err(e) if super::is_gone(&e) => {}
Err(e) => {
warn!(%uid, name, error = %e, "conflict sweep: trashing duplicate failed");
return;
}
}
// Every mount, not just the primary: a conflict copy inside a sync
// folder lives in that fork's inode space, and leaving it hooked up
// there keeps a node we have just trashed remotely visible and
// readable there (`docs/BUGS.md` B74).
self.for_each_state(|st| {
st.forget_or_unlink(uid);
});
// Deliberately *not* `discard_queued_ops`: `duplicate_still_removable`
// proved the queue empty before the trash above, but that proof is stale
// the moment `trash_nodes` goes to the network. A `close(2)` landing in
// that window stages bytes, and discarding here would unlink the staged
// blob holding the only copy of them. Leaving the op queued is the
// non-destructive outcome: the drain re-reads the node, sees it trashed,
// and `revision_conflict` routes it to `keep_as_conflict_copy`.
self.cache.evict(uid);
self.evict_reader(uid);
if let Err(e) = self.db.delete_node(uid) {
warn!(%uid, error = ?e, "conflict sweep: db delete after trash failed");
}
self.invalidate_trash();
self.conflict_notified.lock().remove(uid);
self.log_activity(
ActivityKind::Trash,
name,
format!("auto-removed duplicate of {base}"),
true,
);
info!(%uid, name, base, "conflict sweep: removed identical conflict copy");
}
/// Re-check, immediately before the destructive call, everything the pass
/// decided from its stale snapshot. Any doubt means "leave it alone": a
/// conflict copy that survives to the next pass costs nothing, a wrongly
/// removed one costs the user data.
fn duplicate_still_removable(&self, snapshot: &proton_drive_rs::Node) -> bool {
let uid = &snapshot.uid;
let name = snapshot.name.as_str();
// A queued op means bytes are still owed an upload; trashing the node
// under them turns an ordinary upload into a conflict fork.
match self.db.has_any_op(&uid.to_string()) {
Ok(false) => {}
Ok(true) => {
debug!(%uid, name, "conflict sweep: skipping duplicate with queued ops");
return false;
}
// Cannot prove the queue is empty — assume it is not.
Err(e) => {
warn!(%uid, name, error = ?e, "conflict sweep: op check failed; skipping");
return false;
}
}
// Someone is writing to it, or holds it open. Trashing now would pull the
// node out from under a live handle.
if self.is_busy(uid) {
debug!(%uid, name, "conflict sweep: skipping duplicate that is open or being written");
return false;
}
// Finally, the node itself must not have moved on since the snapshot.
match self.db.node_by_uid(&uid.to_string()) {
Ok(Some(fresh)) => {
if fresh.trashed {
return false;
}
if fresh.name != snapshot.name || fresh.parent_uid != snapshot.parent_uid {
debug!(%uid, name, "conflict sweep: duplicate was renamed or moved; skipping");
return false;
}
if node_revision_id(&fresh) != node_revision_id(snapshot)
|| node_size(&fresh) != node_size(snapshot)
|| node_content_sha1(&fresh) != node_content_sha1(snapshot)
{
debug!(%uid, name, "conflict sweep: duplicate changed since the pass began");
return false;
}
true
}
// Already gone locally, or unreadable — either way, not ours to remove.
Ok(None) => false,
Err(e) => {
warn!(%uid, name, error = ?e, "conflict sweep: re-read failed; skipping");
false
}
}
}
/// Whether a node has a write in flight or a file handle open against it.
/// Read-only opens use fh 0 and are not tracked, so this is specifically the
/// writer/holder check.
/// Asked across **every** mount. The sweep runs on the primary Core but
/// trashes nodes anywhere in the tree, and a file open inside a sync folder
/// is tracked in that fork's `State` — so consulting `self.state` alone
/// answered "idle" for every such file and let the sweep delete one with a
/// writer attached (`docs/BUGS.md` B74). Busy anywhere means busy.
pub(crate) fn is_busy(&self, uid: &NodeUid) -> bool {
let mut busy = false;
self.for_each_state(|st| {
let Some(&ino) = st.by_uid.get(uid) else {
return;
};
busy |=
st.active_writes.contains_key(&ino) || st.handles.values().any(|&held| held == ino);
});
busy
}
/// Surface a conflict copy that needs the user's attention, once. The
/// in-memory set spares the database each pass; the activity log itself is
/// what keeps a restart from logging the same copy again (`docs/BUGS.md`
/// B99). A copy's name carries its own timestamp, so the name identifies it.
fn flag_conflict(&self, uid: &NodeUid, name: &str, detail: String) {
if !self.conflict_notified.lock().insert(uid.clone()) {
return;
}
match self.db.activity_has(ActivityKind::Conflict, name) {
Ok(false) => {}
Ok(true) => return,
// Cannot tell: a repeat entry is harmless, a missing one hides a
// conflict.
Err(e) => warn!(%uid, name, error = ?e, "conflict sweep: activity check failed"),
}
debug!(%uid, name, detail, "conflict sweep: flagged divergent conflict copy");
self.log_activity(ActivityKind::Conflict, name, detail, false);
}
}
/// What the sweep concluded about one `(sync-conflict …)` copy.
#[derive(Debug, Clone, PartialEq, Eq)]
enum SweepAction {
/// Provably redundant: trash it, subject to the runtime interlocks in
/// [`Core::duplicate_still_removable`].
Remove,
/// Only the user can resolve this one; surface it with this detail.
Flag(String),
}
/// Decide what to do about conflict copy `node` given its live sibling (if any)
/// and the configured `mode`.
///
/// Pure, so the policy that governs deletion is testable without a mount, a
/// runtime, or an account — the same split used for `revision_changed` and
/// `is_own_self_supersede`. Everything that can fail or race lives in the caller.
fn decide(
node: &proton_drive_rs::Node,
sibling: Option<&proton_drive_rs::Node>,
base: &str,
mode: SweepMode,
) -> SweepAction {
match sibling {
Some(sib) if is_identical(node, sib) => {
if mode.is_enforcing() {
SweepAction::Remove
} else {
// Report-only: say what would happen, remove nothing. The point is
// to accumulate field evidence that the identity check is right
// before it is trusted with deletion (docs/BUGS.md B71).
SweepAction::Flag(format!(
"identical to {base}; would be removed (sweep is report-only)"
))
}
}
Some(_) => SweepAction::Flag(format!("differs from {base}")),
None => SweepAction::Flag(format!("no original {base} to reconcile against")),
}
}
/// Whether two file nodes provably hold the same bytes: equal size and equal,
/// present content SHA-1. A missing digest on either side is not proof, so the
/// sweep treats it as divergent rather than risk removing a real difference.
fn is_identical(a: &proton_drive_rs::Node, b: &proton_drive_rs::Node) -> bool {
if node_size(a) != node_size(b) {
return false;
}
match (node_content_sha1(a), node_content_sha1(b)) {
(Some(x), Some(y)) => x == y,
_ => false,
}
}
#[cfg(test)]
mod tests {
use super::*;
use proton_drive_rs::NodeKind;
use proton_drive_rs::proton_sdk::ids::{LinkId, VolumeId};
/// A file node with the two fields the sweep's identity check reads.
fn file(name: &str, size: i64, sha1: Option<&str>) -> proton_drive_rs::Node {
proton_drive_rs::Node {
uid: NodeUid::new(VolumeId::from("v"), LinkId::from(name)),
parent_uid: None,
kind: NodeKind::File {
media_type: "text/plain".into(),
total_size_on_storage: size,
active_revision_state: None,
active_revision_id: Some("rev".into()),
claimed_size: Some(size),
claimed_modification_time: None,
content_sha1: sha1.map(String::from),
},
name: name.into(),
creation_time: 0,
modification_time: 0,
trashed: false,
is_shared: false,
is_shared_publicly: false,
signature_email: None,
membership: None,
photo: None,
album: None,
verification: Default::default(),
direct_role: None,
share_id: None,
}
}
/// The only input shape that may ever produce a removal: same size, same
/// present digest, live sibling, enforcing mode.
#[test]
fn removes_only_a_provably_identical_copy_when_enforcing() {
let copy = file("f (sync-conflict 1).txt", 100, Some("aaa"));
let live = file("f.txt", 100, Some("aaa"));
assert_eq!(
decide(&copy, Some(&live), "f.txt", SweepMode::Enforce),
SweepAction::Remove
);
}
#[test]
fn report_only_never_removes_but_still_says_what_it_would_do() {
let copy = file("f (sync-conflict 1).txt", 100, Some("aaa"));
let live = file("f.txt", 100, Some("aaa"));
// Same inputs as the removal case above; only the mode differs.
let SweepAction::Flag(detail) = decide(&copy, Some(&live), "f.txt", SweepMode::Report)
else {
panic!("report mode must never remove");
};
assert!(detail.contains("would be removed"), "{detail}");
assert!(detail.contains("report-only"), "{detail}");
}
#[test]
fn a_differing_size_is_a_real_conflict() {
let copy = file("f (sync-conflict 1).txt", 101, Some("aaa"));
let live = file("f.txt", 100, Some("aaa"));
assert_eq!(
decide(&copy, Some(&live), "f.txt", SweepMode::Enforce),
SweepAction::Flag("differs from f.txt".into())
);
}
#[test]
fn a_missing_digest_is_never_proof_of_identity() {
// Equal sizes are not enough: two different 100-byte files are common.
// A copy the sweep cannot fingerprint must survive, in either direction.
for (a, b) in [(None, Some("aaa")), (Some("aaa"), None), (None, None)] {
let copy = file("f (sync-conflict 1).txt", 100, a);
let live = file("f.txt", 100, b);
assert_eq!(
decide(&copy, Some(&live), "f.txt", SweepMode::Enforce),
SweepAction::Flag("differs from f.txt".into()),
"{a:?} vs {b:?}"
);
}
}
#[test]
fn a_differing_digest_at_equal_size_is_a_real_conflict() {
let copy = file("f (sync-conflict 1).txt", 100, Some("aaa"));
let live = file("f.txt", 100, Some("bbb"));
assert_eq!(
decide(&copy, Some(&live), "f.txt", SweepMode::Enforce),
SweepAction::Flag("differs from f.txt".into())
);
}
#[test]
fn an_orphaned_copy_is_surfaced_not_removed() {
// No live original left to reconcile against — the copy may be the only
// surviving version of the file, so it is never the sweep's to remove.
let copy = file("f (sync-conflict 1).txt", 100, Some("aaa"));
assert_eq!(
decide(&copy, None, "f.txt", SweepMode::Enforce),
SweepAction::Flag("no original f.txt to reconcile against".into())
);
}
#[test]
fn identity_needs_both_size_and_digest_to_match() {
let a = file("a", 100, Some("aaa"));
assert!(is_identical(&a, &file("b", 100, Some("aaa"))));
assert!(!is_identical(&a, &file("b", 101, Some("aaa"))));
assert!(!is_identical(&a, &file("b", 100, Some("bbb"))));
assert!(!is_identical(&a, &file("b", 100, None)));
}
#[test]
fn parses_conflict_names_and_leaves_others_alone() {
assert_eq!(
conflict_base_name("test_export (sync-conflict 1784898786).xml").as_deref(),
Some("test_export.xml")
);
assert_eq!(
conflict_base_name("README (sync-conflict 42)").as_deref(),
Some("README")
);
// The sync engine's de-duplicated suffix form.
assert_eq!(
conflict_base_name("notes (sync-conflict 1700-3).txt").as_deref(),
Some("notes.txt")
);
// Not a conflict copy.
assert_eq!(conflict_base_name("report.xml"), None);
// Looks close but the token is not a stamp.
assert_eq!(conflict_base_name("x (sync-conflict abc).txt"), None);
}
#[test]
fn a_dotted_stem_keeps_its_inner_dots() {
assert_eq!(
conflict_base_name("archive.tar (sync-conflict 7).gz").as_deref(),
Some("archive.tar.gz")
);
}
}
+632 -81
View File
@@ -38,6 +38,7 @@ use tracing::{debug, info, warn};
use crate::transfers::{CountingWriter, OwnedCountingReader}; use crate::transfers::{CountingWriter, OwnedCountingReader};
use crate::{Core, now_secs, parse_uid}; use crate::{Core, now_secs, parse_uid};
use pdfs_core::batch;
use pdfs_core::control::{ActivityKind, TransferDirection}; use pdfs_core::control::{ActivityKind, TransferDirection};
use pdfs_core::db::{StoredSyncEntry, StoredSyncFolder}; use pdfs_core::db::{StoredSyncEntry, StoredSyncFolder};
use pdfs_core::syncignore::IgnoreRules; use pdfs_core::syncignore::IgnoreRules;
@@ -53,7 +54,7 @@ use engine::{classify, settle_with};
#[cfg(test)] #[cfg(test)]
use local_scan::is_write_mode; use local_scan::is_write_mode;
use local_scan::open_for_write_set; use local_scan::open_for_write_set;
use model::{LocalItem, Outcome, PassAbort, Pending, RemoteItem}; use model::{LocalItem, LocalSig, Outcome, PassAbort, Pending, RemoteItem};
pub(crate) use planner::base_name; pub(crate) use planner::base_name;
#[cfg(test)] #[cfg(test)]
use planner::conflict_path; use planner::conflict_path;
@@ -81,6 +82,12 @@ impl Core {
/// is guaranteed — a switch requested over the control socket cannot wait for a /// is guaranteed — a switch requested over the control socket cannot wait for a
/// pass that may run for minutes. /// pass that may run for minutes.
pub(crate) fn reconcile_folder(&self, folder: &StoredSyncFolder) { pub(crate) fn reconcile_folder(&self, folder: &StoredSyncFolder) {
// A paused folder skips the settle too: a queued switch to on-demand
// evicts the local tree, which is only safe after a pass uploaded it.
if self.sync_folder_paused(folder.id) {
debug!(id = folder.id, "sync: folder paused, skipping reconcile");
return;
}
// `ondemand` folders are live FUSE mounts, not mirrored, so there is nothing // `ondemand` folders are live FUSE mounts, not mirrored, so there is nothing
// to reconcile — but one may still carry a queued switch back to `mirror`, // to reconcile — but one may still carry a queued switch back to `mirror`,
// so the settle below is not skipped with it. // so the settle below is not skipped with it.
@@ -513,13 +520,20 @@ impl Core {
let _ = self.db.sync_entry_remove(folder_id, rel); let _ = self.db.sync_entry_remove(folder_id, rel);
continue; continue;
}; };
if let Err(e) = self.rt.block_on(self.client.trash_nodes(&[uid])) { if let Err(e) = self
.rt
.block_on(self.client.trash_nodes(&[uid]))
.and_then(batch::into_unit)
{
warn!(rel, error = %e, "sync: trash remote failed"); warn!(rel, error = %e, "sync: trash remote failed");
self.log_activity(ActivityKind::Trash, base_name(rel), e.to_string(), false); self.log_activity(ActivityKind::Trash, base_name(rel), e.to_string(), false);
outcome.errors += 1; outcome.errors += 1;
continue; continue;
} }
trashed.push(rel.clone()); trashed.push(rel.clone());
if let Some(parent) = parent_uid_of(rel, &remote_dirs, &baseline) {
self.invalidate_children_of(&parent);
}
let _ = self.db.sync_entry_remove(folder_id, rel); let _ = self.db.sync_entry_remove(folder_id, rel);
self.log_activity( self.log_activity(
ActivityKind::Trash, ActivityKind::Trash,
@@ -670,6 +684,28 @@ impl Core {
match plan_file(l, r, baseline.get(rel)) { match plan_file(l, r, baseline.get(rel)) {
FilePlan::Unchanged => {} FilePlan::Unchanged => {}
FilePlan::AdoptBaseline => {
// First sync of an already-identical file: record the baseline
// in place so future divergence is detected, without any
// transfer or a spurious conflict copy.
let local = l.expect("adopt baseline requires a local file");
let remote = r.expect("adopt baseline requires a remote file");
if let Err(e) = self.db.sync_entry_upsert(
folder_id,
&StoredSyncEntry {
rel_path: rel.clone(),
remote_uid: Some(remote.uid.to_string()),
local_mtime: local.mtime,
local_mtime_ns: local.mtime_ns,
local_size: local.size,
remote_rev: Some(remote.mtime.to_string()),
remote_hash: Some(remote.size.to_string()),
},
) {
warn!(rel, error = ?e, "sync: adopt baseline failed");
outcome.errors += 1;
}
}
FilePlan::Deferred => { FilePlan::Deferred => {
debug!( debug!(
rel, rel,
@@ -691,6 +727,9 @@ impl Core {
uid: remote.uid.clone(), uid: remote.uid.clone(),
mtime: remote.mtime, mtime: remote.mtime,
size: remote.size, size: remote.size,
// What the walk saw here, so the apply step can tell a
// stale plan from a live one before it overwrites.
local: l.map(LocalSig::from),
}); });
} }
FilePlan::Conflict => { FilePlan::Conflict => {
@@ -713,7 +752,9 @@ impl Core {
} }
}, },
FilePlan::DeleteLocal => { FilePlan::DeleteLocal => {
if let Err(e) = std::fs::remove_file(local_root.join(rel_to_path(rel))) { if let Err(e) =
removed_locally(std::fs::remove_file(local_root.join(rel_to_path(rel))))
{
warn!(rel, error = %e, "sync: remove local file failed"); warn!(rel, error = %e, "sync: remove local file failed");
self.log_activity( self.log_activity(
ActivityKind::Trash, ActivityKind::Trash,
@@ -733,6 +774,7 @@ impl Core {
if let Err(e) = self if let Err(e) = self
.rt .rt
.block_on(self.client.trash_nodes(std::slice::from_ref(&remote.uid))) .block_on(self.client.trash_nodes(std::slice::from_ref(&remote.uid)))
.and_then(batch::into_unit)
{ {
warn!(rel, error = %e, "sync: trash remote failed"); warn!(rel, error = %e, "sync: trash remote failed");
self.log_activity( self.log_activity(
@@ -744,6 +786,9 @@ impl Core {
outcome.errors += 1; outcome.errors += 1;
continue; continue;
} }
if let Some(parent) = parent_uid_of(rel, &remote_dirs, &baseline) {
self.invalidate_children_of(&parent);
}
let _ = self.db.sync_entry_remove(folder_id, rel); let _ = self.db.sync_entry_remove(folder_id, rel);
self.log_activity( self.log_activity(
ActivityKind::Trash, ActivityKind::Trash,
@@ -770,7 +815,9 @@ impl Core {
// Deferred folder deletions, deepest first. // Deferred folder deletions, deepest first.
delete_local_dirs.sort_by_key(|p| std::cmp::Reverse(p.matches('/').count())); delete_local_dirs.sort_by_key(|p| std::cmp::Reverse(p.matches('/').count()));
for rel in delete_local_dirs { for rel in delete_local_dirs {
if let Err(e) = std::fs::remove_dir_all(local_root.join(rel_to_path(&rel))) { if let Err(e) =
removed_locally(std::fs::remove_dir_all(local_root.join(rel_to_path(&rel))))
{
warn!(rel, error = %e, "sync: remove local folder failed"); warn!(rel, error = %e, "sync: remove local folder failed");
self.log_activity(ActivityKind::Trash, base_name(&rel), e.to_string(), false); self.log_activity(ActivityKind::Trash, base_name(&rel), e.to_string(), false);
outcome.errors += 1; outcome.errors += 1;
@@ -787,12 +834,19 @@ impl Core {
} }
delete_remote_dirs.sort_by_key(|(p, _)| std::cmp::Reverse(p.matches('/').count())); delete_remote_dirs.sort_by_key(|(p, _)| std::cmp::Reverse(p.matches('/').count()));
for (rel, uid) in delete_remote_dirs { for (rel, uid) in delete_remote_dirs {
if let Err(e) = self.rt.block_on(self.client.trash_nodes(&[uid])) { if let Err(e) = self
.rt
.block_on(self.client.trash_nodes(&[uid]))
.and_then(batch::into_unit)
{
warn!(rel, error = %e, "sync: trash remote folder failed"); warn!(rel, error = %e, "sync: trash remote folder failed");
self.log_activity(ActivityKind::Trash, base_name(&rel), e.to_string(), false); self.log_activity(ActivityKind::Trash, base_name(&rel), e.to_string(), false);
outcome.errors += 1; outcome.errors += 1;
continue; continue;
} }
if let Some(parent) = parent_uid_of(&rel, &remote_dirs, &baseline) {
self.invalidate_children_of(&parent);
}
let _ = self.db.sync_entry_remove(folder_id, &rel); let _ = self.db.sync_entry_remove(folder_id, &rel);
self.log_activity( self.log_activity(
ActivityKind::Trash, ActivityKind::Trash,
@@ -818,68 +872,9 @@ impl Core {
writing: &HashSet<PathBuf>, writing: &HashSet<PathBuf>,
out: &mut HashMap<String, LocalItem>, out: &mut HashMap<String, LocalItem>,
) -> Result<(), String> { ) -> Result<(), String> {
let entries = std::fs::read_dir(dir).map_err(|e| format!("read {}: {e}", dir.display()))?; walk_local_tree(root, dir, rules, writing, out, &mut |rel| {
for entry in entries { self.progress_scanned(folder_id, base_name(rel))
let entry = entry.map_err(|e| format!("read entry in {}: {e}", dir.display()))?; })
let path = entry.path();
let meta = std::fs::symlink_metadata(&path)
.map_err(|e| format!("stat {}: {e}", path.display()))?;
let stripped = path.strip_prefix(root).map_err(|e| {
format!(
"local path {} escaped sync root {}: {e}",
path.display(),
root.display()
)
})?;
let rel = stripped.to_str().ok_or_else(|| {
format!(
"local path {} is not valid UTF-8; refusing a destructive sync pass",
path.display()
)
})?;
// Ignore our own in-flight download temp files.
if rel.contains(".pdfs-tmp-") {
continue;
}
// Ignored paths are dropped here as well as from the classification
// union — not for correctness (the union filter is what makes this
// safe) but so an ignored `node_modules/` is never descended into or
// counted against scan progress.
if rules.is_ignored(rel, meta.is_dir()) {
continue;
}
if meta.is_dir() {
out.insert(
rel.to_string(),
LocalItem {
is_dir: true,
mtime: 0,
size: 0,
open_for_write: false,
},
);
self.progress_scanned(folder_id, base_name(rel));
self.walk_local(folder_id, root, &path, rules, writing, out)?;
} else if meta.is_file() {
// Check whether any process holds this file open for writing.
// The canonical path is used because /proc/*/fd links resolve
// to canonical targets.
let ofw = std::fs::canonicalize(&path)
.map(|canon| writing.contains(&canon))
.unwrap_or(false);
out.insert(
rel.to_string(),
LocalItem {
is_dir: false,
mtime: system_mtime(&meta),
size: meta.len() as i64,
open_for_write: ofw,
},
);
self.progress_scanned(folder_id, base_name(rel));
}
}
Ok(())
} }
/// Map every remote folder under `folder` to its uid by rel path, ignoring /// Map every remote folder under `folder` to its uid by rel path, ignoring
@@ -1038,6 +1033,7 @@ impl Core {
rel_path: rel.to_string(), rel_path: rel.to_string(),
remote_uid: Some(uid.to_string()), remote_uid: Some(uid.to_string()),
local_mtime: 0, local_mtime: 0,
local_mtime_ns: None,
local_size: 0, local_size: 0,
remote_rev: None, remote_rev: None,
remote_hash: None, remote_hash: None,
@@ -1144,6 +1140,9 @@ impl Core {
.create_folder(parent, base_name(rel), Some(now_secs())) .create_folder(parent, base_name(rel), Some(now_secs()))
.await .await
.map_err(|e| format!("create remote folder {rel}: {e}"))?; .map_err(|e| format!("create remote folder {rel}: {e}"))?;
// The parent has gained a child that no cached listing of it
// knows about (B86).
self.invalidate_children_of(parent);
self.baseline_dir(folder_id, rel, &uid)?; self.baseline_dir(folder_id, rel, &uid)?;
Ok(Applied::Dir(rel.clone(), uid)) Ok(Applied::Dir(rel.clone(), uid))
} }
@@ -1161,8 +1160,9 @@ impl Core {
uid, uid,
mtime, mtime,
size, size,
local,
} => { } => {
self.download_file(folder_id, local_root, rel, uid, *mtime, *size) self.download_file(folder_id, local_root, rel, uid, *mtime, *size, *local)
.await?; .await?;
Ok(Applied::Downloaded) Ok(Applied::Downloaded)
} }
@@ -1180,7 +1180,11 @@ impl Core {
.map_err(|e| format!("set aside conflict copy for {rel}: {e}"))?; .map_err(|e| format!("set aside conflict copy for {rel}: {e}"))?;
info!(rel, "sync: kept local changes as a conflict copy"); info!(rel, "sync: kept local changes as a conflict copy");
} }
self.download_file(folder_id, local_root, rel, uid, *mtime, *size) // The local copy has just been moved aside, so the state this
// download expects to find at the path is "nothing" — anything
// that appears while it runs is a *new* racing write, and gets
// its own conflict copy rather than being overwritten (B39).
self.download_file(folder_id, local_root, rel, uid, *mtime, *size, None)
.await?; .await?;
Ok(Applied::Conflict) Ok(Applied::Conflict)
} }
@@ -1219,6 +1223,12 @@ impl Core {
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?; let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?; let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
let mtime = system_mtime(&meta); let mtime = system_mtime(&meta);
// Taken from the descriptor being streamed, so it names the content that
// actually goes up rather than whatever is at the path when we finish.
let streamed = LocalSig::from(&meta);
let thumbnails = self
.upload_thumbnails_async(path.clone(), name.to_string())
.await;
// Count the bytes as they are read, so this upload shows up in // Count the bytes as they are read, so this upload shows up in
// `GetQueueStatus` next to a manual one. The uid isn't known until the // `GetQueueStatus` next to a manual one. The uid isn't known until the
// draft is sealed, so the transfer registers without one. // draft is sealed, so the transfer registers without one.
@@ -1239,13 +1249,19 @@ impl Core {
crate::media_type_for(name), crate::media_type_for(name),
reader, reader,
meta.len() as i64, meta.len() as i64,
Vec::new(), thumbnails,
Some(mtime), Some(mtime),
false, false,
) )
.await .await
.map_err(|e| format!("upload {rel}: {e}"))?; .map_err(|e| format!("upload {rel}: {e}"))?;
self.record_file_baseline(folder_id, rel, &path, &uid).await // The parent has gained a child that no cached listing of it knows
// about. Without this, a folder later switched to on-demand serves a
// snapshot that predates this upload (B86).
self.invalidate_children_of(parent);
warn_if_torn(rel, &path, streamed);
self.record_file_baseline(folder_id, rel, Some(streamed), &uid)
.await
} }
/// Upload a changed local file as a new revision of an existing remote node. /// Upload a changed local file as a new revision of an existing remote node.
@@ -1260,6 +1276,10 @@ impl Core {
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?; let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?; let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
let mtime = system_mtime(&meta); let mtime = system_mtime(&meta);
let streamed = LocalSig::from(&meta);
let thumbnails = self
.upload_thumbnails_async(path.clone(), base_name(rel).to_string())
.await;
let reader = OwnedCountingReader::new( let reader = OwnedCountingReader::new(
file, file,
self.transfers.begin( self.transfers.begin(
@@ -1270,14 +1290,23 @@ impl Core {
), ),
); );
self.client self.client
.upload_new_revision_from(uid, reader, meta.len() as i64, Vec::new(), Some(mtime)) .upload_new_revision_from(uid, reader, meta.len() as i64, thumbnails, Some(mtime))
.await .await
.map_err(|e| format!("upload revision {rel}: {e}"))?; .map_err(|e| format!("upload revision {rel}: {e}"))?;
self.record_file_baseline(folder_id, rel, &path, uid).await warn_if_torn(rel, &path, streamed);
self.record_file_baseline(folder_id, rel, Some(streamed), uid)
.await
} }
/// Download a remote file to its local path (atomically via a temp file), /// Download a remote file to its local path (atomically via a temp file),
/// stamp the local mtime to match the remote, then record baseline. /// stamp the local mtime to match the remote, then record baseline.
///
/// `expected_local` is the identity planning saw at the destination. The
/// rename below is the destructive step, and the download in between can run
/// for minutes, so the destination is re-examined right before it: a local
/// file that no longer matches the plan is set aside as a conflict copy
/// instead of being replaced (bugs.md B39).
#[allow(clippy::too_many_arguments)]
async fn download_file( async fn download_file(
&self, &self,
folder_id: i64, folder_id: i64,
@@ -1286,6 +1315,7 @@ impl Core {
uid: &NodeUid, uid: &NodeUid,
mtime: i64, mtime: i64,
size: i64, size: i64,
expected_local: Option<LocalSig>,
) -> Result<(), String> { ) -> Result<(), String> {
let path = local_root.join(rel_to_path(rel)); let path = local_root.join(rel_to_path(rel));
if let Some(parent) = path.parent() { if let Some(parent) = path.parent() {
@@ -1320,6 +1350,13 @@ impl Core {
out.flush().map_err(|e| format!("flush tmp {rel}: {e}"))?; out.flush().map_err(|e| format!("flush tmp {rel}: {e}"))?;
out.sync_all().map_err(|e| format!("sync tmp {rel}: {e}"))?; out.sync_all().map_err(|e| format!("sync tmp {rel}: {e}"))?;
} }
// The last moment at which the destination can still be saved. Anything
// other than the state planning classified means a writer got there
// first, and its bytes are the ones nothing else has a copy of.
if let Err(e) = keep_racing_local_edit(&path, expected_local) {
let _ = std::fs::remove_file(&tmp);
return Err(format!("set aside racing local edit for {rel}: {e}"));
}
std::fs::rename(&tmp, &path).map_err(|e| format!("place {rel}: {e}"))?; std::fs::rename(&tmp, &path).map_err(|e| format!("place {rel}: {e}"))?;
// Match local mtime to the remote's so neither side looks "changed" next pass. // Match local mtime to the remote's so neither side looks "changed" next pass.
let f = std::fs::File::options() let f = std::fs::File::options()
@@ -1335,23 +1372,36 @@ impl Core {
.and_then(|dir| dir.sync_all()) .and_then(|dir| dir.sync_all())
.map_err(|e| format!("sync directory for {rel}: {e}"))?; .map_err(|e| format!("sync directory for {rel}: {e}"))?;
} }
self.record_file_baseline(folder_id, rel, &path, uid).await // Here the file on disk *is* the transferred content: it was published
// by a rename and stamped just above, so its stat is the identity.
self.record_file_baseline(folder_id, rel, local_sig(&path), uid)
.await
} }
/// Refresh a file's baseline from ground truth: the local stat and the /// Refresh a file's baseline from ground truth: the local identity these
/// remote node's reported signature. Called after every upload/download so /// bytes came from and the remote node's reported signature. Called after
/// the very next reconcile sees the path as unchanged. /// every upload/download so the very next reconcile sees the path as
/// unchanged.
///
/// `local` is the identity the *transferred content* corresponds to, which
/// is not always the file's current stat. An upload streams the live file,
/// so it passes the identity it read the first byte at: if a writer changed
/// the file mid-stream, the remote now holds a torn revision, and recording
/// the pre-upload identity is what makes the next pass see the path as
/// locally changed and upload a whole, clean revision over it. Recording the
/// post-upload stat instead declared the torn revision synchronized and left
/// it as the file's content on Drive (bugs.md B40).
async fn record_file_baseline( async fn record_file_baseline(
&self, &self,
folder_id: i64, folder_id: i64,
rel: &str, rel: &str,
local_path: &Path, local: Option<LocalSig>,
uid: &NodeUid, uid: &NodeUid,
) -> Result<(), String> { ) -> Result<(), String> {
let (lmtime, lsize) = match std::fs::metadata(local_path) { let Some(local) = local else {
Ok(m) => (system_mtime(&m), m.len() as i64), return Err(format!("stat {rel}: no local identity to baseline"));
Err(e) => return Err(format!("stat {rel}: {e}")),
}; };
let (lmtime, lsize) = (local.mtime, local.size);
// Re-fetch the node so the stored remote signature is exactly what a walk // Re-fetch the node so the stored remote signature is exactly what a walk
// will report next time. // will report next time.
let (rmtime, rsize) = match self.client.enumerate_nodes(std::slice::from_ref(uid)).await { let (rmtime, rsize) = match self.client.enumerate_nodes(std::slice::from_ref(uid)).await {
@@ -1371,6 +1421,7 @@ impl Core {
rel_path: rel.to_string(), rel_path: rel.to_string(),
remote_uid: Some(uid.to_string()), remote_uid: Some(uid.to_string()),
local_mtime: lmtime, local_mtime: lmtime,
local_mtime_ns: local.mtime_ns,
local_size: lsize, local_size: lsize,
remote_rev: Some(rmtime.to_string()), remote_rev: Some(rmtime.to_string()),
remote_hash: Some(rsize.to_string()), remote_hash: Some(rsize.to_string()),
@@ -1380,6 +1431,177 @@ impl Core {
} }
} }
/// Say so when a file changed under an upload that was streaming it.
///
/// Purely diagnostic — the correction is already in place, because the baseline
/// records the identity the upload *read* rather than the one on disk now, so
/// the next pass classifies the path as locally changed and replaces the torn
/// revision. Without the log there is nothing in the journal to explain the
/// extra revision (bugs.md B40).
fn warn_if_torn(rel: &str, path: &Path, streamed: LocalSig) {
let now = local_sig(path);
if !now.is_some_and(|now| now.same_content(&streamed)) {
warn!(
rel,
?streamed,
?now,
"sync: file changed while uploading; the remote revision may be torn \
and will be replaced on the next pass"
);
}
}
/// The identity a reconcile pass uses to recognise a local file it has already
/// looked at: `None` when nothing is at the path.
fn local_sig(path: &Path) -> Option<LocalSig> {
std::fs::metadata(path).ok().map(|m| LocalSig {
mtime: system_mtime(&m),
mtime_ns: system_mtime_ns(&m),
size: m.len() as i64,
})
}
/// Guard the destination of a download: if what is at `path` is no longer what
/// planning classified, move it aside as a conflict copy before the caller
/// renames over it.
///
/// A reconcile pass decides what to do from a walk that finished before the
/// transfer started, and a download of a large file can run for minutes. Without
/// this, a local edit completed inside that window was replaced by the remote
/// version with no copy of it kept anywhere — the one shape of loss a sync
/// engine must not have (bugs.md B39). Keeping the copy makes it re-upload as a
/// new file on the next pass, which is the same resolution a both-sides-changed
/// conflict already gets.
///
/// A path that is empty when it was expected to be is not a conflict: the
/// download is about to recreate it, and there is nothing to preserve.
fn keep_racing_local_edit(path: &Path, expected: Option<LocalSig>) -> std::io::Result<()> {
let actual = local_sig(path);
let unchanged = match (actual, expected) {
(Some(a), Some(e)) => a.same_content(&e),
// Nothing there: the download is about to create it, and there is
// nothing to preserve either way.
(None, _) => true,
(Some(_), None) => false,
};
if unchanged {
return Ok(());
}
warn!(
path = %path.display(),
?expected,
?actual,
"sync: local file changed while downloading; keeping it as a conflict copy"
);
preserve_conflict_copy(path, now_secs()).map(|_| ())
}
/// Read a local removal as the sync engine has to read it: the *state* it was
/// asking for is "this path is gone", so a path that was already gone is a
/// success, not a failure.
///
/// It matters because the baseline is only dropped on success. Treating
/// `NotFound` as an error left the baseline in place for a file that no longer
/// exists, so the same removal was re-attempted and re-reported as an error on
/// every pass, forever. Every other error is still an error, and keeping the
/// baseline is exactly right there: the surviving file must not be mistaken for
/// a new local creation on the next pass and uploaded back (bugs.md B42).
fn removed_locally(result: std::io::Result<()>) -> std::io::Result<()> {
match result {
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
other => other,
}
}
/// The body of [`Core::walk_local`], without a `Core`: recursively collect the
/// local tree under `dir` into `out`, reporting each entry to `scanned`.
///
/// **Every failure here is fatal to the pass, deliberately.** A `read_dir` that
/// fails, an entry that cannot be stat-ed, a name that is not UTF-8 — each of
/// them would, if skipped, leave `out` describing a *smaller* tree than the one
/// on disk. Reconciliation reads a path present in the baseline and absent from
/// the scan as "the user deleted this", and propagates that to Drive. So an
/// unreadable subdirectory would trash its remote counterpart, and the deeper
/// the failure, the more it takes with it (`docs/BUGS.md` B55). Refusing the
/// whole pass keeps the previous baseline and leaves the remote alone; the next
/// pass retries once the transient condition clears.
///
/// Split out from the method so exactly that can be tested against a real tree
/// with a real unreadable directory in it, which needs no `Core` at all.
fn walk_local_tree(
root: &Path,
dir: &Path,
rules: &IgnoreRules,
writing: &HashSet<PathBuf>,
out: &mut HashMap<String, LocalItem>,
scanned: &mut dyn FnMut(&str),
) -> Result<(), String> {
let entries = std::fs::read_dir(dir).map_err(|e| format!("read {}: {e}", dir.display()))?;
for entry in entries {
let entry = entry.map_err(|e| format!("read entry in {}: {e}", dir.display()))?;
let path = entry.path();
let meta = std::fs::symlink_metadata(&path)
.map_err(|e| format!("stat {}: {e}", path.display()))?;
let stripped = path.strip_prefix(root).map_err(|e| {
format!(
"local path {} escaped sync root {}: {e}",
path.display(),
root.display()
)
})?;
let rel = stripped.to_str().ok_or_else(|| {
format!(
"local path {} is not valid UTF-8; refusing a destructive sync pass",
path.display()
)
})?;
// Ignore our own in-flight download temp files.
if rel.contains(".pdfs-tmp-") {
continue;
}
// Ignored paths are dropped here as well as from the classification
// union — not for correctness (the union filter is what makes this
// safe) but so an ignored `node_modules/` is never descended into or
// counted against scan progress.
if rules.is_ignored(rel, meta.is_dir()) {
continue;
}
if meta.is_dir() {
out.insert(
rel.to_string(),
LocalItem {
is_dir: true,
mtime: 0,
mtime_ns: None,
size: 0,
open_for_write: false,
},
);
scanned(rel);
walk_local_tree(root, &path, rules, writing, out, scanned)?;
} else if meta.is_file() {
// Check whether any process holds this file open for writing.
// The canonical path is used because /proc/*/fd links resolve
// to canonical targets.
let ofw = std::fs::canonicalize(&path)
.map(|canon| writing.contains(&canon))
.unwrap_or(false);
out.insert(
rel.to_string(),
LocalItem {
is_dir: false,
mtime: system_mtime(&meta),
mtime_ns: system_mtime_ns(&meta),
size: meta.len() as i64,
open_for_write: ofw,
},
);
scanned(rel);
}
}
Ok(())
}
/// Publish a conflict copy without overwriting an earlier conflict bearing the /// Publish a conflict copy without overwriting an earlier conflict bearing the
/// same timestamp. The original is removed only after the new copy is durable. /// same timestamp. The original is removed only after the new copy is durable.
fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> { fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> {
@@ -1413,6 +1635,26 @@ fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> {
unreachable!("u32 conflict suffix space exhausted") unreachable!("u32 conflict suffix space exhausted")
} }
/// The remote uid of `rel`'s parent folder: from the pass's folder-uid map
/// first, and from the baseline second for a parent the map never covered.
///
/// Used to name the folder whose cached listing a remote mutation has just
/// invalidated (B86). `None` is not an error — it only means this pass cannot
/// say which folder to invalidate, and the caller skips it.
fn parent_uid_of(
rel: &str,
remote_dirs: &HashMap<String, NodeUid>,
baseline: &HashMap<String, StoredSyncEntry>,
) -> Option<NodeUid> {
let parent = parent_rel(rel);
remote_dirs.get(parent).cloned().or_else(|| {
baseline
.get(parent)
.and_then(|e| e.remote_uid.as_deref())
.and_then(parse_uid)
})
}
fn sync_parent_dir(path: &Path) -> std::io::Result<()> { fn sync_parent_dir(path: &Path) -> std::io::Result<()> {
match path.parent() { match path.parent() {
Some(parent) => std::fs::File::open(parent)?.sync_all(), Some(parent) => std::fs::File::open(parent)?.sync_all(),
@@ -1445,10 +1687,77 @@ fn system_mtime(meta: &std::fs::Metadata) -> i64 {
.map(|d| d.as_secs() as i64) .map(|d| d.as_secs() as i64)
.unwrap_or(0) .unwrap_or(0)
} }
/// The same modification time in nanoseconds, when the filesystem reports one.
///
/// Whole seconds cannot distinguish an edit made in the same second as the last
/// sync from no edit at all, and if the size did not change either, that file
/// was never uploaded (bugs.md B25). `None` for a clock or filesystem that
/// cannot answer, which falls the comparison back to seconds rather than
/// asserting an mtime of zero.
fn system_mtime_ns(meta: &std::fs::Metadata) -> Option<i64> {
meta.modified()
.ok()
.and_then(|t| t.duration_since(SystemTime::UNIX_EPOCH).ok())
.and_then(|d| i64::try_from(d.as_nanos()).ok())
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
/// A removal of something already gone has produced the state the sync
/// engine wanted, so its baseline is dropped rather than retried forever.
/// Anything else stays an error and keeps the baseline (bugs.md B42).
#[test]
fn an_already_absent_path_counts_as_removed() {
let root = std::env::temp_dir().join(format!("pdfs-b42-{}", std::process::id()));
let missing = root.join("never-existed");
assert!(removed_locally(std::fs::remove_file(&missing)).is_ok());
assert!(removed_locally(std::fs::remove_dir_all(&missing)).is_ok());
// A directory in the way of `remove_file` is a real failure, and must
// stay one: the file survives, and its baseline has to survive with it.
let occupied = root.join("a-directory");
std::fs::create_dir_all(&occupied).unwrap();
assert!(removed_locally(std::fs::remove_file(&occupied)).is_err());
let _ = std::fs::remove_dir_all(&root);
}
/// The download guard: a destination that still matches the plan is
/// replaced silently, one that has moved on since is kept (bugs.md B39).
#[test]
fn a_download_keeps_a_local_edit_it_did_not_plan_for() {
let root = std::env::temp_dir().join(format!("pdfs-b39-{}", std::process::id()));
std::fs::create_dir_all(&root).unwrap();
let path = root.join("report.txt");
// Nothing there, nothing expected: the download simply creates it.
assert!(keep_racing_local_edit(&path, None).is_ok());
assert!(!path.exists());
// Unchanged since planning: no copy, the caller overwrites it.
std::fs::write(&path, b"planned").unwrap();
let planned = local_sig(&path).unwrap();
keep_racing_local_edit(&path, Some(planned)).unwrap();
assert_eq!(std::fs::read(&path).unwrap(), b"planned");
// Changed since planning: the racing bytes are the only copy of
// themselves, so they must survive the overwrite as a conflict copy.
std::fs::write(&path, b"a much longer local edit").unwrap();
keep_racing_local_edit(&path, Some(planned)).unwrap();
let kept: Vec<_> = std::fs::read_dir(&root)
.unwrap()
.filter_map(|e| e.ok())
.map(|e| std::fs::read(e.path()).unwrap_or_default())
.collect();
assert!(
kept.iter()
.any(|bytes| bytes == b"a much longer local edit"),
"the racing edit must still exist somewhere: {kept:?}"
);
let _ = std::fs::remove_dir_all(&root);
}
/// Feed `n` `Reconcile` events `gap` apart on another thread, starting /// Feed `n` `Reconcile` events `gap` apart on another thread, starting
/// immediately. Returns the sender's join handle so the test can keep it /// immediately. Returns the sender's join handle so the test can keep it
/// alive (dropping the sender early would look like a disconnect). /// alive (dropping the sender early would look like a disconnect).
@@ -1534,6 +1843,7 @@ mod tests {
fn local_item(is_dir: bool) -> LocalItem { fn local_item(is_dir: bool) -> LocalItem {
LocalItem { LocalItem {
mtime_ns: None,
is_dir, is_dir,
mtime: 1, mtime: 1,
size: 1, size: 1,
@@ -1691,6 +2001,44 @@ mod tests {
assert!(guard_local_wipe(&base, &fresh).is_err()); assert!(guard_local_wipe(&base, &fresh).is_err());
} }
/// B54's remaining case, end to end: a folder whose local root is still
/// *readable* but is no longer the tree it was — an empty directory left
/// where a mount used to be, or a different device mounted over it.
///
/// Nothing fails here. The scan succeeds and honestly reports an empty
/// directory, so the only thing standing between the user and a trashed
/// remote folder is the guard, and it has to fire for a one-file baseline
/// exactly as it does for a large one.
#[test]
fn a_replaced_mountpoint_trips_the_guard_at_every_baseline_size() {
let root = sync_test_dir("wipe-replaced-mount");
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut local: HashMap<String, LocalItem> = HashMap::new();
let mut progress = no_progress();
walk_local_tree(&root, &root, &rules, &writing, &mut local, &mut progress).unwrap();
assert!(local.is_empty(), "the replacement root is genuinely empty");
for size in 1..=3 {
let baseline: HashMap<String, ()> =
(0..size).map(|i| (format!("file{i}.txt"), ())).collect();
assert!(
guard_local_wipe(&filter_baseline(&baseline, &rules), &local).is_err(),
"a {size}-path baseline against an empty root is not a deletion"
);
}
// And the guard steps aside the moment the real tree is back, so a
// remounted folder resumes rather than staying wedged.
std::fs::write(root.join("file0.txt"), b"back").unwrap();
let mut local: HashMap<String, LocalItem> = HashMap::new();
walk_local_tree(&root, &root, &rules, &writing, &mut local, &mut progress).unwrap();
let baseline: HashMap<String, ()> = [("file0.txt".to_string(), ())].into_iter().collect();
assert!(guard_local_wipe(&filter_baseline(&baseline, &rules), &local).is_ok());
std::fs::remove_dir_all(&root).unwrap();
}
/// A restored folder (features.md 5.2) starts with an empty baseline against /// A restored folder (features.md 5.2) starts with an empty baseline against
/// an empty local directory and a full remote. That must reconcile as /// an empty local directory and a full remote. That must reconcile as
/// "download everything" rather than tripping the wipe guard — otherwise /// "download everything" rather than tripping the wipe guard — otherwise
@@ -1733,6 +2081,7 @@ mod tests {
remote_uid: Some("v~l".into()), remote_uid: Some("v~l".into()),
local_mtime: 0, local_mtime: 0,
local_size: 0, local_size: 0,
local_mtime_ns: None,
remote_rev: Some(mtime.to_string()), remote_rev: Some(mtime.to_string()),
remote_hash: Some(size.to_string()), remote_hash: Some(size.to_string()),
}, },
@@ -1770,6 +2119,7 @@ mod tests {
remote_uid: Some("v~l".into()), remote_uid: Some("v~l".into()),
local_mtime: 0, local_mtime: 0,
local_size: 0, local_size: 0,
local_mtime_ns: None,
remote_rev: None, remote_rev: None,
remote_hash: None, remote_hash: None,
}, },
@@ -1784,6 +2134,7 @@ mod tests {
remote_uid: Some("v~l".into()), remote_uid: Some("v~l".into()),
local_mtime: 10, local_mtime: 10,
local_size: 20, local_size: 20,
local_mtime_ns: None,
remote_rev: Some("1700".into()), remote_rev: Some("1700".into()),
remote_hash: Some("4096".into()), remote_hash: Some("4096".into()),
}; };
@@ -1797,6 +2148,7 @@ mod tests {
remote_uid: Some("v~l".into()), remote_uid: Some("v~l".into()),
local_mtime: 0, local_mtime: 0,
local_size: 0, local_size: 0,
local_mtime_ns: None,
remote_rev: None, remote_rev: None,
remote_hash: None, remote_hash: None,
}; };
@@ -1832,6 +2184,7 @@ mod tests {
uid: uid(), uid: uid(),
mtime: 0, mtime: 0,
size: 10, size: 10,
local: None,
}; };
assert_eq!(op.kind(), ActivityKind::Download); assert_eq!(op.kind(), ActivityKind::Download);
} }
@@ -1887,6 +2240,204 @@ mod tests {
std::fs::remove_dir_all(dir).unwrap(); std::fs::remove_dir_all(dir).unwrap();
} }
/// B49's remaining cases. Preservation is the step that makes a download
/// safe to publish, so every way it can fail has to leave the local file
/// where it was — the download is aborted, not the file replaced.
#[test]
fn conflict_preservation_failures_all_keep_the_local_file() {
use std::os::unix::fs::PermissionsExt as _;
// An unwritable directory: the conflict copy cannot be created at all.
let dir = sync_test_dir("conflict-eacces");
let source = dir.join("notes.txt");
std::fs::write(&source, b"the only copy").unwrap();
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o500)).unwrap();
let denied = preserve_conflict_copy(&source, 1700);
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)).unwrap();
assert!(denied.is_err(), "a read-only directory must fail the copy");
assert_eq!(
std::fs::read(&source).unwrap(),
b"the only copy",
"the local file survives byte-identical"
);
std::fs::remove_dir_all(&dir).unwrap();
// A source that vanished between planning and preservation. The pass
// must report it rather than publish the download over the gap.
let dir = sync_test_dir("conflict-enoent");
assert!(preserve_conflict_copy(&dir.join("gone.txt"), 1700).is_err());
std::fs::remove_dir_all(&dir).unwrap();
// A write error partway through the copy — induced by making the
// destination name a directory, so `create_new` fails as something
// other than `AlreadyExists`.
let dir = sync_test_dir("conflict-eisdir");
let source = dir.join("notes.txt");
std::fs::write(&source, b"still the only copy").unwrap();
std::fs::create_dir(conflict_path(&source, 1700)).unwrap();
// Suffix 0 is taken by a directory; `create_new` on it reports
// `AlreadyExists`, so preservation moves on to suffix 1 and succeeds.
// What must not happen is the source being removed without a copy.
let preserved = preserve_conflict_copy(&source, 1700).unwrap();
assert!(preserved.is_file());
assert_eq!(std::fs::read(&preserved).unwrap(), b"still the only copy");
assert!(!source.exists());
std::fs::remove_dir_all(&dir).unwrap();
}
/// A conflict copy keeps the bytes *and* the metadata that identifies them,
/// so the next pass uploads the user's version rather than a file that looks
/// freshly written.
#[test]
fn a_conflict_copy_is_byte_and_mtime_identical() {
use std::os::unix::fs::PermissionsExt as _;
let dir = sync_test_dir("conflict-identity");
let source = dir.join("notes.txt");
std::fs::write(&source, b"local edit that must survive").unwrap();
let before = std::fs::metadata(&source).unwrap();
let preserved = preserve_conflict_copy(&source, 1700).unwrap();
let after = std::fs::metadata(&preserved).unwrap();
assert_eq!(
std::fs::read(&preserved).unwrap(),
b"local edit that must survive"
);
assert_eq!(after.len(), before.len());
assert_eq!(after.modified().unwrap(), before.modified().unwrap());
assert_eq!(after.permissions().mode(), before.permissions().mode());
std::fs::remove_dir_all(&dir).unwrap();
}
// ---- B55: an incomplete local scan is never a set of deletions ---------
fn no_progress() -> impl FnMut(&str) {
|_: &str| {}
}
/// A directory the pass cannot read must fail the pass. Omitting it would
/// hand reconciliation a baseline-minus-subtree, which it reads as the user
/// having deleted that subtree and propagates to Drive (B55).
#[test]
fn an_unreadable_subdirectory_fails_the_scan_instead_of_shrinking_it() {
use std::os::unix::fs::PermissionsExt as _;
let root = sync_test_dir("scan-eacces");
std::fs::write(root.join("keep.txt"), b"a").unwrap();
let locked = root.join("locked");
std::fs::create_dir(&locked).unwrap();
std::fs::write(locked.join("inner.txt"), b"b").unwrap();
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o000)).unwrap();
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut out = HashMap::new();
let mut progress = no_progress();
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o700)).unwrap();
std::fs::remove_dir_all(&root).unwrap();
let error = result.expect_err("an unreadable directory must abort the pass");
assert!(
error.contains("locked"),
"the error should name the subtree it could not read: {error}"
);
}
/// The same rule one level up: if the sync root itself cannot be read, the
/// pass sees nothing, and "nothing" must never mean "delete everything".
#[test]
fn an_unreadable_root_fails_the_scan() {
use std::os::unix::fs::PermissionsExt as _;
let root = sync_test_dir("scan-root-eacces");
std::fs::write(root.join("keep.txt"), b"a").unwrap();
std::fs::set_permissions(&root, std::fs::Permissions::from_mode(0o000)).unwrap();
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut out = HashMap::new();
let mut progress = no_progress();
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
std::fs::set_permissions(&root, std::fs::Permissions::from_mode(0o700)).unwrap();
std::fs::remove_dir_all(&root).unwrap();
assert!(result.is_err());
assert!(out.is_empty());
}
/// A root that is gone entirely — an unmounted device folder, a removed
/// directory — is the sharpest version of the same case: every baseline path
/// is "missing", and the scan must refuse rather than report an empty tree.
#[test]
fn an_absent_root_fails_the_scan() {
let root = sync_test_dir("scan-absent");
std::fs::remove_dir_all(&root).unwrap();
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut out = HashMap::new();
let mut progress = no_progress();
assert!(walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress).is_err());
assert!(out.is_empty());
}
/// A name that is not valid UTF-8 cannot be keyed into the scan, so it
/// cannot be compared against the baseline either. Refusing the pass is the
/// only safe reading: skipping it would classify the file as deleted.
#[test]
fn a_non_utf8_name_fails_the_scan() {
use std::ffi::OsStr;
use std::os::unix::ffi::OsStrExt as _;
let root = sync_test_dir("scan-non-utf8");
std::fs::write(root.join(OsStr::from_bytes(b"bad\xffname")), b"x").unwrap();
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut out = HashMap::new();
let mut progress = no_progress();
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
std::fs::remove_dir_all(&root).unwrap();
let error = result.expect_err("a name the scan cannot key must abort the pass");
assert!(error.contains("UTF-8"), "{error}");
}
/// The complement: a scan that *can* read everything reports everything, so
/// the strictness above does not come at the cost of a working pass. A
/// symlink is skipped by design rather than by failure, and a dangling one
/// stats fine through `symlink_metadata`.
#[test]
fn a_readable_tree_scans_completely() {
let root = sync_test_dir("scan-complete");
std::fs::write(root.join("a.txt"), b"aa").unwrap();
std::fs::create_dir(root.join("sub")).unwrap();
std::fs::write(root.join("sub/b.txt"), b"bbb").unwrap();
std::os::unix::fs::symlink("nowhere", root.join("dangling")).unwrap();
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut out = HashMap::new();
let mut seen: Vec<String> = Vec::new();
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut |rel| {
seen.push(rel.to_string())
});
std::fs::remove_dir_all(&root).unwrap();
result.unwrap();
let mut keys: Vec<&str> = out.keys().map(String::as_str).collect();
keys.sort_unstable();
assert_eq!(keys, vec!["a.txt", "sub", "sub/b.txt"]);
assert_eq!(out["a.txt"].size, 2);
assert_eq!(out["sub/b.txt"].size, 3);
assert!(out["sub"].is_dir);
assert_eq!(seen.len(), 3, "every reported entry ticks scan progress");
}
fn sync_test_dir(label: &str) -> PathBuf { fn sync_test_dir(label: &str) -> PathBuf {
let nonce = SystemTime::now() let nonce = SystemTime::now()
.duration_since(SystemTime::UNIX_EPOCH) .duration_since(SystemTime::UNIX_EPOCH)
+65 -17
View File
@@ -32,17 +32,24 @@ pub(crate) enum SyncMsg {
ReconcileAll, ReconcileAll,
/// Re-read the folder set and adjust the filesystem watches (after add/remove). /// Re-read the folder set and adjust the filesystem watches (after add/remove).
Rewatch, Rewatch,
/// The daemon is tearing down: finish the current pass and end the engine
/// thread, so the mount can join it (bugs.md B44).
Stop,
} }
/// Start the sync engine: a control thread that owns the filesystem watcher and /// Start the sync engine: a control thread that owns the filesystem watcher and
/// serialises reconcile passes, plus a periodic poll thread. The engine receives /// serialises reconcile passes, plus a periodic poll thread. The engine receives
/// on `rx`; senders live in [`Core::sync_tx`]. /// on `rx`; senders live in [`Core::sync_tx`].
pub(crate) fn spawn(core: Core, rx: Receiver<SyncMsg>) { pub(crate) fn spawn(core: Core, rx: Receiver<SyncMsg>) -> Option<std::thread::JoinHandle<()>> {
if let Err(e) = std::thread::Builder::new() match std::thread::Builder::new()
.name("pdfs-sync".into()) .name("pdfs-sync".into())
.spawn(move || engine_loop(core, rx)) .spawn(move || engine_loop(core, rx))
{ {
warn!(error = %e, "failed to start sync engine"); Ok(handle) => Some(handle),
Err(e) => {
warn!(error = %e, "failed to start sync engine");
None
}
} }
} }
@@ -74,11 +81,11 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
// Periodic remote poll. // Periodic remote poll.
{ {
let tx = core.sync_tx.clone(); let tx = core.sync_tx.clone();
let shutdown = core.shutdown.clone();
std::thread::Builder::new() std::thread::Builder::new()
.name("pdfs-sync-poll".into()) .name("pdfs-sync-poll".into())
.spawn(move || { .spawn(move || {
loop { while shutdown.sleep(POLL_INTERVAL) {
std::thread::sleep(POLL_INTERVAL);
if tx.send(SyncMsg::ReconcileAll).is_err() { if tx.send(SyncMsg::ReconcileAll).is_err() {
break; break;
} }
@@ -94,8 +101,12 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
let mut ids: HashSet<i64> = HashSet::new(); let mut ids: HashSet<i64> = HashSet::new();
let mut all = false; let mut all = false;
let mut do_rewatch = false; let mut do_rewatch = false;
classify(msg, &mut ids, &mut all, &mut do_rewatch); if !classify(msg, &mut ids, &mut all, &mut do_rewatch)
settle(&rx, &mut ids, &mut all, &mut do_rewatch); || !settle(&rx, &mut ids, &mut all, &mut do_rewatch)
{
info!("sync engine stopping");
return;
}
if do_rewatch { if do_rewatch {
rewatch(&core, &mut watcher, &watched); rewatch(&core, &mut watcher, &watched);
@@ -104,6 +115,9 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
reconcile_all(&core); reconcile_all(&core);
} else { } else {
for id in ids { for id in ids {
if core.sync_paused() {
break;
}
if let Ok(Some(folder)) = core.db.sync_folder_get(id) { if let Ok(Some(folder)) = core.db.sync_folder_get(id) {
core.reconcile_folder(&folder); core.reconcile_folder(&folder);
} }
@@ -116,9 +130,9 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
fn poll_only_loop(core: &Core, rx: Receiver<SyncMsg>) { fn poll_only_loop(core: &Core, rx: Receiver<SyncMsg>) {
{ {
let tx = core.sync_tx.clone(); let tx = core.sync_tx.clone();
let shutdown = core.shutdown.clone();
std::thread::spawn(move || { std::thread::spawn(move || {
loop { while shutdown.sleep(POLL_INTERVAL) {
std::thread::sleep(POLL_INTERVAL);
if tx.send(SyncMsg::ReconcileAll).is_err() { if tx.send(SyncMsg::ReconcileAll).is_err() {
break; break;
} }
@@ -130,12 +144,19 @@ fn poll_only_loop(core: &Core, rx: Receiver<SyncMsg>) {
let mut ids: HashSet<i64> = HashSet::new(); let mut ids: HashSet<i64> = HashSet::new();
let mut all = false; let mut all = false;
let mut do_rewatch = false; let mut do_rewatch = false;
classify(msg, &mut ids, &mut all, &mut do_rewatch); if !classify(msg, &mut ids, &mut all, &mut do_rewatch)
settle(&rx, &mut ids, &mut all, &mut do_rewatch); || !settle(&rx, &mut ids, &mut all, &mut do_rewatch)
{
info!("sync engine stopping");
return;
}
if all { if all {
reconcile_all(core); reconcile_all(core);
} else { } else {
for id in ids { for id in ids {
if core.sync_paused() {
break;
}
if let Ok(Some(folder)) = core.db.sync_folder_get(id) { if let Ok(Some(folder)) = core.db.sync_folder_get(id) {
core.reconcile_folder(&folder); core.reconcile_folder(&folder);
} }
@@ -158,7 +179,7 @@ pub(super) fn settle_with(
do_rewatch: &mut bool, do_rewatch: &mut bool,
quiet: Duration, quiet: Duration,
cap: Duration, cap: Duration,
) { ) -> bool {
let deadline = Instant::now() + cap; let deadline = Instant::now() + cap;
loop { loop {
let now = Instant::now(); let now = Instant::now();
@@ -170,32 +191,59 @@ pub(super) fn settle_with(
match rx.recv_timeout(wait) { match rx.recv_timeout(wait) {
// Another event: the burst is still going, so the quiet window // Another event: the burst is still going, so the quiet window
// starts again from here. // starts again from here.
Ok(m) => classify(m, ids, all, do_rewatch), // A stop lands mid-burst: give up the rest of the burst rather
// than settling for it, and let the caller end the loop.
Ok(m) => {
if !classify(m, ids, all, do_rewatch) {
return false;
}
}
// Quiet for a full window: whatever was being written has stopped. // Quiet for a full window: whatever was being written has stopped.
Err(RecvTimeoutError::Timeout) => break, Err(RecvTimeoutError::Timeout) => break,
// Every sender is gone; the caller's loop is about to end anyway. // Every sender is gone; the caller's loop is about to end anyway.
Err(RecvTimeoutError::Disconnected) => break, Err(RecvTimeoutError::Disconnected) => break,
} }
} }
true
} }
/// [`settle_with`] at the real timings. /// [`settle_with`] at the real timings.
fn settle(rx: &Receiver<SyncMsg>, ids: &mut HashSet<i64>, all: &mut bool, do_rewatch: &mut bool) { fn settle(
settle_with(rx, ids, all, do_rewatch, DEBOUNCE, MAX_SETTLE); rx: &Receiver<SyncMsg>,
ids: &mut HashSet<i64>,
all: &mut bool,
do_rewatch: &mut bool,
) -> bool {
settle_with(rx, ids, all, do_rewatch, DEBOUNCE, MAX_SETTLE)
} }
pub(super) fn classify(msg: SyncMsg, ids: &mut HashSet<i64>, all: &mut bool, rewatch: &mut bool) { /// Fold one message into the pending work of the current burst. Returns `false`
/// for [`SyncMsg::Stop`], which is not work but an instruction to end the loop.
pub(super) fn classify(
msg: SyncMsg,
ids: &mut HashSet<i64>,
all: &mut bool,
rewatch: &mut bool,
) -> bool {
match msg { match msg {
SyncMsg::Reconcile(id) => { SyncMsg::Reconcile(id) => {
ids.insert(id); ids.insert(id);
} }
SyncMsg::ReconcileAll => *all = true, SyncMsg::ReconcileAll => *all = true,
SyncMsg::Rewatch => *rewatch = true, SyncMsg::Rewatch => *rewatch = true,
SyncMsg::Stop => return false,
} }
true
} }
/// Reconcile every mirror folder in turn. /// Reconcile every mirror folder in turn. Skipped while sync is paused: the
/// resume sends a fresh [`SyncMsg::ReconcileAll`], which walks everything the
/// watcher reported meanwhile.
fn reconcile_all(core: &Core) { fn reconcile_all(core: &Core) {
if core.sync_paused() {
debug!("sync: paused, skipping reconcile");
return;
}
let folders = match core.db.sync_folder_list() { let folders = match core.db.sync_folder_list() {
Ok(f) => f, Ok(f) => f,
Err(e) => { Err(e) => {
+71
View File
@@ -2,9 +2,73 @@
use super::*; use super::*;
/// What a reconcile pass remembers about the local side of one path: enough to
/// recognise the same content again, and no more.
///
/// `mtime_ns` is the whole reason this is a struct rather than a pair — see
/// [`StoredSyncEntry::local_mtime_ns`](pdfs_core::db::StoredSyncEntry).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(super) struct LocalSig {
pub(super) mtime: i64,
pub(super) mtime_ns: Option<i64>,
pub(super) size: i64,
}
impl LocalSig {
/// Whether these two observations describe the same local content.
///
/// Nanoseconds decide it when both sides have them; otherwise the comparison
/// falls back to whole seconds, which is what a baseline row written before
/// schema 28 — or a filesystem that reports no sub-second time — can support
/// (bugs.md B25).
pub(super) fn same_content(&self, other: &Self) -> bool {
if self.size != other.size {
return false;
}
match (self.mtime_ns, other.mtime_ns) {
(Some(a), Some(b)) => a == b,
_ => self.mtime == other.mtime,
}
}
}
impl From<&LocalItem> for LocalSig {
fn from(item: &LocalItem) -> Self {
Self {
mtime: item.mtime,
mtime_ns: item.mtime_ns,
size: item.size,
}
}
}
impl From<&std::fs::Metadata> for LocalSig {
fn from(meta: &std::fs::Metadata) -> Self {
Self {
mtime: system_mtime(meta),
mtime_ns: system_mtime_ns(meta),
size: meta.len() as i64,
}
}
}
/// The baseline's record of the local side, as [`LocalSig`] compares it.
impl From<&StoredSyncEntry> for LocalSig {
fn from(entry: &StoredSyncEntry) -> Self {
Self {
mtime: entry.local_mtime,
mtime_ns: entry.local_mtime_ns,
size: entry.local_size,
}
}
}
pub(super) struct LocalItem { pub(super) struct LocalItem {
pub(super) is_dir: bool, pub(super) is_dir: bool,
pub(super) mtime: i64, pub(super) mtime: i64,
/// [`LocalItem::mtime`] to nanosecond precision, when the filesystem
/// reports one.
pub(super) mtime_ns: Option<i64>,
pub(super) size: i64, pub(super) size: i64,
/// True when another process holds this file open for writing (detected via /// True when another process holds this file open for writing (detected via
/// `/proc/*/fd`). An open-for-write file is kept in the map — so it is not /// `/proc/*/fd`). An open-for-write file is kept in the map — so it is not
@@ -91,11 +155,18 @@ pub(super) enum Pending {
UploadRevision { rel: String, uid: NodeUid }, UploadRevision { rel: String, uid: NodeUid },
/// Download remote `uid` to the local path, stamping `mtime`. `size` is the /// Download remote `uid` to the local path, stamping `mtime`. `size` is the
/// remote's reported size, used as the transfer's expected total. /// remote's reported size, used as the transfer's expected total.
///
/// `local` is what the planning walk saw at this path, or `None` if nothing
/// was there. A download can run for minutes, so the apply step revalidates
/// against it immediately before it renames over the destination: a writer
/// that finished an edit in the meantime must not be overwritten without a
/// conflict copy (bugs.md B39).
Download { Download {
rel: String, rel: String,
uid: NodeUid, uid: NodeUid,
mtime: i64, mtime: i64,
size: i64, size: i64,
local: Option<LocalSig>,
}, },
/// Both sides changed: set the local copy aside, then download remote `uid`. /// Both sides changed: set the local copy aside, then download remote `uid`.
Conflict { Conflict {
+80 -2
View File
@@ -200,6 +200,12 @@ pub(super) enum FilePlan {
DeleteLocal, DeleteLocal,
DeleteRemote, DeleteRemote,
ForgetBaseline, ForgetBaseline,
/// First sync of a file that already exists on both sides with identical
/// content: record the baseline without transferring anything. Without this,
/// a folder synced for the first time cuts a `(sync-conflict)` copy of every
/// pre-existing file, because a missing baseline makes both sides read as
/// "changed" (see the mass first-sync conflict storm in docs/BUGS.md).
AdoptBaseline,
} }
pub(super) fn plan_file( pub(super) fn plan_file(
@@ -212,17 +218,26 @@ pub(super) fn plan_file(
} }
let local_changed = local.is_some_and(|item| { let local_changed = local.is_some_and(|item| {
baseline.is_none_or(|base| base.local_mtime != item.mtime || base.local_size != item.size) // Nanoseconds where both sides have them (bugs.md B25): whole seconds
// could not see an edit made in the same second as the last sync that
// left the file the same length, so it was never uploaded.
baseline.is_none_or(|base| !LocalSig::from(item).same_content(&LocalSig::from(base)))
}); });
let remote_changed = remote.is_some_and(|item| { let remote_changed = remote.is_some_and(|item| {
baseline.is_none_or(|base| remote_sig(base) != Some((item.mtime, item.size))) baseline.is_none_or(|base| remote_sig(base) != Some((item.mtime, item.size)))
}); });
match (local, remote) { match (local, remote) {
(Some(_), Some(_)) => match (local_changed, remote_changed) { (Some(l), Some(r)) => match (local_changed, remote_changed) {
(false, false) => FilePlan::Unchanged, (false, false) => FilePlan::Unchanged,
(true, false) => FilePlan::UploadRevision, (true, false) => FilePlan::UploadRevision,
(false, true) => FilePlan::Download, (false, true) => FilePlan::Download,
// With a baseline, both-changed is a genuine concurrent edit and must
// be preserved as a conflict. Without one, "both changed" only means
// this is the first sync and there was nothing to compare to: adopt
// the baseline when the sizes already match rather than conflict-copy
// an already-identical file. Divergent sizes stay a real conflict.
(true, true) if baseline.is_none() && l.size == r.size => FilePlan::AdoptBaseline,
(true, true) => FilePlan::Conflict, (true, true) => FilePlan::Conflict,
}, },
(Some(_), None) if baseline.is_none() || local_changed => FilePlan::UploadNew, (Some(_), None) if baseline.is_none() || local_changed => FilePlan::UploadNew,
@@ -240,6 +255,7 @@ mod file_plan_tests {
fn local(mtime: i64, size: i64) -> LocalItem { fn local(mtime: i64, size: i64) -> LocalItem {
LocalItem { LocalItem {
mtime_ns: None,
is_dir: false, is_dir: false,
mtime, mtime,
size, size,
@@ -266,12 +282,49 @@ mod file_plan_tests {
rel_path: "file".into(), rel_path: "file".into(),
remote_uid: Some("v~l".into()), remote_uid: Some("v~l".into()),
local_mtime, local_mtime,
local_mtime_ns: None,
local_size, local_size,
remote_rev: Some(remote_mtime.to_string()), remote_rev: Some(remote_mtime.to_string()),
remote_hash: Some(remote_size.to_string()), remote_hash: Some(remote_size.to_string()),
} }
} }
/// Whole seconds cannot see an edit made inside the same second that leaves the
/// file the same length, so a mirror silently stopped uploading it. With a
/// nanosecond time on both sides it is an ordinary local change (bugs.md B25).
#[test]
fn a_same_second_same_size_edit_is_still_a_local_change() {
let mut base = baseline(10, 20, 30, 40);
base.local_mtime_ns = Some(10_000_000_000);
let edited = LocalItem {
mtime_ns: Some(10_400_000_000),
..local(10, 20)
};
let remote = remote(30, 40);
assert_eq!(
plan_file(Some(&edited), Some(&remote), Some(&base)),
FilePlan::UploadRevision
);
// The same file untouched still reads as unchanged.
let same = LocalItem {
mtime_ns: Some(10_000_000_000),
..local(10, 20)
};
assert_eq!(
plan_file(Some(&same), Some(&remote), Some(&base)),
FilePlan::Unchanged
);
// And a row from before the column exists compares on seconds, exactly
// as it did when it was written — no mass re-upload on upgrade.
let old = baseline(10, 20, 30, 40);
assert_eq!(
plan_file(Some(&edited), Some(&remote), Some(&old)),
FilePlan::Unchanged
);
}
#[test] #[test]
fn file_plan_covers_every_presence_and_change_shape() { fn file_plan_covers_every_presence_and_change_shape() {
let same_local = local(10, 20); let same_local = local(10, 20);
@@ -350,6 +403,31 @@ mod file_plan_tests {
} }
} }
#[test]
fn first_sync_of_an_identical_file_adopts_the_baseline_not_a_conflict() {
// No baseline yet (first sync). Same size on both sides is a pre-existing
// identical file, not a concurrent edit: adopt it rather than cut a
// conflict copy of every file in the folder.
let local = local(10, 40);
let remote = remote(30, 40);
assert_eq!(
plan_file(Some(&local), Some(&remote), None),
FilePlan::AdoptBaseline
);
}
#[test]
fn first_sync_of_differing_sizes_is_still_a_conflict() {
// No baseline and the sizes disagree: a genuine both-sides divergence,
// preserved non-destructively as a conflict.
let local = local(10, 20);
let remote = remote(30, 40);
assert_eq!(
plan_file(Some(&local), Some(&remote), None),
FilePlan::Conflict
);
}
#[test] #[test]
fn a_missing_remote_signature_is_treated_as_changed() { fn a_missing_remote_signature_is_treated_as_changed() {
let local = local(10, 20); let local = local(10, 20);
+99
View File
@@ -0,0 +1,99 @@
//! The `sd_notify` half of the service contract, without linking libsystemd.
//!
//! The protocol is one datagram of `KEY=VALUE` lines to the socket in
//! `$NOTIFY_SOCKET`. That is little enough to write directly, and writing it
//! directly keeps the daemon buildable on a machine without systemd headers —
//! everything here degrades to "no socket, do nothing" off systemd.
use std::os::unix::net::UnixDatagram;
use std::time::Duration;
use tracing::debug;
/// Send one notification. `false` when there is no manager to send it to, or the
/// send failed — a caller has nothing useful to do about either.
pub(crate) fn notify(message: &str) -> bool {
let Ok(socket_path) = std::env::var("NOTIFY_SOCKET") else {
return false;
};
if socket_path.is_empty() {
return false;
}
let Ok(socket) = UnixDatagram::unbound() else {
return false;
};
if let Some(name) = socket_path.strip_prefix('@') {
// Abstract namespace: the address is a NUL byte followed by the name,
// which `std`'s path-based API cannot express.
send_abstract(&socket, name, message)
} else {
socket.send_to(message.as_bytes(), &socket_path).is_ok()
}
}
/// `sendto` on an abstract-namespace address, which needs a hand-built
/// `sockaddr_un` because the first byte of the name is NUL.
fn send_abstract(socket: &UnixDatagram, name: &str, message: &str) -> bool {
use std::os::fd::AsRawFd;
let mut addr: libc::sockaddr_un = unsafe { std::mem::zeroed() };
addr.sun_family = libc::AF_UNIX as libc::sa_family_t;
let bytes = name.as_bytes();
// One byte for the leading NUL, and the path must still be addressable.
if bytes.len() + 1 > addr.sun_path.len() {
return false;
}
for (slot, byte) in addr.sun_path.iter_mut().skip(1).zip(bytes) {
*slot = *byte as libc::c_char;
}
let len = std::mem::size_of::<libc::sa_family_t>() + 1 + bytes.len();
let sent = unsafe {
libc::sendto(
socket.as_raw_fd(),
message.as_ptr().cast(),
message.len(),
0,
std::ptr::from_ref(&addr).cast(),
len as libc::socklen_t,
)
};
sent >= 0
}
/// Tell the manager the daemon is up and serving. Called once the mount is live,
/// so `systemctl --user start` returns when the mount actually works rather than
/// when the process exists.
pub(crate) fn ready() {
if notify("READY=1\n") {
debug!("notified systemd: ready");
}
}
/// Tell the manager the daemon is still healthy.
pub(crate) fn watchdog_ping() {
let _ = notify("WATCHDOG=1\n");
}
/// Tell the manager the daemon is stopping, so a slow teardown is not mistaken
/// for a hang by the watchdog it is racing.
pub(crate) fn stopping() {
let _ = notify("STOPPING=1\n");
}
/// How often the manager expects a ping, from `$WATCHDOG_USEC`, or `None` when
/// the watchdog is off or belongs to another process.
///
/// Reports *half* the configured interval, the interval systemd documents for
/// pinging: a ping period equal to the deadline would restart the daemon on the
/// first late scheduling.
pub(crate) fn watchdog_period() -> Option<Duration> {
// `WATCHDOG_PID` is set when the manager wants a specific process to answer;
// in that case a ping from anyone else is not the one it is waiting for.
if let Ok(pid) = std::env::var("WATCHDOG_PID")
&& pid.parse::<u32>().ok() != Some(std::process::id())
{
return None;
}
let usec: u64 = std::env::var("WATCHDOG_USEC").ok()?.parse().ok()?;
(usec > 0).then(|| Duration::from_micros(usec / 2))
}
+632
View File
@@ -0,0 +1,632 @@
//! Importing a Google Photos Takeout export into Proton Photos.
//!
//! [`pdfs_core::takeout`] says what is in the archives; this is the part that
//! talks to Proton. One import runs on its own thread and moves through four
//! phases, reporting into the transfer registry as a job so the GUI's transfer
//! list shows it without a bespoke widget:
//!
//! 1. **Scan** — read the archives' central directories and sidecars.
//! 2. **Deduplicate against the account** — ask the photos volume which of the
//! names already exist ([`ProtonPhotosClient::name_collisions`]), hash only
//! those files, then confirm with the content digest. A library that is
//! already half-synced by Proton Photos on a phone therefore costs one cheap
//! request per 150 names and reads no media at all for the misses.
//! 3. **Upload** the survivors, streaming each entry straight out of its zip —
//! no extraction to disk, so the import needs no scratch space.
//! 4. **Albums** — create the albums the export names that the account lacks,
//! then add every photo of each album, the ones that were already on the
//! account included.
//!
//! Re-running an interrupted import is the resume story: phase 2 finds
//! everything the previous run uploaded and skips it. Nothing is persisted
//! between runs.
use std::collections::{BTreeMap, HashMap};
use std::io::Read;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use pdfs_core::control::{ActivityKind, ImportSummary, TransferDirection};
use pdfs_core::takeout::{Bucket, TakeoutPhoto, TakeoutScan, scan_archives};
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::proton_sdk::ids::NodeUid;
use proton_drive_rs::{PhotoTag, PhotoTagsUpdate, PhotoUploadMetadata};
use sha1::{Digest, Sha1};
use tracing::{info, warn};
use super::Core;
use crate::transfers::CountingReader;
/// The one import this daemon may be running, and how the last one ended.
///
/// Process-global rather than a [`Core`] field because it must be *one* import
/// per daemon however many `Core` clones exist (the forked on-demand mounts each
/// hold one), and because a front-end asking for the result after the import
/// finished must still find it.
static IMPORT: std::sync::LazyLock<parking_lot::Mutex<ImportState>> =
std::sync::LazyLock::new(|| parking_lot::Mutex::new(ImportState::default()));
#[derive(Default)]
struct ImportState {
running: bool,
cancel: Option<Arc<AtomicBool>>,
last: Option<ImportSummary>,
}
/// Whether an import is running, and the last one's report.
pub(crate) fn import_status() -> (bool, Option<ImportSummary>) {
let state = IMPORT.lock();
(state.running, state.last.clone())
}
/// Ask the running import to stop. False when none is running.
pub(crate) fn cancel_import() -> bool {
match &IMPORT.lock().cancel {
Some(flag) => {
flag.store(true, Ordering::Relaxed);
true
}
None => false,
}
}
/// One photo to import, after the copies Google keeps per album are folded into
/// a single upload that joins several albums.
struct Planned {
photo: TakeoutPhoto,
/// Every album this photo belongs to, by export title.
albums: Vec<String>,
/// Filled in phase 2/3: where the photo ended up on the account.
uid: Option<NodeUid>,
}
/// Fold the scan into one entry per distinct photo.
///
/// Google stores a full copy of a photo in every album folder it appears in, so
/// the same file is in the export two or three times over. They are folded on
/// `(name, size)`: a byte-identical copy under the same name is the same photo,
/// and the album memberships of the copies are unioned onto the survivor. Two
/// genuinely different photos sharing both a name and an exact size would be
/// merged — vanishingly unlikely, and the cost of being wrong is one photo
/// filed under an extra album rather than lost data.
fn plan(scan: &TakeoutScan) -> Vec<Planned> {
let mut order: Vec<(String, u64)> = Vec::new();
let mut by_key: HashMap<(String, u64), Planned> = HashMap::new();
for photo in &scan.photos {
let key = (photo.name.clone(), photo.size);
let entry = by_key.entry(key.clone()).or_insert_with(|| {
order.push(key.clone());
Planned {
photo: photo.clone(),
albums: Vec::new(),
uid: None,
}
});
if let Bucket::Album(title) = &photo.bucket
&& !entry.albums.iter().any(|existing| existing == title)
{
entry.albums.push(title.clone());
}
// A copy that carries a capture time wins over one that does not: the
// album copy and the year-folder copy do not always both have a sidecar.
if entry.photo.capture_time.is_none() && photo.capture_time.is_some() {
entry.photo.capture_time = photo.capture_time;
}
entry.photo.favorite |= photo.favorite;
}
order
.into_iter()
.filter_map(|key| by_key.remove(&key))
.collect()
}
/// Open one archive and stream `entry` through `sink`.
fn read_entry<R>(
archive: &mut zip::ZipArchive<std::io::BufReader<std::fs::File>>,
entry: &str,
sink: impl FnOnce(&mut (dyn Read + Send)) -> R,
) -> CoreResult<R> {
let mut file = archive
.by_name(entry)
.map_err(|e| CoreError::invalid(format!("{entry}: {e}")))?;
Ok(sink(&mut file))
}
impl Core {
/// Import Google Photos Takeout archives. Blocking; run on its own thread.
///
/// `cancel` stops the run at the next photo boundary — an upload already on
/// the wire is allowed to finish, so a cancelled import never leaves a
/// half-written revision behind.
pub(crate) fn import_takeout(
&self,
archives: Vec<PathBuf>,
dry_run: bool,
cancel: Arc<AtomicBool>,
) -> CoreResult<ImportSummary> {
let job = self.transfers.begin_job("Importing Google Photos");
job.detail("reading archives");
let scan = scan_archives(&archives)?;
let mut planned = plan(&scan);
let mut summary = ImportSummary {
found: planned.len(),
skipped_trashed: scan.skipped_trashed,
..Default::default()
};
job.set_total(planned.len() as u64);
info!(
archives = archives.len(),
photos = summary.found,
albums = scan.albums.len(),
trashed = summary.skipped_trashed,
"takeout scan complete"
);
let photos = self.photos();
if self
.rt
.block_on(photos.get_photos_root())
.map_err(|e| CoreError::from_api(&e, "resolve photos library"))?
.is_none()
{
return Err(CoreError::invalid(
"this account has no Proton Photos library yet; open Proton Photos once to create it, then import again",
));
}
// Phase 2: which of these are already on the account.
job.detail("checking for photos you already have");
self.mark_existing(&mut planned, &archives, &mut summary, &cancel)?;
if dry_run {
summary.duplicates = planned.iter().filter(|p| p.uid.is_some()).count();
return Ok(summary);
}
// Phase 3: upload what is left, one archive at a time so each zip's
// central directory is parsed once.
self.upload_planned(&mut planned, &archives, &mut summary, &cancel, &job)?;
// Phase 4: albums. Attempted even after a cancel — the photos that did
// upload should still land in their albums.
if !scan.albums.is_empty() {
job.detail("sorting photos into albums");
if let Err(e) = self.link_albums(&planned, &scan.albums, &mut summary) {
warn!(error = %e, "takeout album linking failed");
}
}
self.spawn_timeline_refresh();
summary.cancelled = cancel.load(Ordering::Relaxed);
Ok(summary)
}
/// Fill in `uid` for every planned photo the account already holds.
///
/// Two passes on purpose: the name-only query is one request per 150 photos
/// and needs no file contents, so the expensive SHA-1 pass only runs for the
/// handful of names that actually collide.
fn mark_existing(
&self,
planned: &mut [Planned],
archives: &[PathBuf],
summary: &mut ImportSummary,
cancel: &AtomicBool,
) -> CoreResult<()> {
let photos = self.photos();
let names: Vec<String> = planned.iter().map(|p| p.photo.name.clone()).collect();
let collisions = self
.rt
.block_on(photos.name_collisions(&names))
.map_err(|e| CoreError::from_api(&e, "check for existing photos"))?;
// Hash the colliding files, grouped by archive so each zip opens once.
let mut by_archive: BTreeMap<usize, Vec<usize>> = BTreeMap::new();
for (index, planned) in planned.iter().enumerate() {
if collisions.get(index).copied().unwrap_or(false) {
by_archive
.entry(planned.photo.archive)
.or_default()
.push(index);
}
}
// Kept as a pair list, not a map: the duplicate lookup answers in input
// order, so the order these were built in is what maps answers back to
// planned photos.
let mut digests: Vec<(usize, String)> = Vec::new();
for (archive_index, indices) in by_archive {
if cancel.load(Ordering::Relaxed) {
break;
}
let mut archive = open_archive(archives, archive_index)?;
for index in indices {
let entry = planned[index].photo.entry.clone();
match read_entry(&mut archive, &entry, sha1_of) {
Ok(Ok(digest)) => digests.push((index, digest)),
Ok(Err(e)) | Err(e) => {
warn!(entry, error = %e, "cannot hash takeout entry");
}
}
}
}
if digests.is_empty() {
return Ok(());
}
let items: Vec<(String, String)> = digests
.iter()
.map(|(index, digest)| (planned[*index].photo.name.clone(), digest.clone()))
.collect();
let matches = self
.rt
.block_on(photos.find_duplicates_many(&items))
.map_err(|e| CoreError::from_api(&e, "check for existing photos"))?;
for (slot, (index, _)) in digests.iter().enumerate() {
if let Some(uid) = matches.get(slot).and_then(|uids| uids.first()) {
planned[*index].uid = Some(uid.clone());
summary.duplicates += 1;
}
}
Ok(())
}
/// Upload every planned photo that is not already on the account.
fn upload_planned(
&self,
planned: &mut [Planned],
archives: &[PathBuf],
summary: &mut ImportSummary,
cancel: &AtomicBool,
job: &crate::transfers::JobGuard,
) -> CoreResult<()> {
let photos = self.photos();
let mut favorites: Vec<NodeUid> = Vec::new();
let mut by_archive: BTreeMap<usize, Vec<usize>> = BTreeMap::new();
for (index, item) in planned.iter().enumerate() {
if item.uid.is_none() {
by_archive
.entry(item.photo.archive)
.or_default()
.push(index);
}
}
for (archive_index, indices) in by_archive {
if cancel.load(Ordering::Relaxed) || self.shutdown.is_stopping() {
break;
}
let mut archive = open_archive(archives, archive_index)?;
for index in indices {
if cancel.load(Ordering::Relaxed) || self.shutdown.is_stopping() {
break;
}
let photo = planned[index].photo.clone();
job.detail(photo.name.clone());
let guard = self.transfers.begin(
photo.name.clone(),
"",
TransferDirection::Upload,
photo.size,
);
let metadata = PhotoUploadMetadata {
capture_time: photo.capture_time,
..Default::default()
};
let uploaded = read_entry(&mut archive, &photo.entry, |reader| {
let counting = CountingReader::new(reader, &guard);
self.rt.block_on(photos.upload_photo_from(
&photo.name,
&photo.media_type,
counting,
photo.size as i64,
Vec::new(),
metadata,
false,
))
});
drop(guard);
job.step();
match uploaded {
Ok(Ok(uid)) => {
summary.uploaded += 1;
summary.bytes += photo.size;
if photo.favorite {
favorites.push(uid.clone());
}
planned[index].uid = Some(uid);
}
Ok(Err(e)) => {
summary.failed += 1;
warn!(name = photo.name, error = %e, "takeout photo upload failed");
}
Err(e) => {
summary.failed += 1;
warn!(name = photo.name, error = %e, "cannot read takeout entry");
}
}
}
}
// Favorites are a separate call: the upload seal takes classification
// tags, but `Favorite` is not one of them — it goes through the photos
// volume's dedicated favorite endpoint.
if !favorites.is_empty() {
let updates: Vec<PhotoTagsUpdate> = favorites
.into_iter()
.map(|node_uid| PhotoTagsUpdate {
node_uid,
tags_to_add: vec![PhotoTag::Favorite],
tags_to_remove: Vec::new(),
})
.collect();
if let Err(e) = self.rt.block_on(photos.update_photos(&updates)) {
warn!(error = %e, "marking imported favorites failed");
}
}
Ok(())
}
/// Create the albums the export names that the account lacks, then add each
/// album's photos to it.
fn link_albums(
&self,
planned: &[Planned],
albums: &[String],
summary: &mut ImportSummary,
) -> CoreResult<()> {
let photos = self.photos();
// Existing albums by decrypted name, so a second import of the same
// export adds to the album it made the first time instead of a twin.
let mut existing: HashMap<String, NodeUid> = HashMap::new();
let uids = self
.rt
.block_on(photos.enumerate_album_node_uids())
.map_err(|e| CoreError::from_api(&e, "list albums"))?;
if !uids.is_empty() {
let albums_nodes = self
.rt
.block_on(photos.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "read albums"))?;
for node in albums_nodes {
existing.entry(node.name.clone()).or_insert(node.uid);
}
}
for title in albums {
let members: Vec<NodeUid> = planned
.iter()
.filter(|item| item.albums.iter().any(|album| album == title))
.filter_map(|item| item.uid.clone())
.collect();
if members.is_empty() {
continue;
}
let album_uid = match existing.get(title) {
Some(uid) => uid.clone(),
None => match self.rt.block_on(photos.create_album(title)) {
Ok(uid) => {
summary.albums_created += 1;
existing.insert(title.clone(), uid.clone());
uid
}
Err(e) => {
warn!(album = title, error = %e, "cannot create album");
continue;
}
},
};
match self
.rt
.block_on(photos.add_photos_to_album(&album_uid, &members))
{
Ok(outcomes) => {
for (uid, outcome) in outcomes {
match outcome {
Ok(()) => summary.album_links += 1,
// A photo already in the album is the expected
// outcome of a re-run, not a failure worth counting.
Err(e) => {
warn!(album = title, %uid, error = %e, "cannot add photo to album")
}
}
}
}
Err(e) => warn!(album = title, error = %e, "album addition failed"),
}
}
Ok(())
}
/// Run an import on a background thread, logging its outcome to the activity
/// feed. Returns immediately.
///
/// Refuses while one is already running: two imports of the same export
/// would race each other's duplicate detection and upload everything twice.
pub(crate) fn spawn_takeout_import(
&self,
archives: Vec<PathBuf>,
dry_run: bool,
) -> CoreResult<()> {
let cancel = Arc::new(AtomicBool::new(false));
{
let mut state = IMPORT.lock();
if state.running {
return Err(CoreError::conflict(
"an import is already running; wait for it to finish or cancel it",
));
}
state.running = true;
state.cancel = Some(cancel.clone());
state.last = None;
}
let core = self.clone();
std::thread::spawn(move || {
let outcome = core.import_takeout(archives, dry_run, cancel);
{
let mut state = IMPORT.lock();
state.running = false;
state.cancel = None;
state.last = outcome.as_ref().ok().cloned();
}
match outcome {
Ok(summary) => {
let mut parts = vec![format!("{} uploaded", summary.uploaded)];
if summary.duplicates > 0 {
parts.push(format!("{} already there", summary.duplicates));
}
if summary.albums_created > 0 {
parts.push(format!("{} albums", summary.albums_created));
}
if summary.failed > 0 {
parts.push(format!("{} failed", summary.failed));
}
if summary.cancelled {
parts.push("cancelled".to_string());
}
core.log_activity(
ActivityKind::Upload,
"Google Photos import",
parts.join(" · "),
summary.failed == 0 && !summary.cancelled,
);
}
Err(e) => {
warn!(error = %e, "takeout import failed");
core.log_activity(ActivityKind::Upload, "Google Photos import", &e, false);
}
}
});
Ok(())
}
}
/// Open one of the import's archives by index.
fn open_archive(
archives: &[PathBuf],
index: usize,
) -> CoreResult<zip::ZipArchive<std::io::BufReader<std::fs::File>>> {
let path = archives
.get(index)
.ok_or_else(|| CoreError::internal(format!("archive {index} is not in the import")))?;
let file = std::fs::File::open(path)
.map_err(|e| CoreError::invalid(format!("cannot open {}: {e}", path.display())))?;
zip::ZipArchive::new(std::io::BufReader::new(file))
.map_err(|e| CoreError::invalid(format!("{} is not a readable zip: {e}", path.display())))
}
/// The lowercase-hex SHA-1 of everything `reader` yields — the digest Proton's
/// photo duplicate detection is keyed on.
fn sha1_of(reader: &mut (dyn Read + Send)) -> CoreResult<String> {
let mut hasher = Sha1::new();
let mut buffer = vec![0u8; 256 * 1024];
loop {
let read = reader
.read(&mut buffer)
.map_err(|e| CoreError::invalid(format!("read failed: {e}")))?;
if read == 0 {
break;
}
hasher.update(&buffer[..read]);
}
Ok(hex_lower(&hasher.finalize()))
}
fn hex_lower(bytes: &[u8]) -> String {
let mut out = String::with_capacity(bytes.len() * 2);
for byte in bytes {
out.push_str(&format!("{byte:02x}"));
}
out
}
#[cfg(test)]
mod tests {
use super::*;
use pdfs_core::takeout::TakeoutPhoto;
fn photo(name: &str, size: u64, bucket: Bucket) -> TakeoutPhoto {
TakeoutPhoto {
archive: 0,
entry: format!("Takeout/Google Photos/x/{name}"),
name: name.to_string(),
size,
media_type: "image/jpeg".to_string(),
capture_time: None,
favorite: false,
bucket,
}
}
#[test]
fn album_copies_of_one_photo_fold_into_a_single_upload() {
let scan = TakeoutScan {
photos: vec![
photo("IMG_1.jpg", 100, Bucket::Timeline),
photo("IMG_1.jpg", 100, Bucket::Album("Iceland".into())),
photo("IMG_1.jpg", 100, Bucket::Album("Best of".into())),
photo("IMG_2.jpg", 200, Bucket::Timeline),
],
albums: vec!["Iceland".into(), "Best of".into()],
..Default::default()
};
let planned = plan(&scan);
assert_eq!(planned.len(), 2, "the three copies are one upload");
assert_eq!(planned[0].albums, vec!["Iceland", "Best of"]);
assert!(planned[1].albums.is_empty());
}
#[test]
fn same_name_different_size_stays_two_photos() {
let scan = TakeoutScan {
photos: vec![
photo("IMG_1.jpg", 100, Bucket::Timeline),
photo("IMG_1.jpg", 101, Bucket::Album("Iceland".into())),
],
..Default::default()
};
assert_eq!(plan(&scan).len(), 2);
}
#[test]
fn metadata_from_any_copy_survives_the_fold() {
// Only the album copy carries a sidecar; the fold must not lose it.
let mut bare = photo("IMG_1.jpg", 100, Bucket::Timeline);
bare.capture_time = None;
let mut rich = photo("IMG_1.jpg", 100, Bucket::Album("Iceland".into()));
rich.capture_time = Some(1_560_000_000);
rich.favorite = true;
let planned = plan(&TakeoutScan {
photos: vec![bare, rich],
..Default::default()
});
assert_eq!(planned[0].photo.capture_time, Some(1_560_000_000));
assert!(planned[0].photo.favorite);
}
#[test]
fn sha1_matches_the_known_digest_of_abc() {
let mut reader: &[u8] = b"abc";
assert_eq!(
sha1_of(&mut reader).unwrap(),
"a9993e364706816aba3e25717850c26c9cd0d89d"
);
}
}
+161 -5
View File
@@ -13,14 +13,18 @@
//! always reflects exactly what is in flight — even if the work fails partway //! always reflects exactly what is in flight — even if the work fails partway
//! and unwinds. //! and unwinds.
//! //!
//! The same per-block tick enforces the user's bandwidth limits: every byte a
//! transfer moves is charged to its direction's [`RateLimit`], and the thread
//! moving it sleeps off whatever the charge puts it over the cap.
//!
//! [`Request::GetQueueStatus`]: pdfs_core::control::Request::GetQueueStatus //! [`Request::GetQueueStatus`]: pdfs_core::control::Request::GetQueueStatus
use parking_lot::Mutex; use parking_lot::Mutex;
use std::collections::HashMap; use std::collections::HashMap;
use std::io::{Read, Write}; use std::io::{Read, Write};
use std::sync::Arc; use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::Instant; use std::time::{Duration, Instant};
use pdfs_core::control::{JobItem, TransferDirection, TransferItem}; use pdfs_core::control::{JobItem, TransferDirection, TransferItem};
@@ -52,6 +56,59 @@ pub struct TransferRegistry {
inner: Mutex<HashMap<u64, Arc<Entry>>>, inner: Mutex<HashMap<u64, Arc<Entry>>>,
jobs: Mutex<HashMap<u64, Arc<JobEntry>>>, jobs: Mutex<HashMap<u64, Arc<JobEntry>>>,
next: AtomicU64, next: AtomicU64,
upload_limit: RateLimit,
download_limit: RateLimit,
}
/// How far ahead of its cap a direction may run before it is made to wait, so a
/// small file still goes out at full speed.
const RATE_BURST: Duration = Duration::from_secs(1);
/// A bytes-per-second cap shared by every transfer in one direction; `0` is no
/// cap.
///
/// A virtual clock rather than a refilled bucket: each charge pushes
/// `next_free` forward by the time its bytes take at the cap, so concurrent
/// transfers split the cap between them instead of each getting all of it.
#[derive(Default)]
struct RateLimit {
rate: AtomicU64,
next_free: Mutex<Option<Instant>>,
}
impl RateLimit {
fn set(&self, rate: u64) {
self.rate.store(rate, Ordering::Relaxed);
// A new cap starts from now, not from the debt the old one ran up.
*self.next_free.lock() = None;
}
/// Charge `n` bytes moved at `now`, returning how long the mover must wait
/// to stay under the cap.
fn charge(&self, n: u64, now: Instant) -> Duration {
let rate = self.rate.load(Ordering::Relaxed);
if rate == 0 || n == 0 {
return Duration::ZERO;
}
let mut next_free = self.next_free.lock();
let start = next_free.filter(|t| *t > now).unwrap_or(now);
let done = start + Duration::from_secs_f64(n as f64 / rate as f64);
*next_free = Some(done);
done.saturating_duration_since(now + RATE_BURST)
}
}
/// Sleep off a rate-limit wait. The bulk uploader reads on tokio workers, so
/// the sleep is announced to the runtime there rather than silently holding a
/// worker other tasks — FUSE reads among them — may need.
fn throttle(wait: Duration) {
use tokio::runtime::{Handle, RuntimeFlavor};
match Handle::try_current() {
Ok(handle) if handle.runtime_flavor() == RuntimeFlavor::MultiThread => {
tokio::task::block_in_place(|| std::thread::sleep(wait))
}
_ => std::thread::sleep(wait),
}
} }
impl TransferRegistry { impl TransferRegistry {
@@ -59,6 +116,20 @@ impl TransferRegistry {
Arc::new(Self::default()) Arc::new(Self::default())
} }
/// Cap uploads and downloads at these bytes per second; `0` lifts a cap.
/// Applies to transfers already running from their next block.
pub fn set_limits(&self, upload: u64, download: u64) {
self.upload_limit.set(upload);
self.download_limit.set(download);
}
fn limit(&self, direction: TransferDirection) -> &RateLimit {
match direction {
TransferDirection::Upload => &self.upload_limit,
TransferDirection::Download => &self.download_limit,
}
}
/// Register a new transfer of `total` bytes (`0` = unknown), returning a /// Register a new transfer of `total` bytes (`0` = unknown), returning a
/// guard that tracks progress and deregisters the transfer when dropped. /// guard that tracks progress and deregisters the transfer when dropped.
pub fn begin( pub fn begin(
@@ -169,9 +240,17 @@ pub struct TransferGuard {
} }
impl TransferGuard { impl TransferGuard {
/// Record `n` more bytes moved. /// Record `n` more bytes moved, then wait if they put the direction over
/// its cap.
pub fn add(&self, n: u64) { pub fn add(&self, n: u64) {
self.entry.done.fetch_add(n, Ordering::Relaxed); self.entry.done.fetch_add(n, Ordering::Relaxed);
let wait = self
.reg
.limit(self.entry.direction)
.charge(n, Instant::now());
if !wait.is_zero() {
throttle(wait);
}
} }
} }
@@ -272,22 +351,99 @@ impl<R: Read> Read for OwnedCountingReader<R> {
} }
} }
/// A [`Read`] that tallies bytes read through a [`TransferGuard`] (upload path). /// A [`Read`] that tallies bytes read through a [`TransferGuard`] (upload path),
/// and can be told to stop.
pub struct CountingReader<'a, R> { pub struct CountingReader<'a, R> {
inner: R, inner: R,
guard: &'a TransferGuard, guard: &'a TransferGuard,
cancel: Option<Arc<AtomicBool>>,
} }
impl<'a, R: Read> CountingReader<'a, R> { impl<'a, R: Read> CountingReader<'a, R> {
pub fn new(inner: R, guard: &'a TransferGuard) -> Self { pub fn new(inner: R, guard: &'a TransferGuard) -> Self {
Self { inner, guard } Self {
inner,
guard,
cancel: None,
}
}
/// Abandon the upload as soon as the SDK asks for its next bytes when
/// `cancel` is set.
///
/// This is the only cancellation point an upload has: the SDK takes a plain
/// blocking [`Read`] and pulls from it a block at a time, so refusing to
/// hand over the next block is what stops it. Used when a newer revision of
/// the same file is queued — the bytes on the wire have already been
/// superseded, and finishing them only spends uplink on a revision the next
/// op replaces.
pub fn with_cancel(mut self, cancel: Arc<AtomicBool>) -> Self {
self.cancel = Some(cancel);
self
} }
} }
impl<R: Read> Read for CountingReader<'_, R> { impl<R: Read> Read for CountingReader<'_, R> {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> { fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
if let Some(cancel) = &self.cancel
&& cancel.load(Ordering::Relaxed)
{
// Not `Ok(0)`: a short read is how this reader says "end of file",
// and the SDK would seal a truncated revision from it. Not
// `Interrupted` either: every reader loop, the SDK's included,
// retries that kind, and this flag never clears, so the drain
// thread spun forever with the transfer stuck at 0 bytes (B98).
return Err(std::io::Error::other("upload superseded by a newer write"));
}
let n = self.inner.read(buf)?; let n = self.inner.read(buf)?;
self.guard.add(n as u64); self.guard.add(n as u64);
Ok(n) Ok(n)
} }
} }
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn an_uncapped_direction_never_waits() {
let limit = RateLimit::default();
assert_eq!(limit.charge(u64::MAX / 2, Instant::now()), Duration::ZERO);
}
#[test]
fn a_capped_direction_waits_once_past_its_burst() {
let limit = RateLimit::default();
limit.set(1000);
let now = Instant::now();
// One second of bytes is the burst: no wait yet.
assert_eq!(limit.charge(1000, now), Duration::ZERO);
// Two more seconds' worth at the same instant: wait out all but the burst.
assert_eq!(limit.charge(2000, now), Duration::from_secs(2));
// Lifting the cap drops the debt with it.
limit.set(0);
assert_eq!(limit.charge(5000, now), Duration::ZERO);
}
#[test]
fn concurrent_transfers_share_one_cap() {
let limit = RateLimit::default();
limit.set(1000);
let now = Instant::now();
assert_eq!(limit.charge(1500, now), Duration::from_millis(500));
assert_eq!(limit.charge(1500, now), Duration::from_secs(2));
}
#[test]
fn a_cancelled_upload_fails_its_read_instead_of_asking_for_a_retry() {
let reg = TransferRegistry::new();
let guard = reg.begin("f", "", TransferDirection::Upload, 4);
let cancel = Arc::new(AtomicBool::new(true));
let mut reader = CountingReader::new(&b"data"[..], &guard).with_cancel(cancel);
// `read_exact` retries `Interrupted` the way the SDK's block loop does,
// so this hangs rather than fails if the cancel asks for a retry.
let err = reader.read_exact(&mut [0; 4]).unwrap_err();
assert_ne!(err.kind(), std::io::ErrorKind::Interrupted);
assert_eq!(guard.entry.done.load(Ordering::Relaxed), 0);
}
}
+37 -15
View File
@@ -63,6 +63,15 @@ async fn run_uploads(
.transfers .transfers
.begin(&t.name, "", TransferDirection::Upload, t.size); .begin(&t.name, "", TransferDirection::Upload, t.size);
let reader = OwnedCountingReader::new(file, guard); let reader = OwnedCountingReader::new(file, guard);
if let Err(error) = core.require_uid_writable(&t.parent_uid) {
return Err((
t.name,
format!("destination access denied: errno {}", error.code()),
));
}
let thumbnails = core
.upload_thumbnails_async(t.path.clone(), t.name.clone())
.await;
match core match core
.client .client
.upload_file_from( .upload_file_from(
@@ -71,7 +80,7 @@ async fn run_uploads(
media_type_for(&t.name), media_type_for(&t.name),
reader, reader,
t.size as i64, t.size as i64,
Vec::new(), thumbnails,
mtime, mtime,
false, false,
) )
@@ -110,6 +119,8 @@ impl Core {
let (pino, parent_uid) = self.resolve(parent_rel)?; let (pino, parent_uid) = self.resolve(parent_rel)?;
self.ensure_children(pino) self.ensure_children(pino)
.map_err(|e| self.errno_error(e, "enumerate"))?; .map_err(|e| self.errno_error(e, "enumerate"))?;
self.require_uid_writable(&parent_uid)
.map_err(|error| self.errno_error(error, "upload destination"))?;
// Phase 1 (sequential): build the remote folder skeleton and collect the // Phase 1 (sequential): build the remote folder skeleton and collect the
// flat list of files to upload. Folders must exist before their children, // flat list of files to upload. Folders must exist before their children,
@@ -163,13 +174,15 @@ impl Core {
stats.bytes += size; stats.bytes += size;
match self.fetch_node(&uid) { match self.fetch_node(&uid) {
Ok(node) => { Ok(node) => {
let mut st = self.state.lock(); let mut st = self.state();
let ino = st.intern(parent_ino, node); let ino = st.intern(parent_ino, node);
if let Some(kids) = st.children.get_mut(&parent_ino) if let Some(kids) = st.children.get_mut(&parent_ino)
&& !kids.contains(&ino) && !kids.contains(&ino)
{ {
kids.push(ino); kids.push(ino);
} }
drop(st);
self.flush_access_changes();
} }
// Uploaded fine but the metadata refresh failed; it will // Uploaded fine but the metadata refresh failed; it will
// appear on the next directory enumeration regardless. // appear on the next directory enumeration regardless.
@@ -205,19 +218,24 @@ impl Core {
} }
self.ensure_children(pino) self.ensure_children(pino)
.map_err(|e| self.errno_error(e, "enumerate"))?; .map_err(|e| self.errno_error(e, "enumerate"))?;
{ let existing = {
let st = self.state.lock(); let st = self.state();
if let Some(kids) = st.children.get(&pino) { st.children.get(&pino).and_then(|kids| {
for &ino in kids { kids.iter().find_map(|ino| {
if let Some(e) = st.entries.get(&ino) st.entries
&& e.node.is_folder() .get(ino)
&& e.node.name == name .filter(|entry| entry.node.is_folder() && entry.node.name == name)
{ .map(|entry| (*ino, entry.uid.clone()))
return Ok((ino, e.uid.clone())); })
} })
} };
} if let Some((ino, uid)) = existing {
self.require_uid_writable(&uid)
.map_err(|error| self.errno_error(error, "upload destination"))?;
return Ok((ino, uid));
} }
self.require_uid_writable(parent_uid)
.map_err(|error| self.errno_error(error, "upload destination"))?;
let new_uid = self let new_uid = self
.rt .rt
.block_on( .block_on(
@@ -228,13 +246,17 @@ impl Core {
let node = self let node = self
.fetch_node(&new_uid) .fetch_node(&new_uid)
.map_err(|e| self.errno_error(e, "fetch node"))?; .map_err(|e| self.errno_error(e, "fetch node"))?;
let mut st = self.state.lock(); let mut st = self.state();
let ino = st.intern(pino, node); let ino = st.intern(pino, node);
if let Some(kids) = st.children.get_mut(&pino) if let Some(kids) = st.children.get_mut(&pino)
&& !kids.contains(&ino) && !kids.contains(&ino)
{ {
kids.push(ino); kids.push(ino);
} }
drop(st);
self.flush_access_changes();
self.require_uid_writable(&new_uid)
.map_err(|error| self.errno_error(error, "created upload destination"))?;
Ok((ino, new_uid)) Ok((ino, new_uid))
} }
+323
View File
@@ -0,0 +1,323 @@
//! Synthetic nodes and naming rules for the mounted shared tree.
use std::collections::{HashMap, HashSet};
use std::time::{Duration, Instant};
use proton_drive_rs::proton_sdk::ids::{LinkId, NodeUid, VolumeId};
use proton_drive_rs::{Node, NodeKind};
pub(crate) const VIRTUAL_VOLUME: &str = "virtual";
pub(crate) const SHARED_WITH_ME_LINK: &str = "sharedwithme";
pub(crate) const SHARED_WITH_ME_BASE: &str = "Shared with me";
const MAX_COMPONENT_BYTES: usize = 255;
pub(crate) fn shared_with_me_uid() -> NodeUid {
NodeUid::new(
VolumeId::from(VIRTUAL_VOLUME),
LinkId::from(SHARED_WITH_ME_LINK),
)
}
pub(crate) fn is_virtual_uid(uid: &NodeUid) -> bool {
uid.volume_id.as_str() == VIRTUAL_VOLUME
}
pub(crate) fn is_own_or_virtual_uid(uid: &NodeUid, own_volume: &VolumeId) -> bool {
uid.volume_id == *own_volume || is_virtual_uid(uid)
}
pub(crate) fn is_primary_root_listing(
primary: bool,
folder_uid: &NodeUid,
primary_root_uid: &NodeUid,
) -> bool {
primary && folder_uid == primary_root_uid
}
pub(crate) fn virtual_node(parent_uid: NodeUid, name: String, timestamp: i64) -> Node {
Node {
uid: shared_with_me_uid(),
parent_uid: Some(parent_uid),
name,
kind: NodeKind::Folder,
creation_time: timestamp,
modification_time: timestamp,
trashed: false,
is_shared: false,
is_shared_publicly: false,
signature_email: None,
membership: None,
// A synthetic folder is neither a photo nor an album.
photo: None,
album: None,
verification: Default::default(),
direct_role: None,
share_id: None,
}
}
/// Choose the initial synthetic-root name. Once persisted, `pinned` always wins;
/// a later real collision only controls whether the synthetic dentry is exposed.
pub(crate) fn virtual_root_name(
real_names: &HashSet<String>,
pinned: Option<&str>,
) -> (String, bool) {
if let Some(pinned) = pinned {
return (pinned.to_string(), !real_names.contains(pinned));
}
for n in 0usize.. {
let candidate = match n {
0 => SHARED_WITH_ME_BASE.to_string(),
1 => format!("{SHARED_WITH_ME_BASE} (Proton)"),
_ => format!("{SHARED_WITH_ME_BASE} (Proton {n})"),
};
if !real_names.contains(&candidate) {
return (candidate, true);
}
}
unreachable!("the numbered virtual-root namespace is unbounded")
}
/// Give every shared root a unique Linux component name, independent of API
/// ordering. Literal suffix-shaped names reserve their basename before duplicate
/// names are numbered, so `Photos (2)` is never stolen by a second `Photos`.
pub(crate) fn disambiguate_shared_names(nodes: &mut [Node]) {
nodes.sort_by(|a, b| {
a.name
.cmp(&b.name)
.then_with(|| a.uid.to_string().cmp(&b.uid.to_string()))
});
let reserved: HashSet<String> = nodes
.iter()
.map(|node| component_with_suffix(&node.name, ""))
.collect();
let mut used = HashSet::with_capacity(nodes.len());
for node in nodes {
let original = node.name.clone();
let basename = component_with_suffix(&original, "");
let chosen = if used.insert(basename.clone()) {
basename
} else {
let mut n = 2usize;
loop {
let suffix = format!(" ({n})");
let candidate = component_with_suffix(&original, &suffix);
if !reserved.contains(&candidate) && used.insert(candidate.clone()) {
break candidate;
}
n += 1;
}
};
node.name = chosen;
}
}
pub(crate) fn listing_needs_refresh(online: bool, stale: bool) -> bool {
online && stale
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum SharedListingPlan {
Resident,
Persisted,
Refresh,
}
pub(crate) fn shared_listing_plan(resident: bool, online: bool, stale: bool) -> SharedListingPlan {
if listing_needs_refresh(online, stale) {
SharedListingPlan::Refresh
} else if resident {
SharedListingPlan::Resident
} else {
SharedListingPlan::Persisted
}
}
pub(crate) fn refresh_generation_is_current(started: u64, current: u64) -> bool {
started == current
}
#[derive(Default)]
pub(crate) struct SharedRefreshDeadlines {
deadlines: HashMap<String, Instant>,
}
impl SharedRefreshDeadlines {
pub(crate) fn is_fresh(&mut self, key: &str, now: Instant) -> bool {
self.deadlines.retain(|_, deadline| *deadline > now);
self.deadlines
.get(key)
.is_some_and(|deadline| *deadline > now)
}
pub(crate) fn mark(&mut self, key: &str, now: Instant, ttl: Duration) {
self.deadlines.insert(key.to_string(), now + ttl);
}
pub(crate) fn clear(&mut self) {
self.deadlines.clear();
}
}
fn component_with_suffix(name: &str, suffix: &str) -> String {
let keep = MAX_COMPONENT_BYTES.saturating_sub(suffix.len());
let mut boundary = name.len().min(keep);
while !name.is_char_boundary(boundary) {
boundary -= 1;
}
let mut out = String::with_capacity(boundary + suffix.len());
out.push_str(&name[..boundary]);
out.push_str(suffix);
out
}
#[cfg(test)]
mod tests {
use super::*;
fn folder(volume: &str, link: &str, name: &str) -> Node {
let mut node = virtual_node(
NodeUid::new(VolumeId::from("virtual"), LinkId::from("parent")),
name.to_string(),
1,
);
node.uid = NodeUid::new(VolumeId::from(volume), LinkId::from(link));
node
}
#[test]
fn virtual_uid_and_node_round_trip() {
let parent = NodeUid::new(VolumeId::from("own"), LinkId::from("root"));
let node = virtual_node(parent.clone(), SHARED_WITH_ME_BASE.into(), 42);
assert_eq!(node.uid.to_string(), "virtual~sharedwithme");
assert!(is_virtual_uid(&node.uid));
assert_eq!(node.parent_uid.as_ref(), Some(&parent));
assert!(node.is_folder());
let json = serde_json::to_string(&node).unwrap();
let restored: Node = serde_json::from_str(&json).unwrap();
assert_eq!(restored.uid, node.uid);
assert_eq!(restored.name, SHARED_WITH_ME_BASE);
}
#[test]
fn root_collision_ladder_and_pinned_collision_policy() {
let names = HashSet::from([
SHARED_WITH_ME_BASE.to_string(),
format!("{SHARED_WITH_ME_BASE} (Proton)"),
format!("{SHARED_WITH_ME_BASE} (Proton 2)"),
]);
assert_eq!(
virtual_root_name(&names, None),
(format!("{SHARED_WITH_ME_BASE} (Proton 3)"), true)
);
let (name, visible) = virtual_root_name(&names, Some(SHARED_WITH_ME_BASE));
assert_eq!(name, SHARED_WITH_ME_BASE);
assert!(!visible, "a real later collision suppresses the dentry");
let cleared = HashSet::new();
assert_eq!(
virtual_root_name(&cleared, Some(SHARED_WITH_ME_BASE)),
(SHARED_WITH_ME_BASE.to_string(), true),
"the pinned name returns when the collision clears"
);
}
#[test]
fn duplicate_names_are_stable_under_reordering_and_reserve_suffixes() {
let input = vec![
folder("b", "2", "Photos"),
folder("c", "3", "Photos (2)"),
folder("a", "1", "Photos"),
];
let mut forward = input.clone();
let mut reverse = input;
reverse.reverse();
disambiguate_shared_names(&mut forward);
disambiguate_shared_names(&mut reverse);
let summarize = |nodes: &[Node]| {
nodes
.iter()
.map(|node| (node.uid.to_string(), node.name.clone()))
.collect::<Vec<_>>()
};
assert_eq!(summarize(&forward), summarize(&reverse));
assert_eq!(
forward
.iter()
.map(|node| node.name.as_str())
.collect::<Vec<_>>(),
["Photos", "Photos (3)", "Photos (2)"]
);
}
#[test]
fn disambiguation_respects_utf8_component_limit() {
let long = "é".repeat(200);
let mut nodes = vec![folder("a", "1", &long), folder("b", "2", &long)];
disambiguate_shared_names(&mut nodes);
assert!(nodes.iter().all(|node| node.name.len() <= 255));
assert!(
nodes
.iter()
.all(|node| node.name.is_char_boundary(node.name.len()))
);
assert_ne!(nodes[0].name, nodes[1].name);
assert!(nodes[1].name.ends_with(" (2)"));
}
#[test]
fn ttl_refresh_only_happens_online() {
assert!(!listing_needs_refresh(false, true));
assert!(!listing_needs_refresh(true, false));
assert!(listing_needs_refresh(true, true));
}
#[test]
fn offline_snapshot_suppresses_the_api_and_memory_deadline_suppresses_retries() {
assert_eq!(
shared_listing_plan(false, false, true),
SharedListingPlan::Persisted
);
assert_eq!(
shared_listing_plan(true, true, false),
SharedListingPlan::Resident
);
let now = Instant::now();
let mut deadlines = SharedRefreshDeadlines::default();
deadlines.mark("shared", now, Duration::from_secs(60));
assert!(deadlines.is_fresh("shared", now + Duration::from_secs(30)));
deadlines.clear();
assert!(!deadlines.is_fresh("shared", now + Duration::from_secs(30)));
}
#[test]
fn generation_change_rejects_an_inflight_response() {
assert!(refresh_generation_is_current(7, 7));
assert!(!refresh_generation_is_current(7, 8));
}
#[test]
fn only_the_primary_inode_space_installs_the_synthetic_root() {
let root = NodeUid::new(VolumeId::from("own"), LinkId::from("root"));
assert!(is_primary_root_listing(true, &root, &root));
assert!(!is_primary_root_listing(false, &root, &root));
assert!(!is_primary_root_listing(
true,
&NodeUid::new(VolumeId::from("own"), LinkId::from("device")),
&root
));
}
#[test]
fn ownership_predicate_excludes_foreign_share_volumes() {
let own = VolumeId::from("own");
let own_uid = NodeUid::new(own.clone(), LinkId::from("file"));
let foreign = NodeUid::new(VolumeId::from("foreign"), LinkId::from("share"));
assert!(is_own_or_virtual_uid(&own_uid, &own));
assert!(is_own_or_virtual_uid(&shared_with_me_uid(), &own));
assert!(!is_own_or_virtual_uid(&foreign, &own));
}
}
+333 -33
View File
@@ -12,6 +12,8 @@
use std::collections::VecDeque; use std::collections::VecDeque;
use std::sync::Arc; use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
use std::time::Instant;
use parking_lot::{Condvar, Mutex}; use parking_lot::{Condvar, Mutex};
use tracing::warn; use tracing::warn;
@@ -51,8 +53,25 @@ pub(crate) const FUSE_WORKERS: usize = 11;
/// exactly the blocking this split exists to prevent. /// exactly the blocking this split exists to prevent.
const META_WORKERS: usize = 3; const META_WORKERS: usize = 3;
/// Queue depth that gets a line in the log, and the step at which it repeats.
/// Every job holds a `Reply` and a few fields, so depth is cheap — but a lane
/// this far behind is worth knowing about before the user reports a slow mount.
const QUEUE_WARN_STEP: usize = 256;
/// Whether a queue reaching `depth` is one of the depths worth logging.
fn crosses_warn_step(depth: usize) -> bool {
depth >= QUEUE_WARN_STEP && depth.is_multiple_of(QUEUE_WARN_STEP)
}
type Job = Box<dyn FnOnce() + Send + 'static>; type Job = Box<dyn FnOnce() + Send + 'static>;
/// A queued job and the name its caller gave it, so a job that never finishes
/// can be named in the diagnostics instead of showing up as "a worker is busy".
struct Queued {
label: &'static str,
job: Job,
}
/// Which lane a job belongs in. /// Which lane a job belongs in.
#[derive(Clone, Copy, PartialEq, Eq)] #[derive(Clone, Copy, PartialEq, Eq)]
pub(crate) enum Lane { pub(crate) enum Lane {
@@ -65,12 +84,63 @@ pub(crate) enum Lane {
#[derive(Default)] #[derive(Default)]
struct Queues { struct Queues {
meta: VecDeque<Job>, meta: VecDeque<Queued>,
transfer: VecDeque<Job>, transfer: VecDeque<Queued>,
/// Set when the pool is dropped, to wake and retire every worker. /// Set when the pool is dropped, to wake and retire every worker.
closed: bool, closed: bool,
} }
/// What one worker thread is doing, published for [`Workers::snapshot`].
///
/// Every field is an atomic (or a lock the worker only ever holds for the length
/// of a pointer write) because the whole point of the snapshot is to answer
/// *while the daemon is wedged*: a diagnostic that waits on the same lock the
/// stuck job holds reports nothing at exactly the moment it is needed.
struct Slot {
name: String,
/// True between picking a job up and finishing it.
busy: AtomicBool,
/// When the current state (busy or idle) began, as milliseconds since the
/// pool started. Monotonic, so an age is always a real duration.
since_ms: AtomicU64,
/// The label of the job in hand. Behind a mutex because a `&'static str` is
/// two words and there is no atomic for that; the worker holds it only to
/// swap the pointer, and the snapshot never blocks on it (see
/// [`Workers::snapshot`]).
label: Mutex<&'static str>,
}
impl Slot {
fn new(name: String, now_ms: u64) -> Self {
Self {
name,
busy: AtomicBool::new(false),
since_ms: AtomicU64::new(now_ms),
label: Mutex::new(""),
}
}
}
/// One worker's state at the moment [`Workers::snapshot`] read it.
pub(crate) struct WorkerSnapshot {
pub(crate) name: String,
pub(crate) busy: bool,
/// How long the worker has been in that state.
pub(crate) age_ms: u64,
/// The job in hand, or `""` when idle. `"?"` when the label could not be
/// read without blocking.
pub(crate) label: String,
}
/// The pool's state at the moment [`Workers::snapshot`] read it.
pub(crate) struct PoolSnapshot {
pub(crate) workers: Vec<WorkerSnapshot>,
pub(crate) meta_queued: usize,
pub(crate) transfer_queued: usize,
pub(crate) meta_completed: u64,
pub(crate) transfer_completed: u64,
}
/// The queues and the two wait sets over them. /// The queues and the two wait sets over them.
/// ///
/// Two condvars, not one, because the workers are not interchangeable: a /// Two condvars, not one, because the workers are not interchangeable: a
@@ -86,6 +156,22 @@ struct Pool {
meta_cv: Condvar, meta_cv: Condvar,
/// Waited on by the general workers. /// Waited on by the general workers.
general_cv: Condvar, general_cv: Condvar,
/// Queue depths, mirrored outside the lock so a snapshot can read them
/// without waiting on whatever is holding it.
meta_queued: AtomicUsize,
transfer_queued: AtomicUsize,
/// Jobs finished per lane. A depth that stays high while this stays still is
/// a stalled lane rather than a busy one.
meta_completed: AtomicU64,
transfer_completed: AtomicU64,
slots: Vec<Slot>,
started: Instant,
}
impl Pool {
fn now_ms(&self) -> u64 {
self.started.elapsed().as_millis() as u64
}
} }
/// Bounded thread pool behind the network-touching FUSE handlers. /// Bounded thread pool behind the network-touching FUSE handlers.
@@ -94,25 +180,46 @@ struct Pool {
/// on-demand sync folder), so the bound is per daemon rather than per mount. /// on-demand sync folder), so the bound is per daemon rather than per mount.
pub(crate) struct Workers { pub(crate) struct Workers {
inner: Arc<Pool>, inner: Arc<Pool>,
/// Kept so teardown can *join* the workers, not merely flag them. A worker
/// still inside a job when the process drops its tokio runtime panics with
/// "A Tokio 1.x context was found, but it is being shutdown" (seen
/// 2026-09-18 09:58:35), which is a teardown ordering bug rather than a real
/// failure.
handles: Mutex<Vec<std::thread::JoinHandle<()>>>,
} }
impl Workers { impl Workers {
pub(crate) fn new(n: usize) -> std::io::Result<Self> { pub(crate) fn new(n: usize) -> std::io::Result<Self> {
// Never leave the pool without a general worker, however small `n` is.
let meta_workers = META_WORKERS.min(n.saturating_sub(1));
let names: Vec<String> = (0..n)
.map(|i| {
format!(
"pdfs-fuse-{}{i}",
if i < meta_workers { "meta-" } else { "" }
)
})
.collect();
let inner = Arc::new(Pool { let inner = Arc::new(Pool {
queues: Mutex::new(Queues::default()), queues: Mutex::new(Queues::default()),
meta_cv: Condvar::new(), meta_cv: Condvar::new(),
general_cv: Condvar::new(), general_cv: Condvar::new(),
meta_queued: AtomicUsize::new(0),
transfer_queued: AtomicUsize::new(0),
meta_completed: AtomicU64::new(0),
transfer_completed: AtomicU64::new(0),
slots: names
.iter()
.map(|name| Slot::new(name.clone(), 0))
.collect(),
started: Instant::now(),
}); });
// Never leave the pool without a general worker, however small `n` is. let mut handles = Vec::with_capacity(n);
let meta_workers = META_WORKERS.min(n.saturating_sub(1)); for (i, name) in names.iter().enumerate() {
for i in 0..n {
let inner = inner.clone(); let inner = inner.clone();
let meta_only = i < meta_workers; let meta_only = i < meta_workers;
std::thread::Builder::new() let handle = std::thread::Builder::new()
.name(format!( .name(name.clone())
"pdfs-fuse-{}{i}",
if meta_only { "meta-" } else { "" }
))
.spawn(move || { .spawn(move || {
let cv = if meta_only { let cv = if meta_only {
&inner.meta_cv &inner.meta_cv
@@ -121,17 +228,20 @@ impl Workers {
}; };
loop { loop {
let mut q = inner.queues.lock(); let mut q = inner.queues.lock();
let job = loop { let picked = loop {
// A general worker takes transfers first: metadata // A general worker takes transfers first: metadata
// already has threads of its own, so draining the // already has threads of its own, so draining the
// bulk queue is the useful thing for it to do. // bulk queue is the useful thing for it to do.
let picked = if meta_only { let taken = if meta_only {
q.meta.pop_front() q.meta.pop_front().map(|job| (Lane::Meta, job))
} else { } else {
q.transfer.pop_front().or_else(|| q.meta.pop_front()) q.transfer
.pop_front()
.map(|job| (Lane::Transfer, job))
.or_else(|| q.meta.pop_front().map(|job| (Lane::Meta, job)))
}; };
if let Some(job) = picked { if let Some(taken) = taken {
break Some(job); break Some(taken);
} }
if q.closed { if q.closed {
break None; break None;
@@ -139,7 +249,15 @@ impl Workers {
cv.wait(&mut q); cv.wait(&mut q);
}; };
drop(q); drop(q);
let Some(job) = job else { break }; let Some((lane, queued)) = picked else { break };
match lane {
Lane::Meta => inner.meta_queued.fetch_sub(1, Ordering::Relaxed),
Lane::Transfer => inner.transfer_queued.fetch_sub(1, Ordering::Relaxed),
};
let slot = &inner.slots[i];
*slot.label.lock() = queued.label;
slot.since_ms.store(inner.now_ms(), Ordering::Relaxed);
slot.busy.store(true, Ordering::Relaxed);
// A panicking handler must not cost the pool a worker // A panicking handler must not cost the pool a worker
// for the rest of the run. The dropped `Reply` answers // for the rest of the run. The dropped `Reply` answers
// EIO on its own, so the caller of the failed op is // EIO on its own, so the caller of the failed op is
@@ -164,11 +282,70 @@ impl Workers {
// poisoning at all. Anything new reached from a job owes // poisoning at all. Anything new reached from a job owes
// the same check — an SDK-side `.lock().unwrap()` // the same check — an SDK-side `.lock().unwrap()`
// silently invalidates this comment. // silently invalidates this comment.
let _ = std::panic::catch_unwind(std::panic::AssertUnwindSafe(job)); let _ = std::panic::catch_unwind(std::panic::AssertUnwindSafe(queued.job));
slot.busy.store(false, Ordering::Relaxed);
slot.since_ms.store(inner.now_ms(), Ordering::Relaxed);
*slot.label.lock() = "";
match lane {
Lane::Meta => inner.meta_completed.fetch_add(1, Ordering::Relaxed),
Lane::Transfer => {
inner.transfer_completed.fetch_add(1, Ordering::Relaxed)
}
};
} }
})?; })?;
handles.push(handle);
} }
Ok(Self { inner }) Ok(Self {
inner,
handles: Mutex::new(handles),
})
}
/// Stop accepting work, wake every worker, and wait up to `deadline` for the
/// jobs in flight to finish.
///
/// Bounded because a join is only worth what it costs: a worker wedged on a
/// network call would otherwise hold the whole teardown, and the user asked
/// for the daemon to stop. What is stuck gets named in the log, and the
/// process continues shutting down around it.
pub(crate) fn stop_and_join(&self, deadline: std::time::Duration) {
self.close();
let handles: Vec<_> = std::mem::take(&mut *self.handles.lock());
if handles.is_empty() {
return;
}
let (tx, rx) = std::sync::mpsc::channel();
let joiner = std::thread::Builder::new()
.name("pdfs-fuse-join".into())
.spawn(move || {
for handle in handles {
let _ = handle.join();
}
let _ = tx.send(());
});
if joiner.is_err() {
return;
}
if rx.recv_timeout(deadline).is_err() {
let busy: Vec<String> = self
.snapshot()
.workers
.into_iter()
.filter(|worker| worker.busy)
.map(|worker| format!("{}={}", worker.name, worker.label))
.collect();
warn!(
stuck = busy.join(","),
"fuse workers did not finish in time; shutting down around them"
);
}
}
fn close(&self) {
self.inner.queues.lock().closed = true;
self.inner.meta_cv.notify_all();
self.inner.general_cv.notify_all();
} }
/// Queue `job` in `lane`. /// Queue `job` in `lane`.
@@ -179,7 +356,15 @@ impl Workers {
/// a pending job holds a `Reply` and a few fields, while the 4 MiB block /// a pending job holds a `Reply` and a few fields, while the 4 MiB block
/// buffer is allocated inside the job once it runs, and the SDK's in-flight /// buffer is allocated inside the job once it runs, and the SDK's in-flight
/// semaphore is what bounds how many of those exist at once. /// semaphore is what bounds how many of those exist at once.
pub(crate) fn run(&self, lane: Lane, job: impl FnOnce() + Send + 'static) { /// The queue is *not* bounded, and the depth warning above is deliberately
/// all this does about it: the only caller able to feel backpressure here is
/// fuser's dispatch loop, and blocking that is the stall the whole module
/// exists to avoid. A depth that keeps climbing is a symptom to read in the
/// log, not something to fix by stopping the mount.
///
/// `label` names the work for the diagnostics (`pdfs diagnostics`) and for
/// the stall warnings: it is what a hung daemon reports instead of "busy".
pub(crate) fn run(&self, lane: Lane, label: &'static str, job: impl FnOnce() + Send + 'static) {
let mut q = self.inner.queues.lock(); let mut q = self.inner.queues.lock();
if q.closed { if q.closed {
// Pre-pool behaviour: a shut-down pool degrades to a slow mount // Pre-pool behaviour: a shut-down pool degrades to a slow mount
@@ -189,11 +374,31 @@ impl Workers {
job(); job();
return; return;
} }
match lane { let queued = Queued {
Lane::Meta => q.meta.push_back(Box::new(job)), label,
Lane::Transfer => q.transfer.push_back(Box::new(job)), job: Box::new(job),
} };
let depth = match lane {
Lane::Meta => {
q.meta.push_back(queued);
self.inner.meta_queued.fetch_add(1, Ordering::Relaxed) + 1
}
Lane::Transfer => {
q.transfer.push_back(queued);
self.inner.transfer_queued.fetch_add(1, Ordering::Relaxed) + 1
}
};
drop(q); drop(q);
if crosses_warn_step(depth) {
warn!(
lane = if matches!(lane, Lane::Meta) {
"meta"
} else {
"transfer"
},
depth, label, "fuse worker queue is deep"
);
}
match lane { match lane {
// Either class can serve metadata, and only one of them needs to: // Either class can serve metadata, and only one of them needs to:
// whichever wakes first takes it, and the other finds the queue // whichever wakes first takes it, and the other finds the queue
@@ -209,13 +414,50 @@ impl Workers {
} }
} }
} }
/// What every worker is doing right now, without taking the queue lock.
///
/// A diagnostic is worth having exactly when the daemon is stuck, so this
/// reads atomics only, and `try_lock`s each label rather than waiting for
/// it: a label is held for one pointer write, so failing to get it means a
/// racing worker, not a wedged one, and `"?"` is a better answer than a
/// diagnostics request that hangs too.
pub(crate) fn snapshot(&self) -> PoolSnapshot {
let now = self.inner.now_ms();
let workers = self
.inner
.slots
.iter()
.map(|slot| {
let label = slot
.label
.try_lock()
.map_or_else(|| "?".to_string(), |label| (*label).to_string());
WorkerSnapshot {
name: slot.name.clone(),
busy: slot.busy.load(Ordering::Relaxed),
age_ms: now.saturating_sub(slot.since_ms.load(Ordering::Relaxed)),
label,
}
})
.collect();
PoolSnapshot {
workers,
meta_queued: self.inner.meta_queued.load(Ordering::Relaxed),
transfer_queued: self.inner.transfer_queued.load(Ordering::Relaxed),
meta_completed: self.inner.meta_completed.load(Ordering::Relaxed),
transfer_completed: self.inner.transfer_completed.load(Ordering::Relaxed),
}
}
} }
impl Drop for Workers { impl Drop for Workers {
/// A backstop for the paths that never call
/// [`stop_and_join`](Workers::stop_and_join) — tests, and a mount that fails
/// before teardown is wired up. Flag and wake only: a `Drop` that blocks on
/// a network call is worse than a worker outliving its pool by a moment.
fn drop(&mut self) { fn drop(&mut self) {
self.inner.queues.lock().closed = true; self.close();
self.inner.meta_cv.notify_all();
self.inner.general_cv.notify_all();
} }
} }
@@ -225,6 +467,64 @@ mod tests {
use std::sync::mpsc; use std::sync::mpsc;
use std::time::Duration; use std::time::Duration;
/// The depth warning fires on the step, and only on the step: a lane that
/// oscillates around the threshold must not log on every single job.
#[test]
fn queue_depth_warns_on_each_step() {
assert!(!crosses_warn_step(1));
assert!(!crosses_warn_step(QUEUE_WARN_STEP - 1));
assert!(crosses_warn_step(QUEUE_WARN_STEP));
assert!(!crosses_warn_step(QUEUE_WARN_STEP + 1));
assert!(crosses_warn_step(QUEUE_WARN_STEP * 3));
}
/// Teardown must return even while a job is running, and must not leave the
/// pool accepting work.
#[test]
fn stopping_is_bounded_and_closes_the_pool() {
let pool = Workers::new(2).unwrap();
let (tx, rx) = mpsc::channel();
pool.run(Lane::Meta, "test", move || tx.send(()).unwrap());
rx.recv_timeout(Duration::from_secs(5)).expect("it runs");
pool.stop_and_join(Duration::from_secs(5));
// A job queued after the stop runs inline rather than disappearing.
let (tx, rx) = mpsc::channel();
pool.run(Lane::Meta, "test", move || tx.send(()).unwrap());
rx.recv_timeout(Duration::from_secs(5))
.expect("a closed pool serves inline");
}
/// A diagnostic is only useful if it names the work. A running job must show
/// up under its label, and the queue behind it must show up as depth.
#[test]
fn a_snapshot_names_the_running_job_and_the_queue_behind_it() {
let pool = Workers::new(2).unwrap();
let (release_tx, release_rx) = mpsc::channel::<()>();
let release_rx = Arc::new(Mutex::new(release_rx));
let (started_tx, started_rx) = mpsc::channel();
{
let release_rx = release_rx.clone();
pool.run(Lane::Transfer, "read", move || {
started_tx.send(()).unwrap();
let _ = release_rx.lock().recv();
});
}
started_rx
.recv_timeout(Duration::from_secs(5))
.expect("the job starts");
let snapshot = pool.snapshot();
assert!(
snapshot.workers.iter().any(|w| w.busy && w.label == "read"),
"the running job is named"
);
assert_eq!(snapshot.transfer_completed, 0);
let _ = release_tx.send(());
}
/// Audit A6. Transfers filling every general worker must not delay a /// Audit A6. Transfers filling every general worker must not delay a
/// metadata job: that is what the reserved threads are for. /// metadata job: that is what the reserved threads are for.
#[test] #[test]
@@ -240,7 +540,7 @@ mod tests {
for _ in 0..general { for _ in 0..general {
let release_rx = release_rx.clone(); let release_rx = release_rx.clone();
let started_tx = started_tx.clone(); let started_tx = started_tx.clone();
pool.run(Lane::Transfer, move || { pool.run(Lane::Transfer, "test", move || {
started_tx.send(()).unwrap(); started_tx.send(()).unwrap();
let _ = release_rx.lock().recv(); let _ = release_rx.lock().recv();
}); });
@@ -255,13 +555,13 @@ mod tests {
// in the way that used to starve metadata. // in the way that used to starve metadata.
for _ in 0..16 { for _ in 0..16 {
let release_rx = release_rx.clone(); let release_rx = release_rx.clone();
pool.run(Lane::Transfer, move || { pool.run(Lane::Transfer, "test", move || {
let _ = release_rx.lock().recv(); let _ = release_rx.lock().recv();
}); });
} }
let (meta_tx, meta_rx) = mpsc::channel(); let (meta_tx, meta_rx) = mpsc::channel();
pool.run(Lane::Meta, move || meta_tx.send(()).unwrap()); pool.run(Lane::Meta, "test", move || meta_tx.send(()).unwrap());
meta_rx meta_rx
.recv_timeout(Duration::from_secs(5)) .recv_timeout(Duration::from_secs(5))
.expect("a metadata job runs while every transfer thread is stuck"); .expect("a metadata job runs while every transfer thread is stuck");
@@ -279,7 +579,7 @@ mod tests {
// must have been served by the general worker. // must have been served by the general worker.
let pool = Workers::new(1).unwrap(); let pool = Workers::new(1).unwrap();
let (tx, rx) = mpsc::channel(); let (tx, rx) = mpsc::channel();
pool.run(Lane::Meta, move || tx.send(()).unwrap()); pool.run(Lane::Meta, "test", move || tx.send(()).unwrap());
rx.recv_timeout(Duration::from_secs(5)) rx.recv_timeout(Duration::from_secs(5))
.expect("a general worker takes metadata when no transfer is waiting"); .expect("a general worker takes metadata when no transfer is waiting");
} }
@@ -288,9 +588,9 @@ mod tests {
#[test] #[test]
fn a_panicking_job_does_not_retire_its_worker() { fn a_panicking_job_does_not_retire_its_worker() {
let pool = Workers::new(1).unwrap(); let pool = Workers::new(1).unwrap();
pool.run(Lane::Meta, || panic!("boom")); pool.run(Lane::Meta, "test", || panic!("boom"));
let (tx, rx) = mpsc::channel(); let (tx, rx) = mpsc::channel();
pool.run(Lane::Meta, move || tx.send(()).unwrap()); pool.run(Lane::Meta, "test", move || tx.send(()).unwrap());
rx.recv_timeout(Duration::from_secs(5)) rx.recv_timeout(Duration::from_secs(5))
.expect("the pool still serves work after a panic"); .expect("the pool still serves work after a panic");
} }
+14 -4
View File
@@ -5,6 +5,9 @@ edition.workspace = true
license.workspace = true license.workspace = true
authors.workspace = true authors.workspace = true
rust-version.workspace = true rust-version.workspace = true
repository.workspace = true
homepage.workspace = true
description = "GTK4 desktop app, tray, and search launcher for the Proton Drive Linux client"
[[bin]] [[bin]]
name = "pdfs-tray" name = "pdfs-tray"
@@ -24,10 +27,17 @@ tracing.workspace = true
tracing-subscriber.workspace = true tracing-subscriber.workspace = true
anyhow.workspace = true anyhow.workspace = true
tokio.workspace = true tokio.workspace = true
serde.workspace = true
serde_json.workspace = true serde_json.workspace = true
ksni = "0.2" ksni = { version = "0.3", features = ["blocking"] }
gtk4 = { version = "0.9", features = ["v4_12"] } gtk4 = { version = "0.11", features = ["v4_12"] }
adw = { package = "libadwaita", version = "0.7", features = ["v1_5"] } adw = { package = "libadwaita", version = "0.9", features = ["v1_5"] }
async-channel = "2" async-channel = "2"
exif = { version = "0.6", package = "kamadak-exif" } exif = { version = "0.6", package = "kamadak-exif" }
webkit6 = "0.4" webkit6 = "0.6"
# Town and street names on the Places map, with font fallback for every script.
pangocairo = "0.22"
# Map tiles for the opt-in online map. The same build proton-sdk already pulls in.
reqwest = { version = "0.13", default-features = false, features = ["rustls", "webpki-roots"] }
# The system libintl (glibc) rather than a bundled gettext build.
gettext-rs = { version = "0.7", features = ["gettext-system"] }
+6 -3
View File
@@ -9,6 +9,8 @@ fn main() {
&format!("{}/pdfs.gresource", out_dir), &format!("{}/pdfs.gresource", out_dir),
"--sourcedir", "--sourcedir",
"resources/icons", "resources/icons",
"--sourcedir",
"resources",
"resources/pdfs.gresource.xml", "resources/pdfs.gresource.xml",
]) ])
.status() .status()
@@ -22,7 +24,8 @@ fn main() {
} }
println!("cargo:rerun-if-changed=resources/pdfs.gresource.xml"); println!("cargo:rerun-if-changed=resources/pdfs.gresource.xml");
println!( println!("cargo:rerun-if-changed=resources/icons");
"cargo:rerun-if-changed=resources/icons/scalable/actions/pdfs-folder-upload-symbolic.svg" println!("cargo:rerun-if-changed=resources/style.css");
); println!("cargo:rerun-if-changed=resources/proton-theme.css");
println!("cargo:rerun-if-changed=../../packaging/io.narl.proton-drive-linux.svg");
} }
+393
View File
@@ -0,0 +1,393 @@
47.98,8.00 48.94,9.45 48.94,11.26
47.98,8.00 46.42,6.50 44.94,4.91 43.99,4.95 43.58,4.85 43.13,4.64 42.86,4.32 42.02,4.14 41.88,3.98
34.25,31.21 34.53,31.53 34.48,31.58
35.55,32.40 35.19,32.53 35.07,32.46 34.96,32.16 34.95,31.84 35.20,31.75 34.95,31.60 34.87,31.40 34.91,31.35 35.10,31.37 35.45,31.48
35.11,33.08 35.41,33.08 35.49,33.12 35.53,33.25 35.60,33.24
35.79,32.73 35.89,32.94 35.79,33.29 35.84,33.42
34.00,35.07 33.87,35.09 33.48,35.00 33.38,35.16 33.19,35.17 32.92,35.09 32.71,35.17
77.80,35.50 77.89,35.45 78.04,35.48 78.01,35.25 78.33,34.61 78.67,34.52 78.97,34.30 78.97,34.23 78.73,34.01 78.80,33.50 78.92,33.39
77.05,35.11 77.00,34.99 76.59,34.74 76.04,34.67 75.71,34.50 75.32,34.58
91.63,27.76 91.98,27.73 92.10,27.81 92.55,27.88 92.69,27.99 92.64,28.06 92.70,28.15 93.17,28.36 93.36,28.65 93.90,28.74 93.97,28.86 94.27,28.97 94.29,29.14 94.62,29.31 94.77,29.18 95.39,29.04 95.52,29.21 95.71,29.24 95.77,29.34 96.13,29.38 96.23,29.25 96.36,29.25 96.18,29.12 96.20,28.96 96.44,29.05 96.58,28.76 96.28,28.41 96.39,28.37 96.60,28.46 96.78,28.37 97.08,28.37 97.32,28.22
128.37,38.62 128.17,38.36 128.04,38.31 127.09,38.28 126.67,37.92 126.63,37.78
-8.68,27.66 -8.82,27.66 -8.75,27.19 -8.79,27.12 -9.41,27.09 -9.82,26.85 -10.03,26.91 -10.25,26.86 -10.76,27.02 -11.39,26.88 -11.32,26.74 -11.34,26.63 -11.51,26.47 -11.72,26.10 -12.06,25.99 -12.43,24.83 -12.99,24.47 -13.31,23.98 -13.89,23.69 -14.10,23.10 -14.22,22.31 -14.63,21.86 -14.61,21.75 -14.75,21.50 -14.97,21.44 -16.04,21.50 -17.00,21.42
35.92,4.62 35.76,4.81 35.80,5.16 35.75,5.34 35.47,5.42 35.33,5.36 35.27,5.49
-124.76,48.49 -123.86,48.26 -123.39,48.25 -123.14,48.40 -123.24,48.64 -123.01,48.79 -123.31,48.99
91.63,27.76 91.64,27.92 91.27,28.08 91.02,27.97 90.33,28.12
23.61,51.52 23.61,51.61 23.71,51.64 23.98,51.59 24.36,51.87 25.79,51.92 27.14,51.75 27.30,51.60 27.69,51.57 27.70,51.48 27.86,51.59 28.18,51.61 28.60,51.54 28.73,51.43 29.10,51.63 29.35,51.38 30.16,51.48 30.33,51.33 30.54,51.27 30.63,51.36 30.53,51.60 30.76,51.90 31.08,52.08 31.76,52.10
35.84,33.42 35.60,33.24
35.84,33.42 36.03,33.59 35.94,33.67 35.99,33.75 36.09,33.83 36.37,33.84 36.28,33.93 36.58,34.22 36.50,34.43 36.33,34.50 36.43,34.61 36.38,34.66 35.98,34.63
34.68,-12.01 34.76,-12.03 34.68,-12.12 34.68,-12.01
80.21,42.19 80.26,42.24 80.16,42.63 80.20,42.73 80.54,42.87 80.38,42.97 80.39,43.04 80.62,43.13 80.75,43.10 80.79,43.16 80.67,43.31 80.70,43.43 80.65,43.56 80.36,44.10 80.36,44.55 80.38,44.66 80.48,44.71 80.26,44.81 80.00,44.80 79.87,44.88 80.06,45.01 80.85,45.13 81.69,45.35 81.79,45.23 81.94,45.16 82.27,45.22 82.52,45.13 82.62,45.29 82.61,45.42 82.33,45.52 82.32,45.59 82.97,46.97 83.03,47.19 84.02,46.97 84.67,46.97 84.79,46.83 85.23,47.04 85.48,47.06 85.66,47.25 85.53,47.92 85.75,48.39 86.55,48.53 86.72,48.70 86.73,48.94 86.81,49.05 87.05,49.11 87.32,49.09
23.48,53.94 24.19,53.95 24.32,53.89 24.77,53.97 24.87,54.15 25.05,54.13 25.46,54.29 25.51,54.16 25.75,54.16 25.75,54.26 25.55,54.33 25.72,54.56 25.72,54.72 25.86,54.92 26.18,55.00 26.29,55.14 26.60,55.13 26.78,55.27 26.46,55.34 26.59,55.67
-53.37,-33.74 -53.53,-33.66 -53.53,-33.17 -53.13,-32.74 -53.16,-32.68 -53.60,-32.40 -53.76,-32.06 -54.22,-31.86 -54.59,-31.49 -54.90,-31.39 -55.04,-31.28 -55.17,-31.28 -55.56,-30.88 -55.63,-30.86 -55.81,-31.03
116.68,49.82 116.03,48.78 115.82,48.58 115.79,48.25 115.53,48.13 115.62,47.87 115.90,47.69 116.23,47.86 116.76,47.87 117.07,47.81 117.35,47.65 117.77,47.99 118.15,48.03 118.50,47.98 118.76,47.76 119.08,47.65 119.16,47.53 119.71,47.15 119.90,46.86 119.87,46.67 119.71,46.61 119.33,46.61 118.84,46.76 118.72,46.69 118.31,46.72 118.07,46.67 117.74,46.52 117.44,46.59 117.33,46.36 116.86,46.39 116.56,46.29 116.21,45.89 116.20,45.74 115.68,45.46 114.56,45.39 114.42,45.20 114.08,44.97 113.59,44.75 112.71,44.88 112.41,45.06 111.90,45.06 111.62,44.83 111.40,44.37 111.60,44.11 111.84,43.93 111.93,43.71 111.55,43.50 111.01,43.34 110.40,42.77 109.34,42.44 108.17,42.45 106.77,42.29 105.20,41.74 104.98,41.60 104.50,41.66 104.50,41.88 103.71,41.75 103.07,42.01 102.16,42.16 101.97,42.22 101.66,42.50 101.50,42.54 99.98,42.68 99.47,42.57 97.21,42.79 96.39,42.72 96.30,42.93 95.86,43.28 95.53,43.95 95.33,44.04 95.35,44.28 95.05,44.26 94.71,44.35 94.20,44.65 93.96,44.67 93.52,44.94 92.79,45.04 91.58,45.08 91.05,45.22 90.88,45.20 90.66,45.53 90.71,45.73 91.00,46.04 90.91,46.27 91.03,46.57 90.87,46.95 90.72,47.00 90.50,47.29 90.33,47.66 90.10,47.75 90.03,47.88 89.78,47.83 89.48,48.03 89.05,48.00 88.58,48.22 88.52,48.38 87.98,48.56 88.06,48.71 87.74,48.88 87.87,49.00 87.81,49.16
13.81,48.77 13.55,48.96 13.40,48.98 12.92,49.33 12.68,49.41 12.39,49.74 12.51,49.90 12.21,50.10 12.09,50.30 12.28,50.18 12.55,50.39 12.94,50.41 13.02,50.49 13.44,50.60 13.56,50.70 14.37,50.90 14.26,51.00 14.32,51.04 14.55,50.99 14.66,50.83 14.81,50.86
27.35,57.53 26.97,57.61 26.46,57.54 25.99,57.84 25.28,58.05 25.26,58.00 25.11,58.06 24.32,57.87
27.35,57.53 27.51,57.51 27.54,57.43 27.83,57.29 27.64,56.85 27.85,56.85 28.10,56.55 28.20,56.26 28.15,56.14
21.05,56.07 21.65,56.31 22.08,56.41 22.88,56.40 23.04,56.32 23.20,56.37 24.12,56.26 24.47,56.28 24.84,56.41 25.07,56.20 25.66,56.10 26.28,55.75 26.59,55.67
20.62,69.04 20.12,69.02 20.35,68.85 20.24,68.67 19.97,68.54 20.24,68.48 19.97,68.36 18.30,68.56 18.16,68.53 18.18,68.20 17.92,67.96 17.32,68.10 16.78,67.90 16.57,67.62 16.19,67.51 16.13,67.43 16.43,67.16 16.40,67.05 15.42,66.49 15.48,66.31 15.04,66.17 14.54,66.13 14.63,65.79 14.48,65.30 13.65,64.58 14.08,64.46 14.14,64.17 13.96,64.01 13.20,64.08 12.79,64.00 12.18,63.60 12.21,63.49 12.00,63.29 12.22,63.00 12.11,62.92 12.11,62.59 12.30,62.29 12.16,61.72 12.88,61.35 12.71,61.06 12.29,61.00 12.59,60.45 12.49,60.11 12.17,59.91 11.93,59.86 11.83,59.70 11.68,59.59 11.80,59.29 11.64,58.93 11.47,58.91 11.34,59.09 11.24,59.10 11.14,59.09 11.06,58.96 10.91,58.95
20.62,69.04 21.07,69.04 21.13,69.08 21.07,69.21 21.59,69.27 22.30,68.86 22.41,68.72 23.32,68.65 23.71,68.71 23.85,68.81 24.94,68.59 25.09,68.64 25.25,68.82 25.58,68.89 25.75,68.99 25.77,69.28 26.07,69.69 26.53,69.92 27.13,69.91 27.75,70.06 27.89,70.06 28.41,69.82 29.14,69.67 29.33,69.47 28.85,69.18 28.83,69.12 28.97,69.02
104.43,10.41 104.56,10.52 104.85,10.53 105.05,10.70 105.05,10.91 105.31,10.85 105.41,10.95 105.76,10.99 105.85,10.86 106.16,10.79 106.16,11.04 105.86,11.29 105.85,11.64 106.01,11.76 106.40,11.69 106.41,11.95 106.70,11.98 106.76,12.05 106.93,12.08 107.21,12.30 107.33,12.32 107.39,12.26 107.51,12.36 107.56,12.54 107.48,13.03 107.61,13.44 107.33,14.13 107.36,14.37 107.49,14.55 107.52,14.71
-69.51,-17.50 -69.31,-17.94 -69.09,-18.05 -69.15,-18.14 -68.97,-18.97 -68.46,-19.43 -68.70,-19.72 -68.56,-19.97 -68.60,-20.04 -68.76,-20.09 -68.69,-20.31 -68.76,-20.42 -68.48,-20.63 -68.56,-20.72 -68.56,-20.90 -68.44,-20.95 -68.20,-21.30 -68.19,-21.62 -68.08,-21.98 -67.99,-22.06 -67.88,-22.49 -67.88,-22.82 -67.58,-22.89 -67.19,-22.82
-69.51,-17.51 -69.85,-17.70 -69.80,-17.99 -69.93,-18.21 -70.18,-18.33 -70.42,-18.35
6.34,49.45 6.49,49.80 6.20,49.92 6.11,50.03 6.12,50.12
2.52,51.10 2.60,50.88 2.76,50.75 2.84,50.71 3.11,50.78 3.23,50.66 3.27,50.53 3.60,50.48 3.69,50.31 3.95,50.34 4.17,50.25 4.15,49.97 4.55,49.96 4.82,50.15 4.86,50.14 4.79,49.96 4.87,49.79 5.28,49.68 5.51,49.51 5.79,49.54
6.12,50.12 5.98,50.17 5.74,49.92 5.72,49.81 5.88,49.64 5.79,49.54
46.43,41.89 45.95,42.04 45.64,42.21 45.71,42.50 45.34,42.53 45.16,42.68 44.87,42.76 44.77,42.62 44.64,42.73 44.51,42.75 43.83,42.57 43.74,42.62 43.78,42.75 43.09,42.99 42.99,43.09 42.76,43.17 42.57,43.16 42.42,43.22 41.58,43.22 40.65,43.53 40.15,43.57 39.98,43.42
20.96,40.85 20.93,40.90 20.71,40.93 20.49,41.27 20.45,41.52 20.52,41.57 20.57,41.87
20.57,41.87 20.58,42.01 20.49,42.22 20.24,42.34 20.06,42.55
41.51,41.52 41.82,41.43 41.93,41.50 42.47,41.44 42.61,41.58 42.75,41.58 43.15,41.31 43.21,41.20 43.40,41.18 43.44,41.11
101.14,21.57 101.20,21.52 101.25,21.20 101.54,21.23 101.70,21.15 101.80,21.21 101.72,21.31 101.74,21.83 101.60,21.99 101.52,22.25 101.74,22.50 101.84,22.39 102.02,22.44 102.13,22.38
73.63,39.45 73.82,39.49 73.91,39.58 73.84,39.80 73.99,40.04 74.41,40.14 74.68,40.31 74.83,40.33 74.84,40.48 75.11,40.45 75.56,40.63 75.68,40.31 75.87,40.30 76.26,40.43 76.32,40.35 76.48,40.45 76.66,40.78 76.91,41.02 77.58,40.99 78.12,41.08 78.44,41.42 79.29,41.78 79.77,41.90 79.84,42.00 80.22,42.03 80.21,42.19
43.44,41.11 44.23,41.21 44.84,41.21 44.81,41.26 45.00,41.29
80.21,42.19 79.92,42.41 79.43,42.48 79.13,42.78 76.99,42.97 75.84,42.94 75.64,42.81 75.37,42.84 74.82,42.98 74.21,43.24 73.95,43.20 73.56,43.00 73.42,42.59 73.49,42.41 72.28,42.76 71.76,42.82 71.26,42.73 70.89,42.34 70.95,42.25
-56.48,1.94 -56.84,1.88 -57.12,2.01 -57.32,1.96 -57.55,1.73 -57.98,1.65 -58.03,1.52 -58.34,1.59 -58.40,1.48 -58.51,1.44 -58.51,1.28 -58.68,1.28 -58.82,1.20 -59.23,1.38 -59.54,1.70 -59.67,1.75 -59.67,1.84 -59.76,1.90 -59.76,2.27 -59.89,2.36 -59.99,2.69 -59.97,2.99 -59.83,3.35 -59.85,3.59 -59.68,3.70 -59.55,3.93 -59.74,4.23 -59.70,4.38 -59.83,4.48 -60.15,4.53 -60.03,4.74 -59.99,5.08 -60.14,5.24 -60.74,5.20
8.39,55.10 8.59,54.92 9.25,54.81 9.74,54.83
9.52,30.23 9.90,30.39 10.22,30.78 10.26,30.94 10.11,31.46 10.27,31.68 10.48,31.74 10.61,31.93 10.83,32.08 11.51,32.41 11.53,32.52 11.45,32.64 11.50,33.18
45.00,41.29 45.19,41.15 45.07,41.08 45.42,40.99 45.59,40.85 45.38,40.67 45.45,40.53 45.57,40.42 45.96,40.23 45.89,40.02 45.58,39.98 46.20,39.59 46.48,39.56 46.48,39.48 46.37,39.40 46.58,39.22 46.40,39.19 46.49,39.07 46.49,38.91
44.77,39.70 45.03,39.77 45.17,39.57 45.25,39.60 45.46,39.49 45.78,39.55 45.77,39.38 45.98,39.24 45.95,39.18 46.11,38.88
20.24,46.11 20.51,46.17 20.61,46.13 20.76,46.25 21.12,46.28 21.26,46.41 21.32,46.61 21.50,46.70 21.66,47.04 22.00,47.40 22.00,47.51 22.29,47.73 22.61,47.77 22.88,47.95
70.96,40.24 71.30,40.29 71.63,40.22 71.69,40.15 71.96,40.26 72.13,40.44 72.36,40.40 72.40,40.58 72.60,40.53 73.14,40.81 72.66,40.87 72.36,41.04 72.19,41.03 72.16,41.17 71.88,41.20 71.86,41.31 71.66,41.54 71.60,41.50 71.59,41.33 71.42,41.34 71.39,41.12 70.86,41.22 70.69,41.45 70.47,41.41 70.20,41.51 70.18,41.57 70.63,41.88 70.80,41.92 70.86,42.03 71.23,42.16 71.04,42.28 70.95,42.25
18.83,49.51 18.94,49.50 18.97,49.40 19.15,49.40 19.44,49.60 19.63,49.41 19.77,49.37 19.76,49.20 20.06,49.18 20.16,49.32 20.36,49.39 20.62,49.39 20.87,49.31 21.08,49.42 21.64,49.41 21.89,49.34 22.02,49.21 22.54,49.07
-7.22,55.09 -7.38,55.03 -7.55,54.77 -7.91,54.70 -7.75,54.59 -8.12,54.48 -8.12,54.41 -7.85,54.22 -7.61,54.14 -7.32,54.13 -7.16,54.24 -7.20,54.30 -7.01,54.41 -6.80,54.21 -6.65,54.16 -6.65,54.06 -6.22,54.09
18.83,49.51 18.81,49.61 18.59,49.76 18.56,49.88 18.03,50.04 17.87,49.97 17.63,50.12 17.59,50.16 17.74,50.23 17.70,50.31 17.42,50.25 16.88,50.43 16.99,50.24 16.64,50.10 16.21,50.42 16.42,50.57 16.28,50.66 16.01,50.61 15.73,50.74 15.35,50.81 15.26,50.96 14.99,51.01 14.98,50.89 14.81,50.86
70.96,40.24 70.95,40.19 70.64,40.08 70.62,40.00 70.52,39.95 70.45,40.05 69.97,40.20 69.53,40.10 69.48,39.92 69.31,39.97 69.28,39.92 69.23,39.76 69.30,39.52 70.50,39.59 70.80,39.39 71.00,39.41 71.07,39.49 71.47,39.60 71.50,39.48 71.73,39.42 71.73,39.31 71.78,39.28 72.04,39.35 72.23,39.21 72.36,39.34 72.56,39.38 73.11,39.36 73.63,39.45
20.00,39.71 20.25,39.68 20.31,39.80 20.38,39.80 20.31,39.98 20.66,40.12 20.81,40.45 20.95,40.49 21.03,40.62 20.96,40.85
33.20,-14.01 33.15,-13.94 32.99,-14.02 32.87,-13.82 32.77,-13.76 32.80,-13.69 32.67,-13.59 32.85,-13.46 32.97,-13.22 33.00,-12.90 32.95,-12.80 33.02,-12.63 33.40,-12.49 33.51,-12.35 33.34,-12.31 33.25,-12.11 33.30,-11.69 33.23,-11.42 33.38,-11.16 33.26,-10.89 33.46,-10.78 33.66,-10.55 33.53,-10.23 33.31,-10.04 33.35,-9.86 33.20,-9.63 33.00,-9.62 32.92,-9.41
22.13,48.41 22.14,48.57 22.30,48.69 22.54,49.07
-11.51,6.91 -11.27,7.23 -10.65,7.76 -10.57,8.07 -10.36,8.19 -10.28,8.49
24.15,8.67 24.22,8.61 24.21,8.37 24.29,8.29 24.85,8.14 25.20,7.81 25.25,7.72 25.18,7.56 25.28,7.43 25.89,7.06 26.36,6.64 26.31,6.46 26.51,6.07 27.14,5.72 27.33,5.19
42.92,11.00 43.25,11.50
42.92,11.00 42.56,11.08 41.96,10.94 41.80,10.98 41.79,11.69 42.38,12.47
29.40,-4.45 29.32,-4.90 29.61,-5.72 29.48,-6.03 29.54,-6.31 29.71,-6.62 30.21,-7.04 30.75,-8.19
42.38,12.47 42.48,12.51 42.70,12.38 42.87,12.62 43.12,12.71
13.81,48.77 14.05,48.60 14.69,48.60 14.79,48.75 14.92,48.77 14.97,48.98 15.07,49.00 15.83,48.86 16.06,48.75 16.37,48.74 16.54,48.80 16.88,48.70 16.95,48.60
42.36,37.11 42.77,37.37 42.94,37.32 43.09,37.37 43.68,37.23 44.11,37.30 44.19,37.25 44.20,37.05 44.28,36.98 44.61,37.18 44.77,37.14
16.09,46.86 16.25,46.97 16.45,47.01 16.49,47.12 16.42,47.22 16.44,47.40 16.62,47.45 16.68,47.54 16.64,47.61 16.42,47.67 16.59,47.75 16.79,47.68 17.07,47.71 17.03,47.84 17.15,48.01
7.02,45.93 7.13,45.88 7.54,45.98 7.79,45.92 7.99,46.02 8.13,46.16 8.10,46.27 8.42,46.45 8.46,46.25 8.64,46.11 8.82,46.08 8.78,46.00 8.95,45.83 9.02,45.85 9.00,46.01 9.25,46.29 9.26,46.48 9.43,46.48 9.53,46.31 9.94,46.36 10.04,46.24 10.13,46.24 10.04,46.48 10.09,46.60 10.43,46.55 10.40,46.67 10.45,46.86
10.45,46.86 10.35,46.98 10.13,46.85 9.88,46.94 9.85,47.01 9.58,47.06
44.77,37.14 44.79,37.29 44.57,37.44 44.55,37.66 44.59,37.71 44.21,37.91 44.45,38.33 44.30,38.39 44.27,38.84 44.17,38.93 44.18,39.14 44.08,39.22 44.02,39.38 44.39,39.42 44.46,39.67 44.59,39.77 44.82,39.65
23.48,53.94 23.45,54.14 23.37,54.20 22.89,54.39 22.77,54.36
30.75,-8.19 30.89,-8.47 31.03,-8.60 31.35,-8.61 31.53,-8.71 31.67,-8.91 31.92,-8.94 31.94,-9.05 32.43,-9.16 32.92,-9.41
7.20,53.28 7.19,53.00 7.03,52.65 6.71,52.62 6.75,52.46 6.97,52.44 7.04,52.38 6.98,52.21 6.72,52.08 6.80,51.97 6.52,51.85 6.36,51.82 6.17,51.88 5.95,51.80 6.20,51.45 6.08,51.22 6.13,51.15 5.86,51.03 6.05,50.90 5.99,50.75
9.58,47.06 9.61,47.11 9.53,47.27
-8.49,7.56 -8.43,7.60 -8.23,7.56 -8.01,8.08 -8.05,8.17 -8.26,8.25 -8.24,8.46 -8.05,8.50 -7.82,8.47 -7.74,8.38 -7.68,8.41 -7.72,8.64 -7.95,8.79 -7.94,8.98 -7.78,9.08 -7.92,9.19 -7.90,9.42 -8.03,9.40 -8.14,9.50 -8.16,9.97 -7.99,10.16
7.49,43.77 7.48,43.86 7.68,44.08 7.64,44.16 7.32,44.14 6.90,44.34 6.84,44.51 7.03,44.72 6.99,44.83 6.74,44.92 6.63,45.07 6.69,45.14 6.84,45.14 7.08,45.24 7.15,45.38 6.79,45.74 6.80,45.81 7.02,45.93
28.97,69.02 28.41,68.90 28.77,68.84 28.48,68.54 28.56,68.35 28.69,68.19 29.34,68.06 29.99,67.67 29.94,67.55 29.24,67.10 29.07,66.89 29.90,66.09 30.09,65.79 30.10,65.68 29.72,65.62 29.82,65.57 29.73,65.47 29.71,65.34 29.61,65.25 29.81,65.20 29.83,65.15 29.62,65.04 29.60,64.97 29.78,64.80 30.11,64.73 30.12,64.64 29.99,64.52 30.11,64.37 30.49,64.24 30.53,64.08 30.21,63.80 29.99,63.74 31.18,63.21 31.53,62.89 31.19,62.48 29.25,61.29 28.41,60.90 27.73,60.46
30.87,69.78 30.92,69.61 30.86,69.54 30.62,69.53 30.18,69.64 30.19,69.54 30.09,69.43 29.39,69.30 29.21,69.10 28.97,69.02
55.19,22.70 55.20,23.03 55.53,23.82 55.47,23.94 55.99,24.06 55.93,24.22 55.76,24.24 55.80,24.87 55.92,24.97 56.00,24.95 55.97,24.86 56.06,24.74 56.39,24.98
48.57,41.84 48.39,41.60 48.06,41.46 47.79,41.20 47.32,41.28 47.14,41.52 46.75,41.81 46.57,41.80 46.54,41.87 46.43,41.89
-54.62,2.33 -54.72,2.44 -54.88,2.45 -54.98,2.60 -55.39,2.44 -55.73,2.41 -55.96,2.52 -56.14,2.26 -55.92,2.04 -55.93,1.89 -56.02,1.84 -56.48,1.94
46.43,41.89 46.20,41.74 46.18,41.66 46.31,41.51 46.67,41.29 46.53,41.09 46.43,41.08 46.17,41.20 45.92,41.19 45.73,41.26 45.72,41.34 45.28,41.45 45.00,41.29
-11.39,12.40 -11.45,12.56 -11.39,12.94 -11.63,13.37 -11.76,13.39 -11.83,13.32 -12.05,13.63 -11.96,13.88 -12.02,13.97 -12.02,14.21 -12.23,14.46 -12.19,14.65 -12.28,14.81
-54.62,2.33 -54.13,2.12 -53.77,2.35 -53.51,2.25 -53.33,2.34 -53.23,2.20 -52.90,2.21 -52.58,2.53 -52.33,3.18 -51.99,3.70 -51.65,4.06
2.71,6.37 2.77,6.71 2.72,6.98 2.79,7.48 2.69,7.87 2.77,9.05 3.04,9.08 3.15,9.32 3.14,9.45 3.33,9.67 3.35,9.81 3.56,9.91 3.65,10.16 3.58,10.29 3.65,10.41 3.77,10.42 3.83,10.61 3.72,11.08 3.49,11.40 3.60,11.70
13.07,-4.63 12.80,-4.43 12.38,-4.62 12.31,-4.77 12.20,-4.78 12.02,-5.00
16.52,46.50 16.32,46.53 16.24,46.48 16.23,46.37 16.07,46.37 15.64,46.20 15.59,46.14 15.67,46.05 15.65,45.86 15.28,45.73 15.35,45.66 15.28,45.58 15.34,45.47 14.79,45.48 14.57,45.66 14.37,45.48 13.99,45.51 13.88,45.43 13.58,45.52
18.91,45.93 18.67,45.91 18.53,45.80 18.36,45.75 17.81,45.79 17.31,46.00 16.87,46.34 16.52,46.50
20.24,46.11 19.61,46.17 19.28,46.00 19.09,46.02 18.91,45.93
16.52,46.50 16.28,46.86 16.09,46.86
51.27,24.61 50.97,24.57 50.80,24.79
25.26,-17.79 25.24,-17.97 25.94,-18.94 25.95,-19.08 26.17,-19.54 26.68,-19.89 27.18,-20.10 27.26,-20.23 27.28,-20.48 27.68,-20.50 27.67,-21.06 28.01,-21.55 29.03,-21.80 29.04,-22.02 29.36,-22.19
77.80,35.50 77.45,35.48 76.77,35.66
28.01,41.97 27.53,41.92 27.24,42.09 27.01,42.06 26.58,41.95 26.51,41.83 26.36,41.80 26.32,41.72
102.93,11.71 102.74,12.09 102.76,12.43 102.50,12.67 102.46,13.02 102.33,13.29 102.34,13.56 102.55,13.59 102.54,13.66 102.91,14.14 103.20,14.33 103.60,14.42 103.98,14.36 104.78,14.43 104.97,14.37 105.00,14.25 105.07,14.23 105.18,14.35
88.89,27.32 88.74,27.18 88.86,26.96 89.15,26.82 89.33,26.85 89.55,26.80 89.61,26.72 90.12,26.75 90.35,26.89 90.74,26.77 91.29,26.79 91.43,26.87 91.67,26.80 92.05,26.87 92.07,26.98 91.99,27.10 92.08,27.29 91.99,27.45 91.74,27.44 91.59,27.56 91.63,27.76
104.07,1.28 104.06,1.40 103.82,1.46 103.70,1.44 103.60,1.21
100.12,6.44 100.18,6.67 100.26,6.68 100.35,6.55 100.75,6.46 100.87,6.25 101.05,6.24 101.08,5.96 100.98,5.77 101.11,5.64 101.26,5.79 101.56,5.91 101.68,5.78 101.87,5.83 102.10,6.24
5.99,50.75 6.24,50.60 6.17,50.55 6.20,50.50 6.34,50.45 6.36,50.32 6.18,50.23 6.12,50.12
-71.78,19.72 -71.71,19.49 -71.75,19.29 -71.65,19.16 -71.81,18.99 -71.74,18.73 -71.87,18.61 -72.00,18.60 -71.76,18.34 -71.77,18.04
14.06,13.08 13.61,13.70
46.53,29.10 46.91,29.54 47.15,30.00 47.33,30.08 47.64,30.10 47.98,29.98
88.11,27.87 88.14,27.95 88.58,28.09 88.80,28.01 88.85,27.87 88.75,27.52 88.89,27.32
-90.10,13.74 -90.05,13.90 -89.55,14.24 -89.57,14.39 -89.36,14.42
124.04,-9.34 124.09,-9.42 124.28,-9.43 124.41,-9.31 124.44,-9.19
124.92,-8.94 124.94,-9.05 125.10,-9.00 125.15,-9.04 125.10,-9.19 124.96,-9.21 125.07,-9.51
115.14,4.90 115.27,4.63 115.29,4.35 115.11,4.39 115.03,4.90
109.63,2.03 109.54,1.90 109.64,1.78 109.65,1.61 110.51,0.86 111.10,1.05 111.81,1.01 111.92,1.11 112.08,1.14 112.19,1.44 112.48,1.56 112.94,1.57 113.01,1.43 113.62,1.24 113.90,1.43 114.39,1.50 114.51,1.45 114.70,1.85 114.83,1.98 114.76,2.16 114.79,2.25 115.18,2.52 115.08,2.63 115.12,2.89 115.25,3.03 115.45,3.03 115.57,3.45 115.57,3.94 115.68,4.19 115.84,4.33 116.02,4.29 116.24,4.36 116.41,4.31 116.51,4.37 117.10,4.34 117.54,4.17 117.88,4.19
10.45,46.86 10.99,46.78 11.24,46.98 11.78,46.99 12.17,47.08 12.15,46.94 12.39,46.70 13.70,46.52
13.70,46.52 13.38,46.26 13.63,46.18 13.48,46.01 13.61,45.96 13.58,45.81 13.87,45.61 13.72,45.59
29.01,-2.72 29.10,-2.60 29.30,-2.67 29.39,-2.81 29.70,-2.79 29.87,-2.72 29.93,-2.34 30.12,-2.42 30.41,-2.31 30.55,-2.40
92.57,21.98 92.63,21.31 92.57,21.26 92.33,21.44 92.18,21.29 92.32,20.79
1.43,42.60 1.29,42.71 0.70,42.85 0.63,42.69 -0.04,42.69 -0.30,42.83 -0.59,42.80 -0.76,42.94 -1.18,43.02 -1.30,43.10 -1.46,43.05 -1.41,43.24 -1.71,43.31 -1.79,43.41
1.43,42.60 1.50,42.64 1.71,42.60 1.71,42.50
1.71,42.50 2.03,42.35 2.20,42.42 2.65,42.34 2.89,42.46 3.21,42.43
1.43,42.60 1.45,42.44 1.71,42.50
74.89,37.23 75.12,37.39 74.89,37.60 74.94,37.77 74.78,38.19 74.81,38.46 74.28,38.66 74.13,38.66 74.03,38.54 73.80,38.61 73.70,38.85 73.81,38.97 73.61,39.23 73.63,39.45
74.89,37.23 74.73,37.29 74.37,37.16 74.54,37.02
20.96,40.85 21.40,40.91 21.58,40.87 21.99,41.13 22.60,41.14 22.72,41.18 22.78,41.33 22.92,41.34
21.56,42.25 21.39,42.22 21.29,42.10 21.06,42.17 20.78,42.07 20.72,41.87 20.57,41.87
22.34,42.31 22.24,42.36 21.56,42.25
22.34,42.31 22.52,42.44 22.44,42.63 22.47,42.84 22.71,42.88 22.98,43.19 22.55,43.45 22.37,43.78 22.42,44.01 22.60,44.08 22.63,44.19 22.71,44.24
22.92,41.34 23.00,41.74 22.84,41.99 22.58,42.10 22.34,42.31
18.44,42.56 18.52,42.43
18.44,42.56 17.80,42.90 17.67,42.90
78.92,33.39 79.12,33.23
20.34,42.83 20.47,42.86 20.48,42.95 20.62,43.03 20.62,43.20 20.80,43.26 20.84,43.17 21.06,43.09 21.40,42.83 21.39,42.75 21.75,42.67 21.61,42.39 21.52,42.33 21.56,42.25
20.34,42.83 20.05,42.76 20.06,42.55
19.19,43.53 19.22,43.45 19.61,43.17 20.27,42.94 20.34,42.83
-89.36,14.42 -89.12,14.37 -88.71,14.03 -88.51,13.98 -88.48,13.85 -88.15,13.99 -87.99,13.88 -87.80,13.89 -87.72,13.81 -87.78,13.52 -87.74,13.45 -87.81,13.40
-87.34,12.98 -87.01,13.01 -86.92,13.22 -86.73,13.28 -86.76,13.75 -86.33,13.77 -86.04,14.05 -85.98,13.97 -85.73,13.86 -85.68,13.98 -85.18,14.34 -85.16,14.53 -85.04,14.61 -84.99,14.75 -84.86,14.81 -84.65,14.66 -84.45,14.64 -84.24,14.75 -84.15,14.72 -83.64,14.88 -83.54,14.98 -83.16,14.99
-75.28,-0.11 -75.78,0.09 -76.07,0.35 -76.27,0.44 -76.39,0.40 -76.43,0.26 -76.49,0.24 -76.83,0.25 -77.40,0.39 -77.47,0.64 -77.65,0.72 -77.70,0.84 -77.83,0.83 -78.18,0.97 -78.86,1.46
-57.60,3.37 -57.65,3.52 -57.83,3.68 -58.03,4.00 -58.05,4.17 -57.85,4.67 -57.92,4.82 -57.88,4.88 -57.71,4.99 -57.33,5.02 -57.21,5.20 -57.32,5.34 -57.19,5.55
-54.08,3.33 -53.99,3.59 -54.35,4.05 -54.48,4.84 -54.45,5.01 -54.16,5.36
-58.41,-33.94 -58.48,-33.65 -58.38,-33.14 -58.24,-33.11 -58.13,-32.98 -58.20,-32.47
-57.61,-30.19 -57.87,-30.59 -57.81,-30.86 -57.90,-30.98 -57.89,-31.20 -58.05,-31.49 -57.99,-31.58 -58.01,-31.68 -58.19,-31.92 -58.12,-32.25 -58.20,-32.47
-54.62,-25.58 -54.44,-25.62 -54.15,-25.52 -53.89,-25.67 -53.67,-26.29 -53.75,-26.75 -53.72,-26.88 -53.84,-27.12 -54.16,-27.25 -54.33,-27.42 -54.83,-27.55 -54.91,-27.71 -55.10,-27.87 -55.73,-28.20 -55.69,-28.38 -55.89,-28.37 -56.94,-29.59 -57.22,-29.78 -57.41,-30.03 -57.61,-30.19
20.62,69.04 20.90,68.98 20.92,68.91 22.00,68.52 22.85,68.37 23.10,68.26 23.18,68.14 23.32,68.13 23.64,67.95 23.50,67.88 23.54,67.61 23.45,67.46 23.73,67.42 23.77,67.33 23.63,67.23 23.64,67.13 23.99,66.81 23.67,66.38 23.72,66.22 23.91,66.15 24.16,65.81 24.19,65.61 24.12,65.51
4.23,51.39 4.17,51.31 3.90,51.21 3.58,51.29 3.43,51.25 3.35,51.38
5.99,50.75 5.75,50.76 5.64,50.84 5.75,50.95 5.80,51.15 5.48,51.29 5.21,51.28 5.03,51.47 4.85,51.40 4.76,51.49 4.63,51.42 4.50,51.47 4.38,51.43 4.37,51.36 4.23,51.39
6.34,49.45 6.53,49.39 6.74,49.16 6.89,49.21 7.04,49.11 7.45,49.15 7.61,49.06 8.13,48.97 8.14,48.89 7.84,48.64 7.62,48.16 7.53,47.67 7.62,47.59
9.52,47.52 9.18,47.67 8.88,47.66 8.57,47.78 8.40,47.69 8.57,47.65 8.45,47.60 7.62,47.59
9.53,47.27 9.63,47.47 9.52,47.52
9.58,47.06 9.49,47.06 9.53,47.27
7.02,45.93 6.77,46.17 6.82,46.28 6.76,46.42 6.43,46.43 6.23,46.33 6.27,46.25 6.20,46.19 6.01,46.14 5.97,46.21 6.10,46.28 6.12,46.38 6.07,46.46 6.16,46.61 6.41,46.76 6.46,46.95 6.67,47.03 7.00,47.32 6.90,47.39 6.97,47.45 7.14,47.49 7.34,47.43 7.62,47.59
-7.41,37.18 -7.50,37.59 -7.44,37.73 -7.19,38.01 -7.02,38.04 -6.96,38.19 -7.11,38.18 -7.34,38.46 -7.28,38.71 -7.05,38.91 -7.00,39.06 -7.17,39.14 -7.34,39.47 -7.54,39.66 -7.12,39.68 -6.98,39.80 -6.90,40.02 -7.03,40.17 -6.81,40.34 -6.84,40.78 -6.93,41.01 -6.21,41.53 -6.31,41.64 -6.54,41.67 -6.62,41.94 -7.15,41.98 -7.21,41.90 -7.40,41.83 -7.92,41.88 -8.15,41.81 -8.22,41.90 -8.13,42.02 -8.27,42.14 -8.78,41.94
13.81,48.77 13.79,48.59 13.72,48.54 13.49,48.58 13.37,48.36 12.76,48.11 12.95,47.89 12.90,47.72 13.05,47.66 13.01,47.48 12.81,47.54 12.77,47.64 12.69,47.67 12.48,47.64 12.21,47.72 12.19,47.62 11.72,47.58 11.04,47.39 10.87,47.52 10.44,47.55 10.37,47.37 10.18,47.28 10.20,47.36 10.07,47.39 9.97,47.51 9.75,47.58 9.52,47.52
16.09,46.86 15.98,46.80 15.96,46.68 15.77,46.71 15.44,46.63 14.89,46.61 14.55,46.40 13.70,46.52
18.44,42.56 18.55,42.64 18.47,42.78 18.46,43.00 18.62,43.03 18.67,43.23 18.85,43.35 19.03,43.29 18.95,43.53 19.19,43.53
19.19,43.53 19.30,43.59 19.45,43.56 19.50,43.64 19.25,43.97 19.58,44.01 19.12,44.36 19.35,44.88 18.84,44.88 18.66,45.08 18.00,45.14 17.81,45.08 17.65,45.16 17.50,45.12 17.13,45.17 16.92,45.28 16.79,45.20 16.53,45.22 16.29,45.01 16.03,45.19 15.79,45.18 15.74,44.77 16.10,44.52 16.30,44.12 17.08,43.52 17.27,43.45 17.29,43.31 17.65,43.01 17.59,42.94
19.01,44.87 19.10,44.97 19.06,45.14 19.14,45.20 19.40,45.21 19.00,45.40 19.06,45.52 18.92,45.60 18.95,45.66 18.84,45.84 18.91,45.93
16.95,48.60 17.14,48.84 17.76,48.89 18.09,49.07 18.16,49.26 18.60,49.49 18.83,49.51
17.15,48.01 16.87,48.39 16.95,48.60
17.15,48.01 17.32,47.99 17.76,47.77 18.72,47.79 18.79,48.00 19.47,48.11 19.63,48.22 19.95,48.15 20.33,48.30 20.49,48.53 21.45,48.55 21.72,48.35 22.13,48.41
14.81,50.86 15.02,51.25 14.91,51.46 14.72,51.52 14.74,51.63 14.60,51.83 14.75,52.08 14.55,52.36 14.62,52.53 14.13,52.88 14.41,53.22 14.21,53.95
22.88,47.95 22.77,48.11 22.58,48.13 22.35,48.26 22.25,48.41 22.13,48.41
28.59,43.74 28.22,43.77 28.05,43.82 27.88,43.99 27.74,43.96 27.43,44.02 27.09,44.17 26.22,44.01 25.82,43.77 25.50,43.67 23.22,43.87 22.92,43.83 22.87,43.95 23.03,44.08 22.71,44.24
20.24,46.11 20.71,45.74 20.77,45.75 20.79,45.47 21.49,45.15 21.35,45.01 21.53,44.90 21.36,44.83
22.92,41.34 23.64,41.39 24.01,41.46 24.06,41.53 24.49,41.56 24.77,41.36 24.99,41.36 25.25,41.24 25.92,41.31 26.14,41.39 26.07,41.67 26.11,41.73 26.41,41.70 26.58,41.60 26.62,41.40 26.33,41.24 26.33,40.95 26.04,40.73
20.06,42.55 19.79,42.48 19.73,42.63 19.65,42.63 19.28,42.17 19.36,42.07 19.34,41.87
26.62,48.26 26.79,48.26 26.98,48.16 27.61,47.34 28.07,46.98 28.24,46.64 28.24,46.45 28.10,45.97 28.16,45.65 28.07,45.60 28.21,45.45
22.88,47.95 23.20,48.08 23.41,47.99 24.18,47.91 24.58,47.93 24.89,47.72 24.98,47.72 25.46,47.91 26.16,47.99 26.31,48.20 26.62,48.26
22.54,49.07 22.81,49.02 22.85,49.06 22.71,49.17 22.73,49.30 22.65,49.54 22.71,49.61 23.71,50.38 23.97,50.41 24.09,50.53 23.98,50.79 24.10,50.87 23.66,51.31 23.61,51.52
20.90,55.29 21.39,55.28 22.07,55.06 22.57,55.06 22.82,54.87 22.68,54.56 22.77,54.36
31.76,52.10 31.58,52.31 31.62,52.55 31.52,52.70 31.56,52.76 31.26,53.02 31.42,53.20 31.67,53.20 31.85,53.11 32.14,53.09 32.70,53.34 32.71,53.42 32.47,53.55 32.45,53.69 32.20,53.78 31.75,53.81 31.83,54.03 31.40,54.20 31.18,54.45 31.07,54.49 31.15,54.63 30.80,54.78 30.83,54.91 30.98,55.05 30.81,55.31 30.90,55.40 30.91,55.57 30.23,55.85 29.94,55.85 29.48,55.68 29.35,55.78 29.38,55.94 29.03,56.02 28.79,55.94 28.56,56.09 28.28,56.06 28.15,56.14
31.76,52.10 32.12,52.05 32.28,52.11 32.44,52.31 32.81,52.25 33.15,52.34 33.74,52.34 33.92,52.25 34.11,51.98 34.40,51.78 34.38,51.72 34.12,51.68 34.27,51.34 34.21,51.26 34.71,51.17 35.06,51.20 35.16,51.06 35.31,51.04 35.44,50.73 35.41,50.54 35.59,50.37 35.67,50.35 35.89,50.44 36.12,50.41 36.31,50.28 36.50,50.28 36.62,50.21 36.76,50.29 37.42,50.41 37.70,50.11 38.05,49.92 38.26,50.05 38.92,49.82 39.17,49.86 39.30,49.74 39.46,49.73 39.78,49.57 40.08,49.58 40.11,49.25 39.89,49.06 39.69,49.01 39.75,48.91 40.00,48.82 39.79,48.81 39.64,48.59 39.84,48.54 39.89,48.36 39.85,48.30 39.96,48.27 39.78,47.89 39.66,47.84 38.90,47.86 38.64,47.67 38.37,47.61 38.29,47.56 38.20,47.32 38.28,47.28 38.20,47.18 38.21,47.09
43.44,41.11 43.63,40.93 43.72,40.72 43.57,40.48 43.71,40.17 43.67,40.13 44.01,40.01 44.29,40.04 44.78,39.68
87.82,49.16 88.12,49.26 88.19,49.45 88.83,49.45 88.90,49.54 89.01,49.47 89.11,49.50 89.24,49.63 89.40,49.61 89.65,49.72 89.64,49.90 89.98,49.98 90.05,50.09 90.66,50.22 91.02,50.42 91.42,50.47 91.80,50.69 92.19,50.70 92.35,50.86 92.63,50.69 92.78,50.78 92.94,50.78 93.10,50.60 94.25,50.56 94.35,50.22 94.61,50.02 94.93,50.04 95.11,49.94 95.52,49.91 95.79,50.01 95.94,49.96 96.07,50.00 96.32,49.90 96.99,49.88 97.21,49.73 97.36,49.74 97.59,49.91 98.00,50.01 98.25,50.30 98.28,50.53 98.03,50.64 97.84,51.05 98.04,51.45 98.22,51.51 98.35,51.72 98.64,51.80 98.89,52.12 99.72,51.87 99.92,51.76 100.47,51.73 101.38,51.45 102.11,51.35 102.22,50.83 102.32,50.72 102.29,50.59 102.68,50.39 103.16,50.29 103.30,50.20 103.63,50.14 104.08,50.15 104.47,50.31 105.38,50.47 106.22,50.30 106.71,50.31 107.23,49.99 107.92,49.95 107.97,49.65 108.61,49.32 109.24,49.33 110.71,49.14 111.34,49.36 112.08,49.42 112.49,49.53 112.81,49.52 113.06,49.62 113.16,49.80 113.57,50.01 114.30,50.27 114.74,50.23 115.43,49.90 115.72,49.88 116.22,50.01 116.55,49.92 116.68,49.82
116.68,49.82 117.25,49.62 117.87,49.51 118.45,49.84 119.26,50.07 119.35,50.28 119.16,50.41 119.45,50.70 120.07,51.60 120.51,51.85 120.75,52.10 120.67,52.30 120.70,52.49 120.66,52.57 120.17,52.60 120.07,52.63 120.04,52.72 120.70,53.17 120.99,53.28 122.34,53.49 123.61,53.55 124.81,53.13 125.08,53.20 125.65,53.04 125.73,52.89 125.87,52.87 126.05,52.74 126.02,52.61 126.34,52.36 126.51,51.93 126.70,51.70 126.71,51.57 126.81,51.51 126.92,51.10 127.31,50.71 127.34,50.35 127.59,50.21 127.49,49.98 127.55,49.80 127.71,49.67 128.00,49.57 128.70,49.60 128.82,49.46 129.07,49.37 129.50,49.39 130.20,48.89 130.55,48.86 130.62,48.77 130.55,48.60 130.80,48.34 130.71,48.13 130.73,48.02 130.92,47.84 130.96,47.71 132.48,47.71 132.71,47.95 133.14,48.11 133.47,48.10 133.84,48.27 134.29,48.37 134.67,48.25 134.67,48.15 134.57,48.02 134.75,47.72 134.54,47.49 134.29,47.41 134.17,47.30 134.20,47.13 133.87,46.50 133.86,46.25 133.70,46.14 133.65,45.96 133.51,45.88 133.44,45.60 133.19,45.49 133.11,45.13 132.94,45.03 131.98,45.24 131.85,45.33 131.45,44.98 130.98,44.84 131.26,44.07 131.17,43.70 131.18,43.51 131.26,43.38 131.07,42.90 130.58,42.81 130.42,42.73 130.58,42.62 130.53,42.54
130.53,42.54 130.25,42.74 130.24,42.89 129.90,43.00 129.70,42.45 129.52,42.38 129.31,42.41 129.20,42.22 128.92,42.04 128.05,41.99 128.03,41.90 128.29,41.61 128.15,41.39 128.01,41.45 127.69,41.44 127.18,41.53 126.95,41.77 126.72,41.72 126.60,41.64 126.49,41.36 125.99,40.90 125.73,40.87 125.42,40.66 124.89,40.46 124.39,40.10 124.36,40.00
130.53,42.54 130.69,42.30
70.96,40.24 70.65,40.20 70.37,40.38 70.38,40.45 70.75,40.72 70.40,41.04 70.29,40.89 69.71,40.66 69.41,40.80 69.36,40.77 69.31,40.63 69.21,40.57 69.30,40.33 69.22,40.29 69.27,40.20 68.95,40.22 68.63,40.17 68.97,40.09 68.80,40.05 68.87,39.91 68.64,39.84 68.59,39.63 68.46,39.54 67.72,39.62 67.43,39.47 67.36,39.22 67.65,39.13 67.69,38.99 68.04,38.98 68.13,38.93 68.05,38.67 68.09,38.47 68.35,38.21 68.29,38.03 67.81,37.49 67.76,37.17
35.89,35.92 36.15,35.83 36.20,35.94 36.35,36.00 36.38,36.17 36.64,36.23 36.54,36.46 36.66,36.80 36.94,36.76 37.07,36.65 37.44,36.64 38.19,36.90 38.44,36.86 38.77,36.69 39.36,36.68 40.02,36.83 40.71,37.10 41.52,37.09 42.06,37.21 42.20,37.30 42.31,37.23 42.36,37.11
35.45,31.48 35.40,31.23 35.44,31.13 35.14,30.42 35.14,30.14 34.97,29.56
35.55,32.40 35.56,31.77 35.45,31.48
35.61,32.68 35.55,32.40
88.11,27.87 87.14,27.84 86.61,28.10 86.52,27.96 86.41,27.93 86.14,28.11 86.08,28.08 86.06,27.93 85.99,27.91 85.68,28.28 85.12,28.32 85.09,28.37 85.16,28.59 84.80,28.56 84.23,28.91 84.10,29.22 83.94,29.28 83.58,29.18 83.16,29.61 82.85,29.68 82.22,30.06 82.04,30.33 81.64,30.39 81.42,30.34 81.18,30.04 81.11,30.04 81.01,30.16
97.32,28.22 97.34,27.94 96.88,27.59 96.90,27.44 97.10,27.12 96.95,27.13 96.73,27.33 96.19,27.26 95.46,26.76 95.13,26.60 95.05,26.35 95.13,26.04 95.02,25.91 94.95,25.70 94.58,25.32 94.55,25.22 94.71,25.05 94.29,24.32 94.13,23.88 93.68,24.01 93.49,23.97 93.33,24.06 93.41,23.68 93.39,23.34 93.35,23.08 93.31,23.03 93.16,23.03 93.08,22.72 93.15,22.23 93.04,22.18 92.96,22.00 92.85,22.01 92.69,22.13 92.57,21.98
100.12,20.32 99.89,20.42 99.72,20.33 99.46,20.36 99.49,20.15 99.34,20.08 99.07,20.10 98.92,19.77 98.37,19.69 98.02,19.75 97.82,19.46 97.80,19.13 97.71,18.93 97.75,18.59 97.52,18.50 97.37,18.52 97.45,18.36 97.63,18.29 97.74,17.94 97.71,17.80 98.44,16.98 98.48,16.73 98.66,16.33 98.84,16.42 98.89,16.35 98.82,16.18 98.59,16.05 98.56,15.37 98.19,15.20 98.20,14.98 98.25,14.81 98.57,14.36 98.93,14.05 99.14,13.72 99.18,13.23 99.12,13.03 99.22,12.74 99.41,12.55 99.61,11.78 99.19,11.11 98.76,10.66 98.70,10.19
103.09,18.14 103.09,18.14
103.09,18.14 103.26,18.28 103.29,18.41 103.95,18.32 104.43,17.70 104.74,17.46 104.82,17.30 104.74,16.88 104.82,16.47 105.05,16.16 105.41,15.99 105.40,15.83 105.64,15.66 105.49,15.26 105.55,14.93 105.50,14.59 105.42,14.47 105.18,14.35
100.12,20.32 100.14,20.25 100.32,20.39 100.52,20.18 100.54,20.09 100.40,19.76 100.51,19.55 100.74,19.51 100.97,19.61 101.21,19.55 101.20,19.33 101.29,18.98 101.05,18.44 101.14,18.14 100.91,17.58 100.96,17.54 101.17,17.50 101.69,17.89 101.77,18.03 101.95,18.08 102.10,18.21 102.55,17.97 102.66,17.82 102.81,17.95 103.05,18.03 103.09,18.14
101.14,21.57 101.08,21.47 100.70,21.25 100.54,20.99 100.52,20.92 100.62,20.86 100.25,20.73 100.12,20.32
97.32,28.22 97.54,28.51 97.60,28.52 98.06,28.19 98.30,27.55 98.45,27.66 98.65,27.57 98.74,26.79 98.69,26.19 98.56,26.07 98.66,25.86 98.47,25.79 98.33,25.59 98.14,25.57 98.01,25.29 97.82,25.25 97.71,25.03 97.74,24.87 97.58,24.77 97.53,24.49 97.67,24.38 97.71,24.23 97.56,23.91 97.69,23.90 98.21,24.11 98.58,24.07 98.84,24.12 98.68,23.91 98.83,23.62 98.80,23.52 98.88,23.38 98.86,23.19 99.42,23.07 99.51,22.96 99.39,22.83 99.34,22.50 99.21,22.28 99.19,22.13 99.92,22.03 99.94,21.76 100.10,21.66 100.15,21.48 100.60,21.47 101.08,21.76 101.13,21.74 101.14,21.57
102.13,22.38 102.58,21.90 102.66,21.68 102.74,21.68 102.82,21.81 102.95,21.68 102.85,21.27 103.10,20.89 103.64,20.70 104.10,20.95 104.58,20.65 104.37,20.44 104.62,20.37 104.70,20.21 104.85,20.20 104.93,20.02 104.59,19.62 104.03,19.68 104.06,19.48 103.89,19.30 104.72,18.80 105.15,18.65 105.11,18.41 105.33,18.19 105.46,18.15 105.69,17.74 106.50,16.95 106.55,16.65 106.66,16.49 106.74,16.45 106.85,16.52 106.93,16.35 107.40,16.04 107.39,15.95 107.17,15.80 107.65,15.26 107.48,14.98 107.52,14.71
27.40,5.11 27.07,5.20 26.82,5.06 25.53,5.31 25.40,5.26 25.25,5.02 25.07,4.97 24.77,4.93 24.32,4.99 23.42,4.66 23.12,4.74 22.86,4.72 22.71,4.59 22.42,4.13 21.69,4.28 21.54,4.24 20.56,4.46 20.23,4.83 19.81,5.09 19.50,5.13 19.07,4.89 18.83,4.52 18.59,4.35 18.61,3.48
30.55,-2.40 30.42,-2.64 30.47,-2.69 30.43,-2.87 30.78,-2.98 30.79,-3.27 30.63,-3.35 30.63,-3.42 30.43,-3.59 30.15,-4.09 29.95,-4.31 29.72,-4.46 29.40,-4.45
29.01,-2.72 29.02,-2.80 29.22,-3.05 29.21,-3.83 29.40,-4.45
29.58,-1.39 29.35,-1.52 29.20,-1.72 29.13,-1.86 29.13,-2.20 28.88,-2.40 28.89,-2.64 29.01,-2.72
35.27,5.49 34.98,5.86 34.71,6.66 34.06,7.23 33.90,7.51 33.67,7.67 33.23,7.76 33.00,7.90 33.28,8.44 33.95,8.44 34.07,8.55 34.08,9.46
22.86,10.92 22.49,11.00 22.04,10.82 21.77,10.64 21.71,10.57 21.68,10.29 21.53,10.21 21.40,10.00 21.26,9.97 20.77,9.41 20.34,9.13 20.07,9.13 19.67,9.02 19.15,9.02 18.96,8.94 18.89,8.84 19.11,8.66 18.56,8.05 17.65,7.98 16.78,7.55 16.55,7.87 16.40,7.77 16.38,7.68 15.96,7.51 15.48,7.52
13.29,2.16 13.22,2.26 11.56,2.30 11.35,2.30 11.33,2.17
11.13,-3.92 11.23,-3.69 11.50,-3.52 11.69,-3.68 11.88,-3.67 11.83,-3.53 11.93,-3.32 11.72,-3.18 11.76,-2.98 11.64,-2.86 11.54,-2.84 11.61,-2.34 11.73,-2.39 11.95,-2.34 12.06,-2.41 12.45,-2.33 12.43,-1.93 12.59,-1.83 12.79,-1.93 12.99,-2.31 13.46,-2.40 13.73,-2.14 13.88,-2.33 13.89,-2.47 13.99,-2.49 14.20,-2.35 14.16,-2.22 14.25,-2.00 14.38,-1.89 14.47,-0.57 14.21,-0.43 14.07,-0.27 13.86,-0.20 13.95,0.35 14.09,0.54 14.32,0.62 14.43,0.90 14.18,1.37 13.85,1.42 13.22,1.25 13.25,1.37 13.17,1.79 13.29,2.16
15.48,7.52 15.55,7.79 15.44,7.85 15.12,8.56 14.33,9.20 13.98,9.69 14.24,9.98 14.84,9.94 15.65,10.01 15.28,10.36 15.13,10.65 15.03,11.11 15.12,11.54 15.08,11.85 14.88,12.27 14.85,12.50 14.76,12.66 14.54,12.82 14.46,13.02 14.06,13.08
14.06,13.08 14.20,12.38 14.52,12.30 14.62,12.15 14.58,11.53 14.20,11.27 13.98,11.21 13.70,10.87 13.54,10.61 13.41,10.17 13.27,10.04 13.20,9.56 12.93,9.43 12.78,8.82 12.58,8.62 12.40,8.60 12.23,8.28 12.02,7.59 11.77,7.27 11.86,7.12 11.58,6.89 11.48,6.60 11.24,6.45 11.11,6.46 11.01,6.74 10.61,7.06 10.48,6.89 10.21,6.89 10.14,7.00 9.78,6.76 9.66,6.53 9.00,5.92 8.80,5.20 8.56,4.76
3.60,11.70 3.66,11.76 3.61,11.89 3.65,12.53 3.95,12.78 4.04,12.93 4.15,13.46 4.66,13.73 5.24,13.76 5.49,13.87 6.30,13.66 6.59,13.41 6.87,13.04 7.01,13.00 7.83,13.34 8.10,13.29 8.75,12.91 9.62,12.81 10.05,13.21 10.48,13.33 11.41,13.35 11.99,13.19 12.12,13.09 12.46,13.09 12.65,13.33 12.87,13.45 13.32,13.67 13.61,13.70
1.19,6.09 0.98,6.32 0.74,6.45 0.53,6.85 0.63,7.35 0.51,7.44 0.50,7.55 0.61,7.73 0.58,8.15 0.69,8.35 0.37,8.76 0.49,8.85 0.47,9.12 0.53,9.40 0.41,9.49 0.26,9.43 0.23,9.46 0.34,9.60 0.26,9.64 0.32,9.69 0.38,10.29 -0.09,10.67 0.01,11.02 -0.07,11.12
-2.70,9.48 -2.67,9.28 -2.75,9.05 -2.60,8.80 -2.51,8.21 -2.61,8.15 -2.61,8.05 -2.79,7.93 -2.99,7.20 -3.24,6.81 -3.24,6.65 -3.00,5.71 -2.96,5.64 -2.79,5.60 -2.76,5.36 -2.82,5.15 -3.11,5.09
-5.52,10.43 -5.46,10.77 -5.49,11.04 -5.35,11.13 -5.25,11.38 -5.29,11.83 -5.11,11.97 -4.70,12.08 -4.43,12.34 -4.42,12.58 -4.48,12.67 -4.23,12.79 -4.33,13.12 -4.15,13.31 -3.95,13.40 -3.53,13.18 -3.30,13.28 -3.25,13.66 -2.95,13.65 -2.87,13.95 -2.59,14.23 -2.46,14.27 -2.11,14.17 -1.97,14.46 -1.70,14.51 -1.05,14.82 -0.76,15.05 -0.54,15.08 -0.41,15.01 -0.24,15.06 0.22,14.91
-2.70,9.48 -2.82,9.43 -2.99,9.69 -3.22,9.90 -3.79,9.92 -4.27,9.74 -4.33,9.65 -4.63,9.71 -4.97,9.93 -5.10,10.24 -5.38,10.31 -5.52,10.43
-8.49,7.56 -8.30,6.98 -8.33,6.80 -8.60,6.51 -8.29,6.32 -7.89,6.23 -7.80,5.98 -7.45,5.84 -7.40,5.55 -7.43,5.32 -7.57,5.08 -7.59,4.82 -7.54,4.35
-8.49,7.56 -8.66,7.69 -8.89,7.26 -9.12,7.22 -9.26,7.38 -9.46,7.42 -9.37,7.70 -9.52,8.35 -9.68,8.48 -9.74,8.45 -9.78,8.54 -10.06,8.43 -10.15,8.52 -10.28,8.49
-5.52,10.43 -5.84,10.39 -6.03,10.19 -6.20,10.23 -6.26,10.72 -6.43,10.67 -6.42,10.56 -6.65,10.66 -6.69,10.35 -6.95,10.34 -7.02,10.14 -7.36,10.26 -7.50,10.44 -7.66,10.43 -7.81,10.24 -7.99,10.16
-15.04,10.94 -14.68,11.51 -14.27,11.66 -13.73,11.74 -13.74,12.01 -13.95,12.18 -13.71,12.31 -13.67,12.48 -13.73,12.67
-11.39,12.40 -11.50,12.20 -11.31,12.02 -10.93,12.21 -10.71,11.90 -10.47,12.14 -10.27,12.21 -9.75,12.03 -9.66,12.14 -9.36,12.26 -9.40,12.46 -9.22,12.48 -9.04,12.40 -8.82,11.92 -8.82,11.67 -8.40,11.37 -8.67,11.01 -8.47,11.05 -8.34,10.99 -8.27,10.49 -8.01,10.32 -7.99,10.16
107.52,14.71 107.41,14.56 107.29,14.59 106.94,14.33 106.78,14.34 106.50,14.58 106.35,14.45 106.23,14.48 106.17,14.37 105.98,14.34 106.12,14.05 106.07,13.92 105.90,13.92 105.74,14.08 105.53,14.16 105.35,14.11 105.18,14.35
114.06,4.60 114.22,4.48 114.32,4.26 114.42,4.26 114.57,4.05 114.65,4.04 114.78,4.17 114.84,4.39 114.75,4.72 115.03,4.90
-88.23,15.73 -88.96,15.15 -89.14,15.07 -89.22,14.87 -89.17,14.61 -89.36,14.42
-77.37,8.66 -77.48,8.50 -77.20,7.97 -77.35,7.84 -77.35,7.71 -77.54,7.57 -77.59,7.54 -77.76,7.70 -77.74,7.54 -77.90,7.23
-60.74,5.20 -61.39,5.94 -61.13,6.21 -61.20,6.59 -61.15,6.69 -60.87,6.79 -60.72,6.77 -60.35,7.00 -60.35,7.15 -60.58,7.16 -60.63,7.21 -60.61,7.32 -60.72,7.54 -60.51,7.81 -60.38,7.83 -60.03,8.05 -59.96,8.19 -59.85,8.25 -59.83,8.31 -60.02,8.55
-60.74,5.20 -60.67,5.16 -60.60,4.95 -61.00,4.54 -61.28,4.52 -61.55,4.29 -61.82,4.20 -62.15,4.10 -62.41,4.16 -62.71,4.02 -62.76,3.67 -62.86,3.59 -62.97,3.59 -63.34,3.94 -63.53,3.89 -64.02,3.93 -64.19,4.13 -64.58,4.14 -64.79,4.28 -64.82,4.23 -64.67,4.01 -64.22,3.59 -64.22,3.20 -64.01,2.67 -64.05,2.50 -63.39,2.41 -63.43,2.16 -64.01,1.93 -64.11,1.62 -64.21,1.53 -64.30,1.46 -64.49,1.45 -64.73,1.25 -65.10,1.11 -65.47,0.69 -65.56,0.69 -65.52,0.84 -65.68,0.98 -66.06,0.79 -66.35,0.77 -66.88,1.22
52.49,41.78 53.06,42.15 54.12,42.34 54.85,41.97 55.43,41.30 55.98,41.32
55.98,41.32 57.02,41.26 57.12,41.35 57.02,41.45 56.96,41.86 57.29,42.12 57.81,42.19 58.03,42.49 58.28,42.43 58.40,42.29 58.47,42.30 58.15,42.63 58.26,42.69 58.48,42.66 58.59,42.78 58.88,42.56 59.16,42.51 59.35,42.32 59.86,42.30 59.99,42.21 59.94,41.97 60.20,41.80 60.08,41.76 60.14,41.59 60.09,41.40 60.45,41.22 60.87,41.25 61.24,41.19 61.50,41.28 61.90,41.09 62.10,40.68 62.38,40.33 62.48,39.98 63.51,39.38 63.76,39.16 64.16,38.95 64.31,38.98 65.61,38.24 65.97,38.24 66.57,38.01 66.63,37.93 66.53,37.79 66.52,37.35
49.23,46.34 48.54,46.61 48.50,46.70 48.56,46.76 48.88,46.71 48.96,46.77 48.55,47.32 48.17,47.71 47.48,47.80 47.29,47.74 47.09,47.95 47.12,48.13 47.06,48.23 46.66,48.41 46.61,48.57 46.70,48.81 47.03,49.15 46.80,49.37 46.89,49.70 46.99,49.85 47.25,50.00 47.33,50.27 47.50,50.40 47.71,50.38 48.33,49.86 48.43,49.83 48.76,49.93 48.84,50.01 48.63,50.61 48.81,50.60 49.32,50.85 49.50,51.08 49.82,51.13 50.31,51.32 50.79,51.73 51.16,51.65 51.34,51.48 51.61,51.48 52.22,51.71 52.57,51.48 53.34,51.48 54.14,51.04 54.42,50.78 54.47,50.58 54.56,50.54 54.65,50.66 54.55,50.95 54.64,51.01 55.36,50.67 55.69,50.58 56.05,50.71 56.14,50.84 56.49,51.02 56.62,50.98 57.01,51.07 57.44,50.89 57.65,50.93 57.84,51.09 58.36,51.06 58.88,50.69 59.45,50.62 59.52,50.58 59.52,50.49 59.81,50.58 59.96,50.80 60.06,50.85 60.42,50.68 60.94,50.70 61.39,50.86 61.59,51.23 61.55,51.32 61.36,51.44 60.46,51.65 60.39,51.77 60.03,51.93 60.43,52.13 60.67,52.15 60.99,52.34 60.77,52.68 61.05,52.97 61.97,52.94 62.08,53.01 62.01,53.11 61.66,53.23 61.20,53.29 61.16,53.34 61.23,53.45 61.53,53.52 61.47,53.58 61.25,53.55 60.98,53.62 61.11,53.75 61.14,53.96 61.23,54.02 61.33,54.05 61.93,53.95 63.85,54.24 64.46,54.38 64.93,54.40 65.09,54.34 65.24,54.52 65.48,54.62 65.71,54.62 65.91,54.69 66.22,54.67 68.16,54.98 68.24,55.05 68.21,55.16 68.52,55.20 68.98,55.39 69.49,55.36 70.18,55.16 70.49,55.28 70.74,55.31 70.91,55.13 71.19,54.60 71.15,54.36 71.05,54.26 71.09,54.21 71.34,54.16 72.00,54.21 72.19,54.33 72.39,54.12 72.45,53.94 72.59,54.00 72.56,54.09 72.62,54.13 72.91,54.11 73.23,53.96 73.62,54.07 73.71,54.04 73.68,53.93 73.40,53.81 73.31,53.71 73.29,53.60 73.41,53.45 73.86,53.62 74.21,53.58 74.35,53.49 74.45,53.65 74.83,53.83 75.22,53.89 75.38,53.97 75.44,54.09 76.84,54.44 76.65,54.15 76.42,54.15 76.48,54.02 77.86,53.27 79.47,51.49 79.99,50.77 80.07,50.76 80.09,50.84 80.22,50.91 80.42,50.95 80.45,51.18 80.74,51.29 80.88,51.28 80.97,51.19 81.13,51.19 81.07,50.97 81.39,50.96 81.47,50.74 82.49,50.73 82.76,50.89 83.36,50.99 83.95,50.77 84.18,50.52 84.32,50.24 84.99,50.06 85.00,49.89 85.23,49.62 86.18,49.50 86.68,49.78 86.73,49.70 86.63,49.56 86.81,49.49 87.00,49.29 87.23,49.22 87.32,49.09
34.20,31.32 34.90,29.48
33.98,4.22 34.38,4.62
41.88,3.98 41.14,3.96 40.77,4.27 39.84,3.85 39.54,3.47 39.23,3.48 38.09,3.65 36.91,4.41 36.08,4.45 35.92,4.62
23.38,-17.64 21.42,-18.00 21.11,-17.96 20.75,-18.02 20.39,-17.89 18.96,-17.80 18.72,-17.70 18.40,-17.40 14.02,-17.41 13.48,-17.04 13.10,-16.97 12.55,-17.21 12.01,-17.17 11.74,-17.25
25.26,-17.79 25.64,-17.82 26.00,-17.97 26.14,-17.91 26.33,-17.93 26.78,-18.04 27.02,-17.96 27.93,-16.90 28.76,-16.53 28.83,-16.42 28.91,-15.99 29.49,-15.70 30.40,-15.64
23.38,-17.64 24.27,-17.48 24.73,-17.52 25.00,-17.57 25.26,-17.79
31.29,-22.40 32.43,-21.30 32.35,-21.14 32.48,-20.95 32.49,-20.66 32.67,-20.52 32.99,-19.98 33.01,-19.87 32.78,-19.39 32.85,-19.10 32.70,-18.94 32.72,-18.83 32.88,-18.73 32.99,-18.36 32.97,-17.25 32.88,-16.88 32.95,-16.71 32.24,-16.45 31.94,-16.43 31.69,-16.21 31.49,-16.18 31.24,-16.02 30.44,-16.00 30.40,-15.64
12.28,-6.00 12.42,-6.05 13.18,-5.86 16.43,-5.90 16.70,-6.16 16.74,-6.62 16.92,-6.93 16.98,-7.26 17.58,-8.10 18.01,-8.11 18.56,-7.94 18.94,-8.00 19.34,-7.97 19.53,-7.14 19.66,-7.04 19.88,-6.99 20.59,-6.92 20.54,-7.18 20.61,-7.28 21.78,-7.31 21.84,-7.42 21.80,-8.11 21.90,-8.34 21.91,-8.69 21.81,-9.47 22.20,-10.04 22.27,-10.26 22.31,-10.69 22.18,-10.89 22.23,-11.12 22.31,-11.20 22.56,-11.06 23.08,-11.09 23.46,-10.97 23.83,-11.01 23.97,-10.87
-16.56,13.59 -15.51,13.59 -15.43,13.73 -15.11,13.81 -14.94,13.79 -14.41,13.50 -13.98,13.54 -13.83,13.41 -13.85,13.34 -14.25,13.24 -15.15,13.56 -15.29,13.40 -15.81,13.33 -15.83,13.16 -16.65,13.15 -16.76,13.06
0.22,14.91 0.16,14.50 0.38,14.25 0.35,14.14 0.43,13.97 0.62,13.70 0.90,13.61 1.20,13.36 0.99,13.36 0.99,13.04 1.56,12.64 1.79,12.61 1.96,12.71 2.10,12.70 2.23,12.47 2.07,12.38 2.07,12.31 2.39,11.90
18.61,3.48 18.47,3.62 18.16,3.50 17.49,3.69 16.61,3.51 16.48,3.17 16.47,2.83 16.18,2.27
16.18,2.26 16.08,2.11 16.14,1.72 16.06,1.68 15.74,1.92 15.34,1.94 15.16,2.04 14.90,2.01 14.88,2.08 14.76,2.08 14.58,2.20 14.48,2.15 13.29,2.16
15.48,7.52 15.21,7.21 14.74,6.28 14.51,6.16 14.43,6.04 14.62,5.87 14.56,5.28 14.64,5.18 14.73,4.60 15.06,4.28 15.14,4.04 15.03,4.02 15.13,3.83 15.77,3.13 15.90,3.10 16.06,2.91 16.18,2.26
33.98,4.22 33.49,3.76 33.15,3.77 33.00,3.88 32.34,3.71 32.14,3.52 31.94,3.61 31.80,3.80 31.55,3.68 31.15,3.79 30.93,3.63 30.84,3.49
13.07,-4.63 12.83,-4.74 12.45,-5.07 12.52,-5.15 12.50,-5.70 12.21,-5.76
31.95,-25.96 31.99,-24.46 31.80,-23.89 31.55,-23.48 31.29,-22.40
33.20,-14.01 30.23,-14.99 30.40,-15.64
34.96,-11.58 34.62,-11.62 34.36,-12.16 34.56,-13.36 34.66,-13.49 34.91,-13.55 35.25,-13.90 35.85,-14.67 35.89,-14.89 35.81,-15.27 35.82,-15.68 35.76,-16.06 35.36,-16.16 35.17,-16.56 35.28,-16.81 35.27,-17.12 35.06,-17.08 35.08,-16.83 34.93,-16.76 34.42,-16.25 34.40,-16.08 34.25,-15.89 34.54,-15.30 34.51,-14.60 34.33,-14.41 33.64,-14.57 33.20,-14.01
29.58,-1.39 29.56,-0.98 29.72,0.10 29.93,0.50 29.94,0.82 30.18,0.97 30.32,1.19 30.48,1.24 31.25,2.04 31.27,2.15 31.18,2.27 30.96,2.40 30.73,2.46 30.85,2.89 30.75,3.04 30.91,3.41 30.84,3.49
36.52,14.26 36.43,15.13 36.52,15.25 36.91,16.30 36.89,16.62 36.98,16.80 37.01,17.06 37.41,17.06 37.55,17.32 37.78,17.46 38.25,17.58 38.61,18.01
42.38,12.47 42.13,12.77 41.95,12.88 41.77,13.18 41.36,13.50 40.82,14.11 40.14,14.46 39.90,14.44 39.53,14.54 39.20,14.48 39.02,14.63 38.81,14.48 38.43,14.43 38.22,14.65 37.88,14.85 37.57,14.15 37.26,14.45 37.02,14.27 36.81,14.32 36.52,14.26
27.35,57.53 27.40,57.67 27.54,57.80 27.78,57.87 27.67,57.93 27.50,58.22 27.53,58.44 27.43,58.79 27.76,59.05 27.90,59.28 28.15,59.37 28.01,59.48
74.31,32.81 74.66,32.76 74.69,32.49 74.99,32.46 75.30,32.32
-84.35,10.98 -84.70,11.05 -84.91,10.95 -85.58,11.19 -85.74,11.06
-82.88,8.07 -83.03,8.34 -82.86,8.45 -82.86,8.64 -82.92,8.74 -82.73,8.92 -82.94,9.06 -82.94,9.45 -82.80,9.59 -82.64,9.51 -82.56,9.58
-66.88,1.22 -67.21,2.39 -67.62,2.79 -67.86,2.79 -67.83,2.89 -67.31,3.42 -67.66,3.86 -67.86,4.51 -67.82,5.27 -67.64,5.56 -67.58,5.83 -67.47,5.93 -67.48,6.18 -67.86,6.29 -68.47,6.16 -68.94,6.20 -69.27,6.10 -69.43,6.12 -70.13,6.95 -70.74,7.09 -71.13,6.99 -72.01,7.03 -72.21,7.37 -72.39,7.42 -72.47,7.52 -72.45,7.97 -72.36,8.09 -72.39,8.29 -72.67,8.63 -72.80,9.11 -72.96,9.14 -73.06,9.26 -73.37,9.19 -73.01,9.79 -72.87,10.49 -72.69,10.84 -72.45,11.11 -72.25,11.20 -71.96,11.67 -71.32,11.86
-68.44,-52.36 -68.46,-52.29 -69.96,-52.01 -71.97,-51.96 -71.95,-51.88 -72.41,-51.54 -72.30,-51.30 -72.38,-51.10 -72.28,-50.91 -72.34,-50.68 -72.51,-50.61 -73.15,-50.74 -73.31,-50.36 -73.50,-50.13 -73.48,-49.77
-68.63,-52.65 -68.65,-54.92 -68.06,-54.88 -67.35,-54.93 -66.48,-55.11
22.77,54.36 19.60,54.46
48.44,28.54 47.67,28.53 47.43,28.99 46.53,29.10
44.77,37.14 44.88,36.80 45.02,36.70 45.05,36.47 45.24,36.36 45.36,36.02 45.56,35.98 45.78,35.82 46.27,35.77 46.00,35.61 45.98,35.48 46.11,35.32 46.13,35.13 45.92,35.03 45.68,34.80 45.64,34.57 45.50,34.58 45.44,34.42 45.54,34.22 45.40,33.97 45.74,33.60 45.88,33.61 45.87,33.49 46.02,33.42 46.15,33.23 46.08,33.09 46.11,32.96 46.38,32.93 47.12,32.47 47.37,32.42 47.51,32.15 47.83,31.79 47.68,31.40 47.68,31.00 48.01,30.99 48.01,30.47 48.33,30.29 48.43,30.04 48.55,29.96
61.26,35.62 61.25,35.87 61.15,35.98 61.21,36.19 61.12,36.64 60.34,36.64 60.06,36.96 59.45,37.25 59.30,37.51 58.82,37.68 58.26,37.67 57.98,37.83 57.35,37.97 57.31,38.13 57.19,38.22 56.44,38.25 56.32,38.19 56.27,38.08 55.58,38.10 55.38,38.05 54.90,37.78 54.70,37.47 54.19,37.33 53.91,37.34
140.97,-2.60 140.97,-6.35 140.86,-6.74 140.98,-6.91 140.98,-9.12
9.52,30.23 9.31,30.12 9.64,29.64 9.81,29.18 9.92,27.79 9.75,27.33 9.89,26.85 9.88,26.63 9.86,26.55 9.49,26.33 9.45,26.07 10.00,25.33 10.03,25.05 10.26,24.59 10.40,24.49 10.69,24.55 11.51,24.31 11.97,23.52
-13.73,12.67 -15.20,12.68 -15.57,12.49 -16.71,12.35
-11.39,12.40 -12.40,12.34 -12.93,12.53 -13.06,12.49 -13.08,12.63 -13.73,12.67
0.90,10.99 0.49,10.95 -0.07,11.12
-139.06,60.00 -139.19,60.08 -139.08,60.34 -139.68,60.33 -139.97,60.18 -140.45,60.30 -140.53,60.22 -141.00,60.30 -141.00,69.65
-130.62,54.71 -130.21,55.03 -130.02,55.28 -130.12,55.69 -130.03,55.89 -130.06,56.07 -130.41,56.12 -130.48,56.23 -130.74,56.34 -131.58,56.60 -131.82,56.59 -131.87,56.79 -132.10,56.86 -132.03,57.03 -132.34,57.08 -132.23,57.20 -133.42,58.34 -133.40,58.41 -133.82,58.71 -134.30,58.90 -134.44,59.09 -134.94,59.29 -135.07,59.44 -135.05,59.58 -135.48,59.79 -136.32,59.60 -136.25,59.53 -136.35,59.46 -136.47,59.46 -136.47,59.28 -136.58,59.15 -136.81,59.15 -137.44,58.90 -137.52,58.92 -137.48,58.99 -137.59,59.23 -138.63,59.78 -138.71,59.90 -139.06,60.00
-123.31,48.99 -122.69,48.99
-88.89,15.89 -89.24,15.89 -89.16,17.81
-75.28,-0.11 -75.48,-0.16 -75.63,-0.12 -75.26,-0.59 -75.25,-0.95 -75.41,-0.92 -75.57,-1.53 -76.09,-2.13 -76.68,-2.56 -77.86,-2.98 -78.18,-3.35 -78.16,-3.47 -78.23,-3.49 -78.35,-3.40 -78.42,-3.78 -78.68,-4.33 -78.69,-4.56 -78.91,-4.71 -78.92,-4.86 -79.03,-4.97 -79.33,-4.93 -79.50,-4.67 -79.52,-4.54 -79.64,-4.45 -79.80,-4.48 -80.14,-4.30 -80.38,-4.46 -80.48,-4.43 -80.35,-4.21 -80.49,-4.17 -80.49,-4.01 -80.30,-4.01 -80.18,-3.88 -80.32,-3.39
-69.97,-4.24 -70.34,-3.81 -70.53,-3.87 -70.74,-3.78 -70.07,-2.75 -70.10,-2.66 -70.97,-2.21 -71.40,-2.33 -71.75,-2.15 -71.98,-2.33 -72.22,-2.40 -72.40,-2.43 -72.63,-2.35 -72.94,-2.39 -73.15,-2.28 -73.13,-2.08 -73.20,-1.83 -73.50,-1.69 -73.52,-1.45 -73.66,-1.25 -74.25,-0.97 -74.42,-0.58 -74.80,-0.20 -74.95,-0.19 -75.14,-0.05 -75.28,-0.11
-66.88,1.22 -67.08,1.19 -67.12,1.70 -67.40,2.12 -67.56,2.07 -67.82,1.79 -67.94,1.75 -68.19,1.99 -68.26,1.85 -68.18,1.72 -69.32,1.72 -69.54,1.77 -69.85,1.71 -69.85,1.06 -69.31,1.05 -69.16,0.86 -69.15,0.66 -69.28,0.63 -69.47,0.73 -70.05,0.58 -70.07,-0.14 -69.92,-0.32 -69.63,-0.51 -69.61,-0.76 -69.45,-1.00 -69.40,-1.19 -69.97,-4.24
-69.58,-10.95 -68.69,-12.50 -68.76,-12.69 -68.98,-12.88 -68.98,-13.50 -69.07,-13.68 -68.87,-14.17 -69.00,-14.27 -69.01,-14.38 -69.36,-14.80 -69.37,-14.96 -69.17,-15.24 -69.42,-15.64 -69.22,-16.15 -68.84,-16.34 -69.03,-16.48 -69.02,-16.64 -69.20,-16.77 -69.44,-17.09 -69.62,-17.20 -69.51,-17.50
-69.58,-10.95 -68.85,-11.01 -68.73,-11.12 -68.62,-11.11 -68.31,-10.98 -68.07,-10.70 -67.72,-10.68 -67.42,-10.39 -67.11,-10.27 -66.58,-9.90 -65.92,-9.79 -65.56,-9.80 -65.40,-9.71 -65.31,-9.87 -65.30,-10.15 -65.45,-10.51 -65.32,-11.02 -65.39,-11.25 -65.18,-11.65 -65.19,-11.75 -65.09,-11.74 -64.99,-11.98 -64.51,-12.25 -64.42,-12.44 -63.94,-12.53 -63.69,-12.48 -63.35,-12.68 -63.07,-12.67 -63.02,-12.81 -62.77,-13.00 -62.12,-13.16 -61.79,-13.53 -61.08,-13.49 -60.51,-13.79 -60.40,-14.02 -60.47,-14.18 -60.30,-14.62 -60.27,-15.09 -60.58,-15.10 -60.24,-15.48 -60.18,-16.27 -58.54,-16.33 -58.35,-16.28 -58.35,-16.49 -58.48,-16.70 -58.40,-17.23 -57.99,-17.51 -57.83,-17.51 -57.59,-18.12 -57.50,-18.21 -57.57,-18.28 -57.78,-18.91 -57.73,-18.92 -57.72,-19.04 -57.80,-19.08 -58.13,-19.74 -57.86,-19.98 -58.16,-20.16
-69.97,-4.24 -70.05,-4.33 -70.24,-4.30 -70.40,-4.15 -70.80,-4.17 -70.97,-4.35 -71.84,-4.50 -72.35,-4.79 -72.70,-5.07 -72.89,-5.12 -72.98,-5.63 -73.24,-6.10 -73.14,-6.47 -73.76,-6.91 -73.80,-7.08 -73.72,-7.31 -73.96,-7.38 -74.00,-7.56 -73.72,-7.78 -73.78,-7.94 -73.64,-8.07 -73.55,-8.35 -73.36,-8.48 -73.30,-8.65 -72.97,-8.99 -72.97,-9.12 -73.21,-9.41 -72.81,-9.41 -72.38,-9.51 -72.17,-9.84 -72.14,-10.01 -71.24,-9.97 -70.54,-9.44 -70.64,-9.82 -70.64,-11.01 -70.53,-10.95 -70.29,-11.06 -69.96,-10.93 -69.58,-10.95
-62.65,-22.23 -62.28,-21.07 -62.28,-20.56 -61.92,-20.06 -61.76,-19.65 -60.01,-19.30 -59.09,-19.29 -58.18,-19.82 -58.16,-20.16
-17.05,20.81 -16.96,21.33 -13.02,21.33 -13.15,22.82 -13.03,23.00 -12.62,23.27 -12.02,23.47 -12.02,26.00 -8.68,26.00 -8.68,27.29
-8.68,27.29 -4.82,25.00
-8.68,27.66 -8.66,28.72 -7.69,29.35 -7.49,29.39 -7.16,29.61 -6.64,29.57 -6.52,29.66 -6.48,29.82 -6.00,29.83 -5.45,29.96 -5.18,30.17 -4.97,30.47 -4.32,30.70 -3.99,30.91 -3.67,30.96 -3.62,31.07 -3.83,31.20 -3.79,31.36 -3.83,31.66 -3.02,31.83 -2.86,32.07 -2.45,32.13 -1.28,32.09
-8.68,27.29 -8.68,27.66
-12.28,14.81 -12.86,15.24 -12.93,15.45 -13.21,15.62 -13.41,16.06 -13.51,16.14 -13.87,16.15 -14.09,16.42 -14.30,16.58 -14.53,16.66 -14.99,16.68 -15.77,16.49 -16.24,16.53 -16.44,16.20 -16.54,15.84
-4.82,25.00 -6.59,24.99 -5.63,16.57 -5.36,16.28 -5.51,15.50 -9.18,15.50 -9.34,15.53 -9.35,15.68 -9.43,15.62 -9.45,15.46 -9.94,15.37 -10.70,15.42 -10.95,15.15 -11.50,15.64 -11.76,15.43 -11.94,14.89 -12.10,14.75 -12.28,14.81
4.23,19.14 3.40,18.99 3.26,19.01 3.11,19.15 3.26,19.41 3.20,19.79 3.13,19.85 2.41,20.06 2.22,20.25 1.75,20.33 1.61,20.56 1.17,20.82 1.15,21.10 -4.82,25.00
4.23,19.14 5.84,19.48 7.48,20.87 11.97,23.52
9.52,30.23 9.04,32.07 8.33,32.54 8.11,33.06 7.73,33.27 7.50,33.83 7.51,34.08 7.75,34.25 7.84,34.41 8.12,34.56 8.25,34.73 8.28,34.98 8.39,35.20 8.25,35.80 8.35,36.37 8.21,36.52 8.44,36.76 8.60,36.83 8.58,36.94
23.98,19.50 15.98,23.45 14.98,23.00
24.98,22.00 24.98,20.00 23.98,20.00 23.98,19.50
25.15,31.65 24.85,31.33 24.96,30.68 24.70,30.20 24.98,29.18 24.98,22.00
22.86,10.92 22.92,11.34 22.59,11.58 22.58,11.99 22.47,12.07 22.35,12.66 22.23,12.71 21.93,12.68 21.83,12.79 21.99,13.11 22.23,13.33 22.11,13.80 22.17,13.91 22.54,14.16 22.38,14.55 22.67,14.72 22.76,15.00 22.93,15.16 22.93,15.53 23.11,15.70 23.97,15.72 23.98,19.50
24.98,22.00 31.21,21.99 31.40,22.20 31.49,22.15 31.43,22.00 36.87,22.00
41.88,3.98 41.34,3.20 40.96,2.81 40.98,-0.87 41.52,-1.57 41.53,-1.70
42.92,11.00 42.66,10.60 42.84,10.20 43.07,9.93 43.18,9.88 43.48,9.38 43.62,9.34 43.98,9.01 46.98,8.00 47.98,8.00
30.75,-8.19 28.90,-8.49 28.93,-8.59 28.87,-8.79 28.40,-9.22 28.60,-9.68 28.65,-10.55 28.36,-11.48 28.38,-11.57 28.48,-11.81 29.06,-12.35 29.49,-12.42 29.51,-12.23 29.80,-12.16 29.78,-13.44 29.65,-13.41 29.63,-13.30 29.55,-13.25 29.20,-13.40 29.01,-13.37 28.73,-12.93 28.55,-12.84 28.41,-12.52 27.57,-12.23 27.42,-11.94 27.24,-11.78 27.16,-11.58 27.05,-11.62 26.95,-11.90 26.82,-11.97 26.03,-11.89 25.51,-11.75 25.35,-11.62 25.29,-11.21 24.52,-11.44 24.38,-11.42 24.34,-11.37 24.40,-11.26 24.37,-11.13 24.14,-11.03 24.08,-10.89 23.97,-10.87
23.38,-17.64 22.15,-16.60 22.04,-16.26 21.98,-15.96 21.98,-13.00 23.96,-12.99 23.88,-12.80 23.99,-12.42 23.97,-11.64 24.05,-11.41 23.97,-10.87
33.90,-1.00 33.94,0.17 34.16,0.61 34.41,0.87 34.48,1.04 34.80,1.24 34.78,1.38 34.97,1.64 34.98,1.77 34.91,2.48 34.74,2.82 34.59,2.92 34.52,3.12 34.45,3.16 34.40,3.41 34.44,3.65 34.17,3.81 34.19,3.87 33.98,4.22
33.90,-1.00 37.64,-3.05 37.69,-3.25 37.62,-3.51 39.22,-4.69
11.33,2.17 9.98,2.17 9.80,2.30
11.33,2.17 11.34,1.00 10.03,1.00 9.91,0.96 9.76,1.07 9.59,1.03
25.26,-17.79 24.91,-17.82 24.53,-18.05 24.36,-17.98 24.24,-18.02 23.60,-18.46 23.30,-18.03 23.22,-18.00 20.97,-18.32 20.98,-21.96 19.98,-22.00 19.98,-24.75
19.98,-24.78 19.98,-28.45 19.54,-28.57 19.31,-28.73 19.25,-28.90 19.16,-28.94 18.84,-28.87 18.10,-28.87 17.45,-28.70 17.35,-28.50 17.36,-28.27 17.06,-28.03 16.88,-28.13 16.76,-28.45 16.45,-28.62
19.98,-24.75 19.98,-24.78
0.90,10.99 0.79,10.71 0.76,10.39 1.33,10.00 1.39,9.36 1.60,9.05 1.62,7.00 1.53,6.99 1.60,6.61 1.78,6.29 1.62,6.22
42.80,16.37 43.17,16.69 43.18,16.81 43.12,16.94 43.16,17.21 43.24,17.27 43.19,17.36 43.42,17.52 43.92,17.32 44.16,17.40 45.15,17.43 45.54,17.30 46.31,17.23 46.73,17.27 46.98,16.95 47.14,16.95 47.44,17.11 47.58,17.45 48.17,18.16 49.04,18.58 51.98,19.00
51.98,19.00 52.73,17.30 52.80,17.27 53.09,16.65
55.19,22.70 55.64,22.00 54.98,20.00 51.98,19.00
55.19,22.70 55.10,22.62 52.56,22.93 51.59,24.08 51.57,24.29
42.36,37.11 41.79,36.60 41.42,36.51 41.30,36.38 41.25,36.07 41.35,35.64 41.22,35.29 41.19,34.77 40.99,34.43 40.69,34.33 38.77,33.37
35.79,32.73 36.37,32.39 36.82,32.32 38.77,33.37
35.61,32.68 35.79,32.73
39.15,32.12 39.29,32.24 39.25,32.35 39.04,32.31 38.98,32.47 39.06,32.49 38.77,33.37
34.95,29.35 36.07,29.20 36.48,29.50 36.76,29.87 37.47,30.00 37.65,30.33 37.98,30.50 36.96,31.49 38.96,31.99 39.15,32.12
46.53,29.10 46.36,29.06 44.69,29.20 42.07,31.08 40.37,31.94 39.15,32.12
61.59,25.20 61.66,25.75 61.75,25.84 61.84,26.23 62.13,26.37 62.24,26.36 62.31,26.49 62.44,26.56 63.16,26.65 63.30,27.15 63.17,27.25 62.76,27.25 62.81,27.50 62.76,28.24 62.56,28.24 62.35,28.41 61.89,28.55 61.62,28.79 61.32,29.37 60.84,29.86
60.84,29.86 61.81,30.91 61.76,31.29 61.66,31.38 60.82,31.50 60.83,32.25 60.56,33.06 60.56,33.14 60.92,33.51 60.51,33.64 60.49,34.09 60.64,34.31 60.89,34.32 60.73,34.52 60.95,34.65 61.08,34.86 61.15,35.09 61.10,35.27 61.19,35.31 61.28,35.51 61.26,35.62
73.73,36.89 74.04,36.83 74.54,37.02
60.84,29.86 62.48,29.41 63.57,29.50 64.10,29.39 64.39,29.54 65.10,29.56 66.18,29.84 66.31,29.97 66.24,30.11 66.35,30.80 66.40,30.91 66.60,31.02 66.83,31.26 66.92,31.31 67.45,31.23 67.74,31.34 67.58,31.51 68.02,31.68 68.16,31.80 68.44,31.75 68.60,31.80 68.87,31.63 69.08,31.74 69.28,31.94 69.24,32.43 69.41,32.68 69.50,33.02 69.92,33.11 70.26,33.29 70.28,33.37 70.13,33.62 69.87,33.90 69.89,34.01 69.99,34.05 70.33,33.96 70.65,33.95 71.05,34.05 71.10,34.37 70.97,34.53 71.62,35.18 71.55,35.29 71.60,35.41 71.57,35.55 71.40,35.88 71.19,36.04 71.23,36.12 71.62,36.44 71.77,36.43 72.25,36.73 72.62,36.83 73.73,36.89
70.95,42.25 70.72,42.17 70.58,42.04 70.42,42.08 70.10,41.82 69.15,41.43 68.99,41.21 68.58,40.88 68.57,40.62 68.42,40.62 68.05,40.81 68.11,41.03 67.94,41.20 66.71,41.18 66.50,41.99 66.01,42.00 66.10,42.99 65.80,42.88 65.50,43.31 64.91,43.71 64.44,43.55 63.21,43.63 61.99,43.49 61.16,44.17 61.01,44.39 58.56,45.56 55.98,44.99 55.98,41.32
13.61,13.70 13.45,14.38 14.37,15.75 15.47,16.91 15.74,19.90 15.96,20.35 15.59,20.73 15.54,20.87 15.61,20.95 15.18,21.52 15.17,21.92 14.98,23.00
11.97,23.52 13.48,23.18 14.22,22.62 14.98,23.00
0.22,14.91 0.29,14.98 0.95,14.98 1.30,15.27 3.00,15.34 3.06,15.43 3.50,15.36 3.52,15.48 3.84,15.70 4.12,16.36 4.23,17.00 4.23,19.14
-0.07,11.12 -0.30,11.17 -0.63,10.93 -0.70,10.99 -1.04,11.01 -2.83,11.00 -2.91,10.59 -2.79,10.43 -2.78,9.75 -2.70,9.48
-13.29,9.05 -13.03,9.10 -12.96,9.26 -12.76,9.37 -12.50,9.86 -12.43,9.90 -12.28,9.93 -12.14,9.88 -11.92,9.92 -11.91,9.99 -11.27,10.00 -10.69,9.31 -10.75,9.10 -10.62,9.06 -10.61,8.87 -10.50,8.69 -10.71,8.34 -10.56,8.32 -10.39,8.48 -10.28,8.49
5.79,49.54 6.01,49.45 6.18,49.50 6.34,49.45
56.08,26.06 56.17,26.05 56.14,25.69 56.30,25.65
30.51,-1.07 30.36,-1.07 30.10,-1.37 29.93,-1.47 29.83,-1.34 29.58,-1.39
30.51,-1.07 30.51,-1.21 30.81,-1.56 30.88,-2.14 30.83,-2.34 30.55,-2.40
33.90,-1.00 30.81,-0.99 30.51,-1.07
34.59,-11.97 34.54,-12.02 34.62,-12.07 34.59,-11.97
87.32,49.09 87.82,49.16
45.50,40.66 45.48,40.61 45.55,40.62 45.50,40.66
44.99,41.09 45.00,41.02 44.99,41.09
12.44,43.98 12.40,43.93 12.49,43.90 12.50,43.99 12.44,43.98
70.57,41.02 70.65,40.94 70.57,41.02
-117.13,32.53 -114.72,32.72 -114.84,32.51 -111.04,31.32 -108.21,31.33 -108.21,31.78 -106.45,31.77 -106.15,31.45 -104.92,30.58 -104.68,30.13 -104.62,29.85 -104.40,29.57 -103.26,29.00 -103.09,29.04 -102.89,29.22 -102.73,29.64 -102.34,29.86 -101.99,29.80 -101.55,29.81 -101.38,29.74 -100.75,29.18 -100.33,28.50 -100.30,28.33 -99.75,27.73 -99.51,27.55 -99.46,27.06 -99.23,26.76 -99.11,26.45 -98.28,26.11 -97.80,26.04 -97.38,25.87 -97.15,25.96
-92.24,14.55 -92.16,14.69 -92.16,14.96 -92.07,15.07 -92.20,15.28 -91.74,16.07 -90.45,16.07 -90.42,16.39 -90.63,16.51 -90.71,16.71 -91.41,17.26 -90.99,17.25 -90.99,17.82 -89.16,17.81 -89.13,17.97 -89.05,18.00 -88.90,17.91 -88.81,17.97 -88.46,18.48 -88.04,18.42 -88.04,18.20 -87.83,18.18
-54.62,-25.58 -54.61,-25.43 -54.44,-25.12 -54.28,-24.31 -54.32,-24.13 -54.24,-24.05 -54.63,-23.81 -55.19,-24.02 -55.42,-23.95 -55.65,-22.89 -55.62,-22.67 -55.85,-22.31 -56.25,-22.26 -56.45,-22.08 -56.63,-22.23 -56.94,-22.27 -57.57,-22.18 -57.72,-22.10 -57.96,-22.11 -57.95,-21.49 -57.83,-21.00 -58.16,-20.16
-62.65,-22.23 -62.21,-22.61 -61.80,-23.18 -61.21,-23.56 -61.03,-23.76 -60.51,-23.96 -59.89,-24.09 -59.19,-24.56 -58.37,-24.96 -57.96,-25.05 -57.64,-25.33 -57.57,-25.53 -57.75,-25.70 -57.89,-26.01 -58.11,-26.18 -58.20,-26.38 -58.19,-26.63 -58.32,-26.86 -58.62,-27.13 -58.60,-27.31 -58.17,-27.27 -57.11,-27.47 -56.60,-27.47 -56.44,-27.55 -56.16,-27.32 -55.71,-27.41 -55.43,-27.01 -55.14,-26.93 -54.83,-26.65 -54.68,-26.31 -54.62,-25.58
-62.65,-22.23 -62.84,-22.00 -63.92,-22.03 -64.27,-22.60 -64.33,-22.83 -64.61,-22.23 -64.99,-22.11 -65.77,-22.10 -66.22,-21.80 -66.37,-22.11 -66.71,-22.22 -66.80,-22.41 -67.03,-22.55 -67.19,-22.82
26.62,48.26 26.85,48.39 27.23,48.37 27.55,48.48 27.82,48.42 28.09,48.26 28.29,48.24 28.46,48.09 28.53,48.15 28.77,48.12 28.92,47.95 29.13,47.96 29.21,47.78 29.13,47.49 29.54,47.27 29.51,47.13 29.57,46.96 29.88,46.83 29.94,46.72 29.92,46.54 30.13,46.42 29.84,46.35 29.71,46.45 29.61,46.40 29.30,46.47 29.20,46.38 29.19,46.52 28.96,46.46 28.93,46.36 29.01,46.18 28.95,46.05 28.74,45.94 28.73,45.85 28.49,45.67 28.50,45.52 28.21,45.45
67.76,37.17 67.96,36.97 68.07,36.95 68.91,37.33 69.30,37.12 69.41,37.21 69.40,37.40 69.49,37.55 69.82,37.61 70.04,37.55 70.19,37.58 70.25,37.66 70.21,37.92 70.62,38.33 70.88,38.46 71.26,38.31 71.33,38.17 71.28,37.92 71.58,37.91 71.43,37.13 71.53,36.85 71.67,36.70 71.80,36.69 72.36,36.98 72.66,37.03 72.90,37.27 73.38,37.46 73.72,37.42 73.65,37.24 74.17,37.33 74.35,37.42 74.66,37.39 74.89,37.23
66.52,37.35 66.83,37.37 67.07,37.33 67.20,37.24 67.52,37.27 67.76,37.17
61.26,35.62 61.34,35.63 61.62,35.43 61.98,35.44 62.31,35.17 62.46,35.25 62.69,35.26 63.06,35.45 63.17,35.68 63.13,35.85 64.01,36.01 64.51,36.34 64.82,37.13 65.09,37.24 65.55,37.25 65.64,37.47 65.77,37.57 66.11,37.41 66.52,37.35
92.57,21.98 92.49,22.69 92.36,22.93 92.25,23.68 91.93,23.69 91.94,23.50 91.75,23.29 91.75,23.05 91.62,22.98 91.51,23.03 91.44,23.20 91.37,23.20 91.37,23.07 91.32,23.10 91.16,23.66 91.23,23.92 91.37,24.09 91.88,24.20 91.95,24.36 92.10,24.41 92.23,24.88 92.44,24.85 92.47,24.94 92.05,25.17 90.44,25.16 89.83,25.29 89.82,25.94 89.67,26.21 89.59,26.19 89.55,26.01 89.37,26.01 89.19,26.11 89.02,26.41 88.92,26.38 88.97,26.25 88.83,26.25 88.68,26.29 88.62,26.43 88.42,26.57 88.35,26.50 88.44,26.37 88.15,26.09 88.08,25.89 88.50,25.54 88.77,25.49 88.95,25.26 88.82,25.18 88.46,25.19 88.31,24.88 88.15,24.91 88.02,24.63 88.15,24.49 88.29,24.48 88.72,24.27 88.70,24.00 88.57,23.67 88.74,23.44 88.72,23.25 88.93,23.19 88.85,23.04 89.05,22.09
2.39,11.90 2.37,12.22 2.81,12.38 3.60,11.70
18.61,3.48 18.62,3.30 18.49,2.92 18.21,2.41 18.07,2.01 18.06,1.53 17.90,1.12 17.89,0.23 17.72,-0.28 17.75,-0.55 17.28,-1.00 16.88,-1.23 16.54,-1.84 16.22,-2.18 16.22,-3.03 16.15,-3.46 15.99,-3.77 15.87,-3.93 15.60,-4.03 15.27,-4.31 14.71,-4.88 14.41,-4.83 14.37,-4.59 14.45,-4.45 14.36,-4.30 13.94,-4.48 13.72,-4.45 13.66,-4.72 13.41,-4.84 13.14,-4.60 13.07,-4.63
28.98,-30.00 29.10,-29.92 29.14,-29.70 29.29,-29.57 29.39,-29.27 29.30,-29.09 28.95,-28.88 28.63,-28.58 27.74,-28.94 27.29,-29.52 27.06,-29.63 27.39,-30.32 27.75,-30.60 28.06,-30.63 28.39,-30.15 28.65,-30.13 28.98,-30.00
29.36,-22.19 29.66,-22.15 30.19,-22.29 30.92,-22.29 31.29,-22.40
19.98,-24.78 20.43,-25.15 20.79,-25.92 20.82,-26.16 20.63,-26.44 20.69,-26.82 21.69,-26.84 21.79,-26.71 22.09,-26.58 22.22,-26.39 22.60,-26.13 22.88,-25.46 23.06,-25.31 23.39,-25.29 23.89,-25.60 24.19,-25.63 24.33,-25.74 24.75,-25.82 25.44,-25.71 25.58,-25.61 25.91,-24.75 26.40,-24.61 26.84,-24.24 26.99,-23.70 27.09,-23.58 27.76,-23.20 28.21,-22.69 28.38,-22.59 28.84,-22.48 29.13,-22.21 29.36,-22.19
34.96,-11.58 35.56,-11.60 35.79,-11.45 35.91,-11.45 36.31,-11.71 36.67,-11.68 36.98,-11.57 37.37,-11.71 37.72,-11.58 37.92,-11.29 38.18,-11.28 38.49,-11.41 38.79,-11.23 39.32,-11.12 39.99,-10.82 40.46,-10.46
-84.35,10.98 -84.17,10.78 -83.92,10.74 -83.71,10.79 -83.64,10.92
28.15,56.14 27.90,56.08 27.64,55.91 27.58,55.80 27.05,55.83 26.59,55.67
28.21,45.45 28.45,45.29 28.76,45.23 28.78,45.31 29.40,45.42 29.71,45.26
21.36,44.83 21.91,44.67 22.09,44.54 22.50,44.71 22.64,44.65 22.73,44.57 22.55,44.54 22.49,44.44 22.71,44.24
56.23,25.30 56.24,25.21 56.29,25.28 56.23,25.30
32.11,-26.84 32.89,-26.85
71.74,39.98 71.67,39.95 71.75,39.91 71.79,40.00 71.74,39.98
-122.69,48.99 -95.16,48.99 -95.16,49.37 -94.85,49.30 -94.80,49.00 -94.62,48.74 -94.06,48.66 -93.71,48.53 -93.38,48.62 -93.00,48.61 -92.50,48.44 -92.41,48.28 -92.17,48.34 -92.01,48.30 -91.52,48.06 -90.92,48.21 -90.74,48.10 -90.09,48.12 -89.90,48.00 -89.46,48.00 -88.38,48.30 -84.88,46.90 -84.78,46.64 -84.56,46.46 -84.15,46.54 -84.09,46.23 -83.98,46.08 -83.62,46.12 -83.47,45.99 -83.59,45.82 -82.55,45.35 -82.14,43.57 -82.41,43.07 -82.55,42.62 -83.07,42.30 -83.15,42.14 -83.14,41.98 -83.03,41.83 -82.69,41.68 -82.44,41.67 -81.28,42.21 -80.25,42.37 -79.04,42.80 -78.92,42.91 -79.03,43.02 -79.06,43.28 -79.17,43.47 -78.85,43.58 -76.82,43.63 -76.46,44.06 -76.15,44.30 -75.00,44.97 -71.52,45.01 -71.33,45.29 -71.13,45.26 -71.00,45.34 -70.87,45.27 -70.80,45.40 -70.69,45.43 -70.70,45.55 -70.42,45.74 -70.30,45.91 -70.25,46.25 -70.07,46.44 -70.01,46.71 -69.24,47.46 -69.05,47.43 -69.05,47.27 -68.94,47.21 -68.24,47.35 -67.81,47.08 -67.80,45.73 -67.43,45.60 -67.49,45.47 -67.43,45.38 -67.47,45.28 -67.37,45.17 -67.12,45.17
34.96,-11.58 34.89,-11.39 34.77,-11.34 34.61,-11.08 34.66,-10.71 34.58,-10.53 34.52,-10.03 34.32,-9.73 34.09,-9.54 34.00,-9.50 33.95,-9.66 33.89,-9.67
107.97,21.51 107.76,21.66 107.35,21.61 107.27,21.71 107.06,21.79 106.97,21.92 106.66,21.98 106.65,22.24 106.55,22.34 106.55,22.50 106.78,22.78 106.54,22.91 106.28,22.86 106.15,22.97 105.84,22.92 105.55,23.07 105.28,23.35 104.86,23.14 104.80,22.91 104.69,22.82 104.37,22.70 104.14,22.80 104.05,22.75 103.94,22.54 103.62,22.78 103.49,22.59 103.33,22.77 102.98,22.45 102.83,22.59 102.47,22.75 102.13,22.38
7.44,43.75 7.38,43.73
12.43,41.91 12.43,41.91
113.55,22.22 113.48,22.20
114.02,22.51 114.12,22.57 114.27,22.54
70.61,39.79 70.48,39.88 70.70,39.85 70.61,39.79
71.22,39.91 71.01,39.90 71.04,39.99 70.97,40.04 70.98,40.13 71.23,40.05 71.18,39.98 71.22,39.91
145.94,43.10 145.78,43.48 145.69,43.55 145.54,43.51 145.24,43.78 145.27,43.91 145.62,44.29 145.66,44.45 145.58,44.79
-1.28,32.09 -1.23,32.11 -1.24,32.34 -1.07,32.47 -1.45,32.78 -1.68,33.32 -1.63,33.57 -1.71,33.78 -1.71,34.18 -1.79,34.37 -1.73,34.47 -1.85,34.61 -1.80,34.75 -2.13,34.97 -2.22,35.10
29.00,9.61 29.47,9.77 29.60,9.92 29.61,10.07 30.00,10.28 30.76,9.73 31.22,9.80 31.79,10.38 31.93,10.66 32.42,11.09 32.34,11.31 32.34,11.72 32.07,12.01 32.74,12.01 32.72,12.22 33.20,12.22 33.07,11.59 33.17,10.85 33.13,10.76 33.38,10.65 33.91,10.18 33.96,9.86 33.87,9.51 34.08,9.46
24.15,8.67 24.21,8.77 24.53,8.89 24.67,9.39 24.78,9.53 24.79,9.81 24.96,9.99 25.02,10.24 25.10,10.31 25.86,10.41 25.92,10.17 26.55,9.53 26.66,9.48 27.07,9.61 27.88,9.60
27.88,9.60 28.05,9.33 28.84,9.33 28.84,9.46 29.00,9.61
-56.48,1.94 -56.70,2.04 -57.20,2.85 -57.30,3.38 -57.60,3.37
-54.62,2.33 -54.40,2.46 -54.20,2.82 -54.20,3.14 -54.08,3.33
33.89,-9.67 33.70,-9.60 33.42,-9.61 33.33,-9.52 32.92,-9.41
33.59,46.10 33.66,46.22 34.22,46.10 34.45,45.97 34.69,45.98 34.79,45.89 34.80,45.79 35.00,45.73
77.05,35.11 76.77,35.66
77.05,35.11 77.80,35.50
79.12,33.23 79.38,33.15 79.33,32.98 79.50,32.73 79.38,32.57 79.22,32.50
79.22,32.50 78.92,32.36 78.70,32.60 78.41,32.56 78.49,32.23
78.49,32.23 78.73,31.98 78.72,31.88
78.72,31.88 78.69,31.74 78.80,31.62 78.73,31.47 78.74,31.32 78.84,31.30
78.84,31.30 78.97,31.33 79.01,31.41 79.11,31.40 79.37,31.08
79.37,31.08 79.57,30.95 79.79,30.97 79.91,30.90
79.91,30.90 80.19,30.76 80.19,30.57
80.19,30.57 80.68,30.41 80.99,30.24 81.01,30.16
90.33,28.12 90.35,28.24 90.22,28.28 89.98,28.31 89.80,28.24
89.80,28.24 89.48,28.06 89.12,27.62
89.12,27.62 89.02,27.51
89.02,27.51 88.89,27.32
75.30,32.32 75.32,32.22 75.25,32.14 74.74,31.95 74.56,31.82 74.51,31.71 74.59,31.47 74.52,31.19 74.63,31.03 74.34,30.89 73.90,30.44 73.93,30.22 73.81,30.09 73.38,29.93 73.23,29.55 72.90,29.03 72.34,28.75 72.13,28.35 71.95,28.18 71.87,27.96 70.80,27.71 70.69,27.77 70.63,27.94 70.40,28.03 70.14,27.85 69.90,27.47 69.54,27.12 69.47,26.80 69.51,26.74 70.15,26.51 70.10,25.91 70.26,25.71 70.65,25.67 70.65,25.42 71.05,24.69 70.97,24.57 71.04,24.40 70.72,24.24 70.58,24.28 70.55,24.42 70.49,24.41 70.10,24.29 70.02,24.19 69.81,24.17 69.56,24.27 68.83,24.26 68.78,24.31 68.73,24.27 68.72,23.96 68.38,23.95 68.17,23.86
36.69,45.62 36.69,45.43 36.59,45.30 36.64,45.26 36.51,45.20 36.61,44.95
-55.81,-31.03 -56.00,-31.08 -56.00,-30.85
-56.00,-30.85 -56.18,-30.63 -56.83,-30.11 -57.12,-30.14 -57.21,-30.28 -57.55,-30.26 -57.61,-30.19
-73.48,-49.77 -73.58,-49.58 -73.46,-49.31 -73.14,-49.30
-73.14,-49.30 -73.15,-49.19 -73.03,-49.01 -72.65,-48.84 -72.58,-48.48 -72.35,-48.37 -72.29,-48.23 -72.33,-48.11 -72.51,-47.97 -72.52,-47.88 -72.35,-47.49 -72.04,-47.24 -71.90,-47.20 -71.96,-47.02 -71.94,-46.83 -71.70,-46.65 -71.76,-46.32 -71.88,-46.16 -71.63,-45.95 -71.75,-45.84 -71.75,-45.58 -71.51,-45.51 -71.35,-45.33 -71.35,-45.23 -71.60,-44.98 -72.04,-44.90 -72.06,-44.77 -71.26,-44.76 -71.16,-44.56 -71.21,-44.44 -71.82,-44.38 -71.81,-44.11 -71.68,-43.93 -71.79,-43.75 -71.74,-43.70 -71.75,-43.59 -71.90,-43.44 -71.90,-43.35 -71.75,-43.24 -71.78,-43.17 -72.10,-43.07 -72.14,-42.58 -72.05,-42.47 -72.11,-42.25 -72.03,-42.15 -71.86,-42.15 -71.75,-42.05 -71.91,-41.65 -71.87,-40.89 -71.94,-40.79 -71.80,-40.44 -71.70,-40.34 -71.82,-40.18 -71.66,-40.02 -71.64,-39.89 -71.72,-39.64 -71.54,-39.60 -71.42,-39.29 -71.40,-38.94 -70.95,-38.74 -70.85,-38.54 -70.97,-38.45 -71.03,-38.04 -71.17,-37.76 -71.13,-37.45 -71.20,-37.30 -71.12,-37.11 -71.19,-36.84 -71.06,-36.52 -70.75,-36.39 -70.72,-36.28 -70.40,-36.06 -70.42,-35.52 -70.56,-35.25 -70.39,-35.15 -70.29,-34.73 -70.05,-34.30 -69.85,-34.22 -69.89,-33.73 -69.80,-33.40 -69.82,-33.28 -70.02,-33.27 -70.08,-33.20 -70.09,-33.03 -70.02,-32.88 -70.12,-32.81 -70.17,-32.47 -70.32,-32.27 -70.36,-32.08 -70.25,-31.96 -70.45,-31.84 -70.59,-31.57 -70.52,-31.15 -70.39,-31.12 -70.31,-31.02 -70.35,-30.90 -70.15,-30.36 -69.96,-30.36 -69.84,-30.18 -69.96,-30.08 -69.93,-29.77 -70.03,-29.32 -69.83,-29.10 -69.66,-28.41 -69.17,-27.92 -68.85,-27.15 -68.71,-27.10 -68.59,-27.14 -68.32,-26.97 -68.32,-26.88 -68.59,-26.47 -68.58,-26.35 -68.41,-26.15 -68.59,-25.42 -68.50,-25.16 -68.38,-25.09 -68.56,-24.84 -68.56,-24.75 -68.25,-24.39 -67.36,-24.03 -67.01,-23.00 -67.19,-22.82
75.32,34.58 74.30,34.77 73.96,34.65 73.81,34.42 73.81,34.33 73.97,34.24 73.92,34.04 74.25,33.99 74.22,33.89 74.00,33.79 73.98,33.72 74.15,33.51 73.99,33.22 74.30,32.99 74.31,32.81
23.61,51.52 23.54,51.71 23.63,51.81 23.65,52.04 23.18,52.29 23.41,52.52 23.84,52.66 23.92,52.77 23.86,53.11 23.48,53.94
34.08,9.46 34.16,9.85 34.31,10.19 34.28,10.53 34.34,10.66 34.57,10.88 34.77,10.75 34.93,10.86 34.96,11.28 35.11,11.82 35.67,12.62 36.13,12.76 36.16,13.09 36.39,13.63 36.52,14.26
2.39,11.90 2.29,11.69 1.98,11.42 1.43,11.45 1.32,11.30 1.15,11.25 1.08,11.08 0.99,11.08 0.90,10.99
31.95,-25.96 31.87,-25.98 31.42,-25.75 31.21,-25.84 30.80,-26.41 30.79,-26.76 30.88,-26.79 31.06,-27.11 31.47,-27.30 31.96,-27.31 31.99,-26.82 32.11,-26.84
32.11,-26.84 32.06,-26.02 31.95,-25.96
76.77,35.66 76.56,35.77 76.55,35.89 76.15,35.83 76.07,35.98 75.91,36.05 75.97,36.38
27.33,5.19 27.79,4.64 28.19,4.35 28.43,4.32 28.73,4.50 29.22,4.39 29.47,4.61 29.68,4.59 30.19,3.98 30.51,3.84 30.59,3.62 30.76,3.62 30.84,3.49
22.86,10.92 23.31,10.39 23.65,9.82 23.62,9.34 23.47,9.11 23.55,8.94 23.54,8.82 23.68,8.73 24.15,8.67
46.11,38.88 46.49,38.91
44.78,39.68 45.14,39.25 45.48,39.01 46.11,38.88
46.49,38.91 47.48,39.50 47.77,39.65 48.00,39.68 48.32,39.40 48.10,39.24 48.29,39.02 48.05,38.93 48.00,38.85 48.59,38.41 48.87,38.44
81.01,30.16 80.85,30.14 80.40,29.73 80.25,29.42 80.23,29.19 80.08,28.99 80.07,28.83 80.42,28.61 80.59,28.65 81.85,27.87 81.99,27.91 82.45,27.67 82.68,27.67 82.73,27.52 83.29,27.37 83.45,27.47 83.83,27.38 84.09,27.49 84.61,27.30 84.69,27.04 85.13,26.86 85.24,26.75 85.57,26.84 85.70,26.78 85.79,26.60 86.01,26.65 86.70,26.44 87.02,26.56 87.09,26.43 87.29,26.36 87.41,26.42 87.85,26.44 88.00,26.38 88.16,26.72 87.98,27.13 88.15,27.75 88.11,27.87
75.97,36.38 75.84,36.65 75.67,36.74 75.42,36.74 75.35,36.91 74.54,37.02
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,5 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16">
<!-- Generate-thumbnails action: a picture frame with a sparkle. A single
symbolic fill lets GTK recolour it with the surrounding control. -->
<path fill="#222222" d="M 2,3 C 1.45,3 1,3.45 1,4 V 14 C 1,14.55 1.45,15 2,15 H 14 C 14.55,15 15,14.55 15,14 V 7 H 13 V 13 H 3 V 5 H 9 V 3 Z M 5,6 C 4.45,6 4,6.45 4,7 C 4,7.55 4.45,8 5,8 C 5.55,8 6,7.55 6,7 C 6,6.45 5.55,6 5,6 Z M 10,7 L 7.5,10 L 6,9 L 4,11.5 V 12 H 12 V 9 Z M 13,0 L 12.4,2.4 L 10,3 L 12.4,3.6 L 13,6 L 13.6,3.6 L 16,3 L 13.6,2.4 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 598 B

@@ -0,0 +1,6 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16">
<!-- A copy is on this computer: a check in an outlined circle. Adwaita 50
no longer ships emblem-ok-symbolic. -->
<path fill="#222222" fill-rule="evenodd" d="M 8,1 A 7,7 0 1 0 8,15 A 7,7 0 1 0 8,1 Z M 8,2.5 A 5.5,5.5 0 1 1 8,13.5 A 5.5,5.5 0 1 1 8,2.5 Z"/>
<path fill="#222222" d="M 4.1,8.6 L 5.2,7.5 L 7,9.3 L 10.8,5.4 L 11.9,6.5 L 7,11.5 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 450 B

@@ -0,0 +1,6 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16">
<!-- Show or hide the details pane: a window with its right-hand pane
filled in. Some themes draw sidebar-show-right-symbolic as a solid block. -->
<path fill="#222222" fill-rule="evenodd" d="M 3,2 H 13 A 2,2 0 0 1 15,4 V 12 A 2,2 0 0 1 13,14 H 3 A 2,2 0 0 1 1,12 V 4 A 2,2 0 0 1 3,2 Z M 3,3.5 A 0.5,0.5 0 0 0 2.5,4 V 12 A 0.5,0.5 0 0 0 3,12.5 H 13 A 0.5,0.5 0 0 0 13.5,12 V 4 A 0.5,0.5 0 0 0 13,3.5 Z"/>
<path fill="#222222" d="M 9.5,5 H 12.5 V 11 H 9.5 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 564 B

@@ -0,0 +1,5 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16">
<!-- "Filters": a funnel, wide at the top and narrowing to a short spout.
No stock icon theme ships one under a standard name. -->
<path fill="#222222" d="M 1.5,2 H 14.5 L 9.5,8.5 V 13.5 L 6.5,15 V 8.5 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 309 B

@@ -0,0 +1,8 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16">
<!-- A share link: two open chain ends and the bar that joins them, laid
on the diagonal. Written out in plain coordinates: GTK's symbolic icon
renderer drops a transformed group. -->
<path fill="#222222" d="M 4.82,6.23 L 3.05,8 A 3.5,3.5 0 0 0 8,12.95 L 9.77,11.18 L 8.53,9.94 L 6.76,11.71 A 1.75,1.75 0 0 1 4.29,9.24 L 6.06,7.47 Z"/>
<path fill="#222222" d="M 6.23,4.82 L 8,3.05 A 3.5,3.5 0 0 1 12.95,8 L 11.18,9.77 L 9.94,8.53 L 11.71,6.76 A 1.75,1.75 0 0 0 9.24,4.29 L 7.47,6.06 Z"/>
<path fill="#222222" d="M 5.26,9.5 L 9.5,5.26 L 10.74,6.5 L 6.5,10.74 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 676 B

Loaded 100 of 212 files, more files were not shown because too many files have changed in this diff. Show more