Compare commits

...
19 Commits
Author SHA1 Message Date
nvrl 847ece17d7 version bumped to 1.9.1 2026-08-21 22:48:28 +02:00
nvrl 50d2acdaba Merge pull request #20 from emptyname-org/feature/files-thumbnails-statusbar
feat(gui): add file thumbnails and a browser status bar
2026-08-21 22:44:58 +02:00
emptyname 53caf63ad4 Fix thumbnail lifecycle failures 2026-08-21 12:32:31 +01:00
nvrl 5176dd91e1 fixed icon 2026-08-21 04:13:11 +02:00
nvrl e88903d5ff chore(branding) added branding, best practice to repo 2026-08-21 04:09:58 +02:00
emptyname b3e6af1ae5 docs: document thumbnail runtime support 2026-08-18 13:06:40 +01:00
emptyname b0f9513915 feat(gui): show file thumbnails and browser status bar 2026-08-18 13:06:36 +01:00
emptyname 91b0864fce feat(thumbnails): generate cached previews for Drive files 2026-08-18 13:06:27 +01:00
nvrl 8bd9481902 fix(ui) improved ux 2026-08-18 12:50:49 +02:00
nvrl b2d5870834 fix(ui) improved ux 2026-08-18 12:06:28 +02:00
nvrl 078774f6a1 release(1.8.2) 2026-08-17 00:49:24 +02:00
nvrl 326a9b4e92 chore(docs) updated changelog + bug states 2026-08-17 00:04:54 +02:00
nvrl 78dc72a34d release(1.8.1) 2026-08-16 23:33:43 +02:00
nvrl 438b2b463e chore(docs) updated changelog 2026-08-16 22:03:55 +02:00
nvrl 4cd49168c4 fix(drain) failed attempts 2026-08-16 22:02:27 +02:00
nvrl 481c3fe0c7 version bump + bug docs 2026-08-16 21:34:55 +02:00
nvrl 098da2e97a fix(log) log unreachable nodes 2026-08-16 21:34:12 +02:00
nvrl d03d3eac3f fix(drive) locks + 1.7.1 2026-08-16 20:41:03 +02:00
nvrl 433e206203 chore(changelog) 2026-08-16 00:54:55 +02:00
70 changed files with 11921 additions and 858 deletions

No files matched your search

+76
View File
@@ -0,0 +1,76 @@
name: Bug report
description: Something in the client misbehaves
labels: ["bug"]
body:
- type: markdown
attributes:
value: |
Please check [`docs/BUGS.md`](../blob/main/docs/BUGS.md) first — the issue may already be
tracked there with context.
**Do not report a security vulnerability here.** See [SECURITY.md](../blob/main/SECURITY.md).
- type: textarea
id: what
attributes:
label: What happened
description: What you did, what you expected, and what happened instead.
validations:
required: true
- type: textarea
id: repro
attributes:
label: Steps to reproduce
placeholder: |
1. …
2. …
validations:
required: true
- type: input
id: version
attributes:
label: Version
description: Output of `pdfs --version`, and how you installed it (deb / rpm / AUR / source).
validations:
required: true
- type: input
id: environment
attributes:
label: Distribution and desktop
placeholder: "Fedora 44, GNOME 48 (Wayland)"
validations:
required: true
- type: dropdown
id: surface
attributes:
label: Where does it happen
multiple: true
options:
- FUSE mount / file operations
- Sync or backed-up folders (Computers)
- CLI (pdfs)
- Desktop app (pdfs-app)
- Tray (pdfs-tray)
- Search launcher (pdfs-prompt)
- Login / keyring
- Photos
- Packaging / installation
validations:
required: true
- type: textarea
id: diagnose
attributes:
label: "`pdfs diagnose` output"
description: Redact paths or the account address if you prefer.
render: console
- type: textarea
id: logs
attributes:
label: Daemon logs
description: "`journalctl --user -u proton-drive.service -n 200 --no-pager`"
render: console
- type: checkboxes
id: dataloss
attributes:
label: Data
options:
- label: This bug involved missing, truncated, or overwritten file content.
+11
View File
@@ -0,0 +1,11 @@
blank_issues_enabled: false
contact_links:
- name: Security vulnerability
url: https://github.com/narrrl/proton-drive-linux/security/advisories/new
about: Report privately — never in a public issue.
- name: Known issues
url: https://github.com/narrrl/proton-drive-linux/blob/main/docs/BUGS.md
about: The authoritative ledger of tracked bugs and their status.
- name: Proton Drive service problems
url: https://proton.me/support
about: This is an unofficial client. Account or service issues go to Proton AG.
@@ -0,0 +1,37 @@
name: Feature request
description: Suggest a capability or an improvement
labels: ["enhancement"]
body:
- type: textarea
id: problem
attributes:
label: The problem
description: What are you trying to do that the client makes hard or impossible today?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: What you'd like to see
validations:
required: true
- type: dropdown
id: surface
attributes:
label: Which part of the client
multiple: true
options:
- FUSE mount
- Sync / Computers
- CLI (pdfs)
- Desktop app (pdfs-app)
- Tray (pdfs-tray)
- Search launcher (pdfs-prompt)
- Photos
- Packaging
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives you considered
+26
View File
@@ -0,0 +1,26 @@
## What this changes
<!-- The user-visible effect, in a sentence or two. -->
**Crates touched:**
**Related `docs/BUGS.md` entries:**
## Verification
```bash
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
```
<!-- Plus whatever else you ran: acceptance suite, a live mount, manual steps. -->
## Implications
- [ ] Adds a SQLite migration (new `MIGRATION_V*`, `SCHEMA_VERSION` bumped — no shipped migration edited)
- [ ] Touches recovery, staging, or anything that can be the only copy of user data
- [ ] Changes packaging, the systemd unit, or desktop entries
- [ ] Changes the control protocol (daemon and front ends updated together)
- [ ] None of the above
<!-- Screenshots for GTK changes. -->
+7 -6
View File
@@ -122,13 +122,14 @@ jobs:
Priority: optional
Architecture: amd64
Maintainer: Nils Pukropp <contact@narl.io>
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4, libimage-exiftool-perl
Installed-Size: ${INSTALLED_SIZE}
Description: Proton Drive client for Linux
Proton Drive client for Linux featuring:
- FUSE files-on-demand mount
- Command-line interface (CLI)
- GTK4 system tray, desktop application, and search launcher
Homepage: https://github.com/narrrl/proton-drive-linux
Description: Unofficial Proton Drive client for Linux
Files-on-demand Proton Drive for the Linux desktop, unaffiliated with Proton AG:
- FUSE virtual mount with block-level caching
- Scriptable command-line interface (pdfs)
- GTK4 desktop application, system tray, and search launcher
EOF
# Build Debian package
+32
View File
@@ -0,0 +1,32 @@
# Code of Conduct
## Our pledge
We want participation in this project to be a harassment-free experience for everyone, regardless
of age, body size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socioeconomic status, nationality, personal appearance, race,
religion, or sexual identity and orientation.
## Standards
Examples of behavior that helps:
- being respectful of differing viewpoints and experience levels,
- giving and accepting constructive feedback gracefully,
- focusing on what is best for the project and its users.
Examples of unacceptable behavior:
- sexualized language or imagery, and unwelcome sexual attention,
- trolling, insulting or derogatory comments, and personal or political attacks,
- public or private harassment,
- publishing others' private information without explicit permission.
## Scope and enforcement
This applies in all project spaces — issues, pull requests, discussions — and when representing the
project in public. Report unacceptable behavior to <contact@narl.io>. Reports are handled
confidentially. Maintainers may edit, remove, or reject contributions that violate this document,
and may temporarily or permanently ban any contributor for behavior they deem inappropriate.
Adapted from the [Contributor Covenant](https://www.contributor-covenant.org), version 2.1.
+70
View File
@@ -0,0 +1,70 @@
# Contributing
Thanks for taking the time. This is an unofficial Proton Drive client that people trust with real
files, so the bar for changes that touch the filesystem, the sync engine, or the database is
deliberately high — everything else is ordinary Rust review.
## Before you start
- Read [`AGENTS.md`](AGENTS.md) for repository conventions and
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for how the daemon, front ends, and SDK fit
together.
- Check [`docs/BUGS.md`](docs/BUGS.md) — it is the authoritative issue ledger, and your problem may
already be tracked there with context.
- For anything larger than a fix, open an issue first so the design can be discussed before you
spend the time.
## Development setup
```bash
git clone https://github.com/narrrl/proton-drive-linux.git
cd proton-drive-linux
cargo build --workspace
```
GUI and FUSE targets need the GTK4, libadwaita, WebKitGTK 6, libsecret, D-Bus, and FUSE3
development packages listed in the [README](README.md#prerequisites). Minimum supported Rust
version is 1.96.
## Quality gates
CI runs exactly these, and they must pass with zero warnings:
```bash
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
scripts/fuse-acceptance.sh --offline-only
```
Changes to the FUSE layer, sync, or the pending-operation queue should also be exercised against a
real mount before review — see [`docs/TESTING.md`](docs/TESTING.md). Live runs are destructive:
use a dedicated test account, never one holding data you cannot lose.
## Making changes
- **Keep shared logic in `pdfs-core`.** The CLI and GUI never touch the database or the network;
they speak the control protocol to the daemon.
- **The SQLite schema is forward-only.** Never edit a shipped migration — add a new one and bump
`SCHEMA_VERSION` in `crates/pdfs-core/src/db/migrations.rs`.
- **Typed errors and `?` over panics** on runtime paths. Blocking filesystem, keyring, or
control-socket work stays off GTK's main thread.
- **`staging/` and `recovery/` may hold the only copy of user data.** Nothing may clear or purge
them opportunistically.
## Commits and pull requests
Commits follow scoped [Conventional Commits](https://www.conventionalcommits.org), e.g.
`fix(fuse,core): keep truncate composable with queued revisions`.
A pull request should state:
- the user-visible effect and which crates it touches,
- the `docs/BUGS.md` entries it fixes or affects,
- the commands you ran to verify it,
- screenshots for GTK changes,
- and, explicitly, any migration, recovery, data-loss, or packaging implications.
## Security
Do not open a public issue for a vulnerability — see [SECURITY.md](SECURITY.md).
Generated
+67 -8
View File
@@ -2200,6 +2200,25 @@ dependencies = [
"system-deps",
]
[[package]]
name = "h2"
version = "0.4.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155"
dependencies = [
"atomic-waker",
"bytes",
"fnv",
"futures-core",
"futures-sink",
"http",
"indexmap",
"slab",
"tokio",
"tokio-util",
"tracing",
]
[[package]]
name = "half"
version = "2.7.1"
@@ -2367,6 +2386,7 @@ dependencies = [
"bytes",
"futures-channel",
"futures-core",
"h2",
"http",
"http-body",
"httparse",
@@ -3518,7 +3538,7 @@ checksum = "35fb2e5f958ec131621fdd531e9fc186ed768cbe395337403ae56c17a74c68ec"
[[package]]
name = "pdfs-cli"
version = "1.7.0"
version = "1.9.1"
dependencies = [
"anyhow",
"clap 4.6.4",
@@ -3533,7 +3553,7 @@ dependencies = [
[[package]]
name = "pdfs-core"
version = "1.7.0"
version = "1.9.1"
dependencies = [
"async-trait",
"directories",
@@ -3546,16 +3566,19 @@ dependencies = [
"rusqlite",
"serde",
"serde_json",
"sha1 0.10.7",
"sha2 0.10.9",
"thiserror 2.0.19",
"tokio",
"tracing",
"zip",
]
[[package]]
name = "pdfs-fuse"
version = "1.7.0"
version = "1.9.1"
dependencies = [
"base64",
"fuser",
"futures",
"image",
@@ -3565,13 +3588,15 @@ dependencies = [
"pdfs-core",
"proton-drive-rs",
"serde_json",
"sha1 0.10.7",
"tokio",
"tracing",
"zip",
]
[[package]]
name = "pdfs-gui"
version = "1.7.0"
version = "1.9.1"
dependencies = [
"anyhow",
"async-channel",
@@ -3832,9 +3857,9 @@ dependencies = [
[[package]]
name = "proton-drive-rs"
version = "0.5.1"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3abae5e90e6889b120208159eb2c60b332612e5b708fdf3c14287307ce01715d"
checksum = "58322795538d66137d2eb071590375c5287e69059c429e2309dc37368ab5b588"
dependencies = [
"async-trait",
"base64",
@@ -3859,9 +3884,9 @@ dependencies = [
[[package]]
name = "proton-sdk"
version = "0.5.1"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3fa729310c1a66e12bb41b9f5f036c86b9857e76536556052203ffecdb0b44c3"
checksum = "d77482a3160d14926c65537c64bc95cf27d609597a524c97d97d97d1777e9603"
dependencies = [
"aes-gcm 0.11.0",
"async-trait",
@@ -4252,6 +4277,7 @@ dependencies = [
"bytes",
"futures-core",
"futures-util",
"h2",
"http",
"http-body",
"http-body-util",
@@ -5110,6 +5136,7 @@ dependencies = [
"bytes",
"futures-core",
"futures-sink",
"libc",
"pin-project-lite",
"tokio",
]
@@ -5291,6 +5318,12 @@ dependencies = [
"cipher 0.4.4",
]
[[package]]
name = "typed-path"
version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e28f89b80c87b8fb0cf04ab448d5dd0dd0ade2f8891bae878de66a75a28600e"
[[package]]
name = "typenum"
version = "1.20.1"
@@ -6232,6 +6265,20 @@ dependencies = [
"syn 2.0.119",
]
[[package]]
name = "zip"
version = "8.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d04a6b5381502aa6087c94c669499eb1602eb9c5e8198e534de571f7154809b"
dependencies = [
"crc32fast",
"flate2",
"indexmap",
"memchr",
"typed-path",
"zopfli",
]
[[package]]
name = "zlib-rs"
version = "0.6.6"
@@ -6244,6 +6291,18 @@ version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
[[package]]
name = "zopfli"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f05cd8797d63865425ff89b5c4a48804f35ba0ce8d125800027ad6017d2b5249"
dependencies = [
"bumpalo",
"crc32fast",
"log",
"simd-adler32",
]
[[package]]
name = "zune-core"
version = "0.5.1"
+15 -3
View File
@@ -3,15 +3,20 @@ resolver = "2"
members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"]
[workspace.package]
version = "1.7.0"
version = "1.9.1"
edition = "2024"
license = "MIT"
authors = ["Nils Pukropp <contact@narl.io>"]
rust-version = "1.96"
repository = "https://github.com/narrrl/proton-drive-linux"
homepage = "https://github.com/narrrl/proton-drive-linux"
readme = "README.md"
keywords = ["proton", "drive", "fuse", "sync", "linux"]
categories = ["filesystem", "command-line-utilities"]
[workspace.dependencies]
proton-sdk = { version = "0.5.1" }
proton-drive-rs = { version = "0.5.1" }
proton-sdk = { version = "0.6.1" }
proton-drive-rs = { version = "0.6.1" }
# Internal crates.
pdfs-core = { path = "crates/pdfs-core" }
@@ -20,6 +25,7 @@ pdfs-fuse = { path = "crates/pdfs-fuse" }
tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "fs", "io-util", "time", "signal"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
base64 = "0.22"
thiserror = "2"
anyhow = "1"
# Non-poisoning mutexes. A handler that panics while holding a lock must not
@@ -32,6 +38,12 @@ tracing-subscriber = { version = "0.3", features = ["env-filter"] }
directories = "5"
ignore = "0.4"
sha2 = "0.10"
# Google Takeout exports are zip sets; the importer reads their central
# directories and streams entries out without extracting to disk first.
zip = { version = "8", default-features = false, features = ["deflate"] }
# Proton's photo duplicate detection is keyed on the plaintext SHA-1, so an
# import has to speak SHA-1 whatever its age.
sha1 = "0.10"
# Thumbnail generation for photos the server has none for. Decoders only for the
# formats a camera or phone actually produces — the rest of the crate's codecs
# would be dead weight in the daemon.
+76 -7
View File
@@ -1,12 +1,70 @@
# Proton Drive client for Linux (unofficial)
<div align="center">
A fast, unofficial Proton Drive client for Linux. This client features an advanced files-on-demand FUSE virtual mount with block-level caching, a command-line interface (CLI), and a fully non-blocking GTK4 desktop application with system tray integration.
<img src="images/logo.svg" alt="" width="112" height="112">
# Proton Drive for Linux
**Files-on-demand Proton Drive for the Linux desktop** — a FUSE virtual mount with block-level
caching, a scriptable CLI, and a native GTK4 app with tray and search launcher.
[![CI](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml/badge.svg)](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml)
[![Release](https://img.shields.io/github/v/release/narrrl/proton-drive-linux?sort=semver)](https://github.com/narrrl/proton-drive-linux/releases/latest)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust 1.96+](https://img.shields.io/badge/rust-1.96%2B-orange.svg)](https://www.rust-lang.org)
[![Platform: Linux](https://img.shields.io/badge/platform-linux-lightgrey.svg)](#prerequisites)
<img src="images/files.png" alt="The GTK4 file browser" width="820">
</div>
> [!IMPORTANT]
> This is an **unofficial**, community-built client. It is not affiliated with, endorsed by, or
> supported by Proton AG. "Proton" and "Proton Drive" are trademarks of Proton AG, used here only
> to describe what the software talks to. Use a test account first, and read
> [`docs/RECOVERY.md`](docs/RECOVERY.md) before trusting it with data you cannot lose.
## Install
```bash
# Debian / Ubuntu
sudo apt install ./proton-drive-linux_*.deb
# Fedora
sudo dnf install ./proton-drive-linux-*.rpm
# Arch Linux
cd packaging && makepkg -fi
```
Packages for each release are attached to the [latest
release](https://github.com/narrrl/proton-drive-linux/releases/latest); see
[Installation & Packages](#installation--packages) for details and
[Building from Source](#building-from-source) to compile it yourself.
```bash
pdfs login # sign in (SRP + 2FA, credentials go to the system keyring)
pdfs status # mount state, queued writes, cache usage
pdfs-app # the desktop app
```
## Contents
- [Features](#features) · [Quick search](#quick-search-and-media-opening) · [Opening files](#choosing-how-files-open)
- [Selective sync](#selective-sync-pdfsignore) · [Diagnostics](#diagnostics--maintenance) · [Scripting](#scripting---json)
- [Performance](#performance--caching) · [Filesystem safety](#filesystem-safety) · [Screenshots](#screenshots)
- [Prerequisites](#prerequisites) · [Building](#building-from-source) · [Packages](#installation--packages) · [Releases](#automated-releases-cicd)
**Documentation:** [Architecture](docs/ARCHITECTURE.md) · [Testing](docs/TESTING.md) ·
[Recovery](docs/RECOVERY.md) · [Development](docs/DEVELOPMENT.md) ·
[Known issues](docs/BUGS.md) · [Changelog](docs/CHANGELOG.md) ·
[Contributing](CONTRIBUTING.md) · [Security policy](SECURITY.md)
## Features
- **Files-on-Demand (FUSE)**: Mount your Proton Drive as a virtual filesystem where files are downloaded only when opened, utilizing smart block-level caching and disk-backed writes.
- **Command-Line Interface (CLI)**: Manage your drive, authenticate, and monitor sync status directly from the terminal.
- **Non-Blocking GTK4 Desktop App**: Browse files, manage pins, configure options, and monitor active transfers through a modern, native GUI with a fully non-blocking asynchronous main loop.
- **File Thumbnails**: Browse locally cached image previews in Files, Shared, and Trash, including embedded previews from camera RAW files. Build a whole folder tree from the Files toolbar and resize the grid from the status bar.
- **Shared Links & Invites**: Browse files shared with you by other users, and view/manage your own public shared links directly in the GUI.
- **Local Backup (Computers)**: Sync and back up local directories (like Downloads, Documents, Pictures, etc.) directly to your Proton Drive account.
- **Locations**: One page (and `pdfs locations`) listing every place Proton Drive occupies on this machine — the main mount plus each backed-up folder — with its mode, sync state, and whether it is read-only. Switching a folder between a full local copy and on-demand happens here.
@@ -270,19 +328,21 @@ sudo apt-get install -y \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
libdbus-1-dev \
libimage-exiftool-perl
```
### Arch Linux
```bash
sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus webkitgtk-6.0
sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus webkitgtk-6.0 perl-image-exiftool
```
### Fedora (44+)
```bash
sudo dnf install -y \
pkgconf-pkg-config fuse3-devel gtk4-devel libadwaita-devel \
webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel cargo rust
webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel \
perl-Image-ExifTool cargo rust
```
Runtime extras (pick your desktop):
@@ -302,7 +362,7 @@ Ensure you have Rust and Cargo installed (minimum supported Rust version is 1.96
1. Clone the repository and navigate into the project directory:
```bash
git clone https://github.com/narl/proton-drive-linux.git
git clone https://github.com/narrrl/proton-drive-linux.git
cd proton-drive-linux
```
2. Build the workspace in release mode:
@@ -372,6 +432,15 @@ This project has a GitHub Actions CI workflow configured under `.github/workflow
---
## Contributing
Bug reports, packaging fixes, and patches are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for
the workflow and the quality gates CI enforces, and [SECURITY.md](SECURITY.md) for reporting a
vulnerability privately.
## License
This project is licensed under the [MIT License](LICENSE).
Released under the [MIT License](LICENSE).
Proton Drive is a service of Proton AG. This project is an independent client and carries no
affiliation with or endorsement from Proton AG.
+34
View File
@@ -0,0 +1,34 @@
# Security Policy
## Supported versions
Fixes land on the latest release. Please reproduce on the most recent version before reporting.
## Reporting a vulnerability
**Do not open a public issue.** Report privately through GitHub's
[private vulnerability reporting](https://github.com/narrrl/proton-drive-linux/security/advisories/new),
or by email to <contact@narl.io>.
Please include the version (`pdfs --version`), your distribution and desktop environment, what an
attacker gains, and a reproduction if you have one. You can expect an acknowledgement within a few
days; a fix ships in the next release, and you are credited in the advisory unless you ask
otherwise.
## Scope
This client holds Proton account credentials and decrypted file content on the local machine, so
the interesting boundaries are:
- **Credentials.** Sessions and the mailbox password live in the system Secret Service. Anything
that writes them elsewhere — logs, temp files, crash dumps — is a vulnerability.
- **The control socket.** Anything that can connect to `control.sock` has the daemon's
authenticated session. It is `0600`, and the state, cache, and config directories must be real,
current-user-owned `0700` directories; the daemon fails closed otherwise. Reports that defeat
those checks are in scope.
- **Local cache and staging.** Decrypted block content and staged writes on disk.
- **Cryptography.** Key derivation, signature handling, and anything that could publish plaintext
or the wrong key material to Proton.
Out of scope: vulnerabilities in Proton's own service (report those to Proton AG), and issues that
require an attacker who already has your user account on the machine.
+2
View File
@@ -5,6 +5,8 @@ edition.workspace = true
license.workspace = true
authors.workspace = true
rust-version.workspace = true
repository.workspace = true
homepage.workspace = true
description = "CLI for the Proton Drive Linux client"
[[bin]]
+127
View File
@@ -158,6 +158,28 @@ enum Command {
},
/// List the photo albums (including ones shared with this account).
Albums,
/// Import a Google Photos Takeout export into Proton Photos.
///
/// Pass every zip of the export at once — Google splits one export across
/// numbered parts, and a photo's metadata often sits in a different part
/// than the photo. Albums in the export are recreated as Proton albums, and
/// photos already on the account (same name and same content) are skipped,
/// so an interrupted import can simply be run again.
ImportGooglePhotos {
/// The export's `.zip` files (e.g. `takeout-*.zip`).
#[arg(required = true)]
archives: Vec<PathBuf>,
/// Report what would be imported without uploading anything.
#[arg(long)]
dry_run: bool,
/// Stay attached and print progress until the import finishes.
#[arg(long)]
wait: bool,
},
/// Show how the running (or last) Google Photos import is doing.
ImportStatus,
/// Stop the running Google Photos import.
CancelImport,
/// List the photos in one album (newest capture first).
Album {
/// Album node uid in `volume~link` form (from `pdfs albums`).
@@ -589,6 +611,13 @@ fn main() -> Result<()> {
} => cmd_photos(limit, offset, favorites),
Command::Favorite { uid, remove } => cmd_favorite(uid, remove),
Command::Albums => cmd_albums(),
Command::ImportGooglePhotos {
archives,
dry_run,
wait,
} => cmd_import_google_photos(archives, dry_run, wait),
Command::ImportStatus => cmd_import_status(),
Command::CancelImport => cmd_cancel_import(),
Command::Album { uid, limit, offset } => cmd_album(uid, limit, offset),
Command::OpenPhoto { uid } => cmd_open_photo(uid),
Command::Search { query, limit } => cmd_search(query, limit),
@@ -1711,6 +1740,104 @@ fn cmd_albums() -> Result<()> {
Ok(())
}
fn cmd_import_google_photos(archives: Vec<PathBuf>, dry_run: bool, wait: bool) -> Result<()> {
// The daemon opens the archives itself off the shared filesystem, so it gets
// absolute paths — the same arrangement `pdfs upload` uses.
let mut absolute = Vec::with_capacity(archives.len());
for archive in &archives {
let path = std::fs::canonicalize(archive)
.with_context(|| format!("resolve {}", archive.display()))?;
absolute.push(
path.to_str()
.ok_or_else(|| anyhow!("path is not valid UTF-8: {}", path.display()))?
.to_owned(),
);
}
match control_request(CtlRequest::ImportTakeout {
archives: absolute,
dry_run,
})? {
CtlResponse::Ok { message } => println!("{message}"),
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
}
if !wait {
println!("Run `pdfs import-status` to follow it.");
return Ok(());
}
// Poll rather than stream: the control protocol is request/response, and an
// import runs for hours — a held-open socket would be the only long-lived
// connection the daemon has.
loop {
std::thread::sleep(std::time::Duration::from_secs(2));
match control_request(CtlRequest::ImportStatus)? {
CtlResponse::ImportStatus {
running: true,
summary: _,
} => continue,
CtlResponse::ImportStatus { summary, .. } => {
print_import_summary(summary.as_ref());
return Ok(());
}
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
}
}
}
fn cmd_import_status() -> Result<()> {
match control_request(CtlRequest::ImportStatus)? {
CtlResponse::ImportStatus { running, summary } => {
if running {
println!("Import running. `pdfs transfers` shows its progress.");
}
print_import_summary(summary.as_ref());
}
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
}
Ok(())
}
fn cmd_cancel_import() -> Result<()> {
match control_request(CtlRequest::CancelImport)? {
CtlResponse::Ok { message } => println!("{message}"),
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
other => bail!("unexpected response: {other:?}"),
}
Ok(())
}
fn print_import_summary(summary: Option<&pdfs_core::control::ImportSummary>) {
let Some(summary) = summary else {
println!("No import has run yet.");
return;
};
println!("Photos in export : {}", summary.found);
println!("Uploaded : {}", summary.uploaded);
println!("Already on Proton: {}", summary.duplicates);
if summary.skipped_trashed > 0 {
println!("Skipped (trash) : {}", summary.skipped_trashed);
}
if summary.albums_created > 0 || summary.album_links > 0 {
println!(
"Albums : {} created, {} photos filed",
summary.albums_created, summary.album_links
);
}
if summary.failed > 0 {
println!("Failed : {}", summary.failed);
}
if summary.bytes > 0 {
println!("Uploaded bytes : {}", summary.bytes);
}
if summary.cancelled {
println!("Import was cancelled before it finished.");
}
}
fn cmd_album(uid: String, limit: usize, offset: usize) -> Result<()> {
match control_request(CtlRequest::AlbumPhotos { uid, offset, limit })? {
CtlResponse::Photos { items, .. } if items.is_empty() => println!("No photos."),
+4
View File
@@ -5,6 +5,8 @@ edition.workspace = true
license.workspace = true
authors.workspace = true
rust-version.workspace = true
repository.workspace = true
homepage.workspace = true
description = "Core auth/session/config for the Proton Drive Linux client"
[dependencies]
@@ -18,6 +20,8 @@ tracing.workspace = true
directories.workspace = true
ignore.workspace = true
sha2.workspace = true
sha1.workspace = true
zip.workspace = true
rusqlite.workspace = true
keyring.workspace = true
parking_lot.workspace = true
File diff suppressed because it is too large. Load diff
+254 -1
View File
@@ -93,6 +93,32 @@ pub enum Request {
/// already cached (one batched round-trip) and replying with their on-disk
/// paths. Keep the batch small — it is served on demand, as tiles scroll in.
PhotoThumbs { uids: Vec<String> },
/// Fetch thumbnails for ordinary Drive image files shown outside the Photos
/// timeline. The modification time is the cache validity tag, so replacing
/// an image can never reuse its previous revision's thumbnail.
FileThumbs {
items: Vec<FileThumbRequest>,
/// Identifies the front-end's current listing. Moving to another page or
/// folder advances it, allowing the daemon to stop work for rows that
/// are no longer visible.
#[serde(default)]
generation: u64,
},
/// Reserve a daemon-issued generation for ordinary-file thumbnail work.
/// The daemon is the ordering authority, so GUI restarts and wall-clock
/// corrections cannot make a new listing look older than an earlier one.
ReserveFileThumbGeneration,
/// Stop ordinary-file thumbnail work belonging to older listings. Photos
/// timeline work and an explicit recursive build are deliberately separate.
CancelFileThumbs { generation: u64 },
/// Start building local thumbnails for every supported image below `path`.
/// The request returns immediately; progress is read with
/// [`Request::ThumbnailBuildStatus`].
StartThumbnailBuild { path: String },
/// Stop the current recursive ordinary-file thumbnail build, if any.
CancelThumbnailBuild,
/// Read progress for the recursive ordinary-file thumbnail build.
ThumbnailBuildStatus,
/// Download a photo's full content into the cache; replies with its path.
OpenPhoto { uid: String },
/// Add or remove Proton's `Favorite` tag on a photo. Replies with
@@ -103,6 +129,33 @@ pub enum Request {
/// re-encrypted for our timeline, which the SDK does not implement yet; the
/// daemon reports that as an error rather than silently doing nothing.
SetPhotoFavorite { uid: String, favorite: bool },
/// Import Google Photos **Takeout** archives into Proton Photos.
///
/// `archives` are absolute paths to the export's `.zip` parts on the
/// daemon's own filesystem — the whole set at once, because a photo and its
/// metadata sidecar routinely land in different parts. Photos already on the
/// account (matched by name *and* content digest) are skipped, so a library
/// partly synced by Proton Photos elsewhere does not import twice, and an
/// interrupted import resumes by simply being run again.
///
/// With `dry_run` the daemon scans and reports what it *would* do without
/// uploading anything. Otherwise it acks immediately with
/// [`Response::Ok`] and works in the background — an export is hours of
/// upload, far past any socket timeout. Progress shows up in
/// [`Request::GetQueueStatus`] as a job, and the final counts in
/// [`Request::ImportStatus`].
ImportTakeout {
archives: Vec<String>,
#[serde(default)]
dry_run: bool,
},
/// How the running (or last finished) Takeout import is doing. Replies with
/// [`Response::ImportStatus`].
ImportStatus,
/// Ask the running Takeout import to stop. It finishes the photo on the wire
/// and files what it has uploaded into its albums, then reports
/// `cancelled`. Replies with [`Response::Ok`].
CancelImport,
/// Upload the photo at `source_path` under the given name and media type.
///
/// A path, not the bytes: this protocol is line-delimited JSON, and
@@ -652,6 +705,10 @@ pub struct SharedItem {
/// The shared node's decrypted name.
pub name: String,
pub is_dir: bool,
/// Modification time used to validate an image thumbnail. Older daemons
/// omit it, in which case clients use `0` as a conservative fallback tag.
#[serde(default)]
pub modified: i64,
/// Mountpoint-relative path, when the daemon can resolve it (the node lives in
/// my own tree). Empty when the path is unknown.
#[serde(default)]
@@ -1052,6 +1109,35 @@ pub struct PhotoMonth {
pub count: usize,
}
/// How a Google Photos Takeout import is going, or how the last one ended
/// ([`Response::ImportStatus`]).
///
/// The counts are cumulative for the run, so the same shape serves progress and
/// the final report. `found` counts *distinct* photos: Google stores a separate
/// copy of a photo in each album folder it appears in, and those fold into one
/// upload that joins several albums.
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
pub struct ImportSummary {
/// Distinct photos the export holds.
pub found: usize,
/// Photos uploaded by this run.
pub uploaded: usize,
/// Photos already on the account, matched by name *and* content digest.
pub duplicates: usize,
/// Photos that could not be read or uploaded.
pub failed: usize,
/// Photos in the export's trash bucket, which are never imported.
pub skipped_trashed: usize,
/// Albums created by this run.
pub albums_created: usize,
/// Album memberships added by this run.
pub album_links: usize,
/// Bytes uploaded.
pub bytes: u64,
/// True when the run stopped early because it was cancelled.
pub cancelled: bool,
}
/// One album in a [`Response::Albums`] listing.
///
/// An album is a folder node on the photos volume carrying album properties, so
@@ -1131,6 +1217,76 @@ pub struct PhotoThumb {
pub pending: bool,
}
/// One ordinary Drive file whose thumbnail a front-end wants to paint.
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct FileThumbRequest {
/// Node uid in `volume~link` form.
pub uid: String,
/// File modification time, used as the thumbnail cache validity tag.
pub modified: i64,
/// Original Drive name. Local RAW-preview extraction uses the extension as a
/// format hint; defaulted for wire compatibility with older front-ends.
#[serde(default)]
pub name: String,
}
/// Whether a Drive name is a raster image that the built-in decoder can read,
/// or a concrete camera RAW format whose embedded preview exiftool can extract.
/// Keep this list shared by the daemon and every GUI surface so a visible tile
/// is never advertised without a matching generation path.
pub fn is_thumbnail_image_name(name: &str) -> bool {
let Some((_, extension)) = name.rsplit_once('.') else {
return false;
};
let extension = extension.to_ascii_lowercase();
STANDARD_THUMBNAIL_EXTENSIONS.contains(&extension.as_str())
|| RAW_THUMBNAIL_EXTENSIONS.contains(&extension.as_str())
}
/// Whether `name` is one of the concrete RAW formats handled by the exiftool
/// embedded-preview path. Deliberately excludes generic/legacy grab-bags whose
/// preview availability is not dependable.
pub fn is_raw_image_name(name: &str) -> bool {
let Some((_, extension)) = name.rsplit_once('.') else {
return false;
};
let extension = extension.to_ascii_lowercase();
RAW_THUMBNAIL_EXTENSIONS.contains(&extension.as_str())
}
const STANDARD_THUMBNAIL_EXTENSIONS: &[&str] =
&["bmp", "gif", "jpeg", "jpg", "png", "tif", "tiff", "webp"];
/// Concrete camera RAW formats with embedded previews handled by exiftool.
/// This is the single source used by both RAW classification and the broader
/// thumbnail capability check.
const RAW_THUMBNAIL_EXTENSIONS: &[&str] = &[
"arw", "cr2", "cr3", "crw", "dng", "k25", "kdc", "mrw", "nef", "nrw", "orf", "pef", "raf",
"raw", "rw2", "sr2", "srf", "x3f",
];
/// Progress of the one recursive ordinary-file thumbnail build the daemon may
/// run at a time.
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
pub struct ThumbnailBuildStatus {
pub running: bool,
/// True while folders are still being discovered, before the final image
/// count is known and the progress bar can become determinate.
pub scanning: bool,
/// Mountpoint-relative root selected when the build started.
pub path: String,
pub folders_scanned: u64,
pub images_found: u64,
/// Images already cached or processed during this build. Includes failures,
/// so `completed == images_found` still means the job has finished.
pub completed: u64,
/// Images that could not be downloaded or decoded.
pub failed: u64,
/// A traversal-level problem, such as a folder that could not be listed.
#[serde(default)]
pub message: Option<String>,
}
/// A daemon too old to report connectivity is assumed online — it could not
/// have mounted at all otherwise.
fn default_online() -> bool {
@@ -1270,11 +1426,28 @@ pub enum Response {
available: bool,
items: Vec<AlbumInfo>,
},
/// How the Takeout import is doing (reply to [`Request::ImportStatus`]).
/// `running` is false once it has finished, when `summary` is the final
/// report; `summary` is `None` when no import has run this session.
ImportStatus {
running: bool,
#[serde(default)]
summary: Option<ImportSummary>,
},
/// The months the timeline spans (reply to [`Request::PhotoMonths`]),
/// newest first.
PhotoMonths { months: Vec<PhotoMonth> },
/// Thumbnails for a [`Request::PhotoThumbs`] batch.
Thumbs { items: Vec<PhotoThumb> },
/// A [`Request::FileThumbs`] generation is no longer current. This is
/// retryable after reserving a fresh generation and must never be interpreted
/// as a permanent "no thumbnail" verdict.
FileThumbsStale,
/// Daemon-issued generation reserved by
/// [`Request::ReserveFileThumbGeneration`].
FileThumbGeneration { generation: u64 },
/// Current recursive thumbnail-build progress.
ThumbnailBuild { status: ThumbnailBuildStatus },
/// An on-disk path the front-end can open (e.g. a downloaded photo).
FilePath { path: String },
/// Full-text search results (reply to [`Request::Search`]).
@@ -1409,7 +1582,7 @@ impl ErrorKind {
}
/// Send one [`Request`] to the daemon listening on `socket` and read its
/// [`Response`]. Errors (e.g. [`Error::Io`]) if no daemon is listening.
/// [`Response`]. Returns a crate [`crate::Error`] if no daemon is listening.
///
/// Shared by the CLI and GUI so both speak the wire format identically.
pub fn send(socket: &Path, req: &Request) -> Result<Response> {
@@ -1568,6 +1741,86 @@ mod tests {
}
}
#[test]
fn file_thumbnail_requests_keep_their_revision_tags() {
let request = Request::FileThumbs {
items: vec![
FileThumbRequest {
uid: "volume~first".into(),
modified: 1_700_000_001,
name: "first.jpg".into(),
},
FileThumbRequest {
uid: "volume~second".into(),
modified: 1_700_000_002,
name: "second.nef".into(),
},
],
generation: 42,
};
let line = serde_json::to_string(&request).unwrap();
let decoded: Request = serde_json::from_str(&line).unwrap();
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
}
#[test]
fn legacy_file_thumbnail_requests_default_the_name() {
let decoded: Request = serde_json::from_str(
r#"{"FileThumbs":{"items":[{"uid":"volume~first","modified":1700000001}],"generation":42}}"#,
)
.unwrap();
let Request::FileThumbs { items, generation } = decoded else {
panic!("decoded the wrong request variant");
};
assert_eq!(generation, 42);
assert_eq!(items.len(), 1);
assert!(items[0].name.is_empty());
}
#[test]
fn thumbnail_build_requests_and_progress_roundtrip() {
let requests = [
Request::ReserveFileThumbGeneration,
Request::CancelFileThumbs { generation: 43 },
Request::StartThumbnailBuild {
path: "pictures/events".into(),
},
Request::CancelThumbnailBuild,
Request::ThumbnailBuildStatus,
];
for request in requests {
let line = serde_json::to_string(&request).unwrap();
let decoded: Request = serde_json::from_str(&line).unwrap();
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
}
let response = Response::ThumbnailBuild {
status: ThumbnailBuildStatus {
running: true,
scanning: false,
path: "pictures/events".into(),
folders_scanned: 12,
images_found: 80,
completed: 25,
failed: 1,
message: None,
},
};
let line = serde_json::to_string(&response).unwrap();
let decoded: Response = serde_json::from_str(&line).unwrap();
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
let response = Response::FileThumbGeneration { generation: 44 };
let line = serde_json::to_string(&response).unwrap();
let decoded: Response = serde_json::from_str(&line).unwrap();
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
let response = Response::FileThumbsStale;
let line = serde_json::to_string(&response).unwrap();
let decoded: Response = serde_json::from_str(&line).unwrap();
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
}
#[test]
fn unified_search_request_and_response_roundtrip() {
let request = Request::SearchV2 {
+122 -1
View File
@@ -9,7 +9,7 @@ use super::Db;
use crate::Result;
/// Current schema version. Bump on every forward migration added below.
pub(super) const SCHEMA_VERSION: i64 = 25;
pub(super) const SCHEMA_VERSION: i64 = 28;
impl Db {
pub(super) fn migrate(&self) -> Result<()> {
@@ -161,6 +161,63 @@ impl Db {
tx.execute_batch(MIGRATION_V25)?;
}
}
if current < 26 {
// Same guard as V22: a fixture that never exercised the queue may
// not have `pending_op`, and altering an absent table is an error
// rather than a no-op.
// A fixture built from the current schema and then rewound to an
// older `schema_version` already has the column, and `ADD COLUMN`
// has no `IF NOT EXISTS`.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('pending_op') WHERE name = 'claimed_at'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_ops: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_ops && !has_column {
tx.execute_batch(MIGRATION_V26)?;
}
}
if current < 27 {
// Same guards as V26, for the same two reasons: the table may not
// exist in an old fixture, and a fixture rewound from the current
// schema already has the column.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('pending_op') \
WHERE name = 'access_deferred_since'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_ops: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_ops && !has_column {
tx.execute_batch(MIGRATION_V27)?;
}
}
if current < 28 {
// Same guards as V26/V27, for the same two reasons.
let has_column: bool = tx.query_row(
"SELECT COUNT(*) FROM pragma_table_info('sync_entry') \
WHERE name = 'local_mtime_ns'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
let has_entries: bool = tx.query_row(
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'sync_entry'",
[],
|row| row.get::<_, i64>(0),
)? > 0;
if has_entries && !has_column {
tx.execute_batch(MIGRATION_V28)?;
}
}
tx.execute(
"INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1)
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
@@ -725,3 +782,67 @@ UPDATE nodes
DROP TABLE node_paths;
";
/// V26: `pending_op.claimed_at` — the drain's claim column.
///
/// The drain was one thread taking one op at a time, so a 10 GiB upload held
/// every queued rename and small write behind it for as long as it ran. Several
/// workers can share the queue only if a claim is visible to the others: this
/// column is what makes "already being drained by someone" a fact about the row
/// rather than about one thread's local variable.
///
/// `0` is unclaimed; anything else is the ms at which a worker took it. A
/// claim is process-local state that outlives nothing — the single-writer lock
/// means a non-zero value found at open belongs to a crashed run, and
/// [`Db::clear_op_claims`](super::Db::clear_op_claims) drops the lot.
///
/// The index is partial so it costs nothing on the ordinary row: only the
/// handful of claimed rows are in it, which is exactly the set the claim query
/// has to subtract to keep two workers off one node.
const MIGRATION_V26: &str = "
ALTER TABLE pending_op ADD COLUMN claimed_at INTEGER NOT NULL DEFAULT 0;
CREATE INDEX IF NOT EXISTS idx_pending_op_claimed
ON pending_op(uid) WHERE claimed_at <> 0;
";
/// Schema v27: when an op was first deferred by the local access check.
///
/// An access deferral consumes no attempt and records no error — the op has not
/// failed remotely, it has not been sent at all. That is right for a permission
/// change the user is about to undo, and wrong for a condition that never
/// clears: the row is re-deferred every
/// [`DRAIN_ACCESS_RECHECK`](../../../pdfs_fuse/drain/constant.DRAIN_ACCESS_RECHECK.html)
/// forever while `attempts`, `last_error`, `failing` and the logs all stay
/// empty, so a queue that can never drain is indistinguishable from a busy one.
/// Found in the field as 28 revision ops holding 18 GiB of accepted writes for
/// 30 days, none of them visible anywhere.
///
/// `0` means "not currently access-deferred"; anything else is the ms at which
/// the first consecutive deferral happened, which is what
/// [`Db::defer_op_for_access`](super::Db::defer_op_for_access) measures the
/// deferral against.
const MIGRATION_V27: &str = "
ALTER TABLE pending_op ADD COLUMN access_deferred_since INTEGER NOT NULL DEFAULT 0;
";
/// Schema v28: sub-second precision for a mirror folder's local baseline.
///
/// A reconcile pass decides whether the local side of a path changed by
/// comparing `(local_mtime, local_size)` against the baseline. Both sides of
/// that comparison were whole seconds, so an edit that landed in the same second
/// as the last sync and left the file the same length — a fixed-size record
/// rewritten, an image re-exported, a database page updated in place — read as
/// *unchanged* and was never uploaded. The remote copy silently stayed at the
/// older content until something else about the file moved (bugs.md B25).
///
/// Nullable rather than a converted `local_mtime * 1_000_000_000`: the
/// nanosecond part of an existing baseline is genuinely unknown, and inventing
/// zero for it would make every already-synced file compare as changed and
/// re-upload an entire mirror on first start. `NULL` means "this row predates
/// the column", which the comparison reads as "fall back to whole seconds" —
/// exactly the behaviour that row was written under. Each row gains a real value
/// the next time its path is synced.
const MIGRATION_V28: &str = "
ALTER TABLE sync_entry ADD COLUMN local_mtime_ns INTEGER;
";
+12 -2
View File
@@ -230,7 +230,7 @@ impl Db {
}
Err(e) => return Err(e),
};
Ok(Self {
let db = Self {
conn: Mutex::new(conn),
// No pool for `:memory:`. A second connection to that name is a
// second, empty database rather than another view of this one, so
@@ -240,7 +240,17 @@ impl Db {
idle: Mutex::new(Vec::new()),
}),
_single_writer: single_writer,
})
};
// A drain claim names a worker thread in the process that took it, and
// the single-writer lock above says that process is not running. Any
// claim still on disk is therefore a crashed run's, and leaving it would
// hide those ops from every worker for good.
match db.clear_op_claims() {
Ok(0) => {}
Ok(n) => tracing::info!(ops = n, "released drain claims left by a previous run"),
Err(e) => tracing::warn!(error = %e, "clearing stale drain claims failed"),
}
Ok(db)
}
/// A connection for a `SELECT`-only method.
+159 -2
View File
@@ -512,17 +512,115 @@ impl Db {
Ok(op)
}
/// Take the oldest due op for this worker, marking it claimed so no other
/// drain worker takes it, or `None` when there is nothing to do.
///
/// The drain is several workers over one queue, which is only safe because
/// the claim is a column: a row is offered to exactly one worker, and the
/// worker that took it is the only one that may retire, defer or fail it.
///
/// Two exclusions on top of [`next_due_op`](Self::next_due_op)'s:
///
/// * `claimed_at = 0` — not already in flight somewhere else.
/// * no *other* claimed op shares this `uid`. Ordering only has to hold per
/// node, but there it has to hold absolutely: a queued rename and a queued
/// revision of one file must land in the order they were made, and two
/// workers on one node would also race over its staged blob and its
/// pending entry. `ORDER BY id` plus this exclusion gives per-uid serial
/// order for free — the older op is claimed first and the newer one is
/// invisible until it retires.
///
/// Select and mark are one transaction, so two workers cannot both see the
/// row as unclaimed.
pub fn claim_next_due_op(&self, now: i64) -> Result<Option<PendingOp>> {
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
let op = {
let mut stmt = tx.prepare_cached(&format!(
"SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
attempts, last_error, next_attempt_at
FROM pending_op
WHERE next_attempt_at <= ?1
AND claimed_at = 0
AND (parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~')
AND uid NOT IN (SELECT uid FROM pending_op WHERE claimed_at <> 0)
ORDER BY id LIMIT 1",
v = LOCAL_VOLUME,
n = LOCAL_VOLUME.len() + 1,
))?;
stmt.query_row(params![now], |r| {
Ok(PendingOp {
id: r.get(0)?,
kind: r.get(1)?,
uid: r.get(2)?,
parent_uid: r.get(3)?,
name: r.get(4)?,
blob_path: r.get(5)?,
meta_json: r.get(6)?,
created_at: r.get(7)?,
attempts: r.get(8)?,
last_error: r.get(9)?,
next_attempt_at: r.get(10)?,
})
})
.optional()?
};
if let Some(op) = op.as_ref() {
tx.execute(
"UPDATE pending_op SET claimed_at = ?2 WHERE id = ?1",
params![op.id, now.max(1)],
)?;
}
tx.commit()?;
Ok(op)
}
/// Give a claimed op back to the queue, whatever became of the attempt.
///
/// Called on every path out of a drain attempt rather than only the failing
/// ones: a handler that retires its own row leaves nothing to release (the
/// `UPDATE` matches nothing), and a handler that returns without retiring
/// *and* without failing would otherwise leave the row claimed by a worker
/// that has moved on — invisible to every worker, forever.
pub fn release_op_claim(&self, id: i64) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"UPDATE pending_op SET claimed_at = 0 WHERE id = ?1",
params![id],
)?;
Ok(())
}
/// Drop every claim in the table.
///
/// A claim describes a worker in *this* process, so one surviving on disk
/// means the previous run died holding it. The single-writer lock
/// ([`Db::open`]) is what makes that inference safe: nobody else can be
/// draining this queue while we are opening it.
pub fn clear_op_claims(&self) -> Result<usize> {
let conn = self.conn.lock();
Ok(conn.execute(
"UPDATE pending_op SET claimed_at = 0 WHERE claimed_at <> 0",
[],
)?)
}
/// The earliest `next_attempt_at` among all queued ops, or `None` if the
/// queue is empty. Used by the drain loop to sleep exactly until the next
/// debounced or backed-off op becomes eligible rather than waiting the full
/// idle-poll interval.
///
/// Claimed ops are excluded: another worker is already on them, so they are
/// not work this one is waiting for, and counting them would have an idle
/// worker spin on a row it cannot have.
pub fn earliest_due_at(&self) -> Result<Option<i64>> {
let conn = self.read();
let ts: Option<i64> = conn
.query_row(
&format!(
"SELECT MIN(next_attempt_at) FROM pending_op \
WHERE parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~'",
WHERE claimed_at = 0 \
AND (parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~')",
v = LOCAL_VOLUME,
n = LOCAL_VOLUME.len() + 1,
),
@@ -623,11 +721,16 @@ impl Db {
/// Record a failed attempt and when to next try. Leaves the row in place —
/// the staged bytes are still the only copy of the user's write.
///
/// Clears any access-deferral window: whatever the op was waiting for has
/// now been reported, so the next uncleared deferral starts its own window
/// rather than escalating again on the following recheck.
pub fn record_op_failure(&self, id: i64, error: &str, next_attempt_at: i64) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"UPDATE pending_op
SET attempts = attempts + 1, last_error = ?2, next_attempt_at = ?3
SET attempts = attempts + 1, last_error = ?2, next_attempt_at = ?3,
access_deferred_since = 0
WHERE id = ?1",
params![id, error, next_attempt_at],
)?;
@@ -653,6 +756,60 @@ impl Db {
Ok(())
}
/// Defer an op the local access check refused, and report how long it has
/// been refused for.
///
/// Identical to [`defer_op_without_attempt`](Self::defer_op_without_attempt)
/// on the retry clock, plus the one thing that was missing: the first
/// deferral of a run stamps `access_deferred_since`, and every deferral
/// returns that stamp. A caller can therefore tell a deferral that is one
/// recheck old from one that has been repeating unchanged for days, which is
/// the difference between a permission the user is mid-way through changing
/// and a queue that will never drain.
///
/// Returns the ms at which the current run of deferrals began — `now` on the
/// first one.
pub fn defer_op_for_access(&self, id: i64, now: i64, next_attempt_at: i64) -> Result<i64> {
let conn = self.conn.lock();
conn.execute(
"UPDATE pending_op
SET next_attempt_at = CASE
WHEN next_attempt_at >= ?4 THEN next_attempt_at
ELSE ?3
END,
access_deferred_since = CASE
WHEN access_deferred_since = 0 THEN ?2
ELSE access_deferred_since
END
WHERE id = ?1",
params![id, now, next_attempt_at, PARK_UNTIL],
)?;
let since = conn
.query_row(
"SELECT access_deferred_since FROM pending_op WHERE id = ?1",
params![id],
|r| r.get::<_, i64>(0),
)
.optional()?;
// A row that drained out from under this call leaves nothing to measure;
// reporting `now` says "this run just started", which is the reading that
// never escalates something that no longer exists.
Ok(since.filter(|since| *since != 0).unwrap_or(now))
}
/// Forget an op's access-deferral window, after an attempt the access check
/// let through. The op is no longer blocked, so a later deferral is the
/// start of a new run rather than the continuation of an old one.
pub fn clear_op_access_deferral(&self, id: i64) -> Result<()> {
let conn = self.conn.lock();
conn.execute(
"UPDATE pending_op SET access_deferred_since = 0
WHERE id = ?1 AND access_deferred_since <> 0",
params![id],
)?;
Ok(())
}
/// Every staged blob the queue still refers to.
///
/// The startup staging reconcile subtracts this from what is actually in
+21 -8
View File
@@ -35,6 +35,14 @@ pub struct StoredSyncEntry {
pub remote_uid: Option<String>,
pub local_mtime: i64,
pub local_size: i64,
/// The same modification time to nanosecond precision, when it is known.
///
/// `local_mtime` alone cannot tell an edit that landed inside the same
/// second as the last sync — and left the file the same length — from no
/// edit at all, so those never uploaded (bugs.md B25). `None` on a row
/// written before schema 28, which the comparison reads as "compare whole
/// seconds", so upgrading does not make an entire mirror look changed.
pub local_mtime_ns: Option<i64>,
pub remote_rev: Option<String>,
pub remote_hash: Option<String>,
}
@@ -168,7 +176,8 @@ impl Db {
pub fn sync_entries(&self, folder_id: i64) -> Result<HashMap<String, StoredSyncEntry>> {
let conn = self.read();
let mut stmt = conn.prepare(
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash,
local_mtime_ns
FROM sync_entry WHERE folder_id = ?1",
)?;
let rows = stmt.query_map(params![folder_id], |r| {
@@ -179,6 +188,7 @@ impl Db {
local_size: r.get(3)?,
remote_rev: r.get(4)?,
remote_hash: r.get(5)?,
local_mtime_ns: r.get(6)?,
})
})?;
let mut map = HashMap::new();
@@ -216,14 +226,16 @@ impl Db {
let conn = self.conn.lock();
conn.execute(
"INSERT INTO sync_entry
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev,
remote_hash, local_mtime_ns)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)
ON CONFLICT(folder_id, rel_path) DO UPDATE SET
remote_uid = excluded.remote_uid,
local_mtime = excluded.local_mtime,
local_size = excluded.local_size,
remote_rev = excluded.remote_rev,
remote_hash = excluded.remote_hash",
remote_uid = excluded.remote_uid,
local_mtime = excluded.local_mtime,
local_size = excluded.local_size,
remote_rev = excluded.remote_rev,
remote_hash = excluded.remote_hash,
local_mtime_ns = excluded.local_mtime_ns",
params![
folder_id,
e.rel_path,
@@ -232,6 +244,7 @@ impl Db {
e.local_size,
e.remote_rev,
e.remote_hash,
e.local_mtime_ns,
],
)?;
Ok(())
+356
View File
@@ -1895,6 +1895,123 @@ fn failed_superseding_insert_keeps_the_old_pending_op() {
assert_eq!(ops[0].blob_path.as_deref(), Some("/staging/original"));
}
/// B53's remaining half. The insert-failure regression above proves the *row*
/// rolls back; this proves the two things that make the rollback usable — the
/// old blob is not reported as superseded (so the caller does not delete the
/// bytes the surviving row still points at), and the rollback survives a
/// restart rather than living only in the connection that failed.
#[test]
fn a_rolled_back_supersede_keeps_its_blob_and_survives_reopen() {
let dir = std::env::temp_dir().join(format!("pdfs-db-b53-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join("queue.sqlite");
let op = |blob: &str| PendingOp {
id: 0,
kind: OP_REVISION.to_string(),
uid: uid("durable").to_string(),
parent_uid: None,
name: None,
blob_path: Some(blob.to_string()),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
};
{
let db = Db::open(&path).unwrap();
db.enqueue_op(&op("/staging/original")).unwrap();
db.with_conn(|conn| {
conn.execute_batch(
"CREATE TRIGGER reject_replacement BEFORE INSERT ON pending_op
WHEN NEW.blob_path = '/staging/replacement'
BEGIN SELECT RAISE(ABORT, 'injected insert failure'); END;",
)?;
Ok(())
})
.unwrap();
// The failure must not hand back the old blob path. That value is the
// caller's instruction to delete those bytes, and the row that owns
// them is still queued.
assert!(db.enqueue_op(&op("/staging/replacement")).is_err());
db.with_conn(|conn| {
conn.execute_batch("DROP TRIGGER reject_replacement")?;
Ok(())
})
.unwrap();
}
let db = Db::open(&path).unwrap();
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1, "old or new — never neither");
assert_eq!(ops[0].blob_path.as_deref(), Some("/staging/original"));
// And once the injected failure is gone, superseding works and *does*
// report the blob it retired.
let (_, superseded) = db.enqueue_op(&op("/staging/replacement")).unwrap();
assert_eq!(superseded.as_deref(), Some("/staging/original"));
assert_eq!(db.pending_ops().unwrap().len(), 1);
drop(db);
std::fs::remove_dir_all(&dir).unwrap();
}
/// A full disk, as SQLite reports it. `max_page_count` makes the database
/// refuse to grow, which is the same `SQLITE_FULL` an out-of-space filesystem
/// produces — and the queue must come out of it with the acknowledged work
/// still in it (B53).
#[test]
fn a_full_database_rolls_the_supersede_back_rather_than_erasing_it() {
let db = Db::open_in_memory().unwrap();
let op = |blob: &str| PendingOp {
id: 0,
kind: OP_REVISION.to_string(),
uid: uid("full").to_string(),
parent_uid: None,
name: None,
blob_path: Some(blob.to_string()),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
};
db.enqueue_op(&op("/staging/original")).unwrap();
// Cap the file at its current size, so any page the insert needs fails.
let pages: i64 = db
.with_conn(|conn| Ok(conn.query_row("PRAGMA page_count", [], |r| r.get(0))?))
.unwrap();
db.with_conn(|conn| {
conn.execute_batch(&format!("PRAGMA max_page_count = {pages}"))?;
Ok(())
})
.unwrap();
// A payload far larger than one page, so growth is unavoidable.
let mut fat = op("/staging/replacement");
fat.meta_json = Some("x".repeat(512 * 1024));
let result = db.enqueue_op(&fat);
db.with_conn(|conn| {
conn.execute_batch("PRAGMA max_page_count = 1073741823")?;
Ok(())
})
.unwrap();
assert!(result.is_err(), "a database that cannot grow must refuse");
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1);
assert_eq!(
ops[0].blob_path.as_deref(),
Some("/staging/original"),
"the acknowledged upload is still queued and still owns its bytes"
);
}
#[test]
fn atomic_trash_replacement_returns_owned_blobs_after_commit() {
let db = Db::open_in_memory().unwrap();
@@ -2408,6 +2525,7 @@ fn mirror_uid_lookup_covers_the_root_and_synced_descendants_only() {
local_size: 2,
remote_rev: Some("3".into()),
remote_hash: Some("2".into()),
local_mtime_ns: None,
},
)
.unwrap();
@@ -2424,6 +2542,7 @@ fn mirror_uid_lookup_covers_the_root_and_synced_descendants_only() {
local_size: 2,
remote_rev: Some("3".into()),
remote_hash: Some("2".into()),
local_mtime_ns: None,
},
)
.unwrap();
@@ -3491,6 +3610,132 @@ fn migration_v25_backfills_paths_for_an_existing_tree() {
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
}
/// A queue carried over from V26 keeps its rows, and every one of them starts
/// out not access-deferred — a migration must not invent a deferral window that
/// would report an ordinary queued upload as blocked.
#[test]
fn migration_v27_adds_the_access_deferral_column_to_a_v26_queue() {
let path = std::env::temp_dir().join(format!(
"pdfs-db-v26-fixture-{}-{}.db",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
{
let db = Db::open(&path).unwrap();
db.enqueue_op(&crate::db::PendingOp {
id: 0,
kind: crate::db::OP_REVISION.to_string(),
uid: "v~l".to_string(),
parent_uid: None,
name: None,
blob_path: Some("/tmp/blob".to_string()),
meta_json: Some("{}".to_string()),
created_at: 1,
attempts: 0,
last_error: None,
next_attempt_at: 0,
})
.unwrap();
}
{
// Put the file back in the state a released V26 database was in.
let conn = rusqlite::Connection::open(&path).unwrap();
conn.execute_batch(
"ALTER TABLE pending_op DROP COLUMN access_deferred_since;
UPDATE sync_state SET value = '26' WHERE key = 'schema_version';",
)
.unwrap();
}
let db = Db::open(&path).unwrap();
let ops = db.pending_ops().unwrap();
assert_eq!(ops.len(), 1, "the migration keeps the queued write");
let now = 5_000i64;
assert_eq!(
db.defer_op_for_access(ops[0].id, now, now + 5_000).unwrap(),
now,
"a carried-over op is not already deferred"
);
let _ = std::fs::remove_file(&path);
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
}
/// A mirror baseline carried over from V27 keeps its rows, and each one reads
/// back with no nanosecond time. That `None` is the whole point: it makes the
/// comparison fall back to whole seconds for rows written before the column
/// existed, instead of asserting a sub-second time of zero and re-uploading
/// every already-synced file in the folder (bugs.md B25).
#[test]
fn migration_v28_adds_sub_second_local_times_without_disturbing_a_v27_baseline() {
let path = std::env::temp_dir().join(format!(
"pdfs-db-v27-fixture-{}-{}.db",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
{
let db = Db::open(&path).unwrap();
let folder = db
.sync_folder_add("/home/me/Mirror", "vol~root", "share")
.unwrap();
db.sync_entry_upsert(
folder,
&StoredSyncEntry {
rel_path: "notes.txt".into(),
remote_uid: Some("vol~notes".into()),
local_mtime: 1_700_000_000,
local_mtime_ns: Some(1_700_000_000_123_456_789),
local_size: 42,
remote_rev: Some("1700000000".into()),
remote_hash: Some("42".into()),
},
)
.unwrap();
}
{
// Put the file back in the state a released V27 database was in.
let conn = rusqlite::Connection::open(&path).unwrap();
conn.execute_batch(
"ALTER TABLE sync_entry DROP COLUMN local_mtime_ns;
UPDATE sync_state SET value = '27' WHERE key = 'schema_version';",
)
.unwrap();
}
let db = Db::open(&path).unwrap();
let entries = db.sync_entries(1).unwrap();
let entry = entries.get("notes.txt").expect("the baseline row survives");
assert_eq!(entry.local_mtime, 1_700_000_000);
assert_eq!(entry.local_size, 42);
assert_eq!(
entry.local_mtime_ns, None,
"a carried-over row must not claim a sub-second time it never recorded"
);
// And the column is live: a refreshed row keeps the precision it is given.
db.sync_entry_upsert(
1,
&StoredSyncEntry {
local_mtime_ns: Some(1_700_000_001_000_000_500),
..entry.clone()
},
)
.unwrap();
assert_eq!(
db.sync_entries(1).unwrap()["notes.txt"].local_mtime_ns,
Some(1_700_000_001_000_000_500)
);
let _ = std::fs::remove_file(&path);
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
}
// --- read-only connection pool (DB2) ---------------------------------------
/// A read must not queue behind whatever happens to be writing.
@@ -3545,3 +3790,114 @@ fn a_read_does_not_wait_for_the_write_connection() {
let _ = std::fs::remove_file(&path);
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
}
/// §5: the drain is several workers over one queue, and the claim column is the
/// only thing keeping them off each other's work. Two properties matter — no
/// row is ever handed to two workers, and no *node* is ever worked by two
/// workers even though its ops are separate rows.
#[test]
fn a_claimed_op_is_invisible_to_every_other_worker() {
let db = Db::open_in_memory().unwrap();
let root = uid("root").to_string();
for i in 0..3 {
db.enqueue_op(&bulk_op(i, &root)).unwrap();
}
let first = db.claim_next_due_op(10).unwrap().expect("an op is due");
let second = db
.claim_next_due_op(10)
.unwrap()
.expect("another op is due");
let third = db
.claim_next_due_op(10)
.unwrap()
.expect("a third op is due");
assert!(
db.claim_next_due_op(10).unwrap().is_none(),
"a fourth claim must find nothing left: every queued op is taken"
);
let mut ids = [first.id, second.id, third.id];
ids.sort_unstable();
assert_eq!(
ids,
[1, 2, 3],
"each worker must get a distinct row, oldest first"
);
// Releasing puts one back, and it is the one that comes out next.
db.release_op_claim(second.id).unwrap();
let again = db.claim_next_due_op(10).unwrap().expect("the released op");
assert_eq!(again.id, second.id);
}
#[test]
fn two_workers_never_share_a_node() {
let db = Db::open_in_memory().unwrap();
let root = uid("root").to_string();
// Two ops of different kinds against one node — a queued rename and a
// queued revision, which is an ordinary `mv` plus a save.
let mut revision = bulk_op(0, &root);
revision.parent_uid = None;
db.enqueue_op(&revision).unwrap();
let rename = PendingOp {
kind: OP_RENAME.to_string(),
parent_uid: Some(root.clone()),
blob_path: None,
meta_json: None,
..bulk_op(0, &root)
};
db.enqueue_op(&rename).unwrap();
let claimed = db.claim_next_due_op(10).unwrap().expect("an op is due");
assert_eq!(claimed.kind, OP_REVISION, "the older op goes first");
assert!(
db.claim_next_due_op(10).unwrap().is_none(),
"the second op targets the same node, so it must wait for the first: \
two workers on one uid would race over its staged blob and land its \
changes out of order"
);
db.release_op_claim(claimed.id).unwrap();
db.delete_op(claimed.id).unwrap();
assert_eq!(
db.claim_next_due_op(10).unwrap().map(|o| o.kind),
Some(OP_RENAME.to_string()),
"once the revision retires, the rename is claimable"
);
}
#[test]
fn claims_do_not_survive_the_process_that_took_them() {
let db = Db::open_in_memory().unwrap();
let root = uid("root").to_string();
for i in 0..2 {
db.enqueue_op(&bulk_op(i, &root)).unwrap();
}
db.claim_next_due_op(10).unwrap().unwrap();
db.claim_next_due_op(10).unwrap().unwrap();
assert!(db.claim_next_due_op(10).unwrap().is_none());
// What `Db::open` does after the single-writer lock says the previous run
// is gone. Without it those ops are invisible to every worker forever.
assert_eq!(db.clear_op_claims().unwrap(), 2);
assert!(
db.claim_next_due_op(10).unwrap().is_some(),
"a crashed run's claims must not park its queue"
);
}
#[test]
fn an_idle_worker_does_not_wait_on_an_op_another_worker_has() {
let db = Db::open_in_memory().unwrap();
let root = uid("root").to_string();
db.enqueue_op(&bulk_op(0, &root)).unwrap();
assert_eq!(db.earliest_due_at().unwrap(), Some(0));
db.claim_next_due_op(10).unwrap().unwrap();
assert_eq!(
db.earliest_due_at().unwrap(),
None,
"an op somebody else is draining is not work this worker is waiting for"
);
}
+1
View File
@@ -20,6 +20,7 @@ pub mod search;
pub mod service;
pub mod shell;
pub mod syncignore;
pub mod takeout;
pub use access::{Access, access_for, perm_bits};
pub use error::{CoreError, CoreResult, Error, Result};
+887
View File
@@ -0,0 +1,887 @@
//! Reading a Google Photos **Takeout** export: what is in the archives, which
//! album each photo belongs to, and when it was taken.
//!
//! This module is pure structure — it opens the archives' central directories
//! and reads the small JSON sidecars, never the media bytes. The import itself
//! (hashing, duplicate detection, upload, album creation) lives in the daemon;
//! everything here is offline-testable and has no Proton dependency.
//!
//! # What a Takeout export looks like
//!
//! Google splits an export into numbered zips (`takeout-20260817T...-001.zip`,
//! `-002.zip`, …) that are *independent* archives — no zip64 spanning — but a
//! photo and its JSON sidecar can land in different parts, and one album's
//! folder can be spread over several. So the scan treats the whole set as one
//! namespace keyed by the path inside the export:
//!
//! ```text
//! Takeout/Google Photos/Iceland 2019/IMG_0042.jpg
//! Takeout/Google Photos/Iceland 2019/IMG_0042.jpg.supplemental-metadata.json
//! Takeout/Google Photos/Iceland 2019/metadata.json <- album title
//! Takeout/Google Photos/Photos from 2019/IMG_0043.jpg <- no album
//! Takeout/Google Photos/Trash/IMG_0044.jpg <- skipped
//! ```
//!
//! The service folder (`Google Photos`) is localized, as are the year folders
//! (`Photos from 2019`) and the trash folder, so nothing here matches on the
//! service name: the first component is dropped when it is the export root, the
//! second is the service, and the rest is `<folder>/<file>`. A folder is an
//! **album** when it carries an album metadata sidecar naming it; anything else
//! is a plain bucket that imports into the timeline only. Trash is matched by
//! name against [`TRASH_FOLDERS`] — the one place a localized name is
//! unavoidable, because a trashed photo is not marked as such anywhere else.
use std::collections::{BTreeMap, HashMap};
use std::io::Read;
use std::path::{Path, PathBuf};
use serde::{Deserialize, Serialize};
use crate::error::{CoreError, CoreResult};
/// Folder names Google uses for the trash bucket, lowercased. A Takeout export
/// only contains one when the user asked for it, and its photos are deleted —
/// importing them would resurrect them, so they are skipped.
///
/// Localized, and necessarily incomplete: a locale not listed here imports its
/// trash as ordinary timeline photos. Extend as reports come in rather than
/// guessing at transliterations.
pub const TRASH_FOLDERS: &[&str] = &[
"trash",
"bin",
"papierkorb",
"corbeille",
"papelera",
"papelera de reciclaje",
"cestino",
"lixeira",
"prullenbak",
"kosz",
"koš",
"skräpkorgen",
"papperskorg",
"papirkurv",
"roskakori",
"kuka",
"回收站",
"回收筒",
"ゴミ箱",
"휴지통",
"корзина",
"кошик",
"çöp kutusu",
"σκουπίδια",
];
/// Sidecar file names that describe the *folder* (its album title) rather than
/// one photo, lowercased. Google has renamed this file more than once.
const ALBUM_METADATA_NAMES: &[&str] = &[
"metadata.json",
"album-metadata.json",
"print-subscriptions.json",
"shared_album_comments.json",
"user-generated-memory-titles.json",
];
/// The suffix Google appends to a photo's sidecar, lowercased. Truncation means
/// only a *prefix* of it may survive (see [`strip_sidecar_suffix`]).
const SUPPLEMENTAL: &str = ".supplemental-metadata";
/// File extensions imported as photos or videos. Anything else in the export —
/// `archive_browser.html`, `.csv` activity logs, `print-subscriptions.json` —
/// is not media and is ignored.
const MEDIA_EXTENSIONS: &[(&str, &str)] = &[
("jpg", "image/jpeg"),
("jpeg", "image/jpeg"),
("png", "image/png"),
("gif", "image/gif"),
("webp", "image/webp"),
("bmp", "image/bmp"),
("tif", "image/tiff"),
("tiff", "image/tiff"),
("heic", "image/heic"),
("heif", "image/heif"),
("avif", "image/avif"),
("dng", "image/x-adobe-dng"),
("cr2", "image/x-canon-cr2"),
("cr3", "image/x-canon-cr3"),
("nef", "image/x-nikon-nef"),
("arw", "image/x-sony-arw"),
("orf", "image/x-olympus-orf"),
("rw2", "image/x-panasonic-rw2"),
("raf", "image/x-fuji-raf"),
("mp4", "video/mp4"),
("m4v", "video/x-m4v"),
("mov", "video/quicktime"),
("avi", "video/x-msvideo"),
("mkv", "video/x-matroska"),
("webm", "video/webm"),
("3gp", "video/3gpp"),
("mts", "video/mp2t"),
("m2ts", "video/mp2t"),
("mpg", "video/mpeg"),
("mpeg", "video/mpeg"),
];
/// The media type for `name`'s extension, or `None` when it is not media this
/// import handles.
pub fn media_type_of(name: &str) -> Option<&'static str> {
let ext = Path::new(name)
.extension()
.and_then(|e| e.to_str())?
.to_ascii_lowercase();
MEDIA_EXTENSIONS
.iter()
.find(|(candidate, _)| *candidate == ext)
.map(|(_, media_type)| *media_type)
}
/// Where a photo sits in the export, which decides whether it joins an album.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub enum Bucket {
/// A named album folder — the photo joins that album *and* the timeline.
Album(String),
/// A year folder (`Photos from 2019`) or the export root: timeline only.
Timeline,
/// The archive bucket: timeline only, like Google shows it.
Archived,
}
/// One media file found in the export.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct TakeoutPhoto {
/// Index into the archive list the scan was given.
pub archive: usize,
/// The entry's full path inside that archive — how the importer reopens it.
pub entry: String,
/// The name to upload under: the sidecar's `title` when it has one (Google
/// mangles long names on disk but keeps the original there), else the file
/// name on disk.
pub name: String,
/// Uncompressed size in bytes, from the zip's central directory.
pub size: u64,
/// Media type from the extension.
pub media_type: String,
/// Capture time in epoch seconds, from `photoTakenTime` (falling back to
/// `creationTime`). `None` when the photo has no sidecar — the upload then
/// defaults it to now, as the SDK does.
pub capture_time: Option<i64>,
/// The sidecar's `favorited` flag.
pub favorite: bool,
/// Which bucket the file was found in.
pub bucket: Bucket,
}
/// Everything one scan found, ready to be turned into an import plan.
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
pub struct TakeoutScan {
/// Every importable media file, in archive then entry order.
pub photos: Vec<TakeoutPhoto>,
/// Album names in the order first seen, so the importer creates them in a
/// stable order.
pub albums: Vec<String>,
/// Media files skipped because they sit in the trash bucket.
pub skipped_trashed: usize,
/// Entries that were neither media nor a sidecar (`archive_browser.html`,
/// activity `.csv`s, `.mp` motion-photo parts).
pub skipped_other: usize,
}
impl TakeoutScan {
/// Total bytes of media to be read, before duplicate detection.
pub fn total_bytes(&self) -> u64 {
self.photos.iter().map(|p| p.size).sum()
}
}
/// Google's per-photo sidecar. Only the fields the import uses are read; the
/// file also carries geo data, people tags and view counts.
#[derive(Debug, Default, Deserialize)]
struct PhotoSidecar {
#[serde(default)]
title: Option<String>,
#[serde(rename = "photoTakenTime", default)]
photo_taken_time: Option<SidecarTime>,
#[serde(rename = "creationTime", default)]
creation_time: Option<SidecarTime>,
#[serde(default)]
favorited: bool,
}
/// A sidecar timestamp: epoch seconds, as a *string*.
#[derive(Debug, Deserialize)]
struct SidecarTime {
#[serde(default)]
timestamp: Option<String>,
}
impl SidecarTime {
fn epoch_seconds(&self) -> Option<i64> {
self.timestamp.as_ref()?.parse().ok()
}
}
/// A folder-level sidecar. `title` is the album name as the user typed it,
/// which is what the folder name is derived (and sometimes truncated) from.
#[derive(Debug, Deserialize)]
struct AlbumSidecar {
#[serde(default)]
title: Option<String>,
}
/// One entry as the scan sees it, independent of where it came from.
struct RawEntry {
archive: usize,
entry: String,
/// `<folder>/<file>` with the export root and service folder removed.
relative: String,
size: u64,
}
impl RawEntry {
/// The folder component of `relative`, or `""` at the service root.
fn folder(&self) -> &str {
match self.relative.rfind('/') {
Some(index) => &self.relative[..index],
None => "",
}
}
/// The file name component of `relative`.
fn file_name(&self) -> &str {
match self.relative.rfind('/') {
Some(index) => &self.relative[index + 1..],
None => &self.relative,
}
}
}
/// Scan a set of Takeout zip archives into one plan.
///
/// The archives are treated as one export: an album folder split across parts
/// is one album, and a sidecar in part 3 is matched to its photo in part 1.
/// Reads only the central directory and the JSON sidecars — no media bytes, so
/// this stays fast on a 200 GB export.
pub fn scan_archives(archives: &[PathBuf]) -> CoreResult<TakeoutScan> {
let mut entries: Vec<RawEntry> = Vec::new();
let mut sidecars: HashMap<String, Vec<u8>> = HashMap::new();
for (index, path) in archives.iter().enumerate() {
let file = std::fs::File::open(path)
.map_err(|e| CoreError::invalid(format!("cannot open {}: {e}", path.display())))?;
let mut zip = zip::ZipArchive::new(std::io::BufReader::new(file)).map_err(|e| {
CoreError::invalid(format!("{} is not a readable zip: {e}", path.display()))
})?;
for position in 0..zip.len() {
let mut entry = zip.by_index(position).map_err(|e| {
CoreError::invalid(format!("{}: unreadable entry: {e}", path.display()))
})?;
if entry.is_dir() {
continue;
}
// `enclosed_name` refuses paths that escape the archive root, which
// is exactly the check we want before using one as a key.
let Some(name) = entry.enclosed_name() else {
continue;
};
let name = name.to_string_lossy().replace('\\', "/");
let Some(relative) = strip_export_prefix(&name) else {
continue;
};
let size = entry.size();
// Sidecars are read now, while the entry is open: they are a few
// hundred bytes each and reopening the archive per photo later would
// be a second full pass.
if relative.to_ascii_lowercase().ends_with(".json") {
let mut body = Vec::with_capacity(size.min(64 * 1024) as usize);
if entry.read_to_end(&mut body).is_ok() {
sidecars.insert(relative.to_string(), body);
}
continue;
}
entries.push(RawEntry {
archive: index,
entry: name.clone(),
relative: relative.to_string(),
size,
});
}
}
Ok(assemble(entries, &sidecars))
}
/// Build the scan from raw entries plus the sidecar bodies keyed by their path
/// inside the export. Split out from [`scan_archives`] so the matching rules can
/// be tested without building zip fixtures.
fn assemble(entries: Vec<RawEntry>, sidecars: &HashMap<String, Vec<u8>>) -> TakeoutScan {
// Album titles per folder, from the folder-level sidecars.
let mut album_titles: HashMap<String, String> = HashMap::new();
for (path, body) in sidecars {
let (folder, file) = split_folder(path);
if !ALBUM_METADATA_NAMES.contains(&file.to_ascii_lowercase().as_str()) {
continue;
}
if let Ok(album) = serde_json::from_slice::<AlbumSidecar>(body)
&& let Some(title) = album.title.filter(|t| !t.trim().is_empty())
{
album_titles.insert(folder.to_string(), title);
}
}
// Per-photo sidecars, indexed by folder so a truncated name only ever
// matches within its own folder.
let mut per_folder_sidecars: HashMap<String, BTreeMap<String, &Vec<u8>>> = HashMap::new();
for (path, body) in sidecars {
let (folder, file) = split_folder(path);
if ALBUM_METADATA_NAMES.contains(&file.to_ascii_lowercase().as_str()) {
continue;
}
per_folder_sidecars
.entry(folder.to_string())
.or_default()
.insert(file.to_string(), body);
}
let mut scan = TakeoutScan::default();
let mut seen_albums: HashMap<String, ()> = HashMap::new();
for raw in &entries {
let folder = raw.folder().to_string();
let file_name = raw.file_name().to_string();
if is_trash_folder(&folder) {
if media_type_of(&file_name).is_some() {
scan.skipped_trashed += 1;
}
continue;
}
let Some(media_type) = media_type_of(&file_name) else {
scan.skipped_other += 1;
continue;
};
let sidecar = per_folder_sidecars
.get(&folder)
.and_then(|in_folder| find_sidecar(&file_name, in_folder))
.and_then(|body| serde_json::from_slice::<PhotoSidecar>(body).ok())
.unwrap_or_default();
let bucket = match album_titles.get(&folder) {
Some(title) => Bucket::Album(title.clone()),
None if is_archive_folder(&folder) => Bucket::Archived,
None => Bucket::Timeline,
};
if let Bucket::Album(title) = &bucket
&& seen_albums.insert(title.clone(), ()).is_none()
{
scan.albums.push(title.clone());
}
// The sidecar's `title` is the name Google had before it mangled the
// on-disk one (truncation, `(1)` suffixes, character replacement), so it
// is the better name to upload under — but only when it still looks like
// the same file, i.e. the extension agrees.
let name = sidecar
.title
.as_deref()
.map(str::trim)
.filter(|title| !title.is_empty() && same_extension(title, &file_name))
.unwrap_or(&file_name)
.to_string();
scan.photos.push(TakeoutPhoto {
archive: raw.archive,
entry: raw.entry.clone(),
name,
size: raw.size,
media_type: media_type.to_string(),
capture_time: sidecar
.photo_taken_time
.as_ref()
.and_then(SidecarTime::epoch_seconds)
.or_else(|| {
sidecar
.creation_time
.as_ref()
.and_then(SidecarTime::epoch_seconds)
}),
favorite: sidecar.favorited,
bucket,
});
}
scan
}
/// Drop the export root (`Takeout/`) and the localized service folder from a
/// path inside the archive, leaving `<folder>/<file>` or `<file>`.
///
/// `None` for a path with no service folder at all (the export's own
/// `archive_browser.html`), which carries nothing to import.
fn strip_export_prefix(path: &str) -> Option<&str> {
let mut rest = path;
// The root is usually `Takeout/`, but an export unzipped and re-zipped by
// the user may not have it — so the root is dropped only when a service
// folder follows it, which the component count decides.
let components = rest.split('/').count();
if components >= 3 {
rest = rest.split_once('/')?.1;
}
// Drop the service folder (`Google Photos`, `Google Fotos`, …).
let (_service, tail) = rest.split_once('/')?;
Some(tail)
}
/// Split `<folder>/<file>` into its parts; the folder is `""` at the root.
fn split_folder(path: &str) -> (&str, &str) {
match path.rfind('/') {
Some(index) => (&path[..index], &path[index + 1..]),
None => ("", path),
}
}
fn is_trash_folder(folder: &str) -> bool {
let name = folder.rsplit('/').next().unwrap_or(folder).to_lowercase();
TRASH_FOLDERS.contains(&name.as_str())
}
/// Google's archive bucket, which is not an album but is not the plain timeline
/// either. Only the English name is matched; a missed locale simply imports as
/// [`Bucket::Timeline`], which is the same upload either way.
fn is_archive_folder(folder: &str) -> bool {
let name = folder.rsplit('/').next().unwrap_or(folder).to_lowercase();
name == "archive"
}
/// Whether two file names end in the same extension, case-insensitively.
fn same_extension(a: &str, b: &str) -> bool {
let ext = |name: &str| {
Path::new(name)
.extension()
.and_then(|e| e.to_str())
.map(str::to_ascii_lowercase)
};
ext(a) == ext(b)
}
/// Strip a sidecar's `.json` and any `.supplemental-metadata` suffix, including
/// the *truncated* forms Google produces.
///
/// Google caps the sidecar's file name at 51 characters, so a long photo name
/// leaves only a prefix of the suffix behind — `IMG_1234.jpg.supplemental-me`,
/// `…-metad`, or nothing at all. Any non-empty prefix of the suffix is stripped;
/// a bare `.json` sidecar loses only that.
fn strip_sidecar_suffix(sidecar: &str) -> String {
let stem = sidecar
.strip_suffix(".json")
.or_else(|| sidecar.strip_suffix(".JSON"))
.unwrap_or(sidecar);
// Longest match first so `.supplemental-metadata` wins over `.s`.
for length in (2..=SUPPLEMENTAL.len()).rev() {
let candidate = &SUPPLEMENTAL[..length];
if let Some(head) = stem.strip_suffix(candidate) {
return head.to_string();
}
}
stem.to_string()
}
/// Move a trailing `(N)` from after the extension to before it: Google writes
/// the sidecar of `IMG_1234(1).jpg` as `IMG_1234.jpg(1).json`.
fn unswap_duplicate_marker(stem: &str) -> Option<String> {
let marker_start = stem.rfind('(')?;
let marker = &stem[marker_start..];
if !marker.ends_with(')')
|| !marker[1..marker.len() - 1]
.chars()
.all(|c| c.is_ascii_digit())
{
return None;
}
let head = &stem[..marker_start];
let dot = head.rfind('.')?;
Some(format!("{}{}{}", &head[..dot], marker, &head[dot..]))
}
/// Find the sidecar belonging to `file_name` among the JSON files of its folder.
///
/// Tried in order of confidence: the exact name, the `(N)`-swapped name, then a
/// prefix match for the case where truncation cut into the photo name itself.
fn find_sidecar<'a>(
file_name: &str,
in_folder: &BTreeMap<String, &'a Vec<u8>>,
) -> Option<&'a Vec<u8>> {
let mut prefix_match: Option<&'a Vec<u8>> = None;
let mut prefix_len = 0usize;
for (sidecar_name, body) in in_folder {
let stem = strip_sidecar_suffix(sidecar_name);
if stem == file_name {
return Some(body);
}
if unswap_duplicate_marker(&stem).as_deref() == Some(file_name) {
return Some(body);
}
// Truncated: the stem is a prefix of the photo's name. Keep the longest
// such prefix — a folder holding `IMG_1.jpg` and `IMG_12.jpg` must not
// let the shorter stem claim the longer photo.
if !stem.is_empty() && file_name.starts_with(&stem) && stem.len() > prefix_len {
prefix_len = stem.len();
prefix_match = Some(body);
}
}
prefix_match
}
#[cfg(test)]
mod tests {
use super::*;
fn entry(archive: usize, path: &str, size: u64) -> RawEntry {
RawEntry {
archive,
entry: path.to_string(),
relative: strip_export_prefix(path).unwrap().to_string(),
size,
}
}
fn sidecar(map: &mut HashMap<String, Vec<u8>>, path: &str, body: &str) {
let relative = strip_export_prefix(path).unwrap().to_string();
map.insert(relative, body.as_bytes().to_vec());
}
#[test]
fn strips_export_root_and_localized_service_folder() {
assert_eq!(
strip_export_prefix("Takeout/Google Fotos/Island 2019/IMG_1.jpg"),
Some("Island 2019/IMG_1.jpg")
);
// Already unwrapped by the user: no `Takeout/` root left.
assert_eq!(
strip_export_prefix("Google Photos/IMG_1.jpg"),
Some("IMG_1.jpg")
);
// Nothing below a service folder is nothing to import.
assert_eq!(strip_export_prefix("archive_browser.html"), None);
}
#[test]
fn album_folder_becomes_an_album_year_folder_does_not() {
let mut sidecars = HashMap::new();
sidecar(
&mut sidecars,
"Takeout/Google Photos/Iceland 2019/metadata.json",
r#"{"title":"Iceland 2019"}"#,
);
let scan = assemble(
vec![
entry(0, "Takeout/Google Photos/Iceland 2019/IMG_1.jpg", 10),
entry(0, "Takeout/Google Photos/Photos from 2019/IMG_2.jpg", 20),
],
&sidecars,
);
assert_eq!(scan.albums, vec!["Iceland 2019"]);
assert_eq!(scan.photos[0].bucket, Bucket::Album("Iceland 2019".into()));
assert_eq!(scan.photos[1].bucket, Bucket::Timeline);
assert_eq!(scan.total_bytes(), 30);
}
#[test]
fn trashed_photos_are_skipped_and_counted() {
let scan = assemble(
vec![
entry(0, "Takeout/Google Photos/Papierkorb/IMG_1.jpg", 10),
entry(0, "Takeout/Google Photos/Bin/IMG_2.jpg", 10),
entry(0, "Takeout/Google Photos/Photos from 2019/IMG_3.jpg", 10),
],
&HashMap::new(),
);
assert_eq!(scan.skipped_trashed, 2);
assert_eq!(scan.photos.len(), 1);
assert_eq!(scan.photos[0].name, "IMG_3.jpg");
}
#[test]
fn non_media_entries_are_ignored() {
let scan = assemble(
vec![
entry(0, "Takeout/Google Photos/Photos from 2019/IMG_1.jpg", 10),
entry(0, "Takeout/Google Photos/Photos from 2019/notes.txt", 10),
],
&HashMap::new(),
);
assert_eq!(scan.photos.len(), 1);
assert_eq!(scan.skipped_other, 1);
}
#[test]
fn sidecar_supplies_capture_time_and_favourite() {
let mut sidecars = HashMap::new();
sidecar(
&mut sidecars,
"Takeout/Google Photos/Photos from 2019/IMG_1.jpg.supplemental-metadata.json",
r#"{"title":"IMG_1.jpg","photoTakenTime":{"timestamp":"1560000000"},"favorited":true}"#,
);
let scan = assemble(
vec![entry(
0,
"Takeout/Google Photos/Photos from 2019/IMG_1.jpg",
10,
)],
&sidecars,
);
assert_eq!(scan.photos[0].capture_time, Some(1_560_000_000));
assert!(scan.photos[0].favorite);
}
#[test]
fn creation_time_is_the_fallback_capture_time() {
let mut sidecars = HashMap::new();
sidecar(
&mut sidecars,
"Takeout/Google Photos/Photos from 2019/IMG_1.jpg.json",
r#"{"creationTime":{"timestamp":"1500000000"}}"#,
);
let scan = assemble(
vec![entry(
0,
"Takeout/Google Photos/Photos from 2019/IMG_1.jpg",
10,
)],
&sidecars,
);
assert_eq!(scan.photos[0].capture_time, Some(1_500_000_000));
}
#[test]
fn truncated_supplemental_suffixes_still_match() {
// Google caps the sidecar name at 51 chars, cutting into the suffix.
assert_eq!(
strip_sidecar_suffix("IMG_1.jpg.supplemental-me.json"),
"IMG_1.jpg"
);
assert_eq!(
strip_sidecar_suffix("IMG_1.jpg.supplemental-metadata.json"),
"IMG_1.jpg"
);
assert_eq!(strip_sidecar_suffix("IMG_1.jpg.json"), "IMG_1.jpg");
}
#[test]
fn duplicate_marker_after_the_extension_is_moved_back() {
assert_eq!(
unswap_duplicate_marker("IMG_1.jpg(1)").as_deref(),
Some("IMG_1(1).jpg")
);
assert_eq!(unswap_duplicate_marker("IMG_1.jpg"), None);
let mut sidecars = HashMap::new();
sidecar(
&mut sidecars,
"Takeout/Google Photos/Photos from 2019/IMG_1.jpg(1).json",
r#"{"photoTakenTime":{"timestamp":"1600000000"}}"#,
);
let scan = assemble(
vec![entry(
0,
"Takeout/Google Photos/Photos from 2019/IMG_1(1).jpg",
10,
)],
&sidecars,
);
assert_eq!(scan.photos[0].capture_time, Some(1_600_000_000));
}
#[test]
fn longest_prefix_wins_when_the_photo_name_itself_was_truncated() {
let mut sidecars = HashMap::new();
sidecar(
&mut sidecars,
"Takeout/Google Photos/Photos from 2019/IMG_1.json",
r#"{"photoTakenTime":{"timestamp":"1"}}"#,
);
sidecar(
&mut sidecars,
"Takeout/Google Photos/Photos from 2019/IMG_12.json",
r#"{"photoTakenTime":{"timestamp":"2"}}"#,
);
let scan = assemble(
vec![entry(
0,
"Takeout/Google Photos/Photos from 2019/IMG_12345.jpg",
10,
)],
&sidecars,
);
assert_eq!(scan.photos[0].capture_time, Some(2));
}
#[test]
fn sidecar_title_replaces_a_mangled_on_disk_name() {
let mut sidecars = HashMap::new();
sidecar(
&mut sidecars,
"Takeout/Google Photos/Photos from 2019/a_very_long_name_that_google_.jpg.json",
r#"{"title":"a_very_long_name_that_google_truncated.jpg"}"#,
);
let scan = assemble(
vec![entry(
0,
"Takeout/Google Photos/Photos from 2019/a_very_long_name_that_google_.jpg",
10,
)],
&sidecars,
);
assert_eq!(
scan.photos[0].name,
"a_very_long_name_that_google_truncated.jpg"
);
}
#[test]
fn a_title_with_a_different_extension_is_not_trusted() {
// Motion photos name their `.MP` part after the `.jpg`; taking the title
// verbatim would upload the video under an image name.
let mut sidecars = HashMap::new();
sidecar(
&mut sidecars,
"Takeout/Google Photos/Photos from 2019/IMG_1.mp4.json",
r#"{"title":"IMG_1.jpg"}"#,
);
let scan = assemble(
vec![entry(
0,
"Takeout/Google Photos/Photos from 2019/IMG_1.mp4",
10,
)],
&sidecars,
);
assert_eq!(scan.photos[0].name, "IMG_1.mp4");
assert_eq!(scan.photos[0].media_type, "video/mp4");
}
#[test]
fn an_album_split_across_archives_stays_one_album() {
let mut sidecars = HashMap::new();
sidecar(
&mut sidecars,
"Takeout/Google Photos/Iceland 2019/metadata.json",
r#"{"title":"Iceland 2019"}"#,
);
let scan = assemble(
vec![
entry(0, "Takeout/Google Photos/Iceland 2019/IMG_1.jpg", 10),
entry(1, "Takeout/Google Photos/Iceland 2019/IMG_2.jpg", 10),
],
&sidecars,
);
assert_eq!(scan.albums, vec!["Iceland 2019"]);
assert_eq!(scan.photos.len(), 2);
assert_eq!(scan.photos[1].archive, 1);
}
#[test]
fn archive_bucket_imports_to_the_timeline() {
let scan = assemble(
vec![entry(0, "Takeout/Google Photos/Archive/IMG_1.jpg", 10)],
&HashMap::new(),
);
assert_eq!(scan.photos[0].bucket, Bucket::Archived);
assert!(scan.albums.is_empty());
}
/// Write a zip holding `files` and return its path. No `tempfile`
/// dev-dependency: the daemon crates deliberately carry none, and one
/// uniquely named file under the temp dir is all this needs.
fn write_zip(tag: &str, files: &[(&str, &[u8])]) -> PathBuf {
use std::io::Write;
let path = std::env::temp_dir().join(format!(
"pdfs-takeout-test-{tag}-{}.zip",
std::process::id()
));
let file = std::fs::File::create(&path).unwrap();
let mut zip = zip::ZipWriter::new(file);
let options: zip::write::FileOptions<'_, ()> =
zip::write::FileOptions::default().compression_method(zip::CompressionMethod::Stored);
for (name, body) in files {
zip.start_file(*name, options).unwrap();
zip.write_all(body).unwrap();
}
zip.finish().unwrap();
path
}
#[test]
fn scans_a_two_part_export_where_the_sidecar_is_in_the_other_part() {
// The case that makes the whole set one scan: part 1 has the photo,
// part 2 has its metadata and the album title.
let first = write_zip(
"part1",
&[(
"Takeout/Google Photos/Iceland 2019/IMG_1.jpg",
b"the photo bytes",
)],
);
let second = write_zip(
"part2",
&[
(
"Takeout/Google Photos/Iceland 2019/metadata.json",
br#"{"title":"Iceland 2019"}"#.as_slice(),
),
(
"Takeout/Google Photos/Iceland 2019/IMG_1.jpg.supplemental-metadata.json",
br#"{"title":"IMG_1.jpg","photoTakenTime":{"timestamp":"1560000000"}}"#
.as_slice(),
),
(
"Takeout/Google Photos/Trash/IMG_9.jpg",
b"deleted".as_slice(),
),
],
);
let scan = scan_archives(&[first.clone(), second.clone()]).unwrap();
let _ = std::fs::remove_file(&first);
let _ = std::fs::remove_file(&second);
assert_eq!(scan.photos.len(), 1);
assert_eq!(scan.photos[0].name, "IMG_1.jpg");
assert_eq!(scan.photos[0].archive, 0, "the photo is in part 1");
assert_eq!(scan.photos[0].size, "the photo bytes".len() as u64);
assert_eq!(scan.photos[0].capture_time, Some(1_560_000_000));
assert_eq!(scan.photos[0].bucket, Bucket::Album("Iceland 2019".into()));
assert_eq!(scan.albums, vec!["Iceland 2019"]);
assert_eq!(scan.skipped_trashed, 1);
}
#[test]
fn a_file_that_is_not_a_zip_is_a_readable_error() {
let path =
std::env::temp_dir().join(format!("pdfs-takeout-not-a-zip-{}", std::process::id()));
std::fs::write(&path, b"not a zip at all").unwrap();
let error = scan_archives(std::slice::from_ref(&path)).unwrap_err();
let _ = std::fs::remove_file(&path);
assert!(
error.message.contains("not a readable zip"),
"unexpected message: {}",
error.message
);
}
}
+5
View File
@@ -5,10 +5,14 @@ edition.workspace = true
license.workspace = true
authors.workspace = true
rust-version.workspace = true
repository.workspace = true
homepage.workspace = true
description = "FUSE on-demand filesystem for Proton Drive"
[dependencies]
pdfs-core.workspace = true
zip.workspace = true
sha1.workspace = true
proton-drive-rs.workspace = true
futures.workspace = true
tokio.workspace = true
@@ -16,6 +20,7 @@ tracing.workspace = true
fuser.workspace = true
libc.workspace = true
serde_json.workspace = true
base64.workspace = true
image.workspace = true
notify = "8"
parking_lot.workspace = true
+7 -1
View File
@@ -1,6 +1,7 @@
//! Remote-event invalidation and local-index background workers.
use super::*;
use crate::shutdown::Shutdown;
/// Drop `uid` from one mount's tree and tell that mount's kernel session the
/// entry is gone. Shared by the trash and delete paths, which differ only in
@@ -437,6 +438,7 @@ pub(super) fn run_local_index(
indexing: Arc<AtomicBool>,
transfers: Arc<TransferRegistry>,
mountpoint: PathBuf,
shutdown: Arc<Shutdown>,
) {
loop {
let age = db.local_indexed_at().ok().flatten();
@@ -445,7 +447,11 @@ pub(super) fn run_local_index(
if stale {
scan_local_once(&db, &indexing, &transfers, &mountpoint);
}
std::thread::sleep(LOCAL_INDEX_CHECK);
// Interruptible, so teardown joins this thread instead of leaving it to
// keep rewriting the index for a mount that is gone (bugs.md B44).
if !shutdown.sleep(LOCAL_INDEX_CHECK) {
return;
}
}
}
+144 -7
View File
@@ -11,7 +11,7 @@
use std::io::{BufRead, BufReader, Write as _};
use std::os::unix::net::{UnixListener, UnixStream};
use std::path::{Path, PathBuf};
use std::path::{Component, Path, PathBuf};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::{Duration, Instant};
@@ -104,17 +104,51 @@ fn read_request_line(reader: &mut impl BufRead) -> std::io::Result<Option<String
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))
}
/// Turn a CLI-supplied path into the mount that owns it and the path relative
/// to *that* mount's root.
///
/// [`rel_to_mount`] is the older, primary-only reading, and it is what every
/// mutating request still uses. It rejects anything outside the primary
/// mountpoint, which leaves a secondary on-demand mount — a synced folder
/// switched out of mirror mode — with no way to be named at all: `pdfs ls` and
/// `pdfs refresh` both answered "is not under the mountpoint", so a folder
/// serving a stale listing had no manual escape hatch (`docs/BUGS.md` B86).
///
/// A relative path keeps the old meaning, relative to the primary mount, since
/// there is nothing in it to choose a mount by.
fn route_to_mount(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
let p = Path::new(path);
if !p.is_absolute() {
return Ok((core.clone(), p.to_path_buf()));
}
if let Some(routed) = core.rooted_at(p) {
return Ok(routed);
}
// Fall back to the primary reading so the error text stays the familiar one
// when the path is simply outside everything this daemon has mounted.
rel_to_mount(mountpoint, path).map(|rel| (core.clone(), rel))
}
/// Turn a CLI-supplied path into a mountpoint-relative path. An absolute path
/// must live under `mountpoint`; a relative path is taken as already relative to
/// the mount root.
fn rel_to_mount(mountpoint: &Path, path: &str) -> CoreResult<PathBuf> {
let p = Path::new(path);
if p.is_absolute() {
let relative = if p.is_absolute() {
p.strip_prefix(mountpoint)
.map(Path::to_path_buf)
.map_err(|_| CoreError::invalid(format!("{path} is not under the mountpoint")))
} else {
Ok(p.to_path_buf())
}?;
if relative
.components()
.all(|component| matches!(component, Component::Normal(_) | Component::CurDir))
{
Ok(relative)
} else {
Err(CoreError::invalid(format!("{path} escapes the mountpoint")))
}
}
@@ -193,8 +227,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
Ok(CtlRequest::ListPins) => CtlResponse::Pins {
pins: core.cache.list_pins(),
},
Ok(CtlRequest::ListDir { path }) => match rel_to_mount(mountpoint, &path) {
Ok(rel) => match core.list_dir(&rel) {
Ok(CtlRequest::ListDir { path }) => match route_to_mount(core, mountpoint, &path) {
Ok((core, rel)) => match core.list_dir(&rel) {
Ok(entries) => CtlResponse::Entries { entries },
Err(e) => CtlResponse::error(e),
},
@@ -202,9 +236,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
},
Ok(CtlRequest::Refresh { scope }) => {
let result = match &scope {
RefreshScope::Dir { path } => {
rel_to_mount(mountpoint, path).and_then(|rel| core.refresh_dir(&rel))
}
RefreshScope::Dir { path } => route_to_mount(core, mountpoint, path)
.and_then(|(core, rel)| core.refresh_dir(&rel)),
RefreshScope::Trash => {
core.invalidate_trash();
Ok(())
@@ -296,6 +329,34 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
items: core.photo_thumbs(&parsed),
}
}
Ok(CtlRequest::FileThumbs { items, generation }) => {
match core.file_thumbs(&items, generation) {
Some(items) => CtlResponse::Thumbs { items },
None => CtlResponse::FileThumbsStale,
}
}
Ok(CtlRequest::ReserveFileThumbGeneration) => CtlResponse::FileThumbGeneration {
generation: core.reserve_file_thumb_generation(),
},
Ok(CtlRequest::CancelFileThumbs { generation }) => {
core.cancel_file_thumbs(generation);
CtlResponse::Ok {
message: "thumbnail requests cancelled".to_string(),
}
}
Ok(CtlRequest::StartThumbnailBuild { path }) => match rel_to_mount(mountpoint, &path) {
Ok(rel) => match core.start_thumbnail_build(rel) {
Ok(status) => CtlResponse::ThumbnailBuild { status },
Err(error) => CtlResponse::error(error),
},
Err(error) => CtlResponse::error(error),
},
Ok(CtlRequest::CancelThumbnailBuild) => CtlResponse::ThumbnailBuild {
status: core.cancel_thumbnail_build(),
},
Ok(CtlRequest::ThumbnailBuildStatus) => CtlResponse::ThumbnailBuild {
status: core.thumbnail_build_status(),
},
Ok(CtlRequest::OpenPhoto { uid }) => match parse_uid(&uid) {
Some(u) => match core.open_photo(&u) {
Ok(p) => CtlResponse::FilePath {
@@ -305,6 +366,51 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
},
None => CtlResponse::error(CoreError::invalid(format!("bad uid: {uid}"))),
},
// The scan alone walks every archive's central directory, so even the
// dry run goes to a background thread and is read back through
// `ImportStatus` rather than answered inline.
Ok(CtlRequest::ImportTakeout { archives, dry_run }) => {
if archives.is_empty() {
CtlResponse::error(CoreError::invalid("no Takeout archives given"))
} else {
let paths: Vec<PathBuf> = archives.into_iter().map(PathBuf::from).collect();
let missing: Vec<String> = paths
.iter()
.filter(|path| !path.is_file())
.map(|path| path.display().to_string())
.collect();
if !missing.is_empty() {
CtlResponse::error(CoreError::not_found(format!(
"cannot read: {}",
missing.join(", ")
)))
} else {
let count = paths.len();
match core.spawn_takeout_import(paths, dry_run) {
Ok(()) => CtlResponse::Ok {
message: format!(
"{} {count} archive(s)",
if dry_run { "scanning" } else { "importing" }
),
},
Err(e) => CtlResponse::error(e),
}
}
}
}
Ok(CtlRequest::ImportStatus) => {
let (running, summary) = crate::takeout::import_status();
CtlResponse::ImportStatus { running, summary }
}
Ok(CtlRequest::CancelImport) => {
if crate::takeout::cancel_import() {
CtlResponse::Ok {
message: "stopping the import".to_string(),
}
} else {
CtlResponse::error(CoreError::invalid("no import is running"))
}
}
Ok(CtlRequest::UploadPhoto {
name,
media_type,
@@ -1084,6 +1190,12 @@ pub(crate) fn run_control_socket(
) {
info!("control socket listening");
for conn in listener.incoming() {
// Checked on every accept, including the deliberate poke teardown sends
// to wake this thread out of a blocking accept (bugs.md B44).
if core.shutdown.is_stopping() {
info!("control socket closing");
return;
}
match conn {
Ok(stream) => {
let Some(permit) = ControlHandlerPermit::acquire() else {
@@ -1146,4 +1258,29 @@ mod request_limit_tests {
std::io::ErrorKind::InvalidData
);
}
#[test]
fn accepts_paths_inside_the_mountpoint() {
let mountpoint = Path::new("/mnt/proton");
assert_eq!(
rel_to_mount(mountpoint, "Photos/2026").unwrap(),
Path::new("Photos/2026")
);
assert_eq!(
rel_to_mount(mountpoint, "/mnt/proton/Photos/2026").unwrap(),
Path::new("Photos/2026")
);
assert_eq!(
rel_to_mount(mountpoint, "/mnt/proton").unwrap(),
Path::new("")
);
}
#[test]
fn rejects_paths_that_escape_the_mountpoint() {
let mountpoint = Path::new("/mnt/proton");
assert!(rel_to_mount(mountpoint, "../escape").is_err());
assert!(rel_to_mount(mountpoint, "Photos/../../escape").is_err());
assert!(rel_to_mount(mountpoint, "/mnt/elsewhere").is_err());
}
}
+640 -57
View File
@@ -20,10 +20,14 @@
//! are kept as a conflict copy rather than discarded — see
//! [`Core::revision_conflict`].
use std::collections::HashMap;
use std::fs::File;
use std::path::{Path, PathBuf};
use std::sync::atomic::Ordering;
use std::time::Duration;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::{Duration, Instant};
use parking_lot::Mutex;
use pdfs_core::batch;
use pdfs_core::cache::{Baseline, StagedWrite};
@@ -36,17 +40,132 @@ use tracing::{debug, error, info, warn};
use super::state::Intervals;
use super::transfers::CountingReader;
use super::{
Core, DRAIN_BACKOFF_MAX, DRAIN_BACKOFF_MIN, DRAIN_IDLE_POLL, ROOT_INO, conflict_name,
Core, DRAIN_BACKOFF_MAX, DRAIN_BACKOFF_MIN, DRAIN_IDLE_POLL, DRAIN_REVISION_DEBOUNCE,
DRAIN_REVISION_DEBOUNCE_MAX, ROOT_INO, UPLOAD_TIME_MEMORY, WriteAuthority, conflict_name,
is_already_exists, is_gone, is_local_uid_str, media_type_for, node_revision_id, node_size,
now_millis, now_secs, parse_node_uid,
};
const DRAIN_ACCESS_RECHECK: Duration = Duration::from_secs(5);
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
/// How long an op may stay access-deferred before the deferral is reported as a
/// failure instead of retried in silence.
///
/// An access deferral costs no attempt and records no error, which is right for
/// the case it was written for: a share downgraded while the queue was moving,
/// undone a moment later. It is wrong for anything that does not clear. Until
/// this window existed such an op was re-deferred every
/// [`DRAIN_ACCESS_RECHECK`] indefinitely with `attempts` at zero, `last_error`
/// null and only a `debug!` line to show for it, so `pdfs status` counted it
/// among ordinary queued uploads and nothing ever said otherwise.
///
/// The condition that produced that in practice — a node absent from the local
/// tree, which [`Db::effective_node_access`] used to report indistinguishably
/// from a revoked permission — is now answered directly by
/// [`Core::resolve_unknown_authority`]. This window remains the backstop for
/// every deferral that does not resolve, including a refetch that keeps not
/// helping.
///
/// Five minutes is far longer than a permission change takes to settle and far
/// shorter than a user's patience with bytes that are not moving. Past it each
/// recheck records a failure, so the op enters the ordinary backoff and, at
/// [`FAILING_ATTEMPTS`], is counted and named by `Response::Status`.
///
/// [`Db::effective_node_access`]: pdfs_core::db::Db::effective_node_access
/// [`FAILING_ATTEMPTS`]: pdfs_core::db::FAILING_ATTEMPTS
const DRAIN_ACCESS_DEFER_LIMIT: Duration = Duration::from_secs(300);
/// A registered cancellation flag for one in-flight upload, deregistered when
/// dropped. See [`Core::begin_cancellable_upload`].
struct UploadCancel {
registry: Arc<Mutex<HashMap<NodeUid, Arc<AtomicBool>>>>,
uid: NodeUid,
flag: Arc<AtomicBool>,
}
impl UploadCancel {
/// Whether this upload was told to stop. Distinguishes an upload we
/// abandoned on purpose from one the network broke.
fn cancelled(&self) -> bool {
self.flag.load(Ordering::Relaxed)
}
fn flag(&self) -> Arc<AtomicBool> {
self.flag.clone()
}
}
impl Drop for UploadCancel {
fn drop(&mut self) {
let mut registry = self.registry.lock();
// By identity: a supersede racing this upload's last bytes may already
// have registered the next one under the same uid.
if registry
.get(&self.uid)
.is_some_and(|current| Arc::ptr_eq(current, &self.flag))
{
registry.remove(&self.uid);
}
}
}
/// The debounce arithmetic of [`Core::revision_debounce`], without the map:
/// the last measured upload time for this node, bounded on both sides.
///
/// A node nobody has uploaded yet gets the fixed grace period, which is the
/// same answer the fixed debounce always gave.
fn adaptive_debounce(measured: Option<Duration>) -> Duration {
measured
.unwrap_or(DRAIN_REVISION_DEBOUNCE)
.clamp(DRAIN_REVISION_DEBOUNCE, DRAIN_REVISION_DEBOUNCE_MAX)
}
#[derive(Debug, Clone, PartialEq, Eq)]
enum DrainDisposition {
Applied,
AccessDeferred,
/// An authority this op needs is missing from the local tree, so no access
/// answer exists for it yet. Carries the uid to ask the remote about.
AuthorityUnknown(NodeUid),
}
/// Why an op went back on the clock without consuming an attempt. Only affects
/// what the user is told — every variant is deferred and reported identically.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum AccessDeferral {
/// An authority the tree knows about refuses the write.
NotWritable,
/// An authority missing from the tree was just re-read from the remote and
/// re-interned; the access check gets another look on the next pass.
AuthorityRefetched,
/// An authority missing from the tree could not be asked about.
RemoteUnreachable,
}
impl AccessDeferral {
fn log_reason(self) -> &'static str {
match self {
Self::NotWritable => "the node is not writable",
Self::AuthorityRefetched => "the node was missing locally and has been re-read",
Self::RemoteUnreachable => "the node is missing locally and the remote is unreachable",
}
}
/// Phrased to complete "<reason> for 300s." in a recorded `last_error`, so
/// the text the user sees names the actual condition rather than the
/// permission question it used to be flattened into.
fn failure_reason(self) -> &'static str {
match self {
Self::NotWritable => "the share has not been writable",
Self::AuthorityRefetched => {
"the node has been missing from the local tree, and re-reading it from the remote \
has not restored it"
}
Self::RemoteUnreachable => {
"the node has been missing from the local tree and the remote has been unreachable"
}
}
}
}
fn pending_op_authorities(op: &PendingOp) -> Result<Vec<NodeUid>, Box<dyn std::error::Error>> {
@@ -89,7 +208,7 @@ fn pending_op_authorities(op: &PendingOp) -> Result<Vec<NodeUid>, Box<dyn std::e
fn run_authorized_drain(
op: &PendingOp,
mut require: impl FnMut(&NodeUid) -> Result<(), fuser::Errno>,
mut authority_of: impl FnMut(&NodeUid) -> WriteAuthority,
apply: impl FnOnce() -> Result<(), Box<dyn std::error::Error>>,
) -> Result<DrainDisposition, Box<dyn std::error::Error>> {
// This check is the admission/linearization point. A downgrade that lands
@@ -97,15 +216,14 @@ fn run_authorized_drain(
// attempt. Every later queued operation rechecks. No permission lock is
// held across remote latency.
for authority in pending_op_authorities(op)? {
match require(&authority) {
Ok(()) => {}
Err(error) if error.code() == libc::EACCES => {
return Ok(DrainDisposition::AccessDeferred);
}
Err(error) => {
return Err(
format!("drain access check failed with errno {}", error.code()).into(),
);
match authority_of(&authority) {
WriteAuthority::Writable => {}
WriteAuthority::Denied => return Ok(DrainDisposition::AccessDeferred),
// Nothing in the tree can answer for this node. Deferring on it
// waits for a permission change that has no reason to come, so the
// caller asks the remote instead (B83).
WriteAuthority::Unknown => {
return Ok(DrainDisposition::AuthorityUnknown(authority));
}
}
}
@@ -127,48 +245,85 @@ impl Core {
/// that is due — one file wedged against a folder that no longer exists must
/// not hold up an unrelated upload behind it. The worker only blocks once
/// nothing is due at all.
pub(crate) fn run_pending_drain(&self) {
///
/// Several of these run at once ([`DRAIN_WORKERS`]), sharing the queue
/// through [`Db::claim_next_due_op`]: the alternative was that a 10 GiB
/// upload held every queued rename, trash and small write behind it for as
/// long as it took. Ordering only has to hold *per node*, and the claim
/// query is what guarantees it — no two workers are ever on one uid.
///
/// `primary` marks the one worker that also runs the queue's idle chores.
/// They are cheap, but they are not per-op work and doing them in every
/// worker would just multiply them.
///
/// [`DRAIN_WORKERS`]: super::DRAIN_WORKERS
/// [`Db::claim_next_due_op`]: pdfs_core::db::Db::claim_next_due_op
pub(crate) fn run_pending_drain(&self, primary: bool) {
loop {
// Between ops, never inside one: an op that has started is either
// retired or released, so stopping here cannot leave the queue with
// a row claimed by a worker that no longer exists (bugs.md B44).
if self.shutdown.is_stopping() {
debug!(primary, "drain worker stopping");
return;
}
let now = now_millis();
// One row, chosen by the database. Reading the whole queue to pick
// one op made a long queue quadratic to drain, and held the shared
// connection — and so every FUSE metadata call — for the duration.
let due = self.db.next_due_op(now).unwrap_or_default();
let Some(op) = due.filter(|_| self.online.load(Ordering::Relaxed)) else {
// Nothing to upload is exactly when the connection is free, so
// this is where the read path's buffered LRU touches get written
// (`ContentCache::flush_touches`). Cheap and a no-op when empty.
self.cache.flush_touches();
self.recover_fsynced_writes();
// A debounced or backed-off op may be waiting: sleep only
// until it becomes due rather than the full idle-poll.
self.wait_for_drain_work_or_due();
continue;
// One row, chosen and *claimed* by the database. Reading the whole
// queue to pick one op made a long queue quadratic to drain, and
// held the shared connection — and so every FUSE metadata call —
// for the duration.
let due = match self.db.claim_next_due_op(now) {
Ok(due) => due,
Err(error) => {
error!(%error, "claiming the next pending operation failed");
self.wait_for_drain_work();
continue;
}
};
let online = self.online.load(Ordering::Relaxed);
let op = match due {
Some(op) if online => op,
idle => {
// Claimed but offline: hand it straight back, or it stays
// invisible to every worker until this one happens to be
// the next to notice the reconnect.
if let Some(op) = idle {
self.release_claim(&op);
}
if primary {
// Nothing to upload is exactly when the connection is
// free, so this is where the read path's buffered LRU
// touches get written (`ContentCache::flush_touches`).
// Cheap and a no-op when empty.
self.cache.flush_touches();
self.recover_fsynced_writes();
}
// A debounced or backed-off op may be waiting: sleep only
// until it becomes due rather than the full idle-poll.
self.wait_for_drain_work_or_due();
continue;
}
};
let outcome = run_authorized_drain(
&op,
|uid| self.require_uid_writable(uid),
|uid| self.uid_write_authority(uid),
|| self.drain_op(&op),
);
if matches!(outcome, Ok(DrainDisposition::AccessDeferred)) {
let next_attempt_at = now_millis() + DRAIN_ACCESS_RECHECK.as_millis() as i64;
if let Err(error) = self.db.defer_op_without_attempt(op.id, next_attempt_at) {
error!(
uid = %op.uid,
%error,
"deferring access-blocked pending operation failed"
);
self.wait_for_drain_work();
} else {
debug!(
uid = %op.uid,
next_attempt_at,
"pending operation deferred until access is writable"
);
}
self.defer_for_access(&op, AccessDeferral::NotWritable);
continue;
}
if let Ok(DrainDisposition::AuthorityUnknown(ref authority)) = outcome {
self.resolve_unknown_authority(&op, authority);
continue;
}
// The access check let this attempt through, so any earlier run of
// deferrals is over and must not be counted against the next one.
if matches!(outcome, Ok(DrainDisposition::Applied))
&& let Err(error) = self.db.clear_op_access_deferral(op.id)
{
debug!(uid = %op.uid, %error, "clearing an access-deferral window failed");
}
if let Err(e) = outcome {
let attempts = op.attempts + 1;
let err_str = e.to_string();
@@ -188,9 +343,234 @@ impl Core {
now_millis() + backoff.as_millis() as i64,
) {
error!(uid = %op.uid, error = %e, "recording a drain failure failed");
self.release_claim(&op);
self.wait_for_drain_work();
continue;
}
}
// Unconditionally, on every path: a handler that retired its own row
// leaves nothing to release, and one that returned without retiring
// must not leave the row claimed by a worker that has moved on.
self.release_claim(&op);
}
}
/// Put an op the access check refused back on the clock, and decide whether
/// its refusal is still news.
///
/// Inside [`DRAIN_ACCESS_DEFER_LIMIT`] a deferral is what it always was: no
/// attempt consumed, no error recorded, retried in five seconds. Past it the
/// deferral is reported through [`Db::record_op_failure`] like any other
/// stuck op, which is what puts it in `attempts`, in `last_error`, and — at
/// [`FAILING_ATTEMPTS`] — in the `failing_ops` count and error text that
/// `pdfs status` and the GUI show. The row and its staged blob stay exactly
/// where they are either way; this changes only whether the user can see
/// them.
///
/// [`Db::record_op_failure`]: pdfs_core::db::Db::record_op_failure
/// [`FAILING_ATTEMPTS`]: pdfs_core::db::FAILING_ATTEMPTS
fn defer_for_access(&self, op: &PendingOp, reason: AccessDeferral) {
let now = now_millis();
let next_attempt_at = now + DRAIN_ACCESS_RECHECK.as_millis() as i64;
let since = match self.db.defer_op_for_access(op.id, now, next_attempt_at) {
Ok(since) => since,
Err(error) => {
error!(
uid = %op.uid,
%error,
"deferring access-blocked pending operation failed"
);
self.release_claim(op);
self.wait_for_drain_work();
return;
}
};
let blocked = Duration::from_millis(now.saturating_sub(since).max(0) as u64);
if since == now {
// The first deferral of a run, at `warn!`: a write the mount accepted
// is not going anywhere for now, and that is worth a line even when
// the next recheck clears it.
warn!(
uid = %op.uid,
kind = %op.kind,
reason = reason.log_reason(),
"pending operation deferred"
);
}
if blocked >= DRAIN_ACCESS_DEFER_LIMIT {
let attempts = op.attempts + 1;
let backoff = DRAIN_BACKOFF_MIN
.saturating_mul(1u32 << attempts.min(6))
.min(DRAIN_BACKOFF_MAX);
let error = format!(
"{} for {}s. The staged bytes are kept.",
reason.failure_reason(),
blocked.as_secs()
);
warn!(uid = %op.uid, attempts, blocked_secs = blocked.as_secs(),
"pending operation has been access-blocked past the limit; reporting it");
if let Err(error) =
self.db
.record_op_failure(op.id, &error, now_millis() + backoff.as_millis() as i64)
{
error!(uid = %op.uid, %error, "recording an access-deferral failure failed");
}
} else {
debug!(
uid = %op.uid,
next_attempt_at,
blocked_secs = blocked.as_secs(),
reason = reason.log_reason(),
"pending operation deferred until access is writable"
);
}
self.release_claim(op);
}
/// Answer an authority the local tree has no row for by asking the remote,
/// rather than deferring on a permission question nobody asked (B83).
///
/// Three outcomes, and the point of the whole exercise is that they are
/// three:
///
/// * the node is still there — the tree simply lost it (a pruned subtree, a
/// mount whose registration went away). Re-intern it and the next pass has
/// a real access answer to work with.
/// * the node is gone remotely. That is a permanent, *known* condition, so
/// it is recorded as a failure immediately instead of waiting out
/// [`DRAIN_ACCESS_DEFER_LIMIT`] to say so. The row and its staged blob are
/// kept, as on every other failure path — the bytes are still the user's,
/// and `pdfs recover` is how they get them back.
/// * we could not ask. Ordinary deferral; the network is the network.
fn resolve_unknown_authority(&self, op: &PendingOp, authority: &NodeUid) {
match self.fetch_node_remote(authority) {
Ok(Some(node)) => {
if let Err(error) = self.db.upsert_node(&node) {
error!(uid = %op.uid, %authority, %error,
"re-interning a queued operation's missing authority failed");
} else {
info!(uid = %op.uid, %authority,
"re-interned an authority missing from the local tree; retrying");
}
// Deferred, not applied: the re-read has to go through the same
// access check as everything else, on the next pass. The window
// keeps running, so a refetch that never actually helps still
// gets reported instead of looping in silence.
self.defer_for_access(op, AccessDeferral::AuthorityRefetched);
}
Ok(None) => {
let attempts = op.attempts + 1;
let backoff = DRAIN_BACKOFF_MIN
.saturating_mul(1u32 << attempts.min(6))
.min(DRAIN_BACKOFF_MAX);
let error = format!(
"{authority} no longer exists remotely, so this operation has nothing to \
apply to. The staged bytes are kept."
);
warn!(uid = %op.uid, %authority, attempts,
"a queued operation's authority is gone remotely; reporting it");
if let Err(error) = self.db.record_op_failure(
op.id,
&error,
now_millis() + backoff.as_millis() as i64,
) {
error!(uid = %op.uid, %error, "recording a missing-authority failure failed");
}
self.release_claim(op);
}
Err(error) => {
debug!(uid = %op.uid, %authority, %error,
"could not ask the remote about a missing authority");
self.defer_for_access(op, AccessDeferral::RemoteUnreachable);
}
}
}
/// Hand a claimed op back to the queue, logging rather than propagating —
/// there is no caller left to propagate to, and the drain must not stop.
fn release_claim(&self, op: &PendingOp) {
if let Err(error) = self.db.release_op_claim(op.id) {
error!(uid = %op.uid, %error, "releasing a drain claim failed");
}
}
/// How long a freshly queued revision of `uid` should wait before a worker
/// may pick it up.
///
/// [`DRAIN_REVISION_DEBOUNCE`] exists for a tool that preallocates a file
/// and then writes it, where two seconds is the right scale. It is the wrong
/// scale for a large file on a slow link: an editor saving a 200 MB project
/// every ten seconds queues a write, the drain starts sending it, the next
/// save supersedes it mid-flight, and the file never finishes uploading
/// while the user keeps working. Waiting roughly as long as the last upload
/// of *this* node took moves that supersede into the queue, where replacing
/// a row costs nothing.
///
/// Never shorter than the fixed debounce and never longer than
/// [`DRAIN_REVISION_DEBOUNCE_MAX`]: a node whose upload is genuinely slow
/// still has to reach Drive, and a single pathological stall must not park
/// it.
///
/// [`DRAIN_REVISION_DEBOUNCE`]: super::DRAIN_REVISION_DEBOUNCE
/// [`DRAIN_REVISION_DEBOUNCE_MAX`]: super::DRAIN_REVISION_DEBOUNCE_MAX
pub(crate) fn revision_debounce(&self, uid: &NodeUid) -> Duration {
let measured = self
.upload_times
.lock()
.get(&uid.to_string())
.map(|&(ms, _)| Duration::from_millis(ms));
adaptive_debounce(measured)
}
/// Remember how long an upload of `uid` took, for the next write's
/// debounce.
///
/// Only a *completed* upload is a measurement. A cancelled or failed one
/// says how long it ran before it stopped, which is not how long the file
/// takes to send, and feeding that back would let one network fault widen
/// the debounce for good.
fn record_upload_time(&self, uid: &NodeUid, took: Duration) {
let mut times = self.upload_times.lock();
let now = now_millis();
if times.len() >= UPLOAD_TIME_MEMORY
&& !times.contains_key(&uid.to_string())
&& let Some(oldest) = times
.iter()
.min_by_key(|(_, (_, at))| *at)
.map(|(k, _)| k.clone())
{
times.remove(&oldest);
}
times.insert(uid.to_string(), (took.as_millis() as u64, now));
}
/// Register a cancellation flag for an upload of `uid` that is about to
/// start, returning a guard that deregisters it.
///
/// Deregistration is by identity rather than by key: a supersede that
/// arrives just as this upload finishes may already have registered the
/// *next* upload's flag under the same uid, and removing that one would
/// leave the newer upload uncancellable.
fn begin_cancellable_upload(&self, uid: &NodeUid) -> UploadCancel {
let flag = Arc::new(AtomicBool::new(false));
self.upload_cancel.lock().insert(uid.clone(), flag.clone());
UploadCancel {
registry: self.upload_cancel.clone(),
uid: uid.clone(),
flag,
}
}
/// Tell an upload of `uid` that is on the wire to stop, if there is one.
///
/// Called when a newer revision of the file is queued. The upload does not
/// stop instantly — it stops when the SDK next asks its reader for bytes,
/// which for a 4 MiB block is soon enough to matter and late enough to be
/// safe: the staged blob it is reading stays open until it lets go.
pub(crate) fn cancel_upload(&self, uid: &NodeUid) {
if let Some(flag) = self.upload_cancel.lock().get(uid) {
flag.store(true, Ordering::Relaxed);
debug!(%uid, "cancelling a superseded upload");
}
}
@@ -907,15 +1287,37 @@ impl Core {
let guard =
self.transfers
.begin(&name, meta.uid.clone(), TransferDirection::Upload, meta.len);
let reader = CountingReader::new(File::open(&blob)?, &guard);
self.rt.block_on(self.client.upload_new_revision_from(
// Registered before the first byte moves, so a write that lands while
// this is on the wire can stop it rather than paying for a revision the
// op it queued will immediately replace.
let cancel = self.begin_cancellable_upload(&uid);
let reader = CountingReader::new(File::open(&blob)?, &guard).with_cancel(cancel.flag());
let started = Instant::now();
let sent = self.rt.block_on(self.client.upload_new_revision_from(
&uid,
reader,
meta.len as i64,
Vec::new(),
None,
))?;
));
let took = started.elapsed();
if let Err(e) = sent {
if cancel.cancelled() {
// Not a failure, and not this op's row to retire: the write that
// cancelled us deleted it and took ownership of the staged blob
// when it superseded this revision. Leaving the blob alone here
// is the whole point — it is either already unlinked or it
// belongs to the newer op.
info!(%uid, "upload abandoned; a newer write superseded it");
return Ok(());
}
return Err(e.into());
}
drop(cancel);
drop(guard);
// Only a completed upload times how long this file takes to send, which
// is what the next write's debounce is scaled from.
self.record_upload_time(&uid, took);
// Retire the op before dropping the blob: a crash between the two leaves
// an orphaned file (harmless), whereas the reverse would leave a queued
@@ -1227,12 +1629,10 @@ mod tests {
let applied = Cell::new(false);
let disposition = run_authorized_drain(
&op,
|uid| {
db.effective_node_access(uid)
.unwrap()
.filter(|access| access.writable())
.map(|_| ())
.ok_or(fuser::Errno::EACCES)
|uid| match db.effective_node_access(uid).unwrap() {
Some(access) if access.writable() => WriteAuthority::Writable,
Some(_) => WriteAuthority::Denied,
None => WriteAuthority::Unknown,
},
|| {
applied.set(true);
@@ -1272,7 +1672,7 @@ mod tests {
let disposition = run_authorized_drain(
&op,
|_| Err(fuser::Errno::EACCES),
|_| WriteAuthority::Denied,
|| {
applied.set(true);
Ok(())
@@ -1293,13 +1693,157 @@ mod tests {
let _ = std::fs::remove_dir_all(dir);
}
/// A deferral that never clears has to stop being silent. Found in the field
/// as revision ops whose node had left the local tree: `require_uid_access`
/// reads an unknown node as `EACCES`, so they were re-deferred every five
/// seconds for thirty days with `attempts` at zero, `last_error` null and
/// `failing_ops` at zero — 18 GiB of accepted writes that `pdfs status`
/// reported as ordinary queued uploads.
#[test]
fn an_access_deferral_that_outlives_the_limit_is_reported_as_failing() {
use std::sync::atomic::{AtomicU64, Ordering};
static N: AtomicU64 = AtomicU64::new(0);
let dir = std::env::temp_dir().join(format!(
"pdfs-drain-defer-limit-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed)
));
std::fs::create_dir_all(&dir).unwrap();
let db = Db::open(&dir.join("cache.db")).unwrap();
db.enqueue_op(&pending(OP_REVISION)).unwrap();
let op = db.pending_ops().unwrap().remove(0);
// Two deferrals a recheck apart: the window opens on the first and the
// second is measured against it, not against itself.
let now = 1_000_000i64;
let since = db.defer_op_for_access(op.id, now, now + 5_000).unwrap();
assert_eq!(since, now, "the first deferral opens the window");
let later = now + DRAIN_ACCESS_RECHECK.as_millis() as i64;
assert_eq!(
db.defer_op_for_access(op.id, later, later + 5_000).unwrap(),
now,
"a later deferral keeps the original stamp"
);
let queued = db.pending_ops().unwrap();
assert_eq!(queued[0].attempts, 0, "deferrals consume no attempt");
assert!(queued[0].last_error.is_none());
// Past the limit the same deferral is reported like any other stuck op.
let past = now + DRAIN_ACCESS_DEFER_LIMIT.as_millis() as i64;
assert_eq!(
db.defer_op_for_access(op.id, past, past + 5_000).unwrap(),
now
);
db.record_op_failure(op.id, "not writable locally", past + 10_000)
.unwrap();
let failed = db.pending_ops().unwrap();
assert_eq!(failed.len(), 1, "the row and its staged bytes stay queued");
assert_eq!(failed[0].attempts, 1);
assert_eq!(
failed[0].last_error.as_deref(),
Some("not writable locally")
);
// Reporting closes the window, so the next deferral starts a fresh one
// rather than escalating on every recheck from here on.
let after = past + 20_000;
assert_eq!(
db.defer_op_for_access(op.id, after, after + 5_000).unwrap(),
after,
"recording the failure re-armed the window"
);
drop(db);
let _ = std::fs::remove_dir_all(dir);
}
/// An op that gets through the access check is no longer blocked, so a
/// deferral after it must not inherit the earlier run's age and escalate
/// immediately.
#[test]
fn clearing_the_window_makes_the_next_deferral_a_fresh_run() {
use std::sync::atomic::{AtomicU64, Ordering};
static N: AtomicU64 = AtomicU64::new(0);
let dir = std::env::temp_dir().join(format!(
"pdfs-drain-defer-clear-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed)
));
std::fs::create_dir_all(&dir).unwrap();
let db = Db::open(&dir.join("cache.db")).unwrap();
db.enqueue_op(&pending(OP_REVISION)).unwrap();
let op = db.pending_ops().unwrap().remove(0);
let now = 2_000_000i64;
assert_eq!(
db.defer_op_for_access(op.id, now, now + 5_000).unwrap(),
now
);
db.clear_op_access_deferral(op.id).unwrap();
let later = now + DRAIN_ACCESS_DEFER_LIMIT.as_millis() as i64;
assert_eq!(
db.defer_op_for_access(op.id, later, later + 5_000).unwrap(),
later,
"the window reopens at the new deferral, not the old one"
);
drop(db);
let _ = std::fs::remove_dir_all(dir);
}
#[test]
fn an_authority_the_tree_does_not_have_is_reported_as_unknown_not_deferred() {
// The B83 distinction, at the point it is made: a uid with no row must
// not come back as `AccessDeferred`, because deferring waits on a
// permission change that nothing is going to make. The caller answers
// an `AuthorityUnknown` by asking the remote instead.
let op = pending(OP_REVISION);
let applied = Cell::new(false);
let disposition = run_authorized_drain(
&op,
|_| WriteAuthority::Unknown,
|| {
applied.set(true);
Ok(())
},
)
.unwrap();
assert_eq!(
disposition,
DrainDisposition::AuthorityUnknown(parse_node_uid(&op.uid).unwrap())
);
assert!(!applied.get(), "an unresolved authority never applies");
}
#[test]
fn a_missing_parent_authority_names_the_parent_not_the_node() {
// A create's authority is its parent, so that is the uid the remote has
// to be asked about — reporting the node instead would refetch the
// thing that does not exist yet.
let mut op = pending(OP_CREATE);
let parent = folder_node("vanished-parent", None);
op.parent_uid = Some(parent.uid.to_string());
let disposition =
run_authorized_drain(&op, |_| WriteAuthority::Unknown, || Ok(())).unwrap();
assert_eq!(
disposition,
DrainDisposition::AuthorityUnknown(parent.uid.clone())
);
}
#[test]
fn owner_and_editor_drain_authorities_apply_normally() {
for access in [Access::Owner, Access::Editor] {
let applied = Cell::new(false);
let disposition = run_authorized_drain(
&pending(OP_REVISION),
|_| access.writable().then_some(()).ok_or(fuser::Errno::EACCES),
|_| {
if access.writable() {
WriteAuthority::Writable
} else {
WriteAuthority::Denied
}
},
|| {
applied.set(true);
Ok(())
@@ -1413,3 +1957,42 @@ mod tests {
assert!(!is_own_self_supersede(true, None, None));
}
}
#[cfg(test)]
mod debounce_tests {
use super::*;
#[test]
fn a_node_never_uploaded_keeps_the_fixed_grace_period() {
assert_eq!(adaptive_debounce(None), DRAIN_REVISION_DEBOUNCE);
}
#[test]
fn a_fast_upload_does_not_shorten_the_grace_period() {
// The 2 seconds are not about upload time at all — they are the window
// in which a preallocate-then-write tool supersedes its own first
// close. A file that sends in 50 ms must still get them.
assert_eq!(
adaptive_debounce(Some(Duration::from_millis(50))),
DRAIN_REVISION_DEBOUNCE
);
}
#[test]
fn a_slow_upload_widens_the_grace_period_to_match() {
// The case the fixed debounce got wrong: saves arriving faster than the
// upload completes, each superseding one already on the wire.
assert_eq!(
adaptive_debounce(Some(Duration::from_secs(25))),
Duration::from_secs(25)
);
}
#[test]
fn a_pathological_upload_cannot_park_a_node() {
assert_eq!(
adaptive_debounce(Some(Duration::from_secs(6000))),
DRAIN_REVISION_DEBOUNCE_MAX
);
}
}
+211 -81
View File
@@ -1,6 +1,7 @@
//! Kernel-facing FUSE callback adapter.
use super::*;
use crate::state::DirListing;
impl Filesystem for ProtonFs {
/// Negotiate kernel-side behaviour once, before any other callback.
@@ -162,15 +163,46 @@ impl Filesystem for ProtonFs {
}
}
/// Freeze this directory's listing for the life of the handle.
///
/// `readdir` is paged, and its continuation offset only means anything
/// against a listing that does not move underneath it. Taking the snapshot
/// here — once per `opendir`, which the kernel pairs exactly with one
/// `releasedir` — is what makes the offsets stable, so a create or a trash
/// during an enumeration can no longer make the caller skip or repeat an
/// entry (bugs.md B43).
///
/// Enumerating a cold folder is a remote call, so it goes to a worker for
/// the same reason `lookup` and `readdir` do.
fn opendir(&self, _req: &Request, ino: INodeNo, _flags: OpenFlags, reply: ReplyOpen) {
let ino = ino.0;
if self.core.children_cached(ino) {
self.serve_opendir(ino, reply);
return;
}
let fs = self.clone();
self.core
.workers
.run(Lane::Meta, move || fs.serve_opendir(ino, reply));
}
fn readdir(
&self,
_req: &Request,
ino: INodeNo,
_fh: FileHandle,
fh: FileHandle,
offset: u64,
reply: ReplyDirectory,
) {
let ino = ino.0;
// The snapshot is the whole point: served from it, this is a pure
// in-memory walk with no chance of a remote call, so it never needs a
// worker. Only a handle whose `opendir` failed to snapshot (or a kernel
// that issued `readdir` without one) falls back to the live path.
if let Some(listing) = self.core.state().dir_snapshots.get(&fh.0).cloned() {
reply_listing(&listing, offset, reply);
return;
}
// Same split as `lookup`: only the cold, remote-enumerating path pays
// for a worker.
if self.core.children_cached(ino) {
@@ -183,6 +215,18 @@ impl Filesystem for ProtonFs {
.run(Lane::Meta, move || fs.serve_readdir(ino, offset, reply));
}
fn releasedir(
&self,
_req: &Request,
_ino: INodeNo,
fh: FileHandle,
_flags: OpenFlags,
reply: ReplyEmpty,
) {
self.core.state().dir_snapshots.remove(&fh.0);
reply.ok();
}
/// A read-only open is a map lookup and answers inline; a write open has to
/// allocate a scratch file and, when a queued revision is the base, duplicate
/// a blob that is the whole file — so it is served from a worker rather than
@@ -912,6 +956,58 @@ pub struct ProtonFs {
gid: u32,
}
/// How many times a write `open` will re-read its base after a `release`
/// published a newer queued revision underneath it (bugs.md B32).
///
/// One retry covers the race, which needs a close and an open of the same file
/// to overlap on different threads. More than that is not contention, it is a
/// file being republished continuously, and looping on it would keep a FUSE
/// worker busy indefinitely.
const OPEN_BASE_ATTEMPTS: u32 = 3;
/// One directory's entries as `readdir` reports them: `.` and `..` first, then
/// the interned children in `children` order.
///
/// Split out because it is built from two places that must agree — `opendir`
/// freezes it, and the fallback `readdir` builds it live — and a divergence
/// between them would show up as entries appearing or vanishing depending on
/// whether the kernel happened to send an `opendir`.
fn build_listing(st: &State, ino: u64) -> DirListing {
let parent = st.entries.get(&ino).map_or(ROOT_INO, |e| e.parent);
// "." and ".." occupy offsets 0 and 1; real children follow.
let mut listing = vec![
(ino, true, ".".to_string()),
(parent, true, "..".to_string()),
];
if let Some(kids) = st.children.get(&ino) {
for &kid in kids {
if let Some(e) = st.entries.get(&kid) {
listing.push((kid, e.node.is_folder(), e.node.name.clone()));
}
}
}
listing
}
/// Fill a `readdir` reply from `listing`, resuming at `offset`.
///
/// The cookie handed back is the index of the *next* entry, so a resumed call
/// continues exactly where the last one stopped — which is only true because
/// the listing behind those indexes is a snapshot (bugs.md B43).
fn reply_listing(listing: &DirListing, offset: u64, mut reply: ReplyDirectory) {
for (i, (ino, is_dir, name)) in listing.iter().enumerate().skip(offset as usize) {
let ft = if *is_dir {
FileType::Directory
} else {
FileType::RegularFile
};
if reply.add(INodeNo(*ino), (i + 1) as u64, ft, name) {
break;
}
}
reply.ok();
}
pub(crate) fn access_allowed(is_dir: bool, access: Access, mask: AccessFlags) -> bool {
(!mask.contains(AccessFlags::W_OK) || access.writable())
&& (!mask.contains(AccessFlags::X_OK) || is_dir)
@@ -1027,40 +1123,39 @@ impl ProtonFs {
}
/// The body of [`Filesystem::readdir`], on whichever thread ends up serving it.
fn serve_readdir(&self, ino: u64, offset: u64, mut reply: ReplyDirectory) {
/// The body of [`Filesystem::opendir`]: enumerate the folder if it is cold,
/// then publish one frozen listing under a fresh handle.
///
/// A failure to enumerate is reported as a failed `opendir` rather than
/// deferred to the first `readdir` — the caller gets the errno at the syscall
/// that can still act on it.
fn serve_opendir(&self, ino: u64, reply: ReplyOpen) {
if let Err(e) = self.core.ensure_children(ino) {
reply.error(e);
return;
}
let mut st = self.core.state();
let listing = Arc::new(build_listing(&st, ino));
// Shares the counter with write handles, so a directory handle can never
// collide with a file handle (both are `fh` values in one kernel space).
let fh = st.next_fh;
st.next_fh += 1;
st.dir_snapshots.insert(fh, listing);
drop(st);
reply.opened(FileHandle(fh), FopenFlags::empty());
}
/// The live, un-snapshotted `readdir`. Reached only by a handle that has no
/// snapshot — see [`Filesystem::readdir`].
fn serve_readdir(&self, ino: u64, offset: u64, reply: ReplyDirectory) {
if let Err(e) = self.core.ensure_children(ino) {
reply.error(e);
return;
}
let st = self.core.state();
let parent = st.entries.get(&ino).map_or(ROOT_INO, |e| e.parent);
// "." and ".." occupy offsets 0 and 1; real children follow.
let mut listing: Vec<(u64, FileType, String)> = vec![
(ino, FileType::Directory, ".".to_string()),
(parent, FileType::Directory, "..".to_string()),
];
if let Some(kids) = st.children.get(&ino) {
for &kid in kids {
if let Some(e) = st.entries.get(&kid) {
let ft = if e.node.is_folder() {
FileType::Directory
} else {
FileType::RegularFile
};
listing.push((kid, ft, e.node.name.clone()));
}
}
}
let listing = build_listing(&st, ino);
drop(st);
for (i, (ino, ft, name)) in listing.into_iter().enumerate().skip(offset as usize) {
// The stored offset is that of the *next* entry to resume at.
if reply.add(INodeNo(ino), (i + 1) as u64, ft, &name) {
break;
}
}
reply.ok();
reply_listing(&listing, offset, reply);
}
fn attr(&self, ino: u64, node: &Node, access: Access) -> FileAttr {
@@ -1142,66 +1237,101 @@ impl ProtonFs {
// any network access. An incomplete blob still has gaps referring to
// the remote base; stacking another write on it cannot be represented
// safely by WriteHandle, so fail the open rather than risk corruption.
let pending_base = self.core.pending.lock().get(&uid).cloned();
if pending_base.as_ref().is_some_and(|p| !p.meta.complete) {
if let Some(entry) = self.core.state().entries.get_mut(&ino.0) {
entry.open_count = entry.open_count.saturating_sub(1);
}
error!(%uid, "refusing write over incomplete queued revision");
reply.error(Errno::EIO);
return;
}
let (file, path) = match self.core.cache.create_scratch() {
Ok(x) => x,
Err(e) => {
if let Some(entry) = self.core.state().entries.get_mut(&ino.0) {
entry.open_count = entry.open_count.saturating_sub(1);
//
// Reading that blob and installing the handle cannot be done under one
// lock — the copy is the size of the file — so the base is re-checked at
// the moment the handle goes in, and a `release` that published a newer
// revision in between sends this around again. Without the re-check the
// new handle carried the *older* base, and its own release published
// content that silently discarded the revision closed a moment earlier
// (bugs.md B32). A second attempt is already unlikely and a third would
// mean something is republishing continuously, so give up rather than
// spin: the caller's `open` fails instead of its data being lost.
//
// `reply` is consumed by whichever arm answers, so the loop yields an
// outcome and the answer is sent once, after it.
let outcome = 'install: {
for attempt in 0..OPEN_BASE_ATTEMPTS {
let pending_base = self.core.pending.lock().get(&uid).cloned();
if pending_base.as_ref().is_some_and(|p| !p.meta.complete) {
error!(%uid, "refusing write over incomplete queued revision");
break 'install Err(Errno::EIO);
}
error!(%uid, error = %e, "create scratch file failed");
reply.error(Errno::EIO);
return;
let (file, path) = match self.core.cache.create_scratch() {
Ok(x) => x,
Err(e) => {
error!(%uid, error = %e, "create scratch file failed");
break 'install Err(Errno::EIO);
}
};
let mut initial_written = Intervals::default();
if let Some(pending) = &pending_base {
// Reflink where the filesystem can: this blob is the whole
// file, and a byte-for-byte copy of a multi-GB pending
// revision is the kind of work `open` must not do
// (`ContentCache::clone_or_copy`).
if let Err(e) = ContentCache::clone_or_copy(&pending.path, &path) {
let _ = std::fs::remove_file(&path);
error!(%uid, source = %pending.path.display(), error = %e,
"copy queued revision into write scratch failed");
break 'install Err(Errno::EIO);
}
initial_written.add(0, pending.meta.len);
}
// Sampled *before* the state lock, never under it: no site in
// the daemon holds `pending` and `state` at once (see
// `stamp_pending_sizes`) and this must not become the first.
// What that costs is a window of a few instructions instead of
// a lock-tight comparison — against the multi-gigabyte copy
// above, which is the window that actually loses data.
let current = self.core.pending_blob(&uid);
let mut st = self.core.state();
// Only a handle this open is about to *create* can carry a stale
// base; if one already exists, that handle is authoritative and
// the scratch prepared above is discarded either way.
let creating = !st.active_writes.contains_key(&ino.0);
if creating && current != pending_base.map(|p| p.path) {
drop(st);
let _ = std::fs::remove_file(&path);
warn!(%uid, attempt, "queued revision changed while opening for write; retrying");
continue;
}
let fh = st.next_fh;
st.next_fh += 1;
let aw = st.active_writes.entry(ino.0).or_insert_with(|| {
WriteHandle {
ino: ino.0,
uid: uid.clone(),
file: Arc::new(file),
path,
written: initial_written,
// Starts at the current size; reads in [0, base_size)
// come from the base until overwritten.
len: base_size,
base_size,
base_mtime,
base_revision_id,
dirty: false,
open_count: 0,
}
});
aw.open_count += 1;
st.handles.insert(fh, ino.0);
break 'install Ok(fh);
}
error!(%uid, "queued revision kept changing while opening for write");
Err(Errno::EIO)
};
let mut initial_written = Intervals::default();
if let Some(pending) = &pending_base {
// Reflink where the filesystem can: this blob is the whole file, and
// a byte-for-byte copy of a multi-GB pending revision is the kind of
// work `open` must not do (`ContentCache::clone_or_copy`).
if let Err(e) = ContentCache::clone_or_copy(&pending.path, &path) {
match outcome {
Ok(fh) => reply.opened(FileHandle(fh), FopenFlags::empty()),
Err(error) => {
// The lookup count was raised before any of this could fail.
if let Some(entry) = self.core.state().entries.get_mut(&ino.0) {
entry.open_count = entry.open_count.saturating_sub(1);
}
let _ = std::fs::remove_file(&path);
error!(%uid, source = %pending.path.display(), error = %e,
"copy queued revision into write scratch failed");
reply.error(Errno::EIO);
return;
reply.error(error);
}
initial_written.add(0, pending.meta.len);
}
let mut st = self.core.state();
let fh = st.next_fh;
st.next_fh += 1;
let aw = st.active_writes.entry(ino.0).or_insert_with(|| {
WriteHandle {
ino: ino.0,
uid,
file: Arc::new(file),
path,
written: initial_written,
// Starts at the current size; reads in [0, base_size) come from
// the base until overwritten.
len: base_size,
base_size,
base_mtime,
base_revision_id,
dirty: false,
open_count: 0,
}
});
aw.open_count += 1;
st.handles.insert(fh, ino.0);
reply.opened(FileHandle(fh), FopenFlags::empty());
}
/// The body of [`Filesystem::fsync`], run off the dispatch loop.
+452 -41
View File
@@ -55,12 +55,12 @@ use fuser::{
};
use futures::StreamExt as _;
use pdfs_core::batch;
use pdfs_core::cache::{BLOCK_SIZE, Baseline, ContentCache, StagedWrite};
use pdfs_core::cache::{BLOCK_SIZE, Baseline, BlockGeometry, BlockSpan, ContentCache, StagedWrite};
use pdfs_core::config::{AppDirs, SweepMode};
use pdfs_core::control::{
ActivityEntry, ActivityKind, DirEntry, ErrorKind, LocalHit, PhotoKind, PublicLinkInfo,
SearchFilters, SearchHit, SearchSource, SyncFolderInfo, SyncPhase, SyncProgress,
TransferDirection,
ThumbnailBuildStatus, TransferDirection,
};
use pdfs_core::db::{
Db, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, PARK_UNTIL, PendingOp,
@@ -92,9 +92,11 @@ mod profile;
mod reads;
mod revisions;
mod sharing;
mod shutdown;
mod state;
mod sweep;
mod sync;
mod takeout;
mod transfers;
mod upload;
mod r#virtual;
@@ -166,6 +168,30 @@ const DRAIN_IDLE_POLL: Duration = Duration::from_secs(30);
/// creates a conflict copy of itself. Holding the op for a short window gives
/// the follow-up write time to supersede it.
const DRAIN_REVISION_DEBOUNCE: Duration = Duration::from_secs(2);
/// Ceiling on the *adaptive* part of that grace period.
///
/// The fixed 2 seconds is enough for a preallocate-then-write tool, and wrong
/// for a file that takes half a minute to upload: an editor saving every ten
/// seconds queues a write, watches it start uploading, and supersedes it — each
/// save uploading bytes that the next one replaces, and never catching up.
/// Widening the debounce toward how long this node's last upload actually took
/// makes the supersede happen in the queue, where it is free, instead of on the
/// wire. Bounded so a file on a slow link still reaches Drive in a minute rather
/// than deferring indefinitely, and so a one-off stall cannot park a node.
const DRAIN_REVISION_DEBOUNCE_MAX: Duration = Duration::from_secs(60);
/// How many nodes' measured upload times to remember for that. Bounds a map
/// that would otherwise grow with every file ever written; the debounce falls
/// back to [`DRAIN_REVISION_DEBOUNCE`] for anything evicted, which is the
/// behaviour a node that has never been uploaded gets anyway.
const UPLOAD_TIME_MEMORY: usize = 512;
/// Threads draining the pending-op queue.
///
/// One meant a single 10 GiB upload blocked every queued rename, trash and
/// small write behind it. Ordering only has to hold per node — enforced by the
/// claim query, not by the thread count — so the rest is throughput. Kept small:
/// these are uploads, and past a handful they compete for the same uplink and
/// for the SDK's own per-request concurrency.
const DRAIN_WORKERS: usize = 3;
const ONLINE_PROBE_MIN: Duration = Duration::from_secs(5);
const ONLINE_PROBE_MAX: Duration = Duration::from_secs(300);
@@ -348,6 +374,45 @@ impl SearchRoots {
}
}
/// Negative thumbnail knowledge has two different authorities. A remote miss
/// only says Proton has no thumbnail to download; it must never suppress local
/// generation. A local miss says this exact file revision was successfully
/// inspected and could not be decoded. Both maps are bounded independently.
#[derive(Default)]
struct ThumbnailMissCaches {
remote: HashMap<(NodeUid, i32), i64>,
local: HashMap<(NodeUid, i32), i64>,
}
impl ThumbnailMissCaches {
fn remote_contains(&self, key: &(NodeUid, i32), tag: i64) -> bool {
self.remote.get(key) == Some(&tag)
}
fn local_contains(&self, key: &(NodeUid, i32), tag: i64) -> bool {
self.local.get(key) == Some(&tag)
}
fn remember_remote(&mut self, key: (NodeUid, i32), tag: i64) {
Self::remember(&mut self.remote, key, tag);
}
fn remember_local(&mut self, key: (NodeUid, i32), tag: i64) {
Self::remember(&mut self.local, key, tag);
}
fn forget_local(&mut self, key: &(NodeUid, i32)) {
self.local.remove(key);
}
fn remember(map: &mut HashMap<(NodeUid, i32), i64>, key: (NodeUid, i32), tag: i64) {
if map.len() >= MAX_THUMBNAIL_MISSES && !map.contains_key(&key) {
map.clear();
}
map.insert(key, tag);
}
}
/// `path` expressed relative to `root`, or `None` if it is not below it. The
/// root itself resolves to the empty path; an empty root is the mount itself
/// and covers everything.
@@ -386,6 +451,10 @@ struct Core {
/// kernel read-ahead, our own prefetch) share a single download and decrypt.
/// See [`BlockFlight`].
block_flight: Arc<BlockFlight>,
/// One lock per file being repaired by [`Core::repair_block`], so the
/// concurrent block reads that all discover the same untrustworthy block
/// table share a single whole-file download instead of one each.
content_repairs: Arc<tokio::sync::Mutex<HashMap<NodeUid, Arc<tokio::sync::Mutex<()>>>>>,
/// Per-file sequential-read detection driving [`Core::prefetch`].
prefetch: Arc<Prefetch>,
/// Permits bounding speculative block fetches, so prefetch cannot queue in
@@ -423,9 +492,26 @@ struct Core {
/// Nodes removed through this daemon which a briefly stale remote listing
/// may still return. Uids are not reused; an explicit restore clears one.
hidden: Arc<Mutex<HashSet<NodeUid>>>,
/// Nudges the drain worker: set true and notify to have it re-examine the
/// queue instead of waiting out its backoff.
/// Nudges the drain workers: set true and notify to have them re-examine
/// the queue instead of waiting out their backoff.
drain_wake: Arc<(Mutex<bool>, Condvar)>,
/// The daemon's stop signal, shared with every background loop this `Core`
/// was cloned into so teardown can end and join them (bugs.md B44).
shutdown: Arc<shutdown::Shutdown>,
/// How long this node's last upload actually took, in milliseconds, with
/// the time it was measured for eviction. Feeds
/// [`Core::revision_debounce`]: a file that takes 30 seconds to send needs
/// a grace period on that scale, or every save supersedes an upload that
/// was already on the wire. Bounded by [`UPLOAD_TIME_MEMORY`].
upload_times: Arc<Mutex<HashMap<String, (u64, i64)>>>,
/// Cancellation flags for uploads currently on the wire, keyed by node.
///
/// Set when a newer revision of the same file is queued: the bytes in
/// flight have just been superseded, and the only thing finishing them
/// achieves is spending the user's uplink on a revision the next op
/// replaces. Read by the upload's own reader ([`CountingReader`]), which is
/// the one thing the SDK calls often enough to notice.
upload_cancel: Arc<Mutex<HashMap<NodeUid, Arc<AtomicBool>>>>,
/// True while a background refresh of the photos timeline (resp. the trash) is
/// already running, so a burst of page requests against a stale listing kicks
/// off one refresh rather than one per request.
@@ -464,8 +550,22 @@ struct Core {
/// those downloads is a full-size photo — so an in-flight uid is never started
/// twice.
thumb_gen: Arc<Mutex<HashSet<NodeUid>>>,
/// Nodes the remote has told us have *no* thumbnail of a given type, keyed by
/// `(uid, thumbnail type)` and holding the mtime the answer was learned at.
/// Global permit pool for locally generated thumbnails. The old per-request
/// limit multiplied when the GUI sent several batches, allowing dozens of
/// full-size images to download at once.
thumb_gen_budget: Arc<tokio::sync::Semaphore>,
/// Current ordinary-file listing generation and a wake-up for tasks waiting
/// on a permit or a download when that listing is abandoned.
file_thumb_generation: Arc<AtomicU64>,
file_thumb_cancel: Arc<tokio::sync::Notify>,
/// Progress of the explicit recursive build started from the Files toolbar.
thumbnail_build: Arc<Mutex<ThumbnailBuildStatus>>,
/// Cancellation for the explicit recursive build. Separate from ordinary
/// listing cancellation because the two jobs have independent lifetimes.
thumbnail_build_cancelled: Arc<AtomicBool>,
thumbnail_build_cancel: Arc<tokio::sync::Notify>,
/// Negative thumbnail knowledge, separated by whether the remote lacked a
/// thumbnail or local decoding proved this revision unsupported.
///
/// Absence has to be cached or it costs a round trip every time it is asked
/// for, and it is asked for constantly: an `ls -l` from an xattr-aware lister
@@ -473,7 +573,7 @@ struct Core {
/// of videos re-probed 130 times per listing at ~186 ms each (B5). The mtime
/// is the validity tag — a new revision may well have a thumbnail — matching
/// how [`ContentCache::read_thumbnail`] validates the positive side.
no_thumbnail: Arc<Mutex<HashMap<(NodeUid, i32), i64>>>,
thumbnail_misses: Arc<Mutex<ThumbnailMissCaches>>,
/// Last account quota answer and when it was learned, for `statfs(2)`.
///
/// `df` and the free-space preflight of every file manager and installer ask
@@ -669,6 +769,26 @@ struct MountedState {
/// [`Core::notifier`], still empty until its session is spawned.
notifier: Arc<OnceLock<Notifier>>,
session_live: Arc<AtomicBool>,
/// This mount's in-flight size upgrades, keyed by *its* folder inodes — so
/// a `Core` rebuilt onto this inode space by [`Core::rooted_at`] shares the
/// batch the mount's own session is already running rather than starting a
/// second one against inode numbers that mean nothing to it.
size_upgrades: Arc<Mutex<HashMap<u64, Arc<SizeUpgrade>>>>,
}
/// The per-mount half of a [`Core`] — every field [`Core::fork_state`] gives a
/// fork a fresh copy of — recovered from the registry.
///
/// This is what lets a control request naming a path under a secondary mount be
/// answered *in that mount's inode space* instead of being rejected for not
/// being under the primary mountpoint (`docs/BUGS.md` B86). See
/// [`Core::rooted_at`].
struct MountParts {
mountpoint: PathBuf,
state: Arc<Mutex<State>>,
notifier: Arc<OnceLock<Notifier>>,
session_live: Arc<AtomicBool>,
size_upgrades: Arc<Mutex<HashMap<u64, Arc<SizeUpgrade>>>>,
}
/// Every mounted inode space in the daemon, shared by the primary `Core` and
@@ -691,6 +811,7 @@ impl StateRegistry {
state: &Arc<Mutex<State>>,
notifier: Arc<OnceLock<Notifier>>,
session_live: Arc<AtomicBool>,
size_upgrades: Arc<Mutex<HashMap<u64, Arc<SizeUpgrade>>>>,
) {
let mut states = self.0.lock();
states.retain(|m| m.state.strong_count() > 0);
@@ -705,6 +826,7 @@ impl StateRegistry {
state: Arc::downgrade(state),
notifier,
session_live,
size_upgrades,
});
}
@@ -731,9 +853,9 @@ impl StateRegistry {
///
/// On-demand roots can be nested below another location. Selecting by the
/// longest component prefix ensures the nested session wins rather than the
/// broader primary mount. Callers currently need the selected mount only;
/// a future path router can derive a relative suffix with
/// `path.strip_prefix(mountpoint)` without storing a second path value here.
/// broader primary mount. Callers that need the relative suffix derive it
/// with `path.strip_prefix(mountpoint)`; [`StateRegistry::covering_parts`]
/// is the variant that also hands back enough to *serve* a request there.
fn covering(&self, path: &Path) -> Option<LiveMount> {
self.live()
.into_iter()
@@ -743,6 +865,48 @@ impl StateRegistry {
.max_by_key(|(mountpoint, _, _, _)| mountpoint.components().count())
}
/// [`StateRegistry::register`] with an empty size-upgrade map, for the
/// tests that exercise the reap-and-route rules and nothing else.
#[cfg(test)]
fn register_bare(
&self,
mountpoint: &Path,
state: &Arc<Mutex<State>>,
notifier: Arc<OnceLock<Notifier>>,
session_live: Arc<AtomicBool>,
) {
self.register(
mountpoint,
state,
notifier,
session_live,
Arc::new(Mutex::new(HashMap::new())),
);
}
/// [`StateRegistry::covering`], but returning every per-mount field a
/// [`Core`] needs to be re-rooted onto that inode space.
///
/// Kept separate from `covering` so the common callers — which only ask
/// *whether* a path is mounted — keep the cheaper tuple.
fn covering_parts(&self, path: &Path) -> Option<MountParts> {
let mut states = self.0.lock();
states.retain(|m| m.state.strong_count() > 0);
states
.iter()
.filter(|m| m.session_live.load(Ordering::Acquire) && path.starts_with(&m.mountpoint))
.max_by_key(|m| m.mountpoint.components().count())
.and_then(|m| {
Some(MountParts {
mountpoint: m.mountpoint.clone(),
state: m.state.upgrade()?,
notifier: m.notifier.clone(),
session_live: m.session_live.clone(),
size_upgrades: m.size_upgrades.clone(),
})
})
}
fn is_mounted_at(&self, path: &Path) -> bool {
self.covering(path).is_some_and(|(mountpoint, _, _, live)| {
mountpoint == path && live.load(Ordering::Acquire)
@@ -862,9 +1026,41 @@ impl Core {
&self.state,
self.notifier.clone(),
self.session_live.clone(),
self.size_upgrades.clone(),
);
}
/// This `Core`, re-rooted onto whichever live mount most specifically covers
/// the absolute path `abs`, together with `abs` relative to that mount's
/// root.
///
/// Everything a `Core` holds is per-daemon except the five fields
/// [`Core::fork_state`] replaces, so swapping exactly those turns the
/// primary `Core` into the fork that owns the path — sharing the fork's
/// inode space, notification channel and size-upgrade batches rather than
/// shadowing them. Without this, every path-addressed control request is
/// answered against the primary inode space, so anything under a secondary
/// on-demand mount is rejected as "not under the mountpoint" and the user
/// has no way to re-enumerate a folder they can see (`docs/BUGS.md` B86).
///
/// `None` when no live mount covers `abs`.
fn rooted_at(&self, abs: &Path) -> Option<(Core, PathBuf)> {
let parts = self.states.covering_parts(abs)?;
let rel = abs.strip_prefix(&parts.mountpoint).ok()?.to_path_buf();
let mut core = self.clone();
if !Arc::ptr_eq(&parts.state, &self.state) {
// A different inode space, so adopt its per-mount half wholesale.
// `primary` is a property of the mount, and the only mount this
// `Core` can be the primary of is the one it was built for.
core.primary = false;
core.state = parts.state;
core.notifier = parts.notifier;
core.session_live = parts.session_live;
core.size_upgrades = parts.size_upgrades;
}
Some((core, rel))
}
/// Run `apply` against every live inode space — this daemon's primary mount
/// and each on-demand fork — one lock at a time.
///
@@ -935,6 +1131,27 @@ impl Core {
/// downgrade cannot be hidden by stale live state, and a live downgrade
/// cannot be hidden by a restored DB row. Missing UIDs fail closed.
fn require_uid_writable(&self, uid: &NodeUid) -> Result<(), Errno> {
match self.uid_write_authority(uid) {
WriteAuthority::Writable => Ok(()),
// A uid nobody can speak for is refused here just as a denied one
// is: a syscall is a live request against a live tree, and a stale
// handle must not be admitted because the tree forgot the node.
// The drain, which replays intent recorded long ago, is the one
// caller that has to tell the two apart — see
// [`Core::uid_write_authority`].
WriteAuthority::Denied | WriteAuthority::Unknown => Err(Errno::EACCES),
}
}
/// The full answer behind [`Core::require_uid_writable`]: whether every
/// authority agrees this node is writable, refuses it, or has never heard
/// of it.
///
/// The third case is the one worth naming. A node absent from the local
/// tree yields no access row, and collapsing that into "denied" told the
/// drain to wait for a permission change that was never coming, because
/// nothing about the node's permissions was ever the problem (B83).
pub(crate) fn uid_write_authority(&self, uid: &NodeUid) -> WriteAuthority {
let mut live_access = Vec::new();
{
let state = self.state();
@@ -950,22 +1167,38 @@ impl Core {
live_access.push(access);
}
}
require_uid_access(&self.db, uid, &live_access)
uid_write_authority(&self.db, uid, &live_access)
}
}
fn require_uid_access(db: &Db, uid: &NodeUid, live_access: &[Access]) -> Result<(), Errno> {
/// Whether a node may be written, as agreed by every authority that has an
/// opinion — or the fact that none of them does.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum WriteAuthority {
/// The persisted tree and every resident mount agree: writable.
Writable,
/// At least one authority refuses it, or could not be consulted. Fails
/// closed, so a lookup error lands here.
Denied,
/// The node is not in the local tree, so there is no authority to consult.
/// Not the same as a refusal, and callers that can ask the remote should.
Unknown,
}
fn uid_write_authority(db: &Db, uid: &NodeUid, live_access: &[Access]) -> WriteAuthority {
let persisted = match db.effective_node_access(uid) {
Ok(Some(access)) => access,
Ok(None) => return Err(Errno::EACCES),
Ok(None) => return WriteAuthority::Unknown,
Err(error) => {
error!(%uid, %error, "db effective access lookup failed");
return Err(Errno::EACCES);
return WriteAuthority::Denied;
}
};
(persisted.writable() && live_access.iter().all(|access| access.writable()))
.then_some(())
.ok_or(Errno::EACCES)
if persisted.writable() && live_access.iter().all(|access| access.writable()) {
WriteAuthority::Writable
} else {
WriteAuthority::Denied
}
}
/// One kernel notification, recorded while the State lock is held so it can be
@@ -1633,7 +1866,9 @@ impl Core {
fn run_online_probe(&self) {
let mut delay = ONLINE_PROBE_MIN;
loop {
std::thread::sleep(delay);
if !self.shutdown.sleep(delay) {
return;
}
match self.rt.block_on(self.client.get_my_files_folder()) {
Ok(root) => {
{
@@ -2756,6 +2991,12 @@ impl Core {
}
})?;
let meta_json = serde_json::to_string(&meta).unwrap_or_default();
// Whatever this write supersedes, it supersedes now — including an
// upload already on the wire, whose bytes describe a revision the row
// written just below replaces. Signalled before the enqueue so the
// in-flight reader stops at its next block rather than after however
// much of the file is left.
self.cancel_upload(uid);
let superseded = if is_local_uid(uid) {
// The node has no server-side identity to hang a revision on, so the
// bytes ride on the create that will mint it.
@@ -2789,7 +3030,7 @@ impl Core {
created_at: now_millis(),
attempts: 0,
last_error: None,
next_attempt_at: now_millis() + DRAIN_REVISION_DEBOUNCE.as_millis() as i64,
next_attempt_at: now_millis() + self.revision_debounce(uid).as_millis() as i64,
};
let (_id, superseded) = self.db.enqueue_op(&op).map_err(|e| {
error!(%uid, error = %e, "queueing upload failed");
@@ -3095,6 +3336,12 @@ impl Core {
/// worse in every way.
fn queue_trash(&self, uid: &NodeUid, name: &str) -> Result<(), Errno> {
self.require_uid_writable(uid)?;
// The bytes below are about to be discarded, and a drain worker may be
// reading them onto the wire right now. Stop it first: with several
// workers the trash op this queues can be claimed while that upload is
// still running, and the two would race to decide whether the file
// exists.
self.cancel_upload(uid);
let (_, blobs) = self
.db
.replace_ops_with_trash(&uid.to_string(), name, now_millis())
@@ -3124,6 +3371,9 @@ impl Core {
/// Drop every op queued against a node, and the staged bytes they own.
fn discard_queued_ops(&self, uid: &NodeUid) -> Result<(), Errno> {
// Same reason as `queue_trash`: an upload may be reading the very blobs
// this is about to unlink.
self.cancel_upload(uid);
let blobs = self.db.delete_ops_for_uid(&uid.to_string()).map_err(|e| {
error!(%uid, error = %e, "dropping queued ops failed");
Errno::EIO
@@ -3135,6 +3385,15 @@ impl Core {
Ok(())
}
/// Which staged blob is currently queued for `uid`, if any.
///
/// The path identifies the revision: publication stages a fresh file and
/// swaps the map entry to point at it, so an unchanged path means the base a
/// caller sampled earlier is still the base now (bugs.md B32).
fn pending_blob(&self, uid: &NodeUid) -> Option<PathBuf> {
self.pending.lock().get(uid).map(|p| p.path.clone())
}
/// Nudge the drain worker to re-examine the queue now.
fn wake_drain(&self) {
let (lock, cv) = &*self.drain_wake;
@@ -3374,7 +3633,7 @@ impl Core {
// "This file has no thumbnail" is an answer worth remembering: without
// it every listing pays a round trip per file to be told nothing (B5).
let key = (uid.clone(), ttype.as_i32());
if self.no_thumbnail.lock().get(&key) == Some(&mtime) {
if self.thumbnail_misses.lock().remote_contains(&key, mtime) {
return Ok(None);
}
let bytes = self
@@ -3391,15 +3650,7 @@ impl Core {
.store_thumbnail(&uid, ttype.as_i32(), mtime, bytes);
}
None => {
let mut misses = self.no_thumbnail.lock();
// Bounded by clearing rather than by LRU: the entries are two
// words each, the cap is far above any plausible working set,
// and re-learning a miss costs one round trip. Not worth a
// second data structure to order them.
if misses.len() >= MAX_THUMBNAIL_MISSES {
misses.clear();
}
misses.insert(key, mtime);
self.thumbnail_misses.lock().remember_remote(key, mtime);
}
}
Ok(bytes)
@@ -3825,6 +4076,33 @@ impl Core {
}
}
/// Drop the cached child listing of the folder `parent`, everywhere it is
/// held — naming the folder by uid rather than by inode.
///
/// [`Core::invalidate_parent_listing`] is the path-based sibling, and it
/// only reaches *this* mount's inode space. A remote mutation made by the
/// sync engine needs more than that, for two reasons. The folder it changed
/// is usually not resident in any inode space at the time — a mirror folder
/// has no FUSE session at all — so there is no inode to name. And the stale
/// listing that outlives the pass is not a hot-cache map but the `listed`
/// flag on the folder's DB row, which survives a daemon restart and is what
/// a later `ensure_children` trusts.
///
/// So the flag is cleared unconditionally, and the resident inode spaces are
/// invalidated on top of it. That is what makes a mirror folder switched to
/// on-demand re-enumerate instead of serving a snapshot that predates the
/// engine's own uploads — `docs/BUGS.md` B86.
pub(crate) fn invalidate_children_of(&self, parent: &NodeUid) {
if let Err(e) = self.db.set_listed(parent, false) {
warn!(uid = %parent, error = ?e, "could not clear cached listing");
}
self.for_each_state(|st| {
if let Some(&ino) = st.by_uid.get(parent) {
st.invalidate_listing(ino);
}
});
}
/// Rename a file or folder to `new_name`. `rel` is mountpoint-relative.
/// Mirrors the FUSE `rename` write path: rename on the remote, forget the
/// node so it re-interns under its new name, and drop the parent listing so
@@ -5701,9 +5979,19 @@ mod tests {
preserve_on_access_denied, publish_virtual_root_in_listing,
reconcile_virtual_root_in_listing, release_can_discard_unlinked,
release_must_retain_queued_trash, release_unlinked_entry, rename_needs_queue,
require_node_parent_access, require_rename_access, require_uid_access,
resolve_anywhere_with, shared_with_me_uid, take_self_change, virtual_node,
require_node_parent_access, require_rename_access, resolve_anywhere_with,
shared_with_me_uid, take_self_change, uid_write_authority, virtual_node,
};
use super::{Db, WriteAuthority};
/// The collapse [`Core::require_uid_writable`] performs, without a `Core`:
/// every non-writable authority is one `EACCES` on the syscall path.
fn require_uid_access(db: &Db, uid: &NodeUid, live: &[Access]) -> Result<(), Errno> {
match uid_write_authority(db, uid, live) {
WriteAuthority::Writable => Ok(()),
WriteAuthority::Denied | WriteAuthority::Unknown => Err(Errno::EACCES),
}
}
use crate::filesystem::access_allowed;
use pdfs_core::cache::{Baseline, StagedWrite};
use pdfs_core::db::{OP_REVISION, OP_TRASH, PendingOp};
@@ -5734,7 +6022,7 @@ mod tests {
("/mnt/fork-a", &fork_a),
("/mnt/fork-b", &fork_b),
] {
registry.register(
registry.register_bare(
std::path::Path::new(mountpoint),
state,
std::sync::Arc::new(std::sync::OnceLock::new()),
@@ -5785,19 +6073,19 @@ mod tests {
let nested_live = session_flag(false);
let sibling_live = session_flag(true);
registry.register(
registry.register_bare(
primary_path,
&primary,
std::sync::Arc::new(std::sync::OnceLock::new()),
primary_live.clone(),
);
registry.register(
registry.register_bare(
nested_path,
&nested,
std::sync::Arc::new(std::sync::OnceLock::new()),
nested_live.clone(),
);
registry.register(
registry.register_bare(
sibling_path,
&sibling,
std::sync::Arc::new(std::sync::OnceLock::new()),
@@ -5875,6 +6163,71 @@ mod tests {
assert_eq!(registry.live().len(), 2);
}
/// The routing half of `docs/BUGS.md` B86: a path under a secondary mount
/// has to resolve to *that* mount's inode space and a suffix relative to
/// *its* root, or `pdfs ls` and `pdfs refresh` can only ever name the
/// primary mount and a stale on-demand folder has no escape hatch.
#[test]
fn covering_parts_route_a_path_to_the_mount_that_owns_it() {
let registry = StateRegistry::default();
let (primary, _primary_dir) = rooted_state("primary-volume", "primary");
let (nested, _nested_dir) = rooted_state("nested-volume", "nested");
let primary = std::sync::Arc::new(parking_lot::Mutex::new(primary));
let nested = std::sync::Arc::new(parking_lot::Mutex::new(nested));
let primary_path = std::path::Path::new("/home/me/ProtonDrive");
let nested_path = std::path::Path::new("/home/me/ProtonDrive/Device");
let primary_upgrades = std::sync::Arc::new(parking_lot::Mutex::new(HashMap::new()));
let nested_upgrades = std::sync::Arc::new(parking_lot::Mutex::new(HashMap::new()));
registry.register(
primary_path,
&primary,
std::sync::Arc::new(std::sync::OnceLock::new()),
session_flag(true),
primary_upgrades.clone(),
);
registry.register(
nested_path,
&nested,
std::sync::Arc::new(std::sync::OnceLock::new()),
session_flag(true),
nested_upgrades.clone(),
);
let deep = nested_path.join("sub/file.txt");
let parts = registry.covering_parts(&deep).expect("nested mount covers");
assert_eq!(parts.mountpoint, nested_path, "the nested mount wins");
assert!(
std::sync::Arc::ptr_eq(&parts.state, &nested),
"the request must be answered in the nested inode space"
);
assert!(
std::sync::Arc::ptr_eq(&parts.size_upgrades, &nested_upgrades),
"and share that mount's in-flight size upgrades, not the primary's"
);
assert_eq!(
deep.strip_prefix(&parts.mountpoint).unwrap(),
std::path::Path::new("sub/file.txt"),
"the suffix is relative to the mount that owns it, not the primary"
);
let shallow = primary_path.join("Documents");
let parts = registry
.covering_parts(&shallow)
.expect("primary mount covers");
assert!(
std::sync::Arc::ptr_eq(&parts.state, &primary),
"a path outside the nested root still routes to the primary"
);
assert!(
registry
.covering_parts(std::path::Path::new("/tmp/outside"))
.is_none(),
"a path under no mount routes nowhere, and the caller reports that"
);
}
#[test]
fn state_registry_resolves_uids_from_any_resident_mount() {
let registry = StateRegistry::default();
@@ -5888,13 +6241,13 @@ mod tests {
fork.children.insert(super::ROOT_INO, vec![fork_ino]);
let primary = std::sync::Arc::new(parking_lot::Mutex::new(primary));
let fork = std::sync::Arc::new(parking_lot::Mutex::new(fork));
registry.register(
registry.register_bare(
std::path::Path::new("/mnt/primary"),
&primary,
std::sync::Arc::new(std::sync::OnceLock::new()),
session_flag(true),
);
registry.register(
registry.register_bare(
std::path::Path::new("/mnt/device"),
&fork,
std::sync::Arc::new(std::sync::OnceLock::new()),
@@ -5922,7 +6275,7 @@ mod tests {
let child_ino = state.intern(super::ROOT_INO, child_node);
state.children.insert(super::ROOT_INO, vec![child_ino]);
let state = std::sync::Arc::new(parking_lot::Mutex::new(state));
registry.register(
registry.register_bare(
std::path::Path::new("/mnt/device"),
&state,
std::sync::Arc::new(std::sync::OnceLock::new()),
@@ -5985,7 +6338,7 @@ mod tests {
);
let state = std::sync::Arc::new(parking_lot::Mutex::new(state));
registry.register(
registry.register_bare(
std::path::Path::new("/mnt/device"),
&state,
std::sync::Arc::new(std::sync::OnceLock::new()),
@@ -6253,7 +6606,7 @@ mod tests {
state.children.insert(revoked_ino, vec![child_ino]);
let state = std::sync::Arc::new(parking_lot::Mutex::new(state));
registry.register(
registry.register_bare(
std::path::Path::new("/mnt/device"),
&state,
std::sync::Arc::new(std::sync::OnceLock::new()),
@@ -6307,7 +6660,7 @@ mod tests {
state.children.insert(cycle_b, vec![cycle_a]);
let state = std::sync::Arc::new(parking_lot::Mutex::new(state));
registry.register(
registry.register_bare(
std::path::Path::new("/mnt/device"),
&state,
std::sync::Arc::new(std::sync::OnceLock::new()),
@@ -6710,6 +7063,19 @@ mod tests {
require_uid_access(&db, &stale, &[]).unwrap_err().code(),
libc::EACCES
);
// Same refusal on the syscall path, but a distinguishable one
// underneath: the drain asks the remote about `Unknown` rather than
// waiting on a permission change (B83).
assert_eq!(
uid_write_authority(&db, &stale, &[]),
WriteAuthority::Unknown,
"a uid the tree has no row for is unknown, not denied"
);
assert_eq!(
uid_write_authority(&db, &uid, &[]),
WriteAuthority::Denied,
"a uid the tree knows and refuses is denied"
);
assert_eq!(db.pending_ops().unwrap().len(), before);
}
@@ -6876,6 +7242,19 @@ mod tests {
let trash = function_source(source, "fn queue_trash(");
assert_before(trash, "replace_ops_with_trash", "discard_staged");
// §5, parallel drain: every path that unlinks a staged blob or hands
// the same node's work to another worker has to stop the upload that
// may be reading those bytes *first*. Late is a superseded revision on
// the wire, or two workers deciding whether the file exists.
for (signature, mutation) in [
("fn enqueue_staged_write(", "attach_blob_to_create"),
("fn queue_trash(", "replace_ops_with_trash"),
("fn discard_queued_ops(", "delete_ops_for_uid"),
] {
let function = function_source(source, signature);
assert_before(function, "cancel_upload", mutation);
}
let filesystem = include_str!("filesystem.rs");
for signature in ["fn serve_unlink(", "fn serve_rmdir("] {
let function = function_source(filesystem, signature);
@@ -7212,6 +7591,38 @@ mod tests {
}
}
#[cfg(test)]
mod thumbnail_miss_tests {
use super::*;
#[test]
fn remote_misses_never_suppress_local_generation() {
let uid = parse_uid("volume~node").expect("uid");
let key = (uid, ThumbnailType::Thumbnail.as_i32());
let mut misses = ThumbnailMissCaches::default();
misses.remember_remote(key.clone(), 42);
assert!(misses.remote_contains(&key, 42));
assert!(!misses.local_contains(&key, 42));
misses.remember_local(key.clone(), 42);
assert!(misses.local_contains(&key, 42));
misses.forget_local(&key);
assert!(!misses.local_contains(&key, 42));
assert!(misses.remote_contains(&key, 42));
}
#[test]
fn local_miss_cache_is_bounded_on_gui_only_paths() {
let mut misses = ThumbnailMissCaches::default();
for index in 0..=MAX_THUMBNAIL_MISSES {
let uid = parse_uid(&format!("volume~node-{index}")).expect("uid");
misses.remember_local((uid, ThumbnailType::Thumbnail.as_i32()), 42);
}
assert_eq!(misses.local.len(), 1);
}
}
#[cfg(test)]
mod search_root_tests {
use super::*;
+99 -28
View File
@@ -365,6 +365,7 @@ pub fn mount(
readers: Arc::new(Mutex::new(HashMap::new())),
block_ring: Arc::new(Mutex::new(BlockRing::default())),
block_flight: Arc::new(BlockFlight::default()),
content_repairs: Arc::new(tokio::sync::Mutex::new(HashMap::new())),
prefetch: Arc::new(Prefetch::default()),
prefetch_budget: Arc::new(tokio::sync::Semaphore::new(PREFETCH_BUDGET)),
workers: Arc::new(Workers::new(FUSE_WORKERS)?),
@@ -376,6 +377,9 @@ pub fn mount(
pending: Arc::new(Mutex::new(HashMap::new())),
hidden: Arc::new(Mutex::new(HashSet::new())),
drain_wake: Arc::new((Mutex::new(false), Condvar::new())),
shutdown: Arc::new(crate::shutdown::Shutdown::default()),
upload_times: Arc::new(Mutex::new(HashMap::new())),
upload_cancel: Arc::new(Mutex::new(HashMap::new())),
timeline_refreshing: Arc::new(AtomicBool::new(false)),
albums_refreshing: Arc::new(AtomicBool::new(false)),
trash_refreshing: Arc::new(AtomicBool::new(false)),
@@ -385,7 +389,15 @@ pub fn mount(
own_sealed_revs: Arc::new(Mutex::new(HashMap::new())),
self_changes: Arc::new(Mutex::new(HashMap::new())),
thumb_gen: Arc::new(Mutex::new(HashSet::new())),
no_thumbnail: Arc::new(Mutex::new(HashMap::new())),
thumb_gen_budget: Arc::new(tokio::sync::Semaphore::new(
super::photos::THUMB_GEN_CONCURRENCY,
)),
file_thumb_generation: Arc::new(AtomicU64::new(0)),
file_thumb_cancel: Arc::new(tokio::sync::Notify::new()),
thumbnail_build: Arc::new(Mutex::new(Default::default())),
thumbnail_build_cancelled: Arc::new(AtomicBool::new(false)),
thumbnail_build_cancel: Arc::new(tokio::sync::Notify::new()),
thumbnail_misses: Arc::new(Mutex::new(Default::default())),
quota: Arc::new(Mutex::new(None)),
size_upgrades: Arc::new(Mutex::new(HashMap::new())),
size_waiters: Arc::new(Default::default()),
@@ -413,16 +425,29 @@ pub fn mount(
// And finally the staged blobs neither of those accounts for: bytes a
// failed release left behind with nothing pointing at them.
core.reconcile_staging();
{
// Several drain workers over one queue. One thread meant a single large
// upload held every queued rename, trash and small write behind it for as
// long as it ran; the claim column (`Db::claim_next_due_op`) keeps the
// workers off each other's nodes, which is the only ordering the queue
// actually needs. Worker 0 additionally runs the queue's idle chores.
//
// Every long-lived thread started from here on is kept in `workers` and
// joined at the end of this function. Leaving them running turned an
// in-process remount into a second generation of drain workers, sync engines
// and sweeps all operating on a mount that no longer exists (bugs.md B44).
let mut workers: Vec<std::thread::JoinHandle<()>> = Vec::new();
for worker in 0..DRAIN_WORKERS {
let core = core.clone();
std::thread::Builder::new()
.name("pdfs-drain".into())
.spawn(move || core.run_pending_drain())?;
workers.push(
std::thread::Builder::new()
.name(format!("pdfs-drain-{worker}"))
.spawn(move || core.run_pending_drain(worker == 0))?,
);
}
// Start the folder-sync engine. It watches every mirror folder, polls the
// remotes, and reconciles on its own thread — never in front of a FUSE call.
sync::spawn(core.clone(), sync_rx);
workers.extend(sync::spawn(core.clone(), sync_rx));
// Reconcile leftover `(sync-conflict …)` copies: drop the ones that turned
// out identical to the live file, surface the ones that genuinely diverge.
@@ -434,18 +459,22 @@ pub fn mount(
} else {
info!(mode = ?sweep_mode, "conflict sweep enabled");
let core = core.clone();
std::thread::Builder::new()
.name("pdfs-conflict-sweep".into())
.spawn(move || core.run_conflict_sweep_loop())?;
workers.push(
std::thread::Builder::new()
.name("pdfs-conflict-sweep".into())
.spawn(move || core.run_conflict_sweep_loop())?,
);
}
// Mounted from the cache: watch for the network coming back so the mount can
// stop being read-only-ish without the user restarting the daemon.
if !online {
let core = core.clone();
std::thread::Builder::new()
.name("pdfs-online-probe".into())
.spawn(move || core.run_online_probe())?;
workers.push(
std::thread::Builder::new()
.name("pdfs-online-probe".into())
.spawn(move || core.run_online_probe())?,
);
}
// Keep the launcher prompt's "This computer" index fresh. Its own thread:
@@ -455,9 +484,12 @@ pub fn mount(
let indexing = core.indexing.clone();
let transfers = core.transfers.clone();
let mountpoint = mountpoint.to_path_buf();
std::thread::Builder::new()
.name("pdfs-localindex".into())
.spawn(move || run_local_index(db, indexing, transfers, mountpoint))?;
let shutdown = core.shutdown.clone();
workers.push(
std::thread::Builder::new()
.name("pdfs-localindex".into())
.spawn(move || run_local_index(db, indexing, transfers, mountpoint, shutdown))?,
);
}
// Bind the control socket before the FUSE session takes over the thread. A
@@ -494,23 +526,29 @@ pub fn mount(
let control_core = core.clone();
let username = username.clone();
let mountpoint = mountpoint.to_path_buf();
if let Err(error) = std::thread::Builder::new()
match std::thread::Builder::new()
.name("pdfs-control".into())
.spawn(move || run_control_socket(control_core, username, mountpoint, listener))
{
let secondaries = core.mounts.lock().close_and_drain();
for (id, mount) in secondaries {
if let Err(unmount_error) = mount.teardown() {
warn!(id, error = %unmount_error, "secondary teardown after control startup failure failed");
Ok(handle) => workers.push(handle),
Err(error) => {
// A failure here abandons the mount, so the background threads
// started above have to be told to stop with it.
stop_workers(&core, control_socket, workers);
let secondaries = core.mounts.lock().close_and_drain();
for (id, mount) in secondaries {
if let Err(unmount_error) = mount.teardown() {
warn!(id, error = %unmount_error, "secondary teardown after control startup failure failed");
}
}
if let Err(unmount_error) = teardown_session(&core.session_live, || {
umount_session_unblocked(bg, main_conn)
}) {
warn!(error = %unmount_error, "unmount after control startup failure failed");
}
let _ = std::fs::remove_file(control_socket);
return Err(error);
}
if let Err(unmount_error) = teardown_session(&core.session_live, || {
umount_session_unblocked(bg, main_conn)
}) {
warn!(error = %unmount_error, "unmount after control startup failure failed");
}
let _ = std::fs::remove_file(control_socket);
return Err(error);
}
}
// Re-establish on-demand mounts only after the primary session is live.
@@ -619,10 +657,43 @@ pub fn mount(
}
}
let _ = std::fs::remove_file(control_socket);
// Last, because the workers may legitimately be finishing an upload, and
// nothing they can still do is harmful once the mounts are down.
stop_workers(&core, control_socket, workers);
Ok(outcome)
}
/// Signal every background loop, wake the ones that are blocked, and join them.
///
/// The mount used to return with its drain workers, sync engine, sweep, online
/// probe, indexer and control listener all still running on `Core` clones. The
/// process usually exited straight afterwards and hid it; an in-process remount
/// did not, and started a second full set on top of the first (bugs.md B44).
///
/// Two of the waits need waking rather than merely flagging: the drain workers
/// block on their own condvar (`wake_drain`), and the control listener blocks
/// inside `accept`, which only returns when a connection arrives — so it gets
/// one, from us, after the flag is already set. A join that takes a while is
/// expected and correct: a worker part-way through an upload finishes it rather
/// than abandoning the user's bytes mid-flight.
fn stop_workers(core: &Core, control_socket: &Path, workers: Vec<std::thread::JoinHandle<()>>) {
core.shutdown.stop();
core.wake_drain();
let _ = core.sync_tx.send(sync::SyncMsg::Stop);
// Connect while the socket still exists — that is the whole point of the
// poke — then unlink it so nothing else can arrive behind us. The listener
// re-checks the flag after every accept, so even a connection that races the
// unlink cannot put it back to sleep.
let _ = std::os::unix::net::UnixStream::connect(control_socket);
let _ = std::fs::remove_file(control_socket);
for worker in workers {
if worker.join().is_err() {
warn!("a background worker panicked before shutdown");
}
}
debug!("background workers stopped");
}
#[cfg(test)]
mod tests {
use std::sync::atomic::{AtomicBool, Ordering};
File diff suppressed because it is too large. Load diff
+306 -71
View File
@@ -26,6 +26,11 @@ const PREFETCH_MAX: u64 = 8;
/// Window a sequential read starts at, and returns to after a seek.
const PREFETCH_MIN: u64 = 2;
/// Largest file [`Core::repair_block`] will pull down whole to work around a
/// revision whose block table understates it. Past this a single read would cost
/// gigabytes of download, so the read fails instead and says to pin the file.
pub(super) const REPAIR_MAX: u64 = 512 * 1024 * 1024;
/// Blocks that may be prefetching at once, across every file. Prefetch takes a
/// permit and gives up rather than queueing, so it can never sit in front of a
/// demand read in the SDK's flat block semaphore.
@@ -97,22 +102,38 @@ static NEXT_OPEN_ID: AtomicU64 = AtomicU64::new(0);
/// Validated by the same `(mtime, size)` pair the on-disk caches use: a node
/// whose tag no longer matches has its blocks dropped rather than served, so a
/// new revision can never be stitched together from the old one.
/// A ring entry's identity: the node, and the plaintext range the block covers.
///
/// The range, rather than the block index, because the index alone stopped
/// naming one range when the geometry started coming from the revision — a
/// block cached under the [`BlockGeometry::uniform`] fallback and the block at
/// the same index under the real geometry are different bytes
/// (`docs/BUGS.md` B85). A range is the same identity under either plan, so a
/// re-planned read misses rather than being served the wrong bytes.
type RingKey = (NodeUid, BlockSpan);
#[derive(Default)]
pub(super) struct BlockRing {
blocks: HashMap<(NodeUid, u64), Arc<Vec<u8>>>,
blocks: HashMap<RingKey, Arc<Vec<u8>>>,
/// Keys least-recently-used first, for eviction.
order: VecDeque<(NodeUid, u64)>,
order: VecDeque<RingKey>,
/// Per-node validity tag, `(mtime, size)`.
tags: HashMap<NodeUid, (i64, u64)>,
bytes: u64,
}
impl BlockRing {
fn get(&mut self, uid: &NodeUid, mtime: i64, size: u64, idx: u64) -> Option<Arc<Vec<u8>>> {
fn get(
&mut self,
uid: &NodeUid,
mtime: i64,
size: u64,
span: BlockSpan,
) -> Option<Arc<Vec<u8>>> {
if self.tags.get(uid) != Some(&(mtime, size)) {
return None;
}
let key = (uid.clone(), idx);
let key = (uid.clone(), span);
let hit = self.blocks.get(&key).cloned()?;
// Move to the young end: a block being read repeatedly (the whole reason
// this exists) must not age out under a sequential read passing through.
@@ -123,14 +144,21 @@ impl BlockRing {
Some(hit)
}
fn insert(&mut self, uid: &NodeUid, mtime: i64, size: u64, idx: u64, bytes: Arc<Vec<u8>>) {
fn insert(
&mut self,
uid: &NodeUid,
mtime: i64,
size: u64,
span: BlockSpan,
bytes: Arc<Vec<u8>>,
) {
if self.tags.get(uid) != Some(&(mtime, size)) {
// Revision changed under us (or first sight): anything held for this
// node describes the old one.
self.drop_node(uid);
self.tags.insert(uid.clone(), (mtime, size));
}
let key = (uid.clone(), idx);
let key = (uid.clone(), span);
if self.blocks.contains_key(&key) {
return;
}
@@ -171,8 +199,10 @@ impl BlockRing {
/// The result of fetching one block, shared between everyone who wanted it.
type BlockResult = Result<Arc<Vec<u8>>, Errno>;
/// Identifies a block *of a revision*: node, validity tag, index.
type BlockKey = (NodeUid, i64, u64, u64);
/// Identifies a block *of a revision*: node, validity tag, and the plaintext
/// range the block covers — see [`RingKey`] for why it is the range and not the
/// index.
type BlockKey = (NodeUid, i64, u64, BlockSpan);
/// A claim on a block key: who owns the fetch, and where its result is published.
type Claim = (u64, tokio::sync::watch::Receiver<Option<BlockResult>>);
@@ -412,11 +442,30 @@ impl Core {
self.prefetch.streams.lock().remove(uid);
}
/// How this revision's plaintext is divided into content blocks.
///
/// The revision's own geometry when a previous read has learned and recorded
/// it, and the [`BLOCK_SIZE`] assumption when it has not — which is every
/// first read, because learning it means opening a `RevisionReader`, and
/// opening one per read is precisely the per-file key derivation the block
/// cache exists to avoid (bugs.md B12, B85).
///
/// Being wrong here is not a correctness problem: `read_at` plans over the
/// real sizes and clamps to what was asked for either way. It costs a
/// straddling fetch and a cached block the next read will not ask for, and
/// both settle once the geometry is known.
fn block_geometry(&self, uid: &NodeUid, mtime: i64, fsize: u64) -> BlockGeometry {
self.cache
.block_geometry(uid, mtime, fsize)
.unwrap_or_else(|| BlockGeometry::uniform(fsize))
}
/// Serve bytes `[offset, offset + len)` of `uid`'s active revision, hitting
/// the on-disk caches before the network: a whole-file blob (pinned files)
/// first, then the block cache — fetching only the [`BLOCK_SIZE`]-aligned
/// blocks that overlap the request and caching each. `mtime`/`fsize` validate
/// both caches. Network I/O runs without any lock held.
/// first, then the block cache — fetching only the blocks that overlap the
/// request, on the revision's own boundaries ([`Core::block_geometry`]), and
/// caching each. `mtime`/`fsize` validate both caches. Network I/O runs
/// without any lock held.
pub(super) fn read_range(
&self,
uid: &NodeUid,
@@ -495,7 +544,7 @@ impl Core {
Ok(out)
}
/// Fetch block `bidx`, joining a fetch already in flight for it rather than
/// Fetch the block at `span`, joining a fetch already in flight rather than
/// issuing a second one, and publish the result to everyone who joined.
///
/// See [`BlockFlight`]. A follower whose leader disappears (a panic, or a
@@ -507,10 +556,10 @@ impl Core {
uid: &NodeUid,
mtime: i64,
fsize: u64,
bidx: u64,
span: BlockSpan,
cache_blocks: bool,
) -> BlockResult {
let key: BlockKey = (uid.clone(), mtime, fsize, bidx);
let key: BlockKey = (uid.clone(), mtime, fsize, span);
let claim = {
let mut inflight = self.block_flight.inflight.lock();
match inflight.get(&key) {
@@ -542,12 +591,12 @@ impl Core {
}
}
return self
.read_block(reader, uid, mtime, fsize, bidx, cache_blocks)
.read_block(reader, uid, mtime, fsize, span, cache_blocks)
.await;
}
};
let result = self
.read_block(reader, uid, mtime, fsize, bidx, cache_blocks)
.read_block(reader, uid, mtime, fsize, span, cache_blocks)
.await;
// Retire the claim before publishing, so a racer that wakes and re-enters
// finds no stale entry to join.
@@ -563,36 +612,193 @@ impl Core {
/// The disk store is handed to a blocking thread — it is a 4 MiB write plus
/// a cache-index update, and neither the reactor nor the kernel read waiting
/// on these bytes should be held up by it.
///
/// A block that comes back shorter than the file's size says it must be is
/// never served, cached or promoted: it is repaired from the whole-file
/// download, or the read fails. See [`Core::repair_block`] (bugs.md B84).
async fn read_block(
&self,
reader: &Arc<RevisionReader>,
uid: &NodeUid,
mtime: i64,
fsize: u64,
bidx: u64,
span: BlockSpan,
cache_blocks: bool,
) -> BlockResult {
let bstart = bidx * BLOCK_SIZE;
let blen = BLOCK_SIZE.min(fsize - bstart);
let bytes = reader.read_at(bstart, blen).await.map_err(|e| {
warn!(%uid, bstart, blen, error = %e, "block read failed");
Errno::EIO
})?;
// The reader derives every block boundary from the revision's recorded
// block sizes. If they do not add up to the size the node reports, this
// file's boundaries are wrong, and the failure is not confined to short
// reads: sizes that *overstate* a block shift every block after it and
// return full-length reads of the wrong bytes. So a disagreement of any
// kind takes the file off the range path entirely (bugs.md B84).
let bytes = if reader.size() != fsize {
warn!(
%uid, bidx = span.idx, fsize,
reader_size = reader.size(),
reader_blocks = reader.block_sizes().len(),
"the revision's block table disagrees with the node size"
);
self.repair_block(uid, mtime, fsize, span).await?
} else {
// Learned here rather than on a path of its own: this is where a
// reader is open anyway, and recording the real geometry is what
// lets the *next* read plan its blocks on the revision's own
// boundaries instead of the 4 MiB fallback (bugs.md B85).
if cache_blocks {
self.cache
.store_block_geometry(uid, mtime, fsize, reader.block_sizes());
}
let bytes = reader.read_at(span.start, span.len).await.map_err(|e| {
warn!(%uid, start = span.start, len = span.len, error = %e, "block read failed");
Errno::EIO
})?;
if bytes.len() as u64 == span.len {
bytes
} else {
warn!(
%uid, bidx = span.idx, blen = span.len, got = bytes.len(), fsize,
"block came back shorter than the range asked for"
);
self.repair_block(uid, mtime, fsize, span).await?
}
};
let bytes = Arc::new(bytes);
if cache_blocks {
let cache = self.cache.clone();
let uid = uid.clone();
let bytes = bytes.clone();
tokio::task::spawn_blocking(move || {
let _ = cache.store_block(&uid, mtime, fsize, bidx, &bytes);
let _ = cache.store_block(&uid, mtime, fsize, span, &bytes);
});
}
self.block_ring
.lock()
.insert(uid, mtime, fsize, bidx, bytes.clone());
.insert(uid, mtime, fsize, span, bytes.clone());
Ok(bytes)
}
/// Recover the block at `span` of a file whose per-block reads cannot be
/// by fetching the whole revision through the manifest-verified download and
/// serving the block out of that.
///
/// The range path derives every block boundary from the revision's recorded
/// block sizes. When those understate the file, reads come back short with
/// no error anywhere, and the caller hands userspace a truncated file that
/// looks like a clean EOF (bugs.md B84). `download_file_to` does not do that
/// arithmetic — it streams the block list — so it returns the whole file
/// where the range path cannot, which is why pinning was the workaround.
/// This does the same thing on the user's behalf, once, and the resulting
/// blob answers every later read of the file before the block path is
/// consulted at all.
///
/// Bounded by [`REPAIR_MAX`]: past that, one unlucky read would pull down
/// gigabytes, and a clear `EIO` plus a `pdfs pin` is the better trade.
async fn repair_block(
&self,
uid: &NodeUid,
mtime: i64,
fsize: u64,
span: BlockSpan,
) -> Result<Vec<u8>, Errno> {
let (bstart, blen) = (span.start, span.len);
if fsize > REPAIR_MAX {
warn!(
%uid, fsize,
"file is too large to repair from a whole-file download; pin it to read it"
);
return Err(Errno::EIO);
}
// One download per file, not per block: every block of a file with a bad
// block table discovers the same problem at once.
let lock = {
let mut locks = self.content_repairs.lock().await;
locks.entry(uid.clone()).or_default().clone()
};
let guard = lock.clone().lock_owned().await;
let downloaded = match self.cache.read_range(uid, mtime, fsize, bstart, blen) {
// The first arrival did the work; the rest just read its blob.
Some(bytes) if bytes.len() as u64 == blen => Ok(Some(bytes)),
_ => self
.download_whole_revision(uid, mtime, fsize)
.await
.map(|()| None),
};
// Before releasing, so the map entry is only dropped once no waiter is
// still holding it.
drop(guard);
self.release_repair_lock(uid, lock).await;
if let Some(bytes) = downloaded? {
return Ok(bytes);
}
match self.cache.read_range(uid, mtime, fsize, bstart, blen) {
Some(bytes) if bytes.len() as u64 == blen => {
info!(%uid, bidx = span.idx, "served a short block from a repaired whole-file download");
Ok(bytes)
}
_ => {
error!(%uid, bidx = span.idx, fsize, "repairing a short block did not produce the block");
Err(Errno::EIO)
}
}
}
/// Download the whole of `uid`'s active revision to disk and adopt it as the
/// cached blob, so the range path is bypassed entirely for this file.
///
/// Streamed into a scratch file rather than memory: this runs for files up
/// to [`REPAIR_MAX`], and buffering that per concurrent reader is not a cost
/// a read should carry. A download that does not produce exactly `fsize`
/// bytes is discarded — a short blob would fail the cache's own validity
/// check anyway, and storing it would only hide the disagreement one layer
/// further down.
async fn download_whole_revision(
&self,
uid: &NodeUid,
mtime: i64,
fsize: u64,
) -> Result<(), Errno> {
let (mut file, path) = self.cache.create_scratch().map_err(|error| {
error!(%uid, %error, "creating a scratch file for a repair download failed");
Errno::EIO
})?;
let outcome = self.client.download_file_to(uid, &mut file).await;
let stored = (|| {
outcome.map_err(|error| {
warn!(%uid, %error, "repair download failed");
Errno::EIO
})?;
file.sync_all().map_err(|error| {
warn!(%uid, %error, "syncing a repair download failed");
Errno::EIO
})?;
let got = file.metadata().map(|m| m.len()).unwrap_or(0);
if got != fsize {
warn!(%uid, fsize, got, "repair download did not match the node size");
return Err(Errno::EIO);
}
self.cache
.store_file(uid, mtime, fsize, &path)
.map_err(|error| {
error!(%uid, %error, "adopting a repair download as cached content failed");
Errno::EIO
})
})();
// The blob is a hardlink to the same inode, so unlinking the scratch
// name leaves it intact.
let _ = std::fs::remove_file(&path);
stored
}
/// Drop a repair lock's map entry once nobody else holds it, so a file
/// repaired long ago does not keep a mutex alive forever.
async fn release_repair_lock(&self, uid: &NodeUid, lock: Arc<tokio::sync::Mutex<()>>) {
let mut locks = self.content_repairs.lock().await;
// Two: the map's and ours. Anything more means a waiter still needs it.
if Arc::strong_count(&lock) <= 2 {
locks.remove(uid);
}
}
/// Warm the blocks after a sequential read into the caches, in the
/// background, so the next read finds them instead of paying a round-trip.
///
@@ -619,17 +825,19 @@ impl Core {
if depth == 0 {
return;
}
let last_block = fsize.saturating_sub(1) / BLOCK_SIZE;
let wanted: Vec<u64> = ((last + 1)..=(last + depth).min(last_block))
.filter(|&bidx| {
let geometry = self.block_geometry(uid, mtime, fsize);
let last_block = geometry.blocks().saturating_sub(1);
let wanted: Vec<BlockSpan> = ((last + 1)..=(last + depth).min(last_block))
.filter_map(|bidx| geometry.span(bidx))
.filter(|&span| {
// Blocks already in memory need nothing; blocks already on disk
// are cheap enough to serve that spending a round-trip to move
// them into the ring would be the wrong trade.
self.block_ring
.lock()
.get(uid, mtime, fsize, bidx)
.get(uid, mtime, fsize, span)
.is_none()
&& !(cache_blocks && self.cache.has_block(uid, mtime, fsize, bidx))
&& !(cache_blocks && self.cache.has_block(uid, mtime, fsize, span))
})
.collect();
if wanted.is_empty() {
@@ -646,7 +854,7 @@ impl Core {
// round-trips, and the reader catches up with the window before it
// finishes.
let mut set = tokio::task::JoinSet::new();
for bidx in wanted {
for span in wanted {
let Ok(permit) = core.prefetch_budget.clone().try_acquire_owned() else {
break; // the budget is spent; the demand path can have it
};
@@ -654,10 +862,10 @@ impl Core {
set.spawn(async move {
let _permit = permit;
if let Err(error) = core
.fetch_block(&reader, &uid, mtime, fsize, bidx, cache_blocks)
.fetch_block(&reader, &uid, mtime, fsize, span, cache_blocks)
.await
{
debug!(%uid, bidx, ?error, "prefetch failed");
debug!(%uid, bidx = span.idx, ?error, "prefetch failed");
}
});
}
@@ -687,8 +895,13 @@ impl Core {
}
let end = offset.saturating_add(len).min(fsize);
let mut out = Vec::with_capacity((end - offset) as usize);
let first = offset / BLOCK_SIZE;
let last = (end - 1) / BLOCK_SIZE;
// On the revision's own boundaries where they are known, and on the
// 4 MiB fallback where they are not (bugs.md B85).
let wanted = self.block_geometry(uid, mtime, fsize).spans(offset, end);
let (Some(&head), Some(&tail)) = (wanted.first(), wanted.last()) else {
return Ok(out);
};
let (first, last) = (head.idx, tail.idx);
// Collect the blocks overlapping the request, serving any already cached
// and fetching the rest concurrently. A multi-block read (e.g. a media
@@ -696,31 +909,31 @@ impl Core {
// would otherwise stall on each block round-trip in turn; downloading the
// misses in parallel saturates the connection and bounds latency at the
// slowest single block instead of their sum.
let mut blocks: Vec<Option<Arc<Vec<u8>>>> = Vec::with_capacity((last - first + 1) as usize);
let mut misses: Vec<u64> = Vec::new();
for bidx in first..=last {
let mut blocks: Vec<Option<Arc<Vec<u8>>>> = Vec::with_capacity(wanted.len());
let mut misses: Vec<BlockSpan> = Vec::new();
for &span in &wanted {
// The in-memory ring comes first on both paths: a streaming read has
// no other cache at all, and a cached one would otherwise pay two
// opens, a JSON parse and a 4 MiB read+alloc for every 128 KiB the
// kernel asks for (audit T3).
let hit = self.block_ring.lock().get(uid, mtime, fsize, bidx);
let hit = self.block_ring.lock().get(uid, mtime, fsize, span);
let hit = hit.or_else(|| {
if !cache_blocks {
return None;
}
let bytes = Arc::new(self.cache.cached_block(uid, mtime, fsize, bidx)?);
let bytes = Arc::new(self.cache.cached_block(uid, mtime, fsize, span)?);
// Promote it, so the other ~31 reads of this block are answered
// from memory.
self.block_ring
.lock()
.insert(uid, mtime, fsize, bidx, bytes.clone());
.insert(uid, mtime, fsize, span, bytes.clone());
Some(bytes)
});
match hit {
Some(b) => blocks.push(Some(b)),
None => {
blocks.push(None);
misses.push(bidx);
misses.push(span);
}
}
}
@@ -733,12 +946,12 @@ impl Core {
let reader = self.revision_reader(uid, mtime, fsize).await?;
let mut set = tokio::task::JoinSet::new();
for &bidx in &misses {
for &span in &misses {
let (core, reader, uid) = (self.clone(), reader.clone(), uid.clone());
set.spawn(async move {
core.fetch_block(&reader, &uid, mtime, fsize, bidx, cache_blocks)
core.fetch_block(&reader, &uid, mtime, fsize, span, cache_blocks)
.await
.map(|bytes| (bidx, bytes))
.map(|bytes| (span.idx, bytes))
});
}
let mut out = Vec::with_capacity(misses.len());
@@ -756,11 +969,22 @@ impl Core {
// ahead of a sequential reader instead of stalling once it catches up.
self.prefetch_ahead(uid, mtime, fsize, first, last, cache_blocks);
for (i, block) in blocks.into_iter().enumerate() {
let bidx = first + i as u64;
let bstart = bidx * BLOCK_SIZE;
for (span, block) in wanted.into_iter().zip(blocks) {
let bstart = span.start;
// Every slot is populated: cache hits up front, misses by the fetch above.
let block = block.expect("block fetched or cached");
// Both sources guarantee the length the geometry says the block has
// — the cache validates it on read, the network path in `read_block`
// — so this refuses rather than quietly contributing fewer bytes
// than the range covers, which is what turned a bad block into a
// truncated file (bugs.md B84).
if block.len() as u64 != span.len {
error!(
%uid, bidx = span.idx, len = block.len(), expected = span.len, fsize,
"assembling a read hit a short block"
);
return Err(Errno::EIO);
}
let s = (offset.max(bstart) - bstart) as usize;
let e = (end.min(bstart + block.len() as u64) - bstart) as usize;
if s < e {
@@ -850,13 +1074,13 @@ impl Core {
return None;
}
let mut out = Vec::with_capacity((end - offset) as usize);
for bidx in (offset / BLOCK_SIZE)..=((end - 1) / BLOCK_SIZE) {
let ring = self.block_ring.lock().get(uid, mtime, fsize, bidx);
for span in self.block_geometry(uid, mtime, fsize).spans(offset, end) {
let ring = self.block_ring.lock().get(uid, mtime, fsize, span);
let block = match ring {
Some(bytes) => bytes,
None => Arc::new(self.cache.cached_block(uid, mtime, fsize, bidx)?),
None => Arc::new(self.cache.cached_block(uid, mtime, fsize, span)?),
};
let bstart = bidx * BLOCK_SIZE;
let bstart = span.start;
let s = (offset.max(bstart) - bstart) as usize;
let e = (end.min(bstart + block.len() as u64) - bstart) as usize;
if s >= e {
@@ -976,31 +1200,42 @@ mod block_ring_tests {
RING_BYTES / BLOCK_SIZE
}
/// Block `idx` under the uniform geometry — the shape these tests are
/// written against, since what they exercise is tagging and eviction rather
/// than block layout.
fn span(idx: u64) -> BlockSpan {
BlockSpan {
idx,
start: idx * BLOCK_SIZE,
len: BLOCK_SIZE,
}
}
#[test]
fn a_stored_block_is_served_back() {
let mut ring = BlockRing::default();
let node = uid("a");
ring.insert(&node, 7, 100, 3, block(0xab));
assert_eq!(ring.get(&node, 7, 100, 3).map(|b| b[0]), Some(0xab));
ring.insert(&node, 7, 100, span(3), block(0xab));
assert_eq!(ring.get(&node, 7, 100, span(3)).map(|b| b[0]), Some(0xab));
}
#[test]
fn a_block_of_another_revision_is_never_served() {
let mut ring = BlockRing::default();
let node = uid("a");
ring.insert(&node, 7, 100, 0, block(1));
ring.insert(&node, 7, 100, span(0), block(1));
// Same node, newer revision: the tag moved, so the old bytes go.
assert!(ring.get(&node, 8, 100, 0).is_none());
assert!(ring.get(&node, 7, 101, 0).is_none());
assert!(ring.get(&node, 8, 100, span(0)).is_none());
assert!(ring.get(&node, 7, 101, span(0)).is_none());
}
#[test]
fn a_new_revision_drops_what_the_old_one_left() {
let mut ring = BlockRing::default();
let node = uid("a");
ring.insert(&node, 7, 100, 0, block(1));
ring.insert(&node, 8, 100, 0, block(2));
assert_eq!(ring.get(&node, 8, 100, 0).map(|b| b[0]), Some(2));
ring.insert(&node, 7, 100, span(0), block(1));
ring.insert(&node, 8, 100, span(0), block(2));
assert_eq!(ring.get(&node, 8, 100, span(0)).map(|b| b[0]), Some(2));
assert_eq!(ring.bytes, BLOCK_SIZE);
}
@@ -1009,13 +1244,13 @@ mod block_ring_tests {
let mut ring = BlockRing::default();
let node = uid("a");
for idx in 0..capacity() {
ring.insert(&node, 1, u64::MAX, idx, block(idx as u8));
ring.insert(&node, 1, u64::MAX, span(idx), block(idx as u8));
}
// Re-reading block 0 makes block 1 the oldest instead.
assert!(ring.get(&node, 1, u64::MAX, 0).is_some());
ring.insert(&node, 1, u64::MAX, capacity(), block(0xff));
assert!(ring.get(&node, 1, u64::MAX, 0).is_some());
assert!(ring.get(&node, 1, u64::MAX, 1).is_none());
assert!(ring.get(&node, 1, u64::MAX, span(0)).is_some());
ring.insert(&node, 1, u64::MAX, span(capacity()), block(0xff));
assert!(ring.get(&node, 1, u64::MAX, span(0)).is_some());
assert!(ring.get(&node, 1, u64::MAX, span(1)).is_none());
assert_eq!(ring.bytes, RING_BYTES);
}
@@ -1023,9 +1258,9 @@ mod block_ring_tests {
fn a_node_whose_last_block_ages_out_keeps_no_tag() {
let mut ring = BlockRing::default();
let (old, new) = (uid("old"), uid("new"));
ring.insert(&old, 1, u64::MAX, 0, block(1));
ring.insert(&old, 1, u64::MAX, span(0), block(1));
for idx in 0..capacity() {
ring.insert(&new, 1, u64::MAX, idx, block(2));
ring.insert(&new, 1, u64::MAX, span(idx), block(2));
}
assert!(!ring.tags.contains_key(&old));
assert!(ring.tags.contains_key(&new));
@@ -1035,11 +1270,11 @@ mod block_ring_tests {
fn dropping_a_node_frees_only_its_own_bytes() {
let mut ring = BlockRing::default();
let (a, b) = (uid("a"), uid("b"));
ring.insert(&a, 1, 100, 0, block(1));
ring.insert(&b, 1, 100, 0, block(2));
ring.insert(&a, 1, 100, span(0), block(1));
ring.insert(&b, 1, 100, span(0), block(2));
ring.drop_node(&a);
assert!(ring.get(&a, 1, 100, 0).is_none());
assert_eq!(ring.get(&b, 1, 100, 0).map(|x| x[0]), Some(2));
assert!(ring.get(&a, 1, 100, span(0)).is_none());
assert_eq!(ring.get(&b, 1, 100, span(0)).map(|x| x[0]), Some(2));
assert_eq!(ring.bytes, BLOCK_SIZE);
}
}
+1
View File
@@ -609,6 +609,7 @@ impl Core {
items.push(SharedItem {
uid: uid.to_string(),
is_dir: n.is_folder(),
modified: n.modification_time,
name: n.name,
path: self.rel_path_for_uid(&uid).unwrap_or_default(),
member_count: members,
+96
View File
@@ -0,0 +1,96 @@
//! One shutdown signal shared by every long-lived worker in the daemon.
//!
//! The mount used to end by unmounting and returning, leaving the drain
//! workers, the sync engine, the conflict sweep, the online probe and the local
//! indexer running on `Core` clones. In a process that exits immediately that is
//! invisible; in an in-process remount — which is how the daemon recovers from a
//! mount going away underneath it — it leaks a whole generation of threads that
//! keep writing to the database and calling the API on behalf of a mount that no
//! longer exists (bugs.md B44).
//!
//! The primitive is deliberately small: a flag plus a condvar. Every loop that
//! would otherwise call `thread::sleep` calls [`Shutdown::sleep`] instead, which
//! returns early — and reports `false` — the moment [`Shutdown::stop`] is
//! called, so teardown does not have to wait out a five-minute sweep interval to
//! join a thread.
use std::time::Duration;
use parking_lot::{Condvar, Mutex};
/// The daemon's stop signal. Set once, never cleared: a `Core` that has been
/// told to stop belongs to a mount that is going away.
#[derive(Default)]
pub(crate) struct Shutdown {
stopping: Mutex<bool>,
changed: Condvar,
}
impl Shutdown {
/// Whether teardown has begun. Loops check this where they would otherwise
/// start another unit of work.
pub(crate) fn is_stopping(&self) -> bool {
*self.stopping.lock()
}
/// Begin teardown and wake everything waiting in [`sleep`](Self::sleep).
/// Idempotent, because more than one path can reach teardown (a signal, the
/// kernel mount ending, a failed startup).
pub(crate) fn stop(&self) {
let mut stopping = self.stopping.lock();
*stopping = true;
self.changed.notify_all();
}
/// Wait up to `duration`, or until teardown starts.
///
/// Returns `false` if the daemon is stopping — including when it was already
/// stopping on entry, so `while shutdown.sleep(interval)` is a complete and
/// correct loop condition and no separate check is needed at the top.
pub(crate) fn sleep(&self, duration: Duration) -> bool {
let mut stopping = self.stopping.lock();
if !*stopping {
self.changed.wait_for(&mut stopping, duration);
}
!*stopping
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
use std::time::Instant;
#[test]
fn sleeping_returns_early_once_stopped() {
let shutdown = Arc::new(Shutdown::default());
assert!(!shutdown.is_stopping());
let waker = shutdown.clone();
std::thread::spawn(move || {
std::thread::sleep(Duration::from_millis(20));
waker.stop();
});
let started = Instant::now();
// An hour's worth of interval, cut short by the stop.
assert!(!shutdown.sleep(Duration::from_secs(3600)));
assert!(
started.elapsed() < Duration::from_secs(5),
"stop must interrupt the wait, not wait it out"
);
assert!(shutdown.is_stopping());
// Already stopping: the next wait does not sleep at all.
let started = Instant::now();
assert!(!shutdown.sleep(Duration::from_secs(3600)));
assert!(started.elapsed() < Duration::from_secs(5));
}
#[test]
fn sleeping_runs_to_the_timeout_while_running() {
let shutdown = Shutdown::default();
assert!(shutdown.sleep(Duration::from_millis(10)));
assert!(!shutdown.is_stopping());
}
}
+16
View File
@@ -141,6 +141,11 @@ pub(crate) struct PendingRevision {
pub(crate) meta: StagedWrite,
}
/// One directory's entries as `readdir` reports them: `(ino, is_dir, name)` per
/// entry, in reply order. The kernel's file-type enum is mapped in at reply
/// time so this module stays free of `fuser`.
pub(crate) type DirListing = Vec<(u64, bool, String)>;
/// Mutable inode bookkeeping, guarded by a mutex because fuser drives the
/// `Filesystem` trait through `&self`.
pub(crate) struct State {
@@ -158,6 +163,16 @@ pub(crate) struct State {
/// have no entry here.
pub(crate) handles: HashMap<u64, u64>,
pub(crate) next_fh: u64,
/// One frozen listing per open directory handle, taken at `opendir`.
///
/// `readdir` resumes from a byte offset into the listing it is walking, so
/// that listing has to stay the same across the calls of one `getdents`
/// sequence. Rebuilding it live meant a create or a trash landing between
/// two pages shifted every later entry, and the caller silently skipped or
/// repeated whatever crossed the page boundary. Each entry is
/// `(ino, is_dir, name)` — the kernel type is mapped at reply time so this
/// module stays free of `fuser` (bugs.md B43).
pub(crate) dir_snapshots: HashMap<u64, Arc<DirListing>>,
/// Resident inode attributes whose effective access changed during an
/// intern/root refresh. Core drains this set and notifies the matching
/// kernel only after releasing the State lock.
@@ -317,6 +332,7 @@ impl State {
active_writes: HashMap::new(),
handles: HashMap::new(),
next_fh: 1,
dir_snapshots: HashMap::new(),
access_changes: HashSet::new(),
share_access,
db,
+8 -2
View File
@@ -67,10 +67,16 @@ impl Core {
/// Sweep for stale conflict copies forever, on its own thread. See the module
/// docs for the identical/divergent policy.
pub(crate) fn run_conflict_sweep_loop(&self) {
std::thread::sleep(CONFLICT_SWEEP_WARMUP);
// Both waits are interruptible, so teardown does not have to sit out a
// five-minute interval to join this thread (bugs.md B44).
if !self.shutdown.sleep(CONFLICT_SWEEP_WARMUP) {
return;
}
loop {
self.sweep_conflicts_once();
std::thread::sleep(CONFLICT_SWEEP_INTERVAL);
if !self.shutdown.sleep(CONFLICT_SWEEP_INTERVAL) {
return;
}
}
}
+585 -77
View File
@@ -54,7 +54,7 @@ use engine::{classify, settle_with};
#[cfg(test)]
use local_scan::is_write_mode;
use local_scan::open_for_write_set;
use model::{LocalItem, Outcome, PassAbort, Pending, RemoteItem};
use model::{LocalItem, LocalSig, Outcome, PassAbort, Pending, RemoteItem};
pub(crate) use planner::base_name;
#[cfg(test)]
use planner::conflict_path;
@@ -525,6 +525,9 @@ impl Core {
continue;
}
trashed.push(rel.clone());
if let Some(parent) = parent_uid_of(rel, &remote_dirs, &baseline) {
self.invalidate_children_of(&parent);
}
let _ = self.db.sync_entry_remove(folder_id, rel);
self.log_activity(
ActivityKind::Trash,
@@ -687,6 +690,7 @@ impl Core {
rel_path: rel.clone(),
remote_uid: Some(remote.uid.to_string()),
local_mtime: local.mtime,
local_mtime_ns: local.mtime_ns,
local_size: local.size,
remote_rev: Some(remote.mtime.to_string()),
remote_hash: Some(remote.size.to_string()),
@@ -717,6 +721,9 @@ impl Core {
uid: remote.uid.clone(),
mtime: remote.mtime,
size: remote.size,
// What the walk saw here, so the apply step can tell a
// stale plan from a live one before it overwrites.
local: l.map(LocalSig::from),
});
}
FilePlan::Conflict => {
@@ -739,7 +746,9 @@ impl Core {
}
},
FilePlan::DeleteLocal => {
if let Err(e) = std::fs::remove_file(local_root.join(rel_to_path(rel))) {
if let Err(e) =
removed_locally(std::fs::remove_file(local_root.join(rel_to_path(rel))))
{
warn!(rel, error = %e, "sync: remove local file failed");
self.log_activity(
ActivityKind::Trash,
@@ -771,6 +780,9 @@ impl Core {
outcome.errors += 1;
continue;
}
if let Some(parent) = parent_uid_of(rel, &remote_dirs, &baseline) {
self.invalidate_children_of(&parent);
}
let _ = self.db.sync_entry_remove(folder_id, rel);
self.log_activity(
ActivityKind::Trash,
@@ -797,7 +809,9 @@ impl Core {
// Deferred folder deletions, deepest first.
delete_local_dirs.sort_by_key(|p| std::cmp::Reverse(p.matches('/').count()));
for rel in delete_local_dirs {
if let Err(e) = std::fs::remove_dir_all(local_root.join(rel_to_path(&rel))) {
if let Err(e) =
removed_locally(std::fs::remove_dir_all(local_root.join(rel_to_path(&rel))))
{
warn!(rel, error = %e, "sync: remove local folder failed");
self.log_activity(ActivityKind::Trash, base_name(&rel), e.to_string(), false);
outcome.errors += 1;
@@ -824,6 +838,9 @@ impl Core {
outcome.errors += 1;
continue;
}
if let Some(parent) = parent_uid_of(&rel, &remote_dirs, &baseline) {
self.invalidate_children_of(&parent);
}
let _ = self.db.sync_entry_remove(folder_id, &rel);
self.log_activity(
ActivityKind::Trash,
@@ -849,68 +866,9 @@ impl Core {
writing: &HashSet<PathBuf>,
out: &mut HashMap<String, LocalItem>,
) -> Result<(), String> {
let entries = std::fs::read_dir(dir).map_err(|e| format!("read {}: {e}", dir.display()))?;
for entry in entries {
let entry = entry.map_err(|e| format!("read entry in {}: {e}", dir.display()))?;
let path = entry.path();
let meta = std::fs::symlink_metadata(&path)
.map_err(|e| format!("stat {}: {e}", path.display()))?;
let stripped = path.strip_prefix(root).map_err(|e| {
format!(
"local path {} escaped sync root {}: {e}",
path.display(),
root.display()
)
})?;
let rel = stripped.to_str().ok_or_else(|| {
format!(
"local path {} is not valid UTF-8; refusing a destructive sync pass",
path.display()
)
})?;
// Ignore our own in-flight download temp files.
if rel.contains(".pdfs-tmp-") {
continue;
}
// Ignored paths are dropped here as well as from the classification
// union — not for correctness (the union filter is what makes this
// safe) but so an ignored `node_modules/` is never descended into or
// counted against scan progress.
if rules.is_ignored(rel, meta.is_dir()) {
continue;
}
if meta.is_dir() {
out.insert(
rel.to_string(),
LocalItem {
is_dir: true,
mtime: 0,
size: 0,
open_for_write: false,
},
);
self.progress_scanned(folder_id, base_name(rel));
self.walk_local(folder_id, root, &path, rules, writing, out)?;
} else if meta.is_file() {
// Check whether any process holds this file open for writing.
// The canonical path is used because /proc/*/fd links resolve
// to canonical targets.
let ofw = std::fs::canonicalize(&path)
.map(|canon| writing.contains(&canon))
.unwrap_or(false);
out.insert(
rel.to_string(),
LocalItem {
is_dir: false,
mtime: system_mtime(&meta),
size: meta.len() as i64,
open_for_write: ofw,
},
);
self.progress_scanned(folder_id, base_name(rel));
}
}
Ok(())
walk_local_tree(root, dir, rules, writing, out, &mut |rel| {
self.progress_scanned(folder_id, base_name(rel))
})
}
/// Map every remote folder under `folder` to its uid by rel path, ignoring
@@ -1069,6 +1027,7 @@ impl Core {
rel_path: rel.to_string(),
remote_uid: Some(uid.to_string()),
local_mtime: 0,
local_mtime_ns: None,
local_size: 0,
remote_rev: None,
remote_hash: None,
@@ -1175,6 +1134,9 @@ impl Core {
.create_folder(parent, base_name(rel), Some(now_secs()))
.await
.map_err(|e| format!("create remote folder {rel}: {e}"))?;
// The parent has gained a child that no cached listing of it
// knows about (B86).
self.invalidate_children_of(parent);
self.baseline_dir(folder_id, rel, &uid)?;
Ok(Applied::Dir(rel.clone(), uid))
}
@@ -1192,8 +1154,9 @@ impl Core {
uid,
mtime,
size,
local,
} => {
self.download_file(folder_id, local_root, rel, uid, *mtime, *size)
self.download_file(folder_id, local_root, rel, uid, *mtime, *size, *local)
.await?;
Ok(Applied::Downloaded)
}
@@ -1211,7 +1174,11 @@ impl Core {
.map_err(|e| format!("set aside conflict copy for {rel}: {e}"))?;
info!(rel, "sync: kept local changes as a conflict copy");
}
self.download_file(folder_id, local_root, rel, uid, *mtime, *size)
// The local copy has just been moved aside, so the state this
// download expects to find at the path is "nothing" — anything
// that appears while it runs is a *new* racing write, and gets
// its own conflict copy rather than being overwritten (B39).
self.download_file(folder_id, local_root, rel, uid, *mtime, *size, None)
.await?;
Ok(Applied::Conflict)
}
@@ -1250,6 +1217,9 @@ impl Core {
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
let mtime = system_mtime(&meta);
// Taken from the descriptor being streamed, so it names the content that
// actually goes up rather than whatever is at the path when we finish.
let streamed = LocalSig::from(&meta);
// Count the bytes as they are read, so this upload shows up in
// `GetQueueStatus` next to a manual one. The uid isn't known until the
// draft is sealed, so the transfer registers without one.
@@ -1276,7 +1246,13 @@ impl Core {
)
.await
.map_err(|e| format!("upload {rel}: {e}"))?;
self.record_file_baseline(folder_id, rel, &path, &uid).await
// The parent has gained a child that no cached listing of it knows
// about. Without this, a folder later switched to on-demand serves a
// snapshot that predates this upload (B86).
self.invalidate_children_of(parent);
warn_if_torn(rel, &path, streamed);
self.record_file_baseline(folder_id, rel, Some(streamed), &uid)
.await
}
/// Upload a changed local file as a new revision of an existing remote node.
@@ -1291,6 +1267,7 @@ impl Core {
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
let mtime = system_mtime(&meta);
let streamed = LocalSig::from(&meta);
let reader = OwnedCountingReader::new(
file,
self.transfers.begin(
@@ -1304,11 +1281,20 @@ impl Core {
.upload_new_revision_from(uid, reader, meta.len() as i64, Vec::new(), Some(mtime))
.await
.map_err(|e| format!("upload revision {rel}: {e}"))?;
self.record_file_baseline(folder_id, rel, &path, uid).await
warn_if_torn(rel, &path, streamed);
self.record_file_baseline(folder_id, rel, Some(streamed), uid)
.await
}
/// Download a remote file to its local path (atomically via a temp file),
/// stamp the local mtime to match the remote, then record baseline.
///
/// `expected_local` is the identity planning saw at the destination. The
/// rename below is the destructive step, and the download in between can run
/// for minutes, so the destination is re-examined right before it: a local
/// file that no longer matches the plan is set aside as a conflict copy
/// instead of being replaced (bugs.md B39).
#[allow(clippy::too_many_arguments)]
async fn download_file(
&self,
folder_id: i64,
@@ -1317,6 +1303,7 @@ impl Core {
uid: &NodeUid,
mtime: i64,
size: i64,
expected_local: Option<LocalSig>,
) -> Result<(), String> {
let path = local_root.join(rel_to_path(rel));
if let Some(parent) = path.parent() {
@@ -1351,6 +1338,13 @@ impl Core {
out.flush().map_err(|e| format!("flush tmp {rel}: {e}"))?;
out.sync_all().map_err(|e| format!("sync tmp {rel}: {e}"))?;
}
// The last moment at which the destination can still be saved. Anything
// other than the state planning classified means a writer got there
// first, and its bytes are the ones nothing else has a copy of.
if let Err(e) = keep_racing_local_edit(&path, expected_local) {
let _ = std::fs::remove_file(&tmp);
return Err(format!("set aside racing local edit for {rel}: {e}"));
}
std::fs::rename(&tmp, &path).map_err(|e| format!("place {rel}: {e}"))?;
// Match local mtime to the remote's so neither side looks "changed" next pass.
let f = std::fs::File::options()
@@ -1366,23 +1360,36 @@ impl Core {
.and_then(|dir| dir.sync_all())
.map_err(|e| format!("sync directory for {rel}: {e}"))?;
}
self.record_file_baseline(folder_id, rel, &path, uid).await
// Here the file on disk *is* the transferred content: it was published
// by a rename and stamped just above, so its stat is the identity.
self.record_file_baseline(folder_id, rel, local_sig(&path), uid)
.await
}
/// Refresh a file's baseline from ground truth: the local stat and the
/// remote node's reported signature. Called after every upload/download so
/// the very next reconcile sees the path as unchanged.
/// Refresh a file's baseline from ground truth: the local identity these
/// bytes came from and the remote node's reported signature. Called after
/// every upload/download so the very next reconcile sees the path as
/// unchanged.
///
/// `local` is the identity the *transferred content* corresponds to, which
/// is not always the file's current stat. An upload streams the live file,
/// so it passes the identity it read the first byte at: if a writer changed
/// the file mid-stream, the remote now holds a torn revision, and recording
/// the pre-upload identity is what makes the next pass see the path as
/// locally changed and upload a whole, clean revision over it. Recording the
/// post-upload stat instead declared the torn revision synchronized and left
/// it as the file's content on Drive (bugs.md B40).
async fn record_file_baseline(
&self,
folder_id: i64,
rel: &str,
local_path: &Path,
local: Option<LocalSig>,
uid: &NodeUid,
) -> Result<(), String> {
let (lmtime, lsize) = match std::fs::metadata(local_path) {
Ok(m) => (system_mtime(&m), m.len() as i64),
Err(e) => return Err(format!("stat {rel}: {e}")),
let Some(local) = local else {
return Err(format!("stat {rel}: no local identity to baseline"));
};
let (lmtime, lsize) = (local.mtime, local.size);
// Re-fetch the node so the stored remote signature is exactly what a walk
// will report next time.
let (rmtime, rsize) = match self.client.enumerate_nodes(std::slice::from_ref(uid)).await {
@@ -1402,6 +1409,7 @@ impl Core {
rel_path: rel.to_string(),
remote_uid: Some(uid.to_string()),
local_mtime: lmtime,
local_mtime_ns: local.mtime_ns,
local_size: lsize,
remote_rev: Some(rmtime.to_string()),
remote_hash: Some(rsize.to_string()),
@@ -1411,6 +1419,177 @@ impl Core {
}
}
/// Say so when a file changed under an upload that was streaming it.
///
/// Purely diagnostic — the correction is already in place, because the baseline
/// records the identity the upload *read* rather than the one on disk now, so
/// the next pass classifies the path as locally changed and replaces the torn
/// revision. Without the log there is nothing in the journal to explain the
/// extra revision (bugs.md B40).
fn warn_if_torn(rel: &str, path: &Path, streamed: LocalSig) {
let now = local_sig(path);
if !now.is_some_and(|now| now.same_content(&streamed)) {
warn!(
rel,
?streamed,
?now,
"sync: file changed while uploading; the remote revision may be torn \
and will be replaced on the next pass"
);
}
}
/// The identity a reconcile pass uses to recognise a local file it has already
/// looked at: `None` when nothing is at the path.
fn local_sig(path: &Path) -> Option<LocalSig> {
std::fs::metadata(path).ok().map(|m| LocalSig {
mtime: system_mtime(&m),
mtime_ns: system_mtime_ns(&m),
size: m.len() as i64,
})
}
/// Guard the destination of a download: if what is at `path` is no longer what
/// planning classified, move it aside as a conflict copy before the caller
/// renames over it.
///
/// A reconcile pass decides what to do from a walk that finished before the
/// transfer started, and a download of a large file can run for minutes. Without
/// this, a local edit completed inside that window was replaced by the remote
/// version with no copy of it kept anywhere — the one shape of loss a sync
/// engine must not have (bugs.md B39). Keeping the copy makes it re-upload as a
/// new file on the next pass, which is the same resolution a both-sides-changed
/// conflict already gets.
///
/// A path that is empty when it was expected to be is not a conflict: the
/// download is about to recreate it, and there is nothing to preserve.
fn keep_racing_local_edit(path: &Path, expected: Option<LocalSig>) -> std::io::Result<()> {
let actual = local_sig(path);
let unchanged = match (actual, expected) {
(Some(a), Some(e)) => a.same_content(&e),
// Nothing there: the download is about to create it, and there is
// nothing to preserve either way.
(None, _) => true,
(Some(_), None) => false,
};
if unchanged {
return Ok(());
}
warn!(
path = %path.display(),
?expected,
?actual,
"sync: local file changed while downloading; keeping it as a conflict copy"
);
preserve_conflict_copy(path, now_secs()).map(|_| ())
}
/// Read a local removal as the sync engine has to read it: the *state* it was
/// asking for is "this path is gone", so a path that was already gone is a
/// success, not a failure.
///
/// It matters because the baseline is only dropped on success. Treating
/// `NotFound` as an error left the baseline in place for a file that no longer
/// exists, so the same removal was re-attempted and re-reported as an error on
/// every pass, forever. Every other error is still an error, and keeping the
/// baseline is exactly right there: the surviving file must not be mistaken for
/// a new local creation on the next pass and uploaded back (bugs.md B42).
fn removed_locally(result: std::io::Result<()>) -> std::io::Result<()> {
match result {
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
other => other,
}
}
/// The body of [`Core::walk_local`], without a `Core`: recursively collect the
/// local tree under `dir` into `out`, reporting each entry to `scanned`.
///
/// **Every failure here is fatal to the pass, deliberately.** A `read_dir` that
/// fails, an entry that cannot be stat-ed, a name that is not UTF-8 — each of
/// them would, if skipped, leave `out` describing a *smaller* tree than the one
/// on disk. Reconciliation reads a path present in the baseline and absent from
/// the scan as "the user deleted this", and propagates that to Drive. So an
/// unreadable subdirectory would trash its remote counterpart, and the deeper
/// the failure, the more it takes with it (`docs/BUGS.md` B55). Refusing the
/// whole pass keeps the previous baseline and leaves the remote alone; the next
/// pass retries once the transient condition clears.
///
/// Split out from the method so exactly that can be tested against a real tree
/// with a real unreadable directory in it, which needs no `Core` at all.
fn walk_local_tree(
root: &Path,
dir: &Path,
rules: &IgnoreRules,
writing: &HashSet<PathBuf>,
out: &mut HashMap<String, LocalItem>,
scanned: &mut dyn FnMut(&str),
) -> Result<(), String> {
let entries = std::fs::read_dir(dir).map_err(|e| format!("read {}: {e}", dir.display()))?;
for entry in entries {
let entry = entry.map_err(|e| format!("read entry in {}: {e}", dir.display()))?;
let path = entry.path();
let meta = std::fs::symlink_metadata(&path)
.map_err(|e| format!("stat {}: {e}", path.display()))?;
let stripped = path.strip_prefix(root).map_err(|e| {
format!(
"local path {} escaped sync root {}: {e}",
path.display(),
root.display()
)
})?;
let rel = stripped.to_str().ok_or_else(|| {
format!(
"local path {} is not valid UTF-8; refusing a destructive sync pass",
path.display()
)
})?;
// Ignore our own in-flight download temp files.
if rel.contains(".pdfs-tmp-") {
continue;
}
// Ignored paths are dropped here as well as from the classification
// union — not for correctness (the union filter is what makes this
// safe) but so an ignored `node_modules/` is never descended into or
// counted against scan progress.
if rules.is_ignored(rel, meta.is_dir()) {
continue;
}
if meta.is_dir() {
out.insert(
rel.to_string(),
LocalItem {
is_dir: true,
mtime: 0,
mtime_ns: None,
size: 0,
open_for_write: false,
},
);
scanned(rel);
walk_local_tree(root, &path, rules, writing, out, scanned)?;
} else if meta.is_file() {
// Check whether any process holds this file open for writing.
// The canonical path is used because /proc/*/fd links resolve
// to canonical targets.
let ofw = std::fs::canonicalize(&path)
.map(|canon| writing.contains(&canon))
.unwrap_or(false);
out.insert(
rel.to_string(),
LocalItem {
is_dir: false,
mtime: system_mtime(&meta),
mtime_ns: system_mtime_ns(&meta),
size: meta.len() as i64,
open_for_write: ofw,
},
);
scanned(rel);
}
}
Ok(())
}
/// Publish a conflict copy without overwriting an earlier conflict bearing the
/// same timestamp. The original is removed only after the new copy is durable.
fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> {
@@ -1444,6 +1623,26 @@ fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> {
unreachable!("u32 conflict suffix space exhausted")
}
/// The remote uid of `rel`'s parent folder: from the pass's folder-uid map
/// first, and from the baseline second for a parent the map never covered.
///
/// Used to name the folder whose cached listing a remote mutation has just
/// invalidated (B86). `None` is not an error — it only means this pass cannot
/// say which folder to invalidate, and the caller skips it.
fn parent_uid_of(
rel: &str,
remote_dirs: &HashMap<String, NodeUid>,
baseline: &HashMap<String, StoredSyncEntry>,
) -> Option<NodeUid> {
let parent = parent_rel(rel);
remote_dirs.get(parent).cloned().or_else(|| {
baseline
.get(parent)
.and_then(|e| e.remote_uid.as_deref())
.and_then(parse_uid)
})
}
fn sync_parent_dir(path: &Path) -> std::io::Result<()> {
match path.parent() {
Some(parent) => std::fs::File::open(parent)?.sync_all(),
@@ -1476,10 +1675,77 @@ fn system_mtime(meta: &std::fs::Metadata) -> i64 {
.map(|d| d.as_secs() as i64)
.unwrap_or(0)
}
/// The same modification time in nanoseconds, when the filesystem reports one.
///
/// Whole seconds cannot distinguish an edit made in the same second as the last
/// sync from no edit at all, and if the size did not change either, that file
/// was never uploaded (bugs.md B25). `None` for a clock or filesystem that
/// cannot answer, which falls the comparison back to seconds rather than
/// asserting an mtime of zero.
fn system_mtime_ns(meta: &std::fs::Metadata) -> Option<i64> {
meta.modified()
.ok()
.and_then(|t| t.duration_since(SystemTime::UNIX_EPOCH).ok())
.and_then(|d| i64::try_from(d.as_nanos()).ok())
}
#[cfg(test)]
mod tests {
use super::*;
/// A removal of something already gone has produced the state the sync
/// engine wanted, so its baseline is dropped rather than retried forever.
/// Anything else stays an error and keeps the baseline (bugs.md B42).
#[test]
fn an_already_absent_path_counts_as_removed() {
let root = std::env::temp_dir().join(format!("pdfs-b42-{}", std::process::id()));
let missing = root.join("never-existed");
assert!(removed_locally(std::fs::remove_file(&missing)).is_ok());
assert!(removed_locally(std::fs::remove_dir_all(&missing)).is_ok());
// A directory in the way of `remove_file` is a real failure, and must
// stay one: the file survives, and its baseline has to survive with it.
let occupied = root.join("a-directory");
std::fs::create_dir_all(&occupied).unwrap();
assert!(removed_locally(std::fs::remove_file(&occupied)).is_err());
let _ = std::fs::remove_dir_all(&root);
}
/// The download guard: a destination that still matches the plan is
/// replaced silently, one that has moved on since is kept (bugs.md B39).
#[test]
fn a_download_keeps_a_local_edit_it_did_not_plan_for() {
let root = std::env::temp_dir().join(format!("pdfs-b39-{}", std::process::id()));
std::fs::create_dir_all(&root).unwrap();
let path = root.join("report.txt");
// Nothing there, nothing expected: the download simply creates it.
assert!(keep_racing_local_edit(&path, None).is_ok());
assert!(!path.exists());
// Unchanged since planning: no copy, the caller overwrites it.
std::fs::write(&path, b"planned").unwrap();
let planned = local_sig(&path).unwrap();
keep_racing_local_edit(&path, Some(planned)).unwrap();
assert_eq!(std::fs::read(&path).unwrap(), b"planned");
// Changed since planning: the racing bytes are the only copy of
// themselves, so they must survive the overwrite as a conflict copy.
std::fs::write(&path, b"a much longer local edit").unwrap();
keep_racing_local_edit(&path, Some(planned)).unwrap();
let kept: Vec<_> = std::fs::read_dir(&root)
.unwrap()
.filter_map(|e| e.ok())
.map(|e| std::fs::read(e.path()).unwrap_or_default())
.collect();
assert!(
kept.iter()
.any(|bytes| bytes == b"a much longer local edit"),
"the racing edit must still exist somewhere: {kept:?}"
);
let _ = std::fs::remove_dir_all(&root);
}
/// Feed `n` `Reconcile` events `gap` apart on another thread, starting
/// immediately. Returns the sender's join handle so the test can keep it
/// alive (dropping the sender early would look like a disconnect).
@@ -1565,6 +1831,7 @@ mod tests {
fn local_item(is_dir: bool) -> LocalItem {
LocalItem {
mtime_ns: None,
is_dir,
mtime: 1,
size: 1,
@@ -1722,6 +1989,44 @@ mod tests {
assert!(guard_local_wipe(&base, &fresh).is_err());
}
/// B54's remaining case, end to end: a folder whose local root is still
/// *readable* but is no longer the tree it was — an empty directory left
/// where a mount used to be, or a different device mounted over it.
///
/// Nothing fails here. The scan succeeds and honestly reports an empty
/// directory, so the only thing standing between the user and a trashed
/// remote folder is the guard, and it has to fire for a one-file baseline
/// exactly as it does for a large one.
#[test]
fn a_replaced_mountpoint_trips_the_guard_at_every_baseline_size() {
let root = sync_test_dir("wipe-replaced-mount");
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut local: HashMap<String, LocalItem> = HashMap::new();
let mut progress = no_progress();
walk_local_tree(&root, &root, &rules, &writing, &mut local, &mut progress).unwrap();
assert!(local.is_empty(), "the replacement root is genuinely empty");
for size in 1..=3 {
let baseline: HashMap<String, ()> =
(0..size).map(|i| (format!("file{i}.txt"), ())).collect();
assert!(
guard_local_wipe(&filter_baseline(&baseline, &rules), &local).is_err(),
"a {size}-path baseline against an empty root is not a deletion"
);
}
// And the guard steps aside the moment the real tree is back, so a
// remounted folder resumes rather than staying wedged.
std::fs::write(root.join("file0.txt"), b"back").unwrap();
let mut local: HashMap<String, LocalItem> = HashMap::new();
walk_local_tree(&root, &root, &rules, &writing, &mut local, &mut progress).unwrap();
let baseline: HashMap<String, ()> = [("file0.txt".to_string(), ())].into_iter().collect();
assert!(guard_local_wipe(&filter_baseline(&baseline, &rules), &local).is_ok());
std::fs::remove_dir_all(&root).unwrap();
}
/// A restored folder (features.md 5.2) starts with an empty baseline against
/// an empty local directory and a full remote. That must reconcile as
/// "download everything" rather than tripping the wipe guard — otherwise
@@ -1764,6 +2069,7 @@ mod tests {
remote_uid: Some("v~l".into()),
local_mtime: 0,
local_size: 0,
local_mtime_ns: None,
remote_rev: Some(mtime.to_string()),
remote_hash: Some(size.to_string()),
},
@@ -1801,6 +2107,7 @@ mod tests {
remote_uid: Some("v~l".into()),
local_mtime: 0,
local_size: 0,
local_mtime_ns: None,
remote_rev: None,
remote_hash: None,
},
@@ -1815,6 +2122,7 @@ mod tests {
remote_uid: Some("v~l".into()),
local_mtime: 10,
local_size: 20,
local_mtime_ns: None,
remote_rev: Some("1700".into()),
remote_hash: Some("4096".into()),
};
@@ -1828,6 +2136,7 @@ mod tests {
remote_uid: Some("v~l".into()),
local_mtime: 0,
local_size: 0,
local_mtime_ns: None,
remote_rev: None,
remote_hash: None,
};
@@ -1863,6 +2172,7 @@ mod tests {
uid: uid(),
mtime: 0,
size: 10,
local: None,
};
assert_eq!(op.kind(), ActivityKind::Download);
}
@@ -1918,6 +2228,204 @@ mod tests {
std::fs::remove_dir_all(dir).unwrap();
}
/// B49's remaining cases. Preservation is the step that makes a download
/// safe to publish, so every way it can fail has to leave the local file
/// where it was — the download is aborted, not the file replaced.
#[test]
fn conflict_preservation_failures_all_keep_the_local_file() {
use std::os::unix::fs::PermissionsExt as _;
// An unwritable directory: the conflict copy cannot be created at all.
let dir = sync_test_dir("conflict-eacces");
let source = dir.join("notes.txt");
std::fs::write(&source, b"the only copy").unwrap();
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o500)).unwrap();
let denied = preserve_conflict_copy(&source, 1700);
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)).unwrap();
assert!(denied.is_err(), "a read-only directory must fail the copy");
assert_eq!(
std::fs::read(&source).unwrap(),
b"the only copy",
"the local file survives byte-identical"
);
std::fs::remove_dir_all(&dir).unwrap();
// A source that vanished between planning and preservation. The pass
// must report it rather than publish the download over the gap.
let dir = sync_test_dir("conflict-enoent");
assert!(preserve_conflict_copy(&dir.join("gone.txt"), 1700).is_err());
std::fs::remove_dir_all(&dir).unwrap();
// A write error partway through the copy — induced by making the
// destination name a directory, so `create_new` fails as something
// other than `AlreadyExists`.
let dir = sync_test_dir("conflict-eisdir");
let source = dir.join("notes.txt");
std::fs::write(&source, b"still the only copy").unwrap();
std::fs::create_dir(conflict_path(&source, 1700)).unwrap();
// Suffix 0 is taken by a directory; `create_new` on it reports
// `AlreadyExists`, so preservation moves on to suffix 1 and succeeds.
// What must not happen is the source being removed without a copy.
let preserved = preserve_conflict_copy(&source, 1700).unwrap();
assert!(preserved.is_file());
assert_eq!(std::fs::read(&preserved).unwrap(), b"still the only copy");
assert!(!source.exists());
std::fs::remove_dir_all(&dir).unwrap();
}
/// A conflict copy keeps the bytes *and* the metadata that identifies them,
/// so the next pass uploads the user's version rather than a file that looks
/// freshly written.
#[test]
fn a_conflict_copy_is_byte_and_mtime_identical() {
use std::os::unix::fs::PermissionsExt as _;
let dir = sync_test_dir("conflict-identity");
let source = dir.join("notes.txt");
std::fs::write(&source, b"local edit that must survive").unwrap();
let before = std::fs::metadata(&source).unwrap();
let preserved = preserve_conflict_copy(&source, 1700).unwrap();
let after = std::fs::metadata(&preserved).unwrap();
assert_eq!(
std::fs::read(&preserved).unwrap(),
b"local edit that must survive"
);
assert_eq!(after.len(), before.len());
assert_eq!(after.modified().unwrap(), before.modified().unwrap());
assert_eq!(after.permissions().mode(), before.permissions().mode());
std::fs::remove_dir_all(&dir).unwrap();
}
// ---- B55: an incomplete local scan is never a set of deletions ---------
fn no_progress() -> impl FnMut(&str) {
|_: &str| {}
}
/// A directory the pass cannot read must fail the pass. Omitting it would
/// hand reconciliation a baseline-minus-subtree, which it reads as the user
/// having deleted that subtree and propagates to Drive (B55).
#[test]
fn an_unreadable_subdirectory_fails_the_scan_instead_of_shrinking_it() {
use std::os::unix::fs::PermissionsExt as _;
let root = sync_test_dir("scan-eacces");
std::fs::write(root.join("keep.txt"), b"a").unwrap();
let locked = root.join("locked");
std::fs::create_dir(&locked).unwrap();
std::fs::write(locked.join("inner.txt"), b"b").unwrap();
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o000)).unwrap();
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut out = HashMap::new();
let mut progress = no_progress();
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o700)).unwrap();
std::fs::remove_dir_all(&root).unwrap();
let error = result.expect_err("an unreadable directory must abort the pass");
assert!(
error.contains("locked"),
"the error should name the subtree it could not read: {error}"
);
}
/// The same rule one level up: if the sync root itself cannot be read, the
/// pass sees nothing, and "nothing" must never mean "delete everything".
#[test]
fn an_unreadable_root_fails_the_scan() {
use std::os::unix::fs::PermissionsExt as _;
let root = sync_test_dir("scan-root-eacces");
std::fs::write(root.join("keep.txt"), b"a").unwrap();
std::fs::set_permissions(&root, std::fs::Permissions::from_mode(0o000)).unwrap();
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut out = HashMap::new();
let mut progress = no_progress();
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
std::fs::set_permissions(&root, std::fs::Permissions::from_mode(0o700)).unwrap();
std::fs::remove_dir_all(&root).unwrap();
assert!(result.is_err());
assert!(out.is_empty());
}
/// A root that is gone entirely — an unmounted device folder, a removed
/// directory — is the sharpest version of the same case: every baseline path
/// is "missing", and the scan must refuse rather than report an empty tree.
#[test]
fn an_absent_root_fails_the_scan() {
let root = sync_test_dir("scan-absent");
std::fs::remove_dir_all(&root).unwrap();
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut out = HashMap::new();
let mut progress = no_progress();
assert!(walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress).is_err());
assert!(out.is_empty());
}
/// A name that is not valid UTF-8 cannot be keyed into the scan, so it
/// cannot be compared against the baseline either. Refusing the pass is the
/// only safe reading: skipping it would classify the file as deleted.
#[test]
fn a_non_utf8_name_fails_the_scan() {
use std::ffi::OsStr;
use std::os::unix::ffi::OsStrExt as _;
let root = sync_test_dir("scan-non-utf8");
std::fs::write(root.join(OsStr::from_bytes(b"bad\xffname")), b"x").unwrap();
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut out = HashMap::new();
let mut progress = no_progress();
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
std::fs::remove_dir_all(&root).unwrap();
let error = result.expect_err("a name the scan cannot key must abort the pass");
assert!(error.contains("UTF-8"), "{error}");
}
/// The complement: a scan that *can* read everything reports everything, so
/// the strictness above does not come at the cost of a working pass. A
/// symlink is skipped by design rather than by failure, and a dangling one
/// stats fine through `symlink_metadata`.
#[test]
fn a_readable_tree_scans_completely() {
let root = sync_test_dir("scan-complete");
std::fs::write(root.join("a.txt"), b"aa").unwrap();
std::fs::create_dir(root.join("sub")).unwrap();
std::fs::write(root.join("sub/b.txt"), b"bbb").unwrap();
std::os::unix::fs::symlink("nowhere", root.join("dangling")).unwrap();
let rules = rules_for(&[]);
let writing = HashSet::new();
let mut out = HashMap::new();
let mut seen: Vec<String> = Vec::new();
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut |rel| {
seen.push(rel.to_string())
});
std::fs::remove_dir_all(&root).unwrap();
result.unwrap();
let mut keys: Vec<&str> = out.keys().map(String::as_str).collect();
keys.sort_unstable();
assert_eq!(keys, vec!["a.txt", "sub", "sub/b.txt"]);
assert_eq!(out["a.txt"].size, 2);
assert_eq!(out["sub/b.txt"].size, 3);
assert!(out["sub"].is_dir);
assert_eq!(seen.len(), 3, "every reported entry ticks scan progress");
}
fn sync_test_dir(label: &str) -> PathBuf {
let nonce = SystemTime::now()
.duration_since(SystemTime::UNIX_EPOCH)
+52 -16
View File
@@ -32,17 +32,24 @@ pub(crate) enum SyncMsg {
ReconcileAll,
/// Re-read the folder set and adjust the filesystem watches (after add/remove).
Rewatch,
/// The daemon is tearing down: finish the current pass and end the engine
/// thread, so the mount can join it (bugs.md B44).
Stop,
}
/// Start the sync engine: a control thread that owns the filesystem watcher and
/// serialises reconcile passes, plus a periodic poll thread. The engine receives
/// on `rx`; senders live in [`Core::sync_tx`].
pub(crate) fn spawn(core: Core, rx: Receiver<SyncMsg>) {
if let Err(e) = std::thread::Builder::new()
pub(crate) fn spawn(core: Core, rx: Receiver<SyncMsg>) -> Option<std::thread::JoinHandle<()>> {
match std::thread::Builder::new()
.name("pdfs-sync".into())
.spawn(move || engine_loop(core, rx))
{
warn!(error = %e, "failed to start sync engine");
Ok(handle) => Some(handle),
Err(e) => {
warn!(error = %e, "failed to start sync engine");
None
}
}
}
@@ -74,11 +81,11 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
// Periodic remote poll.
{
let tx = core.sync_tx.clone();
let shutdown = core.shutdown.clone();
std::thread::Builder::new()
.name("pdfs-sync-poll".into())
.spawn(move || {
loop {
std::thread::sleep(POLL_INTERVAL);
while shutdown.sleep(POLL_INTERVAL) {
if tx.send(SyncMsg::ReconcileAll).is_err() {
break;
}
@@ -94,8 +101,12 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
let mut ids: HashSet<i64> = HashSet::new();
let mut all = false;
let mut do_rewatch = false;
classify(msg, &mut ids, &mut all, &mut do_rewatch);
settle(&rx, &mut ids, &mut all, &mut do_rewatch);
if !classify(msg, &mut ids, &mut all, &mut do_rewatch)
|| !settle(&rx, &mut ids, &mut all, &mut do_rewatch)
{
info!("sync engine stopping");
return;
}
if do_rewatch {
rewatch(&core, &mut watcher, &watched);
@@ -116,9 +127,9 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
fn poll_only_loop(core: &Core, rx: Receiver<SyncMsg>) {
{
let tx = core.sync_tx.clone();
let shutdown = core.shutdown.clone();
std::thread::spawn(move || {
loop {
std::thread::sleep(POLL_INTERVAL);
while shutdown.sleep(POLL_INTERVAL) {
if tx.send(SyncMsg::ReconcileAll).is_err() {
break;
}
@@ -130,8 +141,12 @@ fn poll_only_loop(core: &Core, rx: Receiver<SyncMsg>) {
let mut ids: HashSet<i64> = HashSet::new();
let mut all = false;
let mut do_rewatch = false;
classify(msg, &mut ids, &mut all, &mut do_rewatch);
settle(&rx, &mut ids, &mut all, &mut do_rewatch);
if !classify(msg, &mut ids, &mut all, &mut do_rewatch)
|| !settle(&rx, &mut ids, &mut all, &mut do_rewatch)
{
info!("sync engine stopping");
return;
}
if all {
reconcile_all(core);
} else {
@@ -158,7 +173,7 @@ pub(super) fn settle_with(
do_rewatch: &mut bool,
quiet: Duration,
cap: Duration,
) {
) -> bool {
let deadline = Instant::now() + cap;
loop {
let now = Instant::now();
@@ -170,28 +185,49 @@ pub(super) fn settle_with(
match rx.recv_timeout(wait) {
// Another event: the burst is still going, so the quiet window
// starts again from here.
Ok(m) => classify(m, ids, all, do_rewatch),
// A stop lands mid-burst: give up the rest of the burst rather
// than settling for it, and let the caller end the loop.
Ok(m) => {
if !classify(m, ids, all, do_rewatch) {
return false;
}
}
// Quiet for a full window: whatever was being written has stopped.
Err(RecvTimeoutError::Timeout) => break,
// Every sender is gone; the caller's loop is about to end anyway.
Err(RecvTimeoutError::Disconnected) => break,
}
}
true
}
/// [`settle_with`] at the real timings.
fn settle(rx: &Receiver<SyncMsg>, ids: &mut HashSet<i64>, all: &mut bool, do_rewatch: &mut bool) {
settle_with(rx, ids, all, do_rewatch, DEBOUNCE, MAX_SETTLE);
fn settle(
rx: &Receiver<SyncMsg>,
ids: &mut HashSet<i64>,
all: &mut bool,
do_rewatch: &mut bool,
) -> bool {
settle_with(rx, ids, all, do_rewatch, DEBOUNCE, MAX_SETTLE)
}
pub(super) fn classify(msg: SyncMsg, ids: &mut HashSet<i64>, all: &mut bool, rewatch: &mut bool) {
/// Fold one message into the pending work of the current burst. Returns `false`
/// for [`SyncMsg::Stop`], which is not work but an instruction to end the loop.
pub(super) fn classify(
msg: SyncMsg,
ids: &mut HashSet<i64>,
all: &mut bool,
rewatch: &mut bool,
) -> bool {
match msg {
SyncMsg::Reconcile(id) => {
ids.insert(id);
}
SyncMsg::ReconcileAll => *all = true,
SyncMsg::Rewatch => *rewatch = true,
SyncMsg::Stop => return false,
}
true
}
/// Reconcile every mirror folder in turn.
+71
View File
@@ -2,9 +2,73 @@
use super::*;
/// What a reconcile pass remembers about the local side of one path: enough to
/// recognise the same content again, and no more.
///
/// `mtime_ns` is the whole reason this is a struct rather than a pair — see
/// [`StoredSyncEntry::local_mtime_ns`](pdfs_core::db::StoredSyncEntry).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(super) struct LocalSig {
pub(super) mtime: i64,
pub(super) mtime_ns: Option<i64>,
pub(super) size: i64,
}
impl LocalSig {
/// Whether these two observations describe the same local content.
///
/// Nanoseconds decide it when both sides have them; otherwise the comparison
/// falls back to whole seconds, which is what a baseline row written before
/// schema 28 — or a filesystem that reports no sub-second time — can support
/// (bugs.md B25).
pub(super) fn same_content(&self, other: &Self) -> bool {
if self.size != other.size {
return false;
}
match (self.mtime_ns, other.mtime_ns) {
(Some(a), Some(b)) => a == b,
_ => self.mtime == other.mtime,
}
}
}
impl From<&LocalItem> for LocalSig {
fn from(item: &LocalItem) -> Self {
Self {
mtime: item.mtime,
mtime_ns: item.mtime_ns,
size: item.size,
}
}
}
impl From<&std::fs::Metadata> for LocalSig {
fn from(meta: &std::fs::Metadata) -> Self {
Self {
mtime: system_mtime(meta),
mtime_ns: system_mtime_ns(meta),
size: meta.len() as i64,
}
}
}
/// The baseline's record of the local side, as [`LocalSig`] compares it.
impl From<&StoredSyncEntry> for LocalSig {
fn from(entry: &StoredSyncEntry) -> Self {
Self {
mtime: entry.local_mtime,
mtime_ns: entry.local_mtime_ns,
size: entry.local_size,
}
}
}
pub(super) struct LocalItem {
pub(super) is_dir: bool,
pub(super) mtime: i64,
/// [`LocalItem::mtime`] to nanosecond precision, when the filesystem
/// reports one.
pub(super) mtime_ns: Option<i64>,
pub(super) size: i64,
/// True when another process holds this file open for writing (detected via
/// `/proc/*/fd`). An open-for-write file is kept in the map — so it is not
@@ -91,11 +155,18 @@ pub(super) enum Pending {
UploadRevision { rel: String, uid: NodeUid },
/// Download remote `uid` to the local path, stamping `mtime`. `size` is the
/// remote's reported size, used as the transfer's expected total.
///
/// `local` is what the planning walk saw at this path, or `None` if nothing
/// was there. A download can run for minutes, so the apply step revalidates
/// against it immediately before it renames over the destination: a writer
/// that finished an edit in the meantime must not be overwritten without a
/// conflict copy (bugs.md B39).
Download {
rel: String,
uid: NodeUid,
mtime: i64,
size: i64,
local: Option<LocalSig>,
},
/// Both sides changed: set the local copy aside, then download remote `uid`.
Conflict {
+42 -1
View File
@@ -218,7 +218,10 @@ pub(super) fn plan_file(
}
let local_changed = local.is_some_and(|item| {
baseline.is_none_or(|base| base.local_mtime != item.mtime || base.local_size != item.size)
// Nanoseconds where both sides have them (bugs.md B25): whole seconds
// could not see an edit made in the same second as the last sync that
// left the file the same length, so it was never uploaded.
baseline.is_none_or(|base| !LocalSig::from(item).same_content(&LocalSig::from(base)))
});
let remote_changed = remote.is_some_and(|item| {
baseline.is_none_or(|base| remote_sig(base) != Some((item.mtime, item.size)))
@@ -252,6 +255,7 @@ mod file_plan_tests {
fn local(mtime: i64, size: i64) -> LocalItem {
LocalItem {
mtime_ns: None,
is_dir: false,
mtime,
size,
@@ -278,12 +282,49 @@ mod file_plan_tests {
rel_path: "file".into(),
remote_uid: Some("v~l".into()),
local_mtime,
local_mtime_ns: None,
local_size,
remote_rev: Some(remote_mtime.to_string()),
remote_hash: Some(remote_size.to_string()),
}
}
/// Whole seconds cannot see an edit made inside the same second that leaves the
/// file the same length, so a mirror silently stopped uploading it. With a
/// nanosecond time on both sides it is an ordinary local change (bugs.md B25).
#[test]
fn a_same_second_same_size_edit_is_still_a_local_change() {
let mut base = baseline(10, 20, 30, 40);
base.local_mtime_ns = Some(10_000_000_000);
let edited = LocalItem {
mtime_ns: Some(10_400_000_000),
..local(10, 20)
};
let remote = remote(30, 40);
assert_eq!(
plan_file(Some(&edited), Some(&remote), Some(&base)),
FilePlan::UploadRevision
);
// The same file untouched still reads as unchanged.
let same = LocalItem {
mtime_ns: Some(10_000_000_000),
..local(10, 20)
};
assert_eq!(
plan_file(Some(&same), Some(&remote), Some(&base)),
FilePlan::Unchanged
);
// And a row from before the column exists compares on seconds, exactly
// as it did when it was written — no mass re-upload on upgrade.
let old = baseline(10, 20, 30, 40);
assert_eq!(
plan_file(Some(&edited), Some(&remote), Some(&old)),
FilePlan::Unchanged
);
}
#[test]
fn file_plan_covers_every_presence_and_change_shape() {
let same_local = local(10, 20);
+632
View File
@@ -0,0 +1,632 @@
//! Importing a Google Photos Takeout export into Proton Photos.
//!
//! [`pdfs_core::takeout`] says what is in the archives; this is the part that
//! talks to Proton. One import runs on its own thread and moves through four
//! phases, reporting into the transfer registry as a job so the GUI's transfer
//! list shows it without a bespoke widget:
//!
//! 1. **Scan** — read the archives' central directories and sidecars.
//! 2. **Deduplicate against the account** — ask the photos volume which of the
//! names already exist ([`ProtonPhotosClient::name_collisions`]), hash only
//! those files, then confirm with the content digest. A library that is
//! already half-synced by Proton Photos on a phone therefore costs one cheap
//! request per 150 names and reads no media at all for the misses.
//! 3. **Upload** the survivors, streaming each entry straight out of its zip —
//! no extraction to disk, so the import needs no scratch space.
//! 4. **Albums** — create the albums the export names that the account lacks,
//! then add every photo of each album, the ones that were already on the
//! account included.
//!
//! Re-running an interrupted import is the resume story: phase 2 finds
//! everything the previous run uploaded and skips it. Nothing is persisted
//! between runs.
use std::collections::{BTreeMap, HashMap};
use std::io::Read;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use pdfs_core::control::{ActivityKind, ImportSummary, TransferDirection};
use pdfs_core::takeout::{Bucket, TakeoutPhoto, TakeoutScan, scan_archives};
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::proton_sdk::ids::NodeUid;
use proton_drive_rs::{PhotoTag, PhotoTagsUpdate, PhotoUploadMetadata};
use sha1::{Digest, Sha1};
use tracing::{info, warn};
use super::Core;
use crate::transfers::CountingReader;
/// The one import this daemon may be running, and how the last one ended.
///
/// Process-global rather than a [`Core`] field because it must be *one* import
/// per daemon however many `Core` clones exist (the forked on-demand mounts each
/// hold one), and because a front-end asking for the result after the import
/// finished must still find it.
static IMPORT: std::sync::LazyLock<parking_lot::Mutex<ImportState>> =
std::sync::LazyLock::new(|| parking_lot::Mutex::new(ImportState::default()));
#[derive(Default)]
struct ImportState {
running: bool,
cancel: Option<Arc<AtomicBool>>,
last: Option<ImportSummary>,
}
/// Whether an import is running, and the last one's report.
pub(crate) fn import_status() -> (bool, Option<ImportSummary>) {
let state = IMPORT.lock();
(state.running, state.last.clone())
}
/// Ask the running import to stop. False when none is running.
pub(crate) fn cancel_import() -> bool {
match &IMPORT.lock().cancel {
Some(flag) => {
flag.store(true, Ordering::Relaxed);
true
}
None => false,
}
}
/// One photo to import, after the copies Google keeps per album are folded into
/// a single upload that joins several albums.
struct Planned {
photo: TakeoutPhoto,
/// Every album this photo belongs to, by export title.
albums: Vec<String>,
/// Filled in phase 2/3: where the photo ended up on the account.
uid: Option<NodeUid>,
}
/// Fold the scan into one entry per distinct photo.
///
/// Google stores a full copy of a photo in every album folder it appears in, so
/// the same file is in the export two or three times over. They are folded on
/// `(name, size)`: a byte-identical copy under the same name is the same photo,
/// and the album memberships of the copies are unioned onto the survivor. Two
/// genuinely different photos sharing both a name and an exact size would be
/// merged — vanishingly unlikely, and the cost of being wrong is one photo
/// filed under an extra album rather than lost data.
fn plan(scan: &TakeoutScan) -> Vec<Planned> {
let mut order: Vec<(String, u64)> = Vec::new();
let mut by_key: HashMap<(String, u64), Planned> = HashMap::new();
for photo in &scan.photos {
let key = (photo.name.clone(), photo.size);
let entry = by_key.entry(key.clone()).or_insert_with(|| {
order.push(key.clone());
Planned {
photo: photo.clone(),
albums: Vec::new(),
uid: None,
}
});
if let Bucket::Album(title) = &photo.bucket
&& !entry.albums.iter().any(|existing| existing == title)
{
entry.albums.push(title.clone());
}
// A copy that carries a capture time wins over one that does not: the
// album copy and the year-folder copy do not always both have a sidecar.
if entry.photo.capture_time.is_none() && photo.capture_time.is_some() {
entry.photo.capture_time = photo.capture_time;
}
entry.photo.favorite |= photo.favorite;
}
order
.into_iter()
.filter_map(|key| by_key.remove(&key))
.collect()
}
/// Open one archive and stream `entry` through `sink`.
fn read_entry<R>(
archive: &mut zip::ZipArchive<std::io::BufReader<std::fs::File>>,
entry: &str,
sink: impl FnOnce(&mut (dyn Read + Send)) -> R,
) -> CoreResult<R> {
let mut file = archive
.by_name(entry)
.map_err(|e| CoreError::invalid(format!("{entry}: {e}")))?;
Ok(sink(&mut file))
}
impl Core {
/// Import Google Photos Takeout archives. Blocking; run on its own thread.
///
/// `cancel` stops the run at the next photo boundary — an upload already on
/// the wire is allowed to finish, so a cancelled import never leaves a
/// half-written revision behind.
pub(crate) fn import_takeout(
&self,
archives: Vec<PathBuf>,
dry_run: bool,
cancel: Arc<AtomicBool>,
) -> CoreResult<ImportSummary> {
let job = self.transfers.begin_job("Importing Google Photos");
job.detail("reading archives");
let scan = scan_archives(&archives)?;
let mut planned = plan(&scan);
let mut summary = ImportSummary {
found: planned.len(),
skipped_trashed: scan.skipped_trashed,
..Default::default()
};
job.set_total(planned.len() as u64);
info!(
archives = archives.len(),
photos = summary.found,
albums = scan.albums.len(),
trashed = summary.skipped_trashed,
"takeout scan complete"
);
let photos = self.photos();
if self
.rt
.block_on(photos.get_photos_root())
.map_err(|e| CoreError::from_api(&e, "resolve photos library"))?
.is_none()
{
return Err(CoreError::invalid(
"this account has no Proton Photos library yet; open Proton Photos once to create it, then import again",
));
}
// Phase 2: which of these are already on the account.
job.detail("checking for photos you already have");
self.mark_existing(&mut planned, &archives, &mut summary, &cancel)?;
if dry_run {
summary.duplicates = planned.iter().filter(|p| p.uid.is_some()).count();
return Ok(summary);
}
// Phase 3: upload what is left, one archive at a time so each zip's
// central directory is parsed once.
self.upload_planned(&mut planned, &archives, &mut summary, &cancel, &job)?;
// Phase 4: albums. Attempted even after a cancel — the photos that did
// upload should still land in their albums.
if !scan.albums.is_empty() {
job.detail("sorting photos into albums");
if let Err(e) = self.link_albums(&planned, &scan.albums, &mut summary) {
warn!(error = %e, "takeout album linking failed");
}
}
self.spawn_timeline_refresh();
summary.cancelled = cancel.load(Ordering::Relaxed);
Ok(summary)
}
/// Fill in `uid` for every planned photo the account already holds.
///
/// Two passes on purpose: the name-only query is one request per 150 photos
/// and needs no file contents, so the expensive SHA-1 pass only runs for the
/// handful of names that actually collide.
fn mark_existing(
&self,
planned: &mut [Planned],
archives: &[PathBuf],
summary: &mut ImportSummary,
cancel: &AtomicBool,
) -> CoreResult<()> {
let photos = self.photos();
let names: Vec<String> = planned.iter().map(|p| p.photo.name.clone()).collect();
let collisions = self
.rt
.block_on(photos.name_collisions(&names))
.map_err(|e| CoreError::from_api(&e, "check for existing photos"))?;
// Hash the colliding files, grouped by archive so each zip opens once.
let mut by_archive: BTreeMap<usize, Vec<usize>> = BTreeMap::new();
for (index, planned) in planned.iter().enumerate() {
if collisions.get(index).copied().unwrap_or(false) {
by_archive
.entry(planned.photo.archive)
.or_default()
.push(index);
}
}
// Kept as a pair list, not a map: the duplicate lookup answers in input
// order, so the order these were built in is what maps answers back to
// planned photos.
let mut digests: Vec<(usize, String)> = Vec::new();
for (archive_index, indices) in by_archive {
if cancel.load(Ordering::Relaxed) {
break;
}
let mut archive = open_archive(archives, archive_index)?;
for index in indices {
let entry = planned[index].photo.entry.clone();
match read_entry(&mut archive, &entry, sha1_of) {
Ok(Ok(digest)) => digests.push((index, digest)),
Ok(Err(e)) | Err(e) => {
warn!(entry, error = %e, "cannot hash takeout entry");
}
}
}
}
if digests.is_empty() {
return Ok(());
}
let items: Vec<(String, String)> = digests
.iter()
.map(|(index, digest)| (planned[*index].photo.name.clone(), digest.clone()))
.collect();
let matches = self
.rt
.block_on(photos.find_duplicates_many(&items))
.map_err(|e| CoreError::from_api(&e, "check for existing photos"))?;
for (slot, (index, _)) in digests.iter().enumerate() {
if let Some(uid) = matches.get(slot).and_then(|uids| uids.first()) {
planned[*index].uid = Some(uid.clone());
summary.duplicates += 1;
}
}
Ok(())
}
/// Upload every planned photo that is not already on the account.
fn upload_planned(
&self,
planned: &mut [Planned],
archives: &[PathBuf],
summary: &mut ImportSummary,
cancel: &AtomicBool,
job: &crate::transfers::JobGuard,
) -> CoreResult<()> {
let photos = self.photos();
let mut favorites: Vec<NodeUid> = Vec::new();
let mut by_archive: BTreeMap<usize, Vec<usize>> = BTreeMap::new();
for (index, item) in planned.iter().enumerate() {
if item.uid.is_none() {
by_archive
.entry(item.photo.archive)
.or_default()
.push(index);
}
}
for (archive_index, indices) in by_archive {
if cancel.load(Ordering::Relaxed) || self.shutdown.is_stopping() {
break;
}
let mut archive = open_archive(archives, archive_index)?;
for index in indices {
if cancel.load(Ordering::Relaxed) || self.shutdown.is_stopping() {
break;
}
let photo = planned[index].photo.clone();
job.detail(photo.name.clone());
let guard = self.transfers.begin(
photo.name.clone(),
"",
TransferDirection::Upload,
photo.size,
);
let metadata = PhotoUploadMetadata {
capture_time: photo.capture_time,
..Default::default()
};
let uploaded = read_entry(&mut archive, &photo.entry, |reader| {
let counting = CountingReader::new(reader, &guard);
self.rt.block_on(photos.upload_photo_from(
&photo.name,
&photo.media_type,
counting,
photo.size as i64,
Vec::new(),
metadata,
false,
))
});
drop(guard);
job.step();
match uploaded {
Ok(Ok(uid)) => {
summary.uploaded += 1;
summary.bytes += photo.size;
if photo.favorite {
favorites.push(uid.clone());
}
planned[index].uid = Some(uid);
}
Ok(Err(e)) => {
summary.failed += 1;
warn!(name = photo.name, error = %e, "takeout photo upload failed");
}
Err(e) => {
summary.failed += 1;
warn!(name = photo.name, error = %e, "cannot read takeout entry");
}
}
}
}
// Favourites are a separate call: the upload seal takes classification
// tags, but `Favorite` is not one of them — it goes through the photos
// volume's dedicated favourite endpoint.
if !favorites.is_empty() {
let updates: Vec<PhotoTagsUpdate> = favorites
.into_iter()
.map(|node_uid| PhotoTagsUpdate {
node_uid,
tags_to_add: vec![PhotoTag::Favorite],
tags_to_remove: Vec::new(),
})
.collect();
if let Err(e) = self.rt.block_on(photos.update_photos(&updates)) {
warn!(error = %e, "marking imported favourites failed");
}
}
Ok(())
}
/// Create the albums the export names that the account lacks, then add each
/// album's photos to it.
fn link_albums(
&self,
planned: &[Planned],
albums: &[String],
summary: &mut ImportSummary,
) -> CoreResult<()> {
let photos = self.photos();
// Existing albums by decrypted name, so a second import of the same
// export adds to the album it made the first time instead of a twin.
let mut existing: HashMap<String, NodeUid> = HashMap::new();
let uids = self
.rt
.block_on(photos.enumerate_album_node_uids())
.map_err(|e| CoreError::from_api(&e, "list albums"))?;
if !uids.is_empty() {
let albums_nodes = self
.rt
.block_on(photos.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "read albums"))?;
for node in albums_nodes {
existing.entry(node.name.clone()).or_insert(node.uid);
}
}
for title in albums {
let members: Vec<NodeUid> = planned
.iter()
.filter(|item| item.albums.iter().any(|album| album == title))
.filter_map(|item| item.uid.clone())
.collect();
if members.is_empty() {
continue;
}
let album_uid = match existing.get(title) {
Some(uid) => uid.clone(),
None => match self.rt.block_on(photos.create_album(title)) {
Ok(uid) => {
summary.albums_created += 1;
existing.insert(title.clone(), uid.clone());
uid
}
Err(e) => {
warn!(album = title, error = %e, "cannot create album");
continue;
}
},
};
match self
.rt
.block_on(photos.add_photos_to_album(&album_uid, &members))
{
Ok(outcomes) => {
for (uid, outcome) in outcomes {
match outcome {
Ok(()) => summary.album_links += 1,
// A photo already in the album is the expected
// outcome of a re-run, not a failure worth counting.
Err(e) => {
warn!(album = title, %uid, error = %e, "cannot add photo to album")
}
}
}
}
Err(e) => warn!(album = title, error = %e, "album addition failed"),
}
}
Ok(())
}
/// Run an import on a background thread, logging its outcome to the activity
/// feed. Returns immediately.
///
/// Refuses while one is already running: two imports of the same export
/// would race each other's duplicate detection and upload everything twice.
pub(crate) fn spawn_takeout_import(
&self,
archives: Vec<PathBuf>,
dry_run: bool,
) -> CoreResult<()> {
let cancel = Arc::new(AtomicBool::new(false));
{
let mut state = IMPORT.lock();
if state.running {
return Err(CoreError::conflict(
"an import is already running; wait for it to finish or cancel it",
));
}
state.running = true;
state.cancel = Some(cancel.clone());
state.last = None;
}
let core = self.clone();
std::thread::spawn(move || {
let outcome = core.import_takeout(archives, dry_run, cancel);
{
let mut state = IMPORT.lock();
state.running = false;
state.cancel = None;
state.last = outcome.as_ref().ok().cloned();
}
match outcome {
Ok(summary) => {
let mut parts = vec![format!("{} uploaded", summary.uploaded)];
if summary.duplicates > 0 {
parts.push(format!("{} already there", summary.duplicates));
}
if summary.albums_created > 0 {
parts.push(format!("{} albums", summary.albums_created));
}
if summary.failed > 0 {
parts.push(format!("{} failed", summary.failed));
}
if summary.cancelled {
parts.push("cancelled".to_string());
}
core.log_activity(
ActivityKind::Upload,
"Google Photos import",
parts.join(" · "),
summary.failed == 0 && !summary.cancelled,
);
}
Err(e) => {
warn!(error = %e, "takeout import failed");
core.log_activity(ActivityKind::Upload, "Google Photos import", &e, false);
}
}
});
Ok(())
}
}
/// Open one of the import's archives by index.
fn open_archive(
archives: &[PathBuf],
index: usize,
) -> CoreResult<zip::ZipArchive<std::io::BufReader<std::fs::File>>> {
let path = archives
.get(index)
.ok_or_else(|| CoreError::internal(format!("archive {index} is not in the import")))?;
let file = std::fs::File::open(path)
.map_err(|e| CoreError::invalid(format!("cannot open {}: {e}", path.display())))?;
zip::ZipArchive::new(std::io::BufReader::new(file))
.map_err(|e| CoreError::invalid(format!("{} is not a readable zip: {e}", path.display())))
}
/// The lowercase-hex SHA-1 of everything `reader` yields — the digest Proton's
/// photo duplicate detection is keyed on.
fn sha1_of(reader: &mut (dyn Read + Send)) -> CoreResult<String> {
let mut hasher = Sha1::new();
let mut buffer = vec![0u8; 256 * 1024];
loop {
let read = reader
.read(&mut buffer)
.map_err(|e| CoreError::invalid(format!("read failed: {e}")))?;
if read == 0 {
break;
}
hasher.update(&buffer[..read]);
}
Ok(hex_lower(&hasher.finalize()))
}
fn hex_lower(bytes: &[u8]) -> String {
let mut out = String::with_capacity(bytes.len() * 2);
for byte in bytes {
out.push_str(&format!("{byte:02x}"));
}
out
}
#[cfg(test)]
mod tests {
use super::*;
use pdfs_core::takeout::TakeoutPhoto;
fn photo(name: &str, size: u64, bucket: Bucket) -> TakeoutPhoto {
TakeoutPhoto {
archive: 0,
entry: format!("Takeout/Google Photos/x/{name}"),
name: name.to_string(),
size,
media_type: "image/jpeg".to_string(),
capture_time: None,
favorite: false,
bucket,
}
}
#[test]
fn album_copies_of_one_photo_fold_into_a_single_upload() {
let scan = TakeoutScan {
photos: vec![
photo("IMG_1.jpg", 100, Bucket::Timeline),
photo("IMG_1.jpg", 100, Bucket::Album("Iceland".into())),
photo("IMG_1.jpg", 100, Bucket::Album("Best of".into())),
photo("IMG_2.jpg", 200, Bucket::Timeline),
],
albums: vec!["Iceland".into(), "Best of".into()],
..Default::default()
};
let planned = plan(&scan);
assert_eq!(planned.len(), 2, "the three copies are one upload");
assert_eq!(planned[0].albums, vec!["Iceland", "Best of"]);
assert!(planned[1].albums.is_empty());
}
#[test]
fn same_name_different_size_stays_two_photos() {
let scan = TakeoutScan {
photos: vec![
photo("IMG_1.jpg", 100, Bucket::Timeline),
photo("IMG_1.jpg", 101, Bucket::Album("Iceland".into())),
],
..Default::default()
};
assert_eq!(plan(&scan).len(), 2);
}
#[test]
fn metadata_from_any_copy_survives_the_fold() {
// Only the album copy carries a sidecar; the fold must not lose it.
let mut bare = photo("IMG_1.jpg", 100, Bucket::Timeline);
bare.capture_time = None;
let mut rich = photo("IMG_1.jpg", 100, Bucket::Album("Iceland".into()));
rich.capture_time = Some(1_560_000_000);
rich.favorite = true;
let planned = plan(&TakeoutScan {
photos: vec![bare, rich],
..Default::default()
});
assert_eq!(planned[0].photo.capture_time, Some(1_560_000_000));
assert!(planned[0].photo.favorite);
}
#[test]
fn sha1_matches_the_known_digest_of_abc() {
let mut reader: &[u8] = b"abc";
assert_eq!(
sha1_of(&mut reader).unwrap(),
"a9993e364706816aba3e25717850c26c9cd0d89d"
);
}
}
+33 -3
View File
@@ -19,7 +19,7 @@ use parking_lot::Mutex;
use std::collections::HashMap;
use std::io::{Read, Write};
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::time::Instant;
use pdfs_core::control::{JobItem, TransferDirection, TransferItem};
@@ -272,20 +272,50 @@ impl<R: Read> Read for OwnedCountingReader<R> {
}
}
/// A [`Read`] that tallies bytes read through a [`TransferGuard`] (upload path).
/// A [`Read`] that tallies bytes read through a [`TransferGuard`] (upload path),
/// and can be told to stop.
pub struct CountingReader<'a, R> {
inner: R,
guard: &'a TransferGuard,
cancel: Option<Arc<AtomicBool>>,
}
impl<'a, R: Read> CountingReader<'a, R> {
pub fn new(inner: R, guard: &'a TransferGuard) -> Self {
Self { inner, guard }
Self {
inner,
guard,
cancel: None,
}
}
/// Abandon the upload as soon as the SDK asks for its next bytes when
/// `cancel` is set.
///
/// This is the only cancellation point an upload has: the SDK takes a plain
/// blocking [`Read`] and pulls from it a block at a time, so refusing to
/// hand over the next block is what stops it. Used when a newer revision of
/// the same file is queued — the bytes on the wire have already been
/// superseded, and finishing them only spends uplink on a revision the next
/// op replaces.
pub fn with_cancel(mut self, cancel: Arc<AtomicBool>) -> Self {
self.cancel = Some(cancel);
self
}
}
impl<R: Read> Read for CountingReader<'_, R> {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
if let Some(cancel) = &self.cancel
&& cancel.load(Ordering::Relaxed)
{
// Not `Ok(0)`: a short read is how this reader says "end of file",
// and the SDK would seal a truncated revision from it.
return Err(std::io::Error::new(
std::io::ErrorKind::Interrupted,
"upload superseded by a newer write",
));
}
let n = self.inner.read(buf)?;
self.guard.add(n as u64);
Ok(n)
+3
View File
@@ -5,6 +5,9 @@ edition.workspace = true
license.workspace = true
authors.workspace = true
rust-version.workspace = true
repository.workspace = true
homepage.workspace = true
description = "GTK4 desktop app, tray, and search launcher for the Proton Drive Linux client"
[[bin]]
name = "pdfs-tray"
+3
View File
@@ -22,6 +22,9 @@ fn main() {
}
println!("cargo:rerun-if-changed=resources/pdfs.gresource.xml");
println!(
"cargo:rerun-if-changed=resources/icons/scalable/actions/pdfs-build-thumbnails-symbolic.svg"
);
println!(
"cargo:rerun-if-changed=resources/icons/scalable/actions/pdfs-folder-upload-symbolic.svg"
);
@@ -0,0 +1,5 @@
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16">
<!-- Generate-thumbnails action: a picture frame with a sparkle. A single
symbolic fill lets GTK recolour it with the surrounding control. -->
<path fill="#222222" d="M 2,3 C 1.45,3 1,3.45 1,4 V 14 C 1,14.55 1.45,15 2,15 H 14 C 14.55,15 15,14.55 15,14 V 7 H 13 V 13 H 3 V 5 H 9 V 3 Z M 5,6 C 4.45,6 4,6.45 4,7 C 4,7.55 4.45,8 5,8 C 5.55,8 6,7.55 6,7 C 6,6.45 5.55,6 5,6 Z M 10,7 L 7.5,10 L 6,9 L 4,11.5 V 12 H 12 V 9 Z M 13,0 L 12.4,2.4 L 10,3 L 12.4,3.6 L 13,6 L 13.6,3.6 L 16,3 L 13.6,2.4 Z"/>
</svg>

After

Width:  |  Height:  |  Size: 598 B

@@ -4,6 +4,7 @@
icon-theme resource path at startup, so they resolve by name (and recolour
with the theme) exactly like a stock symbolic icon. -->
<gresource prefix="/de/nils/protondrivelinux/icons">
<file preprocess="xml-stripblanks">scalable/actions/pdfs-build-thumbnails-symbolic.svg</file>
<file preprocess="xml-stripblanks">scalable/actions/pdfs-folder-upload-symbolic.svg</file>
</gresource>
</gresources>
+151 -15
View File
@@ -14,10 +14,12 @@ use pages::photos::*;
use pages::shared::*;
use pages::shared_by_me::*;
use pages::status::*;
use pages::takeout::*;
use pages::trash::*;
use pages::verify::*;
use widgets::details::*;
use widgets::share_dialog::*;
use widgets::thumbnails::*;
use widgets::versions_dialog::*;
use std::cell::{Cell, RefCell};
@@ -46,10 +48,10 @@ use pdfs_core::config::AppDirs;
use pdfs_core::control::{
ActivityEntry, ActivityKind, AlbumInfo, BookmarkInfo, DeviceInfo, DirEntry, ErrorKind,
InvitationInfo, JobItem, PhotoItem, PhotoKind, PublicLinkInfo, RefreshScope, Request, Response,
RestorableFolder, RestoreItem, SearchHit, ShareEntry, ShareEntryKind, SharedItem,
SyncFolderInfo, SyncPhase, SyncProgress, TransferDirection, TransferItem, pending_summary,
send,
ImportSummary, InvitationInfo, JobItem, PhotoItem, PhotoKind, PublicLinkInfo, RefreshScope,
Request, Response, RestorableFolder, RestoreItem, SearchHit, ShareEntry, ShareEntryKind,
SharedItem, SyncFolderInfo, SyncPhase, SyncProgress, ThumbnailBuildStatus, TransferDirection,
TransferItem, pending_summary, send,
};
use pdfs_core::mounts::{MountAccess, MountKind, MountMode, MountSpec};
@@ -106,6 +108,10 @@ struct Ui {
/// The sidebar/content split. Collapsed while signed out, so the login page
/// owns the whole window and no destination is reachable without a session.
nav: adw::NavigationSplitView,
/// Shared thumbnails for ordinary image files outside the Photos gallery.
/// One cache and request queue serves Files, search, Shared and Trash, so
/// the same image is downloaded and decoded only once.
pub(crate) file_thumbs: FileThumbnailState,
// Per-page state. Each page module owns its own struct; `Ui` keeps only
// what more than one page genuinely shares.
@@ -120,6 +126,7 @@ struct Ui {
pub(crate) devices: DevicesState,
pub(crate) locations: LocationsState,
pub(crate) activity: ActivityState,
pub(crate) takeout: TakeoutState,
}
impl Ui {
@@ -205,10 +212,15 @@ fn load_proton_theme() {
.file-grid {{ padding: 6px; }}\n\
.file-tile {{ padding: 8px; border-radius: 10px; }}\n\
.file-tile:hover {{ background: alpha({PROTON_PURPLE}, 0.10); }}\n\
.file-thumbnail {{ border-radius: 7px; }}\n\
.file-badge {{ -gtk-icon-shadow: 0 1px 2px rgba(0, 0, 0, 0.5); }}\n\
.badge-pinned {{ color: #f5c211; }}\n\
.badge-cached {{ color: #2ec27e; }}\n\
.badge-cloud {{ color: #9aa0a6; }}\n\
.browser-statusbar {{ background-color: alpha(currentColor, 0.025); }}\n\
scale.browser-status-meter trough, progressbar.browser-status-meter trough {{ min-width: 104px; }}\n\
scale.browser-status-meter trough {{ min-height: 6px; }}\n\
progressbar.browser-status-meter trough, progressbar.browser-status-meter progress {{ min-height: 6px; }}\n\
.photo-viewer-window {{ background-color: #111014; }}\n\
.viewer-top-bar {{ background: linear-gradient(to bottom, rgba(0, 0, 0, 0.75), rgba(0, 0, 0, 0)); padding: 10px 16px 28px 20px; color: white; }}\n\
.viewer-title {{ font-weight: 700; font-size: 1.05rem; text-shadow: 0 1px 3px rgba(0, 0, 0, 0.9); color: white; }}\n\
@@ -246,7 +258,10 @@ fn load_proton_theme() {
.navigation-sidebar row {{ border-radius: 8px; margin: 2px 6px; }}\n\
.navigation-sidebar row:selected {{ background: alpha({PROTON_PURPLE}, 0.16); color: {PROTON_PURPLE}; font-weight: 600; }}\n\
.navigation-sidebar row:selected image {{ color: {PROTON_PURPLE}; }}\n\
.file-tile:selected, .file-tile:hover:selected {{ background: alpha({PROTON_PURPLE}, 0.20); }}\n"
.file-tile:selected, .file-tile:hover:selected {{ background: alpha({PROTON_PURPLE}, 0.20); }}\n\
.bulk-bar {{ padding: 6px 8px; border-radius: 12px; background: alpha({PROTON_PURPLE}, 0.12); }}\n\
.dropzone {{ border: 2px dashed alpha(currentColor, 0.25); border-radius: 16px; padding: 28px 18px; transition: background 160ms ease, border-color 160ms ease; }}\n\
.dropzone-active {{ border-color: {PROTON_PURPLE}; background: alpha({PROTON_PURPLE}, 0.10); }}\n"
);
let provider = gtk4::CssProvider::new();
provider.load_from_string(&css);
@@ -285,6 +300,7 @@ fn build_window(app: &adw::Application) {
let (locations_page, locations_widgets) = build_locations_page();
let (activity_page, activity_widgets) = build_activity_page();
let (trash_page, trash_widgets) = build_trash_page();
let (takeout_page, takeout_widgets) = build_takeout_page();
stack.add_named(&login_page, Some("login"));
stack.add_named(&main_page, Some("main"));
stack.add_named(&browser_page, Some("browser"));
@@ -295,6 +311,7 @@ fn build_window(app: &adw::Application) {
stack.add_named(&locations_page, Some("locations"));
stack.add_named(&activity_page, Some("activity"));
stack.add_named(&trash_page, Some("trash"));
stack.add_named(&takeout_page, Some("takeout"));
// Sidebar: the signed-in destinations. Selecting a row swaps the page stack;
// `sync_sidebar` pushes the other way when navigation happens elsewhere (e.g.
@@ -338,6 +355,7 @@ fn build_window(app: &adw::Application) {
mounted: RefCell::new(false),
sidebar: sidebar_list.clone(),
nav: split.clone(),
file_thumbs: FileThumbnailState::new(),
login: LoginState {
email: login_widgets.0,
password: login_widgets.1,
@@ -362,6 +380,8 @@ fn build_window(app: &adw::Application) {
budget_row: main_widgets.budget_row.clone(),
mountpoint_row: main_widgets.mountpoint_row.clone(),
settings_suppress: Cell::new(false),
budget_source: RefCell::new(None),
pins_expanded: Cell::new(false),
pins_group: main_widgets.pins_group.clone(),
pin_rows: RefCell::new(Vec::new()),
pins_state: RefCell::new(None),
@@ -381,7 +401,29 @@ fn build_window(app: &adw::Application) {
new_folder: browser_widgets.new_folder.clone(),
upload: browser_widgets.upload.clone(),
upload_folder: browser_widgets.upload_folder.clone(),
build_thumbnails: browser_widgets.build_thumbnails.clone(),
thumbnail_build_row: browser_widgets.thumbnail_build_row.clone(),
thumbnail_progress: browser_widgets.thumbnail_progress.clone(),
thumbnail_status: browser_widgets.thumbnail_status.clone(),
thumbnail_poll: RefCell::new(None),
thumbnail_build_running: Cell::new(false),
thumbnail_cancel_pending: Cell::new(false),
search_source: RefCell::new(None),
load_generation: Cell::new(0),
views: browser_widgets.views.clone(),
bulk: browser_widgets.bulk.clone(),
bulk_label: browser_widgets.bulk_label.clone(),
bulk_trash: browser_widgets.bulk_trash.clone(),
bulk_pin: browser_widgets.bulk_pin.clone(),
bulk_unpin: browser_widgets.bulk_unpin.clone(),
empty_actions: browser_widgets.empty_actions.clone(),
summary: browser_widgets.summary.clone(),
zoom: browser_widgets.zoom.clone(),
grid_thumbnail_size: Cell::new(GRID_THUMB_DEFAULT),
grid_tiles: RefCell::new(Vec::new()),
quota_box: browser_widgets.quota_box.clone(),
quota: browser_widgets.quota.clone(),
quota_text: browser_widgets.quota_text.clone(),
},
details: DetailsState {
details: browser_widgets.details,
@@ -407,6 +449,8 @@ fn build_window(app: &adw::Application) {
retry: gallery_widgets.retry.clone(),
more: gallery_widgets.more.clone(),
upload: gallery_widgets.upload.clone(),
import: gallery_widgets.import.clone(),
empty_actions: gallery_widgets.empty_actions.clone(),
title: gallery_widgets.title.clone(),
subtitle: gallery_widgets.subtitle.clone(),
albums: gallery_widgets.albums.clone(),
@@ -494,6 +538,24 @@ fn build_window(app: &adw::Application) {
inflight: Cell::new(false),
key: RefCell::new(None),
},
takeout: TakeoutState {
archives: RefCell::new(Vec::new()),
list_group: takeout_widgets.list_group.clone(),
rows: RefCell::new(Vec::new()),
dropzone: takeout_widgets.dropzone.clone(),
scan_button: takeout_widgets.scan_button.clone(),
import_button: takeout_widgets.import_button.clone(),
cancel_button: takeout_widgets.cancel_button.clone(),
clear_button: takeout_widgets.clear_button.clone(),
progress_group: takeout_widgets.progress_group.clone(),
progress_label: takeout_widgets.progress_label.clone(),
progress_bar: takeout_widgets.progress_bar.clone(),
summary_group: takeout_widgets.summary_group.clone(),
summary_rows: RefCell::new(Vec::new()),
inflight: Cell::new(false),
running: Cell::new(false),
dry_run: Cell::new(false),
},
});
wire_login(&ui);
wire_logout(&ui, &main_widgets.logout_button);
@@ -501,18 +563,31 @@ fn build_window(app: &adw::Application) {
&ui,
&main_widgets.purge_button,
&main_widgets.mountpoint_button,
&main_widgets.import_row,
);
wire_sidebar(&ui);
wire_browser(&ui, &browser_widgets.grid, &browser_widgets.column_view);
wire_bulk(
&ui,
&browser_widgets.bulk_clear,
&browser_widgets.empty_upload,
&browser_widgets.empty_new_folder,
);
wire_browser_actions(
&ui,
&browser_widgets.new_folder,
&browser_widgets.upload,
&browser_widgets.upload_folder,
&browser_widgets.build_thumbnails,
);
wire_details(&ui);
wire_search(&ui);
wire_gallery(&ui, &gallery_widgets.list, &gallery_widgets.scroll);
wire_gallery_empty(
&ui,
&gallery_widgets.empty_upload,
&gallery_widgets.empty_import,
);
wire_albums(&ui);
wire_trash(&ui, &trash_widgets.list, &trash_widgets.empty);
wire_shared(&ui, &shared_widgets.retry, &shared_widgets.add_bookmark);
@@ -520,6 +595,7 @@ fn build_window(app: &adw::Application) {
wire_devices(&ui, &devices_widgets.retry, &devices_widgets.restore);
wire_locations(&ui, &locations_widgets.retry, &locations_widgets.add_folder);
wire_activity(&ui, &activity_widgets.retry);
wire_takeout(&ui, &takeout_widgets);
wire_refresh(
&ui,
&[
@@ -539,6 +615,10 @@ fn build_window(app: &adw::Application) {
// network round-trip only happens on demand rather than on every refresh.
let ui_nav = ui.clone();
stack.connect_visible_child_name_notify(move |st| {
// Rows from the page being left must not keep full-size image downloads
// alive in the daemon. The page being entered establishes a fresh
// thumbnail generation as it paints.
cancel_file_thumbnails(&ui_nav);
sync_sidebar(&ui_nav);
match st.visible_child_name().as_deref() {
Some("browser") => load_browser(&ui_nav),
@@ -559,6 +639,9 @@ fn build_window(app: &adw::Application) {
// every visit to stay live.
Some("activity") => load_activity(&ui_nav),
Some("trash") => load_trash(&ui_nav),
// The import runs in the daemon and outlives this page, so arriving
// here asks straight away whether one is in flight.
Some("takeout") => refresh_takeout(&ui_nav),
_ => {}
}
});
@@ -644,6 +727,17 @@ fn build_sidebar() -> (adw::NavigationPage, gtk4::ListBox) {
(page, list)
}
/// The sidebar destination a page belongs under. Most pages are their own
/// destination; a sub-page reached from one (Import, off Settings) has no row of
/// its own and answers with its parent, so the sidebar highlights where the user
/// came from rather than clearing — which would read as "nowhere".
fn destination_of(page: &str) -> &str {
match page {
"takeout" => "main",
other => other,
}
}
/// Selecting a sidebar row navigates the page stack. The reverse direction (stack
/// → sidebar highlight) is [`sync_sidebar`], so the two can't fight: this handler
/// only ever writes the stack.
@@ -654,7 +748,11 @@ fn wire_sidebar(ui: &Rc<Ui>) {
let Some((page, _, _)) = DESTINATIONS.get(row.index() as usize) else {
return;
};
if ui_row.stack.visible_child_name().as_deref() != Some(*page) {
// Compared by *destination*, not by page: while a sub-page is open its
// parent's row is the selected one, and re-asserting it here would kick
// the user off the sub-page the moment they arrived.
let current = ui_row.stack.visible_child_name();
if current.as_deref().map(destination_of) != Some(*page) {
ui_row.stack.set_visible_child_name(page);
}
});
@@ -755,6 +853,7 @@ fn sync_sidebar(ui: &Rc<Ui>) {
let Some(current) = ui.stack.visible_child_name() else {
return;
};
let current = destination_of(&current);
let index = DESTINATIONS
.iter()
.position(|(page, _, _)| *page == current);
@@ -776,6 +875,22 @@ fn toast(ui: &Rc<Ui>, message: &str) {
ui.toasts.add_toast(adw::Toast::new(message));
}
/// Show a toast that offers one follow-up action — an Undo, or a way to the page
/// the outcome landed on.
///
/// Given a longer timeout than a plain toast: an action the user has to notice,
/// read and reach is not the same ask as a line of confirmation they can ignore.
fn toast_action(ui: &Rc<Ui>, message: &str, label: &str, action: impl Fn(&Rc<Ui>) + 'static) {
let toast = adw::Toast::builder()
.title(message)
.button_label(label)
.timeout(8)
.build();
let ui_action = ui.clone();
toast.connect_button_clicked(move |_| action(&ui_action));
ui.toasts.add_toast(toast);
}
/// Show a toast for a failure. Same surface as [`toast`], but the message is
/// prefixed with what was being attempted, since a bare daemon error ("no such
/// file") reads as noise without it.
@@ -830,7 +945,7 @@ fn toast_failure(ui: &Rc<Ui>, what: &str, message: &str, kind: ErrorKind) {
fn build_primary_menu() -> gtk4::MenuButton {
let menu = gio::Menu::new();
menu.append(Some("Keyboard Shortcuts"), Some("win.shortcuts"));
menu.append(Some("About Proton Drive"), Some("win.about"));
menu.append(Some("About Proton Drive for Linux"), Some("win.about"));
gtk4::MenuButton::builder()
.icon_name("open-menu-symbolic")
.tooltip_text("Main menu")
@@ -849,11 +964,17 @@ fn install_window_actions(window: &adw::ApplicationWindow) {
let win = window.clone();
about.connect_activate(move |_, _| {
let dialog = adw::AboutDialog::builder()
.application_name("Proton Drive")
.application_icon("folder-remote-symbolic")
.application_name("Proton Drive for Linux")
.application_icon("io.narl.proton-drive-linux")
.version(pdfs_core::config::APP_VERSION)
.developer_name("proton-drive-linux")
.comments("On-demand Proton Drive sync for Linux.")
.developer_name("Nils Pukropp")
.website("https://github.com/narrrl/proton-drive-linux")
.issue_url("https://github.com/narrrl/proton-drive-linux/issues")
.license_type(gtk4::License::MitX11)
.comments(
"Files-on-demand Proton Drive for the Linux desktop.\n\n\
Unofficial client — not affiliated with, endorsed by, or supported by Proton AG.",
)
.build();
dialog.present(Some(&win));
});
@@ -913,7 +1034,13 @@ fn install_shortcuts(ui: &Rc<Ui>, window: &adw::ApplicationWindow) {
// browser-only bindings.
gtk4::gdk::Key::F5 => reload_current_page(&ui),
gtk4::gdk::Key::r | gtk4::gdk::Key::R if ctrl => reload_current_page(&ui),
gtk4::gdk::Key::f | gtk4::gdk::Key::F if ctrl && on_browser => {
// Search lives on Files, but the shortcut works from anywhere:
// wanting to find a file is not a reason to first have to remember
// which page owns the search box.
gtk4::gdk::Key::f | gtk4::gdk::Key::F if ctrl => {
if !on_browser {
ui.stack.set_visible_child_name("browser");
}
ui.browser.search.grab_focus();
}
gtk4::gdk::Key::n | gtk4::gdk::Key::N if ctrl && on_browser => prompt_new_folder(&ui),
@@ -923,12 +1050,21 @@ fn install_shortcuts(ui: &Rc<Ui>, window: &adw::ApplicationWindow) {
prompt_rename(&ui, &entry);
}
}
// Acts on the whole selection: Delete with five files highlighted
// means those five, not whichever one the details pane happens to
// be describing.
gtk4::gdk::Key::Delete if on_browser => {
if let Some(entry) = selected_entry(&ui) {
prompt_delete(&ui, &entry);
let entries = selected_entries(&ui);
if !entries.is_empty() {
prompt_delete_many(&ui, entries);
}
}
gtk4::gdk::Key::Escape if on_browser && ui.browser.split.shows_sidebar() => {
gtk4::gdk::Key::Escape
if on_browser
&& (ui.browser.split.shows_sidebar() || ui.browser.bulk.reveals_child()) =>
{
clear_selection(&ui);
sync_bulk_bar(&ui);
hide_details(&ui);
}
_ => return glib::Propagation::Proceed,
File diff suppressed because it is too large. Load diff
+1
View File
@@ -9,5 +9,6 @@ pub(crate) mod photos;
pub(crate) mod shared;
pub(crate) mod shared_by_me;
pub(crate) mod status;
pub(crate) mod takeout;
pub(crate) mod trash;
pub(crate) mod verify;
+71 -1
View File
@@ -19,6 +19,12 @@ pub(crate) struct GalleryState {
pub(crate) retry: gtk4::Button,
pub(crate) more: gtk4::Button,
pub(crate) upload: gtk4::Button,
/// Opens the Google Photos Takeout import chooser.
pub(crate) import: gtk4::Button,
/// Upload / Import offered on the *empty timeline* status page, so a fresh
/// account is a place to start rather than a dead end. Hidden on every other
/// status: a load error is not the moment to offer an upload.
pub(crate) empty_actions: gtk4::Box,
/// "Photos", or the album's name while one is open.
pub(crate) title: gtk4::Label,
/// "1,204 photos" under the page title.
@@ -177,6 +183,10 @@ pub(crate) struct GalleryWidgets {
pub(crate) scroll: gtk4::ScrolledWindow,
pub(crate) retry: gtk4::Button,
pub(crate) upload: gtk4::Button,
pub(crate) import: gtk4::Button,
pub(crate) empty_actions: gtk4::Box,
pub(crate) empty_upload: gtk4::Button,
pub(crate) empty_import: gtk4::Button,
pub(crate) refresh: gtk4::Button,
/// The Albums grid, its own status page and the stack between them, plus the
/// Photos/Albums switcher and the back button out of an album.
@@ -227,10 +237,29 @@ pub(crate) fn build_gallery_page() -> (gtk4::Widget, GalleryWidgets) {
retry.add_css_class("suggested-action");
retry.set_visible(false);
// The empty-timeline state's two ways forward. `StatusPage` takes one child,
// so Retry and these share a box and each is shown only for its own state.
let empty_upload = gtk4::Button::builder().label("Upload photos").build();
empty_upload.add_css_class("pill");
empty_upload.add_css_class("suggested-action");
let empty_import = gtk4::Button::builder()
.label("Import from Google Photos")
.build();
empty_import.add_css_class("pill");
let empty_actions = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
empty_actions.set_halign(gtk4::Align::Center);
empty_actions.set_visible(false);
empty_actions.append(&empty_upload);
empty_actions.append(&empty_import);
let status_child = gtk4::Box::new(gtk4::Orientation::Vertical, 12);
status_child.append(&retry);
status_child.append(&empty_actions);
let status = adw::StatusPage::builder()
.icon_name("image-x-generic-symbolic")
.vexpand(true)
.child(&retry)
.child(&status_child)
.build();
status.add_css_class("compact");
@@ -288,12 +317,23 @@ pub(crate) fn build_gallery_page() -> (gtk4::Widget, GalleryWidgets) {
.build();
upload.add_css_class("pill");
upload.add_css_class("suggested-action");
// Importing an export is a rarer, heavier action than adding one photo, so
// it sits beside Upload as a plain button rather than a second accent one.
let import = gtk4::Button::builder()
.icon_name("folder-download-symbolic")
.tooltip_text("Import a Google Photos Takeout export")
.valign(gtk4::Align::Center)
.build();
import.add_css_class("flat");
import.add_css_class("circular");
let refresh = refresh_button();
let header_box = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
header_box.append(&back);
header_box.append(&titles);
header_box.append(&refresh);
header_box.append(&import);
header_box.append(&upload);
// All / Photos / Videos / Raw filter. Linked toggles acting as one segmented
@@ -428,6 +468,10 @@ pub(crate) fn build_gallery_page() -> (gtk4::Widget, GalleryWidgets) {
scroll,
retry,
upload,
import,
empty_actions,
empty_upload,
empty_import,
refresh,
tabs,
favorites_btn,
@@ -554,6 +598,15 @@ pub(crate) fn thousands(n: usize) -> String {
/// Wire the gallery: install the section factory, the zoom gestures, the pager
/// and the upload button. Activating a thumbnail downloads the photo and opens it
/// in the in-app lightbox.
/// Wire the empty-timeline buttons. Upload re-fires the header's own Upload
/// rather than duplicating its file-chooser, so the two can't drift apart.
pub(crate) fn wire_gallery_empty(ui: &Rc<Ui>, upload: &gtk4::Button, import: &gtk4::Button) {
let ui_upload = ui.clone();
upload.connect_clicked(move |_| ui_upload.gallery.upload.emit_clicked());
let ui_import = ui.clone();
import.connect_clicked(move |_| ui_import.stack.set_visible_child_name("takeout"));
}
pub(crate) fn wire_gallery(ui: &Rc<Ui>, list: &gtk4::ListView, scroll: &gtk4::ScrolledWindow) {
let factory = gtk4::SignalListItemFactory::new();
factory.connect_setup(|_, item| {
@@ -729,6 +782,14 @@ pub(crate) fn wire_gallery(ui: &Rc<Ui>, list: &gtk4::ListView, scroll: &gtk4::Sc
load_gallery(&ui_dates, false);
});
// The import is a staged, hours-long migration, so the button hands over to
// the Import page rather than opening a file chooser here: the archives need
// reviewing before anything is sent, and the run needs somewhere to report.
let ui_import = ui.clone();
ui.gallery
.import
.connect_clicked(move |_| ui_import.stack.set_visible_child_name("takeout"));
let ui_upload = ui.clone();
ui.gallery.upload.connect_clicked(move |_| {
let dialog = gtk4::FileDialog::builder()
@@ -1460,6 +1521,12 @@ pub(crate) fn load_gallery(ui: &Rc<Ui>, append: bool) {
)
};
gallery_status(&ui, "image-x-generic-symbolic", title, description, false);
// Only the whole-timeline empty state: an empty *album* is
// filled by adding existing photos to it, not by uploading
// into it, so the buttons would point the wrong way.
ui.gallery
.empty_actions
.set_visible(ui.gallery.album.borrow().is_none());
return;
}
ui.gallery.content.set_visible_child_name("timeline");
@@ -1502,6 +1569,9 @@ pub(crate) fn gallery_status(ui: &Rc<Ui>, icon: &str, title: &str, description:
ui.gallery.status.set_title(title);
ui.gallery.status.set_description(Some(description));
ui.gallery.retry.set_visible(retry);
// Only the empty timeline offers a way to fill it; every other status turns
// those buttons back off.
ui.gallery.empty_actions.set_visible(false);
ui.gallery.more.set_visible(false);
ui.gallery.content.set_visible_child_name("status");
}
+2 -5
View File
@@ -151,6 +151,7 @@ pub(crate) fn load_shared(ui: &Rc<Ui>) {
if ui.shared.inflight.get() {
return;
}
cancel_file_thumbnails(ui);
let current = ui.shared.nav.borrow().last().cloned();
if let Some((uid, _)) = current {
load_shared_folder(ui, uid);
@@ -308,11 +309,7 @@ fn shared_entry_row(ui: &Rc<Ui>, entry: &DirEntry) -> adw::ActionRow {
if let Some(badge) = role_badge(&entry.role) {
row.add_suffix(&badge);
}
row.add_prefix(&gtk4::Image::from_icon_name(if entry.is_dir {
"folder-symbolic"
} else {
"text-x-generic-symbolic"
}));
row.add_prefix(&file_thumbnail(ui, entry, 40, 24, true));
let ui_act = ui.clone();
let uid = entry.uid.clone();
let name = entry.name.clone();
@@ -113,6 +113,7 @@ pub(crate) fn load_shared_by_me(ui: &Rc<Ui>) {
if ui.shared_by_me.inflight.get() {
return;
}
cancel_file_thumbnails(ui);
ui.shared_by_me.inflight.set(true);
shared_by_me_status(
ui,
@@ -200,15 +201,12 @@ pub(crate) fn repaint_shared_by_me(ui: &Rc<Ui>, items: &[SharedItem]) {
ui.shared_by_me.content.set_visible_child_name("list");
let mut rows: Vec<gtk4::Widget> = Vec::new();
for item in items {
let entry = shared_item_as_entry(item);
let row = adw::ActionRow::builder()
.title(&item.name)
.subtitle(shared_item_summary(item))
.build();
row.add_prefix(&gtk4::Image::from_icon_name(if item.is_dir {
"folder-symbolic"
} else {
"text-x-generic-symbolic"
}));
row.add_prefix(&file_thumbnail(ui, &entry, 40, 24, true));
// Copy / open the public link, when there is one with a recovered URL.
if let Some(url) = item.link.as_ref().and_then(|l| l.url.clone()) {
@@ -245,7 +243,6 @@ pub(crate) fn repaint_shared_by_me(ui: &Rc<Ui>, items: &[SharedItem]) {
.build();
manage.add_css_class("flat");
let ui_manage = ui.clone();
let entry = shared_item_as_entry(item);
manage.connect_clicked(move |_| open_share_dialog(&ui_manage, &entry));
row.add_suffix(&manage);
@@ -289,7 +286,7 @@ pub(crate) fn shared_item_as_entry(item: &SharedItem) -> DirEntry {
name: item.name.clone(),
is_dir: item.is_dir,
size: 0,
modified: 0,
modified: item.modified,
pinned: false,
cached: false,
uid: item.uid.clone(),
+220 -35
View File
@@ -37,7 +37,16 @@ pub(crate) struct StatusState {
/// Set while a settings widget is being populated programmatically, so its
/// change handler skips the IPC/systemd side effect.
pub(crate) settings_suppress: Cell<bool>,
/// Pending debounce for the cache-budget editor. Each `+` click is a value
/// change, and firing a `SetCacheBudget` (and a toast) per click means a
/// stack of toasts and a series of caps the user never asked to apply — so
/// only the value they settle on is sent.
pub(crate) budget_source: RefCell<Option<glib::SourceId>>,
pub(crate) pins_group: adw::PreferencesGroup,
/// Whether the pin list is showing every pin or only the first
/// [`PINS_COLLAPSED`]. A long pin list would otherwise push everything below
/// it — including the Developer group — off the end of the page.
pub(crate) pins_expanded: Cell<bool>,
/// Rows currently shown under [`Self::pins_group`], retained so a refresh can
/// diff against them and only rebuild when the pin set actually changes.
pub(crate) pin_rows: RefCell<Vec<PinRow>>,
@@ -104,6 +113,8 @@ pub(crate) struct MainWidgets {
/// which is where the folder is changed.
pub(crate) mountpoint_row: adw::ActionRow,
pub(crate) mountpoint_button: gtk4::Button,
/// Opens the Google Photos Takeout import page.
pub(crate) import_row: adw::ActionRow,
}
/// The main (logged-in) page: a libadwaita settings surface — account header,
@@ -233,6 +244,22 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
mountpoint_row.add_suffix(&mountpoint_button);
system_group.add(&mountpoint_row);
// Import: a one-off migration rather than a setting, so it is a doorway to
// its own page rather than controls inlined here — staging a set of archives
// and watching an hours-long upload needs the room.
let import_group = adw::PreferencesGroup::builder()
.title("Import")
.description("Bring a photo library from another service into Proton Photos.")
.build();
let import_row = adw::ActionRow::builder()
.title("Import from Google Photos")
.subtitle("Add a Google Takeout export to your timeline.")
.activatable(true)
.build();
import_row.add_prefix(&gtk4::Image::from_icon_name("folder-download-symbolic"));
import_row.add_suffix(&gtk4::Image::from_icon_name("go-next-symbolic"));
import_group.add(&import_row);
// Pins group: filled in by refresh.
let pins_group = adw::PreferencesGroup::builder()
.title("Pinned files")
@@ -265,6 +292,7 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
inner.append(&transfers_group);
inner.append(&storage_group);
inner.append(&system_group);
inner.append(&import_group);
inner.append(&pins_group);
inner.append(&dev_group);
@@ -292,10 +320,20 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
purge_button,
mountpoint_row,
mountpoint_button,
import_row,
},
)
}
/// How many pins the Settings list shows before collapsing the rest behind a
/// "Show all" row. Enough to recognise the list at a glance; short enough that
/// the groups below it stay reachable.
pub(crate) const PINS_COLLAPSED: usize = 6;
/// How long the cache-budget editor waits after the last change before applying
/// it, so a run of `+` clicks is one request rather than one per click.
pub(crate) const BUDGET_DEBOUNCE: Duration = Duration::from_millis(600);
/// Bytes per GiB, for the cache-budget editor's unit conversion.
pub(crate) const GIB: f64 = 1024.0 * 1024.0 * 1024.0;
@@ -308,6 +346,7 @@ pub(crate) fn wire_settings(
ui: &Rc<Ui>,
purge_button: &gtk4::Button,
mountpoint_button: &gtk4::Button,
import_row: &adw::ActionRow,
) {
let config = ui.dirs.load_config();
@@ -329,13 +368,22 @@ pub(crate) fn wire_settings(
if ui_budget.status.settings_suppress.get() {
return;
}
// Replace any pending apply, so only the value the user stops on is sent.
if let Some(src) = ui_budget.status.budget_source.borrow_mut().take() {
src.remove();
}
let bytes = (row.value() * GIB).round() as u64;
settings_request(
&ui_budget,
Request::SetCacheBudget { bytes },
"Cache budget updated",
"Couldn't set cache budget",
);
let ui_fire = ui_budget.clone();
let src = glib::timeout_add_local_once(BUDGET_DEBOUNCE, move || {
ui_fire.status.budget_source.borrow_mut().take();
settings_request(
&ui_fire,
Request::SetCacheBudget { bytes },
"Cache budget updated",
"Couldn't set cache budget",
);
});
*ui_budget.status.budget_source.borrow_mut() = Some(src);
});
// Purge: confirm, then drop all unpinned cached content via the daemon.
@@ -382,6 +430,10 @@ pub(crate) fn wire_settings(
// local path; this button is the way there.
let ui_mp = ui.clone();
mountpoint_button.connect_clicked(move |_| ui_mp.stack.set_visible_child_name("locations"));
// Import: a doorway, not a setting — everything about it lives on its page.
let ui_import = ui.clone();
import_row.connect_activated(move |_| ui_import.stack.set_visible_child_name("takeout"));
}
/// Run a settings control-socket round-trip (budget / purge) on a worker thread,
@@ -513,8 +565,14 @@ pub(crate) fn refresh(ui: &Rc<Ui>) {
refresh_transfers(ui);
// Both of these pages show work as it happens, so they follow the tick while
// they are on screen. Every other page loads on navigation only.
// A Takeout import outlives the page that started it, so it is polled
// wherever the user has navigated to — otherwise leaving the page means
// never being told it finished.
if ui.stack.visible_child_name().as_deref() == Some("takeout") || ui.takeout.running.get() {
refresh_takeout(ui);
}
match ui.stack.visible_child_name().as_deref() {
Some("main") => refresh_quota(ui),
Some("main" | "browser") => refresh_quota(ui),
Some("locations") => refresh_locations(ui),
Some("activity") => refresh_activity(ui),
_ => {}
@@ -522,13 +580,11 @@ pub(crate) fn refresh(ui: &Rc<Ui>) {
}
/// How long a quota reading stays fresh. Account storage barely moves, so the
/// Settings tick refetches it only this often rather than every 2s.
/// active-page tick refetches it only this often rather than every 2s.
const QUOTA_TTL: Duration = Duration::from_secs(60);
/// Fetch the account quota (if the last reading is stale) and paint the Account
/// storage group. Runs only while Settings is on screen. The group stays hidden
/// until a reading lands, so an account the API can't report — or a still-starting
/// daemon — shows nothing rather than an empty bar. A failed fetch leaves the last
/// Fetch the account quota (if the last reading is stale) and paint both the
/// Settings storage group and Files status bar. A failed fetch leaves the last
/// good reading in place.
pub(crate) fn refresh_quota(ui: &Rc<Ui>) {
if ui.status.quota_inflight.get() {
@@ -551,28 +607,71 @@ pub(crate) fn refresh_quota(ui: &Rc<Ui>) {
})) = result
{
ui.status.quota_checked_at.set(Some(Instant::now()));
let used = used_space.max(0) as u64;
if max_space > 0 {
let total = max_space as u64;
let fraction = (used as f64 / total as f64).min(1.0);
ui.status.quota_bar.set_fraction(fraction);
let pct = (fraction * 100.0).round() as u64;
ui.status.quota_label.set_text(&format!(
"{} of {} used ({pct}%)",
human_bytes(used),
human_bytes(total)
));
} else {
ui.status.quota_bar.set_fraction(0.0);
ui.status
.quota_label
.set_text(&format!("{} used", human_bytes(used)));
}
paint_account_quota(&ui, max_space, used_space);
ui.status.quota_group.set_visible(true);
} else if ui.status.quota_checked_at.get().is_none() {
// Match Dolphin: capacity information does not occupy the bar until
// the backing observer has real figures.
ui.browser.quota_box.set_visible(false);
}
});
}
fn paint_account_quota(ui: &Rc<Ui>, max_space: i64, used_space: i64) {
let (fraction, text) = quota_display(max_space, used_space);
ui.status.quota_bar.set_fraction(fraction);
ui.status.quota_label.set_text(&text);
if let Some((fraction, free_text, tooltip)) = quota_status_display(max_space, used_space) {
ui.browser.quota.set_fraction(fraction);
ui.browser.quota.set_tooltip_text(Some(&tooltip));
ui.browser.quota_text.set_label(&free_text);
ui.browser.quota_text.set_tooltip_text(Some(&tooltip));
ui.browser.quota_box.set_visible(true);
} else {
ui.browser.quota_box.set_visible(false);
}
}
fn quota_display(max_space: i64, used_space: i64) -> (f64, String) {
let used = used_space.max(0) as u64;
if max_space <= 0 {
return (0.0, format!("{} used", human_bytes(used)));
}
let total = max_space as u64;
let fraction = (used as f64 / total as f64).clamp(0.0, 1.0);
let pct = (fraction * 100.0).round() as u64;
(
fraction,
format!(
"{} of {} used ({pct}%)",
human_bytes(used),
human_bytes(total)
),
)
}
/// Dolphin's status bar shows a bare capacity bar followed by “X free”; the
/// full free/total/percentage sentence is a tooltip rather than inline bar text.
fn quota_status_display(max_space: i64, used_space: i64) -> Option<(f64, String, String)> {
if max_space <= 0 {
return None;
}
let total = max_space as u64;
let used = (used_space.max(0) as u64).min(total);
let free = total.saturating_sub(used);
let fraction = used as f64 / total as f64;
let pct = (fraction * 100.0).round() as u64;
Some((
fraction,
format!("{} free", human_bytes(free)),
format!(
"{} free out of {} ({pct}% used)",
human_bytes(free),
human_bytes(total)
),
))
}
/// Record the mount state seen by the last status poll: gate every control that
/// needs a live daemon, and notify the desktop when the state actually flips.
///
@@ -584,6 +683,11 @@ pub(crate) fn set_mounted(ui: &Rc<Ui>, mounted: bool) {
ui.browser.new_folder.set_sensitive(mounted);
ui.browser.upload.set_sensitive(mounted);
ui.browser.upload_folder.set_sensitive(mounted);
ui.browser.build_thumbnails.set_sensitive(
mounted
&& (!ui.browser.thumbnail_build_running.get()
|| !ui.browser.thumbnail_cancel_pending.get()),
);
ui.gallery.upload.set_sensitive(mounted);
ui.details.details.pin_row.set_sensitive(mounted);
ui.details.details.rename_button.set_sensitive(mounted);
@@ -653,6 +757,10 @@ pub(crate) fn repaint_transfers(ui: &Rc<Ui>, items: &[TransferItem], jobs: &[Job
// count falling to zero is what "the sync finished" looks like from here.
// Jobs are deliberately not counted — a bulk upload retires its scan job and
// starts its upload job mid-flight, which is not a thing finishing.
// The Import page reads its progress off the same snapshot rather than
// polling the daemon a second time for numbers already on the wire.
takeout_progress(ui, jobs);
let previous = ui.status.active_transfers.replace(items.len());
if items.is_empty() && previous > 0 {
let files = if previous == 1 {
@@ -808,11 +916,14 @@ pub(crate) fn refresh_status(ui: &Rc<Ui>) {
(used as f64 / budget as f64).min(1.0)
};
ui.status.cache_bar.set_fraction(fraction);
ui.status.cache_label.set_text(&format!(
"{} of {} used",
human_bytes(used),
human_bytes(budget)
));
// A 0 budget means *unlimited*, not a zero-byte cap — "of 0 B
// used" reads as a broken read-out, and there is no fraction to
// draw against no limit.
ui.status.cache_label.set_text(&if budget == 0 {
format!("{} cached — no limit set", human_bytes(used))
} else {
format!("{} of {} used", human_bytes(used), human_bytes(budget))
});
repaint_pins(&ui, &pins, true);
}
// Daemon unreachable (still starting, or down): report not-mounted and
@@ -865,7 +976,15 @@ pub(crate) fn repaint_pins(ui: &Rc<Ui>, pins: &[pdfs_core::cache::Pin], mounted:
return;
}
for pin in pins {
// Only the first page of pins is rendered; the rest sit behind the row
// below, so a device with hundreds of pins doesn't bury the groups after it.
let expanded = ui.status.pins_expanded.get();
let shown = if expanded {
pins.len()
} else {
pins.len().min(PINS_COLLAPSED)
};
for pin in &pins[..shown] {
let name = Path::new(&pin.path)
.file_name()
.and_then(|n| n.to_str())
@@ -903,4 +1022,70 @@ pub(crate) fn repaint_pins(ui: &Rc<Ui>, pins: &[pdfs_core::cache::Pin], mounted:
unpin: Some(unpin),
});
}
if pins.len() > PINS_COLLAPSED {
let row = adw::ActionRow::builder()
.title(if expanded {
"Show fewer".to_string()
} else {
format!("Show all {} pinned files", pins.len())
})
.activatable(true)
.build();
row.add_suffix(&gtk4::Image::from_icon_name(if expanded {
"go-up-symbolic"
} else {
"go-down-symbolic"
}));
let ui_more = ui.clone();
row.connect_activated(move |_| {
ui_more
.status
.pins_expanded
.set(!ui_more.status.pins_expanded.get());
// The diff guard compares pin *paths*; the same paths render
// differently now, so the baseline has to be dropped for the next
// repaint to actually rebuild.
*ui_more.status.pins_state.borrow_mut() = None;
refresh(&ui_more);
});
ui.status.pins_group.add(&row);
ui.status
.pin_rows
.borrow_mut()
.push(PinRow { row, unpin: None });
}
}
#[cfg(test)]
mod tests {
use super::{quota_display, quota_status_display};
#[test]
fn quota_display_reports_used_total_and_percentage() {
let gib = 1024_i64.pow(3);
let (fraction, text) = quota_display(4 * gib, gib);
assert!((fraction - 0.25).abs() < f64::EPSILON);
assert_eq!(text, "1.0 GiB of 4.0 GiB used (25%)");
}
#[test]
fn quota_display_clamps_bad_api_values() {
assert_eq!(quota_display(0, -1), (0.0, "0 B used".to_string()));
assert_eq!(quota_display(100, 150).0, 1.0);
}
#[test]
fn quota_status_display_matches_dolphin_wording() {
let gib = 1024_i64.pow(3);
assert_eq!(
quota_status_display(4 * gib, gib),
Some((
0.25,
"3.0 GiB free".to_string(),
"3.0 GiB free out of 4.0 GiB (25% used)".to_string()
))
);
assert_eq!(quota_status_display(0, 0), None);
}
}
+779
View File
@@ -0,0 +1,779 @@
//! The Import page: bringing a Google Photos **Takeout** export into Proton
//! Photos.
//!
//! Reached from Settings → *Import from Google Photos* (and from the Gallery's
//! import button), because it is a one-off migration rather than something the
//! photo timeline does every day. It gets a page of its own instead of a dialog
//! for two reasons: an export is a *set* of archives that has to be staged and
//! checked before anything is sent, and the import that follows runs for hours,
//! so its progress needs somewhere to live that the user can navigate away from
//! and come back to.
//!
//! Archives arrive either by drag-and-drop onto the page or through the file
//! chooser; both funnel into [`add_archives`], which is where the `.zip` filter
//! and de-duplication live. Nothing is sent until the user presses Scan or
//! Import — staging is local, so a mis-drop costs nothing.
//!
//! The daemon owns the run itself ([`Request::ImportTakeout`]): this page only
//! stages paths, launches, polls [`Request::ImportStatus`] for the final report,
//! and reads live progress off the import's job in the transfer snapshot the
//! Settings tick already fetches (see [`takeout_progress`]).
use crate::*;
/// The title the daemon gives the import's job in [`Request::GetQueueStatus`].
/// Matching on it is how this page borrows the progress the Settings tick is
/// already polling instead of adding a second poll of its own.
const IMPORT_JOB: &str = "Importing Google Photos";
/// What the archive list says before anything is staged. `repaint_archives`
/// swaps in the count and total once there is something to report, and swaps
/// this back when the list is emptied again.
/// Where the staged set is remembered between runs, under the app's state
/// directory. Staging an export means picking dozens of files out of a file
/// chooser; losing that to a window close (or a crash mid-import) and having to
/// redo it is the kind of small cruelty that makes a migration not get finished.
fn staged_path(ui: &Rc<Ui>) -> PathBuf {
ui.dirs.state_dir().join("takeout-staged.json")
}
/// Read the remembered set, dropping any archive that has since been moved or
/// deleted — a stale path would only fail at launch, and silently is worse.
fn load_staged(ui: &Rc<Ui>) -> Vec<PathBuf> {
let Ok(text) = std::fs::read_to_string(staged_path(ui)) else {
return Vec::new();
};
let paths: Vec<String> = serde_json::from_str(&text).unwrap_or_default();
paths
.into_iter()
.map(PathBuf::from)
.filter(|p| p.is_file())
.collect()
}
/// Remember the staged set. Best-effort: failing to write it costs the user a
/// re-pick next time, which is not worth interrupting them over now.
fn save_staged(ui: &Rc<Ui>) {
let paths: Vec<String> = ui
.takeout
.archives
.borrow()
.iter()
.map(|p| p.display().to_string())
.collect();
let path = staged_path(ui);
let written = serde_json::to_string(&paths)
.map_err(|e| e.to_string())
.and_then(|text| std::fs::write(&path, text).map_err(|e| e.to_string()));
if let Err(e) = written {
tracing::warn!("cannot remember staged Takeout archives: {e}");
}
}
const EMPTY_LIST_DESCRIPTION: &str = "Photos already in Proton Photos are matched by name and content and skipped, so \
re-running an interrupted import is safe.";
pub(crate) struct TakeoutState {
/// Archives staged for this import, in the order they were added. Sent as
/// one set: Google splits an export across numbered zips and routinely puts
/// a photo in one part and its metadata sidecar in another, so a per-file
/// import would lose album titles and capture times.
pub(crate) archives: RefCell<Vec<PathBuf>>,
/// The staged-archive list and the rows currently rendered in it.
pub(crate) list_group: adw::PreferencesGroup,
pub(crate) rows: RefCell<Vec<adw::PreferencesRow>>,
/// The drop zone, whose `dropzone-active` class follows the drag.
pub(crate) dropzone: gtk4::Box,
pub(crate) scan_button: gtk4::Button,
pub(crate) import_button: gtk4::Button,
pub(crate) cancel_button: gtk4::Button,
pub(crate) clear_button: gtk4::Button,
/// Progress group, hidden unless an import is running.
pub(crate) progress_group: adw::PreferencesGroup,
pub(crate) progress_label: gtk4::Label,
pub(crate) progress_bar: gtk4::ProgressBar,
/// Report group for the last finished run, hidden until there is one.
pub(crate) summary_group: adw::PreferencesGroup,
pub(crate) summary_rows: RefCell<Vec<adw::ActionRow>>,
/// One [`Request::ImportStatus`] in flight at a time, so a wedged daemon
/// can't stack worker threads on the 2s tick.
pub(crate) inflight: Cell<bool>,
/// Whether the daemon reported an import running on the last poll.
pub(crate) running: Cell<bool>,
/// Whether the run in flight was launched as a dry run, so the report can
/// say "would upload" rather than claiming photos were sent. The daemon's
/// summary carries no such flag — it is the same shape either way.
pub(crate) dry_run: Cell<bool>,
}
/// Widgets the Import page hands back for wiring.
pub(crate) struct TakeoutWidgets {
pub(crate) list_group: adw::PreferencesGroup,
pub(crate) dropzone: gtk4::Box,
/// The whole page, which is the drop target: a drag released anywhere on it
/// counts, not only inside the dashed rectangle.
pub(crate) page: gtk4::Widget,
pub(crate) choose_button: gtk4::Button,
pub(crate) scan_button: gtk4::Button,
pub(crate) import_button: gtk4::Button,
pub(crate) cancel_button: gtk4::Button,
pub(crate) clear_button: gtk4::Button,
pub(crate) back_button: gtk4::Button,
pub(crate) progress_group: adw::PreferencesGroup,
pub(crate) progress_label: gtk4::Label,
pub(crate) progress_bar: gtk4::ProgressBar,
pub(crate) summary_group: adw::PreferencesGroup,
}
pub(crate) fn build_takeout_page() -> (gtk4::Widget, TakeoutWidgets) {
// Header: a back button out to Settings, since this page is reached from
// there and has no sidebar row of its own to navigate back with.
let back_button = gtk4::Button::builder()
.icon_name("go-previous-symbolic")
.tooltip_text("Back to Settings")
.valign(gtk4::Align::Center)
.build();
back_button.add_css_class("flat");
let title = gtk4::Label::builder()
.label("Import from Google Photos")
.halign(gtk4::Align::Start)
.build();
title.add_css_class("title-2");
let subtitle = gtk4::Label::builder()
.label("Add a Google Takeout export to your Proton Photos timeline.")
.halign(gtk4::Align::Start)
.wrap(true)
.xalign(0.0)
.build();
subtitle.add_css_class("dim-label");
let titles = gtk4::Box::new(gtk4::Orientation::Vertical, 2);
titles.set_hexpand(true);
titles.append(&title);
titles.append(&subtitle);
let header = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
header.append(&back_button);
header.append(&titles);
let header_clamp = adw::Clamp::builder().child(&header).build();
// Drop zone. It is a hint, not the only way in — the button under it does
// the same thing for anyone not dragging, and the drop target covers the
// whole page so a drag released just outside the rectangle still lands.
let drop_icon = gtk4::Image::from_icon_name("folder-download-symbolic");
drop_icon.set_pixel_size(48);
drop_icon.add_css_class("dim-label");
let drop_title = gtk4::Label::builder()
.label("Drop your Takeout .zip files here")
.build();
drop_title.add_css_class("title-4");
let drop_hint = gtk4::Label::builder()
.label("Add every part of the export at once — a photo and its metadata often sit in different parts.")
.wrap(true)
.justify(gtk4::Justification::Center)
.max_width_chars(48)
.build();
drop_hint.add_css_class("dim-label");
let choose_button = gtk4::Button::builder()
.label("Choose files…")
.halign(gtk4::Align::Center)
.build();
choose_button.add_css_class("pill");
let dropzone = gtk4::Box::new(gtk4::Orientation::Vertical, 10);
dropzone.set_halign(gtk4::Align::Fill);
dropzone.add_css_class("dropzone");
dropzone.append(&drop_icon);
dropzone.append(&drop_title);
dropzone.append(&drop_hint);
dropzone.append(&choose_button);
// Staged archives. Filled by `repaint_archives`, which owns the description
// too (it becomes the count + total once there is something to count); the
// group carries a placeholder row while empty so it never renders as a bare
// title.
let list_group = adw::PreferencesGroup::builder()
.title("Archives to import")
.description(EMPTY_LIST_DESCRIPTION)
.build();
let clear_button = gtk4::Button::builder()
.label("Clear")
.tooltip_text("Remove every staged archive")
.build();
clear_button.add_css_class("flat");
// Actions. Scan is a dry run: it reads the archives' directories and reports
// what an import *would* do, without uploading a byte. It is offered first
// because the import behind it is hours long and hard to take back.
let scan_button = gtk4::Button::builder()
.label("Scan")
.tooltip_text("Check the archives and report what would be imported, without uploading")
.build();
scan_button.add_css_class("pill");
let import_button = gtk4::Button::builder().label("Import").build();
import_button.add_css_class("pill");
import_button.add_css_class("suggested-action");
let cancel_button = gtk4::Button::builder()
.label("Stop import")
.tooltip_text("Finish the photo on the wire, file what was uploaded, then stop")
.build();
cancel_button.add_css_class("pill");
cancel_button.add_css_class("destructive-action");
cancel_button.set_visible(false);
// Clear sits apart from the launch buttons, on the other end of the row: it
// undoes staging, and putting it next to Import invites the wrong click.
let actions = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
let spacer = gtk4::Box::new(gtk4::Orientation::Horizontal, 0);
spacer.set_hexpand(true);
actions.append(&clear_button);
actions.append(&spacer);
actions.append(&cancel_button);
actions.append(&scan_button);
actions.append(&import_button);
// Progress: the import's job from the transfer snapshot, so this page shows
// the same numbers the Settings Activity list does.
let progress_group = adw::PreferencesGroup::builder()
.title("Import in progress")
.visible(false)
.build();
let progress_box = gtk4::Box::new(gtk4::Orientation::Vertical, 6);
progress_box.set_margin_top(8);
progress_box.set_margin_bottom(8);
let progress_label = gtk4::Label::builder().halign(gtk4::Align::Start).build();
progress_label.add_css_class("dim-label");
let progress_bar = gtk4::ProgressBar::new();
progress_box.append(&progress_label);
progress_box.append(&progress_bar);
progress_group.add(
&adw::PreferencesRow::builder()
.activatable(false)
.child(&progress_box)
.build(),
);
// Report for the last finished run, kept on screen after it ends: an import
// outlives the window being on this page, and "what happened" is the whole
// reason to come back to it.
let summary_group = adw::PreferencesGroup::builder()
.title("Last import")
.visible(false)
.build();
let groups = gtk4::Box::new(gtk4::Orientation::Vertical, 18);
groups.append(&dropzone);
groups.append(&list_group);
groups.append(&actions);
groups.append(&progress_group);
groups.append(&summary_group);
let clamp = adw::Clamp::builder()
.maximum_size(640)
.child(&groups)
.build();
let scroll = gtk4::ScrolledWindow::builder()
.vexpand(true)
.hscrollbar_policy(gtk4::PolicyType::Never)
.child(&clamp)
.build();
let page = gtk4::Box::new(gtk4::Orientation::Vertical, 12);
page.set_margin_top(18);
page.set_margin_bottom(18);
page.set_margin_start(18);
page.set_margin_end(18);
page.append(&header_clamp);
page.append(&scroll);
let widgets = TakeoutWidgets {
list_group: list_group.clone(),
dropzone: dropzone.clone(),
page: page.clone().upcast(),
choose_button,
scan_button: scan_button.clone(),
import_button: import_button.clone(),
cancel_button: cancel_button.clone(),
clear_button,
back_button,
progress_group: progress_group.clone(),
progress_label: progress_label.clone(),
progress_bar: progress_bar.clone(),
summary_group: summary_group.clone(),
};
(page.upcast(), widgets)
}
pub(crate) fn wire_takeout(ui: &Rc<Ui>, widgets: &TakeoutWidgets) {
// Drag-and-drop. The target sits on the whole page rather than the dashed
// rectangle, so a drag released anywhere counts; the rectangle only lights
// up to say where the drop is understood.
let target = gtk4::DropTarget::new(
gtk4::gdk::FileList::static_type(),
gtk4::gdk::DragAction::COPY,
);
let ui_enter = ui.clone();
target.connect_enter(move |_, _, _| {
ui_enter.takeout.dropzone.add_css_class("dropzone-active");
gtk4::gdk::DragAction::COPY
});
let ui_leave = ui.clone();
target.connect_leave(move |_| {
ui_leave
.takeout
.dropzone
.remove_css_class("dropzone-active");
});
let ui_drop = ui.clone();
target.connect_drop(move |_, value, _, _| {
ui_drop.takeout.dropzone.remove_css_class("dropzone-active");
let Ok(files) = value.get::<gtk4::gdk::FileList>() else {
return false;
};
let paths: Vec<PathBuf> = files.files().iter().filter_map(|f| f.path()).collect();
add_archives(&ui_drop, paths);
true
});
widgets.page.add_controller(target);
let ui_choose = ui.clone();
widgets.choose_button.connect_clicked(move |_| {
// Multi-select on purpose: the whole export goes in one request.
let dialog = gtk4::FileDialog::builder()
.title("Select your Google Takeout archives")
.build();
let filter = gtk4::FileFilter::new();
filter.set_name(Some("Takeout archives"));
filter.add_pattern("*.zip");
let filters = gio::ListStore::new::<gtk4::FileFilter>();
filters.append(&filter);
dialog.set_filters(Some(&filters));
let ui = ui_choose.clone();
dialog.open_multiple(
ui_window(&ui_choose).as_ref(),
gio::Cancellable::NONE,
move |res| {
let Ok(files) = res else { return };
let paths: Vec<PathBuf> = (0..files.n_items())
.filter_map(|index| files.item(index))
.filter_map(|object| object.downcast::<gio::File>().ok())
.filter_map(|file| file.path())
.collect();
add_archives(&ui, paths);
},
);
});
let ui_clear = ui.clone();
widgets.clear_button.connect_clicked(move |_| {
ui_clear.takeout.archives.borrow_mut().clear();
repaint_archives(&ui_clear);
});
let ui_scan = ui.clone();
widgets
.scan_button
.connect_clicked(move |_| start_import(&ui_scan, true));
// The import itself is confirmed: it uploads a whole photo library, which is
// hours of bandwidth and a large change to the account's timeline.
let ui_import = ui.clone();
widgets.import_button.connect_clicked(move |_| {
let count = ui_import.takeout.archives.borrow().len();
let ui = ui_import.clone();
let dialog = adw::AlertDialog::builder()
.heading("Import this export?")
.body(format!(
"{count} archive(s) will be uploaded to Proton Photos, creating albums as they \
appear in the export. This can take hours; you can stop it at any point and \
what has been uploaded stays. Photos you already have are skipped.",
))
.build();
dialog.add_response("cancel", "Cancel");
dialog.add_response("import", "Import");
dialog.set_response_appearance("import", adw::ResponseAppearance::Suggested);
dialog.set_default_response(Some("cancel"));
dialog.set_close_response("cancel");
dialog.connect_response(None, move |_, resp| {
if resp == "import" {
start_import(&ui, false);
}
});
dialog.present(ui_window(&ui_import).as_ref());
});
let ui_cancel = ui.clone();
widgets.cancel_button.connect_clicked(move |_| {
let ui = ui_cancel.clone();
ui.busy_begin();
let rx = spawn_request(ui.dirs.control_socket(), Request::CancelImport);
glib::spawn_future_local(async move {
let result = rx.recv().await;
ui.busy_end();
match result {
Ok(Ok(Response::Ok { .. })) => toast(&ui, "Stopping the import…"),
Ok(Ok(Response::Error { message, kind })) => {
toast_failure(&ui, "Couldn't stop the import", &message, kind);
}
_ => toast_error(
&ui,
"Couldn't stop the import",
"The mount service didn't respond.",
),
}
});
});
let ui_back = ui.clone();
widgets
.back_button
.connect_clicked(move |_| ui_back.stack.set_visible_child_name("main"));
// Restore whatever was staged when the window last closed, before the first
// paint, so the page comes up as the user left it.
*ui.takeout.archives.borrow_mut() = load_staged(ui);
repaint_archives(ui);
}
/// Stage `paths`, keeping only `.zip` files and skipping ones already staged.
///
/// Both the chooser and the drop target land here, so the filter is stated once.
/// A dropped folder (an already-extracted export) is rejected rather than walked:
/// the scan reads the archives' central directories, and an extracted tree has
/// none — silently importing half of it would be worse than saying no.
pub(crate) fn add_archives(ui: &Rc<Ui>, paths: Vec<PathBuf>) {
let mut added = 0usize;
let mut rejected = 0usize;
{
let mut staged = ui.takeout.archives.borrow_mut();
for path in paths {
let is_zip = path
.extension()
.is_some_and(|ext| ext.eq_ignore_ascii_case("zip"));
if !is_zip || !path.is_file() {
rejected += 1;
continue;
}
if staged.contains(&path) {
continue;
}
staged.push(path);
added += 1;
}
// Numbered parts drop in whatever order the file manager hands them
// over; sorting keeps the list readable and the request stable.
staged.sort();
}
repaint_archives(ui);
if rejected > 0 {
toast(
ui,
"Only Takeout .zip files can be imported — extracted folders aren't supported.",
);
} else if added > 0 {
toast(ui, &format!("Added {added} archive(s)"));
}
}
/// Rebuild the staged-archive list and re-derive which actions are available.
pub(crate) fn repaint_archives(ui: &Rc<Ui>) {
for row in ui.takeout.rows.borrow_mut().drain(..) {
ui.takeout.list_group.remove(&row);
}
let staged = ui.takeout.archives.borrow().clone();
let mut rows: Vec<adw::PreferencesRow> = Vec::new();
if staged.is_empty() {
ui.takeout
.list_group
.set_description(Some(EMPTY_LIST_DESCRIPTION));
let row = dim_row("No archives added yet.");
ui.takeout.list_group.add(&row);
rows.push(row.upcast());
} else {
let mut total = 0u64;
for path in &staged {
let size = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0);
total += size;
let row = adw::ActionRow::builder()
.title(
path.file_name()
.map(|n| n.to_string_lossy().to_string())
.unwrap_or_else(|| path.display().to_string()),
)
.subtitle(human_bytes(size))
.build();
let remove = gtk4::Button::builder()
.icon_name("edit-delete-symbolic")
.tooltip_text("Remove from this import")
.valign(gtk4::Align::Center)
.build();
remove.add_css_class("flat");
let ui_remove = ui.clone();
let target = path.clone();
remove.connect_clicked(move |_| {
ui_remove
.takeout
.archives
.borrow_mut()
.retain(|p| *p != target);
repaint_archives(&ui_remove);
});
row.add_suffix(&remove);
ui.takeout.list_group.add(&row);
rows.push(row.upcast());
}
ui.takeout.list_group.set_description(Some(&format!(
"{} archive(s), {} in total. {EMPTY_LIST_DESCRIPTION}",
staged.len(),
human_bytes(total)
)));
}
*ui.takeout.rows.borrow_mut() = rows;
save_staged(ui);
sync_takeout_actions(ui);
}
/// Enable exactly the actions that make sense right now: nothing to launch
/// without archives, nothing to launch while a run is in flight, and Stop only
/// while there is something to stop.
fn sync_takeout_actions(ui: &Rc<Ui>) {
let has_files = !ui.takeout.archives.borrow().is_empty();
let running = ui.takeout.running.get();
ui.takeout.scan_button.set_sensitive(has_files && !running);
ui.takeout
.import_button
.set_sensitive(has_files && !running);
ui.takeout.clear_button.set_sensitive(has_files && !running);
ui.takeout.cancel_button.set_visible(running);
}
/// Hand the staged set to the daemon. It acks immediately and works in the
/// background — an export is far past any socket timeout — so success here means
/// "started", and the page switches over to reporting progress.
fn start_import(ui: &Rc<Ui>, dry_run: bool) {
let archives: Vec<String> = ui
.takeout
.archives
.borrow()
.iter()
.map(|p| p.display().to_string())
.collect();
if archives.is_empty() {
return;
}
ui.takeout.dry_run.set(dry_run);
ui.busy_begin();
let rx = spawn_request(
ui.dirs.control_socket(),
Request::ImportTakeout { archives, dry_run },
);
let ui = ui.clone();
glib::spawn_future_local(async move {
let result = rx.recv().await;
ui.busy_end();
match result {
Ok(Ok(Response::Ok { .. })) => {
ui.takeout.running.set(true);
ui.takeout.summary_group.set_visible(false);
ui.takeout.progress_group.set_visible(true);
ui.takeout.progress_bar.set_fraction(0.0);
ui.takeout.progress_label.set_text(if dry_run {
"Reading archives…"
} else {
"Starting import…"
});
sync_takeout_actions(&ui);
toast(
&ui,
if dry_run {
"Scanning the export…"
} else {
"Import started — you can leave this page, it keeps running."
},
);
}
Ok(Ok(Response::Error { message, kind })) => {
toast_failure(&ui, "Couldn't start the import", &message, kind);
}
_ => toast_error(
&ui,
"Couldn't start the import",
"The mount service didn't respond.",
),
}
});
}
/// Poll the daemon for the import's state while the page is on screen, and paint
/// the report once a run has finished. Cheap and inflight-guarded, like the other
/// tick polls.
pub(crate) fn refresh_takeout(ui: &Rc<Ui>) {
if ui.takeout.inflight.get() {
return;
}
ui.takeout.inflight.set(true);
let rx = spawn_request(ui.dirs.control_socket(), Request::ImportStatus);
let ui = ui.clone();
glib::spawn_future_local(async move {
let result = rx.recv().await;
ui.takeout.inflight.set(false);
let Ok(Ok(Response::ImportStatus { running, summary })) = result else {
return;
};
let was_running = ui.takeout.running.replace(running);
if running {
ui.takeout.progress_group.set_visible(true);
} else {
ui.takeout.progress_group.set_visible(false);
// An import runs for hours, so its ending is announced rather than
// left to be discovered: a toast for whoever is looking at the
// window, and a desktop notification for whoever is not.
if was_running {
let dry = ui.takeout.dry_run.get();
let headline = if dry {
"Scan finished"
} else {
"Import finished"
};
toast(&ui, headline);
if !dry {
let body = match &summary {
Some(s) if s.failed > 0 => {
format!("{} photos added, {} failed.", s.uploaded, s.failed)
}
Some(s) if s.cancelled => {
format!("Stopped after {} photos.", s.uploaded)
}
Some(s) => format!("{} photos added to Proton Photos.", s.uploaded),
None => "The Google Photos import has finished.".to_string(),
};
notify("takeout-import", headline, &body);
}
}
}
if was_running != running {
sync_takeout_actions(&ui);
}
match summary {
Some(summary) => repaint_summary(&ui, &summary),
None => ui.takeout.summary_group.set_visible(false),
}
});
}
/// Paint the last run's report.
fn repaint_summary(ui: &Rc<Ui>, summary: &ImportSummary) {
for row in ui.takeout.summary_rows.borrow_mut().drain(..) {
ui.takeout.summary_group.remove(&row);
}
let dry = ui.takeout.dry_run.get();
ui.takeout
.summary_group
.set_title(if dry { "Scan result" } else { "Last import" });
ui.takeout.summary_group.set_description(Some(if dry {
"Nothing was uploaded — this is what an import would do."
} else if summary.cancelled {
"Stopped early. Everything uploaded so far is in your timeline; run it again to \
continue where it left off."
} else {
"Finished."
}));
// Only lines that carry information: a run with no failures should not have
// to show a "0 failed" row for the user to read past.
let mut lines: Vec<(String, String)> = vec![
("Photos in the export".into(), summary.found.to_string()),
(
if dry { "Would upload" } else { "Uploaded" }.into(),
summary.uploaded.to_string(),
),
];
if summary.duplicates > 0 {
lines.push((
"Already in Proton Photos".into(),
summary.duplicates.to_string(),
));
}
if summary.albums_created > 0 {
lines.push((
if dry {
"Albums to create"
} else {
"Albums created"
}
.into(),
summary.albums_created.to_string(),
));
}
if summary.album_links > 0 {
lines.push((
"Photos filed into albums".into(),
summary.album_links.to_string(),
));
}
if summary.skipped_trashed > 0 {
lines.push((
"Skipped (in Google's trash)".into(),
summary.skipped_trashed.to_string(),
));
}
if summary.bytes > 0 {
lines.push((
if dry {
"Data to upload"
} else {
"Data uploaded"
}
.into(),
human_bytes(summary.bytes),
));
}
if summary.failed > 0 {
lines.push(("Failed".into(), summary.failed.to_string()));
}
let mut rows = Vec::new();
for (title, value) in lines {
let row = adw::ActionRow::builder()
.title(title)
.subtitle(value)
.build();
row.add_css_class("property");
ui.takeout.summary_group.add(&row);
rows.push(row);
}
*ui.takeout.summary_rows.borrow_mut() = rows;
ui.takeout.summary_group.set_visible(true);
}
/// Paint the import's progress from the transfer snapshot the Settings tick
/// already fetched, so this page costs no extra poll. A job with no known total
/// pulses instead of showing a fraction, the same as the Activity list.
pub(crate) fn takeout_progress(ui: &Rc<Ui>, jobs: &[JobItem]) {
let Some(job) = jobs.iter().find(|job| job.title == IMPORT_JOB) else {
return;
};
ui.takeout.progress_group.set_visible(true);
if job.total > 0 {
ui.takeout
.progress_bar
.set_fraction((job.done as f64 / job.total as f64).min(1.0));
ui.takeout
.progress_label
.set_text(&format!("{} of {} — {}", job.done, job.total, job.detail));
} else {
ui.takeout.progress_bar.pulse();
ui.takeout
.progress_label
.set_text(if job.detail.is_empty() {
"Working…"
} else {
&job.detail
});
}
}
+28 -24
View File
@@ -122,7 +122,7 @@ pub(crate) fn wire_trash(ui: &Rc<Ui>, list: &gtk4::ListView, empty: &gtk4::Butto
factory.connect_setup(move |_, item| {
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
let icon = gtk4::Image::builder().pixel_size(32).build();
let thumbnail = file_thumbnail_widget(40, 32);
let name = gtk4::Label::builder()
.halign(gtk4::Align::Start)
.ellipsize(gtk4::pango::EllipsizeMode::Middle)
@@ -169,35 +169,38 @@ pub(crate) fn wire_trash(ui: &Rc<Ui>, list: &gtk4::ListView, empty: &gtk4::Butto
row.set_margin_bottom(6);
row.set_margin_start(6);
row.set_margin_end(6);
row.append(&icon);
row.append(&thumbnail);
row.append(&text);
row.append(&restore);
row.append(&purge);
item.set_child(Some(&row));
});
factory.connect_bind(|_, item| {
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
let Some(entry) = bound_entry(item) else {
return;
};
let row = item.child().and_downcast::<gtk4::Box>().unwrap();
let Some(icon) = row.first_child().and_downcast::<gtk4::Image>() else {
return;
};
icon.set_icon_name(Some(&format!("{}-symbolic", icon_base_for(&entry))));
let Some(text) = icon.next_sibling().and_downcast::<gtk4::Box>() else {
return;
};
if let Some(name) = text.first_child().and_downcast::<gtk4::Label>() {
name.set_label(&entry.name);
}
if let Some(meta) = text.last_child().and_downcast::<gtk4::Label>() {
let kind = if entry.is_dir {
"Folder".to_string()
} else {
human_bytes(entry.size)
factory.connect_bind({
let ui = ui.clone();
move |_, item| {
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
let Some(entry) = bound_entry(item) else {
return;
};
meta.set_label(&format!("{kind} · {}", format_modified(entry.modified)));
let row = item.child().and_downcast::<gtk4::Box>().unwrap();
let Some(thumbnail) = row.first_child().and_downcast::<gtk4::Overlay>() else {
return;
};
bind_file_thumbnail(&ui, &thumbnail, &entry, true);
let Some(text) = thumbnail.next_sibling().and_downcast::<gtk4::Box>() else {
return;
};
if let Some(name) = text.first_child().and_downcast::<gtk4::Label>() {
name.set_label(&entry.name);
}
if let Some(meta) = text.last_child().and_downcast::<gtk4::Label>() {
let kind = if entry.is_dir {
"Folder".to_string()
} else {
human_bytes(entry.size)
};
meta.set_label(&format!("{kind} · {}", format_modified(entry.modified)));
}
}
});
list.set_factory(Some(&factory));
@@ -216,6 +219,7 @@ pub(crate) fn bound_entry(item: &gtk4::ListItem) -> Option<DirEntry> {
/// Fetch the trash listing and repaint the page.
pub(crate) fn load_trash(ui: &Rc<Ui>) {
cancel_file_thumbnails(ui);
// Drop the old rows first: a stale row here would offer Restore on something
// that may already be gone.
ui.trash.model.remove_all();
+15 -16
View File
@@ -11,8 +11,8 @@ pub(crate) struct DetailsState {
pub(crate) details_suppress: Cell<bool>,
/// The grid's and list's selection models. Both wrap the one browser model, so
/// a selection in either drives the details pane.
pub(crate) grid_selection: gtk4::SingleSelection,
pub(crate) list_selection: gtk4::SingleSelection,
pub(crate) grid_selection: gtk4::MultiSelection,
pub(crate) list_selection: gtk4::MultiSelection,
}
/// The widgets in the browser's details pane that a selection repaints.
@@ -158,24 +158,22 @@ pub(crate) fn wire_details(ui: &Rc<Ui>) {
ui.details.list_selection.clone(),
] {
let ui_sel = ui.clone();
selection.connect_selection_changed(move |sel, _, _| {
match entry_at(sel.model().as_ref(), sel.selected()) {
Some(entry) => show_details(&ui_sel, &entry),
None => hide_details(&ui_sel),
selection.connect_selection_changed(move |_, _, _| {
// The details pane describes *one* entry. A batch is the bulk bar's
// business, so the pane steps aside rather than picking a member of
// the selection to speak for the rest.
let entries = selected_entries(&ui_sel);
match entries.as_slice() {
[entry] => show_details(&ui_sel, entry),
_ => hide_details(&ui_sel),
}
sync_bulk_bar(&ui_sel);
});
}
let ui_close = ui.clone();
ui.details.details.close_button.connect_clicked(move |_| {
ui_close
.details
.grid_selection
.set_selected(gtk4::INVALID_LIST_POSITION);
ui_close
.details
.list_selection
.set_selected(gtk4::INVALID_LIST_POSITION);
clear_selection(&ui_close);
hide_details(&ui_close);
});
@@ -289,8 +287,9 @@ pub(crate) fn hide_details(ui: &Rc<Ui>) {
*ui.details.details_entry.borrow_mut() = None;
}
/// The entry highlighted in whichever browser view is on screen, if any. Backs the
/// F2 / Delete shortcuts.
/// The single entry the details pane is showing, if any. Backs the actions that
/// only make sense on one entry (rename, versions); the batch-capable ones ask
/// [`selected_entries`] instead.
pub(crate) fn selected_entry(ui: &Rc<Ui>) -> Option<DirEntry> {
ui.details.details_entry.borrow().clone()
}
+1
View File
@@ -1,3 +1,4 @@
pub(crate) mod details;
pub(crate) mod share_dialog;
pub(crate) mod thumbnails;
pub(crate) mod versions_dialog;
@@ -0,0 +1,469 @@
//! Shared thumbnails for ordinary Drive image files.
//!
//! The Photos gallery has timeline-specific batching and aspect-ratio state.
//! File listings need a smaller, reusable surface: one loader backs My files,
//! search, Shared and Trash, with modification-time cache keys
//! so a replaced image never flashes its previous contents.
use crate::*;
use pdfs_core::control::{FileThumbRequest, is_thumbnail_image_name};
const THUMB_BATCH: usize = 32;
const THUMB_CACHE_MAX: usize = 256;
const THUMB_MISSING_MAX: usize = 2048;
const THUMB_DEBOUNCE: Duration = Duration::from_millis(40);
const THUMB_RETRY: Duration = Duration::from_secs(2);
pub(crate) struct FileThumbnailState {
textures: RefCell<HashMap<String, gtk4::gdk::Texture>>,
texture_order: RefCell<VecDeque<String>>,
missing: RefCell<HashSet<String>>,
missing_order: RefCell<VecDeque<String>>,
wanted: RefCell<HashMap<String, Vec<gtk4::Picture>>>,
queue: RefCell<VecDeque<FileThumbRequest>>,
inflight: Cell<bool>,
source: RefCell<Option<glib::SourceId>>,
generation: Cell<u64>,
generation_loading: Cell<bool>,
}
#[derive(Debug, PartialEq, Eq)]
enum ReplyDisposition {
Ignore,
Load,
Retry,
Missing,
}
fn reply_disposition(wanted: bool, has_path: bool, pending: bool) -> ReplyDisposition {
if !wanted {
ReplyDisposition::Ignore
} else if has_path {
ReplyDisposition::Load
} else if pending {
ReplyDisposition::Retry
} else {
ReplyDisposition::Missing
}
}
impl FileThumbnailState {
pub(crate) fn new() -> Self {
Self {
textures: RefCell::new(HashMap::new()),
texture_order: RefCell::new(VecDeque::new()),
missing: RefCell::new(HashSet::new()),
missing_order: RefCell::new(VecDeque::new()),
wanted: RefCell::new(HashMap::new()),
queue: RefCell::new(VecDeque::new()),
inflight: Cell::new(false),
source: RefCell::new(None),
generation: Cell::new(0),
generation_loading: Cell::new(false),
}
}
fn store_texture(&self, key: &str, texture: gtk4::gdk::Texture) {
let mut textures = self.textures.borrow_mut();
let mut order = self.texture_order.borrow_mut();
if textures.insert(key.to_string(), texture).is_none() {
order.push_back(key.to_string());
}
while order.len() > THUMB_CACHE_MAX {
if let Some(old) = order.pop_front() {
textures.remove(&old);
}
}
}
fn store_missing(&self, key: String) {
let mut missing = self.missing.borrow_mut();
let mut order = self.missing_order.borrow_mut();
if missing.insert(key.clone()) {
order.push_back(key);
}
while order.len() > THUMB_MISSING_MAX {
if let Some(old) = order.pop_front() {
missing.remove(&old);
}
}
}
}
/// Abandon every opportunistic ordinary-file thumbnail associated with the
/// listing being left. Advancing a wire generation lets the daemon cancel work
/// even when an older request reaches it after this one.
pub(crate) fn cancel_file_thumbnails(ui: &Rc<Ui>) {
if let Some(source) = ui.file_thumbs.source.borrow_mut().take() {
source.remove();
}
ui.file_thumbs.queue.borrow_mut().clear();
ui.file_thumbs.wanted.borrow_mut().clear();
let generation = ui.file_thumbs.generation.replace(0);
// The request is useful even when its reply is not. `spawn_request` owns the
// socket round-trip on a worker thread, so dropping the receiver is safe.
if generation != 0 {
drop(spawn_request(
ui.dirs.control_socket(),
Request::CancelFileThumbs {
generation: generation.saturating_add(1),
},
));
}
}
/// A square thumbnail surface with a generic file icon underneath the picture.
/// Call [`bind_file_thumbnail`] whenever a recycled row is bound to an entry.
pub(crate) fn file_thumbnail_widget(size: i32, fallback_size: i32) -> gtk4::Overlay {
let fallback = gtk4::Image::builder().pixel_size(fallback_size).build();
let picture = gtk4::Picture::builder()
.width_request(size)
.height_request(size)
.content_fit(gtk4::ContentFit::Cover)
.can_shrink(true)
.visible(false)
.build();
picture.add_css_class("file-thumbnail");
let overlay = gtk4::Overlay::new();
overlay.set_width_request(size);
overlay.set_height_request(size);
overlay.set_halign(gtk4::Align::Center);
overlay.set_valign(gtk4::Align::Center);
overlay.set_overflow(gtk4::Overflow::Hidden);
overlay.set_child(Some(&fallback));
overlay.add_overlay(&picture);
overlay
}
/// Resize an existing thumbnail surface. Grid rows are recycled, so changing
/// these requests in place and rebinding the model is enough to implement the
/// status-bar zoom without downloading or decoding an image again.
pub(crate) fn resize_file_thumbnail(widget: &gtk4::Overlay, size: i32, fallback_size: i32) {
widget.set_width_request(size);
widget.set_height_request(size);
let Some(fallback) = widget.child().and_downcast::<gtk4::Image>() else {
return;
};
fallback.set_pixel_size(fallback_size);
if let Some(picture) = fallback.next_sibling().and_downcast::<gtk4::Picture>() {
picture.set_width_request(size);
picture.set_height_request(size);
}
}
/// Bind a thumbnail surface to one entry, painting a cached texture immediately
/// or queueing an on-demand daemon request. Non-images retain the generic icon.
pub(crate) fn bind_file_thumbnail(
ui: &Rc<Ui>,
widget: &gtk4::Overlay,
entry: &DirEntry,
symbolic_fallback: bool,
) {
let Some(fallback) = widget.child().and_downcast::<gtk4::Image>() else {
return;
};
// The browser may add a status badge after the picture. Locate the picture
// immediately after the base fallback instead of assuming it is the last
// overlay child.
let Some(picture) = fallback.next_sibling().and_downcast::<gtk4::Picture>() else {
return;
};
let fallback_name = if symbolic_fallback {
format!("{}-symbolic", icon_base_for(entry))
} else {
icon_base_for(entry).to_string()
};
fallback.set_icon_name(Some(&fallback_name));
fallback.set_visible(true);
picture.set_visible(false);
picture.set_paintable(gtk4::gdk::Paintable::NONE);
// The widget name is a cheap binding token. List factories recycle their
// children; a reply for the previous entry must not paint into the new one.
let key = thumbnail_key(&entry.uid, entry.modified);
picture.set_widget_name(&key);
remove_waiter(ui, &picture);
if entry.is_dir || !is_image_name(&entry.name) || entry.uid.is_empty() {
return;
}
if let Some(texture) = ui.file_thumbs.textures.borrow().get(&key) {
paint(&picture, &key, texture);
return;
}
if ui.file_thumbs.missing.borrow().contains(&key) {
return;
}
ui.file_thumbs
.wanted
.borrow_mut()
.entry(key)
.or_default()
.push(picture);
let mut queue = ui.file_thumbs.queue.borrow_mut();
queue.retain(|item| item.uid != entry.uid);
queue.push_back(FileThumbRequest {
uid: entry.uid.clone(),
modified: entry.modified,
name: entry.name.clone(),
});
drop(queue);
schedule_file_thumbs(ui);
}
/// Create and immediately bind a thumbnail for a non-recycled row.
pub(crate) fn file_thumbnail(
ui: &Rc<Ui>,
entry: &DirEntry,
size: i32,
fallback_size: i32,
symbolic_fallback: bool,
) -> gtk4::Overlay {
let widget = file_thumbnail_widget(size, fallback_size);
bind_file_thumbnail(ui, &widget, entry, symbolic_fallback);
widget
}
/// Whether the file name denotes an image format for which Proton or the local
/// decoder can reasonably provide a thumbnail.
pub(crate) fn is_image_name(name: &str) -> bool {
is_thumbnail_image_name(name)
}
fn thumbnail_key(uid: &str, modified: i64) -> String {
format!("{uid}:{modified}")
}
fn remove_waiter(ui: &Rc<Ui>, picture: &gtk4::Picture) {
let mut wanted = ui.file_thumbs.wanted.borrow_mut();
wanted.retain(|_, pictures| {
pictures.retain(|candidate| candidate != picture);
!pictures.is_empty()
});
}
fn paint(picture: &gtk4::Picture, key: &str, texture: &gtk4::gdk::Texture) {
if picture.widget_name() != key {
return;
}
picture.set_paintable(Some(texture));
picture.set_visible(true);
if let Some(overlay) = picture.parent().and_downcast::<gtk4::Overlay>()
&& let Some(fallback) = overlay.child().and_downcast::<gtk4::Image>()
{
fallback.set_visible(false);
}
}
fn schedule_file_thumbs(ui: &Rc<Ui>) {
if ui.file_thumbs.queue.borrow().is_empty() || ui.file_thumbs.inflight.get() {
return;
}
if ui.file_thumbs.generation.get() == 0 {
reserve_file_thumb_generation(ui);
return;
}
if let Some(source) = ui.file_thumbs.source.borrow_mut().take() {
source.remove();
}
let ui_flush = ui.clone();
let source = glib::timeout_add_local_once(THUMB_DEBOUNCE, move || {
ui_flush.file_thumbs.source.borrow_mut().take();
flush_file_thumbs(&ui_flush);
});
*ui.file_thumbs.source.borrow_mut() = Some(source);
}
fn reserve_file_thumb_generation(ui: &Rc<Ui>) {
if ui.file_thumbs.generation_loading.replace(true) {
return;
}
let rx = spawn_request(
ui.dirs.control_socket(),
Request::ReserveFileThumbGeneration,
);
let ui = ui.clone();
glib::spawn_future_local(async move {
let result = rx.recv().await;
ui.file_thumbs.generation_loading.set(false);
match result {
Ok(Ok(Response::FileThumbGeneration { generation })) if generation != 0 => {
ui.file_thumbs.generation.set(generation);
schedule_file_thumbs(&ui);
}
_ => retry_file_thumb_generation(&ui),
}
});
}
fn retry_file_thumb_generation(ui: &Rc<Ui>) {
if let Some(source) = ui.file_thumbs.source.borrow_mut().take() {
source.remove();
}
let ui_retry = ui.clone();
let source = glib::timeout_add_local_once(THUMB_RETRY, move || {
ui_retry.file_thumbs.source.borrow_mut().take();
schedule_file_thumbs(&ui_retry);
});
*ui.file_thumbs.source.borrow_mut() = Some(source);
}
fn flush_file_thumbs(ui: &Rc<Ui>) {
if ui.file_thumbs.inflight.get() {
return;
}
let items: Vec<FileThumbRequest> = {
let mut queue = ui.file_thumbs.queue.borrow_mut();
let wanted = ui.file_thumbs.wanted.borrow();
let mut batch = Vec::new();
while batch.len() < THUMB_BATCH {
let Some(item) = queue.pop_front() else {
break;
};
if wanted.contains_key(&thumbnail_key(&item.uid, item.modified)) {
batch.push(item);
}
}
batch
};
if items.is_empty() {
return;
}
ui.file_thumbs.inflight.set(true);
let rx = spawn_request(
ui.dirs.control_socket(),
Request::FileThumbs {
items: items.clone(),
generation: ui.file_thumbs.generation.get(),
},
);
let ui = ui.clone();
glib::spawn_future_local(async move {
let result = rx.recv().await;
ui.file_thumbs.inflight.set(false);
match result {
Ok(Ok(Response::Thumbs { items: replies })) => {
let mut requested: HashMap<String, FileThumbRequest> = items
.iter()
.cloned()
.map(|item| (item.uid.clone(), item))
.collect();
let mut pending = Vec::new();
for reply in replies {
let Some(request) = requested.remove(&reply.uid) else {
continue;
};
let key = thumbnail_key(&request.uid, request.modified);
let wanted = ui.file_thumbs.wanted.borrow().contains_key(&key);
match reply_disposition(wanted, reply.path.is_some(), reply.pending) {
ReplyDisposition::Ignore => continue,
ReplyDisposition::Load => {
let Some(path) = reply.path else { continue };
match gtk4::gdk::Texture::from_filename(&path) {
Ok(texture) => {
ui.file_thumbs.store_texture(&key, texture.clone());
if let Some(pictures) =
ui.file_thumbs.wanted.borrow_mut().remove(&key)
{
for picture in pictures {
paint(&picture, &key, &texture);
}
}
}
Err(e) => {
tracing::debug!("cannot decode file thumbnail {path}: {e}");
ui.file_thumbs.store_missing(key.clone());
ui.file_thumbs.wanted.borrow_mut().remove(&key);
}
}
}
ReplyDisposition::Retry => pending.push(request),
ReplyDisposition::Missing => {
ui.file_thumbs.store_missing(key.clone());
ui.file_thumbs.wanted.borrow_mut().remove(&key);
}
}
}
pending.extend(requested.into_values());
if !pending.is_empty() {
retry_file_thumbs(&ui, pending);
}
}
Ok(Ok(Response::FileThumbsStale)) => {
ui.file_thumbs.generation.set(0);
retry_file_thumbs(&ui, items);
}
Ok(Ok(Response::Error { message, .. })) => {
tracing::debug!("file thumbnails failed: {message}");
retry_file_thumbs(&ui, items);
}
Ok(Ok(_)) | Ok(Err(_)) | Err(_) => {
tracing::debug!("file thumbnails: no reply");
retry_file_thumbs(&ui, items);
}
}
schedule_file_thumbs(&ui);
});
}
fn retry_file_thumbs(ui: &Rc<Ui>, items: Vec<FileThumbRequest>) {
let ui = ui.clone();
glib::timeout_add_local_once(THUMB_RETRY, move || {
let wanted = ui.file_thumbs.wanted.borrow();
let retry: Vec<FileThumbRequest> = items
.into_iter()
.filter(|item| wanted.contains_key(&thumbnail_key(&item.uid, item.modified)))
.collect();
drop(wanted);
ui.file_thumbs.queue.borrow_mut().extend(retry);
schedule_file_thumbs(&ui);
});
}
#[cfg(test)]
mod tests {
use super::{ReplyDisposition, is_image_name, reply_disposition};
#[test]
fn image_extensions_are_case_insensitive_and_specific() {
for name in [
"photo.jpg",
"SCAN.PNG",
"animation.Gif",
"archive.tiff",
"camera.nef",
"negative.CR3",
"camera.RAW",
] {
assert!(is_image_name(name), "{name}");
}
for name in [
"notes.txt",
"movie.mp4",
"image.jpg.zip",
"jpg",
"camera.heic",
"frame.avif",
"graphic.svg",
"camera.rw",
] {
assert!(!is_image_name(name), "{name}");
}
}
#[test]
fn cancelled_listing_replies_cannot_become_permanent_misses() {
assert_eq!(
reply_disposition(false, false, false),
ReplyDisposition::Ignore
);
assert_eq!(
reply_disposition(true, false, false),
ReplyDisposition::Missing
);
}
}
+3 -2
View File
@@ -153,8 +153,9 @@ Because Proton Drive does not support partial byte writes, modified files must b
2. **Close/Release (`queue_revision`):** When the application closes the file descriptor, the daemon:
- Fetches any untouched gaps from the remote base file to compile the full file.
- Durably publishes the scratch data and authored-range sidecar into `staging` under a `{uid}-{millis}-{counter}` name. Temporary data and metadata are synced before atomic rename, and their directory is synced before the source is removed.
- Transactionally queues or supersedes a pending database operation (`PendingOp`), so insertion failure cannot erase the previously acknowledged upload. For `OP_REVISION` ops, execution is debounced by 2 seconds (`DRAIN_REVISION_DEBOUNCE = 2s`) to give rapid follow-up writes (e.g. `aria2c` preallocation followed by writing) time to supersede the staged blob before network transmission.
3. **Async Drain Thread (`run_pending_drain`):** The background drain worker picks up the database operations queue, handles revision uploads, resolves conflicts, and cleans up staging files. Upon landing a revision upload (`refresh_after_upload`), it rebaselines both still-queued ops (`rebaseline_pending`) and open write handles targeting the same node to prevent false self-conflict copies on subsequent writes.
- Transactionally queues or supersedes a pending database operation (`PendingOp`), so insertion failure cannot erase the previously acknowledged upload. For `OP_REVISION` ops, execution is debounced to give rapid follow-up writes (e.g. `aria2c` preallocation followed by writing) time to supersede the staged blob before network transmission. The debounce is adaptive (`Core::revision_debounce`): it starts at `DRAIN_REVISION_DEBOUNCE = 2s` and, once a node has been uploaded, widens toward how long that upload actually took, bounded by `DRAIN_REVISION_DEBOUNCE_MAX = 60s`. A file saved faster than it can be sent therefore supersedes in the queue rather than mid-upload.
- Signals `Core::cancel_upload` for the node before touching the queue. A drain worker may already be reading the blob this write supersedes; the flag is read by the upload's `CountingReader`, which refuses the SDK's next block. `queue_trash` and `discard_queued_ops` do the same, because they unlink the blob outright.
3. **Async Drain Threads (`run_pending_drain`):** `DRAIN_WORKERS = 3` background workers share the operations queue. They pick work up through `Db::claim_next_due_op`, which marks the row `claimed_at` in the same transaction that selects it and excludes any op whose uid another worker already holds — ordering only has to hold *per node*, and that exclusion is what guarantees it. Worker 0 additionally runs the queue's idle chores (LRU touch flush, `recover_fsynced_writes`). A claim is process-local state: the single-writer `flock` means one found at open belongs to a crashed run, and `Db::clear_op_claims` drops the lot so those ops are not invisible forever. Each worker handles revision uploads, resolves conflicts, and cleans up staging files. Upon landing a revision upload (`refresh_after_upload`), it rebaselines both still-queued ops (`rebaseline_pending`) and open write handles targeting the same node to prevent false self-conflict copies on subsequent writes.
```mermaid
sequenceDiagram
+878 -19
View File
File diff suppressed because it is too large. Load diff
+517
View File
@@ -0,0 +1,517 @@
# Changelog
All notable changes to the Linux client since the 1.0.0 release. Format follows
[Keep a Changelog](https://keepachangelog.com/1.1.0/); versions follow semver, and every
release is tagged `vX.Y.Z` with `Cargo.toml` and `packaging/PKGBUILD` in agreement.
`Schema:` lines record the SQLite `SCHEMA_VERSION` (`pdfs-core/src/db/migrations.rs`) a
release ships. Migrations are forward-only — a database written by a newer client is a hard
refuse-to-open, not a downgrade, so rolling back a release means restoring the cache from
scratch (user data in `staging/` and `recovery/` is never touched by this).
## [1.9.0] — 2026-08-18
Google Photos import, and a pass over the desktop app's interaction gaps. No schema change.
Schema: **28**. Builds against the published `proton-sdk` / `proton-drive-rs` **0.6.1** — the album
writes this release needs are in the registry crates, so the local `[patch.crates-io]` shim the
feature was developed against is gone and no local path dependency remains.
### Added
- **Import a Google Photos Takeout export into Proton Photos** — `pdfs import-google-photos
takeout-*.zip` (add `--dry-run` to see what it would do, `--wait` to follow it), or the import
button in the Photos header, which multi-selects the export's zips. Albums in the export are
recreated as Proton albums and their photos filed into them; year folders and `Archive/` import
into the timeline; `Trash/` is skipped. Capture times and favourites come from Google's per-photo
JSON sidecars, including the truncated and `(N)`-swapped sidecar names Google produces.
- Photos already on the account are **not re-uploaded**: the import asks the photos volume which of
the export's names exist (batched, 150 per request), reads and SHA-1s only those files, and skips
the ones whose content digest matches too — Proton's own duplicate rule. A library already partly
synced by Proton Photos on a phone therefore imports only what is missing, and an interrupted
import resumes by being run again. Google keeps a full copy of a photo in every album folder it
appears in; those fold into one upload that joins several albums.
- The import streams each photo straight out of its zip, so an export needs no scratch space to
unpack into, and reports as a job in `pdfs transfers` and the GUI's activity view. `pdfs
import-status` shows the counts, `pdfs cancel-import` stops it — a cancelled run still files what
it uploaded into its albums.
- **An Import page in the GUI** (Settings → *Import from Google Photos*, or the import button in
the Photos header). Archives are staged by drag-and-drop onto the page or through the file
chooser, listed with their sizes so the set can be checked before anything is sent, and launched
as either a Scan (dry run) or a confirmed Import. The page shows live progress off the import's
job, keeps the finished run's report on screen, and can stop a run in flight; because the import
outlives the page, it is polled wherever the user navigates and its completion is reported as a
desktop notification.
- SDK (`proton-sdk-rs` **0.6.1**): album **writes** — `ProtonPhotosClient::create_album` and
`add_photos_to_album`, ported from the TypeScript SDK (upstream C# has no album write API), plus
batch duplicate detection (`find_duplicates_many`, `name_collisions`). The workspace deps moved
`"0.6.0"` → `"0.6.1"` and `Cargo.lock` re-resolved to the registry crates. Adding a photo from an
album *shared with us* (a different volume) is still unported and fails that photo's outcome.
- **Multi-select in Files.** Both views select a batch (click-drag, Ctrl-click, Shift-click); a bar
slides in reporting the count and offering Move to Trash and offline on/off for the whole
selection. Delete acts on everything selected, Esc clears it, and right-clicking inside a
selection offers the batch actions rather than the one row under the pointer. Actions that need a
single subject — rename, move, share, revisions — stay single-entry, and the details pane steps
aside for a batch instead of speaking for it.
- **Undo on trash.** "Moved *X* to Trash" now carries an Undo that restores the batch, single files
included.
- **The empty states are a place to act**: an empty folder offers Upload files / New folder, and an
empty photo timeline offers Upload photos / Import from Google Photos, instead of describing what
the user could go and do elsewhere.
- **Ctrl+F works from any page**, navigating to Files and focusing the search box, rather than being
silently inert everywhere but Files.
- Staged Takeout archives survive closing the window; archives that have since been moved or deleted
are dropped from the restored set.
### Fixed
- Settings showed the local cache as "X of 0 B used" when no cache budget was set. An unset budget
means *unlimited*, so it now reads "X cached — no limit set" rather than implying a zero cap.
- The cache-budget editor fired a `SetCacheBudget` round-trip — and a toast — on every step, so a
run of `+` clicks applied a series of caps the user never asked for and stacked a column of
toasts. It now waits for the value to settle.
- A long pin list pushed everything below it off the end of Settings. It shows the first six with a
"Show all N" row for the rest.
## [1.8.2] — 2026-08-17
Closes the two findings 1.8.1's live verification produced (B85, B86) and finishes the data-safety
audit entries that had been sitting as "fixed in code, verification pending" since 1.0 (B49–B55).
No schema change. Schema: **28**.
The read path no longer assumes a revision is a series of 4 MiB blocks; it takes the geometry from
the revision. The B85 investigation also **contradicted its own bug report** — see Notes, because
the conclusion recorded there was that reads of such files were losing data, and they were not.
### Fixed
- A folder switched from mirror to on-demand now shows the files the mirror pass itself uploaded.
The sync engine created nodes on the remote without invalidating the parent's cached child
listing, and that listing is a database flag rather than an in-memory cache — so it survived a
daemon restart, and the files stayed invisible until the folder was switched back. Every remote
mutation the engine makes (folder create, file upload, and the three trash paths) now drops the
parent's cached listing by uid, which reaches a folder that no mount has resident — the case a
mirror folder always is (`docs/BUGS.md` B86).
- `pdfs ls` and `pdfs refresh` accept a path under any mount the daemon owns, not only the primary
one. Both previously answered "is not under the mountpoint" for anything inside a secondary
on-demand mount, which left a folder serving a stale listing with no manual way to re-enumerate
it. A path under nothing this daemon has mounted still gets the same error as before.
- A file whose blocks are not 4 MiB is no longer read twice over. Block offsets came from
`index * 4 MiB`, so on a revision laid out differently every block the client asked for
straddled two of the revision's own, and each was fetched and decrypted to answer for one — then
cached at boundaries the next read would not ask about. Offsets now come from the revision's
recorded block sizes, and the on-disk cache, the in-memory ring and the in-flight fetch key
address a block by the byte range it covers rather than by its index (`docs/BUGS.md` B85).
### Changed
- The block cache records each block's plaintext start alongside its `(mtime, size)` tag, and each
revision's block table in a small sidecar beside its blocks. Blocks cached by an earlier release
carry no start; they are honoured wherever the old 4 MiB assumption and the new plan agree on
the offset, which is every file this client uploaded — so the existing cache survives the
upgrade intact and is quietly refetched only for the files it was placing wrongly.
### Notes
- **B85's diagnosis was wrong, and the entry has been corrected rather than closed.** It read a
set of journal lines as proof that this client was requesting wrong byte ranges and being saved
only by B84's repair path. `RevisionReader::read_at` does not work that way: it plans over the
revision's real block sizes and clamps every block to the range asked for, identically in the
0.6.0 the client builds against and in every earlier version. A read can therefore never return
more bytes than it requested — yet every quoted line shows exactly that (`blen=4194304
got=4939506`), and the one place in the client that logs that shape can only reach it on the
opposite condition. The numbers were either transcribed with two fields swapped or captured from
a build not in the history; they want re-capturing from the journal before anything is concluded
from them. What was real is the assumption itself, and that is what this release removes — a
cost fix, not a correctness one. Nothing here changes what a read returns.
- The first read of a file still plans on the 4 MiB assumption, and learns the real geometry as a
side effect of opening the reader it needs anyway. Resolving a reader merely to *plan* would
reintroduce the per-read key derivation the block cache exists to avoid, so the first read of a
non-4-MiB file pays one straddling fetch and then self-corrects.
- Seven data-safety audit entries (B49–B55) moved from "fixed in code, verification pending" to
verified. The fixes were all genuinely present and needed no change; what they lacked was the
injected-failure testing each entry asked for, and that is what was written — unwritable
directories and `chmod 000` subtrees for permission failures, a real `/dev/shm`↔`/tmp` device
boundary for the cross-filesystem fallbacks, SQLite's `max_page_count` for a full disk, an abort
trigger for a failed insert, and truncated and unpublished sidecars for the crash boundaries.
Two seams were added to make failure arrangeable at all: `stage_write_at`, mirroring the
`preserve_write_at` that already existed, and `walk_local_tree` lifted out of the sync engine so
the local scan can be driven against a real tree without a daemon.
- **Two of those entries are not fully closed, and say so.** B51 and B52 concern crash consistency,
and nothing short of a machine actually losing power tests that; the crash *boundaries* are
covered by reconstructing on disk exactly what a crash at each point leaves. B54 also diverges
from its required fix on purpose: it has no "deliberate delete" confirmation path, so a total
local wipe is refused outright rather than offered for confirmation. That is stricter than asked
for, and the cost is that a user who really did delete everything must remove and re-add the
sync folder.
- B86's fix routes only `ls` and `refresh` to the owning mount. The other path-addressed control
requests — pin, rename, move, delete, sharing — still resolve against the primary mount only.
That is a larger change and was not what blocked the repro; it is recorded as an open follow-up
(`docs/BUGS.md` B88).
- Three findings were opened rather than fixed: the geometry above is learned one read late, so a
non-4-MiB file pays a single wasted fetch on its first read (B87); the remaining path-addressed
control requests are still primary-only (B88); and the container-level `CLAUDE.md` still
describes the SDK dependency as 0.1.11 when 1.7.1 bumped it to 0.6.0, which is what sent the
B85 investigation to the wrong source tree to begin with (B89).
## [1.8.1] — 2026-08-16
Some files read short through the mount and reported success — an application got a truncated
copy with nothing anywhere saying so (`docs/BUGS.md` B84). Found because GNOME's file indexer
kept complaining that PNGs under `~/ProtonDrive/Pictures` were corrupt; it was right. The rest
of this release is the long-standing architectural backlog from the July concurrency audits
(B25, B32, B39, B40, B43, B44), each verified against the current code before being fixed.
Schema: **28** — migration V28 adds `sync_entry.local_mtime_ns`.
### Fixed
- A read no longer trusts a revision whose recorded block sizes disagree with the file's size.
Such a file leaves the block/range path entirely: it is fetched through the whole-file,
manifest-verified download — the same path `pdfs pin` uses, and the reason pinning was the
workaround — and served from the resulting cached blob. One download per file however many
reads discover the problem at once, and files above 512 MiB fail `EIO` with a note to pin
them rather than pulling down gigabytes for one read.
- A block that comes back shorter than the file's size implies is never served, cached or kept
in memory. Assembling a read out of short blocks is what quietly produced the truncation:
the missing bytes were simply not appended, the kernel handed userspace a short read, and
every tool reads a short read as end-of-file.
- A synced folder no longer reports the same deletion as an error on every pass. Deleting a
local file that something else had already deleted counted as a failure, which kept its
baseline and made the next pass try again — forever. A path that is already gone is the
state the sync pass wanted (`docs/BUGS.md` B42).
- A mirror folder no longer misses an edit made in the same second as its last sync. Change
detection compared whole seconds, so rewriting a file in place without changing its length
read as "unchanged" and never uploaded — the remote silently stayed behind until something
else about the file moved. Baselines now record the modification time to nanoseconds
(`docs/BUGS.md` B25).
- A sync download no longer overwrites an edit that finished while it was running. The
destination is re-checked in the last moment before the download is published, and a local
file that has moved on since the pass planned for it is kept as a conflict copy — the same
resolution a both-sides-changed conflict gets (`docs/BUGS.md` B39).
- An upload interrupted by a writer no longer reports itself as synchronized. The baseline now
records the content that actually went up rather than whatever is on disk when the transfer
ends, so a revision torn by a concurrent write is replaced by a clean one on the next pass
instead of standing as the file's content on Drive (`docs/BUGS.md` B40).
- Listing a directory that changes while it is being read no longer skips or repeats entries.
`readdir` resumed by index into a listing rebuilt on every call, so a create or a trash
between two pages shifted everything after it. The listing is now frozen for the life of the
directory handle (`docs/BUGS.md` B43).
- Opening a file for writing while a previous close is still publishing no longer discards that
close's revision. The base is re-checked at the moment the write handle is installed, rather
than trusted across a copy that can be the size of the whole file (`docs/BUGS.md` B32).
- Unmounting now stops and joins the daemon's background workers instead of leaving them
running. The drain, sync engine, conflict sweep, online probe, local indexer and control
listener share a stop signal and interruptible waits, so an in-process remount no longer
starts a second set on top of the first (`docs/BUGS.md` B44).
### Notes
- The size `stat` reports was always the correct one; it is the reader's belief about where
blocks end that was wrong, so nothing about reported sizes changes.
- Verified live on the affected account: four multi-block PNGs that had been reading short now
deliver their full size, and the journal shows each one repaired. The diagnostics also
answered the open question, and not as expected — the revision metadata is fine; those files
simply do not have 4 MiB blocks, and this client assumes they do. Filed as `docs/BUGS.md`
B85. Until that is fixed, a file with non-4-MiB blocks is downloaded whole on its first read.
- Migration V28's new column is nullable rather than converted from the existing seconds. A
baseline row written before this release has no sub-second time to recover, and inventing
zero for it would make every already-synced file compare as changed and re-upload an entire
mirror on first start; `NULL` keeps those rows comparing exactly as they did, and each gains
a real value the next time its path syncs.
- Three audit entries closed without a code change, having been fixed already or being correct
as they stand: sync-folder removal is serialized against reconciliation (B41), the drain's
database contention was removed by the read-only connection pool and the claim column (B30),
and the refusals for symlinks, hard links, device nodes and `chmod` are deliberate — Drive
cannot represent them, and a local-only representation no other Proton client can read would
be worse than a clean refusal (B31).
- Six of this release's fixes were then driven live against the installed 1.8.1 on a real
account: the same-second edit uploads (B25), three rapid rewrites of a 64 MiB file publish the
third one byte for byte (B32), a write racing a download survives as a conflict copy (B39), an
upload torn by a concurrent write heals itself on the following pass (B40), a `getdents64`
walk with a 160-byte buffer against a directory mutated between pages returns every entry
exactly once (B43), and a restart of a daemon with six mounts and a busy drain stops in 110 ms
and comes back at the same thread count (B44). B42 could not be driven from a single machine —
see B86 below.
- One new bug came out of the verification: an on-demand mount serves a folder listing that
predates the sync engine's own uploads, so files the mirror pass created are invisible until
the folder is switched back (`docs/BUGS.md` B86). Nothing is lost — they are on the server and
come back on the next mirror pass — but they cannot be seen or named while the folder is
on-demand, which is also what blocks B42's live verification.
## [1.8.0] — 2026-08-16
A queued write whose node had left the local tree was retried every five seconds forever and
counted as an ordinary pending upload — 28 of them, holding 18.4 GiB, had been doing that for
30 days on the machine that found it (`docs/BUGS.md` B83). Schema: 27 — migration V27 adds
`pending_op.access_deferred_since`.
### Fixed
- The drain no longer reads "I have never heard of this node" as "you may not write this
node". A missing authority is now resolved against the remote: the node is re-interned if
it is still there, and reported as a real failure if it is gone. Ops that had been stuck
invisibly for weeks now say what is wrong with them on the first pass.
- A create or mkdir resolves its *parent* as the authority, so the refetch asks about the
folder rather than the node it has not made yet.
### Added
- Access deferrals are bounded. A deferral still costs no attempt and records no error for
the first five minutes — the case it exists for is a share downgraded mid-queue and undone
a moment later — but past that window each recheck records a failure, so the op enters the
ordinary backoff and appears in `failing_ops` and the `pdfs status` error text like any
other stuck operation. The first deferral of a run is logged at `warn` rather than `debug`.
### Notes
- No staged data is touched by any of this. A reported failure keeps its `pending_op` row and
its staged blob exactly as before; the change is in what the user is told, and in the queue
no longer waiting on a permission change that was never coming.
## [1.7.1] — 2026-08-16
Write-back throughput follow-up to 1.7.0, same audit
(`docs/plans/audit-2026-08-15-perf-locks.md`). Schema: 26 — migration V26 adds
`pending_op.claimed_at`.
### Added
- Parallel write-back drain: three workers share the queue through a `pending_op.claimed_at`
claim column (migration V26), so a large upload no longer holds every queued rename, trash
and small write behind it. Ordering is preserved per node — the claim query never offers
two workers ops for the same uid — and claims left by a crashed run are cleared at open.
- An upload already on the wire is cancelled when a newer write, a trash or a discard
supersedes it, instead of spending the uplink on a revision the next operation replaces.
### Changed
- The revision debounce is adaptive: it widens toward how long that node's last upload
actually took (bounded to 2–60 s), so a file saved faster than it can be sent supersedes
in the queue rather than mid-upload.
- Workspace deps bumped to `proton-sdk` / `proton-drive-rs` 0.6.0.
## [1.7.0] — 2026-08-16
Performance and robustness pass over the mount, from the architecture audit in
`docs/plans/audit-2026-08-15-perf-locks.md`. Schema: 25.
### Added
- Sequential-read prefetch (`Core::prefetch_ahead`): per-file sequential detection, depth
doubling 2→8, reset to 0 after a seek, skipping blocks already cached, and bounded by its
own permit budget so it can never queue ahead of a demand read.
- Single-flight block fetching (`BlockFlight`) keyed on `(uid, mtime, size, block)`, so a
demand read, kernel read-ahead and prefetch of the same block share one download+decrypt.
- In-memory block LRU (`BlockRing`) in front of *both* block paths; a block read off disk is
promoted into it, so the following ~31 kernel reads of that 4 MiB block cost nothing.
- Read-only SQLite connection pool (`ReadPool` / `Db::read`) with every `SELECT`-only method
routed at it, so a read no longer waits on the write connection.
- `nodes.path` persisted and backfilled, replacing the recursive `path_of` walk.
- Single-writer `flock` on the database; a `SQLITE_CORRUPT`/`NOTADB` file is quarantined as
`cache.db.corrupt-<ts>` (WAL sidecars too) and rebuilt rather than failing the mount.
- `VACUUM` and the deep `integrity_check` now ack immediately and run as background jobs;
`pdfs doctor` polls for the verdict.
- Staging reconcile at mount: staged blobs no queued operation refers to are re-queued, or
named in the log when they cannot be addressed.
- Queue/staging visibility — `Status` gained `parked_uploads`, `failing_ops`,
`failing_error`, `staged_bytes`, `staged_oldest_secs`; `pdfs status` prints them when
non-zero.
- `statfs` implemented against the account quota; FUSE `init` negotiates `max_readahead`,
`max_background`, `PARALLEL_DIROPS` and `CACHE_SYMLINKS`; `blksize` raised to 1 MiB.
### Changed
- `State` no longer writes to SQLite under the inode lock — mutations queue into an outbox
applied after the lock is released, so interning a 5000-child listing no longer commits a
transaction with the whole mount frozen.
- Re-listing an unchanged folder no longer touches its subtree; when a subtree really moved,
one recursive walk carries both the new path and the inherited trashed flag.
- `release(2)` no longer gap-fills from the network: it stages an incomplete blob from local
caches only and the drain thread finishes it off the dispatch loop.
- A partial write over an undrained one is merged instead of refused.
- Size-upgrade waiters park on a queue instead of occupying a worker thread, capped at 8
folders in flight; past the cap `stat` answers provisionally.
- Block-cache disk writes moved off the read's critical path; no more fsync per cached block.
- Cache LRU touches buffered in memory and flushed in one transaction.
- Search: sync-folder list and pin lookups hoisted out of the per-hit loop.
- SQLite tuned (`cache_size`, `mmap_size`, `temp_store=MEMORY`, `wal_autocheckpoint`) and
three new indexes added.
- Mount registry lock released before FUSE teardown/join; kernel notifications batched under
the state lock and flushed after it.
### Fixed
- **Data loss:** `ENOSPC` is now propagated from `write`/`fsync`/staging instead of being
swallowed, with emergency eviction wired up and scratch marked durable before any staging
error; a failed `record_pending_write` fails `release` rather than dropping the write.
- **Data loss:** the conflict sweep no longer discards queued operations after the trash
round trip.
- **Data loss:** `rescue_scratch` moves a blob with an unreadable sidecar to `recovery/`
unaccompanied instead of fabricating a sidecar for it.
- Own-sealed revisions are persisted (7-day TTL), so a restart no longer reopens the
transient-download fork window fixed in 1.1.0.
- Depth-capped `is_pinned` (cycle-safe), daemon write timeout, and every client-supplied
`limit` clamped.
- `recover_fsynced_writes` no longer scans the recovery directory on every idle pass.
## [1.6.0] — 2026-08-12
Search-index correctness and ranking (`docs/plans/search-index.md`). Schema: 21 —
migration V21 rebuilds `nodes_fts`.
### Fixed
- **B80** — 29% of the account was missing from the search index. `node_is_indexable_tx`
required a null-parent ancestor, which only the My Files root has, so every node under a
device folder was dropped from `nodes_fts` on its first upsert after the v16 backfill
(2,705 of 9,252 nodes on the account this was diagnosed against). The rule now matches the
backfill, migration V21 replays it, nodes under a trashed folder stay out, and the walk is
depth-capped.
- **B81** — the prompt labelled every Drive hit "My files", including device folders; pin
rows also took `is_dir` from `Pin::recursive`.
- **B82** — a `parent_uid` cycle hung the daemon on any short search: the recursive CTEs in
`path_of` and the pin walk use `UNION ALL`, which does not deduplicate. `path_of` is now
depth-capped at 256 and returns the truncated path.
### Changed
- Local search ranking: whole-query prefix matches get their own lane (newest first) ahead
of trigram rank, which on a home directory full of dependency trees was burying the
obvious answer — 73% of the 500 best-ranked candidates for "test" came from one Go module
cache. The single-character lane, which recovers a typo that destroys every trigram, stays
behind it.
## [1.5.0] — 2026-08-11
Schema: 20.
### Added
- **`pdfs-prompt --dmenu`** — the same search through an external dmenu-style launcher
(fuzzel, rofi, wofi, tofi, bemenu, dmenu) instead of the built-in GTK HUD, so the prompt
matches a tiling-WM desktop. A launcher filters a fixed list rather than re-querying per
keystroke, so searching is a two-step loop; `--query` skips the first step, fuzzel and rofi
get file-type icons. Configurable as `"prompt": { "mode": "dmenu", "menu": [...] }`, with
`--gtk` overriding for one invocation.
- **Configurable open-with rules** (`pdfs-core/src/opener.rs`): an ordered list of name/class
patterns in `config.json` under `open_with`, each mapped to argv and optionally wrapped in
a terminal, first match wins, falling back to `xdg-open`. Lets "open a text file" mean
`nvim` in a terminal without breaking every graphical caller. Absent config reproduces the
old behaviour exactly.
### Fixed
- The prompt and dmenu openers hand applications the cached file while matching rules
against the *original* Drive filename, so e.g. `.md` files stop being routed by their
cache-path extension.
## [1.4.0] — 2026-08-10
Schema: 20.
### Added
- **File version history** (`pdfs-fuse/src/revisions.rs` + a GTK versions dialog): list a
file's revisions, restore one, delete one, or write an old one out to a local path. A
restore re-points the file server-side, so no content crosses the wire and nothing enters
the drain queue; the daemon drops its cached content and repopulates on the next read.
- **Persistent SDK entity cache** (`pdfs-core/src/sdkcache.rs`): the SDK's decrypted node
metadata now survives a daemon restart instead of re-walking the tree. It lives in its own
`sdk_cache.db`, deliberately off the daemon's hot `cache.db` mutex, holds no key material,
needs no migrations, and is cleared on logout.
- Per-node outcomes for batch operations (`pdfs-core/src/batch.rs`): `trash`/`restore`/
`delete`/`move` report one result per node, so a node the server rejects no longer fails
the whole batch.
- Photo viewer page in the GUI and richer photo metadata in the details pane.
### Changed
- SDK telemetry routed into `tracing`, so its transfer/block-storage/request spans land in
the daemon log instead of a no-op sink.
- A file that fits in one block uploads as a single atomic request rather than the
draft/block/commit sequence.
## [1.3.0] — 2026-07-31
Schema: 19.
### Added
- **Photo albums** — `PhotoAlbums` / `AlbumPhotos` control requests, daemon-side `albums.rs`
and a GUI albums page (the former gallery view, renamed). Albums other people share with
you are included; their photos are persisted per album because a shared album's photos
never appear in your own timeline. Served stale-while-refreshing like the timeline.
- **GUI "Locations" page** — one row per mount spec: the primary `~/ProtonDrive` mount
alongside every folder this computer backs up, mirrored or on-demand. Called *Locations*
rather than *Mounts* because a mirror folder is a plain directory with no FUSE session.
It absorbs the old Computers page's actions and the mountpoint chooser from Settings.
- `role` on shared-node metadata (`viewer`/`editor`/`admin`), defaulted for wire-compat with
older clients and daemons.
- FUSE acceptance coverage for the permission-based mounts.
## [1.2.1] — 2026-07-31
Schema: 18.
### Fixed
- Migration bug that could leave the `ProtonDrive` mount unwritable after upgrade.
- Permission error on on-demand mounts.
- B78.
### Changed
- Release pipeline: `.deb` signing added and `.rpm` signing fixed.
## [1.2.0] — 2026-07-29
Sharing and multi-mount release. Schema: 18.
### Added
- **Shared with me**: shares others sent you are listed, verified and mountable, including on
the primary mount.
They appear under a synthetic `Shared with me` directory on a virtual volume
(`pdfs-fuse/src/virtual.rs`), with name collisions and over-long components resolved by
its own naming rules.
- Per-share access model (`db/share_access.rs`): the effective role of each shared root is
persisted and kept resident in mount state, so interning an inode never costs a query. The
database stays the restart/offline authority — every role change is written through before
the cache updates. A share you cannot write is presented POSIX read-only.
- Unified location model (`pdfs-core/src/mounts.rs`, `db/mounts.rs`): one presentation shape
for every local Proton Drive location, joining device sync state with a new `mount` table.
`MountMode` is `Mirror` (full local copy, reconciled both ways) or `OnDemand` (FUSE session
fetching on access), with an `Unknown` fallback so a mode written by a newer client does
not break an older one.
## [1.1.1] — 2026-07-27
Schema: 16.
### Fixed
- Trash refresh did not pull updates.
- Packaging: `PKGBUILD` was left at 1.1.0 by the 1.1.1 version bump, failing the release
workflow's tag/manifest/PKGBUILD agreement check.
## [1.1.0] — 2026-07-26
Conflict-handling and concurrency release. Schema: 16.
### Added
- Conflict sweep (`pdfs-fuse/src/sweep.rs`, 5-minute cadence): removes conflict copies proven
identical by size + `content_sha1`, surfaces divergent ones once as an activity entry, and
never removes what it cannot prove.
- Sweep mode and `.pdfsignore`-driven conflict handling; ignore rules are rebuilt per sync
pass so edits take effect without a restart.
- Substantially extended FUSE acceptance suite.
### Fixed
- **B69** — spurious `(sync-conflict)` copies. Revision identity now uses the server's
`revision_id` rather than `(mtime, size)`, which the server re-stamps on seal; the planner
gained `AdoptBaseline` so an equal-size no-baseline pair is recorded instead of forked on
first reconcile.
- **B70** — browser downloads into an on-demand mount had their in-flight temporaries sealed
as canonical revisions. Transient names are recognised and held until the name is finalized.
- Assorted race conditions and concurrency bugs on the write/drain paths.
### Changed
- Workspace deps bumped to `proton-sdk` / `proton-drive-rs` 0.2.2.
## [1.0.0] — 2026-07-22
First stable release: FUSE files-on-demand mount, sync daemon under `proton-drive.service`,
`pdfs` CLI and the GTK4 app, on top of the pure-Rust Proton Drive SDK. Schema: 16.
### Fixed
- The outstanding FUSE defects tracked in `docs/BUGS.md`, plus a truncate defect, validated
by a new POSIX compliance suite for the filesystem.
[1.8.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.8.2
[1.8.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.8.1
[1.8.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.8.0
[1.7.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.7.1
[1.7.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.7.0
[1.6.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.6.0
[1.5.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.5.0
[1.4.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.4.0
[1.3.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.3.0
[1.2.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.2.1
[1.2.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.2.0
[1.1.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.1.1
[1.1.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.1.0
[1.0.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.0.0
+2 -1
View File
@@ -14,7 +14,8 @@ Consolidated from the historical feature roadmap; current bug status lives in [`
* **Custom FUSE Mount Options (Horizon 1)**: Expose custom mounting parameters in configuration (e.g. `ro` for read-only, `direct_io` to bypass kernel caching, `allow_other`). Auto-generate systemd user mount units upon sync folder configuration.
* **Symbolic Link & Hard Link Virtualization (Horizon 2)**: Add symbolic link virtualization inside the metadata store (`NodeType::Symlink`) by intercepting `readlink(2)` and `symlink(2)`. Targets should sync remotely as small encrypted metadata placeholder files on Drive (`proton-vfs-symlink: <target>`).
* **Desktop File Manager Integration (Horizon 2)**: Develop shell extension plugins for popular Linux file managers (Nautilus Python plugin, Dolphin Service Menus, Thunar custom actions) to offer context-menu shortcuts ("Pin", "Unpin", "Copy Share Link", "View Version History") and file overlay sync badges.
* **Integrated Photo Gallery Enhancements (Horizon 2)**: Support creating and editing Proton Photo Albums (album *writes* are unported in the SDK too) and create an auto-upload Pictures folder pipeline. Exif display, the date-based timeline scrubber and favourites are implemented.
* **Integrated Photo Gallery Enhancements (Horizon 2)**: Album *editing* — rename, cover photo, delete, and removing photos from an album — is still unported in the SDK (TS `updateAlbum` / `deleteAlbum` / `removePhotos`), as is adding a photo that lives on someone else's volume (TS `copyPhoto`). Album **creation** and adding own-volume photos landed with the Google Photos import. Still open: an auto-upload Pictures folder pipeline, and a GUI for creating albums by hand. Exif display, the date-based timeline scrubber and favourites are implemented.
* **Google Photos import follow-ups**: the importer uploads photos one at a time — a large export would finish sooner with a bounded concurrent upload pool. Trash folders are recognized by a hardcoded list of localized names ([`takeout.rs`](../crates/pdfs-core/src/takeout.rs) `TRASH_FOLDERS`); a locale not on it imports its trash as ordinary photos. Motion-photo `.MP`/`.MV` sidecar videos are ignored rather than uploaded as related photos. Live validation against a real export is pending.
* **Random-Access Media Streaming seek support (Horizon 2)**: Intercept out-of-order sparse reads (player seeks) in `pdfs-fuse` and prioritize downloading blocks around the seek offset, aborting/postponing sequential pre-fetches.
* **Dynamic Sync Dashboard & Queue Visualization (Horizon 2)**: GUI real-time transfer list (progress bars, speed, ETA), sync history feed, storage usage breakdown, and global pause/resume sync controls.
* **Interactive Terminal UI (TUI) Mode (Horizon 3)**: Implement `pdfs tui` using the `ratatui` crate to show transfer speed, sync status, active transfer queue with progress bars, and scrolling daemon logs.
-110
View File
@@ -1,110 +0,0 @@
# Search index + prompt display fixes
Working plan. Findings come from an audit of a real `cache.db`
(9,252 Drive nodes, 110,187 local files) on 2026-08-12.
## Evidence
```
nodes 9252
nodes_fts 6547 ← 2705 rows never indexed
nodes in orphan subtrees 2705 ← exactly the missing set
local_files 110187
/home/narl/go/pkg/mod 63399 (58% of the local index)
query "test" 46405 local matches, candidate pool capped at 1000
top 500 by fts rank 367 (73%) from go/pkg/mod
```
`pdfs search tickets` returns only `Documents/Tickets` (My Files); the
`tickets.pdf` sitting in the device folder `Downloads` is invisible.
## P1 — 29% of Drive is unsearchable
`node_is_indexable_tx` (`crates/pdfs-core/src/db/nodes.rs:606`) indexes a node
only when the ancestor walk terminates at a row with `parent_uid IS NULL`. The
DB holds exactly one such row (the My Files root). Device roots live in
`device`/`sync_folder`, never in `nodes`, so every device-folder subtree
(`Downloads`, `Pictures`, `Documents`, `Music`, `Videos` under device
*nthinkpad*) walks up to a `parent_uid` with no row and is judged unindexable.
Stale `pdfs-acceptance-*` roots hit the same path.
- [x] Relax the rule to "the walk terminates and no ancestor is trashed",
keeping the existing cycle guard. A subtree whose top row's parent is
simply not cached is still real, reachable data.
- [x] `MIGRATION_V21`: rebuild `nodes_fts` under the new rule, so existing
installs recover the missing rows instead of waiting for each node to be
upserted again.
- [x] Tests: a node under an uncached parent is indexed; a trashed ancestor
still excludes; the migration backfills.
### Found while testing: a parent cycle hangs the daemon
Writing the cycle test surfaced a live defect, not a test artefact. A query
below the trigram minimum takes the `LIKE` lane, which reads `nodes` directly
and never consults the index, and every row it returns goes through `path_of` —
whose parent walk is an unguarded `UNION ALL`. On a `parent_uid` cycle it never
terminates, while holding the daemon's only SQLite connection.
- [x] Depth-cap `path_of` (`db/utils.rs`), matching the 1024 cap
`path_relative_to` already carried.
- [x] Depth-cap the `descendants` CTE in `upsert_node_tx`, which has the same
exposure on the write side.
- [x] Keep cycles out of the index (part of the P1 rule above).
Still unguarded, not touched here: `pin_is_pinned`'s ancestor walk.
## P2 — local index noise crowds real files out of the candidate pool
`SKIP_DIRS` (`crates/pdfs-core/src/localindex.rs:23`) misses the Go module
cache and build output dirs. `search_v2` clamps candidates to ≤1000 and scores
*after* SQLite, so for any common term the pool is exhausted by vendored files
before a real one is considered.
- [x] Skip `go/pkg/mod` (path-relative, not a bare name), `dist`, `build`,
`.cache`-style artefact dirs.
- [x] Give `search_local_candidates` a name-prefix lane, mirroring what
`search_candidates` already does for Drive, so an exact-prefix match
cannot be starved by trigram rank.
- [x] Tests for both.
## P3 — display lies about where a hit lives
- [x] `Hit::location()` (`crates/pdfs-gui/src/prompt.rs:150`) hardcodes
`My files / …` for every Drive hit. Derive the label from
`mounted_path` when the daemon resolved one (device folder, sync folder)
and fall back to `My files` only for the primary mount.
- [x] Pin rows set `is_dir: pin.recursive` (`dmenu.rs:133`, `prompt.rs:872`) —
a non-recursively pinned folder renders and activates as a file.
- [ ] `pdfs locations` prints `Device folder 1..5` rather than the device and
folder name. (Cosmetic; deferred.)
## Verification
Done, 2026-08-12:
```
cargo fmt --all -- --check clean
cargo clippy --workspace --all-targets -- -D warnings clean
cargo test --workspace --locked 439 passed, 0 failed
```
The V21 backfill was also replayed against a read-only `.backup` of the live
`cache.db`:
```
live_nodes 9276
indexed 9276 (was 6547)
downloads_subtree 300
tickets.pdf -> Downloads/tickets.pdf
```
That run is what caught the last defect: the backfill originally seeded an
uncached-parent root with an empty path, so a node indexed by the migration
(`tickets.pdf`) and the same node after a later upsert (`Downloads/tickets.pdf`)
disagreed. The seed now keeps the name for any row that has a parent, matching
`path_of`.
Still to do, on the live system: rebuild, restart `proton-drive.service`, and
confirm `pdfs search tickets` returns the device folder's `tickets.pdf`. The
schema move to 21 is one-way — an older `pdfs` refuses a v21 database. The
`local_files` shrink (~110k → ~47k) lands with the next background scan.
+26
View File
@@ -0,0 +1,26 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="128" height="128" viewBox="0 0 128 128" fill="none" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Proton Drive for Linux">
<defs>
<linearGradient id="pdl-bg" x1="0" y1="0" x2="0" y2="128" gradientUnits="userSpaceOnUse">
<stop offset="0" stop-color="#8B72FF"/>
<stop offset="1" stop-color="#5A38E8"/>
</linearGradient>
</defs>
<rect width="128" height="128" rx="29" fill="url(#pdl-bg)"/>
<!-- cloud: the remote drive (symmetric about x=64) -->
<g fill="#FFFFFF">
<circle cx="64" cy="42" r="20"/>
<circle cx="44" cy="52" r="14"/>
<circle cx="84" cy="52" r="14"/>
<rect x="32" y="50" width="64" height="16" rx="8"/>
</g>
<!-- pull: files arrive only when opened -->
<g stroke="#FFFFFF" stroke-width="8" stroke-linecap="round" stroke-linejoin="round">
<path d="M64 75v17"/>
<path d="M53 84l11 11 11-11"/>
<path d="M45 108h38" stroke-opacity="0.55"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 990 B

+4 -4
View File
@@ -9,13 +9,13 @@
# Build + install: cd packaging && makepkg -fi
pkgname=proton-drive-linux
pkgver=1.7.0
pkgver=1.9.1
pkgrel=1
pkgdesc="Proton Drive client for Linux: FUSE files-on-demand mount, CLI, GTK4 app + tray"
pkgdesc="Unofficial Proton Drive client for Linux: files-on-demand FUSE mount, CLI, GTK4 app + tray"
arch=('x86_64')
url="https://github.com/narl/proton-drive-linux"
url="https://github.com/narrrl/proton-drive-linux"
license=('MIT')
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0')
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0' 'perl-image-exiftool')
makedepends=('cargo' 'git')
provides=('pdfs')
# Files-on-demand mount + keyring need these at runtime.
@@ -1,11 +1,11 @@
[Desktop Entry]
Type=Application
Name=Proton Drive (Tray)
Name=Proton Drive for Linux (Tray)
GenericName=Cloud Drive Tray
Comment=Proton Drive system tray icon
Comment=Proton Drive status and quick actions (unofficial client)
Exec=pdfs-tray
Icon=io.narl.proton-drive-linux
Terminal=false
Categories=Network;FileTransfer;GTK;
Keywords=proton;drive;cloud;sync;storage;
Keywords=proton;drive;cloud;sync;storage;files;backup;
NoDisplay=true
+3 -3
View File
@@ -1,12 +1,12 @@
[Desktop Entry]
Type=Application
Name=Proton Drive
Name=Proton Drive for Linux
GenericName=Cloud Drive
Comment=Proton Drive files-on-demand for Linux
Comment=Files-on-demand Proton Drive (unofficial client)
Exec=pdfs-app
Icon=io.narl.proton-drive-linux
Terminal=false
Categories=Network;FileTransfer;GTK;
Keywords=proton;drive;cloud;sync;storage;
Keywords=proton;drive;cloud;sync;storage;files;backup;
StartupNotify=true
SingleMainWindow=true
+21 -9
View File
@@ -1,14 +1,26 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="128" height="128" viewBox="0 0 128 128" xmlns="http://www.w3.org/2000/svg">
<svg width="128" height="128" viewBox="0 0 128 128" fill="none" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Proton Drive for Linux">
<defs>
<linearGradient id="g" x1="0" y1="0" x2="0" y2="1">
<stop offset="0" stop-color="#8b6dff"/>
<stop offset="1" stop-color="#6d4aff"/>
<linearGradient id="pdl-bg" x1="0" y1="0" x2="0" y2="128" gradientUnits="userSpaceOnUse">
<stop offset="0" stop-color="#8B72FF"/>
<stop offset="1" stop-color="#5A38E8"/>
</linearGradient>
</defs>
<rect x="8" y="8" width="112" height="112" rx="26" fill="url(#g)"/>
<!-- cloud -->
<path fill="#ffffff" d="M44 84c-11 0-20-9-20-20 0-10 7-18 17-20 3-11 13-19 25-19 13 0 24 10 26 23 9 1 16 9 16 18 0 10-8 18-18 18z"/>
<!-- down arrow (on-demand pull) -->
<path d="M64 50v22m0 0l-9-9m9 9l9-9" stroke="#6d4aff" stroke-width="7" stroke-linecap="round" stroke-linejoin="round" fill="none"/>
<rect width="128" height="128" rx="29" fill="url(#pdl-bg)"/>
<!-- cloud: the remote drive (symmetric about x=64) -->
<g fill="#FFFFFF">
<circle cx="64" cy="42" r="20"/>
<circle cx="44" cy="52" r="14"/>
<circle cx="84" cy="52" r="14"/>
<rect x="32" y="50" width="64" height="16" rx="8"/>
</g>
<!-- pull: files arrive only when opened -->
<g stroke="#FFFFFF" stroke-width="8" stroke-linecap="round" stroke-linejoin="round">
<path d="M64 75v17"/>
<path d="M53 84l11 11 11-11"/>
<path d="M45 108h38" stroke-opacity="0.55"/>
</g>
</svg>

Before

Width:  |  Height:  |  Size: 718 B

After

Width:  |  Height:  |  Size: 990 B

+6 -4
View File
@@ -12,9 +12,9 @@
Name: proton-drive-linux
Version: %{version}
Release: 1%{?dist}
Summary: Proton Drive client for Linux (FUSE, CLI, GTK4 app + tray)
Summary: Unofficial Proton Drive client for Linux (FUSE, CLI, GTK4 app + tray)
License: MIT
URL: https://github.com/narl/proton-drive-linux
URL: https://github.com/narrrl/proton-drive-linux
ExclusiveArch: x86_64
BuildRequires: cargo
@@ -33,6 +33,7 @@ Requires: gtk4
Requires: libadwaita
Requires: libsecret
Requires: webkitgtk6.0
Requires: perl-Image-ExifTool
Requires: xdg-utils
# DE-specific; do not Require a single desktop environment.
@@ -43,8 +44,9 @@ Recommends: kwallet
Provides: pdfs = %{version}-%{release}
%description
Unofficial Proton Drive client featuring a FUSE files-on-demand mount,
CLI, GTK4/Libadwaita GUI, system tray, and search launcher.
Files-on-demand Proton Drive for the Linux desktop: a FUSE virtual mount with
block-level caching, a scriptable CLI, and a GTK4/Libadwaita desktop app with
system tray and search launcher. Unofficial and not affiliated with Proton AG.
%prep
# In-tree build: no Source tarball. Pass --define "git_dir /path/to/checkout".
+1 -1
View File
@@ -1,6 +1,6 @@
[Unit]
Description=Proton Drive Files-On-Demand mount
Documentation=https://github.com/narl/proton-drive-linux
Documentation=https://github.com/narrrl/proton-drive-linux
# Tie the mount to the desktop session: start after it is up and stop with it.
After=graphical-session.target
PartOf=graphical-session.target