mirror of
https://github.com/narrrl/proton-drive-linux.git
synced 2026-11-03 16:41:26 +01:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
847ece17d7 | ||
|
|
50d2acdaba | ||
|
|
53caf63ad4 | ||
|
|
5176dd91e1 | ||
|
|
e88903d5ff | ||
|
|
b3e6af1ae5 | ||
|
|
b0f9513915 | ||
|
|
91b0864fce | ||
|
|
8bd9481902 | ||
|
|
b2d5870834 | ||
|
|
078774f6a1 | ||
|
|
326a9b4e92 | ||
|
|
78dc72a34d | ||
|
|
438b2b463e | ||
|
|
4cd49168c4 | ||
|
|
481c3fe0c7 | ||
|
|
098da2e97a | ||
|
|
d03d3eac3f | ||
|
|
433e206203 |
No files matched your search
@@ -0,0 +1,76 @@
|
||||
name: Bug report
|
||||
description: Something in the client misbehaves
|
||||
labels: ["bug"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Please check [`docs/BUGS.md`](../blob/main/docs/BUGS.md) first — the issue may already be
|
||||
tracked there with context.
|
||||
|
||||
**Do not report a security vulnerability here.** See [SECURITY.md](../blob/main/SECURITY.md).
|
||||
- type: textarea
|
||||
id: what
|
||||
attributes:
|
||||
label: What happened
|
||||
description: What you did, what you expected, and what happened instead.
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: repro
|
||||
attributes:
|
||||
label: Steps to reproduce
|
||||
placeholder: |
|
||||
1. …
|
||||
2. …
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: version
|
||||
attributes:
|
||||
label: Version
|
||||
description: Output of `pdfs --version`, and how you installed it (deb / rpm / AUR / source).
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: environment
|
||||
attributes:
|
||||
label: Distribution and desktop
|
||||
placeholder: "Fedora 44, GNOME 48 (Wayland)"
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: surface
|
||||
attributes:
|
||||
label: Where does it happen
|
||||
multiple: true
|
||||
options:
|
||||
- FUSE mount / file operations
|
||||
- Sync or backed-up folders (Computers)
|
||||
- CLI (pdfs)
|
||||
- Desktop app (pdfs-app)
|
||||
- Tray (pdfs-tray)
|
||||
- Search launcher (pdfs-prompt)
|
||||
- Login / keyring
|
||||
- Photos
|
||||
- Packaging / installation
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: diagnose
|
||||
attributes:
|
||||
label: "`pdfs diagnose` output"
|
||||
description: Redact paths or the account address if you prefer.
|
||||
render: console
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Daemon logs
|
||||
description: "`journalctl --user -u proton-drive.service -n 200 --no-pager`"
|
||||
render: console
|
||||
- type: checkboxes
|
||||
id: dataloss
|
||||
attributes:
|
||||
label: Data
|
||||
options:
|
||||
- label: This bug involved missing, truncated, or overwritten file content.
|
||||
@@ -0,0 +1,11 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: Security vulnerability
|
||||
url: https://github.com/narrrl/proton-drive-linux/security/advisories/new
|
||||
about: Report privately — never in a public issue.
|
||||
- name: Known issues
|
||||
url: https://github.com/narrrl/proton-drive-linux/blob/main/docs/BUGS.md
|
||||
about: The authoritative ledger of tracked bugs and their status.
|
||||
- name: Proton Drive service problems
|
||||
url: https://proton.me/support
|
||||
about: This is an unofficial client. Account or service issues go to Proton AG.
|
||||
@@ -0,0 +1,37 @@
|
||||
name: Feature request
|
||||
description: Suggest a capability or an improvement
|
||||
labels: ["enhancement"]
|
||||
body:
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: The problem
|
||||
description: What are you trying to do that the client makes hard or impossible today?
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: proposal
|
||||
attributes:
|
||||
label: What you'd like to see
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: surface
|
||||
attributes:
|
||||
label: Which part of the client
|
||||
multiple: true
|
||||
options:
|
||||
- FUSE mount
|
||||
- Sync / Computers
|
||||
- CLI (pdfs)
|
||||
- Desktop app (pdfs-app)
|
||||
- Tray (pdfs-tray)
|
||||
- Search launcher (pdfs-prompt)
|
||||
- Photos
|
||||
- Packaging
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Alternatives you considered
|
||||
@@ -0,0 +1,26 @@
|
||||
## What this changes
|
||||
|
||||
<!-- The user-visible effect, in a sentence or two. -->
|
||||
|
||||
**Crates touched:**
|
||||
**Related `docs/BUGS.md` entries:**
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
cargo fmt --all -- --check
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
cargo test --workspace --locked
|
||||
```
|
||||
|
||||
<!-- Plus whatever else you ran: acceptance suite, a live mount, manual steps. -->
|
||||
|
||||
## Implications
|
||||
|
||||
- [ ] Adds a SQLite migration (new `MIGRATION_V*`, `SCHEMA_VERSION` bumped — no shipped migration edited)
|
||||
- [ ] Touches recovery, staging, or anything that can be the only copy of user data
|
||||
- [ ] Changes packaging, the systemd unit, or desktop entries
|
||||
- [ ] Changes the control protocol (daemon and front ends updated together)
|
||||
- [ ] None of the above
|
||||
|
||||
<!-- Screenshots for GTK changes. -->
|
||||
@@ -122,13 +122,14 @@ jobs:
|
||||
Priority: optional
|
||||
Architecture: amd64
|
||||
Maintainer: Nils Pukropp <contact@narl.io>
|
||||
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4
|
||||
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4, libimage-exiftool-perl
|
||||
Installed-Size: ${INSTALLED_SIZE}
|
||||
Description: Proton Drive client for Linux
|
||||
Proton Drive client for Linux featuring:
|
||||
- FUSE files-on-demand mount
|
||||
- Command-line interface (CLI)
|
||||
- GTK4 system tray, desktop application, and search launcher
|
||||
Homepage: https://github.com/narrrl/proton-drive-linux
|
||||
Description: Unofficial Proton Drive client for Linux
|
||||
Files-on-demand Proton Drive for the Linux desktop, unaffiliated with Proton AG:
|
||||
- FUSE virtual mount with block-level caching
|
||||
- Scriptable command-line interface (pdfs)
|
||||
- GTK4 desktop application, system tray, and search launcher
|
||||
EOF
|
||||
|
||||
# Build Debian package
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Code of Conduct
|
||||
|
||||
## Our pledge
|
||||
|
||||
We want participation in this project to be a harassment-free experience for everyone, regardless
|
||||
of age, body size, disability, ethnicity, sex characteristics, gender identity and expression,
|
||||
level of experience, education, socioeconomic status, nationality, personal appearance, race,
|
||||
religion, or sexual identity and orientation.
|
||||
|
||||
## Standards
|
||||
|
||||
Examples of behavior that helps:
|
||||
|
||||
- being respectful of differing viewpoints and experience levels,
|
||||
- giving and accepting constructive feedback gracefully,
|
||||
- focusing on what is best for the project and its users.
|
||||
|
||||
Examples of unacceptable behavior:
|
||||
|
||||
- sexualized language or imagery, and unwelcome sexual attention,
|
||||
- trolling, insulting or derogatory comments, and personal or political attacks,
|
||||
- public or private harassment,
|
||||
- publishing others' private information without explicit permission.
|
||||
|
||||
## Scope and enforcement
|
||||
|
||||
This applies in all project spaces — issues, pull requests, discussions — and when representing the
|
||||
project in public. Report unacceptable behavior to <contact@narl.io>. Reports are handled
|
||||
confidentially. Maintainers may edit, remove, or reject contributions that violate this document,
|
||||
and may temporarily or permanently ban any contributor for behavior they deem inappropriate.
|
||||
|
||||
Adapted from the [Contributor Covenant](https://www.contributor-covenant.org), version 2.1.
|
||||
@@ -0,0 +1,70 @@
|
||||
# Contributing
|
||||
|
||||
Thanks for taking the time. This is an unofficial Proton Drive client that people trust with real
|
||||
files, so the bar for changes that touch the filesystem, the sync engine, or the database is
|
||||
deliberately high — everything else is ordinary Rust review.
|
||||
|
||||
## Before you start
|
||||
|
||||
- Read [`AGENTS.md`](AGENTS.md) for repository conventions and
|
||||
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for how the daemon, front ends, and SDK fit
|
||||
together.
|
||||
- Check [`docs/BUGS.md`](docs/BUGS.md) — it is the authoritative issue ledger, and your problem may
|
||||
already be tracked there with context.
|
||||
- For anything larger than a fix, open an issue first so the design can be discussed before you
|
||||
spend the time.
|
||||
|
||||
## Development setup
|
||||
|
||||
```bash
|
||||
git clone https://github.com/narrrl/proton-drive-linux.git
|
||||
cd proton-drive-linux
|
||||
cargo build --workspace
|
||||
```
|
||||
|
||||
GUI and FUSE targets need the GTK4, libadwaita, WebKitGTK 6, libsecret, D-Bus, and FUSE3
|
||||
development packages listed in the [README](README.md#prerequisites). Minimum supported Rust
|
||||
version is 1.96.
|
||||
|
||||
## Quality gates
|
||||
|
||||
CI runs exactly these, and they must pass with zero warnings:
|
||||
|
||||
```bash
|
||||
cargo fmt --all -- --check
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
cargo test --workspace --locked
|
||||
scripts/fuse-acceptance.sh --offline-only
|
||||
```
|
||||
|
||||
Changes to the FUSE layer, sync, or the pending-operation queue should also be exercised against a
|
||||
real mount before review — see [`docs/TESTING.md`](docs/TESTING.md). Live runs are destructive:
|
||||
use a dedicated test account, never one holding data you cannot lose.
|
||||
|
||||
## Making changes
|
||||
|
||||
- **Keep shared logic in `pdfs-core`.** The CLI and GUI never touch the database or the network;
|
||||
they speak the control protocol to the daemon.
|
||||
- **The SQLite schema is forward-only.** Never edit a shipped migration — add a new one and bump
|
||||
`SCHEMA_VERSION` in `crates/pdfs-core/src/db/migrations.rs`.
|
||||
- **Typed errors and `?` over panics** on runtime paths. Blocking filesystem, keyring, or
|
||||
control-socket work stays off GTK's main thread.
|
||||
- **`staging/` and `recovery/` may hold the only copy of user data.** Nothing may clear or purge
|
||||
them opportunistically.
|
||||
|
||||
## Commits and pull requests
|
||||
|
||||
Commits follow scoped [Conventional Commits](https://www.conventionalcommits.org), e.g.
|
||||
`fix(fuse,core): keep truncate composable with queued revisions`.
|
||||
|
||||
A pull request should state:
|
||||
|
||||
- the user-visible effect and which crates it touches,
|
||||
- the `docs/BUGS.md` entries it fixes or affects,
|
||||
- the commands you ran to verify it,
|
||||
- screenshots for GTK changes,
|
||||
- and, explicitly, any migration, recovery, data-loss, or packaging implications.
|
||||
|
||||
## Security
|
||||
|
||||
Do not open a public issue for a vulnerability — see [SECURITY.md](SECURITY.md).
|
||||
Generated
+67
-8
@@ -2200,6 +2200,25 @@ dependencies = [
|
||||
"system-deps",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "h2"
|
||||
version = "0.4.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155"
|
||||
dependencies = [
|
||||
"atomic-waker",
|
||||
"bytes",
|
||||
"fnv",
|
||||
"futures-core",
|
||||
"futures-sink",
|
||||
"http",
|
||||
"indexmap",
|
||||
"slab",
|
||||
"tokio",
|
||||
"tokio-util",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "half"
|
||||
version = "2.7.1"
|
||||
@@ -2367,6 +2386,7 @@ dependencies = [
|
||||
"bytes",
|
||||
"futures-channel",
|
||||
"futures-core",
|
||||
"h2",
|
||||
"http",
|
||||
"http-body",
|
||||
"httparse",
|
||||
@@ -3518,7 +3538,7 @@ checksum = "35fb2e5f958ec131621fdd531e9fc186ed768cbe395337403ae56c17a74c68ec"
|
||||
|
||||
[[package]]
|
||||
name = "pdfs-cli"
|
||||
version = "1.7.0"
|
||||
version = "1.9.1"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"clap 4.6.4",
|
||||
@@ -3533,7 +3553,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "pdfs-core"
|
||||
version = "1.7.0"
|
||||
version = "1.9.1"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"directories",
|
||||
@@ -3546,16 +3566,19 @@ dependencies = [
|
||||
"rusqlite",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha1 0.10.7",
|
||||
"sha2 0.10.9",
|
||||
"thiserror 2.0.19",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"zip",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pdfs-fuse"
|
||||
version = "1.7.0"
|
||||
version = "1.9.1"
|
||||
dependencies = [
|
||||
"base64",
|
||||
"fuser",
|
||||
"futures",
|
||||
"image",
|
||||
@@ -3565,13 +3588,15 @@ dependencies = [
|
||||
"pdfs-core",
|
||||
"proton-drive-rs",
|
||||
"serde_json",
|
||||
"sha1 0.10.7",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"zip",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pdfs-gui"
|
||||
version = "1.7.0"
|
||||
version = "1.9.1"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-channel",
|
||||
@@ -3832,9 +3857,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "proton-drive-rs"
|
||||
version = "0.5.1"
|
||||
version = "0.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3abae5e90e6889b120208159eb2c60b332612e5b708fdf3c14287307ce01715d"
|
||||
checksum = "58322795538d66137d2eb071590375c5287e69059c429e2309dc37368ab5b588"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64",
|
||||
@@ -3859,9 +3884,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "proton-sdk"
|
||||
version = "0.5.1"
|
||||
version = "0.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3fa729310c1a66e12bb41b9f5f036c86b9857e76536556052203ffecdb0b44c3"
|
||||
checksum = "d77482a3160d14926c65537c64bc95cf27d609597a524c97d97d97d1777e9603"
|
||||
dependencies = [
|
||||
"aes-gcm 0.11.0",
|
||||
"async-trait",
|
||||
@@ -4252,6 +4277,7 @@ dependencies = [
|
||||
"bytes",
|
||||
"futures-core",
|
||||
"futures-util",
|
||||
"h2",
|
||||
"http",
|
||||
"http-body",
|
||||
"http-body-util",
|
||||
@@ -5110,6 +5136,7 @@ dependencies = [
|
||||
"bytes",
|
||||
"futures-core",
|
||||
"futures-sink",
|
||||
"libc",
|
||||
"pin-project-lite",
|
||||
"tokio",
|
||||
]
|
||||
@@ -5291,6 +5318,12 @@ dependencies = [
|
||||
"cipher 0.4.4",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "typed-path"
|
||||
version = "0.12.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8e28f89b80c87b8fb0cf04ab448d5dd0dd0ade2f8891bae878de66a75a28600e"
|
||||
|
||||
[[package]]
|
||||
name = "typenum"
|
||||
version = "1.20.1"
|
||||
@@ -6232,6 +6265,20 @@ dependencies = [
|
||||
"syn 2.0.119",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zip"
|
||||
version = "8.6.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2d04a6b5381502aa6087c94c669499eb1602eb9c5e8198e534de571f7154809b"
|
||||
dependencies = [
|
||||
"crc32fast",
|
||||
"flate2",
|
||||
"indexmap",
|
||||
"memchr",
|
||||
"typed-path",
|
||||
"zopfli",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zlib-rs"
|
||||
version = "0.6.6"
|
||||
@@ -6244,6 +6291,18 @@ version = "1.0.23"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
|
||||
|
||||
[[package]]
|
||||
name = "zopfli"
|
||||
version = "0.8.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f05cd8797d63865425ff89b5c4a48804f35ba0ce8d125800027ad6017d2b5249"
|
||||
dependencies = [
|
||||
"bumpalo",
|
||||
"crc32fast",
|
||||
"log",
|
||||
"simd-adler32",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zune-core"
|
||||
version = "0.5.1"
|
||||
|
||||
+15
-3
@@ -3,15 +3,20 @@ resolver = "2"
|
||||
members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"]
|
||||
|
||||
[workspace.package]
|
||||
version = "1.7.0"
|
||||
version = "1.9.1"
|
||||
edition = "2024"
|
||||
license = "MIT"
|
||||
authors = ["Nils Pukropp <contact@narl.io>"]
|
||||
rust-version = "1.96"
|
||||
repository = "https://github.com/narrrl/proton-drive-linux"
|
||||
homepage = "https://github.com/narrrl/proton-drive-linux"
|
||||
readme = "README.md"
|
||||
keywords = ["proton", "drive", "fuse", "sync", "linux"]
|
||||
categories = ["filesystem", "command-line-utilities"]
|
||||
|
||||
[workspace.dependencies]
|
||||
proton-sdk = { version = "0.5.1" }
|
||||
proton-drive-rs = { version = "0.5.1" }
|
||||
proton-sdk = { version = "0.6.1" }
|
||||
proton-drive-rs = { version = "0.6.1" }
|
||||
|
||||
# Internal crates.
|
||||
pdfs-core = { path = "crates/pdfs-core" }
|
||||
@@ -20,6 +25,7 @@ pdfs-fuse = { path = "crates/pdfs-fuse" }
|
||||
tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "fs", "io-util", "time", "signal"] }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
base64 = "0.22"
|
||||
thiserror = "2"
|
||||
anyhow = "1"
|
||||
# Non-poisoning mutexes. A handler that panics while holding a lock must not
|
||||
@@ -32,6 +38,12 @@ tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||
directories = "5"
|
||||
ignore = "0.4"
|
||||
sha2 = "0.10"
|
||||
# Google Takeout exports are zip sets; the importer reads their central
|
||||
# directories and streams entries out without extracting to disk first.
|
||||
zip = { version = "8", default-features = false, features = ["deflate"] }
|
||||
# Proton's photo duplicate detection is keyed on the plaintext SHA-1, so an
|
||||
# import has to speak SHA-1 whatever its age.
|
||||
sha1 = "0.10"
|
||||
# Thumbnail generation for photos the server has none for. Decoders only for the
|
||||
# formats a camera or phone actually produces — the rest of the crate's codecs
|
||||
# would be dead weight in the daemon.
|
||||
|
||||
@@ -1,12 +1,70 @@
|
||||
# Proton Drive client for Linux (unofficial)
|
||||
<div align="center">
|
||||
|
||||
A fast, unofficial Proton Drive client for Linux. This client features an advanced files-on-demand FUSE virtual mount with block-level caching, a command-line interface (CLI), and a fully non-blocking GTK4 desktop application with system tray integration.
|
||||
<img src="images/logo.svg" alt="" width="112" height="112">
|
||||
|
||||
# Proton Drive for Linux
|
||||
|
||||
**Files-on-demand Proton Drive for the Linux desktop** — a FUSE virtual mount with block-level
|
||||
caching, a scriptable CLI, and a native GTK4 app with tray and search launcher.
|
||||
|
||||
[](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml)
|
||||
[](https://github.com/narrrl/proton-drive-linux/releases/latest)
|
||||
[](LICENSE)
|
||||
[](https://www.rust-lang.org)
|
||||
[](#prerequisites)
|
||||
|
||||
<img src="images/files.png" alt="The GTK4 file browser" width="820">
|
||||
|
||||
</div>
|
||||
|
||||
> [!IMPORTANT]
|
||||
> This is an **unofficial**, community-built client. It is not affiliated with, endorsed by, or
|
||||
> supported by Proton AG. "Proton" and "Proton Drive" are trademarks of Proton AG, used here only
|
||||
> to describe what the software talks to. Use a test account first, and read
|
||||
> [`docs/RECOVERY.md`](docs/RECOVERY.md) before trusting it with data you cannot lose.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# Debian / Ubuntu
|
||||
sudo apt install ./proton-drive-linux_*.deb
|
||||
|
||||
# Fedora
|
||||
sudo dnf install ./proton-drive-linux-*.rpm
|
||||
|
||||
# Arch Linux
|
||||
cd packaging && makepkg -fi
|
||||
```
|
||||
|
||||
Packages for each release are attached to the [latest
|
||||
release](https://github.com/narrrl/proton-drive-linux/releases/latest); see
|
||||
[Installation & Packages](#installation--packages) for details and
|
||||
[Building from Source](#building-from-source) to compile it yourself.
|
||||
|
||||
```bash
|
||||
pdfs login # sign in (SRP + 2FA, credentials go to the system keyring)
|
||||
pdfs status # mount state, queued writes, cache usage
|
||||
pdfs-app # the desktop app
|
||||
```
|
||||
|
||||
## Contents
|
||||
|
||||
- [Features](#features) · [Quick search](#quick-search-and-media-opening) · [Opening files](#choosing-how-files-open)
|
||||
- [Selective sync](#selective-sync-pdfsignore) · [Diagnostics](#diagnostics--maintenance) · [Scripting](#scripting---json)
|
||||
- [Performance](#performance--caching) · [Filesystem safety](#filesystem-safety) · [Screenshots](#screenshots)
|
||||
- [Prerequisites](#prerequisites) · [Building](#building-from-source) · [Packages](#installation--packages) · [Releases](#automated-releases-cicd)
|
||||
|
||||
**Documentation:** [Architecture](docs/ARCHITECTURE.md) · [Testing](docs/TESTING.md) ·
|
||||
[Recovery](docs/RECOVERY.md) · [Development](docs/DEVELOPMENT.md) ·
|
||||
[Known issues](docs/BUGS.md) · [Changelog](docs/CHANGELOG.md) ·
|
||||
[Contributing](CONTRIBUTING.md) · [Security policy](SECURITY.md)
|
||||
|
||||
## Features
|
||||
|
||||
- **Files-on-Demand (FUSE)**: Mount your Proton Drive as a virtual filesystem where files are downloaded only when opened, utilizing smart block-level caching and disk-backed writes.
|
||||
- **Command-Line Interface (CLI)**: Manage your drive, authenticate, and monitor sync status directly from the terminal.
|
||||
- **Non-Blocking GTK4 Desktop App**: Browse files, manage pins, configure options, and monitor active transfers through a modern, native GUI with a fully non-blocking asynchronous main loop.
|
||||
- **File Thumbnails**: Browse locally cached image previews in Files, Shared, and Trash, including embedded previews from camera RAW files. Build a whole folder tree from the Files toolbar and resize the grid from the status bar.
|
||||
- **Shared Links & Invites**: Browse files shared with you by other users, and view/manage your own public shared links directly in the GUI.
|
||||
- **Local Backup (Computers)**: Sync and back up local directories (like Downloads, Documents, Pictures, etc.) directly to your Proton Drive account.
|
||||
- **Locations**: One page (and `pdfs locations`) listing every place Proton Drive occupies on this machine — the main mount plus each backed-up folder — with its mode, sync state, and whether it is read-only. Switching a folder between a full local copy and on-demand happens here.
|
||||
@@ -270,19 +328,21 @@ sudo apt-get install -y \
|
||||
libadwaita-1-dev \
|
||||
libwebkitgtk-6.0-dev \
|
||||
libsecret-1-dev \
|
||||
libdbus-1-dev
|
||||
libdbus-1-dev \
|
||||
libimage-exiftool-perl
|
||||
```
|
||||
|
||||
### Arch Linux
|
||||
```bash
|
||||
sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus webkitgtk-6.0
|
||||
sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus webkitgtk-6.0 perl-image-exiftool
|
||||
```
|
||||
|
||||
### Fedora (44+)
|
||||
```bash
|
||||
sudo dnf install -y \
|
||||
pkgconf-pkg-config fuse3-devel gtk4-devel libadwaita-devel \
|
||||
webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel cargo rust
|
||||
webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel \
|
||||
perl-Image-ExifTool cargo rust
|
||||
```
|
||||
|
||||
Runtime extras (pick your desktop):
|
||||
@@ -302,7 +362,7 @@ Ensure you have Rust and Cargo installed (minimum supported Rust version is 1.96
|
||||
|
||||
1. Clone the repository and navigate into the project directory:
|
||||
```bash
|
||||
git clone https://github.com/narl/proton-drive-linux.git
|
||||
git clone https://github.com/narrrl/proton-drive-linux.git
|
||||
cd proton-drive-linux
|
||||
```
|
||||
2. Build the workspace in release mode:
|
||||
@@ -372,6 +432,15 @@ This project has a GitHub Actions CI workflow configured under `.github/workflow
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
Bug reports, packaging fixes, and patches are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for
|
||||
the workflow and the quality gates CI enforces, and [SECURITY.md](SECURITY.md) for reporting a
|
||||
vulnerability privately.
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the [MIT License](LICENSE).
|
||||
Released under the [MIT License](LICENSE).
|
||||
|
||||
Proton Drive is a service of Proton AG. This project is an independent client and carries no
|
||||
affiliation with or endorsement from Proton AG.
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
# Security Policy
|
||||
|
||||
## Supported versions
|
||||
|
||||
Fixes land on the latest release. Please reproduce on the most recent version before reporting.
|
||||
|
||||
## Reporting a vulnerability
|
||||
|
||||
**Do not open a public issue.** Report privately through GitHub's
|
||||
[private vulnerability reporting](https://github.com/narrrl/proton-drive-linux/security/advisories/new),
|
||||
or by email to <contact@narl.io>.
|
||||
|
||||
Please include the version (`pdfs --version`), your distribution and desktop environment, what an
|
||||
attacker gains, and a reproduction if you have one. You can expect an acknowledgement within a few
|
||||
days; a fix ships in the next release, and you are credited in the advisory unless you ask
|
||||
otherwise.
|
||||
|
||||
## Scope
|
||||
|
||||
This client holds Proton account credentials and decrypted file content on the local machine, so
|
||||
the interesting boundaries are:
|
||||
|
||||
- **Credentials.** Sessions and the mailbox password live in the system Secret Service. Anything
|
||||
that writes them elsewhere — logs, temp files, crash dumps — is a vulnerability.
|
||||
- **The control socket.** Anything that can connect to `control.sock` has the daemon's
|
||||
authenticated session. It is `0600`, and the state, cache, and config directories must be real,
|
||||
current-user-owned `0700` directories; the daemon fails closed otherwise. Reports that defeat
|
||||
those checks are in scope.
|
||||
- **Local cache and staging.** Decrypted block content and staged writes on disk.
|
||||
- **Cryptography.** Key derivation, signature handling, and anything that could publish plaintext
|
||||
or the wrong key material to Proton.
|
||||
|
||||
Out of scope: vulnerabilities in Proton's own service (report those to Proton AG), and issues that
|
||||
require an attacker who already has your user account on the machine.
|
||||
@@ -5,6 +5,8 @@ edition.workspace = true
|
||||
license.workspace = true
|
||||
authors.workspace = true
|
||||
rust-version.workspace = true
|
||||
repository.workspace = true
|
||||
homepage.workspace = true
|
||||
description = "CLI for the Proton Drive Linux client"
|
||||
|
||||
[[bin]]
|
||||
|
||||
@@ -158,6 +158,28 @@ enum Command {
|
||||
},
|
||||
/// List the photo albums (including ones shared with this account).
|
||||
Albums,
|
||||
/// Import a Google Photos Takeout export into Proton Photos.
|
||||
///
|
||||
/// Pass every zip of the export at once — Google splits one export across
|
||||
/// numbered parts, and a photo's metadata often sits in a different part
|
||||
/// than the photo. Albums in the export are recreated as Proton albums, and
|
||||
/// photos already on the account (same name and same content) are skipped,
|
||||
/// so an interrupted import can simply be run again.
|
||||
ImportGooglePhotos {
|
||||
/// The export's `.zip` files (e.g. `takeout-*.zip`).
|
||||
#[arg(required = true)]
|
||||
archives: Vec<PathBuf>,
|
||||
/// Report what would be imported without uploading anything.
|
||||
#[arg(long)]
|
||||
dry_run: bool,
|
||||
/// Stay attached and print progress until the import finishes.
|
||||
#[arg(long)]
|
||||
wait: bool,
|
||||
},
|
||||
/// Show how the running (or last) Google Photos import is doing.
|
||||
ImportStatus,
|
||||
/// Stop the running Google Photos import.
|
||||
CancelImport,
|
||||
/// List the photos in one album (newest capture first).
|
||||
Album {
|
||||
/// Album node uid in `volume~link` form (from `pdfs albums`).
|
||||
@@ -589,6 +611,13 @@ fn main() -> Result<()> {
|
||||
} => cmd_photos(limit, offset, favorites),
|
||||
Command::Favorite { uid, remove } => cmd_favorite(uid, remove),
|
||||
Command::Albums => cmd_albums(),
|
||||
Command::ImportGooglePhotos {
|
||||
archives,
|
||||
dry_run,
|
||||
wait,
|
||||
} => cmd_import_google_photos(archives, dry_run, wait),
|
||||
Command::ImportStatus => cmd_import_status(),
|
||||
Command::CancelImport => cmd_cancel_import(),
|
||||
Command::Album { uid, limit, offset } => cmd_album(uid, limit, offset),
|
||||
Command::OpenPhoto { uid } => cmd_open_photo(uid),
|
||||
Command::Search { query, limit } => cmd_search(query, limit),
|
||||
@@ -1711,6 +1740,104 @@ fn cmd_albums() -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn cmd_import_google_photos(archives: Vec<PathBuf>, dry_run: bool, wait: bool) -> Result<()> {
|
||||
// The daemon opens the archives itself off the shared filesystem, so it gets
|
||||
// absolute paths — the same arrangement `pdfs upload` uses.
|
||||
let mut absolute = Vec::with_capacity(archives.len());
|
||||
for archive in &archives {
|
||||
let path = std::fs::canonicalize(archive)
|
||||
.with_context(|| format!("resolve {}", archive.display()))?;
|
||||
absolute.push(
|
||||
path.to_str()
|
||||
.ok_or_else(|| anyhow!("path is not valid UTF-8: {}", path.display()))?
|
||||
.to_owned(),
|
||||
);
|
||||
}
|
||||
|
||||
match control_request(CtlRequest::ImportTakeout {
|
||||
archives: absolute,
|
||||
dry_run,
|
||||
})? {
|
||||
CtlResponse::Ok { message } => println!("{message}"),
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
if !wait {
|
||||
println!("Run `pdfs import-status` to follow it.");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Poll rather than stream: the control protocol is request/response, and an
|
||||
// import runs for hours — a held-open socket would be the only long-lived
|
||||
// connection the daemon has.
|
||||
loop {
|
||||
std::thread::sleep(std::time::Duration::from_secs(2));
|
||||
match control_request(CtlRequest::ImportStatus)? {
|
||||
CtlResponse::ImportStatus {
|
||||
running: true,
|
||||
summary: _,
|
||||
} => continue,
|
||||
CtlResponse::ImportStatus { summary, .. } => {
|
||||
print_import_summary(summary.as_ref());
|
||||
return Ok(());
|
||||
}
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn cmd_import_status() -> Result<()> {
|
||||
match control_request(CtlRequest::ImportStatus)? {
|
||||
CtlResponse::ImportStatus { running, summary } => {
|
||||
if running {
|
||||
println!("Import running. `pdfs transfers` shows its progress.");
|
||||
}
|
||||
print_import_summary(summary.as_ref());
|
||||
}
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn cmd_cancel_import() -> Result<()> {
|
||||
match control_request(CtlRequest::CancelImport)? {
|
||||
CtlResponse::Ok { message } => println!("{message}"),
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn print_import_summary(summary: Option<&pdfs_core::control::ImportSummary>) {
|
||||
let Some(summary) = summary else {
|
||||
println!("No import has run yet.");
|
||||
return;
|
||||
};
|
||||
println!("Photos in export : {}", summary.found);
|
||||
println!("Uploaded : {}", summary.uploaded);
|
||||
println!("Already on Proton: {}", summary.duplicates);
|
||||
if summary.skipped_trashed > 0 {
|
||||
println!("Skipped (trash) : {}", summary.skipped_trashed);
|
||||
}
|
||||
if summary.albums_created > 0 || summary.album_links > 0 {
|
||||
println!(
|
||||
"Albums : {} created, {} photos filed",
|
||||
summary.albums_created, summary.album_links
|
||||
);
|
||||
}
|
||||
if summary.failed > 0 {
|
||||
println!("Failed : {}", summary.failed);
|
||||
}
|
||||
if summary.bytes > 0 {
|
||||
println!("Uploaded bytes : {}", summary.bytes);
|
||||
}
|
||||
if summary.cancelled {
|
||||
println!("Import was cancelled before it finished.");
|
||||
}
|
||||
}
|
||||
|
||||
fn cmd_album(uid: String, limit: usize, offset: usize) -> Result<()> {
|
||||
match control_request(CtlRequest::AlbumPhotos { uid, offset, limit })? {
|
||||
CtlResponse::Photos { items, .. } if items.is_empty() => println!("No photos."),
|
||||
|
||||
@@ -5,6 +5,8 @@ edition.workspace = true
|
||||
license.workspace = true
|
||||
authors.workspace = true
|
||||
rust-version.workspace = true
|
||||
repository.workspace = true
|
||||
homepage.workspace = true
|
||||
description = "Core auth/session/config for the Proton Drive Linux client"
|
||||
|
||||
[dependencies]
|
||||
@@ -18,6 +20,8 @@ tracing.workspace = true
|
||||
directories.workspace = true
|
||||
ignore.workspace = true
|
||||
sha2.workspace = true
|
||||
sha1.workspace = true
|
||||
zip.workspace = true
|
||||
rusqlite.workspace = true
|
||||
keyring.workspace = true
|
||||
parking_lot.workspace = true
|
||||
|
||||
+918
-60
File diff suppressed because it is too large.
Load diff
@@ -93,6 +93,32 @@ pub enum Request {
|
||||
/// already cached (one batched round-trip) and replying with their on-disk
|
||||
/// paths. Keep the batch small — it is served on demand, as tiles scroll in.
|
||||
PhotoThumbs { uids: Vec<String> },
|
||||
/// Fetch thumbnails for ordinary Drive image files shown outside the Photos
|
||||
/// timeline. The modification time is the cache validity tag, so replacing
|
||||
/// an image can never reuse its previous revision's thumbnail.
|
||||
FileThumbs {
|
||||
items: Vec<FileThumbRequest>,
|
||||
/// Identifies the front-end's current listing. Moving to another page or
|
||||
/// folder advances it, allowing the daemon to stop work for rows that
|
||||
/// are no longer visible.
|
||||
#[serde(default)]
|
||||
generation: u64,
|
||||
},
|
||||
/// Reserve a daemon-issued generation for ordinary-file thumbnail work.
|
||||
/// The daemon is the ordering authority, so GUI restarts and wall-clock
|
||||
/// corrections cannot make a new listing look older than an earlier one.
|
||||
ReserveFileThumbGeneration,
|
||||
/// Stop ordinary-file thumbnail work belonging to older listings. Photos
|
||||
/// timeline work and an explicit recursive build are deliberately separate.
|
||||
CancelFileThumbs { generation: u64 },
|
||||
/// Start building local thumbnails for every supported image below `path`.
|
||||
/// The request returns immediately; progress is read with
|
||||
/// [`Request::ThumbnailBuildStatus`].
|
||||
StartThumbnailBuild { path: String },
|
||||
/// Stop the current recursive ordinary-file thumbnail build, if any.
|
||||
CancelThumbnailBuild,
|
||||
/// Read progress for the recursive ordinary-file thumbnail build.
|
||||
ThumbnailBuildStatus,
|
||||
/// Download a photo's full content into the cache; replies with its path.
|
||||
OpenPhoto { uid: String },
|
||||
/// Add or remove Proton's `Favorite` tag on a photo. Replies with
|
||||
@@ -103,6 +129,33 @@ pub enum Request {
|
||||
/// re-encrypted for our timeline, which the SDK does not implement yet; the
|
||||
/// daemon reports that as an error rather than silently doing nothing.
|
||||
SetPhotoFavorite { uid: String, favorite: bool },
|
||||
/// Import Google Photos **Takeout** archives into Proton Photos.
|
||||
///
|
||||
/// `archives` are absolute paths to the export's `.zip` parts on the
|
||||
/// daemon's own filesystem — the whole set at once, because a photo and its
|
||||
/// metadata sidecar routinely land in different parts. Photos already on the
|
||||
/// account (matched by name *and* content digest) are skipped, so a library
|
||||
/// partly synced by Proton Photos elsewhere does not import twice, and an
|
||||
/// interrupted import resumes by simply being run again.
|
||||
///
|
||||
/// With `dry_run` the daemon scans and reports what it *would* do without
|
||||
/// uploading anything. Otherwise it acks immediately with
|
||||
/// [`Response::Ok`] and works in the background — an export is hours of
|
||||
/// upload, far past any socket timeout. Progress shows up in
|
||||
/// [`Request::GetQueueStatus`] as a job, and the final counts in
|
||||
/// [`Request::ImportStatus`].
|
||||
ImportTakeout {
|
||||
archives: Vec<String>,
|
||||
#[serde(default)]
|
||||
dry_run: bool,
|
||||
},
|
||||
/// How the running (or last finished) Takeout import is doing. Replies with
|
||||
/// [`Response::ImportStatus`].
|
||||
ImportStatus,
|
||||
/// Ask the running Takeout import to stop. It finishes the photo on the wire
|
||||
/// and files what it has uploaded into its albums, then reports
|
||||
/// `cancelled`. Replies with [`Response::Ok`].
|
||||
CancelImport,
|
||||
/// Upload the photo at `source_path` under the given name and media type.
|
||||
///
|
||||
/// A path, not the bytes: this protocol is line-delimited JSON, and
|
||||
@@ -652,6 +705,10 @@ pub struct SharedItem {
|
||||
/// The shared node's decrypted name.
|
||||
pub name: String,
|
||||
pub is_dir: bool,
|
||||
/// Modification time used to validate an image thumbnail. Older daemons
|
||||
/// omit it, in which case clients use `0` as a conservative fallback tag.
|
||||
#[serde(default)]
|
||||
pub modified: i64,
|
||||
/// Mountpoint-relative path, when the daemon can resolve it (the node lives in
|
||||
/// my own tree). Empty when the path is unknown.
|
||||
#[serde(default)]
|
||||
@@ -1052,6 +1109,35 @@ pub struct PhotoMonth {
|
||||
pub count: usize,
|
||||
}
|
||||
|
||||
/// How a Google Photos Takeout import is going, or how the last one ended
|
||||
/// ([`Response::ImportStatus`]).
|
||||
///
|
||||
/// The counts are cumulative for the run, so the same shape serves progress and
|
||||
/// the final report. `found` counts *distinct* photos: Google stores a separate
|
||||
/// copy of a photo in each album folder it appears in, and those fold into one
|
||||
/// upload that joins several albums.
|
||||
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
|
||||
pub struct ImportSummary {
|
||||
/// Distinct photos the export holds.
|
||||
pub found: usize,
|
||||
/// Photos uploaded by this run.
|
||||
pub uploaded: usize,
|
||||
/// Photos already on the account, matched by name *and* content digest.
|
||||
pub duplicates: usize,
|
||||
/// Photos that could not be read or uploaded.
|
||||
pub failed: usize,
|
||||
/// Photos in the export's trash bucket, which are never imported.
|
||||
pub skipped_trashed: usize,
|
||||
/// Albums created by this run.
|
||||
pub albums_created: usize,
|
||||
/// Album memberships added by this run.
|
||||
pub album_links: usize,
|
||||
/// Bytes uploaded.
|
||||
pub bytes: u64,
|
||||
/// True when the run stopped early because it was cancelled.
|
||||
pub cancelled: bool,
|
||||
}
|
||||
|
||||
/// One album in a [`Response::Albums`] listing.
|
||||
///
|
||||
/// An album is a folder node on the photos volume carrying album properties, so
|
||||
@@ -1131,6 +1217,76 @@ pub struct PhotoThumb {
|
||||
pub pending: bool,
|
||||
}
|
||||
|
||||
/// One ordinary Drive file whose thumbnail a front-end wants to paint.
|
||||
#[derive(Serialize, Deserialize, Debug, Clone)]
|
||||
pub struct FileThumbRequest {
|
||||
/// Node uid in `volume~link` form.
|
||||
pub uid: String,
|
||||
/// File modification time, used as the thumbnail cache validity tag.
|
||||
pub modified: i64,
|
||||
/// Original Drive name. Local RAW-preview extraction uses the extension as a
|
||||
/// format hint; defaulted for wire compatibility with older front-ends.
|
||||
#[serde(default)]
|
||||
pub name: String,
|
||||
}
|
||||
|
||||
/// Whether a Drive name is a raster image that the built-in decoder can read,
|
||||
/// or a concrete camera RAW format whose embedded preview exiftool can extract.
|
||||
/// Keep this list shared by the daemon and every GUI surface so a visible tile
|
||||
/// is never advertised without a matching generation path.
|
||||
pub fn is_thumbnail_image_name(name: &str) -> bool {
|
||||
let Some((_, extension)) = name.rsplit_once('.') else {
|
||||
return false;
|
||||
};
|
||||
let extension = extension.to_ascii_lowercase();
|
||||
STANDARD_THUMBNAIL_EXTENSIONS.contains(&extension.as_str())
|
||||
|| RAW_THUMBNAIL_EXTENSIONS.contains(&extension.as_str())
|
||||
}
|
||||
|
||||
/// Whether `name` is one of the concrete RAW formats handled by the exiftool
|
||||
/// embedded-preview path. Deliberately excludes generic/legacy grab-bags whose
|
||||
/// preview availability is not dependable.
|
||||
pub fn is_raw_image_name(name: &str) -> bool {
|
||||
let Some((_, extension)) = name.rsplit_once('.') else {
|
||||
return false;
|
||||
};
|
||||
let extension = extension.to_ascii_lowercase();
|
||||
RAW_THUMBNAIL_EXTENSIONS.contains(&extension.as_str())
|
||||
}
|
||||
|
||||
const STANDARD_THUMBNAIL_EXTENSIONS: &[&str] =
|
||||
&["bmp", "gif", "jpeg", "jpg", "png", "tif", "tiff", "webp"];
|
||||
|
||||
/// Concrete camera RAW formats with embedded previews handled by exiftool.
|
||||
/// This is the single source used by both RAW classification and the broader
|
||||
/// thumbnail capability check.
|
||||
const RAW_THUMBNAIL_EXTENSIONS: &[&str] = &[
|
||||
"arw", "cr2", "cr3", "crw", "dng", "k25", "kdc", "mrw", "nef", "nrw", "orf", "pef", "raf",
|
||||
"raw", "rw2", "sr2", "srf", "x3f",
|
||||
];
|
||||
|
||||
/// Progress of the one recursive ordinary-file thumbnail build the daemon may
|
||||
/// run at a time.
|
||||
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
|
||||
pub struct ThumbnailBuildStatus {
|
||||
pub running: bool,
|
||||
/// True while folders are still being discovered, before the final image
|
||||
/// count is known and the progress bar can become determinate.
|
||||
pub scanning: bool,
|
||||
/// Mountpoint-relative root selected when the build started.
|
||||
pub path: String,
|
||||
pub folders_scanned: u64,
|
||||
pub images_found: u64,
|
||||
/// Images already cached or processed during this build. Includes failures,
|
||||
/// so `completed == images_found` still means the job has finished.
|
||||
pub completed: u64,
|
||||
/// Images that could not be downloaded or decoded.
|
||||
pub failed: u64,
|
||||
/// A traversal-level problem, such as a folder that could not be listed.
|
||||
#[serde(default)]
|
||||
pub message: Option<String>,
|
||||
}
|
||||
|
||||
/// A daemon too old to report connectivity is assumed online — it could not
|
||||
/// have mounted at all otherwise.
|
||||
fn default_online() -> bool {
|
||||
@@ -1270,11 +1426,28 @@ pub enum Response {
|
||||
available: bool,
|
||||
items: Vec<AlbumInfo>,
|
||||
},
|
||||
/// How the Takeout import is doing (reply to [`Request::ImportStatus`]).
|
||||
/// `running` is false once it has finished, when `summary` is the final
|
||||
/// report; `summary` is `None` when no import has run this session.
|
||||
ImportStatus {
|
||||
running: bool,
|
||||
#[serde(default)]
|
||||
summary: Option<ImportSummary>,
|
||||
},
|
||||
/// The months the timeline spans (reply to [`Request::PhotoMonths`]),
|
||||
/// newest first.
|
||||
PhotoMonths { months: Vec<PhotoMonth> },
|
||||
/// Thumbnails for a [`Request::PhotoThumbs`] batch.
|
||||
Thumbs { items: Vec<PhotoThumb> },
|
||||
/// A [`Request::FileThumbs`] generation is no longer current. This is
|
||||
/// retryable after reserving a fresh generation and must never be interpreted
|
||||
/// as a permanent "no thumbnail" verdict.
|
||||
FileThumbsStale,
|
||||
/// Daemon-issued generation reserved by
|
||||
/// [`Request::ReserveFileThumbGeneration`].
|
||||
FileThumbGeneration { generation: u64 },
|
||||
/// Current recursive thumbnail-build progress.
|
||||
ThumbnailBuild { status: ThumbnailBuildStatus },
|
||||
/// An on-disk path the front-end can open (e.g. a downloaded photo).
|
||||
FilePath { path: String },
|
||||
/// Full-text search results (reply to [`Request::Search`]).
|
||||
@@ -1409,7 +1582,7 @@ impl ErrorKind {
|
||||
}
|
||||
|
||||
/// Send one [`Request`] to the daemon listening on `socket` and read its
|
||||
/// [`Response`]. Errors (e.g. [`Error::Io`]) if no daemon is listening.
|
||||
/// [`Response`]. Returns a crate [`crate::Error`] if no daemon is listening.
|
||||
///
|
||||
/// Shared by the CLI and GUI so both speak the wire format identically.
|
||||
pub fn send(socket: &Path, req: &Request) -> Result<Response> {
|
||||
@@ -1568,6 +1741,86 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn file_thumbnail_requests_keep_their_revision_tags() {
|
||||
let request = Request::FileThumbs {
|
||||
items: vec![
|
||||
FileThumbRequest {
|
||||
uid: "volume~first".into(),
|
||||
modified: 1_700_000_001,
|
||||
name: "first.jpg".into(),
|
||||
},
|
||||
FileThumbRequest {
|
||||
uid: "volume~second".into(),
|
||||
modified: 1_700_000_002,
|
||||
name: "second.nef".into(),
|
||||
},
|
||||
],
|
||||
generation: 42,
|
||||
};
|
||||
let line = serde_json::to_string(&request).unwrap();
|
||||
let decoded: Request = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_file_thumbnail_requests_default_the_name() {
|
||||
let decoded: Request = serde_json::from_str(
|
||||
r#"{"FileThumbs":{"items":[{"uid":"volume~first","modified":1700000001}],"generation":42}}"#,
|
||||
)
|
||||
.unwrap();
|
||||
let Request::FileThumbs { items, generation } = decoded else {
|
||||
panic!("decoded the wrong request variant");
|
||||
};
|
||||
assert_eq!(generation, 42);
|
||||
assert_eq!(items.len(), 1);
|
||||
assert!(items[0].name.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn thumbnail_build_requests_and_progress_roundtrip() {
|
||||
let requests = [
|
||||
Request::ReserveFileThumbGeneration,
|
||||
Request::CancelFileThumbs { generation: 43 },
|
||||
Request::StartThumbnailBuild {
|
||||
path: "pictures/events".into(),
|
||||
},
|
||||
Request::CancelThumbnailBuild,
|
||||
Request::ThumbnailBuildStatus,
|
||||
];
|
||||
for request in requests {
|
||||
let line = serde_json::to_string(&request).unwrap();
|
||||
let decoded: Request = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
|
||||
}
|
||||
|
||||
let response = Response::ThumbnailBuild {
|
||||
status: ThumbnailBuildStatus {
|
||||
running: true,
|
||||
scanning: false,
|
||||
path: "pictures/events".into(),
|
||||
folders_scanned: 12,
|
||||
images_found: 80,
|
||||
completed: 25,
|
||||
failed: 1,
|
||||
message: None,
|
||||
},
|
||||
};
|
||||
let line = serde_json::to_string(&response).unwrap();
|
||||
let decoded: Response = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
|
||||
|
||||
let response = Response::FileThumbGeneration { generation: 44 };
|
||||
let line = serde_json::to_string(&response).unwrap();
|
||||
let decoded: Response = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
|
||||
|
||||
let response = Response::FileThumbsStale;
|
||||
let line = serde_json::to_string(&response).unwrap();
|
||||
let decoded: Response = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unified_search_request_and_response_roundtrip() {
|
||||
let request = Request::SearchV2 {
|
||||
|
||||
@@ -9,7 +9,7 @@ use super::Db;
|
||||
use crate::Result;
|
||||
|
||||
/// Current schema version. Bump on every forward migration added below.
|
||||
pub(super) const SCHEMA_VERSION: i64 = 25;
|
||||
pub(super) const SCHEMA_VERSION: i64 = 28;
|
||||
|
||||
impl Db {
|
||||
pub(super) fn migrate(&self) -> Result<()> {
|
||||
@@ -161,6 +161,63 @@ impl Db {
|
||||
tx.execute_batch(MIGRATION_V25)?;
|
||||
}
|
||||
}
|
||||
if current < 26 {
|
||||
// Same guard as V22: a fixture that never exercised the queue may
|
||||
// not have `pending_op`, and altering an absent table is an error
|
||||
// rather than a no-op.
|
||||
// A fixture built from the current schema and then rewound to an
|
||||
// older `schema_version` already has the column, and `ADD COLUMN`
|
||||
// has no `IF NOT EXISTS`.
|
||||
let has_column: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM pragma_table_info('pending_op') WHERE name = 'claimed_at'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
let has_ops: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
if has_ops && !has_column {
|
||||
tx.execute_batch(MIGRATION_V26)?;
|
||||
}
|
||||
}
|
||||
if current < 27 {
|
||||
// Same guards as V26, for the same two reasons: the table may not
|
||||
// exist in an old fixture, and a fixture rewound from the current
|
||||
// schema already has the column.
|
||||
let has_column: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM pragma_table_info('pending_op') \
|
||||
WHERE name = 'access_deferred_since'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
let has_ops: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
if has_ops && !has_column {
|
||||
tx.execute_batch(MIGRATION_V27)?;
|
||||
}
|
||||
}
|
||||
if current < 28 {
|
||||
// Same guards as V26/V27, for the same two reasons.
|
||||
let has_column: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM pragma_table_info('sync_entry') \
|
||||
WHERE name = 'local_mtime_ns'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
let has_entries: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'sync_entry'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
if has_entries && !has_column {
|
||||
tx.execute_batch(MIGRATION_V28)?;
|
||||
}
|
||||
}
|
||||
tx.execute(
|
||||
"INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1)
|
||||
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
|
||||
@@ -725,3 +782,67 @@ UPDATE nodes
|
||||
|
||||
DROP TABLE node_paths;
|
||||
";
|
||||
|
||||
/// V26: `pending_op.claimed_at` — the drain's claim column.
|
||||
///
|
||||
/// The drain was one thread taking one op at a time, so a 10 GiB upload held
|
||||
/// every queued rename and small write behind it for as long as it ran. Several
|
||||
/// workers can share the queue only if a claim is visible to the others: this
|
||||
/// column is what makes "already being drained by someone" a fact about the row
|
||||
/// rather than about one thread's local variable.
|
||||
///
|
||||
/// `0` is unclaimed; anything else is the ms at which a worker took it. A
|
||||
/// claim is process-local state that outlives nothing — the single-writer lock
|
||||
/// means a non-zero value found at open belongs to a crashed run, and
|
||||
/// [`Db::clear_op_claims`](super::Db::clear_op_claims) drops the lot.
|
||||
///
|
||||
/// The index is partial so it costs nothing on the ordinary row: only the
|
||||
/// handful of claimed rows are in it, which is exactly the set the claim query
|
||||
/// has to subtract to keep two workers off one node.
|
||||
const MIGRATION_V26: &str = "
|
||||
ALTER TABLE pending_op ADD COLUMN claimed_at INTEGER NOT NULL DEFAULT 0;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_pending_op_claimed
|
||||
ON pending_op(uid) WHERE claimed_at <> 0;
|
||||
";
|
||||
|
||||
/// Schema v27: when an op was first deferred by the local access check.
|
||||
///
|
||||
/// An access deferral consumes no attempt and records no error — the op has not
|
||||
/// failed remotely, it has not been sent at all. That is right for a permission
|
||||
/// change the user is about to undo, and wrong for a condition that never
|
||||
/// clears: the row is re-deferred every
|
||||
/// [`DRAIN_ACCESS_RECHECK`](../../../pdfs_fuse/drain/constant.DRAIN_ACCESS_RECHECK.html)
|
||||
/// forever while `attempts`, `last_error`, `failing` and the logs all stay
|
||||
/// empty, so a queue that can never drain is indistinguishable from a busy one.
|
||||
/// Found in the field as 28 revision ops holding 18 GiB of accepted writes for
|
||||
/// 30 days, none of them visible anywhere.
|
||||
///
|
||||
/// `0` means "not currently access-deferred"; anything else is the ms at which
|
||||
/// the first consecutive deferral happened, which is what
|
||||
/// [`Db::defer_op_for_access`](super::Db::defer_op_for_access) measures the
|
||||
/// deferral against.
|
||||
const MIGRATION_V27: &str = "
|
||||
ALTER TABLE pending_op ADD COLUMN access_deferred_since INTEGER NOT NULL DEFAULT 0;
|
||||
";
|
||||
|
||||
/// Schema v28: sub-second precision for a mirror folder's local baseline.
|
||||
///
|
||||
/// A reconcile pass decides whether the local side of a path changed by
|
||||
/// comparing `(local_mtime, local_size)` against the baseline. Both sides of
|
||||
/// that comparison were whole seconds, so an edit that landed in the same second
|
||||
/// as the last sync and left the file the same length — a fixed-size record
|
||||
/// rewritten, an image re-exported, a database page updated in place — read as
|
||||
/// *unchanged* and was never uploaded. The remote copy silently stayed at the
|
||||
/// older content until something else about the file moved (bugs.md B25).
|
||||
///
|
||||
/// Nullable rather than a converted `local_mtime * 1_000_000_000`: the
|
||||
/// nanosecond part of an existing baseline is genuinely unknown, and inventing
|
||||
/// zero for it would make every already-synced file compare as changed and
|
||||
/// re-upload an entire mirror on first start. `NULL` means "this row predates
|
||||
/// the column", which the comparison reads as "fall back to whole seconds" —
|
||||
/// exactly the behaviour that row was written under. Each row gains a real value
|
||||
/// the next time its path is synced.
|
||||
const MIGRATION_V28: &str = "
|
||||
ALTER TABLE sync_entry ADD COLUMN local_mtime_ns INTEGER;
|
||||
";
|
||||
@@ -230,7 +230,7 @@ impl Db {
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
};
|
||||
Ok(Self {
|
||||
let db = Self {
|
||||
conn: Mutex::new(conn),
|
||||
// No pool for `:memory:`. A second connection to that name is a
|
||||
// second, empty database rather than another view of this one, so
|
||||
@@ -240,7 +240,17 @@ impl Db {
|
||||
idle: Mutex::new(Vec::new()),
|
||||
}),
|
||||
_single_writer: single_writer,
|
||||
})
|
||||
};
|
||||
// A drain claim names a worker thread in the process that took it, and
|
||||
// the single-writer lock above says that process is not running. Any
|
||||
// claim still on disk is therefore a crashed run's, and leaving it would
|
||||
// hide those ops from every worker for good.
|
||||
match db.clear_op_claims() {
|
||||
Ok(0) => {}
|
||||
Ok(n) => tracing::info!(ops = n, "released drain claims left by a previous run"),
|
||||
Err(e) => tracing::warn!(error = %e, "clearing stale drain claims failed"),
|
||||
}
|
||||
Ok(db)
|
||||
}
|
||||
|
||||
/// A connection for a `SELECT`-only method.
|
||||
|
||||
@@ -512,17 +512,115 @@ impl Db {
|
||||
Ok(op)
|
||||
}
|
||||
|
||||
/// Take the oldest due op for this worker, marking it claimed so no other
|
||||
/// drain worker takes it, or `None` when there is nothing to do.
|
||||
///
|
||||
/// The drain is several workers over one queue, which is only safe because
|
||||
/// the claim is a column: a row is offered to exactly one worker, and the
|
||||
/// worker that took it is the only one that may retire, defer or fail it.
|
||||
///
|
||||
/// Two exclusions on top of [`next_due_op`](Self::next_due_op)'s:
|
||||
///
|
||||
/// * `claimed_at = 0` — not already in flight somewhere else.
|
||||
/// * no *other* claimed op shares this `uid`. Ordering only has to hold per
|
||||
/// node, but there it has to hold absolutely: a queued rename and a queued
|
||||
/// revision of one file must land in the order they were made, and two
|
||||
/// workers on one node would also race over its staged blob and its
|
||||
/// pending entry. `ORDER BY id` plus this exclusion gives per-uid serial
|
||||
/// order for free — the older op is claimed first and the newer one is
|
||||
/// invisible until it retires.
|
||||
///
|
||||
/// Select and mark are one transaction, so two workers cannot both see the
|
||||
/// row as unclaimed.
|
||||
pub fn claim_next_due_op(&self, now: i64) -> Result<Option<PendingOp>> {
|
||||
let mut conn = self.conn.lock();
|
||||
let tx = conn.transaction()?;
|
||||
let op = {
|
||||
let mut stmt = tx.prepare_cached(&format!(
|
||||
"SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
|
||||
attempts, last_error, next_attempt_at
|
||||
FROM pending_op
|
||||
WHERE next_attempt_at <= ?1
|
||||
AND claimed_at = 0
|
||||
AND (parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~')
|
||||
AND uid NOT IN (SELECT uid FROM pending_op WHERE claimed_at <> 0)
|
||||
ORDER BY id LIMIT 1",
|
||||
v = LOCAL_VOLUME,
|
||||
n = LOCAL_VOLUME.len() + 1,
|
||||
))?;
|
||||
stmt.query_row(params![now], |r| {
|
||||
Ok(PendingOp {
|
||||
id: r.get(0)?,
|
||||
kind: r.get(1)?,
|
||||
uid: r.get(2)?,
|
||||
parent_uid: r.get(3)?,
|
||||
name: r.get(4)?,
|
||||
blob_path: r.get(5)?,
|
||||
meta_json: r.get(6)?,
|
||||
created_at: r.get(7)?,
|
||||
attempts: r.get(8)?,
|
||||
last_error: r.get(9)?,
|
||||
next_attempt_at: r.get(10)?,
|
||||
})
|
||||
})
|
||||
.optional()?
|
||||
};
|
||||
if let Some(op) = op.as_ref() {
|
||||
tx.execute(
|
||||
"UPDATE pending_op SET claimed_at = ?2 WHERE id = ?1",
|
||||
params![op.id, now.max(1)],
|
||||
)?;
|
||||
}
|
||||
tx.commit()?;
|
||||
Ok(op)
|
||||
}
|
||||
|
||||
/// Give a claimed op back to the queue, whatever became of the attempt.
|
||||
///
|
||||
/// Called on every path out of a drain attempt rather than only the failing
|
||||
/// ones: a handler that retires its own row leaves nothing to release (the
|
||||
/// `UPDATE` matches nothing), and a handler that returns without retiring
|
||||
/// *and* without failing would otherwise leave the row claimed by a worker
|
||||
/// that has moved on — invisible to every worker, forever.
|
||||
pub fn release_op_claim(&self, id: i64) -> Result<()> {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute(
|
||||
"UPDATE pending_op SET claimed_at = 0 WHERE id = ?1",
|
||||
params![id],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Drop every claim in the table.
|
||||
///
|
||||
/// A claim describes a worker in *this* process, so one surviving on disk
|
||||
/// means the previous run died holding it. The single-writer lock
|
||||
/// ([`Db::open`]) is what makes that inference safe: nobody else can be
|
||||
/// draining this queue while we are opening it.
|
||||
pub fn clear_op_claims(&self) -> Result<usize> {
|
||||
let conn = self.conn.lock();
|
||||
Ok(conn.execute(
|
||||
"UPDATE pending_op SET claimed_at = 0 WHERE claimed_at <> 0",
|
||||
[],
|
||||
)?)
|
||||
}
|
||||
|
||||
/// The earliest `next_attempt_at` among all queued ops, or `None` if the
|
||||
/// queue is empty. Used by the drain loop to sleep exactly until the next
|
||||
/// debounced or backed-off op becomes eligible rather than waiting the full
|
||||
/// idle-poll interval.
|
||||
///
|
||||
/// Claimed ops are excluded: another worker is already on them, so they are
|
||||
/// not work this one is waiting for, and counting them would have an idle
|
||||
/// worker spin on a row it cannot have.
|
||||
pub fn earliest_due_at(&self) -> Result<Option<i64>> {
|
||||
let conn = self.read();
|
||||
let ts: Option<i64> = conn
|
||||
.query_row(
|
||||
&format!(
|
||||
"SELECT MIN(next_attempt_at) FROM pending_op \
|
||||
WHERE parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~'",
|
||||
WHERE claimed_at = 0 \
|
||||
AND (parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~')",
|
||||
v = LOCAL_VOLUME,
|
||||
n = LOCAL_VOLUME.len() + 1,
|
||||
),
|
||||
@@ -623,11 +721,16 @@ impl Db {
|
||||
|
||||
/// Record a failed attempt and when to next try. Leaves the row in place —
|
||||
/// the staged bytes are still the only copy of the user's write.
|
||||
///
|
||||
/// Clears any access-deferral window: whatever the op was waiting for has
|
||||
/// now been reported, so the next uncleared deferral starts its own window
|
||||
/// rather than escalating again on the following recheck.
|
||||
pub fn record_op_failure(&self, id: i64, error: &str, next_attempt_at: i64) -> Result<()> {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute(
|
||||
"UPDATE pending_op
|
||||
SET attempts = attempts + 1, last_error = ?2, next_attempt_at = ?3
|
||||
SET attempts = attempts + 1, last_error = ?2, next_attempt_at = ?3,
|
||||
access_deferred_since = 0
|
||||
WHERE id = ?1",
|
||||
params![id, error, next_attempt_at],
|
||||
)?;
|
||||
@@ -653,6 +756,60 @@ impl Db {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Defer an op the local access check refused, and report how long it has
|
||||
/// been refused for.
|
||||
///
|
||||
/// Identical to [`defer_op_without_attempt`](Self::defer_op_without_attempt)
|
||||
/// on the retry clock, plus the one thing that was missing: the first
|
||||
/// deferral of a run stamps `access_deferred_since`, and every deferral
|
||||
/// returns that stamp. A caller can therefore tell a deferral that is one
|
||||
/// recheck old from one that has been repeating unchanged for days, which is
|
||||
/// the difference between a permission the user is mid-way through changing
|
||||
/// and a queue that will never drain.
|
||||
///
|
||||
/// Returns the ms at which the current run of deferrals began — `now` on the
|
||||
/// first one.
|
||||
pub fn defer_op_for_access(&self, id: i64, now: i64, next_attempt_at: i64) -> Result<i64> {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute(
|
||||
"UPDATE pending_op
|
||||
SET next_attempt_at = CASE
|
||||
WHEN next_attempt_at >= ?4 THEN next_attempt_at
|
||||
ELSE ?3
|
||||
END,
|
||||
access_deferred_since = CASE
|
||||
WHEN access_deferred_since = 0 THEN ?2
|
||||
ELSE access_deferred_since
|
||||
END
|
||||
WHERE id = ?1",
|
||||
params![id, now, next_attempt_at, PARK_UNTIL],
|
||||
)?;
|
||||
let since = conn
|
||||
.query_row(
|
||||
"SELECT access_deferred_since FROM pending_op WHERE id = ?1",
|
||||
params![id],
|
||||
|r| r.get::<_, i64>(0),
|
||||
)
|
||||
.optional()?;
|
||||
// A row that drained out from under this call leaves nothing to measure;
|
||||
// reporting `now` says "this run just started", which is the reading that
|
||||
// never escalates something that no longer exists.
|
||||
Ok(since.filter(|since| *since != 0).unwrap_or(now))
|
||||
}
|
||||
|
||||
/// Forget an op's access-deferral window, after an attempt the access check
|
||||
/// let through. The op is no longer blocked, so a later deferral is the
|
||||
/// start of a new run rather than the continuation of an old one.
|
||||
pub fn clear_op_access_deferral(&self, id: i64) -> Result<()> {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute(
|
||||
"UPDATE pending_op SET access_deferred_since = 0
|
||||
WHERE id = ?1 AND access_deferred_since <> 0",
|
||||
params![id],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Every staged blob the queue still refers to.
|
||||
///
|
||||
/// The startup staging reconcile subtracts this from what is actually in
|
||||
|
||||
@@ -35,6 +35,14 @@ pub struct StoredSyncEntry {
|
||||
pub remote_uid: Option<String>,
|
||||
pub local_mtime: i64,
|
||||
pub local_size: i64,
|
||||
/// The same modification time to nanosecond precision, when it is known.
|
||||
///
|
||||
/// `local_mtime` alone cannot tell an edit that landed inside the same
|
||||
/// second as the last sync — and left the file the same length — from no
|
||||
/// edit at all, so those never uploaded (bugs.md B25). `None` on a row
|
||||
/// written before schema 28, which the comparison reads as "compare whole
|
||||
/// seconds", so upgrading does not make an entire mirror look changed.
|
||||
pub local_mtime_ns: Option<i64>,
|
||||
pub remote_rev: Option<String>,
|
||||
pub remote_hash: Option<String>,
|
||||
}
|
||||
@@ -168,7 +176,8 @@ impl Db {
|
||||
pub fn sync_entries(&self, folder_id: i64) -> Result<HashMap<String, StoredSyncEntry>> {
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash
|
||||
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash,
|
||||
local_mtime_ns
|
||||
FROM sync_entry WHERE folder_id = ?1",
|
||||
)?;
|
||||
let rows = stmt.query_map(params![folder_id], |r| {
|
||||
@@ -179,6 +188,7 @@ impl Db {
|
||||
local_size: r.get(3)?,
|
||||
remote_rev: r.get(4)?,
|
||||
remote_hash: r.get(5)?,
|
||||
local_mtime_ns: r.get(6)?,
|
||||
})
|
||||
})?;
|
||||
let mut map = HashMap::new();
|
||||
@@ -216,14 +226,16 @@ impl Db {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute(
|
||||
"INSERT INTO sync_entry
|
||||
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
|
||||
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev,
|
||||
remote_hash, local_mtime_ns)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)
|
||||
ON CONFLICT(folder_id, rel_path) DO UPDATE SET
|
||||
remote_uid = excluded.remote_uid,
|
||||
local_mtime = excluded.local_mtime,
|
||||
local_size = excluded.local_size,
|
||||
remote_rev = excluded.remote_rev,
|
||||
remote_hash = excluded.remote_hash",
|
||||
remote_uid = excluded.remote_uid,
|
||||
local_mtime = excluded.local_mtime,
|
||||
local_size = excluded.local_size,
|
||||
remote_rev = excluded.remote_rev,
|
||||
remote_hash = excluded.remote_hash,
|
||||
local_mtime_ns = excluded.local_mtime_ns",
|
||||
params![
|
||||
folder_id,
|
||||
e.rel_path,
|
||||
@@ -232,6 +244,7 @@ impl Db {
|
||||
e.local_size,
|
||||
e.remote_rev,
|
||||
e.remote_hash,
|
||||
e.local_mtime_ns,
|
||||
],
|
||||
)?;
|
||||
Ok(())
|
||||
|
||||
@@ -1895,6 +1895,123 @@ fn failed_superseding_insert_keeps_the_old_pending_op() {
|
||||
assert_eq!(ops[0].blob_path.as_deref(), Some("/staging/original"));
|
||||
}
|
||||
|
||||
/// B53's remaining half. The insert-failure regression above proves the *row*
|
||||
/// rolls back; this proves the two things that make the rollback usable — the
|
||||
/// old blob is not reported as superseded (so the caller does not delete the
|
||||
/// bytes the surviving row still points at), and the rollback survives a
|
||||
/// restart rather than living only in the connection that failed.
|
||||
#[test]
|
||||
fn a_rolled_back_supersede_keeps_its_blob_and_survives_reopen() {
|
||||
let dir = std::env::temp_dir().join(format!("pdfs-db-b53-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let path = dir.join("queue.sqlite");
|
||||
let op = |blob: &str| PendingOp {
|
||||
id: 0,
|
||||
kind: OP_REVISION.to_string(),
|
||||
uid: uid("durable").to_string(),
|
||||
parent_uid: None,
|
||||
name: None,
|
||||
blob_path: Some(blob.to_string()),
|
||||
meta_json: Some("{}".to_string()),
|
||||
created_at: 1,
|
||||
attempts: 0,
|
||||
last_error: None,
|
||||
next_attempt_at: 0,
|
||||
};
|
||||
|
||||
{
|
||||
let db = Db::open(&path).unwrap();
|
||||
db.enqueue_op(&op("/staging/original")).unwrap();
|
||||
db.with_conn(|conn| {
|
||||
conn.execute_batch(
|
||||
"CREATE TRIGGER reject_replacement BEFORE INSERT ON pending_op
|
||||
WHEN NEW.blob_path = '/staging/replacement'
|
||||
BEGIN SELECT RAISE(ABORT, 'injected insert failure'); END;",
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
|
||||
// The failure must not hand back the old blob path. That value is the
|
||||
// caller's instruction to delete those bytes, and the row that owns
|
||||
// them is still queued.
|
||||
assert!(db.enqueue_op(&op("/staging/replacement")).is_err());
|
||||
db.with_conn(|conn| {
|
||||
conn.execute_batch("DROP TRIGGER reject_replacement")?;
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let db = Db::open(&path).unwrap();
|
||||
let ops = db.pending_ops().unwrap();
|
||||
assert_eq!(ops.len(), 1, "old or new — never neither");
|
||||
assert_eq!(ops[0].blob_path.as_deref(), Some("/staging/original"));
|
||||
|
||||
// And once the injected failure is gone, superseding works and *does*
|
||||
// report the blob it retired.
|
||||
let (_, superseded) = db.enqueue_op(&op("/staging/replacement")).unwrap();
|
||||
assert_eq!(superseded.as_deref(), Some("/staging/original"));
|
||||
assert_eq!(db.pending_ops().unwrap().len(), 1);
|
||||
|
||||
drop(db);
|
||||
std::fs::remove_dir_all(&dir).unwrap();
|
||||
}
|
||||
|
||||
/// A full disk, as SQLite reports it. `max_page_count` makes the database
|
||||
/// refuse to grow, which is the same `SQLITE_FULL` an out-of-space filesystem
|
||||
/// produces — and the queue must come out of it with the acknowledged work
|
||||
/// still in it (B53).
|
||||
#[test]
|
||||
fn a_full_database_rolls_the_supersede_back_rather_than_erasing_it() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
let op = |blob: &str| PendingOp {
|
||||
id: 0,
|
||||
kind: OP_REVISION.to_string(),
|
||||
uid: uid("full").to_string(),
|
||||
parent_uid: None,
|
||||
name: None,
|
||||
blob_path: Some(blob.to_string()),
|
||||
meta_json: Some("{}".to_string()),
|
||||
created_at: 1,
|
||||
attempts: 0,
|
||||
last_error: None,
|
||||
next_attempt_at: 0,
|
||||
};
|
||||
db.enqueue_op(&op("/staging/original")).unwrap();
|
||||
|
||||
// Cap the file at its current size, so any page the insert needs fails.
|
||||
let pages: i64 = db
|
||||
.with_conn(|conn| Ok(conn.query_row("PRAGMA page_count", [], |r| r.get(0))?))
|
||||
.unwrap();
|
||||
db.with_conn(|conn| {
|
||||
conn.execute_batch(&format!("PRAGMA max_page_count = {pages}"))?;
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
|
||||
// A payload far larger than one page, so growth is unavoidable.
|
||||
let mut fat = op("/staging/replacement");
|
||||
fat.meta_json = Some("x".repeat(512 * 1024));
|
||||
let result = db.enqueue_op(&fat);
|
||||
|
||||
db.with_conn(|conn| {
|
||||
conn.execute_batch("PRAGMA max_page_count = 1073741823")?;
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
|
||||
assert!(result.is_err(), "a database that cannot grow must refuse");
|
||||
let ops = db.pending_ops().unwrap();
|
||||
assert_eq!(ops.len(), 1);
|
||||
assert_eq!(
|
||||
ops[0].blob_path.as_deref(),
|
||||
Some("/staging/original"),
|
||||
"the acknowledged upload is still queued and still owns its bytes"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn atomic_trash_replacement_returns_owned_blobs_after_commit() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
@@ -2408,6 +2525,7 @@ fn mirror_uid_lookup_covers_the_root_and_synced_descendants_only() {
|
||||
local_size: 2,
|
||||
remote_rev: Some("3".into()),
|
||||
remote_hash: Some("2".into()),
|
||||
local_mtime_ns: None,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
@@ -2424,6 +2542,7 @@ fn mirror_uid_lookup_covers_the_root_and_synced_descendants_only() {
|
||||
local_size: 2,
|
||||
remote_rev: Some("3".into()),
|
||||
remote_hash: Some("2".into()),
|
||||
local_mtime_ns: None,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
@@ -3491,6 +3610,132 @@ fn migration_v25_backfills_paths_for_an_existing_tree() {
|
||||
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
|
||||
}
|
||||
|
||||
/// A queue carried over from V26 keeps its rows, and every one of them starts
|
||||
/// out not access-deferred — a migration must not invent a deferral window that
|
||||
/// would report an ordinary queued upload as blocked.
|
||||
#[test]
|
||||
fn migration_v27_adds_the_access_deferral_column_to_a_v26_queue() {
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"pdfs-db-v26-fixture-{}-{}.db",
|
||||
std::process::id(),
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_nanos()
|
||||
));
|
||||
{
|
||||
let db = Db::open(&path).unwrap();
|
||||
db.enqueue_op(&crate::db::PendingOp {
|
||||
id: 0,
|
||||
kind: crate::db::OP_REVISION.to_string(),
|
||||
uid: "v~l".to_string(),
|
||||
parent_uid: None,
|
||||
name: None,
|
||||
blob_path: Some("/tmp/blob".to_string()),
|
||||
meta_json: Some("{}".to_string()),
|
||||
created_at: 1,
|
||||
attempts: 0,
|
||||
last_error: None,
|
||||
next_attempt_at: 0,
|
||||
})
|
||||
.unwrap();
|
||||
}
|
||||
{
|
||||
// Put the file back in the state a released V26 database was in.
|
||||
let conn = rusqlite::Connection::open(&path).unwrap();
|
||||
conn.execute_batch(
|
||||
"ALTER TABLE pending_op DROP COLUMN access_deferred_since;
|
||||
UPDATE sync_state SET value = '26' WHERE key = 'schema_version';",
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let db = Db::open(&path).unwrap();
|
||||
let ops = db.pending_ops().unwrap();
|
||||
assert_eq!(ops.len(), 1, "the migration keeps the queued write");
|
||||
let now = 5_000i64;
|
||||
assert_eq!(
|
||||
db.defer_op_for_access(ops[0].id, now, now + 5_000).unwrap(),
|
||||
now,
|
||||
"a carried-over op is not already deferred"
|
||||
);
|
||||
|
||||
let _ = std::fs::remove_file(&path);
|
||||
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
|
||||
}
|
||||
|
||||
/// A mirror baseline carried over from V27 keeps its rows, and each one reads
|
||||
/// back with no nanosecond time. That `None` is the whole point: it makes the
|
||||
/// comparison fall back to whole seconds for rows written before the column
|
||||
/// existed, instead of asserting a sub-second time of zero and re-uploading
|
||||
/// every already-synced file in the folder (bugs.md B25).
|
||||
#[test]
|
||||
fn migration_v28_adds_sub_second_local_times_without_disturbing_a_v27_baseline() {
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"pdfs-db-v27-fixture-{}-{}.db",
|
||||
std::process::id(),
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_nanos()
|
||||
));
|
||||
{
|
||||
let db = Db::open(&path).unwrap();
|
||||
let folder = db
|
||||
.sync_folder_add("/home/me/Mirror", "vol~root", "share")
|
||||
.unwrap();
|
||||
db.sync_entry_upsert(
|
||||
folder,
|
||||
&StoredSyncEntry {
|
||||
rel_path: "notes.txt".into(),
|
||||
remote_uid: Some("vol~notes".into()),
|
||||
local_mtime: 1_700_000_000,
|
||||
local_mtime_ns: Some(1_700_000_000_123_456_789),
|
||||
local_size: 42,
|
||||
remote_rev: Some("1700000000".into()),
|
||||
remote_hash: Some("42".into()),
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
{
|
||||
// Put the file back in the state a released V27 database was in.
|
||||
let conn = rusqlite::Connection::open(&path).unwrap();
|
||||
conn.execute_batch(
|
||||
"ALTER TABLE sync_entry DROP COLUMN local_mtime_ns;
|
||||
UPDATE sync_state SET value = '27' WHERE key = 'schema_version';",
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let db = Db::open(&path).unwrap();
|
||||
let entries = db.sync_entries(1).unwrap();
|
||||
let entry = entries.get("notes.txt").expect("the baseline row survives");
|
||||
assert_eq!(entry.local_mtime, 1_700_000_000);
|
||||
assert_eq!(entry.local_size, 42);
|
||||
assert_eq!(
|
||||
entry.local_mtime_ns, None,
|
||||
"a carried-over row must not claim a sub-second time it never recorded"
|
||||
);
|
||||
|
||||
// And the column is live: a refreshed row keeps the precision it is given.
|
||||
db.sync_entry_upsert(
|
||||
1,
|
||||
&StoredSyncEntry {
|
||||
local_mtime_ns: Some(1_700_000_001_000_000_500),
|
||||
..entry.clone()
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
db.sync_entries(1).unwrap()["notes.txt"].local_mtime_ns,
|
||||
Some(1_700_000_001_000_000_500)
|
||||
);
|
||||
|
||||
let _ = std::fs::remove_file(&path);
|
||||
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
|
||||
}
|
||||
|
||||
// --- read-only connection pool (DB2) ---------------------------------------
|
||||
|
||||
/// A read must not queue behind whatever happens to be writing.
|
||||
@@ -3545,3 +3790,114 @@ fn a_read_does_not_wait_for_the_write_connection() {
|
||||
let _ = std::fs::remove_file(&path);
|
||||
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
|
||||
}
|
||||
|
||||
/// §5: the drain is several workers over one queue, and the claim column is the
|
||||
/// only thing keeping them off each other's work. Two properties matter — no
|
||||
/// row is ever handed to two workers, and no *node* is ever worked by two
|
||||
/// workers even though its ops are separate rows.
|
||||
#[test]
|
||||
fn a_claimed_op_is_invisible_to_every_other_worker() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
let root = uid("root").to_string();
|
||||
for i in 0..3 {
|
||||
db.enqueue_op(&bulk_op(i, &root)).unwrap();
|
||||
}
|
||||
|
||||
let first = db.claim_next_due_op(10).unwrap().expect("an op is due");
|
||||
let second = db
|
||||
.claim_next_due_op(10)
|
||||
.unwrap()
|
||||
.expect("another op is due");
|
||||
let third = db
|
||||
.claim_next_due_op(10)
|
||||
.unwrap()
|
||||
.expect("a third op is due");
|
||||
assert!(
|
||||
db.claim_next_due_op(10).unwrap().is_none(),
|
||||
"a fourth claim must find nothing left: every queued op is taken"
|
||||
);
|
||||
|
||||
let mut ids = [first.id, second.id, third.id];
|
||||
ids.sort_unstable();
|
||||
assert_eq!(
|
||||
ids,
|
||||
[1, 2, 3],
|
||||
"each worker must get a distinct row, oldest first"
|
||||
);
|
||||
|
||||
// Releasing puts one back, and it is the one that comes out next.
|
||||
db.release_op_claim(second.id).unwrap();
|
||||
let again = db.claim_next_due_op(10).unwrap().expect("the released op");
|
||||
assert_eq!(again.id, second.id);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn two_workers_never_share_a_node() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
let root = uid("root").to_string();
|
||||
// Two ops of different kinds against one node — a queued rename and a
|
||||
// queued revision, which is an ordinary `mv` plus a save.
|
||||
let mut revision = bulk_op(0, &root);
|
||||
revision.parent_uid = None;
|
||||
db.enqueue_op(&revision).unwrap();
|
||||
let rename = PendingOp {
|
||||
kind: OP_RENAME.to_string(),
|
||||
parent_uid: Some(root.clone()),
|
||||
blob_path: None,
|
||||
meta_json: None,
|
||||
..bulk_op(0, &root)
|
||||
};
|
||||
db.enqueue_op(&rename).unwrap();
|
||||
|
||||
let claimed = db.claim_next_due_op(10).unwrap().expect("an op is due");
|
||||
assert_eq!(claimed.kind, OP_REVISION, "the older op goes first");
|
||||
assert!(
|
||||
db.claim_next_due_op(10).unwrap().is_none(),
|
||||
"the second op targets the same node, so it must wait for the first: \
|
||||
two workers on one uid would race over its staged blob and land its \
|
||||
changes out of order"
|
||||
);
|
||||
|
||||
db.release_op_claim(claimed.id).unwrap();
|
||||
db.delete_op(claimed.id).unwrap();
|
||||
assert_eq!(
|
||||
db.claim_next_due_op(10).unwrap().map(|o| o.kind),
|
||||
Some(OP_RENAME.to_string()),
|
||||
"once the revision retires, the rename is claimable"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn claims_do_not_survive_the_process_that_took_them() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
let root = uid("root").to_string();
|
||||
for i in 0..2 {
|
||||
db.enqueue_op(&bulk_op(i, &root)).unwrap();
|
||||
}
|
||||
db.claim_next_due_op(10).unwrap().unwrap();
|
||||
db.claim_next_due_op(10).unwrap().unwrap();
|
||||
assert!(db.claim_next_due_op(10).unwrap().is_none());
|
||||
|
||||
// What `Db::open` does after the single-writer lock says the previous run
|
||||
// is gone. Without it those ops are invisible to every worker forever.
|
||||
assert_eq!(db.clear_op_claims().unwrap(), 2);
|
||||
assert!(
|
||||
db.claim_next_due_op(10).unwrap().is_some(),
|
||||
"a crashed run's claims must not park its queue"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_idle_worker_does_not_wait_on_an_op_another_worker_has() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
let root = uid("root").to_string();
|
||||
db.enqueue_op(&bulk_op(0, &root)).unwrap();
|
||||
|
||||
assert_eq!(db.earliest_due_at().unwrap(), Some(0));
|
||||
db.claim_next_due_op(10).unwrap().unwrap();
|
||||
assert_eq!(
|
||||
db.earliest_due_at().unwrap(),
|
||||
None,
|
||||
"an op somebody else is draining is not work this worker is waiting for"
|
||||
);
|
||||
}
|
||||
@@ -20,6 +20,7 @@ pub mod search;
|
||||
pub mod service;
|
||||
pub mod shell;
|
||||
pub mod syncignore;
|
||||
pub mod takeout;
|
||||
|
||||
pub use access::{Access, access_for, perm_bits};
|
||||
pub use error::{CoreError, CoreResult, Error, Result};
|
||||
|
||||
@@ -0,0 +1,887 @@
|
||||
//! Reading a Google Photos **Takeout** export: what is in the archives, which
|
||||
//! album each photo belongs to, and when it was taken.
|
||||
//!
|
||||
//! This module is pure structure — it opens the archives' central directories
|
||||
//! and reads the small JSON sidecars, never the media bytes. The import itself
|
||||
//! (hashing, duplicate detection, upload, album creation) lives in the daemon;
|
||||
//! everything here is offline-testable and has no Proton dependency.
|
||||
//!
|
||||
//! # What a Takeout export looks like
|
||||
//!
|
||||
//! Google splits an export into numbered zips (`takeout-20260817T...-001.zip`,
|
||||
//! `-002.zip`, …) that are *independent* archives — no zip64 spanning — but a
|
||||
//! photo and its JSON sidecar can land in different parts, and one album's
|
||||
//! folder can be spread over several. So the scan treats the whole set as one
|
||||
//! namespace keyed by the path inside the export:
|
||||
//!
|
||||
//! ```text
|
||||
//! Takeout/Google Photos/Iceland 2019/IMG_0042.jpg
|
||||
//! Takeout/Google Photos/Iceland 2019/IMG_0042.jpg.supplemental-metadata.json
|
||||
//! Takeout/Google Photos/Iceland 2019/metadata.json <- album title
|
||||
//! Takeout/Google Photos/Photos from 2019/IMG_0043.jpg <- no album
|
||||
//! Takeout/Google Photos/Trash/IMG_0044.jpg <- skipped
|
||||
//! ```
|
||||
//!
|
||||
//! The service folder (`Google Photos`) is localized, as are the year folders
|
||||
//! (`Photos from 2019`) and the trash folder, so nothing here matches on the
|
||||
//! service name: the first component is dropped when it is the export root, the
|
||||
//! second is the service, and the rest is `<folder>/<file>`. A folder is an
|
||||
//! **album** when it carries an album metadata sidecar naming it; anything else
|
||||
//! is a plain bucket that imports into the timeline only. Trash is matched by
|
||||
//! name against [`TRASH_FOLDERS`] — the one place a localized name is
|
||||
//! unavoidable, because a trashed photo is not marked as such anywhere else.
|
||||
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::io::Read;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::error::{CoreError, CoreResult};
|
||||
|
||||
/// Folder names Google uses for the trash bucket, lowercased. A Takeout export
|
||||
/// only contains one when the user asked for it, and its photos are deleted —
|
||||
/// importing them would resurrect them, so they are skipped.
|
||||
///
|
||||
/// Localized, and necessarily incomplete: a locale not listed here imports its
|
||||
/// trash as ordinary timeline photos. Extend as reports come in rather than
|
||||
/// guessing at transliterations.
|
||||
pub const TRASH_FOLDERS: &[&str] = &[
|
||||
"trash",
|
||||
"bin",
|
||||
"papierkorb",
|
||||
"corbeille",
|
||||
"papelera",
|
||||
"papelera de reciclaje",
|
||||
"cestino",
|
||||
"lixeira",
|
||||
"prullenbak",
|
||||
"kosz",
|
||||
"koš",
|
||||
"skräpkorgen",
|
||||
"papperskorg",
|
||||
"papirkurv",
|
||||
"roskakori",
|
||||
"kuka",
|
||||
"回收站",
|
||||
"回收筒",
|
||||
"ゴミ箱",
|
||||
"휴지통",
|
||||
"корзина",
|
||||
"кошик",
|
||||
"çöp kutusu",
|
||||
"σκουπίδια",
|
||||
];
|
||||
|
||||
/// Sidecar file names that describe the *folder* (its album title) rather than
|
||||
/// one photo, lowercased. Google has renamed this file more than once.
|
||||
const ALBUM_METADATA_NAMES: &[&str] = &[
|
||||
"metadata.json",
|
||||
"album-metadata.json",
|
||||
"print-subscriptions.json",
|
||||
"shared_album_comments.json",
|
||||
"user-generated-memory-titles.json",
|
||||
];
|
||||
|
||||
/// The suffix Google appends to a photo's sidecar, lowercased. Truncation means
|
||||
/// only a *prefix* of it may survive (see [`strip_sidecar_suffix`]).
|
||||
const SUPPLEMENTAL: &str = ".supplemental-metadata";
|
||||
|
||||
/// File extensions imported as photos or videos. Anything else in the export —
|
||||
/// `archive_browser.html`, `.csv` activity logs, `print-subscriptions.json` —
|
||||
/// is not media and is ignored.
|
||||
const MEDIA_EXTENSIONS: &[(&str, &str)] = &[
|
||||
("jpg", "image/jpeg"),
|
||||
("jpeg", "image/jpeg"),
|
||||
("png", "image/png"),
|
||||
("gif", "image/gif"),
|
||||
("webp", "image/webp"),
|
||||
("bmp", "image/bmp"),
|
||||
("tif", "image/tiff"),
|
||||
("tiff", "image/tiff"),
|
||||
("heic", "image/heic"),
|
||||
("heif", "image/heif"),
|
||||
("avif", "image/avif"),
|
||||
("dng", "image/x-adobe-dng"),
|
||||
("cr2", "image/x-canon-cr2"),
|
||||
("cr3", "image/x-canon-cr3"),
|
||||
("nef", "image/x-nikon-nef"),
|
||||
("arw", "image/x-sony-arw"),
|
||||
("orf", "image/x-olympus-orf"),
|
||||
("rw2", "image/x-panasonic-rw2"),
|
||||
("raf", "image/x-fuji-raf"),
|
||||
("mp4", "video/mp4"),
|
||||
("m4v", "video/x-m4v"),
|
||||
("mov", "video/quicktime"),
|
||||
("avi", "video/x-msvideo"),
|
||||
("mkv", "video/x-matroska"),
|
||||
("webm", "video/webm"),
|
||||
("3gp", "video/3gpp"),
|
||||
("mts", "video/mp2t"),
|
||||
("m2ts", "video/mp2t"),
|
||||
("mpg", "video/mpeg"),
|
||||
("mpeg", "video/mpeg"),
|
||||
];
|
||||
|
||||
/// The media type for `name`'s extension, or `None` when it is not media this
|
||||
/// import handles.
|
||||
pub fn media_type_of(name: &str) -> Option<&'static str> {
|
||||
let ext = Path::new(name)
|
||||
.extension()
|
||||
.and_then(|e| e.to_str())?
|
||||
.to_ascii_lowercase();
|
||||
MEDIA_EXTENSIONS
|
||||
.iter()
|
||||
.find(|(candidate, _)| *candidate == ext)
|
||||
.map(|(_, media_type)| *media_type)
|
||||
}
|
||||
|
||||
/// Where a photo sits in the export, which decides whether it joins an album.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub enum Bucket {
|
||||
/// A named album folder — the photo joins that album *and* the timeline.
|
||||
Album(String),
|
||||
/// A year folder (`Photos from 2019`) or the export root: timeline only.
|
||||
Timeline,
|
||||
/// The archive bucket: timeline only, like Google shows it.
|
||||
Archived,
|
||||
}
|
||||
|
||||
/// One media file found in the export.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct TakeoutPhoto {
|
||||
/// Index into the archive list the scan was given.
|
||||
pub archive: usize,
|
||||
/// The entry's full path inside that archive — how the importer reopens it.
|
||||
pub entry: String,
|
||||
/// The name to upload under: the sidecar's `title` when it has one (Google
|
||||
/// mangles long names on disk but keeps the original there), else the file
|
||||
/// name on disk.
|
||||
pub name: String,
|
||||
/// Uncompressed size in bytes, from the zip's central directory.
|
||||
pub size: u64,
|
||||
/// Media type from the extension.
|
||||
pub media_type: String,
|
||||
/// Capture time in epoch seconds, from `photoTakenTime` (falling back to
|
||||
/// `creationTime`). `None` when the photo has no sidecar — the upload then
|
||||
/// defaults it to now, as the SDK does.
|
||||
pub capture_time: Option<i64>,
|
||||
/// The sidecar's `favorited` flag.
|
||||
pub favorite: bool,
|
||||
/// Which bucket the file was found in.
|
||||
pub bucket: Bucket,
|
||||
}
|
||||
|
||||
/// Everything one scan found, ready to be turned into an import plan.
|
||||
#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
|
||||
pub struct TakeoutScan {
|
||||
/// Every importable media file, in archive then entry order.
|
||||
pub photos: Vec<TakeoutPhoto>,
|
||||
/// Album names in the order first seen, so the importer creates them in a
|
||||
/// stable order.
|
||||
pub albums: Vec<String>,
|
||||
/// Media files skipped because they sit in the trash bucket.
|
||||
pub skipped_trashed: usize,
|
||||
/// Entries that were neither media nor a sidecar (`archive_browser.html`,
|
||||
/// activity `.csv`s, `.mp` motion-photo parts).
|
||||
pub skipped_other: usize,
|
||||
}
|
||||
|
||||
impl TakeoutScan {
|
||||
/// Total bytes of media to be read, before duplicate detection.
|
||||
pub fn total_bytes(&self) -> u64 {
|
||||
self.photos.iter().map(|p| p.size).sum()
|
||||
}
|
||||
}
|
||||
|
||||
/// Google's per-photo sidecar. Only the fields the import uses are read; the
|
||||
/// file also carries geo data, people tags and view counts.
|
||||
#[derive(Debug, Default, Deserialize)]
|
||||
struct PhotoSidecar {
|
||||
#[serde(default)]
|
||||
title: Option<String>,
|
||||
#[serde(rename = "photoTakenTime", default)]
|
||||
photo_taken_time: Option<SidecarTime>,
|
||||
#[serde(rename = "creationTime", default)]
|
||||
creation_time: Option<SidecarTime>,
|
||||
#[serde(default)]
|
||||
favorited: bool,
|
||||
}
|
||||
|
||||
/// A sidecar timestamp: epoch seconds, as a *string*.
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct SidecarTime {
|
||||
#[serde(default)]
|
||||
timestamp: Option<String>,
|
||||
}
|
||||
|
||||
impl SidecarTime {
|
||||
fn epoch_seconds(&self) -> Option<i64> {
|
||||
self.timestamp.as_ref()?.parse().ok()
|
||||
}
|
||||
}
|
||||
|
||||
/// A folder-level sidecar. `title` is the album name as the user typed it,
|
||||
/// which is what the folder name is derived (and sometimes truncated) from.
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct AlbumSidecar {
|
||||
#[serde(default)]
|
||||
title: Option<String>,
|
||||
}
|
||||
|
||||
/// One entry as the scan sees it, independent of where it came from.
|
||||
struct RawEntry {
|
||||
archive: usize,
|
||||
entry: String,
|
||||
/// `<folder>/<file>` with the export root and service folder removed.
|
||||
relative: String,
|
||||
size: u64,
|
||||
}
|
||||
|
||||
impl RawEntry {
|
||||
/// The folder component of `relative`, or `""` at the service root.
|
||||
fn folder(&self) -> &str {
|
||||
match self.relative.rfind('/') {
|
||||
Some(index) => &self.relative[..index],
|
||||
None => "",
|
||||
}
|
||||
}
|
||||
|
||||
/// The file name component of `relative`.
|
||||
fn file_name(&self) -> &str {
|
||||
match self.relative.rfind('/') {
|
||||
Some(index) => &self.relative[index + 1..],
|
||||
None => &self.relative,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Scan a set of Takeout zip archives into one plan.
|
||||
///
|
||||
/// The archives are treated as one export: an album folder split across parts
|
||||
/// is one album, and a sidecar in part 3 is matched to its photo in part 1.
|
||||
/// Reads only the central directory and the JSON sidecars — no media bytes, so
|
||||
/// this stays fast on a 200 GB export.
|
||||
pub fn scan_archives(archives: &[PathBuf]) -> CoreResult<TakeoutScan> {
|
||||
let mut entries: Vec<RawEntry> = Vec::new();
|
||||
let mut sidecars: HashMap<String, Vec<u8>> = HashMap::new();
|
||||
|
||||
for (index, path) in archives.iter().enumerate() {
|
||||
let file = std::fs::File::open(path)
|
||||
.map_err(|e| CoreError::invalid(format!("cannot open {}: {e}", path.display())))?;
|
||||
let mut zip = zip::ZipArchive::new(std::io::BufReader::new(file)).map_err(|e| {
|
||||
CoreError::invalid(format!("{} is not a readable zip: {e}", path.display()))
|
||||
})?;
|
||||
|
||||
for position in 0..zip.len() {
|
||||
let mut entry = zip.by_index(position).map_err(|e| {
|
||||
CoreError::invalid(format!("{}: unreadable entry: {e}", path.display()))
|
||||
})?;
|
||||
if entry.is_dir() {
|
||||
continue;
|
||||
}
|
||||
// `enclosed_name` refuses paths that escape the archive root, which
|
||||
// is exactly the check we want before using one as a key.
|
||||
let Some(name) = entry.enclosed_name() else {
|
||||
continue;
|
||||
};
|
||||
let name = name.to_string_lossy().replace('\\', "/");
|
||||
let Some(relative) = strip_export_prefix(&name) else {
|
||||
continue;
|
||||
};
|
||||
let size = entry.size();
|
||||
|
||||
// Sidecars are read now, while the entry is open: they are a few
|
||||
// hundred bytes each and reopening the archive per photo later would
|
||||
// be a second full pass.
|
||||
if relative.to_ascii_lowercase().ends_with(".json") {
|
||||
let mut body = Vec::with_capacity(size.min(64 * 1024) as usize);
|
||||
if entry.read_to_end(&mut body).is_ok() {
|
||||
sidecars.insert(relative.to_string(), body);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
entries.push(RawEntry {
|
||||
archive: index,
|
||||
entry: name.clone(),
|
||||
relative: relative.to_string(),
|
||||
size,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
Ok(assemble(entries, &sidecars))
|
||||
}
|
||||
|
||||
/// Build the scan from raw entries plus the sidecar bodies keyed by their path
|
||||
/// inside the export. Split out from [`scan_archives`] so the matching rules can
|
||||
/// be tested without building zip fixtures.
|
||||
fn assemble(entries: Vec<RawEntry>, sidecars: &HashMap<String, Vec<u8>>) -> TakeoutScan {
|
||||
// Album titles per folder, from the folder-level sidecars.
|
||||
let mut album_titles: HashMap<String, String> = HashMap::new();
|
||||
for (path, body) in sidecars {
|
||||
let (folder, file) = split_folder(path);
|
||||
if !ALBUM_METADATA_NAMES.contains(&file.to_ascii_lowercase().as_str()) {
|
||||
continue;
|
||||
}
|
||||
if let Ok(album) = serde_json::from_slice::<AlbumSidecar>(body)
|
||||
&& let Some(title) = album.title.filter(|t| !t.trim().is_empty())
|
||||
{
|
||||
album_titles.insert(folder.to_string(), title);
|
||||
}
|
||||
}
|
||||
|
||||
// Per-photo sidecars, indexed by folder so a truncated name only ever
|
||||
// matches within its own folder.
|
||||
let mut per_folder_sidecars: HashMap<String, BTreeMap<String, &Vec<u8>>> = HashMap::new();
|
||||
for (path, body) in sidecars {
|
||||
let (folder, file) = split_folder(path);
|
||||
if ALBUM_METADATA_NAMES.contains(&file.to_ascii_lowercase().as_str()) {
|
||||
continue;
|
||||
}
|
||||
per_folder_sidecars
|
||||
.entry(folder.to_string())
|
||||
.or_default()
|
||||
.insert(file.to_string(), body);
|
||||
}
|
||||
|
||||
let mut scan = TakeoutScan::default();
|
||||
let mut seen_albums: HashMap<String, ()> = HashMap::new();
|
||||
|
||||
for raw in &entries {
|
||||
let folder = raw.folder().to_string();
|
||||
let file_name = raw.file_name().to_string();
|
||||
|
||||
if is_trash_folder(&folder) {
|
||||
if media_type_of(&file_name).is_some() {
|
||||
scan.skipped_trashed += 1;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
let Some(media_type) = media_type_of(&file_name) else {
|
||||
scan.skipped_other += 1;
|
||||
continue;
|
||||
};
|
||||
|
||||
let sidecar = per_folder_sidecars
|
||||
.get(&folder)
|
||||
.and_then(|in_folder| find_sidecar(&file_name, in_folder))
|
||||
.and_then(|body| serde_json::from_slice::<PhotoSidecar>(body).ok())
|
||||
.unwrap_or_default();
|
||||
|
||||
let bucket = match album_titles.get(&folder) {
|
||||
Some(title) => Bucket::Album(title.clone()),
|
||||
None if is_archive_folder(&folder) => Bucket::Archived,
|
||||
None => Bucket::Timeline,
|
||||
};
|
||||
if let Bucket::Album(title) = &bucket
|
||||
&& seen_albums.insert(title.clone(), ()).is_none()
|
||||
{
|
||||
scan.albums.push(title.clone());
|
||||
}
|
||||
|
||||
// The sidecar's `title` is the name Google had before it mangled the
|
||||
// on-disk one (truncation, `(1)` suffixes, character replacement), so it
|
||||
// is the better name to upload under — but only when it still looks like
|
||||
// the same file, i.e. the extension agrees.
|
||||
let name = sidecar
|
||||
.title
|
||||
.as_deref()
|
||||
.map(str::trim)
|
||||
.filter(|title| !title.is_empty() && same_extension(title, &file_name))
|
||||
.unwrap_or(&file_name)
|
||||
.to_string();
|
||||
|
||||
scan.photos.push(TakeoutPhoto {
|
||||
archive: raw.archive,
|
||||
entry: raw.entry.clone(),
|
||||
name,
|
||||
size: raw.size,
|
||||
media_type: media_type.to_string(),
|
||||
capture_time: sidecar
|
||||
.photo_taken_time
|
||||
.as_ref()
|
||||
.and_then(SidecarTime::epoch_seconds)
|
||||
.or_else(|| {
|
||||
sidecar
|
||||
.creation_time
|
||||
.as_ref()
|
||||
.and_then(SidecarTime::epoch_seconds)
|
||||
}),
|
||||
favorite: sidecar.favorited,
|
||||
bucket,
|
||||
});
|
||||
}
|
||||
|
||||
scan
|
||||
}
|
||||
|
||||
/// Drop the export root (`Takeout/`) and the localized service folder from a
|
||||
/// path inside the archive, leaving `<folder>/<file>` or `<file>`.
|
||||
///
|
||||
/// `None` for a path with no service folder at all (the export's own
|
||||
/// `archive_browser.html`), which carries nothing to import.
|
||||
fn strip_export_prefix(path: &str) -> Option<&str> {
|
||||
let mut rest = path;
|
||||
// The root is usually `Takeout/`, but an export unzipped and re-zipped by
|
||||
// the user may not have it — so the root is dropped only when a service
|
||||
// folder follows it, which the component count decides.
|
||||
let components = rest.split('/').count();
|
||||
if components >= 3 {
|
||||
rest = rest.split_once('/')?.1;
|
||||
}
|
||||
// Drop the service folder (`Google Photos`, `Google Fotos`, …).
|
||||
let (_service, tail) = rest.split_once('/')?;
|
||||
Some(tail)
|
||||
}
|
||||
|
||||
/// Split `<folder>/<file>` into its parts; the folder is `""` at the root.
|
||||
fn split_folder(path: &str) -> (&str, &str) {
|
||||
match path.rfind('/') {
|
||||
Some(index) => (&path[..index], &path[index + 1..]),
|
||||
None => ("", path),
|
||||
}
|
||||
}
|
||||
|
||||
fn is_trash_folder(folder: &str) -> bool {
|
||||
let name = folder.rsplit('/').next().unwrap_or(folder).to_lowercase();
|
||||
TRASH_FOLDERS.contains(&name.as_str())
|
||||
}
|
||||
|
||||
/// Google's archive bucket, which is not an album but is not the plain timeline
|
||||
/// either. Only the English name is matched; a missed locale simply imports as
|
||||
/// [`Bucket::Timeline`], which is the same upload either way.
|
||||
fn is_archive_folder(folder: &str) -> bool {
|
||||
let name = folder.rsplit('/').next().unwrap_or(folder).to_lowercase();
|
||||
name == "archive"
|
||||
}
|
||||
|
||||
/// Whether two file names end in the same extension, case-insensitively.
|
||||
fn same_extension(a: &str, b: &str) -> bool {
|
||||
let ext = |name: &str| {
|
||||
Path::new(name)
|
||||
.extension()
|
||||
.and_then(|e| e.to_str())
|
||||
.map(str::to_ascii_lowercase)
|
||||
};
|
||||
ext(a) == ext(b)
|
||||
}
|
||||
|
||||
/// Strip a sidecar's `.json` and any `.supplemental-metadata` suffix, including
|
||||
/// the *truncated* forms Google produces.
|
||||
///
|
||||
/// Google caps the sidecar's file name at 51 characters, so a long photo name
|
||||
/// leaves only a prefix of the suffix behind — `IMG_1234.jpg.supplemental-me`,
|
||||
/// `…-metad`, or nothing at all. Any non-empty prefix of the suffix is stripped;
|
||||
/// a bare `.json` sidecar loses only that.
|
||||
fn strip_sidecar_suffix(sidecar: &str) -> String {
|
||||
let stem = sidecar
|
||||
.strip_suffix(".json")
|
||||
.or_else(|| sidecar.strip_suffix(".JSON"))
|
||||
.unwrap_or(sidecar);
|
||||
// Longest match first so `.supplemental-metadata` wins over `.s`.
|
||||
for length in (2..=SUPPLEMENTAL.len()).rev() {
|
||||
let candidate = &SUPPLEMENTAL[..length];
|
||||
if let Some(head) = stem.strip_suffix(candidate) {
|
||||
return head.to_string();
|
||||
}
|
||||
}
|
||||
stem.to_string()
|
||||
}
|
||||
|
||||
/// Move a trailing `(N)` from after the extension to before it: Google writes
|
||||
/// the sidecar of `IMG_1234(1).jpg` as `IMG_1234.jpg(1).json`.
|
||||
fn unswap_duplicate_marker(stem: &str) -> Option<String> {
|
||||
let marker_start = stem.rfind('(')?;
|
||||
let marker = &stem[marker_start..];
|
||||
if !marker.ends_with(')')
|
||||
|| !marker[1..marker.len() - 1]
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_digit())
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let head = &stem[..marker_start];
|
||||
let dot = head.rfind('.')?;
|
||||
Some(format!("{}{}{}", &head[..dot], marker, &head[dot..]))
|
||||
}
|
||||
|
||||
/// Find the sidecar belonging to `file_name` among the JSON files of its folder.
|
||||
///
|
||||
/// Tried in order of confidence: the exact name, the `(N)`-swapped name, then a
|
||||
/// prefix match for the case where truncation cut into the photo name itself.
|
||||
fn find_sidecar<'a>(
|
||||
file_name: &str,
|
||||
in_folder: &BTreeMap<String, &'a Vec<u8>>,
|
||||
) -> Option<&'a Vec<u8>> {
|
||||
let mut prefix_match: Option<&'a Vec<u8>> = None;
|
||||
let mut prefix_len = 0usize;
|
||||
|
||||
for (sidecar_name, body) in in_folder {
|
||||
let stem = strip_sidecar_suffix(sidecar_name);
|
||||
if stem == file_name {
|
||||
return Some(body);
|
||||
}
|
||||
if unswap_duplicate_marker(&stem).as_deref() == Some(file_name) {
|
||||
return Some(body);
|
||||
}
|
||||
// Truncated: the stem is a prefix of the photo's name. Keep the longest
|
||||
// such prefix — a folder holding `IMG_1.jpg` and `IMG_12.jpg` must not
|
||||
// let the shorter stem claim the longer photo.
|
||||
if !stem.is_empty() && file_name.starts_with(&stem) && stem.len() > prefix_len {
|
||||
prefix_len = stem.len();
|
||||
prefix_match = Some(body);
|
||||
}
|
||||
}
|
||||
prefix_match
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn entry(archive: usize, path: &str, size: u64) -> RawEntry {
|
||||
RawEntry {
|
||||
archive,
|
||||
entry: path.to_string(),
|
||||
relative: strip_export_prefix(path).unwrap().to_string(),
|
||||
size,
|
||||
}
|
||||
}
|
||||
|
||||
fn sidecar(map: &mut HashMap<String, Vec<u8>>, path: &str, body: &str) {
|
||||
let relative = strip_export_prefix(path).unwrap().to_string();
|
||||
map.insert(relative, body.as_bytes().to_vec());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn strips_export_root_and_localized_service_folder() {
|
||||
assert_eq!(
|
||||
strip_export_prefix("Takeout/Google Fotos/Island 2019/IMG_1.jpg"),
|
||||
Some("Island 2019/IMG_1.jpg")
|
||||
);
|
||||
// Already unwrapped by the user: no `Takeout/` root left.
|
||||
assert_eq!(
|
||||
strip_export_prefix("Google Photos/IMG_1.jpg"),
|
||||
Some("IMG_1.jpg")
|
||||
);
|
||||
// Nothing below a service folder is nothing to import.
|
||||
assert_eq!(strip_export_prefix("archive_browser.html"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn album_folder_becomes_an_album_year_folder_does_not() {
|
||||
let mut sidecars = HashMap::new();
|
||||
sidecar(
|
||||
&mut sidecars,
|
||||
"Takeout/Google Photos/Iceland 2019/metadata.json",
|
||||
r#"{"title":"Iceland 2019"}"#,
|
||||
);
|
||||
let scan = assemble(
|
||||
vec![
|
||||
entry(0, "Takeout/Google Photos/Iceland 2019/IMG_1.jpg", 10),
|
||||
entry(0, "Takeout/Google Photos/Photos from 2019/IMG_2.jpg", 20),
|
||||
],
|
||||
&sidecars,
|
||||
);
|
||||
|
||||
assert_eq!(scan.albums, vec!["Iceland 2019"]);
|
||||
assert_eq!(scan.photos[0].bucket, Bucket::Album("Iceland 2019".into()));
|
||||
assert_eq!(scan.photos[1].bucket, Bucket::Timeline);
|
||||
assert_eq!(scan.total_bytes(), 30);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn trashed_photos_are_skipped_and_counted() {
|
||||
let scan = assemble(
|
||||
vec![
|
||||
entry(0, "Takeout/Google Photos/Papierkorb/IMG_1.jpg", 10),
|
||||
entry(0, "Takeout/Google Photos/Bin/IMG_2.jpg", 10),
|
||||
entry(0, "Takeout/Google Photos/Photos from 2019/IMG_3.jpg", 10),
|
||||
],
|
||||
&HashMap::new(),
|
||||
);
|
||||
|
||||
assert_eq!(scan.skipped_trashed, 2);
|
||||
assert_eq!(scan.photos.len(), 1);
|
||||
assert_eq!(scan.photos[0].name, "IMG_3.jpg");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn non_media_entries_are_ignored() {
|
||||
let scan = assemble(
|
||||
vec![
|
||||
entry(0, "Takeout/Google Photos/Photos from 2019/IMG_1.jpg", 10),
|
||||
entry(0, "Takeout/Google Photos/Photos from 2019/notes.txt", 10),
|
||||
],
|
||||
&HashMap::new(),
|
||||
);
|
||||
|
||||
assert_eq!(scan.photos.len(), 1);
|
||||
assert_eq!(scan.skipped_other, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sidecar_supplies_capture_time_and_favourite() {
|
||||
let mut sidecars = HashMap::new();
|
||||
sidecar(
|
||||
&mut sidecars,
|
||||
"Takeout/Google Photos/Photos from 2019/IMG_1.jpg.supplemental-metadata.json",
|
||||
r#"{"title":"IMG_1.jpg","photoTakenTime":{"timestamp":"1560000000"},"favorited":true}"#,
|
||||
);
|
||||
let scan = assemble(
|
||||
vec![entry(
|
||||
0,
|
||||
"Takeout/Google Photos/Photos from 2019/IMG_1.jpg",
|
||||
10,
|
||||
)],
|
||||
&sidecars,
|
||||
);
|
||||
|
||||
assert_eq!(scan.photos[0].capture_time, Some(1_560_000_000));
|
||||
assert!(scan.photos[0].favorite);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn creation_time_is_the_fallback_capture_time() {
|
||||
let mut sidecars = HashMap::new();
|
||||
sidecar(
|
||||
&mut sidecars,
|
||||
"Takeout/Google Photos/Photos from 2019/IMG_1.jpg.json",
|
||||
r#"{"creationTime":{"timestamp":"1500000000"}}"#,
|
||||
);
|
||||
let scan = assemble(
|
||||
vec![entry(
|
||||
0,
|
||||
"Takeout/Google Photos/Photos from 2019/IMG_1.jpg",
|
||||
10,
|
||||
)],
|
||||
&sidecars,
|
||||
);
|
||||
|
||||
assert_eq!(scan.photos[0].capture_time, Some(1_500_000_000));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn truncated_supplemental_suffixes_still_match() {
|
||||
// Google caps the sidecar name at 51 chars, cutting into the suffix.
|
||||
assert_eq!(
|
||||
strip_sidecar_suffix("IMG_1.jpg.supplemental-me.json"),
|
||||
"IMG_1.jpg"
|
||||
);
|
||||
assert_eq!(
|
||||
strip_sidecar_suffix("IMG_1.jpg.supplemental-metadata.json"),
|
||||
"IMG_1.jpg"
|
||||
);
|
||||
assert_eq!(strip_sidecar_suffix("IMG_1.jpg.json"), "IMG_1.jpg");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn duplicate_marker_after_the_extension_is_moved_back() {
|
||||
assert_eq!(
|
||||
unswap_duplicate_marker("IMG_1.jpg(1)").as_deref(),
|
||||
Some("IMG_1(1).jpg")
|
||||
);
|
||||
assert_eq!(unswap_duplicate_marker("IMG_1.jpg"), None);
|
||||
|
||||
let mut sidecars = HashMap::new();
|
||||
sidecar(
|
||||
&mut sidecars,
|
||||
"Takeout/Google Photos/Photos from 2019/IMG_1.jpg(1).json",
|
||||
r#"{"photoTakenTime":{"timestamp":"1600000000"}}"#,
|
||||
);
|
||||
let scan = assemble(
|
||||
vec![entry(
|
||||
0,
|
||||
"Takeout/Google Photos/Photos from 2019/IMG_1(1).jpg",
|
||||
10,
|
||||
)],
|
||||
&sidecars,
|
||||
);
|
||||
|
||||
assert_eq!(scan.photos[0].capture_time, Some(1_600_000_000));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn longest_prefix_wins_when_the_photo_name_itself_was_truncated() {
|
||||
let mut sidecars = HashMap::new();
|
||||
sidecar(
|
||||
&mut sidecars,
|
||||
"Takeout/Google Photos/Photos from 2019/IMG_1.json",
|
||||
r#"{"photoTakenTime":{"timestamp":"1"}}"#,
|
||||
);
|
||||
sidecar(
|
||||
&mut sidecars,
|
||||
"Takeout/Google Photos/Photos from 2019/IMG_12.json",
|
||||
r#"{"photoTakenTime":{"timestamp":"2"}}"#,
|
||||
);
|
||||
let scan = assemble(
|
||||
vec![entry(
|
||||
0,
|
||||
"Takeout/Google Photos/Photos from 2019/IMG_12345.jpg",
|
||||
10,
|
||||
)],
|
||||
&sidecars,
|
||||
);
|
||||
|
||||
assert_eq!(scan.photos[0].capture_time, Some(2));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sidecar_title_replaces_a_mangled_on_disk_name() {
|
||||
let mut sidecars = HashMap::new();
|
||||
sidecar(
|
||||
&mut sidecars,
|
||||
"Takeout/Google Photos/Photos from 2019/a_very_long_name_that_google_.jpg.json",
|
||||
r#"{"title":"a_very_long_name_that_google_truncated.jpg"}"#,
|
||||
);
|
||||
let scan = assemble(
|
||||
vec![entry(
|
||||
0,
|
||||
"Takeout/Google Photos/Photos from 2019/a_very_long_name_that_google_.jpg",
|
||||
10,
|
||||
)],
|
||||
&sidecars,
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
scan.photos[0].name,
|
||||
"a_very_long_name_that_google_truncated.jpg"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_title_with_a_different_extension_is_not_trusted() {
|
||||
// Motion photos name their `.MP` part after the `.jpg`; taking the title
|
||||
// verbatim would upload the video under an image name.
|
||||
let mut sidecars = HashMap::new();
|
||||
sidecar(
|
||||
&mut sidecars,
|
||||
"Takeout/Google Photos/Photos from 2019/IMG_1.mp4.json",
|
||||
r#"{"title":"IMG_1.jpg"}"#,
|
||||
);
|
||||
let scan = assemble(
|
||||
vec![entry(
|
||||
0,
|
||||
"Takeout/Google Photos/Photos from 2019/IMG_1.mp4",
|
||||
10,
|
||||
)],
|
||||
&sidecars,
|
||||
);
|
||||
|
||||
assert_eq!(scan.photos[0].name, "IMG_1.mp4");
|
||||
assert_eq!(scan.photos[0].media_type, "video/mp4");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_album_split_across_archives_stays_one_album() {
|
||||
let mut sidecars = HashMap::new();
|
||||
sidecar(
|
||||
&mut sidecars,
|
||||
"Takeout/Google Photos/Iceland 2019/metadata.json",
|
||||
r#"{"title":"Iceland 2019"}"#,
|
||||
);
|
||||
let scan = assemble(
|
||||
vec![
|
||||
entry(0, "Takeout/Google Photos/Iceland 2019/IMG_1.jpg", 10),
|
||||
entry(1, "Takeout/Google Photos/Iceland 2019/IMG_2.jpg", 10),
|
||||
],
|
||||
&sidecars,
|
||||
);
|
||||
|
||||
assert_eq!(scan.albums, vec!["Iceland 2019"]);
|
||||
assert_eq!(scan.photos.len(), 2);
|
||||
assert_eq!(scan.photos[1].archive, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn archive_bucket_imports_to_the_timeline() {
|
||||
let scan = assemble(
|
||||
vec![entry(0, "Takeout/Google Photos/Archive/IMG_1.jpg", 10)],
|
||||
&HashMap::new(),
|
||||
);
|
||||
|
||||
assert_eq!(scan.photos[0].bucket, Bucket::Archived);
|
||||
assert!(scan.albums.is_empty());
|
||||
}
|
||||
|
||||
/// Write a zip holding `files` and return its path. No `tempfile`
|
||||
/// dev-dependency: the daemon crates deliberately carry none, and one
|
||||
/// uniquely named file under the temp dir is all this needs.
|
||||
fn write_zip(tag: &str, files: &[(&str, &[u8])]) -> PathBuf {
|
||||
use std::io::Write;
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"pdfs-takeout-test-{tag}-{}.zip",
|
||||
std::process::id()
|
||||
));
|
||||
let file = std::fs::File::create(&path).unwrap();
|
||||
let mut zip = zip::ZipWriter::new(file);
|
||||
let options: zip::write::FileOptions<'_, ()> =
|
||||
zip::write::FileOptions::default().compression_method(zip::CompressionMethod::Stored);
|
||||
for (name, body) in files {
|
||||
zip.start_file(*name, options).unwrap();
|
||||
zip.write_all(body).unwrap();
|
||||
}
|
||||
zip.finish().unwrap();
|
||||
path
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn scans_a_two_part_export_where_the_sidecar_is_in_the_other_part() {
|
||||
// The case that makes the whole set one scan: part 1 has the photo,
|
||||
// part 2 has its metadata and the album title.
|
||||
let first = write_zip(
|
||||
"part1",
|
||||
&[(
|
||||
"Takeout/Google Photos/Iceland 2019/IMG_1.jpg",
|
||||
b"the photo bytes",
|
||||
)],
|
||||
);
|
||||
let second = write_zip(
|
||||
"part2",
|
||||
&[
|
||||
(
|
||||
"Takeout/Google Photos/Iceland 2019/metadata.json",
|
||||
br#"{"title":"Iceland 2019"}"#.as_slice(),
|
||||
),
|
||||
(
|
||||
"Takeout/Google Photos/Iceland 2019/IMG_1.jpg.supplemental-metadata.json",
|
||||
br#"{"title":"IMG_1.jpg","photoTakenTime":{"timestamp":"1560000000"}}"#
|
||||
.as_slice(),
|
||||
),
|
||||
(
|
||||
"Takeout/Google Photos/Trash/IMG_9.jpg",
|
||||
b"deleted".as_slice(),
|
||||
),
|
||||
],
|
||||
);
|
||||
|
||||
let scan = scan_archives(&[first.clone(), second.clone()]).unwrap();
|
||||
let _ = std::fs::remove_file(&first);
|
||||
let _ = std::fs::remove_file(&second);
|
||||
|
||||
assert_eq!(scan.photos.len(), 1);
|
||||
assert_eq!(scan.photos[0].name, "IMG_1.jpg");
|
||||
assert_eq!(scan.photos[0].archive, 0, "the photo is in part 1");
|
||||
assert_eq!(scan.photos[0].size, "the photo bytes".len() as u64);
|
||||
assert_eq!(scan.photos[0].capture_time, Some(1_560_000_000));
|
||||
assert_eq!(scan.photos[0].bucket, Bucket::Album("Iceland 2019".into()));
|
||||
assert_eq!(scan.albums, vec!["Iceland 2019"]);
|
||||
assert_eq!(scan.skipped_trashed, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_file_that_is_not_a_zip_is_a_readable_error() {
|
||||
let path =
|
||||
std::env::temp_dir().join(format!("pdfs-takeout-not-a-zip-{}", std::process::id()));
|
||||
std::fs::write(&path, b"not a zip at all").unwrap();
|
||||
let error = scan_archives(std::slice::from_ref(&path)).unwrap_err();
|
||||
let _ = std::fs::remove_file(&path);
|
||||
assert!(
|
||||
error.message.contains("not a readable zip"),
|
||||
"unexpected message: {}",
|
||||
error.message
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -5,10 +5,14 @@ edition.workspace = true
|
||||
license.workspace = true
|
||||
authors.workspace = true
|
||||
rust-version.workspace = true
|
||||
repository.workspace = true
|
||||
homepage.workspace = true
|
||||
description = "FUSE on-demand filesystem for Proton Drive"
|
||||
|
||||
[dependencies]
|
||||
pdfs-core.workspace = true
|
||||
zip.workspace = true
|
||||
sha1.workspace = true
|
||||
proton-drive-rs.workspace = true
|
||||
futures.workspace = true
|
||||
tokio.workspace = true
|
||||
@@ -16,6 +20,7 @@ tracing.workspace = true
|
||||
fuser.workspace = true
|
||||
libc.workspace = true
|
||||
serde_json.workspace = true
|
||||
base64.workspace = true
|
||||
image.workspace = true
|
||||
notify = "8"
|
||||
parking_lot.workspace = true
|
||||
@@ -1,6 +1,7 @@
|
||||
//! Remote-event invalidation and local-index background workers.
|
||||
|
||||
use super::*;
|
||||
use crate::shutdown::Shutdown;
|
||||
|
||||
/// Drop `uid` from one mount's tree and tell that mount's kernel session the
|
||||
/// entry is gone. Shared by the trash and delete paths, which differ only in
|
||||
@@ -437,6 +438,7 @@ pub(super) fn run_local_index(
|
||||
indexing: Arc<AtomicBool>,
|
||||
transfers: Arc<TransferRegistry>,
|
||||
mountpoint: PathBuf,
|
||||
shutdown: Arc<Shutdown>,
|
||||
) {
|
||||
loop {
|
||||
let age = db.local_indexed_at().ok().flatten();
|
||||
@@ -445,7 +447,11 @@ pub(super) fn run_local_index(
|
||||
if stale {
|
||||
scan_local_once(&db, &indexing, &transfers, &mountpoint);
|
||||
}
|
||||
std::thread::sleep(LOCAL_INDEX_CHECK);
|
||||
// Interruptible, so teardown joins this thread instead of leaving it to
|
||||
// keep rewriting the index for a mount that is gone (bugs.md B44).
|
||||
if !shutdown.sleep(LOCAL_INDEX_CHECK) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
|
||||
use std::io::{BufRead, BufReader, Write as _};
|
||||
use std::os::unix::net::{UnixListener, UnixStream};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::path::{Component, Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
@@ -104,17 +104,51 @@ fn read_request_line(reader: &mut impl BufRead) -> std::io::Result<Option<String
|
||||
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))
|
||||
}
|
||||
|
||||
/// Turn a CLI-supplied path into the mount that owns it and the path relative
|
||||
/// to *that* mount's root.
|
||||
///
|
||||
/// [`rel_to_mount`] is the older, primary-only reading, and it is what every
|
||||
/// mutating request still uses. It rejects anything outside the primary
|
||||
/// mountpoint, which leaves a secondary on-demand mount — a synced folder
|
||||
/// switched out of mirror mode — with no way to be named at all: `pdfs ls` and
|
||||
/// `pdfs refresh` both answered "is not under the mountpoint", so a folder
|
||||
/// serving a stale listing had no manual escape hatch (`docs/BUGS.md` B86).
|
||||
///
|
||||
/// A relative path keeps the old meaning, relative to the primary mount, since
|
||||
/// there is nothing in it to choose a mount by.
|
||||
fn route_to_mount(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
|
||||
let p = Path::new(path);
|
||||
if !p.is_absolute() {
|
||||
return Ok((core.clone(), p.to_path_buf()));
|
||||
}
|
||||
if let Some(routed) = core.rooted_at(p) {
|
||||
return Ok(routed);
|
||||
}
|
||||
// Fall back to the primary reading so the error text stays the familiar one
|
||||
// when the path is simply outside everything this daemon has mounted.
|
||||
rel_to_mount(mountpoint, path).map(|rel| (core.clone(), rel))
|
||||
}
|
||||
|
||||
/// Turn a CLI-supplied path into a mountpoint-relative path. An absolute path
|
||||
/// must live under `mountpoint`; a relative path is taken as already relative to
|
||||
/// the mount root.
|
||||
fn rel_to_mount(mountpoint: &Path, path: &str) -> CoreResult<PathBuf> {
|
||||
let p = Path::new(path);
|
||||
if p.is_absolute() {
|
||||
let relative = if p.is_absolute() {
|
||||
p.strip_prefix(mountpoint)
|
||||
.map(Path::to_path_buf)
|
||||
.map_err(|_| CoreError::invalid(format!("{path} is not under the mountpoint")))
|
||||
} else {
|
||||
Ok(p.to_path_buf())
|
||||
}?;
|
||||
|
||||
if relative
|
||||
.components()
|
||||
.all(|component| matches!(component, Component::Normal(_) | Component::CurDir))
|
||||
{
|
||||
Ok(relative)
|
||||
} else {
|
||||
Err(CoreError::invalid(format!("{path} escapes the mountpoint")))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -193,8 +227,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
Ok(CtlRequest::ListPins) => CtlResponse::Pins {
|
||||
pins: core.cache.list_pins(),
|
||||
},
|
||||
Ok(CtlRequest::ListDir { path }) => match rel_to_mount(mountpoint, &path) {
|
||||
Ok(rel) => match core.list_dir(&rel) {
|
||||
Ok(CtlRequest::ListDir { path }) => match route_to_mount(core, mountpoint, &path) {
|
||||
Ok((core, rel)) => match core.list_dir(&rel) {
|
||||
Ok(entries) => CtlResponse::Entries { entries },
|
||||
Err(e) => CtlResponse::error(e),
|
||||
},
|
||||
@@ -202,9 +236,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
},
|
||||
Ok(CtlRequest::Refresh { scope }) => {
|
||||
let result = match &scope {
|
||||
RefreshScope::Dir { path } => {
|
||||
rel_to_mount(mountpoint, path).and_then(|rel| core.refresh_dir(&rel))
|
||||
}
|
||||
RefreshScope::Dir { path } => route_to_mount(core, mountpoint, path)
|
||||
.and_then(|(core, rel)| core.refresh_dir(&rel)),
|
||||
RefreshScope::Trash => {
|
||||
core.invalidate_trash();
|
||||
Ok(())
|
||||
@@ -296,6 +329,34 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
items: core.photo_thumbs(&parsed),
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::FileThumbs { items, generation }) => {
|
||||
match core.file_thumbs(&items, generation) {
|
||||
Some(items) => CtlResponse::Thumbs { items },
|
||||
None => CtlResponse::FileThumbsStale,
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::ReserveFileThumbGeneration) => CtlResponse::FileThumbGeneration {
|
||||
generation: core.reserve_file_thumb_generation(),
|
||||
},
|
||||
Ok(CtlRequest::CancelFileThumbs { generation }) => {
|
||||
core.cancel_file_thumbs(generation);
|
||||
CtlResponse::Ok {
|
||||
message: "thumbnail requests cancelled".to_string(),
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::StartThumbnailBuild { path }) => match rel_to_mount(mountpoint, &path) {
|
||||
Ok(rel) => match core.start_thumbnail_build(rel) {
|
||||
Ok(status) => CtlResponse::ThumbnailBuild { status },
|
||||
Err(error) => CtlResponse::error(error),
|
||||
},
|
||||
Err(error) => CtlResponse::error(error),
|
||||
},
|
||||
Ok(CtlRequest::CancelThumbnailBuild) => CtlResponse::ThumbnailBuild {
|
||||
status: core.cancel_thumbnail_build(),
|
||||
},
|
||||
Ok(CtlRequest::ThumbnailBuildStatus) => CtlResponse::ThumbnailBuild {
|
||||
status: core.thumbnail_build_status(),
|
||||
},
|
||||
Ok(CtlRequest::OpenPhoto { uid }) => match parse_uid(&uid) {
|
||||
Some(u) => match core.open_photo(&u) {
|
||||
Ok(p) => CtlResponse::FilePath {
|
||||
@@ -305,6 +366,51 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
},
|
||||
None => CtlResponse::error(CoreError::invalid(format!("bad uid: {uid}"))),
|
||||
},
|
||||
// The scan alone walks every archive's central directory, so even the
|
||||
// dry run goes to a background thread and is read back through
|
||||
// `ImportStatus` rather than answered inline.
|
||||
Ok(CtlRequest::ImportTakeout { archives, dry_run }) => {
|
||||
if archives.is_empty() {
|
||||
CtlResponse::error(CoreError::invalid("no Takeout archives given"))
|
||||
} else {
|
||||
let paths: Vec<PathBuf> = archives.into_iter().map(PathBuf::from).collect();
|
||||
let missing: Vec<String> = paths
|
||||
.iter()
|
||||
.filter(|path| !path.is_file())
|
||||
.map(|path| path.display().to_string())
|
||||
.collect();
|
||||
if !missing.is_empty() {
|
||||
CtlResponse::error(CoreError::not_found(format!(
|
||||
"cannot read: {}",
|
||||
missing.join(", ")
|
||||
)))
|
||||
} else {
|
||||
let count = paths.len();
|
||||
match core.spawn_takeout_import(paths, dry_run) {
|
||||
Ok(()) => CtlResponse::Ok {
|
||||
message: format!(
|
||||
"{} {count} archive(s)",
|
||||
if dry_run { "scanning" } else { "importing" }
|
||||
),
|
||||
},
|
||||
Err(e) => CtlResponse::error(e),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::ImportStatus) => {
|
||||
let (running, summary) = crate::takeout::import_status();
|
||||
CtlResponse::ImportStatus { running, summary }
|
||||
}
|
||||
Ok(CtlRequest::CancelImport) => {
|
||||
if crate::takeout::cancel_import() {
|
||||
CtlResponse::Ok {
|
||||
message: "stopping the import".to_string(),
|
||||
}
|
||||
} else {
|
||||
CtlResponse::error(CoreError::invalid("no import is running"))
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::UploadPhoto {
|
||||
name,
|
||||
media_type,
|
||||
@@ -1084,6 +1190,12 @@ pub(crate) fn run_control_socket(
|
||||
) {
|
||||
info!("control socket listening");
|
||||
for conn in listener.incoming() {
|
||||
// Checked on every accept, including the deliberate poke teardown sends
|
||||
// to wake this thread out of a blocking accept (bugs.md B44).
|
||||
if core.shutdown.is_stopping() {
|
||||
info!("control socket closing");
|
||||
return;
|
||||
}
|
||||
match conn {
|
||||
Ok(stream) => {
|
||||
let Some(permit) = ControlHandlerPermit::acquire() else {
|
||||
@@ -1146,4 +1258,29 @@ mod request_limit_tests {
|
||||
std::io::ErrorKind::InvalidData
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accepts_paths_inside_the_mountpoint() {
|
||||
let mountpoint = Path::new("/mnt/proton");
|
||||
assert_eq!(
|
||||
rel_to_mount(mountpoint, "Photos/2026").unwrap(),
|
||||
Path::new("Photos/2026")
|
||||
);
|
||||
assert_eq!(
|
||||
rel_to_mount(mountpoint, "/mnt/proton/Photos/2026").unwrap(),
|
||||
Path::new("Photos/2026")
|
||||
);
|
||||
assert_eq!(
|
||||
rel_to_mount(mountpoint, "/mnt/proton").unwrap(),
|
||||
Path::new("")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_paths_that_escape_the_mountpoint() {
|
||||
let mountpoint = Path::new("/mnt/proton");
|
||||
assert!(rel_to_mount(mountpoint, "../escape").is_err());
|
||||
assert!(rel_to_mount(mountpoint, "Photos/../../escape").is_err());
|
||||
assert!(rel_to_mount(mountpoint, "/mnt/elsewhere").is_err());
|
||||
}
|
||||
}
|
||||
+640
-57
@@ -20,10 +20,14 @@
|
||||
//! are kept as a conflict copy rather than discarded — see
|
||||
//! [`Core::revision_conflict`].
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::fs::File;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::Ordering;
|
||||
use std::time::Duration;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use parking_lot::Mutex;
|
||||
|
||||
use pdfs_core::batch;
|
||||
use pdfs_core::cache::{Baseline, StagedWrite};
|
||||
@@ -36,17 +40,132 @@ use tracing::{debug, error, info, warn};
|
||||
use super::state::Intervals;
|
||||
use super::transfers::CountingReader;
|
||||
use super::{
|
||||
Core, DRAIN_BACKOFF_MAX, DRAIN_BACKOFF_MIN, DRAIN_IDLE_POLL, ROOT_INO, conflict_name,
|
||||
Core, DRAIN_BACKOFF_MAX, DRAIN_BACKOFF_MIN, DRAIN_IDLE_POLL, DRAIN_REVISION_DEBOUNCE,
|
||||
DRAIN_REVISION_DEBOUNCE_MAX, ROOT_INO, UPLOAD_TIME_MEMORY, WriteAuthority, conflict_name,
|
||||
is_already_exists, is_gone, is_local_uid_str, media_type_for, node_revision_id, node_size,
|
||||
now_millis, now_secs, parse_node_uid,
|
||||
};
|
||||
|
||||
const DRAIN_ACCESS_RECHECK: Duration = Duration::from_secs(5);
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
/// How long an op may stay access-deferred before the deferral is reported as a
|
||||
/// failure instead of retried in silence.
|
||||
///
|
||||
/// An access deferral costs no attempt and records no error, which is right for
|
||||
/// the case it was written for: a share downgraded while the queue was moving,
|
||||
/// undone a moment later. It is wrong for anything that does not clear. Until
|
||||
/// this window existed such an op was re-deferred every
|
||||
/// [`DRAIN_ACCESS_RECHECK`] indefinitely with `attempts` at zero, `last_error`
|
||||
/// null and only a `debug!` line to show for it, so `pdfs status` counted it
|
||||
/// among ordinary queued uploads and nothing ever said otherwise.
|
||||
///
|
||||
/// The condition that produced that in practice — a node absent from the local
|
||||
/// tree, which [`Db::effective_node_access`] used to report indistinguishably
|
||||
/// from a revoked permission — is now answered directly by
|
||||
/// [`Core::resolve_unknown_authority`]. This window remains the backstop for
|
||||
/// every deferral that does not resolve, including a refetch that keeps not
|
||||
/// helping.
|
||||
///
|
||||
/// Five minutes is far longer than a permission change takes to settle and far
|
||||
/// shorter than a user's patience with bytes that are not moving. Past it each
|
||||
/// recheck records a failure, so the op enters the ordinary backoff and, at
|
||||
/// [`FAILING_ATTEMPTS`], is counted and named by `Response::Status`.
|
||||
///
|
||||
/// [`Db::effective_node_access`]: pdfs_core::db::Db::effective_node_access
|
||||
/// [`FAILING_ATTEMPTS`]: pdfs_core::db::FAILING_ATTEMPTS
|
||||
const DRAIN_ACCESS_DEFER_LIMIT: Duration = Duration::from_secs(300);
|
||||
|
||||
/// A registered cancellation flag for one in-flight upload, deregistered when
|
||||
/// dropped. See [`Core::begin_cancellable_upload`].
|
||||
struct UploadCancel {
|
||||
registry: Arc<Mutex<HashMap<NodeUid, Arc<AtomicBool>>>>,
|
||||
uid: NodeUid,
|
||||
flag: Arc<AtomicBool>,
|
||||
}
|
||||
|
||||
impl UploadCancel {
|
||||
/// Whether this upload was told to stop. Distinguishes an upload we
|
||||
/// abandoned on purpose from one the network broke.
|
||||
fn cancelled(&self) -> bool {
|
||||
self.flag.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
fn flag(&self) -> Arc<AtomicBool> {
|
||||
self.flag.clone()
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for UploadCancel {
|
||||
fn drop(&mut self) {
|
||||
let mut registry = self.registry.lock();
|
||||
// By identity: a supersede racing this upload's last bytes may already
|
||||
// have registered the next one under the same uid.
|
||||
if registry
|
||||
.get(&self.uid)
|
||||
.is_some_and(|current| Arc::ptr_eq(current, &self.flag))
|
||||
{
|
||||
registry.remove(&self.uid);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The debounce arithmetic of [`Core::revision_debounce`], without the map:
|
||||
/// the last measured upload time for this node, bounded on both sides.
|
||||
///
|
||||
/// A node nobody has uploaded yet gets the fixed grace period, which is the
|
||||
/// same answer the fixed debounce always gave.
|
||||
fn adaptive_debounce(measured: Option<Duration>) -> Duration {
|
||||
measured
|
||||
.unwrap_or(DRAIN_REVISION_DEBOUNCE)
|
||||
.clamp(DRAIN_REVISION_DEBOUNCE, DRAIN_REVISION_DEBOUNCE_MAX)
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
enum DrainDisposition {
|
||||
Applied,
|
||||
AccessDeferred,
|
||||
/// An authority this op needs is missing from the local tree, so no access
|
||||
/// answer exists for it yet. Carries the uid to ask the remote about.
|
||||
AuthorityUnknown(NodeUid),
|
||||
}
|
||||
|
||||
/// Why an op went back on the clock without consuming an attempt. Only affects
|
||||
/// what the user is told — every variant is deferred and reported identically.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum AccessDeferral {
|
||||
/// An authority the tree knows about refuses the write.
|
||||
NotWritable,
|
||||
/// An authority missing from the tree was just re-read from the remote and
|
||||
/// re-interned; the access check gets another look on the next pass.
|
||||
AuthorityRefetched,
|
||||
/// An authority missing from the tree could not be asked about.
|
||||
RemoteUnreachable,
|
||||
}
|
||||
|
||||
impl AccessDeferral {
|
||||
fn log_reason(self) -> &'static str {
|
||||
match self {
|
||||
Self::NotWritable => "the node is not writable",
|
||||
Self::AuthorityRefetched => "the node was missing locally and has been re-read",
|
||||
Self::RemoteUnreachable => "the node is missing locally and the remote is unreachable",
|
||||
}
|
||||
}
|
||||
|
||||
/// Phrased to complete "<reason> for 300s." in a recorded `last_error`, so
|
||||
/// the text the user sees names the actual condition rather than the
|
||||
/// permission question it used to be flattened into.
|
||||
fn failure_reason(self) -> &'static str {
|
||||
match self {
|
||||
Self::NotWritable => "the share has not been writable",
|
||||
Self::AuthorityRefetched => {
|
||||
"the node has been missing from the local tree, and re-reading it from the remote \
|
||||
has not restored it"
|
||||
}
|
||||
Self::RemoteUnreachable => {
|
||||
"the node has been missing from the local tree and the remote has been unreachable"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn pending_op_authorities(op: &PendingOp) -> Result<Vec<NodeUid>, Box<dyn std::error::Error>> {
|
||||
@@ -89,7 +208,7 @@ fn pending_op_authorities(op: &PendingOp) -> Result<Vec<NodeUid>, Box<dyn std::e
|
||||
|
||||
fn run_authorized_drain(
|
||||
op: &PendingOp,
|
||||
mut require: impl FnMut(&NodeUid) -> Result<(), fuser::Errno>,
|
||||
mut authority_of: impl FnMut(&NodeUid) -> WriteAuthority,
|
||||
apply: impl FnOnce() -> Result<(), Box<dyn std::error::Error>>,
|
||||
) -> Result<DrainDisposition, Box<dyn std::error::Error>> {
|
||||
// This check is the admission/linearization point. A downgrade that lands
|
||||
@@ -97,15 +216,14 @@ fn run_authorized_drain(
|
||||
// attempt. Every later queued operation rechecks. No permission lock is
|
||||
// held across remote latency.
|
||||
for authority in pending_op_authorities(op)? {
|
||||
match require(&authority) {
|
||||
Ok(()) => {}
|
||||
Err(error) if error.code() == libc::EACCES => {
|
||||
return Ok(DrainDisposition::AccessDeferred);
|
||||
}
|
||||
Err(error) => {
|
||||
return Err(
|
||||
format!("drain access check failed with errno {}", error.code()).into(),
|
||||
);
|
||||
match authority_of(&authority) {
|
||||
WriteAuthority::Writable => {}
|
||||
WriteAuthority::Denied => return Ok(DrainDisposition::AccessDeferred),
|
||||
// Nothing in the tree can answer for this node. Deferring on it
|
||||
// waits for a permission change that has no reason to come, so the
|
||||
// caller asks the remote instead (B83).
|
||||
WriteAuthority::Unknown => {
|
||||
return Ok(DrainDisposition::AuthorityUnknown(authority));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -127,48 +245,85 @@ impl Core {
|
||||
/// that is due — one file wedged against a folder that no longer exists must
|
||||
/// not hold up an unrelated upload behind it. The worker only blocks once
|
||||
/// nothing is due at all.
|
||||
pub(crate) fn run_pending_drain(&self) {
|
||||
///
|
||||
/// Several of these run at once ([`DRAIN_WORKERS`]), sharing the queue
|
||||
/// through [`Db::claim_next_due_op`]: the alternative was that a 10 GiB
|
||||
/// upload held every queued rename, trash and small write behind it for as
|
||||
/// long as it took. Ordering only has to hold *per node*, and the claim
|
||||
/// query is what guarantees it — no two workers are ever on one uid.
|
||||
///
|
||||
/// `primary` marks the one worker that also runs the queue's idle chores.
|
||||
/// They are cheap, but they are not per-op work and doing them in every
|
||||
/// worker would just multiply them.
|
||||
///
|
||||
/// [`DRAIN_WORKERS`]: super::DRAIN_WORKERS
|
||||
/// [`Db::claim_next_due_op`]: pdfs_core::db::Db::claim_next_due_op
|
||||
pub(crate) fn run_pending_drain(&self, primary: bool) {
|
||||
loop {
|
||||
// Between ops, never inside one: an op that has started is either
|
||||
// retired or released, so stopping here cannot leave the queue with
|
||||
// a row claimed by a worker that no longer exists (bugs.md B44).
|
||||
if self.shutdown.is_stopping() {
|
||||
debug!(primary, "drain worker stopping");
|
||||
return;
|
||||
}
|
||||
let now = now_millis();
|
||||
// One row, chosen by the database. Reading the whole queue to pick
|
||||
// one op made a long queue quadratic to drain, and held the shared
|
||||
// connection — and so every FUSE metadata call — for the duration.
|
||||
let due = self.db.next_due_op(now).unwrap_or_default();
|
||||
|
||||
let Some(op) = due.filter(|_| self.online.load(Ordering::Relaxed)) else {
|
||||
// Nothing to upload is exactly when the connection is free, so
|
||||
// this is where the read path's buffered LRU touches get written
|
||||
// (`ContentCache::flush_touches`). Cheap and a no-op when empty.
|
||||
self.cache.flush_touches();
|
||||
self.recover_fsynced_writes();
|
||||
// A debounced or backed-off op may be waiting: sleep only
|
||||
// until it becomes due rather than the full idle-poll.
|
||||
self.wait_for_drain_work_or_due();
|
||||
continue;
|
||||
// One row, chosen and *claimed* by the database. Reading the whole
|
||||
// queue to pick one op made a long queue quadratic to drain, and
|
||||
// held the shared connection — and so every FUSE metadata call —
|
||||
// for the duration.
|
||||
let due = match self.db.claim_next_due_op(now) {
|
||||
Ok(due) => due,
|
||||
Err(error) => {
|
||||
error!(%error, "claiming the next pending operation failed");
|
||||
self.wait_for_drain_work();
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let online = self.online.load(Ordering::Relaxed);
|
||||
let op = match due {
|
||||
Some(op) if online => op,
|
||||
idle => {
|
||||
// Claimed but offline: hand it straight back, or it stays
|
||||
// invisible to every worker until this one happens to be
|
||||
// the next to notice the reconnect.
|
||||
if let Some(op) = idle {
|
||||
self.release_claim(&op);
|
||||
}
|
||||
if primary {
|
||||
// Nothing to upload is exactly when the connection is
|
||||
// free, so this is where the read path's buffered LRU
|
||||
// touches get written (`ContentCache::flush_touches`).
|
||||
// Cheap and a no-op when empty.
|
||||
self.cache.flush_touches();
|
||||
self.recover_fsynced_writes();
|
||||
}
|
||||
// A debounced or backed-off op may be waiting: sleep only
|
||||
// until it becomes due rather than the full idle-poll.
|
||||
self.wait_for_drain_work_or_due();
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let outcome = run_authorized_drain(
|
||||
&op,
|
||||
|uid| self.require_uid_writable(uid),
|
||||
|uid| self.uid_write_authority(uid),
|
||||
|| self.drain_op(&op),
|
||||
);
|
||||
if matches!(outcome, Ok(DrainDisposition::AccessDeferred)) {
|
||||
let next_attempt_at = now_millis() + DRAIN_ACCESS_RECHECK.as_millis() as i64;
|
||||
if let Err(error) = self.db.defer_op_without_attempt(op.id, next_attempt_at) {
|
||||
error!(
|
||||
uid = %op.uid,
|
||||
%error,
|
||||
"deferring access-blocked pending operation failed"
|
||||
);
|
||||
self.wait_for_drain_work();
|
||||
} else {
|
||||
debug!(
|
||||
uid = %op.uid,
|
||||
next_attempt_at,
|
||||
"pending operation deferred until access is writable"
|
||||
);
|
||||
}
|
||||
self.defer_for_access(&op, AccessDeferral::NotWritable);
|
||||
continue;
|
||||
}
|
||||
if let Ok(DrainDisposition::AuthorityUnknown(ref authority)) = outcome {
|
||||
self.resolve_unknown_authority(&op, authority);
|
||||
continue;
|
||||
}
|
||||
// The access check let this attempt through, so any earlier run of
|
||||
// deferrals is over and must not be counted against the next one.
|
||||
if matches!(outcome, Ok(DrainDisposition::Applied))
|
||||
&& let Err(error) = self.db.clear_op_access_deferral(op.id)
|
||||
{
|
||||
debug!(uid = %op.uid, %error, "clearing an access-deferral window failed");
|
||||
}
|
||||
if let Err(e) = outcome {
|
||||
let attempts = op.attempts + 1;
|
||||
let err_str = e.to_string();
|
||||
@@ -188,9 +343,234 @@ impl Core {
|
||||
now_millis() + backoff.as_millis() as i64,
|
||||
) {
|
||||
error!(uid = %op.uid, error = %e, "recording a drain failure failed");
|
||||
self.release_claim(&op);
|
||||
self.wait_for_drain_work();
|
||||
continue;
|
||||
}
|
||||
}
|
||||
// Unconditionally, on every path: a handler that retired its own row
|
||||
// leaves nothing to release, and one that returned without retiring
|
||||
// must not leave the row claimed by a worker that has moved on.
|
||||
self.release_claim(&op);
|
||||
}
|
||||
}
|
||||
|
||||
/// Put an op the access check refused back on the clock, and decide whether
|
||||
/// its refusal is still news.
|
||||
///
|
||||
/// Inside [`DRAIN_ACCESS_DEFER_LIMIT`] a deferral is what it always was: no
|
||||
/// attempt consumed, no error recorded, retried in five seconds. Past it the
|
||||
/// deferral is reported through [`Db::record_op_failure`] like any other
|
||||
/// stuck op, which is what puts it in `attempts`, in `last_error`, and — at
|
||||
/// [`FAILING_ATTEMPTS`] — in the `failing_ops` count and error text that
|
||||
/// `pdfs status` and the GUI show. The row and its staged blob stay exactly
|
||||
/// where they are either way; this changes only whether the user can see
|
||||
/// them.
|
||||
///
|
||||
/// [`Db::record_op_failure`]: pdfs_core::db::Db::record_op_failure
|
||||
/// [`FAILING_ATTEMPTS`]: pdfs_core::db::FAILING_ATTEMPTS
|
||||
fn defer_for_access(&self, op: &PendingOp, reason: AccessDeferral) {
|
||||
let now = now_millis();
|
||||
let next_attempt_at = now + DRAIN_ACCESS_RECHECK.as_millis() as i64;
|
||||
let since = match self.db.defer_op_for_access(op.id, now, next_attempt_at) {
|
||||
Ok(since) => since,
|
||||
Err(error) => {
|
||||
error!(
|
||||
uid = %op.uid,
|
||||
%error,
|
||||
"deferring access-blocked pending operation failed"
|
||||
);
|
||||
self.release_claim(op);
|
||||
self.wait_for_drain_work();
|
||||
return;
|
||||
}
|
||||
};
|
||||
let blocked = Duration::from_millis(now.saturating_sub(since).max(0) as u64);
|
||||
if since == now {
|
||||
// The first deferral of a run, at `warn!`: a write the mount accepted
|
||||
// is not going anywhere for now, and that is worth a line even when
|
||||
// the next recheck clears it.
|
||||
warn!(
|
||||
uid = %op.uid,
|
||||
kind = %op.kind,
|
||||
reason = reason.log_reason(),
|
||||
"pending operation deferred"
|
||||
);
|
||||
}
|
||||
if blocked >= DRAIN_ACCESS_DEFER_LIMIT {
|
||||
let attempts = op.attempts + 1;
|
||||
let backoff = DRAIN_BACKOFF_MIN
|
||||
.saturating_mul(1u32 << attempts.min(6))
|
||||
.min(DRAIN_BACKOFF_MAX);
|
||||
let error = format!(
|
||||
"{} for {}s. The staged bytes are kept.",
|
||||
reason.failure_reason(),
|
||||
blocked.as_secs()
|
||||
);
|
||||
warn!(uid = %op.uid, attempts, blocked_secs = blocked.as_secs(),
|
||||
"pending operation has been access-blocked past the limit; reporting it");
|
||||
if let Err(error) =
|
||||
self.db
|
||||
.record_op_failure(op.id, &error, now_millis() + backoff.as_millis() as i64)
|
||||
{
|
||||
error!(uid = %op.uid, %error, "recording an access-deferral failure failed");
|
||||
}
|
||||
} else {
|
||||
debug!(
|
||||
uid = %op.uid,
|
||||
next_attempt_at,
|
||||
blocked_secs = blocked.as_secs(),
|
||||
reason = reason.log_reason(),
|
||||
"pending operation deferred until access is writable"
|
||||
);
|
||||
}
|
||||
self.release_claim(op);
|
||||
}
|
||||
|
||||
/// Answer an authority the local tree has no row for by asking the remote,
|
||||
/// rather than deferring on a permission question nobody asked (B83).
|
||||
///
|
||||
/// Three outcomes, and the point of the whole exercise is that they are
|
||||
/// three:
|
||||
///
|
||||
/// * the node is still there — the tree simply lost it (a pruned subtree, a
|
||||
/// mount whose registration went away). Re-intern it and the next pass has
|
||||
/// a real access answer to work with.
|
||||
/// * the node is gone remotely. That is a permanent, *known* condition, so
|
||||
/// it is recorded as a failure immediately instead of waiting out
|
||||
/// [`DRAIN_ACCESS_DEFER_LIMIT`] to say so. The row and its staged blob are
|
||||
/// kept, as on every other failure path — the bytes are still the user's,
|
||||
/// and `pdfs recover` is how they get them back.
|
||||
/// * we could not ask. Ordinary deferral; the network is the network.
|
||||
fn resolve_unknown_authority(&self, op: &PendingOp, authority: &NodeUid) {
|
||||
match self.fetch_node_remote(authority) {
|
||||
Ok(Some(node)) => {
|
||||
if let Err(error) = self.db.upsert_node(&node) {
|
||||
error!(uid = %op.uid, %authority, %error,
|
||||
"re-interning a queued operation's missing authority failed");
|
||||
} else {
|
||||
info!(uid = %op.uid, %authority,
|
||||
"re-interned an authority missing from the local tree; retrying");
|
||||
}
|
||||
// Deferred, not applied: the re-read has to go through the same
|
||||
// access check as everything else, on the next pass. The window
|
||||
// keeps running, so a refetch that never actually helps still
|
||||
// gets reported instead of looping in silence.
|
||||
self.defer_for_access(op, AccessDeferral::AuthorityRefetched);
|
||||
}
|
||||
Ok(None) => {
|
||||
let attempts = op.attempts + 1;
|
||||
let backoff = DRAIN_BACKOFF_MIN
|
||||
.saturating_mul(1u32 << attempts.min(6))
|
||||
.min(DRAIN_BACKOFF_MAX);
|
||||
let error = format!(
|
||||
"{authority} no longer exists remotely, so this operation has nothing to \
|
||||
apply to. The staged bytes are kept."
|
||||
);
|
||||
warn!(uid = %op.uid, %authority, attempts,
|
||||
"a queued operation's authority is gone remotely; reporting it");
|
||||
if let Err(error) = self.db.record_op_failure(
|
||||
op.id,
|
||||
&error,
|
||||
now_millis() + backoff.as_millis() as i64,
|
||||
) {
|
||||
error!(uid = %op.uid, %error, "recording a missing-authority failure failed");
|
||||
}
|
||||
self.release_claim(op);
|
||||
}
|
||||
Err(error) => {
|
||||
debug!(uid = %op.uid, %authority, %error,
|
||||
"could not ask the remote about a missing authority");
|
||||
self.defer_for_access(op, AccessDeferral::RemoteUnreachable);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Hand a claimed op back to the queue, logging rather than propagating —
|
||||
/// there is no caller left to propagate to, and the drain must not stop.
|
||||
fn release_claim(&self, op: &PendingOp) {
|
||||
if let Err(error) = self.db.release_op_claim(op.id) {
|
||||
error!(uid = %op.uid, %error, "releasing a drain claim failed");
|
||||
}
|
||||
}
|
||||
|
||||
/// How long a freshly queued revision of `uid` should wait before a worker
|
||||
/// may pick it up.
|
||||
///
|
||||
/// [`DRAIN_REVISION_DEBOUNCE`] exists for a tool that preallocates a file
|
||||
/// and then writes it, where two seconds is the right scale. It is the wrong
|
||||
/// scale for a large file on a slow link: an editor saving a 200 MB project
|
||||
/// every ten seconds queues a write, the drain starts sending it, the next
|
||||
/// save supersedes it mid-flight, and the file never finishes uploading
|
||||
/// while the user keeps working. Waiting roughly as long as the last upload
|
||||
/// of *this* node took moves that supersede into the queue, where replacing
|
||||
/// a row costs nothing.
|
||||
///
|
||||
/// Never shorter than the fixed debounce and never longer than
|
||||
/// [`DRAIN_REVISION_DEBOUNCE_MAX`]: a node whose upload is genuinely slow
|
||||
/// still has to reach Drive, and a single pathological stall must not park
|
||||
/// it.
|
||||
///
|
||||
/// [`DRAIN_REVISION_DEBOUNCE`]: super::DRAIN_REVISION_DEBOUNCE
|
||||
/// [`DRAIN_REVISION_DEBOUNCE_MAX`]: super::DRAIN_REVISION_DEBOUNCE_MAX
|
||||
pub(crate) fn revision_debounce(&self, uid: &NodeUid) -> Duration {
|
||||
let measured = self
|
||||
.upload_times
|
||||
.lock()
|
||||
.get(&uid.to_string())
|
||||
.map(|&(ms, _)| Duration::from_millis(ms));
|
||||
adaptive_debounce(measured)
|
||||
}
|
||||
|
||||
/// Remember how long an upload of `uid` took, for the next write's
|
||||
/// debounce.
|
||||
///
|
||||
/// Only a *completed* upload is a measurement. A cancelled or failed one
|
||||
/// says how long it ran before it stopped, which is not how long the file
|
||||
/// takes to send, and feeding that back would let one network fault widen
|
||||
/// the debounce for good.
|
||||
fn record_upload_time(&self, uid: &NodeUid, took: Duration) {
|
||||
let mut times = self.upload_times.lock();
|
||||
let now = now_millis();
|
||||
if times.len() >= UPLOAD_TIME_MEMORY
|
||||
&& !times.contains_key(&uid.to_string())
|
||||
&& let Some(oldest) = times
|
||||
.iter()
|
||||
.min_by_key(|(_, (_, at))| *at)
|
||||
.map(|(k, _)| k.clone())
|
||||
{
|
||||
times.remove(&oldest);
|
||||
}
|
||||
times.insert(uid.to_string(), (took.as_millis() as u64, now));
|
||||
}
|
||||
|
||||
/// Register a cancellation flag for an upload of `uid` that is about to
|
||||
/// start, returning a guard that deregisters it.
|
||||
///
|
||||
/// Deregistration is by identity rather than by key: a supersede that
|
||||
/// arrives just as this upload finishes may already have registered the
|
||||
/// *next* upload's flag under the same uid, and removing that one would
|
||||
/// leave the newer upload uncancellable.
|
||||
fn begin_cancellable_upload(&self, uid: &NodeUid) -> UploadCancel {
|
||||
let flag = Arc::new(AtomicBool::new(false));
|
||||
self.upload_cancel.lock().insert(uid.clone(), flag.clone());
|
||||
UploadCancel {
|
||||
registry: self.upload_cancel.clone(),
|
||||
uid: uid.clone(),
|
||||
flag,
|
||||
}
|
||||
}
|
||||
|
||||
/// Tell an upload of `uid` that is on the wire to stop, if there is one.
|
||||
///
|
||||
/// Called when a newer revision of the file is queued. The upload does not
|
||||
/// stop instantly — it stops when the SDK next asks its reader for bytes,
|
||||
/// which for a 4 MiB block is soon enough to matter and late enough to be
|
||||
/// safe: the staged blob it is reading stays open until it lets go.
|
||||
pub(crate) fn cancel_upload(&self, uid: &NodeUid) {
|
||||
if let Some(flag) = self.upload_cancel.lock().get(uid) {
|
||||
flag.store(true, Ordering::Relaxed);
|
||||
debug!(%uid, "cancelling a superseded upload");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -907,15 +1287,37 @@ impl Core {
|
||||
let guard =
|
||||
self.transfers
|
||||
.begin(&name, meta.uid.clone(), TransferDirection::Upload, meta.len);
|
||||
let reader = CountingReader::new(File::open(&blob)?, &guard);
|
||||
self.rt.block_on(self.client.upload_new_revision_from(
|
||||
// Registered before the first byte moves, so a write that lands while
|
||||
// this is on the wire can stop it rather than paying for a revision the
|
||||
// op it queued will immediately replace.
|
||||
let cancel = self.begin_cancellable_upload(&uid);
|
||||
let reader = CountingReader::new(File::open(&blob)?, &guard).with_cancel(cancel.flag());
|
||||
let started = Instant::now();
|
||||
let sent = self.rt.block_on(self.client.upload_new_revision_from(
|
||||
&uid,
|
||||
reader,
|
||||
meta.len as i64,
|
||||
Vec::new(),
|
||||
None,
|
||||
))?;
|
||||
));
|
||||
let took = started.elapsed();
|
||||
if let Err(e) = sent {
|
||||
if cancel.cancelled() {
|
||||
// Not a failure, and not this op's row to retire: the write that
|
||||
// cancelled us deleted it and took ownership of the staged blob
|
||||
// when it superseded this revision. Leaving the blob alone here
|
||||
// is the whole point — it is either already unlinked or it
|
||||
// belongs to the newer op.
|
||||
info!(%uid, "upload abandoned; a newer write superseded it");
|
||||
return Ok(());
|
||||
}
|
||||
return Err(e.into());
|
||||
}
|
||||
drop(cancel);
|
||||
drop(guard);
|
||||
// Only a completed upload times how long this file takes to send, which
|
||||
// is what the next write's debounce is scaled from.
|
||||
self.record_upload_time(&uid, took);
|
||||
|
||||
// Retire the op before dropping the blob: a crash between the two leaves
|
||||
// an orphaned file (harmless), whereas the reverse would leave a queued
|
||||
@@ -1227,12 +1629,10 @@ mod tests {
|
||||
let applied = Cell::new(false);
|
||||
let disposition = run_authorized_drain(
|
||||
&op,
|
||||
|uid| {
|
||||
db.effective_node_access(uid)
|
||||
.unwrap()
|
||||
.filter(|access| access.writable())
|
||||
.map(|_| ())
|
||||
.ok_or(fuser::Errno::EACCES)
|
||||
|uid| match db.effective_node_access(uid).unwrap() {
|
||||
Some(access) if access.writable() => WriteAuthority::Writable,
|
||||
Some(_) => WriteAuthority::Denied,
|
||||
None => WriteAuthority::Unknown,
|
||||
},
|
||||
|| {
|
||||
applied.set(true);
|
||||
@@ -1272,7 +1672,7 @@ mod tests {
|
||||
|
||||
let disposition = run_authorized_drain(
|
||||
&op,
|
||||
|_| Err(fuser::Errno::EACCES),
|
||||
|_| WriteAuthority::Denied,
|
||||
|| {
|
||||
applied.set(true);
|
||||
Ok(())
|
||||
@@ -1293,13 +1693,157 @@ mod tests {
|
||||
let _ = std::fs::remove_dir_all(dir);
|
||||
}
|
||||
|
||||
/// A deferral that never clears has to stop being silent. Found in the field
|
||||
/// as revision ops whose node had left the local tree: `require_uid_access`
|
||||
/// reads an unknown node as `EACCES`, so they were re-deferred every five
|
||||
/// seconds for thirty days with `attempts` at zero, `last_error` null and
|
||||
/// `failing_ops` at zero — 18 GiB of accepted writes that `pdfs status`
|
||||
/// reported as ordinary queued uploads.
|
||||
#[test]
|
||||
fn an_access_deferral_that_outlives_the_limit_is_reported_as_failing() {
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
static N: AtomicU64 = AtomicU64::new(0);
|
||||
let dir = std::env::temp_dir().join(format!(
|
||||
"pdfs-drain-defer-limit-{}-{}",
|
||||
std::process::id(),
|
||||
N.fetch_add(1, Ordering::Relaxed)
|
||||
));
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let db = Db::open(&dir.join("cache.db")).unwrap();
|
||||
db.enqueue_op(&pending(OP_REVISION)).unwrap();
|
||||
let op = db.pending_ops().unwrap().remove(0);
|
||||
|
||||
// Two deferrals a recheck apart: the window opens on the first and the
|
||||
// second is measured against it, not against itself.
|
||||
let now = 1_000_000i64;
|
||||
let since = db.defer_op_for_access(op.id, now, now + 5_000).unwrap();
|
||||
assert_eq!(since, now, "the first deferral opens the window");
|
||||
let later = now + DRAIN_ACCESS_RECHECK.as_millis() as i64;
|
||||
assert_eq!(
|
||||
db.defer_op_for_access(op.id, later, later + 5_000).unwrap(),
|
||||
now,
|
||||
"a later deferral keeps the original stamp"
|
||||
);
|
||||
let queued = db.pending_ops().unwrap();
|
||||
assert_eq!(queued[0].attempts, 0, "deferrals consume no attempt");
|
||||
assert!(queued[0].last_error.is_none());
|
||||
|
||||
// Past the limit the same deferral is reported like any other stuck op.
|
||||
let past = now + DRAIN_ACCESS_DEFER_LIMIT.as_millis() as i64;
|
||||
assert_eq!(
|
||||
db.defer_op_for_access(op.id, past, past + 5_000).unwrap(),
|
||||
now
|
||||
);
|
||||
db.record_op_failure(op.id, "not writable locally", past + 10_000)
|
||||
.unwrap();
|
||||
let failed = db.pending_ops().unwrap();
|
||||
assert_eq!(failed.len(), 1, "the row and its staged bytes stay queued");
|
||||
assert_eq!(failed[0].attempts, 1);
|
||||
assert_eq!(
|
||||
failed[0].last_error.as_deref(),
|
||||
Some("not writable locally")
|
||||
);
|
||||
|
||||
// Reporting closes the window, so the next deferral starts a fresh one
|
||||
// rather than escalating on every recheck from here on.
|
||||
let after = past + 20_000;
|
||||
assert_eq!(
|
||||
db.defer_op_for_access(op.id, after, after + 5_000).unwrap(),
|
||||
after,
|
||||
"recording the failure re-armed the window"
|
||||
);
|
||||
drop(db);
|
||||
let _ = std::fs::remove_dir_all(dir);
|
||||
}
|
||||
|
||||
/// An op that gets through the access check is no longer blocked, so a
|
||||
/// deferral after it must not inherit the earlier run's age and escalate
|
||||
/// immediately.
|
||||
#[test]
|
||||
fn clearing_the_window_makes_the_next_deferral_a_fresh_run() {
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
static N: AtomicU64 = AtomicU64::new(0);
|
||||
let dir = std::env::temp_dir().join(format!(
|
||||
"pdfs-drain-defer-clear-{}-{}",
|
||||
std::process::id(),
|
||||
N.fetch_add(1, Ordering::Relaxed)
|
||||
));
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let db = Db::open(&dir.join("cache.db")).unwrap();
|
||||
db.enqueue_op(&pending(OP_REVISION)).unwrap();
|
||||
let op = db.pending_ops().unwrap().remove(0);
|
||||
|
||||
let now = 2_000_000i64;
|
||||
assert_eq!(
|
||||
db.defer_op_for_access(op.id, now, now + 5_000).unwrap(),
|
||||
now
|
||||
);
|
||||
db.clear_op_access_deferral(op.id).unwrap();
|
||||
let later = now + DRAIN_ACCESS_DEFER_LIMIT.as_millis() as i64;
|
||||
assert_eq!(
|
||||
db.defer_op_for_access(op.id, later, later + 5_000).unwrap(),
|
||||
later,
|
||||
"the window reopens at the new deferral, not the old one"
|
||||
);
|
||||
drop(db);
|
||||
let _ = std::fs::remove_dir_all(dir);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_authority_the_tree_does_not_have_is_reported_as_unknown_not_deferred() {
|
||||
// The B83 distinction, at the point it is made: a uid with no row must
|
||||
// not come back as `AccessDeferred`, because deferring waits on a
|
||||
// permission change that nothing is going to make. The caller answers
|
||||
// an `AuthorityUnknown` by asking the remote instead.
|
||||
let op = pending(OP_REVISION);
|
||||
let applied = Cell::new(false);
|
||||
let disposition = run_authorized_drain(
|
||||
&op,
|
||||
|_| WriteAuthority::Unknown,
|
||||
|| {
|
||||
applied.set(true);
|
||||
Ok(())
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
disposition,
|
||||
DrainDisposition::AuthorityUnknown(parse_node_uid(&op.uid).unwrap())
|
||||
);
|
||||
assert!(!applied.get(), "an unresolved authority never applies");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_parent_authority_names_the_parent_not_the_node() {
|
||||
// A create's authority is its parent, so that is the uid the remote has
|
||||
// to be asked about — reporting the node instead would refetch the
|
||||
// thing that does not exist yet.
|
||||
let mut op = pending(OP_CREATE);
|
||||
let parent = folder_node("vanished-parent", None);
|
||||
op.parent_uid = Some(parent.uid.to_string());
|
||||
let disposition =
|
||||
run_authorized_drain(&op, |_| WriteAuthority::Unknown, || Ok(())).unwrap();
|
||||
assert_eq!(
|
||||
disposition,
|
||||
DrainDisposition::AuthorityUnknown(parent.uid.clone())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn owner_and_editor_drain_authorities_apply_normally() {
|
||||
for access in [Access::Owner, Access::Editor] {
|
||||
let applied = Cell::new(false);
|
||||
let disposition = run_authorized_drain(
|
||||
&pending(OP_REVISION),
|
||||
|_| access.writable().then_some(()).ok_or(fuser::Errno::EACCES),
|
||||
|_| {
|
||||
if access.writable() {
|
||||
WriteAuthority::Writable
|
||||
} else {
|
||||
WriteAuthority::Denied
|
||||
}
|
||||
},
|
||||
|| {
|
||||
applied.set(true);
|
||||
Ok(())
|
||||
@@ -1413,3 +1957,42 @@ mod tests {
|
||||
assert!(!is_own_self_supersede(true, None, None));
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod debounce_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_node_never_uploaded_keeps_the_fixed_grace_period() {
|
||||
assert_eq!(adaptive_debounce(None), DRAIN_REVISION_DEBOUNCE);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_fast_upload_does_not_shorten_the_grace_period() {
|
||||
// The 2 seconds are not about upload time at all — they are the window
|
||||
// in which a preallocate-then-write tool supersedes its own first
|
||||
// close. A file that sends in 50 ms must still get them.
|
||||
assert_eq!(
|
||||
adaptive_debounce(Some(Duration::from_millis(50))),
|
||||
DRAIN_REVISION_DEBOUNCE
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_slow_upload_widens_the_grace_period_to_match() {
|
||||
// The case the fixed debounce got wrong: saves arriving faster than the
|
||||
// upload completes, each superseding one already on the wire.
|
||||
assert_eq!(
|
||||
adaptive_debounce(Some(Duration::from_secs(25))),
|
||||
Duration::from_secs(25)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_pathological_upload_cannot_park_a_node() {
|
||||
assert_eq!(
|
||||
adaptive_debounce(Some(Duration::from_secs(6000))),
|
||||
DRAIN_REVISION_DEBOUNCE_MAX
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
//! Kernel-facing FUSE callback adapter.
|
||||
|
||||
use super::*;
|
||||
use crate::state::DirListing;
|
||||
|
||||
impl Filesystem for ProtonFs {
|
||||
/// Negotiate kernel-side behaviour once, before any other callback.
|
||||
@@ -162,15 +163,46 @@ impl Filesystem for ProtonFs {
|
||||
}
|
||||
}
|
||||
|
||||
/// Freeze this directory's listing for the life of the handle.
|
||||
///
|
||||
/// `readdir` is paged, and its continuation offset only means anything
|
||||
/// against a listing that does not move underneath it. Taking the snapshot
|
||||
/// here — once per `opendir`, which the kernel pairs exactly with one
|
||||
/// `releasedir` — is what makes the offsets stable, so a create or a trash
|
||||
/// during an enumeration can no longer make the caller skip or repeat an
|
||||
/// entry (bugs.md B43).
|
||||
///
|
||||
/// Enumerating a cold folder is a remote call, so it goes to a worker for
|
||||
/// the same reason `lookup` and `readdir` do.
|
||||
fn opendir(&self, _req: &Request, ino: INodeNo, _flags: OpenFlags, reply: ReplyOpen) {
|
||||
let ino = ino.0;
|
||||
if self.core.children_cached(ino) {
|
||||
self.serve_opendir(ino, reply);
|
||||
return;
|
||||
}
|
||||
let fs = self.clone();
|
||||
self.core
|
||||
.workers
|
||||
.run(Lane::Meta, move || fs.serve_opendir(ino, reply));
|
||||
}
|
||||
|
||||
fn readdir(
|
||||
&self,
|
||||
_req: &Request,
|
||||
ino: INodeNo,
|
||||
_fh: FileHandle,
|
||||
fh: FileHandle,
|
||||
offset: u64,
|
||||
reply: ReplyDirectory,
|
||||
) {
|
||||
let ino = ino.0;
|
||||
// The snapshot is the whole point: served from it, this is a pure
|
||||
// in-memory walk with no chance of a remote call, so it never needs a
|
||||
// worker. Only a handle whose `opendir` failed to snapshot (or a kernel
|
||||
// that issued `readdir` without one) falls back to the live path.
|
||||
if let Some(listing) = self.core.state().dir_snapshots.get(&fh.0).cloned() {
|
||||
reply_listing(&listing, offset, reply);
|
||||
return;
|
||||
}
|
||||
// Same split as `lookup`: only the cold, remote-enumerating path pays
|
||||
// for a worker.
|
||||
if self.core.children_cached(ino) {
|
||||
@@ -183,6 +215,18 @@ impl Filesystem for ProtonFs {
|
||||
.run(Lane::Meta, move || fs.serve_readdir(ino, offset, reply));
|
||||
}
|
||||
|
||||
fn releasedir(
|
||||
&self,
|
||||
_req: &Request,
|
||||
_ino: INodeNo,
|
||||
fh: FileHandle,
|
||||
_flags: OpenFlags,
|
||||
reply: ReplyEmpty,
|
||||
) {
|
||||
self.core.state().dir_snapshots.remove(&fh.0);
|
||||
reply.ok();
|
||||
}
|
||||
|
||||
/// A read-only open is a map lookup and answers inline; a write open has to
|
||||
/// allocate a scratch file and, when a queued revision is the base, duplicate
|
||||
/// a blob that is the whole file — so it is served from a worker rather than
|
||||
@@ -912,6 +956,58 @@ pub struct ProtonFs {
|
||||
gid: u32,
|
||||
}
|
||||
|
||||
/// How many times a write `open` will re-read its base after a `release`
|
||||
/// published a newer queued revision underneath it (bugs.md B32).
|
||||
///
|
||||
/// One retry covers the race, which needs a close and an open of the same file
|
||||
/// to overlap on different threads. More than that is not contention, it is a
|
||||
/// file being republished continuously, and looping on it would keep a FUSE
|
||||
/// worker busy indefinitely.
|
||||
const OPEN_BASE_ATTEMPTS: u32 = 3;
|
||||
|
||||
/// One directory's entries as `readdir` reports them: `.` and `..` first, then
|
||||
/// the interned children in `children` order.
|
||||
///
|
||||
/// Split out because it is built from two places that must agree — `opendir`
|
||||
/// freezes it, and the fallback `readdir` builds it live — and a divergence
|
||||
/// between them would show up as entries appearing or vanishing depending on
|
||||
/// whether the kernel happened to send an `opendir`.
|
||||
fn build_listing(st: &State, ino: u64) -> DirListing {
|
||||
let parent = st.entries.get(&ino).map_or(ROOT_INO, |e| e.parent);
|
||||
// "." and ".." occupy offsets 0 and 1; real children follow.
|
||||
let mut listing = vec![
|
||||
(ino, true, ".".to_string()),
|
||||
(parent, true, "..".to_string()),
|
||||
];
|
||||
if let Some(kids) = st.children.get(&ino) {
|
||||
for &kid in kids {
|
||||
if let Some(e) = st.entries.get(&kid) {
|
||||
listing.push((kid, e.node.is_folder(), e.node.name.clone()));
|
||||
}
|
||||
}
|
||||
}
|
||||
listing
|
||||
}
|
||||
|
||||
/// Fill a `readdir` reply from `listing`, resuming at `offset`.
|
||||
///
|
||||
/// The cookie handed back is the index of the *next* entry, so a resumed call
|
||||
/// continues exactly where the last one stopped — which is only true because
|
||||
/// the listing behind those indexes is a snapshot (bugs.md B43).
|
||||
fn reply_listing(listing: &DirListing, offset: u64, mut reply: ReplyDirectory) {
|
||||
for (i, (ino, is_dir, name)) in listing.iter().enumerate().skip(offset as usize) {
|
||||
let ft = if *is_dir {
|
||||
FileType::Directory
|
||||
} else {
|
||||
FileType::RegularFile
|
||||
};
|
||||
if reply.add(INodeNo(*ino), (i + 1) as u64, ft, name) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
reply.ok();
|
||||
}
|
||||
|
||||
pub(crate) fn access_allowed(is_dir: bool, access: Access, mask: AccessFlags) -> bool {
|
||||
(!mask.contains(AccessFlags::W_OK) || access.writable())
|
||||
&& (!mask.contains(AccessFlags::X_OK) || is_dir)
|
||||
@@ -1027,40 +1123,39 @@ impl ProtonFs {
|
||||
}
|
||||
|
||||
/// The body of [`Filesystem::readdir`], on whichever thread ends up serving it.
|
||||
fn serve_readdir(&self, ino: u64, offset: u64, mut reply: ReplyDirectory) {
|
||||
/// The body of [`Filesystem::opendir`]: enumerate the folder if it is cold,
|
||||
/// then publish one frozen listing under a fresh handle.
|
||||
///
|
||||
/// A failure to enumerate is reported as a failed `opendir` rather than
|
||||
/// deferred to the first `readdir` — the caller gets the errno at the syscall
|
||||
/// that can still act on it.
|
||||
fn serve_opendir(&self, ino: u64, reply: ReplyOpen) {
|
||||
if let Err(e) = self.core.ensure_children(ino) {
|
||||
reply.error(e);
|
||||
return;
|
||||
}
|
||||
let mut st = self.core.state();
|
||||
let listing = Arc::new(build_listing(&st, ino));
|
||||
// Shares the counter with write handles, so a directory handle can never
|
||||
// collide with a file handle (both are `fh` values in one kernel space).
|
||||
let fh = st.next_fh;
|
||||
st.next_fh += 1;
|
||||
st.dir_snapshots.insert(fh, listing);
|
||||
drop(st);
|
||||
reply.opened(FileHandle(fh), FopenFlags::empty());
|
||||
}
|
||||
|
||||
/// The live, un-snapshotted `readdir`. Reached only by a handle that has no
|
||||
/// snapshot — see [`Filesystem::readdir`].
|
||||
fn serve_readdir(&self, ino: u64, offset: u64, reply: ReplyDirectory) {
|
||||
if let Err(e) = self.core.ensure_children(ino) {
|
||||
reply.error(e);
|
||||
return;
|
||||
}
|
||||
let st = self.core.state();
|
||||
let parent = st.entries.get(&ino).map_or(ROOT_INO, |e| e.parent);
|
||||
|
||||
// "." and ".." occupy offsets 0 and 1; real children follow.
|
||||
let mut listing: Vec<(u64, FileType, String)> = vec![
|
||||
(ino, FileType::Directory, ".".to_string()),
|
||||
(parent, FileType::Directory, "..".to_string()),
|
||||
];
|
||||
if let Some(kids) = st.children.get(&ino) {
|
||||
for &kid in kids {
|
||||
if let Some(e) = st.entries.get(&kid) {
|
||||
let ft = if e.node.is_folder() {
|
||||
FileType::Directory
|
||||
} else {
|
||||
FileType::RegularFile
|
||||
};
|
||||
listing.push((kid, ft, e.node.name.clone()));
|
||||
}
|
||||
}
|
||||
}
|
||||
let listing = build_listing(&st, ino);
|
||||
drop(st);
|
||||
|
||||
for (i, (ino, ft, name)) in listing.into_iter().enumerate().skip(offset as usize) {
|
||||
// The stored offset is that of the *next* entry to resume at.
|
||||
if reply.add(INodeNo(ino), (i + 1) as u64, ft, &name) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
reply.ok();
|
||||
reply_listing(&listing, offset, reply);
|
||||
}
|
||||
|
||||
fn attr(&self, ino: u64, node: &Node, access: Access) -> FileAttr {
|
||||
@@ -1142,66 +1237,101 @@ impl ProtonFs {
|
||||
// any network access. An incomplete blob still has gaps referring to
|
||||
// the remote base; stacking another write on it cannot be represented
|
||||
// safely by WriteHandle, so fail the open rather than risk corruption.
|
||||
let pending_base = self.core.pending.lock().get(&uid).cloned();
|
||||
if pending_base.as_ref().is_some_and(|p| !p.meta.complete) {
|
||||
if let Some(entry) = self.core.state().entries.get_mut(&ino.0) {
|
||||
entry.open_count = entry.open_count.saturating_sub(1);
|
||||
}
|
||||
error!(%uid, "refusing write over incomplete queued revision");
|
||||
reply.error(Errno::EIO);
|
||||
return;
|
||||
}
|
||||
let (file, path) = match self.core.cache.create_scratch() {
|
||||
Ok(x) => x,
|
||||
Err(e) => {
|
||||
if let Some(entry) = self.core.state().entries.get_mut(&ino.0) {
|
||||
entry.open_count = entry.open_count.saturating_sub(1);
|
||||
//
|
||||
// Reading that blob and installing the handle cannot be done under one
|
||||
// lock — the copy is the size of the file — so the base is re-checked at
|
||||
// the moment the handle goes in, and a `release` that published a newer
|
||||
// revision in between sends this around again. Without the re-check the
|
||||
// new handle carried the *older* base, and its own release published
|
||||
// content that silently discarded the revision closed a moment earlier
|
||||
// (bugs.md B32). A second attempt is already unlikely and a third would
|
||||
// mean something is republishing continuously, so give up rather than
|
||||
// spin: the caller's `open` fails instead of its data being lost.
|
||||
//
|
||||
// `reply` is consumed by whichever arm answers, so the loop yields an
|
||||
// outcome and the answer is sent once, after it.
|
||||
let outcome = 'install: {
|
||||
for attempt in 0..OPEN_BASE_ATTEMPTS {
|
||||
let pending_base = self.core.pending.lock().get(&uid).cloned();
|
||||
if pending_base.as_ref().is_some_and(|p| !p.meta.complete) {
|
||||
error!(%uid, "refusing write over incomplete queued revision");
|
||||
break 'install Err(Errno::EIO);
|
||||
}
|
||||
error!(%uid, error = %e, "create scratch file failed");
|
||||
reply.error(Errno::EIO);
|
||||
return;
|
||||
let (file, path) = match self.core.cache.create_scratch() {
|
||||
Ok(x) => x,
|
||||
Err(e) => {
|
||||
error!(%uid, error = %e, "create scratch file failed");
|
||||
break 'install Err(Errno::EIO);
|
||||
}
|
||||
};
|
||||
let mut initial_written = Intervals::default();
|
||||
if let Some(pending) = &pending_base {
|
||||
// Reflink where the filesystem can: this blob is the whole
|
||||
// file, and a byte-for-byte copy of a multi-GB pending
|
||||
// revision is the kind of work `open` must not do
|
||||
// (`ContentCache::clone_or_copy`).
|
||||
if let Err(e) = ContentCache::clone_or_copy(&pending.path, &path) {
|
||||
let _ = std::fs::remove_file(&path);
|
||||
error!(%uid, source = %pending.path.display(), error = %e,
|
||||
"copy queued revision into write scratch failed");
|
||||
break 'install Err(Errno::EIO);
|
||||
}
|
||||
initial_written.add(0, pending.meta.len);
|
||||
}
|
||||
// Sampled *before* the state lock, never under it: no site in
|
||||
// the daemon holds `pending` and `state` at once (see
|
||||
// `stamp_pending_sizes`) and this must not become the first.
|
||||
// What that costs is a window of a few instructions instead of
|
||||
// a lock-tight comparison — against the multi-gigabyte copy
|
||||
// above, which is the window that actually loses data.
|
||||
let current = self.core.pending_blob(&uid);
|
||||
let mut st = self.core.state();
|
||||
// Only a handle this open is about to *create* can carry a stale
|
||||
// base; if one already exists, that handle is authoritative and
|
||||
// the scratch prepared above is discarded either way.
|
||||
let creating = !st.active_writes.contains_key(&ino.0);
|
||||
if creating && current != pending_base.map(|p| p.path) {
|
||||
drop(st);
|
||||
let _ = std::fs::remove_file(&path);
|
||||
warn!(%uid, attempt, "queued revision changed while opening for write; retrying");
|
||||
continue;
|
||||
}
|
||||
let fh = st.next_fh;
|
||||
st.next_fh += 1;
|
||||
let aw = st.active_writes.entry(ino.0).or_insert_with(|| {
|
||||
WriteHandle {
|
||||
ino: ino.0,
|
||||
uid: uid.clone(),
|
||||
file: Arc::new(file),
|
||||
path,
|
||||
written: initial_written,
|
||||
// Starts at the current size; reads in [0, base_size)
|
||||
// come from the base until overwritten.
|
||||
len: base_size,
|
||||
base_size,
|
||||
base_mtime,
|
||||
base_revision_id,
|
||||
dirty: false,
|
||||
open_count: 0,
|
||||
}
|
||||
});
|
||||
aw.open_count += 1;
|
||||
st.handles.insert(fh, ino.0);
|
||||
break 'install Ok(fh);
|
||||
}
|
||||
error!(%uid, "queued revision kept changing while opening for write");
|
||||
Err(Errno::EIO)
|
||||
};
|
||||
let mut initial_written = Intervals::default();
|
||||
if let Some(pending) = &pending_base {
|
||||
// Reflink where the filesystem can: this blob is the whole file, and
|
||||
// a byte-for-byte copy of a multi-GB pending revision is the kind of
|
||||
// work `open` must not do (`ContentCache::clone_or_copy`).
|
||||
if let Err(e) = ContentCache::clone_or_copy(&pending.path, &path) {
|
||||
match outcome {
|
||||
Ok(fh) => reply.opened(FileHandle(fh), FopenFlags::empty()),
|
||||
Err(error) => {
|
||||
// The lookup count was raised before any of this could fail.
|
||||
if let Some(entry) = self.core.state().entries.get_mut(&ino.0) {
|
||||
entry.open_count = entry.open_count.saturating_sub(1);
|
||||
}
|
||||
let _ = std::fs::remove_file(&path);
|
||||
error!(%uid, source = %pending.path.display(), error = %e,
|
||||
"copy queued revision into write scratch failed");
|
||||
reply.error(Errno::EIO);
|
||||
return;
|
||||
reply.error(error);
|
||||
}
|
||||
initial_written.add(0, pending.meta.len);
|
||||
}
|
||||
let mut st = self.core.state();
|
||||
let fh = st.next_fh;
|
||||
st.next_fh += 1;
|
||||
let aw = st.active_writes.entry(ino.0).or_insert_with(|| {
|
||||
WriteHandle {
|
||||
ino: ino.0,
|
||||
uid,
|
||||
file: Arc::new(file),
|
||||
path,
|
||||
written: initial_written,
|
||||
// Starts at the current size; reads in [0, base_size) come from
|
||||
// the base until overwritten.
|
||||
len: base_size,
|
||||
base_size,
|
||||
base_mtime,
|
||||
base_revision_id,
|
||||
dirty: false,
|
||||
open_count: 0,
|
||||
}
|
||||
});
|
||||
aw.open_count += 1;
|
||||
st.handles.insert(fh, ino.0);
|
||||
reply.opened(FileHandle(fh), FopenFlags::empty());
|
||||
}
|
||||
|
||||
/// The body of [`Filesystem::fsync`], run off the dispatch loop.
|
||||
|
||||
+452
-41
@@ -55,12 +55,12 @@ use fuser::{
|
||||
};
|
||||
use futures::StreamExt as _;
|
||||
use pdfs_core::batch;
|
||||
use pdfs_core::cache::{BLOCK_SIZE, Baseline, ContentCache, StagedWrite};
|
||||
use pdfs_core::cache::{BLOCK_SIZE, Baseline, BlockGeometry, BlockSpan, ContentCache, StagedWrite};
|
||||
use pdfs_core::config::{AppDirs, SweepMode};
|
||||
use pdfs_core::control::{
|
||||
ActivityEntry, ActivityKind, DirEntry, ErrorKind, LocalHit, PhotoKind, PublicLinkInfo,
|
||||
SearchFilters, SearchHit, SearchSource, SyncFolderInfo, SyncPhase, SyncProgress,
|
||||
TransferDirection,
|
||||
ThumbnailBuildStatus, TransferDirection,
|
||||
};
|
||||
use pdfs_core::db::{
|
||||
Db, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH, PARK_UNTIL, PendingOp,
|
||||
@@ -92,9 +92,11 @@ mod profile;
|
||||
mod reads;
|
||||
mod revisions;
|
||||
mod sharing;
|
||||
mod shutdown;
|
||||
mod state;
|
||||
mod sweep;
|
||||
mod sync;
|
||||
mod takeout;
|
||||
mod transfers;
|
||||
mod upload;
|
||||
mod r#virtual;
|
||||
@@ -166,6 +168,30 @@ const DRAIN_IDLE_POLL: Duration = Duration::from_secs(30);
|
||||
/// creates a conflict copy of itself. Holding the op for a short window gives
|
||||
/// the follow-up write time to supersede it.
|
||||
const DRAIN_REVISION_DEBOUNCE: Duration = Duration::from_secs(2);
|
||||
/// Ceiling on the *adaptive* part of that grace period.
|
||||
///
|
||||
/// The fixed 2 seconds is enough for a preallocate-then-write tool, and wrong
|
||||
/// for a file that takes half a minute to upload: an editor saving every ten
|
||||
/// seconds queues a write, watches it start uploading, and supersedes it — each
|
||||
/// save uploading bytes that the next one replaces, and never catching up.
|
||||
/// Widening the debounce toward how long this node's last upload actually took
|
||||
/// makes the supersede happen in the queue, where it is free, instead of on the
|
||||
/// wire. Bounded so a file on a slow link still reaches Drive in a minute rather
|
||||
/// than deferring indefinitely, and so a one-off stall cannot park a node.
|
||||
const DRAIN_REVISION_DEBOUNCE_MAX: Duration = Duration::from_secs(60);
|
||||
/// How many nodes' measured upload times to remember for that. Bounds a map
|
||||
/// that would otherwise grow with every file ever written; the debounce falls
|
||||
/// back to [`DRAIN_REVISION_DEBOUNCE`] for anything evicted, which is the
|
||||
/// behaviour a node that has never been uploaded gets anyway.
|
||||
const UPLOAD_TIME_MEMORY: usize = 512;
|
||||
/// Threads draining the pending-op queue.
|
||||
///
|
||||
/// One meant a single 10 GiB upload blocked every queued rename, trash and
|
||||
/// small write behind it. Ordering only has to hold per node — enforced by the
|
||||
/// claim query, not by the thread count — so the rest is throughput. Kept small:
|
||||
/// these are uploads, and past a handful they compete for the same uplink and
|
||||
/// for the SDK's own per-request concurrency.
|
||||
const DRAIN_WORKERS: usize = 3;
|
||||
|
||||
const ONLINE_PROBE_MIN: Duration = Duration::from_secs(5);
|
||||
const ONLINE_PROBE_MAX: Duration = Duration::from_secs(300);
|
||||
@@ -348,6 +374,45 @@ impl SearchRoots {
|
||||
}
|
||||
}
|
||||
|
||||
/// Negative thumbnail knowledge has two different authorities. A remote miss
|
||||
/// only says Proton has no thumbnail to download; it must never suppress local
|
||||
/// generation. A local miss says this exact file revision was successfully
|
||||
/// inspected and could not be decoded. Both maps are bounded independently.
|
||||
#[derive(Default)]
|
||||
struct ThumbnailMissCaches {
|
||||
remote: HashMap<(NodeUid, i32), i64>,
|
||||
local: HashMap<(NodeUid, i32), i64>,
|
||||
}
|
||||
|
||||
impl ThumbnailMissCaches {
|
||||
fn remote_contains(&self, key: &(NodeUid, i32), tag: i64) -> bool {
|
||||
self.remote.get(key) == Some(&tag)
|
||||
}
|
||||
|
||||
fn local_contains(&self, key: &(NodeUid, i32), tag: i64) -> bool {
|
||||
self.local.get(key) == Some(&tag)
|
||||
}
|
||||
|
||||
fn remember_remote(&mut self, key: (NodeUid, i32), tag: i64) {
|
||||
Self::remember(&mut self.remote, key, tag);
|
||||
}
|
||||
|
||||
fn remember_local(&mut self, key: (NodeUid, i32), tag: i64) {
|
||||
Self::remember(&mut self.local, key, tag);
|
||||
}
|
||||
|
||||
fn forget_local(&mut self, key: &(NodeUid, i32)) {
|
||||
self.local.remove(key);
|
||||
}
|
||||
|
||||
fn remember(map: &mut HashMap<(NodeUid, i32), i64>, key: (NodeUid, i32), tag: i64) {
|
||||
if map.len() >= MAX_THUMBNAIL_MISSES && !map.contains_key(&key) {
|
||||
map.clear();
|
||||
}
|
||||
map.insert(key, tag);
|
||||
}
|
||||
}
|
||||
|
||||
/// `path` expressed relative to `root`, or `None` if it is not below it. The
|
||||
/// root itself resolves to the empty path; an empty root is the mount itself
|
||||
/// and covers everything.
|
||||
@@ -386,6 +451,10 @@ struct Core {
|
||||
/// kernel read-ahead, our own prefetch) share a single download and decrypt.
|
||||
/// See [`BlockFlight`].
|
||||
block_flight: Arc<BlockFlight>,
|
||||
/// One lock per file being repaired by [`Core::repair_block`], so the
|
||||
/// concurrent block reads that all discover the same untrustworthy block
|
||||
/// table share a single whole-file download instead of one each.
|
||||
content_repairs: Arc<tokio::sync::Mutex<HashMap<NodeUid, Arc<tokio::sync::Mutex<()>>>>>,
|
||||
/// Per-file sequential-read detection driving [`Core::prefetch`].
|
||||
prefetch: Arc<Prefetch>,
|
||||
/// Permits bounding speculative block fetches, so prefetch cannot queue in
|
||||
@@ -423,9 +492,26 @@ struct Core {
|
||||
/// Nodes removed through this daemon which a briefly stale remote listing
|
||||
/// may still return. Uids are not reused; an explicit restore clears one.
|
||||
hidden: Arc<Mutex<HashSet<NodeUid>>>,
|
||||
/// Nudges the drain worker: set true and notify to have it re-examine the
|
||||
/// queue instead of waiting out its backoff.
|
||||
/// Nudges the drain workers: set true and notify to have them re-examine
|
||||
/// the queue instead of waiting out their backoff.
|
||||
drain_wake: Arc<(Mutex<bool>, Condvar)>,
|
||||
/// The daemon's stop signal, shared with every background loop this `Core`
|
||||
/// was cloned into so teardown can end and join them (bugs.md B44).
|
||||
shutdown: Arc<shutdown::Shutdown>,
|
||||
/// How long this node's last upload actually took, in milliseconds, with
|
||||
/// the time it was measured for eviction. Feeds
|
||||
/// [`Core::revision_debounce`]: a file that takes 30 seconds to send needs
|
||||
/// a grace period on that scale, or every save supersedes an upload that
|
||||
/// was already on the wire. Bounded by [`UPLOAD_TIME_MEMORY`].
|
||||
upload_times: Arc<Mutex<HashMap<String, (u64, i64)>>>,
|
||||
/// Cancellation flags for uploads currently on the wire, keyed by node.
|
||||
///
|
||||
/// Set when a newer revision of the same file is queued: the bytes in
|
||||
/// flight have just been superseded, and the only thing finishing them
|
||||
/// achieves is spending the user's uplink on a revision the next op
|
||||
/// replaces. Read by the upload's own reader ([`CountingReader`]), which is
|
||||
/// the one thing the SDK calls often enough to notice.
|
||||
upload_cancel: Arc<Mutex<HashMap<NodeUid, Arc<AtomicBool>>>>,
|
||||
/// True while a background refresh of the photos timeline (resp. the trash) is
|
||||
/// already running, so a burst of page requests against a stale listing kicks
|
||||
/// off one refresh rather than one per request.
|
||||
@@ -464,8 +550,22 @@ struct Core {
|
||||
/// those downloads is a full-size photo — so an in-flight uid is never started
|
||||
/// twice.
|
||||
thumb_gen: Arc<Mutex<HashSet<NodeUid>>>,
|
||||
/// Nodes the remote has told us have *no* thumbnail of a given type, keyed by
|
||||
/// `(uid, thumbnail type)` and holding the mtime the answer was learned at.
|
||||
/// Global permit pool for locally generated thumbnails. The old per-request
|
||||
/// limit multiplied when the GUI sent several batches, allowing dozens of
|
||||
/// full-size images to download at once.
|
||||
thumb_gen_budget: Arc<tokio::sync::Semaphore>,
|
||||
/// Current ordinary-file listing generation and a wake-up for tasks waiting
|
||||
/// on a permit or a download when that listing is abandoned.
|
||||
file_thumb_generation: Arc<AtomicU64>,
|
||||
file_thumb_cancel: Arc<tokio::sync::Notify>,
|
||||
/// Progress of the explicit recursive build started from the Files toolbar.
|
||||
thumbnail_build: Arc<Mutex<ThumbnailBuildStatus>>,
|
||||
/// Cancellation for the explicit recursive build. Separate from ordinary
|
||||
/// listing cancellation because the two jobs have independent lifetimes.
|
||||
thumbnail_build_cancelled: Arc<AtomicBool>,
|
||||
thumbnail_build_cancel: Arc<tokio::sync::Notify>,
|
||||
/// Negative thumbnail knowledge, separated by whether the remote lacked a
|
||||
/// thumbnail or local decoding proved this revision unsupported.
|
||||
///
|
||||
/// Absence has to be cached or it costs a round trip every time it is asked
|
||||
/// for, and it is asked for constantly: an `ls -l` from an xattr-aware lister
|
||||
@@ -473,7 +573,7 @@ struct Core {
|
||||
/// of videos re-probed 130 times per listing at ~186 ms each (B5). The mtime
|
||||
/// is the validity tag — a new revision may well have a thumbnail — matching
|
||||
/// how [`ContentCache::read_thumbnail`] validates the positive side.
|
||||
no_thumbnail: Arc<Mutex<HashMap<(NodeUid, i32), i64>>>,
|
||||
thumbnail_misses: Arc<Mutex<ThumbnailMissCaches>>,
|
||||
/// Last account quota answer and when it was learned, for `statfs(2)`.
|
||||
///
|
||||
/// `df` and the free-space preflight of every file manager and installer ask
|
||||
@@ -669,6 +769,26 @@ struct MountedState {
|
||||
/// [`Core::notifier`], still empty until its session is spawned.
|
||||
notifier: Arc<OnceLock<Notifier>>,
|
||||
session_live: Arc<AtomicBool>,
|
||||
/// This mount's in-flight size upgrades, keyed by *its* folder inodes — so
|
||||
/// a `Core` rebuilt onto this inode space by [`Core::rooted_at`] shares the
|
||||
/// batch the mount's own session is already running rather than starting a
|
||||
/// second one against inode numbers that mean nothing to it.
|
||||
size_upgrades: Arc<Mutex<HashMap<u64, Arc<SizeUpgrade>>>>,
|
||||
}
|
||||
|
||||
/// The per-mount half of a [`Core`] — every field [`Core::fork_state`] gives a
|
||||
/// fork a fresh copy of — recovered from the registry.
|
||||
///
|
||||
/// This is what lets a control request naming a path under a secondary mount be
|
||||
/// answered *in that mount's inode space* instead of being rejected for not
|
||||
/// being under the primary mountpoint (`docs/BUGS.md` B86). See
|
||||
/// [`Core::rooted_at`].
|
||||
struct MountParts {
|
||||
mountpoint: PathBuf,
|
||||
state: Arc<Mutex<State>>,
|
||||
notifier: Arc<OnceLock<Notifier>>,
|
||||
session_live: Arc<AtomicBool>,
|
||||
size_upgrades: Arc<Mutex<HashMap<u64, Arc<SizeUpgrade>>>>,
|
||||
}
|
||||
|
||||
/// Every mounted inode space in the daemon, shared by the primary `Core` and
|
||||
@@ -691,6 +811,7 @@ impl StateRegistry {
|
||||
state: &Arc<Mutex<State>>,
|
||||
notifier: Arc<OnceLock<Notifier>>,
|
||||
session_live: Arc<AtomicBool>,
|
||||
size_upgrades: Arc<Mutex<HashMap<u64, Arc<SizeUpgrade>>>>,
|
||||
) {
|
||||
let mut states = self.0.lock();
|
||||
states.retain(|m| m.state.strong_count() > 0);
|
||||
@@ -705,6 +826,7 @@ impl StateRegistry {
|
||||
state: Arc::downgrade(state),
|
||||
notifier,
|
||||
session_live,
|
||||
size_upgrades,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -731,9 +853,9 @@ impl StateRegistry {
|
||||
///
|
||||
/// On-demand roots can be nested below another location. Selecting by the
|
||||
/// longest component prefix ensures the nested session wins rather than the
|
||||
/// broader primary mount. Callers currently need the selected mount only;
|
||||
/// a future path router can derive a relative suffix with
|
||||
/// `path.strip_prefix(mountpoint)` without storing a second path value here.
|
||||
/// broader primary mount. Callers that need the relative suffix derive it
|
||||
/// with `path.strip_prefix(mountpoint)`; [`StateRegistry::covering_parts`]
|
||||
/// is the variant that also hands back enough to *serve* a request there.
|
||||
fn covering(&self, path: &Path) -> Option<LiveMount> {
|
||||
self.live()
|
||||
.into_iter()
|
||||
@@ -743,6 +865,48 @@ impl StateRegistry {
|
||||
.max_by_key(|(mountpoint, _, _, _)| mountpoint.components().count())
|
||||
}
|
||||
|
||||
/// [`StateRegistry::register`] with an empty size-upgrade map, for the
|
||||
/// tests that exercise the reap-and-route rules and nothing else.
|
||||
#[cfg(test)]
|
||||
fn register_bare(
|
||||
&self,
|
||||
mountpoint: &Path,
|
||||
state: &Arc<Mutex<State>>,
|
||||
notifier: Arc<OnceLock<Notifier>>,
|
||||
session_live: Arc<AtomicBool>,
|
||||
) {
|
||||
self.register(
|
||||
mountpoint,
|
||||
state,
|
||||
notifier,
|
||||
session_live,
|
||||
Arc::new(Mutex::new(HashMap::new())),
|
||||
);
|
||||
}
|
||||
|
||||
/// [`StateRegistry::covering`], but returning every per-mount field a
|
||||
/// [`Core`] needs to be re-rooted onto that inode space.
|
||||
///
|
||||
/// Kept separate from `covering` so the common callers — which only ask
|
||||
/// *whether* a path is mounted — keep the cheaper tuple.
|
||||
fn covering_parts(&self, path: &Path) -> Option<MountParts> {
|
||||
let mut states = self.0.lock();
|
||||
states.retain(|m| m.state.strong_count() > 0);
|
||||
states
|
||||
.iter()
|
||||
.filter(|m| m.session_live.load(Ordering::Acquire) && path.starts_with(&m.mountpoint))
|
||||
.max_by_key(|m| m.mountpoint.components().count())
|
||||
.and_then(|m| {
|
||||
Some(MountParts {
|
||||
mountpoint: m.mountpoint.clone(),
|
||||
state: m.state.upgrade()?,
|
||||
notifier: m.notifier.clone(),
|
||||
session_live: m.session_live.clone(),
|
||||
size_upgrades: m.size_upgrades.clone(),
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
fn is_mounted_at(&self, path: &Path) -> bool {
|
||||
self.covering(path).is_some_and(|(mountpoint, _, _, live)| {
|
||||
mountpoint == path && live.load(Ordering::Acquire)
|
||||
@@ -862,9 +1026,41 @@ impl Core {
|
||||
&self.state,
|
||||
self.notifier.clone(),
|
||||
self.session_live.clone(),
|
||||
self.size_upgrades.clone(),
|
||||
);
|
||||
}
|
||||
|
||||
/// This `Core`, re-rooted onto whichever live mount most specifically covers
|
||||
/// the absolute path `abs`, together with `abs` relative to that mount's
|
||||
/// root.
|
||||
///
|
||||
/// Everything a `Core` holds is per-daemon except the five fields
|
||||
/// [`Core::fork_state`] replaces, so swapping exactly those turns the
|
||||
/// primary `Core` into the fork that owns the path — sharing the fork's
|
||||
/// inode space, notification channel and size-upgrade batches rather than
|
||||
/// shadowing them. Without this, every path-addressed control request is
|
||||
/// answered against the primary inode space, so anything under a secondary
|
||||
/// on-demand mount is rejected as "not under the mountpoint" and the user
|
||||
/// has no way to re-enumerate a folder they can see (`docs/BUGS.md` B86).
|
||||
///
|
||||
/// `None` when no live mount covers `abs`.
|
||||
fn rooted_at(&self, abs: &Path) -> Option<(Core, PathBuf)> {
|
||||
let parts = self.states.covering_parts(abs)?;
|
||||
let rel = abs.strip_prefix(&parts.mountpoint).ok()?.to_path_buf();
|
||||
let mut core = self.clone();
|
||||
if !Arc::ptr_eq(&parts.state, &self.state) {
|
||||
// A different inode space, so adopt its per-mount half wholesale.
|
||||
// `primary` is a property of the mount, and the only mount this
|
||||
// `Core` can be the primary of is the one it was built for.
|
||||
core.primary = false;
|
||||
core.state = parts.state;
|
||||
core.notifier = parts.notifier;
|
||||
core.session_live = parts.session_live;
|
||||
core.size_upgrades = parts.size_upgrades;
|
||||
}
|
||||
Some((core, rel))
|
||||
}
|
||||
|
||||
/// Run `apply` against every live inode space — this daemon's primary mount
|
||||
/// and each on-demand fork — one lock at a time.
|
||||
///
|
||||
@@ -935,6 +1131,27 @@ impl Core {
|
||||
/// downgrade cannot be hidden by stale live state, and a live downgrade
|
||||
/// cannot be hidden by a restored DB row. Missing UIDs fail closed.
|
||||
fn require_uid_writable(&self, uid: &NodeUid) -> Result<(), Errno> {
|
||||
match self.uid_write_authority(uid) {
|
||||
WriteAuthority::Writable => Ok(()),
|
||||
// A uid nobody can speak for is refused here just as a denied one
|
||||
// is: a syscall is a live request against a live tree, and a stale
|
||||
// handle must not be admitted because the tree forgot the node.
|
||||
// The drain, which replays intent recorded long ago, is the one
|
||||
// caller that has to tell the two apart — see
|
||||
// [`Core::uid_write_authority`].
|
||||
WriteAuthority::Denied | WriteAuthority::Unknown => Err(Errno::EACCES),
|
||||
}
|
||||
}
|
||||
|
||||
/// The full answer behind [`Core::require_uid_writable`]: whether every
|
||||
/// authority agrees this node is writable, refuses it, or has never heard
|
||||
/// of it.
|
||||
///
|
||||
/// The third case is the one worth naming. A node absent from the local
|
||||
/// tree yields no access row, and collapsing that into "denied" told the
|
||||
/// drain to wait for a permission change that was never coming, because
|
||||
/// nothing about the node's permissions was ever the problem (B83).
|
||||
pub(crate) fn uid_write_authority(&self, uid: &NodeUid) -> WriteAuthority {
|
||||
let mut live_access = Vec::new();
|
||||
{
|
||||
let state = self.state();
|
||||
@@ -950,22 +1167,38 @@ impl Core {
|
||||
live_access.push(access);
|
||||
}
|
||||
}
|
||||
require_uid_access(&self.db, uid, &live_access)
|
||||
uid_write_authority(&self.db, uid, &live_access)
|
||||
}
|
||||
}
|
||||
|
||||
fn require_uid_access(db: &Db, uid: &NodeUid, live_access: &[Access]) -> Result<(), Errno> {
|
||||
/// Whether a node may be written, as agreed by every authority that has an
|
||||
/// opinion — or the fact that none of them does.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub(crate) enum WriteAuthority {
|
||||
/// The persisted tree and every resident mount agree: writable.
|
||||
Writable,
|
||||
/// At least one authority refuses it, or could not be consulted. Fails
|
||||
/// closed, so a lookup error lands here.
|
||||
Denied,
|
||||
/// The node is not in the local tree, so there is no authority to consult.
|
||||
/// Not the same as a refusal, and callers that can ask the remote should.
|
||||
Unknown,
|
||||
}
|
||||
|
||||
fn uid_write_authority(db: &Db, uid: &NodeUid, live_access: &[Access]) -> WriteAuthority {
|
||||
let persisted = match db.effective_node_access(uid) {
|
||||
Ok(Some(access)) => access,
|
||||
Ok(None) => return Err(Errno::EACCES),
|
||||
Ok(None) => return WriteAuthority::Unknown,
|
||||
Err(error) => {
|
||||
error!(%uid, %error, "db effective access lookup failed");
|
||||
return Err(Errno::EACCES);
|
||||
return WriteAuthority::Denied;
|
||||
}
|
||||
};
|
||||
(persisted.writable() && live_access.iter().all(|access| access.writable()))
|
||||
.then_some(())
|
||||
.ok_or(Errno::EACCES)
|
||||
if persisted.writable() && live_access.iter().all(|access| access.writable()) {
|
||||
WriteAuthority::Writable
|
||||
} else {
|
||||
WriteAuthority::Denied
|
||||
}
|
||||
}
|
||||
|
||||
/// One kernel notification, recorded while the State lock is held so it can be
|
||||
@@ -1633,7 +1866,9 @@ impl Core {
|
||||
fn run_online_probe(&self) {
|
||||
let mut delay = ONLINE_PROBE_MIN;
|
||||
loop {
|
||||
std::thread::sleep(delay);
|
||||
if !self.shutdown.sleep(delay) {
|
||||
return;
|
||||
}
|
||||
match self.rt.block_on(self.client.get_my_files_folder()) {
|
||||
Ok(root) => {
|
||||
{
|
||||
@@ -2756,6 +2991,12 @@ impl Core {
|
||||
}
|
||||
})?;
|
||||
let meta_json = serde_json::to_string(&meta).unwrap_or_default();
|
||||
// Whatever this write supersedes, it supersedes now — including an
|
||||
// upload already on the wire, whose bytes describe a revision the row
|
||||
// written just below replaces. Signalled before the enqueue so the
|
||||
// in-flight reader stops at its next block rather than after however
|
||||
// much of the file is left.
|
||||
self.cancel_upload(uid);
|
||||
let superseded = if is_local_uid(uid) {
|
||||
// The node has no server-side identity to hang a revision on, so the
|
||||
// bytes ride on the create that will mint it.
|
||||
@@ -2789,7 +3030,7 @@ impl Core {
|
||||
created_at: now_millis(),
|
||||
attempts: 0,
|
||||
last_error: None,
|
||||
next_attempt_at: now_millis() + DRAIN_REVISION_DEBOUNCE.as_millis() as i64,
|
||||
next_attempt_at: now_millis() + self.revision_debounce(uid).as_millis() as i64,
|
||||
};
|
||||
let (_id, superseded) = self.db.enqueue_op(&op).map_err(|e| {
|
||||
error!(%uid, error = %e, "queueing upload failed");
|
||||
@@ -3095,6 +3336,12 @@ impl Core {
|
||||
/// worse in every way.
|
||||
fn queue_trash(&self, uid: &NodeUid, name: &str) -> Result<(), Errno> {
|
||||
self.require_uid_writable(uid)?;
|
||||
// The bytes below are about to be discarded, and a drain worker may be
|
||||
// reading them onto the wire right now. Stop it first: with several
|
||||
// workers the trash op this queues can be claimed while that upload is
|
||||
// still running, and the two would race to decide whether the file
|
||||
// exists.
|
||||
self.cancel_upload(uid);
|
||||
let (_, blobs) = self
|
||||
.db
|
||||
.replace_ops_with_trash(&uid.to_string(), name, now_millis())
|
||||
@@ -3124,6 +3371,9 @@ impl Core {
|
||||
|
||||
/// Drop every op queued against a node, and the staged bytes they own.
|
||||
fn discard_queued_ops(&self, uid: &NodeUid) -> Result<(), Errno> {
|
||||
// Same reason as `queue_trash`: an upload may be reading the very blobs
|
||||
// this is about to unlink.
|
||||
self.cancel_upload(uid);
|
||||
let blobs = self.db.delete_ops_for_uid(&uid.to_string()).map_err(|e| {
|
||||
error!(%uid, error = %e, "dropping queued ops failed");
|
||||
Errno::EIO
|
||||
@@ -3135,6 +3385,15 @@ impl Core {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Which staged blob is currently queued for `uid`, if any.
|
||||
///
|
||||
/// The path identifies the revision: publication stages a fresh file and
|
||||
/// swaps the map entry to point at it, so an unchanged path means the base a
|
||||
/// caller sampled earlier is still the base now (bugs.md B32).
|
||||
fn pending_blob(&self, uid: &NodeUid) -> Option<PathBuf> {
|
||||
self.pending.lock().get(uid).map(|p| p.path.clone())
|
||||
}
|
||||
|
||||
/// Nudge the drain worker to re-examine the queue now.
|
||||
fn wake_drain(&self) {
|
||||
let (lock, cv) = &*self.drain_wake;
|
||||
@@ -3374,7 +3633,7 @@ impl Core {
|
||||
// "This file has no thumbnail" is an answer worth remembering: without
|
||||
// it every listing pays a round trip per file to be told nothing (B5).
|
||||
let key = (uid.clone(), ttype.as_i32());
|
||||
if self.no_thumbnail.lock().get(&key) == Some(&mtime) {
|
||||
if self.thumbnail_misses.lock().remote_contains(&key, mtime) {
|
||||
return Ok(None);
|
||||
}
|
||||
let bytes = self
|
||||
@@ -3391,15 +3650,7 @@ impl Core {
|
||||
.store_thumbnail(&uid, ttype.as_i32(), mtime, bytes);
|
||||
}
|
||||
None => {
|
||||
let mut misses = self.no_thumbnail.lock();
|
||||
// Bounded by clearing rather than by LRU: the entries are two
|
||||
// words each, the cap is far above any plausible working set,
|
||||
// and re-learning a miss costs one round trip. Not worth a
|
||||
// second data structure to order them.
|
||||
if misses.len() >= MAX_THUMBNAIL_MISSES {
|
||||
misses.clear();
|
||||
}
|
||||
misses.insert(key, mtime);
|
||||
self.thumbnail_misses.lock().remember_remote(key, mtime);
|
||||
}
|
||||
}
|
||||
Ok(bytes)
|
||||
@@ -3825,6 +4076,33 @@ impl Core {
|
||||
}
|
||||
}
|
||||
|
||||
/// Drop the cached child listing of the folder `parent`, everywhere it is
|
||||
/// held — naming the folder by uid rather than by inode.
|
||||
///
|
||||
/// [`Core::invalidate_parent_listing`] is the path-based sibling, and it
|
||||
/// only reaches *this* mount's inode space. A remote mutation made by the
|
||||
/// sync engine needs more than that, for two reasons. The folder it changed
|
||||
/// is usually not resident in any inode space at the time — a mirror folder
|
||||
/// has no FUSE session at all — so there is no inode to name. And the stale
|
||||
/// listing that outlives the pass is not a hot-cache map but the `listed`
|
||||
/// flag on the folder's DB row, which survives a daemon restart and is what
|
||||
/// a later `ensure_children` trusts.
|
||||
///
|
||||
/// So the flag is cleared unconditionally, and the resident inode spaces are
|
||||
/// invalidated on top of it. That is what makes a mirror folder switched to
|
||||
/// on-demand re-enumerate instead of serving a snapshot that predates the
|
||||
/// engine's own uploads — `docs/BUGS.md` B86.
|
||||
pub(crate) fn invalidate_children_of(&self, parent: &NodeUid) {
|
||||
if let Err(e) = self.db.set_listed(parent, false) {
|
||||
warn!(uid = %parent, error = ?e, "could not clear cached listing");
|
||||
}
|
||||
self.for_each_state(|st| {
|
||||
if let Some(&ino) = st.by_uid.get(parent) {
|
||||
st.invalidate_listing(ino);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Rename a file or folder to `new_name`. `rel` is mountpoint-relative.
|
||||
/// Mirrors the FUSE `rename` write path: rename on the remote, forget the
|
||||
/// node so it re-interns under its new name, and drop the parent listing so
|
||||
@@ -5701,9 +5979,19 @@ mod tests {
|
||||
preserve_on_access_denied, publish_virtual_root_in_listing,
|
||||
reconcile_virtual_root_in_listing, release_can_discard_unlinked,
|
||||
release_must_retain_queued_trash, release_unlinked_entry, rename_needs_queue,
|
||||
require_node_parent_access, require_rename_access, require_uid_access,
|
||||
resolve_anywhere_with, shared_with_me_uid, take_self_change, virtual_node,
|
||||
require_node_parent_access, require_rename_access, resolve_anywhere_with,
|
||||
shared_with_me_uid, take_self_change, uid_write_authority, virtual_node,
|
||||
};
|
||||
use super::{Db, WriteAuthority};
|
||||
|
||||
/// The collapse [`Core::require_uid_writable`] performs, without a `Core`:
|
||||
/// every non-writable authority is one `EACCES` on the syscall path.
|
||||
fn require_uid_access(db: &Db, uid: &NodeUid, live: &[Access]) -> Result<(), Errno> {
|
||||
match uid_write_authority(db, uid, live) {
|
||||
WriteAuthority::Writable => Ok(()),
|
||||
WriteAuthority::Denied | WriteAuthority::Unknown => Err(Errno::EACCES),
|
||||
}
|
||||
}
|
||||
use crate::filesystem::access_allowed;
|
||||
use pdfs_core::cache::{Baseline, StagedWrite};
|
||||
use pdfs_core::db::{OP_REVISION, OP_TRASH, PendingOp};
|
||||
@@ -5734,7 +6022,7 @@ mod tests {
|
||||
("/mnt/fork-a", &fork_a),
|
||||
("/mnt/fork-b", &fork_b),
|
||||
] {
|
||||
registry.register(
|
||||
registry.register_bare(
|
||||
std::path::Path::new(mountpoint),
|
||||
state,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
@@ -5785,19 +6073,19 @@ mod tests {
|
||||
let nested_live = session_flag(false);
|
||||
let sibling_live = session_flag(true);
|
||||
|
||||
registry.register(
|
||||
registry.register_bare(
|
||||
primary_path,
|
||||
&primary,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
primary_live.clone(),
|
||||
);
|
||||
registry.register(
|
||||
registry.register_bare(
|
||||
nested_path,
|
||||
&nested,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
nested_live.clone(),
|
||||
);
|
||||
registry.register(
|
||||
registry.register_bare(
|
||||
sibling_path,
|
||||
&sibling,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
@@ -5875,6 +6163,71 @@ mod tests {
|
||||
assert_eq!(registry.live().len(), 2);
|
||||
}
|
||||
|
||||
/// The routing half of `docs/BUGS.md` B86: a path under a secondary mount
|
||||
/// has to resolve to *that* mount's inode space and a suffix relative to
|
||||
/// *its* root, or `pdfs ls` and `pdfs refresh` can only ever name the
|
||||
/// primary mount and a stale on-demand folder has no escape hatch.
|
||||
#[test]
|
||||
fn covering_parts_route_a_path_to_the_mount_that_owns_it() {
|
||||
let registry = StateRegistry::default();
|
||||
let (primary, _primary_dir) = rooted_state("primary-volume", "primary");
|
||||
let (nested, _nested_dir) = rooted_state("nested-volume", "nested");
|
||||
let primary = std::sync::Arc::new(parking_lot::Mutex::new(primary));
|
||||
let nested = std::sync::Arc::new(parking_lot::Mutex::new(nested));
|
||||
let primary_path = std::path::Path::new("/home/me/ProtonDrive");
|
||||
let nested_path = std::path::Path::new("/home/me/ProtonDrive/Device");
|
||||
let primary_upgrades = std::sync::Arc::new(parking_lot::Mutex::new(HashMap::new()));
|
||||
let nested_upgrades = std::sync::Arc::new(parking_lot::Mutex::new(HashMap::new()));
|
||||
|
||||
registry.register(
|
||||
primary_path,
|
||||
&primary,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
session_flag(true),
|
||||
primary_upgrades.clone(),
|
||||
);
|
||||
registry.register(
|
||||
nested_path,
|
||||
&nested,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
session_flag(true),
|
||||
nested_upgrades.clone(),
|
||||
);
|
||||
|
||||
let deep = nested_path.join("sub/file.txt");
|
||||
let parts = registry.covering_parts(&deep).expect("nested mount covers");
|
||||
assert_eq!(parts.mountpoint, nested_path, "the nested mount wins");
|
||||
assert!(
|
||||
std::sync::Arc::ptr_eq(&parts.state, &nested),
|
||||
"the request must be answered in the nested inode space"
|
||||
);
|
||||
assert!(
|
||||
std::sync::Arc::ptr_eq(&parts.size_upgrades, &nested_upgrades),
|
||||
"and share that mount's in-flight size upgrades, not the primary's"
|
||||
);
|
||||
assert_eq!(
|
||||
deep.strip_prefix(&parts.mountpoint).unwrap(),
|
||||
std::path::Path::new("sub/file.txt"),
|
||||
"the suffix is relative to the mount that owns it, not the primary"
|
||||
);
|
||||
|
||||
let shallow = primary_path.join("Documents");
|
||||
let parts = registry
|
||||
.covering_parts(&shallow)
|
||||
.expect("primary mount covers");
|
||||
assert!(
|
||||
std::sync::Arc::ptr_eq(&parts.state, &primary),
|
||||
"a path outside the nested root still routes to the primary"
|
||||
);
|
||||
|
||||
assert!(
|
||||
registry
|
||||
.covering_parts(std::path::Path::new("/tmp/outside"))
|
||||
.is_none(),
|
||||
"a path under no mount routes nowhere, and the caller reports that"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn state_registry_resolves_uids_from_any_resident_mount() {
|
||||
let registry = StateRegistry::default();
|
||||
@@ -5888,13 +6241,13 @@ mod tests {
|
||||
fork.children.insert(super::ROOT_INO, vec![fork_ino]);
|
||||
let primary = std::sync::Arc::new(parking_lot::Mutex::new(primary));
|
||||
let fork = std::sync::Arc::new(parking_lot::Mutex::new(fork));
|
||||
registry.register(
|
||||
registry.register_bare(
|
||||
std::path::Path::new("/mnt/primary"),
|
||||
&primary,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
session_flag(true),
|
||||
);
|
||||
registry.register(
|
||||
registry.register_bare(
|
||||
std::path::Path::new("/mnt/device"),
|
||||
&fork,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
@@ -5922,7 +6275,7 @@ mod tests {
|
||||
let child_ino = state.intern(super::ROOT_INO, child_node);
|
||||
state.children.insert(super::ROOT_INO, vec![child_ino]);
|
||||
let state = std::sync::Arc::new(parking_lot::Mutex::new(state));
|
||||
registry.register(
|
||||
registry.register_bare(
|
||||
std::path::Path::new("/mnt/device"),
|
||||
&state,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
@@ -5985,7 +6338,7 @@ mod tests {
|
||||
);
|
||||
|
||||
let state = std::sync::Arc::new(parking_lot::Mutex::new(state));
|
||||
registry.register(
|
||||
registry.register_bare(
|
||||
std::path::Path::new("/mnt/device"),
|
||||
&state,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
@@ -6253,7 +6606,7 @@ mod tests {
|
||||
state.children.insert(revoked_ino, vec![child_ino]);
|
||||
|
||||
let state = std::sync::Arc::new(parking_lot::Mutex::new(state));
|
||||
registry.register(
|
||||
registry.register_bare(
|
||||
std::path::Path::new("/mnt/device"),
|
||||
&state,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
@@ -6307,7 +6660,7 @@ mod tests {
|
||||
state.children.insert(cycle_b, vec![cycle_a]);
|
||||
|
||||
let state = std::sync::Arc::new(parking_lot::Mutex::new(state));
|
||||
registry.register(
|
||||
registry.register_bare(
|
||||
std::path::Path::new("/mnt/device"),
|
||||
&state,
|
||||
std::sync::Arc::new(std::sync::OnceLock::new()),
|
||||
@@ -6710,6 +7063,19 @@ mod tests {
|
||||
require_uid_access(&db, &stale, &[]).unwrap_err().code(),
|
||||
libc::EACCES
|
||||
);
|
||||
// Same refusal on the syscall path, but a distinguishable one
|
||||
// underneath: the drain asks the remote about `Unknown` rather than
|
||||
// waiting on a permission change (B83).
|
||||
assert_eq!(
|
||||
uid_write_authority(&db, &stale, &[]),
|
||||
WriteAuthority::Unknown,
|
||||
"a uid the tree has no row for is unknown, not denied"
|
||||
);
|
||||
assert_eq!(
|
||||
uid_write_authority(&db, &uid, &[]),
|
||||
WriteAuthority::Denied,
|
||||
"a uid the tree knows and refuses is denied"
|
||||
);
|
||||
assert_eq!(db.pending_ops().unwrap().len(), before);
|
||||
}
|
||||
|
||||
@@ -6876,6 +7242,19 @@ mod tests {
|
||||
let trash = function_source(source, "fn queue_trash(");
|
||||
assert_before(trash, "replace_ops_with_trash", "discard_staged");
|
||||
|
||||
// §5, parallel drain: every path that unlinks a staged blob or hands
|
||||
// the same node's work to another worker has to stop the upload that
|
||||
// may be reading those bytes *first*. Late is a superseded revision on
|
||||
// the wire, or two workers deciding whether the file exists.
|
||||
for (signature, mutation) in [
|
||||
("fn enqueue_staged_write(", "attach_blob_to_create"),
|
||||
("fn queue_trash(", "replace_ops_with_trash"),
|
||||
("fn discard_queued_ops(", "delete_ops_for_uid"),
|
||||
] {
|
||||
let function = function_source(source, signature);
|
||||
assert_before(function, "cancel_upload", mutation);
|
||||
}
|
||||
|
||||
let filesystem = include_str!("filesystem.rs");
|
||||
for signature in ["fn serve_unlink(", "fn serve_rmdir("] {
|
||||
let function = function_source(filesystem, signature);
|
||||
@@ -7212,6 +7591,38 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod thumbnail_miss_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn remote_misses_never_suppress_local_generation() {
|
||||
let uid = parse_uid("volume~node").expect("uid");
|
||||
let key = (uid, ThumbnailType::Thumbnail.as_i32());
|
||||
let mut misses = ThumbnailMissCaches::default();
|
||||
|
||||
misses.remember_remote(key.clone(), 42);
|
||||
assert!(misses.remote_contains(&key, 42));
|
||||
assert!(!misses.local_contains(&key, 42));
|
||||
|
||||
misses.remember_local(key.clone(), 42);
|
||||
assert!(misses.local_contains(&key, 42));
|
||||
misses.forget_local(&key);
|
||||
assert!(!misses.local_contains(&key, 42));
|
||||
assert!(misses.remote_contains(&key, 42));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn local_miss_cache_is_bounded_on_gui_only_paths() {
|
||||
let mut misses = ThumbnailMissCaches::default();
|
||||
for index in 0..=MAX_THUMBNAIL_MISSES {
|
||||
let uid = parse_uid(&format!("volume~node-{index}")).expect("uid");
|
||||
misses.remember_local((uid, ThumbnailType::Thumbnail.as_i32()), 42);
|
||||
}
|
||||
assert_eq!(misses.local.len(), 1);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod search_root_tests {
|
||||
use super::*;
|
||||
|
||||
@@ -365,6 +365,7 @@ pub fn mount(
|
||||
readers: Arc::new(Mutex::new(HashMap::new())),
|
||||
block_ring: Arc::new(Mutex::new(BlockRing::default())),
|
||||
block_flight: Arc::new(BlockFlight::default()),
|
||||
content_repairs: Arc::new(tokio::sync::Mutex::new(HashMap::new())),
|
||||
prefetch: Arc::new(Prefetch::default()),
|
||||
prefetch_budget: Arc::new(tokio::sync::Semaphore::new(PREFETCH_BUDGET)),
|
||||
workers: Arc::new(Workers::new(FUSE_WORKERS)?),
|
||||
@@ -376,6 +377,9 @@ pub fn mount(
|
||||
pending: Arc::new(Mutex::new(HashMap::new())),
|
||||
hidden: Arc::new(Mutex::new(HashSet::new())),
|
||||
drain_wake: Arc::new((Mutex::new(false), Condvar::new())),
|
||||
shutdown: Arc::new(crate::shutdown::Shutdown::default()),
|
||||
upload_times: Arc::new(Mutex::new(HashMap::new())),
|
||||
upload_cancel: Arc::new(Mutex::new(HashMap::new())),
|
||||
timeline_refreshing: Arc::new(AtomicBool::new(false)),
|
||||
albums_refreshing: Arc::new(AtomicBool::new(false)),
|
||||
trash_refreshing: Arc::new(AtomicBool::new(false)),
|
||||
@@ -385,7 +389,15 @@ pub fn mount(
|
||||
own_sealed_revs: Arc::new(Mutex::new(HashMap::new())),
|
||||
self_changes: Arc::new(Mutex::new(HashMap::new())),
|
||||
thumb_gen: Arc::new(Mutex::new(HashSet::new())),
|
||||
no_thumbnail: Arc::new(Mutex::new(HashMap::new())),
|
||||
thumb_gen_budget: Arc::new(tokio::sync::Semaphore::new(
|
||||
super::photos::THUMB_GEN_CONCURRENCY,
|
||||
)),
|
||||
file_thumb_generation: Arc::new(AtomicU64::new(0)),
|
||||
file_thumb_cancel: Arc::new(tokio::sync::Notify::new()),
|
||||
thumbnail_build: Arc::new(Mutex::new(Default::default())),
|
||||
thumbnail_build_cancelled: Arc::new(AtomicBool::new(false)),
|
||||
thumbnail_build_cancel: Arc::new(tokio::sync::Notify::new()),
|
||||
thumbnail_misses: Arc::new(Mutex::new(Default::default())),
|
||||
quota: Arc::new(Mutex::new(None)),
|
||||
size_upgrades: Arc::new(Mutex::new(HashMap::new())),
|
||||
size_waiters: Arc::new(Default::default()),
|
||||
@@ -413,16 +425,29 @@ pub fn mount(
|
||||
// And finally the staged blobs neither of those accounts for: bytes a
|
||||
// failed release left behind with nothing pointing at them.
|
||||
core.reconcile_staging();
|
||||
{
|
||||
// Several drain workers over one queue. One thread meant a single large
|
||||
// upload held every queued rename, trash and small write behind it for as
|
||||
// long as it ran; the claim column (`Db::claim_next_due_op`) keeps the
|
||||
// workers off each other's nodes, which is the only ordering the queue
|
||||
// actually needs. Worker 0 additionally runs the queue's idle chores.
|
||||
//
|
||||
// Every long-lived thread started from here on is kept in `workers` and
|
||||
// joined at the end of this function. Leaving them running turned an
|
||||
// in-process remount into a second generation of drain workers, sync engines
|
||||
// and sweeps all operating on a mount that no longer exists (bugs.md B44).
|
||||
let mut workers: Vec<std::thread::JoinHandle<()>> = Vec::new();
|
||||
for worker in 0..DRAIN_WORKERS {
|
||||
let core = core.clone();
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-drain".into())
|
||||
.spawn(move || core.run_pending_drain())?;
|
||||
workers.push(
|
||||
std::thread::Builder::new()
|
||||
.name(format!("pdfs-drain-{worker}"))
|
||||
.spawn(move || core.run_pending_drain(worker == 0))?,
|
||||
);
|
||||
}
|
||||
|
||||
// Start the folder-sync engine. It watches every mirror folder, polls the
|
||||
// remotes, and reconciles on its own thread — never in front of a FUSE call.
|
||||
sync::spawn(core.clone(), sync_rx);
|
||||
workers.extend(sync::spawn(core.clone(), sync_rx));
|
||||
|
||||
// Reconcile leftover `(sync-conflict …)` copies: drop the ones that turned
|
||||
// out identical to the live file, surface the ones that genuinely diverge.
|
||||
@@ -434,18 +459,22 @@ pub fn mount(
|
||||
} else {
|
||||
info!(mode = ?sweep_mode, "conflict sweep enabled");
|
||||
let core = core.clone();
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-conflict-sweep".into())
|
||||
.spawn(move || core.run_conflict_sweep_loop())?;
|
||||
workers.push(
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-conflict-sweep".into())
|
||||
.spawn(move || core.run_conflict_sweep_loop())?,
|
||||
);
|
||||
}
|
||||
|
||||
// Mounted from the cache: watch for the network coming back so the mount can
|
||||
// stop being read-only-ish without the user restarting the daemon.
|
||||
if !online {
|
||||
let core = core.clone();
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-online-probe".into())
|
||||
.spawn(move || core.run_online_probe())?;
|
||||
workers.push(
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-online-probe".into())
|
||||
.spawn(move || core.run_online_probe())?,
|
||||
);
|
||||
}
|
||||
|
||||
// Keep the launcher prompt's "This computer" index fresh. Its own thread:
|
||||
@@ -455,9 +484,12 @@ pub fn mount(
|
||||
let indexing = core.indexing.clone();
|
||||
let transfers = core.transfers.clone();
|
||||
let mountpoint = mountpoint.to_path_buf();
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-localindex".into())
|
||||
.spawn(move || run_local_index(db, indexing, transfers, mountpoint))?;
|
||||
let shutdown = core.shutdown.clone();
|
||||
workers.push(
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-localindex".into())
|
||||
.spawn(move || run_local_index(db, indexing, transfers, mountpoint, shutdown))?,
|
||||
);
|
||||
}
|
||||
|
||||
// Bind the control socket before the FUSE session takes over the thread. A
|
||||
@@ -494,23 +526,29 @@ pub fn mount(
|
||||
let control_core = core.clone();
|
||||
let username = username.clone();
|
||||
let mountpoint = mountpoint.to_path_buf();
|
||||
if let Err(error) = std::thread::Builder::new()
|
||||
match std::thread::Builder::new()
|
||||
.name("pdfs-control".into())
|
||||
.spawn(move || run_control_socket(control_core, username, mountpoint, listener))
|
||||
{
|
||||
let secondaries = core.mounts.lock().close_and_drain();
|
||||
for (id, mount) in secondaries {
|
||||
if let Err(unmount_error) = mount.teardown() {
|
||||
warn!(id, error = %unmount_error, "secondary teardown after control startup failure failed");
|
||||
Ok(handle) => workers.push(handle),
|
||||
Err(error) => {
|
||||
// A failure here abandons the mount, so the background threads
|
||||
// started above have to be told to stop with it.
|
||||
stop_workers(&core, control_socket, workers);
|
||||
let secondaries = core.mounts.lock().close_and_drain();
|
||||
for (id, mount) in secondaries {
|
||||
if let Err(unmount_error) = mount.teardown() {
|
||||
warn!(id, error = %unmount_error, "secondary teardown after control startup failure failed");
|
||||
}
|
||||
}
|
||||
if let Err(unmount_error) = teardown_session(&core.session_live, || {
|
||||
umount_session_unblocked(bg, main_conn)
|
||||
}) {
|
||||
warn!(error = %unmount_error, "unmount after control startup failure failed");
|
||||
}
|
||||
let _ = std::fs::remove_file(control_socket);
|
||||
return Err(error);
|
||||
}
|
||||
if let Err(unmount_error) = teardown_session(&core.session_live, || {
|
||||
umount_session_unblocked(bg, main_conn)
|
||||
}) {
|
||||
warn!(error = %unmount_error, "unmount after control startup failure failed");
|
||||
}
|
||||
let _ = std::fs::remove_file(control_socket);
|
||||
return Err(error);
|
||||
}
|
||||
}
|
||||
// Re-establish on-demand mounts only after the primary session is live.
|
||||
@@ -619,10 +657,43 @@ pub fn mount(
|
||||
}
|
||||
}
|
||||
|
||||
let _ = std::fs::remove_file(control_socket);
|
||||
// Last, because the workers may legitimately be finishing an upload, and
|
||||
// nothing they can still do is harmful once the mounts are down.
|
||||
stop_workers(&core, control_socket, workers);
|
||||
Ok(outcome)
|
||||
}
|
||||
|
||||
/// Signal every background loop, wake the ones that are blocked, and join them.
|
||||
///
|
||||
/// The mount used to return with its drain workers, sync engine, sweep, online
|
||||
/// probe, indexer and control listener all still running on `Core` clones. The
|
||||
/// process usually exited straight afterwards and hid it; an in-process remount
|
||||
/// did not, and started a second full set on top of the first (bugs.md B44).
|
||||
///
|
||||
/// Two of the waits need waking rather than merely flagging: the drain workers
|
||||
/// block on their own condvar (`wake_drain`), and the control listener blocks
|
||||
/// inside `accept`, which only returns when a connection arrives — so it gets
|
||||
/// one, from us, after the flag is already set. A join that takes a while is
|
||||
/// expected and correct: a worker part-way through an upload finishes it rather
|
||||
/// than abandoning the user's bytes mid-flight.
|
||||
fn stop_workers(core: &Core, control_socket: &Path, workers: Vec<std::thread::JoinHandle<()>>) {
|
||||
core.shutdown.stop();
|
||||
core.wake_drain();
|
||||
let _ = core.sync_tx.send(sync::SyncMsg::Stop);
|
||||
// Connect while the socket still exists — that is the whole point of the
|
||||
// poke — then unlink it so nothing else can arrive behind us. The listener
|
||||
// re-checks the flag after every accept, so even a connection that races the
|
||||
// unlink cannot put it back to sleep.
|
||||
let _ = std::os::unix::net::UnixStream::connect(control_socket);
|
||||
let _ = std::fs::remove_file(control_socket);
|
||||
for worker in workers {
|
||||
if worker.join().is_err() {
|
||||
warn!("a background worker panicked before shutdown");
|
||||
}
|
||||
}
|
||||
debug!("background workers stopped");
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
+983
-39
File diff suppressed because it is too large.
Load diff
+306
-71
@@ -26,6 +26,11 @@ const PREFETCH_MAX: u64 = 8;
|
||||
/// Window a sequential read starts at, and returns to after a seek.
|
||||
const PREFETCH_MIN: u64 = 2;
|
||||
|
||||
/// Largest file [`Core::repair_block`] will pull down whole to work around a
|
||||
/// revision whose block table understates it. Past this a single read would cost
|
||||
/// gigabytes of download, so the read fails instead and says to pin the file.
|
||||
pub(super) const REPAIR_MAX: u64 = 512 * 1024 * 1024;
|
||||
|
||||
/// Blocks that may be prefetching at once, across every file. Prefetch takes a
|
||||
/// permit and gives up rather than queueing, so it can never sit in front of a
|
||||
/// demand read in the SDK's flat block semaphore.
|
||||
@@ -97,22 +102,38 @@ static NEXT_OPEN_ID: AtomicU64 = AtomicU64::new(0);
|
||||
/// Validated by the same `(mtime, size)` pair the on-disk caches use: a node
|
||||
/// whose tag no longer matches has its blocks dropped rather than served, so a
|
||||
/// new revision can never be stitched together from the old one.
|
||||
/// A ring entry's identity: the node, and the plaintext range the block covers.
|
||||
///
|
||||
/// The range, rather than the block index, because the index alone stopped
|
||||
/// naming one range when the geometry started coming from the revision — a
|
||||
/// block cached under the [`BlockGeometry::uniform`] fallback and the block at
|
||||
/// the same index under the real geometry are different bytes
|
||||
/// (`docs/BUGS.md` B85). A range is the same identity under either plan, so a
|
||||
/// re-planned read misses rather than being served the wrong bytes.
|
||||
type RingKey = (NodeUid, BlockSpan);
|
||||
|
||||
#[derive(Default)]
|
||||
pub(super) struct BlockRing {
|
||||
blocks: HashMap<(NodeUid, u64), Arc<Vec<u8>>>,
|
||||
blocks: HashMap<RingKey, Arc<Vec<u8>>>,
|
||||
/// Keys least-recently-used first, for eviction.
|
||||
order: VecDeque<(NodeUid, u64)>,
|
||||
order: VecDeque<RingKey>,
|
||||
/// Per-node validity tag, `(mtime, size)`.
|
||||
tags: HashMap<NodeUid, (i64, u64)>,
|
||||
bytes: u64,
|
||||
}
|
||||
|
||||
impl BlockRing {
|
||||
fn get(&mut self, uid: &NodeUid, mtime: i64, size: u64, idx: u64) -> Option<Arc<Vec<u8>>> {
|
||||
fn get(
|
||||
&mut self,
|
||||
uid: &NodeUid,
|
||||
mtime: i64,
|
||||
size: u64,
|
||||
span: BlockSpan,
|
||||
) -> Option<Arc<Vec<u8>>> {
|
||||
if self.tags.get(uid) != Some(&(mtime, size)) {
|
||||
return None;
|
||||
}
|
||||
let key = (uid.clone(), idx);
|
||||
let key = (uid.clone(), span);
|
||||
let hit = self.blocks.get(&key).cloned()?;
|
||||
// Move to the young end: a block being read repeatedly (the whole reason
|
||||
// this exists) must not age out under a sequential read passing through.
|
||||
@@ -123,14 +144,21 @@ impl BlockRing {
|
||||
Some(hit)
|
||||
}
|
||||
|
||||
fn insert(&mut self, uid: &NodeUid, mtime: i64, size: u64, idx: u64, bytes: Arc<Vec<u8>>) {
|
||||
fn insert(
|
||||
&mut self,
|
||||
uid: &NodeUid,
|
||||
mtime: i64,
|
||||
size: u64,
|
||||
span: BlockSpan,
|
||||
bytes: Arc<Vec<u8>>,
|
||||
) {
|
||||
if self.tags.get(uid) != Some(&(mtime, size)) {
|
||||
// Revision changed under us (or first sight): anything held for this
|
||||
// node describes the old one.
|
||||
self.drop_node(uid);
|
||||
self.tags.insert(uid.clone(), (mtime, size));
|
||||
}
|
||||
let key = (uid.clone(), idx);
|
||||
let key = (uid.clone(), span);
|
||||
if self.blocks.contains_key(&key) {
|
||||
return;
|
||||
}
|
||||
@@ -171,8 +199,10 @@ impl BlockRing {
|
||||
/// The result of fetching one block, shared between everyone who wanted it.
|
||||
type BlockResult = Result<Arc<Vec<u8>>, Errno>;
|
||||
|
||||
/// Identifies a block *of a revision*: node, validity tag, index.
|
||||
type BlockKey = (NodeUid, i64, u64, u64);
|
||||
/// Identifies a block *of a revision*: node, validity tag, and the plaintext
|
||||
/// range the block covers — see [`RingKey`] for why it is the range and not the
|
||||
/// index.
|
||||
type BlockKey = (NodeUid, i64, u64, BlockSpan);
|
||||
|
||||
/// A claim on a block key: who owns the fetch, and where its result is published.
|
||||
type Claim = (u64, tokio::sync::watch::Receiver<Option<BlockResult>>);
|
||||
@@ -412,11 +442,30 @@ impl Core {
|
||||
self.prefetch.streams.lock().remove(uid);
|
||||
}
|
||||
|
||||
/// How this revision's plaintext is divided into content blocks.
|
||||
///
|
||||
/// The revision's own geometry when a previous read has learned and recorded
|
||||
/// it, and the [`BLOCK_SIZE`] assumption when it has not — which is every
|
||||
/// first read, because learning it means opening a `RevisionReader`, and
|
||||
/// opening one per read is precisely the per-file key derivation the block
|
||||
/// cache exists to avoid (bugs.md B12, B85).
|
||||
///
|
||||
/// Being wrong here is not a correctness problem: `read_at` plans over the
|
||||
/// real sizes and clamps to what was asked for either way. It costs a
|
||||
/// straddling fetch and a cached block the next read will not ask for, and
|
||||
/// both settle once the geometry is known.
|
||||
fn block_geometry(&self, uid: &NodeUid, mtime: i64, fsize: u64) -> BlockGeometry {
|
||||
self.cache
|
||||
.block_geometry(uid, mtime, fsize)
|
||||
.unwrap_or_else(|| BlockGeometry::uniform(fsize))
|
||||
}
|
||||
|
||||
/// Serve bytes `[offset, offset + len)` of `uid`'s active revision, hitting
|
||||
/// the on-disk caches before the network: a whole-file blob (pinned files)
|
||||
/// first, then the block cache — fetching only the [`BLOCK_SIZE`]-aligned
|
||||
/// blocks that overlap the request and caching each. `mtime`/`fsize` validate
|
||||
/// both caches. Network I/O runs without any lock held.
|
||||
/// first, then the block cache — fetching only the blocks that overlap the
|
||||
/// request, on the revision's own boundaries ([`Core::block_geometry`]), and
|
||||
/// caching each. `mtime`/`fsize` validate both caches. Network I/O runs
|
||||
/// without any lock held.
|
||||
pub(super) fn read_range(
|
||||
&self,
|
||||
uid: &NodeUid,
|
||||
@@ -495,7 +544,7 @@ impl Core {
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Fetch block `bidx`, joining a fetch already in flight for it rather than
|
||||
/// Fetch the block at `span`, joining a fetch already in flight rather than
|
||||
/// issuing a second one, and publish the result to everyone who joined.
|
||||
///
|
||||
/// See [`BlockFlight`]. A follower whose leader disappears (a panic, or a
|
||||
@@ -507,10 +556,10 @@ impl Core {
|
||||
uid: &NodeUid,
|
||||
mtime: i64,
|
||||
fsize: u64,
|
||||
bidx: u64,
|
||||
span: BlockSpan,
|
||||
cache_blocks: bool,
|
||||
) -> BlockResult {
|
||||
let key: BlockKey = (uid.clone(), mtime, fsize, bidx);
|
||||
let key: BlockKey = (uid.clone(), mtime, fsize, span);
|
||||
let claim = {
|
||||
let mut inflight = self.block_flight.inflight.lock();
|
||||
match inflight.get(&key) {
|
||||
@@ -542,12 +591,12 @@ impl Core {
|
||||
}
|
||||
}
|
||||
return self
|
||||
.read_block(reader, uid, mtime, fsize, bidx, cache_blocks)
|
||||
.read_block(reader, uid, mtime, fsize, span, cache_blocks)
|
||||
.await;
|
||||
}
|
||||
};
|
||||
let result = self
|
||||
.read_block(reader, uid, mtime, fsize, bidx, cache_blocks)
|
||||
.read_block(reader, uid, mtime, fsize, span, cache_blocks)
|
||||
.await;
|
||||
// Retire the claim before publishing, so a racer that wakes and re-enters
|
||||
// finds no stale entry to join.
|
||||
@@ -563,36 +612,193 @@ impl Core {
|
||||
/// The disk store is handed to a blocking thread — it is a 4 MiB write plus
|
||||
/// a cache-index update, and neither the reactor nor the kernel read waiting
|
||||
/// on these bytes should be held up by it.
|
||||
///
|
||||
/// A block that comes back shorter than the file's size says it must be is
|
||||
/// never served, cached or promoted: it is repaired from the whole-file
|
||||
/// download, or the read fails. See [`Core::repair_block`] (bugs.md B84).
|
||||
async fn read_block(
|
||||
&self,
|
||||
reader: &Arc<RevisionReader>,
|
||||
uid: &NodeUid,
|
||||
mtime: i64,
|
||||
fsize: u64,
|
||||
bidx: u64,
|
||||
span: BlockSpan,
|
||||
cache_blocks: bool,
|
||||
) -> BlockResult {
|
||||
let bstart = bidx * BLOCK_SIZE;
|
||||
let blen = BLOCK_SIZE.min(fsize - bstart);
|
||||
let bytes = reader.read_at(bstart, blen).await.map_err(|e| {
|
||||
warn!(%uid, bstart, blen, error = %e, "block read failed");
|
||||
Errno::EIO
|
||||
})?;
|
||||
// The reader derives every block boundary from the revision's recorded
|
||||
// block sizes. If they do not add up to the size the node reports, this
|
||||
// file's boundaries are wrong, and the failure is not confined to short
|
||||
// reads: sizes that *overstate* a block shift every block after it and
|
||||
// return full-length reads of the wrong bytes. So a disagreement of any
|
||||
// kind takes the file off the range path entirely (bugs.md B84).
|
||||
let bytes = if reader.size() != fsize {
|
||||
warn!(
|
||||
%uid, bidx = span.idx, fsize,
|
||||
reader_size = reader.size(),
|
||||
reader_blocks = reader.block_sizes().len(),
|
||||
"the revision's block table disagrees with the node size"
|
||||
);
|
||||
self.repair_block(uid, mtime, fsize, span).await?
|
||||
} else {
|
||||
// Learned here rather than on a path of its own: this is where a
|
||||
// reader is open anyway, and recording the real geometry is what
|
||||
// lets the *next* read plan its blocks on the revision's own
|
||||
// boundaries instead of the 4 MiB fallback (bugs.md B85).
|
||||
if cache_blocks {
|
||||
self.cache
|
||||
.store_block_geometry(uid, mtime, fsize, reader.block_sizes());
|
||||
}
|
||||
let bytes = reader.read_at(span.start, span.len).await.map_err(|e| {
|
||||
warn!(%uid, start = span.start, len = span.len, error = %e, "block read failed");
|
||||
Errno::EIO
|
||||
})?;
|
||||
if bytes.len() as u64 == span.len {
|
||||
bytes
|
||||
} else {
|
||||
warn!(
|
||||
%uid, bidx = span.idx, blen = span.len, got = bytes.len(), fsize,
|
||||
"block came back shorter than the range asked for"
|
||||
);
|
||||
self.repair_block(uid, mtime, fsize, span).await?
|
||||
}
|
||||
};
|
||||
let bytes = Arc::new(bytes);
|
||||
if cache_blocks {
|
||||
let cache = self.cache.clone();
|
||||
let uid = uid.clone();
|
||||
let bytes = bytes.clone();
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let _ = cache.store_block(&uid, mtime, fsize, bidx, &bytes);
|
||||
let _ = cache.store_block(&uid, mtime, fsize, span, &bytes);
|
||||
});
|
||||
}
|
||||
self.block_ring
|
||||
.lock()
|
||||
.insert(uid, mtime, fsize, bidx, bytes.clone());
|
||||
.insert(uid, mtime, fsize, span, bytes.clone());
|
||||
Ok(bytes)
|
||||
}
|
||||
|
||||
/// Recover the block at `span` of a file whose per-block reads cannot be
|
||||
/// by fetching the whole revision through the manifest-verified download and
|
||||
/// serving the block out of that.
|
||||
///
|
||||
/// The range path derives every block boundary from the revision's recorded
|
||||
/// block sizes. When those understate the file, reads come back short with
|
||||
/// no error anywhere, and the caller hands userspace a truncated file that
|
||||
/// looks like a clean EOF (bugs.md B84). `download_file_to` does not do that
|
||||
/// arithmetic — it streams the block list — so it returns the whole file
|
||||
/// where the range path cannot, which is why pinning was the workaround.
|
||||
/// This does the same thing on the user's behalf, once, and the resulting
|
||||
/// blob answers every later read of the file before the block path is
|
||||
/// consulted at all.
|
||||
///
|
||||
/// Bounded by [`REPAIR_MAX`]: past that, one unlucky read would pull down
|
||||
/// gigabytes, and a clear `EIO` plus a `pdfs pin` is the better trade.
|
||||
async fn repair_block(
|
||||
&self,
|
||||
uid: &NodeUid,
|
||||
mtime: i64,
|
||||
fsize: u64,
|
||||
span: BlockSpan,
|
||||
) -> Result<Vec<u8>, Errno> {
|
||||
let (bstart, blen) = (span.start, span.len);
|
||||
if fsize > REPAIR_MAX {
|
||||
warn!(
|
||||
%uid, fsize,
|
||||
"file is too large to repair from a whole-file download; pin it to read it"
|
||||
);
|
||||
return Err(Errno::EIO);
|
||||
}
|
||||
// One download per file, not per block: every block of a file with a bad
|
||||
// block table discovers the same problem at once.
|
||||
let lock = {
|
||||
let mut locks = self.content_repairs.lock().await;
|
||||
locks.entry(uid.clone()).or_default().clone()
|
||||
};
|
||||
let guard = lock.clone().lock_owned().await;
|
||||
let downloaded = match self.cache.read_range(uid, mtime, fsize, bstart, blen) {
|
||||
// The first arrival did the work; the rest just read its blob.
|
||||
Some(bytes) if bytes.len() as u64 == blen => Ok(Some(bytes)),
|
||||
_ => self
|
||||
.download_whole_revision(uid, mtime, fsize)
|
||||
.await
|
||||
.map(|()| None),
|
||||
};
|
||||
// Before releasing, so the map entry is only dropped once no waiter is
|
||||
// still holding it.
|
||||
drop(guard);
|
||||
self.release_repair_lock(uid, lock).await;
|
||||
if let Some(bytes) = downloaded? {
|
||||
return Ok(bytes);
|
||||
}
|
||||
match self.cache.read_range(uid, mtime, fsize, bstart, blen) {
|
||||
Some(bytes) if bytes.len() as u64 == blen => {
|
||||
info!(%uid, bidx = span.idx, "served a short block from a repaired whole-file download");
|
||||
Ok(bytes)
|
||||
}
|
||||
_ => {
|
||||
error!(%uid, bidx = span.idx, fsize, "repairing a short block did not produce the block");
|
||||
Err(Errno::EIO)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Download the whole of `uid`'s active revision to disk and adopt it as the
|
||||
/// cached blob, so the range path is bypassed entirely for this file.
|
||||
///
|
||||
/// Streamed into a scratch file rather than memory: this runs for files up
|
||||
/// to [`REPAIR_MAX`], and buffering that per concurrent reader is not a cost
|
||||
/// a read should carry. A download that does not produce exactly `fsize`
|
||||
/// bytes is discarded — a short blob would fail the cache's own validity
|
||||
/// check anyway, and storing it would only hide the disagreement one layer
|
||||
/// further down.
|
||||
async fn download_whole_revision(
|
||||
&self,
|
||||
uid: &NodeUid,
|
||||
mtime: i64,
|
||||
fsize: u64,
|
||||
) -> Result<(), Errno> {
|
||||
let (mut file, path) = self.cache.create_scratch().map_err(|error| {
|
||||
error!(%uid, %error, "creating a scratch file for a repair download failed");
|
||||
Errno::EIO
|
||||
})?;
|
||||
let outcome = self.client.download_file_to(uid, &mut file).await;
|
||||
let stored = (|| {
|
||||
outcome.map_err(|error| {
|
||||
warn!(%uid, %error, "repair download failed");
|
||||
Errno::EIO
|
||||
})?;
|
||||
file.sync_all().map_err(|error| {
|
||||
warn!(%uid, %error, "syncing a repair download failed");
|
||||
Errno::EIO
|
||||
})?;
|
||||
let got = file.metadata().map(|m| m.len()).unwrap_or(0);
|
||||
if got != fsize {
|
||||
warn!(%uid, fsize, got, "repair download did not match the node size");
|
||||
return Err(Errno::EIO);
|
||||
}
|
||||
self.cache
|
||||
.store_file(uid, mtime, fsize, &path)
|
||||
.map_err(|error| {
|
||||
error!(%uid, %error, "adopting a repair download as cached content failed");
|
||||
Errno::EIO
|
||||
})
|
||||
})();
|
||||
// The blob is a hardlink to the same inode, so unlinking the scratch
|
||||
// name leaves it intact.
|
||||
let _ = std::fs::remove_file(&path);
|
||||
stored
|
||||
}
|
||||
|
||||
/// Drop a repair lock's map entry once nobody else holds it, so a file
|
||||
/// repaired long ago does not keep a mutex alive forever.
|
||||
async fn release_repair_lock(&self, uid: &NodeUid, lock: Arc<tokio::sync::Mutex<()>>) {
|
||||
let mut locks = self.content_repairs.lock().await;
|
||||
// Two: the map's and ours. Anything more means a waiter still needs it.
|
||||
if Arc::strong_count(&lock) <= 2 {
|
||||
locks.remove(uid);
|
||||
}
|
||||
}
|
||||
|
||||
/// Warm the blocks after a sequential read into the caches, in the
|
||||
/// background, so the next read finds them instead of paying a round-trip.
|
||||
///
|
||||
@@ -619,17 +825,19 @@ impl Core {
|
||||
if depth == 0 {
|
||||
return;
|
||||
}
|
||||
let last_block = fsize.saturating_sub(1) / BLOCK_SIZE;
|
||||
let wanted: Vec<u64> = ((last + 1)..=(last + depth).min(last_block))
|
||||
.filter(|&bidx| {
|
||||
let geometry = self.block_geometry(uid, mtime, fsize);
|
||||
let last_block = geometry.blocks().saturating_sub(1);
|
||||
let wanted: Vec<BlockSpan> = ((last + 1)..=(last + depth).min(last_block))
|
||||
.filter_map(|bidx| geometry.span(bidx))
|
||||
.filter(|&span| {
|
||||
// Blocks already in memory need nothing; blocks already on disk
|
||||
// are cheap enough to serve that spending a round-trip to move
|
||||
// them into the ring would be the wrong trade.
|
||||
self.block_ring
|
||||
.lock()
|
||||
.get(uid, mtime, fsize, bidx)
|
||||
.get(uid, mtime, fsize, span)
|
||||
.is_none()
|
||||
&& !(cache_blocks && self.cache.has_block(uid, mtime, fsize, bidx))
|
||||
&& !(cache_blocks && self.cache.has_block(uid, mtime, fsize, span))
|
||||
})
|
||||
.collect();
|
||||
if wanted.is_empty() {
|
||||
@@ -646,7 +854,7 @@ impl Core {
|
||||
// round-trips, and the reader catches up with the window before it
|
||||
// finishes.
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for bidx in wanted {
|
||||
for span in wanted {
|
||||
let Ok(permit) = core.prefetch_budget.clone().try_acquire_owned() else {
|
||||
break; // the budget is spent; the demand path can have it
|
||||
};
|
||||
@@ -654,10 +862,10 @@ impl Core {
|
||||
set.spawn(async move {
|
||||
let _permit = permit;
|
||||
if let Err(error) = core
|
||||
.fetch_block(&reader, &uid, mtime, fsize, bidx, cache_blocks)
|
||||
.fetch_block(&reader, &uid, mtime, fsize, span, cache_blocks)
|
||||
.await
|
||||
{
|
||||
debug!(%uid, bidx, ?error, "prefetch failed");
|
||||
debug!(%uid, bidx = span.idx, ?error, "prefetch failed");
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -687,8 +895,13 @@ impl Core {
|
||||
}
|
||||
let end = offset.saturating_add(len).min(fsize);
|
||||
let mut out = Vec::with_capacity((end - offset) as usize);
|
||||
let first = offset / BLOCK_SIZE;
|
||||
let last = (end - 1) / BLOCK_SIZE;
|
||||
// On the revision's own boundaries where they are known, and on the
|
||||
// 4 MiB fallback where they are not (bugs.md B85).
|
||||
let wanted = self.block_geometry(uid, mtime, fsize).spans(offset, end);
|
||||
let (Some(&head), Some(&tail)) = (wanted.first(), wanted.last()) else {
|
||||
return Ok(out);
|
||||
};
|
||||
let (first, last) = (head.idx, tail.idx);
|
||||
|
||||
// Collect the blocks overlapping the request, serving any already cached
|
||||
// and fetching the rest concurrently. A multi-block read (e.g. a media
|
||||
@@ -696,31 +909,31 @@ impl Core {
|
||||
// would otherwise stall on each block round-trip in turn; downloading the
|
||||
// misses in parallel saturates the connection and bounds latency at the
|
||||
// slowest single block instead of their sum.
|
||||
let mut blocks: Vec<Option<Arc<Vec<u8>>>> = Vec::with_capacity((last - first + 1) as usize);
|
||||
let mut misses: Vec<u64> = Vec::new();
|
||||
for bidx in first..=last {
|
||||
let mut blocks: Vec<Option<Arc<Vec<u8>>>> = Vec::with_capacity(wanted.len());
|
||||
let mut misses: Vec<BlockSpan> = Vec::new();
|
||||
for &span in &wanted {
|
||||
// The in-memory ring comes first on both paths: a streaming read has
|
||||
// no other cache at all, and a cached one would otherwise pay two
|
||||
// opens, a JSON parse and a 4 MiB read+alloc for every 128 KiB the
|
||||
// kernel asks for (audit T3).
|
||||
let hit = self.block_ring.lock().get(uid, mtime, fsize, bidx);
|
||||
let hit = self.block_ring.lock().get(uid, mtime, fsize, span);
|
||||
let hit = hit.or_else(|| {
|
||||
if !cache_blocks {
|
||||
return None;
|
||||
}
|
||||
let bytes = Arc::new(self.cache.cached_block(uid, mtime, fsize, bidx)?);
|
||||
let bytes = Arc::new(self.cache.cached_block(uid, mtime, fsize, span)?);
|
||||
// Promote it, so the other ~31 reads of this block are answered
|
||||
// from memory.
|
||||
self.block_ring
|
||||
.lock()
|
||||
.insert(uid, mtime, fsize, bidx, bytes.clone());
|
||||
.insert(uid, mtime, fsize, span, bytes.clone());
|
||||
Some(bytes)
|
||||
});
|
||||
match hit {
|
||||
Some(b) => blocks.push(Some(b)),
|
||||
None => {
|
||||
blocks.push(None);
|
||||
misses.push(bidx);
|
||||
misses.push(span);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -733,12 +946,12 @@ impl Core {
|
||||
let reader = self.revision_reader(uid, mtime, fsize).await?;
|
||||
|
||||
let mut set = tokio::task::JoinSet::new();
|
||||
for &bidx in &misses {
|
||||
for &span in &misses {
|
||||
let (core, reader, uid) = (self.clone(), reader.clone(), uid.clone());
|
||||
set.spawn(async move {
|
||||
core.fetch_block(&reader, &uid, mtime, fsize, bidx, cache_blocks)
|
||||
core.fetch_block(&reader, &uid, mtime, fsize, span, cache_blocks)
|
||||
.await
|
||||
.map(|bytes| (bidx, bytes))
|
||||
.map(|bytes| (span.idx, bytes))
|
||||
});
|
||||
}
|
||||
let mut out = Vec::with_capacity(misses.len());
|
||||
@@ -756,11 +969,22 @@ impl Core {
|
||||
// ahead of a sequential reader instead of stalling once it catches up.
|
||||
self.prefetch_ahead(uid, mtime, fsize, first, last, cache_blocks);
|
||||
|
||||
for (i, block) in blocks.into_iter().enumerate() {
|
||||
let bidx = first + i as u64;
|
||||
let bstart = bidx * BLOCK_SIZE;
|
||||
for (span, block) in wanted.into_iter().zip(blocks) {
|
||||
let bstart = span.start;
|
||||
// Every slot is populated: cache hits up front, misses by the fetch above.
|
||||
let block = block.expect("block fetched or cached");
|
||||
// Both sources guarantee the length the geometry says the block has
|
||||
// — the cache validates it on read, the network path in `read_block`
|
||||
// — so this refuses rather than quietly contributing fewer bytes
|
||||
// than the range covers, which is what turned a bad block into a
|
||||
// truncated file (bugs.md B84).
|
||||
if block.len() as u64 != span.len {
|
||||
error!(
|
||||
%uid, bidx = span.idx, len = block.len(), expected = span.len, fsize,
|
||||
"assembling a read hit a short block"
|
||||
);
|
||||
return Err(Errno::EIO);
|
||||
}
|
||||
let s = (offset.max(bstart) - bstart) as usize;
|
||||
let e = (end.min(bstart + block.len() as u64) - bstart) as usize;
|
||||
if s < e {
|
||||
@@ -850,13 +1074,13 @@ impl Core {
|
||||
return None;
|
||||
}
|
||||
let mut out = Vec::with_capacity((end - offset) as usize);
|
||||
for bidx in (offset / BLOCK_SIZE)..=((end - 1) / BLOCK_SIZE) {
|
||||
let ring = self.block_ring.lock().get(uid, mtime, fsize, bidx);
|
||||
for span in self.block_geometry(uid, mtime, fsize).spans(offset, end) {
|
||||
let ring = self.block_ring.lock().get(uid, mtime, fsize, span);
|
||||
let block = match ring {
|
||||
Some(bytes) => bytes,
|
||||
None => Arc::new(self.cache.cached_block(uid, mtime, fsize, bidx)?),
|
||||
None => Arc::new(self.cache.cached_block(uid, mtime, fsize, span)?),
|
||||
};
|
||||
let bstart = bidx * BLOCK_SIZE;
|
||||
let bstart = span.start;
|
||||
let s = (offset.max(bstart) - bstart) as usize;
|
||||
let e = (end.min(bstart + block.len() as u64) - bstart) as usize;
|
||||
if s >= e {
|
||||
@@ -976,31 +1200,42 @@ mod block_ring_tests {
|
||||
RING_BYTES / BLOCK_SIZE
|
||||
}
|
||||
|
||||
/// Block `idx` under the uniform geometry — the shape these tests are
|
||||
/// written against, since what they exercise is tagging and eviction rather
|
||||
/// than block layout.
|
||||
fn span(idx: u64) -> BlockSpan {
|
||||
BlockSpan {
|
||||
idx,
|
||||
start: idx * BLOCK_SIZE,
|
||||
len: BLOCK_SIZE,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_stored_block_is_served_back() {
|
||||
let mut ring = BlockRing::default();
|
||||
let node = uid("a");
|
||||
ring.insert(&node, 7, 100, 3, block(0xab));
|
||||
assert_eq!(ring.get(&node, 7, 100, 3).map(|b| b[0]), Some(0xab));
|
||||
ring.insert(&node, 7, 100, span(3), block(0xab));
|
||||
assert_eq!(ring.get(&node, 7, 100, span(3)).map(|b| b[0]), Some(0xab));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_block_of_another_revision_is_never_served() {
|
||||
let mut ring = BlockRing::default();
|
||||
let node = uid("a");
|
||||
ring.insert(&node, 7, 100, 0, block(1));
|
||||
ring.insert(&node, 7, 100, span(0), block(1));
|
||||
// Same node, newer revision: the tag moved, so the old bytes go.
|
||||
assert!(ring.get(&node, 8, 100, 0).is_none());
|
||||
assert!(ring.get(&node, 7, 101, 0).is_none());
|
||||
assert!(ring.get(&node, 8, 100, span(0)).is_none());
|
||||
assert!(ring.get(&node, 7, 101, span(0)).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_new_revision_drops_what_the_old_one_left() {
|
||||
let mut ring = BlockRing::default();
|
||||
let node = uid("a");
|
||||
ring.insert(&node, 7, 100, 0, block(1));
|
||||
ring.insert(&node, 8, 100, 0, block(2));
|
||||
assert_eq!(ring.get(&node, 8, 100, 0).map(|b| b[0]), Some(2));
|
||||
ring.insert(&node, 7, 100, span(0), block(1));
|
||||
ring.insert(&node, 8, 100, span(0), block(2));
|
||||
assert_eq!(ring.get(&node, 8, 100, span(0)).map(|b| b[0]), Some(2));
|
||||
assert_eq!(ring.bytes, BLOCK_SIZE);
|
||||
}
|
||||
|
||||
@@ -1009,13 +1244,13 @@ mod block_ring_tests {
|
||||
let mut ring = BlockRing::default();
|
||||
let node = uid("a");
|
||||
for idx in 0..capacity() {
|
||||
ring.insert(&node, 1, u64::MAX, idx, block(idx as u8));
|
||||
ring.insert(&node, 1, u64::MAX, span(idx), block(idx as u8));
|
||||
}
|
||||
// Re-reading block 0 makes block 1 the oldest instead.
|
||||
assert!(ring.get(&node, 1, u64::MAX, 0).is_some());
|
||||
ring.insert(&node, 1, u64::MAX, capacity(), block(0xff));
|
||||
assert!(ring.get(&node, 1, u64::MAX, 0).is_some());
|
||||
assert!(ring.get(&node, 1, u64::MAX, 1).is_none());
|
||||
assert!(ring.get(&node, 1, u64::MAX, span(0)).is_some());
|
||||
ring.insert(&node, 1, u64::MAX, span(capacity()), block(0xff));
|
||||
assert!(ring.get(&node, 1, u64::MAX, span(0)).is_some());
|
||||
assert!(ring.get(&node, 1, u64::MAX, span(1)).is_none());
|
||||
assert_eq!(ring.bytes, RING_BYTES);
|
||||
}
|
||||
|
||||
@@ -1023,9 +1258,9 @@ mod block_ring_tests {
|
||||
fn a_node_whose_last_block_ages_out_keeps_no_tag() {
|
||||
let mut ring = BlockRing::default();
|
||||
let (old, new) = (uid("old"), uid("new"));
|
||||
ring.insert(&old, 1, u64::MAX, 0, block(1));
|
||||
ring.insert(&old, 1, u64::MAX, span(0), block(1));
|
||||
for idx in 0..capacity() {
|
||||
ring.insert(&new, 1, u64::MAX, idx, block(2));
|
||||
ring.insert(&new, 1, u64::MAX, span(idx), block(2));
|
||||
}
|
||||
assert!(!ring.tags.contains_key(&old));
|
||||
assert!(ring.tags.contains_key(&new));
|
||||
@@ -1035,11 +1270,11 @@ mod block_ring_tests {
|
||||
fn dropping_a_node_frees_only_its_own_bytes() {
|
||||
let mut ring = BlockRing::default();
|
||||
let (a, b) = (uid("a"), uid("b"));
|
||||
ring.insert(&a, 1, 100, 0, block(1));
|
||||
ring.insert(&b, 1, 100, 0, block(2));
|
||||
ring.insert(&a, 1, 100, span(0), block(1));
|
||||
ring.insert(&b, 1, 100, span(0), block(2));
|
||||
ring.drop_node(&a);
|
||||
assert!(ring.get(&a, 1, 100, 0).is_none());
|
||||
assert_eq!(ring.get(&b, 1, 100, 0).map(|x| x[0]), Some(2));
|
||||
assert!(ring.get(&a, 1, 100, span(0)).is_none());
|
||||
assert_eq!(ring.get(&b, 1, 100, span(0)).map(|x| x[0]), Some(2));
|
||||
assert_eq!(ring.bytes, BLOCK_SIZE);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -609,6 +609,7 @@ impl Core {
|
||||
items.push(SharedItem {
|
||||
uid: uid.to_string(),
|
||||
is_dir: n.is_folder(),
|
||||
modified: n.modification_time,
|
||||
name: n.name,
|
||||
path: self.rel_path_for_uid(&uid).unwrap_or_default(),
|
||||
member_count: members,
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
//! One shutdown signal shared by every long-lived worker in the daemon.
|
||||
//!
|
||||
//! The mount used to end by unmounting and returning, leaving the drain
|
||||
//! workers, the sync engine, the conflict sweep, the online probe and the local
|
||||
//! indexer running on `Core` clones. In a process that exits immediately that is
|
||||
//! invisible; in an in-process remount — which is how the daemon recovers from a
|
||||
//! mount going away underneath it — it leaks a whole generation of threads that
|
||||
//! keep writing to the database and calling the API on behalf of a mount that no
|
||||
//! longer exists (bugs.md B44).
|
||||
//!
|
||||
//! The primitive is deliberately small: a flag plus a condvar. Every loop that
|
||||
//! would otherwise call `thread::sleep` calls [`Shutdown::sleep`] instead, which
|
||||
//! returns early — and reports `false` — the moment [`Shutdown::stop`] is
|
||||
//! called, so teardown does not have to wait out a five-minute sweep interval to
|
||||
//! join a thread.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use parking_lot::{Condvar, Mutex};
|
||||
|
||||
/// The daemon's stop signal. Set once, never cleared: a `Core` that has been
|
||||
/// told to stop belongs to a mount that is going away.
|
||||
#[derive(Default)]
|
||||
pub(crate) struct Shutdown {
|
||||
stopping: Mutex<bool>,
|
||||
changed: Condvar,
|
||||
}
|
||||
|
||||
impl Shutdown {
|
||||
/// Whether teardown has begun. Loops check this where they would otherwise
|
||||
/// start another unit of work.
|
||||
pub(crate) fn is_stopping(&self) -> bool {
|
||||
*self.stopping.lock()
|
||||
}
|
||||
|
||||
/// Begin teardown and wake everything waiting in [`sleep`](Self::sleep).
|
||||
/// Idempotent, because more than one path can reach teardown (a signal, the
|
||||
/// kernel mount ending, a failed startup).
|
||||
pub(crate) fn stop(&self) {
|
||||
let mut stopping = self.stopping.lock();
|
||||
*stopping = true;
|
||||
self.changed.notify_all();
|
||||
}
|
||||
|
||||
/// Wait up to `duration`, or until teardown starts.
|
||||
///
|
||||
/// Returns `false` if the daemon is stopping — including when it was already
|
||||
/// stopping on entry, so `while shutdown.sleep(interval)` is a complete and
|
||||
/// correct loop condition and no separate check is needed at the top.
|
||||
pub(crate) fn sleep(&self, duration: Duration) -> bool {
|
||||
let mut stopping = self.stopping.lock();
|
||||
if !*stopping {
|
||||
self.changed.wait_for(&mut stopping, duration);
|
||||
}
|
||||
!*stopping
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
#[test]
|
||||
fn sleeping_returns_early_once_stopped() {
|
||||
let shutdown = Arc::new(Shutdown::default());
|
||||
assert!(!shutdown.is_stopping());
|
||||
|
||||
let waker = shutdown.clone();
|
||||
std::thread::spawn(move || {
|
||||
std::thread::sleep(Duration::from_millis(20));
|
||||
waker.stop();
|
||||
});
|
||||
|
||||
let started = Instant::now();
|
||||
// An hour's worth of interval, cut short by the stop.
|
||||
assert!(!shutdown.sleep(Duration::from_secs(3600)));
|
||||
assert!(
|
||||
started.elapsed() < Duration::from_secs(5),
|
||||
"stop must interrupt the wait, not wait it out"
|
||||
);
|
||||
assert!(shutdown.is_stopping());
|
||||
// Already stopping: the next wait does not sleep at all.
|
||||
let started = Instant::now();
|
||||
assert!(!shutdown.sleep(Duration::from_secs(3600)));
|
||||
assert!(started.elapsed() < Duration::from_secs(5));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sleeping_runs_to_the_timeout_while_running() {
|
||||
let shutdown = Shutdown::default();
|
||||
assert!(shutdown.sleep(Duration::from_millis(10)));
|
||||
assert!(!shutdown.is_stopping());
|
||||
}
|
||||
}
|
||||
@@ -141,6 +141,11 @@ pub(crate) struct PendingRevision {
|
||||
pub(crate) meta: StagedWrite,
|
||||
}
|
||||
|
||||
/// One directory's entries as `readdir` reports them: `(ino, is_dir, name)` per
|
||||
/// entry, in reply order. The kernel's file-type enum is mapped in at reply
|
||||
/// time so this module stays free of `fuser`.
|
||||
pub(crate) type DirListing = Vec<(u64, bool, String)>;
|
||||
|
||||
/// Mutable inode bookkeeping, guarded by a mutex because fuser drives the
|
||||
/// `Filesystem` trait through `&self`.
|
||||
pub(crate) struct State {
|
||||
@@ -158,6 +163,16 @@ pub(crate) struct State {
|
||||
/// have no entry here.
|
||||
pub(crate) handles: HashMap<u64, u64>,
|
||||
pub(crate) next_fh: u64,
|
||||
/// One frozen listing per open directory handle, taken at `opendir`.
|
||||
///
|
||||
/// `readdir` resumes from a byte offset into the listing it is walking, so
|
||||
/// that listing has to stay the same across the calls of one `getdents`
|
||||
/// sequence. Rebuilding it live meant a create or a trash landing between
|
||||
/// two pages shifted every later entry, and the caller silently skipped or
|
||||
/// repeated whatever crossed the page boundary. Each entry is
|
||||
/// `(ino, is_dir, name)` — the kernel type is mapped at reply time so this
|
||||
/// module stays free of `fuser` (bugs.md B43).
|
||||
pub(crate) dir_snapshots: HashMap<u64, Arc<DirListing>>,
|
||||
/// Resident inode attributes whose effective access changed during an
|
||||
/// intern/root refresh. Core drains this set and notifies the matching
|
||||
/// kernel only after releasing the State lock.
|
||||
@@ -317,6 +332,7 @@ impl State {
|
||||
active_writes: HashMap::new(),
|
||||
handles: HashMap::new(),
|
||||
next_fh: 1,
|
||||
dir_snapshots: HashMap::new(),
|
||||
access_changes: HashSet::new(),
|
||||
share_access,
|
||||
db,
|
||||
|
||||
@@ -67,10 +67,16 @@ impl Core {
|
||||
/// Sweep for stale conflict copies forever, on its own thread. See the module
|
||||
/// docs for the identical/divergent policy.
|
||||
pub(crate) fn run_conflict_sweep_loop(&self) {
|
||||
std::thread::sleep(CONFLICT_SWEEP_WARMUP);
|
||||
// Both waits are interruptible, so teardown does not have to sit out a
|
||||
// five-minute interval to join this thread (bugs.md B44).
|
||||
if !self.shutdown.sleep(CONFLICT_SWEEP_WARMUP) {
|
||||
return;
|
||||
}
|
||||
loop {
|
||||
self.sweep_conflicts_once();
|
||||
std::thread::sleep(CONFLICT_SWEEP_INTERVAL);
|
||||
if !self.shutdown.sleep(CONFLICT_SWEEP_INTERVAL) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+585
-77
@@ -54,7 +54,7 @@ use engine::{classify, settle_with};
|
||||
#[cfg(test)]
|
||||
use local_scan::is_write_mode;
|
||||
use local_scan::open_for_write_set;
|
||||
use model::{LocalItem, Outcome, PassAbort, Pending, RemoteItem};
|
||||
use model::{LocalItem, LocalSig, Outcome, PassAbort, Pending, RemoteItem};
|
||||
pub(crate) use planner::base_name;
|
||||
#[cfg(test)]
|
||||
use planner::conflict_path;
|
||||
@@ -525,6 +525,9 @@ impl Core {
|
||||
continue;
|
||||
}
|
||||
trashed.push(rel.clone());
|
||||
if let Some(parent) = parent_uid_of(rel, &remote_dirs, &baseline) {
|
||||
self.invalidate_children_of(&parent);
|
||||
}
|
||||
let _ = self.db.sync_entry_remove(folder_id, rel);
|
||||
self.log_activity(
|
||||
ActivityKind::Trash,
|
||||
@@ -687,6 +690,7 @@ impl Core {
|
||||
rel_path: rel.clone(),
|
||||
remote_uid: Some(remote.uid.to_string()),
|
||||
local_mtime: local.mtime,
|
||||
local_mtime_ns: local.mtime_ns,
|
||||
local_size: local.size,
|
||||
remote_rev: Some(remote.mtime.to_string()),
|
||||
remote_hash: Some(remote.size.to_string()),
|
||||
@@ -717,6 +721,9 @@ impl Core {
|
||||
uid: remote.uid.clone(),
|
||||
mtime: remote.mtime,
|
||||
size: remote.size,
|
||||
// What the walk saw here, so the apply step can tell a
|
||||
// stale plan from a live one before it overwrites.
|
||||
local: l.map(LocalSig::from),
|
||||
});
|
||||
}
|
||||
FilePlan::Conflict => {
|
||||
@@ -739,7 +746,9 @@ impl Core {
|
||||
}
|
||||
},
|
||||
FilePlan::DeleteLocal => {
|
||||
if let Err(e) = std::fs::remove_file(local_root.join(rel_to_path(rel))) {
|
||||
if let Err(e) =
|
||||
removed_locally(std::fs::remove_file(local_root.join(rel_to_path(rel))))
|
||||
{
|
||||
warn!(rel, error = %e, "sync: remove local file failed");
|
||||
self.log_activity(
|
||||
ActivityKind::Trash,
|
||||
@@ -771,6 +780,9 @@ impl Core {
|
||||
outcome.errors += 1;
|
||||
continue;
|
||||
}
|
||||
if let Some(parent) = parent_uid_of(rel, &remote_dirs, &baseline) {
|
||||
self.invalidate_children_of(&parent);
|
||||
}
|
||||
let _ = self.db.sync_entry_remove(folder_id, rel);
|
||||
self.log_activity(
|
||||
ActivityKind::Trash,
|
||||
@@ -797,7 +809,9 @@ impl Core {
|
||||
// Deferred folder deletions, deepest first.
|
||||
delete_local_dirs.sort_by_key(|p| std::cmp::Reverse(p.matches('/').count()));
|
||||
for rel in delete_local_dirs {
|
||||
if let Err(e) = std::fs::remove_dir_all(local_root.join(rel_to_path(&rel))) {
|
||||
if let Err(e) =
|
||||
removed_locally(std::fs::remove_dir_all(local_root.join(rel_to_path(&rel))))
|
||||
{
|
||||
warn!(rel, error = %e, "sync: remove local folder failed");
|
||||
self.log_activity(ActivityKind::Trash, base_name(&rel), e.to_string(), false);
|
||||
outcome.errors += 1;
|
||||
@@ -824,6 +838,9 @@ impl Core {
|
||||
outcome.errors += 1;
|
||||
continue;
|
||||
}
|
||||
if let Some(parent) = parent_uid_of(&rel, &remote_dirs, &baseline) {
|
||||
self.invalidate_children_of(&parent);
|
||||
}
|
||||
let _ = self.db.sync_entry_remove(folder_id, &rel);
|
||||
self.log_activity(
|
||||
ActivityKind::Trash,
|
||||
@@ -849,68 +866,9 @@ impl Core {
|
||||
writing: &HashSet<PathBuf>,
|
||||
out: &mut HashMap<String, LocalItem>,
|
||||
) -> Result<(), String> {
|
||||
let entries = std::fs::read_dir(dir).map_err(|e| format!("read {}: {e}", dir.display()))?;
|
||||
for entry in entries {
|
||||
let entry = entry.map_err(|e| format!("read entry in {}: {e}", dir.display()))?;
|
||||
let path = entry.path();
|
||||
let meta = std::fs::symlink_metadata(&path)
|
||||
.map_err(|e| format!("stat {}: {e}", path.display()))?;
|
||||
let stripped = path.strip_prefix(root).map_err(|e| {
|
||||
format!(
|
||||
"local path {} escaped sync root {}: {e}",
|
||||
path.display(),
|
||||
root.display()
|
||||
)
|
||||
})?;
|
||||
let rel = stripped.to_str().ok_or_else(|| {
|
||||
format!(
|
||||
"local path {} is not valid UTF-8; refusing a destructive sync pass",
|
||||
path.display()
|
||||
)
|
||||
})?;
|
||||
// Ignore our own in-flight download temp files.
|
||||
if rel.contains(".pdfs-tmp-") {
|
||||
continue;
|
||||
}
|
||||
// Ignored paths are dropped here as well as from the classification
|
||||
// union — not for correctness (the union filter is what makes this
|
||||
// safe) but so an ignored `node_modules/` is never descended into or
|
||||
// counted against scan progress.
|
||||
if rules.is_ignored(rel, meta.is_dir()) {
|
||||
continue;
|
||||
}
|
||||
if meta.is_dir() {
|
||||
out.insert(
|
||||
rel.to_string(),
|
||||
LocalItem {
|
||||
is_dir: true,
|
||||
mtime: 0,
|
||||
size: 0,
|
||||
open_for_write: false,
|
||||
},
|
||||
);
|
||||
self.progress_scanned(folder_id, base_name(rel));
|
||||
self.walk_local(folder_id, root, &path, rules, writing, out)?;
|
||||
} else if meta.is_file() {
|
||||
// Check whether any process holds this file open for writing.
|
||||
// The canonical path is used because /proc/*/fd links resolve
|
||||
// to canonical targets.
|
||||
let ofw = std::fs::canonicalize(&path)
|
||||
.map(|canon| writing.contains(&canon))
|
||||
.unwrap_or(false);
|
||||
out.insert(
|
||||
rel.to_string(),
|
||||
LocalItem {
|
||||
is_dir: false,
|
||||
mtime: system_mtime(&meta),
|
||||
size: meta.len() as i64,
|
||||
open_for_write: ofw,
|
||||
},
|
||||
);
|
||||
self.progress_scanned(folder_id, base_name(rel));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
walk_local_tree(root, dir, rules, writing, out, &mut |rel| {
|
||||
self.progress_scanned(folder_id, base_name(rel))
|
||||
})
|
||||
}
|
||||
|
||||
/// Map every remote folder under `folder` to its uid by rel path, ignoring
|
||||
@@ -1069,6 +1027,7 @@ impl Core {
|
||||
rel_path: rel.to_string(),
|
||||
remote_uid: Some(uid.to_string()),
|
||||
local_mtime: 0,
|
||||
local_mtime_ns: None,
|
||||
local_size: 0,
|
||||
remote_rev: None,
|
||||
remote_hash: None,
|
||||
@@ -1175,6 +1134,9 @@ impl Core {
|
||||
.create_folder(parent, base_name(rel), Some(now_secs()))
|
||||
.await
|
||||
.map_err(|e| format!("create remote folder {rel}: {e}"))?;
|
||||
// The parent has gained a child that no cached listing of it
|
||||
// knows about (B86).
|
||||
self.invalidate_children_of(parent);
|
||||
self.baseline_dir(folder_id, rel, &uid)?;
|
||||
Ok(Applied::Dir(rel.clone(), uid))
|
||||
}
|
||||
@@ -1192,8 +1154,9 @@ impl Core {
|
||||
uid,
|
||||
mtime,
|
||||
size,
|
||||
local,
|
||||
} => {
|
||||
self.download_file(folder_id, local_root, rel, uid, *mtime, *size)
|
||||
self.download_file(folder_id, local_root, rel, uid, *mtime, *size, *local)
|
||||
.await?;
|
||||
Ok(Applied::Downloaded)
|
||||
}
|
||||
@@ -1211,7 +1174,11 @@ impl Core {
|
||||
.map_err(|e| format!("set aside conflict copy for {rel}: {e}"))?;
|
||||
info!(rel, "sync: kept local changes as a conflict copy");
|
||||
}
|
||||
self.download_file(folder_id, local_root, rel, uid, *mtime, *size)
|
||||
// The local copy has just been moved aside, so the state this
|
||||
// download expects to find at the path is "nothing" — anything
|
||||
// that appears while it runs is a *new* racing write, and gets
|
||||
// its own conflict copy rather than being overwritten (B39).
|
||||
self.download_file(folder_id, local_root, rel, uid, *mtime, *size, None)
|
||||
.await?;
|
||||
Ok(Applied::Conflict)
|
||||
}
|
||||
@@ -1250,6 +1217,9 @@ impl Core {
|
||||
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
|
||||
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
|
||||
let mtime = system_mtime(&meta);
|
||||
// Taken from the descriptor being streamed, so it names the content that
|
||||
// actually goes up rather than whatever is at the path when we finish.
|
||||
let streamed = LocalSig::from(&meta);
|
||||
// Count the bytes as they are read, so this upload shows up in
|
||||
// `GetQueueStatus` next to a manual one. The uid isn't known until the
|
||||
// draft is sealed, so the transfer registers without one.
|
||||
@@ -1276,7 +1246,13 @@ impl Core {
|
||||
)
|
||||
.await
|
||||
.map_err(|e| format!("upload {rel}: {e}"))?;
|
||||
self.record_file_baseline(folder_id, rel, &path, &uid).await
|
||||
// The parent has gained a child that no cached listing of it knows
|
||||
// about. Without this, a folder later switched to on-demand serves a
|
||||
// snapshot that predates this upload (B86).
|
||||
self.invalidate_children_of(parent);
|
||||
warn_if_torn(rel, &path, streamed);
|
||||
self.record_file_baseline(folder_id, rel, Some(streamed), &uid)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Upload a changed local file as a new revision of an existing remote node.
|
||||
@@ -1291,6 +1267,7 @@ impl Core {
|
||||
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
|
||||
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
|
||||
let mtime = system_mtime(&meta);
|
||||
let streamed = LocalSig::from(&meta);
|
||||
let reader = OwnedCountingReader::new(
|
||||
file,
|
||||
self.transfers.begin(
|
||||
@@ -1304,11 +1281,20 @@ impl Core {
|
||||
.upload_new_revision_from(uid, reader, meta.len() as i64, Vec::new(), Some(mtime))
|
||||
.await
|
||||
.map_err(|e| format!("upload revision {rel}: {e}"))?;
|
||||
self.record_file_baseline(folder_id, rel, &path, uid).await
|
||||
warn_if_torn(rel, &path, streamed);
|
||||
self.record_file_baseline(folder_id, rel, Some(streamed), uid)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Download a remote file to its local path (atomically via a temp file),
|
||||
/// stamp the local mtime to match the remote, then record baseline.
|
||||
///
|
||||
/// `expected_local` is the identity planning saw at the destination. The
|
||||
/// rename below is the destructive step, and the download in between can run
|
||||
/// for minutes, so the destination is re-examined right before it: a local
|
||||
/// file that no longer matches the plan is set aside as a conflict copy
|
||||
/// instead of being replaced (bugs.md B39).
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
async fn download_file(
|
||||
&self,
|
||||
folder_id: i64,
|
||||
@@ -1317,6 +1303,7 @@ impl Core {
|
||||
uid: &NodeUid,
|
||||
mtime: i64,
|
||||
size: i64,
|
||||
expected_local: Option<LocalSig>,
|
||||
) -> Result<(), String> {
|
||||
let path = local_root.join(rel_to_path(rel));
|
||||
if let Some(parent) = path.parent() {
|
||||
@@ -1351,6 +1338,13 @@ impl Core {
|
||||
out.flush().map_err(|e| format!("flush tmp {rel}: {e}"))?;
|
||||
out.sync_all().map_err(|e| format!("sync tmp {rel}: {e}"))?;
|
||||
}
|
||||
// The last moment at which the destination can still be saved. Anything
|
||||
// other than the state planning classified means a writer got there
|
||||
// first, and its bytes are the ones nothing else has a copy of.
|
||||
if let Err(e) = keep_racing_local_edit(&path, expected_local) {
|
||||
let _ = std::fs::remove_file(&tmp);
|
||||
return Err(format!("set aside racing local edit for {rel}: {e}"));
|
||||
}
|
||||
std::fs::rename(&tmp, &path).map_err(|e| format!("place {rel}: {e}"))?;
|
||||
// Match local mtime to the remote's so neither side looks "changed" next pass.
|
||||
let f = std::fs::File::options()
|
||||
@@ -1366,23 +1360,36 @@ impl Core {
|
||||
.and_then(|dir| dir.sync_all())
|
||||
.map_err(|e| format!("sync directory for {rel}: {e}"))?;
|
||||
}
|
||||
self.record_file_baseline(folder_id, rel, &path, uid).await
|
||||
// Here the file on disk *is* the transferred content: it was published
|
||||
// by a rename and stamped just above, so its stat is the identity.
|
||||
self.record_file_baseline(folder_id, rel, local_sig(&path), uid)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Refresh a file's baseline from ground truth: the local stat and the
|
||||
/// remote node's reported signature. Called after every upload/download so
|
||||
/// the very next reconcile sees the path as unchanged.
|
||||
/// Refresh a file's baseline from ground truth: the local identity these
|
||||
/// bytes came from and the remote node's reported signature. Called after
|
||||
/// every upload/download so the very next reconcile sees the path as
|
||||
/// unchanged.
|
||||
///
|
||||
/// `local` is the identity the *transferred content* corresponds to, which
|
||||
/// is not always the file's current stat. An upload streams the live file,
|
||||
/// so it passes the identity it read the first byte at: if a writer changed
|
||||
/// the file mid-stream, the remote now holds a torn revision, and recording
|
||||
/// the pre-upload identity is what makes the next pass see the path as
|
||||
/// locally changed and upload a whole, clean revision over it. Recording the
|
||||
/// post-upload stat instead declared the torn revision synchronized and left
|
||||
/// it as the file's content on Drive (bugs.md B40).
|
||||
async fn record_file_baseline(
|
||||
&self,
|
||||
folder_id: i64,
|
||||
rel: &str,
|
||||
local_path: &Path,
|
||||
local: Option<LocalSig>,
|
||||
uid: &NodeUid,
|
||||
) -> Result<(), String> {
|
||||
let (lmtime, lsize) = match std::fs::metadata(local_path) {
|
||||
Ok(m) => (system_mtime(&m), m.len() as i64),
|
||||
Err(e) => return Err(format!("stat {rel}: {e}")),
|
||||
let Some(local) = local else {
|
||||
return Err(format!("stat {rel}: no local identity to baseline"));
|
||||
};
|
||||
let (lmtime, lsize) = (local.mtime, local.size);
|
||||
// Re-fetch the node so the stored remote signature is exactly what a walk
|
||||
// will report next time.
|
||||
let (rmtime, rsize) = match self.client.enumerate_nodes(std::slice::from_ref(uid)).await {
|
||||
@@ -1402,6 +1409,7 @@ impl Core {
|
||||
rel_path: rel.to_string(),
|
||||
remote_uid: Some(uid.to_string()),
|
||||
local_mtime: lmtime,
|
||||
local_mtime_ns: local.mtime_ns,
|
||||
local_size: lsize,
|
||||
remote_rev: Some(rmtime.to_string()),
|
||||
remote_hash: Some(rsize.to_string()),
|
||||
@@ -1411,6 +1419,177 @@ impl Core {
|
||||
}
|
||||
}
|
||||
|
||||
/// Say so when a file changed under an upload that was streaming it.
|
||||
///
|
||||
/// Purely diagnostic — the correction is already in place, because the baseline
|
||||
/// records the identity the upload *read* rather than the one on disk now, so
|
||||
/// the next pass classifies the path as locally changed and replaces the torn
|
||||
/// revision. Without the log there is nothing in the journal to explain the
|
||||
/// extra revision (bugs.md B40).
|
||||
fn warn_if_torn(rel: &str, path: &Path, streamed: LocalSig) {
|
||||
let now = local_sig(path);
|
||||
if !now.is_some_and(|now| now.same_content(&streamed)) {
|
||||
warn!(
|
||||
rel,
|
||||
?streamed,
|
||||
?now,
|
||||
"sync: file changed while uploading; the remote revision may be torn \
|
||||
and will be replaced on the next pass"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The identity a reconcile pass uses to recognise a local file it has already
|
||||
/// looked at: `None` when nothing is at the path.
|
||||
fn local_sig(path: &Path) -> Option<LocalSig> {
|
||||
std::fs::metadata(path).ok().map(|m| LocalSig {
|
||||
mtime: system_mtime(&m),
|
||||
mtime_ns: system_mtime_ns(&m),
|
||||
size: m.len() as i64,
|
||||
})
|
||||
}
|
||||
|
||||
/// Guard the destination of a download: if what is at `path` is no longer what
|
||||
/// planning classified, move it aside as a conflict copy before the caller
|
||||
/// renames over it.
|
||||
///
|
||||
/// A reconcile pass decides what to do from a walk that finished before the
|
||||
/// transfer started, and a download of a large file can run for minutes. Without
|
||||
/// this, a local edit completed inside that window was replaced by the remote
|
||||
/// version with no copy of it kept anywhere — the one shape of loss a sync
|
||||
/// engine must not have (bugs.md B39). Keeping the copy makes it re-upload as a
|
||||
/// new file on the next pass, which is the same resolution a both-sides-changed
|
||||
/// conflict already gets.
|
||||
///
|
||||
/// A path that is empty when it was expected to be is not a conflict: the
|
||||
/// download is about to recreate it, and there is nothing to preserve.
|
||||
fn keep_racing_local_edit(path: &Path, expected: Option<LocalSig>) -> std::io::Result<()> {
|
||||
let actual = local_sig(path);
|
||||
let unchanged = match (actual, expected) {
|
||||
(Some(a), Some(e)) => a.same_content(&e),
|
||||
// Nothing there: the download is about to create it, and there is
|
||||
// nothing to preserve either way.
|
||||
(None, _) => true,
|
||||
(Some(_), None) => false,
|
||||
};
|
||||
if unchanged {
|
||||
return Ok(());
|
||||
}
|
||||
warn!(
|
||||
path = %path.display(),
|
||||
?expected,
|
||||
?actual,
|
||||
"sync: local file changed while downloading; keeping it as a conflict copy"
|
||||
);
|
||||
preserve_conflict_copy(path, now_secs()).map(|_| ())
|
||||
}
|
||||
|
||||
/// Read a local removal as the sync engine has to read it: the *state* it was
|
||||
/// asking for is "this path is gone", so a path that was already gone is a
|
||||
/// success, not a failure.
|
||||
///
|
||||
/// It matters because the baseline is only dropped on success. Treating
|
||||
/// `NotFound` as an error left the baseline in place for a file that no longer
|
||||
/// exists, so the same removal was re-attempted and re-reported as an error on
|
||||
/// every pass, forever. Every other error is still an error, and keeping the
|
||||
/// baseline is exactly right there: the surviving file must not be mistaken for
|
||||
/// a new local creation on the next pass and uploaded back (bugs.md B42).
|
||||
fn removed_locally(result: std::io::Result<()>) -> std::io::Result<()> {
|
||||
match result {
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
|
||||
other => other,
|
||||
}
|
||||
}
|
||||
|
||||
/// The body of [`Core::walk_local`], without a `Core`: recursively collect the
|
||||
/// local tree under `dir` into `out`, reporting each entry to `scanned`.
|
||||
///
|
||||
/// **Every failure here is fatal to the pass, deliberately.** A `read_dir` that
|
||||
/// fails, an entry that cannot be stat-ed, a name that is not UTF-8 — each of
|
||||
/// them would, if skipped, leave `out` describing a *smaller* tree than the one
|
||||
/// on disk. Reconciliation reads a path present in the baseline and absent from
|
||||
/// the scan as "the user deleted this", and propagates that to Drive. So an
|
||||
/// unreadable subdirectory would trash its remote counterpart, and the deeper
|
||||
/// the failure, the more it takes with it (`docs/BUGS.md` B55). Refusing the
|
||||
/// whole pass keeps the previous baseline and leaves the remote alone; the next
|
||||
/// pass retries once the transient condition clears.
|
||||
///
|
||||
/// Split out from the method so exactly that can be tested against a real tree
|
||||
/// with a real unreadable directory in it, which needs no `Core` at all.
|
||||
fn walk_local_tree(
|
||||
root: &Path,
|
||||
dir: &Path,
|
||||
rules: &IgnoreRules,
|
||||
writing: &HashSet<PathBuf>,
|
||||
out: &mut HashMap<String, LocalItem>,
|
||||
scanned: &mut dyn FnMut(&str),
|
||||
) -> Result<(), String> {
|
||||
let entries = std::fs::read_dir(dir).map_err(|e| format!("read {}: {e}", dir.display()))?;
|
||||
for entry in entries {
|
||||
let entry = entry.map_err(|e| format!("read entry in {}: {e}", dir.display()))?;
|
||||
let path = entry.path();
|
||||
let meta = std::fs::symlink_metadata(&path)
|
||||
.map_err(|e| format!("stat {}: {e}", path.display()))?;
|
||||
let stripped = path.strip_prefix(root).map_err(|e| {
|
||||
format!(
|
||||
"local path {} escaped sync root {}: {e}",
|
||||
path.display(),
|
||||
root.display()
|
||||
)
|
||||
})?;
|
||||
let rel = stripped.to_str().ok_or_else(|| {
|
||||
format!(
|
||||
"local path {} is not valid UTF-8; refusing a destructive sync pass",
|
||||
path.display()
|
||||
)
|
||||
})?;
|
||||
// Ignore our own in-flight download temp files.
|
||||
if rel.contains(".pdfs-tmp-") {
|
||||
continue;
|
||||
}
|
||||
// Ignored paths are dropped here as well as from the classification
|
||||
// union — not for correctness (the union filter is what makes this
|
||||
// safe) but so an ignored `node_modules/` is never descended into or
|
||||
// counted against scan progress.
|
||||
if rules.is_ignored(rel, meta.is_dir()) {
|
||||
continue;
|
||||
}
|
||||
if meta.is_dir() {
|
||||
out.insert(
|
||||
rel.to_string(),
|
||||
LocalItem {
|
||||
is_dir: true,
|
||||
mtime: 0,
|
||||
mtime_ns: None,
|
||||
size: 0,
|
||||
open_for_write: false,
|
||||
},
|
||||
);
|
||||
scanned(rel);
|
||||
walk_local_tree(root, &path, rules, writing, out, scanned)?;
|
||||
} else if meta.is_file() {
|
||||
// Check whether any process holds this file open for writing.
|
||||
// The canonical path is used because /proc/*/fd links resolve
|
||||
// to canonical targets.
|
||||
let ofw = std::fs::canonicalize(&path)
|
||||
.map(|canon| writing.contains(&canon))
|
||||
.unwrap_or(false);
|
||||
out.insert(
|
||||
rel.to_string(),
|
||||
LocalItem {
|
||||
is_dir: false,
|
||||
mtime: system_mtime(&meta),
|
||||
mtime_ns: system_mtime_ns(&meta),
|
||||
size: meta.len() as i64,
|
||||
open_for_write: ofw,
|
||||
},
|
||||
);
|
||||
scanned(rel);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Publish a conflict copy without overwriting an earlier conflict bearing the
|
||||
/// same timestamp. The original is removed only after the new copy is durable.
|
||||
fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> {
|
||||
@@ -1444,6 +1623,26 @@ fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> {
|
||||
unreachable!("u32 conflict suffix space exhausted")
|
||||
}
|
||||
|
||||
/// The remote uid of `rel`'s parent folder: from the pass's folder-uid map
|
||||
/// first, and from the baseline second for a parent the map never covered.
|
||||
///
|
||||
/// Used to name the folder whose cached listing a remote mutation has just
|
||||
/// invalidated (B86). `None` is not an error — it only means this pass cannot
|
||||
/// say which folder to invalidate, and the caller skips it.
|
||||
fn parent_uid_of(
|
||||
rel: &str,
|
||||
remote_dirs: &HashMap<String, NodeUid>,
|
||||
baseline: &HashMap<String, StoredSyncEntry>,
|
||||
) -> Option<NodeUid> {
|
||||
let parent = parent_rel(rel);
|
||||
remote_dirs.get(parent).cloned().or_else(|| {
|
||||
baseline
|
||||
.get(parent)
|
||||
.and_then(|e| e.remote_uid.as_deref())
|
||||
.and_then(parse_uid)
|
||||
})
|
||||
}
|
||||
|
||||
fn sync_parent_dir(path: &Path) -> std::io::Result<()> {
|
||||
match path.parent() {
|
||||
Some(parent) => std::fs::File::open(parent)?.sync_all(),
|
||||
@@ -1476,10 +1675,77 @@ fn system_mtime(meta: &std::fs::Metadata) -> i64 {
|
||||
.map(|d| d.as_secs() as i64)
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// The same modification time in nanoseconds, when the filesystem reports one.
|
||||
///
|
||||
/// Whole seconds cannot distinguish an edit made in the same second as the last
|
||||
/// sync from no edit at all, and if the size did not change either, that file
|
||||
/// was never uploaded (bugs.md B25). `None` for a clock or filesystem that
|
||||
/// cannot answer, which falls the comparison back to seconds rather than
|
||||
/// asserting an mtime of zero.
|
||||
fn system_mtime_ns(meta: &std::fs::Metadata) -> Option<i64> {
|
||||
meta.modified()
|
||||
.ok()
|
||||
.and_then(|t| t.duration_since(SystemTime::UNIX_EPOCH).ok())
|
||||
.and_then(|d| i64::try_from(d.as_nanos()).ok())
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A removal of something already gone has produced the state the sync
|
||||
/// engine wanted, so its baseline is dropped rather than retried forever.
|
||||
/// Anything else stays an error and keeps the baseline (bugs.md B42).
|
||||
#[test]
|
||||
fn an_already_absent_path_counts_as_removed() {
|
||||
let root = std::env::temp_dir().join(format!("pdfs-b42-{}", std::process::id()));
|
||||
let missing = root.join("never-existed");
|
||||
assert!(removed_locally(std::fs::remove_file(&missing)).is_ok());
|
||||
assert!(removed_locally(std::fs::remove_dir_all(&missing)).is_ok());
|
||||
|
||||
// A directory in the way of `remove_file` is a real failure, and must
|
||||
// stay one: the file survives, and its baseline has to survive with it.
|
||||
let occupied = root.join("a-directory");
|
||||
std::fs::create_dir_all(&occupied).unwrap();
|
||||
assert!(removed_locally(std::fs::remove_file(&occupied)).is_err());
|
||||
let _ = std::fs::remove_dir_all(&root);
|
||||
}
|
||||
|
||||
/// The download guard: a destination that still matches the plan is
|
||||
/// replaced silently, one that has moved on since is kept (bugs.md B39).
|
||||
#[test]
|
||||
fn a_download_keeps_a_local_edit_it_did_not_plan_for() {
|
||||
let root = std::env::temp_dir().join(format!("pdfs-b39-{}", std::process::id()));
|
||||
std::fs::create_dir_all(&root).unwrap();
|
||||
let path = root.join("report.txt");
|
||||
|
||||
// Nothing there, nothing expected: the download simply creates it.
|
||||
assert!(keep_racing_local_edit(&path, None).is_ok());
|
||||
assert!(!path.exists());
|
||||
|
||||
// Unchanged since planning: no copy, the caller overwrites it.
|
||||
std::fs::write(&path, b"planned").unwrap();
|
||||
let planned = local_sig(&path).unwrap();
|
||||
keep_racing_local_edit(&path, Some(planned)).unwrap();
|
||||
assert_eq!(std::fs::read(&path).unwrap(), b"planned");
|
||||
|
||||
// Changed since planning: the racing bytes are the only copy of
|
||||
// themselves, so they must survive the overwrite as a conflict copy.
|
||||
std::fs::write(&path, b"a much longer local edit").unwrap();
|
||||
keep_racing_local_edit(&path, Some(planned)).unwrap();
|
||||
let kept: Vec<_> = std::fs::read_dir(&root)
|
||||
.unwrap()
|
||||
.filter_map(|e| e.ok())
|
||||
.map(|e| std::fs::read(e.path()).unwrap_or_default())
|
||||
.collect();
|
||||
assert!(
|
||||
kept.iter()
|
||||
.any(|bytes| bytes == b"a much longer local edit"),
|
||||
"the racing edit must still exist somewhere: {kept:?}"
|
||||
);
|
||||
let _ = std::fs::remove_dir_all(&root);
|
||||
}
|
||||
|
||||
/// Feed `n` `Reconcile` events `gap` apart on another thread, starting
|
||||
/// immediately. Returns the sender's join handle so the test can keep it
|
||||
/// alive (dropping the sender early would look like a disconnect).
|
||||
@@ -1565,6 +1831,7 @@ mod tests {
|
||||
|
||||
fn local_item(is_dir: bool) -> LocalItem {
|
||||
LocalItem {
|
||||
mtime_ns: None,
|
||||
is_dir,
|
||||
mtime: 1,
|
||||
size: 1,
|
||||
@@ -1722,6 +1989,44 @@ mod tests {
|
||||
assert!(guard_local_wipe(&base, &fresh).is_err());
|
||||
}
|
||||
|
||||
/// B54's remaining case, end to end: a folder whose local root is still
|
||||
/// *readable* but is no longer the tree it was — an empty directory left
|
||||
/// where a mount used to be, or a different device mounted over it.
|
||||
///
|
||||
/// Nothing fails here. The scan succeeds and honestly reports an empty
|
||||
/// directory, so the only thing standing between the user and a trashed
|
||||
/// remote folder is the guard, and it has to fire for a one-file baseline
|
||||
/// exactly as it does for a large one.
|
||||
#[test]
|
||||
fn a_replaced_mountpoint_trips_the_guard_at_every_baseline_size() {
|
||||
let root = sync_test_dir("wipe-replaced-mount");
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut local: HashMap<String, LocalItem> = HashMap::new();
|
||||
let mut progress = no_progress();
|
||||
walk_local_tree(&root, &root, &rules, &writing, &mut local, &mut progress).unwrap();
|
||||
assert!(local.is_empty(), "the replacement root is genuinely empty");
|
||||
|
||||
for size in 1..=3 {
|
||||
let baseline: HashMap<String, ()> =
|
||||
(0..size).map(|i| (format!("file{i}.txt"), ())).collect();
|
||||
assert!(
|
||||
guard_local_wipe(&filter_baseline(&baseline, &rules), &local).is_err(),
|
||||
"a {size}-path baseline against an empty root is not a deletion"
|
||||
);
|
||||
}
|
||||
|
||||
// And the guard steps aside the moment the real tree is back, so a
|
||||
// remounted folder resumes rather than staying wedged.
|
||||
std::fs::write(root.join("file0.txt"), b"back").unwrap();
|
||||
let mut local: HashMap<String, LocalItem> = HashMap::new();
|
||||
walk_local_tree(&root, &root, &rules, &writing, &mut local, &mut progress).unwrap();
|
||||
let baseline: HashMap<String, ()> = [("file0.txt".to_string(), ())].into_iter().collect();
|
||||
assert!(guard_local_wipe(&filter_baseline(&baseline, &rules), &local).is_ok());
|
||||
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
}
|
||||
|
||||
/// A restored folder (features.md 5.2) starts with an empty baseline against
|
||||
/// an empty local directory and a full remote. That must reconcile as
|
||||
/// "download everything" rather than tripping the wipe guard — otherwise
|
||||
@@ -1764,6 +2069,7 @@ mod tests {
|
||||
remote_uid: Some("v~l".into()),
|
||||
local_mtime: 0,
|
||||
local_size: 0,
|
||||
local_mtime_ns: None,
|
||||
remote_rev: Some(mtime.to_string()),
|
||||
remote_hash: Some(size.to_string()),
|
||||
},
|
||||
@@ -1801,6 +2107,7 @@ mod tests {
|
||||
remote_uid: Some("v~l".into()),
|
||||
local_mtime: 0,
|
||||
local_size: 0,
|
||||
local_mtime_ns: None,
|
||||
remote_rev: None,
|
||||
remote_hash: None,
|
||||
},
|
||||
@@ -1815,6 +2122,7 @@ mod tests {
|
||||
remote_uid: Some("v~l".into()),
|
||||
local_mtime: 10,
|
||||
local_size: 20,
|
||||
local_mtime_ns: None,
|
||||
remote_rev: Some("1700".into()),
|
||||
remote_hash: Some("4096".into()),
|
||||
};
|
||||
@@ -1828,6 +2136,7 @@ mod tests {
|
||||
remote_uid: Some("v~l".into()),
|
||||
local_mtime: 0,
|
||||
local_size: 0,
|
||||
local_mtime_ns: None,
|
||||
remote_rev: None,
|
||||
remote_hash: None,
|
||||
};
|
||||
@@ -1863,6 +2172,7 @@ mod tests {
|
||||
uid: uid(),
|
||||
mtime: 0,
|
||||
size: 10,
|
||||
local: None,
|
||||
};
|
||||
assert_eq!(op.kind(), ActivityKind::Download);
|
||||
}
|
||||
@@ -1918,6 +2228,204 @@ mod tests {
|
||||
std::fs::remove_dir_all(dir).unwrap();
|
||||
}
|
||||
|
||||
/// B49's remaining cases. Preservation is the step that makes a download
|
||||
/// safe to publish, so every way it can fail has to leave the local file
|
||||
/// where it was — the download is aborted, not the file replaced.
|
||||
#[test]
|
||||
fn conflict_preservation_failures_all_keep_the_local_file() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
|
||||
// An unwritable directory: the conflict copy cannot be created at all.
|
||||
let dir = sync_test_dir("conflict-eacces");
|
||||
let source = dir.join("notes.txt");
|
||||
std::fs::write(&source, b"the only copy").unwrap();
|
||||
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o500)).unwrap();
|
||||
let denied = preserve_conflict_copy(&source, 1700);
|
||||
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)).unwrap();
|
||||
assert!(denied.is_err(), "a read-only directory must fail the copy");
|
||||
assert_eq!(
|
||||
std::fs::read(&source).unwrap(),
|
||||
b"the only copy",
|
||||
"the local file survives byte-identical"
|
||||
);
|
||||
std::fs::remove_dir_all(&dir).unwrap();
|
||||
|
||||
// A source that vanished between planning and preservation. The pass
|
||||
// must report it rather than publish the download over the gap.
|
||||
let dir = sync_test_dir("conflict-enoent");
|
||||
assert!(preserve_conflict_copy(&dir.join("gone.txt"), 1700).is_err());
|
||||
std::fs::remove_dir_all(&dir).unwrap();
|
||||
|
||||
// A write error partway through the copy — induced by making the
|
||||
// destination name a directory, so `create_new` fails as something
|
||||
// other than `AlreadyExists`.
|
||||
let dir = sync_test_dir("conflict-eisdir");
|
||||
let source = dir.join("notes.txt");
|
||||
std::fs::write(&source, b"still the only copy").unwrap();
|
||||
std::fs::create_dir(conflict_path(&source, 1700)).unwrap();
|
||||
// Suffix 0 is taken by a directory; `create_new` on it reports
|
||||
// `AlreadyExists`, so preservation moves on to suffix 1 and succeeds.
|
||||
// What must not happen is the source being removed without a copy.
|
||||
let preserved = preserve_conflict_copy(&source, 1700).unwrap();
|
||||
assert!(preserved.is_file());
|
||||
assert_eq!(std::fs::read(&preserved).unwrap(), b"still the only copy");
|
||||
assert!(!source.exists());
|
||||
std::fs::remove_dir_all(&dir).unwrap();
|
||||
}
|
||||
|
||||
/// A conflict copy keeps the bytes *and* the metadata that identifies them,
|
||||
/// so the next pass uploads the user's version rather than a file that looks
|
||||
/// freshly written.
|
||||
#[test]
|
||||
fn a_conflict_copy_is_byte_and_mtime_identical() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
|
||||
let dir = sync_test_dir("conflict-identity");
|
||||
let source = dir.join("notes.txt");
|
||||
std::fs::write(&source, b"local edit that must survive").unwrap();
|
||||
let before = std::fs::metadata(&source).unwrap();
|
||||
|
||||
let preserved = preserve_conflict_copy(&source, 1700).unwrap();
|
||||
let after = std::fs::metadata(&preserved).unwrap();
|
||||
|
||||
assert_eq!(
|
||||
std::fs::read(&preserved).unwrap(),
|
||||
b"local edit that must survive"
|
||||
);
|
||||
assert_eq!(after.len(), before.len());
|
||||
assert_eq!(after.modified().unwrap(), before.modified().unwrap());
|
||||
assert_eq!(after.permissions().mode(), before.permissions().mode());
|
||||
std::fs::remove_dir_all(&dir).unwrap();
|
||||
}
|
||||
|
||||
// ---- B55: an incomplete local scan is never a set of deletions ---------
|
||||
|
||||
fn no_progress() -> impl FnMut(&str) {
|
||||
|_: &str| {}
|
||||
}
|
||||
|
||||
/// A directory the pass cannot read must fail the pass. Omitting it would
|
||||
/// hand reconciliation a baseline-minus-subtree, which it reads as the user
|
||||
/// having deleted that subtree and propagates to Drive (B55).
|
||||
#[test]
|
||||
fn an_unreadable_subdirectory_fails_the_scan_instead_of_shrinking_it() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
|
||||
let root = sync_test_dir("scan-eacces");
|
||||
std::fs::write(root.join("keep.txt"), b"a").unwrap();
|
||||
let locked = root.join("locked");
|
||||
std::fs::create_dir(&locked).unwrap();
|
||||
std::fs::write(locked.join("inner.txt"), b"b").unwrap();
|
||||
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o000)).unwrap();
|
||||
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut out = HashMap::new();
|
||||
let mut progress = no_progress();
|
||||
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
|
||||
|
||||
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o700)).unwrap();
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
|
||||
let error = result.expect_err("an unreadable directory must abort the pass");
|
||||
assert!(
|
||||
error.contains("locked"),
|
||||
"the error should name the subtree it could not read: {error}"
|
||||
);
|
||||
}
|
||||
|
||||
/// The same rule one level up: if the sync root itself cannot be read, the
|
||||
/// pass sees nothing, and "nothing" must never mean "delete everything".
|
||||
#[test]
|
||||
fn an_unreadable_root_fails_the_scan() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
|
||||
let root = sync_test_dir("scan-root-eacces");
|
||||
std::fs::write(root.join("keep.txt"), b"a").unwrap();
|
||||
std::fs::set_permissions(&root, std::fs::Permissions::from_mode(0o000)).unwrap();
|
||||
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut out = HashMap::new();
|
||||
let mut progress = no_progress();
|
||||
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
|
||||
|
||||
std::fs::set_permissions(&root, std::fs::Permissions::from_mode(0o700)).unwrap();
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
|
||||
assert!(result.is_err());
|
||||
assert!(out.is_empty());
|
||||
}
|
||||
|
||||
/// A root that is gone entirely — an unmounted device folder, a removed
|
||||
/// directory — is the sharpest version of the same case: every baseline path
|
||||
/// is "missing", and the scan must refuse rather than report an empty tree.
|
||||
#[test]
|
||||
fn an_absent_root_fails_the_scan() {
|
||||
let root = sync_test_dir("scan-absent");
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut out = HashMap::new();
|
||||
let mut progress = no_progress();
|
||||
assert!(walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress).is_err());
|
||||
assert!(out.is_empty());
|
||||
}
|
||||
|
||||
/// A name that is not valid UTF-8 cannot be keyed into the scan, so it
|
||||
/// cannot be compared against the baseline either. Refusing the pass is the
|
||||
/// only safe reading: skipping it would classify the file as deleted.
|
||||
#[test]
|
||||
fn a_non_utf8_name_fails_the_scan() {
|
||||
use std::ffi::OsStr;
|
||||
use std::os::unix::ffi::OsStrExt as _;
|
||||
|
||||
let root = sync_test_dir("scan-non-utf8");
|
||||
std::fs::write(root.join(OsStr::from_bytes(b"bad\xffname")), b"x").unwrap();
|
||||
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut out = HashMap::new();
|
||||
let mut progress = no_progress();
|
||||
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
|
||||
let error = result.expect_err("a name the scan cannot key must abort the pass");
|
||||
assert!(error.contains("UTF-8"), "{error}");
|
||||
}
|
||||
|
||||
/// The complement: a scan that *can* read everything reports everything, so
|
||||
/// the strictness above does not come at the cost of a working pass. A
|
||||
/// symlink is skipped by design rather than by failure, and a dangling one
|
||||
/// stats fine through `symlink_metadata`.
|
||||
#[test]
|
||||
fn a_readable_tree_scans_completely() {
|
||||
let root = sync_test_dir("scan-complete");
|
||||
std::fs::write(root.join("a.txt"), b"aa").unwrap();
|
||||
std::fs::create_dir(root.join("sub")).unwrap();
|
||||
std::fs::write(root.join("sub/b.txt"), b"bbb").unwrap();
|
||||
std::os::unix::fs::symlink("nowhere", root.join("dangling")).unwrap();
|
||||
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut out = HashMap::new();
|
||||
let mut seen: Vec<String> = Vec::new();
|
||||
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut |rel| {
|
||||
seen.push(rel.to_string())
|
||||
});
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
|
||||
result.unwrap();
|
||||
let mut keys: Vec<&str> = out.keys().map(String::as_str).collect();
|
||||
keys.sort_unstable();
|
||||
assert_eq!(keys, vec!["a.txt", "sub", "sub/b.txt"]);
|
||||
assert_eq!(out["a.txt"].size, 2);
|
||||
assert_eq!(out["sub/b.txt"].size, 3);
|
||||
assert!(out["sub"].is_dir);
|
||||
assert_eq!(seen.len(), 3, "every reported entry ticks scan progress");
|
||||
}
|
||||
|
||||
fn sync_test_dir(label: &str) -> PathBuf {
|
||||
let nonce = SystemTime::now()
|
||||
.duration_since(SystemTime::UNIX_EPOCH)
|
||||
|
||||
@@ -32,17 +32,24 @@ pub(crate) enum SyncMsg {
|
||||
ReconcileAll,
|
||||
/// Re-read the folder set and adjust the filesystem watches (after add/remove).
|
||||
Rewatch,
|
||||
/// The daemon is tearing down: finish the current pass and end the engine
|
||||
/// thread, so the mount can join it (bugs.md B44).
|
||||
Stop,
|
||||
}
|
||||
|
||||
/// Start the sync engine: a control thread that owns the filesystem watcher and
|
||||
/// serialises reconcile passes, plus a periodic poll thread. The engine receives
|
||||
/// on `rx`; senders live in [`Core::sync_tx`].
|
||||
pub(crate) fn spawn(core: Core, rx: Receiver<SyncMsg>) {
|
||||
if let Err(e) = std::thread::Builder::new()
|
||||
pub(crate) fn spawn(core: Core, rx: Receiver<SyncMsg>) -> Option<std::thread::JoinHandle<()>> {
|
||||
match std::thread::Builder::new()
|
||||
.name("pdfs-sync".into())
|
||||
.spawn(move || engine_loop(core, rx))
|
||||
{
|
||||
warn!(error = %e, "failed to start sync engine");
|
||||
Ok(handle) => Some(handle),
|
||||
Err(e) => {
|
||||
warn!(error = %e, "failed to start sync engine");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -74,11 +81,11 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
|
||||
// Periodic remote poll.
|
||||
{
|
||||
let tx = core.sync_tx.clone();
|
||||
let shutdown = core.shutdown.clone();
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-sync-poll".into())
|
||||
.spawn(move || {
|
||||
loop {
|
||||
std::thread::sleep(POLL_INTERVAL);
|
||||
while shutdown.sleep(POLL_INTERVAL) {
|
||||
if tx.send(SyncMsg::ReconcileAll).is_err() {
|
||||
break;
|
||||
}
|
||||
@@ -94,8 +101,12 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
|
||||
let mut ids: HashSet<i64> = HashSet::new();
|
||||
let mut all = false;
|
||||
let mut do_rewatch = false;
|
||||
classify(msg, &mut ids, &mut all, &mut do_rewatch);
|
||||
settle(&rx, &mut ids, &mut all, &mut do_rewatch);
|
||||
if !classify(msg, &mut ids, &mut all, &mut do_rewatch)
|
||||
|| !settle(&rx, &mut ids, &mut all, &mut do_rewatch)
|
||||
{
|
||||
info!("sync engine stopping");
|
||||
return;
|
||||
}
|
||||
|
||||
if do_rewatch {
|
||||
rewatch(&core, &mut watcher, &watched);
|
||||
@@ -116,9 +127,9 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
|
||||
fn poll_only_loop(core: &Core, rx: Receiver<SyncMsg>) {
|
||||
{
|
||||
let tx = core.sync_tx.clone();
|
||||
let shutdown = core.shutdown.clone();
|
||||
std::thread::spawn(move || {
|
||||
loop {
|
||||
std::thread::sleep(POLL_INTERVAL);
|
||||
while shutdown.sleep(POLL_INTERVAL) {
|
||||
if tx.send(SyncMsg::ReconcileAll).is_err() {
|
||||
break;
|
||||
}
|
||||
@@ -130,8 +141,12 @@ fn poll_only_loop(core: &Core, rx: Receiver<SyncMsg>) {
|
||||
let mut ids: HashSet<i64> = HashSet::new();
|
||||
let mut all = false;
|
||||
let mut do_rewatch = false;
|
||||
classify(msg, &mut ids, &mut all, &mut do_rewatch);
|
||||
settle(&rx, &mut ids, &mut all, &mut do_rewatch);
|
||||
if !classify(msg, &mut ids, &mut all, &mut do_rewatch)
|
||||
|| !settle(&rx, &mut ids, &mut all, &mut do_rewatch)
|
||||
{
|
||||
info!("sync engine stopping");
|
||||
return;
|
||||
}
|
||||
if all {
|
||||
reconcile_all(core);
|
||||
} else {
|
||||
@@ -158,7 +173,7 @@ pub(super) fn settle_with(
|
||||
do_rewatch: &mut bool,
|
||||
quiet: Duration,
|
||||
cap: Duration,
|
||||
) {
|
||||
) -> bool {
|
||||
let deadline = Instant::now() + cap;
|
||||
loop {
|
||||
let now = Instant::now();
|
||||
@@ -170,28 +185,49 @@ pub(super) fn settle_with(
|
||||
match rx.recv_timeout(wait) {
|
||||
// Another event: the burst is still going, so the quiet window
|
||||
// starts again from here.
|
||||
Ok(m) => classify(m, ids, all, do_rewatch),
|
||||
// A stop lands mid-burst: give up the rest of the burst rather
|
||||
// than settling for it, and let the caller end the loop.
|
||||
Ok(m) => {
|
||||
if !classify(m, ids, all, do_rewatch) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
// Quiet for a full window: whatever was being written has stopped.
|
||||
Err(RecvTimeoutError::Timeout) => break,
|
||||
// Every sender is gone; the caller's loop is about to end anyway.
|
||||
Err(RecvTimeoutError::Disconnected) => break,
|
||||
}
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
/// [`settle_with`] at the real timings.
|
||||
fn settle(rx: &Receiver<SyncMsg>, ids: &mut HashSet<i64>, all: &mut bool, do_rewatch: &mut bool) {
|
||||
settle_with(rx, ids, all, do_rewatch, DEBOUNCE, MAX_SETTLE);
|
||||
fn settle(
|
||||
rx: &Receiver<SyncMsg>,
|
||||
ids: &mut HashSet<i64>,
|
||||
all: &mut bool,
|
||||
do_rewatch: &mut bool,
|
||||
) -> bool {
|
||||
settle_with(rx, ids, all, do_rewatch, DEBOUNCE, MAX_SETTLE)
|
||||
}
|
||||
|
||||
pub(super) fn classify(msg: SyncMsg, ids: &mut HashSet<i64>, all: &mut bool, rewatch: &mut bool) {
|
||||
/// Fold one message into the pending work of the current burst. Returns `false`
|
||||
/// for [`SyncMsg::Stop`], which is not work but an instruction to end the loop.
|
||||
pub(super) fn classify(
|
||||
msg: SyncMsg,
|
||||
ids: &mut HashSet<i64>,
|
||||
all: &mut bool,
|
||||
rewatch: &mut bool,
|
||||
) -> bool {
|
||||
match msg {
|
||||
SyncMsg::Reconcile(id) => {
|
||||
ids.insert(id);
|
||||
}
|
||||
SyncMsg::ReconcileAll => *all = true,
|
||||
SyncMsg::Rewatch => *rewatch = true,
|
||||
SyncMsg::Stop => return false,
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
/// Reconcile every mirror folder in turn.
|
||||
|
||||
@@ -2,9 +2,73 @@
|
||||
|
||||
use super::*;
|
||||
|
||||
/// What a reconcile pass remembers about the local side of one path: enough to
|
||||
/// recognise the same content again, and no more.
|
||||
///
|
||||
/// `mtime_ns` is the whole reason this is a struct rather than a pair — see
|
||||
/// [`StoredSyncEntry::local_mtime_ns`](pdfs_core::db::StoredSyncEntry).
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(super) struct LocalSig {
|
||||
pub(super) mtime: i64,
|
||||
pub(super) mtime_ns: Option<i64>,
|
||||
pub(super) size: i64,
|
||||
}
|
||||
|
||||
impl LocalSig {
|
||||
/// Whether these two observations describe the same local content.
|
||||
///
|
||||
/// Nanoseconds decide it when both sides have them; otherwise the comparison
|
||||
/// falls back to whole seconds, which is what a baseline row written before
|
||||
/// schema 28 — or a filesystem that reports no sub-second time — can support
|
||||
/// (bugs.md B25).
|
||||
pub(super) fn same_content(&self, other: &Self) -> bool {
|
||||
if self.size != other.size {
|
||||
return false;
|
||||
}
|
||||
match (self.mtime_ns, other.mtime_ns) {
|
||||
(Some(a), Some(b)) => a == b,
|
||||
_ => self.mtime == other.mtime,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<&LocalItem> for LocalSig {
|
||||
fn from(item: &LocalItem) -> Self {
|
||||
Self {
|
||||
mtime: item.mtime,
|
||||
mtime_ns: item.mtime_ns,
|
||||
size: item.size,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<&std::fs::Metadata> for LocalSig {
|
||||
fn from(meta: &std::fs::Metadata) -> Self {
|
||||
Self {
|
||||
mtime: system_mtime(meta),
|
||||
mtime_ns: system_mtime_ns(meta),
|
||||
size: meta.len() as i64,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The baseline's record of the local side, as [`LocalSig`] compares it.
|
||||
impl From<&StoredSyncEntry> for LocalSig {
|
||||
fn from(entry: &StoredSyncEntry) -> Self {
|
||||
Self {
|
||||
mtime: entry.local_mtime,
|
||||
mtime_ns: entry.local_mtime_ns,
|
||||
size: entry.local_size,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) struct LocalItem {
|
||||
pub(super) is_dir: bool,
|
||||
pub(super) mtime: i64,
|
||||
/// [`LocalItem::mtime`] to nanosecond precision, when the filesystem
|
||||
/// reports one.
|
||||
pub(super) mtime_ns: Option<i64>,
|
||||
pub(super) size: i64,
|
||||
/// True when another process holds this file open for writing (detected via
|
||||
/// `/proc/*/fd`). An open-for-write file is kept in the map — so it is not
|
||||
@@ -91,11 +155,18 @@ pub(super) enum Pending {
|
||||
UploadRevision { rel: String, uid: NodeUid },
|
||||
/// Download remote `uid` to the local path, stamping `mtime`. `size` is the
|
||||
/// remote's reported size, used as the transfer's expected total.
|
||||
///
|
||||
/// `local` is what the planning walk saw at this path, or `None` if nothing
|
||||
/// was there. A download can run for minutes, so the apply step revalidates
|
||||
/// against it immediately before it renames over the destination: a writer
|
||||
/// that finished an edit in the meantime must not be overwritten without a
|
||||
/// conflict copy (bugs.md B39).
|
||||
Download {
|
||||
rel: String,
|
||||
uid: NodeUid,
|
||||
mtime: i64,
|
||||
size: i64,
|
||||
local: Option<LocalSig>,
|
||||
},
|
||||
/// Both sides changed: set the local copy aside, then download remote `uid`.
|
||||
Conflict {
|
||||
|
||||
@@ -218,7 +218,10 @@ pub(super) fn plan_file(
|
||||
}
|
||||
|
||||
let local_changed = local.is_some_and(|item| {
|
||||
baseline.is_none_or(|base| base.local_mtime != item.mtime || base.local_size != item.size)
|
||||
// Nanoseconds where both sides have them (bugs.md B25): whole seconds
|
||||
// could not see an edit made in the same second as the last sync that
|
||||
// left the file the same length, so it was never uploaded.
|
||||
baseline.is_none_or(|base| !LocalSig::from(item).same_content(&LocalSig::from(base)))
|
||||
});
|
||||
let remote_changed = remote.is_some_and(|item| {
|
||||
baseline.is_none_or(|base| remote_sig(base) != Some((item.mtime, item.size)))
|
||||
@@ -252,6 +255,7 @@ mod file_plan_tests {
|
||||
|
||||
fn local(mtime: i64, size: i64) -> LocalItem {
|
||||
LocalItem {
|
||||
mtime_ns: None,
|
||||
is_dir: false,
|
||||
mtime,
|
||||
size,
|
||||
@@ -278,12 +282,49 @@ mod file_plan_tests {
|
||||
rel_path: "file".into(),
|
||||
remote_uid: Some("v~l".into()),
|
||||
local_mtime,
|
||||
local_mtime_ns: None,
|
||||
local_size,
|
||||
remote_rev: Some(remote_mtime.to_string()),
|
||||
remote_hash: Some(remote_size.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Whole seconds cannot see an edit made inside the same second that leaves the
|
||||
/// file the same length, so a mirror silently stopped uploading it. With a
|
||||
/// nanosecond time on both sides it is an ordinary local change (bugs.md B25).
|
||||
#[test]
|
||||
fn a_same_second_same_size_edit_is_still_a_local_change() {
|
||||
let mut base = baseline(10, 20, 30, 40);
|
||||
base.local_mtime_ns = Some(10_000_000_000);
|
||||
let edited = LocalItem {
|
||||
mtime_ns: Some(10_400_000_000),
|
||||
..local(10, 20)
|
||||
};
|
||||
let remote = remote(30, 40);
|
||||
assert_eq!(
|
||||
plan_file(Some(&edited), Some(&remote), Some(&base)),
|
||||
FilePlan::UploadRevision
|
||||
);
|
||||
|
||||
// The same file untouched still reads as unchanged.
|
||||
let same = LocalItem {
|
||||
mtime_ns: Some(10_000_000_000),
|
||||
..local(10, 20)
|
||||
};
|
||||
assert_eq!(
|
||||
plan_file(Some(&same), Some(&remote), Some(&base)),
|
||||
FilePlan::Unchanged
|
||||
);
|
||||
|
||||
// And a row from before the column exists compares on seconds, exactly
|
||||
// as it did when it was written — no mass re-upload on upgrade.
|
||||
let old = baseline(10, 20, 30, 40);
|
||||
assert_eq!(
|
||||
plan_file(Some(&edited), Some(&remote), Some(&old)),
|
||||
FilePlan::Unchanged
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn file_plan_covers_every_presence_and_change_shape() {
|
||||
let same_local = local(10, 20);
|
||||
|
||||
@@ -0,0 +1,632 @@
|
||||
//! Importing a Google Photos Takeout export into Proton Photos.
|
||||
//!
|
||||
//! [`pdfs_core::takeout`] says what is in the archives; this is the part that
|
||||
//! talks to Proton. One import runs on its own thread and moves through four
|
||||
//! phases, reporting into the transfer registry as a job so the GUI's transfer
|
||||
//! list shows it without a bespoke widget:
|
||||
//!
|
||||
//! 1. **Scan** — read the archives' central directories and sidecars.
|
||||
//! 2. **Deduplicate against the account** — ask the photos volume which of the
|
||||
//! names already exist ([`ProtonPhotosClient::name_collisions`]), hash only
|
||||
//! those files, then confirm with the content digest. A library that is
|
||||
//! already half-synced by Proton Photos on a phone therefore costs one cheap
|
||||
//! request per 150 names and reads no media at all for the misses.
|
||||
//! 3. **Upload** the survivors, streaming each entry straight out of its zip —
|
||||
//! no extraction to disk, so the import needs no scratch space.
|
||||
//! 4. **Albums** — create the albums the export names that the account lacks,
|
||||
//! then add every photo of each album, the ones that were already on the
|
||||
//! account included.
|
||||
//!
|
||||
//! Re-running an interrupted import is the resume story: phase 2 finds
|
||||
//! everything the previous run uploaded and skips it. Nothing is persisted
|
||||
//! between runs.
|
||||
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::io::Read;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
use pdfs_core::control::{ActivityKind, ImportSummary, TransferDirection};
|
||||
use pdfs_core::takeout::{Bucket, TakeoutPhoto, TakeoutScan, scan_archives};
|
||||
use pdfs_core::{CoreError, CoreResult};
|
||||
use proton_drive_rs::proton_sdk::ids::NodeUid;
|
||||
use proton_drive_rs::{PhotoTag, PhotoTagsUpdate, PhotoUploadMetadata};
|
||||
use sha1::{Digest, Sha1};
|
||||
use tracing::{info, warn};
|
||||
|
||||
use super::Core;
|
||||
use crate::transfers::CountingReader;
|
||||
|
||||
/// The one import this daemon may be running, and how the last one ended.
|
||||
///
|
||||
/// Process-global rather than a [`Core`] field because it must be *one* import
|
||||
/// per daemon however many `Core` clones exist (the forked on-demand mounts each
|
||||
/// hold one), and because a front-end asking for the result after the import
|
||||
/// finished must still find it.
|
||||
static IMPORT: std::sync::LazyLock<parking_lot::Mutex<ImportState>> =
|
||||
std::sync::LazyLock::new(|| parking_lot::Mutex::new(ImportState::default()));
|
||||
|
||||
#[derive(Default)]
|
||||
struct ImportState {
|
||||
running: bool,
|
||||
cancel: Option<Arc<AtomicBool>>,
|
||||
last: Option<ImportSummary>,
|
||||
}
|
||||
|
||||
/// Whether an import is running, and the last one's report.
|
||||
pub(crate) fn import_status() -> (bool, Option<ImportSummary>) {
|
||||
let state = IMPORT.lock();
|
||||
(state.running, state.last.clone())
|
||||
}
|
||||
|
||||
/// Ask the running import to stop. False when none is running.
|
||||
pub(crate) fn cancel_import() -> bool {
|
||||
match &IMPORT.lock().cancel {
|
||||
Some(flag) => {
|
||||
flag.store(true, Ordering::Relaxed);
|
||||
true
|
||||
}
|
||||
None => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// One photo to import, after the copies Google keeps per album are folded into
|
||||
/// a single upload that joins several albums.
|
||||
struct Planned {
|
||||
photo: TakeoutPhoto,
|
||||
/// Every album this photo belongs to, by export title.
|
||||
albums: Vec<String>,
|
||||
/// Filled in phase 2/3: where the photo ended up on the account.
|
||||
uid: Option<NodeUid>,
|
||||
}
|
||||
|
||||
/// Fold the scan into one entry per distinct photo.
|
||||
///
|
||||
/// Google stores a full copy of a photo in every album folder it appears in, so
|
||||
/// the same file is in the export two or three times over. They are folded on
|
||||
/// `(name, size)`: a byte-identical copy under the same name is the same photo,
|
||||
/// and the album memberships of the copies are unioned onto the survivor. Two
|
||||
/// genuinely different photos sharing both a name and an exact size would be
|
||||
/// merged — vanishingly unlikely, and the cost of being wrong is one photo
|
||||
/// filed under an extra album rather than lost data.
|
||||
fn plan(scan: &TakeoutScan) -> Vec<Planned> {
|
||||
let mut order: Vec<(String, u64)> = Vec::new();
|
||||
let mut by_key: HashMap<(String, u64), Planned> = HashMap::new();
|
||||
|
||||
for photo in &scan.photos {
|
||||
let key = (photo.name.clone(), photo.size);
|
||||
let entry = by_key.entry(key.clone()).or_insert_with(|| {
|
||||
order.push(key.clone());
|
||||
Planned {
|
||||
photo: photo.clone(),
|
||||
albums: Vec::new(),
|
||||
uid: None,
|
||||
}
|
||||
});
|
||||
if let Bucket::Album(title) = &photo.bucket
|
||||
&& !entry.albums.iter().any(|existing| existing == title)
|
||||
{
|
||||
entry.albums.push(title.clone());
|
||||
}
|
||||
// A copy that carries a capture time wins over one that does not: the
|
||||
// album copy and the year-folder copy do not always both have a sidecar.
|
||||
if entry.photo.capture_time.is_none() && photo.capture_time.is_some() {
|
||||
entry.photo.capture_time = photo.capture_time;
|
||||
}
|
||||
entry.photo.favorite |= photo.favorite;
|
||||
}
|
||||
|
||||
order
|
||||
.into_iter()
|
||||
.filter_map(|key| by_key.remove(&key))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Open one archive and stream `entry` through `sink`.
|
||||
fn read_entry<R>(
|
||||
archive: &mut zip::ZipArchive<std::io::BufReader<std::fs::File>>,
|
||||
entry: &str,
|
||||
sink: impl FnOnce(&mut (dyn Read + Send)) -> R,
|
||||
) -> CoreResult<R> {
|
||||
let mut file = archive
|
||||
.by_name(entry)
|
||||
.map_err(|e| CoreError::invalid(format!("{entry}: {e}")))?;
|
||||
Ok(sink(&mut file))
|
||||
}
|
||||
|
||||
impl Core {
|
||||
/// Import Google Photos Takeout archives. Blocking; run on its own thread.
|
||||
///
|
||||
/// `cancel` stops the run at the next photo boundary — an upload already on
|
||||
/// the wire is allowed to finish, so a cancelled import never leaves a
|
||||
/// half-written revision behind.
|
||||
pub(crate) fn import_takeout(
|
||||
&self,
|
||||
archives: Vec<PathBuf>,
|
||||
dry_run: bool,
|
||||
cancel: Arc<AtomicBool>,
|
||||
) -> CoreResult<ImportSummary> {
|
||||
let job = self.transfers.begin_job("Importing Google Photos");
|
||||
job.detail("reading archives");
|
||||
|
||||
let scan = scan_archives(&archives)?;
|
||||
let mut planned = plan(&scan);
|
||||
let mut summary = ImportSummary {
|
||||
found: planned.len(),
|
||||
skipped_trashed: scan.skipped_trashed,
|
||||
..Default::default()
|
||||
};
|
||||
job.set_total(planned.len() as u64);
|
||||
info!(
|
||||
archives = archives.len(),
|
||||
photos = summary.found,
|
||||
albums = scan.albums.len(),
|
||||
trashed = summary.skipped_trashed,
|
||||
"takeout scan complete"
|
||||
);
|
||||
|
||||
let photos = self.photos();
|
||||
if self
|
||||
.rt
|
||||
.block_on(photos.get_photos_root())
|
||||
.map_err(|e| CoreError::from_api(&e, "resolve photos library"))?
|
||||
.is_none()
|
||||
{
|
||||
return Err(CoreError::invalid(
|
||||
"this account has no Proton Photos library yet; open Proton Photos once to create it, then import again",
|
||||
));
|
||||
}
|
||||
|
||||
// Phase 2: which of these are already on the account.
|
||||
job.detail("checking for photos you already have");
|
||||
self.mark_existing(&mut planned, &archives, &mut summary, &cancel)?;
|
||||
|
||||
if dry_run {
|
||||
summary.duplicates = planned.iter().filter(|p| p.uid.is_some()).count();
|
||||
return Ok(summary);
|
||||
}
|
||||
|
||||
// Phase 3: upload what is left, one archive at a time so each zip's
|
||||
// central directory is parsed once.
|
||||
self.upload_planned(&mut planned, &archives, &mut summary, &cancel, &job)?;
|
||||
|
||||
// Phase 4: albums. Attempted even after a cancel — the photos that did
|
||||
// upload should still land in their albums.
|
||||
if !scan.albums.is_empty() {
|
||||
job.detail("sorting photos into albums");
|
||||
if let Err(e) = self.link_albums(&planned, &scan.albums, &mut summary) {
|
||||
warn!(error = %e, "takeout album linking failed");
|
||||
}
|
||||
}
|
||||
|
||||
self.spawn_timeline_refresh();
|
||||
summary.cancelled = cancel.load(Ordering::Relaxed);
|
||||
Ok(summary)
|
||||
}
|
||||
|
||||
/// Fill in `uid` for every planned photo the account already holds.
|
||||
///
|
||||
/// Two passes on purpose: the name-only query is one request per 150 photos
|
||||
/// and needs no file contents, so the expensive SHA-1 pass only runs for the
|
||||
/// handful of names that actually collide.
|
||||
fn mark_existing(
|
||||
&self,
|
||||
planned: &mut [Planned],
|
||||
archives: &[PathBuf],
|
||||
summary: &mut ImportSummary,
|
||||
cancel: &AtomicBool,
|
||||
) -> CoreResult<()> {
|
||||
let photos = self.photos();
|
||||
let names: Vec<String> = planned.iter().map(|p| p.photo.name.clone()).collect();
|
||||
let collisions = self
|
||||
.rt
|
||||
.block_on(photos.name_collisions(&names))
|
||||
.map_err(|e| CoreError::from_api(&e, "check for existing photos"))?;
|
||||
|
||||
// Hash the colliding files, grouped by archive so each zip opens once.
|
||||
let mut by_archive: BTreeMap<usize, Vec<usize>> = BTreeMap::new();
|
||||
for (index, planned) in planned.iter().enumerate() {
|
||||
if collisions.get(index).copied().unwrap_or(false) {
|
||||
by_archive
|
||||
.entry(planned.photo.archive)
|
||||
.or_default()
|
||||
.push(index);
|
||||
}
|
||||
}
|
||||
|
||||
// Kept as a pair list, not a map: the duplicate lookup answers in input
|
||||
// order, so the order these were built in is what maps answers back to
|
||||
// planned photos.
|
||||
let mut digests: Vec<(usize, String)> = Vec::new();
|
||||
for (archive_index, indices) in by_archive {
|
||||
if cancel.load(Ordering::Relaxed) {
|
||||
break;
|
||||
}
|
||||
let mut archive = open_archive(archives, archive_index)?;
|
||||
for index in indices {
|
||||
let entry = planned[index].photo.entry.clone();
|
||||
match read_entry(&mut archive, &entry, sha1_of) {
|
||||
Ok(Ok(digest)) => digests.push((index, digest)),
|
||||
Ok(Err(e)) | Err(e) => {
|
||||
warn!(entry, error = %e, "cannot hash takeout entry");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if digests.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let items: Vec<(String, String)> = digests
|
||||
.iter()
|
||||
.map(|(index, digest)| (planned[*index].photo.name.clone(), digest.clone()))
|
||||
.collect();
|
||||
let matches = self
|
||||
.rt
|
||||
.block_on(photos.find_duplicates_many(&items))
|
||||
.map_err(|e| CoreError::from_api(&e, "check for existing photos"))?;
|
||||
|
||||
for (slot, (index, _)) in digests.iter().enumerate() {
|
||||
if let Some(uid) = matches.get(slot).and_then(|uids| uids.first()) {
|
||||
planned[*index].uid = Some(uid.clone());
|
||||
summary.duplicates += 1;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Upload every planned photo that is not already on the account.
|
||||
fn upload_planned(
|
||||
&self,
|
||||
planned: &mut [Planned],
|
||||
archives: &[PathBuf],
|
||||
summary: &mut ImportSummary,
|
||||
cancel: &AtomicBool,
|
||||
job: &crate::transfers::JobGuard,
|
||||
) -> CoreResult<()> {
|
||||
let photos = self.photos();
|
||||
let mut favorites: Vec<NodeUid> = Vec::new();
|
||||
|
||||
let mut by_archive: BTreeMap<usize, Vec<usize>> = BTreeMap::new();
|
||||
for (index, item) in planned.iter().enumerate() {
|
||||
if item.uid.is_none() {
|
||||
by_archive
|
||||
.entry(item.photo.archive)
|
||||
.or_default()
|
||||
.push(index);
|
||||
}
|
||||
}
|
||||
|
||||
for (archive_index, indices) in by_archive {
|
||||
if cancel.load(Ordering::Relaxed) || self.shutdown.is_stopping() {
|
||||
break;
|
||||
}
|
||||
let mut archive = open_archive(archives, archive_index)?;
|
||||
|
||||
for index in indices {
|
||||
if cancel.load(Ordering::Relaxed) || self.shutdown.is_stopping() {
|
||||
break;
|
||||
}
|
||||
let photo = planned[index].photo.clone();
|
||||
job.detail(photo.name.clone());
|
||||
|
||||
let guard = self.transfers.begin(
|
||||
photo.name.clone(),
|
||||
"",
|
||||
TransferDirection::Upload,
|
||||
photo.size,
|
||||
);
|
||||
let metadata = PhotoUploadMetadata {
|
||||
capture_time: photo.capture_time,
|
||||
..Default::default()
|
||||
};
|
||||
let uploaded = read_entry(&mut archive, &photo.entry, |reader| {
|
||||
let counting = CountingReader::new(reader, &guard);
|
||||
self.rt.block_on(photos.upload_photo_from(
|
||||
&photo.name,
|
||||
&photo.media_type,
|
||||
counting,
|
||||
photo.size as i64,
|
||||
Vec::new(),
|
||||
metadata,
|
||||
false,
|
||||
))
|
||||
});
|
||||
drop(guard);
|
||||
job.step();
|
||||
|
||||
match uploaded {
|
||||
Ok(Ok(uid)) => {
|
||||
summary.uploaded += 1;
|
||||
summary.bytes += photo.size;
|
||||
if photo.favorite {
|
||||
favorites.push(uid.clone());
|
||||
}
|
||||
planned[index].uid = Some(uid);
|
||||
}
|
||||
Ok(Err(e)) => {
|
||||
summary.failed += 1;
|
||||
warn!(name = photo.name, error = %e, "takeout photo upload failed");
|
||||
}
|
||||
Err(e) => {
|
||||
summary.failed += 1;
|
||||
warn!(name = photo.name, error = %e, "cannot read takeout entry");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Favourites are a separate call: the upload seal takes classification
|
||||
// tags, but `Favorite` is not one of them — it goes through the photos
|
||||
// volume's dedicated favourite endpoint.
|
||||
if !favorites.is_empty() {
|
||||
let updates: Vec<PhotoTagsUpdate> = favorites
|
||||
.into_iter()
|
||||
.map(|node_uid| PhotoTagsUpdate {
|
||||
node_uid,
|
||||
tags_to_add: vec![PhotoTag::Favorite],
|
||||
tags_to_remove: Vec::new(),
|
||||
})
|
||||
.collect();
|
||||
if let Err(e) = self.rt.block_on(photos.update_photos(&updates)) {
|
||||
warn!(error = %e, "marking imported favourites failed");
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Create the albums the export names that the account lacks, then add each
|
||||
/// album's photos to it.
|
||||
fn link_albums(
|
||||
&self,
|
||||
planned: &[Planned],
|
||||
albums: &[String],
|
||||
summary: &mut ImportSummary,
|
||||
) -> CoreResult<()> {
|
||||
let photos = self.photos();
|
||||
|
||||
// Existing albums by decrypted name, so a second import of the same
|
||||
// export adds to the album it made the first time instead of a twin.
|
||||
let mut existing: HashMap<String, NodeUid> = HashMap::new();
|
||||
let uids = self
|
||||
.rt
|
||||
.block_on(photos.enumerate_album_node_uids())
|
||||
.map_err(|e| CoreError::from_api(&e, "list albums"))?;
|
||||
if !uids.is_empty() {
|
||||
let albums_nodes = self
|
||||
.rt
|
||||
.block_on(photos.enumerate_nodes(&uids))
|
||||
.map_err(|e| CoreError::from_api(&e, "read albums"))?;
|
||||
for node in albums_nodes {
|
||||
existing.entry(node.name.clone()).or_insert(node.uid);
|
||||
}
|
||||
}
|
||||
|
||||
for title in albums {
|
||||
let members: Vec<NodeUid> = planned
|
||||
.iter()
|
||||
.filter(|item| item.albums.iter().any(|album| album == title))
|
||||
.filter_map(|item| item.uid.clone())
|
||||
.collect();
|
||||
if members.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let album_uid = match existing.get(title) {
|
||||
Some(uid) => uid.clone(),
|
||||
None => match self.rt.block_on(photos.create_album(title)) {
|
||||
Ok(uid) => {
|
||||
summary.albums_created += 1;
|
||||
existing.insert(title.clone(), uid.clone());
|
||||
uid
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(album = title, error = %e, "cannot create album");
|
||||
continue;
|
||||
}
|
||||
},
|
||||
};
|
||||
|
||||
match self
|
||||
.rt
|
||||
.block_on(photos.add_photos_to_album(&album_uid, &members))
|
||||
{
|
||||
Ok(outcomes) => {
|
||||
for (uid, outcome) in outcomes {
|
||||
match outcome {
|
||||
Ok(()) => summary.album_links += 1,
|
||||
// A photo already in the album is the expected
|
||||
// outcome of a re-run, not a failure worth counting.
|
||||
Err(e) => {
|
||||
warn!(album = title, %uid, error = %e, "cannot add photo to album")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => warn!(album = title, error = %e, "album addition failed"),
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Run an import on a background thread, logging its outcome to the activity
|
||||
/// feed. Returns immediately.
|
||||
///
|
||||
/// Refuses while one is already running: two imports of the same export
|
||||
/// would race each other's duplicate detection and upload everything twice.
|
||||
pub(crate) fn spawn_takeout_import(
|
||||
&self,
|
||||
archives: Vec<PathBuf>,
|
||||
dry_run: bool,
|
||||
) -> CoreResult<()> {
|
||||
let cancel = Arc::new(AtomicBool::new(false));
|
||||
{
|
||||
let mut state = IMPORT.lock();
|
||||
if state.running {
|
||||
return Err(CoreError::conflict(
|
||||
"an import is already running; wait for it to finish or cancel it",
|
||||
));
|
||||
}
|
||||
state.running = true;
|
||||
state.cancel = Some(cancel.clone());
|
||||
state.last = None;
|
||||
}
|
||||
|
||||
let core = self.clone();
|
||||
std::thread::spawn(move || {
|
||||
let outcome = core.import_takeout(archives, dry_run, cancel);
|
||||
{
|
||||
let mut state = IMPORT.lock();
|
||||
state.running = false;
|
||||
state.cancel = None;
|
||||
state.last = outcome.as_ref().ok().cloned();
|
||||
}
|
||||
match outcome {
|
||||
Ok(summary) => {
|
||||
let mut parts = vec![format!("{} uploaded", summary.uploaded)];
|
||||
if summary.duplicates > 0 {
|
||||
parts.push(format!("{} already there", summary.duplicates));
|
||||
}
|
||||
if summary.albums_created > 0 {
|
||||
parts.push(format!("{} albums", summary.albums_created));
|
||||
}
|
||||
if summary.failed > 0 {
|
||||
parts.push(format!("{} failed", summary.failed));
|
||||
}
|
||||
if summary.cancelled {
|
||||
parts.push("cancelled".to_string());
|
||||
}
|
||||
core.log_activity(
|
||||
ActivityKind::Upload,
|
||||
"Google Photos import",
|
||||
parts.join(" · "),
|
||||
summary.failed == 0 && !summary.cancelled,
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(error = %e, "takeout import failed");
|
||||
core.log_activity(ActivityKind::Upload, "Google Photos import", &e, false);
|
||||
}
|
||||
}
|
||||
});
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Open one of the import's archives by index.
|
||||
fn open_archive(
|
||||
archives: &[PathBuf],
|
||||
index: usize,
|
||||
) -> CoreResult<zip::ZipArchive<std::io::BufReader<std::fs::File>>> {
|
||||
let path = archives
|
||||
.get(index)
|
||||
.ok_or_else(|| CoreError::internal(format!("archive {index} is not in the import")))?;
|
||||
let file = std::fs::File::open(path)
|
||||
.map_err(|e| CoreError::invalid(format!("cannot open {}: {e}", path.display())))?;
|
||||
zip::ZipArchive::new(std::io::BufReader::new(file))
|
||||
.map_err(|e| CoreError::invalid(format!("{} is not a readable zip: {e}", path.display())))
|
||||
}
|
||||
|
||||
/// The lowercase-hex SHA-1 of everything `reader` yields — the digest Proton's
|
||||
/// photo duplicate detection is keyed on.
|
||||
fn sha1_of(reader: &mut (dyn Read + Send)) -> CoreResult<String> {
|
||||
let mut hasher = Sha1::new();
|
||||
let mut buffer = vec![0u8; 256 * 1024];
|
||||
loop {
|
||||
let read = reader
|
||||
.read(&mut buffer)
|
||||
.map_err(|e| CoreError::invalid(format!("read failed: {e}")))?;
|
||||
if read == 0 {
|
||||
break;
|
||||
}
|
||||
hasher.update(&buffer[..read]);
|
||||
}
|
||||
Ok(hex_lower(&hasher.finalize()))
|
||||
}
|
||||
|
||||
fn hex_lower(bytes: &[u8]) -> String {
|
||||
let mut out = String::with_capacity(bytes.len() * 2);
|
||||
for byte in bytes {
|
||||
out.push_str(&format!("{byte:02x}"));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use pdfs_core::takeout::TakeoutPhoto;
|
||||
|
||||
fn photo(name: &str, size: u64, bucket: Bucket) -> TakeoutPhoto {
|
||||
TakeoutPhoto {
|
||||
archive: 0,
|
||||
entry: format!("Takeout/Google Photos/x/{name}"),
|
||||
name: name.to_string(),
|
||||
size,
|
||||
media_type: "image/jpeg".to_string(),
|
||||
capture_time: None,
|
||||
favorite: false,
|
||||
bucket,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn album_copies_of_one_photo_fold_into_a_single_upload() {
|
||||
let scan = TakeoutScan {
|
||||
photos: vec![
|
||||
photo("IMG_1.jpg", 100, Bucket::Timeline),
|
||||
photo("IMG_1.jpg", 100, Bucket::Album("Iceland".into())),
|
||||
photo("IMG_1.jpg", 100, Bucket::Album("Best of".into())),
|
||||
photo("IMG_2.jpg", 200, Bucket::Timeline),
|
||||
],
|
||||
albums: vec!["Iceland".into(), "Best of".into()],
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let planned = plan(&scan);
|
||||
assert_eq!(planned.len(), 2, "the three copies are one upload");
|
||||
assert_eq!(planned[0].albums, vec!["Iceland", "Best of"]);
|
||||
assert!(planned[1].albums.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_name_different_size_stays_two_photos() {
|
||||
let scan = TakeoutScan {
|
||||
photos: vec![
|
||||
photo("IMG_1.jpg", 100, Bucket::Timeline),
|
||||
photo("IMG_1.jpg", 101, Bucket::Album("Iceland".into())),
|
||||
],
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
assert_eq!(plan(&scan).len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn metadata_from_any_copy_survives_the_fold() {
|
||||
// Only the album copy carries a sidecar; the fold must not lose it.
|
||||
let mut bare = photo("IMG_1.jpg", 100, Bucket::Timeline);
|
||||
bare.capture_time = None;
|
||||
let mut rich = photo("IMG_1.jpg", 100, Bucket::Album("Iceland".into()));
|
||||
rich.capture_time = Some(1_560_000_000);
|
||||
rich.favorite = true;
|
||||
|
||||
let planned = plan(&TakeoutScan {
|
||||
photos: vec![bare, rich],
|
||||
..Default::default()
|
||||
});
|
||||
|
||||
assert_eq!(planned[0].photo.capture_time, Some(1_560_000_000));
|
||||
assert!(planned[0].photo.favorite);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sha1_matches_the_known_digest_of_abc() {
|
||||
let mut reader: &[u8] = b"abc";
|
||||
assert_eq!(
|
||||
sha1_of(&mut reader).unwrap(),
|
||||
"a9993e364706816aba3e25717850c26c9cd0d89d"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -19,7 +19,7 @@ use parking_lot::Mutex;
|
||||
use std::collections::HashMap;
|
||||
use std::io::{Read, Write};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||
use std::time::Instant;
|
||||
|
||||
use pdfs_core::control::{JobItem, TransferDirection, TransferItem};
|
||||
@@ -272,20 +272,50 @@ impl<R: Read> Read for OwnedCountingReader<R> {
|
||||
}
|
||||
}
|
||||
|
||||
/// A [`Read`] that tallies bytes read through a [`TransferGuard`] (upload path).
|
||||
/// A [`Read`] that tallies bytes read through a [`TransferGuard`] (upload path),
|
||||
/// and can be told to stop.
|
||||
pub struct CountingReader<'a, R> {
|
||||
inner: R,
|
||||
guard: &'a TransferGuard,
|
||||
cancel: Option<Arc<AtomicBool>>,
|
||||
}
|
||||
|
||||
impl<'a, R: Read> CountingReader<'a, R> {
|
||||
pub fn new(inner: R, guard: &'a TransferGuard) -> Self {
|
||||
Self { inner, guard }
|
||||
Self {
|
||||
inner,
|
||||
guard,
|
||||
cancel: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Abandon the upload as soon as the SDK asks for its next bytes when
|
||||
/// `cancel` is set.
|
||||
///
|
||||
/// This is the only cancellation point an upload has: the SDK takes a plain
|
||||
/// blocking [`Read`] and pulls from it a block at a time, so refusing to
|
||||
/// hand over the next block is what stops it. Used when a newer revision of
|
||||
/// the same file is queued — the bytes on the wire have already been
|
||||
/// superseded, and finishing them only spends uplink on a revision the next
|
||||
/// op replaces.
|
||||
pub fn with_cancel(mut self, cancel: Arc<AtomicBool>) -> Self {
|
||||
self.cancel = Some(cancel);
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
impl<R: Read> Read for CountingReader<'_, R> {
|
||||
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
|
||||
if let Some(cancel) = &self.cancel
|
||||
&& cancel.load(Ordering::Relaxed)
|
||||
{
|
||||
// Not `Ok(0)`: a short read is how this reader says "end of file",
|
||||
// and the SDK would seal a truncated revision from it.
|
||||
return Err(std::io::Error::new(
|
||||
std::io::ErrorKind::Interrupted,
|
||||
"upload superseded by a newer write",
|
||||
));
|
||||
}
|
||||
let n = self.inner.read(buf)?;
|
||||
self.guard.add(n as u64);
|
||||
Ok(n)
|
||||
|
||||
@@ -5,6 +5,9 @@ edition.workspace = true
|
||||
license.workspace = true
|
||||
authors.workspace = true
|
||||
rust-version.workspace = true
|
||||
repository.workspace = true
|
||||
homepage.workspace = true
|
||||
description = "GTK4 desktop app, tray, and search launcher for the Proton Drive Linux client"
|
||||
|
||||
[[bin]]
|
||||
name = "pdfs-tray"
|
||||
|
||||
@@ -22,6 +22,9 @@ fn main() {
|
||||
}
|
||||
|
||||
println!("cargo:rerun-if-changed=resources/pdfs.gresource.xml");
|
||||
println!(
|
||||
"cargo:rerun-if-changed=resources/icons/scalable/actions/pdfs-build-thumbnails-symbolic.svg"
|
||||
);
|
||||
println!(
|
||||
"cargo:rerun-if-changed=resources/icons/scalable/actions/pdfs-folder-upload-symbolic.svg"
|
||||
);
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16">
|
||||
<!-- Generate-thumbnails action: a picture frame with a sparkle. A single
|
||||
symbolic fill lets GTK recolour it with the surrounding control. -->
|
||||
<path fill="#222222" d="M 2,3 C 1.45,3 1,3.45 1,4 V 14 C 1,14.55 1.45,15 2,15 H 14 C 14.55,15 15,14.55 15,14 V 7 H 13 V 13 H 3 V 5 H 9 V 3 Z M 5,6 C 4.45,6 4,6.45 4,7 C 4,7.55 4.45,8 5,8 C 5.55,8 6,7.55 6,7 C 6,6.45 5.55,6 5,6 Z M 10,7 L 7.5,10 L 6,9 L 4,11.5 V 12 H 12 V 9 Z M 13,0 L 12.4,2.4 L 10,3 L 12.4,3.6 L 13,6 L 13.6,3.6 L 16,3 L 13.6,2.4 Z"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 598 B |
@@ -4,6 +4,7 @@
|
||||
icon-theme resource path at startup, so they resolve by name (and recolour
|
||||
with the theme) exactly like a stock symbolic icon. -->
|
||||
<gresource prefix="/de/nils/protondrivelinux/icons">
|
||||
<file preprocess="xml-stripblanks">scalable/actions/pdfs-build-thumbnails-symbolic.svg</file>
|
||||
<file preprocess="xml-stripblanks">scalable/actions/pdfs-folder-upload-symbolic.svg</file>
|
||||
</gresource>
|
||||
</gresources>
|
||||
+151
-15
@@ -14,10 +14,12 @@ use pages::photos::*;
|
||||
use pages::shared::*;
|
||||
use pages::shared_by_me::*;
|
||||
use pages::status::*;
|
||||
use pages::takeout::*;
|
||||
use pages::trash::*;
|
||||
use pages::verify::*;
|
||||
use widgets::details::*;
|
||||
use widgets::share_dialog::*;
|
||||
use widgets::thumbnails::*;
|
||||
use widgets::versions_dialog::*;
|
||||
|
||||
use std::cell::{Cell, RefCell};
|
||||
@@ -46,10 +48,10 @@ use pdfs_core::config::AppDirs;
|
||||
|
||||
use pdfs_core::control::{
|
||||
ActivityEntry, ActivityKind, AlbumInfo, BookmarkInfo, DeviceInfo, DirEntry, ErrorKind,
|
||||
InvitationInfo, JobItem, PhotoItem, PhotoKind, PublicLinkInfo, RefreshScope, Request, Response,
|
||||
RestorableFolder, RestoreItem, SearchHit, ShareEntry, ShareEntryKind, SharedItem,
|
||||
SyncFolderInfo, SyncPhase, SyncProgress, TransferDirection, TransferItem, pending_summary,
|
||||
send,
|
||||
ImportSummary, InvitationInfo, JobItem, PhotoItem, PhotoKind, PublicLinkInfo, RefreshScope,
|
||||
Request, Response, RestorableFolder, RestoreItem, SearchHit, ShareEntry, ShareEntryKind,
|
||||
SharedItem, SyncFolderInfo, SyncPhase, SyncProgress, ThumbnailBuildStatus, TransferDirection,
|
||||
TransferItem, pending_summary, send,
|
||||
};
|
||||
|
||||
use pdfs_core::mounts::{MountAccess, MountKind, MountMode, MountSpec};
|
||||
@@ -106,6 +108,10 @@ struct Ui {
|
||||
/// The sidebar/content split. Collapsed while signed out, so the login page
|
||||
/// owns the whole window and no destination is reachable without a session.
|
||||
nav: adw::NavigationSplitView,
|
||||
/// Shared thumbnails for ordinary image files outside the Photos gallery.
|
||||
/// One cache and request queue serves Files, search, Shared and Trash, so
|
||||
/// the same image is downloaded and decoded only once.
|
||||
pub(crate) file_thumbs: FileThumbnailState,
|
||||
|
||||
// Per-page state. Each page module owns its own struct; `Ui` keeps only
|
||||
// what more than one page genuinely shares.
|
||||
@@ -120,6 +126,7 @@ struct Ui {
|
||||
pub(crate) devices: DevicesState,
|
||||
pub(crate) locations: LocationsState,
|
||||
pub(crate) activity: ActivityState,
|
||||
pub(crate) takeout: TakeoutState,
|
||||
}
|
||||
|
||||
impl Ui {
|
||||
@@ -205,10 +212,15 @@ fn load_proton_theme() {
|
||||
.file-grid {{ padding: 6px; }}\n\
|
||||
.file-tile {{ padding: 8px; border-radius: 10px; }}\n\
|
||||
.file-tile:hover {{ background: alpha({PROTON_PURPLE}, 0.10); }}\n\
|
||||
.file-thumbnail {{ border-radius: 7px; }}\n\
|
||||
.file-badge {{ -gtk-icon-shadow: 0 1px 2px rgba(0, 0, 0, 0.5); }}\n\
|
||||
.badge-pinned {{ color: #f5c211; }}\n\
|
||||
.badge-cached {{ color: #2ec27e; }}\n\
|
||||
.badge-cloud {{ color: #9aa0a6; }}\n\
|
||||
.browser-statusbar {{ background-color: alpha(currentColor, 0.025); }}\n\
|
||||
scale.browser-status-meter trough, progressbar.browser-status-meter trough {{ min-width: 104px; }}\n\
|
||||
scale.browser-status-meter trough {{ min-height: 6px; }}\n\
|
||||
progressbar.browser-status-meter trough, progressbar.browser-status-meter progress {{ min-height: 6px; }}\n\
|
||||
.photo-viewer-window {{ background-color: #111014; }}\n\
|
||||
.viewer-top-bar {{ background: linear-gradient(to bottom, rgba(0, 0, 0, 0.75), rgba(0, 0, 0, 0)); padding: 10px 16px 28px 20px; color: white; }}\n\
|
||||
.viewer-title {{ font-weight: 700; font-size: 1.05rem; text-shadow: 0 1px 3px rgba(0, 0, 0, 0.9); color: white; }}\n\
|
||||
@@ -246,7 +258,10 @@ fn load_proton_theme() {
|
||||
.navigation-sidebar row {{ border-radius: 8px; margin: 2px 6px; }}\n\
|
||||
.navigation-sidebar row:selected {{ background: alpha({PROTON_PURPLE}, 0.16); color: {PROTON_PURPLE}; font-weight: 600; }}\n\
|
||||
.navigation-sidebar row:selected image {{ color: {PROTON_PURPLE}; }}\n\
|
||||
.file-tile:selected, .file-tile:hover:selected {{ background: alpha({PROTON_PURPLE}, 0.20); }}\n"
|
||||
.file-tile:selected, .file-tile:hover:selected {{ background: alpha({PROTON_PURPLE}, 0.20); }}\n\
|
||||
.bulk-bar {{ padding: 6px 8px; border-radius: 12px; background: alpha({PROTON_PURPLE}, 0.12); }}\n\
|
||||
.dropzone {{ border: 2px dashed alpha(currentColor, 0.25); border-radius: 16px; padding: 28px 18px; transition: background 160ms ease, border-color 160ms ease; }}\n\
|
||||
.dropzone-active {{ border-color: {PROTON_PURPLE}; background: alpha({PROTON_PURPLE}, 0.10); }}\n"
|
||||
);
|
||||
let provider = gtk4::CssProvider::new();
|
||||
provider.load_from_string(&css);
|
||||
@@ -285,6 +300,7 @@ fn build_window(app: &adw::Application) {
|
||||
let (locations_page, locations_widgets) = build_locations_page();
|
||||
let (activity_page, activity_widgets) = build_activity_page();
|
||||
let (trash_page, trash_widgets) = build_trash_page();
|
||||
let (takeout_page, takeout_widgets) = build_takeout_page();
|
||||
stack.add_named(&login_page, Some("login"));
|
||||
stack.add_named(&main_page, Some("main"));
|
||||
stack.add_named(&browser_page, Some("browser"));
|
||||
@@ -295,6 +311,7 @@ fn build_window(app: &adw::Application) {
|
||||
stack.add_named(&locations_page, Some("locations"));
|
||||
stack.add_named(&activity_page, Some("activity"));
|
||||
stack.add_named(&trash_page, Some("trash"));
|
||||
stack.add_named(&takeout_page, Some("takeout"));
|
||||
|
||||
// Sidebar: the signed-in destinations. Selecting a row swaps the page stack;
|
||||
// `sync_sidebar` pushes the other way when navigation happens elsewhere (e.g.
|
||||
@@ -338,6 +355,7 @@ fn build_window(app: &adw::Application) {
|
||||
mounted: RefCell::new(false),
|
||||
sidebar: sidebar_list.clone(),
|
||||
nav: split.clone(),
|
||||
file_thumbs: FileThumbnailState::new(),
|
||||
login: LoginState {
|
||||
email: login_widgets.0,
|
||||
password: login_widgets.1,
|
||||
@@ -362,6 +380,8 @@ fn build_window(app: &adw::Application) {
|
||||
budget_row: main_widgets.budget_row.clone(),
|
||||
mountpoint_row: main_widgets.mountpoint_row.clone(),
|
||||
settings_suppress: Cell::new(false),
|
||||
budget_source: RefCell::new(None),
|
||||
pins_expanded: Cell::new(false),
|
||||
pins_group: main_widgets.pins_group.clone(),
|
||||
pin_rows: RefCell::new(Vec::new()),
|
||||
pins_state: RefCell::new(None),
|
||||
@@ -381,7 +401,29 @@ fn build_window(app: &adw::Application) {
|
||||
new_folder: browser_widgets.new_folder.clone(),
|
||||
upload: browser_widgets.upload.clone(),
|
||||
upload_folder: browser_widgets.upload_folder.clone(),
|
||||
build_thumbnails: browser_widgets.build_thumbnails.clone(),
|
||||
thumbnail_build_row: browser_widgets.thumbnail_build_row.clone(),
|
||||
thumbnail_progress: browser_widgets.thumbnail_progress.clone(),
|
||||
thumbnail_status: browser_widgets.thumbnail_status.clone(),
|
||||
thumbnail_poll: RefCell::new(None),
|
||||
thumbnail_build_running: Cell::new(false),
|
||||
thumbnail_cancel_pending: Cell::new(false),
|
||||
search_source: RefCell::new(None),
|
||||
load_generation: Cell::new(0),
|
||||
views: browser_widgets.views.clone(),
|
||||
bulk: browser_widgets.bulk.clone(),
|
||||
bulk_label: browser_widgets.bulk_label.clone(),
|
||||
bulk_trash: browser_widgets.bulk_trash.clone(),
|
||||
bulk_pin: browser_widgets.bulk_pin.clone(),
|
||||
bulk_unpin: browser_widgets.bulk_unpin.clone(),
|
||||
empty_actions: browser_widgets.empty_actions.clone(),
|
||||
summary: browser_widgets.summary.clone(),
|
||||
zoom: browser_widgets.zoom.clone(),
|
||||
grid_thumbnail_size: Cell::new(GRID_THUMB_DEFAULT),
|
||||
grid_tiles: RefCell::new(Vec::new()),
|
||||
quota_box: browser_widgets.quota_box.clone(),
|
||||
quota: browser_widgets.quota.clone(),
|
||||
quota_text: browser_widgets.quota_text.clone(),
|
||||
},
|
||||
details: DetailsState {
|
||||
details: browser_widgets.details,
|
||||
@@ -407,6 +449,8 @@ fn build_window(app: &adw::Application) {
|
||||
retry: gallery_widgets.retry.clone(),
|
||||
more: gallery_widgets.more.clone(),
|
||||
upload: gallery_widgets.upload.clone(),
|
||||
import: gallery_widgets.import.clone(),
|
||||
empty_actions: gallery_widgets.empty_actions.clone(),
|
||||
title: gallery_widgets.title.clone(),
|
||||
subtitle: gallery_widgets.subtitle.clone(),
|
||||
albums: gallery_widgets.albums.clone(),
|
||||
@@ -494,6 +538,24 @@ fn build_window(app: &adw::Application) {
|
||||
inflight: Cell::new(false),
|
||||
key: RefCell::new(None),
|
||||
},
|
||||
takeout: TakeoutState {
|
||||
archives: RefCell::new(Vec::new()),
|
||||
list_group: takeout_widgets.list_group.clone(),
|
||||
rows: RefCell::new(Vec::new()),
|
||||
dropzone: takeout_widgets.dropzone.clone(),
|
||||
scan_button: takeout_widgets.scan_button.clone(),
|
||||
import_button: takeout_widgets.import_button.clone(),
|
||||
cancel_button: takeout_widgets.cancel_button.clone(),
|
||||
clear_button: takeout_widgets.clear_button.clone(),
|
||||
progress_group: takeout_widgets.progress_group.clone(),
|
||||
progress_label: takeout_widgets.progress_label.clone(),
|
||||
progress_bar: takeout_widgets.progress_bar.clone(),
|
||||
summary_group: takeout_widgets.summary_group.clone(),
|
||||
summary_rows: RefCell::new(Vec::new()),
|
||||
inflight: Cell::new(false),
|
||||
running: Cell::new(false),
|
||||
dry_run: Cell::new(false),
|
||||
},
|
||||
});
|
||||
wire_login(&ui);
|
||||
wire_logout(&ui, &main_widgets.logout_button);
|
||||
@@ -501,18 +563,31 @@ fn build_window(app: &adw::Application) {
|
||||
&ui,
|
||||
&main_widgets.purge_button,
|
||||
&main_widgets.mountpoint_button,
|
||||
&main_widgets.import_row,
|
||||
);
|
||||
wire_sidebar(&ui);
|
||||
wire_browser(&ui, &browser_widgets.grid, &browser_widgets.column_view);
|
||||
wire_bulk(
|
||||
&ui,
|
||||
&browser_widgets.bulk_clear,
|
||||
&browser_widgets.empty_upload,
|
||||
&browser_widgets.empty_new_folder,
|
||||
);
|
||||
wire_browser_actions(
|
||||
&ui,
|
||||
&browser_widgets.new_folder,
|
||||
&browser_widgets.upload,
|
||||
&browser_widgets.upload_folder,
|
||||
&browser_widgets.build_thumbnails,
|
||||
);
|
||||
wire_details(&ui);
|
||||
wire_search(&ui);
|
||||
wire_gallery(&ui, &gallery_widgets.list, &gallery_widgets.scroll);
|
||||
wire_gallery_empty(
|
||||
&ui,
|
||||
&gallery_widgets.empty_upload,
|
||||
&gallery_widgets.empty_import,
|
||||
);
|
||||
wire_albums(&ui);
|
||||
wire_trash(&ui, &trash_widgets.list, &trash_widgets.empty);
|
||||
wire_shared(&ui, &shared_widgets.retry, &shared_widgets.add_bookmark);
|
||||
@@ -520,6 +595,7 @@ fn build_window(app: &adw::Application) {
|
||||
wire_devices(&ui, &devices_widgets.retry, &devices_widgets.restore);
|
||||
wire_locations(&ui, &locations_widgets.retry, &locations_widgets.add_folder);
|
||||
wire_activity(&ui, &activity_widgets.retry);
|
||||
wire_takeout(&ui, &takeout_widgets);
|
||||
wire_refresh(
|
||||
&ui,
|
||||
&[
|
||||
@@ -539,6 +615,10 @@ fn build_window(app: &adw::Application) {
|
||||
// network round-trip only happens on demand rather than on every refresh.
|
||||
let ui_nav = ui.clone();
|
||||
stack.connect_visible_child_name_notify(move |st| {
|
||||
// Rows from the page being left must not keep full-size image downloads
|
||||
// alive in the daemon. The page being entered establishes a fresh
|
||||
// thumbnail generation as it paints.
|
||||
cancel_file_thumbnails(&ui_nav);
|
||||
sync_sidebar(&ui_nav);
|
||||
match st.visible_child_name().as_deref() {
|
||||
Some("browser") => load_browser(&ui_nav),
|
||||
@@ -559,6 +639,9 @@ fn build_window(app: &adw::Application) {
|
||||
// every visit to stay live.
|
||||
Some("activity") => load_activity(&ui_nav),
|
||||
Some("trash") => load_trash(&ui_nav),
|
||||
// The import runs in the daemon and outlives this page, so arriving
|
||||
// here asks straight away whether one is in flight.
|
||||
Some("takeout") => refresh_takeout(&ui_nav),
|
||||
_ => {}
|
||||
}
|
||||
});
|
||||
@@ -644,6 +727,17 @@ fn build_sidebar() -> (adw::NavigationPage, gtk4::ListBox) {
|
||||
(page, list)
|
||||
}
|
||||
|
||||
/// The sidebar destination a page belongs under. Most pages are their own
|
||||
/// destination; a sub-page reached from one (Import, off Settings) has no row of
|
||||
/// its own and answers with its parent, so the sidebar highlights where the user
|
||||
/// came from rather than clearing — which would read as "nowhere".
|
||||
fn destination_of(page: &str) -> &str {
|
||||
match page {
|
||||
"takeout" => "main",
|
||||
other => other,
|
||||
}
|
||||
}
|
||||
|
||||
/// Selecting a sidebar row navigates the page stack. The reverse direction (stack
|
||||
/// → sidebar highlight) is [`sync_sidebar`], so the two can't fight: this handler
|
||||
/// only ever writes the stack.
|
||||
@@ -654,7 +748,11 @@ fn wire_sidebar(ui: &Rc<Ui>) {
|
||||
let Some((page, _, _)) = DESTINATIONS.get(row.index() as usize) else {
|
||||
return;
|
||||
};
|
||||
if ui_row.stack.visible_child_name().as_deref() != Some(*page) {
|
||||
// Compared by *destination*, not by page: while a sub-page is open its
|
||||
// parent's row is the selected one, and re-asserting it here would kick
|
||||
// the user off the sub-page the moment they arrived.
|
||||
let current = ui_row.stack.visible_child_name();
|
||||
if current.as_deref().map(destination_of) != Some(*page) {
|
||||
ui_row.stack.set_visible_child_name(page);
|
||||
}
|
||||
});
|
||||
@@ -755,6 +853,7 @@ fn sync_sidebar(ui: &Rc<Ui>) {
|
||||
let Some(current) = ui.stack.visible_child_name() else {
|
||||
return;
|
||||
};
|
||||
let current = destination_of(¤t);
|
||||
let index = DESTINATIONS
|
||||
.iter()
|
||||
.position(|(page, _, _)| *page == current);
|
||||
@@ -776,6 +875,22 @@ fn toast(ui: &Rc<Ui>, message: &str) {
|
||||
ui.toasts.add_toast(adw::Toast::new(message));
|
||||
}
|
||||
|
||||
/// Show a toast that offers one follow-up action — an Undo, or a way to the page
|
||||
/// the outcome landed on.
|
||||
///
|
||||
/// Given a longer timeout than a plain toast: an action the user has to notice,
|
||||
/// read and reach is not the same ask as a line of confirmation they can ignore.
|
||||
fn toast_action(ui: &Rc<Ui>, message: &str, label: &str, action: impl Fn(&Rc<Ui>) + 'static) {
|
||||
let toast = adw::Toast::builder()
|
||||
.title(message)
|
||||
.button_label(label)
|
||||
.timeout(8)
|
||||
.build();
|
||||
let ui_action = ui.clone();
|
||||
toast.connect_button_clicked(move |_| action(&ui_action));
|
||||
ui.toasts.add_toast(toast);
|
||||
}
|
||||
|
||||
/// Show a toast for a failure. Same surface as [`toast`], but the message is
|
||||
/// prefixed with what was being attempted, since a bare daemon error ("no such
|
||||
/// file") reads as noise without it.
|
||||
@@ -830,7 +945,7 @@ fn toast_failure(ui: &Rc<Ui>, what: &str, message: &str, kind: ErrorKind) {
|
||||
fn build_primary_menu() -> gtk4::MenuButton {
|
||||
let menu = gio::Menu::new();
|
||||
menu.append(Some("Keyboard Shortcuts"), Some("win.shortcuts"));
|
||||
menu.append(Some("About Proton Drive"), Some("win.about"));
|
||||
menu.append(Some("About Proton Drive for Linux"), Some("win.about"));
|
||||
gtk4::MenuButton::builder()
|
||||
.icon_name("open-menu-symbolic")
|
||||
.tooltip_text("Main menu")
|
||||
@@ -849,11 +964,17 @@ fn install_window_actions(window: &adw::ApplicationWindow) {
|
||||
let win = window.clone();
|
||||
about.connect_activate(move |_, _| {
|
||||
let dialog = adw::AboutDialog::builder()
|
||||
.application_name("Proton Drive")
|
||||
.application_icon("folder-remote-symbolic")
|
||||
.application_name("Proton Drive for Linux")
|
||||
.application_icon("io.narl.proton-drive-linux")
|
||||
.version(pdfs_core::config::APP_VERSION)
|
||||
.developer_name("proton-drive-linux")
|
||||
.comments("On-demand Proton Drive sync for Linux.")
|
||||
.developer_name("Nils Pukropp")
|
||||
.website("https://github.com/narrrl/proton-drive-linux")
|
||||
.issue_url("https://github.com/narrrl/proton-drive-linux/issues")
|
||||
.license_type(gtk4::License::MitX11)
|
||||
.comments(
|
||||
"Files-on-demand Proton Drive for the Linux desktop.\n\n\
|
||||
Unofficial client — not affiliated with, endorsed by, or supported by Proton AG.",
|
||||
)
|
||||
.build();
|
||||
dialog.present(Some(&win));
|
||||
});
|
||||
@@ -913,7 +1034,13 @@ fn install_shortcuts(ui: &Rc<Ui>, window: &adw::ApplicationWindow) {
|
||||
// browser-only bindings.
|
||||
gtk4::gdk::Key::F5 => reload_current_page(&ui),
|
||||
gtk4::gdk::Key::r | gtk4::gdk::Key::R if ctrl => reload_current_page(&ui),
|
||||
gtk4::gdk::Key::f | gtk4::gdk::Key::F if ctrl && on_browser => {
|
||||
// Search lives on Files, but the shortcut works from anywhere:
|
||||
// wanting to find a file is not a reason to first have to remember
|
||||
// which page owns the search box.
|
||||
gtk4::gdk::Key::f | gtk4::gdk::Key::F if ctrl => {
|
||||
if !on_browser {
|
||||
ui.stack.set_visible_child_name("browser");
|
||||
}
|
||||
ui.browser.search.grab_focus();
|
||||
}
|
||||
gtk4::gdk::Key::n | gtk4::gdk::Key::N if ctrl && on_browser => prompt_new_folder(&ui),
|
||||
@@ -923,12 +1050,21 @@ fn install_shortcuts(ui: &Rc<Ui>, window: &adw::ApplicationWindow) {
|
||||
prompt_rename(&ui, &entry);
|
||||
}
|
||||
}
|
||||
// Acts on the whole selection: Delete with five files highlighted
|
||||
// means those five, not whichever one the details pane happens to
|
||||
// be describing.
|
||||
gtk4::gdk::Key::Delete if on_browser => {
|
||||
if let Some(entry) = selected_entry(&ui) {
|
||||
prompt_delete(&ui, &entry);
|
||||
let entries = selected_entries(&ui);
|
||||
if !entries.is_empty() {
|
||||
prompt_delete_many(&ui, entries);
|
||||
}
|
||||
}
|
||||
gtk4::gdk::Key::Escape if on_browser && ui.browser.split.shows_sidebar() => {
|
||||
gtk4::gdk::Key::Escape
|
||||
if on_browser
|
||||
&& (ui.browser.split.shows_sidebar() || ui.browser.bulk.reveals_child()) =>
|
||||
{
|
||||
clear_selection(&ui);
|
||||
sync_bulk_bar(&ui);
|
||||
hide_details(&ui);
|
||||
}
|
||||
_ => return glib::Propagation::Proceed,
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -9,5 +9,6 @@ pub(crate) mod photos;
|
||||
pub(crate) mod shared;
|
||||
pub(crate) mod shared_by_me;
|
||||
pub(crate) mod status;
|
||||
pub(crate) mod takeout;
|
||||
pub(crate) mod trash;
|
||||
pub(crate) mod verify;
|
||||
@@ -19,6 +19,12 @@ pub(crate) struct GalleryState {
|
||||
pub(crate) retry: gtk4::Button,
|
||||
pub(crate) more: gtk4::Button,
|
||||
pub(crate) upload: gtk4::Button,
|
||||
/// Opens the Google Photos Takeout import chooser.
|
||||
pub(crate) import: gtk4::Button,
|
||||
/// Upload / Import offered on the *empty timeline* status page, so a fresh
|
||||
/// account is a place to start rather than a dead end. Hidden on every other
|
||||
/// status: a load error is not the moment to offer an upload.
|
||||
pub(crate) empty_actions: gtk4::Box,
|
||||
/// "Photos", or the album's name while one is open.
|
||||
pub(crate) title: gtk4::Label,
|
||||
/// "1,204 photos" under the page title.
|
||||
@@ -177,6 +183,10 @@ pub(crate) struct GalleryWidgets {
|
||||
pub(crate) scroll: gtk4::ScrolledWindow,
|
||||
pub(crate) retry: gtk4::Button,
|
||||
pub(crate) upload: gtk4::Button,
|
||||
pub(crate) import: gtk4::Button,
|
||||
pub(crate) empty_actions: gtk4::Box,
|
||||
pub(crate) empty_upload: gtk4::Button,
|
||||
pub(crate) empty_import: gtk4::Button,
|
||||
pub(crate) refresh: gtk4::Button,
|
||||
/// The Albums grid, its own status page and the stack between them, plus the
|
||||
/// Photos/Albums switcher and the back button out of an album.
|
||||
@@ -227,10 +237,29 @@ pub(crate) fn build_gallery_page() -> (gtk4::Widget, GalleryWidgets) {
|
||||
retry.add_css_class("suggested-action");
|
||||
retry.set_visible(false);
|
||||
|
||||
// The empty-timeline state's two ways forward. `StatusPage` takes one child,
|
||||
// so Retry and these share a box and each is shown only for its own state.
|
||||
let empty_upload = gtk4::Button::builder().label("Upload photos").build();
|
||||
empty_upload.add_css_class("pill");
|
||||
empty_upload.add_css_class("suggested-action");
|
||||
let empty_import = gtk4::Button::builder()
|
||||
.label("Import from Google Photos")
|
||||
.build();
|
||||
empty_import.add_css_class("pill");
|
||||
let empty_actions = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
|
||||
empty_actions.set_halign(gtk4::Align::Center);
|
||||
empty_actions.set_visible(false);
|
||||
empty_actions.append(&empty_upload);
|
||||
empty_actions.append(&empty_import);
|
||||
|
||||
let status_child = gtk4::Box::new(gtk4::Orientation::Vertical, 12);
|
||||
status_child.append(&retry);
|
||||
status_child.append(&empty_actions);
|
||||
|
||||
let status = adw::StatusPage::builder()
|
||||
.icon_name("image-x-generic-symbolic")
|
||||
.vexpand(true)
|
||||
.child(&retry)
|
||||
.child(&status_child)
|
||||
.build();
|
||||
status.add_css_class("compact");
|
||||
|
||||
@@ -288,12 +317,23 @@ pub(crate) fn build_gallery_page() -> (gtk4::Widget, GalleryWidgets) {
|
||||
.build();
|
||||
upload.add_css_class("pill");
|
||||
upload.add_css_class("suggested-action");
|
||||
|
||||
// Importing an export is a rarer, heavier action than adding one photo, so
|
||||
// it sits beside Upload as a plain button rather than a second accent one.
|
||||
let import = gtk4::Button::builder()
|
||||
.icon_name("folder-download-symbolic")
|
||||
.tooltip_text("Import a Google Photos Takeout export")
|
||||
.valign(gtk4::Align::Center)
|
||||
.build();
|
||||
import.add_css_class("flat");
|
||||
import.add_css_class("circular");
|
||||
let refresh = refresh_button();
|
||||
|
||||
let header_box = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
|
||||
header_box.append(&back);
|
||||
header_box.append(&titles);
|
||||
header_box.append(&refresh);
|
||||
header_box.append(&import);
|
||||
header_box.append(&upload);
|
||||
|
||||
// All / Photos / Videos / Raw filter. Linked toggles acting as one segmented
|
||||
@@ -428,6 +468,10 @@ pub(crate) fn build_gallery_page() -> (gtk4::Widget, GalleryWidgets) {
|
||||
scroll,
|
||||
retry,
|
||||
upload,
|
||||
import,
|
||||
empty_actions,
|
||||
empty_upload,
|
||||
empty_import,
|
||||
refresh,
|
||||
tabs,
|
||||
favorites_btn,
|
||||
@@ -554,6 +598,15 @@ pub(crate) fn thousands(n: usize) -> String {
|
||||
/// Wire the gallery: install the section factory, the zoom gestures, the pager
|
||||
/// and the upload button. Activating a thumbnail downloads the photo and opens it
|
||||
/// in the in-app lightbox.
|
||||
/// Wire the empty-timeline buttons. Upload re-fires the header's own Upload
|
||||
/// rather than duplicating its file-chooser, so the two can't drift apart.
|
||||
pub(crate) fn wire_gallery_empty(ui: &Rc<Ui>, upload: >k4::Button, import: >k4::Button) {
|
||||
let ui_upload = ui.clone();
|
||||
upload.connect_clicked(move |_| ui_upload.gallery.upload.emit_clicked());
|
||||
let ui_import = ui.clone();
|
||||
import.connect_clicked(move |_| ui_import.stack.set_visible_child_name("takeout"));
|
||||
}
|
||||
|
||||
pub(crate) fn wire_gallery(ui: &Rc<Ui>, list: >k4::ListView, scroll: >k4::ScrolledWindow) {
|
||||
let factory = gtk4::SignalListItemFactory::new();
|
||||
factory.connect_setup(|_, item| {
|
||||
@@ -729,6 +782,14 @@ pub(crate) fn wire_gallery(ui: &Rc<Ui>, list: >k4::ListView, scroll: >k4::Sc
|
||||
load_gallery(&ui_dates, false);
|
||||
});
|
||||
|
||||
// The import is a staged, hours-long migration, so the button hands over to
|
||||
// the Import page rather than opening a file chooser here: the archives need
|
||||
// reviewing before anything is sent, and the run needs somewhere to report.
|
||||
let ui_import = ui.clone();
|
||||
ui.gallery
|
||||
.import
|
||||
.connect_clicked(move |_| ui_import.stack.set_visible_child_name("takeout"));
|
||||
|
||||
let ui_upload = ui.clone();
|
||||
ui.gallery.upload.connect_clicked(move |_| {
|
||||
let dialog = gtk4::FileDialog::builder()
|
||||
@@ -1460,6 +1521,12 @@ pub(crate) fn load_gallery(ui: &Rc<Ui>, append: bool) {
|
||||
)
|
||||
};
|
||||
gallery_status(&ui, "image-x-generic-symbolic", title, description, false);
|
||||
// Only the whole-timeline empty state: an empty *album* is
|
||||
// filled by adding existing photos to it, not by uploading
|
||||
// into it, so the buttons would point the wrong way.
|
||||
ui.gallery
|
||||
.empty_actions
|
||||
.set_visible(ui.gallery.album.borrow().is_none());
|
||||
return;
|
||||
}
|
||||
ui.gallery.content.set_visible_child_name("timeline");
|
||||
@@ -1502,6 +1569,9 @@ pub(crate) fn gallery_status(ui: &Rc<Ui>, icon: &str, title: &str, description:
|
||||
ui.gallery.status.set_title(title);
|
||||
ui.gallery.status.set_description(Some(description));
|
||||
ui.gallery.retry.set_visible(retry);
|
||||
// Only the empty timeline offers a way to fill it; every other status turns
|
||||
// those buttons back off.
|
||||
ui.gallery.empty_actions.set_visible(false);
|
||||
ui.gallery.more.set_visible(false);
|
||||
ui.gallery.content.set_visible_child_name("status");
|
||||
}
|
||||
|
||||
@@ -151,6 +151,7 @@ pub(crate) fn load_shared(ui: &Rc<Ui>) {
|
||||
if ui.shared.inflight.get() {
|
||||
return;
|
||||
}
|
||||
cancel_file_thumbnails(ui);
|
||||
let current = ui.shared.nav.borrow().last().cloned();
|
||||
if let Some((uid, _)) = current {
|
||||
load_shared_folder(ui, uid);
|
||||
@@ -308,11 +309,7 @@ fn shared_entry_row(ui: &Rc<Ui>, entry: &DirEntry) -> adw::ActionRow {
|
||||
if let Some(badge) = role_badge(&entry.role) {
|
||||
row.add_suffix(&badge);
|
||||
}
|
||||
row.add_prefix(>k4::Image::from_icon_name(if entry.is_dir {
|
||||
"folder-symbolic"
|
||||
} else {
|
||||
"text-x-generic-symbolic"
|
||||
}));
|
||||
row.add_prefix(&file_thumbnail(ui, entry, 40, 24, true));
|
||||
let ui_act = ui.clone();
|
||||
let uid = entry.uid.clone();
|
||||
let name = entry.name.clone();
|
||||
|
||||
@@ -113,6 +113,7 @@ pub(crate) fn load_shared_by_me(ui: &Rc<Ui>) {
|
||||
if ui.shared_by_me.inflight.get() {
|
||||
return;
|
||||
}
|
||||
cancel_file_thumbnails(ui);
|
||||
ui.shared_by_me.inflight.set(true);
|
||||
shared_by_me_status(
|
||||
ui,
|
||||
@@ -200,15 +201,12 @@ pub(crate) fn repaint_shared_by_me(ui: &Rc<Ui>, items: &[SharedItem]) {
|
||||
ui.shared_by_me.content.set_visible_child_name("list");
|
||||
let mut rows: Vec<gtk4::Widget> = Vec::new();
|
||||
for item in items {
|
||||
let entry = shared_item_as_entry(item);
|
||||
let row = adw::ActionRow::builder()
|
||||
.title(&item.name)
|
||||
.subtitle(shared_item_summary(item))
|
||||
.build();
|
||||
row.add_prefix(>k4::Image::from_icon_name(if item.is_dir {
|
||||
"folder-symbolic"
|
||||
} else {
|
||||
"text-x-generic-symbolic"
|
||||
}));
|
||||
row.add_prefix(&file_thumbnail(ui, &entry, 40, 24, true));
|
||||
|
||||
// Copy / open the public link, when there is one with a recovered URL.
|
||||
if let Some(url) = item.link.as_ref().and_then(|l| l.url.clone()) {
|
||||
@@ -245,7 +243,6 @@ pub(crate) fn repaint_shared_by_me(ui: &Rc<Ui>, items: &[SharedItem]) {
|
||||
.build();
|
||||
manage.add_css_class("flat");
|
||||
let ui_manage = ui.clone();
|
||||
let entry = shared_item_as_entry(item);
|
||||
manage.connect_clicked(move |_| open_share_dialog(&ui_manage, &entry));
|
||||
row.add_suffix(&manage);
|
||||
|
||||
@@ -289,7 +286,7 @@ pub(crate) fn shared_item_as_entry(item: &SharedItem) -> DirEntry {
|
||||
name: item.name.clone(),
|
||||
is_dir: item.is_dir,
|
||||
size: 0,
|
||||
modified: 0,
|
||||
modified: item.modified,
|
||||
pinned: false,
|
||||
cached: false,
|
||||
uid: item.uid.clone(),
|
||||
|
||||
@@ -37,7 +37,16 @@ pub(crate) struct StatusState {
|
||||
/// Set while a settings widget is being populated programmatically, so its
|
||||
/// change handler skips the IPC/systemd side effect.
|
||||
pub(crate) settings_suppress: Cell<bool>,
|
||||
/// Pending debounce for the cache-budget editor. Each `+` click is a value
|
||||
/// change, and firing a `SetCacheBudget` (and a toast) per click means a
|
||||
/// stack of toasts and a series of caps the user never asked to apply — so
|
||||
/// only the value they settle on is sent.
|
||||
pub(crate) budget_source: RefCell<Option<glib::SourceId>>,
|
||||
pub(crate) pins_group: adw::PreferencesGroup,
|
||||
/// Whether the pin list is showing every pin or only the first
|
||||
/// [`PINS_COLLAPSED`]. A long pin list would otherwise push everything below
|
||||
/// it — including the Developer group — off the end of the page.
|
||||
pub(crate) pins_expanded: Cell<bool>,
|
||||
/// Rows currently shown under [`Self::pins_group`], retained so a refresh can
|
||||
/// diff against them and only rebuild when the pin set actually changes.
|
||||
pub(crate) pin_rows: RefCell<Vec<PinRow>>,
|
||||
@@ -104,6 +113,8 @@ pub(crate) struct MainWidgets {
|
||||
/// which is where the folder is changed.
|
||||
pub(crate) mountpoint_row: adw::ActionRow,
|
||||
pub(crate) mountpoint_button: gtk4::Button,
|
||||
/// Opens the Google Photos Takeout import page.
|
||||
pub(crate) import_row: adw::ActionRow,
|
||||
}
|
||||
|
||||
/// The main (logged-in) page: a libadwaita settings surface — account header,
|
||||
@@ -233,6 +244,22 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
|
||||
mountpoint_row.add_suffix(&mountpoint_button);
|
||||
system_group.add(&mountpoint_row);
|
||||
|
||||
// Import: a one-off migration rather than a setting, so it is a doorway to
|
||||
// its own page rather than controls inlined here — staging a set of archives
|
||||
// and watching an hours-long upload needs the room.
|
||||
let import_group = adw::PreferencesGroup::builder()
|
||||
.title("Import")
|
||||
.description("Bring a photo library from another service into Proton Photos.")
|
||||
.build();
|
||||
let import_row = adw::ActionRow::builder()
|
||||
.title("Import from Google Photos")
|
||||
.subtitle("Add a Google Takeout export to your timeline.")
|
||||
.activatable(true)
|
||||
.build();
|
||||
import_row.add_prefix(>k4::Image::from_icon_name("folder-download-symbolic"));
|
||||
import_row.add_suffix(>k4::Image::from_icon_name("go-next-symbolic"));
|
||||
import_group.add(&import_row);
|
||||
|
||||
// Pins group: filled in by refresh.
|
||||
let pins_group = adw::PreferencesGroup::builder()
|
||||
.title("Pinned files")
|
||||
@@ -265,6 +292,7 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
|
||||
inner.append(&transfers_group);
|
||||
inner.append(&storage_group);
|
||||
inner.append(&system_group);
|
||||
inner.append(&import_group);
|
||||
inner.append(&pins_group);
|
||||
inner.append(&dev_group);
|
||||
|
||||
@@ -292,10 +320,20 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
|
||||
purge_button,
|
||||
mountpoint_row,
|
||||
mountpoint_button,
|
||||
import_row,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
/// How many pins the Settings list shows before collapsing the rest behind a
|
||||
/// "Show all" row. Enough to recognise the list at a glance; short enough that
|
||||
/// the groups below it stay reachable.
|
||||
pub(crate) const PINS_COLLAPSED: usize = 6;
|
||||
|
||||
/// How long the cache-budget editor waits after the last change before applying
|
||||
/// it, so a run of `+` clicks is one request rather than one per click.
|
||||
pub(crate) const BUDGET_DEBOUNCE: Duration = Duration::from_millis(600);
|
||||
|
||||
/// Bytes per GiB, for the cache-budget editor's unit conversion.
|
||||
pub(crate) const GIB: f64 = 1024.0 * 1024.0 * 1024.0;
|
||||
|
||||
@@ -308,6 +346,7 @@ pub(crate) fn wire_settings(
|
||||
ui: &Rc<Ui>,
|
||||
purge_button: >k4::Button,
|
||||
mountpoint_button: >k4::Button,
|
||||
import_row: &adw::ActionRow,
|
||||
) {
|
||||
let config = ui.dirs.load_config();
|
||||
|
||||
@@ -329,13 +368,22 @@ pub(crate) fn wire_settings(
|
||||
if ui_budget.status.settings_suppress.get() {
|
||||
return;
|
||||
}
|
||||
// Replace any pending apply, so only the value the user stops on is sent.
|
||||
if let Some(src) = ui_budget.status.budget_source.borrow_mut().take() {
|
||||
src.remove();
|
||||
}
|
||||
let bytes = (row.value() * GIB).round() as u64;
|
||||
settings_request(
|
||||
&ui_budget,
|
||||
Request::SetCacheBudget { bytes },
|
||||
"Cache budget updated",
|
||||
"Couldn't set cache budget",
|
||||
);
|
||||
let ui_fire = ui_budget.clone();
|
||||
let src = glib::timeout_add_local_once(BUDGET_DEBOUNCE, move || {
|
||||
ui_fire.status.budget_source.borrow_mut().take();
|
||||
settings_request(
|
||||
&ui_fire,
|
||||
Request::SetCacheBudget { bytes },
|
||||
"Cache budget updated",
|
||||
"Couldn't set cache budget",
|
||||
);
|
||||
});
|
||||
*ui_budget.status.budget_source.borrow_mut() = Some(src);
|
||||
});
|
||||
|
||||
// Purge: confirm, then drop all unpinned cached content via the daemon.
|
||||
@@ -382,6 +430,10 @@ pub(crate) fn wire_settings(
|
||||
// local path; this button is the way there.
|
||||
let ui_mp = ui.clone();
|
||||
mountpoint_button.connect_clicked(move |_| ui_mp.stack.set_visible_child_name("locations"));
|
||||
|
||||
// Import: a doorway, not a setting — everything about it lives on its page.
|
||||
let ui_import = ui.clone();
|
||||
import_row.connect_activated(move |_| ui_import.stack.set_visible_child_name("takeout"));
|
||||
}
|
||||
|
||||
/// Run a settings control-socket round-trip (budget / purge) on a worker thread,
|
||||
@@ -513,8 +565,14 @@ pub(crate) fn refresh(ui: &Rc<Ui>) {
|
||||
refresh_transfers(ui);
|
||||
// Both of these pages show work as it happens, so they follow the tick while
|
||||
// they are on screen. Every other page loads on navigation only.
|
||||
// A Takeout import outlives the page that started it, so it is polled
|
||||
// wherever the user has navigated to — otherwise leaving the page means
|
||||
// never being told it finished.
|
||||
if ui.stack.visible_child_name().as_deref() == Some("takeout") || ui.takeout.running.get() {
|
||||
refresh_takeout(ui);
|
||||
}
|
||||
match ui.stack.visible_child_name().as_deref() {
|
||||
Some("main") => refresh_quota(ui),
|
||||
Some("main" | "browser") => refresh_quota(ui),
|
||||
Some("locations") => refresh_locations(ui),
|
||||
Some("activity") => refresh_activity(ui),
|
||||
_ => {}
|
||||
@@ -522,13 +580,11 @@ pub(crate) fn refresh(ui: &Rc<Ui>) {
|
||||
}
|
||||
|
||||
/// How long a quota reading stays fresh. Account storage barely moves, so the
|
||||
/// Settings tick refetches it only this often rather than every 2s.
|
||||
/// active-page tick refetches it only this often rather than every 2s.
|
||||
const QUOTA_TTL: Duration = Duration::from_secs(60);
|
||||
|
||||
/// Fetch the account quota (if the last reading is stale) and paint the Account
|
||||
/// storage group. Runs only while Settings is on screen. The group stays hidden
|
||||
/// until a reading lands, so an account the API can't report — or a still-starting
|
||||
/// daemon — shows nothing rather than an empty bar. A failed fetch leaves the last
|
||||
/// Fetch the account quota (if the last reading is stale) and paint both the
|
||||
/// Settings storage group and Files status bar. A failed fetch leaves the last
|
||||
/// good reading in place.
|
||||
pub(crate) fn refresh_quota(ui: &Rc<Ui>) {
|
||||
if ui.status.quota_inflight.get() {
|
||||
@@ -551,28 +607,71 @@ pub(crate) fn refresh_quota(ui: &Rc<Ui>) {
|
||||
})) = result
|
||||
{
|
||||
ui.status.quota_checked_at.set(Some(Instant::now()));
|
||||
let used = used_space.max(0) as u64;
|
||||
if max_space > 0 {
|
||||
let total = max_space as u64;
|
||||
let fraction = (used as f64 / total as f64).min(1.0);
|
||||
ui.status.quota_bar.set_fraction(fraction);
|
||||
let pct = (fraction * 100.0).round() as u64;
|
||||
ui.status.quota_label.set_text(&format!(
|
||||
"{} of {} used ({pct}%)",
|
||||
human_bytes(used),
|
||||
human_bytes(total)
|
||||
));
|
||||
} else {
|
||||
ui.status.quota_bar.set_fraction(0.0);
|
||||
ui.status
|
||||
.quota_label
|
||||
.set_text(&format!("{} used", human_bytes(used)));
|
||||
}
|
||||
paint_account_quota(&ui, max_space, used_space);
|
||||
ui.status.quota_group.set_visible(true);
|
||||
} else if ui.status.quota_checked_at.get().is_none() {
|
||||
// Match Dolphin: capacity information does not occupy the bar until
|
||||
// the backing observer has real figures.
|
||||
ui.browser.quota_box.set_visible(false);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
fn paint_account_quota(ui: &Rc<Ui>, max_space: i64, used_space: i64) {
|
||||
let (fraction, text) = quota_display(max_space, used_space);
|
||||
ui.status.quota_bar.set_fraction(fraction);
|
||||
ui.status.quota_label.set_text(&text);
|
||||
if let Some((fraction, free_text, tooltip)) = quota_status_display(max_space, used_space) {
|
||||
ui.browser.quota.set_fraction(fraction);
|
||||
ui.browser.quota.set_tooltip_text(Some(&tooltip));
|
||||
ui.browser.quota_text.set_label(&free_text);
|
||||
ui.browser.quota_text.set_tooltip_text(Some(&tooltip));
|
||||
ui.browser.quota_box.set_visible(true);
|
||||
} else {
|
||||
ui.browser.quota_box.set_visible(false);
|
||||
}
|
||||
}
|
||||
|
||||
fn quota_display(max_space: i64, used_space: i64) -> (f64, String) {
|
||||
let used = used_space.max(0) as u64;
|
||||
if max_space <= 0 {
|
||||
return (0.0, format!("{} used", human_bytes(used)));
|
||||
}
|
||||
let total = max_space as u64;
|
||||
let fraction = (used as f64 / total as f64).clamp(0.0, 1.0);
|
||||
let pct = (fraction * 100.0).round() as u64;
|
||||
(
|
||||
fraction,
|
||||
format!(
|
||||
"{} of {} used ({pct}%)",
|
||||
human_bytes(used),
|
||||
human_bytes(total)
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
/// Dolphin's status bar shows a bare capacity bar followed by “X free”; the
|
||||
/// full free/total/percentage sentence is a tooltip rather than inline bar text.
|
||||
fn quota_status_display(max_space: i64, used_space: i64) -> Option<(f64, String, String)> {
|
||||
if max_space <= 0 {
|
||||
return None;
|
||||
}
|
||||
let total = max_space as u64;
|
||||
let used = (used_space.max(0) as u64).min(total);
|
||||
let free = total.saturating_sub(used);
|
||||
let fraction = used as f64 / total as f64;
|
||||
let pct = (fraction * 100.0).round() as u64;
|
||||
Some((
|
||||
fraction,
|
||||
format!("{} free", human_bytes(free)),
|
||||
format!(
|
||||
"{} free out of {} ({pct}% used)",
|
||||
human_bytes(free),
|
||||
human_bytes(total)
|
||||
),
|
||||
))
|
||||
}
|
||||
|
||||
/// Record the mount state seen by the last status poll: gate every control that
|
||||
/// needs a live daemon, and notify the desktop when the state actually flips.
|
||||
///
|
||||
@@ -584,6 +683,11 @@ pub(crate) fn set_mounted(ui: &Rc<Ui>, mounted: bool) {
|
||||
ui.browser.new_folder.set_sensitive(mounted);
|
||||
ui.browser.upload.set_sensitive(mounted);
|
||||
ui.browser.upload_folder.set_sensitive(mounted);
|
||||
ui.browser.build_thumbnails.set_sensitive(
|
||||
mounted
|
||||
&& (!ui.browser.thumbnail_build_running.get()
|
||||
|| !ui.browser.thumbnail_cancel_pending.get()),
|
||||
);
|
||||
ui.gallery.upload.set_sensitive(mounted);
|
||||
ui.details.details.pin_row.set_sensitive(mounted);
|
||||
ui.details.details.rename_button.set_sensitive(mounted);
|
||||
@@ -653,6 +757,10 @@ pub(crate) fn repaint_transfers(ui: &Rc<Ui>, items: &[TransferItem], jobs: &[Job
|
||||
// count falling to zero is what "the sync finished" looks like from here.
|
||||
// Jobs are deliberately not counted — a bulk upload retires its scan job and
|
||||
// starts its upload job mid-flight, which is not a thing finishing.
|
||||
// The Import page reads its progress off the same snapshot rather than
|
||||
// polling the daemon a second time for numbers already on the wire.
|
||||
takeout_progress(ui, jobs);
|
||||
|
||||
let previous = ui.status.active_transfers.replace(items.len());
|
||||
if items.is_empty() && previous > 0 {
|
||||
let files = if previous == 1 {
|
||||
@@ -808,11 +916,14 @@ pub(crate) fn refresh_status(ui: &Rc<Ui>) {
|
||||
(used as f64 / budget as f64).min(1.0)
|
||||
};
|
||||
ui.status.cache_bar.set_fraction(fraction);
|
||||
ui.status.cache_label.set_text(&format!(
|
||||
"{} of {} used",
|
||||
human_bytes(used),
|
||||
human_bytes(budget)
|
||||
));
|
||||
// A 0 budget means *unlimited*, not a zero-byte cap — "of 0 B
|
||||
// used" reads as a broken read-out, and there is no fraction to
|
||||
// draw against no limit.
|
||||
ui.status.cache_label.set_text(&if budget == 0 {
|
||||
format!("{} cached — no limit set", human_bytes(used))
|
||||
} else {
|
||||
format!("{} of {} used", human_bytes(used), human_bytes(budget))
|
||||
});
|
||||
repaint_pins(&ui, &pins, true);
|
||||
}
|
||||
// Daemon unreachable (still starting, or down): report not-mounted and
|
||||
@@ -865,7 +976,15 @@ pub(crate) fn repaint_pins(ui: &Rc<Ui>, pins: &[pdfs_core::cache::Pin], mounted:
|
||||
return;
|
||||
}
|
||||
|
||||
for pin in pins {
|
||||
// Only the first page of pins is rendered; the rest sit behind the row
|
||||
// below, so a device with hundreds of pins doesn't bury the groups after it.
|
||||
let expanded = ui.status.pins_expanded.get();
|
||||
let shown = if expanded {
|
||||
pins.len()
|
||||
} else {
|
||||
pins.len().min(PINS_COLLAPSED)
|
||||
};
|
||||
for pin in &pins[..shown] {
|
||||
let name = Path::new(&pin.path)
|
||||
.file_name()
|
||||
.and_then(|n| n.to_str())
|
||||
@@ -903,4 +1022,70 @@ pub(crate) fn repaint_pins(ui: &Rc<Ui>, pins: &[pdfs_core::cache::Pin], mounted:
|
||||
unpin: Some(unpin),
|
||||
});
|
||||
}
|
||||
|
||||
if pins.len() > PINS_COLLAPSED {
|
||||
let row = adw::ActionRow::builder()
|
||||
.title(if expanded {
|
||||
"Show fewer".to_string()
|
||||
} else {
|
||||
format!("Show all {} pinned files", pins.len())
|
||||
})
|
||||
.activatable(true)
|
||||
.build();
|
||||
row.add_suffix(>k4::Image::from_icon_name(if expanded {
|
||||
"go-up-symbolic"
|
||||
} else {
|
||||
"go-down-symbolic"
|
||||
}));
|
||||
let ui_more = ui.clone();
|
||||
row.connect_activated(move |_| {
|
||||
ui_more
|
||||
.status
|
||||
.pins_expanded
|
||||
.set(!ui_more.status.pins_expanded.get());
|
||||
// The diff guard compares pin *paths*; the same paths render
|
||||
// differently now, so the baseline has to be dropped for the next
|
||||
// repaint to actually rebuild.
|
||||
*ui_more.status.pins_state.borrow_mut() = None;
|
||||
refresh(&ui_more);
|
||||
});
|
||||
ui.status.pins_group.add(&row);
|
||||
ui.status
|
||||
.pin_rows
|
||||
.borrow_mut()
|
||||
.push(PinRow { row, unpin: None });
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{quota_display, quota_status_display};
|
||||
|
||||
#[test]
|
||||
fn quota_display_reports_used_total_and_percentage() {
|
||||
let gib = 1024_i64.pow(3);
|
||||
let (fraction, text) = quota_display(4 * gib, gib);
|
||||
assert!((fraction - 0.25).abs() < f64::EPSILON);
|
||||
assert_eq!(text, "1.0 GiB of 4.0 GiB used (25%)");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn quota_display_clamps_bad_api_values() {
|
||||
assert_eq!(quota_display(0, -1), (0.0, "0 B used".to_string()));
|
||||
assert_eq!(quota_display(100, 150).0, 1.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn quota_status_display_matches_dolphin_wording() {
|
||||
let gib = 1024_i64.pow(3);
|
||||
assert_eq!(
|
||||
quota_status_display(4 * gib, gib),
|
||||
Some((
|
||||
0.25,
|
||||
"3.0 GiB free".to_string(),
|
||||
"3.0 GiB free out of 4.0 GiB (25% used)".to_string()
|
||||
))
|
||||
);
|
||||
assert_eq!(quota_status_display(0, 0), None);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,779 @@
|
||||
//! The Import page: bringing a Google Photos **Takeout** export into Proton
|
||||
//! Photos.
|
||||
//!
|
||||
//! Reached from Settings → *Import from Google Photos* (and from the Gallery's
|
||||
//! import button), because it is a one-off migration rather than something the
|
||||
//! photo timeline does every day. It gets a page of its own instead of a dialog
|
||||
//! for two reasons: an export is a *set* of archives that has to be staged and
|
||||
//! checked before anything is sent, and the import that follows runs for hours,
|
||||
//! so its progress needs somewhere to live that the user can navigate away from
|
||||
//! and come back to.
|
||||
//!
|
||||
//! Archives arrive either by drag-and-drop onto the page or through the file
|
||||
//! chooser; both funnel into [`add_archives`], which is where the `.zip` filter
|
||||
//! and de-duplication live. Nothing is sent until the user presses Scan or
|
||||
//! Import — staging is local, so a mis-drop costs nothing.
|
||||
//!
|
||||
//! The daemon owns the run itself ([`Request::ImportTakeout`]): this page only
|
||||
//! stages paths, launches, polls [`Request::ImportStatus`] for the final report,
|
||||
//! and reads live progress off the import's job in the transfer snapshot the
|
||||
//! Settings tick already fetches (see [`takeout_progress`]).
|
||||
|
||||
use crate::*;
|
||||
|
||||
/// The title the daemon gives the import's job in [`Request::GetQueueStatus`].
|
||||
/// Matching on it is how this page borrows the progress the Settings tick is
|
||||
/// already polling instead of adding a second poll of its own.
|
||||
const IMPORT_JOB: &str = "Importing Google Photos";
|
||||
|
||||
/// What the archive list says before anything is staged. `repaint_archives`
|
||||
/// swaps in the count and total once there is something to report, and swaps
|
||||
/// this back when the list is emptied again.
|
||||
/// Where the staged set is remembered between runs, under the app's state
|
||||
/// directory. Staging an export means picking dozens of files out of a file
|
||||
/// chooser; losing that to a window close (or a crash mid-import) and having to
|
||||
/// redo it is the kind of small cruelty that makes a migration not get finished.
|
||||
fn staged_path(ui: &Rc<Ui>) -> PathBuf {
|
||||
ui.dirs.state_dir().join("takeout-staged.json")
|
||||
}
|
||||
|
||||
/// Read the remembered set, dropping any archive that has since been moved or
|
||||
/// deleted — a stale path would only fail at launch, and silently is worse.
|
||||
fn load_staged(ui: &Rc<Ui>) -> Vec<PathBuf> {
|
||||
let Ok(text) = std::fs::read_to_string(staged_path(ui)) else {
|
||||
return Vec::new();
|
||||
};
|
||||
let paths: Vec<String> = serde_json::from_str(&text).unwrap_or_default();
|
||||
paths
|
||||
.into_iter()
|
||||
.map(PathBuf::from)
|
||||
.filter(|p| p.is_file())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Remember the staged set. Best-effort: failing to write it costs the user a
|
||||
/// re-pick next time, which is not worth interrupting them over now.
|
||||
fn save_staged(ui: &Rc<Ui>) {
|
||||
let paths: Vec<String> = ui
|
||||
.takeout
|
||||
.archives
|
||||
.borrow()
|
||||
.iter()
|
||||
.map(|p| p.display().to_string())
|
||||
.collect();
|
||||
let path = staged_path(ui);
|
||||
let written = serde_json::to_string(&paths)
|
||||
.map_err(|e| e.to_string())
|
||||
.and_then(|text| std::fs::write(&path, text).map_err(|e| e.to_string()));
|
||||
if let Err(e) = written {
|
||||
tracing::warn!("cannot remember staged Takeout archives: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
const EMPTY_LIST_DESCRIPTION: &str = "Photos already in Proton Photos are matched by name and content and skipped, so \
|
||||
re-running an interrupted import is safe.";
|
||||
|
||||
pub(crate) struct TakeoutState {
|
||||
/// Archives staged for this import, in the order they were added. Sent as
|
||||
/// one set: Google splits an export across numbered zips and routinely puts
|
||||
/// a photo in one part and its metadata sidecar in another, so a per-file
|
||||
/// import would lose album titles and capture times.
|
||||
pub(crate) archives: RefCell<Vec<PathBuf>>,
|
||||
/// The staged-archive list and the rows currently rendered in it.
|
||||
pub(crate) list_group: adw::PreferencesGroup,
|
||||
pub(crate) rows: RefCell<Vec<adw::PreferencesRow>>,
|
||||
/// The drop zone, whose `dropzone-active` class follows the drag.
|
||||
pub(crate) dropzone: gtk4::Box,
|
||||
pub(crate) scan_button: gtk4::Button,
|
||||
pub(crate) import_button: gtk4::Button,
|
||||
pub(crate) cancel_button: gtk4::Button,
|
||||
pub(crate) clear_button: gtk4::Button,
|
||||
/// Progress group, hidden unless an import is running.
|
||||
pub(crate) progress_group: adw::PreferencesGroup,
|
||||
pub(crate) progress_label: gtk4::Label,
|
||||
pub(crate) progress_bar: gtk4::ProgressBar,
|
||||
/// Report group for the last finished run, hidden until there is one.
|
||||
pub(crate) summary_group: adw::PreferencesGroup,
|
||||
pub(crate) summary_rows: RefCell<Vec<adw::ActionRow>>,
|
||||
/// One [`Request::ImportStatus`] in flight at a time, so a wedged daemon
|
||||
/// can't stack worker threads on the 2s tick.
|
||||
pub(crate) inflight: Cell<bool>,
|
||||
/// Whether the daemon reported an import running on the last poll.
|
||||
pub(crate) running: Cell<bool>,
|
||||
/// Whether the run in flight was launched as a dry run, so the report can
|
||||
/// say "would upload" rather than claiming photos were sent. The daemon's
|
||||
/// summary carries no such flag — it is the same shape either way.
|
||||
pub(crate) dry_run: Cell<bool>,
|
||||
}
|
||||
|
||||
/// Widgets the Import page hands back for wiring.
|
||||
pub(crate) struct TakeoutWidgets {
|
||||
pub(crate) list_group: adw::PreferencesGroup,
|
||||
pub(crate) dropzone: gtk4::Box,
|
||||
/// The whole page, which is the drop target: a drag released anywhere on it
|
||||
/// counts, not only inside the dashed rectangle.
|
||||
pub(crate) page: gtk4::Widget,
|
||||
pub(crate) choose_button: gtk4::Button,
|
||||
pub(crate) scan_button: gtk4::Button,
|
||||
pub(crate) import_button: gtk4::Button,
|
||||
pub(crate) cancel_button: gtk4::Button,
|
||||
pub(crate) clear_button: gtk4::Button,
|
||||
pub(crate) back_button: gtk4::Button,
|
||||
pub(crate) progress_group: adw::PreferencesGroup,
|
||||
pub(crate) progress_label: gtk4::Label,
|
||||
pub(crate) progress_bar: gtk4::ProgressBar,
|
||||
pub(crate) summary_group: adw::PreferencesGroup,
|
||||
}
|
||||
|
||||
pub(crate) fn build_takeout_page() -> (gtk4::Widget, TakeoutWidgets) {
|
||||
// Header: a back button out to Settings, since this page is reached from
|
||||
// there and has no sidebar row of its own to navigate back with.
|
||||
let back_button = gtk4::Button::builder()
|
||||
.icon_name("go-previous-symbolic")
|
||||
.tooltip_text("Back to Settings")
|
||||
.valign(gtk4::Align::Center)
|
||||
.build();
|
||||
back_button.add_css_class("flat");
|
||||
|
||||
let title = gtk4::Label::builder()
|
||||
.label("Import from Google Photos")
|
||||
.halign(gtk4::Align::Start)
|
||||
.build();
|
||||
title.add_css_class("title-2");
|
||||
let subtitle = gtk4::Label::builder()
|
||||
.label("Add a Google Takeout export to your Proton Photos timeline.")
|
||||
.halign(gtk4::Align::Start)
|
||||
.wrap(true)
|
||||
.xalign(0.0)
|
||||
.build();
|
||||
subtitle.add_css_class("dim-label");
|
||||
|
||||
let titles = gtk4::Box::new(gtk4::Orientation::Vertical, 2);
|
||||
titles.set_hexpand(true);
|
||||
titles.append(&title);
|
||||
titles.append(&subtitle);
|
||||
|
||||
let header = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
|
||||
header.append(&back_button);
|
||||
header.append(&titles);
|
||||
let header_clamp = adw::Clamp::builder().child(&header).build();
|
||||
|
||||
// Drop zone. It is a hint, not the only way in — the button under it does
|
||||
// the same thing for anyone not dragging, and the drop target covers the
|
||||
// whole page so a drag released just outside the rectangle still lands.
|
||||
let drop_icon = gtk4::Image::from_icon_name("folder-download-symbolic");
|
||||
drop_icon.set_pixel_size(48);
|
||||
drop_icon.add_css_class("dim-label");
|
||||
let drop_title = gtk4::Label::builder()
|
||||
.label("Drop your Takeout .zip files here")
|
||||
.build();
|
||||
drop_title.add_css_class("title-4");
|
||||
let drop_hint = gtk4::Label::builder()
|
||||
.label("Add every part of the export at once — a photo and its metadata often sit in different parts.")
|
||||
.wrap(true)
|
||||
.justify(gtk4::Justification::Center)
|
||||
.max_width_chars(48)
|
||||
.build();
|
||||
drop_hint.add_css_class("dim-label");
|
||||
let choose_button = gtk4::Button::builder()
|
||||
.label("Choose files…")
|
||||
.halign(gtk4::Align::Center)
|
||||
.build();
|
||||
choose_button.add_css_class("pill");
|
||||
|
||||
let dropzone = gtk4::Box::new(gtk4::Orientation::Vertical, 10);
|
||||
dropzone.set_halign(gtk4::Align::Fill);
|
||||
dropzone.add_css_class("dropzone");
|
||||
dropzone.append(&drop_icon);
|
||||
dropzone.append(&drop_title);
|
||||
dropzone.append(&drop_hint);
|
||||
dropzone.append(&choose_button);
|
||||
|
||||
// Staged archives. Filled by `repaint_archives`, which owns the description
|
||||
// too (it becomes the count + total once there is something to count); the
|
||||
// group carries a placeholder row while empty so it never renders as a bare
|
||||
// title.
|
||||
let list_group = adw::PreferencesGroup::builder()
|
||||
.title("Archives to import")
|
||||
.description(EMPTY_LIST_DESCRIPTION)
|
||||
.build();
|
||||
|
||||
let clear_button = gtk4::Button::builder()
|
||||
.label("Clear")
|
||||
.tooltip_text("Remove every staged archive")
|
||||
.build();
|
||||
clear_button.add_css_class("flat");
|
||||
|
||||
// Actions. Scan is a dry run: it reads the archives' directories and reports
|
||||
// what an import *would* do, without uploading a byte. It is offered first
|
||||
// because the import behind it is hours long and hard to take back.
|
||||
let scan_button = gtk4::Button::builder()
|
||||
.label("Scan")
|
||||
.tooltip_text("Check the archives and report what would be imported, without uploading")
|
||||
.build();
|
||||
scan_button.add_css_class("pill");
|
||||
let import_button = gtk4::Button::builder().label("Import").build();
|
||||
import_button.add_css_class("pill");
|
||||
import_button.add_css_class("suggested-action");
|
||||
let cancel_button = gtk4::Button::builder()
|
||||
.label("Stop import")
|
||||
.tooltip_text("Finish the photo on the wire, file what was uploaded, then stop")
|
||||
.build();
|
||||
cancel_button.add_css_class("pill");
|
||||
cancel_button.add_css_class("destructive-action");
|
||||
cancel_button.set_visible(false);
|
||||
|
||||
// Clear sits apart from the launch buttons, on the other end of the row: it
|
||||
// undoes staging, and putting it next to Import invites the wrong click.
|
||||
let actions = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
|
||||
let spacer = gtk4::Box::new(gtk4::Orientation::Horizontal, 0);
|
||||
spacer.set_hexpand(true);
|
||||
actions.append(&clear_button);
|
||||
actions.append(&spacer);
|
||||
actions.append(&cancel_button);
|
||||
actions.append(&scan_button);
|
||||
actions.append(&import_button);
|
||||
|
||||
// Progress: the import's job from the transfer snapshot, so this page shows
|
||||
// the same numbers the Settings Activity list does.
|
||||
let progress_group = adw::PreferencesGroup::builder()
|
||||
.title("Import in progress")
|
||||
.visible(false)
|
||||
.build();
|
||||
let progress_box = gtk4::Box::new(gtk4::Orientation::Vertical, 6);
|
||||
progress_box.set_margin_top(8);
|
||||
progress_box.set_margin_bottom(8);
|
||||
let progress_label = gtk4::Label::builder().halign(gtk4::Align::Start).build();
|
||||
progress_label.add_css_class("dim-label");
|
||||
let progress_bar = gtk4::ProgressBar::new();
|
||||
progress_box.append(&progress_label);
|
||||
progress_box.append(&progress_bar);
|
||||
progress_group.add(
|
||||
&adw::PreferencesRow::builder()
|
||||
.activatable(false)
|
||||
.child(&progress_box)
|
||||
.build(),
|
||||
);
|
||||
|
||||
// Report for the last finished run, kept on screen after it ends: an import
|
||||
// outlives the window being on this page, and "what happened" is the whole
|
||||
// reason to come back to it.
|
||||
let summary_group = adw::PreferencesGroup::builder()
|
||||
.title("Last import")
|
||||
.visible(false)
|
||||
.build();
|
||||
|
||||
let groups = gtk4::Box::new(gtk4::Orientation::Vertical, 18);
|
||||
groups.append(&dropzone);
|
||||
groups.append(&list_group);
|
||||
groups.append(&actions);
|
||||
groups.append(&progress_group);
|
||||
groups.append(&summary_group);
|
||||
let clamp = adw::Clamp::builder()
|
||||
.maximum_size(640)
|
||||
.child(&groups)
|
||||
.build();
|
||||
let scroll = gtk4::ScrolledWindow::builder()
|
||||
.vexpand(true)
|
||||
.hscrollbar_policy(gtk4::PolicyType::Never)
|
||||
.child(&clamp)
|
||||
.build();
|
||||
|
||||
let page = gtk4::Box::new(gtk4::Orientation::Vertical, 12);
|
||||
page.set_margin_top(18);
|
||||
page.set_margin_bottom(18);
|
||||
page.set_margin_start(18);
|
||||
page.set_margin_end(18);
|
||||
page.append(&header_clamp);
|
||||
page.append(&scroll);
|
||||
|
||||
let widgets = TakeoutWidgets {
|
||||
list_group: list_group.clone(),
|
||||
dropzone: dropzone.clone(),
|
||||
page: page.clone().upcast(),
|
||||
choose_button,
|
||||
scan_button: scan_button.clone(),
|
||||
import_button: import_button.clone(),
|
||||
cancel_button: cancel_button.clone(),
|
||||
clear_button,
|
||||
back_button,
|
||||
progress_group: progress_group.clone(),
|
||||
progress_label: progress_label.clone(),
|
||||
progress_bar: progress_bar.clone(),
|
||||
summary_group: summary_group.clone(),
|
||||
};
|
||||
(page.upcast(), widgets)
|
||||
}
|
||||
|
||||
pub(crate) fn wire_takeout(ui: &Rc<Ui>, widgets: &TakeoutWidgets) {
|
||||
// Drag-and-drop. The target sits on the whole page rather than the dashed
|
||||
// rectangle, so a drag released anywhere counts; the rectangle only lights
|
||||
// up to say where the drop is understood.
|
||||
let target = gtk4::DropTarget::new(
|
||||
gtk4::gdk::FileList::static_type(),
|
||||
gtk4::gdk::DragAction::COPY,
|
||||
);
|
||||
let ui_enter = ui.clone();
|
||||
target.connect_enter(move |_, _, _| {
|
||||
ui_enter.takeout.dropzone.add_css_class("dropzone-active");
|
||||
gtk4::gdk::DragAction::COPY
|
||||
});
|
||||
let ui_leave = ui.clone();
|
||||
target.connect_leave(move |_| {
|
||||
ui_leave
|
||||
.takeout
|
||||
.dropzone
|
||||
.remove_css_class("dropzone-active");
|
||||
});
|
||||
let ui_drop = ui.clone();
|
||||
target.connect_drop(move |_, value, _, _| {
|
||||
ui_drop.takeout.dropzone.remove_css_class("dropzone-active");
|
||||
let Ok(files) = value.get::<gtk4::gdk::FileList>() else {
|
||||
return false;
|
||||
};
|
||||
let paths: Vec<PathBuf> = files.files().iter().filter_map(|f| f.path()).collect();
|
||||
add_archives(&ui_drop, paths);
|
||||
true
|
||||
});
|
||||
widgets.page.add_controller(target);
|
||||
|
||||
let ui_choose = ui.clone();
|
||||
widgets.choose_button.connect_clicked(move |_| {
|
||||
// Multi-select on purpose: the whole export goes in one request.
|
||||
let dialog = gtk4::FileDialog::builder()
|
||||
.title("Select your Google Takeout archives")
|
||||
.build();
|
||||
let filter = gtk4::FileFilter::new();
|
||||
filter.set_name(Some("Takeout archives"));
|
||||
filter.add_pattern("*.zip");
|
||||
let filters = gio::ListStore::new::<gtk4::FileFilter>();
|
||||
filters.append(&filter);
|
||||
dialog.set_filters(Some(&filters));
|
||||
|
||||
let ui = ui_choose.clone();
|
||||
dialog.open_multiple(
|
||||
ui_window(&ui_choose).as_ref(),
|
||||
gio::Cancellable::NONE,
|
||||
move |res| {
|
||||
let Ok(files) = res else { return };
|
||||
let paths: Vec<PathBuf> = (0..files.n_items())
|
||||
.filter_map(|index| files.item(index))
|
||||
.filter_map(|object| object.downcast::<gio::File>().ok())
|
||||
.filter_map(|file| file.path())
|
||||
.collect();
|
||||
add_archives(&ui, paths);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
let ui_clear = ui.clone();
|
||||
widgets.clear_button.connect_clicked(move |_| {
|
||||
ui_clear.takeout.archives.borrow_mut().clear();
|
||||
repaint_archives(&ui_clear);
|
||||
});
|
||||
|
||||
let ui_scan = ui.clone();
|
||||
widgets
|
||||
.scan_button
|
||||
.connect_clicked(move |_| start_import(&ui_scan, true));
|
||||
|
||||
// The import itself is confirmed: it uploads a whole photo library, which is
|
||||
// hours of bandwidth and a large change to the account's timeline.
|
||||
let ui_import = ui.clone();
|
||||
widgets.import_button.connect_clicked(move |_| {
|
||||
let count = ui_import.takeout.archives.borrow().len();
|
||||
let ui = ui_import.clone();
|
||||
let dialog = adw::AlertDialog::builder()
|
||||
.heading("Import this export?")
|
||||
.body(format!(
|
||||
"{count} archive(s) will be uploaded to Proton Photos, creating albums as they \
|
||||
appear in the export. This can take hours; you can stop it at any point and \
|
||||
what has been uploaded stays. Photos you already have are skipped.",
|
||||
))
|
||||
.build();
|
||||
dialog.add_response("cancel", "Cancel");
|
||||
dialog.add_response("import", "Import");
|
||||
dialog.set_response_appearance("import", adw::ResponseAppearance::Suggested);
|
||||
dialog.set_default_response(Some("cancel"));
|
||||
dialog.set_close_response("cancel");
|
||||
dialog.connect_response(None, move |_, resp| {
|
||||
if resp == "import" {
|
||||
start_import(&ui, false);
|
||||
}
|
||||
});
|
||||
dialog.present(ui_window(&ui_import).as_ref());
|
||||
});
|
||||
|
||||
let ui_cancel = ui.clone();
|
||||
widgets.cancel_button.connect_clicked(move |_| {
|
||||
let ui = ui_cancel.clone();
|
||||
ui.busy_begin();
|
||||
let rx = spawn_request(ui.dirs.control_socket(), Request::CancelImport);
|
||||
glib::spawn_future_local(async move {
|
||||
let result = rx.recv().await;
|
||||
ui.busy_end();
|
||||
match result {
|
||||
Ok(Ok(Response::Ok { .. })) => toast(&ui, "Stopping the import…"),
|
||||
Ok(Ok(Response::Error { message, kind })) => {
|
||||
toast_failure(&ui, "Couldn't stop the import", &message, kind);
|
||||
}
|
||||
_ => toast_error(
|
||||
&ui,
|
||||
"Couldn't stop the import",
|
||||
"The mount service didn't respond.",
|
||||
),
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
let ui_back = ui.clone();
|
||||
widgets
|
||||
.back_button
|
||||
.connect_clicked(move |_| ui_back.stack.set_visible_child_name("main"));
|
||||
|
||||
// Restore whatever was staged when the window last closed, before the first
|
||||
// paint, so the page comes up as the user left it.
|
||||
*ui.takeout.archives.borrow_mut() = load_staged(ui);
|
||||
repaint_archives(ui);
|
||||
}
|
||||
|
||||
/// Stage `paths`, keeping only `.zip` files and skipping ones already staged.
|
||||
///
|
||||
/// Both the chooser and the drop target land here, so the filter is stated once.
|
||||
/// A dropped folder (an already-extracted export) is rejected rather than walked:
|
||||
/// the scan reads the archives' central directories, and an extracted tree has
|
||||
/// none — silently importing half of it would be worse than saying no.
|
||||
pub(crate) fn add_archives(ui: &Rc<Ui>, paths: Vec<PathBuf>) {
|
||||
let mut added = 0usize;
|
||||
let mut rejected = 0usize;
|
||||
{
|
||||
let mut staged = ui.takeout.archives.borrow_mut();
|
||||
for path in paths {
|
||||
let is_zip = path
|
||||
.extension()
|
||||
.is_some_and(|ext| ext.eq_ignore_ascii_case("zip"));
|
||||
if !is_zip || !path.is_file() {
|
||||
rejected += 1;
|
||||
continue;
|
||||
}
|
||||
if staged.contains(&path) {
|
||||
continue;
|
||||
}
|
||||
staged.push(path);
|
||||
added += 1;
|
||||
}
|
||||
// Numbered parts drop in whatever order the file manager hands them
|
||||
// over; sorting keeps the list readable and the request stable.
|
||||
staged.sort();
|
||||
}
|
||||
repaint_archives(ui);
|
||||
if rejected > 0 {
|
||||
toast(
|
||||
ui,
|
||||
"Only Takeout .zip files can be imported — extracted folders aren't supported.",
|
||||
);
|
||||
} else if added > 0 {
|
||||
toast(ui, &format!("Added {added} archive(s)"));
|
||||
}
|
||||
}
|
||||
|
||||
/// Rebuild the staged-archive list and re-derive which actions are available.
|
||||
pub(crate) fn repaint_archives(ui: &Rc<Ui>) {
|
||||
for row in ui.takeout.rows.borrow_mut().drain(..) {
|
||||
ui.takeout.list_group.remove(&row);
|
||||
}
|
||||
|
||||
let staged = ui.takeout.archives.borrow().clone();
|
||||
let mut rows: Vec<adw::PreferencesRow> = Vec::new();
|
||||
if staged.is_empty() {
|
||||
ui.takeout
|
||||
.list_group
|
||||
.set_description(Some(EMPTY_LIST_DESCRIPTION));
|
||||
let row = dim_row("No archives added yet.");
|
||||
ui.takeout.list_group.add(&row);
|
||||
rows.push(row.upcast());
|
||||
} else {
|
||||
let mut total = 0u64;
|
||||
for path in &staged {
|
||||
let size = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0);
|
||||
total += size;
|
||||
let row = adw::ActionRow::builder()
|
||||
.title(
|
||||
path.file_name()
|
||||
.map(|n| n.to_string_lossy().to_string())
|
||||
.unwrap_or_else(|| path.display().to_string()),
|
||||
)
|
||||
.subtitle(human_bytes(size))
|
||||
.build();
|
||||
let remove = gtk4::Button::builder()
|
||||
.icon_name("edit-delete-symbolic")
|
||||
.tooltip_text("Remove from this import")
|
||||
.valign(gtk4::Align::Center)
|
||||
.build();
|
||||
remove.add_css_class("flat");
|
||||
let ui_remove = ui.clone();
|
||||
let target = path.clone();
|
||||
remove.connect_clicked(move |_| {
|
||||
ui_remove
|
||||
.takeout
|
||||
.archives
|
||||
.borrow_mut()
|
||||
.retain(|p| *p != target);
|
||||
repaint_archives(&ui_remove);
|
||||
});
|
||||
row.add_suffix(&remove);
|
||||
ui.takeout.list_group.add(&row);
|
||||
rows.push(row.upcast());
|
||||
}
|
||||
ui.takeout.list_group.set_description(Some(&format!(
|
||||
"{} archive(s), {} in total. {EMPTY_LIST_DESCRIPTION}",
|
||||
staged.len(),
|
||||
human_bytes(total)
|
||||
)));
|
||||
}
|
||||
*ui.takeout.rows.borrow_mut() = rows;
|
||||
|
||||
save_staged(ui);
|
||||
sync_takeout_actions(ui);
|
||||
}
|
||||
|
||||
/// Enable exactly the actions that make sense right now: nothing to launch
|
||||
/// without archives, nothing to launch while a run is in flight, and Stop only
|
||||
/// while there is something to stop.
|
||||
fn sync_takeout_actions(ui: &Rc<Ui>) {
|
||||
let has_files = !ui.takeout.archives.borrow().is_empty();
|
||||
let running = ui.takeout.running.get();
|
||||
ui.takeout.scan_button.set_sensitive(has_files && !running);
|
||||
ui.takeout
|
||||
.import_button
|
||||
.set_sensitive(has_files && !running);
|
||||
ui.takeout.clear_button.set_sensitive(has_files && !running);
|
||||
ui.takeout.cancel_button.set_visible(running);
|
||||
}
|
||||
|
||||
/// Hand the staged set to the daemon. It acks immediately and works in the
|
||||
/// background — an export is far past any socket timeout — so success here means
|
||||
/// "started", and the page switches over to reporting progress.
|
||||
fn start_import(ui: &Rc<Ui>, dry_run: bool) {
|
||||
let archives: Vec<String> = ui
|
||||
.takeout
|
||||
.archives
|
||||
.borrow()
|
||||
.iter()
|
||||
.map(|p| p.display().to_string())
|
||||
.collect();
|
||||
if archives.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
ui.takeout.dry_run.set(dry_run);
|
||||
ui.busy_begin();
|
||||
let rx = spawn_request(
|
||||
ui.dirs.control_socket(),
|
||||
Request::ImportTakeout { archives, dry_run },
|
||||
);
|
||||
let ui = ui.clone();
|
||||
glib::spawn_future_local(async move {
|
||||
let result = rx.recv().await;
|
||||
ui.busy_end();
|
||||
match result {
|
||||
Ok(Ok(Response::Ok { .. })) => {
|
||||
ui.takeout.running.set(true);
|
||||
ui.takeout.summary_group.set_visible(false);
|
||||
ui.takeout.progress_group.set_visible(true);
|
||||
ui.takeout.progress_bar.set_fraction(0.0);
|
||||
ui.takeout.progress_label.set_text(if dry_run {
|
||||
"Reading archives…"
|
||||
} else {
|
||||
"Starting import…"
|
||||
});
|
||||
sync_takeout_actions(&ui);
|
||||
toast(
|
||||
&ui,
|
||||
if dry_run {
|
||||
"Scanning the export…"
|
||||
} else {
|
||||
"Import started — you can leave this page, it keeps running."
|
||||
},
|
||||
);
|
||||
}
|
||||
Ok(Ok(Response::Error { message, kind })) => {
|
||||
toast_failure(&ui, "Couldn't start the import", &message, kind);
|
||||
}
|
||||
_ => toast_error(
|
||||
&ui,
|
||||
"Couldn't start the import",
|
||||
"The mount service didn't respond.",
|
||||
),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Poll the daemon for the import's state while the page is on screen, and paint
|
||||
/// the report once a run has finished. Cheap and inflight-guarded, like the other
|
||||
/// tick polls.
|
||||
pub(crate) fn refresh_takeout(ui: &Rc<Ui>) {
|
||||
if ui.takeout.inflight.get() {
|
||||
return;
|
||||
}
|
||||
ui.takeout.inflight.set(true);
|
||||
let rx = spawn_request(ui.dirs.control_socket(), Request::ImportStatus);
|
||||
let ui = ui.clone();
|
||||
glib::spawn_future_local(async move {
|
||||
let result = rx.recv().await;
|
||||
ui.takeout.inflight.set(false);
|
||||
let Ok(Ok(Response::ImportStatus { running, summary })) = result else {
|
||||
return;
|
||||
};
|
||||
let was_running = ui.takeout.running.replace(running);
|
||||
if running {
|
||||
ui.takeout.progress_group.set_visible(true);
|
||||
} else {
|
||||
ui.takeout.progress_group.set_visible(false);
|
||||
// An import runs for hours, so its ending is announced rather than
|
||||
// left to be discovered: a toast for whoever is looking at the
|
||||
// window, and a desktop notification for whoever is not.
|
||||
if was_running {
|
||||
let dry = ui.takeout.dry_run.get();
|
||||
let headline = if dry {
|
||||
"Scan finished"
|
||||
} else {
|
||||
"Import finished"
|
||||
};
|
||||
toast(&ui, headline);
|
||||
if !dry {
|
||||
let body = match &summary {
|
||||
Some(s) if s.failed > 0 => {
|
||||
format!("{} photos added, {} failed.", s.uploaded, s.failed)
|
||||
}
|
||||
Some(s) if s.cancelled => {
|
||||
format!("Stopped after {} photos.", s.uploaded)
|
||||
}
|
||||
Some(s) => format!("{} photos added to Proton Photos.", s.uploaded),
|
||||
None => "The Google Photos import has finished.".to_string(),
|
||||
};
|
||||
notify("takeout-import", headline, &body);
|
||||
}
|
||||
}
|
||||
}
|
||||
if was_running != running {
|
||||
sync_takeout_actions(&ui);
|
||||
}
|
||||
match summary {
|
||||
Some(summary) => repaint_summary(&ui, &summary),
|
||||
None => ui.takeout.summary_group.set_visible(false),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Paint the last run's report.
|
||||
fn repaint_summary(ui: &Rc<Ui>, summary: &ImportSummary) {
|
||||
for row in ui.takeout.summary_rows.borrow_mut().drain(..) {
|
||||
ui.takeout.summary_group.remove(&row);
|
||||
}
|
||||
let dry = ui.takeout.dry_run.get();
|
||||
ui.takeout
|
||||
.summary_group
|
||||
.set_title(if dry { "Scan result" } else { "Last import" });
|
||||
ui.takeout.summary_group.set_description(Some(if dry {
|
||||
"Nothing was uploaded — this is what an import would do."
|
||||
} else if summary.cancelled {
|
||||
"Stopped early. Everything uploaded so far is in your timeline; run it again to \
|
||||
continue where it left off."
|
||||
} else {
|
||||
"Finished."
|
||||
}));
|
||||
|
||||
// Only lines that carry information: a run with no failures should not have
|
||||
// to show a "0 failed" row for the user to read past.
|
||||
let mut lines: Vec<(String, String)> = vec![
|
||||
("Photos in the export".into(), summary.found.to_string()),
|
||||
(
|
||||
if dry { "Would upload" } else { "Uploaded" }.into(),
|
||||
summary.uploaded.to_string(),
|
||||
),
|
||||
];
|
||||
if summary.duplicates > 0 {
|
||||
lines.push((
|
||||
"Already in Proton Photos".into(),
|
||||
summary.duplicates.to_string(),
|
||||
));
|
||||
}
|
||||
if summary.albums_created > 0 {
|
||||
lines.push((
|
||||
if dry {
|
||||
"Albums to create"
|
||||
} else {
|
||||
"Albums created"
|
||||
}
|
||||
.into(),
|
||||
summary.albums_created.to_string(),
|
||||
));
|
||||
}
|
||||
if summary.album_links > 0 {
|
||||
lines.push((
|
||||
"Photos filed into albums".into(),
|
||||
summary.album_links.to_string(),
|
||||
));
|
||||
}
|
||||
if summary.skipped_trashed > 0 {
|
||||
lines.push((
|
||||
"Skipped (in Google's trash)".into(),
|
||||
summary.skipped_trashed.to_string(),
|
||||
));
|
||||
}
|
||||
if summary.bytes > 0 {
|
||||
lines.push((
|
||||
if dry {
|
||||
"Data to upload"
|
||||
} else {
|
||||
"Data uploaded"
|
||||
}
|
||||
.into(),
|
||||
human_bytes(summary.bytes),
|
||||
));
|
||||
}
|
||||
if summary.failed > 0 {
|
||||
lines.push(("Failed".into(), summary.failed.to_string()));
|
||||
}
|
||||
|
||||
let mut rows = Vec::new();
|
||||
for (title, value) in lines {
|
||||
let row = adw::ActionRow::builder()
|
||||
.title(title)
|
||||
.subtitle(value)
|
||||
.build();
|
||||
row.add_css_class("property");
|
||||
ui.takeout.summary_group.add(&row);
|
||||
rows.push(row);
|
||||
}
|
||||
*ui.takeout.summary_rows.borrow_mut() = rows;
|
||||
ui.takeout.summary_group.set_visible(true);
|
||||
}
|
||||
|
||||
/// Paint the import's progress from the transfer snapshot the Settings tick
|
||||
/// already fetched, so this page costs no extra poll. A job with no known total
|
||||
/// pulses instead of showing a fraction, the same as the Activity list.
|
||||
pub(crate) fn takeout_progress(ui: &Rc<Ui>, jobs: &[JobItem]) {
|
||||
let Some(job) = jobs.iter().find(|job| job.title == IMPORT_JOB) else {
|
||||
return;
|
||||
};
|
||||
ui.takeout.progress_group.set_visible(true);
|
||||
if job.total > 0 {
|
||||
ui.takeout
|
||||
.progress_bar
|
||||
.set_fraction((job.done as f64 / job.total as f64).min(1.0));
|
||||
ui.takeout
|
||||
.progress_label
|
||||
.set_text(&format!("{} of {} — {}", job.done, job.total, job.detail));
|
||||
} else {
|
||||
ui.takeout.progress_bar.pulse();
|
||||
ui.takeout
|
||||
.progress_label
|
||||
.set_text(if job.detail.is_empty() {
|
||||
"Working…"
|
||||
} else {
|
||||
&job.detail
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -122,7 +122,7 @@ pub(crate) fn wire_trash(ui: &Rc<Ui>, list: >k4::ListView, empty: >k4::Butto
|
||||
factory.connect_setup(move |_, item| {
|
||||
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
|
||||
|
||||
let icon = gtk4::Image::builder().pixel_size(32).build();
|
||||
let thumbnail = file_thumbnail_widget(40, 32);
|
||||
let name = gtk4::Label::builder()
|
||||
.halign(gtk4::Align::Start)
|
||||
.ellipsize(gtk4::pango::EllipsizeMode::Middle)
|
||||
@@ -169,35 +169,38 @@ pub(crate) fn wire_trash(ui: &Rc<Ui>, list: >k4::ListView, empty: >k4::Butto
|
||||
row.set_margin_bottom(6);
|
||||
row.set_margin_start(6);
|
||||
row.set_margin_end(6);
|
||||
row.append(&icon);
|
||||
row.append(&thumbnail);
|
||||
row.append(&text);
|
||||
row.append(&restore);
|
||||
row.append(&purge);
|
||||
item.set_child(Some(&row));
|
||||
});
|
||||
factory.connect_bind(|_, item| {
|
||||
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
|
||||
let Some(entry) = bound_entry(item) else {
|
||||
return;
|
||||
};
|
||||
let row = item.child().and_downcast::<gtk4::Box>().unwrap();
|
||||
let Some(icon) = row.first_child().and_downcast::<gtk4::Image>() else {
|
||||
return;
|
||||
};
|
||||
icon.set_icon_name(Some(&format!("{}-symbolic", icon_base_for(&entry))));
|
||||
let Some(text) = icon.next_sibling().and_downcast::<gtk4::Box>() else {
|
||||
return;
|
||||
};
|
||||
if let Some(name) = text.first_child().and_downcast::<gtk4::Label>() {
|
||||
name.set_label(&entry.name);
|
||||
}
|
||||
if let Some(meta) = text.last_child().and_downcast::<gtk4::Label>() {
|
||||
let kind = if entry.is_dir {
|
||||
"Folder".to_string()
|
||||
} else {
|
||||
human_bytes(entry.size)
|
||||
factory.connect_bind({
|
||||
let ui = ui.clone();
|
||||
move |_, item| {
|
||||
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
|
||||
let Some(entry) = bound_entry(item) else {
|
||||
return;
|
||||
};
|
||||
meta.set_label(&format!("{kind} · {}", format_modified(entry.modified)));
|
||||
let row = item.child().and_downcast::<gtk4::Box>().unwrap();
|
||||
let Some(thumbnail) = row.first_child().and_downcast::<gtk4::Overlay>() else {
|
||||
return;
|
||||
};
|
||||
bind_file_thumbnail(&ui, &thumbnail, &entry, true);
|
||||
let Some(text) = thumbnail.next_sibling().and_downcast::<gtk4::Box>() else {
|
||||
return;
|
||||
};
|
||||
if let Some(name) = text.first_child().and_downcast::<gtk4::Label>() {
|
||||
name.set_label(&entry.name);
|
||||
}
|
||||
if let Some(meta) = text.last_child().and_downcast::<gtk4::Label>() {
|
||||
let kind = if entry.is_dir {
|
||||
"Folder".to_string()
|
||||
} else {
|
||||
human_bytes(entry.size)
|
||||
};
|
||||
meta.set_label(&format!("{kind} · {}", format_modified(entry.modified)));
|
||||
}
|
||||
}
|
||||
});
|
||||
list.set_factory(Some(&factory));
|
||||
@@ -216,6 +219,7 @@ pub(crate) fn bound_entry(item: >k4::ListItem) -> Option<DirEntry> {
|
||||
|
||||
/// Fetch the trash listing and repaint the page.
|
||||
pub(crate) fn load_trash(ui: &Rc<Ui>) {
|
||||
cancel_file_thumbnails(ui);
|
||||
// Drop the old rows first: a stale row here would offer Restore on something
|
||||
// that may already be gone.
|
||||
ui.trash.model.remove_all();
|
||||
|
||||
@@ -11,8 +11,8 @@ pub(crate) struct DetailsState {
|
||||
pub(crate) details_suppress: Cell<bool>,
|
||||
/// The grid's and list's selection models. Both wrap the one browser model, so
|
||||
/// a selection in either drives the details pane.
|
||||
pub(crate) grid_selection: gtk4::SingleSelection,
|
||||
pub(crate) list_selection: gtk4::SingleSelection,
|
||||
pub(crate) grid_selection: gtk4::MultiSelection,
|
||||
pub(crate) list_selection: gtk4::MultiSelection,
|
||||
}
|
||||
|
||||
/// The widgets in the browser's details pane that a selection repaints.
|
||||
@@ -158,24 +158,22 @@ pub(crate) fn wire_details(ui: &Rc<Ui>) {
|
||||
ui.details.list_selection.clone(),
|
||||
] {
|
||||
let ui_sel = ui.clone();
|
||||
selection.connect_selection_changed(move |sel, _, _| {
|
||||
match entry_at(sel.model().as_ref(), sel.selected()) {
|
||||
Some(entry) => show_details(&ui_sel, &entry),
|
||||
None => hide_details(&ui_sel),
|
||||
selection.connect_selection_changed(move |_, _, _| {
|
||||
// The details pane describes *one* entry. A batch is the bulk bar's
|
||||
// business, so the pane steps aside rather than picking a member of
|
||||
// the selection to speak for the rest.
|
||||
let entries = selected_entries(&ui_sel);
|
||||
match entries.as_slice() {
|
||||
[entry] => show_details(&ui_sel, entry),
|
||||
_ => hide_details(&ui_sel),
|
||||
}
|
||||
sync_bulk_bar(&ui_sel);
|
||||
});
|
||||
}
|
||||
|
||||
let ui_close = ui.clone();
|
||||
ui.details.details.close_button.connect_clicked(move |_| {
|
||||
ui_close
|
||||
.details
|
||||
.grid_selection
|
||||
.set_selected(gtk4::INVALID_LIST_POSITION);
|
||||
ui_close
|
||||
.details
|
||||
.list_selection
|
||||
.set_selected(gtk4::INVALID_LIST_POSITION);
|
||||
clear_selection(&ui_close);
|
||||
hide_details(&ui_close);
|
||||
});
|
||||
|
||||
@@ -289,8 +287,9 @@ pub(crate) fn hide_details(ui: &Rc<Ui>) {
|
||||
*ui.details.details_entry.borrow_mut() = None;
|
||||
}
|
||||
|
||||
/// The entry highlighted in whichever browser view is on screen, if any. Backs the
|
||||
/// F2 / Delete shortcuts.
|
||||
/// The single entry the details pane is showing, if any. Backs the actions that
|
||||
/// only make sense on one entry (rename, versions); the batch-capable ones ask
|
||||
/// [`selected_entries`] instead.
|
||||
pub(crate) fn selected_entry(ui: &Rc<Ui>) -> Option<DirEntry> {
|
||||
ui.details.details_entry.borrow().clone()
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
pub(crate) mod details;
|
||||
pub(crate) mod share_dialog;
|
||||
pub(crate) mod thumbnails;
|
||||
pub(crate) mod versions_dialog;
|
||||
@@ -0,0 +1,469 @@
|
||||
//! Shared thumbnails for ordinary Drive image files.
|
||||
//!
|
||||
//! The Photos gallery has timeline-specific batching and aspect-ratio state.
|
||||
//! File listings need a smaller, reusable surface: one loader backs My files,
|
||||
//! search, Shared and Trash, with modification-time cache keys
|
||||
//! so a replaced image never flashes its previous contents.
|
||||
|
||||
use crate::*;
|
||||
use pdfs_core::control::{FileThumbRequest, is_thumbnail_image_name};
|
||||
|
||||
const THUMB_BATCH: usize = 32;
|
||||
const THUMB_CACHE_MAX: usize = 256;
|
||||
const THUMB_MISSING_MAX: usize = 2048;
|
||||
const THUMB_DEBOUNCE: Duration = Duration::from_millis(40);
|
||||
const THUMB_RETRY: Duration = Duration::from_secs(2);
|
||||
|
||||
pub(crate) struct FileThumbnailState {
|
||||
textures: RefCell<HashMap<String, gtk4::gdk::Texture>>,
|
||||
texture_order: RefCell<VecDeque<String>>,
|
||||
missing: RefCell<HashSet<String>>,
|
||||
missing_order: RefCell<VecDeque<String>>,
|
||||
wanted: RefCell<HashMap<String, Vec<gtk4::Picture>>>,
|
||||
queue: RefCell<VecDeque<FileThumbRequest>>,
|
||||
inflight: Cell<bool>,
|
||||
source: RefCell<Option<glib::SourceId>>,
|
||||
generation: Cell<u64>,
|
||||
generation_loading: Cell<bool>,
|
||||
}
|
||||
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
enum ReplyDisposition {
|
||||
Ignore,
|
||||
Load,
|
||||
Retry,
|
||||
Missing,
|
||||
}
|
||||
|
||||
fn reply_disposition(wanted: bool, has_path: bool, pending: bool) -> ReplyDisposition {
|
||||
if !wanted {
|
||||
ReplyDisposition::Ignore
|
||||
} else if has_path {
|
||||
ReplyDisposition::Load
|
||||
} else if pending {
|
||||
ReplyDisposition::Retry
|
||||
} else {
|
||||
ReplyDisposition::Missing
|
||||
}
|
||||
}
|
||||
|
||||
impl FileThumbnailState {
|
||||
pub(crate) fn new() -> Self {
|
||||
Self {
|
||||
textures: RefCell::new(HashMap::new()),
|
||||
texture_order: RefCell::new(VecDeque::new()),
|
||||
missing: RefCell::new(HashSet::new()),
|
||||
missing_order: RefCell::new(VecDeque::new()),
|
||||
wanted: RefCell::new(HashMap::new()),
|
||||
queue: RefCell::new(VecDeque::new()),
|
||||
inflight: Cell::new(false),
|
||||
source: RefCell::new(None),
|
||||
generation: Cell::new(0),
|
||||
generation_loading: Cell::new(false),
|
||||
}
|
||||
}
|
||||
|
||||
fn store_texture(&self, key: &str, texture: gtk4::gdk::Texture) {
|
||||
let mut textures = self.textures.borrow_mut();
|
||||
let mut order = self.texture_order.borrow_mut();
|
||||
if textures.insert(key.to_string(), texture).is_none() {
|
||||
order.push_back(key.to_string());
|
||||
}
|
||||
while order.len() > THUMB_CACHE_MAX {
|
||||
if let Some(old) = order.pop_front() {
|
||||
textures.remove(&old);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn store_missing(&self, key: String) {
|
||||
let mut missing = self.missing.borrow_mut();
|
||||
let mut order = self.missing_order.borrow_mut();
|
||||
if missing.insert(key.clone()) {
|
||||
order.push_back(key);
|
||||
}
|
||||
while order.len() > THUMB_MISSING_MAX {
|
||||
if let Some(old) = order.pop_front() {
|
||||
missing.remove(&old);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Abandon every opportunistic ordinary-file thumbnail associated with the
|
||||
/// listing being left. Advancing a wire generation lets the daemon cancel work
|
||||
/// even when an older request reaches it after this one.
|
||||
pub(crate) fn cancel_file_thumbnails(ui: &Rc<Ui>) {
|
||||
if let Some(source) = ui.file_thumbs.source.borrow_mut().take() {
|
||||
source.remove();
|
||||
}
|
||||
ui.file_thumbs.queue.borrow_mut().clear();
|
||||
ui.file_thumbs.wanted.borrow_mut().clear();
|
||||
let generation = ui.file_thumbs.generation.replace(0);
|
||||
// The request is useful even when its reply is not. `spawn_request` owns the
|
||||
// socket round-trip on a worker thread, so dropping the receiver is safe.
|
||||
if generation != 0 {
|
||||
drop(spawn_request(
|
||||
ui.dirs.control_socket(),
|
||||
Request::CancelFileThumbs {
|
||||
generation: generation.saturating_add(1),
|
||||
},
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
/// A square thumbnail surface with a generic file icon underneath the picture.
|
||||
/// Call [`bind_file_thumbnail`] whenever a recycled row is bound to an entry.
|
||||
pub(crate) fn file_thumbnail_widget(size: i32, fallback_size: i32) -> gtk4::Overlay {
|
||||
let fallback = gtk4::Image::builder().pixel_size(fallback_size).build();
|
||||
let picture = gtk4::Picture::builder()
|
||||
.width_request(size)
|
||||
.height_request(size)
|
||||
.content_fit(gtk4::ContentFit::Cover)
|
||||
.can_shrink(true)
|
||||
.visible(false)
|
||||
.build();
|
||||
picture.add_css_class("file-thumbnail");
|
||||
|
||||
let overlay = gtk4::Overlay::new();
|
||||
overlay.set_width_request(size);
|
||||
overlay.set_height_request(size);
|
||||
overlay.set_halign(gtk4::Align::Center);
|
||||
overlay.set_valign(gtk4::Align::Center);
|
||||
overlay.set_overflow(gtk4::Overflow::Hidden);
|
||||
overlay.set_child(Some(&fallback));
|
||||
overlay.add_overlay(&picture);
|
||||
overlay
|
||||
}
|
||||
|
||||
/// Resize an existing thumbnail surface. Grid rows are recycled, so changing
|
||||
/// these requests in place and rebinding the model is enough to implement the
|
||||
/// status-bar zoom without downloading or decoding an image again.
|
||||
pub(crate) fn resize_file_thumbnail(widget: >k4::Overlay, size: i32, fallback_size: i32) {
|
||||
widget.set_width_request(size);
|
||||
widget.set_height_request(size);
|
||||
let Some(fallback) = widget.child().and_downcast::<gtk4::Image>() else {
|
||||
return;
|
||||
};
|
||||
fallback.set_pixel_size(fallback_size);
|
||||
if let Some(picture) = fallback.next_sibling().and_downcast::<gtk4::Picture>() {
|
||||
picture.set_width_request(size);
|
||||
picture.set_height_request(size);
|
||||
}
|
||||
}
|
||||
|
||||
/// Bind a thumbnail surface to one entry, painting a cached texture immediately
|
||||
/// or queueing an on-demand daemon request. Non-images retain the generic icon.
|
||||
pub(crate) fn bind_file_thumbnail(
|
||||
ui: &Rc<Ui>,
|
||||
widget: >k4::Overlay,
|
||||
entry: &DirEntry,
|
||||
symbolic_fallback: bool,
|
||||
) {
|
||||
let Some(fallback) = widget.child().and_downcast::<gtk4::Image>() else {
|
||||
return;
|
||||
};
|
||||
// The browser may add a status badge after the picture. Locate the picture
|
||||
// immediately after the base fallback instead of assuming it is the last
|
||||
// overlay child.
|
||||
let Some(picture) = fallback.next_sibling().and_downcast::<gtk4::Picture>() else {
|
||||
return;
|
||||
};
|
||||
|
||||
let fallback_name = if symbolic_fallback {
|
||||
format!("{}-symbolic", icon_base_for(entry))
|
||||
} else {
|
||||
icon_base_for(entry).to_string()
|
||||
};
|
||||
fallback.set_icon_name(Some(&fallback_name));
|
||||
fallback.set_visible(true);
|
||||
picture.set_visible(false);
|
||||
picture.set_paintable(gtk4::gdk::Paintable::NONE);
|
||||
|
||||
// The widget name is a cheap binding token. List factories recycle their
|
||||
// children; a reply for the previous entry must not paint into the new one.
|
||||
let key = thumbnail_key(&entry.uid, entry.modified);
|
||||
picture.set_widget_name(&key);
|
||||
remove_waiter(ui, &picture);
|
||||
|
||||
if entry.is_dir || !is_image_name(&entry.name) || entry.uid.is_empty() {
|
||||
return;
|
||||
}
|
||||
if let Some(texture) = ui.file_thumbs.textures.borrow().get(&key) {
|
||||
paint(&picture, &key, texture);
|
||||
return;
|
||||
}
|
||||
if ui.file_thumbs.missing.borrow().contains(&key) {
|
||||
return;
|
||||
}
|
||||
|
||||
ui.file_thumbs
|
||||
.wanted
|
||||
.borrow_mut()
|
||||
.entry(key)
|
||||
.or_default()
|
||||
.push(picture);
|
||||
let mut queue = ui.file_thumbs.queue.borrow_mut();
|
||||
queue.retain(|item| item.uid != entry.uid);
|
||||
queue.push_back(FileThumbRequest {
|
||||
uid: entry.uid.clone(),
|
||||
modified: entry.modified,
|
||||
name: entry.name.clone(),
|
||||
});
|
||||
drop(queue);
|
||||
schedule_file_thumbs(ui);
|
||||
}
|
||||
|
||||
/// Create and immediately bind a thumbnail for a non-recycled row.
|
||||
pub(crate) fn file_thumbnail(
|
||||
ui: &Rc<Ui>,
|
||||
entry: &DirEntry,
|
||||
size: i32,
|
||||
fallback_size: i32,
|
||||
symbolic_fallback: bool,
|
||||
) -> gtk4::Overlay {
|
||||
let widget = file_thumbnail_widget(size, fallback_size);
|
||||
bind_file_thumbnail(ui, &widget, entry, symbolic_fallback);
|
||||
widget
|
||||
}
|
||||
|
||||
/// Whether the file name denotes an image format for which Proton or the local
|
||||
/// decoder can reasonably provide a thumbnail.
|
||||
pub(crate) fn is_image_name(name: &str) -> bool {
|
||||
is_thumbnail_image_name(name)
|
||||
}
|
||||
|
||||
fn thumbnail_key(uid: &str, modified: i64) -> String {
|
||||
format!("{uid}:{modified}")
|
||||
}
|
||||
|
||||
fn remove_waiter(ui: &Rc<Ui>, picture: >k4::Picture) {
|
||||
let mut wanted = ui.file_thumbs.wanted.borrow_mut();
|
||||
wanted.retain(|_, pictures| {
|
||||
pictures.retain(|candidate| candidate != picture);
|
||||
!pictures.is_empty()
|
||||
});
|
||||
}
|
||||
|
||||
fn paint(picture: >k4::Picture, key: &str, texture: >k4::gdk::Texture) {
|
||||
if picture.widget_name() != key {
|
||||
return;
|
||||
}
|
||||
picture.set_paintable(Some(texture));
|
||||
picture.set_visible(true);
|
||||
if let Some(overlay) = picture.parent().and_downcast::<gtk4::Overlay>()
|
||||
&& let Some(fallback) = overlay.child().and_downcast::<gtk4::Image>()
|
||||
{
|
||||
fallback.set_visible(false);
|
||||
}
|
||||
}
|
||||
|
||||
fn schedule_file_thumbs(ui: &Rc<Ui>) {
|
||||
if ui.file_thumbs.queue.borrow().is_empty() || ui.file_thumbs.inflight.get() {
|
||||
return;
|
||||
}
|
||||
if ui.file_thumbs.generation.get() == 0 {
|
||||
reserve_file_thumb_generation(ui);
|
||||
return;
|
||||
}
|
||||
if let Some(source) = ui.file_thumbs.source.borrow_mut().take() {
|
||||
source.remove();
|
||||
}
|
||||
let ui_flush = ui.clone();
|
||||
let source = glib::timeout_add_local_once(THUMB_DEBOUNCE, move || {
|
||||
ui_flush.file_thumbs.source.borrow_mut().take();
|
||||
flush_file_thumbs(&ui_flush);
|
||||
});
|
||||
*ui.file_thumbs.source.borrow_mut() = Some(source);
|
||||
}
|
||||
|
||||
fn reserve_file_thumb_generation(ui: &Rc<Ui>) {
|
||||
if ui.file_thumbs.generation_loading.replace(true) {
|
||||
return;
|
||||
}
|
||||
let rx = spawn_request(
|
||||
ui.dirs.control_socket(),
|
||||
Request::ReserveFileThumbGeneration,
|
||||
);
|
||||
let ui = ui.clone();
|
||||
glib::spawn_future_local(async move {
|
||||
let result = rx.recv().await;
|
||||
ui.file_thumbs.generation_loading.set(false);
|
||||
match result {
|
||||
Ok(Ok(Response::FileThumbGeneration { generation })) if generation != 0 => {
|
||||
ui.file_thumbs.generation.set(generation);
|
||||
schedule_file_thumbs(&ui);
|
||||
}
|
||||
_ => retry_file_thumb_generation(&ui),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
fn retry_file_thumb_generation(ui: &Rc<Ui>) {
|
||||
if let Some(source) = ui.file_thumbs.source.borrow_mut().take() {
|
||||
source.remove();
|
||||
}
|
||||
let ui_retry = ui.clone();
|
||||
let source = glib::timeout_add_local_once(THUMB_RETRY, move || {
|
||||
ui_retry.file_thumbs.source.borrow_mut().take();
|
||||
schedule_file_thumbs(&ui_retry);
|
||||
});
|
||||
*ui.file_thumbs.source.borrow_mut() = Some(source);
|
||||
}
|
||||
|
||||
fn flush_file_thumbs(ui: &Rc<Ui>) {
|
||||
if ui.file_thumbs.inflight.get() {
|
||||
return;
|
||||
}
|
||||
let items: Vec<FileThumbRequest> = {
|
||||
let mut queue = ui.file_thumbs.queue.borrow_mut();
|
||||
let wanted = ui.file_thumbs.wanted.borrow();
|
||||
let mut batch = Vec::new();
|
||||
while batch.len() < THUMB_BATCH {
|
||||
let Some(item) = queue.pop_front() else {
|
||||
break;
|
||||
};
|
||||
if wanted.contains_key(&thumbnail_key(&item.uid, item.modified)) {
|
||||
batch.push(item);
|
||||
}
|
||||
}
|
||||
batch
|
||||
};
|
||||
if items.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
ui.file_thumbs.inflight.set(true);
|
||||
let rx = spawn_request(
|
||||
ui.dirs.control_socket(),
|
||||
Request::FileThumbs {
|
||||
items: items.clone(),
|
||||
generation: ui.file_thumbs.generation.get(),
|
||||
},
|
||||
);
|
||||
let ui = ui.clone();
|
||||
glib::spawn_future_local(async move {
|
||||
let result = rx.recv().await;
|
||||
ui.file_thumbs.inflight.set(false);
|
||||
match result {
|
||||
Ok(Ok(Response::Thumbs { items: replies })) => {
|
||||
let mut requested: HashMap<String, FileThumbRequest> = items
|
||||
.iter()
|
||||
.cloned()
|
||||
.map(|item| (item.uid.clone(), item))
|
||||
.collect();
|
||||
let mut pending = Vec::new();
|
||||
for reply in replies {
|
||||
let Some(request) = requested.remove(&reply.uid) else {
|
||||
continue;
|
||||
};
|
||||
let key = thumbnail_key(&request.uid, request.modified);
|
||||
let wanted = ui.file_thumbs.wanted.borrow().contains_key(&key);
|
||||
match reply_disposition(wanted, reply.path.is_some(), reply.pending) {
|
||||
ReplyDisposition::Ignore => continue,
|
||||
ReplyDisposition::Load => {
|
||||
let Some(path) = reply.path else { continue };
|
||||
match gtk4::gdk::Texture::from_filename(&path) {
|
||||
Ok(texture) => {
|
||||
ui.file_thumbs.store_texture(&key, texture.clone());
|
||||
if let Some(pictures) =
|
||||
ui.file_thumbs.wanted.borrow_mut().remove(&key)
|
||||
{
|
||||
for picture in pictures {
|
||||
paint(&picture, &key, &texture);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::debug!("cannot decode file thumbnail {path}: {e}");
|
||||
ui.file_thumbs.store_missing(key.clone());
|
||||
ui.file_thumbs.wanted.borrow_mut().remove(&key);
|
||||
}
|
||||
}
|
||||
}
|
||||
ReplyDisposition::Retry => pending.push(request),
|
||||
ReplyDisposition::Missing => {
|
||||
ui.file_thumbs.store_missing(key.clone());
|
||||
ui.file_thumbs.wanted.borrow_mut().remove(&key);
|
||||
}
|
||||
}
|
||||
}
|
||||
pending.extend(requested.into_values());
|
||||
if !pending.is_empty() {
|
||||
retry_file_thumbs(&ui, pending);
|
||||
}
|
||||
}
|
||||
Ok(Ok(Response::FileThumbsStale)) => {
|
||||
ui.file_thumbs.generation.set(0);
|
||||
retry_file_thumbs(&ui, items);
|
||||
}
|
||||
Ok(Ok(Response::Error { message, .. })) => {
|
||||
tracing::debug!("file thumbnails failed: {message}");
|
||||
retry_file_thumbs(&ui, items);
|
||||
}
|
||||
Ok(Ok(_)) | Ok(Err(_)) | Err(_) => {
|
||||
tracing::debug!("file thumbnails: no reply");
|
||||
retry_file_thumbs(&ui, items);
|
||||
}
|
||||
}
|
||||
schedule_file_thumbs(&ui);
|
||||
});
|
||||
}
|
||||
|
||||
fn retry_file_thumbs(ui: &Rc<Ui>, items: Vec<FileThumbRequest>) {
|
||||
let ui = ui.clone();
|
||||
glib::timeout_add_local_once(THUMB_RETRY, move || {
|
||||
let wanted = ui.file_thumbs.wanted.borrow();
|
||||
let retry: Vec<FileThumbRequest> = items
|
||||
.into_iter()
|
||||
.filter(|item| wanted.contains_key(&thumbnail_key(&item.uid, item.modified)))
|
||||
.collect();
|
||||
drop(wanted);
|
||||
ui.file_thumbs.queue.borrow_mut().extend(retry);
|
||||
schedule_file_thumbs(&ui);
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{ReplyDisposition, is_image_name, reply_disposition};
|
||||
|
||||
#[test]
|
||||
fn image_extensions_are_case_insensitive_and_specific() {
|
||||
for name in [
|
||||
"photo.jpg",
|
||||
"SCAN.PNG",
|
||||
"animation.Gif",
|
||||
"archive.tiff",
|
||||
"camera.nef",
|
||||
"negative.CR3",
|
||||
"camera.RAW",
|
||||
] {
|
||||
assert!(is_image_name(name), "{name}");
|
||||
}
|
||||
for name in [
|
||||
"notes.txt",
|
||||
"movie.mp4",
|
||||
"image.jpg.zip",
|
||||
"jpg",
|
||||
"camera.heic",
|
||||
"frame.avif",
|
||||
"graphic.svg",
|
||||
"camera.rw",
|
||||
] {
|
||||
assert!(!is_image_name(name), "{name}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cancelled_listing_replies_cannot_become_permanent_misses() {
|
||||
assert_eq!(
|
||||
reply_disposition(false, false, false),
|
||||
ReplyDisposition::Ignore
|
||||
);
|
||||
assert_eq!(
|
||||
reply_disposition(true, false, false),
|
||||
ReplyDisposition::Missing
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -153,8 +153,9 @@ Because Proton Drive does not support partial byte writes, modified files must b
|
||||
2. **Close/Release (`queue_revision`):** When the application closes the file descriptor, the daemon:
|
||||
- Fetches any untouched gaps from the remote base file to compile the full file.
|
||||
- Durably publishes the scratch data and authored-range sidecar into `staging` under a `{uid}-{millis}-{counter}` name. Temporary data and metadata are synced before atomic rename, and their directory is synced before the source is removed.
|
||||
- Transactionally queues or supersedes a pending database operation (`PendingOp`), so insertion failure cannot erase the previously acknowledged upload. For `OP_REVISION` ops, execution is debounced by 2 seconds (`DRAIN_REVISION_DEBOUNCE = 2s`) to give rapid follow-up writes (e.g. `aria2c` preallocation followed by writing) time to supersede the staged blob before network transmission.
|
||||
3. **Async Drain Thread (`run_pending_drain`):** The background drain worker picks up the database operations queue, handles revision uploads, resolves conflicts, and cleans up staging files. Upon landing a revision upload (`refresh_after_upload`), it rebaselines both still-queued ops (`rebaseline_pending`) and open write handles targeting the same node to prevent false self-conflict copies on subsequent writes.
|
||||
- Transactionally queues or supersedes a pending database operation (`PendingOp`), so insertion failure cannot erase the previously acknowledged upload. For `OP_REVISION` ops, execution is debounced to give rapid follow-up writes (e.g. `aria2c` preallocation followed by writing) time to supersede the staged blob before network transmission. The debounce is adaptive (`Core::revision_debounce`): it starts at `DRAIN_REVISION_DEBOUNCE = 2s` and, once a node has been uploaded, widens toward how long that upload actually took, bounded by `DRAIN_REVISION_DEBOUNCE_MAX = 60s`. A file saved faster than it can be sent therefore supersedes in the queue rather than mid-upload.
|
||||
- Signals `Core::cancel_upload` for the node before touching the queue. A drain worker may already be reading the blob this write supersedes; the flag is read by the upload's `CountingReader`, which refuses the SDK's next block. `queue_trash` and `discard_queued_ops` do the same, because they unlink the blob outright.
|
||||
3. **Async Drain Threads (`run_pending_drain`):** `DRAIN_WORKERS = 3` background workers share the operations queue. They pick work up through `Db::claim_next_due_op`, which marks the row `claimed_at` in the same transaction that selects it and excludes any op whose uid another worker already holds — ordering only has to hold *per node*, and that exclusion is what guarantees it. Worker 0 additionally runs the queue's idle chores (LRU touch flush, `recover_fsynced_writes`). A claim is process-local state: the single-writer `flock` means one found at open belongs to a crashed run, and `Db::clear_op_claims` drops the lot so those ops are not invisible forever. Each worker handles revision uploads, resolves conflicts, and cleans up staging files. Upon landing a revision upload (`refresh_after_upload`), it rebaselines both still-queued ops (`rebaseline_pending`) and open write handles targeting the same node to prevent false self-conflict copies on subsequent writes.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
|
||||
+878
-19
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,517 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to the Linux client since the 1.0.0 release. Format follows
|
||||
[Keep a Changelog](https://keepachangelog.com/1.1.0/); versions follow semver, and every
|
||||
release is tagged `vX.Y.Z` with `Cargo.toml` and `packaging/PKGBUILD` in agreement.
|
||||
|
||||
`Schema:` lines record the SQLite `SCHEMA_VERSION` (`pdfs-core/src/db/migrations.rs`) a
|
||||
release ships. Migrations are forward-only — a database written by a newer client is a hard
|
||||
refuse-to-open, not a downgrade, so rolling back a release means restoring the cache from
|
||||
scratch (user data in `staging/` and `recovery/` is never touched by this).
|
||||
|
||||
## [1.9.0] — 2026-08-18
|
||||
|
||||
Google Photos import, and a pass over the desktop app's interaction gaps. No schema change.
|
||||
Schema: **28**. Builds against the published `proton-sdk` / `proton-drive-rs` **0.6.1** — the album
|
||||
writes this release needs are in the registry crates, so the local `[patch.crates-io]` shim the
|
||||
feature was developed against is gone and no local path dependency remains.
|
||||
|
||||
### Added
|
||||
- **Import a Google Photos Takeout export into Proton Photos** — `pdfs import-google-photos
|
||||
takeout-*.zip` (add `--dry-run` to see what it would do, `--wait` to follow it), or the import
|
||||
button in the Photos header, which multi-selects the export's zips. Albums in the export are
|
||||
recreated as Proton albums and their photos filed into them; year folders and `Archive/` import
|
||||
into the timeline; `Trash/` is skipped. Capture times and favourites come from Google's per-photo
|
||||
JSON sidecars, including the truncated and `(N)`-swapped sidecar names Google produces.
|
||||
- Photos already on the account are **not re-uploaded**: the import asks the photos volume which of
|
||||
the export's names exist (batched, 150 per request), reads and SHA-1s only those files, and skips
|
||||
the ones whose content digest matches too — Proton's own duplicate rule. A library already partly
|
||||
synced by Proton Photos on a phone therefore imports only what is missing, and an interrupted
|
||||
import resumes by being run again. Google keeps a full copy of a photo in every album folder it
|
||||
appears in; those fold into one upload that joins several albums.
|
||||
- The import streams each photo straight out of its zip, so an export needs no scratch space to
|
||||
unpack into, and reports as a job in `pdfs transfers` and the GUI's activity view. `pdfs
|
||||
import-status` shows the counts, `pdfs cancel-import` stops it — a cancelled run still files what
|
||||
it uploaded into its albums.
|
||||
- **An Import page in the GUI** (Settings → *Import from Google Photos*, or the import button in
|
||||
the Photos header). Archives are staged by drag-and-drop onto the page or through the file
|
||||
chooser, listed with their sizes so the set can be checked before anything is sent, and launched
|
||||
as either a Scan (dry run) or a confirmed Import. The page shows live progress off the import's
|
||||
job, keeps the finished run's report on screen, and can stop a run in flight; because the import
|
||||
outlives the page, it is polled wherever the user navigates and its completion is reported as a
|
||||
desktop notification.
|
||||
|
||||
- SDK (`proton-sdk-rs` **0.6.1**): album **writes** — `ProtonPhotosClient::create_album` and
|
||||
`add_photos_to_album`, ported from the TypeScript SDK (upstream C# has no album write API), plus
|
||||
batch duplicate detection (`find_duplicates_many`, `name_collisions`). The workspace deps moved
|
||||
`"0.6.0"` → `"0.6.1"` and `Cargo.lock` re-resolved to the registry crates. Adding a photo from an
|
||||
album *shared with us* (a different volume) is still unported and fails that photo's outcome.
|
||||
- **Multi-select in Files.** Both views select a batch (click-drag, Ctrl-click, Shift-click); a bar
|
||||
slides in reporting the count and offering Move to Trash and offline on/off for the whole
|
||||
selection. Delete acts on everything selected, Esc clears it, and right-clicking inside a
|
||||
selection offers the batch actions rather than the one row under the pointer. Actions that need a
|
||||
single subject — rename, move, share, revisions — stay single-entry, and the details pane steps
|
||||
aside for a batch instead of speaking for it.
|
||||
- **Undo on trash.** "Moved *X* to Trash" now carries an Undo that restores the batch, single files
|
||||
included.
|
||||
- **The empty states are a place to act**: an empty folder offers Upload files / New folder, and an
|
||||
empty photo timeline offers Upload photos / Import from Google Photos, instead of describing what
|
||||
the user could go and do elsewhere.
|
||||
- **Ctrl+F works from any page**, navigating to Files and focusing the search box, rather than being
|
||||
silently inert everywhere but Files.
|
||||
- Staged Takeout archives survive closing the window; archives that have since been moved or deleted
|
||||
are dropped from the restored set.
|
||||
|
||||
### Fixed
|
||||
- Settings showed the local cache as "X of 0 B used" when no cache budget was set. An unset budget
|
||||
means *unlimited*, so it now reads "X cached — no limit set" rather than implying a zero cap.
|
||||
- The cache-budget editor fired a `SetCacheBudget` round-trip — and a toast — on every step, so a
|
||||
run of `+` clicks applied a series of caps the user never asked for and stacked a column of
|
||||
toasts. It now waits for the value to settle.
|
||||
- A long pin list pushed everything below it off the end of Settings. It shows the first six with a
|
||||
"Show all N" row for the rest.
|
||||
|
||||
## [1.8.2] — 2026-08-17
|
||||
|
||||
Closes the two findings 1.8.1's live verification produced (B85, B86) and finishes the data-safety
|
||||
audit entries that had been sitting as "fixed in code, verification pending" since 1.0 (B49–B55).
|
||||
No schema change. Schema: **28**.
|
||||
|
||||
The read path no longer assumes a revision is a series of 4 MiB blocks; it takes the geometry from
|
||||
the revision. The B85 investigation also **contradicted its own bug report** — see Notes, because
|
||||
the conclusion recorded there was that reads of such files were losing data, and they were not.
|
||||
|
||||
### Fixed
|
||||
- A folder switched from mirror to on-demand now shows the files the mirror pass itself uploaded.
|
||||
The sync engine created nodes on the remote without invalidating the parent's cached child
|
||||
listing, and that listing is a database flag rather than an in-memory cache — so it survived a
|
||||
daemon restart, and the files stayed invisible until the folder was switched back. Every remote
|
||||
mutation the engine makes (folder create, file upload, and the three trash paths) now drops the
|
||||
parent's cached listing by uid, which reaches a folder that no mount has resident — the case a
|
||||
mirror folder always is (`docs/BUGS.md` B86).
|
||||
- `pdfs ls` and `pdfs refresh` accept a path under any mount the daemon owns, not only the primary
|
||||
one. Both previously answered "is not under the mountpoint" for anything inside a secondary
|
||||
on-demand mount, which left a folder serving a stale listing with no manual way to re-enumerate
|
||||
it. A path under nothing this daemon has mounted still gets the same error as before.
|
||||
- A file whose blocks are not 4 MiB is no longer read twice over. Block offsets came from
|
||||
`index * 4 MiB`, so on a revision laid out differently every block the client asked for
|
||||
straddled two of the revision's own, and each was fetched and decrypted to answer for one — then
|
||||
cached at boundaries the next read would not ask about. Offsets now come from the revision's
|
||||
recorded block sizes, and the on-disk cache, the in-memory ring and the in-flight fetch key
|
||||
address a block by the byte range it covers rather than by its index (`docs/BUGS.md` B85).
|
||||
|
||||
### Changed
|
||||
- The block cache records each block's plaintext start alongside its `(mtime, size)` tag, and each
|
||||
revision's block table in a small sidecar beside its blocks. Blocks cached by an earlier release
|
||||
carry no start; they are honoured wherever the old 4 MiB assumption and the new plan agree on
|
||||
the offset, which is every file this client uploaded — so the existing cache survives the
|
||||
upgrade intact and is quietly refetched only for the files it was placing wrongly.
|
||||
|
||||
### Notes
|
||||
- **B85's diagnosis was wrong, and the entry has been corrected rather than closed.** It read a
|
||||
set of journal lines as proof that this client was requesting wrong byte ranges and being saved
|
||||
only by B84's repair path. `RevisionReader::read_at` does not work that way: it plans over the
|
||||
revision's real block sizes and clamps every block to the range asked for, identically in the
|
||||
0.6.0 the client builds against and in every earlier version. A read can therefore never return
|
||||
more bytes than it requested — yet every quoted line shows exactly that (`blen=4194304
|
||||
got=4939506`), and the one place in the client that logs that shape can only reach it on the
|
||||
opposite condition. The numbers were either transcribed with two fields swapped or captured from
|
||||
a build not in the history; they want re-capturing from the journal before anything is concluded
|
||||
from them. What was real is the assumption itself, and that is what this release removes — a
|
||||
cost fix, not a correctness one. Nothing here changes what a read returns.
|
||||
- The first read of a file still plans on the 4 MiB assumption, and learns the real geometry as a
|
||||
side effect of opening the reader it needs anyway. Resolving a reader merely to *plan* would
|
||||
reintroduce the per-read key derivation the block cache exists to avoid, so the first read of a
|
||||
non-4-MiB file pays one straddling fetch and then self-corrects.
|
||||
- Seven data-safety audit entries (B49–B55) moved from "fixed in code, verification pending" to
|
||||
verified. The fixes were all genuinely present and needed no change; what they lacked was the
|
||||
injected-failure testing each entry asked for, and that is what was written — unwritable
|
||||
directories and `chmod 000` subtrees for permission failures, a real `/dev/shm`↔`/tmp` device
|
||||
boundary for the cross-filesystem fallbacks, SQLite's `max_page_count` for a full disk, an abort
|
||||
trigger for a failed insert, and truncated and unpublished sidecars for the crash boundaries.
|
||||
Two seams were added to make failure arrangeable at all: `stage_write_at`, mirroring the
|
||||
`preserve_write_at` that already existed, and `walk_local_tree` lifted out of the sync engine so
|
||||
the local scan can be driven against a real tree without a daemon.
|
||||
- **Two of those entries are not fully closed, and say so.** B51 and B52 concern crash consistency,
|
||||
and nothing short of a machine actually losing power tests that; the crash *boundaries* are
|
||||
covered by reconstructing on disk exactly what a crash at each point leaves. B54 also diverges
|
||||
from its required fix on purpose: it has no "deliberate delete" confirmation path, so a total
|
||||
local wipe is refused outright rather than offered for confirmation. That is stricter than asked
|
||||
for, and the cost is that a user who really did delete everything must remove and re-add the
|
||||
sync folder.
|
||||
- B86's fix routes only `ls` and `refresh` to the owning mount. The other path-addressed control
|
||||
requests — pin, rename, move, delete, sharing — still resolve against the primary mount only.
|
||||
That is a larger change and was not what blocked the repro; it is recorded as an open follow-up
|
||||
(`docs/BUGS.md` B88).
|
||||
- Three findings were opened rather than fixed: the geometry above is learned one read late, so a
|
||||
non-4-MiB file pays a single wasted fetch on its first read (B87); the remaining path-addressed
|
||||
control requests are still primary-only (B88); and the container-level `CLAUDE.md` still
|
||||
describes the SDK dependency as 0.1.11 when 1.7.1 bumped it to 0.6.0, which is what sent the
|
||||
B85 investigation to the wrong source tree to begin with (B89).
|
||||
|
||||
## [1.8.1] — 2026-08-16
|
||||
|
||||
Some files read short through the mount and reported success — an application got a truncated
|
||||
copy with nothing anywhere saying so (`docs/BUGS.md` B84). Found because GNOME's file indexer
|
||||
kept complaining that PNGs under `~/ProtonDrive/Pictures` were corrupt; it was right. The rest
|
||||
of this release is the long-standing architectural backlog from the July concurrency audits
|
||||
(B25, B32, B39, B40, B43, B44), each verified against the current code before being fixed.
|
||||
Schema: **28** — migration V28 adds `sync_entry.local_mtime_ns`.
|
||||
|
||||
### Fixed
|
||||
- A read no longer trusts a revision whose recorded block sizes disagree with the file's size.
|
||||
Such a file leaves the block/range path entirely: it is fetched through the whole-file,
|
||||
manifest-verified download — the same path `pdfs pin` uses, and the reason pinning was the
|
||||
workaround — and served from the resulting cached blob. One download per file however many
|
||||
reads discover the problem at once, and files above 512 MiB fail `EIO` with a note to pin
|
||||
them rather than pulling down gigabytes for one read.
|
||||
- A block that comes back shorter than the file's size implies is never served, cached or kept
|
||||
in memory. Assembling a read out of short blocks is what quietly produced the truncation:
|
||||
the missing bytes were simply not appended, the kernel handed userspace a short read, and
|
||||
every tool reads a short read as end-of-file.
|
||||
- A synced folder no longer reports the same deletion as an error on every pass. Deleting a
|
||||
local file that something else had already deleted counted as a failure, which kept its
|
||||
baseline and made the next pass try again — forever. A path that is already gone is the
|
||||
state the sync pass wanted (`docs/BUGS.md` B42).
|
||||
- A mirror folder no longer misses an edit made in the same second as its last sync. Change
|
||||
detection compared whole seconds, so rewriting a file in place without changing its length
|
||||
read as "unchanged" and never uploaded — the remote silently stayed behind until something
|
||||
else about the file moved. Baselines now record the modification time to nanoseconds
|
||||
(`docs/BUGS.md` B25).
|
||||
- A sync download no longer overwrites an edit that finished while it was running. The
|
||||
destination is re-checked in the last moment before the download is published, and a local
|
||||
file that has moved on since the pass planned for it is kept as a conflict copy — the same
|
||||
resolution a both-sides-changed conflict gets (`docs/BUGS.md` B39).
|
||||
- An upload interrupted by a writer no longer reports itself as synchronized. The baseline now
|
||||
records the content that actually went up rather than whatever is on disk when the transfer
|
||||
ends, so a revision torn by a concurrent write is replaced by a clean one on the next pass
|
||||
instead of standing as the file's content on Drive (`docs/BUGS.md` B40).
|
||||
- Listing a directory that changes while it is being read no longer skips or repeats entries.
|
||||
`readdir` resumed by index into a listing rebuilt on every call, so a create or a trash
|
||||
between two pages shifted everything after it. The listing is now frozen for the life of the
|
||||
directory handle (`docs/BUGS.md` B43).
|
||||
- Opening a file for writing while a previous close is still publishing no longer discards that
|
||||
close's revision. The base is re-checked at the moment the write handle is installed, rather
|
||||
than trusted across a copy that can be the size of the whole file (`docs/BUGS.md` B32).
|
||||
- Unmounting now stops and joins the daemon's background workers instead of leaving them
|
||||
running. The drain, sync engine, conflict sweep, online probe, local indexer and control
|
||||
listener share a stop signal and interruptible waits, so an in-process remount no longer
|
||||
starts a second set on top of the first (`docs/BUGS.md` B44).
|
||||
|
||||
### Notes
|
||||
- The size `stat` reports was always the correct one; it is the reader's belief about where
|
||||
blocks end that was wrong, so nothing about reported sizes changes.
|
||||
- Verified live on the affected account: four multi-block PNGs that had been reading short now
|
||||
deliver their full size, and the journal shows each one repaired. The diagnostics also
|
||||
answered the open question, and not as expected — the revision metadata is fine; those files
|
||||
simply do not have 4 MiB blocks, and this client assumes they do. Filed as `docs/BUGS.md`
|
||||
B85. Until that is fixed, a file with non-4-MiB blocks is downloaded whole on its first read.
|
||||
- Migration V28's new column is nullable rather than converted from the existing seconds. A
|
||||
baseline row written before this release has no sub-second time to recover, and inventing
|
||||
zero for it would make every already-synced file compare as changed and re-upload an entire
|
||||
mirror on first start; `NULL` keeps those rows comparing exactly as they did, and each gains
|
||||
a real value the next time its path syncs.
|
||||
- Three audit entries closed without a code change, having been fixed already or being correct
|
||||
as they stand: sync-folder removal is serialized against reconciliation (B41), the drain's
|
||||
database contention was removed by the read-only connection pool and the claim column (B30),
|
||||
and the refusals for symlinks, hard links, device nodes and `chmod` are deliberate — Drive
|
||||
cannot represent them, and a local-only representation no other Proton client can read would
|
||||
be worse than a clean refusal (B31).
|
||||
- Six of this release's fixes were then driven live against the installed 1.8.1 on a real
|
||||
account: the same-second edit uploads (B25), three rapid rewrites of a 64 MiB file publish the
|
||||
third one byte for byte (B32), a write racing a download survives as a conflict copy (B39), an
|
||||
upload torn by a concurrent write heals itself on the following pass (B40), a `getdents64`
|
||||
walk with a 160-byte buffer against a directory mutated between pages returns every entry
|
||||
exactly once (B43), and a restart of a daemon with six mounts and a busy drain stops in 110 ms
|
||||
and comes back at the same thread count (B44). B42 could not be driven from a single machine —
|
||||
see B86 below.
|
||||
- One new bug came out of the verification: an on-demand mount serves a folder listing that
|
||||
predates the sync engine's own uploads, so files the mirror pass created are invisible until
|
||||
the folder is switched back (`docs/BUGS.md` B86). Nothing is lost — they are on the server and
|
||||
come back on the next mirror pass — but they cannot be seen or named while the folder is
|
||||
on-demand, which is also what blocks B42's live verification.
|
||||
|
||||
## [1.8.0] — 2026-08-16
|
||||
|
||||
A queued write whose node had left the local tree was retried every five seconds forever and
|
||||
counted as an ordinary pending upload — 28 of them, holding 18.4 GiB, had been doing that for
|
||||
30 days on the machine that found it (`docs/BUGS.md` B83). Schema: 27 — migration V27 adds
|
||||
`pending_op.access_deferred_since`.
|
||||
|
||||
### Fixed
|
||||
- The drain no longer reads "I have never heard of this node" as "you may not write this
|
||||
node". A missing authority is now resolved against the remote: the node is re-interned if
|
||||
it is still there, and reported as a real failure if it is gone. Ops that had been stuck
|
||||
invisibly for weeks now say what is wrong with them on the first pass.
|
||||
- A create or mkdir resolves its *parent* as the authority, so the refetch asks about the
|
||||
folder rather than the node it has not made yet.
|
||||
|
||||
### Added
|
||||
- Access deferrals are bounded. A deferral still costs no attempt and records no error for
|
||||
the first five minutes — the case it exists for is a share downgraded mid-queue and undone
|
||||
a moment later — but past that window each recheck records a failure, so the op enters the
|
||||
ordinary backoff and appears in `failing_ops` and the `pdfs status` error text like any
|
||||
other stuck operation. The first deferral of a run is logged at `warn` rather than `debug`.
|
||||
|
||||
### Notes
|
||||
- No staged data is touched by any of this. A reported failure keeps its `pending_op` row and
|
||||
its staged blob exactly as before; the change is in what the user is told, and in the queue
|
||||
no longer waiting on a permission change that was never coming.
|
||||
|
||||
## [1.7.1] — 2026-08-16
|
||||
|
||||
Write-back throughput follow-up to 1.7.0, same audit
|
||||
(`docs/plans/audit-2026-08-15-perf-locks.md`). Schema: 26 — migration V26 adds
|
||||
`pending_op.claimed_at`.
|
||||
|
||||
### Added
|
||||
- Parallel write-back drain: three workers share the queue through a `pending_op.claimed_at`
|
||||
claim column (migration V26), so a large upload no longer holds every queued rename, trash
|
||||
and small write behind it. Ordering is preserved per node — the claim query never offers
|
||||
two workers ops for the same uid — and claims left by a crashed run are cleared at open.
|
||||
- An upload already on the wire is cancelled when a newer write, a trash or a discard
|
||||
supersedes it, instead of spending the uplink on a revision the next operation replaces.
|
||||
|
||||
### Changed
|
||||
- The revision debounce is adaptive: it widens toward how long that node's last upload
|
||||
actually took (bounded to 2–60 s), so a file saved faster than it can be sent supersedes
|
||||
in the queue rather than mid-upload.
|
||||
- Workspace deps bumped to `proton-sdk` / `proton-drive-rs` 0.6.0.
|
||||
|
||||
## [1.7.0] — 2026-08-16
|
||||
|
||||
Performance and robustness pass over the mount, from the architecture audit in
|
||||
`docs/plans/audit-2026-08-15-perf-locks.md`. Schema: 25.
|
||||
|
||||
### Added
|
||||
- Sequential-read prefetch (`Core::prefetch_ahead`): per-file sequential detection, depth
|
||||
doubling 2→8, reset to 0 after a seek, skipping blocks already cached, and bounded by its
|
||||
own permit budget so it can never queue ahead of a demand read.
|
||||
- Single-flight block fetching (`BlockFlight`) keyed on `(uid, mtime, size, block)`, so a
|
||||
demand read, kernel read-ahead and prefetch of the same block share one download+decrypt.
|
||||
- In-memory block LRU (`BlockRing`) in front of *both* block paths; a block read off disk is
|
||||
promoted into it, so the following ~31 kernel reads of that 4 MiB block cost nothing.
|
||||
- Read-only SQLite connection pool (`ReadPool` / `Db::read`) with every `SELECT`-only method
|
||||
routed at it, so a read no longer waits on the write connection.
|
||||
- `nodes.path` persisted and backfilled, replacing the recursive `path_of` walk.
|
||||
- Single-writer `flock` on the database; a `SQLITE_CORRUPT`/`NOTADB` file is quarantined as
|
||||
`cache.db.corrupt-<ts>` (WAL sidecars too) and rebuilt rather than failing the mount.
|
||||
- `VACUUM` and the deep `integrity_check` now ack immediately and run as background jobs;
|
||||
`pdfs doctor` polls for the verdict.
|
||||
- Staging reconcile at mount: staged blobs no queued operation refers to are re-queued, or
|
||||
named in the log when they cannot be addressed.
|
||||
- Queue/staging visibility — `Status` gained `parked_uploads`, `failing_ops`,
|
||||
`failing_error`, `staged_bytes`, `staged_oldest_secs`; `pdfs status` prints them when
|
||||
non-zero.
|
||||
- `statfs` implemented against the account quota; FUSE `init` negotiates `max_readahead`,
|
||||
`max_background`, `PARALLEL_DIROPS` and `CACHE_SYMLINKS`; `blksize` raised to 1 MiB.
|
||||
|
||||
### Changed
|
||||
- `State` no longer writes to SQLite under the inode lock — mutations queue into an outbox
|
||||
applied after the lock is released, so interning a 5000-child listing no longer commits a
|
||||
transaction with the whole mount frozen.
|
||||
- Re-listing an unchanged folder no longer touches its subtree; when a subtree really moved,
|
||||
one recursive walk carries both the new path and the inherited trashed flag.
|
||||
- `release(2)` no longer gap-fills from the network: it stages an incomplete blob from local
|
||||
caches only and the drain thread finishes it off the dispatch loop.
|
||||
- A partial write over an undrained one is merged instead of refused.
|
||||
- Size-upgrade waiters park on a queue instead of occupying a worker thread, capped at 8
|
||||
folders in flight; past the cap `stat` answers provisionally.
|
||||
- Block-cache disk writes moved off the read's critical path; no more fsync per cached block.
|
||||
- Cache LRU touches buffered in memory and flushed in one transaction.
|
||||
- Search: sync-folder list and pin lookups hoisted out of the per-hit loop.
|
||||
- SQLite tuned (`cache_size`, `mmap_size`, `temp_store=MEMORY`, `wal_autocheckpoint`) and
|
||||
three new indexes added.
|
||||
- Mount registry lock released before FUSE teardown/join; kernel notifications batched under
|
||||
the state lock and flushed after it.
|
||||
|
||||
### Fixed
|
||||
- **Data loss:** `ENOSPC` is now propagated from `write`/`fsync`/staging instead of being
|
||||
swallowed, with emergency eviction wired up and scratch marked durable before any staging
|
||||
error; a failed `record_pending_write` fails `release` rather than dropping the write.
|
||||
- **Data loss:** the conflict sweep no longer discards queued operations after the trash
|
||||
round trip.
|
||||
- **Data loss:** `rescue_scratch` moves a blob with an unreadable sidecar to `recovery/`
|
||||
unaccompanied instead of fabricating a sidecar for it.
|
||||
- Own-sealed revisions are persisted (7-day TTL), so a restart no longer reopens the
|
||||
transient-download fork window fixed in 1.1.0.
|
||||
- Depth-capped `is_pinned` (cycle-safe), daemon write timeout, and every client-supplied
|
||||
`limit` clamped.
|
||||
- `recover_fsynced_writes` no longer scans the recovery directory on every idle pass.
|
||||
|
||||
## [1.6.0] — 2026-08-12
|
||||
|
||||
Search-index correctness and ranking (`docs/plans/search-index.md`). Schema: 21 —
|
||||
migration V21 rebuilds `nodes_fts`.
|
||||
|
||||
### Fixed
|
||||
- **B80** — 29% of the account was missing from the search index. `node_is_indexable_tx`
|
||||
required a null-parent ancestor, which only the My Files root has, so every node under a
|
||||
device folder was dropped from `nodes_fts` on its first upsert after the v16 backfill
|
||||
(2,705 of 9,252 nodes on the account this was diagnosed against). The rule now matches the
|
||||
backfill, migration V21 replays it, nodes under a trashed folder stay out, and the walk is
|
||||
depth-capped.
|
||||
- **B81** — the prompt labelled every Drive hit "My files", including device folders; pin
|
||||
rows also took `is_dir` from `Pin::recursive`.
|
||||
- **B82** — a `parent_uid` cycle hung the daemon on any short search: the recursive CTEs in
|
||||
`path_of` and the pin walk use `UNION ALL`, which does not deduplicate. `path_of` is now
|
||||
depth-capped at 256 and returns the truncated path.
|
||||
|
||||
### Changed
|
||||
- Local search ranking: whole-query prefix matches get their own lane (newest first) ahead
|
||||
of trigram rank, which on a home directory full of dependency trees was burying the
|
||||
obvious answer — 73% of the 500 best-ranked candidates for "test" came from one Go module
|
||||
cache. The single-character lane, which recovers a typo that destroys every trigram, stays
|
||||
behind it.
|
||||
|
||||
## [1.5.0] — 2026-08-11
|
||||
|
||||
Schema: 20.
|
||||
|
||||
### Added
|
||||
- **`pdfs-prompt --dmenu`** — the same search through an external dmenu-style launcher
|
||||
(fuzzel, rofi, wofi, tofi, bemenu, dmenu) instead of the built-in GTK HUD, so the prompt
|
||||
matches a tiling-WM desktop. A launcher filters a fixed list rather than re-querying per
|
||||
keystroke, so searching is a two-step loop; `--query` skips the first step, fuzzel and rofi
|
||||
get file-type icons. Configurable as `"prompt": { "mode": "dmenu", "menu": [...] }`, with
|
||||
`--gtk` overriding for one invocation.
|
||||
- **Configurable open-with rules** (`pdfs-core/src/opener.rs`): an ordered list of name/class
|
||||
patterns in `config.json` under `open_with`, each mapped to argv and optionally wrapped in
|
||||
a terminal, first match wins, falling back to `xdg-open`. Lets "open a text file" mean
|
||||
`nvim` in a terminal without breaking every graphical caller. Absent config reproduces the
|
||||
old behaviour exactly.
|
||||
|
||||
### Fixed
|
||||
- The prompt and dmenu openers hand applications the cached file while matching rules
|
||||
against the *original* Drive filename, so e.g. `.md` files stop being routed by their
|
||||
cache-path extension.
|
||||
|
||||
## [1.4.0] — 2026-08-10
|
||||
|
||||
Schema: 20.
|
||||
|
||||
### Added
|
||||
- **File version history** (`pdfs-fuse/src/revisions.rs` + a GTK versions dialog): list a
|
||||
file's revisions, restore one, delete one, or write an old one out to a local path. A
|
||||
restore re-points the file server-side, so no content crosses the wire and nothing enters
|
||||
the drain queue; the daemon drops its cached content and repopulates on the next read.
|
||||
- **Persistent SDK entity cache** (`pdfs-core/src/sdkcache.rs`): the SDK's decrypted node
|
||||
metadata now survives a daemon restart instead of re-walking the tree. It lives in its own
|
||||
`sdk_cache.db`, deliberately off the daemon's hot `cache.db` mutex, holds no key material,
|
||||
needs no migrations, and is cleared on logout.
|
||||
- Per-node outcomes for batch operations (`pdfs-core/src/batch.rs`): `trash`/`restore`/
|
||||
`delete`/`move` report one result per node, so a node the server rejects no longer fails
|
||||
the whole batch.
|
||||
- Photo viewer page in the GUI and richer photo metadata in the details pane.
|
||||
|
||||
### Changed
|
||||
- SDK telemetry routed into `tracing`, so its transfer/block-storage/request spans land in
|
||||
the daemon log instead of a no-op sink.
|
||||
- A file that fits in one block uploads as a single atomic request rather than the
|
||||
draft/block/commit sequence.
|
||||
|
||||
## [1.3.0] — 2026-07-31
|
||||
|
||||
Schema: 19.
|
||||
|
||||
### Added
|
||||
- **Photo albums** — `PhotoAlbums` / `AlbumPhotos` control requests, daemon-side `albums.rs`
|
||||
and a GUI albums page (the former gallery view, renamed). Albums other people share with
|
||||
you are included; their photos are persisted per album because a shared album's photos
|
||||
never appear in your own timeline. Served stale-while-refreshing like the timeline.
|
||||
- **GUI "Locations" page** — one row per mount spec: the primary `~/ProtonDrive` mount
|
||||
alongside every folder this computer backs up, mirrored or on-demand. Called *Locations*
|
||||
rather than *Mounts* because a mirror folder is a plain directory with no FUSE session.
|
||||
It absorbs the old Computers page's actions and the mountpoint chooser from Settings.
|
||||
- `role` on shared-node metadata (`viewer`/`editor`/`admin`), defaulted for wire-compat with
|
||||
older clients and daemons.
|
||||
- FUSE acceptance coverage for the permission-based mounts.
|
||||
|
||||
## [1.2.1] — 2026-07-31
|
||||
|
||||
Schema: 18.
|
||||
|
||||
### Fixed
|
||||
- Migration bug that could leave the `ProtonDrive` mount unwritable after upgrade.
|
||||
- Permission error on on-demand mounts.
|
||||
- B78.
|
||||
|
||||
### Changed
|
||||
- Release pipeline: `.deb` signing added and `.rpm` signing fixed.
|
||||
|
||||
## [1.2.0] — 2026-07-29
|
||||
|
||||
Sharing and multi-mount release. Schema: 18.
|
||||
|
||||
### Added
|
||||
- **Shared with me**: shares others sent you are listed, verified and mountable, including on
|
||||
the primary mount.
|
||||
They appear under a synthetic `Shared with me` directory on a virtual volume
|
||||
(`pdfs-fuse/src/virtual.rs`), with name collisions and over-long components resolved by
|
||||
its own naming rules.
|
||||
- Per-share access model (`db/share_access.rs`): the effective role of each shared root is
|
||||
persisted and kept resident in mount state, so interning an inode never costs a query. The
|
||||
database stays the restart/offline authority — every role change is written through before
|
||||
the cache updates. A share you cannot write is presented POSIX read-only.
|
||||
- Unified location model (`pdfs-core/src/mounts.rs`, `db/mounts.rs`): one presentation shape
|
||||
for every local Proton Drive location, joining device sync state with a new `mount` table.
|
||||
`MountMode` is `Mirror` (full local copy, reconciled both ways) or `OnDemand` (FUSE session
|
||||
fetching on access), with an `Unknown` fallback so a mode written by a newer client does
|
||||
not break an older one.
|
||||
|
||||
## [1.1.1] — 2026-07-27
|
||||
|
||||
Schema: 16.
|
||||
|
||||
### Fixed
|
||||
- Trash refresh did not pull updates.
|
||||
- Packaging: `PKGBUILD` was left at 1.1.0 by the 1.1.1 version bump, failing the release
|
||||
workflow's tag/manifest/PKGBUILD agreement check.
|
||||
|
||||
## [1.1.0] — 2026-07-26
|
||||
|
||||
Conflict-handling and concurrency release. Schema: 16.
|
||||
|
||||
### Added
|
||||
- Conflict sweep (`pdfs-fuse/src/sweep.rs`, 5-minute cadence): removes conflict copies proven
|
||||
identical by size + `content_sha1`, surfaces divergent ones once as an activity entry, and
|
||||
never removes what it cannot prove.
|
||||
- Sweep mode and `.pdfsignore`-driven conflict handling; ignore rules are rebuilt per sync
|
||||
pass so edits take effect without a restart.
|
||||
- Substantially extended FUSE acceptance suite.
|
||||
|
||||
### Fixed
|
||||
- **B69** — spurious `(sync-conflict)` copies. Revision identity now uses the server's
|
||||
`revision_id` rather than `(mtime, size)`, which the server re-stamps on seal; the planner
|
||||
gained `AdoptBaseline` so an equal-size no-baseline pair is recorded instead of forked on
|
||||
first reconcile.
|
||||
- **B70** — browser downloads into an on-demand mount had their in-flight temporaries sealed
|
||||
as canonical revisions. Transient names are recognised and held until the name is finalized.
|
||||
- Assorted race conditions and concurrency bugs on the write/drain paths.
|
||||
|
||||
### Changed
|
||||
- Workspace deps bumped to `proton-sdk` / `proton-drive-rs` 0.2.2.
|
||||
|
||||
## [1.0.0] — 2026-07-22
|
||||
|
||||
First stable release: FUSE files-on-demand mount, sync daemon under `proton-drive.service`,
|
||||
`pdfs` CLI and the GTK4 app, on top of the pure-Rust Proton Drive SDK. Schema: 16.
|
||||
|
||||
### Fixed
|
||||
- The outstanding FUSE defects tracked in `docs/BUGS.md`, plus a truncate defect, validated
|
||||
by a new POSIX compliance suite for the filesystem.
|
||||
|
||||
[1.8.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.8.2
|
||||
[1.8.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.8.1
|
||||
[1.8.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.8.0
|
||||
[1.7.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.7.1
|
||||
[1.7.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.7.0
|
||||
[1.6.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.6.0
|
||||
[1.5.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.5.0
|
||||
[1.4.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.4.0
|
||||
[1.3.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.3.0
|
||||
[1.2.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.2.1
|
||||
[1.2.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.2.0
|
||||
[1.1.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.1.1
|
||||
[1.1.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.1.0
|
||||
[1.0.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.0.0
|
||||
+2
-1
@@ -14,7 +14,8 @@ Consolidated from the historical feature roadmap; current bug status lives in [`
|
||||
* **Custom FUSE Mount Options (Horizon 1)**: Expose custom mounting parameters in configuration (e.g. `ro` for read-only, `direct_io` to bypass kernel caching, `allow_other`). Auto-generate systemd user mount units upon sync folder configuration.
|
||||
* **Symbolic Link & Hard Link Virtualization (Horizon 2)**: Add symbolic link virtualization inside the metadata store (`NodeType::Symlink`) by intercepting `readlink(2)` and `symlink(2)`. Targets should sync remotely as small encrypted metadata placeholder files on Drive (`proton-vfs-symlink: <target>`).
|
||||
* **Desktop File Manager Integration (Horizon 2)**: Develop shell extension plugins for popular Linux file managers (Nautilus Python plugin, Dolphin Service Menus, Thunar custom actions) to offer context-menu shortcuts ("Pin", "Unpin", "Copy Share Link", "View Version History") and file overlay sync badges.
|
||||
* **Integrated Photo Gallery Enhancements (Horizon 2)**: Support creating and editing Proton Photo Albums (album *writes* are unported in the SDK too) and create an auto-upload Pictures folder pipeline. Exif display, the date-based timeline scrubber and favourites are implemented.
|
||||
* **Integrated Photo Gallery Enhancements (Horizon 2)**: Album *editing* — rename, cover photo, delete, and removing photos from an album — is still unported in the SDK (TS `updateAlbum` / `deleteAlbum` / `removePhotos`), as is adding a photo that lives on someone else's volume (TS `copyPhoto`). Album **creation** and adding own-volume photos landed with the Google Photos import. Still open: an auto-upload Pictures folder pipeline, and a GUI for creating albums by hand. Exif display, the date-based timeline scrubber and favourites are implemented.
|
||||
* **Google Photos import follow-ups**: the importer uploads photos one at a time — a large export would finish sooner with a bounded concurrent upload pool. Trash folders are recognized by a hardcoded list of localized names ([`takeout.rs`](../crates/pdfs-core/src/takeout.rs) `TRASH_FOLDERS`); a locale not on it imports its trash as ordinary photos. Motion-photo `.MP`/`.MV` sidecar videos are ignored rather than uploaded as related photos. Live validation against a real export is pending.
|
||||
* **Random-Access Media Streaming seek support (Horizon 2)**: Intercept out-of-order sparse reads (player seeks) in `pdfs-fuse` and prioritize downloading blocks around the seek offset, aborting/postponing sequential pre-fetches.
|
||||
* **Dynamic Sync Dashboard & Queue Visualization (Horizon 2)**: GUI real-time transfer list (progress bars, speed, ETA), sync history feed, storage usage breakdown, and global pause/resume sync controls.
|
||||
* **Interactive Terminal UI (TUI) Mode (Horizon 3)**: Implement `pdfs tui` using the `ratatui` crate to show transfer speed, sync status, active transfer queue with progress bars, and scrolling daemon logs.
|
||||
|
||||
@@ -1,110 +0,0 @@
|
||||
# Search index + prompt display fixes
|
||||
|
||||
Working plan. Findings come from an audit of a real `cache.db`
|
||||
(9,252 Drive nodes, 110,187 local files) on 2026-08-12.
|
||||
|
||||
## Evidence
|
||||
|
||||
```
|
||||
nodes 9252
|
||||
nodes_fts 6547 ← 2705 rows never indexed
|
||||
nodes in orphan subtrees 2705 ← exactly the missing set
|
||||
local_files 110187
|
||||
/home/narl/go/pkg/mod 63399 (58% of the local index)
|
||||
query "test" 46405 local matches, candidate pool capped at 1000
|
||||
top 500 by fts rank 367 (73%) from go/pkg/mod
|
||||
```
|
||||
|
||||
`pdfs search tickets` returns only `Documents/Tickets` (My Files); the
|
||||
`tickets.pdf` sitting in the device folder `Downloads` is invisible.
|
||||
|
||||
## P1 — 29% of Drive is unsearchable
|
||||
|
||||
`node_is_indexable_tx` (`crates/pdfs-core/src/db/nodes.rs:606`) indexes a node
|
||||
only when the ancestor walk terminates at a row with `parent_uid IS NULL`. The
|
||||
DB holds exactly one such row (the My Files root). Device roots live in
|
||||
`device`/`sync_folder`, never in `nodes`, so every device-folder subtree
|
||||
(`Downloads`, `Pictures`, `Documents`, `Music`, `Videos` under device
|
||||
*nthinkpad*) walks up to a `parent_uid` with no row and is judged unindexable.
|
||||
Stale `pdfs-acceptance-*` roots hit the same path.
|
||||
|
||||
- [x] Relax the rule to "the walk terminates and no ancestor is trashed",
|
||||
keeping the existing cycle guard. A subtree whose top row's parent is
|
||||
simply not cached is still real, reachable data.
|
||||
- [x] `MIGRATION_V21`: rebuild `nodes_fts` under the new rule, so existing
|
||||
installs recover the missing rows instead of waiting for each node to be
|
||||
upserted again.
|
||||
- [x] Tests: a node under an uncached parent is indexed; a trashed ancestor
|
||||
still excludes; the migration backfills.
|
||||
|
||||
### Found while testing: a parent cycle hangs the daemon
|
||||
|
||||
Writing the cycle test surfaced a live defect, not a test artefact. A query
|
||||
below the trigram minimum takes the `LIKE` lane, which reads `nodes` directly
|
||||
and never consults the index, and every row it returns goes through `path_of` —
|
||||
whose parent walk is an unguarded `UNION ALL`. On a `parent_uid` cycle it never
|
||||
terminates, while holding the daemon's only SQLite connection.
|
||||
|
||||
- [x] Depth-cap `path_of` (`db/utils.rs`), matching the 1024 cap
|
||||
`path_relative_to` already carried.
|
||||
- [x] Depth-cap the `descendants` CTE in `upsert_node_tx`, which has the same
|
||||
exposure on the write side.
|
||||
- [x] Keep cycles out of the index (part of the P1 rule above).
|
||||
|
||||
Still unguarded, not touched here: `pin_is_pinned`'s ancestor walk.
|
||||
|
||||
## P2 — local index noise crowds real files out of the candidate pool
|
||||
|
||||
`SKIP_DIRS` (`crates/pdfs-core/src/localindex.rs:23`) misses the Go module
|
||||
cache and build output dirs. `search_v2` clamps candidates to ≤1000 and scores
|
||||
*after* SQLite, so for any common term the pool is exhausted by vendored files
|
||||
before a real one is considered.
|
||||
|
||||
- [x] Skip `go/pkg/mod` (path-relative, not a bare name), `dist`, `build`,
|
||||
`.cache`-style artefact dirs.
|
||||
- [x] Give `search_local_candidates` a name-prefix lane, mirroring what
|
||||
`search_candidates` already does for Drive, so an exact-prefix match
|
||||
cannot be starved by trigram rank.
|
||||
- [x] Tests for both.
|
||||
|
||||
## P3 — display lies about where a hit lives
|
||||
|
||||
- [x] `Hit::location()` (`crates/pdfs-gui/src/prompt.rs:150`) hardcodes
|
||||
`My files / …` for every Drive hit. Derive the label from
|
||||
`mounted_path` when the daemon resolved one (device folder, sync folder)
|
||||
and fall back to `My files` only for the primary mount.
|
||||
- [x] Pin rows set `is_dir: pin.recursive` (`dmenu.rs:133`, `prompt.rs:872`) —
|
||||
a non-recursively pinned folder renders and activates as a file.
|
||||
- [ ] `pdfs locations` prints `Device folder 1..5` rather than the device and
|
||||
folder name. (Cosmetic; deferred.)
|
||||
|
||||
## Verification
|
||||
|
||||
Done, 2026-08-12:
|
||||
|
||||
```
|
||||
cargo fmt --all -- --check clean
|
||||
cargo clippy --workspace --all-targets -- -D warnings clean
|
||||
cargo test --workspace --locked 439 passed, 0 failed
|
||||
```
|
||||
|
||||
The V21 backfill was also replayed against a read-only `.backup` of the live
|
||||
`cache.db`:
|
||||
|
||||
```
|
||||
live_nodes 9276
|
||||
indexed 9276 (was 6547)
|
||||
downloads_subtree 300
|
||||
tickets.pdf -> Downloads/tickets.pdf
|
||||
```
|
||||
|
||||
That run is what caught the last defect: the backfill originally seeded an
|
||||
uncached-parent root with an empty path, so a node indexed by the migration
|
||||
(`tickets.pdf`) and the same node after a later upsert (`Downloads/tickets.pdf`)
|
||||
disagreed. The seed now keeps the name for any row that has a parent, matching
|
||||
`path_of`.
|
||||
|
||||
Still to do, on the live system: rebuild, restart `proton-drive.service`, and
|
||||
confirm `pdfs search tickets` returns the device folder's `tickets.pdf`. The
|
||||
schema move to 21 is one-way — an older `pdfs` refuses a v21 database. The
|
||||
`local_files` shrink (~110k → ~47k) lands with the next background scan.
|
||||
@@ -0,0 +1,26 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg width="128" height="128" viewBox="0 0 128 128" fill="none" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Proton Drive for Linux">
|
||||
<defs>
|
||||
<linearGradient id="pdl-bg" x1="0" y1="0" x2="0" y2="128" gradientUnits="userSpaceOnUse">
|
||||
<stop offset="0" stop-color="#8B72FF"/>
|
||||
<stop offset="1" stop-color="#5A38E8"/>
|
||||
</linearGradient>
|
||||
</defs>
|
||||
|
||||
<rect width="128" height="128" rx="29" fill="url(#pdl-bg)"/>
|
||||
|
||||
<!-- cloud: the remote drive (symmetric about x=64) -->
|
||||
<g fill="#FFFFFF">
|
||||
<circle cx="64" cy="42" r="20"/>
|
||||
<circle cx="44" cy="52" r="14"/>
|
||||
<circle cx="84" cy="52" r="14"/>
|
||||
<rect x="32" y="50" width="64" height="16" rx="8"/>
|
||||
</g>
|
||||
|
||||
<!-- pull: files arrive only when opened -->
|
||||
<g stroke="#FFFFFF" stroke-width="8" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M64 75v17"/>
|
||||
<path d="M53 84l11 11 11-11"/>
|
||||
<path d="M45 108h38" stroke-opacity="0.55"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 990 B |
+4
-4
@@ -9,13 +9,13 @@
|
||||
# Build + install: cd packaging && makepkg -fi
|
||||
|
||||
pkgname=proton-drive-linux
|
||||
pkgver=1.7.0
|
||||
pkgver=1.9.1
|
||||
pkgrel=1
|
||||
pkgdesc="Proton Drive client for Linux: FUSE files-on-demand mount, CLI, GTK4 app + tray"
|
||||
pkgdesc="Unofficial Proton Drive client for Linux: files-on-demand FUSE mount, CLI, GTK4 app + tray"
|
||||
arch=('x86_64')
|
||||
url="https://github.com/narl/proton-drive-linux"
|
||||
url="https://github.com/narrrl/proton-drive-linux"
|
||||
license=('MIT')
|
||||
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0')
|
||||
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0' 'perl-image-exiftool')
|
||||
makedepends=('cargo' 'git')
|
||||
provides=('pdfs')
|
||||
# Files-on-demand mount + keyring need these at runtime.
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
[Desktop Entry]
|
||||
Type=Application
|
||||
Name=Proton Drive (Tray)
|
||||
Name=Proton Drive for Linux (Tray)
|
||||
GenericName=Cloud Drive Tray
|
||||
Comment=Proton Drive system tray icon
|
||||
Comment=Proton Drive status and quick actions (unofficial client)
|
||||
Exec=pdfs-tray
|
||||
Icon=io.narl.proton-drive-linux
|
||||
Terminal=false
|
||||
Categories=Network;FileTransfer;GTK;
|
||||
Keywords=proton;drive;cloud;sync;storage;
|
||||
Keywords=proton;drive;cloud;sync;storage;files;backup;
|
||||
NoDisplay=true
|
||||
@@ -1,12 +1,12 @@
|
||||
[Desktop Entry]
|
||||
Type=Application
|
||||
Name=Proton Drive
|
||||
Name=Proton Drive for Linux
|
||||
GenericName=Cloud Drive
|
||||
Comment=Proton Drive files-on-demand for Linux
|
||||
Comment=Files-on-demand Proton Drive (unofficial client)
|
||||
Exec=pdfs-app
|
||||
Icon=io.narl.proton-drive-linux
|
||||
Terminal=false
|
||||
Categories=Network;FileTransfer;GTK;
|
||||
Keywords=proton;drive;cloud;sync;storage;
|
||||
Keywords=proton;drive;cloud;sync;storage;files;backup;
|
||||
StartupNotify=true
|
||||
SingleMainWindow=true
|
||||
@@ -1,14 +1,26 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg width="128" height="128" viewBox="0 0 128 128" xmlns="http://www.w3.org/2000/svg">
|
||||
<svg width="128" height="128" viewBox="0 0 128 128" fill="none" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Proton Drive for Linux">
|
||||
<defs>
|
||||
<linearGradient id="g" x1="0" y1="0" x2="0" y2="1">
|
||||
<stop offset="0" stop-color="#8b6dff"/>
|
||||
<stop offset="1" stop-color="#6d4aff"/>
|
||||
<linearGradient id="pdl-bg" x1="0" y1="0" x2="0" y2="128" gradientUnits="userSpaceOnUse">
|
||||
<stop offset="0" stop-color="#8B72FF"/>
|
||||
<stop offset="1" stop-color="#5A38E8"/>
|
||||
</linearGradient>
|
||||
</defs>
|
||||
<rect x="8" y="8" width="112" height="112" rx="26" fill="url(#g)"/>
|
||||
<!-- cloud -->
|
||||
<path fill="#ffffff" d="M44 84c-11 0-20-9-20-20 0-10 7-18 17-20 3-11 13-19 25-19 13 0 24 10 26 23 9 1 16 9 16 18 0 10-8 18-18 18z"/>
|
||||
<!-- down arrow (on-demand pull) -->
|
||||
<path d="M64 50v22m0 0l-9-9m9 9l9-9" stroke="#6d4aff" stroke-width="7" stroke-linecap="round" stroke-linejoin="round" fill="none"/>
|
||||
|
||||
<rect width="128" height="128" rx="29" fill="url(#pdl-bg)"/>
|
||||
|
||||
<!-- cloud: the remote drive (symmetric about x=64) -->
|
||||
<g fill="#FFFFFF">
|
||||
<circle cx="64" cy="42" r="20"/>
|
||||
<circle cx="44" cy="52" r="14"/>
|
||||
<circle cx="84" cy="52" r="14"/>
|
||||
<rect x="32" y="50" width="64" height="16" rx="8"/>
|
||||
</g>
|
||||
|
||||
<!-- pull: files arrive only when opened -->
|
||||
<g stroke="#FFFFFF" stroke-width="8" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M64 75v17"/>
|
||||
<path d="M53 84l11 11 11-11"/>
|
||||
<path d="M45 108h38" stroke-opacity="0.55"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 718 B After Width: | Height: | Size: 990 B |
@@ -12,9 +12,9 @@
|
||||
Name: proton-drive-linux
|
||||
Version: %{version}
|
||||
Release: 1%{?dist}
|
||||
Summary: Proton Drive client for Linux (FUSE, CLI, GTK4 app + tray)
|
||||
Summary: Unofficial Proton Drive client for Linux (FUSE, CLI, GTK4 app + tray)
|
||||
License: MIT
|
||||
URL: https://github.com/narl/proton-drive-linux
|
||||
URL: https://github.com/narrrl/proton-drive-linux
|
||||
ExclusiveArch: x86_64
|
||||
|
||||
BuildRequires: cargo
|
||||
@@ -33,6 +33,7 @@ Requires: gtk4
|
||||
Requires: libadwaita
|
||||
Requires: libsecret
|
||||
Requires: webkitgtk6.0
|
||||
Requires: perl-Image-ExifTool
|
||||
Requires: xdg-utils
|
||||
|
||||
# DE-specific; do not Require a single desktop environment.
|
||||
@@ -43,8 +44,9 @@ Recommends: kwallet
|
||||
Provides: pdfs = %{version}-%{release}
|
||||
|
||||
%description
|
||||
Unofficial Proton Drive client featuring a FUSE files-on-demand mount,
|
||||
CLI, GTK4/Libadwaita GUI, system tray, and search launcher.
|
||||
Files-on-demand Proton Drive for the Linux desktop: a FUSE virtual mount with
|
||||
block-level caching, a scriptable CLI, and a GTK4/Libadwaita desktop app with
|
||||
system tray and search launcher. Unofficial and not affiliated with Proton AG.
|
||||
|
||||
%prep
|
||||
# In-tree build: no Source tarball. Pass --define "git_dir /path/to/checkout".
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[Unit]
|
||||
Description=Proton Drive Files-On-Demand mount
|
||||
Documentation=https://github.com/narl/proton-drive-linux
|
||||
Documentation=https://github.com/narrrl/proton-drive-linux
|
||||
# Tie the mount to the desktop session: start after it is up and stop with it.
|
||||
After=graphical-session.target
|
||||
PartOf=graphical-session.target
|
||||
|
||||
Reference in new issue
Block a user