mirror of
https://github.com/narrrl/proton-drive-linux.git
synced 2026-11-03 16:41:26 +01:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f4d6580cd3 | ||
|
|
b2f62f70e1 | ||
|
|
dcb2ab7fac | ||
|
|
eb56f4a1e0 | ||
|
|
c28803b037 | ||
|
|
cd8474954b | ||
|
|
d7e07d042d | ||
|
|
45852082c0 | ||
|
|
99ad3838e1 | ||
|
|
637b512f6a | ||
|
|
16ad6548c8 | ||
|
|
847ece17d7 | ||
|
|
50d2acdaba | ||
|
|
53caf63ad4 | ||
|
|
5176dd91e1 | ||
|
|
e88903d5ff | ||
|
|
b3e6af1ae5 | ||
|
|
b0f9513915 | ||
|
|
91b0864fce | ||
|
|
8bd9481902 | ||
|
|
b2d5870834 | ||
|
|
078774f6a1 | ||
|
|
326a9b4e92 | ||
|
|
78dc72a34d | ||
|
|
438b2b463e | ||
|
|
4cd49168c4 | ||
|
|
481c3fe0c7 | ||
|
|
098da2e97a | ||
|
|
d03d3eac3f | ||
|
|
433e206203 | ||
|
|
35418eea3f | ||
|
|
e48c4bb4b2 | ||
|
|
7ad9231858 | ||
|
|
e396c6b4d6 |
No files matched your search
@@ -0,0 +1,76 @@
|
||||
name: Bug report
|
||||
description: Something in the client misbehaves
|
||||
labels: ["bug"]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Please check [`docs/BUGS.md`](../blob/main/docs/BUGS.md) first — the issue may already be
|
||||
tracked there with context.
|
||||
|
||||
**Do not report a security vulnerability here.** See [SECURITY.md](../blob/main/SECURITY.md).
|
||||
- type: textarea
|
||||
id: what
|
||||
attributes:
|
||||
label: What happened
|
||||
description: What you did, what you expected, and what happened instead.
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: repro
|
||||
attributes:
|
||||
label: Steps to reproduce
|
||||
placeholder: |
|
||||
1. …
|
||||
2. …
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: version
|
||||
attributes:
|
||||
label: Version
|
||||
description: Output of `pdfs --version`, and how you installed it (deb / rpm / AUR / source).
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: environment
|
||||
attributes:
|
||||
label: Distribution and desktop
|
||||
placeholder: "Fedora 44, GNOME 48 (Wayland)"
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: surface
|
||||
attributes:
|
||||
label: Where does it happen
|
||||
multiple: true
|
||||
options:
|
||||
- FUSE mount / file operations
|
||||
- Sync or backed-up folders (Computers)
|
||||
- CLI (pdfs)
|
||||
- Desktop app (pdfs-app)
|
||||
- Tray (pdfs-tray)
|
||||
- Search launcher (pdfs-prompt)
|
||||
- Login / keyring
|
||||
- Photos
|
||||
- Packaging / installation
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: diagnose
|
||||
attributes:
|
||||
label: "`pdfs diagnose` output"
|
||||
description: Redact paths or the account address if you prefer.
|
||||
render: console
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Daemon logs
|
||||
description: "`journalctl --user -u proton-drive.service -n 200 --no-pager`"
|
||||
render: console
|
||||
- type: checkboxes
|
||||
id: dataloss
|
||||
attributes:
|
||||
label: Data
|
||||
options:
|
||||
- label: This bug involved missing, truncated, or overwritten file content.
|
||||
@@ -0,0 +1,11 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: Security vulnerability
|
||||
url: https://github.com/narrrl/proton-drive-linux/security/advisories/new
|
||||
about: Report privately — never in a public issue.
|
||||
- name: Known issues
|
||||
url: https://github.com/narrrl/proton-drive-linux/blob/main/docs/BUGS.md
|
||||
about: The authoritative ledger of tracked bugs and their status.
|
||||
- name: Proton Drive service problems
|
||||
url: https://proton.me/support
|
||||
about: This is an unofficial client. Account or service issues go to Proton AG.
|
||||
@@ -0,0 +1,37 @@
|
||||
name: Feature request
|
||||
description: Suggest a capability or an improvement
|
||||
labels: ["enhancement"]
|
||||
body:
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: The problem
|
||||
description: What are you trying to do that the client makes hard or impossible today?
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: proposal
|
||||
attributes:
|
||||
label: What you'd like to see
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: surface
|
||||
attributes:
|
||||
label: Which part of the client
|
||||
multiple: true
|
||||
options:
|
||||
- FUSE mount
|
||||
- Sync / Computers
|
||||
- CLI (pdfs)
|
||||
- Desktop app (pdfs-app)
|
||||
- Tray (pdfs-tray)
|
||||
- Search launcher (pdfs-prompt)
|
||||
- Photos
|
||||
- Packaging
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Alternatives you considered
|
||||
@@ -0,0 +1,26 @@
|
||||
## What this changes
|
||||
|
||||
<!-- The user-visible effect, in a sentence or two. -->
|
||||
|
||||
**Crates touched:**
|
||||
**Related `docs/BUGS.md` entries:**
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
cargo fmt --all -- --check
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
cargo test --workspace --locked
|
||||
```
|
||||
|
||||
<!-- Plus whatever else you ran: acceptance suite, a live mount, manual steps. -->
|
||||
|
||||
## Implications
|
||||
|
||||
- [ ] Adds a SQLite migration (new `MIGRATION_V*`, `SCHEMA_VERSION` bumped — no shipped migration edited)
|
||||
- [ ] Touches recovery, staging, or anything that can be the only copy of user data
|
||||
- [ ] Changes packaging, the systemd unit, or desktop entries
|
||||
- [ ] Changes the control protocol (daemon and front ends updated together)
|
||||
- [ ] None of the above
|
||||
|
||||
<!-- Screenshots for GTK changes. -->
|
||||
@@ -122,13 +122,14 @@ jobs:
|
||||
Priority: optional
|
||||
Architecture: amd64
|
||||
Maintainer: Nils Pukropp <contact@narl.io>
|
||||
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4
|
||||
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4, libimage-exiftool-perl
|
||||
Installed-Size: ${INSTALLED_SIZE}
|
||||
Description: Proton Drive client for Linux
|
||||
Proton Drive client for Linux featuring:
|
||||
- FUSE files-on-demand mount
|
||||
- Command-line interface (CLI)
|
||||
- GTK4 system tray, desktop application, and search launcher
|
||||
Homepage: https://github.com/narrrl/proton-drive-linux
|
||||
Description: Unofficial Proton Drive client for Linux
|
||||
Files-on-demand Proton Drive for the Linux desktop, unaffiliated with Proton AG:
|
||||
- FUSE virtual mount with block-level caching
|
||||
- Scriptable command-line interface (pdfs)
|
||||
- GTK4 desktop application, system tray, and search launcher
|
||||
EOF
|
||||
|
||||
# Build Debian package
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Code of Conduct
|
||||
|
||||
## Our pledge
|
||||
|
||||
We want participation in this project to be a harassment-free experience for everyone, regardless
|
||||
of age, body size, disability, ethnicity, sex characteristics, gender identity and expression,
|
||||
level of experience, education, socioeconomic status, nationality, personal appearance, race,
|
||||
religion, or sexual identity and orientation.
|
||||
|
||||
## Standards
|
||||
|
||||
Examples of behavior that helps:
|
||||
|
||||
- being respectful of differing viewpoints and experience levels,
|
||||
- giving and accepting constructive feedback gracefully,
|
||||
- focusing on what is best for the project and its users.
|
||||
|
||||
Examples of unacceptable behavior:
|
||||
|
||||
- sexualized language or imagery, and unwelcome sexual attention,
|
||||
- trolling, insulting or derogatory comments, and personal or political attacks,
|
||||
- public or private harassment,
|
||||
- publishing others' private information without explicit permission.
|
||||
|
||||
## Scope and enforcement
|
||||
|
||||
This applies in all project spaces — issues, pull requests, discussions — and when representing the
|
||||
project in public. Report unacceptable behavior to <contact@narl.io>. Reports are handled
|
||||
confidentially. Maintainers may edit, remove, or reject contributions that violate this document,
|
||||
and may temporarily or permanently ban any contributor for behavior they deem inappropriate.
|
||||
|
||||
Adapted from the [Contributor Covenant](https://www.contributor-covenant.org), version 2.1.
|
||||
@@ -0,0 +1,70 @@
|
||||
# Contributing
|
||||
|
||||
Thanks for taking the time. This is an unofficial Proton Drive client that people trust with real
|
||||
files, so the bar for changes that touch the filesystem, the sync engine, or the database is
|
||||
deliberately high — everything else is ordinary Rust review.
|
||||
|
||||
## Before you start
|
||||
|
||||
- Read [`AGENTS.md`](AGENTS.md) for repository conventions and
|
||||
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for how the daemon, front ends, and SDK fit
|
||||
together.
|
||||
- Check [`docs/BUGS.md`](docs/BUGS.md) — it is the authoritative issue ledger, and your problem may
|
||||
already be tracked there with context.
|
||||
- For anything larger than a fix, open an issue first so the design can be discussed before you
|
||||
spend the time.
|
||||
|
||||
## Development setup
|
||||
|
||||
```bash
|
||||
git clone https://github.com/narrrl/proton-drive-linux.git
|
||||
cd proton-drive-linux
|
||||
cargo build --workspace
|
||||
```
|
||||
|
||||
GUI and FUSE targets need the GTK4, libadwaita, WebKitGTK 6, libsecret, D-Bus, and FUSE3
|
||||
development packages listed in the [README](README.md#prerequisites). Minimum supported Rust
|
||||
version is 1.96.
|
||||
|
||||
## Quality gates
|
||||
|
||||
CI runs exactly these, and they must pass with zero warnings:
|
||||
|
||||
```bash
|
||||
cargo fmt --all -- --check
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
cargo test --workspace --locked
|
||||
scripts/fuse-acceptance.sh --offline-only
|
||||
```
|
||||
|
||||
Changes to the FUSE layer, sync, or the pending-operation queue should also be exercised against a
|
||||
real mount before review — see [`docs/TESTING.md`](docs/TESTING.md). Live runs are destructive:
|
||||
use a dedicated test account, never one holding data you cannot lose.
|
||||
|
||||
## Making changes
|
||||
|
||||
- **Keep shared logic in `pdfs-core`.** The CLI and GUI never touch the database or the network;
|
||||
they speak the control protocol to the daemon.
|
||||
- **The SQLite schema is forward-only.** Never edit a shipped migration — add a new one and bump
|
||||
`SCHEMA_VERSION` in `crates/pdfs-core/src/db/migrations.rs`.
|
||||
- **Typed errors and `?` over panics** on runtime paths. Blocking filesystem, keyring, or
|
||||
control-socket work stays off GTK's main thread.
|
||||
- **`staging/` and `recovery/` may hold the only copy of user data.** Nothing may clear or purge
|
||||
them opportunistically.
|
||||
|
||||
## Commits and pull requests
|
||||
|
||||
Commits follow scoped [Conventional Commits](https://www.conventionalcommits.org), e.g.
|
||||
`fix(fuse,core): keep truncate composable with queued revisions`.
|
||||
|
||||
A pull request should state:
|
||||
|
||||
- the user-visible effect and which crates it touches,
|
||||
- the `docs/BUGS.md` entries it fixes or affects,
|
||||
- the commands you ran to verify it,
|
||||
- screenshots for GTK changes,
|
||||
- and, explicitly, any migration, recovery, data-loss, or packaging implications.
|
||||
|
||||
## Security
|
||||
|
||||
Do not open a public issue for a vulnerability — see [SECURITY.md](SECURITY.md).
|
||||
Generated
+568
-789
File diff suppressed because it is too large.
Load diff
+24
-8
@@ -3,15 +3,20 @@ resolver = "2"
|
||||
members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"]
|
||||
|
||||
[workspace.package]
|
||||
version = "1.6.0"
|
||||
version = "1.10.1"
|
||||
edition = "2024"
|
||||
license = "MIT"
|
||||
authors = ["Nils Pukropp <contact@narl.io>"]
|
||||
rust-version = "1.96"
|
||||
repository = "https://github.com/narrrl/proton-drive-linux"
|
||||
homepage = "https://github.com/narrrl/proton-drive-linux"
|
||||
readme = "README.md"
|
||||
keywords = ["proton", "drive", "fuse", "sync", "linux"]
|
||||
categories = ["filesystem", "command-line-utilities"]
|
||||
|
||||
[workspace.dependencies]
|
||||
proton-sdk = { version = "0.5.1" }
|
||||
proton-drive-rs = { version = "0.5.1" }
|
||||
proton-sdk = { version = "0.6.4" }
|
||||
proton-drive-rs = { version = "0.6.4" }
|
||||
|
||||
# Internal crates.
|
||||
pdfs-core = { path = "crates/pdfs-core" }
|
||||
@@ -20,6 +25,7 @@ pdfs-fuse = { path = "crates/pdfs-fuse" }
|
||||
tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "fs", "io-util", "time", "signal"] }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
base64 = "0.23"
|
||||
thiserror = "2"
|
||||
anyhow = "1"
|
||||
# Non-poisoning mutexes. A handler that panics while holding a lock must not
|
||||
@@ -29,18 +35,28 @@ anyhow = "1"
|
||||
parking_lot = "0.12"
|
||||
tracing = "0.1"
|
||||
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
|
||||
directories = "5"
|
||||
directories = "6"
|
||||
ignore = "0.4"
|
||||
sha2 = "0.10"
|
||||
sha2 = "0.11"
|
||||
# Google Takeout exports are zip sets; the importer reads their central
|
||||
# directories and streams entries out without extracting to disk first.
|
||||
zip = { version = "8", default-features = false, features = ["deflate"] }
|
||||
# Proton's photo duplicate detection is keyed on the plaintext SHA-1, so an
|
||||
# import has to speak SHA-1 whatever its age.
|
||||
sha1 = "0.11"
|
||||
# Thumbnail generation for photos the server has none for. Decoders only for the
|
||||
# formats a camera or phone actually produces — the rest of the crate's codecs
|
||||
# would be dead weight in the daemon.
|
||||
image = { version = "0.25", default-features = false, features = ["jpeg", "png", "webp", "gif", "tiff", "bmp", "rayon"] }
|
||||
rusqlite = { version = "0.32", features = ["bundled"] }
|
||||
keyring = { version = "3", features = ["sync-secret-service", "crypto-rust"] }
|
||||
rusqlite = { version = "0.40", features = ["bundled"] }
|
||||
# keyring 4 split into `keyring-core` plus per-platform stores. The libdbus
|
||||
# Secret Service store is the one keyring 3's `sync-secret-service` wrapped, and
|
||||
# with no `target` it matches items v3 wrote, so existing logins survive.
|
||||
keyring-core = "1"
|
||||
dbus-secret-service-keyring-store = { version = "1", features = ["crypto-rust"] }
|
||||
clap = { version = "4", features = ["derive"] }
|
||||
rpassword = "7"
|
||||
async-trait = "0.1"
|
||||
futures = "0.3"
|
||||
fuser = "0.17"
|
||||
fuser = "0.18"
|
||||
libc = "0.2"
|
||||
@@ -1,17 +1,75 @@
|
||||
# Proton Drive client for Linux (unofficial)
|
||||
<div align="center">
|
||||
|
||||
A fast, unofficial Proton Drive client for Linux. This client features an advanced files-on-demand FUSE virtual mount with block-level caching, a command-line interface (CLI), and a fully non-blocking GTK4 desktop application with system tray integration.
|
||||
<img src="images/logo.svg" alt="" width="112" height="112">
|
||||
|
||||
# Proton Drive for Linux
|
||||
|
||||
**Files-on-demand Proton Drive for the Linux desktop** — a FUSE virtual mount with block-level
|
||||
caching, a scriptable CLI, and a native GTK4 app with tray and search launcher.
|
||||
|
||||
[](https://github.com/narrrl/proton-drive-linux/actions/workflows/ci.yml)
|
||||
[](https://github.com/narrrl/proton-drive-linux/releases/latest)
|
||||
[](LICENSE)
|
||||
[](https://www.rust-lang.org)
|
||||
[](#prerequisites)
|
||||
|
||||
<img src="images/files.png" alt="The GTK4 file browser" width="820">
|
||||
|
||||
</div>
|
||||
|
||||
> [!IMPORTANT]
|
||||
> This is an **unofficial**, community-built client. It is not affiliated with, endorsed by, or
|
||||
> supported by Proton AG. "Proton" and "Proton Drive" are trademarks of Proton AG, used here only
|
||||
> to describe what the software talks to. Use a test account first, and read
|
||||
> [`docs/RECOVERY.md`](docs/RECOVERY.md) before trusting it with data you cannot lose.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# Debian / Ubuntu
|
||||
sudo apt install ./proton-drive-linux_*.deb
|
||||
|
||||
# Fedora
|
||||
sudo dnf install ./proton-drive-linux-*.rpm
|
||||
|
||||
# Arch Linux
|
||||
cd packaging && makepkg -fi
|
||||
```
|
||||
|
||||
Packages for each release are attached to the [latest
|
||||
release](https://github.com/narrrl/proton-drive-linux/releases/latest); see
|
||||
[Installation & Packages](#installation--packages) for details and
|
||||
[Building from Source](#building-from-source) to compile it yourself.
|
||||
|
||||
```bash
|
||||
pdfs login # sign in (SRP + 2FA, credentials go to the system keyring)
|
||||
pdfs status # mount state, queued writes, cache usage
|
||||
pdfs-app # the desktop app
|
||||
```
|
||||
|
||||
## Contents
|
||||
|
||||
- [Features](#features) · [Quick search](#quick-search-and-media-opening) · [Opening files](#choosing-how-files-open)
|
||||
- [Selective sync](#selective-sync-pdfsignore) · [Diagnostics](#diagnostics--maintenance) · [Scripting](#scripting---json)
|
||||
- [Performance](#performance--caching) · [Filesystem safety](#filesystem-safety) · [Screenshots](#screenshots)
|
||||
- [Prerequisites](#prerequisites) · [Building](#building-from-source) · [Packages](#installation--packages) · [Releases](#automated-releases-cicd)
|
||||
|
||||
**Documentation:** [Architecture](docs/ARCHITECTURE.md) · [Testing](docs/TESTING.md) ·
|
||||
[Recovery](docs/RECOVERY.md) · [Development](docs/DEVELOPMENT.md) ·
|
||||
[Known issues](docs/BUGS.md) · [Changelog](docs/CHANGELOG.md) ·
|
||||
[Contributing](CONTRIBUTING.md) · [Security policy](SECURITY.md)
|
||||
|
||||
## Features
|
||||
|
||||
- **Files-on-Demand (FUSE)**: Mount your Proton Drive as a virtual filesystem where files are downloaded only when opened, utilizing smart block-level caching and disk-backed writes.
|
||||
- **Command-Line Interface (CLI)**: Manage your drive, authenticate, and monitor sync status directly from the terminal.
|
||||
- **Non-Blocking GTK4 Desktop App**: Browse files, manage pins, configure options, and monitor active transfers through a modern, native GUI with a fully non-blocking asynchronous main loop.
|
||||
- **File Thumbnails**: Browse locally cached image previews in Files, Shared, and Trash, including embedded previews from camera RAW files. Build a whole folder tree from the Files toolbar and resize the grid from the status bar.
|
||||
- **Shared Links & Invites**: Browse files shared with you by other users, and view/manage your own public shared links directly in the GUI.
|
||||
- **Local Backup (Computers)**: Sync and back up local directories (like Downloads, Documents, Pictures, etc.) directly to your Proton Drive account.
|
||||
- **Locations**: One page (and `pdfs locations`) listing every place Proton Drive occupies on this machine — the main mount plus each backed-up folder — with its mode, sync state, and whether it is read-only. Switching a folder between a full local copy and on-demand happens here.
|
||||
- **System Tray Integration**: Background indicator for status monitoring, quick actions, and fast sync controls.
|
||||
- **Unified Search Launcher (HUD)**: A resident Google Drive-style launcher (`pdfs-prompt`) that searches Proton Drive and local files together, ranks the best matches, tolerates abbreviations and typos, and is ideal for a system-wide hotkey.
|
||||
- **Unified Search Launcher (HUD)**: A resident Google Drive-style launcher (`pdfs-prompt`) that searches Proton Drive and local files together, ranks the best matches, tolerates abbreviations and typos, and is ideal for a system-wide hotkey. It can also present the same search in your own launcher (`--dmenu`) or, searching live as you type, in `fzf` (`--fzf`).
|
||||
- **Secure Credential Storage**: Integrates with the system Secret Service (GNOME Keyring, KWallet, etc.) with smart in-memory credential caching to avoid UI thread blockages.
|
||||
- **Proton Photos Support**: Access your Proton Photos timeline, view thumbnails, and download backed-up media natively (available in the GUI as a navigation tab and via the CLI).
|
||||
- **File Version History**: Every revision Proton Drive still holds for a file, from the browser's details pane (**Versions**) or `pdfs versions list|restore|save|rm`. Restoring is server-side — no re-upload — and an old version can be written out to a local file without touching the live one.
|
||||
@@ -66,6 +124,51 @@ launcher command, set them in `config.json`:
|
||||
anywhere in `menu` is replaced by the prompt text; without one, the launcher's
|
||||
own prompt flag is appended.
|
||||
|
||||
### Live launcher search (fzf)
|
||||
|
||||
The two-step above exists because no dmenu-style launcher can call back for new
|
||||
results per keystroke. `fzf` can, so `pdfs-prompt --fzf` searches as you type,
|
||||
exactly like the built-in window:
|
||||
|
||||
```bash
|
||||
pdfs-prompt --fzf # pinned files, then live results as you type
|
||||
pdfs-prompt --fzf --query invoice # start on a search
|
||||
```
|
||||
|
||||
Every keystroke re-queries the daemon (debounced, like the GTK prompt) and fzf's
|
||||
own matcher is switched off, so the ordering you see is the daemon's ranking —
|
||||
abbreviations and typos included, which a launcher filtering a fixed list cannot
|
||||
do. Enter opens; Escape closes.
|
||||
|
||||
fzf is a terminal program, so when it is started without one — from a WM
|
||||
keybinding — it spawns a terminal for itself: foot, ghostty, kitty, alacritty,
|
||||
wezterm or xterm, whichever is installed. Each is launched with the app-id/class
|
||||
`pdfs-prompt`, so one window rule floats it whichever terminal you have. In
|
||||
Hyprland:
|
||||
|
||||
```
|
||||
windowrulev2 = float, class:^(pdfs-prompt)$
|
||||
windowrulev2 = size 900 500, class:^(pdfs-prompt)$
|
||||
windowrulev2 = center, class:^(pdfs-prompt)$
|
||||
bind = SUPER, space, exec, pdfs-prompt --fzf
|
||||
```
|
||||
|
||||
Set it as the default front end, and pick the terminal explicitly, in
|
||||
`config.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"prompt": {
|
||||
"mode": "fzf",
|
||||
"terminal": ["foot", "--app-id=pdfs-prompt", "--"],
|
||||
"menu_limit": 50
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A `{cmd}` token anywhere in `terminal` is replaced by the command to run;
|
||||
without one it is appended, which is what every detected terminal expects.
|
||||
|
||||
## Choosing How Files Open
|
||||
|
||||
By default every result is handed to `xdg-open`. An `open_with` block in
|
||||
@@ -270,19 +373,21 @@ sudo apt-get install -y \
|
||||
libadwaita-1-dev \
|
||||
libwebkitgtk-6.0-dev \
|
||||
libsecret-1-dev \
|
||||
libdbus-1-dev
|
||||
libdbus-1-dev \
|
||||
libimage-exiftool-perl
|
||||
```
|
||||
|
||||
### Arch Linux
|
||||
```bash
|
||||
sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus webkitgtk-6.0
|
||||
sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus webkitgtk-6.0 perl-image-exiftool
|
||||
```
|
||||
|
||||
### Fedora (44+)
|
||||
```bash
|
||||
sudo dnf install -y \
|
||||
pkgconf-pkg-config fuse3-devel gtk4-devel libadwaita-devel \
|
||||
webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel cargo rust
|
||||
webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel \
|
||||
perl-Image-ExifTool cargo rust
|
||||
```
|
||||
|
||||
Runtime extras (pick your desktop):
|
||||
@@ -302,7 +407,7 @@ Ensure you have Rust and Cargo installed (minimum supported Rust version is 1.96
|
||||
|
||||
1. Clone the repository and navigate into the project directory:
|
||||
```bash
|
||||
git clone https://github.com/narl/proton-drive-linux.git
|
||||
git clone https://github.com/narrrl/proton-drive-linux.git
|
||||
cd proton-drive-linux
|
||||
```
|
||||
2. Build the workspace in release mode:
|
||||
@@ -372,6 +477,15 @@ This project has a GitHub Actions CI workflow configured under `.github/workflow
|
||||
|
||||
---
|
||||
|
||||
## Contributing
|
||||
|
||||
Bug reports, packaging fixes, and patches are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for
|
||||
the workflow and the quality gates CI enforces, and [SECURITY.md](SECURITY.md) for reporting a
|
||||
vulnerability privately.
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under the [MIT License](LICENSE).
|
||||
Released under the [MIT License](LICENSE).
|
||||
|
||||
Proton Drive is a service of Proton AG. This project is an independent client and carries no
|
||||
affiliation with or endorsement from Proton AG.
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
# Security Policy
|
||||
|
||||
## Supported versions
|
||||
|
||||
Fixes land on the latest release. Please reproduce on the most recent version before reporting.
|
||||
|
||||
## Reporting a vulnerability
|
||||
|
||||
**Do not open a public issue.** Report privately through GitHub's
|
||||
[private vulnerability reporting](https://github.com/narrrl/proton-drive-linux/security/advisories/new),
|
||||
or by email to <contact@narl.io>.
|
||||
|
||||
Please include the version (`pdfs --version`), your distribution and desktop environment, what an
|
||||
attacker gains, and a reproduction if you have one. You can expect an acknowledgement within a few
|
||||
days; a fix ships in the next release, and you are credited in the advisory unless you ask
|
||||
otherwise.
|
||||
|
||||
## Scope
|
||||
|
||||
This client holds Proton account credentials and decrypted file content on the local machine, so
|
||||
the interesting boundaries are:
|
||||
|
||||
- **Credentials.** Sessions and the mailbox password live in the system Secret Service. Anything
|
||||
that writes them elsewhere — logs, temp files, crash dumps — is a vulnerability.
|
||||
- **The control socket.** Anything that can connect to `control.sock` has the daemon's
|
||||
authenticated session. It is `0600`, and the state, cache, and config directories must be real,
|
||||
current-user-owned `0700` directories; the daemon fails closed otherwise. Reports that defeat
|
||||
those checks are in scope.
|
||||
- **Local cache and staging.** Decrypted block content and staged writes on disk.
|
||||
- **Cryptography.** Key derivation, signature handling, and anything that could publish plaintext
|
||||
or the wrong key material to Proton.
|
||||
|
||||
Out of scope: vulnerabilities in Proton's own service (report those to Proton AG), and issues that
|
||||
require an attacker who already has your user account on the machine.
|
||||
@@ -5,6 +5,8 @@ edition.workspace = true
|
||||
license.workspace = true
|
||||
authors.workspace = true
|
||||
rust-version.workspace = true
|
||||
repository.workspace = true
|
||||
homepage.workspace = true
|
||||
description = "CLI for the Proton Drive Linux client"
|
||||
|
||||
[[bin]]
|
||||
|
||||
+557
-12
@@ -17,6 +17,7 @@ use pdfs_core::control::{
|
||||
RestoreItem as CtlRestoreItem, ShareEntryKind, SyncPhase, pending_summary,
|
||||
};
|
||||
use pdfs_core::db::Db;
|
||||
use pdfs_core::service;
|
||||
|
||||
#[derive(Parser)]
|
||||
#[command(
|
||||
@@ -158,6 +159,65 @@ enum Command {
|
||||
},
|
||||
/// List the photo albums (including ones shared with this account).
|
||||
Albums,
|
||||
/// Import a Google Photos Takeout export into Proton Photos.
|
||||
///
|
||||
/// Pass every zip of the export at once — Google splits one export across
|
||||
/// numbered parts, and a photo's metadata often sits in a different part
|
||||
/// than the photo. Albums in the export are recreated as Proton albums, and
|
||||
/// photos already on the account (same name and same content) are skipped,
|
||||
/// so an interrupted import can simply be run again.
|
||||
ImportGooglePhotos {
|
||||
/// The export's `.zip` files (e.g. `takeout-*.zip`).
|
||||
#[arg(required = true)]
|
||||
archives: Vec<PathBuf>,
|
||||
/// Report what would be imported without uploading anything.
|
||||
#[arg(long)]
|
||||
dry_run: bool,
|
||||
/// Stay attached and print progress until the import finishes.
|
||||
#[arg(long)]
|
||||
wait: bool,
|
||||
},
|
||||
/// Fix photos filed under the day they were imported instead of the day
|
||||
/// they were taken.
|
||||
///
|
||||
/// A Takeout whose metadata sidecars did not match imports with no capture
|
||||
/// time, and Proton then stamps every photo with the moment it arrived. The
|
||||
/// date is still in the file name (`IMG-20230219-WA0001.jpg`), so this finds
|
||||
/// photos whose name disagrees with their capture time by more than a day
|
||||
/// and rewrites them at the right time.
|
||||
///
|
||||
/// Proton cannot edit a sealed capture time, so a rewrite means re-uploading
|
||||
/// the photo and trashing the original — favourites and albums are carried
|
||||
/// across, and the original goes to Proton's trash, not away. Run it with
|
||||
/// `--dry-run` first.
|
||||
RedatePhotos {
|
||||
/// Report what would be re-dated without uploading or trashing anything.
|
||||
#[arg(long)]
|
||||
dry_run: bool,
|
||||
/// Stop after this many photos.
|
||||
#[arg(long)]
|
||||
limit: Option<usize>,
|
||||
/// Only touch photos whose current capture time is on or after this
|
||||
/// date (`YYYY-MM-DD`, UTC) — aim the repair at the days an import
|
||||
/// landed on.
|
||||
#[arg(long)]
|
||||
from: Option<String>,
|
||||
/// Only touch photos whose current capture time is before this date
|
||||
/// (`YYYY-MM-DD`, UTC).
|
||||
#[arg(long)]
|
||||
to: Option<String>,
|
||||
/// Stay attached and print the report when the run finishes.
|
||||
#[arg(long)]
|
||||
wait: bool,
|
||||
},
|
||||
/// Show how the running (or last) photo re-date is doing.
|
||||
RedateStatus,
|
||||
/// Stop the running photo re-date.
|
||||
CancelRedate,
|
||||
/// Show how the running (or last) Google Photos import is doing.
|
||||
ImportStatus,
|
||||
/// Stop the running Google Photos import.
|
||||
CancelImport,
|
||||
/// List the photos in one album (newest capture first).
|
||||
Album {
|
||||
/// Album node uid in `volume~link` form (from `pdfs albums`).
|
||||
@@ -589,6 +649,22 @@ fn main() -> Result<()> {
|
||||
} => cmd_photos(limit, offset, favorites),
|
||||
Command::Favorite { uid, remove } => cmd_favorite(uid, remove),
|
||||
Command::Albums => cmd_albums(),
|
||||
Command::ImportGooglePhotos {
|
||||
archives,
|
||||
dry_run,
|
||||
wait,
|
||||
} => cmd_import_google_photos(archives, dry_run, wait),
|
||||
Command::RedatePhotos {
|
||||
dry_run,
|
||||
limit,
|
||||
from,
|
||||
to,
|
||||
wait,
|
||||
} => cmd_redate_photos(dry_run, limit, from, to, wait),
|
||||
Command::RedateStatus => cmd_redate_status(),
|
||||
Command::CancelRedate => cmd_cancel_redate(),
|
||||
Command::ImportStatus => cmd_import_status(),
|
||||
Command::CancelImport => cmd_cancel_import(),
|
||||
Command::Album { uid, limit, offset } => cmd_album(uid, limit, offset),
|
||||
Command::OpenPhoto { uid } => cmd_open_photo(uid),
|
||||
Command::Search { query, limit } => cmd_search(query, limit),
|
||||
@@ -1053,6 +1129,9 @@ fn cmd_shared_with_me(uid: Option<String>) -> Result<()> {
|
||||
format!(" ({})", e.role)
|
||||
};
|
||||
println!("{kind} {:>12} {}{role} [{}]", e.size, e.name, e.uid);
|
||||
if let Some(line) = shared_by_line(&e) {
|
||||
println!("{:>15}{line}", "");
|
||||
}
|
||||
}
|
||||
}
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
@@ -1061,6 +1140,39 @@ fn cmd_shared_with_me(uid: Option<String>) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Who shared an entry with me, and when, as a line printed under it — `None`
|
||||
/// when the invitation said nothing (owned content, or a node inside a shared
|
||||
/// folder rather than its root). An inviter whose signature did not verify is
|
||||
/// said to be claimed, not stated: the name may be forged.
|
||||
fn shared_by_line(entry: &pdfs_core::control::DirEntry) -> Option<String> {
|
||||
if entry.shared_by.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let mut line = format!("shared by {}", entry.shared_by);
|
||||
if entry.shared_at > 0 {
|
||||
line.push_str(&format!(" at {}", format_epoch_utc(entry.shared_at)));
|
||||
}
|
||||
if entry.shared_by_unverified {
|
||||
line.push_str(" WARNING: invitation signature did not verify; the sender may be forged");
|
||||
}
|
||||
Some(line)
|
||||
}
|
||||
|
||||
/// `YYYY-MM-DD` (UTC) for epoch seconds, without pulling a date crate into the
|
||||
/// CLI for one column. Civil-from-days per Howard Hinnant's algorithm.
|
||||
fn format_epoch_utc(secs: i64) -> String {
|
||||
let z = secs.div_euclid(86_400) + 719_468;
|
||||
let era = z.div_euclid(146_097);
|
||||
let doe = z - era * 146_097;
|
||||
let yoe = (doe - doe / 1_460 + doe / 36_524 - doe / 146_096) / 365;
|
||||
let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
|
||||
let mp = (5 * doy + 2) / 153;
|
||||
let day = doy - (153 * mp + 2) / 5 + 1;
|
||||
let month = if mp < 10 { mp + 3 } else { mp - 9 };
|
||||
let year = yoe + era * 400 + i64::from(month <= 2);
|
||||
format!("{year:04}-{month:02}-{day:02}")
|
||||
}
|
||||
|
||||
/// Download a file shared with me. The daemon puts the plaintext in its content
|
||||
/// cache and replies with that path; with a `dest` we copy it out, so the caller
|
||||
/// gets a file it owns rather than one the cache may later evict.
|
||||
@@ -1280,6 +1392,15 @@ fn cmd_login(username: Option<String>) -> Result<()> {
|
||||
other => anyhow::Error::new(other).context("login failed"),
|
||||
})?;
|
||||
|
||||
// A daemon that is already up is parked in `wait_for_session`, sleeping
|
||||
// between keyring checks; restart it so the mount appears now instead of
|
||||
// whenever the next poll fires. Only when it is already active — `pdfs
|
||||
// login` deliberately does not enable the unit the way the GUI does, so a
|
||||
// headless user driving `pdfs daemon` by hand keeps that setup.
|
||||
if service::is_active() {
|
||||
service::restart();
|
||||
}
|
||||
|
||||
println!("Logged in as {username}. Session stored in the system keyring.");
|
||||
Ok(())
|
||||
}
|
||||
@@ -1336,6 +1457,11 @@ fn cmd_status() -> Result<()> {
|
||||
online,
|
||||
pending_uploads,
|
||||
pending_changes,
|
||||
parked_uploads,
|
||||
failing_ops,
|
||||
failing_error,
|
||||
staged_bytes,
|
||||
staged_oldest_secs,
|
||||
..
|
||||
}) => {
|
||||
let state = if online { "" } else { ", offline" };
|
||||
@@ -1344,6 +1470,25 @@ fn cmd_status() -> Result<()> {
|
||||
None => String::new(),
|
||||
};
|
||||
println!("Mounted at {mountpoint} ({pinned} pinned{state}{queued})");
|
||||
// Everything below is a queue that is not making progress on its
|
||||
// own. Printed only when it exists, because in the ordinary case
|
||||
// there is nothing to say.
|
||||
if staged_bytes > 0 {
|
||||
println!(
|
||||
"Staged {} not yet uploaded (oldest {})",
|
||||
human_bytes(staged_bytes),
|
||||
human_age(staged_oldest_secs),
|
||||
);
|
||||
}
|
||||
if parked_uploads > 0 {
|
||||
println!("Parked {parked_uploads} upload(s) waiting for a final filename");
|
||||
}
|
||||
if failing_ops > 0 {
|
||||
println!(
|
||||
"Stuck {failing_ops} queued operation(s) keep failing: {}",
|
||||
failing_error.as_deref().unwrap_or("no error recorded"),
|
||||
);
|
||||
}
|
||||
}
|
||||
Ok(other) => println!("Mount: unexpected response {other:?}"),
|
||||
Err(_) => println!("Mount: not running."),
|
||||
@@ -1395,6 +1540,59 @@ fn cmd_mount(mountpoint: Option<PathBuf>) -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// How long to wait before the next login check, after `attempts` checks have
|
||||
/// already come back [`pdfs_core::Error::NotLoggedIn`].
|
||||
///
|
||||
/// Doubles from 3s to a 15s ceiling. The interval is not just a sleep:
|
||||
/// [`auth::load`] builds a fresh `keyring::Entry` on every call, so each check
|
||||
/// opens a new D-Bus connection to the Secret Service and negotiates an
|
||||
/// encrypted session over it. A flat 3s poll meant 28,800 of those a day from a
|
||||
/// daemon that had nothing to do; the ceiling cuts that by 80%.
|
||||
///
|
||||
/// The ceiling stays low because the sleep is also the worst-case login →
|
||||
/// mount latency for anything that does *not* poke systemd: front-ends restart
|
||||
/// the unit on login (`service::enable_start`), which interrupts the sleep
|
||||
/// immediately, but a hand-run `pdfs daemon` has no such wake-up path.
|
||||
fn login_poll_delay(attempts: u32) -> std::time::Duration {
|
||||
const BASE_SECS: u64 = 3;
|
||||
const MAX_SECS: u64 = 15;
|
||||
|
||||
// `checked_shl` rather than `<<`: the daemon can sit unauthenticated for
|
||||
// weeks, and a shift past the width of the type is a panic, not a big number.
|
||||
let factor = 1u64.checked_shl(attempts).unwrap_or(u64::MAX);
|
||||
std::time::Duration::from_secs(BASE_SECS.saturating_mul(factor).min(MAX_SECS))
|
||||
}
|
||||
|
||||
/// Block until a session is stored, backing off between checks.
|
||||
///
|
||||
/// `load` and `sleep` are parameters rather than direct calls so the backoff
|
||||
/// schedule can be tested without a keyring and without real time passing.
|
||||
fn wait_for_session<L, S>(mut load: L, mut sleep: S) -> Result<()>
|
||||
where
|
||||
L: FnMut() -> std::result::Result<(), pdfs_core::Error>,
|
||||
S: FnMut(std::time::Duration),
|
||||
{
|
||||
let mut attempts: u32 = 0;
|
||||
loop {
|
||||
match load() {
|
||||
Ok(()) => return Ok(()),
|
||||
Err(pdfs_core::Error::NotLoggedIn) => {
|
||||
let delay = login_poll_delay(attempts);
|
||||
// Say it once at INFO. Repeating it every interval buried the
|
||||
// journal in tens of thousands of identical lines a day.
|
||||
if attempts == 0 {
|
||||
tracing::info!("not logged in; waiting for a session…");
|
||||
} else {
|
||||
tracing::debug!(delay_secs = delay.as_secs(), "still not logged in");
|
||||
}
|
||||
sleep(delay);
|
||||
attempts = attempts.saturating_add(1);
|
||||
}
|
||||
Err(e) => return Err(e.into()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Run the auto-mount daemon loop. Waits for a stored login, mounts, and keeps
|
||||
/// the mount alive: a clean stop (SIGTERM via `systemctl --user stop`) exits 0;
|
||||
/// an external unmount triggers a remount; errors back off and retry. This is
|
||||
@@ -1403,16 +1601,7 @@ fn cmd_daemon(mountpoint: Option<PathBuf>) -> Result<()> {
|
||||
loop {
|
||||
// Wait until a session is stored. The GUI enables this service on login,
|
||||
// but the service may also start at boot before the user has logged in.
|
||||
loop {
|
||||
match auth::load() {
|
||||
Ok(_) => break,
|
||||
Err(pdfs_core::Error::NotLoggedIn) => {
|
||||
tracing::info!("not logged in; waiting…");
|
||||
std::thread::sleep(std::time::Duration::from_secs(3));
|
||||
}
|
||||
Err(e) => return Err(e.into()),
|
||||
}
|
||||
}
|
||||
wait_for_session(|| auth::load().map(|_| ()), std::thread::sleep)?;
|
||||
|
||||
match mount_once(mountpoint.clone()) {
|
||||
Ok(pdfs_fuse::MountOutcome::Shutdown) => {
|
||||
@@ -1687,6 +1876,213 @@ fn cmd_albums() -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn cmd_import_google_photos(archives: Vec<PathBuf>, dry_run: bool, wait: bool) -> Result<()> {
|
||||
// The daemon opens the archives itself off the shared filesystem, so it gets
|
||||
// absolute paths — the same arrangement `pdfs upload` uses.
|
||||
let mut absolute = Vec::with_capacity(archives.len());
|
||||
for archive in &archives {
|
||||
let path = std::fs::canonicalize(archive)
|
||||
.with_context(|| format!("resolve {}", archive.display()))?;
|
||||
absolute.push(
|
||||
path.to_str()
|
||||
.ok_or_else(|| anyhow!("path is not valid UTF-8: {}", path.display()))?
|
||||
.to_owned(),
|
||||
);
|
||||
}
|
||||
|
||||
match control_request(CtlRequest::ImportTakeout {
|
||||
archives: absolute,
|
||||
dry_run,
|
||||
})? {
|
||||
CtlResponse::Ok { message } => println!("{message}"),
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
if !wait {
|
||||
println!("Run `pdfs import-status` to follow it.");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Poll rather than stream: the control protocol is request/response, and an
|
||||
// import runs for hours — a held-open socket would be the only long-lived
|
||||
// connection the daemon has.
|
||||
loop {
|
||||
std::thread::sleep(std::time::Duration::from_secs(2));
|
||||
match control_request(CtlRequest::ImportStatus)? {
|
||||
CtlResponse::ImportStatus {
|
||||
running: true,
|
||||
summary: _,
|
||||
} => continue,
|
||||
CtlResponse::ImportStatus { summary, .. } => {
|
||||
print_import_summary(summary.as_ref());
|
||||
return Ok(());
|
||||
}
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn cmd_redate_photos(
|
||||
dry_run: bool,
|
||||
limit: Option<usize>,
|
||||
from: Option<String>,
|
||||
to: Option<String>,
|
||||
wait: bool,
|
||||
) -> Result<()> {
|
||||
let range = match (from.as_deref(), to.as_deref()) {
|
||||
(None, None) => None,
|
||||
(from, to) => Some((
|
||||
from.map(parse_day).transpose()?.unwrap_or(i64::MIN),
|
||||
to.map(parse_day).transpose()?.unwrap_or(i64::MAX),
|
||||
)),
|
||||
};
|
||||
match control_request(CtlRequest::RedatePhotos {
|
||||
dry_run,
|
||||
limit,
|
||||
range,
|
||||
})? {
|
||||
CtlResponse::Ok { message } => println!("{message}"),
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
if !wait {
|
||||
println!("Run `pdfs redate-status` to follow it.");
|
||||
return Ok(());
|
||||
}
|
||||
// Polled for the same reason the import is: request/response protocol, and a
|
||||
// run over a whole library is a download plus an upload per photo.
|
||||
loop {
|
||||
std::thread::sleep(std::time::Duration::from_secs(2));
|
||||
match control_request(CtlRequest::RedateStatus)? {
|
||||
CtlResponse::RedateStatus { running: true, .. } => continue,
|
||||
CtlResponse::RedateStatus { summary, .. } => {
|
||||
print_redate_summary(summary.as_ref());
|
||||
return Ok(());
|
||||
}
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A `YYYY-MM-DD` day as epoch seconds at UTC midnight. Reuses the Takeout
|
||||
/// scanner's date reader, which already accepts exactly this spelling.
|
||||
fn parse_day(day: &str) -> Result<i64> {
|
||||
pdfs_core::takeout::capture_time_from_name(day)
|
||||
.ok_or_else(|| anyhow!("not a date (expected YYYY-MM-DD): {day}"))
|
||||
}
|
||||
|
||||
fn cmd_redate_status() -> Result<()> {
|
||||
match control_request(CtlRequest::RedateStatus)? {
|
||||
CtlResponse::RedateStatus { running, summary } => {
|
||||
if running {
|
||||
println!("Re-date running. `pdfs transfers` shows its progress.");
|
||||
}
|
||||
print_redate_summary(summary.as_ref());
|
||||
}
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn cmd_cancel_redate() -> Result<()> {
|
||||
match control_request(CtlRequest::CancelRedate)? {
|
||||
CtlResponse::Ok { message } => println!("{message}"),
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn print_redate_summary(summary: Option<&pdfs_core::control::RedateSummary>) {
|
||||
let Some(summary) = summary else {
|
||||
println!("No re-date has run yet.");
|
||||
return;
|
||||
};
|
||||
println!("Photos examined : {}", summary.examined);
|
||||
println!("Mis-dated : {}", summary.candidates);
|
||||
if summary.dry_run {
|
||||
println!("(dry run — nothing was uploaded or trashed)");
|
||||
} else {
|
||||
println!("Re-dated : {}", summary.redated);
|
||||
if summary.album_links > 0 {
|
||||
println!("Albums refiled : {}", summary.album_links);
|
||||
}
|
||||
if summary.failed > 0 {
|
||||
println!("Failed : {}", summary.failed);
|
||||
}
|
||||
if summary.bytes > 0 {
|
||||
println!("Uploaded bytes : {}", summary.bytes);
|
||||
}
|
||||
}
|
||||
for (name, stored, wanted) in &summary.samples {
|
||||
// Epoch seconds, as `pdfs photos` prints capture times.
|
||||
println!(" {name}: {stored} -> {wanted}");
|
||||
}
|
||||
if summary.candidates > summary.samples.len() {
|
||||
println!(
|
||||
" … and {} more",
|
||||
summary.candidates - summary.samples.len()
|
||||
);
|
||||
}
|
||||
if summary.cancelled {
|
||||
println!("Run was cancelled before it finished.");
|
||||
}
|
||||
}
|
||||
|
||||
fn cmd_import_status() -> Result<()> {
|
||||
match control_request(CtlRequest::ImportStatus)? {
|
||||
CtlResponse::ImportStatus { running, summary } => {
|
||||
if running {
|
||||
println!("Import running. `pdfs transfers` shows its progress.");
|
||||
}
|
||||
print_import_summary(summary.as_ref());
|
||||
}
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn cmd_cancel_import() -> Result<()> {
|
||||
match control_request(CtlRequest::CancelImport)? {
|
||||
CtlResponse::Ok { message } => println!("{message}"),
|
||||
CtlResponse::Error { message, kind } => bail!("{}", cli_error(kind, &message)),
|
||||
other => bail!("unexpected response: {other:?}"),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn print_import_summary(summary: Option<&pdfs_core::control::ImportSummary>) {
|
||||
let Some(summary) = summary else {
|
||||
println!("No import has run yet.");
|
||||
return;
|
||||
};
|
||||
println!("Photos in export : {}", summary.found);
|
||||
println!("Uploaded : {}", summary.uploaded);
|
||||
println!("Already on Proton: {}", summary.duplicates);
|
||||
if summary.skipped_trashed > 0 {
|
||||
println!("Skipped (trash) : {}", summary.skipped_trashed);
|
||||
}
|
||||
if summary.albums_created > 0 || summary.album_links > 0 {
|
||||
println!(
|
||||
"Albums : {} created, {} photos filed",
|
||||
summary.albums_created, summary.album_links
|
||||
);
|
||||
}
|
||||
if summary.failed > 0 {
|
||||
println!("Failed : {}", summary.failed);
|
||||
}
|
||||
if summary.bytes > 0 {
|
||||
println!("Uploaded bytes : {}", summary.bytes);
|
||||
}
|
||||
if summary.cancelled {
|
||||
println!("Import was cancelled before it finished.");
|
||||
}
|
||||
}
|
||||
|
||||
fn cmd_album(uid: String, limit: usize, offset: usize) -> Result<()> {
|
||||
match control_request(CtlRequest::AlbumPhotos { uid, offset, limit })? {
|
||||
CtlResponse::Photos { items, .. } if items.is_empty() => println!("No photos."),
|
||||
@@ -1886,6 +2282,18 @@ fn path_arg(path: &Path) -> Result<String> {
|
||||
.ok_or_else(|| anyhow!("path is not valid UTF-8"))
|
||||
}
|
||||
|
||||
/// Format an age in seconds for a report line, coarsely — the point of the
|
||||
/// number is whether it is minutes or weeks.
|
||||
fn human_age(secs: u64) -> String {
|
||||
const UNITS: [(&str, u64); 4] = [("d", 86_400), ("h", 3_600), ("m", 60), ("s", 1)];
|
||||
for (unit, scale) in UNITS {
|
||||
if secs >= scale {
|
||||
return format!("{}{unit}", secs / scale);
|
||||
}
|
||||
}
|
||||
"just now".to_string()
|
||||
}
|
||||
|
||||
/// Format a byte count for a report line.
|
||||
fn human_bytes(bytes: u64) -> String {
|
||||
const UNITS: [(&str, f64); 4] = [
|
||||
@@ -1925,6 +2333,32 @@ fn cmd_cache(action: CacheCmd) -> Result<()> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Ask for a deep integrity check and wait for it to finish.
|
||||
///
|
||||
/// The daemon answers the request immediately and runs the check on a thread —
|
||||
/// it holds the single database connection for as long as it takes, which would
|
||||
/// otherwise stall the mount — so a caller that needs the verdict polls for it.
|
||||
/// `pdfs doctor` is that caller: it is a diagnostic the user is watching, so
|
||||
/// waiting is right, and the bound is generous enough for a large database
|
||||
/// without hanging forever if the daemon never finishes.
|
||||
fn await_integrity_check() -> Result<CtlResponse> {
|
||||
const POLL: std::time::Duration = std::time::Duration::from_secs(1);
|
||||
const GIVE_UP_AFTER: std::time::Duration = std::time::Duration::from_secs(120);
|
||||
|
||||
let started = std::time::Instant::now();
|
||||
loop {
|
||||
let response = control_request(CtlRequest::CacheInspect { deep: true })?;
|
||||
match &response {
|
||||
CtlResponse::CacheReport {
|
||||
integrity_running: true,
|
||||
integrity_checked: false,
|
||||
..
|
||||
} if started.elapsed() < GIVE_UP_AFTER => std::thread::sleep(POLL),
|
||||
_ => return Ok(response),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn cmd_cache_inspect(deep: bool) -> Result<()> {
|
||||
let response = control_request(CtlRequest::CacheInspect { deep })?;
|
||||
if emit_json(&response)? {
|
||||
@@ -1940,6 +2374,7 @@ fn cmd_cache_inspect(deep: bool) -> Result<()> {
|
||||
cache_budget,
|
||||
integrity_problems,
|
||||
integrity_checked,
|
||||
integrity_running,
|
||||
} => {
|
||||
println!(
|
||||
"Database schema v{schema_version}, {}",
|
||||
@@ -1972,6 +2407,11 @@ fn cmd_cache_inspect(deep: bool) -> Result<()> {
|
||||
println!(" {problem}");
|
||||
}
|
||||
}
|
||||
} else if integrity_running {
|
||||
// The daemon runs the check on a thread rather than holding its
|
||||
// only database connection for the length of it, so the answer
|
||||
// belongs to a later request.
|
||||
println!("\nIntegrity checking in the background — run this again for the result");
|
||||
} else {
|
||||
println!("\nIntegrity not checked (pass --deep)");
|
||||
}
|
||||
@@ -2209,14 +2649,24 @@ fn cmd_diagnose() -> Result<()> {
|
||||
format!("{pending} waiting to upload")
|
||||
},
|
||||
);
|
||||
match control_request(CtlRequest::CacheInspect { deep: true }) {
|
||||
match await_integrity_check() {
|
||||
Ok(CtlResponse::CacheReport {
|
||||
integrity_problems, ..
|
||||
integrity_problems,
|
||||
integrity_checked: true,
|
||||
..
|
||||
}) if integrity_problems.is_empty() => report.finding(
|
||||
DiagnoseLevel::Ok,
|
||||
" database integrity",
|
||||
"SQLite integrity_check passed",
|
||||
),
|
||||
Ok(CtlResponse::CacheReport {
|
||||
integrity_checked: false,
|
||||
..
|
||||
}) => report.finding(
|
||||
DiagnoseLevel::Warn,
|
||||
" database integrity",
|
||||
"still checking — run `pdfs cache inspect --deep` for the result",
|
||||
),
|
||||
Ok(CtlResponse::CacheReport {
|
||||
integrity_problems, ..
|
||||
}) => report.finding(
|
||||
@@ -2332,3 +2782,98 @@ mod diagnose_tests {
|
||||
std::fs::remove_dir(dir).unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod login_wait_tests {
|
||||
use super::*;
|
||||
use std::time::Duration;
|
||||
|
||||
#[test]
|
||||
fn poll_delay_doubles_then_holds_at_the_ceiling() {
|
||||
assert_eq!(login_poll_delay(0), Duration::from_secs(3));
|
||||
assert_eq!(login_poll_delay(1), Duration::from_secs(6));
|
||||
assert_eq!(login_poll_delay(2), Duration::from_secs(12));
|
||||
assert_eq!(login_poll_delay(3), Duration::from_secs(15));
|
||||
assert_eq!(login_poll_delay(4), Duration::from_secs(15));
|
||||
// A daemon left unauthenticated for weeks reaches attempt counts well
|
||||
// past the width of the shift; that must cap, not panic.
|
||||
assert_eq!(login_poll_delay(64), Duration::from_secs(15));
|
||||
assert_eq!(login_poll_delay(u32::MAX), Duration::from_secs(15));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn waiting_for_a_session_stops_hammering_the_keyring() {
|
||||
let mut checks_left = 20;
|
||||
let mut slept: Vec<Duration> = Vec::new();
|
||||
|
||||
wait_for_session(
|
||||
|| {
|
||||
if checks_left == 0 {
|
||||
Ok(())
|
||||
} else {
|
||||
checks_left -= 1;
|
||||
Err(pdfs_core::Error::NotLoggedIn)
|
||||
}
|
||||
},
|
||||
|delay| slept.push(delay),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(slept.len(), 20, "one wait per failed check");
|
||||
assert!(
|
||||
slept.windows(2).all(|w| w[1] >= w[0]),
|
||||
"the interval must never shrink: {slept:?}"
|
||||
);
|
||||
|
||||
// The point of the change. Every check is a fresh D-Bus session against
|
||||
// the Secret Service, so 20 of them must not fit inside a single minute
|
||||
// the way a flat 3s poll did.
|
||||
let total: Duration = slept.iter().sum();
|
||||
assert!(
|
||||
total >= Duration::from_secs(4 * 60),
|
||||
"20 checks should span at least 4 minutes, spanned {total:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_stored_session_is_not_waited_for_at_all() {
|
||||
let mut slept = 0;
|
||||
wait_for_session(|| Ok(()), |_| slept += 1).unwrap();
|
||||
assert_eq!(slept, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn epoch_seconds_format_as_utc_calendar_dates() {
|
||||
assert_eq!(format_epoch_utc(0), "1970-01-01");
|
||||
assert_eq!(format_epoch_utc(951_782_400), "2000-02-29");
|
||||
assert_eq!(format_epoch_utc(1_700_000_000), "2023-11-14");
|
||||
assert_eq!(format_epoch_utc(-86_400), "1969-12-31");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_forged_looking_invitation_is_flagged_not_stated() {
|
||||
let mut entry = pdfs_core::control::DirEntry {
|
||||
name: "Budget.ods".into(),
|
||||
is_dir: false,
|
||||
size: 1,
|
||||
modified: 0,
|
||||
pinned: false,
|
||||
cached: false,
|
||||
uid: "vol~link".into(),
|
||||
path: String::new(),
|
||||
role: "viewer".into(),
|
||||
shared_by: String::new(),
|
||||
shared_at: 0,
|
||||
shared_by_unverified: false,
|
||||
};
|
||||
assert_eq!(shared_by_line(&entry), None, "no invitation, no line");
|
||||
entry.shared_by = "alice@proton.me".into();
|
||||
entry.shared_at = 1_700_000_000;
|
||||
assert_eq!(
|
||||
shared_by_line(&entry).as_deref(),
|
||||
Some("shared by alice@proton.me at 2023-11-14")
|
||||
);
|
||||
entry.shared_by_unverified = true;
|
||||
assert!(shared_by_line(&entry).unwrap().contains("WARNING"));
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,8 @@ edition.workspace = true
|
||||
license.workspace = true
|
||||
authors.workspace = true
|
||||
rust-version.workspace = true
|
||||
repository.workspace = true
|
||||
homepage.workspace = true
|
||||
description = "Core auth/session/config for the Proton Drive Linux client"
|
||||
|
||||
[dependencies]
|
||||
@@ -18,8 +20,11 @@ tracing.workspace = true
|
||||
directories.workspace = true
|
||||
ignore.workspace = true
|
||||
sha2.workspace = true
|
||||
sha1.workspace = true
|
||||
zip.workspace = true
|
||||
rusqlite.workspace = true
|
||||
keyring.workspace = true
|
||||
keyring-core.workspace = true
|
||||
dbus-secret-service-keyring-store.workspace = true
|
||||
parking_lot.workspace = true
|
||||
async-trait.workspace = true
|
||||
libc.workspace = true
|
||||
@@ -6,7 +6,7 @@
|
||||
//! interactive step; the refresh token auto-renews via the HTTP client's 401
|
||||
//! path, so no fresh 2FA is required until the refresh token itself expires.
|
||||
|
||||
use keyring::Entry;
|
||||
use keyring_core::Entry;
|
||||
use proton_drive_rs::{KeySalt, ProtonDriveClient};
|
||||
use proton_sdk::api::{HumanVerification, HumanVerificationCredential};
|
||||
use proton_sdk::cache::EncryptedCacheRepository;
|
||||
@@ -81,6 +81,11 @@ fn client_config() -> ProtonClientConfiguration {
|
||||
}
|
||||
|
||||
fn keyring_entry() -> Result<Entry> {
|
||||
// Installed lazily rather than once at startup so a session bus that was not
|
||||
// up yet on the first attempt is retried on the next one.
|
||||
if keyring_core::get_default_store().is_none() {
|
||||
keyring_core::set_default_store(dbus_secret_service_keyring_store::Store::new()?);
|
||||
}
|
||||
Ok(Entry::new(KEYRING_SERVICE, KEYRING_USER)?)
|
||||
}
|
||||
|
||||
@@ -215,7 +220,7 @@ pub fn load() -> Result<StoredSession> {
|
||||
let entry = keyring_entry()?;
|
||||
match entry.get_password() {
|
||||
Ok(json) => Ok(serde_json::from_str(&json)?),
|
||||
Err(keyring::Error::NoEntry) => Err(Error::NotLoggedIn),
|
||||
Err(keyring_core::Error::NoEntry) => Err(Error::NotLoggedIn),
|
||||
Err(e) => Err(e.into()),
|
||||
}
|
||||
}
|
||||
@@ -238,7 +243,7 @@ pub fn logout() -> Result<()> {
|
||||
}
|
||||
}
|
||||
match keyring_entry()?.delete_credential() {
|
||||
Ok(()) | Err(keyring::Error::NoEntry) => Ok(()),
|
||||
Ok(()) | Err(keyring_core::Error::NoEntry) => Ok(()),
|
||||
Err(e) => Err(e.into()),
|
||||
}
|
||||
}
|
||||
|
||||
+1235
-94
File diff suppressed because it is too large.
Load diff
@@ -93,6 +93,32 @@ pub enum Request {
|
||||
/// already cached (one batched round-trip) and replying with their on-disk
|
||||
/// paths. Keep the batch small — it is served on demand, as tiles scroll in.
|
||||
PhotoThumbs { uids: Vec<String> },
|
||||
/// Fetch thumbnails for ordinary Drive image files shown outside the Photos
|
||||
/// timeline. The modification time is the cache validity tag, so replacing
|
||||
/// an image can never reuse its previous revision's thumbnail.
|
||||
FileThumbs {
|
||||
items: Vec<FileThumbRequest>,
|
||||
/// Identifies the front-end's current listing. Moving to another page or
|
||||
/// folder advances it, allowing the daemon to stop work for rows that
|
||||
/// are no longer visible.
|
||||
#[serde(default)]
|
||||
generation: u64,
|
||||
},
|
||||
/// Reserve a daemon-issued generation for ordinary-file thumbnail work.
|
||||
/// The daemon is the ordering authority, so GUI restarts and wall-clock
|
||||
/// corrections cannot make a new listing look older than an earlier one.
|
||||
ReserveFileThumbGeneration,
|
||||
/// Stop ordinary-file thumbnail work belonging to older listings. Photos
|
||||
/// timeline work and an explicit recursive build are deliberately separate.
|
||||
CancelFileThumbs { generation: u64 },
|
||||
/// Start building local thumbnails for every supported image below `path`.
|
||||
/// The request returns immediately; progress is read with
|
||||
/// [`Request::ThumbnailBuildStatus`].
|
||||
StartThumbnailBuild { path: String },
|
||||
/// Stop the current recursive ordinary-file thumbnail build, if any.
|
||||
CancelThumbnailBuild,
|
||||
/// Read progress for the recursive ordinary-file thumbnail build.
|
||||
ThumbnailBuildStatus,
|
||||
/// Download a photo's full content into the cache; replies with its path.
|
||||
OpenPhoto { uid: String },
|
||||
/// Add or remove Proton's `Favorite` tag on a photo. Replies with
|
||||
@@ -103,6 +129,70 @@ pub enum Request {
|
||||
/// re-encrypted for our timeline, which the SDK does not implement yet; the
|
||||
/// daemon reports that as an error rather than silently doing nothing.
|
||||
SetPhotoFavorite { uid: String, favorite: bool },
|
||||
/// Import Google Photos **Takeout** archives into Proton Photos.
|
||||
///
|
||||
/// `archives` are absolute paths to the export's `.zip` parts on the
|
||||
/// daemon's own filesystem — the whole set at once, because a photo and its
|
||||
/// metadata sidecar routinely land in different parts. Photos already on the
|
||||
/// account (matched by name *and* content digest) are skipped, so a library
|
||||
/// partly synced by Proton Photos elsewhere does not import twice, and an
|
||||
/// interrupted import resumes by simply being run again.
|
||||
///
|
||||
/// With `dry_run` the daemon scans and reports what it *would* do without
|
||||
/// uploading anything. Otherwise it acks immediately with
|
||||
/// [`Response::Ok`] and works in the background — an export is hours of
|
||||
/// upload, far past any socket timeout. Progress shows up in
|
||||
/// [`Request::GetQueueStatus`] as a job, and the final counts in
|
||||
/// [`Request::ImportStatus`].
|
||||
ImportTakeout {
|
||||
archives: Vec<String>,
|
||||
#[serde(default)]
|
||||
dry_run: bool,
|
||||
},
|
||||
/// Re-date photos whose capture time is when they were *imported* rather
|
||||
/// than when they were taken.
|
||||
///
|
||||
/// Proton has no API for editing a photo's capture time — it is sealed into
|
||||
/// the revision — so the only repair is to upload the photo again with the
|
||||
/// right time and trash the original. That is what this does, for photos
|
||||
/// whose file name carries a date that disagrees with their stored capture
|
||||
/// time by more than a day: `IMG-20230219-WA0001.jpg` filed under 2026 is
|
||||
/// the signature of an import whose metadata sidecars did not match.
|
||||
///
|
||||
/// Favourites and album memberships are carried across; the original is
|
||||
/// **trashed**, not deleted, so a bad run is recoverable from Proton's
|
||||
/// trash. With `dry_run` nothing is uploaded or trashed and the report says
|
||||
/// what would have been done.
|
||||
///
|
||||
/// Acks immediately with [`Response::Ok`] and works in the background — each
|
||||
/// photo is a download plus an upload. Progress shows up in
|
||||
/// [`Request::GetQueueStatus`] as a job, and the counts in
|
||||
/// [`Request::RedateStatus`].
|
||||
RedatePhotos {
|
||||
#[serde(default)]
|
||||
dry_run: bool,
|
||||
/// Stop after this many photos. `None` is the whole library.
|
||||
#[serde(default)]
|
||||
limit: Option<usize>,
|
||||
/// Only consider photos whose *stored* capture time falls in this
|
||||
/// `[from, to)` window — the way to aim the repair at the days an
|
||||
/// import landed on and leave the rest of the library untouched.
|
||||
#[serde(default)]
|
||||
range: Option<(i64, i64)>,
|
||||
},
|
||||
/// How the running (or last finished) re-date is doing. Replies with
|
||||
/// [`Response::RedateStatus`].
|
||||
RedateStatus,
|
||||
/// Ask the running re-date to stop. It finishes the photo on the wire, then
|
||||
/// reports `cancelled`. Replies with [`Response::Ok`].
|
||||
CancelRedate,
|
||||
/// How the running (or last finished) Takeout import is doing. Replies with
|
||||
/// [`Response::ImportStatus`].
|
||||
ImportStatus,
|
||||
/// Ask the running Takeout import to stop. It finishes the photo on the wire
|
||||
/// and files what it has uploaded into its albums, then reports
|
||||
/// `cancelled`. Replies with [`Response::Ok`].
|
||||
CancelImport,
|
||||
/// Upload the photo at `source_path` under the given name and media type.
|
||||
///
|
||||
/// A path, not the bytes: this protocol is line-delimited JSON, and
|
||||
@@ -652,6 +742,10 @@ pub struct SharedItem {
|
||||
/// The shared node's decrypted name.
|
||||
pub name: String,
|
||||
pub is_dir: bool,
|
||||
/// Modification time used to validate an image thumbnail. Older daemons
|
||||
/// omit it, in which case clients use `0` as a conservative fallback tag.
|
||||
#[serde(default)]
|
||||
pub modified: i64,
|
||||
/// Mountpoint-relative path, when the daemon can resolve it (the node lives in
|
||||
/// my own tree). Empty when the path is unknown.
|
||||
#[serde(default)]
|
||||
@@ -812,6 +906,22 @@ pub struct DirEntry {
|
||||
/// report. Defaulted for wire-compat with clients/daemons predating it.
|
||||
#[serde(default)]
|
||||
pub role: String,
|
||||
/// Who shared this node with me, as the invitation claims it — empty for
|
||||
/// content I own, for a node reached *inside* a shared folder (only the share
|
||||
/// root carries the invitation), and when the API did not say. Trust it only
|
||||
/// as far as [`shared_by_unverified`](Self::shared_by_unverified) allows.
|
||||
/// Defaulted for wire-compat with clients/daemons predating it.
|
||||
#[serde(default)]
|
||||
pub shared_by: String,
|
||||
/// When it was shared with me, epoch seconds; 0 when unknown.
|
||||
#[serde(default)]
|
||||
pub shared_at: i64,
|
||||
/// The invitation's signature did not check out against `shared_by`'s keys,
|
||||
/// so the name may be forged. False when it verified *and* when it could not
|
||||
/// be checked at all (no signature, no keys): only an actual mismatch is a
|
||||
/// warning worth showing.
|
||||
#[serde(default)]
|
||||
pub shared_by_unverified: bool,
|
||||
}
|
||||
|
||||
/// One hit in a [`Request::Search`] result. Like [`DirEntry`] but carries the
|
||||
@@ -1052,6 +1162,64 @@ pub struct PhotoMonth {
|
||||
pub count: usize,
|
||||
}
|
||||
|
||||
/// How a photo re-date is going, or how the last one ended
|
||||
/// ([`Response::RedateStatus`]). Cumulative for the run, so the same shape
|
||||
/// serves progress and the final report.
|
||||
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
|
||||
pub struct RedateSummary {
|
||||
/// Photos examined.
|
||||
pub examined: usize,
|
||||
/// Photos whose name implies a capture time more than a day from the one
|
||||
/// they carry — what the run would (or did) repair.
|
||||
pub candidates: usize,
|
||||
/// Photos re-uploaded with the right capture time, their original trashed.
|
||||
pub redated: usize,
|
||||
/// Photos that could not be downloaded, re-uploaded, or trashed. The
|
||||
/// original is left alone in every failure case.
|
||||
pub failed: usize,
|
||||
/// Album memberships re-created on the new copies.
|
||||
pub album_links: usize,
|
||||
/// Bytes uploaded.
|
||||
pub bytes: u64,
|
||||
/// True when the run stopped early because it was cancelled.
|
||||
pub cancelled: bool,
|
||||
/// True when nothing was actually uploaded or trashed.
|
||||
pub dry_run: bool,
|
||||
/// Up to a handful of examples, as `(name, stored capture time, the time the
|
||||
/// name implies)` — what a dry run is read for.
|
||||
#[serde(default)]
|
||||
pub samples: Vec<(String, i64, i64)>,
|
||||
}
|
||||
|
||||
/// How a Google Photos Takeout import is going, or how the last one ended
|
||||
/// ([`Response::ImportStatus`]).
|
||||
///
|
||||
/// The counts are cumulative for the run, so the same shape serves progress and
|
||||
/// the final report. `found` counts *distinct* photos: Google stores a separate
|
||||
/// copy of a photo in each album folder it appears in, and those fold into one
|
||||
/// upload that joins several albums.
|
||||
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
|
||||
pub struct ImportSummary {
|
||||
/// Distinct photos the export holds.
|
||||
pub found: usize,
|
||||
/// Photos uploaded by this run.
|
||||
pub uploaded: usize,
|
||||
/// Photos already on the account, matched by name *and* content digest.
|
||||
pub duplicates: usize,
|
||||
/// Photos that could not be read or uploaded.
|
||||
pub failed: usize,
|
||||
/// Photos in the export's trash bucket, which are never imported.
|
||||
pub skipped_trashed: usize,
|
||||
/// Albums created by this run.
|
||||
pub albums_created: usize,
|
||||
/// Album memberships added by this run.
|
||||
pub album_links: usize,
|
||||
/// Bytes uploaded.
|
||||
pub bytes: u64,
|
||||
/// True when the run stopped early because it was cancelled.
|
||||
pub cancelled: bool,
|
||||
}
|
||||
|
||||
/// One album in a [`Response::Albums`] listing.
|
||||
///
|
||||
/// An album is a folder node on the photos volume carrying album properties, so
|
||||
@@ -1131,6 +1299,76 @@ pub struct PhotoThumb {
|
||||
pub pending: bool,
|
||||
}
|
||||
|
||||
/// One ordinary Drive file whose thumbnail a front-end wants to paint.
|
||||
#[derive(Serialize, Deserialize, Debug, Clone)]
|
||||
pub struct FileThumbRequest {
|
||||
/// Node uid in `volume~link` form.
|
||||
pub uid: String,
|
||||
/// File modification time, used as the thumbnail cache validity tag.
|
||||
pub modified: i64,
|
||||
/// Original Drive name. Local RAW-preview extraction uses the extension as a
|
||||
/// format hint; defaulted for wire compatibility with older front-ends.
|
||||
#[serde(default)]
|
||||
pub name: String,
|
||||
}
|
||||
|
||||
/// Whether a Drive name is a raster image that the built-in decoder can read,
|
||||
/// or a concrete camera RAW format whose embedded preview exiftool can extract.
|
||||
/// Keep this list shared by the daemon and every GUI surface so a visible tile
|
||||
/// is never advertised without a matching generation path.
|
||||
pub fn is_thumbnail_image_name(name: &str) -> bool {
|
||||
let Some((_, extension)) = name.rsplit_once('.') else {
|
||||
return false;
|
||||
};
|
||||
let extension = extension.to_ascii_lowercase();
|
||||
STANDARD_THUMBNAIL_EXTENSIONS.contains(&extension.as_str())
|
||||
|| RAW_THUMBNAIL_EXTENSIONS.contains(&extension.as_str())
|
||||
}
|
||||
|
||||
/// Whether `name` is one of the concrete RAW formats handled by the exiftool
|
||||
/// embedded-preview path. Deliberately excludes generic/legacy grab-bags whose
|
||||
/// preview availability is not dependable.
|
||||
pub fn is_raw_image_name(name: &str) -> bool {
|
||||
let Some((_, extension)) = name.rsplit_once('.') else {
|
||||
return false;
|
||||
};
|
||||
let extension = extension.to_ascii_lowercase();
|
||||
RAW_THUMBNAIL_EXTENSIONS.contains(&extension.as_str())
|
||||
}
|
||||
|
||||
const STANDARD_THUMBNAIL_EXTENSIONS: &[&str] =
|
||||
&["bmp", "gif", "jpeg", "jpg", "png", "tif", "tiff", "webp"];
|
||||
|
||||
/// Concrete camera RAW formats with embedded previews handled by exiftool.
|
||||
/// This is the single source used by both RAW classification and the broader
|
||||
/// thumbnail capability check.
|
||||
const RAW_THUMBNAIL_EXTENSIONS: &[&str] = &[
|
||||
"arw", "cr2", "cr3", "crw", "dng", "k25", "kdc", "mrw", "nef", "nrw", "orf", "pef", "raf",
|
||||
"raw", "rw2", "sr2", "srf", "x3f",
|
||||
];
|
||||
|
||||
/// Progress of the one recursive ordinary-file thumbnail build the daemon may
|
||||
/// run at a time.
|
||||
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
|
||||
pub struct ThumbnailBuildStatus {
|
||||
pub running: bool,
|
||||
/// True while folders are still being discovered, before the final image
|
||||
/// count is known and the progress bar can become determinate.
|
||||
pub scanning: bool,
|
||||
/// Mountpoint-relative root selected when the build started.
|
||||
pub path: String,
|
||||
pub folders_scanned: u64,
|
||||
pub images_found: u64,
|
||||
/// Images already cached or processed during this build. Includes failures,
|
||||
/// so `completed == images_found` still means the job has finished.
|
||||
pub completed: u64,
|
||||
/// Images that could not be downloaded or decoded.
|
||||
pub failed: u64,
|
||||
/// A traversal-level problem, such as a folder that could not be listed.
|
||||
#[serde(default)]
|
||||
pub message: Option<String>,
|
||||
}
|
||||
|
||||
/// A daemon too old to report connectivity is assumed online — it could not
|
||||
/// have mounted at all otherwise.
|
||||
fn default_online() -> bool {
|
||||
@@ -1193,6 +1431,28 @@ pub enum Response {
|
||||
/// calling a queued `mkdir` an upload is a lie.
|
||||
#[serde(default)]
|
||||
pending_changes: u64,
|
||||
/// Queued writes held back because the file still wears a transient
|
||||
/// name (a browser's `*.crdownload`, an editor's `*.swp`). They are not
|
||||
/// waiting on the network and will not drain until the finalising
|
||||
/// rename, so they are counted apart from `pending_uploads`.
|
||||
#[serde(default)]
|
||||
parked_uploads: u64,
|
||||
/// Queued ops that have failed enough times to be considered stuck.
|
||||
/// They keep retrying forever — correctly, since their staged bytes are
|
||||
/// the only copy — which is why they have to be visible.
|
||||
#[serde(default)]
|
||||
failing_ops: u64,
|
||||
/// The most recent error from a stuck op, when there is one.
|
||||
#[serde(default)]
|
||||
failing_error: Option<String>,
|
||||
/// Bytes held in `staging/`: writes accepted from the kernel whose only
|
||||
/// copy is on local disk. Never evictable, never counted against the
|
||||
/// cache budget — hence reported separately.
|
||||
#[serde(default)]
|
||||
staged_bytes: u64,
|
||||
/// Age of the oldest staged write, in seconds.
|
||||
#[serde(default)]
|
||||
staged_oldest_secs: u64,
|
||||
},
|
||||
/// A human-readable success message.
|
||||
Ok { message: String },
|
||||
@@ -1217,6 +1477,12 @@ pub enum Response {
|
||||
/// Whether the integrity check actually ran, so an empty problem list
|
||||
/// is not mistaken for a clean bill of health it never gave.
|
||||
integrity_checked: bool,
|
||||
/// Whether a deep check is running right now. It reads every page of
|
||||
/// the database under the daemon's only connection, so it is answered
|
||||
/// asynchronously: this says "ask again shortly", which is a different
|
||||
/// thing to say than either of the two above.
|
||||
#[serde(default)]
|
||||
integrity_running: bool,
|
||||
},
|
||||
/// The pin registry.
|
||||
Pins { pins: Vec<Pin> },
|
||||
@@ -1242,11 +1508,36 @@ pub enum Response {
|
||||
available: bool,
|
||||
items: Vec<AlbumInfo>,
|
||||
},
|
||||
/// How the Takeout import is doing (reply to [`Request::ImportStatus`]).
|
||||
/// `running` is false once it has finished, when `summary` is the final
|
||||
/// report; `summary` is `None` when no import has run this session.
|
||||
ImportStatus {
|
||||
running: bool,
|
||||
#[serde(default)]
|
||||
summary: Option<ImportSummary>,
|
||||
},
|
||||
/// How the photo re-date is doing (reply to [`Request::RedateStatus`]).
|
||||
/// `running` is false once it has finished, when `summary` is the final
|
||||
/// report; `summary` is `None` when no re-date has run this session.
|
||||
RedateStatus {
|
||||
running: bool,
|
||||
#[serde(default)]
|
||||
summary: Option<RedateSummary>,
|
||||
},
|
||||
/// The months the timeline spans (reply to [`Request::PhotoMonths`]),
|
||||
/// newest first.
|
||||
PhotoMonths { months: Vec<PhotoMonth> },
|
||||
/// Thumbnails for a [`Request::PhotoThumbs`] batch.
|
||||
Thumbs { items: Vec<PhotoThumb> },
|
||||
/// A [`Request::FileThumbs`] generation is no longer current. This is
|
||||
/// retryable after reserving a fresh generation and must never be interpreted
|
||||
/// as a permanent "no thumbnail" verdict.
|
||||
FileThumbsStale,
|
||||
/// Daemon-issued generation reserved by
|
||||
/// [`Request::ReserveFileThumbGeneration`].
|
||||
FileThumbGeneration { generation: u64 },
|
||||
/// Current recursive thumbnail-build progress.
|
||||
ThumbnailBuild { status: ThumbnailBuildStatus },
|
||||
/// An on-disk path the front-end can open (e.g. a downloaded photo).
|
||||
FilePath { path: String },
|
||||
/// Full-text search results (reply to [`Request::Search`]).
|
||||
@@ -1381,7 +1672,7 @@ impl ErrorKind {
|
||||
}
|
||||
|
||||
/// Send one [`Request`] to the daemon listening on `socket` and read its
|
||||
/// [`Response`]. Errors (e.g. [`Error::Io`]) if no daemon is listening.
|
||||
/// [`Response`]. Returns a crate [`crate::Error`] if no daemon is listening.
|
||||
///
|
||||
/// Shared by the CLI and GUI so both speak the wire format identically.
|
||||
pub fn send(socket: &Path, req: &Request) -> Result<Response> {
|
||||
@@ -1540,6 +1831,86 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn file_thumbnail_requests_keep_their_revision_tags() {
|
||||
let request = Request::FileThumbs {
|
||||
items: vec![
|
||||
FileThumbRequest {
|
||||
uid: "volume~first".into(),
|
||||
modified: 1_700_000_001,
|
||||
name: "first.jpg".into(),
|
||||
},
|
||||
FileThumbRequest {
|
||||
uid: "volume~second".into(),
|
||||
modified: 1_700_000_002,
|
||||
name: "second.nef".into(),
|
||||
},
|
||||
],
|
||||
generation: 42,
|
||||
};
|
||||
let line = serde_json::to_string(&request).unwrap();
|
||||
let decoded: Request = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_file_thumbnail_requests_default_the_name() {
|
||||
let decoded: Request = serde_json::from_str(
|
||||
r#"{"FileThumbs":{"items":[{"uid":"volume~first","modified":1700000001}],"generation":42}}"#,
|
||||
)
|
||||
.unwrap();
|
||||
let Request::FileThumbs { items, generation } = decoded else {
|
||||
panic!("decoded the wrong request variant");
|
||||
};
|
||||
assert_eq!(generation, 42);
|
||||
assert_eq!(items.len(), 1);
|
||||
assert!(items[0].name.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn thumbnail_build_requests_and_progress_roundtrip() {
|
||||
let requests = [
|
||||
Request::ReserveFileThumbGeneration,
|
||||
Request::CancelFileThumbs { generation: 43 },
|
||||
Request::StartThumbnailBuild {
|
||||
path: "pictures/events".into(),
|
||||
},
|
||||
Request::CancelThumbnailBuild,
|
||||
Request::ThumbnailBuildStatus,
|
||||
];
|
||||
for request in requests {
|
||||
let line = serde_json::to_string(&request).unwrap();
|
||||
let decoded: Request = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
|
||||
}
|
||||
|
||||
let response = Response::ThumbnailBuild {
|
||||
status: ThumbnailBuildStatus {
|
||||
running: true,
|
||||
scanning: false,
|
||||
path: "pictures/events".into(),
|
||||
folders_scanned: 12,
|
||||
images_found: 80,
|
||||
completed: 25,
|
||||
failed: 1,
|
||||
message: None,
|
||||
},
|
||||
};
|
||||
let line = serde_json::to_string(&response).unwrap();
|
||||
let decoded: Response = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
|
||||
|
||||
let response = Response::FileThumbGeneration { generation: 44 };
|
||||
let line = serde_json::to_string(&response).unwrap();
|
||||
let decoded: Response = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
|
||||
|
||||
let response = Response::FileThumbsStale;
|
||||
let line = serde_json::to_string(&response).unwrap();
|
||||
let decoded: Response = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(line, serde_json::to_string(&decoded).unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unified_search_request_and_response_roundtrip() {
|
||||
let request = Request::SearchV2 {
|
||||
|
||||
@@ -33,7 +33,7 @@ impl Db {
|
||||
/// whose stored `kind` no longer parses (written by an older build) are
|
||||
/// skipped rather than failing the whole read.
|
||||
pub fn activity_list(&self, limit: usize) -> Result<Vec<ActivityEntry>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT time, kind, target, detail, ok FROM activity
|
||||
ORDER BY id DESC LIMIT ?1",
|
||||
|
||||
@@ -76,7 +76,7 @@ impl Db {
|
||||
|
||||
/// The persisted album listing, in stored order.
|
||||
pub fn albums_list(&self) -> Result<Vec<StoredAlbum>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT uid, name, photo_count, cover_uid, last_activity, shared
|
||||
FROM albums ORDER BY seq",
|
||||
@@ -98,7 +98,7 @@ impl Db {
|
||||
|
||||
/// Number of albums in the persisted listing.
|
||||
pub fn albums_count(&self) -> Result<usize> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let n: i64 = conn.query_row("SELECT COUNT(*) FROM albums", [], |r| r.get(0))?;
|
||||
Ok(n.max(0) as usize)
|
||||
}
|
||||
@@ -164,7 +164,7 @@ impl Db {
|
||||
offset: usize,
|
||||
limit: usize,
|
||||
) -> Result<Vec<StoredPhoto>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT uid, capture_time, name, ratio, thumb_state, kind FROM album_photos
|
||||
WHERE album_uid = ?1 ORDER BY seq LIMIT ?2 OFFSET ?3",
|
||||
@@ -188,9 +188,22 @@ impl Db {
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
/// Which albums hold `uid`, by album node uid. Read by the photo re-date,
|
||||
/// which has to put the re-uploaded copy back into the albums the original
|
||||
/// was in.
|
||||
pub fn albums_of_photo(&self, uid: &str) -> Result<Vec<String>> {
|
||||
let conn = self.read();
|
||||
let mut stmt =
|
||||
conn.prepare("SELECT DISTINCT album_uid FROM album_photos WHERE uid = ?1")?;
|
||||
let rows = stmt
|
||||
.query_map([uid], |r| r.get(0))?
|
||||
.collect::<rusqlite::Result<_>>()?;
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
/// How many photos of `album_uid` are persisted.
|
||||
pub fn album_photos_count(&self, album_uid: &str) -> Result<usize> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let n: i64 = conn.query_row(
|
||||
"SELECT COUNT(*) FROM album_photos WHERE album_uid = ?1",
|
||||
[album_uid],
|
||||
@@ -207,7 +220,7 @@ impl Db {
|
||||
if uids.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let placeholders = vec!["?"; uids.len()].join(",");
|
||||
let mut stmt = conn.prepare(&format!(
|
||||
"SELECT uid, capture_time, name, ratio, thumb_state, kind FROM album_photos
|
||||
|
||||
@@ -42,6 +42,30 @@ impl Db {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Bump `last_accessed` for many keys in one transaction.
|
||||
///
|
||||
/// The read path touches a key per 128 KiB the kernel asks for — 32 times
|
||||
/// per 4 MiB block — and each of those was its own write transaction under
|
||||
/// the single connection mutex, contending with every `lookup` and
|
||||
/// `readdir`. Callers accumulate touches in memory and flush them here.
|
||||
pub fn cache_accessed_batch(&self, touches: &[(String, i64)]) -> Result<()> {
|
||||
if touches.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let mut conn = self.conn.lock();
|
||||
let tx = conn.transaction()?;
|
||||
{
|
||||
let mut stmt = tx.prepare_cached(
|
||||
"UPDATE cache_entries SET last_accessed = ?2 WHERE cache_key = ?1",
|
||||
)?;
|
||||
for (key, at) in touches {
|
||||
stmt.execute(params![key, at])?;
|
||||
}
|
||||
}
|
||||
tx.commit()?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Drop a single cache-entry row (one evicted blob or block).
|
||||
pub fn cache_remove(&self, key: &str) -> Result<()> {
|
||||
let conn = self.conn.lock();
|
||||
@@ -75,7 +99,7 @@ impl Db {
|
||||
/// because `enforce_block_budget` runs on every cached block — once per
|
||||
/// 4 MiB of every cold read — while holding the shared connection.
|
||||
pub fn cache_total_bytes(&self, kind: &str) -> Result<u64> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let total: i64 = conn.query_row(
|
||||
"SELECT COALESCE(SUM(size_bytes), 0) FROM cache_entries WHERE kind = ?1",
|
||||
params![kind],
|
||||
@@ -95,7 +119,7 @@ impl Db {
|
||||
kind: &str,
|
||||
limit: usize,
|
||||
) -> Result<Vec<(String, u64)>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare_cached(
|
||||
"SELECT cache_key, size_bytes FROM cache_entries
|
||||
WHERE kind = ?1 ORDER BY last_accessed ASC LIMIT ?2",
|
||||
@@ -115,7 +139,7 @@ impl Db {
|
||||
/// (`clear_unpinned`); incremental eviction wants
|
||||
/// [`cache_eviction_candidates`](Self::cache_eviction_candidates).
|
||||
pub fn cache_entries_by_kind(&self, kind: &str) -> Result<Vec<(String, u64)>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT cache_key, size_bytes FROM cache_entries
|
||||
WHERE kind = ?1 ORDER BY last_accessed ASC",
|
||||
|
||||
@@ -15,7 +15,7 @@ pub struct StoredDevice {
|
||||
|
||||
impl Db {
|
||||
pub fn device_get(&self) -> Result<Option<StoredDevice>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
conn.query_row(
|
||||
"SELECT uid, share_id, root_uid, name, created FROM device LIMIT 1",
|
||||
[],
|
||||
|
||||
@@ -41,32 +41,43 @@ impl Db {
|
||||
Ok(next)
|
||||
}
|
||||
|
||||
/// Write one batch of walked entries under scan generation `generation`. The
|
||||
/// FTS index is *not* touched here — it is rebuilt once in
|
||||
/// [`local_finish_scan`](Self::local_finish_scan).
|
||||
/// Write one batch of walked entries under scan generation `generation`,
|
||||
/// keeping the FTS index in step with them.
|
||||
///
|
||||
/// The index is maintained incrementally rather than rebuilt at the end of
|
||||
/// the scan, because a rebuild reads and re-tokenises every row in the table
|
||||
/// — trigrams over a few hundred thousand paths — inside one transaction on
|
||||
/// the single shared connection, which stalls the whole mount for as long as
|
||||
/// it takes. Incremental maintenance is possible here for a reason specific
|
||||
/// to this table: `local_fts` covers `name` and `path`, `path` is the
|
||||
/// table's key, and `name` is derived from it, so an existing row's indexed
|
||||
/// content can never change. Only insertions and deletions touch the index;
|
||||
/// a re-scan that finds the same files does no index work at all.
|
||||
///
|
||||
/// Hence the update-then-insert shape below rather than an upsert: the
|
||||
/// distinction between "already indexed" and "new row" is exactly what
|
||||
/// decides whether the index needs a write.
|
||||
pub fn local_upsert_batch(&self, generation: i64, entries: &[LocalEntry]) -> Result<()> {
|
||||
let mut conn = self.conn.lock();
|
||||
let tx = conn.transaction()?;
|
||||
{
|
||||
let mut stmt = tx.prepare_cached(
|
||||
"INSERT INTO local_files (path, name, is_dir, size, mtime, scan_gen)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6)
|
||||
ON CONFLICT(path) DO UPDATE SET
|
||||
name = excluded.name,
|
||||
is_dir = excluded.is_dir,
|
||||
size = excluded.size,
|
||||
mtime = excluded.mtime,
|
||||
scan_gen = excluded.scan_gen",
|
||||
let mut update = tx.prepare_cached(
|
||||
"UPDATE local_files
|
||||
SET name = ?2, is_dir = ?3, size = ?4, mtime = ?5, scan_gen = ?6
|
||||
WHERE path = ?1",
|
||||
)?;
|
||||
let mut insert = tx.prepare_cached(
|
||||
"INSERT INTO local_files (path, name, is_dir, size, mtime, scan_gen)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6)",
|
||||
)?;
|
||||
let mut index =
|
||||
tx.prepare_cached("INSERT INTO local_fts (rowid, name, path) VALUES (?1, ?2, ?3)")?;
|
||||
for e in entries {
|
||||
stmt.execute(params![
|
||||
e.path,
|
||||
e.name,
|
||||
e.is_dir as i64,
|
||||
e.size,
|
||||
e.mtime,
|
||||
generation
|
||||
])?;
|
||||
let row = params![e.path, e.name, e.is_dir as i64, e.size, e.mtime, generation];
|
||||
if update.execute(row)? == 0 {
|
||||
insert.execute(row)?;
|
||||
index.execute(params![tx.last_insert_rowid(), e.name, e.path])?;
|
||||
}
|
||||
}
|
||||
}
|
||||
tx.commit()?;
|
||||
@@ -74,16 +85,25 @@ impl Db {
|
||||
}
|
||||
|
||||
/// Close scan generation `generation`: drop every row an older scan wrote
|
||||
/// (those paths are gone from disk), rebuild the FTS index over what remains,
|
||||
/// and stamp the completion time. Returns the number of indexed entries.
|
||||
/// (those paths are gone from disk), drop their FTS entries, and stamp the
|
||||
/// completion time. Returns the number of indexed entries.
|
||||
///
|
||||
/// The FTS deletes name the old values explicitly, which is how an
|
||||
/// external-content FTS5 table is told to retract a row — see
|
||||
/// [`local_upsert_batch`](Self::local_upsert_batch) for why this is done
|
||||
/// per-row instead of by rebuilding the index.
|
||||
pub fn local_finish_scan(&self, generation: i64, finished_at: i64) -> Result<i64> {
|
||||
let mut conn = self.conn.lock();
|
||||
let tx = conn.transaction()?;
|
||||
tx.execute(
|
||||
"INSERT INTO local_fts (local_fts, rowid, name, path)
|
||||
SELECT 'delete', id, name, path FROM local_files WHERE scan_gen != ?1",
|
||||
params![generation],
|
||||
)?;
|
||||
tx.execute(
|
||||
"DELETE FROM local_files WHERE scan_gen != ?1",
|
||||
params![generation],
|
||||
)?;
|
||||
tx.execute_batch("INSERT INTO local_fts(local_fts) VALUES('rebuild');")?;
|
||||
tx.execute(
|
||||
"INSERT INTO sync_state (key, value) VALUES ('local_indexed_at', ?1)
|
||||
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
|
||||
@@ -98,7 +118,7 @@ impl Db {
|
||||
/// has never been built. The daemon uses this to decide whether a fresh mount
|
||||
/// needs an immediate rescan or can serve the existing index.
|
||||
pub fn local_indexed_at(&self) -> Result<Option<i64>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
Ok(conn
|
||||
.query_row(
|
||||
"SELECT value FROM sync_state WHERE key = 'local_indexed_at'",
|
||||
@@ -118,7 +138,7 @@ impl Db {
|
||||
if query.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt;
|
||||
let rows = if query.chars().count() < TRIGRAM_MIN {
|
||||
let pat = format!("%{}%", like_escape(query));
|
||||
@@ -157,7 +177,7 @@ impl Db {
|
||||
if query.is_empty() || limit == 0 {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let terms = candidate_trigrams(query);
|
||||
let lane_limit = limit.div_ceil(2);
|
||||
let mut rows: Vec<LocalFileHit> = if terms.is_empty() {
|
||||
|
||||
@@ -9,7 +9,7 @@ use super::Db;
|
||||
use crate::Result;
|
||||
|
||||
/// Current schema version. Bump on every forward migration added below.
|
||||
pub(super) const SCHEMA_VERSION: i64 = 21;
|
||||
pub(super) const SCHEMA_VERSION: i64 = 28;
|
||||
|
||||
impl Db {
|
||||
pub(super) fn migrate(&self) -> Result<()> {
|
||||
@@ -113,6 +113,111 @@ impl Db {
|
||||
tx.execute_batch(MIGRATION_V21)?;
|
||||
}
|
||||
}
|
||||
if current < 22 {
|
||||
// Older fixtures (and a database that never grew a table because
|
||||
// its feature was never used) may be missing any of these, and an
|
||||
// index on an absent table is a hard error rather than a no-op.
|
||||
for (table, sql) in MIGRATION_V22 {
|
||||
let present: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?1",
|
||||
[table],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
if present {
|
||||
tx.execute_batch(sql)?;
|
||||
}
|
||||
}
|
||||
// `activity_time` indexes `time DESC` while both activity queries
|
||||
// order by `id`: it has only ever cost writes.
|
||||
tx.execute_batch("DROP INDEX IF EXISTS activity_time;")?;
|
||||
}
|
||||
if current < 23 {
|
||||
// One last full rebuild of the local index, because from here on the
|
||||
// scan maintains it incrementally (`local_upsert_batch`) and so can
|
||||
// no longer repair a `local_fts` that drifted from `local_files`.
|
||||
// Whatever it costs, it costs once, at startup, before the mount
|
||||
// serves anything.
|
||||
let present: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'local_fts'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
if present {
|
||||
tx.execute_batch("INSERT INTO local_fts(local_fts) VALUES('rebuild');")?;
|
||||
}
|
||||
}
|
||||
if current < 24 {
|
||||
tx.execute_batch(MIGRATION_V24)?;
|
||||
}
|
||||
if current < 25 {
|
||||
// Same guard as V21: an old fixture may not have `nodes` at all,
|
||||
// and there is no path to materialise without it.
|
||||
let has_nodes: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'nodes'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
if has_nodes {
|
||||
tx.execute_batch(MIGRATION_V25)?;
|
||||
}
|
||||
}
|
||||
if current < 26 {
|
||||
// Same guard as V22: a fixture that never exercised the queue may
|
||||
// not have `pending_op`, and altering an absent table is an error
|
||||
// rather than a no-op.
|
||||
// A fixture built from the current schema and then rewound to an
|
||||
// older `schema_version` already has the column, and `ADD COLUMN`
|
||||
// has no `IF NOT EXISTS`.
|
||||
let has_column: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM pragma_table_info('pending_op') WHERE name = 'claimed_at'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
let has_ops: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
if has_ops && !has_column {
|
||||
tx.execute_batch(MIGRATION_V26)?;
|
||||
}
|
||||
}
|
||||
if current < 27 {
|
||||
// Same guards as V26, for the same two reasons: the table may not
|
||||
// exist in an old fixture, and a fixture rewound from the current
|
||||
// schema already has the column.
|
||||
let has_column: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM pragma_table_info('pending_op') \
|
||||
WHERE name = 'access_deferred_since'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
let has_ops: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'pending_op'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
if has_ops && !has_column {
|
||||
tx.execute_batch(MIGRATION_V27)?;
|
||||
}
|
||||
}
|
||||
if current < 28 {
|
||||
// Same guards as V26/V27, for the same two reasons.
|
||||
let has_column: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM pragma_table_info('sync_entry') \
|
||||
WHERE name = 'local_mtime_ns'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
let has_entries: bool = tx.query_row(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'sync_entry'",
|
||||
[],
|
||||
|row| row.get::<_, i64>(0),
|
||||
)? > 0;
|
||||
if has_entries && !has_column {
|
||||
tx.execute_batch(MIGRATION_V28)?;
|
||||
}
|
||||
}
|
||||
tx.execute(
|
||||
"INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1)
|
||||
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
|
||||
@@ -583,3 +688,161 @@ SELECT n.rowid, n.name, paths.path
|
||||
FROM nodes n JOIN paths ON paths.rowid = n.rowid
|
||||
WHERE paths.trashed = 0;
|
||||
";
|
||||
|
||||
/// V22: indexes for the lookups that were scanning, each paired with the table
|
||||
/// it needs to exist for.
|
||||
///
|
||||
/// Each backs a query on a hot path that had no index to use:
|
||||
///
|
||||
/// - `sync_entry(remote_uid)` — the reconcile resolves a remote uid to its
|
||||
/// local entry once per node, per pass.
|
||||
/// - `local_files(scan_gen)` — the sweep that retires the previous generation
|
||||
/// after a local index scan.
|
||||
/// - `nodes(parent_uid) WHERE trashed = 0` — every listing, with the partial
|
||||
/// index keeping it to the live rows a listing actually wants.
|
||||
///
|
||||
/// Deliberately *not* here: an index on `pending_op(next_attempt_at, id)`.
|
||||
/// `next_due_op` orders by `id`, which is the rowid, so it already stops at the
|
||||
/// first due row; offering the planner a `next_attempt_at` index makes it scan
|
||||
/// that index and then sort the matches by `id` instead —
|
||||
/// `next_due_op_does_not_scale_with_queue_length` measures the regression.
|
||||
const MIGRATION_V22: [(&str, &str); 3] = [
|
||||
(
|
||||
"sync_entry",
|
||||
"CREATE INDEX IF NOT EXISTS idx_sync_entry_remote_uid ON sync_entry(remote_uid);",
|
||||
),
|
||||
(
|
||||
"local_files",
|
||||
"CREATE INDEX IF NOT EXISTS idx_local_files_scan_gen ON local_files(scan_gen);",
|
||||
),
|
||||
(
|
||||
"nodes",
|
||||
"CREATE INDEX IF NOT EXISTS idx_nodes_parent_live ON nodes(parent_uid) WHERE trashed = 0;",
|
||||
),
|
||||
];
|
||||
|
||||
/// V24: the revisions this daemon sealed itself, so restarting does not reopen
|
||||
/// the conflict hole B70 layer B closed.
|
||||
///
|
||||
/// `Core::own_sealed_revs` held these in memory: a queued write whose baseline
|
||||
/// names an earlier revision is a conflict *unless* the remote sits at a
|
||||
/// revision we sealed, in which case it is one writer stalling and resuming and
|
||||
/// must supersede rather than fork a `(sync-conflict)` copy. Losing the map at
|
||||
/// shutdown meant the first drain after a restart could not tell the two apart.
|
||||
///
|
||||
/// `sealed_at` is what bounds the table: entries older than the window in which
|
||||
/// a stall→resume is plausible are pruned on write.
|
||||
const MIGRATION_V24: &str = "
|
||||
CREATE TABLE IF NOT EXISTS own_sealed_rev (
|
||||
uid TEXT PRIMARY KEY,
|
||||
revision_id TEXT NOT NULL,
|
||||
sealed_at INTEGER NOT NULL
|
||||
);
|
||||
";
|
||||
|
||||
/// V25: `nodes.path` — a node's mountpoint-relative path, stored rather than
|
||||
/// walked.
|
||||
///
|
||||
/// `path_of` was a recursive CTE per node, and search ran it once per candidate
|
||||
/// (up to a thousand per keystroke) on top of the one the write path already ran
|
||||
/// per descendant of every folder upsert. The value only changes when a node's
|
||||
/// name or parent changes, which is exactly when the write path is already
|
||||
/// walking the subtree — so it belongs in a column.
|
||||
///
|
||||
/// The backfill is the same walk V21 used for `nodes_fts`, minus the trashed
|
||||
/// filter: a trashed node stays out of the index but still has a path. Roots get
|
||||
/// `''` (their name is the mount), and a node whose parent is merely uncached
|
||||
/// keeps its own name — both matching what `path_of` returned at runtime.
|
||||
///
|
||||
/// Materialised into a temp table rather than written by a correlated subquery,
|
||||
/// so the recursive walk runs once instead of once per row.
|
||||
const MIGRATION_V25: &str = "
|
||||
ALTER TABLE nodes ADD COLUMN path TEXT;
|
||||
|
||||
CREATE TEMP TABLE node_paths (uid TEXT PRIMARY KEY, path TEXT NOT NULL);
|
||||
|
||||
INSERT INTO node_paths (uid, path)
|
||||
WITH RECURSIVE paths(uid, path, depth) AS (
|
||||
SELECT n.uid, CASE WHEN n.parent_uid IS NULL THEN '' ELSE n.name END, 0
|
||||
FROM nodes n
|
||||
WHERE n.parent_uid IS NULL
|
||||
OR NOT EXISTS (SELECT 1 FROM nodes p WHERE p.uid = n.parent_uid)
|
||||
UNION ALL
|
||||
SELECT n.uid,
|
||||
CASE WHEN paths.path = '' THEN n.name ELSE paths.path || '/' || n.name END,
|
||||
paths.depth + 1
|
||||
FROM nodes n JOIN paths ON n.parent_uid = paths.uid
|
||||
WHERE paths.depth < 256
|
||||
)
|
||||
SELECT uid, path FROM paths;
|
||||
|
||||
UPDATE nodes
|
||||
SET path = (SELECT p.path FROM node_paths p WHERE p.uid = nodes.uid)
|
||||
WHERE EXISTS (SELECT 1 FROM node_paths p WHERE p.uid = nodes.uid);
|
||||
|
||||
DROP TABLE node_paths;
|
||||
";
|
||||
|
||||
/// V26: `pending_op.claimed_at` — the drain's claim column.
|
||||
///
|
||||
/// The drain was one thread taking one op at a time, so a 10 GiB upload held
|
||||
/// every queued rename and small write behind it for as long as it ran. Several
|
||||
/// workers can share the queue only if a claim is visible to the others: this
|
||||
/// column is what makes "already being drained by someone" a fact about the row
|
||||
/// rather than about one thread's local variable.
|
||||
///
|
||||
/// `0` is unclaimed; anything else is the ms at which a worker took it. A
|
||||
/// claim is process-local state that outlives nothing — the single-writer lock
|
||||
/// means a non-zero value found at open belongs to a crashed run, and
|
||||
/// [`Db::clear_op_claims`](super::Db::clear_op_claims) drops the lot.
|
||||
///
|
||||
/// The index is partial so it costs nothing on the ordinary row: only the
|
||||
/// handful of claimed rows are in it, which is exactly the set the claim query
|
||||
/// has to subtract to keep two workers off one node.
|
||||
const MIGRATION_V26: &str = "
|
||||
ALTER TABLE pending_op ADD COLUMN claimed_at INTEGER NOT NULL DEFAULT 0;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_pending_op_claimed
|
||||
ON pending_op(uid) WHERE claimed_at <> 0;
|
||||
";
|
||||
|
||||
/// Schema v27: when an op was first deferred by the local access check.
|
||||
///
|
||||
/// An access deferral consumes no attempt and records no error — the op has not
|
||||
/// failed remotely, it has not been sent at all. That is right for a permission
|
||||
/// change the user is about to undo, and wrong for a condition that never
|
||||
/// clears: the row is re-deferred every
|
||||
/// [`DRAIN_ACCESS_RECHECK`](../../../pdfs_fuse/drain/constant.DRAIN_ACCESS_RECHECK.html)
|
||||
/// forever while `attempts`, `last_error`, `failing` and the logs all stay
|
||||
/// empty, so a queue that can never drain is indistinguishable from a busy one.
|
||||
/// Found in the field as 28 revision ops holding 18 GiB of accepted writes for
|
||||
/// 30 days, none of them visible anywhere.
|
||||
///
|
||||
/// `0` means "not currently access-deferred"; anything else is the ms at which
|
||||
/// the first consecutive deferral happened, which is what
|
||||
/// [`Db::defer_op_for_access`](super::Db::defer_op_for_access) measures the
|
||||
/// deferral against.
|
||||
const MIGRATION_V27: &str = "
|
||||
ALTER TABLE pending_op ADD COLUMN access_deferred_since INTEGER NOT NULL DEFAULT 0;
|
||||
";
|
||||
|
||||
/// Schema v28: sub-second precision for a mirror folder's local baseline.
|
||||
///
|
||||
/// A reconcile pass decides whether the local side of a path changed by
|
||||
/// comparing `(local_mtime, local_size)` against the baseline. Both sides of
|
||||
/// that comparison were whole seconds, so an edit that landed in the same second
|
||||
/// as the last sync and left the file the same length — a fixed-size record
|
||||
/// rewritten, an image re-exported, a database page updated in place — read as
|
||||
/// *unchanged* and was never uploaded. The remote copy silently stayed at the
|
||||
/// older content until something else about the file moved (bugs.md B25).
|
||||
///
|
||||
/// Nullable rather than a converted `local_mtime * 1_000_000_000`: the
|
||||
/// nanosecond part of an existing baseline is genuinely unknown, and inventing
|
||||
/// zero for it would make every already-synced file compare as changed and
|
||||
/// re-upload an entire mirror on first start. `NULL` means "this row predates
|
||||
/// the column", which the comparison reads as "fall back to whole seconds" —
|
||||
/// exactly the behaviour that row was written under. Each row gains a real value
|
||||
/// the next time its path is synced.
|
||||
const MIGRATION_V28: &str = "
|
||||
ALTER TABLE sync_entry ADD COLUMN local_mtime_ns INTEGER;
|
||||
";
|
||||
@@ -2,9 +2,17 @@
|
||||
//! inode bookkeeping, full-text search, content-cache LRU tracking, and pins.
|
||||
//!
|
||||
//! Only the daemon (`pdfs-fuse`) opens this for writes; the GUI and CLI reach
|
||||
//! the same data through the control socket. The connection is wrapped in a
|
||||
//! `Mutex` because the FUSE callbacks are synchronous and already serialize
|
||||
//! behind the `State` lock, so a connection pool would be overkill.
|
||||
//! the same data through the control socket. There is one write connection
|
||||
//! behind a `Mutex` — SQLite allows exactly one writer, and the single-instance
|
||||
//! `flock` makes sure it is ours — and a small pool of read-only connections
|
||||
//! behind [`Db::read`], which every `SELECT`-only method uses.
|
||||
//!
|
||||
//! The pool is what makes WAL worth having. The original justification for a
|
||||
//! single connection was that "FUSE callbacks already serialize behind the
|
||||
//! `State` lock"; that stopped being true when reads moved onto an 11-thread
|
||||
//! worker pool, and a lookup then queued behind whatever listing happened to be
|
||||
//! committing. Under WAL a reader never blocks the writer and vice versa, so the
|
||||
//! only thing left serialising them was this process's own mutex.
|
||||
//!
|
||||
//! This module is the P0 foundation: it opens the database, enables WAL, and
|
||||
//! applies the forward-only schema migrations. Write-through of nodes, the
|
||||
@@ -18,8 +26,10 @@
|
||||
use parking_lot::Mutex;
|
||||
use rusqlite::Connection;
|
||||
|
||||
use crate::Result;
|
||||
use std::path::Path;
|
||||
use crate::{Error, Result};
|
||||
use std::fs::File;
|
||||
use std::os::fd::AsRawFd;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
mod activity;
|
||||
mod albums;
|
||||
@@ -66,18 +76,208 @@ const WAL_SIZE_LIMIT: i64 = 64 * 1024 * 1024;
|
||||
/// genuinely stuck holder surfaces as an error rather than an apparent hang.
|
||||
const BUSY_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
|
||||
|
||||
/// Page cache, in KiB (applied negated, which is how SQLite reads a size in KiB
|
||||
/// rather than in pages). 32 MiB against a database that is typically tens of
|
||||
/// MiB: large enough to hold the working set of a listing-heavy session, small
|
||||
/// enough to be unremarkable in a desktop daemon's RSS.
|
||||
const CACHE_SIZE_KIB: i64 = 32 * 1024;
|
||||
|
||||
/// Ceiling on the memory-mapped window over the database file. 256 MiB covers
|
||||
/// any realistic index while keeping the mapping bounded.
|
||||
const MMAP_SIZE: i64 = 256 * 1024 * 1024;
|
||||
|
||||
/// WAL pages between automatic checkpoints (default 1000, i.e. ~4 MiB). Halved
|
||||
/// so the synchronous checkpoint a committing thread inherits stays short.
|
||||
const WAL_AUTOCHECKPOINT_PAGES: i64 = 500;
|
||||
|
||||
/// Handle to the unified metadata database.
|
||||
///
|
||||
/// Cheap to wrap in an `Arc`; clone the `Arc`, not this. All access goes through
|
||||
/// the inner `Mutex<Connection>`.
|
||||
pub struct Db {
|
||||
conn: Mutex<Connection>,
|
||||
/// Read-only connections for `SELECT`-only methods. `None` for an in-memory
|
||||
/// database, where a second connection would open a different, empty one —
|
||||
/// those fall back to the write connection.
|
||||
readers: Option<ReadPool>,
|
||||
/// The single-writer lock, held open for as long as this handle lives.
|
||||
///
|
||||
/// Never read; dropping the `File` is what releases the `flock`, so it has
|
||||
/// to be owned here rather than by `open`'s stack frame. `None` for
|
||||
/// in-memory databases, which nothing else can reach.
|
||||
_single_writer: Option<File>,
|
||||
}
|
||||
|
||||
/// Idle read-only connections, opened on demand and kept for reuse.
|
||||
///
|
||||
/// Not a bounded pool: the number in flight is bounded by the number of threads
|
||||
/// that can be inside a read at once (the FUSE worker lanes plus the control
|
||||
/// handlers), and blocking one of those on a permit would reintroduce exactly
|
||||
/// the queueing this exists to remove. Only the *idle* set is capped, so a burst
|
||||
/// does not leave a connection per thread parked forever.
|
||||
struct ReadPool {
|
||||
path: PathBuf,
|
||||
idle: Mutex<Vec<Connection>>,
|
||||
}
|
||||
|
||||
/// How many idle read connections to keep. Enough that the steady state never
|
||||
/// reopens, small enough to be unremarkable.
|
||||
const MAX_IDLE_READERS: usize = 4;
|
||||
|
||||
impl ReadPool {
|
||||
fn take(&self) -> Option<Connection> {
|
||||
if let Some(conn) = self.idle.lock().pop() {
|
||||
return Some(conn);
|
||||
}
|
||||
match open_read_only(&self.path) {
|
||||
Ok(conn) => Some(conn),
|
||||
Err(error) => {
|
||||
// Falling back to the write connection is correct, just slower,
|
||||
// so this is a warning and not a failed read.
|
||||
tracing::warn!(%error, "could not open a read-only connection");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn put(&self, conn: Connection) {
|
||||
let mut idle = self.idle.lock();
|
||||
if idle.len() < MAX_IDLE_READERS {
|
||||
idle.push(conn);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A connection to read through: one borrowed from [`ReadPool`], or the write
|
||||
/// connection when there is no pool (in-memory) or opening one failed.
|
||||
///
|
||||
/// Derefs to `Connection`, so a method converts by swapping `self.conn.lock()`
|
||||
/// for `self.read()` and nothing else.
|
||||
pub(super) struct Reader<'a> {
|
||||
pooled: Option<Connection>,
|
||||
writer: Option<parking_lot::MutexGuard<'a, Connection>>,
|
||||
pool: Option<&'a ReadPool>,
|
||||
}
|
||||
|
||||
impl std::ops::Deref for Reader<'_> {
|
||||
type Target = Connection;
|
||||
|
||||
fn deref(&self) -> &Connection {
|
||||
self.pooled
|
||||
.as_ref()
|
||||
.or(self.writer.as_deref())
|
||||
.expect("a Reader always holds one of the two")
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Reader<'_> {
|
||||
fn drop(&mut self) {
|
||||
if let (Some(conn), Some(pool)) = (self.pooled.take(), self.pool) {
|
||||
pool.put(conn);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Open a second connection to the same file for reads only.
|
||||
///
|
||||
/// `SQLITE_OPEN_READ_ONLY` plus `query_only` is belt and braces: the flag stops
|
||||
/// a routing mistake at the file, the pragma stops it at the statement. The rest
|
||||
/// mirror [`Db::open_configured`] — a read connection has its own page cache,
|
||||
/// its own temp store and its own busy timeout, and gets nothing from the
|
||||
/// writer's.
|
||||
///
|
||||
/// `journal_mode` is deliberately absent: it is a property of the database file,
|
||||
/// which the writer already set to WAL, and a read-only connection cannot change
|
||||
/// it anyway.
|
||||
fn open_read_only(path: &Path) -> Result<Connection> {
|
||||
use rusqlite::OpenFlags;
|
||||
let conn = Connection::open_with_flags(
|
||||
path,
|
||||
OpenFlags::SQLITE_OPEN_READ_ONLY | OpenFlags::SQLITE_OPEN_NO_MUTEX,
|
||||
)?;
|
||||
conn.busy_timeout(BUSY_TIMEOUT)?;
|
||||
conn.pragma_update(None, "cache_size", -CACHE_SIZE_KIB)?;
|
||||
conn.pragma_update(None, "temp_store", "MEMORY")?;
|
||||
conn.pragma_update(None, "mmap_size", MMAP_SIZE)?;
|
||||
conn.pragma_update(None, "query_only", true)?;
|
||||
Ok(conn)
|
||||
}
|
||||
|
||||
impl Db {
|
||||
/// Open (creating if absent) the database at `path`, enable WAL, and bring
|
||||
/// the schema up to [`SCHEMA_VERSION`].
|
||||
///
|
||||
/// Takes the single-instance lock first (see
|
||||
/// [`acquire_single_writer_lock`]), and rebuilds from empty if what is on
|
||||
/// disk turns out not to be a readable database (see [`is_corrupt`]).
|
||||
pub fn open(path: &Path) -> Result<Self> {
|
||||
let single_writer = acquire_single_writer_lock(path)?;
|
||||
let conn = match Self::open_configured(path) {
|
||||
Ok(conn) => conn,
|
||||
Err(e) if is_corrupt(&e) => {
|
||||
// Every byte in here is derived from the account or from the
|
||||
// content cache directory, so the recovery is to start over —
|
||||
// loudly, and keeping the damaged file for a post-mortem rather
|
||||
// than deleting evidence.
|
||||
tracing::error!(
|
||||
db = %path.display(),
|
||||
error = %e,
|
||||
"cache database is corrupt — moving it aside and rebuilding \
|
||||
(metadata will be re-fetched; no user data is stored here)"
|
||||
);
|
||||
quarantine_corrupt_db(path)?;
|
||||
Self::open_configured(path)?
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
};
|
||||
let db = Self {
|
||||
conn: Mutex::new(conn),
|
||||
// No pool for `:memory:`. A second connection to that name is a
|
||||
// second, empty database rather than another view of this one, so
|
||||
// every routed read would find a schema that does not exist.
|
||||
readers: (!is_in_memory(path)).then(|| ReadPool {
|
||||
path: path.to_path_buf(),
|
||||
idle: Mutex::new(Vec::new()),
|
||||
}),
|
||||
_single_writer: single_writer,
|
||||
};
|
||||
// A drain claim names a worker thread in the process that took it, and
|
||||
// the single-writer lock above says that process is not running. Any
|
||||
// claim still on disk is therefore a crashed run's, and leaving it would
|
||||
// hide those ops from every worker for good.
|
||||
match db.clear_op_claims() {
|
||||
Ok(0) => {}
|
||||
Ok(n) => tracing::info!(ops = n, "released drain claims left by a previous run"),
|
||||
Err(e) => tracing::warn!(error = %e, "clearing stale drain claims failed"),
|
||||
}
|
||||
Ok(db)
|
||||
}
|
||||
|
||||
/// A connection for a `SELECT`-only method.
|
||||
///
|
||||
/// Reads see the last committed state, which is the same thing they saw
|
||||
/// through the write connection: every method here commits before it
|
||||
/// returns, and nothing holds a transaction open across calls.
|
||||
pub(super) fn read(&self) -> Reader<'_> {
|
||||
if let Some(pool) = &self.readers
|
||||
&& let Some(conn) = pool.take()
|
||||
{
|
||||
return Reader {
|
||||
pooled: Some(conn),
|
||||
writer: None,
|
||||
pool: Some(pool),
|
||||
};
|
||||
}
|
||||
Reader {
|
||||
pooled: None,
|
||||
writer: Some(self.conn.lock()),
|
||||
pool: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Open the file, apply every PRAGMA, and migrate. Split out of
|
||||
/// [`open`](Self::open) so the corruption path can retry it verbatim.
|
||||
fn open_configured(path: &Path) -> Result<Connection> {
|
||||
let conn = Connection::open(path)?;
|
||||
// WAL: readers never block the single writer. NORMAL sync is the
|
||||
// standard durability/throughput tradeoff for WAL.
|
||||
@@ -102,12 +302,39 @@ impl Db {
|
||||
// (Bare SQLite does default to 0, which is where the belief that this
|
||||
// was unset came from. That default is not what we get.)
|
||||
conn.busy_timeout(BUSY_TIMEOUT)?;
|
||||
// Page cache. The default is 2 MiB, against a schema whose hot table
|
||||
// stores a JSON blob per node — a listing of a few thousand files
|
||||
// evicts itself while it is being read.
|
||||
conn.pragma_update(None, "cache_size", -CACHE_SIZE_KIB)?;
|
||||
// Sorts and recursive CTEs materialise their intermediate results.
|
||||
// Every `ORDER BY … COLLATE NOCASE` listing and every ancestor walk
|
||||
// spills those to a file in `/tmp` by default; they are small and
|
||||
// short-lived, so memory is both faster and less surprising.
|
||||
conn.pragma_update(None, "temp_store", "MEMORY")?;
|
||||
// Read the database through the page cache of the OS rather than
|
||||
// `pread` per page. Bounded rather than unlimited so the daemon's RSS
|
||||
// does not track the database size.
|
||||
conn.pragma_update(None, "mmap_size", MMAP_SIZE)?;
|
||||
// Checkpointing is synchronous in whichever thread happens to commit
|
||||
// the transaction that crosses the threshold — often a FUSE callback.
|
||||
// A smaller autocheckpoint makes that debt smaller and more frequent
|
||||
// rather than rare and multi-second.
|
||||
conn.pragma_update(None, "wal_autocheckpoint", WAL_AUTOCHECKPOINT_PAGES)?;
|
||||
|
||||
let db = Self {
|
||||
conn: Mutex::new(conn),
|
||||
readers: None,
|
||||
_single_writer: None,
|
||||
};
|
||||
db.migrate()?;
|
||||
Ok(db)
|
||||
Ok(db.into_conn())
|
||||
}
|
||||
|
||||
/// Unwrap the connection back out of a temporary handle. Only
|
||||
/// [`open_configured`](Self::open_configured) uses it, to run `migrate`
|
||||
/// (which is written against `&self`) before the real handle exists.
|
||||
fn into_conn(self) -> Connection {
|
||||
self.conn.into_inner()
|
||||
}
|
||||
|
||||
/// Open an in-memory database. For tests.
|
||||
@@ -120,6 +347,8 @@ impl Db {
|
||||
conn.pragma_update(None, "foreign_keys", "ON")?;
|
||||
let db = Self {
|
||||
conn: Mutex::new(conn),
|
||||
readers: None,
|
||||
_single_writer: None,
|
||||
};
|
||||
db.migrate()?;
|
||||
Ok(db)
|
||||
@@ -133,5 +362,102 @@ impl Db {
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether `path` names the in-memory database rather than a file. Both
|
||||
/// spellings SQLite accepts are checked, because neither has a lock file or a
|
||||
/// corruption story.
|
||||
fn is_in_memory(path: &Path) -> bool {
|
||||
let p = path.as_os_str();
|
||||
p == ":memory:" || p.is_empty()
|
||||
}
|
||||
|
||||
/// Take the single-instance write lock for the database at `path`.
|
||||
///
|
||||
/// The daemon is the only writer by design — the front-ends go through the
|
||||
/// control socket — but nothing stopped a hand-run `pdfs mount` from starting
|
||||
/// alongside the systemd unit, at which point two processes own the same inode
|
||||
/// space, the same content cache, and the same drain queue. SQLite's own
|
||||
/// locking makes each *statement* safe and does nothing about that.
|
||||
///
|
||||
/// `flock` on a sibling `.lock` file rather than on the database itself:
|
||||
/// SQLite uses POSIX record locks, so the two never interact, and a lock file
|
||||
/// left behind carries no state — the lock lives in the kernel and dies with
|
||||
/// the process, however it exits.
|
||||
fn acquire_single_writer_lock(path: &Path) -> Result<Option<File>> {
|
||||
if is_in_memory(path) {
|
||||
return Ok(None);
|
||||
}
|
||||
let lock_path = lock_path_for(path);
|
||||
let file = std::fs::OpenOptions::new()
|
||||
.create(true)
|
||||
.read(true)
|
||||
.truncate(false)
|
||||
.write(true)
|
||||
.open(&lock_path)?;
|
||||
// SAFETY: `file` owns the descriptor and outlives the call.
|
||||
let rc = unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX | libc::LOCK_NB) };
|
||||
if rc != 0 {
|
||||
let e = std::io::Error::last_os_error();
|
||||
if e.raw_os_error() == Some(libc::EWOULDBLOCK) {
|
||||
return Err(Error::Other(format!(
|
||||
"another Proton Drive daemon is already using {} — \
|
||||
stop it first (`systemctl --user stop proton-drive.service`)",
|
||||
path.display()
|
||||
)));
|
||||
}
|
||||
return Err(Error::Io(e));
|
||||
}
|
||||
Ok(Some(file))
|
||||
}
|
||||
|
||||
fn lock_path_for(path: &Path) -> PathBuf {
|
||||
let mut name = path.as_os_str().to_owned();
|
||||
name.push(".lock");
|
||||
PathBuf::from(name)
|
||||
}
|
||||
|
||||
/// Whether `e` says the file is not a usable database, as opposed to any other
|
||||
/// failure. These are the only two conditions rebuilding fixes; a disk error, a
|
||||
/// permission problem or a newer-than-supported schema must all still fail.
|
||||
fn is_corrupt(e: &Error) -> bool {
|
||||
matches!(
|
||||
e,
|
||||
Error::Db(rusqlite::Error::SqliteFailure(inner, _))
|
||||
if inner.code == rusqlite::ErrorCode::DatabaseCorrupt
|
||||
|| inner.code == rusqlite::ErrorCode::NotADatabase
|
||||
)
|
||||
}
|
||||
|
||||
/// Move a damaged database (and its WAL sidecars) out of the way so the next
|
||||
/// open starts from empty.
|
||||
///
|
||||
/// Renamed rather than deleted: this is a cache, but it is also the only
|
||||
/// evidence of what went wrong, and it costs one file the user can remove.
|
||||
fn quarantine_corrupt_db(path: &Path) -> Result<()> {
|
||||
let stamp = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map(|d| d.as_secs())
|
||||
.unwrap_or(0);
|
||||
for suffix in ["", "-wal", "-shm"] {
|
||||
let mut from = path.as_os_str().to_owned();
|
||||
from.push(suffix);
|
||||
let from = PathBuf::from(from);
|
||||
if !from.exists() {
|
||||
continue;
|
||||
}
|
||||
let mut to = path.as_os_str().to_owned();
|
||||
to.push(format!(".corrupt-{stamp}{suffix}"));
|
||||
let to = PathBuf::from(to);
|
||||
// A failure to rename the sidecars is survivable — SQLite discards a
|
||||
// WAL whose database header no longer matches — but failing to move the
|
||||
// database itself is not, because the retry would hit the same file.
|
||||
match std::fs::rename(&from, &to) {
|
||||
Ok(()) => tracing::warn!(from = %from.display(), to = %to.display(), "quarantined"),
|
||||
Err(e) if suffix.is_empty() => return Err(Error::Io(e)),
|
||||
Err(e) => tracing::warn!(file = %from.display(), error = %e, "could not quarantine"),
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
@@ -74,7 +74,7 @@ impl Db {
|
||||
|
||||
/// Every local location, with device fields joined from `sync_folder`.
|
||||
pub fn mount_list(&self) -> Result<Vec<MountSpec>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT
|
||||
m.id,
|
||||
|
||||
@@ -10,7 +10,9 @@ use crate::{Access, Result};
|
||||
use proton_drive_rs::proton_sdk::ids::{LinkId, NodeUid, VolumeId};
|
||||
use proton_drive_rs::{Node, NodeKind};
|
||||
|
||||
use super::utils::{TRIGRAM_MIN, collect_pairs, like_escape, pair, path_of};
|
||||
use super::utils::{
|
||||
HitRow, TRIGRAM_MIN, collect_hits, hit_row, join_path, like_escape, path_of, walk_path_of,
|
||||
};
|
||||
|
||||
pub struct StoredNode {
|
||||
pub node: Node,
|
||||
@@ -59,7 +61,7 @@ impl Db {
|
||||
/// flag. Synthetic shared listings use this to serve the last completed
|
||||
/// snapshot while offline, including after a live event expired its TTL.
|
||||
pub fn visible_children(&self, parent: &NodeUid) -> Result<Vec<Node>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT node_json FROM nodes
|
||||
WHERE parent_uid = ?1 AND trashed = 0 AND node_json IS NOT NULL
|
||||
@@ -285,7 +287,7 @@ impl Db {
|
||||
/// Check if a folder node has any non-trashed children in the database.
|
||||
pub fn has_children(&self, parent_uid: &NodeUid) -> Result<bool> {
|
||||
let uid_str = parent_uid.to_string();
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let count: i64 = conn.query_row(
|
||||
"SELECT COUNT(*) FROM nodes WHERE parent_uid = ?1 AND trashed = 0",
|
||||
params![uid_str],
|
||||
@@ -294,6 +296,25 @@ impl Db {
|
||||
Ok(count > 0)
|
||||
}
|
||||
|
||||
/// A node's mountpoint-relative path, or `None` when it is not cached.
|
||||
///
|
||||
/// One indexed read of the stored `path`. Callers that need to relate many
|
||||
/// nodes to a handful of roots — search decorating each hit with its
|
||||
/// mountpoint — resolve the roots once through this and do the rest as
|
||||
/// string arithmetic, instead of a `path_relative_to` walk per pair.
|
||||
pub fn node_path(&self, uid: &str) -> Result<Option<String>> {
|
||||
let conn = self.read();
|
||||
let exists: Option<i64> = conn
|
||||
.query_row("SELECT 1 FROM nodes WHERE uid = ?1", params![uid], |r| {
|
||||
r.get(0)
|
||||
})
|
||||
.optional()?;
|
||||
if exists.is_none() {
|
||||
return Ok(None);
|
||||
}
|
||||
Ok(Some(path_of(&conn, uid)?))
|
||||
}
|
||||
|
||||
/// Resolve `uid` relative to an ancestor node.
|
||||
///
|
||||
/// This is used when a remote subtree has its own local sync mount: the
|
||||
@@ -302,7 +323,7 @@ impl Db {
|
||||
/// chain keeps callers from accidentally joining a Drive-wide path onto the
|
||||
/// wrong mountpoint. The ancestor itself resolves to the empty path.
|
||||
pub fn path_relative_to(&self, ancestor_uid: &str, uid: &str) -> Result<Option<String>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"WITH RECURSIVE anc(uid, parent_uid, name, depth) AS (
|
||||
SELECT uid, parent_uid, name, 0 FROM nodes WHERE uid = ?1
|
||||
@@ -338,15 +359,15 @@ impl Db {
|
||||
if query.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let conn = self.conn.lock();
|
||||
let rows: Vec<(String, String)> = if query.chars().count() < TRIGRAM_MIN {
|
||||
let conn = self.read();
|
||||
let rows: Vec<HitRow> = if query.chars().count() < TRIGRAM_MIN {
|
||||
let pat = format!("%{}%", like_escape(query));
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT node_json, uid FROM nodes
|
||||
"SELECT node_json, uid, path FROM nodes
|
||||
WHERE name LIKE ?1 ESCAPE '\\' AND trashed = 0 AND node_json IS NOT NULL
|
||||
ORDER BY name LIMIT ?2",
|
||||
)?;
|
||||
collect_pairs(stmt.query_map(params![pat, limit as i64], pair)?)?
|
||||
collect_hits(stmt.query_map(params![pat, limit as i64], hit_row)?)?
|
||||
} else {
|
||||
// Escape double quotes and quote each term, then combine with AND so
|
||||
// all terms must match but can appear in any order or position.
|
||||
@@ -356,18 +377,21 @@ impl Db {
|
||||
.collect::<Vec<_>>()
|
||||
.join(" AND ");
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT n.node_json, n.uid
|
||||
"SELECT n.node_json, n.uid, n.path
|
||||
FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid
|
||||
WHERE f.name MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL
|
||||
ORDER BY f.rank LIMIT ?2",
|
||||
)?;
|
||||
collect_pairs(stmt.query_map(params![phrase, limit as i64], pair)?)?
|
||||
collect_hits(stmt.query_map(params![phrase, limit as i64], hit_row)?)?
|
||||
};
|
||||
|
||||
let mut hits = Vec::with_capacity(rows.len());
|
||||
for (json, uid) in rows {
|
||||
for (json, uid, path) in rows {
|
||||
let node: Node = serde_json::from_str(&json)?;
|
||||
let path = path_of(&conn, &uid)?;
|
||||
let path = match path {
|
||||
Some(path) => path,
|
||||
None => walk_path_of(&conn, &uid)?,
|
||||
};
|
||||
hits.push(SearchHit { node, path });
|
||||
}
|
||||
Ok(hits)
|
||||
@@ -382,26 +406,26 @@ impl Db {
|
||||
if query.is_empty() || limit == 0 {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let terms = candidate_trigrams(query);
|
||||
let lane_limit = limit.div_ceil(2);
|
||||
let mut rows: Vec<(String, String)> = if terms.is_empty() {
|
||||
let mut rows: Vec<HitRow> = if terms.is_empty() {
|
||||
let pat = format!("%{}%", like_escape(query));
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT node_json, uid FROM nodes
|
||||
"SELECT node_json, uid, path FROM nodes
|
||||
WHERE name LIKE ?1 ESCAPE '\\' AND trashed = 0 AND node_json IS NOT NULL
|
||||
ORDER BY name LIMIT ?2",
|
||||
)?;
|
||||
collect_pairs(stmt.query_map(params![pat, limit as i64], pair)?)?
|
||||
collect_hits(stmt.query_map(params![pat, limit as i64], hit_row)?)?
|
||||
} else {
|
||||
let expression = terms.join(" OR ");
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT n.node_json, n.uid
|
||||
"SELECT n.node_json, n.uid, n.path
|
||||
FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid
|
||||
WHERE nodes_fts MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL
|
||||
ORDER BY f.rank LIMIT ?2",
|
||||
)?;
|
||||
collect_pairs(stmt.query_map(params![expression, lane_limit as i64], pair)?)?
|
||||
collect_hits(stmt.query_map(params![expression, lane_limit as i64], hit_row)?)?
|
||||
};
|
||||
// Trigram rank does not know that a name *starts* with the query, and a
|
||||
// short substitution can destroy every trigram outright (`vedio` vs
|
||||
@@ -411,7 +435,7 @@ impl Db {
|
||||
&& let Some(first) = query.chars().next()
|
||||
{
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT node_json, uid FROM nodes
|
||||
"SELECT node_json, uid, path FROM nodes
|
||||
WHERE name COLLATE NOCASE LIKE ?1 ESCAPE '\\'
|
||||
AND trashed = 0 AND node_json IS NOT NULL
|
||||
ORDER BY name COLLATE NOCASE LIMIT ?2",
|
||||
@@ -421,23 +445,26 @@ impl Db {
|
||||
format!("{}%", like_escape(query)),
|
||||
format!("{}%", like_escape(&first.to_string())),
|
||||
] {
|
||||
extra.extend(collect_pairs(
|
||||
stmt.query_map(params![pattern, lane_limit as i64], pair)?,
|
||||
extra.extend(collect_hits(
|
||||
stmt.query_map(params![pattern, lane_limit as i64], hit_row)?,
|
||||
)?);
|
||||
}
|
||||
let trigram = std::mem::take(&mut rows);
|
||||
for row in extra.into_iter().chain(trigram) {
|
||||
if !rows.iter().any(|(_, uid)| uid == &row.1) {
|
||||
if !rows.iter().any(|(_, uid, _)| uid == &row.1) {
|
||||
rows.push(row);
|
||||
}
|
||||
}
|
||||
rows.truncate(limit);
|
||||
}
|
||||
rows.into_iter()
|
||||
.map(|(json, uid)| {
|
||||
.map(|(json, uid, path)| {
|
||||
Ok(SearchHit {
|
||||
node: serde_json::from_str(&json)?,
|
||||
path: path_of(&conn, &uid)?,
|
||||
path: match path {
|
||||
Some(path) => path,
|
||||
None => walk_path_of(&conn, &uid)?,
|
||||
},
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
@@ -457,7 +484,7 @@ impl Db {
|
||||
|
||||
/// Load every persisted node for cold-start hydration of the `State` maps.
|
||||
pub fn load_all(&self) -> Result<Vec<StoredNode>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt =
|
||||
conn.prepare("SELECT node_json, listed FROM nodes WHERE node_json IS NOT NULL")?;
|
||||
let rows = stmt.query_map([], |row| {
|
||||
@@ -478,7 +505,7 @@ impl Db {
|
||||
/// when the API is unreachable, so the mount can still serve the cached tree
|
||||
/// (offline.md Phase 1).
|
||||
pub fn node_by_uid(&self, uid: &str) -> Result<Option<Node>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let json: Option<String> = conn
|
||||
.query_row(
|
||||
"SELECT node_json FROM nodes WHERE uid = ?1 AND node_json IS NOT NULL",
|
||||
@@ -496,7 +523,7 @@ impl Db {
|
||||
/// The daemon resumes from this on restart instead of reseeding to the
|
||||
/// server head, so changes made while unmounted are still applied (P2).
|
||||
pub fn children_if_listed(&self, parent: &NodeUid) -> Result<Option<Vec<Node>>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let listed: Option<i64> = conn
|
||||
.query_row(
|
||||
"SELECT listed FROM nodes WHERE uid = ?1",
|
||||
@@ -529,13 +556,57 @@ impl Db {
|
||||
// in sync on every store/read/evict, so it is authoritative for eviction.
|
||||
}
|
||||
|
||||
/// What a node's row looked like before this upsert, for the fields that decide
|
||||
/// whether the subtree below it has to be touched at all.
|
||||
struct PriorRow {
|
||||
parent_uid: Option<String>,
|
||||
name: String,
|
||||
path: Option<String>,
|
||||
trashed: bool,
|
||||
}
|
||||
|
||||
fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
|
||||
let json = serde_json::to_string(node)?;
|
||||
let uid = node.uid.to_string();
|
||||
let parent_uid = node.parent_uid.as_ref().map(|u| u.to_string());
|
||||
|
||||
let prior: Option<PriorRow> = tx
|
||||
.query_row(
|
||||
"SELECT parent_uid, name, path, trashed FROM nodes WHERE uid = ?1",
|
||||
params![uid],
|
||||
|row| {
|
||||
Ok(PriorRow {
|
||||
parent_uid: row.get(0)?,
|
||||
name: row.get(1)?,
|
||||
path: row.get(2)?,
|
||||
trashed: row.get::<_, i64>(3)? != 0,
|
||||
})
|
||||
},
|
||||
)
|
||||
.optional()?;
|
||||
|
||||
// A node's path is its parent's path plus its own name — one indexed lookup,
|
||||
// where resolving it from scratch is a recursive walk to the root. A node
|
||||
// whose parent is not cached (a device folder's root lives in `device`, never
|
||||
// in `nodes`) starts a path of its own, which is what the walk did too.
|
||||
let path = match &parent_uid {
|
||||
None => String::new(),
|
||||
Some(parent) => {
|
||||
let parent_path: Option<Option<String>> = tx
|
||||
.query_row(
|
||||
"SELECT path FROM nodes WHERE uid = ?1",
|
||||
params![parent],
|
||||
|row| row.get(0),
|
||||
)
|
||||
.optional()?;
|
||||
join_path(parent_path.flatten().as_deref().unwrap_or(""), &node.name)
|
||||
}
|
||||
};
|
||||
|
||||
tx.execute(
|
||||
"INSERT INTO nodes
|
||||
(uid, parent_uid, name, is_dir, size, mtime, trashed, node_json)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)
|
||||
(uid, parent_uid, name, is_dir, size, mtime, trashed, node_json, path)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)
|
||||
ON CONFLICT(uid) DO UPDATE SET
|
||||
parent_uid = excluded.parent_uid,
|
||||
name = excluded.name,
|
||||
@@ -543,65 +614,116 @@ fn upsert_node_tx(tx: &Transaction<'_>, node: &Node) -> Result<()> {
|
||||
size = excluded.size,
|
||||
mtime = excluded.mtime,
|
||||
trashed = excluded.trashed,
|
||||
node_json = excluded.node_json",
|
||||
node_json = excluded.node_json,
|
||||
path = excluded.path",
|
||||
params![
|
||||
uid,
|
||||
node.parent_uid.as_ref().map(|u| u.to_string()),
|
||||
parent_uid,
|
||||
node.name,
|
||||
node.is_folder() as i64,
|
||||
node_size(node),
|
||||
node.modification_time,
|
||||
node.trashed as i64,
|
||||
json,
|
||||
path,
|
||||
],
|
||||
)?;
|
||||
// FTS5 has no UPSERT. Folder moves also change every descendant path, so
|
||||
// refresh the whole affected subtree while keeping rowid deletes indexed.
|
||||
let rowid: i64 = tx.query_row(
|
||||
"SELECT rowid FROM nodes WHERE uid = ?1",
|
||||
params![uid],
|
||||
|row| row.get(0),
|
||||
)?;
|
||||
let mut affected = vec![rowid];
|
||||
if node.is_folder() {
|
||||
// The depth cap is a liveness guard, not a policy: `UNION ALL` over a
|
||||
// parent cycle (a's parent is b, b's parent is a — corrupt data, but the
|
||||
// API can hand it to us) never terminates, and this runs inside the
|
||||
// write transaction that holds the daemon's only SQLite connection.
|
||||
// Real Drive trees are nowhere near this deep.
|
||||
let mut stmt = tx.prepare(
|
||||
"WITH RECURSIVE descendants(rowid, uid, depth) AS (
|
||||
SELECT rowid, uid, 0 FROM nodes WHERE parent_uid = ?1
|
||||
UNION ALL
|
||||
SELECT n.rowid, n.uid, d.depth + 1 FROM nodes n
|
||||
JOIN descendants d ON n.parent_uid = d.uid
|
||||
WHERE d.depth < 256
|
||||
) SELECT rowid FROM descendants",
|
||||
)?;
|
||||
for descendant in stmt.query_map(params![uid], |row| row.get(0))? {
|
||||
affected.push(descendant?);
|
||||
}
|
||||
}
|
||||
for affected_rowid in affected {
|
||||
|
||||
// FTS5 has no UPSERT, so the node's own row is always replaced.
|
||||
let indexable = node_is_indexable_tx(tx, &uid)?;
|
||||
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![rowid])?;
|
||||
if indexable {
|
||||
tx.execute(
|
||||
"DELETE FROM nodes_fts WHERE rowid = ?1",
|
||||
params![affected_rowid],
|
||||
"INSERT INTO nodes_fts (rowid, name, path) VALUES (?1, ?2, ?3)",
|
||||
params![rowid, node.name, path],
|
||||
)?;
|
||||
let indexed: Option<(String, String)> = tx
|
||||
.query_row(
|
||||
"SELECT name, uid FROM nodes WHERE rowid = ?1",
|
||||
params![affected_rowid],
|
||||
|row| Ok((row.get(0)?, row.get(1)?)),
|
||||
)
|
||||
.optional()?;
|
||||
if let Some((name, indexed_uid)) = indexed
|
||||
&& node_is_indexable_tx(tx, &indexed_uid)?
|
||||
{
|
||||
let path = path_of(tx, &indexed_uid)?;
|
||||
tx.execute(
|
||||
"INSERT INTO nodes_fts (rowid, name, path) VALUES (?1, ?2, ?3)",
|
||||
params![affected_rowid, name, path],
|
||||
)?;
|
||||
}
|
||||
|
||||
// Everything below only matters when the subtree's paths or reachability
|
||||
// actually moved. Re-listing a folder upserts it unchanged, and paying for a
|
||||
// full descendant walk on each of those was most of what a large tree spent
|
||||
// its time on.
|
||||
let subtree_moved = prior.is_none_or(|prior| {
|
||||
prior.parent_uid != parent_uid
|
||||
|| prior.name != node.name
|
||||
|| prior.path.as_deref() != Some(path.as_str())
|
||||
|| prior.trashed != node.trashed
|
||||
});
|
||||
if node.is_folder() && subtree_moved {
|
||||
reindex_subtree_tx(tx, &uid, &path, indexable)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Rewrite every descendant's stored path and search-index row after `folder`
|
||||
/// moved, was renamed, or changed reachability.
|
||||
///
|
||||
/// One recursive walk carries both the new path and whether anything on the way
|
||||
/// down is trashed, so each descendant costs two trivial statements instead of a
|
||||
/// `path_of` walk plus an ancestor walk of its own. `folder_indexable` is the
|
||||
/// answer `node_is_indexable_tx` gave for the folder itself: what is above the
|
||||
/// subtree is the same for every node in it.
|
||||
///
|
||||
/// The depth cap is a liveness guard, not a policy: `UNION ALL` over a parent
|
||||
/// cycle (a's parent is b, b's parent is a — corrupt data, but the API can hand
|
||||
/// it to us) never terminates, and this runs inside the write transaction that
|
||||
/// holds the daemon's only SQLite connection. Real Drive trees are nowhere near
|
||||
/// this deep.
|
||||
fn reindex_subtree_tx(
|
||||
tx: &Transaction<'_>,
|
||||
folder: &str,
|
||||
folder_path: &str,
|
||||
folder_indexable: bool,
|
||||
) -> Result<()> {
|
||||
let descendants: Vec<(i64, String, bool)> = {
|
||||
let mut stmt = tx.prepare(
|
||||
"WITH RECURSIVE sub(rowid, uid, name, path, blocked, depth) AS (
|
||||
SELECT n.rowid, n.uid, n.name,
|
||||
CASE WHEN ?2 = '' THEN n.name ELSE ?2 || '/' || n.name END,
|
||||
n.trashed, 0
|
||||
FROM nodes n WHERE n.parent_uid = ?1
|
||||
UNION ALL
|
||||
SELECT n.rowid, n.uid, n.name,
|
||||
sub.path || '/' || n.name,
|
||||
MAX(sub.blocked, n.trashed), sub.depth + 1
|
||||
FROM nodes n JOIN sub ON n.parent_uid = sub.uid
|
||||
WHERE sub.depth < 256
|
||||
)
|
||||
SELECT rowid, path, blocked FROM sub",
|
||||
)?;
|
||||
let rows = stmt.query_map(params![folder, folder_path], |row| {
|
||||
Ok((
|
||||
row.get::<_, i64>(0)?,
|
||||
row.get::<_, String>(1)?,
|
||||
row.get::<_, i64>(2)? != 0,
|
||||
))
|
||||
})?;
|
||||
let mut out = Vec::new();
|
||||
for row in rows {
|
||||
out.push(row?);
|
||||
}
|
||||
out
|
||||
};
|
||||
if descendants.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let mut set_path = tx.prepare("UPDATE nodes SET path = ?2 WHERE rowid = ?1")?;
|
||||
let mut unindex = tx.prepare("DELETE FROM nodes_fts WHERE rowid = ?1")?;
|
||||
let mut index = tx.prepare(
|
||||
"INSERT INTO nodes_fts (rowid, name, path)
|
||||
SELECT rowid, name, ?2 FROM nodes WHERE rowid = ?1",
|
||||
)?;
|
||||
for (rowid, path, blocked) in descendants {
|
||||
set_path.execute(params![rowid, path])?;
|
||||
unindex.execute(params![rowid])?;
|
||||
if folder_indexable && !blocked {
|
||||
index.execute(params![rowid, path])?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
|
||||
+286
-12
@@ -119,15 +119,32 @@ pub struct PendingOp {
|
||||
}
|
||||
|
||||
/// How much work the queue owes the server, by whether it carries bytes.
|
||||
#[derive(Debug, Clone, Copy, Default)]
|
||||
#[derive(Debug, Clone, Default)]
|
||||
pub struct PendingCounts {
|
||||
/// Queued `create`/`revision` ops: files whose content is not on the remote.
|
||||
pub uploads: i64,
|
||||
/// Queued `mkdir`/`rename`/`trash` ops: metadata the remote has not been
|
||||
/// told about.
|
||||
pub changes: i64,
|
||||
/// Ops parked at [`PARK_UNTIL`]: a transient file's bytes, held back until
|
||||
/// the rename that gives it its finished name. They are queued but will
|
||||
/// never drain on their own, so counting them as "waiting to upload" would
|
||||
/// have the front-end promise progress that is not coming.
|
||||
pub parked: i64,
|
||||
/// Ops that have failed at least [`FAILING_ATTEMPTS`] times. A permanently
|
||||
/// failing op never wedges the queue — the backoff sees to that — which is
|
||||
/// exactly why it needs surfacing: it retries forever, invisibly.
|
||||
pub failing: i64,
|
||||
/// The most recent error text from a failing op, for the front-end to show
|
||||
/// instead of the bare count.
|
||||
pub last_error: Option<String>,
|
||||
}
|
||||
|
||||
/// Failed attempts after which an op is reported as failing rather than merely
|
||||
/// retrying. Six is where the backoff has reached its ceiling, so an op past it
|
||||
/// is retrying at the slowest rate it ever will and has been for minutes.
|
||||
pub const FAILING_ATTEMPTS: i64 = 6;
|
||||
|
||||
/// The outcome of folding freshly written bytes into a queued create.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct AttachedBlob {
|
||||
@@ -140,7 +157,7 @@ pub struct AttachedBlob {
|
||||
impl Db {
|
||||
/// Whether a specific desired-state operation is still queued for a node.
|
||||
pub fn has_pending_op(&self, uid: &str, kind: &str) -> Result<bool> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
conn.query_row(
|
||||
"SELECT EXISTS(
|
||||
SELECT 1 FROM pending_op WHERE uid = ?1 AND kind = ?2
|
||||
@@ -153,7 +170,7 @@ impl Db {
|
||||
|
||||
/// Read an existing operation's metadata before a superseding enqueue.
|
||||
pub fn pending_op_meta(&self, uid: &str, kind: &str) -> Result<Option<String>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
conn.query_row(
|
||||
"SELECT meta_json FROM pending_op WHERE uid = ?1 AND kind = ?2",
|
||||
params![uid, kind],
|
||||
@@ -352,7 +369,7 @@ impl Db {
|
||||
|
||||
/// Check if a queued create or mkdir op exists for `uid`.
|
||||
pub fn has_create_op(&self, uid: &str) -> Result<bool> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let count: i64 = conn.query_row(
|
||||
"SELECT COUNT(*) FROM pending_op WHERE uid = ?1 AND kind IN (?2, ?3)",
|
||||
params![uid, OP_CREATE, OP_MKDIR],
|
||||
@@ -368,7 +385,7 @@ impl Db {
|
||||
/// bytes are still owed an upload, and dropping it would discard the staged
|
||||
/// blob that holds them (`docs/BUGS.md` B71).
|
||||
pub fn has_any_op(&self, uid: &str) -> Result<bool> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let count: i64 = conn.query_row(
|
||||
"SELECT COUNT(*) FROM pending_op WHERE uid = ?1",
|
||||
params![uid],
|
||||
@@ -464,7 +481,7 @@ impl Db {
|
||||
/// sort, and no separate index to maintain. The common case (something is
|
||||
/// due) exits on the first row.
|
||||
pub fn next_due_op(&self, now: i64) -> Result<Option<PendingOp>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare_cached(&format!(
|
||||
"SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
|
||||
attempts, last_error, next_attempt_at
|
||||
@@ -495,17 +512,115 @@ impl Db {
|
||||
Ok(op)
|
||||
}
|
||||
|
||||
/// Take the oldest due op for this worker, marking it claimed so no other
|
||||
/// drain worker takes it, or `None` when there is nothing to do.
|
||||
///
|
||||
/// The drain is several workers over one queue, which is only safe because
|
||||
/// the claim is a column: a row is offered to exactly one worker, and the
|
||||
/// worker that took it is the only one that may retire, defer or fail it.
|
||||
///
|
||||
/// Two exclusions on top of [`next_due_op`](Self::next_due_op)'s:
|
||||
///
|
||||
/// * `claimed_at = 0` — not already in flight somewhere else.
|
||||
/// * no *other* claimed op shares this `uid`. Ordering only has to hold per
|
||||
/// node, but there it has to hold absolutely: a queued rename and a queued
|
||||
/// revision of one file must land in the order they were made, and two
|
||||
/// workers on one node would also race over its staged blob and its
|
||||
/// pending entry. `ORDER BY id` plus this exclusion gives per-uid serial
|
||||
/// order for free — the older op is claimed first and the newer one is
|
||||
/// invisible until it retires.
|
||||
///
|
||||
/// Select and mark are one transaction, so two workers cannot both see the
|
||||
/// row as unclaimed.
|
||||
pub fn claim_next_due_op(&self, now: i64) -> Result<Option<PendingOp>> {
|
||||
let mut conn = self.conn.lock();
|
||||
let tx = conn.transaction()?;
|
||||
let op = {
|
||||
let mut stmt = tx.prepare_cached(&format!(
|
||||
"SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
|
||||
attempts, last_error, next_attempt_at
|
||||
FROM pending_op
|
||||
WHERE next_attempt_at <= ?1
|
||||
AND claimed_at = 0
|
||||
AND (parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~')
|
||||
AND uid NOT IN (SELECT uid FROM pending_op WHERE claimed_at <> 0)
|
||||
ORDER BY id LIMIT 1",
|
||||
v = LOCAL_VOLUME,
|
||||
n = LOCAL_VOLUME.len() + 1,
|
||||
))?;
|
||||
stmt.query_row(params![now], |r| {
|
||||
Ok(PendingOp {
|
||||
id: r.get(0)?,
|
||||
kind: r.get(1)?,
|
||||
uid: r.get(2)?,
|
||||
parent_uid: r.get(3)?,
|
||||
name: r.get(4)?,
|
||||
blob_path: r.get(5)?,
|
||||
meta_json: r.get(6)?,
|
||||
created_at: r.get(7)?,
|
||||
attempts: r.get(8)?,
|
||||
last_error: r.get(9)?,
|
||||
next_attempt_at: r.get(10)?,
|
||||
})
|
||||
})
|
||||
.optional()?
|
||||
};
|
||||
if let Some(op) = op.as_ref() {
|
||||
tx.execute(
|
||||
"UPDATE pending_op SET claimed_at = ?2 WHERE id = ?1",
|
||||
params![op.id, now.max(1)],
|
||||
)?;
|
||||
}
|
||||
tx.commit()?;
|
||||
Ok(op)
|
||||
}
|
||||
|
||||
/// Give a claimed op back to the queue, whatever became of the attempt.
|
||||
///
|
||||
/// Called on every path out of a drain attempt rather than only the failing
|
||||
/// ones: a handler that retires its own row leaves nothing to release (the
|
||||
/// `UPDATE` matches nothing), and a handler that returns without retiring
|
||||
/// *and* without failing would otherwise leave the row claimed by a worker
|
||||
/// that has moved on — invisible to every worker, forever.
|
||||
pub fn release_op_claim(&self, id: i64) -> Result<()> {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute(
|
||||
"UPDATE pending_op SET claimed_at = 0 WHERE id = ?1",
|
||||
params![id],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Drop every claim in the table.
|
||||
///
|
||||
/// A claim describes a worker in *this* process, so one surviving on disk
|
||||
/// means the previous run died holding it. The single-writer lock
|
||||
/// ([`Db::open`]) is what makes that inference safe: nobody else can be
|
||||
/// draining this queue while we are opening it.
|
||||
pub fn clear_op_claims(&self) -> Result<usize> {
|
||||
let conn = self.conn.lock();
|
||||
Ok(conn.execute(
|
||||
"UPDATE pending_op SET claimed_at = 0 WHERE claimed_at <> 0",
|
||||
[],
|
||||
)?)
|
||||
}
|
||||
|
||||
/// The earliest `next_attempt_at` among all queued ops, or `None` if the
|
||||
/// queue is empty. Used by the drain loop to sleep exactly until the next
|
||||
/// debounced or backed-off op becomes eligible rather than waiting the full
|
||||
/// idle-poll interval.
|
||||
///
|
||||
/// Claimed ops are excluded: another worker is already on them, so they are
|
||||
/// not work this one is waiting for, and counting them would have an idle
|
||||
/// worker spin on a row it cannot have.
|
||||
pub fn earliest_due_at(&self) -> Result<Option<i64>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let ts: Option<i64> = conn
|
||||
.query_row(
|
||||
&format!(
|
||||
"SELECT MIN(next_attempt_at) FROM pending_op \
|
||||
WHERE parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~'",
|
||||
WHERE claimed_at = 0 \
|
||||
AND (parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~')",
|
||||
v = LOCAL_VOLUME,
|
||||
n = LOCAL_VOLUME.len() + 1,
|
||||
),
|
||||
@@ -520,7 +635,7 @@ impl Db {
|
||||
/// Every queued op, oldest first. For status read-outs and the CLI — the
|
||||
/// drain wants [`next_due_op`](Self::next_due_op) instead.
|
||||
pub fn pending_ops(&self) -> Result<Vec<PendingOp>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT id, kind, uid, parent_uid, name, blob_path, meta_json, created_at,
|
||||
attempts, last_error, next_attempt_at
|
||||
@@ -553,7 +668,7 @@ impl Db {
|
||||
/// mount owes the server, but it carries no bytes and reporting it as an
|
||||
/// upload is simply untrue.
|
||||
pub fn pending_op_counts(&self) -> Result<PendingCounts> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let uploads = conn.query_row(
|
||||
"SELECT COUNT(*) FROM pending_op WHERE kind IN (?1, ?2)",
|
||||
params![OP_REVISION, OP_CREATE],
|
||||
@@ -564,7 +679,33 @@ impl Db {
|
||||
params![OP_REVISION, OP_CREATE],
|
||||
|r| r.get(0),
|
||||
)?;
|
||||
Ok(PendingCounts { uploads, changes })
|
||||
let parked = conn.query_row(
|
||||
"SELECT COUNT(*) FROM pending_op WHERE next_attempt_at >= ?1",
|
||||
params![PARK_UNTIL],
|
||||
|r| r.get(0),
|
||||
)?;
|
||||
let failing = conn.query_row(
|
||||
"SELECT COUNT(*) FROM pending_op WHERE attempts >= ?1",
|
||||
params![FAILING_ATTEMPTS],
|
||||
|r| r.get(0),
|
||||
)?;
|
||||
let last_error = conn
|
||||
.query_row(
|
||||
"SELECT last_error FROM pending_op
|
||||
WHERE attempts >= ?1 AND last_error IS NOT NULL
|
||||
ORDER BY attempts DESC LIMIT 1",
|
||||
params![FAILING_ATTEMPTS],
|
||||
|r| r.get::<_, Option<String>>(0),
|
||||
)
|
||||
.optional()?
|
||||
.flatten();
|
||||
Ok(PendingCounts {
|
||||
uploads,
|
||||
changes,
|
||||
parked,
|
||||
failing,
|
||||
last_error,
|
||||
})
|
||||
}
|
||||
|
||||
/// Nothing bearing this volume may be handed to the API — it would 404. The
|
||||
@@ -580,11 +721,16 @@ impl Db {
|
||||
|
||||
/// Record a failed attempt and when to next try. Leaves the row in place —
|
||||
/// the staged bytes are still the only copy of the user's write.
|
||||
///
|
||||
/// Clears any access-deferral window: whatever the op was waiting for has
|
||||
/// now been reported, so the next uncleared deferral starts its own window
|
||||
/// rather than escalating again on the following recheck.
|
||||
pub fn record_op_failure(&self, id: i64, error: &str, next_attempt_at: i64) -> Result<()> {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute(
|
||||
"UPDATE pending_op
|
||||
SET attempts = attempts + 1, last_error = ?2, next_attempt_at = ?3
|
||||
SET attempts = attempts + 1, last_error = ?2, next_attempt_at = ?3,
|
||||
access_deferred_since = 0
|
||||
WHERE id = ?1",
|
||||
params![id, error, next_attempt_at],
|
||||
)?;
|
||||
@@ -609,4 +755,132 @@ impl Db {
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Defer an op the local access check refused, and report how long it has
|
||||
/// been refused for.
|
||||
///
|
||||
/// Identical to [`defer_op_without_attempt`](Self::defer_op_without_attempt)
|
||||
/// on the retry clock, plus the one thing that was missing: the first
|
||||
/// deferral of a run stamps `access_deferred_since`, and every deferral
|
||||
/// returns that stamp. A caller can therefore tell a deferral that is one
|
||||
/// recheck old from one that has been repeating unchanged for days, which is
|
||||
/// the difference between a permission the user is mid-way through changing
|
||||
/// and a queue that will never drain.
|
||||
///
|
||||
/// Returns the ms at which the current run of deferrals began — `now` on the
|
||||
/// first one.
|
||||
pub fn defer_op_for_access(&self, id: i64, now: i64, next_attempt_at: i64) -> Result<i64> {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute(
|
||||
"UPDATE pending_op
|
||||
SET next_attempt_at = CASE
|
||||
WHEN next_attempt_at >= ?4 THEN next_attempt_at
|
||||
ELSE ?3
|
||||
END,
|
||||
access_deferred_since = CASE
|
||||
WHEN access_deferred_since = 0 THEN ?2
|
||||
ELSE access_deferred_since
|
||||
END
|
||||
WHERE id = ?1",
|
||||
params![id, now, next_attempt_at, PARK_UNTIL],
|
||||
)?;
|
||||
let since = conn
|
||||
.query_row(
|
||||
"SELECT access_deferred_since FROM pending_op WHERE id = ?1",
|
||||
params![id],
|
||||
|r| r.get::<_, i64>(0),
|
||||
)
|
||||
.optional()?;
|
||||
// A row that drained out from under this call leaves nothing to measure;
|
||||
// reporting `now` says "this run just started", which is the reading that
|
||||
// never escalates something that no longer exists.
|
||||
Ok(since.filter(|since| *since != 0).unwrap_or(now))
|
||||
}
|
||||
|
||||
/// Forget an op's access-deferral window, after an attempt the access check
|
||||
/// let through. The op is no longer blocked, so a later deferral is the
|
||||
/// start of a new run rather than the continuation of an old one.
|
||||
pub fn clear_op_access_deferral(&self, id: i64) -> Result<()> {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute(
|
||||
"UPDATE pending_op SET access_deferred_since = 0
|
||||
WHERE id = ?1 AND access_deferred_since <> 0",
|
||||
params![id],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Every staged blob the queue still refers to.
|
||||
///
|
||||
/// The startup staging reconcile subtracts this from what is actually in
|
||||
/// `staging/`: the difference is bytes the kernel was told were written and
|
||||
/// that nothing is going to upload.
|
||||
pub fn op_blob_paths(&self) -> Result<std::collections::HashSet<String>> {
|
||||
let conn = self.read();
|
||||
let mut stmt =
|
||||
conn.prepare("SELECT blob_path FROM pending_op WHERE blob_path IS NOT NULL")?;
|
||||
let rows = stmt.query_map([], |r| r.get::<_, String>(0))?;
|
||||
let mut out = std::collections::HashSet::new();
|
||||
for row in rows {
|
||||
out.insert(row?);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Record that this daemon sealed `revision_id` on `uid`, and prune entries
|
||||
/// past [`OWN_SEALED_TTL_MS`].
|
||||
///
|
||||
/// One row per node, overwritten as the node advances. See
|
||||
/// [`own_sealed_rev`](Self::own_sealed_rev) for what reads it.
|
||||
pub fn set_own_sealed_rev(&self, uid: &str, revision_id: &str, now_ms: i64) -> Result<()> {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute(
|
||||
"INSERT INTO own_sealed_rev (uid, revision_id, sealed_at)
|
||||
VALUES (?1, ?2, ?3)
|
||||
ON CONFLICT(uid) DO UPDATE SET
|
||||
revision_id = excluded.revision_id,
|
||||
sealed_at = excluded.sealed_at",
|
||||
params![uid, revision_id, now_ms],
|
||||
)?;
|
||||
conn.execute(
|
||||
"DELETE FROM own_sealed_rev WHERE sealed_at < ?1",
|
||||
params![now_ms - OWN_SEALED_TTL_MS],
|
||||
)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The last revision of `uid` this daemon sealed itself, if it is still
|
||||
/// within [`OWN_SEALED_TTL_MS`].
|
||||
///
|
||||
/// The drain asks before treating a moved-on remote as a conflict: a
|
||||
/// revision we sealed means one writer stalled and resumed, which
|
||||
/// supersedes rather than forking a `(sync-conflict)` copy (docs/BUGS.md
|
||||
/// B70 layer B).
|
||||
pub fn own_sealed_rev(&self, uid: &str, now_ms: i64) -> Result<Option<String>> {
|
||||
let conn = self.read();
|
||||
Ok(conn
|
||||
.query_row(
|
||||
"SELECT revision_id FROM own_sealed_rev
|
||||
WHERE uid = ?1 AND sealed_at >= ?2",
|
||||
params![uid, now_ms - OWN_SEALED_TTL_MS],
|
||||
|r| r.get(0),
|
||||
)
|
||||
.optional()?)
|
||||
}
|
||||
|
||||
/// Forget what we sealed on `uid` — it has been trashed, so there is no
|
||||
/// revision left to chain onto.
|
||||
pub fn clear_own_sealed_rev(&self, uid: &str) -> Result<()> {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute("DELETE FROM own_sealed_rev WHERE uid = ?1", params![uid])?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// How long a revision this daemon sealed stays recognisable as its own.
|
||||
///
|
||||
/// Bounds a table that would otherwise grow with every write for the life of
|
||||
/// the install. A week is far longer than any stall→resume — a browser download
|
||||
/// that pauses and continues, an editor that reopens its handle — and short
|
||||
/// enough that the table tracks recent work rather than accumulating history.
|
||||
pub const OWN_SEALED_TTL_MS: i64 = 7 * 24 * 60 * 60 * 1000;
|
||||
@@ -115,7 +115,7 @@ impl Db {
|
||||
range: Option<(i64, i64)>,
|
||||
favorites: bool,
|
||||
) -> Result<Vec<StoredPhoto>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
// Built up so any combination of the optional filters is one indexed
|
||||
// query rather than a statement per case.
|
||||
let mut sql = String::from(
|
||||
@@ -165,6 +165,31 @@ impl Db {
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
/// The whole persisted timeline, in stored order. Only for whole-library
|
||||
/// passes (the photo re-date) — the gallery pages with
|
||||
/// [`Db::photos_page`].
|
||||
pub fn photos_all(&self) -> Result<Vec<StoredPhoto>> {
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT uid, capture_time, name, ratio, thumb_state, kind, favorite \
|
||||
FROM photos ORDER BY seq",
|
||||
)?;
|
||||
let rows = stmt
|
||||
.query_map([], |r| {
|
||||
Ok(StoredPhoto {
|
||||
uid: r.get(0)?,
|
||||
capture_time: r.get(1)?,
|
||||
name: r.get(2)?,
|
||||
ratio: r.get(3)?,
|
||||
thumb_state: r.get(4)?,
|
||||
kind: crate::control::PhotoKind::from_i64(r.get(5)?),
|
||||
favorite: r.get::<_, i64>(6)? != 0,
|
||||
})
|
||||
})?
|
||||
.collect::<rusqlite::Result<_>>()?;
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
/// The months the timeline spans, newest first, each with how many photos it
|
||||
/// holds — the data behind the date scrubber. Buckets are local-time
|
||||
/// `(year, month)` so they line up with the day headings the gallery draws
|
||||
@@ -174,7 +199,7 @@ impl Db {
|
||||
&self,
|
||||
kind: Option<crate::control::PhotoKind>,
|
||||
) -> Result<Vec<(i32, i32, usize)>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let (filter, binds): (&str, Vec<i64>) = match kind {
|
||||
Some(k) => (" WHERE kind = ?1", vec![k.as_i64()]),
|
||||
None => ("", Vec::new()),
|
||||
@@ -201,7 +226,7 @@ impl Db {
|
||||
/// Per-tab counts for the Photos page subtitle: `(photos, videos, raw)`.
|
||||
pub fn photos_counts(&self) -> Result<(usize, usize, usize)> {
|
||||
use crate::control::PhotoKind;
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare("SELECT kind, COUNT(*) FROM photos GROUP BY kind")?;
|
||||
let rows = stmt.query_map([], |r| Ok((r.get::<_, i64>(0)?, r.get::<_, i64>(1)?)))?;
|
||||
let (mut photos, mut videos, mut raw) = (0usize, 0usize, 0usize);
|
||||
@@ -223,7 +248,7 @@ impl Db {
|
||||
if uids.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let placeholders = vec!["?"; uids.len()].join(",");
|
||||
let mut stmt = conn.prepare(&format!(
|
||||
"SELECT uid, capture_time, name, ratio, thumb_state, kind, favorite FROM photos
|
||||
@@ -273,7 +298,7 @@ impl Db {
|
||||
|
||||
/// Number of photos in the persisted timeline.
|
||||
pub fn photos_count(&self) -> Result<usize> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let n: i64 = conn.query_row("SELECT COUNT(*) FROM photos", [], |r| r.get(0))?;
|
||||
Ok(n.max(0) as usize)
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
use rusqlite::{OptionalExtension, params};
|
||||
|
||||
use super::Db;
|
||||
use super::utils::MAX_PATH_DEPTH;
|
||||
use crate::Result;
|
||||
|
||||
/// One row of [`Db::pin_list`]. `recursive` is pin *policy* — whether the whole
|
||||
@@ -39,7 +40,7 @@ impl Db {
|
||||
|
||||
/// Every directly-pinned entry, ordered by uid.
|
||||
pub fn pin_list(&self) -> Result<Vec<PinRow>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
// `recursive` is pin policy; the node's kind comes from `nodes`, and is
|
||||
// NULL for a pin whose node was never cached.
|
||||
let mut stmt = conn.prepare(
|
||||
@@ -66,8 +67,13 @@ impl Db {
|
||||
/// is pinned recursively. The ancestor check walks `parent_uid` to the root
|
||||
/// via a CTE; a direct pin is honoured even when the node has no `nodes` row
|
||||
/// yet (e.g. a CLI that never hydrates the node cache).
|
||||
///
|
||||
/// The walk is depth-capped like [`super::utils::path_of`]: a `parent_uid`
|
||||
/// cycle is corrupt data the API can hand us, and `UNION ALL` over one never
|
||||
/// terminates — while holding the daemon's only SQLite connection, on a path
|
||||
/// that runs per cached read.
|
||||
pub fn is_pinned(&self, uid: &str) -> Result<bool> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let direct: Option<i64> = conn
|
||||
.query_row("SELECT 1 FROM pins WHERE uid = ?1", params![uid], |r| {
|
||||
r.get(0)
|
||||
@@ -78,14 +84,18 @@ impl Db {
|
||||
}
|
||||
let anc: Option<i64> = conn
|
||||
.query_row(
|
||||
"WITH RECURSIVE anc(uid, parent_uid) AS (
|
||||
SELECT uid, parent_uid FROM nodes WHERE uid = ?1
|
||||
UNION ALL
|
||||
SELECT n.uid, n.parent_uid FROM nodes n JOIN anc ON n.uid = anc.parent_uid
|
||||
)
|
||||
SELECT 1 FROM anc a JOIN pins p ON p.uid = a.uid
|
||||
WHERE a.uid != ?1 AND p.recursive = 1
|
||||
LIMIT 1",
|
||||
&format!(
|
||||
"WITH RECURSIVE anc(uid, parent_uid, depth) AS (
|
||||
SELECT uid, parent_uid, 0 FROM nodes WHERE uid = ?1
|
||||
UNION ALL
|
||||
SELECT n.uid, n.parent_uid, anc.depth + 1
|
||||
FROM nodes n JOIN anc ON n.uid = anc.parent_uid
|
||||
WHERE anc.depth < {MAX_PATH_DEPTH}
|
||||
)
|
||||
SELECT 1 FROM anc a JOIN pins p ON p.uid = a.uid
|
||||
WHERE a.uid != ?1 AND p.recursive = 1
|
||||
LIMIT 1"
|
||||
),
|
||||
params![uid],
|
||||
|r| r.get(0),
|
||||
)
|
||||
@@ -97,7 +107,7 @@ impl Db {
|
||||
/// every descendant of a recursively-pinned folder. Used to build the
|
||||
/// eviction-exempt set (the budget enforcer hashes these into cache keys).
|
||||
pub fn pinned_uids(&self) -> Result<Vec<String>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"WITH RECURSIVE sub(uid) AS (
|
||||
SELECT uid FROM pins WHERE recursive = 1
|
||||
@@ -119,7 +129,7 @@ impl Db {
|
||||
/// Every descendant uid of `folder` (all depths), via a `parent_uid` CTE.
|
||||
/// Used to evict a recursively-pinned subtree's cached blobs on unpin.
|
||||
pub fn descendants(&self, folder: &str) -> Result<Vec<String>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"WITH RECURSIVE sub(uid) AS (
|
||||
SELECT uid FROM nodes WHERE parent_uid = ?1
|
||||
|
||||
@@ -16,7 +16,7 @@ impl Db {
|
||||
/// remains the restart/offline authority and every explicit role change is
|
||||
/// written through before this cache is updated.
|
||||
pub fn all_share_access(&self) -> Result<HashMap<NodeUid, Access>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare("SELECT root_uid, access FROM share_access")?;
|
||||
let rows = stmt.query_map([], |row| {
|
||||
Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?))
|
||||
@@ -42,7 +42,7 @@ impl Db {
|
||||
|
||||
/// Read the effective access recorded for a shared-tree root.
|
||||
pub fn share_access(&self, uid: &NodeUid) -> Result<Option<Access>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let value = conn
|
||||
.query_row(
|
||||
"SELECT access FROM share_access WHERE root_uid = ?1",
|
||||
@@ -96,7 +96,7 @@ impl Db {
|
||||
/// share root. `None` means the node itself is not persisted, which callers
|
||||
/// must not treat as owned: stale handles need to fail closed.
|
||||
pub fn effective_node_access(&self, uid: &NodeUid) -> Result<Option<Access>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let exists: bool = conn.query_row(
|
||||
"SELECT EXISTS(SELECT 1 FROM nodes WHERE uid = ?1)",
|
||||
[uid.to_string()],
|
||||
|
||||
@@ -8,7 +8,7 @@ use crate::Result;
|
||||
|
||||
impl Db {
|
||||
pub fn get_event_cursor(&self) -> Result<Option<String>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let v = conn
|
||||
.query_row(
|
||||
"SELECT value FROM sync_state WHERE key = 'event_cursor'",
|
||||
@@ -32,7 +32,7 @@ impl Db {
|
||||
|
||||
/// Read a `sync_state` value as a string.
|
||||
pub fn state_str(&self, key: &str) -> Result<Option<String>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let v = conn
|
||||
.query_row(
|
||||
"SELECT value FROM sync_state WHERE key = ?1",
|
||||
@@ -66,7 +66,7 @@ impl Db {
|
||||
/// dropping it would leave a file that exists nowhere but this machine with
|
||||
/// nothing left to create it. (Writes to a node that is itself still queued
|
||||
pub fn state_i64(&self, key: &str) -> Result<Option<i64>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let v: Option<String> = conn
|
||||
.query_row(
|
||||
"SELECT value FROM sync_state WHERE key = ?1",
|
||||
|
||||
@@ -35,6 +35,14 @@ pub struct StoredSyncEntry {
|
||||
pub remote_uid: Option<String>,
|
||||
pub local_mtime: i64,
|
||||
pub local_size: i64,
|
||||
/// The same modification time to nanosecond precision, when it is known.
|
||||
///
|
||||
/// `local_mtime` alone cannot tell an edit that landed inside the same
|
||||
/// second as the last sync — and left the file the same length — from no
|
||||
/// edit at all, so those never uploaded (bugs.md B25). `None` on a row
|
||||
/// written before schema 28, which the comparison reads as "compare whole
|
||||
/// seconds", so upgrading does not make an entire mirror look changed.
|
||||
pub local_mtime_ns: Option<i64>,
|
||||
pub remote_rev: Option<String>,
|
||||
pub remote_hash: Option<String>,
|
||||
}
|
||||
@@ -57,7 +65,7 @@ impl Db {
|
||||
|
||||
/// Every synced folder, oldest first.
|
||||
pub fn sync_folder_list(&self) -> Result<Vec<StoredSyncFolder>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT id, local_path, remote_uid, remote_share_id, mode, pending_mode, state, last_sync
|
||||
FROM sync_folder ORDER BY id",
|
||||
@@ -83,7 +91,7 @@ impl Db {
|
||||
|
||||
/// Look up one synced folder by id.
|
||||
pub fn sync_folder_get(&self, id: i64) -> Result<Option<StoredSyncFolder>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
conn.query_row(
|
||||
"SELECT id, local_path, remote_uid, remote_share_id, mode, pending_mode, state, last_sync
|
||||
FROM sync_folder WHERE id = ?1",
|
||||
@@ -166,9 +174,10 @@ impl Db {
|
||||
|
||||
/// The whole per-file sync baseline for a folder, keyed by relative path.
|
||||
pub fn sync_entries(&self, folder_id: i64) -> Result<HashMap<String, StoredSyncEntry>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash
|
||||
"SELECT rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash,
|
||||
local_mtime_ns
|
||||
FROM sync_entry WHERE folder_id = ?1",
|
||||
)?;
|
||||
let rows = stmt.query_map(params![folder_id], |r| {
|
||||
@@ -179,6 +188,7 @@ impl Db {
|
||||
local_size: r.get(3)?,
|
||||
remote_rev: r.get(4)?,
|
||||
remote_hash: r.get(5)?,
|
||||
local_mtime_ns: r.get(6)?,
|
||||
})
|
||||
})?;
|
||||
let mut map = HashMap::new();
|
||||
@@ -196,7 +206,7 @@ impl Db {
|
||||
/// `sync_folder` and descendants come from the last successful
|
||||
/// `sync_entry` baseline.
|
||||
pub fn mirror_contains_uid(&self, uid: &str) -> Result<bool> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
conn.query_row(
|
||||
"SELECT EXISTS(
|
||||
SELECT 1
|
||||
@@ -216,14 +226,16 @@ impl Db {
|
||||
let conn = self.conn.lock();
|
||||
conn.execute(
|
||||
"INSERT INTO sync_entry
|
||||
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev, remote_hash)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
|
||||
(folder_id, rel_path, remote_uid, local_mtime, local_size, remote_rev,
|
||||
remote_hash, local_mtime_ns)
|
||||
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)
|
||||
ON CONFLICT(folder_id, rel_path) DO UPDATE SET
|
||||
remote_uid = excluded.remote_uid,
|
||||
local_mtime = excluded.local_mtime,
|
||||
local_size = excluded.local_size,
|
||||
remote_rev = excluded.remote_rev,
|
||||
remote_hash = excluded.remote_hash",
|
||||
remote_uid = excluded.remote_uid,
|
||||
local_mtime = excluded.local_mtime,
|
||||
local_size = excluded.local_size,
|
||||
remote_rev = excluded.remote_rev,
|
||||
remote_hash = excluded.remote_hash,
|
||||
local_mtime_ns = excluded.local_mtime_ns",
|
||||
params![
|
||||
folder_id,
|
||||
e.rel_path,
|
||||
@@ -232,6 +244,7 @@ impl Db {
|
||||
e.local_size,
|
||||
e.remote_rev,
|
||||
e.remote_hash,
|
||||
e.local_mtime_ns,
|
||||
],
|
||||
)?;
|
||||
Ok(())
|
||||
|
||||
@@ -867,6 +867,67 @@ fn shared_root_pinned_name_survives_database_reopen() {
|
||||
let _ = std::fs::remove_file(path);
|
||||
}
|
||||
|
||||
/// Two writers on one database is the failure mode the single-instance lock
|
||||
/// exists to prevent — a hand-run `pdfs mount` next to the systemd unit gives
|
||||
/// two daemons the same inode space, content cache, and drain queue. The second
|
||||
/// open has to fail, and it has to stop failing once the first handle is gone.
|
||||
#[test]
|
||||
fn a_second_writer_is_refused_while_the_first_holds_the_database() {
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"pdfs-single-writer-{}-{}.db",
|
||||
std::process::id(),
|
||||
now_test_id()
|
||||
));
|
||||
let first = Db::open(&path).unwrap();
|
||||
let second = Db::open(&path);
|
||||
assert!(second.is_err(), "a second writer was allowed in");
|
||||
|
||||
drop(first);
|
||||
// The lock lives in the kernel and dies with the descriptor, so nothing has
|
||||
// to be cleaned up for the next daemon to start.
|
||||
Db::open(&path).expect("reopen after the first handle is dropped");
|
||||
|
||||
let _ = std::fs::remove_file(&path);
|
||||
let _ = std::fs::remove_file(path.with_extension("db.lock"));
|
||||
}
|
||||
|
||||
/// `cache.db` is derived state — every row can be re-fetched or re-scanned — so
|
||||
/// a file that is not a database at all must not stop the daemon starting. It
|
||||
/// is moved aside (not deleted: it is the only evidence) and rebuilt.
|
||||
#[test]
|
||||
fn a_corrupt_database_is_moved_aside_and_rebuilt() {
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"pdfs-corrupt-{}-{}.db",
|
||||
std::process::id(),
|
||||
now_test_id()
|
||||
));
|
||||
std::fs::write(&path, b"this is not an SQLite database").unwrap();
|
||||
|
||||
let db = Db::open(&path).unwrap();
|
||||
db.set_state_str("shared_with_me_name", "rebuilt").unwrap();
|
||||
assert_eq!(
|
||||
db.state_str("shared_with_me_name").unwrap().as_deref(),
|
||||
Some("rebuilt")
|
||||
);
|
||||
drop(db);
|
||||
|
||||
let dir = path.parent().unwrap();
|
||||
let stem = path.file_name().unwrap().to_string_lossy().to_string();
|
||||
let quarantined: Vec<_> = std::fs::read_dir(dir)
|
||||
.unwrap()
|
||||
.filter_map(|e| e.ok())
|
||||
.map(|e| e.file_name().to_string_lossy().to_string())
|
||||
.filter(|name| name.starts_with(&format!("{stem}.corrupt-")))
|
||||
.collect();
|
||||
assert_eq!(quarantined.len(), 1, "the damaged file was not kept");
|
||||
|
||||
for name in quarantined {
|
||||
let _ = std::fs::remove_file(dir.join(name));
|
||||
}
|
||||
let _ = std::fs::remove_file(&path);
|
||||
let _ = std::fs::remove_file(path.with_extension("db.lock"));
|
||||
}
|
||||
|
||||
fn local(path: &str, name: &str, is_dir: bool) -> LocalEntry {
|
||||
LocalEntry {
|
||||
path: path.into(),
|
||||
@@ -911,6 +972,25 @@ fn local_scan_indexes_then_prunes_stale_paths() {
|
||||
assert_eq!(db.search_local("notes", 10).unwrap().len(), 1);
|
||||
}
|
||||
|
||||
/// The index is maintained incrementally rather than rebuilt at the end of a
|
||||
/// scan, so the case that used to be covered for free — a file the scan sees
|
||||
/// again — needs its own test: a re-scan must neither lose the entry nor
|
||||
/// duplicate it.
|
||||
#[test]
|
||||
fn rescanning_an_unchanged_file_keeps_exactly_one_index_entry() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
for (generation, mtime) in [(1_000, 5), (2_000, 9)] {
|
||||
let g = db.local_begin_scan().unwrap();
|
||||
let mut e = local("/home/u/docs/report.pdf", "report.pdf", false);
|
||||
e.mtime = mtime;
|
||||
db.local_upsert_batch(g, &[e]).unwrap();
|
||||
assert_eq!(db.local_finish_scan(g, generation).unwrap(), 1);
|
||||
let hits = db.search_local("report", 10).unwrap();
|
||||
assert_eq!(hits.len(), 1, "generation {generation}");
|
||||
assert_eq!(hits[0].mtime, mtime);
|
||||
}
|
||||
}
|
||||
|
||||
/// Queries below the trigram minimum still match, via the `LIKE` fallback.
|
||||
#[test]
|
||||
fn local_search_short_query_like_fallback() {
|
||||
@@ -1429,6 +1509,7 @@ fn migration_v17_adds_share_access_to_a_v16_fixture() {
|
||||
DROP TABLE albums;
|
||||
DROP INDEX idx_photos_favorite;
|
||||
ALTER TABLE photos DROP COLUMN favorite;
|
||||
ALTER TABLE nodes DROP COLUMN path;
|
||||
UPDATE sync_state SET value = '16' WHERE key = 'schema_version';",
|
||||
)
|
||||
.unwrap();
|
||||
@@ -1814,6 +1895,123 @@ fn failed_superseding_insert_keeps_the_old_pending_op() {
|
||||
assert_eq!(ops[0].blob_path.as_deref(), Some("/staging/original"));
|
||||
}
|
||||
|
||||
/// B53's remaining half. The insert-failure regression above proves the *row*
|
||||
/// rolls back; this proves the two things that make the rollback usable — the
|
||||
/// old blob is not reported as superseded (so the caller does not delete the
|
||||
/// bytes the surviving row still points at), and the rollback survives a
|
||||
/// restart rather than living only in the connection that failed.
|
||||
#[test]
|
||||
fn a_rolled_back_supersede_keeps_its_blob_and_survives_reopen() {
|
||||
let dir = std::env::temp_dir().join(format!("pdfs-db-b53-{}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let path = dir.join("queue.sqlite");
|
||||
let op = |blob: &str| PendingOp {
|
||||
id: 0,
|
||||
kind: OP_REVISION.to_string(),
|
||||
uid: uid("durable").to_string(),
|
||||
parent_uid: None,
|
||||
name: None,
|
||||
blob_path: Some(blob.to_string()),
|
||||
meta_json: Some("{}".to_string()),
|
||||
created_at: 1,
|
||||
attempts: 0,
|
||||
last_error: None,
|
||||
next_attempt_at: 0,
|
||||
};
|
||||
|
||||
{
|
||||
let db = Db::open(&path).unwrap();
|
||||
db.enqueue_op(&op("/staging/original")).unwrap();
|
||||
db.with_conn(|conn| {
|
||||
conn.execute_batch(
|
||||
"CREATE TRIGGER reject_replacement BEFORE INSERT ON pending_op
|
||||
WHEN NEW.blob_path = '/staging/replacement'
|
||||
BEGIN SELECT RAISE(ABORT, 'injected insert failure'); END;",
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
|
||||
// The failure must not hand back the old blob path. That value is the
|
||||
// caller's instruction to delete those bytes, and the row that owns
|
||||
// them is still queued.
|
||||
assert!(db.enqueue_op(&op("/staging/replacement")).is_err());
|
||||
db.with_conn(|conn| {
|
||||
conn.execute_batch("DROP TRIGGER reject_replacement")?;
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let db = Db::open(&path).unwrap();
|
||||
let ops = db.pending_ops().unwrap();
|
||||
assert_eq!(ops.len(), 1, "old or new — never neither");
|
||||
assert_eq!(ops[0].blob_path.as_deref(), Some("/staging/original"));
|
||||
|
||||
// And once the injected failure is gone, superseding works and *does*
|
||||
// report the blob it retired.
|
||||
let (_, superseded) = db.enqueue_op(&op("/staging/replacement")).unwrap();
|
||||
assert_eq!(superseded.as_deref(), Some("/staging/original"));
|
||||
assert_eq!(db.pending_ops().unwrap().len(), 1);
|
||||
|
||||
drop(db);
|
||||
std::fs::remove_dir_all(&dir).unwrap();
|
||||
}
|
||||
|
||||
/// A full disk, as SQLite reports it. `max_page_count` makes the database
|
||||
/// refuse to grow, which is the same `SQLITE_FULL` an out-of-space filesystem
|
||||
/// produces — and the queue must come out of it with the acknowledged work
|
||||
/// still in it (B53).
|
||||
#[test]
|
||||
fn a_full_database_rolls_the_supersede_back_rather_than_erasing_it() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
let op = |blob: &str| PendingOp {
|
||||
id: 0,
|
||||
kind: OP_REVISION.to_string(),
|
||||
uid: uid("full").to_string(),
|
||||
parent_uid: None,
|
||||
name: None,
|
||||
blob_path: Some(blob.to_string()),
|
||||
meta_json: Some("{}".to_string()),
|
||||
created_at: 1,
|
||||
attempts: 0,
|
||||
last_error: None,
|
||||
next_attempt_at: 0,
|
||||
};
|
||||
db.enqueue_op(&op("/staging/original")).unwrap();
|
||||
|
||||
// Cap the file at its current size, so any page the insert needs fails.
|
||||
let pages: i64 = db
|
||||
.with_conn(|conn| Ok(conn.query_row("PRAGMA page_count", [], |r| r.get(0))?))
|
||||
.unwrap();
|
||||
db.with_conn(|conn| {
|
||||
conn.execute_batch(&format!("PRAGMA max_page_count = {pages}"))?;
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
|
||||
// A payload far larger than one page, so growth is unavoidable.
|
||||
let mut fat = op("/staging/replacement");
|
||||
fat.meta_json = Some("x".repeat(512 * 1024));
|
||||
let result = db.enqueue_op(&fat);
|
||||
|
||||
db.with_conn(|conn| {
|
||||
conn.execute_batch("PRAGMA max_page_count = 1073741823")?;
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
|
||||
assert!(result.is_err(), "a database that cannot grow must refuse");
|
||||
let ops = db.pending_ops().unwrap();
|
||||
assert_eq!(ops.len(), 1);
|
||||
assert_eq!(
|
||||
ops[0].blob_path.as_deref(),
|
||||
Some("/staging/original"),
|
||||
"the acknowledged upload is still queued and still owns its bytes"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn atomic_trash_replacement_returns_owned_blobs_after_commit() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
@@ -2327,6 +2525,7 @@ fn mirror_uid_lookup_covers_the_root_and_synced_descendants_only() {
|
||||
local_size: 2,
|
||||
remote_rev: Some("3".into()),
|
||||
remote_hash: Some("2".into()),
|
||||
local_mtime_ns: None,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
@@ -2343,6 +2542,7 @@ fn mirror_uid_lookup_covers_the_root_and_synced_descendants_only() {
|
||||
local_size: 2,
|
||||
remote_rev: Some("3".into()),
|
||||
remote_hash: Some("2".into()),
|
||||
local_mtime_ns: None,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
@@ -2628,6 +2828,26 @@ fn non_recursive_folder_pin_does_not_cover_children() {
|
||||
assert!(!db.is_pinned(&du("rep")).unwrap());
|
||||
}
|
||||
|
||||
/// A `parent_uid` cycle is corrupt data the API can hand us, and `is_pinned`'s
|
||||
/// ancestor walk is `UNION ALL` — uncapped it never terminates, while holding
|
||||
/// the daemon's only SQLite connection, on a path that runs per cached read.
|
||||
/// The answer here matters less than the fact that there is one.
|
||||
#[test]
|
||||
fn is_pinned_terminates_on_a_parent_cycle() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
let du = |l: &str| uid(l).to_string();
|
||||
// a -> b -> a, with nothing pinned above either.
|
||||
db.upsert_node(&folder("a", Some("b"), "A")).unwrap();
|
||||
db.upsert_node(&folder("b", Some("a"), "B")).unwrap();
|
||||
db.upsert_node(&file("leaf", "a", "leaf.txt", 1)).unwrap();
|
||||
|
||||
assert!(!db.is_pinned(&du("leaf")).unwrap());
|
||||
|
||||
// And a pin *inside* the cycle is still found rather than walked past.
|
||||
db.pin_add(&du("a"), "A", true).unwrap();
|
||||
assert!(db.is_pinned(&du("leaf")).unwrap());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn schema_objects_exist() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
@@ -2981,6 +3201,8 @@ fn test_db_has_children_and_trashed_filtering() {
|
||||
photo: None,
|
||||
album: None,
|
||||
verification: Default::default(),
|
||||
direct_role: None,
|
||||
share_id: None,
|
||||
};
|
||||
db.upsert_node(&parent_node).unwrap();
|
||||
|
||||
@@ -3007,6 +3229,8 @@ fn test_db_has_children_and_trashed_filtering() {
|
||||
photo: None,
|
||||
album: None,
|
||||
verification: Default::default(),
|
||||
direct_role: None,
|
||||
share_id: None,
|
||||
};
|
||||
db.upsert_node(&child_node).unwrap();
|
||||
|
||||
@@ -3182,3 +3406,502 @@ fn clear_state_prefix_takes_only_the_prefixed_keys() {
|
||||
"the listing's own stamp is not a per-album one"
|
||||
);
|
||||
}
|
||||
|
||||
// --- `nodes.path`, and the subtree work it replaces (DB3) -------------------
|
||||
|
||||
/// Read the stored path column directly: the point of these tests is that the
|
||||
/// column itself is right, not that `path_of` can still fall back to a walk.
|
||||
fn stored_path(db: &Db, link: &str) -> Option<String> {
|
||||
use rusqlite::OptionalExtension;
|
||||
db.with_conn(|conn| {
|
||||
Ok(conn
|
||||
.query_row(
|
||||
"SELECT path FROM nodes WHERE uid = ?1",
|
||||
[uid(link).to_string()],
|
||||
|row| row.get::<_, Option<String>>(0),
|
||||
)
|
||||
.optional()?
|
||||
.flatten())
|
||||
})
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_node_stores_its_path_when_it_is_written() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
db.upsert_node(&folder("root", None, "My Files")).unwrap();
|
||||
db.upsert_node(&folder("docs", Some("root"), "Documents"))
|
||||
.unwrap();
|
||||
db.upsert_node(&file("f1", "docs", "report.pdf", 1))
|
||||
.unwrap();
|
||||
|
||||
// The root's name is the mount, so it contributes nothing.
|
||||
assert_eq!(stored_path(&db, "root").as_deref(), Some(""));
|
||||
assert_eq!(stored_path(&db, "docs").as_deref(), Some("Documents"));
|
||||
assert_eq!(
|
||||
stored_path(&db, "f1").as_deref(),
|
||||
Some("Documents/report.pdf")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_node_below_an_uncached_parent_starts_its_own_path() {
|
||||
// A device folder's root lives in `device`/`sync_folder` and never gets a
|
||||
// `nodes` row, so its children are the top of the cached chain.
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
db.upsert_node(&folder("orphan", Some("never-cached"), "Work"))
|
||||
.unwrap();
|
||||
db.upsert_node(&file("f1", "orphan", "notes.txt", 1))
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(stored_path(&db, "orphan").as_deref(), Some("Work"));
|
||||
assert_eq!(stored_path(&db, "f1").as_deref(), Some("Work/notes.txt"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn renaming_a_folder_rewrites_every_descendant_path() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
db.upsert_node(&folder("root", None, "My Files")).unwrap();
|
||||
db.upsert_node(&folder("a", Some("root"), "Before"))
|
||||
.unwrap();
|
||||
db.upsert_node(&folder("b", Some("a"), "Inner")).unwrap();
|
||||
db.upsert_node(&file("f1", "b", "deep.txt", 1)).unwrap();
|
||||
assert_eq!(
|
||||
stored_path(&db, "f1").as_deref(),
|
||||
Some("Before/Inner/deep.txt")
|
||||
);
|
||||
|
||||
db.upsert_node(&folder("a", Some("root"), "After")).unwrap();
|
||||
|
||||
assert_eq!(stored_path(&db, "a").as_deref(), Some("After"));
|
||||
assert_eq!(stored_path(&db, "b").as_deref(), Some("After/Inner"));
|
||||
assert_eq!(
|
||||
stored_path(&db, "f1").as_deref(),
|
||||
Some("After/Inner/deep.txt")
|
||||
);
|
||||
let hit = &db.search("deep.txt", 10).unwrap()[0];
|
||||
assert_eq!(
|
||||
hit.path, "After/Inner/deep.txt",
|
||||
"the search index must carry the rewritten path too"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn moving_a_folder_rewrites_every_descendant_path() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
db.upsert_node(&folder("root", None, "My Files")).unwrap();
|
||||
db.upsert_node(&folder("old", Some("root"), "Old")).unwrap();
|
||||
db.upsert_node(&folder("new", Some("root"), "New")).unwrap();
|
||||
db.upsert_node(&folder("moved", Some("old"), "Moved"))
|
||||
.unwrap();
|
||||
db.upsert_node(&file("f1", "moved", "deep.txt", 1)).unwrap();
|
||||
|
||||
db.upsert_node(&folder("moved", Some("new"), "Moved"))
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(stored_path(&db, "moved").as_deref(), Some("New/Moved"));
|
||||
assert_eq!(
|
||||
stored_path(&db, "f1").as_deref(),
|
||||
Some("New/Moved/deep.txt")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn trashing_a_folder_drops_its_subtree_from_the_index() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
db.upsert_node(&folder("root", None, "My Files")).unwrap();
|
||||
db.upsert_node(&folder("a", Some("root"), "Holder"))
|
||||
.unwrap();
|
||||
db.upsert_node(&file("f1", "a", "hidden.txt", 1)).unwrap();
|
||||
assert_eq!(db.search("hidden", 10).unwrap().len(), 1);
|
||||
|
||||
let mut trashed = folder("a", Some("root"), "Holder");
|
||||
trashed.trashed = true;
|
||||
db.upsert_node(&trashed).unwrap();
|
||||
|
||||
assert_eq!(
|
||||
db.search("hidden", 10).unwrap().len(),
|
||||
0,
|
||||
"a file under a trashed folder is not reachable and must not be findable"
|
||||
);
|
||||
|
||||
// …and comes back when the folder does.
|
||||
db.upsert_node(&folder("a", Some("root"), "Holder"))
|
||||
.unwrap();
|
||||
assert_eq!(db.search("hidden", 10).unwrap().len(), 1);
|
||||
}
|
||||
|
||||
/// The re-listing case, which is most of what the write path actually does: a
|
||||
/// folder upserted with the same name and parent must not touch its subtree.
|
||||
///
|
||||
/// Written as a ratio between two subtree sizes, so it fails on the scaling
|
||||
/// regression rather than on a slow machine. Before this, every re-listing paid
|
||||
/// a `path_of` walk plus an ancestor walk per descendant.
|
||||
#[test]
|
||||
fn re_listing_a_folder_does_not_scale_with_its_subtree() {
|
||||
fn elapsed_for(children: usize) -> std::time::Duration {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
db.upsert_node(&folder("root", None, "My Files")).unwrap();
|
||||
db.upsert_node(&folder("big", Some("root"), "Big")).unwrap();
|
||||
let kids: Vec<Node> = (0..children)
|
||||
.map(|i| file(&format!("k{i}"), "big", &format!("child-{i}.txt"), 1))
|
||||
.collect();
|
||||
db.upsert_nodes(&kids).unwrap();
|
||||
|
||||
let unchanged = folder("big", Some("root"), "Big");
|
||||
let start = std::time::Instant::now();
|
||||
for _ in 0..20 {
|
||||
db.upsert_node(&unchanged).unwrap();
|
||||
}
|
||||
start.elapsed()
|
||||
}
|
||||
|
||||
let small = elapsed_for(100);
|
||||
let large = elapsed_for(2_000);
|
||||
assert!(
|
||||
large < small * 8,
|
||||
"re-listing a folder scaled with its subtree: {small:?} for 100 children, \
|
||||
{large:?} for 2000 (20x the descendants)"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn migration_v25_backfills_paths_for_an_existing_tree() {
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"pdfs-db-v24-fixture-{}-{}.db",
|
||||
std::process::id(),
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_nanos()
|
||||
));
|
||||
{
|
||||
let db = Db::open(&path).unwrap();
|
||||
db.upsert_node(&folder("root", None, "My Files")).unwrap();
|
||||
db.upsert_node(&folder("docs", Some("root"), "Documents"))
|
||||
.unwrap();
|
||||
db.upsert_node(&file("f1", "docs", "report.pdf", 1))
|
||||
.unwrap();
|
||||
db.upsert_node(&folder("orphan", Some("never-cached"), "Work"))
|
||||
.unwrap();
|
||||
db.upsert_node(&file("f2", "orphan", "notes.txt", 1))
|
||||
.unwrap();
|
||||
}
|
||||
{
|
||||
// Put the file back in the state a released V24 database was in.
|
||||
let conn = rusqlite::Connection::open(&path).unwrap();
|
||||
conn.execute_batch(
|
||||
"ALTER TABLE nodes DROP COLUMN path;
|
||||
UPDATE sync_state SET value = '24' WHERE key = 'schema_version';",
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let db = Db::open(&path).unwrap();
|
||||
assert_eq!(stored_path(&db, "root").as_deref(), Some(""));
|
||||
assert_eq!(stored_path(&db, "docs").as_deref(), Some("Documents"));
|
||||
assert_eq!(
|
||||
stored_path(&db, "f1").as_deref(),
|
||||
Some("Documents/report.pdf"),
|
||||
"an existing tree must come out of the migration with the same paths \
|
||||
the walk used to produce"
|
||||
);
|
||||
// A subtree whose top has an uncached parent is ordinary data, not an error.
|
||||
assert_eq!(stored_path(&db, "orphan").as_deref(), Some("Work"));
|
||||
assert_eq!(stored_path(&db, "f2").as_deref(), Some("Work/notes.txt"));
|
||||
|
||||
let _ = std::fs::remove_file(&path);
|
||||
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
|
||||
}
|
||||
|
||||
/// A queue carried over from V26 keeps its rows, and every one of them starts
|
||||
/// out not access-deferred — a migration must not invent a deferral window that
|
||||
/// would report an ordinary queued upload as blocked.
|
||||
#[test]
|
||||
fn migration_v27_adds_the_access_deferral_column_to_a_v26_queue() {
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"pdfs-db-v26-fixture-{}-{}.db",
|
||||
std::process::id(),
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_nanos()
|
||||
));
|
||||
{
|
||||
let db = Db::open(&path).unwrap();
|
||||
db.enqueue_op(&crate::db::PendingOp {
|
||||
id: 0,
|
||||
kind: crate::db::OP_REVISION.to_string(),
|
||||
uid: "v~l".to_string(),
|
||||
parent_uid: None,
|
||||
name: None,
|
||||
blob_path: Some("/tmp/blob".to_string()),
|
||||
meta_json: Some("{}".to_string()),
|
||||
created_at: 1,
|
||||
attempts: 0,
|
||||
last_error: None,
|
||||
next_attempt_at: 0,
|
||||
})
|
||||
.unwrap();
|
||||
}
|
||||
{
|
||||
// Put the file back in the state a released V26 database was in.
|
||||
let conn = rusqlite::Connection::open(&path).unwrap();
|
||||
conn.execute_batch(
|
||||
"ALTER TABLE pending_op DROP COLUMN access_deferred_since;
|
||||
UPDATE sync_state SET value = '26' WHERE key = 'schema_version';",
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let db = Db::open(&path).unwrap();
|
||||
let ops = db.pending_ops().unwrap();
|
||||
assert_eq!(ops.len(), 1, "the migration keeps the queued write");
|
||||
let now = 5_000i64;
|
||||
assert_eq!(
|
||||
db.defer_op_for_access(ops[0].id, now, now + 5_000).unwrap(),
|
||||
now,
|
||||
"a carried-over op is not already deferred"
|
||||
);
|
||||
|
||||
let _ = std::fs::remove_file(&path);
|
||||
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
|
||||
}
|
||||
|
||||
/// A mirror baseline carried over from V27 keeps its rows, and each one reads
|
||||
/// back with no nanosecond time. That `None` is the whole point: it makes the
|
||||
/// comparison fall back to whole seconds for rows written before the column
|
||||
/// existed, instead of asserting a sub-second time of zero and re-uploading
|
||||
/// every already-synced file in the folder (bugs.md B25).
|
||||
#[test]
|
||||
fn migration_v28_adds_sub_second_local_times_without_disturbing_a_v27_baseline() {
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"pdfs-db-v27-fixture-{}-{}.db",
|
||||
std::process::id(),
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_nanos()
|
||||
));
|
||||
{
|
||||
let db = Db::open(&path).unwrap();
|
||||
let folder = db
|
||||
.sync_folder_add("/home/me/Mirror", "vol~root", "share")
|
||||
.unwrap();
|
||||
db.sync_entry_upsert(
|
||||
folder,
|
||||
&StoredSyncEntry {
|
||||
rel_path: "notes.txt".into(),
|
||||
remote_uid: Some("vol~notes".into()),
|
||||
local_mtime: 1_700_000_000,
|
||||
local_mtime_ns: Some(1_700_000_000_123_456_789),
|
||||
local_size: 42,
|
||||
remote_rev: Some("1700000000".into()),
|
||||
remote_hash: Some("42".into()),
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
{
|
||||
// Put the file back in the state a released V27 database was in.
|
||||
let conn = rusqlite::Connection::open(&path).unwrap();
|
||||
conn.execute_batch(
|
||||
"ALTER TABLE sync_entry DROP COLUMN local_mtime_ns;
|
||||
UPDATE sync_state SET value = '27' WHERE key = 'schema_version';",
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let db = Db::open(&path).unwrap();
|
||||
let entries = db.sync_entries(1).unwrap();
|
||||
let entry = entries.get("notes.txt").expect("the baseline row survives");
|
||||
assert_eq!(entry.local_mtime, 1_700_000_000);
|
||||
assert_eq!(entry.local_size, 42);
|
||||
assert_eq!(
|
||||
entry.local_mtime_ns, None,
|
||||
"a carried-over row must not claim a sub-second time it never recorded"
|
||||
);
|
||||
|
||||
// And the column is live: a refreshed row keeps the precision it is given.
|
||||
db.sync_entry_upsert(
|
||||
1,
|
||||
&StoredSyncEntry {
|
||||
local_mtime_ns: Some(1_700_000_001_000_000_500),
|
||||
..entry.clone()
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
db.sync_entries(1).unwrap()["notes.txt"].local_mtime_ns,
|
||||
Some(1_700_000_001_000_000_500)
|
||||
);
|
||||
|
||||
let _ = std::fs::remove_file(&path);
|
||||
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
|
||||
}
|
||||
|
||||
// --- read-only connection pool (DB2) ---------------------------------------
|
||||
|
||||
/// A read must not queue behind whatever happens to be writing.
|
||||
///
|
||||
/// The write connection is held for the whole measurement — which is what a
|
||||
/// large listing's commit does to it — while a reader asks for a search. Before
|
||||
/// the pool this was the same mutex, so the read waited for the writer;
|
||||
/// `busy_timeout` never entered into it, because the contention was this
|
||||
/// process's own lock rather than SQLite's.
|
||||
#[test]
|
||||
fn a_read_does_not_wait_for_the_write_connection() {
|
||||
let path = std::env::temp_dir().join(format!(
|
||||
"pdfs-db-readpool-{}-{}.db",
|
||||
std::process::id(),
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_nanos()
|
||||
));
|
||||
let db = std::sync::Arc::new(Db::open(&path).unwrap());
|
||||
db.upsert_node(&folder("root", None, "My Files")).unwrap();
|
||||
db.upsert_node(&file("f1", "root", "findme.txt", 1))
|
||||
.unwrap();
|
||||
|
||||
let held = std::time::Duration::from_millis(500);
|
||||
let (tx, rx) = std::sync::mpsc::channel();
|
||||
let writer = {
|
||||
let db = db.clone();
|
||||
std::thread::spawn(move || {
|
||||
db.with_conn(|_conn| {
|
||||
tx.send(()).unwrap();
|
||||
std::thread::sleep(held);
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
})
|
||||
};
|
||||
rx.recv().unwrap();
|
||||
|
||||
let start = std::time::Instant::now();
|
||||
let hits = db.search("findme", 10).unwrap();
|
||||
let waited = start.elapsed();
|
||||
writer.join().unwrap();
|
||||
|
||||
assert_eq!(hits.len(), 1, "the read must still see committed data");
|
||||
assert!(
|
||||
waited < held / 2,
|
||||
"the read waited {waited:?} for a write connection held {held:?}"
|
||||
);
|
||||
|
||||
drop(db);
|
||||
let _ = std::fs::remove_file(&path);
|
||||
let _ = std::fs::remove_file(format!("{}.lock", path.display()));
|
||||
}
|
||||
|
||||
/// §5: the drain is several workers over one queue, and the claim column is the
|
||||
/// only thing keeping them off each other's work. Two properties matter — no
|
||||
/// row is ever handed to two workers, and no *node* is ever worked by two
|
||||
/// workers even though its ops are separate rows.
|
||||
#[test]
|
||||
fn a_claimed_op_is_invisible_to_every_other_worker() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
let root = uid("root").to_string();
|
||||
for i in 0..3 {
|
||||
db.enqueue_op(&bulk_op(i, &root)).unwrap();
|
||||
}
|
||||
|
||||
let first = db.claim_next_due_op(10).unwrap().expect("an op is due");
|
||||
let second = db
|
||||
.claim_next_due_op(10)
|
||||
.unwrap()
|
||||
.expect("another op is due");
|
||||
let third = db
|
||||
.claim_next_due_op(10)
|
||||
.unwrap()
|
||||
.expect("a third op is due");
|
||||
assert!(
|
||||
db.claim_next_due_op(10).unwrap().is_none(),
|
||||
"a fourth claim must find nothing left: every queued op is taken"
|
||||
);
|
||||
|
||||
let mut ids = [first.id, second.id, third.id];
|
||||
ids.sort_unstable();
|
||||
assert_eq!(
|
||||
ids,
|
||||
[1, 2, 3],
|
||||
"each worker must get a distinct row, oldest first"
|
||||
);
|
||||
|
||||
// Releasing puts one back, and it is the one that comes out next.
|
||||
db.release_op_claim(second.id).unwrap();
|
||||
let again = db.claim_next_due_op(10).unwrap().expect("the released op");
|
||||
assert_eq!(again.id, second.id);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn two_workers_never_share_a_node() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
let root = uid("root").to_string();
|
||||
// Two ops of different kinds against one node — a queued rename and a
|
||||
// queued revision, which is an ordinary `mv` plus a save.
|
||||
let mut revision = bulk_op(0, &root);
|
||||
revision.parent_uid = None;
|
||||
db.enqueue_op(&revision).unwrap();
|
||||
let rename = PendingOp {
|
||||
kind: OP_RENAME.to_string(),
|
||||
parent_uid: Some(root.clone()),
|
||||
blob_path: None,
|
||||
meta_json: None,
|
||||
..bulk_op(0, &root)
|
||||
};
|
||||
db.enqueue_op(&rename).unwrap();
|
||||
|
||||
let claimed = db.claim_next_due_op(10).unwrap().expect("an op is due");
|
||||
assert_eq!(claimed.kind, OP_REVISION, "the older op goes first");
|
||||
assert!(
|
||||
db.claim_next_due_op(10).unwrap().is_none(),
|
||||
"the second op targets the same node, so it must wait for the first: \
|
||||
two workers on one uid would race over its staged blob and land its \
|
||||
changes out of order"
|
||||
);
|
||||
|
||||
db.release_op_claim(claimed.id).unwrap();
|
||||
db.delete_op(claimed.id).unwrap();
|
||||
assert_eq!(
|
||||
db.claim_next_due_op(10).unwrap().map(|o| o.kind),
|
||||
Some(OP_RENAME.to_string()),
|
||||
"once the revision retires, the rename is claimable"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn claims_do_not_survive_the_process_that_took_them() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
let root = uid("root").to_string();
|
||||
for i in 0..2 {
|
||||
db.enqueue_op(&bulk_op(i, &root)).unwrap();
|
||||
}
|
||||
db.claim_next_due_op(10).unwrap().unwrap();
|
||||
db.claim_next_due_op(10).unwrap().unwrap();
|
||||
assert!(db.claim_next_due_op(10).unwrap().is_none());
|
||||
|
||||
// What `Db::open` does after the single-writer lock says the previous run
|
||||
// is gone. Without it those ops are invisible to every worker forever.
|
||||
assert_eq!(db.clear_op_claims().unwrap(), 2);
|
||||
assert!(
|
||||
db.claim_next_due_op(10).unwrap().is_some(),
|
||||
"a crashed run's claims must not park its queue"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_idle_worker_does_not_wait_on_an_op_another_worker_has() {
|
||||
let db = Db::open_in_memory().unwrap();
|
||||
let root = uid("root").to_string();
|
||||
db.enqueue_op(&bulk_op(0, &root)).unwrap();
|
||||
|
||||
assert_eq!(db.earliest_due_at().unwrap(), Some(0));
|
||||
db.claim_next_due_op(10).unwrap().unwrap();
|
||||
assert_eq!(
|
||||
db.earliest_due_at().unwrap(),
|
||||
None,
|
||||
"an op somebody else is draining is not work this worker is waiting for"
|
||||
);
|
||||
}
|
||||
@@ -39,7 +39,7 @@ impl Db {
|
||||
/// The persisted trash listing, folders first then by name — the order the
|
||||
/// Trash page shows it in.
|
||||
pub fn trash_list(&self) -> Result<Vec<StoredTrash>> {
|
||||
let conn = self.conn.lock();
|
||||
let conn = self.read();
|
||||
let mut stmt = conn.prepare(
|
||||
"SELECT uid, name, is_dir, size, mtime FROM trash
|
||||
ORDER BY is_dir DESC, name COLLATE NOCASE",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
//! Small query helpers shared across the table modules.
|
||||
|
||||
use rusqlite::{Connection, params};
|
||||
use rusqlite::{Connection, OptionalExtension, params};
|
||||
|
||||
use crate::Result;
|
||||
|
||||
@@ -8,14 +8,19 @@ use crate::Result;
|
||||
/// grams), so short queries fall back to a `LIKE` scan over `nodes.name`.
|
||||
pub(super) const TRIGRAM_MIN: usize = 3;
|
||||
|
||||
pub(super) fn pair(row: &rusqlite::Row) -> rusqlite::Result<(String, String)> {
|
||||
Ok((row.get(0)?, row.get(1)?))
|
||||
/// One raw search row: the node's JSON, its uid, and its stored path — `None`
|
||||
/// only for a row written before the `path` column existed, which
|
||||
/// [`path_of`] repairs by walking.
|
||||
pub(super) type HitRow = (String, String, Option<String>);
|
||||
|
||||
pub(super) fn hit_row(row: &rusqlite::Row) -> rusqlite::Result<HitRow> {
|
||||
Ok((row.get(0)?, row.get(1)?, row.get(2)?))
|
||||
}
|
||||
|
||||
/// Drain a `query_map` of [`pair`] rows into a `Vec`, propagating row errors.
|
||||
pub(super) fn collect_pairs(
|
||||
rows: impl Iterator<Item = rusqlite::Result<(String, String)>>,
|
||||
) -> Result<Vec<(String, String)>> {
|
||||
/// Drain a `query_map` of [`hit_row`] rows into a `Vec`, propagating row errors.
|
||||
pub(super) fn collect_hits(
|
||||
rows: impl Iterator<Item = rusqlite::Result<HitRow>>,
|
||||
) -> Result<Vec<HitRow>> {
|
||||
let mut out = Vec::new();
|
||||
for r in rows {
|
||||
out.push(r?);
|
||||
@@ -35,18 +40,44 @@ pub(super) fn like_escape(s: &str) -> String {
|
||||
/// nowhere near this; a parent *cycle* is unbounded, and this query is on the
|
||||
/// path of every search result, including the `LIKE` lane that reads `nodes`
|
||||
/// directly and so never passed through the index's cycle check.
|
||||
const MAX_PATH_DEPTH: usize = 256;
|
||||
pub(super) const MAX_PATH_DEPTH: usize = 256;
|
||||
|
||||
/// Resolve a node's mountpoint-relative path by walking `parent_uid` to the
|
||||
/// root via a recursive CTE. The root (the node with no parent) is excluded, so
|
||||
/// a top-level file `report.pdf` yields `"report.pdf"`, not `"My Files/report.pdf"`.
|
||||
/// Append `name` to a parent's stored path. An empty parent path is the root,
|
||||
/// whose name is the mount and so contributes nothing.
|
||||
pub(super) fn join_path(parent_path: &str, name: &str) -> String {
|
||||
if parent_path.is_empty() {
|
||||
name.to_string()
|
||||
} else {
|
||||
format!("{parent_path}/{name}")
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve a node's mountpoint-relative path. The root (the node with no
|
||||
/// parent) is excluded, so a top-level file `report.pdf` yields
|
||||
/// `"report.pdf"`, not `"My Files/report.pdf"`.
|
||||
///
|
||||
/// Reads the stored `nodes.path`, which the write path maintains (see
|
||||
/// `upsert_node_tx`). Falls back to [`walk_path_of`] for a row that predates the
|
||||
/// column or was never written through — a search serving a keystroke used to
|
||||
/// run that walk once per candidate, which is the reason the column exists.
|
||||
pub(super) fn path_of(conn: &Connection, uid: &str) -> Result<String> {
|
||||
let stored: Option<Option<String>> = conn
|
||||
.query_row("SELECT path FROM nodes WHERE uid = ?1", params![uid], |r| {
|
||||
r.get(0)
|
||||
})
|
||||
.optional()?;
|
||||
match stored.flatten() {
|
||||
Some(path) => Ok(path),
|
||||
None => walk_path_of(conn, uid),
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve a path by walking `parent_uid` to the root via a recursive CTE.
|
||||
///
|
||||
/// A cycle in `parent_uid` is corrupt data the API can still hand us, and
|
||||
/// `UNION ALL` over one never terminates — the walk is therefore depth-capped
|
||||
/// and returns the truncated path rather than hanging the caller. That caller is
|
||||
/// usually a search serving a keystroke, holding the daemon's only SQLite
|
||||
/// connection.
|
||||
pub(super) fn path_of(conn: &Connection, uid: &str) -> Result<String> {
|
||||
/// and returns the truncated path rather than hanging the caller.
|
||||
pub(super) fn walk_path_of(conn: &Connection, uid: &str) -> Result<String> {
|
||||
let mut stmt = conn.prepare(&format!(
|
||||
"WITH RECURSIVE anc(uid, parent_uid, name, depth) AS (
|
||||
SELECT uid, parent_uid, name, 0 FROM nodes WHERE uid = ?1
|
||||
|
||||
@@ -11,7 +11,7 @@ pub enum Error {
|
||||
Proton(#[from] proton_sdk::ProtonError),
|
||||
|
||||
#[error("keyring: {0}")]
|
||||
Keyring(#[from] keyring::Error),
|
||||
Keyring(#[from] keyring_core::Error),
|
||||
|
||||
#[error("serialization: {0}")]
|
||||
Serde(#[from] serde_json::Error),
|
||||
@@ -45,6 +45,27 @@ pub enum Error {
|
||||
Other(String),
|
||||
}
|
||||
|
||||
impl Error {
|
||||
/// Whether this is the local disk being full (`ENOSPC`), as opposed to any
|
||||
/// other I/O failure.
|
||||
///
|
||||
/// Worth distinguishing because it is the one storage failure the user can
|
||||
/// act on, and because the caller's response differs: a full disk is met by
|
||||
/// evicting cached blobs and answering `ENOSPC`, where `EIO` would read as a
|
||||
/// broken file. `std::io::ErrorKind::StorageFull` is still unstable, so this
|
||||
/// compares the raw code.
|
||||
pub fn is_disk_full(&self) -> bool {
|
||||
match self {
|
||||
Error::Io(e) => e.raw_os_error() == Some(libc::ENOSPC),
|
||||
// SQLite reports it as its own code rather than surfacing the errno.
|
||||
Error::Db(rusqlite::Error::SqliteFailure(e, _)) => {
|
||||
e.code == rusqlite::ErrorCode::DiskFull
|
||||
}
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub type Result<T> = std::result::Result<T, Error>;
|
||||
|
||||
/// A failure on its way to a front-end: prose for the user, plus the
|
||||
|
||||
@@ -20,6 +20,7 @@ pub mod search;
|
||||
pub mod service;
|
||||
pub mod shell;
|
||||
pub mod syncignore;
|
||||
pub mod takeout;
|
||||
|
||||
pub use access::{Access, access_for, perm_bits};
|
||||
pub use error::{CoreError, CoreResult, Error, Result};
|
||||
|
||||
@@ -72,6 +72,25 @@ const KNOWN_MENUS: [MenuFlavor; 6] = [
|
||||
},
|
||||
];
|
||||
|
||||
/// Terminals that can host `pdfs-prompt --fzf`, most-preferred first.
|
||||
///
|
||||
/// The fzf front end is a terminal program, so a keybinding that has no tty
|
||||
/// needs one spawned for it. Each entry sets a stable app-id/class
|
||||
/// ([`TERMINAL_APP_ID`]) so a tiling WM can be told to float exactly this
|
||||
/// window, and ends with the flag after which the command to run follows.
|
||||
const KNOWN_TERMINALS: [&[&str]; 6] = [
|
||||
&["foot", "--app-id=pdfs-prompt", "--"],
|
||||
&["ghostty", "--class=pdfs-prompt", "-e"],
|
||||
&["kitty", "--class", "pdfs-prompt"],
|
||||
&["alacritty", "--class", "pdfs-prompt", "-e"],
|
||||
&["wezterm", "start", "--class", "pdfs-prompt", "--"],
|
||||
&["xterm", "-class", "pdfs-prompt", "-e"],
|
||||
];
|
||||
|
||||
/// The app-id/class every [`KNOWN_TERMINALS`] entry sets, so the window rule a
|
||||
/// user writes for one terminal keeps working if they switch.
|
||||
pub const TERMINAL_APP_ID: &str = "pdfs-prompt";
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum PromptFlag {
|
||||
/// `--prompt=TEXT`
|
||||
@@ -105,6 +124,13 @@ pub struct PromptConfig {
|
||||
/// the prompt text; without one the launcher's own prompt flag is appended.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub menu: Option<Vec<String>>,
|
||||
/// Terminal argv for `pdfs-prompt --fzf` when it is started without a tty.
|
||||
/// Unset means "find one" — see [`resolve_terminal`]. A `{cmd}` token
|
||||
/// anywhere in the argv is replaced by the command to run; without one the
|
||||
/// command is appended, which is what every [`KNOWN_TERMINALS`] entry
|
||||
/// expects.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub terminal: Option<Vec<String>>,
|
||||
/// How many results to feed the launcher. Unset means
|
||||
/// [`DEFAULT_MENU_LIMIT`].
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
@@ -124,6 +150,8 @@ pub enum PromptMode {
|
||||
Gtk,
|
||||
/// The user's own launcher, driven over stdin/stdout.
|
||||
Dmenu,
|
||||
/// `fzf` in a terminal, re-querying the daemon on every keystroke.
|
||||
Fzf,
|
||||
}
|
||||
|
||||
/// Rows an external launcher can show without becoming a scroll exercise. Larger
|
||||
@@ -186,6 +214,43 @@ pub fn resolve_menu(configured: Option<&Vec<String>>) -> Option<Vec<String>> {
|
||||
Some(argv)
|
||||
}
|
||||
|
||||
/// The terminal to host `pdfs-prompt --fzf` in: the configured one, or the
|
||||
/// first [`KNOWN_TERMINALS`] entry on `PATH`. Returns the argv up to and
|
||||
/// including the "run this command" flag — the caller appends the command, or
|
||||
/// substitutes it for a `{cmd}` token, via [`with_command`].
|
||||
pub fn resolve_terminal(configured: Option<&Vec<String>>) -> Option<Vec<String>> {
|
||||
if let Some(argv) = configured.filter(|argv| !argv.is_empty()) {
|
||||
return Some(argv.clone());
|
||||
}
|
||||
let flavor = KNOWN_TERMINALS
|
||||
.iter()
|
||||
.find(|argv| argv.first().is_some_and(|program| on_path(program)))?;
|
||||
Some(flavor.iter().map(|arg| (*arg).to_string()).collect())
|
||||
}
|
||||
|
||||
/// Place `command` into a terminal argv: substituted for a `{cmd}` token if the
|
||||
/// user wrote one, otherwise appended.
|
||||
///
|
||||
/// The token exists for terminals that want the command in the middle of their
|
||||
/// own arguments; appending is right for every [`KNOWN_TERMINALS`] entry, whose
|
||||
/// last element is the flag the command follows.
|
||||
pub fn with_command(argv: &[String], command: &[String]) -> Vec<String> {
|
||||
if !argv.iter().any(|arg| arg == "{cmd}") {
|
||||
let mut argv = argv.to_vec();
|
||||
argv.extend_from_slice(command);
|
||||
return argv;
|
||||
}
|
||||
let mut out = Vec::with_capacity(argv.len() + command.len());
|
||||
for arg in argv {
|
||||
if arg == "{cmd}" {
|
||||
out.extend_from_slice(command);
|
||||
} else {
|
||||
out.push(arg.clone());
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Show `items` in the launcher named by `argv` and wait for a choice.
|
||||
///
|
||||
/// Blocking by design: the caller is a one-shot CLI invocation whose entire job
|
||||
@@ -305,7 +370,8 @@ fn flavor_of(program: &str) -> Option<&'static MenuFlavor> {
|
||||
KNOWN_MENUS.iter().find(|flavor| flavor.program == base)
|
||||
}
|
||||
|
||||
fn on_path(program: &str) -> bool {
|
||||
/// Whether `program` is an executable file on `PATH`.
|
||||
pub fn on_path(program: &str) -> bool {
|
||||
let Some(path) = std::env::var_os("PATH") else {
|
||||
return false;
|
||||
};
|
||||
@@ -408,6 +474,47 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_configured_terminal_is_used_verbatim() {
|
||||
let configured = vec!["my-term".to_string(), "-e".to_string()];
|
||||
assert_eq!(
|
||||
resolve_terminal(Some(&configured)),
|
||||
Some(configured.clone())
|
||||
);
|
||||
// An empty list is "not configured", not "run nothing".
|
||||
assert_eq!(
|
||||
resolve_terminal(Some(&Vec::new())),
|
||||
resolve_terminal(None),
|
||||
"an empty terminal list must fall back to detection"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_command_is_appended_unless_the_user_placed_it() {
|
||||
let command = vec!["pdfs-prompt".to_string(), "--fzf".to_string()];
|
||||
assert_eq!(
|
||||
with_command(&["foot".into(), "--".into()], &command),
|
||||
vec!["foot", "--", "pdfs-prompt", "--fzf"]
|
||||
);
|
||||
assert_eq!(
|
||||
with_command(
|
||||
&["term".into(), "-e".into(), "{cmd}".into(), "--hold".into()],
|
||||
&command
|
||||
),
|
||||
vec!["term", "-e", "pdfs-prompt", "--fzf", "--hold"]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_known_terminal_sets_the_same_app_id() {
|
||||
for argv in KNOWN_TERMINALS {
|
||||
assert!(
|
||||
argv.iter().any(|arg| arg.contains(TERMINAL_APP_ID)),
|
||||
"{argv:?} must be floatable by one window rule"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_configured_launcher_is_used_verbatim() {
|
||||
let configured = vec!["my-menu".to_string(), "--pick".to_string()];
|
||||
|
||||
@@ -23,8 +23,9 @@
|
||||
//! ```
|
||||
|
||||
use std::ffi::OsStr;
|
||||
use std::os::unix::process::CommandExt;
|
||||
use std::path::Path;
|
||||
use std::process::Command;
|
||||
use std::process::{Command, Stdio};
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
@@ -165,12 +166,30 @@ pub fn open_named(policy: &OpenWith, path: &Path, name: &str, is_dir: bool) {
|
||||
spawn(&argv, path);
|
||||
}
|
||||
|
||||
/// Launch `argv`, detached from whatever is opening it.
|
||||
///
|
||||
/// The child gets its own process group and no inherited stdio, because the
|
||||
/// caller may be about to exit and take its terminal with it: `pdfs-prompt
|
||||
/// --fzf` runs inside a terminal it spawned for itself, and that terminal
|
||||
/// closes the moment the selection is made. A child left in the terminal's
|
||||
/// foreground process group is sent `SIGHUP` when the pty hangs up, so
|
||||
/// "open this folder in a terminal" would open a window that dies immediately
|
||||
/// — which is exactly what it did. Nothing here needs the caller's stdio in the
|
||||
/// first place: every `argv` is another application, and a terminal one is
|
||||
/// wrapped in its own emulator by the `terminal` rule flag.
|
||||
fn spawn(argv: &[String], path: &Path) {
|
||||
let Some((program, args)) = argv.split_first() else {
|
||||
return;
|
||||
};
|
||||
tracing::info!(path = %path.display(), command = ?argv, "opening");
|
||||
if let Err(e) = Command::new(program).args(args).spawn() {
|
||||
let spawned = Command::new(program)
|
||||
.args(args)
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::null())
|
||||
.process_group(0)
|
||||
.spawn();
|
||||
if let Err(e) = spawned {
|
||||
tracing::error!(command = ?argv, "open failed: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -47,10 +47,18 @@ fn systemctl_query(args: &[&str]) -> bool {
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
/// Enable the service so it survives reboots/logins and start it now. Called on
|
||||
/// successful login.
|
||||
/// Enable the service so it survives reboots/logins, and bring it up now.
|
||||
/// Called on successful login.
|
||||
///
|
||||
/// `restart` rather than `enable --now`: the unit may already be active but
|
||||
/// parked in the daemon's login-wait backoff (`pdfs daemon` sleeps between
|
||||
/// keyring checks). `--now` is a no-op on an active unit, so the daemon would
|
||||
/// keep sleeping and the mount would not appear until the next poll fired.
|
||||
/// Restarting makes the new session take effect immediately, and starts the
|
||||
/// unit when it is not running at all.
|
||||
pub fn enable_start() -> bool {
|
||||
systemctl(&["enable", "--now"])
|
||||
let enabled = systemctl(&["enable"]);
|
||||
systemctl(&["restart"]) && enabled
|
||||
}
|
||||
|
||||
/// Stop the running mount without disabling it. Called by the tray's
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -5,10 +5,14 @@ edition.workspace = true
|
||||
license.workspace = true
|
||||
authors.workspace = true
|
||||
rust-version.workspace = true
|
||||
repository.workspace = true
|
||||
homepage.workspace = true
|
||||
description = "FUSE on-demand filesystem for Proton Drive"
|
||||
|
||||
[dependencies]
|
||||
pdfs-core.workspace = true
|
||||
zip.workspace = true
|
||||
sha1.workspace = true
|
||||
proton-drive-rs.workspace = true
|
||||
futures.workspace = true
|
||||
tokio.workspace = true
|
||||
@@ -16,6 +20,7 @@ tracing.workspace = true
|
||||
fuser.workspace = true
|
||||
libc.workspace = true
|
||||
serde_json.workspace = true
|
||||
base64.workspace = true
|
||||
image.workspace = true
|
||||
notify = "8"
|
||||
parking_lot.workspace = true
|
||||
@@ -1,6 +1,7 @@
|
||||
//! Remote-event invalidation and local-index background workers.
|
||||
|
||||
use super::*;
|
||||
use crate::shutdown::Shutdown;
|
||||
|
||||
/// Drop `uid` from one mount's tree and tell that mount's kernel session the
|
||||
/// entry is gone. Shared by the trash and delete paths, which differ only in
|
||||
@@ -8,35 +9,20 @@ use super::*;
|
||||
///
|
||||
/// Returns whether this mount actually held the node, so the caller can evict
|
||||
/// its content blob once if *any* mount did.
|
||||
fn forget_and_notify(st: &mut State, notifier: Option<&Notifier>, uid: &NodeUid) -> bool {
|
||||
fn forget_and_notify(st: &mut State, notify: &mut NotifyBatch, uid: &NodeUid) -> bool {
|
||||
// Capture the inode before `forget` clears the uid mapping.
|
||||
let child = st.by_uid.get(uid).copied();
|
||||
let Some((parent, name)) = st.forget(uid) else {
|
||||
return false;
|
||||
};
|
||||
if let Some(notifier) = notifier {
|
||||
match child {
|
||||
Some(child) => {
|
||||
let _ = notifier.delete(INodeNo(parent), INodeNo(child), OsStr::new(&name));
|
||||
}
|
||||
None => {
|
||||
let _ = notifier.inval_entry(INodeNo(parent), OsStr::new(&name));
|
||||
}
|
||||
}
|
||||
match child {
|
||||
Some(child) => notify.delete(parent, child, name),
|
||||
None => notify.inval_entry(parent, name),
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
fn invalidate_access_changes(notifier: Option<&Notifier>, changed: &[u64]) {
|
||||
let Some(notifier) = notifier else {
|
||||
return;
|
||||
};
|
||||
for &ino in changed {
|
||||
let _ = notifier.inval_inode(INodeNo(ino), 0, 0);
|
||||
}
|
||||
}
|
||||
|
||||
fn hide_foreign_deleted_and_notify(st: &mut State, notifier: Option<&Notifier>, uid: &NodeUid) {
|
||||
fn hide_foreign_deleted_and_notify(st: &mut State, notify: &mut NotifyBatch, uid: &NodeUid) {
|
||||
let child = st.by_uid.get(uid).copied();
|
||||
let dentry = child.and_then(|ino| {
|
||||
st.entries
|
||||
@@ -45,9 +31,9 @@ fn hide_foreign_deleted_and_notify(st: &mut State, notifier: Option<&Notifier>,
|
||||
});
|
||||
let changed = st.downgrade_shared_subtree(uid);
|
||||
st.hide_foreign_subtree(uid);
|
||||
invalidate_access_changes(notifier, &changed);
|
||||
if let (Some(notifier), Some((parent, name)), Some(child)) = (notifier, dentry, child) {
|
||||
let _ = notifier.delete(INodeNo(parent), INodeNo(child), OsStr::new(&name));
|
||||
notify.extend_inodes(&changed);
|
||||
if let (Some((parent, name)), Some(child)) = (dentry, child) {
|
||||
notify.delete(parent, child, name);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -132,18 +118,18 @@ fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdf
|
||||
&& foreign_delete
|
||||
{
|
||||
apply_foreign_delete(&core.db, node_uid, |uid| {
|
||||
core.for_each_mount(|st, notifier| {
|
||||
hide_foreign_deleted_and_notify(st, notifier, uid);
|
||||
core.for_each_mount(|st, notify| {
|
||||
hide_foreign_deleted_and_notify(st, notify, uid);
|
||||
});
|
||||
})
|
||||
} else {
|
||||
apply_access_downgrade(&core.db, event, |root| {
|
||||
core.for_each_mount(|st, notifier| {
|
||||
core.for_each_mount(|st, notify| {
|
||||
let changed = match root {
|
||||
Some(uid) => st.downgrade_shared_subtree(uid),
|
||||
None => st.downgrade_known_shared_access(),
|
||||
};
|
||||
invalidate_access_changes(notifier, &changed);
|
||||
notify.extend_inodes(&changed);
|
||||
});
|
||||
})
|
||||
};
|
||||
@@ -173,19 +159,17 @@ fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdf
|
||||
debug!(uid = %node_uid, "ignoring remote event for a node with a queued write");
|
||||
}
|
||||
let mut had_node = false;
|
||||
core.for_each_mount(|st, notifier| {
|
||||
core.for_each_mount(|st, notify| {
|
||||
if *is_trashed {
|
||||
// Trashing makes a node vanish from its parent listing.
|
||||
had_node |= forget_and_notify(st, notifier, node_uid);
|
||||
had_node |= forget_and_notify(st, notify, node_uid);
|
||||
} else if !ours && let Some(&ino) = st.by_uid.get(node_uid) {
|
||||
// Known node changed: drop its cached attrs/data (and
|
||||
// listing if it is a directory) so the next access
|
||||
// re-fetches. Its content blob may now be stale too.
|
||||
had_node = true;
|
||||
st.invalidate_listing(ino);
|
||||
if let Some(notifier) = notifier {
|
||||
let _ = notifier.inval_inode(INodeNo(ino), 0, 0);
|
||||
}
|
||||
notify.inval_inode(ino);
|
||||
}
|
||||
});
|
||||
// Once, not once per mount: the content cache is per-daemon.
|
||||
@@ -209,8 +193,8 @@ fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdf
|
||||
DriveEvent::NodeDeleted { node_uid, .. } => {
|
||||
core.cache.evict(node_uid);
|
||||
if !foreign_delete {
|
||||
core.for_each_mount(|st, notifier| {
|
||||
forget_and_notify(st, notifier, node_uid);
|
||||
core.for_each_mount(|st, notify| {
|
||||
forget_and_notify(st, notify, node_uid);
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -223,13 +207,11 @@ fn apply_event(core: &Core, event: &DriveEvent, dirty: &mut DirtyParents) -> pdf
|
||||
| DriveEvent::SharedWithMeUpdated { .. } => {
|
||||
warn!("event access continuity lost; downgrading shared trees and resyncing lazily");
|
||||
apply_result = apply_result.and(core.db.clear_state(SHARED_WITH_ME_SYNCED_MS));
|
||||
core.for_each_mount(|st, notifier| {
|
||||
core.for_each_mount(|st, notify| {
|
||||
let dirs: Vec<u64> = st.children.keys().copied().collect();
|
||||
for &ino in &dirs {
|
||||
st.invalidate_listing(ino);
|
||||
if let Some(notifier) = notifier {
|
||||
let _ = notifier.inval_inode(INodeNo(ino), 0, 0);
|
||||
}
|
||||
notify.inval_inode(ino);
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -303,18 +285,16 @@ fn flush_dirty_parents(core: &Core, dirty: &DirtyParents) {
|
||||
if dirty.foreign.is_empty() && dirty.ours.is_empty() {
|
||||
return;
|
||||
}
|
||||
core.for_each_mount(|st, notifier| {
|
||||
let drop_listing = |st: &mut State, parent_uid: &NodeUid| {
|
||||
let Some(&parent) = st.by_uid.get(parent_uid) else {
|
||||
return;
|
||||
};
|
||||
st.invalidate_listing(parent);
|
||||
if let Some(notifier) = notifier {
|
||||
let _ = notifier.inval_inode(INodeNo(parent), 0, 0);
|
||||
}
|
||||
fn drop_listing(st: &mut State, notify: &mut NotifyBatch, parent_uid: &NodeUid) {
|
||||
let Some(&parent) = st.by_uid.get(parent_uid) else {
|
||||
return;
|
||||
};
|
||||
st.invalidate_listing(parent);
|
||||
notify.inval_inode(parent);
|
||||
}
|
||||
core.for_each_mount(|st, notify| {
|
||||
for parent_uid in &dirty.foreign {
|
||||
drop_listing(st, parent_uid);
|
||||
drop_listing(st, notify, parent_uid);
|
||||
}
|
||||
for (parent_uid, children) in &dirty.ours {
|
||||
if dirty.foreign.contains(parent_uid) {
|
||||
@@ -328,7 +308,7 @@ fn flush_dirty_parents(core: &Core, dirty: &DirtyParents) {
|
||||
{
|
||||
continue;
|
||||
}
|
||||
drop_listing(st, parent_uid);
|
||||
drop_listing(st, notify, parent_uid);
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -458,6 +438,7 @@ pub(super) fn run_local_index(
|
||||
indexing: Arc<AtomicBool>,
|
||||
transfers: Arc<TransferRegistry>,
|
||||
mountpoint: PathBuf,
|
||||
shutdown: Arc<Shutdown>,
|
||||
) {
|
||||
loop {
|
||||
let age = db.local_indexed_at().ok().flatten();
|
||||
@@ -466,7 +447,11 @@ pub(super) fn run_local_index(
|
||||
if stale {
|
||||
scan_local_once(&db, &indexing, &transfers, &mountpoint);
|
||||
}
|
||||
std::thread::sleep(LOCAL_INDEX_CHECK);
|
||||
// Interruptible, so teardown joins this thread instead of leaving it to
|
||||
// keep rewriting the index for a mount that is gone (bugs.md B44).
|
||||
if !shutdown.sleep(LOCAL_INDEX_CHECK) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -556,6 +541,8 @@ mod tests {
|
||||
photo: None,
|
||||
album: None,
|
||||
verification: Default::default(),
|
||||
direct_role: None,
|
||||
share_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+234
-39
@@ -11,7 +11,7 @@
|
||||
|
||||
use std::io::{BufRead, BufReader, Write as _};
|
||||
use std::os::unix::net::{UnixListener, UnixStream};
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::path::{Component, Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
@@ -29,7 +29,23 @@ use super::{Core, count_noun, human_bytes, human_duration, parse_uid};
|
||||
|
||||
const MAX_CONTROL_REQUEST_BYTES: u64 = 1024 * 1024;
|
||||
const CONTROL_READ_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
/// A client that connects, sends a request and then never reads its reply used
|
||||
/// to park a handler thread forever in `write_all`; 64 of those exhaust
|
||||
/// [`MAX_CONTROL_HANDLERS`] permanently. Generous enough for the largest
|
||||
/// responses (photo bytes) over a local socket, finite so a wedged peer can't
|
||||
/// hold a permit.
|
||||
const CONTROL_WRITE_TIMEOUT: Duration = Duration::from_secs(30);
|
||||
const MAX_CONTROL_HANDLERS: usize = 64;
|
||||
/// Ceiling on any client-supplied row `limit`. Every in-tree caller asks for far
|
||||
/// less (200 for activity, 60 for a photo page), but the value goes straight
|
||||
/// into `SQL LIMIT ?` under the daemon's single connection — an unclamped one
|
||||
/// materialises the whole account, and then serialises it into one JSON line.
|
||||
const MAX_ROW_LIMIT: usize = 1000;
|
||||
|
||||
/// Clamp a client-supplied row limit to [`MAX_ROW_LIMIT`].
|
||||
fn clamp_limit(limit: usize) -> usize {
|
||||
limit.min(MAX_ROW_LIMIT)
|
||||
}
|
||||
static ACTIVE_CONTROL_HANDLERS: AtomicUsize = AtomicUsize::new(0);
|
||||
|
||||
struct ControlHandlerPermit;
|
||||
@@ -88,17 +104,51 @@ fn read_request_line(reader: &mut impl BufRead) -> std::io::Result<Option<String
|
||||
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))
|
||||
}
|
||||
|
||||
/// Turn a CLI-supplied path into the mount that owns it and the path relative
|
||||
/// to *that* mount's root.
|
||||
///
|
||||
/// [`rel_to_mount`] is the older, primary-only reading, and it is what every
|
||||
/// mutating request still uses. It rejects anything outside the primary
|
||||
/// mountpoint, which leaves a secondary on-demand mount — a synced folder
|
||||
/// switched out of mirror mode — with no way to be named at all: `pdfs ls` and
|
||||
/// `pdfs refresh` both answered "is not under the mountpoint", so a folder
|
||||
/// serving a stale listing had no manual escape hatch (`docs/BUGS.md` B86).
|
||||
///
|
||||
/// A relative path keeps the old meaning, relative to the primary mount, since
|
||||
/// there is nothing in it to choose a mount by.
|
||||
fn route_to_mount(core: &Core, mountpoint: &Path, path: &str) -> CoreResult<(Core, PathBuf)> {
|
||||
let p = Path::new(path);
|
||||
if !p.is_absolute() {
|
||||
return Ok((core.clone(), p.to_path_buf()));
|
||||
}
|
||||
if let Some(routed) = core.rooted_at(p) {
|
||||
return Ok(routed);
|
||||
}
|
||||
// Fall back to the primary reading so the error text stays the familiar one
|
||||
// when the path is simply outside everything this daemon has mounted.
|
||||
rel_to_mount(mountpoint, path).map(|rel| (core.clone(), rel))
|
||||
}
|
||||
|
||||
/// Turn a CLI-supplied path into a mountpoint-relative path. An absolute path
|
||||
/// must live under `mountpoint`; a relative path is taken as already relative to
|
||||
/// the mount root.
|
||||
fn rel_to_mount(mountpoint: &Path, path: &str) -> CoreResult<PathBuf> {
|
||||
let p = Path::new(path);
|
||||
if p.is_absolute() {
|
||||
let relative = if p.is_absolute() {
|
||||
p.strip_prefix(mountpoint)
|
||||
.map(Path::to_path_buf)
|
||||
.map_err(|_| CoreError::invalid(format!("{path} is not under the mountpoint")))
|
||||
} else {
|
||||
Ok(p.to_path_buf())
|
||||
}?;
|
||||
|
||||
if relative
|
||||
.components()
|
||||
.all(|component| matches!(component, Component::Normal(_) | Component::CurDir))
|
||||
{
|
||||
Ok(relative)
|
||||
} else {
|
||||
Err(CoreError::invalid(format!("{path} escapes the mountpoint")))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -109,6 +159,10 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
warn!(error = %e, "control: could not set request timeout");
|
||||
return;
|
||||
}
|
||||
if let Err(e) = stream.set_write_timeout(Some(CONTROL_WRITE_TIMEOUT)) {
|
||||
warn!(error = %e, "control: could not set response timeout");
|
||||
return;
|
||||
}
|
||||
let mut reader = BufReader::new(match stream.try_clone() {
|
||||
Ok(s) => s,
|
||||
Err(e) => {
|
||||
@@ -128,7 +182,13 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
Ok(CtlRequest::Status) => {
|
||||
let pins = core.cache.list_pins();
|
||||
let queued = core.db.pending_op_counts().unwrap_or_default();
|
||||
let staging = core.cache.staging_usage();
|
||||
CtlResponse::Status {
|
||||
parked_uploads: queued.parked.max(0) as u64,
|
||||
failing_ops: queued.failing.max(0) as u64,
|
||||
failing_error: queued.last_error.clone(),
|
||||
staged_bytes: staging.bytes,
|
||||
staged_oldest_secs: staging.oldest_secs,
|
||||
username: username.to_string(),
|
||||
mountpoint: mountpoint.display().to_string(),
|
||||
pinned: pins.len(),
|
||||
@@ -167,8 +227,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
Ok(CtlRequest::ListPins) => CtlResponse::Pins {
|
||||
pins: core.cache.list_pins(),
|
||||
},
|
||||
Ok(CtlRequest::ListDir { path }) => match rel_to_mount(mountpoint, &path) {
|
||||
Ok(rel) => match core.list_dir(&rel) {
|
||||
Ok(CtlRequest::ListDir { path }) => match route_to_mount(core, mountpoint, &path) {
|
||||
Ok((core, rel)) => match core.list_dir(&rel) {
|
||||
Ok(entries) => CtlResponse::Entries { entries },
|
||||
Err(e) => CtlResponse::error(e),
|
||||
},
|
||||
@@ -176,9 +236,8 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
},
|
||||
Ok(CtlRequest::Refresh { scope }) => {
|
||||
let result = match &scope {
|
||||
RefreshScope::Dir { path } => {
|
||||
rel_to_mount(mountpoint, path).and_then(|rel| core.refresh_dir(&rel))
|
||||
}
|
||||
RefreshScope::Dir { path } => route_to_mount(core, mountpoint, path)
|
||||
.and_then(|(core, rel)| core.refresh_dir(&rel)),
|
||||
RefreshScope::Trash => {
|
||||
core.invalidate_trash();
|
||||
Ok(())
|
||||
@@ -204,7 +263,7 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
kind,
|
||||
range,
|
||||
favorites,
|
||||
}) => match core.photos_timeline(offset, limit, kind, range, favorites) {
|
||||
}) => match core.photos_timeline(offset, clamp_limit(limit), kind, range, favorites) {
|
||||
Ok(Some(items)) => CtlResponse::Photos {
|
||||
available: true,
|
||||
items,
|
||||
@@ -245,7 +304,7 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
// An album page is a timeline page as far as the front-end is
|
||||
// concerned, so it comes back in the same reply — with `counts`
|
||||
// unset, which describe the whole timeline rather than this album.
|
||||
Some(album) => match core.album_photos(&album, offset, limit) {
|
||||
Some(album) => match core.album_photos(&album, offset, clamp_limit(limit)) {
|
||||
Ok(items) => CtlResponse::Photos {
|
||||
available: true,
|
||||
items,
|
||||
@@ -270,6 +329,34 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
items: core.photo_thumbs(&parsed),
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::FileThumbs { items, generation }) => {
|
||||
match core.file_thumbs(&items, generation) {
|
||||
Some(items) => CtlResponse::Thumbs { items },
|
||||
None => CtlResponse::FileThumbsStale,
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::ReserveFileThumbGeneration) => CtlResponse::FileThumbGeneration {
|
||||
generation: core.reserve_file_thumb_generation(),
|
||||
},
|
||||
Ok(CtlRequest::CancelFileThumbs { generation }) => {
|
||||
core.cancel_file_thumbs(generation);
|
||||
CtlResponse::Ok {
|
||||
message: "thumbnail requests cancelled".to_string(),
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::StartThumbnailBuild { path }) => match rel_to_mount(mountpoint, &path) {
|
||||
Ok(rel) => match core.start_thumbnail_build(rel) {
|
||||
Ok(status) => CtlResponse::ThumbnailBuild { status },
|
||||
Err(error) => CtlResponse::error(error),
|
||||
},
|
||||
Err(error) => CtlResponse::error(error),
|
||||
},
|
||||
Ok(CtlRequest::CancelThumbnailBuild) => CtlResponse::ThumbnailBuild {
|
||||
status: core.cancel_thumbnail_build(),
|
||||
},
|
||||
Ok(CtlRequest::ThumbnailBuildStatus) => CtlResponse::ThumbnailBuild {
|
||||
status: core.thumbnail_build_status(),
|
||||
},
|
||||
Ok(CtlRequest::OpenPhoto { uid }) => match parse_uid(&uid) {
|
||||
Some(u) => match core.open_photo(&u) {
|
||||
Ok(p) => CtlResponse::FilePath {
|
||||
@@ -279,6 +366,81 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
},
|
||||
None => CtlResponse::error(CoreError::invalid(format!("bad uid: {uid}"))),
|
||||
},
|
||||
// The scan alone walks every archive's central directory, so even the
|
||||
// dry run goes to a background thread and is read back through
|
||||
// `ImportStatus` rather than answered inline.
|
||||
Ok(CtlRequest::ImportTakeout { archives, dry_run }) => {
|
||||
if archives.is_empty() {
|
||||
CtlResponse::error(CoreError::invalid("no Takeout archives given"))
|
||||
} else {
|
||||
let paths: Vec<PathBuf> = archives.into_iter().map(PathBuf::from).collect();
|
||||
let missing: Vec<String> = paths
|
||||
.iter()
|
||||
.filter(|path| !path.is_file())
|
||||
.map(|path| path.display().to_string())
|
||||
.collect();
|
||||
if !missing.is_empty() {
|
||||
CtlResponse::error(CoreError::not_found(format!(
|
||||
"cannot read: {}",
|
||||
missing.join(", ")
|
||||
)))
|
||||
} else {
|
||||
let count = paths.len();
|
||||
match core.spawn_takeout_import(paths, dry_run) {
|
||||
Ok(()) => CtlResponse::Ok {
|
||||
message: format!(
|
||||
"{} {count} archive(s)",
|
||||
if dry_run { "scanning" } else { "importing" }
|
||||
),
|
||||
},
|
||||
Err(e) => CtlResponse::error(e),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::ImportStatus) => {
|
||||
let (running, summary) = crate::takeout::import_status();
|
||||
CtlResponse::ImportStatus { running, summary }
|
||||
}
|
||||
Ok(CtlRequest::CancelImport) => {
|
||||
if crate::takeout::cancel_import() {
|
||||
CtlResponse::Ok {
|
||||
message: "stopping the import".to_string(),
|
||||
}
|
||||
} else {
|
||||
CtlResponse::error(CoreError::invalid("no import is running"))
|
||||
}
|
||||
}
|
||||
// The scan reads the whole persisted timeline and every candidate is a
|
||||
// download plus an upload, so this goes to a background thread and is
|
||||
// read back through `RedateStatus`.
|
||||
Ok(CtlRequest::RedatePhotos {
|
||||
dry_run,
|
||||
limit,
|
||||
range,
|
||||
}) => match core.spawn_redate(dry_run, limit, range) {
|
||||
Ok(()) => CtlResponse::Ok {
|
||||
message: if dry_run {
|
||||
"checking photo dates".to_string()
|
||||
} else {
|
||||
"fixing photo dates".to_string()
|
||||
},
|
||||
},
|
||||
Err(e) => CtlResponse::error(e),
|
||||
},
|
||||
Ok(CtlRequest::RedateStatus) => {
|
||||
let (running, summary) = crate::redate::redate_status();
|
||||
CtlResponse::RedateStatus { running, summary }
|
||||
}
|
||||
Ok(CtlRequest::CancelRedate) => {
|
||||
if crate::redate::cancel_redate() {
|
||||
CtlResponse::Ok {
|
||||
message: "stopping the re-date".to_string(),
|
||||
}
|
||||
} else {
|
||||
CtlResponse::error(CoreError::invalid("no photo re-date is running"))
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::UploadPhoto {
|
||||
name,
|
||||
media_type,
|
||||
@@ -346,22 +508,24 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
Err(e) => CtlResponse::error(e),
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::Search { query, limit }) => match core.search(&query, limit) {
|
||||
Ok(CtlRequest::Search { query, limit }) => match core.search(&query, clamp_limit(limit)) {
|
||||
Ok(hits) => CtlResponse::SearchResults { hits },
|
||||
Err(e) => CtlResponse::error(e),
|
||||
},
|
||||
Ok(CtlRequest::SearchLocal { query, limit }) => match core.search_local(&query, limit) {
|
||||
Ok(hits) => CtlResponse::LocalResults {
|
||||
hits,
|
||||
indexing: core.indexing.load(Ordering::Relaxed),
|
||||
},
|
||||
Err(e) => CtlResponse::error(e),
|
||||
},
|
||||
Ok(CtlRequest::SearchLocal { query, limit }) => {
|
||||
match core.search_local(&query, clamp_limit(limit)) {
|
||||
Ok(hits) => CtlResponse::LocalResults {
|
||||
hits,
|
||||
indexing: core.indexing.load(Ordering::Relaxed),
|
||||
},
|
||||
Err(e) => CtlResponse::error(e),
|
||||
}
|
||||
}
|
||||
Ok(CtlRequest::SearchV2 {
|
||||
query,
|
||||
limit,
|
||||
filters,
|
||||
}) => match core.search_v2(&query, limit, &filters) {
|
||||
}) => match core.search_v2(&query, clamp_limit(limit), &filters) {
|
||||
Ok((drive_hits, local_hits)) => CtlResponse::SearchResultsV2 {
|
||||
drive_hits,
|
||||
local_hits,
|
||||
@@ -540,19 +704,20 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
}
|
||||
Ok(CtlRequest::CacheInspect { deep }) => match core.db.stats() {
|
||||
Ok(stats) => {
|
||||
// The deep check reads every page, so it runs only on request;
|
||||
// `integrity_checked` tells the caller which answer it is
|
||||
// holding, since "no problems found" and "never looked" are
|
||||
// both an empty list.
|
||||
let (integrity_problems, integrity_checked) = if deep {
|
||||
match core.db.integrity_check() {
|
||||
Ok(problems) => (problems, true),
|
||||
Err(e) => (vec![format!("integrity check failed: {e:?}")], true),
|
||||
}
|
||||
// The deep check reads every page — tens of seconds on a real
|
||||
// database, all of it holding the connection every FUSE handler
|
||||
// needs — so it runs on a thread and this request returns
|
||||
// whatever the last completed check found. `integrity_checked`
|
||||
// tells the caller which answer it is holding, since "no
|
||||
// problems found" and "never looked" are both an empty list;
|
||||
// `integrity_running` says a fresh answer is on its way.
|
||||
let (integrity_problems, integrity_checked, integrity_running) = if deep {
|
||||
core.start_or_read_integrity_check()
|
||||
} else {
|
||||
(Vec::new(), false)
|
||||
(Vec::new(), false, false)
|
||||
};
|
||||
CtlResponse::CacheReport {
|
||||
integrity_running,
|
||||
schema_version: stats.schema_version,
|
||||
db_bytes: stats.total_bytes(),
|
||||
db_reclaimable_bytes: stats.reclaimable_bytes(),
|
||||
@@ -565,17 +730,16 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
}
|
||||
Err(e) => CtlResponse::error(CoreError::internal(format!("read db stats: {e:?}"))),
|
||||
},
|
||||
Ok(CtlRequest::CacheVacuum) => match core.db.vacuum() {
|
||||
Ok(outcome) => CtlResponse::Ok {
|
||||
message: format!(
|
||||
"vacuumed: {:.1} MiB freed ({:.1} MiB → {:.1} MiB), {} WAL frame(s) checkpointed",
|
||||
outcome.freed_bytes() as f64 / 1_048_576.0,
|
||||
outcome.before_bytes as f64 / 1_048_576.0,
|
||||
outcome.after_bytes as f64 / 1_048_576.0,
|
||||
outcome.wal_frames_checkpointed,
|
||||
),
|
||||
// Ack and vacuum on a thread. `VACUUM` rewrites the whole database
|
||||
// under the connection mutex — minutes on a large one — and the
|
||||
// request's own socket timeout is 10 s, so running it inline both froze
|
||||
// the mount and lost its own answer. Progress shows up as a job in
|
||||
// `GetQueueStatus`; the outcome goes to the log.
|
||||
Ok(CtlRequest::CacheVacuum) => match core.start_vacuum() {
|
||||
Ok(()) => CtlResponse::Ok {
|
||||
message: "compacting the cache database in the background".into(),
|
||||
},
|
||||
Err(e) => CtlResponse::error(CoreError::internal(format!("vacuum: {e:?}"))),
|
||||
Err(e) => CtlResponse::error(e),
|
||||
},
|
||||
Ok(CtlRequest::GetQueueStatus) => CtlResponse::Transfers {
|
||||
items: core.transfers.snapshot(),
|
||||
@@ -1023,7 +1187,7 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
|
||||
Err(e) => CtlResponse::error(e),
|
||||
},
|
||||
Ok(CtlRequest::ListActivity { limit }) => CtlResponse::Activity {
|
||||
items: core.list_activity(limit),
|
||||
items: core.list_activity(clamp_limit(limit)),
|
||||
},
|
||||
// The request did not parse: the caller sent something malformed.
|
||||
Err(e) => CtlResponse::error(CoreError::invalid(format!("bad request: {e}"))),
|
||||
@@ -1056,6 +1220,12 @@ pub(crate) fn run_control_socket(
|
||||
) {
|
||||
info!("control socket listening");
|
||||
for conn in listener.incoming() {
|
||||
// Checked on every accept, including the deliberate poke teardown sends
|
||||
// to wake this thread out of a blocking accept (bugs.md B44).
|
||||
if core.shutdown.is_stopping() {
|
||||
info!("control socket closing");
|
||||
return;
|
||||
}
|
||||
match conn {
|
||||
Ok(stream) => {
|
||||
let Some(permit) = ControlHandlerPermit::acquire() else {
|
||||
@@ -1118,4 +1288,29 @@ mod request_limit_tests {
|
||||
std::io::ErrorKind::InvalidData
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accepts_paths_inside_the_mountpoint() {
|
||||
let mountpoint = Path::new("/mnt/proton");
|
||||
assert_eq!(
|
||||
rel_to_mount(mountpoint, "Photos/2026").unwrap(),
|
||||
Path::new("Photos/2026")
|
||||
);
|
||||
assert_eq!(
|
||||
rel_to_mount(mountpoint, "/mnt/proton/Photos/2026").unwrap(),
|
||||
Path::new("Photos/2026")
|
||||
);
|
||||
assert_eq!(
|
||||
rel_to_mount(mountpoint, "/mnt/proton").unwrap(),
|
||||
Path::new("")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_paths_that_escape_the_mountpoint() {
|
||||
let mountpoint = Path::new("/mnt/proton");
|
||||
assert!(rel_to_mount(mountpoint, "../escape").is_err());
|
||||
assert!(rel_to_mount(mountpoint, "Photos/../../escape").is_err());
|
||||
assert!(rel_to_mount(mountpoint, "/mnt/elsewhere").is_err());
|
||||
}
|
||||
}
|
||||
@@ -536,7 +536,14 @@ impl Core {
|
||||
// folder the daemon no longer tracks, and nothing would ever unmount it.
|
||||
// Tear it down first — including before trashing the remote tree it
|
||||
// serves, which would otherwise leave it answering for deleted nodes.
|
||||
if let Some(mount) = self.mounts.lock().remove(&id)
|
||||
// Taken in its own statement so the registry lock is released before the
|
||||
// teardown: an edition-2024 `if let` chain holds the guard for the whole
|
||||
// body, and `teardown` unmounts and joins the session thread — which can
|
||||
// be inside a multi-second transfer. The shutdown path takes the same
|
||||
// lock, so holding it here risks systemd's stop timeout landing mid-
|
||||
// unmount and stranding the endpoint.
|
||||
let taken = self.mounts.lock().remove(&id);
|
||||
if let Some(mount) = taken
|
||||
&& let Err(error) = commit_after_teardown(mount.teardown(), || ())
|
||||
{
|
||||
let _ = self.db.sync_folder_set_state(id, "error", now_secs());
|
||||
@@ -584,19 +591,8 @@ impl Core {
|
||||
pub(crate) fn fork_state(&self) -> Core {
|
||||
let mut fork = self.clone();
|
||||
fork.primary = false;
|
||||
let share_access = self.state.lock().share_access.clone();
|
||||
fork.state = Arc::new(Mutex::new(State {
|
||||
entries: HashMap::new(),
|
||||
by_uid: HashMap::new(),
|
||||
children: HashMap::new(),
|
||||
next_ino: 2,
|
||||
active_writes: HashMap::new(),
|
||||
handles: HashMap::new(),
|
||||
next_fh: 1,
|
||||
access_changes: std::collections::HashSet::new(),
|
||||
share_access,
|
||||
db: self.db.clone(),
|
||||
}));
|
||||
let share_access = self.state().share_access.clone();
|
||||
fork.state = Arc::new(Mutex::new(State::new(self.db.clone(), share_access, 2)));
|
||||
// A fresh inode space needs a fresh notification channel: this fork's
|
||||
// session is the only one that knows these inodes, so it must be the one
|
||||
// notified about them. Filled in by `spawn_ondemand_mount`.
|
||||
@@ -866,8 +862,11 @@ impl Core {
|
||||
Ok(format!("{} is now on-demand", local.display()))
|
||||
}
|
||||
ApplicableMountMode::Mirror => {
|
||||
// ondemand→mirror: tear down the secondary mount first.
|
||||
if let Some(mount) = self.mounts.lock().remove(&id)
|
||||
// ondemand→mirror: tear down the secondary mount first. Taken
|
||||
// before the teardown so the registry lock is not held across
|
||||
// the unmount and thread join (see `remove_sync_folder`).
|
||||
let taken = self.mounts.lock().remove(&id);
|
||||
if let Some(mount) = taken
|
||||
&& let Err(error) = commit_after_teardown(mount.teardown(), || ())
|
||||
{
|
||||
let _ = self.db.sync_folder_set_state(id, "error", now_secs());
|
||||
|
||||
+777
-109
File diff suppressed because it is too large.
Load diff
+517
-254
File diff suppressed because it is too large.
Load diff
+1665
-274
File diff suppressed because it is too large.
Load diff
+118
-41
@@ -255,6 +255,14 @@ pub(super) fn spawn_session(
|
||||
root: Node,
|
||||
) -> std::io::Result<BackgroundSession> {
|
||||
let mut config = Config::default();
|
||||
// `n_threads` stays at fuser's default of 1 (with `clone_fd` off) on
|
||||
// purpose, for now. A concurrent dispatch loop is worth having, but
|
||||
// `release` currently relies on being serialised against the `open`+`read`
|
||||
// an application issues right after `close(2)` — see the comment on
|
||||
// `Filesystem::release`. Raising it before that ordering is made explicit
|
||||
// (a per-node staging-in-flight barrier) reintroduces the acceptance
|
||||
// suite's "concurrent file 1 mismatch". Kernel-side caps that do *not*
|
||||
// depend on that ordering are negotiated in `ProtonFs::init`.
|
||||
config.mount_options = vec![
|
||||
MountOption::FSName("protondrive".to_string()),
|
||||
MountOption::Subtype("protondrive".to_string()),
|
||||
@@ -349,23 +357,17 @@ pub fn mount(
|
||||
let core = Core {
|
||||
client: client.clone(),
|
||||
rt: rt.clone(),
|
||||
maintenance: Arc::new(Mutex::new(Default::default())),
|
||||
primary_root_uid: root.uid.clone(),
|
||||
primary: true,
|
||||
state: Arc::new(Mutex::new(State {
|
||||
entries: HashMap::new(),
|
||||
by_uid: HashMap::new(),
|
||||
children: HashMap::new(),
|
||||
next_ino: 2,
|
||||
active_writes: HashMap::new(),
|
||||
handles: HashMap::new(),
|
||||
next_fh: 1,
|
||||
access_changes: HashSet::new(),
|
||||
share_access,
|
||||
db: db.clone(),
|
||||
})),
|
||||
state: Arc::new(Mutex::new(State::new(db.clone(), share_access, 2))),
|
||||
cache: Arc::new(cache),
|
||||
readers: Arc::new(Mutex::new(HashMap::new())),
|
||||
stream_ring: Arc::new(Mutex::new(StreamRing::default())),
|
||||
block_ring: Arc::new(Mutex::new(BlockRing::default())),
|
||||
block_flight: Arc::new(BlockFlight::default()),
|
||||
content_repairs: Arc::new(tokio::sync::Mutex::new(HashMap::new())),
|
||||
prefetch: Arc::new(Prefetch::default()),
|
||||
prefetch_budget: Arc::new(tokio::sync::Semaphore::new(PREFETCH_BUDGET)),
|
||||
workers: Arc::new(Workers::new(FUSE_WORKERS)?),
|
||||
db,
|
||||
shared_publication: Arc::new(Mutex::new(())),
|
||||
@@ -375,6 +377,9 @@ pub fn mount(
|
||||
pending: Arc::new(Mutex::new(HashMap::new())),
|
||||
hidden: Arc::new(Mutex::new(HashSet::new())),
|
||||
drain_wake: Arc::new((Mutex::new(false), Condvar::new())),
|
||||
shutdown: Arc::new(crate::shutdown::Shutdown::default()),
|
||||
upload_times: Arc::new(Mutex::new(HashMap::new())),
|
||||
upload_cancel: Arc::new(Mutex::new(HashMap::new())),
|
||||
timeline_refreshing: Arc::new(AtomicBool::new(false)),
|
||||
albums_refreshing: Arc::new(AtomicBool::new(false)),
|
||||
trash_refreshing: Arc::new(AtomicBool::new(false)),
|
||||
@@ -384,8 +389,18 @@ pub fn mount(
|
||||
own_sealed_revs: Arc::new(Mutex::new(HashMap::new())),
|
||||
self_changes: Arc::new(Mutex::new(HashMap::new())),
|
||||
thumb_gen: Arc::new(Mutex::new(HashSet::new())),
|
||||
no_thumbnail: Arc::new(Mutex::new(HashMap::new())),
|
||||
thumb_gen_budget: Arc::new(tokio::sync::Semaphore::new(
|
||||
super::photos::THUMB_GEN_CONCURRENCY,
|
||||
)),
|
||||
file_thumb_generation: Arc::new(AtomicU64::new(0)),
|
||||
file_thumb_cancel: Arc::new(tokio::sync::Notify::new()),
|
||||
thumbnail_build: Arc::new(Mutex::new(Default::default())),
|
||||
thumbnail_build_cancelled: Arc::new(AtomicBool::new(false)),
|
||||
thumbnail_build_cancel: Arc::new(tokio::sync::Notify::new()),
|
||||
thumbnail_misses: Arc::new(Mutex::new(Default::default())),
|
||||
quota: Arc::new(Mutex::new(None)),
|
||||
size_upgrades: Arc::new(Mutex::new(HashMap::new())),
|
||||
size_waiters: Arc::new(Default::default()),
|
||||
notifier: Arc::new(OnceLock::new()),
|
||||
session_live: Arc::new(AtomicBool::new(false)),
|
||||
transfers: TransferRegistry::new(),
|
||||
@@ -407,16 +422,32 @@ pub fn mount(
|
||||
// aside at open. After `hydrate_pending`, so a recovered partial write can
|
||||
// see an already-queued write to the same node.
|
||||
core.recover_fsynced_writes();
|
||||
{
|
||||
// And finally the staged blobs neither of those accounts for: bytes a
|
||||
// failed release left behind with nothing pointing at them.
|
||||
core.reconcile_staging();
|
||||
// Several drain workers over one queue. One thread meant a single large
|
||||
// upload held every queued rename, trash and small write behind it for as
|
||||
// long as it ran; the claim column (`Db::claim_next_due_op`) keeps the
|
||||
// workers off each other's nodes, which is the only ordering the queue
|
||||
// actually needs. Worker 0 additionally runs the queue's idle chores.
|
||||
//
|
||||
// Every long-lived thread started from here on is kept in `workers` and
|
||||
// joined at the end of this function. Leaving them running turned an
|
||||
// in-process remount into a second generation of drain workers, sync engines
|
||||
// and sweeps all operating on a mount that no longer exists (bugs.md B44).
|
||||
let mut workers: Vec<std::thread::JoinHandle<()>> = Vec::new();
|
||||
for worker in 0..DRAIN_WORKERS {
|
||||
let core = core.clone();
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-drain".into())
|
||||
.spawn(move || core.run_pending_drain())?;
|
||||
workers.push(
|
||||
std::thread::Builder::new()
|
||||
.name(format!("pdfs-drain-{worker}"))
|
||||
.spawn(move || core.run_pending_drain(worker == 0))?,
|
||||
);
|
||||
}
|
||||
|
||||
// Start the folder-sync engine. It watches every mirror folder, polls the
|
||||
// remotes, and reconciles on its own thread — never in front of a FUSE call.
|
||||
sync::spawn(core.clone(), sync_rx);
|
||||
workers.extend(sync::spawn(core.clone(), sync_rx));
|
||||
|
||||
// Reconcile leftover `(sync-conflict …)` copies: drop the ones that turned
|
||||
// out identical to the live file, surface the ones that genuinely diverge.
|
||||
@@ -428,18 +459,22 @@ pub fn mount(
|
||||
} else {
|
||||
info!(mode = ?sweep_mode, "conflict sweep enabled");
|
||||
let core = core.clone();
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-conflict-sweep".into())
|
||||
.spawn(move || core.run_conflict_sweep_loop())?;
|
||||
workers.push(
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-conflict-sweep".into())
|
||||
.spawn(move || core.run_conflict_sweep_loop())?,
|
||||
);
|
||||
}
|
||||
|
||||
// Mounted from the cache: watch for the network coming back so the mount can
|
||||
// stop being read-only-ish without the user restarting the daemon.
|
||||
if !online {
|
||||
let core = core.clone();
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-online-probe".into())
|
||||
.spawn(move || core.run_online_probe())?;
|
||||
workers.push(
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-online-probe".into())
|
||||
.spawn(move || core.run_online_probe())?,
|
||||
);
|
||||
}
|
||||
|
||||
// Keep the launcher prompt's "This computer" index fresh. Its own thread:
|
||||
@@ -449,9 +484,12 @@ pub fn mount(
|
||||
let indexing = core.indexing.clone();
|
||||
let transfers = core.transfers.clone();
|
||||
let mountpoint = mountpoint.to_path_buf();
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-localindex".into())
|
||||
.spawn(move || run_local_index(db, indexing, transfers, mountpoint))?;
|
||||
let shutdown = core.shutdown.clone();
|
||||
workers.push(
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-localindex".into())
|
||||
.spawn(move || run_local_index(db, indexing, transfers, mountpoint, shutdown))?,
|
||||
);
|
||||
}
|
||||
|
||||
// Bind the control socket before the FUSE session takes over the thread. A
|
||||
@@ -488,23 +526,29 @@ pub fn mount(
|
||||
let control_core = core.clone();
|
||||
let username = username.clone();
|
||||
let mountpoint = mountpoint.to_path_buf();
|
||||
if let Err(error) = std::thread::Builder::new()
|
||||
match std::thread::Builder::new()
|
||||
.name("pdfs-control".into())
|
||||
.spawn(move || run_control_socket(control_core, username, mountpoint, listener))
|
||||
{
|
||||
let secondaries = core.mounts.lock().close_and_drain();
|
||||
for (id, mount) in secondaries {
|
||||
if let Err(unmount_error) = mount.teardown() {
|
||||
warn!(id, error = %unmount_error, "secondary teardown after control startup failure failed");
|
||||
Ok(handle) => workers.push(handle),
|
||||
Err(error) => {
|
||||
// A failure here abandons the mount, so the background threads
|
||||
// started above have to be told to stop with it.
|
||||
stop_workers(&core, control_socket, workers);
|
||||
let secondaries = core.mounts.lock().close_and_drain();
|
||||
for (id, mount) in secondaries {
|
||||
if let Err(unmount_error) = mount.teardown() {
|
||||
warn!(id, error = %unmount_error, "secondary teardown after control startup failure failed");
|
||||
}
|
||||
}
|
||||
if let Err(unmount_error) = teardown_session(&core.session_live, || {
|
||||
umount_session_unblocked(bg, main_conn)
|
||||
}) {
|
||||
warn!(error = %unmount_error, "unmount after control startup failure failed");
|
||||
}
|
||||
let _ = std::fs::remove_file(control_socket);
|
||||
return Err(error);
|
||||
}
|
||||
if let Err(unmount_error) = teardown_session(&core.session_live, || {
|
||||
umount_session_unblocked(bg, main_conn)
|
||||
}) {
|
||||
warn!(error = %unmount_error, "unmount after control startup failure failed");
|
||||
}
|
||||
let _ = std::fs::remove_file(control_socket);
|
||||
return Err(error);
|
||||
}
|
||||
}
|
||||
// Re-establish on-demand mounts only after the primary session is live.
|
||||
@@ -613,10 +657,43 @@ pub fn mount(
|
||||
}
|
||||
}
|
||||
|
||||
let _ = std::fs::remove_file(control_socket);
|
||||
// Last, because the workers may legitimately be finishing an upload, and
|
||||
// nothing they can still do is harmful once the mounts are down.
|
||||
stop_workers(&core, control_socket, workers);
|
||||
Ok(outcome)
|
||||
}
|
||||
|
||||
/// Signal every background loop, wake the ones that are blocked, and join them.
|
||||
///
|
||||
/// The mount used to return with its drain workers, sync engine, sweep, online
|
||||
/// probe, indexer and control listener all still running on `Core` clones. The
|
||||
/// process usually exited straight afterwards and hid it; an in-process remount
|
||||
/// did not, and started a second full set on top of the first (bugs.md B44).
|
||||
///
|
||||
/// Two of the waits need waking rather than merely flagging: the drain workers
|
||||
/// block on their own condvar (`wake_drain`), and the control listener blocks
|
||||
/// inside `accept`, which only returns when a connection arrives — so it gets
|
||||
/// one, from us, after the flag is already set. A join that takes a while is
|
||||
/// expected and correct: a worker part-way through an upload finishes it rather
|
||||
/// than abandoning the user's bytes mid-flight.
|
||||
fn stop_workers(core: &Core, control_socket: &Path, workers: Vec<std::thread::JoinHandle<()>>) {
|
||||
core.shutdown.stop();
|
||||
core.wake_drain();
|
||||
let _ = core.sync_tx.send(sync::SyncMsg::Stop);
|
||||
// Connect while the socket still exists — that is the whole point of the
|
||||
// poke — then unlink it so nothing else can arrive behind us. The listener
|
||||
// re-checks the flag after every accept, so even a connection that races the
|
||||
// unlink cannot put it back to sleep.
|
||||
let _ = std::os::unix::net::UnixStream::connect(control_socket);
|
||||
let _ = std::fs::remove_file(control_socket);
|
||||
for worker in workers {
|
||||
if worker.join().is_err() {
|
||||
warn!("a background worker panicked before shutdown");
|
||||
}
|
||||
}
|
||||
debug!("background workers stopped");
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
+983
-39
File diff suppressed because it is too large.
Load diff
+797
-107
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,327 @@
|
||||
//! Repairing photos whose capture time is when they were *imported* rather than
|
||||
//! when they were taken.
|
||||
//!
|
||||
//! A Google Photos Takeout whose metadata sidecars did not match — a localized
|
||||
//! `.supplemental-metadata` spelling, say — imports with no capture time at all,
|
||||
//! and the SDK then seals "now" into every revision. A decade of photos lands on
|
||||
//! one day.
|
||||
//!
|
||||
//! Proton has no API for editing a sealed capture time, so the repair is a
|
||||
//! rewrite: download the photo, upload it again with the capture time its file
|
||||
//! name implies, put the new copy back into the albums and favourites the old
|
||||
//! one was in, then **trash** (not delete) the original. Trash rather than
|
||||
//! delete so a bad run is recoverable from Proton's own trash for 30 days.
|
||||
//!
|
||||
//! A photo is a candidate only when its name carries a date that disagrees with
|
||||
//! its stored capture time by more than [`DRIFT`]. That is deliberately narrow:
|
||||
//! a photo whose name says nothing about its date is left alone, because there
|
||||
//! is nothing better to set it to.
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
use pdfs_core::control::{RedateSummary, TransferDirection};
|
||||
use pdfs_core::takeout::{capture_time_from_name, media_type_of};
|
||||
use pdfs_core::{CoreError, CoreResult};
|
||||
use proton_drive_rs::proton_sdk::ids::NodeUid;
|
||||
use proton_drive_rs::{PhotoTag, PhotoTagsUpdate, PhotoUploadMetadata};
|
||||
use tracing::{info, warn};
|
||||
|
||||
use super::Core;
|
||||
use crate::parse_uid as parse_node_uid;
|
||||
use crate::transfers::CountingReader;
|
||||
|
||||
/// How far a stored capture time may sit from the one the file name implies
|
||||
/// before the photo counts as mis-dated. A day, because a name usually carries
|
||||
/// only the date and the stored time carries a clock — and because a timezone
|
||||
/// can legitimately move a photo by hours.
|
||||
const DRIFT: i64 = 86_400;
|
||||
|
||||
/// How many examples a report carries.
|
||||
const SAMPLES: usize = 10;
|
||||
|
||||
/// The one re-date this daemon may be running, and how the last one ended.
|
||||
/// Process-global for the same reasons as the Takeout import's state: one run
|
||||
/// per daemon however many `Core` clones exist, and a front-end asking after it
|
||||
/// finished must still find the report.
|
||||
static REDATE: std::sync::LazyLock<parking_lot::Mutex<RedateState>> =
|
||||
std::sync::LazyLock::new(|| parking_lot::Mutex::new(RedateState::default()));
|
||||
|
||||
#[derive(Default)]
|
||||
struct RedateState {
|
||||
running: bool,
|
||||
cancel: Option<Arc<AtomicBool>>,
|
||||
last: Option<RedateSummary>,
|
||||
}
|
||||
|
||||
/// Whether a re-date is running, and the last one's report.
|
||||
pub(crate) fn redate_status() -> (bool, Option<RedateSummary>) {
|
||||
let state = REDATE.lock();
|
||||
(state.running, state.last.clone())
|
||||
}
|
||||
|
||||
/// Ask the running re-date to stop. False when none is running.
|
||||
pub(crate) fn cancel_redate() -> bool {
|
||||
match &REDATE.lock().cancel {
|
||||
Some(flag) => {
|
||||
flag.store(true, Ordering::Relaxed);
|
||||
true
|
||||
}
|
||||
None => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// One photo to repair.
|
||||
struct Candidate {
|
||||
uid: NodeUid,
|
||||
name: String,
|
||||
/// The capture time it carries now.
|
||||
stored: i64,
|
||||
/// The capture time its name implies.
|
||||
wanted: i64,
|
||||
favorite: bool,
|
||||
/// Album node uids the original is filed under.
|
||||
albums: Vec<String>,
|
||||
}
|
||||
|
||||
impl Core {
|
||||
/// Re-date mis-dated photos. Blocking; run on its own thread.
|
||||
///
|
||||
/// `cancel` stops the run at the next photo boundary. A photo is only ever
|
||||
/// trashed after its replacement has uploaded, so a cancel — or a crash —
|
||||
/// can leave a duplicate, never a hole.
|
||||
pub(crate) fn redate_photos(
|
||||
&self,
|
||||
dry_run: bool,
|
||||
limit: Option<usize>,
|
||||
range: Option<(i64, i64)>,
|
||||
cancel: Arc<AtomicBool>,
|
||||
) -> CoreResult<RedateSummary> {
|
||||
let job = self.transfers.begin_job("Fixing photo dates");
|
||||
job.detail("looking for mis-dated photos");
|
||||
|
||||
let stored = self
|
||||
.db
|
||||
.photos_all()
|
||||
.map_err(|e| CoreError::internal(format!("read timeline: {e}")))?;
|
||||
let mut summary = RedateSummary {
|
||||
examined: stored.len(),
|
||||
dry_run,
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let mut candidates = Vec::new();
|
||||
for photo in stored {
|
||||
if let Some((from, to)) = range
|
||||
&& !(from..to).contains(&photo.capture_time)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let Some(name) = photo.name.filter(|n| !n.is_empty()) else {
|
||||
continue;
|
||||
};
|
||||
let Some(wanted) = capture_time_from_name(&name) else {
|
||||
continue;
|
||||
};
|
||||
if (wanted - photo.capture_time).abs() <= DRIFT {
|
||||
continue;
|
||||
}
|
||||
let Some(uid) = parse_node_uid(&photo.uid) else {
|
||||
continue;
|
||||
};
|
||||
candidates.push(Candidate {
|
||||
uid,
|
||||
name,
|
||||
stored: photo.capture_time,
|
||||
wanted,
|
||||
favorite: photo.favorite,
|
||||
albums: self.db.albums_of_photo(&photo.uid).unwrap_or_default(),
|
||||
});
|
||||
}
|
||||
if let Some(limit) = limit {
|
||||
candidates.truncate(limit);
|
||||
}
|
||||
summary.candidates = candidates.len();
|
||||
// Sampled after the limit, so the examples are of photos the run would
|
||||
// actually touch.
|
||||
summary.samples = candidates
|
||||
.iter()
|
||||
.take(SAMPLES)
|
||||
.map(|c| (c.name.clone(), c.stored, c.wanted))
|
||||
.collect();
|
||||
job.set_total(candidates.len() as u64);
|
||||
info!(
|
||||
examined = summary.examined,
|
||||
candidates = summary.candidates,
|
||||
dry_run,
|
||||
"photo re-date scan complete"
|
||||
);
|
||||
|
||||
if dry_run || candidates.is_empty() {
|
||||
return Ok(summary);
|
||||
}
|
||||
|
||||
for candidate in candidates {
|
||||
if cancel.load(Ordering::Relaxed) || self.shutdown.is_stopping() {
|
||||
break;
|
||||
}
|
||||
job.detail(candidate.name.clone());
|
||||
if let Err(e) = self.redate_one(&candidate, &mut summary) {
|
||||
summary.failed += 1;
|
||||
warn!(name = candidate.name, error = %e, "cannot re-date photo");
|
||||
}
|
||||
job.step();
|
||||
}
|
||||
|
||||
self.spawn_timeline_refresh();
|
||||
summary.cancelled = cancel.load(Ordering::Relaxed);
|
||||
Ok(summary)
|
||||
}
|
||||
|
||||
/// Rewrite one photo at the capture time its name implies.
|
||||
///
|
||||
/// Order is load-bearing: upload first, file the new copy, and only then
|
||||
/// trash the original. Every earlier step failing leaves the account exactly
|
||||
/// as it was.
|
||||
fn redate_one(&self, candidate: &Candidate, summary: &mut RedateSummary) -> CoreResult<()> {
|
||||
let photos = self.photos();
|
||||
let path = self.open_photo(&candidate.uid)?;
|
||||
let size = std::fs::metadata(&path)
|
||||
.map_err(|e| CoreError::internal(format!("stat {}: {e}", path.display())))?
|
||||
.len();
|
||||
let media_type = media_type_of(&candidate.name).unwrap_or("application/octet-stream");
|
||||
|
||||
let guard =
|
||||
self.transfers
|
||||
.begin(candidate.name.clone(), "", TransferDirection::Upload, size);
|
||||
let file = std::fs::File::open(&path)
|
||||
.map_err(|e| CoreError::internal(format!("open {}: {e}", path.display())))?;
|
||||
let reader = CountingReader::new(std::io::BufReader::new(file), &guard);
|
||||
let metadata = PhotoUploadMetadata {
|
||||
capture_time: Some(candidate.wanted),
|
||||
..Default::default()
|
||||
};
|
||||
let uploaded = self.rt.block_on(photos.upload_photo_from(
|
||||
&candidate.name,
|
||||
media_type,
|
||||
reader,
|
||||
size as i64,
|
||||
Vec::new(),
|
||||
metadata,
|
||||
false,
|
||||
));
|
||||
drop(guard);
|
||||
let new_uid = uploaded.map_err(|e| CoreError::from_api(&e, "re-upload photo"))?;
|
||||
|
||||
// Favourite and albums before the trash: a failure here leaves both
|
||||
// copies on the account, which is recoverable by hand. Losing the
|
||||
// original first would not be.
|
||||
if candidate.favorite {
|
||||
let update = PhotoTagsUpdate {
|
||||
node_uid: new_uid.clone(),
|
||||
tags_to_add: vec![PhotoTag::Favorite],
|
||||
tags_to_remove: Vec::new(),
|
||||
};
|
||||
if let Err(e) = self.rt.block_on(photos.update_photos(&[update])) {
|
||||
warn!(name = candidate.name, error = %e, "cannot re-favourite re-dated photo");
|
||||
}
|
||||
}
|
||||
for album in &candidate.albums {
|
||||
let Some(album_uid) = parse_node_uid(album) else {
|
||||
continue;
|
||||
};
|
||||
match self
|
||||
.rt
|
||||
.block_on(photos.add_photos_to_album(&album_uid, std::slice::from_ref(&new_uid)))
|
||||
{
|
||||
Ok(_) => summary.album_links += 1,
|
||||
Err(e) => {
|
||||
warn!(name = candidate.name, error = %e, "cannot re-file re-dated photo")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let outcomes = self
|
||||
.rt
|
||||
.block_on(
|
||||
self.client
|
||||
.trash_nodes(std::slice::from_ref(&candidate.uid)),
|
||||
)
|
||||
.map_err(|e| CoreError::from_api(&e, "trash the mis-dated original"))?;
|
||||
if let Some((_, Err(e))) = outcomes.into_iter().next() {
|
||||
return Err(CoreError::from_api(&e, "trash the mis-dated original"));
|
||||
}
|
||||
|
||||
summary.redated += 1;
|
||||
summary.bytes += size;
|
||||
info!(
|
||||
name = candidate.name,
|
||||
from = candidate.stored,
|
||||
to = candidate.wanted,
|
||||
"photo re-dated"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Start a re-date on its own thread. Errors when one is already running.
|
||||
pub(crate) fn spawn_redate(
|
||||
&self,
|
||||
dry_run: bool,
|
||||
limit: Option<usize>,
|
||||
range: Option<(i64, i64)>,
|
||||
) -> CoreResult<()> {
|
||||
let cancel = Arc::new(AtomicBool::new(false));
|
||||
{
|
||||
let mut state = REDATE.lock();
|
||||
if state.running {
|
||||
return Err(CoreError::conflict(
|
||||
"a photo re-date is already running; wait for it to finish or cancel it",
|
||||
));
|
||||
}
|
||||
state.running = true;
|
||||
state.cancel = Some(cancel.clone());
|
||||
state.last = None;
|
||||
}
|
||||
|
||||
let core = self.clone();
|
||||
std::thread::spawn(move || {
|
||||
let outcome = core.redate_photos(dry_run, limit, range, cancel);
|
||||
let mut state = REDATE.lock();
|
||||
state.running = false;
|
||||
state.cancel = None;
|
||||
state.last = outcome.as_ref().ok().cloned();
|
||||
match &outcome {
|
||||
Ok(summary) => info!(
|
||||
candidates = summary.candidates,
|
||||
redated = summary.redated,
|
||||
failed = summary.failed,
|
||||
"photo re-date finished"
|
||||
),
|
||||
Err(e) => warn!(error = %e, "photo re-date failed"),
|
||||
}
|
||||
});
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn a_uid_without_both_halves_is_not_a_node() {
|
||||
assert!(parse_node_uid("vol~link").is_some());
|
||||
assert!(parse_node_uid("vol~").is_none());
|
||||
assert!(parse_node_uid("~link").is_none());
|
||||
assert!(parse_node_uid("nothing").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn only_a_disagreement_larger_than_a_day_counts() {
|
||||
// The name says 2023-02-19; a stored time inside that day is fine.
|
||||
let wanted = capture_time_from_name("IMG-20230219-WA0001.jpg").unwrap();
|
||||
assert!((wanted - (wanted + 3_600)).abs() <= DRIFT);
|
||||
// An import three years later is not.
|
||||
assert!((wanted - (wanted + 3 * 365 * 86_400)).abs() > DRIFT);
|
||||
}
|
||||
}
|
||||
@@ -12,6 +12,10 @@ use pdfs_core::control::{
|
||||
BookmarkInfo, DirEntry, InvitationInfo, PublicLinkInfo, ShareEntry, ShareEntryKind, SharedItem,
|
||||
};
|
||||
use pdfs_core::{CoreError, CoreResult};
|
||||
use tracing::debug;
|
||||
|
||||
use crate::QUOTA_TTL;
|
||||
use proton_drive_rs::proton_sdk::crypto::VerificationStatus;
|
||||
use proton_drive_rs::proton_sdk::ids::NodeUid;
|
||||
use proton_drive_rs::{MemberRole, NodeKind};
|
||||
|
||||
@@ -44,6 +48,18 @@ fn shared_entry_of(core: &Core, n: Node) -> DirEntry {
|
||||
.map(role_to_str)
|
||||
.unwrap_or_default()
|
||||
.to_string();
|
||||
let invitation = n.membership.as_ref();
|
||||
let shared_by = invitation
|
||||
.and_then(|m| m.inviter_email.clone())
|
||||
.unwrap_or_default();
|
||||
let shared_at = invitation.and_then(|m| m.invite_time).unwrap_or(0);
|
||||
// Only a signature that is present and wrong is a warning. An invitation
|
||||
// that could not be checked — unsigned, or no keys for the claimed inviter —
|
||||
// is not evidence of forgery, and flagging every such share would teach the
|
||||
// user to ignore the flag.
|
||||
let shared_by_unverified = invitation
|
||||
.and_then(|m| m.inviter_verification)
|
||||
.is_some_and(|status| status == VerificationStatus::Failed);
|
||||
DirEntry {
|
||||
name: n.name,
|
||||
is_dir,
|
||||
@@ -54,6 +70,9 @@ fn shared_entry_of(core: &Core, n: Node) -> DirEntry {
|
||||
path: core.rel_path_for_uid(&n.uid).unwrap_or_default(),
|
||||
uid: n.uid.to_string(),
|
||||
role,
|
||||
shared_by,
|
||||
shared_at,
|
||||
shared_by_unverified,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -533,9 +552,33 @@ impl Core {
|
||||
.rt
|
||||
.block_on(self.client.quota())
|
||||
.map_err(|e| CoreError::from_api(&e, "account quota"))?;
|
||||
*self.quota.lock() = Some((std::time::Instant::now(), q.max_space, q.used_space));
|
||||
Ok((q.max_space, q.used_space))
|
||||
}
|
||||
|
||||
/// The account quota as `statfs(2)` needs it: cached for [`QUOTA_TTL`],
|
||||
/// refetched after that, and `None` when there is no usable answer at all —
|
||||
/// offline with nothing cached.
|
||||
///
|
||||
/// Never blocks on the network for a *fresh* value when a recent one exists,
|
||||
/// because the callers are `df` and every "is there room?" preflight, and a
|
||||
/// stale byte count is worth more to them than a stalled syscall. Offline,
|
||||
/// the last known figures keep being served rather than degrading to zero.
|
||||
pub(crate) fn account_quota_cached(&self) -> Option<(i64, i64)> {
|
||||
if let Some((at, max_space, used_space)) = *self.quota.lock()
|
||||
&& at.elapsed() < QUOTA_TTL
|
||||
{
|
||||
return Some((max_space, used_space));
|
||||
}
|
||||
match self.account_quota() {
|
||||
Ok(q) => Some(q),
|
||||
Err(error) => {
|
||||
debug!(%error, "account quota refresh failed; serving the last known figures");
|
||||
self.quota.lock().map(|(_, max, used)| (max, used))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- shared by me -----------------------------------------------------
|
||||
|
||||
/// List the nodes I have shared with others, each with a summary of its share
|
||||
@@ -582,6 +625,7 @@ impl Core {
|
||||
items.push(SharedItem {
|
||||
uid: uid.to_string(),
|
||||
is_dir: n.is_folder(),
|
||||
modified: n.modification_time,
|
||||
name: n.name,
|
||||
path: self.rel_path_for_uid(&uid).unwrap_or_default(),
|
||||
member_count: members,
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
//! One shutdown signal shared by every long-lived worker in the daemon.
|
||||
//!
|
||||
//! The mount used to end by unmounting and returning, leaving the drain
|
||||
//! workers, the sync engine, the conflict sweep, the online probe and the local
|
||||
//! indexer running on `Core` clones. In a process that exits immediately that is
|
||||
//! invisible; in an in-process remount — which is how the daemon recovers from a
|
||||
//! mount going away underneath it — it leaks a whole generation of threads that
|
||||
//! keep writing to the database and calling the API on behalf of a mount that no
|
||||
//! longer exists (bugs.md B44).
|
||||
//!
|
||||
//! The primitive is deliberately small: a flag plus a condvar. Every loop that
|
||||
//! would otherwise call `thread::sleep` calls [`Shutdown::sleep`] instead, which
|
||||
//! returns early — and reports `false` — the moment [`Shutdown::stop`] is
|
||||
//! called, so teardown does not have to wait out a five-minute sweep interval to
|
||||
//! join a thread.
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use parking_lot::{Condvar, Mutex};
|
||||
|
||||
/// The daemon's stop signal. Set once, never cleared: a `Core` that has been
|
||||
/// told to stop belongs to a mount that is going away.
|
||||
#[derive(Default)]
|
||||
pub(crate) struct Shutdown {
|
||||
stopping: Mutex<bool>,
|
||||
changed: Condvar,
|
||||
}
|
||||
|
||||
impl Shutdown {
|
||||
/// Whether teardown has begun. Loops check this where they would otherwise
|
||||
/// start another unit of work.
|
||||
pub(crate) fn is_stopping(&self) -> bool {
|
||||
*self.stopping.lock()
|
||||
}
|
||||
|
||||
/// Begin teardown and wake everything waiting in [`sleep`](Self::sleep).
|
||||
/// Idempotent, because more than one path can reach teardown (a signal, the
|
||||
/// kernel mount ending, a failed startup).
|
||||
pub(crate) fn stop(&self) {
|
||||
let mut stopping = self.stopping.lock();
|
||||
*stopping = true;
|
||||
self.changed.notify_all();
|
||||
}
|
||||
|
||||
/// Wait up to `duration`, or until teardown starts.
|
||||
///
|
||||
/// Returns `false` if the daemon is stopping — including when it was already
|
||||
/// stopping on entry, so `while shutdown.sleep(interval)` is a complete and
|
||||
/// correct loop condition and no separate check is needed at the top.
|
||||
pub(crate) fn sleep(&self, duration: Duration) -> bool {
|
||||
let mut stopping = self.stopping.lock();
|
||||
if !*stopping {
|
||||
self.changed.wait_for(&mut stopping, duration);
|
||||
}
|
||||
!*stopping
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::sync::Arc;
|
||||
use std::time::Instant;
|
||||
|
||||
#[test]
|
||||
fn sleeping_returns_early_once_stopped() {
|
||||
let shutdown = Arc::new(Shutdown::default());
|
||||
assert!(!shutdown.is_stopping());
|
||||
|
||||
let waker = shutdown.clone();
|
||||
std::thread::spawn(move || {
|
||||
std::thread::sleep(Duration::from_millis(20));
|
||||
waker.stop();
|
||||
});
|
||||
|
||||
let started = Instant::now();
|
||||
// An hour's worth of interval, cut short by the stop.
|
||||
assert!(!shutdown.sleep(Duration::from_secs(3600)));
|
||||
assert!(
|
||||
started.elapsed() < Duration::from_secs(5),
|
||||
"stop must interrupt the wait, not wait it out"
|
||||
);
|
||||
assert!(shutdown.is_stopping());
|
||||
// Already stopping: the next wait does not sleep at all.
|
||||
let started = Instant::now();
|
||||
assert!(!shutdown.sleep(Duration::from_secs(3600)));
|
||||
assert!(started.elapsed() < Duration::from_secs(5));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sleeping_runs_to_the_timeout_while_running() {
|
||||
let shutdown = Shutdown::default();
|
||||
assert!(shutdown.sleep(Duration::from_millis(10)));
|
||||
assert!(!shutdown.is_stopping());
|
||||
}
|
||||
}
|
||||
+272
-85
@@ -141,6 +141,11 @@ pub(crate) struct PendingRevision {
|
||||
pub(crate) meta: StagedWrite,
|
||||
}
|
||||
|
||||
/// One directory's entries as `readdir` reports them: `(ino, is_dir, name)` per
|
||||
/// entry, in reply order. The kernel's file-type enum is mapped in at reply
|
||||
/// time so this module stays free of `fuser`.
|
||||
pub(crate) type DirListing = Vec<(u64, bool, String)>;
|
||||
|
||||
/// Mutable inode bookkeeping, guarded by a mutex because fuser drives the
|
||||
/// `Filesystem` trait through `&self`.
|
||||
pub(crate) struct State {
|
||||
@@ -158,6 +163,16 @@ pub(crate) struct State {
|
||||
/// have no entry here.
|
||||
pub(crate) handles: HashMap<u64, u64>,
|
||||
pub(crate) next_fh: u64,
|
||||
/// One frozen listing per open directory handle, taken at `opendir`.
|
||||
///
|
||||
/// `readdir` resumes from a byte offset into the listing it is walking, so
|
||||
/// that listing has to stay the same across the calls of one `getdents`
|
||||
/// sequence. Rebuilding it live meant a create or a trash landing between
|
||||
/// two pages shifted every later entry, and the caller silently skipped or
|
||||
/// repeated whatever crossed the page boundary. Each entry is
|
||||
/// `(ino, is_dir, name)` — the kernel type is mapped at reply time so this
|
||||
/// module stays free of `fuser` (bugs.md B43).
|
||||
pub(crate) dir_snapshots: HashMap<u64, Arc<DirListing>>,
|
||||
/// Resident inode attributes whose effective access changed during an
|
||||
/// intern/root refresh. Core drains this set and notifies the matching
|
||||
/// kernel only after releasing the State lock.
|
||||
@@ -166,13 +181,164 @@ pub(crate) struct State {
|
||||
/// on every explicit role/tombstone change. Descendant inheritance reads
|
||||
/// this map instead of issuing one SQLite query per interned inode.
|
||||
pub(crate) share_access: HashMap<NodeUid, Access>,
|
||||
/// Unified SQLite metadata cache. Every map mutation below writes through to
|
||||
/// it inside the `State` lock so the DB stays the authoritative copy across
|
||||
/// restarts (see plan.md P1).
|
||||
/// Unified SQLite metadata cache. Read from here (`has_children`) but never
|
||||
/// written: mutations go to [`State::outbox`] and are applied after the lock
|
||||
/// is released. See [`StateGuard`].
|
||||
pub(crate) db: Arc<Db>,
|
||||
/// Write-throughs this mutation owes SQLite, applied by [`StateGuard`] once
|
||||
/// the inode lock is released.
|
||||
///
|
||||
/// The DB stays the authoritative copy across restarts, so every map
|
||||
/// mutation still writes through — it just no longer does so with the
|
||||
/// whole-mount lock held. A single listing's write-through is one
|
||||
/// transaction over up to a few thousand rows, and every other FUSE callback
|
||||
/// used to wait behind it for no reason: the maps were already updated
|
||||
/// before the first row was written.
|
||||
outbox: Vec<DbWrite>,
|
||||
}
|
||||
|
||||
/// A write-through queued by a `State` mutation, applied after the lock drops.
|
||||
///
|
||||
/// Deliberately a small closed set rather than a boxed closure: these run
|
||||
/// outside the lock, and what they can touch should be readable here.
|
||||
pub(crate) enum DbWrite {
|
||||
/// One node or a whole listing. Both go through `upsert_nodes`, which is a
|
||||
/// single transaction either way, so there is nothing to gain by splitting
|
||||
/// them — and a `Node` is large enough that a variant holding one inline
|
||||
/// would set the size of every other.
|
||||
Upsert(Vec<Node>),
|
||||
Delete(NodeUid),
|
||||
SetListed(NodeUid, bool),
|
||||
SetShareAccess(NodeUid, Access),
|
||||
}
|
||||
|
||||
impl DbWrite {
|
||||
fn apply(self, db: &Db) {
|
||||
let outcome = match &self {
|
||||
Self::Upsert(nodes) => db.upsert_nodes(nodes),
|
||||
Self::Delete(uid) => db.delete_node(uid),
|
||||
Self::SetListed(uid, listed) => db.set_listed(uid, *listed),
|
||||
Self::SetShareAccess(uid, access) => db.set_share_access(uid, *access),
|
||||
};
|
||||
if let Err(error) = outcome {
|
||||
let what = match &self {
|
||||
Self::Upsert(_) => "upsert_nodes",
|
||||
Self::Delete(_) => "delete_node",
|
||||
Self::SetListed(..) => "set_listed",
|
||||
Self::SetShareAccess(..) => "set_share_access",
|
||||
};
|
||||
warn!(%error, write = what, "db write-through failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The inode lock, held for the duration of a mutation, plus the write-throughs
|
||||
/// that mutation owed the database.
|
||||
///
|
||||
/// Dropping it releases the lock *first* and only then writes, which is the
|
||||
/// whole point: interning a five-thousand-child listing commits one transaction
|
||||
/// — several fsyncs' worth — and nothing about that needs the maps frozen. The
|
||||
/// write still happens synchronously on this thread before the caller proceeds,
|
||||
/// so a read that follows a mutation still sees it.
|
||||
pub(crate) struct StateGuard<'a> {
|
||||
guard: Option<parking_lot::MutexGuard<'a, State>>,
|
||||
db: &'a Arc<Db>,
|
||||
}
|
||||
|
||||
/// Take a mount's inode lock through a [`StateGuard`], for the loops that walk
|
||||
/// every live mount rather than this `Core`'s own.
|
||||
pub(crate) fn lock_state<'a>(
|
||||
state: &'a parking_lot::Mutex<State>,
|
||||
db: &'a Arc<Db>,
|
||||
) -> StateGuard<'a> {
|
||||
StateGuard::new(state.lock(), db)
|
||||
}
|
||||
|
||||
impl<'a> StateGuard<'a> {
|
||||
pub(crate) fn new(guard: parking_lot::MutexGuard<'a, State>, db: &'a Arc<Db>) -> Self {
|
||||
Self {
|
||||
guard: Some(guard),
|
||||
db,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::ops::Deref for StateGuard<'_> {
|
||||
type Target = State;
|
||||
|
||||
fn deref(&self) -> &State {
|
||||
self.guard.as_ref().expect("held until drop")
|
||||
}
|
||||
}
|
||||
|
||||
impl std::ops::DerefMut for StateGuard<'_> {
|
||||
fn deref_mut(&mut self) -> &mut State {
|
||||
self.guard.as_mut().expect("held until drop")
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for StateGuard<'_> {
|
||||
fn drop(&mut self) {
|
||||
let Some(mut guard) = self.guard.take() else {
|
||||
return;
|
||||
};
|
||||
let writes = std::mem::take(&mut guard.outbox);
|
||||
// The lock goes before the writes do. That ordering is the feature.
|
||||
drop(guard);
|
||||
for write in writes {
|
||||
write.apply(self.db);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl State {
|
||||
/// A fresh, empty inode space over `db`.
|
||||
///
|
||||
/// `next_ino` differs by caller: a real mount hands out [`crate::ROOT_INO`]
|
||||
/// to its root and starts allocating above it, while a bare state under test
|
||||
/// has no root and starts at 1.
|
||||
/// Apply the write-throughs this state owes SQLite, here and now.
|
||||
///
|
||||
/// Production code never calls this: [`StateGuard`] releases the inode lock
|
||||
/// first and then applies them, which is the whole point. A test driving a
|
||||
/// bare `State` has no guard, so it calls this where the daemon would have
|
||||
/// dropped one.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn flush_outbox(&mut self) {
|
||||
let db = self.db.clone();
|
||||
for write in std::mem::take(&mut self.outbox) {
|
||||
write.apply(&db);
|
||||
}
|
||||
}
|
||||
|
||||
/// The database, with everything this state owes it already written.
|
||||
///
|
||||
/// What a test means by `st.db` is almost always "the database as of now",
|
||||
/// which in the daemon is the state after the guard dropped. Handing back an
|
||||
/// owned `Arc` rather than a borrow keeps it usable inside an `assert_eq!`
|
||||
/// that also touches `st`.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn flushed_db(&mut self) -> Arc<Db> {
|
||||
self.flush_outbox();
|
||||
self.db.clone()
|
||||
}
|
||||
|
||||
pub(crate) fn new(db: Arc<Db>, share_access: HashMap<NodeUid, Access>, next_ino: u64) -> Self {
|
||||
Self {
|
||||
entries: HashMap::new(),
|
||||
by_uid: HashMap::new(),
|
||||
children: HashMap::new(),
|
||||
next_ino,
|
||||
active_writes: HashMap::new(),
|
||||
handles: HashMap::new(),
|
||||
next_fh: 1,
|
||||
dir_snapshots: HashMap::new(),
|
||||
access_changes: HashSet::new(),
|
||||
share_access,
|
||||
db,
|
||||
outbox: Vec::new(),
|
||||
}
|
||||
}
|
||||
/// Whether `uid` names an owned node currently reachable from this mount's
|
||||
/// root. Resident entries can outlive their dentries for open-handle and
|
||||
/// access-revocation semantics, so `by_uid` membership alone is not proof
|
||||
@@ -278,11 +444,9 @@ impl State {
|
||||
|| stored.unwrap_or(Access::Viewer),
|
||||
|membership| access_for(membership.role_exact(), parent_access, true),
|
||||
);
|
||||
if let Err(error) = self.db.set_share_access(&node.uid, access) {
|
||||
warn!(uid = %node.uid, ?access, %error, "db set_share_access failed");
|
||||
} else {
|
||||
self.share_access.insert(node.uid.clone(), access);
|
||||
}
|
||||
self.outbox
|
||||
.push(DbWrite::SetShareAccess(node.uid.clone(), access));
|
||||
self.share_access.insert(node.uid.clone(), access);
|
||||
return access;
|
||||
}
|
||||
parent_access
|
||||
@@ -313,11 +477,9 @@ impl State {
|
||||
None => (None, Access::Unknown),
|
||||
};
|
||||
let access = access_for(role, parent_access, true);
|
||||
if let Err(error) = self.db.set_share_access(uid, access) {
|
||||
warn!(%uid, ?access, %error, "db set observed share access failed");
|
||||
} else {
|
||||
self.share_access.insert(uid.clone(), access);
|
||||
}
|
||||
self.outbox
|
||||
.push(DbWrite::SetShareAccess(uid.clone(), access));
|
||||
self.share_access.insert(uid.clone(), access);
|
||||
let Some(root) = root else {
|
||||
return Vec::new();
|
||||
};
|
||||
@@ -602,18 +764,14 @@ impl State {
|
||||
})
|
||||
.collect();
|
||||
for (uid, access) in discovered {
|
||||
if let Err(error) = self.db.set_share_access(&uid, access) {
|
||||
warn!(%uid, ?access, %error, "db set hydrated share access failed");
|
||||
} else {
|
||||
self.share_access.insert(uid, access);
|
||||
}
|
||||
self.outbox
|
||||
.push(DbWrite::SetShareAccess(uid.clone(), access));
|
||||
self.share_access.insert(uid, access);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn intern(&mut self, parent: u64, node: Node) -> u64 {
|
||||
if let Err(e) = self.db.upsert_node(&node) {
|
||||
warn!(uid = %node.uid, error = %e, "db upsert_node failed");
|
||||
}
|
||||
self.outbox.push(DbWrite::Upsert(vec![node.clone()]));
|
||||
self.intern_mem(parent, node)
|
||||
}
|
||||
|
||||
@@ -685,9 +843,7 @@ impl State {
|
||||
/// This is what keeps `ls` on a large folder quick: one commit for the
|
||||
/// listing, rather than one autocommit — and one fsync — per child.
|
||||
pub(crate) fn intern_batch(&mut self, parent: u64, nodes: Vec<Node>) -> Vec<u64> {
|
||||
if let Err(e) = self.db.upsert_nodes(&nodes) {
|
||||
warn!(error = %e, "db upsert_nodes failed");
|
||||
}
|
||||
self.outbox.push(DbWrite::Upsert(nodes.clone()));
|
||||
nodes
|
||||
.into_iter()
|
||||
.map(|node| self.intern_mem(parent, node))
|
||||
@@ -769,9 +925,7 @@ impl State {
|
||||
/// `(parent_ino, name)` when the node was known, so the caller can notify
|
||||
/// the kernel.
|
||||
pub(crate) fn forget(&mut self, uid: &NodeUid) -> Option<(u64, String)> {
|
||||
if let Err(e) = self.db.delete_node(uid) {
|
||||
warn!(%uid, error = %e, "db delete_node failed");
|
||||
}
|
||||
self.outbox.push(DbWrite::Delete(uid.clone()));
|
||||
self.forget_mem(uid)
|
||||
}
|
||||
|
||||
@@ -834,9 +988,7 @@ impl State {
|
||||
kids.push(ino);
|
||||
}
|
||||
}
|
||||
if let Err(e) = self.db.upsert_node(&node) {
|
||||
warn!(uid = %node.uid, error = %e, "db upsert_node failed for a queued rename");
|
||||
}
|
||||
self.outbox.push(DbWrite::Upsert(vec![node]));
|
||||
self.recompute_descendant_access(ino);
|
||||
}
|
||||
|
||||
@@ -851,9 +1003,7 @@ impl State {
|
||||
self.children.remove(&ino);
|
||||
if let Some(e) = self.entries.get(&ino) {
|
||||
let uid = e.uid.clone();
|
||||
if let Err(err) = self.db.set_listed(&uid, false) {
|
||||
warn!(%uid, error = %err, "db set_listed(false) failed");
|
||||
}
|
||||
self.outbox.push(DbWrite::SetListed(uid, false));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -920,14 +1070,15 @@ impl State {
|
||||
/// file is as long as the caller was told it is. Without it a restart serves
|
||||
/// the stale size and the file reads as truncated — or empty — while its
|
||||
/// bytes sit safely in staging (offline.md Phase 3).
|
||||
pub(crate) fn record_pending_write(&mut self, ino: u64, size: u64, mtime: i64) {
|
||||
/// Returns the node to persist rather than queueing it in the outbox, and is
|
||||
/// the one mutation here whose write-through the caller must do itself:
|
||||
/// failure has to be reported, not logged. Dropping the row silently means
|
||||
/// the daemon acknowledged a write and then serves the old size for it, so
|
||||
/// the caller has to be able to turn it into a failed `release`.
|
||||
pub(crate) fn record_pending_write(&mut self, ino: u64, size: u64, mtime: i64) -> Option<Node> {
|
||||
self.set_size(ino, size);
|
||||
self.touch_mtime(ino, mtime);
|
||||
if let Some(e) = self.entries.get(&ino)
|
||||
&& let Err(err) = self.db.upsert_node(&e.node)
|
||||
{
|
||||
warn!(uid = %e.uid, error = %err, "db upsert_node failed for a queued write");
|
||||
}
|
||||
self.entries.get(&ino).map(|e| e.node.clone())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -969,6 +1120,8 @@ mod tests {
|
||||
photo: None,
|
||||
album: None,
|
||||
verification: Default::default(),
|
||||
direct_role: None,
|
||||
share_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -977,6 +1130,9 @@ mod tests {
|
||||
share_id: ShareId::from("share"),
|
||||
membership_id: ShareMembershipId::from("membership"),
|
||||
permissions,
|
||||
invite_time: None,
|
||||
inviter_email: None,
|
||||
inviter_verification: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1008,18 +1164,7 @@ mod tests {
|
||||
let dir = TempDir::new();
|
||||
let db = Db::open(&dir.0.join("cache.db")).unwrap();
|
||||
let share_access = db.all_share_access().unwrap();
|
||||
let st = State {
|
||||
entries: HashMap::new(),
|
||||
by_uid: HashMap::new(),
|
||||
children: HashMap::new(),
|
||||
next_ino: 1,
|
||||
active_writes: HashMap::new(),
|
||||
handles: HashMap::new(),
|
||||
next_fh: 1,
|
||||
access_changes: HashSet::new(),
|
||||
share_access,
|
||||
db: Arc::new(db),
|
||||
};
|
||||
let st = State::new(Arc::new(db), share_access, 1);
|
||||
(st, dir)
|
||||
}
|
||||
|
||||
@@ -1032,8 +1177,8 @@ mod tests {
|
||||
let f = st.intern(src, node("f", "src", "f.txt", false));
|
||||
st.children.insert(src, vec![f]);
|
||||
st.children.insert(dst, vec![]);
|
||||
st.db.set_listed(&uid("src"), true).unwrap();
|
||||
st.db.set_listed(&uid("dst"), true).unwrap();
|
||||
st.flushed_db().set_listed(&uid("src"), true).unwrap();
|
||||
st.flushed_db().set_listed(&uid("dst"), true).unwrap();
|
||||
(st, dir, src, dst)
|
||||
}
|
||||
|
||||
@@ -1047,11 +1192,17 @@ mod tests {
|
||||
st.relocate(f, src, dst, &uid("dst"), "f.txt");
|
||||
|
||||
assert!(
|
||||
st.db.children_if_listed(&uid("dst")).unwrap().is_none(),
|
||||
st.flushed_db()
|
||||
.children_if_listed(&uid("dst"))
|
||||
.unwrap()
|
||||
.is_none(),
|
||||
"destination re-enumerates: it gained a child whose row was just deleted"
|
||||
);
|
||||
assert!(
|
||||
st.db.children_if_listed(&uid("src")).unwrap().is_none(),
|
||||
st.flushed_db()
|
||||
.children_if_listed(&uid("src"))
|
||||
.unwrap()
|
||||
.is_none(),
|
||||
"source re-enumerates: its listing predates the move, and a stale \
|
||||
`listed` flag would hide every later remote change under it"
|
||||
);
|
||||
@@ -1074,12 +1225,41 @@ mod tests {
|
||||
let (mut st, _dir, src, _dst) = two_folders();
|
||||
let f = st.by_uid[&uid("f")];
|
||||
st.relocate(f, src, src, &uid("src"), "renamed.txt");
|
||||
assert!(st.db.children_if_listed(&uid("src")).unwrap().is_none());
|
||||
assert!(
|
||||
st.flushed_db()
|
||||
.children_if_listed(&uid("src"))
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
assert!(!st.children.contains_key(&src));
|
||||
assert_eq!(st.by_uid.get(&uid("f")), Some(&f), "the inode is stable");
|
||||
assert_eq!(st.entries[&f].node.name, "renamed.txt");
|
||||
}
|
||||
|
||||
/// DB1: a mutation's write-through must not happen with the inode lock
|
||||
/// held. Interning a listing commits a transaction over every child, and
|
||||
/// every other FUSE callback used to wait behind that for no reason — the
|
||||
/// maps it needs were updated before the first row was written.
|
||||
#[test]
|
||||
fn a_mutation_writes_through_only_after_the_lock_is_released() {
|
||||
let (st, _dir) = state();
|
||||
let db = st.db.clone();
|
||||
let root = uid("root").to_string();
|
||||
let state = parking_lot::Mutex::new(st);
|
||||
{
|
||||
let mut guard = lock_state(&state, &db);
|
||||
guard.intern(0, node("root", "none", "My Files", true));
|
||||
assert!(
|
||||
db.node_by_uid(&root).unwrap().is_none(),
|
||||
"the row must not be written while the lock is held"
|
||||
);
|
||||
}
|
||||
assert!(
|
||||
db.node_by_uid(&root).unwrap().is_some(),
|
||||
"…and must be written by the time the guard is gone"
|
||||
);
|
||||
}
|
||||
|
||||
/// The failure mode the fix exists for, stated directly: forgetting the node
|
||||
/// prunes it from the resident listing but leaves the DB claiming the source
|
||||
/// is fully enumerated. This pins *why* `relocate` cannot just call `forget`.
|
||||
@@ -1087,7 +1267,7 @@ mod tests {
|
||||
fn forget_alone_leaves_the_source_claiming_a_complete_listing() {
|
||||
let (mut st, _dir, _src, _dst) = two_folders();
|
||||
st.forget(&uid("f"));
|
||||
let listed = st.db.children_if_listed(&uid("src")).unwrap();
|
||||
let listed = st.flushed_db().children_if_listed(&uid("src")).unwrap();
|
||||
assert!(
|
||||
listed.is_some(),
|
||||
"forget does not clear the flag — which is exactly why relocate must"
|
||||
@@ -1136,7 +1316,7 @@ mod tests {
|
||||
assert!(!st.has_children(folder), "cleared memory children");
|
||||
|
||||
// DB children
|
||||
st.db
|
||||
st.flushed_db()
|
||||
.upsert_node(&node("file2_uid", "dir_uid", "file2.txt", false))
|
||||
.unwrap();
|
||||
assert!(
|
||||
@@ -1162,7 +1342,7 @@ mod tests {
|
||||
assert_eq!(st.entries[&shared_ino].access, Access::Editor);
|
||||
assert_eq!(st.entries[&child].access, Access::Editor);
|
||||
assert_eq!(
|
||||
st.db.share_access(&uid("shared")).unwrap(),
|
||||
st.flushed_db().share_access(&uid("shared")).unwrap(),
|
||||
Some(Access::Editor)
|
||||
);
|
||||
}
|
||||
@@ -1174,8 +1354,8 @@ mod tests {
|
||||
let virtual_uid = NodeUid::new(VolumeId::from("virtual"), LinkId::from("sharedwithme"));
|
||||
let mut virtual_root = node("placeholder", "root", "Shared with me", true);
|
||||
virtual_root.uid = virtual_uid.clone();
|
||||
st.db.upsert_node(&virtual_root).unwrap();
|
||||
st.db
|
||||
st.flushed_db().upsert_node(&virtual_root).unwrap();
|
||||
st.flushed_db()
|
||||
.set_share_access(&virtual_uid, Access::Viewer)
|
||||
.unwrap();
|
||||
let virtual_ino = st.intern_published_share_root(owned_root, virtual_root, Access::Viewer);
|
||||
@@ -1183,15 +1363,17 @@ mod tests {
|
||||
let mut shared = node("shared", "placeholder", "Retained", true);
|
||||
shared.parent_uid = Some(virtual_uid.clone());
|
||||
shared.membership = Some(membership(6));
|
||||
st.db.upsert_node(&shared).unwrap();
|
||||
st.db.set_share_access(&shared.uid, Access::Editor).unwrap();
|
||||
st.flushed_db().upsert_node(&shared).unwrap();
|
||||
st.flushed_db()
|
||||
.set_share_access(&shared.uid, Access::Editor)
|
||||
.unwrap();
|
||||
let shared_ino =
|
||||
st.intern_published_share_root(virtual_ino, shared.clone(), Access::Editor);
|
||||
let child = st.intern(
|
||||
shared_ino,
|
||||
node("shared-child", "shared", "pending.txt", false),
|
||||
);
|
||||
st.db
|
||||
st.flushed_db()
|
||||
.enqueue_op(&pdfs_core::db::PendingOp {
|
||||
id: 0,
|
||||
kind: pdfs_core::db::OP_RENAME.to_string(),
|
||||
@@ -1209,20 +1391,20 @@ mod tests {
|
||||
assert_eq!(st.entries[&shared_ino].access, Access::Editor);
|
||||
assert_eq!(st.entries[&child].access, Access::Editor);
|
||||
|
||||
st.db
|
||||
st.flushed_db()
|
||||
.publish_shared_roots(&virtual_uid, std::slice::from_ref(&shared.uid), &[])
|
||||
.unwrap();
|
||||
let snapshot = st.db.visible_children(&virtual_uid).unwrap();
|
||||
let snapshot = st.flushed_db().visible_children(&virtual_uid).unwrap();
|
||||
assert_eq!(snapshot.len(), 1);
|
||||
st.intern_published_share_root(virtual_ino, snapshot[0].clone(), Access::Viewer);
|
||||
|
||||
assert_eq!(
|
||||
st.db.share_access(&shared.uid).unwrap(),
|
||||
st.flushed_db().share_access(&shared.uid).unwrap(),
|
||||
Some(Access::Viewer)
|
||||
);
|
||||
assert_eq!(st.entries[&shared_ino].access, Access::Viewer);
|
||||
assert_eq!(st.entries[&child].access, Access::Viewer);
|
||||
assert_eq!(st.db.pending_ops().unwrap().len(), 1);
|
||||
assert_eq!(st.flushed_db().pending_ops().unwrap().len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1233,12 +1415,12 @@ mod tests {
|
||||
let child_node = node("offline-child", "offline-share", "nested folder", true);
|
||||
let leaf_node = node("offline-leaf", "offline-child", "child.txt", false);
|
||||
for node in [&root_node, &shared_node, &child_node, &leaf_node] {
|
||||
st.db.upsert_node(node).unwrap();
|
||||
st.flushed_db().upsert_node(node).unwrap();
|
||||
}
|
||||
st.db
|
||||
st.flushed_db()
|
||||
.set_share_access(&uid("offline-share"), Access::Viewer)
|
||||
.unwrap();
|
||||
st.share_access = st.db.all_share_access().unwrap();
|
||||
st.share_access = st.flushed_db().all_share_access().unwrap();
|
||||
|
||||
// Reconstruct in deliberately non-topological order, matching the
|
||||
// database hydration path that first allocates every inode and only
|
||||
@@ -1296,7 +1478,7 @@ mod tests {
|
||||
/// `ORPHAN_INO`. `nodes[0]` becomes the mount root.
|
||||
fn hydrate_from(st: &mut State, nodes: &[Node]) -> Vec<u64> {
|
||||
for node in nodes {
|
||||
st.db.upsert_node(node).unwrap();
|
||||
st.flushed_db().upsert_node(node).unwrap();
|
||||
}
|
||||
let inos: Vec<u64> = nodes
|
||||
.iter()
|
||||
@@ -1394,10 +1576,10 @@ mod tests {
|
||||
#[test]
|
||||
fn persisted_authority_outranks_the_own_volume_fail_open() {
|
||||
let (mut st, _dir) = state();
|
||||
st.db
|
||||
st.flushed_db()
|
||||
.set_share_access(&uid("device-root"), Access::Viewer)
|
||||
.unwrap();
|
||||
st.share_access = st.db.all_share_access().unwrap();
|
||||
st.share_access = st.flushed_db().all_share_access().unwrap();
|
||||
let nodes = [
|
||||
node("root", "none", "My Files", true),
|
||||
node("documents", "device-root", "Documents", true),
|
||||
@@ -1431,7 +1613,7 @@ mod tests {
|
||||
assert_eq!(st.require_writable(ino).unwrap_err().code(), libc::EACCES);
|
||||
}
|
||||
assert_eq!(
|
||||
st.db.share_access(&uid("shared")).unwrap(),
|
||||
st.flushed_db().share_access(&uid("shared")).unwrap(),
|
||||
Some(Access::Viewer)
|
||||
);
|
||||
}
|
||||
@@ -1441,8 +1623,10 @@ mod tests {
|
||||
for persisted in [Access::Viewer, Access::Editor] {
|
||||
let (mut st, _dir) = state();
|
||||
let owned_root = st.intern(0, node("root", "none", "My Files", true));
|
||||
st.db.set_share_access(&uid("shared"), persisted).unwrap();
|
||||
st.share_access = st.db.all_share_access().unwrap();
|
||||
st.flushed_db()
|
||||
.set_share_access(&uid("shared"), persisted)
|
||||
.unwrap();
|
||||
st.share_access = st.flushed_db().all_share_access().unwrap();
|
||||
let shared = st.intern(
|
||||
owned_root,
|
||||
node("shared", "root", "Shared without membership", true),
|
||||
@@ -1454,7 +1638,10 @@ mod tests {
|
||||
|
||||
assert_eq!(st.entries[&shared].access, persisted);
|
||||
assert_eq!(st.entries[&child].access, persisted);
|
||||
assert_eq!(st.db.share_access(&uid("shared")).unwrap(), Some(persisted));
|
||||
assert_eq!(
|
||||
st.flushed_db().share_access(&uid("shared")).unwrap(),
|
||||
Some(persisted)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1462,7 +1649,7 @@ mod tests {
|
||||
fn explicit_missing_share_role_fails_closed_and_marks_kernel_attrs() {
|
||||
let (mut st, _dir) = state();
|
||||
let owned_root = st.intern(0, node("root", "none", "My Files", true));
|
||||
st.db
|
||||
st.flushed_db()
|
||||
.set_share_access(&uid("shared"), Access::Editor)
|
||||
.unwrap();
|
||||
let shared = st.intern(owned_root, node("shared", "root", "Shared", true));
|
||||
@@ -1474,7 +1661,7 @@ mod tests {
|
||||
assert_eq!(st.entries[&shared].access, Access::Viewer);
|
||||
assert_eq!(st.entries[&child].access, Access::Viewer);
|
||||
assert_eq!(
|
||||
st.db.share_access(&uid("shared")).unwrap(),
|
||||
st.flushed_db().share_access(&uid("shared")).unwrap(),
|
||||
Some(Access::Viewer)
|
||||
);
|
||||
assert!(changed.contains(&shared));
|
||||
@@ -1491,7 +1678,7 @@ mod tests {
|
||||
let virtual_uid = NodeUid::new(VolumeId::from("virtual"), LinkId::from("sharedwithme"));
|
||||
let mut virtual_node = node("sharedwithme", "root", "Shared with me", true);
|
||||
virtual_node.uid = virtual_uid.clone();
|
||||
st.db
|
||||
st.flushed_db()
|
||||
.set_share_access(&virtual_uid, Access::Viewer)
|
||||
.unwrap();
|
||||
st.share_access.insert(virtual_uid.clone(), Access::Viewer);
|
||||
@@ -1558,7 +1745,7 @@ mod tests {
|
||||
let child = st.intern(shared, node("child", "shared", "folder", true));
|
||||
let leaf = st.intern(child, node("leaf", "child", "leaf.txt", false));
|
||||
|
||||
st.db
|
||||
st.flushed_db()
|
||||
.set_share_access(&uid("shared"), Access::Viewer)
|
||||
.unwrap();
|
||||
let changed = st.downgrade_shared_subtree(&uid("shared"));
|
||||
@@ -1572,7 +1759,7 @@ mod tests {
|
||||
assert_eq!(st.entries[&leaf].access, Access::Viewer);
|
||||
assert_eq!(st.require_writable(leaf).unwrap_err().code(), libc::EACCES);
|
||||
assert_eq!(
|
||||
st.db.effective_node_access(&uid("leaf")).unwrap(),
|
||||
st.flushed_db().effective_node_access(&uid("leaf")).unwrap(),
|
||||
Some(Access::Viewer)
|
||||
);
|
||||
|
||||
@@ -1628,7 +1815,7 @@ mod tests {
|
||||
let shared = st.intern(owned_root, shared_node);
|
||||
let shared_child = st.intern(shared, node("child", "shared", "child.txt", false));
|
||||
|
||||
st.db.downgrade_all_share_access().unwrap();
|
||||
st.flushed_db().downgrade_all_share_access().unwrap();
|
||||
let changed = st.downgrade_known_shared_access();
|
||||
|
||||
assert!(changed.contains(&shared));
|
||||
@@ -1652,7 +1839,7 @@ mod tests {
|
||||
let parent_link = st.entries[&parent].uid.link_id.to_string();
|
||||
parent = st.intern(parent, node(&link, &parent_link, &link, true));
|
||||
}
|
||||
st.db.downgrade_all_share_access().unwrap();
|
||||
st.flushed_db().downgrade_all_share_access().unwrap();
|
||||
|
||||
let changed = st.downgrade_known_shared_access();
|
||||
|
||||
|
||||
@@ -67,10 +67,16 @@ impl Core {
|
||||
/// Sweep for stale conflict copies forever, on its own thread. See the module
|
||||
/// docs for the identical/divergent policy.
|
||||
pub(crate) fn run_conflict_sweep_loop(&self) {
|
||||
std::thread::sleep(CONFLICT_SWEEP_WARMUP);
|
||||
// Both waits are interruptible, so teardown does not have to sit out a
|
||||
// five-minute interval to join this thread (bugs.md B44).
|
||||
if !self.shutdown.sleep(CONFLICT_SWEEP_WARMUP) {
|
||||
return;
|
||||
}
|
||||
loop {
|
||||
self.sweep_conflicts_once();
|
||||
std::thread::sleep(CONFLICT_SWEEP_INTERVAL);
|
||||
if !self.shutdown.sleep(CONFLICT_SWEEP_INTERVAL) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -138,9 +144,9 @@ impl Core {
|
||||
/// enumerated every node up front and has been doing network I/O since. Before
|
||||
/// touching anything remote this re-reads the node and refuses to act unless
|
||||
/// it is *still* the same revision, still untrashed, still unwritten and still
|
||||
/// unopened. Without that, a user edit landing inside the window is destroyed
|
||||
/// — `discard_queued_ops` below drops the queued write *and its staged blob*,
|
||||
/// which may hold the only copy of those bytes (`docs/BUGS.md` B71).
|
||||
/// unopened (`docs/BUGS.md` B71). Nothing on this path discards a queued write
|
||||
/// or its staged blob: a write that arrives after that check is left for the
|
||||
/// drain to resolve against the now-trashed node.
|
||||
fn remove_conflict_copy(&self, snapshot: &proton_drive_rs::Node, base: &str) {
|
||||
let uid = &snapshot.uid;
|
||||
let name = snapshot.name.as_str();
|
||||
@@ -166,10 +172,13 @@ impl Core {
|
||||
self.for_each_state(|st| {
|
||||
st.forget_or_unlink(uid);
|
||||
});
|
||||
if let Err(error) = self.discard_queued_ops(uid) {
|
||||
warn!(%uid, ?error, "conflict sweep: queued-op cleanup failed");
|
||||
return;
|
||||
}
|
||||
// Deliberately *not* `discard_queued_ops`: `duplicate_still_removable`
|
||||
// proved the queue empty before the trash above, but that proof is stale
|
||||
// the moment `trash_nodes` goes to the network. A `close(2)` landing in
|
||||
// that window stages bytes, and discarding here would unlink the staged
|
||||
// blob holding the only copy of them. Leaving the op queued is the
|
||||
// non-destructive outcome: the drain re-reads the node, sees it trashed,
|
||||
// and `revision_conflict` routes it to `keep_as_conflict_copy`.
|
||||
self.cache.evict(uid);
|
||||
self.evict_reader(uid);
|
||||
if let Err(e) = self.db.delete_node(uid) {
|
||||
@@ -194,8 +203,8 @@ impl Core {
|
||||
let uid = &snapshot.uid;
|
||||
let name = snapshot.name.as_str();
|
||||
|
||||
// A queued op means bytes are still owed an upload. `discard_queued_ops`
|
||||
// would throw those away along with the staged blob holding them.
|
||||
// A queued op means bytes are still owed an upload; trashing the node
|
||||
// under them turns an ordinary upload into a conflict fork.
|
||||
match self.db.has_any_op(&uid.to_string()) {
|
||||
Ok(false) => {}
|
||||
Ok(true) => {
|
||||
@@ -359,6 +368,8 @@ mod tests {
|
||||
photo: None,
|
||||
album: None,
|
||||
verification: Default::default(),
|
||||
direct_role: None,
|
||||
share_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+585
-77
@@ -54,7 +54,7 @@ use engine::{classify, settle_with};
|
||||
#[cfg(test)]
|
||||
use local_scan::is_write_mode;
|
||||
use local_scan::open_for_write_set;
|
||||
use model::{LocalItem, Outcome, PassAbort, Pending, RemoteItem};
|
||||
use model::{LocalItem, LocalSig, Outcome, PassAbort, Pending, RemoteItem};
|
||||
pub(crate) use planner::base_name;
|
||||
#[cfg(test)]
|
||||
use planner::conflict_path;
|
||||
@@ -525,6 +525,9 @@ impl Core {
|
||||
continue;
|
||||
}
|
||||
trashed.push(rel.clone());
|
||||
if let Some(parent) = parent_uid_of(rel, &remote_dirs, &baseline) {
|
||||
self.invalidate_children_of(&parent);
|
||||
}
|
||||
let _ = self.db.sync_entry_remove(folder_id, rel);
|
||||
self.log_activity(
|
||||
ActivityKind::Trash,
|
||||
@@ -687,6 +690,7 @@ impl Core {
|
||||
rel_path: rel.clone(),
|
||||
remote_uid: Some(remote.uid.to_string()),
|
||||
local_mtime: local.mtime,
|
||||
local_mtime_ns: local.mtime_ns,
|
||||
local_size: local.size,
|
||||
remote_rev: Some(remote.mtime.to_string()),
|
||||
remote_hash: Some(remote.size.to_string()),
|
||||
@@ -717,6 +721,9 @@ impl Core {
|
||||
uid: remote.uid.clone(),
|
||||
mtime: remote.mtime,
|
||||
size: remote.size,
|
||||
// What the walk saw here, so the apply step can tell a
|
||||
// stale plan from a live one before it overwrites.
|
||||
local: l.map(LocalSig::from),
|
||||
});
|
||||
}
|
||||
FilePlan::Conflict => {
|
||||
@@ -739,7 +746,9 @@ impl Core {
|
||||
}
|
||||
},
|
||||
FilePlan::DeleteLocal => {
|
||||
if let Err(e) = std::fs::remove_file(local_root.join(rel_to_path(rel))) {
|
||||
if let Err(e) =
|
||||
removed_locally(std::fs::remove_file(local_root.join(rel_to_path(rel))))
|
||||
{
|
||||
warn!(rel, error = %e, "sync: remove local file failed");
|
||||
self.log_activity(
|
||||
ActivityKind::Trash,
|
||||
@@ -771,6 +780,9 @@ impl Core {
|
||||
outcome.errors += 1;
|
||||
continue;
|
||||
}
|
||||
if let Some(parent) = parent_uid_of(rel, &remote_dirs, &baseline) {
|
||||
self.invalidate_children_of(&parent);
|
||||
}
|
||||
let _ = self.db.sync_entry_remove(folder_id, rel);
|
||||
self.log_activity(
|
||||
ActivityKind::Trash,
|
||||
@@ -797,7 +809,9 @@ impl Core {
|
||||
// Deferred folder deletions, deepest first.
|
||||
delete_local_dirs.sort_by_key(|p| std::cmp::Reverse(p.matches('/').count()));
|
||||
for rel in delete_local_dirs {
|
||||
if let Err(e) = std::fs::remove_dir_all(local_root.join(rel_to_path(&rel))) {
|
||||
if let Err(e) =
|
||||
removed_locally(std::fs::remove_dir_all(local_root.join(rel_to_path(&rel))))
|
||||
{
|
||||
warn!(rel, error = %e, "sync: remove local folder failed");
|
||||
self.log_activity(ActivityKind::Trash, base_name(&rel), e.to_string(), false);
|
||||
outcome.errors += 1;
|
||||
@@ -824,6 +838,9 @@ impl Core {
|
||||
outcome.errors += 1;
|
||||
continue;
|
||||
}
|
||||
if let Some(parent) = parent_uid_of(&rel, &remote_dirs, &baseline) {
|
||||
self.invalidate_children_of(&parent);
|
||||
}
|
||||
let _ = self.db.sync_entry_remove(folder_id, &rel);
|
||||
self.log_activity(
|
||||
ActivityKind::Trash,
|
||||
@@ -849,68 +866,9 @@ impl Core {
|
||||
writing: &HashSet<PathBuf>,
|
||||
out: &mut HashMap<String, LocalItem>,
|
||||
) -> Result<(), String> {
|
||||
let entries = std::fs::read_dir(dir).map_err(|e| format!("read {}: {e}", dir.display()))?;
|
||||
for entry in entries {
|
||||
let entry = entry.map_err(|e| format!("read entry in {}: {e}", dir.display()))?;
|
||||
let path = entry.path();
|
||||
let meta = std::fs::symlink_metadata(&path)
|
||||
.map_err(|e| format!("stat {}: {e}", path.display()))?;
|
||||
let stripped = path.strip_prefix(root).map_err(|e| {
|
||||
format!(
|
||||
"local path {} escaped sync root {}: {e}",
|
||||
path.display(),
|
||||
root.display()
|
||||
)
|
||||
})?;
|
||||
let rel = stripped.to_str().ok_or_else(|| {
|
||||
format!(
|
||||
"local path {} is not valid UTF-8; refusing a destructive sync pass",
|
||||
path.display()
|
||||
)
|
||||
})?;
|
||||
// Ignore our own in-flight download temp files.
|
||||
if rel.contains(".pdfs-tmp-") {
|
||||
continue;
|
||||
}
|
||||
// Ignored paths are dropped here as well as from the classification
|
||||
// union — not for correctness (the union filter is what makes this
|
||||
// safe) but so an ignored `node_modules/` is never descended into or
|
||||
// counted against scan progress.
|
||||
if rules.is_ignored(rel, meta.is_dir()) {
|
||||
continue;
|
||||
}
|
||||
if meta.is_dir() {
|
||||
out.insert(
|
||||
rel.to_string(),
|
||||
LocalItem {
|
||||
is_dir: true,
|
||||
mtime: 0,
|
||||
size: 0,
|
||||
open_for_write: false,
|
||||
},
|
||||
);
|
||||
self.progress_scanned(folder_id, base_name(rel));
|
||||
self.walk_local(folder_id, root, &path, rules, writing, out)?;
|
||||
} else if meta.is_file() {
|
||||
// Check whether any process holds this file open for writing.
|
||||
// The canonical path is used because /proc/*/fd links resolve
|
||||
// to canonical targets.
|
||||
let ofw = std::fs::canonicalize(&path)
|
||||
.map(|canon| writing.contains(&canon))
|
||||
.unwrap_or(false);
|
||||
out.insert(
|
||||
rel.to_string(),
|
||||
LocalItem {
|
||||
is_dir: false,
|
||||
mtime: system_mtime(&meta),
|
||||
size: meta.len() as i64,
|
||||
open_for_write: ofw,
|
||||
},
|
||||
);
|
||||
self.progress_scanned(folder_id, base_name(rel));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
walk_local_tree(root, dir, rules, writing, out, &mut |rel| {
|
||||
self.progress_scanned(folder_id, base_name(rel))
|
||||
})
|
||||
}
|
||||
|
||||
/// Map every remote folder under `folder` to its uid by rel path, ignoring
|
||||
@@ -1069,6 +1027,7 @@ impl Core {
|
||||
rel_path: rel.to_string(),
|
||||
remote_uid: Some(uid.to_string()),
|
||||
local_mtime: 0,
|
||||
local_mtime_ns: None,
|
||||
local_size: 0,
|
||||
remote_rev: None,
|
||||
remote_hash: None,
|
||||
@@ -1175,6 +1134,9 @@ impl Core {
|
||||
.create_folder(parent, base_name(rel), Some(now_secs()))
|
||||
.await
|
||||
.map_err(|e| format!("create remote folder {rel}: {e}"))?;
|
||||
// The parent has gained a child that no cached listing of it
|
||||
// knows about (B86).
|
||||
self.invalidate_children_of(parent);
|
||||
self.baseline_dir(folder_id, rel, &uid)?;
|
||||
Ok(Applied::Dir(rel.clone(), uid))
|
||||
}
|
||||
@@ -1192,8 +1154,9 @@ impl Core {
|
||||
uid,
|
||||
mtime,
|
||||
size,
|
||||
local,
|
||||
} => {
|
||||
self.download_file(folder_id, local_root, rel, uid, *mtime, *size)
|
||||
self.download_file(folder_id, local_root, rel, uid, *mtime, *size, *local)
|
||||
.await?;
|
||||
Ok(Applied::Downloaded)
|
||||
}
|
||||
@@ -1211,7 +1174,11 @@ impl Core {
|
||||
.map_err(|e| format!("set aside conflict copy for {rel}: {e}"))?;
|
||||
info!(rel, "sync: kept local changes as a conflict copy");
|
||||
}
|
||||
self.download_file(folder_id, local_root, rel, uid, *mtime, *size)
|
||||
// The local copy has just been moved aside, so the state this
|
||||
// download expects to find at the path is "nothing" — anything
|
||||
// that appears while it runs is a *new* racing write, and gets
|
||||
// its own conflict copy rather than being overwritten (B39).
|
||||
self.download_file(folder_id, local_root, rel, uid, *mtime, *size, None)
|
||||
.await?;
|
||||
Ok(Applied::Conflict)
|
||||
}
|
||||
@@ -1250,6 +1217,9 @@ impl Core {
|
||||
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
|
||||
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
|
||||
let mtime = system_mtime(&meta);
|
||||
// Taken from the descriptor being streamed, so it names the content that
|
||||
// actually goes up rather than whatever is at the path when we finish.
|
||||
let streamed = LocalSig::from(&meta);
|
||||
// Count the bytes as they are read, so this upload shows up in
|
||||
// `GetQueueStatus` next to a manual one. The uid isn't known until the
|
||||
// draft is sealed, so the transfer registers without one.
|
||||
@@ -1276,7 +1246,13 @@ impl Core {
|
||||
)
|
||||
.await
|
||||
.map_err(|e| format!("upload {rel}: {e}"))?;
|
||||
self.record_file_baseline(folder_id, rel, &path, &uid).await
|
||||
// The parent has gained a child that no cached listing of it knows
|
||||
// about. Without this, a folder later switched to on-demand serves a
|
||||
// snapshot that predates this upload (B86).
|
||||
self.invalidate_children_of(parent);
|
||||
warn_if_torn(rel, &path, streamed);
|
||||
self.record_file_baseline(folder_id, rel, Some(streamed), &uid)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Upload a changed local file as a new revision of an existing remote node.
|
||||
@@ -1291,6 +1267,7 @@ impl Core {
|
||||
let file = std::fs::File::open(&path).map_err(|e| format!("open {rel}: {e}"))?;
|
||||
let meta = file.metadata().map_err(|e| format!("stat {rel}: {e}"))?;
|
||||
let mtime = system_mtime(&meta);
|
||||
let streamed = LocalSig::from(&meta);
|
||||
let reader = OwnedCountingReader::new(
|
||||
file,
|
||||
self.transfers.begin(
|
||||
@@ -1304,11 +1281,20 @@ impl Core {
|
||||
.upload_new_revision_from(uid, reader, meta.len() as i64, Vec::new(), Some(mtime))
|
||||
.await
|
||||
.map_err(|e| format!("upload revision {rel}: {e}"))?;
|
||||
self.record_file_baseline(folder_id, rel, &path, uid).await
|
||||
warn_if_torn(rel, &path, streamed);
|
||||
self.record_file_baseline(folder_id, rel, Some(streamed), uid)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Download a remote file to its local path (atomically via a temp file),
|
||||
/// stamp the local mtime to match the remote, then record baseline.
|
||||
///
|
||||
/// `expected_local` is the identity planning saw at the destination. The
|
||||
/// rename below is the destructive step, and the download in between can run
|
||||
/// for minutes, so the destination is re-examined right before it: a local
|
||||
/// file that no longer matches the plan is set aside as a conflict copy
|
||||
/// instead of being replaced (bugs.md B39).
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
async fn download_file(
|
||||
&self,
|
||||
folder_id: i64,
|
||||
@@ -1317,6 +1303,7 @@ impl Core {
|
||||
uid: &NodeUid,
|
||||
mtime: i64,
|
||||
size: i64,
|
||||
expected_local: Option<LocalSig>,
|
||||
) -> Result<(), String> {
|
||||
let path = local_root.join(rel_to_path(rel));
|
||||
if let Some(parent) = path.parent() {
|
||||
@@ -1351,6 +1338,13 @@ impl Core {
|
||||
out.flush().map_err(|e| format!("flush tmp {rel}: {e}"))?;
|
||||
out.sync_all().map_err(|e| format!("sync tmp {rel}: {e}"))?;
|
||||
}
|
||||
// The last moment at which the destination can still be saved. Anything
|
||||
// other than the state planning classified means a writer got there
|
||||
// first, and its bytes are the ones nothing else has a copy of.
|
||||
if let Err(e) = keep_racing_local_edit(&path, expected_local) {
|
||||
let _ = std::fs::remove_file(&tmp);
|
||||
return Err(format!("set aside racing local edit for {rel}: {e}"));
|
||||
}
|
||||
std::fs::rename(&tmp, &path).map_err(|e| format!("place {rel}: {e}"))?;
|
||||
// Match local mtime to the remote's so neither side looks "changed" next pass.
|
||||
let f = std::fs::File::options()
|
||||
@@ -1366,23 +1360,36 @@ impl Core {
|
||||
.and_then(|dir| dir.sync_all())
|
||||
.map_err(|e| format!("sync directory for {rel}: {e}"))?;
|
||||
}
|
||||
self.record_file_baseline(folder_id, rel, &path, uid).await
|
||||
// Here the file on disk *is* the transferred content: it was published
|
||||
// by a rename and stamped just above, so its stat is the identity.
|
||||
self.record_file_baseline(folder_id, rel, local_sig(&path), uid)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Refresh a file's baseline from ground truth: the local stat and the
|
||||
/// remote node's reported signature. Called after every upload/download so
|
||||
/// the very next reconcile sees the path as unchanged.
|
||||
/// Refresh a file's baseline from ground truth: the local identity these
|
||||
/// bytes came from and the remote node's reported signature. Called after
|
||||
/// every upload/download so the very next reconcile sees the path as
|
||||
/// unchanged.
|
||||
///
|
||||
/// `local` is the identity the *transferred content* corresponds to, which
|
||||
/// is not always the file's current stat. An upload streams the live file,
|
||||
/// so it passes the identity it read the first byte at: if a writer changed
|
||||
/// the file mid-stream, the remote now holds a torn revision, and recording
|
||||
/// the pre-upload identity is what makes the next pass see the path as
|
||||
/// locally changed and upload a whole, clean revision over it. Recording the
|
||||
/// post-upload stat instead declared the torn revision synchronized and left
|
||||
/// it as the file's content on Drive (bugs.md B40).
|
||||
async fn record_file_baseline(
|
||||
&self,
|
||||
folder_id: i64,
|
||||
rel: &str,
|
||||
local_path: &Path,
|
||||
local: Option<LocalSig>,
|
||||
uid: &NodeUid,
|
||||
) -> Result<(), String> {
|
||||
let (lmtime, lsize) = match std::fs::metadata(local_path) {
|
||||
Ok(m) => (system_mtime(&m), m.len() as i64),
|
||||
Err(e) => return Err(format!("stat {rel}: {e}")),
|
||||
let Some(local) = local else {
|
||||
return Err(format!("stat {rel}: no local identity to baseline"));
|
||||
};
|
||||
let (lmtime, lsize) = (local.mtime, local.size);
|
||||
// Re-fetch the node so the stored remote signature is exactly what a walk
|
||||
// will report next time.
|
||||
let (rmtime, rsize) = match self.client.enumerate_nodes(std::slice::from_ref(uid)).await {
|
||||
@@ -1402,6 +1409,7 @@ impl Core {
|
||||
rel_path: rel.to_string(),
|
||||
remote_uid: Some(uid.to_string()),
|
||||
local_mtime: lmtime,
|
||||
local_mtime_ns: local.mtime_ns,
|
||||
local_size: lsize,
|
||||
remote_rev: Some(rmtime.to_string()),
|
||||
remote_hash: Some(rsize.to_string()),
|
||||
@@ -1411,6 +1419,177 @@ impl Core {
|
||||
}
|
||||
}
|
||||
|
||||
/// Say so when a file changed under an upload that was streaming it.
|
||||
///
|
||||
/// Purely diagnostic — the correction is already in place, because the baseline
|
||||
/// records the identity the upload *read* rather than the one on disk now, so
|
||||
/// the next pass classifies the path as locally changed and replaces the torn
|
||||
/// revision. Without the log there is nothing in the journal to explain the
|
||||
/// extra revision (bugs.md B40).
|
||||
fn warn_if_torn(rel: &str, path: &Path, streamed: LocalSig) {
|
||||
let now = local_sig(path);
|
||||
if !now.is_some_and(|now| now.same_content(&streamed)) {
|
||||
warn!(
|
||||
rel,
|
||||
?streamed,
|
||||
?now,
|
||||
"sync: file changed while uploading; the remote revision may be torn \
|
||||
and will be replaced on the next pass"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The identity a reconcile pass uses to recognise a local file it has already
|
||||
/// looked at: `None` when nothing is at the path.
|
||||
fn local_sig(path: &Path) -> Option<LocalSig> {
|
||||
std::fs::metadata(path).ok().map(|m| LocalSig {
|
||||
mtime: system_mtime(&m),
|
||||
mtime_ns: system_mtime_ns(&m),
|
||||
size: m.len() as i64,
|
||||
})
|
||||
}
|
||||
|
||||
/// Guard the destination of a download: if what is at `path` is no longer what
|
||||
/// planning classified, move it aside as a conflict copy before the caller
|
||||
/// renames over it.
|
||||
///
|
||||
/// A reconcile pass decides what to do from a walk that finished before the
|
||||
/// transfer started, and a download of a large file can run for minutes. Without
|
||||
/// this, a local edit completed inside that window was replaced by the remote
|
||||
/// version with no copy of it kept anywhere — the one shape of loss a sync
|
||||
/// engine must not have (bugs.md B39). Keeping the copy makes it re-upload as a
|
||||
/// new file on the next pass, which is the same resolution a both-sides-changed
|
||||
/// conflict already gets.
|
||||
///
|
||||
/// A path that is empty when it was expected to be is not a conflict: the
|
||||
/// download is about to recreate it, and there is nothing to preserve.
|
||||
fn keep_racing_local_edit(path: &Path, expected: Option<LocalSig>) -> std::io::Result<()> {
|
||||
let actual = local_sig(path);
|
||||
let unchanged = match (actual, expected) {
|
||||
(Some(a), Some(e)) => a.same_content(&e),
|
||||
// Nothing there: the download is about to create it, and there is
|
||||
// nothing to preserve either way.
|
||||
(None, _) => true,
|
||||
(Some(_), None) => false,
|
||||
};
|
||||
if unchanged {
|
||||
return Ok(());
|
||||
}
|
||||
warn!(
|
||||
path = %path.display(),
|
||||
?expected,
|
||||
?actual,
|
||||
"sync: local file changed while downloading; keeping it as a conflict copy"
|
||||
);
|
||||
preserve_conflict_copy(path, now_secs()).map(|_| ())
|
||||
}
|
||||
|
||||
/// Read a local removal as the sync engine has to read it: the *state* it was
|
||||
/// asking for is "this path is gone", so a path that was already gone is a
|
||||
/// success, not a failure.
|
||||
///
|
||||
/// It matters because the baseline is only dropped on success. Treating
|
||||
/// `NotFound` as an error left the baseline in place for a file that no longer
|
||||
/// exists, so the same removal was re-attempted and re-reported as an error on
|
||||
/// every pass, forever. Every other error is still an error, and keeping the
|
||||
/// baseline is exactly right there: the surviving file must not be mistaken for
|
||||
/// a new local creation on the next pass and uploaded back (bugs.md B42).
|
||||
fn removed_locally(result: std::io::Result<()>) -> std::io::Result<()> {
|
||||
match result {
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()),
|
||||
other => other,
|
||||
}
|
||||
}
|
||||
|
||||
/// The body of [`Core::walk_local`], without a `Core`: recursively collect the
|
||||
/// local tree under `dir` into `out`, reporting each entry to `scanned`.
|
||||
///
|
||||
/// **Every failure here is fatal to the pass, deliberately.** A `read_dir` that
|
||||
/// fails, an entry that cannot be stat-ed, a name that is not UTF-8 — each of
|
||||
/// them would, if skipped, leave `out` describing a *smaller* tree than the one
|
||||
/// on disk. Reconciliation reads a path present in the baseline and absent from
|
||||
/// the scan as "the user deleted this", and propagates that to Drive. So an
|
||||
/// unreadable subdirectory would trash its remote counterpart, and the deeper
|
||||
/// the failure, the more it takes with it (`docs/BUGS.md` B55). Refusing the
|
||||
/// whole pass keeps the previous baseline and leaves the remote alone; the next
|
||||
/// pass retries once the transient condition clears.
|
||||
///
|
||||
/// Split out from the method so exactly that can be tested against a real tree
|
||||
/// with a real unreadable directory in it, which needs no `Core` at all.
|
||||
fn walk_local_tree(
|
||||
root: &Path,
|
||||
dir: &Path,
|
||||
rules: &IgnoreRules,
|
||||
writing: &HashSet<PathBuf>,
|
||||
out: &mut HashMap<String, LocalItem>,
|
||||
scanned: &mut dyn FnMut(&str),
|
||||
) -> Result<(), String> {
|
||||
let entries = std::fs::read_dir(dir).map_err(|e| format!("read {}: {e}", dir.display()))?;
|
||||
for entry in entries {
|
||||
let entry = entry.map_err(|e| format!("read entry in {}: {e}", dir.display()))?;
|
||||
let path = entry.path();
|
||||
let meta = std::fs::symlink_metadata(&path)
|
||||
.map_err(|e| format!("stat {}: {e}", path.display()))?;
|
||||
let stripped = path.strip_prefix(root).map_err(|e| {
|
||||
format!(
|
||||
"local path {} escaped sync root {}: {e}",
|
||||
path.display(),
|
||||
root.display()
|
||||
)
|
||||
})?;
|
||||
let rel = stripped.to_str().ok_or_else(|| {
|
||||
format!(
|
||||
"local path {} is not valid UTF-8; refusing a destructive sync pass",
|
||||
path.display()
|
||||
)
|
||||
})?;
|
||||
// Ignore our own in-flight download temp files.
|
||||
if rel.contains(".pdfs-tmp-") {
|
||||
continue;
|
||||
}
|
||||
// Ignored paths are dropped here as well as from the classification
|
||||
// union — not for correctness (the union filter is what makes this
|
||||
// safe) but so an ignored `node_modules/` is never descended into or
|
||||
// counted against scan progress.
|
||||
if rules.is_ignored(rel, meta.is_dir()) {
|
||||
continue;
|
||||
}
|
||||
if meta.is_dir() {
|
||||
out.insert(
|
||||
rel.to_string(),
|
||||
LocalItem {
|
||||
is_dir: true,
|
||||
mtime: 0,
|
||||
mtime_ns: None,
|
||||
size: 0,
|
||||
open_for_write: false,
|
||||
},
|
||||
);
|
||||
scanned(rel);
|
||||
walk_local_tree(root, &path, rules, writing, out, scanned)?;
|
||||
} else if meta.is_file() {
|
||||
// Check whether any process holds this file open for writing.
|
||||
// The canonical path is used because /proc/*/fd links resolve
|
||||
// to canonical targets.
|
||||
let ofw = std::fs::canonicalize(&path)
|
||||
.map(|canon| writing.contains(&canon))
|
||||
.unwrap_or(false);
|
||||
out.insert(
|
||||
rel.to_string(),
|
||||
LocalItem {
|
||||
is_dir: false,
|
||||
mtime: system_mtime(&meta),
|
||||
mtime_ns: system_mtime_ns(&meta),
|
||||
size: meta.len() as i64,
|
||||
open_for_write: ofw,
|
||||
},
|
||||
);
|
||||
scanned(rel);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Publish a conflict copy without overwriting an earlier conflict bearing the
|
||||
/// same timestamp. The original is removed only after the new copy is durable.
|
||||
fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> {
|
||||
@@ -1444,6 +1623,26 @@ fn preserve_conflict_copy(path: &Path, stamp: i64) -> std::io::Result<PathBuf> {
|
||||
unreachable!("u32 conflict suffix space exhausted")
|
||||
}
|
||||
|
||||
/// The remote uid of `rel`'s parent folder: from the pass's folder-uid map
|
||||
/// first, and from the baseline second for a parent the map never covered.
|
||||
///
|
||||
/// Used to name the folder whose cached listing a remote mutation has just
|
||||
/// invalidated (B86). `None` is not an error — it only means this pass cannot
|
||||
/// say which folder to invalidate, and the caller skips it.
|
||||
fn parent_uid_of(
|
||||
rel: &str,
|
||||
remote_dirs: &HashMap<String, NodeUid>,
|
||||
baseline: &HashMap<String, StoredSyncEntry>,
|
||||
) -> Option<NodeUid> {
|
||||
let parent = parent_rel(rel);
|
||||
remote_dirs.get(parent).cloned().or_else(|| {
|
||||
baseline
|
||||
.get(parent)
|
||||
.and_then(|e| e.remote_uid.as_deref())
|
||||
.and_then(parse_uid)
|
||||
})
|
||||
}
|
||||
|
||||
fn sync_parent_dir(path: &Path) -> std::io::Result<()> {
|
||||
match path.parent() {
|
||||
Some(parent) => std::fs::File::open(parent)?.sync_all(),
|
||||
@@ -1476,10 +1675,77 @@ fn system_mtime(meta: &std::fs::Metadata) -> i64 {
|
||||
.map(|d| d.as_secs() as i64)
|
||||
.unwrap_or(0)
|
||||
}
|
||||
|
||||
/// The same modification time in nanoseconds, when the filesystem reports one.
|
||||
///
|
||||
/// Whole seconds cannot distinguish an edit made in the same second as the last
|
||||
/// sync from no edit at all, and if the size did not change either, that file
|
||||
/// was never uploaded (bugs.md B25). `None` for a clock or filesystem that
|
||||
/// cannot answer, which falls the comparison back to seconds rather than
|
||||
/// asserting an mtime of zero.
|
||||
fn system_mtime_ns(meta: &std::fs::Metadata) -> Option<i64> {
|
||||
meta.modified()
|
||||
.ok()
|
||||
.and_then(|t| t.duration_since(SystemTime::UNIX_EPOCH).ok())
|
||||
.and_then(|d| i64::try_from(d.as_nanos()).ok())
|
||||
}
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A removal of something already gone has produced the state the sync
|
||||
/// engine wanted, so its baseline is dropped rather than retried forever.
|
||||
/// Anything else stays an error and keeps the baseline (bugs.md B42).
|
||||
#[test]
|
||||
fn an_already_absent_path_counts_as_removed() {
|
||||
let root = std::env::temp_dir().join(format!("pdfs-b42-{}", std::process::id()));
|
||||
let missing = root.join("never-existed");
|
||||
assert!(removed_locally(std::fs::remove_file(&missing)).is_ok());
|
||||
assert!(removed_locally(std::fs::remove_dir_all(&missing)).is_ok());
|
||||
|
||||
// A directory in the way of `remove_file` is a real failure, and must
|
||||
// stay one: the file survives, and its baseline has to survive with it.
|
||||
let occupied = root.join("a-directory");
|
||||
std::fs::create_dir_all(&occupied).unwrap();
|
||||
assert!(removed_locally(std::fs::remove_file(&occupied)).is_err());
|
||||
let _ = std::fs::remove_dir_all(&root);
|
||||
}
|
||||
|
||||
/// The download guard: a destination that still matches the plan is
|
||||
/// replaced silently, one that has moved on since is kept (bugs.md B39).
|
||||
#[test]
|
||||
fn a_download_keeps_a_local_edit_it_did_not_plan_for() {
|
||||
let root = std::env::temp_dir().join(format!("pdfs-b39-{}", std::process::id()));
|
||||
std::fs::create_dir_all(&root).unwrap();
|
||||
let path = root.join("report.txt");
|
||||
|
||||
// Nothing there, nothing expected: the download simply creates it.
|
||||
assert!(keep_racing_local_edit(&path, None).is_ok());
|
||||
assert!(!path.exists());
|
||||
|
||||
// Unchanged since planning: no copy, the caller overwrites it.
|
||||
std::fs::write(&path, b"planned").unwrap();
|
||||
let planned = local_sig(&path).unwrap();
|
||||
keep_racing_local_edit(&path, Some(planned)).unwrap();
|
||||
assert_eq!(std::fs::read(&path).unwrap(), b"planned");
|
||||
|
||||
// Changed since planning: the racing bytes are the only copy of
|
||||
// themselves, so they must survive the overwrite as a conflict copy.
|
||||
std::fs::write(&path, b"a much longer local edit").unwrap();
|
||||
keep_racing_local_edit(&path, Some(planned)).unwrap();
|
||||
let kept: Vec<_> = std::fs::read_dir(&root)
|
||||
.unwrap()
|
||||
.filter_map(|e| e.ok())
|
||||
.map(|e| std::fs::read(e.path()).unwrap_or_default())
|
||||
.collect();
|
||||
assert!(
|
||||
kept.iter()
|
||||
.any(|bytes| bytes == b"a much longer local edit"),
|
||||
"the racing edit must still exist somewhere: {kept:?}"
|
||||
);
|
||||
let _ = std::fs::remove_dir_all(&root);
|
||||
}
|
||||
|
||||
/// Feed `n` `Reconcile` events `gap` apart on another thread, starting
|
||||
/// immediately. Returns the sender's join handle so the test can keep it
|
||||
/// alive (dropping the sender early would look like a disconnect).
|
||||
@@ -1565,6 +1831,7 @@ mod tests {
|
||||
|
||||
fn local_item(is_dir: bool) -> LocalItem {
|
||||
LocalItem {
|
||||
mtime_ns: None,
|
||||
is_dir,
|
||||
mtime: 1,
|
||||
size: 1,
|
||||
@@ -1722,6 +1989,44 @@ mod tests {
|
||||
assert!(guard_local_wipe(&base, &fresh).is_err());
|
||||
}
|
||||
|
||||
/// B54's remaining case, end to end: a folder whose local root is still
|
||||
/// *readable* but is no longer the tree it was — an empty directory left
|
||||
/// where a mount used to be, or a different device mounted over it.
|
||||
///
|
||||
/// Nothing fails here. The scan succeeds and honestly reports an empty
|
||||
/// directory, so the only thing standing between the user and a trashed
|
||||
/// remote folder is the guard, and it has to fire for a one-file baseline
|
||||
/// exactly as it does for a large one.
|
||||
#[test]
|
||||
fn a_replaced_mountpoint_trips_the_guard_at_every_baseline_size() {
|
||||
let root = sync_test_dir("wipe-replaced-mount");
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut local: HashMap<String, LocalItem> = HashMap::new();
|
||||
let mut progress = no_progress();
|
||||
walk_local_tree(&root, &root, &rules, &writing, &mut local, &mut progress).unwrap();
|
||||
assert!(local.is_empty(), "the replacement root is genuinely empty");
|
||||
|
||||
for size in 1..=3 {
|
||||
let baseline: HashMap<String, ()> =
|
||||
(0..size).map(|i| (format!("file{i}.txt"), ())).collect();
|
||||
assert!(
|
||||
guard_local_wipe(&filter_baseline(&baseline, &rules), &local).is_err(),
|
||||
"a {size}-path baseline against an empty root is not a deletion"
|
||||
);
|
||||
}
|
||||
|
||||
// And the guard steps aside the moment the real tree is back, so a
|
||||
// remounted folder resumes rather than staying wedged.
|
||||
std::fs::write(root.join("file0.txt"), b"back").unwrap();
|
||||
let mut local: HashMap<String, LocalItem> = HashMap::new();
|
||||
walk_local_tree(&root, &root, &rules, &writing, &mut local, &mut progress).unwrap();
|
||||
let baseline: HashMap<String, ()> = [("file0.txt".to_string(), ())].into_iter().collect();
|
||||
assert!(guard_local_wipe(&filter_baseline(&baseline, &rules), &local).is_ok());
|
||||
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
}
|
||||
|
||||
/// A restored folder (features.md 5.2) starts with an empty baseline against
|
||||
/// an empty local directory and a full remote. That must reconcile as
|
||||
/// "download everything" rather than tripping the wipe guard — otherwise
|
||||
@@ -1764,6 +2069,7 @@ mod tests {
|
||||
remote_uid: Some("v~l".into()),
|
||||
local_mtime: 0,
|
||||
local_size: 0,
|
||||
local_mtime_ns: None,
|
||||
remote_rev: Some(mtime.to_string()),
|
||||
remote_hash: Some(size.to_string()),
|
||||
},
|
||||
@@ -1801,6 +2107,7 @@ mod tests {
|
||||
remote_uid: Some("v~l".into()),
|
||||
local_mtime: 0,
|
||||
local_size: 0,
|
||||
local_mtime_ns: None,
|
||||
remote_rev: None,
|
||||
remote_hash: None,
|
||||
},
|
||||
@@ -1815,6 +2122,7 @@ mod tests {
|
||||
remote_uid: Some("v~l".into()),
|
||||
local_mtime: 10,
|
||||
local_size: 20,
|
||||
local_mtime_ns: None,
|
||||
remote_rev: Some("1700".into()),
|
||||
remote_hash: Some("4096".into()),
|
||||
};
|
||||
@@ -1828,6 +2136,7 @@ mod tests {
|
||||
remote_uid: Some("v~l".into()),
|
||||
local_mtime: 0,
|
||||
local_size: 0,
|
||||
local_mtime_ns: None,
|
||||
remote_rev: None,
|
||||
remote_hash: None,
|
||||
};
|
||||
@@ -1863,6 +2172,7 @@ mod tests {
|
||||
uid: uid(),
|
||||
mtime: 0,
|
||||
size: 10,
|
||||
local: None,
|
||||
};
|
||||
assert_eq!(op.kind(), ActivityKind::Download);
|
||||
}
|
||||
@@ -1918,6 +2228,204 @@ mod tests {
|
||||
std::fs::remove_dir_all(dir).unwrap();
|
||||
}
|
||||
|
||||
/// B49's remaining cases. Preservation is the step that makes a download
|
||||
/// safe to publish, so every way it can fail has to leave the local file
|
||||
/// where it was — the download is aborted, not the file replaced.
|
||||
#[test]
|
||||
fn conflict_preservation_failures_all_keep_the_local_file() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
|
||||
// An unwritable directory: the conflict copy cannot be created at all.
|
||||
let dir = sync_test_dir("conflict-eacces");
|
||||
let source = dir.join("notes.txt");
|
||||
std::fs::write(&source, b"the only copy").unwrap();
|
||||
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o500)).unwrap();
|
||||
let denied = preserve_conflict_copy(&source, 1700);
|
||||
std::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)).unwrap();
|
||||
assert!(denied.is_err(), "a read-only directory must fail the copy");
|
||||
assert_eq!(
|
||||
std::fs::read(&source).unwrap(),
|
||||
b"the only copy",
|
||||
"the local file survives byte-identical"
|
||||
);
|
||||
std::fs::remove_dir_all(&dir).unwrap();
|
||||
|
||||
// A source that vanished between planning and preservation. The pass
|
||||
// must report it rather than publish the download over the gap.
|
||||
let dir = sync_test_dir("conflict-enoent");
|
||||
assert!(preserve_conflict_copy(&dir.join("gone.txt"), 1700).is_err());
|
||||
std::fs::remove_dir_all(&dir).unwrap();
|
||||
|
||||
// A write error partway through the copy — induced by making the
|
||||
// destination name a directory, so `create_new` fails as something
|
||||
// other than `AlreadyExists`.
|
||||
let dir = sync_test_dir("conflict-eisdir");
|
||||
let source = dir.join("notes.txt");
|
||||
std::fs::write(&source, b"still the only copy").unwrap();
|
||||
std::fs::create_dir(conflict_path(&source, 1700)).unwrap();
|
||||
// Suffix 0 is taken by a directory; `create_new` on it reports
|
||||
// `AlreadyExists`, so preservation moves on to suffix 1 and succeeds.
|
||||
// What must not happen is the source being removed without a copy.
|
||||
let preserved = preserve_conflict_copy(&source, 1700).unwrap();
|
||||
assert!(preserved.is_file());
|
||||
assert_eq!(std::fs::read(&preserved).unwrap(), b"still the only copy");
|
||||
assert!(!source.exists());
|
||||
std::fs::remove_dir_all(&dir).unwrap();
|
||||
}
|
||||
|
||||
/// A conflict copy keeps the bytes *and* the metadata that identifies them,
|
||||
/// so the next pass uploads the user's version rather than a file that looks
|
||||
/// freshly written.
|
||||
#[test]
|
||||
fn a_conflict_copy_is_byte_and_mtime_identical() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
|
||||
let dir = sync_test_dir("conflict-identity");
|
||||
let source = dir.join("notes.txt");
|
||||
std::fs::write(&source, b"local edit that must survive").unwrap();
|
||||
let before = std::fs::metadata(&source).unwrap();
|
||||
|
||||
let preserved = preserve_conflict_copy(&source, 1700).unwrap();
|
||||
let after = std::fs::metadata(&preserved).unwrap();
|
||||
|
||||
assert_eq!(
|
||||
std::fs::read(&preserved).unwrap(),
|
||||
b"local edit that must survive"
|
||||
);
|
||||
assert_eq!(after.len(), before.len());
|
||||
assert_eq!(after.modified().unwrap(), before.modified().unwrap());
|
||||
assert_eq!(after.permissions().mode(), before.permissions().mode());
|
||||
std::fs::remove_dir_all(&dir).unwrap();
|
||||
}
|
||||
|
||||
// ---- B55: an incomplete local scan is never a set of deletions ---------
|
||||
|
||||
fn no_progress() -> impl FnMut(&str) {
|
||||
|_: &str| {}
|
||||
}
|
||||
|
||||
/// A directory the pass cannot read must fail the pass. Omitting it would
|
||||
/// hand reconciliation a baseline-minus-subtree, which it reads as the user
|
||||
/// having deleted that subtree and propagates to Drive (B55).
|
||||
#[test]
|
||||
fn an_unreadable_subdirectory_fails_the_scan_instead_of_shrinking_it() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
|
||||
let root = sync_test_dir("scan-eacces");
|
||||
std::fs::write(root.join("keep.txt"), b"a").unwrap();
|
||||
let locked = root.join("locked");
|
||||
std::fs::create_dir(&locked).unwrap();
|
||||
std::fs::write(locked.join("inner.txt"), b"b").unwrap();
|
||||
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o000)).unwrap();
|
||||
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut out = HashMap::new();
|
||||
let mut progress = no_progress();
|
||||
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
|
||||
|
||||
std::fs::set_permissions(&locked, std::fs::Permissions::from_mode(0o700)).unwrap();
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
|
||||
let error = result.expect_err("an unreadable directory must abort the pass");
|
||||
assert!(
|
||||
error.contains("locked"),
|
||||
"the error should name the subtree it could not read: {error}"
|
||||
);
|
||||
}
|
||||
|
||||
/// The same rule one level up: if the sync root itself cannot be read, the
|
||||
/// pass sees nothing, and "nothing" must never mean "delete everything".
|
||||
#[test]
|
||||
fn an_unreadable_root_fails_the_scan() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
|
||||
let root = sync_test_dir("scan-root-eacces");
|
||||
std::fs::write(root.join("keep.txt"), b"a").unwrap();
|
||||
std::fs::set_permissions(&root, std::fs::Permissions::from_mode(0o000)).unwrap();
|
||||
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut out = HashMap::new();
|
||||
let mut progress = no_progress();
|
||||
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
|
||||
|
||||
std::fs::set_permissions(&root, std::fs::Permissions::from_mode(0o700)).unwrap();
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
|
||||
assert!(result.is_err());
|
||||
assert!(out.is_empty());
|
||||
}
|
||||
|
||||
/// A root that is gone entirely — an unmounted device folder, a removed
|
||||
/// directory — is the sharpest version of the same case: every baseline path
|
||||
/// is "missing", and the scan must refuse rather than report an empty tree.
|
||||
#[test]
|
||||
fn an_absent_root_fails_the_scan() {
|
||||
let root = sync_test_dir("scan-absent");
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut out = HashMap::new();
|
||||
let mut progress = no_progress();
|
||||
assert!(walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress).is_err());
|
||||
assert!(out.is_empty());
|
||||
}
|
||||
|
||||
/// A name that is not valid UTF-8 cannot be keyed into the scan, so it
|
||||
/// cannot be compared against the baseline either. Refusing the pass is the
|
||||
/// only safe reading: skipping it would classify the file as deleted.
|
||||
#[test]
|
||||
fn a_non_utf8_name_fails_the_scan() {
|
||||
use std::ffi::OsStr;
|
||||
use std::os::unix::ffi::OsStrExt as _;
|
||||
|
||||
let root = sync_test_dir("scan-non-utf8");
|
||||
std::fs::write(root.join(OsStr::from_bytes(b"bad\xffname")), b"x").unwrap();
|
||||
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut out = HashMap::new();
|
||||
let mut progress = no_progress();
|
||||
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut progress);
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
|
||||
let error = result.expect_err("a name the scan cannot key must abort the pass");
|
||||
assert!(error.contains("UTF-8"), "{error}");
|
||||
}
|
||||
|
||||
/// The complement: a scan that *can* read everything reports everything, so
|
||||
/// the strictness above does not come at the cost of a working pass. A
|
||||
/// symlink is skipped by design rather than by failure, and a dangling one
|
||||
/// stats fine through `symlink_metadata`.
|
||||
#[test]
|
||||
fn a_readable_tree_scans_completely() {
|
||||
let root = sync_test_dir("scan-complete");
|
||||
std::fs::write(root.join("a.txt"), b"aa").unwrap();
|
||||
std::fs::create_dir(root.join("sub")).unwrap();
|
||||
std::fs::write(root.join("sub/b.txt"), b"bbb").unwrap();
|
||||
std::os::unix::fs::symlink("nowhere", root.join("dangling")).unwrap();
|
||||
|
||||
let rules = rules_for(&[]);
|
||||
let writing = HashSet::new();
|
||||
let mut out = HashMap::new();
|
||||
let mut seen: Vec<String> = Vec::new();
|
||||
let result = walk_local_tree(&root, &root, &rules, &writing, &mut out, &mut |rel| {
|
||||
seen.push(rel.to_string())
|
||||
});
|
||||
std::fs::remove_dir_all(&root).unwrap();
|
||||
|
||||
result.unwrap();
|
||||
let mut keys: Vec<&str> = out.keys().map(String::as_str).collect();
|
||||
keys.sort_unstable();
|
||||
assert_eq!(keys, vec!["a.txt", "sub", "sub/b.txt"]);
|
||||
assert_eq!(out["a.txt"].size, 2);
|
||||
assert_eq!(out["sub/b.txt"].size, 3);
|
||||
assert!(out["sub"].is_dir);
|
||||
assert_eq!(seen.len(), 3, "every reported entry ticks scan progress");
|
||||
}
|
||||
|
||||
fn sync_test_dir(label: &str) -> PathBuf {
|
||||
let nonce = SystemTime::now()
|
||||
.duration_since(SystemTime::UNIX_EPOCH)
|
||||
|
||||
@@ -32,17 +32,24 @@ pub(crate) enum SyncMsg {
|
||||
ReconcileAll,
|
||||
/// Re-read the folder set and adjust the filesystem watches (after add/remove).
|
||||
Rewatch,
|
||||
/// The daemon is tearing down: finish the current pass and end the engine
|
||||
/// thread, so the mount can join it (bugs.md B44).
|
||||
Stop,
|
||||
}
|
||||
|
||||
/// Start the sync engine: a control thread that owns the filesystem watcher and
|
||||
/// serialises reconcile passes, plus a periodic poll thread. The engine receives
|
||||
/// on `rx`; senders live in [`Core::sync_tx`].
|
||||
pub(crate) fn spawn(core: Core, rx: Receiver<SyncMsg>) {
|
||||
if let Err(e) = std::thread::Builder::new()
|
||||
pub(crate) fn spawn(core: Core, rx: Receiver<SyncMsg>) -> Option<std::thread::JoinHandle<()>> {
|
||||
match std::thread::Builder::new()
|
||||
.name("pdfs-sync".into())
|
||||
.spawn(move || engine_loop(core, rx))
|
||||
{
|
||||
warn!(error = %e, "failed to start sync engine");
|
||||
Ok(handle) => Some(handle),
|
||||
Err(e) => {
|
||||
warn!(error = %e, "failed to start sync engine");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -74,11 +81,11 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
|
||||
// Periodic remote poll.
|
||||
{
|
||||
let tx = core.sync_tx.clone();
|
||||
let shutdown = core.shutdown.clone();
|
||||
std::thread::Builder::new()
|
||||
.name("pdfs-sync-poll".into())
|
||||
.spawn(move || {
|
||||
loop {
|
||||
std::thread::sleep(POLL_INTERVAL);
|
||||
while shutdown.sleep(POLL_INTERVAL) {
|
||||
if tx.send(SyncMsg::ReconcileAll).is_err() {
|
||||
break;
|
||||
}
|
||||
@@ -94,8 +101,12 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
|
||||
let mut ids: HashSet<i64> = HashSet::new();
|
||||
let mut all = false;
|
||||
let mut do_rewatch = false;
|
||||
classify(msg, &mut ids, &mut all, &mut do_rewatch);
|
||||
settle(&rx, &mut ids, &mut all, &mut do_rewatch);
|
||||
if !classify(msg, &mut ids, &mut all, &mut do_rewatch)
|
||||
|| !settle(&rx, &mut ids, &mut all, &mut do_rewatch)
|
||||
{
|
||||
info!("sync engine stopping");
|
||||
return;
|
||||
}
|
||||
|
||||
if do_rewatch {
|
||||
rewatch(&core, &mut watcher, &watched);
|
||||
@@ -116,9 +127,9 @@ fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
|
||||
fn poll_only_loop(core: &Core, rx: Receiver<SyncMsg>) {
|
||||
{
|
||||
let tx = core.sync_tx.clone();
|
||||
let shutdown = core.shutdown.clone();
|
||||
std::thread::spawn(move || {
|
||||
loop {
|
||||
std::thread::sleep(POLL_INTERVAL);
|
||||
while shutdown.sleep(POLL_INTERVAL) {
|
||||
if tx.send(SyncMsg::ReconcileAll).is_err() {
|
||||
break;
|
||||
}
|
||||
@@ -130,8 +141,12 @@ fn poll_only_loop(core: &Core, rx: Receiver<SyncMsg>) {
|
||||
let mut ids: HashSet<i64> = HashSet::new();
|
||||
let mut all = false;
|
||||
let mut do_rewatch = false;
|
||||
classify(msg, &mut ids, &mut all, &mut do_rewatch);
|
||||
settle(&rx, &mut ids, &mut all, &mut do_rewatch);
|
||||
if !classify(msg, &mut ids, &mut all, &mut do_rewatch)
|
||||
|| !settle(&rx, &mut ids, &mut all, &mut do_rewatch)
|
||||
{
|
||||
info!("sync engine stopping");
|
||||
return;
|
||||
}
|
||||
if all {
|
||||
reconcile_all(core);
|
||||
} else {
|
||||
@@ -158,7 +173,7 @@ pub(super) fn settle_with(
|
||||
do_rewatch: &mut bool,
|
||||
quiet: Duration,
|
||||
cap: Duration,
|
||||
) {
|
||||
) -> bool {
|
||||
let deadline = Instant::now() + cap;
|
||||
loop {
|
||||
let now = Instant::now();
|
||||
@@ -170,28 +185,49 @@ pub(super) fn settle_with(
|
||||
match rx.recv_timeout(wait) {
|
||||
// Another event: the burst is still going, so the quiet window
|
||||
// starts again from here.
|
||||
Ok(m) => classify(m, ids, all, do_rewatch),
|
||||
// A stop lands mid-burst: give up the rest of the burst rather
|
||||
// than settling for it, and let the caller end the loop.
|
||||
Ok(m) => {
|
||||
if !classify(m, ids, all, do_rewatch) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
// Quiet for a full window: whatever was being written has stopped.
|
||||
Err(RecvTimeoutError::Timeout) => break,
|
||||
// Every sender is gone; the caller's loop is about to end anyway.
|
||||
Err(RecvTimeoutError::Disconnected) => break,
|
||||
}
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
/// [`settle_with`] at the real timings.
|
||||
fn settle(rx: &Receiver<SyncMsg>, ids: &mut HashSet<i64>, all: &mut bool, do_rewatch: &mut bool) {
|
||||
settle_with(rx, ids, all, do_rewatch, DEBOUNCE, MAX_SETTLE);
|
||||
fn settle(
|
||||
rx: &Receiver<SyncMsg>,
|
||||
ids: &mut HashSet<i64>,
|
||||
all: &mut bool,
|
||||
do_rewatch: &mut bool,
|
||||
) -> bool {
|
||||
settle_with(rx, ids, all, do_rewatch, DEBOUNCE, MAX_SETTLE)
|
||||
}
|
||||
|
||||
pub(super) fn classify(msg: SyncMsg, ids: &mut HashSet<i64>, all: &mut bool, rewatch: &mut bool) {
|
||||
/// Fold one message into the pending work of the current burst. Returns `false`
|
||||
/// for [`SyncMsg::Stop`], which is not work but an instruction to end the loop.
|
||||
pub(super) fn classify(
|
||||
msg: SyncMsg,
|
||||
ids: &mut HashSet<i64>,
|
||||
all: &mut bool,
|
||||
rewatch: &mut bool,
|
||||
) -> bool {
|
||||
match msg {
|
||||
SyncMsg::Reconcile(id) => {
|
||||
ids.insert(id);
|
||||
}
|
||||
SyncMsg::ReconcileAll => *all = true,
|
||||
SyncMsg::Rewatch => *rewatch = true,
|
||||
SyncMsg::Stop => return false,
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
/// Reconcile every mirror folder in turn.
|
||||
|
||||
@@ -2,9 +2,73 @@
|
||||
|
||||
use super::*;
|
||||
|
||||
/// What a reconcile pass remembers about the local side of one path: enough to
|
||||
/// recognise the same content again, and no more.
|
||||
///
|
||||
/// `mtime_ns` is the whole reason this is a struct rather than a pair — see
|
||||
/// [`StoredSyncEntry::local_mtime_ns`](pdfs_core::db::StoredSyncEntry).
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(super) struct LocalSig {
|
||||
pub(super) mtime: i64,
|
||||
pub(super) mtime_ns: Option<i64>,
|
||||
pub(super) size: i64,
|
||||
}
|
||||
|
||||
impl LocalSig {
|
||||
/// Whether these two observations describe the same local content.
|
||||
///
|
||||
/// Nanoseconds decide it when both sides have them; otherwise the comparison
|
||||
/// falls back to whole seconds, which is what a baseline row written before
|
||||
/// schema 28 — or a filesystem that reports no sub-second time — can support
|
||||
/// (bugs.md B25).
|
||||
pub(super) fn same_content(&self, other: &Self) -> bool {
|
||||
if self.size != other.size {
|
||||
return false;
|
||||
}
|
||||
match (self.mtime_ns, other.mtime_ns) {
|
||||
(Some(a), Some(b)) => a == b,
|
||||
_ => self.mtime == other.mtime,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<&LocalItem> for LocalSig {
|
||||
fn from(item: &LocalItem) -> Self {
|
||||
Self {
|
||||
mtime: item.mtime,
|
||||
mtime_ns: item.mtime_ns,
|
||||
size: item.size,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<&std::fs::Metadata> for LocalSig {
|
||||
fn from(meta: &std::fs::Metadata) -> Self {
|
||||
Self {
|
||||
mtime: system_mtime(meta),
|
||||
mtime_ns: system_mtime_ns(meta),
|
||||
size: meta.len() as i64,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The baseline's record of the local side, as [`LocalSig`] compares it.
|
||||
impl From<&StoredSyncEntry> for LocalSig {
|
||||
fn from(entry: &StoredSyncEntry) -> Self {
|
||||
Self {
|
||||
mtime: entry.local_mtime,
|
||||
mtime_ns: entry.local_mtime_ns,
|
||||
size: entry.local_size,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) struct LocalItem {
|
||||
pub(super) is_dir: bool,
|
||||
pub(super) mtime: i64,
|
||||
/// [`LocalItem::mtime`] to nanosecond precision, when the filesystem
|
||||
/// reports one.
|
||||
pub(super) mtime_ns: Option<i64>,
|
||||
pub(super) size: i64,
|
||||
/// True when another process holds this file open for writing (detected via
|
||||
/// `/proc/*/fd`). An open-for-write file is kept in the map — so it is not
|
||||
@@ -91,11 +155,18 @@ pub(super) enum Pending {
|
||||
UploadRevision { rel: String, uid: NodeUid },
|
||||
/// Download remote `uid` to the local path, stamping `mtime`. `size` is the
|
||||
/// remote's reported size, used as the transfer's expected total.
|
||||
///
|
||||
/// `local` is what the planning walk saw at this path, or `None` if nothing
|
||||
/// was there. A download can run for minutes, so the apply step revalidates
|
||||
/// against it immediately before it renames over the destination: a writer
|
||||
/// that finished an edit in the meantime must not be overwritten without a
|
||||
/// conflict copy (bugs.md B39).
|
||||
Download {
|
||||
rel: String,
|
||||
uid: NodeUid,
|
||||
mtime: i64,
|
||||
size: i64,
|
||||
local: Option<LocalSig>,
|
||||
},
|
||||
/// Both sides changed: set the local copy aside, then download remote `uid`.
|
||||
Conflict {
|
||||
|
||||
@@ -218,7 +218,10 @@ pub(super) fn plan_file(
|
||||
}
|
||||
|
||||
let local_changed = local.is_some_and(|item| {
|
||||
baseline.is_none_or(|base| base.local_mtime != item.mtime || base.local_size != item.size)
|
||||
// Nanoseconds where both sides have them (bugs.md B25): whole seconds
|
||||
// could not see an edit made in the same second as the last sync that
|
||||
// left the file the same length, so it was never uploaded.
|
||||
baseline.is_none_or(|base| !LocalSig::from(item).same_content(&LocalSig::from(base)))
|
||||
});
|
||||
let remote_changed = remote.is_some_and(|item| {
|
||||
baseline.is_none_or(|base| remote_sig(base) != Some((item.mtime, item.size)))
|
||||
@@ -252,6 +255,7 @@ mod file_plan_tests {
|
||||
|
||||
fn local(mtime: i64, size: i64) -> LocalItem {
|
||||
LocalItem {
|
||||
mtime_ns: None,
|
||||
is_dir: false,
|
||||
mtime,
|
||||
size,
|
||||
@@ -278,12 +282,49 @@ mod file_plan_tests {
|
||||
rel_path: "file".into(),
|
||||
remote_uid: Some("v~l".into()),
|
||||
local_mtime,
|
||||
local_mtime_ns: None,
|
||||
local_size,
|
||||
remote_rev: Some(remote_mtime.to_string()),
|
||||
remote_hash: Some(remote_size.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Whole seconds cannot see an edit made inside the same second that leaves the
|
||||
/// file the same length, so a mirror silently stopped uploading it. With a
|
||||
/// nanosecond time on both sides it is an ordinary local change (bugs.md B25).
|
||||
#[test]
|
||||
fn a_same_second_same_size_edit_is_still_a_local_change() {
|
||||
let mut base = baseline(10, 20, 30, 40);
|
||||
base.local_mtime_ns = Some(10_000_000_000);
|
||||
let edited = LocalItem {
|
||||
mtime_ns: Some(10_400_000_000),
|
||||
..local(10, 20)
|
||||
};
|
||||
let remote = remote(30, 40);
|
||||
assert_eq!(
|
||||
plan_file(Some(&edited), Some(&remote), Some(&base)),
|
||||
FilePlan::UploadRevision
|
||||
);
|
||||
|
||||
// The same file untouched still reads as unchanged.
|
||||
let same = LocalItem {
|
||||
mtime_ns: Some(10_000_000_000),
|
||||
..local(10, 20)
|
||||
};
|
||||
assert_eq!(
|
||||
plan_file(Some(&same), Some(&remote), Some(&base)),
|
||||
FilePlan::Unchanged
|
||||
);
|
||||
|
||||
// And a row from before the column exists compares on seconds, exactly
|
||||
// as it did when it was written — no mass re-upload on upgrade.
|
||||
let old = baseline(10, 20, 30, 40);
|
||||
assert_eq!(
|
||||
plan_file(Some(&edited), Some(&remote), Some(&old)),
|
||||
FilePlan::Unchanged
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn file_plan_covers_every_presence_and_change_shape() {
|
||||
let same_local = local(10, 20);
|
||||
|
||||
@@ -0,0 +1,632 @@
|
||||
//! Importing a Google Photos Takeout export into Proton Photos.
|
||||
//!
|
||||
//! [`pdfs_core::takeout`] says what is in the archives; this is the part that
|
||||
//! talks to Proton. One import runs on its own thread and moves through four
|
||||
//! phases, reporting into the transfer registry as a job so the GUI's transfer
|
||||
//! list shows it without a bespoke widget:
|
||||
//!
|
||||
//! 1. **Scan** — read the archives' central directories and sidecars.
|
||||
//! 2. **Deduplicate against the account** — ask the photos volume which of the
|
||||
//! names already exist ([`ProtonPhotosClient::name_collisions`]), hash only
|
||||
//! those files, then confirm with the content digest. A library that is
|
||||
//! already half-synced by Proton Photos on a phone therefore costs one cheap
|
||||
//! request per 150 names and reads no media at all for the misses.
|
||||
//! 3. **Upload** the survivors, streaming each entry straight out of its zip —
|
||||
//! no extraction to disk, so the import needs no scratch space.
|
||||
//! 4. **Albums** — create the albums the export names that the account lacks,
|
||||
//! then add every photo of each album, the ones that were already on the
|
||||
//! account included.
|
||||
//!
|
||||
//! Re-running an interrupted import is the resume story: phase 2 finds
|
||||
//! everything the previous run uploaded and skips it. Nothing is persisted
|
||||
//! between runs.
|
||||
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::io::Read;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
|
||||
use pdfs_core::control::{ActivityKind, ImportSummary, TransferDirection};
|
||||
use pdfs_core::takeout::{Bucket, TakeoutPhoto, TakeoutScan, scan_archives};
|
||||
use pdfs_core::{CoreError, CoreResult};
|
||||
use proton_drive_rs::proton_sdk::ids::NodeUid;
|
||||
use proton_drive_rs::{PhotoTag, PhotoTagsUpdate, PhotoUploadMetadata};
|
||||
use sha1::{Digest, Sha1};
|
||||
use tracing::{info, warn};
|
||||
|
||||
use super::Core;
|
||||
use crate::transfers::CountingReader;
|
||||
|
||||
/// The one import this daemon may be running, and how the last one ended.
|
||||
///
|
||||
/// Process-global rather than a [`Core`] field because it must be *one* import
|
||||
/// per daemon however many `Core` clones exist (the forked on-demand mounts each
|
||||
/// hold one), and because a front-end asking for the result after the import
|
||||
/// finished must still find it.
|
||||
static IMPORT: std::sync::LazyLock<parking_lot::Mutex<ImportState>> =
|
||||
std::sync::LazyLock::new(|| parking_lot::Mutex::new(ImportState::default()));
|
||||
|
||||
#[derive(Default)]
|
||||
struct ImportState {
|
||||
running: bool,
|
||||
cancel: Option<Arc<AtomicBool>>,
|
||||
last: Option<ImportSummary>,
|
||||
}
|
||||
|
||||
/// Whether an import is running, and the last one's report.
|
||||
pub(crate) fn import_status() -> (bool, Option<ImportSummary>) {
|
||||
let state = IMPORT.lock();
|
||||
(state.running, state.last.clone())
|
||||
}
|
||||
|
||||
/// Ask the running import to stop. False when none is running.
|
||||
pub(crate) fn cancel_import() -> bool {
|
||||
match &IMPORT.lock().cancel {
|
||||
Some(flag) => {
|
||||
flag.store(true, Ordering::Relaxed);
|
||||
true
|
||||
}
|
||||
None => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// One photo to import, after the copies Google keeps per album are folded into
|
||||
/// a single upload that joins several albums.
|
||||
struct Planned {
|
||||
photo: TakeoutPhoto,
|
||||
/// Every album this photo belongs to, by export title.
|
||||
albums: Vec<String>,
|
||||
/// Filled in phase 2/3: where the photo ended up on the account.
|
||||
uid: Option<NodeUid>,
|
||||
}
|
||||
|
||||
/// Fold the scan into one entry per distinct photo.
|
||||
///
|
||||
/// Google stores a full copy of a photo in every album folder it appears in, so
|
||||
/// the same file is in the export two or three times over. They are folded on
|
||||
/// `(name, size)`: a byte-identical copy under the same name is the same photo,
|
||||
/// and the album memberships of the copies are unioned onto the survivor. Two
|
||||
/// genuinely different photos sharing both a name and an exact size would be
|
||||
/// merged — vanishingly unlikely, and the cost of being wrong is one photo
|
||||
/// filed under an extra album rather than lost data.
|
||||
fn plan(scan: &TakeoutScan) -> Vec<Planned> {
|
||||
let mut order: Vec<(String, u64)> = Vec::new();
|
||||
let mut by_key: HashMap<(String, u64), Planned> = HashMap::new();
|
||||
|
||||
for photo in &scan.photos {
|
||||
let key = (photo.name.clone(), photo.size);
|
||||
let entry = by_key.entry(key.clone()).or_insert_with(|| {
|
||||
order.push(key.clone());
|
||||
Planned {
|
||||
photo: photo.clone(),
|
||||
albums: Vec::new(),
|
||||
uid: None,
|
||||
}
|
||||
});
|
||||
if let Bucket::Album(title) = &photo.bucket
|
||||
&& !entry.albums.iter().any(|existing| existing == title)
|
||||
{
|
||||
entry.albums.push(title.clone());
|
||||
}
|
||||
// A copy that carries a capture time wins over one that does not: the
|
||||
// album copy and the year-folder copy do not always both have a sidecar.
|
||||
if entry.photo.capture_time.is_none() && photo.capture_time.is_some() {
|
||||
entry.photo.capture_time = photo.capture_time;
|
||||
}
|
||||
entry.photo.favorite |= photo.favorite;
|
||||
}
|
||||
|
||||
order
|
||||
.into_iter()
|
||||
.filter_map(|key| by_key.remove(&key))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Open one archive and stream `entry` through `sink`.
|
||||
fn read_entry<R>(
|
||||
archive: &mut zip::ZipArchive<std::io::BufReader<std::fs::File>>,
|
||||
entry: &str,
|
||||
sink: impl FnOnce(&mut (dyn Read + Send)) -> R,
|
||||
) -> CoreResult<R> {
|
||||
let mut file = archive
|
||||
.by_name(entry)
|
||||
.map_err(|e| CoreError::invalid(format!("{entry}: {e}")))?;
|
||||
Ok(sink(&mut file))
|
||||
}
|
||||
|
||||
impl Core {
|
||||
/// Import Google Photos Takeout archives. Blocking; run on its own thread.
|
||||
///
|
||||
/// `cancel` stops the run at the next photo boundary — an upload already on
|
||||
/// the wire is allowed to finish, so a cancelled import never leaves a
|
||||
/// half-written revision behind.
|
||||
pub(crate) fn import_takeout(
|
||||
&self,
|
||||
archives: Vec<PathBuf>,
|
||||
dry_run: bool,
|
||||
cancel: Arc<AtomicBool>,
|
||||
) -> CoreResult<ImportSummary> {
|
||||
let job = self.transfers.begin_job("Importing Google Photos");
|
||||
job.detail("reading archives");
|
||||
|
||||
let scan = scan_archives(&archives)?;
|
||||
let mut planned = plan(&scan);
|
||||
let mut summary = ImportSummary {
|
||||
found: planned.len(),
|
||||
skipped_trashed: scan.skipped_trashed,
|
||||
..Default::default()
|
||||
};
|
||||
job.set_total(planned.len() as u64);
|
||||
info!(
|
||||
archives = archives.len(),
|
||||
photos = summary.found,
|
||||
albums = scan.albums.len(),
|
||||
trashed = summary.skipped_trashed,
|
||||
"takeout scan complete"
|
||||
);
|
||||
|
||||
let photos = self.photos();
|
||||
if self
|
||||
.rt
|
||||
.block_on(photos.get_photos_root())
|
||||
.map_err(|e| CoreError::from_api(&e, "resolve photos library"))?
|
||||
.is_none()
|
||||
{
|
||||
return Err(CoreError::invalid(
|
||||
"this account has no Proton Photos library yet; open Proton Photos once to create it, then import again",
|
||||
));
|
||||
}
|
||||
|
||||
// Phase 2: which of these are already on the account.
|
||||
job.detail("checking for photos you already have");
|
||||
self.mark_existing(&mut planned, &archives, &mut summary, &cancel)?;
|
||||
|
||||
if dry_run {
|
||||
summary.duplicates = planned.iter().filter(|p| p.uid.is_some()).count();
|
||||
return Ok(summary);
|
||||
}
|
||||
|
||||
// Phase 3: upload what is left, one archive at a time so each zip's
|
||||
// central directory is parsed once.
|
||||
self.upload_planned(&mut planned, &archives, &mut summary, &cancel, &job)?;
|
||||
|
||||
// Phase 4: albums. Attempted even after a cancel — the photos that did
|
||||
// upload should still land in their albums.
|
||||
if !scan.albums.is_empty() {
|
||||
job.detail("sorting photos into albums");
|
||||
if let Err(e) = self.link_albums(&planned, &scan.albums, &mut summary) {
|
||||
warn!(error = %e, "takeout album linking failed");
|
||||
}
|
||||
}
|
||||
|
||||
self.spawn_timeline_refresh();
|
||||
summary.cancelled = cancel.load(Ordering::Relaxed);
|
||||
Ok(summary)
|
||||
}
|
||||
|
||||
/// Fill in `uid` for every planned photo the account already holds.
|
||||
///
|
||||
/// Two passes on purpose: the name-only query is one request per 150 photos
|
||||
/// and needs no file contents, so the expensive SHA-1 pass only runs for the
|
||||
/// handful of names that actually collide.
|
||||
fn mark_existing(
|
||||
&self,
|
||||
planned: &mut [Planned],
|
||||
archives: &[PathBuf],
|
||||
summary: &mut ImportSummary,
|
||||
cancel: &AtomicBool,
|
||||
) -> CoreResult<()> {
|
||||
let photos = self.photos();
|
||||
let names: Vec<String> = planned.iter().map(|p| p.photo.name.clone()).collect();
|
||||
let collisions = self
|
||||
.rt
|
||||
.block_on(photos.name_collisions(&names))
|
||||
.map_err(|e| CoreError::from_api(&e, "check for existing photos"))?;
|
||||
|
||||
// Hash the colliding files, grouped by archive so each zip opens once.
|
||||
let mut by_archive: BTreeMap<usize, Vec<usize>> = BTreeMap::new();
|
||||
for (index, planned) in planned.iter().enumerate() {
|
||||
if collisions.get(index).copied().unwrap_or(false) {
|
||||
by_archive
|
||||
.entry(planned.photo.archive)
|
||||
.or_default()
|
||||
.push(index);
|
||||
}
|
||||
}
|
||||
|
||||
// Kept as a pair list, not a map: the duplicate lookup answers in input
|
||||
// order, so the order these were built in is what maps answers back to
|
||||
// planned photos.
|
||||
let mut digests: Vec<(usize, String)> = Vec::new();
|
||||
for (archive_index, indices) in by_archive {
|
||||
if cancel.load(Ordering::Relaxed) {
|
||||
break;
|
||||
}
|
||||
let mut archive = open_archive(archives, archive_index)?;
|
||||
for index in indices {
|
||||
let entry = planned[index].photo.entry.clone();
|
||||
match read_entry(&mut archive, &entry, sha1_of) {
|
||||
Ok(Ok(digest)) => digests.push((index, digest)),
|
||||
Ok(Err(e)) | Err(e) => {
|
||||
warn!(entry, error = %e, "cannot hash takeout entry");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if digests.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let items: Vec<(String, String)> = digests
|
||||
.iter()
|
||||
.map(|(index, digest)| (planned[*index].photo.name.clone(), digest.clone()))
|
||||
.collect();
|
||||
let matches = self
|
||||
.rt
|
||||
.block_on(photos.find_duplicates_many(&items))
|
||||
.map_err(|e| CoreError::from_api(&e, "check for existing photos"))?;
|
||||
|
||||
for (slot, (index, _)) in digests.iter().enumerate() {
|
||||
if let Some(uid) = matches.get(slot).and_then(|uids| uids.first()) {
|
||||
planned[*index].uid = Some(uid.clone());
|
||||
summary.duplicates += 1;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Upload every planned photo that is not already on the account.
|
||||
fn upload_planned(
|
||||
&self,
|
||||
planned: &mut [Planned],
|
||||
archives: &[PathBuf],
|
||||
summary: &mut ImportSummary,
|
||||
cancel: &AtomicBool,
|
||||
job: &crate::transfers::JobGuard,
|
||||
) -> CoreResult<()> {
|
||||
let photos = self.photos();
|
||||
let mut favorites: Vec<NodeUid> = Vec::new();
|
||||
|
||||
let mut by_archive: BTreeMap<usize, Vec<usize>> = BTreeMap::new();
|
||||
for (index, item) in planned.iter().enumerate() {
|
||||
if item.uid.is_none() {
|
||||
by_archive
|
||||
.entry(item.photo.archive)
|
||||
.or_default()
|
||||
.push(index);
|
||||
}
|
||||
}
|
||||
|
||||
for (archive_index, indices) in by_archive {
|
||||
if cancel.load(Ordering::Relaxed) || self.shutdown.is_stopping() {
|
||||
break;
|
||||
}
|
||||
let mut archive = open_archive(archives, archive_index)?;
|
||||
|
||||
for index in indices {
|
||||
if cancel.load(Ordering::Relaxed) || self.shutdown.is_stopping() {
|
||||
break;
|
||||
}
|
||||
let photo = planned[index].photo.clone();
|
||||
job.detail(photo.name.clone());
|
||||
|
||||
let guard = self.transfers.begin(
|
||||
photo.name.clone(),
|
||||
"",
|
||||
TransferDirection::Upload,
|
||||
photo.size,
|
||||
);
|
||||
let metadata = PhotoUploadMetadata {
|
||||
capture_time: photo.capture_time,
|
||||
..Default::default()
|
||||
};
|
||||
let uploaded = read_entry(&mut archive, &photo.entry, |reader| {
|
||||
let counting = CountingReader::new(reader, &guard);
|
||||
self.rt.block_on(photos.upload_photo_from(
|
||||
&photo.name,
|
||||
&photo.media_type,
|
||||
counting,
|
||||
photo.size as i64,
|
||||
Vec::new(),
|
||||
metadata,
|
||||
false,
|
||||
))
|
||||
});
|
||||
drop(guard);
|
||||
job.step();
|
||||
|
||||
match uploaded {
|
||||
Ok(Ok(uid)) => {
|
||||
summary.uploaded += 1;
|
||||
summary.bytes += photo.size;
|
||||
if photo.favorite {
|
||||
favorites.push(uid.clone());
|
||||
}
|
||||
planned[index].uid = Some(uid);
|
||||
}
|
||||
Ok(Err(e)) => {
|
||||
summary.failed += 1;
|
||||
warn!(name = photo.name, error = %e, "takeout photo upload failed");
|
||||
}
|
||||
Err(e) => {
|
||||
summary.failed += 1;
|
||||
warn!(name = photo.name, error = %e, "cannot read takeout entry");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Favourites are a separate call: the upload seal takes classification
|
||||
// tags, but `Favorite` is not one of them — it goes through the photos
|
||||
// volume's dedicated favourite endpoint.
|
||||
if !favorites.is_empty() {
|
||||
let updates: Vec<PhotoTagsUpdate> = favorites
|
||||
.into_iter()
|
||||
.map(|node_uid| PhotoTagsUpdate {
|
||||
node_uid,
|
||||
tags_to_add: vec![PhotoTag::Favorite],
|
||||
tags_to_remove: Vec::new(),
|
||||
})
|
||||
.collect();
|
||||
if let Err(e) = self.rt.block_on(photos.update_photos(&updates)) {
|
||||
warn!(error = %e, "marking imported favourites failed");
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Create the albums the export names that the account lacks, then add each
|
||||
/// album's photos to it.
|
||||
fn link_albums(
|
||||
&self,
|
||||
planned: &[Planned],
|
||||
albums: &[String],
|
||||
summary: &mut ImportSummary,
|
||||
) -> CoreResult<()> {
|
||||
let photos = self.photos();
|
||||
|
||||
// Existing albums by decrypted name, so a second import of the same
|
||||
// export adds to the album it made the first time instead of a twin.
|
||||
let mut existing: HashMap<String, NodeUid> = HashMap::new();
|
||||
let uids = self
|
||||
.rt
|
||||
.block_on(photos.enumerate_album_node_uids())
|
||||
.map_err(|e| CoreError::from_api(&e, "list albums"))?;
|
||||
if !uids.is_empty() {
|
||||
let albums_nodes = self
|
||||
.rt
|
||||
.block_on(photos.enumerate_nodes(&uids))
|
||||
.map_err(|e| CoreError::from_api(&e, "read albums"))?;
|
||||
for node in albums_nodes {
|
||||
existing.entry(node.name.clone()).or_insert(node.uid);
|
||||
}
|
||||
}
|
||||
|
||||
for title in albums {
|
||||
let members: Vec<NodeUid> = planned
|
||||
.iter()
|
||||
.filter(|item| item.albums.iter().any(|album| album == title))
|
||||
.filter_map(|item| item.uid.clone())
|
||||
.collect();
|
||||
if members.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let album_uid = match existing.get(title) {
|
||||
Some(uid) => uid.clone(),
|
||||
None => match self.rt.block_on(photos.create_album(title)) {
|
||||
Ok(uid) => {
|
||||
summary.albums_created += 1;
|
||||
existing.insert(title.clone(), uid.clone());
|
||||
uid
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(album = title, error = %e, "cannot create album");
|
||||
continue;
|
||||
}
|
||||
},
|
||||
};
|
||||
|
||||
match self
|
||||
.rt
|
||||
.block_on(photos.add_photos_to_album(&album_uid, &members))
|
||||
{
|
||||
Ok(outcomes) => {
|
||||
for (uid, outcome) in outcomes {
|
||||
match outcome {
|
||||
Ok(()) => summary.album_links += 1,
|
||||
// A photo already in the album is the expected
|
||||
// outcome of a re-run, not a failure worth counting.
|
||||
Err(e) => {
|
||||
warn!(album = title, %uid, error = %e, "cannot add photo to album")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => warn!(album = title, error = %e, "album addition failed"),
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Run an import on a background thread, logging its outcome to the activity
|
||||
/// feed. Returns immediately.
|
||||
///
|
||||
/// Refuses while one is already running: two imports of the same export
|
||||
/// would race each other's duplicate detection and upload everything twice.
|
||||
pub(crate) fn spawn_takeout_import(
|
||||
&self,
|
||||
archives: Vec<PathBuf>,
|
||||
dry_run: bool,
|
||||
) -> CoreResult<()> {
|
||||
let cancel = Arc::new(AtomicBool::new(false));
|
||||
{
|
||||
let mut state = IMPORT.lock();
|
||||
if state.running {
|
||||
return Err(CoreError::conflict(
|
||||
"an import is already running; wait for it to finish or cancel it",
|
||||
));
|
||||
}
|
||||
state.running = true;
|
||||
state.cancel = Some(cancel.clone());
|
||||
state.last = None;
|
||||
}
|
||||
|
||||
let core = self.clone();
|
||||
std::thread::spawn(move || {
|
||||
let outcome = core.import_takeout(archives, dry_run, cancel);
|
||||
{
|
||||
let mut state = IMPORT.lock();
|
||||
state.running = false;
|
||||
state.cancel = None;
|
||||
state.last = outcome.as_ref().ok().cloned();
|
||||
}
|
||||
match outcome {
|
||||
Ok(summary) => {
|
||||
let mut parts = vec![format!("{} uploaded", summary.uploaded)];
|
||||
if summary.duplicates > 0 {
|
||||
parts.push(format!("{} already there", summary.duplicates));
|
||||
}
|
||||
if summary.albums_created > 0 {
|
||||
parts.push(format!("{} albums", summary.albums_created));
|
||||
}
|
||||
if summary.failed > 0 {
|
||||
parts.push(format!("{} failed", summary.failed));
|
||||
}
|
||||
if summary.cancelled {
|
||||
parts.push("cancelled".to_string());
|
||||
}
|
||||
core.log_activity(
|
||||
ActivityKind::Upload,
|
||||
"Google Photos import",
|
||||
parts.join(" · "),
|
||||
summary.failed == 0 && !summary.cancelled,
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(error = %e, "takeout import failed");
|
||||
core.log_activity(ActivityKind::Upload, "Google Photos import", &e, false);
|
||||
}
|
||||
}
|
||||
});
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Open one of the import's archives by index.
|
||||
fn open_archive(
|
||||
archives: &[PathBuf],
|
||||
index: usize,
|
||||
) -> CoreResult<zip::ZipArchive<std::io::BufReader<std::fs::File>>> {
|
||||
let path = archives
|
||||
.get(index)
|
||||
.ok_or_else(|| CoreError::internal(format!("archive {index} is not in the import")))?;
|
||||
let file = std::fs::File::open(path)
|
||||
.map_err(|e| CoreError::invalid(format!("cannot open {}: {e}", path.display())))?;
|
||||
zip::ZipArchive::new(std::io::BufReader::new(file))
|
||||
.map_err(|e| CoreError::invalid(format!("{} is not a readable zip: {e}", path.display())))
|
||||
}
|
||||
|
||||
/// The lowercase-hex SHA-1 of everything `reader` yields — the digest Proton's
|
||||
/// photo duplicate detection is keyed on.
|
||||
fn sha1_of(reader: &mut (dyn Read + Send)) -> CoreResult<String> {
|
||||
let mut hasher = Sha1::new();
|
||||
let mut buffer = vec![0u8; 256 * 1024];
|
||||
loop {
|
||||
let read = reader
|
||||
.read(&mut buffer)
|
||||
.map_err(|e| CoreError::invalid(format!("read failed: {e}")))?;
|
||||
if read == 0 {
|
||||
break;
|
||||
}
|
||||
hasher.update(&buffer[..read]);
|
||||
}
|
||||
Ok(hex_lower(&hasher.finalize()))
|
||||
}
|
||||
|
||||
fn hex_lower(bytes: &[u8]) -> String {
|
||||
let mut out = String::with_capacity(bytes.len() * 2);
|
||||
for byte in bytes {
|
||||
out.push_str(&format!("{byte:02x}"));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use pdfs_core::takeout::TakeoutPhoto;
|
||||
|
||||
fn photo(name: &str, size: u64, bucket: Bucket) -> TakeoutPhoto {
|
||||
TakeoutPhoto {
|
||||
archive: 0,
|
||||
entry: format!("Takeout/Google Photos/x/{name}"),
|
||||
name: name.to_string(),
|
||||
size,
|
||||
media_type: "image/jpeg".to_string(),
|
||||
capture_time: None,
|
||||
favorite: false,
|
||||
bucket,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn album_copies_of_one_photo_fold_into_a_single_upload() {
|
||||
let scan = TakeoutScan {
|
||||
photos: vec![
|
||||
photo("IMG_1.jpg", 100, Bucket::Timeline),
|
||||
photo("IMG_1.jpg", 100, Bucket::Album("Iceland".into())),
|
||||
photo("IMG_1.jpg", 100, Bucket::Album("Best of".into())),
|
||||
photo("IMG_2.jpg", 200, Bucket::Timeline),
|
||||
],
|
||||
albums: vec!["Iceland".into(), "Best of".into()],
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let planned = plan(&scan);
|
||||
assert_eq!(planned.len(), 2, "the three copies are one upload");
|
||||
assert_eq!(planned[0].albums, vec!["Iceland", "Best of"]);
|
||||
assert!(planned[1].albums.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_name_different_size_stays_two_photos() {
|
||||
let scan = TakeoutScan {
|
||||
photos: vec![
|
||||
photo("IMG_1.jpg", 100, Bucket::Timeline),
|
||||
photo("IMG_1.jpg", 101, Bucket::Album("Iceland".into())),
|
||||
],
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
assert_eq!(plan(&scan).len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn metadata_from_any_copy_survives_the_fold() {
|
||||
// Only the album copy carries a sidecar; the fold must not lose it.
|
||||
let mut bare = photo("IMG_1.jpg", 100, Bucket::Timeline);
|
||||
bare.capture_time = None;
|
||||
let mut rich = photo("IMG_1.jpg", 100, Bucket::Album("Iceland".into()));
|
||||
rich.capture_time = Some(1_560_000_000);
|
||||
rich.favorite = true;
|
||||
|
||||
let planned = plan(&TakeoutScan {
|
||||
photos: vec![bare, rich],
|
||||
..Default::default()
|
||||
});
|
||||
|
||||
assert_eq!(planned[0].photo.capture_time, Some(1_560_000_000));
|
||||
assert!(planned[0].photo.favorite);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sha1_matches_the_known_digest_of_abc() {
|
||||
let mut reader: &[u8] = b"abc";
|
||||
assert_eq!(
|
||||
sha1_of(&mut reader).unwrap(),
|
||||
"a9993e364706816aba3e25717850c26c9cd0d89d"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -19,7 +19,7 @@ use parking_lot::Mutex;
|
||||
use std::collections::HashMap;
|
||||
use std::io::{Read, Write};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||
use std::time::Instant;
|
||||
|
||||
use pdfs_core::control::{JobItem, TransferDirection, TransferItem};
|
||||
@@ -272,20 +272,50 @@ impl<R: Read> Read for OwnedCountingReader<R> {
|
||||
}
|
||||
}
|
||||
|
||||
/// A [`Read`] that tallies bytes read through a [`TransferGuard`] (upload path).
|
||||
/// A [`Read`] that tallies bytes read through a [`TransferGuard`] (upload path),
|
||||
/// and can be told to stop.
|
||||
pub struct CountingReader<'a, R> {
|
||||
inner: R,
|
||||
guard: &'a TransferGuard,
|
||||
cancel: Option<Arc<AtomicBool>>,
|
||||
}
|
||||
|
||||
impl<'a, R: Read> CountingReader<'a, R> {
|
||||
pub fn new(inner: R, guard: &'a TransferGuard) -> Self {
|
||||
Self { inner, guard }
|
||||
Self {
|
||||
inner,
|
||||
guard,
|
||||
cancel: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Abandon the upload as soon as the SDK asks for its next bytes when
|
||||
/// `cancel` is set.
|
||||
///
|
||||
/// This is the only cancellation point an upload has: the SDK takes a plain
|
||||
/// blocking [`Read`] and pulls from it a block at a time, so refusing to
|
||||
/// hand over the next block is what stops it. Used when a newer revision of
|
||||
/// the same file is queued — the bytes on the wire have already been
|
||||
/// superseded, and finishing them only spends uplink on a revision the next
|
||||
/// op replaces.
|
||||
pub fn with_cancel(mut self, cancel: Arc<AtomicBool>) -> Self {
|
||||
self.cancel = Some(cancel);
|
||||
self
|
||||
}
|
||||
}
|
||||
|
||||
impl<R: Read> Read for CountingReader<'_, R> {
|
||||
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
|
||||
if let Some(cancel) = &self.cancel
|
||||
&& cancel.load(Ordering::Relaxed)
|
||||
{
|
||||
// Not `Ok(0)`: a short read is how this reader says "end of file",
|
||||
// and the SDK would seal a truncated revision from it.
|
||||
return Err(std::io::Error::new(
|
||||
std::io::ErrorKind::Interrupted,
|
||||
"upload superseded by a newer write",
|
||||
));
|
||||
}
|
||||
let n = self.inner.read(buf)?;
|
||||
self.guard.add(n as u64);
|
||||
Ok(n)
|
||||
|
||||
@@ -171,7 +171,7 @@ impl Core {
|
||||
stats.bytes += size;
|
||||
match self.fetch_node(&uid) {
|
||||
Ok(node) => {
|
||||
let mut st = self.state.lock();
|
||||
let mut st = self.state();
|
||||
let ino = st.intern(parent_ino, node);
|
||||
if let Some(kids) = st.children.get_mut(&parent_ino)
|
||||
&& !kids.contains(&ino)
|
||||
@@ -216,7 +216,7 @@ impl Core {
|
||||
self.ensure_children(pino)
|
||||
.map_err(|e| self.errno_error(e, "enumerate"))?;
|
||||
let existing = {
|
||||
let st = self.state.lock();
|
||||
let st = self.state();
|
||||
st.children.get(&pino).and_then(|kids| {
|
||||
kids.iter().find_map(|ino| {
|
||||
st.entries
|
||||
@@ -243,7 +243,7 @@ impl Core {
|
||||
let node = self
|
||||
.fetch_node(&new_uid)
|
||||
.map_err(|e| self.errno_error(e, "fetch node"))?;
|
||||
let mut st = self.state.lock();
|
||||
let mut st = self.state();
|
||||
let ino = st.intern(pino, node);
|
||||
if let Some(kids) = st.children.get_mut(&pino)
|
||||
&& !kids.contains(&ino)
|
||||
|
||||
@@ -51,6 +51,8 @@ pub(crate) fn virtual_node(parent_uid: NodeUid, name: String, timestamp: i64) ->
|
||||
photo: None,
|
||||
album: None,
|
||||
verification: Default::default(),
|
||||
direct_role: None,
|
||||
share_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -5,6 +5,9 @@ edition.workspace = true
|
||||
license.workspace = true
|
||||
authors.workspace = true
|
||||
rust-version.workspace = true
|
||||
repository.workspace = true
|
||||
homepage.workspace = true
|
||||
description = "GTK4 desktop app, tray, and search launcher for the Proton Drive Linux client"
|
||||
|
||||
[[bin]]
|
||||
name = "pdfs-tray"
|
||||
@@ -24,10 +27,11 @@ tracing.workspace = true
|
||||
tracing-subscriber.workspace = true
|
||||
anyhow.workspace = true
|
||||
tokio.workspace = true
|
||||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
ksni = "0.2"
|
||||
gtk4 = { version = "0.9", features = ["v4_12"] }
|
||||
adw = { package = "libadwaita", version = "0.7", features = ["v1_5"] }
|
||||
ksni = { version = "0.3", features = ["blocking"] }
|
||||
gtk4 = { version = "0.11", features = ["v4_12"] }
|
||||
adw = { package = "libadwaita", version = "0.9", features = ["v1_5"] }
|
||||
async-channel = "2"
|
||||
exif = { version = "0.6", package = "kamadak-exif" }
|
||||
webkit6 = "0.4"
|
||||
webkit6 = "0.6"
|
||||
@@ -22,6 +22,9 @@ fn main() {
|
||||
}
|
||||
|
||||
println!("cargo:rerun-if-changed=resources/pdfs.gresource.xml");
|
||||
println!(
|
||||
"cargo:rerun-if-changed=resources/icons/scalable/actions/pdfs-build-thumbnails-symbolic.svg"
|
||||
);
|
||||
println!(
|
||||
"cargo:rerun-if-changed=resources/icons/scalable/actions/pdfs-folder-upload-symbolic.svg"
|
||||
);
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" viewBox="0 0 16 16">
|
||||
<!-- Generate-thumbnails action: a picture frame with a sparkle. A single
|
||||
symbolic fill lets GTK recolour it with the surrounding control. -->
|
||||
<path fill="#222222" d="M 2,3 C 1.45,3 1,3.45 1,4 V 14 C 1,14.55 1.45,15 2,15 H 14 C 14.55,15 15,14.55 15,14 V 7 H 13 V 13 H 3 V 5 H 9 V 3 Z M 5,6 C 4.45,6 4,6.45 4,7 C 4,7.55 4.45,8 5,8 C 5.55,8 6,7.55 6,7 C 6,6.45 5.55,6 5,6 Z M 10,7 L 7.5,10 L 6,9 L 4,11.5 V 12 H 12 V 9 Z M 13,0 L 12.4,2.4 L 10,3 L 12.4,3.6 L 13,6 L 13.6,3.6 L 16,3 L 13.6,2.4 Z"/>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 598 B |
@@ -4,6 +4,7 @@
|
||||
icon-theme resource path at startup, so they resolve by name (and recolour
|
||||
with the theme) exactly like a stock symbolic icon. -->
|
||||
<gresource prefix="/de/nils/protondrivelinux/icons">
|
||||
<file preprocess="xml-stripblanks">scalable/actions/pdfs-build-thumbnails-symbolic.svg</file>
|
||||
<file preprocess="xml-stripblanks">scalable/actions/pdfs-folder-upload-symbolic.svg</file>
|
||||
</gresource>
|
||||
</gresources>
|
||||
+154
-17
@@ -14,15 +14,17 @@ use pages::photos::*;
|
||||
use pages::shared::*;
|
||||
use pages::shared_by_me::*;
|
||||
use pages::status::*;
|
||||
use pages::takeout::*;
|
||||
use pages::trash::*;
|
||||
use pages::verify::*;
|
||||
use widgets::details::*;
|
||||
use widgets::share_dialog::*;
|
||||
use widgets::thumbnails::*;
|
||||
use widgets::versions_dialog::*;
|
||||
|
||||
use std::cell::{Cell, RefCell};
|
||||
|
||||
use std::collections::{HashMap, HashSet, VecDeque};
|
||||
use std::collections::{BTreeMap, HashMap, HashSet, VecDeque};
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
@@ -46,10 +48,10 @@ use pdfs_core::config::AppDirs;
|
||||
|
||||
use pdfs_core::control::{
|
||||
ActivityEntry, ActivityKind, AlbumInfo, BookmarkInfo, DeviceInfo, DirEntry, ErrorKind,
|
||||
InvitationInfo, JobItem, PhotoItem, PhotoKind, PublicLinkInfo, RefreshScope, Request, Response,
|
||||
RestorableFolder, RestoreItem, SearchHit, ShareEntry, ShareEntryKind, SharedItem,
|
||||
SyncFolderInfo, SyncPhase, SyncProgress, TransferDirection, TransferItem, pending_summary,
|
||||
send,
|
||||
ImportSummary, InvitationInfo, JobItem, PhotoItem, PhotoKind, PublicLinkInfo, RefreshScope,
|
||||
Request, Response, RestorableFolder, RestoreItem, SearchHit, ShareEntry, ShareEntryKind,
|
||||
SharedItem, SyncFolderInfo, SyncPhase, SyncProgress, ThumbnailBuildStatus, TransferDirection,
|
||||
TransferItem, pending_summary, send,
|
||||
};
|
||||
|
||||
use pdfs_core::mounts::{MountAccess, MountKind, MountMode, MountSpec};
|
||||
@@ -106,6 +108,10 @@ struct Ui {
|
||||
/// The sidebar/content split. Collapsed while signed out, so the login page
|
||||
/// owns the whole window and no destination is reachable without a session.
|
||||
nav: adw::NavigationSplitView,
|
||||
/// Shared thumbnails for ordinary image files outside the Photos gallery.
|
||||
/// One cache and request queue serves Files, search, Shared and Trash, so
|
||||
/// the same image is downloaded and decoded only once.
|
||||
pub(crate) file_thumbs: FileThumbnailState,
|
||||
|
||||
// Per-page state. Each page module owns its own struct; `Ui` keeps only
|
||||
// what more than one page genuinely shares.
|
||||
@@ -120,6 +126,7 @@ struct Ui {
|
||||
pub(crate) devices: DevicesState,
|
||||
pub(crate) locations: LocationsState,
|
||||
pub(crate) activity: ActivityState,
|
||||
pub(crate) takeout: TakeoutState,
|
||||
}
|
||||
|
||||
impl Ui {
|
||||
@@ -205,10 +212,15 @@ fn load_proton_theme() {
|
||||
.file-grid {{ padding: 6px; }}\n\
|
||||
.file-tile {{ padding: 8px; border-radius: 10px; }}\n\
|
||||
.file-tile:hover {{ background: alpha({PROTON_PURPLE}, 0.10); }}\n\
|
||||
.file-thumbnail {{ border-radius: 7px; }}\n\
|
||||
.file-badge {{ -gtk-icon-shadow: 0 1px 2px rgba(0, 0, 0, 0.5); }}\n\
|
||||
.badge-pinned {{ color: #f5c211; }}\n\
|
||||
.badge-cached {{ color: #2ec27e; }}\n\
|
||||
.badge-cloud {{ color: #9aa0a6; }}\n\
|
||||
.browser-statusbar {{ background-color: alpha(currentColor, 0.025); }}\n\
|
||||
scale.browser-status-meter trough, progressbar.browser-status-meter trough {{ min-width: 104px; }}\n\
|
||||
scale.browser-status-meter trough {{ min-height: 6px; }}\n\
|
||||
progressbar.browser-status-meter trough, progressbar.browser-status-meter progress {{ min-height: 6px; }}\n\
|
||||
.photo-viewer-window {{ background-color: #111014; }}\n\
|
||||
.viewer-top-bar {{ background: linear-gradient(to bottom, rgba(0, 0, 0, 0.75), rgba(0, 0, 0, 0)); padding: 10px 16px 28px 20px; color: white; }}\n\
|
||||
.viewer-title {{ font-weight: 700; font-size: 1.05rem; text-shadow: 0 1px 3px rgba(0, 0, 0, 0.9); color: white; }}\n\
|
||||
@@ -246,7 +258,10 @@ fn load_proton_theme() {
|
||||
.navigation-sidebar row {{ border-radius: 8px; margin: 2px 6px; }}\n\
|
||||
.navigation-sidebar row:selected {{ background: alpha({PROTON_PURPLE}, 0.16); color: {PROTON_PURPLE}; font-weight: 600; }}\n\
|
||||
.navigation-sidebar row:selected image {{ color: {PROTON_PURPLE}; }}\n\
|
||||
.file-tile:selected, .file-tile:hover:selected {{ background: alpha({PROTON_PURPLE}, 0.20); }}\n"
|
||||
.file-tile:selected, .file-tile:hover:selected {{ background: alpha({PROTON_PURPLE}, 0.20); }}\n\
|
||||
.bulk-bar {{ padding: 6px 8px; border-radius: 12px; background: alpha({PROTON_PURPLE}, 0.12); }}\n\
|
||||
.dropzone {{ border: 2px dashed alpha(currentColor, 0.25); border-radius: 16px; padding: 28px 18px; transition: background 160ms ease, border-color 160ms ease; }}\n\
|
||||
.dropzone-active {{ border-color: {PROTON_PURPLE}; background: alpha({PROTON_PURPLE}, 0.10); }}\n"
|
||||
);
|
||||
let provider = gtk4::CssProvider::new();
|
||||
provider.load_from_string(&css);
|
||||
@@ -285,6 +300,7 @@ fn build_window(app: &adw::Application) {
|
||||
let (locations_page, locations_widgets) = build_locations_page();
|
||||
let (activity_page, activity_widgets) = build_activity_page();
|
||||
let (trash_page, trash_widgets) = build_trash_page();
|
||||
let (takeout_page, takeout_widgets) = build_takeout_page();
|
||||
stack.add_named(&login_page, Some("login"));
|
||||
stack.add_named(&main_page, Some("main"));
|
||||
stack.add_named(&browser_page, Some("browser"));
|
||||
@@ -295,6 +311,7 @@ fn build_window(app: &adw::Application) {
|
||||
stack.add_named(&locations_page, Some("locations"));
|
||||
stack.add_named(&activity_page, Some("activity"));
|
||||
stack.add_named(&trash_page, Some("trash"));
|
||||
stack.add_named(&takeout_page, Some("takeout"));
|
||||
|
||||
// Sidebar: the signed-in destinations. Selecting a row swaps the page stack;
|
||||
// `sync_sidebar` pushes the other way when navigation happens elsewhere (e.g.
|
||||
@@ -338,6 +355,7 @@ fn build_window(app: &adw::Application) {
|
||||
mounted: RefCell::new(false),
|
||||
sidebar: sidebar_list.clone(),
|
||||
nav: split.clone(),
|
||||
file_thumbs: FileThumbnailState::new(),
|
||||
login: LoginState {
|
||||
email: login_widgets.0,
|
||||
password: login_widgets.1,
|
||||
@@ -362,6 +380,8 @@ fn build_window(app: &adw::Application) {
|
||||
budget_row: main_widgets.budget_row.clone(),
|
||||
mountpoint_row: main_widgets.mountpoint_row.clone(),
|
||||
settings_suppress: Cell::new(false),
|
||||
budget_source: RefCell::new(None),
|
||||
pins_expanded: Cell::new(false),
|
||||
pins_group: main_widgets.pins_group.clone(),
|
||||
pin_rows: RefCell::new(Vec::new()),
|
||||
pins_state: RefCell::new(None),
|
||||
@@ -381,7 +401,29 @@ fn build_window(app: &adw::Application) {
|
||||
new_folder: browser_widgets.new_folder.clone(),
|
||||
upload: browser_widgets.upload.clone(),
|
||||
upload_folder: browser_widgets.upload_folder.clone(),
|
||||
build_thumbnails: browser_widgets.build_thumbnails.clone(),
|
||||
thumbnail_build_row: browser_widgets.thumbnail_build_row.clone(),
|
||||
thumbnail_progress: browser_widgets.thumbnail_progress.clone(),
|
||||
thumbnail_status: browser_widgets.thumbnail_status.clone(),
|
||||
thumbnail_poll: RefCell::new(None),
|
||||
thumbnail_build_running: Cell::new(false),
|
||||
thumbnail_cancel_pending: Cell::new(false),
|
||||
search_source: RefCell::new(None),
|
||||
load_generation: Cell::new(0),
|
||||
views: browser_widgets.views.clone(),
|
||||
bulk: browser_widgets.bulk.clone(),
|
||||
bulk_label: browser_widgets.bulk_label.clone(),
|
||||
bulk_trash: browser_widgets.bulk_trash.clone(),
|
||||
bulk_pin: browser_widgets.bulk_pin.clone(),
|
||||
bulk_unpin: browser_widgets.bulk_unpin.clone(),
|
||||
empty_actions: browser_widgets.empty_actions.clone(),
|
||||
summary: browser_widgets.summary.clone(),
|
||||
zoom: browser_widgets.zoom.clone(),
|
||||
grid_thumbnail_size: Cell::new(GRID_THUMB_DEFAULT),
|
||||
grid_tiles: RefCell::new(Vec::new()),
|
||||
quota_box: browser_widgets.quota_box.clone(),
|
||||
quota: browser_widgets.quota.clone(),
|
||||
quota_text: browser_widgets.quota_text.clone(),
|
||||
},
|
||||
details: DetailsState {
|
||||
details: browser_widgets.details,
|
||||
@@ -407,6 +449,8 @@ fn build_window(app: &adw::Application) {
|
||||
retry: gallery_widgets.retry.clone(),
|
||||
more: gallery_widgets.more.clone(),
|
||||
upload: gallery_widgets.upload.clone(),
|
||||
import: gallery_widgets.import.clone(),
|
||||
empty_actions: gallery_widgets.empty_actions.clone(),
|
||||
title: gallery_widgets.title.clone(),
|
||||
subtitle: gallery_widgets.subtitle.clone(),
|
||||
albums: gallery_widgets.albums.clone(),
|
||||
@@ -440,7 +484,8 @@ fn build_window(app: &adw::Application) {
|
||||
decode_idle: Cell::new(false),
|
||||
thumb_source: RefCell::new(None),
|
||||
relayout_source: RefCell::new(None),
|
||||
bound: RefCell::new(HashMap::new()),
|
||||
bound: RefCell::new(BTreeMap::new()),
|
||||
list: gallery_widgets.list.clone(),
|
||||
},
|
||||
shared: SharedState {
|
||||
content: shared_widgets.content.clone(),
|
||||
@@ -494,6 +539,24 @@ fn build_window(app: &adw::Application) {
|
||||
inflight: Cell::new(false),
|
||||
key: RefCell::new(None),
|
||||
},
|
||||
takeout: TakeoutState {
|
||||
archives: RefCell::new(Vec::new()),
|
||||
list_group: takeout_widgets.list_group.clone(),
|
||||
rows: RefCell::new(Vec::new()),
|
||||
dropzone: takeout_widgets.dropzone.clone(),
|
||||
scan_button: takeout_widgets.scan_button.clone(),
|
||||
import_button: takeout_widgets.import_button.clone(),
|
||||
cancel_button: takeout_widgets.cancel_button.clone(),
|
||||
clear_button: takeout_widgets.clear_button.clone(),
|
||||
progress_group: takeout_widgets.progress_group.clone(),
|
||||
progress_label: takeout_widgets.progress_label.clone(),
|
||||
progress_bar: takeout_widgets.progress_bar.clone(),
|
||||
summary_group: takeout_widgets.summary_group.clone(),
|
||||
summary_rows: RefCell::new(Vec::new()),
|
||||
inflight: Cell::new(false),
|
||||
running: Cell::new(false),
|
||||
dry_run: Cell::new(false),
|
||||
},
|
||||
});
|
||||
wire_login(&ui);
|
||||
wire_logout(&ui, &main_widgets.logout_button);
|
||||
@@ -501,18 +564,31 @@ fn build_window(app: &adw::Application) {
|
||||
&ui,
|
||||
&main_widgets.purge_button,
|
||||
&main_widgets.mountpoint_button,
|
||||
&main_widgets.import_row,
|
||||
);
|
||||
wire_sidebar(&ui);
|
||||
wire_browser(&ui, &browser_widgets.grid, &browser_widgets.column_view);
|
||||
wire_bulk(
|
||||
&ui,
|
||||
&browser_widgets.bulk_clear,
|
||||
&browser_widgets.empty_upload,
|
||||
&browser_widgets.empty_new_folder,
|
||||
);
|
||||
wire_browser_actions(
|
||||
&ui,
|
||||
&browser_widgets.new_folder,
|
||||
&browser_widgets.upload,
|
||||
&browser_widgets.upload_folder,
|
||||
&browser_widgets.build_thumbnails,
|
||||
);
|
||||
wire_details(&ui);
|
||||
wire_search(&ui);
|
||||
wire_gallery(&ui, &gallery_widgets.list, &gallery_widgets.scroll);
|
||||
wire_gallery_empty(
|
||||
&ui,
|
||||
&gallery_widgets.empty_upload,
|
||||
&gallery_widgets.empty_import,
|
||||
);
|
||||
wire_albums(&ui);
|
||||
wire_trash(&ui, &trash_widgets.list, &trash_widgets.empty);
|
||||
wire_shared(&ui, &shared_widgets.retry, &shared_widgets.add_bookmark);
|
||||
@@ -520,6 +596,7 @@ fn build_window(app: &adw::Application) {
|
||||
wire_devices(&ui, &devices_widgets.retry, &devices_widgets.restore);
|
||||
wire_locations(&ui, &locations_widgets.retry, &locations_widgets.add_folder);
|
||||
wire_activity(&ui, &activity_widgets.retry);
|
||||
wire_takeout(&ui, &takeout_widgets);
|
||||
wire_refresh(
|
||||
&ui,
|
||||
&[
|
||||
@@ -539,6 +616,10 @@ fn build_window(app: &adw::Application) {
|
||||
// network round-trip only happens on demand rather than on every refresh.
|
||||
let ui_nav = ui.clone();
|
||||
stack.connect_visible_child_name_notify(move |st| {
|
||||
// Rows from the page being left must not keep full-size image downloads
|
||||
// alive in the daemon. The page being entered establishes a fresh
|
||||
// thumbnail generation as it paints.
|
||||
cancel_file_thumbnails(&ui_nav);
|
||||
sync_sidebar(&ui_nav);
|
||||
match st.visible_child_name().as_deref() {
|
||||
Some("browser") => load_browser(&ui_nav),
|
||||
@@ -559,6 +640,9 @@ fn build_window(app: &adw::Application) {
|
||||
// every visit to stay live.
|
||||
Some("activity") => load_activity(&ui_nav),
|
||||
Some("trash") => load_trash(&ui_nav),
|
||||
// The import runs in the daemon and outlives this page, so arriving
|
||||
// here asks straight away whether one is in flight.
|
||||
Some("takeout") => refresh_takeout(&ui_nav),
|
||||
_ => {}
|
||||
}
|
||||
});
|
||||
@@ -644,6 +728,17 @@ fn build_sidebar() -> (adw::NavigationPage, gtk4::ListBox) {
|
||||
(page, list)
|
||||
}
|
||||
|
||||
/// The sidebar destination a page belongs under. Most pages are their own
|
||||
/// destination; a sub-page reached from one (Import, off Settings) has no row of
|
||||
/// its own and answers with its parent, so the sidebar highlights where the user
|
||||
/// came from rather than clearing — which would read as "nowhere".
|
||||
fn destination_of(page: &str) -> &str {
|
||||
match page {
|
||||
"takeout" => "main",
|
||||
other => other,
|
||||
}
|
||||
}
|
||||
|
||||
/// Selecting a sidebar row navigates the page stack. The reverse direction (stack
|
||||
/// → sidebar highlight) is [`sync_sidebar`], so the two can't fight: this handler
|
||||
/// only ever writes the stack.
|
||||
@@ -654,7 +749,11 @@ fn wire_sidebar(ui: &Rc<Ui>) {
|
||||
let Some((page, _, _)) = DESTINATIONS.get(row.index() as usize) else {
|
||||
return;
|
||||
};
|
||||
if ui_row.stack.visible_child_name().as_deref() != Some(*page) {
|
||||
// Compared by *destination*, not by page: while a sub-page is open its
|
||||
// parent's row is the selected one, and re-asserting it here would kick
|
||||
// the user off the sub-page the moment they arrived.
|
||||
let current = ui_row.stack.visible_child_name();
|
||||
if current.as_deref().map(destination_of) != Some(*page) {
|
||||
ui_row.stack.set_visible_child_name(page);
|
||||
}
|
||||
});
|
||||
@@ -755,6 +854,7 @@ fn sync_sidebar(ui: &Rc<Ui>) {
|
||||
let Some(current) = ui.stack.visible_child_name() else {
|
||||
return;
|
||||
};
|
||||
let current = destination_of(¤t);
|
||||
let index = DESTINATIONS
|
||||
.iter()
|
||||
.position(|(page, _, _)| *page == current);
|
||||
@@ -776,6 +876,22 @@ fn toast(ui: &Rc<Ui>, message: &str) {
|
||||
ui.toasts.add_toast(adw::Toast::new(message));
|
||||
}
|
||||
|
||||
/// Show a toast that offers one follow-up action — an Undo, or a way to the page
|
||||
/// the outcome landed on.
|
||||
///
|
||||
/// Given a longer timeout than a plain toast: an action the user has to notice,
|
||||
/// read and reach is not the same ask as a line of confirmation they can ignore.
|
||||
fn toast_action(ui: &Rc<Ui>, message: &str, label: &str, action: impl Fn(&Rc<Ui>) + 'static) {
|
||||
let toast = adw::Toast::builder()
|
||||
.title(message)
|
||||
.button_label(label)
|
||||
.timeout(8)
|
||||
.build();
|
||||
let ui_action = ui.clone();
|
||||
toast.connect_button_clicked(move |_| action(&ui_action));
|
||||
ui.toasts.add_toast(toast);
|
||||
}
|
||||
|
||||
/// Show a toast for a failure. Same surface as [`toast`], but the message is
|
||||
/// prefixed with what was being attempted, since a bare daemon error ("no such
|
||||
/// file") reads as noise without it.
|
||||
@@ -830,7 +946,7 @@ fn toast_failure(ui: &Rc<Ui>, what: &str, message: &str, kind: ErrorKind) {
|
||||
fn build_primary_menu() -> gtk4::MenuButton {
|
||||
let menu = gio::Menu::new();
|
||||
menu.append(Some("Keyboard Shortcuts"), Some("win.shortcuts"));
|
||||
menu.append(Some("About Proton Drive"), Some("win.about"));
|
||||
menu.append(Some("About Proton Drive for Linux"), Some("win.about"));
|
||||
gtk4::MenuButton::builder()
|
||||
.icon_name("open-menu-symbolic")
|
||||
.tooltip_text("Main menu")
|
||||
@@ -849,11 +965,17 @@ fn install_window_actions(window: &adw::ApplicationWindow) {
|
||||
let win = window.clone();
|
||||
about.connect_activate(move |_, _| {
|
||||
let dialog = adw::AboutDialog::builder()
|
||||
.application_name("Proton Drive")
|
||||
.application_icon("folder-remote-symbolic")
|
||||
.application_name("Proton Drive for Linux")
|
||||
.application_icon("io.narl.proton-drive-linux")
|
||||
.version(pdfs_core::config::APP_VERSION)
|
||||
.developer_name("proton-drive-linux")
|
||||
.comments("On-demand Proton Drive sync for Linux.")
|
||||
.developer_name("Nils Pukropp")
|
||||
.website("https://github.com/narrrl/proton-drive-linux")
|
||||
.issue_url("https://github.com/narrrl/proton-drive-linux/issues")
|
||||
.license_type(gtk4::License::MitX11)
|
||||
.comments(
|
||||
"Files-on-demand Proton Drive for the Linux desktop.\n\n\
|
||||
Unofficial client — not affiliated with, endorsed by, or supported by Proton AG.",
|
||||
)
|
||||
.build();
|
||||
dialog.present(Some(&win));
|
||||
});
|
||||
@@ -913,7 +1035,13 @@ fn install_shortcuts(ui: &Rc<Ui>, window: &adw::ApplicationWindow) {
|
||||
// browser-only bindings.
|
||||
gtk4::gdk::Key::F5 => reload_current_page(&ui),
|
||||
gtk4::gdk::Key::r | gtk4::gdk::Key::R if ctrl => reload_current_page(&ui),
|
||||
gtk4::gdk::Key::f | gtk4::gdk::Key::F if ctrl && on_browser => {
|
||||
// Search lives on Files, but the shortcut works from anywhere:
|
||||
// wanting to find a file is not a reason to first have to remember
|
||||
// which page owns the search box.
|
||||
gtk4::gdk::Key::f | gtk4::gdk::Key::F if ctrl => {
|
||||
if !on_browser {
|
||||
ui.stack.set_visible_child_name("browser");
|
||||
}
|
||||
ui.browser.search.grab_focus();
|
||||
}
|
||||
gtk4::gdk::Key::n | gtk4::gdk::Key::N if ctrl && on_browser => prompt_new_folder(&ui),
|
||||
@@ -923,12 +1051,21 @@ fn install_shortcuts(ui: &Rc<Ui>, window: &adw::ApplicationWindow) {
|
||||
prompt_rename(&ui, &entry);
|
||||
}
|
||||
}
|
||||
// Acts on the whole selection: Delete with five files highlighted
|
||||
// means those five, not whichever one the details pane happens to
|
||||
// be describing.
|
||||
gtk4::gdk::Key::Delete if on_browser => {
|
||||
if let Some(entry) = selected_entry(&ui) {
|
||||
prompt_delete(&ui, &entry);
|
||||
let entries = selected_entries(&ui);
|
||||
if !entries.is_empty() {
|
||||
prompt_delete_many(&ui, entries);
|
||||
}
|
||||
}
|
||||
gtk4::gdk::Key::Escape if on_browser && ui.browser.split.shows_sidebar() => {
|
||||
gtk4::gdk::Key::Escape
|
||||
if on_browser
|
||||
&& (ui.browser.split.shows_sidebar() || ui.browser.bulk.reveals_child()) =>
|
||||
{
|
||||
clear_selection(&ui);
|
||||
sync_bulk_bar(&ui);
|
||||
hide_details(&ui);
|
||||
}
|
||||
_ => return glib::Propagation::Proceed,
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -9,5 +9,6 @@ pub(crate) mod photos;
|
||||
pub(crate) mod shared;
|
||||
pub(crate) mod shared_by_me;
|
||||
pub(crate) mod status;
|
||||
pub(crate) mod takeout;
|
||||
pub(crate) mod trash;
|
||||
pub(crate) mod verify;
|
||||
@@ -5,8 +5,12 @@ pub(crate) struct GalleryState {
|
||||
/// Every photo loaded so far, newest first — the order the lightbox's
|
||||
/// prev/next walks. The visible day sections are derived from this.
|
||||
pub(crate) model: gio::ListStore,
|
||||
/// The day sections rendered by the Photos ListView, rebuilt from
|
||||
/// [`Self::model`] by [`repaint_gallery`].
|
||||
/// The Photos ListView's own model: one item per *rendered row* — a day
|
||||
/// heading, or one row of that day's tile grid — rebuilt from
|
||||
/// [`Self::model`] by [`repaint_gallery`]. Rows rather than whole days
|
||||
/// because the ListView only realises what is on screen, and a day holding
|
||||
/// 1,600 photos (a Takeout import lands one) would otherwise be 1,600
|
||||
/// widgets built in a single bind.
|
||||
pub(crate) groups: gio::ListStore,
|
||||
/// Target tile edge in px, retuned by Ctrl+scroll / Ctrl+± (see
|
||||
/// [`zoom_gallery`]). The grid fits as many square tiles of about this size
|
||||
@@ -19,6 +23,12 @@ pub(crate) struct GalleryState {
|
||||
pub(crate) retry: gtk4::Button,
|
||||
pub(crate) more: gtk4::Button,
|
||||
pub(crate) upload: gtk4::Button,
|
||||
/// Opens the Google Photos Takeout import chooser.
|
||||
pub(crate) import: gtk4::Button,
|
||||
/// Upload / Import offered on the *empty timeline* status page, so a fresh
|
||||
/// account is a place to start rather than a dead end. Hidden on every other
|
||||
/// status: a load error is not the moment to offer an upload.
|
||||
pub(crate) empty_actions: gtk4::Box,
|
||||
/// "Photos", or the album's name while one is open.
|
||||
pub(crate) title: gtk4::Label,
|
||||
/// "1,204 photos" under the page title.
|
||||
@@ -104,11 +114,14 @@ pub(crate) struct GalleryState {
|
||||
/// re-flow, replaced on each new trigger so only the last one fires.
|
||||
pub(crate) thumb_source: RefCell<Option<glib::SourceId>>,
|
||||
pub(crate) relayout_source: RefCell<Option<glib::SourceId>>,
|
||||
/// The day sections currently realised by the ListView, by their index in
|
||||
/// [`Self::groups`]. A resize or a zoom step re-flows these in place —
|
||||
/// rebuilding the ListStore instead would reset the scroll position out from
|
||||
/// under the user.
|
||||
pub(crate) bound: RefCell<HashMap<u32, gtk4::Box>>,
|
||||
/// The rows currently realised by the ListView, as row index -> the uid of
|
||||
/// its first photo (absent for a heading row). A resize or a zoom step
|
||||
/// changes how many tiles fit per row, so the row model has to be rebuilt —
|
||||
/// this is what lets the rebuild put the user back where they were.
|
||||
pub(crate) bound: RefCell<BTreeMap<u32, Option<String>>>,
|
||||
/// The ListView itself, so a rebuild can scroll back to the row the user was
|
||||
/// looking at.
|
||||
pub(crate) list: gtk4::ListView,
|
||||
}
|
||||
|
||||
/// How many photos to pull per [`Request::PhotosTimeline`] page.
|
||||
@@ -152,13 +165,71 @@ pub(crate) const RELAYOUT_DEBOUNCE: Duration = Duration::from_millis(80);
|
||||
|
||||
/// One day-section of the photos timeline: a heading plus the photos captured
|
||||
/// that day, in timeline order. Built from the flat [`Ui::gallery_model`] by
|
||||
/// [`group_photos`] and rendered as one [`gtk4::ListView`] row.
|
||||
/// [`group_photos`], then flattened into [`GalleryRow`]s for rendering.
|
||||
pub(crate) struct PhotoGroup {
|
||||
/// "Today", "Yesterday", or e.g. "3 June 2026".
|
||||
pub(crate) heading: String,
|
||||
pub(crate) photos: Vec<PhotoItem>,
|
||||
}
|
||||
|
||||
/// One item of the ListView's model — the unit the gallery virtualises at.
|
||||
///
|
||||
/// A whole day is *not* one item: a Google Photos Takeout drops thousands of
|
||||
/// photos onto a single date, and one ListView item per day means GTK builds
|
||||
/// every one of those tiles in one bind, on the main thread, before the row can
|
||||
/// be shown. Splitting the day into its grid rows keeps a bind to a handful of
|
||||
/// widgets no matter how big the day is.
|
||||
pub(crate) enum GalleryRow {
|
||||
/// A day heading: "Today", "3 June 2026".
|
||||
Heading(String),
|
||||
/// One row of a day's grid, already laid out to the current width and zoom.
|
||||
Tiles {
|
||||
photos: Vec<PhotoItem>,
|
||||
edge: i32,
|
||||
/// True for the last row of its day, which carries the section's bottom
|
||||
/// margin so days stay visually separated.
|
||||
last: bool,
|
||||
},
|
||||
}
|
||||
|
||||
impl GalleryRow {
|
||||
/// The uid of the row's first photo, or `None` for a heading — the anchor a
|
||||
/// relayout scrolls back to.
|
||||
fn anchor(&self) -> Option<String> {
|
||||
match self {
|
||||
GalleryRow::Heading(_) => None,
|
||||
GalleryRow::Tiles { photos, .. } => photos.first().map(|p| p.uid.clone()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether re-rendering `self` would produce exactly what `other` shows —
|
||||
/// the test [`repaint_gallery`] diffs on, so appending a page only touches
|
||||
/// the rows that actually changed.
|
||||
fn same_as(&self, other: &GalleryRow) -> bool {
|
||||
match (self, other) {
|
||||
(GalleryRow::Heading(a), GalleryRow::Heading(b)) => a == b,
|
||||
(
|
||||
GalleryRow::Tiles {
|
||||
photos: a,
|
||||
edge: ae,
|
||||
last: al,
|
||||
},
|
||||
GalleryRow::Tiles {
|
||||
photos: b,
|
||||
edge: be,
|
||||
last: bl,
|
||||
},
|
||||
) => {
|
||||
ae == be
|
||||
&& al == bl
|
||||
&& a.len() == b.len()
|
||||
&& a.iter().zip(b).all(|(x, y)| x.uid == y.uid)
|
||||
}
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The widgets [`build_gallery_page`] hands back to [`build_window`].
|
||||
pub(crate) struct GalleryWidgets {
|
||||
/// Flat, newest-first list of every loaded photo. Backs the lightbox's
|
||||
@@ -177,6 +248,10 @@ pub(crate) struct GalleryWidgets {
|
||||
pub(crate) scroll: gtk4::ScrolledWindow,
|
||||
pub(crate) retry: gtk4::Button,
|
||||
pub(crate) upload: gtk4::Button,
|
||||
pub(crate) import: gtk4::Button,
|
||||
pub(crate) empty_actions: gtk4::Box,
|
||||
pub(crate) empty_upload: gtk4::Button,
|
||||
pub(crate) empty_import: gtk4::Button,
|
||||
pub(crate) refresh: gtk4::Button,
|
||||
/// The Albums grid, its own status page and the stack between them, plus the
|
||||
/// Photos/Albums switcher and the back button out of an album.
|
||||
@@ -227,10 +302,29 @@ pub(crate) fn build_gallery_page() -> (gtk4::Widget, GalleryWidgets) {
|
||||
retry.add_css_class("suggested-action");
|
||||
retry.set_visible(false);
|
||||
|
||||
// The empty-timeline state's two ways forward. `StatusPage` takes one child,
|
||||
// so Retry and these share a box and each is shown only for its own state.
|
||||
let empty_upload = gtk4::Button::builder().label("Upload photos").build();
|
||||
empty_upload.add_css_class("pill");
|
||||
empty_upload.add_css_class("suggested-action");
|
||||
let empty_import = gtk4::Button::builder()
|
||||
.label("Import from Google Photos")
|
||||
.build();
|
||||
empty_import.add_css_class("pill");
|
||||
let empty_actions = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
|
||||
empty_actions.set_halign(gtk4::Align::Center);
|
||||
empty_actions.set_visible(false);
|
||||
empty_actions.append(&empty_upload);
|
||||
empty_actions.append(&empty_import);
|
||||
|
||||
let status_child = gtk4::Box::new(gtk4::Orientation::Vertical, 12);
|
||||
status_child.append(&retry);
|
||||
status_child.append(&empty_actions);
|
||||
|
||||
let status = adw::StatusPage::builder()
|
||||
.icon_name("image-x-generic-symbolic")
|
||||
.vexpand(true)
|
||||
.child(&retry)
|
||||
.child(&status_child)
|
||||
.build();
|
||||
status.add_css_class("compact");
|
||||
|
||||
@@ -288,12 +382,23 @@ pub(crate) fn build_gallery_page() -> (gtk4::Widget, GalleryWidgets) {
|
||||
.build();
|
||||
upload.add_css_class("pill");
|
||||
upload.add_css_class("suggested-action");
|
||||
|
||||
// Importing an export is a rarer, heavier action than adding one photo, so
|
||||
// it sits beside Upload as a plain button rather than a second accent one.
|
||||
let import = gtk4::Button::builder()
|
||||
.icon_name("folder-download-symbolic")
|
||||
.tooltip_text("Import a Google Photos Takeout export")
|
||||
.valign(gtk4::Align::Center)
|
||||
.build();
|
||||
import.add_css_class("flat");
|
||||
import.add_css_class("circular");
|
||||
let refresh = refresh_button();
|
||||
|
||||
let header_box = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
|
||||
header_box.append(&back);
|
||||
header_box.append(&titles);
|
||||
header_box.append(&refresh);
|
||||
header_box.append(&import);
|
||||
header_box.append(&upload);
|
||||
|
||||
// All / Photos / Videos / Raw filter. Linked toggles acting as one segmented
|
||||
@@ -428,6 +533,10 @@ pub(crate) fn build_gallery_page() -> (gtk4::Widget, GalleryWidgets) {
|
||||
scroll,
|
||||
retry,
|
||||
upload,
|
||||
import,
|
||||
empty_actions,
|
||||
empty_upload,
|
||||
empty_import,
|
||||
refresh,
|
||||
tabs,
|
||||
favorites_btn,
|
||||
@@ -554,44 +663,77 @@ pub(crate) fn thousands(n: usize) -> String {
|
||||
/// Wire the gallery: install the section factory, the zoom gestures, the pager
|
||||
/// and the upload button. Activating a thumbnail downloads the photo and opens it
|
||||
/// in the in-app lightbox.
|
||||
/// Wire the empty-timeline buttons. Upload re-fires the header's own Upload
|
||||
/// rather than duplicating its file-chooser, so the two can't drift apart.
|
||||
pub(crate) fn wire_gallery_empty(ui: &Rc<Ui>, upload: >k4::Button, import: >k4::Button) {
|
||||
let ui_upload = ui.clone();
|
||||
upload.connect_clicked(move |_| ui_upload.gallery.upload.emit_clicked());
|
||||
let ui_import = ui.clone();
|
||||
import.connect_clicked(move |_| ui_import.stack.set_visible_child_name("takeout"));
|
||||
}
|
||||
|
||||
pub(crate) fn wire_gallery(ui: &Rc<Ui>, list: >k4::ListView, scroll: >k4::ScrolledWindow) {
|
||||
let factory = gtk4::SignalListItemFactory::new();
|
||||
factory.connect_setup(|_, item| {
|
||||
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
|
||||
let section = gtk4::Box::new(gtk4::Orientation::Vertical, 8);
|
||||
section.set_margin_bottom(16);
|
||||
let heading = gtk4::Label::builder().halign(gtk4::Align::Start).build();
|
||||
heading.add_css_class("heading");
|
||||
heading.add_css_class("gallery-day");
|
||||
section.append(&heading);
|
||||
item.set_child(Some(§ion));
|
||||
// One container for both row kinds: the bind fills it with either a
|
||||
// heading label or a strip of tiles. The alternative — two factories, or
|
||||
// a Stack per row — buys nothing, since a row's children are rebuilt on
|
||||
// bind either way.
|
||||
let row = gtk4::Box::new(gtk4::Orientation::Horizontal, TILE_GAP);
|
||||
row.set_halign(gtk4::Align::Start);
|
||||
item.set_child(Some(&row));
|
||||
item.set_activatable(false);
|
||||
});
|
||||
|
||||
let ui_bind = ui.clone();
|
||||
factory.connect_bind(move |_, item| {
|
||||
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
|
||||
let section = item.child().and_downcast::<gtk4::Box>().unwrap();
|
||||
let row_box = item.child().and_downcast::<gtk4::Box>().unwrap();
|
||||
let obj = item.item().and_downcast::<BoxedAnyObject>().unwrap();
|
||||
let group = obj.borrow::<PhotoGroup>();
|
||||
let row = obj.borrow::<GalleryRow>();
|
||||
|
||||
let heading = section.first_child().and_downcast::<gtk4::Label>().unwrap();
|
||||
heading.set_label(&group.heading);
|
||||
|
||||
fill_section(&ui_bind, §ion, &group.photos);
|
||||
// Remember the realised section so a learned aspect ratio or a resize can
|
||||
// re-justify it in place, without rebuilding the ListStore (which would
|
||||
// yank the scroll position back to the top).
|
||||
while let Some(child) = row_box.first_child() {
|
||||
row_box.remove(&child);
|
||||
}
|
||||
match &*row {
|
||||
GalleryRow::Heading(heading) => {
|
||||
row_box.set_margin_top(16);
|
||||
row_box.set_margin_bottom(8);
|
||||
let label = gtk4::Label::builder()
|
||||
.label(heading)
|
||||
.halign(gtk4::Align::Start)
|
||||
.build();
|
||||
label.add_css_class("heading");
|
||||
label.add_css_class("gallery-day");
|
||||
row_box.append(&label);
|
||||
}
|
||||
GalleryRow::Tiles { photos, edge, last } => {
|
||||
row_box.set_margin_top(0);
|
||||
// The last row of a day carries the gap to the next heading.
|
||||
row_box.set_margin_bottom(if *last { 8 } else { TILE_GAP });
|
||||
for photo in photos {
|
||||
row_box.append(&photo_tile(
|
||||
&ui_bind,
|
||||
Tile {
|
||||
photo: photo.clone(),
|
||||
edge: *edge,
|
||||
},
|
||||
));
|
||||
}
|
||||
schedule_thumbs(&ui_bind);
|
||||
}
|
||||
}
|
||||
ui_bind
|
||||
.gallery
|
||||
.bound
|
||||
.borrow_mut()
|
||||
.insert(item.position(), section);
|
||||
.insert(item.position(), row.anchor());
|
||||
});
|
||||
|
||||
// ListView recycles section widgets, so a scrolled-away day must give up its
|
||||
// claim on the widgets — otherwise a thumbnail landing late would paint into
|
||||
// a tile that now shows a different day.
|
||||
// ListView recycles row widgets, so a scrolled-away row must give up its
|
||||
// claim on them — otherwise a thumbnail landing late would paint into a tile
|
||||
// that now shows a different photo.
|
||||
let ui_unbind = ui.clone();
|
||||
factory.connect_unbind(move |_, item| {
|
||||
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
|
||||
@@ -600,10 +742,11 @@ pub(crate) fn wire_gallery(ui: &Rc<Ui>, list: >k4::ListView, scroll: >k4::Sc
|
||||
.bound
|
||||
.borrow_mut()
|
||||
.remove(&item.position());
|
||||
if let Some(obj) = item.item().and_downcast::<BoxedAnyObject>() {
|
||||
let group = obj.borrow::<PhotoGroup>();
|
||||
if let Some(obj) = item.item().and_downcast::<BoxedAnyObject>()
|
||||
&& let GalleryRow::Tiles { photos, .. } = &*obj.borrow::<GalleryRow>()
|
||||
{
|
||||
let mut wanted = ui_unbind.gallery.thumb_wanted.borrow_mut();
|
||||
for photo in &group.photos {
|
||||
for photo in photos {
|
||||
wanted.remove(&photo.uid);
|
||||
}
|
||||
}
|
||||
@@ -729,6 +872,14 @@ pub(crate) fn wire_gallery(ui: &Rc<Ui>, list: >k4::ListView, scroll: >k4::Sc
|
||||
load_gallery(&ui_dates, false);
|
||||
});
|
||||
|
||||
// The import is a staged, hours-long migration, so the button hands over to
|
||||
// the Import page rather than opening a file chooser here: the archives need
|
||||
// reviewing before anything is sent, and the run needs somewhere to report.
|
||||
let ui_import = ui.clone();
|
||||
ui.gallery
|
||||
.import
|
||||
.connect_clicked(move |_| ui_import.stack.set_visible_child_name("takeout"));
|
||||
|
||||
let ui_upload = ui.clone();
|
||||
ui.gallery.upload.connect_clicked(move |_| {
|
||||
let dialog = gtk4::FileDialog::builder()
|
||||
@@ -856,34 +1007,6 @@ pub(crate) fn plan_grid(target: i32, width: i32) -> (usize, i32) {
|
||||
(columns, edge)
|
||||
}
|
||||
|
||||
/// (Re)build a bound day-section's tiles: lay this day's photos out to the
|
||||
/// current content width and hand each tile whatever thumbnail is already in
|
||||
/// memory, queueing the rest. Replaces the section's rows in place, leaving the
|
||||
/// heading — so a re-flow never touches the ListView's model or scroll.
|
||||
pub(crate) fn fill_section(ui: &Rc<Ui>, section: >k4::Box, photos: &[PhotoItem]) {
|
||||
let Some(heading) = section.first_child() else {
|
||||
return;
|
||||
};
|
||||
while let Some(old) = heading.next_sibling() {
|
||||
section.remove(&old);
|
||||
}
|
||||
|
||||
let width = gallery_width(ui);
|
||||
let rows = gtk4::Box::new(gtk4::Orientation::Vertical, TILE_GAP);
|
||||
for row in grid_rows(ui, photos, width) {
|
||||
let row_box = gtk4::Box::new(gtk4::Orientation::Horizontal, TILE_GAP);
|
||||
// A short last row stays left-aligned at the same tile size rather than
|
||||
// stretching: an even grid is the whole point of the square layout.
|
||||
row_box.set_halign(gtk4::Align::Start);
|
||||
for tile in row {
|
||||
row_box.append(&photo_tile(ui, tile));
|
||||
}
|
||||
rows.append(&row_box);
|
||||
}
|
||||
section.append(&rows);
|
||||
schedule_thumbs(ui);
|
||||
}
|
||||
|
||||
/// The width the grid is laid out to: the ListView's own width, less a couple of
|
||||
/// px so a rounding error can't push a row into a horizontal overflow.
|
||||
/// Falls back to a sane guess before the first allocation.
|
||||
@@ -1177,27 +1300,35 @@ pub(crate) fn schedule_relayout(ui: &Rc<Ui>) {
|
||||
*ui.gallery.relayout_source.borrow_mut() = Some(source);
|
||||
}
|
||||
|
||||
/// Rebuild the tiles of the day sections currently on screen, at the current
|
||||
/// width and zoom. Sections that are *not* realised need no work: they lay
|
||||
/// themselves out against the current state when the ListView binds them.
|
||||
/// Re-flow the timeline at the current width and zoom.
|
||||
///
|
||||
/// A different column count means different rows, so unlike the old
|
||||
/// section-per-day model there is nothing to patch in place — the row model is
|
||||
/// rebuilt. What that would cost the user is their scroll position, so the
|
||||
/// topmost realised row's first photo is remembered and scrolled back to.
|
||||
pub(crate) fn relayout_gallery(ui: &Rc<Ui>) {
|
||||
let bound: Vec<(u32, gtk4::Box)> = ui
|
||||
.gallery
|
||||
.bound
|
||||
.borrow()
|
||||
.iter()
|
||||
.map(|(pos, section)| (*pos, section.clone()))
|
||||
.collect();
|
||||
for (pos, section) in bound {
|
||||
let Some(obj) = ui.gallery.groups.item(pos) else {
|
||||
continue;
|
||||
};
|
||||
let Some(boxed) = obj.downcast_ref::<BoxedAnyObject>() else {
|
||||
continue;
|
||||
};
|
||||
let photos = boxed.borrow::<PhotoGroup>().photos.clone();
|
||||
fill_section(ui, §ion, &photos);
|
||||
}
|
||||
let anchor: Option<String> = ui.gallery.bound.borrow().values().flatten().next().cloned();
|
||||
repaint_gallery(ui);
|
||||
let Some(anchor) = anchor else { return };
|
||||
let Some(row) = row_of_photo(&ui.gallery.groups, &anchor) else {
|
||||
return;
|
||||
};
|
||||
ui.gallery
|
||||
.list
|
||||
.scroll_to(row, gtk4::ListScrollFlags::empty(), None);
|
||||
}
|
||||
|
||||
/// Which row of the rendered model holds `uid`, if any.
|
||||
fn row_of_photo(store: &gio::ListStore, uid: &str) -> Option<u32> {
|
||||
(0..store.n_items()).find(|i| {
|
||||
store
|
||||
.item(*i)
|
||||
.and_downcast::<BoxedAnyObject>()
|
||||
.is_some_and(|obj| match &*obj.borrow::<GalleryRow>() {
|
||||
GalleryRow::Heading(_) => false,
|
||||
GalleryRow::Tiles { photos, .. } => photos.iter().any(|p| p.uid == uid),
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
/// Step the tile size by `delta` px and re-flow, clamped to the zoom range.
|
||||
@@ -1215,46 +1346,75 @@ pub(crate) fn set_gallery_tile(ui: &Rc<Ui>, tile: i32) {
|
||||
schedule_relayout(ui);
|
||||
}
|
||||
|
||||
/// Rebuild the day sections from the flat photo model. The timeline arrives
|
||||
/// newest-first, so photos of the same day are already contiguous — one pass
|
||||
/// splits them.
|
||||
/// Rebuild the ListView's row model from the flat photo model: each day becomes
|
||||
/// a heading row followed by its grid rows, laid out to the current width and
|
||||
/// zoom.
|
||||
///
|
||||
/// The groups are diffed into the existing store rather than replacing it: a
|
||||
/// "load more" only really changes the last day (the one the new page continues)
|
||||
/// and appends after it, and clearing the store instead would scroll the user
|
||||
/// back to the top of the timeline at the exact moment they asked for more.
|
||||
/// The rows are diffed into the existing store rather than replacing it: a "load
|
||||
/// more" only really changes the last day (the one the new page continues) and
|
||||
/// appends after it, and clearing the store instead would scroll the user back
|
||||
/// to the top of the timeline at the exact moment they asked for more.
|
||||
pub(crate) fn repaint_gallery(ui: &Rc<Ui>) {
|
||||
let groups = group_photos(&ui.gallery.model);
|
||||
let rows = build_rows(ui);
|
||||
let store = &ui.gallery.groups;
|
||||
|
||||
for (i, group) in groups.iter().enumerate() {
|
||||
for (i, row) in rows.iter().enumerate() {
|
||||
let i = i as u32;
|
||||
let unchanged = store
|
||||
.item(i)
|
||||
.and_downcast::<BoxedAnyObject>()
|
||||
.is_some_and(|old| {
|
||||
let old = old.borrow::<PhotoGroup>();
|
||||
old.heading == group.heading && old.photos.len() == group.photos.len()
|
||||
});
|
||||
.is_some_and(|old| row.same_as(&old.borrow::<GalleryRow>()));
|
||||
if unchanged {
|
||||
continue;
|
||||
}
|
||||
let boxed = BoxedAnyObject::new(PhotoGroup {
|
||||
heading: group.heading.clone(),
|
||||
photos: group.photos.clone(),
|
||||
});
|
||||
let boxed = BoxedAnyObject::new(clone_row(row));
|
||||
if i < store.n_items() {
|
||||
store.splice(i, 1, &[boxed]);
|
||||
} else {
|
||||
store.append(&boxed);
|
||||
}
|
||||
}
|
||||
// Photos only ever get appended, so a shorter model means a fresh load.
|
||||
if store.n_items() > groups.len() as u32 {
|
||||
let len = groups.len() as u32;
|
||||
if store.n_items() > rows.len() as u32 {
|
||||
let len = rows.len() as u32;
|
||||
store.splice(len, store.n_items() - len, &[] as &[BoxedAnyObject]);
|
||||
}
|
||||
|
||||
update_gallery_subtitle(ui);
|
||||
}
|
||||
|
||||
/// Flatten the loaded photos into the rows the ListView renders.
|
||||
fn build_rows(ui: &Rc<Ui>) -> Vec<GalleryRow> {
|
||||
let width = gallery_width(ui);
|
||||
let mut rows = Vec::new();
|
||||
for group in group_photos(&ui.gallery.model) {
|
||||
rows.push(GalleryRow::Heading(group.heading));
|
||||
let grid = grid_rows(ui, &group.photos, width);
|
||||
let last_index = grid.len().saturating_sub(1);
|
||||
for (index, tiles) in grid.into_iter().enumerate() {
|
||||
let edge = tiles.first().map(|t| t.edge).unwrap_or(TILE_DEFAULT);
|
||||
rows.push(GalleryRow::Tiles {
|
||||
photos: tiles.into_iter().map(|t| t.photo).collect(),
|
||||
edge,
|
||||
last: index == last_index,
|
||||
});
|
||||
}
|
||||
}
|
||||
rows
|
||||
}
|
||||
|
||||
fn clone_row(row: &GalleryRow) -> GalleryRow {
|
||||
match row {
|
||||
GalleryRow::Heading(heading) => GalleryRow::Heading(heading.clone()),
|
||||
GalleryRow::Tiles { photos, edge, last } => GalleryRow::Tiles {
|
||||
photos: photos.clone(),
|
||||
edge: *edge,
|
||||
last: *last,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// "1,204 photos" under the page title.
|
||||
fn update_gallery_subtitle(ui: &Rc<Ui>) {
|
||||
let loaded = ui.gallery.model.n_items() as usize;
|
||||
ui.gallery.subtitle.set_visible(loaded > 0);
|
||||
// An album counts what the server says it holds, not how much of it has been
|
||||
@@ -1460,6 +1620,12 @@ pub(crate) fn load_gallery(ui: &Rc<Ui>, append: bool) {
|
||||
)
|
||||
};
|
||||
gallery_status(&ui, "image-x-generic-symbolic", title, description, false);
|
||||
// Only the whole-timeline empty state: an empty *album* is
|
||||
// filled by adding existing photos to it, not by uploading
|
||||
// into it, so the buttons would point the wrong way.
|
||||
ui.gallery
|
||||
.empty_actions
|
||||
.set_visible(ui.gallery.album.borrow().is_none());
|
||||
return;
|
||||
}
|
||||
ui.gallery.content.set_visible_child_name("timeline");
|
||||
@@ -1502,6 +1668,9 @@ pub(crate) fn gallery_status(ui: &Rc<Ui>, icon: &str, title: &str, description:
|
||||
ui.gallery.status.set_title(title);
|
||||
ui.gallery.status.set_description(Some(description));
|
||||
ui.gallery.retry.set_visible(retry);
|
||||
// Only the empty timeline offers a way to fill it; every other status turns
|
||||
// those buttons back off.
|
||||
ui.gallery.empty_actions.set_visible(false);
|
||||
ui.gallery.more.set_visible(false);
|
||||
ui.gallery.content.set_visible_child_name("status");
|
||||
}
|
||||
|
||||
@@ -151,6 +151,7 @@ pub(crate) fn load_shared(ui: &Rc<Ui>) {
|
||||
if ui.shared.inflight.get() {
|
||||
return;
|
||||
}
|
||||
cancel_file_thumbnails(ui);
|
||||
let current = ui.shared.nav.borrow().last().cloned();
|
||||
if let Some((uid, _)) = current {
|
||||
load_shared_folder(ui, uid);
|
||||
@@ -308,11 +309,18 @@ fn shared_entry_row(ui: &Rc<Ui>, entry: &DirEntry) -> adw::ActionRow {
|
||||
if let Some(badge) = role_badge(&entry.role) {
|
||||
row.add_suffix(&badge);
|
||||
}
|
||||
row.add_prefix(>k4::Image::from_icon_name(if entry.is_dir {
|
||||
"folder-symbolic"
|
||||
} else {
|
||||
"text-x-generic-symbolic"
|
||||
}));
|
||||
// The "shared by" in the subtitle is what the invitation claims. When its
|
||||
// signature does not check out against that person's keys, the claim must
|
||||
// not read as fact — someone may be impersonating a contact to get a file
|
||||
// opened.
|
||||
if entry.shared_by_unverified {
|
||||
let warning = gtk4::Image::from_icon_name("dialog-warning-symbolic");
|
||||
warning.add_css_class("warning");
|
||||
warning.set_valign(gtk4::Align::Center);
|
||||
warning.set_tooltip_text(Some(UNVERIFIED_INVITER_TOOLTIP));
|
||||
row.add_suffix(&warning);
|
||||
}
|
||||
row.add_prefix(&file_thumbnail(ui, entry, 40, 24, true));
|
||||
let ui_act = ui.clone();
|
||||
let uid = entry.uid.clone();
|
||||
let name = entry.name.clone();
|
||||
@@ -332,12 +340,29 @@ fn shared_entry_row(ui: &Rc<Ui>, entry: &DirEntry) -> adw::ActionRow {
|
||||
row
|
||||
}
|
||||
|
||||
/// A shared row's subtitle: its size (files only), and where it can be reached
|
||||
/// in the local mount once the synthetic `Shared with me/` directory has interned
|
||||
/// it. An unlisted share has no local path yet, and says nothing rather than
|
||||
/// pointing at a directory that is not there.
|
||||
/// Tooltip on the warning shown next to a share whose invitation did not verify.
|
||||
const UNVERIFIED_INVITER_TOOLTIP: &str = "The invitation's signature doesn't match the sender's keys. \
|
||||
It may not really be from them.";
|
||||
|
||||
/// A shared row's subtitle: who shared it and when (share roots only — that is
|
||||
/// where the invitation lives), its size (files only), and where it can be
|
||||
/// reached in the local mount once the synthetic `Shared with me/` directory has
|
||||
/// interned it. An unlisted share has no local path yet, and says nothing rather
|
||||
/// than pointing at a directory that is not there.
|
||||
pub(crate) fn shared_row_subtitle(entry: &DirEntry) -> String {
|
||||
let mut parts: Vec<String> = Vec::new();
|
||||
if !entry.shared_by.is_empty() {
|
||||
let date = if entry.shared_at > 0 {
|
||||
format_modified(entry.shared_at)
|
||||
} else {
|
||||
String::new()
|
||||
};
|
||||
parts.push(if date.is_empty() {
|
||||
format!("Shared by {}", entry.shared_by)
|
||||
} else {
|
||||
format!("Shared by {} on {date}", entry.shared_by)
|
||||
});
|
||||
}
|
||||
if !entry.is_dir {
|
||||
parts.push(human_bytes(entry.size));
|
||||
}
|
||||
@@ -726,9 +751,25 @@ mod tests {
|
||||
uid: "vol~link".into(),
|
||||
path: path.into(),
|
||||
role: role.into(),
|
||||
shared_by: String::new(),
|
||||
shared_at: 0,
|
||||
shared_by_unverified: false,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_share_root_says_who_shared_it() {
|
||||
let mut root = entry(true, "", "viewer");
|
||||
root.shared_by = "alice@proton.me".into();
|
||||
assert_eq!(shared_row_subtitle(&root), "Shared by alice@proton.me");
|
||||
root.shared_at = 1_700_000_000;
|
||||
let subtitle = shared_row_subtitle(&root);
|
||||
assert!(
|
||||
subtitle.starts_with("Shared by alice@proton.me on "),
|
||||
"{subtitle}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_share_with_no_local_path_says_nothing_about_one() {
|
||||
// Before the synthetic directory has been listed, the daemon cannot
|
||||
|
||||
@@ -113,6 +113,7 @@ pub(crate) fn load_shared_by_me(ui: &Rc<Ui>) {
|
||||
if ui.shared_by_me.inflight.get() {
|
||||
return;
|
||||
}
|
||||
cancel_file_thumbnails(ui);
|
||||
ui.shared_by_me.inflight.set(true);
|
||||
shared_by_me_status(
|
||||
ui,
|
||||
@@ -200,15 +201,12 @@ pub(crate) fn repaint_shared_by_me(ui: &Rc<Ui>, items: &[SharedItem]) {
|
||||
ui.shared_by_me.content.set_visible_child_name("list");
|
||||
let mut rows: Vec<gtk4::Widget> = Vec::new();
|
||||
for item in items {
|
||||
let entry = shared_item_as_entry(item);
|
||||
let row = adw::ActionRow::builder()
|
||||
.title(&item.name)
|
||||
.subtitle(shared_item_summary(item))
|
||||
.build();
|
||||
row.add_prefix(>k4::Image::from_icon_name(if item.is_dir {
|
||||
"folder-symbolic"
|
||||
} else {
|
||||
"text-x-generic-symbolic"
|
||||
}));
|
||||
row.add_prefix(&file_thumbnail(ui, &entry, 40, 24, true));
|
||||
|
||||
// Copy / open the public link, when there is one with a recovered URL.
|
||||
if let Some(url) = item.link.as_ref().and_then(|l| l.url.clone()) {
|
||||
@@ -245,7 +243,6 @@ pub(crate) fn repaint_shared_by_me(ui: &Rc<Ui>, items: &[SharedItem]) {
|
||||
.build();
|
||||
manage.add_css_class("flat");
|
||||
let ui_manage = ui.clone();
|
||||
let entry = shared_item_as_entry(item);
|
||||
manage.connect_clicked(move |_| open_share_dialog(&ui_manage, &entry));
|
||||
row.add_suffix(&manage);
|
||||
|
||||
@@ -289,12 +286,15 @@ pub(crate) fn shared_item_as_entry(item: &SharedItem) -> DirEntry {
|
||||
name: item.name.clone(),
|
||||
is_dir: item.is_dir,
|
||||
size: 0,
|
||||
modified: 0,
|
||||
modified: item.modified,
|
||||
pinned: false,
|
||||
cached: false,
|
||||
uid: item.uid.clone(),
|
||||
path: item.path.clone(),
|
||||
// Shared *by* me: I own it, so there is no role of mine to report.
|
||||
role: String::new(),
|
||||
shared_by: String::new(),
|
||||
shared_at: 0,
|
||||
shared_by_unverified: false,
|
||||
}
|
||||
}
|
||||
@@ -37,7 +37,16 @@ pub(crate) struct StatusState {
|
||||
/// Set while a settings widget is being populated programmatically, so its
|
||||
/// change handler skips the IPC/systemd side effect.
|
||||
pub(crate) settings_suppress: Cell<bool>,
|
||||
/// Pending debounce for the cache-budget editor. Each `+` click is a value
|
||||
/// change, and firing a `SetCacheBudget` (and a toast) per click means a
|
||||
/// stack of toasts and a series of caps the user never asked to apply — so
|
||||
/// only the value they settle on is sent.
|
||||
pub(crate) budget_source: RefCell<Option<glib::SourceId>>,
|
||||
pub(crate) pins_group: adw::PreferencesGroup,
|
||||
/// Whether the pin list is showing every pin or only the first
|
||||
/// [`PINS_COLLAPSED`]. A long pin list would otherwise push everything below
|
||||
/// it — including the Developer group — off the end of the page.
|
||||
pub(crate) pins_expanded: Cell<bool>,
|
||||
/// Rows currently shown under [`Self::pins_group`], retained so a refresh can
|
||||
/// diff against them and only rebuild when the pin set actually changes.
|
||||
pub(crate) pin_rows: RefCell<Vec<PinRow>>,
|
||||
@@ -104,6 +113,8 @@ pub(crate) struct MainWidgets {
|
||||
/// which is where the folder is changed.
|
||||
pub(crate) mountpoint_row: adw::ActionRow,
|
||||
pub(crate) mountpoint_button: gtk4::Button,
|
||||
/// Opens the Google Photos Takeout import page.
|
||||
pub(crate) import_row: adw::ActionRow,
|
||||
}
|
||||
|
||||
/// The main (logged-in) page: a libadwaita settings surface — account header,
|
||||
@@ -233,6 +244,22 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
|
||||
mountpoint_row.add_suffix(&mountpoint_button);
|
||||
system_group.add(&mountpoint_row);
|
||||
|
||||
// Import: a one-off migration rather than a setting, so it is a doorway to
|
||||
// its own page rather than controls inlined here — staging a set of archives
|
||||
// and watching an hours-long upload needs the room.
|
||||
let import_group = adw::PreferencesGroup::builder()
|
||||
.title("Import")
|
||||
.description("Bring a photo library from another service into Proton Photos.")
|
||||
.build();
|
||||
let import_row = adw::ActionRow::builder()
|
||||
.title("Import from Google Photos")
|
||||
.subtitle("Add a Google Takeout export to your timeline.")
|
||||
.activatable(true)
|
||||
.build();
|
||||
import_row.add_prefix(>k4::Image::from_icon_name("folder-download-symbolic"));
|
||||
import_row.add_suffix(>k4::Image::from_icon_name("go-next-symbolic"));
|
||||
import_group.add(&import_row);
|
||||
|
||||
// Pins group: filled in by refresh.
|
||||
let pins_group = adw::PreferencesGroup::builder()
|
||||
.title("Pinned files")
|
||||
@@ -265,6 +292,7 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
|
||||
inner.append(&transfers_group);
|
||||
inner.append(&storage_group);
|
||||
inner.append(&system_group);
|
||||
inner.append(&import_group);
|
||||
inner.append(&pins_group);
|
||||
inner.append(&dev_group);
|
||||
|
||||
@@ -292,10 +320,20 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
|
||||
purge_button,
|
||||
mountpoint_row,
|
||||
mountpoint_button,
|
||||
import_row,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
/// How many pins the Settings list shows before collapsing the rest behind a
|
||||
/// "Show all" row. Enough to recognise the list at a glance; short enough that
|
||||
/// the groups below it stay reachable.
|
||||
pub(crate) const PINS_COLLAPSED: usize = 6;
|
||||
|
||||
/// How long the cache-budget editor waits after the last change before applying
|
||||
/// it, so a run of `+` clicks is one request rather than one per click.
|
||||
pub(crate) const BUDGET_DEBOUNCE: Duration = Duration::from_millis(600);
|
||||
|
||||
/// Bytes per GiB, for the cache-budget editor's unit conversion.
|
||||
pub(crate) const GIB: f64 = 1024.0 * 1024.0 * 1024.0;
|
||||
|
||||
@@ -308,6 +346,7 @@ pub(crate) fn wire_settings(
|
||||
ui: &Rc<Ui>,
|
||||
purge_button: >k4::Button,
|
||||
mountpoint_button: >k4::Button,
|
||||
import_row: &adw::ActionRow,
|
||||
) {
|
||||
let config = ui.dirs.load_config();
|
||||
|
||||
@@ -329,13 +368,22 @@ pub(crate) fn wire_settings(
|
||||
if ui_budget.status.settings_suppress.get() {
|
||||
return;
|
||||
}
|
||||
// Replace any pending apply, so only the value the user stops on is sent.
|
||||
if let Some(src) = ui_budget.status.budget_source.borrow_mut().take() {
|
||||
src.remove();
|
||||
}
|
||||
let bytes = (row.value() * GIB).round() as u64;
|
||||
settings_request(
|
||||
&ui_budget,
|
||||
Request::SetCacheBudget { bytes },
|
||||
"Cache budget updated",
|
||||
"Couldn't set cache budget",
|
||||
);
|
||||
let ui_fire = ui_budget.clone();
|
||||
let src = glib::timeout_add_local_once(BUDGET_DEBOUNCE, move || {
|
||||
ui_fire.status.budget_source.borrow_mut().take();
|
||||
settings_request(
|
||||
&ui_fire,
|
||||
Request::SetCacheBudget { bytes },
|
||||
"Cache budget updated",
|
||||
"Couldn't set cache budget",
|
||||
);
|
||||
});
|
||||
*ui_budget.status.budget_source.borrow_mut() = Some(src);
|
||||
});
|
||||
|
||||
// Purge: confirm, then drop all unpinned cached content via the daemon.
|
||||
@@ -382,6 +430,10 @@ pub(crate) fn wire_settings(
|
||||
// local path; this button is the way there.
|
||||
let ui_mp = ui.clone();
|
||||
mountpoint_button.connect_clicked(move |_| ui_mp.stack.set_visible_child_name("locations"));
|
||||
|
||||
// Import: a doorway, not a setting — everything about it lives on its page.
|
||||
let ui_import = ui.clone();
|
||||
import_row.connect_activated(move |_| ui_import.stack.set_visible_child_name("takeout"));
|
||||
}
|
||||
|
||||
/// Run a settings control-socket round-trip (budget / purge) on a worker thread,
|
||||
@@ -513,8 +565,14 @@ pub(crate) fn refresh(ui: &Rc<Ui>) {
|
||||
refresh_transfers(ui);
|
||||
// Both of these pages show work as it happens, so they follow the tick while
|
||||
// they are on screen. Every other page loads on navigation only.
|
||||
// A Takeout import outlives the page that started it, so it is polled
|
||||
// wherever the user has navigated to — otherwise leaving the page means
|
||||
// never being told it finished.
|
||||
if ui.stack.visible_child_name().as_deref() == Some("takeout") || ui.takeout.running.get() {
|
||||
refresh_takeout(ui);
|
||||
}
|
||||
match ui.stack.visible_child_name().as_deref() {
|
||||
Some("main") => refresh_quota(ui),
|
||||
Some("main" | "browser") => refresh_quota(ui),
|
||||
Some("locations") => refresh_locations(ui),
|
||||
Some("activity") => refresh_activity(ui),
|
||||
_ => {}
|
||||
@@ -522,13 +580,11 @@ pub(crate) fn refresh(ui: &Rc<Ui>) {
|
||||
}
|
||||
|
||||
/// How long a quota reading stays fresh. Account storage barely moves, so the
|
||||
/// Settings tick refetches it only this often rather than every 2s.
|
||||
/// active-page tick refetches it only this often rather than every 2s.
|
||||
const QUOTA_TTL: Duration = Duration::from_secs(60);
|
||||
|
||||
/// Fetch the account quota (if the last reading is stale) and paint the Account
|
||||
/// storage group. Runs only while Settings is on screen. The group stays hidden
|
||||
/// until a reading lands, so an account the API can't report — or a still-starting
|
||||
/// daemon — shows nothing rather than an empty bar. A failed fetch leaves the last
|
||||
/// Fetch the account quota (if the last reading is stale) and paint both the
|
||||
/// Settings storage group and Files status bar. A failed fetch leaves the last
|
||||
/// good reading in place.
|
||||
pub(crate) fn refresh_quota(ui: &Rc<Ui>) {
|
||||
if ui.status.quota_inflight.get() {
|
||||
@@ -551,28 +607,71 @@ pub(crate) fn refresh_quota(ui: &Rc<Ui>) {
|
||||
})) = result
|
||||
{
|
||||
ui.status.quota_checked_at.set(Some(Instant::now()));
|
||||
let used = used_space.max(0) as u64;
|
||||
if max_space > 0 {
|
||||
let total = max_space as u64;
|
||||
let fraction = (used as f64 / total as f64).min(1.0);
|
||||
ui.status.quota_bar.set_fraction(fraction);
|
||||
let pct = (fraction * 100.0).round() as u64;
|
||||
ui.status.quota_label.set_text(&format!(
|
||||
"{} of {} used ({pct}%)",
|
||||
human_bytes(used),
|
||||
human_bytes(total)
|
||||
));
|
||||
} else {
|
||||
ui.status.quota_bar.set_fraction(0.0);
|
||||
ui.status
|
||||
.quota_label
|
||||
.set_text(&format!("{} used", human_bytes(used)));
|
||||
}
|
||||
paint_account_quota(&ui, max_space, used_space);
|
||||
ui.status.quota_group.set_visible(true);
|
||||
} else if ui.status.quota_checked_at.get().is_none() {
|
||||
// Match Dolphin: capacity information does not occupy the bar until
|
||||
// the backing observer has real figures.
|
||||
ui.browser.quota_box.set_visible(false);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
fn paint_account_quota(ui: &Rc<Ui>, max_space: i64, used_space: i64) {
|
||||
let (fraction, text) = quota_display(max_space, used_space);
|
||||
ui.status.quota_bar.set_fraction(fraction);
|
||||
ui.status.quota_label.set_text(&text);
|
||||
if let Some((fraction, free_text, tooltip)) = quota_status_display(max_space, used_space) {
|
||||
ui.browser.quota.set_fraction(fraction);
|
||||
ui.browser.quota.set_tooltip_text(Some(&tooltip));
|
||||
ui.browser.quota_text.set_label(&free_text);
|
||||
ui.browser.quota_text.set_tooltip_text(Some(&tooltip));
|
||||
ui.browser.quota_box.set_visible(true);
|
||||
} else {
|
||||
ui.browser.quota_box.set_visible(false);
|
||||
}
|
||||
}
|
||||
|
||||
fn quota_display(max_space: i64, used_space: i64) -> (f64, String) {
|
||||
let used = used_space.max(0) as u64;
|
||||
if max_space <= 0 {
|
||||
return (0.0, format!("{} used", human_bytes(used)));
|
||||
}
|
||||
let total = max_space as u64;
|
||||
let fraction = (used as f64 / total as f64).clamp(0.0, 1.0);
|
||||
let pct = (fraction * 100.0).round() as u64;
|
||||
(
|
||||
fraction,
|
||||
format!(
|
||||
"{} of {} used ({pct}%)",
|
||||
human_bytes(used),
|
||||
human_bytes(total)
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
/// Dolphin's status bar shows a bare capacity bar followed by “X free”; the
|
||||
/// full free/total/percentage sentence is a tooltip rather than inline bar text.
|
||||
fn quota_status_display(max_space: i64, used_space: i64) -> Option<(f64, String, String)> {
|
||||
if max_space <= 0 {
|
||||
return None;
|
||||
}
|
||||
let total = max_space as u64;
|
||||
let used = (used_space.max(0) as u64).min(total);
|
||||
let free = total.saturating_sub(used);
|
||||
let fraction = used as f64 / total as f64;
|
||||
let pct = (fraction * 100.0).round() as u64;
|
||||
Some((
|
||||
fraction,
|
||||
format!("{} free", human_bytes(free)),
|
||||
format!(
|
||||
"{} free out of {} ({pct}% used)",
|
||||
human_bytes(free),
|
||||
human_bytes(total)
|
||||
),
|
||||
))
|
||||
}
|
||||
|
||||
/// Record the mount state seen by the last status poll: gate every control that
|
||||
/// needs a live daemon, and notify the desktop when the state actually flips.
|
||||
///
|
||||
@@ -584,6 +683,11 @@ pub(crate) fn set_mounted(ui: &Rc<Ui>, mounted: bool) {
|
||||
ui.browser.new_folder.set_sensitive(mounted);
|
||||
ui.browser.upload.set_sensitive(mounted);
|
||||
ui.browser.upload_folder.set_sensitive(mounted);
|
||||
ui.browser.build_thumbnails.set_sensitive(
|
||||
mounted
|
||||
&& (!ui.browser.thumbnail_build_running.get()
|
||||
|| !ui.browser.thumbnail_cancel_pending.get()),
|
||||
);
|
||||
ui.gallery.upload.set_sensitive(mounted);
|
||||
ui.details.details.pin_row.set_sensitive(mounted);
|
||||
ui.details.details.rename_button.set_sensitive(mounted);
|
||||
@@ -653,6 +757,10 @@ pub(crate) fn repaint_transfers(ui: &Rc<Ui>, items: &[TransferItem], jobs: &[Job
|
||||
// count falling to zero is what "the sync finished" looks like from here.
|
||||
// Jobs are deliberately not counted — a bulk upload retires its scan job and
|
||||
// starts its upload job mid-flight, which is not a thing finishing.
|
||||
// The Import page reads its progress off the same snapshot rather than
|
||||
// polling the daemon a second time for numbers already on the wire.
|
||||
takeout_progress(ui, jobs);
|
||||
|
||||
let previous = ui.status.active_transfers.replace(items.len());
|
||||
if items.is_empty() && previous > 0 {
|
||||
let files = if previous == 1 {
|
||||
@@ -808,11 +916,14 @@ pub(crate) fn refresh_status(ui: &Rc<Ui>) {
|
||||
(used as f64 / budget as f64).min(1.0)
|
||||
};
|
||||
ui.status.cache_bar.set_fraction(fraction);
|
||||
ui.status.cache_label.set_text(&format!(
|
||||
"{} of {} used",
|
||||
human_bytes(used),
|
||||
human_bytes(budget)
|
||||
));
|
||||
// A 0 budget means *unlimited*, not a zero-byte cap — "of 0 B
|
||||
// used" reads as a broken read-out, and there is no fraction to
|
||||
// draw against no limit.
|
||||
ui.status.cache_label.set_text(&if budget == 0 {
|
||||
format!("{} cached — no limit set", human_bytes(used))
|
||||
} else {
|
||||
format!("{} of {} used", human_bytes(used), human_bytes(budget))
|
||||
});
|
||||
repaint_pins(&ui, &pins, true);
|
||||
}
|
||||
// Daemon unreachable (still starting, or down): report not-mounted and
|
||||
@@ -865,7 +976,15 @@ pub(crate) fn repaint_pins(ui: &Rc<Ui>, pins: &[pdfs_core::cache::Pin], mounted:
|
||||
return;
|
||||
}
|
||||
|
||||
for pin in pins {
|
||||
// Only the first page of pins is rendered; the rest sit behind the row
|
||||
// below, so a device with hundreds of pins doesn't bury the groups after it.
|
||||
let expanded = ui.status.pins_expanded.get();
|
||||
let shown = if expanded {
|
||||
pins.len()
|
||||
} else {
|
||||
pins.len().min(PINS_COLLAPSED)
|
||||
};
|
||||
for pin in &pins[..shown] {
|
||||
let name = Path::new(&pin.path)
|
||||
.file_name()
|
||||
.and_then(|n| n.to_str())
|
||||
@@ -903,4 +1022,70 @@ pub(crate) fn repaint_pins(ui: &Rc<Ui>, pins: &[pdfs_core::cache::Pin], mounted:
|
||||
unpin: Some(unpin),
|
||||
});
|
||||
}
|
||||
|
||||
if pins.len() > PINS_COLLAPSED {
|
||||
let row = adw::ActionRow::builder()
|
||||
.title(if expanded {
|
||||
"Show fewer".to_string()
|
||||
} else {
|
||||
format!("Show all {} pinned files", pins.len())
|
||||
})
|
||||
.activatable(true)
|
||||
.build();
|
||||
row.add_suffix(>k4::Image::from_icon_name(if expanded {
|
||||
"go-up-symbolic"
|
||||
} else {
|
||||
"go-down-symbolic"
|
||||
}));
|
||||
let ui_more = ui.clone();
|
||||
row.connect_activated(move |_| {
|
||||
ui_more
|
||||
.status
|
||||
.pins_expanded
|
||||
.set(!ui_more.status.pins_expanded.get());
|
||||
// The diff guard compares pin *paths*; the same paths render
|
||||
// differently now, so the baseline has to be dropped for the next
|
||||
// repaint to actually rebuild.
|
||||
*ui_more.status.pins_state.borrow_mut() = None;
|
||||
refresh(&ui_more);
|
||||
});
|
||||
ui.status.pins_group.add(&row);
|
||||
ui.status
|
||||
.pin_rows
|
||||
.borrow_mut()
|
||||
.push(PinRow { row, unpin: None });
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{quota_display, quota_status_display};
|
||||
|
||||
#[test]
|
||||
fn quota_display_reports_used_total_and_percentage() {
|
||||
let gib = 1024_i64.pow(3);
|
||||
let (fraction, text) = quota_display(4 * gib, gib);
|
||||
assert!((fraction - 0.25).abs() < f64::EPSILON);
|
||||
assert_eq!(text, "1.0 GiB of 4.0 GiB used (25%)");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn quota_display_clamps_bad_api_values() {
|
||||
assert_eq!(quota_display(0, -1), (0.0, "0 B used".to_string()));
|
||||
assert_eq!(quota_display(100, 150).0, 1.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn quota_status_display_matches_dolphin_wording() {
|
||||
let gib = 1024_i64.pow(3);
|
||||
assert_eq!(
|
||||
quota_status_display(4 * gib, gib),
|
||||
Some((
|
||||
0.25,
|
||||
"3.0 GiB free".to_string(),
|
||||
"3.0 GiB free out of 4.0 GiB (25% used)".to_string()
|
||||
))
|
||||
);
|
||||
assert_eq!(quota_status_display(0, 0), None);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,779 @@
|
||||
//! The Import page: bringing a Google Photos **Takeout** export into Proton
|
||||
//! Photos.
|
||||
//!
|
||||
//! Reached from Settings → *Import from Google Photos* (and from the Gallery's
|
||||
//! import button), because it is a one-off migration rather than something the
|
||||
//! photo timeline does every day. It gets a page of its own instead of a dialog
|
||||
//! for two reasons: an export is a *set* of archives that has to be staged and
|
||||
//! checked before anything is sent, and the import that follows runs for hours,
|
||||
//! so its progress needs somewhere to live that the user can navigate away from
|
||||
//! and come back to.
|
||||
//!
|
||||
//! Archives arrive either by drag-and-drop onto the page or through the file
|
||||
//! chooser; both funnel into [`add_archives`], which is where the `.zip` filter
|
||||
//! and de-duplication live. Nothing is sent until the user presses Scan or
|
||||
//! Import — staging is local, so a mis-drop costs nothing.
|
||||
//!
|
||||
//! The daemon owns the run itself ([`Request::ImportTakeout`]): this page only
|
||||
//! stages paths, launches, polls [`Request::ImportStatus`] for the final report,
|
||||
//! and reads live progress off the import's job in the transfer snapshot the
|
||||
//! Settings tick already fetches (see [`takeout_progress`]).
|
||||
|
||||
use crate::*;
|
||||
|
||||
/// The title the daemon gives the import's job in [`Request::GetQueueStatus`].
|
||||
/// Matching on it is how this page borrows the progress the Settings tick is
|
||||
/// already polling instead of adding a second poll of its own.
|
||||
const IMPORT_JOB: &str = "Importing Google Photos";
|
||||
|
||||
/// What the archive list says before anything is staged. `repaint_archives`
|
||||
/// swaps in the count and total once there is something to report, and swaps
|
||||
/// this back when the list is emptied again.
|
||||
/// Where the staged set is remembered between runs, under the app's state
|
||||
/// directory. Staging an export means picking dozens of files out of a file
|
||||
/// chooser; losing that to a window close (or a crash mid-import) and having to
|
||||
/// redo it is the kind of small cruelty that makes a migration not get finished.
|
||||
fn staged_path(ui: &Rc<Ui>) -> PathBuf {
|
||||
ui.dirs.state_dir().join("takeout-staged.json")
|
||||
}
|
||||
|
||||
/// Read the remembered set, dropping any archive that has since been moved or
|
||||
/// deleted — a stale path would only fail at launch, and silently is worse.
|
||||
fn load_staged(ui: &Rc<Ui>) -> Vec<PathBuf> {
|
||||
let Ok(text) = std::fs::read_to_string(staged_path(ui)) else {
|
||||
return Vec::new();
|
||||
};
|
||||
let paths: Vec<String> = serde_json::from_str(&text).unwrap_or_default();
|
||||
paths
|
||||
.into_iter()
|
||||
.map(PathBuf::from)
|
||||
.filter(|p| p.is_file())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Remember the staged set. Best-effort: failing to write it costs the user a
|
||||
/// re-pick next time, which is not worth interrupting them over now.
|
||||
fn save_staged(ui: &Rc<Ui>) {
|
||||
let paths: Vec<String> = ui
|
||||
.takeout
|
||||
.archives
|
||||
.borrow()
|
||||
.iter()
|
||||
.map(|p| p.display().to_string())
|
||||
.collect();
|
||||
let path = staged_path(ui);
|
||||
let written = serde_json::to_string(&paths)
|
||||
.map_err(|e| e.to_string())
|
||||
.and_then(|text| std::fs::write(&path, text).map_err(|e| e.to_string()));
|
||||
if let Err(e) = written {
|
||||
tracing::warn!("cannot remember staged Takeout archives: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
const EMPTY_LIST_DESCRIPTION: &str = "Photos already in Proton Photos are matched by name and content and skipped, so \
|
||||
re-running an interrupted import is safe.";
|
||||
|
||||
pub(crate) struct TakeoutState {
|
||||
/// Archives staged for this import, in the order they were added. Sent as
|
||||
/// one set: Google splits an export across numbered zips and routinely puts
|
||||
/// a photo in one part and its metadata sidecar in another, so a per-file
|
||||
/// import would lose album titles and capture times.
|
||||
pub(crate) archives: RefCell<Vec<PathBuf>>,
|
||||
/// The staged-archive list and the rows currently rendered in it.
|
||||
pub(crate) list_group: adw::PreferencesGroup,
|
||||
pub(crate) rows: RefCell<Vec<adw::PreferencesRow>>,
|
||||
/// The drop zone, whose `dropzone-active` class follows the drag.
|
||||
pub(crate) dropzone: gtk4::Box,
|
||||
pub(crate) scan_button: gtk4::Button,
|
||||
pub(crate) import_button: gtk4::Button,
|
||||
pub(crate) cancel_button: gtk4::Button,
|
||||
pub(crate) clear_button: gtk4::Button,
|
||||
/// Progress group, hidden unless an import is running.
|
||||
pub(crate) progress_group: adw::PreferencesGroup,
|
||||
pub(crate) progress_label: gtk4::Label,
|
||||
pub(crate) progress_bar: gtk4::ProgressBar,
|
||||
/// Report group for the last finished run, hidden until there is one.
|
||||
pub(crate) summary_group: adw::PreferencesGroup,
|
||||
pub(crate) summary_rows: RefCell<Vec<adw::ActionRow>>,
|
||||
/// One [`Request::ImportStatus`] in flight at a time, so a wedged daemon
|
||||
/// can't stack worker threads on the 2s tick.
|
||||
pub(crate) inflight: Cell<bool>,
|
||||
/// Whether the daemon reported an import running on the last poll.
|
||||
pub(crate) running: Cell<bool>,
|
||||
/// Whether the run in flight was launched as a dry run, so the report can
|
||||
/// say "would upload" rather than claiming photos were sent. The daemon's
|
||||
/// summary carries no such flag — it is the same shape either way.
|
||||
pub(crate) dry_run: Cell<bool>,
|
||||
}
|
||||
|
||||
/// Widgets the Import page hands back for wiring.
|
||||
pub(crate) struct TakeoutWidgets {
|
||||
pub(crate) list_group: adw::PreferencesGroup,
|
||||
pub(crate) dropzone: gtk4::Box,
|
||||
/// The whole page, which is the drop target: a drag released anywhere on it
|
||||
/// counts, not only inside the dashed rectangle.
|
||||
pub(crate) page: gtk4::Widget,
|
||||
pub(crate) choose_button: gtk4::Button,
|
||||
pub(crate) scan_button: gtk4::Button,
|
||||
pub(crate) import_button: gtk4::Button,
|
||||
pub(crate) cancel_button: gtk4::Button,
|
||||
pub(crate) clear_button: gtk4::Button,
|
||||
pub(crate) back_button: gtk4::Button,
|
||||
pub(crate) progress_group: adw::PreferencesGroup,
|
||||
pub(crate) progress_label: gtk4::Label,
|
||||
pub(crate) progress_bar: gtk4::ProgressBar,
|
||||
pub(crate) summary_group: adw::PreferencesGroup,
|
||||
}
|
||||
|
||||
pub(crate) fn build_takeout_page() -> (gtk4::Widget, TakeoutWidgets) {
|
||||
// Header: a back button out to Settings, since this page is reached from
|
||||
// there and has no sidebar row of its own to navigate back with.
|
||||
let back_button = gtk4::Button::builder()
|
||||
.icon_name("go-previous-symbolic")
|
||||
.tooltip_text("Back to Settings")
|
||||
.valign(gtk4::Align::Center)
|
||||
.build();
|
||||
back_button.add_css_class("flat");
|
||||
|
||||
let title = gtk4::Label::builder()
|
||||
.label("Import from Google Photos")
|
||||
.halign(gtk4::Align::Start)
|
||||
.build();
|
||||
title.add_css_class("title-2");
|
||||
let subtitle = gtk4::Label::builder()
|
||||
.label("Add a Google Takeout export to your Proton Photos timeline.")
|
||||
.halign(gtk4::Align::Start)
|
||||
.wrap(true)
|
||||
.xalign(0.0)
|
||||
.build();
|
||||
subtitle.add_css_class("dim-label");
|
||||
|
||||
let titles = gtk4::Box::new(gtk4::Orientation::Vertical, 2);
|
||||
titles.set_hexpand(true);
|
||||
titles.append(&title);
|
||||
titles.append(&subtitle);
|
||||
|
||||
let header = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
|
||||
header.append(&back_button);
|
||||
header.append(&titles);
|
||||
let header_clamp = adw::Clamp::builder().child(&header).build();
|
||||
|
||||
// Drop zone. It is a hint, not the only way in — the button under it does
|
||||
// the same thing for anyone not dragging, and the drop target covers the
|
||||
// whole page so a drag released just outside the rectangle still lands.
|
||||
let drop_icon = gtk4::Image::from_icon_name("folder-download-symbolic");
|
||||
drop_icon.set_pixel_size(48);
|
||||
drop_icon.add_css_class("dim-label");
|
||||
let drop_title = gtk4::Label::builder()
|
||||
.label("Drop your Takeout .zip files here")
|
||||
.build();
|
||||
drop_title.add_css_class("title-4");
|
||||
let drop_hint = gtk4::Label::builder()
|
||||
.label("Add every part of the export at once — a photo and its metadata often sit in different parts.")
|
||||
.wrap(true)
|
||||
.justify(gtk4::Justification::Center)
|
||||
.max_width_chars(48)
|
||||
.build();
|
||||
drop_hint.add_css_class("dim-label");
|
||||
let choose_button = gtk4::Button::builder()
|
||||
.label("Choose files…")
|
||||
.halign(gtk4::Align::Center)
|
||||
.build();
|
||||
choose_button.add_css_class("pill");
|
||||
|
||||
let dropzone = gtk4::Box::new(gtk4::Orientation::Vertical, 10);
|
||||
dropzone.set_halign(gtk4::Align::Fill);
|
||||
dropzone.add_css_class("dropzone");
|
||||
dropzone.append(&drop_icon);
|
||||
dropzone.append(&drop_title);
|
||||
dropzone.append(&drop_hint);
|
||||
dropzone.append(&choose_button);
|
||||
|
||||
// Staged archives. Filled by `repaint_archives`, which owns the description
|
||||
// too (it becomes the count + total once there is something to count); the
|
||||
// group carries a placeholder row while empty so it never renders as a bare
|
||||
// title.
|
||||
let list_group = adw::PreferencesGroup::builder()
|
||||
.title("Archives to import")
|
||||
.description(EMPTY_LIST_DESCRIPTION)
|
||||
.build();
|
||||
|
||||
let clear_button = gtk4::Button::builder()
|
||||
.label("Clear")
|
||||
.tooltip_text("Remove every staged archive")
|
||||
.build();
|
||||
clear_button.add_css_class("flat");
|
||||
|
||||
// Actions. Scan is a dry run: it reads the archives' directories and reports
|
||||
// what an import *would* do, without uploading a byte. It is offered first
|
||||
// because the import behind it is hours long and hard to take back.
|
||||
let scan_button = gtk4::Button::builder()
|
||||
.label("Scan")
|
||||
.tooltip_text("Check the archives and report what would be imported, without uploading")
|
||||
.build();
|
||||
scan_button.add_css_class("pill");
|
||||
let import_button = gtk4::Button::builder().label("Import").build();
|
||||
import_button.add_css_class("pill");
|
||||
import_button.add_css_class("suggested-action");
|
||||
let cancel_button = gtk4::Button::builder()
|
||||
.label("Stop import")
|
||||
.tooltip_text("Finish the photo on the wire, file what was uploaded, then stop")
|
||||
.build();
|
||||
cancel_button.add_css_class("pill");
|
||||
cancel_button.add_css_class("destructive-action");
|
||||
cancel_button.set_visible(false);
|
||||
|
||||
// Clear sits apart from the launch buttons, on the other end of the row: it
|
||||
// undoes staging, and putting it next to Import invites the wrong click.
|
||||
let actions = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
|
||||
let spacer = gtk4::Box::new(gtk4::Orientation::Horizontal, 0);
|
||||
spacer.set_hexpand(true);
|
||||
actions.append(&clear_button);
|
||||
actions.append(&spacer);
|
||||
actions.append(&cancel_button);
|
||||
actions.append(&scan_button);
|
||||
actions.append(&import_button);
|
||||
|
||||
// Progress: the import's job from the transfer snapshot, so this page shows
|
||||
// the same numbers the Settings Activity list does.
|
||||
let progress_group = adw::PreferencesGroup::builder()
|
||||
.title("Import in progress")
|
||||
.visible(false)
|
||||
.build();
|
||||
let progress_box = gtk4::Box::new(gtk4::Orientation::Vertical, 6);
|
||||
progress_box.set_margin_top(8);
|
||||
progress_box.set_margin_bottom(8);
|
||||
let progress_label = gtk4::Label::builder().halign(gtk4::Align::Start).build();
|
||||
progress_label.add_css_class("dim-label");
|
||||
let progress_bar = gtk4::ProgressBar::new();
|
||||
progress_box.append(&progress_label);
|
||||
progress_box.append(&progress_bar);
|
||||
progress_group.add(
|
||||
&adw::PreferencesRow::builder()
|
||||
.activatable(false)
|
||||
.child(&progress_box)
|
||||
.build(),
|
||||
);
|
||||
|
||||
// Report for the last finished run, kept on screen after it ends: an import
|
||||
// outlives the window being on this page, and "what happened" is the whole
|
||||
// reason to come back to it.
|
||||
let summary_group = adw::PreferencesGroup::builder()
|
||||
.title("Last import")
|
||||
.visible(false)
|
||||
.build();
|
||||
|
||||
let groups = gtk4::Box::new(gtk4::Orientation::Vertical, 18);
|
||||
groups.append(&dropzone);
|
||||
groups.append(&list_group);
|
||||
groups.append(&actions);
|
||||
groups.append(&progress_group);
|
||||
groups.append(&summary_group);
|
||||
let clamp = adw::Clamp::builder()
|
||||
.maximum_size(640)
|
||||
.child(&groups)
|
||||
.build();
|
||||
let scroll = gtk4::ScrolledWindow::builder()
|
||||
.vexpand(true)
|
||||
.hscrollbar_policy(gtk4::PolicyType::Never)
|
||||
.child(&clamp)
|
||||
.build();
|
||||
|
||||
let page = gtk4::Box::new(gtk4::Orientation::Vertical, 12);
|
||||
page.set_margin_top(18);
|
||||
page.set_margin_bottom(18);
|
||||
page.set_margin_start(18);
|
||||
page.set_margin_end(18);
|
||||
page.append(&header_clamp);
|
||||
page.append(&scroll);
|
||||
|
||||
let widgets = TakeoutWidgets {
|
||||
list_group: list_group.clone(),
|
||||
dropzone: dropzone.clone(),
|
||||
page: page.clone().upcast(),
|
||||
choose_button,
|
||||
scan_button: scan_button.clone(),
|
||||
import_button: import_button.clone(),
|
||||
cancel_button: cancel_button.clone(),
|
||||
clear_button,
|
||||
back_button,
|
||||
progress_group: progress_group.clone(),
|
||||
progress_label: progress_label.clone(),
|
||||
progress_bar: progress_bar.clone(),
|
||||
summary_group: summary_group.clone(),
|
||||
};
|
||||
(page.upcast(), widgets)
|
||||
}
|
||||
|
||||
pub(crate) fn wire_takeout(ui: &Rc<Ui>, widgets: &TakeoutWidgets) {
|
||||
// Drag-and-drop. The target sits on the whole page rather than the dashed
|
||||
// rectangle, so a drag released anywhere counts; the rectangle only lights
|
||||
// up to say where the drop is understood.
|
||||
let target = gtk4::DropTarget::new(
|
||||
gtk4::gdk::FileList::static_type(),
|
||||
gtk4::gdk::DragAction::COPY,
|
||||
);
|
||||
let ui_enter = ui.clone();
|
||||
target.connect_enter(move |_, _, _| {
|
||||
ui_enter.takeout.dropzone.add_css_class("dropzone-active");
|
||||
gtk4::gdk::DragAction::COPY
|
||||
});
|
||||
let ui_leave = ui.clone();
|
||||
target.connect_leave(move |_| {
|
||||
ui_leave
|
||||
.takeout
|
||||
.dropzone
|
||||
.remove_css_class("dropzone-active");
|
||||
});
|
||||
let ui_drop = ui.clone();
|
||||
target.connect_drop(move |_, value, _, _| {
|
||||
ui_drop.takeout.dropzone.remove_css_class("dropzone-active");
|
||||
let Ok(files) = value.get::<gtk4::gdk::FileList>() else {
|
||||
return false;
|
||||
};
|
||||
let paths: Vec<PathBuf> = files.files().iter().filter_map(|f| f.path()).collect();
|
||||
add_archives(&ui_drop, paths);
|
||||
true
|
||||
});
|
||||
widgets.page.add_controller(target);
|
||||
|
||||
let ui_choose = ui.clone();
|
||||
widgets.choose_button.connect_clicked(move |_| {
|
||||
// Multi-select on purpose: the whole export goes in one request.
|
||||
let dialog = gtk4::FileDialog::builder()
|
||||
.title("Select your Google Takeout archives")
|
||||
.build();
|
||||
let filter = gtk4::FileFilter::new();
|
||||
filter.set_name(Some("Takeout archives"));
|
||||
filter.add_pattern("*.zip");
|
||||
let filters = gio::ListStore::new::<gtk4::FileFilter>();
|
||||
filters.append(&filter);
|
||||
dialog.set_filters(Some(&filters));
|
||||
|
||||
let ui = ui_choose.clone();
|
||||
dialog.open_multiple(
|
||||
ui_window(&ui_choose).as_ref(),
|
||||
gio::Cancellable::NONE,
|
||||
move |res| {
|
||||
let Ok(files) = res else { return };
|
||||
let paths: Vec<PathBuf> = (0..files.n_items())
|
||||
.filter_map(|index| files.item(index))
|
||||
.filter_map(|object| object.downcast::<gio::File>().ok())
|
||||
.filter_map(|file| file.path())
|
||||
.collect();
|
||||
add_archives(&ui, paths);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
let ui_clear = ui.clone();
|
||||
widgets.clear_button.connect_clicked(move |_| {
|
||||
ui_clear.takeout.archives.borrow_mut().clear();
|
||||
repaint_archives(&ui_clear);
|
||||
});
|
||||
|
||||
let ui_scan = ui.clone();
|
||||
widgets
|
||||
.scan_button
|
||||
.connect_clicked(move |_| start_import(&ui_scan, true));
|
||||
|
||||
// The import itself is confirmed: it uploads a whole photo library, which is
|
||||
// hours of bandwidth and a large change to the account's timeline.
|
||||
let ui_import = ui.clone();
|
||||
widgets.import_button.connect_clicked(move |_| {
|
||||
let count = ui_import.takeout.archives.borrow().len();
|
||||
let ui = ui_import.clone();
|
||||
let dialog = adw::AlertDialog::builder()
|
||||
.heading("Import this export?")
|
||||
.body(format!(
|
||||
"{count} archive(s) will be uploaded to Proton Photos, creating albums as they \
|
||||
appear in the export. This can take hours; you can stop it at any point and \
|
||||
what has been uploaded stays. Photos you already have are skipped.",
|
||||
))
|
||||
.build();
|
||||
dialog.add_response("cancel", "Cancel");
|
||||
dialog.add_response("import", "Import");
|
||||
dialog.set_response_appearance("import", adw::ResponseAppearance::Suggested);
|
||||
dialog.set_default_response(Some("cancel"));
|
||||
dialog.set_close_response("cancel");
|
||||
dialog.connect_response(None, move |_, resp| {
|
||||
if resp == "import" {
|
||||
start_import(&ui, false);
|
||||
}
|
||||
});
|
||||
dialog.present(ui_window(&ui_import).as_ref());
|
||||
});
|
||||
|
||||
let ui_cancel = ui.clone();
|
||||
widgets.cancel_button.connect_clicked(move |_| {
|
||||
let ui = ui_cancel.clone();
|
||||
ui.busy_begin();
|
||||
let rx = spawn_request(ui.dirs.control_socket(), Request::CancelImport);
|
||||
glib::spawn_future_local(async move {
|
||||
let result = rx.recv().await;
|
||||
ui.busy_end();
|
||||
match result {
|
||||
Ok(Ok(Response::Ok { .. })) => toast(&ui, "Stopping the import…"),
|
||||
Ok(Ok(Response::Error { message, kind })) => {
|
||||
toast_failure(&ui, "Couldn't stop the import", &message, kind);
|
||||
}
|
||||
_ => toast_error(
|
||||
&ui,
|
||||
"Couldn't stop the import",
|
||||
"The mount service didn't respond.",
|
||||
),
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
let ui_back = ui.clone();
|
||||
widgets
|
||||
.back_button
|
||||
.connect_clicked(move |_| ui_back.stack.set_visible_child_name("main"));
|
||||
|
||||
// Restore whatever was staged when the window last closed, before the first
|
||||
// paint, so the page comes up as the user left it.
|
||||
*ui.takeout.archives.borrow_mut() = load_staged(ui);
|
||||
repaint_archives(ui);
|
||||
}
|
||||
|
||||
/// Stage `paths`, keeping only `.zip` files and skipping ones already staged.
|
||||
///
|
||||
/// Both the chooser and the drop target land here, so the filter is stated once.
|
||||
/// A dropped folder (an already-extracted export) is rejected rather than walked:
|
||||
/// the scan reads the archives' central directories, and an extracted tree has
|
||||
/// none — silently importing half of it would be worse than saying no.
|
||||
pub(crate) fn add_archives(ui: &Rc<Ui>, paths: Vec<PathBuf>) {
|
||||
let mut added = 0usize;
|
||||
let mut rejected = 0usize;
|
||||
{
|
||||
let mut staged = ui.takeout.archives.borrow_mut();
|
||||
for path in paths {
|
||||
let is_zip = path
|
||||
.extension()
|
||||
.is_some_and(|ext| ext.eq_ignore_ascii_case("zip"));
|
||||
if !is_zip || !path.is_file() {
|
||||
rejected += 1;
|
||||
continue;
|
||||
}
|
||||
if staged.contains(&path) {
|
||||
continue;
|
||||
}
|
||||
staged.push(path);
|
||||
added += 1;
|
||||
}
|
||||
// Numbered parts drop in whatever order the file manager hands them
|
||||
// over; sorting keeps the list readable and the request stable.
|
||||
staged.sort();
|
||||
}
|
||||
repaint_archives(ui);
|
||||
if rejected > 0 {
|
||||
toast(
|
||||
ui,
|
||||
"Only Takeout .zip files can be imported — extracted folders aren't supported.",
|
||||
);
|
||||
} else if added > 0 {
|
||||
toast(ui, &format!("Added {added} archive(s)"));
|
||||
}
|
||||
}
|
||||
|
||||
/// Rebuild the staged-archive list and re-derive which actions are available.
|
||||
pub(crate) fn repaint_archives(ui: &Rc<Ui>) {
|
||||
for row in ui.takeout.rows.borrow_mut().drain(..) {
|
||||
ui.takeout.list_group.remove(&row);
|
||||
}
|
||||
|
||||
let staged = ui.takeout.archives.borrow().clone();
|
||||
let mut rows: Vec<adw::PreferencesRow> = Vec::new();
|
||||
if staged.is_empty() {
|
||||
ui.takeout
|
||||
.list_group
|
||||
.set_description(Some(EMPTY_LIST_DESCRIPTION));
|
||||
let row = dim_row("No archives added yet.");
|
||||
ui.takeout.list_group.add(&row);
|
||||
rows.push(row.upcast());
|
||||
} else {
|
||||
let mut total = 0u64;
|
||||
for path in &staged {
|
||||
let size = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0);
|
||||
total += size;
|
||||
let row = adw::ActionRow::builder()
|
||||
.title(
|
||||
path.file_name()
|
||||
.map(|n| n.to_string_lossy().to_string())
|
||||
.unwrap_or_else(|| path.display().to_string()),
|
||||
)
|
||||
.subtitle(human_bytes(size))
|
||||
.build();
|
||||
let remove = gtk4::Button::builder()
|
||||
.icon_name("edit-delete-symbolic")
|
||||
.tooltip_text("Remove from this import")
|
||||
.valign(gtk4::Align::Center)
|
||||
.build();
|
||||
remove.add_css_class("flat");
|
||||
let ui_remove = ui.clone();
|
||||
let target = path.clone();
|
||||
remove.connect_clicked(move |_| {
|
||||
ui_remove
|
||||
.takeout
|
||||
.archives
|
||||
.borrow_mut()
|
||||
.retain(|p| *p != target);
|
||||
repaint_archives(&ui_remove);
|
||||
});
|
||||
row.add_suffix(&remove);
|
||||
ui.takeout.list_group.add(&row);
|
||||
rows.push(row.upcast());
|
||||
}
|
||||
ui.takeout.list_group.set_description(Some(&format!(
|
||||
"{} archive(s), {} in total. {EMPTY_LIST_DESCRIPTION}",
|
||||
staged.len(),
|
||||
human_bytes(total)
|
||||
)));
|
||||
}
|
||||
*ui.takeout.rows.borrow_mut() = rows;
|
||||
|
||||
save_staged(ui);
|
||||
sync_takeout_actions(ui);
|
||||
}
|
||||
|
||||
/// Enable exactly the actions that make sense right now: nothing to launch
|
||||
/// without archives, nothing to launch while a run is in flight, and Stop only
|
||||
/// while there is something to stop.
|
||||
fn sync_takeout_actions(ui: &Rc<Ui>) {
|
||||
let has_files = !ui.takeout.archives.borrow().is_empty();
|
||||
let running = ui.takeout.running.get();
|
||||
ui.takeout.scan_button.set_sensitive(has_files && !running);
|
||||
ui.takeout
|
||||
.import_button
|
||||
.set_sensitive(has_files && !running);
|
||||
ui.takeout.clear_button.set_sensitive(has_files && !running);
|
||||
ui.takeout.cancel_button.set_visible(running);
|
||||
}
|
||||
|
||||
/// Hand the staged set to the daemon. It acks immediately and works in the
|
||||
/// background — an export is far past any socket timeout — so success here means
|
||||
/// "started", and the page switches over to reporting progress.
|
||||
fn start_import(ui: &Rc<Ui>, dry_run: bool) {
|
||||
let archives: Vec<String> = ui
|
||||
.takeout
|
||||
.archives
|
||||
.borrow()
|
||||
.iter()
|
||||
.map(|p| p.display().to_string())
|
||||
.collect();
|
||||
if archives.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
ui.takeout.dry_run.set(dry_run);
|
||||
ui.busy_begin();
|
||||
let rx = spawn_request(
|
||||
ui.dirs.control_socket(),
|
||||
Request::ImportTakeout { archives, dry_run },
|
||||
);
|
||||
let ui = ui.clone();
|
||||
glib::spawn_future_local(async move {
|
||||
let result = rx.recv().await;
|
||||
ui.busy_end();
|
||||
match result {
|
||||
Ok(Ok(Response::Ok { .. })) => {
|
||||
ui.takeout.running.set(true);
|
||||
ui.takeout.summary_group.set_visible(false);
|
||||
ui.takeout.progress_group.set_visible(true);
|
||||
ui.takeout.progress_bar.set_fraction(0.0);
|
||||
ui.takeout.progress_label.set_text(if dry_run {
|
||||
"Reading archives…"
|
||||
} else {
|
||||
"Starting import…"
|
||||
});
|
||||
sync_takeout_actions(&ui);
|
||||
toast(
|
||||
&ui,
|
||||
if dry_run {
|
||||
"Scanning the export…"
|
||||
} else {
|
||||
"Import started — you can leave this page, it keeps running."
|
||||
},
|
||||
);
|
||||
}
|
||||
Ok(Ok(Response::Error { message, kind })) => {
|
||||
toast_failure(&ui, "Couldn't start the import", &message, kind);
|
||||
}
|
||||
_ => toast_error(
|
||||
&ui,
|
||||
"Couldn't start the import",
|
||||
"The mount service didn't respond.",
|
||||
),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Poll the daemon for the import's state while the page is on screen, and paint
|
||||
/// the report once a run has finished. Cheap and inflight-guarded, like the other
|
||||
/// tick polls.
|
||||
pub(crate) fn refresh_takeout(ui: &Rc<Ui>) {
|
||||
if ui.takeout.inflight.get() {
|
||||
return;
|
||||
}
|
||||
ui.takeout.inflight.set(true);
|
||||
let rx = spawn_request(ui.dirs.control_socket(), Request::ImportStatus);
|
||||
let ui = ui.clone();
|
||||
glib::spawn_future_local(async move {
|
||||
let result = rx.recv().await;
|
||||
ui.takeout.inflight.set(false);
|
||||
let Ok(Ok(Response::ImportStatus { running, summary })) = result else {
|
||||
return;
|
||||
};
|
||||
let was_running = ui.takeout.running.replace(running);
|
||||
if running {
|
||||
ui.takeout.progress_group.set_visible(true);
|
||||
} else {
|
||||
ui.takeout.progress_group.set_visible(false);
|
||||
// An import runs for hours, so its ending is announced rather than
|
||||
// left to be discovered: a toast for whoever is looking at the
|
||||
// window, and a desktop notification for whoever is not.
|
||||
if was_running {
|
||||
let dry = ui.takeout.dry_run.get();
|
||||
let headline = if dry {
|
||||
"Scan finished"
|
||||
} else {
|
||||
"Import finished"
|
||||
};
|
||||
toast(&ui, headline);
|
||||
if !dry {
|
||||
let body = match &summary {
|
||||
Some(s) if s.failed > 0 => {
|
||||
format!("{} photos added, {} failed.", s.uploaded, s.failed)
|
||||
}
|
||||
Some(s) if s.cancelled => {
|
||||
format!("Stopped after {} photos.", s.uploaded)
|
||||
}
|
||||
Some(s) => format!("{} photos added to Proton Photos.", s.uploaded),
|
||||
None => "The Google Photos import has finished.".to_string(),
|
||||
};
|
||||
notify("takeout-import", headline, &body);
|
||||
}
|
||||
}
|
||||
}
|
||||
if was_running != running {
|
||||
sync_takeout_actions(&ui);
|
||||
}
|
||||
match summary {
|
||||
Some(summary) => repaint_summary(&ui, &summary),
|
||||
None => ui.takeout.summary_group.set_visible(false),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Paint the last run's report.
|
||||
fn repaint_summary(ui: &Rc<Ui>, summary: &ImportSummary) {
|
||||
for row in ui.takeout.summary_rows.borrow_mut().drain(..) {
|
||||
ui.takeout.summary_group.remove(&row);
|
||||
}
|
||||
let dry = ui.takeout.dry_run.get();
|
||||
ui.takeout
|
||||
.summary_group
|
||||
.set_title(if dry { "Scan result" } else { "Last import" });
|
||||
ui.takeout.summary_group.set_description(Some(if dry {
|
||||
"Nothing was uploaded — this is what an import would do."
|
||||
} else if summary.cancelled {
|
||||
"Stopped early. Everything uploaded so far is in your timeline; run it again to \
|
||||
continue where it left off."
|
||||
} else {
|
||||
"Finished."
|
||||
}));
|
||||
|
||||
// Only lines that carry information: a run with no failures should not have
|
||||
// to show a "0 failed" row for the user to read past.
|
||||
let mut lines: Vec<(String, String)> = vec![
|
||||
("Photos in the export".into(), summary.found.to_string()),
|
||||
(
|
||||
if dry { "Would upload" } else { "Uploaded" }.into(),
|
||||
summary.uploaded.to_string(),
|
||||
),
|
||||
];
|
||||
if summary.duplicates > 0 {
|
||||
lines.push((
|
||||
"Already in Proton Photos".into(),
|
||||
summary.duplicates.to_string(),
|
||||
));
|
||||
}
|
||||
if summary.albums_created > 0 {
|
||||
lines.push((
|
||||
if dry {
|
||||
"Albums to create"
|
||||
} else {
|
||||
"Albums created"
|
||||
}
|
||||
.into(),
|
||||
summary.albums_created.to_string(),
|
||||
));
|
||||
}
|
||||
if summary.album_links > 0 {
|
||||
lines.push((
|
||||
"Photos filed into albums".into(),
|
||||
summary.album_links.to_string(),
|
||||
));
|
||||
}
|
||||
if summary.skipped_trashed > 0 {
|
||||
lines.push((
|
||||
"Skipped (in Google's trash)".into(),
|
||||
summary.skipped_trashed.to_string(),
|
||||
));
|
||||
}
|
||||
if summary.bytes > 0 {
|
||||
lines.push((
|
||||
if dry {
|
||||
"Data to upload"
|
||||
} else {
|
||||
"Data uploaded"
|
||||
}
|
||||
.into(),
|
||||
human_bytes(summary.bytes),
|
||||
));
|
||||
}
|
||||
if summary.failed > 0 {
|
||||
lines.push(("Failed".into(), summary.failed.to_string()));
|
||||
}
|
||||
|
||||
let mut rows = Vec::new();
|
||||
for (title, value) in lines {
|
||||
let row = adw::ActionRow::builder()
|
||||
.title(title)
|
||||
.subtitle(value)
|
||||
.build();
|
||||
row.add_css_class("property");
|
||||
ui.takeout.summary_group.add(&row);
|
||||
rows.push(row);
|
||||
}
|
||||
*ui.takeout.summary_rows.borrow_mut() = rows;
|
||||
ui.takeout.summary_group.set_visible(true);
|
||||
}
|
||||
|
||||
/// Paint the import's progress from the transfer snapshot the Settings tick
|
||||
/// already fetched, so this page costs no extra poll. A job with no known total
|
||||
/// pulses instead of showing a fraction, the same as the Activity list.
|
||||
pub(crate) fn takeout_progress(ui: &Rc<Ui>, jobs: &[JobItem]) {
|
||||
let Some(job) = jobs.iter().find(|job| job.title == IMPORT_JOB) else {
|
||||
return;
|
||||
};
|
||||
ui.takeout.progress_group.set_visible(true);
|
||||
if job.total > 0 {
|
||||
ui.takeout
|
||||
.progress_bar
|
||||
.set_fraction((job.done as f64 / job.total as f64).min(1.0));
|
||||
ui.takeout
|
||||
.progress_label
|
||||
.set_text(&format!("{} of {} — {}", job.done, job.total, job.detail));
|
||||
} else {
|
||||
ui.takeout.progress_bar.pulse();
|
||||
ui.takeout
|
||||
.progress_label
|
||||
.set_text(if job.detail.is_empty() {
|
||||
"Working…"
|
||||
} else {
|
||||
&job.detail
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -122,7 +122,7 @@ pub(crate) fn wire_trash(ui: &Rc<Ui>, list: >k4::ListView, empty: >k4::Butto
|
||||
factory.connect_setup(move |_, item| {
|
||||
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
|
||||
|
||||
let icon = gtk4::Image::builder().pixel_size(32).build();
|
||||
let thumbnail = file_thumbnail_widget(40, 32);
|
||||
let name = gtk4::Label::builder()
|
||||
.halign(gtk4::Align::Start)
|
||||
.ellipsize(gtk4::pango::EllipsizeMode::Middle)
|
||||
@@ -169,35 +169,38 @@ pub(crate) fn wire_trash(ui: &Rc<Ui>, list: >k4::ListView, empty: >k4::Butto
|
||||
row.set_margin_bottom(6);
|
||||
row.set_margin_start(6);
|
||||
row.set_margin_end(6);
|
||||
row.append(&icon);
|
||||
row.append(&thumbnail);
|
||||
row.append(&text);
|
||||
row.append(&restore);
|
||||
row.append(&purge);
|
||||
item.set_child(Some(&row));
|
||||
});
|
||||
factory.connect_bind(|_, item| {
|
||||
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
|
||||
let Some(entry) = bound_entry(item) else {
|
||||
return;
|
||||
};
|
||||
let row = item.child().and_downcast::<gtk4::Box>().unwrap();
|
||||
let Some(icon) = row.first_child().and_downcast::<gtk4::Image>() else {
|
||||
return;
|
||||
};
|
||||
icon.set_icon_name(Some(&format!("{}-symbolic", icon_base_for(&entry))));
|
||||
let Some(text) = icon.next_sibling().and_downcast::<gtk4::Box>() else {
|
||||
return;
|
||||
};
|
||||
if let Some(name) = text.first_child().and_downcast::<gtk4::Label>() {
|
||||
name.set_label(&entry.name);
|
||||
}
|
||||
if let Some(meta) = text.last_child().and_downcast::<gtk4::Label>() {
|
||||
let kind = if entry.is_dir {
|
||||
"Folder".to_string()
|
||||
} else {
|
||||
human_bytes(entry.size)
|
||||
factory.connect_bind({
|
||||
let ui = ui.clone();
|
||||
move |_, item| {
|
||||
let item = item.downcast_ref::<gtk4::ListItem>().unwrap();
|
||||
let Some(entry) = bound_entry(item) else {
|
||||
return;
|
||||
};
|
||||
meta.set_label(&format!("{kind} · {}", format_modified(entry.modified)));
|
||||
let row = item.child().and_downcast::<gtk4::Box>().unwrap();
|
||||
let Some(thumbnail) = row.first_child().and_downcast::<gtk4::Overlay>() else {
|
||||
return;
|
||||
};
|
||||
bind_file_thumbnail(&ui, &thumbnail, &entry, true);
|
||||
let Some(text) = thumbnail.next_sibling().and_downcast::<gtk4::Box>() else {
|
||||
return;
|
||||
};
|
||||
if let Some(name) = text.first_child().and_downcast::<gtk4::Label>() {
|
||||
name.set_label(&entry.name);
|
||||
}
|
||||
if let Some(meta) = text.last_child().and_downcast::<gtk4::Label>() {
|
||||
let kind = if entry.is_dir {
|
||||
"Folder".to_string()
|
||||
} else {
|
||||
human_bytes(entry.size)
|
||||
};
|
||||
meta.set_label(&format!("{kind} · {}", format_modified(entry.modified)));
|
||||
}
|
||||
}
|
||||
});
|
||||
list.set_factory(Some(&factory));
|
||||
@@ -216,6 +219,7 @@ pub(crate) fn bound_entry(item: >k4::ListItem) -> Option<DirEntry> {
|
||||
|
||||
/// Fetch the trash listing and repaint the page.
|
||||
pub(crate) fn load_trash(ui: &Rc<Ui>) {
|
||||
cancel_file_thumbnails(ui);
|
||||
// Drop the old rows first: a stale row here would offer Restore on something
|
||||
// that may already be gone.
|
||||
ui.trash.model.remove_all();
|
||||
|
||||
@@ -11,8 +11,8 @@ pub(crate) struct DetailsState {
|
||||
pub(crate) details_suppress: Cell<bool>,
|
||||
/// The grid's and list's selection models. Both wrap the one browser model, so
|
||||
/// a selection in either drives the details pane.
|
||||
pub(crate) grid_selection: gtk4::SingleSelection,
|
||||
pub(crate) list_selection: gtk4::SingleSelection,
|
||||
pub(crate) grid_selection: gtk4::MultiSelection,
|
||||
pub(crate) list_selection: gtk4::MultiSelection,
|
||||
}
|
||||
|
||||
/// The widgets in the browser's details pane that a selection repaints.
|
||||
@@ -158,24 +158,22 @@ pub(crate) fn wire_details(ui: &Rc<Ui>) {
|
||||
ui.details.list_selection.clone(),
|
||||
] {
|
||||
let ui_sel = ui.clone();
|
||||
selection.connect_selection_changed(move |sel, _, _| {
|
||||
match entry_at(sel.model().as_ref(), sel.selected()) {
|
||||
Some(entry) => show_details(&ui_sel, &entry),
|
||||
None => hide_details(&ui_sel),
|
||||
selection.connect_selection_changed(move |_, _, _| {
|
||||
// The details pane describes *one* entry. A batch is the bulk bar's
|
||||
// business, so the pane steps aside rather than picking a member of
|
||||
// the selection to speak for the rest.
|
||||
let entries = selected_entries(&ui_sel);
|
||||
match entries.as_slice() {
|
||||
[entry] => show_details(&ui_sel, entry),
|
||||
_ => hide_details(&ui_sel),
|
||||
}
|
||||
sync_bulk_bar(&ui_sel);
|
||||
});
|
||||
}
|
||||
|
||||
let ui_close = ui.clone();
|
||||
ui.details.details.close_button.connect_clicked(move |_| {
|
||||
ui_close
|
||||
.details
|
||||
.grid_selection
|
||||
.set_selected(gtk4::INVALID_LIST_POSITION);
|
||||
ui_close
|
||||
.details
|
||||
.list_selection
|
||||
.set_selected(gtk4::INVALID_LIST_POSITION);
|
||||
clear_selection(&ui_close);
|
||||
hide_details(&ui_close);
|
||||
});
|
||||
|
||||
@@ -289,8 +287,9 @@ pub(crate) fn hide_details(ui: &Rc<Ui>) {
|
||||
*ui.details.details_entry.borrow_mut() = None;
|
||||
}
|
||||
|
||||
/// The entry highlighted in whichever browser view is on screen, if any. Backs the
|
||||
/// F2 / Delete shortcuts.
|
||||
/// The single entry the details pane is showing, if any. Backs the actions that
|
||||
/// only make sense on one entry (rename, versions); the batch-capable ones ask
|
||||
/// [`selected_entries`] instead.
|
||||
pub(crate) fn selected_entry(ui: &Rc<Ui>) -> Option<DirEntry> {
|
||||
ui.details.details_entry.borrow().clone()
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
pub(crate) mod details;
|
||||
pub(crate) mod share_dialog;
|
||||
pub(crate) mod thumbnails;
|
||||
pub(crate) mod versions_dialog;
|
||||
@@ -0,0 +1,469 @@
|
||||
//! Shared thumbnails for ordinary Drive image files.
|
||||
//!
|
||||
//! The Photos gallery has timeline-specific batching and aspect-ratio state.
|
||||
//! File listings need a smaller, reusable surface: one loader backs My files,
|
||||
//! search, Shared and Trash, with modification-time cache keys
|
||||
//! so a replaced image never flashes its previous contents.
|
||||
|
||||
use crate::*;
|
||||
use pdfs_core::control::{FileThumbRequest, is_thumbnail_image_name};
|
||||
|
||||
const THUMB_BATCH: usize = 32;
|
||||
const THUMB_CACHE_MAX: usize = 256;
|
||||
const THUMB_MISSING_MAX: usize = 2048;
|
||||
const THUMB_DEBOUNCE: Duration = Duration::from_millis(40);
|
||||
const THUMB_RETRY: Duration = Duration::from_secs(2);
|
||||
|
||||
pub(crate) struct FileThumbnailState {
|
||||
textures: RefCell<HashMap<String, gtk4::gdk::Texture>>,
|
||||
texture_order: RefCell<VecDeque<String>>,
|
||||
missing: RefCell<HashSet<String>>,
|
||||
missing_order: RefCell<VecDeque<String>>,
|
||||
wanted: RefCell<HashMap<String, Vec<gtk4::Picture>>>,
|
||||
queue: RefCell<VecDeque<FileThumbRequest>>,
|
||||
inflight: Cell<bool>,
|
||||
source: RefCell<Option<glib::SourceId>>,
|
||||
generation: Cell<u64>,
|
||||
generation_loading: Cell<bool>,
|
||||
}
|
||||
|
||||
#[derive(Debug, PartialEq, Eq)]
|
||||
enum ReplyDisposition {
|
||||
Ignore,
|
||||
Load,
|
||||
Retry,
|
||||
Missing,
|
||||
}
|
||||
|
||||
fn reply_disposition(wanted: bool, has_path: bool, pending: bool) -> ReplyDisposition {
|
||||
if !wanted {
|
||||
ReplyDisposition::Ignore
|
||||
} else if has_path {
|
||||
ReplyDisposition::Load
|
||||
} else if pending {
|
||||
ReplyDisposition::Retry
|
||||
} else {
|
||||
ReplyDisposition::Missing
|
||||
}
|
||||
}
|
||||
|
||||
impl FileThumbnailState {
|
||||
pub(crate) fn new() -> Self {
|
||||
Self {
|
||||
textures: RefCell::new(HashMap::new()),
|
||||
texture_order: RefCell::new(VecDeque::new()),
|
||||
missing: RefCell::new(HashSet::new()),
|
||||
missing_order: RefCell::new(VecDeque::new()),
|
||||
wanted: RefCell::new(HashMap::new()),
|
||||
queue: RefCell::new(VecDeque::new()),
|
||||
inflight: Cell::new(false),
|
||||
source: RefCell::new(None),
|
||||
generation: Cell::new(0),
|
||||
generation_loading: Cell::new(false),
|
||||
}
|
||||
}
|
||||
|
||||
fn store_texture(&self, key: &str, texture: gtk4::gdk::Texture) {
|
||||
let mut textures = self.textures.borrow_mut();
|
||||
let mut order = self.texture_order.borrow_mut();
|
||||
if textures.insert(key.to_string(), texture).is_none() {
|
||||
order.push_back(key.to_string());
|
||||
}
|
||||
while order.len() > THUMB_CACHE_MAX {
|
||||
if let Some(old) = order.pop_front() {
|
||||
textures.remove(&old);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn store_missing(&self, key: String) {
|
||||
let mut missing = self.missing.borrow_mut();
|
||||
let mut order = self.missing_order.borrow_mut();
|
||||
if missing.insert(key.clone()) {
|
||||
order.push_back(key);
|
||||
}
|
||||
while order.len() > THUMB_MISSING_MAX {
|
||||
if let Some(old) = order.pop_front() {
|
||||
missing.remove(&old);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Abandon every opportunistic ordinary-file thumbnail associated with the
|
||||
/// listing being left. Advancing a wire generation lets the daemon cancel work
|
||||
/// even when an older request reaches it after this one.
|
||||
pub(crate) fn cancel_file_thumbnails(ui: &Rc<Ui>) {
|
||||
if let Some(source) = ui.file_thumbs.source.borrow_mut().take() {
|
||||
source.remove();
|
||||
}
|
||||
ui.file_thumbs.queue.borrow_mut().clear();
|
||||
ui.file_thumbs.wanted.borrow_mut().clear();
|
||||
let generation = ui.file_thumbs.generation.replace(0);
|
||||
// The request is useful even when its reply is not. `spawn_request` owns the
|
||||
// socket round-trip on a worker thread, so dropping the receiver is safe.
|
||||
if generation != 0 {
|
||||
drop(spawn_request(
|
||||
ui.dirs.control_socket(),
|
||||
Request::CancelFileThumbs {
|
||||
generation: generation.saturating_add(1),
|
||||
},
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
/// A square thumbnail surface with a generic file icon underneath the picture.
|
||||
/// Call [`bind_file_thumbnail`] whenever a recycled row is bound to an entry.
|
||||
pub(crate) fn file_thumbnail_widget(size: i32, fallback_size: i32) -> gtk4::Overlay {
|
||||
let fallback = gtk4::Image::builder().pixel_size(fallback_size).build();
|
||||
let picture = gtk4::Picture::builder()
|
||||
.width_request(size)
|
||||
.height_request(size)
|
||||
.content_fit(gtk4::ContentFit::Cover)
|
||||
.can_shrink(true)
|
||||
.visible(false)
|
||||
.build();
|
||||
picture.add_css_class("file-thumbnail");
|
||||
|
||||
let overlay = gtk4::Overlay::new();
|
||||
overlay.set_width_request(size);
|
||||
overlay.set_height_request(size);
|
||||
overlay.set_halign(gtk4::Align::Center);
|
||||
overlay.set_valign(gtk4::Align::Center);
|
||||
overlay.set_overflow(gtk4::Overflow::Hidden);
|
||||
overlay.set_child(Some(&fallback));
|
||||
overlay.add_overlay(&picture);
|
||||
overlay
|
||||
}
|
||||
|
||||
/// Resize an existing thumbnail surface. Grid rows are recycled, so changing
|
||||
/// these requests in place and rebinding the model is enough to implement the
|
||||
/// status-bar zoom without downloading or decoding an image again.
|
||||
pub(crate) fn resize_file_thumbnail(widget: >k4::Overlay, size: i32, fallback_size: i32) {
|
||||
widget.set_width_request(size);
|
||||
widget.set_height_request(size);
|
||||
let Some(fallback) = widget.child().and_downcast::<gtk4::Image>() else {
|
||||
return;
|
||||
};
|
||||
fallback.set_pixel_size(fallback_size);
|
||||
if let Some(picture) = fallback.next_sibling().and_downcast::<gtk4::Picture>() {
|
||||
picture.set_width_request(size);
|
||||
picture.set_height_request(size);
|
||||
}
|
||||
}
|
||||
|
||||
/// Bind a thumbnail surface to one entry, painting a cached texture immediately
|
||||
/// or queueing an on-demand daemon request. Non-images retain the generic icon.
|
||||
pub(crate) fn bind_file_thumbnail(
|
||||
ui: &Rc<Ui>,
|
||||
widget: >k4::Overlay,
|
||||
entry: &DirEntry,
|
||||
symbolic_fallback: bool,
|
||||
) {
|
||||
let Some(fallback) = widget.child().and_downcast::<gtk4::Image>() else {
|
||||
return;
|
||||
};
|
||||
// The browser may add a status badge after the picture. Locate the picture
|
||||
// immediately after the base fallback instead of assuming it is the last
|
||||
// overlay child.
|
||||
let Some(picture) = fallback.next_sibling().and_downcast::<gtk4::Picture>() else {
|
||||
return;
|
||||
};
|
||||
|
||||
let fallback_name = if symbolic_fallback {
|
||||
format!("{}-symbolic", icon_base_for(entry))
|
||||
} else {
|
||||
icon_base_for(entry).to_string()
|
||||
};
|
||||
fallback.set_icon_name(Some(&fallback_name));
|
||||
fallback.set_visible(true);
|
||||
picture.set_visible(false);
|
||||
picture.set_paintable(gtk4::gdk::Paintable::NONE);
|
||||
|
||||
// The widget name is a cheap binding token. List factories recycle their
|
||||
// children; a reply for the previous entry must not paint into the new one.
|
||||
let key = thumbnail_key(&entry.uid, entry.modified);
|
||||
picture.set_widget_name(&key);
|
||||
remove_waiter(ui, &picture);
|
||||
|
||||
if entry.is_dir || !is_image_name(&entry.name) || entry.uid.is_empty() {
|
||||
return;
|
||||
}
|
||||
if let Some(texture) = ui.file_thumbs.textures.borrow().get(&key) {
|
||||
paint(&picture, &key, texture);
|
||||
return;
|
||||
}
|
||||
if ui.file_thumbs.missing.borrow().contains(&key) {
|
||||
return;
|
||||
}
|
||||
|
||||
ui.file_thumbs
|
||||
.wanted
|
||||
.borrow_mut()
|
||||
.entry(key)
|
||||
.or_default()
|
||||
.push(picture);
|
||||
let mut queue = ui.file_thumbs.queue.borrow_mut();
|
||||
queue.retain(|item| item.uid != entry.uid);
|
||||
queue.push_back(FileThumbRequest {
|
||||
uid: entry.uid.clone(),
|
||||
modified: entry.modified,
|
||||
name: entry.name.clone(),
|
||||
});
|
||||
drop(queue);
|
||||
schedule_file_thumbs(ui);
|
||||
}
|
||||
|
||||
/// Create and immediately bind a thumbnail for a non-recycled row.
|
||||
pub(crate) fn file_thumbnail(
|
||||
ui: &Rc<Ui>,
|
||||
entry: &DirEntry,
|
||||
size: i32,
|
||||
fallback_size: i32,
|
||||
symbolic_fallback: bool,
|
||||
) -> gtk4::Overlay {
|
||||
let widget = file_thumbnail_widget(size, fallback_size);
|
||||
bind_file_thumbnail(ui, &widget, entry, symbolic_fallback);
|
||||
widget
|
||||
}
|
||||
|
||||
/// Whether the file name denotes an image format for which Proton or the local
|
||||
/// decoder can reasonably provide a thumbnail.
|
||||
pub(crate) fn is_image_name(name: &str) -> bool {
|
||||
is_thumbnail_image_name(name)
|
||||
}
|
||||
|
||||
fn thumbnail_key(uid: &str, modified: i64) -> String {
|
||||
format!("{uid}:{modified}")
|
||||
}
|
||||
|
||||
fn remove_waiter(ui: &Rc<Ui>, picture: >k4::Picture) {
|
||||
let mut wanted = ui.file_thumbs.wanted.borrow_mut();
|
||||
wanted.retain(|_, pictures| {
|
||||
pictures.retain(|candidate| candidate != picture);
|
||||
!pictures.is_empty()
|
||||
});
|
||||
}
|
||||
|
||||
fn paint(picture: >k4::Picture, key: &str, texture: >k4::gdk::Texture) {
|
||||
if picture.widget_name() != key {
|
||||
return;
|
||||
}
|
||||
picture.set_paintable(Some(texture));
|
||||
picture.set_visible(true);
|
||||
if let Some(overlay) = picture.parent().and_downcast::<gtk4::Overlay>()
|
||||
&& let Some(fallback) = overlay.child().and_downcast::<gtk4::Image>()
|
||||
{
|
||||
fallback.set_visible(false);
|
||||
}
|
||||
}
|
||||
|
||||
fn schedule_file_thumbs(ui: &Rc<Ui>) {
|
||||
if ui.file_thumbs.queue.borrow().is_empty() || ui.file_thumbs.inflight.get() {
|
||||
return;
|
||||
}
|
||||
if ui.file_thumbs.generation.get() == 0 {
|
||||
reserve_file_thumb_generation(ui);
|
||||
return;
|
||||
}
|
||||
if let Some(source) = ui.file_thumbs.source.borrow_mut().take() {
|
||||
source.remove();
|
||||
}
|
||||
let ui_flush = ui.clone();
|
||||
let source = glib::timeout_add_local_once(THUMB_DEBOUNCE, move || {
|
||||
ui_flush.file_thumbs.source.borrow_mut().take();
|
||||
flush_file_thumbs(&ui_flush);
|
||||
});
|
||||
*ui.file_thumbs.source.borrow_mut() = Some(source);
|
||||
}
|
||||
|
||||
fn reserve_file_thumb_generation(ui: &Rc<Ui>) {
|
||||
if ui.file_thumbs.generation_loading.replace(true) {
|
||||
return;
|
||||
}
|
||||
let rx = spawn_request(
|
||||
ui.dirs.control_socket(),
|
||||
Request::ReserveFileThumbGeneration,
|
||||
);
|
||||
let ui = ui.clone();
|
||||
glib::spawn_future_local(async move {
|
||||
let result = rx.recv().await;
|
||||
ui.file_thumbs.generation_loading.set(false);
|
||||
match result {
|
||||
Ok(Ok(Response::FileThumbGeneration { generation })) if generation != 0 => {
|
||||
ui.file_thumbs.generation.set(generation);
|
||||
schedule_file_thumbs(&ui);
|
||||
}
|
||||
_ => retry_file_thumb_generation(&ui),
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
fn retry_file_thumb_generation(ui: &Rc<Ui>) {
|
||||
if let Some(source) = ui.file_thumbs.source.borrow_mut().take() {
|
||||
source.remove();
|
||||
}
|
||||
let ui_retry = ui.clone();
|
||||
let source = glib::timeout_add_local_once(THUMB_RETRY, move || {
|
||||
ui_retry.file_thumbs.source.borrow_mut().take();
|
||||
schedule_file_thumbs(&ui_retry);
|
||||
});
|
||||
*ui.file_thumbs.source.borrow_mut() = Some(source);
|
||||
}
|
||||
|
||||
fn flush_file_thumbs(ui: &Rc<Ui>) {
|
||||
if ui.file_thumbs.inflight.get() {
|
||||
return;
|
||||
}
|
||||
let items: Vec<FileThumbRequest> = {
|
||||
let mut queue = ui.file_thumbs.queue.borrow_mut();
|
||||
let wanted = ui.file_thumbs.wanted.borrow();
|
||||
let mut batch = Vec::new();
|
||||
while batch.len() < THUMB_BATCH {
|
||||
let Some(item) = queue.pop_front() else {
|
||||
break;
|
||||
};
|
||||
if wanted.contains_key(&thumbnail_key(&item.uid, item.modified)) {
|
||||
batch.push(item);
|
||||
}
|
||||
}
|
||||
batch
|
||||
};
|
||||
if items.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
ui.file_thumbs.inflight.set(true);
|
||||
let rx = spawn_request(
|
||||
ui.dirs.control_socket(),
|
||||
Request::FileThumbs {
|
||||
items: items.clone(),
|
||||
generation: ui.file_thumbs.generation.get(),
|
||||
},
|
||||
);
|
||||
let ui = ui.clone();
|
||||
glib::spawn_future_local(async move {
|
||||
let result = rx.recv().await;
|
||||
ui.file_thumbs.inflight.set(false);
|
||||
match result {
|
||||
Ok(Ok(Response::Thumbs { items: replies })) => {
|
||||
let mut requested: HashMap<String, FileThumbRequest> = items
|
||||
.iter()
|
||||
.cloned()
|
||||
.map(|item| (item.uid.clone(), item))
|
||||
.collect();
|
||||
let mut pending = Vec::new();
|
||||
for reply in replies {
|
||||
let Some(request) = requested.remove(&reply.uid) else {
|
||||
continue;
|
||||
};
|
||||
let key = thumbnail_key(&request.uid, request.modified);
|
||||
let wanted = ui.file_thumbs.wanted.borrow().contains_key(&key);
|
||||
match reply_disposition(wanted, reply.path.is_some(), reply.pending) {
|
||||
ReplyDisposition::Ignore => continue,
|
||||
ReplyDisposition::Load => {
|
||||
let Some(path) = reply.path else { continue };
|
||||
match gtk4::gdk::Texture::from_filename(&path) {
|
||||
Ok(texture) => {
|
||||
ui.file_thumbs.store_texture(&key, texture.clone());
|
||||
if let Some(pictures) =
|
||||
ui.file_thumbs.wanted.borrow_mut().remove(&key)
|
||||
{
|
||||
for picture in pictures {
|
||||
paint(&picture, &key, &texture);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::debug!("cannot decode file thumbnail {path}: {e}");
|
||||
ui.file_thumbs.store_missing(key.clone());
|
||||
ui.file_thumbs.wanted.borrow_mut().remove(&key);
|
||||
}
|
||||
}
|
||||
}
|
||||
ReplyDisposition::Retry => pending.push(request),
|
||||
ReplyDisposition::Missing => {
|
||||
ui.file_thumbs.store_missing(key.clone());
|
||||
ui.file_thumbs.wanted.borrow_mut().remove(&key);
|
||||
}
|
||||
}
|
||||
}
|
||||
pending.extend(requested.into_values());
|
||||
if !pending.is_empty() {
|
||||
retry_file_thumbs(&ui, pending);
|
||||
}
|
||||
}
|
||||
Ok(Ok(Response::FileThumbsStale)) => {
|
||||
ui.file_thumbs.generation.set(0);
|
||||
retry_file_thumbs(&ui, items);
|
||||
}
|
||||
Ok(Ok(Response::Error { message, .. })) => {
|
||||
tracing::debug!("file thumbnails failed: {message}");
|
||||
retry_file_thumbs(&ui, items);
|
||||
}
|
||||
Ok(Ok(_)) | Ok(Err(_)) | Err(_) => {
|
||||
tracing::debug!("file thumbnails: no reply");
|
||||
retry_file_thumbs(&ui, items);
|
||||
}
|
||||
}
|
||||
schedule_file_thumbs(&ui);
|
||||
});
|
||||
}
|
||||
|
||||
fn retry_file_thumbs(ui: &Rc<Ui>, items: Vec<FileThumbRequest>) {
|
||||
let ui = ui.clone();
|
||||
glib::timeout_add_local_once(THUMB_RETRY, move || {
|
||||
let wanted = ui.file_thumbs.wanted.borrow();
|
||||
let retry: Vec<FileThumbRequest> = items
|
||||
.into_iter()
|
||||
.filter(|item| wanted.contains_key(&thumbnail_key(&item.uid, item.modified)))
|
||||
.collect();
|
||||
drop(wanted);
|
||||
ui.file_thumbs.queue.borrow_mut().extend(retry);
|
||||
schedule_file_thumbs(&ui);
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{ReplyDisposition, is_image_name, reply_disposition};
|
||||
|
||||
#[test]
|
||||
fn image_extensions_are_case_insensitive_and_specific() {
|
||||
for name in [
|
||||
"photo.jpg",
|
||||
"SCAN.PNG",
|
||||
"animation.Gif",
|
||||
"archive.tiff",
|
||||
"camera.nef",
|
||||
"negative.CR3",
|
||||
"camera.RAW",
|
||||
] {
|
||||
assert!(is_image_name(name), "{name}");
|
||||
}
|
||||
for name in [
|
||||
"notes.txt",
|
||||
"movie.mp4",
|
||||
"image.jpg.zip",
|
||||
"jpg",
|
||||
"camera.heic",
|
||||
"frame.avif",
|
||||
"graphic.svg",
|
||||
"camera.rw",
|
||||
] {
|
||||
assert!(!is_image_name(name), "{name}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cancelled_listing_replies_cannot_become_permanent_misses() {
|
||||
assert_eq!(
|
||||
reply_disposition(false, false, false),
|
||||
ReplyDisposition::Ignore
|
||||
);
|
||||
assert_eq!(
|
||||
reply_disposition(true, false, false),
|
||||
ReplyDisposition::Missing
|
||||
);
|
||||
}
|
||||
}
|
||||
+15
-172
@@ -1,26 +1,23 @@
|
||||
//! `pdfs-prompt --dmenu`: the same search, presented in the user's own launcher.
|
||||
//!
|
||||
//! Everything here is blocking and GTK-free — the process exists for one
|
||||
//! round-trip and then exits, so there is no main loop to keep off. It reuses
|
||||
//! the GTK prompt's [`Hit`](crate::Hit) model and ranking, and its Drive
|
||||
//! activation policy, so the two front ends cannot drift on what "open" means.
|
||||
//! round-trip and then exits, so there is no main loop to keep off. Talking to
|
||||
//! the daemon and opening a chosen file are [`crate::query`]'s, shared with the
|
||||
//! fzf front end and built on the GTK prompt's own [`Hit`](crate::Hit) model
|
||||
//! and activation policy; this module is only presentation.
|
||||
//!
|
||||
//! Launchers filter a static list; they cannot call back for a new one per
|
||||
//! keystroke. So the interaction is a loop: the first menu shows pinned files
|
||||
//! and accepts free text, and any text that matches no entry becomes the next
|
||||
//! search. Escape (an empty selection) ends it.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
//! search. Escape (an empty selection) ends it. `--fzf` ([`crate::fzf`]) is the
|
||||
//! front end for users who want results to appear while they type instead.
|
||||
|
||||
use pdfs_core::config::AppDirs;
|
||||
use pdfs_core::control::{
|
||||
Request, Response, SearchFilters, SearchHit, SearchKind, SearchSource, send,
|
||||
};
|
||||
use pdfs_core::menu::{self, MenuChoice, MenuItem, PromptConfig};
|
||||
use pdfs_core::opener::{self, OpenWith};
|
||||
use pdfs_core::opener::OpenWith;
|
||||
|
||||
use crate::activation::{DriveActivation, drive_activation, mounted_or_relative, mounted_target};
|
||||
use crate::{Hit, file_name, is_document, is_image, is_media, rank_hits};
|
||||
use crate::Hit;
|
||||
use crate::query;
|
||||
|
||||
/// Guard against a launcher that keeps handing back text we cannot satisfy —
|
||||
/// without it a scripted (non-interactive) menu could spin forever.
|
||||
@@ -48,10 +45,7 @@ pub(crate) fn run(options: Options) -> Result<(), String> {
|
||||
.to_string()
|
||||
})?;
|
||||
|
||||
let mountpoint = match request(&socket, Request::Status)? {
|
||||
Response::Status { mountpoint, .. } => PathBuf::from(mountpoint),
|
||||
_ => dirs.default_mountpoint(),
|
||||
};
|
||||
let mountpoint = query::mountpoint(&socket, &dirs);
|
||||
|
||||
let mut query = options
|
||||
.query
|
||||
@@ -59,8 +53,8 @@ pub(crate) fn run(options: Options) -> Result<(), String> {
|
||||
.filter(|q| !q.is_empty());
|
||||
for _ in 0..MAX_ROUNDS {
|
||||
let hits = match &query {
|
||||
Some(text) => search(&socket, text, prompt.resolved_menu_limit())?,
|
||||
None => pins(&socket)?,
|
||||
Some(text) => query::search(&socket, text, prompt.resolved_menu_limit())?,
|
||||
None => query::pins(&socket)?,
|
||||
};
|
||||
let items: Vec<MenuItem> = if hits.is_empty() {
|
||||
vec![MenuItem::new(placeholder(query.as_deref()), None)]
|
||||
@@ -77,7 +71,7 @@ pub(crate) fn run(options: Options) -> Result<(), String> {
|
||||
query = None;
|
||||
continue;
|
||||
};
|
||||
return open(&socket, &mountpoint, &policy, hit, query.is_none());
|
||||
return query::open(&socket, &mountpoint, &policy, hit, query.is_none());
|
||||
}
|
||||
// No entry matched what was typed: treat it as the next query.
|
||||
MenuChoice::Custom(text) => query = Some(text),
|
||||
@@ -118,57 +112,6 @@ fn placeholder(query: Option<&str>) -> String {
|
||||
}
|
||||
}
|
||||
|
||||
/// Pinned files — the same "you asked to keep these" set the GTK prompt opens
|
||||
/// with, and the only listing the daemon offers without a query.
|
||||
fn pins(socket: &Path) -> Result<Vec<Hit>, String> {
|
||||
let Response::Pins { pins } = request(socket, Request::ListPins)? else {
|
||||
return Ok(Vec::new());
|
||||
};
|
||||
Ok(pins
|
||||
.into_iter()
|
||||
.map(|pin| {
|
||||
Hit::Drive(SearchHit {
|
||||
name: file_name(&pin.path),
|
||||
path: pin.path,
|
||||
is_dir: pin.is_dir.unwrap_or(pin.recursive),
|
||||
size: 0,
|
||||
modified: 0,
|
||||
pinned: true,
|
||||
uid: pin.uid,
|
||||
mounted_path: None,
|
||||
score: 0,
|
||||
})
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
fn search(socket: &Path, query: &str, limit: usize) -> Result<Vec<Hit>, String> {
|
||||
let reply = request(
|
||||
socket,
|
||||
Request::SearchV2 {
|
||||
query: query.to_string(),
|
||||
limit,
|
||||
filters: SearchFilters {
|
||||
sources: vec![SearchSource::Drive, SearchSource::Local],
|
||||
kind: SearchKind::All,
|
||||
},
|
||||
},
|
||||
)?;
|
||||
let Response::SearchResultsV2 {
|
||||
drive_hits,
|
||||
local_hits,
|
||||
..
|
||||
} = reply
|
||||
else {
|
||||
return Ok(Vec::new());
|
||||
};
|
||||
let mut hits: Vec<Hit> = drive_hits.into_iter().map(Hit::Drive).collect();
|
||||
hits.extend(local_hits.into_iter().map(Hit::Local));
|
||||
rank_hits(&mut hits);
|
||||
hits.truncate(limit);
|
||||
Ok(hits)
|
||||
}
|
||||
|
||||
/// One launcher line per hit. Labels must be unique: they are how the selection
|
||||
/// is matched back, and two files of the same name in the same folder listing
|
||||
/// would otherwise be indistinguishable.
|
||||
@@ -181,101 +124,15 @@ fn label_all(hits: &[Hit]) -> Vec<MenuItem> {
|
||||
label = format!("{} · {} ({suffix})", hit.name(), hit.location());
|
||||
suffix += 1;
|
||||
}
|
||||
items.push(MenuItem::new(label, Some(icon_name(hit))));
|
||||
items.push(MenuItem::new(label, Some(query::icon_name(hit))));
|
||||
}
|
||||
items
|
||||
}
|
||||
|
||||
/// Icon theme name for a hit.
|
||||
///
|
||||
/// Coarse on purpose: `content_type_guess` would give an exact MIME name, but
|
||||
/// only with file *contents* to sniff — a Drive hit is metadata, and passing it
|
||||
/// an empty buffer answers `application-x-zerosize` for everything. The generic
|
||||
/// names below exist in every icon theme.
|
||||
fn icon_name(hit: &Hit) -> String {
|
||||
if hit.is_dir() {
|
||||
return "folder".to_string();
|
||||
}
|
||||
let name = hit.name();
|
||||
if is_image(name) {
|
||||
"image-x-generic"
|
||||
} else if is_media(name) {
|
||||
"video-x-generic"
|
||||
} else if is_document(name) {
|
||||
"text-x-generic"
|
||||
} else {
|
||||
"application-x-generic"
|
||||
}
|
||||
.to_string()
|
||||
}
|
||||
|
||||
/// Open a chosen hit, mirroring the GTK prompt: local files are already on
|
||||
/// disk, a Drive folder or streamable file goes through the mount, and anything
|
||||
/// else is materialised by the daemon first.
|
||||
fn open(
|
||||
socket: &Path,
|
||||
mountpoint: &Path,
|
||||
policy: &OpenWith,
|
||||
hit: &Hit,
|
||||
from_pins: bool,
|
||||
) -> Result<(), String> {
|
||||
match hit {
|
||||
Hit::Local(local) => {
|
||||
opener::open(policy, Path::new(&local.path), local.is_dir);
|
||||
Ok(())
|
||||
}
|
||||
Hit::Drive(drive) => {
|
||||
// A folder has no materialised form, and a pin row's `is_dir` is pin
|
||||
// policy rather than node kind, so both go to the mount unchecked.
|
||||
let unconditional = from_pins
|
||||
|| matches!(
|
||||
drive_activation(&drive.name, drive.is_dir),
|
||||
DriveActivation::Folder | DriveActivation::MountedMedia
|
||||
);
|
||||
if unconditional {
|
||||
opener::open(
|
||||
policy,
|
||||
&mounted_or_relative(mountpoint, drive),
|
||||
drive.is_dir,
|
||||
);
|
||||
return Ok(());
|
||||
}
|
||||
// Anything else the mount exposes opens there too, so an editor
|
||||
// saves back into Drive instead of into a cache blob.
|
||||
if let Some(path) = mounted_target(mountpoint, drive) {
|
||||
opener::open(policy, &path, false);
|
||||
return Ok(());
|
||||
}
|
||||
match request(
|
||||
socket,
|
||||
Request::OpenFile {
|
||||
path: drive.path.clone(),
|
||||
uid: Some(drive.uid.clone()),
|
||||
},
|
||||
)? {
|
||||
Response::FilePath { path } => {
|
||||
// The reply is a content-hash blob; the rules key off the
|
||||
// Drive name, which is what the user actually opened.
|
||||
opener::open_named(policy, Path::new(&path), &drive.name, false);
|
||||
Ok(())
|
||||
}
|
||||
Response::Error { message, .. } => Err(format!("could not open: {message}")),
|
||||
_ => Err("unexpected reply from the daemon".to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One control-socket round-trip, with the daemon-is-down case phrased for a
|
||||
/// terminal rather than a status page.
|
||||
fn request(socket: &Path, request: Request) -> Result<Response, String> {
|
||||
send(socket, &request).map_err(|e| format!("cannot reach the Proton Drive daemon: {e}"))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use pdfs_core::control::LocalHit;
|
||||
use pdfs_core::control::{LocalHit, SearchHit};
|
||||
|
||||
fn drive(name: &str, path: &str, is_dir: bool) -> Hit {
|
||||
Hit::Drive(SearchHit {
|
||||
@@ -331,18 +188,4 @@ mod tests {
|
||||
assert!(placeholder(None).contains("type to search"));
|
||||
assert!(placeholder(Some("md")).contains("md"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn icons_are_theme_names_the_launcher_can_look_up() {
|
||||
assert_eq!(icon_name(&drive("Photos", "Photos", true)), "folder");
|
||||
assert_eq!(
|
||||
icon_name(&drive("a.png", "a.png", false)),
|
||||
"image-x-generic"
|
||||
);
|
||||
assert_eq!(icon_name(&drive("a.md", "a.md", false)), "text-x-generic");
|
||||
assert_eq!(
|
||||
icon_name(&drive("a.bin", "a.bin", false)),
|
||||
"application-x-generic"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,372 @@
|
||||
//! `pdfs-prompt --fzf`: launcher search that queries the daemon per keystroke.
|
||||
//!
|
||||
//! A dmenu-protocol launcher (fuzzel, rofi, wofi, …) is handed one static list
|
||||
//! and filters it itself; it cannot ask for a new one as the user types, which
|
||||
//! is why [`crate::dmenu`] has to make the search a two-step. `fzf` can:
|
||||
//! `--bind change:reload(…)` re-runs a command on every keystroke. So this
|
||||
//! front end re-executes *this binary* in `--feed` mode, and the daemon's own
|
||||
//! ranking ([`pdfs_core::control::Request::SearchV2`]) decides the order —
|
||||
//! `--disabled` turns fzf's matcher off so it does not re-sort what came back.
|
||||
//!
|
||||
//! Three roles, all the same binary:
|
||||
//!
|
||||
//! * **outer** — `--fzf` with no tty (a WM keybinding): spawn a terminal that
|
||||
//! runs the inner role, and exit.
|
||||
//! * **inner** — `--fzf` with a tty: run fzf, open what was chosen.
|
||||
//! * **feed** — `--feed <TEXT>`: print one line per hit and exit. fzf spawns
|
||||
//! this per keystroke and kills the one it superseded.
|
||||
//!
|
||||
//! A line is `display\tpayload`, shown through `--with-nth=1` so only the left
|
||||
//! half is visible. The payload is the JSON [`Hit`], which is what makes the
|
||||
//! selection unambiguous: unlike the dmenu path, identity does not have to be
|
||||
//! recovered from the label, so [`crate::dmenu::label_all`]'s uniquifying
|
||||
//! suffix has no counterpart here. JSON escapes tabs and newlines, so no file
|
||||
//! name can split a line.
|
||||
|
||||
use std::io::{BufRead, BufReader, IsTerminal, Write};
|
||||
use std::path::PathBuf;
|
||||
use std::process::{Command, Stdio};
|
||||
|
||||
use pdfs_core::config::AppDirs;
|
||||
use pdfs_core::menu::{self, PromptConfig};
|
||||
use pdfs_core::opener::OpenWith;
|
||||
|
||||
use crate::query;
|
||||
use crate::{Hit, SEARCH_DEBOUNCE};
|
||||
|
||||
/// Separates the visible half of a feed line from its payload.
|
||||
const FIELD: char = '\t';
|
||||
|
||||
/// What the launcher prompt reads, matching the dmenu front end's wording.
|
||||
const PROMPT: &str = "Search Drive › ";
|
||||
|
||||
pub(crate) struct Options {
|
||||
/// Search for this immediately instead of opening on the pinned files.
|
||||
pub query: Option<String>,
|
||||
/// Set by the terminal we spawned: run fzf here even without a tty rather
|
||||
/// than spawning a second terminal, which would recurse forever if the
|
||||
/// terminal does not give its child one.
|
||||
pub inner: bool,
|
||||
}
|
||||
|
||||
/// Run the fzf flow. Returns an error only for conditions the user must act on
|
||||
/// (no fzf, no terminal, no daemon); a plain cancel is `Ok`.
|
||||
pub(crate) fn run(options: Options) -> Result<(), String> {
|
||||
if options.inner || std::io::stdin().is_terminal() {
|
||||
return interactive(options.query);
|
||||
}
|
||||
spawn_terminal(options.query)
|
||||
}
|
||||
|
||||
/// Re-launch this binary's inner role inside a terminal.
|
||||
///
|
||||
/// Not waited on: a keybinding should return immediately, and the terminal
|
||||
/// owns the rest of the interaction — including opening the chosen file, which
|
||||
/// `opener` detaches anyway.
|
||||
fn spawn_terminal(query: Option<String>) -> Result<(), String> {
|
||||
let dirs = AppDirs::new().map_err(|e| format!("cannot resolve app dirs: {e}"))?;
|
||||
let prompt: PromptConfig = dirs.load_config().resolved_prompt();
|
||||
let terminal = menu::resolve_terminal(prompt.terminal.as_ref()).ok_or_else(|| {
|
||||
"no terminal found — install foot/kitty/alacritty, or set prompt.terminal in config.json"
|
||||
.to_string()
|
||||
})?;
|
||||
|
||||
let argv = menu::with_command(&terminal, &inner_command(self_exe()?, query.as_deref()));
|
||||
let (program, args) = argv
|
||||
.split_first()
|
||||
.ok_or_else(|| "no terminal command configured".to_string())?;
|
||||
Command::new(program)
|
||||
.args(args)
|
||||
.spawn()
|
||||
.map_err(|e| format!("cannot run terminal {program}: {e}"))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The argv the spawned terminal runs.
|
||||
fn inner_command(exe: PathBuf, query: Option<&str>) -> Vec<String> {
|
||||
let mut argv = vec![
|
||||
exe.display().to_string(),
|
||||
"--fzf".to_string(),
|
||||
"--inner".to_string(),
|
||||
];
|
||||
if let Some(text) = query.filter(|text| !text.trim().is_empty()) {
|
||||
argv.push("--query".to_string());
|
||||
argv.push(text.to_string());
|
||||
}
|
||||
argv
|
||||
}
|
||||
|
||||
/// Run fzf here and act on the choice.
|
||||
fn interactive(query: Option<String>) -> Result<(), String> {
|
||||
if !menu::on_path("fzf") {
|
||||
return Err("fzf is not installed — install it, or use --dmenu".to_string());
|
||||
}
|
||||
let dirs = AppDirs::new().map_err(|e| format!("cannot resolve app dirs: {e}"))?;
|
||||
let config = dirs.load_config();
|
||||
let prompt: PromptConfig = config.resolved_prompt();
|
||||
let policy: OpenWith = config.resolved_open_with();
|
||||
let socket = dirs.control_socket();
|
||||
|
||||
let query = query
|
||||
.map(|q| q.trim().to_string())
|
||||
.filter(|q| !q.is_empty());
|
||||
// fzf only fires `change` for a *keystroke*, so the first screen — pins, or
|
||||
// the results for an initial --query — has to arrive on stdin.
|
||||
let initial = hits(&socket, query.as_deref(), prompt.resolved_menu_limit());
|
||||
|
||||
let argv = fzf_argv(&self_exe()?, query.as_deref());
|
||||
let (program, args) = argv.split_first().expect("fzf_argv is never empty");
|
||||
let mut child = Command::new(program)
|
||||
.args(args)
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(Stdio::piped())
|
||||
.spawn()
|
||||
.map_err(|e| format!("cannot run fzf: {e}"))?;
|
||||
|
||||
// Written on a worker for the same reason the dmenu path does it: fzf reads
|
||||
// stdin only once its UI is up, and a full pipe buffer would deadlock us.
|
||||
let payload: String = initial.iter().map(encode_line).collect();
|
||||
let mut stdin = child
|
||||
.stdin
|
||||
.take()
|
||||
.ok_or_else(|| "fzf stdin unavailable".to_string())?;
|
||||
let writer = std::thread::spawn(move || {
|
||||
let _ = stdin.write_all(payload.as_bytes());
|
||||
let _ = stdin.flush();
|
||||
});
|
||||
|
||||
let mut selected = String::new();
|
||||
if let Some(stdout) = child.stdout.take() {
|
||||
let mut reader = BufReader::new(stdout);
|
||||
let mut line = String::new();
|
||||
while reader.read_line(&mut line).unwrap_or(0) > 0 {
|
||||
if selected.is_empty() {
|
||||
selected = line.trim_end_matches('\n').to_string();
|
||||
}
|
||||
line.clear();
|
||||
}
|
||||
}
|
||||
let _ = child.wait();
|
||||
let _ = writer.join();
|
||||
|
||||
// Escape, or Enter on the "cannot reach the daemon" row, which carries no
|
||||
// payload — neither is an error.
|
||||
let Some(hit) = decode_line(&selected) else {
|
||||
return Ok(());
|
||||
};
|
||||
let mountpoint = query::mountpoint(&socket, &dirs);
|
||||
// Only the no-query screen is the pin list, and only there does a row's
|
||||
// `is_dir` mean pin policy rather than node kind.
|
||||
query::open(&socket, &mountpoint, &policy, &hit, query.is_none())
|
||||
}
|
||||
|
||||
/// `--feed`: one round-trip, printed for fzf to swallow.
|
||||
///
|
||||
/// Always succeeds. A reload command that exits non-zero leaves fzf showing an
|
||||
/// empty list with no explanation, so a failure becomes a visible row instead.
|
||||
pub(crate) fn feed(query: Option<&str>) {
|
||||
let Ok(dirs) = AppDirs::new() else {
|
||||
println!("(cannot resolve app dirs)");
|
||||
return;
|
||||
};
|
||||
let limit = dirs.load_config().resolved_prompt().resolved_menu_limit();
|
||||
let socket = dirs.control_socket();
|
||||
let query = query.map(str::trim).filter(|text| !text.is_empty());
|
||||
|
||||
let stdout = std::io::stdout();
|
||||
let mut out = stdout.lock();
|
||||
for line in hits(&socket, query, limit).iter().map(encode_line) {
|
||||
let _ = out.write_all(line.as_bytes());
|
||||
}
|
||||
}
|
||||
|
||||
/// The rows for one screen: pins when there is nothing typed, search results
|
||||
/// otherwise — the same split the GTK prompt makes on an empty entry.
|
||||
fn hits(socket: &std::path::Path, query: Option<&str>, limit: usize) -> Vec<Hit> {
|
||||
let result = match query {
|
||||
Some(text) => query::search(socket, text, limit),
|
||||
None => query::pins(socket),
|
||||
};
|
||||
match result {
|
||||
Ok(hits) => hits,
|
||||
Err(message) => {
|
||||
// Printed as a row rather than to stderr: fzf's alternate screen
|
||||
// hides stderr, so this is the only place the user would see it.
|
||||
println!("({message})");
|
||||
Vec::new()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn fzf_argv(exe: &std::path::Path, query: Option<&str>) -> Vec<String> {
|
||||
let feed = format!("{} --feed {{q}}", shell_quote(&exe.display().to_string()));
|
||||
// The sleep is the debounce: fzf kills a reload child that a later keystroke
|
||||
// superseded, so a burst of typing costs one daemon round-trip, exactly as
|
||||
// the GTK prompt's timeout does.
|
||||
let debounce = SEARCH_DEBOUNCE.as_secs_f32();
|
||||
let mut argv: Vec<String> = [
|
||||
"fzf",
|
||||
// fzf's own fuzzy matcher would re-order and re-filter what the daemon
|
||||
// already ranked, and would hide rows it does not think match.
|
||||
"--disabled",
|
||||
"--no-sort",
|
||||
"--no-multi",
|
||||
"--layout=reverse",
|
||||
"--info=inline",
|
||||
"--delimiter=\t",
|
||||
"--with-nth=1",
|
||||
]
|
||||
.iter()
|
||||
.map(|arg| (*arg).to_string())
|
||||
.collect();
|
||||
argv.push(format!("--prompt={PROMPT}"));
|
||||
// `reload:` rather than `reload(…)`: the unparenthesised form runs to the
|
||||
// end of the argument, so an install path containing a bracket cannot
|
||||
// truncate the command fzf parses out of it.
|
||||
argv.push(format!("--bind=change:reload:sleep {debounce}; {feed}"));
|
||||
if let Some(text) = query {
|
||||
argv.push(format!("--query={text}"));
|
||||
}
|
||||
argv
|
||||
}
|
||||
|
||||
/// `display\tpayload\n`. The display half is scrubbed of the delimiter and of
|
||||
/// newlines; the payload is JSON, which escapes both.
|
||||
fn encode_line(hit: &Hit) -> String {
|
||||
let display =
|
||||
format!("{} · {}", hit.name(), hit.location()).replace(['\n', '\r', FIELD], " ");
|
||||
let payload = serde_json::to_string(hit).unwrap_or_default();
|
||||
format!("{display}{FIELD}{payload}\n")
|
||||
}
|
||||
|
||||
/// The hit a selected line stands for, or `None` for a line that carries no
|
||||
/// payload — the error/empty row, or an empty selection (Escape).
|
||||
fn decode_line(line: &str) -> Option<Hit> {
|
||||
let (_, payload) = line.trim_end_matches('\n').rsplit_once(FIELD)?;
|
||||
serde_json::from_str(payload).ok()
|
||||
}
|
||||
|
||||
fn self_exe() -> Result<PathBuf, String> {
|
||||
std::env::current_exe().map_err(|e| format!("cannot locate pdfs-prompt: {e}"))
|
||||
}
|
||||
|
||||
/// POSIX single-quoting, for the one string that goes through fzf's `sh -c`.
|
||||
fn shell_quote(text: &str) -> String {
|
||||
format!("'{}'", text.replace('\'', r"'\''"))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use pdfs_core::control::{LocalHit, SearchHit};
|
||||
|
||||
fn drive(name: &str, path: &str) -> Hit {
|
||||
Hit::Drive(SearchHit {
|
||||
name: name.into(),
|
||||
path: path.into(),
|
||||
is_dir: false,
|
||||
size: 7,
|
||||
modified: 9,
|
||||
pinned: false,
|
||||
uid: "uid~1".into(),
|
||||
mounted_path: None,
|
||||
score: 42,
|
||||
})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_line_round_trips_through_the_payload() {
|
||||
let hit = drive("notes.md", "Docs/notes.md");
|
||||
let line = encode_line(&hit);
|
||||
assert!(line.ends_with('\n'));
|
||||
let back = decode_line(&line).expect("payload decodes");
|
||||
assert_eq!(back.name(), "notes.md");
|
||||
assert_eq!(back.key(), hit.key());
|
||||
assert_eq!(back.score(), 42);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_local_hit_round_trips_as_itself() {
|
||||
let hit = Hit::Local(LocalHit {
|
||||
name: "notes.md".into(),
|
||||
path: "/home/me/Docs/notes.md".into(),
|
||||
is_dir: false,
|
||||
size: 1,
|
||||
modified: 2,
|
||||
score: 3,
|
||||
});
|
||||
let back = decode_line(&encode_line(&hit)).expect("payload decodes");
|
||||
assert!(matches!(back, Hit::Local(_)));
|
||||
assert_eq!(back.key(), hit.key());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_name_with_a_tab_or_newline_stays_one_line() {
|
||||
let hit = drive("two\tparts\nfile.md", "Docs/two\tparts\nfile.md");
|
||||
let line = encode_line(&hit);
|
||||
assert_eq!(line.matches('\n').count(), 1, "{line:?}");
|
||||
assert_eq!(line.matches(FIELD).count(), 1, "{line:?}");
|
||||
assert_eq!(
|
||||
decode_line(&line).expect("payload decodes").name(),
|
||||
"two\tparts\nfile.md"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_row_without_a_payload_is_not_a_hit() {
|
||||
assert!(decode_line("").is_none());
|
||||
assert!(decode_line("(cannot reach the Proton Drive daemon)").is_none());
|
||||
assert!(decode_line("bogus\tnot json").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fzf_is_told_not_to_do_its_own_matching() {
|
||||
let argv = fzf_argv(std::path::Path::new("/usr/bin/pdfs-prompt"), None);
|
||||
assert!(argv.contains(&"--disabled".to_string()), "{argv:?}");
|
||||
assert!(argv.contains(&"--no-sort".to_string()), "{argv:?}");
|
||||
assert!(
|
||||
argv.iter().any(|arg| arg.starts_with("--with-nth=")),
|
||||
"the payload field must stay hidden: {argv:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_keystroke_re_runs_the_feed_after_the_debounce() {
|
||||
let argv = fzf_argv(std::path::Path::new("/usr/bin/pdfs-prompt"), Some("md"));
|
||||
let bind = argv
|
||||
.iter()
|
||||
.find(|arg| arg.starts_with("--bind=change:"))
|
||||
.expect("a change binding");
|
||||
assert!(bind.contains("reload"), "{bind}");
|
||||
assert!(bind.contains("--feed {q}"), "{bind}");
|
||||
assert!(bind.contains(&format!("sleep {}", SEARCH_DEBOUNCE.as_secs_f32())));
|
||||
assert!(argv.contains(&"--query=md".to_string()), "{argv:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_path_with_a_quote_cannot_break_out_of_the_reload_command() {
|
||||
let argv = fzf_argv(std::path::Path::new("/home/o'brien/my prompt"), None);
|
||||
let bind = argv
|
||||
.iter()
|
||||
.find(|arg| arg.starts_with("--bind=change:"))
|
||||
.expect("a change binding");
|
||||
assert!(bind.contains(r"'/home/o'\''brien/my prompt'"), "{bind}");
|
||||
// The unparenthesised reload form: a bracket in the path must not end
|
||||
// the command as far as fzf's own parser is concerned.
|
||||
assert!(bind.starts_with("--bind=change:reload:"), "{bind}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_terminal_runs_the_inner_role_so_it_cannot_recurse() {
|
||||
let argv = inner_command(PathBuf::from("/usr/bin/pdfs-prompt"), Some("md"));
|
||||
assert_eq!(
|
||||
argv,
|
||||
vec!["/usr/bin/pdfs-prompt", "--fzf", "--inner", "--query", "md"]
|
||||
);
|
||||
assert_eq!(
|
||||
inner_command(PathBuf::from("/usr/bin/pdfs-prompt"), Some(" ")),
|
||||
vec!["/usr/bin/pdfs-prompt", "--fzf", "--inner"],
|
||||
"a blank query is no query"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -14,8 +14,9 @@ use std::process::Command;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use ksni::blocking::TrayMethods;
|
||||
use ksni::menu::StandardItem;
|
||||
use ksni::{MenuItem, Tray, TrayService};
|
||||
use ksni::{MenuItem, Tray};
|
||||
use pdfs_core::auth;
|
||||
use pdfs_core::config::AppDirs;
|
||||
use pdfs_core::control::{
|
||||
@@ -282,13 +283,13 @@ fn main() -> Result<()> {
|
||||
state: poll_state(&socket, &default_mountpoint),
|
||||
};
|
||||
|
||||
let service = TrayService::new(tray);
|
||||
let handle = service.handle();
|
||||
std::thread::spawn(move || {
|
||||
if let Err(e) = service.run() {
|
||||
tracing::error!("tray service stopped: {e:?}");
|
||||
}
|
||||
});
|
||||
// The tray autostarts with the session and can beat the desktop's SNI
|
||||
// watcher onto the bus. Without `assume_sni_available` that race is a hard
|
||||
// spawn error; with it the item waits and registers once the watcher shows.
|
||||
let handle = tray
|
||||
.assume_sni_available(true)
|
||||
.spawn()
|
||||
.context("start tray service")?;
|
||||
|
||||
// Poll the daemon forever, pushing each fresh snapshot into the tray so the
|
||||
// menu reflects mount/login changes made elsewhere (e.g. via the CLI).
|
||||
|
||||
@@ -24,6 +24,8 @@ use pdfs_core::opener::{self, OpenWith};
|
||||
|
||||
mod activation;
|
||||
mod dmenu;
|
||||
mod fzf;
|
||||
mod query;
|
||||
use activation::{DriveActivation, drive_activation, mounted_or_relative, mounted_target};
|
||||
|
||||
const APP_ID: &str = "io.narl.proton-drive-linux-prompt";
|
||||
@@ -125,7 +127,7 @@ fn is_media(name: &str) -> bool {
|
||||
/// One row in the unified result list. The two sections hold different payloads
|
||||
/// — a Drive hit must be hydrated through the daemon before it can be opened, a
|
||||
/// local file is already on disk — but they share one keyboard cursor.
|
||||
#[derive(Clone)]
|
||||
#[derive(Clone, serde::Serialize, serde::Deserialize)]
|
||||
enum Hit {
|
||||
Drive(SearchHit),
|
||||
Local(LocalHit),
|
||||
@@ -341,12 +343,20 @@ Options:
|
||||
rofi, …) instead of the built-in GTK window. Also
|
||||
settable permanently as \"prompt\": { \"mode\": \"dmenu\" }
|
||||
in config.json.
|
||||
--fzf Search in fzf, in a terminal, re-querying the daemon on
|
||||
every keystroke — results appear as you type, unlike
|
||||
--dmenu. Also settable as \"prompt\": { \"mode\": \"fzf\" }.
|
||||
--gtk Force the built-in window, overriding that setting.
|
||||
--menu <COMMAND> Launcher command line for --dmenu, e.g.
|
||||
--menu 'fuzzel --dmenu --width 60'. Overrides
|
||||
\"prompt\": { \"menu\": [...] }.
|
||||
--query <TEXT> Search for TEXT immediately instead of opening on the
|
||||
pinned-files list (--dmenu only).
|
||||
pinned-files list (--dmenu and --fzf only).
|
||||
--feed <TEXT> Print the hits for TEXT, one per line, and exit. This is
|
||||
what --fzf re-runs on each keystroke; it is not meant to
|
||||
be typed.
|
||||
--inner Run fzf here rather than spawning a terminal. Set by the
|
||||
terminal --fzf spawns; not meant to be typed.
|
||||
-h, --help Show this help.
|
||||
";
|
||||
|
||||
@@ -356,6 +366,12 @@ struct Args {
|
||||
mode: Option<PromptMode>,
|
||||
menu: Option<Vec<String>>,
|
||||
query: Option<String>,
|
||||
/// `--feed`: print hits for this text and exit. `Some(None)` is `--feed`
|
||||
/// with an empty argument, which fzf sends for an empty input — a real
|
||||
/// request for the pinned list, not an absent flag.
|
||||
feed: Option<Option<String>>,
|
||||
/// `--inner`: we are the process a spawned terminal is running.
|
||||
inner: bool,
|
||||
}
|
||||
|
||||
fn parse_args(argv: impl Iterator<Item = String>) -> Result<Args, String> {
|
||||
@@ -363,12 +379,19 @@ fn parse_args(argv: impl Iterator<Item = String>) -> Result<Args, String> {
|
||||
mode: None,
|
||||
menu: None,
|
||||
query: None,
|
||||
feed: None,
|
||||
inner: false,
|
||||
};
|
||||
let mut argv = argv.peekable();
|
||||
while let Some(arg) = argv.next() {
|
||||
match arg.as_str() {
|
||||
"--dmenu" => args.mode = Some(PromptMode::Dmenu),
|
||||
"--fzf" => args.mode = Some(PromptMode::Fzf),
|
||||
"--gtk" => args.mode = Some(PromptMode::Gtk),
|
||||
"--inner" => args.inner = true,
|
||||
// fzf passes an empty argument for an empty input, so a missing
|
||||
// value here is that case rather than a user error.
|
||||
"--feed" => args.feed = Some(argv.next().filter(|text| !text.trim().is_empty())),
|
||||
// A launcher command is one shell-ish string so it can live in a
|
||||
// keybinding; splitting on whitespace is enough for flags, and a
|
||||
// launcher argument needing spaces belongs in config.json.
|
||||
@@ -416,16 +439,31 @@ fn main() -> glib::ExitCode {
|
||||
}
|
||||
};
|
||||
|
||||
// A feed is fzf's own reload child, not a front end: it must print hits and
|
||||
// nothing else, so it is answered before the stored mode is even read.
|
||||
if let Some(query) = args.feed {
|
||||
fzf::feed(query.as_deref());
|
||||
return glib::ExitCode::SUCCESS;
|
||||
}
|
||||
|
||||
// The stored mode makes an existing keybinding switch front ends without
|
||||
// being re-bound; an explicit flag still wins.
|
||||
let stored = AppDirs::new()
|
||||
.map(|dirs| dirs.load_config().resolved_prompt().resolved_mode())
|
||||
.unwrap_or_default();
|
||||
if args.mode.unwrap_or(stored) == PromptMode::Dmenu {
|
||||
return match dmenu::run(dmenu::Options {
|
||||
let launcher = match args.mode.unwrap_or(stored) {
|
||||
PromptMode::Dmenu => Some(dmenu::run(dmenu::Options {
|
||||
menu: args.menu,
|
||||
query: args.query,
|
||||
}) {
|
||||
})),
|
||||
PromptMode::Fzf => Some(fzf::run(fzf::Options {
|
||||
query: args.query,
|
||||
inner: args.inner,
|
||||
})),
|
||||
PromptMode::Gtk => None,
|
||||
};
|
||||
if let Some(result) = launcher {
|
||||
return match result {
|
||||
Ok(()) => glib::ExitCode::SUCCESS,
|
||||
Err(message) => {
|
||||
eprintln!("pdfs-prompt: {message}");
|
||||
@@ -1326,7 +1364,7 @@ fn icon_for(name: &str, is_dir: bool) -> gio::Icon {
|
||||
if is_dir {
|
||||
return gio::ThemedIcon::new("folder").upcast();
|
||||
}
|
||||
let (content_type, _uncertain) = gio::functions::content_type_guess(Some(name), &[]);
|
||||
let (content_type, _uncertain) = gio::functions::content_type_guess(Some(name), None);
|
||||
gio::functions::content_type_get_icon(&content_type)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
//! The daemon side of both launcher front ends: ask for hits, open a chosen one.
|
||||
//!
|
||||
//! `--dmenu` and `--fzf` differ only in how they present a list and collect a
|
||||
//! choice. Everything between the control socket and the chosen file lives here
|
||||
//! so the two cannot drift on what a result set is or what "open" means — the
|
||||
//! same reason both already share [`Hit`] and [`rank_hits`] with the GTK prompt.
|
||||
//!
|
||||
//! Everything here is blocking and GTK-free: the callers are one-shot processes
|
||||
//! with no main loop to keep free.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use pdfs_core::config::AppDirs;
|
||||
use pdfs_core::control::{
|
||||
Request, Response, SearchFilters, SearchHit, SearchKind, SearchSource, send,
|
||||
};
|
||||
use pdfs_core::opener::{self, OpenWith};
|
||||
|
||||
use crate::activation::{DriveActivation, drive_activation, mounted_or_relative, mounted_target};
|
||||
use crate::{Hit, file_name, is_document, is_image, is_media, rank_hits};
|
||||
|
||||
/// Where the mount lives, for turning a Drive path into something an
|
||||
/// application can open. Falls back to the configured default so a momentarily
|
||||
/// unreachable daemon still produces a plausible path rather than an error.
|
||||
pub(crate) fn mountpoint(socket: &Path, dirs: &AppDirs) -> PathBuf {
|
||||
match request(socket, Request::Status) {
|
||||
Ok(Response::Status { mountpoint, .. }) => PathBuf::from(mountpoint),
|
||||
_ => dirs.default_mountpoint(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Pinned files — the same "you asked to keep these" set the GTK prompt opens
|
||||
/// with, and the only listing the daemon offers without a query.
|
||||
pub(crate) fn pins(socket: &Path) -> Result<Vec<Hit>, String> {
|
||||
let Response::Pins { pins } = request(socket, Request::ListPins)? else {
|
||||
return Ok(Vec::new());
|
||||
};
|
||||
Ok(pins
|
||||
.into_iter()
|
||||
.map(|pin| {
|
||||
Hit::Drive(SearchHit {
|
||||
name: file_name(&pin.path),
|
||||
path: pin.path,
|
||||
is_dir: pin.is_dir.unwrap_or(pin.recursive),
|
||||
size: 0,
|
||||
modified: 0,
|
||||
pinned: true,
|
||||
uid: pin.uid,
|
||||
mounted_path: None,
|
||||
score: 0,
|
||||
})
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
pub(crate) fn search(socket: &Path, query: &str, limit: usize) -> Result<Vec<Hit>, String> {
|
||||
let reply = request(
|
||||
socket,
|
||||
Request::SearchV2 {
|
||||
query: query.to_string(),
|
||||
limit,
|
||||
filters: SearchFilters {
|
||||
sources: vec![SearchSource::Drive, SearchSource::Local],
|
||||
kind: SearchKind::All,
|
||||
},
|
||||
},
|
||||
)?;
|
||||
let Response::SearchResultsV2 {
|
||||
drive_hits,
|
||||
local_hits,
|
||||
..
|
||||
} = reply
|
||||
else {
|
||||
return Ok(Vec::new());
|
||||
};
|
||||
let mut hits: Vec<Hit> = drive_hits.into_iter().map(Hit::Drive).collect();
|
||||
hits.extend(local_hits.into_iter().map(Hit::Local));
|
||||
rank_hits(&mut hits);
|
||||
hits.truncate(limit);
|
||||
Ok(hits)
|
||||
}
|
||||
|
||||
/// Icon theme name for a hit.
|
||||
///
|
||||
/// Coarse on purpose: `content_type_guess` would give an exact MIME name, but
|
||||
/// only with file *contents* to sniff — a Drive hit is metadata, and passing it
|
||||
/// an empty buffer answers `application-x-zerosize` for everything. The generic
|
||||
/// names below exist in every icon theme.
|
||||
pub(crate) fn icon_name(hit: &Hit) -> String {
|
||||
if hit.is_dir() {
|
||||
return "folder".to_string();
|
||||
}
|
||||
let name = hit.name();
|
||||
if is_image(name) {
|
||||
"image-x-generic"
|
||||
} else if is_media(name) {
|
||||
"video-x-generic"
|
||||
} else if is_document(name) {
|
||||
"text-x-generic"
|
||||
} else {
|
||||
"application-x-generic"
|
||||
}
|
||||
.to_string()
|
||||
}
|
||||
|
||||
/// Open a chosen hit, mirroring the GTK prompt: local files are already on
|
||||
/// disk, a Drive folder or streamable file goes through the mount, and anything
|
||||
/// else is materialised by the daemon first.
|
||||
pub(crate) fn open(
|
||||
socket: &Path,
|
||||
mountpoint: &Path,
|
||||
policy: &OpenWith,
|
||||
hit: &Hit,
|
||||
from_pins: bool,
|
||||
) -> Result<(), String> {
|
||||
match hit {
|
||||
Hit::Local(local) => {
|
||||
opener::open(policy, Path::new(&local.path), local.is_dir);
|
||||
Ok(())
|
||||
}
|
||||
Hit::Drive(drive) => {
|
||||
// A folder has no materialised form, and a pin row's `is_dir` is pin
|
||||
// policy rather than node kind, so both go to the mount unchecked.
|
||||
let unconditional = from_pins
|
||||
|| matches!(
|
||||
drive_activation(&drive.name, drive.is_dir),
|
||||
DriveActivation::Folder | DriveActivation::MountedMedia
|
||||
);
|
||||
if unconditional {
|
||||
opener::open(
|
||||
policy,
|
||||
&mounted_or_relative(mountpoint, drive),
|
||||
drive.is_dir,
|
||||
);
|
||||
return Ok(());
|
||||
}
|
||||
// Anything else the mount exposes opens there too, so an editor
|
||||
// saves back into Drive instead of into a cache blob.
|
||||
if let Some(path) = mounted_target(mountpoint, drive) {
|
||||
opener::open(policy, &path, false);
|
||||
return Ok(());
|
||||
}
|
||||
match request(
|
||||
socket,
|
||||
Request::OpenFile {
|
||||
path: drive.path.clone(),
|
||||
uid: Some(drive.uid.clone()),
|
||||
},
|
||||
)? {
|
||||
Response::FilePath { path } => {
|
||||
// The reply is a content-hash blob; the rules key off the
|
||||
// Drive name, which is what the user actually opened.
|
||||
opener::open_named(policy, Path::new(&path), &drive.name, false);
|
||||
Ok(())
|
||||
}
|
||||
Response::Error { message, .. } => Err(format!("could not open: {message}")),
|
||||
_ => Err("unexpected reply from the daemon".to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One control-socket round-trip, with the daemon-is-down case phrased for a
|
||||
/// terminal rather than a status page.
|
||||
pub(crate) fn request(socket: &Path, request: Request) -> Result<Response, String> {
|
||||
send(socket, &request).map_err(|e| format!("cannot reach the Proton Drive daemon: {e}"))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn drive(name: &str, is_dir: bool) -> Hit {
|
||||
Hit::Drive(SearchHit {
|
||||
name: name.into(),
|
||||
path: name.into(),
|
||||
is_dir,
|
||||
size: 0,
|
||||
modified: 0,
|
||||
pinned: false,
|
||||
uid: "uid".into(),
|
||||
mounted_path: None,
|
||||
score: 0,
|
||||
})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn icons_are_theme_names_the_launcher_can_look_up() {
|
||||
assert_eq!(icon_name(&drive("Photos", true)), "folder");
|
||||
assert_eq!(icon_name(&drive("a.png", false)), "image-x-generic");
|
||||
assert_eq!(icon_name(&drive("a.md", false)), "text-x-generic");
|
||||
assert_eq!(icon_name(&drive("a.bin", false)), "application-x-generic");
|
||||
}
|
||||
}
|
||||
@@ -153,8 +153,9 @@ Because Proton Drive does not support partial byte writes, modified files must b
|
||||
2. **Close/Release (`queue_revision`):** When the application closes the file descriptor, the daemon:
|
||||
- Fetches any untouched gaps from the remote base file to compile the full file.
|
||||
- Durably publishes the scratch data and authored-range sidecar into `staging` under a `{uid}-{millis}-{counter}` name. Temporary data and metadata are synced before atomic rename, and their directory is synced before the source is removed.
|
||||
- Transactionally queues or supersedes a pending database operation (`PendingOp`), so insertion failure cannot erase the previously acknowledged upload. For `OP_REVISION` ops, execution is debounced by 2 seconds (`DRAIN_REVISION_DEBOUNCE = 2s`) to give rapid follow-up writes (e.g. `aria2c` preallocation followed by writing) time to supersede the staged blob before network transmission.
|
||||
3. **Async Drain Thread (`run_pending_drain`):** The background drain worker picks up the database operations queue, handles revision uploads, resolves conflicts, and cleans up staging files. Upon landing a revision upload (`refresh_after_upload`), it rebaselines both still-queued ops (`rebaseline_pending`) and open write handles targeting the same node to prevent false self-conflict copies on subsequent writes.
|
||||
- Transactionally queues or supersedes a pending database operation (`PendingOp`), so insertion failure cannot erase the previously acknowledged upload. For `OP_REVISION` ops, execution is debounced to give rapid follow-up writes (e.g. `aria2c` preallocation followed by writing) time to supersede the staged blob before network transmission. The debounce is adaptive (`Core::revision_debounce`): it starts at `DRAIN_REVISION_DEBOUNCE = 2s` and, once a node has been uploaded, widens toward how long that upload actually took, bounded by `DRAIN_REVISION_DEBOUNCE_MAX = 60s`. A file saved faster than it can be sent therefore supersedes in the queue rather than mid-upload.
|
||||
- Signals `Core::cancel_upload` for the node before touching the queue. A drain worker may already be reading the blob this write supersedes; the flag is read by the upload's `CountingReader`, which refuses the SDK's next block. `queue_trash` and `discard_queued_ops` do the same, because they unlink the blob outright.
|
||||
3. **Async Drain Threads (`run_pending_drain`):** `DRAIN_WORKERS = 3` background workers share the operations queue. They pick work up through `Db::claim_next_due_op`, which marks the row `claimed_at` in the same transaction that selects it and excludes any op whose uid another worker already holds — ordering only has to hold *per node*, and that exclusion is what guarantees it. Worker 0 additionally runs the queue's idle chores (LRU touch flush, `recover_fsynced_writes`). A claim is process-local state: the single-writer `flock` means one found at open belongs to a crashed run, and `Db::clear_op_claims` drops the lot so those ops are not invisible forever. Each worker handles revision uploads, resolves conflicts, and cleans up staging files. Upon landing a revision upload (`refresh_after_upload`), it rebaselines both still-queued ops (`rebaseline_pending`) and open write handles targeting the same node to prevent false self-conflict copies on subsequent writes.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
|
||||
+888
-19
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,717 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to the Linux client since the 1.0.0 release. Format follows
|
||||
[Keep a Changelog](https://keepachangelog.com/1.1.0/); versions follow semver, and every
|
||||
release is tagged `vX.Y.Z` with `Cargo.toml` and `packaging/PKGBUILD` in agreement.
|
||||
|
||||
`Schema:` lines record the SQLite `SCHEMA_VERSION` (`pdfs-core/src/db/migrations.rs`) a
|
||||
release ships. Migrations are forward-only — a database written by a newer client is a hard
|
||||
refuse-to-open, not a downgrade, so rolling back a release means restoring the cache from
|
||||
scratch (user data in `staging/` and `recovery/` is never touched by this).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
Nothing yet.
|
||||
|
||||
## [1.10.1] — 2026-09-15
|
||||
|
||||
Dependency upgrade release, clearing both open Dependabot advisories. No schema change.
|
||||
Schema: **28**; SDK bumped to `proton-sdk` / `proton-drive-rs` **0.6.4**.
|
||||
|
||||
### Security
|
||||
- **`lru` IterMut Stacked Borrows advisory.** The SDK now depends on `lru` 0.18.
|
||||
- **`atty` unaligned read advisory.** The tray moved to `ksni` 0.3, which no longer pulls
|
||||
`dbus-codegen` → `clap` 2 → `atty`.
|
||||
|
||||
### Changed
|
||||
- **Keyring backend is `keyring-core` + `dbus-secret-service-keyring-store`** (keyring 4's split).
|
||||
Same libdbus Secret Service as before; stored sessions written by earlier releases are still
|
||||
found, so no re-login is needed.
|
||||
- **The tray waits for a late StatusNotifierWatcher** instead of failing to start when it beats
|
||||
the desktop's tray host onto the session bus.
|
||||
- Upgraded `gtk4` 0.11, `libadwaita` 0.9, `webkit6` 0.6, `rusqlite` 0.40 (bundled SQLite),
|
||||
`fuser` 0.18, `directories` 6, `sha1`/`sha2` 0.11, `base64` 0.23.
|
||||
|
||||
## [1.10.0] — 2026-09-15
|
||||
|
||||
One-request move-and-rename in the drain, and who shared an item with you. No schema change.
|
||||
Schema: **28**; SDK bumped to `proton-sdk` / `proton-drive-rs` **0.6.3**.
|
||||
|
||||
### Added
|
||||
- **Shared with me says who shared an item, and when.** The GUI row subtitle and `pdfs
|
||||
shared-with-me` show the inviter and invite date for a share root. The invitation's signature is
|
||||
checked against the claimed inviter's keys; when it does not verify, the GUI shows a warning
|
||||
and the CLI prints one, since the sender may be forged. `DirEntry` gains `shared_by`,
|
||||
`shared_at` and `shared_by_unverified` (defaulted, so older front-ends and daemons interoperate).
|
||||
|
||||
### Changed
|
||||
- **A queued move-and-rename lands as one request.** `drain_rename` used to rename in the source
|
||||
folder and then move, with a rename-out-of-the-way step when the destination held the name. It
|
||||
now calls `move-multiple` with a target name, so there is no half-applied state between the two
|
||||
halves and no second call for the encrypted-name requirements to go stale against (B46). A
|
||||
rename that stays in the same folder still uses the rename endpoint.
|
||||
|
||||
### Fixed
|
||||
- **Moving a node with no signature email no longer fails.** Such nodes (created anonymously, e.g.
|
||||
through a public link) need the passphrase signed by the mover; SDK 0.6.3 sends that signature.
|
||||
- **A node trashed through the newer API field is recognised as trashed.** The SDK now reads
|
||||
`TrashTime` as well as the legacy `Trashed` field, so the drain no longer tries to rename a node
|
||||
that is already in the trash.
|
||||
|
||||
## [1.9.3] — 2026-09-01
|
||||
|
||||
The daemon's wait-for-login loop, and the latency that fixing it introduced. No schema change.
|
||||
Schema: **28**; SDK unchanged at `proton-sdk` / `proton-drive-rs` **0.6.2**.
|
||||
|
||||
### Fixed
|
||||
- **An enabled service with no saved session no longer polls the keyring 28,800 times a day.**
|
||||
`wait_for_session` slept a flat 3 s between checks, and a check is not just a sleep — `auth::load`
|
||||
builds a fresh `keyring::Entry`, so every iteration opened a new D-Bus connection to the Secret
|
||||
Service and negotiated a session on it. The wait now backs off, and "not logged in" is said once
|
||||
at `info` instead of on every pass, which also stops the journal filling with tens of thousands
|
||||
of identical lines (`#21`).
|
||||
- **Signing in wakes the parked daemon instead of leaving the mount up to an interval late.** The
|
||||
backoff had no wake-up path: a unit started at boot reaches the ceiling within ~90 s, and
|
||||
`enable --now` is a no-op on an already-active unit, so the GUI login and the tray's Connect left
|
||||
it sleeping. `service::enable_start` now runs `enable` + `restart`, which interrupts the sleep and
|
||||
still starts a unit that is down, and `pdfs login` restarts the unit when it is already active —
|
||||
deliberately without touching `enable`, so a headless setup driving `pdfs daemon` by hand is left
|
||||
alone. The ceiling also drops from 60 s to 15 s, still an 80% cut in keyring/D-Bus wakeups
|
||||
(28,800/day → 5,760/day), bounding the worst case for paths that never reach systemd.
|
||||
|
||||
### Changed
|
||||
- The wait is extracted into `wait_for_session()`, taking the load and sleep calls as parameters,
|
||||
so the schedule is testable without a keyring and without real time passing.
|
||||
|
||||
## [1.9.2] — 2026-08-28
|
||||
|
||||
Repair for photos a Takeout import filed under the day they were imported, an `fzf` launcher front
|
||||
end, and a gallery that survives 1,600 photos on one date. No schema change. Schema: **28**; SDK
|
||||
`proton-sdk` / `proton-drive-rs` **0.6.2**.
|
||||
|
||||
### Added
|
||||
- **`pdfs redate-photos`: repair photos filed under the day they were imported.** A Takeout whose
|
||||
metadata sidecars did not match imports with no capture time at all, and the SDK then seals
|
||||
"now" into every revision — a decade of photos lands on one date. The date is still in the file
|
||||
name, so this finds photos whose name disagrees with their stored capture time by more than a
|
||||
day and rewrites them: download, re-upload at the right time, carry favourites and album
|
||||
memberships across, then **trash** (never delete) the original — Proton has no API for editing a
|
||||
sealed capture time. `--dry-run` reports without touching anything, `--from`/`--to` aim the run
|
||||
at the days an import landed on, and `--limit` caps it. Followed with `pdfs redate-status`,
|
||||
stopped with `pdfs cancel-redate`.
|
||||
|
||||
- **`pdfs-prompt --fzf`: launcher search that updates as you type.** A dmenu-style launcher
|
||||
(fuzzel, rofi, wofi, …) is handed one static list and filters it itself, which is why `--dmenu`
|
||||
has to make searching two steps — type, Enter, *then* results. `fzf` can re-run a command on
|
||||
every keystroke, so the new mode re-queries the daemon per keystroke (debounced to match the GTK
|
||||
prompt) and keeps `--disabled` on so the daemon's ranking, not fzf's matcher, decides the order.
|
||||
Abbreviation and typo tolerance therefore work here exactly as they do in the built-in window.
|
||||
- Started without a tty — from a WM keybinding — `--fzf` spawns a terminal for itself (foot,
|
||||
ghostty, kitty, alacritty, wezterm or xterm, whichever is installed), each with the app-id/class
|
||||
`pdfs-prompt` so a single window rule floats it. Override with `"prompt": { "terminal": [...] }`,
|
||||
where a `{cmd}` token places the command and its absence appends it. `"prompt": { "mode": "fzf" }`
|
||||
makes it the default front end for an existing keybinding.
|
||||
|
||||
### Fixed
|
||||
- **Google Photos Takeout: sidecars with a localized suffix are matched again.** Google localizes
|
||||
the `.supplemental-metadata` it appends to a photo's JSON sidecar — a German export writes
|
||||
`.ergänzende-Metadaten` — and the scanner only knew the English spelling. No sidecar matched, so
|
||||
no photo had a capture time, so every photo in the export was stamped with the moment it was
|
||||
uploaded. Matching now hangs on the photo name in front of the suffix rather than on the suffix
|
||||
itself, which is locale-agnostic, and covers the misspelt variants Google has also shipped.
|
||||
- **A photo with no usable sidecar now takes its date from its file name.** `PXL_20260818_171030868.jpg`,
|
||||
`IMG_20230219_171030.jpg`, `Screenshot_20230101-102950.png`, `IMG-20230219-WA0001.jpg` and
|
||||
`Screenshot from 2023-01-01 10-29-50.png` all carry the date the photo was taken. Read as UTC, so
|
||||
it can be hours out — but never the years out that "the moment it was imported" is. A sidecar
|
||||
still wins.
|
||||
|
||||
- **An opened file or folder no longer dies with the window that opened it.** Everything launched
|
||||
through `open_with` was started in the caller's process group with the caller's stdio, which was
|
||||
invisible from a resident GUI but fatal from `--fzf`: the prompt exits the moment you choose
|
||||
something, its terminal closes, and the pty hangup killed the application it had just started —
|
||||
so an `@dir` rule like `alacritty --working-directory {}` opened nothing at all. Opened children
|
||||
now get their own process group and no inherited stdio. Affects every front end; only `--fzf`
|
||||
could observe the bug.
|
||||
|
||||
### Changed
|
||||
- **The gallery virtualises at the row, not the day.** The Photos ListView had one item per day,
|
||||
and a bind built every one of that day's tiles — fine for a normal day, ruinous for the ~1,600
|
||||
photos a Takeout import drops onto a single date, which was 1,600 widgets constructed on the main
|
||||
thread before the section could be shown. Days are now flattened into heading rows and grid rows,
|
||||
so a bind is a handful of widgets whatever the day holds. A resize or a zoom step rebuilds the row
|
||||
model (the column count decides which photos share a row) and scrolls back to the photo that was
|
||||
at the top.
|
||||
- Updated to `proton-sdk` / `proton-drive-rs` 0.6.2 (photos save-to-timeline, `enumerate_events`).
|
||||
|
||||
- The daemon side of both launcher front ends — search, pins, mountpoint lookup and the
|
||||
open-a-result policy — moved out of `pdfs-gui/src/dmenu.rs` into a shared `pdfs-gui/src/query.rs`,
|
||||
so `--dmenu` and `--fzf` cannot drift on what a result set is or what "open" means. No behaviour
|
||||
change to `--dmenu`.
|
||||
|
||||
## [1.9.1] — 2026-08-21
|
||||
|
||||
Thumbnails everywhere the GUI lists files, a status bar under the Files browser, and the
|
||||
repository's contribution/branding files. No schema change. Schema: **28**; SDK unchanged at
|
||||
`proton-sdk` / `proton-drive-rs` **0.6.1**.
|
||||
|
||||
`exiftool` (`libimage-exiftool-perl` / `perl-image-exiftool` / `perl-Image-ExifTool`) is a new
|
||||
runtime dependency, declared in the `.deb`, `.rpm` and `PKGBUILD` packaging. It is only used to
|
||||
pull the embedded preview out of a camera RAW; without it every other image format still gets a
|
||||
thumbnail.
|
||||
|
||||
### Added
|
||||
- **Thumbnails for ordinary Drive files**, not just the Photos timeline — painted in Files (both
|
||||
the grid and the Name column of the list), Shared, Shared by me, and Trash. Standard raster
|
||||
formats (`bmp`, `gif`, `jpeg`, `png`, `tiff`, `webp`) are decoded in the daemon; camera RAWs
|
||||
(`arw`, `cr2`, `cr3`, `dng`, `nef`, `orf`, `raf`, `rw2`, and the rest of the concrete list) are
|
||||
served from their embedded preview via `exiftool`. Previews are cached beside the existing photo
|
||||
thumbnails, outside the cache-budget scan, and tagged with the node's modification time — so a
|
||||
new revision of an image can never be shown with its predecessor's thumbnail.
|
||||
- Thumbnails for visible rows are requested opportunistically as tiles scroll in, and the daemon
|
||||
drops work for listings the user has already navigated away from. The listing generation is
|
||||
issued by the daemon (`ReserveFileThumbGeneration`), not seeded from the front-end's clock.
|
||||
- **Build a whole folder tree ahead of time** from the Files toolbar: one recursive local-thumbnail
|
||||
job per daemon, with live scan/found/completed/failed progress and a Cancel that appears only
|
||||
once the daemon has acknowledged the start. The job outlives navigation — it is deliberate work,
|
||||
separate from the opportunistic per-tile requests.
|
||||
- **A Dolphin-style status bar under the Files browser**: the current listing's counts, a Zoom
|
||||
slider for the icon grid, and account storage usage. Zoom resizes the realised grid cells rather
|
||||
than rebuilding the model, so dragging the slider does not tear down the selection; the default
|
||||
size reproduces the previous fixed 72 px tiles. Capacity stays out of the bar until a real
|
||||
reading lands, and the quota fetch (60 s TTL) now also runs while Files is on screen instead of
|
||||
only in Settings.
|
||||
- Repository housekeeping: `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, `SECURITY.md`, issue and pull
|
||||
request templates, a project logo, and workspace crate metadata (`repository`, `homepage`,
|
||||
`readme`, `keywords`, `categories`).
|
||||
|
||||
### Fixed
|
||||
- A remote thumbnail miss no longer suppresses local generation. The two negative results are now
|
||||
held in separate bounded caches: "Proton has no thumbnail for this node" and "this exact revision
|
||||
was decoded locally and could not be read" mean different things, and only the second is a
|
||||
permanent verdict for that revision.
|
||||
- A RAW whose preview could not be extracted because of the *environment* — `exiftool` missing, a
|
||||
failed download — is retried instead of being remembered as undecodable. Only bytes conclusively
|
||||
outside the supported formats are cached as a negative.
|
||||
- A recursive build that finds a uid an opportunistic tile job already owns waits briefly for it
|
||||
rather than either duplicating the work or blocking on it forever, and a cancellation racing a
|
||||
waiter's registration is caught by a backstop instead of leaving the waiter parked.
|
||||
- Starting a build for a root that is already building attaches to that job; starting one for a
|
||||
*different* root is refused explicitly rather than silently reporting the running job's progress
|
||||
as the caller's.
|
||||
- Decrypted camera RAWs are staged for `exiftool` in an owner-only (`0700`) directory of their own,
|
||||
removed as soon as the preview is out and swept at cache open after an unclean shutdown.
|
||||
- The RAW extension list had `rw` (not a format) and was missing `raw`; both classification helpers
|
||||
now read from one shared table instead of two hand-maintained `matches!` arms.
|
||||
- The tray and desktop-entry icons were wrong/missing; `.desktop` files and the packaged SVG were
|
||||
corrected.
|
||||
|
||||
## [1.9.0] — 2026-08-18
|
||||
|
||||
Google Photos import, and a pass over the desktop app's interaction gaps. No schema change.
|
||||
Schema: **28**. Builds against the published `proton-sdk` / `proton-drive-rs` **0.6.1** — the album
|
||||
writes this release needs are in the registry crates, so the local `[patch.crates-io]` shim the
|
||||
feature was developed against is gone and no local path dependency remains.
|
||||
|
||||
### Added
|
||||
- **Import a Google Photos Takeout export into Proton Photos** — `pdfs import-google-photos
|
||||
takeout-*.zip` (add `--dry-run` to see what it would do, `--wait` to follow it), or the import
|
||||
button in the Photos header, which multi-selects the export's zips. Albums in the export are
|
||||
recreated as Proton albums and their photos filed into them; year folders and `Archive/` import
|
||||
into the timeline; `Trash/` is skipped. Capture times and favourites come from Google's per-photo
|
||||
JSON sidecars, including the truncated and `(N)`-swapped sidecar names Google produces.
|
||||
- Photos already on the account are **not re-uploaded**: the import asks the photos volume which of
|
||||
the export's names exist (batched, 150 per request), reads and SHA-1s only those files, and skips
|
||||
the ones whose content digest matches too — Proton's own duplicate rule. A library already partly
|
||||
synced by Proton Photos on a phone therefore imports only what is missing, and an interrupted
|
||||
import resumes by being run again. Google keeps a full copy of a photo in every album folder it
|
||||
appears in; those fold into one upload that joins several albums.
|
||||
- The import streams each photo straight out of its zip, so an export needs no scratch space to
|
||||
unpack into, and reports as a job in `pdfs transfers` and the GUI's activity view. `pdfs
|
||||
import-status` shows the counts, `pdfs cancel-import` stops it — a cancelled run still files what
|
||||
it uploaded into its albums.
|
||||
- **An Import page in the GUI** (Settings → *Import from Google Photos*, or the import button in
|
||||
the Photos header). Archives are staged by drag-and-drop onto the page or through the file
|
||||
chooser, listed with their sizes so the set can be checked before anything is sent, and launched
|
||||
as either a Scan (dry run) or a confirmed Import. The page shows live progress off the import's
|
||||
job, keeps the finished run's report on screen, and can stop a run in flight; because the import
|
||||
outlives the page, it is polled wherever the user navigates and its completion is reported as a
|
||||
desktop notification.
|
||||
|
||||
- SDK (`proton-sdk-rs` **0.6.1**): album **writes** — `ProtonPhotosClient::create_album` and
|
||||
`add_photos_to_album`, ported from the TypeScript SDK (upstream C# has no album write API), plus
|
||||
batch duplicate detection (`find_duplicates_many`, `name_collisions`). The workspace deps moved
|
||||
`"0.6.0"` → `"0.6.1"` and `Cargo.lock` re-resolved to the registry crates. Adding a photo from an
|
||||
album *shared with us* (a different volume) is still unported and fails that photo's outcome.
|
||||
- **Multi-select in Files.** Both views select a batch (click-drag, Ctrl-click, Shift-click); a bar
|
||||
slides in reporting the count and offering Move to Trash and offline on/off for the whole
|
||||
selection. Delete acts on everything selected, Esc clears it, and right-clicking inside a
|
||||
selection offers the batch actions rather than the one row under the pointer. Actions that need a
|
||||
single subject — rename, move, share, revisions — stay single-entry, and the details pane steps
|
||||
aside for a batch instead of speaking for it.
|
||||
- **Undo on trash.** "Moved *X* to Trash" now carries an Undo that restores the batch, single files
|
||||
included.
|
||||
- **The empty states are a place to act**: an empty folder offers Upload files / New folder, and an
|
||||
empty photo timeline offers Upload photos / Import from Google Photos, instead of describing what
|
||||
the user could go and do elsewhere.
|
||||
- **Ctrl+F works from any page**, navigating to Files and focusing the search box, rather than being
|
||||
silently inert everywhere but Files.
|
||||
- Staged Takeout archives survive closing the window; archives that have since been moved or deleted
|
||||
are dropped from the restored set.
|
||||
|
||||
### Fixed
|
||||
- Settings showed the local cache as "X of 0 B used" when no cache budget was set. An unset budget
|
||||
means *unlimited*, so it now reads "X cached — no limit set" rather than implying a zero cap.
|
||||
- The cache-budget editor fired a `SetCacheBudget` round-trip — and a toast — on every step, so a
|
||||
run of `+` clicks applied a series of caps the user never asked for and stacked a column of
|
||||
toasts. It now waits for the value to settle.
|
||||
- A long pin list pushed everything below it off the end of Settings. It shows the first six with a
|
||||
"Show all N" row for the rest.
|
||||
|
||||
## [1.8.2] — 2026-08-17
|
||||
|
||||
Closes the two findings 1.8.1's live verification produced (B85, B86) and finishes the data-safety
|
||||
audit entries that had been sitting as "fixed in code, verification pending" since 1.0 (B49–B55).
|
||||
No schema change. Schema: **28**.
|
||||
|
||||
The read path no longer assumes a revision is a series of 4 MiB blocks; it takes the geometry from
|
||||
the revision. The B85 investigation also **contradicted its own bug report** — see Notes, because
|
||||
the conclusion recorded there was that reads of such files were losing data, and they were not.
|
||||
|
||||
### Fixed
|
||||
- A folder switched from mirror to on-demand now shows the files the mirror pass itself uploaded.
|
||||
The sync engine created nodes on the remote without invalidating the parent's cached child
|
||||
listing, and that listing is a database flag rather than an in-memory cache — so it survived a
|
||||
daemon restart, and the files stayed invisible until the folder was switched back. Every remote
|
||||
mutation the engine makes (folder create, file upload, and the three trash paths) now drops the
|
||||
parent's cached listing by uid, which reaches a folder that no mount has resident — the case a
|
||||
mirror folder always is (`docs/BUGS.md` B86).
|
||||
- `pdfs ls` and `pdfs refresh` accept a path under any mount the daemon owns, not only the primary
|
||||
one. Both previously answered "is not under the mountpoint" for anything inside a secondary
|
||||
on-demand mount, which left a folder serving a stale listing with no manual way to re-enumerate
|
||||
it. A path under nothing this daemon has mounted still gets the same error as before.
|
||||
- A file whose blocks are not 4 MiB is no longer read twice over. Block offsets came from
|
||||
`index * 4 MiB`, so on a revision laid out differently every block the client asked for
|
||||
straddled two of the revision's own, and each was fetched and decrypted to answer for one — then
|
||||
cached at boundaries the next read would not ask about. Offsets now come from the revision's
|
||||
recorded block sizes, and the on-disk cache, the in-memory ring and the in-flight fetch key
|
||||
address a block by the byte range it covers rather than by its index (`docs/BUGS.md` B85).
|
||||
|
||||
### Changed
|
||||
- The block cache records each block's plaintext start alongside its `(mtime, size)` tag, and each
|
||||
revision's block table in a small sidecar beside its blocks. Blocks cached by an earlier release
|
||||
carry no start; they are honoured wherever the old 4 MiB assumption and the new plan agree on
|
||||
the offset, which is every file this client uploaded — so the existing cache survives the
|
||||
upgrade intact and is quietly refetched only for the files it was placing wrongly.
|
||||
|
||||
### Notes
|
||||
- **B85's diagnosis was wrong, and the entry has been corrected rather than closed.** It read a
|
||||
set of journal lines as proof that this client was requesting wrong byte ranges and being saved
|
||||
only by B84's repair path. `RevisionReader::read_at` does not work that way: it plans over the
|
||||
revision's real block sizes and clamps every block to the range asked for, identically in the
|
||||
0.6.0 the client builds against and in every earlier version. A read can therefore never return
|
||||
more bytes than it requested — yet every quoted line shows exactly that (`blen=4194304
|
||||
got=4939506`), and the one place in the client that logs that shape can only reach it on the
|
||||
opposite condition. The numbers were either transcribed with two fields swapped or captured from
|
||||
a build not in the history; they want re-capturing from the journal before anything is concluded
|
||||
from them. What was real is the assumption itself, and that is what this release removes — a
|
||||
cost fix, not a correctness one. Nothing here changes what a read returns.
|
||||
- The first read of a file still plans on the 4 MiB assumption, and learns the real geometry as a
|
||||
side effect of opening the reader it needs anyway. Resolving a reader merely to *plan* would
|
||||
reintroduce the per-read key derivation the block cache exists to avoid, so the first read of a
|
||||
non-4-MiB file pays one straddling fetch and then self-corrects.
|
||||
- Seven data-safety audit entries (B49–B55) moved from "fixed in code, verification pending" to
|
||||
verified. The fixes were all genuinely present and needed no change; what they lacked was the
|
||||
injected-failure testing each entry asked for, and that is what was written — unwritable
|
||||
directories and `chmod 000` subtrees for permission failures, a real `/dev/shm`↔`/tmp` device
|
||||
boundary for the cross-filesystem fallbacks, SQLite's `max_page_count` for a full disk, an abort
|
||||
trigger for a failed insert, and truncated and unpublished sidecars for the crash boundaries.
|
||||
Two seams were added to make failure arrangeable at all: `stage_write_at`, mirroring the
|
||||
`preserve_write_at` that already existed, and `walk_local_tree` lifted out of the sync engine so
|
||||
the local scan can be driven against a real tree without a daemon.
|
||||
- **Two of those entries are not fully closed, and say so.** B51 and B52 concern crash consistency,
|
||||
and nothing short of a machine actually losing power tests that; the crash *boundaries* are
|
||||
covered by reconstructing on disk exactly what a crash at each point leaves. B54 also diverges
|
||||
from its required fix on purpose: it has no "deliberate delete" confirmation path, so a total
|
||||
local wipe is refused outright rather than offered for confirmation. That is stricter than asked
|
||||
for, and the cost is that a user who really did delete everything must remove and re-add the
|
||||
sync folder.
|
||||
- B86's fix routes only `ls` and `refresh` to the owning mount. The other path-addressed control
|
||||
requests — pin, rename, move, delete, sharing — still resolve against the primary mount only.
|
||||
That is a larger change and was not what blocked the repro; it is recorded as an open follow-up
|
||||
(`docs/BUGS.md` B88).
|
||||
- Three findings were opened rather than fixed: the geometry above is learned one read late, so a
|
||||
non-4-MiB file pays a single wasted fetch on its first read (B87); the remaining path-addressed
|
||||
control requests are still primary-only (B88); and the container-level `CLAUDE.md` still
|
||||
describes the SDK dependency as 0.1.11 when 1.7.1 bumped it to 0.6.0, which is what sent the
|
||||
B85 investigation to the wrong source tree to begin with (B89).
|
||||
|
||||
## [1.8.1] — 2026-08-16
|
||||
|
||||
Some files read short through the mount and reported success — an application got a truncated
|
||||
copy with nothing anywhere saying so (`docs/BUGS.md` B84). Found because GNOME's file indexer
|
||||
kept complaining that PNGs under `~/ProtonDrive/Pictures` were corrupt; it was right. The rest
|
||||
of this release is the long-standing architectural backlog from the July concurrency audits
|
||||
(B25, B32, B39, B40, B43, B44), each verified against the current code before being fixed.
|
||||
Schema: **28** — migration V28 adds `sync_entry.local_mtime_ns`.
|
||||
|
||||
### Fixed
|
||||
- A read no longer trusts a revision whose recorded block sizes disagree with the file's size.
|
||||
Such a file leaves the block/range path entirely: it is fetched through the whole-file,
|
||||
manifest-verified download — the same path `pdfs pin` uses, and the reason pinning was the
|
||||
workaround — and served from the resulting cached blob. One download per file however many
|
||||
reads discover the problem at once, and files above 512 MiB fail `EIO` with a note to pin
|
||||
them rather than pulling down gigabytes for one read.
|
||||
- A block that comes back shorter than the file's size implies is never served, cached or kept
|
||||
in memory. Assembling a read out of short blocks is what quietly produced the truncation:
|
||||
the missing bytes were simply not appended, the kernel handed userspace a short read, and
|
||||
every tool reads a short read as end-of-file.
|
||||
- A synced folder no longer reports the same deletion as an error on every pass. Deleting a
|
||||
local file that something else had already deleted counted as a failure, which kept its
|
||||
baseline and made the next pass try again — forever. A path that is already gone is the
|
||||
state the sync pass wanted (`docs/BUGS.md` B42).
|
||||
- A mirror folder no longer misses an edit made in the same second as its last sync. Change
|
||||
detection compared whole seconds, so rewriting a file in place without changing its length
|
||||
read as "unchanged" and never uploaded — the remote silently stayed behind until something
|
||||
else about the file moved. Baselines now record the modification time to nanoseconds
|
||||
(`docs/BUGS.md` B25).
|
||||
- A sync download no longer overwrites an edit that finished while it was running. The
|
||||
destination is re-checked in the last moment before the download is published, and a local
|
||||
file that has moved on since the pass planned for it is kept as a conflict copy — the same
|
||||
resolution a both-sides-changed conflict gets (`docs/BUGS.md` B39).
|
||||
- An upload interrupted by a writer no longer reports itself as synchronized. The baseline now
|
||||
records the content that actually went up rather than whatever is on disk when the transfer
|
||||
ends, so a revision torn by a concurrent write is replaced by a clean one on the next pass
|
||||
instead of standing as the file's content on Drive (`docs/BUGS.md` B40).
|
||||
- Listing a directory that changes while it is being read no longer skips or repeats entries.
|
||||
`readdir` resumed by index into a listing rebuilt on every call, so a create or a trash
|
||||
between two pages shifted everything after it. The listing is now frozen for the life of the
|
||||
directory handle (`docs/BUGS.md` B43).
|
||||
- Opening a file for writing while a previous close is still publishing no longer discards that
|
||||
close's revision. The base is re-checked at the moment the write handle is installed, rather
|
||||
than trusted across a copy that can be the size of the whole file (`docs/BUGS.md` B32).
|
||||
- Unmounting now stops and joins the daemon's background workers instead of leaving them
|
||||
running. The drain, sync engine, conflict sweep, online probe, local indexer and control
|
||||
listener share a stop signal and interruptible waits, so an in-process remount no longer
|
||||
starts a second set on top of the first (`docs/BUGS.md` B44).
|
||||
|
||||
### Notes
|
||||
- The size `stat` reports was always the correct one; it is the reader's belief about where
|
||||
blocks end that was wrong, so nothing about reported sizes changes.
|
||||
- Verified live on the affected account: four multi-block PNGs that had been reading short now
|
||||
deliver their full size, and the journal shows each one repaired. The diagnostics also
|
||||
answered the open question, and not as expected — the revision metadata is fine; those files
|
||||
simply do not have 4 MiB blocks, and this client assumes they do. Filed as `docs/BUGS.md`
|
||||
B85. Until that is fixed, a file with non-4-MiB blocks is downloaded whole on its first read.
|
||||
- Migration V28's new column is nullable rather than converted from the existing seconds. A
|
||||
baseline row written before this release has no sub-second time to recover, and inventing
|
||||
zero for it would make every already-synced file compare as changed and re-upload an entire
|
||||
mirror on first start; `NULL` keeps those rows comparing exactly as they did, and each gains
|
||||
a real value the next time its path syncs.
|
||||
- Three audit entries closed without a code change, having been fixed already or being correct
|
||||
as they stand: sync-folder removal is serialized against reconciliation (B41), the drain's
|
||||
database contention was removed by the read-only connection pool and the claim column (B30),
|
||||
and the refusals for symlinks, hard links, device nodes and `chmod` are deliberate — Drive
|
||||
cannot represent them, and a local-only representation no other Proton client can read would
|
||||
be worse than a clean refusal (B31).
|
||||
- Six of this release's fixes were then driven live against the installed 1.8.1 on a real
|
||||
account: the same-second edit uploads (B25), three rapid rewrites of a 64 MiB file publish the
|
||||
third one byte for byte (B32), a write racing a download survives as a conflict copy (B39), an
|
||||
upload torn by a concurrent write heals itself on the following pass (B40), a `getdents64`
|
||||
walk with a 160-byte buffer against a directory mutated between pages returns every entry
|
||||
exactly once (B43), and a restart of a daemon with six mounts and a busy drain stops in 110 ms
|
||||
and comes back at the same thread count (B44). B42 could not be driven from a single machine —
|
||||
see B86 below.
|
||||
- One new bug came out of the verification: an on-demand mount serves a folder listing that
|
||||
predates the sync engine's own uploads, so files the mirror pass created are invisible until
|
||||
the folder is switched back (`docs/BUGS.md` B86). Nothing is lost — they are on the server and
|
||||
come back on the next mirror pass — but they cannot be seen or named while the folder is
|
||||
on-demand, which is also what blocks B42's live verification.
|
||||
|
||||
## [1.8.0] — 2026-08-16
|
||||
|
||||
A queued write whose node had left the local tree was retried every five seconds forever and
|
||||
counted as an ordinary pending upload — 28 of them, holding 18.4 GiB, had been doing that for
|
||||
30 days on the machine that found it (`docs/BUGS.md` B83). Schema: 27 — migration V27 adds
|
||||
`pending_op.access_deferred_since`.
|
||||
|
||||
### Fixed
|
||||
- The drain no longer reads "I have never heard of this node" as "you may not write this
|
||||
node". A missing authority is now resolved against the remote: the node is re-interned if
|
||||
it is still there, and reported as a real failure if it is gone. Ops that had been stuck
|
||||
invisibly for weeks now say what is wrong with them on the first pass.
|
||||
- A create or mkdir resolves its *parent* as the authority, so the refetch asks about the
|
||||
folder rather than the node it has not made yet.
|
||||
|
||||
### Added
|
||||
- Access deferrals are bounded. A deferral still costs no attempt and records no error for
|
||||
the first five minutes — the case it exists for is a share downgraded mid-queue and undone
|
||||
a moment later — but past that window each recheck records a failure, so the op enters the
|
||||
ordinary backoff and appears in `failing_ops` and the `pdfs status` error text like any
|
||||
other stuck operation. The first deferral of a run is logged at `warn` rather than `debug`.
|
||||
|
||||
### Notes
|
||||
- No staged data is touched by any of this. A reported failure keeps its `pending_op` row and
|
||||
its staged blob exactly as before; the change is in what the user is told, and in the queue
|
||||
no longer waiting on a permission change that was never coming.
|
||||
|
||||
## [1.7.1] — 2026-08-16
|
||||
|
||||
Write-back throughput follow-up to 1.7.0, same audit
|
||||
(`docs/plans/audit-2026-08-15-perf-locks.md`). Schema: 26 — migration V26 adds
|
||||
`pending_op.claimed_at`.
|
||||
|
||||
### Added
|
||||
- Parallel write-back drain: three workers share the queue through a `pending_op.claimed_at`
|
||||
claim column (migration V26), so a large upload no longer holds every queued rename, trash
|
||||
and small write behind it. Ordering is preserved per node — the claim query never offers
|
||||
two workers ops for the same uid — and claims left by a crashed run are cleared at open.
|
||||
- An upload already on the wire is cancelled when a newer write, a trash or a discard
|
||||
supersedes it, instead of spending the uplink on a revision the next operation replaces.
|
||||
|
||||
### Changed
|
||||
- The revision debounce is adaptive: it widens toward how long that node's last upload
|
||||
actually took (bounded to 2–60 s), so a file saved faster than it can be sent supersedes
|
||||
in the queue rather than mid-upload.
|
||||
- Workspace deps bumped to `proton-sdk` / `proton-drive-rs` 0.6.0.
|
||||
|
||||
## [1.7.0] — 2026-08-16
|
||||
|
||||
Performance and robustness pass over the mount, from the architecture audit in
|
||||
`docs/plans/audit-2026-08-15-perf-locks.md`. Schema: 25.
|
||||
|
||||
### Added
|
||||
- Sequential-read prefetch (`Core::prefetch_ahead`): per-file sequential detection, depth
|
||||
doubling 2→8, reset to 0 after a seek, skipping blocks already cached, and bounded by its
|
||||
own permit budget so it can never queue ahead of a demand read.
|
||||
- Single-flight block fetching (`BlockFlight`) keyed on `(uid, mtime, size, block)`, so a
|
||||
demand read, kernel read-ahead and prefetch of the same block share one download+decrypt.
|
||||
- In-memory block LRU (`BlockRing`) in front of *both* block paths; a block read off disk is
|
||||
promoted into it, so the following ~31 kernel reads of that 4 MiB block cost nothing.
|
||||
- Read-only SQLite connection pool (`ReadPool` / `Db::read`) with every `SELECT`-only method
|
||||
routed at it, so a read no longer waits on the write connection.
|
||||
- `nodes.path` persisted and backfilled, replacing the recursive `path_of` walk.
|
||||
- Single-writer `flock` on the database; a `SQLITE_CORRUPT`/`NOTADB` file is quarantined as
|
||||
`cache.db.corrupt-<ts>` (WAL sidecars too) and rebuilt rather than failing the mount.
|
||||
- `VACUUM` and the deep `integrity_check` now ack immediately and run as background jobs;
|
||||
`pdfs doctor` polls for the verdict.
|
||||
- Staging reconcile at mount: staged blobs no queued operation refers to are re-queued, or
|
||||
named in the log when they cannot be addressed.
|
||||
- Queue/staging visibility — `Status` gained `parked_uploads`, `failing_ops`,
|
||||
`failing_error`, `staged_bytes`, `staged_oldest_secs`; `pdfs status` prints them when
|
||||
non-zero.
|
||||
- `statfs` implemented against the account quota; FUSE `init` negotiates `max_readahead`,
|
||||
`max_background`, `PARALLEL_DIROPS` and `CACHE_SYMLINKS`; `blksize` raised to 1 MiB.
|
||||
|
||||
### Changed
|
||||
- `State` no longer writes to SQLite under the inode lock — mutations queue into an outbox
|
||||
applied after the lock is released, so interning a 5000-child listing no longer commits a
|
||||
transaction with the whole mount frozen.
|
||||
- Re-listing an unchanged folder no longer touches its subtree; when a subtree really moved,
|
||||
one recursive walk carries both the new path and the inherited trashed flag.
|
||||
- `release(2)` no longer gap-fills from the network: it stages an incomplete blob from local
|
||||
caches only and the drain thread finishes it off the dispatch loop.
|
||||
- A partial write over an undrained one is merged instead of refused.
|
||||
- Size-upgrade waiters park on a queue instead of occupying a worker thread, capped at 8
|
||||
folders in flight; past the cap `stat` answers provisionally.
|
||||
- Block-cache disk writes moved off the read's critical path; no more fsync per cached block.
|
||||
- Cache LRU touches buffered in memory and flushed in one transaction.
|
||||
- Search: sync-folder list and pin lookups hoisted out of the per-hit loop.
|
||||
- SQLite tuned (`cache_size`, `mmap_size`, `temp_store=MEMORY`, `wal_autocheckpoint`) and
|
||||
three new indexes added.
|
||||
- Mount registry lock released before FUSE teardown/join; kernel notifications batched under
|
||||
the state lock and flushed after it.
|
||||
|
||||
### Fixed
|
||||
- **Data loss:** `ENOSPC` is now propagated from `write`/`fsync`/staging instead of being
|
||||
swallowed, with emergency eviction wired up and scratch marked durable before any staging
|
||||
error; a failed `record_pending_write` fails `release` rather than dropping the write.
|
||||
- **Data loss:** the conflict sweep no longer discards queued operations after the trash
|
||||
round trip.
|
||||
- **Data loss:** `rescue_scratch` moves a blob with an unreadable sidecar to `recovery/`
|
||||
unaccompanied instead of fabricating a sidecar for it.
|
||||
- Own-sealed revisions are persisted (7-day TTL), so a restart no longer reopens the
|
||||
transient-download fork window fixed in 1.1.0.
|
||||
- Depth-capped `is_pinned` (cycle-safe), daemon write timeout, and every client-supplied
|
||||
`limit` clamped.
|
||||
- `recover_fsynced_writes` no longer scans the recovery directory on every idle pass.
|
||||
|
||||
## [1.6.0] — 2026-08-12
|
||||
|
||||
Search-index correctness and ranking (`docs/plans/search-index.md`). Schema: 21 —
|
||||
migration V21 rebuilds `nodes_fts`.
|
||||
|
||||
### Fixed
|
||||
- **B80** — 29% of the account was missing from the search index. `node_is_indexable_tx`
|
||||
required a null-parent ancestor, which only the My Files root has, so every node under a
|
||||
device folder was dropped from `nodes_fts` on its first upsert after the v16 backfill
|
||||
(2,705 of 9,252 nodes on the account this was diagnosed against). The rule now matches the
|
||||
backfill, migration V21 replays it, nodes under a trashed folder stay out, and the walk is
|
||||
depth-capped.
|
||||
- **B81** — the prompt labelled every Drive hit "My files", including device folders; pin
|
||||
rows also took `is_dir` from `Pin::recursive`.
|
||||
- **B82** — a `parent_uid` cycle hung the daemon on any short search: the recursive CTEs in
|
||||
`path_of` and the pin walk use `UNION ALL`, which does not deduplicate. `path_of` is now
|
||||
depth-capped at 256 and returns the truncated path.
|
||||
|
||||
### Changed
|
||||
- Local search ranking: whole-query prefix matches get their own lane (newest first) ahead
|
||||
of trigram rank, which on a home directory full of dependency trees was burying the
|
||||
obvious answer — 73% of the 500 best-ranked candidates for "test" came from one Go module
|
||||
cache. The single-character lane, which recovers a typo that destroys every trigram, stays
|
||||
behind it.
|
||||
|
||||
## [1.5.0] — 2026-08-11
|
||||
|
||||
Schema: 20.
|
||||
|
||||
### Added
|
||||
- **`pdfs-prompt --dmenu`** — the same search through an external dmenu-style launcher
|
||||
(fuzzel, rofi, wofi, tofi, bemenu, dmenu) instead of the built-in GTK HUD, so the prompt
|
||||
matches a tiling-WM desktop. A launcher filters a fixed list rather than re-querying per
|
||||
keystroke, so searching is a two-step loop; `--query` skips the first step, fuzzel and rofi
|
||||
get file-type icons. Configurable as `"prompt": { "mode": "dmenu", "menu": [...] }`, with
|
||||
`--gtk` overriding for one invocation.
|
||||
- **Configurable open-with rules** (`pdfs-core/src/opener.rs`): an ordered list of name/class
|
||||
patterns in `config.json` under `open_with`, each mapped to argv and optionally wrapped in
|
||||
a terminal, first match wins, falling back to `xdg-open`. Lets "open a text file" mean
|
||||
`nvim` in a terminal without breaking every graphical caller. Absent config reproduces the
|
||||
old behaviour exactly.
|
||||
|
||||
### Fixed
|
||||
- The prompt and dmenu openers hand applications the cached file while matching rules
|
||||
against the *original* Drive filename, so e.g. `.md` files stop being routed by their
|
||||
cache-path extension.
|
||||
|
||||
## [1.4.0] — 2026-08-10
|
||||
|
||||
Schema: 20.
|
||||
|
||||
### Added
|
||||
- **File version history** (`pdfs-fuse/src/revisions.rs` + a GTK versions dialog): list a
|
||||
file's revisions, restore one, delete one, or write an old one out to a local path. A
|
||||
restore re-points the file server-side, so no content crosses the wire and nothing enters
|
||||
the drain queue; the daemon drops its cached content and repopulates on the next read.
|
||||
- **Persistent SDK entity cache** (`pdfs-core/src/sdkcache.rs`): the SDK's decrypted node
|
||||
metadata now survives a daemon restart instead of re-walking the tree. It lives in its own
|
||||
`sdk_cache.db`, deliberately off the daemon's hot `cache.db` mutex, holds no key material,
|
||||
needs no migrations, and is cleared on logout.
|
||||
- Per-node outcomes for batch operations (`pdfs-core/src/batch.rs`): `trash`/`restore`/
|
||||
`delete`/`move` report one result per node, so a node the server rejects no longer fails
|
||||
the whole batch.
|
||||
- Photo viewer page in the GUI and richer photo metadata in the details pane.
|
||||
|
||||
### Changed
|
||||
- SDK telemetry routed into `tracing`, so its transfer/block-storage/request spans land in
|
||||
the daemon log instead of a no-op sink.
|
||||
- A file that fits in one block uploads as a single atomic request rather than the
|
||||
draft/block/commit sequence.
|
||||
|
||||
## [1.3.0] — 2026-07-31
|
||||
|
||||
Schema: 19.
|
||||
|
||||
### Added
|
||||
- **Photo albums** — `PhotoAlbums` / `AlbumPhotos` control requests, daemon-side `albums.rs`
|
||||
and a GUI albums page (the former gallery view, renamed). Albums other people share with
|
||||
you are included; their photos are persisted per album because a shared album's photos
|
||||
never appear in your own timeline. Served stale-while-refreshing like the timeline.
|
||||
- **GUI "Locations" page** — one row per mount spec: the primary `~/ProtonDrive` mount
|
||||
alongside every folder this computer backs up, mirrored or on-demand. Called *Locations*
|
||||
rather than *Mounts* because a mirror folder is a plain directory with no FUSE session.
|
||||
It absorbs the old Computers page's actions and the mountpoint chooser from Settings.
|
||||
- `role` on shared-node metadata (`viewer`/`editor`/`admin`), defaulted for wire-compat with
|
||||
older clients and daemons.
|
||||
- FUSE acceptance coverage for the permission-based mounts.
|
||||
|
||||
## [1.2.1] — 2026-07-31
|
||||
|
||||
Schema: 18.
|
||||
|
||||
### Fixed
|
||||
- Migration bug that could leave the `ProtonDrive` mount unwritable after upgrade.
|
||||
- Permission error on on-demand mounts.
|
||||
- B78.
|
||||
|
||||
### Changed
|
||||
- Release pipeline: `.deb` signing added and `.rpm` signing fixed.
|
||||
|
||||
## [1.2.0] — 2026-07-29
|
||||
|
||||
Sharing and multi-mount release. Schema: 18.
|
||||
|
||||
### Added
|
||||
- **Shared with me**: shares others sent you are listed, verified and mountable, including on
|
||||
the primary mount.
|
||||
They appear under a synthetic `Shared with me` directory on a virtual volume
|
||||
(`pdfs-fuse/src/virtual.rs`), with name collisions and over-long components resolved by
|
||||
its own naming rules.
|
||||
- Per-share access model (`db/share_access.rs`): the effective role of each shared root is
|
||||
persisted and kept resident in mount state, so interning an inode never costs a query. The
|
||||
database stays the restart/offline authority — every role change is written through before
|
||||
the cache updates. A share you cannot write is presented POSIX read-only.
|
||||
- Unified location model (`pdfs-core/src/mounts.rs`, `db/mounts.rs`): one presentation shape
|
||||
for every local Proton Drive location, joining device sync state with a new `mount` table.
|
||||
`MountMode` is `Mirror` (full local copy, reconciled both ways) or `OnDemand` (FUSE session
|
||||
fetching on access), with an `Unknown` fallback so a mode written by a newer client does
|
||||
not break an older one.
|
||||
|
||||
## [1.1.1] — 2026-07-27
|
||||
|
||||
Schema: 16.
|
||||
|
||||
### Fixed
|
||||
- Trash refresh did not pull updates.
|
||||
- Packaging: `PKGBUILD` was left at 1.1.0 by the 1.1.1 version bump, failing the release
|
||||
workflow's tag/manifest/PKGBUILD agreement check.
|
||||
|
||||
## [1.1.0] — 2026-07-26
|
||||
|
||||
Conflict-handling and concurrency release. Schema: 16.
|
||||
|
||||
### Added
|
||||
- Conflict sweep (`pdfs-fuse/src/sweep.rs`, 5-minute cadence): removes conflict copies proven
|
||||
identical by size + `content_sha1`, surfaces divergent ones once as an activity entry, and
|
||||
never removes what it cannot prove.
|
||||
- Sweep mode and `.pdfsignore`-driven conflict handling; ignore rules are rebuilt per sync
|
||||
pass so edits take effect without a restart.
|
||||
- Substantially extended FUSE acceptance suite.
|
||||
|
||||
### Fixed
|
||||
- **B69** — spurious `(sync-conflict)` copies. Revision identity now uses the server's
|
||||
`revision_id` rather than `(mtime, size)`, which the server re-stamps on seal; the planner
|
||||
gained `AdoptBaseline` so an equal-size no-baseline pair is recorded instead of forked on
|
||||
first reconcile.
|
||||
- **B70** — browser downloads into an on-demand mount had their in-flight temporaries sealed
|
||||
as canonical revisions. Transient names are recognised and held until the name is finalized.
|
||||
- Assorted race conditions and concurrency bugs on the write/drain paths.
|
||||
|
||||
### Changed
|
||||
- Workspace deps bumped to `proton-sdk` / `proton-drive-rs` 0.2.2.
|
||||
|
||||
## [1.0.0] — 2026-07-22
|
||||
|
||||
First stable release: FUSE files-on-demand mount, sync daemon under `proton-drive.service`,
|
||||
`pdfs` CLI and the GTK4 app, on top of the pure-Rust Proton Drive SDK. Schema: 16.
|
||||
|
||||
### Fixed
|
||||
- The outstanding FUSE defects tracked in `docs/BUGS.md`, plus a truncate defect, validated
|
||||
by a new POSIX compliance suite for the filesystem.
|
||||
|
||||
[1.9.3]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.9.3
|
||||
[1.9.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.9.2
|
||||
[1.9.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.9.1
|
||||
[1.9.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.9.0
|
||||
[1.8.2]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.8.2
|
||||
[1.8.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.8.1
|
||||
[1.8.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.8.0
|
||||
[1.7.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.7.1
|
||||
[1.7.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.7.0
|
||||
[1.6.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.6.0
|
||||
[1.5.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.5.0
|
||||
[1.4.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.4.0
|
||||
[1.3.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.3.0
|
||||
[1.2.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.2.1
|
||||
[1.2.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.2.0
|
||||
[1.1.1]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.1.1
|
||||
[1.1.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.1.0
|
||||
[1.0.0]: https://github.com/narrrl/proton-drive-linux/releases/tag/v1.0.0
|
||||
+2
-1
@@ -14,7 +14,8 @@ Consolidated from the historical feature roadmap; current bug status lives in [`
|
||||
* **Custom FUSE Mount Options (Horizon 1)**: Expose custom mounting parameters in configuration (e.g. `ro` for read-only, `direct_io` to bypass kernel caching, `allow_other`). Auto-generate systemd user mount units upon sync folder configuration.
|
||||
* **Symbolic Link & Hard Link Virtualization (Horizon 2)**: Add symbolic link virtualization inside the metadata store (`NodeType::Symlink`) by intercepting `readlink(2)` and `symlink(2)`. Targets should sync remotely as small encrypted metadata placeholder files on Drive (`proton-vfs-symlink: <target>`).
|
||||
* **Desktop File Manager Integration (Horizon 2)**: Develop shell extension plugins for popular Linux file managers (Nautilus Python plugin, Dolphin Service Menus, Thunar custom actions) to offer context-menu shortcuts ("Pin", "Unpin", "Copy Share Link", "View Version History") and file overlay sync badges.
|
||||
* **Integrated Photo Gallery Enhancements (Horizon 2)**: Support creating and editing Proton Photo Albums (album *writes* are unported in the SDK too) and create an auto-upload Pictures folder pipeline. Exif display, the date-based timeline scrubber and favourites are implemented.
|
||||
* **Integrated Photo Gallery Enhancements (Horizon 2)**: Album *editing* — rename, cover photo, delete, and removing photos from an album — is still unported in the SDK (TS `updateAlbum` / `deleteAlbum` / `removePhotos`), as is adding a photo that lives on someone else's volume (TS `copyPhoto`). Album **creation** and adding own-volume photos landed with the Google Photos import. Still open: an auto-upload Pictures folder pipeline, and a GUI for creating albums by hand. Exif display, the date-based timeline scrubber and favourites are implemented.
|
||||
* **Google Photos import follow-ups**: the importer uploads photos one at a time — a large export would finish sooner with a bounded concurrent upload pool. Trash folders are recognized by a hardcoded list of localized names ([`takeout.rs`](../crates/pdfs-core/src/takeout.rs) `TRASH_FOLDERS`); a locale not on it imports its trash as ordinary photos. Motion-photo `.MP`/`.MV` sidecar videos are ignored rather than uploaded as related photos. Live validation against a real export is pending.
|
||||
* **Random-Access Media Streaming seek support (Horizon 2)**: Intercept out-of-order sparse reads (player seeks) in `pdfs-fuse` and prioritize downloading blocks around the seek offset, aborting/postponing sequential pre-fetches.
|
||||
* **Dynamic Sync Dashboard & Queue Visualization (Horizon 2)**: GUI real-time transfer list (progress bars, speed, ETA), sync history feed, storage usage breakdown, and global pause/resume sync controls.
|
||||
* **Interactive Terminal UI (TUI) Mode (Horizon 3)**: Implement `pdfs tui` using the `ratatui` crate to show transfer speed, sync status, active transfer queue with progress bars, and scrolling daemon logs.
|
||||
|
||||
@@ -1,110 +0,0 @@
|
||||
# Search index + prompt display fixes
|
||||
|
||||
Working plan. Findings come from an audit of a real `cache.db`
|
||||
(9,252 Drive nodes, 110,187 local files) on 2026-08-12.
|
||||
|
||||
## Evidence
|
||||
|
||||
```
|
||||
nodes 9252
|
||||
nodes_fts 6547 ← 2705 rows never indexed
|
||||
nodes in orphan subtrees 2705 ← exactly the missing set
|
||||
local_files 110187
|
||||
/home/narl/go/pkg/mod 63399 (58% of the local index)
|
||||
query "test" 46405 local matches, candidate pool capped at 1000
|
||||
top 500 by fts rank 367 (73%) from go/pkg/mod
|
||||
```
|
||||
|
||||
`pdfs search tickets` returns only `Documents/Tickets` (My Files); the
|
||||
`tickets.pdf` sitting in the device folder `Downloads` is invisible.
|
||||
|
||||
## P1 — 29% of Drive is unsearchable
|
||||
|
||||
`node_is_indexable_tx` (`crates/pdfs-core/src/db/nodes.rs:606`) indexes a node
|
||||
only when the ancestor walk terminates at a row with `parent_uid IS NULL`. The
|
||||
DB holds exactly one such row (the My Files root). Device roots live in
|
||||
`device`/`sync_folder`, never in `nodes`, so every device-folder subtree
|
||||
(`Downloads`, `Pictures`, `Documents`, `Music`, `Videos` under device
|
||||
*nthinkpad*) walks up to a `parent_uid` with no row and is judged unindexable.
|
||||
Stale `pdfs-acceptance-*` roots hit the same path.
|
||||
|
||||
- [x] Relax the rule to "the walk terminates and no ancestor is trashed",
|
||||
keeping the existing cycle guard. A subtree whose top row's parent is
|
||||
simply not cached is still real, reachable data.
|
||||
- [x] `MIGRATION_V21`: rebuild `nodes_fts` under the new rule, so existing
|
||||
installs recover the missing rows instead of waiting for each node to be
|
||||
upserted again.
|
||||
- [x] Tests: a node under an uncached parent is indexed; a trashed ancestor
|
||||
still excludes; the migration backfills.
|
||||
|
||||
### Found while testing: a parent cycle hangs the daemon
|
||||
|
||||
Writing the cycle test surfaced a live defect, not a test artefact. A query
|
||||
below the trigram minimum takes the `LIKE` lane, which reads `nodes` directly
|
||||
and never consults the index, and every row it returns goes through `path_of` —
|
||||
whose parent walk is an unguarded `UNION ALL`. On a `parent_uid` cycle it never
|
||||
terminates, while holding the daemon's only SQLite connection.
|
||||
|
||||
- [x] Depth-cap `path_of` (`db/utils.rs`), matching the 1024 cap
|
||||
`path_relative_to` already carried.
|
||||
- [x] Depth-cap the `descendants` CTE in `upsert_node_tx`, which has the same
|
||||
exposure on the write side.
|
||||
- [x] Keep cycles out of the index (part of the P1 rule above).
|
||||
|
||||
Still unguarded, not touched here: `pin_is_pinned`'s ancestor walk.
|
||||
|
||||
## P2 — local index noise crowds real files out of the candidate pool
|
||||
|
||||
`SKIP_DIRS` (`crates/pdfs-core/src/localindex.rs:23`) misses the Go module
|
||||
cache and build output dirs. `search_v2` clamps candidates to ≤1000 and scores
|
||||
*after* SQLite, so for any common term the pool is exhausted by vendored files
|
||||
before a real one is considered.
|
||||
|
||||
- [x] Skip `go/pkg/mod` (path-relative, not a bare name), `dist`, `build`,
|
||||
`.cache`-style artefact dirs.
|
||||
- [x] Give `search_local_candidates` a name-prefix lane, mirroring what
|
||||
`search_candidates` already does for Drive, so an exact-prefix match
|
||||
cannot be starved by trigram rank.
|
||||
- [x] Tests for both.
|
||||
|
||||
## P3 — display lies about where a hit lives
|
||||
|
||||
- [x] `Hit::location()` (`crates/pdfs-gui/src/prompt.rs:150`) hardcodes
|
||||
`My files / …` for every Drive hit. Derive the label from
|
||||
`mounted_path` when the daemon resolved one (device folder, sync folder)
|
||||
and fall back to `My files` only for the primary mount.
|
||||
- [x] Pin rows set `is_dir: pin.recursive` (`dmenu.rs:133`, `prompt.rs:872`) —
|
||||
a non-recursively pinned folder renders and activates as a file.
|
||||
- [ ] `pdfs locations` prints `Device folder 1..5` rather than the device and
|
||||
folder name. (Cosmetic; deferred.)
|
||||
|
||||
## Verification
|
||||
|
||||
Done, 2026-08-12:
|
||||
|
||||
```
|
||||
cargo fmt --all -- --check clean
|
||||
cargo clippy --workspace --all-targets -- -D warnings clean
|
||||
cargo test --workspace --locked 439 passed, 0 failed
|
||||
```
|
||||
|
||||
The V21 backfill was also replayed against a read-only `.backup` of the live
|
||||
`cache.db`:
|
||||
|
||||
```
|
||||
live_nodes 9276
|
||||
indexed 9276 (was 6547)
|
||||
downloads_subtree 300
|
||||
tickets.pdf -> Downloads/tickets.pdf
|
||||
```
|
||||
|
||||
That run is what caught the last defect: the backfill originally seeded an
|
||||
uncached-parent root with an empty path, so a node indexed by the migration
|
||||
(`tickets.pdf`) and the same node after a later upsert (`Downloads/tickets.pdf`)
|
||||
disagreed. The seed now keeps the name for any row that has a parent, matching
|
||||
`path_of`.
|
||||
|
||||
Still to do, on the live system: rebuild, restart `proton-drive.service`, and
|
||||
confirm `pdfs search tickets` returns the device folder's `tickets.pdf`. The
|
||||
schema move to 21 is one-way — an older `pdfs` refuses a v21 database. The
|
||||
`local_files` shrink (~110k → ~47k) lands with the next background scan.
|
||||
@@ -0,0 +1,26 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg width="128" height="128" viewBox="0 0 128 128" fill="none" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Proton Drive for Linux">
|
||||
<defs>
|
||||
<linearGradient id="pdl-bg" x1="0" y1="0" x2="0" y2="128" gradientUnits="userSpaceOnUse">
|
||||
<stop offset="0" stop-color="#8B72FF"/>
|
||||
<stop offset="1" stop-color="#5A38E8"/>
|
||||
</linearGradient>
|
||||
</defs>
|
||||
|
||||
<rect width="128" height="128" rx="29" fill="url(#pdl-bg)"/>
|
||||
|
||||
<!-- cloud: the remote drive (symmetric about x=64) -->
|
||||
<g fill="#FFFFFF">
|
||||
<circle cx="64" cy="42" r="20"/>
|
||||
<circle cx="44" cy="52" r="14"/>
|
||||
<circle cx="84" cy="52" r="14"/>
|
||||
<rect x="32" y="50" width="64" height="16" rx="8"/>
|
||||
</g>
|
||||
|
||||
<!-- pull: files arrive only when opened -->
|
||||
<g stroke="#FFFFFF" stroke-width="8" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M64 75v17"/>
|
||||
<path d="M53 84l11 11 11-11"/>
|
||||
<path d="M45 108h38" stroke-opacity="0.55"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 990 B |
+7
-5
@@ -9,18 +9,20 @@
|
||||
# Build + install: cd packaging && makepkg -fi
|
||||
|
||||
pkgname=proton-drive-linux
|
||||
pkgver=1.6.0
|
||||
pkgver=1.10.1
|
||||
pkgrel=1
|
||||
pkgdesc="Proton Drive client for Linux: FUSE files-on-demand mount, CLI, GTK4 app + tray"
|
||||
pkgdesc="Unofficial Proton Drive client for Linux: files-on-demand FUSE mount, CLI, GTK4 app + tray"
|
||||
arch=('x86_64')
|
||||
url="https://github.com/narl/proton-drive-linux"
|
||||
url="https://github.com/narrrl/proton-drive-linux"
|
||||
license=('MIT')
|
||||
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0')
|
||||
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0' 'perl-image-exiftool')
|
||||
makedepends=('cargo' 'git')
|
||||
provides=('pdfs')
|
||||
# Files-on-demand mount + keyring need these at runtime.
|
||||
optdepends=('gnome-keyring: credential storage via Secret Service'
|
||||
'xdg-utils: "Open folder" action from the tray')
|
||||
'xdg-utils: "Open folder" action from the tray'
|
||||
'fzf: live search-as-you-type launcher (pdfs-prompt --fzf)'
|
||||
'foot: terminal for pdfs-prompt --fzf started from a keybinding')
|
||||
options=('!lto' '!debug')
|
||||
|
||||
# Built in place from the checkout; nothing to fetch or checksum.
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
[Desktop Entry]
|
||||
Type=Application
|
||||
Name=Proton Drive (Tray)
|
||||
Name=Proton Drive for Linux (Tray)
|
||||
GenericName=Cloud Drive Tray
|
||||
Comment=Proton Drive system tray icon
|
||||
Comment=Proton Drive status and quick actions (unofficial client)
|
||||
Exec=pdfs-tray
|
||||
Icon=io.narl.proton-drive-linux
|
||||
Terminal=false
|
||||
Categories=Network;FileTransfer;GTK;
|
||||
Keywords=proton;drive;cloud;sync;storage;
|
||||
Keywords=proton;drive;cloud;sync;storage;files;backup;
|
||||
NoDisplay=true
|
||||
@@ -1,12 +1,12 @@
|
||||
[Desktop Entry]
|
||||
Type=Application
|
||||
Name=Proton Drive
|
||||
Name=Proton Drive for Linux
|
||||
GenericName=Cloud Drive
|
||||
Comment=Proton Drive files-on-demand for Linux
|
||||
Comment=Files-on-demand Proton Drive (unofficial client)
|
||||
Exec=pdfs-app
|
||||
Icon=io.narl.proton-drive-linux
|
||||
Terminal=false
|
||||
Categories=Network;FileTransfer;GTK;
|
||||
Keywords=proton;drive;cloud;sync;storage;
|
||||
Keywords=proton;drive;cloud;sync;storage;files;backup;
|
||||
StartupNotify=true
|
||||
SingleMainWindow=true
|
||||
@@ -1,14 +1,26 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg width="128" height="128" viewBox="0 0 128 128" xmlns="http://www.w3.org/2000/svg">
|
||||
<svg width="128" height="128" viewBox="0 0 128 128" fill="none" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Proton Drive for Linux">
|
||||
<defs>
|
||||
<linearGradient id="g" x1="0" y1="0" x2="0" y2="1">
|
||||
<stop offset="0" stop-color="#8b6dff"/>
|
||||
<stop offset="1" stop-color="#6d4aff"/>
|
||||
<linearGradient id="pdl-bg" x1="0" y1="0" x2="0" y2="128" gradientUnits="userSpaceOnUse">
|
||||
<stop offset="0" stop-color="#8B72FF"/>
|
||||
<stop offset="1" stop-color="#5A38E8"/>
|
||||
</linearGradient>
|
||||
</defs>
|
||||
<rect x="8" y="8" width="112" height="112" rx="26" fill="url(#g)"/>
|
||||
<!-- cloud -->
|
||||
<path fill="#ffffff" d="M44 84c-11 0-20-9-20-20 0-10 7-18 17-20 3-11 13-19 25-19 13 0 24 10 26 23 9 1 16 9 16 18 0 10-8 18-18 18z"/>
|
||||
<!-- down arrow (on-demand pull) -->
|
||||
<path d="M64 50v22m0 0l-9-9m9 9l9-9" stroke="#6d4aff" stroke-width="7" stroke-linecap="round" stroke-linejoin="round" fill="none"/>
|
||||
|
||||
<rect width="128" height="128" rx="29" fill="url(#pdl-bg)"/>
|
||||
|
||||
<!-- cloud: the remote drive (symmetric about x=64) -->
|
||||
<g fill="#FFFFFF">
|
||||
<circle cx="64" cy="42" r="20"/>
|
||||
<circle cx="44" cy="52" r="14"/>
|
||||
<circle cx="84" cy="52" r="14"/>
|
||||
<rect x="32" y="50" width="64" height="16" rx="8"/>
|
||||
</g>
|
||||
|
||||
<!-- pull: files arrive only when opened -->
|
||||
<g stroke="#FFFFFF" stroke-width="8" stroke-linecap="round" stroke-linejoin="round">
|
||||
<path d="M64 75v17"/>
|
||||
<path d="M53 84l11 11 11-11"/>
|
||||
<path d="M45 108h38" stroke-opacity="0.55"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 718 B After Width: | Height: | Size: 990 B |
@@ -12,9 +12,9 @@
|
||||
Name: proton-drive-linux
|
||||
Version: %{version}
|
||||
Release: 1%{?dist}
|
||||
Summary: Proton Drive client for Linux (FUSE, CLI, GTK4 app + tray)
|
||||
Summary: Unofficial Proton Drive client for Linux (FUSE, CLI, GTK4 app + tray)
|
||||
License: MIT
|
||||
URL: https://github.com/narl/proton-drive-linux
|
||||
URL: https://github.com/narrrl/proton-drive-linux
|
||||
ExclusiveArch: x86_64
|
||||
|
||||
BuildRequires: cargo
|
||||
@@ -33,6 +33,7 @@ Requires: gtk4
|
||||
Requires: libadwaita
|
||||
Requires: libsecret
|
||||
Requires: webkitgtk6.0
|
||||
Requires: perl-Image-ExifTool
|
||||
Requires: xdg-utils
|
||||
|
||||
# DE-specific; do not Require a single desktop environment.
|
||||
@@ -40,11 +41,16 @@ Recommends: gnome-keyring
|
||||
Recommends: gnome-shell-extension-appindicator
|
||||
Recommends: kwallet
|
||||
|
||||
# Only for `pdfs-prompt --fzf`; every other front end works without them.
|
||||
Suggests: fzf
|
||||
Suggests: foot
|
||||
|
||||
Provides: pdfs = %{version}-%{release}
|
||||
|
||||
%description
|
||||
Unofficial Proton Drive client featuring a FUSE files-on-demand mount,
|
||||
CLI, GTK4/Libadwaita GUI, system tray, and search launcher.
|
||||
Files-on-demand Proton Drive for the Linux desktop: a FUSE virtual mount with
|
||||
block-level caching, a scriptable CLI, and a GTK4/Libadwaita desktop app with
|
||||
system tray and search launcher. Unofficial and not affiliated with Proton AG.
|
||||
|
||||
%prep
|
||||
# In-tree build: no Source tarball. Pass --define "git_dir /path/to/checkout".
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[Unit]
|
||||
Description=Proton Drive Files-On-Demand mount
|
||||
Documentation=https://github.com/narl/proton-drive-linux
|
||||
Documentation=https://github.com/narrrl/proton-drive-linux
|
||||
# Tie the mount to the desktop session: start after it is up and stop with it.
|
||||
After=graphical-session.target
|
||||
PartOf=graphical-session.target
|
||||
|
||||
Reference in new issue
Block a user