Compare commits

...
Author SHA1 Message Date
nvrl 26375885ed updated workflow 2026-07-22 03:04:59 +02:00
nvrl 544d0913e8 added fuse compliance testing 2026-07-22 03:04:40 +02:00
nvrl bef39f75cf sync refactor 2026-07-22 02:22:40 +02:00
nvrl 0a1fa95eb4 refactor 2026-07-22 02:17:19 +02:00
nvrl 242a28bb8f refactor from monolithic structure 2026-07-22 02:02:26 +02:00
nvrl bd17d506f8 fixed rm before online del, mirror bug 2026-07-22 01:52:10 +02:00
nvrl 1cd6a26be3 fix(fuse,core): remediate audit bugs B23, B26-B29, B33 and update BUGS tracking
- B23: Generate synthetic StagedWrite sidecars in rescue_scratch when sidecar is corrupted
- B26: Add open_count and unlinked state fields to defer node removal and cache eviction on unlinked open files until release
- B27: Discard orphaned write revisions for local UIDs unlinked prior to upload
- B28: Add emergency_evict to ContentCache for LRU blob eviction upon ENOSPC
- B29: Implement FUSE forget handler and forget_lookup in state to prune unreferenced inodes
- B33: Quarantine failing drain ops on HTTP 40x / max retries to unblock offline sync queue
- docs: Track audit items B18-B36 in docs/BUGS.md with fix details and verification results
2026-07-22 01:18:35 +02:00
nvrl fe75998cea fix(filesystem): resolve critical data loss, POSIX non-compliance, and deadlock vulnerabilities
- Standardize POSIX unlink (EISDIR) and rmdir (ENOTDIR/ENOTEMPTY) checks
- Defer local directory eviction in mode switch until FUSE mount verification succeeds
- Enforce fsync before atomic renames in ContentCache store operations
- Add directory ancestor loop check in rename (EINVAL)
- Implement punch_hole interval tracking in fallocate
- Purge un-created local node write revisions on unlink
- Fix control socket permission race with octal umask 0o77
- Add unit tests for state, DB, and FUSE POSIX logic (208/208 tests passing)
2026-07-22 00:23:55 +02:00
nvrl 9d55ffb9f6 fixed io lock with alloced files from for example aria2c 2026-07-21 23:23:20 +02:00
nvrl 2275004f79 impplemented fallocate 2026-07-21 22:39:35 +02:00
nvrl 914ff2b5ae added ioctl handler 2026-07-21 16:42:41 +02:00
nvrl 01677f2670 updated documentation on bugs/features 2026-07-21 10:35:53 +02:00
nvrl 3f9d36607a updated readme 2026-07-21 02:12:50 +02:00
nvrl f1399aa661 Merge pull request #13 from narrrl/feature/selective-sync
Feature/selective sync + diagnostics
2026-07-20 23:09:47 +02:00
nvrl a3d9d05777 added settings for this device + quota in settings 2026-07-20 23:08:23 +02:00
nvrl bf625bf8d9 added functionality to open shared files with you + fixed bug where mid transfer the SIGTERM signal can corrupt the mount 2026-07-20 22:30:16 +02:00
nvrl e0f0c7e0b7 added json output 2026-07-20 20:44:37 +02:00
nvrlandClaude Opus 4.8 a6a925667a add --json output to the query commands
Scripts and health checks had to scrape formatted text, which changes whenever
the display does.

--json is global and applies to status, ls, pins, sync list, devices list,
transfers, activity, and cache inspect. Action commands keep their human
output; a script that needs to know whether one worked has the exit code.

Two decisions worth recording:

The daemon's own reply is serialized rather than a hand-built parallel
structure, so new fields reach the JSON without anyone remembering to add them
-- but the variant tag is stripped first. Serializing CtlResponse directly
yields {"SyncFolders": {"items": [...]}}, which would make every caller write
jq '.SyncFolders.items[]' and couple it to the name of an internal enum
variant. Callers get {"items": [...]}.

A daemon-side error prints its payload and then fails the command. The first
cut exited 0 after printing the error body, which was caught by running
`pdfs --json ls NoSuchFolder`: a script would have read a failed lookup as
success. The body is still emitted so `kind` is available, but the exit code
is what `set -e` and `if pdfs ...` actually test.

status composes its object by hand instead, being the one query that merges
the local keyring with the daemon -- including the logged-in-but-no-daemon
case, which has no daemon reply to pass through at all.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFzNQyb4TWrCRWcnegqWxY
2026-07-20 20:38:48 +02:00
nvrlandClaude Opus 4.8 e259016a00 checkpoint the WAL after the vacuum, not before
Verified against a live install: vacuum reported "34.3 MiB freed (170.3 MiB
-> 136.0 MiB), 0 WAL frame(s) checkpointed" while the WAL grew from 67 MiB to
143 MiB. Net disk use barely moved, and the frame count was zero because the
checkpoint ran before the VACUUM, when there was nothing to fold.

Reordering is strictly better, but the commit message should not overstate it:
the live inflation happens because the daemon's other connections hold read
transactions and a TRUNCATE checkpoint waits on readers, so under a busy
daemon this can still move little. journal_size_limit stays the backstop --
the live WAL did fall back to its 64 MiB cap on its own afterwards.

The accompanying test deliberately does not assert on WAL size or frame count.
Measured both ways, a single-connection test reports frames=0 and a 0-byte WAL
under either ordering, because SQLite folds the log itself without a
concurrent reader to block it; an assertion there would pass on the bug and be
decoration. It asserts what is reliably true instead -- that a file-backed
vacuum reclaims space and leaves the database sound -- and the ordering
rationale plus the residual uncertainty are documented on vacuum() itself.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFzNQyb4TWrCRWcnegqWxY
2026-07-20 20:32:18 +02:00
nvrlandClaude Opus 4.8 471f57b8e1 add cache inspect/vacuum and a diagnose self-check
A corrupt database or an oversized cache previously left the user with no
answer but "delete the state directory and re-sync", which for a large
account means re-downloading everything to fix a problem that may have been
a few stale rows.

pdfs diagnose reports on paths, keyring, daemon, and mountpoint, and exits
non-zero if any check fails so it works in a health check. Every check is
written to run with no daemon, because that is the situation it gets run in:
a stopped daemon is reported as a finding rather than aborting the report.
Writability is tested by writing, since permission bits say nothing about a
full disk or a read-only remount.

pdfs cache inspect reports database size, reclaimable space, per-table row
counts, and cache usage; --deep adds SQLite's integrity check, opt-in because
it reads every page. The report distinguishes "checked and clean" from "not
checked" rather than letting an empty problem list imply a clean bill of
health it never gave.

pdfs cache vacuum checkpoints the WAL and compacts the database.

Deliberately no orphaned-cache-row pruning: ContentCache::reconcile already
rebuilds the index from what is on disk at every open, so a second pass here
would duplicate it against a worse view of the truth. The WAL is likewise
left to journal_size_limit, which already caps it -- a periodic checkpoint
would paper over a problem that no longer exists.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFzNQyb4TWrCRWcnegqWxY
2026-07-20 19:54:42 +02:00
nvrlandClaude Opus 4.8 3ade754ea2 add .pdfsignore selective sync for mirror folders
Mirror folders walked and uploaded every path under their root, so syncing a
project directory pushed node_modules/, target/, and .git/ to Drive -- quota
spent on files nobody wants there, and enough API traffic to invite rate
limiting.

Rules come from a .pdfsignore (or .protonignore) at the folder root, unioned
with an ignore_patterns list in config.json that defaults to the usual build,
VCS, and editor-leftover globs. Both are re-read at the start of every pass,
so edits apply without restarting the daemon.

The filter deliberately sits on the classification union in
classification_order(), not on walk_local alone. do_reconcile classifies
local | remote | baseline, so a file synced before a rule started matching it
is absent from local but present in the other two -- the "local deleted,
remote untouched" arm, which calls trash_nodes. Filtering only the walk would
have meant that adding a line to .pdfsignore silently trashed files on Drive
on the next pass. Ignored baseline rows are left in place so un-ignoring
re-adopts the existing remote file instead of re-uploading it.

guard_local_wipe takes a filtered baseline for the same reason: an ignored
path is absent from the walk by rule rather than by loss, and counting those
as missing would trip the guard on every pass once a rule covered the tree,
wedging that folder's sync permanently.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFzNQyb4TWrCRWcnegqWxY
2026-07-20 19:40:40 +02:00
nvrl 661ae3cf7a fixed incorrect versioning for rpm build 2026-07-20 17:06:36 +02:00
nvrl f803bf3cd7 fixed github ci 2026-07-20 15:06:17 +02:00
nvrl 084740ad95 version bump to 0.6.2 2026-07-20 14:54:13 +02:00
nvrl 65eed0cc72 Merge pull request #12 from narrrl/release/0.6.2
release/0.6.2
2026-07-20 14:53:06 +02:00
nvrl 1e0e284d5f per-chunk wakeup, B13 + B14 2026-07-20 14:52:31 +02:00
nvrl f0fbd0775c fixed long getattr load times with ls -l 2026-07-20 13:37:59 +02:00
nvrl d25a573e8f fixed mount-kind correlation 2026-07-20 00:09:00 +02:00
nvrl eafe1307b8 Merge pull request #11 from narrrl/fix/return-prompt
Fix/return prompt
2026-07-19 23:53:04 +02:00
nvrl 056ae0463f Merge pull request #10 from narrrl/fix/capture #
added webgtk recaptcha
2026-07-19 23:52:39 +02:00
nvrl b2b0fe7b99 fixed return key not opening file 2026-07-19 23:39:24 +02:00
55 changed files with 11236 additions and 3465 deletions

No files matched your search

+56
View File
@@ -0,0 +1,56 @@
name: CI
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
rust:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- name: Install native dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
pkg-config \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
- name: Install minimum supported Rust
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
toolchain: '1.96'
components: rustfmt, clippy
- uses: swatinem/rust-cache@v2
- name: Formatting
run: cargo fmt --all -- --check
- name: Lints
run: cargo clippy --workspace --all-targets -- -D warnings
- name: Unit tests
run: cargo test --workspace --locked
- name: Acceptance runner safety guard
run: |
if scripts/fuse-acceptance.sh /tmp >acceptance.out 2>&1; then
echo "runner unexpectedly accepted a non-FUSE path"
exit 1
fi
grep -q "refusing primary non-FUSE path" acceptance.out
# Real Proton credentials and /dev/fuse deliberately do not run on pull
# requests. Follow docs/TESTING.md on a dedicated test-account runner.
+4 -1
View File
@@ -26,6 +26,7 @@ jobs:
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev \
desktop-file-utils \
@@ -98,7 +99,7 @@ jobs:
Priority: optional
Architecture: amd64
Maintainer: Nils Pukropp <contact@narl.io>
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0
Depends: fuse3, libgtk-4-1 (>= 4.12.0), libadwaita-1-0 (>= 1.5.0), libsecret-1-0, libwebkitgtk-6.0-4
Installed-Size: ${INSTALLED_SIZE}
Description: Proton Drive client for Linux
Proton Drive client for Linux featuring:
@@ -120,8 +121,10 @@ jobs:
# Build RPM package
rpmbuild -bb \
--nodeps \
--define "_topdir $PWD/rpmbuild" \
--define "_sourcedir $PWD" \
--define "git_dir $PWD" \
--define "version ${RPM_VERSION}" \
--define "_dbpath $PWD/rpmbuild/db" \
packaging/proton-drive-linux.spec
Generated
+4 -4
View File
@@ -3518,7 +3518,7 @@ checksum = "35fb2e5f958ec131621fdd531e9fc186ed768cbe395337403ae56c17a74c68ec"
[[package]]
name = "pdfs-cli"
version = "0.6.1"
version = "0.7.1"
dependencies = [
"anyhow",
"clap 4.6.2",
@@ -3533,7 +3533,7 @@ dependencies = [
[[package]]
name = "pdfs-core"
version = "0.6.1"
version = "0.7.1"
dependencies = [
"directories",
"ignore",
@@ -3552,7 +3552,7 @@ dependencies = [
[[package]]
name = "pdfs-fuse"
version = "0.6.1"
version = "0.7.1"
dependencies = [
"fuser",
"image",
@@ -3568,7 +3568,7 @@ dependencies = [
[[package]]
name = "pdfs-gui"
version = "0.6.1"
version = "0.7.1"
dependencies = [
"anyhow",
"async-channel",
+1 -1
View File
@@ -3,7 +3,7 @@ resolver = "2"
members = ["crates/pdfs-core", "crates/pdfs-fuse", "crates/pdfs-cli", "crates/pdfs-gui"]
[workspace.package]
version = "0.6.1"
version = "0.7.1"
edition = "2024"
license = "MIT"
authors = ["Nils Pukropp <contact@narl.io>"]
+123 -6
View File
@@ -6,11 +6,122 @@ A fast, unofficial Proton Drive client for Linux. This client features an advanc
- **Files-on-Demand (FUSE)**: Mount your Proton Drive as a virtual filesystem where files are downloaded only when opened, utilizing smart block-level caching and disk-backed writes.
- **Command-Line Interface (CLI)**: Manage your drive, authenticate, and monitor sync status directly from the terminal.
- **Non-Blocking GTK4 Desktop App**: Browse files, manage pins, and configure options through a modern, native GUI with a fully non-blocking asynchronous main loop.
- **Non-Blocking GTK4 Desktop App**: Browse files, manage pins, configure options, and monitor active transfers through a modern, native GUI with a fully non-blocking asynchronous main loop.
- **Shared Links & Invites**: Browse files shared with you by other users, and view/manage your own public shared links directly in the GUI.
- **Local Backup (Computers)**: Sync and back up local directories (like Downloads, Documents, Pictures, etc.) directly to your Proton Drive account.
- **System Tray Integration**: Background indicator for status monitoring, quick actions, and fast sync controls.
- **Fuzzy Search Launcher Prompt (HUD)**: A Google Drive-style search launcher (`pdfs-prompt`) for finding and opening files or folders instantly, ideal for binding to a system-wide hotkey.
- **Secure Credential Storage**: Integrates with the system Secret Service (GNOME Keyring, KWallet, etc.) with smart in-memory credential caching to avoid UI thread blockages.
- **Proton Photos Support**: Access your Proton Photos timeline, view thumbnails, and download backed-up media natively (available in the GUI as a navigation tab and via the CLI).
- **Human Verification (CAPTCHA) Recovery**: Detects sign-in gates (VPN/new IP challenges) and launches an embedded `WebKitWebView` dialog to safely complete the challenge, transparently retrying authentication with the earned token.
- **Selective Sync (`.pdfsignore`)**: Keep build trees, dependency directories, and editor leftovers out of synced folders using gitignore-style rules.
## Selective Sync (`.pdfsignore`)
Two-way synced folders skip paths matched by ignore rules, so syncing a project
directory does not upload `node_modules/`, `target/`, or `.git/`.
Rules come from two places, and both apply:
1. **Per folder** — a `.pdfsignore` file at the root of the synced folder
(`.protonignore` also works). Gitignore syntax, including negation:
```gitignore
# everything build-related
build/
*.log
# ...except this one
!important.log
```
2. **Globally** — an `ignore_patterns` list in `config.json`, applied to every
synced folder. When unset, sensible defaults apply: `.git/`, `.hg/`, `.svn/`,
`node_modules/`, `target/`, `.venv/`, `__pycache__/`, `*~`, `*.swp`, `*.tmp`,
`.DS_Store`, and `Thumbs.db`.
```json
{
"ignore_patterns": ["node_modules/", "target/", "*.iso"]
}
```
Set it to `[]` to opt out of the defaults entirely.
Rules are re-read at the start of every sync pass, so edits take effect on the
next pass without restarting the daemon.
**Ignoring is never destructive.** If a rule starts matching a file that was
already synced, its copy on Drive is left untouched — the file simply stops
being tracked. Removing the rule later picks the existing remote file back up
rather than re-uploading it.
## Diagnostics & Maintenance
When something looks wrong, `pdfs diagnose` checks the installation and prints a
report. It runs without a daemon on purpose — the state worth diagnosing is
usually the state where the daemon will not start:
```console
$ pdfs diagnose
Paths
[ok ] state dir: /home/you/.local/state/proton-drive-linux
[ok ] database: /home/you/.local/state/proton-drive-linux/cache.db (170.3 MiB)
Account
[ok ] keyring session: you@proton.me
Daemon
[ok ] daemon responding
[ok ] mounted at: /home/you/ProtonDrive
[ok ] queued writes: none
No problems found.
```
It exits non-zero if any check fails, so it works in a health-check script.
For the local metadata database and content cache:
| Command | What it does |
|---|---|
| `pdfs cache inspect` | Database size, reclaimable space, per-table row counts, cache usage against budget |
| `pdfs cache inspect --deep` | Also runs SQLite's integrity check — reads every page, so it is slow on a large database |
| `pdfs cache vacuum` | Checkpoints the write-ahead log and compacts the database |
| `pdfs cache clear` | Deletes cached file content, keeping pinned files |
`vacuum` takes a write lock for its duration and needs room for a second copy of
the database while it runs, so it is a deliberate operation rather than
something the daemon does on a timer.
Before a release, run the real-kernel acceptance suite against a dedicated test
account and mount. See [`docs/TESTING.md`](docs/TESTING.md); it includes optional
two-mount convergence checks and confines destructive operations to a fresh
test directory.
## Scripting (`--json`)
Query commands accept a global `--json` flag and emit machine-readable output:
```console
$ pdfs --json sync list | jq '.items[] | select(.state != "idle") | .local_path'
$ pdfs --json cache inspect | jq '.db_reclaimable_bytes'
$ pdfs --json status | jq -r '.mount.mountpoint'
```
Supported on `status`, `ls`, `pins`, `sync list`, `devices list`, `transfers`,
`activity`, and `cache inspect`. Commands that perform an action keep their
human output — a script that needs to know whether one worked has the exit code.
Two things worth relying on:
- **The payload is unwrapped.** Output is `{"items": […]}`, not the daemon's
internal `{"SyncFolders": {"items": […]}}`, so scripts never name an internal
variant.
- **Errors still fail.** A daemon-side error prints its JSON body (with a
machine-readable `kind`) on stdout *and* exits non-zero, so `set -e` and
`if pdfs …` behave as expected rather than treating a failed lookup as
success.
## Performance & Caching
@@ -32,11 +143,16 @@ The client includes several optimizations designed for high efficiency, a low me
<td align="center" width="50%"><img src="images/files.png" alt="Files Browser" width="100%"><br><sub><b>Files Browser</b></sub></td>
</tr>
<tr>
<td align="center" width="50%"><img src="images/photos.png" alt="Photos Timeline" width="100%"><br><sub><b>Photos Timeline</b></sub></td>
<td align="center" width="50%"><img src="images/prompt.png" alt="Search Launcher Prompt" width="100%"><br><sub><b>Search Launcher Prompt</b></sub></td>
<td align="center" width="50%"><img src="images/shared.png" alt="Shared Links" width="100%"><br><sub><b>Shared Links</b></sub></td>
<td align="center" width="50%"><img src="images/shared_with_me.png" alt="Shared with me" width="100%"><br><sub><b>Shared with me</b></sub></td>
</tr>
<tr>
<td align="center" colspan="2"><img src="images/settings.png" alt="Settings" width="50%"><br><sub><b>Settings</b></sub></td>
<td align="center" width="50%"><img src="images/computers.png" alt="Local Backups (Computers)" width="100%"><br><sub><b>Local Backups (Computers)</b></sub></td>
<td align="center" width="50%"><img src="images/photos.png" alt="Photos Timeline" width="100%"><br><sub><b>Photos Timeline</b></sub></td>
</tr>
<tr>
<td align="center" width="50%"><img src="images/prompt.png" alt="Search Launcher Prompt" width="100%"><br><sub><b>Search Launcher Prompt</b></sub></td>
<td align="center" width="50%"><img src="images/settings.png" alt="Settings" width="100%"><br><sub><b>Settings</b></sub></td>
</tr>
</table>
@@ -54,20 +170,21 @@ sudo apt-get install -y \
libfuse3-dev \
libgtk-4-dev \
libadwaita-1-dev \
libwebkitgtk-6.0-dev \
libsecret-1-dev \
libdbus-1-dev
```
### Arch Linux
```bash
sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus
sudo pacman -S --needed pkgconf fuse3 gtk4 libadwaita libsecret dbus webkitgtk-6.0
```
### Fedora (44+)
```bash
sudo dnf install -y \
pkgconf-pkg-config fuse3-devel gtk4-devel libadwaita-devel \
libsecret-devel dbus-devel glib2-devel cargo rust
webkitgtk6.0-devel libsecret-devel dbus-devel glib2-devel cargo rust
```
Runtime extras (pick your desktop):
File diff suppressed because it is too large. Load diff
+79 -9
View File
@@ -153,10 +153,12 @@ pub struct StagedWrite {
/// The identity of a remote revision, as far as we can observe it: a file whose
/// size and mtime both still match the ones a queued change was made against has
/// not been rewritten by anyone else in the meantime.
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct Baseline {
pub mtime: i64,
pub size: u64,
#[serde(default)]
pub hash: Option<String>,
}
/// Content cache rooted at a directory, with a sibling pin-registry file.
@@ -431,7 +433,12 @@ impl ContentCache {
// 64-char hex key with no `.`, so this appends rather than replaces.
// `store_thumbnail` cannot use the same idiom — see the note there.
let tmp = blob.with_extension("tmp");
std::fs::write(&tmp, bytes)?;
{
let mut f = std::fs::File::create(&tmp)?;
use std::io::Write;
f.write_all(bytes)?;
f.sync_all()?;
}
std::fs::rename(&tmp, &blob)?;
let meta = serde_json::to_vec(&Meta { mtime, size })?;
std::fs::write(self.meta_path(uid), meta)?;
@@ -464,6 +471,9 @@ impl ContentCache {
if std::fs::hard_link(src, &tmp).is_err() {
std::fs::copy(src, &tmp)?;
}
if let Ok(f) = std::fs::File::open(&tmp) {
let _ = f.sync_all();
}
let bytes = std::fs::metadata(&tmp)?.len();
std::fs::rename(&tmp, &blob)?;
let meta = serde_json::to_vec(&Meta { mtime, size })?;
@@ -526,7 +536,12 @@ impl ContentCache {
let tmp = self
.block_dir
.join(format!("{}.b{idx}.tmp", Self::key(uid)));
std::fs::write(&tmp, bytes)?;
{
let mut f = std::fs::File::create(&tmp)?;
use std::io::Write;
f.write_all(bytes)?;
f.sync_all()?;
}
std::fs::rename(&tmp, &blob)?;
std::fs::write(
self.block_meta(uid, idx),
@@ -733,14 +748,40 @@ impl ContentCache {
continue;
}
let side = Self::scratch_sidecar(&blob);
// Parsed, not merely present: an unreadable sidecar cannot drive a
// recovery, so the blob is rubbish like any other leftover.
if std::fs::read(&side)
if !side.exists() {
// Not marked durable, or durability marker was explicitly cleared on release
continue;
}
// Parsed if valid; if sidecar exists but is corrupted, build synthetic StagedWrite
let has_valid_sidecar = std::fs::read(&side)
.ok()
.and_then(|b| serde_json::from_slice::<StagedWrite>(&b).ok())
.is_none()
{
continue;
.is_some();
if !has_valid_sidecar {
if let Ok(meta) = std::fs::metadata(&blob)
&& meta.len() > 0
{
let blob_name = blob
.file_name()
.map(|n| n.to_string_lossy().to_string())
.unwrap_or_default();
let synthetic = StagedWrite {
uid: format!("recovered~{blob_name}"),
len: meta.len(),
base_size: meta.len(),
base_mtime: 0,
authored: vec![(0, meta.len())],
complete: true,
based_on: None,
};
if let Ok(data) = serde_json::to_vec(&synthetic) {
let _ = std::fs::write(&side, data);
} else {
continue;
}
} else {
continue;
}
}
let Some(name) = blob.file_name() else {
continue;
@@ -755,6 +796,34 @@ impl ContentCache {
}
}
/// Emergency eviction of unpinned blobs when local disk runs out of space (ENOSPC).
pub fn emergency_evict(&self) {
let pinned: HashSet<String> = self
.db
.pinned_uids()
.unwrap_or_default()
.iter()
.map(|uid| Self::key_str(uid))
.collect();
let content_dir = self.content_dir.clone();
// Force pool enforcement ignoring current cap check to free disk space immediately
let Ok(list) = self.db.cache_eviction_candidates(KIND_BLOB, 10) else {
return;
};
for (key, bytes) in list {
if pinned.contains(&key) {
continue;
}
let p = content_dir.join(&key);
let m = content_dir.join(format!("{key}.meta"));
let _ = std::fs::remove_file(&p);
let _ = std::fs::remove_file(&m);
let _ = self.db.cache_remove(&key);
self.blob_bytes.fetch_sub(bytes, Ordering::Relaxed);
break; // Evict one item at a time until space is freed
}
}
/// Writes rescued from an unclean shutdown, as `(blob, meta)` pairs for the
/// daemon to hand to the upload queue. A blob whose sidecar did not survive
/// the rescue is deleted here — nothing can address it.
@@ -1759,6 +1828,7 @@ mod tests {
based_on: Some(Baseline {
mtime: 100,
size: 40,
hash: None,
}),
};
let staged = c.stage_write(&meta, &scratch).unwrap();
+58
View File
@@ -6,6 +6,7 @@ use directories::ProjectDirs;
use serde::{Deserialize, Serialize};
use crate::error::{Error, Result};
use crate::syncignore::DEFAULT_IGNORE_PATTERNS;
/// Proton requires `external-drive-{name}@{semver}-{channel}` (channel ∈
/// stable/beta/alpha); a malformed value trips the 422 anti-abuse path.
@@ -34,6 +35,19 @@ pub struct AppConfig {
/// path here. Defaulted for configs predating the field.
#[serde(default)]
pub mountpoint: Option<String>,
/// Ignore patterns (gitignore syntax) applied to every mirror sync folder,
/// on top of each folder's own `.pdfsignore`. `None` means "use
/// [`DEFAULT_IGNORE_PATTERNS`]"; an explicit empty list disables the
/// defaults. Defaulted for configs predating the field.
#[serde(default)]
pub ignore_patterns: Option<Vec<String>>,
/// Device uid this machine has been *adopted* into, overriding the hostname
/// heuristic in `ensure_device`. Lives here rather than in `cache.db` so it
/// survives a fresh install — on a new machine the DB is empty by
/// definition, which is exactly when the pin matters. `None` means "resolve
/// by hostname". Defaulted for configs predating the field.
#[serde(default)]
pub device_uid: Option<String>,
}
impl Default for AppConfig {
@@ -43,6 +57,8 @@ impl Default for AppConfig {
user_agent: USER_AGENT.to_string(),
cache_budget: None,
mountpoint: None,
ignore_patterns: None,
device_uid: None,
}
}
}
@@ -53,6 +69,20 @@ impl AppConfig {
pub fn resolved_cache_budget(&self) -> u64 {
self.cache_budget.unwrap_or(DEFAULT_CACHE_BUDGET_BYTES)
}
/// The effective global sync ignore patterns: the user's explicit list, or
/// [`DEFAULT_IGNORE_PATTERNS`] when unset. An explicit empty list is
/// honoured as "no global patterns", which is why this is not a
/// `unwrap_or_default`.
pub fn resolved_ignore_patterns(&self) -> Vec<String> {
match &self.ignore_patterns {
Some(patterns) => patterns.clone(),
None => DEFAULT_IGNORE_PATTERNS
.iter()
.map(|s| (*s).to_string())
.collect(),
}
}
}
/// Resolved XDG locations for state, cache, and the default mountpoint.
@@ -271,6 +301,8 @@ mod tests {
user_agent: "test-agent/1.0".to_string(),
cache_budget: Some(1234),
mountpoint: Some("/tmp/x".to_string()),
ignore_patterns: Some(vec!["build/".to_string()]),
device_uid: Some("dev-uid".to_string()),
};
let json = serde_json::to_string(&config).unwrap();
let decoded: AppConfig = serde_json::from_str(&json).unwrap();
@@ -284,4 +316,30 @@ mod tests {
assert_eq!(default_config.app_version, APP_VERSION);
assert_eq!(default_config.user_agent, USER_AGENT);
}
#[test]
fn a_config_predating_ignore_patterns_still_loads() {
// Every installed config on disk looks like this; the field must not be
// required or the daemon would fall back to a default config on upgrade.
let json = r#"{"app_version":"v","user_agent":"u"}"#;
let decoded: AppConfig = serde_json::from_str(json).unwrap();
assert!(decoded.ignore_patterns.is_none());
assert_eq!(
decoded.resolved_ignore_patterns().len(),
DEFAULT_IGNORE_PATTERNS.len(),
"an absent list means the defaults apply"
);
}
#[test]
fn an_explicit_empty_ignore_list_disables_the_defaults() {
let config = AppConfig {
ignore_patterns: Some(Vec::new()),
..AppConfig::default()
};
assert!(
config.resolved_ignore_patterns().is_empty(),
"opting out must be expressible, so this cannot be unwrap_or_default"
);
}
}
+126
View File
@@ -136,6 +136,22 @@ pub enum Request {
/// persist it to config so the next mount keeps it. Replies with
/// [`Response::Ok`].
SetCacheBudget { bytes: u64 },
/// Report on the health of the metadata database and content cache: sizes,
/// row counts, and — when `deep` — SQLite's own integrity check. Replies
/// with [`Response::CacheReport`].
///
/// `deep` reads every page of the database, so it is opt-in: worth it when
/// diagnosing a suspected corruption, wasteful as a routine status call.
CacheInspect {
#[serde(default)]
deep: bool,
},
/// Checkpoint the write-ahead log and compact the database, replying with
/// [`Response::Ok`] reporting the bytes reclaimed.
///
/// Takes a write lock for the duration and needs room for a second copy of
/// the database, which is why it is user-invoked rather than periodic.
CacheVacuum,
/// Snapshot what the daemon is working on: in-flight transfers (active
/// uploads/downloads) and the longer jobs around them (scans, folder
/// skeletons, the local index, sync passes). Replies with
@@ -172,6 +188,11 @@ pub enum Request {
RenameDevice { uid: String, name: String },
/// Delete (deregister) a device by its uid. Replies with [`Response::Ok`].
DeleteDevice { uid: String },
/// Adopt an existing device as this machine's, pinning its uid in
/// `config.json` so a hostname change or reinstall re-attaches to it instead
/// of registering a duplicate (features.md 5.1). `uid: None` clears the pin.
/// Replies with [`Response::Ok`].
AdoptDevice { uid: Option<String> },
// ---- device folder sync (devices.md) ----------------------------------
/// Add a local folder to this machine's device, uploading its tree and
@@ -188,6 +209,13 @@ pub enum Request {
/// Force a reconcile pass: one folder by id, or all when `id` is `None`.
/// Replies with [`Response::Ok`].
SyncNow { id: Option<i64> },
/// List the folders under this machine's device that can be synced here,
/// each with a proposed local path (features.md 5.2). Replies with
/// [`Response::RestorableFolders`].
ListRestorableFolders,
/// Attach the given remote device folders to local paths and sync them down.
/// Replies with [`Response::Ok`].
RestoreSyncFolders { items: Vec<RestoreItem> },
// ---- sharing a node ---------------------------------------------------
/// Invite `emails` (Proton and/or external addresses, auto-detected) to the
@@ -243,6 +271,16 @@ pub enum Request {
/// [`Response::Entries`] (each entry carries its `uid`; `path` is empty since
/// the item lives outside the mount tree).
ListSharedWithMe,
/// List the children of a folder shared with me, addressed by `uid`. Replies
/// with [`Response::Entries`]; like [`Request::ListSharedWithMe`] the entries
/// carry a `uid` and an empty `path`, since the subtree lives outside the
/// mount. Lets a front-end browse into a shared folder.
ListSharedFolder { uid: String },
/// Download a file shared with me into the content cache, addressed by `uid`.
/// Replies with [`Response::FilePath`] so the front-end can open it with the
/// default app. The by-uid twin of [`Request::OpenFile`], which can only
/// address nodes inside the mount.
OpenSharedFile { uid: String },
/// Leave a shared node by its `uid`, giving up my access. Replies with
/// [`Response::Ok`].
LeaveShared { uid: String },
@@ -269,6 +307,11 @@ pub enum Request {
/// Remove a saved bookmark by its `token`. Replies with [`Response::Ok`].
DeleteBookmark { token: String },
// ---- account ----------------------------------------------------------
/// Total account storage usage (used/total, all Proton products, not just
/// Drive). Replies with [`Response::AccountQuota`].
AccountQuota,
// ---- activity ---------------------------------------------------------
/// Fetch the daemon's recent activity log, newest first, capped at `limit`
/// entries. Replies with [`Response::Activity`]. The log is persisted, so it
@@ -324,6 +367,11 @@ pub struct DeviceInfo {
/// front-end must not offer that as casually as removing another computer.
#[serde(default)]
pub this_device: bool,
/// Whether this device is the one *explicitly adopted* in `config.json`
/// ([`Request::AdoptDevice`]), as opposed to one matched by hostname. Only
/// an adopted device survives a hostname change or a reinstall.
#[serde(default)]
pub adopted: bool,
}
/// One synced local folder on this machine's device (in [`Response::SyncFolders`]).
@@ -353,6 +401,36 @@ pub struct SyncFolderInfo {
pub progress: Option<SyncProgress>,
}
/// A folder under this machine's device offered for restore (in
/// [`Response::RestorableFolders`]).
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct RestorableFolder {
/// Uid of the remote folder — the handle for [`RestoreItem`].
pub remote_uid: String,
/// Its remote name.
pub name: String,
/// Where the daemon *proposes* to put it: the path the profile recorded when
/// that path makes sense on this machine, else `~/<name>`. A front-end shows
/// this as an editable default, never applies it silently.
pub local_path: String,
/// Proposed mode, from the profile; `mirror` when unknown.
pub mode: String,
/// Already synced on this machine, so restoring it again would be a no-op.
/// Listed anyway so the picker shows the whole device.
#[serde(default)]
pub already_synced: bool,
}
/// One folder to restore, in [`Request::RestoreSyncFolders`].
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct RestoreItem {
pub remote_uid: String,
/// Absolute local path to sync it to; created if missing.
pub local_path: String,
/// `mirror` or `ondemand`.
pub mode: String,
}
/// Which stage a running sync pass is in, in a [`SyncProgress`].
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
pub enum SyncPhase {
@@ -817,6 +895,28 @@ pub enum Response {
},
/// A human-readable success message.
Ok { message: String },
/// Health report for the metadata database and content cache (reply to
/// [`Request::CacheInspect`]).
CacheReport {
/// Schema version the database file is at.
schema_version: i64,
/// Bytes the database file accounts for.
db_bytes: u64,
/// Bytes a vacuum could hand back.
db_reclaimable_bytes: u64,
/// Row counts per table, in display order.
tables: Vec<(String, i64)>,
/// Bytes of cached content blobs, blocks, and thumbnails.
cache_used: u64,
/// Configured soft byte cap (`0` = unlimited).
cache_budget: u64,
/// Problems SQLite reported. Empty means sound; absent when the caller
/// did not ask for a deep check.
integrity_problems: Vec<String>,
/// Whether the integrity check actually ran, so an empty problem list
/// is not mistaken for a clean bill of health it never gave.
integrity_checked: bool,
},
/// The pin registry.
Pins { pins: Vec<Pin> },
/// A directory listing (reply to [`Request::ListDir`]) or a trash listing
@@ -860,6 +960,8 @@ pub enum Response {
Devices { items: Vec<DeviceInfo> },
/// This device's synced folders (reply to [`Request::ListSyncFolders`]).
SyncFolders { items: Vec<SyncFolderInfo> },
/// Folders offered for restore (reply to [`Request::ListRestorableFolders`]).
RestorableFolders { items: Vec<RestorableFolder> },
/// A node's share: members + pending invitations, and its public link if any
/// (reply to [`Request::ListShare`]).
Share {
@@ -873,6 +975,11 @@ pub enum Response {
Invitations { items: Vec<InvitationInfo> },
/// Saved public links (reply to [`Request::ListBookmarks`]).
Bookmarks { items: Vec<BookmarkInfo> },
/// Account storage usage in bytes (reply to [`Request::AccountQuota`]).
/// `max_space` is the total; `used_space` what is consumed. `max_space == 0`
/// means the total is unknown, which a front-end shows as a plain used figure
/// rather than a full bar.
AccountQuota { max_space: i64, used_space: i64 },
/// Nodes I have shared with others (reply to [`Request::ListSharedByMe`]).
SharedByMe { items: Vec<SharedItem> },
/// The daemon's recent activity, newest first (reply to
@@ -1123,6 +1230,18 @@ mod tests {
mode: "ondemand".into(),
},
Request::SyncNow { id: Some(3) },
Request::AdoptDevice {
uid: Some("dev-1".into()),
},
Request::AdoptDevice { uid: None },
Request::ListRestorableFolders,
Request::RestoreSyncFolders {
items: vec![RestoreItem {
remote_uid: "vol~link".into(),
local_path: "/home/me/Docs".into(),
mode: "mirror".into(),
}],
},
Request::ShareNode {
path: "a/b".into(),
emails: vec!["x@proton.me".into(), "y@example.com".into()],
@@ -1154,6 +1273,12 @@ mod tests {
Request::ListSharedByMe,
Request::ListActivity { limit: 100 },
Request::ListSharedWithMe,
Request::ListSharedFolder {
uid: "vol~link".into(),
},
Request::OpenSharedFile {
uid: "vol~link".into(),
},
Request::LeaveShared {
uid: "vol~link".into(),
},
@@ -1168,6 +1293,7 @@ mod tests {
Request::DeleteBookmark {
token: "tok".into(),
},
Request::AccountQuota,
];
for req in reqs {
let line = serde_json::to_string(&req).unwrap();
+335
View File
@@ -0,0 +1,335 @@
//! Database health reporting and repair (features.md 4.3).
//!
//! Everything here exists for the case where something has already gone wrong:
//! a crash mid-write, a full disk, a cache that has grown out of proportion to
//! what it holds. The daemon never calls these on its own — they are what
//! `pdfs cache inspect`, `pdfs cache vacuum`, and `pdfs diagnose` are built on,
//! so a user with a suspect install can get an answer without being told to
//! delete their state directory and re-sync from scratch.
use rusqlite::OptionalExtension;
use super::Db;
use crate::Result;
/// A point-in-time picture of the database's size and contents.
///
/// Sizes are computed from SQLite's own page accounting rather than by stat'ing
/// the file, so they stay right when the file is sparse or a checkpoint is
/// pending.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct DbStats {
/// Schema version the file is currently at.
pub schema_version: i64,
/// Total pages allocated to the database.
pub page_count: i64,
/// Pages allocated but unused — what a `VACUUM` would reclaim.
pub free_pages: i64,
/// Bytes per page.
pub page_size: i64,
/// Row counts for the tables worth reporting, in display order.
pub tables: Vec<(String, i64)>,
}
impl DbStats {
/// Bytes the database file accounts for.
pub fn total_bytes(&self) -> u64 {
(self.page_count * self.page_size).max(0) as u64
}
/// Bytes a `VACUUM` could hand back to the filesystem.
pub fn reclaimable_bytes(&self) -> u64 {
(self.free_pages * self.page_size).max(0) as u64
}
}
/// What a maintenance run reclaimed.
///
/// Deliberately does not carry an orphaned-row count: the content cache already
/// rebuilds its index from what is actually on disk every time it opens (see
/// [`crate::cache::ContentCache`]'s `reconcile`), so a second pruning pass here
/// would be a duplicate of that logic with a worse view of the truth.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub struct VacuumOutcome {
/// Database bytes before the run.
pub before_bytes: u64,
/// Database bytes after it.
pub after_bytes: u64,
/// WAL frames written back into the database by the checkpoint.
pub wal_frames_checkpointed: i64,
}
impl VacuumOutcome {
/// Bytes handed back to the filesystem. Saturating: a database can legitimately
/// grow across a vacuum (a checkpoint folds WAL frames in), and reporting that
/// as a huge positive number via wraparound would be worse than reporting zero.
pub fn freed_bytes(&self) -> u64 {
self.before_bytes.saturating_sub(self.after_bytes)
}
}
/// The tables `inspect` reports on, in the order a reader wants them.
const REPORTED_TABLES: &[&str] = &[
"nodes",
"cache_entries",
"pins",
"photos",
"trash",
"device",
"sync_folder",
"sync_entry",
"pending_op",
"activity",
"local_files",
];
impl Db {
/// SQLite's own consistency check.
///
/// Returns an empty vector when the database is sound; otherwise each entry
/// is one problem SQLite found. Read-only and safe against a live daemon,
/// though it reads every page, so it is not free on a large file.
pub fn integrity_check(&self) -> Result<Vec<String>> {
let conn = self.conn.lock();
let mut stmt = conn.prepare("PRAGMA integrity_check")?;
let rows = stmt.query_map([], |r| r.get::<_, String>(0))?;
let mut problems = Vec::new();
for row in rows {
let row = row?;
// A sound database reports the single line "ok"; anything else is a
// finding.
if row != "ok" {
problems.push(row);
}
}
Ok(problems)
}
/// Size and row-count snapshot for `pdfs cache inspect`.
pub fn stats(&self) -> Result<DbStats> {
let conn = self.conn.lock();
let page_count: i64 = conn.query_row("PRAGMA page_count", [], |r| r.get(0))?;
let free_pages: i64 = conn.query_row("PRAGMA freelist_count", [], |r| r.get(0))?;
let page_size: i64 = conn.query_row("PRAGMA page_size", [], |r| r.get(0))?;
// The schema version lives in the `sync_state` key/value table, not in
// `PRAGMA user_version` — see `migrations`.
let schema_version: i64 = conn
.query_row(
"SELECT value FROM sync_state WHERE key = 'schema_version'",
[],
|r| r.get::<_, String>(0),
)
.optional()?
.and_then(|v| v.parse().ok())
.unwrap_or(0);
let mut tables = Vec::new();
for name in REPORTED_TABLES {
// A table listed here may not exist yet on an older schema, and a
// report is not worth failing over a missing one.
let exists: Option<String> = conn
.query_row(
"SELECT name FROM sqlite_master WHERE type='table' AND name=?1",
[name],
|r| r.get(0),
)
.optional()?;
if exists.is_none() {
continue;
}
let count: i64 =
conn.query_row(&format!("SELECT count(*) FROM \"{name}\""), [], |r| {
r.get(0)
})?;
tables.push(((*name).to_string(), count));
}
Ok(DbStats {
schema_version,
page_count,
free_pages,
page_size,
tables,
})
}
/// Fold the WAL back into the database and compact the file.
///
/// `VACUUM` rewrites the whole database into a fresh file, so it needs room
/// for a second copy and takes a write lock for the duration. That makes it
/// a deliberate, user-invoked operation rather than something the daemon
/// does on a timer.
///
/// The checkpoint runs *after* the vacuum. Checkpointing only beforehand
/// folds a log that the vacuum is about to rewrite anyway, and leaves the
/// rewrite itself sitting in the WAL: measured on a live install, a 170 MiB
/// database vacuumed to 136 MiB while its WAL grew 67 MiB → 143 MiB,
/// erasing the reclaim in net disk terms and reporting `0 frames`
/// checkpointed because it ran when there was nothing to fold.
///
/// Ordering it afterwards is strictly better but is *not* a proven fix for
/// that inflation. The live case inflates because the daemon's other
/// connections hold read transactions, and a `TRUNCATE` checkpoint waits on
/// readers — so under a busy daemon this can still return having moved
/// little. `journal_size_limit` remains the backstop that caps the file.
/// Confirming the fix needs a measurement against a live daemon, not a
/// single-connection test, which folds the WAL on its own either way.
pub fn vacuum(&self) -> Result<VacuumOutcome> {
let before = self.stats()?.total_bytes();
{
let conn = self.conn.lock();
conn.execute_batch("VACUUM")?;
}
let wal_frames_checkpointed = self.checkpoint_wal()?;
let after = self.stats()?.total_bytes();
Ok(VacuumOutcome {
before_bytes: before,
after_bytes: after,
wal_frames_checkpointed,
})
}
/// Checkpoint the WAL and truncate it, returning the number of frames
/// written back into the database.
///
/// `TRUNCATE` blocks on readers rather than giving up part-way, which is
/// what makes the result meaningful: a `PASSIVE` checkpoint against a busy
/// daemon can legitimately move nothing and report success.
pub fn checkpoint_wal(&self) -> Result<i64> {
let conn = self.conn.lock();
// Columns are (busy, log frames, frames checkpointed).
let frames: i64 = conn.query_row("PRAGMA wal_checkpoint(TRUNCATE)", [], |r| r.get(2))?;
Ok(frames.max(0))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_fresh_database_passes_its_integrity_check() {
let db = Db::open_in_memory().unwrap();
assert!(db.integrity_check().unwrap().is_empty());
}
#[test]
fn stats_report_the_schema_and_the_known_tables() {
let db = Db::open_in_memory().unwrap();
let stats = db.stats().unwrap();
assert!(stats.schema_version > 0, "migrations set a user_version");
assert!(stats.page_size > 0);
assert!(
stats.tables.iter().any(|(name, _)| name == "nodes"),
"the nodes table should be reported; got {:?}",
stats.tables
);
// A fresh database has no rows in any of them.
assert!(stats.tables.iter().all(|(_, count)| *count == 0));
}
#[test]
fn total_and_reclaimable_bytes_derive_from_the_page_counts() {
let stats = DbStats {
schema_version: 1,
page_count: 10,
free_pages: 4,
page_size: 4096,
tables: Vec::new(),
};
assert_eq!(stats.total_bytes(), 40960);
assert_eq!(stats.reclaimable_bytes(), 16384);
}
/// A vacuum that leaves the file larger (a checkpoint folded WAL frames in)
/// must report nothing freed, not an underflowed enormous number.
#[test]
fn freed_bytes_does_not_underflow_when_the_file_grew() {
let outcome = VacuumOutcome {
before_bytes: 100,
after_bytes: 400,
..VacuumOutcome::default()
};
assert_eq!(outcome.freed_bytes(), 0);
}
#[test]
fn freed_bytes_reports_the_difference_when_the_file_shrank() {
let outcome = VacuumOutcome {
before_bytes: 4096,
after_bytes: 1024,
..VacuumOutcome::default()
};
assert_eq!(outcome.freed_bytes(), 3072);
}
#[test]
fn vacuuming_a_fresh_database_leaves_it_sound() {
let db = Db::open_in_memory().unwrap();
db.vacuum().unwrap();
assert!(db.integrity_check().unwrap().is_empty());
}
/// A unique temp directory removed on drop; avoids a dev-dependency, as in
/// [`crate::cache`]'s tests. Needed here because an in-memory database has
/// no WAL file to measure.
struct TempDir(std::path::PathBuf);
impl TempDir {
fn new() -> Self {
use std::sync::atomic::{AtomicU64, Ordering};
static N: AtomicU64 = AtomicU64::new(0);
let p = std::env::temp_dir().join(format!(
"pdfs-maint-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed)
));
std::fs::create_dir_all(&p).unwrap();
TempDir(p)
}
}
impl Drop for TempDir {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
/// A file-backed vacuum reclaims space and leaves the file sound.
///
/// Note what this does *not* pin: the WAL-inflation behaviour that motivated
/// the checkpoint ordering in [`Db::vacuum`]. Measured here, both orderings
/// report `frames=0` and leave a 0-byte WAL, because a single-connection
/// test lets SQLite fold the log itself. The live case inflated only because
/// the daemon holds concurrent readers that block a checkpoint, and
/// reproducing that needs a second connection parked in a read transaction.
/// An assertion on frame count or WAL size here would pass under both
/// orderings and so would be decoration — see the note in `vacuum`.
#[test]
fn a_file_backed_vacuum_reclaims_space_and_stays_sound() {
let dir = TempDir::new();
let path = dir.0.join("cache.db");
let db = Db::open(&path).unwrap();
for i in 0..2000 {
db.set_state_str(&format!("k{i}"), &"v".repeat(256))
.unwrap();
}
let full = db.stats().unwrap().total_bytes();
for i in 0..2000 {
db.set_state_str(&format!("k{i}"), "").unwrap();
}
let outcome = db.vacuum().unwrap();
assert!(
outcome.after_bytes < full,
"vacuum did not reclaim: {} → {} (peak {full})",
outcome.before_bytes,
outcome.after_bytes
);
assert!(db.integrity_check().unwrap().is_empty());
}
}
+30 -1
View File
@@ -9,7 +9,7 @@ use super::Db;
use crate::Result;
/// Current schema version. Bump on every forward migration added below.
pub(super) const SCHEMA_VERSION: i64 = 14;
pub(super) const SCHEMA_VERSION: i64 = 15;
impl Db {
pub(super) fn migrate(&self) -> Result<()> {
@@ -78,6 +78,9 @@ impl Db {
if current < 14 {
tx.execute_batch(MIGRATION_V14)?;
}
if current < 15 {
tx.execute_batch(MIGRATION_V15)?;
}
tx.execute(
"INSERT INTO sync_state (key, value) VALUES ('schema_version', ?1)
ON CONFLICT(key) DO UPDATE SET value = excluded.value",
@@ -340,3 +343,29 @@ ALTER TABLE photos ADD COLUMN kind INTEGER NOT NULL DEFAULT 0;
const MIGRATION_V14: &str = "
CREATE INDEX cache_entries_lru ON cache_entries(kind, last_accessed);
";
/// Schema v15: key the search index by rowid so a row can be replaced without
/// scanning the whole index (B12).
///
/// v3 built `nodes_fts` with the node's `uid` as an `UNINDEXED` column and
/// `upsert_nodes` refreshed a row with `DELETE FROM nodes_fts WHERE uid = ?`.
/// `UNINDEXED` means exactly what it says — the column is retrievable but not
/// searchable — so that predicate could only ever be a full scan of the index
/// (`EXPLAIN QUERY PLAN`: `SCAN nodes_fts VIRTUAL TABLE`). Every node written
/// scanned every node indexed: measured at 6.5 ms per node against a 17k-node
/// index, about half the cost of a cold listing, and *growing with the size of
/// the account* rather than the size of the listing.
///
/// FTS5 deletes by `rowid` efficiently, and `nodes` is an ordinary rowid table
/// (`uid` is its `TEXT PRIMARY KEY`, not `WITHOUT ROWID`), so its rowid is a
/// stable per-node key that `ON CONFLICT DO UPDATE` preserves. Rebuild the index
/// keyed by it and drop the `uid` column: searches now join on `rowid`, which is
/// also a cheaper join than the old one on a TEXT uid.
const MIGRATION_V15: &str = "
DROP TABLE IF EXISTS nodes_fts;
CREATE VIRTUAL TABLE nodes_fts USING fts5(
name, tokenize='trigram'
);
INSERT INTO nodes_fts (rowid, name)
SELECT rowid, name FROM nodes WHERE trashed = 0;
";
+2
View File
@@ -25,6 +25,7 @@ mod activity;
mod cache;
mod devices;
mod local;
mod maintenance;
mod migrations;
mod nodes;
mod ops;
@@ -38,6 +39,7 @@ mod utils;
pub use cache::CacheEntryInput;
pub use devices::StoredDevice;
pub use local::LocalFileHit;
pub use maintenance::{DbStats, VacuumOutcome};
pub use nodes::{SearchHit, StoredNode};
pub use ops::{
AttachedBlob, LOCAL_VOLUME, OP_CREATE, OP_MKDIR, OP_RENAME, OP_REVISION, OP_TRASH,
+41 -5
View File
@@ -70,11 +70,22 @@ impl Db {
// FTS5 has no UPSERT, so refresh the row by delete-then-insert.
// Trashed nodes are kept out of the index entirely so they never
// surface in search results.
tx.execute("DELETE FROM nodes_fts WHERE uid = ?1", params![uid])?;
//
// Keyed by the node's rowid, never by its uid: the uid is not an
// indexed FTS column, so deleting by it scans the entire index once
// per node written (B12). The rowid is stable across the upsert
// above — `ON CONFLICT DO UPDATE` keeps the existing row — so it is
// the same key before and after.
let rowid: i64 = tx.query_row(
"SELECT rowid FROM nodes WHERE uid = ?1",
params![uid],
|row| row.get(0),
)?;
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![rowid])?;
if !node.trashed {
tx.execute(
"INSERT INTO nodes_fts (uid, name) VALUES (?1, ?2)",
params![uid, node.name],
"INSERT INTO nodes_fts (rowid, name) VALUES (?1, ?2)",
params![rowid, node.name],
)?;
}
}
@@ -88,12 +99,37 @@ impl Db {
let uid = uid.to_string();
let mut conn = self.conn.lock();
let tx = conn.transaction()?;
// Read the rowid before dropping the node: it is the search index's key
// (B12), and it is gone once the row is. A node with no row leaves
// nothing to unindex — the index is keyed off `nodes`, so it cannot
// hold an entry the table never had.
let rowid: Option<i64> = tx
.query_row(
"SELECT rowid FROM nodes WHERE uid = ?1",
params![uid],
|row| row.get(0),
)
.optional()?;
tx.execute("DELETE FROM nodes WHERE uid = ?1", params![uid])?;
tx.execute("DELETE FROM nodes_fts WHERE uid = ?1", params![uid])?;
if let Some(rowid) = rowid {
tx.execute("DELETE FROM nodes_fts WHERE rowid = ?1", params![rowid])?;
}
tx.commit()?;
Ok(())
}
/// Check if a folder node has any non-trashed children in the database.
pub fn has_children(&self, parent_uid: &NodeUid) -> Result<bool> {
let uid_str = parent_uid.to_string();
let conn = self.conn.lock();
let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM nodes WHERE parent_uid = ?1 AND trashed = 0",
params![uid_str],
|row| row.get(0),
)?;
Ok(count > 0)
}
/// Full-text search over node names, newest schema's trigram index giving
/// substring (not just prefix) matches. Returns up to `limit` non-trashed
/// hits, each with its mountpoint-relative path resolved. Queries shorter
@@ -123,7 +159,7 @@ impl Db {
.join(" AND ");
let mut stmt = conn.prepare(
"SELECT n.node_json, n.uid
FROM nodes_fts f JOIN nodes n ON n.uid = f.uid
FROM nodes_fts f JOIN nodes n ON n.rowid = f.rowid
WHERE f.name MATCH ?1 AND n.trashed = 0 AND n.node_json IS NOT NULL
ORDER BY f.rank LIMIT ?2",
)?;
+37 -3
View File
@@ -138,8 +138,8 @@ impl Db {
};
conn.execute(
"INSERT INTO pending_op
(kind, uid, parent_uid, name, blob_path, meta_json, created_at)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)",
(kind, uid, parent_uid, name, blob_path, meta_json, created_at, next_attempt_at)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)",
params![
op.kind,
op.uid,
@@ -147,7 +147,8 @@ impl Db {
op.name,
op.blob_path,
op.meta_json,
op.created_at
op.created_at,
op.next_attempt_at
],
)?;
Ok((conn.last_insert_rowid(), superseded))
@@ -229,6 +230,17 @@ impl Db {
Ok(n > 0)
}
/// Check if a queued create or mkdir op exists for `uid`.
pub fn has_create_op(&self, uid: &str) -> Result<bool> {
let conn = self.conn.lock();
let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM pending_op WHERE uid = ?1 AND kind IN (?2, ?3)",
params![uid, OP_CREATE, OP_MKDIR],
|row| row.get(0),
)?;
Ok(count > 0)
}
/// Drop every op targeting a node **or anything queued beneath it**,
/// returning the staged blobs they held so the caller can delete them.
///
@@ -347,6 +359,28 @@ impl Db {
Ok(op)
}
/// The earliest `next_attempt_at` among all queued ops, or `None` if the
/// queue is empty. Used by the drain loop to sleep exactly until the next
/// debounced or backed-off op becomes eligible rather than waiting the full
/// idle-poll interval.
pub fn earliest_due_at(&self) -> Result<Option<i64>> {
let conn = self.conn.lock();
let ts: Option<i64> = conn
.query_row(
&format!(
"SELECT MIN(next_attempt_at) FROM pending_op \
WHERE parent_uid IS NULL OR substr(parent_uid, 1, {n}) <> '{v}~'",
v = LOCAL_VOLUME,
n = LOCAL_VOLUME.len() + 1,
),
[],
|r| r.get(0),
)
.optional()?
.flatten();
Ok(ts)
}
/// Every queued op, oldest first. For status read-outs and the CLI — the
/// drain wants [`next_due_op`](Self::next_due_op) instead.
pub fn pending_ops(&self) -> Result<Vec<PendingOp>> {
+160
View File
@@ -564,6 +564,78 @@ fn search_drops_fts_row_on_delete_and_trash() {
assert_eq!(db.search("unique", 10).unwrap().len(), 0);
}
/// Renaming re-indexes the node, which is the case delete-by-rowid has to get
/// right: the *old* name must leave the index. A stale row here would be
/// invisible in every other test — the new name is findable either way.
#[test]
fn renaming_a_node_replaces_the_name_in_the_search_index() {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
db.upsert_node(&file("f1", "root", "beforename.txt", 1))
.unwrap();
assert_eq!(db.search("beforename", 10).unwrap().len(), 1);
db.upsert_node(&file("f1", "root", "aftername.txt", 1))
.unwrap();
assert_eq!(
db.search("aftername", 10).unwrap().len(),
1,
"the new name must be searchable"
);
assert_eq!(
db.search("beforename", 10).unwrap().len(),
0,
"the old name must have left the index, not merely been added alongside"
);
}
/// B12: the cost of writing a listing must not scale with how much is already
/// indexed. The old `DELETE FROM nodes_fts WHERE uid = ?` scanned the whole
/// index once per node written, so a large account paid more to write the *same*
/// folder — measured at 6.5 ms per node against a 17k-node index on the live DB.
///
/// Written as a ratio between two index sizes rather than an absolute time, so
/// it fails on the scaling regression and not on a slow machine.
#[test]
fn writing_nodes_does_not_scale_with_the_size_of_the_index() {
use std::time::Instant;
/// Fill `db` with `n` indexed nodes, then time re-writing a fixed batch.
fn cost_against_index_of(n: usize) -> std::time::Duration {
let db = Db::open_in_memory().unwrap();
db.upsert_node(&folder("root", None, "My Files")).unwrap();
let filler: Vec<_> = (0..n)
.map(|i| file(&format!("bg{i}"), "root", &format!("background{i}.txt"), 1))
.collect();
db.upsert_nodes(&filler).unwrap();
// The batch under test is identical in both runs; only the amount of
// already-indexed data around it differs.
let batch: Vec<_> = (0..200)
.map(|i| file(&format!("m{i}"), "root", &format!("measured{i}.txt"), 1))
.collect();
db.upsert_nodes(&batch).unwrap();
let t = Instant::now();
db.upsert_nodes(&batch).unwrap();
t.elapsed()
}
let small = cost_against_index_of(200);
let large = cost_against_index_of(5000);
// Calibration, measured while writing this: over the same 25× growth in
// index size, the old full-scan path cost 6.4× more (46 ms → 294 ms) while
// the rowid path costs 1.04× (39 ms → 41 ms). 3× sits well clear of both.
println!("B12: 200-node write — against 200 indexed {small:?}, against 5000 {large:?}");
assert!(
large < small * 3,
"writing a listing got dramatically more expensive as the index grew, \
which is the B12 full-scan regression; small={small:?} large={large:?}"
);
}
fn activity(target: &str, kind: ActivityKind, ok: bool) -> ActivityEntry {
ActivityEntry {
time: 1700,
@@ -1450,3 +1522,91 @@ fn db_contention_under_fuse_worker_load() {
serial / total as u32
);
}
#[test]
fn test_db_has_children_and_trashed_filtering() {
use proton_drive_rs::proton_sdk::ids::{LinkId, VolumeId};
let db = Db::open_in_memory().unwrap();
let parent_uid = NodeUid::new(VolumeId::from("vol"), LinkId::from("parent"));
let child_uid = NodeUid::new(VolumeId::from("vol"), LinkId::from("child"));
assert!(
!db.has_children(&parent_uid).unwrap(),
"no children initially"
);
let parent_node = Node {
uid: parent_uid.clone(),
parent_uid: None,
name: "parent".into(),
kind: NodeKind::Folder,
creation_time: 100,
modification_time: 100,
trashed: false,
is_shared: false,
is_shared_publicly: false,
signature_email: None,
verification: Default::default(),
};
db.upsert_node(&parent_node).unwrap();
let mut child_node = Node {
uid: child_uid.clone(),
parent_uid: Some(parent_uid.clone()),
name: "child.txt".into(),
kind: NodeKind::File {
media_type: "text/plain".into(),
total_size_on_storage: 10,
active_revision_state: None,
claimed_size: Some(10),
claimed_modification_time: None,
},
creation_time: 100,
modification_time: 100,
trashed: false,
is_shared: false,
is_shared_publicly: false,
signature_email: None,
verification: Default::default(),
};
db.upsert_node(&child_node).unwrap();
assert!(db.has_children(&parent_uid).unwrap(), "child node present");
// Trashing child removes it from active children
child_node.trashed = true;
db.upsert_node(&child_node).unwrap();
assert!(
!db.has_children(&parent_uid).unwrap(),
"trashed child ignored"
);
}
#[test]
fn test_db_has_create_op() {
let db = Db::open_in_memory().unwrap();
let local_uid = "local~testnode12345678901234567890";
assert!(!db.has_create_op(local_uid).unwrap(), "no op initially");
let op = ops::PendingOp {
id: 0,
kind: ops::OP_CREATE.to_string(),
uid: local_uid.to_string(),
parent_uid: Some("vol~parent".to_string()),
name: Some("test.txt".to_string()),
blob_path: None,
meta_json: None,
created_at: 1000,
attempts: 0,
last_error: None,
next_attempt_at: 0,
};
db.enqueue_op(&op).unwrap();
assert!(db.has_create_op(local_uid).unwrap(), "create op present");
db.delete_ops_for_uid(local_uid).unwrap();
assert!(!db.has_create_op(local_uid).unwrap(), "op deleted");
}
+2
View File
@@ -9,8 +9,10 @@ pub mod control;
pub mod db;
pub mod error;
pub mod localindex;
pub mod profile;
pub mod service;
pub mod shell;
pub mod syncignore;
pub use error::{CoreError, CoreResult, Error, Result};
+171
View File
@@ -0,0 +1,171 @@
//! The machine profile that travels with a device (features.md 5.3).
//!
//! Sync folder mappings, pins, and the settings that shape them live in
//! `cache.db` and `config.json` — both on the local disk. When that disk dies,
//! the remote data survives but the *arrangement* of it does not: which folders
//! were synced, where they lived locally, which were on-demand, what was pinned.
//!
//! So the daemon mirrors that arrangement into a `profile.json` in its device's
//! root folder on Drive. It goes up the normal upload path, which means it is
//! end-to-end encrypted with the account keys like any other file — 5.3's
//! "encrypt with the user's Proton keys" is satisfied by *not* special-casing
//! it. It lives under the device rather than the Drive root because it describes
//! that device; adopting the device is what makes it relevant.
//!
//! This module is pure data. The upload/download and the restore flow live in
//! `pdfs-fuse`, next to the client that performs them.
use serde::{Deserialize, Serialize};
/// File name of the profile inside the device root folder.
pub const PROFILE_FILE_NAME: &str = "profile.json";
/// Format version of the profile document.
///
/// Bumped only for changes a *reader* cannot survive. Adding fields does not
/// qualify — unknown fields are ignored on the way in, so an older client reads
/// a newer profile's common subset. A newer `version` than this is refused
/// outright rather than half-applied: a restore that silently drops folders it
/// did not understand is worse than one that tells you to upgrade.
pub const PROFILE_VERSION: u32 = 1;
/// One synced folder, as it was arranged on the machine that wrote the profile.
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct ProfileFolder {
/// Uid of the folder's remote root, under the device root. The stable
/// identity — the local path may not exist on the restoring machine, but
/// this does.
pub remote_uid: String,
/// Absolute local path it was synced to. A *suggestion* on restore: another
/// machine may have a different username, home, or mount layout.
pub local_path: String,
/// `mirror` or `ondemand`.
pub mode: String,
}
/// One pinned node (P5 pins), as `pins` stores it.
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct ProfilePin {
pub uid: String,
/// Mountpoint-relative path, kept for display: a pin whose node has been
/// deleted should still be nameable in a restore summary.
pub path: String,
pub recursive: bool,
}
/// The full arrangement of one machine.
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)]
pub struct Profile {
pub version: u32,
/// The device this profile describes.
pub device_uid: String,
/// Hostname of the machine that wrote it, so a restore prompt can say
/// *"restore settings from laptop-xyz?"* rather than quoting a uid.
pub hostname: String,
/// When it was written, epoch seconds.
pub saved_at: i64,
#[serde(default)]
pub folders: Vec<ProfileFolder>,
#[serde(default)]
pub pins: Vec<ProfilePin>,
/// Global sync ignore patterns. Safe to restore as-is.
#[serde(default)]
pub ignore_patterns: Option<Vec<String>>,
/// Machine-shaped settings. Carried, but never applied without asking: a
/// laptop's 5 GB cache budget and `~/ProtonDrive` mountpoint are not
/// obviously right on the next machine.
#[serde(default)]
pub cache_budget: Option<u64>,
#[serde(default)]
pub mountpoint: Option<String>,
}
impl Profile {
/// Parse a profile document, refusing one written by a newer client.
pub fn parse(bytes: &[u8]) -> Result<Self, String> {
let profile: Profile =
serde_json::from_slice(bytes).map_err(|e| format!("malformed profile.json: {e}"))?;
if profile.version > PROFILE_VERSION {
return Err(format!(
"profile.json was written by a newer client (version {}, this client understands {PROFILE_VERSION}); upgrade to restore it",
profile.version
));
}
Ok(profile)
}
/// Serialize for upload. Pretty-printed: it is a small file a user may well
/// open in the web UI to see what their machine had.
pub fn to_bytes(&self) -> Result<Vec<u8>, String> {
serde_json::to_vec_pretty(self).map_err(|e| format!("serialize profile: {e}"))
}
}
#[cfg(test)]
mod tests {
use super::*;
fn sample() -> Profile {
Profile {
version: PROFILE_VERSION,
device_uid: "dev-1".to_string(),
hostname: "laptop-xyz".to_string(),
saved_at: 1_700_000_000,
folders: vec![ProfileFolder {
remote_uid: "vol~link".to_string(),
local_path: "/home/nils/Documents".to_string(),
mode: "mirror".to_string(),
}],
pins: vec![ProfilePin {
uid: "vol~pin".to_string(),
path: "Photos/2024".to_string(),
recursive: true,
}],
ignore_patterns: Some(vec!["target/".to_string()]),
cache_budget: Some(5_000_000_000),
mountpoint: Some("/home/nils/ProtonDrive".to_string()),
}
}
#[test]
fn round_trips() {
let p = sample();
assert_eq!(Profile::parse(&p.to_bytes().unwrap()).unwrap(), p);
}
/// A field added by a later client must not break this one's restore: the
/// common subset is still usable, which is the whole reason `version` is
/// reserved for breaking changes only.
#[test]
fn unknown_fields_are_ignored() {
let json = br#"{
"version": 1,
"device_uid": "dev-1",
"hostname": "laptop-xyz",
"saved_at": 1,
"folders": [],
"something_from_the_future": {"a": 1}
}"#;
let p = Profile::parse(json).unwrap();
assert_eq!(p.device_uid, "dev-1");
assert!(p.folders.is_empty());
}
/// Missing optional sections read as empty rather than failing: a profile
/// written before pins existed is still a valid folder mapping.
#[test]
fn absent_sections_default() {
let json = br#"{"version":1,"device_uid":"d","hostname":"h","saved_at":0}"#;
let p = Profile::parse(json).unwrap();
assert!(p.pins.is_empty());
assert!(p.ignore_patterns.is_none());
}
#[test]
fn future_version_is_refused() {
let mut p = sample();
p.version = PROFILE_VERSION + 1;
let err = Profile::parse(&p.to_bytes().unwrap()).unwrap_err();
assert!(err.contains("newer client"), "{err}");
}
}
+275
View File
@@ -0,0 +1,275 @@
//! Selective-sync ignore rules for mirror folders (features.md 1.1).
//!
//! A mirror folder otherwise uploads every path beneath its root, which for a
//! project directory means `node_modules/`, `.git/`, and `target/` — quota
//! spent on files nobody wants on Drive, and enough API traffic to invite rate
//! limiting.
//!
//! Rules come from two places, unioned:
//!
//! * `.pdfsignore` (or `.protonignore`) at the root of the synced folder,
//! gitignore syntax.
//! * A global pattern list from the config, applied to every synced folder.
//!
//! Both are read fresh at the start of each reconcile pass, so editing the file
//! takes effect on the next pass without restarting the daemon.
//!
//! # Ignoring is not deleting
//!
//! A path that becomes ignored after it was already synced keeps its remote
//! copy and its baseline row. The sync engine drops ignored paths from
//! classification entirely rather than treating them as locally deleted —
//! see the union filter in `pdfs-fuse`'s `do_reconcile`. Editing a text file
//! must never trash data on Drive.
use std::path::Path;
use ignore::gitignore::{Gitignore, GitignoreBuilder};
/// Patterns applied to every synced folder when the config does not override
/// them. Version-control metadata, dependency and build trees, editor
/// leftovers, and OS turds — none of which belong on Drive, all of which are
/// large or rewritten constantly.
pub const DEFAULT_IGNORE_PATTERNS: &[&str] = &[
".git/",
".hg/",
".svn/",
"node_modules/",
"target/",
".venv/",
"__pycache__/",
"*~",
"*.swp",
"*.tmp",
".DS_Store",
"Thumbs.db",
];
/// Filenames accepted as a folder's ignore file, in precedence order.
const IGNORE_FILE_NAMES: &[&str] = &[".pdfsignore", ".protonignore"];
/// Compiled ignore rules for one synced folder.
pub struct IgnoreRules {
matcher: Option<Gitignore>,
}
impl IgnoreRules {
/// Build the rules for a folder rooted at `root`: `globals` first, then any
/// ignore file found at the root, so a folder-local rule can negate a
/// global one with `!pattern`.
///
/// Malformed patterns are skipped individually rather than failing the
/// load — a typo in `.pdfsignore` must not take the folder's sync down.
pub fn load(root: &Path, globals: &[String]) -> Self {
let mut builder = GitignoreBuilder::new(root);
let mut any = false;
for pattern in globals {
if builder.add_line(None, pattern).is_ok() {
any = true;
}
}
for name in IGNORE_FILE_NAMES {
let path = root.join(name);
if !path.is_file() {
continue;
}
// `add` returns the first parse error but still takes the lines it
// could read, so a bad line costs that line and nothing more.
let _ = builder.add(&path);
any = true;
break;
}
if !any {
return Self { matcher: None };
}
Self {
matcher: builder.build().ok(),
}
}
/// Rules matching nothing, for callers with no folder context.
pub fn empty() -> Self {
Self { matcher: None }
}
/// Whether any rule is loaded. When false, [`is_ignored`](Self::is_ignored)
/// is always false and callers can skip filtering entirely.
pub fn is_empty(&self) -> bool {
self.matcher.is_none()
}
/// Whether `rel` — a `/`-joined path relative to the folder root — is
/// ignored.
///
/// A path under an ignored directory is ignored too. `Gitignore` matches
/// one path against one rule set and does not walk parents itself, so each
/// ancestor is tested as a directory. This is what makes filtering the
/// classification union sound: the engine can ask about any path in
/// isolation, in any order, without having walked its parents first.
pub fn is_ignored(&self, rel: &str, is_dir: bool) -> bool {
let Some(matcher) = &self.matcher else {
return false;
};
if rel.is_empty() {
return false;
}
let mut prefix = String::new();
let mut parts = rel.split('/').peekable();
while let Some(part) = parts.next() {
if !prefix.is_empty() {
prefix.push('/');
}
prefix.push_str(part);
let last = parts.peek().is_none();
// Every ancestor is a directory; only the final component's kind
// is the caller's to say.
let as_dir = if last { is_dir } else { true };
if matcher.matched(&prefix, as_dir).is_ignore() {
return true;
}
}
false
}
}
impl std::fmt::Debug for IgnoreRules {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("IgnoreRules")
.field("active", &!self.is_empty())
.finish()
}
}
#[cfg(test)]
mod tests {
use super::*;
/// A unique temp directory removed on drop; avoids a dev-dependency, as in
/// [`crate::cache`]'s tests.
struct TempDir(std::path::PathBuf);
impl TempDir {
fn new() -> Self {
use std::sync::atomic::{AtomicU64, Ordering};
static N: AtomicU64 = AtomicU64::new(0);
let p = std::env::temp_dir().join(format!(
"pdfs-ignore-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed)
));
std::fs::create_dir_all(&p).unwrap();
TempDir(p)
}
fn path(&self) -> &Path {
&self.0
}
}
impl Drop for TempDir {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
fn globals() -> Vec<String> {
DEFAULT_IGNORE_PATTERNS
.iter()
.map(|s| (*s).to_string())
.collect()
}
fn write(root: &Path, name: &str, body: &str) {
std::fs::write(root.join(name), body).unwrap();
}
#[test]
fn no_rules_at_all_ignores_nothing() {
let dir = TempDir::new();
let rules = IgnoreRules::load(dir.path(), &[]);
assert!(rules.is_empty());
assert!(!rules.is_ignored("anything", false));
assert!(!rules.is_ignored("deep/nested/path.rs", false));
}
#[test]
fn global_defaults_match_without_an_ignore_file() {
let dir = TempDir::new();
let rules = IgnoreRules::load(dir.path(), &globals());
assert!(rules.is_ignored("node_modules", true));
assert!(rules.is_ignored("target", true));
assert!(rules.is_ignored("notes.txt~", false));
assert!(!rules.is_ignored("src/main.rs", false));
}
#[test]
fn a_path_under_an_ignored_directory_is_ignored() {
let dir = TempDir::new();
let rules = IgnoreRules::load(dir.path(), &globals());
// The engine asks about deep paths in isolation, so each must answer
// correctly without its parents having been tested first.
assert!(rules.is_ignored("node_modules/left-pad/index.js", false));
assert!(rules.is_ignored("node_modules/left-pad", true));
assert!(rules.is_ignored("target/debug/build/x/out", false));
}
#[test]
fn a_folder_ignore_file_adds_to_the_globals() {
let dir = TempDir::new();
write(dir.path(), ".pdfsignore", "secrets/\n*.log\n");
let rules = IgnoreRules::load(dir.path(), &globals());
assert!(rules.is_ignored("secrets/key.pem", false));
assert!(rules.is_ignored("run.log", false));
// Globals still apply alongside the file.
assert!(rules.is_ignored("node_modules", true));
assert!(!rules.is_ignored("run.txt", false));
}
#[test]
fn a_folder_rule_can_negate_a_global_one() {
let dir = TempDir::new();
write(dir.path(), ".pdfsignore", "!keep.tmp\n");
let rules = IgnoreRules::load(dir.path(), &globals());
assert!(rules.is_ignored("scratch.tmp", false));
assert!(!rules.is_ignored("keep.tmp", false));
}
#[test]
fn protonignore_is_accepted_as_an_alias() {
let dir = TempDir::new();
write(dir.path(), ".protonignore", "build/\n");
let rules = IgnoreRules::load(dir.path(), &[]);
assert!(rules.is_ignored("build/out.o", false));
}
#[test]
fn pdfsignore_wins_when_both_files_exist() {
let dir = TempDir::new();
write(dir.path(), ".pdfsignore", "from-pdfs/\n");
write(dir.path(), ".protonignore", "from-proton/\n");
let rules = IgnoreRules::load(dir.path(), &[]);
assert!(rules.is_ignored("from-pdfs/x", false));
assert!(!rules.is_ignored("from-proton/x", false));
}
#[test]
fn a_malformed_pattern_does_not_take_the_rest_down() {
let dir = TempDir::new();
// An unclosed character class is not a valid glob.
write(dir.path(), ".pdfsignore", "[unclosed\nvalid/\n");
let rules = IgnoreRules::load(dir.path(), &globals());
assert!(rules.is_ignored("valid/x", false));
assert!(rules.is_ignored("node_modules", true));
}
#[test]
fn the_root_itself_is_never_ignored() {
let dir = TempDir::new();
write(dir.path(), ".pdfsignore", "*\n");
let rules = IgnoreRules::load(dir.path(), &[]);
// A blanket rule must not classify the folder root, or the pass would
// have nothing to reconcile against.
assert!(!rules.is_ignored("", true));
}
}
+268
View File
@@ -0,0 +1,268 @@
//! Remote-event invalidation and local-index background workers.
use super::*;
/// Apply one remote event to the local cache and notify the kernel so it drops
/// any stale cached metadata/data for the affected inodes.
///
/// The cache is authoritative-by-absence: dropping a directory's `children`
/// entry forces the next `lookup`/`readdir` to re-enumerate from the remote, so
/// most events only need to invalidate listings rather than re-fetch eagerly.
fn apply_event(
state: &Mutex<State>,
content: &ContentCache,
pending: &Mutex<HashMap<NodeUid, PendingRevision>>,
notifier: &Notifier,
event: &DriveEvent,
) {
match event {
DriveEvent::NodeUpdated {
node_uid,
parent_node_uid,
is_trashed,
..
} => {
let mut st = state.lock();
if *is_trashed {
// Trashing makes a node vanish from its parent listing.
let child = st.by_uid.get(node_uid).copied();
if let Some((parent, name)) = st.forget(node_uid) {
content.evict(node_uid);
match child {
Some(child) => {
let _ =
notifier.delete(INodeNo(parent), INodeNo(child), OsStr::new(&name));
}
None => {
let _ = notifier.inval_entry(INodeNo(parent), OsStr::new(&name));
}
}
}
} else if pending.lock().contains_key(node_uid) {
// A node we owe an upload for is *ahead* of the remote, not
// behind it: this event is almost always the echo of our own
// empty-file create, and re-fetching would replace the size and
// mtime of the write we just accepted with the stale revision's
// — making a file that was copied in seconds ago read as empty
// until its upload lands (offline.md Phase 3).
debug!(uid = %node_uid, "ignoring remote event for a node with a queued write");
} else if let Some(&ino) = st.by_uid.get(node_uid) {
// Known node changed: drop its cached attrs/data (and listing if
// it is a directory) so the next access re-fetches. Its content
// blob may now be stale, so evict it too.
st.invalidate_listing(ino);
content.evict(node_uid);
let _ = notifier.inval_inode(INodeNo(ino), 0, 0);
}
// A create (or move-in) shows up as a change to the parent listing;
// drop it so the new child is picked up on the next readdir.
if let Some(parent_uid) = parent_node_uid
&& let Some(&parent) = st.by_uid.get(parent_uid)
{
st.invalidate_listing(parent);
let _ = notifier.inval_inode(INodeNo(parent), 0, 0);
}
}
DriveEvent::NodeDeleted { node_uid, .. } => {
let mut st = state.lock();
// Capture the inode before `forget` clears the uid mapping.
let child = st.by_uid.get(node_uid).copied();
content.evict(node_uid);
if let Some((parent, name)) = st.forget(node_uid) {
match child {
Some(child) => {
let _ = notifier.delete(INodeNo(parent), INodeNo(child), OsStr::new(&name));
}
None => {
let _ = notifier.inval_entry(INodeNo(parent), OsStr::new(&name));
}
}
}
}
// Continuity or scope was lost: our cached listings may be arbitrarily
// stale, so drop every listing and tell the kernel to forget all
// metadata. Inodes stay stable; dirs simply re-enumerate on next access.
DriveEvent::ContinuityLost { .. } | DriveEvent::ScopeAccessLost { .. } => {
warn!("event continuity lost; dropping all cached listings, resyncing lazily");
let mut st = state.lock();
let dirs: Vec<u64> = st.children.keys().copied().collect();
for &ino in &dirs {
st.invalidate_listing(ino);
let _ = notifier.inval_inode(INodeNo(ino), 0, 0);
}
}
// No substantive local change; the cursor advance is handled by the
// caller persisting the event id.
DriveEvent::CursorAdvanced { .. } | DriveEvent::SharedWithMeUpdated { .. } => {}
}
}
/// Poll the remote event cursor forever, applying each batch to the shared
/// state. Resumes from the cursor persisted in the DB so changes made while
/// unmounted are applied; only a first-ever mount seeds from the server head.
/// The cursor is persisted after every batch. Runs as a Tokio task; returns
/// only on fatal error.
pub(super) async fn run_event_sync(
client: ProtonDriveClient,
scope: DriveEventScopeId,
state: Arc<Mutex<State>>,
content: Arc<ContentCache>,
db: Arc<Db>,
pending: Arc<Mutex<HashMap<NodeUid, PendingRevision>>>,
notifier: Notifier,
) {
let mut cursor: Option<DriveEventId> = match db.get_event_cursor() {
// Resume: pick up exactly where the last run left off.
Ok(Some(saved)) => Some(DriveEventId::from(saved)),
// First mount: a `None` cursor yields a single `CursorAdvanced` at the
// server head; persist it so the next restart resumes instead of
// reseeding (which would skip everything that changed offline).
// Seeding needs the network, and this task also runs on mounts that
// started offline (offline.md Phase 1) — so retry rather than giving up,
// which used to disable live sync for the life of the daemon.
Ok(None) => {
let mut delay = ONLINE_PROBE_MIN;
loop {
match client.enumerate_events(&scope, None).await {
Ok(events) => {
let head = events.last().map(|e| e.id().clone());
if let Some(c) = &head
&& let Err(e) = db.set_event_cursor(c.as_str())
{
warn!(error = %e, "persist seed cursor failed");
}
break head;
}
Err(e) => {
warn!(error = %e, ?delay, "seed event cursor failed; retrying");
tokio::time::sleep(delay).await;
delay = (delay * 2).min(ONLINE_PROBE_MAX);
}
}
}
}
Err(e) => {
error!(error = %e, "read persisted cursor failed; live sync disabled");
return;
}
};
info!(?cursor, "event sync started");
loop {
tokio::time::sleep(POLL_INTERVAL).await;
let events = match client.enumerate_events(&scope, cursor.as_ref()).await {
Ok(events) => events,
Err(e) => {
warn!(error = %e, "event poll failed; retrying after interval");
continue;
}
};
if events.is_empty() {
continue;
}
debug!(count = events.len(), "applying remote events");
for event in &events {
// Converge the SDK's own caches (folder keys, entity cache) on the
// server before applying the event to our tree. Without this, a node
// re-keyed/moved by another client keeps a stale key in the SDK for
// the life of the daemon (SDK plan #9). `apply_event` only touches
// our FUSE state, so nothing else does this.
if let Err(e) = client.invalidate_caches_for_event(event).await {
warn!(error = %e, "sdk cache invalidation for event failed");
}
apply_event(&state, &content, &pending, &notifier, event);
}
cursor = events.last().map(|e| e.id().clone());
if let Some(c) = &cursor
&& let Err(e) = db.set_event_cursor(c.as_str())
{
warn!(error = %e, "persist event cursor failed");
}
}
}
/// Keep the local-file index fresh for the launcher prompt's "This computer"
/// results. Rebuilds the index whenever it is older than [`LOCAL_INDEX_TTL`],
/// then sleeps; runs on its own thread for the life of the daemon.
///
/// The walk is the one part of the daemon that touches the wider filesystem, so
/// it is deliberately kept off every hot path: it never runs on a FUSE or
/// control-socket thread, and it excludes the mountpoint (walking it would fault
/// every remote node in through FUSE, defeating on-demand hydration).
pub(super) fn run_local_index(
db: Arc<Db>,
indexing: Arc<AtomicBool>,
transfers: Arc<TransferRegistry>,
mountpoint: PathBuf,
) {
loop {
let age = db.local_indexed_at().ok().flatten();
let stale =
age.is_none_or(|at| now_secs().saturating_sub(at) >= LOCAL_INDEX_TTL.as_secs() as i64);
if stale {
scan_local_once(&db, &indexing, &transfers, &mountpoint);
}
std::thread::sleep(LOCAL_INDEX_CHECK);
}
}
/// Walk `$HOME` once and replace the local-file index with what it finds.
/// Batches stream straight into SQLite, so peak memory is one batch — not the
/// whole home directory.
///
/// Reports itself as a job: the first scan after a fresh install walks the whole
/// home directory, and `indexing` alone only tells the launcher prompt to say
/// "still indexing" — nothing else showed that the daemon was busy.
fn scan_local_once(
db: &Db,
indexing: &AtomicBool,
transfers: &Arc<TransferRegistry>,
mountpoint: &Path,
) {
let dirs = match AppDirs::new() {
Ok(d) => d,
Err(e) => {
warn!(error = %e, "local index: cannot resolve app dirs");
return;
}
};
let Some(home) = dirs.home_dir() else {
warn!("local index: cannot resolve home directory");
return;
};
let generation = match db.local_begin_scan() {
Ok(g) => g,
Err(e) => {
warn!(error = %e, "local index: cannot open scan generation");
return;
}
};
let excludes = localindex::default_excludes(mountpoint, &dirs.state_dir(), &dirs.cache_dir());
indexing.store(true, Ordering::Relaxed);
let started = Instant::now();
// The walk has no idea how many files it will find, so the job counts what it
// has seen and stays indeterminate.
let job = transfers.begin_job("Indexing this computer");
job.detail("Scanning your files");
let walked = localindex::scan(&[home], &excludes, |batch| {
if let Err(e) = db.local_upsert_batch(generation, &batch) {
warn!(error = %e, "local index: batch write failed");
}
});
// Prune what this scan did not see and rebuild the FTS index over the rest,
// even if some batches failed — a partial index still beats none.
job.detail("Building the search index");
match db.local_finish_scan(generation, now_secs()) {
Ok(indexed) => info!(
walked,
indexed,
elapsed_ms = started.elapsed().as_millis() as u64,
"local index rebuilt"
),
Err(e) => warn!(error = %e, "local index: finish failed"),
}
indexing.store(false, Ordering::Relaxed);
}
+108 -12
View File
@@ -73,18 +73,24 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
}
Ok(CtlRequest::Pin { path }) => match rel_to_mount(mountpoint, &path) {
Ok(rel) => match core.pin(&rel) {
Ok(name) => CtlResponse::Ok {
message: format!("pinned {name}"),
},
Ok(name) => {
core.touch_profile();
CtlResponse::Ok {
message: format!("pinned {name}"),
}
}
Err(e) => CtlResponse::error(e),
},
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::Unpin { path }) => match rel_to_mount(mountpoint, &path) {
Ok(rel) => match core.unpin(&rel) {
Ok(name) => CtlResponse::Ok {
message: format!("unpinned {name}"),
},
Ok(name) => {
core.touch_profile();
CtlResponse::Ok {
message: format!("unpinned {name}"),
}
}
Err(e) => CtlResponse::error(e),
},
Err(e) => CtlResponse::error(e),
@@ -404,6 +410,45 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
),
}
}
Ok(CtlRequest::CacheInspect { deep }) => match core.db.stats() {
Ok(stats) => {
// The deep check reads every page, so it runs only on request;
// `integrity_checked` tells the caller which answer it is
// holding, since "no problems found" and "never looked" are
// both an empty list.
let (integrity_problems, integrity_checked) = if deep {
match core.db.integrity_check() {
Ok(problems) => (problems, true),
Err(e) => (vec![format!("integrity check failed: {e:?}")], true),
}
} else {
(Vec::new(), false)
};
CtlResponse::CacheReport {
schema_version: stats.schema_version,
db_bytes: stats.total_bytes(),
db_reclaimable_bytes: stats.reclaimable_bytes(),
tables: stats.tables,
cache_used: core.cache.usage(),
cache_budget: core.cache.budget(),
integrity_problems,
integrity_checked,
}
}
Err(e) => CtlResponse::error(CoreError::internal(format!("read db stats: {e:?}"))),
},
Ok(CtlRequest::CacheVacuum) => match core.db.vacuum() {
Ok(outcome) => CtlResponse::Ok {
message: format!(
"vacuumed: {:.1} MiB freed ({:.1} MiB → {:.1} MiB), {} WAL frame(s) checkpointed",
outcome.freed_bytes() as f64 / 1_048_576.0,
outcome.before_bytes as f64 / 1_048_576.0,
outcome.after_bytes as f64 / 1_048_576.0,
outcome.wal_frames_checkpointed,
),
},
Err(e) => CtlResponse::error(CoreError::internal(format!("vacuum: {e:?}"))),
},
Ok(CtlRequest::GetQueueStatus) => CtlResponse::Transfers {
items: core.transfers.snapshot(),
jobs: core.jobs_snapshot(),
@@ -417,9 +462,12 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
cfg.cache_budget = Some(bytes);
dirs.save_config(&cfg)
}) {
Ok(Ok(())) => CtlResponse::Ok {
message: format!("cache budget set to {bytes} bytes"),
},
Ok(Ok(())) => {
core.touch_profile();
CtlResponse::Ok {
message: format!("cache budget set to {bytes} bytes"),
}
}
Ok(Err(e)) => CtlResponse::error(CoreError::internal(format!(
"budget applied but config write failed: {e}"
))),
@@ -444,6 +492,10 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
},
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::AdoptDevice { uid }) => match core.adopt_device(uid.as_deref()) {
Ok(message) => CtlResponse::Ok { message },
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::AddSyncFolder { local_path }) => {
// Registering the device and uploading a folder tree far outlasts the
// socket read timeout, so ack immediately and work on a background
@@ -455,6 +507,7 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
let started = Instant::now();
match core.add_sync_folder(&path) {
Ok(folder) => {
core.touch_profile();
let name = Path::new(&folder.local_path)
.file_name()
.and_then(|n| n.to_str())
@@ -483,15 +536,19 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
},
Ok(CtlRequest::RemoveSyncFolder { id, delete_remote }) => {
match core.remove_sync_folder(id, delete_remote) {
Ok(()) => CtlResponse::Ok {
message: "removed synced folder".to_string(),
},
Ok(()) => {
core.touch_profile();
CtlResponse::Ok {
message: "removed synced folder".to_string(),
}
}
Err(e) => CtlResponse::error(e),
}
}
Ok(CtlRequest::SetSyncFolderMode { id, mode }) => {
match core.request_sync_folder_mode(id, &mode) {
Ok(message) => {
core.touch_profile();
core.log_activity(ActivityKind::Upload, &message, "", true);
CtlResponse::Ok { message }
}
@@ -507,6 +564,28 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
},
}
}
Ok(CtlRequest::ListRestorableFolders) => match core.list_restorable_folders() {
Ok(items) => CtlResponse::RestorableFolders { items },
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::RestoreSyncFolders { items }) => {
// Like AddSyncFolder: each restored folder downloads a whole tree,
// which outlasts the socket timeout. Ack, then work.
let core = core.clone();
std::thread::spawn(move || match core.restore_sync_folders(&items) {
Ok(message) => {
core.touch_profile();
core.log_activity(ActivityKind::Download, &message, "", true);
}
Err(e) => {
warn!(error = %e, "restore failed");
core.log_activity(ActivityKind::Download, "restore folders", &e, false);
}
});
CtlResponse::Ok {
message: "restoring folders".to_string(),
}
}
Ok(CtlRequest::ShareNode {
path,
emails,
@@ -601,6 +680,23 @@ fn handle_control_conn(core: &Core, username: &str, mountpoint: &Path, stream: U
Ok(entries) => CtlResponse::Entries { entries },
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::AccountQuota) => match core.account_quota() {
Ok((max_space, used_space)) => CtlResponse::AccountQuota {
max_space,
used_space,
},
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::ListSharedFolder { uid }) => match core.list_shared_folder(&uid) {
Ok(entries) => CtlResponse::Entries { entries },
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::OpenSharedFile { uid }) => match core.open_shared_file(&uid) {
Ok(p) => CtlResponse::FilePath {
path: p.display().to_string(),
},
Err(e) => CtlResponse::error(e),
},
Ok(CtlRequest::LeaveShared { uid }) => match core.leave_shared(&uid) {
Ok(()) => {
core.log_activity(ActivityKind::Unshare, "shared item", "left", true);
+157 -30
View File
@@ -20,14 +20,40 @@ use pdfs_core::db::{StoredDevice, StoredSyncFolder};
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::proton_sdk::ids::{DeviceUid, NodeUid};
use proton_drive_rs::{DeviceType, Node};
use std::sync::OnceLock;
use tracing::{info, warn};
use super::sync::{self, base_name};
use super::{
BackgroundSession, Core, ProtonFs, State, SwitchBlocked, clear_stale_mount, device_type_str,
dir_is_empty, evict_dir_contents, now_secs, parse_uid, sync_folder_info, this_hostname,
dir_is_empty, evict_dir_contents, fuse_connection_id, is_stale_mount, now_secs, parse_uid,
sync_folder_info, this_hostname, umount_session_unblocked,
};
/// The device uid this machine has been adopted into, from `config.json`.
///
/// Read fresh each time rather than cached on [`Core`]: adoption rewrites the
/// config from the control socket, and the next `ensure_device` must see it
/// without a daemon restart.
pub(crate) fn adopted_device_uid() -> Option<String> {
let uid = pdfs_core::config::AppDirs::new()
.ok()?
.load_config()
.device_uid?;
let uid = uid.trim().to_string();
(!uid.is_empty()).then_some(uid)
}
/// Pin (or, with `None`, unpin) the adopted device uid in `config.json`.
pub(crate) fn set_adopted_device_uid(uid: Option<&str>) -> CoreResult<()> {
let dirs = pdfs_core::config::AppDirs::new()
.map_err(|e| CoreError::internal(format!("config dirs: {e}")))?;
let mut cfg = dirs.load_config();
cfg.device_uid = uid.map(|u| u.to_string());
dirs.save_config(&cfg)
.map_err(|e| CoreError::internal(format!("write config: {e}")))
}
impl Core {
// ---- devices ----------------------------------------------------------
@@ -42,12 +68,14 @@ impl Core {
// No cached device row yet means this machine syncs nothing, so none of
// the listed devices is ours.
let this_uid = self.db.device_get().ok().flatten().map(|d| d.uid);
let pinned = adopted_device_uid();
Ok(devices
.into_iter()
.map(|d| {
let uid = d.uid.to_string();
DeviceInfo {
this_device: this_uid.as_deref() == Some(uid.as_str()),
adopted: pinned.as_deref() == Some(uid.as_str()),
uid,
name: d.name.unwrap_or_else(|_| "(unnamed device)".to_string()),
device_type: device_type_str(d.device_type).to_string(),
@@ -78,21 +106,97 @@ impl Core {
Ok(())
}
/// Adopt an existing device as *this* machine's, pinning it in `config.json`
/// so hostname changes and reinstalls stop registering duplicates
/// (features.md 5.1). `None` clears the pin and returns to hostname matching.
///
/// Refuses a uid that is not in the account: the whole point of the pin is
/// that it is authoritative, so accepting a bad one would defer the failure
/// to the next `ensure_device` with no obvious cause.
pub(crate) fn adopt_device(&self, uid: Option<&str>) -> CoreResult<String> {
let Some(uid) = uid else {
set_adopted_device_uid(None)?;
return Ok("adoption cleared; device now resolved by hostname".to_string());
};
let remote = self
.rt
.block_on(self.client.enumerate_devices())
.map_err(|e| CoreError::from_api(&e, "enumerate devices"))?;
let dev = remote
.into_iter()
.find(|d| d.uid.to_string() == uid)
.ok_or_else(|| CoreError::not_found(format!("no device {uid} in this account")))?;
let name = dev
.name
.as_deref()
.ok()
.map(|n| n.to_string())
.unwrap_or_else(this_hostname);
set_adopted_device_uid(Some(uid))?;
// Write the row through too, so callers that read the cached device
// (and `ListSyncFolders`' "this device" flag) agree immediately rather
// than after the next `ensure_device`.
self.db
.device_set(&StoredDevice {
uid: uid.to_string(),
share_id: dev.share_id.to_string(),
root_uid: dev.root_folder_uid.to_string(),
name: name.clone(),
created: dev.creation_time,
})
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?;
info!(uid, name, "adopted device");
Ok(format!("adopted device {name}"))
}
// ---- device folder sync (devices.md, Phase 1) -------------------------
/// Auto-register (or recover) this machine as a Proton Drive Device, caching
/// it so restarts reuse the same device. Recovery matches an existing remote
/// Linux device by name before creating a new one, so a lost local record
/// doesn't orphan the device's root folder.
///
/// Resolution order: the *adopted* uid pinned in `config.json`, then the
/// cached DB row (validated remotely), then a hostname match, then create.
/// The pin exists because the hostname heuristic silently registers a second
/// device after a rename or reinstall, orphaning the first one's folders
/// (features.md 5.1).
pub(crate) fn ensure_device(&self) -> CoreResult<StoredDevice> {
let name = this_hostname();
// Enumerate the remote devices once: used both to validate any cached
// record and to recover an existing device by name.
// Enumerate the remote devices once: used to validate the pin, validate
// any cached record, and recover an existing device by name.
let remote = self
.rt
.block_on(self.client.enumerate_devices())
.map_err(|e| CoreError::from_api(&e, "enumerate devices"))?;
// An adopted uid is an explicit instruction, so it outranks everything
// and it fails loudly: falling back to the heuristic here would create
// the duplicate device the user adopted specifically to avoid.
if let Some(pin) = adopted_device_uid() {
let d = remote
.iter()
.find(|d| d.uid.to_string() == pin)
.ok_or_else(|| {
CoreError::not_found(format!(
"adopted device {pin} not found in this account; \
run `pdfs devices adopt --clear` to return to hostname matching"
))
})?;
let dev = StoredDevice {
uid: d.uid.to_string(),
share_id: d.share_id.to_string(),
root_uid: d.root_folder_uid.to_string(),
name: d.name.as_deref().ok().unwrap_or(&name).to_string(),
created: d.creation_time,
};
self.db
.device_set(&dev)
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?;
return Ok(dev);
}
// A cached device is only trustworthy if it still exists remotely. A
// device deleted from another client (or the web UI) leaves a stale row
// whose root folder is gone, so creating folders under it fails with
@@ -345,8 +449,8 @@ impl Core {
// folder the daemon no longer tracks, and nothing would ever unmount it.
// Tear it down first — including before trashing the remote tree it
// serves, which would otherwise leave it answering for deleted nodes.
if let Some(session) = self.mounts.lock().remove(&id) {
if let Err(e) = session.umount_and_join() {
if let Some((session, conn)) = self.mounts.lock().remove(&id) {
if let Err(e) = umount_session_unblocked(session, conn) {
warn!(id, error = %e, "unmount on-demand folder failed");
} else {
info!(id, path = %folder.local_path, "unmounted on-demand folder");
@@ -390,10 +494,17 @@ impl Core {
by_uid: HashMap::new(),
children: HashMap::new(),
next_ino: 2,
active_writes: HashMap::new(),
handles: HashMap::new(),
next_fh: 1,
db: self.db.clone(),
}));
// A fresh inode space needs a fresh notification channel: this fork's
// session is the only one that knows these inodes, so it must be the one
// notified about them. Filled in by `spawn_ondemand_mount`.
fork.notifier = Arc::new(OnceLock::new());
// Size upgrades are keyed by inode, which is per-fork too.
fork.size_upgrades = Arc::new(Mutex::new(HashMap::new()));
fork
}
@@ -572,27 +683,25 @@ impl Core {
// fails, since eviction may already have deleted part of the tree.
// Recovery is the user's explicit switch back to `mirror`, whose arm
// clears the baseline and re-downloads.
self.db
.sync_folder_set_mode(id, "ondemand")
.map_err(|e| SwitchBlocked::Failed(format!("db: {e:?}")))?;
// Reclaim the disk: empty the local dir (keep it as the mountpoint).
if let Err(e) = evict_dir_contents(&local) {
let _ = self.db.sync_folder_set_state(id, "error", now_secs());
return Err(SwitchBlocked::Failed(format!(
"evict {}: {e}",
local.display()
)));
if let Err(e) = self.db.sync_folder_set_mode(id, "ondemand") {
return Err(SwitchBlocked::Failed(format!("db: {e:?}")));
}
let session = match self.spawn_ondemand_mount(&local, root) {
Ok(session) => session,
Err(e) => {
let _ = self.db.sync_folder_set_state(id, "error", now_secs());
return Err(SwitchBlocked::Failed(e.to_string()));
}
};
self.mounts.lock().insert(id, session);
// Reclaim the disk before mounting. Once FUSE covers `local`, a
// directory walk addresses the remote namespace, so evicting at
// that point sends unlink/rmdir requests to Drive and leaves the
// underlying mirror hidden and untouched.
if let Err(e) = evict_dir_contents(&local) {
warn!(id, path = %local.display(), error = %e, "evict local dir contents failed");
}
let session = self.spawn_ondemand_mount(&local, root).map_err(|e| {
SwitchBlocked::Failed(format!(
"mount ondemand failed after local cache eviction; switch back to mirror to restore it: {e}"
))
})?;
self.mounts
.lock()
.insert(id, (session, fuse_connection_id(&local)));
let _ = self.sync_tx.send(sync::SyncMsg::Rewatch);
self.db.sync_folder_set_state(id, "idle", now_secs()).ok();
info!(id, path = %local.display(), "mounted sync folder on-demand");
@@ -600,8 +709,8 @@ impl Core {
}
_ => {
// ondemand→mirror: tear down the secondary mount first.
if let Some(session) = self.mounts.lock().remove(&id)
&& let Err(e) = session.umount_and_join()
if let Some((session, conn)) = self.mounts.lock().remove(&id)
&& let Err(e) = umount_session_unblocked(session, conn)
{
warn!(id, error = %e, "unmount on-demand folder failed");
}
@@ -639,10 +748,15 @@ impl Core {
MountOption::Subtype("protondrive".to_string()),
MountOption::DefaultPermissions,
];
let fs = ProtonFs::new(self.fork_state(), root);
Session::new(fs, local, &config)
let core = self.fork_state();
let fs = ProtonFs::new(core.clone(), root);
let session = Session::new(fs, local, &config)
.and_then(|s| s.spawn())
.map_err(|e| CoreError::internal(format!("mount {}: {e}", local.display())))
.map_err(|e| CoreError::internal(format!("mount {}: {e}", local.display())))?;
// Hand this fork its own session's notification channel, so a background
// size upgrade can drop the kernel's cached attrs for these inodes.
let _ = core.notifier.set(session.notifier());
Ok(session)
}
/// Re-establish FUSE mounts for folders left in `ondemand` mode across a daemon
@@ -664,6 +778,17 @@ impl Core {
continue;
}
let local = PathBuf::from(&folder.local_path);
// A daemon killed before it could unmount (systemd's stop timeout,
// say) leaves this path attached to a dead FUSE connection: it still
// exists, but every stat on it fails with ENOTCONN — so `is_dir()`
// reads it as *missing* and the folder gets parked in `error` on
// every subsequent start, with no way back but a manual
// `fusermount -u`. Detach the corpse first; the directory
// underneath then restores normally.
if is_stale_mount(&local) {
warn!(id = folder.id, path = %local.display(), "restore on-demand: clearing stale mount");
clear_stale_mount(&local);
}
if !local.is_dir() {
warn!(id = folder.id, path = %local.display(), "restore on-demand: local path missing");
let _ = self
@@ -725,7 +850,9 @@ impl Core {
};
match self.spawn_ondemand_mount(&local, root) {
Ok(session) => {
self.mounts.lock().insert(folder.id, session);
self.mounts
.lock()
.insert(folder.id, (session, fuse_connection_id(&local)));
let _ = self.db.sync_folder_set_state(folder.id, "idle", now_secs());
info!(id = folder.id, path = %local.display(), "remounted on-demand folder");
}
+66 -9
View File
@@ -23,6 +23,7 @@
use std::fs::File;
use std::path::{Path, PathBuf};
use std::sync::atomic::Ordering;
use std::time::Duration;
use pdfs_core::cache::{Baseline, StagedWrite};
use pdfs_core::control::{ActivityKind, TransferDirection};
@@ -62,23 +63,30 @@ impl Core {
let due = self.db.next_due_op(now).unwrap_or_default();
let Some(op) = due.filter(|_| self.online.load(Ordering::Relaxed)) else {
self.wait_for_drain_work();
self.recover_fsynced_writes();
// A debounced or backed-off op may be waiting: sleep only
// until it becomes due rather than the full idle-poll.
self.wait_for_drain_work_or_due();
continue;
};
if let Err(e) = self.drain_op(&op) {
let attempts = op.attempts + 1;
let err_str = e.to_string();
// Never infer that user data is disposable from an error string
// or an arbitrary retry count. The staged blob and this row are
// the only durable description of an accepted write. Even a
// permission/quota error can become recoverable after the user
// changes account state, and deleting the row here made the blob
// unreachable after five ordinary network failures.
let backoff = DRAIN_BACKOFF_MIN
.saturating_mul(1u32 << attempts.min(6))
.min(DRAIN_BACKOFF_MAX);
warn!(uid = %op.uid, attempts, error = %e, "pending upload failed; will retry");
warn!(uid = %op.uid, attempts, error = %err_str, "pending upload failed; will retry");
if let Err(e) = self.db.record_op_failure(
op.id,
&e.to_string(),
&err_str,
now_millis() + backoff.as_millis() as i64,
) {
// The backoff is the only thing keeping a failing op from
// being picked again immediately, so without it the loop
// would spin on this op as fast as the API can refuse it.
error!(uid = %op.uid, error = %e, "recording a drain failure failed");
self.wait_for_drain_work();
}
@@ -97,6 +105,25 @@ impl Core {
*woken = false;
}
/// Like [`wait_for_drain_work`](Self::wait_for_drain_work), but first
/// checks whether a debounced or backed-off op is due before the idle-poll
/// ceiling. A 2-second debounce must not sleep 30 seconds.
fn wait_for_drain_work_or_due(&self) {
let timeout = match self.db.earliest_due_at() {
Ok(Some(ts)) => {
let remaining = (ts - now_millis()).max(0) as u64;
Duration::from_millis(remaining).min(DRAIN_IDLE_POLL)
}
_ => DRAIN_IDLE_POLL,
};
let (lock, cv) = &*self.drain_wake;
let mut woken = lock.lock();
if !*woken {
cv.wait_for(&mut woken, timeout);
}
*woken = false;
}
/// Perform one queued op and retire it.
pub(crate) fn drain_op(&self, op: &PendingOp) -> Result<(), Box<dyn std::error::Error>> {
match op.kind.as_str() {
@@ -467,7 +494,7 @@ impl Core {
uid: &NodeUid,
meta: &StagedWrite,
) -> Result<Option<String>, Box<dyn std::error::Error>> {
let Some(base) = meta.based_on else {
let Some(ref base) = meta.based_on else {
return Ok(None);
};
let Some(node) = self.fetch_node_remote(uid)? else {
@@ -727,7 +754,14 @@ impl Core {
// an orphaned file (harmless), whereas the reverse would leave a queued
// op pointing at nothing.
self.db.delete_op(op.id)?;
self.pending.lock().remove(&uid);
{
let mut p = self.pending.lock();
if let Some(op_in_mem) = p.get(&uid)
&& op_in_mem.path == blob
{
p.remove(&uid);
}
}
// The staged blob now matches the sealed revision, so a pinned file keeps
// it as its cached content rather than re-downloading what we just sent.
@@ -780,6 +814,28 @@ impl Core {
return;
}
};
// Rebase any *open* write handles targeting this node. A handle opened
// before this upload carries a stale base — its `base_mtime`/`base_size`
// name the revision we just replaced. Without this update,
// `remote_baseline` will build a `based_on` from those stale values, the
// drain will compare them against the revision we sealed, and the write
// will be diverted into a conflict copy of itself.
//
// This is the open-handle counterpart of `rebaseline_pending`, which
// covers the same gap for *queued* ops.
{
let sealed_mtime = node.modification_time;
let sealed_size = node_size(&node);
let mut st = self.state.lock();
for aw in st.active_writes.values_mut() {
if aw.uid == *uid {
aw.base_mtime = sealed_mtime;
aw.base_size = sealed_size;
debug!(%uid, mtime = sealed_mtime, size = sealed_size,
"rebased open write handle onto the revision just uploaded");
}
}
}
// Ordered so that a write queued *during* the fetch above is still
// caught: it took its baseline from the node's optimistic stamp, and
// this overwrites it with the revision the server actually holds.
@@ -818,12 +874,13 @@ impl Core {
let base = Baseline {
mtime: sealed.modification_time,
size: node_size(sealed),
hash: None,
};
let mut pending = self.pending.lock();
let Some(p) = pending.get_mut(uid) else {
return false;
};
p.meta.based_on = Some(base);
p.meta.based_on = Some(base.clone());
match serde_json::to_string(&p.meta) {
Ok(json) => {
if let Err(e) = self.db.update_op_meta(&uid.to_string(), OP_REVISION, &json) {
File diff suppressed because it is too large. Load diff
+929 -2650
View File
File diff suppressed because it is too large. Load diff
+392
View File
@@ -0,0 +1,392 @@
//! Mount lifecycle and background-service orchestration.
use super::*;
/// Why a [`mount`] call returned. Lets the daemon decide whether to exit (clean
/// shutdown) or remount (the mount went away under it).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum MountOutcome {
/// The daemon was asked to stop (SIGTERM/SIGINT) and we lazily unmounted
/// ourselves. The caller should exit cleanly.
Shutdown,
/// The kernel mount ended on its own (e.g. an external `fusermount -u`).
/// The caller may want to remount.
Unmounted,
}
/// Whether `path` is a mountpoint whose FUSE connection is dead — the state a
/// daemon killed before it could unmount leaves behind.
///
/// The kernel answers every operation on such a path with `ENOTCONN`, which the
/// ordinary existence checks (`is_dir`, `exists`) report as plain `false`: the
/// path looks *absent* rather than broken. Callers that would otherwise treat
/// that as "nothing to mount here" use this to tell the two apart.
pub(crate) fn is_stale_mount(path: &Path) -> bool {
matches!(
std::fs::metadata(path).map_err(|e| e.raw_os_error()),
Err(Some(libc::ENOTCONN))
)
}
/// A secondary (on-demand sync folder) FUSE session, paired with its FUSE
/// connection id so teardown can abort a mid-transfer connection rather than
/// block on it. See [`Core::mounts`] and [`umount_session_unblocked`].
pub(super) type SecondaryMount = (BackgroundSession, Option<u32>);
/// The kernel's id for the FUSE connection backing the mount at `mountpoint` —
/// the directory name under `/sys/fs/fuse/connections`, which is the minor
/// number of the mountpoint's device. Must be read *while still mounted*; after
/// unmount the path resolves to the underlying directory on another device.
/// `None` when the path can't be stat'd.
pub(super) fn fuse_connection_id(mountpoint: &Path) -> Option<u32> {
use std::os::unix::fs::MetadataExt;
let dev = std::fs::metadata(mountpoint).ok()?.dev();
Some(libc::minor(dev))
}
/// Force the kernel to abort FUSE connection `id`, erroring every in-flight
/// request.
///
/// On a stop signal we unmount lazily (`MNT_DETACH`) so the call succeeds even
/// mid-transfer — but detach only removes the mountpoint; it does *not* end the
/// connection while a request is still in flight. fuser's session loop then
/// blocks on `/dev/fuse` waiting for a next request that never comes, so `join`
/// hangs — long enough during a transfer that systemd's stop timeout SIGKILLs
/// the daemon mid-unmount, stranding the on-demand mounts as dead endpoints.
/// Writing the connection's `abort` file makes the pending reads fail with
/// `ENODEV`, so the loop returns and `join` completes at once. Best-effort:
/// there is nothing more to do on the shutdown path if it fails.
pub(super) fn abort_fuse_connection(id: u32) {
let path = format!("/sys/fs/fuse/connections/{id}/abort");
if std::fs::write(&path, b"1").is_ok() {
info!(id, "aborted FUSE connection to unblock unmount");
}
}
/// Unmount a background session that may be mid-transfer without wedging: abort
/// its connection first (so the session loop exits promptly), then lazily
/// unmount and join. `conn` is the id captured at mount time; `None` skips the
/// abort and just unmounts (a healthy idle mount joins on its own `Destroy`).
pub(super) fn umount_session_unblocked(
session: BackgroundSession,
conn: Option<u32>,
) -> std::io::Result<()> {
if let Some(id) = conn {
abort_fuse_connection(id);
}
session.umount_and_join()
}
/// Best-effort teardown of a stale mount left behind by a crashed daemon. A
/// previous run that died without unmounting leaves the kernel mount in place,
/// so the fresh `Session::new` below would fail with EBUSY ("Device or resource
/// busy"). `fusermount3 -u -z` is the lazy (detach) unmount, which succeeds even
/// when the old mount is still busy. Swallow all output/errors: if there is no
/// stale mount this is simply a no-op.
pub(super) fn clear_stale_mount(mountpoint: &Path) {
for bin in ["fusermount3", "fusermount"] {
let ok = std::process::Command::new(bin)
.arg("-u")
.arg("-z")
.arg(mountpoint)
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false);
if ok {
info!(mountpoint = %mountpoint.display(), "cleared stale mount before remount");
return;
}
}
}
/// `sync_state` key holding the uid of the My Files root, so a later run can
/// recover it from `nodes` without the API (offline.md Phase 1).
const ROOT_UID_KEY: &str = "root_uid";
/// The My Files root, and whether we got it from the API (`true`) or from the
/// cache because the API was unreachable (`false`).
///
/// A successful fetch also records the root's uid, which is what makes the
/// fallback possible on a later run: `nodes` is keyed by uid, so without this we
/// would have the root's row on disk and no way to tell which one it is.
///
/// Failing to fetch is only fatal on a genuinely cold start — no cached root
/// means an empty tree, and mounting that would show the user an empty Drive
/// rather than an honest error.
fn fetch_or_recall_root(
client: &ProtonDriveClient,
rt: &tokio::runtime::Handle,
db: &Db,
) -> std::io::Result<(Node, bool)> {
let err = match rt.block_on(client.get_my_files_folder()) {
Ok(root) => {
if let Err(e) = db.set_state_str(ROOT_UID_KEY, &root.uid.to_string()) {
warn!(error = %e, "persist root uid failed");
}
return Ok((root, true));
}
Err(e) => e,
};
let cached = db
.state_str(ROOT_UID_KEY)
.ok()
.flatten()
.and_then(|uid| db.node_by_uid(&uid).ok().flatten());
match cached {
Some(root) => {
warn!(error = %err, "fetch My Files root failed; mounting from cache (offline)");
Ok((root, false))
}
None => Err(std::io::Error::other(format!("fetch My Files root: {err}"))),
}
}
/// Mount the filesystem at `mountpoint` and block until it is unmounted or the
/// daemon is asked to stop.
///
/// Resolves the My Files root up front — from the API, or from the cached tree
/// when the API is unreachable (`fetch_or_recall_root`) — then spawns the
/// Phase 2 event-sync task, the
/// Phase 4 control socket, and runs the FUSE session on its own thread while
/// this thread waits for either a stop signal (SIGTERM/SIGINT) or the kernel
/// mount ending on its own. On a stop signal we lazily unmount ourselves
/// (`umount_and_join`, the MNT_DETACH path that succeeds even while downloads
/// are in flight), so `systemctl --user stop` is always a clean teardown.
/// `rt` must be a handle to a *running* multi-threaded runtime.
pub fn mount(
client: ProtonDriveClient,
rt: tokio::runtime::Handle,
mountpoint: &Path,
cache: ContentCache,
control_socket: &Path,
db: Arc<Db>,
username: String,
) -> std::io::Result<MountOutcome> {
let (root, online) = fetch_or_recall_root(&client, &rt, &db)?;
let scope = root.tree_event_scope_id();
// The folder-sync engine (devices.md Phase 2) runs on its own thread and is
// nudged over this channel; the sender lives in Core so control-socket
// handlers can trigger reconciles.
let (sync_tx, sync_rx) = std::sync::mpsc::channel::<sync::SyncMsg>();
let core = Core {
client: client.clone(),
rt: rt.clone(),
state: Arc::new(Mutex::new(State {
entries: HashMap::new(),
by_uid: HashMap::new(),
children: HashMap::new(),
next_ino: 2,
active_writes: HashMap::new(),
handles: HashMap::new(),
next_fh: 1,
db: db.clone(),
})),
cache: Arc::new(cache),
readers: Arc::new(Mutex::new(HashMap::new())),
stream_ring: Arc::new(Mutex::new(StreamRing::default())),
workers: Arc::new(Workers::new(FUSE_WORKERS)?),
db,
online: Arc::new(AtomicBool::new(online)),
pending: Arc::new(Mutex::new(HashMap::new())),
drain_wake: Arc::new((Mutex::new(false), Condvar::new())),
timeline_refreshing: Arc::new(AtomicBool::new(false)),
trash_refreshing: Arc::new(AtomicBool::new(false)),
thumb_gen: Arc::new(Mutex::new(HashSet::new())),
no_thumbnail: Arc::new(Mutex::new(HashMap::new())),
size_upgrades: Arc::new(Mutex::new(HashMap::new())),
notifier: Arc::new(OnceLock::new()),
transfers: TransferRegistry::new(),
indexing: Arc::new(AtomicBool::new(false)),
sync_progress: Arc::new(Mutex::new(HashMap::new())),
sync_tx,
mounts: Arc::new(Mutex::new(HashMap::new())),
sync_locks: Arc::new(Mutex::new(HashMap::new())),
};
// Writes queued by a previous run (or left behind by a crash) are still owed
// an upload, and reads must be served from their staged blobs until they land.
core.hydrate_pending();
// Then the writes that were fsynced but never closed, which the cache moved
// aside at open. After `hydrate_pending`, so a recovered partial write can
// see an already-queued write to the same node.
core.recover_fsynced_writes();
{
let core = core.clone();
std::thread::Builder::new()
.name("pdfs-drain".into())
.spawn(move || core.run_pending_drain())?;
}
// Start the folder-sync engine. It watches every mirror folder, polls the
// remotes, and reconciles on its own thread — never in front of a FUSE call.
sync::spawn(core.clone(), sync_rx);
// Mounted from the cache: watch for the network coming back so the mount can
// stop being read-only-ish without the user restarting the daemon.
if !online {
let core = core.clone();
std::thread::Builder::new()
.name("pdfs-online-probe".into())
.spawn(move || core.run_online_probe())?;
}
// Re-establish on-demand mounts left over from a previous run (devices.md
// Phase 4). On its own thread: each remount fetches a remote node, and we
// must not block the main mount below on the network.
{
let core = core.clone();
std::thread::Builder::new()
.name("pdfs-restore-ondemand".into())
.spawn(move || core.restore_ondemand_mounts())?;
}
// Keep the launcher prompt's "This computer" index fresh. Its own thread:
// the walk is I/O-heavy and must never sit in front of a FUSE callback.
{
let db = core.db.clone();
let indexing = core.indexing.clone();
let transfers = core.transfers.clone();
let mountpoint = mountpoint.to_path_buf();
std::thread::Builder::new()
.name("pdfs-localindex".into())
.spawn(move || run_local_index(db, indexing, transfers, mountpoint))?;
}
let mut config = Config::default();
config.mount_options = vec![
MountOption::FSName("protondrive".to_string()),
MountOption::Subtype("protondrive".to_string()),
MountOption::DefaultPermissions,
];
// A crashed previous run can leave the kernel mount in place, which makes
// the fresh mount below fail with EBUSY. Lazily detach any leftover first.
clear_stale_mount(mountpoint);
info!(mountpoint = %mountpoint.display(), "mounting Proton Drive");
// Bind the control socket before the FUSE session takes over the thread. A
// stale socket file from a previous run would block the bind, so clear it.
let _ = std::fs::remove_file(control_socket);
let old_umask = unsafe { libc::umask(0o77) };
let listener_res = UnixListener::bind(control_socket);
unsafe { libc::umask(old_umask) };
let listener = listener_res?;
// Owner-only before anything can connect: a peer on this socket commands the
// daemon's authenticated session without a credential of its own (B6).
if let Err(e) = pdfs_core::config::restrict_socket(control_socket) {
error!(error = %e, "cannot restrict control socket permissions; refusing to serve");
let _ = std::fs::remove_file(control_socket);
return Err(std::io::Error::other(format!(
"control socket permissions: {e}"
)));
}
{
let core = core.clone();
let username = username.clone();
let mountpoint = mountpoint.to_path_buf();
std::thread::Builder::new()
.name("pdfs-control".into())
.spawn(move || run_control_socket(core, username, mountpoint, listener))?;
}
let fs = ProtonFs::new(core.clone(), root);
// Warm the in-memory maps from the DB so a cold start serves previously
// seen metadata without re-hitting the API (plan.md P1).
core.hydrate();
// Build the session explicitly (not `mount2`) so we can grab a `Notifier`
// for the event task. `spawn` runs the session loop on its own background
// thread; we then wait here for either a stop signal or the mount ending.
let bg = Session::new(fs, mountpoint, &config)?.spawn()?;
// The connection id, captured now while the mount is live, so a stop signal
// mid-transfer can abort it rather than block `join` (see `abort_fuse_connection`).
let main_conn = fuse_connection_id(mountpoint);
let notifier = bg.notifier();
// Same channel, kept on the `Core` so background work (a size upgrade, say)
// can invalidate kernel-cached metadata without threading a handle through.
let _ = core.notifier.set(notifier.clone());
rt.spawn(run_event_sync(
client,
scope,
core.state,
core.cache,
core.db,
core.pending,
notifier,
));
// Stop signals (SIGTERM from `systemctl --user stop`, SIGINT from Ctrl-C)
// are delivered onto the async runtime; bridge them onto a sync channel so
// the loop below can react without blocking a worker thread. A bounded
// channel of 1 is enough — we only need to know that *a* stop arrived.
let (sig_tx, sig_rx) = std::sync::mpsc::sync_channel::<()>(1);
rt.spawn(async move {
let mut sigterm =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGTERM handler failed");
return;
}
};
let mut sigint =
match tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt()) {
Ok(s) => s,
Err(e) => {
warn!(error = %e, "install SIGINT handler failed");
return;
}
};
tokio::select! {
_ = sigterm.recv() => info!("received SIGTERM"),
_ = sigint.recv() => info!("received SIGINT"),
}
let _ = sig_tx.try_send(());
});
// Wait for whichever happens first: a stop signal (→ we unmount ourselves
// via the lazy MNT_DETACH path, clean even mid-download), or the kernel
// mount ending on its own (→ the session thread finishes). Poll instead of
// blocking on `join` so we can also notice the signal.
let outcome = loop {
match sig_rx.recv_timeout(Duration::from_millis(500)) {
Ok(()) => {
info!("stop requested; unmounting");
if let Err(e) = umount_session_unblocked(bg, main_conn) {
warn!(error = %e, "umount_and_join failed");
}
break MountOutcome::Shutdown;
}
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {
if bg.guard.is_finished() {
info!("mount ended externally");
if let Err(e) = bg.join() {
warn!(error = %e, "session join failed");
}
break MountOutcome::Unmounted;
}
}
Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => {
// Signal task gone (failed to install); fall back to join.
let _ = bg.join();
break MountOutcome::Unmounted;
}
}
};
// Unmount every on-demand sync folder too, or the kernel mounts linger as
// stale and the next start fails with EBUSY (devices.md Phase 3).
let secondaries: Vec<_> = core.mounts.lock().drain().collect();
for (id, (session, conn)) in secondaries {
if let Err(e) = umount_session_unblocked(session, conn) {
warn!(id, error = %e, "unmount on-demand folder failed");
}
}
let _ = std::fs::remove_file(control_socket);
Ok(outcome)
}
+127 -2
View File
@@ -18,10 +18,78 @@ use proton_drive_rs::{NodeKind, ThumbnailType};
use tracing::{info, warn};
use super::{
Core, PHOTOS_AVAILABLE, PHOTOS_SYNCED_MS, THUMB_GEN_CONCURRENCY, TIMELINE_ENRICH_CHUNK,
TIMELINE_TTL, ThumbAttempt, node_size, now_ms, parse_uid, ratio_of, scale_thumbnail,
Core, PHOTOS_AVAILABLE, PHOTOS_SYNCED_MS, TIMELINE_ENRICH_CHUNK, TIMELINE_TTL, node_size,
now_ms, parse_uid,
};
/// Longest edge, in px, of a thumbnail generated locally for a photo the server
/// has none for. Matches the server's own thumbnail scale closely enough that a
/// tile can't tell them apart.
const THUMB_EDGE: u32 = 512;
/// JPEG quality of a locally generated thumbnail.
const THUMB_QUALITY: u8 = 82;
/// How many photos may be downloaded at once to generate their missing
/// thumbnails. Bounded: a screenful of 20 MB digicam JPEGs would otherwise
/// saturate the link and starve the rest of the daemon.
const THUMB_GEN_CONCURRENCY: usize = 4;
/// The aspect ratio (w/h) of an encoded image, read from its header alone — no
/// pixels are decoded. `None` when the format is unknown or the header is torn.
fn ratio_of(bytes: &[u8]) -> Option<f64> {
let (width, height) = image::ImageReader::new(std::io::Cursor::new(bytes))
.with_guessed_format()
.ok()?
.into_dimensions()
.ok()?;
(height > 0).then(|| f64::from(width) / f64::from(height))
}
/// A thumbnail the daemon made itself, and the aspect ratio of the photo it was
/// made from (free at that point — the full image had to be decoded anyway).
struct GeneratedThumb {
bytes: Vec<u8>,
ratio: f64,
}
/// How one attempt at generating a missing thumbnail ended. The distinction that
/// matters is *permanent* versus *transient*: only bytes we cannot decode prove
/// the photo will never have a thumbnail, and only that verdict is persisted.
enum ThumbAttempt {
Made(GeneratedThumb),
/// Decoded nothing — a format this build has no decoder for. Permanent.
Undecodable,
/// The photo couldn't be downloaded. Transient: try again next time.
Unavailable,
}
/// Scale a full-size photo down to a thumbnail: at most [`THUMB_EDGE`] on its
/// longest side, JPEG, aspect ratio preserved. `None` when the bytes aren't an
/// image this build can decode — the caller then writes the photo off as
/// un-thumbnailable.
///
/// CPU-bound (a 20 MP JPEG is real work), so callers run it on the blocking pool.
fn scale_thumbnail(bytes: &[u8]) -> Option<GeneratedThumb> {
let image = image::ImageReader::new(std::io::Cursor::new(bytes))
.with_guessed_format()
.ok()?
.decode()
.ok()?;
let (width, height) = (image.width(), image.height());
if width == 0 || height == 0 {
return None;
}
let ratio = f64::from(width) / f64::from(height);
// `thumbnail` fits the image *inside* the box, so the longest edge lands on
// THUMB_EDGE and the ratio is untouched.
let thumb = image.thumbnail(THUMB_EDGE, THUMB_EDGE).to_rgb8();
let mut bytes = Vec::new();
image::codecs::jpeg::JpegEncoder::new_with_quality(&mut bytes, THUMB_QUALITY)
.encode_image(&thumb)
.ok()?;
Some(GeneratedThumb { bytes, ratio })
}
impl Core {
pub(crate) fn photos_timeline(
&self,
@@ -361,3 +429,60 @@ impl Core {
Ok(self.cache.content_path(uid))
}
}
#[cfg(test)]
mod thumb_tests {
use super::{THUMB_EDGE, ratio_of, scale_thumbnail};
/// A `width`×`height` JPEG, standing in for a camera photo the server never
/// generated a thumbnail for.
fn jpeg(width: u32, height: u32) -> Vec<u8> {
let image = image::RgbImage::from_fn(width, height, |x, y| {
image::Rgb([(x % 256) as u8, (y % 256) as u8, 128])
});
let mut bytes = Vec::new();
image::codecs::jpeg::JpegEncoder::new(&mut bytes)
.encode_image(&image)
.unwrap();
bytes
}
#[test]
fn scaling_fits_the_long_edge_and_keeps_the_aspect_ratio() {
let photo = jpeg(4000, 3000);
let thumb = scale_thumbnail(&photo).expect("a JPEG scales");
let (width, height) = image::ImageReader::new(std::io::Cursor::new(&thumb.bytes))
.with_guessed_format()
.unwrap()
.into_dimensions()
.unwrap();
assert_eq!(width, THUMB_EDGE, "the long edge lands on the target");
assert_eq!(height, THUMB_EDGE * 3 / 4, "and nothing is cropped");
assert!((thumb.ratio - 4.0 / 3.0).abs() < 1e-6);
assert!(
thumb.bytes.len() < photo.len(),
"a thumbnail that isn't smaller than its photo is no thumbnail"
);
}
#[test]
fn a_portrait_photo_fits_its_long_edge_too() {
let thumb = scale_thumbnail(&jpeg(1000, 2000)).expect("a JPEG scales");
assert!(thumb.ratio < 1.0, "portrait stays portrait");
assert_eq!(ratio_of(&thumb.bytes).map(|r| r < 1.0), Some(true));
}
#[test]
fn undecodable_bytes_are_not_a_thumbnail() {
// What a photo in a format this build has no decoder for looks like: the
// caller writes it off as un-thumbnailable rather than retrying forever.
assert!(scale_thumbnail(b"not an image at all").is_none());
assert!(ratio_of(b"not an image at all").is_none());
}
#[test]
fn ratio_is_read_from_the_header_alone() {
assert_eq!(ratio_of(&jpeg(300, 200)), Some(1.5));
}
}
+394
View File
@@ -0,0 +1,394 @@
//! Reading and writing this machine's `profile.json` in its device root, and
//! the restore that rebuilds a machine from one (features.md 5.2 and 5.3).
//!
//! The document itself is defined in [`pdfs_core::profile`]; this is the half
//! that talks to Drive. Saving is best-effort and never blocks a user action:
//! the profile is a convenience for a *future* machine, so failing to write it
//! must not fail the folder add that triggered it.
use std::io::Cursor;
use std::path::{Path, PathBuf};
use std::sync::OnceLock;
use std::sync::atomic::{AtomicBool, Ordering};
use pdfs_core::control::RestorableFolder;
use pdfs_core::profile::{PROFILE_FILE_NAME, PROFILE_VERSION, Profile, ProfileFolder, ProfilePin};
use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::proton_sdk::ids::NodeUid;
use tracing::{info, warn};
use super::sync;
use super::{Core, now_secs, parse_uid, this_hostname};
/// Coalescing flags for background profile saves.
///
/// A single burst of user actions (adding three folders, pinning a tree) should
/// cost one upload, not one per action. `SAVING` admits one worker at a time and
/// `DIRTY` records that something changed while it was busy, so the worker takes
/// exactly one more lap afterwards.
///
/// Process-global rather than a [`Core`] field because on-demand mounts fork
/// their own `Core` (`Core::fork_state`) and all of them describe the same
/// machine — there is one profile per daemon, so there is one flag pair.
static SAVE_STATE: OnceLock<(AtomicBool, AtomicBool)> = OnceLock::new();
fn save_state() -> &'static (AtomicBool, AtomicBool) {
SAVE_STATE.get_or_init(|| (AtomicBool::new(false), AtomicBool::new(false)))
}
impl Core {
/// Note that the arrangement this machine's profile describes has changed,
/// and upload a fresh profile in the background.
///
/// Fire-and-forget by design: callers are mutation paths (add/remove a sync
/// folder, switch a mode, pin a node, save settings) whose success does not
/// depend on the backup succeeding.
pub(crate) fn touch_profile(&self) {
let (saving, dirty) = save_state();
dirty.store(true, Ordering::SeqCst);
// Someone is already uploading; they will see DIRTY and take another lap.
if saving.swap(true, Ordering::SeqCst) {
return;
}
let core = self.clone();
std::thread::spawn(move || {
let (saving, dirty) = save_state();
while dirty.swap(false, Ordering::SeqCst) {
if let Err(e) = core.save_profile() {
warn!(error = %e.message, "profile backup failed");
break;
}
}
saving.store(false, Ordering::SeqCst);
});
}
/// Serialize this machine's arrangement and upload it to `profile.json` in
/// the device root, replacing any existing one.
///
/// Does nothing when no device is registered: a machine that has never
/// synced a folder has nothing to describe, and registering a device just to
/// hold an empty profile would put a stray computer in the user's Drive.
pub(crate) fn save_profile(&self) -> CoreResult<()> {
let Some(device) = self
.db
.device_get()
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?
else {
return Ok(());
};
let root_uid = parse_uid(&device.root_uid).ok_or_else(|| {
CoreError::internal(format!("bad device root uid: {}", device.root_uid))
})?;
let profile = self.build_profile(&device.uid)?;
let bytes = profile.to_bytes().map_err(CoreError::internal)?;
let len = bytes.len() as i64;
// Replace the existing document rather than adding a second file of the
// same name: the profile is a single mutable record, and a device root
// with `profile.json` and `profile (1).json` in it teaches the restore
// nothing about which to trust.
let existing = self.find_device_child_file(&root_uid, PROFILE_FILE_NAME)?;
match existing {
Some(uid) => self
.rt
.block_on(self.client.upload_new_revision_from(
&uid,
Cursor::new(bytes),
len,
Vec::new(),
Some(now_secs()),
))
.map(|_| ())
.map_err(|e| CoreError::from_api(&e, "upload profile revision"))?,
None => {
self.rt
.block_on(self.client.upload_file_replacing_draft_from(
&root_uid,
PROFILE_FILE_NAME,
"application/json",
Cursor::new(bytes),
len,
Vec::new(),
Some(now_secs()),
false,
))
.map_err(|e| CoreError::from_api(&e, "upload profile"))?;
}
}
info!(folders = profile.folders.len(), "profile backed up");
Ok(())
}
/// Snapshot the local arrangement into a [`Profile`].
fn build_profile(&self, device_uid: &str) -> CoreResult<Profile> {
let folders = self
.db
.sync_folder_list()
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?
.into_iter()
.map(|f| ProfileFolder {
remote_uid: f.remote_uid,
local_path: f.local_path,
// A queued switch is where the user asked the folder to be, so
// that is what the next machine should set up.
mode: f.pending_mode.unwrap_or(f.mode),
})
.collect();
let pins = self
.db
.pin_list()
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?
.into_iter()
.map(|(uid, path, recursive)| ProfilePin {
uid,
path,
recursive,
})
.collect();
let cfg = pdfs_core::config::AppDirs::new()
.map(|d| d.load_config())
.unwrap_or_default();
Ok(Profile {
version: PROFILE_VERSION,
device_uid: device_uid.to_string(),
hostname: this_hostname(),
saved_at: now_secs(),
folders,
pins,
ignore_patterns: cfg.ignore_patterns,
cache_budget: cfg.cache_budget,
mountpoint: cfg.mountpoint,
})
}
/// An untrashed *file* named `name` directly under the device root.
///
/// The sibling of `find_device_child_folder`; kept separate because a folder
/// named `profile.json` must not be mistaken for the profile.
fn find_device_child_file(
&self,
root_uid: &NodeUid,
name: &str,
) -> CoreResult<Option<NodeUid>> {
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(root_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?;
if uids.is_empty() {
return Ok(None);
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?;
Ok(nodes
.into_iter()
.find(|n| !n.is_folder() && !n.trashed && n.name == name)
.map(|n| n.uid))
}
/// Download and parse the profile in the device root, if there is one.
fn load_profile(&self, root_uid: &NodeUid) -> CoreResult<Option<Profile>> {
let Some(uid) = self.find_device_child_file(root_uid, PROFILE_FILE_NAME)? else {
return Ok(None);
};
let mut buf: Vec<u8> = Vec::new();
self.rt
.block_on(self.client.download_file_to(&uid, &mut buf))
.map_err(|e| CoreError::from_api(&e, "download profile"))?;
Profile::parse(&buf).map(Some).map_err(CoreError::invalid)
}
// ---- restore (features.md 5.2) ----------------------------------------
/// The folders under this machine's device that could be synced here, each
/// carrying a proposed local path.
///
/// The proposal comes from the profile when one exists and its path is
/// usable on this machine, and from `~/<name>` otherwise — a restored
/// `/home/other-user/Documents` would be wrong on a machine where that user
/// does not exist, and silently creating it would be worse.
pub(crate) fn list_restorable_folders(&self) -> CoreResult<Vec<RestorableFolder>> {
let device = self.ensure_device()?;
let root_uid = parse_uid(&device.root_uid).ok_or_else(|| {
CoreError::internal(format!("bad device root uid: {}", device.root_uid))
})?;
let profile = match self.load_profile(&root_uid) {
Ok(p) => p,
// A missing or unreadable profile is not fatal: the remote tree
// alone is enough to offer a restore, just with plainer defaults.
Err(e) => {
warn!(error = %e.message, "could not read profile; proposing defaults");
None
}
};
let uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(&root_uid))
.map_err(|e| CoreError::from_api(&e, "list device root"))?;
let nodes = if uids.is_empty() {
Vec::new()
} else {
self.rt
.block_on(self.client.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "resolve device root children"))?
};
// Folders already synced here are not restorable — they are already
// restored — but they are still listed, flagged, so the picker can show
// the whole device rather than a confusing subset.
let existing: Vec<String> = self
.db
.sync_folder_list()
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?
.into_iter()
.map(|f| f.remote_uid)
.collect();
Ok(nodes
.into_iter()
.filter(|n| n.is_folder() && !n.trashed)
.map(|n| {
let uid = n.uid.to_string();
let saved = profile
.as_ref()
.and_then(|p| p.folders.iter().find(|f| f.remote_uid == uid));
let proposed = saved
.map(|f| PathBuf::from(&f.local_path))
.filter(|p| path_is_usable_here(p))
.unwrap_or_else(|| default_local_path(&n.name));
RestorableFolder {
already_synced: existing.iter().any(|e| e == &uid),
remote_uid: uid,
name: n.name,
local_path: proposed.to_string_lossy().to_string(),
mode: saved
.map(|f| f.mode.clone())
.unwrap_or_else(|| "mirror".into()),
}
})
.collect())
}
/// Attach remote device folders to local paths and let the sync engine fill
/// them in.
///
/// The download is not written here: a new sync-folder row starts with an
/// empty baseline, and an empty baseline against an empty local directory
/// and a populated remote already classifies as "download everything". The
/// restore is therefore just bookkeeping plus a `Reconcile`, and it reuses
/// the engine's conflict handling for the case where the local directory
/// turns out not to be empty after all.
pub(crate) fn restore_sync_folders(
&self,
items: &[pdfs_core::control::RestoreItem],
) -> CoreResult<String> {
if items.is_empty() {
return Err(CoreError::invalid("nothing to restore"));
}
let device = self.ensure_device()?;
let existing = self
.db
.sync_folder_list()
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?;
let mut restored = 0usize;
let mut skipped: Vec<String> = Vec::new();
for item in items {
let local = PathBuf::from(&item.local_path);
if !local.is_absolute() {
skipped.push(format!("{}: local path must be absolute", item.local_path));
continue;
}
if existing.iter().any(|f| f.remote_uid == item.remote_uid) {
skipped.push(format!("{}: already synced", item.local_path));
continue;
}
if existing
.iter()
.any(|f| Path::new(&f.local_path) == local.as_path())
{
skipped.push(format!("{}: path already in use", item.local_path));
continue;
}
if let Err(e) = std::fs::create_dir_all(&local) {
skipped.push(format!("{}: {e}", item.local_path));
continue;
}
let mode = match item.mode.as_str() {
"mirror" | "ondemand" => item.mode.as_str(),
other => {
skipped.push(format!("{}: unknown mode {other}", item.local_path));
continue;
}
};
let id = self
.db
.sync_folder_add(&local.to_string_lossy(), &item.remote_uid, &device.share_id)
.map_err(|e| CoreError::internal(format!("db: {e:?}")))?;
// `ondemand` is applied through the normal request path, which
// mounts the FUSE session and handles the folder being busy — the
// restore does not get its own copy of that logic.
if mode == "ondemand" {
let _ = self.request_sync_folder_mode(id, "ondemand");
}
let _ = self.sync_tx.send(sync::SyncMsg::Reconcile(id));
restored += 1;
}
let _ = self.sync_tx.send(sync::SyncMsg::Rewatch);
let mut message = format!("restored {restored} folder(s)");
if !skipped.is_empty() {
message.push_str(&format!("; skipped: {}", skipped.join(", ")));
}
info!(restored, skipped = skipped.len(), "restore complete");
Ok(message)
}
}
/// Whether a path recorded on another machine can be reused verbatim here.
///
/// Requires the *parent* to exist: `~/Documents` on a machine with the same
/// username is a fine proposal even though the directory itself is gone (the
/// restore creates it), but `/home/someone-else/Documents` is not.
fn path_is_usable_here(path: &Path) -> bool {
path.is_absolute() && path.parent().is_some_and(|p| p.is_dir())
}
/// `~/<name>`, or `/<name>` if the home directory cannot be determined.
fn default_local_path(name: &str) -> PathBuf {
match pdfs_core::config::AppDirs::new()
.ok()
.and_then(|d| d.home_dir())
{
Some(home) => home.join(name),
None => PathBuf::from("/").join(name),
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn foreign_home_is_not_reused() {
assert!(!path_is_usable_here(Path::new(
"/home/definitely-not-a-user-here/Documents"
)));
assert!(!path_is_usable_here(Path::new("relative/path")));
}
/// A path whose parent exists but which does not itself is exactly the
/// restore case — the directory is about to be created.
#[test]
fn missing_leaf_under_existing_parent_is_reused() {
let dir = std::env::temp_dir().join("pdfs-restore-probe");
std::fs::create_dir_all(&dir).unwrap();
assert!(path_is_usable_here(&dir.join("Documents")));
let _ = std::fs::remove_dir_all(&dir);
}
}
+645
View File
@@ -0,0 +1,645 @@
//! Revision-reader cache and streamed range-read subsystem.
use super::*;
/// A read of an unpinned video at least this large streams *without* persisting
/// its blocks to the on-disk cache. Playing a 2 GB film would otherwise pour it
/// through the block LRU and evict everything else the user actually wants kept —
/// and it re-streams cheaply enough on a rewatch that keeping it was never worth
/// that. Pinned videos (kept offline on purpose) and anything smaller cache as
/// usual.
pub(super) const STREAM_BYPASS_MIN: u64 = 256 * 1024 * 1024;
/// How many bytes of streamed blocks the in-memory ring keeps. Bypassing the
/// on-disk cache must not mean re-fetching: the kernel asks for a streamed file
/// in reads far smaller than [`BLOCK_SIZE`], so without a ring every 128 KiB the
/// player consumes would download and decrypt a whole 4 MiB block again. Sized
/// for a handful of blocks per concurrently-streamed file.
const STREAM_RING_BYTES: u64 = 128 * 1024 * 1024;
/// Blocks fetched *past* the one a sequential streaming read needed, warmed into
/// the ring in the background so the player's next read is already in memory
/// instead of waiting a round-trip. Kept small so it cannot crowd out the
/// in-flight block governor.
const STREAM_READAHEAD: u64 = 4;
/// How many [`RevisionReader`]s stay open at once.
///
/// A reader holds its revision's content key and block table — a few KB even for
/// a large file, so this is bounded for tidiness and staleness rather than
/// memory. Evicted least-recently-used; a dropped reader costs one re-open
/// (two API calls and a node-key unlock) the next time that file is read.
const MAX_OPEN_READERS: usize = 64;
/// An open reader plus the node metadata it was opened against.
///
/// The SDK pins a reader to the revision that was active at `open_revision`, so
/// a reader is only reusable while the node still reports the same
/// `(mtime, size)` — the same validity pair the content cache uses (a new
/// revision bumps mtime). On a mismatch the reader is dropped and reopened.
pub(super) struct CachedReader {
reader: Arc<RevisionReader>,
mtime: i64,
size: u64,
/// For LRU eviction.
last_used: Instant,
}
/// What the `readers` map holds for a node: an open reader, or the fact that
/// someone is already opening one.
///
/// The `Pending` arm is what makes the map single-flight rather than merely a
/// cache. Opening resolves link details, ancestor keys, an S2K node-key unlock
/// and the block table — five round-trips — and read-ahead on a cold file puts
/// several workers into the open at the same moment, so without this each
/// racer would redo all of it and discard every result but one.
pub(super) enum ReaderSlot {
Ready(CachedReader),
Pending(PendingOpen),
}
/// An `open_revision` in flight. Racers clone `rx` and await the leader's result
/// instead of opening their own.
pub(super) struct PendingOpen {
/// Identifies this attempt so the leader can tell "my slot is still there"
/// from "someone evicted or replaced it while I was on the network", and
/// only publish its reader in the first case.
id: u64,
/// What the leader is opening against. A racer wanting a different revision
/// must not join this open — it would get a reader for the wrong bytes.
mtime: i64,
size: u64,
rx: tokio::sync::watch::Receiver<Option<Result<Arc<RevisionReader>, Errno>>>,
}
/// Distinguishes `PendingOpen`s. Only uniqueness matters, so a plain counter
/// does; ids are never compared for order.
static NEXT_OPEN_ID: AtomicU64 = AtomicU64::new(0);
/// Drive client, a Tokio handle to bridge the synchronous FUSE/socket threads
/// to the async SDK, the inode bookkeeping, and the on-disk content cache.
///
/// Cheaply cloneable (every field is a handle/`Arc`), so the control-socket task
/// gets its own copy while the FUSE session keeps another.
/// In-memory LRU of blocks belonging to a file streaming past the on-disk cache
/// (see [`STREAM_BYPASS_MIN`]). Bypassing the disk stops one film evicting the
/// block LRU; it must not also mean the same 4 MiB block is downloaded once per
/// 128 KiB the player reads out of it.
///
/// Validated by the same `(mtime, size)` pair the on-disk caches use: a node
/// whose tag no longer matches has its blocks dropped rather than served, so a
/// new revision can never be stitched together from the old one.
#[derive(Default)]
pub(super) struct StreamRing {
blocks: HashMap<(NodeUid, u64), Arc<Vec<u8>>>,
/// Keys oldest-first, for eviction.
order: VecDeque<(NodeUid, u64)>,
/// Per-node validity tag, `(mtime, size)`.
tags: HashMap<NodeUid, (i64, u64)>,
bytes: u64,
}
impl StreamRing {
fn get(&mut self, uid: &NodeUid, mtime: i64, size: u64, idx: u64) -> Option<Arc<Vec<u8>>> {
if self.tags.get(uid) != Some(&(mtime, size)) {
return None;
}
self.blocks.get(&(uid.clone(), idx)).cloned()
}
fn insert(&mut self, uid: &NodeUid, mtime: i64, size: u64, idx: u64, bytes: Arc<Vec<u8>>) {
if self.tags.get(uid) != Some(&(mtime, size)) {
// Revision changed under us (or first sight): anything held for this
// node describes the old one.
self.drop_node(uid);
self.tags.insert(uid.clone(), (mtime, size));
}
let key = (uid.clone(), idx);
if self.blocks.contains_key(&key) {
return;
}
self.bytes += bytes.len() as u64;
self.blocks.insert(key.clone(), bytes);
self.order.push_back(key);
while self.bytes > STREAM_RING_BYTES {
let Some(victim) = self.order.pop_front() else {
break;
};
if let Some(dropped) = self.blocks.remove(&victim) {
self.bytes -= dropped.len() as u64;
}
}
}
fn drop_node(&mut self, uid: &NodeUid) {
self.order.retain(|(u, _)| u != uid);
let mut freed = 0u64;
self.blocks.retain(|(u, _), bytes| {
if u == uid {
freed += bytes.len() as u64;
false
} else {
true
}
});
self.bytes -= freed;
self.tags.remove(uid);
}
}
impl Core {
/// An open [`RevisionReader`] for `uid`, reusing the cached one when it is
/// still valid for `(mtime, fsize)` and opening a fresh one otherwise.
///
/// Opening resolves the file's link details, ancestor keys, node key (an S2K
/// unlock) and block table. Doing that once per file rather than once per
/// block is the whole point: a cold 100 MB read is 25 block misses, which
/// used to mean 25 full resolutions (50 API calls, 25 unlocks) and now means
/// one.
///
/// Racing reads of one cold file open it **once**: the first caller claims
/// the slot and the rest await its result. That matters now that `read` is
/// served off the dispatch loop, because read-ahead on a first touch issues
/// several parallel reads of the same file as a matter of course.
async fn revision_reader(
&self,
uid: &NodeUid,
mtime: i64,
fsize: u64,
) -> Result<Arc<RevisionReader>, Errno> {
loop {
// Decide under the lock, act outside it — network I/O must never run
// under a std `Mutex`, and an await must never hold one.
enum Act {
Hit(Arc<RevisionReader>),
Join(tokio::sync::watch::Receiver<Option<Result<Arc<RevisionReader>, Errno>>>),
Lead(
u64,
tokio::sync::watch::Sender<Option<Result<Arc<RevisionReader>, Errno>>>,
),
}
let act = {
let mut readers = self.readers.lock();
match readers.get_mut(uid) {
Some(ReaderSlot::Ready(entry))
if entry.mtime == mtime && entry.size == fsize =>
{
entry.last_used = Instant::now();
Act::Hit(entry.reader.clone())
}
Some(ReaderSlot::Pending(p)) if p.mtime == mtime && p.size == fsize => {
Act::Join(p.rx.clone())
}
// Either nothing here, or something for a revision we no
// longer want (a stale reader, or an open for one). Ours
// replaces it: the newer `(mtime, size)` is the truth.
_ => {
let id = NEXT_OPEN_ID.fetch_add(1, Ordering::Relaxed);
let (tx, rx) = tokio::sync::watch::channel(None);
readers.insert(
uid.clone(),
ReaderSlot::Pending(PendingOpen {
id,
mtime,
size: fsize,
rx,
}),
);
Act::Lead(id, tx)
}
}
};
match act {
Act::Hit(reader) => return Ok(reader),
Act::Join(mut rx) => {
// `changed()` errors only if the leader dropped its sender
// without publishing — it panicked. Retry from the top
// rather than fail: the slot it left is gone, so this pass
// leads its own open.
loop {
if let Some(result) = rx.borrow_and_update().clone() {
return result;
}
if rx.changed().await.is_err() {
break;
}
}
continue;
}
Act::Lead(id, tx) => {
debug!(%uid, mtime, fsize, "opening revision");
let result = match self.client.open_revision(uid).await {
Ok(reader) => Ok(Arc::new(reader)),
Err(e) => {
warn!(%uid, error = %e, "open_revision failed");
Err(Errno::EIO)
}
};
{
let mut readers = self.readers.lock();
// Publish only into the slot we claimed. If it is gone
// or belongs to a later attempt, an eviction or a newer
// revision overtook us: our reader still answers the
// callers waiting on it, but it must not be cached.
let ours = matches!(
readers.get(uid),
Some(ReaderSlot::Pending(p)) if p.id == id
);
if ours {
match &result {
Ok(reader) => {
readers.insert(
uid.clone(),
ReaderSlot::Ready(CachedReader {
reader: reader.clone(),
mtime,
size: fsize,
last_used: Instant::now(),
}),
);
}
// Never cache a failure: the next read retries.
Err(_) => {
readers.remove(uid);
}
}
}
Self::trim_readers(&mut readers);
}
// After the map is updated, so a racer that wakes and
// re-enters finds the finished slot rather than racing us.
let _ = tx.send(Some(result.clone()));
return result;
}
}
}
}
/// Bound the reader map: drop least-recently-used entries until it is back
/// at [`MAX_OPEN_READERS`]. Only `Ready` slots are eviction candidates — a
/// `Pending` has callers waiting on it and no reader to drop yet.
fn trim_readers(readers: &mut HashMap<NodeUid, ReaderSlot>) {
while readers.len() > MAX_OPEN_READERS {
let victim = readers
.iter()
.filter_map(|(uid, slot)| match slot {
ReaderSlot::Ready(entry) => Some((uid, entry.last_used)),
ReaderSlot::Pending(_) => None,
})
.min_by_key(|(_, last_used)| *last_used)
.map(|(uid, _)| uid.clone());
let Some(victim) = victim else {
break; // every slot is an open in flight; nothing to drop
};
readers.remove(&victim);
}
}
/// Drop any open reader for `uid`, so the next read reopens against the
/// current revision. Called wherever cached content is evicted.
///
/// Drops an in-flight open too: its leader finds the slot gone and declines
/// to cache what it opened, which is what we want — the eviction says that
/// revision is no longer the truth.
pub(super) fn evict_reader(&self, uid: &NodeUid) {
self.readers.lock().remove(uid);
}
/// Serve bytes `[offset, offset + len)` of `uid`'s active revision, hitting
/// the on-disk caches before the network: a whole-file blob (pinned files)
/// first, then the block cache — fetching only the [`BLOCK_SIZE`]-aligned
/// blocks that overlap the request and caching each. `mtime`/`fsize` validate
/// both caches. Network I/O runs without any lock held.
pub(super) fn read_range(
&self,
uid: &NodeUid,
mtime: i64,
fsize: u64,
offset: u64,
len: u64,
cache_blocks: bool,
) -> Result<Vec<u8>, Errno> {
// A queued write has not reached the remote yet, so the remote's current
// revision is stale and the staged blob is the truth. Serve from it until
// the drain worker lands the upload (offline.md Phase 3).
if let Some(pending) = self.pending.lock().get(uid).cloned() {
return self.read_pending(&pending, offset, len);
}
// A node created offline and never written has no blob and no remote: it
// is an empty file, and asking the API about a `local~` uid would only
// earn a 404 (offline.md Phase 3b).
if is_local_uid(uid) {
return Ok(Vec::new());
}
self.read_range_remote(uid, mtime, fsize, offset, len, cache_blocks)
}
/// Serve a read from a staged blob, falling back to the remote base for any
/// range the write did not author (an incomplete [`StagedWrite`] holds zeros
/// there, which must never be handed out as content).
fn read_pending(
&self,
pending: &PendingRevision,
offset: u64,
len: u64,
) -> Result<Vec<u8>, Errno> {
let m = &pending.meta;
if offset >= m.len || len == 0 {
return Ok(Vec::new());
}
let uid = parse_node_uid(&m.uid).ok_or(Errno::EIO)?;
let file = File::open(&pending.path).map_err(|e| {
error!(%uid, path = %pending.path.display(), error = %e, "open staged blob failed");
Errno::EIO
})?;
let mut written = Intervals::default();
for &(s, e) in &m.authored {
written.add(s, e);
}
// Same merge as `serve_open_read`, but resolving gaps against the remote
// rather than through `read_range` — going through `read_range` would find
// this very pending op and recurse.
let end = offset.saturating_add(len).min(m.len);
let mut out = Vec::with_capacity((end - offset) as usize);
for (s, e, authored) in written.segments(offset, end) {
if authored {
let mut buf = vec![0u8; (e - s) as usize];
file.read_exact_at(&mut buf, s).map_err(|err| {
warn!(%uid, error = %err, "staged blob read failed");
Errno::EIO
})?;
out.extend_from_slice(&buf);
continue;
}
let bend = e.min(m.base_size);
if s < bend {
out.extend_from_slice(&self.read_range_remote(
&uid,
m.base_mtime,
m.base_size,
s,
bend - s,
true,
)?);
}
// Past the base: a hole the write extended over.
out.resize(out.len() + e.saturating_sub(s.max(m.base_size)) as usize, 0);
}
Ok(out)
}
/// Warm the [`STREAM_READAHEAD`] blocks after `last` into the ring, in the
/// background, so a player reading forward finds its next block already in
/// memory rather than paying a fetch + decrypt round-trip at each boundary.
///
/// Fire-and-forget: a failure here is not an error, it only means the read
/// that needs those bytes will fetch them itself.
fn stream_readahead(&self, uid: &NodeUid, mtime: i64, fsize: u64, last: u64) {
let last_block = (fsize.saturating_sub(1)) / BLOCK_SIZE;
let wanted: Vec<u64> = ((last + 1)..=(last + STREAM_READAHEAD).min(last_block))
.filter(|&bidx| {
self.stream_ring
.lock()
.get(uid, mtime, fsize, bidx)
.is_none()
})
.collect();
if wanted.is_empty() {
return;
}
let core = self.clone();
let uid = uid.clone();
self.rt.spawn(async move {
let Ok(reader) = core.revision_reader(&uid, mtime, fsize).await else {
return;
};
// Concurrently, for the same reason the demand path fetches its
// misses concurrently: serially awaited blocks cost the sum of their
// round-trips, and the player catches up with the readahead before it
// finishes. The SDK's in-flight block governor bounds the fan-out.
let mut set = tokio::task::JoinSet::new();
for bidx in wanted {
let reader = reader.clone();
let bstart = bidx * BLOCK_SIZE;
let blen = BLOCK_SIZE.min(fsize - bstart);
set.spawn(async move { (bidx, reader.read_at(bstart, blen).await) });
}
while let Some(joined) = set.join_next().await {
let Ok((bidx, read)) = joined else { continue };
match read {
Ok(bytes) => {
core.stream_ring
.lock()
.insert(&uid, mtime, fsize, bidx, Arc::new(bytes))
}
Err(e) => debug!(%uid, bidx, error = %e, "stream readahead failed"),
}
}
});
}
/// Read from the content cache, else the remote. The base-content path, with
/// no awareness of queued writes — callers wanting the file's *current*
/// content want [`Core::read_range`]. Gap-filling a staged blob is the one
/// caller that genuinely means "the base", since that is what its zeroed
/// ranges have to be filled from.
fn read_range_remote(
&self,
uid: &NodeUid,
mtime: i64,
fsize: u64,
offset: u64,
len: u64,
cache_blocks: bool,
) -> Result<Vec<u8>, Errno> {
if let Some(bytes) = self.cache.read_range(uid, mtime, fsize, offset, len) {
return Ok(bytes);
}
if offset >= fsize || len == 0 {
return Ok(Vec::new());
}
let end = offset.saturating_add(len).min(fsize);
let mut out = Vec::with_capacity((end - offset) as usize);
let first = offset / BLOCK_SIZE;
let last = (end - 1) / BLOCK_SIZE;
// Collect the blocks overlapping the request, serving any already cached
// and fetching the rest concurrently. A multi-block read (e.g. a media
// player buffering, or a large sequential read split into one FUSE call)
// would otherwise stall on each block round-trip in turn; downloading the
// misses in parallel saturates the connection and bounds latency at the
// slowest single block instead of their sum.
let mut blocks: Vec<Option<Arc<Vec<u8>>>> = Vec::with_capacity((last - first + 1) as usize);
let mut misses: Vec<u64> = Vec::new();
for bidx in first..=last {
// A streaming read keeps its blocks only in memory, so the ring is
// the only cache it has — check it before the disk (which will not
// hold them) and before the network (which would refetch the same
// 4 MiB block for every 128 KiB the player consumes).
let hit = if cache_blocks {
self.cache
.cached_block(uid, mtime, fsize, bidx)
.map(Arc::new)
} else {
self.stream_ring.lock().get(uid, mtime, fsize, bidx)
};
match hit {
Some(b) => blocks.push(Some(b)),
None => {
blocks.push(None);
misses.push(bidx);
}
}
}
if !misses.is_empty() {
let fetched = self.rt.block_on(async {
// Resolve the file's keys and block table once, then read every
// missing block through the shared reader. Previously each block
// called `download_range`, which redid that resolution per block.
let reader = self.revision_reader(uid, mtime, fsize).await?;
let mut set = tokio::task::JoinSet::new();
for &bidx in &misses {
let reader = reader.clone();
let uid = uid.clone();
let bstart = bidx * BLOCK_SIZE;
let blen = BLOCK_SIZE.min(fsize - bstart);
set.spawn(async move {
reader
.read_at(bstart, blen)
.await
.map(|bytes| (bidx, bytes))
.map_err(|e| {
warn!(%uid, bstart, blen, error = %e, "block read failed");
Errno::EIO
})
});
}
let mut out = Vec::with_capacity(misses.len());
while let Some(joined) = set.join_next().await {
// A join error means the task panicked; surface it as EIO.
out.push(joined.map_err(|_| Errno::EIO)??);
}
Ok::<_, Errno>(out)
})?;
for (bidx, bytes) in fetched {
// A streaming read (large unpinned video) skips the on-disk cache
// so it can't evict the rest of it; it keeps the block in the
// in-memory ring instead, which is bounded and dies with the read.
let bytes = Arc::new(bytes);
if cache_blocks {
let _ = self.cache.store_block(uid, mtime, fsize, bidx, &bytes);
} else {
self.stream_ring
.lock()
.insert(uid, mtime, fsize, bidx, bytes.clone());
}
blocks[(bidx - first) as usize] = Some(bytes);
}
if !cache_blocks {
self.stream_readahead(uid, mtime, fsize, last);
}
}
for (i, block) in blocks.into_iter().enumerate() {
let bidx = first + i as u64;
let bstart = bidx * BLOCK_SIZE;
// Every slot is populated: cache hits up front, misses by the fetch above.
let block = block.expect("block fetched or cached");
let s = (offset.max(bstart) - bstart) as usize;
let e = (end.min(bstart + block.len() as u64) - bstart) as usize;
if s < e {
out.extend_from_slice(&block[s..e]);
}
}
Ok(out)
}
/// Serve a read against an open write handle: stitch authored ranges (from
/// the scratch file) and untouched ranges (from the remote base via the
/// block cache) in order, zero-filling any region past the base.
#[allow(clippy::too_many_arguments)]
pub(super) fn serve_open_read(
&self,
file: &Arc<File>,
len: u64,
uid: &NodeUid,
base_mtime: i64,
base_size: u64,
written: &Intervals,
offset: u64,
size: u64,
) -> Result<Vec<u8>, Errno> {
if offset >= len || size == 0 {
return Ok(Vec::new());
}
let end = offset.saturating_add(size).min(len);
let mut out = Vec::with_capacity((end - offset) as usize);
for (s, e, authored) in written.segments(offset, end) {
if authored {
let mut buf = vec![0u8; (e - s) as usize];
file.read_exact_at(&mut buf, s).map_err(|err| {
warn!(%uid, error = %err, "scratch read failed");
Errno::EIO
})?;
out.extend_from_slice(&buf);
} else {
let bend = e.min(base_size);
if s < bend {
out.extend_from_slice(&self.read_range(
uid,
base_mtime,
base_size,
s,
bend - s,
true,
)?);
}
// Anything past the base is a hole: zero-fill.
let zeros = e.saturating_sub(s.max(base_size));
out.resize(out.len() + zeros as usize, 0);
}
}
Ok(out)
}
/// Fill every unauthored range of a scratch/staged file that overlaps its
/// base with the base's bytes, so the file becomes the complete new content.
///
/// This is the step a partial overwrite cannot skip: only the authored bytes
/// were ever written to disk, and a revision upload sends the whole file.
/// The gaps come from the *remote base* (through the block cache, so a small
/// edit of a large file does not pull all of it), which is exactly why this
/// can fail with no network — see `StagedWrite`.
pub(super) fn fill_gaps(
&self,
uid: &NodeUid,
file: &File,
len: u64,
base_mtime: i64,
base_size: u64,
written: &Intervals,
) -> Result<(), Errno> {
file.set_len(len).map_err(|e| {
error!(%uid, error = %e, "resize scratch file failed");
Errno::EIO
})?;
for (s, e, authored) in written.segments(0, len) {
if authored {
continue;
}
let bend = e.min(base_size);
if s >= bend {
continue; // wholly past the base: already zero-filled on disk
}
let bytes = self.read_range_remote(uid, base_mtime, base_size, s, bend - s, true)?;
file.write_all_at(&bytes, s).map_err(|err| {
error!(%uid, error = %err, "scratch gap-fill write failed");
Errno::EIO
})?;
}
Ok(())
}
}
+97 -25
View File
@@ -15,7 +15,34 @@ use pdfs_core::{CoreError, CoreResult};
use proton_drive_rs::proton_sdk::ids::NodeUid;
use proton_drive_rs::{MemberRole, NodeKind};
use super::{Core, ROOT_INO, parse_uid, public_link_info, role_from_str, role_to_str};
use proton_drive_rs::Node;
use super::{Core, ROOT_INO, node_size, parse_uid, public_link_info, role_from_str, role_to_str};
/// A node from a *shared* listing as a [`DirEntry`]. Shared nodes live outside
/// the mount, so the entry is uid-addressed: `path` is empty, and the local-state
/// flags (`pinned`, `cached`) are false — they describe my own tree, not this one.
fn shared_entry(n: Node) -> DirEntry {
let is_dir = n.is_folder();
let size = match &n.kind {
NodeKind::File {
claimed_size,
total_size_on_storage,
..
} => claimed_size.unwrap_or(*total_size_on_storage).max(0) as u64,
NodeKind::Folder => 0,
};
DirEntry {
name: n.name,
is_dir,
size,
modified: n.modification_time,
pinned: false,
cached: false,
uid: n.uid.to_string(),
path: String::new(),
}
}
impl Core {
// ---- sharing a node ---------------------------------------------------
@@ -232,30 +259,62 @@ impl Core {
.rt
.block_on(self.client.enumerate_nodes(&uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes
.into_iter()
.map(|n| {
let is_dir = n.is_folder();
let size = match &n.kind {
NodeKind::File {
claimed_size,
total_size_on_storage,
..
} => claimed_size.unwrap_or(*total_size_on_storage).max(0) as u64,
NodeKind::Folder => 0,
};
DirEntry {
name: n.name,
is_dir,
size,
modified: 0,
pinned: false,
cached: false,
uid: n.uid.to_string(),
path: String::new(),
}
})
.collect())
Ok(nodes.into_iter().map(shared_entry).collect())
}
/// List the children of a folder shared with me, addressed by uid.
///
/// A shared subtree has no path in this account's mount — the FUSE tree only
/// spans my own volume — so browsing into one is uid-addressed the whole way
/// down, each listing handing the front-end the uids it descends with next.
pub(crate) fn list_shared_folder(&self, uid: &str) -> CoreResult<Vec<DirEntry>> {
let uid =
parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?;
let child_uids = self
.rt
.block_on(self.client.enumerate_folder_children_node_uids(&uid))
.map_err(|e| CoreError::from_api(&e, "enumerate shared children"))?;
if child_uids.is_empty() {
return Ok(Vec::new());
}
let nodes = self
.rt
.block_on(self.client.enumerate_nodes(&child_uids))
.map_err(|e| CoreError::from_api(&e, "enumerate nodes"))?;
Ok(nodes.into_iter().map(shared_entry).collect())
}
/// Download a file shared with me into the content cache, returning its
/// on-disk path (served from cache when a fresh blob already exists).
///
/// The by-uid twin of [`Core::open_file`]: same cache and same tracked
/// download, but the node is fetched from the server rather than resolved
/// through the inode tree, which does not cover other people's volumes.
///
/// [`Core::open_file`]: super::Core::open_file
pub(crate) fn open_shared_file(&self, uid: &str) -> CoreResult<std::path::PathBuf> {
let uid =
parse_uid(uid).ok_or_else(|| CoreError::invalid(format!("invalid uid: {uid}")))?;
let node = self
.rt
.block_on(self.client.get_node(&uid))
.map_err(|e| CoreError::from_api(&e, "get shared node"))?
.ok_or_else(|| CoreError::not_found("shared node not found"))?;
if !node.is_file() {
return Err(CoreError::invalid("not a regular file"));
}
let size = node_size(&node);
let mtime = node.modification_time;
if let Some(p) = self.cache.cached_content_path(&uid, mtime, size) {
return Ok(p);
}
let bytes = self
.download_file_tracked(&uid, &node.name, size)
.map_err(|e| CoreError::from_api(&e, "download"))?;
self.cache
.store(&uid, mtime, size, &bytes)
.map_err(|e| CoreError::internal(format!("cache store: {e}")))?;
Ok(self.cache.content_path(&uid))
}
/// Leave a shared node by its uid.
@@ -333,6 +392,19 @@ impl Core {
.map_err(|e| CoreError::from_api(&e, "delete bookmark"))
}
// ---- account ----------------------------------------------------------
/// Total account storage usage `(max_space, used_space)` in bytes, across all
/// Proton products (not Drive-only). A remote round-trip; nothing here is
/// cached.
pub(crate) fn account_quota(&self) -> CoreResult<(i64, i64)> {
let q = self
.rt
.block_on(self.client.quota())
.map_err(|e| CoreError::from_api(&e, "account quota"))?;
Ok((q.max_space, q.used_space))
}
// ---- shared by me -----------------------------------------------------
/// List the nodes I have shared with others, each with a summary of its share
+119 -2
View File
@@ -18,6 +18,9 @@ pub(crate) struct Entry {
pub(crate) uid: NodeUid,
pub(crate) parent: u64,
pub(crate) node: Node,
pub(crate) lookup_count: u64,
pub(crate) open_count: u32,
pub(crate) unlinked: bool,
}
/// A set of non-overlapping `[start, end)` byte ranges, kept sorted and merged.
@@ -105,6 +108,8 @@ pub(crate) struct WriteHandle {
pub(crate) base_mtime: i64,
/// Whether anything diverged from the remote and needs an upload.
pub(crate) dirty: bool,
/// Number of file handles currently sharing this scratch state.
pub(crate) open_count: usize,
}
/// A released write whose upload has not happened yet (offline.md Phase 3).
@@ -131,9 +136,11 @@ pub(crate) struct State {
/// key means the directory has been enumerated.
pub(crate) children: HashMap<u64, Vec<u64>>,
pub(crate) next_ino: u64,
/// Open write handles keyed by file handle id. Read-only opens use fh 0 and
/// Shared write state keyed by inode. Concurrent writers share the scratch file.
pub(crate) active_writes: HashMap<u64, WriteHandle>,
/// Maps file handle id (fh) to inode (ino). Read-only opens use fh 0 and
/// have no entry here.
pub(crate) handles: HashMap<u64, WriteHandle>,
pub(crate) handles: HashMap<u64, u64>,
pub(crate) next_fh: u64,
/// Unified SQLite metadata cache. Every map mutation below writes through to
/// it inside the `State` lock so the DB stays the authoritative copy across
@@ -169,11 +176,26 @@ impl State {
uid: node.uid.clone(),
parent,
node,
lookup_count: 1,
open_count: 0,
unlinked: false,
},
);
ino
}
/// Decrement lookup count for an inode and prune if lookup_count == 0 && open_count == 0 && unlinked.
pub(crate) fn forget_lookup(&mut self, ino: u64, nlookup: u64) -> Option<(u64, String)> {
if let Some(entry) = self.entries.get_mut(&ino) {
entry.lookup_count = entry.lookup_count.saturating_sub(nlookup);
if entry.lookup_count == 0 && entry.open_count == 0 && entry.unlinked {
let uid = entry.uid.clone();
return self.forget(&uid);
}
}
None
}
/// Allocate (or reuse) a stable inode for a node that came *from* the
/// database, which is why nothing is written back.
pub(crate) fn intern_from_db(&mut self, parent: u64, node: Node) -> u64 {
@@ -195,6 +217,57 @@ impl State {
.collect()
}
/// Check if a directory inode has any child nodes in memory or in the database.
pub(crate) fn has_children(&self, parent: u64) -> bool {
if let Some(kids) = self.children.get(&parent)
&& !kids.is_empty()
{
return true;
}
if let Some(entry) = self.entries.get(&parent)
&& let Ok(has_kids) = self.db.has_children(&entry.uid)
{
return has_kids;
}
false
}
/// Check if `ancestor_ino` is `target_ino` or an ancestor of `target_ino` (for rename cycle prevention).
pub(crate) fn is_ancestor_of(&self, ancestor_ino: u64, mut target_ino: u64) -> bool {
if ancestor_ino == target_ino {
return true;
}
let mut visited = std::collections::HashSet::new();
visited.insert(target_ino);
while let Some(entry) = self.entries.get(&target_ino) {
let parent = entry.parent;
if parent == ancestor_ino {
return true;
}
if parent == 0 || !visited.insert(parent) {
break;
}
target_ino = parent;
}
false
}
/// Forget a node or, if open handles exist (open_count > 0), mark it unlinked
/// and remove it from parent children so lookups fail while open reads succeed.
pub(crate) fn forget_or_unlink(&mut self, uid: &NodeUid) -> Option<(u64, String)> {
if let Some(&ino) = self.by_uid.get(uid)
&& let Some(entry) = self.entries.get_mut(&ino)
&& entry.open_count > 0
{
entry.unlinked = true;
if let Some(kids) = self.children.get_mut(&entry.parent) {
kids.retain(|&k| k != ino);
}
return Some((entry.parent, entry.node.name.clone()));
}
self.forget(uid)
}
/// Forget a node entirely: drop its inode, its uid mapping, its own cached
/// listing, its slot in its parent's listing, and its DB row. Returns
/// `(parent_ino, name)` when the node was known, so the caller can notify
@@ -410,6 +483,7 @@ mod tests {
by_uid: HashMap::new(),
children: HashMap::new(),
next_ino: 1,
active_writes: HashMap::new(),
handles: HashMap::new(),
next_fh: 1,
db: Arc::new(db),
@@ -492,4 +566,47 @@ mod tests {
with no way to notice anything else changed"
);
}
#[test]
fn test_is_ancestor_of_hierarchy() {
let (mut st, _dir) = state();
let root = st.intern(0, node("root_id", "none", "root", true));
let p1 = st.intern(root, node("p1_id", "root_id", "p1", true));
let p2 = st.intern(p1, node("p2_id", "p1_id", "p2", true));
let child = st.intern(p2, node("child_id", "p2_id", "child", true));
assert!(st.is_ancestor_of(root, root), "self is ancestor");
assert!(st.is_ancestor_of(root, p1), "root is ancestor of p1");
assert!(
st.is_ancestor_of(root, child),
"root is ancestor of deep child"
);
assert!(st.is_ancestor_of(p1, child), "p1 is ancestor of deep child");
assert!(
!st.is_ancestor_of(child, root),
"child is not ancestor of root"
);
assert!(!st.is_ancestor_of(child, p1), "child is not ancestor of p1");
}
#[test]
fn test_state_has_children_mem_and_db() {
let (mut st, _dir) = state();
let folder = st.intern_mem(0, node("dir_uid", "none", "dir", true));
assert!(!st.has_children(folder), "initially empty");
// Memory-only children (using intern_mem so DB is not populated yet)
let f1 = st.intern_mem(folder, node("file1_uid", "dir_uid", "file1.txt", false));
st.children.insert(folder, vec![f1]);
assert!(st.has_children(folder), "has child in memory");
st.children.insert(folder, vec![]);
assert!(!st.has_children(folder), "cleared memory children");
// DB children
st.db
.upsert_node(&node("file2_uid", "dir_uid", "file2.txt", false))
.unwrap();
assert!(st.has_children(folder), "has child in db");
}
}
File diff suppressed because it is too large. Load diff
+251
View File
@@ -0,0 +1,251 @@
//! Filesystem watcher, polling, and debounce control plane.
use super::*;
/// How long the watcher must go quiet before a burst counts as finished.
///
/// Measured from the *last* event, not the first: applications do not write a
/// file once. An editor saves by writing a temp file and renaming it over the
/// target; an export or a database dump writes continuously for as long as it
/// takes. Settling a fixed interval after the first event walks the tree while
/// the file is still growing, and uploads a torn snapshot as a real revision —
/// corrected on the next pass, but only after spending the encrypt-and-upload
/// on a file that was never in that state.
const DEBOUNCE: Duration = Duration::from_secs(2);
/// Ceiling on the settle wait, so a directory under sustained change still
/// syncs. Without it, copying a large tree in — events arriving forever —
/// would postpone the reconcile for as long as the copy ran.
const MAX_SETTLE: Duration = Duration::from_secs(30);
/// How often to re-walk remotes so changes made on other clients are pulled in.
const POLL_INTERVAL: Duration = Duration::from_secs(120);
/// How many uploads/downloads/folder-creations a reconcile runs at once. The
/// per-item work is a network round-trip, so a folder with thousands of files is
/// unusable done one-at-a-time; this bounds concurrency like the bulk-upload
/// engine does.
pub(super) const SYNC_CONCURRENCY: usize = 8;
/// A message to the sync engine's control thread.
pub(crate) enum SyncMsg {
/// Reconcile one folder by id (a local change, or a targeted `SyncNow`).
Reconcile(i64),
/// Reconcile every mirror folder (periodic poll, or startup).
ReconcileAll,
/// Re-read the folder set and adjust the filesystem watches (after add/remove).
Rewatch,
}
/// Start the sync engine: a control thread that owns the filesystem watcher and
/// serialises reconcile passes, plus a periodic poll thread. The engine receives
/// on `rx`; senders live in [`Core::sync_tx`].
pub(crate) fn spawn(core: Core, rx: Receiver<SyncMsg>) {
if let Err(e) = std::thread::Builder::new()
.name("pdfs-sync".into())
.spawn(move || engine_loop(core, rx))
{
warn!(error = %e, "failed to start sync engine");
}
}
/// The engine control loop. Single-threaded so reconcile passes never overlap.
fn engine_loop(core: Core, rx: Receiver<SyncMsg>) {
// Paths the watcher currently covers, mapped to their folder id, so an event
// path can be resolved back to the folder it belongs to.
let watched: Mutex<Vec<(PathBuf, i64)>> = Mutex::new(Vec::new());
let tx_events = core.sync_tx.clone();
let mut watcher =
match notify::recommended_watcher(move |res: notify::Result<notify::Event>| {
let Ok(event) = res else { return };
// Ignore pure metadata/access noise; content changes are what matter.
if matches!(event.kind, notify::EventKind::Access(_)) {
return;
}
let _ = tx_events.send(SyncMsg::ReconcileAll);
}) {
Ok(w) => w,
Err(e) => {
warn!(error = %e, "filesystem watcher unavailable; sync is poll-only");
// Still run the loop so polling and explicit SyncNow work.
poll_only_loop(&core, rx);
return;
}
};
// Periodic remote poll.
{
let tx = core.sync_tx.clone();
std::thread::Builder::new()
.name("pdfs-sync-poll".into())
.spawn(move || {
loop {
std::thread::sleep(POLL_INTERVAL);
if tx.send(SyncMsg::ReconcileAll).is_err() {
break;
}
}
})
.ok();
}
rewatch(&core, &mut watcher, &watched);
reconcile_all(&core);
while let Ok(msg) = rx.recv() {
let mut ids: HashSet<i64> = HashSet::new();
let mut all = false;
let mut do_rewatch = false;
classify(msg, &mut ids, &mut all, &mut do_rewatch);
settle(&rx, &mut ids, &mut all, &mut do_rewatch);
if do_rewatch {
rewatch(&core, &mut watcher, &watched);
}
if all {
reconcile_all(&core);
} else {
for id in ids {
if let Ok(Some(folder)) = core.db.sync_folder_get(id) {
core.reconcile_folder(&folder);
}
}
}
}
}
/// Fallback loop when no filesystem watcher could be created: poll + SyncNow only.
fn poll_only_loop(core: &Core, rx: Receiver<SyncMsg>) {
{
let tx = core.sync_tx.clone();
std::thread::spawn(move || {
loop {
std::thread::sleep(POLL_INTERVAL);
if tx.send(SyncMsg::ReconcileAll).is_err() {
break;
}
}
});
}
reconcile_all(core);
while let Ok(msg) = rx.recv() {
let mut ids: HashSet<i64> = HashSet::new();
let mut all = false;
let mut do_rewatch = false;
classify(msg, &mut ids, &mut all, &mut do_rewatch);
settle(&rx, &mut ids, &mut all, &mut do_rewatch);
if all {
reconcile_all(core);
} else {
for id in ids {
if let Ok(Some(folder)) = core.db.sync_folder_get(id) {
core.reconcile_folder(&folder);
}
}
}
}
}
/// Absorb the rest of an event burst, returning once the watcher has been quiet
/// for `quiet` or `cap` has elapsed since the first event — whichever comes
/// first. The caller has already classified the event that opened the burst.
///
/// Split out with its timings as parameters so the settling behaviour can be
/// tested in milliseconds rather than in the tens of seconds the real constants
/// describe.
pub(super) fn settle_with(
rx: &Receiver<SyncMsg>,
ids: &mut HashSet<i64>,
all: &mut bool,
do_rewatch: &mut bool,
quiet: Duration,
cap: Duration,
) {
let deadline = Instant::now() + cap;
loop {
let now = Instant::now();
if now >= deadline {
break;
}
// Never wait past the cap, however quiet the watcher goes.
let wait = quiet.min(deadline - now);
match rx.recv_timeout(wait) {
// Another event: the burst is still going, so the quiet window
// starts again from here.
Ok(m) => classify(m, ids, all, do_rewatch),
// Quiet for a full window: whatever was being written has stopped.
Err(RecvTimeoutError::Timeout) => break,
// Every sender is gone; the caller's loop is about to end anyway.
Err(RecvTimeoutError::Disconnected) => break,
}
}
}
/// [`settle_with`] at the real timings.
fn settle(rx: &Receiver<SyncMsg>, ids: &mut HashSet<i64>, all: &mut bool, do_rewatch: &mut bool) {
settle_with(rx, ids, all, do_rewatch, DEBOUNCE, MAX_SETTLE);
}
pub(super) fn classify(msg: SyncMsg, ids: &mut HashSet<i64>, all: &mut bool, rewatch: &mut bool) {
match msg {
SyncMsg::Reconcile(id) => {
ids.insert(id);
}
SyncMsg::ReconcileAll => *all = true,
SyncMsg::Rewatch => *rewatch = true,
}
}
/// Reconcile every mirror folder in turn.
fn reconcile_all(core: &Core) {
let folders = match core.db.sync_folder_list() {
Ok(f) => f,
Err(e) => {
warn!(error = ?e, "sync: cannot list folders");
return;
}
};
for folder in folders {
core.reconcile_folder(&folder);
}
}
/// Bring the filesystem watches in line with the current mirror-folder set:
/// watch newly-added folders, drop removed ones.
fn rewatch(
core: &Core,
watcher: &mut notify::RecommendedWatcher,
watched: &Mutex<Vec<(PathBuf, i64)>>,
) {
let want: Vec<(PathBuf, i64)> = match core.db.sync_folder_list() {
Ok(f) => f
.into_iter()
.filter(|f| f.mode == "mirror")
.map(|f| (PathBuf::from(f.local_path), f.id))
.collect(),
Err(e) => {
warn!(error = ?e, "sync: cannot list folders to watch");
return;
}
};
let mut have = watched.lock();
// Drop watches no longer wanted.
have.retain(|(path, _)| {
if want.iter().any(|(p, _)| p == path) {
true
} else {
let _ = watcher.unwatch(path);
false
}
});
// Add newly wanted watches.
for (path, id) in &want {
if have.iter().any(|(p, _)| p == path) {
continue;
}
match watcher.watch(path, RecursiveMode::Recursive) {
Ok(()) => have.push((path.clone(), *id)),
Err(e) => warn!(path = %path.display(), error = %e, "sync: watch failed"),
}
}
}
// ---- reconcile ------------------------------------------------------------
+81
View File
@@ -0,0 +1,81 @@
//! Local open-for-write detection used to defer torn uploads.
use super::*;
// ---- open-for-write detection ---------------------------------------------
/// Scan `/proc/*/fd` once and return the set of **canonical** paths under
/// `root` that any process currently holds open for writing (`O_WRONLY` or
/// `O_RDWR`). The scan is best-effort: unreadable `/proc/*/fd` directories
/// (other users' processes, or kernel threads) are silently skipped.
///
/// Cost: one `readdir` of `/proc`, then for every process one `readdir` of its
/// `/proc/<pid>/fd/`, and one `readlink` + one `open` of `/proc/<pid>/fdinfo/<n>`
/// per open fd that resolves under `root`. For a typical desktop with a few
/// hundred processes and a handful of files inside the sync root, this
/// completes in low single-digit milliseconds.
pub(super) fn open_for_write_set(root: &Path) -> HashSet<PathBuf> {
let mut result = HashSet::new();
let Ok(root_canonical) = std::fs::canonicalize(root) else {
return result;
};
let Ok(proc_entries) = std::fs::read_dir("/proc") else {
return result;
};
for proc_entry in proc_entries.filter_map(|e| e.ok()) {
let pid_name = proc_entry.file_name();
// Skip non-numeric entries (kernel threads, /proc/self, etc.)
if !pid_name
.to_str()
.is_some_and(|s| s.bytes().all(|b| b.is_ascii_digit()))
{
continue;
}
let fd_dir = PathBuf::from("/proc").join(&pid_name).join("fd");
let Ok(fds) = std::fs::read_dir(&fd_dir) else {
continue;
};
for fd_entry in fds.filter_map(|e| e.ok()) {
// readlink on /proc/<pid>/fd/<n> gives the target path.
let link = fd_dir.join(fd_entry.file_name());
let Ok(target) = std::fs::read_link(&link) else {
continue;
};
// Quick prefix check before the more expensive fdinfo read.
if !target.starts_with(&root_canonical) {
continue;
}
// Read the fd's flags from /proc/<pid>/fdinfo/<n> to determine
// the access mode. The first line is "pos:\t<offset>", the second
// is "flags:\t<octal>". We only need the low two bits of flags:
// 0 = O_RDONLY, 1 = O_WRONLY, 2 = O_RDWR
let fdinfo_path = PathBuf::from("/proc")
.join(&pid_name)
.join("fdinfo")
.join(fd_entry.file_name());
if let Ok(contents) = std::fs::read_to_string(&fdinfo_path)
&& is_write_mode(&contents)
{
result.insert(target);
}
}
}
result
}
/// Parse the `flags:` line out of a `/proc/<pid>/fdinfo/<n>` file and return
/// true if the low two bits indicate write access (`O_WRONLY = 1` or
/// `O_RDWR = 2`).
pub(super) fn is_write_mode(fdinfo: &str) -> bool {
for line in fdinfo.lines() {
if let Some(rest) = line.strip_prefix("flags:\t") {
if let Ok(flags) = u32::from_str_radix(rest.trim_start_matches('0'), 8) {
let access = flags & 0o3; // O_ACCMODE
return access == 1 || access == 2; // O_WRONLY or O_RDWR
}
// An empty octal string ("0" stripped to "") means flags == 0 == O_RDONLY.
return false;
}
}
false
}
+164
View File
@@ -0,0 +1,164 @@
//! Reconciliation snapshots, queued operations, and pass outcomes.
use super::*;
pub(super) struct LocalItem {
pub(super) is_dir: bool,
pub(super) mtime: i64,
pub(super) size: i64,
/// True when another process holds this file open for writing (detected via
/// `/proc/*/fd`). An open-for-write file is kept in the map — so it is not
/// misread as a deletion — but treated as unchanged for upload purposes,
/// deferring the upload until the writer closes the file.
pub(super) open_for_write: bool,
}
/// One item found while walking a remote tree.
pub(super) struct RemoteItem {
pub(super) uid: NodeUid,
pub(super) is_dir: bool,
pub(super) mtime: i64,
pub(super) size: i64,
}
/// The result of a reconcile pass: what it moved, how many paths were kept as
/// conflict copies, and how many failed to apply (and so still need another
/// pass). The counts drive both the folder's state and its activity summary.
#[derive(Default)]
pub(super) struct Outcome {
pub(super) uploaded: usize,
pub(super) downloaded: usize,
pub(super) created: usize,
pub(super) deleted: usize,
pub(super) conflicts: usize,
pub(super) errors: usize,
/// Files skipped because another process held them open for writing.
pub(super) deferred: usize,
}
impl Outcome {
/// Fold in one applied op.
pub(super) fn record(&mut self, applied: &Applied) {
match applied {
Applied::Dir(..) => self.created += 1,
Applied::Uploaded => self.uploaded += 1,
Applied::Downloaded => self.downloaded += 1,
Applied::Conflict => self.conflicts += 1,
}
}
/// Whether the pass moved nothing at all — the common case on a poll of an
/// unchanged folder, which should not add a line to the activity feed.
pub(super) fn is_empty(&self) -> bool {
self.uploaded == 0
&& self.downloaded == 0
&& self.created == 0
&& self.deleted == 0
&& self.conflicts == 0
&& self.errors == 0
&& self.deferred == 0
}
/// A human summary of the pass: "3 uploaded, 1 downloaded, 2 failed".
pub(super) fn summary(&self) -> String {
let mut parts = Vec::new();
for (n, label) in [
(self.uploaded, "uploaded"),
(self.downloaded, "downloaded"),
(self.created, "folder(s) created"),
(self.deleted, "deleted"),
(self.conflicts, "conflicted"),
(self.deferred, "deferred (open for write)"),
(self.errors, "failed"),
] {
if n > 0 {
parts.push(format!("{n} {label}"));
}
}
parts.join(", ")
}
}
/// A network operation queued during classification and run concurrently in a
/// per-depth batch. Parent uids are resolved up front (the parent folder is one
/// depth shallower and already created), so tasks share nothing mutable.
pub(super) enum Pending {
/// Create a new remote folder under `parent`.
CreateDir { rel: String, parent: NodeUid },
/// Upload a brand-new local file into `parent`.
UploadNew { rel: String, parent: NodeUid },
/// Upload a changed local file as a new revision of `uid`.
UploadRevision { rel: String, uid: NodeUid },
/// Download remote `uid` to the local path, stamping `mtime`. `size` is the
/// remote's reported size, used as the transfer's expected total.
Download {
rel: String,
uid: NodeUid,
mtime: i64,
size: i64,
},
/// Both sides changed: set the local copy aside, then download remote `uid`.
Conflict {
rel: String,
uid: NodeUid,
mtime: i64,
size: i64,
},
/// Both sides changed during a [`push pass`](Core::push_pass): upload the local
/// copy into `parent` under a conflict name, leaving the remote file as it is.
PushConflict { rel: String, parent: NodeUid },
}
impl Pending {
/// The path this op acts on, relative to the folder root.
pub(super) fn rel(&self) -> &str {
match self {
Pending::CreateDir { rel, .. }
| Pending::UploadNew { rel, .. }
| Pending::UploadRevision { rel, .. }
| Pending::Download { rel, .. }
| Pending::Conflict { rel, .. }
| Pending::PushConflict { rel, .. } => rel,
}
}
/// How this op reads in the activity feed.
pub(super) fn kind(&self) -> ActivityKind {
match self {
Pending::CreateDir { .. } => ActivityKind::CreateFolder,
Pending::UploadNew { .. }
| Pending::UploadRevision { .. }
| Pending::PushConflict { .. } => ActivityKind::Upload,
Pending::Download { .. } | Pending::Conflict { .. } => ActivityKind::Download,
}
}
/// The activity line's detail, which distinguishes ops that share a kind.
pub(super) fn detail(&self) -> &'static str {
match self {
Pending::CreateDir { .. } => "on Drive",
Pending::UploadNew { .. } => "new file",
Pending::UploadRevision { .. } => "new version",
Pending::Download { .. } => "from Drive",
Pending::Conflict { .. } => "local changes kept as a conflict copy",
Pending::PushConflict { .. } => "local changes uploaded as a conflict copy",
}
}
}
/// Why a full reconcile pass stopped without a result.
pub(super) enum PassAbort {
/// An on-demand switch was queued while the pass was running. Everything left
/// to do is work on a local copy about to be evicted, so the pass gives up its
/// remaining work to a [`push pass`](Core::push_pass) instead. Not a failure —
/// nothing is left half-applied, because each step is applied whole.
Interrupted,
/// The pass could not establish its diff (a walk or the baseline load failed).
Failed(String),
}
impl From<String> for PassAbort {
fn from(e: String) -> Self {
PassAbort::Failed(e)
}
}
+353
View File
@@ -0,0 +1,353 @@
//! Pure reconciliation ordering, safety guards, signatures, and path rules.
use super::*;
/// The paths a reconcile pass will classify: the union of the local, remote,
/// and baseline states, minus anything the ignore rules exclude, shallowest
/// first so a parent folder is created before its children are placed in it.
///
/// The ignore filter belongs *here*, on the union, rather than on the local walk
/// alone — filtering the walk alone would be actively destructive. A file synced
/// before it became ignored is absent from `local` but present in `remote` and
/// `baseline`, which is precisely the "local deleted, remote untouched" shape
/// `do_reconcile` responds to by trashing the remote copy. Adding a line to
/// `.pdfsignore` must never delete anything from Drive.
///
/// Ignored baseline rows are deliberately left in the database rather than
/// removed: with the row intact, un-ignoring the path later re-adopts the
/// existing remote file instead of reading as a brand-new local one.
pub(super) fn classification_order<L, R, B>(
local: &HashMap<String, L>,
remote: &HashMap<String, R>,
baseline: &HashMap<String, B>,
rules: &IgnoreRules,
) -> Vec<String>
where
L: HasKind,
R: HasKind,
{
let mut paths: HashSet<String> = HashSet::new();
paths.extend(local.keys().cloned());
paths.extend(remote.keys().cloned());
paths.extend(baseline.keys().cloned());
let mut order: Vec<String> = paths.into_iter().collect();
if !rules.is_empty() {
order.retain(|rel| {
// A baseline-only path has no kind recorded, so it is tested as
// both; see `filter_baseline` for why erring towards "ignored" is
// the safe direction here.
let kind = local
.get(rel)
.map(HasKind::is_dir)
.or_else(|| remote.get(rel).map(HasKind::is_dir));
match kind {
Some(is_dir) => !rules.is_ignored(rel, is_dir),
None => !rules.is_ignored(rel, false) && !rules.is_ignored(rel, true),
}
});
}
order.sort_by_key(|p| p.matches('/').count());
order
}
/// Lets [`classification_order`] ask either walk's item whether it is a
/// directory without knowing which walk it came from.
pub(super) trait HasKind {
fn is_dir(&self) -> bool;
}
impl HasKind for LocalItem {
fn is_dir(&self) -> bool {
self.is_dir
}
}
impl HasKind for RemoteItem {
fn is_dir(&self) -> bool {
self.is_dir
}
}
/// The baseline minus its ignored paths, for [`guard_local_wipe`].
///
/// The guard asks "did every synced path vanish locally?", and an ignored path
/// is absent from the local walk by rule rather than by loss. Left in, a rule
/// covering the whole tree would trip the guard on every pass and wedge that
/// folder's sync for good.
///
/// A baseline row does not record whether it was a file or a folder, so a path
/// counts as ignored if it matches as either. Erring towards "ignored" only ever
/// shrinks the set the guard checks, which weakens a safety net rather than
/// causing a deletion — the wrong direction to be wrong in is the other one.
pub(super) fn filter_baseline<'a, B>(
baseline: &'a HashMap<String, B>,
rules: &IgnoreRules,
) -> HashMap<String, &'a B> {
baseline
.iter()
.filter(|(rel, _)| !rules.is_ignored(rel, false) && !rules.is_ignored(rel, true))
.map(|(rel, entry)| (rel.clone(), entry))
.collect()
}
/// Refuse to run a pass whose local side has vanished in its entirety.
///
/// When every baseline path is absent locally, the likely cause is an unavailable
/// mount or unreadable folder rather than a deliberate whole-tree deletion.
/// One surviving path is enough to clear the guard.
pub(super) fn guard_local_wipe<B, L>(
baseline: &HashMap<String, B>,
local: &HashMap<String, L>,
) -> Result<(), String> {
if baseline.len() >= 2 && baseline.keys().all(|rel| !local.contains_key(rel)) {
return Err(format!(
"every one of the {} synced paths is missing locally; refusing to trash \
them on Drive. Check that the folder is mounted and readable.",
baseline.len()
));
}
Ok(())
}
/// The size the baseline recorded for `rel`, if its remote signature's mtime
/// still matches `mtime`.
pub(super) fn unchanged_remote_size(
baseline: &HashMap<String, StoredSyncEntry>,
rel: &str,
mtime: i64,
) -> Option<i64> {
match baseline.get(rel).and_then(remote_sig) {
Some((recorded, size)) if recorded == mtime => Some(size),
_ => None,
}
}
/// Join a child `name` onto a walk's `prefix`, giving a rel path.
pub(super) fn join_rel(prefix: &str, name: &str) -> String {
if prefix.is_empty() {
name.to_string()
} else {
format!("{prefix}/{name}")
}
}
/// The stored remote signature of a baseline row, if it has one.
pub(super) fn remote_sig(e: &StoredSyncEntry) -> Option<(i64, i64)> {
match (&e.remote_rev, &e.remote_hash) {
(Some(m), Some(s)) => Some((m.parse().ok()?, s.parse().ok()?)),
_ => None,
}
}
/// The parent of a `/`-joined relative path (`""` for a top-level entry).
pub(super) fn parent_rel(rel: &str) -> &str {
match rel.rfind('/') {
Some(i) => &rel[..i],
None => "",
}
}
/// The final component of a `/`-joined relative path.
pub(crate) fn base_name(rel: &str) -> &str {
match rel.rfind('/') {
Some(i) => &rel[i + 1..],
None => rel,
}
}
/// Turn a `/`-joined relative path into an OS path (`/` is already the separator
/// on Linux, this keeps the intent explicit).
pub(super) fn rel_to_path(rel: &str) -> PathBuf {
rel.split('/').collect()
}
/// The name for a conflict copy of `path`, e.g. `notes (sync-conflict 1700000000).txt`.
pub(super) fn conflict_path(path: &Path, stamp: i64) -> PathBuf {
let stem = path.file_stem().and_then(|s| s.to_str()).unwrap_or("file");
let ext = path.extension().and_then(|s| s.to_str());
let name = match ext {
Some(ext) => format!("{stem} (sync-conflict {stamp}).{ext}"),
None => format!("{stem} (sync-conflict {stamp})"),
};
match path.parent() {
Some(dir) => dir.join(name),
None => PathBuf::from(name),
}
}
/// A pure decision for one file-shaped path. Transfer identities and filesystem
/// effects are attached by the executor after this classification.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(super) enum FilePlan {
Unchanged,
Deferred,
UploadNew,
UploadRevision,
Download,
Conflict,
DeleteLocal,
DeleteRemote,
ForgetBaseline,
}
pub(super) fn plan_file(
local: Option<&LocalItem>,
remote: Option<&RemoteItem>,
baseline: Option<&StoredSyncEntry>,
) -> FilePlan {
if local.is_some_and(|item| item.open_for_write) {
return FilePlan::Deferred;
}
let local_changed = local.is_some_and(|item| {
baseline.is_none_or(|base| base.local_mtime != item.mtime || base.local_size != item.size)
});
let remote_changed = remote.is_some_and(|item| {
baseline.is_none_or(|base| remote_sig(base) != Some((item.mtime, item.size)))
});
match (local, remote) {
(Some(_), Some(_)) => match (local_changed, remote_changed) {
(false, false) => FilePlan::Unchanged,
(true, false) => FilePlan::UploadRevision,
(false, true) => FilePlan::Download,
(true, true) => FilePlan::Conflict,
},
(Some(_), None) if baseline.is_none() || local_changed => FilePlan::UploadNew,
(Some(_), None) => FilePlan::DeleteLocal,
(None, Some(_)) if baseline.is_none() || remote_changed => FilePlan::Download,
(None, Some(_)) => FilePlan::DeleteRemote,
(None, None) => FilePlan::ForgetBaseline,
}
}
#[cfg(test)]
mod file_plan_tests {
use super::*;
use proton_drive_rs::proton_sdk::ids::{LinkId, VolumeId};
fn local(mtime: i64, size: i64) -> LocalItem {
LocalItem {
is_dir: false,
mtime,
size,
open_for_write: false,
}
}
fn remote(mtime: i64, size: i64) -> RemoteItem {
RemoteItem {
uid: NodeUid::new(VolumeId::from("v"), LinkId::from("l")),
is_dir: false,
mtime,
size,
}
}
fn baseline(
local_mtime: i64,
local_size: i64,
remote_mtime: i64,
remote_size: i64,
) -> StoredSyncEntry {
StoredSyncEntry {
rel_path: "file".into(),
remote_uid: Some("v~l".into()),
local_mtime,
local_size,
remote_rev: Some(remote_mtime.to_string()),
remote_hash: Some(remote_size.to_string()),
}
}
#[test]
fn file_plan_covers_every_presence_and_change_shape() {
let same_local = local(10, 20);
let changed_local = local(11, 20);
let same_remote = remote(30, 40);
let changed_remote = remote(31, 40);
let base = baseline(10, 20, 30, 40);
let cases = [
(None, None, None, FilePlan::ForgetBaseline),
(None, None, Some(&base), FilePlan::ForgetBaseline),
(Some(&same_local), None, None, FilePlan::UploadNew),
(Some(&same_local), None, Some(&base), FilePlan::DeleteLocal),
(Some(&changed_local), None, Some(&base), FilePlan::UploadNew),
(None, Some(&same_remote), None, FilePlan::Download),
(
None,
Some(&same_remote),
Some(&base),
FilePlan::DeleteRemote,
),
(None, Some(&changed_remote), Some(&base), FilePlan::Download),
(
Some(&same_local),
Some(&same_remote),
None,
FilePlan::Conflict,
),
(
Some(&same_local),
Some(&same_remote),
Some(&base),
FilePlan::Unchanged,
),
(
Some(&changed_local),
Some(&same_remote),
Some(&base),
FilePlan::UploadRevision,
),
(
Some(&same_local),
Some(&changed_remote),
Some(&base),
FilePlan::Download,
),
(
Some(&changed_local),
Some(&changed_remote),
Some(&base),
FilePlan::Conflict,
),
];
for (local, remote, baseline, expected) in cases {
assert_eq!(plan_file(local, remote, baseline), expected);
}
}
#[test]
fn an_open_writer_overrides_every_other_file_decision() {
let mut local = local(11, 20);
local.open_for_write = true;
let remote = remote(31, 40);
let base = baseline(10, 20, 30, 40);
for (remote, baseline) in [
(None, None),
(None, Some(&base)),
(Some(&remote), None),
(Some(&remote), Some(&base)),
] {
assert_eq!(
plan_file(Some(&local), remote, baseline),
FilePlan::Deferred
);
}
}
#[test]
fn a_missing_remote_signature_is_treated_as_changed() {
let local = local(10, 20);
let remote = remote(30, 40);
let mut base = baseline(10, 20, 30, 40);
base.remote_hash = None;
assert_eq!(
plan_file(Some(&local), Some(&remote), Some(&base)),
FilePlan::Download
);
}
}
+298
View File
@@ -0,0 +1,298 @@
//! Bulk local-to-Drive upload workflow.
use super::*;
/// How many files the bulk uploader ships at once. Overlaps the per-file network
/// round-trips without letting an unbounded number of block buffers pile up.
const UPLOAD_CONCURRENCY: usize = 4;
/// One file queued for bulk upload, resolved during the directory walk so the
/// concurrent phase carries everything it needs (no shared state, no `block_on`).
struct UploadTask {
/// Inode of the (already-created) remote parent folder, for interning the
/// uploaded node afterwards.
parent_ino: u64,
parent_uid: NodeUid,
name: String,
/// Local filesystem path to stream from.
path: PathBuf,
size: u64,
}
/// Tally of a completed [`Core::upload_paths`] batch, for the daemon log.
#[derive(Default)]
pub(super) struct UploadStats {
pub(super) uploaded: usize,
pub(super) failed: usize,
/// Total plaintext bytes of the files that uploaded successfully.
pub(super) bytes: u64,
/// Folders created (or reused) to mirror the local tree.
pub(super) folders: usize,
}
/// Upload every [`UploadTask`] with at most `limit` in flight at once, each
/// streamed straight from disk and ticking its own transfer-registry guard.
/// Returns, per file, either `(parent_ino, new_uid)` for the caller to intern or
/// `(name, error)` to log — one failure never sinks the batch.
///
/// `job` counts files finished (either way: a failure is still one file the batch
/// no longer waits on), so a front-end can show "12 of 40" over the per-file bars.
async fn run_uploads(
core: Core,
tasks: Vec<UploadTask>,
limit: usize,
job: Arc<JobGuard>,
) -> Vec<Result<(u64, NodeUid, u64), (String, String)>> {
let sem = Arc::new(tokio::sync::Semaphore::new(limit));
let mut set = tokio::task::JoinSet::new();
for t in tasks {
let core = core.clone();
let sem = sem.clone();
set.spawn(async move {
let _permit = sem.acquire_owned().await.expect("semaphore not closed");
let file = match std::fs::File::open(&t.path) {
Ok(f) => f,
Err(e) => return Err((t.name, format!("open {}: {e}", t.path.display()))),
};
let mtime = std::fs::metadata(&t.path)
.and_then(|m| m.modified())
.ok()
.and_then(|mt| mt.duration_since(std::time::UNIX_EPOCH).ok())
.map(|d| d.as_secs() as i64);
let guard = core
.transfers
.begin(&t.name, "", TransferDirection::Upload, t.size);
let reader = OwnedCountingReader::new(file, guard);
match core
.client
.upload_file_from(
&t.parent_uid,
&t.name,
media_type_for(&t.name),
reader,
t.size as i64,
Vec::new(),
mtime,
false,
)
.await
{
Ok(uid) => Ok((t.parent_ino, uid, t.size)),
Err(e) => Err((t.name, format!("upload: {e}"))),
}
});
}
let mut out = Vec::new();
while let Some(joined) = set.join_next().await {
job.step();
match joined {
Ok(result) => out.push(result),
Err(e) => warn!(error = %e, "upload task panicked"),
}
}
out
}
impl Core {
/// Bulk-upload local files and directory trees under `sources` into the
/// mountpoint-relative `parent_rel` folder. Directories are recreated (or
/// merged into an existing same-named folder) and walked; the resulting flat
/// set of files is uploaded with bounded concurrency, each ticking the
/// transfer registry so a front-end sees live progress. Runs on a background
/// thread — a large tree far outlasts the control socket's read timeout — so
/// it reports only a summary for the log. Individual failures are counted and
/// logged rather than aborting the whole batch.
pub(super) fn upload_paths(
&self,
parent_rel: &Path,
sources: &[PathBuf],
) -> CoreResult<UploadStats> {
let (pino, parent_uid) = self.resolve(parent_rel)?;
self.ensure_children(pino)
.map_err(|e| self.errno_error(e, "enumerate"))?;
// Phase 1 (sequential): build the remote folder skeleton and collect the
// flat list of files to upload. Folders must exist before their children,
// so this can't be parallelised. On a deep tree this is a folder-creation
// round-trip per directory before a single byte moves — long enough that
// it needs a job of its own, or the daemon looks idle for minutes.
let mut tasks = Vec::new();
let mut folders = 0usize;
{
let job = self.transfers.begin_job("Preparing upload");
for src in sources {
if let Err(e) =
self.collect_uploads(pino, &parent_uid, src, &mut tasks, &mut folders, &job)
{
warn!(source = %src.display(), error = %e, "skipping source");
}
}
}
// Phase 2 (concurrent): upload the files, up to UPLOAD_CONCURRENCY at once.
// Each file reports its own bytes; this job is the batch's "N of M files",
// which is the number a user actually waits on.
let job = Arc::new(self.transfers.begin_job(match tasks.len() {
1 => "Uploading 1 file".to_string(),
n => format!("Uploading {n} files"),
}));
job.set_total(tasks.len() as u64);
let outcomes = self.rt.block_on(run_uploads(
self.clone(),
tasks,
UPLOAD_CONCURRENCY,
job.clone(),
));
drop(job);
// Phase 3 (sequential): intern each uploaded node so it shows up in the
// listing without a re-enumeration. fetch_node uses `block_on`, so it must
// run here rather than inside the async batch — and it is a round-trip per
// file, so it too gets a job rather than a silent tail.
let mut stats = UploadStats {
folders,
..UploadStats::default()
};
let job = self.transfers.begin_job("Finishing upload");
job.set_total(outcomes.len() as u64);
for outcome in outcomes {
job.step();
match outcome {
Ok((parent_ino, uid, size)) => {
stats.uploaded += 1;
stats.bytes += size;
match self.fetch_node(&uid) {
Ok(node) => {
let mut st = self.state.lock();
let ino = st.intern(parent_ino, node);
if let Some(kids) = st.children.get_mut(&parent_ino)
&& !kids.contains(&ino)
{
kids.push(ino);
}
}
// Uploaded fine but the metadata refresh failed; it will
// appear on the next directory enumeration regardless.
Err(e) => warn!(%uid, error = ?e, "uploaded node metadata refresh failed"),
}
}
Err((name, msg)) => {
stats.failed += 1;
warn!(name, error = %msg, "file upload failed");
}
}
}
info!(
uploaded = stats.uploaded,
failed = stats.failed,
"bulk upload finished"
);
Ok(stats)
}
/// Resolve a remote child folder named `name` under `pino`, creating it if it
/// doesn't exist, and return its `(inode, uid)`. Reusing an existing same-named
/// folder makes re-uploading a directory merge into it rather than fail on a
/// duplicate name.
fn ensure_remote_folder(
&self,
pino: u64,
parent_uid: &NodeUid,
name: &str,
) -> CoreResult<(u64, NodeUid)> {
if name.is_empty() || name.contains('/') {
return Err(CoreError::invalid(format!("invalid folder name: {name:?}")));
}
self.ensure_children(pino)
.map_err(|e| self.errno_error(e, "enumerate"))?;
{
let st = self.state.lock();
if let Some(kids) = st.children.get(&pino) {
for &ino in kids {
if let Some(e) = st.entries.get(&ino)
&& e.node.is_folder()
&& e.node.name == name
{
return Ok((ino, e.uid.clone()));
}
}
}
}
let new_uid = self
.rt
.block_on(
self.client
.create_folder(parent_uid, name, Some(now_secs())),
)
.map_err(|e| CoreError::from_api(&e, &format!("create folder {name}")))?;
let node = self
.fetch_node(&new_uid)
.map_err(|e| self.errno_error(e, "fetch node"))?;
let mut st = self.state.lock();
let ino = st.intern(pino, node);
if let Some(kids) = st.children.get_mut(&pino)
&& !kids.contains(&ino)
{
kids.push(ino);
}
Ok((ino, new_uid))
}
/// Walk one local source path, appending its files to `tasks`. A file becomes
/// one task; a directory is recreated remotely and recursed into (children
/// sorted for a stable order). Symlinks and other special files are skipped.
///
/// `job` narrates the walk with the folder currently being mirrored. It stays
/// indeterminate: nothing knows the size of the tree until the walk has ended.
fn collect_uploads(
&self,
pino: u64,
parent_uid: &NodeUid,
src: &Path,
tasks: &mut Vec<UploadTask>,
folders: &mut usize,
job: &JobGuard,
) -> CoreResult<()> {
let meta = std::fs::symlink_metadata(src)
.map_err(|e| CoreError::internal(format!("stat {}: {e}", src.display())))?;
let name = src
.file_name()
.and_then(|n| n.to_str())
.ok_or_else(|| CoreError::invalid(format!("unusable name: {}", src.display())))?
.to_string();
if meta.is_dir() {
job.detail(&name);
let (child_ino, child_uid) = self.ensure_remote_folder(pino, parent_uid, &name)?;
*folders += 1;
let mut entries: Vec<PathBuf> = std::fs::read_dir(src)
.map_err(|e| CoreError::internal(format!("read dir {}: {e}", src.display())))?
.filter_map(|e| e.ok())
.map(|e| e.path())
.collect();
entries.sort();
for entry in entries {
if let Err(e) =
self.collect_uploads(child_ino, &child_uid, &entry, tasks, folders, job)
{
warn!(source = %entry.display(), error = %e, "skipping entry");
}
}
// Deeper folders have retitled the job by now; put this one back so the
// line tracks the walk's position rather than its deepest leaf.
job.detail(&name);
} else if meta.is_file() {
if name.contains('/') {
return Err(CoreError::invalid(format!("invalid file name: {name:?}")));
}
tasks.push(UploadTask {
parent_ino: pino,
parent_uid: parent_uid.clone(),
name,
path: src.to_path_buf(),
size: meta.len(),
});
}
Ok(())
}
}
+18 -4
View File
@@ -41,9 +41,10 @@ use pdfs_core::config::AppDirs;
use pdfs_core::control::{
ActivityEntry, ActivityKind, BookmarkInfo, DeviceInfo, DirEntry, ErrorKind, InvitationInfo,
JobItem, PhotoItem, PhotoKind, PublicLinkInfo, RefreshScope, Request, Response, SearchHit,
ShareEntry, ShareEntryKind, SharedItem, SyncFolderInfo, SyncPhase, SyncProgress,
TransferDirection, TransferItem, pending_summary, send,
JobItem, PhotoItem, PhotoKind, PublicLinkInfo, RefreshScope, Request, Response,
RestorableFolder, RestoreItem, SearchHit, ShareEntry, ShareEntryKind, SharedItem,
SyncFolderInfo, SyncPhase, SyncProgress, TransferDirection, TransferItem, pending_summary,
send,
};
use pdfs_core::service;
@@ -399,6 +400,11 @@ fn build_window(app: &adw::Application) {
transfers_inflight: Cell::new(false),
cache_bar: main_widgets.cache_bar.clone(),
cache_label: main_widgets.cache_label.clone(),
quota_group: main_widgets.quota_group.clone(),
quota_bar: main_widgets.quota_bar.clone(),
quota_label: main_widgets.quota_label.clone(),
quota_inflight: Cell::new(false),
quota_checked_at: Cell::new(None),
autostart_row: main_widgets.autostart_row.clone(),
budget_row: main_widgets.budget_row.clone(),
mountpoint_row: main_widgets.mountpoint_row.clone(),
@@ -478,6 +484,7 @@ fn build_window(app: &adw::Application) {
with_me_group: shared_widgets.shared_with_me.clone(),
invitations_group: shared_widgets.invitations.clone(),
bookmarks_group: shared_widgets.bookmarks.clone(),
nav: RefCell::new(Vec::new()),
rows: RefCell::new(Vec::new()),
inflight: Cell::new(false),
loaded_at: Cell::new(None),
@@ -499,6 +506,8 @@ fn build_window(app: &adw::Application) {
rows: RefCell::new(Vec::new()),
sync_group: devices_widgets.sync_group.clone(),
sync_rows: RefCell::new(Vec::new()),
rename_this: devices_widgets.rename_this.clone(),
this_device: RefCell::new(None),
inflight: Cell::new(false),
loaded_at: Cell::new(None),
},
@@ -535,7 +544,12 @@ fn build_window(app: &adw::Application) {
wire_trash(&ui, &trash_widgets.list, &trash_widgets.empty);
wire_shared(&ui, &shared_widgets.retry, &shared_widgets.add_bookmark);
wire_shared_by_me(&ui, &shared_by_me_widgets.retry);
wire_devices(&ui, &devices_widgets.retry, &devices_widgets.add_folder);
wire_devices(
&ui,
&devices_widgets.retry,
&devices_widgets.add_folder,
&devices_widgets.restore,
);
wire_activity(&ui, &activity_widgets.retry);
wire_refresh(
&ui,
+221 -1
View File
@@ -10,6 +10,13 @@ pub(crate) struct DevicesState {
pub(crate) rows: RefCell<Vec<gtk4::Widget>>,
pub(crate) sync_group: adw::PreferencesGroup,
pub(crate) sync_rows: RefCell<Vec<gtk4::Widget>>,
/// "Rename" action in the "This computer" header. Insensitive until the
/// device list identifies this machine's own device.
pub(crate) rename_this: gtk4::Button,
/// This machine's own device `(uid, name)`, from the last device list. The
/// current device is filtered out of the "Other computers" rows, so this is
/// where its rename target lives. `None` until identified.
pub(crate) this_device: RefCell<Option<(String, String)>>,
pub(crate) inflight: Cell<bool>,
pub(crate) loaded_at: Cell<Option<Instant>>,
}
@@ -20,11 +27,16 @@ pub(crate) struct DevicesWidgets {
pub(crate) status: adw::StatusPage,
/// "This computer" — the local folders synced to this machine's device.
pub(crate) sync_group: adw::PreferencesGroup,
/// "Rename" in the "This computer" header — renames this machine's device.
pub(crate) rename_this: gtk4::Button,
/// "Other computers" — the account's *other* registered devices. This
/// machine's own device is deliberately not among them; see
/// [`repaint_devices`].
pub(crate) group: adw::PreferencesGroup,
pub(crate) add_folder: gtk4::Button,
/// "Restore Folders" — re-attach this device's remote folders to local
/// directories after adopting it on a new machine.
pub(crate) restore: gtk4::Button,
pub(crate) retry: gtk4::Button,
pub(crate) refresh: gtk4::Button,
}
@@ -47,11 +59,18 @@ pub(crate) fn build_devices_page() -> (gtk4::Widget, DevicesWidgets) {
.valign(gtk4::Align::Center)
.build();
add_folder.add_css_class("flat");
let restore = gtk4::Button::builder()
.label("Restore Folders")
.tooltip_text("Sync this computer's Drive folders back to local directories")
.valign(gtk4::Align::Center)
.build();
restore.add_css_class("flat");
let refresh = refresh_button();
let header = gtk4::Box::new(gtk4::Orientation::Horizontal, 8);
header.append(&titles);
header.append(&refresh);
header.append(&restore);
header.append(&add_folder);
// The description spells out what the per-row On-demand switch does to the
@@ -65,6 +84,17 @@ pub(crate) fn build_devices_page() -> (gtk4::Widget, DevicesWidgets) {
them as you open them.",
)
.build();
// Rename control for *this* machine's device, in the section header. The
// current device is filtered out of "Other computers", so this is the only
// place it can be renamed; insensitive until the device list identifies it.
let rename_this = gtk4::Button::builder()
.icon_name("document-edit-symbolic")
.tooltip_text("Rename this computer")
.valign(gtk4::Align::Center)
.sensitive(false)
.build();
rename_this.add_css_class("flat");
sync_group.set_header_suffix(Some(&rename_this));
let group = adw::PreferencesGroup::builder()
.title("Other computers")
.description("Other devices backing up to this account.")
@@ -113,8 +143,10 @@ pub(crate) fn build_devices_page() -> (gtk4::Widget, DevicesWidgets) {
content,
status,
sync_group,
rename_this,
group,
add_folder,
restore,
retry,
refresh,
},
@@ -122,7 +154,12 @@ pub(crate) fn build_devices_page() -> (gtk4::Widget, DevicesWidgets) {
}
/// Install the Devices page's retry button and the "Add Folder" action.
pub(crate) fn wire_devices(ui: &Rc<Ui>, retry: &gtk4::Button, add_folder: &gtk4::Button) {
pub(crate) fn wire_devices(
ui: &Rc<Ui>,
retry: &gtk4::Button,
add_folder: &gtk4::Button,
restore: &gtk4::Button,
) {
let ui_retry = ui.clone();
retry.connect_clicked(move |_| {
service::restart();
@@ -130,6 +167,14 @@ pub(crate) fn wire_devices(ui: &Rc<Ui>, retry: &gtk4::Button, add_folder: &gtk4:
});
let ui_add = ui.clone();
add_folder.connect_clicked(move |_| prompt_add_sync_folder(&ui_add));
let ui_restore = ui.clone();
restore.connect_clicked(move |_| prompt_restore_folders(&ui_restore));
let ui_ren = ui.clone();
ui.devices.rename_this.connect_clicked(move |_| {
if let Some((uid, name)) = ui_ren.devices.this_device.borrow().clone() {
prompt_rename_device(&ui_ren, &uid, &name);
}
});
}
/// Show a status page in place of the Devices list.
@@ -264,6 +309,22 @@ pub(crate) fn repaint_devices(ui: &Rc<Ui>, devices: &[DeviceInfo]) {
for row in ui.devices.rows.borrow_mut().drain(..) {
ui.devices.group.remove(&row);
}
// Identify this machine's own device so the "This computer" header's Rename
// action has a target. It never appears in the rows below, so this is the
// only place its name/uid is captured.
match devices.iter().find(|d| d.this_device) {
Some(me) => {
*ui.devices.this_device.borrow_mut() = Some((me.uid.clone(), me.name.clone()));
ui.devices.rename_this.set_sensitive(true);
ui.devices
.rename_this
.set_tooltip_text(Some(&format!("Rename this computer ({})", me.name)));
}
None => {
*ui.devices.this_device.borrow_mut() = None;
ui.devices.rename_this.set_sensitive(false);
}
}
let others: Vec<&DeviceInfo> = devices.iter().filter(|d| !d.this_device).collect();
if others.is_empty() {
let row = adw::ActionRow::builder()
@@ -294,6 +355,21 @@ pub(crate) fn repaint_devices(ui: &Rc<Ui>, devices: &[DeviceInfo]) {
.valign(gtk4::Align::Center)
.build();
rename.add_css_class("flat");
// Adoption is how a reinstalled or renamed machine re-attaches to the
// device it used to be, instead of registering a duplicate. It only
// makes sense on *another* device's row, which is the only place this
// loop paints.
let adopt = gtk4::Button::builder()
.icon_name("insert-object-symbolic")
.tooltip_text("Use this computer's identity for this machine")
.valign(gtk4::Align::Center)
.build();
adopt.add_css_class("flat");
let ui_ad = ui.clone();
let uid_ad = dev.uid.clone();
let name_ad = dev.name.clone();
adopt.connect_clicked(move |_| prompt_adopt_device(&ui_ad, &uid_ad, &name_ad));
row.add_suffix(&adopt);
let ui_ren = ui.clone();
let uid_ren = dev.uid.clone();
let name_ren = dev.name.clone();
@@ -533,6 +609,110 @@ pub(crate) fn prompt_add_sync_folder(ui: &Rc<Ui>) {
});
}
/// Ask the daemon what this machine's device holds, then show a picker mapping
/// each remote folder onto a local directory.
///
/// The daemon's paths are proposals — from the device's `profile.json` when it
/// makes sense here, else `~/<name>` — so every one of them is editable and
/// nothing is restored without being ticked.
pub(crate) fn prompt_restore_folders(ui: &Rc<Ui>) {
ui.busy_begin();
let rx = spawn_request(ui.dirs.control_socket(), Request::ListRestorableFolders);
let ui = ui.clone();
glib::spawn_future_local(async move {
let result = rx.recv().await;
ui.busy_end();
match result {
Ok(Ok(Response::RestorableFolders { items })) => show_restore_picker(&ui, items),
Ok(Ok(Response::Error { message, kind })) => {
toast_failure(&ui, "Couldn't list folders to restore", &message, kind)
}
_ => toast_error(
&ui,
"Couldn't list folders to restore",
"The mount service didn't respond.",
),
}
});
}
/// The restore picker itself: one editable row per restorable folder.
fn show_restore_picker(ui: &Rc<Ui>, items: Vec<RestorableFolder>) {
let win = ui_window(ui);
let candidates: Vec<RestorableFolder> =
items.into_iter().filter(|f| !f.already_synced).collect();
if candidates.is_empty() {
toast(
ui,
"Nothing to restore — this computer's folders are all synced here.",
);
return;
}
let group = adw::PreferencesGroup::builder()
.description(
"Tick the folders to sync to this machine, and adjust where each one should live.",
)
.build();
// Held so the response handler can read back what the user ticked and typed.
let mut controls: Vec<(String, String, gtk4::CheckButton, adw::EntryRow)> = Vec::new();
for f in candidates {
let check = gtk4::CheckButton::builder()
.active(true)
.valign(gtk4::Align::Center)
.build();
let row = adw::EntryRow::builder().title(&f.name).build();
row.set_text(&f.local_path);
row.add_prefix(&check);
group.add(&row);
controls.push((f.remote_uid, f.mode, check, row));
}
let scroll = gtk4::ScrolledWindow::builder()
.propagate_natural_height(true)
.max_content_height(420)
.child(&group)
.build();
let dialog = adw::AlertDialog::builder()
.heading("Restore folders")
.body("These folders are backed up under this computer in Proton Drive.")
.extra_child(&scroll)
.build();
dialog.add_response("cancel", "Cancel");
dialog.add_response("restore", "Restore");
dialog.set_response_appearance("restore", adw::ResponseAppearance::Suggested);
dialog.set_default_response(Some("restore"));
dialog.set_close_response("cancel");
let ui = ui.clone();
dialog.connect_response(None, move |_, resp| {
if resp != "restore" {
return;
}
let items: Vec<RestoreItem> = controls
.iter()
.filter(|(_, _, check, _)| check.is_active())
.map(|(uid, mode, _, row)| RestoreItem {
remote_uid: uid.clone(),
local_path: row.text().to_string(),
mode: mode.clone(),
})
.collect();
if items.is_empty() {
return;
}
// The daemon acks before the downloads finish, like AddSyncFolder — the
// rows appear as the periodic refresh picks them up.
run_devices_mutation(
&ui,
Request::RestoreSyncFolders { items },
"Restoring folders…",
"Couldn't restore folders",
);
});
dialog.present(win.as_ref());
}
/// Flip a synced folder between `mirror` and `ondemand`. Reloads after so the
/// row's subtitle and switch reflect the daemon's real state (the request may be
/// rejected, e.g. switching to on-demand while a folder is mid-sync).
@@ -698,6 +878,46 @@ pub(crate) fn prompt_remove_device(ui: &Rc<Ui>, uid: &str, name: &str) {
dialog.present(win.as_ref());
}
/// Confirm, then adopt another device as this machine's identity.
///
/// Worth a confirmation rather than a plain click: adoption re-points this
/// machine's syncing at another computer's device folder, and the folders
/// already synced here keep pointing at the old one until they are removed. The
/// dialog says so, because the alternative is a user discovering it by watching
/// their folders diverge.
pub(crate) fn prompt_adopt_device(ui: &Rc<Ui>, uid: &str, name: &str) {
let win = ui_window(ui);
let dialog = adw::AlertDialog::builder()
.heading("Use this computer's identity")
.body(format!(
"Treat this machine as “{name}”?\n\nNew synced folders are created under that \
computer in Proton Drive, and this machine keeps that identity even if its hostname \
changes. Folders already synced here are not moved.\n\nUse “Restore folders” \
afterwards to bring back what “{name}” was syncing."
))
.build();
dialog.add_response("cancel", "Cancel");
dialog.add_response("adopt", "Use identity");
dialog.set_response_appearance("adopt", adw::ResponseAppearance::Suggested);
dialog.set_default_response(Some("cancel"));
dialog.set_close_response("cancel");
let ui = ui.clone();
let uid = uid.to_string();
dialog.connect_response(None, move |_, resp| {
if resp == "adopt" {
run_devices_mutation(
&ui,
Request::AdoptDevice {
uid: Some(uid.clone()),
},
"Identity adopted",
"Couldn't adopt that computer",
);
}
});
dialog.present(win.as_ref());
}
/// Run a mutation raised from the Devices page and reload the page on success.
pub(crate) fn run_devices_mutation(
ui: &Rc<Ui>,
+174 -6
View File
@@ -10,6 +10,10 @@ pub(crate) struct SharedState {
pub(crate) with_me_group: adw::PreferencesGroup,
pub(crate) invitations_group: adw::PreferencesGroup,
pub(crate) bookmarks_group: adw::PreferencesGroup,
/// Where in a shared folder the page currently is, as `(uid, name)` from the
/// top level down. Empty = the top level. Shared subtrees have no path in the
/// mount, so descending is uid-addressed and the stack *is* the breadcrumb.
pub(crate) nav: RefCell<Vec<(String, String)>>,
pub(crate) rows: RefCell<Vec<(adw::PreferencesGroup, gtk4::Widget)>>,
/// Guards the Shared page's load so overlapping navigations don't stack.
pub(crate) inflight: Cell<bool>,
@@ -139,10 +143,19 @@ pub(crate) fn shared_status(ui: &Rc<Ui>, icon: &str, title: &str, description: &
/// Fetch the three Shared sections (shared-with-me, invitations, bookmarks) in
/// parallel and repaint the page once all three land.
///
/// Inside a shared folder ([`SharedState::nav`] non-empty) only that folder's
/// children are fetched: invitations and bookmarks belong to the top level, and
/// carrying them down a subtree would read as if they lived there.
pub(crate) fn load_shared(ui: &Rc<Ui>) {
if ui.shared.inflight.get() {
return;
}
let current = ui.shared.nav.borrow().last().cloned();
if let Some((uid, _)) = current {
load_shared_folder(ui, uid);
return;
}
ui.shared.inflight.set(true);
shared_status(
ui,
@@ -192,6 +205,163 @@ pub(crate) fn load_shared(ui: &Rc<Ui>) {
});
}
/// List one shared folder's children and repaint the page as that folder's view.
/// The uid comes from the row that was activated (or from the nav stack on a
/// reload) — a shared subtree is reachable no other way.
fn load_shared_folder(ui: &Rc<Ui>, uid: String) {
ui.shared.inflight.set(true);
shared_status(
ui,
"folder-symbolic",
"Loading…",
"Reading this shared folder.",
false,
);
ui.busy_begin();
let rx = spawn_request(ui.dirs.control_socket(), Request::ListSharedFolder { uid });
let ui = ui.clone();
glib::spawn_future_local(async move {
let result = rx.recv().await;
ui.busy_end();
ui.shared.inflight.set(false);
match result {
Ok(Ok(Response::Entries { entries })) => {
repaint_shared_folder(&ui, &entries);
ui.shared.loaded_at.set(Some(Instant::now()));
}
Ok(Ok(Response::Error { message, kind })) => {
// The folder is gone or access was revoked: fall back to the top
// level rather than stranding the page on a dead uid.
ui.shared.nav.borrow_mut().pop();
toast_failure(&ui, "Couldn't open shared folder", &message, kind);
load_shared(&ui);
}
_ => {
ui.shared.loaded_at.set(None);
shared_unreachable(&ui);
}
}
});
}
/// Paint the contents of the shared folder at the top of the nav stack: a row to
/// go back up, then the children. Invitations and bookmarks are hidden here —
/// they are top-level things, not contents of this folder.
fn repaint_shared_folder(ui: &Rc<Ui>, entries: &[DirEntry]) {
for (group, row) in ui.shared.rows.borrow_mut().drain(..) {
group.remove(&row);
}
ui.shared.content.set_visible_child_name("list");
ui.shared.invitations_group.set_visible(false);
ui.shared.bookmarks_group.set_visible(false);
let nav = ui.shared.nav.borrow().clone();
let title = nav
.iter()
.map(|(_, name)| name.as_str())
.collect::<Vec<_>>()
.join(" / ");
ui.shared.with_me_group.set_title(&title);
let mut rows: Vec<(adw::PreferencesGroup, gtk4::Widget)> = Vec::new();
let up = adw::ActionRow::builder()
.title("Back")
.activatable(true)
.build();
up.add_prefix(&gtk4::Image::from_icon_name("go-up-symbolic"));
let ui_up = ui.clone();
up.connect_activated(move |_| {
ui_up.shared.nav.borrow_mut().pop();
load_shared(&ui_up);
});
ui.shared.with_me_group.add(&up);
rows.push((ui.shared.with_me_group.clone(), up.upcast()));
if entries.is_empty() {
let row = dim_row("This folder is empty.");
ui.shared.with_me_group.add(&row);
rows.push((ui.shared.with_me_group.clone(), row.upcast()));
} else {
for entry in entries {
let row = shared_entry_row(ui, entry);
ui.shared.with_me_group.add(&row);
rows.push((ui.shared.with_me_group.clone(), row.upcast()));
}
}
*ui.shared.rows.borrow_mut() = rows;
}
/// A row for one node shared with me: folders descend into, files download and
/// open with the user's default application.
fn shared_entry_row(ui: &Rc<Ui>, entry: &DirEntry) -> adw::ActionRow {
let row = adw::ActionRow::builder()
.title(&entry.name)
.activatable(true)
.build();
if !entry.is_dir {
row.set_subtitle(&human_bytes(entry.size));
}
row.add_prefix(&gtk4::Image::from_icon_name(if entry.is_dir {
"folder-symbolic"
} else {
"text-x-generic-symbolic"
}));
let ui_act = ui.clone();
let uid = entry.uid.clone();
let name = entry.name.clone();
let is_dir = entry.is_dir;
row.connect_activated(move |_| {
if is_dir {
ui_act
.shared
.nav
.borrow_mut()
.push((uid.clone(), name.clone()));
load_shared(&ui_act);
} else {
open_shared_file(&ui_act, &uid, &name);
}
});
row
}
/// Download a file shared with me into the daemon's cache and hand it to the
/// user's default application — the shared-item twin of the browser's
/// download-and-open, addressed by uid because the file lives outside the mount.
fn open_shared_file(ui: &Rc<Ui>, uid: &str, name: &str) {
// Ignore a repeat activation of a file already downloading, so an impatient
// double-click doesn't kick off a second round-trip.
if !ui.opening.borrow_mut().insert(uid.to_string()) {
return;
}
ui.busy_begin();
toast(ui, &format!("Downloading “{name}”…"));
let rx = spawn_request(
ui.dirs.control_socket(),
Request::OpenSharedFile {
uid: uid.to_string(),
},
);
let ui = ui.clone();
let uid = uid.to_string();
glib::spawn_future_local(async move {
let result = rx.recv().await;
ui.busy_end();
ui.opening.borrow_mut().remove(&uid);
match result {
Ok(Ok(Response::FilePath { path })) => open_path(&path),
Ok(Ok(Response::Error { message, kind })) => {
toast_failure(&ui, "Couldn't open file", &message, kind)
}
_ => toast_error(
&ui,
"Couldn't open file",
"The mount service didn't respond.",
),
}
});
}
/// The daemon didn't answer the Shared page. Same still-starting vs. down split
/// as the other pages.
pub(crate) fn shared_unreachable(ui: &Rc<Ui>) {
@@ -233,6 +403,9 @@ pub(crate) fn repaint_shared(
group.remove(&row);
}
ui.shared.content.set_visible_child_name("list");
ui.shared.invitations_group.set_visible(true);
ui.shared.bookmarks_group.set_visible(true);
ui.shared.with_me_group.set_title("Shared with me");
let mut rows: Vec<(adw::PreferencesGroup, gtk4::Widget)> = Vec::new();
// Shared with me: name + Leave.
@@ -242,12 +415,7 @@ pub(crate) fn repaint_shared(
rows.push((ui.shared.with_me_group.clone(), row.upcast()));
} else {
for entry in shared {
let row = adw::ActionRow::builder().title(&entry.name).build();
row.add_prefix(&gtk4::Image::from_icon_name(if entry.is_dir {
"folder-symbolic"
} else {
"text-x-generic-symbolic"
}));
let row = shared_entry_row(ui, entry);
let leave = gtk4::Button::builder()
.label("Leave")
.valign(gtk4::Align::Center)
+92
View File
@@ -17,6 +17,14 @@ pub(crate) struct StatusState {
pub(crate) mount_row: adw::ActionRow,
pub(crate) cache_bar: gtk4::ProgressBar,
pub(crate) cache_label: gtk4::Label,
/// Account-quota group + its bar/label. Hidden until the first reading.
pub(crate) quota_group: adw::PreferencesGroup,
pub(crate) quota_bar: gtk4::ProgressBar,
pub(crate) quota_label: gtk4::Label,
/// One `AccountQuota` in flight at a time, and when it last succeeded — quota
/// barely moves, so [`refresh_quota`] refetches on a long TTL, not every tick.
pub(crate) quota_inflight: Cell<bool>,
pub(crate) quota_checked_at: Cell<Option<Instant>>,
/// "Start on login" toggle. [`Self::settings_suppress`] guards programmatic
/// sets so reflecting the systemd state doesn't fire the toggle handler.
pub(crate) autostart_row: adw::SwitchRow,
@@ -78,6 +86,10 @@ pub(crate) struct MainWidgets {
pub(crate) mount_row: adw::ActionRow,
/// Live upload/download progress, populated by the refresh loop.
pub(crate) transfers_group: adw::PreferencesGroup,
/// Account-quota group; hidden until `refresh_quota` gets a reading.
pub(crate) quota_group: adw::PreferencesGroup,
pub(crate) quota_bar: gtk4::ProgressBar,
pub(crate) quota_label: gtk4::Label,
pub(crate) cache_bar: gtk4::ProgressBar,
pub(crate) cache_label: gtk4::Label,
pub(crate) pins_group: adw::PreferencesGroup,
@@ -111,6 +123,29 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
account_row.add_suffix(&logout_button);
account_group.add(&account_row);
// Account storage: the Proton account quota (used of total), distinct from
// the local content cache below. A progress bar + "X of Y used" line, painted
// by `refresh_quota`. Hidden until the first successful read so a cold start
// (or an account the API can't report) shows nothing rather than an empty bar.
let quota_group = adw::PreferencesGroup::builder()
.title("Account storage")
.description("Your Proton storage across all products.")
.visible(false)
.build();
let quota_box = gtk4::Box::new(gtk4::Orientation::Vertical, 6);
quota_box.set_margin_top(6);
quota_box.set_margin_bottom(6);
let quota_bar = gtk4::ProgressBar::new();
let quota_label = gtk4::Label::builder().halign(gtk4::Align::Start).build();
quota_label.add_css_class("dim-label");
quota_box.append(&quota_bar);
quota_box.append(&quota_label);
let quota_row = adw::PreferencesRow::builder()
.activatable(false)
.child(&quota_box)
.build();
quota_group.add(&quota_row);
// Mount group: a read-only status line. The mount is managed automatically
// by the systemd user service; there is no toggle to fiddle with.
let mount_group = adw::PreferencesGroup::builder().title("Drive").build();
@@ -219,6 +254,7 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
inner.set_margin_start(12);
inner.set_margin_end(12);
inner.append(&account_group);
inner.append(&quota_group);
inner.append(&mount_group);
inner.append(&transfers_group);
inner.append(&storage_group);
@@ -238,6 +274,9 @@ pub(crate) fn build_main_page() -> (gtk4::Widget, MainWidgets) {
account_row,
mount_row,
transfers_group,
quota_group,
quota_bar,
quota_label,
cache_bar,
cache_label,
pins_group,
@@ -463,12 +502,65 @@ pub(crate) fn refresh(ui: &Rc<Ui>) {
// Both of these pages show work as it happens, so they follow the tick while
// they are on screen. Every other page loads on navigation only.
match ui.stack.visible_child_name().as_deref() {
Some("main") => refresh_quota(ui),
Some("devices") => refresh_sync_folders(ui),
Some("activity") => refresh_activity(ui),
_ => {}
}
}
/// How long a quota reading stays fresh. Account storage barely moves, so the
/// Settings tick refetches it only this often rather than every 2s.
const QUOTA_TTL: Duration = Duration::from_secs(60);
/// Fetch the account quota (if the last reading is stale) and paint the Account
/// storage group. Runs only while Settings is on screen. The group stays hidden
/// until a reading lands, so an account the API can't report — or a still-starting
/// daemon — shows nothing rather than an empty bar. A failed fetch leaves the last
/// good reading in place.
pub(crate) fn refresh_quota(ui: &Rc<Ui>) {
if ui.status.quota_inflight.get() {
return;
}
if let Some(at) = ui.status.quota_checked_at.get()
&& at.elapsed() < QUOTA_TTL
{
return;
}
ui.status.quota_inflight.set(true);
let rx = spawn_request(ui.dirs.control_socket(), Request::AccountQuota);
let ui = ui.clone();
glib::spawn_future_local(async move {
let result = rx.recv().await;
ui.status.quota_inflight.set(false);
if let Ok(Ok(Response::AccountQuota {
max_space,
used_space,
})) = result
{
ui.status.quota_checked_at.set(Some(Instant::now()));
let used = used_space.max(0) as u64;
if max_space > 0 {
let total = max_space as u64;
let fraction = (used as f64 / total as f64).min(1.0);
ui.status.quota_bar.set_fraction(fraction);
let pct = (fraction * 100.0).round() as u64;
ui.status.quota_label.set_text(&format!(
"{} of {} used ({pct}%)",
human_bytes(used),
human_bytes(total)
));
} else {
ui.status.quota_bar.set_fraction(0.0);
ui.status
.quota_label
.set_text(&format!("{} used", human_bytes(used)));
}
ui.status.quota_group.set_visible(true);
}
});
}
/// Record the mount state seen by the last status poll: gate every control that
/// needs a live daemon, and notify the desktop when the state actually flips.
///
+18 -3
View File
@@ -70,7 +70,7 @@ pub(crate) fn prompt_human_verification(
let token_tx = Rc::new(RefCell::new(Some(token_tx)));
let tx = token_tx.clone();
let dlg = dialog.clone();
let dlg_weak = dialog.downgrade();
content.connect_script_message_received(Some("hv"), move |_, value| {
let Some(token) = extract_token(&value.to_str()) else {
// Every `postMessage` on the page reaches here, most of them the
@@ -81,7 +81,9 @@ pub(crate) fn prompt_human_verification(
if let Some(tx) = tx.borrow_mut().take() {
let _ = tx.send(token);
}
dlg.close();
if let Some(dlg) = dlg_weak.upgrade() {
dlg.close();
}
});
// Covers the close button, Escape, and the user giving up.
@@ -104,7 +106,14 @@ pub(crate) fn prompt_human_verification(
/// unrecognised shape as success would hand the API an empty token and fail the
/// login with a confusing error.
fn extract_token(raw: &str) -> Option<String> {
let value: serde_json::Value = serde_json::from_str(raw).ok()?;
let mut value: serde_json::Value = serde_json::from_str(raw).ok()?;
// If the message was stringified twice (e.g. raw is a double-quoted JSON string),
// parse the inner string.
if let Some(inner) = value.as_str()
&& let Ok(parsed) = serde_json::from_str(inner)
{
value = parsed;
}
// The payload is nested under `payload` and the message names itself in
// `type`; both spellings of the success type have shipped.
let kind = value.get("type")?.as_str()?;
@@ -133,6 +142,12 @@ mod tests {
assert_eq!(extract_token(raw).as_deref(), Some("tok-1"));
}
#[test]
fn a_double_serialized_completion_message_yields_its_token() {
let raw = r#""{\"type\":\"HUMAN_VERIFICATION_SUCCESS\",\"payload\":{\"token\":\"tok-1\",\"type\":\"captcha\"}}""#;
assert_eq!(extract_token(raw).as_deref(), Some("tok-1"));
}
/// The page posts plenty that is not a completion; none of it may be
/// mistaken for one, or the login retries with a garbage token.
#[test]
+20 -6
View File
@@ -515,12 +515,10 @@ fn build_window(app: &adw::Application) {
ui_key.cycle_filter(back);
glib::Propagation::Stop
}
gtk4::gdk::Key::Return | gtk4::gdk::Key::KP_Enter => {
// Fall back to the top row when nothing is explicitly selected
// (e.g. results haven't rendered yet); `open` guards the rest.
ui_key.open(ui_key.cursor.get().unwrap_or(0), &window_key);
glib::Propagation::Stop
}
// Return is deliberately absent: `GtkText` binds it to `activate`
// and consumes it, so a bubble-phase controller on the window never
// sees it while the entry has focus (which is always, here). It is
// handled on the entry's `activate` signal instead — see below.
_ => glib::Propagation::Proceed,
}
});
@@ -528,6 +526,22 @@ fn build_window(app: &adw::Application) {
// pointer has moved focus into it.
window.add_controller(keys);
// Enter opens the row under the cursor. This lives on the entry rather than
// with the other keys on the window because `GtkText` claims Return for its
// own `activate` binding, so a bubble-phase controller upstream of the
// focused entry is never reached. Enter with focus in the list is covered by
// `row_activated` above.
let ui_activate = ui.clone();
let window_activate = window.clone();
entry.connect_activate(move |_| {
if ui_activate.opening.get() {
return;
}
// Fall back to the top row when nothing is explicitly selected (e.g.
// results haven't rendered yet); `open` guards the rest.
ui_activate.open(ui_activate.cursor.get().unwrap_or(0), &window_activate);
});
let debounce: Rc<RefCell<Option<glib::SourceId>>> = Rc::new(RefCell::new(None));
let ui_changed = ui.clone();
entry.connect_changed(move |entry| {
+63 -3
View File
@@ -33,7 +33,7 @@ graph TD
| [`pdfs-core`](file:///home/narl/dev/private/proton-drive-linux/crates/pdfs-core) | Core Infrastructure & Services | Cache Bookkeeping, Database migrations/schemas, IPC protocol payloads. | Holds the unified SQLite DB (`Db`) connection and the on-disk cache metadata (`ContentCache`). |
| [`pdfs-fuse`](file:///home/narl/dev/private/proton-drive-linux/crates/pdfs-fuse) | VFS Layer & Reconciliation | FUSE callbacks, background upload queue (`drain`), two-way sync runner. | Manages in-memory inode maps (`State`), active descriptors (`WriteHandle`), and background task threads. |
| [`pdfs-cli`](file:///home/narl/dev/private/proton-drive-linux/crates/pdfs-cli) | Command Line Interface | Command routing, daemon launcher, IPC client wrapper. | Stateless; communicates with daemon over IPC control socket. |
| [`pdfs-gui`](file:///home/narl/dev/private/proton-drive-linux/crates/pdfs-gui) | Graphical Interface | GTK Page timelines (Files, Photos, Shares, Status). | Stateless; polls daemon for status and lists timelines via IPC socket. |
| [`pdfs-gui`](file:///home/narl/dev/private/proton-drive-linux/crates/pdfs-gui) | Graphical Interface | GTK Page timelines (My files, Shared, Shared with me, Computers, Photos, Activity, Trash, Settings). | Stateless; polls daemon for status and lists timelines via IPC socket. |
---
@@ -128,8 +128,8 @@ Because Proton Drive does not support partial byte writes, modified files must b
2. **Close/Release (`queue_revision`):** When the application closes the file descriptor, the daemon:
- Fetches any untouched gaps from the remote base file to compile the full file.
- Moves the scratch file to `staging` under a `{uid}-{millis}-{counter}` name, so a staged blob can be tied back to its node without consulting the database.
- Queues a pending database operation (`PendingOp`).
3. **Async Drain Thread (`run_pending_drain`):** The background drain worker picks up the database operations queue, handles revisions uploads, resolves conflicts, and cleans up staging files.
- Queues a pending database operation (`PendingOp`). For `OP_REVISION` ops, execution is debounced by 2 seconds (`DRAIN_REVISION_DEBOUNCE = 2s`) to give rapid follow-up writes (e.g. `aria2c` preallocation followed by writing) time to supersede the staged blob before network transmission.
3. **Async Drain Thread (`run_pending_drain`):** The background drain worker picks up the database operations queue, handles revision uploads, resolves conflicts, and cleans up staging files. Upon landing a revision upload (`refresh_after_upload`), it rebaselines both still-queued ops (`rebaseline_pending`) and open write handles targeting the same node to prevent false self-conflict copies on subsequent writes.
```mermaid
sequenceDiagram
@@ -284,3 +284,63 @@ See [RECOVERY.md](RECOVERY.md) for what a lost machine means for the plaintext d
- Treat the cache and state directories as being as sensitive as the Drive contents themselves.
- Multi-user machines warrant checking `~/.cache` and `~/.local/state` are `0700` until §8.5 is addressed.
- Purging the cache (`pdfs` settings, or `PurgeCache` over IPC) removes content but **not** `cache.db`, and deliberately never removes undrained `staging/` or `recovery/` blobs.
---
## 9. Human Verification (CAPTCHA) Flow
When logging in from an unfamiliar IP address or VPN, the Proton API may gate the sign-in with a human verification challenge (CAPTCHA). This client handles this asynchronously and interactively.
### 9.1 Sequence of Verification and Re-Authentication
```mermaid
sequenceDiagram
autonumber
actor User as User (GUI)
participant Core as pdfs-core (Auth)
participant UI as pdfs-gui (Login Page)
participant Web as WebKitWebView Dialog
participant API as Proton API Server
User->>UI: Enter credentials & click Sign In
UI->>Core: login_interactive()
Core->>API: auth/v4 (Initial SRP Handshake)
API-->>Core: HTTP 422 (Error 9001: CAPTCHA Challenge URL)
Core->>UI: Error::HumanVerificationRequired(hv)
UI->>UI: Block login thread, dispatch to GTK main loop
UI->>Web: Create dialog & load verification URL
Note over User, Web: User completes CAPTCHA in embedded WebView
Web->>UI: window.postMessage(HUMAN_VERIFICATION_SUCCESS)
UI->>UI: Extract token (with double-serialization safety)
UI->>UI: Close dialog & send token to blocked login thread
UI->>Core: login_verified(with verification token)
Core->>API: auth/v4 (SRP retry with x-pm-human-verification-token)
API-->>Core: Returns session tokens
Core-->>UI: Login Successful
```
### 9.2 Key Technical Design Decisions
1. **Weak Reference UI Binding:** To prevent memory leaks and strong reference cycles between the parent dialog, the child `WebKitWebView`, the script message manager, and the connection callback, the dialog is downgraded to a `WeakRef` inside the callback:
```rust
let dlg_weak = dialog.downgrade();
content.connect_script_message_received(Some("hv"), move |_, value| {
// ...
if let Some(dlg) = dlg_weak.upgrade() {
dlg.close();
}
});
```
2. **Double-Serialization Tolerance:** The JavaScript message listener forwards event data as a JSON string to the native handler. Since the underlying page may post either JS objects or pre-serialized JSON strings, the Rust side performs dual-phase parsing:
```rust
let mut value = serde_json::from_str(raw).ok()?;
if let Some(inner) = value.as_str() {
if let Ok(parsed) = serde_json::from_str(inner) {
value = parsed;
}
}
```
This ensures compatibility with all versions of Proton's client verification scripts.
3. **SRP Handshake Reset:** Because a gated login burns the SRP handshake on the API side, the client cannot simply resume the previous request. Instead, `auth::login_interactive` restarts the SRP process from scratch with the verification credentials attached, keeping the complex handshake details isolated from the front-end.
4. **CLI Fallback:** Since the CLI has no native web browser engine, hitting the CAPTCHA gate fails immediately with a user-friendly message directing the user to sign in once via the GUI (`pdfs-app`) to persist the authenticated session keys to the system keyring.
+1631
View File
File diff suppressed because it is too large. Load diff
+130
View File
@@ -0,0 +1,130 @@
# Pending Development & Verification Tasks
This document compiles all pending, deferred, and unstarted tasks from the various roadmap, audit, and bug-tracking markdown files in the repository. Completed items have been filtered out.
---
## 1. Core Feature Roadmap (Horizon Tasks)
Derived from: [features.md](file:///home/narl/dev/private/proton-drive-linux/features.md) & [features-plan.md](file:///home/narl/dev/private/proton-drive-linux/features-plan.md)
* **Local Cache & Metadata Encryption at Rest (Priority 5 / Horizon 1)**: Encrypt the SQLite database `cache.db` on disk using SQLCipher (`rusqlite` feature `bundled-sqlcipher`), and encrypt raw content cache blocks (`content/blocks/` and `content/scratch/`) using a fast symmetric scheme like AES-GCM or ChaCha20-Poly1305. The encryption key should be derived from the OS Keyring.
* **Active Bandwidth Throttling & Traffic Shaping (Horizon 1)**: Add speed governors for daemon uploads and downloads. Support configuration variables `max_upload_rate_kbps` and `max_download_rate_kbps` in `config.json` that can be dynamically adjusted over the Unix control socket.
* **Interactive & Policy-Based Conflict Resolution (Horizon 1)**: Implement configurable conflict resolution policies (`rename-local`, `rename-remote`, `prefer-local`, `prefer-remote`, `interactive`). Add a "Conflicts" tab to the GUI to notify the user, block sync, and launch external visual diff tools (like Meld or KDiff3).
* **Multi-Account / Multi-Profile Support (Horizon 3)**: Enable running multiple profiles concurrently. Support separate configuration namespaces (e.g. `profiles/<profile_name>/`), independent SQLite databases, distinct mount paths (e.g. `~/ProtonDrive/Personal` and `~/ProtonDrive/Work`), and multiple daemon processes.
* **Custom FUSE Mount Options (Horizon 1)**: Expose custom mounting parameters in configuration (e.g. `ro` for read-only, `direct_io` to bypass kernel caching, `allow_other`). Auto-generate systemd user mount units upon sync folder configuration.
* **Symbolic Link & Hard Link Virtualization (Horizon 2)**: Add symbolic link virtualization inside the metadata store (`NodeType::Symlink`) by intercepting `readlink(2)` and `symlink(2)`. Targets should sync remotely as small encrypted metadata placeholder files on Drive (`proton-vfs-symlink: <target>`).
* **Desktop File Manager Integration (Horizon 2)**: Develop shell extension plugins for popular Linux file managers (Nautilus Python plugin, Dolphin Service Menus, Thunar custom actions) to offer context-menu shortcuts ("Pin", "Unpin", "Copy Share Link", "View Version History") and file overlay sync badges.
* **Integrated Photo Gallery Enhancements (Horizon 2)**: Extract and display Exif metadata (camera model, exposure, coordinates) from photos locally, support creating and editing Proton Photo Albums, build a date-based timeline scrollbar, and create an auto-upload Pictures folder pipeline.
* **Random-Access Media Streaming seek support (Horizon 2)**: Intercept out-of-order sparse reads (player seeks) in `pdfs-fuse` and prioritize downloading blocks around the seek offset, aborting/postponing sequential pre-fetches.
* **Dynamic Sync Dashboard & Queue Visualization (Horizon 2)**: GUI real-time transfer list (progress bars, speed, ETA), sync history feed, storage usage breakdown, and global pause/resume sync controls.
* **Interactive Terminal UI (TUI) Mode (Horizon 3)**: Implement `pdfs tui` using the `ratatui` crate to show transfer speed, sync status, active transfer queue with progress bars, and scrolling daemon logs.
* **Shutdown Safety & Write Queue Visibility (Horizon 1 / P4)**: Register systemd inhibitor locks in `pdfs-fuse` when there are outstanding staging writes to prevent shutdown or sleep, pop up warnings when exiting the GUI, and build a blocking `pdfs sync flush` command.
* **Block-Level Deduplication (Horizon 3)**: Transition to a Content-Addressable block cache where cached blocks are stored on disk by their SHA-256 hash. Map logical ranges in the database: `(node_uid, block_idx) -> block_hash` to avoid storing or downloading identical blocks.
* **Pre-emptive Sync Debouncing & File System Events (Horizon 3)**: Group filesystem write events on paths and delay sync queue insertion until a quiet period has elapsed (e.g. 5 seconds of inactivity) to prevent thrashing the Proton API.
* **File Version History (Horizon 2)**: Expose past file revisions and version history inside the GUI browser tabs.
---
## 2. Performance, Correctness & Robustness Items
Derived from: [plan.md](file:///home/narl/dev/private/proton-drive-linux/plan.md), [improvements.md](file:///home/narl/dev/private/proton-drive-linux/improvements.md), & [photos-video-streaming.md](file:///home/narl/dev/private/proton-drive-linux/photos-video-streaming.md)
* **B6. Debounce the LRU access touch**: Keep last-touch times in memory and flush them in batches (debounce window of 30-60 seconds) to avoid performing a SQLite `UPDATE` to the `cache_entries` table on every cache hit.
* **E12. Client in-flight cap shape**: Address the lock discrepancy between `MAX_CONCURRENT_BLOCK_DOWNLOADS = 10` (per-file) and `DEFAULT_MAX_INFLIGHT_BLOCKS = 12` (global client-wide). Either raise the global cap, lower the per-file window, or dynamically calculate the window to prevent single files from starving concurrent transfers.
* **Pipeline `enumerate_nodes_detail` (P2.8 / 8)**: Introduce pipelined node enumeration to speed up directory listings.
* **Local ancestor-chain walk (P3.4 / 4)**: Optimize recursive path lookups where full batching is impossible.
* **Name Search for Photos/Videos**: Add search by photo/video name (deferred from Phase 3 gallery implementation).
* **Sequential-Read Prefetch for Media Streaming**: Implement sequential read prefetch for media files to avoid stalls when buffering.
* **Minor sweep items (Phase F)**:
* **F1. `StreamRing.tags` Leak**: Clean up tag entries in `StreamRing.tags` when blocks are evicted. Currently, it only shrinks on `drop_node` (revision mismatch), leading to tag accumulation.
* **F2. Non-saturating bytes subtraction**: Use `saturating_sub` in `lib.rs:382` (`self.bytes -= dropped.len()`) to prevent release mode panics/wrapping if accounting drifts.
* **F3. Short block yields a short read / EOF**: Address the kernel EOF interpretation on short block reads. Either pad the block or fail loudly rather than silently serving short data.
* **F4. `stream_readahead` check-then-spawn race**: Prevent duplicate concurrent readahead tasks for the same block. Check-then-spawn is currently racey.
* **F5. `refresh_blocks` permit lock (SDK)**: Ensure URL refreshing (`refresh_blocks`) does not hold an in-flight block permit, preventing expired URLs from pinning client permits.
* **F6. Uncapped Unix control socket thread spawns**: Replace the unbounded `std::thread::spawn` per accepted socket connection in `run_control_socket` with a bounded pool or handler limits to prevent thread exhaustion.
* **F7. Document temporary cache file safety**: Add explanatory comment to `cache.rs:362` explaining why `with_extension("tmp")` is safe there but not in `store_thumbnail`.
---
## 3. Open & Unverified Bugs
Derived from: [docs/bugs.md](file:///home/narl/dev/private/proton-drive-linux/docs/bugs.md)
* **B2. Unattributed Trash Origin**: Investigate why deleted/moved files are occasionally sent to trash instead of vanishing on older rename operations.
* **B10. GLib Critical Warning on HUD Close**: Assertion failure `g_list_store_remove: assertion '!g_sequence_iter_is_end (it)' failed` when prompt window closes during active launch (`xdg-open`). Needs investigation under `G_DEBUG=fatal-criticals`.
* **B15. Empty-but-listed folders and duplicate UIDs**: Investigate folder-identity issues (e.g. duplicate remote folders like `Music` with different UIDs) and how folders obtain `listed = 1` in the database without matching child rows.
* **B8 (CAPTCHA sign-in bridge)**: Verify a real gated sign-in using Webkit CAPTCHA completion.
* **Draft Revision Upload loop**: Verify SDK-side deletion and retry logic when a conflicting draft revision exists under the same client UID.
* **Optimistic Size Loss on Restart**: Verify that pending uploads retain their optimistic sizes across a daemon restart.
---
## 4. Testing, Verification & Disaster Recovery Drills
Derived from: [testing.md](file:///home/narl/dev/private/proton-drive-linux/testing.md) & [docs/RECOVERY.md](file:///home/narl/dev/private/proton-drive-linux/docs/RECOVERY.md)
* **IPC Memory and Performance Stress Tests (Phase 3)**: Verify memory allocations do not balloon when sending large `UploadPhoto` JSON payloads. Test control socket timeout behaviors under long-running block operations.
* **Fault Injection and Resiliency Tests (Phase 4)**:
* **Network degradation**: Verify that interrupted uploads leave readable staging buffers and result in clean backoff retries.
* **Disk full (ENOSPC)**: Verify that write staging and gap-filling report correct errors to FUSE and preserve staged work.
* **Concurrent reader starvation**: Test that parallel sequential reads do not freeze directory lookups or metadata handlers.
* **Disaster Recovery Drill (P5 / Phase 5)**: Execute a complete manual disaster recovery test:
1. Register a device.
2. Sync a folder and upload data.
3. Wipe the local state directory and SQLite cache.
4. Restart the daemon.
5. Run `pdfs sync restore` and assert the recovered filesystem matches the source byte-for-byte.
---
## 5. Fixed & Resolved Bugs
Derived from: [docs/bugs.md](file:///home/narl/dev/private/proton-drive-linux/docs/bugs.md)
### Completely Fixed & Verified
* **B1 — FUSE rename loses the file (data loss)**: In `rename`, the node's database row was deleted while only updating the in-memory state, causing the file to vanish on next listing sync. Fixed by correctly calling `st.invalidate_listing(newparent)` to refresh SQLite cache states.
* **B4 — `invalidate_listing` silently skipped non-resident folders**: Folders not loaded in memory were being skipped during listing invalidation, leaving stale cache flags in the DB. Fixed by removing the residency check guard.
* **B5 — `ls -l` costs a network round trip per file (thumbnail xattr probes)**: File listings triggered blocking network checks for thumbnails on unsupported files. Fixed by caching negative thumbnail results and restricting xattr advertisement to supported image/media files.
* **B6 — Daemon sets no file modes: `control.sock` is an unguarded authority**: The Unix control socket was created without restricting file permissions, exposing daemon controls to other local users. Fixed by applying proper creation permission masks.
* **B7 — renamed directory reads as missing until the entry TTL expires**: Kernel cache maps remained stale after a FUSE directory re-anchor. Fixed by proactively sending entry invalidation notifications to the kernel.
* **B9 — Enter did not open the selected result in the launcher**: Hooked up keyboard execution (Enter key) in `pdfs-prompt` search HUD to launch the selected item using `xdg-open`.
* **B11 — a file moved while its create is still queued reads as empty**: Reconciling a local rename on a file that had not finished uploading yet uploaded empty blocks. Fixed by carrying over and staging the correct logical file size.
* **B12 — cold enumeration is slow per entry, and goes superlinear past ~500**: S2K decryption and thumbnail resolution loops were serial. Fixed by parallelizing key derivations and caching metadata.
* **B13 — `rename` over an existing destination fails instead of replacing it**: Fixed POSIX rename target overwrite behavior to replace existing destinations.
* **B14 — provisional (ciphertext) sizes make rsync read short and abort the file**: Adjusted size reporting to align logical and ciphertext sizing, preventing premature EOF.
### Code Fixed / Implemented (Pending Verification)
* **B8 — no way to complete a CAPTCHA, so a gated sign-in is unrecoverable**: Implemented a Webkit-based bridge to prompt the user to resolve interactive CAPTCHAs during gated logins. (Needs verification on a real gate request).
### Closed / Not a Bug
* **B3 — Activity log timestamps written in seconds, read as milliseconds**: Closed as investigator error (timestamps are consistent).
---
## 6. Recently Implemented Features
### Open-for-Write Deferral for Mirror Sync (Implemented)
**File**: [`sync.rs`](file:///home/narl/dev/private/proton-drive-linux/crates/pdfs-fuse/src/sync.rs)
The mirror folder sync engine now defers uploading any file that is currently held open for writing by another process, matching the guarantee the FUSE mount path already provides (where uploads are deferred until `close(fd)`).
**Problem**: Previously, the mirror sync path relied solely on a 2-second trailing-edge debounce (with a 30-second ceiling) to avoid uploading files mid-write. This was insufficient for slow continuous writers (e.g., database dumps, large exports, or editors that keep files open for extended periods), which could have their incomplete state uploaded as a real revision.
**Solution**: Before each reconcile pass, the sync engine scans `/proc/*/fd` once to build a set of canonical paths within the sync root that any process holds open for writing (`O_WRONLY` or `O_RDWR`). Files in this set are:
- **Kept in the local walk** — so they are not misclassified as deletions
- **Treated as unchanged** — so no upload is queued for them
- **Counted as `deferred`** — so the activity summary reports them (e.g., "3 uploaded, 1 deferred (open for write)")
- **Picked up on the next pass** — after the writer has closed the file
**How it works**:
1. `open_for_write_set(root)` reads `/proc/*/fd` → `readlink` for each fd → prefix-checks against the sync root → reads `/proc/<pid>/fdinfo/<n>` to check the `flags:` line's low two bits (O_ACCMODE)
2. `walk_local` sets `LocalItem.open_for_write = true` for matching files
3. Both `do_reconcile` and `push_pass` skip upload classification for flagged files
4. `Outcome.deferred` tracks the count for the activity summary
**Performance**: The `/proc` scan completes in low single-digit milliseconds on a typical desktop. Only fds whose `readlink` target falls under the sync root incur the `fdinfo` read.
**Tests added**: 8 unit tests covering `is_write_mode` parsing of various fdinfo flag combinations, and `Outcome` summary formatting with deferred counts.
| Sync Path | Open-for-write protection | Mechanism |
|---|---|---|
| FUSE mount | ✅ Perfect | Upload deferred until `close(fd)` |
| Mirror folders | ✅ Perfect | `/proc/*/fd` scan per reconcile pass |
+55
View File
@@ -0,0 +1,55 @@
# Testing
The normal Rust suite exercises the metadata database, sync planner, offline
queue, write staging, cache, and FUSE handler state without requiring a Proton
account:
```console
cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings
```
## Live FUSE acceptance suite
Kernel behavior and remote convergence need a real mount. The live suite runs
destructive POSIX operations only below a fresh `pdfs-acceptance-*` directory.
It refuses paths that are not FUSE mounts and removes its directory on success,
failure, or interruption.
Use a dedicated test account without irreplaceable data:
```console
scripts/fuse-acceptance.sh /mnt/on-demand-testmount
```
Passing more paths runs the same filesystem suite independently against each.
The first path must be the FUSE mount under test; later paths may be another
FUSE mount or a normal local mirror filesystem:
```console
PDFS_ACCEPTANCE_SYNC_TIMEOUT=180 \
scripts/fuse-acceptance.sh /mnt/on-demand-testmount /mnt/testmount
```
If two paths really are views of the **same remote folder**, enable an
additional byte-for-byte convergence check explicitly:
```console
PDFS_ACCEPTANCE_CONVERGENCE=1 \
scripts/fuse-acceptance.sh /mnt/first-view /mnt/second-view
```
Do not enable that flag merely because two sync folders belong to the same
account. Their configured remote UIDs must be identical.
The suite covers create/read/rewrite/truncate, multi-megabyte binary content,
unaligned random writes, rename replacement, cross-directory moves, important
POSIX error cases, open-then-unlink behavior, concurrent writers, directory
enumeration, and optional cross-mount byte-for-byte convergence. A pass is
strong evidence for those paths, not a blanket data-loss guarantee. Before a
release, also run the recovery drills in `docs/RECOVERY.md` and inspect
`docs/BUGS.md` for live-verification items.
The runner accepts already-mounted paths instead of starting a daemon. This
keeps authentication and keyring setup explicit and prevents it from silently
reusing or replacing the normal application state directory.
Binary file not shown.

After

Width:  |  Height:  |  Size: 91 KiB

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 190 KiB

After

Width:  |  Height:  |  Size: 46 KiB

BIN
View File
Binary file not shown.

Before

Width:  |  Height:  |  Size: 190 KiB

After

Width:  |  Height:  |  Size: 170 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 239 KiB

After

Width:  |  Height:  |  Size: 108 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 52 KiB

+2 -2
View File
@@ -9,13 +9,13 @@
# Build + install: cd packaging && makepkg -fi
pkgname=proton-drive-linux
pkgver=0.5.0
pkgver=0.7.1
pkgrel=1
pkgdesc="Proton Drive client for Linux: FUSE files-on-demand mount, CLI, GTK4 app + tray"
arch=('x86_64')
url="https://github.com/narl/proton-drive-linux"
license=('MIT')
depends=('fuse3' 'gtk4' 'libadwaita')
depends=('fuse3' 'gtk4' 'libadwaita' 'webkitgtk-6.0')
makedepends=('cargo' 'git')
provides=('pdfs')
# Files-on-demand mount + keyring need these at runtime.
+5 -1
View File
@@ -7,8 +7,10 @@
# Match Arch PKGBUILD `!lto` — LTO has broken some GTK/Rust links in practice.
%global _lto_cflags %{nil}
%{!?version: %global version 0.4.0}
Name: proton-drive-linux
Version: 0.4.0
Version: %{version}
Release: 1%{?dist}
Summary: Proton Drive client for Linux (FUSE, CLI, GTK4 app + tray)
License: MIT
@@ -24,11 +26,13 @@ BuildRequires: libadwaita-devel
BuildRequires: libsecret-devel
BuildRequires: dbus-devel
BuildRequires: glib2-devel
BuildRequires: webkitgtk6.0-devel
Requires: fuse3
Requires: gtk4
Requires: libadwaita
Requires: libsecret
Requires: webkitgtk6.0
Requires: xdg-utils
# DE-specific; do not Require a single desktop environment.
+150
View File
@@ -0,0 +1,150 @@
#!/usr/bin/env bash
# Destructive only inside a uniquely named directory created by this script.
set -Eeuo pipefail
usage() {
echo "usage: $0 MOUNTPOINT [MOUNTPOINT ...]" >&2
echo "example: $0 /mnt/on-demand-testmount /mnt/testmount" >&2
exit 2
}
(( $# > 0 )) || usage
command -v python3 >/dev/null || { echo "python3 is required" >&2; exit 2; }
command -v findmnt >/dev/null || { echo "findmnt is required" >&2; exit 2; }
mounts=("$@")
run_id="pdfs-acceptance-$(date +%Y%m%dT%H%M%S)-$$"
roots=()
cleanup() {
local root
for root in "${roots[@]}"; do
[[ -n "$root" && "$root" == */pdfs-acceptance-* ]] || continue
rm -rf -- "$root" 2>/dev/null || true
done
}
trap cleanup EXIT INT TERM
for index in "${!mounts[@]}"; do
mountpoint="${mounts[$index]}"
[[ -d "$mountpoint" ]] || { echo "not a directory: $mountpoint" >&2; exit 2; }
fs_type="$(findmnt -T "$mountpoint" -n -o FSTYPE | head -n 1)"
if (( index == 0 )) && [[ "$fs_type" != fuse* ]]; then
echo "refusing primary non-FUSE path $mountpoint (type: ${fs_type:-unknown})" >&2
exit 2
fi
roots+=("${mountpoint%/}/$run_id")
done
step() { printf '[test] %s\n' "$1"; }
assert_text() {
python3 - "$1" "$2" <<'PY'
import sys
p, expected = sys.argv[1], sys.argv[2].encode()
with open(p, "rb") as f: got = f.read()
if got != expected:
raise AssertionError(
f"{p}: expected {expected!r} ({len(expected)} bytes), "
f"got {got!r} ({len(got)} bytes)"
)
PY
}
run_suite() {
local primary="$1"
printf '[mount] testing %s\n' "${primary%/*}"
mkdir -- "$primary"
step "create, read, overwrite, truncate"
printf 'alpha\n' > "$primary/plain.txt"
assert_text "$primary/plain.txt" $'alpha\n'
printf 'a much longer replacement\n' > "$primary/plain.txt"
truncate -s 4 "$primary/plain.txt"
assert_text "$primary/plain.txt" "a mu"
step "binary round-trip and random-access writes"
python3 - "$primary/binary.dat" <<'PY'
import hashlib, sys
p = sys.argv[1]
data = bytes(range(256)) * 16384
with open(p, "wb") as f: f.write(data)
with open(p, "r+b", buffering=0) as f:
f.seek(4093); f.write(b"boundary-write")
with open(p, "rb") as f: got = f.read()
expected = data[:4093] + b"boundary-write" + data[4107:]
assert got == expected
print(hashlib.sha256(got).hexdigest())
PY
step "rename, move, and replacement semantics"
mkdir -- "$primary/dir-a" "$primary/dir-b"
printf 'source' > "$primary/source"
printf 'victim' > "$primary/victim"
mv -- "$primary/source" "$primary/victim"
assert_text "$primary/victim" "source"
[[ ! -e "$primary/source" ]]
mv -- "$primary/victim" "$primary/dir-a/moved"
assert_text "$primary/dir-a/moved" "source"
step "POSIX error behavior"
if rmdir -- "$primary/dir-a/moved" 2>/dev/null; then echo "rmdir removed a file" >&2; exit 1; fi
if rm -- "$primary/dir-a" 2>/dev/null; then echo "rm removed a directory" >&2; exit 1; fi
if mv -- "$primary/dir-a" "$primary/dir-a/child" 2>/dev/null; then
echo "moving a directory below itself succeeded" >&2; exit 1
fi
step "open file remains readable after unlink"
python3 - "$primary/open-unlink" <<'PY'
import os, sys
p = sys.argv[1]; data = b"held-open\0" * 4096
with open(p, "wb") as f: f.write(data)
f = open(p, "rb", buffering=0); os.unlink(p)
assert not os.path.exists(p) and f.read() == data
f.close()
PY
step "concurrent independent writers"
for i in $(seq 1 12); do
(python3 - "$primary/concurrent-$i" "$i" <<'PY'
import sys
p, seed = sys.argv[1], int(sys.argv[2]); data = bytes([seed]) * (131072 + seed)
with open(p, "wb") as f: f.write(data)
with open(p, "rb") as f: assert f.read() == data
PY
) &
done
wait
step "directory enumeration agrees with stat"
python3 - "$primary" <<'PY'
import os, sys
for name in os.listdir(sys.argv[1]): os.lstat(os.path.join(sys.argv[1], name))
PY
}
for root in "${roots[@]}"; do
run_suite "$root"
done
if [[ "${PDFS_ACCEPTANCE_CONVERGENCE:-0}" == 1 ]] && (( ${#roots[@]} > 1 )); then
step "cross-mount convergence"
primary="${roots[0]}"
expected="$(sha256sum "$primary/binary.dat" | cut -d' ' -f1)"
for secondary in "${roots[@]:1}"; do
deadline=$((SECONDS + ${PDFS_ACCEPTANCE_SYNC_TIMEOUT:-120}))
while (( SECONDS < deadline )); do
if [[ -f "$secondary/binary.dat" ]] &&
[[ "$(sha256sum "$secondary/binary.dat" 2>/dev/null | cut -d' ' -f1)" == "$expected" ]]; then break; fi
sleep 2
done
[[ -f "$secondary/binary.dat" ]] || { echo "did not appear on $secondary" >&2; exit 1; }
[[ "$(sha256sum "$secondary/binary.dat" | cut -d' ' -f1)" == "$expected" ]] || {
echo "content mismatch on $secondary" >&2; exit 1
}
done
fi
step "cleanup"
cleanup
roots=()
echo "PASS: FUSE acceptance suite completed ($run_id)"